vmware's replay framework and gdb

vmware's replay framework and gdb

[Edward Peschko]

hey..

I watched the following Google tech talk with much interest:

http://www.youtube.com/watch?v=RvMlihjqlhY

It describes a very generic, scalable, environment-agnostic way of
recording programs' execution via vmware's virtualization, and doing
reverse tracing (which is one of the reasons I'd like to record
programs in the first place). I heard gdb mentioned in the video in
passing, so I'm assuming someone in the list is familiar with vmware's
effort, but how far along is gdb in supporting this framework?

Thanks much,

Ed

reverse trace [was: vmware's replay framework and gdb]

[EBo]

I Cannot comment on VMWares effort, but reading this I had an idea and thought
I would share...

The overall problem with reverse tracing as I see it is one of caching the old
values basically as they change.  One way to get at this is to integrate a SQL
database with transaction support into the trace subsystem.  The you can view
the entire history back and forth.  

So, if you are willing to do a little brainstorming along these lines the
following questions arise:

*) is there some easy way to get the symbol table and integrate that info back
into gdb and associate the variables/memory blocks with each line of code?

*) is there some computationally inexpensive way to trap memory changes and
associate the timing with source lines and execution times?

Once all this information is funneled through a relational database, you can
then either grab the current state of each variable or reconstruct it on the fly.

Just an idea, and I hope this kind of speculative response is considered
acceptable to the group.

  EBo --

Edward Peschko <horos11@gmail.com> said:

> hey..
> 
> I watched the following Google tech talk with much interest:
> 
> http://www.youtube.com/watch?v=RvMlihjqlhY
> 
> It describes a very generic, scalable, environment-agnostic way of
> recording programs' execution via vmware's virtualization, and doing
> reverse tracing (which is one of the reasons I'd like to record
> programs in the first place). I heard gdb mentioned in the video in
> passing, so I'm assuming someone in the list is familiar with vmware's
> effort, but how far along is gdb in supporting this framework?
> 
> Thanks much,
> 
> Ed
> 



-- 

Re: reverse trace [was: vmware's replay framework and gdb]

[Thiago Jung Bauermann]

El mié, 05-11-2008 a las 18:01 -0700, EBo escribió:
> The overall problem with reverse tracing as I see it is one of caching the old
> values basically as they change.  One way to get at this is to integrate a SQL
> database with transaction support into the trace subsystem.  The you can view
> the entire history back and forth.  
<snip>
> Once all this information is funneled through a relational database, you can
> then either grab the current state of each variable or reconstruct it on the fly.
> 
> Just an idea, and I hope this kind of speculative response is considered
> acceptable to the group.

Since we're just brainstorming:

Have you seen Chronicle and Chronomancer? I didn't try them, but from
what I read they seem to go in the direction that you suggest here:

http://code.google.com/p/chronicle-recorder/
http://code.google.com/p/chronomancer/
http://weblogs.mozillazine.org/roc/archives/2007/08/announcing_chro.html
-- 
[]'s
Thiago Jung Bauermann
IBM Linux Technology Center

Re: reverse trace [was: vmware's replay framework and gdb]

[EBo]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain, Size: 1421 bytes --]

Thanks Thiago,

That is so wicked cool!  If this project was really ongoing I would be
interested, but as it is it looks orphaned...  Who's to know though.

Basically what I was thinking would be behind the scenes with gdb:

  * turn reversable tracing on
  * step forward
  * step back

..

  EBo --


Thiago Jung Bauermann <bauerman@br.ibm.com> said:

> El mié, 05-11-2008 a las 18:01 -0700, EBo escribió:
> > The overall problem with reverse tracing as I see it is one of caching the old
> > values basically as they change.  One way to get at this is to integrate a SQL
> > database with transaction support into the trace subsystem.  The you can view
> > the entire history back and forth.  
> <snip>
> > Once all this information is funneled through a relational database, you can
> > then either grab the current state of each variable or reconstruct it on
the fly.
> > 
> > Just an idea, and I hope this kind of speculative response is considered
> > acceptable to the group.
> 
> Since we're just brainstorming:
> 
> Have you seen Chronicle and Chronomancer? I didn't try them, but from
> what I read they seem to go in the direction that you suggest here:
> 
> http://code.google.com/p/chronicle-recorder/
> http://code.google.com/p/chronomancer/
> http://weblogs.mozillazine.org/roc/archives/2007/08/announcing_chro.html
> -- 
> []'s
> Thiago Jung Bauermann
> IBM Linux Technology Center
> 
> 



-- 

RE: reverse trace [was: vmware's replay framework and gdb]

[Jakob Engblom]

> I Cannot comment on VMWares effort, but reading this I had an idea and thought
> I would share...
> 
> The overall problem with reverse tracing as I see it is one of caching the old
> values basically as they change.  One way to get at this is to integrate a SQL
> database with transaction support into the trace subsystem.  The you can view
> the entire history back and forth.

Well, the amonut of data we are talking about is very very large.  Recording a
complete execution trace very quickly eats up memory.  

If you look at what Lauterbach and GreenHills have in their hardware-based
reverse debuggers, it is essentialy that: a recording of execution step by step.
Using some very clever compression techniques, they can put a few hundred
million clock cycles of instructions into a gigabyte-level memory in their trace
boxes.  You need to consider a data need for well over 32 bits per instruction.

Even quite trivial workloads quickly build up tens of gigabytes of data if you
trace all, and that means that you are basically doing something that is more
expensive than high-def video in terms of data rates.

Pushing that into a database is not really feasible for any real world scenario
involving multiple gigahertz-level processors crunching real code and not
idling. 

That's why the solutions to reversing from VmWare and Virtutech both hinge on
deterministic replay rather than complete recording -- it is far cheaper to go
back and reconstruct the state at some point in time than trying to store it. 

If you also include fnu things like memory state, I/O system state changes,
graphics memory, etc. it is clear why this is the only sensible thing to do.
 
> So, if you are willing to do a little brainstorming along these lines the
> following questions arise:
> 
> *) is there some easy way to get the symbol table and integrate that info back
> into gdb and associate the variables/memory blocks with each line of code?
> 
> *) is there some computationally inexpensive way to trap memory changes and
> associate the timing with source lines and execution times?
> 
> Once all this information is funneled through a relational database, you can
> then either grab the current state of each variable or reconstruct it on the
fly.
> 
> Just an idea, and I hope this kind of speculative response is considered
> acceptable to the group.

Have been thinking along he same lines, but the volume really is a killer here.


/jakob
 

RE: reverse trace [was: vmware's replay framework and gdb]

[EBo]

One of the reasons I was thinking of relational databases was that they can
deal with the volume and put whatever the decide to cache into dis instead of
memory (but maybe you were thinking disk space when you said memory anyway).  

Besides compression of data stream there is also the possibility of only
caching at certain sync points and tracing between sync points you have to
regenerate the temp variables, etc.  Like I said, I was just brain storming
here.  I had no idea how far things had gotten along the lines of
Chronomancer/Chronicle.

Regardless of how it eventually ends up working, it is going to either use a
LOT of space, or a lot of CPU power to recreate.  

Cheers,

  EBo --

Jakob Engblom <jakob@virtutech.com> said:

> Well, the amonut of data we are talking about is very very large.  Recording a
> complete execution trace very quickly eats up memory.  
>
> If you look at what Lauterbach and GreenHills have in their hardware-based
> reverse debuggers, it is essentialy that: a recording of execution step by step.
> Using some very clever compression techniques, they can put a few hundred
> million clock cycles of instructions into a gigabyte-level memory in their trace
> boxes.  You need to consider a data need for well over 32 bits per instruction.
> 
> Even quite trivial workloads quickly build up tens of gigabytes of data if you
> trace all, and that means that you are basically doing something that is more
> expensive than high-def video in terms of data rates.
> 
> Pushing that into a database is not really feasible for any real world scenario
> involving multiple gigahertz-level processors crunching real code and not
> idling. 
> 
> That's why the solutions to reversing from VmWare and Virtutech both hinge on
> deterministic replay rather than complete recording -- it is far cheaper to go
> back and reconstruct the state at some point in time than trying to store it. 
> 
> If you also include fnu things like memory state, I/O system state changes,
> graphics memory, etc. it is clear why this is the only sensible thing to do.
>  
> > So, if you are willing to do a little brainstorming along these lines the
> > following questions arise:
> > 
> > *) is there some easy way to get the symbol table and integrate that info back
> > into gdb and associate the variables/memory blocks with each line of code?
> > 
> > *) is there some computationally inexpensive way to trap memory changes and
> > associate the timing with source lines and execution times?
> > 
> > Once all this information is funneled through a relational database, you can
> > then either grab the current state of each variable or reconstruct it on the
> fly.
> > 
> > Just an idea, and I hope this kind of speculative response is considered
> > acceptable to the group.
> 
> Have been thinking along he same lines, but the volume really is a killer here.
> 
> 
> /jakob
>  
> 
> 
> 



-- 

RE: reverse trace [was: vmware's replay framework and gdb]

[Jakob Engblom]

> One of the reasons I was thinking of relational databases was that they can
> deal with the volume and put whatever the decide to cache into dis instead of
> memory (but maybe you were thinking disk space when you said memory anyway).

RAM, disk, does not matter -- size is enormous either case. 
 
> Besides compression of data stream there is also the possibility of only
> caching at certain sync points and tracing between sync points you have to
> regenerate the temp variables, etc.  Like I said, I was just brain storming
> here.  I had no idea how far things had gotten along the lines of
> Chronomancer/Chronicle.
> 
> Regardless of how it eventually ends up working, it is going to either use a
> LOT of space, or a lot of CPU power to recreate.

The CPU power is actually easier to handle, especially if you have a few
snapshots to return to along the way.  We have looked at this hard, and the
execution to recreate is much easier to make scalable than the complete trace
approach.

/jakob

Re: vmware's replay framework and gdb

[Michael Snyder]

Edward Peschko wrote:
 > hey..
 >
 > I watched the following Google tech talk with much interest:
 >
 > http://www.youtube.com/watch?v=RvMlihjqlhY
 >
 > It describes a very generic, scalable, environment-agnostic way of
 > recording programs' execution via vmware's virtualization, and doing
 > reverse tracing (which is one of the reasons I'd like to record
 > programs in the first place). I heard gdb mentioned in the video in
 > passing, so I'm assuming someone in the list is familiar with vmware's
 > effort,

Me -- I'm a member of the replay debugging team at VMware,
and a public gdb maintainer, and I was at that tech talk.

 > but how far along is gdb in supporting this framework?

I'm working on it.   ;-)

If you look at the recent threads on this list about reverse
debugging in gdb, you'll get an idea of where we're headed.
We want to let a user debug a recorded execution, deterministic-
ally, and make it run both forward and backward using the same
reverse debugging commands as described in those threads
(reverse-continue, reverse-step, reverse-next etc.)

We don't have any product release date for it, but you can
already do some rudimentary replay debugging with the
released version of VMware Worksatation and the standard gdb.
Here's a blog article about it:

http://stackframe.blogspot.com/

RE: vmware's replay framework and gdb

[Jakob Engblom]

>  > but how far along is gdb in supporting this framework?
> 
> I'm working on it.   ;-)
> 
> If you look at the recent threads on this list about reverse
> debugging in gdb, you'll get an idea of where we're headed.
> We want to let a user debug a recorded execution, deterministic-
> ally, and make it run both forward and backward using the same
> reverse debugging commands as described in those threads
> (reverse-continue, reverse-step, reverse-next etc.)
> 
> We don't have any product release date for it, but you can
> already do some rudimentary replay debugging with the
> released version of VMware Worksatation and the standard gdb.
> Here's a blog article about it:
> 
> http://stackframe.blogspot.com/

Not to be outdone: Simics, at www.virtutech.com, also has support for reverse
debug with gdb, using our own patched version of gdb 6.8.  This has been in use
since 2005. 

Simics is free for academia, if you are an academic, and we are also quite happy
to provide access to Simics licenses to the gdb maintainers. We are looking into
how to integrate the upcoming reversible gdb with our backend, looks fairly
similar. 


Best regards,

/jakob

_______________________________________________________

Jakob Engblom, PhD, Technical Marketing Manager

Virtutech                   Direct: +46 8 690 07 47    
Drottningholmsvägen 14      Mobile: +46 709 242 646   
11243 Stockholm             Web:    www.virtutech.com  
Sweden
________________________________________________________