real world reverse debugging success story

real world reverse debugging success story

[Michael Snyder]

Hey guys,

Just wanted to make note that I used reverse debugging to
solve a real world gdb bug (the recently submitted one for
dcache.c).

First I found where the lists were cross-linked, then I put
watchpoints on the list elements that were crossed, and ran
backwards.  One watchpoint took me to dcache_alloc, and the
other to dcache_invalidate.

Then I used some bookmarks to look at the state of the list
before and after, and close in on the actual problem (which
was in fact in another function).

Doug rewrote my patch (thanks Doug), but he verified that
the actual bug was exactly what process record revealed it
to be.

Whee!
Michael

Stats: used an 8 million instruction cache, running as a
ring buffer.  Had to record over 80 million instructions
before I tripped the bug.  Saved core file with record log
was 250 megabytes, and reloaded fine.

RE: real world reverse debugging success story

[Marc Khouzam]

[...]
> 
> Stats: used an 8 million instruction cache, running as a
> ring buffer.  Had to record over 80 million instructions
> before I tripped the bug.  Saved core file with record log
> was 250 megabytes, and reloaded fine.

Very impressive!

How was the responsiveness?  I assume you didn't step
over all those instuctions ;-)  So, you must have run the program
and have it be recorded for a while.  I'm wondering if the execution
was annoyingly slow, or if it was ok.

Thanks
 

Re: real world reverse debugging success story

[Michael Snyder]

Marc Khouzam wrote:
> [...]
>> Stats: used an 8 million instruction cache, running as a
>> ring buffer.  Had to record over 80 million instructions
>> before I tripped the bug.  Saved core file with record log
>> was 250 megabytes, and reloaded fine.
> 
> Very impressive!
> 
> How was the responsiveness?  I assume you didn't step
> over all those instuctions ;-)  So, you must have run the program
> and have it be recorded for a while.  I'm wondering if the execution
> was annoyingly slow, or if it was ok.

The record phase was kind of slow, but I'm sure that was
impacted by a very large number of notifications to the effect
that process record would not record some memory because it
could not get the segment register.

Replay was not bad at all, about 15 seconds to get from
"goto-bookmark begin" to "goto-bookmark end".  For 80
million instructions, that's about 5 million insns / sec.

Much faster than a first gen IBM PC, for instance!   ;-)

Re: real world reverse debugging success story

[Michael Snyder]

Michael Snyder wrote:
> Marc Khouzam wrote:
>> [...]
>>> Stats: used an 8 million instruction cache, running as a
>>> ring buffer.  Had to record over 80 million instructions
>>> before I tripped the bug.  Saved core file with record log
>>> was 250 megabytes, and reloaded fine.
>> Very impressive!
>>
>> How was the responsiveness?  I assume you didn't step
>> over all those instuctions ;-)  So, you must have run the program
>> and have it be recorded for a while.  I'm wondering if the execution
>> was annoyingly slow, or if it was ok.
> 
> The record phase was kind of slow, but I'm sure that was
> impacted by a very large number of notifications to the effect
> that process record would not record some memory because it
> could not get the segment register.
> 
> Replay was not bad at all, about 15 seconds to get from
> "goto-bookmark begin" to "goto-bookmark end".  For 80
> million instructions, that's about 5 million insns / sec.
> 
> Much faster than a first gen IBM PC, for instance!   ;-)

Oh, sorry, it was 8 million insns, so about 0.5M / sec.

RE: real world reverse debugging success story

[Marc Khouzam]

 

> -----Original Message-----
> From: Michael Snyder [mailto:msnyder@vmware.com] 
> Sent: November-11-09 8:08 PM
> To: Michael Snyder
> Cc: Marc Khouzam; gdb@sourceware.org
> Subject: Re: real world reverse debugging success story
> 
> Michael Snyder wrote:
> > Marc Khouzam wrote:
> >> [...]
> >>> Stats: used an 8 million instruction cache, running as a
> >>> ring buffer.  Had to record over 80 million instructions
> >>> before I tripped the bug.  Saved core file with record log
> >>> was 250 megabytes, and reloaded fine.
> >> Very impressive!
> >>
> >> How was the responsiveness?  I assume you didn't step
> >> over all those instuctions ;-)  So, you must have run the program
> >> and have it be recorded for a while.  I'm wondering if the 
> execution
> >> was annoyingly slow, or if it was ok.
> > 
> > The record phase was kind of slow, but I'm sure that was
> > impacted by a very large number of notifications to the effect
> > that process record would not record some memory because it
> > could not get the segment register.
> > 
> > Replay was not bad at all, about 15 seconds to get from
> > "goto-bookmark begin" to "goto-bookmark end".  For 80
> > million instructions, that's about 5 million insns / sec.

You know what, it just dawned on me that I never really payed
attention to the replaying spead :-O  I'm glad to hear it is good.

It is really the recording speed that I'm curious about.  I think
right now, this is a big limiting factor (understandably of course),
that would prevent the use of PRecord for real world applications.

I've had one person want to use PRecord to learn about a piece
of software.  They figured that they would record execution up to
a point in the code they know would hit, and then could go backwards
to find a more proper starting point for their investigation.  But
the recording was to slow to allow the application to reach far enough
fast enough.

I was happy to see that Hui had some ideas on how to improve the
recording speed.

This reverse 'thingy' as so much potential! :-)

> > 
> > Much faster than a first gen IBM PC, for instance!   ;-)
> 
> Oh, sorry, it was 8 million insns, so about 0.5M / sec.
> 
> 

Re: real world reverse debugging success story

[Sean Chen]

> The record phase was kind of slow, but I'm sure that was
> impacted by a very large number of notifications to the effect
> that process record would not record some memory because it
> could not get the segment register.

Valuable data. What about your CPU and memory?

How slow is the record phase in your experiment? Minutes?

Thanks.