[FYI] tutorial for process record and reverse debugging

[FYI] tutorial for process record and reverse debugging

[Michael Snyder]

FYI, there is now a tutorial for process record/replay and reverse 
debugging on the gdb wiki:

http://sourceware.org/gdb/wiki/ProcessRecord/Tutorial

Re: [FYI] tutorial for process record and reverse debugging

[Hui Zhu]

It's really clear.  Thanks a lot.  :)

Hui

On Sun, Oct 18, 2009 at 06:57, Michael Snyder <msnyder@vmware.com> wrote:
> FYI, there is now a tutorial for process record/replay and reverse debugging
> on the gdb wiki:
>
> http://sourceware.org/gdb/wiki/ProcessRecord/Tutorial
>
>

RE: [FYI] tutorial for process record and reverse debugging

[Marc Khouzam]

> -----Original Message-----
> From: gdb-owner@sourceware.org 
> [mailto:gdb-owner@sourceware.org] On Behalf Of Michael Snyder
> Sent: Saturday, October 17, 2009 6:57 PM
> To: gdb@sourceware.org
> Subject: [FYI] tutorial for process record and reverse debugging
> 
> FYI, there is now a tutorial for process record/replay and reverse 
> debugging on the gdb wiki:
> 
> http://sourceware.org/gdb/wiki/ProcessRecord/Tutorial

Very clearly written.  It should be very useful.

I even learned that the following was a wanted behavior:

"We can go directly to the point at which the recording currently ends, by disabling 
all our breakpoints and then telling gdb to continue. Process record will replay 
until it reaches the current end of the recording, and then stop."

To be honest, I never cared much for this behavior because from an Eclipse 
user-experience, it is not very clear why the execution/replay suddenly stops in 
this case.

But from reading the tutorial I see that it may be of interest in some situations.
What bothers me though is that one must disable all breakpoints and then re-enable 
them.  This can be a bit of an annoyance, especially if some bps were already disabled.
I got to wonder if this doesn't call for a new command; something like
"record live", which would take us to the end of the recording while not needing the
user to deal with existing breakpoints.

What do you think?

Marc

Re: [FYI] tutorial for process record and reverse debugging

[Hui Zhu]

> But from reading the tutorial I see that it may be of interest in some situations.
> What bothers me though is that one must disable all breakpoints and then re-enable
> them.  This can be a bit of an annoyance, especially if some bps were already disabled.
> I got to wonder if this doesn't call for a new command; something like
> "record live", which would take us to the end of the recording while not needing the
> user to deal with existing breakpoints.
>
> What do you think?
>
> Marc
>

Sorry Marc,  I cannot understand it.   Could you please tell me which
status need "disable all breakpoints and then re-enable them"?

Thanks,
Hui

RE: [FYI] tutorial for process record and reverse debugging

[Marc Khouzam]

> -----Original Message-----
> From: gdb-owner@sourceware.org 
> [mailto:gdb-owner@sourceware.org] On Behalf Of Hui Zhu
> Sent: Monday, October 19, 2009 8:57 AM
> To: Marc Khouzam
> Cc: Michael Snyder; gdb@sourceware.org
> Subject: Re: [FYI] tutorial for process record and reverse debugging
> 
> > But from reading the tutorial I see that it may be of 
> interest in some situations.
> > What bothers me though is that one must disable all 
> breakpoints and then re-enable
> > them.  This can be a bit of an annoyance, especially if 
> some bps were already disabled.
> > I got to wonder if this doesn't call for a new command; 
> something like
> > "record live", which would take us to the end of the 
> recording while not needing the
> > user to deal with existing breakpoints.
> >
> > What do you think?
> >
> > Marc
> >
> 
> Sorry Marc,  I cannot understand it.   Could you please tell me which
> status need "disable all breakpoints and then re-enable them"?

In the tutorial, around the middle of section 
"Continue, reverse-continue, and breakpoints"
it mentions that after doing some reverse execution, we may be
"now ready to resume where we left off".

From what I understand this means that the user may decide that she
is done with reverse execution and wants to go back to executing
the live program.  To do this, we need to disable all bps, do a
'continue' and then (optionally) re-enable all bps.

If going straight to the end of the recording log is a valuable
feature, I was thinking it may deserve its own command.

Ultimately, I'm hoping that with such a command, we can then see
if we can have PRecord directly jump from recorded execution
to live execution without stopping.

I'm hoping to get opinions from you experts, as I'm not sure it is
a good idea.

Marc

Re: [FYI] tutorial for process record and reverse debugging

[Hui Zhu]

Thanks Marc, I think I got you mean.  You mean is when the user don't
want keep play in replay mode, we give him a way to go the the record
mode directly.  I think this idea is not bad.

Michael, what do you think about it?

Thanks,
Hui

On Mon, Oct 19, 2009 at 21:06, Marc Khouzam <marc.khouzam@ericsson.com> wrote:
>> -----Original Message-----
>> From: gdb-owner@sourceware.org
>> [mailto:gdb-owner@sourceware.org] On Behalf Of Hui Zhu
>> Sent: Monday, October 19, 2009 8:57 AM
>> To: Marc Khouzam
>> Cc: Michael Snyder; gdb@sourceware.org
>> Subject: Re: [FYI] tutorial for process record and reverse debugging
>>
>> > But from reading the tutorial I see that it may be of
>> interest in some situations.
>> > What bothers me though is that one must disable all
>> breakpoints and then re-enable
>> > them.  This can be a bit of an annoyance, especially if
>> some bps were already disabled.
>> > I got to wonder if this doesn't call for a new command;
>> something like
>> > "record live", which would take us to the end of the
>> recording while not needing the
>> > user to deal with existing breakpoints.
>> >
>> > What do you think?
>> >
>> > Marc
>> >
>>
>> Sorry Marc,  I cannot understand it.   Could you please tell me which
>> status need "disable all breakpoints and then re-enable them"?
>
> In the tutorial, around the middle of section
> "Continue, reverse-continue, and breakpoints"
> it mentions that after doing some reverse execution, we may be
> "now ready to resume where we left off".
>
> From what I understand this means that the user may decide that she
> is done with reverse execution and wants to go back to executing
> the live program.  To do this, we need to disable all bps, do a
> 'continue' and then (optionally) re-enable all bps.
>
> If going straight to the end of the recording log is a valuable
> feature, I was thinking it may deserve its own command.
>
> Ultimately, I'm hoping that with such a command, we can then see
> if we can have PRecord directly jump from recorded execution
> to live execution without stopping.
>
> I'm hoping to get opinions from you experts, as I'm not sure it is
> a good idea.
>
> Marc
>

Re: [FYI] tutorial for process record and reverse debugging

[Michael Snyder]

Marc Khouzam wrote:
>> -----Original Message-----
>> From: gdb-owner@sourceware.org 
>> [mailto:gdb-owner@sourceware.org] On Behalf Of Michael Snyder
>> Sent: Saturday, October 17, 2009 6:57 PM
>> To: gdb@sourceware.org
>> Subject: [FYI] tutorial for process record and reverse debugging
>>
>> FYI, there is now a tutorial for process record/replay and reverse 
>> debugging on the gdb wiki:
>>
>> http://sourceware.org/gdb/wiki/ProcessRecord/Tutorial
> 
> Very clearly written.  It should be very useful.
> 
> I even learned that the following was a wanted behavior:
> 
> "We can go directly to the point at which the recording currently ends, by disabling 
> all our breakpoints and then telling gdb to continue. Process record will replay 
> until it reaches the current end of the recording, and then stop."

I don't know how well thought-out the current behavior is.
It could be worth discussing.  What I described in my tutorial
is the behavior we've got, not necessarily the behavior we set out
to achieve.

> To be honest, I never cared much for this behavior because from an Eclipse 
> user-experience, it is not very clear why the execution/replay suddenly stops in 
> this case.

OK, well, let's talk about it.

Seems to me that if you're replaying, and you reach the end of the
recorded log, at least one natural thing to do is to stop.  Could be
that another natural thing to do would be to automatically switch
from replay mode to record mode and keep going -- but I'm not sure
if I could safely assume that that was the user's desire.

Sounds like a case for a query, and/or a user-settable mode.

It sometimes seems to me that we have too many queries already,
but maybe we can keep them under control with user-settable modes.

Helpful to talk about them in advance, though...

> But from reading the tutorial I see that it may be of interest in some situations.
> What bothers me though is that one must disable all breakpoints and then re-enable 
> them.  This can be a bit of an annoyance, especially if some bps were already disabled.
> I got to wonder if this doesn't call for a new command; something like
> "record live", which would take us to the end of the recording while not needing the
> user to deal with existing breakpoints.

Yeah, the "disable all breakpoints and continue" business was something
that I just came up with on the spot.

However, rather than such a specific command as you describe, I had been
thinking about a more general alternative, which in my mind I have been
referring to by the code name "goto".  As in:

   (gdb) record goto end
   (gdb) record goto beginning
   (gdb) record goto instruction 12345
   (gdb) record goto bookmark 7

Easy enough to make this command ignore breakpoints watchpoints etc.

Re: [FYI] tutorial for process record and reverse debugging

[Michael Snyder]

Marc Khouzam wrote:
>> -----Original Message-----
>> From: gdb-owner@sourceware.org 
>> [mailto:gdb-owner@sourceware.org] On Behalf Of Hui Zhu
>> Sent: Monday, October 19, 2009 8:57 AM
>> To: Marc Khouzam
>> Cc: Michael Snyder; gdb@sourceware.org
>> Subject: Re: [FYI] tutorial for process record and reverse debugging
>>
>>> But from reading the tutorial I see that it may be of 
>> interest in some situations.
>>> What bothers me though is that one must disable all 
>> breakpoints and then re-enable
>>> them.  This can be a bit of an annoyance, especially if 
>> some bps were already disabled.
>>> I got to wonder if this doesn't call for a new command; 
>> something like
>>> "record live", which would take us to the end of the 
>> recording while not needing the
>>> user to deal with existing breakpoints.
>>>
>>> What do you think?
>>>
>>> Marc
>>>
>> Sorry Marc,  I cannot understand it.   Could you please tell me which
>> status need "disable all breakpoints and then re-enable them"?
> 
> In the tutorial, around the middle of section 
> "Continue, reverse-continue, and breakpoints"
> it mentions that after doing some reverse execution, we may be
> "now ready to resume where we left off".
> 
> From what I understand this means that the user may decide that she
> is done with reverse execution and wants to go back to executing
> the live program.  To do this, we need to disable all bps, do a
> 'continue' and then (optionally) re-enable all bps.
> 
> If going straight to the end of the recording log is a valuable
> feature, I was thinking it may deserve its own command.
> 
> Ultimately, I'm hoping that with such a command, we can then see
> if we can have PRecord directly jump from recorded execution
> to live execution without stopping.
> 
> I'm hoping to get opinions from you experts, as I'm not sure it is
> a good idea.

OK, so good discussion.  Let's cover some bases here.

1) I'm in record mode, and I want to stay in record mode.
No brainer -- that's the default behavior.

2) I'm in record mode, and I want to go to replay mode.
Currently the only way to do that is to give a "reverse"
command (reverse step, reverse continue...)

That's not too bad, but sometimes I might want to simply
go to the beginning of the log and start replaying forward
from the beginning (ie. not backwards from the end.
Or, I might even want to goto the middle before I start
to replay (in either direction).

We can do that now by using breakpoints, but we might have
to disable other breakpoints, if there are any.

But we COULD do it if we had a command like "goto beginning",
or "goto bookmark 12".

3) I'm in replay mode, possibly in the middle of the recording,
and I want to switch to record mode.  Now there are several
branching possibilities:  Do I want to:

   a) Go to the end and start appending to the existing log?
   b) Truncate the existing log at the point where I am, and
      start appending to the prefix?
   c) Discard the existing log and start a new log from the
      point where I am?
   d) Go back to the beginning, discard the existing log, and
      start a new log from the beginning?

These are all things that I think we can do at present, but
not generally with a single command.  Maybe we should kick
them around.  They're all useful, and not far out of reach.
They wouldn't, for instance, require that we keep several
branching logs around at the same time (although just think
what would be possible if we could do that!!! <g>)

Re: [FYI] tutorial for process record and reverse debugging

[Michael Snyder]

Hui Zhu wrote:
> Thanks Marc, I think I got you mean.  You mean is when the user don't
> want keep play in replay mode, we give him a way to go the the record
> mode directly.  I think this idea is not bad.
> 
> Michael, what do you think about it?

Been thinking about it all along -- just waiting to get some more
urgent issues handled first.  ;-)

See my reply to Marc.

RE: [FYI] tutorial for process record and reverse debugging

[Marc Khouzam]

 
>    (gdb) record goto end
>    (gdb) record goto beginning
>    (gdb) record goto instruction 12345
>    (gdb) record goto bookmark 7
> 
> Easy enough to make this command ignore breakpoints watchpoints etc.

This sounds neat.  Seems like it has a lot of potential.
 

> > In the tutorial, around the middle of section 
> > "Continue, reverse-continue, and breakpoints"
> > it mentions that after doing some reverse execution, we may be
> > "now ready to resume where we left off".
> > 
> > From what I understand this means that the user may decide that she
> > is done with reverse execution and wants to go back to executing
> > the live program.  To do this, we need to disable all bps, do a
> > 'continue' and then (optionally) re-enable all bps.
> > 
> > If going straight to the end of the recording log is a valuable
> > feature, I was thinking it may deserve its own command.
> > 
> > Ultimately, I'm hoping that with such a command, we can then see
> > if we can have PRecord directly jump from recorded execution
> > to live execution without stopping.
> > 
> > I'm hoping to get opinions from you experts, as I'm not sure it is
> > a good idea.
> 
> OK, so good discussion.  Let's cover some bases here.
> 
> 1) I'm in record mode, and I want to stay in record mode.
> No brainer -- that's the default behavior.
> 
> 2) I'm in record mode, and I want to go to replay mode.
> Currently the only way to do that is to give a "reverse"
> command (reverse step, reverse continue...)
> 
> That's not too bad, but sometimes I might want to simply
> go to the beginning of the log and start replaying forward
> from the beginning (ie. not backwards from the end.
> Or, I might even want to goto the middle before I start
> to replay (in either direction).
> 
> We can do that now by using breakpoints, but we might have
> to disable other breakpoints, if there are any.

And for long executions, jumping in the recorded log is probably
faster than using breakpoints.  I agree that this seem valuable.

> But we COULD do it if we had a command like "goto beginning",
> or "goto bookmark 12".

Again, this seems neat.  I do think it is somewhat of an
'advanced' feature, as it requires more understanding of PRecord
than using breakpoints and reverse-continue/reverse-step/etc

> 3) I'm in replay mode, possibly in the middle of the recording,
> and I want to switch to record mode.  Now there are several
> branching possibilities:  Do I want to:
> 
>    a) Go to the end and start appending to the existing log?

I can understand someone wanting this.

>    b) Truncate the existing log at the point where I am, and
>       start appending to the prefix?

I never thought of this case.  I see now that for non-deterministic
executions this could have value.

>    c) Discard the existing log and start a new log from the
>       point where I am?

I think this one is simply to re-issue the 'record' command.
Also, besides saving some space, I don't really see a big value
compared to point b) above.

>    d) Go back to the beginning, discard the existing log, and
>       start a new log from the beginning?

I think this is
run
record

In Eclipse we have a 'restart' button that calls the 'run' command
and we have a way to automatically turn on 'record' if the user wants.
So that is one feature the frontend can do without gdb support
(I don't believe that is the case for any other scenarios you brought
up).
 
> These are all things that I think we can do at present, but
> not generally with a single command.  Maybe we should kick
> them around.  They're all useful, and not far out of reach.
> They wouldn't, for instance, require that we keep several
> branching logs around at the same time (although just think
> what would be possible if we could do that!!! <g>)

Wow, that would be something to see.  The UI stuff might
be tricky though.

Now, let me describe the case I am imagining.
It is as simple as it gets.
The user simply enables the 'reverse debugging' feature.
After that, the user should not need to pay attention to
record logs and such.  What they should see is that they
can go forward or backwards as if everything was true 'execution'.
We don't need to differentiate between 'execution' and 'replay'.

For example, when changing memory, the user doesn't need to know
that we are moving away from replay into a new execution.  All 
they see is that the program moves forward with the new memory
value.

And that is why, in this scenario, I thought it seemed
unintuitive to stop execution when
arriving at the end of the replay log; instead, the user
pressed 'continue' and the 'execution' should continue until
a breakpoint or the end of the program, as if a true execution.

The only limitation to this, is that we cannot go backwards
past the start of the recording.  But I think this can be easily
understood by the user.

I don't think this scenario is good for everyone, but I think
for average users, it makes reverse debugging very fluid.

Thoughts?

Marc

Re: [FYI] tutorial for process record and reverse debugging

[Michael Snyder]

Marc Khouzam wrote:

>> OK, so good discussion.  Let's cover some bases here.
>>
>> 1) I'm in record mode, and I want to stay in record mode.
>> No brainer -- that's the default behavior.
>>
>> 2) I'm in record mode, and I want to go to replay mode.
>> Currently the only way to do that is to give a "reverse"
>> command (reverse step, reverse continue...)
>>
>> That's not too bad, but sometimes I might want to simply
>> go to the beginning of the log and start replaying forward
>> from the beginning (ie. not backwards from the end.
>> Or, I might even want to goto the middle before I start
>> to replay (in either direction).
>>
>> We can do that now by using breakpoints, but we might have
>> to disable other breakpoints, if there are any.
> 
> And for long executions, jumping in the recorded log is probably
> faster than using breakpoints.  I agree that this seem valuable.
> 
>> But we COULD do it if we had a command like "goto beginning",
>> or "goto bookmark 12".
> 
> Again, this seems neat.  I do think it is somewhat of an
> 'advanced' feature, as it requires more understanding of PRecord
> than using breakpoints and reverse-continue/reverse-step/etc
> 
>> 3) I'm in replay mode, possibly in the middle of the recording,
>> and I want to switch to record mode.  Now there are several
>> branching possibilities:  Do I want to:
>>
>>    a) Go to the end and start appending to the existing log?
> 
> I can understand someone wanting this.
> 
>>    b) Truncate the existing log at the point where I am, and
>>       start appending to the prefix?
> 
> I never thought of this case.  I see now that for non-deterministic
> executions this could have value.

Not just that, though.  This is also what happens if we
change a memory or register value, eg. a variable that
controls a conditional branch.  We auto-delete the trailing
part of the execution log, because now we're going to go
forward in a different direction.


>>    c) Discard the existing log and start a new log from the
>>       point where I am?
> 
> I think this one is simply to re-issue the 'record' command.
> Also, besides saving some space, I don't really see a big value
> compared to point b) above.

It's a minor case (because it's easy).  I'm just being
exhaustive.

[...]
> Now, let me describe the case I am imagining.
> It is as simple as it gets.
> The user simply enables the 'reverse debugging' feature.
> After that, the user should not need to pay attention to
> record logs and such.  What they should see is that they
> can go forward or backwards as if everything was true 'execution'.
> We don't need to differentiate between 'execution' and 'replay'.
> 
> For example, when changing memory, the user doesn't need to know
> that we are moving away from replay into a new execution.  All 
> they see is that the program moves forward with the new memory
> value.
> 
> And that is why, in this scenario, I thought it seemed
> unintuitive to stop execution when
> arriving at the end of the replay log; instead, the user
> pressed 'continue' and the 'execution' should continue until
> a breakpoint or the end of the program, as if a true execution.
> 
> The only limitation to this, is that we cannot go backwards
> past the start of the recording.  But I think this can be easily
> understood by the user.
> 
> I don't think this scenario is good for everyone, but I think
> for average users, it makes reverse debugging very fluid.

I think that's a great scenario -- just not the only scenario.
We could call that Marc-mode, for devel purposes.   ;-)

How would you suggest we might turn on Marc-mode with a
single command?

Or do you imagine it being the default?

Re: [FYI] tutorial for process record and reverse debugging

[Greg Law]

Michael Snyder wrote:
> [...]
>>
>>> 3) I'm in replay mode, possibly in the middle of the recording,
>>> and I want to switch to record mode.  Now there are several
>>> branching possibilities:  Do I want to:
>>>
>>>    a) Go to the end and start appending to the existing log?
>>
>> I can understand someone wanting this.
>>
>>>    b) Truncate the existing log at the point where I am, and
>>>       start appending to the prefix?
>>
>> I never thought of this case.  I see now that for non-deterministic
>> executions this could have value.
> 
> Not just that, though.  This is also what happens if we
> change a memory or register value, eg. a variable that
> controls a conditional branch.  We auto-delete the trailing
> part of the execution log, because now we're going to go
> forward in a different direction.

But what would we do about "external state"?

Example, say I'm debugging a web-server, and half way through a 
connection I go back, then truncate the log and start again in a new 
direction.  And clients "mid-session" with the webserver are liable to 
get mighty confused.


> 
> 
>>>    c) Discard the existing log and start a new log from the
>>>       point where I am?
>>
>> I think this one is simply to re-issue the 'record' command.
>> Also, besides saving some space, I don't really see a big value
>> compared to point b) above.
> 
> It's a minor case (because it's easy).  I'm just being
> exhaustive.
> 
> [...]
>> Now, let me describe the case I am imagining.
>> It is as simple as it gets.
>> The user simply enables the 'reverse debugging' feature.
>> After that, the user should not need to pay attention to
>> record logs and such.  What they should see is that they
>> can go forward or backwards as if everything was true 'execution'.
>> We don't need to differentiate between 'execution' and 'replay'.
>>
>> For example, when changing memory, the user doesn't need to know
>> that we are moving away from replay into a new execution.  All they 
>> see is that the program moves forward with the new memory
>> value.
>>
>> And that is why, in this scenario, I thought it seemed
>> unintuitive to stop execution when
>> arriving at the end of the replay log; instead, the user
>> pressed 'continue' and the 'execution' should continue until
>> a breakpoint or the end of the program, as if a true execution.
>>
>> The only limitation to this, is that we cannot go backwards
>> past the start of the recording.  But I think this can be easily
>> understood by the user.
>>
>> I don't think this scenario is good for everyone, but I think
>> for average users, it makes reverse debugging very fluid.
> 
> I think that's a great scenario -- just not the only scenario.
> We could call that Marc-mode, for devel purposes.   ;-)
> 
> How would you suggest we might turn on Marc-mode with a
> single command?
> 
> Or do you imagine it being the default?
> 

FWIW, early versions of UndoDB operated in "Marc-mode".  We changed it 
because replay mode and record are quite different, particularly w.r.t. 
to the program's interaction with the outside world. "Silent" transition 
from replay to record mode could be quite confusing/surprising.

Greg

-- 
Greg Law, Undo Software                       http://undo-software.com/

RE: [FYI] tutorial for process record and reverse debugging

[Marc Khouzam]

 

> -----Original Message-----
> From: gdb-owner@sourceware.org 
> [mailto:gdb-owner@sourceware.org] On Behalf Of Michael Snyder
> Sent: Tuesday, October 20, 2009 4:06 PM
> To: Marc Khouzam
> Cc: 'Hui Zhu'; 'gdb@sourceware.org'
> Subject: Re: [FYI] tutorial for process record and reverse debugging
> 
> [...]
> > Now, let me describe the case I am imagining.
> > It is as simple as it gets.
> > The user simply enables the 'reverse debugging' feature.
> > After that, the user should not need to pay attention to
> > record logs and such.  What they should see is that they
> > can go forward or backwards as if everything was true 'execution'.
> > We don't need to differentiate between 'execution' and 'replay'.
> > 
> > For example, when changing memory, the user doesn't need to know
> > that we are moving away from replay into a new execution.  All 
> > they see is that the program moves forward with the new memory
> > value.
> > 
> > And that is why, in this scenario, I thought it seemed
> > unintuitive to stop execution when
> > arriving at the end of the replay log; instead, the user
> > pressed 'continue' and the 'execution' should continue until
> > a breakpoint or the end of the program, as if a true execution.
> > 
> > The only limitation to this, is that we cannot go backwards
> > past the start of the recording.  But I think this can be easily
> > understood by the user.
> > 
> > I don't think this scenario is good for everyone, but I think
> > for average users, it makes reverse debugging very fluid.
> 
> I think that's a great scenario -- just not the only scenario.
> We could call that Marc-mode, for devel purposes.   ;-)

Just so it doesn't go to my head, how about "auto-mode"
as in "transitions from replay to record are done automatically"
(and without querying the user).

> How would you suggest we might turn on Marc-mode with a
> single command?

Maybe we can introduce a user setting such as
"record mode <auto|someothermode>"

> Or do you imagine it being the default?

Greg's follow-up email made me re-think a bit how intuitive
auto-mode would be.  I'll reply to his email with my comments.

Marc

RE: [FYI] tutorial for process record and reverse debugging

[Marc Khouzam]

 

> -----Original Message-----
> From: Greg Law [mailto:glaw@undo-software.com] 
> Sent: Tuesday, October 20, 2009 5:01 PM
> To: Michael Snyder
> Cc: Marc Khouzam; 'Hui Zhu'; 'gdb@sourceware.org'
> Subject: Re: [FYI] tutorial for process record and reverse debugging
> 
> Michael Snyder wrote:
> > [...]
> >>
> >>> 3) I'm in replay mode, possibly in the middle of the recording,
> >>> and I want to switch to record mode.  Now there are several
> >>> branching possibilities:  Do I want to:
> >>>
> >>>    a) Go to the end and start appending to the existing log?
> >>
> >> I can understand someone wanting this.
> >>
> >>>    b) Truncate the existing log at the point where I am, and
> >>>       start appending to the prefix?
> >>
> >> I never thought of this case.  I see now that for non-deterministic
> >> executions this could have value.
> > 
> > Not just that, though.  This is also what happens if we
> > change a memory or register value, eg. a variable that
> > controls a conditional branch.  We auto-delete the trailing
> > part of the execution log, because now we're going to go
> > forward in a different direction.
> 
> But what would we do about "external state"?
> 
> Example, say I'm debugging a web-server, and half way through a 
> connection I go back, then truncate the log and start again in a new 
> direction.  And clients "mid-session" with the webserver are 
> liable to 
> get mighty confused.

To be honest, I hadn't been dealing with any "external state" much.
But now that you mention it, yes, that would make "auto-mode"
(going from replay to record automatically) confusing.
See below please.
 

> > [...]
> >> Now, let me describe the case I am imagining.
> >> It is as simple as it gets.
> >> The user simply enables the 'reverse debugging' feature.
> >> After that, the user should not need to pay attention to
> >> record logs and such.  What they should see is that they
> >> can go forward or backwards as if everything was true 'execution'.
> >> We don't need to differentiate between 'execution' and 'replay'.
> >>
> >> For example, when changing memory, the user doesn't need to know
> >> that we are moving away from replay into a new execution.  
> All they 
> >> see is that the program moves forward with the new memory
> >> value.
> >>
> >> And that is why, in this scenario, I thought it seemed
> >> unintuitive to stop execution when
> >> arriving at the end of the replay log; instead, the user
> >> pressed 'continue' and the 'execution' should continue until
> >> a breakpoint or the end of the program, as if a true execution.
> >>
> >> The only limitation to this, is that we cannot go backwards
> >> past the start of the recording.  But I think this can be easily
> >> understood by the user.
> >>
> >> I don't think this scenario is good for everyone, but I think
> >> for average users, it makes reverse debugging very fluid.
> > 
> > I think that's a great scenario -- just not the only scenario.
> > We could call that Marc-mode, for devel purposes.   ;-)
> > 
> > How would you suggest we might turn on Marc-mode with a
> > single command?
> > 
> > Or do you imagine it being the default?
> > 
> 
> FWIW, early versions of UndoDB operated in "Marc-mode".  We 
> changed it 
> because replay mode and record are quite different, 
> particularly w.r.t. 
> to the program's interaction with the outside world. "Silent" 
> transition 
> from replay to record mode could be quite confusing/surprising.

In cases where the program does interact with the outside world, 
I agree that "auto-mode" could be confusing.  In fact, "record mode"
may not be the one we want in this case either.  For instance, in
some cases I may want to re-execute instructions that affect the outside
world, even if I'm gone backwards.  In this case I would want to use
the recorded  data to go backwards, but never to go forward (never use
"replay mode").

This is giving me two ideas:
1- the frontend (my problem) would benefit in showing to the user if
we are currently "executing" or "replaying".  some minimal support from
GDB would help there.

2- we could define a set of behaviors or modes for these scenarios
record mode auto
record mode replay
record mode execute
...

Thanks

Marc

Re: [FYI] tutorial for process record and reverse debugging

[Michael Snyder]

Greg Law wrote:
> Michael Snyder wrote:
>> [...]
>>>> 3) I'm in replay mode, possibly in the middle of the recording,
>>>> and I want to switch to record mode.  Now there are several
>>>> branching possibilities:  Do I want to:
>>>>
>>>>    a) Go to the end and start appending to the existing log?
>>> I can understand someone wanting this.
>>>
>>>>    b) Truncate the existing log at the point where I am, and
>>>>       start appending to the prefix?
>>> I never thought of this case.  I see now that for non-deterministic
>>> executions this could have value.
>> Not just that, though.  This is also what happens if we
>> change a memory or register value, eg. a variable that
>> controls a conditional branch.  We auto-delete the trailing
>> part of the execution log, because now we're going to go
>> forward in a different direction.
> 
> But what would we do about "external state"?

Well, I had only been talking in the context of process record,
which doesn't really deal all that well with external state,
as far as I know.

It's fine to open up the discussion to general reverse debugging,
but I think at some point different implementations are going to
have to have their own separate, dedicated commands (as process
record presently does).

Of course, the more generality, the better!


> Example, say I'm debugging a web-server, and half way through a 
> connection I go back, then truncate the log and start again in a new 
> direction.  And clients "mid-session" with the webserver are liable to 
> get mighty confused.

Speaking only for process record, I don't think process record
could handle that situation well.  It's called process record
for one reason because it only handles one process at a time
(in fact, currently only one thread at a time).

Re: [FYI] tutorial for process record and reverse debugging

[Greg Law]

Marc Khouzam wrote:
> 
> 
>> [...]
>>> Or do you imagine it being the default?
>>> 
>> FWIW, early versions of UndoDB operated in "Marc-mode".  We changed
>> it because replay mode and record are quite different, particularly
>> w.r.t. to the program's interaction with the outside world.
>> "Silent" transition from replay to record mode could be quite
>> confusing/surprising.
> 
> In cases where the program does interact with the outside world, I
> agree that "auto-mode" could be confusing.  In fact, "record mode" 
> may not be the one we want in this case either.  For instance, in 
> some cases I may want to re-execute instructions that affect the
> outside world, even if I'm gone backwards.  In this case I would want
> to use the recorded  data to go backwards, but never to go forward
> (never use "replay mode").

I'm not completely sure of the behaviour of process record here, but
UndoDB in replay mode is completely "neutered", in that the results of
all system calls are 'synthesised'.  So if, say, in record mode your
program unlinks a file, then in replay mode the fact the filename is no
longer present on the filesystem doesn't matter - we don't really do an 
unlink, we just replay the results of whatever unlink returned when in 
record mode.

I *believe* process record does the same, as would any reasonably sane
reversible debugging approach.   If you can step back over an
instruction, you do so without actually executing the instructions in
question.  So it seems reasonable that in replay mode you can also step
forwards, but without actually executing the instruction.

> 
> This is giving me two ideas: 1- the frontend (my problem) would
> benefit in showing to the user if we are currently "executing" or
> "replaying".  some minimal support from GDB would help there.

I agree this would be very useful for the user.

> 
> 2- we could define a set of behaviors or modes for these scenarios 
> record mode auto record mode replay record mode execute ...

Yes, that should work.  I can see giving the user the ability to tweak
it would help.  I think the default should be to stop, but what the 
default is could always be overridden by the frontend.

Greg
-- 
Greg Law, Undo Software                       http://undo-software.com/

Re: [FYI] tutorial for process record and reverse debugging

[Michael Snyder]

Greg Law wrote:
> Marc Khouzam wrote:
>>
>>> [...]
>>>> Or do you imagine it being the default?
>>>>
>>> FWIW, early versions of UndoDB operated in "Marc-mode".  We changed
>>> it because replay mode and record are quite different, particularly
>>> w.r.t. to the program's interaction with the outside world.
>>> "Silent" transition from replay to record mode could be quite
>>> confusing/surprising.
>> In cases where the program does interact with the outside world, I
>> agree that "auto-mode" could be confusing.  In fact, "record mode" 
>> may not be the one we want in this case either.  For instance, in 
>> some cases I may want to re-execute instructions that affect the
>> outside world, even if I'm gone backwards.  In this case I would want
>> to use the recorded  data to go backwards, but never to go forward
>> (never use "replay mode").
> 
> I'm not completely sure of the behaviour of process record here, but
> UndoDB in replay mode is completely "neutered", in that the results of
> all system calls are 'synthesised'.  So if, say, in record mode your
> program unlinks a file, then in replay mode the fact the filename is no
> longer present on the filesystem doesn't matter - we don't really do an 
> unlink, we just replay the results of whatever unlink returned when in 
> record mode.

It's the same for process record.

RE: [FYI] tutorial for process record and reverse debugging

[Marc Khouzam]

> I'm not completely sure of the behaviour of process record here, but
> UndoDB in replay mode is completely "neutered", in that the results of
> all system calls are 'synthesised'.  So if, say, in record mode your
> program unlinks a file, then in replay mode the fact the 
> filename is no
> longer present on the filesystem doesn't matter - we don't 
> really do an 
> unlink, we just replay the results of whatever unlink 
> returned when in 
> record mode.
> 
> I *believe* process record does the same, as would any reasonably sane
> reversible debugging approach.   If you can step back over an
> instruction, you do so without actually executing the instructions in
> question.  So it seems reasonable that in replay mode you can 
> also step
> forwards, but without actually executing the instruction.

I just wanted to clarify what I meant.
I agree with you that the behavior you describe is probably the most
useful one.
But I was thinking of certain cases, most probably very rare, such as:
I have a client/server setup where my server answers simple queries;
when running the client in the debugger I would find it useful to
execute the sending of a query, then go backwards and (intead of
replaying "neutered") go forwards again and re-send the query.
The idea here is that I'm more testing my server than my client
by doing this.
Granted, this example is pretty simplistic and probably not that
useful, but I just thought it illustrated that some users may
want to not use replay but instead re-execute.  So it may be nice
to have a command to do that, or a 'mode' that always did that.

Marc

Re: [FYI] tutorial for process record and reverse debugging

[Hui Zhu]

>
> However, rather than such a specific command as you describe, I had been
> thinking about a more general alternative, which in my mind I have been
> referring to by the code name "goto".  As in:
>
>  (gdb) record goto end
>  (gdb) record goto beginning
>  (gdb) record goto instruction 12345
>  (gdb) record goto bookmark 7
>
> Easy enough to make this command ignore breakpoints watchpoints etc.
>
>
>

I really like this idea.  :)

Hui

Re: [FYI] tutorial for process record and reverse debugging

[Hui Zhu]

On Wed, Oct 21, 2009 at 04:05, Michael Snyder <msnyder@vmware.com> wrote:
> Marc Khouzam wrote:
>
>>> OK, so good discussion.  Let's cover some bases here.
>>>
>>> 1) I'm in record mode, and I want to stay in record mode.
>>> No brainer -- that's the default behavior.
>>>
>>> 2) I'm in record mode, and I want to go to replay mode.
>>> Currently the only way to do that is to give a "reverse"
>>> command (reverse step, reverse continue...)
>>>
>>> That's not too bad, but sometimes I might want to simply
>>> go to the beginning of the log and start replaying forward
>>> from the beginning (ie. not backwards from the end.
>>> Or, I might even want to goto the middle before I start
>>> to replay (in either direction).
>>>
>>> We can do that now by using breakpoints, but we might have
>>> to disable other breakpoints, if there are any.
>>
>> And for long executions, jumping in the recorded log is probably
>> faster than using breakpoints.  I agree that this seem valuable.
>>
>>> But we COULD do it if we had a command like "goto beginning",
>>> or "goto bookmark 12".
>>
>> Again, this seems neat.  I do think it is somewhat of an
>> 'advanced' feature, as it requires more understanding of PRecord
>> than using breakpoints and reverse-continue/reverse-step/etc
>>
>>> 3) I'm in replay mode, possibly in the middle of the recording,
>>> and I want to switch to record mode.  Now there are several
>>> branching possibilities:  Do I want to:
>>>
>>>   a) Go to the end and start appending to the existing log?
>>
>> I can understand someone wanting this.
>>
>>>   b) Truncate the existing log at the point where I am, and
>>>      start appending to the prefix?
>>
>> I never thought of this case.  I see now that for non-deterministic
>> executions this could have value.
>
> Not just that, though.  This is also what happens if we
> change a memory or register value, eg. a variable that
> controls a conditional branch.  We auto-delete the trailing
> part of the execution log, because now we're going to go
> forward in a different direction.
>
>
>>>   c) Discard the existing log and start a new log from the
>>>      point where I am?
>>
>> I think this one is simply to re-issue the 'record' command.
>> Also, besides saving some space, I don't really see a big value
>> compared to point b) above.
>
> It's a minor case (because it's easy).  I'm just being
> exhaustive.
>
> [...]
>>
>> Now, let me describe the case I am imagining.
>> It is as simple as it gets.
>> The user simply enables the 'reverse debugging' feature.
>> After that, the user should not need to pay attention to
>> record logs and such.  What they should see is that they
>> can go forward or backwards as if everything was true 'execution'.
>> We don't need to differentiate between 'execution' and 'replay'.
>>
>> For example, when changing memory, the user doesn't need to know
>> that we are moving away from replay into a new execution.  All they see is
>> that the program moves forward with the new memory
>> value.
>>
>> And that is why, in this scenario, I thought it seemed
>> unintuitive to stop execution when
>> arriving at the end of the replay log; instead, the user
>> pressed 'continue' and the 'execution' should continue until
>> a breakpoint or the end of the program, as if a true execution.
>>
>> The only limitation to this, is that we cannot go backwards
>> past the start of the recording.  But I think this can be easily
>> understood by the user.
>>
>> I don't think this scenario is good for everyone, but I think
>> for average users, it makes reverse debugging very fluid.
>
> I think that's a great scenario -- just not the only scenario.
> We could call that Marc-mode, for devel purposes.   ;-)
>
> How would you suggest we might turn on Marc-mode with a
> single command?
>
> Or do you imagine it being the default?
>
>

For current status,  this record still have a lot of limit.  I think
make it to default is not very fit for the current record.

Thanks,
Hui

RE: [FYI] tutorial for process record and reverse debugging

[Jakob Engblom]

> If going straight to the end of the recording log is a valuable
> feature, I was thinking it may deserve its own command.
> 
> Ultimately, I'm hoping that with such a command, we can then see
> if we can have PRecord directly jump from recorded execution
> to live execution without stopping.

Note that we had a discussion a while ago about adding a general "go to point in
time in recording/reverse execution history" command, which would kind of solve
this as a special case (provided you can find the "last" time).   We can already
do this in Simics, but there is no support in the gdb-serial protocol or the gdb
CLI or MI to access the function.  The undodb people had the same issue.

/jakob

RE: [FYI] tutorial for process record and reverse debugging

[Jakob Engblom]

> 
> Seems to me that if you're replaying, and you reach the end of the
> recorded log, at least one natural thing to do is to stop.  Could be
> that another natural thing to do would be to automatically switch
> from replay mode to record mode and keep going -- but I'm not sure
> if I could safely assume that that was the user's desire.
> 
> Sounds like a case for a query, and/or a user-settable mode.

Unless the backend tool is one that silently goes on recording here, like I know
Simics does and I presume any other simulation-based solution will. And
VMWare-style virtual-machine-based solutions.  So a query for how a target
behaves would  be nice. 

In our experience, going on recording seems to be the natural thing to do.  You
can look at this way: the end of recording is nothing a user knows about, as the
extent of the recording in time is a tool implementation aspect. As far as the
user is concerned, the program is being reversed and run forward, without much
care for how far into the execution they have gotten at any particular point in
user time.  If you want to execute for a bounded amount, use a "continue N
cycles"-style command (which gdb does not have either). 

/jakob

RE: [FYI] tutorial for process record and reverse debugging

[Jakob Engblom]

> >>    b) Truncate the existing log at the point where I am, and
> >>       start appending to the prefix?
> >
> > I never thought of this case.  I see now that for non-deterministic
> > executions this could have value.
> 
> Not just that, though.  This is also what happens if we
> change a memory or register value, eg. a variable that
> controls a conditional branch.  We auto-delete the trailing
> part of the execution log, because now we're going to go
> forward in a different direction.

This is pretty useful, actually.  We have done that in Simics for a long time,
go back in time, and delete the recording of activity, to take a different path.


You can use it to "undo" mistakes (it is pretty funny to sit on a target
console, type "rm -rf /*", see the system implode... and then go back and just
undo the delete, delete the recording, and move on with an intact file system). 

You can use it to systematically explore system behavior, such as going back and
changing program inputs, hardware latencies, or other controllable aspects.  It
is neat for fault injection: go back, chagnge value or register or delete a
network packet, and then compare the results.  

Best regards,

/jakob

_______________________________________________________

Jakob Engblom, PhD, Technical Marketing Manager

Virtutech                   Direct: +46 8 690 07 47   
Drottningholmsvägen 22      Mobile: +46 709 242 646  
11243 Stockholm             Web:    www.virtutech.com 
Sweden
________________________________________________________
  


So yes, it is something that is pretty useful.

RE: [FYI] tutorial for process record and reverse debugging

[Jakob Engblom]

> > FWIW, early versions of UndoDB operated in "Marc-mode".  We
> > changed it
> > because replay mode and record are quite different,
> > particularly w.r.t.
> > to the program's interaction with the outside world. "Silent"
> > transition
> > from replay to record mode could be quite confusing/surprising.
> 
> In cases where the program does interact with the outside world,
> I agree that "auto-mode" could be confusing.  In fact, "record mode"
> may not be the one we want in this case either.  For instance, in
> some cases I may want to re-execute instructions that affect the outside
> world, even if I'm gone backwards.  In this case I would want to use
> the recorded  data to go backwards, but never to go forward (never use
> "replay mode").

Well, it actually can be made to work:

* Make sure the target is isolated, and that the debugger system can catch
anything going to the outside. This is trivial in a simulator or VM.

* When working with past time, don't send on things to the outside world. This
means reverse, replay, and going back again.

* When in progression forward, do connect to the outside world. The only problem
here are time-outs. 

Obviously, my favorite solution is to just get around the entire problem by
including a sufficient component of the world inside the reversible system,
inside a system simulator.


Best regards,

/jakob

_______________________________________________________

Jakob Engblom, PhD, Technical Marketing Manager

Virtutech                   Direct: +46 8 690 07 47   
Drottningholmsvägen 22      Mobile: +46 709 242 646  
11243 Stockholm             Web:    www.virtutech.com 
Sweden
________________________________________________________
  

Re: [FYI] tutorial for process record and reverse debugging

[Michael Snyder]

Jakob Engblom wrote:
>> If going straight to the end of the recording log is a valuable
>> feature, I was thinking it may deserve its own command.
>>
>> Ultimately, I'm hoping that with such a command, we can then see
>> if we can have PRecord directly jump from recorded execution
>> to live execution without stopping.
> 
> Note that we had a discussion a while ago about adding a general "go to point in
> time in recording/reverse execution history" command, which would kind of solve
> this as a special case (provided you can find the "last" time).   We can already
> do this in Simics, but there is no support in the gdb-serial protocol or the gdb
> CLI or MI to access the function.  The undodb people had the same issue.

What would you think of "qBookmark" to request a bookmark id,
and "QBookmark:id" to "set" the bookmark (that is, to go back
to that machine state)?

RE: [FYI] tutorial for process record and reverse debugging

[Jakob Engblom]

> > Note that we had a discussion a while ago about adding a general "go to
point
> in
> > time in recording/reverse execution history" command, which would kind of
solve
> > this as a special case (provided you can find the "last" time).   We can
> already
> > do this in Simics, but there is no support in the gdb-serial protocol or the
> gdb
> > CLI or MI to access the function.  The undodb people had the same issue.
> 
> What would you think of "qBookmark" to request a bookmark id,
> and "QBookmark:id" to "set" the bookmark (that is, to go back
> to that machine state)?

That would certainly work with Simics. 

/jakob