Stopping reverse debugging behaves differently with btrace

Stopping reverse debugging behaves differently with btrace

[Marc Khouzam]

Hi,

I noticed a difference of behaviour when stopping reverse debugging
with btrace vs record/replay (full mode).

If full mode, if I step to line 200 and then back to line 150 then give
the record stop command, real execution will start from line 150 as 
I step my program. (This is really cool btw.)

In btrace mode (I tried bts), after I step back to line 150 and send 
the record stop command, GDB appears to still be at line 150 but on
the next step, execution jumps to line 200 before doing the step 
operation.

I'm guessing this has to do with the fact that btrace does not store 
registers and memory, so cannot restart execution just anywhere.  

I find it strange though that when turning off record, every indication 
to the user is that we are still at line 150, when in reality, GDB is 
effectively back at line 200.  This is particularly noticeable in a 
frontends when execution jumps (unexpectedly) when the first step
is requested.  

Variables also remain unavailable until the next step (or strangely,
until I send some register command).

I was wondering if GDB should reset its execution to the proper
place upon a 'record stop' for btrace?  And notify the frontend of
that change.

Any opinions?

Thanks

Marc

RE: Stopping reverse debugging behaves differently with btrace

[Metzger, Markus T]

Hi Marc,

> I noticed a difference of behaviour when stopping reverse debugging
> with btrace vs record/replay (full mode).
> 
> If full mode, if I step to line 200 and then back to line 150 then give
> the record stop command, real execution will start from line 150 as
> I step my program. (This is really cool btw.)
> 
> In btrace mode (I tried bts), after I step back to line 150 and send
> the record stop command, GDB appears to still be at line 150 but on
> the next step, execution jumps to line 200 before doing the step
> operation.
> 
> I'm guessing this has to do with the fact that btrace does not store
> registers and memory, so cannot restart execution just anywhere.

That's correct.


> I find it strange though that when turning off record, every indication
> to the user is that we are still at line 150, when in reality, GDB is
> effectively back at line 200.  This is particularly noticeable in a
> frontends when execution jumps (unexpectedly) when the first step
> is requested.
> 
> Variables also remain unavailable until the next step (or strangely,
> until I send some register command).
> 
> I was wondering if GDB should reset its execution to the proper
> place upon a 'record stop' for btrace?  And notify the frontend of
> that change.

I agree.  I'll look into it.  Thanks for pointing it out.

Regards,
Markus.
Intel Deutschland GmbH
Registered Address: Am Campeon 10-12, 85579 Neubiberg, Germany
Tel: +49 89 99 8853-0, www.intel.de
Managing Directors: Christin Eisenschmid, Christian Lamprechter
Chairperson of the Supervisory Board: Nicole Lau
Registered Office: Munich
Commercial Register: Amtsgericht Muenchen HRB 186928

RE: Stopping reverse debugging behaves differently with btrace

[Metzger, Markus T]

Hi Marc, Pedro,

> > I find it strange though that when turning off record, every indication
> > to the user is that we are still at line 150, when in reality, GDB is
> > effectively back at line 200.  This is particularly noticeable in a
> > frontends when execution jumps (unexpectedly) when the first step
> > is requested.
> >
> > Variables also remain unavailable until the next step (or strangely,
> > until I send some register command).
> >
> > I was wondering if GDB should reset its execution to the proper
> > place upon a 'record stop' for btrace?  And notify the frontend of
> > that change.
> 
> I agree.  I'll look into it.  Thanks for pointing it out.

I have a fix for the missing STOP_PC update which may result in all
kinds of strange effects.  The front-end notification sounds more
tricky, though.

If I understand the code correctly, infrun's normal_stop calls a normal_stop
observer to which front-ends register.  The front-end is then responsible
for printing the current source location.  In the case of MI, this should also
give the stopped notification to GUIs.  I assume it's this notification you're
talking about.

We also shouldn't get a notification for the "record goto" command.  This
affects all record targets.  Historically, the "record goto" command does not call
the normal_stop observer and instead calls print_stack_frame directly.  Are you
seeing this, as well, Marc?


When I try to either call observer_notify_normal_stop (NULL, 1) directly or
indirectly via normal_stop, I get repeated breakpoint notifications for the
breakpoint at which the thread stopped originally.  Looking at those functions
I may get other notifications, as well, depending on global state.

It looks like the trick is to do what's normally done after target_wait.  I found
some code in infrun that seems to do something like that:

      /* Go through handle_inferior_event/normal_stop, so we always
	 have consistent output as if the stop event had been
	 reported.  */
      ecs->ptid = info->ptid;
      ecs->event_thread = info;
      ecs->ws.kind = TARGET_WAITKIND_STOPPED;
      ecs->ws.value.sig = GDB_SIGNAL_0;

      handle_inferior_event (ecs);

      if (!ecs->wait_some_more)
	{
	  /* Cancel any running execution command.  */
	  thread_cancel_execution_command (info);

	  normal_stop ();
	}


When I do that for "record goto" and "record stop" I get "Thread x stopped." as
well as "Switching to thread x" notifications.  I'm not sure if we want them.

Pedro, is this the right direction?  Would we need to go through the entire
proceed/resume/wait flow?  Or is the above already too complicated?

Thanks,
Markus.

Intel Deutschland GmbH
Registered Address: Am Campeon 10-12, 85579 Neubiberg, Germany
Tel: +49 89 99 8853-0, www.intel.de
Managing Directors: Christin Eisenschmid, Christian Lamprechter
Chairperson of the Supervisory Board: Nicole Lau
Registered Office: Munich
Commercial Register: Amtsgericht Muenchen HRB 186928

RE: Stopping reverse debugging behaves differently with btrace

[Metzger, Markus T]

Hi Marc, Pedro,


> > > I find it strange though that when turning off record, every indication
> > > to the user is that we are still at line 150, when in reality, GDB is
> > > effectively back at line 200.  This is particularly noticeable in a
> > > frontends when execution jumps (unexpectedly) when the first step
> > > is requested.
> > >
> > > Variables also remain unavailable until the next step (or strangely,
> > > until I send some register command).
> > >
> > > I was wondering if GDB should reset its execution to the proper
> > > place upon a 'record stop' for btrace?  And notify the frontend of
> > > that change.
> >
> > I agree.  I'll look into it.  Thanks for pointing it out.
> 
> I have a fix for the missing STOP_PC update which may result in all
> kinds of strange effects.  The front-end notification sounds more
> tricky, though.

Looks like the trick is to clear the proceed state of the moving thread
before calling the normal-stop observer.  I'm omitting the about-to-
proceed notification and I'm not calling proceed or normal_stop.
I hope I'm not breaking something.

GDB is now sending a normal-stop without stop reason  on every record
goto command:

(gdb) 
rec go 12
&"rec go 12\n"
~"0x00007ffff762c671 in _dl_addr () from /lib64/libc.so.6\n"
*stopped,frame={addr="0x00007ffff762c671",func="_dl_addr",args=[],from="/lib64/libc.so.6"},thread-id="1",stopped-threads="all",core="1"
^done

We can also invent a new stop-reason if necessary.  The record-btrace
target also sends such a normal-stop notification on "record stop" for
each replaying thread:

(gdb) 
rec stop
&"rec stop\n"
~"test (arg=0x0) at gdb.btrace/multi-thread-step.c:34\n"
~"34\t  global = 42; /* bp.2 */\n"
*stopped,frame={addr="0x000000000040078a",func="test",args=[{name="arg",value="0x0"}],file="gdb.btrace/multi-thread-step.c",fullname="gdb.btrace/multi-thread-step.c",line="34"},thread-id="1",stopped-threads="all",core="1"
~"test (arg=0x0) at gdb.btrace/multi-thread-step.c:34\n"
~"34\t  global = 42; /* bp.2 */\n"
*stopped,frame={addr="0x000000000040078a",func="test",args=[{name="arg",value="0x0"}],file="gdb.btrace/multi-thread-step.c",fullname="gdb.btrace/multi-thread-step.c",line="34"},thread-id="2",stopped-threads="all",core="1"
~"Process record is stopped and all execution logs are deleted.\n"
=record-stopped,thread-group="i1"
^done

Marc, is this what you were expecting?

The CLI output for "record stop" needs more polishing.  It currently prints the
updated source location for every replaying thread without indicating the thread
the output belongs to.  Not sure how much output we really want on the CLI for
"record stop".

Regards,
Markus.

Intel Deutschland GmbH
Registered Address: Am Campeon 10-12, 85579 Neubiberg, Germany
Tel: +49 89 99 8853-0, www.intel.de
Managing Directors: Christin Eisenschmid, Christian Lamprechter
Chairperson of the Supervisory Board: Nicole Lau
Registered Office: Munich
Commercial Register: Amtsgericht Muenchen HRB 186928

RE: Stopping reverse debugging behaves differently with btrace

[Marc Khouzam]

> Hi Marc, Pedro,
>
> > > > I find it strange though that when turning off record, every indication
> > > > to the user is that we are still at line 150, when in reality, GDB is
> > > > effectively back at line 200.  This is particularly noticeable in a
> > > > frontends when execution jumps (unexpectedly) when the first step
> > > > is requested.
> > > >
> > > > Variables also remain unavailable until the next step (or strangely,
> > > > until I send some register command).
> > > >
> > > > I was wondering if GDB should reset its execution to the proper
> > > > place upon a 'record stop' for btrace?  And notify the frontend of
> > > > that change.
> > >
> > > I agree.  I'll look into it.  Thanks for pointing it out.
> >
> > I have a fix for the missing STOP_PC update which may result in all
> > kinds of strange effects.  The front-end notification sounds more
> > tricky, though.
> 
> Looks like the trick is to clear the proceed state of the moving thread
> before calling the normal-stop observer.  I'm omitting the about-to-
> proceed notification and I'm not calling proceed or normal_stop.
> I hope I'm not breaking something.
> 
> GDB is now sending a normal-stop without stop reason  on every record
> goto command:
> 
> (gdb)
> rec go 12
> &"rec go 12\n"
> ~"0x00007ffff762c671 in _dl_addr () from /lib64/libc.so.6\n"
> *stopped,frame={addr="0x00007ffff762c671",func="_dl_addr",args=[],from="/lib64/libc.so.6"},thread-id="1",stopped-threads="all",core="1"
> ^done

I'd have to try it to be sure, but this looks right.  If a record command 
changes the line at which GDB is stopped, the frontend needs a *stopped
event like that.  Sending multiple *stopped events for the same thread without
a *running event in between may be a new case; I don't know if that is something 
frontends are supposed to be ready for.  I'm almost certain Eclipse can handle
it though.

> We can also invent a new stop-reason if necessary.  

That is probably a nice clarification for the frontend.
However, looking at how Eclipse handles the 'reason' field, we are not ready
for a new one and will ignore any message with an unexpected 'reason'.
Again, I'd have to try it to be sure.  It would be easy to fix and I would make
Eclipse accept any unkown reason as a generic *stopped event, but older
versions of Eclipse won't have that fix.

Please keep me in the loop about such a change.

> The record-btrace
> target also sends such a normal-stop notification on "record stop" for
> each replaying thread:
> 
> (gdb)
> rec stop
> &"rec stop\n"
> ~"test (arg=0x0) at gdb.btrace/multi-thread-step.c:34\n"
> ~"34\t  global = 42; /* bp.2 */\n"
> *stopped,frame={addr="0x000000000040078a",func="test",args=[{name="arg",value="0x0"}],file="gdb.btrace/multi-thread-step.c",fullname="gdb.btrace/multi-thread-step.c",line="34"},thread-id="1",stopped-threads="all",core="1"
> ~"test (arg=0x0) at gdb.btrace/multi-thread-step.c:34\n"
> ~"34\t  global = 42; /* bp.2 */\n"
> *stopped,frame={addr="0x000000000040078a",func="test",args=[{name="arg",value="0x0"}],file="gdb.btrace/multi-thread-step.c",fullname="gdb.btrace/multi-thread-step.c",line="34"},thread-id="2",stopped-threads="all",core="1"
> ~"Process record is stopped and all execution logs are deleted.\n"
> =record-stopped,thread-group="i1"
> ^done
> 
> Marc, is this what you were expecting?

That also seems right.
Eclipse does not yet support reverse debugging with non-stop, but we are
working on it.

> The CLI output for "record stop" needs more polishing.  It currently prints the
> updated source location for every replaying thread without indicating the thread
> the output belongs to.  Not sure how much output we really want on the CLI for
> "record stop".
> 
> Regards,
> Markus.

Thanks for looking into this so quickly.

Marc

RE: Stopping reverse debugging behaves differently with btrace

[Metzger, Markus T]

Hi Marc,

> > GDB is now sending a normal-stop without stop reason  on every record
> > goto command:
> >
> > (gdb)
> > rec go 12
> > &"rec go 12\n"
> > ~"0x00007ffff762c671 in _dl_addr () from /lib64/libc.so.6\n"
> >
> *stopped,frame={addr="0x00007ffff762c671",func="_dl_addr",args=[],from="/li
> b64/libc.so.6"},thread-id="1",stopped-threads="all",core="1"
> > ^done
> 
> I'd have to try it to be sure, but this looks right.  If a record command
> changes the line at which GDB is stopped, the frontend needs a *stopped
> event like that.  Sending multiple *stopped events for the same thread without
> a *running event in between may be a new case; I don't know if that is
> something
> frontends are supposed to be ready for.  I'm almost certain Eclipse can handle
> it though.

Hmmm, the alternative would be to go through the full proceed/resume/
wait/normal_stop flow for "record goto".  This would give a ^running and a
prompt followed by a *stopped notification.  There's a problem with this,
though - see below.


> > We can also invent a new stop-reason if necessary.
> 
> That is probably a nice clarification for the frontend.
> However, looking at how Eclipse handles the 'reason' field, we are not ready
> for a new one and will ignore any message with an unexpected 'reason'.
> Again, I'd have to try it to be sure.  It would be easy to fix and I would make
> Eclipse accept any unkown reason as a generic *stopped event, but older
> versions of Eclipse won't have that fix.

OK, let's go without a stop reason for now.


> > The record-btrace
> > target also sends such a normal-stop notification on "record stop" for
> > each replaying thread:
> >
> > (gdb)
> > rec stop
> > &"rec stop\n"
> > ~"test (arg=0x0) at gdb.btrace/multi-thread-step.c:34\n"
> > ~"34\t  global = 42; /* bp.2 */\n"
> >
> *stopped,frame={addr="0x000000000040078a",func="test",args=[{name="arg",v
> alue="0x0"}],file="gdb.btrace/multi-thread-step.c",fullname="gdb.btrace/multi-
> thread-step.c",line="34"},thread-id="1",stopped-threads="all",core="1"
> > ~"test (arg=0x0) at gdb.btrace/multi-thread-step.c:34\n"
> > ~"34\t  global = 42; /* bp.2 */\n"
> >
> *stopped,frame={addr="0x000000000040078a",func="test",args=[{name="arg",v
> alue="0x0"}],file="gdb.btrace/multi-thread-step.c",fullname="gdb.btrace/multi-
> thread-step.c",line="34"},thread-id="2",stopped-threads="all",core="1"
> > ~"Process record is stopped and all execution logs are deleted.\n"
> > =record-stopped,thread-group="i1"
> > ^done
> 
> That also seems right.
> Eclipse does not yet support reverse debugging with non-stop, but we are
> working on it.

That's the output for stop-all.  On "record stop" GDB is iterating over all threads to
stop recording and I'm sending a notification for every thread that moved due to
this.

If we wanted a stepping-like behavior (as sketched above), GDB would report a
single event in stop-all mode and queue events of other threads to be reported
when they are resumed the next time.

That's not desirable since we would need multiple "continue" after "record stop"
to drain all those implicit record goto stopped events.

To the contrary, GDB implicitly stops replaying other threads if the selected thread
is resumed at the end of its execution history.  This is to avoid having to switch to
all replaying threads and do an explicit "record goto end".

Those implicit moves also do not trigger *stopped notifications.  I don't think they
should.  We're getting

    ^running
    *running,thread-id="all"

which already indicates that other threads may have moved, as well.

If we look at another command that affects multiple threads

    (gdb) 
    kill
    &"kill\n"
    =thread-exited,id="1",group-id="i1"
    =thread-exited,id="2",group-id="i1"
    =thread-group-exited,id="i1"
    ^done

I think that's what we'd want for "record stop" as well.  If *stopped doesn't
work, we may need a new record.

Regards,
Markus.

Intel Deutschland GmbH
Registered Address: Am Campeon 10-12, 85579 Neubiberg, Germany
Tel: +49 89 99 8853-0, www.intel.de
Managing Directors: Christin Eisenschmid, Christian Lamprechter
Chairperson of the Supervisory Board: Nicole Lau
Registered Office: Munich
Commercial Register: Amtsgericht Muenchen HRB 186928

RE: Stopping reverse debugging behaves differently with btrace

[Metzger, Markus T]

Hi Marc,

> > > GDB is now sending a normal-stop without stop reason  on every record
> > > goto command:
> > >
> > > (gdb)
> > > rec go 12
> > > &"rec go 12\n"
> > > ~"0x00007ffff762c671 in _dl_addr () from /lib64/libc.so.6\n"
> > >
> >
> *stopped,frame={addr="0x00007ffff762c671",func="_dl_addr",args=[],from="/li
> > b64/libc.so.6"},thread-id="1",stopped-threads="all",core="1"
> > > ^done
> >
> > I'd have to try it to be sure, but this looks right.  If a record command
> > changes the line at which GDB is stopped, the frontend needs a *stopped
> > event like that.  Sending multiple *stopped events for the same thread without
> > a *running event in between may be a new case; I don't know if that is
> > something
> > frontends are supposed to be ready for.  I'm almost certain Eclipse can handle
> > it though.

Eclipse Mars.2 seems to be OK with it.


> > > The record-btrace
> > > target also sends such a normal-stop notification on "record stop" for
> > > each replaying thread:
> > >
> > > (gdb)
> > > rec stop
> > > &"rec stop\n"
> > > ~"test (arg=0x0) at gdb.btrace/multi-thread-step.c:34\n"
> > > ~"34\t  global = 42; /* bp.2 */\n"
> > >
> >
> *stopped,frame={addr="0x000000000040078a",func="test",args=[{name="arg",v
> > alue="0x0"}],file="gdb.btrace/multi-thread-
> step.c",fullname="gdb.btrace/multi-
> > thread-step.c",line="34"},thread-id="1",stopped-threads="all",core="1"
> > > ~"test (arg=0x0) at gdb.btrace/multi-thread-step.c:34\n"
> > > ~"34\t  global = 42; /* bp.2 */\n"
> > >
> >
> *stopped,frame={addr="0x000000000040078a",func="test",args=[{name="arg",v
> > alue="0x0"}],file="gdb.btrace/multi-thread-
> step.c",fullname="gdb.btrace/multi-
> > thread-step.c",line="34"},thread-id="2",stopped-threads="all",core="1"
> > > ~"Process record is stopped and all execution logs are deleted.\n"
> > > =record-stopped,thread-group="i1"
> > > ^done
> >
> > That also seems right.
> > Eclipse does not yet support reverse debugging with non-stop, but we are
> > working on it.
> 
> That's the output for stop-all.  On "record stop" GDB is iterating over all threads to
> stop recording and I'm sending a notification for every thread that moved due to
> this.

Eclipse seems to be able to handle it but it may cause a thread switch.  Plus this
causes lots of output on the CLI.  I don't think we want that.

For stop-all mode, I changed this to send a single *stopped for the selected thread -
provided it is replaying.  This works fine as long as the selected thread actually is
replaying.  All threads are updated and we're not switching away from the selected
thread.

If the selected thread isn't replaying, however, GDB won't send a *stopped.  This is
OK for the selected thread as its state didn't change, but it won't cause other threads
that had been replaying to be updated.

I added another *stopped event without thread information for that case.  Eclipse
seems to be OK with it.  We're not switching threads but other threads seem to be
updated correctly.


I pushed the series into users/mmetzger/record-goto-mi.  Let me know what you think
and if this is working for you.  I will send the patches out for review in a week or two.

Regards,
Markus.

Intel Deutschland GmbH
Registered Address: Am Campeon 10-12, 85579 Neubiberg, Germany
Tel: +49 89 99 8853-0, www.intel.de
Managing Directors: Christin Eisenschmid, Christian Lamprechter
Chairperson of the Supervisory Board: Nicole Lau
Registered Office: Munich
Commercial Register: Amtsgericht Muenchen HRB 186928