Unneeded displaced single stepping causes issues: Performance drop and range stepping killer

Unneeded displaced single stepping causes issues: Performance drop and range stepping killer

[Raphael Zulliger]

Hi

I've been playing around with the 'range stepping' patch set. (Thanks 
for that great feature!) Thereby I encountered the following behavior:
     Targets, having a 'gdbarch->software_single_step' function set, are 
always using displaced single stepping even when "normal" single 
stepping would be sufficient. (sure, displaced single-stepping may have 
the advantage of less breakpoint hits by foreign task, but...)
While this is not a problem as such, it may be sub-optimal because:
     Issue a) At least on extended-remote ppc target displaced single 
stepping causes much more RSP traffic
     Issue b) It renders 'range stepping' useless

Let me explain Issue a)
Not sure whether this statement is correct for all platform out there, 
but on my PPC603/EABI extended remote target, the difference between 
non- vs. displaced single stepping is quite big.

         (gdb) set displaced-stepping 0
         (gdb) si
         Sending packet: $m832d0,4#fe...Packet received: 9421ffc8
         Sending packet: $vCont;s:p1.dca118#83...Packet received: OK
           Notification received:
Stop:T05thread:p1.dca118;0:00dcdea0;1:00dcde40;2:00dca118;3:00dcdea4;4:00dcdea0;5:00000142;6:00dcde68;7:00000002;8:00000005;9:00dcdea4;a:00579e90;b:00ed4b34;c:00000000;d:f7ffffff;e:00000357;f:f1010000;10:00000000;11:fff71658;12:0068e738;13:001dd964;14:00000000;15:00000000;16:0069c120;17:00000000;18:00000037;19:00000000;1a:00000032;1b:00e6b2c0;1c:00087e48;1d:00ed12a0;1e:00ed1b68;1f:00dcde78;20:00000006347fd013;21:3ff199999999999a;22:3ff199999999999a;23:0000000000000000;24:0000000000000000;25:0000000000000000;26:0000000000000000;27:0000000000000000;28:0000000000000000;29:0000000000000000;2a:0000000000000000;2b:0000000000000000;2c:401900a3d70a3d71;2d:3ff199999999999a;2e:0000000000000000;2f:0000000000000000;30:0000000000000000;31:0000000000000000;32:0000000000000000;33:0000000000000000;34:0000000000000000;35:0000000000000000;36:0000000000000000;37:0000000000000000;38:0000000000000000;39:0000000000000000;3a:0000000000000000;3b:0000000000000000;3c:0000000000000000;3d:0000000000000000;3e:0000000000000000;3f:0000000000000000;46:efbeadde;40:000832d4;41:0008a030;42:20000004;43:00049cd0;44:00000000;45:00000000;
         Sending packet: $vStopped#55...Packet received: OK
         0x000832d4    20    SourceLine::SourceLine( const std::string 
&fileName,

versus:

         (gdb) set displaced-stepping 1
         (gdb) si
         Sending packet: $m1508,4#9b...Packet received: 3c000058
         Sending packet: $m832d4,4#02...Packet received: 7c0802a6
         Sending packet: $X1508,0:#bc...Packet received:
         binary downloading NOT supported by target
         Sending packet: $M1508,4:7c0802a6#b0...Packet received: OK
         Sending packet: $g#67...Packet received: 
00dcdea000dcde4000dca11800dcdea400dcdea00000014200dcde68000000020000000500dcdea400579e9000ed4b3400000000f7ffffff00000357f101000000000000fff716580068e738001dd96400000000000000000069c1200000000000000037000000000000003200e6b2c000087e4800ed12a000ed1b6800dcde7800000006347fd0133ff199999999999a3ff199999999999a000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000401900a3d70a3d713ff199999999999a000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000832d40008a0302000000400049cd00000000000000000efbeadde
         Sending packet: $P40=00001508#7f...Packet received: OK
         Packet P (set-register) is supported
         Sending packet: $vCont;s:p1.dca118#83...Packet received: OK
           Notification received: 
Stop:T05thread:p1.dca118;0:00049cd0;1:00dcde40;2:00dca118;3:00dcdea4;4:00dcdea0;5:00000142;6:00dcde68;7:00000002;8:00000005;9:00dcdea4;a:00579e90;b:00ed4b34;c:00000000;d:f7ffffff;e:00000357;f:f1010000;10:00000000;11:fff71658;12:0068e738;13:001dd964;14:00000000;15:00000000;16:0069c120;17:00000000;18:00000037;19:00000000;1a:00000032;1b:00e6b2c0;1c:00087e48;1d:00ed12a0;1e:00ed1b68;1f:00dcde78;20:00000006347fd013;21:3ff199999999999a;22:3ff199999999999a;23:0000000000000000;24:0000000000000000;25:0000000000000000;26:0000000000000000;27:0000000000000000;28:0000000000000000;29:0000000000000000;2a:0000000000000000;2b:0000000000000000;2c:401900a3d70a3d71;2d:3ff199999999999a;2e:0000000000000000;2f:0000000000000000;30:0000000000000000;31:0000000000000000;32:0000000000000000;33:0000000000000000;34:0000000000000000;35:0000000000000000;36:0000000000000000;37:0000000000000000;38:0000000000000000;39:0000000000000000;3a:0000000000000000;3b:0000000000000000;3c:0000000000000000;3d:0000000000000000;3e:0000000000000000;3f:0000000000000000;46:efbeadde;40:0000150c;41:0008a030;42:20000004;43:00049cd0;44:00000000;45:00000000;
         Sending packet: $vStopped#55...Packet received: OK
         Sending packet: $M1508,4:3c000058#78...Packet received: OK
         Sending packet: $P40=000832d8#ba...Packet received: OK
         0x000832d8    20    SourceLine::SourceLine( const std::string 
&fileName,
         (gdb)

Issue b)
According to my current understanding (please correct me if I'm wrong): 
Range stepping decides whether to send a 'vCont;r' package by comparing 
the current program counter with the range that needs to be executed 
(control.step_range_start and control.step_range_end). Using displaced 
single stepping means that the program counter is, by definition, always 
outside the range that needs to be single stepped and thus 'range 
stepping' will never be used.

What causes the problem?
The following statement (from infrun.c, resume(), git master) decides 
whether to use displaced single stepping or not:
   if (use_displaced_stepping (gdbarch)
       && (tp->control.trap_expected
       || (step && gdbarch_software_single_step_p (gdbarch)))
       && sig == GDB_SIGNAL_0
       && !current_inferior ()->waiting_for_vfork_done)

According to my experiments:
     - Using gdb/gdbserver on x86/64, that statement is 'true' when we 
step over a breakpoint, but is 'false' otherwise ->range stepping is 
used when possible
     - Using extended remote to ppc603/EABI embedded system, that 
statement is always 'true' when we step because 
'gdbarch_software_single_step_p' returns 'true' ->range stepping is 
never used
     - When patching GDB so that 'gdbarch->software_single_step = NULL', 
then the behavior is like on x86/64 and thus 'range stepping' works 
->range stepping is used when possible


Finally, my request: Could someone with more insight into GDB internals 
please decide whether we have to fix something here and if yes, how? Or, 
in case that my conclusions are wrong, could you help me to really 
understand the problem? (FYI: My ultimate goal is to speedup remote 
debugging for our ppc603/EABI target.)

Thanks,
Raphael

Re: Unneeded displaced single stepping causes issues: Performance drop and range stepping killer

[Pedro Alves]

On 04/22/2013 11:54 AM, Raphael Zulliger wrote:

> According to my current understanding (please correct me if I'm wrong): Range stepping decides whether to send a 'vCont;r' package by comparing the current program counter with the range that needs to be executed (control.step_range_start and control.step_range_end). Using displaced single stepping means that the program counter is, by definition, always outside the range that needs to be single stepped and thus 'range stepping' will never be used.

I've been hacking on the range stepping series, and I've actually
changed how GDB decides to issue a range-step or not.
By "PC-is-in-range" has problems.  But, that's largely unrelated.
The root issue is the forced displaced step, and that'll remain
the same.

> What causes the problem?
> The following statement (from infrun.c, resume(), git master) decides whether to use displaced single stepping or not:
>   if (use_displaced_stepping (gdbarch)
>       && (tp->control.trap_expected
>       || (step && gdbarch_software_single_step_p (gdbarch)))
>       && sig == GDB_SIGNAL_0
>       && !current_inferior ()->waiting_for_vfork_done)
> 
> According to my experiments:
>     - Using gdb/gdbserver on x86/64, that statement is 'true' when we step over a breakpoint, but is 'false' otherwise ->range stepping is used when possible
>     - Using extended remote to ppc603/EABI embedded system, that statement is always 'true' when we step because 'gdbarch_software_single_step_p' returns 'true' ->range stepping is never used
>     - When patching GDB so that 'gdbarch->software_single_step = NULL', then the behavior is like on x86/64 and thus 'range stepping' works ->range stepping is used when possible
> 
> 
> Finally, my request: Could someone with more insight into GDB internals please decide whether we have to fix something here and if yes, how? Or, in case that my conclusions are wrong, could you help me to really understand the problem? (FYI: My ultimate goal is to speedup remote debugging for our ppc603/EABI target.)

There are several things to consider here.  The
"(step && gdbarch_software_single_step_p (gdbarch))" part
was added because GDB can't currently do multiple software
single-steps simultaneously, which is a problem for non-stop
mode (which enables displaced stepping).  The workaround was to force
displaced stepping for all single-steps.  GDB knows how to queue
displaced step requests (necessary because there's only one scratch
pad for all threads), and have threads wait for their turn in the
queue.  I was working on making GDB aware of multiple software
single-steps a while ago, but I got distracted with other things
and haven't finished that yet...

Then, as you note, gdbarch_software_single_step_p isn't really the
right predicate.  A better one would be "do I need to software
single-step this instruction".  There's no such predicate today,
because the gdbarch_software_single_step hook is responsible for
installing the breakpoints itself, instead of returning a vector
of the locations where breakpoints should be set (which I think
would be better).

-- 
Pedro Alves

Re: Unneeded displaced single stepping causes issues: Performance drop and range stepping killer

[Yao Qi]

On 04/22/2013 06:54 PM, Raphael Zulliger wrote:
> Hi
> 
> I've been playing around with the 'range stepping' patch set. (Thanks
> for that great feature!) Thereby I encountered the following behavior:

Thanks for playing with range stepping!

>       Targets, having a 'gdbarch->software_single_step' function set, are
> always using displaced single stepping even when "normal" single
> stepping would be sufficient. (sure, displaced single-stepping may have
> the advantage of less breakpoint hits by foreign task, but...)
> While this is not a problem as such, it may be sub-optimal because:
>       Issue a) At least on extended-remote ppc target displaced single
> stepping causes much more RSP traffic
>       Issue b) It renders 'range stepping' useless
> 
> Let me explain Issue a)
> Not sure whether this statement is correct for all platform out there,
> but on my PPC603/EABI extended remote target, the difference between
> non- vs. displaced single stepping is quite big.
> 
>           (gdb) set displaced-stepping 0
>           (gdb) si
>           Sending packet: $m832d0,4#fe...Packet received: 9421ffc8
>           Sending packet: $vCont;s:p1.dca118#83...Packet received: OK
>             Notification received:
> Stop:T05thread:p1.dca118;0:00dcdea0;1:00dcde40;2:00dca118;3:00dcdea4;4:00dcdea0;5:00000142;6:00dcde68;7:00000002;8:00000005;9:00dcdea4;a:00579e90;b:00ed4b34;c:00000000;d:f7ffffff;e:00000357;f:f1010000;10:00000000;11:fff71658;12:0068e738;13:001dd964;14:00000000;15:00000000;16:0069c120;17:00000000;18:00000037;19:00000000;1a:00000032;1b:00e6b2c0;1c:00087e48;1d:00ed12a0;1e:00ed1b68;1f:00dcde78;20:00000006347fd013;21:3ff199999999999a;22:3ff199999999999a;23:0000000000000000;24:0000000000000000;25:0000000000000000;26:0000000000000000;27:0000000000000000;28:0000000000000000;29:0000000000000000;2a:0000000000000000;2b:0000000000000000;2c:401900a3d70a3d71;2d:3ff199999999999a;2e:0000000000000000;2f:0000000000000000;30:0000000000000000;31:0000000000000000;32:0000000000000000;33:0000000000000000;34:0000000000000000;35:0000000000000000;36:0000000000000000;37:0000000000000000;38:0000000000000000;39:0000000000000000;3a:0000000000000000;3b:0000000000000000;3c:0000000000000000;3d:0000000000000000;3e:0
000000000000000;3f:0000000000000000;46:efbeadde;40:000832d4;41:0008a030;42:20000004;43:00049cd0;44:00000000;45:00000000;
>           Sending packet: $vStopped#55...Packet received: OK
>           0x000832d4    20    SourceLine::SourceLine( const std::string
> &fileName,
> 
> versus:
> 
>           (gdb) set displaced-stepping 1
>           (gdb) si
>           Sending packet: $m1508,4#9b...Packet received: 3c000058
>           Sending packet: $m832d4,4#02...Packet received: 7c0802a6
>           Sending packet: $X1508,0:#bc...Packet received:
>           binary downloading NOT supported by target
>           Sending packet: $M1508,4:7c0802a6#b0...Packet received: OK
>           Sending packet: $g#67...Packet received:
> 00dcdea000dcde4000dca11800dcdea400dcdea00000014200dcde68000000020000000500dcdea400579e9000ed4b3400000000f7ffffff00000357f101000000000000fff716580068e738001dd96400000000000000000069c1200000000000000037000000000000003200e6b2c000087e4800ed12a000ed1b6800dcde7800000006347fd0133ff199999999999a3ff199999999999a000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000401900a3d70a3d713ff199999999999a000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000832d40008a0302000000400049cd00000000000000000efbeadde
>           Sending packet: $P40=00001508#7f...Packet received: OK
>           Packet P (set-register) is supported
>           Sending packet: $vCont;s:p1.dca118#83...Packet received: OK
>             Notification received:
> Stop:T05thread:p1.dca118;0:00049cd0;1:00dcde40;2:00dca118;3:00dcdea4;4:00dcdea0;5:00000142;6:00dcde68;7:00000002;8:00000005;9:00dcdea4;a:00579e90;b:00ed4b34;c:00000000;d:f7ffffff;e:00000357;f:f1010000;10:00000000;11:fff71658;12:0068e738;13:001dd964;14:00000000;15:00000000;16:0069c120;17:00000000;18:00000037;19:00000000;1a:00000032;1b:00e6b2c0;1c:00087e48;1d:00ed12a0;1e:00ed1b68;1f:00dcde78;20:00000006347fd013;21:3ff199999999999a;22:3ff199999999999a;23:0000000000000000;24:0000000000000000;25:0000000000000000;26:0000000000000000;27:0000000000000000;28:0000000000000000;29:0000000000000000;2a:0000000000000000;2b:0000000000000000;2c:401900a3d70a3d71;2d:3ff199999999999a;2e:0000000000000000;2f:0000000000000000;30:0000000000000000;31:0000000000000000;32:0000000000000000;33:0000000000000000;34:0000000000000000;35:0000000000000000;36:0000000000000000;37:0000000000000000;38:0000000000000000;39:0000000000000000;3a:0000000000000000;3b:0000000000000000;3c:0000000000000000;3d:0000000000000000;3e:0
000000000000000;3f:0000000000000000;46:efbeadde;40:0000150c;41:0008a030;42:20000004;43:00049cd0;44:00000000;45:00000000;
>           Sending packet: $vStopped#55...Packet received: OK
>           Sending packet: $M1508,4:3c000058#78...Packet received: OK
>           Sending packet: $P40=000832d8#ba...Packet received: OK
>           0x000832d8    20    SourceLine::SourceLine( const std::string
> &fileName,
>           (gdb)

Without displaced stepping, when GDB steps over breakpoint for one
thread, other threads will be affected.  With displaced stepping, when
GDB steps over breakppoint, other threads won't be affected, but as you
say, there are more RSP traffic.  We need to balance on thread-stop
delay vs. rsp traffic delay.  Probably, you can try to disable
displaced stepping, to see if it helps to speed up.

> 
> Issue b)
> According to my current understanding (please correct me if I'm wrong):
> Range stepping decides whether to send a 'vCont;r' package by comparing
> the current program counter with the range that needs to be executed
> (control.step_range_start and control.step_range_end). Using displaced

Yes, you are right.

> single stepping means that the program counter is, by definition, always
> outside the range that needs to be single stepped and thus 'range
> stepping' will never be used.

Once GDB decides to do displaced stepping, it will place equivalent
instructions at scratch pad to emulate the original instruction,
resumes inferior from scratch pad, and expects the stop.  That means
GDB is aware of every stop after the execution of each instruction, so
range stepping can't be used.  The rationale of range stepping is that,
GDB tells the remote stub to do single in a range of address, and
report a stop once PC is out of the range, in order to reduce the rsp
traffic.  During the range stepping, GDB is not aware of how many stops
occurred in the range.

> 
> What causes the problem?
> The following statement (from infrun.c, resume(), git master) decides
> whether to use displaced single stepping or not:
>     if (use_displaced_stepping (gdbarch)
>         && (tp->control.trap_expected
>         || (step && gdbarch_software_single_step_p (gdbarch)))
>         && sig == GDB_SIGNAL_0
>         && !current_inferior ()->waiting_for_vfork_done)
> 
> According to my experiments:
>       - Using gdb/gdbserver on x86/64, that statement is 'true' when we
> step over a breakpoint, but is 'false' otherwise ->range stepping is
> used when possible
>       - Using extended remote to ppc603/EABI embedded system, that
> statement is always 'true' when we step because
> 'gdbarch_software_single_step_p' returns 'true' ->range stepping is
> never used

Once GDB decides to do software single step, it will insert a software
single step breakpoint at the next instruction of PC, resume inferior,
and expect a stop on that address.  This will cause a lot of RSP
traffic, which can't be reduced.  Then, range stepping is useless.

>       - When patching GDB so that 'gdbarch->software_single_step = NULL',
> then the behavior is like on x86/64 and thus 'range stepping' works
> ->range stepping is used when possible
> 
> 
> Finally, my request: Could someone with more insight into GDB internals
> please decide whether we have to fix something here and if yes, how? Or,
> in case that my conclusions are wrong, could you help me to really
> understand the problem? (FYI: My ultimate goal is to speedup remote
> debugging for our ppc603/EABI target.)

Considering your goal, I suggest that please make range stepping on,
and probably turn displaced stepping off.  If you are using breakpoint
with conditions, please 'set breakpoint condition-evaluation target'
(it requires your remote stub support condition evaluation), it will
improve the performance to some extent.

On the other hand, we have a local patch to prefer range stepping to
software single step:

diff --git a/gdb/infrun.c b/gdb/infrun.c
index cbc11f7..b606384 100644
--- a/gdb/infrun.c
+++ b/gdb/infrun.c
@@ -1824,7 +1824,9 @@ a command like `return' or `jump' to continue execution."));
     }
 
   /* Do we need to do it the hard way, w/temp breakpoints?  */
-  else if (step)
+  else if (step
+	   && !(target supports range stepping () /* undefined */
+		&& THREAD_WITHIN_SINGLE_STEP_RANGE (tp, pc)))
     step = maybe_software_singlestep (gdbarch, pc);
 
   /* Currently, our software single-step implementation leads to different

target_supports_range_stepping can be hacked to 1 in your case.  It
helps in your case that the remote stub is able to do single step
while the GDB side uses software single step.

My range stepping series didn't include this patch above, because it is a
step-2" patch series.  My plan is to draft and post the "step-2"
patches series after getting some comments on range stepping patches.
I'd like to get the existing patches reviewed and committed first, and the
start to work on "step-2" patches.

B.T.W, we are looking for the opportunities to improve the performance
of remote debugging, "target-side condition evaluation" and
"range stepping" are of this kind.  If you know some cases are extremely
slow, please let us know.

-- 
Yao (齐尧)

Re: Unneeded displaced single stepping causes issues: Performance drop and range stepping killer

[Tom Tromey]

>>>>> "Raphael" == Raphael Zulliger <zulliger@indel.ch> writes:

Raphael> versus:

Raphael>         (gdb) set displaced-stepping 1
Raphael>         (gdb) si
Raphael>         Sending packet: $m1508,4#9b...Packet received: 3c000058
Raphael>         Sending packet: $m832d4,4#02...Packet received: 7c0802a6
Raphael>         Sending packet: $X1508,0:#bc...Packet received:
Raphael>         binary downloading NOT supported by target
Raphael>         Sending packet: $M1508,4:7c0802a6#b0...Packet received: OK
Raphael>         Sending packet: $g#67...Packet received:
[...]

I guess this is copying the instruction and stepping it?  I wonder if we
could put more information about displaced stepping into the remote
protocol, to avoid the extra traffic.

Tom

Re: Unneeded displaced single stepping causes issues: Performance drop and range stepping killer

[Raphael Zulliger]

On 04/24/2013 04:24 AM, Yao Qi wrote:
> Considering your goal, I suggest that please make range stepping on,
> and probably turn displaced stepping off.
I'm definitely going to turn range stepping on, as we benefit much from it!

Turning off displaced stepping is not an option for our system 
(multithreaded
real-time embedded), as it would mean that other threads could miss 
breakpoints
(which got temporarily removed by the single-stepping mechanism)
> If you are using breakpoint
> with conditions, please 'set breakpoint condition-evaluation target'
> (it requires your remote stub support condition evaluation), it will
> improve the performance to some extent.
>
> On the other hand, we have a local patch to prefer range stepping to
> software single step:
>
> diff --git a/gdb/infrun.c b/gdb/infrun.c
> index cbc11f7..b606384 100644
> --- a/gdb/infrun.c
> +++ b/gdb/infrun.c
> @@ -1824,7 +1824,9 @@ a command like `return' or `jump' to continue execution."));
>       }
>   
>     /* Do we need to do it the hard way, w/temp breakpoints?  */
> -  else if (step)
> +  else if (step
> +	   && !(target supports range stepping () /* undefined */
> +		&& THREAD_WITHIN_SINGLE_STEP_RANGE (tp, pc)))
>       step = maybe_software_singlestep (gdbarch, pc);
>   
>     /* Currently, our software single-step implementation leads to different
Unfortunately, this doesn't help. In my use-case in which "unwanted" 
displaced
single-stepping happens, we already took the "if" path of the "else if" 
you're
mentioning here :-(.
>
> target_supports_range_stepping can be hacked to 1 in your case.  It
> helps in your case that the remote stub is able to do single step
> while the GDB side uses software single step.
I don't understand this. (Maybe I already misunderstood the intention of
the small patch above?).
We use our own gdbstub implementation (completely independent from
gdbserver). In order to try the range stepping patches, I already extended
our stub to return "vCont;c;C;s;S;t;r"
>
> My range stepping series didn't include this patch above, because it is a
> step-2" patch series.  My plan is to draft and post the "step-2"
> patches series after getting some comments on range stepping patches.
> I'd like to get the existing patches reviewed and committed first, and the
> start to work on "step-2" patches.
>
> B.T.W, we are looking for the opportunities to improve the performance
> of remote debugging, "target-side condition evaluation" and
> "range stepping" are of this kind.  If you know some cases are extremely
> slow, please let us know.
- "target-side condition evaluation" is certainly something we'd love to
have in our system... on the other side, using conditional breakpoints
is not the main use case when debugging our system so far. (Probably
because we didn't have that feature at all in the past).
- There's something I hope we could improve (but I haven't looked
into the code yet): When GDB receives a stop reply, it issues many "memory
reads". Sometimes it read consecutive memory by multiple RSP packages.
Sometime it even reads the same memory region twice. The following
shows a typical sequence after 1 single-step (or range step) on our system

Sending packet: $vCont;r49c38,49c40:p1.dca318#1f...Packet received: OK
   Notification received: 
Stop:T05thread:p1.dca318;0:00000002;1:00dce078;2:00dca318;3:00dce09c;4:d8434b16;5:005a21b4;6:00dce098;7:00000002;8:00000005;9:000002e4;a:d8434b16;b:00dce050;c:00000000;d:f5ffffff;e:00000357;f:f1010000;10:00000000;11:fff71658;12:0068e9f0;13:001dd964;14:00000000;15:00000000;16:0069c3d8;17:00000000;18:00000037;19:00000000;1a:00000032;1b:00e6b4c0;1c:00087e48;1d:00ecd2d0;1e:00ecdb98;1f:00dce078;20:00000003d8434b16;21:3ff199999999999a;22:3ff199999999999a;23:0000000000000000;24:0000000000000000;25:0000000000000000;26:0000000000000000;27:0000000000000000;28:0000000000000000;29:0000000000000000;2a:0000000000000000;2b:0000000000000000;2c:401900a3d70a3d71;2d:3ff199999999999a;2e:0000000000000000;2f:0000000000000000;30:0000000000000000;31:0000000000000000;32:0000000000000000;33:0000000000000000;34:0000000000000000;35:0000000000000000;36:0000000000000000;37:0000000000000000;38:0000000000000000;39:0000000000000000;3a:0000000000000000;3b:0000000000000000;3c:0000000000000000;3d:0000000000000000;3e:0000000000000000;3f:0000000000000000;46:efbeadde;40:00049c40;41:0008a030;42:20000008;43:00049d58;44:00000000;45:00000000;
Sending packet: $vStopped#55...Packet received: OK
Sending packet: $mdce0c0,40#ec...Packet received: 
00e6b488000020300000a03000dce0d000dce0e8004a066800e6b4c0000002020000000100dce0e800dce100004d0aa800dce45000ecd90000b36b7400dce100
Sending packet: $mdce0c0,40#ec...Packet received: 
00e6b488000020300000a03000dce0d000dce0e8004a066800e6b4c0000002020000000100dce0e800dce100004d0aa800dce45000ecd90000b36b7400dce100
Sending packet: $mdce100,40#ba...Packet received: 
00dce3780008148800ecd9c400000037000000000000003200e6b4c000087e4800dce35800dce35800dce2f800dce154001c748c00dce4008080808064630000
Sending packet: $mdce340,40#c0...Packet received: 
0000a030000814580000000400dce3500000000000dce35800ecd2d000dce45000dce40000dce36800dce39000ecd2d000ecdb9800dce37800dce390004cbc38
Sending packet: $mdce380,40#c4...Packet received: 
00ecdb9800dce3ac00dce3a000dce39000dce3f8000830e8ffffffff0000000100dce45000ecdb9801dce3b800ecdb9800ecd9280000000800ecd97000ecd970
Sending packet: $mdce3c0,40#ef...Packet received: 
00ecdb7000ecd93400ecd97400ecd97000ecdb7000ecd93400e6b6a800dce45000dce40000e6b4c000dce4f40000003500e6b4c000dce3f800dce42800087ec8
Sending packet: $mdce400,40#bd...Packet received: 
00e6b4c000dce4f40063d47800dce44c00dce4f400dce45000e6b4c000dce44c00e6b4c000dce42800dce4a8000847b000ecce3c00ecd34c00ecd38800000000
Sending packet: $mdce480,40#c5...Packet received: 
00e6b4c000dce4f4000000350000000000dce4a800377a5000000000000000000000006400dce4a800dce4c00008522800dce4d000dce4f40000006400dce4c0
Sending packet: $mdce4c0,40#f0...Packet received: 
00dce7980007ec586464646464646464005cc40000e6b4c000dce4f06464646400ecc92064646464005c481800dce4f000dce508005ccb0000eccdb800eccdf0
Sending packet: $mdce780,40#c8...Packet received: 
00dce79800dce6d000dca31800e6b4c00000006400dce79800dce7b00007e89c00dca31800ecc7f80000006400dce7b000dcede80034288400dce99000dce930

(Note that this comes from a slightly patched gdb. But I'm pretty sure 
theses patches shouldn't have any influence on these memory reads)

Thanks,
Raphael

Re: Unneeded displaced single stepping causes issues: Performance drop and range stepping killer

[Raphael Zulliger]

On 04/24/2013 06:53 AM, Tom Tromey wrote:
>>>>>> "Raphael" == Raphael Zulliger <zulliger@indel.ch> writes:
> Raphael> versus:
>
> Raphael>         (gdb) set displaced-stepping 1
> Raphael>         (gdb) si
> Raphael>         Sending packet: $m1508,4#9b...Packet received: 3c000058
> Raphael>         Sending packet: $m832d4,4#02...Packet received: 7c0802a6
> Raphael>         Sending packet: $X1508,0:#bc...Packet received:
> Raphael>         binary downloading NOT supported by target
> Raphael>         Sending packet: $M1508,4:7c0802a6#b0...Packet received: OK
> Raphael>         Sending packet: $g#67...Packet received:
> [...]
>
> I guess this is copying the instruction and stepping it?
Yes.
>   I wonder if we
> could put more information about displaced stepping into the remote
> protocol, to avoid the extra traffic.
Good point! Something like:
   vCont;d:p1.dca118
Would perfectly work for our system. (For our system it'd be required 
that GDB does the displaced step fixup, as we do not (yet) support this 
on the target side - others systems may even avoid that step).

Speaking in RSP, it could then look like this:
         (gdb) si
         Sending packet: $vCont;d:p1.dca118#83...Packet received: OK
           Notification received: 
Stop:T05thread:p1.dca118;0:00049cd0;1:00dcde40;2:00dca118;3:00dcdea4;4:00dcdea0;5:00000142;6:00dcde68;7:00000002;8:00000005;9:00dcdea4;a:00579e90;b:00ed4b34;c:00000000;d:f7ffffff;e:00000357;f:f1010000;10:00000000;11:fff71658;12:0068e738;13:001dd964;14:00000000;15:00000000;16:0069c120;17:00000000;18:00000037;19:00000000;1a:00000032;1b:00e6b2c0;1c:00087e48;1d:00ed12a0;1e:00ed1b68;1f:00dcde78;20:00000006347fd013;21:3ff199999999999a;22:3ff199999999999a;23:0000000000000000;24:0000000000000000;25:0000000000000000;26:0000000000000000;27:0000000000000000;28:0000000000000000;29:0000000000000000;2a:0000000000000000;2b:0000000000000000;2c:401900a3d70a3d71;2d:3ff199999999999a;2e:0000000000000000;2f:0000000000000000;30:0000000000000000;31:0000000000000000;32:0000000000000000;33:0000000000000000;34:0000000000000000;35:0000000000000000;36:0000000000000000;37:0000000000000000;38:0000000000000000;39:0000000000000000;3a:0000000000000000;3b:0000000000000000;3c:0000000000000000;3d:0000000000000000;3e:0000000000000000;3f:0000000000000000;46:efbeadde;40:0000150c;41:0008a030;42:20000004;43:00049cd0;44:00000000;45:00000000;
         Sending packet: $vStopped#55...Packet received: OK
         Sending packet: $P40=000832d8#ba...Packet received: OK

vs:

  (gdb) si
         Sending packet: $m1508,4#9b...Packet received: 3c000058
         Sending packet: $m832d4,4#02...Packet received: 7c0802a6
         Sending packet: $M1508,4:7c0802a6#b0...Packet received: OK
         Sending packet: $g#67...Packet received: 
00dcdea000dcde4000dca11800dcdea400dcdea00000014200dcde68000000020000000500dcdea400579e9000ed4b3400000000f7ffffff00000357f101000000000000fff716580068e738001dd96400000000000000000069c1200000000000000037000000000000003200e6b2c000087e4800ed12a000ed1b6800dcde7800000006347fd0133ff199999999999a3ff199999999999a000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000401900a3d70a3d713ff199999999999a000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000832d40008a0302000000400049cd00000000000000000efbeadde
         Sending packet: $P40=00001508#7f...Packet received: OK
         Sending packet: $vCont;s:p1.dca118#83...Packet received: OK
           Notification received: 
Stop:T05thread:p1.dca118;0:00049cd0;1:00dcde40;2:00dca118;3:00dcdea4;4:00dcdea0;5:00000142;6:00dcde68;7:00000002;8:00000005;9:00dcdea4;a:00579e90;b:00ed4b34;c:00000000;d:f7ffffff;e:00000357;f:f1010000;10:00000000;11:fff71658;12:0068e738;13:001dd964;14:00000000;15:00000000;16:0069c120;17:00000000;18:00000037;19:00000000;1a:00000032;1b:00e6b2c0;1c:00087e48;1d:00ed12a0;1e:00ed1b68;1f:00dcde78;20:00000006347fd013;21:3ff199999999999a;22:3ff199999999999a;23:0000000000000000;24:0000000000000000;25:0000000000000000;26:0000000000000000;27:0000000000000000;28:0000000000000000;29:0000000000000000;2a:0000000000000000;2b:0000000000000000;2c:401900a3d70a3d71;2d:3ff199999999999a;2e:0000000000000000;2f:0000000000000000;30:0000000000000000;31:0000000000000000;32:0000000000000000;33:0000000000000000;34:0000000000000000;35:0000000000000000;36:0000000000000000;37:0000000000000000;38:0000000000000000;39:0000000000000000;3a:0000000000000000;3b:0000000000000000;3c:0000000000000000;3d:0000000000000000;3e:0000000000000000;3f:0000000000000000;46:efbeadde;40:0000150c;41:0008a030;42:20000004;43:00049cd0;44:00000000;45:00000000;
         Sending packet: $vStopped#55...Packet received: OK
         Sending packet: $M1508,4:3c000058#78...Packet received: OK
         Sending packet: $P40=000832d8#ba...Packet received: OK

That would be quite an improvement!

While we are at it... I've got another note related to single-stepping: 
For our system, once a 'vCont;s:' returns with "OK", the single-step has 
already been performed. Maybe this is also the case for the majority of 
the systems out there? For systems like ours, we could get rid of the 
stop notification, by returning the content of the stop reply (sig 
number & registers) right in the vCont reply. (And, of course, the same 
would be true for range stepping.) That would again reduce the 
communication to something like:

         (gdb) si
         Sending packet: $vCont;d:p1.dca118#83...Packet received: 
T05thread:p1.dca118;0:00049cd0;1:00dcde40;2:00dca118;3:00dcdea4;4:00dcdea0;5:00000142;6:00dcde68;7:00000002;8:00000005;9:00dcdea4;a:00579e90;b:00ed4b34;c:00000000;d:f7ffffff;e:00000357;f:f1010000;10:00000000;11:fff71658;12:0068e738;13:001dd964;14:00000000;15:00000000;16:0069c120;17:00000000;18:00000037;19:00000000;1a:00000032;1b:00e6b2c0;1c:00087e48;1d:00ed12a0;1e:00ed1b68;1f:00dcde78;20:00000006347fd013;21:3ff199999999999a;22:3ff199999999999a;23:0000000000000000;24:0000000000000000;25:0000000000000000;26:0000000000000000;27:0000000000000000;28:0000000000000000;29:0000000000000000;2a:0000000000000000;2b:0000000000000000;2c:401900a3d70a3d71;2d:3ff199999999999a;2e:0000000000000000;2f:0000000000000000;30:0000000000000000;31:0000000000000000;32:0000000000000000;33:0000000000000000;34:0000000000000000;35:0000000000000000;36:0000000000000000;37:0000000000000000;38:0000000000000000;39:0000000000000000;3a:0000000000000000;3b:0000000000000000;3c:0000000000000000;3d:0000000000000000;3e:0000000000000000;3f:0000000000000000;46:efbeadde;40:0000150c;41:0008a030;42:20000004;43:00049cd0;44:00000000;45:00000000;
         Sending packet: $P40=000832d8#ba...Packet received: OK

>
> Tom
>

Re: Unneeded displaced single stepping causes issues: Performance drop and range stepping killer

[Raphael Zulliger]

On 04/23/2013 09:03 PM, Pedro Alves wrote:
> On 04/22/2013 11:54 AM, Raphael Zulliger wrote:
>
>> According to my current understanding (please correct me if I'm wrong): Range stepping decides whether to send a 'vCont;r' package by comparing the current program counter with the range that needs to be executed (control.step_range_start and control.step_range_end). Using displaced single stepping means that the program counter is, by definition, always outside the range that needs to be single stepped and thus 'range stepping' will never be used.
> I've been hacking on the range stepping series, and I've actually
> changed how GDB decides to issue a range-step or not.
> By "PC-is-in-range" has problems.  But, that's largely unrelated.
> The root issue is the forced displaced step, and that'll remain
> the same.
If you have something working, I'd like to try it out.
>
>> What causes the problem?
>> The following statement (from infrun.c, resume(), git master) decides whether to use displaced single stepping or not:
>>    if (use_displaced_stepping (gdbarch)
>>        && (tp->control.trap_expected
>>        || (step && gdbarch_software_single_step_p (gdbarch)))
>>        && sig == GDB_SIGNAL_0
>>        && !current_inferior ()->waiting_for_vfork_done)
>>
>> According to my experiments:
>>      - Using gdb/gdbserver on x86/64, that statement is 'true' when we step over a breakpoint, but is 'false' otherwise ->range stepping is used when possible
>>      - Using extended remote to ppc603/EABI embedded system, that statement is always 'true' when we step because 'gdbarch_software_single_step_p' returns 'true' ->range stepping is never used
>>      - When patching GDB so that 'gdbarch->software_single_step = NULL', then the behavior is like on x86/64 and thus 'range stepping' works ->range stepping is used when possible
>>
>>
>> Finally, my request: Could someone with more insight into GDB internals please decide whether we have to fix something here and if yes, how? Or, in case that my conclusions are wrong, could you help me to really understand the problem? (FYI: My ultimate goal is to speedup remote debugging for our ppc603/EABI target.)
> There are several things to consider here.  The
> "(step && gdbarch_software_single_step_p (gdbarch))" part
> was added because GDB can't currently do multiple software
> single-steps simultaneously, which is a problem for non-stop
> mode (which enables displaced stepping).  The workaround was to force
> displaced stepping for all single-steps.  GDB knows how to queue
> displaced step requests (necessary because there's only one scratch
> pad for all threads), and have threads wait for their turn in the
> queue.  I was working on making GDB aware of multiple software
> single-steps a while ago, but I got distracted with other things
> and haven't finished that yet...
Thanks for this useful clarification. (Unfortunately, it
was not obvious to me by reading the code)
>
> Then, as you note, gdbarch_software_single_step_p isn't really the
> right predicate.  A better one would be "do I need to software
> single-step this instruction".  There's no such predicate today,
> because the gdbarch_software_single_step hook is responsible for
> installing the breakpoints itself, instead of returning a vector
> of the locations where breakpoints should be set (which I think
> would be better).
>
Sounds good.

Re: Unneeded displaced single stepping causes issues: Performance drop and range stepping killer

[Doug Evans]

On Tue, Apr 23, 2013 at 11:58 PM, Raphael Zulliger <zulliger@indel.ch> wrote:
> - There's something I hope we could improve (but I haven't looked
> into the code yet): When GDB receives a stop reply, it issues many "memory
> reads". Sometimes it read consecutive memory by multiple RSP packages.
> Sometime it even reads the same memory region twice.

It's unfortunate that gdb's remote caching support doesn't work better
"out of the box".

Re: Unneeded displaced single stepping causes issues: Performance drop and range stepping killer

[Yao Qi]

On 04/24/2013 02:58 PM, Raphael Zulliger wrote:
> Sometimes it read consecutive memory by multiple RSP packages.
> Sometime it even reads the same memory region twice. The following
> shows a typical sequence after 1 single-step (or range step) on our system
>
> Sending packet: $vCont;r49c38,49c40:p1.dca318#1f...Packet received: OK
>     Notification received:
> Stop:T05thread:p1.dca318;0:00000002;1:00dce078;2:00dca318;3:00dce09c;4:d8434b16;5:005a21b4;6:00dce098;7:00000002;8:00000005;9:000002e4;a:d8434b16;b:00dce050;c:00000000;d:f5ffffff;e:00000357;f:f1010000;10:00000000;11:fff71658;12:0068e9f0;13:001dd964;14:00000000;15:00000000;16:0069c3d8;17:00000000;18:00000037;19:00000000;1a:00000032;1b:00e6b4c0;1c:00087e48;1d:00ecd2d0;1e:00ecdb98;1f:00dce078;20:00000003d8434b16;21:3ff199999999999a;22:3ff199999999999a;23:0000000000000000;24:0000000000000000;25:0000000000000000;26:0000000000000000;27:0000000000000000;28:0000000000000000;29:0000000000000000;2a:0000000000000000;2b:0000000000000000;2c:401900a3d70a3d71;2d:3ff199999999999a;2e:0000000000000000;2f:0000000000000000;30:0000000000000000;31:0000000000000000;32:0000000000000000;33:0000000000000000;34:0000000000000000;35:0000000000000000;36:0000000000000000;37:0000000000000000;38:0000000000000000;39:0000000000000000;3a:0000000000000000;3b:0000000000000000;3c:0000000000000000;3d:0000000000000000;3e:0
000000000000000;3f:0000000000000000;46:efbeadde;40:00049c40;41:0008a030;42:20000008;43:00049d58;44:00000000;45:00000000;
> Sending packet: $vStopped#55...Packet received: OK
> Sending packet: $mdce0c0,40#ec...Packet received:
> 00e6b488000020300000a03000dce0d000dce0e8004a066800e6b4c0000002020000000100dce0e800dce100004d0aa800dce45000ecd90000b36b7400dce100
> Sending packet: $mdce0c0,40#ec...Packet received:
> 00e6b488000020300000a03000dce0d000dce0e8004a066800e6b4c0000002020000000100dce0e800dce100004d0aa800dce45000ecd90000b36b7400dce100
> Sending packet: $mdce100,40#ba...Packet received:
> 00dce3780008148800ecd9c400000037000000000000003200e6b4c000087e4800dce35800dce35800dce2f800dce154001c748c00dce4008080808064630000
> Sending packet: $mdce340,40#c0...Packet received:
> 0000a030000814580000000400dce3500000000000dce35800ecd2d000dce45000dce40000dce36800dce39000ecd2d000ecdb9800dce37800dce390004cbc38
> Sending packet: $mdce380,40#c4...Packet received:
> 00ecdb9800dce3ac00dce3a000dce39000dce3f8000830e8ffffffff0000000100dce45000ecdb9801dce3b800ecdb9800ecd9280000000800ecd97000ecd970
> Sending packet: $mdce3c0,40#ef...Packet received:
> 00ecdb7000ecd93400ecd97400ecd97000ecdb7000ecd93400e6b6a800dce45000dce40000e6b4c000dce4f40000003500e6b4c000dce3f800dce42800087ec8
> Sending packet: $mdce400,40#bd...Packet received:
> 00e6b4c000dce4f40063d47800dce44c00dce4f400dce45000e6b4c000dce44c00e6b4c000dce42800dce4a8000847b000ecce3c00ecd34c00ecd38800000000
> Sending packet: $mdce480,40#c5...Packet received:
> 00e6b4c000dce4f4000000350000000000dce4a800377a5000000000000000000000006400dce4a800dce4c00008522800dce4d000dce4f40000006400dce4c0
> Sending packet: $mdce4c0,40#f0...Packet received:
> 00dce7980007ec586464646464646464005cc40000e6b4c000dce4f06464646400ecc92064646464005c481800dce4f000dce508005ccb0000eccdb800eccdf0
> Sending packet: $mdce780,40#c8...Packet received:
> 00dce79800dce6d000dca31800e6b4c00000006400dce79800dce7b00007e89c00dca31800ecc7f80000006400dce7b000dcede80034288400dce99000dce930
>
> (Note that this comes from a slightly patched gdb. But I'm pretty sure
> theses patches shouldn't have any influence on these memory reads)

These memory reads are quite target specific, and I can't see similar 
pattern on x86.  However, I find some duplicated reads on x86 caused by 
GDB calling 'gdbarch_skip_prologue', which reads several instructions 
from the remote target.  Please try command 'set trust-readonly-sections 
on' 
(http://sourceware.org/gdb/current/onlinedocs/gdb/Files.html#index-set-trust_002dreadonly_002dsections-1073), 
by which GDB will fetch instructions from sections instead of the remote 
target.  It helps to reduce the number of rsp packets in traffic.  Hope 
it helps.

-- 
Yao (齐尧)