huge remote debug traffic with multi-thread program

huge remote debug traffic with multi-thread program

[Atsushi Nemoto]

[-- Attachment #1: Type: Text/Plain, Size: 3379 bytes --]

When remote-debugging a multi-thread program on mips-linux, I see huge
remote traffic after interrupting the remote program.

My target environment is mips(el)-linux (Debian sarge) and current gdb
snapshot (20050216).

Here is my test program.

#include <pthread.h>
pthread_mutex_t pm;
static void *func(void *arg)
{
	while (1) {
		pthread_mutex_lock(&pm);
		pthread_mutex_unlock(&pm);
	}
}
int main(int argc, char *argv[])
{
	int i;
	pthread_t tid[2];
	pthread_mutex_init(&pm, NULL);
	for (i = 0; i < 2; i++)
		pthread_create(&tid[i], NULL, func, NULL);
	for (i = 0; i < 2; i++)
		pthread_join(tid[i], NULL);
}

When I typed Ctrl-C and 'c' (continue) on gdb, traffic between gdb and
gdbserver start eating network bandwidth and gdbserver eats whole CPU
power on the target.  Is this a normal behavior?

If I replace last 2 line in main() with following code (i.e. not
use pthread_join), the traffic disappear soon.

	while (1)
		sleep(100);

With gdb 6.3, the traffic disappear soon without this change.


With "set debug remote 1", I got following logs after the 'c' command.
(please refer hostlog-join.txt for complete log)

(gdb) c
Continuing.
Sending packet: $m404e18,4#33...Ack
Packet received: 03e00008
Sending packet: $m404e18,4#33...Ack
Packet received: 03e00008
Sending packet: $m404a78,4#35...Ack
Packet received: 8fbc0010
Sending packet: $M404a78,4:0005000d#08...Ack
Packet received: OK
Sending packet: $vCont;c#a8...Ack
Packet received: T0525:00404a78;1d:7f7ff998;thread:4002;
...
Sending packet: $vCont;c:4002#a8...Ack
Packet received: T0525:00404a7c;1d:7f7ff998;thread:4002;
...
Sending packet: $vCont;c#a8...Ack
Packet received: T0525:00404a78;1d:7f5ff998;thread:8003;
...
Sending packet: $vCont;c:8003#ad...Ack
Packet received: T0525:00404a7c;1d:7f5ff998;thread:8003;
...
Sending packet: $vCont;c#a8...Ack
Packet received: T0525:00404a78;1d:7f7ff998;thread:4002;
...
Sending packet: $vCont;c:4002#a8...Ack
Packet received: T0525:00404a7c;1d:7f7ff998;thread:4002;
...

0x00404a78 and 0x00404a7c are within __pthread_wait_for_restart_signal().

0x00404a70 <__pthread_wait_for_restart_signal+124>:     jalr    t9	# __pthread_sigsuspend()
0x00404a74 <__pthread_wait_for_restart_signal+128>:     nop
0x00404a78 <__pthread_wait_for_restart_signal+132>:     lw      gp,16(sp)
0x00404a7c <__pthread_wait_for_restart_signal+136>:     lw      v1,96(s0)
0x00404a80 <__pthread_wait_for_restart_signal+140>:     lw      v0,-32364(gp)
0x00404a84 <__pthread_wait_for_restart_signal+144>:     nop
0x00404a88 <__pthread_wait_for_restart_signal+148>:     lw      v0,0(v0)
0x00404a8c <__pthread_wait_for_restart_signal+152>:     nop
0x00404a90 <__pthread_wait_for_restart_signal+156>:     bne     v1,v0,0x404a68 <__pthread_wait_for_restart_signal+116>


Using sleep() instead of pthread_join(), I got any packets after
'$vCont' until I type Ctrl-C again.
(please refer hostlog-sleep.txt for complete log)

Attached is a complete logs for both side.  The test program is
statically linked.  This problem is same with dynamically linked
program.

hostlog-join.txt	-- gdb log with 'set debug remote 1' (pthread_join)
serverlog-join.txt	-- gdbserver log with debug_threads = 1 (pthread_join)
hostlog-sleep.txt	-- gdb log with 'set debug remote 1' (sleep)
serverlog-sleep.txt	-- gdbserver log with debug_threads = 1 (sleep)

Could someone look into this?

Thank you.
---
Atsushi Nemoto

[-- Attachment #2: log.tar.gz --]
[-- Type: Application/Octet-Stream, Size: 4483 bytes --]

Re: huge remote debug traffic with multi-thread program

[Daniel Jacobowitz]

On Wed, Feb 16, 2005 at 09:04:32PM +0900, Atsushi Nemoto wrote:
> When I typed Ctrl-C and 'c' (continue) on gdb, traffic between gdb and
> gdbserver start eating network bandwidth and gdbserver eats whole CPU
> power on the target.  Is this a normal behavior?

The normal cause of this sort of problem is GDB's lame software
single-step support.  You have two threads iterating in the same piece
of code, and probably GDB is continually getting a trap from the wrong
one.  Or it may get confused about where it has put the breakpoint. 
Does the problem go away if each thread is in a separate identical copy
of the function?

-- 
Daniel Jacobowitz
CodeSourcery, LLC

Re: huge remote debug traffic with multi-thread program

[Atsushi Nemoto]

>>>>> On Wed, 16 Feb 2005 11:59:09 -0500, Daniel Jacobowitz <drow@false.org> said:
drow> The normal cause of this sort of problem is GDB's lame software
drow> single-step support.  You have two threads iterating in the same
drow> piece of code, and probably GDB is continually getting a trap
drow> from the wrong one.  Or it may get confused about where it has
drow> put the breakpoint.

Thanks for your suggestion.  It looks latter.

drow> Does the problem go away if each thread is in a separate
drow> identical copy of the function?

No, this problem still remains if each thread have its own function.

Looking the log, gdbserver keep getting SIGTRAP on 0x00404a78
(__pthread_wait_for_restart_signal+132).  This is a return address
from __pthread_sigsuspend() which is a stopping place of SIGINT
(Ctrl-C).  I'm wondering why gdb set breakpoint to the return address
of __pthread_sigsuspend() ...


--- excerpt from hostlog-join.txt ---
remote_interrupt called
remote_stop called
Packet received: T0225:00404e18;1d:7fff7a08;thread:4000;

Program received signal SIGINT, Interrupt.
...
Sending packet: $g#67...Ack
Packet received: 0000000010005e18000000047fffffff7fff7a20000000100000000100000001000000018678e008ffffffff00200200001001000000ffff87ff5e74ffffffff1000006010008dc07f7ffbe000004002000000007fff7e04ffffffff000000000000000100404e0000000000000000001000d1507fff7a087fff7b9000404a78000000000000070800001c20004093d01000002000404e18ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff0000000000002d30000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000	# pc=0x00404e18, ra=0x00404a78
...
0x00404e18 in __pthread_sigsuspend (set=0x7fff7a20)
    at ../linuxthreads/sysdeps/unix/sysv/linux/pt-sigsuspend.c:54
54	../linuxthreads/sysdeps/unix/sysv/linux/pt-sigsuspend.c: No such file or directory.
	in ../linuxthreads/sysdeps/unix/sysv/linux/pt-sigsuspend.c
Current language:  auto; currently c
(gdb) c
Continuing.
...
Sending packet: $m404a78,4#35...Ack
Packet received: 8fbc0010
Sending packet: $M404a78,4:0005000d#08...Ack	# set breakpoint on 0x00404a78

---
Atsushi Nemoto

Re: huge remote debug traffic with multi-thread program

[Atsushi Nemoto]

>>>>> On Thu, 17 Feb 2005 10:42:00 +0900 (JST), Atsushi Nemoto <anemo@mba.ocn.ne.jp> said:
anemo> Looking the log, gdbserver keep getting SIGTRAP on 0x00404a78
anemo> (__pthread_wait_for_restart_signal+132).  This is a return
anemo> address from __pthread_sigsuspend() which is a stopping place
anemo> of SIGINT (Ctrl-C).  I'm wondering why gdb set breakpoint to
anemo> the return address of __pthread_sigsuspend() ...

The reason was gdb is doing software single-step on resuming.

The instruction on stop_pc is 'jr ra' in __pthread_sigsuspend.  When
resuming, gdb set breakpoint of the return address to do software
single-stop.

Next question is why gdb try to do single step here.  It seems due to
this code in infrun.c:proceed().

712:      else if (gdbarch_single_step_through_delay_p (current_gdbarch)
713:              && gdbarch_single_step_through_delay (current_gdbarch,
714:                                                    get_current_frame ()))
715:        /* We stepped onto an instruction that needs to be stepped
716:           again before re-inserting the breakpoint, do so.  */
717:        oneproc = 1;

On mips, gdbarch_single_step_through_delay return 1 for 'jr' instruction.

In gdb 6.3, it was:

732:      if (STEP_SKIPS_DELAY_P
732:          && breakpoint_here_p (read_pc () + 4)
732:          && STEP_SKIPS_DELAY (read_pc ()))
732:        oneproc = 1;

and ChangeLog saied:

2004-10-31  Orjan Friberg <organ.friberg@axis.com>
	    Andrew Cagney  <cagney@gnu.org>

	* gdbarch.sh (single_step_through_delay): Add.
	* gdbarch.h, gdbarch.c: Re-generate.
	* config/mips/tm-mips.h (STEP_SKIPS_DELAY_P, STEP_SKIPS_DELAY)
	(mips_step_skips_delay): Delete.
	* mips-tdep.c (mips_single_step_through_delay): Replace
	mips_step_skips_delay.
	(mips_gdbarch_init): Set single_step_through_delay.
	(mips_dump_tdep): Do not print STEP_SKIPS_DELAY.

It seems "breakpoint_here_p (read_pc() + 4)" test was lost.
Is this intentional?

Thank you.
---
Atsushi Nemoto

Re: huge remote debug traffic with multi-thread program

[Atsushi Nemoto]

>>>>> On Fri, 18 Feb 2005 19:33:33 +0900 (JST), Atsushi Nemoto <anemo@mba.ocn.ne.jp> said:
anemo> It seems "breakpoint_here_p (read_pc() + 4)" test was lost.  Is
anemo> this intentional?

This patch fixed my problem.  Please review and apply if appropriate.
Thank you.

--- gdb/mips-tdep.c.orig	2005-02-21 13:31:58.000000000 +0900
+++ gdb/mips-tdep.c	2005-02-22 11:45:07.159141141 +0900
@@ -4148,6 +4148,8 @@ mips_single_step_through_delay (struct g
   if (mips_pc_is_mips16 (pc))
     return 0;
 
+  if (!breakpoint_here_p (pc + 4))
+    return 0;
   if (!safe_frame_unwind_memory (frame, pc, buf, sizeof buf))
     /* If error reading memory, guess that it is not a delayed
        branch.  */

Re: huge remote debug traffic with multi-thread program

[Atsushi Nemoto]

>>>>> On Tue, 22 Feb 2005 16:46:14 +0900 (JST), Atsushi Nemoto <anemo@mba.ocn.ne.jp> said:
anemo> This patch fixed my problem.  Please review and apply if
anemo> appropriate.  Thank you.

Ping.  This patch add a breakpoint test which was lost on 2004-10-31
change:

> 2004-10-31  Orjan Friberg <organ.friberg@axis.com>
> 	    Andrew Cagney  <cagney@gnu.org>
> 
> 	* gdbarch.sh (single_step_through_delay): Add.
> 	* gdbarch.h, gdbarch.c: Re-generate.
> 	* config/mips/tm-mips.h (STEP_SKIPS_DELAY_P, STEP_SKIPS_DELAY)
> 	(mips_step_skips_delay): Delete.
> 	* mips-tdep.c (mips_single_step_through_delay): Replace
> 	mips_step_skips_delay.
> 	(mips_gdbarch_init): Set single_step_through_delay.
> 	(mips_dump_tdep): Do not print STEP_SKIPS_DELAY.

Please review.  Thank you.

--- gdb/mips-tdep.c.orig	2005-02-21 13:31:58.000000000 +0900
+++ gdb/mips-tdep.c	2005-02-22 11:45:07.159141141 +0900
@@ -4148,6 +4148,8 @@ mips_single_step_through_delay (struct g
   if (mips_pc_is_mips16 (pc))
     return 0;
 
+  if (!breakpoint_here_p (pc + 4))
+    return 0;
   if (!safe_frame_unwind_memory (frame, pc, buf, sizeof buf))
     /* If error reading memory, guess that it is not a delayed
        branch.  */

---
Atsushi Nemoto

Re: huge remote debug traffic with multi-thread program

[Bob Rossi]

On Thu, Mar 17, 2005 at 02:43:38PM +0900, Atsushi Nemoto wrote:
> >>>>> On Tue, 22 Feb 2005 16:46:14 +0900 (JST), Atsushi Nemoto <anemo@mba.ocn.ne.jp> said:
> anemo> This patch fixed my problem.  Please review and apply if
> anemo> appropriate.  Thank you.
> 
> Ping.  This patch add a breakpoint test which was lost on 2004-10-31
> change:

Would a testcase help ensure this doesn't get removed again?

Bob Rossi

Re: huge remote debug traffic with multi-thread program

[Mark Kettenis]

   Date: Thu, 17 Mar 2005 14:43:38 +0900 (JST)
   From: Atsushi Nemoto <anemo@mba.ocn.ne.jp>

   >>>>> On Tue, 22 Feb 2005 16:46:14 +0900 (JST), Atsushi Nemoto <anemo@mba.ocn.ne.jp> said:
   anemo> This patch fixed my problem.  Please review and apply if
   anemo> appropriate.  Thank you.

   Please review.  Thank you.

   --- gdb/mips-tdep.c.orig	2005-02-21 13:31:58.000000000 +0900
   +++ gdb/mips-tdep.c	2005-02-22 11:45:07.159141141 +0900
   @@ -4148,6 +4148,8 @@ mips_single_step_through_delay (struct g
      if (mips_pc_is_mips16 (pc))
	return 0;

   +  if (!breakpoint_here_p (pc + 4))
   +    return 0;
      if (!safe_frame_unwind_memory (frame, pc, buf, sizeof buf))
	/* If error reading memory, guess that it is not a delayed
	   branch.  */

Sorry, missed your first post when travelling.  I had the same patch
in my tree for OpenBSD/mips64 support.  There are some problems with
the mips_single_step_through_delay though, and I intended to solve
those first.  However, this doesn't make matters worse, so I've
checked it in.

Thanks,

Mark