remote debugging a multi-threaded program with signal

Atsushi Nemoto
2004-03-03T15:59:04Z

[-- Attachment #1: Type: Text/Plain, Size: 3040 bytes --]

There is a program on remote-debugging a multi-threaded program which
uses signals. After receiving a signal (configured to "nostop"), the
thread which receives it resumes normally but other threads leave
stopped.

I'm using gdb/snapshots/branch/gdb-6.0.90.tar.bz2 (Mar 1) on i386-linux.

The test program to reproduce this problem is attached (sigtest.c).
It creates a thread which waits SIGUSR1 and main thread sends SIGUSR1
to the thread every 10 secs. Usual output is:

$ ./sigtest
[sigtest:26616:1024]:main
[sigtest:26618:1026]:func
[sigtest:26616:1024]:send SIGUSR1
[sigtest:26618:1026]:sigwait 10
[sigtest:26616:1024]:send SIGUSR1
[sigtest:26618:1026]:sigwait 10
...

gdb debug session log is:

(gdb) target remote localhost:50000
Remote debugging using localhost:50000
0x080480e0 in _start ()
(gdb) handle SIGUSR1 nostop
Signal Stop Print Pass to program Description
SIGUSR1 No Yes Yes User defined signal 1
(gdb) c
Continuing.
[New Thread 1024]
[New Thread 1026]

and gdbserver side log is:

$ ./gdbserver host:50000 ./sigtest
Process ./sigtest created; pid = 26599
Listening on port 50000
Remote debugging from host 127.0.0.1
[sigtest:26599:1024]:main
[sigtest:26601:1026]:func
[sigtest:26599:1024]:send SIGUSR1
[sigtest:26601:1026]:sigwait 10
(no more output)

ps output is:

$ ps a
PID TTY STAT TIME COMMAND
26598 pts/4 S 0:00 ./gdbserver host:50000 ./sigtest
26599 pts/4 T 0:00 ./sigtest
26600 pts/4 T 0:00 ./sigtest
26601 pts/4 S 0:00 ./sigtest

PID 26601 is a thread which receives SIGUSR1. This was resumed but
others are still stopped.

I tried "set remote verbose-resume-packet off" and got same results.

It seems gdbserver does not handle 'continue with signal' case
correctly. I tried to fix this problem and create a patch below, and
it seems work, but I'm not sure this is a correct fix (or correct
place).

diff -up gdb-6.0.90.org/gdb/gdbserver/server.c gdb-6.0.90/gdb/gdbserver/server.c
--- gdb-6.0.90.org/gdb/gdbserver/server.c Mon Mar 1 01:49:38 2004
+++ gdb-6.0.90/gdb/gdbserver/server.c Wed Mar 3 23:49:42 2004
@@ -235,6 +235,8 @@ handle_v_cont (char *own_buf, char *stat
cont_thread = resume_info[0].thread;
else
cont_thread = -1;
+ if (cont_thread != -1 && n == 1 && !resume_info[0].step)
+ resume_info[i].leave_stopped = 0; /* signal the thread and run others */
set_desired_inferior (0);

(*the_target->resume) (resume_info);
@@ -292,7 +294,7 @@ myresume (int step, int sig)
resume_info[n].thread = -1;
resume_info[n].step = 0;
resume_info[n].sig = 0;
- resume_info[n].leave_stopped = (cont_thread > 0);
+ resume_info[n].leave_stopped = step && (cont_thread > 0);

(*the_target->resume) (resume_info);
}

Other attached files are some session logs for gdb (set debug remote
1) and gdbserver (debug_thread = 1).

gdb.output1 and gdbserver.output1 are logs with "verbose-resume on".
gdb.output2 and gdbserver.output2 are logs with "verbose-resume off".

---
Atsushi Nemoto

[-- Attachment #2: sigtest.c --]
[-- Type: Text/Plain, Size: 849 bytes --]

#include <stdio.h>
#include <pthread.h>
#include <signal.h>

static void *func(void *arg)
{
sigset_t sig;
int s;
sigemptyset(&sig);
sigaddset(&sig, SIGUSR1);
pthread_sigmask(SIG_BLOCK, &sig, NULL);
printf("[sigtest:%d:%d]:func\n", getpid(), pthread_self());
sleep(3);
while (1) {
sigwait(&sig, &s);
printf("[sigtest:%d:%d]:sigwait %d\n", getpid(), pthread_self(), s);
}
}

int
main(int argc, char *argv[])
{
pthread_t tid;
sigset_t sig;
int s, i;
sigemptyset(&sig);
sigaddset(&sig, SIGUSR1);
pthread_sigmask(SIG_BLOCK, &sig, NULL);
printf("[sigtest:%d:%d]:main\n", getpid(), pthread_self());
pthread_create(&tid, NULL, func, (void *)0);

while (1) {
for (i = 10; (i = sleep(i)) > 0; ) /* sleep 10 sec */
;
printf("[sigtest:%d:%d]:send SIGUSR1\n", getpid(), pthread_self());
pthread_kill(tid, SIGUSR1);
}
return 0;
}

[-- Attachment #3: gdb.output1 --]
[-- Type: Text/Plain, Size: 2352 bytes --]

(gdb) target remote localhost:50000
Remote debugging using localhost:50000
0x080480e0 in _start ()
(gdb) handle SIGUSR1 nostop
Signal Stop Print Pass to program Description
SIGUSR1 No Yes Yes User defined signal 1
(gdb) set debug remote 1
(gdb) c
Continuing.
Sending packet: $vCont?#49...Ack
Packet received: vCont;c;C;s;S
Packet vCont (verbose-resume) is supported
Sending packet: $vCont;s#b8...Ack
Packet received: T0505:00000000;04:e0fbffbf;08:e2800408;thread:400;
[New Thread 1024]
Sending packet: $g#67...Ack
Packet received: 00000000000000000000000000000000e0fbffbf000000000000000000000000e280040846030000230000002b0000002b0000002b000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000007f03000000000000ffff000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000801f0000ffffffff
Sending packet: $Z0,80480e0,1#ac...Ack
Packet received:
Packet Z0 (software-breakpoint) is NOT supported
Sending packet: $m80480e0,1#63...Ack
Packet received: 31
Sending packet: $X80480e0,0:#87...Ack
Packet received:
binary downloading NOT suppported by target
Sending packet: $M80480e0,1:cc#43...Ack
Packet received: OK
Sending packet: $vCont;c#a8...Ack
Packet received: T1e05:64f87fbf;04:48f87fbf;08:6fe70408;thread:402;
[New Thread 1026]
Sending packet: $g#67...Ack
Packet received: fcffffff0800000088fc7fbfc0f97fbf48f87fbf64f87fbfc0f97fbfc0f97fbf6fe7040897020000230000002b0000002b0000002b000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000007f03000000000000ffff000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000801f0000b3000000
Sending packet: $vCont;C1e:402#ee...Ack

[-- Attachment #4: gdbserver.output1 --]
[-- Type: Text/Plain, Size: 3981 bytes --]

~/work$ ./gdbserver host:50000 ./sigtest
Process ./sigtest created; pid = 26290
Got an event from 26290 (57f)
stop pc (before any decrement) is 080480e0
stop pc (before any decrement) is 080480e0
Hit a (non-reinsert) breakpoint.
Listening on port 50000
Remote debugging from host 127.0.0.1
Writing ffffff80 to 080c06dc
Writing ffffffcc to 0804ac08
Writing 01 to 080b66c8
Resuming, no pending status
Resuming process 26290 (step, signal 0, stop not expected)
stop pc (before any decrement) is 080480e0
Got an event from 26290 (57f)
stop pc (before any decrement) is 080480e2
stop pc (before any decrement) is 080480e2
Hit a (non-reinsert) breakpoint.
Writing resume reply for 1024

Writing ffffffcc to 080480e0
Resuming, no pending status
Resuming process 26290 (continue, signal 0, stop not expected)
stop pc (before any decrement) is 080480e2
[sigtest:26290:1024]:main
Got an event from 26290 (57f)
stop pc (before any decrement) is 0804ac09
stop pc (before any decrement) is 0804ac09
Hit a (non-reinsert) breakpoint.
Thread creation event.
Writing 00 to 080b6acc
Attaching to thread 2049 (LWP 26291)
Writing 01 to 080b6ac8
Writing 55 to 0804ac08
Resuming process 26290 (step, signal 0, stop not expected)
pending reinsert at 0804ac08Checking for breakpoint.
stop pc (before any decrement) is 0804ac09
Removed breakpoint.
set pc to 0804ac08
stop pc (before any decrement) is 0804ac08
Got an event from 26291 (137f)
stop pc (before any decrement) is 0804e76f
Expected stop.
Resuming process 26291 (continue, signal 0, stop not expected)
stop pc (before any decrement) is 0804e76f
Got an event from 26290 (57f)
stop pc (before any decrement) is 0804ac09
stop pc (before any decrement) is 0804ac09
Reinserted breakpoint.
Writing ffffffcc to 0804ac08
Resuming process 26290 (continue, signal 0, stop not expected)
stop pc (before any decrement) is 0804ac09
stop pc (before any decrement) is 0804e76f
Ignored signal 32 for 2049 (LWP 26291).
Resuming process 26291 (continue, signal 32, stop not expected)
stop pc (before any decrement) is 0804e76f
Got an event from 26291 (57f)
stop pc (before any decrement) is 0804ac09
stop pc (before any decrement) is 0804ac09
Hit a (non-reinsert) breakpoint.
Thread creation event.
Writing 00 to bf7ffdcc
Attaching to thread 1026 (LWP 26292)
Writing 01 to bf7ffdc8
Writing 55 to 0804ac08
Resuming process 26291 (step, signal 0, stop not expected)
pending reinsert at 0804ac08Checking for breakpoint.
stop pc (before any decrement) is 0804ac09
Removed breakpoint.
set pc to 0804ac08
stop pc (before any decrement) is 0804ac08
Got an event from 26292 (137f)
stop pc (before any decrement) is 0804e76f
Expected stop.
Resuming process 26292 (continue, signal 0, stop not expected)
stop pc (before any decrement) is 0804e76f
Got an event from 26291 (57f)
stop pc (before any decrement) is 0804ac09
stop pc (before any decrement) is 0804ac09
Reinserted breakpoint.
Writing ffffffcc to 0804ac08
Resuming process 26291 (continue, signal 0, stop not expected)
stop pc (before any decrement) is 0804ac09
stop pc (before any decrement) is 0804e76f
Ignored signal 32 for 1024 (LWP 26290).
Resuming process 26290 (continue, signal 32, stop not expected)
stop pc (before any decrement) is 0804e76f
stop pc (before any decrement) is 0804e76f
Ignored signal 32 for 1026 (LWP 26292).
Resuming process 26292 (continue, signal 32, stop not expected)
stop pc (before any decrement) is 0804e76f
[sigtest:26292:1026]:func
[sigtest:26290:1024]:send SIGUSR1
Got an event from 26292 (a7f)
stop pc (before any decrement) is 0804e76f
Sending sigstop to process 26290
Sending sigstop to process 26291
Got an event from 26290 (137f)
stop pc (before any decrement) is 08053f51
Got an event from 26291 (137f)
stop pc (before any decrement) is 08054a66
Writing resume reply for 1026

Resuming, no pending status
Resuming process 26292 (continue, signal 10, stop not expected)
stop pc (before any decrement) is 0804e76f
[sigtest:26292:1026]:sigwait 10

[-- Attachment #5: gdb.output2 --]
[-- Type: Text/Plain, Size: 2418 bytes --]

(gdb) set remote verbose-resume-packet off
(gdb) target remote localhost:50000
Remote debugging using localhost:50000
0x080480e0 in _start ()
(gdb) handle SIGUSR1 nostop
Signal Stop Print Pass to program Description
SIGUSR1 No Yes Yes User defined signal 1
(gdb) set debug remote 1
(gdb) c
Continuing.
Sending packet: $Hc0#db...Ack
Packet received: OK
Sending packet: $s#73...Ack
Packet received: T0505:00000000;04:e0fbffbf;08:e2800408;thread:400;
[New Thread 1024]
Sending packet: $g#67...Ack
Packet received: 00000000000000000000000000000000e0fbffbf000000000000000000000000e280040846030000230000002b0000002b0000002b000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000007f03000000000000ffff000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000801f0000ffffffff
Sending packet: $Z0,80480e0,1#ac...Ack
Packet received:
Packet Z0 (software-breakpoint) is NOT supported
Sending packet: $m80480e0,1#63...Ack
Packet received: 31
Sending packet: $X80480e0,0:#87...Ack
Packet received:
binary downloading NOT suppported by target
Sending packet: $M80480e0,1:cc#43...Ack
Packet received: OK
Sending packet: $Hc0#db...Ack
Packet received: OK
Sending packet: $c#63...Ack
Packet received: T1e05:64f87fbf;04:48f87fbf;08:6fe70408;thread:402;
[New Thread 1026]
Sending packet: $g#67...Ack
Packet received: fcffffff0800000088fc7fbfc0f97fbf48f87fbf64f87fbfc0f97fbfc0f97fbf6fe7040897020000230000002b0000002b0000002b000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000007f03000000000000ffff000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000801f0000b3000000
Sending packet: $Hc402#41...Ack
Packet received: OK
Sending packet: $C1e#d9...Ack

[-- Attachment #6: gdbserver.output2 --]
[-- Type: Text/Plain, Size: 3789 bytes --]

~/work$ ./gdbserver host:50000 ./sigtest
Process ./sigtest created; pid = 26306
Got an event from 26306 (57f)
stop pc (before any decrement) is 080480e0
stop pc (before any decrement) is 080480e0
Hit a (non-reinsert) breakpoint.
Listening on port 50000
Remote debugging from host 127.0.0.1
Writing ffffff80 to 080c06dc
Writing ffffffcc to 0804ac08
Writing 01 to 080b66c8
Resuming, no pending status
Resuming process 26306 (step, signal 0, stop not expected)
stop pc (before any decrement) is 080480e0
Got an event from 26306 (57f)
stop pc (before any decrement) is 080480e2
stop pc (before any decrement) is 080480e2
Hit a (non-reinsert) breakpoint.
Writing resume reply for 1024

Writing ffffffcc to 080480e0
Resuming, no pending status
Resuming process 26306 (continue, signal 0, stop not expected)
stop pc (before any decrement) is 080480e2
[sigtest:26306:1024]:main
Got an event from 26306 (57f)
stop pc (before any decrement) is 0804ac09
stop pc (before any decrement) is 0804ac09
Hit a (non-reinsert) breakpoint.
Thread creation event.
Writing 00 to 080b6acc
Attaching to thread 2049 (LWP 26307)
Writing 01 to 080b6ac8
Writing 55 to 0804ac08
Resuming process 26306 (step, signal 0, stop not expected)
pending reinsert at 0804ac08Checking for breakpoint.
stop pc (before any decrement) is 0804ac09
Removed breakpoint.
set pc to 0804ac08
stop pc (before any decrement) is 0804ac08
Got an event from 26306 (57f)
stop pc (before any decrement) is 0804ac09
stop pc (before any decrement) is 0804ac09
Reinserted breakpoint.
Writing ffffffcc to 0804ac08
Resuming process 26306 (continue, signal 0, stop not expected)
stop pc (before any decrement) is 0804ac09
Got an event from 26307 (137f)
stop pc (before any decrement) is 0805536a
Expected stop.
Resuming process 26307 (continue, signal 0, stop not expected)
stop pc (before any decrement) is 0805536a
Got an event from 26307 (57f)
stop pc (before any decrement) is 0804ac09
stop pc (before any decrement) is 0804ac09
Hit a (non-reinsert) breakpoint.
Thread creation event.
Writing 00 to bf7ffdcc
Attaching to thread 1026 (LWP 26308)
Writing 01 to bf7ffdc8
Writing 55 to 0804ac08
Resuming process 26307 (step, signal 0, stop not expected)
pending reinsert at 0804ac08Checking for breakpoint.
stop pc (before any decrement) is 0804ac09
Removed breakpoint.
set pc to 0804ac08
stop pc (before any decrement) is 0804ac08
Got an event from 26308 (137f)
stop pc (before any decrement) is 0804e76f
Expected stop.
Resuming process 26308 (continue, signal 0, stop not expected)
stop pc (before any decrement) is 0804e76f
Got an event from 26307 (57f)
stop pc (before any decrement) is 0804ac09
stop pc (before any decrement) is 0804ac09
Reinserted breakpoint.
Writing ffffffcc to 0804ac08
Resuming process 26307 (continue, signal 0, stop not expected)
stop pc (before any decrement) is 0804ac09
stop pc (before any decrement) is 0804e76f
Ignored signal 32 for 1024 (LWP 26306).
Resuming process 26306 (continue, signal 32, stop not expected)
stop pc (before any decrement) is 0804e76f
stop pc (before any decrement) is 0804e76f
Ignored signal 32 for 1026 (LWP 26308).
Resuming process 26308 (continue, signal 32, stop not expected)
stop pc (before any decrement) is 0804e76f
[sigtest:26308:1026]:func
[sigtest:26306:1024]:send SIGUSR1
Got an event from 26308 (a7f)
stop pc (before any decrement) is 0804e76f
Sending sigstop to process 26306
Sending sigstop to process 26307
Got an event from 26306 (137f)
stop pc (before any decrement) is 08053f51
Got an event from 26307 (137f)
stop pc (before any decrement) is 08054a66
Writing resume reply for 1026

Resuming, no pending status
Resuming process 26308 (continue, signal 10, stop not expected)
stop pc (before any decrement) is 0804e76f
[sigtest:26308:1026]:sigwait 10