* thread-debuging with gdbserver on ARM/NPTL
@ 2006-10-24 18:29 Benedikt Spranger
2006-10-24 18:34 ` Daniel Jacobowitz
0 siblings, 1 reply; 6+ messages in thread
From: Benedikt Spranger @ 2006-10-24 18:29 UTC (permalink / raw)
To: gdb
Hi,
trying to debug a simple multithreaded application with gdbserver on an
ARM eabi/NPTL based System leads to a completely confused gdbserver.
thread_code() {
BP2 do_thread_init();
pthread_lock();
do_thread_work();
pthread_unlock();
}
main() {
pthread_lock()
pthread_create()
BP1 do_something()
pthread_unlock()
}
Attaching gdb to gdbserver, set the above breakpoints and issue "cont",
then the program stops at breakpoint BP1. BP2 is never reached, although
the thread is visible via "ps" on the target system. The thread is stuck
in sys_futex and not as expected in the ptrace breakpoint trap. Also gdb
does not show that the thread was created.
This can be reproduced with glibc 2.3.6 -> 2.5, kernel 2.6.14 ->
2.6.19-rc2, gcc 3.4 -> 4.1.1 and binutils up to cvs-latest.
Benedikt Spranger
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: thread-debuging with gdbserver on ARM/NPTL 2006-10-24 18:29 thread-debuging with gdbserver on ARM/NPTL Benedikt Spranger @ 2006-10-24 18:34 ` Daniel Jacobowitz 2006-10-25 14:46 ` Robert Schwebel 0 siblings, 1 reply; 6+ messages in thread From: Daniel Jacobowitz @ 2006-10-24 18:34 UTC (permalink / raw) To: Benedikt Spranger; +Cc: gdb On Tue, Oct 24, 2006 at 08:29:33PM +0200, Benedikt Spranger wrote: > Attaching gdb to gdbserver, set the above breakpoints and issue "cont", > then the program stops at breakpoint BP1. BP2 is never reached, although > the thread is visible via "ps" on the target system. The thread is stuck > in sys_futex and not as expected in the ptrace breakpoint trap. Also gdb > does not show that the thread was created. > > This can be reproduced with glibc 2.3.6 -> 2.5, kernel 2.6.14 -> > 2.6.19-rc2, gcc 3.4 -> 4.1.1 and binutils up to cvs-latest. Well, it works just fine for me, so you may want to investigate your setup. Have you set solib-absolute-prefix, the most frequent problem? Check if "info shared" and "info threads" show sensible output. -- Daniel Jacobowitz CodeSourcery ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: thread-debuging with gdbserver on ARM/NPTL 2006-10-24 18:34 ` Daniel Jacobowitz @ 2006-10-25 14:46 ` Robert Schwebel 2006-10-25 15:03 ` Daniel Jacobowitz 0 siblings, 1 reply; 6+ messages in thread From: Robert Schwebel @ 2006-10-25 14:46 UTC (permalink / raw) To: Daniel Jacobowitz; +Cc: Benedikt Spranger, gdb, Marc Kleine-Budde, Luotao Fu [-- Attachment #1: Type: text/plain, Size: 1350 bytes --] On Tue, Oct 24, 2006 at 02:34:04PM -0400, Daniel Jacobowitz wrote: > On Tue, Oct 24, 2006 at 08:29:33PM +0200, Benedikt Spranger wrote: > > Attaching gdb to gdbserver, set the above breakpoints and issue "cont", > > then the program stops at breakpoint BP1. BP2 is never reached, although > > the thread is visible via "ps" on the target system. The thread is stuck > > in sys_futex and not as expected in the ptrace breakpoint trap. Also gdb > > does not show that the thread was created. > > > > This can be reproduced with glibc 2.3.6 -> 2.5, kernel 2.6.14 -> > > 2.6.19-rc2, gcc 3.4 -> 4.1.1 and binutils up to cvs-latest. > > Well, it works just fine for me, so you may want to investigate your > setup. Have you set solib-absolute-prefix, the most frequent problem? > Check if "info shared" and "info threads" show sensible output. Here are some more details about the problem. It turned out that it works if you link against libpthread dynamically and doesn't with -static. The test program and log reports of the static and dynamic case are attached. Robert -- Dipl.-Ing. Robert Schwebel | http://www.pengutronix.de Pengutronix - Linux Solutions for Science and Industry Handelsregister: Amtsgericht Hildesheim, HRA 2686 Hannoversche Str. 2, 31134 Hildesheim, Germany Phone: +49-5121-206917-0 | Fax: +49-5121-206917-9 [-- Attachment #2: gdb_test.c --] [-- Type: text/x-csrc, Size: 1946 bytes --] #define _XOPEN_SOURCE 600 #include <unistd.h> #include <stdio.h> #include <stdlib.h> #include <semaphore.h> #include <time.h> #include <assert.h> #include <errno.h> #include <signal.h> #include <pthread.h> #include <signal.h> #define die(msg) {perror (msg); exit (EXIT_FAILURE); } sem_t sem; int stop; void sig_handler (int sig) { stop = sig; } void *thread1 (void *data) { int t = (int) data; while (!stop) { while (sem_wait (&sem) == -1 && errno == EINTR) continue; printf ("My name is %d. thread %d\n", t, t); fflush (stdout); sem_post (&sem); sleep (1); } return NULL; } void *thread2 (void *data) { int t = (int) data; while (!stop) { while (sem_wait (&sem) == -1 && errno == EINTR) continue; printf ("My name is %d. thread %d\n", t, t); fflush (stdout); sem_post (&sem); sleep (1); } return NULL; } void *thread3 (void *data) { int t = (int) data; while (!stop) { while (sem_wait (&sem) == -1 && errno == EINTR) continue; printf ("My name is %d. thread %d\n", t, t); fflush (stdout); sem_post (&sem); sleep (1); } return NULL; } void *thread4 (void *data) { int t = (int) data; while (!stop) { while (sem_wait (&sem) == -1 && errno == EINTR) continue; printf ("My name is %d. thread %d\n", t, t); fflush (stdout); sem_post (&sem); sleep (1); } return NULL; } int main (void) { pthread_t p1, p2, p3, p4; signal (SIGINT, sig_handler); stop = 0; if (sem_init (&sem, 0, 0) == -1) die ("sem_init"); pthread_create (&p1, NULL, thread1, (void *) 1); pthread_create (&p2, NULL, thread2, (void *) 2); pthread_create (&p3, NULL, thread3, (void *) 3); pthread_create (&p4, NULL, thread4, (void *) 4); while (!stop) { printf ("main\n"); fflush (stdout); sem_post (&sem); sleep (2); } sem_post (&sem); pthread_join (p1, NULL); pthread_join (p2, NULL); pthread_join (p3, NULL); pthread_join (p4, NULL); return 0; } [-- Attachment #3: static_bin.gdb.log --] [-- Type: text/plain, Size: 1979 bytes --] build-flag: -g -Wall gdb_test.c -o gdb_test -lpthread -static gdb init script (Auszug): ............ target remote 192.168.23.216:2345 continue info share continue continue continue info thread continue lfu@himalia:/ptx/work/mkl/sprecher/testcases/test_case_3 /opt/OSELAS.Toolchain-trunk/arm-ep93xx-linux-gnueabi/gcc-4.1.1-glibc-2.5/bin/arm-ep93xx-linux-gnueabi-gdb ./gdb_test GNU gdb 6.5.50.20061010 Copyright (C) 2006 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "--host=i686-host-linux-gnu --target=arm-ep93xx-linux-gnueabi"... Breakpoint 1 at 0x8294: file gdb_test.c, line 32. Breakpoint 2 at 0x8340: file gdb_test.c, line 49. Breakpoint 3 at 0x83ec: file gdb_test.c, line 66. Breakpoint 4 at 0x8498: file gdb_test.c, line 83. Breakpoint 5 at 0x8550: file gdb_test.c, line 101. Breakpoint 6 at 0x8568: file gdb_test.c, line 102. Breakpoint 7 at 0x8580: file gdb_test.c, line 103. Breakpoint 8 at 0x8598: file gdb_test.c, line 104. 0x00008110 in _start () [New thread 420] [Switching to thread 420] Breakpoint 5, main () at gdb_test.c:101 101 pthread_create (&p1, NULL, thread1, (void *) 1); No shared libraries loaded at this time. Breakpoint 6, main () at gdb_test.c:102 102 pthread_create (&p2, NULL, thread2, (void *) 2); Breakpoint 7, main () at gdb_test.c:103 103 pthread_create (&p3, NULL, thread3, (void *) 3); Breakpoint 8, main () at gdb_test.c:104 104 pthread_create (&p4, NULL, thread4, (void *) 4); * 1 thread 420 main () at gdb_test.c:104 Program terminated with signal SIGTRAP, Trace/breakpoint trap. The program no longer exists. (gdb) [-- Attachment #4: shared_bin.gdb.log --] [-- Type: text/plain, Size: 4142 bytes --] build-flag: -g -Wall gdb_test.c -o gdb_test -lpthread gdb init script (Auszug): ............ target remote 192.168.23.216:2345 continue info share continue continue continue info thread continue continue ...............(10 x Wiederholungen) ####################################################### lfu@himalia:/ptx/work/mkl/sprecher/testcases/test_case_3 /opt/OSELAS.Toolchain-trunk/arm-ep93xx-linux-gnueabi/gcc-4.1.1-glibc-2.5/bin/arm-ep93xx-linux-gnueabi-gdb ./gdb_test GNU gdb 6.5.50.20061010 Copyright (C) 2006 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "--host=i686-host-linux-gnu --target=arm-ep93xx-linux-gnueabi"... Breakpoint 1 at 0x86f8: file gdb_test.c, line 32. Breakpoint 2 at 0x87a4: file gdb_test.c, line 49. Breakpoint 3 at 0x8850: file gdb_test.c, line 66. Breakpoint 4 at 0x88fc: file gdb_test.c, line 83. Breakpoint 5 at 0x89b4: file gdb_test.c, line 101. Breakpoint 6 at 0x89cc: file gdb_test.c, line 102. Breakpoint 7 at 0x89e4: file gdb_test.c, line 103. Breakpoint 8 at 0x89fc: file gdb_test.c, line 104. 0x400007e0 in ?? () [New thread 442] [Switching to thread 442] Breakpoint 5, main () at gdb_test.c:101 101 pthread_create (&p1, NULL, thread1, (void *) 1); From To Syms Read Shared Object Library 0x400007e0 0x40017334 Yes /ptx/work/mkl/sprecher/OSELAS.BSP-Phytec-phyCORE-PXA270-trunk/root-debug/lib/ld-linux.so.3 0x40028ea0 0x40035a88 Yes /ptx/work/mkl/sprecher/OSELAS.BSP-Phytec-phyCORE-PXA270-trunk/root-debug/lib/libpthread.so.0 0x40055ee0 0x401326f0 Yes /ptx/work/mkl/sprecher/OSELAS.BSP-Phytec-phyCORE-PXA270-trunk/root-debug/lib/libc.so.6 Breakpoint 6, main () at gdb_test.c:102 102 pthread_create (&p2, NULL, thread2, (void *) 2); Breakpoint 7, main () at gdb_test.c:103 103 pthread_create (&p3, NULL, thread3, (void *) 3); Breakpoint 8, main () at gdb_test.c:104 104 pthread_create (&p4, NULL, thread4, (void *) 4); 4 thread 446 0x400315bc in __new_sem_wait (sem=0x10cf8) at ../nptl/sysdeps/unix/sysv/linux/sem_wait.c:48 3 thread 445 0x400315bc in __new_sem_wait (sem=0x10cf8) at ../nptl/sysdeps/unix/sysv/linux/sem_wait.c:48 2 thread 444 0x400315bc in __new_sem_wait (sem=0x10cf8) at ../nptl/sysdeps/unix/sysv/linux/sem_wait.c:48 * 1 thread 442 main () at gdb_test.c:104 [Switching to thread 444] Breakpoint 1, thread1 (data=0x1) at gdb_test.c:32 32 printf ("My name is %d. thread %d\n", t, t); ---Type <return> to continue, or q <return> to quit--- [Switching to thread 445] Breakpoint 2, thread2 (data=0x2) at gdb_test.c:49 49 printf ("My name is %d. thread %d\n", t, t); [Switching to thread 446] Breakpoint 3, thread3 (data=0x3) at gdb_test.c:66 66 printf ("My name is %d. thread %d\n", t, t); [Switching to thread 444] Breakpoint 1, thread1 (data=0x1) at gdb_test.c:32 32 printf ("My name is %d. thread %d\n", t, t); [New thread 447] [Switching to thread 447] Breakpoint 4, thread4 (data=0x4) at gdb_test.c:83 83 printf ("My name is %d. thread %d\n", t, t); [Switching to thread 445] Breakpoint 2, thread2 (data=0x2) at gdb_test.c:49 49 printf ("My name is %d. thread %d\n", t, t); [Switching to thread 446] Breakpoint 3, thread3 (data=0x3) at gdb_test.c:66 66 printf ("My name is %d. thread %d\n", t, t); [Switching to thread 444] Breakpoint 1, thread1 (data=0x1) at gdb_test.c:32 32 printf ("My name is %d. thread %d\n", t, t); [Switching to thread 447] Breakpoint 4, thread4 (data=0x4) at gdb_test.c:83 83 printf ("My name is %d. thread %d\n", t, t); [Switching to thread 445] Breakpoint 2, thread2 (data=0x2) at gdb_test.c:49 49 printf ("My name is %d. thread %d\n", t, t); ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: thread-debuging with gdbserver on ARM/NPTL 2006-10-25 14:46 ` Robert Schwebel @ 2006-10-25 15:03 ` Daniel Jacobowitz 2006-10-26 9:55 ` Luotao Fu 0 siblings, 1 reply; 6+ messages in thread From: Daniel Jacobowitz @ 2006-10-25 15:03 UTC (permalink / raw) To: Robert Schwebel; +Cc: Benedikt Spranger, gdb, Marc Kleine-Budde, Luotao Fu On Wed, Oct 25, 2006 at 04:45:29PM +0200, Robert Schwebel wrote: > Here are some more details about the problem. It turned out that it > works if you link against libpthread dynamically and doesn't with > -static. Oh, right. Thread debugging of static applications using NPTL is particularly problematic. Take a look at this, for starters: http://sourceware.org/bugzilla/show_bug.cgi?id=2696 -- Daniel Jacobowitz CodeSourcery ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: thread-debuging with gdbserver on ARM/NPTL 2006-10-25 15:03 ` Daniel Jacobowitz @ 2006-10-26 9:55 ` Luotao Fu 2006-10-26 12:22 ` Daniel Jacobowitz 0 siblings, 1 reply; 6+ messages in thread From: Luotao Fu @ 2006-10-26 9:55 UTC (permalink / raw) To: Daniel Jacobowitz Cc: Franz MEINDL, Manfred Gruber, Marc Kleine-Budde, gdb, Luotao Fu, Robert Schwebel, Benedikt Spranger, tglx [-- Attachment #1.1: Type: text/plain, Size: 1654 bytes --] Hi, On Wed, Oct 25, 2006 at 11:03:44AM -0400, Daniel Jacobowitz wrote: > On Wed, Oct 25, 2006 at 04:45:29PM +0200, Robert Schwebel wrote: > > Here are some more details about the problem. It turned out that it > > works if you link against libpthread dynamically and doesn't with > > -static. > > Oh, right. Thread debugging of static applications using NPTL is > particularly problematic. Take a look at this, for starters: > > http://sourceware.org/bugzilla/show_bug.cgi?id=2696 We built a Toolchain with gcc4.1.1+glibc2.5+gdb-6.5.50.20061025 using the patch posted on http://sourceware.org/ml/libc-alpha/2006-03/msg00014.html and did the same test as already described in robert's mail again. Unluckiely the problems with static binary seems to still consist. On the host side gdb would stuck at first continue, while on the target side gdbserver prints something like "Cannot enable thread event reporting for [PID of the testapp]: generic error". I however noticed that the test can be run if there's no symboltable given on the host side. (say gdb is started without any parameter or arguments) In this case I can process between the continues sending SIGINT. I logged the outputs and used commands to a log file and enclose it in the attachment. > > -- > Daniel Jacobowitz > CodeSourcery Regards Luotao Fu -- Dipl.-Ing. Luotao Fu | http://www.pengutronix.de Pengutronix - Linux Solutions for Science and Industry Handelsregister: Amtsgericht Hildesheim, HRA 2686 Hannoversche Str. 2, 31134 Hildesheim, Germany Phone: +49-5121-206917-0 | Fax: +49-5121-206917-9 [-- Attachment #1.2: static_bin.gdb_patched.log --] [-- Type: text/plain, Size: 4545 bytes --] gdb (host): lfu@himalia:/ptx/work/mkl/sprecher/testcases/test_case_3 /opt/OSELAS.Toolchain-trunk/arm-ep93xx-linux-gnueabi/gcc-4.1.1-glibc-2.5-staticfix/bin/arm-ep93xx-linux-gnueabi-gdb gdb_test GNU gdb 6.5.50.20061025 Copyright (C) 2006 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "--host=i686-host-linux-gnu --target=arm-ep93xx-linux-gnueabi"... Breakpoint 1 at 0x8294: file gdb_test.c, line 32. Breakpoint 2 at 0x8340: file gdb_test.c, line 49. Breakpoint 3 at 0x83ec: file gdb_test.c, line 66. Breakpoint 4 at 0x8498: file gdb_test.c, line 83. Breakpoint 5 at 0x8550: file gdb_test.c, line 101. Breakpoint 6 at 0x8568: file gdb_test.c, line 102. Breakpoint 7 at 0x8580: file gdb_test.c, line 103. Breakpoint 8 at 0x8598: file gdb_test.c, line 104. 0x00008110 in _start () Ignoring packet error, continuing... warning: Invalid remote reply: qSymbol:5f5f6e70746c5f6c6173745f6576656e74 Interrupted while waiting for the program. Give up (and stop debugging it)? (y or n) [answered Y; input not from terminal] gdbserver (testboard): root@testboard:~> gdbserver 192.168.23.1:2345 gdb_test Process gdb_test created; pid = 365 Listening on port 2345 Remote debugging from host 192.168.23.1 Cannot enable thread event reporting for 365: generic error input_interrupt, count = 0 c = 0 ('') readchar: Got EOF putpkt(read): Got EOF thread getmsg err: debugger service failed Segmentation fault ############################################################################################# gdb(host) malia:/ptx/work/mkl/sprecher/testcases/test_case_3 /opt/OSELAS.Toolchain-trunk/arm-ep93xx-linux-gnueabi/gcc-4.1.1-glibc-2.5-staticfix/bin/arm-ep93xx-linux-gnueabi-gdb GNU gdb 6.5.50.20061025 Copyright (C) 2006 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "--host=i686-host-linux-gnu --target=arm-ep93xx-linux-gnueabi". No symbol table is loaded. Use the "file" command. No symbol table is loaded. Use the "file" command. No symbol table is loaded. Use the "file" command. No symbol table is loaded. Use the "file" command. No symbol table is loaded. Use the "file" command. No symbol table is loaded. Use the "file" command. No symbol table is loaded. Use the "file" command. No symbol table is loaded. Use the "file" command. 0x00008110 in ?? () Program received signal SIGINT, Interrupt. 0x00021774 in ?? () No shared libraries loaded at this time. Program received signal SIGINT, Interrupt. 0xbeeaf9b4 in ?? () Program received signal SIGINT, Interrupt. 0x00021774 in ?? () Program received signal SIGINT, Interrupt. 0xbeeaf9b4 in ?? () 1 Thread 367 0xbeeaf9b4 in ?? () warning: Couldn't restore frame in current thread, at frame 0 0xbeeaf9b4 in ?? () Program received signal SIGINT, Interrupt. 0x00021774 in ?? () Program received signal SIGINT, Interrupt. 0xbeeaf9b4 in ?? () Program received signal SIGINT, Interrupt. 0xbeeaf9a8 in ?? () Program received signal SIGINT, Interrupt. 0xbeeaf99c in ?? () The program is running. Exit anyway? (y or n) [answered Y; input not from terminal] gdbserver (testboard): root@phyCORE-PXA270:~> gdbserver 192.168.23.1:2345 gdb_test Process gdb_test created; pid = 367 Listening on port 2345 Remote debugging from host 192.168.23.1 main My name is 1. thread 1 My name is 2. thread 2 My name is 3. thread 3 My name is 4. thread 4 My name is 1. thread 1 My name is 2. thread 2 My name is 3. thread 3 My name is 4. thread 4 main My name is 1. thread 1 My name is 2. thread 2 My name is 3. thread 3 My name is 4. thread 4 My name is 1. thread 1 My name is 2. thread 2 My name is 3. thread 3 My name is 4. thread 4 main My name is 1. thread 1 My name is 2. thread 2 My name is 3. thread 3 My name is 4. thread 4 My name is 1. thread 1 My name is 2. thread 2 My name is 3. thread 3 My name is 4. thread 4 main My name is 1. thread 1 My name is 2. thread 2 My name is 3. thread 3 My name is 4. thread 4 My name is 1. thread 1 My name is 2. thread 2 My name is 3. thread 3 My name is 4. thread 4 Killing inferior [-- Attachment #2: Digital signature --] [-- Type: application/pgp-signature, Size: 189 bytes --] ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: thread-debuging with gdbserver on ARM/NPTL 2006-10-26 9:55 ` Luotao Fu @ 2006-10-26 12:22 ` Daniel Jacobowitz 0 siblings, 0 replies; 6+ messages in thread From: Daniel Jacobowitz @ 2006-10-26 12:22 UTC (permalink / raw) To: Luotao Fu Cc: Franz MEINDL, Manfred Gruber, Marc Kleine-Budde, gdb, Luotao Fu, Robert Schwebel, Benedikt Spranger, tglx On Thu, Oct 26, 2006 at 11:57:36AM +0200, Luotao Fu wrote: > We built a Toolchain with gcc4.1.1+glibc2.5+gdb-6.5.50.20061025 using > the patch posted on > http://sourceware.org/ml/libc-alpha/2006-03/msg00014.html and did the > same test as already described in robert's mail again. Unluckiely the > problems with static binary seems to still consist. On the host > side gdb would stuck at first continue, while on the target side > gdbserver prints something like "Cannot enable thread event reporting > for [PID of the testapp]: generic error". Sorry, I have no idea what's wrong now. You shouldn't ever see the qSymbol error from your log. -- Daniel Jacobowitz CodeSourcery ^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2006-10-26 12:22 UTC | newest] Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2006-10-24 18:29 thread-debuging with gdbserver on ARM/NPTL Benedikt Spranger 2006-10-24 18:34 ` Daniel Jacobowitz 2006-10-25 14:46 ` Robert Schwebel 2006-10-25 15:03 ` Daniel Jacobowitz 2006-10-26 9:55 ` Luotao Fu 2006-10-26 12:22 ` Daniel Jacobowitz
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox