Cross-platform, multithreaded debugging (x86 to ARM) with gdb and gdbserver not recognizing threads

Cross-platform, multithreaded debugging (x86 to ARM) with gdb and gdbserver not recognizing threads

[Jason Machacek]

Hello everyone,

I recently ran into an issue where my embedded application became too
large to run natively under GDB.  Unfortunately, my SBC provider doesn't
supply a precompiled version of GDB, so I've been trying to build a
GDB/gdbserver combination myself.  My application is a C++ app with
several threads running.

I can successfully build both GDB (native x86 with --target=arm-linux
specified) and gdbserver (ARM-native).  However, when I connect to
gdbserver on my target with gdb, it doesn't understand that it's a
multithreaded application--it breaks on each context switch (SIG32) and
can't figure out what instruction it's looking at.

I found a 2005 thread from this mailing list
(http://www.cygwin.com/ml/gdb/2005-11/msg00364.html) and several other
messages from other places that set me on the right track, but I'm still
not quite there.

Here are the commands I used to build GDB (x86):

# cd gdb-6.7.1/
# PATH=$PATH:/opt/crosstool/gcc-3.4.4-glibc-2.3.2/arm-linux/bin/
./configure --target=arm-linux --disable-sim --disable-tcl
--enable-threads --enable-shared
# make

Here are the commands I used to build gdbserver (ARM):

# cd gdb/gdbserver/
# PATH=$PATH:/opt/crosstool/gcc-3.4.4-glibc-2.3.2/arm-linux/bin
CC=/opt/crosstool/gcc-3.4.4-glibc-2.3.2/arm-linux/bin/arm-linux-gcc
./configure --host=arm-linux --target=arm-linux --enable-threads
--enable-shared
# make

I execute the gdbserver binary on the target:

$ ./gdbserver :12345 app
Process app created; pid = 270
Listening on port 12345


I then connect to the target from the host like so:

# ../gdb-6.7.1/gdb/gdb app
GNU gdb 6.7.1
Copyright (C) 2007 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
<http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show
copying"
and "show warranty" for details.
This GDB was configured as "--host=i686-pc-linux-gnu
--target=arm-linux"...
(gdb) set solib-absolute-prefix
/opt/crosstool/gcc-3.4.4-glibc-2.3.2/arm-linux/arm-linux/
(gdb) file app
Load new symbol table from "/home/jmachacek/juice/juice/src/app"? (y or
n) y
Reading symbols from /home/jmachacek/juice/juice/src/app...done.
(gdb) target remote 10.0.150.55:12345
Remote debugging using 10.0.150.55:12345
0x2aaaba70 in _start () from
/opt/crosstool/gcc-3.4.4-glibc-2.3.2/arm-linux/arm-linux/lib/ld-linux.so
.2
(gdb)


So far so good.  "info threads" and "info shared" look okay:

(gdb) info threads
* 1 Thread 270  0x2aaaba70 in _start ()
   from
/opt/crosstool/gcc-3.4.4-glibc-2.3.2/arm-linux/arm-linux/lib/ld-linux.so
.2
(gdb) info shared
From        To          Syms Read   Shared Object Library
0x2aaaba70  0x2aabd888  Yes
/opt/crosstool/gcc-3.4.4-glibc-2.3.2/arm-linux/arm-linux/lib/ld-linux.so
.2
(gdb)


I can debug to main() with no problem:

(gdb) b main
Breakpoint 1 at 0x20808: file Main/Main.cpp, line 41.
(gdb) c
Continuing.

Breakpoint 1, main (argc=1, argv=0x7ffffec4) at Main/Main.cpp:41
41	    BSP_init(argc, argv); 
(gdb) bt
#0  main (argc=1, argv=0x7ffffec4) at Main/Main.cpp:41
(gdb)


At this point, I use 'c' to start running and GDB stops when it receives
SIG32 (which appears to be generated when the CPU switches between
threads):

(gdb) c
Continuing.

Program received signal SIG32, Real-time event 32.
0x2abd49e8 in ?? ()
(gdb) bt
#0  0x2abd49e8 in ?? ()
(gdb) info threads
* 1 Thread 270  0x2abd49e8 in ?? ()
warning: Couldn't restore frame in current thread, at frame 0
0x2abd49e8 in ?? ()
(gdb) info shared
warning: .dynamic section for
"/opt/crosstool/gcc-3.4.4-glibc-2.3.2/arm-linux/arm-linux/lib/libm.so.6"
is not at the expected address (wrong library or version mismatch?)
warning: .dynamic section for
"/opt/crosstool/gcc-3.4.4-glibc-2.3.2/arm-linux/arm-linux/lib/libgcc_s.s
o.1" is not at the expected address (wrong library or version mismatch?)
warning: .dynamic section for
"/opt/crosstool/gcc-3.4.4-glibc-2.3.2/arm-linux/arm-linux/lib/libc.so.6"
is not at the expected address (wrong library or version mismatch?)
From        To          Syms Read   Shared Object Library
0x2aaaba70  0x2aabd888  Yes
/opt/crosstool/gcc-3.4.4-glibc-2.3.2/arm-linux/arm-linux/lib/ld-linux.so
.2
0x2aacc050  0x2aad5e1c  No
/opt/crosstool/gcc-3.4.4-glibc-2.3.2/arm-linux/arm-linux/lib/libpthread.
so.0
0x2ab254c4  0x2ab5d5d0  No
/opt/crosstool/gcc-3.4.4-glibc-2.3.2/arm-linux/arm-linux/lib/libm.so.6
0x2ab9ab84  0x2aba12d8  No
/opt/crosstool/gcc-3.4.4-glibc-2.3.2/arm-linux/arm-linux/lib/libgcc_s.so
.1
0x2abc0f80  0x2acbd404  No
/opt/crosstool/gcc-3.4.4-glibc-2.3.2/arm-linux/arm-linux/lib/libc.so.6
(gdb)


The warnings in "info shared" are due to libc being statically linked in
my application.  Can anyone help me figure out why I'm unable to get
gdb/gdbserver to work properly with my target?


Thanks very much,
Jason


Jason Machacek  |  Stratos Product Development  |  Software Engineer  |
(206) 576-7312

Re: Cross-platform, multithreaded debugging (x86 to ARM) with gdb and gdbserver not recognizing threads

[Paul Pluzhnikov]

On Wed, Dec 16, 2009 at 3:01 PM, Jason Machacek <jmachacek@stratos.com> wrote:

> Can anyone help me figure out why I'm unable to get
> gdb/gdbserver to work properly with my target?

A couple of questions:
- Does gdbserver config.log mention libthread_db.so, and if so, which one?
- Does ldd gdbserver on target show libthread_db.so, and if so, is the
right one? (Should come from the same compilation from which libc.so.6
and libpthread.so.0 came.)
- Is libpthread.so.0 on target stripped?

Most of the "SIG32" problems come from stripped libpthread.so.0 or
libpthread/libthread_db mismatch.

Good luck!
-- 
Paul Pluzhnikov

RE: Cross-platform, multithreaded debugging (x86 to ARM) with gdb and gdbserver not recognizing threads

[Jason Machacek]

Hi Paul,

Thanks for the reply.


> A couple of questions:
> - Does gdbserver config.log mention libthread_db.so, and if so, which
one?

$ cat config.log | grep libthread_db -A8
configure:3888: checking for libthread_db
configure:3919:
/opt/crosstool/gcc-3.4.4-glibc-2.3.2/arm-linux/bin/arm-linux-gcc -o
conftest -g -O2   conftest.c  -lthread_db >&5
configure:3925: $? = 0
configure:3929: test -z 
			 || test ! -s conftest.err
configure:3932: $? = 0
configure:3935: test -s conftest
configure:3938: $? = 0
configure:4014: result: -lthread_db

It appears gdbserver with the correct libthread_db--my toolchain is
mentioned, so I'm assuming it's linking the correct library.  It's the
only ARM-compatible libthread_db.so on my system. 



> - Does ldd gdbserver on target show libthread_db.so, and if so, is the
> right one? (Should come from the same compilation from which libc.so.6
> and libpthread.so.0 came.)

I unfortunately don't have ldd on my target, but gdbserver's maps file
shows that it's grabbing libthread_db.so.  I installed that version of
libthread_db.so myself from the toolchain along with libpthread.so, and
libc.so.6 is statically linked into my application.

$ cat /proc/212/maps 
00008000-00016000 r-xp 00000000 1f:01 549        /root/gdbserver
0001e000-0001f000 rw-p 0000e000 1f:01 549        /root/gdbserver
0001f000-00042000 rwxp 00000000 00:00 0 
2aaab000-2aac0000 r-xp 00000000 1f:01 607        /lib/ld-2.3.2.so
2aac0000-2aac1000 rw-p 00000000 00:00 0 
2aac7000-2aac8000 rw-p 00014000 1f:01 607        /lib/ld-2.3.2.so
2aac8000-2aacc000 r-xp 00000000 1f:01 799
/lib/libthread_db-1.0.so
2aacc000-2aad0000 ---p 00004000 1f:01 799
/lib/libthread_db-1.0.so
2aad0000-2aad4000 rw-p 00000000 1f:01 799
/lib/libthread_db-1.0.so
2aad4000-2abec000 r-xp 00000000 1f:01 687        /lib/libc-2.3.2.so
2abec000-2abf6000 rw-p 00110000 1f:01 687        /lib/libc-2.3.2.so
2abf6000-2abf9000 rw-p 00000000 00:00 0 
7fffd000-80000000 rwxp ffffe000 00:00 0



> - Is libpthread.so.0 on target stripped?

It doesn't appear to be.  I made a copy of libthread_db-1.0.so and ran
'strip' on it, and the file size dropped.  Is this the correct approach?

$ cp
/opt/crosstool/gcc-3.4.4-glibc-2.3.2/arm-linux/arm-linux/lib/libthread_d
b-1.0.so .
$ ls -l libthread_db-1.0.so 
-rwxr-xr-x 1 jason jason 29579 2009-12-17 09:49 libthread_db-1.0.so
$ /opt/crosstool/gcc-3.4.4-glibc-2.3.2/arm-linux/bin/arm-linux-strip
libthread_db-1.0.so 
$ ls -l libthread_db-1.0.so 
-rwxr-xr-x 1 jason jason 18296 2009-12-17 09:50 libthread_db-1.0.so

I tried just stripping debugging information as well, and that also
decreased the file size:

$ ls -l libthread_db-1.0.so 
-rwxr-xr-x 1 jason jason 29579 2009-12-17 09:53 libthread_db-1.0.so
$ /opt/crosstool/gcc-3.4.4-glibc-2.3.2/arm-linux/bin/arm-linux-strip
--strip-debug libthread_db-1.0.so 
$ ls -l libthread_db-1.0.so
-rwxr-xr-x 1 jason jason 23781 2009-12-17 09:54 libthread_db-1.0.so


Does this indicate that I've got a library version mismatch somewhere in
my system?  I'm using a toolchain from my SBC supplier, but perhaps that
wasn't the toolchain they used to build all the libraries included on
the target.

Is there any problem with setting up a second set of libraries on my
system?  E.g. /lib2 that contains the libraries copied from my
toolchain.  Would I have to make changes to my toolchain's directory
structure to ensure the directories matched up with those on the target
(for GDB that is)?


Thanks,
Jason

Re: Cross-platform, multithreaded debugging (x86 to ARM) with gdb and gdbserver not recognizing threads

[Paul Pluzhnikov]

On Thu, Dec 17, 2009 at 10:01 AM, Jason Machacek <jmachacek@stratos.com> wrote:

>> - Is libpthread.so.0 on target stripped?
>
> It doesn't appear to be.  I made a copy of libthread_db-1.0.so and ran
> 'strip' on it, and the file size dropped.  Is this the correct approach?

It's not libthread_db, it's the libpthread that matters.

You want to copy /lib/libpthread.so.0 (which should be a symlink to
/lib/libpthread-2.3.2.so) from target to host, and then:

  arm-linux-nm libpthread.so.0 | egrep 'version|threads_events'

You should see one of: nptl_version, __linuxthreads_version or
__pthread_threads_events.

If you see 'no symbols' instead, then your libpthread is stripped, and
that explains your problems.

> and libc.so.6 is statically linked into my application.

It is impossible to statically link libc.so.6 into an application. Perhaps
you mean that your app is linked with libc.a ?

In general GDB may have harder time debugging statically-linked threaded
executables, and you should always prefer to link against system libraries
dynamically.

Cheers,
-- 
Paul Pluzhnikov

RE: Cross-platform, multithreaded debugging (x86 to ARM) with gdb and gdbserver not recognizing threads

[Jason Machacek]

> You want to copy /lib/libpthread.so.0 (which should be a symlink to
> /lib/libpthread-2.3.2.so) from target to host, and then:
>
>   arm-linux-nm libpthread.so.0 | egrep 'version|threads_events'
>
> You should see one of: nptl_version, __linuxthreads_version or
> __pthread_threads_events.

Sorry about that, I misread your last message.  libpthread.so isn't
stripped:

$ /opt/crosstool/gcc-3.4.4-glibc-2.3.2/arm-linux/bin/arm-linux-nm
libpthread-0.10.so | egrep 'version|threads_events'
0000de40 r __linuxthreads_version
00018e68 b __pthread_threads_events


> It is impossible to statically link libc.so.6 into an application.
Perhaps
> you mean that your app is linked with libc.a ?
>
> In general GDB may have harder time debugging statically-linked
threaded
> executables, and you should always prefer to link against system
libraries
> dynamically.

Yes, I did mean that I statically linked my application with libc.a.
Perhaps I'll reexamine my build environment as this may be the root of
my problems with GDB.  When running GDB natively on my target, I had no
problems debugging my application despite it being statically linked.
I'll do some tests to check whether addressing this allows gdb to
recognize threads in my remote application.

Thanks again,
Jason

Re: Cross-platform, multithreaded debugging (x86 to ARM) with gdb and gdbserver not recognizing threads

[Paul Pluzhnikov]

On Thu, Dec 17, 2009 at 10:38 AM, Jason Machacek <jmachacek@stratos.com> wrote:

> Sorry about that, I misread your last message.  libpthread.so isn't
> stripped:
>
> $ /opt/crosstool/gcc-3.4.4-glibc-2.3.2/arm-linux/bin/arm-linux-nm
> libpthread-0.10.so | egrep 'version|threads_events'
> 0000de40 r __linuxthreads_version
> 00018e68 b __pthread_threads_events

AFAIU, this is the host copy of libpthread.

One common mistake is stripping it *on target*. Hence I asked to copy it
back and run nm on what is actually on target.

Of course you could just 'ls -l' or md5sum to verify that host and target
copies are identical.

> Yes, I did mean that I statically linked my application with libc.a.
> Perhaps I'll reexamine my build environment as this may be the root of
> my problems with GDB.  When running GDB natively on my target, I had no
> problems debugging my application despite it being statically linked.

Was it on the same target you are using now? (That would rule out stripped
libpthread, or libpthread/libthread_db mismatch, as gdb and gdbserver
should be similarly broken.)

Cheers,
-- 
Paul Pluzhnikov

RE: Cross-platform, multithreaded debugging (x86 to ARM) with gdb and gdbserver not recognizing threads

[Jason Machacek]

>> $ /opt/crosstool/gcc-3.4.4-glibc-2.3.2/arm-linux/bin/arm-linux-nm
>> libpthread-0.10.so | egrep 'version|threads_events'
>> 0000de40 r __linuxthreads_version
>> 00018e68 b __pthread_threads_events
>
> AFAIU, this is the host copy of libpthread.

That was the target's copy of libpthread.so--I did copy it back to the
host before running arm-linux-nm on it as you suggested.


> Was it on the same target you are using now? (That would rule out
stripped
> libpthread, or libpthread/libthread_db mismatch, as gdb and gdbserver
> should be similarly broken.)

Yes, it was the same target.  Running GDB natively works perfectly with
multiple threads, but when running GDB from my host and connecting to
gdbserver on my target GDB gets confused when the program reaches the
first pthread_create() call.  Unfortunately, my application has recently
become too large to fit in the target's memory while GDB is running, so
I can no longer run GDB natively on my target.

Best regards,
Jason