Trouble with gdbserver from gdb-5.3, glibc-2.3.2 on ppc32

Trouble with gdbserver from gdb-5.3, glibc-2.3.2 on ppc32

[Peter Barada]

I'm trying to bring up gdbserver2 on an ppc32 embedded linux platform, and
I'm running in to the following message.  It doesn't alway appear, so
it has me pretty confused....

sh-2.05b# cat /proc/version
Linux version 2.4.21-rc1 (pbarada@hyper) (gcc version 3.3.1) #21 Tue Sep 9 15:43
sh-2.05b# gdbserver2 :3001 /usr/local/bin/page.strip -m16m -J < /dev/lp0
Process /usr/local/bin/page.strip created; pid = 35
Remote debugging from host 10.1.1.2
gdb: error initializing thread_db library.

I modified gdbserver/thread-db.c::thread_db_init to print the value of
the err that is returned form td_ta_new, and the result is 0x16 which
if I read glibc-2.3.2/linuxthreads_db/thread_db.h correctly, the 0x16
returned from td_ta_new() is TD_VERSION which means the version of
libpthread doesn't match the version of libthread_db.

If I look in gdb for the type of TD_VERSION I don't see TD_VERSION
even thos I see it in linuxthreads_db/thread_db.h:

(gdb) ptype TD_OK
type = enum {TD_OK, TD_ERR, TD_NOTHR, TD_NOSV, TD_NOLWP, TD_BADPH, TD_BADTH, TD_BADSH, 
    TD_BADTA, TD_BADKEY, TD_NOMSG, TD_NOFPREGS, TD_NOLIBTHREAD, TD_NOEVENT, 
    TD_NOCAPAB, TD_DBERR, TD_NOAPLIC, TD_NOTSD, TD_MALLOC, TD_PARTIALREG, TD_NOXREGS}
(gdb) ptype TD_VERSION
No symbol "TD_VERSION" in current context.

My question is how I can tell why gdbserver believes that the verion
of libpthread.so.0 doesn't match the version of libthread_db.so.1?
Whath options to readelf do I need?

execution doesn't work after this (i.e. 'inf thr' doesn't show me my
thread, etc).  I'm wondering if there's a deeper problem...

-- 
Peter Barada                             Peter.Barada@motorola.com
Wizard                                   781-852-2768 (direct)
WaveMark Solutions(A Motorola Company)   781-270-0193 (fax)

Re: Trouble with gdbserver from gdb-5.3, glibc-2.3.2 on ppc32

[Roland McGrath]

libpthread and libthread_db are very closely tied.  You have to use a
libthread_db that is part of the same glibc package that libpthread came
from.  The situation is not cross-friendly, sorry.

Re: Trouble with gdbserver from gdb-5.3, glibc-2.3.2 on ppc32

[Peter Barada]

>libpthread and libthread_db are very closely tied.  You have to use a
>libthread_db that is part of the same glibc package that libpthread came
>from.  The situation is not cross-friendly, sorry.

I completely understand.  I downloaded glibc-2.3.2 and
glibc-linuxthreads-2.3.2 from sources.redhat.com and built up the
toolchain --target=powerpc-linux using Dan Kegel's excellent
crosstools-0.22 script.

How do I find out the version of libpthread.so and libthread_db.so so
I can figure out why td_ta_new() fails?

-- 
Peter Barada                             Peter.Barada@motorola.com
Wizard                                   781-852-2768 (direct)
WaveMark Solutions(A Motorola Company)   781-270-0193 (fax)

Re: Trouble with gdbserver from gdb-5.3, glibc-2.3.2 on ppc32

[Roland McGrath]

> How do I find out the version of libpthread.so and libthread_db.so so

You need to know what you've installed on your target and host.

Re: Trouble with gdbserver from gdb-5.3, glibc-2.3.2 on ppc32

[Daniel Jacobowitz]

On Tue, Sep 09, 2003 at 04:31:54PM -0400, Peter Barada wrote:
> 
> I'm trying to bring up gdbserver2 on an ppc32 embedded linux platform, and

What's gdbserver2?  I'll just quietly assume you mean gdbserver...

> I'm running in to the following message.  It doesn't alway appear, so
> it has me pretty confused....
> 
> sh-2.05b# cat /proc/version
> Linux version 2.4.21-rc1 (pbarada@hyper) (gcc version 3.3.1) #21 Tue Sep 9 15:43
> sh-2.05b# gdbserver2 :3001 /usr/local/bin/page.strip -m16m -J < /dev/lp0
> Process /usr/local/bin/page.strip created; pid = 35
> Remote debugging from host 10.1.1.2
> gdb: error initializing thread_db library.
> 
> I modified gdbserver/thread-db.c::thread_db_init to print the value of
> the err that is returned form td_ta_new, and the result is 0x16 which
> if I read glibc-2.3.2/linuxthreads_db/thread_db.h correctly, the 0x16
> returned from td_ta_new() is TD_VERSION which means the version of
> libpthread doesn't match the version of libthread_db.
> 
> If I look in gdb for the type of TD_VERSION I don't see TD_VERSION
> even thos I see it in linuxthreads_db/thread_db.h:
> 
> (gdb) ptype TD_OK
> type = enum {TD_OK, TD_ERR, TD_NOTHR, TD_NOSV, TD_NOLWP, TD_BADPH, TD_BADTH, TD_BADSH, 
>     TD_BADTA, TD_BADKEY, TD_NOMSG, TD_NOFPREGS, TD_NOLIBTHREAD, TD_NOEVENT, 
>     TD_NOCAPAB, TD_DBERR, TD_NOAPLIC, TD_NOTSD, TD_MALLOC, TD_PARTIALREG, TD_NOXREGS}
> (gdb) ptype TD_VERSION
> No symbol "TD_VERSION" in current context.
> 
> My question is how I can tell why gdbserver believes that the verion
> of libpthread.so.0 doesn't match the version of libthread_db.so.1?
> Whath options to readelf do I need?
> 
> execution doesn't work after this (i.e. 'inf thr' doesn't show me my
> thread, etc).  I'm wondering if there's a deeper problem...

Read the source of libthread_db?  No, really.  Then start another
gdbserver debugging gdbserver, and step through td_ta_new.  It fetches
the version.  This could be a symptom of symbol lookup being messed up,
that's almost always the problem.  Versions of the library on host and
target match?  GDB found the rght target libraries?

-- 
Daniel Jacobowitz
MontaVista Software                         Debian GNU/Linux Developer

Re: Trouble with gdbserver from gdb-5.3, glibc-2.3.2 on ppc32

[Daniel Jacobowitz]

On Tue, Sep 09, 2003 at 05:22:41PM -0400, Daniel Jacobowitz wrote:
> On Tue, Sep 09, 2003 at 04:31:54PM -0400, Peter Barada wrote:
> > 
> > I'm trying to bring up gdbserver2 on an ppc32 embedded linux platform, and
> 
> What's gdbserver2?  I'll just quietly assume you mean gdbserver...
> 
> > I'm running in to the following message.  It doesn't alway appear, so
> > it has me pretty confused....
> > 
> > sh-2.05b# cat /proc/version
> > Linux version 2.4.21-rc1 (pbarada@hyper) (gcc version 3.3.1) #21 Tue Sep 9 15:43
> > sh-2.05b# gdbserver2 :3001 /usr/local/bin/page.strip -m16m -J < /dev/lp0
> > Process /usr/local/bin/page.strip created; pid = 35
> > Remote debugging from host 10.1.1.2
> > gdb: error initializing thread_db library.
> > 
> > I modified gdbserver/thread-db.c::thread_db_init to print the value of
> > the err that is returned form td_ta_new, and the result is 0x16 which
> > if I read glibc-2.3.2/linuxthreads_db/thread_db.h correctly, the 0x16
> > returned from td_ta_new() is TD_VERSION which means the version of
> > libpthread doesn't match the version of libthread_db.
> > 
> > If I look in gdb for the type of TD_VERSION I don't see TD_VERSION
> > even thos I see it in linuxthreads_db/thread_db.h:
> > 
> > (gdb) ptype TD_OK
> > type = enum {TD_OK, TD_ERR, TD_NOTHR, TD_NOSV, TD_NOLWP, TD_BADPH, TD_BADTH, TD_BADSH, 
> >     TD_BADTA, TD_BADKEY, TD_NOMSG, TD_NOFPREGS, TD_NOLIBTHREAD, TD_NOEVENT, 
> >     TD_NOCAPAB, TD_DBERR, TD_NOAPLIC, TD_NOTSD, TD_MALLOC, TD_PARTIALREG, TD_NOXREGS}
> > (gdb) ptype TD_VERSION
> > No symbol "TD_VERSION" in current context.
> > 
> > My question is how I can tell why gdbserver believes that the verion
> > of libpthread.so.0 doesn't match the version of libthread_db.so.1?
> > Whath options to readelf do I need?
> > 
> > execution doesn't work after this (i.e. 'inf thr' doesn't show me my
> > thread, etc).  I'm wondering if there's a deeper problem...
> 
> Read the source of libthread_db?  No, really.  Then start another
> gdbserver debugging gdbserver, and step through td_ta_new.  It fetches
> the version.  This could be a symptom of symbol lookup being messed up,
> that's almost always the problem.  Versions of the library on host and
> target match?  GDB found the rght target libraries?

Amusingly, after writing that message, I just got:
warning: Cannot initialize thread debugging library: unknown thread_db error '22'

on my native setup.  This happens when debugging a particular
statically linked executable that was built against a different version
of linuxthreads than is present on my host.  Is that binary statically
linked?

-- 
Daniel Jacobowitz
MontaVista Software                         Debian GNU/Linux Developer

Re: Trouble with gdbserver from gdb-5.3, glibc-2.3.2 on ppc32

[Roland McGrath]

> warning: Cannot initialize thread debugging library: unknown thread_db error '22'
> 
> on my native setup.  This happens when debugging a particular
> statically linked executable that was built against a different version
> of linuxthreads than is present on my host.  

That is (unfortunately) the expected behavior.

Re: Trouble with gdbserver from gdb-5.3, glibc-2.3.2 on ppc32

[Daniel Jacobowitz]

On Tue, Sep 09, 2003 at 02:39:06PM -0700, Roland McGrath wrote:
> > warning: Cannot initialize thread debugging library: unknown thread_db error '22'
> > 
> > on my native setup.  This happens when debugging a particular
> > statically linked executable that was built against a different version
> > of linuxthreads than is present on my host.  
> 
> That is (unfortunately) the expected behavior.

Yes, I know.

Another reason to finish my libthread_db-free debugging support.

-- 
Daniel Jacobowitz
MontaVista Software                         Debian GNU/Linux Developer

Re: Trouble with gdbserver from gdb-5.3, glibc-2.3.2 on ppc32

[Peter Barada]

>Amusingly, after writing that message, I just got:
>warning: Cannot initialize thread debugging library: unknown thread_db error '22'
>
>on my native setup.  This happens when debugging a particular
>statically linked executable that was built against a different version
>of linuxthreads than is present on my host.  Is that binary statically
>linked?

No, its dynamically linked.

-- 
Peter Barada
peter@baradas.org

Re: Trouble with gdbserver from gdb-5.3, glibc-2.3.2 on ppc32

[Peter Barada]

>>on my native setup.  This happens when debugging a particular
>>statically linked executable that was built against a different version
>>of linuxthreads than is present on my host.  Is that binary statically
>>linked?
>
>No, its dynamically linked.

Now that the power is back on(and our network is up and runnig again),
I looked again and indeed its statically linked.  I'll try dynamically
linking it and see what happens.

What's still strange is that the toolchain I used to build the
executable is the same on I used to build gdbserver...

-- 
Peter Barada                             Peter.Barada@motorola.com
Wizard                                   781-852-2768 (direct)
WaveMark Solutions(A Motorola Company)   781-270-0193 (fax)

Re: Trouble with gdbserver from gdb-5.3, glibc-2.3.2 on ppc32

[Peter Barada]

>Now that the power is back on(and our network is up and runnig again),
>I looked again and indeed its statically linked.  I'll try dynamically
>linking it and see what happens.

Ok, my applications runs if its dynamically linked, but if I debug it
using gdb-5.3/gdbserver I get the following:

(gdb) set solib-absolute-prefix /dev/null
(gdb) set solib-search-path /usr/local/wave/x-ppc-linux2/powerpc-linux/lib
(gdb) tar remo 10.1.1.3:3001
Remote debugging using 10.1.1.3:3001
0x3000f430 in ?? ()
(gdb) i b
No breakpoints or watchpoints.
(gdb) b main
Breakpoint 4 at 0x10001eb0: file ../mn/mn.c, line 709.
(gdb) c
Continuing.
Error while reading shared library symbols:
DW_FORM_strp pointing outside of .debug_str section
Error while reading shared library symbols:
DW_FORM_strp pointing outside of .debug_str section
[New Thread 16384]
Error while reading shared library symbols:
DW_FORM_strp pointing outside of .debug_str section
Error while reading shared library symbols:
DW_FORM_strp pointing outside of .debug_str section
Error while reading shared library symbols:
DW_FORM_strp pointing outside of .debug_str section
Error while reading shared library symbols:
DW_FORM_strp pointing outside of .debug_str section
[Switching to Thread 16384]

Breakpoint 4, main (argc=3, argv=0x7ffffe04) at ../mn/mn.c:709
(gdb) i sh
From        To          Syms Read   Shared Object Library
0x0ff732dc  0x0ffa3994  Yes         /home/mylocal/local/wave/x-ppc-linux2/powerpc-linux/lib/libm.so.6
0x0fefc0f4  0x0ff05140  Yes         /home/mylocal/local/wave/x-ppc-linux2/powerpc-linux/lib/libpthread.so.0
0x0fdbea20  0x0feb64e4  No          /home/mylocal/local/wave/x-ppc-linux2/powerpc-linux/lib/libc.so.6
0x30001e00  0x300126f0  No          /home/mylocal/local/wave/x-ppc-linux2/powerpc-linux/lib/ld.so.1
(gdb) i thr
* 1 Thread 16384  main (argc=3, argv=0x7ffffe04) at ../mn/mn.c:709
(gdb) s
(gdb) s
(gdb) c
Continuing.
[New Thread 32769]

Program received signal ?, Unknown signal.
[Switching to Thread 32769]
0x0fdd4d5c in ?? ()
   from /home/mylocal/local/wave/x-ppc-linux2/powerpc-linux/lib/libc.so.6
(gdb) where
#0  0x0fdd4d5c in ?? ()
   from /home/mylocal/local/wave/x-ppc-linux2/powerpc-linux/lib/libc.so.6
(gdb) i thr
* 2 Thread 32769  0x0fdd4d5c in ?? ()
   from /home/mylocal/local/wave/x-ppc-linux2/powerpc-linux/lib/libc.so.6
  1 Thread 16384  0x0fdd4d5c in ?? ()
   from /home/mylocal/local/wave/x-ppc-linux2/powerpc-linux/lib/libc.so.6
(gdb) disp/3i $pc
1: x/3i $pc
0xfdd4d5c:	mfcr	r0
0xfdd4d60:	mr	r31,r3
0xfdd4d64:	andis.	r9,r0,4096
(gdb) si


And gdb never comes back to me(whether I us step, stepi, continue, and
it is interruptable with a C-c, but the pc is still at 0xfdd4d5c).

Any suggestions as to why its not executing under gdb are highly appreciated!  

-- 
Peter Barada                             Peter.Barada@motorola.com
Wizard                                   781-852-2768 (direct)
WaveMark Solutions(A Motorola Company)   781-270-0193 (fax)

Re: Trouble with gdbserver from gdb-5.3, glibc-2.3.2 on ppc32

[Peter Barada]

> > I'm trying to bring up gdbserver2 on an ppc32 embedded linux platform, and
> 
> What's gdbserver2?  I'll just quietly assume you mean gdbserver...

Duh.  I have two versions of gdbserver around, won back from gdb-41.7
that I went through massive pains to get to build , and a newer one
from gdb-5.3.

> Read the source of libthread_db?  No, really.  Then start another
> gdbserver debugging gdbserver, and step through td_ta_new.  It fetches
> the version.  This could be a symptom of symbol lookup being messed up,
> that's almost always the problem.  Versions of the library on host and
> target match?  GDB found the rght target libraries?

Uggh.  Ok, I'll try this method(and add code to td_ta_new to dump why
its tossing TD_VERSION).  I believe that the versions have to match
since I copy the librairs from the cross-compiled toolchain into the
target's /lib directory; how they can differ isn't obvious to me.

I would have thought that building up a toolchain using Dan's
crosstool script would get me what I want.  I read the description of
solib-absolute-prefix, but it doesn't shed much light on why the
*target* is complaining.  I don't see in the remote protocol how it
can ask for a particular library, so I'll dig depper and then get back
to the mailing list.

But back to my original(and probably stupid) question.  Is there any
way to extract information(using readelf, ldd, etc) to determine the
version of the cross-built libpthread.so and libthread_db.so libraries?

-- 
Peter Barada
peter@baradas.org

Re: Trouble with gdbserver from gdb-5.3, glibc-2.3.2 on ppc32

[Dan Kegel]

Peter Barada wrote:
> I would have thought that building up a toolchain using Dan's
> crosstool script would get me what I want.

I've been meaning to add gdb to crosstool (including
its regression test); maybe I can shake loose some
time to do that this week, and help put Peter out of his misery...
- Dan

-- 
Dan Kegel
http://www.kegel.com
http://counter.li.org/cgi-bin/runscript/display-person.cgi?user=78045

Re: Trouble with gdbserver from gdb-5.3, glibc-2.3.2 on ppc32

[Peter Barada]

>I've been meaning to add gdb to crosstool (including
>its regression test); maybe I can shake loose some
>time to do that this week, and help put Peter out of his misery...

Just shoot me now, it will be less painful. :-)

-- 
Peter Barada
peter@baradas.org

Re: Trouble with gdbserver from gdb-5.3, glibc-2.3.2 on ppc32

[Dan Kegel]

Peter Barada wrote:
> sh-2.05b# gdbserver2 :3001 /usr/local/bin/page.strip -m16m -J < /dev/lp0
> Process /usr/local/bin/page.strip created; pid = 35
> Remote debugging from host 10.1.1.2
> gdb: error initializing thread_db library.

Hmm.  I seem to recall running into this, and it being a problem
with solib-absolute-prefix.  Daniel has helped many a
person with this; see e.g.
http://sources.redhat.com/ml/gdb/2003-06/msg00487.html
I wrote up a note about this at
http://www.kegel.com/linux/gdbserver.html
last year when I was going through the same thing.
- Dan

-- 
Dan Kegel
http://www.kegel.com
http://counter.li.org/cgi-bin/runscript/display-person.cgi?user=78045