gdb doesn't show all threads, but gdbserver does

gdb doesn't show all threads, but gdbserver does

[N. van Bolhuis]

I built a gdb (v6.8) for powerpc to be executed on powerpc target
(This GDB was configured as "powerpc-e300c3-linux"...).
If I use this target gdb to debug my powerpc program and hit a
breakpoint "info threads" shows only one thread.
This is incorrect since there are 7 threads (cat /proc/<pid>/status
shows 7 threads).

If I debug the same program using gdbserver on target and gdb on a 
host=i686 (This GDB was configured as "--host=i686-linux
--target=powerpc-e300c3-linux"...) "info threads" shows the expected 7
threads.

How can this be ?

(I noticed the target gdbserver needs libthread_db.so.1, the target gdb
  does not. Maybe this is what's causing this).

Re: gdb doesn't show all threads, but gdbserver does

[teawater]

Both gdbserver and gdb use libthread_db.

gdbserver link together with the lib.
gdb dlopen the lib when it need it.

So I think maybe gdb didn't find the lib or something else. Do you gdb
have some error message?

On Tue, Jan 20, 2009 at 05:09, N. van Bolhuis <nvbolhuis@aimvalley.nl> wrote:
>
> I built a gdb (v6.8) for powerpc to be executed on powerpc target
> (This GDB was configured as "powerpc-e300c3-linux"...).
> If I use this target gdb to debug my powerpc program and hit a
> breakpoint "info threads" shows only one thread.
> This is incorrect since there are 7 threads (cat /proc/<pid>/status
> shows 7 threads).
>
> If I debug the same program using gdbserver on target and gdb on a host=i686 (This GDB was configured as "--host=i686-linux
> --target=powerpc-e300c3-linux"...) "info threads" shows the expected 7
> threads.
>
> How can this be ?
>
> (I noticed the target gdbserver needs libthread_db.so.1, the target gdb
>  does not. Maybe this is what's causing this).
>
>
>

Re: gdb doesn't show all threads, but gdbserver does

[Norbert van Bolhuis]

thanks for your reply teawater.

Btw. I'm using linux-2.6.28, gcc-4.2.4, glibc-2.7 and gdb-6.8.

There is no error message from gdb, see below:

# ./gdb /mnt/norbert/mainctrl.exe
GNU gdb 6.8
Copyright (C) 2008 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "powerpc-e300c3-linux"...
(gdb) br PisIdInterpreter.C:265
Breakpoint 1 at 0x10046644: file /local/common_2/pis-2.0/pis/src/PisIdInterpreter.C, line 265.
(gdb) run
Starting program: /mnt/norbert/mainctrl.exe
========
Build information:
Date: 09-01-15 Time: 13:14:32
CVS tag: (none)
User: nvbolhuis

Ali Model:
Date: 09-01-13 Time: 10:01:26
========


InitProcesses: applications initialized
Username:root
[New LWP 928]
[Switching to LWP 928]

Breakpoint 1, PisIdInterpreter::interpreter (this=0x10365d68)
     at /local/common_2/pis-2.0/pis/src/PisIdInterpreter.C:265
265                 if (!userInput) {
(gdb) info thr
* 1 LWP 928  PisIdInterpreter::interpreter (this=0x10365d68)
     at /local/common_2/pis-2.0/pis/src/PisIdInterpreter.C:265
(gdb) [1] + Stopped                    ./gdb /mnt/norbert/mainctrl.exe
# ps
   PID USER       VSZ STAT COMMAND
     1 root      3184 S    init
     2 root         0 SW<  [kthreadd]
     3 root         0 SW<  [ksoftirqd/0]
     4 root         0 SW<  [watchdog/0]
     5 root         0 SW<  [events/0]
     6 root         0 SW<  [khelper]
    82 root         0 SW<  [kblockd/0]
    95 root         0 SW<  [khubd]
   141 root         0 SW   [pdflush]
   142 root         0 SW   [pdflush]
   143 root         0 SW<  [kswapd0]
   191 root         0 SW<  [aio/0]
   198 root         0 SW<  [nfsiod]
   799 root         0 SW<  [mtdblockd]
   874 root         0 SW<  [hid_compat]
   879 root         0 SW<  [rpciod/0]
   886 root      3188 R    -sh
   892 root         0 SWN  [jffs2_gcd_mtd6]
   897 root      3184 S    telnetd
   919 root     19588 T    ./gdb /mnt/norbert/mainctrl.exe
   920 root     13352 T N  /mnt/norbert/mainctrl.exe
   930 root      3188 R    ps
# cat /proc/920/status
Name:   mainctrl.exe
State:  T (tracing stop)
Tgid:   920
Pid:    920
PPid:   919
TracerPid:      919
Uid:    0       0       0       0
Gid:    0       0       0       0
FDSize: 32
Groups: 0 1 2 3 4 6 10
VmPeak:    13352 kB
VmSize:    11428 kB
VmLck:         0 kB
VmHWM:      3404 kB
VmRSS:      3404 kB
VmData:     6696 kB
VmStk:        84 kB
VmExe:       528 kB
VmLib:      3592 kB
VmPTE:        28 kB
Threads:        7
SigQ:   1/1024
SigPnd: 0000000000000000
ShdPnd: 0000000000000000
SigBlk: 0000000000001000
SigIgn: 0000000000080004
SigCgt: 0000000188102002
CapInh: 0000000000000000
CapPrm: fffffffffffffeff
CapEff: fffffffffffffeff
CapBnd: fffffffffffffeff
voluntary_ctxt_switches:        11
nonvoluntary_ctxt_switches:     2
# fg
./gdb /mnt/norbert/mainctrl.exe

* 1 LWP 928  PisIdInterpreter::interpreter (this=0x10365d68)
     at /local/common_2/pis-2.0/pis/src/PisIdInterpreter.C:265
(gdb)

Re: gdb doesn't show all threads, but gdbserver does

[Paul Pluzhnikov]

On Mon, Jan 19, 2009 at 11:28 PM, Norbert van Bolhuis
<nvbolhuis@aimvalley.nl> wrote:

> # cat /proc/920/status

What did you hope to find from status?

What you really want is /proc/919/maps: this will show whether
libthread_db.so.1 was loaded into gdb, and if so, which one.

It should be the same as the one which shows up in
'ldd gdbserver' output.

Cheers,
-- 
Paul Pluzhnikov

Re: gdb doesn't show all threads, but gdbserver does

[N. van Bolhuis]

Paul Pluzhnikov wrote:
> On Mon, Jan 19, 2009 at 11:28 PM, Norbert van Bolhuis
> <nvbolhuis@aimvalley.nl> wrote:
> 
>> # cat /proc/920/status
> 
> What did you hope to find from status?
> 

proof that 7 threads are really there.

> What you really want is /proc/919/maps: this will show whether
> libthread_db.so.1 was loaded into gdb, and if so, which one.
> 
> It should be the same as the one which shows up in
> 'ldd gdbserver' output.
> 

In fact I did just that. I didn't mention it since it's there
(libthread_db.so.1 is loaded into gdb) so this is not the issue.

Re: gdb doesn't show all threads, but gdbserver does

[Paul Pluzhnikov]

On Tue, Jan 20, 2009 at 12:17 PM, N. van Bolhuis <nvbolhuis@aimvalley.nl> wrote:

> In fact I did just that. I didn't mention it since it's there
> (libthread_db.so.1 is loaded into gdb) so this is not the issue.

Hmm, I wonder if the "stripped libpthread.so.0" struck again.

Could you please do 'ldd /mnt/norbert/mainctrl.exe' on target,
note which libpthread.so.0 it is using, and then

  nm /path/to/libpthread.so.0 | egrep '_version|pthread_threads_events'

If this shows 'no symbols', you have stripped libpthread.so.0 on
target (but not on host). Don't do that. Do "strip --strip-debug"
instead.

Cheers,
-- 
Paul Pluzhnikov

Re: gdb doesn't show all threads, but gdbserver does

[N. van Bolhuis]

Paul Pluzhnikov wrote:
> On Tue, Jan 20, 2009 at 12:17 PM, N. van Bolhuis <nvbolhuis@aimvalley.nl> wrote:
> 
>> In fact I did just that. I didn't mention it since it's there
>> (libthread_db.so.1 is loaded into gdb) so this is not the issue.
> 
> Hmm, I wonder if the "stripped libpthread.so.0" struck again.
> 
> Could you please do 'ldd /mnt/norbert/mainctrl.exe' on target,
> note which libpthread.so.0 it is using, and then
> 
>   nm /path/to/libpthread.so.0 | egrep '_version|pthread_threads_events'
> 
> If this shows 'no symbols', you have stripped libpthread.so.0 on
> target (but not on host). Don't do that. Do "strip --strip-debug"
> instead.
> 

aha, that must be it. I did (fully) strip all libraries on target.

yes if I do "strip --strip-debug" it works.

thanks a lot!

I did not realize gdb needs libpthread debug symbols to "discover"
threads.
Hmm, if I think about it I still not fully understand why.
Anyway it work now and I'm happy.

Re: gdb doesn't show all threads, but gdbserver does

[Andreas Schwab]

"N. van Bolhuis" <nvbolhuis@aimvalley.nl> writes:

> I did not realize gdb needs libpthread debug symbols to "discover"
> threads.

It's actually libthread_db that needs a non-global symbol from
libpthread.

Andreas.

-- 
Andreas Schwab, SuSE Labs, schwab@suse.de
SuSE Linux Products GmbH, Maxfeldstraße 5, 90409 Nürnberg, Germany
PGP key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: gdb doesn't show all threads, but gdbserver does

[Paul Pluzhnikov]

On Tue, Jan 20, 2009 at 2:24 PM, Andreas Schwab <schwab@suse.de> wrote:

> It's actually libthread_db that needs a non-global symbol from
> libpthread.

IOW, it's a glibc feature :)

Stripping libpthread appears to be a very common mistake:

- we've seen it twice on this list in the last two weeks,
- I made it myself 4 weeks ago, even though I *knew* that libpthread
  must not be stripped.

I wonder if we should warn glibc users? Something like:

  Warning: your /lib64/libpthread.so.0 has been stripped.
  GDB will not see other threads, and multithreaded debugging will
  be unreliable.
  See http://sourceware.org/gdb/wiki/... for further info.

-- 
Paul Pluzhnikov

Re: gdb doesn't show all threads, but gdbserver does

[teawater]

Agree. Most of multi-thread bug that I meet are stripped libpthread.

On Wed, Jan 21, 2009 at 06:45, Paul Pluzhnikov <ppluzhnikov@google.com> wrote:
> On Tue, Jan 20, 2009 at 2:24 PM, Andreas Schwab <schwab@suse.de> wrote:
>
>> It's actually libthread_db that needs a non-global symbol from
>> libpthread.
>
> IOW, it's a glibc feature :)
>
> Stripping libpthread appears to be a very common mistake:
>
> - we've seen it twice on this list in the last two weeks,
> - I made it myself 4 weeks ago, even though I *knew* that libpthread
>  must not be stripped.
>
> I wonder if we should warn glibc users? Something like:
>
>  Warning: your /lib64/libpthread.so.0 has been stripped.
>  GDB will not see other threads, and multithreaded debugging will
>  be unreliable.
>  See http://sourceware.org/gdb/wiki/... for further info.
>
> --
> Paul Pluzhnikov
>

Re: gdb doesn't show all threads, but gdbserver does

[Norbert van Bolhuis]

Paul Pluzhnikov wrote:
> On Tue, Jan 20, 2009 at 2:24 PM, Andreas Schwab <schwab@suse.de> wrote:
> 
>> It's actually libthread_db that needs a non-global symbol from
>> libpthread.
> 
> IOW, it's a glibc feature :)
> 
> Stripping libpthread appears to be a very common mistake:
> 
> - we've seen it twice on this list in the last two weeks,
> - I made it myself 4 weeks ago, even though I *knew* that libpthread
>   must not be stripped.
> 
> I wonder if we should warn glibc users? Something like:
> 
>   Warning: your /lib64/libpthread.so.0 has been stripped.
>   GDB will not see other threads, and multithreaded debugging will
>   be unreliable.
>   See http://sourceware.org/gdb/wiki/... for further info.
> 

I consulted the GDB manual first, then google.

http://sourceware.org/gdb/wiki is an interesting source of information.
I expect to see this common mistake being mentioned there, but I can't
find it.

Anyway, thanks to everybody for all answers.

Re: gdb doesn't show all threads, but gdbserver does

[teawater]

What about FAQ?

On Wed, Jan 21, 2009 at 18:32, Norbert van Bolhuis
<nvbolhuis@aimvalley.nl> wrote:
>
> Paul Pluzhnikov wrote:
>>
>> On Tue, Jan 20, 2009 at 2:24 PM, Andreas Schwab <schwab@suse.de> wrote:
>>
>>> It's actually libthread_db that needs a non-global symbol from
>>> libpthread.
>>
>> IOW, it's a glibc feature :)
>>
>> Stripping libpthread appears to be a very common mistake:
>>
>> - we've seen it twice on this list in the last two weeks,
>> - I made it myself 4 weeks ago, even though I *knew* that libpthread
>>  must not be stripped.
>>
>> I wonder if we should warn glibc users? Something like:
>>
>>  Warning: your /lib64/libpthread.so.0 has been stripped.
>>  GDB will not see other threads, and multithreaded debugging will
>>  be unreliable.
>>  See http://sourceware.org/gdb/wiki/... for further info.
>>
>
> I consulted the GDB manual first, then google.
>
> http://sourceware.org/gdb/wiki is an interesting source of information.
> I expect to see this common mistake being mentioned there, but I can't
> find it.
>
> Anyway, thanks to everybody for all answers.
>
>

Re: gdb doesn't show all threads, but gdbserver does

[teawater]

My mean is maybe you can update this question and answer to FAQ in wiki. :)

On Wed, Jan 21, 2009 at 19:24, teawater <teawater@gmail.com> wrote:
> What about FAQ?
>
> On Wed, Jan 21, 2009 at 18:32, Norbert van Bolhuis
> <nvbolhuis@aimvalley.nl> wrote:
>>
>> Paul Pluzhnikov wrote:
>>>
>>> On Tue, Jan 20, 2009 at 2:24 PM, Andreas Schwab <schwab@suse.de> wrote:
>>>
>>>> It's actually libthread_db that needs a non-global symbol from
>>>> libpthread.
>>>
>>> IOW, it's a glibc feature :)
>>>
>>> Stripping libpthread appears to be a very common mistake:
>>>
>>> - we've seen it twice on this list in the last two weeks,
>>> - I made it myself 4 weeks ago, even though I *knew* that libpthread
>>>  must not be stripped.
>>>
>>> I wonder if we should warn glibc users? Something like:
>>>
>>>  Warning: your /lib64/libpthread.so.0 has been stripped.
>>>  GDB will not see other threads, and multithreaded debugging will
>>>  be unreliable.
>>>  See http://sourceware.org/gdb/wiki/... for further info.
>>>
>>
>> I consulted the GDB manual first, then google.
>>
>> http://sourceware.org/gdb/wiki is an interesting source of information.
>> I expect to see this common mistake being mentioned there, but I can't
>> find it.
>>
>> Anyway, thanks to everybody for all answers.
>>
>>
>

Re: gdb doesn't show all threads, but gdbserver does

[Paul Pluzhnikov]

I just updated the wiki ...

On Wed, Jan 21, 2009 at 5:45 AM, teawater <teawater@gmail.com> wrote:
> My mean is maybe you can update this question and answer to FAQ in wiki. :)
>
> On Wed, Jan 21, 2009 at 19:24, teawater <teawater@gmail.com> wrote:
>> What about FAQ?
>>
>> On Wed, Jan 21, 2009 at 18:32, Norbert van Bolhuis
>> <nvbolhuis@aimvalley.nl> wrote:
>>>
>>> Paul Pluzhnikov wrote:
>>>>
>>>> On Tue, Jan 20, 2009 at 2:24 PM, Andreas Schwab <schwab@suse.de> wrote:
>>>>
>>>>> It's actually libthread_db that needs a non-global symbol from
>>>>> libpthread.
>>>>
>>>> IOW, it's a glibc feature :)
>>>>
>>>> Stripping libpthread appears to be a very common mistake:
>>>>
>>>> - we've seen it twice on this list in the last two weeks,
>>>> - I made it myself 4 weeks ago, even though I *knew* that libpthread
>>>>  must not be stripped.
>>>>
>>>> I wonder if we should warn glibc users? Something like:
>>>>
>>>>  Warning: your /lib64/libpthread.so.0 has been stripped.
>>>>  GDB will not see other threads, and multithreaded debugging will
>>>>  be unreliable.
>>>>  See http://sourceware.org/gdb/wiki/... for further info.
>>>>
>>>
>>> I consulted the GDB manual first, then google.
>>>
>>> http://sourceware.org/gdb/wiki is an interesting source of information.
>>> I expect to see this common mistake being mentioned there, but I can't
>>> find it.
>>>
>>> Anyway, thanks to everybody for all answers.
>>>
>>>
>>
>



-- 
Paul Pluzhnikov

Re: gdb doesn't show all threads, but gdbserver does

[teawater]

I make a patch to output a warning.
http://sourceware.org/ml/gdb-patches/2009-02/msg00234.html

Thanks,
Hui

On Wed, Jan 21, 2009 at 06:45, Paul Pluzhnikov <ppluzhnikov@google.com> wrote:
> On Tue, Jan 20, 2009 at 2:24 PM, Andreas Schwab <schwab@suse.de> wrote:
>
>> It's actually libthread_db that needs a non-global symbol from
>> libpthread.
>
> IOW, it's a glibc feature :)
>
> Stripping libpthread appears to be a very common mistake:
>
> - we've seen it twice on this list in the last two weeks,
> - I made it myself 4 weeks ago, even though I *knew* that libpthread
>  must not be stripped.
>
> I wonder if we should warn glibc users? Something like:
>
>  Warning: your /lib64/libpthread.so.0 has been stripped.
>  GDB will not see other threads, and multithreaded debugging will
>  be unreliable.
>  See http://sourceware.org/gdb/wiki/... for further info.
>
> --
> Paul Pluzhnikov
>

Re: gdb doesn't show all threads, but gdbserver does

[teawater]

http://sourceware.org/ml/gdb-patches/2009-02/msg00232.html

On Thu, Feb 12, 2009 at 11:07, teawater <teawater@gmail.com> wrote:
> I make a patch to output a warning.
> http://sourceware.org/ml/gdb-patches/2009-02/msg00234.html
>
> Thanks,
> Hui
>
> On Wed, Jan 21, 2009 at 06:45, Paul Pluzhnikov <ppluzhnikov@google.com> wrote:
>> On Tue, Jan 20, 2009 at 2:24 PM, Andreas Schwab <schwab@suse.de> wrote:
>>
>>> It's actually libthread_db that needs a non-global symbol from
>>> libpthread.
>>
>> IOW, it's a glibc feature :)
>>
>> Stripping libpthread appears to be a very common mistake:
>>
>> - we've seen it twice on this list in the last two weeks,
>> - I made it myself 4 weeks ago, even though I *knew* that libpthread
>>  must not be stripped.
>>
>> I wonder if we should warn glibc users? Something like:
>>
>>  Warning: your /lib64/libpthread.so.0 has been stripped.
>>  GDB will not see other threads, and multithreaded debugging will
>>  be unreliable.
>>  See http://sourceware.org/gdb/wiki/... for further info.
>>
>> --
>> Paul Pluzhnikov
>>
>

Re: gdb doesn't show all threads, but gdbserver does

[teawater]

Really?
Why your gdbserver can be use? Is it static link?

On Wed, Jan 21, 2009 at 05:17, N. van Bolhuis <nvbolhuis@aimvalley.nl> wrote:
> Paul Pluzhnikov wrote:
>>
>> On Tue, Jan 20, 2009 at 12:17 PM, N. van Bolhuis <nvbolhuis@aimvalley.nl>
>> wrote:
>>
>>> In fact I did just that. I didn't mention it since it's there
>>> (libthread_db.so.1 is loaded into gdb) so this is not the issue.
>>
>> Hmm, I wonder if the "stripped libpthread.so.0" struck again.
>>
>> Could you please do 'ldd /mnt/norbert/mainctrl.exe' on target,
>> note which libpthread.so.0 it is using, and then
>>
>>  nm /path/to/libpthread.so.0 | egrep '_version|pthread_threads_events'
>>
>> If this shows 'no symbols', you have stripped libpthread.so.0 on
>> target (but not on host). Don't do that. Do "strip --strip-debug"
>> instead.
>>
>
> aha, that must be it. I did (fully) strip all libraries on target.
>
> yes if I do "strip --strip-debug" it works.
>
> thanks a lot!
>
> I did not realize gdb needs libpthread debug symbols to "discover"
> threads.
> Hmm, if I think about it I still not fully understand why.
> Anyway it work now and I'm happy.
>
>
>

Re: gdb doesn't show all threads, but gdbserver does

[Paul Pluzhnikov]

On Tue, Jan 20, 2009 at 4:05 PM, teawater <teawater@gmail.com> wrote:
> Really?
> Why your gdbserver can be use? Is it static link?

I think it works for OP because with gdbserver, the host GDB is
doing symbol lookups, and the copy of target libpthread which host
GDB uses has not been stripped.

Cheers,
-- 
Paul Pluzhnikov

Re: gdb doesn't show all threads, but gdbserver does

[teawater]

I think gdbserver need non-strip libpthread too. I met this issue in before.
So it make me confused. :)

On Wed, Jan 21, 2009 at 08:14, Paul Pluzhnikov <ppluzhnikov@google.com> wrote:
> On Tue, Jan 20, 2009 at 4:05 PM, teawater <teawater@gmail.com> wrote:
>> Really?
>> Why your gdbserver can be use? Is it static link?
>
> I think it works for OP because with gdbserver, the host GDB is
> doing symbol lookups, and the copy of target libpthread which host
> GDB uses has not been stripped.
>
> Cheers,
> --
> Paul Pluzhnikov
>

Re: gdb doesn't show all threads, but gdbserver does

[teawater]

This part of code in gdb/gdbserver/thread-db.c:thread_db_init.

On Wed, Jan 21, 2009 at 08:25, teawater <teawater@gmail.com> wrote:
> I think gdbserver need non-strip libpthread too. I met this issue in before.
> So it make me confused. :)
>
> On Wed, Jan 21, 2009 at 08:14, Paul Pluzhnikov <ppluzhnikov@google.com> wrote:
>> On Tue, Jan 20, 2009 at 4:05 PM, teawater <teawater@gmail.com> wrote:
>>> Really?
>>> Why your gdbserver can be use? Is it static link?
>>
>> I think it works for OP because with gdbserver, the host GDB is
>> doing symbol lookups, and the copy of target libpthread which host
>> GDB uses has not been stripped.
>>
>> Cheers,
>> --
>> Paul Pluzhnikov
>>
>

Re: gdb doesn't show all threads, but gdbserver does

[Daniel Jacobowitz]

On Wed, Jan 21, 2009 at 10:35:12AM +0800, teawater wrote:
> This part of code in gdb/gdbserver/thread-db.c:thread_db_init.

No, what Paul said is correct.  The symbol lookup is performed by the
GDB client, not by gdbserver - so it uses the copy of libpthread.so on
the host.

-- 
Daniel Jacobowitz
CodeSourcery

Re: gdb doesn't show all threads, but gdbserver does

[teawater]

Thank you for help me make it clean, Daniel. :)

Hui

On Wed, Jan 21, 2009 at 12:16, Daniel Jacobowitz <drow@false.org> wrote:
> On Wed, Jan 21, 2009 at 10:35:12AM +0800, teawater wrote:
>> This part of code in gdb/gdbserver/thread-db.c:thread_db_init.
>
> No, what Paul said is correct.  The symbol lookup is performed by the
> GDB client, not by gdbserver - so it uses the copy of libpthread.so on
> the host.
>
> --
> Daniel Jacobowitz
> CodeSourcery
>