"Cannot find new threads" on Fedora 9, but not on CentOS 5 (?)

"Cannot find new threads" on Fedora 9, but not on CentOS 5 (?)

[Andrew Lofthouse]

I'm trying to debug an application that throws a SEGFAULT in a shared 
library.  On Fedora 9 (and on Ubuntu 8.04), I get the following error:

[Thread debugging using libthread_db enabled]
Error while reading shared library symbols:
Cannot find new threads: generic error
Cannot find new threads: generic error

However, when using CentOS 5, the debugging session seems to work fine 
and gives the following output:

[Detaching after fork from child process 1973]
[Detaching after fork from child process 1992]
[Detaching after fork from child process 2004]

Program received signal SIGSEGV, Segmentation fault.

etc, etc--basically gdb works and tells me where the segmentation fault is.

So, I know that Fedora 9/Ubuntu 8.04 use gdb version 6.8, while CentOS 5 
uses version 6.5.  Why can an earlier version of gdb work fine, while 
the later version doesn't work?  What can I do so that the later version 
works?

Thanks,

Andrew L.

Re: "Cannot find new threads" on Fedora 9, but not on CentOS 5 (?)

[Paul Pluzhnikov]

On Mon, Oct 27, 2008 at 5:24 PM, Andrew Lofthouse <loftyhauser@gmail.com> wrote:
> I'm trying to debug an application that throws a SEGFAULT in a shared
> library.  On Fedora 9 (and on Ubuntu 8.04), I get the following error:
>
> [Thread debugging using libthread_db enabled]
> Error while reading shared library symbols:
> Cannot find new threads: generic error
> Cannot find new threads: generic error

Please see:
  http://sourceware.org/ml/gdb/2008-08/msg00208.html
for possible reasons.

> However, when using CentOS 5, the debugging session seems to work fine and
> gives the following output:
>
> [Detaching after fork from child process 1973]
> [Detaching after fork from child process 1992]
> [Detaching after fork from child process 2004]
>
> Program received signal SIGSEGV, Segmentation fault.
>
> etc, etc--basically gdb works and tells me where the segmentation fault is.
>
> So, I know that Fedora 9/Ubuntu 8.04 use gdb version 6.8, while CentOS 5
> uses version 6.5.  Why can an earlier version of gdb work fine, while the
> later version doesn't work?  What can I do so that the later version works?

This is most likely result of "pilot error" (stripping
libpthread.so.0, or incomplete installation of glibc), and has
nothing to do with actual version of GDB.

Cheers,
-- 
Paul Pluzhnikov

Re: "Cannot find new threads" on Fedora 9, but not on CentOS 5 (?)

[Andrew Lofthouse]

Sorry for the delay; I don't have access to my personal e-mail account 
at work...

Paul Pluzhnikov wrote:
> - What does 'ldd /path/to/app' say?
> - What does 'nm /path/to/libpthread.so.0 | grep _version' say (where
>   /path/to/libpthread.so.0 is the one which ldd prints)?

Hmm, ldd does not list libpthread.so.0:

[andrew@fedora-vm ufs]$ ldd /usr/local/bin/ufs2oogl2D
         linux-gate.so.1 =>  (0x0012e000)
         libufs2D-0.9.so.2 => /usr/local/lib/libufs2D-0.9.so.2 (0x0012f000)
         libgts-0.7.so.5 => /usr/local/lib/libgts-0.7.so.5 (0x001d8000)
         libgmodule-2.0.so.0 => /lib/libgmodule-2.0.so.0 (0x00228000)
         libdl.so.2 => /lib/libdl.so.2 (0x0022c000)
         libglib-2.0.so.0 => /lib/libglib-2.0.so.0 (0x00231000)
         libm.so.6 => /lib/libm.so.6 (0x00312000)
         libc.so.6 => /lib/libc.so.6 (0x0033b000)
         libselinux.so.1 => /lib/libselinux.so.1 (0x004a4000)
         /lib/ld-linux.so.2 (0x00110000)

But, gdb still seems to use libpthread when debugging the application 
(see below).

> 
>>> This is most likely result of "pilot error" (stripping
>>> libpthread.so.0, or incomplete installation of glibc), and has
>>> nothing to do with actual version of GDB.
>> Except that libpthread.so.0 is not stripped (on any of the installations).
> 
> How do you know that?

Now I'm confused.  It looks like the non-debug library is not stripped, 
but the debug library is stripped (?):

[andrew@fedora-vm ufs]$ file /lib/libpthread-2.8.so
/lib/libpthread-2.8.so: ELF 32-bit LSB shared object, Intel 80386, 
version 1 (SYSV), for GNU/Linux 2.6.9, not stripped

[andrew@fedora-vm ufs]$ file /usr/lib/debug/lib/libpthread-2.8.so.debug
/usr/lib/debug/lib/libpthread-2.8.so.debug: ELF 32-bit LSB shared 
object, Intel 80386, version 1 (SYSV), for GNU/Linux 2.6.9, stripped

Now, looking in gdb, it looks like gdb is using /lib/libpthread.so.0, 
but can't find its symbols:

[andrew@fedora-vm ufs]$ gdb /usr/local/bin/ufs2oogl2D
GNU gdb Fedora (6.8-1.fc9)
Copyright (C) 2008 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later 
<http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "i386-redhat-linux-gnu"...
(gdb) run -T "rho" < UAM10_002_UFS-005000.sim > test.plt
Starting program: /usr/local/bin/ufs2oogl2D -T "rho" < 
UAM10_002_UFS-005000.sim > test.plt

Detaching after fork from child process 8745.
[Thread debugging using libthread_db enabled]
Error while reading shared library symbols:
Cannot find new threads: generic error
Cannot find new threads: generic error
(gdb) info share
 From        To          Syms Read   Shared Object Library
0x00110810  0x0012724f  Yes         /lib/ld-linux.so.2
0x0013ef00  0x001ca5d8  Yes         /usr/local/lib/libufs2D-0.9.so.2
0x001dfb80  0x00221638  Yes         /usr/local/lib/libgts-0.7.so.5
0x00228c50  0x00229cc8  Yes         /lib/libgmodule-2.0.so.0
0x0022ca60  0x0022da68  Yes         /lib/libdl.so.2
0x002401f0  0x002bf4d8  Yes         /lib/libglib-2.0.so.0
0x00315420  0x0032fe48  Yes         /lib/libm.so.6
0x003513b0  0x0045dc88  Yes         /lib/libc.so.6
0x004a7bb0  0x004b7ce8  Yes         /lib/libselinux.so.1
0x004c0430  0x004c05a8  Yes         /tmp/gfsZ54KNU
0x004c2f50  0x004c4748  Yes         /lib/libgthread-2.0.so.0
0x004c8940  0x004cd088  Yes         /lib/librt.so.1
0x004d51f0  0x004e0b58  No          /lib/libpthread.so.0
(gdb) share
Symbols already loaded for /lib/ld-linux.so.2
Symbols already loaded for /usr/local/lib/libufs2D-0.9.so.2
Symbols already loaded for /usr/local/lib/libgts-0.7.so.5
Symbols already loaded for /lib/libgmodule-2.0.so.0
Symbols already loaded for /lib/libdl.so.2
Symbols already loaded for /lib/libglib-2.0.so.0
Symbols already loaded for /lib/libm.so.6
Symbols already loaded for /lib/libc.so.6
Symbols already loaded for /lib/libselinux.so.1
Symbols already loaded for /tmp/gfsZ54KNU
Symbols already loaded for /lib/libgthread-2.0.so.0
Symbols already loaded for /lib/librt.so.1
Loaded symbols for /lib/libpthread.so.0
(gdb) info share
 From        To          Syms Read   Shared Object Library
0x00110810  0x0012724f  Yes         /lib/ld-linux.so.2
0x0013ef00  0x001ca5d8  Yes         /usr/local/lib/libufs2D-0.9.so.2
0x001dfb80  0x00221638  Yes         /usr/local/lib/libgts-0.7.so.5
0x00228c50  0x00229cc8  Yes         /lib/libgmodule-2.0.so.0
0x0022ca60  0x0022da68  Yes         /lib/libdl.so.2
0x002401f0  0x002bf4d8  Yes         /lib/libglib-2.0.so.0
0x00315420  0x0032fe48  Yes         /lib/libm.so.6
0x003513b0  0x0045dc88  Yes         /lib/libc.so.6
0x004a7bb0  0x004b7ce8  Yes         /lib/libselinux.so.1
0x004c0430  0x004c05a8  Yes         /tmp/gfsZ54KNU
0x004c2f50  0x004c4748  Yes         /lib/libgthread-2.0.so.0
0x004c8940  0x004cd088  Yes         /lib/librt.so.1
0x004d51f0  0x004e0b58  Yes         /lib/libpthread.so.0
(gdb) run -T "rho" < UAM10_002_UFS-005000.sim > test.plt
The program being debugged has been started already.
Start it from the beginning? (y or n) y

Starting program: /usr/local/bin/ufs2oogl2D -T "rho" < 
UAM10_002_UFS-005000.sim > test.plt

Detaching after fork from child process 8775.
[Thread debugging using libthread_db enabled]
Error while reading shared library symbols:
Cannot find new threads: generic error
Cannot find new threads: generic error
(gdb) info share
 From        To          Syms Read   Shared Object Library
0x00110810  0x0012724f  Yes         /lib/ld-linux.so.2
0x0013ef00  0x001ca5d8  Yes         /usr/local/lib/libufs2D-0.9.so.2
0x001dfb80  0x00221638  Yes         /usr/local/lib/libgts-0.7.so.5
0x00228c50  0x00229cc8  Yes         /lib/libgmodule-2.0.so.0
0x0022ca60  0x0022da68  Yes         /lib/libdl.so.2
0x002401f0  0x002bf4d8  Yes         /lib/libglib-2.0.so.0
0x00315420  0x0032fe48  Yes         /lib/libm.so.6
0x003513b0  0x0045dc88  Yes         /lib/libc.so.6
0x004a7bb0  0x004b7ce8  Yes         /lib/libselinux.so.1
0x004c0430  0x004c05a8  Yes         /tmp/gfsZ54KNU
0x004c2f50  0x004c4748  Yes         /lib/libgthread-2.0.so.0
0x004c8940  0x004cd088  Yes         /lib/librt.so.1
0x004d51f0  0x004e0b58  No          /lib/libpthread.so.0

Following your example:

> I have a fresh out-of-the-box installation of Fedora 9 for x86_64.
> Here is what I see:
> 
> $ cat t.c
> #include <pthread.h>
> 
> void *fn(void *p)
> {
>    char *cp = (char *)p;
>    cp[0] = 'a';
>    return p;
> }
> 
> int main()
> {
>     pthread_t tid;
>     pthread_create(&tid, 0, fn, 0);
>     pthread_join(tid, 0);
>     return 0;
> }
> 
> $ gcc -g -pthread t.c && gdb ./a.out
> GNU gdb Fedora (6.8-1.fc9)
> Copyright (C) 2008 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
> and "show warranty" for details.
> This GDB was configured as "x86_64-redhat-linux-gnu"...
> (gdb) run
> Starting program: a.out
> [Thread debugging using libthread_db enabled]
> [New Thread 0x7ff617b6c6f0 (LWP 2907)]
> [New Thread 0x42533950 (LWP 2910)]
> 
> Program received signal SIGSEGV, Segmentation fault.
> [Switching to Thread 0x42533950 (LWP 2910)]
> 0x0000000000400590 in fn (p=0x0) at t.c:6
> 6          cp[0] = 'a';
> Missing separate debuginfos, use: debuginfo-install glibc.x86_64
> (gdb) info thread
> * 2 Thread 0x42533950 (LWP 2910)  0x0000000000400590 in fn (p=0x0) at t.c:6
>   1 Thread 0x7ff617b6c6f0 (LWP 2907)  0x0000003c9f807b75 in
> pthread_join () from /lib64/libpthread.so.0
> (gdb) quit
> The program is running.  Exit anyway? (y or n) y
> 
> $ rpm -qf /lib64/libpthread.so.0 /lib64/libthread_db.so.1
> glibc-2.8-3.x86_64
> glibc-2.8-3.x86_64
> 
> I did 'debuginfo-install glibc.x86_64' just to check if that makes
> any difference... It didn't:
> 
> (gdb) run
> Starting program: /root/a.out
> [Thread debugging using libthread_db enabled]
> [New Thread 0x7f7e9e9ce6f0 (LWP 2948)]
> [New Thread 0x426f2950 (LWP 2951)]
> 
> Program received signal SIGSEGV, Segmentation fault.
> [Switching to Thread 0x426f2950 (LWP 2951)]
> 0x0000000000400590 in fn (p=0x0) at t.c:6
> 6          cp[0] = 'a';
> (gdb) inf thread
> * 2 Thread 0x426f2950 (LWP 2951)  0x0000000000400590 in fn (p=0x0) at t.c:6
>   1 Thread 0x7f7e9e9ce6f0 (LWP 2948)  0x0000003c9f807b75 in
> pthread_join (threadid=<value optimized out>,
>     thread_return=<value optimized out>) at pthread_join.c:89
> (gdb)
> 

I get essentially the same output.

It looks like something to do with gdb not finding the symbols for 
/lib/libpthread.so.0.  Does this have something to do with ldd not 
listing libpthread as a shared lib?  Or perhaps something with 
libpthread really being stripped (but file shows otherwise)?


This is quite perplexing, and I would appreciate any help!

Regards,

Andrew L.

Re: "Cannot find new threads" on Fedora 9, but not on CentOS 5 (?)

[Daniel Jacobowitz]

On Tue, Oct 28, 2008 at 09:15:01PM -0400, Andrew Lofthouse wrote:
> It looks like something to do with gdb not finding the symbols for  
> /lib/libpthread.so.0.  Does this have something to do with ldd not  
> listing libpthread as a shared lib?  Or perhaps something with libpthread 
> really being stripped (but file shows otherwise)?

Yeah - something having to do with a library, probably libgthread,
dynamically loading it.  Sounds like it might be a bug in GDB.
Dynamically loading the thread library is unusual; it also stresses
bits of GLIBC that are not ordinarily used.

-- 
Daniel Jacobowitz
CodeSourcery

Re: "Cannot find new threads" on Fedora 9, but not on CentOS 5 (?)

[Paul Pluzhnikov]

On Tue, Oct 28, 2008 at 6:15 PM, Andrew Lofthouse <loftyhauser@gmail.com> wrote:

>> - What does 'ldd /path/to/app' say?
>> - What does 'nm /path/to/libpthread.so.0 | grep _version' say (where
>>  /path/to/libpthread.so.0 is the one which ldd prints)?
>
> Hmm, ldd does not list libpthread.so.0:
>
> [andrew@fedora-vm ufs]$ ldd /usr/local/bin/ufs2oogl2D
>        linux-gate.so.1 =>  (0x0012e000)
>        libufs2D-0.9.so.2 => /usr/local/lib/libufs2D-0.9.so.2 (0x0012f000)
>        libgts-0.7.so.5 => /usr/local/lib/libgts-0.7.so.5 (0x001d8000)
>        libgmodule-2.0.so.0 => /lib/libgmodule-2.0.so.0 (0x00228000)
>        libdl.so.2 => /lib/libdl.so.2 (0x0022c000)
>        libglib-2.0.so.0 => /lib/libglib-2.0.so.0 (0x00231000)
>        libm.so.6 => /lib/libm.so.6 (0x00312000)
>        libc.so.6 => /lib/libc.so.6 (0x0033b000)
>        libselinux.so.1 => /lib/libselinux.so.1 (0x004a4000)
>        /lib/ld-linux.so.2 (0x00110000)
>
> But, gdb still seems to use libpthread when debugging the application (see
> below).

The application apparently dynamically loads /tmp/gfsZ54KNU,
which in turn depends (perhaps indirectly) on libpthread.so.0

This is supposed to work in theory, but apparently has glitches on
Fedora 9, though I can't reproduce that on my x86_64 Fedora 9.

See what you get with a test case at the bottom.

As a workaround, you may try to build your ufs2oogl2D with '-pthread'.

The other thing which may provide clues is to 'set verbose on'
before running the program.

> Now I'm confused.  It looks like the non-debug library is not stripped, but
> the debug library is stripped (?):

This is good: libpthread.so.0 is not stripped:

> [andrew@fedora-vm ufs]$ file /lib/libpthread-2.8.so
> /lib/libpthread-2.8.so: ELF 32-bit LSB shared object, Intel 80386, version 1
> (SYSV), for GNU/Linux 2.6.9, not stripped

This isn't a library at all; this is just the debug bits for it.
Since these bits do not have a symbol table, file(1) says that it's
stripped. You can ignore that:

> [andrew@fedora-vm ufs]$ file /usr/lib/debug/lib/libpthread-2.8.so.debug
> /usr/lib/debug/lib/libpthread-2.8.so.debug: ELF 32-bit LSB shared object,
> Intel 80386, version 1 (SYSV), for GNU/Linux 2.6.9, stripped

> Now, looking in gdb, it looks like gdb is using /lib/libpthread.so.0, but
> can't find its symbols:

It can find the symbols all right, but while GDB is adding symbols,
it hits some error, likely having to do with the process "suddenly"
becoming multithreaded.

Anyway, do you get any problems with the test case below.
What does "info threads" at crash point say?

--- cut --- thread-so-main.c ---
// compile with "gcc thread-so-main.c -ldl"
#include <dlfcn.h>

int main()
{
  void *h = dlopen("./thread-so.so", RTLD_LAZY);
  void (*fn)(void) = dlsym(h, "foo");
  fn();
  return 0;
}
--- cut ---

--- cut --- thread-so.c ---
// compile with "gcc -g -fPIC -shared -o thread-so.so thread-so.c
/lib/libgthread-2.0.so.0"
#include <pthread.h>

void *fn(void *p)
{
  char *cp = (char *)p;
  cp[0] = 'a';
  return p;
}

void foo()
{
  pthread_t tid;
  pthread_create(&tid, 0, fn, 0);
  pthread_join(tid, 0);
}
--- cut ---

For reference, here is what I see (all good):

GNU gdb Fedora (6.8-1.fc9)
Copyright (C) 2008 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu"...
(no debugging symbols found)
(gdb) r
Starting program: /mnt/ppluzhnikov/tmp/a.out
[Thread debugging using libthread_db enabled]
[New Thread 0x7f1a2d0136f0 (LWP 3004)]
[New Thread 0x416de950 (LWP 3007)]

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x416de950 (LWP 3007)]
0x0000000000110628 in fn (p=0x0) at thread-so.c:6
6        cp[0] = 'a';
Missing separate debuginfos, use: debuginfo-install glib2.x86_64
libselinux.x86_64
(gdb) inf thread
* 2 Thread 0x416de950 (LWP 3007)  0x0000000000110628 in fn (p=0x0) at
thread-so.c:6
  1 Thread 0x7f1a2d0136f0 (LWP 3004)  0x0000003c9f807b75 in
pthread_join (threadid=<value optimized out>, thread_return=<value
optimized out>)
    at pthread_join.c:89
(gdb) bt
#0  0x0000000000110628 in fn (p=0x0) at thread-so.c:6
#1  0x0000003c9f80729a in start_thread (arg=<value optimized out>) at
pthread_create.c:297
#2  0x0000003c9e8e42cd in clone () from /lib64/libc.so.6
(gdb) thread 1
[Switching to thread 1 (Thread 0x7f1a2d0136f0 (LWP 3004))]#0
0x0000003c9f807b75 in pthread_join (threadid=<value optimized out>,
    thread_return=<value optimized out>) at pthread_join.c:89
89         lll_wait_tid (pd->tid);
(gdb) bt
#0  0x0000003c9f807b75 in pthread_join (threadid=<value optimized
out>, thread_return=<value optimized out>) at pthread_join.c:89
#1  0x0000000000110661 in foo () at thread-so.c:14
#2  0x000000000040059f in main ()
(gdb) q
The program is running.  Exit anyway? (y or n) y



-- 
Paul Pluzhnikov

Re: "Cannot find new threads" on Fedora 9, but not on CentOS 5 (?)

[Andrew Lofthouse]

Paul Pluzhnikov wrote:

  > The application apparently dynamically loads /tmp/gfsZ54KNU,
> which in turn depends (perhaps indirectly) on libpthread.so.0
> 
> This is supposed to work in theory, but apparently has glitches on
> Fedora 9, though I can't reproduce that on my x86_64 Fedora 9.
> 
> See what you get with a test case at the bottom.
> 
> As a workaround, you may try to build your ufs2oogl2D with '-pthread'.

This indeed works ("./configure LIBS=-pthread"):

[andrew@fedora-vm ufs]$ gdb /usr/local/bin/ufs2oogl2D
GNU gdb Fedora (6.8-1.fc9)
Copyright (C) 2008 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later 
<http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "i386-redhat-linux-gnu"...
(gdb) run -T "rho" < UAM10_002_UFS-005000.sim > test.plt
Starting program: /usr/local/bin/ufs2oogl2D -T "rho" < 
UAM10_002_UFS-005000.sim > test.plt
[Thread debugging using libthread_db enabled]

Detaching after fork from child process 16872.
[New Thread 0xb7f7c710 (LWP 16869)]

Detaching after fork from child process 16883.
Detaching after fork from child process 16894.
Detaching after fork from child process 16905.
Detaching after fork from child process 16916.
Detaching after fork from child process 16927.

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0xb7f7c710 (LWP 16869)]
0x00137d17 in symmetry_gasdynamics (f=0xbff90c30, b=0x9683570)
     at boundary_gasdynamics.c:282
282	  n = floor((b->v->i - ufs->rho[0]->i)/7.);
Missing separate debuginfos, use: debuginfo-install glib2.i386 
glibc.i686 libselinux.i386

> Anyway, do you get any problems with the test case below.

Not exactly the same output as yours in the backtrace (due to 
differences between i386 and x86-64?), but no apparent problems:

[andrew@fedora-vm ufs]$ gdb ./a.out
GNU gdb Fedora (6.8-1.fc9)
Copyright (C) 2008 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later 
<http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "i386-redhat-linux-gnu"...
(no debugging symbols found)
(gdb) run
Starting program: /home/andrew/ufs/a.out
(no debugging symbols found)
(no debugging symbols found)
(no debugging symbols found)
(no debugging symbols found)
[Thread debugging using libthread_db enabled]
[New Thread 0xb80e56c0 (LWP 16859)]
[New Thread 0xb80e4b90 (LWP 16862)]

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0xb80e4b90 (LWP 16862)]
0x0011147b in fn (p=0x0) at thread-so.c:8
8	   cp[0] = 'a';
Missing separate debuginfos, use: debuginfo-install glib2.i386 
glibc.i686 libselinux.i386
(gdb) info thread
* 2 Thread 0xb80e4b90 (LWP 16862)  0x0011147b in fn (p=0x0) at thread-so.c:8
   1 Thread 0xb80e56c0 (LWP 16859)  0x00110416 in __kernel_vsyscall ()
(gdb) bt
#0  0x0011147b in fn (p=0x0) at thread-so.c:8
#1  0x008bc32f in start_thread () from /lib/libpthread.so.0
#2  0x007d927e in clone () from /lib/libc.so.6
(gdb) thread 1
[Switching to thread 1 (Thread 0xb80e56c0 (LWP 16859))]#0  0x00110416 in 
__kernel_vsyscall ()
(gdb) bt
#0  0x00110416 in __kernel_vsyscall ()
#1  0x008c2916 in __lll_unlock_wake_private () from /lib/libpthread.so.0
#2  0x008bc7d4 in _L_unlock_2792 () from /lib/libpthread.so.0
#3  0x008bbbd6 in pthread_create@@GLIBC_2.1 () from /lib/libpthread.so.0
#4  0x008bc0f8 in pthread_create@GLIBC_2.0 () from /lib/libpthread.so.0
#5  0x001114ba in foo () at thread-so.c:15
#6  0x08048497 in main ()

> What does "info threads" at crash point say?

Nothing seemingly useful:

(gdb) info threads
* 1 Thread 0xb7f7c710 (LWP 16869)  0x00137d17 in symmetry_gasdynamics (
     f=0xbff90c30, b=0x9683570) at boundary_gasdynamics.c:282


It seems to me that there is some difference in the linking of shared 
libraries between Fedora 9 and CentOS 5.  With the same exact code and 
the same exact configure, make sequence, CentOS 5 ldd shows libpthread 
(without explicitly specifying it, i.e. LIBS=-pthread), while on Fedora, 
there is no explicit link.  Thus, gdb doesn't handle it on Fedora 9.  I 
wonder what is different? And, yet, it handles your test case as well...

Even if we don't come to the final answer, at least I can work around 
the problem--thanks!

Regards,

Andrew

Re: "Cannot find new threads" on Fedora 9, but not on CentOS 5 (?)

[Paul Pluzhnikov]

On Wed, Oct 29, 2008 at 6:11 PM, Andrew Lofthouse <loftyhauser@gmail.com> wrote:

> It seems to me that there is some difference in the linking of shared
> libraries between Fedora 9 and CentOS 5.  With the same exact code and the
> same exact configure, make sequence, CentOS 5 ldd shows libpthread (without
> explicitly specifying it, i.e. LIBS=-pthread), while on Fedora, there is no
> explicit link.  Thus, gdb doesn't handle it on Fedora 9.  I wonder what is
> different?

You can tell: for each library in ldd output on Fedora 9, namely:

       /usr/local/lib/libufs2D-0.9.so.2
       /usr/local/lib/libgts-0.7.so.5
       /lib/libgmodule-2.0.so.0
       /lib/libdl.so.2
       /lib/libglib-2.0.so.0
       /lib/libselinux.so.1

do 'ldd /path/to/lib' on CentOS.
At least one of them (I expect) will show libpthread.so.0 dependency,
which implies that they were built differently.

> And, yet, it handles your test case as well...

Since we can't replicate "bad GDB behavior" on a test case, I am
afraid the only way forward is for you to debug gdb with itself.

This isn't as bad as it sounds, but probably would require some
effort from you.

Something like this:

  gdb -ex 'set prompt (top) ' --args gdb /usr/local/bin/ufs2oogl2D

  (top) rbreak throw_
  # should set a couple of breakpoints

  (top) run
  (gdb) set verbose on
  (gdb) run -T "rho" < UAM10_002_UFS-005000.sim > test.plt
  # expect one of the throw_ breakpoints to fire and you should see
  # "(top)" prompt.

  (top) where


-- 
Paul Pluzhnikov

Re: "Cannot find new threads" on Fedora 9, but not on CentOS 5 (?)

[Andrew Lofthouse]

Paul Pluzhnikov wrote:
  > You can tell: for each library in ldd output on Fedora 9, namely:
> 
>        /usr/local/lib/libufs2D-0.9.so.2
>        /usr/local/lib/libgts-0.7.so.5
>        /lib/libgmodule-2.0.so.0
>        /lib/libdl.so.2
>        /lib/libglib-2.0.so.0
>        /lib/libselinux.so.1
> 
> do 'ldd /path/to/lib' on CentOS.
> At least one of them (I expect) will show libpthread.so.0 dependency,
> which implies that they were built differently.

Yes, /lib/libglib-2.0.so.0 depends on libpthread.so.0
> 
>> And, yet, it handles your test case as well...
> 
> Since we can't replicate "bad GDB behavior" on a test case, I am
> afraid the only way forward is for you to debug gdb with itself.

Last night I tried to follow your example and try to debug gdb, but to 
do so, I would have to replicate my earlier problem, but I was unable to 
do so (meaning, gdb did not have any problems following the new threads, 
or finding libpthread symbols).  I thought this was because I had 
explicitly linked with -pthread previously.  I rebooted, rebuilt, and 
was still unable to replicate the problem.  I re-installed Fedora 9 from 
scratch, and I'm still unable to replicate it.  I have no idea what changed.

Ubuntu 8.04, however, still gives a "cannot find new threads: generic 
error" when not explicitly linking to libpthread. So, the following 
comes from Ubuntu.  Unfortunately, I cannot find any debug symbols for 
gdb for Ubuntu (Fedora seems to have them).  For what it's worth:

==> andrew@ubuntu-vm ~/codes/ufs Fri Oct 31 22:17:01
532 $ gdb -ex 'set prompt (top) ' --args gdb /usr/local/bin/ufs2oogl2D
GNU gdb 6.8-debian
Copyright (C) 2008 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later 
<http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "i486-linux-gnu"...
(no debugging symbols found)
(top) rbreak throw_
Breakpoint 1 at 0x8132024
<function, no debug info> throw_exception;
Breakpoint 2 at 0x8132136
<function, no debug info> throw_error;
Breakpoint 3 at 0x8132158
<function, no debug info> throw_vfatal;
Breakpoint 4 at 0x8132176
<function, no debug info> throw_verror;
Breakpoint 5 at 0x8132196
<function, no debug info> deprecated_throw_reason;
(top) run
Starting program: /usr/bin/gdb /usr/local/bin/ufs2oogl2D
(no debugging symbols found)
(no debugging symbols found)
(no debugging symbols found)
(no debugging symbols found)
(no debugging symbols found)
[Thread debugging using libthread_db enabled]
[New Thread 0xb7d396b0 (LWP 31952)]
GNU gdb 6.8-debian
Copyright (C) 2008 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later 
<http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "i486-linux-gnu"...
(gdb) set verbose on
(gdb) run -T "rho" < UAM10_002_UFS-005000.sim > test.plt
Starting program: /usr/local/bin/ufs2oogl2D -T "rho" < 
UAM10_002_UFS-005000.sim > test.plt
Reading symbols from /lib/ld-linux.so.2...Reading symbols from 
/usr/lib/debug/lib/ld-2.7.so...done.
done.
Reading symbols from system-supplied DSO at 0xb7f0e000...done.
Reading symbols from /usr/local/lib/libufs2D-0.9.so.2...done.
Reading symbols from /usr/local/lib/libgts-0.7.so.5...done.
Reading symbols from /usr/lib/libgmodule-2.0.so.0...Reading symbols from 
/usr/lib/debug/usr/lib/libgmodule-2.0.so.0.1600.6...done.
done.
Reading symbols from /usr/lib/debug/libdl.so.2...done.
Reading symbols from /usr/lib/libglib-2.0.so.0...Reading symbols from 
/usr/lib/debug/usr/lib/libglib-2.0.so.0.1600.6...done.
done.
Reading symbols from /usr/lib/debug/libm.so.6...done.
Reading symbols from /usr/lib/debug/libc.so.6...done.
Reading symbols from /usr/lib/libpcre.so.3...Reading symbols from 
/usr/lib/debug/usr/lib/libpcre.so.3.12.1...done.
done.

Detaching after fork from child process 31962.
Reading symbols from /tmp/gfs2O2C03...done.
Reading symbols from /usr/lib/libgthread-2.0.so.0...Reading symbols from 
/usr/lib/debug/usr/lib/libgthread-2.0.so.0.1600.6...done.
done.
Reading symbols from /usr/lib/debug/librt.so.1...done.
Reading symbols from /usr/lib/debug/libpthread.so.0...done.
[Thread debugging using libthread_db enabled]
[Switching to Thread 0xb7d396b0 (LWP 31952)]

Breakpoint 4, 0x08132176 in throw_verror ()
(top) where
#0  0x08132176 in throw_verror ()
#1  0x0808f403 in error ()
#2  0x080a2580 in ?? ()
#3  0x080a3d4e in check_for_thread_db ()
#4  0x0808782f in ?? ()
#5  0x08087987 in observer_notify_new_objfile ()
#6  0x08119aaa in ?? ()
#7  0x0809aad0 in ?? ()
#8  0x08132393 in catch_errors ()
#9  0x0809a69c in solib_read_symbols ()
#10 0x0809b062 in solib_add ()
#11 0x081284eb in handle_inferior_event ()
#12 0x08129f77 in wait_for_inferior ()
#13 0x0812a147 in proceed ()
#14 0x08124dc3 in ?? ()
#15 0x0808b531 in execute_command ()
#16 0x08135dff in ?? ()
#17 0x08136b53 in ?? ()
#18 0xb7f27962 in rl_callback_read_char () from /lib/libreadline.so.5
#19 0x08135fcb in ?? ()
#20 0x0813594e in ?? ()
---Type <return> to continue, or q <return> to quit---
#21 0x08134d90 in ?? ()
#22 0x08135630 in gdb_do_one_event ()
#23 0x08132393 in catch_errors ()
#24 0x080d63c4 in ?? ()
#25 0x0813299f in current_interp_command_loop ()
#26 0x0808330b in ?? ()
#27 0x08132393 in catch_errors ()
#28 0x08083c0c in ?? ()
#29 0x08132393 in catch_errors ()
#30 0x080832f1 in gdb_main ()
#31 0x080832b5 in main ()
(top)


If I have some more time, I'll see what else I can do.  But, I do at 
least have a work-around...

Regards,

Andrew

Re: "Cannot find new threads" on Fedora 9, but not on CentOS 5 (?)

[Paul Pluzhnikov]

On Fri, Oct 31, 2008 at 7:29 PM, Andrew Lofthouse <loftyhauser@gmail.com> wrote:

> Last night I tried to follow your example and try to debug gdb, but to do
> so, I would have to replicate my earlier problem, but I was unable to do so
> (meaning, gdb did not have any problems following the new threads, or
> finding libpthread symbols).  I thought this was because I had explicitly
> linked with -pthread previously.  I rebooted, rebuilt, and was still unable
> to replicate the problem.  I re-installed Fedora 9 from scratch, and I'm
> still unable to replicate it.  I have no idea what changed.

Don't you just hate that?

One thing that may have changed is prelink addresses --
IIRC, Fedora runs prelink at 2 week intervals.

You may want to try:

  /usr/sbin/prelink -a
  gdb ...

several times, and see if one particular randomized layout
triggers the bug again.

> Ubuntu 8.04, however, still gives a "cannot find new threads: generic error"
> when not explicitly linking to libpthread.

Are you running a 64-bit kernel perchance?

There is a known kernel bug related to ptrace()ing 32-bit processes,
and it sounds from your trace that that might be what you are hitting.

> So, the following comes from
> Ubuntu.  Unfortunately, I cannot find any debug symbols for gdb for Ubuntu
> (Fedora seems to have them).

It's worse: your GDB has been stripped :(

> For what it's worth:
>
> ==> andrew@ubuntu-vm ~/codes/ufs Fri Oct 31 22:17:01
> 532 $ gdb -ex 'set prompt (top) ' --args gdb /usr/local/bin/ufs2oogl2D
> GNU gdb 6.8-debian
> Copyright (C) 2008 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later
> <http://gnu.org/licenses/gpl.html>
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
> and "show warranty" for details.
> This GDB was configured as "i486-linux-gnu"...
> (no debugging symbols found)
> (top) rbreak throw_
> Breakpoint 1 at 0x8132024
> <function, no debug info> throw_exception;
> Breakpoint 2 at 0x8132136
> <function, no debug info> throw_error;
> Breakpoint 3 at 0x8132158
> <function, no debug info> throw_vfatal;
> Breakpoint 4 at 0x8132176
> <function, no debug info> throw_verror;
> Breakpoint 5 at 0x8132196
> <function, no debug info> deprecated_throw_reason;
> (top) run
> Starting program: /usr/bin/gdb /usr/local/bin/ufs2oogl2D
> (no debugging symbols found)
> (no debugging symbols found)
> (no debugging symbols found)
> (no debugging symbols found)
> (no debugging symbols found)
> [Thread debugging using libthread_db enabled]
> [New Thread 0xb7d396b0 (LWP 31952)]
> GNU gdb 6.8-debian
> Copyright (C) 2008 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later
> <http://gnu.org/licenses/gpl.html>
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
> and "show warranty" for details.
> This GDB was configured as "i486-linux-gnu"...
> (gdb) set verbose on
> (gdb) run -T "rho" < UAM10_002_UFS-005000.sim > test.plt
> Starting program: /usr/local/bin/ufs2oogl2D -T "rho" <
> UAM10_002_UFS-005000.sim > test.plt
> Reading symbols from /lib/ld-linux.so.2...Reading symbols from
> /usr/lib/debug/lib/ld-2.7.so...done.
> done.
> Reading symbols from system-supplied DSO at 0xb7f0e000...done.
> Reading symbols from /usr/local/lib/libufs2D-0.9.so.2...done.
> Reading symbols from /usr/local/lib/libgts-0.7.so.5...done.
> Reading symbols from /usr/lib/libgmodule-2.0.so.0...Reading symbols from
> /usr/lib/debug/usr/lib/libgmodule-2.0.so.0.1600.6...done.
> done.
> Reading symbols from /usr/lib/debug/libdl.so.2...done.
> Reading symbols from /usr/lib/libglib-2.0.so.0...Reading symbols from
> /usr/lib/debug/usr/lib/libglib-2.0.so.0.1600.6...done.
> done.
> Reading symbols from /usr/lib/debug/libm.so.6...done.
> Reading symbols from /usr/lib/debug/libc.so.6...done.
> Reading symbols from /usr/lib/libpcre.so.3...Reading symbols from
> /usr/lib/debug/usr/lib/libpcre.so.3.12.1...done.
> done.
>
> Detaching after fork from child process 31962.
> Reading symbols from /tmp/gfs2O2C03...done.
> Reading symbols from /usr/lib/libgthread-2.0.so.0...Reading symbols from
> /usr/lib/debug/usr/lib/libgthread-2.0.so.0.1600.6...done.
> done.
> Reading symbols from /usr/lib/debug/librt.so.1...done.
> Reading symbols from /usr/lib/debug/libpthread.so.0...done.
> [Thread debugging using libthread_db enabled]
> [Switching to Thread 0xb7d396b0 (LWP 31952)]
>
> Breakpoint 4, 0x08132176 in throw_verror ()
> (top) where
> #0  0x08132176 in throw_verror ()
> #1  0x0808f403 in error ()
> #2  0x080a2580 in ?? ()
> #3  0x080a3d4e in check_for_thread_db ()
> #4  0x0808782f in ?? ()
> #5  0x08087987 in observer_notify_new_objfile ()
> #6  0x08119aaa in ?? ()
> #7  0x0809aad0 in ?? ()
> #8  0x08132393 in catch_errors ()
> #9  0x0809a69c in solib_read_symbols ()
> #10 0x0809b062 in solib_add ()
> #11 0x081284eb in handle_inferior_event ()
> #12 0x08129f77 in wait_for_inferior ()
> #13 0x0812a147 in proceed ()
> #14 0x08124dc3 in ?? ()
> #15 0x0808b531 in execute_command ()
> #16 0x08135dff in ?? ()
> #17 0x08136b53 in ?? ()
> #18 0xb7f27962 in rl_callback_read_char () from /lib/libreadline.so.5
> #19 0x08135fcb in ?? ()
> #20 0x0813594e in ?? ()
> ---Type <return> to continue, or q <return> to quit---
> #21 0x08134d90 in ?? ()
> #22 0x08135630 in gdb_do_one_event ()
> #23 0x08132393 in catch_errors ()
> #24 0x080d63c4 in ?? ()
> #25 0x0813299f in current_interp_command_loop ()
> #26 0x0808330b in ?? ()
> #27 0x08132393 in catch_errors ()
> #28 0x08083c0c in ?? ()
> #29 0x08132393 in catch_errors ()
> #30 0x080832f1 in gdb_main ()
> #31 0x080832b5 in main ()
> (top)
>
>
> If I have some more time, I'll see what else I can do.

You may want to build GDB from sources (either 6.8, or better
yet CVS head) on Ubuntu. Then we'll at least have a complete
stack trace.

>  But, I do at least have a work-around...

Yes, but we still don't know if there is a GDB bug hiding underneath ...

Cheers,
-- 
Paul Pluzhnikov

Re: "Cannot find new threads" on Fedora 9, but not on CentOS 5 (?)

[Andrew Lofthouse]

Paul Pluzhnikov wrote:
> On Fri, Oct 31, 2008 at 7:29 PM, Andrew Lofthouse <loftyhauser@gmail.com> wrote:

I'm going to be getting a bunch of spam, now, huh?   ^^^^^^^^^^^^^^^^^^^
(Since the messages are archived...)

>> Last night I tried to follow your example and try to debug gdb, but to do
>> so, I would have to replicate my earlier problem, but I was unable to do so
>> (meaning, gdb did not have any problems following the new threads, or
>> finding libpthread symbols).  I thought this was because I had explicitly
>> linked with -pthread previously.  I rebooted, rebuilt, and was still unable
>> to replicate the problem.  I re-installed Fedora 9 from scratch, and I'm
>> still unable to replicate it.  I have no idea what changed.
> 
> Don't you just hate that?
> 
> One thing that may have changed is prelink addresses --
> IIRC, Fedora runs prelink at 2 week intervals.

Okay, it must have been prelink.  I did a clean install, deleted the 
prelink cronjob (not that it had time to run) and I can reproduce the 
behavior again:

[andrew@fedora-vm ufs]$ gdb /usr/local/bin/ufs2oogl2D
GNU gdb Fedora (6.8-1.fc9)
Copyright (C) 2008 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later 
<http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "i386-redhat-linux-gnu"...
(gdb) run -T "rho" < UAM10_002_UFS-005000.sim > test.plt
Starting program: /usr/local/bin/ufs2oogl2D -T "rho" < 
UAM10_002_UFS-005000.sim > test.plt

Detaching after fork from child process 2444.
[Thread debugging using libthread_db enabled]
Error while reading shared library symbols:
Cannot find new threads: generic error
Cannot find new threads: generic error
(gdb) quit
The program is running.  Exit anyway? (y or n) y

After installing the debug symbols for gdb, here is what I get:

[andrew@fedora-vm ufs]$ gdb -ex 'set prompt (top) ' --args gdb 
/usr/local/bin/ufs2oogl2D
GNU gdb Fedora (6.8-1.fc9)
Copyright (C) 2008 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later 
<http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "i386-redhat-linux-gnu"...
(top) rbreak throw_
Breakpoint 1 at 0x812fb56: file ../../gdb/exceptions.c, line 241.
void deprecated_throw_reason(enum return_reason);
Breakpoint 2 at 0x812f6f6: file ../../gdb/exceptions.c, line 413.
void throw_error(enum errors, const char *, ...);
Breakpoint 3 at 0x812f5e7: file ../../gdb/exceptions.c, line 220.
void throw_exception(struct gdb_exception);
Breakpoint 4 at 0x812f736: file ../../gdb/exceptions.c, line 399.
void throw_verror(enum errors, const char *, va_list);
Breakpoint 5 at 0x812f718: file ../../gdb/exceptions.c, line 405.
void throw_vfatal(const char *, va_list);
Breakpoint 6 at 0x812f6ab: file ../../gdb/exceptions.c, line 383.
static void throw_it(enum return_reason, enum errors, const char *, 
va_list);
(top) run
Starting program: /usr/bin/gdb /usr/local/bin/ufs2oogl2D
[Thread debugging using libthread_db enabled]
[New Thread 0xb7f21710 (LWP 2457)]
GNU gdb Fedora (6.8-1.fc9)
Copyright (C) 2008 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later 
<http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "i386-redhat-linux-gnu"...
(gdb) set verbose on
(gdb) run -T "rho" < UAM10_002_UFS-005000.sim > test.plt
Starting program: /usr/local/bin/ufs2oogl2D -T "rho" < 
UAM10_002_UFS-005000.sim > test.plt
Detaching after fork from child process 2461.
Detaching after fork from child process 2462.
Reading symbols from /lib/ld-linux.so.2...Reading symbols from 
/usr/lib/debug/lib/ld-2.8.so.debug...done.
done.
Reading symbols from system-supplied DSO at 0x12e000...done.
Reading in symbols for rtld.c...done.
Reading in symbols for dl-debug.c...done.
Reading symbols from /usr/local/lib/libufs2D-0.9.so.2...done.
Reading symbols from /usr/local/lib/libgts-0.7.so.5...done.
Reading symbols from /lib/libgmodule-2.0.so.0...Reading symbols from 
/usr/lib/debug/lib/libgmodule-2.0.so.0.1600.3.debug...done.
done.
Reading symbols from /lib/libdl.so.2...Reading symbols from 
/usr/lib/debug/lib/libdl-2.8.so.debug...done.
done.
Reading symbols from /lib/libglib-2.0.so.0...Reading symbols from 
/usr/lib/debug/lib/libglib-2.0.so.0.1600.3.debug...done.
done.
Reading symbols from /lib/libm.so.6...Reading symbols from 
/usr/lib/debug/lib/libm-2.8.so.debug...done.
done.
Reading symbols from /lib/libc.so.6...Reading symbols from 
/usr/lib/debug/lib/libc-2.8.so.debug...done.
done.
Reading symbols from /lib/libselinux.so.1...Reading symbols from 
/usr/lib/debug/lib/libselinux.so.1.debug...done.
done.

Detaching after fork from child process 2465.
Reading symbols from /tmp/gfskZMmwG...done.
Reading symbols from /lib/libgthread-2.0.so.0...Reading symbols from 
/usr/lib/debug/lib/libgthread-2.0.so.0.1600.3.debug...done.
done.
Reading symbols from /lib/librt.so.1...Reading symbols from 
/usr/lib/debug/lib/librt-2.8.so.debug...done.
done.
Reading symbols from /lib/libpthread.so.0...Reading symbols from 
/usr/lib/debug/lib/libpthread-2.8.so.debug...done.
[Thread debugging using libthread_db enabled]
[Switching to Thread 0xb7f21710 (LWP 2457)]

Breakpoint 4, throw_verror (error=GENERIC_ERROR,
     fmt=0x8271b5c "Cannot find new threads: %s", ap=0xbf935ce4 
"�\030'\b\005")
     at ../../gdb/exceptions.c:399
399	  throw_it (RETURN_ERROR, error, fmt, ap);
Missing separate debuginfos, use: debuginfo-install bzip2.i386 
elfutils.i386 expat.i386 gcc.i386 ncurses.i386 nspr.i386 nss.i386 
popt.i386 readline.i386 rpm.i386 sqlite.i386 zlib.i386
(top) where
#0  throw_verror (error=GENERIC_ERROR,
     fmt=0x8271b5c "Cannot find new threads: %s", ap=0xbf935ce4 
"�\030'\b\005")
     at ../../gdb/exceptions.c:399
#1  0x080882c0 in error (string=0x8271b5c "Cannot find new threads: %s")
     at ../../gdb/utils.c:635
#2  0x0809c1df in thread_db_find_new_threads ()
     at ../../gdb/linux-thread-db.c:975
#3  0x0809daad in check_for_thread_db () at ../../gdb/linux-thread-db.c:650
#4  0x080800d7 in observer_notify_new_objfile (objfile=0xa34cb90)
     at ../../gdb/observer.c:166
#5  0x081162f7 in symbol_file_add_with_addrs_or_offsets (abfd=0xa25a968,
     from_tty=0, addrs=0xa23e250, offsets=0x0, num_offsets=0, mainline=0,
     flags=<value optimized out>) at ../../gdb/symfile.c:1179
#6  0x08116548 in symbol_file_add_with_addrs_or_offsets (
     abfd=<value optimized out>, from_tty=0, addrs=0xa060d60, offsets=0x0,
     num_offsets=0, mainline=0, flags=<value optimized out>)
     at ../../gdb/symfile.c:1127
#7  0x08093ed2 in symbol_add_stub (arg=0x9ecac88) at ../../gdb/solib.c:447
#8  0x0812f8eb in catch_errors (func=0x8093de0 <symbol_add_stub>,
     func_args=0x9ecac88,
     errstring=0x826fa60 "Error while reading shared library symbols:\n",
     mask=6) at ../../gdb/exceptions.c:513
#9  0x08093aa2 in solib_read_symbols (so=0x9ecac88, from_tty=0)
---Type <return> to continue, or q <return> to quit---
     at ../../gdb/solib.c:474
#10 0x080943af in solib_add (pattern=0x0, from_tty=0, target=0x83741c0,
     readsyms=1) at ../../gdb/solib.c:708
#11 0x0812604d in handle_inferior_event (ecs=0xbf936034)
     at ../../gdb/infrun.c:2276
#12 0x08127a17 in wait_for_inferior (treat_exec_as_sigtrap=0)
     at ../../gdb/infrun.c:1052
#13 0x08127d1d in proceed (addr=4294967295, siggnal=TARGET_SIGNAL_0, step=0)
     at ../../gdb/infrun.c:854
#14 0x08121fe3 in run_command_1 (
     args=0x9a915dc "-T \"rho\" < UAM10_002_UFS-005000.sim > test.plt",
     from_tty=1, tbreak_at_main=<value optimized out>) at 
../../gdb/infcmd.c:563
#15 0x08084141 in execute_command (p=0x9a91609 "t", from_tty=1)
     at ../../gdb/top.c:449
#16 0x0813357f in command_handler (command=0x9a915d8 "")
     at ../../gdb/event-top.c:523
#17 0x081344c2 in command_line_handler (
     rl=0x9acd388 "run -T \"rho\" < UAM10_002_UFS-005000.sim > test.plt ")
     at ../../gdb/event-top.c:809
#18 0x00150507 in rl_callback_read_char () from /lib/libreadline.so.5
#19 0x0813386b in rl_callback_read_char_wrapper (client_data=0x0)
     at ../../gdb/event-top.c:178
#20 0x08133184 in handle_file_event (event_file_desc=0)
---Type <return> to continue, or q <return> to quit---
     at ../../gdb/event-loop.c:728
#21 0x08132572 in process_event () at ../../gdb/event-loop.c:341
#22 0x08132e50 in gdb_do_one_event (data=0x0) at ../../gdb/event-loop.c:378
#23 0x0812f8eb in catch_errors (func=0x8132c80 <gdb_do_one_event>,
     func_args=0x0, errstring=0x8274428 "", mask=6)
     at ../../gdb/exceptions.c:513
#24 0x080d352a in tui_command_loop (data=0x0) at 
../../gdb/tui/tui-interp.c:156
#25 0x0812ffb4 in current_interp_command_loop () at ../../gdb/interps.c:276
#26 0x0807b8db in captured_command_loop (data=0x0) at ../../gdb/main.c:99
#27 0x0812f8eb in catch_errors (func=0x807b8d0 <captured_command_loop>,
     func_args=0x0, errstring=0x8274428 "", mask=6)
     at ../../gdb/exceptions.c:513
#28 0x0807c5ac in captured_main (data=0xbf936594) at ../../gdb/main.c:884
#29 0x0812f8eb in catch_errors (func=0x807b910 <captured_main>,
     func_args=0xbf936594, errstring=0x8274428 "", mask=6)
     at ../../gdb/exceptions.c:513
#30 0x0807b8c1 in gdb_main (args=0xbf936594) at ../../gdb/main.c:893
#31 0x0807b885 in main (argc=1852727619, argv=0x6620746f) at 
../../gdb/gdb.c:33
(top) quit
The program is running.  Exit anyway? (y or n) y

> Yes, but we still don't know if there is a GDB bug hiding underneath ...

Can you tell now? :)

Regards,

Andrew L.

Re: "Cannot find new threads" on Fedora 9, but not on CentOS 5 (?)

[Paul Pluzhnikov]

On Sat, Nov 1, 2008 at 8:28 AM, Andrew Lofthouse wrote:

> I'm going to be getting a bunch of spam, now, huh?

This wasn't the first message so quoted.

I find gmail spam filtering to be extremely effective:
I only get about 1 spam/month through.

>> One thing that may have changed is prelink addresses --
>> IIRC, Fedora runs prelink at 2 week intervals.
>
> Okay, it must have been prelink.

Indeed. I did '/usr/sbin/prelink -ua', and now I can reproduce
this on the thread-so test case (with 32-bit kernel).

Reproduces with current GDB CVS head...

The problem is a (now almost 3 year old) bug in glibc:
http://sources.redhat.com/bugzilla/show_bug.cgi?id=2328
for which a patch has been available for 2 years, but came
from the "wrong" person :-(

In this case, GDB gets notified about libpthread load
before libpthread has been relocated:

GNU gdb Fedora (6.8-1.fc9)
Copyright (C) 2008 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "i386-redhat-linux-gnu"...
(gdb) r
Starting program: /tmp/a.out
[Thread debugging using libthread_db enabled]
Error while reading shared library symbols:
Cannot find new threads: generic error
Cannot find new threads: generic error
(gdb) bt
#0  0x0011f1a1 in _dl_debug_state () at dl-debug.c:76
#1  0x00122a2b in dl_open_worker (a=0xbfc4a9ec) at dl-open.c:366
#2  0x0011e4f6 in _dl_catch_error (objname=0xbfc4aa14,
errstring=0xbfc4aa10, mallocedp=0xbfc4aa1b,
    operate=0x122820 <dl_open_worker>, args=0xbfc4a9ec) at dl-error.c:178
#3  0x001222f2 in _dl_open (file=0x8048574 "./thread-so.so",
mode=<value optimized out>, caller_dlopen=0x8048479, nsid=-2, argc=1,
    argv=0xbfc4ac64, env=0xbfc4ac6c) at dl-open.c:596
#4  0x0012fc2c in dlopen_doit (a=0xbfc4ab80) at dlopen.c:66
#5  0x0011e4f6 in _dl_catch_error (objname=0x133058,
errstring=0x13305c, mallocedp=0x133054, operate=0x12fb90
<dlopen_doit>,
    args=0xbfc4ab80) at dl-error.c:178
#6  0x0013005c in _dlerror_run (operate=0x12fb90 <dlopen_doit>,
args=<value optimized out>) at dlerror.c:164
#7  0x0012fb61 in __dlopen (file=0x8048574 "./thread-so.so", mode=1)
at dlopen.c:87
#8  0x08048479 in main () at thread-so-main.c:6
(gdb) p &stack_used
$1 = (list_t *) 0x2ba110
(gdb) x/x $1
0x2ba110 <stack_used>:	0x00016110
(gdb) shell nm /lib/libpthread.so.0 | grep stack_used
00016110 d stack_used   ### NOTE: stack_used points to
                        ### non-relocated (and unreadable) self!

(gdb) fr 8
#8  0x08048479 in main () at thread-so-main.c:6
6	 void *h = dlopen("./thread-so.so", RTLD_LAZY);
(gdb) b 7
Breakpoint 1 at 0x804847c: file thread-so-main.c, line 7.
(gdb) c
Continuing.
[New Thread 0xb803e6c0 (LWP 32044)]
[Switching to Thread 0xb803e6c0 (LWP 32044)]

Breakpoint 1, main () at thread-so-main.c:7
7	 void (*fn)(void) = dlsym(h, "foo");
(gdb) x/x $1
0x2ba110 <stack_used>:	0x002ba110   ### NOTE: stack_used now points to
relocated self.

The 'stack_used' a head of thread list, and libthread_db can not
work correctly if it observes the wrong (unreadable) value.

I don't know what it takes to make Uli apply the patch to fix this :(
Perhaps Tom can have a friendly chat with him?


Cheers,
-- 
Paul Pluzhnikov

Re: "Cannot find new threads" on Fedora 9, but not on CentOS 5 (?)

[Andrew Lofthouse]

Paul Pluzhnikov wrote:
>> I'm going to be getting a bunch of spam, now, huh?
> 
> This wasn't the first message so quoted.

No, it was the first I noticed though.

> I find gmail spam filtering to be extremely effective:
> I only get about 1 spam/month through.

Thanks goodness for that--I haven't noticed any increase in my spam 
count since I posted theses messages.  Hopefully it will stay that way.

>>> One thing that may have changed is prelink addresses --
>>> IIRC, Fedora runs prelink at 2 week intervals.
>> Okay, it must have been prelink.
> 
> Indeed. I did '/usr/sbin/prelink -ua', and now I can reproduce
> this on the thread-so test case (with 32-bit kernel).
> 
> Reproduces with current GDB CVS head...
> 
> The problem is a (now almost 3 year old) bug in glibc:
> http://sources.redhat.com/bugzilla/show_bug.cgi?id=2328
> for which a patch has been available for 2 years, but came
> from the "wrong" person :-(

Good.  At least we know what the problem is (and there is a solution). 
In the mean-time, I have a temporary work-around.

Thanks again for the help!

Regards,

Andrew L.

Re: "Cannot find new threads" on Fedora 9, but not on CentOS 5 (?)

[Tom Tromey]

>>>>> "Paul" == Paul Pluzhnikov <ppluzhnikov@google.com> writes:

Paul> Reproduces with current GDB CVS head...

Paul> The problem is a (now almost 3 year old) bug in glibc:
Paul> http://sources.redhat.com/bugzilla/show_bug.cgi?id=2328
Paul> for which a patch has been available for 2 years, but came
Paul> from the "wrong" person :-(

Paul> I don't know what it takes to make Uli apply the patch to fix this :(
Paul> Perhaps Tom can have a friendly chat with him?

I will ask about it and get back to you.

Tom

Re: "Cannot find new threads" on Fedora 9, but not on CentOS 5 (?)

[Tom Tromey]

>>>>> "Tom" == Tom Tromey <tromey@redhat.com> writes:

Paul> The problem is a (now almost 3 year old) bug in glibc:
Paul> http://sources.redhat.com/bugzilla/show_bug.cgi?id=2328
Paul> for which a patch has been available for 2 years, but came
Paul> from the "wrong" person :-(

Paul> I don't know what it takes to make Uli apply the patch to fix this :(
Paul> Perhaps Tom can have a friendly chat with him?

Tom> I will ask about it and get back to you.

Sorry for the long delay on this.

My summary of the discussion is that someone will have to dig into the
problem in depth.  My understanding is that Ulrich does not like that
patch.  But, neither of us knows enough about this part of gdb to say
what it may be doing wrong.

Tom

Re: "Cannot find new threads" on Fedora 9, but not on CentOS 5 (?)

[Daniel Jacobowitz]

On Fri, Dec 12, 2008 at 03:49:37PM -0700, Tom Tromey wrote:
> >>>>> "Tom" == Tom Tromey <tromey@redhat.com> writes:
> 
> Paul> The problem is a (now almost 3 year old) bug in glibc:
> Paul> http://sources.redhat.com/bugzilla/show_bug.cgi?id=2328
> Paul> for which a patch has been available for 2 years, but came
> Paul> from the "wrong" person :-(
> 
> Paul> I don't know what it takes to make Uli apply the patch to fix this :(
> Paul> Perhaps Tom can have a friendly chat with him?
> 
> Tom> I will ask about it and get back to you.
> 
> Sorry for the long delay on this.
> 
> My summary of the discussion is that someone will have to dig into the
> problem in depth.  My understanding is that Ulrich does not like that
> patch.  But, neither of us knows enough about this part of gdb to say
> what it may be doing wrong.

I don't suppose you can be any more specific?  I agree that the
dl-load.c bit is ugly, but at least the other portion seems clearly
correct and I don't understand how the fault token shifted back to
GDB.

-- 
Daniel Jacobowitz
CodeSourcery

Re: "Cannot find new threads" on Fedora 9, but not on CentOS 5 (?)

[Paul Pluzhnikov]

On Fri, Dec 12, 2008 at 2:49 PM, Tom Tromey <tromey@redhat.com> wrote:

> Tom> I will ask about it and get back to you.
>
> My summary of the discussion is that someone will have to dig into the
> problem in depth.  My understanding is that Ulrich does not like that
> patch.

Looking some more, I actually got confused :(

Here is the patch which is "obviously correct" (and I think
largely non-controversial):

  https://bugzilla.redhat.com/attachment.cgi?id=123756

It is attached to
  https://bugzilla.redhat.com/show_bug.cgi?id=179072#c1

> But, neither of us knows enough about this part of gdb to say
> what it may be doing wrong.

-- 
Paul Pluzhnikov

Re: "Cannot find new threads" on Fedora 9, but not on CentOS 5 (?)

[Tom Tromey]

>>>>> "Daniel" == Daniel Jacobowitz <drow@false.org> writes:

Tom> My summary of the discussion is that someone will have to dig into the
Tom> problem in depth.  My understanding is that Ulrich does not like that
Tom> patch.  But, neither of us knows enough about this part of gdb to say
Tom> what it may be doing wrong.

Daniel> I don't suppose you can be any more specific?  I agree that the
Daniel> dl-load.c bit is ugly, but at least the other portion seems clearly
Daniel> correct and I don't understand how the fault token shifted back to
Daniel> GDB.

The above really does convey the gist of our conversation.

I'm somewhat uncomfortable trying to act as an intermediary in a
discussion where I really know nothing about the problem.  I will try
to educate myself a bit so I can come up with good followup questions.

Tom

Re: "Cannot find new threads" on Fedora 9, but not on CentOS 5 (?)

[Michael Snyder]

Tom Tromey wrote:
>>>>>> "Daniel" == Daniel Jacobowitz <drow@false.org> writes:
> 
> Tom> My summary of the discussion is that someone will have to dig into the
> Tom> problem in depth.  My understanding is that Ulrich does not like that
> Tom> patch.  But, neither of us knows enough about this part of gdb to say
> Tom> what it may be doing wrong.
> 
> Daniel> I don't suppose you can be any more specific?  I agree that the
> Daniel> dl-load.c bit is ugly, but at least the other portion seems clearly
> Daniel> correct and I don't understand how the fault token shifted back to
> Daniel> GDB.
> 
> The above really does convey the gist of our conversation.
> 
> I'm somewhat uncomfortable trying to act as an intermediary in a
> discussion where I really know nothing about the problem.  I will try
> to educate myself a bit so I can come up with good followup questions.

Tom, I do appreciate your position...

Could you possibly summarize the problem and the proposed patch?