Problem debugging multi-threaded application

[Arvid Brodin <arvid.brodin@enea.com>]

We have a program, a big beast, which on some platforms cannot be debugged, and I'm trying to figure out why.

The program is supposed to print out some welcome/init messages and then display a prompt. It also starts a bunch of threads (about 200 of them). When run under gdb on some platforms, I only get the [New Thread 0x... (LWP ...)] messages, but no text output. The program seems to run - only I get no text output (or input). 

If I invoke gdb under strace, i.e. 'strace -f -o strace-out.txt gdb <my-program>', and then run my program in gdb, I instead get no messages about created threads, but I do get some printouts from my program before it hangs. The strace log among other things contain messages about "--- SIGTTOU (Stopped (tty output)) @ 0 (0) ---" - I don't know if this is important.

If run under gdbtui - 'gdbtui <my-program>' - as soon as I 'run' the program gdbtui exits to the shell with
'[1]+  Stopped       gdbtui <my-program>'.

If I comment out either the call to create_thread_event_breakpoint() in gdb/linux-thread-db.c:468 or the call to td_thr_event_enable() in gdb/linux-thread-db.c:716 I get the printouts from my program (although I guess it also messes something up - the program hangs shortly after initialization in this case).

The problem exists with both gdb-6.3.0 and gdb-6.7.1.

The problem exists on these platforms: 

---------------
$ uname -r
2.4.21-53.ELsmp

$ /lib/tls/libc.so.6 
GNU C Library stable release version 2.3.2, by Roland McGrath et al.
Copyright (C) 2003 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.
Compiled by GNU CC version 3.2.3 20030502 (Red Hat Linux 3.2.3-58).
Compiled on a Linux 2.4.20 system on 2007-04-19.
Available extensions:
        GNU libio by Per Bothner
        crypt add-on version 2.1 by Michael Glad and others
        NPTL 0.60 by Ulrich Drepper
        RT using linux kernel aio
        The C stubs add-on version 2.1.2.
        BIND-8.2.3-T5B
        NIS(YP)/NIS+ NSS modules 0.19 by Thorsten Kukuk
        Glibc-2.0 compatibility add-on by Cristian Gafton 
Thread-local storage support included.
---------------

and

---------------
$~> uname -r
2.6.5-7.287.3-smp

$~> /lib/tls/libc.so.6 
GNU C Library stable release version 2.3.5 (20050720), by Roland McGrath et al.
Copyright (C) 2005 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.
Configured for i686-suse-linux.
Compiled by GNU CC version 3.3.3 (SuSE Linux).
Compiled on a Linux 2.6.5 system on 2007-10-10.
Available extensions:
        GNU libio by Per Bothner
        crypt add-on version 2.1 by Michael Glad and others
        Native POSIX Threads Library by Ulrich Drepper et al
        GNU Libidn by Simon Josefsson
        NoVersion patch for broken glibc 2.0 binaries
        BIND-8.2.3-T5B
        NIS(YP)/NIS+ NSS modules 0.19 by Thorsten Kukuk
Thread-local storage support included.
---------------

but not on

---------------
$~> uname -r
2.6.16.46-0.14-smp

$~> /lib/libc.so.6 
GNU C Library development release version 2.4 (20070506), by Roland McGrath et al.
Copyright (C) 2006 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.
Configured for i686-suse-linux.
Compiled by GNU CC version 4.1.0 (SUSE Linux).
Compiled on a Linux 2.6.16 system on 2007-05-06.
Available extensions:
        crypt add-on version 2.1 by Michael Glad and others
        GNU Libidn by Simon Josefsson
        GNU libio by Per Bothner
        NIS(YP)/NIS+ NSS modules 0.19 by Thorsten Kukuk
        NoVersion patch for broken glibc 2.0 binaries
        Native POSIX Threads Library by Ulrich Drepper et al
        BIND-8.2.3-T5B
Thread-local storage support included.
---------------

The paths to glibc above is taken from 'ldd <path to libthread_db.so.1 as printed by gdb>, i.e. on the systems that fails /lib/tls/libc.so.1 and /lib/tls/libthread_db.so.1 are used, while on the working system the files are found under /lib.

To recapitulate, the program runs fine on all these platforms when not being debugged, and works under gdb on the "newest" one. On the older platforms the symptoms are text input and output does not work when the program is being debugged, unless gdb is run under strace, when it works "better", at least.

I have no idea if this is a bug in gdb, if we do something bad in our program, if it is a bug in nptl, or ... 

Any ideas?

--
Arvid Brodin
Enea LCC