[patch] Fix for gdb.threads/staticthreads.exp failure on Linux

[patch] Fix for gdb.threads/staticthreads.exp failure on Linux

[Paul Pluzhnikov]

Greetings,

The following tests always fail for us:

FAIL: gdb.threads/staticthreads.exp: running to main in runto
FAIL: gdb.threads/staticthreads.exp: Continue to main's call of sem_post
FAIL: gdb.threads/staticthreads.exp: handle SIG32 helps

This is happening because when loading a statically-linked executable, gdb
detects threads right away, but attempt to enumerate threads at that time
fails, because "thread subsystem" in the inferior has not been initialized
yet. This causes spurious error:

  [Thread debugging using libthread_db enabled]
  find_new_threads_callback: cannot get thread info: generic error

and makes subsequent debugging impossible.

Attached patch silently ignores such errors, and makes staticthreads PASS.

Tested on Linux/x86_64.

Ok to check in?

Thanks,

--
Paul Pluzhnikov

2009-06-05  Paul Pluzhnikov  <ppluzhnikov@google.com>

	* linux-thread-db.c (thread_db_find_new_threads_silently):
	New function.
	(try_thread_db_load_1): Call it.


Index: linux-thread-db.c
===================================================================
RCS file: /cvs/src/src/gdb/linux-thread-db.c,v
retrieving revision 1.60
diff -u -p -u -r1.60 linux-thread-db.c
--- linux-thread-db.c	24 May 2009 21:06:53 -0000	1.60
+++ linux-thread-db.c	5 Jun 2009 22:01:37 -0000
@@ -588,6 +588,25 @@ enable_thread_event_reporting (void)
     }
 }
 
+/* Same as thread_db_find_new_threads_1, but silently ignore errors.  */
+
+static void
+thread_db_find_new_threads_silently (ptid_t ptid)
+{
+  volatile struct gdb_exception except;
+
+  TRY_CATCH (except, RETURN_MASK_ERROR)
+    {
+      thread_db_find_new_threads_1 (ptid);
+    }
+
+  if (except.reason < 0 && info_verbose)
+  {
+    exception_fprintf (gdb_stderr, except,
+                       "Warning: thread_db_find_new_threads_silently: ");
+  }
+}
+
 /* Lookup a library in which given symbol resides.
    Note: this is looking in GDB process, not in the inferior.
    Returns library name, or NULL.  */
@@ -705,7 +724,7 @@ try_thread_db_load_1 (struct thread_db_i
     push_target (&thread_db_ops);
 
   enable_thread_event_reporting ();
-  thread_db_find_new_threads_1 (inferior_ptid);
+  thread_db_find_new_threads_silently (inferior_ptid);
   return 1;
 }
 

Re: [patch] Fix for gdb.threads/staticthreads.exp failure on Linux

[Paul Pluzhnikov]

On Fri, Jun 5, 2009 at 3:22 PM, Paul Pluzhnikov<ppluzhnikov@google.com> wrote:

> Attached patch silently ignores such errors, and makes staticthreads PASS.

Ping?


-- 
Paul Pluzhnikov

Re: [patch] Fix for gdb.threads/staticthreads.exp failure on Linux

[Paul Pluzhnikov]

On Mon, Jun 15, 2009 at 11:39 AM, Paul Pluzhnikov<ppluzhnikov@google.com> wrote:
> On Fri, Jun 5, 2009 at 3:22 PM, Paul Pluzhnikov<ppluzhnikov@google.com> wrote:
>
>> Attached patch silently ignores such errors, and makes staticthreads PASS.
>
> Ping?

Ping, Ping?
-- 
Paul Pluzhnikov

Re: [patch] Fix for gdb.threads/staticthreads.exp failure on Linux

[Michael Snyder]

Paul Pluzhnikov wrote:
> On Mon, Jun 15, 2009 at 11:39 AM, Paul Pluzhnikov<ppluzhnikov@google.com> wrote:
>> On Fri, Jun 5, 2009 at 3:22 PM, Paul Pluzhnikov<ppluzhnikov@google.com> wrote:
>>
>>> Attached patch silently ignores such errors, and makes staticthreads PASS.
>> Ping?
> 
> Ping, Ping?

Paul, I don't see these FAILs on i686, so it's difficult to have an 
opinion.  Have you, or could you, run regression testing in i686?

Re: [patch] Fix for gdb.threads/staticthreads.exp failure on Linux

[Paul Pluzhnikov]

On Mon, Jun 22, 2009 at 11:09 AM, Michael Snyder<msnyder@vmware.com> wrote:

> Have you, or could you, run regression testing in i686?

This is glibc-version related, rather than x86_64 vs. i686:

Using glibc-2.7: "runtest gdb.threads/staticthreads.exp" gives:

  # of expected passes            8
  # of known failures             1

in both i686 and x86_64 modes.

But using glibc-2.3.6 (which is my target):

  FAIL: gdb.threads/staticthreads.exp: running to main in runto
  FAIL: gdb.threads/staticthreads.exp: Continue to main's call of sem_post
  FAIL: gdb.threads/staticthreads.exp: handle SIG32 helps

		  === gdb Summary ===

  # of expected passes            6
  # of unexpected failures        3
  # of known failures             1

again in both i686 and x86_64 modes.

Thanks,
-- 
Paul Pluzhnikov

Re: [patch] Fix for gdb.threads/staticthreads.exp failure on Linux

[Paul Pluzhnikov]

On Mon, Jun 22, 2009 at 1:21 PM, Paul Pluzhnikov<ppluzhnikov@google.com> wrote:
> On Mon, Jun 22, 2009 at 11:09 AM, Michael Snyder<msnyder@vmware.com> wrote:
>
>> Have you, or could you, run regression testing in i686?
>
> This is glibc-version related, rather than x86_64 vs. i686:

Ping? Ping? Ping?
-- 
Paul Pluzhnikov

Re: [patch] Fix for gdb.threads/staticthreads.exp failure on Linux

[Paul Pluzhnikov]

On Mon, Jun 29, 2009 at 7:36 AM, Paul Pluzhnikov<ppluzhnikov@google.com> wrote:

> Ping? Ping? Ping?

4x Ping?

-- 
Paul Pluzhnikov

Re: [patch] Fix for gdb.threads/staticthreads.exp failure on Linux

[Doug Evans]

On Mon, Jun 22, 2009 at 1:21 PM, Paul Pluzhnikov<ppluzhnikov@google.com> wrote:
> On Mon, Jun 22, 2009 at 11:09 AM, Michael Snyder<msnyder@vmware.com> wrote:
>
>> Have you, or could you, run regression testing in i686?
>
> This is glibc-version related, rather than x86_64 vs. i686:
>
> Using glibc-2.7: "runtest gdb.threads/staticthreads.exp" gives:
>
>  # of expected passes            8
>  # of known failures             1
>
> in both i686 and x86_64 modes.
>
> But using glibc-2.3.6 (which is my target):
>
>  FAIL: gdb.threads/staticthreads.exp: running to main in runto
>  FAIL: gdb.threads/staticthreads.exp: Continue to main's call of sem_post
>  FAIL: gdb.threads/staticthreads.exp: handle SIG32 helps
>
>                  === gdb Summary ===
>
>  # of expected passes            6
>  # of unexpected failures        3
>  # of known failures             1
>
> again in both i686 and x86_64 modes.
>
> Thanks,
> --
> Paul Pluzhnikov
>

I looked into this a bit.

I think(!) this is the glibc patch that fixes things.

2007-05-16  Roland McGrath  <roland@redhat.com>

        * td_ta_thr_iter.c (iterate_thread_list): Make FAKE_EMPTY bool.
        Use th_unique=0 in fake descriptor before initialization.

Regardless, this is indeed glibc specific, and older glibcs fail while
newer ones pass.
I like the patch.  If thread enumeration fails during
try_thread_db_load_1 we ignore it and leave gdb to try again later.
When we get to this point we've already verified libthread_db is
(mostly) happy.
I kinda wonder, though, if that works then maybe gdb shouldn't do
thread enumeration at all here.
Running the testcase with --target_board=native-gdbserver doesn't have
this problem.  Some research as to what gdbserver is doing would be
illuminating.  "consistency is good": can you research what's
happening in gdbserver that it works there, and see if it's reasonable
to do the same thing in gdb?

[digression: GDB does use glibc's support for determining the glibc
version, but here we want the glibc version of libthread_db so that's
out.  Hmmm, though it looks like the existing use of
gnu_get_libc_version is to get the libthread_db version which can be
different now that we have libthread-db-search-path.  ...  This seems
like a problem, though in practice I suspect it's ok.  [grep for
gnu_get_libc_version in linux-thread-db.c]]

I think a comment needs to be added to your patch, probably to the
call site of thread_db_find_new_threads_silently, that thoroughly
explains what's going on (assuming that's still the best solution).

btw, the reason for the 1 known failure is, I'm guessing, because gdb
was built with the same toolchain that uses the older glibc, but the
test was run using a native toolchain that uses a newer glibc and the
mismatch is sufficient to trigger the failure.  At least that explains
things in my sandbox.  If I rebuild gdb with native gcc, or set
libthread-db-search-path to find the newer glibc, the kfail goes away.

Re: [patch] Fix for gdb.threads/staticthreads.exp failure on Linux

[Doug Evans]

On Wed, Jul 15, 2009 at 6:25 PM, Doug Evans<dje@google.com> wrote:
> Running the testcase with --target_board=native-gdbserver doesn't have
> this problem.  Some research as to what gdbserver is doing would be
> illuminating.  "consistency is good": can you research what's
> happening in gdbserver that it works there, and see if it's reasonable
> to do the same thing in gdb?

I spoke too soon.  gdbserver does have the same problem, I just needed
to look at gdb.log to see it.

Process /usr/local/g3/gnu/sourceware/static-threads/build/obj32/gdb/testsuite/gdb.threads/staticthreads
created; pid = 19691^M
Listening on port 2347^M
target remote localhost:2347^M
Remote debugging using localhost:2347^M
Remote debugging from host 127.0.0.1^M
Cannot get thread handle for LWP 19691: generic error^M
_start () at ../sysdeps/i386/elf/start.S:65^M
65              xorl %ebp, %ebp^M
(gdb) Cannot get thread handle for LWP 19691: generic error^M
continue^M
Continuing.^M
^M
Breakpoint 1, main (argc=<value optimized out>, argv=<value optimized
out>) at ../../../../src/gdb/testsuite/gdb.threads/staticthreads.c:51^M
51        pthread_attr_init (&attr);^M
Current language:  auto; currently c^M
(gdb) PASS: gdb.threads/staticthreads.exp: rerun to main

[The "value optimized out" seems like a bug, but a different bug. :-)]

Re: [patch] Fix for gdb.threads/staticthreads.exp failure on Linux

[Paul Pluzhnikov]

[-- Attachment #1: Type: text/plain, Size: 2276 bytes --]

On Wed, Jul 15, 2009 at 6:25 PM, Doug Evans<dje@google.com> wrote:

> I like the patch. If thread enumeration fails during
> try_thread_db_load_1 we ignore it and leave gdb to try again later.
> When we get to this point we've already verified libthread_db is
> (mostly) happy.
> I kinda wonder, though, if that works then maybe gdb shouldn't do
> thread enumeration at all here.

There is the 'attach' scenario under which (I believe) we *should* do
thread enumeration here.

> I think a comment needs to be added to your patch, probably to the
> call site of thread_db_find_new_threads_silently, that thoroughly
> explains what's going on (assuming that's still the best solution).

Added.

> btw, the reason for the 1 known failure is, I'm guessing, because gdb
> was built with the same toolchain that uses the older glibc, but the
> test was run using a native toolchain that uses a newer glibc and the
> mismatch is sufficient to trigger the failure.  At least that explains
> things in my sandbox.  If I rebuild gdb with native gcc, or set
> libthread-db-search-path to find the newer glibc, the kfail goes away.

When gdb is built with native toolchain (glibc-2.7) and the test is built
with the same, then there is no failure.

When gdb is built with native toolchain, but the test is built against
glibc-2.3.6, I see one KFAIL.

When both gdb and the test are built against glibc-2.3.6, I see this:

FAIL: gdb.threads/staticthreads.exp: running to main in runto
FAIL: gdb.threads/staticthreads.exp: Continue to main's call of sem_post
FAIL: gdb.threads/staticthreads.exp: handle SIG32 helps

# of expected passes            6
# of unexpected failures        3
# of known failures             1

When gdb is built against glibc-2.3.6, but test with glibc-2.7, I see
one KFAIL.

I've retested attached patch under all four combinations above, which
results in:

gdb-glibc-2.7   test-glibc-2.7:   no failures
gdb-glibc-2.7   test-glibc-2.3.6: KFAIL:
gdb.threads/staticthreads.exp: info threads (PRMS: gdb/1328)
gdb-glibc-2.3.6 test-glibc-2.3.6: no failures
gdb-glibc-2.3.6 test-glibc-2.7:   KFAIL:
gdb.threads/staticthreads.exp: info threads (PRMS: gdb/1328)

and checked this in.

Thanks,
-- 
Paul Pluzhnikov

[-- Attachment #2: gdb-staticthreads-20090716.txt --]
[-- Type: text/plain, Size: 1495 bytes --]

Index: linux-thread-db.c
===================================================================
RCS file: /cvs/src/src/gdb/linux-thread-db.c,v
retrieving revision 1.62
diff -u -p -u -r1.62 linux-thread-db.c
--- linux-thread-db.c	2 Jul 2009 17:12:25 -0000	1.62
+++ linux-thread-db.c	16 Jul 2009 19:40:07 -0000
@@ -588,6 +588,25 @@ enable_thread_event_reporting (void)
     }
 }
 
+/* Same as thread_db_find_new_threads_1, but silently ignore errors.  */
+
+static void
+thread_db_find_new_threads_silently (ptid_t ptid)
+{
+  volatile struct gdb_exception except;
+
+  TRY_CATCH (except, RETURN_MASK_ERROR)
+    {
+      thread_db_find_new_threads_1 (ptid);
+    }
+
+  if (except.reason < 0 && info_verbose)
+  {
+    exception_fprintf (gdb_stderr, except,
+                       "Warning: thread_db_find_new_threads_silently: ");
+  }
+}
+
 /* Lookup a library in which given symbol resides.
    Note: this is looking in GDB process, not in the inferior.
    Returns library name, or NULL.  */
@@ -705,7 +724,13 @@ try_thread_db_load_1 (struct thread_db_i
     push_target (&thread_db_ops);
 
   enable_thread_event_reporting ();
-  thread_db_find_new_threads_1 (inferior_ptid);
+
+  /* There appears to be a bug glibc-2.3.6: call to td_thr_get_info fails
+     with TD_ERR for statically linked executables if td_thr_get_info is
+     called before glibc has initialized itself.  Silently ignore such
+     errors.  */
+
+  thread_db_find_new_threads_silently (inferior_ptid);
   return 1;
 }
 

Re: [patch] Fix for gdb.threads/staticthreads.exp failure on Linux

[Paul Pluzhnikov]

On Thu, Jul 16, 2009 at 12:48 PM, Paul Pluzhnikov<ppluzhnikov@google.com> wrote:

> gdb-glibc-2.7   test-glibc-2.3.6: KFAIL:
> gdb-glibc-2.3.6 test-glibc-2.3.6: no failures

Why should "how gdb built" matter here?
Because the test suite doesn't set libthread-db-search-path.

When I set verbose on, for KFAIL I see this:

  (gdb) run
  Starting program: staticthreads
  Trying host libthread_db library: libthread_db.so.1.
  Host libthread_db.so.1 resolved to: /lib/libthread_db.so.1.
  td_ta_new failed: versions of libpthread and libthread_db do not match
  Trying host libthread_db library: libthread_db.so.1.
  Host libthread_db.so.1 resolved to: /lib/libthread_db.so.1.
  td_ta_new failed: versions of libpthread and libthread_db do not match

  Breakpoint 1, main (argc=1, argv=0x7fffffffdaa8) at staticthreads.c:51
  51        pthread_attr_init (&attr);

When I set libthread-db-search-path correctly for the target program,
then I see the same behavior with gdb-glibc-2.7 and gdb-glibc-2.3.6:

# before the patch:

  (gdb) run
  Starting program: staticthreads
  Trying host libthread_db library: /usr/glibc-2.3.6/lib64/libthread_db.so.1.
  [Thread debugging using libthread_db enabled]
  Using host libthread_db library "/usr/glibc-2.3.6/lib64/libthread_db.so.1".
  find_new_threads_callback: cannot get thread info: generic error
  (gdb) c
  Continuing.
  Cannot execute this command while the selected thread is running.

     # no further debugging possible

# after the patch:

  (gdb) run
  Starting program: staticthreads
  Trying host libthread_db library "/usr/glibc-2.3.6/lib64/libthread_db.so.1".
  [Thread debugging using libthread_db enabled]
  Warning: thread_db_find_new_threads_silently:
find_new_threads_callback: cannot get thread info: generic error
  Warning: guessed host libthread_db library
"/usr/glibc-2.3.6/lib64/libthread_db.so.1".
  Using host libthread_db library "/usr/glibc-2.3.6/lib64/libthread_db.so.1".
  [New Thread 0x693860 (LWP 10575)]
  [Switching to Thread 0x693860 (LWP 10575)]

  Breakpoint 1, main (argc=1, argv=0x7fffffffdab8) at
../../../src/gdb/testsuite/gdb.threads/staticthreads.c:51
  51        pthread_attr_init (&attr);

-- 
Paul Pluzhnikov

Re: [patch] Fix for gdb.threads/staticthreads.exp failure on Linux

[Doug Evans]

On Thu, Jul 16, 2009 at 12:48 PM, Paul Pluzhnikov<ppluzhnikov@google.com> wrote:
> On Wed, Jul 15, 2009 at 6:25 PM, Doug Evans<dje@google.com> wrote:
>
>> I like the patch. If thread enumeration fails during
>> try_thread_db_load_1 we ignore it and leave gdb to try again later.
>> When we get to this point we've already verified libthread_db is
>> (mostly) happy.
>> I kinda wonder, though, if that works then maybe gdb shouldn't do
>> thread enumeration at all here.
>
> There is the 'attach' scenario under which (I believe) we *should* do
> thread enumeration here.

Right.  But for "run", that's my question.

>> I think a comment needs to be added to your patch, probably to the
>> call site of thread_db_find_new_threads_silently, that thoroughly
>> explains what's going on (assuming that's still the best solution).
>
> Added.
>
>> btw, the reason for the 1 known failure is, I'm guessing, because gdb
>> was built with the same toolchain that uses the older glibc, but the
>> test was run using a native toolchain that uses a newer glibc and the
>> mismatch is sufficient to trigger the failure.  At least that explains
>> things in my sandbox.  If I rebuild gdb with native gcc, or set
>> libthread-db-search-path to find the newer glibc, the kfail goes away.
>
> When gdb is built with native toolchain (glibc-2.7) and the test is built
> with the same, then there is no failure.
>
> When gdb is built with native toolchain, but the test is built against
> glibc-2.3.6, I see one KFAIL.
>
> When both gdb and the test are built against glibc-2.3.6, I see this:
>
> FAIL: gdb.threads/staticthreads.exp: running to main in runto
> FAIL: gdb.threads/staticthreads.exp: Continue to main's call of sem_post
> FAIL: gdb.threads/staticthreads.exp: handle SIG32 helps
>
> # of expected passes            6
> # of unexpected failures        3
> # of known failures             1
>
> When gdb is built against glibc-2.3.6, but test with glibc-2.7, I see
> one KFAIL.

That's what I see too.

> I've retested attached patch under all four combinations above, which
> results in:
>
> gdb-glibc-2.7   test-glibc-2.7:   no failures
> gdb-glibc-2.7   test-glibc-2.3.6: KFAIL:
> gdb.threads/staticthreads.exp: info threads (PRMS: gdb/1328)
> gdb-glibc-2.3.6 test-glibc-2.3.6: no failures
> gdb-glibc-2.3.6 test-glibc-2.7:   KFAIL:
> gdb.threads/staticthreads.exp: info threads (PRMS: gdb/1328)
>
> and checked this in.

Thanks.
I tweaked the comment a bit.