[patch][rfc] Allow GDB to search for the right libthread_db.so.1

[patch][rfc] Allow GDB to search for the right libthread_db.so.1

[Paul Pluzhnikov]

Greetings,

We have perhaps uncommon setup here, where we have several installed
versions of glibc, and need to debug executables which are compiled
and linked against them (using -rpath).

Currently, GDB will dlopen("libthread_db.so.1", ...), which means that
in order debug "non-standard" binary, one has to set LD_LIBRARY_PATH to
point to correct libthread_db before invoking GDB, or it will refuse to
see threads in the inferior. This is not automatic, and error prone.

Attached patch fixes this, by 

- first looking for libthread_db in the same directory from which
  libpthread.so came, and

- allowing user to set libthread-db-search-path
  at runtime (or maintainer to set LIBTHREAD_DB_SEARCH_PATH at GDB compile
  time) which GDB uses to iterate over available libthread_db's, until
  it finds one which "accepts" the inferior.

We've been running with this patch for a couple of month now; it works
well for us, but I'd like to get it upstream so we don't have to deal with
merge conflicts.

If this looks reasonable, I'll work on documentation next.
There is also a matching change to gdbserver, which I am postponing until
a decision on this patch is made.

Tested on Linux/x86_64 without new failures.

Thanks,

--
Paul Pluzhnikov

2009-04-06  Paul Pluzhnikov  <ppluzhnikov@google.com>

	    * gdb_thread_db.h (LIBTHREAD_DB_SEARCH_PATH): New define.
	    (LIBTHREAD_DB_SO): Moved from linux-thread-db.c

	    * linux-thread-db.c (try_thread_db_load_1): New function.
	    (try_thread_db_load, thread_db_load_search): Likewise.
	    (thread_db_load): Iterate over possibly multiple libthread_db's.

Index: gdb_thread_db.h
===================================================================
RCS file: /cvs/src/src/gdb/gdb_thread_db.h,v
retrieving revision 1.12
diff -u -p -u -r1.12 gdb_thread_db.h
--- gdb_thread_db.h	18 Mar 2009 08:51:11 -0000	1.12
+++ gdb_thread_db.h	6 Apr 2009 20:05:01 -0000
@@ -1,5 +1,14 @@
 #ifdef HAVE_THREAD_DB_H
 #include <thread_db.h>
+
+#ifndef LIBTHREAD_DB_SO
+#define LIBTHREAD_DB_SO "libthread_db.so.1"
+#endif
+
+#ifndef LIBTHREAD_DB_SEARCH_PATH
+#define LIBTHREAD_DB_SEARCH_PATH ""
+#endif
+
 #else
 
 /* Copyright (C) 1999, 2000, 2007, 2008, 2009 Free Software Foundation, Inc.
Index: linux-thread-db.c
===================================================================
RCS file: /cvs/src/src/gdb/linux-thread-db.c,v
retrieving revision 1.54
diff -u -p -u -r1.54 linux-thread-db.c
--- linux-thread-db.c	27 Feb 2009 20:34:41 -0000	1.54
+++ linux-thread-db.c	6 Apr 2009 20:05:01 -0000
@@ -26,13 +26,16 @@
 #include "gdb_thread_db.h"
 
 #include "bfd.h"
+#include "command.h"
 #include "exceptions.h"
+#include "gdbcmd.h"
 #include "gdbthread.h"
 #include "inferior.h"
 #include "symfile.h"
 #include "objfiles.h"
 #include "target.h"
 #include "regcache.h"
+#include "solib.h"
 #include "solib-svr4.h"
 #include "gdbcore.h"
 #include "observer.h"
@@ -44,10 +47,6 @@
 #include <gnu/libc-version.h>
 #endif
 
-#ifndef LIBTHREAD_DB_SO
-#define LIBTHREAD_DB_SO "libthread_db.so.1"
-#endif
-
 /* GNU/Linux libthread_db support.
 
    libthread_db is a library, provided along with libpthread.so, which
@@ -74,6 +73,8 @@
    of the ptid_t prevents thread IDs changing when libpthread is
    loaded or unloaded.  */
 
+static char *libthread_db_search_path;
+
 /* If we're running on GNU/Linux, we must explicitly attach to any new
    threads.  */
 
@@ -81,7 +82,7 @@
 static struct target_ops thread_db_ops;
 
 /* Non-zero if we're using this module's target vector.  */
-static int using_thread_db;
+static void *using_thread_db;
 
 /* Non-zero if we have determined the signals used by the threads
    library.  */
@@ -143,6 +144,10 @@ static void thread_db_find_new_threads_1
 static void attach_thread (ptid_t ptid, const td_thrhandle_t *th_p,
 			   const td_thrinfo_t *ti_p);
 static void detach_thread (ptid_t ptid);
+static void init_thread_db_ops (void);
+static td_err_e enable_thread_event (td_thragent_t *thread_agent,
+                                     int event, CORE_ADDR *bp);
+static void enable_thread_event_reporting (void);
 \f
 
 /* Use "struct private_thread_info" to cache thread state.  This is
@@ -386,32 +391,38 @@ verbose_dlsym (void *handle, const char 
 }
 
 static int
-thread_db_load (void)
+try_thread_db_load_1(void *handle)
 {
-  void *handle;
   td_err_e err;
 
-  handle = dlopen (LIBTHREAD_DB_SO, RTLD_NOW);
-  if (handle == NULL)
-    {
-      fprintf_filtered (gdb_stderr, "\n\ndlopen failed on '%s' - %s\n",
-			LIBTHREAD_DB_SO, dlerror ());
-      fprintf_filtered (gdb_stderr,
-			"GDB will not be able to debug pthreads.\n\n");
-      return 0;
-    }
-
   /* Initialize pointers to the dynamic library functions we will use.
      Essential functions first.  */
 
   td_init_p = verbose_dlsym (handle, "td_init");
   if (td_init_p == NULL)
     return 0;
+  err = td_init_p ();
+  if (err != TD_OK)
+    return 0;
 
   td_ta_new_p = verbose_dlsym (handle, "td_ta_new");
   if (td_ta_new_p == NULL)
     return 0;
 
+  /* Initialize the structure that identifies the child process.  */
+  proc_handle.ptid = inferior_ptid;
+
+  /* Now attempt to open a connection to the thread library.  */
+  err = td_ta_new_p (&proc_handle, &thread_agent);
+  if (err != TD_OK)
+    {
+      td_ta_new_p = NULL;
+      if (info_verbose)
+	printf_unfiltered (_("td_ta_new(): %s.\n"),
+			   thread_db_err_str (err));
+      return 0;
+    }
+
   td_ta_map_id2thr_p = verbose_dlsym (handle, "td_ta_map_id2thr");
   if (td_ta_map_id2thr_p == NULL)
     return 0;
@@ -432,14 +443,6 @@ thread_db_load (void)
   if (td_thr_get_info_p == NULL)
     return 0;
 
-  /* Initialize the library.  */
-  err = td_init_p ();
-  if (err != TD_OK)
-    {
-      warning (_("Cannot initialize libthread_db: %s"), thread_db_err_str (err));
-      return 0;
-    }
-
   /* These are not essential.  */
   td_ta_event_addr_p = dlsym (handle, "td_ta_event_addr");
   td_ta_set_event_p = dlsym (handle, "td_ta_set_event");
@@ -447,9 +450,141 @@ thread_db_load (void)
   td_thr_event_enable_p = dlsym (handle, "td_thr_event_enable");
   td_thr_tls_get_addr_p = dlsym (handle, "td_thr_tls_get_addr");
 
+  printf_unfiltered (_("[Thread debugging using libthread_db enabled]\n"));
+
+  init_thread_db_ops ();
+  add_target (&thread_db_ops);
+
+  /* The thread library was detected.  Activate the thread_db target.  */
+  push_target (&thread_db_ops);
+  using_thread_db = handle;
+
+  enable_thread_event_reporting ();
+  thread_db_find_new_threads_1 ();
   return 1;
 }
 
+static int
+try_thread_db_load (const char *library)
+{
+  void *handle;
+
+  if (info_verbose)
+    printf_unfiltered (_("Trying host libthread_db library \"%s\".\n"),
+                       library);
+  handle = dlopen (library, RTLD_NOW);
+  if (handle == NULL)
+    {
+      if (info_verbose)
+	printf_unfiltered (_("dlopen(): %s.\n"), dlerror ());
+      return 0;
+    }
+  if (try_thread_db_load_1 (handle))
+    return 1;
+
+  /* This library "refused" to work on current inferior.  */
+  dlclose (handle);
+  return 0;
+}
+
+static int
+thread_db_load_search ()
+{
+  char path[PATH_MAX];
+  const char *search_path = libthread_db_search_path;
+  int rc = 0;
+
+  while (*search_path)
+    {
+      const char *end = strchr (search_path, ':');
+      if (end)
+	{
+	  size_t len = end - search_path;
+	  strncpy (path, search_path, len);
+	  path[len] = '\0';
+	  search_path += len + 1;
+	}
+      else
+	{
+	  strcpy (path, search_path);
+	  search_path += strlen (search_path);
+	}
+      strcat (path, "/");
+      strcat (path, LIBTHREAD_DB_SO);
+      if (try_thread_db_load (path))
+	{
+	  rc = 1;
+	  break;
+	}
+    }
+  if (!rc)
+    rc = try_thread_db_load (LIBTHREAD_DB_SO);
+  if (rc)
+    {
+      Dl_info info;
+      const char *library = NULL;
+      if (dladdr ((*td_ta_new_p), &info) != 0)
+	library = info.dli_fname;
+      if (library == NULL)
+	library = LIBTHREAD_DB_SO;
+      printf_unfiltered (_("Warning: guessed host libthread_db "
+			   "library \"%s\".\n"), library);
+    }
+  else
+      printf_unfiltered (_("Warning: unable to guess libthread_db, "
+			   "thread debugging will not be available.\n"));
+  return rc;
+}
+
+static int
+thread_db_load (void)
+{
+  const char *soname;
+  struct minimal_symbol *msym;
+
+  if (using_thread_db)
+    return 1;
+
+  /* Don't attempt to use thread_db on targets which can not run
+     (executables not running yet, core files) for now.  */
+  if (!target_has_execution)
+    return 0;
+
+  /* Don't attempt to use thread_db for remote targets.  */
+  if (!target_can_run (&current_target))
+    return 0;
+
+  msym = lookup_minimal_symbol ("nptl_version", NULL, NULL);
+  if (!msym)
+    msym = lookup_minimal_symbol ("__linuxthreads_version", NULL, NULL);
+
+  if (!msym)
+    /* No threads yet */
+    return 0;
+
+  soname = solib_name_from_address (SYMBOL_VALUE_ADDRESS (msym));
+  if (soname)
+    {
+      /* Attempt to load libthread_db from the same directory. */
+      char path[PATH_MAX], *cp;
+      strcpy (path, soname);
+      cp = strrchr (path, '/');
+      if (cp == NULL)
+	{
+	  /* Expected to get fully resolved pathname, but got
+	     something else. Hope for the best.  */
+	  printf_unfiltered (_("warning: (Internal error: "
+			       "solib_name_from_address() returned \"%s\".\n"),
+			     soname);
+	  return thread_db_load_search ();
+	}
+      strcpy (cp + 1, LIBTHREAD_DB_SO);
+      if (try_thread_db_load (path))
+	return 1;
+    }
+  return thread_db_load_search ();
+}
+
 static td_err_e
 enable_thread_event (td_thragent_t *thread_agent, int event, CORE_ADDR *bp)
 {
@@ -593,17 +728,20 @@ void
 check_for_thread_db (void)
 {
   td_err_e err;
-  static int already_loaded;
+  static void *last_loaded;
 
   /* Do nothing if we couldn't load libthread_db.so.1.  */
-  if (td_ta_new_p == NULL)
+  if (!thread_db_load ())
     return;
 
   /* First time through, report that libthread_db was successfuly
      loaded.  Can't print this in in thread_db_load as, at that stage,
-     the interpreter and it's console haven't started.  */
+     the interpreter and it's console haven't started.
+     We track td_ta_new_p because the user may switch executables,
+     and as a result we may decide to use a different version of
+     libthread_db. */
 
-  if (!already_loaded)
+  if (last_loaded != td_ta_new_p)
     {
       Dl_info info;
       const char *library = NULL;
@@ -616,52 +754,9 @@ check_for_thread_db (void)
 	/* Paranoid - don't let a NULL path slip through.  */
 	library = LIBTHREAD_DB_SO;
 
-      if (info_verbose)
-	printf_unfiltered (_("Using host libthread_db library \"%s\".\n"),
-			   library);
-      already_loaded = 1;
-    }
-
-  if (using_thread_db)
-    /* Nothing to do.  The thread library was already detected and the
-       target vector was already activated.  */
-    return;
-
-  /* Don't attempt to use thread_db on targets which can not run
-     (executables not running yet, core files) for now.  */
-  if (!target_has_execution)
-    return;
-
-  /* Don't attempt to use thread_db for remote targets.  */
-  if (!target_can_run (&current_target))
-    return;
-
-  /* Initialize the structure that identifies the child process.  */
-  proc_handle.ptid = inferior_ptid;
-
-  /* Now attempt to open a connection to the thread library.  */
-  err = td_ta_new_p (&proc_handle, &thread_agent);
-  switch (err)
-    {
-    case TD_NOLIBTHREAD:
-      /* No thread library was detected.  */
-      break;
-
-    case TD_OK:
-      printf_unfiltered (_("[Thread debugging using libthread_db enabled]\n"));
-
-      /* The thread library was detected.  Activate the thread_db target.  */
-      push_target (&thread_db_ops);
-      using_thread_db = 1;
-
-      enable_thread_event_reporting ();
-      thread_db_find_new_threads_1 ();
-      break;
-
-    default:
-      warning (_("Cannot initialize thread debugging library: %s"),
-	       thread_db_err_str (err));
-      break;
+      printf_unfiltered (_("Using host libthread_db library \"%s\".\n"),
+                         library);
+      last_loaded = td_ta_new_p;
     }
 }
 
@@ -783,6 +878,8 @@ thread_db_detach (struct target_ops *ops
 
   /* Detach thread_db target ops.  */
   unpush_target (&thread_db_ops);
+  if (using_thread_db)
+    dlclose (using_thread_db);
   using_thread_db = 0;
 
   target_beneath->to_detach (target_beneath, args, from_tty);
@@ -896,7 +993,10 @@ thread_db_wait (struct target_ops *ops,
     {
       remove_thread_event_breakpoints ();
       unpush_target (&thread_db_ops);
+      if (using_thread_db)
+	dlclose (using_thread_db);
       using_thread_db = 0;
+      no_shared_libraries (NULL, 0);
 
       return ptid;
     }
@@ -944,6 +1044,8 @@ thread_db_mourn_inferior (struct target_
 
   /* Detach thread_db target ops.  */
   unpush_target (ops);
+  if (using_thread_db)
+    dlclose (using_thread_db);
   using_thread_db = 0;
 }
 
@@ -1187,13 +1289,24 @@ extern initialize_file_ftype _initialize
 void
 _initialize_thread_db (void)
 {
-  /* Only initialize the module if we can load libthread_db.  */
-  if (thread_db_load ())
-    {
-      init_thread_db_ops ();
-      add_target (&thread_db_ops);
-
-      /* Add ourselves to objfile event chain.  */
-      observer_attach_new_objfile (thread_db_new_objfile);
-    }
+  /* Defer loading of libthread_db.so until inferior is running.
+     This allows gdb to load correct libthread_db for a given
+     executable -- there could be mutiple versions of glibc,
+     compiled with LinuxThreads or NPTL, and until there is
+     a running inferior, we can't tell which libthread_db is
+     the correct one to load. */
+
+  libthread_db_search_path = xstrdup(LIBTHREAD_DB_SEARCH_PATH);
+
+  add_setshow_filename_cmd ("libthread-db-search-path", class_support,
+			    &libthread_db_search_path, _("\
+Set search path for libthread_db."), _("\
+Show the current search path or libthread_db."), _("\
+This path is used to search for libthread_db to be loaded into \
+gdb itself."),
+			    NULL,
+			    NULL,
+			    &setlist, &showlist);
+  /* Add ourselves to objfile event chain.  */
+  observer_attach_new_objfile (thread_db_new_objfile);
 }

Re: [patch][rfc] Allow GDB to search for the right libthread_db.so.1

[Thiago Jung Bauermann]

Hi Paul,

El lun, 06-04-2009 a las 13:39 -0700, Paul Pluzhnikov escribió:
> We have perhaps uncommon setup here, where we have several installed
> versions of glibc, and need to debug executables which are compiled
> and linked against them (using -rpath).

Many thanks for posting this patch! This is an issue for glibc
developers too, and I already considered working on exactly this
problem. I'm glad you did it first. :-)

> If this looks reasonable, I'll work on documentation next.
> There is also a matching change to gdbserver, which I am postponing until
> a decision on this patch is made.

Looks reasonable to me in general, but I have some comments. I wasn't
familiar with linux-thread-db.c before reviewing this patch, so I'd
appreciate if someone with more familiarity with this code would verify
that I'm not talking nonsense. :-)

> 2009-04-06  Paul Pluzhnikov  <ppluzhnikov@google.com>
> 
> 	    * gdb_thread_db.h (LIBTHREAD_DB_SEARCH_PATH): New define.
> 	    (LIBTHREAD_DB_SO): Moved from linux-thread-db.c
> 
> 	    * linux-thread-db.c (try_thread_db_load_1): New function.
> 	    (try_thread_db_load, thread_db_load_search): Likewise.
> 	    (thread_db_load): Iterate over possibly multiple libthread_db's.

I think the ChangeLog should mention that thread_db_load is now called
from check_for_thread_db instead of _initialize_thread_db. Also, there
are changes in the patch that this ChangeLog doesn't mention.

>  /* Non-zero if we're using this module's target vector.  */
> -static int using_thread_db;
> +static void *using_thread_db;

The comment for this variable needs to be updated. It's not a binary
flag anymore, so it should mention what the void * value is. The
variable should have a different name too, to reflect its new meaning.
 
>  static int
> -thread_db_load (void)
> +try_thread_db_load_1(void *handle)
>  {

I know that this function was undocumented already, but would you mind
adding a brief comment describing what it does, and what its return
value means?

Since you are changing a bit its behaviour and purpose, it seems fair
that you get to (briefly) document it. :-)

> +  /* Now attempt to open a connection to the thread library.  */
> +  err = td_ta_new_p (&proc_handle, &thread_agent);
> +  if (err != TD_OK)
> +    {
> +      td_ta_new_p = NULL;
> +      if (info_verbose)
> +	printf_unfiltered (_("td_ta_new(): %s.\n"),
> +			   thread_db_err_str (err));
> +      return 0;
> +    }

If err != TD_OK && err != TD_NOLIBTHREAD, a warning should be emitted,
like in the current version of the code:

      warning (_("Cannot initialize thread debugging library: %s"),
	       thread_db_err_str (err));

In your version of the code, this message is gone.

> @@ -447,9 +450,141 @@ thread_db_load (void)
>    td_thr_event_enable_p = dlsym (handle, "td_thr_event_enable");
>    td_thr_tls_get_addr_p = dlsym (handle, "td_thr_tls_get_addr");
>  
> +  printf_unfiltered (_("[Thread debugging using libthread_db enabled]\n"));
> +
> +  init_thread_db_ops ();
> +  add_target (&thread_db_ops);

Why can't you keep these calls in _initialize_thread_db? Isn't it wrong
to call add_target whenever a libthread_db is loaded (I admit I don't
know much about the target infrastructure in GDB)?

> +  handle = dlopen (library, RTLD_NOW);
> +  if (handle == NULL)
> +    {
> +      if (info_verbose)
> +	printf_unfiltered (_("dlopen(): %s.\n"), dlerror ());

The parenthesis shouldn't be in the message. Also, suggest being a bit
more descriptive: "dlopen failed: %s.\n"

The other error and information messages which currently have
parenthesis in them should also be fixed.

> +static int
> +thread_db_load_search ()
> +{
> +  char path[PATH_MAX];

This function can overflow path. Serious problem, IMHO.

> +      Dl_info info;
> +      const char *library = NULL;
> +      if (dladdr ((*td_ta_new_p), &info) != 0)
> +	library = info.dli_fname;
> +      if (library == NULL)
> +	library = LIBTHREAD_DB_SO;
> +      printf_unfiltered (_("Warning: guessed host libthread_db "
> +			   "library \"%s\".\n"), library);

Not sure this should be a warning. It's possible that the guessed
libthread_db is correct. Or is it more likely that it's not?

Also, I think it's clearer to rephrase it as:

"Searched libthread_db library to use, selected %s"

That is, avoid the (IMHO confusing) term "guessed library".

> +    }
> +  else
> +      printf_unfiltered (_("Warning: unable to guess libthread_db, "
> +			   "thread debugging will not be available.\n"));

Again, printf_unfiltered vs warning. I'd appreciate other opinions on
this topic. Also, suggest rewording to avoid "guess":

"Unable to find libthread_db matching inferior's thread library, thread
debugging will not be available.\n".

> +static int
> +thread_db_load (void)
> +{

Please add a comment describing the function and its return value.
Please also do this to the other functions which you introduced.

> +  msym = lookup_minimal_symbol ("nptl_version", NULL, NULL);
> +  if (!msym)
> +    msym = lookup_minimal_symbol ("__linuxthreads_version", NULL, NULL);
> +
> +  if (!msym)
> +    /* No threads yet */
> +    return 0;

Clever way to detect if a thread library is present. I can't comment on
its correctness and reliability though. Perhaps others will have more
insight. My only comment is that an alternative would be to search
through the inferior's link map. Don't know if it would be better or
worse.

Also, I assume you tested your patch in both NPTL systems and
LinuxThread systems, right?

> +  soname = solib_name_from_address (SYMBOL_VALUE_ADDRESS (msym));
> +  if (soname)
> +    {
> +      /* Attempt to load libthread_db from the same directory. */
> +      char path[PATH_MAX], *cp;
> +      strcpy (path, soname);
> +      cp = strrchr (path, '/');
> +      if (cp == NULL)
> +	{
> +	  /* Expected to get fully resolved pathname, but got
> +	     something else. Hope for the best.  */
> +	  printf_unfiltered (_("warning: (Internal error: "
> +			       "solib_name_from_address() returned \"%s\".\n"),
> +			     soname);
> +	  return thread_db_load_search ();
> +	}

"Internal error" has a specific meaning in GDB already, and it's not
what you use it for here. I suggest rewording to:

warning (_("Cannot obtain absolute path of thread library: %s"));

Note that I changed from calling printf_unfiltered to warning. I'm not
sure, but I gather the latter is preferred, right? (/me looks nervously
at other GDB developers, seeking a nod.)

Also, please clarify what "hope for the best" means here.

> +      printf_unfiltered (_("Using host libthread_db library \"%s\".\n"),
> +                         library);
> +      last_loaded = td_ta_new_p;

This message is currently only printed when verbose is on, but you
changed it to be always printed. Please provide a rationale for the
change.

> @@ -896,7 +993,10 @@ thread_db_wait (struct target_ops *ops,
>      {
>        remove_thread_event_breakpoints ();
>        unpush_target (&thread_db_ops);
> +      if (using_thread_db)
> +	dlclose (using_thread_db);
>        using_thread_db = 0;
> +      no_shared_libraries (NULL, 0);

I don't know much about GDB's mourning process, so this is a genuine
question: is this the appropriate place to call no_shared_libraries?
Doesn't feel right to me.
-- 
[]'s
Thiago Jung Bauermann
IBM Linux Technology Center

Re: [patch][rfc] Allow GDB to search for the right libthread_db.so.1

[Paul Pluzhnikov]

[-- Attachment #1: Type: text/plain, Size: 7982 bytes --]

On Wed, Apr 8, 2009 at 2:22 PM, Thiago Jung Bauermann
<bauerman@br.ibm.com> wrote:

Thank you for comments and interest in this patch :-)

Attached is a revised version, which I believe addresses all the issues
you have raised.

>>  /* Non-zero if we're using this module's target vector.  */
>> -static int using_thread_db;
>> +static void *using_thread_db;
>
> The comment for this variable needs to be updated. It's not a binary
> flag anymore, so it should mention what the void * value is. The
> variable should have a different name too, to reflect its new meaning.

Done.

>>  static int
>> -thread_db_load (void)
>> +try_thread_db_load_1(void *handle)
>>  {
>
> I know that this function was undocumented already, but would you mind
> adding a brief comment describing what it does, and what its return
> value means?
>
> Since you are changing a bit its behaviour and purpose, it seems fair
> that you get to (briefly) document it. :-)

Done.

>> +  /* Now attempt to open a connection to the thread library.  */
>> +  err = td_ta_new_p (&proc_handle, &thread_agent);
>> +  if (err != TD_OK)
>> +    {
>> +      td_ta_new_p = NULL;
>> +      if (info_verbose)
>> +     printf_unfiltered (_("td_ta_new(): %s.\n"),
>> +                        thread_db_err_str (err));
>> +      return 0;
>> +    }
>
> If err != TD_OK && err != TD_NOLIBTHREAD, a warning should be emitted,
> like in the current version of the code:
>
>      warning (_("Cannot initialize thread debugging library: %s"),
>               thread_db_err_str (err));

That's not quite right: you could also see TD_VERSION here (if you try the
"wrong" libthread_db first). I've changed the code to emit a warning for
other cases.

>> @@ -447,9 +450,141 @@ thread_db_load (void)
>>    td_thr_event_enable_p = dlsym (handle, "td_thr_event_enable");
>>    td_thr_tls_get_addr_p = dlsym (handle, "td_thr_tls_get_addr");
>>
>> +  printf_unfiltered (_("[Thread debugging using libthread_db enabled]\n"));
>> +
>> +  init_thread_db_ops ();
>> +  add_target (&thread_db_ops);
>
> Why can't you keep these calls in _initialize_thread_db? Isn't it wrong
> to call add_target whenever a libthread_db is loaded (I admit I don't
> know much about the target infrastructure in GDB)?

I believe you are quite correct. Fixed.

>> +  handle = dlopen (library, RTLD_NOW);
>> +  if (handle == NULL)
>> +    {
>> +      if (info_verbose)
>> +     printf_unfiltered (_("dlopen(): %s.\n"), dlerror ());
>
> The parenthesis shouldn't be in the message. Also, suggest being a bit
> more descriptive: "dlopen failed: %s.\n"

Fixed.

> The other error and information messages which currently have
> parenthesis in them should also be fixed.

Fixed.

>> +static int
>> +thread_db_load_search ()
>> +{
>> +  char path[PATH_MAX];
>
> This function can overflow path. Serious problem, IMHO.

Fixed.

>> +      Dl_info info;
>> +      const char *library = NULL;
>> +      if (dladdr ((*td_ta_new_p), &info) != 0)
>> +     library = info.dli_fname;
>> +      if (library == NULL)
>> +     library = LIBTHREAD_DB_SO;
>> +      printf_unfiltered (_("Warning: guessed host libthread_db "
>> +                        "library \"%s\".\n"), library);
>
> Not sure this should be a warning. It's possible that the guessed
> libthread_db is correct. Or is it more likely that it's not?

With our Google-specific setting of LIBTHREAD_DB_SEARCH_PATH, we expect
correct libthread_db to always be found on that path; hence the warning.

But with empty search path (as in this patch) it definitely should not be.
Fixed accordingly.

> Also, I think it's clearer to rephrase it as:
>
> "Searched libthread_db library to use, selected %s"
>
> That is, avoid the (IMHO confusing) term "guessed library".

Done.

>> +    }
>> +  else
>> +      printf_unfiltered (_("Warning: unable to guess libthread_db, "
>> +                        "thread debugging will not be available.\n"));
>
> Again, printf_unfiltered vs warning. I'd appreciate other opinions on
> this topic. Also, suggest rewording to avoid "guess":
>
> "Unable to find libthread_db matching inferior's thread library, thread
> debugging will not be available.\n".

Done.

>> +static int
>> +thread_db_load (void)
>> +{
>
> Please add a comment describing the function and its return value.
> Please also do this to the other functions which you introduced.

Done.

>> +  msym = lookup_minimal_symbol ("nptl_version", NULL, NULL);
>> +  if (!msym)
>> +    msym = lookup_minimal_symbol ("__linuxthreads_version", NULL, NULL);
>> +
>> +  if (!msym)
>> +    /* No threads yet */
>> +    return 0;
>
> Clever way to detect if a thread library is present. I can't comment on
> its correctness and reliability though.

I added one more symbol: __pthread_threads_events; really old versions of
LinuxThreads libpthread lacked the __linuxthreads_version.

> Perhaps others will have more
> insight. My only comment is that an alternative would be to search
> through the inferior's link map. Don't know if it would be better or
> worse.

Looking through link_map fails for statically-linked executables.

It is desireable to have a single GDB that "just works", regardless of
whether the executable was built statically or dynamically.

> Also, I assume you tested your patch in both NPTL systems and
> LinuxThread systems, right?

Yes: we have a mixture and I tested both.

>> +  soname = solib_name_from_address (SYMBOL_VALUE_ADDRESS (msym));
>> +  if (soname)
>> +    {
>> +      /* Attempt to load libthread_db from the same directory. */
>> +      char path[PATH_MAX], *cp;
>> +      strcpy (path, soname);
>> +      cp = strrchr (path, '/');
>> +      if (cp == NULL)
>> +     {
>> +       /* Expected to get fully resolved pathname, but got
>> +          something else. Hope for the best.  */
>> +       printf_unfiltered (_("warning: (Internal error: "
>> +                            "solib_name_from_address() returned \"%s\".\n"),
>> +                          soname);
>> +       return thread_db_load_search ();
>> +     }
>
> "Internal error" has a specific meaning in GDB already, and it's not
> what you use it for here. I suggest rewording to:
>
> warning (_("Cannot obtain absolute path of thread library: %s"));

Fixed.

> Note that I changed from calling printf_unfiltered to warning. I'm not
> sure, but I gather the latter is preferred, right? (/me looks nervously
> at other GDB developers, seeking a nod.)
>
> Also, please clarify what "hope for the best" means here.

Done.

>> +      printf_unfiltered (_("Using host libthread_db library \"%s\".\n"),
>> +                         library);
>> +      last_loaded = td_ta_new_p;
>
> This message is currently only printed when verbose is on, but you
> changed it to be always printed. Please provide a rationale for the
> change.

It's an "internal support" issue: we want to know at a glance which
libthread_db got loaded.

I changed this back to "verbose on" only.

>> @@ -896,7 +993,10 @@ thread_db_wait (struct target_ops *ops,
>>      {
>>        remove_thread_event_breakpoints ();
>>        unpush_target (&thread_db_ops);
>> +      if (using_thread_db)
>> +     dlclose (using_thread_db);
>>        using_thread_db = 0;
>> +      no_shared_libraries (NULL, 0);
>
> I don't know much about GDB's mourning process, so this is a genuine
> question: is this the appropriate place to call no_shared_libraries?
> Doesn't feel right to me.

This was a workaround for earlier bug, which (AFAICT) is fixed in current
source. I removed the call.

I've tried to add documentation changes for this patch, but can't seem to
find appropriate place for it :-(

I think "Debugging Programs with Multiple Threads" section is the most
appropriate place, but I am not sure.

Thanks,
-- 
Paul Pluzhnikov

[-- Attachment #2: gdb-thread-db.20090409.txt --]
[-- Type: text/plain, Size: 15860 bytes --]

Index: gdb_thread_db.h
===================================================================
RCS file: /cvs/src/src/gdb/gdb_thread_db.h,v
retrieving revision 1.12
diff -u -p -u -r1.12 gdb_thread_db.h
--- gdb_thread_db.h	18 Mar 2009 08:51:11 -0000	1.12
+++ gdb_thread_db.h	10 Apr 2009 00:28:27 -0000
@@ -1,5 +1,14 @@
 #ifdef HAVE_THREAD_DB_H
 #include <thread_db.h>
+
+#ifndef LIBTHREAD_DB_SO
+#define LIBTHREAD_DB_SO "libthread_db.so.1"
+#endif
+
+#ifndef LIBTHREAD_DB_SEARCH_PATH
+#define LIBTHREAD_DB_SEARCH_PATH ""
+#endif
+
 #else
 
 /* Copyright (C) 1999, 2000, 2007, 2008, 2009 Free Software Foundation, Inc.
Index: linux-thread-db.c
===================================================================
RCS file: /cvs/src/src/gdb/linux-thread-db.c,v
retrieving revision 1.54
diff -u -p -u -r1.54 linux-thread-db.c
--- linux-thread-db.c	27 Feb 2009 20:34:41 -0000	1.54
+++ linux-thread-db.c	10 Apr 2009 00:28:28 -0000
@@ -26,13 +26,16 @@
 #include "gdb_thread_db.h"
 
 #include "bfd.h"
+#include "command.h"
 #include "exceptions.h"
+#include "gdbcmd.h"
 #include "gdbthread.h"
 #include "inferior.h"
 #include "symfile.h"
 #include "objfiles.h"
 #include "target.h"
 #include "regcache.h"
+#include "solib.h"
 #include "solib-svr4.h"
 #include "gdbcore.h"
 #include "observer.h"
@@ -44,10 +47,6 @@
 #include <gnu/libc-version.h>
 #endif
 
-#ifndef LIBTHREAD_DB_SO
-#define LIBTHREAD_DB_SO "libthread_db.so.1"
-#endif
-
 /* GNU/Linux libthread_db support.
 
    libthread_db is a library, provided along with libpthread.so, which
@@ -74,14 +73,17 @@
    of the ptid_t prevents thread IDs changing when libpthread is
    loaded or unloaded.  */
 
+static char *libthread_db_search_path;
+
 /* If we're running on GNU/Linux, we must explicitly attach to any new
    threads.  */
 
 /* This module's target vector.  */
 static struct target_ops thread_db_ops;
 
-/* Non-zero if we're using this module's target vector.  */
-static int using_thread_db;
+/* Handle from dlopen for libthread_db.so.  Not NULL if we're using this
+   module's target vector.  */
+static void *thread_db_handle;
 
 /* Non-zero if we have determined the signals used by the threads
    library.  */
@@ -143,6 +145,9 @@ static void thread_db_find_new_threads_1
 static void attach_thread (ptid_t ptid, const td_thrhandle_t *th_p,
 			   const td_thrinfo_t *ti_p);
 static void detach_thread (ptid_t ptid);
+static td_err_e enable_thread_event (td_thragent_t *thread_agent,
+                                     int event, CORE_ADDR *bp);
+static void enable_thread_event_reporting (void);
 \f
 
 /* Use "struct private_thread_info" to cache thread state.  This is
@@ -344,7 +349,7 @@ thread_db_attach_lwp (ptid_t ptid)
   td_thrinfo_t ti;
   td_err_e err;
 
-  if (!using_thread_db)
+  if (thread_db_handle == NULL)
     return 0;
 
   /* This ptid comes from linux-nat.c, which should always fill in the
@@ -385,22 +390,17 @@ verbose_dlsym (void *handle, const char 
   return sym;
 }
 
+/* Attempt to initialize dlopen()ed libthread_db, described by HANDLE.
+   Return 1 on success.
+   Failure could happen if libthread_db does not have symbols we expect,
+   or when it refuses to work with the current inferior (e.g. due to
+   version mismatch between libthread_db and libpthread).  */
+
 static int
-thread_db_load (void)
+try_thread_db_load_1(void *handle)
 {
-  void *handle;
   td_err_e err;
 
-  handle = dlopen (LIBTHREAD_DB_SO, RTLD_NOW);
-  if (handle == NULL)
-    {
-      fprintf_filtered (gdb_stderr, "\n\ndlopen failed on '%s' - %s\n",
-			LIBTHREAD_DB_SO, dlerror ());
-      fprintf_filtered (gdb_stderr,
-			"GDB will not be able to debug pthreads.\n\n");
-      return 0;
-    }
-
   /* Initialize pointers to the dynamic library functions we will use.
      Essential functions first.  */
 
@@ -408,10 +408,45 @@ thread_db_load (void)
   if (td_init_p == NULL)
     return 0;
 
+  err = td_init_p ();
+  if (err != TD_OK)
+    {
+      warning (_("Cannot initialize libthread_db: %s"), thread_db_err_str (err));
+      return 0;
+    }
+
   td_ta_new_p = verbose_dlsym (handle, "td_ta_new");
   if (td_ta_new_p == NULL)
     return 0;
 
+  /* Initialize the structure that identifies the child process.  */
+  proc_handle.ptid = inferior_ptid;
+
+  /* Now attempt to open a connection to the thread library.  */
+  err = td_ta_new_p (&proc_handle, &thread_agent);
+  if (err != TD_OK)
+    {
+      td_ta_new_p = NULL;
+      if (info_verbose)
+	printf_unfiltered (_("td_ta_new failed: %s\n"),
+			   thread_db_err_str (err));
+      else
+        switch (err)
+          {
+            case TD_NOLIBTHREAD:
+#ifdef THREAD_DB_HAS_TD_VERSION
+            case TD_VERSION:
+#endif
+              /* The errors above are not unexpected and silently ignored:
+                 they just mean we haven't found correct version of
+                 libthread_db yet.  */
+              break;
+            default:
+              warning (_("td_ta_new failed: %s"), thread_db_err_str (err));
+          }
+      return 0;
+    }
+
   td_ta_map_id2thr_p = verbose_dlsym (handle, "td_ta_map_id2thr");
   if (td_ta_map_id2thr_p == NULL)
     return 0;
@@ -432,14 +467,6 @@ thread_db_load (void)
   if (td_thr_get_info_p == NULL)
     return 0;
 
-  /* Initialize the library.  */
-  err = td_init_p ();
-  if (err != TD_OK)
-    {
-      warning (_("Cannot initialize libthread_db: %s"), thread_db_err_str (err));
-      return 0;
-    }
-
   /* These are not essential.  */
   td_ta_event_addr_p = dlsym (handle, "td_ta_event_addr");
   td_ta_set_event_p = dlsym (handle, "td_ta_set_event");
@@ -447,9 +474,191 @@ thread_db_load (void)
   td_thr_event_enable_p = dlsym (handle, "td_thr_event_enable");
   td_thr_tls_get_addr_p = dlsym (handle, "td_thr_tls_get_addr");
 
+  printf_unfiltered (_("[Thread debugging using libthread_db enabled]\n"));
+
+  /* The thread library was detected.  Activate the thread_db target.  */
+  push_target (&thread_db_ops);
+  thread_db_handle = handle;
+
+  enable_thread_event_reporting ();
+  thread_db_find_new_threads_1 ();
   return 1;
 }
 
+/* Lookup a library in which given symbol resides.
+   Note: this is looking in GDB process, not in the inferior.
+   Returns library name, or NULL.  */
+
+static const char *
+dladdr_to_soname (const void *addr)
+{
+  Dl_info info;
+
+  if (dladdr (addr, &info) != 0)
+    return info.dli_fname;
+  return NULL;
+}
+
+/* Attempt to use LIBRARY as libthread_db.  LIBRARY could be absolute,
+   relative, or just LIBTHREAD_DB.  */
+
+static int
+try_thread_db_load (const char *library)
+{
+  void *handle;
+
+  if (info_verbose)
+    printf_unfiltered (_("Trying host libthread_db library: %s.\n"),
+                       library);
+  handle = dlopen (library, RTLD_NOW);
+  if (handle == NULL)
+    {
+      if (info_verbose)
+	printf_unfiltered (_("dlopen failed: %s.\n"), dlerror ());
+      return 0;
+    }
+
+  if (info_verbose && strchr(library, '/') == NULL)
+    {
+      void *td_init;
+
+      td_init = dlsym (handle, "td_init");
+      if (td_init != NULL)
+        {
+          const char *const libpath = dladdr_to_soname (td_init);
+
+          if (libpath != NULL)
+            printf_unfiltered (_("Host %s resolved to: %s.\n"),
+                               library, libpath);
+        }
+    }
+
+  if (try_thread_db_load_1 (handle))
+    return 1;
+
+  /* This library "refused" to work on current inferior.  */
+  dlclose (handle);
+  return 0;
+}
+
+
+/* Search libthread_db_search_path for libthread_db which "agrees"
+   to work on current inferior.  */
+
+static int
+thread_db_load_search ()
+{
+  char path[PATH_MAX];
+  const char *search_path = libthread_db_search_path;
+  int rc = 0;
+
+  while (*search_path)
+    {
+      const char *end = strchr (search_path, ':');
+      if (end)
+	{
+	  size_t len = end - search_path;
+          if (len + 1 + strlen (LIBTHREAD_DB_SO) + 1 > sizeof (path))
+            {
+              char *cp = xmalloc (len + 1);
+              memcpy (cp, search_path, len);
+              cp[len] = '\0';
+              warning (_("libthread_db_search_path component too long,"
+                         " ignored: %s."), cp);
+              xfree (cp);
+              search_path += len + 1;
+              continue;
+            }
+	  memcpy (path, search_path, len);
+	  path[len] = '\0';
+	  search_path += len + 1;
+	}
+      else
+	{
+          size_t len = strlen (search_path);
+
+          if (len + 1 + strlen (LIBTHREAD_DB_SO) + 1 > sizeof (path))
+            {
+              warning (_("libthread_db_search_path component too long,"
+                         " ignored: %s."), search_path);
+              break;
+            }
+	  memcpy (path, search_path, len + 1);
+	  search_path += len;
+	}
+      strcat (path, "/");
+      strcat (path, LIBTHREAD_DB_SO);
+      if (try_thread_db_load (path))
+	{
+	  rc = 1;
+	  break;
+	}
+    }
+  if (rc == 0)
+    rc = try_thread_db_load (LIBTHREAD_DB_SO);
+  if (rc == 0)
+    warning (_("Unable to find libthread_db matching inferior's thread"
+               " library, thread debugging will not be available."));
+  return rc;
+}
+
+/* Attempt to load and initialize libthread_db.
+   Return 1 on success.
+ */
+
+static int
+thread_db_load (void)
+{
+  const char *soname;
+  struct minimal_symbol *msym;
+
+  if (thread_db_handle != NULL)
+    return 1;
+
+  /* Don't attempt to use thread_db on targets which can not run
+     (executables not running yet, core files) for now.  */
+  if (!target_has_execution)
+    return 0;
+
+  /* Don't attempt to use thread_db for remote targets.  */
+  if (!target_can_run (&current_target))
+    return 0;
+
+  msym = lookup_minimal_symbol ("nptl_version", NULL, NULL);
+  if (!msym)
+    msym = lookup_minimal_symbol ("__linuxthreads_version", NULL, NULL);
+
+  /* Some really old libpthread versions do not have either of the above.  */
+  if (!msym)
+    msym = lookup_minimal_symbol ("__pthread_threads_events", NULL, NULL);
+
+  if (!msym)
+    /* No threads yet */
+    return 0;
+
+  soname = solib_name_from_address (SYMBOL_VALUE_ADDRESS (msym));
+  if (soname)
+    {
+      /* Attempt to load libthread_db from the same directory. */
+      char path[PATH_MAX], *cp;
+      strcpy (path, soname);
+      cp = strrchr (path, '/');
+      if (cp == NULL)
+	{
+	  /* Expected to get fully resolved pathname for libpthread,
+             but got something else.  Search for matching libthread_db and
+             hope there is one that matches current libpthread.  */
+	  warning (_("Cannot obtain absolute path of thread library: %s."),
+                   soname);
+	  return thread_db_load_search ();
+	}
+      strcpy (cp + 1, LIBTHREAD_DB_SO);
+      if (try_thread_db_load (path))
+	return 1;
+    }
+  return thread_db_load_search ();
+}
+
 static td_err_e
 enable_thread_event (td_thragent_t *thread_agent, int event, CORE_ADDR *bp)
 {
@@ -593,75 +802,34 @@ void
 check_for_thread_db (void)
 {
   td_err_e err;
-  static int already_loaded;
+  static void *last_loaded;
 
   /* Do nothing if we couldn't load libthread_db.so.1.  */
-  if (td_ta_new_p == NULL)
+  if (!thread_db_load ())
     return;
 
   /* First time through, report that libthread_db was successfuly
      loaded.  Can't print this in in thread_db_load as, at that stage,
-     the interpreter and it's console haven't started.  */
+     the interpreter and it's console haven't started.
+     We track td_ta_new_p because the user may switch executables,
+     and as a result we may decide to use a different version of
+     libthread_db. */
 
-  if (!already_loaded)
+  if (last_loaded != td_ta_new_p)
     {
-      Dl_info info;
-      const char *library = NULL;
-      if (dladdr ((*td_ta_new_p), &info) != 0)
-	library = info.dli_fname;
-
-      /* Try dlinfo?  */
-
-      if (library == NULL)
-	/* Paranoid - don't let a NULL path slip through.  */
-	library = LIBTHREAD_DB_SO;
+      last_loaded = td_ta_new_p;
 
       if (info_verbose)
-	printf_unfiltered (_("Using host libthread_db library \"%s\".\n"),
-			   library);
-      already_loaded = 1;
-    }
-
-  if (using_thread_db)
-    /* Nothing to do.  The thread library was already detected and the
-       target vector was already activated.  */
-    return;
-
-  /* Don't attempt to use thread_db on targets which can not run
-     (executables not running yet, core files) for now.  */
-  if (!target_has_execution)
-    return;
-
-  /* Don't attempt to use thread_db for remote targets.  */
-  if (!target_can_run (&current_target))
-    return;
-
-  /* Initialize the structure that identifies the child process.  */
-  proc_handle.ptid = inferior_ptid;
-
-  /* Now attempt to open a connection to the thread library.  */
-  err = td_ta_new_p (&proc_handle, &thread_agent);
-  switch (err)
-    {
-    case TD_NOLIBTHREAD:
-      /* No thread library was detected.  */
-      break;
-
-    case TD_OK:
-      printf_unfiltered (_("[Thread debugging using libthread_db enabled]\n"));
-
-      /* The thread library was detected.  Activate the thread_db target.  */
-      push_target (&thread_db_ops);
-      using_thread_db = 1;
+        {
+          const char *library;
 
-      enable_thread_event_reporting ();
-      thread_db_find_new_threads_1 ();
-      break;
-
-    default:
-      warning (_("Cannot initialize thread debugging library: %s"),
-	       thread_db_err_str (err));
-      break;
+          library = dladdr_to_soname (*td_ta_new_p);
+          if (library == NULL)
+            library = LIBTHREAD_DB_SO;
+
+          printf_unfiltered (_("Using host libthread_db library \"%s\".\n"),
+                             library);
+        }
     }
 }
 
@@ -783,7 +951,9 @@ thread_db_detach (struct target_ops *ops
 
   /* Detach thread_db target ops.  */
   unpush_target (&thread_db_ops);
-  using_thread_db = 0;
+  if (thread_db_handle)
+    dlclose (thread_db_handle);
+  thread_db_handle = NULL;
 
   target_beneath->to_detach (target_beneath, args, from_tty);
 }
@@ -896,7 +1066,9 @@ thread_db_wait (struct target_ops *ops,
     {
       remove_thread_event_breakpoints ();
       unpush_target (&thread_db_ops);
-      using_thread_db = 0;
+      if (thread_db_handle)
+	dlclose (thread_db_handle);
+      thread_db_handle = NULL;
 
       return ptid;
     }
@@ -944,7 +1116,9 @@ thread_db_mourn_inferior (struct target_
 
   /* Detach thread_db target ops.  */
   unpush_target (ops);
-  using_thread_db = 0;
+  if (thread_db_handle)
+    dlclose (thread_db_handle);
+  thread_db_handle = NULL;
 }
 
 static int
@@ -1187,13 +1361,27 @@ extern initialize_file_ftype _initialize
 void
 _initialize_thread_db (void)
 {
-  /* Only initialize the module if we can load libthread_db.  */
-  if (thread_db_load ())
-    {
-      init_thread_db_ops ();
-      add_target (&thread_db_ops);
+  init_thread_db_ops ();
+  add_target (&thread_db_ops);
 
-      /* Add ourselves to objfile event chain.  */
-      observer_attach_new_objfile (thread_db_new_objfile);
-    }
+  /* Defer loading of libthread_db.so until inferior is running.
+     This allows gdb to load correct libthread_db for a given
+     executable -- there could be mutiple versions of glibc,
+     compiled with LinuxThreads or NPTL, and until there is
+     a running inferior, we can't tell which libthread_db is
+     the correct one to load. */
+
+  libthread_db_search_path = xstrdup(LIBTHREAD_DB_SEARCH_PATH);
+
+  add_setshow_filename_cmd ("libthread-db-search-path", class_support,
+			    &libthread_db_search_path, _("\
+Set search path for libthread_db."), _("\
+Show the current search path or libthread_db."), _("\
+This path is used to search for libthread_db to be loaded into \
+gdb itself."),
+			    NULL,
+			    NULL,
+			    &setlist, &showlist);
+  /* Add ourselves to objfile event chain.  */
+  observer_attach_new_objfile (thread_db_new_objfile);
 }

Re: [patch][rfc] Allow GDB to search for the right libthread_db.so.1

[Tom Tromey]

>>>>> "Paul" == Paul Pluzhnikov <ppluzhnikov@google.com> writes:

Paul> Attached is a revised version, which I believe addresses all the issues
Paul> you have raised.

I think this seems like reasonable functionality to have.

>> This message is currently only printed when verbose is on, but you
>> changed it to be always printed. Please provide a rationale for the
>> change.

Paul> It's an "internal support" issue: we want to know at a glance which
Paul> libthread_db got loaded.

Paul> I changed this back to "verbose on" only.

If you want, I think it would make sense to unconditionally print this
message if a search path was specified.

Paul> I've tried to add documentation changes for this patch, but can't seem to
Paul> find appropriate place for it :-(

Paul> I think "Debugging Programs with Multiple Threads" section is the most
Paul> appropriate place, but I am not sure.

It seems reasonable to me, but perhaps Eli could provide his opinion.

Tom

Re: [patch][rfc] Allow GDB to search for the right libthread_db.so.1

[Eli Zaretskii]

> Cc: Thiago Jung Bauermann <bauerman@br.ibm.com>,         gdb-patches ml <gdb-patches@sourceware.org>
> From: Tom Tromey <tromey@redhat.com>
> Date: Thu, 16 Apr 2009 11:53:58 -0600
> 
> Paul> I think "Debugging Programs with Multiple Threads" section is the most
> Paul> appropriate place, but I am not sure.
> 
> It seems reasonable to me, but perhaps Eli could provide his opinion.

Yes, that's reasonable: we generally document such variables together
with the features to which they are tied, cf. "set solib-search-path",
for example.

Re: [patch][rfc] Allow GDB to search for the right libthread_db.so.1

[Paul Pluzhnikov]

[-- Attachment #1: Type: text/plain, Size: 1251 bytes --]

On Thu, Apr 16, 2009 at 10:53 AM, Tom Tromey <tromey@redhat.com> wrote:

> If you want, I think it would make sense to unconditionally print this
> message if a search path was specified.

Good idea. Fixed.

Attached is a revised patch, with updated ChangeLog and documentation
changes.

Tested on Linux/x86_64 with no regressions.

Thanks,
-- 
Paul Pluzhnikov

2009-04-17  Paul Pluzhnikov  <ppluzhnikov@google.com>

	* gdb_thread_db.h (LIBTHREAD_DB_SEARCH_PATH): New define.
	(LIBTHREAD_DB_SO): Moved from linux-thread-db.c

	* linux-thread-db.c (libthread_db_search_path): New setting.
	(thread_db_handle): New variable (replaces using_thread_db).
	(try_thread_db_load_1): New function.
	(try_thread_db_load, thread_db_load_search): Likewise.
	(dladdr_to_soname): Likewise.
	(thread_db_load): Iterate over possibly multiple libthread_db's.
	(check_for_thread_db): Attempt to load new libthread_db.
	(thread_db_detach, thread_db_wait): Unload libthread_db.
	(thread_db_mourn_inferior): Likewise.
	(_initialize_thread_db): Add new libthread-db-search-path option.
	Defer loading of libthread_db to check_for_thread_db.

doc/ChangeLog

2009-04-17  Paul Pluzhnikov  <ppluzhnikov@google.com>

	* gdb.texinfo (Threads): Document libthread-db-search-path.

[-- Attachment #2: gdb-thread-db.20090417.txt --]
[-- Type: text/plain, Size: 19766 bytes --]

Index: gdb_thread_db.h
===================================================================
RCS file: /cvs/src/src/gdb/gdb_thread_db.h,v
retrieving revision 1.12
diff -u -p -u -r1.12 gdb_thread_db.h
--- gdb_thread_db.h	18 Mar 2009 08:51:11 -0000	1.12
+++ gdb_thread_db.h	17 Apr 2009 18:27:43 -0000
@@ -1,5 +1,14 @@
 #ifdef HAVE_THREAD_DB_H
 #include <thread_db.h>
+
+#ifndef LIBTHREAD_DB_SO
+#define LIBTHREAD_DB_SO "libthread_db.so.1"
+#endif
+
+#ifndef LIBTHREAD_DB_SEARCH_PATH
+#define LIBTHREAD_DB_SEARCH_PATH ""
+#endif
+
 #else
 
 /* Copyright (C) 1999, 2000, 2007, 2008, 2009 Free Software Foundation, Inc.
Index: linux-thread-db.c
===================================================================
RCS file: /cvs/src/src/gdb/linux-thread-db.c,v
retrieving revision 1.54
diff -u -p -u -r1.54 linux-thread-db.c
--- linux-thread-db.c	27 Feb 2009 20:34:41 -0000	1.54
+++ linux-thread-db.c	17 Apr 2009 18:27:43 -0000
@@ -26,13 +26,16 @@
 #include "gdb_thread_db.h"
 
 #include "bfd.h"
+#include "command.h"
 #include "exceptions.h"
+#include "gdbcmd.h"
 #include "gdbthread.h"
 #include "inferior.h"
 #include "symfile.h"
 #include "objfiles.h"
 #include "target.h"
 #include "regcache.h"
+#include "solib.h"
 #include "solib-svr4.h"
 #include "gdbcore.h"
 #include "observer.h"
@@ -44,10 +47,6 @@
 #include <gnu/libc-version.h>
 #endif
 
-#ifndef LIBTHREAD_DB_SO
-#define LIBTHREAD_DB_SO "libthread_db.so.1"
-#endif
-
 /* GNU/Linux libthread_db support.
 
    libthread_db is a library, provided along with libpthread.so, which
@@ -74,14 +73,17 @@
    of the ptid_t prevents thread IDs changing when libpthread is
    loaded or unloaded.  */
 
+static char *libthread_db_search_path;
+
 /* If we're running on GNU/Linux, we must explicitly attach to any new
    threads.  */
 
 /* This module's target vector.  */
 static struct target_ops thread_db_ops;
 
-/* Non-zero if we're using this module's target vector.  */
-static int using_thread_db;
+/* Handle from dlopen for libthread_db.so.  Not NULL if we're using this
+   module's target vector.  */
+static void *thread_db_handle;
 
 /* Non-zero if we have determined the signals used by the threads
    library.  */
@@ -344,7 +346,7 @@ thread_db_attach_lwp (ptid_t ptid)
   td_thrinfo_t ti;
   td_err_e err;
 
-  if (!using_thread_db)
+  if (thread_db_handle == NULL)
     return 0;
 
   /* This ptid comes from linux-nat.c, which should always fill in the
@@ -385,71 +387,6 @@ verbose_dlsym (void *handle, const char 
   return sym;
 }
 
-static int
-thread_db_load (void)
-{
-  void *handle;
-  td_err_e err;
-
-  handle = dlopen (LIBTHREAD_DB_SO, RTLD_NOW);
-  if (handle == NULL)
-    {
-      fprintf_filtered (gdb_stderr, "\n\ndlopen failed on '%s' - %s\n",
-			LIBTHREAD_DB_SO, dlerror ());
-      fprintf_filtered (gdb_stderr,
-			"GDB will not be able to debug pthreads.\n\n");
-      return 0;
-    }
-
-  /* Initialize pointers to the dynamic library functions we will use.
-     Essential functions first.  */
-
-  td_init_p = verbose_dlsym (handle, "td_init");
-  if (td_init_p == NULL)
-    return 0;
-
-  td_ta_new_p = verbose_dlsym (handle, "td_ta_new");
-  if (td_ta_new_p == NULL)
-    return 0;
-
-  td_ta_map_id2thr_p = verbose_dlsym (handle, "td_ta_map_id2thr");
-  if (td_ta_map_id2thr_p == NULL)
-    return 0;
-
-  td_ta_map_lwp2thr_p = verbose_dlsym (handle, "td_ta_map_lwp2thr");
-  if (td_ta_map_lwp2thr_p == NULL)
-    return 0;
-
-  td_ta_thr_iter_p = verbose_dlsym (handle, "td_ta_thr_iter");
-  if (td_ta_thr_iter_p == NULL)
-    return 0;
-
-  td_thr_validate_p = verbose_dlsym (handle, "td_thr_validate");
-  if (td_thr_validate_p == NULL)
-    return 0;
-
-  td_thr_get_info_p = verbose_dlsym (handle, "td_thr_get_info");
-  if (td_thr_get_info_p == NULL)
-    return 0;
-
-  /* Initialize the library.  */
-  err = td_init_p ();
-  if (err != TD_OK)
-    {
-      warning (_("Cannot initialize libthread_db: %s"), thread_db_err_str (err));
-      return 0;
-    }
-
-  /* These are not essential.  */
-  td_ta_event_addr_p = dlsym (handle, "td_ta_event_addr");
-  td_ta_set_event_p = dlsym (handle, "td_ta_set_event");
-  td_ta_event_getmsg_p = dlsym (handle, "td_ta_event_getmsg");
-  td_thr_event_enable_p = dlsym (handle, "td_thr_event_enable");
-  td_thr_tls_get_addr_p = dlsym (handle, "td_thr_tls_get_addr");
-
-  return 1;
-}
-
 static td_err_e
 enable_thread_event (td_thragent_t *thread_agent, int event, CORE_ADDR *bp)
 {
@@ -541,6 +478,275 @@ enable_thread_event_reporting (void)
     }
 }
 
+/* Attempt to initialize dlopen()ed libthread_db, described by HANDLE.
+   Return 1 on success.
+   Failure could happen if libthread_db does not have symbols we expect,
+   or when it refuses to work with the current inferior (e.g. due to
+   version mismatch between libthread_db and libpthread).  */
+
+static int
+try_thread_db_load_1(void *handle)
+{
+  td_err_e err;
+
+  /* Initialize pointers to the dynamic library functions we will use.
+     Essential functions first.  */
+
+  td_init_p = verbose_dlsym (handle, "td_init");
+  if (td_init_p == NULL)
+    return 0;
+
+  err = td_init_p ();
+  if (err != TD_OK)
+    {
+      warning (_("Cannot initialize libthread_db: %s"), thread_db_err_str (err));
+      return 0;
+    }
+
+  td_ta_new_p = verbose_dlsym (handle, "td_ta_new");
+  if (td_ta_new_p == NULL)
+    return 0;
+
+  /* Initialize the structure that identifies the child process.  */
+  proc_handle.ptid = inferior_ptid;
+
+  /* Now attempt to open a connection to the thread library.  */
+  err = td_ta_new_p (&proc_handle, &thread_agent);
+  if (err != TD_OK)
+    {
+      td_ta_new_p = NULL;
+      if (info_verbose)
+	printf_unfiltered (_("td_ta_new failed: %s\n"),
+			   thread_db_err_str (err));
+      else
+        switch (err)
+          {
+            case TD_NOLIBTHREAD:
+#ifdef THREAD_DB_HAS_TD_VERSION
+            case TD_VERSION:
+#endif
+              /* The errors above are not unexpected and silently ignored:
+                 they just mean we haven't found correct version of
+                 libthread_db yet.  */
+              break;
+            default:
+              warning (_("td_ta_new failed: %s"), thread_db_err_str (err));
+          }
+      return 0;
+    }
+
+  td_ta_map_id2thr_p = verbose_dlsym (handle, "td_ta_map_id2thr");
+  if (td_ta_map_id2thr_p == NULL)
+    return 0;
+
+  td_ta_map_lwp2thr_p = verbose_dlsym (handle, "td_ta_map_lwp2thr");
+  if (td_ta_map_lwp2thr_p == NULL)
+    return 0;
+
+  td_ta_thr_iter_p = verbose_dlsym (handle, "td_ta_thr_iter");
+  if (td_ta_thr_iter_p == NULL)
+    return 0;
+
+  td_thr_validate_p = verbose_dlsym (handle, "td_thr_validate");
+  if (td_thr_validate_p == NULL)
+    return 0;
+
+  td_thr_get_info_p = verbose_dlsym (handle, "td_thr_get_info");
+  if (td_thr_get_info_p == NULL)
+    return 0;
+
+  /* These are not essential.  */
+  td_ta_event_addr_p = dlsym (handle, "td_ta_event_addr");
+  td_ta_set_event_p = dlsym (handle, "td_ta_set_event");
+  td_ta_event_getmsg_p = dlsym (handle, "td_ta_event_getmsg");
+  td_thr_event_enable_p = dlsym (handle, "td_thr_event_enable");
+  td_thr_tls_get_addr_p = dlsym (handle, "td_thr_tls_get_addr");
+
+  printf_unfiltered (_("[Thread debugging using libthread_db enabled]\n"));
+
+  /* The thread library was detected.  Activate the thread_db target.  */
+  push_target (&thread_db_ops);
+  thread_db_handle = handle;
+
+  enable_thread_event_reporting ();
+  thread_db_find_new_threads_1 ();
+  return 1;
+}
+
+/* Lookup a library in which given symbol resides.
+   Note: this is looking in GDB process, not in the inferior.
+   Returns library name, or NULL.  */
+
+static const char *
+dladdr_to_soname (const void *addr)
+{
+  Dl_info info;
+
+  if (dladdr (addr, &info) != 0)
+    return info.dli_fname;
+  return NULL;
+}
+
+/* Attempt to use LIBRARY as libthread_db.  LIBRARY could be absolute,
+   relative, or just LIBTHREAD_DB.  */
+
+static int
+try_thread_db_load (const char *library)
+{
+  void *handle;
+
+  if (info_verbose)
+    printf_unfiltered (_("Trying host libthread_db library: %s.\n"),
+                       library);
+  handle = dlopen (library, RTLD_NOW);
+  if (handle == NULL)
+    {
+      if (info_verbose)
+	printf_unfiltered (_("dlopen failed: %s.\n"), dlerror ());
+      return 0;
+    }
+
+  if (info_verbose && strchr(library, '/') == NULL)
+    {
+      void *td_init;
+
+      td_init = dlsym (handle, "td_init");
+      if (td_init != NULL)
+        {
+          const char *const libpath = dladdr_to_soname (td_init);
+
+          if (libpath != NULL)
+            printf_unfiltered (_("Host %s resolved to: %s.\n"),
+                               library, libpath);
+        }
+    }
+
+  if (try_thread_db_load_1 (handle))
+    return 1;
+
+  /* This library "refused" to work on current inferior.  */
+  dlclose (handle);
+  return 0;
+}
+
+
+/* Search libthread_db_search_path for libthread_db which "agrees"
+   to work on current inferior.  */
+
+static int
+thread_db_load_search ()
+{
+  char path[PATH_MAX];
+  const char *search_path = libthread_db_search_path;
+  int rc = 0;
+
+  while (*search_path)
+    {
+      const char *end = strchr (search_path, ':');
+      if (end)
+	{
+	  size_t len = end - search_path;
+          if (len + 1 + strlen (LIBTHREAD_DB_SO) + 1 > sizeof (path))
+            {
+              char *cp = xmalloc (len + 1);
+              memcpy (cp, search_path, len);
+              cp[len] = '\0';
+              warning (_("libthread_db_search_path component too long,"
+                         " ignored: %s."), cp);
+              xfree (cp);
+              search_path += len + 1;
+              continue;
+            }
+	  memcpy (path, search_path, len);
+	  path[len] = '\0';
+	  search_path += len + 1;
+	}
+      else
+	{
+          size_t len = strlen (search_path);
+
+          if (len + 1 + strlen (LIBTHREAD_DB_SO) + 1 > sizeof (path))
+            {
+              warning (_("libthread_db_search_path component too long,"
+                         " ignored: %s."), search_path);
+              break;
+            }
+	  memcpy (path, search_path, len + 1);
+	  search_path += len;
+	}
+      strcat (path, "/");
+      strcat (path, LIBTHREAD_DB_SO);
+      if (try_thread_db_load (path))
+	{
+	  rc = 1;
+	  break;
+	}
+    }
+  if (rc == 0)
+    rc = try_thread_db_load (LIBTHREAD_DB_SO);
+  if (rc == 0)
+    warning (_("Unable to find libthread_db matching inferior's thread"
+               " library, thread debugging will not be available."));
+  return rc;
+}
+
+/* Attempt to load and initialize libthread_db.
+   Return 1 on success.
+ */
+
+static int
+thread_db_load (void)
+{
+  const char *soname;
+  struct minimal_symbol *msym;
+
+  if (thread_db_handle != NULL)
+    return 1;
+
+  /* Don't attempt to use thread_db on targets which can not run
+     (executables not running yet, core files) for now.  */
+  if (!target_has_execution)
+    return 0;
+
+  /* Don't attempt to use thread_db for remote targets.  */
+  if (!target_can_run (&current_target))
+    return 0;
+
+  msym = lookup_minimal_symbol ("nptl_version", NULL, NULL);
+  if (!msym)
+    msym = lookup_minimal_symbol ("__linuxthreads_version", NULL, NULL);
+
+  /* Some really old libpthread versions do not have either of the above.  */
+  if (!msym)
+    msym = lookup_minimal_symbol ("__pthread_threads_events", NULL, NULL);
+
+  if (!msym)
+    /* No threads yet */
+    return 0;
+
+  soname = solib_name_from_address (SYMBOL_VALUE_ADDRESS (msym));
+  if (soname)
+    {
+      /* Attempt to load libthread_db from the same directory. */
+      char path[PATH_MAX], *cp;
+      strcpy (path, soname);
+      cp = strrchr (path, '/');
+      if (cp == NULL)
+	{
+	  /* Expected to get fully resolved pathname for libpthread,
+             but got something else.  Search for matching libthread_db and
+             hope there is one that matches current libpthread.  */
+	  warning (_("Cannot obtain absolute path of thread library: %s."),
+                   soname);
+	  return thread_db_load_search ();
+	}
+      strcpy (cp + 1, LIBTHREAD_DB_SO);
+      if (try_thread_db_load (path))
+	return 1;
+    }
+  return thread_db_load_search ();
+}
+
 static void
 disable_thread_event_reporting (void)
 {
@@ -593,75 +799,34 @@ void
 check_for_thread_db (void)
 {
   td_err_e err;
-  static int already_loaded;
+  static void *last_loaded;
 
   /* Do nothing if we couldn't load libthread_db.so.1.  */
-  if (td_ta_new_p == NULL)
+  if (!thread_db_load ())
     return;
 
   /* First time through, report that libthread_db was successfuly
      loaded.  Can't print this in in thread_db_load as, at that stage,
-     the interpreter and it's console haven't started.  */
+     the interpreter and it's console haven't started.
+     We track td_ta_new_p because the user may switch executables,
+     and as a result we may decide to use a different version of
+     libthread_db. */
 
-  if (!already_loaded)
+  if (last_loaded != td_ta_new_p)
     {
-      Dl_info info;
-      const char *library = NULL;
-      if (dladdr ((*td_ta_new_p), &info) != 0)
-	library = info.dli_fname;
+      last_loaded = td_ta_new_p;
 
-      /* Try dlinfo?  */
-
-      if (library == NULL)
-	/* Paranoid - don't let a NULL path slip through.  */
-	library = LIBTHREAD_DB_SO;
-
-      if (info_verbose)
-	printf_unfiltered (_("Using host libthread_db library \"%s\".\n"),
-			   library);
-      already_loaded = 1;
-    }
-
-  if (using_thread_db)
-    /* Nothing to do.  The thread library was already detected and the
-       target vector was already activated.  */
-    return;
-
-  /* Don't attempt to use thread_db on targets which can not run
-     (executables not running yet, core files) for now.  */
-  if (!target_has_execution)
-    return;
-
-  /* Don't attempt to use thread_db for remote targets.  */
-  if (!target_can_run (&current_target))
-    return;
-
-  /* Initialize the structure that identifies the child process.  */
-  proc_handle.ptid = inferior_ptid;
-
-  /* Now attempt to open a connection to the thread library.  */
-  err = td_ta_new_p (&proc_handle, &thread_agent);
-  switch (err)
-    {
-    case TD_NOLIBTHREAD:
-      /* No thread library was detected.  */
-      break;
-
-    case TD_OK:
-      printf_unfiltered (_("[Thread debugging using libthread_db enabled]\n"));
-
-      /* The thread library was detected.  Activate the thread_db target.  */
-      push_target (&thread_db_ops);
-      using_thread_db = 1;
-
-      enable_thread_event_reporting ();
-      thread_db_find_new_threads_1 ();
-      break;
-
-    default:
-      warning (_("Cannot initialize thread debugging library: %s"),
-	       thread_db_err_str (err));
-      break;
+      if (info_verbose || *libthread_db_search_path)
+        {
+          const char *library;
+
+          library = dladdr_to_soname (*td_ta_new_p);
+          if (library == NULL)
+            library = LIBTHREAD_DB_SO;
+
+          printf_unfiltered (_("Using host libthread_db library \"%s\".\n"),
+                             library);
+        }
     }
 }
 
@@ -783,7 +948,9 @@ thread_db_detach (struct target_ops *ops
 
   /* Detach thread_db target ops.  */
   unpush_target (&thread_db_ops);
-  using_thread_db = 0;
+  if (thread_db_handle)
+    dlclose (thread_db_handle);
+  thread_db_handle = NULL;
 
   target_beneath->to_detach (target_beneath, args, from_tty);
 }
@@ -896,7 +1063,9 @@ thread_db_wait (struct target_ops *ops,
     {
       remove_thread_event_breakpoints ();
       unpush_target (&thread_db_ops);
-      using_thread_db = 0;
+      if (thread_db_handle)
+	dlclose (thread_db_handle);
+      thread_db_handle = NULL;
 
       return ptid;
     }
@@ -944,7 +1113,9 @@ thread_db_mourn_inferior (struct target_
 
   /* Detach thread_db target ops.  */
   unpush_target (ops);
-  using_thread_db = 0;
+  if (thread_db_handle)
+    dlclose (thread_db_handle);
+  thread_db_handle = NULL;
 }
 
 static int
@@ -1187,13 +1358,28 @@ extern initialize_file_ftype _initialize
 void
 _initialize_thread_db (void)
 {
-  /* Only initialize the module if we can load libthread_db.  */
-  if (thread_db_load ())
-    {
-      init_thread_db_ops ();
-      add_target (&thread_db_ops);
+  init_thread_db_ops ();
+  add_target (&thread_db_ops);
 
-      /* Add ourselves to objfile event chain.  */
-      observer_attach_new_objfile (thread_db_new_objfile);
-    }
+  /* Defer loading of libthread_db.so until inferior is running.
+     This allows gdb to load correct libthread_db for a given
+     executable -- there could be mutiple versions of glibc,
+     compiled with LinuxThreads or NPTL, and until there is
+     a running inferior, we can't tell which libthread_db is
+     the correct one to load. */
+
+  libthread_db_search_path = xstrdup(LIBTHREAD_DB_SEARCH_PATH);
+
+  add_setshow_optional_filename_cmd ("libthread-db-search-path",
+				     class_support,
+				     &libthread_db_search_path, _("\
+Set search path for libthread_db."), _("\
+Show the current search path or libthread_db."), _("\
+This path is used to search for libthread_db to be loaded into \
+gdb itself."),
+			    NULL,
+			    NULL,
+			    &setlist, &showlist);
+  /* Add ourselves to objfile event chain.  */
+  observer_attach_new_objfile (thread_db_new_objfile);
 }
Index: doc/gdb.texinfo
===================================================================
RCS file: /cvs/src/src/gdb/doc/gdb.texinfo,v
retrieving revision 1.580
diff -u -p -u -r1.580 gdb.texinfo
--- doc/gdb.texinfo	15 Apr 2009 22:20:32 -0000	1.580
+++ doc/gdb.texinfo	17 Apr 2009 18:27:44 -0000
@@ -2429,6 +2429,9 @@ a command to apply a command to a list o
 @item thread-specific breakpoints
 @item @samp{set print thread-events}, which controls printing of 
 messages on thread start and exit.
+@item @samp{set libthread-db-search-path @var{path}}, which lets
+the user specify which @code{libthread_db} to use if the default choice
+isn't compatible with the program.
 @end itemize
 
 @quotation
@@ -2647,6 +2650,39 @@ programs with multiple threads.
 @xref{Set Watchpoints,,Setting Watchpoints}, for information about
 watchpoints in programs with multiple threads.
 
+@table @code
+@kindex set libthread-db-search-path
+@item set libthread-db-search-path @r{[}@var{path}@r{]}
+If this variable is set, @var{path} is a colon-separated list of
+directories @value{GDBN} will use to search for @code{libthread_db}.
+If you omit @var{path}, @samp{libthread-db-search-path} will be reset to
+empty list.
+
+On @sc{gnu}/Linux and Solaris systems, @value{GDBN} uses a ``helper''
+@code{libthread_db} library to obtain information about threads in the
+inferior process.  @value{GDBN} first attempts to use
+@code{libthread_db} located in the same directory, from which
+@code{libpthread} was loaded in the inferior process.  If that fails,
+@value{GDBN} will use @samp{libthread-db-search-path},
+and then default system shared library directories, to find
+@code{libthread_db}.
+
+For any @code{libthread_db} library @value{GDBN} finds in above directories,
+@value{GDBN} attempts to initialize it with the current inferior process.
+If this initialization fails (which could happen because of a version
+mismatch between @code{libthread_db} and @code{libpthread}), @value{GDBN}
+will unload @code{libthread_db}, and continue with the next directory.
+If none of @code{libthread_db} libraries initialize successfully,
+thread debugging will be disabled.
+
+Setting @code{libthread-db-search-path} is currently implemented
+only for @sc{gnu}/Linux targets.
+
+@kindex show libthread-db-search-path 
+@item show libthread-db-search-path 
+Display current libthread_db search path.
+@end table
+
 @node Processes
 @section Debugging Programs with Multiple Processes
 

Re: [patch][rfc] Allow GDB to search for the right libthread_db.so.1

[Eli Zaretskii]

> Date: Fri, 17 Apr 2009 12:13:08 -0700
> From: Paul Pluzhnikov <ppluzhnikov@google.com>
> Cc: Thiago Jung Bauermann <bauerman@br.ibm.com>,         gdb-patches ml <gdb-patches@sourceware.org>
> 
> Attached is a revised patch, with updated ChangeLog and documentation
> changes.

Thanks.  Please also add a new entry to NEWS, in the "New options"
section.  We have just decided to have all the new options mentioned
there.

> doc/ChangeLog
> 
> 2009-04-17  Paul Pluzhnikov  <ppluzhnikov@google.com>
> 
> 	* gdb.texinfo (Threads): Document libthread-db-search-path.

This part is approved, with a few comments:

> +@table @code
> +@kindex set libthread-db-search-path
> +@item set libthread-db-search-path @r{[}@var{path}@r{]}
> +If this variable is set, @var{path} is a colon-separated list of
> +directories @value{GDBN} will use to search for @code{libthread_db}.

I would add here some suitable @cindex entry, for those who don't know
or don't remember the name of the variable.  Something like

  @cindex search path for @code{libthread_db}

> +If you omit @var{path}, @samp{libthread-db-search-path} will be reset to
> +empty list.

"will be reset to an empty list", with the article.

> +inferior process.  @value{GDBN} first attempts to use
> +@code{libthread_db} located in the same directory, from which
                                                    ^
In English, there's no need for a comma here.  Actually, it would
sound a bit better if you remove the word "same" from this sentence.

> +If none of @code{libthread_db} libraries initialize successfully,
> +thread debugging will be disabled.

I think we should mention here that GDB displays a warning in this
case.

> +Setting @code{libthread-db-search-path} is currently implemented
> +only for @sc{gnu}/Linux targets.

In general, mentioning specific platforms in the manual is a
maintenance nightmare, because we must remember to make changes in the
manual to track the support of the feature on other platforms.  It is
much better to say that this feature is available "only on some
platforms."  That's assuming that on other platforms, using the
command will cause a suitable error message; does it?

Btw, what are the Linux-specific aspects that this feature depends on?
Why can't we implement it on Solaris as well?

Re: [patch][rfc] Allow GDB to search for the right libthread_db.so.1

[Thiago Jung Bauermann]

El vie, 17-04-2009 a las 12:13 -0700, Paul Pluzhnikov escribió:
> Attached is a revised patch, with updated ChangeLog and documentation
> changes.

This version looks great to me. My only comment is that you are using
dladdr, which is a glibc extension. Since linux-thread-db.c is only used
in native Linux targets, this doesn't seem much of a problem. But do we
care about Linux systems with other libc implementations (e.g., diet
libc, uClibc (I didn't check whether theses libraries have dladdr or
not))?
-- 
[]'s
Thiago Jung Bauermann
IBM Linux Technology Center

Re: [patch][rfc] Allow GDB to search for the right libthread_db.so.1

[Paul Pluzhnikov]

On Sun, Apr 19, 2009 at 7:58 AM, Thiago Jung Bauermann
<bauerman@br.ibm.com> wrote:

> My only comment is that you are using dladdr, which is a glibc extension.

I am not adding this as a new dependency though; currently checked source
already depends on it.

> But do we care about Linux systems with other libc implementations
> (e.g., diet libc, uClibc ?

AFAICT, both diet libc and uClibc have it (uClibc only if it has shared
library support compiled in; but if it doesn't, then libthread_db isn't
usable anyway).

-- 
Paul Pluzhnikov

Re: [patch][rfc] Allow GDB to search for the right libthread_db.so.1

[Daniel Jacobowitz]

I have just a few minor comments on this.  Overall, I am heartily
in favor of the idea - I have carried a less elegant version of this
feature in the Debian GDB packages for two years or so.

On Fri, Apr 17, 2009 at 12:13:08PM -0700, Paul Pluzhnikov wrote:
> +#ifndef LIBTHREAD_DB_SO
> +#define LIBTHREAD_DB_SO "libthread_db.so.1"
> +#endif
> +
> +#ifndef LIBTHREAD_DB_SEARCH_PATH
> +#define LIBTHREAD_DB_SEARCH_PATH ""
> +#endif

Do we intend to let anything override these?  Otherwise IMHO the
ifndef just confuses the issue.

> +static int
> +try_thread_db_load_1(void *handle)

Space before paren :-)

> +  if (info_verbose && strchr(library, '/') == NULL)

Oh, here too.

> +  msym = lookup_minimal_symbol ("nptl_version", NULL, NULL);
> +  if (!msym)
> +    msym = lookup_minimal_symbol ("__linuxthreads_version", NULL, NULL);
> +
> +  /* Some really old libpthread versions do not have either of the above.  */
> +  if (!msym)
> +    msym = lookup_minimal_symbol ("__pthread_threads_events", NULL, NULL);
> +
> +  if (!msym)
> +    /* No threads yet */
> +    return 0;

Why is this symbol lookup necessary?  Is it accomplishing the same
thing that searching for a shared library matching "libpthread[-.]"
would?  Also see libpthread_solib_p, which uses basically that
method.

> +  libthread_db_search_path = xstrdup(LIBTHREAD_DB_SEARCH_PATH);

Space there too :-)

-- 
Daniel Jacobowitz
CodeSourcery

Re: [patch][rfc] Allow GDB to search for the right libthread_db.so.1

[Paul Pluzhnikov]

[-- Attachment #1: Type: text/plain, Size: 3830 bytes --]

On Mon, Apr 20, 2009 at 6:17 AM, Daniel Jacobowitz <drow@false.org> wrote:

> I have just a few minor comments on this.  Overall, I am heartily
> in favor of the idea - I have carried a less elegant version of this
> feature in the Debian GDB packages for two years or so.

Could you tell a bit more about this? I thought out set up with
multiple incompatible (WRT libthread_db) libc versions was more an
exception than the rule.

> On Fri, Apr 17, 2009 at 12:13:08PM -0700, Paul Pluzhnikov wrote:
>> +#ifndef LIBTHREAD_DB_SO
>> +#define LIBTHREAD_DB_SO "libthread_db.so.1"
>> +#endif
>> +
>> +#ifndef LIBTHREAD_DB_SEARCH_PATH
>> +#define LIBTHREAD_DB_SEARCH_PATH ""
>> +#endif
>
> Do we intend to let anything override these?  Otherwise IMHO the
> ifndef just confuses the issue.

It might be reasonable to override these via CFLAGS. In particular,
we intend to override the LIBTHREAD_DB_SEARCH_PATH with an appropriate
local default. Of course we could just as easily patch the source.

> Space before paren :-)

Sorry about that. Fixed everywhere.

>> +  if (!msym)
>> +    msym = lookup_minimal_symbol ("__pthread_threads_events", NULL, NULL);
>
> Why is this symbol lookup necessary?  Is it accomplishing the same
> thing that searching for a shared library matching "libpthread[-.]"
> would?

The symbol lookup works for statically linked executables (which we
also have and would like to support). Searching for libpthread solib
doesn't.

On Sat, Apr 18, 2009 at 10:00 AM, Eli Zaretskii <eliz@gnu.org> wrote:

>> +Setting @code{libthread-db-search-path} is currently implemented
>> +only for @sc{gnu}/Linux targets.
>
> In general, mentioning specific platforms in the manual is a
> maintenance nightmare, because we must remember to make changes in the
> manual to track the support of the feature on other platforms.  It is
> much better to say that this feature is available "only on some
> platforms."

Done.

>  That's assuming that on other platforms, using the
> command will cause a suitable error message; does it?

Yes: since the add_setshow... is not done on other platforms, you'll
get the same error as if you tried to set unknown variable (which is
actually quite confusing!):

   ./gdb -q -nx /bin/date
  (no debugging symbols found)
  (gdb) set libthread-db-search-path
  No symbol table is loaded.  Use the "file" command.
  (gdb)

> Btw, what are the Linux-specific aspects that this feature depends on?
> Why can't we implement it on Solaris as well?

This could quite easily be implemented on Solaris as well, although I
expect that there is significantly smaller number of users who need
this on Solaris. OTOH, anyone using glibc on Solaris will likely want
it (if he debugs both glibc and native-libc binaries).


Attached is a revised patch that fixes all the comments so far.

Thanks,
-- 
Paul Pluzhnikov

2009-04-20  Paul Pluzhnikov  <ppluzhnikov@google.com>

	* gdb_thread_db.h (LIBTHREAD_DB_SEARCH_PATH): New define.
	(LIBTHREAD_DB_SO): Moved from linux-thread-db.c

	* linux-thread-db.c (libthread_db_search_path): New setting.
	(thread_db_handle): New variable (replaces using_thread_db).
	(try_thread_db_load_1): New function.
	(try_thread_db_load, thread_db_load_search): Likewise.
	(dladdr_to_soname): Likewise.
	(thread_db_load): Iterate over possibly multiple libthread_db's.
	(check_for_thread_db): Attempt to load new libthread_db.
	(thread_db_detach, thread_db_wait): Unload libthread_db.
	(thread_db_mourn_inferior): Likewise.
	(_initialize_thread_db): Add new libthread-db-search-path option.
	Defer loading of libthread_db to check_for_thread_db.

doc/ChangeLog

2009-04-20  Paul Pluzhnikov  <ppluzhnikov@google.com>

	* gdb.texinfo (Threads): Document libthread-db-search-path.

[-- Attachment #2: gdb-thread-db.20090420.txt --]
[-- Type: text/plain, Size: 19840 bytes --]

Index: gdb_thread_db.h
===================================================================
RCS file: /cvs/src/src/gdb/gdb_thread_db.h,v
retrieving revision 1.12
diff -u -p -u -r1.12 gdb_thread_db.h
--- gdb_thread_db.h	18 Mar 2009 08:51:11 -0000	1.12
+++ gdb_thread_db.h	20 Apr 2009 16:41:55 -0000
@@ -1,5 +1,14 @@
 #ifdef HAVE_THREAD_DB_H
 #include <thread_db.h>
+
+#ifndef LIBTHREAD_DB_SO
+#define LIBTHREAD_DB_SO "libthread_db.so.1"
+#endif
+
+#ifndef LIBTHREAD_DB_SEARCH_PATH
+#define LIBTHREAD_DB_SEARCH_PATH ""
+#endif
+
 #else
 
 /* Copyright (C) 1999, 2000, 2007, 2008, 2009 Free Software Foundation, Inc.
Index: linux-thread-db.c
===================================================================
RCS file: /cvs/src/src/gdb/linux-thread-db.c,v
retrieving revision 1.54
diff -u -p -u -r1.54 linux-thread-db.c
--- linux-thread-db.c	27 Feb 2009 20:34:41 -0000	1.54
+++ linux-thread-db.c	20 Apr 2009 16:41:55 -0000
@@ -26,13 +26,16 @@
 #include "gdb_thread_db.h"
 
 #include "bfd.h"
+#include "command.h"
 #include "exceptions.h"
+#include "gdbcmd.h"
 #include "gdbthread.h"
 #include "inferior.h"
 #include "symfile.h"
 #include "objfiles.h"
 #include "target.h"
 #include "regcache.h"
+#include "solib.h"
 #include "solib-svr4.h"
 #include "gdbcore.h"
 #include "observer.h"
@@ -44,10 +47,6 @@
 #include <gnu/libc-version.h>
 #endif
 
-#ifndef LIBTHREAD_DB_SO
-#define LIBTHREAD_DB_SO "libthread_db.so.1"
-#endif
-
 /* GNU/Linux libthread_db support.
 
    libthread_db is a library, provided along with libpthread.so, which
@@ -74,14 +73,17 @@
    of the ptid_t prevents thread IDs changing when libpthread is
    loaded or unloaded.  */
 
+static char *libthread_db_search_path;
+
 /* If we're running on GNU/Linux, we must explicitly attach to any new
    threads.  */
 
 /* This module's target vector.  */
 static struct target_ops thread_db_ops;
 
-/* Non-zero if we're using this module's target vector.  */
-static int using_thread_db;
+/* Handle from dlopen for libthread_db.so.  Not NULL if we're using this
+   module's target vector.  */
+static void *thread_db_handle;
 
 /* Non-zero if we have determined the signals used by the threads
    library.  */
@@ -344,7 +346,7 @@ thread_db_attach_lwp (ptid_t ptid)
   td_thrinfo_t ti;
   td_err_e err;
 
-  if (!using_thread_db)
+  if (thread_db_handle == NULL)
     return 0;
 
   /* This ptid comes from linux-nat.c, which should always fill in the
@@ -385,71 +387,6 @@ verbose_dlsym (void *handle, const char 
   return sym;
 }
 
-static int
-thread_db_load (void)
-{
-  void *handle;
-  td_err_e err;
-
-  handle = dlopen (LIBTHREAD_DB_SO, RTLD_NOW);
-  if (handle == NULL)
-    {
-      fprintf_filtered (gdb_stderr, "\n\ndlopen failed on '%s' - %s\n",
-			LIBTHREAD_DB_SO, dlerror ());
-      fprintf_filtered (gdb_stderr,
-			"GDB will not be able to debug pthreads.\n\n");
-      return 0;
-    }
-
-  /* Initialize pointers to the dynamic library functions we will use.
-     Essential functions first.  */
-
-  td_init_p = verbose_dlsym (handle, "td_init");
-  if (td_init_p == NULL)
-    return 0;
-
-  td_ta_new_p = verbose_dlsym (handle, "td_ta_new");
-  if (td_ta_new_p == NULL)
-    return 0;
-
-  td_ta_map_id2thr_p = verbose_dlsym (handle, "td_ta_map_id2thr");
-  if (td_ta_map_id2thr_p == NULL)
-    return 0;
-
-  td_ta_map_lwp2thr_p = verbose_dlsym (handle, "td_ta_map_lwp2thr");
-  if (td_ta_map_lwp2thr_p == NULL)
-    return 0;
-
-  td_ta_thr_iter_p = verbose_dlsym (handle, "td_ta_thr_iter");
-  if (td_ta_thr_iter_p == NULL)
-    return 0;
-
-  td_thr_validate_p = verbose_dlsym (handle, "td_thr_validate");
-  if (td_thr_validate_p == NULL)
-    return 0;
-
-  td_thr_get_info_p = verbose_dlsym (handle, "td_thr_get_info");
-  if (td_thr_get_info_p == NULL)
-    return 0;
-
-  /* Initialize the library.  */
-  err = td_init_p ();
-  if (err != TD_OK)
-    {
-      warning (_("Cannot initialize libthread_db: %s"), thread_db_err_str (err));
-      return 0;
-    }
-
-  /* These are not essential.  */
-  td_ta_event_addr_p = dlsym (handle, "td_ta_event_addr");
-  td_ta_set_event_p = dlsym (handle, "td_ta_set_event");
-  td_ta_event_getmsg_p = dlsym (handle, "td_ta_event_getmsg");
-  td_thr_event_enable_p = dlsym (handle, "td_thr_event_enable");
-  td_thr_tls_get_addr_p = dlsym (handle, "td_thr_tls_get_addr");
-
-  return 1;
-}
-
 static td_err_e
 enable_thread_event (td_thragent_t *thread_agent, int event, CORE_ADDR *bp)
 {
@@ -541,6 +478,275 @@ enable_thread_event_reporting (void)
     }
 }
 
+/* Attempt to initialize dlopen()ed libthread_db, described by HANDLE.
+   Return 1 on success.
+   Failure could happen if libthread_db does not have symbols we expect,
+   or when it refuses to work with the current inferior (e.g. due to
+   version mismatch between libthread_db and libpthread).  */
+
+static int
+try_thread_db_load_1 (void *handle)
+{
+  td_err_e err;
+
+  /* Initialize pointers to the dynamic library functions we will use.
+     Essential functions first.  */
+
+  td_init_p = verbose_dlsym (handle, "td_init");
+  if (td_init_p == NULL)
+    return 0;
+
+  err = td_init_p ();
+  if (err != TD_OK)
+    {
+      warning (_("Cannot initialize libthread_db: %s"), thread_db_err_str (err));
+      return 0;
+    }
+
+  td_ta_new_p = verbose_dlsym (handle, "td_ta_new");
+  if (td_ta_new_p == NULL)
+    return 0;
+
+  /* Initialize the structure that identifies the child process.  */
+  proc_handle.ptid = inferior_ptid;
+
+  /* Now attempt to open a connection to the thread library.  */
+  err = td_ta_new_p (&proc_handle, &thread_agent);
+  if (err != TD_OK)
+    {
+      td_ta_new_p = NULL;
+      if (info_verbose)
+	printf_unfiltered (_("td_ta_new failed: %s\n"),
+			   thread_db_err_str (err));
+      else
+        switch (err)
+          {
+            case TD_NOLIBTHREAD:
+#ifdef THREAD_DB_HAS_TD_VERSION
+            case TD_VERSION:
+#endif
+              /* The errors above are not unexpected and silently ignored:
+                 they just mean we haven't found correct version of
+                 libthread_db yet.  */
+              break;
+            default:
+              warning (_("td_ta_new failed: %s"), thread_db_err_str (err));
+          }
+      return 0;
+    }
+
+  td_ta_map_id2thr_p = verbose_dlsym (handle, "td_ta_map_id2thr");
+  if (td_ta_map_id2thr_p == NULL)
+    return 0;
+
+  td_ta_map_lwp2thr_p = verbose_dlsym (handle, "td_ta_map_lwp2thr");
+  if (td_ta_map_lwp2thr_p == NULL)
+    return 0;
+
+  td_ta_thr_iter_p = verbose_dlsym (handle, "td_ta_thr_iter");
+  if (td_ta_thr_iter_p == NULL)
+    return 0;
+
+  td_thr_validate_p = verbose_dlsym (handle, "td_thr_validate");
+  if (td_thr_validate_p == NULL)
+    return 0;
+
+  td_thr_get_info_p = verbose_dlsym (handle, "td_thr_get_info");
+  if (td_thr_get_info_p == NULL)
+    return 0;
+
+  /* These are not essential.  */
+  td_ta_event_addr_p = dlsym (handle, "td_ta_event_addr");
+  td_ta_set_event_p = dlsym (handle, "td_ta_set_event");
+  td_ta_event_getmsg_p = dlsym (handle, "td_ta_event_getmsg");
+  td_thr_event_enable_p = dlsym (handle, "td_thr_event_enable");
+  td_thr_tls_get_addr_p = dlsym (handle, "td_thr_tls_get_addr");
+
+  printf_unfiltered (_("[Thread debugging using libthread_db enabled]\n"));
+
+  /* The thread library was detected.  Activate the thread_db target.  */
+  push_target (&thread_db_ops);
+  thread_db_handle = handle;
+
+  enable_thread_event_reporting ();
+  thread_db_find_new_threads_1 ();
+  return 1;
+}
+
+/* Lookup a library in which given symbol resides.
+   Note: this is looking in GDB process, not in the inferior.
+   Returns library name, or NULL.  */
+
+static const char *
+dladdr_to_soname (const void *addr)
+{
+  Dl_info info;
+
+  if (dladdr (addr, &info) != 0)
+    return info.dli_fname;
+  return NULL;
+}
+
+/* Attempt to use LIBRARY as libthread_db.  LIBRARY could be absolute,
+   relative, or just LIBTHREAD_DB.  */
+
+static int
+try_thread_db_load (const char *library)
+{
+  void *handle;
+
+  if (info_verbose)
+    printf_unfiltered (_("Trying host libthread_db library: %s.\n"),
+                       library);
+  handle = dlopen (library, RTLD_NOW);
+  if (handle == NULL)
+    {
+      if (info_verbose)
+	printf_unfiltered (_("dlopen failed: %s.\n"), dlerror ());
+      return 0;
+    }
+
+  if (info_verbose && strchr (library, '/') == NULL)
+    {
+      void *td_init;
+
+      td_init = dlsym (handle, "td_init");
+      if (td_init != NULL)
+        {
+          const char *const libpath = dladdr_to_soname (td_init);
+
+          if (libpath != NULL)
+            printf_unfiltered (_("Host %s resolved to: %s.\n"),
+                               library, libpath);
+        }
+    }
+
+  if (try_thread_db_load_1 (handle))
+    return 1;
+
+  /* This library "refused" to work on current inferior.  */
+  dlclose (handle);
+  return 0;
+}
+
+
+/* Search libthread_db_search_path for libthread_db which "agrees"
+   to work on current inferior.  */
+
+static int
+thread_db_load_search ()
+{
+  char path[PATH_MAX];
+  const char *search_path = libthread_db_search_path;
+  int rc = 0;
+
+  while (*search_path)
+    {
+      const char *end = strchr (search_path, ':');
+      if (end)
+	{
+	  size_t len = end - search_path;
+          if (len + 1 + strlen (LIBTHREAD_DB_SO) + 1 > sizeof (path))
+            {
+              char *cp = xmalloc (len + 1);
+              memcpy (cp, search_path, len);
+              cp[len] = '\0';
+              warning (_("libthread_db_search_path component too long,"
+                         " ignored: %s."), cp);
+              xfree (cp);
+              search_path += len + 1;
+              continue;
+            }
+	  memcpy (path, search_path, len);
+	  path[len] = '\0';
+	  search_path += len + 1;
+	}
+      else
+	{
+          size_t len = strlen (search_path);
+
+          if (len + 1 + strlen (LIBTHREAD_DB_SO) + 1 > sizeof (path))
+            {
+              warning (_("libthread_db_search_path component too long,"
+                         " ignored: %s."), search_path);
+              break;
+            }
+	  memcpy (path, search_path, len + 1);
+	  search_path += len;
+	}
+      strcat (path, "/");
+      strcat (path, LIBTHREAD_DB_SO);
+      if (try_thread_db_load (path))
+	{
+	  rc = 1;
+	  break;
+	}
+    }
+  if (rc == 0)
+    rc = try_thread_db_load (LIBTHREAD_DB_SO);
+  if (rc == 0)
+    warning (_("Unable to find libthread_db matching inferior's thread"
+               " library, thread debugging will not be available."));
+  return rc;
+}
+
+/* Attempt to load and initialize libthread_db.
+   Return 1 on success.
+ */
+
+static int
+thread_db_load (void)
+{
+  const char *soname;
+  struct minimal_symbol *msym;
+
+  if (thread_db_handle != NULL)
+    return 1;
+
+  /* Don't attempt to use thread_db on targets which can not run
+     (executables not running yet, core files) for now.  */
+  if (!target_has_execution)
+    return 0;
+
+  /* Don't attempt to use thread_db for remote targets.  */
+  if (!target_can_run (&current_target))
+    return 0;
+
+  msym = lookup_minimal_symbol ("nptl_version", NULL, NULL);
+  if (!msym)
+    msym = lookup_minimal_symbol ("__linuxthreads_version", NULL, NULL);
+
+  /* Some really old libpthread versions do not have either of the above.  */
+  if (!msym)
+    msym = lookup_minimal_symbol ("__pthread_threads_events", NULL, NULL);
+
+  if (!msym)
+    /* No threads yet */
+    return 0;
+
+  soname = solib_name_from_address (SYMBOL_VALUE_ADDRESS (msym));
+  if (soname)
+    {
+      /* Attempt to load libthread_db from the same directory. */
+      char path[PATH_MAX], *cp;
+      strcpy (path, soname);
+      cp = strrchr (path, '/');
+      if (cp == NULL)
+	{
+	  /* Expected to get fully resolved pathname for libpthread,
+             but got something else.  Search for matching libthread_db and
+             hope there is one that matches current libpthread.  */
+	  warning (_("Cannot obtain absolute path of thread library: %s."),
+                   soname);
+	  return thread_db_load_search ();
+	}
+      strcpy (cp + 1, LIBTHREAD_DB_SO);
+      if (try_thread_db_load (path))
+	return 1;
+    }
+  return thread_db_load_search ();
+}
+
 static void
 disable_thread_event_reporting (void)
 {
@@ -593,75 +799,34 @@ void
 check_for_thread_db (void)
 {
   td_err_e err;
-  static int already_loaded;
+  static void *last_loaded;
 
   /* Do nothing if we couldn't load libthread_db.so.1.  */
-  if (td_ta_new_p == NULL)
+  if (!thread_db_load ())
     return;
 
   /* First time through, report that libthread_db was successfuly
      loaded.  Can't print this in in thread_db_load as, at that stage,
-     the interpreter and it's console haven't started.  */
+     the interpreter and it's console haven't started.
+     We track td_ta_new_p because the user may switch executables,
+     and as a result we may decide to use a different version of
+     libthread_db. */
 
-  if (!already_loaded)
+  if (last_loaded != td_ta_new_p)
     {
-      Dl_info info;
-      const char *library = NULL;
-      if (dladdr ((*td_ta_new_p), &info) != 0)
-	library = info.dli_fname;
+      last_loaded = td_ta_new_p;
 
-      /* Try dlinfo?  */
-
-      if (library == NULL)
-	/* Paranoid - don't let a NULL path slip through.  */
-	library = LIBTHREAD_DB_SO;
-
-      if (info_verbose)
-	printf_unfiltered (_("Using host libthread_db library \"%s\".\n"),
-			   library);
-      already_loaded = 1;
-    }
-
-  if (using_thread_db)
-    /* Nothing to do.  The thread library was already detected and the
-       target vector was already activated.  */
-    return;
-
-  /* Don't attempt to use thread_db on targets which can not run
-     (executables not running yet, core files) for now.  */
-  if (!target_has_execution)
-    return;
-
-  /* Don't attempt to use thread_db for remote targets.  */
-  if (!target_can_run (&current_target))
-    return;
-
-  /* Initialize the structure that identifies the child process.  */
-  proc_handle.ptid = inferior_ptid;
-
-  /* Now attempt to open a connection to the thread library.  */
-  err = td_ta_new_p (&proc_handle, &thread_agent);
-  switch (err)
-    {
-    case TD_NOLIBTHREAD:
-      /* No thread library was detected.  */
-      break;
-
-    case TD_OK:
-      printf_unfiltered (_("[Thread debugging using libthread_db enabled]\n"));
-
-      /* The thread library was detected.  Activate the thread_db target.  */
-      push_target (&thread_db_ops);
-      using_thread_db = 1;
-
-      enable_thread_event_reporting ();
-      thread_db_find_new_threads_1 ();
-      break;
-
-    default:
-      warning (_("Cannot initialize thread debugging library: %s"),
-	       thread_db_err_str (err));
-      break;
+      if (info_verbose || *libthread_db_search_path)
+        {
+          const char *library;
+
+          library = dladdr_to_soname (*td_ta_new_p);
+          if (library == NULL)
+            library = LIBTHREAD_DB_SO;
+
+          printf_unfiltered (_("Using host libthread_db library \"%s\".\n"),
+                             library);
+        }
     }
 }
 
@@ -783,7 +948,9 @@ thread_db_detach (struct target_ops *ops
 
   /* Detach thread_db target ops.  */
   unpush_target (&thread_db_ops);
-  using_thread_db = 0;
+  if (thread_db_handle)
+    dlclose (thread_db_handle);
+  thread_db_handle = NULL;
 
   target_beneath->to_detach (target_beneath, args, from_tty);
 }
@@ -896,7 +1063,9 @@ thread_db_wait (struct target_ops *ops,
     {
       remove_thread_event_breakpoints ();
       unpush_target (&thread_db_ops);
-      using_thread_db = 0;
+      if (thread_db_handle)
+	dlclose (thread_db_handle);
+      thread_db_handle = NULL;
 
       return ptid;
     }
@@ -944,7 +1113,9 @@ thread_db_mourn_inferior (struct target_
 
   /* Detach thread_db target ops.  */
   unpush_target (ops);
-  using_thread_db = 0;
+  if (thread_db_handle)
+    dlclose (thread_db_handle);
+  thread_db_handle = NULL;
 }
 
 static int
@@ -1187,13 +1358,28 @@ extern initialize_file_ftype _initialize
 void
 _initialize_thread_db (void)
 {
-  /* Only initialize the module if we can load libthread_db.  */
-  if (thread_db_load ())
-    {
-      init_thread_db_ops ();
-      add_target (&thread_db_ops);
+  init_thread_db_ops ();
+  add_target (&thread_db_ops);
 
-      /* Add ourselves to objfile event chain.  */
-      observer_attach_new_objfile (thread_db_new_objfile);
-    }
+  /* Defer loading of libthread_db.so until inferior is running.
+     This allows gdb to load correct libthread_db for a given
+     executable -- there could be mutiple versions of glibc,
+     compiled with LinuxThreads or NPTL, and until there is
+     a running inferior, we can't tell which libthread_db is
+     the correct one to load. */
+
+  libthread_db_search_path = xstrdup (LIBTHREAD_DB_SEARCH_PATH);
+
+  add_setshow_optional_filename_cmd ("libthread-db-search-path",
+				     class_support,
+				     &libthread_db_search_path, _("\
+Set search path for libthread_db."), _("\
+Show the current search path or libthread_db."), _("\
+This path is used to search for libthread_db to be loaded into \
+gdb itself."),
+			    NULL,
+			    NULL,
+			    &setlist, &showlist);
+  /* Add ourselves to objfile event chain.  */
+  observer_attach_new_objfile (thread_db_new_objfile);
 }
Index: doc/gdb.texinfo
===================================================================
RCS file: /cvs/src/src/gdb/doc/gdb.texinfo,v
retrieving revision 1.580
diff -u -p -u -r1.580 gdb.texinfo
--- doc/gdb.texinfo	15 Apr 2009 22:20:32 -0000	1.580
+++ doc/gdb.texinfo	20 Apr 2009 16:41:56 -0000
@@ -2429,6 +2429,9 @@ a command to apply a command to a list o
 @item thread-specific breakpoints
 @item @samp{set print thread-events}, which controls printing of 
 messages on thread start and exit.
+@item @samp{set libthread-db-search-path @var{path}}, which lets
+the user specify which @code{libthread_db} to use if the default choice
+isn't compatible with the program.
 @end itemize
 
 @quotation
@@ -2647,6 +2650,40 @@ programs with multiple threads.
 @xref{Set Watchpoints,,Setting Watchpoints}, for information about
 watchpoints in programs with multiple threads.
 
+@table @code
+@kindex set libthread-db-search-path
+@cindex search path for @code{libthread_db}
+@item set libthread-db-search-path @r{[}@var{path}@r{]}
+If this variable is set, @var{path} is a colon-separated list of
+directories @value{GDBN} will use to search for @code{libthread_db}.
+If you omit @var{path}, @samp{libthread-db-search-path} will be reset to
+an empty list.
+
+On @sc{gnu}/Linux and Solaris systems, @value{GDBN} uses a ``helper''
+@code{libthread_db} library to obtain information about threads in the
+inferior process.  @value{GDBN} first attempts to use
+@code{libthread_db} located in the directory from which
+@code{libpthread} was loaded in the inferior process.  If that fails,
+@value{GDBN} will use @samp{libthread-db-search-path},
+and then default system shared library directories, to find
+@code{libthread_db}.
+
+For any @code{libthread_db} library @value{GDBN} finds in above directories,
+@value{GDBN} attempts to initialize it with the current inferior process.
+If this initialization fails (which could happen because of a version
+mismatch between @code{libthread_db} and @code{libpthread}), @value{GDBN}
+will unload @code{libthread_db}, and continue with the next directory.
+If none of @code{libthread_db} libraries initialize successfully,
+@value{GDBN} will issue a warning and thread debugging will be disabled.
+
+Setting @code{libthread-db-search-path} is currently implemented
+only on some platforms.
+
+@kindex show libthread-db-search-path 
+@item show libthread-db-search-path 
+Display current libthread_db search path.
+@end table
+
 @node Processes
 @section Debugging Programs with Multiple Processes
 

Re: [patch][rfc] Allow GDB to search for the right libthread_db.so.1

[Daniel Jacobowitz]

On Mon, Apr 20, 2009 at 09:47:30AM -0700, Paul Pluzhnikov wrote:
> On Mon, Apr 20, 2009 at 6:17 AM, Daniel Jacobowitz <drow@false.org> wrote:
> 
> > I have just a few minor comments on this.  Overall, I am heartily
> > in favor of the idea - I have carried a less elegant version of this
> > feature in the Debian GDB packages for two years or so.
> 
> Could you tell a bit more about this? I thought out set up with
> multiple incompatible (WRT libthread_db) libc versions was more an
> exception than the rule.

LinuxThreads and NPTL on the same system.  It's a lot less common now
- which is why I was going to let the patch fade away instead of
merging it - but I used to need this regularly.

> It might be reasonable to override these via CFLAGS. In particular,
> we intend to override the LIBTHREAD_DB_SEARCH_PATH with an appropriate
> local default. Of course we could just as easily patch the source.

OK, makes sense.  We use configure options for that mostly, but it may
not be worth the effort here.

> >> +  if (!msym)
> >> +    msym = lookup_minimal_symbol ("__pthread_threads_events", NULL, NULL);
> >
> > Why is this symbol lookup necessary?  Is it accomplishing the same
> > thing that searching for a shared library matching "libpthread[-.]"
> > would?
> 
> The symbol lookup works for statically linked executables (which we
> also have and would like to support). Searching for libpthread solib
> doesn't.

I see.  Do you ship a libthread_db.so alongside a static executable?

-- 
Daniel Jacobowitz
CodeSourcery

Re: [patch][rfc] Allow GDB to search for the right libthread_db.so.1

[Paul Pluzhnikov]

On Mon, Apr 20, 2009 at 10:01 AM, Daniel Jacobowitz <drow@false.org> wrote:

>> Could you tell a bit more about this? I thought out set up with
>> multiple incompatible (WRT libthread_db) libc versions was more an
>> exception than the rule.
>
> LinuxThreads and NPTL on the same system.  It's a lot less common now
> - which is why I was going to let the patch fade away instead of
> merging it - but I used to need this regularly.

Oh, right.

If the executable selects LinuxThreads via something like
LD_ASSUME_KERNEL, then GDB must also be invoked with the same, or it
will bind to the wrong libthread_db. And of course "old" static
threaded binaries will continue to use LinuxThreads forever :-(

>> The symbol lookup works for statically linked executables ...
>
> I see.  Do you ship a libthread_db.so alongside a static executable?

Generally we don't ship anything to the outside world (with the
exception of Google search appliance). For our production environment,
we already have several versions of libthread_db.so installed in
known locations, and (with this patch) it's just a matter of iterating
over possible locations until the right libthread_db is found.

-- 
Paul Pluzhnikov

Re: [patch][rfc] Allow GDB to search for the right libthread_db.so.1

[Daniel Jacobowitz]

On Mon, Apr 20, 2009 at 10:19:49AM -0700, Paul Pluzhnikov wrote:
> Oh, right.
> 
> If the executable selects LinuxThreads via something like
> LD_ASSUME_KERNEL, then GDB must also be invoked with the same, or it
> will bind to the wrong libthread_db. And of course "old" static
> threaded binaries will continue to use LinuxThreads forever :-(

Right.  For a long time -static selected LinuxThreads but runtime
generally selected NPTL.  Thus the search.

> >> The symbol lookup works for statically linked executables ...
> >
> > I see.  Do you ship a libthread_db.so alongside a static executable?
> 
> Generally we don't ship anything to the outside world (with the
> exception of Google search appliance). For our production environment,
> we already have several versions of libthread_db.so installed in
> known locations, and (with this patch) it's just a matter of iterating
> over possible locations until the right libthread_db is found.

I don't see why you need the symbol list for this; shouldn't we do the
search at the same time we'd previously have pushed libthread_db?  We
went through a lot of pain to get the timing right for both dynamic
and static binaries.

-- 
Daniel Jacobowitz
CodeSourcery

Re: [patch][rfc] Allow GDB to search for the right libthread_db.so.1

[Paul Pluzhnikov]

On Mon, Apr 20, 2009 at 11:03 AM, Daniel Jacobowitz <drow@false.org> wrote:

> I don't see why you need the symbol list for this; shouldn't we do the
> search at the same time we'd previously have pushed libthread_db?

Previously, dlopen was performed at initialization time, and then
thread_db_ops was pushed when (and only when) td_ta_new returned TD_OK
(checked on attach, and on every new solib).

If td_ta_new returned TD_NOLIBTHREAD, we assumed that there are no threads
at all (at least not yet), and any other error produced a warning.

One example where this fails: static LinuxThreads exe on NPTL system:
td_ta_new returns TD_NOLIBTHREAD (which is semi-correct -- no NPTL
libpthread), so no warning is issued; but then GDB doesn't work :-(

The symbol lookup is there in the new code to firmly establish a point
where threads are known to be present, so we can issue correct warning if
at that point no working libthread_db can be found.

I don't see how to establish that "threads are present" point without
symbol lookup, nor how to keep the proper warning if we don't know for
sure whether they are.

>  We
> went through a lot of pain to get the timing right for both dynamic
> and static binaries.

Yes, it caused me a bit of trouble as well :-(
But (a variant of) the current patch has been tested in all combinations :-)

-- 
Paul Pluzhnikov

Re: [patch][rfc] Allow GDB to search for the right libthread_db.so.1

[Daniel Jacobowitz]

On Mon, Apr 20, 2009 at 12:08:54PM -0700, Paul Pluzhnikov wrote:
> Previously, dlopen was performed at initialization time, and then
> thread_db_ops was pushed when (and only when) td_ta_new returned TD_OK
> (checked on attach, and on every new solib).
> 
> If td_ta_new returned TD_NOLIBTHREAD, we assumed that there are no threads
> at all (at least not yet), and any other error produced a warning.
> 
> One example where this fails: static LinuxThreads exe on NPTL system:
> td_ta_new returns TD_NOLIBTHREAD (which is semi-correct -- no NPTL
> libpthread), so no warning is issued; but then GDB doesn't work :-(
> 
> The symbol lookup is there in the new code to firmly establish a point
> where threads are known to be present, so we can issue correct warning if
> at that point no working libthread_db can be found.

... if we have symbols.

> I don't see how to establish that "threads are present" point without
> symbol lookup, nor how to keep the proper warning if we don't know for
> sure whether they are.

If you want to generate a warning when there are threads - not just
threads, but also a thread library - and no matching libthread_db,
then I suppose this is the only way.

Personally, I'd rather ditch the warning for static binaries.  Check
the shared library list for libpthread if all available libthread_db's
have failed, and if there is one, warn then.  Any thoughts on that?

Otherwise, I won't object to what you've got.  I am just generally
dissatisfied with hard-coding a list of symbols from the
implementation.  e.g. uClibc could legitimately use different names.

-- 
Daniel Jacobowitz
CodeSourcery

Re: [patch][rfc] Allow GDB to search for the right libthread_db.so.1

[Paul Pluzhnikov]

On Wed, Apr 22, 2009 at 10:25 AM, Daniel Jacobowitz <drow@false.org> wrote:
> On Mon, Apr 20, 2009 at 12:08:54PM -0700, Paul Pluzhnikov wrote:
>
>> The symbol lookup is there in the new code to firmly establish a point
>> where threads are known to be present, so we can issue correct warning if
>> at that point no working libthread_db can be found.
>
> ... if we have symbols.

But if we don't have symbols, libthread_db will not work at all
(the way linux libthread_db finds out whether it is compatible
with the inferior: it asks GDB to lookup one of the three symbols
in this patch. If GDB says there is no such symbol, then td_ta_new
returns TD_NOLIBTHREAD).

In fact there was a patch to add a warning if we detect
stripped libpthread.so, as this appears to be a common path
to non-working GDB:
http://sourceware.org/ml/gdb-patches/2009-02/msg00232.html

The patch didn't make it, though.

> Personally, I'd rather ditch the warning for static binaries.  Check
> the shared library list for libpthread if all available libthread_db's
> have failed, and if there is one, warn then.  Any thoughts on that?
>
> Otherwise, I won't object to what you've got.  I am just generally
> dissatisfied with hard-coding a list of symbols from the
> implementation.  e.g. uClibc could legitimately use different names.

Good point.

uClibc currently uses __linuxthreads_version (and diet-libc
doesn't provide libthread_db at all, AFAICT), but that certainly
isn't guaranteed.

OTOH, uClibc could have named its pthread library libmt.so or
anything else, and we have to make some assumptions about names
anyway, or just not issue warnings at all.

I think warning only for dynamic case is a good compromise. It will
also subsume teawater patch mentioned above.

I'll send updated patch next week when I come back from vacation.

Thanks,
-- 
Paul Pluzhnikov

Re: [patch][rfc] Allow GDB to search for the right libthread_db.so.1

[Tom Tromey]

>>>>> "Paul" == Paul Pluzhnikov <ppluzhnikov@google.com> writes:

Paul> In fact there was a patch to add a warning if we detect
Paul> stripped libpthread.so, as this appears to be a common path
Paul> to non-working GDB:
Paul> http://sourceware.org/ml/gdb-patches/2009-02/msg00232.html
Paul> The patch didn't make it, though.

Did I miss the rejection, or was it just not reviewed?

I've tried to keep track of patches that have gone unreviewed.  But, I
do miss some (I missed this one).  I recommend that everybody send
pings for patches that go unanswered after some time (1 or 2 weeks).

I would like us as a group to review all submitted patches before the
release (with the option of choosing to defer a patch, but not its
review, to post-7.0).  I don't know if this is achievable, but I
thought I'd toss it out there as a goal.

Tom

Re: [patch][rfc] Allow GDB to search for the right libthread_db.so.1

[Hui Zhu]

On Thu, Apr 23, 2009 at 09:32, Tom Tromey <tromey@redhat.com> wrote:
>>>>>> "Paul" == Paul Pluzhnikov <ppluzhnikov@google.com> writes:
>
> Paul> In fact there was a patch to add a warning if we detect
> Paul> stripped libpthread.so, as this appears to be a common path
> Paul> to non-working GDB:
> Paul> http://sourceware.org/ml/gdb-patches/2009-02/msg00232.html
> Paul> The patch didn't make it, though.
>
> Did I miss the rejection, or was it just not reviewed?

It is not reviewed.

>
> I've tried to keep track of patches that have gone unreviewed.  But, I
> do miss some (I missed this one).  I recommend that everybody send
> pings for patches that go unanswered after some time (1 or 2 weeks).
>
> I would like us as a group to review all submitted patches before the
> release (with the option of choosing to defer a patch, but not its
> review, to post-7.0).  I don't know if this is achievable, but I
> thought I'd toss it out there as a goal.

Maybe we can post a mail to let everybody ping their patch that is not reviewed.


Thanks,
Hui

Re: [patch][rfc] Allow GDB to search for the right libthread_db.so.1

[Hui Zhu]

On Thu, Apr 23, 2009 at 09:10, Paul Pluzhnikov <ppluzhnikov@google.com> wrote:
> On Wed, Apr 22, 2009 at 10:25 AM, Daniel Jacobowitz <drow@false.org> wrote:
>> On Mon, Apr 20, 2009 at 12:08:54PM -0700, Paul Pluzhnikov wrote:
>>
>>> The symbol lookup is there in the new code to firmly establish a point
>>> where threads are known to be present, so we can issue correct warning if
>>> at that point no working libthread_db can be found.
>>
>> ... if we have symbols.
>
> But if we don't have symbols, libthread_db will not work at all
> (the way linux libthread_db finds out whether it is compatible
> with the inferior: it asks GDB to lookup one of the three symbols
> in this patch. If GDB says there is no such symbol, then td_ta_new
> returns TD_NOLIBTHREAD).
>
> In fact there was a patch to add a warning if we detect
> stripped libpthread.so, as this appears to be a common path
> to non-working GDB:
> http://sourceware.org/ml/gdb-patches/2009-02/msg00232.html
>
> The patch didn't make it, though.

Yes, maybe we can give user some warning and suggests in gdb when they
have trouble with multi-thread debug.

>
>> Personally, I'd rather ditch the warning for static binaries.  Check
>> the shared library list for libpthread if all available libthread_db's
>> have failed, and if there is one, warn then.  Any thoughts on that?
>>
>> Otherwise, I won't object to what you've got.  I am just generally
>> dissatisfied with hard-coding a list of symbols from the
>> implementation.  e.g. uClibc could legitimately use different names.
>
> Good point.
>
> uClibc currently uses __linuxthreads_version (and diet-libc
> doesn't provide libthread_db at all, AFAICT), but that certainly
> isn't guaranteed.
>
> OTOH, uClibc could have named its pthread library libmt.so or
> anything else, and we have to make some assumptions about names
> anyway, or just not issue warnings at all.
>
> I think warning only for dynamic case is a good compromise. It will
> also subsume teawater patch mentioned above.
>

About your patch, I think let user choice load which libthread_db is
very cool idea.
Why not let they set which file they want to load directly?

For example:
set libthread-db /xxx_dir/libxxx

Let they choice, why not let they choice everything?  :)


Thanks,
Hui

Re: [patch][rfc] Allow GDB to search for the right libthread_db.so.1

[Paul Pluzhnikov]

On Wed, Apr 22, 2009 at 11:21 PM, Hui Zhu <teawater@gmail.com> wrote:

> About your patch, I think let user choice load which libthread_db is
> very cool idea.
> Why not let they set which file they want to load directly?
>
> For example:
> set libthread-db /xxx_dir/libxxx

As I stated at the start of this thread, we have a mixture of
executables: some are linked statically, some dynamically, and
against several (incompatible WRT libthread_db) glibc versions.

I'd like GDB to work "automagically" for all such executables,
without the end user having to understand and specify exactly
which libthread_db must be loaded for each one [1].

The patch allows me to achieve that (all I need to do is provide
appropriate local definition of LIBTHREAD_DB_SEARCH_PATH).

[1] Understanding this requires the user to understand how
GDB uses libthread_db and the relationship between libpthread and
libthread_db. Also, the mapping is complicated by the fact that
64-bit GDB is sometimes used to debug 32-bit inferiors.

Cheers,
-- 
Paul Pluzhnikov

Re: [patch][rfc] Allow GDB to search for the right libthread_db.so.1

[Hui Zhu]

On Thu, Apr 23, 2009 at 15:01, Paul Pluzhnikov <ppluzhnikov@google.com> wrote:
> On Wed, Apr 22, 2009 at 11:21 PM, Hui Zhu <teawater@gmail.com> wrote:
>
>> About your patch, I think let user choice load which libthread_db is
>> very cool idea.
>> Why not let they set which file they want to load directly?
>>
>> For example:
>> set libthread-db /xxx_dir/libxxx
>
> As I stated at the start of this thread, we have a mixture of
> executables: some are linked statically, some dynamically, and
> against several (incompatible WRT libthread_db) glibc versions.
>
> I'd like GDB to work "automagically" for all such executables,
> without the end user having to understand and specify exactly
> which libthread_db must be loaded for each one [1].
>
> The patch allows me to achieve that (all I need to do is provide
> appropriate local definition of LIBTHREAD_DB_SEARCH_PATH).
>
> [1] Understanding this requires the user to understand how
> GDB uses libthread_db and the relationship between libpthread and
> libthread_db. Also, the mapping is complicated by the fact that
> 64-bit GDB is sometimes used to debug 32-bit inferiors.
>

I read your patch again, And I think let user set file name is not
conflict with your patch.
In thread_db_load_search:
+  while (*search_path)
+    {
+      const char *end = strchr (search_path, ':');
+      if (end)
+	{
+	  size_t len = end - search_path;
+          if (len + 1 + strlen (LIBTHREAD_DB_SO) + 1 > sizeof (path))
+            {
+              char *cp = xmalloc (len + 1);
+              memcpy (cp, search_path, len);
+              cp[len] = '\0';
+              warning (_("libthread_db_search_path component too long,"
+                         " ignored: %s."), cp);
+              xfree (cp);
+              search_path += len + 1;
+              continue;
+            }
+	  memcpy (path, search_path, len);
+	  path[len] = '\0';
+	  search_path += len + 1;
+	}
+      else
+	{
+          size_t len = strlen (search_path);
+
+          if (len + 1 + strlen (LIBTHREAD_DB_SO) + 1 > sizeof (path))
+            {
+              warning (_("libthread_db_search_path component too long,"
+                         " ignored: %s."), search_path);
+              break;
+            }
+	  memcpy (path, search_path, len + 1);
+	  search_path += len;
+	}

When you get a path, you can check if this is a directory.

If this is a directory, do following job.

+      strcat (path, "/");
+      strcat (path, LIBTHREAD_DB_SO);
+      if (try_thread_db_load (path))
+	{
+	  rc = 1;
+	  break;
+	}

If this is a normal file, try_thread_db_load (path) without strcat
LIBTHREAD_DB_SO.

What do you think about it?


And in thread_db_load:
+  msym = lookup_minimal_symbol ("nptl_version", NULL, NULL);
+  if (!msym)
+    msym = lookup_minimal_symbol ("__linuxthreads_version", NULL, NULL);
+
+  /* Some really old libpthread versions do not have either of the above.  */
+  if (!msym)
+    msym = lookup_minimal_symbol ("__pthread_threads_events", NULL, NULL);
+
+  if (!msym)
+    /* No threads yet */
+    return 0;

You really don't want gdb try it with libthread_db?
If in the future, this code doesn't cover everything.  And I think let
gdb try will not affect anything.  :)

Thanks,
Hui

Re: [patch][rfc] Allow GDB to search for the right libthread_db.so.1

[Hui Zhu]

In function thread_db_load, I suggest try_thread_db_load with
libthread_db_search_path first.
Cause I think if user set a directory, he must want it be loaded.


Thanks,
Hui

On Thu, Apr 23, 2009 at 16:05, Hui Zhu <teawater@gmail.com> wrote:
> On Thu, Apr 23, 2009 at 15:01, Paul Pluzhnikov <ppluzhnikov@google.com> wrote:
>> On Wed, Apr 22, 2009 at 11:21 PM, Hui Zhu <teawater@gmail.com> wrote:
>>
>>> About your patch, I think let user choice load which libthread_db is
>>> very cool idea.
>>> Why not let they set which file they want to load directly?
>>>
>>> For example:
>>> set libthread-db /xxx_dir/libxxx
>>
>> As I stated at the start of this thread, we have a mixture of
>> executables: some are linked statically, some dynamically, and
>> against several (incompatible WRT libthread_db) glibc versions.
>>
>> I'd like GDB to work "automagically" for all such executables,
>> without the end user having to understand and specify exactly
>> which libthread_db must be loaded for each one [1].
>>
>> The patch allows me to achieve that (all I need to do is provide
>> appropriate local definition of LIBTHREAD_DB_SEARCH_PATH).
>>
>> [1] Understanding this requires the user to understand how
>> GDB uses libthread_db and the relationship between libpthread and
>> libthread_db. Also, the mapping is complicated by the fact that
>> 64-bit GDB is sometimes used to debug 32-bit inferiors.
>>
>
> I read your patch again, And I think let user set file name is not
> conflict with your patch.
> In thread_db_load_search:
> +  while (*search_path)
> +    {
> +      const char *end = strchr (search_path, ':');
> +      if (end)
> +       {
> +         size_t len = end - search_path;
> +          if (len + 1 + strlen (LIBTHREAD_DB_SO) + 1 > sizeof (path))
> +            {
> +              char *cp = xmalloc (len + 1);
> +              memcpy (cp, search_path, len);
> +              cp[len] = '\0';
> +              warning (_("libthread_db_search_path component too long,"
> +                         " ignored: %s."), cp);
> +              xfree (cp);
> +              search_path += len + 1;
> +              continue;
> +            }
> +         memcpy (path, search_path, len);
> +         path[len] = '\0';
> +         search_path += len + 1;
> +       }
> +      else
> +       {
> +          size_t len = strlen (search_path);
> +
> +          if (len + 1 + strlen (LIBTHREAD_DB_SO) + 1 > sizeof (path))
> +            {
> +              warning (_("libthread_db_search_path component too long,"
> +                         " ignored: %s."), search_path);
> +              break;
> +            }
> +         memcpy (path, search_path, len + 1);
> +         search_path += len;
> +       }
>
> When you get a path, you can check if this is a directory.
>
> If this is a directory, do following job.
>
> +      strcat (path, "/");
> +      strcat (path, LIBTHREAD_DB_SO);
> +      if (try_thread_db_load (path))
> +       {
> +         rc = 1;
> +         break;
> +       }
>
> If this is a normal file, try_thread_db_load (path) without strcat
> LIBTHREAD_DB_SO.
>
> What do you think about it?
>
>
> And in thread_db_load:
> +  msym = lookup_minimal_symbol ("nptl_version", NULL, NULL);
> +  if (!msym)
> +    msym = lookup_minimal_symbol ("__linuxthreads_version", NULL, NULL);
> +
> +  /* Some really old libpthread versions do not have either of the above.  */
> +  if (!msym)
> +    msym = lookup_minimal_symbol ("__pthread_threads_events", NULL, NULL);
> +
> +  if (!msym)
> +    /* No threads yet */
> +    return 0;
>
> You really don't want gdb try it with libthread_db?
> If in the future, this code doesn't cover everything.  And I think let
> gdb try will not affect anything.  :)
>
> Thanks,
> Hui
>

Re: [patch][rfc] Allow GDB to search for the right libthread_db.so.1

[Paul Pluzhnikov]

On Thu, Apr 23, 2009 at 4:32 AM, Hui Zhu <teawater@gmail.com> wrote:
> In function thread_db_load, I suggest try_thread_db_load with
> libthread_db_search_path first.
> Cause I think if user set a directory, he must want it be loaded.

Hui made two suggestions:

1. If libthread_db_search_path is set, use it *before* trying to load
   libthread_db from the same directory where libpthread was loaded in the
   inferior.

2. Allow libthread_db_search_path contain files as well as directories.

Both are (IMHO) reasonable, though I don't particularly like either.

Implementing 1) will cause a bit of searching, because most of the time
the libthread_db which "parallels" libpthread is the right one (at least
for us), and the search is really there mostly for static executables
(which are somewhat rare here).

Implementing 2) "clouds" the meaning of libthread_db_search_path somewhat,
and it is quite unlikely (though certainlyh possible) that libthread_db
will be called anything other than libthread_db.so.1 on Linux.

Comments?

Thanks,
-- 
Paul Pluzhnikov

Re: [patch][rfc] Allow GDB to search for the right libthread_db.so.1

[Hui Zhu]

Hi Paul,

On Thu, Apr 30, 2009 at 04:30, Paul Pluzhnikov <ppluzhnikov@google.com> wrote:
> On Thu, Apr 23, 2009 at 4:32 AM, Hui Zhu <teawater@gmail.com> wrote:
>> In function thread_db_load, I suggest try_thread_db_load with
>> libthread_db_search_path first.
>> Cause I think if user set a directory, he must want it be loaded.
>
> Hui made two suggestions:
>
> 1. If libthread_db_search_path is set, use it *before* trying to load
>   libthread_db from the same directory where libpthread was loaded in the
>   inferior.
>
> 2. Allow libthread_db_search_path contain files as well as directories.

3. And in thread_db_load:
+  msym = lookup_minimal_symbol ("nptl_version", NULL, NULL);
+  if (!msym)
+    msym = lookup_minimal_symbol ("__linuxthreads_version", NULL, NULL);
+
+  /* Some really old libpthread versions do not have either of the above.  */
+  if (!msym)
+    msym = lookup_minimal_symbol ("__pthread_threads_events", NULL, NULL);
+
+  if (!msym)
+    /* No threads yet */
+    return 0;

You really don't want gdb try it with libthread_db?
If in the future, this code doesn't cover everything.  And I think let
gdb try will not affect anything.  :)


>
> Both are (IMHO) reasonable, though I don't particularly like either.
>
> Implementing 1) will cause a bit of searching, because most of the time
> the libthread_db which "parallels" libpthread is the right one (at least
> for us), and the search is really there mostly for static executables
> (which are somewhat rare here).

If user set the directory, I think he must want gdb use it directly.

>
> Implementing 2) "clouds" the meaning of libthread_db_search_path somewhat,
> and it is quite unlikely (though certainlyh possible) that libthread_db
> will be called anything other than libthread_db.so.1 on Linux.

Why not let user can set a file to be libthread_db?  Maybe this
directory doesn't have soft link, just a "libthread_db-1.0.so".


Thanks,
Hui

Re: [patch][rfc] Allow GDB to search for the right libthread_db.so.1

[Joel Brobecker]

Just my 2 cents, from an outsider's point of view:

> 1. If libthread_db_search_path is set, use it *before* trying to load
>    libthread_db from the same directory where libpthread was loaded in the
>    inferior.

From my naive perspective (never really had a need for this feature),
it seems better to check the path before checking the path that was
used by the inferior.  Otherwise, if the local host has a library
at the same location but that's different from the library used
by the inferior, we wouldn't be able to force the debugger to use
a different library, would we?

> 2. Allow libthread_db_search_path contain files as well as directories.
[...]
> Implementing 2) "clouds" the meaning of libthread_db_search_path somewhat,
> and it is quite unlikely (though certainlyh possible) that libthread_db
> will be called anything other than libthread_db.so.1 on Linux.

This one seems much less important, if at all, to me, especially if
it is unlikely that libthread_db might have a different name. And
even in this case, it's easy to create a symbolic link in a user
area, and update the path to point to that directory.

-- 
Joel

Re: [patch][rfc] Allow GDB to search for the right libthread_db.so.1

[Paul Pluzhnikov]

On Thu, Apr 30, 2009 at 11:55 AM, Joel Brobecker <brobecker@adacore.com> wrote:

>> 1. If libthread_db_search_path is set, use it *before* trying to load
>>    libthread_db from the same directory where libpthread was loaded in the
>>    inferior.
>
> From my naive perspective (never really had a need for this feature),
> it seems better to check the path before checking the path that was
> used by the inferior.  Otherwise, if the local host has a library
> at the same location but that's different from the library used
> by the inferior, we wouldn't be able to force the debugger to use
> a different library, would we?

Assume inferior is using remote:/a/b/c/libpthread.so.0.

If host has local:/a/b/c/libthread_db.so.0 which is incompatible with
remote:/a/b/c/libpthread.so.0, then
try_thread_db_load("/a/b/c/libthread_db.so.0")
will fail, and GDB will proceed to try other paths in
libthread_db_search_path.

But I see your point: if local:/a/b/c/libthread_db.so.0 "agrees"
to work with the inferior (td_ta_new returns TD_OK), yet is the wrong one
to use for some reason, there is no way for the user to force GDB to use
an alternate libthread_db.

I'll invert the order of search, get rid of symbol lookup, and send an
updated patch shortly.

Thanks,
-- 
Paul Pluzhnikov

Re: [patch][rfc] Allow GDB to search for the right libthread_db.so.1

[Doug Evans]

On Thu, Apr 30, 2009 at 12:11 PM, Paul Pluzhnikov
<ppluzhnikov@google.com> wrote:
> On Thu, Apr 30, 2009 at 11:55 AM, Joel Brobecker <brobecker@adacore.com> wrote:
>
>>> 1. If libthread_db_search_path is set, use it *before* trying to load
>>>    libthread_db from the same directory where libpthread was loaded in the
>>>    inferior.
>>
>> From my naive perspective (never really had a need for this feature),
>> it seems better to check the path before checking the path that was
>> used by the inferior.  Otherwise, if the local host has a library
>> at the same location but that's different from the library used
>> by the inferior, we wouldn't be able to force the debugger to use
>> a different library, would we?
>
> Assume inferior is using remote:/a/b/c/libpthread.so.0.
>
> If host has local:/a/b/c/libthread_db.so.0 which is incompatible with
> remote:/a/b/c/libpthread.so.0, then
> try_thread_db_load("/a/b/c/libthread_db.so.0")
> will fail, and GDB will proceed to try other paths in
> libthread_db_search_path.
>
> But I see your point: if local:/a/b/c/libthread_db.so.0 "agrees"
> to work with the inferior (td_ta_new returns TD_OK), yet is the wrong one
> to use for some reason, there is no way for the user to force GDB to use
> an alternate libthread_db.
>
> I'll invert the order of search, get rid of symbol lookup, and send an
> updated patch shortly.

One could recognize a special path that means the path used by the inferior.
That would also collapse the algorithm from "check A, then check B"
(where A and B are libthread-db-path, inferior-path in some order)
to just "check libthread-db-path".
It's kinda similar to "dir" where "dir" by itself resets the path to $cdir:$cwd,
though in this case one would always set the entire libthread-db-path
whereas the "dir" command prepends to the current path.

Just a thought.

Re: [patch][rfc] Allow GDB to search for the right libthread_db.so.1

[Paul Pluzhnikov]

[-- Attachment #1: Type: text/plain, Size: 2342 bytes --]

On Thu, Apr 30, 2009 at 12:11 PM, Paul Pluzhnikov
<ppluzhnikov@google.com> wrote:

> On Thu, Apr 30, 2009 at 11:55 AM, Joel Brobecker <brobecker@adacore.com> wrote:
>
>>> 1. If libthread_db_search_path is set, use it *before* trying to load
>>>    libthread_db from the same directory where libpthread was loaded in the
>>>    inferior.
>>
>> From my naive perspective (never really had a need for this feature),
>> it seems better to check the path before checking the path that was
>> used by the inferior.  Otherwise, if the local host has a library
>> at the same location but that's different from the library used
>> by the inferior, we wouldn't be able to force the debugger to use
>> a different library, would we?
>
> Assume inferior is using remote:/a/b/c/libpthread.so.0.

Actually, in case of remote debugging gdb will not use libthread_db at all
(gdbserver uses it).

> I'll invert the order of search, get rid of symbol lookup, and send an
> updated patch shortly.

Attached.

Note that documentation changed slightly to reflect the new search order.

Tested on Linux/x86_64, no regressions.
Also tested by running with several incompatible versions of libpthread.

Thanks,
-- 
Paul Pluzhnikov

2009-04-30  Paul Pluzhnikov  <ppluzhnikov@google.com>

	* NEWS: Mention set/show libthread-db-search-path.

	* gdb_thread_db.h (LIBTHREAD_DB_SEARCH_PATH): New define.
	(LIBTHREAD_DB_SO): Moved from linux-thread-db.c

	* linux-thread-db.c (libthread_db_search_path): New setting.
	(thread_db_handle): New variable (replaces using_thread_db).
	(try_thread_db_load_1): New function.
	(try_thread_db_load, thread_db_load_search): Likewise.
	(dladdr_to_soname): Likewise.
	(thread_db_load): Iterate over possibly multiple libthread_db's.
	(check_for_thread_db): Attempt to load new libthread_db.
	(thread_db_detach, thread_db_wait): Unload libthread_db.
	(thread_db_mourn_inferior): Likewise.
	(_initialize_thread_db): Add new libthread-db-search-path option.
	Defer loading of libthread_db to check_for_thread_db.

	* solib.c (libpthread_name_p): New function.
	(libpthread_solib_p): Call it.

	* solib.h (libpthread_name_p): New prototype.

doc/ChangeLog

2009-04-30  Paul Pluzhnikov  <ppluzhnikov@google.com>

	* gdb.texinfo (Threads): Document libthread-db-search-path.

[-- Attachment #2: gdb-thread-db.20090430.txt --]
[-- Type: text/plain, Size: 21957 bytes --]

Index: NEWS
===================================================================
RCS file: /cvs/src/src/gdb/NEWS,v
retrieving revision 1.308
diff -u -p -u -r1.308 NEWS
--- NEWS	20 Apr 2009 21:11:05 -0000	1.308
+++ NEWS	30 Apr 2009 22:08:35 -0000
@@ -288,6 +288,11 @@ show tcp connect-timeout
   with a specified timeout period; this is useful if the stub is launched
   in parallel with GDB but may not be ready to accept connections immediately.
 
+set libthread-db-search-path
+show libthread-db-search-path
+  Control list of directories which GDB will search for appropriate
+  libthread_db.
+
 * New native configurations
 
 x86/x86_64 Darwin		i[34567]86-*-darwin*
Index: gdb_thread_db.h
===================================================================
RCS file: /cvs/src/src/gdb/gdb_thread_db.h,v
retrieving revision 1.12
diff -u -p -u -r1.12 gdb_thread_db.h
--- gdb_thread_db.h	18 Mar 2009 08:51:11 -0000	1.12
+++ gdb_thread_db.h	30 Apr 2009 22:08:35 -0000
@@ -1,5 +1,14 @@
 #ifdef HAVE_THREAD_DB_H
 #include <thread_db.h>
+
+#ifndef LIBTHREAD_DB_SO
+#define LIBTHREAD_DB_SO "libthread_db.so.1"
+#endif
+
+#ifndef LIBTHREAD_DB_SEARCH_PATH
+#define LIBTHREAD_DB_SEARCH_PATH ""
+#endif
+
 #else
 
 /* Copyright (C) 1999, 2000, 2007, 2008, 2009 Free Software Foundation, Inc.
Index: linux-thread-db.c
===================================================================
RCS file: /cvs/src/src/gdb/linux-thread-db.c,v
retrieving revision 1.54
diff -u -p -u -r1.54 linux-thread-db.c
--- linux-thread-db.c	27 Feb 2009 20:34:41 -0000	1.54
+++ linux-thread-db.c	30 Apr 2009 22:08:35 -0000
@@ -26,13 +26,16 @@
 #include "gdb_thread_db.h"
 
 #include "bfd.h"
+#include "command.h"
 #include "exceptions.h"
+#include "gdbcmd.h"
 #include "gdbthread.h"
 #include "inferior.h"
 #include "symfile.h"
 #include "objfiles.h"
 #include "target.h"
 #include "regcache.h"
+#include "solib.h"
 #include "solib-svr4.h"
 #include "gdbcore.h"
 #include "observer.h"
@@ -44,10 +47,6 @@
 #include <gnu/libc-version.h>
 #endif
 
-#ifndef LIBTHREAD_DB_SO
-#define LIBTHREAD_DB_SO "libthread_db.so.1"
-#endif
-
 /* GNU/Linux libthread_db support.
 
    libthread_db is a library, provided along with libpthread.so, which
@@ -74,14 +73,17 @@
    of the ptid_t prevents thread IDs changing when libpthread is
    loaded or unloaded.  */
 
+static char *libthread_db_search_path;
+
 /* If we're running on GNU/Linux, we must explicitly attach to any new
    threads.  */
 
 /* This module's target vector.  */
 static struct target_ops thread_db_ops;
 
-/* Non-zero if we're using this module's target vector.  */
-static int using_thread_db;
+/* Handle from dlopen for libthread_db.so.  Not NULL if we're using this
+   module's target vector.  */
+static void *thread_db_handle;
 
 /* Non-zero if we have determined the signals used by the threads
    library.  */
@@ -344,7 +346,7 @@ thread_db_attach_lwp (ptid_t ptid)
   td_thrinfo_t ti;
   td_err_e err;
 
-  if (!using_thread_db)
+  if (thread_db_handle == NULL)
     return 0;
 
   /* This ptid comes from linux-nat.c, which should always fill in the
@@ -385,71 +387,6 @@ verbose_dlsym (void *handle, const char 
   return sym;
 }
 
-static int
-thread_db_load (void)
-{
-  void *handle;
-  td_err_e err;
-
-  handle = dlopen (LIBTHREAD_DB_SO, RTLD_NOW);
-  if (handle == NULL)
-    {
-      fprintf_filtered (gdb_stderr, "\n\ndlopen failed on '%s' - %s\n",
-			LIBTHREAD_DB_SO, dlerror ());
-      fprintf_filtered (gdb_stderr,
-			"GDB will not be able to debug pthreads.\n\n");
-      return 0;
-    }
-
-  /* Initialize pointers to the dynamic library functions we will use.
-     Essential functions first.  */
-
-  td_init_p = verbose_dlsym (handle, "td_init");
-  if (td_init_p == NULL)
-    return 0;
-
-  td_ta_new_p = verbose_dlsym (handle, "td_ta_new");
-  if (td_ta_new_p == NULL)
-    return 0;
-
-  td_ta_map_id2thr_p = verbose_dlsym (handle, "td_ta_map_id2thr");
-  if (td_ta_map_id2thr_p == NULL)
-    return 0;
-
-  td_ta_map_lwp2thr_p = verbose_dlsym (handle, "td_ta_map_lwp2thr");
-  if (td_ta_map_lwp2thr_p == NULL)
-    return 0;
-
-  td_ta_thr_iter_p = verbose_dlsym (handle, "td_ta_thr_iter");
-  if (td_ta_thr_iter_p == NULL)
-    return 0;
-
-  td_thr_validate_p = verbose_dlsym (handle, "td_thr_validate");
-  if (td_thr_validate_p == NULL)
-    return 0;
-
-  td_thr_get_info_p = verbose_dlsym (handle, "td_thr_get_info");
-  if (td_thr_get_info_p == NULL)
-    return 0;
-
-  /* Initialize the library.  */
-  err = td_init_p ();
-  if (err != TD_OK)
-    {
-      warning (_("Cannot initialize libthread_db: %s"), thread_db_err_str (err));
-      return 0;
-    }
-
-  /* These are not essential.  */
-  td_ta_event_addr_p = dlsym (handle, "td_ta_event_addr");
-  td_ta_set_event_p = dlsym (handle, "td_ta_set_event");
-  td_ta_event_getmsg_p = dlsym (handle, "td_ta_event_getmsg");
-  td_thr_event_enable_p = dlsym (handle, "td_thr_event_enable");
-  td_thr_tls_get_addr_p = dlsym (handle, "td_thr_tls_get_addr");
-
-  return 1;
-}
-
 static td_err_e
 enable_thread_event (td_thragent_t *thread_agent, int event, CORE_ADDR *bp)
 {
@@ -541,6 +478,278 @@ enable_thread_event_reporting (void)
     }
 }
 
+/* Attempt to initialize dlopen()ed libthread_db, described by HANDLE.
+   Return 1 on success.
+   Failure could happen if libthread_db does not have symbols we expect,
+   or when it refuses to work with the current inferior (e.g. due to
+   version mismatch between libthread_db and libpthread).  */
+
+static int
+try_thread_db_load_1 (void *handle)
+{
+  td_err_e err;
+
+  /* Initialize pointers to the dynamic library functions we will use.
+     Essential functions first.  */
+
+  td_init_p = verbose_dlsym (handle, "td_init");
+  if (td_init_p == NULL)
+    return 0;
+
+  err = td_init_p ();
+  if (err != TD_OK)
+    {
+      warning (_("Cannot initialize libthread_db: %s"), thread_db_err_str (err));
+      return 0;
+    }
+
+  td_ta_new_p = verbose_dlsym (handle, "td_ta_new");
+  if (td_ta_new_p == NULL)
+    return 0;
+
+  /* Initialize the structure that identifies the child process.  */
+  proc_handle.ptid = inferior_ptid;
+
+  /* Now attempt to open a connection to the thread library.  */
+  err = td_ta_new_p (&proc_handle, &thread_agent);
+  if (err != TD_OK)
+    {
+      td_ta_new_p = NULL;
+      if (info_verbose)
+	printf_unfiltered (_("td_ta_new failed: %s\n"),
+			   thread_db_err_str (err));
+      else
+        switch (err)
+          {
+            case TD_NOLIBTHREAD:
+#ifdef THREAD_DB_HAS_TD_VERSION
+            case TD_VERSION:
+#endif
+              /* The errors above are not unexpected and silently ignored:
+                 they just mean we haven't found correct version of
+                 libthread_db yet.  */
+              break;
+            default:
+              warning (_("td_ta_new failed: %s"), thread_db_err_str (err));
+          }
+      return 0;
+    }
+
+  td_ta_map_id2thr_p = verbose_dlsym (handle, "td_ta_map_id2thr");
+  if (td_ta_map_id2thr_p == NULL)
+    return 0;
+
+  td_ta_map_lwp2thr_p = verbose_dlsym (handle, "td_ta_map_lwp2thr");
+  if (td_ta_map_lwp2thr_p == NULL)
+    return 0;
+
+  td_ta_thr_iter_p = verbose_dlsym (handle, "td_ta_thr_iter");
+  if (td_ta_thr_iter_p == NULL)
+    return 0;
+
+  td_thr_validate_p = verbose_dlsym (handle, "td_thr_validate");
+  if (td_thr_validate_p == NULL)
+    return 0;
+
+  td_thr_get_info_p = verbose_dlsym (handle, "td_thr_get_info");
+  if (td_thr_get_info_p == NULL)
+    return 0;
+
+  /* These are not essential.  */
+  td_ta_event_addr_p = dlsym (handle, "td_ta_event_addr");
+  td_ta_set_event_p = dlsym (handle, "td_ta_set_event");
+  td_ta_event_getmsg_p = dlsym (handle, "td_ta_event_getmsg");
+  td_thr_event_enable_p = dlsym (handle, "td_thr_event_enable");
+  td_thr_tls_get_addr_p = dlsym (handle, "td_thr_tls_get_addr");
+
+  printf_unfiltered (_("[Thread debugging using libthread_db enabled]\n"));
+
+  /* The thread library was detected.  Activate the thread_db target.  */
+  push_target (&thread_db_ops);
+  thread_db_handle = handle;
+
+  enable_thread_event_reporting ();
+  thread_db_find_new_threads_1 ();
+  return 1;
+}
+
+/* Lookup a library in which given symbol resides.
+   Note: this is looking in GDB process, not in the inferior.
+   Returns library name, or NULL.  */
+
+static const char *
+dladdr_to_soname (const void *addr)
+{
+  Dl_info info;
+
+  if (dladdr (addr, &info) != 0)
+    return info.dli_fname;
+  return NULL;
+}
+
+/* Attempt to use LIBRARY as libthread_db.  LIBRARY could be absolute,
+   relative, or just LIBTHREAD_DB.  */
+
+static int
+try_thread_db_load (const char *library)
+{
+  void *handle;
+
+  if (info_verbose)
+    printf_unfiltered (_("Trying host libthread_db library: %s.\n"),
+                       library);
+  handle = dlopen (library, RTLD_NOW);
+  if (handle == NULL)
+    {
+      if (info_verbose)
+	printf_unfiltered (_("dlopen failed: %s.\n"), dlerror ());
+      return 0;
+    }
+
+  if (info_verbose && strchr (library, '/') == NULL)
+    {
+      void *td_init;
+
+      td_init = dlsym (handle, "td_init");
+      if (td_init != NULL)
+        {
+          const char *const libpath = dladdr_to_soname (td_init);
+
+          if (libpath != NULL)
+            printf_unfiltered (_("Host %s resolved to: %s.\n"),
+                               library, libpath);
+        }
+    }
+
+  if (try_thread_db_load_1 (handle))
+    return 1;
+
+  /* This library "refused" to work on current inferior.  */
+  dlclose (handle);
+  return 0;
+}
+
+
+/* Search libthread_db_search_path for libthread_db which "agrees"
+   to work on current inferior.  */
+
+static int
+thread_db_load_search ()
+{
+  char path[PATH_MAX];
+  const char *search_path = libthread_db_search_path;
+  int rc = 0;
+
+  while (*search_path)
+    {
+      const char *end = strchr (search_path, ':');
+      if (end)
+	{
+	  size_t len = end - search_path;
+          if (len + 1 + strlen (LIBTHREAD_DB_SO) + 1 > sizeof (path))
+            {
+              char *cp = xmalloc (len + 1);
+              memcpy (cp, search_path, len);
+              cp[len] = '\0';
+              warning (_("libthread_db_search_path component too long,"
+                         " ignored: %s."), cp);
+              xfree (cp);
+              search_path += len + 1;
+              continue;
+            }
+	  memcpy (path, search_path, len);
+	  path[len] = '\0';
+	  search_path += len + 1;
+	}
+      else
+	{
+          size_t len = strlen (search_path);
+
+          if (len + 1 + strlen (LIBTHREAD_DB_SO) + 1 > sizeof (path))
+            {
+              warning (_("libthread_db_search_path component too long,"
+                         " ignored: %s."), search_path);
+              break;
+            }
+	  memcpy (path, search_path, len + 1);
+	  search_path += len;
+	}
+      strcat (path, "/");
+      strcat (path, LIBTHREAD_DB_SO);
+      if (try_thread_db_load (path))
+	{
+	  rc = 1;
+	  break;
+	}
+    }
+  if (rc == 0)
+    rc = try_thread_db_load (LIBTHREAD_DB_SO);
+  return rc;
+}
+
+/* Attempt to load and initialize libthread_db.
+   Return 1 on success.
+ */
+
+static int
+thread_db_load (void)
+{
+  struct objfile *obj;
+
+  if (thread_db_handle != NULL)
+    return 1;
+
+  /* Don't attempt to use thread_db on targets which can not run
+     (executables not running yet, core files) for now.  */
+  if (!target_has_execution)
+    return 0;
+
+  /* Don't attempt to use thread_db for remote targets.  */
+  if (!target_can_run (&current_target))
+    return 0;
+
+  if (thread_db_load_search ())
+    return 1;
+
+  /* None of the libthread_db's on our search path, not the system default
+     ones worked.  If the executable is dynamically linked against
+     libpthread, try loading libthread_db from the same directory.  */
+
+  ALL_OBJFILES (obj)
+    if (libpthread_name_p (obj->name))
+      {
+	char path[PATH_MAX], *cp;
+
+	gdb_assert (strlen (obj->name) < sizeof (path));
+	strcpy (path, obj->name);
+	cp = strrchr (path, '/');
+
+	if (cp == NULL)
+	  {
+	    warning (_("Expected absolute pathname for libpthread in the"
+		       " inferior, but got %s."), path);
+	  }
+	else if (cp + 1 + strlen (LIBTHREAD_DB_SO) + 1 > path + sizeof (path))
+	  {
+	    warning (_("Unexpected: path to libpthread in the inferior is"
+		       " too long: %s"), path);
+	  }
+	else
+	  {
+	    strcpy (cp + 1, LIBTHREAD_DB_SO);
+	    if (try_thread_db_load (path))
+	      return 1;
+	  }
+	warning (_("Unable to find libthread_db matching inferior's thread"
+		   " library, thread debugging will not be available."));
+	return 0;
+    }
+  /* Either this executable isn't using libpthread at all, or it is
+     statically linked.  Since we can't easily distinguish these two cases,
+     no warning is issued.  */
+  return 0;
+}
+
 static void
 disable_thread_event_reporting (void)
 {
@@ -593,75 +802,34 @@ void
 check_for_thread_db (void)
 {
   td_err_e err;
-  static int already_loaded;
+  static void *last_loaded;
 
   /* Do nothing if we couldn't load libthread_db.so.1.  */
-  if (td_ta_new_p == NULL)
+  if (!thread_db_load ())
     return;
 
   /* First time through, report that libthread_db was successfuly
      loaded.  Can't print this in in thread_db_load as, at that stage,
-     the interpreter and it's console haven't started.  */
-
-  if (!already_loaded)
-    {
-      Dl_info info;
-      const char *library = NULL;
-      if (dladdr ((*td_ta_new_p), &info) != 0)
-	library = info.dli_fname;
-
-      /* Try dlinfo?  */
-
-      if (library == NULL)
-	/* Paranoid - don't let a NULL path slip through.  */
-	library = LIBTHREAD_DB_SO;
-
-      if (info_verbose)
-	printf_unfiltered (_("Using host libthread_db library \"%s\".\n"),
-			   library);
-      already_loaded = 1;
-    }
+     the interpreter and it's console haven't started.
+     We track td_ta_new_p because the user may switch executables,
+     and as a result we may decide to use a different version of
+     libthread_db. */
 
-  if (using_thread_db)
-    /* Nothing to do.  The thread library was already detected and the
-       target vector was already activated.  */
-    return;
-
-  /* Don't attempt to use thread_db on targets which can not run
-     (executables not running yet, core files) for now.  */
-  if (!target_has_execution)
-    return;
-
-  /* Don't attempt to use thread_db for remote targets.  */
-  if (!target_can_run (&current_target))
-    return;
-
-  /* Initialize the structure that identifies the child process.  */
-  proc_handle.ptid = inferior_ptid;
-
-  /* Now attempt to open a connection to the thread library.  */
-  err = td_ta_new_p (&proc_handle, &thread_agent);
-  switch (err)
+  if (last_loaded != td_ta_new_p)
     {
-    case TD_NOLIBTHREAD:
-      /* No thread library was detected.  */
-      break;
-
-    case TD_OK:
-      printf_unfiltered (_("[Thread debugging using libthread_db enabled]\n"));
-
-      /* The thread library was detected.  Activate the thread_db target.  */
-      push_target (&thread_db_ops);
-      using_thread_db = 1;
+      last_loaded = td_ta_new_p;
 
-      enable_thread_event_reporting ();
-      thread_db_find_new_threads_1 ();
-      break;
-
-    default:
-      warning (_("Cannot initialize thread debugging library: %s"),
-	       thread_db_err_str (err));
-      break;
+      if (info_verbose || *libthread_db_search_path)
+        {
+          const char *library;
+
+          library = dladdr_to_soname (*td_ta_new_p);
+          if (library == NULL)
+            library = LIBTHREAD_DB_SO;
+
+          printf_unfiltered (_("Using host libthread_db library \"%s\".\n"),
+                             library);
+        }
     }
 }
 
@@ -783,7 +951,9 @@ thread_db_detach (struct target_ops *ops
 
   /* Detach thread_db target ops.  */
   unpush_target (&thread_db_ops);
-  using_thread_db = 0;
+  if (thread_db_handle)
+    dlclose (thread_db_handle);
+  thread_db_handle = NULL;
 
   target_beneath->to_detach (target_beneath, args, from_tty);
 }
@@ -896,7 +1066,9 @@ thread_db_wait (struct target_ops *ops,
     {
       remove_thread_event_breakpoints ();
       unpush_target (&thread_db_ops);
-      using_thread_db = 0;
+      if (thread_db_handle)
+	dlclose (thread_db_handle);
+      thread_db_handle = NULL;
 
       return ptid;
     }
@@ -944,7 +1116,9 @@ thread_db_mourn_inferior (struct target_
 
   /* Detach thread_db target ops.  */
   unpush_target (ops);
-  using_thread_db = 0;
+  if (thread_db_handle)
+    dlclose (thread_db_handle);
+  thread_db_handle = NULL;
 }
 
 static int
@@ -1187,13 +1361,28 @@ extern initialize_file_ftype _initialize
 void
 _initialize_thread_db (void)
 {
-  /* Only initialize the module if we can load libthread_db.  */
-  if (thread_db_load ())
-    {
-      init_thread_db_ops ();
-      add_target (&thread_db_ops);
+  init_thread_db_ops ();
+  add_target (&thread_db_ops);
 
-      /* Add ourselves to objfile event chain.  */
-      observer_attach_new_objfile (thread_db_new_objfile);
-    }
+  /* Defer loading of libthread_db.so until inferior is running.
+     This allows gdb to load correct libthread_db for a given
+     executable -- there could be mutiple versions of glibc,
+     compiled with LinuxThreads or NPTL, and until there is
+     a running inferior, we can't tell which libthread_db is
+     the correct one to load. */
+
+  libthread_db_search_path = xstrdup (LIBTHREAD_DB_SEARCH_PATH);
+
+  add_setshow_optional_filename_cmd ("libthread-db-search-path",
+				     class_support,
+				     &libthread_db_search_path, _("\
+Set search path for libthread_db."), _("\
+Show the current search path or libthread_db."), _("\
+This path is used to search for libthread_db to be loaded into \
+gdb itself."),
+			    NULL,
+			    NULL,
+			    &setlist, &showlist);
+  /* Add ourselves to objfile event chain.  */
+  observer_attach_new_objfile (thread_db_new_objfile);
 }
Index: solib.c
===================================================================
RCS file: /cvs/src/src/gdb/solib.c,v
retrieving revision 1.115
diff -u -p -u -r1.115 solib.c
--- solib.c	9 Mar 2009 22:38:37 -0000	1.115
+++ solib.c	30 Apr 2009 22:08:35 -0000
@@ -684,16 +684,24 @@ update_solib_list (int from_tty, struct 
     }
 }
 
-/* Return non-zero if SO is the libpthread shared library.
+
+/* Return non-zero if NAME is the libpthread shared library.
 
    Uses a fairly simplistic heuristic approach where we check
    the file name against "/libpthread".  This can lead to false
    positives, but this should be good enough in practice.  */
 
+int libpthread_name_p (const char *name)
+{
+  return (strstr (name, "/libpthread") != NULL);
+}
+
+/* Return non-zero if SO is the libpthread shared library.  */
+
 static int
 libpthread_solib_p (struct so_list *so)
 {
-  return (strstr (so->so_name, "/libpthread") != NULL);
+  return libpthread_name_p(so->so_name);
 }
 
 /* GLOBAL FUNCTION
Index: solib.h
===================================================================
RCS file: /cvs/src/src/gdb/solib.h,v
retrieving revision 1.23
diff -u -p -u -r1.23 solib.h
--- solib.h	9 Mar 2009 22:38:37 -0000	1.23
+++ solib.h	30 Apr 2009 22:08:35 -0000
@@ -65,4 +65,8 @@ extern void no_shared_libraries (char *i
 extern void set_solib_ops (struct gdbarch *gdbarch,
 			   struct target_so_ops *new_ops);
 
+/* Return non-zero if NAME is the libpthread shared library.  */
+
+extern int libpthread_name_p (const char *name);
+
 #endif /* SOLIB_H */
Index: doc/gdb.texinfo
===================================================================
RCS file: /cvs/src/src/gdb/doc/gdb.texinfo,v
retrieving revision 1.588
diff -u -p -u -r1.588 gdb.texinfo
--- doc/gdb.texinfo	30 Apr 2009 03:24:48 -0000	1.588
+++ doc/gdb.texinfo	30 Apr 2009 22:08:36 -0000
@@ -2430,6 +2430,9 @@ a command to apply a command to a list o
 @item thread-specific breakpoints
 @item @samp{set print thread-events}, which controls printing of 
 messages on thread start and exit.
+@item @samp{set libthread-db-search-path @var{path}}, which lets
+the user specify which @code{libthread_db} to use if the default choice
+isn't compatible with the program.
 @end itemize
 
 @quotation
@@ -2648,6 +2651,38 @@ programs with multiple threads.
 @xref{Set Watchpoints,,Setting Watchpoints}, for information about
 watchpoints in programs with multiple threads.
 
+@table @code
+@kindex set libthread-db-search-path
+@cindex search path for @code{libthread_db}
+@item set libthread-db-search-path @r{[}@var{path}@r{]}
+If this variable is set, @var{path} is a colon-separated list of
+directories @value{GDBN} will use to search for @code{libthread_db}.
+If you omit @var{path}, @samp{libthread-db-search-path} will be reset to
+an empty list.
+
+On @sc{gnu}/Linux and Solaris systems, @value{GDBN} uses a ``helper''
+@code{libthread_db} library to obtain information about threads in the
+inferior process.  @value{GDBN} will use @samp{libthread-db-search-path}
+to find @code{libthread_db}.  If that fails, @value{GDBN} will continue
+with default system shared library directories, and finally the directory
+from which @code{libpthread} was loaded in the inferior process.
+
+For any @code{libthread_db} library @value{GDBN} finds in above directories,
+@value{GDBN} attempts to initialize it with the current inferior process.
+If this initialization fails (which could happen because of a version
+mismatch between @code{libthread_db} and @code{libpthread}), @value{GDBN}
+will unload @code{libthread_db}, and continue with the next directory.
+If none of @code{libthread_db} libraries initialize successfully,
+@value{GDBN} will issue a warning and thread debugging will be disabled.
+
+Setting @code{libthread-db-search-path} is currently implemented
+only on some platforms.
+
+@kindex show libthread-db-search-path 
+@item show libthread-db-search-path 
+Display current libthread_db search path.
+@end table
+
 @node Processes
 @section Debugging Programs with Multiple Processes
 

Re: [patch][rfc] Allow GDB to search for the right libthread_db.so.1

[Paul Pluzhnikov]

[-- Attachment #1: Type: text/plain, Size: 258 bytes --]

On Thu, Apr 30, 2009 at 4:18 PM, Paul Pluzhnikov <ppluzhnikov@google.com> wrote:

> Attached.

Sorry, missed "space before parenth":

< +  return libpthread_name_p(so->so_name);
> +  return libpthread_name_p (so->so_name);

Fix attached.
-- 
Paul Pluzhnikov

[-- Attachment #2: gdb-thread-db.20090430-1.txt --]
[-- Type: text/plain, Size: 21952 bytes --]

Index: NEWS
===================================================================
RCS file: /cvs/src/src/gdb/NEWS,v
retrieving revision 1.308
diff -u -p -u -r1.308 NEWS
--- NEWS	20 Apr 2009 21:11:05 -0000	1.308
+++ NEWS	1 May 2009 00:14:58 -0000
@@ -288,6 +288,11 @@ show tcp connect-timeout
   with a specified timeout period; this is useful if the stub is launched
   in parallel with GDB but may not be ready to accept connections immediately.
 
+set libthread-db-search-path
+show libthread-db-search-path
+  Control list of directories which GDB will search for appropriate
+  libthread_db.
+
 * New native configurations
 
 x86/x86_64 Darwin		i[34567]86-*-darwin*
Index: gdb_thread_db.h
===================================================================
RCS file: /cvs/src/src/gdb/gdb_thread_db.h,v
retrieving revision 1.12
diff -u -p -u -r1.12 gdb_thread_db.h
--- gdb_thread_db.h	18 Mar 2009 08:51:11 -0000	1.12
+++ gdb_thread_db.h	1 May 2009 00:14:58 -0000
@@ -1,5 +1,14 @@
 #ifdef HAVE_THREAD_DB_H
 #include <thread_db.h>
+
+#ifndef LIBTHREAD_DB_SO
+#define LIBTHREAD_DB_SO "libthread_db.so.1"
+#endif
+
+#ifndef LIBTHREAD_DB_SEARCH_PATH
+#define LIBTHREAD_DB_SEARCH_PATH ""
+#endif
+
 #else
 
 /* Copyright (C) 1999, 2000, 2007, 2008, 2009 Free Software Foundation, Inc.
Index: linux-thread-db.c
===================================================================
RCS file: /cvs/src/src/gdb/linux-thread-db.c,v
retrieving revision 1.54
diff -u -p -u -r1.54 linux-thread-db.c
--- linux-thread-db.c	27 Feb 2009 20:34:41 -0000	1.54
+++ linux-thread-db.c	1 May 2009 00:14:58 -0000
@@ -26,13 +26,16 @@
 #include "gdb_thread_db.h"
 
 #include "bfd.h"
+#include "command.h"
 #include "exceptions.h"
+#include "gdbcmd.h"
 #include "gdbthread.h"
 #include "inferior.h"
 #include "symfile.h"
 #include "objfiles.h"
 #include "target.h"
 #include "regcache.h"
+#include "solib.h"
 #include "solib-svr4.h"
 #include "gdbcore.h"
 #include "observer.h"
@@ -44,10 +47,6 @@
 #include <gnu/libc-version.h>
 #endif
 
-#ifndef LIBTHREAD_DB_SO
-#define LIBTHREAD_DB_SO "libthread_db.so.1"
-#endif
-
 /* GNU/Linux libthread_db support.
 
    libthread_db is a library, provided along with libpthread.so, which
@@ -74,14 +73,17 @@
    of the ptid_t prevents thread IDs changing when libpthread is
    loaded or unloaded.  */
 
+static char *libthread_db_search_path;
+
 /* If we're running on GNU/Linux, we must explicitly attach to any new
    threads.  */
 
 /* This module's target vector.  */
 static struct target_ops thread_db_ops;
 
-/* Non-zero if we're using this module's target vector.  */
-static int using_thread_db;
+/* Handle from dlopen for libthread_db.so.  Not NULL if we're using this
+   module's target vector.  */
+static void *thread_db_handle;
 
 /* Non-zero if we have determined the signals used by the threads
    library.  */
@@ -344,7 +346,7 @@ thread_db_attach_lwp (ptid_t ptid)
   td_thrinfo_t ti;
   td_err_e err;
 
-  if (!using_thread_db)
+  if (thread_db_handle == NULL)
     return 0;
 
   /* This ptid comes from linux-nat.c, which should always fill in the
@@ -385,71 +387,6 @@ verbose_dlsym (void *handle, const char 
   return sym;
 }
 
-static int
-thread_db_load (void)
-{
-  void *handle;
-  td_err_e err;
-
-  handle = dlopen (LIBTHREAD_DB_SO, RTLD_NOW);
-  if (handle == NULL)
-    {
-      fprintf_filtered (gdb_stderr, "\n\ndlopen failed on '%s' - %s\n",
-			LIBTHREAD_DB_SO, dlerror ());
-      fprintf_filtered (gdb_stderr,
-			"GDB will not be able to debug pthreads.\n\n");
-      return 0;
-    }
-
-  /* Initialize pointers to the dynamic library functions we will use.
-     Essential functions first.  */
-
-  td_init_p = verbose_dlsym (handle, "td_init");
-  if (td_init_p == NULL)
-    return 0;
-
-  td_ta_new_p = verbose_dlsym (handle, "td_ta_new");
-  if (td_ta_new_p == NULL)
-    return 0;
-
-  td_ta_map_id2thr_p = verbose_dlsym (handle, "td_ta_map_id2thr");
-  if (td_ta_map_id2thr_p == NULL)
-    return 0;
-
-  td_ta_map_lwp2thr_p = verbose_dlsym (handle, "td_ta_map_lwp2thr");
-  if (td_ta_map_lwp2thr_p == NULL)
-    return 0;
-
-  td_ta_thr_iter_p = verbose_dlsym (handle, "td_ta_thr_iter");
-  if (td_ta_thr_iter_p == NULL)
-    return 0;
-
-  td_thr_validate_p = verbose_dlsym (handle, "td_thr_validate");
-  if (td_thr_validate_p == NULL)
-    return 0;
-
-  td_thr_get_info_p = verbose_dlsym (handle, "td_thr_get_info");
-  if (td_thr_get_info_p == NULL)
-    return 0;
-
-  /* Initialize the library.  */
-  err = td_init_p ();
-  if (err != TD_OK)
-    {
-      warning (_("Cannot initialize libthread_db: %s"), thread_db_err_str (err));
-      return 0;
-    }
-
-  /* These are not essential.  */
-  td_ta_event_addr_p = dlsym (handle, "td_ta_event_addr");
-  td_ta_set_event_p = dlsym (handle, "td_ta_set_event");
-  td_ta_event_getmsg_p = dlsym (handle, "td_ta_event_getmsg");
-  td_thr_event_enable_p = dlsym (handle, "td_thr_event_enable");
-  td_thr_tls_get_addr_p = dlsym (handle, "td_thr_tls_get_addr");
-
-  return 1;
-}
-
 static td_err_e
 enable_thread_event (td_thragent_t *thread_agent, int event, CORE_ADDR *bp)
 {
@@ -541,6 +478,278 @@ enable_thread_event_reporting (void)
     }
 }
 
+/* Attempt to initialize dlopen()ed libthread_db, described by HANDLE.
+   Return 1 on success.
+   Failure could happen if libthread_db does not have symbols we expect,
+   or when it refuses to work with the current inferior (e.g. due to
+   version mismatch between libthread_db and libpthread).  */
+
+static int
+try_thread_db_load_1 (void *handle)
+{
+  td_err_e err;
+
+  /* Initialize pointers to the dynamic library functions we will use.
+     Essential functions first.  */
+
+  td_init_p = verbose_dlsym (handle, "td_init");
+  if (td_init_p == NULL)
+    return 0;
+
+  err = td_init_p ();
+  if (err != TD_OK)
+    {
+      warning (_("Cannot initialize libthread_db: %s"), thread_db_err_str (err));
+      return 0;
+    }
+
+  td_ta_new_p = verbose_dlsym (handle, "td_ta_new");
+  if (td_ta_new_p == NULL)
+    return 0;
+
+  /* Initialize the structure that identifies the child process.  */
+  proc_handle.ptid = inferior_ptid;
+
+  /* Now attempt to open a connection to the thread library.  */
+  err = td_ta_new_p (&proc_handle, &thread_agent);
+  if (err != TD_OK)
+    {
+      td_ta_new_p = NULL;
+      if (info_verbose)
+	printf_unfiltered (_("td_ta_new failed: %s\n"),
+			   thread_db_err_str (err));
+      else
+        switch (err)
+          {
+            case TD_NOLIBTHREAD:
+#ifdef THREAD_DB_HAS_TD_VERSION
+            case TD_VERSION:
+#endif
+              /* The errors above are not unexpected and silently ignored:
+                 they just mean we haven't found correct version of
+                 libthread_db yet.  */
+              break;
+            default:
+              warning (_("td_ta_new failed: %s"), thread_db_err_str (err));
+          }
+      return 0;
+    }
+
+  td_ta_map_id2thr_p = verbose_dlsym (handle, "td_ta_map_id2thr");
+  if (td_ta_map_id2thr_p == NULL)
+    return 0;
+
+  td_ta_map_lwp2thr_p = verbose_dlsym (handle, "td_ta_map_lwp2thr");
+  if (td_ta_map_lwp2thr_p == NULL)
+    return 0;
+
+  td_ta_thr_iter_p = verbose_dlsym (handle, "td_ta_thr_iter");
+  if (td_ta_thr_iter_p == NULL)
+    return 0;
+
+  td_thr_validate_p = verbose_dlsym (handle, "td_thr_validate");
+  if (td_thr_validate_p == NULL)
+    return 0;
+
+  td_thr_get_info_p = verbose_dlsym (handle, "td_thr_get_info");
+  if (td_thr_get_info_p == NULL)
+    return 0;
+
+  /* These are not essential.  */
+  td_ta_event_addr_p = dlsym (handle, "td_ta_event_addr");
+  td_ta_set_event_p = dlsym (handle, "td_ta_set_event");
+  td_ta_event_getmsg_p = dlsym (handle, "td_ta_event_getmsg");
+  td_thr_event_enable_p = dlsym (handle, "td_thr_event_enable");
+  td_thr_tls_get_addr_p = dlsym (handle, "td_thr_tls_get_addr");
+
+  printf_unfiltered (_("[Thread debugging using libthread_db enabled]\n"));
+
+  /* The thread library was detected.  Activate the thread_db target.  */
+  push_target (&thread_db_ops);
+  thread_db_handle = handle;
+
+  enable_thread_event_reporting ();
+  thread_db_find_new_threads_1 ();
+  return 1;
+}
+
+/* Lookup a library in which given symbol resides.
+   Note: this is looking in GDB process, not in the inferior.
+   Returns library name, or NULL.  */
+
+static const char *
+dladdr_to_soname (const void *addr)
+{
+  Dl_info info;
+
+  if (dladdr (addr, &info) != 0)
+    return info.dli_fname;
+  return NULL;
+}
+
+/* Attempt to use LIBRARY as libthread_db.  LIBRARY could be absolute,
+   relative, or just LIBTHREAD_DB.  */
+
+static int
+try_thread_db_load (const char *library)
+{
+  void *handle;
+
+  if (info_verbose)
+    printf_unfiltered (_("Trying host libthread_db library: %s.\n"),
+                       library);
+  handle = dlopen (library, RTLD_NOW);
+  if (handle == NULL)
+    {
+      if (info_verbose)
+	printf_unfiltered (_("dlopen failed: %s.\n"), dlerror ());
+      return 0;
+    }
+
+  if (info_verbose && strchr (library, '/') == NULL)
+    {
+      void *td_init;
+
+      td_init = dlsym (handle, "td_init");
+      if (td_init != NULL)
+        {
+          const char *const libpath = dladdr_to_soname (td_init);
+
+          if (libpath != NULL)
+            printf_unfiltered (_("Host %s resolved to: %s.\n"),
+                               library, libpath);
+        }
+    }
+
+  if (try_thread_db_load_1 (handle))
+    return 1;
+
+  /* This library "refused" to work on current inferior.  */
+  dlclose (handle);
+  return 0;
+}
+
+
+/* Search libthread_db_search_path for libthread_db which "agrees"
+   to work on current inferior.  */
+
+static int
+thread_db_load_search ()
+{
+  char path[PATH_MAX];
+  const char *search_path = libthread_db_search_path;
+  int rc = 0;
+
+  while (*search_path)
+    {
+      const char *end = strchr (search_path, ':');
+      if (end)
+	{
+	  size_t len = end - search_path;
+          if (len + 1 + strlen (LIBTHREAD_DB_SO) + 1 > sizeof (path))
+            {
+              char *cp = xmalloc (len + 1);
+              memcpy (cp, search_path, len);
+              cp[len] = '\0';
+              warning (_("libthread_db_search_path component too long,"
+                         " ignored: %s."), cp);
+              xfree (cp);
+              search_path += len + 1;
+              continue;
+            }
+	  memcpy (path, search_path, len);
+	  path[len] = '\0';
+	  search_path += len + 1;
+	}
+      else
+	{
+          size_t len = strlen (search_path);
+
+          if (len + 1 + strlen (LIBTHREAD_DB_SO) + 1 > sizeof (path))
+            {
+              warning (_("libthread_db_search_path component too long,"
+                         " ignored: %s."), search_path);
+              break;
+            }
+	  memcpy (path, search_path, len + 1);
+	  search_path += len;
+	}
+      strcat (path, "/");
+      strcat (path, LIBTHREAD_DB_SO);
+      if (try_thread_db_load (path))
+	{
+	  rc = 1;
+	  break;
+	}
+    }
+  if (rc == 0)
+    rc = try_thread_db_load (LIBTHREAD_DB_SO);
+  return rc;
+}
+
+/* Attempt to load and initialize libthread_db.
+   Return 1 on success.
+ */
+
+static int
+thread_db_load (void)
+{
+  struct objfile *obj;
+
+  if (thread_db_handle != NULL)
+    return 1;
+
+  /* Don't attempt to use thread_db on targets which can not run
+     (executables not running yet, core files) for now.  */
+  if (!target_has_execution)
+    return 0;
+
+  /* Don't attempt to use thread_db for remote targets.  */
+  if (!target_can_run (&current_target))
+    return 0;
+
+  if (thread_db_load_search ())
+    return 1;
+
+  /* None of the libthread_db's on our search path, not the system default
+     ones worked.  If the executable is dynamically linked against
+     libpthread, try loading libthread_db from the same directory.  */
+
+  ALL_OBJFILES (obj)
+    if (libpthread_name_p (obj->name))
+      {
+	char path[PATH_MAX], *cp;
+
+	gdb_assert (strlen (obj->name) < sizeof (path));
+	strcpy (path, obj->name);
+	cp = strrchr (path, '/');
+
+	if (cp == NULL)
+	  {
+	    warning (_("Expected absolute pathname for libpthread in the"
+		       " inferior, but got %s."), path);
+	  }
+	else if (cp + 1 + strlen (LIBTHREAD_DB_SO) + 1 > path + sizeof (path))
+	  {
+	    warning (_("Unexpected: path to libpthread in the inferior is"
+		       " too long: %s"), path);
+	  }
+	else
+	  {
+	    strcpy (cp + 1, LIBTHREAD_DB_SO);
+	    if (try_thread_db_load (path))
+	      return 1;
+	  }
+	warning (_("Unable to find libthread_db matching inferior's thread"
+		   " library, thread debugging will not be available."));
+	return 0;
+    }
+  /* Either this executable isn't using libpthread at all, or it is
+     statically linked.  Since we can't easily distinguish these two cases,
+     no warning is issued.  */
+  return 0;
+}
+
 static void
 disable_thread_event_reporting (void)
 {
@@ -593,75 +802,34 @@ void
 check_for_thread_db (void)
 {
   td_err_e err;
-  static int already_loaded;
+  static void *last_loaded;
 
   /* Do nothing if we couldn't load libthread_db.so.1.  */
-  if (td_ta_new_p == NULL)
+  if (!thread_db_load ())
     return;
 
   /* First time through, report that libthread_db was successfuly
      loaded.  Can't print this in in thread_db_load as, at that stage,
-     the interpreter and it's console haven't started.  */
-
-  if (!already_loaded)
-    {
-      Dl_info info;
-      const char *library = NULL;
-      if (dladdr ((*td_ta_new_p), &info) != 0)
-	library = info.dli_fname;
-
-      /* Try dlinfo?  */
-
-      if (library == NULL)
-	/* Paranoid - don't let a NULL path slip through.  */
-	library = LIBTHREAD_DB_SO;
-
-      if (info_verbose)
-	printf_unfiltered (_("Using host libthread_db library \"%s\".\n"),
-			   library);
-      already_loaded = 1;
-    }
+     the interpreter and it's console haven't started.
+     We track td_ta_new_p because the user may switch executables,
+     and as a result we may decide to use a different version of
+     libthread_db. */
 
-  if (using_thread_db)
-    /* Nothing to do.  The thread library was already detected and the
-       target vector was already activated.  */
-    return;
-
-  /* Don't attempt to use thread_db on targets which can not run
-     (executables not running yet, core files) for now.  */
-  if (!target_has_execution)
-    return;
-
-  /* Don't attempt to use thread_db for remote targets.  */
-  if (!target_can_run (&current_target))
-    return;
-
-  /* Initialize the structure that identifies the child process.  */
-  proc_handle.ptid = inferior_ptid;
-
-  /* Now attempt to open a connection to the thread library.  */
-  err = td_ta_new_p (&proc_handle, &thread_agent);
-  switch (err)
+  if (last_loaded != td_ta_new_p)
     {
-    case TD_NOLIBTHREAD:
-      /* No thread library was detected.  */
-      break;
-
-    case TD_OK:
-      printf_unfiltered (_("[Thread debugging using libthread_db enabled]\n"));
-
-      /* The thread library was detected.  Activate the thread_db target.  */
-      push_target (&thread_db_ops);
-      using_thread_db = 1;
+      last_loaded = td_ta_new_p;
 
-      enable_thread_event_reporting ();
-      thread_db_find_new_threads_1 ();
-      break;
-
-    default:
-      warning (_("Cannot initialize thread debugging library: %s"),
-	       thread_db_err_str (err));
-      break;
+      if (info_verbose || *libthread_db_search_path)
+        {
+          const char *library;
+
+          library = dladdr_to_soname (*td_ta_new_p);
+          if (library == NULL)
+            library = LIBTHREAD_DB_SO;
+
+          printf_unfiltered (_("Using host libthread_db library \"%s\".\n"),
+                             library);
+        }
     }
 }
 
@@ -783,7 +951,9 @@ thread_db_detach (struct target_ops *ops
 
   /* Detach thread_db target ops.  */
   unpush_target (&thread_db_ops);
-  using_thread_db = 0;
+  if (thread_db_handle)
+    dlclose (thread_db_handle);
+  thread_db_handle = NULL;
 
   target_beneath->to_detach (target_beneath, args, from_tty);
 }
@@ -896,7 +1066,9 @@ thread_db_wait (struct target_ops *ops,
     {
       remove_thread_event_breakpoints ();
       unpush_target (&thread_db_ops);
-      using_thread_db = 0;
+      if (thread_db_handle)
+	dlclose (thread_db_handle);
+      thread_db_handle = NULL;
 
       return ptid;
     }
@@ -944,7 +1116,9 @@ thread_db_mourn_inferior (struct target_
 
   /* Detach thread_db target ops.  */
   unpush_target (ops);
-  using_thread_db = 0;
+  if (thread_db_handle)
+    dlclose (thread_db_handle);
+  thread_db_handle = NULL;
 }
 
 static int
@@ -1187,13 +1361,28 @@ extern initialize_file_ftype _initialize
 void
 _initialize_thread_db (void)
 {
-  /* Only initialize the module if we can load libthread_db.  */
-  if (thread_db_load ())
-    {
-      init_thread_db_ops ();
-      add_target (&thread_db_ops);
+  init_thread_db_ops ();
+  add_target (&thread_db_ops);
 
-      /* Add ourselves to objfile event chain.  */
-      observer_attach_new_objfile (thread_db_new_objfile);
-    }
+  /* Defer loading of libthread_db.so until inferior is running.
+     This allows gdb to load correct libthread_db for a given
+     executable -- there could be mutiple versions of glibc,
+     compiled with LinuxThreads or NPTL, and until there is
+     a running inferior, we can't tell which libthread_db is
+     the correct one to load. */
+
+  libthread_db_search_path = xstrdup (LIBTHREAD_DB_SEARCH_PATH);
+
+  add_setshow_optional_filename_cmd ("libthread-db-search-path",
+				     class_support,
+				     &libthread_db_search_path, _("\
+Set search path for libthread_db."), _("\
+Show the current search path or libthread_db."), _("\
+This path is used to search for libthread_db to be loaded into \
+gdb itself."),
+			    NULL,
+			    NULL,
+			    &setlist, &showlist);
+  /* Add ourselves to objfile event chain.  */
+  observer_attach_new_objfile (thread_db_new_objfile);
 }
Index: solib.c
===================================================================
RCS file: /cvs/src/src/gdb/solib.c,v
retrieving revision 1.115
diff -u -p -u -r1.115 solib.c
--- solib.c	9 Mar 2009 22:38:37 -0000	1.115
+++ solib.c	1 May 2009 00:14:58 -0000
@@ -684,16 +684,24 @@ update_solib_list (int from_tty, struct 
     }
 }
 
-/* Return non-zero if SO is the libpthread shared library.
+
+/* Return non-zero if NAME is the libpthread shared library.
 
    Uses a fairly simplistic heuristic approach where we check
    the file name against "/libpthread".  This can lead to false
    positives, but this should be good enough in practice.  */
 
+int libpthread_name_p (const char *name)
+{
+  return (strstr (name, "/libpthread") != NULL);
+}
+
+/* Return non-zero if SO is the libpthread shared library.  */
+
 static int
 libpthread_solib_p (struct so_list *so)
 {
-  return (strstr (so->so_name, "/libpthread") != NULL);
+  return libpthread_name_p (so->so_name);
 }
 
 /* GLOBAL FUNCTION
Index: solib.h
===================================================================
RCS file: /cvs/src/src/gdb/solib.h,v
retrieving revision 1.23
diff -u -p -u -r1.23 solib.h
--- solib.h	9 Mar 2009 22:38:37 -0000	1.23
+++ solib.h	1 May 2009 00:14:58 -0000
@@ -65,4 +65,8 @@ extern void no_shared_libraries (char *i
 extern void set_solib_ops (struct gdbarch *gdbarch,
 			   struct target_so_ops *new_ops);
 
+/* Return non-zero if NAME is the libpthread shared library.  */
+
+extern int libpthread_name_p (const char *name);
+
 #endif /* SOLIB_H */
Index: doc/gdb.texinfo
===================================================================
RCS file: /cvs/src/src/gdb/doc/gdb.texinfo,v
retrieving revision 1.588
diff -u -p -u -r1.588 gdb.texinfo
--- doc/gdb.texinfo	30 Apr 2009 03:24:48 -0000	1.588
+++ doc/gdb.texinfo	1 May 2009 00:14:59 -0000
@@ -2430,6 +2430,9 @@ a command to apply a command to a list o
 @item thread-specific breakpoints
 @item @samp{set print thread-events}, which controls printing of 
 messages on thread start and exit.
+@item @samp{set libthread-db-search-path @var{path}}, which lets
+the user specify which @code{libthread_db} to use if the default choice
+isn't compatible with the program.
 @end itemize
 
 @quotation
@@ -2648,6 +2651,38 @@ programs with multiple threads.
 @xref{Set Watchpoints,,Setting Watchpoints}, for information about
 watchpoints in programs with multiple threads.
 
+@table @code
+@kindex set libthread-db-search-path
+@cindex search path for @code{libthread_db}
+@item set libthread-db-search-path @r{[}@var{path}@r{]}
+If this variable is set, @var{path} is a colon-separated list of
+directories @value{GDBN} will use to search for @code{libthread_db}.
+If you omit @var{path}, @samp{libthread-db-search-path} will be reset to
+an empty list.
+
+On @sc{gnu}/Linux and Solaris systems, @value{GDBN} uses a ``helper''
+@code{libthread_db} library to obtain information about threads in the
+inferior process.  @value{GDBN} will use @samp{libthread-db-search-path}
+to find @code{libthread_db}.  If that fails, @value{GDBN} will continue
+with default system shared library directories, and finally the directory
+from which @code{libpthread} was loaded in the inferior process.
+
+For any @code{libthread_db} library @value{GDBN} finds in above directories,
+@value{GDBN} attempts to initialize it with the current inferior process.
+If this initialization fails (which could happen because of a version
+mismatch between @code{libthread_db} and @code{libpthread}), @value{GDBN}
+will unload @code{libthread_db}, and continue with the next directory.
+If none of @code{libthread_db} libraries initialize successfully,
+@value{GDBN} will issue a warning and thread debugging will be disabled.
+
+Setting @code{libthread-db-search-path} is currently implemented
+only on some platforms.
+
+@kindex show libthread-db-search-path 
+@item show libthread-db-search-path 
+Display current libthread_db search path.
+@end table
+
 @node Processes
 @section Debugging Programs with Multiple Processes
 

Re: [patch][rfc] Allow GDB to search for the right libthread_db.so.1

[Pedro Alves]

On Friday 01 May 2009 01:19:43, Paul Pluzhnikov wrote:
> Sorry, missed "space before parenth":
> 
> < +  return libpthread_name_p(so->so_name);
> > +  return libpthread_name_p (so->so_name);
> 
> Fix attached.

On Friday 01 May 2009 01:19:43, Paul Pluzhnikov wrote:

> handle = dlopen (library, RTLD_NOW);

I wonder if making this RTLD_LAZY until you found the correct one
wouldn't make sense?

> +static int
> +thread_db_load_search ()

                         ^ (void)

  (this function could be made to use `openat' at some point, but
   gdb is already assumes PATH_MAX is largest path possible elsewhere
   anyway)

I also wonder if `set sysroot' should affect this search path: I think
not, but I'm not 100% sure.

> +int libpthread_name_p (const char *name)
> +{

      ^ function name at column 0, please.


Other that those nits, this version looks good to me.  Everyone else
has already picked on what else could have been picked on.  :-)
AFAICS, this version addressed all of Daniel's concerns.  Daniel,
what do you think?

-- 
Pedro Alves

Re: [patch][rfc] Allow GDB to search for the right libthread_db.so.1

[Paul Pluzhnikov]

[-- Attachment #1: Type: text/plain, Size: 1354 bytes --]

On Mon, May 11, 2009 at 6:12 AM, Pedro Alves <pedro@codesourcery.com> wrote:

>> handle = dlopen (library, RTLD_NOW);
>
> I wonder if making this RTLD_LAZY until you found the correct one
> wouldn't make sense?

I don't believe so.

AFAICT, the reason for RTLD_NOW is to make sure that this
libthread_db is really compatible with this GDB (doesn't require
any symbols GDB doesn't provide); and also to prevent GDB from
dying half way through with "unable to resolve symbol ...".

Both of these still apply to whatever the "final" libthread_db is
going to be.

Why would we want to dlopen(... RTLD_LAZY) and try to initialize
libthread_db if we are going to reject it as unusable in the end?

>> +static int
>> +thread_db_load_search ()
>
>                         ^ (void)

Fixed, thanks.

>  (this function could be made to use `openat' at some point, but
>   gdb is already assumes PATH_MAX is largest path possible elsewhere
>   anyway)
>
> I also wonder if `set sysroot' should affect this search path: I think
> not, but I'm not 100% sure.

I don't believe it makes sense for 'set sysroot' to affect this
search path.

>> +int libpthread_name_p (const char *name)
>> +{
>
>      ^ function name at column 0, please.

Fixed, thanks.

Anxiously waiting for Daniel's verdict now ...

-- 
Paul Pluzhnikov

[-- Attachment #2: gdb-thread-db.20090511.txt --]
[-- Type: text/plain, Size: 21960 bytes --]

Index: NEWS
===================================================================
RCS file: /cvs/src/src/gdb/NEWS,v
retrieving revision 1.310
diff -u -p -u -r1.310 NEWS
--- NEWS	6 May 2009 02:32:42 -0000	1.310
+++ NEWS	11 May 2009 17:56:13 -0000
@@ -300,6 +300,11 @@ show tcp connect-timeout
   with a specified timeout period; this is useful if the stub is launched
   in parallel with GDB but may not be ready to accept connections immediately.
 
+set libthread-db-search-path
+show libthread-db-search-path
+  Control list of directories which GDB will search for appropriate
+  libthread_db.
+
 * New native configurations
 
 x86/x86_64 Darwin		i[34567]86-*-darwin*
Index: gdb_thread_db.h
===================================================================
RCS file: /cvs/src/src/gdb/gdb_thread_db.h,v
retrieving revision 1.12
diff -u -p -u -r1.12 gdb_thread_db.h
--- gdb_thread_db.h	18 Mar 2009 08:51:11 -0000	1.12
+++ gdb_thread_db.h	11 May 2009 17:56:13 -0000
@@ -1,5 +1,14 @@
 #ifdef HAVE_THREAD_DB_H
 #include <thread_db.h>
+
+#ifndef LIBTHREAD_DB_SO
+#define LIBTHREAD_DB_SO "libthread_db.so.1"
+#endif
+
+#ifndef LIBTHREAD_DB_SEARCH_PATH
+#define LIBTHREAD_DB_SEARCH_PATH ""
+#endif
+
 #else
 
 /* Copyright (C) 1999, 2000, 2007, 2008, 2009 Free Software Foundation, Inc.
Index: linux-thread-db.c
===================================================================
RCS file: /cvs/src/src/gdb/linux-thread-db.c,v
retrieving revision 1.55
diff -u -p -u -r1.55 linux-thread-db.c
--- linux-thread-db.c	1 May 2009 22:05:46 -0000	1.55
+++ linux-thread-db.c	11 May 2009 17:56:13 -0000
@@ -26,13 +26,16 @@
 #include "gdb_thread_db.h"
 
 #include "bfd.h"
+#include "command.h"
 #include "exceptions.h"
+#include "gdbcmd.h"
 #include "gdbthread.h"
 #include "inferior.h"
 #include "symfile.h"
 #include "objfiles.h"
 #include "target.h"
 #include "regcache.h"
+#include "solib.h"
 #include "solib-svr4.h"
 #include "gdbcore.h"
 #include "observer.h"
@@ -44,10 +47,6 @@
 #include <gnu/libc-version.h>
 #endif
 
-#ifndef LIBTHREAD_DB_SO
-#define LIBTHREAD_DB_SO "libthread_db.so.1"
-#endif
-
 /* GNU/Linux libthread_db support.
 
    libthread_db is a library, provided along with libpthread.so, which
@@ -74,14 +73,17 @@
    of the ptid_t prevents thread IDs changing when libpthread is
    loaded or unloaded.  */
 
+static char *libthread_db_search_path;
+
 /* If we're running on GNU/Linux, we must explicitly attach to any new
    threads.  */
 
 /* This module's target vector.  */
 static struct target_ops thread_db_ops;
 
-/* Non-zero if we're using this module's target vector.  */
-static int using_thread_db;
+/* Handle from dlopen for libthread_db.so.  Not NULL if we're using this
+   module's target vector.  */
+static void *thread_db_handle;
 
 /* Non-zero if we have determined the signals used by the threads
    library.  */
@@ -344,7 +346,7 @@ thread_db_attach_lwp (ptid_t ptid)
   td_thrinfo_t ti;
   td_err_e err;
 
-  if (!using_thread_db)
+  if (thread_db_handle == NULL)
     return 0;
 
   /* This ptid comes from linux-nat.c, which should always fill in the
@@ -385,71 +387,6 @@ verbose_dlsym (void *handle, const char 
   return sym;
 }
 
-static int
-thread_db_load (void)
-{
-  void *handle;
-  td_err_e err;
-
-  handle = dlopen (LIBTHREAD_DB_SO, RTLD_NOW);
-  if (handle == NULL)
-    {
-      fprintf_filtered (gdb_stderr, "\n\ndlopen failed on '%s' - %s\n",
-			LIBTHREAD_DB_SO, dlerror ());
-      fprintf_filtered (gdb_stderr,
-			"GDB will not be able to debug pthreads.\n\n");
-      return 0;
-    }
-
-  /* Initialize pointers to the dynamic library functions we will use.
-     Essential functions first.  */
-
-  td_init_p = verbose_dlsym (handle, "td_init");
-  if (td_init_p == NULL)
-    return 0;
-
-  td_ta_new_p = verbose_dlsym (handle, "td_ta_new");
-  if (td_ta_new_p == NULL)
-    return 0;
-
-  td_ta_map_id2thr_p = verbose_dlsym (handle, "td_ta_map_id2thr");
-  if (td_ta_map_id2thr_p == NULL)
-    return 0;
-
-  td_ta_map_lwp2thr_p = verbose_dlsym (handle, "td_ta_map_lwp2thr");
-  if (td_ta_map_lwp2thr_p == NULL)
-    return 0;
-
-  td_ta_thr_iter_p = verbose_dlsym (handle, "td_ta_thr_iter");
-  if (td_ta_thr_iter_p == NULL)
-    return 0;
-
-  td_thr_validate_p = verbose_dlsym (handle, "td_thr_validate");
-  if (td_thr_validate_p == NULL)
-    return 0;
-
-  td_thr_get_info_p = verbose_dlsym (handle, "td_thr_get_info");
-  if (td_thr_get_info_p == NULL)
-    return 0;
-
-  /* Initialize the library.  */
-  err = td_init_p ();
-  if (err != TD_OK)
-    {
-      warning (_("Cannot initialize libthread_db: %s"), thread_db_err_str (err));
-      return 0;
-    }
-
-  /* These are not essential.  */
-  td_ta_event_addr_p = dlsym (handle, "td_ta_event_addr");
-  td_ta_set_event_p = dlsym (handle, "td_ta_set_event");
-  td_ta_event_getmsg_p = dlsym (handle, "td_ta_event_getmsg");
-  td_thr_event_enable_p = dlsym (handle, "td_thr_event_enable");
-  td_thr_tls_get_addr_p = dlsym (handle, "td_thr_tls_get_addr");
-
-  return 1;
-}
-
 static td_err_e
 enable_thread_event (td_thragent_t *thread_agent, int event, CORE_ADDR *bp)
 {
@@ -541,6 +478,278 @@ enable_thread_event_reporting (void)
     }
 }
 
+/* Attempt to initialize dlopen()ed libthread_db, described by HANDLE.
+   Return 1 on success.
+   Failure could happen if libthread_db does not have symbols we expect,
+   or when it refuses to work with the current inferior (e.g. due to
+   version mismatch between libthread_db and libpthread).  */
+
+static int
+try_thread_db_load_1 (void *handle)
+{
+  td_err_e err;
+
+  /* Initialize pointers to the dynamic library functions we will use.
+     Essential functions first.  */
+
+  td_init_p = verbose_dlsym (handle, "td_init");
+  if (td_init_p == NULL)
+    return 0;
+
+  err = td_init_p ();
+  if (err != TD_OK)
+    {
+      warning (_("Cannot initialize libthread_db: %s"), thread_db_err_str (err));
+      return 0;
+    }
+
+  td_ta_new_p = verbose_dlsym (handle, "td_ta_new");
+  if (td_ta_new_p == NULL)
+    return 0;
+
+  /* Initialize the structure that identifies the child process.  */
+  proc_handle.ptid = inferior_ptid;
+
+  /* Now attempt to open a connection to the thread library.  */
+  err = td_ta_new_p (&proc_handle, &thread_agent);
+  if (err != TD_OK)
+    {
+      td_ta_new_p = NULL;
+      if (info_verbose)
+	printf_unfiltered (_("td_ta_new failed: %s\n"),
+			   thread_db_err_str (err));
+      else
+        switch (err)
+          {
+            case TD_NOLIBTHREAD:
+#ifdef THREAD_DB_HAS_TD_VERSION
+            case TD_VERSION:
+#endif
+              /* The errors above are not unexpected and silently ignored:
+                 they just mean we haven't found correct version of
+                 libthread_db yet.  */
+              break;
+            default:
+              warning (_("td_ta_new failed: %s"), thread_db_err_str (err));
+          }
+      return 0;
+    }
+
+  td_ta_map_id2thr_p = verbose_dlsym (handle, "td_ta_map_id2thr");
+  if (td_ta_map_id2thr_p == NULL)
+    return 0;
+
+  td_ta_map_lwp2thr_p = verbose_dlsym (handle, "td_ta_map_lwp2thr");
+  if (td_ta_map_lwp2thr_p == NULL)
+    return 0;
+
+  td_ta_thr_iter_p = verbose_dlsym (handle, "td_ta_thr_iter");
+  if (td_ta_thr_iter_p == NULL)
+    return 0;
+
+  td_thr_validate_p = verbose_dlsym (handle, "td_thr_validate");
+  if (td_thr_validate_p == NULL)
+    return 0;
+
+  td_thr_get_info_p = verbose_dlsym (handle, "td_thr_get_info");
+  if (td_thr_get_info_p == NULL)
+    return 0;
+
+  /* These are not essential.  */
+  td_ta_event_addr_p = dlsym (handle, "td_ta_event_addr");
+  td_ta_set_event_p = dlsym (handle, "td_ta_set_event");
+  td_ta_event_getmsg_p = dlsym (handle, "td_ta_event_getmsg");
+  td_thr_event_enable_p = dlsym (handle, "td_thr_event_enable");
+  td_thr_tls_get_addr_p = dlsym (handle, "td_thr_tls_get_addr");
+
+  printf_unfiltered (_("[Thread debugging using libthread_db enabled]\n"));
+
+  /* The thread library was detected.  Activate the thread_db target.  */
+  push_target (&thread_db_ops);
+  thread_db_handle = handle;
+
+  enable_thread_event_reporting ();
+  thread_db_find_new_threads_1 ();
+  return 1;
+}
+
+/* Lookup a library in which given symbol resides.
+   Note: this is looking in GDB process, not in the inferior.
+   Returns library name, or NULL.  */
+
+static const char *
+dladdr_to_soname (const void *addr)
+{
+  Dl_info info;
+
+  if (dladdr (addr, &info) != 0)
+    return info.dli_fname;
+  return NULL;
+}
+
+/* Attempt to use LIBRARY as libthread_db.  LIBRARY could be absolute,
+   relative, or just LIBTHREAD_DB.  */
+
+static int
+try_thread_db_load (const char *library)
+{
+  void *handle;
+
+  if (info_verbose)
+    printf_unfiltered (_("Trying host libthread_db library: %s.\n"),
+                       library);
+  handle = dlopen (library, RTLD_NOW);
+  if (handle == NULL)
+    {
+      if (info_verbose)
+	printf_unfiltered (_("dlopen failed: %s.\n"), dlerror ());
+      return 0;
+    }
+
+  if (info_verbose && strchr (library, '/') == NULL)
+    {
+      void *td_init;
+
+      td_init = dlsym (handle, "td_init");
+      if (td_init != NULL)
+        {
+          const char *const libpath = dladdr_to_soname (td_init);
+
+          if (libpath != NULL)
+            printf_unfiltered (_("Host %s resolved to: %s.\n"),
+                               library, libpath);
+        }
+    }
+
+  if (try_thread_db_load_1 (handle))
+    return 1;
+
+  /* This library "refused" to work on current inferior.  */
+  dlclose (handle);
+  return 0;
+}
+
+
+/* Search libthread_db_search_path for libthread_db which "agrees"
+   to work on current inferior.  */
+
+static int
+thread_db_load_search (void)
+{
+  char path[PATH_MAX];
+  const char *search_path = libthread_db_search_path;
+  int rc = 0;
+
+  while (*search_path)
+    {
+      const char *end = strchr (search_path, ':');
+      if (end)
+	{
+	  size_t len = end - search_path;
+          if (len + 1 + strlen (LIBTHREAD_DB_SO) + 1 > sizeof (path))
+            {
+              char *cp = xmalloc (len + 1);
+              memcpy (cp, search_path, len);
+              cp[len] = '\0';
+              warning (_("libthread_db_search_path component too long,"
+                         " ignored: %s."), cp);
+              xfree (cp);
+              search_path += len + 1;
+              continue;
+            }
+	  memcpy (path, search_path, len);
+	  path[len] = '\0';
+	  search_path += len + 1;
+	}
+      else
+	{
+          size_t len = strlen (search_path);
+
+          if (len + 1 + strlen (LIBTHREAD_DB_SO) + 1 > sizeof (path))
+            {
+              warning (_("libthread_db_search_path component too long,"
+                         " ignored: %s."), search_path);
+              break;
+            }
+	  memcpy (path, search_path, len + 1);
+	  search_path += len;
+	}
+      strcat (path, "/");
+      strcat (path, LIBTHREAD_DB_SO);
+      if (try_thread_db_load (path))
+	{
+	  rc = 1;
+	  break;
+	}
+    }
+  if (rc == 0)
+    rc = try_thread_db_load (LIBTHREAD_DB_SO);
+  return rc;
+}
+
+/* Attempt to load and initialize libthread_db.
+   Return 1 on success.
+ */
+
+static int
+thread_db_load (void)
+{
+  struct objfile *obj;
+
+  if (thread_db_handle != NULL)
+    return 1;
+
+  /* Don't attempt to use thread_db on targets which can not run
+     (executables not running yet, core files) for now.  */
+  if (!target_has_execution)
+    return 0;
+
+  /* Don't attempt to use thread_db for remote targets.  */
+  if (!target_can_run (&current_target))
+    return 0;
+
+  if (thread_db_load_search ())
+    return 1;
+
+  /* None of the libthread_db's on our search path, not the system default
+     ones worked.  If the executable is dynamically linked against
+     libpthread, try loading libthread_db from the same directory.  */
+
+  ALL_OBJFILES (obj)
+    if (libpthread_name_p (obj->name))
+      {
+	char path[PATH_MAX], *cp;
+
+	gdb_assert (strlen (obj->name) < sizeof (path));
+	strcpy (path, obj->name);
+	cp = strrchr (path, '/');
+
+	if (cp == NULL)
+	  {
+	    warning (_("Expected absolute pathname for libpthread in the"
+		       " inferior, but got %s."), path);
+	  }
+	else if (cp + 1 + strlen (LIBTHREAD_DB_SO) + 1 > path + sizeof (path))
+	  {
+	    warning (_("Unexpected: path to libpthread in the inferior is"
+		       " too long: %s"), path);
+	  }
+	else
+	  {
+	    strcpy (cp + 1, LIBTHREAD_DB_SO);
+	    if (try_thread_db_load (path))
+	      return 1;
+	  }
+	warning (_("Unable to find libthread_db matching inferior's thread"
+		   " library, thread debugging will not be available."));
+	return 0;
+    }
+  /* Either this executable isn't using libpthread at all, or it is
+     statically linked.  Since we can't easily distinguish these two cases,
+     no warning is issued.  */
+  return 0;
+}
+
 static void
 disable_thread_event_reporting (void)
 {
@@ -593,75 +802,34 @@ void
 check_for_thread_db (void)
 {
   td_err_e err;
-  static int already_loaded;
+  static void *last_loaded;
 
   /* Do nothing if we couldn't load libthread_db.so.1.  */
-  if (td_ta_new_p == NULL)
+  if (!thread_db_load ())
     return;
 
   /* First time through, report that libthread_db was successfuly
      loaded.  Can't print this in in thread_db_load as, at that stage,
-     the interpreter and it's console haven't started.  */
-
-  if (!already_loaded)
-    {
-      Dl_info info;
-      const char *library = NULL;
-      if (dladdr ((*td_ta_new_p), &info) != 0)
-	library = info.dli_fname;
-
-      /* Try dlinfo?  */
-
-      if (library == NULL)
-	/* Paranoid - don't let a NULL path slip through.  */
-	library = LIBTHREAD_DB_SO;
-
-      if (info_verbose)
-	printf_unfiltered (_("Using host libthread_db library \"%s\".\n"),
-			   library);
-      already_loaded = 1;
-    }
+     the interpreter and it's console haven't started.
+     We track td_ta_new_p because the user may switch executables,
+     and as a result we may decide to use a different version of
+     libthread_db. */
 
-  if (using_thread_db)
-    /* Nothing to do.  The thread library was already detected and the
-       target vector was already activated.  */
-    return;
-
-  /* Don't attempt to use thread_db on targets which can not run
-     (executables not running yet, core files) for now.  */
-  if (!target_has_execution)
-    return;
-
-  /* Don't attempt to use thread_db for remote targets.  */
-  if (!target_can_run (&current_target))
-    return;
-
-  /* Initialize the structure that identifies the child process.  */
-  proc_handle.ptid = inferior_ptid;
-
-  /* Now attempt to open a connection to the thread library.  */
-  err = td_ta_new_p (&proc_handle, &thread_agent);
-  switch (err)
+  if (last_loaded != td_ta_new_p)
     {
-    case TD_NOLIBTHREAD:
-      /* No thread library was detected.  */
-      break;
-
-    case TD_OK:
-      printf_unfiltered (_("[Thread debugging using libthread_db enabled]\n"));
-
-      /* The thread library was detected.  Activate the thread_db target.  */
-      push_target (&thread_db_ops);
-      using_thread_db = 1;
+      last_loaded = td_ta_new_p;
 
-      enable_thread_event_reporting ();
-      thread_db_find_new_threads_1 ();
-      break;
-
-    default:
-      warning (_("Cannot initialize thread debugging library: %s"),
-	       thread_db_err_str (err));
-      break;
+      if (info_verbose || *libthread_db_search_path)
+        {
+          const char *library;
+
+          library = dladdr_to_soname (*td_ta_new_p);
+          if (library == NULL)
+            library = LIBTHREAD_DB_SO;
+
+          printf_unfiltered (_("Using host libthread_db library \"%s\".\n"),
+                             library);
+        }
     }
 }
 
@@ -783,7 +951,9 @@ thread_db_detach (struct target_ops *ops
 
   /* Detach thread_db target ops.  */
   unpush_target (&thread_db_ops);
-  using_thread_db = 0;
+  if (thread_db_handle)
+    dlclose (thread_db_handle);
+  thread_db_handle = NULL;
 
   target_beneath->to_detach (target_beneath, args, from_tty);
 }
@@ -896,7 +1066,9 @@ thread_db_wait (struct target_ops *ops,
     {
       remove_thread_event_breakpoints ();
       unpush_target (&thread_db_ops);
-      using_thread_db = 0;
+      if (thread_db_handle)
+	dlclose (thread_db_handle);
+      thread_db_handle = NULL;
 
       return ptid;
     }
@@ -944,7 +1116,9 @@ thread_db_mourn_inferior (struct target_
 
   /* Detach thread_db target ops.  */
   unpush_target (ops);
-  using_thread_db = 0;
+  if (thread_db_handle)
+    dlclose (thread_db_handle);
+  thread_db_handle = NULL;
 }
 
 static int
@@ -1186,13 +1360,28 @@ extern initialize_file_ftype _initialize
 void
 _initialize_thread_db (void)
 {
-  /* Only initialize the module if we can load libthread_db.  */
-  if (thread_db_load ())
-    {
-      init_thread_db_ops ();
-      add_target (&thread_db_ops);
+  init_thread_db_ops ();
+  add_target (&thread_db_ops);
 
-      /* Add ourselves to objfile event chain.  */
-      observer_attach_new_objfile (thread_db_new_objfile);
-    }
+  /* Defer loading of libthread_db.so until inferior is running.
+     This allows gdb to load correct libthread_db for a given
+     executable -- there could be mutiple versions of glibc,
+     compiled with LinuxThreads or NPTL, and until there is
+     a running inferior, we can't tell which libthread_db is
+     the correct one to load. */
+
+  libthread_db_search_path = xstrdup (LIBTHREAD_DB_SEARCH_PATH);
+
+  add_setshow_optional_filename_cmd ("libthread-db-search-path",
+				     class_support,
+				     &libthread_db_search_path, _("\
+Set search path for libthread_db."), _("\
+Show the current search path or libthread_db."), _("\
+This path is used to search for libthread_db to be loaded into \
+gdb itself."),
+			    NULL,
+			    NULL,
+			    &setlist, &showlist);
+  /* Add ourselves to objfile event chain.  */
+  observer_attach_new_objfile (thread_db_new_objfile);
 }
Index: solib.c
===================================================================
RCS file: /cvs/src/src/gdb/solib.c,v
retrieving revision 1.115
diff -u -p -u -r1.115 solib.c
--- solib.c	9 Mar 2009 22:38:37 -0000	1.115
+++ solib.c	11 May 2009 17:56:13 -0000
@@ -684,16 +684,25 @@ update_solib_list (int from_tty, struct 
     }
 }
 
-/* Return non-zero if SO is the libpthread shared library.
+
+/* Return non-zero if NAME is the libpthread shared library.
 
    Uses a fairly simplistic heuristic approach where we check
    the file name against "/libpthread".  This can lead to false
    positives, but this should be good enough in practice.  */
 
+int
+libpthread_name_p (const char *name)
+{
+  return (strstr (name, "/libpthread") != NULL);
+}
+
+/* Return non-zero if SO is the libpthread shared library.  */
+
 static int
 libpthread_solib_p (struct so_list *so)
 {
-  return (strstr (so->so_name, "/libpthread") != NULL);
+  return libpthread_name_p (so->so_name);
 }
 
 /* GLOBAL FUNCTION
Index: solib.h
===================================================================
RCS file: /cvs/src/src/gdb/solib.h,v
retrieving revision 1.23
diff -u -p -u -r1.23 solib.h
--- solib.h	9 Mar 2009 22:38:37 -0000	1.23
+++ solib.h	11 May 2009 17:56:13 -0000
@@ -65,4 +65,8 @@ extern void no_shared_libraries (char *i
 extern void set_solib_ops (struct gdbarch *gdbarch,
 			   struct target_so_ops *new_ops);
 
+/* Return non-zero if NAME is the libpthread shared library.  */
+
+extern int libpthread_name_p (const char *name);
+
 #endif /* SOLIB_H */
Index: doc/gdb.texinfo
===================================================================
RCS file: /cvs/src/src/gdb/doc/gdb.texinfo,v
retrieving revision 1.590
diff -u -p -u -r1.590 gdb.texinfo
--- doc/gdb.texinfo	8 May 2009 09:47:53 -0000	1.590
+++ doc/gdb.texinfo	11 May 2009 17:56:14 -0000
@@ -2430,6 +2430,9 @@ a command to apply a command to a list o
 @item thread-specific breakpoints
 @item @samp{set print thread-events}, which controls printing of 
 messages on thread start and exit.
+@item @samp{set libthread-db-search-path @var{path}}, which lets
+the user specify which @code{libthread_db} to use if the default choice
+isn't compatible with the program.
 @end itemize
 
 @quotation
@@ -2648,6 +2651,38 @@ programs with multiple threads.
 @xref{Set Watchpoints,,Setting Watchpoints}, for information about
 watchpoints in programs with multiple threads.
 
+@table @code
+@kindex set libthread-db-search-path
+@cindex search path for @code{libthread_db}
+@item set libthread-db-search-path @r{[}@var{path}@r{]}
+If this variable is set, @var{path} is a colon-separated list of
+directories @value{GDBN} will use to search for @code{libthread_db}.
+If you omit @var{path}, @samp{libthread-db-search-path} will be reset to
+an empty list.
+
+On @sc{gnu}/Linux and Solaris systems, @value{GDBN} uses a ``helper''
+@code{libthread_db} library to obtain information about threads in the
+inferior process.  @value{GDBN} will use @samp{libthread-db-search-path}
+to find @code{libthread_db}.  If that fails, @value{GDBN} will continue
+with default system shared library directories, and finally the directory
+from which @code{libpthread} was loaded in the inferior process.
+
+For any @code{libthread_db} library @value{GDBN} finds in above directories,
+@value{GDBN} attempts to initialize it with the current inferior process.
+If this initialization fails (which could happen because of a version
+mismatch between @code{libthread_db} and @code{libpthread}), @value{GDBN}
+will unload @code{libthread_db}, and continue with the next directory.
+If none of @code{libthread_db} libraries initialize successfully,
+@value{GDBN} will issue a warning and thread debugging will be disabled.
+
+Setting @code{libthread-db-search-path} is currently implemented
+only on some platforms.
+
+@kindex show libthread-db-search-path 
+@item show libthread-db-search-path 
+Display current libthread_db search path.
+@end table
+
 @node Processes
 @section Debugging Programs with Multiple Processes
 

Re: [patch][rfc] Allow GDB to search for the right libthread_db.so.1

[Pedro Alves]

On Monday 11 May 2009 19:08:46, Paul Pluzhnikov wrote:
> On Mon, May 11, 2009 at 6:12 AM, Pedro Alves <pedro@codesourcery.com> wrote:
> 
> >> handle = dlopen (library, RTLD_NOW);
> >
> > I wonder if making this RTLD_LAZY until you found the correct one
> > wouldn't make sense?
> 
> I don't believe so.
> 
> AFAICT, the reason for RTLD_NOW is to make sure that this
> libthread_db is really compatible with this GDB (doesn't require
> any symbols GDB doesn't provide); and also to prevent GDB from
> dying half way through with "unable to resolve symbol ...".

Yeah.

> Both of these still apply to whatever the "final" libthread_db is
> going to be.
> 
> Why would we want to dlopen(... RTLD_LAZY) and try to initialize
> libthread_db if we are going to reject it as unusable in the end?

To avoid immediate binding of random libc's, but yeah, brain f*rt.

-- 
Pedro Alves

Re: [patch][rfc] Allow GDB to search for the right libthread_db.so.1

[Hui Zhu]

1.  Could you please add a debug interface like "set debug
linux-thread 1" to let linux-thread-db.c output some debug message?
I met a lot of people have trouble with linux-multi-thread debug.  I
think it will help us a lot.  :)

2.  I still peddle my idea: let libthread-db-search-path can support
single file.  And a make a patch follow your patch. Wish you like it.
:)
---
 linux-thread-db.c |   13 +++++++++++--
 1 file changed, 11 insertions(+), 2 deletions(-)

--- a/linux-thread-db.c
+++ b/linux-thread-db.c
@@ -642,6 +642,7 @@ thread_db_load_search (void)

   while (*search_path)
     {
+      struct stat buf;
       const char *end = strchr (search_path, ':');
       if (end)
 	{
@@ -674,8 +675,16 @@ thread_db_load_search (void)
 	  memcpy (path, search_path, len + 1);
 	  search_path += len;
 	}
-      strcat (path, "/");
-      strcat (path, LIBTHREAD_DB_SO);
+      if (stat (path, &buf))
+        {
+	  warning (_("Stats the file %s failied."), path);
+	  continue;
+	}
+      if (S_ISDIR (buf.st_mode))
+        {
+          strcat (path, "/");
+          strcat (path, LIBTHREAD_DB_SO);
+	}
       if (try_thread_db_load (path))
 	{
 	  rc = 1;


And thanks for you work.  Your patch is very cool.


Thanks,
Hui



On Tue, May 12, 2009 at 02:08, Paul Pluzhnikov <ppluzhnikov@google.com> wrote:
> On Mon, May 11, 2009 at 6:12 AM, Pedro Alves <pedro@codesourcery.com> wrote:
>
>>> handle = dlopen (library, RTLD_NOW);
>>
>> I wonder if making this RTLD_LAZY until you found the correct one
>> wouldn't make sense?
>
> I don't believe so.
>
> AFAICT, the reason for RTLD_NOW is to make sure that this
> libthread_db is really compatible with this GDB (doesn't require
> any symbols GDB doesn't provide); and also to prevent GDB from
> dying half way through with "unable to resolve symbol ...".
>
> Both of these still apply to whatever the "final" libthread_db is
> going to be.
>
> Why would we want to dlopen(... RTLD_LAZY) and try to initialize
> libthread_db if we are going to reject it as unusable in the end?
>
>>> +static int
>>> +thread_db_load_search ()
>>
>>                         ^ (void)
>
> Fixed, thanks.
>
>>  (this function could be made to use `openat' at some point, but
>>   gdb is already assumes PATH_MAX is largest path possible elsewhere
>>   anyway)
>>
>> I also wonder if `set sysroot' should affect this search path: I think
>> not, but I'm not 100% sure.
>
> I don't believe it makes sense for 'set sysroot' to affect this
> search path.
>
>>> +int libpthread_name_p (const char *name)
>>> +{
>>
>>      ^ function name at column 0, please.
>
> Fixed, thanks.
>
> Anxiously waiting for Daniel's verdict now ...
>
> --
> Paul Pluzhnikov
>

Re: [patch][rfc] Allow GDB to search for the right libthread_db.so.1

[Paul Pluzhnikov]

On Tue, May 12, 2009 at 12:16 AM, Hui Zhu <teawater@gmail.com> wrote:

> 1.  Could you please add a debug interface like "set debug
> linux-thread 1" to let linux-thread-db.c output some debug message?

This will already happen with 'set verbose on':

  try_thread_db_load_1 (void *handle)
  ...
	if (info_verbose)
	  printf_unfiltered (_("td_ta_new failed: %s\n"),
			     thread_db_err_str (err));

  try_thread_db_load (const char *library)
  ...
    if (info_verbose)
      printf_unfiltered (_("Trying host libthread_db library: %s.\n"),
			 library);

etc, etc.

I don't think a separate "debug linux-thread" is really necessary,
though it is trivial to add if others "vote" for it.

> 2.  I still peddle my idea: let libthread-db-search-path can support
> single file.  And a make a patch follow your patch. Wish you like it.
> :)

I don't object to the idea per se; but I believe it really solves
a non-existant problem, and makes documentation more confusing.

> ---
>  linux-thread-db.c |   13 +++++++++++--
>  1 file changed, 11 insertions(+), 2 deletions(-)
>
> --- a/linux-thread-db.c
> +++ b/linux-thread-db.c
> @@ -642,6 +642,7 @@ thread_db_load_search (void)

Unrelated question: how did you generate above diff?
Do you maintain a separate git mirror?

Thanks,
-- 
Paul Pluzhnikov

Re: [patch][rfc] Allow GDB to search for the right libthread_db.so.1

[Hui Zhu]

On Wed, May 13, 2009 at 00:42, Paul Pluzhnikov <ppluzhnikov@google.com> wrote:
> On Tue, May 12, 2009 at 12:16 AM, Hui Zhu <teawater@gmail.com> wrote:
>
>> 1.  Could you please add a debug interface like "set debug
>> linux-thread 1" to let linux-thread-db.c output some debug message?
>
> This will already happen with 'set verbose on':
>
>  try_thread_db_load_1 (void *handle)
>  ...
>        if (info_verbose)
>          printf_unfiltered (_("td_ta_new failed: %s\n"),
>                             thread_db_err_str (err));
>
>  try_thread_db_load (const char *library)
>  ...
>    if (info_verbose)
>      printf_unfiltered (_("Trying host libthread_db library: %s.\n"),
>                         library);
>
> etc, etc.
>
> I don't think a separate "debug linux-thread" is really necessary,
> though it is trivial to add if others "vote" for it.

This part is OK with me.

>
>> 2.  I still peddle my idea: let libthread-db-search-path can support
>> single file.  And a make a patch follow your patch. Wish you like it.
>> :)
>
> I don't object to the idea per se; but I believe it really solves
> a non-existant problem, and makes documentation more confusing.
>

In a linux that use readonly filesystem and don't config tmpfs with
kernel, user will more like this function than regenerate the image
and write it to flash.

>> ---
>>  linux-thread-db.c |   13 +++++++++++--
>>  1 file changed, 11 insertions(+), 2 deletions(-)
>>
>> --- a/linux-thread-db.c
>> +++ b/linux-thread-db.c
>> @@ -642,6 +642,7 @@ thread_db_load_search (void)
>
> Unrelated question: how did you generate above diff?
> Do you maintain a separate git mirror?
>
I use quilt, a very powerful patch manage tools.

Thanks,
Hui

Re: [patch][rfc] Allow GDB to search for the right libthread_db.so.1

[Paul Pluzhnikov]

On Tue, May 12, 2009 at 7:56 PM, Hui Zhu <teawater@gmail.com> wrote:

>> I don't object to the idea per se; but I believe it really solves
>> a non-existant problem, and makes documentation more confusing.
>>
>
> In a linux that use readonly filesystem and don't config tmpfs with
> kernel, user will more like this function than regenerate the image
> and write it to flash.

Well, let's take this example. The user has generated a filesystem
on a flash, and is now debugging something natively (which is already
somewhat unlikely), and there is no writeable FS in sight.

Chances are, there is no libthread_db.so.1 anywhere in sight either.
If there is one, the user could use it by setting
libthread-db-search-path to containing directory.

The only case your proposal fixes is when libthread_db.so.1 *is*
present on the system, but for some mysterious reason is called
libsomething_else.so instead.

Under what realistic conditions would this happen?

Thanks,
-- 
Paul Pluzhnikov

Re: [patch][rfc] Allow GDB to search for the right libthread_db.so.1

[Hui Zhu]

On Wed, May 13, 2009 at 11:28, Paul Pluzhnikov <ppluzhnikov@google.com> wrote:
> On Tue, May 12, 2009 at 7:56 PM, Hui Zhu <teawater@gmail.com> wrote:
>
>>> I don't object to the idea per se; but I believe it really solves
>>> a non-existant problem, and makes documentation more confusing.
>>>
>>
>> In a linux that use readonly filesystem and don't config tmpfs with
>> kernel, user will more like this function than regenerate the image
>> and write it to flash.
>
> Well, let's take this example. The user has generated a filesystem
> on a flash, and is now debugging something natively (which is already
> somewhat unlikely), and there is no writeable FS in sight.
>
> Chances are, there is no libthread_db.so.1 anywhere in sight either.
> If there is one, the user could use it by setting
> libthread-db-search-path to containing directory.
>
> The only case your proposal fixes is when libthread_db.so.1 *is*
> present on the system, but for some mysterious reason is called
> libsomething_else.so instead.
>
> Under what realistic conditions would this happen?
>
Embeded environment.

Re: [patch][rfc] Allow GDB to search for the right libthread_db.so.1

[Daniel Jacobowitz]

On Mon, May 11, 2009 at 11:08:46AM -0700, Paul Pluzhnikov wrote:
> Anxiously waiting for Daniel's verdict now ...

Sorry, making things wait for me is unfortunate now; I've got almost
no time for GDB :-(

This seems fine to me.

-- 
Daniel Jacobowitz
CodeSourcery

Re: [patch][rfc] Allow GDB to search for the right libthread_db.so.1

[Paul Pluzhnikov]

On Fri, May 15, 2009 at 7:37 AM, Daniel Jacobowitz <drow@false.org> wrote:

> This seems fine to me.

Thanks, I've committed the gdb-thread-db.20090511.txt patch.

-- 
Paul Pluzhnikov

Re: [patch][rfc] Allow GDB to search for the right libthread_db.so.1

[Eli Zaretskii]

> Date: Thu, 30 Apr 2009 16:18:26 -0700
> From: Paul Pluzhnikov <ppluzhnikov@google.com>
> Cc: Hui Zhu <teawater@gmail.com>, tromey@redhat.com,         Thiago Jung Bauermann <bauerman@br.ibm.com>,         gdb-patches ml <gdb-patches@sourceware.org>,         Eli Zaretskii <eliz@gnu.org>
> 
> 	* NEWS: Mention set/show libthread-db-search-path.

This part is okay.

> doc/ChangeLog
> 
> 2009-04-30  Paul Pluzhnikov  <ppluzhnikov@google.com>
> 
> 	* gdb.texinfo (Threads): Document libthread-db-search-path.

This part is also okay.

> +	warning (_("Unable to find libthread_db matching inferior's thread"
> +		   " library, thread debugging will not be available."));

Is thread debugging indeed completely impossible without libthread_db?
If not, perhaps this warning (and the corresponding description in the
manual) should be amended somehow.

Thanks.

Re: [patch][rfc] Allow GDB to search for the right libthread_db.so.1

[Paul Pluzhnikov]

On Fri, May 1, 2009 at 12:20 AM, Eli Zaretskii <eliz@gnu.org> wrote:

> Is thread debugging indeed completely impossible without libthread_db?

Pretty much: GDB will not know about any threads other than the one
that is stopped or crashed; with LinuxThreads GDB will stop with SIG32
all the time in unexpected places, etc. etc.

Debugging experience for anyone but libpthread implementer will be
quite miserable.

> If not, perhaps this warning (and the corresponding description in the
> manual) should be amended somehow.

Given above, I don't believe that's warranted.

Thanks,
-- 
Paul Pluzhnikov

Re: [patch][rfc] Allow GDB to search for the right libthread_db.so.1

[Daniel Jacobowitz]

On Fri, May 01, 2009 at 08:49:07AM -0700, Paul Pluzhnikov wrote:
> On Fri, May 1, 2009 at 12:20 AM, Eli Zaretskii <eliz@gnu.org> wrote:
> 
> > Is thread debugging indeed completely impossible without libthread_db?
> 
> Pretty much: GDB will not know about any threads other than the one
> that is stopped or crashed; with LinuxThreads GDB will stop with SIG32
> all the time in unexpected places, etc. etc.
> 
> Debugging experience for anyone but libpthread implementer will be
> quite miserable.

Have you tried this?  I know it'll work better than that in gdbserver,
at least if you run the application rather than attaching.  We
recognize thread creation events from the kernel.  I don't remember
what happens in GDB, though.

-- 
Daniel Jacobowitz
CodeSourcery

Re: [patch][rfc] Allow GDB to search for the right libthread_db.so.1

[Paul Pluzhnikov]

On Fri, May 1, 2009 at 9:49 AM, Daniel Jacobowitz <drow@false.org> wrote:

> Have you tried this?

Yes. It's slightly better than what I said: GDB knows about the original
(main) thread, and the thread that crashed, but knows no other threads.

Here is a sample session.

$ cat thread-crash.c
#include <pthread.h>

void *fn(void *p)
{
    if (p) {
        char *cp = 0;
        cp[1] = 'a'; /* crash! */
    }
    sleep(60);
    return 0;
}

#define N 5
int main()
{
    pthread_t tid[N];
    int i;

    for (i = 0; i < N; ++i)
        pthread_create(tid+i, 0, fn, i == N-1 ? &i : 0);
    for (i = 0; i < N; ++i)
        pthread_join(tid[i], 0);
    return 0;
}


First with a patched gdb:

$ /home/ppluzhnikov/Archive/sourceware.org/gdb/build/gdb/gdb
./thread-crash-64-v10
GNU gdb (GDB) 6.8.50.20090430-cvs
Copyright (C) 2009 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-unknown-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
(gdb) r
[Thread debugging using libthread_db enabled]
[New Thread 0x40800950 (LWP 32344)]
[New Thread 0x41001950 (LWP 32345)]
[New Thread 0x41802950 (LWP 32346)]
[New Thread 0x42003950 (LWP 32347)]
[New Thread 0x42804950 (LWP 32348)]

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x42804950 (LWP 32348)]
0x00000000004005ab in fn (p=0x7fffffffda0c) at thread-crash.c:7
7               cp[1] = 'a'; /* crash! */
(gdb) inf thread
* 6 Thread 0x42804950 (LWP 32348)  0x00000000004005ab in fn
(p=0x7fffffffda0c) at thread-crash.c:7
  5 Thread 0x42003950 (LWP 32347)  0x00007ffff78ffb81 in nanosleep ()
from /lib/libc.so.6
  4 Thread 0x41802950 (LWP 32346)  0x00007ffff78ffb81 in nanosleep ()
from /lib/libc.so.6
  3 Thread 0x41001950 (LWP 32345)  0x00007ffff78ffb81 in nanosleep ()
from /lib/libc.so.6
  2 Thread 0x40800950 (LWP 32344)  0x00007ffff78ffb81 in nanosleep ()
from /lib/libc.so.6
  1 Thread 0x7ffff7fda6e0 (LWP 32341)  0x00007ffff7bd127f in
__lll_unlock_wake_private () from /lib/libpthread.so.0
(gdb) q


And now with current CVS Head:

$ /home/ppluzhnikov/Archive/sourceware.org/gdb/build2/gdb/gdb
./thread-crash-64-v10
GNU gdb (GDB) 6.8.50.20090501-cvs
Copyright (C) 2009 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-unknown-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
(gdb) r
warning: Cannot initialize thread debugging library: versions of
libpthread and libthread_db do not match
warning: Cannot initialize thread debugging library: versions of
libpthread and libthread_db do not match
[New LWP 32375]

Program received signal SIGSEGV, Segmentation fault.
[Switching to LWP 32375]
0x00000000004005ab in fn (p=0x7fffffffda0c) at thread-crash.c:7
7               cp[1] = 'a'; /* crash! */
(gdb) inf thread
* 2 LWP 32375  0x00000000004005ab in fn (p=0x7fffffffda0c) at thread-crash.c:7
  1 LWP 32368  0x00007ffff7939b01 in clone () from /lib/libc.so.6


-- 
Paul Pluzhnikov

Re: [patch][rfc] Allow GDB to search for the right libthread_db.so.1

[Daniel Jacobowitz]

On Fri, May 01, 2009 at 10:02:04AM -0700, Paul Pluzhnikov wrote:
> On Fri, May 1, 2009 at 9:49 AM, Daniel Jacobowitz <drow@false.org> wrote:
> 
> > Have you tried this?
> 
> Yes. It's slightly better than what I said: GDB knows about the original
> (main) thread, and the thread that crashed, but knows no other threads.

OK, thanks for checking.  An enhancement for a brave soul someday :-)

-- 
Daniel Jacobowitz
CodeSourcery

Re: [patch][rfc] Allow GDB to search for the right libthread_db.so.1

[Pedro Alves]

On Friday 01 May 2009 18:02:04, Paul Pluzhnikov wrote:
> On Fri, May 1, 2009 at 9:49 AM, Daniel Jacobowitz <drow@false.org> wrote:
> 
> > Have you tried this?
> 
> Yes. It's slightly better than what I said: GDB knows about the original
> (main) thread, and the thread that crashed, but knows no other threads.
> 

> $ /home/ppluzhnikov/Archive/sourceware.org/gdb/build2/gdb/gdb
> ./thread-crash-64-v10
> GNU gdb (GDB) 6.8.50.20090501-cvs
> Copyright (C) 2009 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
> and "show warranty" for details.
> This GDB was configured as "x86_64-unknown-linux-gnu".
> For bug reporting instructions, please see:
> <http://www.gnu.org/software/gdb/bugs/>...
> (gdb) r
> warning: Cannot initialize thread debugging library: versions of
> libpthread and libthread_db do not match
> warning: Cannot initialize thread debugging library: versions of
> libpthread and libthread_db do not match
> [New LWP 32375]
> 
> Program received signal SIGSEGV, Segmentation fault.
> [Switching to LWP 32375]
> 0x00000000004005ab in fn (p=0x7fffffffda0c) at thread-crash.c:7
> 7               cp[1] = 'a'; /* crash! */
> (gdb) inf thread
> * 2 LWP 32375  0x00000000004005ab in fn (p=0x7fffffffda0c) at thread-crash.c:7
>   1 LWP 32368  0x00007ffff7939b01 in clone () from /lib/libc.so.6

This is because even though linux_handle_extended_wait keeps track of
new cloned LWPs through PTRACE_EVENT_CLONE, it only adds them to the
lwp list.  It doesn't add them to the thread list (it does so non-stop
mode, but your example is all-stop mode).

linux-nat.c doesn't implement the target_find_new_threads callback, only
linux-thread-db.c does, but  it would be easy to write a
linux-nat.c:linux_nat_find_new_threads function that adds lwps in the
lwp_list to the thread list that aren't listed already.

The tricker case is if you had just attached to a running
multi-threaded program, and thread_db debugging fails.  There'd be
no PTRACE_EVENT_CLONE for threads that are there already, of
course, so unless there's a simpler way I don't know about, you'd
have to extract the lwp list from /proc/ info.

-- 
Pedro Alves

Re: [patch][rfc] Allow GDB to search for the right libthread_db.so.1

[Doug Evans]

On Fri, May 1, 2009 at 10:17 AM, Pedro Alves <pedro@codesourcery.com> wrote:
> The tricker case is if you had just attached to a running
> multi-threaded program, and thread_db debugging fails.  There'd be
> no PTRACE_EVENT_CLONE for threads that are there already, of
> course, so unless there's a simpler way I don't know about, you'd
> have to extract the lwp list from /proc/ info.

fwiw, I did this in a libgdbserver patch that we use here.

gdbserver needs to call back to gdb to do symbol resolution, which is
needed as part of libthread_db init, but the program that links in
libgdbserver may want to attach to all threads before gdb has
connected, so I opendir /proc/<pid>/task and read in the lwps.

Re: [patch][rfc] Allow GDB to search for the right libthread_db.so.1

[Hui Zhu]

On Fri, May 1, 2009 at 02:55, Joel Brobecker <brobecker@adacore.com> wrote:
> Just my 2 cents, from an outsider's point of view:
>
>> 1. If libthread_db_search_path is set, use it *before* trying to load
>>    libthread_db from the same directory where libpthread was loaded in the
>>    inferior.
>
> From my naive perspective (never really had a need for this feature),
> it seems better to check the path before checking the path that was
> used by the inferior.  Otherwise, if the local host has a library
> at the same location but that's different from the library used
> by the inferior, we wouldn't be able to force the debugger to use
> a different library, would we?
>
>> 2. Allow libthread_db_search_path contain files as well as directories.
> [...]
>> Implementing 2) "clouds" the meaning of libthread_db_search_path somewhat,
>> and it is quite unlikely (though certainlyh possible) that libthread_db
>> will be called anything other than libthread_db.so.1 on Linux.
>
> This one seems much less important, if at all, to me, especially if
> it is unlikely that libthread_db might have a different name. And
> even in this case, it's easy to create a symbolic link in a user
> area, and update the path to point to that directory.

If we can let user create a slink here,  User can create slink to
right lib with themselves.  Maybe they don't need set
libthread_db_search_path then.

Re: [patch][rfc] Allow GDB to search for the right libthread_db.so.1

[Paul Pluzhnikov]

On Sun, May 3, 2009 at 5:06 PM, Hui Zhu <teawater@gmail.com> wrote:

> If we can let user create a slink here,  User can create slink to
> right lib with themselves.  Maybe they don't need set
> libthread_db_search_path then.

Hui, I think you are loosing track of what this patch is for.

The goal of this patch is to allow GDB to "just work" in the presence
of multiple incompatible libthread_db's. If you remove
libthread_db_search_path, then *no matter* which symlinks you create,
GDB will not work for some executables, unless you also reset
LD_LIBRARY_PATH, or reset the symlink before each invocation, neither
of which is automatic or desireable.

Cheers,
-- 
Paul Pluzhnikov

Re: [patch][rfc] Allow GDB to search for the right libthread_db.so.1

[Hui Zhu]

On Mon, May 4, 2009 at 11:31, Paul Pluzhnikov <ppluzhnikov@google.com> wrote:
> On Sun, May 3, 2009 at 5:06 PM, Hui Zhu <teawater@gmail.com> wrote:
>
>> If we can let user create a slink here,  User can create slink to
>> right lib with themselves.  Maybe they don't need set
>> libthread_db_search_path then.
>
> Hui, I think you are loosing track of what this patch is for.
>
> The goal of this patch is to allow GDB to "just work" in the presence
> of multiple incompatible libthread_db's. If you remove
> libthread_db_search_path, then *no matter* which symlinks you create,
> GDB will not work for some executables, unless you also reset
> LD_LIBRARY_PATH, or reset the symlink before each invocation, neither
> of which is automatic or desireable.
>

Without this function, what I need to do is:
mv /lib/tls/i686/cmov/libthread_db.so.1 /lib/tls/i686/cmov/libthread_db.so.1.old
mv right_libthread_db /lib/tls/i686/cmov/libthread_db.so.1

Actually, I think libthread_db_search_path is cool function,  not
because it can't be substituted, but because it can make gdb more
powerful.
But if after we have this function, we still need:
> even in this case, it's easy to create a symbolic link in a user
> area, and update the path to point to that directory.
I don't think it is very well.

Sorry for my words is directly.

Thanks,
Hui

Re: [patch][rfc] Allow GDB to search for the right libthread_db.so.1

[Joel Brobecker]

> Actually, I think libthread_db_search_path is cool function,  not
> because it can't be substituted, but because it can make gdb more
> powerful.
> But if after we have this function, we still need:
> > even in this case, it's easy to create a symbolic link in a user
> > area, and update the path to point to that directory.
> I don't think it is very well.
> 
> Sorry for my words is directly.

The problem is that you don't necessarily have root priviledges
to do the move that you describe. You might also want to *not*
disturb the rest of the system by moving out the thread_db lib
on the host just to put the one that was used on the target.

On the other hand, my understanding is that it's unlikely that
this library will have a different name; so the feature of being
able to specify the name in the search path is only hypothetical.
And even if it happens, it's easy to create a symbolic link in
an area that you have write access to (say, your home directory),
and then point the search path to link.

That's why we're suggesting that it's not worth allowing files
in the path.

-- 
Joel

Re: [patch][rfc] Allow GDB to search for the right libthread_db.so.1

[Hui Zhu]

On Tue, May 5, 2009 at 11:38, Joel Brobecker <brobecker@adacore.com> wrote:
>> Actually, I think libthread_db_search_path is cool function,  not
>> because it can't be substituted, but because it can make gdb more
>> powerful.
>> But if after we have this function, we still need:
>> > even in this case, it's easy to create a symbolic link in a user
>> > area, and update the path to point to that directory.
>> I don't think it is very well.
>>
>> Sorry for my words is directly.
>
> The problem is that you don't necessarily have root priviledges
> to do the move that you describe. You might also want to *not*
> disturb the rest of the system by moving out the thread_db lib
> on the host just to put the one that was used on the target.

I forgot it.  :P
sudo mv /lib/tls/i686/cmov/libthread_db.so.1
/lib/tls/i686/cmov/libthread_db.so.1.old
sudo mv right_libthread_db /lib/tls/i686/cmov/libthread_db.so.1



>
> On the other hand, my understanding is that it's unlikely that
> this library will have a different name; so the feature of being
> able to specify the name in the search path is only hypothetical.
> And even if it happens, it's easy to create a symbolic link in
> an area that you have write access to (say, your home directory),
> and then point the search path to link.

If the filesystem is readonly, they will need it.

Re: [patch][rfc] Allow GDB to search for the right libthread_db.so.1

[Pedro Alves]

Hi guys, what's the status on this patch?

This conflicts (patch-wise, that is) much with one of the
base linux multi-process patches:

 http://sourceware.org/ml/gdb-patches/2009-04/msg00132.html

I'm trying to figure out how to proceed here.  I'll
happily adjust my patch to apply on top of Paul's,
provided it isn't stuck forever.  :-)

-- 
Pedro Alves

Re: [patch][rfc] Allow GDB to search for the right libthread_db.so.1

[Joel Brobecker]

> Hi guys, what's the status on this patch?

As far as I know, we are now in agreement of what the behavior
should be, but it's awaiting review...

-- 
Joel

Re: [patch][rfc] Allow GDB to search for the right libthread_db.so.1

[Paul Pluzhnikov]

[-- Attachment #1: Type: text/plain, Size: 1152 bytes --]

On Mon, Apr 20, 2009 at 9:47 AM, Paul Pluzhnikov <ppluzhnikov@google.com> wrote:

> Attached is a revised patch that fixes all the comments so far.

Sorry, missed the NEWS update.

Thanks,
-- 
Paul Pluzhnikov

2009-04-20  Paul Pluzhnikov  <ppluzhnikov@google.com>

	* NEWS: Mention set/show libthread-db-search-path.

	* gdb_thread_db.h (LIBTHREAD_DB_SEARCH_PATH): New define.
	(LIBTHREAD_DB_SO): Moved from linux-thread-db.c

	* linux-thread-db.c (libthread_db_search_path): New setting.
	(thread_db_handle): New variable (replaces using_thread_db).
	(try_thread_db_load_1): New function.
	(try_thread_db_load, thread_db_load_search): Likewise.
	(dladdr_to_soname): Likewise.
	(thread_db_load): Iterate over possibly multiple libthread_db's.
	(check_for_thread_db): Attempt to load new libthread_db.
	(thread_db_detach, thread_db_wait): Unload libthread_db.
	(thread_db_mourn_inferior): Likewise.
	(_initialize_thread_db): Add new libthread-db-search-path option.
	Defer loading of libthread_db to check_for_thread_db.

doc/ChangeLog

2009-04-20  Paul Pluzhnikov  <ppluzhnikov@google.com>

	* gdb.texinfo (Threads): Document libthread-db-search-path.

[-- Attachment #2: gdb-thread-db.20090420-1.txt --]
[-- Type: text/plain, Size: 20509 bytes --]

Index: NEWS
===================================================================
RCS file: /cvs/src/src/gdb/NEWS,v
retrieving revision 1.307
diff -u -p -u -r1.307 NEWS
--- NEWS	19 Apr 2009 20:05:23 -0000	1.307
+++ NEWS	20 Apr 2009 17:35:40 -0000
@@ -284,6 +284,11 @@ show tcp connect-timeout
   with a specified timeout period; this is useful if the stub is launched
   in parallel with GDB but may not be ready to accept connections immediately.
 
+set libthread-db-search-path
+show libthread-db-search-path
+  Control list of directories which GDB will search for appropriate
+  libthread_db.
+
 * New native configurations
 
 x86/x86_64 Darwin		i[34567]86-*-darwin*
Index: gdb_thread_db.h
===================================================================
RCS file: /cvs/src/src/gdb/gdb_thread_db.h,v
retrieving revision 1.12
diff -u -p -u -r1.12 gdb_thread_db.h
--- gdb_thread_db.h	18 Mar 2009 08:51:11 -0000	1.12
+++ gdb_thread_db.h	20 Apr 2009 17:35:40 -0000
@@ -1,5 +1,14 @@
 #ifdef HAVE_THREAD_DB_H
 #include <thread_db.h>
+
+#ifndef LIBTHREAD_DB_SO
+#define LIBTHREAD_DB_SO "libthread_db.so.1"
+#endif
+
+#ifndef LIBTHREAD_DB_SEARCH_PATH
+#define LIBTHREAD_DB_SEARCH_PATH ""
+#endif
+
 #else
 
 /* Copyright (C) 1999, 2000, 2007, 2008, 2009 Free Software Foundation, Inc.
Index: linux-thread-db.c
===================================================================
RCS file: /cvs/src/src/gdb/linux-thread-db.c,v
retrieving revision 1.54
diff -u -p -u -r1.54 linux-thread-db.c
--- linux-thread-db.c	27 Feb 2009 20:34:41 -0000	1.54
+++ linux-thread-db.c	20 Apr 2009 17:35:40 -0000
@@ -26,13 +26,16 @@
 #include "gdb_thread_db.h"
 
 #include "bfd.h"
+#include "command.h"
 #include "exceptions.h"
+#include "gdbcmd.h"
 #include "gdbthread.h"
 #include "inferior.h"
 #include "symfile.h"
 #include "objfiles.h"
 #include "target.h"
 #include "regcache.h"
+#include "solib.h"
 #include "solib-svr4.h"
 #include "gdbcore.h"
 #include "observer.h"
@@ -44,10 +47,6 @@
 #include <gnu/libc-version.h>
 #endif
 
-#ifndef LIBTHREAD_DB_SO
-#define LIBTHREAD_DB_SO "libthread_db.so.1"
-#endif
-
 /* GNU/Linux libthread_db support.
 
    libthread_db is a library, provided along with libpthread.so, which
@@ -74,14 +73,17 @@
    of the ptid_t prevents thread IDs changing when libpthread is
    loaded or unloaded.  */
 
+static char *libthread_db_search_path;
+
 /* If we're running on GNU/Linux, we must explicitly attach to any new
    threads.  */
 
 /* This module's target vector.  */
 static struct target_ops thread_db_ops;
 
-/* Non-zero if we're using this module's target vector.  */
-static int using_thread_db;
+/* Handle from dlopen for libthread_db.so.  Not NULL if we're using this
+   module's target vector.  */
+static void *thread_db_handle;
 
 /* Non-zero if we have determined the signals used by the threads
    library.  */
@@ -344,7 +346,7 @@ thread_db_attach_lwp (ptid_t ptid)
   td_thrinfo_t ti;
   td_err_e err;
 
-  if (!using_thread_db)
+  if (thread_db_handle == NULL)
     return 0;
 
   /* This ptid comes from linux-nat.c, which should always fill in the
@@ -385,71 +387,6 @@ verbose_dlsym (void *handle, const char 
   return sym;
 }
 
-static int
-thread_db_load (void)
-{
-  void *handle;
-  td_err_e err;
-
-  handle = dlopen (LIBTHREAD_DB_SO, RTLD_NOW);
-  if (handle == NULL)
-    {
-      fprintf_filtered (gdb_stderr, "\n\ndlopen failed on '%s' - %s\n",
-			LIBTHREAD_DB_SO, dlerror ());
-      fprintf_filtered (gdb_stderr,
-			"GDB will not be able to debug pthreads.\n\n");
-      return 0;
-    }
-
-  /* Initialize pointers to the dynamic library functions we will use.
-     Essential functions first.  */
-
-  td_init_p = verbose_dlsym (handle, "td_init");
-  if (td_init_p == NULL)
-    return 0;
-
-  td_ta_new_p = verbose_dlsym (handle, "td_ta_new");
-  if (td_ta_new_p == NULL)
-    return 0;
-
-  td_ta_map_id2thr_p = verbose_dlsym (handle, "td_ta_map_id2thr");
-  if (td_ta_map_id2thr_p == NULL)
-    return 0;
-
-  td_ta_map_lwp2thr_p = verbose_dlsym (handle, "td_ta_map_lwp2thr");
-  if (td_ta_map_lwp2thr_p == NULL)
-    return 0;
-
-  td_ta_thr_iter_p = verbose_dlsym (handle, "td_ta_thr_iter");
-  if (td_ta_thr_iter_p == NULL)
-    return 0;
-
-  td_thr_validate_p = verbose_dlsym (handle, "td_thr_validate");
-  if (td_thr_validate_p == NULL)
-    return 0;
-
-  td_thr_get_info_p = verbose_dlsym (handle, "td_thr_get_info");
-  if (td_thr_get_info_p == NULL)
-    return 0;
-
-  /* Initialize the library.  */
-  err = td_init_p ();
-  if (err != TD_OK)
-    {
-      warning (_("Cannot initialize libthread_db: %s"), thread_db_err_str (err));
-      return 0;
-    }
-
-  /* These are not essential.  */
-  td_ta_event_addr_p = dlsym (handle, "td_ta_event_addr");
-  td_ta_set_event_p = dlsym (handle, "td_ta_set_event");
-  td_ta_event_getmsg_p = dlsym (handle, "td_ta_event_getmsg");
-  td_thr_event_enable_p = dlsym (handle, "td_thr_event_enable");
-  td_thr_tls_get_addr_p = dlsym (handle, "td_thr_tls_get_addr");
-
-  return 1;
-}
-
 static td_err_e
 enable_thread_event (td_thragent_t *thread_agent, int event, CORE_ADDR *bp)
 {
@@ -541,6 +478,275 @@ enable_thread_event_reporting (void)
     }
 }
 
+/* Attempt to initialize dlopen()ed libthread_db, described by HANDLE.
+   Return 1 on success.
+   Failure could happen if libthread_db does not have symbols we expect,
+   or when it refuses to work with the current inferior (e.g. due to
+   version mismatch between libthread_db and libpthread).  */
+
+static int
+try_thread_db_load_1 (void *handle)
+{
+  td_err_e err;
+
+  /* Initialize pointers to the dynamic library functions we will use.
+     Essential functions first.  */
+
+  td_init_p = verbose_dlsym (handle, "td_init");
+  if (td_init_p == NULL)
+    return 0;
+
+  err = td_init_p ();
+  if (err != TD_OK)
+    {
+      warning (_("Cannot initialize libthread_db: %s"), thread_db_err_str (err));
+      return 0;
+    }
+
+  td_ta_new_p = verbose_dlsym (handle, "td_ta_new");
+  if (td_ta_new_p == NULL)
+    return 0;
+
+  /* Initialize the structure that identifies the child process.  */
+  proc_handle.ptid = inferior_ptid;
+
+  /* Now attempt to open a connection to the thread library.  */
+  err = td_ta_new_p (&proc_handle, &thread_agent);
+  if (err != TD_OK)
+    {
+      td_ta_new_p = NULL;
+      if (info_verbose)
+	printf_unfiltered (_("td_ta_new failed: %s\n"),
+			   thread_db_err_str (err));
+      else
+        switch (err)
+          {
+            case TD_NOLIBTHREAD:
+#ifdef THREAD_DB_HAS_TD_VERSION
+            case TD_VERSION:
+#endif
+              /* The errors above are not unexpected and silently ignored:
+                 they just mean we haven't found correct version of
+                 libthread_db yet.  */
+              break;
+            default:
+              warning (_("td_ta_new failed: %s"), thread_db_err_str (err));
+          }
+      return 0;
+    }
+
+  td_ta_map_id2thr_p = verbose_dlsym (handle, "td_ta_map_id2thr");
+  if (td_ta_map_id2thr_p == NULL)
+    return 0;
+
+  td_ta_map_lwp2thr_p = verbose_dlsym (handle, "td_ta_map_lwp2thr");
+  if (td_ta_map_lwp2thr_p == NULL)
+    return 0;
+
+  td_ta_thr_iter_p = verbose_dlsym (handle, "td_ta_thr_iter");
+  if (td_ta_thr_iter_p == NULL)
+    return 0;
+
+  td_thr_validate_p = verbose_dlsym (handle, "td_thr_validate");
+  if (td_thr_validate_p == NULL)
+    return 0;
+
+  td_thr_get_info_p = verbose_dlsym (handle, "td_thr_get_info");
+  if (td_thr_get_info_p == NULL)
+    return 0;
+
+  /* These are not essential.  */
+  td_ta_event_addr_p = dlsym (handle, "td_ta_event_addr");
+  td_ta_set_event_p = dlsym (handle, "td_ta_set_event");
+  td_ta_event_getmsg_p = dlsym (handle, "td_ta_event_getmsg");
+  td_thr_event_enable_p = dlsym (handle, "td_thr_event_enable");
+  td_thr_tls_get_addr_p = dlsym (handle, "td_thr_tls_get_addr");
+
+  printf_unfiltered (_("[Thread debugging using libthread_db enabled]\n"));
+
+  /* The thread library was detected.  Activate the thread_db target.  */
+  push_target (&thread_db_ops);
+  thread_db_handle = handle;
+
+  enable_thread_event_reporting ();
+  thread_db_find_new_threads_1 ();
+  return 1;
+}
+
+/* Lookup a library in which given symbol resides.
+   Note: this is looking in GDB process, not in the inferior.
+   Returns library name, or NULL.  */
+
+static const char *
+dladdr_to_soname (const void *addr)
+{
+  Dl_info info;
+
+  if (dladdr (addr, &info) != 0)
+    return info.dli_fname;
+  return NULL;
+}
+
+/* Attempt to use LIBRARY as libthread_db.  LIBRARY could be absolute,
+   relative, or just LIBTHREAD_DB.  */
+
+static int
+try_thread_db_load (const char *library)
+{
+  void *handle;
+
+  if (info_verbose)
+    printf_unfiltered (_("Trying host libthread_db library: %s.\n"),
+                       library);
+  handle = dlopen (library, RTLD_NOW);
+  if (handle == NULL)
+    {
+      if (info_verbose)
+	printf_unfiltered (_("dlopen failed: %s.\n"), dlerror ());
+      return 0;
+    }
+
+  if (info_verbose && strchr (library, '/') == NULL)
+    {
+      void *td_init;
+
+      td_init = dlsym (handle, "td_init");
+      if (td_init != NULL)
+        {
+          const char *const libpath = dladdr_to_soname (td_init);
+
+          if (libpath != NULL)
+            printf_unfiltered (_("Host %s resolved to: %s.\n"),
+                               library, libpath);
+        }
+    }
+
+  if (try_thread_db_load_1 (handle))
+    return 1;
+
+  /* This library "refused" to work on current inferior.  */
+  dlclose (handle);
+  return 0;
+}
+
+
+/* Search libthread_db_search_path for libthread_db which "agrees"
+   to work on current inferior.  */
+
+static int
+thread_db_load_search ()
+{
+  char path[PATH_MAX];
+  const char *search_path = libthread_db_search_path;
+  int rc = 0;
+
+  while (*search_path)
+    {
+      const char *end = strchr (search_path, ':');
+      if (end)
+	{
+	  size_t len = end - search_path;
+          if (len + 1 + strlen (LIBTHREAD_DB_SO) + 1 > sizeof (path))
+            {
+              char *cp = xmalloc (len + 1);
+              memcpy (cp, search_path, len);
+              cp[len] = '\0';
+              warning (_("libthread_db_search_path component too long,"
+                         " ignored: %s."), cp);
+              xfree (cp);
+              search_path += len + 1;
+              continue;
+            }
+	  memcpy (path, search_path, len);
+	  path[len] = '\0';
+	  search_path += len + 1;
+	}
+      else
+	{
+          size_t len = strlen (search_path);
+
+          if (len + 1 + strlen (LIBTHREAD_DB_SO) + 1 > sizeof (path))
+            {
+              warning (_("libthread_db_search_path component too long,"
+                         " ignored: %s."), search_path);
+              break;
+            }
+	  memcpy (path, search_path, len + 1);
+	  search_path += len;
+	}
+      strcat (path, "/");
+      strcat (path, LIBTHREAD_DB_SO);
+      if (try_thread_db_load (path))
+	{
+	  rc = 1;
+	  break;
+	}
+    }
+  if (rc == 0)
+    rc = try_thread_db_load (LIBTHREAD_DB_SO);
+  if (rc == 0)
+    warning (_("Unable to find libthread_db matching inferior's thread"
+               " library, thread debugging will not be available."));
+  return rc;
+}
+
+/* Attempt to load and initialize libthread_db.
+   Return 1 on success.
+ */
+
+static int
+thread_db_load (void)
+{
+  const char *soname;
+  struct minimal_symbol *msym;
+
+  if (thread_db_handle != NULL)
+    return 1;
+
+  /* Don't attempt to use thread_db on targets which can not run
+     (executables not running yet, core files) for now.  */
+  if (!target_has_execution)
+    return 0;
+
+  /* Don't attempt to use thread_db for remote targets.  */
+  if (!target_can_run (&current_target))
+    return 0;
+
+  msym = lookup_minimal_symbol ("nptl_version", NULL, NULL);
+  if (!msym)
+    msym = lookup_minimal_symbol ("__linuxthreads_version", NULL, NULL);
+
+  /* Some really old libpthread versions do not have either of the above.  */
+  if (!msym)
+    msym = lookup_minimal_symbol ("__pthread_threads_events", NULL, NULL);
+
+  if (!msym)
+    /* No threads yet */
+    return 0;
+
+  soname = solib_name_from_address (SYMBOL_VALUE_ADDRESS (msym));
+  if (soname)
+    {
+      /* Attempt to load libthread_db from the same directory. */
+      char path[PATH_MAX], *cp;
+      strcpy (path, soname);
+      cp = strrchr (path, '/');
+      if (cp == NULL)
+	{
+	  /* Expected to get fully resolved pathname for libpthread,
+             but got something else.  Search for matching libthread_db and
+             hope there is one that matches current libpthread.  */
+	  warning (_("Cannot obtain absolute path of thread library: %s."),
+                   soname);
+	  return thread_db_load_search ();
+	}
+      strcpy (cp + 1, LIBTHREAD_DB_SO);
+      if (try_thread_db_load (path))
+	return 1;
+    }
+  return thread_db_load_search ();
+}
+
 static void
 disable_thread_event_reporting (void)
 {
@@ -593,75 +799,34 @@ void
 check_for_thread_db (void)
 {
   td_err_e err;
-  static int already_loaded;
+  static void *last_loaded;
 
   /* Do nothing if we couldn't load libthread_db.so.1.  */
-  if (td_ta_new_p == NULL)
+  if (!thread_db_load ())
     return;
 
   /* First time through, report that libthread_db was successfuly
      loaded.  Can't print this in in thread_db_load as, at that stage,
-     the interpreter and it's console haven't started.  */
+     the interpreter and it's console haven't started.
+     We track td_ta_new_p because the user may switch executables,
+     and as a result we may decide to use a different version of
+     libthread_db. */
 
-  if (!already_loaded)
+  if (last_loaded != td_ta_new_p)
     {
-      Dl_info info;
-      const char *library = NULL;
-      if (dladdr ((*td_ta_new_p), &info) != 0)
-	library = info.dli_fname;
+      last_loaded = td_ta_new_p;
 
-      /* Try dlinfo?  */
-
-      if (library == NULL)
-	/* Paranoid - don't let a NULL path slip through.  */
-	library = LIBTHREAD_DB_SO;
-
-      if (info_verbose)
-	printf_unfiltered (_("Using host libthread_db library \"%s\".\n"),
-			   library);
-      already_loaded = 1;
-    }
-
-  if (using_thread_db)
-    /* Nothing to do.  The thread library was already detected and the
-       target vector was already activated.  */
-    return;
-
-  /* Don't attempt to use thread_db on targets which can not run
-     (executables not running yet, core files) for now.  */
-  if (!target_has_execution)
-    return;
-
-  /* Don't attempt to use thread_db for remote targets.  */
-  if (!target_can_run (&current_target))
-    return;
-
-  /* Initialize the structure that identifies the child process.  */
-  proc_handle.ptid = inferior_ptid;
-
-  /* Now attempt to open a connection to the thread library.  */
-  err = td_ta_new_p (&proc_handle, &thread_agent);
-  switch (err)
-    {
-    case TD_NOLIBTHREAD:
-      /* No thread library was detected.  */
-      break;
-
-    case TD_OK:
-      printf_unfiltered (_("[Thread debugging using libthread_db enabled]\n"));
-
-      /* The thread library was detected.  Activate the thread_db target.  */
-      push_target (&thread_db_ops);
-      using_thread_db = 1;
-
-      enable_thread_event_reporting ();
-      thread_db_find_new_threads_1 ();
-      break;
-
-    default:
-      warning (_("Cannot initialize thread debugging library: %s"),
-	       thread_db_err_str (err));
-      break;
+      if (info_verbose || *libthread_db_search_path)
+        {
+          const char *library;
+
+          library = dladdr_to_soname (*td_ta_new_p);
+          if (library == NULL)
+            library = LIBTHREAD_DB_SO;
+
+          printf_unfiltered (_("Using host libthread_db library \"%s\".\n"),
+                             library);
+        }
     }
 }
 
@@ -783,7 +948,9 @@ thread_db_detach (struct target_ops *ops
 
   /* Detach thread_db target ops.  */
   unpush_target (&thread_db_ops);
-  using_thread_db = 0;
+  if (thread_db_handle)
+    dlclose (thread_db_handle);
+  thread_db_handle = NULL;
 
   target_beneath->to_detach (target_beneath, args, from_tty);
 }
@@ -896,7 +1063,9 @@ thread_db_wait (struct target_ops *ops,
     {
       remove_thread_event_breakpoints ();
       unpush_target (&thread_db_ops);
-      using_thread_db = 0;
+      if (thread_db_handle)
+	dlclose (thread_db_handle);
+      thread_db_handle = NULL;
 
       return ptid;
     }
@@ -944,7 +1113,9 @@ thread_db_mourn_inferior (struct target_
 
   /* Detach thread_db target ops.  */
   unpush_target (ops);
-  using_thread_db = 0;
+  if (thread_db_handle)
+    dlclose (thread_db_handle);
+  thread_db_handle = NULL;
 }
 
 static int
@@ -1187,13 +1358,28 @@ extern initialize_file_ftype _initialize
 void
 _initialize_thread_db (void)
 {
-  /* Only initialize the module if we can load libthread_db.  */
-  if (thread_db_load ())
-    {
-      init_thread_db_ops ();
-      add_target (&thread_db_ops);
+  init_thread_db_ops ();
+  add_target (&thread_db_ops);
 
-      /* Add ourselves to objfile event chain.  */
-      observer_attach_new_objfile (thread_db_new_objfile);
-    }
+  /* Defer loading of libthread_db.so until inferior is running.
+     This allows gdb to load correct libthread_db for a given
+     executable -- there could be mutiple versions of glibc,
+     compiled with LinuxThreads or NPTL, and until there is
+     a running inferior, we can't tell which libthread_db is
+     the correct one to load. */
+
+  libthread_db_search_path = xstrdup (LIBTHREAD_DB_SEARCH_PATH);
+
+  add_setshow_optional_filename_cmd ("libthread-db-search-path",
+				     class_support,
+				     &libthread_db_search_path, _("\
+Set search path for libthread_db."), _("\
+Show the current search path or libthread_db."), _("\
+This path is used to search for libthread_db to be loaded into \
+gdb itself."),
+			    NULL,
+			    NULL,
+			    &setlist, &showlist);
+  /* Add ourselves to objfile event chain.  */
+  observer_attach_new_objfile (thread_db_new_objfile);
 }
Index: doc/gdb.texinfo
===================================================================
RCS file: /cvs/src/src/gdb/doc/gdb.texinfo,v
retrieving revision 1.580
diff -u -p -u -r1.580 gdb.texinfo
--- doc/gdb.texinfo	15 Apr 2009 22:20:32 -0000	1.580
+++ doc/gdb.texinfo	20 Apr 2009 17:35:40 -0000
@@ -2429,6 +2429,9 @@ a command to apply a command to a list o
 @item thread-specific breakpoints
 @item @samp{set print thread-events}, which controls printing of 
 messages on thread start and exit.
+@item @samp{set libthread-db-search-path @var{path}}, which lets
+the user specify which @code{libthread_db} to use if the default choice
+isn't compatible with the program.
 @end itemize
 
 @quotation
@@ -2647,6 +2650,40 @@ programs with multiple threads.
 @xref{Set Watchpoints,,Setting Watchpoints}, for information about
 watchpoints in programs with multiple threads.
 
+@table @code
+@kindex set libthread-db-search-path
+@cindex search path for @code{libthread_db}
+@item set libthread-db-search-path @r{[}@var{path}@r{]}
+If this variable is set, @var{path} is a colon-separated list of
+directories @value{GDBN} will use to search for @code{libthread_db}.
+If you omit @var{path}, @samp{libthread-db-search-path} will be reset to
+an empty list.
+
+On @sc{gnu}/Linux and Solaris systems, @value{GDBN} uses a ``helper''
+@code{libthread_db} library to obtain information about threads in the
+inferior process.  @value{GDBN} first attempts to use
+@code{libthread_db} located in the directory from which
+@code{libpthread} was loaded in the inferior process.  If that fails,
+@value{GDBN} will use @samp{libthread-db-search-path},
+and then default system shared library directories, to find
+@code{libthread_db}.
+
+For any @code{libthread_db} library @value{GDBN} finds in above directories,
+@value{GDBN} attempts to initialize it with the current inferior process.
+If this initialization fails (which could happen because of a version
+mismatch between @code{libthread_db} and @code{libpthread}), @value{GDBN}
+will unload @code{libthread_db}, and continue with the next directory.
+If none of @code{libthread_db} libraries initialize successfully,
+@value{GDBN} will issue a warning and thread debugging will be disabled.
+
+Setting @code{libthread-db-search-path} is currently implemented
+only on some platforms.
+
+@kindex show libthread-db-search-path 
+@item show libthread-db-search-path 
+Display current libthread_db search path.
+@end table
+
 @node Processes
 @section Debugging Programs with Multiple Processes
 

Re: [patch][rfc] Allow GDB to search for the right libthread_db.so.1

[Eli Zaretskii]

> Date: Mon, 20 Apr 2009 10:37:08 -0700
> From: Paul Pluzhnikov <ppluzhnikov@google.com>
> 
> On Mon, Apr 20, 2009 at 9:47 AM, Paul Pluzhnikov <ppluzhnikov@google.com> wrote:
> 
> > Attached is a revised patch that fixes all the comments so far.
> 
> Sorry, missed the NEWS update.

Thanks.  The patches for NEWS and the manual are approved.

Re: [patch][rfc] Allow GDB to search for the right libthread_db.so.1

[Jan Kratochvil]

On Mon, 06 Apr 2009 22:39:20 +0200, Paul Pluzhnikov wrote:
> We have perhaps uncommon setup here, where we have several installed
> versions of glibc, and need to debug executables which are compiled
> and linked against them (using -rpath).
> 
> Currently, GDB will dlopen("libthread_db.so.1", ...), which means that
> in order debug "non-standard" binary, one has to set LD_LIBRARY_PATH to
> point to correct libthread_db before invoking GDB, or it will refuse to
> see threads in the inferior. This is not automatic, and error prone.

Just out of curiosity:

libthread_db should be compatible with any libpthread thanks to:

$ nm /lib64/libpthread.so.0|grep _thread_db_
0000003d1b812010 r _thread_db_dtv_dtv
0000003d1b812004 r _thread_db_dtv_t_pointer_val
0000003d1b81201c r _thread_db_link_map_l_tls_modid
0000003d1b8120c4 r _thread_db_list_t_next
0000003d1b8120b8 r _thread_db_list_t_prev
[39 lines total]

Do you use non-glibc libthread_db/libpthread (Bionic?) or this _thread_db_*
compatibility API does not work even for glibc versions?


Thanks,
Jan

Re: [patch][rfc] Allow GDB to search for the right libthread_db.so.1

[Paul Pluzhnikov]

On Tue, Aug 25, 2015 at 11:01 AM, Jan Kratochvil
<jan.kratochvil@redhat.com> wrote:
> On Mon, 06 Apr 2009 22:39:20 +0200, Paul Pluzhnikov wrote:
>> We have perhaps uncommon setup here, where we have several installed
>> versions of glibc, and need to debug executables which are compiled
>> and linked against them (using -rpath).
>>
>> Currently, GDB will dlopen("libthread_db.so.1", ...), which means that
>> in order debug "non-standard" binary, one has to set LD_LIBRARY_PATH to
>> point to correct libthread_db before invoking GDB, or it will refuse to
>> see threads in the inferior. This is not automatic, and error prone.
>
> Just out of curiosity:
>
> libthread_db should be compatible with any libpthread thanks to:
>
> $ nm /lib64/libpthread.so.0|grep _thread_db_
> 0000003d1b812010 r _thread_db_dtv_dtv
> 0000003d1b812004 r _thread_db_dtv_t_pointer_val
> 0000003d1b81201c r _thread_db_link_map_l_tls_modid
> 0000003d1b8120c4 r _thread_db_list_t_next
> 0000003d1b8120b8 r _thread_db_list_t_prev
> [39 lines total]

I don't understand what above symbols do, but ...

We've had a lot of systems where installed /lib64/libthread_db was
from glibc-2.7, while target binary was using glibc-2.3.6, and I am
99.99% sure that combination didn't work.

Currently we have most systems using glibc-2.19 in both /lib64 and
target, but when transitioning between OS releases (upgrading
desktops) we frequently have system libthread_db being older than
target libc.

Also, fully-static links are used sometimes, and that often presents
its own challenges.


> Do you use non-glibc libthread_db/libpthread (Bionic?) or this _thread_db_*
> compatibility API does not work even for glibc versions?

We do not use non-glibc libthread_db/libpthread.

Cheers,
-- 
Paul Pluzhnikov

Re: [patch][rfc] Allow GDB to search for the right libthread_db.so.1

[Jan Kratochvil]

On Tue, 25 Aug 2015 20:13:54 +0200, Paul Pluzhnikov wrote:
> We've had a lot of systems where installed /lib64/libthread_db was
> from glibc-2.7, while target binary was using glibc-2.3.6, and I am
> 99.99% sure that combination didn't work.

Pedro explained that glibc also does:

# $ grep -rn nptl_version nptl*
# nptl/ChangeLog.old:7461:        * init.c (nptl_version): Add __attribute_used__ to nptl_version.
# nptl/nptl-init.c:68:static const char nptl_version[] __attribute_used__ = VERSION;
# nptl_db/td_ta_new.c:42:  if (td_lookup (ps, SYM_nptl_version, &versaddr) != PS_OK)
# nptl_db/structs.def:53:DB_SYMBOL (nptl_version)
#
# So that alone makes a libthread_db.so for glibc 2.x refuse to work with glibc 2.y.

BTW the nptl_version check seems to predate the _thread_db_* symbols.


Thanks,
Jan