Re: [patch][rfc] Allow GDB to search for the right libthread_db.so.1

[Paul Pluzhnikov <ppluzhnikov@google.com>]

[-- Attachment #1: Type: text/plain, Size: 3830 bytes --]

On Mon, Apr 20, 2009 at 6:17 AM, Daniel Jacobowitz <drow@false.org> wrote:

> I have just a few minor comments on this.  Overall, I am heartily
> in favor of the idea - I have carried a less elegant version of this
> feature in the Debian GDB packages for two years or so.

Could you tell a bit more about this? I thought out set up with
multiple incompatible (WRT libthread_db) libc versions was more an
exception than the rule.

> On Fri, Apr 17, 2009 at 12:13:08PM -0700, Paul Pluzhnikov wrote:
>> +#ifndef LIBTHREAD_DB_SO
>> +#define LIBTHREAD_DB_SO "libthread_db.so.1"
>> +#endif
>> +
>> +#ifndef LIBTHREAD_DB_SEARCH_PATH
>> +#define LIBTHREAD_DB_SEARCH_PATH ""
>> +#endif
>
> Do we intend to let anything override these?  Otherwise IMHO the
> ifndef just confuses the issue.

It might be reasonable to override these via CFLAGS. In particular,
we intend to override the LIBTHREAD_DB_SEARCH_PATH with an appropriate
local default. Of course we could just as easily patch the source.

> Space before paren :-)

Sorry about that. Fixed everywhere.

>> +  if (!msym)
>> +    msym = lookup_minimal_symbol ("__pthread_threads_events", NULL, NULL);
>
> Why is this symbol lookup necessary?  Is it accomplishing the same
> thing that searching for a shared library matching "libpthread[-.]"
> would?

The symbol lookup works for statically linked executables (which we
also have and would like to support). Searching for libpthread solib
doesn't.

On Sat, Apr 18, 2009 at 10:00 AM, Eli Zaretskii <eliz@gnu.org> wrote:

>> +Setting @code{libthread-db-search-path} is currently implemented
>> +only for @sc{gnu}/Linux targets.
>
> In general, mentioning specific platforms in the manual is a
> maintenance nightmare, because we must remember to make changes in the
> manual to track the support of the feature on other platforms.  It is
> much better to say that this feature is available "only on some
> platforms."

Done.

>  That's assuming that on other platforms, using the
> command will cause a suitable error message; does it?

Yes: since the add_setshow... is not done on other platforms, you'll
get the same error as if you tried to set unknown variable (which is
actually quite confusing!):

   ./gdb -q -nx /bin/date
  (no debugging symbols found)
  (gdb) set libthread-db-search-path
  No symbol table is loaded.  Use the "file" command.
  (gdb)

> Btw, what are the Linux-specific aspects that this feature depends on?
> Why can't we implement it on Solaris as well?

This could quite easily be implemented on Solaris as well, although I
expect that there is significantly smaller number of users who need
this on Solaris. OTOH, anyone using glibc on Solaris will likely want
it (if he debugs both glibc and native-libc binaries).


Attached is a revised patch that fixes all the comments so far.

Thanks,
-- 
Paul Pluzhnikov

2009-04-20  Paul Pluzhnikov  <ppluzhnikov@google.com>

	* gdb_thread_db.h (LIBTHREAD_DB_SEARCH_PATH): New define.
	(LIBTHREAD_DB_SO): Moved from linux-thread-db.c

	* linux-thread-db.c (libthread_db_search_path): New setting.
	(thread_db_handle): New variable (replaces using_thread_db).
	(try_thread_db_load_1): New function.
	(try_thread_db_load, thread_db_load_search): Likewise.
	(dladdr_to_soname): Likewise.
	(thread_db_load): Iterate over possibly multiple libthread_db's.
	(check_for_thread_db): Attempt to load new libthread_db.
	(thread_db_detach, thread_db_wait): Unload libthread_db.
	(thread_db_mourn_inferior): Likewise.
	(_initialize_thread_db): Add new libthread-db-search-path option.
	Defer loading of libthread_db to check_for_thread_db.

doc/ChangeLog

2009-04-20  Paul Pluzhnikov  <ppluzhnikov@google.com>

	* gdb.texinfo (Threads): Document libthread-db-search-path.

[-- Attachment #2: gdb-thread-db.20090420.txt --]
[-- Type: text/plain, Size: 19840 bytes --]

Index: gdb_thread_db.h
===================================================================
RCS file: /cvs/src/src/gdb/gdb_thread_db.h,v
retrieving revision 1.12
diff -u -p -u -r1.12 gdb_thread_db.h
--- gdb_thread_db.h	18 Mar 2009 08:51:11 -0000	1.12
+++ gdb_thread_db.h	20 Apr 2009 16:41:55 -0000
@@ -1,5 +1,14 @@
 #ifdef HAVE_THREAD_DB_H
 #include <thread_db.h>
+
+#ifndef LIBTHREAD_DB_SO
+#define LIBTHREAD_DB_SO "libthread_db.so.1"
+#endif
+
+#ifndef LIBTHREAD_DB_SEARCH_PATH
+#define LIBTHREAD_DB_SEARCH_PATH ""
+#endif
+
 #else
 
 /* Copyright (C) 1999, 2000, 2007, 2008, 2009 Free Software Foundation, Inc.
Index: linux-thread-db.c
===================================================================
RCS file: /cvs/src/src/gdb/linux-thread-db.c,v
retrieving revision 1.54
diff -u -p -u -r1.54 linux-thread-db.c
--- linux-thread-db.c	27 Feb 2009 20:34:41 -0000	1.54
+++ linux-thread-db.c	20 Apr 2009 16:41:55 -0000
@@ -26,13 +26,16 @@
 #include "gdb_thread_db.h"
 
 #include "bfd.h"
+#include "command.h"
 #include "exceptions.h"
+#include "gdbcmd.h"
 #include "gdbthread.h"
 #include "inferior.h"
 #include "symfile.h"
 #include "objfiles.h"
 #include "target.h"
 #include "regcache.h"
+#include "solib.h"
 #include "solib-svr4.h"
 #include "gdbcore.h"
 #include "observer.h"
@@ -44,10 +47,6 @@
 #include <gnu/libc-version.h>
 #endif
 
-#ifndef LIBTHREAD_DB_SO
-#define LIBTHREAD_DB_SO "libthread_db.so.1"
-#endif
-
 /* GNU/Linux libthread_db support.
 
    libthread_db is a library, provided along with libpthread.so, which
@@ -74,14 +73,17 @@
    of the ptid_t prevents thread IDs changing when libpthread is
    loaded or unloaded.  */
 
+static char *libthread_db_search_path;
+
 /* If we're running on GNU/Linux, we must explicitly attach to any new
    threads.  */
 
 /* This module's target vector.  */
 static struct target_ops thread_db_ops;
 
-/* Non-zero if we're using this module's target vector.  */
-static int using_thread_db;
+/* Handle from dlopen for libthread_db.so.  Not NULL if we're using this
+   module's target vector.  */
+static void *thread_db_handle;
 
 /* Non-zero if we have determined the signals used by the threads
    library.  */
@@ -344,7 +346,7 @@ thread_db_attach_lwp (ptid_t ptid)
   td_thrinfo_t ti;
   td_err_e err;
 
-  if (!using_thread_db)
+  if (thread_db_handle == NULL)
     return 0;
 
   /* This ptid comes from linux-nat.c, which should always fill in the
@@ -385,71 +387,6 @@ verbose_dlsym (void *handle, const char 
   return sym;
 }
 
-static int
-thread_db_load (void)
-{
-  void *handle;
-  td_err_e err;
-
-  handle = dlopen (LIBTHREAD_DB_SO, RTLD_NOW);
-  if (handle == NULL)
-    {
-      fprintf_filtered (gdb_stderr, "\n\ndlopen failed on '%s' - %s\n",
-			LIBTHREAD_DB_SO, dlerror ());
-      fprintf_filtered (gdb_stderr,
-			"GDB will not be able to debug pthreads.\n\n");
-      return 0;
-    }
-
-  /* Initialize pointers to the dynamic library functions we will use.
-     Essential functions first.  */
-
-  td_init_p = verbose_dlsym (handle, "td_init");
-  if (td_init_p == NULL)
-    return 0;
-
-  td_ta_new_p = verbose_dlsym (handle, "td_ta_new");
-  if (td_ta_new_p == NULL)
-    return 0;
-
-  td_ta_map_id2thr_p = verbose_dlsym (handle, "td_ta_map_id2thr");
-  if (td_ta_map_id2thr_p == NULL)
-    return 0;
-
-  td_ta_map_lwp2thr_p = verbose_dlsym (handle, "td_ta_map_lwp2thr");
-  if (td_ta_map_lwp2thr_p == NULL)
-    return 0;
-
-  td_ta_thr_iter_p = verbose_dlsym (handle, "td_ta_thr_iter");
-  if (td_ta_thr_iter_p == NULL)
-    return 0;
-
-  td_thr_validate_p = verbose_dlsym (handle, "td_thr_validate");
-  if (td_thr_validate_p == NULL)
-    return 0;
-
-  td_thr_get_info_p = verbose_dlsym (handle, "td_thr_get_info");
-  if (td_thr_get_info_p == NULL)
-    return 0;
-
-  /* Initialize the library.  */
-  err = td_init_p ();
-  if (err != TD_OK)
-    {
-      warning (_("Cannot initialize libthread_db: %s"), thread_db_err_str (err));
-      return 0;
-    }
-
-  /* These are not essential.  */
-  td_ta_event_addr_p = dlsym (handle, "td_ta_event_addr");
-  td_ta_set_event_p = dlsym (handle, "td_ta_set_event");
-  td_ta_event_getmsg_p = dlsym (handle, "td_ta_event_getmsg");
-  td_thr_event_enable_p = dlsym (handle, "td_thr_event_enable");
-  td_thr_tls_get_addr_p = dlsym (handle, "td_thr_tls_get_addr");
-
-  return 1;
-}
-
 static td_err_e
 enable_thread_event (td_thragent_t *thread_agent, int event, CORE_ADDR *bp)
 {
@@ -541,6 +478,275 @@ enable_thread_event_reporting (void)
     }
 }
 
+/* Attempt to initialize dlopen()ed libthread_db, described by HANDLE.
+   Return 1 on success.
+   Failure could happen if libthread_db does not have symbols we expect,
+   or when it refuses to work with the current inferior (e.g. due to
+   version mismatch between libthread_db and libpthread).  */
+
+static int
+try_thread_db_load_1 (void *handle)
+{
+  td_err_e err;
+
+  /* Initialize pointers to the dynamic library functions we will use.
+     Essential functions first.  */
+
+  td_init_p = verbose_dlsym (handle, "td_init");
+  if (td_init_p == NULL)
+    return 0;
+
+  err = td_init_p ();
+  if (err != TD_OK)
+    {
+      warning (_("Cannot initialize libthread_db: %s"), thread_db_err_str (err));
+      return 0;
+    }
+
+  td_ta_new_p = verbose_dlsym (handle, "td_ta_new");
+  if (td_ta_new_p == NULL)
+    return 0;
+
+  /* Initialize the structure that identifies the child process.  */
+  proc_handle.ptid = inferior_ptid;
+
+  /* Now attempt to open a connection to the thread library.  */
+  err = td_ta_new_p (&proc_handle, &thread_agent);
+  if (err != TD_OK)
+    {
+      td_ta_new_p = NULL;
+      if (info_verbose)
+	printf_unfiltered (_("td_ta_new failed: %s\n"),
+			   thread_db_err_str (err));
+      else
+        switch (err)
+          {
+            case TD_NOLIBTHREAD:
+#ifdef THREAD_DB_HAS_TD_VERSION
+            case TD_VERSION:
+#endif
+              /* The errors above are not unexpected and silently ignored:
+                 they just mean we haven't found correct version of
+                 libthread_db yet.  */
+              break;
+            default:
+              warning (_("td_ta_new failed: %s"), thread_db_err_str (err));
+          }
+      return 0;
+    }
+
+  td_ta_map_id2thr_p = verbose_dlsym (handle, "td_ta_map_id2thr");
+  if (td_ta_map_id2thr_p == NULL)
+    return 0;
+
+  td_ta_map_lwp2thr_p = verbose_dlsym (handle, "td_ta_map_lwp2thr");
+  if (td_ta_map_lwp2thr_p == NULL)
+    return 0;
+
+  td_ta_thr_iter_p = verbose_dlsym (handle, "td_ta_thr_iter");
+  if (td_ta_thr_iter_p == NULL)
+    return 0;
+
+  td_thr_validate_p = verbose_dlsym (handle, "td_thr_validate");
+  if (td_thr_validate_p == NULL)
+    return 0;
+
+  td_thr_get_info_p = verbose_dlsym (handle, "td_thr_get_info");
+  if (td_thr_get_info_p == NULL)
+    return 0;
+
+  /* These are not essential.  */
+  td_ta_event_addr_p = dlsym (handle, "td_ta_event_addr");
+  td_ta_set_event_p = dlsym (handle, "td_ta_set_event");
+  td_ta_event_getmsg_p = dlsym (handle, "td_ta_event_getmsg");
+  td_thr_event_enable_p = dlsym (handle, "td_thr_event_enable");
+  td_thr_tls_get_addr_p = dlsym (handle, "td_thr_tls_get_addr");
+
+  printf_unfiltered (_("[Thread debugging using libthread_db enabled]\n"));
+
+  /* The thread library was detected.  Activate the thread_db target.  */
+  push_target (&thread_db_ops);
+  thread_db_handle = handle;
+
+  enable_thread_event_reporting ();
+  thread_db_find_new_threads_1 ();
+  return 1;
+}
+
+/* Lookup a library in which given symbol resides.
+   Note: this is looking in GDB process, not in the inferior.
+   Returns library name, or NULL.  */
+
+static const char *
+dladdr_to_soname (const void *addr)
+{
+  Dl_info info;
+
+  if (dladdr (addr, &info) != 0)
+    return info.dli_fname;
+  return NULL;
+}
+
+/* Attempt to use LIBRARY as libthread_db.  LIBRARY could be absolute,
+   relative, or just LIBTHREAD_DB.  */
+
+static int
+try_thread_db_load (const char *library)
+{
+  void *handle;
+
+  if (info_verbose)
+    printf_unfiltered (_("Trying host libthread_db library: %s.\n"),
+                       library);
+  handle = dlopen (library, RTLD_NOW);
+  if (handle == NULL)
+    {
+      if (info_verbose)
+	printf_unfiltered (_("dlopen failed: %s.\n"), dlerror ());
+      return 0;
+    }
+
+  if (info_verbose && strchr (library, '/') == NULL)
+    {
+      void *td_init;
+
+      td_init = dlsym (handle, "td_init");
+      if (td_init != NULL)
+        {
+          const char *const libpath = dladdr_to_soname (td_init);
+
+          if (libpath != NULL)
+            printf_unfiltered (_("Host %s resolved to: %s.\n"),
+                               library, libpath);
+        }
+    }
+
+  if (try_thread_db_load_1 (handle))
+    return 1;
+
+  /* This library "refused" to work on current inferior.  */
+  dlclose (handle);
+  return 0;
+}
+
+
+/* Search libthread_db_search_path for libthread_db which "agrees"
+   to work on current inferior.  */
+
+static int
+thread_db_load_search ()
+{
+  char path[PATH_MAX];
+  const char *search_path = libthread_db_search_path;
+  int rc = 0;
+
+  while (*search_path)
+    {
+      const char *end = strchr (search_path, ':');
+      if (end)
+	{
+	  size_t len = end - search_path;
+          if (len + 1 + strlen (LIBTHREAD_DB_SO) + 1 > sizeof (path))
+            {
+              char *cp = xmalloc (len + 1);
+              memcpy (cp, search_path, len);
+              cp[len] = '\0';
+              warning (_("libthread_db_search_path component too long,"
+                         " ignored: %s."), cp);
+              xfree (cp);
+              search_path += len + 1;
+              continue;
+            }
+	  memcpy (path, search_path, len);
+	  path[len] = '\0';
+	  search_path += len + 1;
+	}
+      else
+	{
+          size_t len = strlen (search_path);
+
+          if (len + 1 + strlen (LIBTHREAD_DB_SO) + 1 > sizeof (path))
+            {
+              warning (_("libthread_db_search_path component too long,"
+                         " ignored: %s."), search_path);
+              break;
+            }
+	  memcpy (path, search_path, len + 1);
+	  search_path += len;
+	}
+      strcat (path, "/");
+      strcat (path, LIBTHREAD_DB_SO);
+      if (try_thread_db_load (path))
+	{
+	  rc = 1;
+	  break;
+	}
+    }
+  if (rc == 0)
+    rc = try_thread_db_load (LIBTHREAD_DB_SO);
+  if (rc == 0)
+    warning (_("Unable to find libthread_db matching inferior's thread"
+               " library, thread debugging will not be available."));
+  return rc;
+}
+
+/* Attempt to load and initialize libthread_db.
+   Return 1 on success.
+ */
+
+static int
+thread_db_load (void)
+{
+  const char *soname;
+  struct minimal_symbol *msym;
+
+  if (thread_db_handle != NULL)
+    return 1;
+
+  /* Don't attempt to use thread_db on targets which can not run
+     (executables not running yet, core files) for now.  */
+  if (!target_has_execution)
+    return 0;
+
+  /* Don't attempt to use thread_db for remote targets.  */
+  if (!target_can_run (&current_target))
+    return 0;
+
+  msym = lookup_minimal_symbol ("nptl_version", NULL, NULL);
+  if (!msym)
+    msym = lookup_minimal_symbol ("__linuxthreads_version", NULL, NULL);
+
+  /* Some really old libpthread versions do not have either of the above.  */
+  if (!msym)
+    msym = lookup_minimal_symbol ("__pthread_threads_events", NULL, NULL);
+
+  if (!msym)
+    /* No threads yet */
+    return 0;
+
+  soname = solib_name_from_address (SYMBOL_VALUE_ADDRESS (msym));
+  if (soname)
+    {
+      /* Attempt to load libthread_db from the same directory. */
+      char path[PATH_MAX], *cp;
+      strcpy (path, soname);
+      cp = strrchr (path, '/');
+      if (cp == NULL)
+	{
+	  /* Expected to get fully resolved pathname for libpthread,
+             but got something else.  Search for matching libthread_db and
+             hope there is one that matches current libpthread.  */
+	  warning (_("Cannot obtain absolute path of thread library: %s."),
+                   soname);
+	  return thread_db_load_search ();
+	}
+      strcpy (cp + 1, LIBTHREAD_DB_SO);
+      if (try_thread_db_load (path))
+	return 1;
+    }
+  return thread_db_load_search ();
+}
+
 static void
 disable_thread_event_reporting (void)
 {
@@ -593,75 +799,34 @@ void
 check_for_thread_db (void)
 {
   td_err_e err;
-  static int already_loaded;
+  static void *last_loaded;
 
   /* Do nothing if we couldn't load libthread_db.so.1.  */
-  if (td_ta_new_p == NULL)
+  if (!thread_db_load ())
     return;
 
   /* First time through, report that libthread_db was successfuly
      loaded.  Can't print this in in thread_db_load as, at that stage,
-     the interpreter and it's console haven't started.  */
+     the interpreter and it's console haven't started.
+     We track td_ta_new_p because the user may switch executables,
+     and as a result we may decide to use a different version of
+     libthread_db. */
 
-  if (!already_loaded)
+  if (last_loaded != td_ta_new_p)
     {
-      Dl_info info;
-      const char *library = NULL;
-      if (dladdr ((*td_ta_new_p), &info) != 0)
-	library = info.dli_fname;
+      last_loaded = td_ta_new_p;
 
-      /* Try dlinfo?  */
-
-      if (library == NULL)
-	/* Paranoid - don't let a NULL path slip through.  */
-	library = LIBTHREAD_DB_SO;
-
-      if (info_verbose)
-	printf_unfiltered (_("Using host libthread_db library \"%s\".\n"),
-			   library);
-      already_loaded = 1;
-    }
-
-  if (using_thread_db)
-    /* Nothing to do.  The thread library was already detected and the
-       target vector was already activated.  */
-    return;
-
-  /* Don't attempt to use thread_db on targets which can not run
-     (executables not running yet, core files) for now.  */
-  if (!target_has_execution)
-    return;
-
-  /* Don't attempt to use thread_db for remote targets.  */
-  if (!target_can_run (&current_target))
-    return;
-
-  /* Initialize the structure that identifies the child process.  */
-  proc_handle.ptid = inferior_ptid;
-
-  /* Now attempt to open a connection to the thread library.  */
-  err = td_ta_new_p (&proc_handle, &thread_agent);
-  switch (err)
-    {
-    case TD_NOLIBTHREAD:
-      /* No thread library was detected.  */
-      break;
-
-    case TD_OK:
-      printf_unfiltered (_("[Thread debugging using libthread_db enabled]\n"));
-
-      /* The thread library was detected.  Activate the thread_db target.  */
-      push_target (&thread_db_ops);
-      using_thread_db = 1;
-
-      enable_thread_event_reporting ();
-      thread_db_find_new_threads_1 ();
-      break;
-
-    default:
-      warning (_("Cannot initialize thread debugging library: %s"),
-	       thread_db_err_str (err));
-      break;
+      if (info_verbose || *libthread_db_search_path)
+        {
+          const char *library;
+
+          library = dladdr_to_soname (*td_ta_new_p);
+          if (library == NULL)
+            library = LIBTHREAD_DB_SO;
+
+          printf_unfiltered (_("Using host libthread_db library \"%s\".\n"),
+                             library);
+        }
     }
 }
 
@@ -783,7 +948,9 @@ thread_db_detach (struct target_ops *ops
 
   /* Detach thread_db target ops.  */
   unpush_target (&thread_db_ops);
-  using_thread_db = 0;
+  if (thread_db_handle)
+    dlclose (thread_db_handle);
+  thread_db_handle = NULL;
 
   target_beneath->to_detach (target_beneath, args, from_tty);
 }
@@ -896,7 +1063,9 @@ thread_db_wait (struct target_ops *ops,
     {
       remove_thread_event_breakpoints ();
       unpush_target (&thread_db_ops);
-      using_thread_db = 0;
+      if (thread_db_handle)
+	dlclose (thread_db_handle);
+      thread_db_handle = NULL;
 
       return ptid;
     }
@@ -944,7 +1113,9 @@ thread_db_mourn_inferior (struct target_
 
   /* Detach thread_db target ops.  */
   unpush_target (ops);
-  using_thread_db = 0;
+  if (thread_db_handle)
+    dlclose (thread_db_handle);
+  thread_db_handle = NULL;
 }
 
 static int
@@ -1187,13 +1358,28 @@ extern initialize_file_ftype _initialize
 void
 _initialize_thread_db (void)
 {
-  /* Only initialize the module if we can load libthread_db.  */
-  if (thread_db_load ())
-    {
-      init_thread_db_ops ();
-      add_target (&thread_db_ops);
+  init_thread_db_ops ();
+  add_target (&thread_db_ops);
 
-      /* Add ourselves to objfile event chain.  */
-      observer_attach_new_objfile (thread_db_new_objfile);
-    }
+  /* Defer loading of libthread_db.so until inferior is running.
+     This allows gdb to load correct libthread_db for a given
+     executable -- there could be mutiple versions of glibc,
+     compiled with LinuxThreads or NPTL, and until there is
+     a running inferior, we can't tell which libthread_db is
+     the correct one to load. */
+
+  libthread_db_search_path = xstrdup (LIBTHREAD_DB_SEARCH_PATH);
+
+  add_setshow_optional_filename_cmd ("libthread-db-search-path",
+				     class_support,
+				     &libthread_db_search_path, _("\
+Set search path for libthread_db."), _("\
+Show the current search path or libthread_db."), _("\
+This path is used to search for libthread_db to be loaded into \
+gdb itself."),
+			    NULL,
+			    NULL,
+			    &setlist, &showlist);
+  /* Add ourselves to objfile event chain.  */
+  observer_attach_new_objfile (thread_db_new_objfile);
 }
Index: doc/gdb.texinfo
===================================================================
RCS file: /cvs/src/src/gdb/doc/gdb.texinfo,v
retrieving revision 1.580
diff -u -p -u -r1.580 gdb.texinfo
--- doc/gdb.texinfo	15 Apr 2009 22:20:32 -0000	1.580
+++ doc/gdb.texinfo	20 Apr 2009 16:41:56 -0000
@@ -2429,6 +2429,9 @@ a command to apply a command to a list o
 @item thread-specific breakpoints
 @item @samp{set print thread-events}, which controls printing of 
 messages on thread start and exit.
+@item @samp{set libthread-db-search-path @var{path}}, which lets
+the user specify which @code{libthread_db} to use if the default choice
+isn't compatible with the program.
 @end itemize
 
 @quotation
@@ -2647,6 +2650,40 @@ programs with multiple threads.
 @xref{Set Watchpoints,,Setting Watchpoints}, for information about
 watchpoints in programs with multiple threads.
 
+@table @code
+@kindex set libthread-db-search-path
+@cindex search path for @code{libthread_db}
+@item set libthread-db-search-path @r{[}@var{path}@r{]}
+If this variable is set, @var{path} is a colon-separated list of
+directories @value{GDBN} will use to search for @code{libthread_db}.
+If you omit @var{path}, @samp{libthread-db-search-path} will be reset to
+an empty list.
+
+On @sc{gnu}/Linux and Solaris systems, @value{GDBN} uses a ``helper''
+@code{libthread_db} library to obtain information about threads in the
+inferior process.  @value{GDBN} first attempts to use
+@code{libthread_db} located in the directory from which
+@code{libpthread} was loaded in the inferior process.  If that fails,
+@value{GDBN} will use @samp{libthread-db-search-path},
+and then default system shared library directories, to find
+@code{libthread_db}.
+
+For any @code{libthread_db} library @value{GDBN} finds in above directories,
+@value{GDBN} attempts to initialize it with the current inferior process.
+If this initialization fails (which could happen because of a version
+mismatch between @code{libthread_db} and @code{libpthread}), @value{GDBN}
+will unload @code{libthread_db}, and continue with the next directory.
+If none of @code{libthread_db} libraries initialize successfully,
+@value{GDBN} will issue a warning and thread debugging will be disabled.
+
+Setting @code{libthread-db-search-path} is currently implemented
+only on some platforms.
+
+@kindex show libthread-db-search-path 
+@item show libthread-db-search-path 
+Display current libthread_db search path.
+@end table
+
 @node Processes
 @section Debugging Programs with Multiple Processes