(linux/threads) Interesting side-effect of "auto-solib-add 0"

(linux/threads) Interesting side-effect of "auto-solib-add 0"

[Joel Brobecker]

Hello,

One of our customers noticed that when you set auto-solib-add to off
and then start your program, GDB usually breaks down like so:

    (gdb) c
    Continuing.
    
    Program terminated with signal SIGTRAP, Trace/breakpoint trap.
    The program no longer exists.

This is on x86-linux.

The important clue here is that the program is using threads, and
that turning off auto-solib-add causes GDB to not push the thread
support module.

I did my analysis with a debugger derived from gdb-6.4. With the
current GDB from CVS, the errors are not so obvious - at least
I can't seem to reproduce them. But I did double-check that GDB
indeed doesn't start the thread module (by checking with another
debugger the value of "current_target") either.

 From what I can tell, the thread support is pushed by check_for_thread_db
which is called (through several calls) by solib_read_symbols, which
itself is called by solib_add:

    for (gdb = so_list_head; gdb; gdb = gdb->next)
      if (! pattern || re_exec (gdb->so_name))
        {
          any_matches = 1;
          if (readsyms && solib_read_symbols (gdb, from_tty))
            loaded_any_symbols = 1;
        }

The call to check_for_thread_db occurs through a hook:

        /* Call predecessor on chain, if any.  */
        if (remote_new_objfile_chain != 0 &&
            remote_desc == 0)
          remote_new_objfile_chain (objfile);

Which is set as follow in linux-thread-db.c:

        /* Add ourselves to objfile event chain.  */
        target_new_objfile_chain = deprecated_target_new_objfile_hook;
        deprecated_target_new_objfile_hook = thread_db_new_objfile;

and remote.c:

        /* Hook into new objfile notification.  */
        remote_new_objfile_chain = deprecated_target_new_objfile_hook;
        deprecated_target_new_objfile_hook  = remote_new_objfile;

(ugh! another candidate for a new observer???)

So basically, the linux thread layer hooks itself up to the objfile
creation callback. But unfortunately for us, the objfile for
libpthread.so does not get created since we turned auto-solib-add off,
the hook doesn't get called, and as a result, thread support stays off.

It seems to me that we should be able to fix this by adding an observer
of the `solib_loaded' event to replace the hook that linux-thread-db
installed. I actually gave it a try, using the sources from CVS, and
found out something surprising: Although the solib_loaded observer
I just created gets called as expected, the linux thread layers does
not detect that threads are in use:

        /* Now attempt to open a connection to the thread library.  */
        err = td_ta_new_p (&proc_handle, &thread_agent);
        switch (err)
          {
          case TD_NOLIBTHREAD:
            /* No thread library was detected.  */
            break;

For some reason that I cannot understand, "td_ta_new" doesn't detect
thread usage if this function is called from the observer.

I made another experiment: Still with the old hook deactivated, I added
the following loop at the end of "solib_add", which basically causes
a second round of notifications after solib_add had time to do its
thing:

        so = so_list_head;
        while (so != NULL)
          {
            observer_notify_solib_loaded (so);
            so = so->next;
          }

I first did a run with "auto-solib-add" activated (unchanged), and
linux-thread-db.c DID detect thread usage. I then turned off
auto-solib-add, and this time thread usages was NOT detected.

The only difference I can see between the two runs is that symbols
are read in one case, and not read in the other case. Is libthread_db.so
dependent on having the symbol table being loaded or something of that
sort? Any idea as to using the solib_loaded observer didn't work?

Thanks,
-- 
Joel

Re: (linux/threads) Interesting side-effect of "auto-solib-add 0"

[Daniel Jacobowitz]

On Tue, Sep 12, 2006 at 05:27:13PM -0700, Joel Brobecker wrote:
> It seems to me that we should be able to fix this by adding an observer
> of the `solib_loaded' event to replace the hook that linux-thread-db
> installed. I actually gave it a try, using the sources from CVS, and
> found out something surprising: Although the solib_loaded observer
> I just created gets called as expected, the linux thread layers does
> not detect that threads are in use:

You've missed the pertinent fact here.  We're not waiting for
libpthread.so to be loaded just for kicks!  The only way libthread_db
can operate is by querying back through GDB for the locations of
symbols in the thread library.  If we don't load that objfile, then
it can't initialize.

> The only difference I can see between the two runs is that symbols
> are read in one case, and not read in the other case. Is libthread_db.so
> dependent on having the symbol table being loaded or something of that
> sort?

Yes.  Don't load libpthread, won't get threads.

-- 
Daniel Jacobowitz
CodeSourcery

Re: (linux/threads) Interesting side-effect of "auto-solib-add 0"

[Joel Brobecker]

> You've missed the pertinent fact here.  We're not waiting for
> libpthread.so to be loaded just for kicks!  The only way libthread_db
> can operate is by querying back through GDB for the locations of
> symbols in the thread library.  If we don't load that objfile, then
> it can't initialize.

It's really strange. I put traces everywhere inside linux-thread-db
so that if there was any callbacks called during the detection,
I would see them. As you can see, I'm not very familiar with how
this library works. Do you know where the callbacks are in the
debugger code? I've been trying to find a document that explains
how the pthread_db library works, is there one?

> > The only difference I can see between the two runs is that symbols
> > are read in one case, and not read in the other case. Is libthread_db.so
> > dependent on having the symbol table being loaded or something of that
> > sort?
> 
> Yes.  Don't load libpthread, won't get threads.

Sounds fair enough to me.

Eli, everyone,

Would it be something worth documenting somewhere in the manual?
I could probably write a small paragraph somewhere after the description
of the auto-solib-add setting.

Thanks!
-- 
Joel

Re: (linux/threads) Interesting side-effect of "auto-solib-add 0"

[Daniel Jacobowitz]

On Tue, Sep 12, 2006 at 05:47:20PM -0700, Joel Brobecker wrote:
> > You've missed the pertinent fact here.  We're not waiting for
> > libpthread.so to be loaded just for kicks!  The only way libthread_db
> > can operate is by querying back through GDB for the locations of
> > symbols in the thread library.  If we don't load that objfile, then
> > it can't initialize.
> 
> It's really strange. I put traces everywhere inside linux-thread-db
> so that if there was any callbacks called during the detection,
> I would see them. As you can see, I'm not very familiar with how
> this library works. Do you know where the callbacks are in the
> debugger code?

Ah, simple: they're not there.  Take a look at the ps_* routines in
proc-service.c.  In particular, ps_pglobal_lookup.  It's not at all
surprising that you couldn't find them; this is a very rare library
interface.  The shared library actually has undefined symbols which
must be satisfied by GDB.

Do you want to stick a comment in linux-thread-db.c somewhere
suggesting a look over there too?

> I've been trying to find a document that explains
> how the pthread_db library works, is there one?

Well, the way we use glibc's isn't documented anywhere, but this
library is historically Sun's - their documentation may be enlightening.

> > > The only difference I can see between the two runs is that symbols
> > > are read in one case, and not read in the other case. Is libthread_db.so
> > > dependent on having the symbol table being loaded or something of that
> > > sort?
> > 
> > Yes.  Don't load libpthread, won't get threads.
> 
> Sounds fair enough to me.

The technical alternative, if you want to implement it, would be to
always load symbols for libpthread.  What do you think?  We don't even
really need debugging symbols - just the minimal (ELF) symbol table.

> Eli, everyone,
> 
> Would it be something worth documenting somewhere in the manual?
> I could probably write a small paragraph somewhere after the description
> of the auto-solib-add setting.

If we leave it the way it is, I think this is a great idea.  Please be
sure to mark it as a GNU/Linux specific issue.

-- 
Daniel Jacobowitz
CodeSourcery

Re: (linux/threads) Interesting side-effect of "auto-solib-add 0"

[Joel Brobecker]

> Ah, simple: they're not there.  Take a look at the ps_* routines in
> proc-service.c.  In particular, ps_pglobal_lookup.  It's not at all
> surprising that you couldn't find them; this is a very rare library
> interface.  The shared library actually has undefined symbols which
> must be satisfied by GDB.

Ah ha, I see! I didn't realize it was using undefined references...
I've been used to seeing callbacks...

> Do you want to stick a comment in linux-thread-db.c somewhere
> suggesting a look over there too?

Absolutely, might be useful to someone else. I'll do that tomorrow.

> The technical alternative, if you want to implement it, would be to
> always load symbols for libpthread.  What do you think?  We don't even
> really need debugging symbols - just the minimal (ELF) symbol table.

I have been thinking about that, but there were several little factors
that made me reconsider. The one that I didn't like the most was the
fact that the only way I could see to detect the pthread library being
loaded is by looking at its name. It doesn't feel very elegant to have
to search for "/libpthread.so" or somesuch in GDB. I don't know very
well how GNU/Linux distributions are made, but couldn't that library
be named libpthread-1.8.0.so, for instance, making its identification
harder?

At one point, I considered the idea of detecting libpthread.so, and
printing a warning saying that, because auto-solib-add has been
deactivated, thread support hasn't been enabled. But that's almost
no less work than the above, so pointless.

In the end, because I wasn't able to produce any unexpected behavior
except maybe lack of thread support, and given the probable little
number of times this capability is used (I'm guessing, though), it
didn't seem worth the effort.

I do realize, however, that it shouldn't be too much work, though.
I'm open to both alternatives:

  1. Enhance GDB
  2. Document the limitation (Eli also agreed with the idea)

-- 
Joel

Re: (linux/threads) Interesting side-effect of "auto-solib-add 0"

[Daniel Jacobowitz]

On Tue, Sep 12, 2006 at 08:45:18PM -0700, Joel Brobecker wrote:
> I have been thinking about that, but there were several little factors
> that made me reconsider. The one that I didn't like the most was the
> fact that the only way I could see to detect the pthread library being
> loaded is by looking at its name. It doesn't feel very elegant to have
> to search for "/libpthread.so" or somesuch in GDB. I don't know very
> well how GNU/Linux distributions are made, but couldn't that library
> be named libpthread-1.8.0.so, for instance, making its identification
> harder?

Well, if you wanted to be that precise about it, you could do it by
SONAME.  But I think matching libpthread wouldn't be that bad of a
hack.

>   2. Document the limitation (Eli also agreed with the idea)

Let's go with this for now then.

-- 
Daniel Jacobowitz
CodeSourcery

Re: (linux/threads) Interesting side-effect of "auto-solib-add 0"

[Eli Zaretskii]

> Date: Tue, 12 Sep 2006 17:47:20 -0700
> From: Joel Brobecker <brobecker@adacore.com>
> 
> Eli, everyone,
> 
> Would it be something worth documenting somewhere in the manual?
> I could probably write a small paragraph somewhere after the description
> of the auto-solib-add setting.

Yes, please do.