expected behavior of GNU/Linux gcore and corefiles

expected behavior of GNU/Linux gcore and corefiles

[Andrew Cagney]

(notes from a hallway conversation, I had a recollection that, at some 
stage, the attempt to load libthread db over a core file on GNU/Linux 
was disabled.)

When using GDB on a live threaded program that puts all threads into 
tight infinite loops (while (1);), I'll do something like:

     $ ./a.out &
     Pid 1234
     $ gdb ./a.out 1234
     (gdb) info threads
     ....
     (gdb) quit

As a user I'd also expect sequences such as:

     $ kill -QUIT 1234
     $ gdb ./a.out core
     (gdb) info threads
     ....
     (gdb) quit

and:

     $ gcore 1234
     $ gdb ./a.out core
     (gdb) info threads
     ....
     (gdb) quit

and:

     $ gdb ./a.out 1234
     (gdb) gcore
     (gdb) quit
     $ gdb ./a.out core
     (gdb) info threads

to all come back with effectively the same output.  Further, on both 
live and corefile targets, I'd expect to be able to select/examine each 
thread vis:

     (gdb) thread 5
     11    i = i + 1;
     (gdb) list
     10    __thread__ i = 1;
     11    i = i + 1;
     (gdb) print i
     $1 = 1

(which would involve thread local storage).

Two problems come to mind:

- Is the kernel including all the raw information needed to do this in 
the core file?

- For GDB to completly implement the above, is it forced to use 
libthread-db?

My instincts tell me that, to completly implement the above 
functionality, GDB is always going to need libthread-db.  If GDB could 
implement the above on a core file without using libthread-db, then GDB 
could also implement the above on a live target also without using 
libthread-db.  This is because a core file is always going to contain a 
subset of the information made available via ptrace et.al.

Andrew

Re: expected behavior of GNU/Linux gcore and corefiles

[Daniel Jacobowitz]

On Sat, Mar 01, 2003 at 10:16:39PM -0500, Andrew Cagney wrote:
> (notes from a hallway conversation, I had a recollection that, at some 
> stage, the attempt to load libthread db over a core file on GNU/Linux 
> was disabled.)
> 
> When using GDB on a live threaded program that puts all threads into 
> tight infinite loops (while (1);), I'll do something like:
> 
>     $ ./a.out &
>     Pid 1234
>     $ gdb ./a.out 1234
>     (gdb) info threads
>     ....
>     (gdb) quit
> 
> As a user I'd also expect sequences such as:
> 
>     $ kill -QUIT 1234
>     $ gdb ./a.out core
>     (gdb) info threads
>     ....
>     (gdb) quit
> 
> and:
> 
>     $ gcore 1234
>     $ gdb ./a.out core
>     (gdb) info threads
>     ....
>     (gdb) quit

Hmm, what platform is that?  I've never seen a GNU/Linux
implementation, but I'm a youngun.

> 
> and:
> 
>     $ gdb ./a.out 1234
>     (gdb) gcore
>     (gdb) quit
>     $ gdb ./a.out core
>     (gdb) info threads
> 
> to all come back with effectively the same output.  Further, on both 
> live and corefile targets, I'd expect to be able to select/examine each 
> thread vis:
> 
>     (gdb) thread 5
>     11    i = i + 1;
>     (gdb) list
>     10    __thread__ i = 1;
>     11    i = i + 1;
>     (gdb) print i
>     $1 = 1
> 
> (which would involve thread local storage).
> 
> Two problems come to mind:
> 
> - Is the kernel including all the raw information needed to do this in 
> the core file?

Some platforms yes, some platforms no.  I've had Linux patches to do
this for some time.  I believe kernels as of 2.5.50 or so do it also.

> - For GDB to completly implement the above, is it forced to use 
> libthread-db?

Not necessarily.
Here's the added value we get from libthread_db right now, on "normal"
GNU/Linux threading (either LinuxThreads or NPTL, but not including the
M-N stuff that NGPT does):
  - Thread IDs, instead of just LWP IDs.  Big deal.
  - Thread local storage offsets.

We could do find TLS areas with a per-architecture hook; libthread_db
finds the TLS area by grubbing through the thread descriptor, but given
that we have each LWP's state we could just do it in the
platform-mandated way (i.e. examining the registers in question, etc.)

I don't know offhand if the necessary information for Alpha, i386, etc.
TLS is in core files - i.e. platforms where it's more than just a
register.

> My instincts tell me that, to completly implement the above 
> functionality, GDB is always going to need libthread-db.  If GDB could 
> implement the above on a core file without using libthread-db, then GDB 
> could also implement the above on a live target also without using 
> libthread-db.  This is because a core file is always going to contain a 
> subset of the information made available via ptrace et.al.

In my opinion, there are two things wrong with that instinct:
  - What we need for live debugging is a superset of what we need for
    core debugging.  For instance we use libthread_db to find new
    threads at the moment.
  - Current GNU/Linux kernels can implement an entirely thread_db free
    debugging model for threads.  We can trap on each clone, acquire
    each thread as it is created, map them to our internal numbering,
    etc.  We can't easily figure out TIDs (1024, 2049, etc.) - but it's
    not clear how much that matters.

    Doing it this way is more like the old linux-thread code than the
    current lin-lwp/thread-db combination; but it's vastly more
    reliable than either.  In my opinion it's preferable.  I just
    haven't had the time to finish it; and it isn't clear how to
    integrate it with thread_db.

Thread_db is a great interface for querying threads.  It's not so good
for debugging them; for a case in point see my messages about the FAIL
at the end of print-threads.exp.  Thread_db does not interact well with
debugging the thread manager itself, which is a very important feature
for me.

-- 
Daniel Jacobowitz
MontaVista Software                         Debian GNU/Linux Developer

Re: expected behavior of GNU/Linux gcore and corefiles

[Daniel Jacobowitz]

On Sat, Mar 01, 2003 at 10:39:00PM -0500, Daniel Jacobowitz wrote:
> > My instincts tell me that, to completly implement the above 
> > functionality, GDB is always going to need libthread-db.  If GDB could 
> > implement the above on a core file without using libthread-db, then GDB 
> > could also implement the above on a live target also without using 
> > libthread-db.  This is because a core file is always going to contain a 
> > subset of the information made available via ptrace et.al.

Oh, and one other thing that I like to mention when this comes up.  My
previous message was the implementation issues involved; this one's the
motivational issue.  Thread_db is not, and can't/shouldn't be,
available in a cross environment.  We have done a lot of work to make
GDB read corefiles in a cross environment; and at MontaVista we've seen
a large demand for this functionality from our customers.  So using
thread_db with corefiles doesn't meet our (GDB developers') goals, I
think.


-- 
Daniel Jacobowitz
MontaVista Software                         Debian GNU/Linux Developer

Re: expected behavior of GNU/Linux gcore and corefiles

[Andrew Cagney]

> On Sat, Mar 01, 2003 at 10:39:00PM -0500, Daniel Jacobowitz wrote:
> 
>> > My instincts tell me that, to completly implement the above 
>> > functionality, GDB is always going to need libthread-db.  If GDB could 
>> > implement the above on a core file without using libthread-db, then GDB 
>> > could also implement the above on a live target also without using 
>> > libthread-db.  This is because a core file is always going to contain a 
>> > subset of the information made available via ptrace et.al.
> 
> 
> Oh, and one other thing that I like to mention when this comes up.  My
> previous message was the implementation issues involved; this one's the
> motivational issue.  Thread_db is not, and can't/shouldn't be,
> available in a cross environment.  We have done a lot of work to make
> GDB read corefiles in a cross environment; and at MontaVista we've seen
> a large demand for this functionality from our customers.  So using
> thread_db with corefiles doesn't meet our (GDB developers') goals, I
> think.

Sorry, on this point, I'm lost.  What are you suggesting here?

Andrew

Re: expected behavior of GNU/Linux gcore and corefiles

[Daniel Jacobowitz]

On Sun, Mar 02, 2003 at 10:56:08AM -0500, Andrew Cagney wrote:
> >On Sat, Mar 01, 2003 at 10:39:00PM -0500, Daniel Jacobowitz wrote:
> >
> >>> My instincts tell me that, to completly implement the above 
> >>> functionality, GDB is always going to need libthread-db.  If GDB could 
> >>> implement the above on a core file without using libthread-db, then GDB 
> >>> could also implement the above on a live target also without using 
> >>> libthread-db.  This is because a core file is always going to contain a 
> >>> subset of the information made available via ptrace et.al.
> >
> >
> >Oh, and one other thing that I like to mention when this comes up.  My
> >previous message was the implementation issues involved; this one's the
> >motivational issue.  Thread_db is not, and can't/shouldn't be,
> >available in a cross environment.  We have done a lot of work to make
> >GDB read corefiles in a cross environment; and at MontaVista we've seen
> >a large demand for this functionality from our customers.  So using
> >thread_db with corefiles doesn't meet our (GDB developers') goals, I
> >think.
> 
> Sorry, on this point, I'm lost.  What are you suggesting here?

My point is just that it's important for GDB to not require thread_db
when dealing with core files.  A lot of people seem to use e.g. Solaris
to debug core files and expect everything they would get from using GDB
natively; that's their expectation and so far we've been able to stay
pretty much there.

-- 
Daniel Jacobowitz
MontaVista Software                         Debian GNU/Linux Developer

Re: expected behavior of GNU/Linux gcore and corefiles

[Andrew Cagney]

> On Sun, Mar 02, 2003 at 10:56:08AM -0500, Andrew Cagney wrote:
> 
>> >On Sat, Mar 01, 2003 at 10:39:00PM -0500, Daniel Jacobowitz wrote:
>> >
> 
>> >>> My instincts tell me that, to completly implement the above 
>> >>> functionality, GDB is always going to need libthread-db.  If GDB could 
>> >>> implement the above on a core file without using libthread-db, then GDB 
>> >>> could also implement the above on a live target also without using 
>> >>> libthread-db.  This is because a core file is always going to contain a 
>> >>> subset of the information made available via ptrace et.al.
> 
>> >
>> >
>> >Oh, and one other thing that I like to mention when this comes up.  My
>> >previous message was the implementation issues involved; this one's the
>> >motivational issue.  Thread_db is not, and can't/shouldn't be,
>> >available in a cross environment.  We have done a lot of work to make
>> >GDB read corefiles in a cross environment; and at MontaVista we've seen
>> >a large demand for this functionality from our customers.  So using
>> >thread_db with corefiles doesn't meet our (GDB developers') goals, I
>> >think.
> 
>> 
>> Sorry, on this point, I'm lost.  What are you suggesting here?
> 
> 
> My point is just that it's important for GDB to not require thread_db
> when dealing with core files.   A lot of people seem to use e.g. Solaris
> to debug core files and expect everything they would get from using GDB
> natively; that's their expectation and so far we've been able to stay
> pretty much there.

If that position means precluding certain native-only functionality?

Andrew

Re: expected behavior of GNU/Linux gcore and corefiles

[Daniel Jacobowitz]

On Sun, Mar 02, 2003 at 01:31:08PM -0500, Andrew Cagney wrote:
> >On Sun, Mar 02, 2003 at 10:56:08AM -0500, Andrew Cagney wrote:
> >
> >>>On Sat, Mar 01, 2003 at 10:39:00PM -0500, Daniel Jacobowitz wrote:
> >>>
> >
> >>>>> My instincts tell me that, to completly implement the above 
> >>>>> functionality, GDB is always going to need libthread-db.  If GDB 
> >>could >>> implement the above on a core file without using libthread-db, 
> >>then GDB >>> could also implement the above on a live target also without 
> >>using >>> libthread-db.  This is because a core file is always going to 
> >>contain a >>> subset of the information made available via ptrace et.al.
> >
> >>>
> >>>
> >>>Oh, and one other thing that I like to mention when this comes up.  My
> >>>previous message was the implementation issues involved; this one's the
> >>>motivational issue.  Thread_db is not, and can't/shouldn't be,
> >>>available in a cross environment.  We have done a lot of work to make
> >>>GDB read corefiles in a cross environment; and at MontaVista we've seen
> >>>a large demand for this functionality from our customers.  So using
> >>>thread_db with corefiles doesn't meet our (GDB developers') goals, I
> >>>think.
> >
> >>
> >>Sorry, on this point, I'm lost.  What are you suggesting here?
> >
> >
> >My point is just that it's important for GDB to not require thread_db
> >when dealing with core files.   A lot of people seem to use e.g. Solaris
> >to debug core files and expect everything they would get from using GDB
> >natively; that's their expectation and so far we've been able to stay
> >pretty much there.
> 
> If that position means precluding certain native-only functionality?

Not exactly; if that native-only functionality precludes keeping what
we have now for non-thread_db setups and extending it.  For instance, I
think we should always be able to print LWP state (assuming a 1-1
mapping when we have no thread_db to ask for more information).  And I
think we should add support for TLS based only on LWPs, since it's not
dependent on the thread manager.  If we can't "info mutex" on a cross
corefile, well, that's the same as now.

-- 
Daniel Jacobowitz
MontaVista Software                         Debian GNU/Linux Developer

Re: expected behavior of GNU/Linux gcore and corefiles

[Andrew Cagney]

>> 
>> If that position means precluding certain native-only functionality?
> 
> 
> Not exactly; if that native-only functionality precludes keeping what
> we have now for non-thread_db setups and extending it.



> For instance, I
> think we should always be able to print LWP state (assuming a 1-1
> mapping when we have no thread_db to ask for more information).  And I
> think we should add support for TLS based only on LWPs, since it's not
> dependent on the thread manager.  If we can't "info mutex" on a cross
> corefile, well, that's the same as now.

To draw this back to the original post.  What of a native GDB loading 
libthread-db over a core file to determine thread IDs and TLS?

Andrew

Re: expected behavior of GNU/Linux gcore and corefiles

[Daniel Jacobowitz]

On Sun, Mar 02, 2003 at 03:05:12PM -0500, Andrew Cagney wrote:
> 
> >>
> >>If that position means precluding certain native-only functionality?
> >
> >
> >Not exactly; if that native-only functionality precludes keeping what
> >we have now for non-thread_db setups and extending it.
> 
> 
> 
> >For instance, I
> >think we should always be able to print LWP state (assuming a 1-1
> >mapping when we have no thread_db to ask for more information).  And I
> >think we should add support for TLS based only on LWPs, since it's not
> >dependent on the thread manager.  If we can't "info mutex" on a cross
> >corefile, well, that's the same as now.
> 
> To draw this back to the original post.  What of a native GDB loading 
> libthread-db over a core file to determine thread IDs and TLS?

Well, it makes me pretty nervous - we (have to) load libthread_db from
/lib.  Not from solib-absolute-prefix.  I have several setups where I
debug core files from one machine on another.

That said, I don't see any reason we shouldn't support it.  We need to
get it to work, of course.

-- 
Daniel Jacobowitz
MontaVista Software                         Debian GNU/Linux Developer

Re: expected behavior of GNU/Linux gcore and corefiles

[Andrew Cagney]

> On Sat, Mar 01, 2003 at 10:16:39PM -0500, Andrew Cagney wrote:
> 
>> (notes from a hallway conversation, I had a recollection that, at some 
>> stage, the attempt to load libthread db over a core file on GNU/Linux 
>> was disabled.)
>> 
>> When using GDB on a live threaded program that puts all threads into 
>> tight infinite loops (while (1);), I'll do something like:
>> 
>>     $ ./a.out &
>>     Pid 1234
>>     $ gdb ./a.out 1234
>>     (gdb) info threads
>>     ....
>>     (gdb) quit
>> 
>> As a user I'd also expect sequences such as:
>> 
>>     $ kill -QUIT 1234
>>     $ gdb ./a.out core
>>     (gdb) info threads
>>     ....
>>     (gdb) quit
>> 
>> and:
>> 
>>     $ gcore 1234
>>     $ gdb ./a.out core
>>     (gdb) info threads
>>     ....
>>     (gdb) quit
> 
> 
> Hmm, what platform is that?  I've never seen a GNU/Linux
> implementation, but I'm a youngun.

FreeBSD, solaris.  This is the `expected behavior'.

>> 
>> and:
>> 
>>     $ gdb ./a.out 1234
>>     (gdb) gcore
>>     (gdb) quit
>>     $ gdb ./a.out core
>>     (gdb) info threads
>> 
>> to all come back with effectively the same output.  Further, on both 
>> live and corefile targets, I'd expect to be able to select/examine each 
>> thread vis:
>> 
>>     (gdb) thread 5
>>     11    i = i + 1;
>>     (gdb) list
>>     10    __thread__ i = 1;
>>     11    i = i + 1;
>>     (gdb) print i
>>     $1 = 1
>> 
>> (which would involve thread local storage).
>> 
>> Two problems come to mind:
>> 
>> - Is the kernel including all the raw information needed to do this in 
>> the core file?
> 
> 
> Some platforms yes, some platforms no.  I've had Linux patches to do
> this for some time.  I believe kernels as of 2.5.50 or so do it also.

Sounds like bugs.  If GDB is to give the user the best ptrace experience 
then GDB will need that information.

>> - For GDB to completly implement the above, is it forced to use 
>> libthread-db?
> 
> 
> Not necessarily.
> Here's the added value we get from libthread_db right now, on "normal"
> GNU/Linux threading (either LinuxThreads or NPTL, but not including the
> M-N stuff that NGPT does):
>   - Thread IDs, instead of just LWP IDs.  Big deal.

My point is that this _is_ a big deal.

GDB, in conjunction with libthread-db, should provide this interface. 
In fact, GDB+libthread-db should be providing a lot more:

	`info mutex'
	`info lwp'

It's the same situtation as for source code debugging.  While the user 
normally only views source code and variables (i.e., threads), GDB does 
also allow the user to open the bonnet and poke around in the registers 
and assembly (i.e., the lwp layer).

>   - Thread local storage offsets.
> 
> We could do find TLS areas with a per-architecture hook; libthread_db
> finds the TLS area by grubbing through the thread descriptor, but given
> that we have each LWP's state we could just do it in the
> platform-mandated way (i.e. examining the registers in question, etc.)
> 
> I don't know offhand if the necessary information for Alpha, i386, etc.
> TLS is in core files - i.e. platforms where it's more than just a
> register.

Per above, sounds like a bug.

[...]


> Thread_db is a great interface for querying threads.  It's not so good
> for debugging them; for a case in point see my messages about the FAIL
> at the end of print-threads.exp.  Thread_db does not interact well with
> debugging the thread manager itself, which is a very important feature
> for me.

And that was a bug.  I'm told that the new libthread-db code dropped the 
manager thread and, in the process, eliminated the problem.

Andrew

Re: expected behavior of GNU/Linux gcore and corefiles

[Andrew Cagney]

Dig dig, this has come up before.  Up until this change:

> 2002-01-04  Daniel Jacobowitz  <drow@mvista.com>
> 
>         * thread-db.c (thread_db_new_objfile): Do not enable thread_db
>         for core files.
> 

GDB was barfing when trying to load the thread DB.  Looking back through 
that thread, part of the problem appears to have stemed from GDB 
thinking that the GNU/Linux core file contained threads instead of LWPs :-(

Andrew

http://sources.redhat.com/ml/gdb-patches/2002-01/msg00017.html
http://sources.redhat.com/ml/gdb-patches/2001-12/msg00345.html

Re: expected behavior of GNU/Linux gcore and corefiles

[Daniel Jacobowitz]

On Sun, Mar 02, 2003 at 03:17:31PM -0500, Andrew Cagney wrote:
> Dig dig, this has come up before.  Up until this change:
> 
> >2002-01-04  Daniel Jacobowitz  <drow@mvista.com>
> >
> >        * thread-db.c (thread_db_new_objfile): Do not enable thread_db
> >        for core files.
> >
> 
> GDB was barfing when trying to load the thread DB.  Looking back through 
> that thread, part of the problem appears to have stemed from GDB 
> thinking that the GNU/Linux core file contained threads instead of LWPs :-(
> 
> Andrew
> 
> http://sources.redhat.com/ml/gdb-patches/2002-01/msg00017.html
> http://sources.redhat.com/ml/gdb-patches/2001-12/msg00345.html

Yep.  Among other things the problems arise because thread_db passes
requests to lin-lwp instead of properly along the target stack; lin-lwp
should be sitting in the "process" slot on the target stack.

There are also load/unload issues; observers are the perfect way to fix
that.

-- 
Daniel Jacobowitz
MontaVista Software                         Debian GNU/Linux Developer