thread ptids when debugging from core file (x86-linux)

thread ptids when debugging from core file (x86-linux)

[Joel Brobecker]

Hello,

We recently noticed that Ada task switching doesn't work when debugging
a program using a core file. This is on x86-linux.

ada-tasks.c is just a module similar to thread*.c. It shows the list of
Ada tasks (high-level equivalent of C threads), and allows a user to
switch from one task to the other. Ada tasks are usually mapped
one-on-one on threads, so task switching is implemented by using
the thread layer: Given a certain task to switch to, we deduce from
some data stored in the GNAT runtime memory (+ the inferior ptid)
the PTID of the associated thread, and then ask the thread layer
to switch to that PTID.

Unfortunately, that doesn't work when debugging from a core file,
because the PTID of the threads are not the same as when debugging
live. According to corelow.c:add_to_thread_list(), the thread PTID
is a degenerated {pid, 0, 0} where pid is just the number in the section
name where the thread has been dumped (ie section ".reg/1234" => thread
ptid = {1234, 0, 0}).

This breaks the mapping that we computed in the ada-tasks layer,
and thus causes the task switch to fail, because ada-tasks asks
thread to switch to a thread it doesn't know about.

I was wondering whether I could be able to dig the original thread
PTIDs back from the data stored in the core file. Would anybody know?
Is there some sort of documentation on how this core file is created
and what information it contains?  How are the number in the .reg/n
sections generated? Are they PID+n where PID is the inferior PID and
n the thread number?

If I can't find the original PTIDs, I really don't know how I am going
to be able to restore the mapping between the Ada tasks and the
underlying threads. Does anybody have some ideas of how this could be
done? We don't have much information in the GNAT runtime. It's all
stored in the Ada Task Control Block (ATCB) and basically our only
link is that thread ID.

Thanks,
-- 
Joel

Re: thread ptids when debugging from core file (x86-linux)

[Nathan J. Williams]

Joel Brobecker <brobecker@gnat.com> writes:

> According to corelow.c:add_to_thread_list(), the thread PTID
> is a degenerated {pid, 0, 0} where pid is just the number in the section
> name where the thread has been dumped (ie section ".reg/1234" => thread
> ptid = {1234, 0, 0}).

I've always assumed that corelow.c's mapping between .reg sections and
threads was intended as a minimal fallback. The module I wrote for
NetBSD's threads (which I'll get into shape for the FSF tree Real Soon
Now) punts that list and generates its own list of threads from the
memory structures of libpthread in the core file, just as it would for
a live process.

        - Nathan

Re: thread ptids when debugging from core file (x86-linux)

[Andrew Cagney]

> Joel Brobecker <brobecker@gnat.com> writes:
> 
> 
>>> According to corelow.c:add_to_thread_list(), the thread PTID
>>> is a degenerated {pid, 0, 0} where pid is just the number in the section
>>> name where the thread has been dumped (ie section ".reg/1234" => thread
>>> ptid = {1234, 0, 0}).
> 
> 
> I've always assumed that corelow.c's mapping between .reg sections and
> threads was intended as a minimal fallback. The module I wrote for
> NetBSD's threads (which I'll get into shape for the FSF tree Real Soon
> Now) punts that list and generates its own list of threads from the
> memory structures of libpthread in the core file, just as it would for
> a live process.

Yes.

Joel, this observation of yours is the key:

> Unfortunately, that doesn't work when debugging from a core file,
> because the PTID of the threads are not the same as when debugging
> live. 

The best way to appreciate this is to consider:

(gdb) attach program
(gdb) info threads
thread output #1
(gdb) info lwp -- yes ok, that cmd doesn't exist :-)
lwp output #1 (different to the thread output)
(gdb) gcore program.gcore
(gdb) detach
Segmentation fault core dumped to program.core
(gdb) corefile program.core
(gdb) info threads
identical thread output #2
(gdb) info lwp
identical lwp output #2
(gdb) corefile program.gcore
(gdb) info threads
also identical output #3
(gdb) info lwp
also identical lwp output #3

Andrew

Re: thread ptids when debugging from core file (x86-linux)

[Joel Brobecker]

> >I've always assumed that corelow.c's mapping between .reg sections and
> >threads was intended as a minimal fallback. The module I wrote for
> >NetBSD's threads (which I'll get into shape for the FSF tree Real Soon
> >Now) punts that list and generates its own list of threads from the
> >memory structures of libpthread in the core file, just as it would for
> >a live process.
> 
> Yes.

So do you think I'll have to dig into the inferior memory for data used
by the libpthread library to find that information? Is this something
that would be portable across NPTL versions? Supposing we want to go
that route, does anybody know where to find some documentation (or some
code) that exposes the data structures, so I can try digging into them?

> Joel, this observation of yours is the key:
> 
> >Unfortunately, that doesn't work when debugging from a core file,
> >because the PTID of the threads are not the same as when debugging
> >live. 
> 
> The best way to appreciate this is to consider:
> 
> (gdb) attach program
> (gdb) info threads
> thread output #1
> (gdb) info lwp -- yes ok, that cmd doesn't exist :-)
> lwp output #1 (different to the thread output)
> (gdb) gcore program.gcore
> (gdb) detach
> Segmentation fault core dumped to program.core
> (gdb) corefile program.core
> (gdb) info threads
> identical thread output #2
> (gdb) info lwp
> identical lwp output #2
> (gdb) corefile program.gcore
> (gdb) info threads
> also identical output #3
> (gdb) info lwp
> also identical lwp output #3

I am sorry, I don't understand what you are saying. What do you
mean by "identical output #n"? What is it identical to? to output
#1? (right now, it is not).

Thanks,
-- 
Joel

Re: thread ptids when debugging from core file (x86-linux)

[Marcel Moolenaar]

On Fri, Aug 06, 2004 at 10:58:18AM -0700, Joel Brobecker wrote:
> > >I've always assumed that corelow.c's mapping between .reg sections and
> > >threads was intended as a minimal fallback. The module I wrote for
> > >NetBSD's threads (which I'll get into shape for the FSF tree Real Soon
> > >Now) punts that list and generates its own list of threads from the
> > >memory structures of libpthread in the core file, just as it would for
> > >a live process.
> > 
> So do you think I'll have to dig into the inferior memory for data used
> by the libpthread library to find that information? Is this something
> that would be portable across NPTL versions? Supposing we want to go
> that route, does anybody know where to find some documentation (or some
> code) that exposes the data structures, so I can try digging into them?

[not strictly Linux, but more genericly]

But how do you map register contents in the various .reg/# sections to
the right thread if there's no way to correlate .reg/# sections to light-
weight processes? Would this in fact imply that the IDs needs to match?

Or am I missing something here?

-- 
 Marcel Moolenaar	  USPA: A-39004		 marcel@xcllnt.net

Re: thread ptids when debugging from core file (x86-linux)

[Joel Brobecker]

> > So do you think I'll have to dig into the inferior memory for data used
> > by the libpthread library to find that information? Is this something
> > that would be portable across NPTL versions? Supposing we want to go
> > that route, does anybody know where to find some documentation (or some
> > code) that exposes the data structures, so I can try digging into them?
> 
> [not strictly Linux, but more genericly]
> 
> But how do you map register contents in the various .reg/# sections to
> the right thread if there's no way to correlate .reg/# sections to light-
> weight processes? Would this in fact imply that the IDs needs to match?
> 
> Or am I missing something here?

If anybody is missing something, it must be me. This is all new
territory to me. I'd be interested in seeing how Nathan did it for
NetBSD, that would give me some clues. But I think I also need to
look closer at the NPTL and core files... I'm still not sure that
doing what I want to do will be possible at all.

-- 
Joel

Re: thread ptids when debugging from core file (x86-linux)

[Nathan J. Williams]

Marcel Moolenaar <marcel@xcllnt.net> writes:

> On Fri, Aug 06, 2004 at 10:58:18AM -0700, Joel Brobecker wrote:
> > > >I've always assumed that corelow.c's mapping between .reg sections and
> > > >threads was intended as a minimal fallback. The module I wrote for
> > > >NetBSD's threads (which I'll get into shape for the FSF tree Real Soon
> > > >Now) punts that list and generates its own list of threads from the
> > > >memory structures of libpthread in the core file, just as it would for
> > > >a live process.
> > > 
> > So do you think I'll have to dig into the inferior memory for data used
> > by the libpthread library to find that information? Is this something
> > that would be portable across NPTL versions? Supposing we want to go
> > that route, does anybody know where to find some documentation (or some
> > code) that exposes the data structures, so I can try digging into them?
> 
> [not strictly Linux, but more genericly]
> 
> But how do you map register contents in the various .reg/# sections to
> the right thread if there's no way to correlate .reg/# sections to light-
> weight processes? Would this in fact imply that the IDs needs to match?

The Linux thread code (thread-db.c) already knows how to poke around
in memory and find all the threads. That memory also includes the
thread-to-LWP mapping, in some form. The trick is getting the process
callbacks in proc-service.c to look up registers of LWPs in the
matching core file section. Here's the getregs callback in
nbsd-thread.c:

nbsd_thread_proc_getregs (void *arg, int regset, int lwp, void *buf)
{
  struct cleanup *old_chain;
  int ret;

  old_chain = save_inferior_ptid ();
  
  if (target_has_execution)
    {
      inferior_ptid = BUILD_LWP (lwp, main_ptid);
      child_ops.to_fetch_registers (-1);
    }
  else
    {
      inferior_ptid = pid_to_ptid ((lwp << 16) | GET_PID (main_ptid));
      orig_core_ops.to_fetch_registers (-1);
    }

  ret = 0;
  switch (regset)
    {
    case 0:
      fill_gregset ((gregset_t *)buf, -1);
      break;
    case 1:
      fill_fpregset ((fpregset_t *)buf, -1);
      break;
    default: /* XXX need to handle other reg sets: SSE, AltiVec, etc. */
      ret = TD_ERR_INVAL;
    }

  do_cleanups (old_chain);
}

On Solaris and NetBSD, the .reg/NNN sections have NNN values equal to
PID + (LWPID << 16), so it's straightforward to map from a LWP to a
core section. I'm not sure how Linux names them.

It also helps to use init_thread_list() to blow away the list that
corelow.c generates.

        - Nathan

Re: thread ptids when debugging from core file (x86-linux)

[Andrew Cagney]

>>Joel, this observation of yours is the key:
>>> 
>>
>>>> >Unfortunately, that doesn't work when debugging from a core file,
>>>> >because the PTID of the threads are not the same as when debugging
>>>> >live. 
>>
>>> 
>>> The best way to appreciate this is to consider:
>>> 
>>> (gdb) attach program
>>> (gdb) info threads
>>> thread output #1
>>> (gdb) info lwp -- yes ok, that cmd doesn't exist :-)
>>> lwp output #1 (different to the thread output)
>>> (gdb) gcore program.gcore
>>> (gdb) detach
>>> Segmentation fault core dumped to program.core
>>> (gdb) corefile program.core
>>> (gdb) info threads
>>> identical thread output #2
>>> (gdb) info lwp
>>> identical lwp output #2
>>> (gdb) corefile program.gcore
>>> (gdb) info threads
>>> also identical output #3
>>> (gdb) info lwp
>>> also identical lwp output #3
> 
> 
> I am sorry, I don't understand what you are saying. What do you
> mean by "identical output #n"? What is it identical to? to output
> #1? (right now, it is not).

Yes.  ``thread output'' #1, #2, an #3 should be identical.  Similarly 
``lwp output'' 1-3 should also be identical.

The're not, its a bug (and one everyone has been trying to sweep under 
the carpet).  The corefile code should be loading libthread_db so that 
GDB can use the .reg/<LWPID> along with memory information to re-create 
the thread list.  At one stage it even did that but it was then disabled 
because it caused a core dump and ...

Andrew

Re: thread ptids when debugging from core file (x86-linux)

[Marcel Moolenaar]

On Fri, Aug 06, 2004 at 03:10:52PM -0400, Nathan J. Williams wrote:
> Marcel Moolenaar <marcel@xcllnt.net> writes:
> 
> The Linux thread code (thread-db.c) already knows how to poke around
> in memory and find all the threads. That memory also includes the
> thread-to-LWP mapping, in some form. The trick is getting the process
> callbacks in proc-service.c to look up registers of LWPs in the
> matching core file section.

Yes.

> On Solaris and NetBSD, the .reg/NNN sections have NNN values equal to
> PID + (LWPID << 16), so it's straightforward to map from a LWP to a
> core section. I'm not sure how Linux names them.

This doesn't work on FreeBSD and I also don't think it works on Linux.
PIDs go beyond 2^16, and LWPIDs are higher numbered than PIDs on FreeBSD.
I solved it on FreeBSD (which given the release schedule means that
impact had to be minimal) by using the PID field in the note section to
hold the LWPID and redefinition inferior_ptid in the proc_service "layer"
to use the LWPID as the PID. This yielded a minimal impact for the core,
inferior and remote targets as well as the ptrace(2) syscall.

> It also helps to use init_thread_list() to blow away the list that
> corelow.c generates.

I happen to do that on FreeBSD as well, but that's mostly because of
MxN threading. For unbound threads the correlation between LWPID and
TID is non-constant and there's significant overhead to have the PTID
include both. So, the PTID only has the TID (and of course the PID,
but that's constant), which means that the thread list created by
corelow, which has the LWPID in the PID field on FreeBSD, needs to
be replaced by a thread list that has TID filled in. For 1xN or 1x1
threading you don't have to throw away the thread list that corelow.c
creates, but the purpose of libthread_db is to abstract the difference
so we don't try to avoid it :-)

Anyway, to get back to Joel's original mail: a logical solution that
would minimize knowledge of the threading implementation is to use
the libthread_db API to access thread information, and to keep the
Ada task to TID mapping inside the Ada-tasks layer. If the Ada-tasks
layer is a target on top of the thread target, then it makes sense
to have the thread list in GDB use Ada-tasks specific IDs. It may
then even be possible to use target_beneath->[method]() without
having to use the libthread_db API.

Put differently, one can view the Ada-tasks layer as a 1:1 threading
target on top of the threads target.

Just a thought,

-- 
 Marcel Moolenaar	  USPA: A-39004		 marcel@xcllnt.net