Re: GDB doesn't display thread_id while debugging a core file

Re: GDB doesn't display thread_id while debugging a core file

[Icarus Sparry]

Last November there was some discussion on this topic, which ended with
Daniel Jacobowitz saying in

http://sourceware.org/ml/gdb/2007-08/msg00068.html

	If we assume that the host's libthread_db will either recognize
the
	core file and do the right thing, or reject the core file, then
we can
	write a small target layer that uses it on top of corelow.c in a
	similar way to how linux-thread-db.c / proc-service.c use
linux-nat.c.

	It's just a matter of testing that on a couple of different
setups,
	like LinuxThreads and cross debuggers, to see how it behaves.
Or
	doesn't behave.


Being able to access variables declared with __thread in core files
would certainly be useful. Could someone give some reasonable guess of
the amount of effort required to do this? In particular for a powerpc32
corefile from a linux process with NPTL being debugged on an x86 linux
box.

Thanks.

Re: GDB doesn't display thread_id while debugging a core file

[Michael Snyder]

On Mon, 2008-04-14 at 18:01 -0700, Icarus Sparry wrote:
> Last November there was some discussion on this topic, which ended with
> Daniel Jacobowitz saying in
> 
> http://sourceware.org/ml/gdb/2007-08/msg00068.html
> 
> 	If we assume that the host's libthread_db will either recognize
> the
> 	core file and do the right thing, or reject the core file, then
> we can
> 	write a small target layer that uses it on top of corelow.c in a
> 	similar way to how linux-thread-db.c / proc-service.c use
> linux-nat.c.
> 
> 	It's just a matter of testing that on a couple of different
> setups,
> 	like LinuxThreads and cross debuggers, to see how it behaves.
> Or
> 	doesn't behave.
> 
> 
> Being able to access variables declared with __thread in core files
> would certainly be useful. Could someone give some reasonable guess of
> the amount of effort required to do this? In particular for a powerpc32
> corefile from a linux process with NPTL being debugged on an x86 linux
> box.

Off hand, I would not expect the libthread_db library to 
be able to do anything useful with a corefile from a different
architecture.

RE: GDB doesn't display thread_id while debugging a core file

[Icarus Sparry]

> -----Original Message-----
> From: Michael Snyder [mailto:msnyder@specifix.com]
> Sent: Monday, April 14, 2008 6:24 PM
> To: Icarus Sparry
> Cc: gdb@sourceware.org
> Subject: Re: GDB doesn't display thread_id while debugging a core file
> 
> On Mon, 2008-04-14 at 18:01 -0700, Icarus Sparry wrote:
> > Last November there was some discussion on this topic, which ended
with
> > Daniel Jacobowitz saying in
> >
> > http://sourceware.org/ml/gdb/2007-08/msg00068.html
> >
> > 	If we assume that the host's libthread_db will either recognize
> > the
> > 	core file and do the right thing, or reject the core file, then
> > we can
> > 	write a small target layer that uses it on top of corelow.c in a
> > 	similar way to how linux-thread-db.c / proc-service.c use
> > linux-nat.c.
> >
> > 	It's just a matter of testing that on a couple of different
> > setups,
> > 	like LinuxThreads and cross debuggers, to see how it behaves.
> > Or
> > 	doesn't behave.
> >
> >
> > Being able to access variables declared with __thread in core files
> > would certainly be useful. Could someone give some reasonable guess
of
> > the amount of effort required to do this? In particular for a
powerpc32
> > corefile from a linux process with NPTL being debugged on an x86
linux
> > box.
> 
> Off hand, I would not expect the libthread_db library to
> be able to do anything useful with a corefile from a different
> architecture.

If I understand what Daniel was saying correctly, the libthread_db file
would be an x86 shared library, but it would be configured to handle
ppc32 elf core files, in the same way as the executable
powerpc-linux-gdb program that currently we have running on the x86
machines.

Re: GDB doesn't display thread_id while debugging a core file

[Daniel Jacobowitz]

On Mon, Apr 14, 2008 at 08:04:52PM -0700, Icarus Sparry wrote:
> If I understand what Daniel was saying correctly, the libthread_db file
> would be an x86 shared library, but it would be configured to handle
> ppc32 elf core files, in the same way as the executable
> powerpc-linux-gdb program that currently we have running on the x86
> machines.

No.  You would have to rely on the native x86 library luckily doing
the right thing.

-- 
Daniel Jacobowitz
CodeSourcery

Re: GDB doesn't display thread_id while debugging a core file

[Michael Snyder]

On Mon, 2008-04-14 at 23:10 -0400, Daniel Jacobowitz wrote:
> On Mon, Apr 14, 2008 at 08:04:52PM -0700, Icarus Sparry wrote:
> > If I understand what Daniel was saying correctly, the libthread_db file
> > would be an x86 shared library, but it would be configured to handle
> > ppc32 elf core files, in the same way as the executable
> > powerpc-linux-gdb program that currently we have running on the x86
> > machines.
> 
> No.  You would have to rely on the native x86 library luckily doing
> the right thing.

And that's highly unlikely.  Data structures would likely
have different fields and sizes and such.

If someone wanted to make a project of building a cross-libthread-db, 
I'm sure it could be done -- but it would be a project.

RE: GDB doesn't display thread_id while debugging a core file

[Icarus Sparry]

> -----Original Message-----
> From: Michael Snyder [mailto:msnyder@specifix.com]
> Sent: Tuesday, April 15, 2008 11:05 AM
> To: Daniel Jacobowitz
> Cc: Icarus Sparry; gdb@sourceware.org
> Subject: Re: GDB doesn't display thread_id while debugging a core file
> 
> On Mon, 2008-04-14 at 23:10 -0400, Daniel Jacobowitz wrote:
> > On Mon, Apr 14, 2008 at 08:04:52PM -0700, Icarus Sparry wrote:
> > > If I understand what Daniel was saying correctly, the libthread_db
> file
> > > would be an x86 shared library, but it would be configured to
handle
> > > ppc32 elf core files, in the same way as the executable
> > > powerpc-linux-gdb program that currently we have running on the
x86
> > > machines.
> >
> > No.  You would have to rely on the native x86 library luckily doing
> > the right thing.
> 
> And that's highly unlikely.  Data structures would likely
> have different fields and sizes and such.
> 
> If someone wanted to make a project of building a cross-libthread-db,
> I'm sure it could be done -- but it would be a project.

I am obviously not making myself clear! It is "obvious" to me that one
needs a "cross-libthread-db" library, for example one that is an x86
executable but is configured to handle threading on a ppc32 with NPTL as
I described above. I do not expect things like this to come for free on
most systems. The question I hoped I had asked was "How much effort is
it?"

The program I am interested in at the moment only has a single __thread
variable, which is used to index many arrays. I have a number of
choices, ranging from write a function in gdb to get the value of this
particular variable, to fixing gdb so it can get the information
correctly from the core for all thread local variables. The latter is
going to be more work, but makes gdb a better debugger for everyone. 

Re: GDB doesn't display thread_id while debugging a core file

[Daniel Jacobowitz]

On Tue, Apr 15, 2008 at 01:16:41PM -0700, Icarus Sparry wrote:
> The program I am interested in at the moment only has a single __thread
> variable, which is used to index many arrays. I have a number of
> choices, ranging from write a function in gdb to get the value of this
> particular variable, to fixing gdb so it can get the information
> correctly from the core for all thread local variables. The latter is
> going to be more work, but makes gdb a better debugger for everyone. 

We've talked about accessing __thread storage without passing through
libthread_db.  In my opinion, that is likely to be a better solution.
However, it's a bit tricky to figure out the module numbers assigned
by the C library in this case...

-- 
Daniel Jacobowitz
CodeSourcery

Re: GDB doesn't display thread_id while debugging a core file

[msnyder]

Daniel writes:
> On Mon, Aug 06, 2007 at 06:20:58PM -0300, Carlos Eduardo Seo wrote:
> > > The thread ID is produced by NPTL's libthread_db library,
> > Yes, and this number is also an address, the thread pointer used to
> > access the TCB, as it is defined in the TLS definitions in the ABI.
> > So, there's no magic involved. :)
>
> That's incorrect.  The fact that pthread_self, and thus libthread_db,
> use a pointer to the "struct pthread" as their thread ID is an
> internal implementation detail of NPTL, i.e. subject to change.  That
> means it's the province of libthread_db.

Agree with Daniel.  This is an abi spec.  We are not allowed
to presume that we know the internal details of implementation.
That is what the libthread_db API is for.

> > This may be a silly question, but, how can we use libthread_db in
> > order to get the thread ID from threads within a core file?
>
> If we assume that the host's libthread_db will either recognize the
> core file and do the right thing, or reject the core file, then we can
> write a small target layer that uses it on top of corelow.c in a
> similar way to how linux-thread-db.c / proc-service.c use linux-nat.c.

Daniel, I think it's even simpler than that.

libthread_db works via callbacks to gdb that do nothing more
complicated than read memory.  GDB can read memory from the corefile.

The only question is whether the versions of libthread and
libthread_db match, on the host and (corefile-generating) target.

If they do, "it should just work" (tm).

And if there is a version string somewhere in the library's read/write
memory sections, it should also exist in the corefile.

Michael

Re: GDB doesn't display thread_id while debugging a core file

[Daniel Jacobowitz]

On Wed, Aug 08, 2007 at 12:12:56PM -0700, Michael Snyder wrote:
> Daniel, I think it's even simpler than that.
> 
> libthread_db works via callbacks to gdb that do nothing more
> complicated than read memory.  GDB can read memory from the corefile.

Yes, but our current linux thread-db support is hardwired to sit on
top of a live target :-)  It's fixable.

> And if there is a version string somewhere in the library's read/write
> memory sections, it should also exist in the corefile.

Unfortunately I suspect it is in the read-only sections.  Maybe the
--build-id stuff Roland has been working on will suffice to prevent
mismatches though.

Someone want to give it a try? :-)

-- 
Daniel Jacobowitz
CodeSourcery

GDB doesn't display thread_id while debugging a core file

[Carlos Eduardo Seo]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hello

While debugging a core file from a multi-threaded application, I noticed
that GDB doesn't show the thread_id:

(gdb) thread
[Current thread is 4 (process 8466)]

instead of something like this:

(gdb) thread
[Current thread is 5 (Thread 2199033284976 (LWP 4124))]

As far as I'm aware of, this big number is actually the thread pointer
minus an offset value.

For ppc64, I can get the thread pointer from register r13 (which is in
the core file) and then subtract (TLS_PRE_TCB_SIZE + 0x7000) to get the
thread_id. 0x7000 is fixed per the ABI and TLS_PRE_TCB_SIZE can be
calculated from the size of two structs defined in GLIBC (so, I must
assume that the user has a GLIBC with debug information in order to get
the size of those structs).

I know this is an issue that has a solution which is arch-dependent, so
my concern here is to make a fix that's easily extensible to other archs.

Is a solution like this acceptable?

Thanks and regards,

- --
Carlos Eduardo Seo
Software Engineer
IBM Linux Technology Center
E-Mail: cseo@linux.vnet.ibm.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.5 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFGt4QFqvq7Aov/qQARAt12AJ9szUdI05rLYuJ+r+oiCwPKZjI+BwCfaHpz
YUp9BcgLXvfsoWJ7Id9DDBk=
=ayr3
-----END PGP SIGNATURE-----

Re: GDB doesn't display thread_id while debugging a core file

[Daniel Jacobowitz]

On Mon, Aug 06, 2007 at 05:26:45PM -0300, Carlos Eduardo Seo wrote:
> Is a solution like this acceptable?

No, sorry.  The thread ID is produced by NPTL's libthread_db library,
which we do not use on core files because it might not match the
version in use when the program was running; and thus it might produce
garbage (or more often, fail gracefully).

Now that most platforms have moved from LinuxThreads to NPTL, this
might be worth another look.  Opportunistically, sometimes we can use
libthread_db and get sensible answers.

-- 
Daniel Jacobowitz
CodeSourcery

Re: GDB doesn't display thread_id while debugging a core file

[Carlos Eduardo Seo]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Daniel Jacobowitz wrote:
> The thread ID is produced by NPTL's libthread_db library,
Yes, and this number is also an address, the thread pointer used to
access the TCB, as it is defined in the TLS definitions in the ABI.
So, there's no magic involved. :) I can read that value from the
register and get the thread ID from there. I know that this approach
works on ppc32, ppc64 and s390. Don't know about other archs, though.
That's why I'm worried about providing a fix that can be extended to
other archs.

> Now that most platforms have moved from LinuxThreads to NPTL, this
> might be worth another look.  Opportunistically, sometimes we can use
> libthread_db and get sensible answers.
This may be a silly question, but, how can we use libthread_db in
order to get the thread ID from threads within a core file?

Regards,

- --
Carlos Eduardo Seo
Software Engineer
IBM Linux Technology Center Brazil
Phone: +55 19-2132-4339
T/L: 839-4339
E-Mail: cseo@linux.vnet.ibm.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.5 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFGt5C5qvq7Aov/qQARAuYjAJ4nx36Jm1Ix2Mgmz3KlIsTptvBRlACgjIQT
vjdq9KtChupvNmrtxHloKAQ=
=/E52
-----END PGP SIGNATURE-----

Re: GDB doesn't display thread_id while debugging a core file

[Daniel Jacobowitz]

On Mon, Aug 06, 2007 at 06:20:58PM -0300, Carlos Eduardo Seo wrote:
> > The thread ID is produced by NPTL's libthread_db library,
> Yes, and this number is also an address, the thread pointer used to
> access the TCB, as it is defined in the TLS definitions in the ABI.
> So, there's no magic involved. :)

That's incorrect.  The fact that pthread_self, and thus libthread_db,
use a pointer to the "struct pthread" as their thread ID is an
internal implementation detail of NPTL, i.e. subject to change.  That
means it's the province of libthread_db.  We don't have any flag to
indicate that it will be the case unless we ask the library for the ID.

For instance, if you do this then you'll start showing the TLS pointer
as a thread ID for LinuxThreads applications.  But they used a
different numbering scheme.

An architecture method to get at the thread pointer directly would
still be useful, by the way - we've talked about that for accessing
__thread variables without debug info.

> > Now that most platforms have moved from LinuxThreads to NPTL, this
> > might be worth another look.  Opportunistically, sometimes we can use
> > libthread_db and get sensible answers.
> This may be a silly question, but, how can we use libthread_db in
> order to get the thread ID from threads within a core file?

If we assume that the host's libthread_db will either recognize the
core file and do the right thing, or reject the core file, then we can
write a small target layer that uses it on top of corelow.c in a
similar way to how linux-thread-db.c / proc-service.c use linux-nat.c.

It's just a matter of testing that on a couple of different setups,
like LinuxThreads and cross debuggers, to see how it behaves.  Or
doesn't behave.

-- 
Daniel Jacobowitz
CodeSourcery