gdb / thread_db / multiple ABI question

gdb / thread_db / multiple ABI question

[Steve Ellcey]

I have a question about debugging, the thread_db library, and multiple ABI's.
I am trying to support an ILP32 ABI to the aarch64 architecture.  Now I am
still building gdb as a LP64 object.  In most cases this seems fine, I can use
an LP64 gdb to debug an ILP32 object.  But when I have threaded objects and
need to use libthread_db.so, things don't seem to work.  When I try this
and set libthread_db_debug to 1, I get:

Trying host libthread_db library: /home/sellcey/gdb-ilp32/install/libilp32/libth
read_db.so.1.
dlopen failed: /home/sellcey/gdb-ilp32/install/libilp32/libthread_db.so.1: wrong
 ELF class: ELFCLASS32.
thread_db_load_search returning 0
warning: Unable to find libthread_db matching inferior's thread library, thread 
debugging will not be available.

It makes sense that the LP64 gdb cannot dlopen the ILP32 thread_db that
corresponds to the ILP32 thread library that the program being debugged
uses but I don't know what to do about it.

I looked at x86 and when I compile a 32 bit program there and then debug
it with a 64 bit gdb I see:

	[Thread debugging using libthread_db enabled]
	Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".

So I am guessing that aarch64 needs to use the 64 bit libthread_db.so.1 even
when debugging 32 bit programs, is that right?  What do I need to do to tell
gdb to do that?  I tried adding /home/sellcey/gdb-ilp32/install/lib64 to
solib-search-path but that did not help.  I am not sure what triggers the
difference in behaviour between aarch64 and x86.

Steve Ellcey
sellcey@cavium.com

Re: gdb / thread_db / multiple ABI question

[Pedro Alves]

On 07/07/2017 07:20 PM, Steve Ellcey wrote:

> It makes sense that the LP64 gdb cannot dlopen the ILP32 thread_db that
> corresponds to the ILP32 thread library that the program being debugged
> uses but I don't know what to do about it.
> 
> I looked at x86 and when I compile a 32 bit program there and then debug
> it with a 64 bit gdb I see:
> 
> 	[Thread debugging using libthread_db enabled]
> 	Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
> 
> So I am guessing that aarch64 needs to use the 64 bit libthread_db.so.1 even
> when debugging 32 bit programs, is that right?  

Yes.  

Though I have no idea whether an Aarch64 build of libthread_db is
capable of biarch debugging.  It may be that GDB is already trying to
the load the 64-bit libthread_db, but then libthread_db rejects
the inferior for some reason.  The "set debug libthread-db" option should
help you figure that out.  See below.

> What do I need to do to tell
> gdb to do that?  I tried adding /home/sellcey/gdb-ilp32/install/lib64 to
> solib-search-path but that did not help.  I am not sure what triggers the
> difference in behaviour between aarch64 and x86.

See:

(gdb) apropos libthread
info auto-load libthread-db -- Print the list of loaded inferior specific libthread_db
set auto-load libthread-db -- Enable or disable auto-loading of inferior specific libthread_db
set debug libthread-db -- Set libthread-db debugging
set libthread-db-search-path -- Set search path for libthread_db
show auto-load libthread-db -- Show whether auto-loading inferior specific libthread_db is enabled
show debug libthread-db -- Show libthread-db debugging
show libthread-db-search-path -- Show the current search path or libthread_db

The related GDB code is in gdb/linux-thread-db.c.

Thanks,
Pedro Alves

Re: gdb / thread_db / multiple ABI question

[Steve Ellcey]

On Fri, 2017-07-07 at 19:36 +0100, Pedro Alves wrote:
> 
> Though I have no idea whether an Aarch64 build of libthread_db is
> capable of biarch debugging.  It may be that GDB is already trying to
> the load the 64-bit libthread_db, but then libthread_db rejects
> the inferior for some reason.  The "set debug libthread-db" option
> should
> help you figure that out.  See below.

OK, setting libthread-db-search-path and debug libthread-db got me
a bit further.  I don't get any errors or warnings about not being
able to find or load libthread_db.  It's still not reading everything
correctly though:

(gdb) info threads
Found 0 new threads in iteration 0.
  Id   Target Id         Frame 
* 3    Thread 0xf739ef70 (LWP 8660) 0x00400680 in doSomeThing ()
  4    Thread 0xf69bf490 (LWP 8662) warning: Couldn't find general-purpose registers in core file.
PC register is not available


I don't see any code in gdb/linux-thread-db.c that seems to be
conditional on the word size (ILP32 vs LP64).  I also don't see
anything in glibc nptl-db directory that appears to be conditional on
word size.  Things like pthread_t and __SIZEOF_PTHREAD_ATTR_T are 
different for the two different ABIs though.  But that is true for x86
too and things seem to be working there.  I am not sure what (if
anything) would need to be done to libthread_db to support biarch
debugging.

Steve Ellcey
sellcey@cavium.com

Re: gdb / thread_db / multiple ABI question

[Yao Qi]

Steve Ellcey <sellcey@cavium.com> writes:

> I don't see any code in gdb/linux-thread-db.c that seems to be
> conditional on the word size (ILP32 vs LP64).  I also don't see

Take a look at aarch64-linux-nat.c:ps_get_thread_area.  It calls
aarch64_ps_get_thread_area.

> anything in glibc nptl-db directory that appears to be conditional on
> word size.  Things like pthread_t and __SIZEOF_PTHREAD_ATTR_T are 
> different for the two different ABIs though.  But that is true for x86
> too and things seem to be working there.  I am not sure what (if
> anything) would need to be done to libthread_db to support biarch
> debugging.

AArch64 linux GDB is able to debug arm-linux 32-bit program.

-- 
Yao (齐尧)

Re: gdb / thread_db / multiple ABI question

[Steve Ellcey]

On Mon, 2017-07-10 at 08:39 +0100, Yao Qi wrote:
> Steve Ellcey <sellcey@cavium.com> writes:
> 
> > 
> > I don't see any code in gdb/linux-thread-db.c that seems to be
> > conditional on the word size (ILP32 vs LP64).  I also don't see
> Take a look at aarch64-linux-nat.c:ps_get_thread_area.  It calls
> aarch64_ps_get_thread_area.
> 
> > 
> > anything in glibc nptl-db directory that appears to be conditional
> > on
> > word size.  Things like pthread_t and __SIZEOF_PTHREAD_ATTR_T are 
> > different for the two different ABIs though.  But that is true for
> > x86
> > too and things seem to be working there.  I am not sure what (if
> > anything) would need to be done to libthread_db to support biarch
> > debugging.
> AArch64 linux GDB is able to debug arm-linux 32-bit program.

OK, there are actually changes in this file where as part of the
earlier ILP32 work, we changed:

	(gdbarch_bfd_arch_info (gdbarch)->bits_per_word == 32)
to
	(gdbarch_bfd_arch_info(gdbarch)->arch == bfd_arch_arm)

But maybe, for ps_get_thread_area (and other places?) we need the
original test for word size and not the arch check.

Steve Ellcey
sellcey@cavium.com

Re: gdb / thread_db / multiple ABI question

[Andreas Schwab]

On Jul 07 2017, Steve Ellcey <sellcey@cavium.com> wrote:

>  I am not sure what (if anything) would need to be done to
> libthread_db to support biarch debugging.

libthread_db is looking up the offsets and sizes in the symbol table of
the target's libpthread.

Andreas.

-- 
Andreas Schwab, SUSE Labs, schwab@suse.de
GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE  1748 E4D4 88E3 0EEA B9D7
"And now for something completely different."

Re: gdb / thread_db / multiple ABI question

[Pedro Alves]

> OK, setting libthread-db-search-path and debug libthread-db got me
> a bit further.  I don't get any errors or warnings about not being
> able to find or load libthread_db.  It's still not reading everything
> correctly though:
> 
> (gdb) info threads
> Found 0 new threads in iteration 0.
>   Id   Target Id         Frame 
> * 3    Thread 0xf739ef70 (LWP 8660) 0x00400680 in doSomeThing ()
>   4    Thread 0xf69bf490 (LWP 8662) warning: Couldn't find general-purpose registers in core file.
> PC register is not available

Do you get that same warning without libthread_db?  It sounds more
like an issue on the gdb<->bfd side than on the libthread_db side
of things.  (Put a breakpoint on "warning".  I believe it comes from
corelow.c:get_core_register_section.)

Thanks,
Pedro Alves

Re: gdb / thread_db / multiple ABI question

[Steve Ellcey]

On Mon, 2017-07-10 at 10:43 +0100, Pedro Alves wrote:
> > 
> > OK, setting libthread-db-search-path and debug libthread-db got me
> > a bit further.  I don't get any errors or warnings about not being
> > able to find or load libthread_db.  It's still not reading
> > everything
> > correctly though:
> > 
> > (gdb) info threads
> > Found 0 new threads in iteration 0.
> >   Id   Target Id         Frame 
> > * 3    Thread 0xf739ef70 (LWP 8660) 0x00400680 in doSomeThing ()
> >   4    Thread 0xf69bf490 (LWP 8662) warning: Couldn't find general-
> > purpose registers in core file.
> > PC register is not available
> Do you get that same warning without libthread_db?  It sounds more
> like an issue on the gdb<->bfd side than on the libthread_db side
> of things.  (Put a breakpoint on "warning".  I believe it comes from
> corelow.c:get_core_register_section.)
> 
> Thanks,
> Pedro Alves

No, without thread_db I get this:

(gdb) file m32
Reading symbols from m32...done.
(gdb) core c_32
[New LWP 79425]
[New LWP 79425]
[New LWP 79425]
warning: Unable to find libthread_db matching inferior's thread
library, thread debugging will not be available.
warning: Unable to find libthread_db matching inferior's thread
library, thread debugging will not be available.
Core was generated by `./m32'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00400680 in doSomeThing ()
(gdb) info threads
  Id   Target Id         Frame 
* 3    LWP 79425         0x00400680 in doSomeThing ()



Steve Ellcey
sellcey@cavium.com

Re: gdb / thread_db / multiple ABI question

[Pedro Alves]

On 07/10/2017 07:02 PM, Steve Ellcey wrote:
> No, without thread_db I get this:
> 
> (gdb) file m32
> Reading symbols from m32...done.
> (gdb) core c_32
> [New LWP 79425]
> [New LWP 79425]
> [New LWP 79425]

This means that GDB/bfd thinks that there are 3 threads
in the core, but they all have the same TID, which is of course
bogus.  So this still looks to me like something is wrong on
the bfd side.  Try "objdump -h c_32".  You should see
one ".reg/NNN" section per thread, like e.g.:

 $ objdump -h core.1904 | grep "reg/"
   1 .reg/1904     000000d8  0000000000000000  0000000000000000  000006a0  2**2
   9 .reg/1909     000000d8  0000000000000000  0000000000000000  00000e00  2**2
  13 .reg/1910     000000d8  0000000000000000  0000000000000000  00001560  2**2

Put a breakpoint on elf.c:elfcore_grok_note, and step through it,
checking that the proper NT_PRSTATUS / NT_PSINFO hooks get called
via the elf_backend_data vector, and check that they're extracting
the right fields/offsets out of the core structures.

Once you get 3 threads without libthread_db., then this:

> (gdb) info threads
Found 0 new threads in iteration 0.
>   Id   Target Id         Frame 
> * 3    Thread 0xf739ef70 (LWP 8660) 0x00400680 in doSomeThing ()
>   4    Thread 0xf69bf490 (LWP 8662) warning: Couldn't find general-purpose registers in core file.
> PC register is not available

should not longer happen.  The above with libthread_db is very likely
happening because libthread_db found a thread that maps to LWP 8662
in libpthread's internal data structures, but then GDB/bfd can't
find the ".reg/8662" note/section in the core dump.

Thanks,
Pedro Alves

Re: gdb / thread_db / multiple ABI question

[Steve Ellcey]

I never noticed that the 3 TIDs were all the same.  I am looking
at elf.c:elfcore_grok_note and I noticed something odd that doesn't
seem related to the ILP32 work.  If I look at the note->type
values that come in and get used in the switch statement, I see 0x401,
0x402, 0x403, and 0x404.  But 0x404 is not defined
in include/elf/common.h.  There is a 0x405, but no 0x404.  Any
idea why?

Steve Ellcey
sellcey@cavium.com


On Mon, 2017-07-10 at 19:21 +0100, Pedro Alves wrote:
> On 07/10/2017 07:02 PM, Steve Ellcey wrote:
> > 
> > No, without thread_db I get this:
> > 
> > (gdb) file m32
> > Reading symbols from m32...done.
> > (gdb) core c_32
> > [New LWP 79425]
> > [New LWP 79425]
> > [New LWP 79425]
> This means that GDB/bfd thinks that there are 3 threads
> in the core, but they all have the same TID, which is of course
> bogus.  So this still looks to me like something is wrong on
> the bfd side.  Try "objdump -h c_32".  You should see
> one ".reg/NNN" section per thread, like e.g.:
> 
>  $ objdump -h core.1904 | grep "reg/"
>    1
> .reg/1904     000000d8  0000000000000000  0000000000000000  000006a0 
>  2**2
>    9
> .reg/1909     000000d8  0000000000000000  0000000000000000  00000e00 
>  2**2
>   13
> .reg/1910     000000d8  0000000000000000  0000000000000000  00001560 
>  2**2
> 
> Put a breakpoint on elf.c:elfcore_grok_note, and step through it,
> checking that the proper NT_PRSTATUS / NT_PSINFO hooks get called
> via the elf_backend_data vector, and check that they're extracting
> the right fields/offsets out of the core structures.
> 
> Once you get 3 threads without libthread_db., then this:
> 
> > 
> > (gdb) info threads
> Found 0 new threads in iteration 0.
> > 
> >   Id   Target Id         Frame 
> > * 3    Thread 0xf739ef70 (LWP 8660) 0x00400680 in doSomeThing ()
> >   4    Thread 0xf69bf490 (LWP 8662) warning: Couldn't find general-
> > purpose registers in core file.
> > PC register is not available
> should not longer happen.  The above with libthread_db is very likely
> happening because libthread_db found a thread that maps to LWP 8662
> in libpthread's internal data structures, but then GDB/bfd can't
> find the ".reg/8662" note/section in the core dump.
> 
> Thanks,
> Pedro Alves

Re: gdb / thread_db / multiple ABI question

[Yao Qi]

Steve Ellcey <sellcey@cavium.com> writes:

> seem related to the ILP32 work.  If I look at the note->type
> values that come in and get used in the switch statement, I see 0x401,
> 0x402, 0x403, and 0x404.  But 0x404 is not defined
> in include/elf/common.h.  There is a 0x405, but no 0x404.  Any
> idea why?

0x404 is NT_ARM_SYSTEM_CALL, defined in kernel include/uapi/linux/elf.h.

-- 
Yao (齐尧)

Re: gdb / thread_db / multiple ABI question

[Steve Ellcey]

On Mon, 2017-07-10 at 19:21 +0100, Pedro Alves wrote:
> On 07/10/2017 07:02 PM, Steve Ellcey wrote:
> > 
> > No, without thread_db I get this:
> > 
> > (gdb) file m32
> > Reading symbols from m32...done.
> > (gdb) core c_32
> > [New LWP 79425]
> > [New LWP 79425]
> > [New LWP 79425]
> This means that GDB/bfd thinks that there are 3 threads
> in the core, but they all have the same TID, which is of course
> bogus.  So this still looks to me like something is wrong on
> the bfd side.  Try "objdump -h c_32".  You should see
> one ".reg/NNN" section per thread, like e.g.:


FYI: I think I found the cause of this problem.  In bfd/elfxx-
aarch64.c:_bfd_aarch64_elf_grok_prstatus there is code to
differentiate LP64 and ILP32 based on the different sizes of
elf_prstatus (note->descsz).  But after that differentiation I was
using 12 as the offset of pr_cursig in elf_prstatus and 32 for the
offset of pr_reg in both the ILP32 and LP64 cases.  I think the offset
of 12 for pr_cursig is OK because there are 3 int fields in front of
it and those are 4 bytes in both modes.  But the offset of pr_reg
should be 32 for LP64 and 24 for ILP32 because there are two long
variables between pr_cursig and pr_pid so the offset to pr_pid will
differ based on the ABI.  After this change:

(gdb) file m32
Reading symbols from m32...done.
(gdb) core c_32
[New LWP 15129]
[New LWP 15130]
[New LWP 15128]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/home/sellcey/gdb-ilp32/install/lib64/libthread_db.so.1".
Core was generated by `./m32'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00400680 in doSomeThing ()
[Current thread is 1 (Thread 0xf725f490 (LWP 15129))]
(gdb) info threads
  Id   Target Id         Frame 
* 1    Thread 0xf725f490 (LWP 15129) 0x00400680 in doSomeThing ()
  2    Thread 0xf6a5f490 (LWP 15130) 0xf72fa2d0 in __GI___nanosleep (
    requested_time=requested_time@entry=0xf6a5eeb8, 
    remaining=remaining@entry=0xf6a5eeb8)
    at ../sysdeps/unix/sysv/linux/nanosleep.c:27
  3    Thread 0xf743ef70 (LWP 15128) 0xf72fa2d0 in __GI___nanosleep (
    requested_time=requested_time@entry=0xffa1b1c8, 
    remaining=remaining@entry=0xffa1b1c8)
    at ../sysdeps/unix/sysv/linux/nanosleep.c:27