Troubleshooting broken gdbserver/remote-target

Troubleshooting broken gdbserver/remote-target

[Grant Edwards]

I've given up for now on trying to get threaded debugging
working. :(

Now I'm just trying to get normal symbolic debugging and
backtrace working.  As long as execution is in user-application
code, everything is fine.  I can do "next", "finish", set
breakpoints, display variables, backtraces, etc.

However, if I stop the program while it's in libc (e.g.
blocking on a read() call), gdb has absolutely no clue. Gdb is
unable to display any backtrace or figure out what the current
function is.

I've set solib-absolute-prefix to point to the directory
containing target libraries that are unstripped and contain
debug symbols.  On startup, everything works (gdb knows the PC
is pointing to line 332 in ldso/ldso/arm/elfinterp.c:332.

But when the program is stopped while blocked in a call to
read() inside gets(), gdb throws up its hands:

Here's the program:

   #include <stdio.h>
   
   char buffer[1024];
   int main(void)
   {
     while (1)
       {
         gets(buffer);
         puts(buffer);
       }
     return 0;
   }

And here's the debugging session:


   GNU gdb 6.8
   [...]
   This GDB was configured as "--host=i386-pc-linux-gnu --target=arm-linux-uclibc".
   [New Thread 815]
   0x40001350 in _start () at ldso/ldso/arm/elfinterp.c:332
   332     }
   (gdb) c
   Continuing.
   ^C
   Program received signal SIGINT, Interrupt.
   0x40031914 in ?? ()
   (gdb) bt
   #0  0x40031914 in ?? ()

   
If I look at the maps or smaps file for the process, address
0x40031914 is clearly in uclibc.so:

   40025000-4008a000 r-xp 00000000 01:00 637         /lib/libuClibc-0.9.30.1.so

40031914 is offset 0x0000c914 in /lib/libuClibc-0.9.30.1.so, and
looking at the symbols in that file shows that offset 0x0000c914 
is in __libc_read()

   0000c5e8 T __libc_pselect
   0000c6d8 T ptrace
   0000c858 T quotactl
   0000c8e0 T __libc_read
   0000c968 T readlink
   0000c9f4 T __libc_readv
   0000ca78 T reboot

Why can't gdb figure out that the process is blocked in
__libc_read() and print a proper backtrace?

AFAICT from strace output gdbserver never tries to read
/proc/<pid>/[s]maps.  What mechanism is supposed to be used to
determine what executable file a particular address is in?

-- 
Grant Edwards                   grante             Yow! INSIDE, I have the
                                  at               same personality disorder
                               visi.com            as LUCY RICARDO!!

Re: Troubleshooting broken gdbserver/remote-target

[Grant Edwards]

On 2009-06-23, Grant Edwards <grant.b.edwards@gmail.com> wrote:

> But when the program is stopped while blocked in a call to
> read() inside gets(), gdb throws up its hands:

[...]

>    GNU gdb 6.8
>    [...]
>    This GDB was configured as "--host=i386-pc-linux-gnu --target=arm-linux-uclibc".
>    [New Thread 815]
>    0x40001350 in _start () at ldso/ldso/arm/elfinterp.c:332
>    332     }
>    (gdb) c
>    Continuing.
>    ^C
>    Program received signal SIGINT, Interrupt.
>    0x40031914 in ?? ()
>    (gdb) bt
>    #0  0x40031914 in ?? ()
>
>    
> If I look at the maps or smaps file for the process, address
> 0x40031914 is clearly in uclibc.so:
>
>    40025000-4008a000 r-xp 00000000 01:00 637         /lib/libuClibc-0.9.30.1.so
>
> 40031914 is offset 0x0000c914 in /lib/libuClibc-0.9.30.1.so, and
> looking at the symbols in that file shows that offset 0x0000c914 
> is in __libc_read()
[...]

> Why can't gdb figure out that the process is blocked in
> __libc_read() and print a proper backtrace?

I think I've found a clue why gdb is unable to figure out what
function contains address 0x40031914.  Gdb seems to be aware of
the mappig of only one of the library files (/lib/ld-uClibc-0.9.30.1.so),
and doesn't appear to know that libuClibc-0.9.30.1.so is mapped:

   (gdb) info file
   Symbols from "/home/grante/hello/mecho".
   Remote serial target in gdb-specific protocol:
   Debugging a target over a serial line.
           While running this, GDB does not access memory from...
   Local exec file:
           `/home/grante/hello/mecho', file type elf32-littlearm.
           Entry point: 0x8420     0x000080f4 - 0x00008108 @ 0x000000f4 is .interp
           0x00008108 - 0x00008160 @ 0x00000108 is .hash
           0x00008160 - 0x00008270 @ 0x00000160 is .dynsym
           0x00008270 - 0x0000833a @ 0x00000270 is .dynstr
           0x0000833a - 0x0000835c @ 0x0000033a is .gnu.version
           0x0000835c - 0x0000837c @ 0x0000035c is .gnu.version_r
           0x0000837c - 0x000083ac @ 0x0000037c is .rel.plt
           0x000083ac - 0x000083c4 @ 0x000003ac is .init
           0x000083c4 - 0x00008420 @ 0x000003c4 is .plt
           0x00008420 - 0x000085a0 @ 0x00000420 is .text
           0x000085a0 - 0x000085b4 @ 0x000005a0 is .fini
           0x000085b4 - 0x000085b8 @ 0x000005b4 is .eh_frame
           0x000105b8 - 0x000105c0 @ 0x000005b8 is .ctors
           0x000105c0 - 0x000105c8 @ 0x000005c0 is .dtors
           0x000105c8 - 0x000105cc @ 0x000005c8 is .jcr
           0x000105cc - 0x00010684 @ 0x000005cc is .dynamic
           0x00010684 - 0x000106a8 @ 0x00000684 is .got
           0x000106a8 - 0x000106b8 @ 0x000006a8 is .data
           0x000106b8 - 0x00010ad8 @ 0x000006b8 is .bss
           0x400000d4 - 0x40000240 @ 0x000000d4 is .hash in /home/grante/processors/atmel/RM9200/buildroot-2009.05/build_arm/staging_dir/lib/ld-uClibc.so.0
           0x40000240 - 0x40000580 @ 0x00000240 is .dynsym in /home/grante/processors/atmel/RM9200/buildroot-2009.05/build_arm/staging_dir/lib/ld-uClibc.so.0
           0x40000580 - 0x40000888 @ 0x00000580 is .dynstr in /home/grante/processors/atmel/RM9200/buildroot-2009.05/build_arm/staging_dir/lib/ld-uClibc.so.0
           0x40000888 - 0x40000930 @ 0x00000888 is .rel.dyn in /home/grante/processors/atmel/RM9200/buildroot-2009.05/build_arm/staging_dir/lib/ld-uClibc.so.0
           0x40000930 - 0x40008b70 @ 0x00000930 is .text in /home/grante/processors/atmel/RM9200/buildroot-2009.05/build_arm/staging_dir/lib/ld-uClibc.so.0
           0x40008b70 - 0x40008ffc @ 0x00008b70 is .rodata in /home/grante/processors/atmel/RM9200/buildroot-2009.05/build_arm/staging_dir/lib/ld-uClibc.so.0
           0x40011f14 - 0x40011f18 @ 0x00009f14 is .dtors in /home/grante/processors/atmel/RM9200/buildroot-2009.05/build_arm/staging_dir/lib/ld-uClibc.so.0
           0x40011f18 - 0x40011fa8 @ 0x00009f18 is .dynamic in /home/grante/processors/atmel/RM9200/buildroot-2009.05/build_arm/staging_dir/lib/ld-uClibc.so.0
           0x40011fa8 - 0x40012000 @ 0x00009fa8 is .got in /home/grante/processors/atmel/RM9200/buildroot-2009.05/build_arm/staging_dir/lib/ld-uClibc.so.0
           0x40012000 - 0x40012008 @ 0x0000a000 is .data in /home/grante/processors/atmel/RM9200/buildroot-2009.05/build_arm/staging_dir/lib/ld-uClibc.so.0
           0x40012008 - 0x4001205c @ 0x0000a008 is .bss in /home/grante/processors/atmel/RM9200/buildroot-2009.05/build_arm/staging_dir/lib/ld-uClibc.so.0

Here's the contents of /proc/<pid>/maps for the process being
debuged:

   00008000-00009000 r-xp 00000000 01:00 130        /usr/bin/mecho
   00010000-00011000 rw-p 00000000 01:00 130        /usr/bin/mecho
   40000000-40009000 r-xp 00000000 01:00 642        /lib/ld-uClibc-0.9.30.1.so
   40009000-4000a000 rw-p 40009000 00:00 0 
   40011000-40012000 r--p 00009000 01:00 642        /lib/ld-uClibc-0.9.30.1.so
   40012000-40013000 rw-p 0000a000 01:00 642        /lib/ld-uClibc-0.9.30.1.so
   40013000-4001c000 r-xp 00000000 01:00 212        /lib/libgcc_s.so.1
   4001c000-40024000 ---p 4001c000 00:00 0 
   40024000-40025000 rw-p 00009000 01:00 212        /lib/libgcc_s.so.1
   40025000-4008a000 r-xp 00000000 01:00 637        /lib/libuClibc-0.9.30.1.so
   4008a000-40092000 ---p 4008a000 00:00 0 
   40092000-40093000 r--p 00065000 01:00 637        /lib/libuClibc-0.9.30.1.so
   40093000-40094000 rw-p 00066000 01:00 637        /lib/libuClibc-0.9.30.1.so
   40094000-40098000 rw-p 40094000 00:00 0 
   be98b000-be9a0000 rw-p befeb000 00:00 0          [stack]

Gdb figured out the /lib/ld-uClibc-0.9.30.1.so, but it's
missing /lib/libgcc_s.so.1 and /lib/libuClibc-0.9.30.1.so.

What would cause gdb to miss out on the fact that /lib/libuClibc-0.9.30.1.so
is mapped?

Where does gdb get mapping info?

-- 
Grant

Re: Troubleshooting broken gdbserver/remote-target

[Paul Pluzhnikov]

On Mon, Jun 22, 2009 at 10:09 PM, Grant Edwards
<grant.b.edwards@gmail.com> wrote:

> What would cause gdb to miss out on the fact that /lib/libuClibc-0.9.30.1.so
> is mapped?
>
> Where does gdb get mapping info?

I believe the code GDB should be using on Linux is in solib-svr4.c

Gdb looks for _r_debug symbol and DT_DEBUG Elf32_Dyn entry to find the
'struct r_debug', which contains a linked list of loaded shared libraries,
and traverses that list.

My guess: either GDB is not finding _r_debug, or (more likely) is not
traversing the list correctly.

Cheers,
--
Paul Pluzhnikov

Re: Troubleshooting broken gdbserver/remote-target

[Grant Edwards]

On 2009-06-23, Paul Pluzhnikov <ppluzhnikov@google.com> wrote:

>> What would cause gdb to miss out on the fact that
>> /lib/libuClibc-0.9.30.1.so is mapped?
>>
>> Where does gdb get mapping info?
>
> I believe the code GDB should be using on Linux is in solib-svr4.c
>
> Gdb looks for _r_debug symbol and DT_DEBUG Elf32_Dyn entry to
> find the 'struct r_debug', which contains a linked list of
> loaded shared libraries, and traverses that list.

Thanks for the hint.  When I have a chance I'll see if I can
figure out what's wrong.  I presume gdb is looking for those
symbols in the ELF executable file?

> My guess: either GDB is not finding _r_debug, or (more likely)
> is not traversing the list correctly.

I found other reports that 6.8 doesn't work for ARM targets,
but 6.7.1 does.  I switched from 6.8 to 6.7.1, and that fixed
the problem.

Now to see if that fixed the problems with thread support
as well...

-- 
Grant Edwards                   grante             Yow! I know things about
                                  at               TROY DONAHUE that can't
                               visi.com            even be PRINTED!!

Re: Troubleshooting broken gdbserver/remote-target

[Grant Edwards]

On 2009-06-23, Grant Edwards <grant.b.edwards@gmail.com> wrote:
> On 2009-06-23, Paul Pluzhnikov <ppluzhnikov@google.com> wrote:
>
>>> What would cause gdb to miss out on the fact that
>>> /lib/libuClibc-0.9.30.1.so is mapped?
>>>
>>> Where does gdb get mapping info?
>>
>> I believe the code GDB should be using on Linux is in solib-svr4.c
>>
>> Gdb looks for _r_debug symbol and DT_DEBUG Elf32_Dyn entry to
>> find the 'struct r_debug', which contains a linked list of
>> loaded shared libraries, and traverses that list.
>
> Thanks for the hint.  When I have a chance I'll see if I can
> figure out what's wrong.  I presume gdb is looking for those
> symbols in the ELF executable file?
>
>> My guess: either GDB is not finding _r_debug, or (more likely)
>> is not traversing the list correctly.
>
> I found other reports that 6.8 doesn't work for ARM targets,
> but 6.7.1 does.  I switched from 6.8 to 6.7.1, and that fixed
> the problem.
>
> Now to see if that fixed the problems with thread support
> as well...

Indeed it does!

Multithreaded debugging using gdbserver on an ARM9 target works
fine with 6.7.1, but it with 6.8 you get SIG32 and a variety of
other problems.

-- 
Grant Edwards                   grante             Yow! ... I want FORTY-TWO
                                  at               TRYNEL FLOATATION SYSTEMS
                               visi.com            installed within SIX AND A
                                                   HALF HOURS!!!

Re: Troubleshooting broken gdbserver/remote-target

[Grant Edwards]

On 2009-06-23, Grant Edwards <grant.b.edwards@gmail.com> wrote:

>> I found other reports that 6.8 doesn't work for ARM targets,
>> but 6.7.1 does.  I switched from 6.8 to 6.7.1, and that fixed
>> the problem.
>>
>> Now to see if that fixed the problems with thread support as
>> well...
>
> Indeed it does!
>
> Multithreaded debugging using gdbserver on an ARM9 target
> works fine with 6.7.1, but it with 6.8 you get SIG32 and a
> variety of other problems.

FWIW, I've logged a bug against 6.8: http://sourceware.org/bugzilla/show_bug.cgi?id=10328
but I'd be curious to know if the same problems occur with
glibc (I was using uclibc)...
  
-- 
Grant Edwards                   grante             Yow! Hmmm ... an arrogant
                                  at               bouquet with a subtle
                               visi.com            suggestion of POLYVINYL
                                                   CHLORIDE ...