debugging a dynamically loaded library

debugging a dynamically loaded library

[Mike Krogh]

Can someone explain how to debug subroutines that are loaded via
dlopen()/dlsym()?  Specifically I'd like to compile and link dso.c
into dso.so and then compile main.c and have it dynamically load
dso.so.  Once that is done, I'd like to get gdb to stop in
dso_init(int i) from ./dso.so.

I'm compiling with the following:

cc -g -c dso.c
cc -shared -g -o dso.so dso.o
cc -rdynamic -o main main.c -ldl

Executing ./main works fine.  I just can't get gdb to stop anywhere
within dso.c.  I've tried using 'add-symbol-file dso.so <address>'
without success.

I'm running under RedHat 7.1 and RedHat 6.2.  I've also tried the
8/29/01 gdb dev snapshot.

Thanks,

Mike


/********************** main.c **********************/
#include <stdio.h>
#include <dlfcn.h>


#include "dso.h"


int
main(int argc, char *argv[]) {

  void *handle;
  void (*init)(int);
  void (*incr)(int);
  char *error;
  int i;


  if (!(handle = dlopen("./dso.so", RTLD_NOW|RTLD_GLOBAL))) {
    fputs(dlerror(), stderr);
    exit(1);
  }

  init = dlsym(handle, "dso_init");
  if ((error = dlerror()) != NULL) {
    fprintf(stderr, "%s\n", error);
    exit(1);
  } else
    fprintf(stderr, "Do `add-symbol-file dso.so 0x%x`", init);

  incr = dlsym(handle, "dso_increment");
  if ((error = dlerror()) != NULL) {
    fprintf(stderr, "%s\n", error);
    exit(1);
  }

  (*init)(5);

  for (i=0; i<10; i++)
    (*incr)(2);

  dlclose(handle);        

  return 0;

}



/********************** dso.c **********************/
#include <stdio.h>


static int value;


void
dso_init(int i) {

  value = i;
  fprintf(stderr, "dso_init: initialized with %d\n", i);

}


void
dso_increment(int i) {

  value += i;

  fprintf(stderr, "dso_increment: i = %d value = %d\n", i, value);

}


/********************** dso.c **********************/
#ifndef DSO_H
#define DSO_H

void dso_init(int i);

void dso_increment(int i);


#endif /* DSO_H */


-- 
-
+-----------------------------------------------------------------+
| Mike Krogh                       .~.               919-363-0883 |
| Computational Engineering Intl.  /V\           fax 919-363-0833 |
| 2166 N. Salem St., Suite 101    // \\          krogh@ceintl.com |
| Apex, NC 27502                 /(   )\           www.ceintl.com |
| EnSight/EnLiten/EnVideo         ^`~'^                           |
+-----------------------------------------------------------------+

Re: debugging a dynamically loaded library

[Kevin Buettner]

On Aug 29,  4:44pm, Mike Krogh wrote:

> Can someone explain how to debug subroutines that are loaded via
> dlopen()/dlsym()?

Sure.  Read on...

> Specifically I'd like to compile and link dso.c
> into dso.so and then compile main.c and have it dynamically load
> dso.so.  Once that is done, I'd like to get gdb to stop in
> dso_init(int i) from ./dso.so.
> 
> I'm compiling with the following:
> 
> cc -g -c dso.c
> cc -shared -g -o dso.so dso.o
> cc -rdynamic -o main main.c -ldl

You'll have an easier time of it if you compile main.c with -g.

[...]

> int
> main(int argc, char *argv[]) {
> 
>   void *handle;
>   void (*init)(int);
>   void (*incr)(int);
>   char *error;
>   int i;
> 
> 
>   if (!(handle = dlopen("./dso.so", RTLD_NOW|RTLD_GLOBAL))) {
>     fputs(dlerror(), stderr);
>     exit(1);
>   }
> 
>   init = dlsym(handle, "dso_init");

Once you're running gdb, place a breakpoint on the above line.  The
point of doing this is that it's after the call to dlopen().  GDB
will automatically read dso.so's symbols after the dlopen() call and
then you'll be able to place breakpoints in your dynamically loaded
library.

Here's a test run using your example:

saguaro:dynlib$ /saguaro1/sourceware/bld-saguaro/gdb/gdb main
GNU gdb 2001-08-21-cvs (MI_OUT)
Copyright 2001 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i686-pc-linux-gnu"...
(gdb) b main.c:23
Breakpoint 1 at 0x80487fc: file main.c, line 23.
(gdb) r
Starting program: /home/kev/ctests/dynlib/main 

Breakpoint 1, main (argc=1, argv=0xbffffa2c) at main.c:23
23        init = dlsym(handle, "dso_init");
(gdb) b dso_init
Breakpoint 2 at 0x400188a2: file dso.c, line 10.
(gdb) b dso_increment
Breakpoint 3 at 0x400188ce: file dso.c, line 19.
(gdb) c
Continuing.
Do `add-symbol-file dso.so 0x4001889c`
Breakpoint 2, dso_init (i=5) at dso.c:10
10        value = i;
(gdb) c
Continuing.
dso_init: initialized with 5

Breakpoint 3, dso_increment (i=2) at dso.c:19
19        value += i;
(gdb) c
Continuing.
dso_increment: i = 2 value = 7

Breakpoint 3, dso_increment (i=2) at dso.c:19
19        value += i;

etc...

Re: debugging a dynamically loaded library

[H . J . Lu]

On Wed, Aug 29, 2001 at 03:41:26PM -0700, Kevin Buettner wrote:
> 
> saguaro:dynlib$ /saguaro1/sourceware/bld-saguaro/gdb/gdb main
> GNU gdb 2001-08-21-cvs (MI_OUT)
> Copyright 2001 Free Software Foundation, Inc.
> GDB is free software, covered by the GNU General Public License, and you are
> welcome to change it and/or distribute copies of it under certain conditions.
> Type "show copying" to see the conditions.
> There is absolutely no warranty for GDB.  Type "show warranty" for details.
> This GDB was configured as "i686-pc-linux-gnu"...
> (gdb) b main.c:23
> Breakpoint 1 at 0x80487fc: file main.c, line 23.
> (gdb) r
> Starting program: /home/kev/ctests/dynlib/main 
> 
> Breakpoint 1, main (argc=1, argv=0xbffffa2c) at main.c:23
> 23        init = dlsym(handle, "dso_init");
> (gdb) b dso_init
> Breakpoint 2 at 0x400188a2: file dso.c, line 10.

For some reason, it doesn't work for me:

GNU gdb 2001-08-29-cvs (MI_OUT)
....
(gdb) b main.c:23
Breakpoint 1 at 0x8048820: file main.c, line 23.
(gdb) r
Starting program: /home/hjl/bugs/gdb/dlopen/main 

Breakpoint 1, main (argc=1, argv=0xbffff804) at main.c:23
23        init = dlsym(handle, "dso_init");
(gdb) b dso_init
Cannot access memory at address 0x123
(gdb) p dso_init
$1 = {<text variable, no debug info>} 0x40017800 <dso_init>
(gdb) info shared
From        To          Syms Read   Shared Object Library
0x400281d0  0x40028fc0  Yes         /lib/libdl.so.2
0x40047400  0x40147080  Yes         /lib/i686/libc.so.6
0x40001eb0  0x40012f20  Yes         /lib/ld-linux.so.2
0x400176d0  0x400178b0  Yes         ./dso.so

I am using glibc 2.2.4 on x86. It seems that gdb doesn't add the base
address of ./dso.so.


H.J. 

Re: debugging a dynamically loaded library

[Kevin Buettner]

On Aug 29,  4:07pm, H . J . Lu wrote:

> For some reason, it doesn't work for me:
[...]
> (gdb) b dso_init
> Cannot access memory at address 0x123

I'm seeing the same thing on my Red Hat 7.1 machine.  I'm looking
into it...

Kevin

Re: debugging a dynamically loaded library

[Kevin Buettner]

On Aug 29,  4:07pm, H . J . Lu wrote:

> > (gdb) b main.c:23
> > Breakpoint 1 at 0x80487fc: file main.c, line 23.
> > (gdb) r
> > Starting program: /home/kev/ctests/dynlib/main 
> > 
> > Breakpoint 1, main (argc=1, argv=0xbffffa2c) at main.c:23
> > 23        init = dlsym(handle, "dso_init");
> > (gdb) b dso_init
> > Breakpoint 2 at 0x400188a2: file dso.c, line 10.
> 
> For some reason, it doesn't work for me:
> 
> GNU gdb 2001-08-29-cvs (MI_OUT)
> ....
> (gdb) b main.c:23
> Breakpoint 1 at 0x8048820: file main.c, line 23.
> (gdb) r
> Starting program: /home/hjl/bugs/gdb/dlopen/main 
> 
> Breakpoint 1, main (argc=1, argv=0xbffff804) at main.c:23
> 23        init = dlsym(handle, "dso_init");
> (gdb) b dso_init
> Cannot access memory at address 0x123
> (gdb) p dso_init
> $1 = {<text variable, no debug info>} 0x40017800 <dso_init>
> (gdb) info shared
> >From        To          Syms Read   Shared Object Library
> 0x400281d0  0x40028fc0  Yes         /lib/libdl.so.2
> 0x40047400  0x40147080  Yes         /lib/i686/libc.so.6
> 0x40001eb0  0x40012f20  Yes         /lib/ld-linux.so.2
> 0x400176d0  0x400178b0  Yes         ./dso.so
> 
> I am using glibc 2.2.4 on x86. It seems that gdb doesn't add the base
> address of ./dso.so.

H. J.,

Could you try 

    http://sources.redhat.com/ml/gdb-patches/2001-09/msg00003.html

and let me know if it fixes the above problem?

Thanks,

Kevin

Re: debugging a dynamically loaded library

[H . J . Lu]

On Mon, Sep 03, 2001 at 03:58:50PM -0700, Kevin Buettner wrote:
> On Aug 29,  4:07pm, H . J . Lu wrote:
> 
> > > (gdb) b main.c:23
> > > Breakpoint 1 at 0x80487fc: file main.c, line 23.
> > > (gdb) r
> > > Starting program: /home/kev/ctests/dynlib/main 
> > > 
> > > Breakpoint 1, main (argc=1, argv=0xbffffa2c) at main.c:23
> > > 23        init = dlsym(handle, "dso_init");
> > > (gdb) b dso_init
> > > Breakpoint 2 at 0x400188a2: file dso.c, line 10.
> > 
> > For some reason, it doesn't work for me:
> > 
> > GNU gdb 2001-08-29-cvs (MI_OUT)
> > ....
> > (gdb) b main.c:23
> > Breakpoint 1 at 0x8048820: file main.c, line 23.
> > (gdb) r
> > Starting program: /home/hjl/bugs/gdb/dlopen/main 
> > 
> > Breakpoint 1, main (argc=1, argv=0xbffff804) at main.c:23
> > 23        init = dlsym(handle, "dso_init");
> > (gdb) b dso_init
> > Cannot access memory at address 0x123
> > (gdb) p dso_init
> > $1 = {<text variable, no debug info>} 0x40017800 <dso_init>
> > (gdb) info shared
> > >From        To          Syms Read   Shared Object Library
> > 0x400281d0  0x40028fc0  Yes         /lib/libdl.so.2
> > 0x40047400  0x40147080  Yes         /lib/i686/libc.so.6
> > 0x40001eb0  0x40012f20  Yes         /lib/ld-linux.so.2
> > 0x400176d0  0x400178b0  Yes         ./dso.so
> > 
> > I am using glibc 2.2.4 on x86. It seems that gdb doesn't add the base
> > address of ./dso.so.
> 
> H. J.,
> 
> Could you try 
> 
>     http://sources.redhat.com/ml/gdb-patches/2001-09/msg00003.html
> 
> and let me know if it fixes the above problem?
> 

I just came back from a trip. I will give it try tomorrow. I know there
was at least one gcc bug, which generated bogus debug code for glibc.
I am wondering if

1327   FUN    0      36     00000000 31372  __strtol_internal:F(0,3)

is a valid stab entry. It it is valid, why does it trigger the
SOFUN_ADDRESS_MAYBE_MISSING code in gdb. Is that possible a linker
or assembler bug?


H.J.

Re: debugging a dynamically loaded library

[Kevin Buettner]

On Sep 5, 10:29pm, H . J . Lu wrote:

> I am wondering if
> 
> 1327   FUN    0      36     00000000 31372  __strtol_internal:F(0,3)
> 
> is a valid stab entry. It it is valid, why does it trigger the
> SOFUN_ADDRESS_MAYBE_MISSING code in gdb. Is that possible a linker
> or assembler bug?

It's always possible that there's a bug elsewhere, but my guess is
that __strtol_internal() is merely the first function in the section
in which case the above entry makes sense to me.

Kevin

Re: debugging a dynamically loaded library

[H . J . Lu]

On Wed, Sep 05, 2001 at 11:04:03PM -0700, Kevin Buettner wrote:
> On Sep 5, 10:29pm, H . J . Lu wrote:
> 
> > I am wondering if
> > 
> > 1327   FUN    0      36     00000000 31372  __strtol_internal:F(0,3)
> > 
> > is a valid stab entry. It it is valid, why does it trigger the
> > SOFUN_ADDRESS_MAYBE_MISSING code in gdb. Is that possible a linker
> > or assembler bug?
> 
> It's always possible that there's a bug elsewhere, but my guess is
> that __strtol_internal() is merely the first function in the section
> in which case the above entry makes sense to me.

No. That is not the case. In glibc 2.2, which is used in RedHat 7.1,
__strtol_internal is a weak definition in the object files used to
build libdl.so.2:

# objdump --sym eval.os | grep __strtol_internal
00000440  w    F .text  00000137 __strtol_internal

But when libdl.so.2 is generated against libc.so.6:

# objdump --sym ../libc.so.6 | grep __strtol_internal
00032d04 g     F .text  000006ab              __strtol_internal

the linker generates

# objdump --sym libdl.so.2 | grep __strtol_internal
00000000  w    F *UND*  000006ab              __strtol_internal@@GLIBC_2.0

in libdl.so.2. I am 100% sure where the bug is. It can be in as, ld,
gcc and/or gdb. I tend to think it is a linker bug since the intention
of glibc is to make __strtol_internal weak defined, not weak undefined.
My second thought a weak undefined __strtol_internal in this case may
be the right thing to do, given how weak should be treated. Now the
question is how as/ld/gcc/gdb should deal a symbol which is turned from
weak defined to weak undefined. Does stabs/gdb support the notation of
weak defined symbol, which may be changed between compile-time,
link-time and run-time?

As for how useful a weak defined __strtol_internal is in libdl.so, that
is an entirely different question.


H.J.

Re: debugging a dynamically loaded library

[Mike Krogh]

Kevin,

Can you tell me if any more progress has been made with this problem?
Do you have any ETA on when a resolution might be available?  Since
your email address is at 'cygnus.com', I'm assuming that you might be
one of the maintainers of gdb.  If not, any suggestions on who I might
contact would be appreciated!

Anything I might be able to do?

Thanks!

Mike

 > On Aug 29,  4:07pm, H . J . Lu wrote:
>> > (gdb) b main.c:23
>> > Breakpoint 1 at 0x80487fc: file main.c, line 23.
>> > (gdb) r
>> > Starting program: /home/kev/ctests/dynlib/main 
>> > 
>> > Breakpoint 1, main (argc=1, argv=0xbffffa2c) at main.c:23
>> > 23        init = dlsym(handle, "dso_init");
>> > (gdb) b dso_init
>> > Breakpoint 2 at 0x400188a2: file dso.c, line 10.
>> 
>> For some reason, it doesn't work for me:
>> 
>> GNU gdb 2001-08-29-cvs (MI_OUT)
>> ....
>> (gdb) b main.c:23
>> Breakpoint 1 at 0x8048820: file main.c, line 23.
>> (gdb) r
>> Starting program: /home/hjl/bugs/gdb/dlopen/main 
>> 
>> Breakpoint 1, main (argc=1, argv=0xbffff804) at main.c:23
>> 23        init = dlsym(handle, "dso_init");
>> (gdb) b dso_init
>> Cannot access memory at address 0x123
>> (gdb) p dso_init
>> $1 = {<text variable, no debug info>} 0x40017800 <dso_init>
>> (gdb) info shared
>> >From        To          Syms Read   Shared Object Library
>> 0x400281d0  0x40028fc0  Yes         /lib/libdl.so.2
>> 0x40047400  0x40147080  Yes         /lib/i686/libc.so.6
>> 0x40001eb0  0x40012f20  Yes         /lib/ld-linux.so.2
>> 0x400176d0  0x400178b0  Yes         ./dso.so
>> 
>> I am using glibc 2.2.4 on x86. It seems that gdb doesn't add the base
>> address of ./dso.so.

 > H. J.,

 > Could you try 

 >     http://sources.redhat.com/ml/gdb-patches/2001-09/msg00003.html

 > and let me know if it fixes the above problem?

 > Thanks,

 > Kevin

-- 
-
+-----------------------------------------------------------------+
| Mike Krogh                       .~.               919-363-0883 |
| Computational Engineering Intl.  /V\           fax 919-363-0833 |
| 2166 N. Salem St., Suite 101    // \\          krogh@ceintl.com |
| Apex, NC 27502                 /(   )\           www.ceintl.com |
| EnSight/EnLiten/EnVideo         ^`~'^                           |
+-----------------------------------------------------------------+

Re: debugging a dynamically loaded library

[Kevin Buettner]

On Sep 18,  5:10pm, Mike Krogh wrote:

> Can you tell me if any more progress has been made with this problem?
> Do you have any ETA on when a resolution might be available?

A patch has been checked into the GDB development sources which "fixes"
the problem noted by H.J.  The reason that I put "fixes" in quotes is
that there is also a bug in the linker (or possibly some other part
of the toolchain) which causes a weak defined symbol to be changed to
a weak undefined symbol under some circumstances.  The stabs entries
for this (now undefined) symbol are still retained however.  Thus,
under these conditions, it may still be possible for GDB to produce
unexpected behavior, but I don't think this is unreasonable since GDB's
sources of debugging information are inconsistent with each other. 
I.e, one source says that the symbol is undefined while the other, in
essence, is saying that the symbol is defined by providing line number
information for the function.

In any event, the patch in question does allow me to successfully debug
your example program on a Red Hat Linux 7.1 system.  I don't think that
there were any issues with debugging your example program on a Red Hat
Linux 6.2 system.  (These were the two platforms that you were testing
on, right?)

> Anything I might be able to do?

Yes!  Fetch a copy of the development sources and test GDB against
your test case as well the application that caused you to report
this problem in the first place.

BTW, the ChangeLog entry for the patch in question is as follows:

2001-09-06  Kevin Buettner  <kevinb@redhat.com>
	
	* dbxread.c (process_one_symbol): Don't use error result from
	find_stab_function_addr().
	* partial-stab.h (case 'F'): Likewise.

	* partial-stab.h (case 'f'): Make SOFUN_ADDRESS_MAYBE_MISSING
	code match that used for case 'F'.  This fixes the divergence
	that was introduced by my 1999-09-14 changes to partial-stab.h.

Thanks,

Kevin