examining a core dump on a machine with other base addresses in shared libraries

examining a core dump on a machine with other base addresses in shared libraries

[Drago Krznaric]

Hi,

I have a program and a core dump created on machine A that I copy to 
machine B. I can open the core dump and get a nice backtrace on A but
not on B. The reason appears to be that the shared libraries on A and B
have different base addresses. They are otherwise exactly identical in
hardware and software (RedHat 4). I guess that the addresses are
different because of the prelink cron job or "address space layout
randomization". Normally I don't have access to machine A, so I would
really want to open the core dump on B without having to copy all used
shared libraries from A to B. 

Does anybody know if and how this can be done?

When opening core dump with gdb-6.8 on B machine, I get something like:

warning: .dynamic section for "/usr/lib/libstdc++.so.6" is not at the 
expected address (wrong library or version mismatch?)
warning: .dynamic section for "/lib/tls/libm.so.6" is not at the
expected address
warning: difference appears to be caused by prelink, adjusting
expectations
...

and when I do a backtrace I get something useless as:

#0  0x003d17a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2
#1  0x004117d5 in __longjmp () from /lib/tls/libc.so.6
#2  0x00000000 in ?? ()
(gdb)

Thanks in advance,
Drago 

Re: examining a core dump on a machine with other base addresses in shared libraries

[Jan Kratochvil]

On Wed, 27 May 2009 22:52:35 +0200, Drago Krznaric wrote:
> software (RedHat 4).

expecting RHEL (Red Hat Enterprise Linux) release 4 update <unknown>.
Red Hat Linux 4.0/4.1/4.2 is sometimes still in use but it is very old now.

IMO this looks as system specific issue more suitable for:
	https://bugzilla.redhat.com


> Does anybody know if and how this can be done?

This is a standard feature of GDB and verified now RHEL-4.8 i386 that it
really works as expect both with system gdb-6.3.0.0-1.162.el4.i386 and built
FSF GDB 6.8.


> When opening core dump with gdb-6.8 on B machine, I get something like:
> 
> warning: .dynamic section for "/usr/lib/libstdc++.so.6" is not at the expected address (wrong library or version mismatch?)

As printed by GDB LM_ADDR_CHECK() it suggests the build (=library rpm
versions) of your two machines differ.  core file cannot be debugged without
the exact build of the libraries.  Check it really matches by:
	rpm --qf '%{name}-%{version}-%{release}.%{arch}\n' -qf FILES...

build-id feature was not available for RHEL-4 so one can never be for sure but
this mismatch means something is wrong, expecting a different version/build.


> and when I do a backtrace I get something useless as:

Attached below my tests of differently prelinked libraries.


Regards,
Jan

------------------------------------------------------------------------------

$ echo -e "set debug-file-directory $PWD\nfile /bin/sleep\ncore-file ./core.18978" >1;~/redhat/gdb-6.8/gdb/gdb -x ./1
GNU gdb 6.8
Copyright (C) 2008 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "i686-pc-linux-gnu".
warning: .dynamic section for "/root/jkratoch/redhat/libc.so.6" is not at the expected address
warning: difference appears to be caused by prelink, adjusting expectations
Core was generated by `sleep 1h'.
Program terminated with signal 11, Segmentation fault.
[New process 18978]
#0  0x005fd7a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2
(gdb) bt
#0  0x005fd7a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2
#1  0x006a7bc0 in __nanosleep_nocancel () from /root/jkratoch/redhat/libc.so.6
#2  0x0804a6c8 in xnanosleep (seconds=3600) at xnanosleep.c:169
#3  0x080490ce in main (argc=2, argv=0xbff84104) at sleep.c:166
(gdb) q
$ echo -e "set debug-file-directory $PWD\nfile /bin/sleep\ncore-file ./core.18978" >1;gdb -x ./1
GNU gdb Red Hat Linux (6.3.0.0-1.162.el4rh)
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i386-redhat-linux-gnu".
Using host libthread_db library "/lib/tls/libthread_db.so.1".
Core was generated by `sleep 1h'.
Program terminated with signal 11, Segmentation fault.
warning: .dynamic section for "/root/jkratoch/redhat/libc.so.6" is not at the expected address
warning: difference appears to be caused by prelink, adjusting expectations
#0  0x005fd7a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2
(gdb) bt
#0  0x005fd7a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2
#1  0x006a7bc0 in __nanosleep_nocancel () from /root/jkratoch/redhat/libc.so.6
#2  0x0804a6c8 in xnanosleep (seconds=3600) at xnanosleep.c:169
#3  0x080490ce in main (argc=2, argv=0xbff84104) at sleep.c:166
#4  0x00630df3 in __libc_start_main () from /root/jkratoch/redhat/libc.so.6
#5  0x08048cf1 in _start ()
(gdb) q

Re: examining a core dump on a machine with other base addresses in shared libraries

[Drago Krznaric]

I'm pretty sure the library versions are identical because everything on
the two computers have been installed from the same CDs and they have
exactly the same hardware. But if I understand you correctly, gdb can
handle prelinked libraries. That's good. I will continue to do some
investigation. Perhaps its an old version of prelink that doesn't do
what gdb exepects. Otherwise I think that we can turn off prelink.
Indeed, if I turn it off and issue a prelink undo command, then I don't
have any problems debugging the core dumps. Upgrading to newer version
of RedHat is not possible at the moment, but perhaps we can upgrade the
prelink version if the problem is there.

Regards,
Drago


On Wed, 2009-05-27 at 23:41 +0200, Jan Kratochvil wrote:
> On Wed, 27 May 2009 22:52:35 +0200, Drago Krznaric wrote:
> > software (RedHat 4).
> 
> expecting RHEL (Red Hat Enterprise Linux) release 4 update <unknown>.
> Red Hat Linux 4.0/4.1/4.2 is sometimes still in use but it is very old now.
> 
> IMO this looks as system specific issue more suitable for:
> 	https://bugzilla.redhat.com
> 
> 
> > Does anybody know if and how this can be done?
> 
> This is a standard feature of GDB and verified now RHEL-4.8 i386 that it
> really works as expect both with system gdb-6.3.0.0-1.162.el4.i386 and built
> FSF GDB 6.8.
> 
> 
> > When opening core dump with gdb-6.8 on B machine, I get something like:
> > 
> > warning: .dynamic section for "/usr/lib/libstdc++.so.6" is not at the expected address (wrong library or version mismatch?)
> 
> As printed by GDB LM_ADDR_CHECK() it suggests the build (=library rpm
> versions) of your two machines differ.  core file cannot be debugged without
> the exact build of the libraries.  Check it really matches by:
> 	rpm --qf '%{name}-%{version}-%{release}.%{arch}\n' -qf FILES...
> 
> build-id feature was not available for RHEL-4 so one can never be for sure but
> this mismatch means something is wrong, expecting a different version/build.
> 
> 
> > and when I do a backtrace I get something useless as:
> 
> Attached below my tests of differently prelinked libraries.
> 
> 
> Regards,
> Jan
> 
> ------------------------------------------------------------------------------
> 
> $ echo -e "set debug-file-directory $PWD\nfile /bin/sleep\ncore-file ./core.18978" >1;~/redhat/gdb-6.8/gdb/gdb -x ./1
> GNU gdb 6.8
> Copyright (C) 2008 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
> and "show warranty" for details.
> This GDB was configured as "i686-pc-linux-gnu".
> warning: .dynamic section for "/root/jkratoch/redhat/libc.so.6" is not at the expected address
> warning: difference appears to be caused by prelink, adjusting expectations
> Core was generated by `sleep 1h'.
> Program terminated with signal 11, Segmentation fault.
> [New process 18978]
> #0  0x005fd7a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2
> (gdb) bt
> #0  0x005fd7a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2
> #1  0x006a7bc0 in __nanosleep_nocancel () from /root/jkratoch/redhat/libc.so.6
> #2  0x0804a6c8 in xnanosleep (seconds=3600) at xnanosleep.c:169
> #3  0x080490ce in main (argc=2, argv=0xbff84104) at sleep.c:166
> (gdb) q
> $ echo -e "set debug-file-directory $PWD\nfile /bin/sleep\ncore-file ./core.18978" >1;gdb -x ./1
> GNU gdb Red Hat Linux (6.3.0.0-1.162.el4rh)
> Copyright 2004 Free Software Foundation, Inc.
> GDB is free software, covered by the GNU General Public License, and you are
> welcome to change it and/or distribute copies of it under certain conditions.
> Type "show copying" to see the conditions.
> There is absolutely no warranty for GDB.  Type "show warranty" for details.
> This GDB was configured as "i386-redhat-linux-gnu".
> Using host libthread_db library "/lib/tls/libthread_db.so.1".
> Core was generated by `sleep 1h'.
> Program terminated with signal 11, Segmentation fault.
> warning: .dynamic section for "/root/jkratoch/redhat/libc.so.6" is not at the expected address
> warning: difference appears to be caused by prelink, adjusting expectations
> #0  0x005fd7a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2
> (gdb) bt
> #0  0x005fd7a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2
> #1  0x006a7bc0 in __nanosleep_nocancel () from /root/jkratoch/redhat/libc.so.6
> #2  0x0804a6c8 in xnanosleep (seconds=3600) at xnanosleep.c:169
> #3  0x080490ce in main (argc=2, argv=0xbff84104) at sleep.c:166
> #4  0x00630df3 in __libc_start_main () from /root/jkratoch/redhat/libc.so.6
> #5  0x08048cf1 in _start ()
> (gdb) q

Re: examining a core dump on a machine with other base addresses in shared libraries

[Paul Pluzhnikov]

On Wed, May 27, 2009 at 1:52 PM, Drago Krznaric <drago.krznaric@spray.se> wrote:

> I have a program and a core dump created on machine A that I copy to
> machine B. I can open the core dump and get a nice backtrace on A but
> not on B. The reason appears to be that the shared libraries on A and B
> have different base addresses. They are otherwise exactly identical in
> hardware and software (RedHat 4).

Are you absolutely sure the libc binaries are identical between A and B?
We see a lot of similar GDB failures due to mismatch between libc versions
(but we don't use prelink).

You could verify whether prelink is the problem, by copying B:libc.so.6 and
B:ld-linux.so.2 to e.g. /tmp/lib, manually prelinking them to the same
address A:libc.so.6 and A:ld-linux.so.2 were prelinked at, and verifying
they "work" (set solib-absolute-prefix /tmp in GDB before loading the core).

I think you should also get identical (as in md5sum-identical) binary
for B:/tmp/lib/libc.so.6 and A:/lib/libc.so.6

Cheers,
-- 
Paul Pluzhnikov

Re: examining a core dump on a machine with other base addresses in shared libraries

[Drago Krznaric]

Yes and no... It turns out that they're identical on A and B but on
those machines I had an old gdb that couldn't deal with the prelinked
libraries. When i ran the newer gdb I was on another machine that had
some different libraries. However, running a newer gdb on A and B did
indeed give a readable backtrace even with prelinked libraries.
Thanx,
drago 

On Wed, 2009-05-27 at 16:46 -0700, Paul Pluzhnikov wrote:
> On Wed, May 27, 2009 at 1:52 PM, Drago Krznaric <drago.krznaric@spray.se> wrote:

> > I have a program and a core dump created on machine A that I copy to
> > machine B. I can open the core dump and get a nice backtrace on A but
> > not on B. The reason appears to be that the shared libraries on A and B
> > have different base addresses. They are otherwise exactly identical in
> > hardware and software (RedHat 4).
> 
> Are you absolutely sure the libc binaries are identical between A and B?
> We see a lot of similar GDB failures due to mismatch between libc versions
> (but we don't use prelink).
> 
> You could verify whether prelink is the problem, by copying B:libc.so.6 and
> B:ld-linux.so.2 to e.g. /tmp/lib, manually prelinking them to the same
> address A:libc.so.6 and A:ld-linux.so.2 were prelinked at, and verifying
> they "work" (set solib-absolute-prefix /tmp in GDB before loading the core).
> 
> I think you should also get identical (as in md5sum-identical) binary
> for B:/tmp/lib/libc.so.6 and A:/lib/libc.so.6
> 
> Cheers,

Re: examining a core dump on a machine with other base addresses in shared libraries

[Hui Zhu]

I think you can copy the share lib in A to B, and call command "set
sysroot dirname" in gdb to let gdb know wich directory it can find
right lib in B.

Hui

On Thu, May 28, 2009 at 04:52, Drago Krznaric <drago.krznaric@spray.se> wrote:
> Hi,
>
> I have a program and a core dump created on machine A that I copy to
> machine B. I can open the core dump and get a nice backtrace on A but
> not on B. The reason appears to be that the shared libraries on A and B
> have different base addresses. They are otherwise exactly identical in
> hardware and software (RedHat 4). I guess that the addresses are
> different because of the prelink cron job or "address space layout
> randomization". Normally I don't have access to machine A, so I would
> really want to open the core dump on B without having to copy all used
> shared libraries from A to B.
>
> Does anybody know if and how this can be done?
>
> When opening core dump with gdb-6.8 on B machine, I get something like:
>
> warning: .dynamic section for "/usr/lib/libstdc++.so.6" is not at the
> expected address (wrong library or version mismatch?)
> warning: .dynamic section for "/lib/tls/libm.so.6" is not at the
> expected address
> warning: difference appears to be caused by prelink, adjusting
> expectations
> ...
>
> and when I do a backtrace I get something useless as:
>
> #0  0x003d17a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2
> #1  0x004117d5 in __longjmp () from /lib/tls/libc.so.6
> #2  0x00000000 in ?? ()
> (gdb)
>
> Thanks in advance,
> Drago
>
>