Decoding stack in core file without correct libs?

Decoding stack in core file without correct libs?

[Pierre Ossman]

Hi,

I'd like to see if there is a way to produce binaries so that gdb can 
walk the stack in a core dump even if the libraries gdb sees doesn't 
match the libraries when the core dump was generated.

My scenario is simply that we might get crashes at customer sites. 
Rather than getting some kind of remote access up an running, it would 
be easier to have them send us a core dump from the crash. We then load 
the core file together with a binary with debug symbols on our end.

Unfortunately gdb doesn't traverse the stack correctly if the crash is 
in a system library. We just get random addresses for each frame.

It's okay that we cannot get the proper symbols or local variables for 
frames that are in system libraries, but is there some way to get access 
to the frames that are in our binary?

Regards
-- 
Pierre Ossman           Software Development
Cendio AB               https://cendio.com
Teknikringen 8          https://twitter.com/ThinLinc
583 30 Linköping        https://facebook.com/ThinLinc
Phone: +46-13-214600    https://plus.google.com/+CendioThinLinc

A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?

Re: Decoding stack in core file without correct libs?

[Dmitry Samersoff]

[-- Attachment #1.1: Type: text/plain, Size: 1336 bytes --]

Pierre,

You can run command (below) on your own machine:

gdb -batch --eval "info shared"  binary core 2> /dev/null |\
 sed -n -e 's/^.*Yes[^\/]*\//\//p' -e 's/^.*No[^\/]*\//\//p'  > filelist

and then do

cat filelist | zip zipme.zip -@

on client's one.

PS: this small article might be helpful.

http://www.samersoff.net/do/index.php/blog/19-articles/56-how-to-open-java-coredump-in-gdb


-Dmitry

On 07/13/2017 12:24 PM, Pierre Ossman wrote:
> Hi,
> 
> I'd like to see if there is a way to produce binaries so that gdb can
> walk the stack in a core dump even if the libraries gdb sees doesn't
> match the libraries when the core dump was generated.
> 
> My scenario is simply that we might get crashes at customer sites.
> Rather than getting some kind of remote access up an running, it would
> be easier to have them send us a core dump from the crash. We then load
> the core file together with a binary with debug symbols on our end.
> 
> Unfortunately gdb doesn't traverse the stack correctly if the crash is
> in a system library. We just get random addresses for each frame.
> 
> It's okay that we cannot get the proper symbols or local variables for
> frames that are in system libraries, but is there some way to get access
> to the frames that are in our binary?
> 
> Regards



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

Re: Decoding stack in core file without correct libs?

[Pierre Ossman]

On 13/07/17 11:55, Dmitry Samersoff wrote:
> Pierre,
> 
> You can run command (below) on your own machine:
> 
> gdb -batch --eval "info shared"  binary core 2> /dev/null |\
>   sed -n -e 's/^.*Yes[^\/]*\//\//p' -e 's/^.*No[^\/]*\//\//p'  > filelist
> 
> and then do
> 
> cat filelist | zip zipme.zip -@
> 
> on client's one.
> 
> PS: this small article might be helpful.
> 
> http://www.samersoff.net/do/index.php/blog/19-articles/56-how-to-open-java-coredump-in-gdb
> 
> 
> -Dmitry
> 

I was hoping there was a way without having to involve more steps for 
the customer. :/

Regards
-- 
Pierre Ossman           Software Development
Cendio AB               https://cendio.com
Teknikringen 8          https://twitter.com/ThinLinc
583 30 Linköping        https://facebook.com/ThinLinc
Phone: +46-13-214600    https://plus.google.com/+CendioThinLinc

A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?

Re: Decoding stack in core file without correct libs?

[Jan Kratochvil]

On Thu, 13 Jul 2017 11:24:50 +0200, Pierre Ossman wrote:
> I'd like to see if there is a way to produce binaries so that gdb can walk
> the stack in a core dump even if the libraries gdb sees doesn't match the
> libraries when the core dump was generated.

Features described below are present at least in Fedora (and some of them on
RHEL-7).

Basic requirement is so-called build-id:
$ readelf -n /bin/bash
...
Displaying notes found at file offset 0x00000274 with length 0x00000024:
  Owner                 Data size	Description
  GNU                  0x00000014	NT_GNU_BUILD_ID (unique build ID bitstring)
    Build ID: bad838505c566d846ec9571e4f592d108064cf92

Then Fedora GDB; not upstream GDB - this feature was never reviewed upstream
	[PATCH v12 00/32] Validate binary before use
	https://sourceware.org/ml/gdb-patches/2015-08/msg00590.html
will guide one to provide the matching packages for it:
$ gdb core.21935
...
Missing separate debuginfo for the main executable file
Try: dnf --enablerepo='*debug*' install /usr/lib/debug/.build-id/07/379f708a44a91477b1407baf59c9bc1426d69e
Core was generated by `bc'.
...
# dnf --enablerepo='*debug*' install /usr/lib/debug/.build-id/07/379f708a44a91477b1407baf59c9bc1426d69e
...
  Installing  : bc-debuginfo-1.06.95-16.fc24.x86_64                                                                                  1/1 
...
$ gdb core.21935
...
Missing separate debuginfos, use: dnf debuginfo-install glibc-2.24-9.fc25.x86_64 readline-6.3-8.fc24.x86_64
...

Installation of such libraries is sure a problem if you run on a system with
different libraries - you do not want to replace your system libraries just to
make a backtrace.  For such case there is ABRT - it will automatically build
a new chroot with libraries matching build-ids from the core file.
	https://developer.fedoraproject.org/tools/abrt/about.html

Otherwise one can also try store all the binaries into the core file:
	(ulimit -c unlimited;echo 0xff >/proc/self/coredump_filter;exec myapp appargs)
->
	eu-stack --core=core.21760

eu-stack from elfutils can extract debug information from binaries contained
in the core file (GDB cannot do that).  Although during my experiment it
succeeded with the libraries but it still failed to backtrace executable of
a different version on disk, there may be some bug.

Regular core file backtraced on a different system:
	$ eu-stack --core=core.21935
	PID 21935 - core
	TID 21935:
	#0  0x00007f88835d3a80
	eu-stack: dwfl_thread_getframes tid 21935 at 0x7f88835d3a80 in /usr/lib64/libc-2.24.so: Callback returned failure
coredump_filter-extended core file backtraced on a different system:
	eu-stack --core=core.21760 
	PID 21760 - core
	TID 21760:
	#0  0x00007ff4ee365a80 __read
	#1  0x00007ff4eeab4d1d rl_getc
	#2  0x00007ff4eeab559a rl_read_key
	#3  0x00007ff4eea9ee32 readline_internal_char
	#4  0x00007ff4eea9f595 readline
	#5  0x0000559a6c1ad47c
	eu-stack: dwfl_thread_getframes tid 21760 at 0x559a6c1ad47b in /usr/bin/bc: Callback returned failure
	^^^ some bug, #5 is in /usr/bin/bc which should be present in the core file.


Jan