Classifying core files?

Classifying core files?

[Paul Smith]

Hi all;

I have a situation where I'm collecting cores from a wide variety of
(well-defined) systems: there are cores from other architectures (where
I need a cross debugger), cores from different executables, cores that
require special extra shared objects to be available (because they
dlopen() things), etc.

What I have at my disposal is a bunch of local and cross-debuggers and
gcc/binutils suites, a bunch of potential executables that cores could
come from, and a bunch of shared objects that might or might not have
been loaded.  And, a core file.


What I'm trying to do is generate a script that can determine the right
set of GDB executable, GDB command line options (to "set sysroot",
etc.), etc., starting with only the core file itself as information,
then invoke GDB properly.

What options do I have to help me do this?  About all I have is using
"file" or maybe "readelf" to find the hardware type, so I can guess
more-or-less which cross-GDB I might need.  Then when I start GDB with
the core it will say something like "Core was generated by `someapp'."
Once I have the executable I can use "info shared" to find missing
dynamic objects.

This seems brittle, esp. the first part using "file" and parsing the
output of GDB to find the executable.  Does anyone have any better
options?  Are there any "info" operations in GDB I could use, given
_only_ the core file, that can help me?  What about some way of tagging
the executables themselves, either with a string or somehow in the
executable ELF image that would get added to the core, that I could
query given just the resulting core file?

Any ideas would be welcome.  Thanks!

Re: Classifying core files?

[Daniel Jacobowitz]

On Tue, Dec 09, 2008 at 12:09:08PM -0500, Paul Smith wrote:
> This seems brittle, esp. the first part using "file" and parsing the
> output of GDB to find the executable.  Does anyone have any better
> options?  Are there any "info" operations in GDB I could use, given
> _only_ the core file, that can help me?  What about some way of tagging
> the executables themselves, either with a string or somehow in the
> executable ELF image that would get added to the core, that I could
> query given just the resulting core file?

Sounds like you should take a look at Roland's and Jan's work on
build-id notes.  There's also a Linux kernel patch that causes the ELF
notes for the executable to be dumped to the core file; it should be
easy to replicate onto another system that supports core dumping.

-- 
Daniel Jacobowitz
CodeSourcery

Re: Classifying core files?

[Jan Kratochvil]

On Tue, 09 Dec 2008 18:09:08 +0100, Paul Smith wrote:
> What I have at my disposal is a bunch of local and cross-debuggers and
> gcc/binutils suites, a bunch of potential executables that cores could
> come from, and a bunch of shared objects that might or might not have
> been loaded.  And, a core file.

With old and/or non-Linux cores/executables there is no way to safely associate
them.  It may be some heuristics such as checking some _r_debug linklist
validity but it is nothing much reliable, if possible at all.  The primary
problem is that in the core files the readonly (commonly .text and .rodata)
pages are just omitted and readwrite pages cannot be compared as sure they
could be / usually are already modified in the core dumping time.

I do not understand much if you can generate new core files or if you are just
interested in the existing core files you have.  In the former case there are
some possibilities:

For recent Linux kernels you may get the full dumps (incl. the readonly pages)
using /proc/<pid>/coredump_filter (see its doc).  But still it will be
complicated to find the appropriate file containing the debuginfo for it.

More efficient and recommended would be build-ids:

With recent (2007 Oct+) Linux kernel the core files now contain also the first
page of the mapped ELF files.
http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=82df39738ba9e02c057fa99b7461a56117d36119
If you build your distro using `ld --build-id' (default for Fedora since
Fedora 8) each binary has its unique id and so this unique id is present also
in the core files.  Described at:
http://fedoraproject.org/wiki/Releases/FeatureBuildId

FSF GDB currently contains only the faster verification the separate .debug
files.  There is a patch going to be hopefully integrated one day which
supports automatically finding the right binary just being given a bare core
file (`gdb -c ./core.1234').  For gdb-6.8:
http://cvs.fedora.redhat.com/viewvc/rpms/gdb/devel/gdb-6.6-buildid-locate.patch?view=co
For GDB HEAD:
http://people.redhat.com/jkratoch/gdb-6.8.50.20081209-buildid-locate.patch

It still has currently merged two unrelated functionalities:
(a) The expected locating of appropriate binaries from a bare core file.
(b) Reporting missing debuginfo rpms from missing .debug files.


> What I'm trying to do is generate a script that can determine the right
> set of GDB executable, GDB command line options (to "set sysroot",
> etc.), etc., starting with only the core file itself as information,
> then invoke GDB properly.

Yes, it works as described although it understandably requires the appropriate
symlinks (files) for the build-id hash mapping:

lrwxrwxrwx 1 root root      23 2008-10-30 11:57 /usr/lib/debug/.build-id/29/93b8ca46989e7434eef8daf61f31e9f8d6a0ba -> ../../../../../bin/bash
-rwxr-xr-x 1 root root  834520 2008-10-28 22:37 /bin/bash
lrwxrwxrwx 1 root root      20 2008-10-30 11:57 /usr/lib/debug/.build-id/29/93b8ca46989e7434eef8daf61f31e9f8d6a0ba.debug -> ../../bin/bash.debug
-rwxr-xr-x 1 root root 1903112 2008-10-28 22:37 /usr/lib/debug/bin/bash.debug

(possibly placed in a separate sysroot)

Patched GDB also loads the shared libraries according to their build-id hashes.



Regards,
Jan