Tools to classify / uniquify core dumps or stack traces?

Tools to classify / uniquify core dumps or stack traces?

[Paul Smith]

Hi all.  I've got an environment where I'm getting lots of core dumps
from various places and it's very tedious to go through them and
determine which ones are for unique problems, and which are essentially
duplicates (same bug causing the core dump).

I was thinking of throwing together some kind of Perl or Python script
that could compare and categorize stack traces, but I thought surely
someone must have done something like this before.

Anyone have any pointers or thoughts about something like this?

Re: Tools to classify / uniquify core dumps or stack traces?

[Jan Kratochvil]

On Thu, 01 Aug 2013 21:26:21 +0200, Paul Smith wrote:
> Hi all.  I've got an environment where I'm getting lots of core dumps
> from various places and it's very tedious to go through them and
> determine which ones are for unique problems, and which are essentially
> duplicates (same bug causing the core dump).
> 
> I was thinking of throwing together some kind of Perl or Python script
> that could compare and categorize stack traces, but I thought surely
> someone must have done something like this before.
> 
> Anyone have any pointers or thoughts about something like this?

ABRT, it has several backends how to report the results (the typical one is
Bugzilla), it also supports heuristic duplicates detection etc.
It is shipped in all recent Fedora releases by default and it is also
a project deployable on any OS:
	https://fedorahosted.org/abrt/

There is also Apport (I do not have experience with it).


Jan

Re: Tools to classify / uniquify core dumps or stack traces?

[Aurelian Melinte]

On 01/08/2013 3:35 PM, Jan Kratochvil wrote:
> On Thu, 01 Aug 2013 21:26:21 +0200, Paul Smith wrote:
>    
>> Hi all.  I've got an environment where I'm getting lots of core dumps
>> from various places and it's very tedious to go through them and
>> determine which ones are for unique problems, and which are essentially
>> duplicates (same bug causing the core dump).
>>
>> I was thinking of throwing together some kind of Perl or Python script
>> that could compare and categorize stack traces, but I thought surely
>> someone must have done something like this before.
>>
>> Anyone have any pointers or thoughts about something like this?
>>      

I use a home-brewed script - pasted below - to turn a collection of core 
dumps into a report for each core. Then I sift manually through but 
still much faster than analyzing core by core.

Regards,
a.


#!/bin/bash

#
# A script to extract core-file informations
#


if  [  $#  -ne  1  ]
then
   echo  "Usage:`basename $0`  <for-binary-image>"
   exit  -1
else
   binimg=$1
fi


# Today and yesterdays cores
cores=`find  .-name  '*.core'  -mtime  -1`

#cores=`find . -name '*.core'`


for  corein  $cores
do
   gdblogfile="$core-gdb.log"
   rm  $gdblogfile

   bininfo=`ls  -l  $binimg`
   coreinfo=`ls  -l  $core`

   gdb  -batch  \
       -ex  "set logging file$gdblogfile"  \
       -ex  "set logging on"  \
       -ex  "set pagination off"  \
       -ex  "printf\"**\n** Process info for$binimg  -$core  \n** Generated`date`\n\""  \
       -ex  "printf\"**\n**$bininfo  \n**$coreinfo\n**\n\""  \
       -ex  "file$binimg"  \
       -ex  "core-file$core"  \
       -ex  "bt"  \
       -ex  "info proc"  \
       -ex  "printf\"*\n* Libraries\n*\n\""  \
       -ex  "info sharedlib"  \
       -ex  "printf\"*\n* Memory map\n*\n\""  \
       -ex  "info target"  \
       -ex  "printf\"*\n* Registers\n*\n\""  \
       -ex  "info registers"  \
       -ex  "printf\"*\n* Current instructions\n*\n\""  -ex  "x/16i\$pc"  \
       -ex  "printf\"*\n* Threads (full)\n*\n\""  \
       -ex  "info threads"  \
       -ex  "bt"  \
       -ex  "thread apply all bt full"  \
       -ex  "printf\"*\n* Threads (basic)\n*\n\""  \
       -ex  "info threads"  \
       -ex  "thread apply all bt"  \
       -ex  "printf\"*\n* Done\n*\n\""  \
       -ex  "quit"
done

Re: Tools to classify / uniquify core dumps or stack traces?

[Paul Smith]

On Thu, 2013-08-01 at 21:35 +0200, Jan Kratochvil wrote:
> On Thu, 01 Aug 2013 21:26:21 +0200, Paul Smith wrote:
> > Hi all.  I've got an environment where I'm getting lots of core dumps
> > from various places and it's very tedious to go through them and
> > determine which ones are for unique problems, and which are essentially
> > duplicates (same bug causing the core dump).
> > 
> > I was thinking of throwing together some kind of Perl or Python script
> > that could compare and categorize stack traces, but I thought surely
> > someone must have done something like this before.
> > 
> > Anyone have any pointers or thoughts about something like this?
> 
> ABRT, it has several backends how to report the results (the typical one is
> Bugzilla), it also supports heuristic duplicates detection etc.
> It is shipped in all recent Fedora releases by default and it is also
> a project deployable on any OS:
> 	https://fedorahosted.org/abrt/
> 
> There is also Apport (I do not have experience with it).

I don't need to collect the cores (at least not yet), they are being
obtained through other methods (which I don't have a lot of control
over, currently) and I have bunches of them sitting in a directory.
Along with them I have the binary they were generated from, a source
tree it was built from, and a sysroot containing the runtime libraries
it was invoked with.  I can easily script up something to generate GDB
stack traces from them.  I just need something to reduce/classify them.

It looks like btparser is close to what I need.  It's an unusual choice,
IMHO, to use C to implement something which so fundamentally depends on
text manipulation; I would have chosen Perl or Python myself.  But I'll
definitely take a look.

I notice that "satyr" is said to be "the next generation btparser", but
from my very brief overview it appears to be more tightly bound up with
abrt and the way that abrt packages things, and I'm not using abrt.


Thanks for the pointer!

Re: Tools to classify / uniquify core dumps or stack traces?

[Phil Muldoon]

On 02/08/13 02:13, Paul Smith wrote:
> On Thu, 2013-08-01 at 21:35 +0200, Jan Kratochvil wrote:
>> On Thu, 01 Aug 2013 21:26:21 +0200, Paul Smith wrote:
>>> Hi all.  I've got an environment where I'm getting lots of core dumps
>>> from various places and it's very tedious to go through them and
>>> determine which ones are for unique problems, and which are essentially
>>> duplicates (same bug causing the core dump).
>>>
>>> I was thinking of throwing together some kind of Perl or Python script
>>> that could compare and categorize stack traces, but I thought surely
>>> someone must have done something like this before.
>>>
>>> Anyone have any pointers or thoughts about something like this?
>>
>> ABRT, it has several backends how to report the results (the typical one is
>> Bugzilla), it also supports heuristic duplicates detection etc.
>> It is shipped in all recent Fedora releases by default and it is also
>> a project deployable on any OS:
>> 	https://fedorahosted.org/abrt/
>>
>> There is also Apport (I do not have experience with it).
> 
> It looks like btparser is close to what I need.  It's an unusual choice,
> IMHO, to use C to implement something which so fundamentally depends on
> text manipulation; I would have chosen Perl or Python myself.  But I'll
> definitely take a look.

Why not use the Python API in GDB to automate your task?

http://sourceware.org/gdb/current/onlinedocs/gdb/Python.html#Python

If there are things missing in the API regarding what you need for
your task, I will be happy to add them.


Cheers,

Phil