gdb on Linux Kernel dumps (gdb-kdump)

gdb on Linux Kernel dumps (gdb-kdump)

[Ales Novak]

Hello,

as was discussed at Andreas Arnez's talk during the GNU Cauldron 2015,
it would be nice to be able to work with Linux kernel dumps using the
ordinary gdb.  Linux kernel dumps may appear in multiple formats
(KDUMP/DISKDUMP being most efficient, but LKCD, Xen, .vmss, ... are
possible as well), while gdb does understand almost only the usual ELF.

The only tool to work with those dumps is crash-utility, which in fact
has an older version of gdb embedded in, yet doesn't use it effectively,
is not able to work multiarch, etc.

My colleagues and myself have explored the ways how to enable gdb to
open kdumps, and as a result, Petr Tesarik introduced a whole new
library "libkdumpfile", not only able to open the dumps in various
formats, but furthermore doing the virtual to physical memory mapping
(which gdb doesn't do).

My efforts resulted in a fork of gdb called "gdb-kdump". This one adds a
new target called "kdump", which uses libkdumpfile to access specified
kernel dump.  For each task from the dumped kernel a new gdb's thread
structure is created.  Currently it implements two commands "kdump ps"
which prints out the tasks, and "kdump modules /some/path" which does
find and load debuginfo files for the modules loaded in the dumped
kernel.

The gdb-kdump should be considered rather as a proof of concept, yet it
shows that enabling the kdumps in gdb should be principally easy. It is
able to open the x86_64 and s390x kdumps. I'd imagine that most of the
commands of the crash-utility could be reimplemented using the gdb's
Python interface.

The projects reside on github:

* https://github.com/ptesarik/libkdumpfile

* https://github.com/alesax/gdb-kdump

The example session looks like that:

------------------------------------------------------------------+
ales@pixla:~/dump1> gdb vmlinux.debug                             |
GNU gdb (GDB) 7.10                                                |
[...]                                                             |
For help, type "help".                                            |
Type "apropos word" to search for commands related to "word"...   |
Reading symbols from vmlinux.debug...done.                        |
(gdb) target kdump vmcore                                         |
Loaded processes: 346                                             |
(gdb) thread 322                                                  |
[Switching to thread 322 (pid 17626)]                             |
#0  worker_thread (__worker=<optimized out>)                      |
     at /usr/src/debug/kernel-default-3.0.101/linux-3.0/kernel/work|
(gdb) bt f                                                        |
#0  worker_thread (__worker=<optimized out>)                      |
     at /usr/src/debug/kernel-default-3.0.101/linux-3.0/kernel/work|
         worker = <unavailable>                                    |
         gcwq = (struct global_cwq*) 0xffff8807f6900c40            |
#1  0xffffffff810829a6 in kthread (_create=<optimized out>)       |
     at /usr/src/debug/kernel-default-3.0.101/linux-3.0/kernel/kthr|
         threadfn = 0xffffffff8107e4c0 <worker_thread>             |
         data = 0xffff8807f6900c40                                 |
         self = {should_stop = 0, data = 0xffff8807f6900c40, exited|
                     raw_lock = {slock = 0}}}}, task_list = {next =|
                 prev = (struct list_head*) 0xffff8807cc0e9f10}}}} |
         ret = <optimized out>                                     |
#2  0xffffffff8146b164 in kernel_thread_helper ()                 |
------------------------------------------------------------------+

We'd be thankful for any comments!

-- 
Ales Novak
SUSE L3

Re: gdb on Linux Kernel dumps (gdb-kdump)

[Andreas Arnez]

On Thu, Sep 17 2015, Ales Novak wrote:

> Hello,
>
> as was discussed at Andreas Arnez's talk during the GNU Cauldron 2015,
> it would be nice to be able to work with Linux kernel dumps using the
> ordinary gdb.

Agreed ;-)

> [...]
> 
> The gdb-kdump should be considered rather as a proof of concept, yet it
> shows that enabling the kdumps in gdb should be principally easy. It is
> able to open the x86_64 and s390x kdumps. I'd imagine that most of the
> commands of the crash-utility could be reimplemented using the gdb's
> Python interface.

Right.  As pointed out in the talk at the Cauldron, there are already a
few examples in the Linux kernel source tree under "scripts/gdb".  In
addition, this post mentions another GDB-enhancing project for Linux
kernel debugging, named "LKD":

  https://sourceware.org/ml/gdb-patches/2015-06/msg00040.html

The LKD project focuses more on live debugging and thus might be a
useful complement to gdb-kdump.

>
> The example session looks like that:
>
> [...]

Very nice!

> We'd be thankful for any comments!

IMHO, it would be very useful if GDB had support for various Linux
kernel dump formats and for the Linux kernel runtime.  It seems that
gdb-kdump covers both to some extent.

So, how to continue?  Would you be able to convert gdb-kdump into
patches against upstream GDB and send those to the gdb-patches mailing
list?

--
Andreas

Re: gdb on Linux Kernel dumps (gdb-kdump)

[Ales Novak]

On 2015-9-22 19:56, Andreas Arnez wrote:

> Right.  As pointed out in the talk at the Cauldron, there are already a
> few examples in the Linux kernel source tree under "scripts/gdb".  In
> addition, this post mentions another GDB-enhancing project for Linux
> kernel debugging, named "LKD":
>
>  https://sourceware.org/ml/gdb-patches/2015-06/msg00040.html
>
> The LKD project focuses more on live debugging and thus might be a
> useful complement to gdb-kdump.

Yes. The scripts in scripts/gdb/linux can be used, if they match the 
dumped kernel. LKD sounds interesting, being slightly in overlap with 
gdb-kdump.

> So, how to continue?  Would you be able to convert gdb-kdump into
> patches against upstream GDB and send those to the gdb-patches mailing
> list?

The github repo is based on the very recent 7.10 release, i.e. it can be 
applied almost harmlessly. Yet it brings dependance on the libkdumpfile, 
which in turn brings dependance on liblzo2, libz, libsnappy (these are 
used in the dump formats), which I don't know how upstream will like. 
Should I try to send it anyway?

-- 
Ales Novak

Re: gdb on Linux Kernel dumps (gdb-kdump)

[Mike Frysinger]

[-- Attachment #1: Type: text/plain, Size: 1533 bytes --]

On 23 Sep 2015 00:10, Ales Novak wrote:
> On 2015-9-22 19:56, Andreas Arnez wrote:
> > Right.  As pointed out in the talk at the Cauldron, there are already a
> > few examples in the Linux kernel source tree under "scripts/gdb".  In
> > addition, this post mentions another GDB-enhancing project for Linux
> > kernel debugging, named "LKD":
> >
> >  https://sourceware.org/ml/gdb-patches/2015-06/msg00040.html
> >
> > The LKD project focuses more on live debugging and thus might be a
> > useful complement to gdb-kdump.
> 
> Yes. The scripts in scripts/gdb/linux can be used, if they match the 
> dumped kernel. LKD sounds interesting, being slightly in overlap with 
> gdb-kdump.
> 
> > So, how to continue?  Would you be able to convert gdb-kdump into
> > patches against upstream GDB and send those to the gdb-patches mailing
> > list?
> 
> The github repo is based on the very recent 7.10 release, i.e. it can be 
> applied almost harmlessly. Yet it brings dependance on the libkdumpfile, 
> which in turn brings dependance on liblzo2, libz, libsnappy (these are 
> used in the dump formats), which I don't know how upstream will like. 
> Should I try to send it anyway?

as long as support is behind a configure flag/check, it shouldn't be a
problem.  gdb doesn't generally care about transitive deps either -- in
this case it merely relies on libkdumpfile.  i'd note that libkdumpfile
has all those libs listed as optional too.  its license is GPL, so that
shouldn't be a problem either.
-mike

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]