[ltt-dev] Mini Design and Roadmap of LTT-Kdump

[ltt-dev] Mini Design and Roadmap of LTT-Kdump

[Lai Jiangshan]

Mini Design and Roadmap of LTT-Kdump

-----------
	People in enterprise need to be able to diagnose why the system
failed. Failing once is acceptable from a customer perspective, but
failing again isn't. In this case, being able to extract the last events
before the crash can be very valuable and helps solving the problems
before they happen again.

	Create tools to simplify extraction of traces from crashed kernel.
The core file of crashed kernel is provided by kdump.
-----------

	We will implement it. This tool include two parts.

Part1: Core-file analyser.
	Analyse(needs kernel-debuginfo) the core-file and read
ltt-relay files.
	This part will use elf-libs for analysing, but we use crash(8)
instead at first. crash(8) can simplify this work and crash(8) can perform
on a compressed core-file(http://sourceforge.net/projects/makedumpfile/).
	When we use crash(8), we will write a gdb script for analysing.
crash(8) loads this gdb script, core-file and kernel-debuginfo then listens
to a pipe and does works.
	crash(8) works very well when pages are vmap()ed into to a continuous
memmory region. But ltt-relay's pages are not vmap()ed, it'll very slow.
so we may use elf-libs or enhance crash(8) at last.

Part2: ltt-relay extracter
	Extracter calls analyser's API to travel the debugfs tree in core-file,
and copies all ltt-relay files to the disk. The written files are the same
format exactly as the files what lttd writes, so we can use lttv or other
lttng tools to read the events.


Any comments and ideas are welcome!

Thanks, Lai

[ltt-dev] Mini Design and Roadmap of LTT-Kdump

[Mathieu Desnoyers]

* Lai Jiangshan (laijs at cn.fujitsu.com) wrote:
> 
> 
> Mini Design and Roadmap of LTT-Kdump
> 

Hi Lai,

This sounds very interesting and useful !

> -----------
> 	People in enterprise need to be able to diagnose why the system
> failed. Failing once is acceptable from a customer perspective, but
> failing again isn't. In this case, being able to extract the last events
> before the crash can be very valuable and helps solving the problems
> before they happen again.
> 
> 	Create tools to simplify extraction of traces from crashed kernel.
> The core file of crashed kernel is provided by kdump.
> -----------
> 
> 	We will implement it. This tool include two parts.
> 
> Part1: Core-file analyser.
> 	Analyse(needs kernel-debuginfo) the core-file and read
> ltt-relay files.
> 	This part will use elf-libs for analysing, but we use crash(8)
> instead at first. crash(8) can simplify this work and crash(8) can perform
> on a compressed core-file(http://sourceforge.net/projects/makedumpfile/).
> 	When we use crash(8), we will write a gdb script for analysing.
> crash(8) loads this gdb script, core-file and kernel-debuginfo then listens
> to a pipe and does works.
> 	crash(8) works very well when pages are vmap()ed into to a continuous
> memmory region. But ltt-relay's pages are not vmap()ed, it'll very slow.
> so we may use elf-libs or enhance crash(8) at last.

Why would it be so slow ? We don't have a contiguous memory mapping, but
we have a linked list of pages we can walk. I am not very familiar with
the crash(8) internals though, so there could be a limitation I do not
foresee...

> 
> Part2: ltt-relay extracter
> 	Extracter calls analyser's API to travel the debugfs tree in core-file,
> and copies all ltt-relay files to the disk. The written files are the same
> format exactly as the files what lttd writes, so we can use lttv or other
> lttng tools to read the events.

Yes, and we should think about giving the ability to copy the tracefiles
created from the crash buffers into an existing trace. So we get the
following scenario :

Tracing to disk.. trace gets written partially. The crash may happen
while the trace is being written.

The tool should detect these incomplete trace subbuffers on the disk and
truncate them. It should then add the information extracted from the
crash. This information can be added in separate tracefiles if
necessary, e.g. :

trace/sched_0 (normal tracefile for scheduler activity)
trace/crash/sched_0 (scheduler activity extracted from crash dump)

Best regards,

Mathieu




> 
> 
> Any comments and ideas are welcome!
> 
> Thanks, Lai
> 
> 

-- 
Mathieu Desnoyers
OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68

[ltt-dev] Mini Design and Roadmap of LTT-Kdump

[Lai Jiangshan]

Mathieu Desnoyers wrote:
> * Lai Jiangshan (laijs at cn.fujitsu.com) wrote:
>>
>> Mini Design and Roadmap of LTT-Kdump
>>
> 
> Hi Lai,
> 
> This sounds very interesting and useful !
> 
>> -----------
>> 	People in enterprise need to be able to diagnose why the system
>> failed. Failing once is acceptable from a customer perspective, but
>> failing again isn't. In this case, being able to extract the last events
>> before the crash can be very valuable and helps solving the problems
>> before they happen again.
>>
>> 	Create tools to simplify extraction of traces from crashed kernel.
>> The core file of crashed kernel is provided by kdump.
>> -----------
>>
>> 	We will implement it. This tool include two parts.
>>
>> Part1: Core-file analyser.
>> 	Analyse(needs kernel-debuginfo) the core-file and read
>> ltt-relay files.
>> 	This part will use elf-libs for analysing, but we use crash(8)
>> instead at first. crash(8) can simplify this work and crash(8) can perform
>> on a compressed core-file(http://sourceforge.net/projects/makedumpfile/).
>> 	When we use crash(8), we will write a gdb script for analysing.
>> crash(8) loads this gdb script, core-file and kernel-debuginfo then listens
>> to a pipe and does works.
>> 	crash(8) works very well when pages are vmap()ed into to a continuous
>> memmory region. But ltt-relay's pages are not vmap()ed, it'll very slow.
>> so we may use elf-libs or enhance crash(8) at last.
> 
> Why would it be so slow ? We don't have a contiguous memory mapping, but
> we have a linked list of pages we can walk. I am not very familiar with
> the crash(8) internals though, so there could be a limitation I do not
> foresee...

We use "append memory" to get memory:
append memory part2_pipe_name $buf->start $buf->start + $buf->chan->alloc_size
(this example for relay, not ltt-relay)

The length is not important, but if we call "append memory" millions times,
it will need several hours.

I use crash(8) instead hardcode to traveling structures in core-file by using
elf-libs. I hardly know elf-libs. I can learn it and use it.
But two things:

1) except un-vmaped pages, crash(8) does every thing well and simple.
example for traveling debugfs:
define debugfsls
        set $father = (struct dentry *)$arg1
        set $child_off = (size_t)&((struct dentry *)0)->d_u.d_child
        set $father_head = (void *)&$father->d_subdirs
        set $child_head = (void *)$father->d_subdirs.next
        while ($child_head != $father_head)
                set $child = (struct dentry *)($child_head - $child_off)
                if ($child->d_inode->i_fop == &relay_file_operations)
                        __dump_dentry__ $arg0 $RELAY_FILE $child
                end
                if (($child->d_inode->i_sb->s_magic == $DEBUGFS_MAGIC) && \
                                (($child->d_inode->i_mode & $S_IFDIR) != 0))
                        __dump_dentry__ $arg0 $DEBUGFS_DIR $child
                end
                set $child_head = (void *)$child->d_u.d_child.next
        end
        append value $arg0 $LIST_END
end

and crash(8) can handle several kinds of core-file.

2) Jobs should be done by professional guys.
   My code will be migrated to elf-libs very easy if anyone need.
   I will wait elf-professional to implement it or I implement it at last.

Lai