[lttng-dev] Extract lttng trace from kernel coredump

[mathieu.desnoyers@efficios.com (Mathieu Desnoyers)]

----- Original Message -----
> From: "Corey Minyard" <minyard@acm.org>
> To: "Mathieu Desnoyers" <mathieu.desnoyers at efficios.com>
> Cc: "David Goulet" <dgoulet at efficios.com>, lttng-dev at lists.lttng.org
> Sent: Tuesday, February 18, 2014 2:29:23 PM
> Subject: Re: [lttng-dev] Extract lttng trace from kernel coredump
> 
> I have attached some gdb macros for dumping LTT buffers from a kernel
> coredump.  I haven't done a lot of testing, though.

Very cool! I'll have a look when things get less busy here. I hope my
earlier reply helps clearing things out.

> 
> I do have one question.  I've been getting the metadata by taking a
> snapshot right when the trace starts and saving that to stick into the
> trace output from the coredump.  Is that the best way, or is there some
> other way I can extract it?  There doesn't seem to be a metadata channel
> running in snapshot mode.

Actually, it's better if you grab the metadata near when you grab the
coredump. Grabbing the metadata at the beginning of your tracing session
will not have the info about other modules that might be coming and going
while tracing is active (which might contain tracepoints).

You could grab the metadata from the coredump actually, this would be the
best approach. Looking at lttng modules 2.4 rc, you will want to look at

 /lttng-events.h

struct lttng_session {
  ...
  struct lttng_metadata_cache *metadata_cache;
  ...
};

Which has the metadata string:

struct lttng_metadata_cache {
        char *data;                     /* Metadata cache */
        unsigned int cache_alloc;       /* Metadata allocated size (bytes) */
        unsigned int metadata_written;  /* Number of bytes written in metadata cache */
        struct kref refcount;           /* Metadata cache usage */
        struct list_head metadata_stream;       /* Metadata stream list */
};

You simply need to dump this metadata string into a "metadata" file, add a
"/* CTF 1.8 */\n line at the beginning, and this will generate a "text only
CTF metadata", which can be read by babeltrace.

If you happen to core dump while the metadata cache is being resized, you might
encounter a corrupted metadata. It should not happen frequently, but it's possible.
We could eventually change lttng_metadata_printf() so it makes sure to always
keep a readable metadata around (by e.g. using kcalloc rather than krealloc and
dealing carefully with cache_alloc vs data fields updates).

Thoughts ?

Thanks,

Mathieu

-- 
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com