[lttng-dev] Extract lttng trace from kernel coredump

[lttng-dev] Extract lttng trace from kernel coredump

[Corey Minyard]

On 01/18/2014 01:00 PM, Mathieu Desnoyers wrote:
> You can use the flight recorder mode in recent LTTng for this (2.3 and
> newer). It simply writes to memory, without any output. I understand that
> you want to create a contiguous ring buffer memory layout. However, you
> have to be aware that this will probably be done using either
>
> a) statically allocated memory at boot time (not very flexible),
> b) vmalloc() (very flexible, but can triggers minor page faults, which
>    can interact badly with page fault instrumentation. vmalloc() space
>    is often limited by a kernel boot time parameter, and is putting
>    quite important limitations on systems with 32-bit address spaces).
>
>> But I am certainly open to suggestions on how to do this, and happy to
>> have anything included back into the mainline.
>>
>> And I'm still learning about the internals of LTT.
> One option would be to modify the tool to understand the LTTng 2.x buffer
> layout by stitching pages together by software using the LTTng
> libringbuffer "subbuffer table". You can think of it as a 2-level page
> table, but one level indexes the sub-buffers, and the next level indexes
> the pages within a sub-buffer.

I'm finally back to this.  I discovered that /proc/vmcore did not map
vmalloc-ed memory, so I had to come up with something to handle that
before I could continue to work on this.

This information was very useful, and snapshot mode is definitely the
way to go.  I just want to make sure I understand this before I go on. I
have some specific question:

consumed is where the data starts and offset is where the data ends.  So
just go through the subbufs through a double index.  Calculate the
start/end location by taking the consumed/offset value, dividing that by
the size of a subbuffer, looking up that value in backend.buf_wsb,
getting the index from the id there, then indexing into the
backend.array with the index.  Once you have the subbuffer, you mod the
location by the size of a subbuffer and that's the subbuffer offset. 
Divide the subbuffer offset by a page size to get the page index in the
subbuffer, and mod by a page size to get the offset into the page.

Starting from the consumed position, dump data from pages until you hit
subbufer->data_size, then move to the next subbuffer.  On the last
subbuffer, you have to fill in the header and dump up to the offset.

I think I'm missing something, though, because the data size of the last
subbuffer doesn't match the offset location in that subbuffer.  It's a
pretty good distance away.

Thanks,

-corey

> A good way to understand its layout is to look at:
>
> lttng-modules (master)
> lib/ringbuffer/ring_buffer_backend.c
>
> lib_ring_buffer_backend_allocate()
>
> lib/ringbuffer/backend_types.h
>
> struct lib_ring_buffer_backend
> struct lib_ring_buffer_backend_subbuffer
> struct lib_ring_buffer_backend_pages
> struct lib_ring_buffer_backend_page
>
> In your case, you never care about the bufb->buf_rsb (read-side owned subbuffer),
> because you always ever just write into it. buf_rsb is only useful when taking
> snapshots.
>
> bufb->buf_wsb[] has the mapping from sub-buffers write-side index within
> the buffer to the associated index into bufb->array[], which allows getting the
> actual sub-buffers and memory pages associated to each buffer.
>
> You'll notice that the "id" field within struct lib_ring_buffer_backend_subbuffer
> is actually made of a mask of many fields. In order to understand how to use it,
> see 
>
> lib/ringbuffer/backend_types.h
>
> where we provide helpers to get and set the various information elements
> contained within the "id" field. See subbuffer_id*() functions and comments
> surrounding them.
>
> So you'll need to use the structures presented above to make sense of the memory
> layout of a buffer, and reorganize it into a CTF file that can be read by
> Babeltrace or other CTF trace readers.
>
> The algorithm you want to end up doing (offline, on a vmcore) is pretty much
> the same as grabbing an online snapshot (iterate from the consumer position up to
> the producer position, see lib/ringbuffer/frontend.h:ib_ring_buffer_snapshot() ).
> You will need an extra trick to handle the sub-buffer that was being written to
> at the time of the crash, by using the
>
> lib/ringbuffer/frontend_types.h struct commit_counters_hot "seq" field
>
> which is designed to track the contiguously committed data within the currently
> written buffer. This can be used at any point in time (whenever a crash occurs) to
> populate the last sub-buffer's content size, packet size, see:
>
> lttng-ring-buffer-client.h: client_buffer_end()
>
> and find out how much of the last sub-buffer needs to be copied into the output
> CTF trace.
>
> Thanks,
>
> Mathieu
>
>> Thanks,
>>
>> -corey
>>

[lttng-dev] Extract lttng trace from kernel coredump

[Mathieu Desnoyers]

----- Original Message -----
> From: "Corey Minyard" <minyard@acm.org>
> To: "Mathieu Desnoyers" <mathieu.desnoyers at efficios.com>
> Cc: "David Goulet" <dgoulet at efficios.com>, lttng-dev at lists.lttng.org
> Sent: Thursday, February 13, 2014 2:31:13 PM
> Subject: Re: [lttng-dev] Extract lttng trace from kernel coredump
> 
> On 01/18/2014 01:00 PM, Mathieu Desnoyers wrote:
> > You can use the flight recorder mode in recent LTTng for this (2.3 and
> > newer). It simply writes to memory, without any output. I understand that
> > you want to create a contiguous ring buffer memory layout. However, you
> > have to be aware that this will probably be done using either
> >
> > a) statically allocated memory at boot time (not very flexible),
> > b) vmalloc() (very flexible, but can triggers minor page faults, which
> >    can interact badly with page fault instrumentation. vmalloc() space
> >    is often limited by a kernel boot time parameter, and is putting
> >    quite important limitations on systems with 32-bit address spaces).
> >
> >> But I am certainly open to suggestions on how to do this, and happy to
> >> have anything included back into the mainline.
> >>
> >> And I'm still learning about the internals of LTT.
> > One option would be to modify the tool to understand the LTTng 2.x buffer
> > layout by stitching pages together by software using the LTTng
> > libringbuffer "subbuffer table". You can think of it as a 2-level page
> > table, but one level indexes the sub-buffers, and the next level indexes
> > the pages within a sub-buffer.
> 
> I'm finally back to this.  I discovered that /proc/vmcore did not map
> vmalloc-ed memory, so I had to come up with something to handle that
> before I could continue to work on this.
> 
> This information was very useful, and snapshot mode is definitely the
> way to go.  I just want to make sure I understand this before I go on. I
> have some specific question:

Sorry for the lag... ;)

I'll reply to your questions below just to make sure we're in sync.

> 
> consumed is where the data starts and offset is where the data ends.

Yes, approximately. This is right for consumed (this is where 
consumable data starts), but "offset" is either:
- where the data ends, or
- slightly beyond where the last contiguously committed data end.

It means we have to take extra care for buffers where the amount of
commit_seq count is not a multiple of the sub-buffer size.

> So
> just go through the subbufs through a double index.  Calculate the
> start/end location by taking the consumed/offset value,

First apply a module buffer size, otherwise the free running counters
divided by subbuffer size will go beyond the backend.buf_wsb array.

> dividing that by
> the size of a subbuffer,

Yes,

> looking up that value in backend.buf_wsb,
> getting the index from the id there, then indexing into the
> backend.array with the index.

Yes.

> Once you have the subbuffer, you mod the
> location by the size of a subbuffer and that's the subbuffer offset.

Exactly.

> Divide the subbuffer offset by a page size to get the page index in the
> subbuffer, and mod by a page size to get the offset into the page.

Yes.

> 
> Starting from the consumed position, dump data from pages until you hit
> subbufer->data_size, then move to the next subbuffer.  On the last
> subbuffer, you have to fill in the header and dump up to the offset.

Not quite exactly. It's better to do:

for each subbuffer between consumed and offset (inclusive)
  - if commit_seq is multiple of subbuffer size
    dump subbuffer->data_size
  - if commit_seq is not a multiple of subbuffer size,
    dump commit_seq % subbuf size bytes of data, filling up the
    header accordingly.

Basically, all data that was "being written" (between commit_seq % subbuf size
and write offset % subbuf size) cannot be read, because it likely contains
holes and/or incomplete data.

> 
> I think I'm missing something, though, because the data size of the last
> subbuffer doesn't match the offset location in that subbuffer.  It's a
> pretty good distance away.

Does my explanation above help clear things out ?

Thanks,

Mathieu

> 
> Thanks,
> 
> -corey
> 
> > A good way to understand its layout is to look at:
> >
> > lttng-modules (master)
> > lib/ringbuffer/ring_buffer_backend.c
> >
> > lib_ring_buffer_backend_allocate()
> >
> > lib/ringbuffer/backend_types.h
> >
> > struct lib_ring_buffer_backend
> > struct lib_ring_buffer_backend_subbuffer
> > struct lib_ring_buffer_backend_pages
> > struct lib_ring_buffer_backend_page
> >
> > In your case, you never care about the bufb->buf_rsb (read-side owned
> > subbuffer),
> > because you always ever just write into it. buf_rsb is only useful when
> > taking
> > snapshots.
> >
> > bufb->buf_wsb[] has the mapping from sub-buffers write-side index within
> > the buffer to the associated index into bufb->array[], which allows getting
> > the
> > actual sub-buffers and memory pages associated to each buffer.
> >
> > You'll notice that the "id" field within struct
> > lib_ring_buffer_backend_subbuffer
> > is actually made of a mask of many fields. In order to understand how to
> > use it,
> > see
> >
> > lib/ringbuffer/backend_types.h
> >
> > where we provide helpers to get and set the various information elements
> > contained within the "id" field. See subbuffer_id*() functions and comments
> > surrounding them.
> >
> > So you'll need to use the structures presented above to make sense of the
> > memory
> > layout of a buffer, and reorganize it into a CTF file that can be read by
> > Babeltrace or other CTF trace readers.
> >
> > The algorithm you want to end up doing (offline, on a vmcore) is pretty
> > much
> > the same as grabbing an online snapshot (iterate from the consumer position
> > up to
> > the producer position, see
> > lib/ringbuffer/frontend.h:ib_ring_buffer_snapshot() ).
> > You will need an extra trick to handle the sub-buffer that was being
> > written to
> > at the time of the crash, by using the
> >
> > lib/ringbuffer/frontend_types.h struct commit_counters_hot "seq" field
> >
> > which is designed to track the contiguously committed data within the
> > currently
> > written buffer. This can be used at any point in time (whenever a crash
> > occurs) to
> > populate the last sub-buffer's content size, packet size, see:
> >
> > lttng-ring-buffer-client.h: client_buffer_end()
> >
> > and find out how much of the last sub-buffer needs to be copied into the
> > output
> > CTF trace.
> >
> > Thanks,
> >
> > Mathieu
> >
> >> Thanks,
> >>
> >> -corey
> >>
> 
> 

-- 
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com

[lttng-dev] Extract lttng trace from kernel coredump

[Corey Minyard]

On 02/18/2014 02:02 PM, Mathieu Desnoyers wrote:
>> Starting from the consumed position, dump data from pages until you hit
>> subbufer->data_size, then move to the next subbuffer.  On the last
>> subbuffer, you have to fill in the header and dump up to the offset.
> Not quite exactly. It's better to do:
>
> for each subbuffer between consumed and offset (inclusive)
>   - if commit_seq is multiple of subbuffer size
>     dump subbuffer->data_size
>   - if commit_seq is not a multiple of subbuffer size,
>     dump commit_seq % subbuf size bytes of data, filling up the
>     header accordingly.
>
> Basically, all data that was "being written" (between commit_seq % subbuf size
> and write offset % subbuf size) cannot be read, because it likely contains
> holes and/or incomplete data.
>
>> I think I'm missing something, though, because the data size of the last
>> subbuffer doesn't match the offset location in that subbuffer.  It's a
>> pretty good distance away.
> Does my explanation above help clear things out ?

Yes, I've modified the code a bit to use the hot commit data if
necessary, and to dump the metadata from the metadata cache.  A new
version is attached.

I also modified to use the flushed value if present.  The probabilities
of that situation causing a problem are pretty low, but this will avoid
any issue.  It looks correct to me.

-corey

> Thanks,
>
> Mathieu
>
>> Thanks,
>>
>> -corey
>>
>>> A good way to understand its layout is to look at:
>>>
>>> lttng-modules (master)
>>> lib/ringbuffer/ring_buffer_backend.c
>>>
>>> lib_ring_buffer_backend_allocate()
>>>
>>> lib/ringbuffer/backend_types.h
>>>
>>> struct lib_ring_buffer_backend
>>> struct lib_ring_buffer_backend_subbuffer
>>> struct lib_ring_buffer_backend_pages
>>> struct lib_ring_buffer_backend_page
>>>
>>> In your case, you never care about the bufb->buf_rsb (read-side owned
>>> subbuffer),
>>> because you always ever just write into it. buf_rsb is only useful when
>>> taking
>>> snapshots.
>>>
>>> bufb->buf_wsb[] has the mapping from sub-buffers write-side index within
>>> the buffer to the associated index into bufb->array[], which allows getting
>>> the
>>> actual sub-buffers and memory pages associated to each buffer.
>>>
>>> You'll notice that the "id" field within struct
>>> lib_ring_buffer_backend_subbuffer
>>> is actually made of a mask of many fields. In order to understand how to
>>> use it,
>>> see
>>>
>>> lib/ringbuffer/backend_types.h
>>>
>>> where we provide helpers to get and set the various information elements
>>> contained within the "id" field. See subbuffer_id*() functions and comments
>>> surrounding them.
>>>
>>> So you'll need to use the structures presented above to make sense of the
>>> memory
>>> layout of a buffer, and reorganize it into a CTF file that can be read by
>>> Babeltrace or other CTF trace readers.
>>>
>>> The algorithm you want to end up doing (offline, on a vmcore) is pretty
>>> much
>>> the same as grabbing an online snapshot (iterate from the consumer position
>>> up to
>>> the producer position, see
>>> lib/ringbuffer/frontend.h:ib_ring_buffer_snapshot() ).
>>> You will need an extra trick to handle the sub-buffer that was being
>>> written to
>>> at the time of the crash, by using the
>>>
>>> lib/ringbuffer/frontend_types.h struct commit_counters_hot "seq" field
>>>
>>> which is designed to track the contiguously committed data within the
>>> currently
>>> written buffer. This can be used at any point in time (whenever a crash
>>> occurs) to
>>> populate the last sub-buffer's content size, packet size, see:
>>>
>>> lttng-ring-buffer-client.h: client_buffer_end()
>>>
>>> and find out how much of the last sub-buffer needs to be copied into the
>>> output
>>> CTF trace.
>>>
>>> Thanks,
>>>
>>> Mathieu
>>>
>>>> Thanks,
>>>>
>>>> -corey
>>>>
>>

-------------- next part --------------
A non-text attachment was scrubbed...
Name: ltt.py
Type: text/x-python
Size: 17137 bytes
Desc: not available
URL: <http://lists.lttng.org/pipermail/lttng-dev/attachments/20140219/379ed30a/attachment.py>

[lttng-dev] Extract lttng trace from kernel coredump

[Corey Minyard]

I have attached some gdb macros for dumping LTT buffers from a kernel
coredump.  I haven't done a lot of testing, though.

I do have one question.  I've been getting the metadata by taking a
snapshot right when the trace starts and saving that to stick into the
trace output from the coredump.  Is that the best way, or is there some
other way I can extract it?  There doesn't seem to be a metadata channel
running in snapshot mode.

Thanks,

-corey
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ltt.py
Type: text/x-python
Size: 16072 bytes
Desc: not available
URL: <http://lists.lttng.org/pipermail/lttng-dev/attachments/20140218/00922103/attachment.py>

[lttng-dev] Extract lttng trace from kernel coredump

[Mathieu Desnoyers]

----- Original Message -----
> From: "Corey Minyard" <minyard@acm.org>
> To: "Mathieu Desnoyers" <mathieu.desnoyers at efficios.com>
> Cc: "David Goulet" <dgoulet at efficios.com>, lttng-dev at lists.lttng.org
> Sent: Tuesday, February 18, 2014 2:29:23 PM
> Subject: Re: [lttng-dev] Extract lttng trace from kernel coredump
> 
> I have attached some gdb macros for dumping LTT buffers from a kernel
> coredump.  I haven't done a lot of testing, though.

Very cool! I'll have a look when things get less busy here. I hope my
earlier reply helps clearing things out.

> 
> I do have one question.  I've been getting the metadata by taking a
> snapshot right when the trace starts and saving that to stick into the
> trace output from the coredump.  Is that the best way, or is there some
> other way I can extract it?  There doesn't seem to be a metadata channel
> running in snapshot mode.

Actually, it's better if you grab the metadata near when you grab the
coredump. Grabbing the metadata at the beginning of your tracing session
will not have the info about other modules that might be coming and going
while tracing is active (which might contain tracepoints).

You could grab the metadata from the coredump actually, this would be the
best approach. Looking at lttng modules 2.4 rc, you will want to look at

 /lttng-events.h

struct lttng_session {
  ...
  struct lttng_metadata_cache *metadata_cache;
  ...
};

Which has the metadata string:

struct lttng_metadata_cache {
        char *data;                     /* Metadata cache */
        unsigned int cache_alloc;       /* Metadata allocated size (bytes) */
        unsigned int metadata_written;  /* Number of bytes written in metadata cache */
        struct kref refcount;           /* Metadata cache usage */
        struct list_head metadata_stream;       /* Metadata stream list */
};

You simply need to dump this metadata string into a "metadata" file, add a
"/* CTF 1.8 */\n line at the beginning, and this will generate a "text only
CTF metadata", which can be read by babeltrace.

If you happen to core dump while the metadata cache is being resized, you might
encounter a corrupted metadata. It should not happen frequently, but it's possible.
We could eventually change lttng_metadata_printf() so it makes sure to always
keep a readable metadata around (by e.g. using kcalloc rather than krealloc and
dealing carefully with cache_alloc vs data fields updates).

Thoughts ?

Thanks,

Mathieu

-- 
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com

[lttng-dev] Extract lttng trace from kernel coredump

[Corey Minyard]

On 02/18/2014 03:02 PM, Mathieu Desnoyers wrote:
> ----- Original Message -----
>> From: "Corey Minyard" <minyard@acm.org>
>> To: "Mathieu Desnoyers" <mathieu.desnoyers at efficios.com>
>> Cc: "David Goulet" <dgoulet at efficios.com>, lttng-dev at lists.lttng.org
>> Sent: Tuesday, February 18, 2014 2:29:23 PM
>> Subject: Re: [lttng-dev] Extract lttng trace from kernel coredump
>>
>> I have attached some gdb macros for dumping LTT buffers from a kernel
>> coredump.  I haven't done a lot of testing, though.
> Very cool! I'll have a look when things get less busy here. I hope my
> earlier reply helps clearing things out.

Yes, I'm going through that and the code I have to get things clean.

>> I do have one question.  I've been getting the metadata by taking a
>> snapshot right when the trace starts and saving that to stick into the
>> trace output from the coredump.  Is that the best way, or is there some
>> other way I can extract it?  There doesn't seem to be a metadata channel
>> running in snapshot mode.
> Actually, it's better if you grab the metadata near when you grab the
> coredump. Grabbing the metadata at the beginning of your tracing session
> will not have the info about other modules that might be coming and going
> while tracing is active (which might contain tracepoints).
>
> You could grab the metadata from the coredump actually, this would be the
> best approach. Looking at lttng modules 2.4 rc, you will want to look at
>
>  /lttng-events.h
>
> struct lttng_session {
>   ...
>   struct lttng_metadata_cache *metadata_cache;
>   ...
> };
>
> Which has the metadata string:
>
> struct lttng_metadata_cache {
>         char *data;                     /* Metadata cache */
>         unsigned int cache_alloc;       /* Metadata allocated size (bytes) */
>         unsigned int metadata_written;  /* Number of bytes written in metadata cache */
>         struct kref refcount;           /* Metadata cache usage */
>         struct list_head metadata_stream;       /* Metadata stream list */
> };
>
> You simply need to dump this metadata string into a "metadata" file, add a
> "/* CTF 1.8 */\n line at the beginning, and this will generate a "text only
> CTF metadata", which can be read by babeltrace.
>
> If you happen to core dump while the metadata cache is being resized, you might
> encounter a corrupted metadata. It should not happen frequently, but it's possible.
> We could eventually change lttng_metadata_printf() so it makes sure to always
> keep a readable metadata around (by e.g. using kcalloc rather than krealloc and
> dealing carefully with cache_alloc vs data fields updates).
>
> Thoughts ?

Perfect, that's easy.  I'll do that.

Thanks again.

-corey

[lttng-dev] Extract lttng trace from kernel coredump

[Mathieu Desnoyers]

----- Original Message -----
> From: "Mathieu Desnoyers" <mathieu.desnoyers@efficios.com>
> To: minyard at acm.org
> Cc: lttng-dev at lists.lttng.org
> Sent: Tuesday, February 18, 2014 4:02:15 PM
> Subject: Re: [lttng-dev] Extract lttng trace from kernel coredump
> 
> ----- Original Message -----
> > From: "Corey Minyard" <minyard@acm.org>
> > To: "Mathieu Desnoyers" <mathieu.desnoyers at efficios.com>
> > Cc: "David Goulet" <dgoulet at efficios.com>, lttng-dev at lists.lttng.org
> > Sent: Tuesday, February 18, 2014 2:29:23 PM
> > Subject: Re: [lttng-dev] Extract lttng trace from kernel coredump
> > 
> > I have attached some gdb macros for dumping LTT buffers from a kernel
> > coredump.  I haven't done a lot of testing, though.
> 
> Very cool! I'll have a look when things get less busy here. I hope my
> earlier reply helps clearing things out.
> 
> > 
> > I do have one question.  I've been getting the metadata by taking a
> > snapshot right when the trace starts and saving that to stick into the
> > trace output from the coredump.  Is that the best way, or is there some
> > other way I can extract it?  There doesn't seem to be a metadata channel
> > running in snapshot mode.
> 
> Actually, it's better if you grab the metadata near when you grab the
> coredump. Grabbing the metadata at the beginning of your tracing session
> will not have the info about other modules that might be coming and going
> while tracing is active (which might contain tracepoints).
> 
> You could grab the metadata from the coredump actually, this would be the
> best approach. Looking at lttng modules 2.4 rc, you will want to look at
> 
>  /lttng-events.h
> 
> struct lttng_session {
>   ...
>   struct lttng_metadata_cache *metadata_cache;
>   ...
> };
> 
> Which has the metadata string:
> 
> struct lttng_metadata_cache {
>         char *data;                     /* Metadata cache */
>         unsigned int cache_alloc;       /* Metadata allocated size (bytes) */
>         unsigned int metadata_written;  /* Number of bytes written in
>         metadata cache */
>         struct kref refcount;           /* Metadata cache usage */
>         struct list_head metadata_stream;       /* Metadata stream list */
> };
> 
> You simply need to dump this metadata string into a "metadata" file, add a
> "/* CTF 1.8 */\n line at the beginning, and this will generate a "text only
> CTF metadata", which can be read by babeltrace.
> 
> If you happen to core dump while the metadata cache is being resized, you
> might
> encounter a corrupted metadata. It should not happen frequently, but it's
> possible.
> We could eventually change lttng_metadata_printf() so it makes sure to always
> keep a readable metadata around (by e.g. using kcalloc rather than krealloc
> and
> dealing carefully with cache_alloc vs data fields updates).

Oops, I meant "kzalloc".

I'm attaching a patch that should take care of making sure it is always in
a valid state when a core dump occurs. You should be able to safely read
"metadata_flushed" bytes of data at any time. Whatever has been flushed
should be parseable by Babeltrace.

Thoughts ?

Thanks,

Mathieu

> 
> Thoughts ?
> 
> Thanks,
> 
> Mathieu
> 
> --
> Mathieu Desnoyers
> EfficiOS Inc.
> http://www.efficios.com
> 
> _______________________________________________
> lttng-dev mailing list
> lttng-dev at lists.lttng.org
> http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
> 

-- 
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com
-------------- next part --------------
A non-text attachment was scrubbed...
Name: metadata-core-dump.patch
Type: text/x-patch
Size: 6041 bytes
Desc: not available
URL: <http://lists.lttng.org/pipermail/lttng-dev/attachments/20140218/45a87053/attachment-0001.bin>

[lttng-dev] Extract lttng trace from kernel coredump

[Mathieu Desnoyers]

----- Original Message -----
> From: "Mathieu Desnoyers" <mathieu.desnoyers@efficios.com>
> To: minyard at acm.org
> Cc: lttng-dev at lists.lttng.org
> Sent: Tuesday, February 18, 2014 4:42:06 PM
> Subject: Re: [lttng-dev] Extract lttng trace from kernel coredump
> 
> ----- Original Message -----
> > From: "Mathieu Desnoyers" <mathieu.desnoyers@efficios.com>
> > To: minyard at acm.org
> > Cc: lttng-dev at lists.lttng.org
> > Sent: Tuesday, February 18, 2014 4:02:15 PM
> > Subject: Re: [lttng-dev] Extract lttng trace from kernel coredump
> > 
> > ----- Original Message -----
> > > From: "Corey Minyard" <minyard@acm.org>
> > > To: "Mathieu Desnoyers" <mathieu.desnoyers at efficios.com>
> > > Cc: "David Goulet" <dgoulet at efficios.com>, lttng-dev at lists.lttng.org
> > > Sent: Tuesday, February 18, 2014 2:29:23 PM
> > > Subject: Re: [lttng-dev] Extract lttng trace from kernel coredump
> > > 
> > > I have attached some gdb macros for dumping LTT buffers from a kernel
> > > coredump.  I haven't done a lot of testing, though.
> > 
> > Very cool! I'll have a look when things get less busy here. I hope my
> > earlier reply helps clearing things out.
> > 
> > > 
> > > I do have one question.  I've been getting the metadata by taking a
> > > snapshot right when the trace starts and saving that to stick into the
> > > trace output from the coredump.  Is that the best way, or is there some
> > > other way I can extract it?  There doesn't seem to be a metadata channel
> > > running in snapshot mode.
> > 
> > Actually, it's better if you grab the metadata near when you grab the
> > coredump. Grabbing the metadata at the beginning of your tracing session
> > will not have the info about other modules that might be coming and going
> > while tracing is active (which might contain tracepoints).
> > 
> > You could grab the metadata from the coredump actually, this would be the
> > best approach. Looking at lttng modules 2.4 rc, you will want to look at
> > 
> >  /lttng-events.h
> > 
> > struct lttng_session {
> >   ...
> >   struct lttng_metadata_cache *metadata_cache;
> >   ...
> > };
> > 
> > Which has the metadata string:
> > 
> > struct lttng_metadata_cache {
> >         char *data;                     /* Metadata cache */
> >         unsigned int cache_alloc;       /* Metadata allocated size (bytes)
> >         */
> >         unsigned int metadata_written;  /* Number of bytes written in
> >         metadata cache */
> >         struct kref refcount;           /* Metadata cache usage */
> >         struct list_head metadata_stream;       /* Metadata stream list */
> > };
> > 
> > You simply need to dump this metadata string into a "metadata" file, add a
> > "/* CTF 1.8 */\n line at the beginning, and this will generate a "text only
> > CTF metadata", which can be read by babeltrace.
> > 
> > If you happen to core dump while the metadata cache is being resized, you
> > might
> > encounter a corrupted metadata. It should not happen frequently, but it's
> > possible.
> > We could eventually change lttng_metadata_printf() so it makes sure to
> > always
> > keep a readable metadata around (by e.g. using kcalloc rather than krealloc
> > and
> > dealing carefully with cache_alloc vs data fields updates).
> 
> Oops, I meant "kzalloc".
> 
> I'm attaching a patch that should take care of making sure it is always in
> a valid state when a core dump occurs. You should be able to safely read
> "metadata_flushed" bytes of data at any time. Whatever has been flushed
> should be parseable by Babeltrace.

Updated patch (fixing poll).

Thanks,

Mathieu

> 
> Thoughts ?
> 
> Thanks,
> 
> Mathieu
> 
> > 
> > Thoughts ?
> > 
> > Thanks,
> > 
> > Mathieu
> > 
> > --
> > Mathieu Desnoyers
> > EfficiOS Inc.
> > http://www.efficios.com
> > 
> > _______________________________________________
> > lttng-dev mailing list
> > lttng-dev at lists.lttng.org
> > http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
> > 
> 
> --
> Mathieu Desnoyers
> EfficiOS Inc.
> http://www.efficios.com
> 
> _______________________________________________
> lttng-dev mailing list
> lttng-dev at lists.lttng.org
> http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
> 

-- 
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com
-------------- next part --------------
A non-text attachment was scrubbed...
Name: metadata-core-dump.patch
Type: text/x-patch
Size: 6421 bytes
Desc: not available
URL: <http://lists.lttng.org/pipermail/lttng-dev/attachments/20140218/1610ac23/attachment.bin>