[ltt-dev] Babel Trace

[mathieu.desnoyers@efficios.com (Mathieu Desnoyers)]

* Mathieu Desnoyers (mathieu.desnoyers at efficios.com) wrote:
> * Klein, Russell (russell_klein at mentor.com) wrote:
> > Mathieu,
> > 
> > I have found a need to use a very old version of LTTng, which produces
> > a trace file format of 1.0.  Do you think it would be practical to use
> > Babel trace to convert the file from 1.0 format to 2.6 for analysis
> > and viewing?  Would you have any advice on how to approach this?
> 
> Hi Russell,
> 
> I'm taking the liberty to CC ltt-dev at lists.casi.polymtl.ca as others
> will benefit from this explanation:
> 
> Sure, babeltrace could handle this. Not sure which "1.0" version you
> refer to (probably a very very old LTTng) ;-). There would seem to be
> more interest in converting from LTTng 0.x trace format 2.6 to the new
> CTF, or in your case convert from the very old LTTng trace format to
> CTF, rather than from the (very old) LTTng 1.0 to the (old) LTTng 0.x
> trace format 2.6. One of the major advantage of converting to CTF is
> that you only need to convert the metadata, not the whole trace streams.
> 
> Considering conversion to CTF, the way I would recommend handling this
> task would be:
> 
> 1) look at lttv ltt/marker.c parse_c_type() and parse_trace_type():
> these parse the marker metadata.
> 
> 2) You'll need to create a babeltrace input-side plugin, e.g.
> "lttng-legacy-0.x", that takes the LTTng traces as input, and converts
> the marker metadata into CTF metadata format. Have a look at babeltrace
> converter/babeltrace-log.c to see an example of how to convert from a
> format (here dmesg) to CTF. In your case though, you'll only need to
> convert the LTTng 0.x "metadata_NN" files into a CTF "metadata" file (a
> single file). The plugin "lttng-legacy-0.x" could detect the LTTng
> major/minor trace version numbers in the input trace and handle
> 1.0/2.5/2.6 formats appropriately within the same plugin.
> 
> I recommend you do not even convert the LTTng 0.x trace streams per se:
> you just need to create the new "metadata" file and delete the old ones
> (metadata_NN). However, you'll have to generate CTF metadata that
> describes the LTTng 0.x packet header (functionally equivalent to the
> lttng 0.x subbuffer header) and event header, which might both differ
> slightly from the LTTng 2.0 headers in terms of layout. You'll have to
> add description of these headers in the metadata file you create.

Some extra info that might help you start: using lttng-modules git head,
with lttng-tools, and babeltrace -d -v tracepath, you can extract the
text metadata exported by lttng 2.0 into its packet-based metadata file.
The metadata describing older lttng versions will be very similar, with
small changes in the header layout. An example of lttng 2.0 metadata
taken on my machine follows. Note that you can simply create a
text-based metadata file (the packet-based metadata is only useful for
convenient transport from the tracer).

typealias integer { size = 8; align = 8; signed = false; } := uint8_t;
typealias integer { size = 16; align = 8; signed = false; } := uint16_t;
typealias integer { size = 32; align = 8; signed = false; } := uint32_t;
typealias integer { size = 64; align = 8; signed = false; } := uint64_t;
typealias integer { size = 5; align = 1; signed = false; } := uint5_t;
typealias integer { size = 27; align = 1; signed = false; } := uint27_t;

trace {
        major = 0;
        minor = 1;
        uuid = "cd783712-adf2-034c-ab4b-5846380dc229";
        byte_order = le;
        packet.header := struct {
                uint32_t magic;
                uint8_t  uuid[16];
                uint32_t stream_id;
        };
};

struct packet_context {
        uint64_t timestamp_begin;
        uint64_t timestamp_end;
        uint32_t events_discarded;
        uint32_t content_size;
        uint32_t packet_size;
        uint32_t cpu_id;
};

struct event_header_compact {
        enum : uint5_t { compact = 0 ... 30, extended = 31 } id;
        variant <id> {
                struct {
                        uint27_t timestamp;
                } compact;
                struct {
                        uint32_t id;
                        uint64_t timestamp;
                } extended;
        } v;
} align(8);

struct event_header_large {
        enum : uint16_t { compact = 0 ... 65534, extended = 65535 } id;
        variant <id> {
                struct {
                        uint32_t timestamp;
                } compact;
                struct {
                        uint32_t id;
                        uint64_t timestamp;
                } extended;
        } v;
} align(8);

stream {
        id = 1;
        event.header := struct event_header_compact;
        packet.context := struct packet_context;
        event.context := struct {
                integer { size = 32; align = 8; signed = 1; encoding = none; base = 10; } pid;
                integer { size = 8; align = 8; signed = 1; encoding = UTF8; base = 10; } comm[16];
                integer { size = 32; align = 8; signed = 1; encoding = none; base = 10; } prio;
                integer { size = 32; align = 8; signed = 1; encoding = none; base = 10; } nice;
                integer { size = 64; align = 8; signed = 0; encoding = none; base = 10; } branches;
        };
};

event {
        name = sys_close_ftrace;
        id = 7;
        stream_id = 1;
        fields := struct {
                integer { size = 64; align = 8; signed = 0; encoding = none; base = 16; } ip;
                integer { size = 64; align = 8; signed = 0; encoding = none; base = 16; } parent_ip;
        };
};

event {
        name = sys_open_kprobe;
        id = 6;
        stream_id = 1;
        fields := struct {
                integer { size = 64; align = 8; signed = 0; encoding = none; base = 16; } ip;
        };
};

event {
        name = sys_exit;
        id = 5;
        stream_id = 1;
        fields := struct {
                integer { size = 64; align = 8; signed = 1; encoding = none; base = 10; } id;
                integer { size = 64; align = 8; signed = 1; encoding = none; base = 10; } ret;
        };
};

event {
        name = sys_enter;
        id = 4;
        stream_id = 1;
        fields := struct {
                integer { size = 64; align = 8; signed = 1; encoding = none; base = 10; } id;
                integer { size = 64; align = 8; signed = 0; encoding = none; base = 10; } args[6];
        };
};

event {
        name = sched_stat_wait;
        id = 3;
        stream_id = 1;
        fields := struct {
                integer { size = 8; align = 8; signed = 1; encoding = UTF8; base = 10; } comm[16];
                integer { size = 32; align = 8; signed = 1; encoding = none; base = 10; } pid;
                integer { size = 64; align = 8; signed = 0; encoding = none; base = 10; } delay;
        };
};

event {
        name = sched_stat_runtime;
        id = 2;
        stream_id = 1;
        fields := struct {
                integer { size = 8; align = 8; signed = 1; encoding = UTF8; base = 10; } comm[16];
                integer { size = 32; align = 8; signed = 1; encoding = none; base = 10; } pid;
                integer { size = 64; align = 8; signed = 0; encoding = none; base = 10; } runtime;
                integer { size = 64; align = 8; signed = 0; encoding = none; base = 10; } vruntime;
        };
};

event {
        name = sched_kthread_stop;
        id = 1;
        stream_id = 1;
        fields := struct {
                integer { size = 8; align = 8; signed = 1; encoding = UTF8; base = 10; } comm[16];
                integer { size = 32; align = 8; signed = 1; encoding = none; base = 10; } pid;
        };
};

event {
        name = sched_switch;
        id = 0;
        stream_id = 1;
        fields := struct {
                integer { size = 8; align = 8; signed = 1; encoding = UTF8; base = 10; } prev_comm[16];
                integer { size = 32; align = 8; signed = 1; encoding = none; base = 10; } prev_pid;
                integer { size = 32; align = 8; signed = 1; encoding = none; base = 10; } prev_prio;
                integer { size = 64; align = 8; signed = 1; encoding = none; base = 10; } prev_state;
                integer { size = 8; align = 8; signed = 1; encoding = UTF8; base = 10; } next_comm[16];
                integer { size = 32; align = 8; signed = 1; encoding = none; base = 10; } next_pid;
                integer { size = 32; align = 8; signed = 1; encoding = none; base = 10; } next_prio;
        };
};

Best regards,

Mathieu


> 
> Given that the plugin you'll create will be derived from LTTV marker.c,
> licensed under LGPLv2.1, it would seem appropriate to keep this license
> for the plugin. I would very much welcome this contribution into the
> babeltrace source tree.
> 
> Thanks,
> 
> Mathieu
> 
> -- 
> Mathieu Desnoyers
> Operating System Efficiency R&D Consultant
> EfficiOS Inc.
> http://www.efficios.com

-- 
Mathieu Desnoyers
Operating System Efficiency R&D Consultant
EfficiOS Inc.
http://www.efficios.com