[ltt-dev] sub roadmap for text output

[ltt-dev] sub roadmap for text output

[Lai Jiangshan]

1) Marker ID management
   Marker ID identify every type of event,
   a) ASCII text output needs get name&fmt from eID faster.
   b) Marker ID and its data must be permanent.

2) ltt-relay events traveling
   current ltt-relay does not support iterator. it seems that ltt-relay is too
   complex to do this things. Is any plan to revise ltt-relay?

3) event reader
   get significative binary data from event. and convert them to a va_list
   or other type. binary in event is raw data, we must know its meaning for
   formating.

4) event text formator
   format data to text.

5) create files for text output and control

In all, I think text output is urgency for lttng, but
"ltt-relay events traveling" is the bottleneck.

Lai.

[ltt-dev] sub roadmap for text output

[Mathieu Desnoyers]

* Lai Jiangshan (laijs at cn.fujitsu.com) wrote:
> 
> 1) Marker ID management
>    Marker ID identify every type of event,

Beware that I am currently doing a big refactoring of LTTng which will
change the following :

- Channels will be declared in markers, e.g. :

  trace_mark(channel_name, event_name, "format", ...);

Those channels are dynamically allocated when the trace is "setup".

Event IDs are dynamically assigned by the marker/channel infrastructure.
Those are unique per-channel.

I am currently in the testing phase. Tell me if you are interested to do
an early review of a prerelease of LTTng (the patchset does not
currently bissect in the last changes and may still contain a few bugs).

>    a) ASCII text output needs get name&fmt from eID faster.

We'll probably have to add the following to a new marker-to-string.c :

const struct marker_to_string *
markers_get_marker_from_id(u16 channel_id, u16 event_id);

static struct kref trace_refcount;

That means we have to keep around an hash table of 

struct marker_to_string {
  u16 channel_id;
  u16 event_id;
  const char *channel;
  const char *name;
  struct kref refcount;
};

around. 

>    b) Marker ID and its data must be permanent.
> 

It does not need to be exactly "permanent". These entries must stay live
until those two conditions occurs :
- The given marker is not referencing the entry anymore.
- All traces are inactive (destroyed)

The new ltt/ltt-channel.c file I am currently working on uses a similar
semantic to deal with channel persistency, so derivating from this will
be easier.


> 2) ltt-relay events traveling
>    current ltt-relay does not support iterator. it seems that ltt-relay is too
>    complex to do this things. Is any plan to revise ltt-relay?
> 

ltt-relay task is to write data into buffers. I would create a different
module which task is to do just like lttd does (get subbuf, read, put
subbuf), but within the kernel. We might have to extend the API beyond
the current ioctl/poll-based interface to allow in-kernel modules to do
this.

Side-note : I think I would need to hook into open()/close() to keep
refcounts on the channel reads rather that to use the get subbuf/put
subbuf for refcounting. The currently implementation will behave a bit
weirdly if multiple processes try to hook on the same trace channels.
The resulting trace outputs will only each have part of the subbuffers.

> 3) event reader
>    get significative binary data from event. and convert them to a va_list
>    or other type. binary in event is raw data, we must know its meaning for
>    formating.
> 

The event reader is fairly easy to do. You fork from
ltt/ltt-serialize.c, change it so it does binary to ascii conversion.
It takes the format string as input. That's about it.

> 4) event text formator
>    format data to text.

I am not sure how it differs from 3 ?

> 
> 5) create files for text output and control
> 

That will go on top of Zhaolei debugfs control implementation.

> In all, I think text output is urgency for lttng, but
> "ltt-relay events traveling" is the bottleneck.
> 

Please tell me if you need more information or to discuss some elements
more.

Best regards,

Mathieu

> Lai.
> 
> 
> _______________________________________________
> ltt-dev mailing list
> ltt-dev at lists.casi.polymtl.ca
> http://lists.casi.polymtl.ca/cgi-bin/mailman/listinfo/ltt-dev
> 

-- 
Mathieu Desnoyers
OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68

[ltt-dev] sub roadmap for text output

[Lai Jiangshan]

Mathieu Desnoyers wrote:
> 
>> 2) ltt-relay events traveling
>>    current ltt-relay does not support iterator. it seems that ltt-relay is too
>>    complex to do this things. Is any plan to revise ltt-relay?
>>
> 
> ltt-relay task is to write data into buffers. I would create a different
> module which task is to do just like lttd does (get subbuf, read, put
> subbuf), but within the kernel. We might have to extend the API beyond
> the current ioctl/poll-based interface to allow in-kernel modules to do
> this.
> 
> Side-note : I think I would need to hook into open()/close() to keep
> refcounts on the channel reads rather that to use the get subbuf/put
> subbuf for refcounting. The currently implementation will behave a bit
> weirdly if multiple processes try to hook on the same trace channels.
> The resulting trace outputs will only each have part of the subbuffers.
> 

Very shocked!
I thought we should iterate very events directly in kernel, and merge them.
What you do is moving a user-space tool to kernel.

Any way, I will use these new APIs when they are applied.

>> 3) event reader
>>    get significative binary data from event. and convert them to a va_list
>>    or other type. binary in event is raw data, we must know its meaning for
>>    formating.
>>
> 
> The event reader is fairly easy to do. You fork from
> ltt/ltt-serialize.c, change it so it does binary to ascii conversion.
> It takes the format string as input. That's about it.
> 
>> 4) event text formator
>>    format data to text.
> 
> I am not sure how it differs from 3 ?

I think getting data and using data are two phases.

In Lttv
marker.c is the phase 1: event reader.
print.c is the phase 2: event text formator.

By the way: my patch for ftrace is also two phases.

> 
>> 5) create files for text output and control
>>
> 
> That will go on top of Zhaolei debugfs control implementation.
> 
>> In all, I think text output is urgency for lttng, but
>> "ltt-relay events traveling" is the bottleneck.
>>
> 
> Please tell me if you need more information or to discuss some elements
> more.
> 
> Best regards,
> 

[ltt-dev] sub roadmap for text output

[Lai Jiangshan]

Lai Jiangshan wrote:

> 2) ltt-relay events traveling
>    current ltt-relay does not support iterator. it seems that ltt-relay is too
>    complex to do this things. Is any plan to revise ltt-relay?
> 

Hi, Mathieu Desnoyers

	Merry Christmas!

	I still don't know how to deal with corrupted data which is brought when
subbuffer is pushed by the writer in the kernel. Any idea?
The iterator is hard to be designed congruously with reserver side.

	Thanks.
Lai.

[ltt-dev] sub roadmap for text output

[Mathieu Desnoyers]

* Lai Jiangshan (laijs at cn.fujitsu.com) wrote:
> Lai Jiangshan wrote:
> 
> > 2) ltt-relay events traveling
> >    current ltt-relay does not support iterator. it seems that ltt-relay is too
> >    complex to do this things. Is any plan to revise ltt-relay?
> > 
> 
> Hi, Mathieu Desnoyers
> 
> 	Merry Christmas!
> 
> 	I still don't know how to deal with corrupted data which is brought when
> subbuffer is pushed by the writer in the kernel. Any idea?
> The iterator is hard to be designed congruously with reserver side.
> 

Merry Christmas Lai,

Here is how to deal with it :

- Allow iterators only in the following cases :
  - non-overwrite channel
  - writing is stopped (paused) in overwrite channel

- _Never_ allow iterators on overwrite channels with non-paused writers.
  That's the only scenario which can corrupt a channel due to a writer
  push.

Best regards,

Mathieu

P.S. I'll be on vacation and probably unreachable until January 2nd.


> 	Thanks.
> Lai.
> 
> 
> _______________________________________________
> ltt-dev mailing list
> ltt-dev at lists.casi.polymtl.ca
> http://lists.casi.polymtl.ca/cgi-bin/mailman/listinfo/ltt-dev
> 

-- 
Mathieu Desnoyers
OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68