[ltt-dev] [Fwd: Re: sub roadmap for text output]

[ltt-dev] [Fwd: Re: sub roadmap for text output]

[Mathieu Desnoyers]

(the Fujitsu mail servers still seems to have problems sending to my
krystal.dyndns.org address. I just updated the ltt-dev mailing list so
it forward the mails to my account even if I am a recipient of the given
mail)

Lai wrote:
> Mathieu Desnoyers wrote:
> > 
> >> 2) ltt-relay events traveling
> >>    current ltt-relay does not support iterator. it seems that ltt-relay is too
> >>    complex to do this things. Is any plan to revise ltt-relay?
> >>
> > 
> > ltt-relay task is to write data into buffers. I would create a different
> > module which task is to do just like lttd does (get subbuf, read, put
> > subbuf), but within the kernel. We might have to extend the API beyond
> > the current ioctl/poll-based interface to allow in-kernel modules to do
> > this.
> > 
> > Side-note : I think I would need to hook into open()/close() to keep
> > refcounts on the channel reads rather that to use the get subbuf/put
> > subbuf for refcounting. The currently implementation will behave a bit
> > weirdly if multiple processes try to hook on the same trace channels.
> > The resulting trace outputs will only each have part of the subbuffers.
> > 
> 
> Very shocked!
> I thought we should iterate very events directly in kernel, and merge them.
> What you do is moving a user-space tool to kernel.
> 

Hrm, I think I have not been clear enough then. Sorry. My plan is to
iterate on every events directly in kernel. Those events will be stored
in ltt-relay buffers. We will _not_ use the poll() and ioctl() methods.
Instead, we will create new in-kernel APIs which will act _similarly_ to
those primitives. But everything will be in-kernel.

Those two different methods of accessing the buffers would be called
"data consumers".

(current) method 1:
- poll, ioctl, splice to export the data

(new) text output consumer :
- api to get events from a trace channel, merging the events by
  increasing timestamp
- exports the data to a userspace seq file (single stream per trace)

> Any way, I will use these new APIs when they are applied.
> 
> >> 3) event reader
> >>    get significative binary data from event. and convert them to a va_list
> >>    or other type. binary in event is raw data, we must know its meaning for
> >>    formating.
> >>
> > 
> > The event reader is fairly easy to do. You fork from
> > ltt/ltt-serialize.c, change it so it does binary to ascii conversion.
> > It takes the format string as input. That's about it.
> > 
> >> 4) event text formator
> >>    format data to text.
> > 
> > I am not sure how it differs from 3 ?
> 
> I think getting data and using data are two phases.
> 
> In Lttv
> marker.c is the phase 1: event reader.
> print.c is the phase 2: event text formator.
> 
> By the way: my patch for ftrace is also two phases.
> 

Agreed.

Does my explanation above clarify things ?

Mathieu

> > 
> >> 5) create files for text output and control
> >>
> > 
> > That will go on top of Zhaolei debugfs control implementation.
> > 
> >> In all, I think text output is urgency for lttng, but
> >> "ltt-relay events traveling" is the bottleneck.
> >>
> > 
> > Please tell me if you need more information or to discuss some elements
> > more.
> > 
> > Best regards,
> 

-- 
Mathieu Desnoyers
OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68

[ltt-dev] [Fwd: Re: sub roadmap for text output]

[Lai Jiangshan]

Mathieu Desnoyers wrote:
> (the Fujitsu mail servers still seems to have problems sending to my
> krystal.dyndns.org address. I just updated the ltt-dev mailing list so
> it forward the mails to my account even if I am a recipient of the given
> mail)
> 
> Lai wrote:
>> Mathieu Desnoyers wrote:
>>>> 2) ltt-relay events traveling
>>>>    current ltt-relay does not support iterator. it seems that ltt-relay is too
>>>>    complex to do this things. Is any plan to revise ltt-relay?
>>>>
>>> ltt-relay task is to write data into buffers. I would create a different
>>> module which task is to do just like lttd does (get subbuf, read, put
>>> subbuf), but within the kernel. We might have to extend the API beyond
>>> the current ioctl/poll-based interface to allow in-kernel modules to do
>>> this.
>>>
>>> Side-note : I think I would need to hook into open()/close() to keep
>>> refcounts on the channel reads rather that to use the get subbuf/put
>>> subbuf for refcounting. The currently implementation will behave a bit
>>> weirdly if multiple processes try to hook on the same trace channels.
>>> The resulting trace outputs will only each have part of the subbuffers.
>>>
>> Very shocked!
>> I thought we should iterate very events directly in kernel, and merge them.
>> What you do is moving a user-space tool to kernel.
>>
> 
> Hrm, I think I have not been clear enough then. Sorry. My plan is to
> iterate on every events directly in kernel. Those events will be stored
> in ltt-relay buffers. We will _not_ use the poll() and ioctl() methods.
> Instead, we will create new in-kernel APIs which will act _similarly_ to
> those primitives. But everything will be in-kernel.
> 
> Those two different methods of accessing the buffers would be called
> "data consumers".
> 
> (current) method 1:
> - poll, ioctl, splice to export the data
> 
> (new) text output consumer :
> - api to get events from a trace channel, merging the events by
>   increasing timestamp
> - exports the data to a userspace seq file (single stream per trace)

I understood. I had said, we need revise ltt-relay a lot:
>>>> 2) ltt-relay events traveling
>>>>    current ltt-relay does not support iterator. it seems that ltt-relay is too
>>>>    complex to do this things. Is any plan to revise ltt-relay?
But ltt-relay is so complicated that I can't write iterator for it.

> 
>> Any way, I will use these new APIs when they are applied.
>>
>>>> 3) event reader
>>>>    get significative binary data from event. and convert them to a va_list
>>>>    or other type. binary in event is raw data, we must know its meaning for
>>>>    formating.
>>>>
>>> The event reader is fairly easy to do. You fork from
>>> ltt/ltt-serialize.c, change it so it does binary to ascii conversion.
>>> It takes the format string as input. That's about it.
>>>
>>>> 4) event text formator
>>>>    format data to text.
>>> I am not sure how it differs from 3 ?
>> I think getting data and using data are two phases.
>>
>> In Lttv
>> marker.c is the phase 1: event reader.
>> print.c is the phase 2: event text formator.
>>
>> By the way: my patch for ftrace is also two phases.
>>
> 
> Agreed.
> 
> Does my explanation above clarify things ?
> 
> Mathieu
> 
>>>> 5) create files for text output and control
>>>>
>>> That will go on top of Zhaolei debugfs control implementation.
>>>
>>>> In all, I think text output is urgency for lttng, but
>>>> "ltt-relay events traveling" is the bottleneck.
>>>>
>>> Please tell me if you need more information or to discuss some elements
>>> more.
>>>
>>> Best regards,
> 

[ltt-dev] [Fwd: Re: sub roadmap for text output]

[Mathieu Desnoyers]

* Lai Jiangshan (laijs at cn.fujitsu.com) wrote:
> Mathieu Desnoyers wrote:
[...]
> > Lai wrote:
> >> Mathieu Desnoyers wrote:
> >>>> 2) ltt-relay events traveling
> >>>>    current ltt-relay does not support iterator. it seems that ltt-relay is too
> >>>>    complex to do this things. Is any plan to revise ltt-relay?
> >>>>
> >>> ltt-relay task is to write data into buffers. I would create a different
> >>> module which task is to do just like lttd does (get subbuf, read, put
> >>> subbuf), but within the kernel. We might have to extend the API beyond
> >>> the current ioctl/poll-based interface to allow in-kernel modules to do
> >>> this.
> >>>
> >>> Side-note : I think I would need to hook into open()/close() to keep
> >>> refcounts on the channel reads rather that to use the get subbuf/put
> >>> subbuf for refcounting. The currently implementation will behave a bit
> >>> weirdly if multiple processes try to hook on the same trace channels.
> >>> The resulting trace outputs will only each have part of the subbuffers.
> >>>
> >> Very shocked!
> >> I thought we should iterate very events directly in kernel, and merge them.
> >> What you do is moving a user-space tool to kernel.
> >>
> > 
> > Hrm, I think I have not been clear enough then. Sorry. My plan is to
> > iterate on every events directly in kernel. Those events will be stored
> > in ltt-relay buffers. We will _not_ use the poll() and ioctl() methods.
> > Instead, we will create new in-kernel APIs which will act _similarly_ to
> > those primitives. But everything will be in-kernel.
> > 
> > Those two different methods of accessing the buffers would be called
> > "data consumers".
> > 
> > (current) method 1:
> > - poll, ioctl, splice to export the data
> > 
> > (new) text output consumer :
> > - api to get events from a trace channel, merging the events by
> >   increasing timestamp
> > - exports the data to a userspace seq file (single stream per trace)
> 
> I understood. I had said, we need revise ltt-relay a lot:
> >>>> 2) ltt-relay events traveling
> >>>>    current ltt-relay does not support iterator. it seems that ltt-relay is too
> >>>>    complex to do this things. Is any plan to revise ltt-relay?
> But ltt-relay is so complicated that I can't write iterator for it.
> 

That's why I suggested that we first look at how ltt_ioctl and ltt_poll
works, because this is about the only thing we really need to understand
to create iterators. Because an iterator can be seen as a data
"consumer", and all the code that is responsible for synchronizing the
data consumption sits in ltt_ioctl and ltt_poll.

Mathieu

-- 
Mathieu Desnoyers
OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68