[ltt-dev] lttng development plan

[compudj@krystal.dyndns.org (Mathieu Desnoyers)]

* Zhaolei (zhaolei at cn.fujitsu.com) wrote:
> Hello, Mathieu
> 
> Sorry fot trouble you, how about my design of ltt-filter?
> It no objection, I'd like to starting it.
> 
> B.R.
> Zhaolei
> 
> * Zhaolei Wrote:
> > * Mathieu Desnoyers wrote:
> >> * Zhaolei (zhaolei at cn.fujitsu.com) wrote:
> >>> * Mathieu Desnoyers Wrote:
> >>>> * Zhaolei (zhaolei at cn.fujitsu.com) wrote:
> >>>>> * Mathieu Desnoyers Wrote:
> >>>>>>   - Create an in-kernel event filtering module which connects on
> >>>>>>     LTTng.
> >>>>>>
> >>>>>> Well, I think this last item would be a generalization of the filtering
> >>>>>> module I created for ext4 and jbd2 recently. The one I did provided
> >>>>>> basic filtering for inode and device number.
> >>>>> Hello, Matuieu
> >>>>>
> >>>>> I readed filter code of ext4 and jbd2 module,
> >>>>> and I have some questions of generalization filter implementation.
> >>>>>
> >>>>>> The idea would be to extend this and to connect it as a filter callback
> >>>>>> with 
> >>>>>> ltt_module_register(LTT_FUNCTION_RUN_FILTER, filter_callback,
> >>>>>>                     THIS_MODULE);
> >>>>> Now ltt_module_register can only register one filter_callback.
> >>>>> But I think when we make one filter for all types of tracepoint, it is hard to
> >>>>> maintenance.
> >>>>>
> >>>> Not exactly. That would be one filter for all types of markers. And it
> >>>> would iterate on the marker format string to filter by field name. So I
> >>>> don't see where the maintenant burden is ?
> >>>>
> >>> Hello, Mathieu
> >>>
> >>> Thanks for your response.
> >>>
> >>> Do you means we only need one filter callback function in all lttng source,
> >>> and this callback is used for every event in one trace?
> >>>
> >>> And programmer of ltt/probes/* don't need to consider about filter function
> >>> as ext4 and jbd2?
> >>>
> >> Exactly.
> >>
> >> Mathieu
> >>
> > Hello, Mathieu
> > 
> > I'm planing to make a in-kernel-filter for lttng.
> > This is my image of function and schedule:
> > 
> > Function:
> >    Build config dir-struct in debugfs
> > 
> >    Dir can dynamic updated when add/remove a module with matkers.
> > 
> >    Data's name is a symlink to __argX, because some data is noname, or some
> >    data's name is not filename, or some data have samename, for ex:
> >      Normal: "name1 %d name2 %d" -> __arg1(name1) __arg2(name2)
> >      No name: "%d %d" -> __arg1() __arg2()
> >      Not filenane: "* %d" -> __arg1()
> >      Same name: "name %d name %d" -> __arg1(name) __arg2()
> >      
> >    __arg0 is format string, to make user easy to read:
> >    # echo [debugfs]/ltt/filter/allocate_blocks/__arg0:
> >      dev %s block %llu flags %u len %u ino %lu logical %llu goal %llu lleft %llu lright %llu pleft %llu pright %llu
> > 
> >    Dir struct is like this:
> >    [debugfs]/ltt/filter
> >        |-- allocate_blocks
> >        |   |-- __arg0
> >        |   |-- __arg1
> >        |   |-- __arg2
> >        |   |-- __arg3
> >        |   |-- __arg4
> >        |   |-- __arg5
> >        |   |-- __arg6
> >        |   |-- __arg7
> >        |   |-- __arg8
> >        |   |-- __arg9
> >        |   |-- __arg10
> >        |   |-- __arg11
> >        |   |-- dev -> __arg1
> >        |   |-- block -> __arg2
> >        |   |-- flags -> __arg3
> >        |   |-- len -> __arg4
> >        |   |-- ino -> __arg5
> >        |   |-- logical -> __arg6
> >        |   |-- goal -> __arg7
> >        |   |-- lleft -> __arg8
> >        |   |-- lright -> __arg9
> >        |   |-- pleft -> __arg10
> >        |   `-- pright -> __arg11
> >        |-- allocate_inode
> >        |   |-- __arg0
> >        |   |-- __arg1
> >        |   |-- __arg2
> >        |   |-- __arg3
> >        |   |-- __arg4
> >        |   |-- dev -> __arg1
> >        |   |-- dir -> __arg3
> >        |   |-- ino -> __arg2
> >        |   `-- mode -> __arg4
> >        |-- bio_backmerge
> >        |   `-- ...
> >        |-- bio_bounce
> >        |   `-- ...
> >        |-- bio_complete
> >        |   `-- ...
> >        |-- bio_frontmerge
> >        |   `-- ...
> >        `-- ...
> > 
> >    User can config filter by write something to following debugfs files:
> >    Ex:
> >      echo "*" > [debugfs]/ltt/filter/allocate_blocks/len # enable all len
> >      echo ">100" > [debugfs]/ltt/filter/allocate_blocks/len # enable len>100
> > 
> > Schedule:
> >   1: Make basic function work
> >   2: Make support for per-trace-filter
> >      Make dir struct from:
> >      [debugfs]/ltt/filter/[marker_name]/[arg_name]
> >      to
> >      [debugfs]/ltt/filter/[trace]/[marker_name]/[arg_name]
> >      Make and use callback when add/delete trace.
> >      Update filter control dir when add/delete trace.
> >   3: Otimize for speed(use hlist, rcu, ...)
> >   4: (Maybe) Support echo "(>0 && <10) || (>50 && <100)" > len
> >      and echo "eth*" > name
> > 
> > Do you have some suggestion?
> > 

This proposal sounds great. I look forward to see the implementation. :)
The patches from Lai I am currently merging (making marker id permanent
for the whole trace session) will be useful to you when you try to
express your filter data structures.

Mathieu

> > B.R.
> > Zhaolei
> > 
> >>> B.R.
> >>> Zhaolei
> >>>
> >>>>> Maybe it is better to make each filter for each type of event, for ex:
> >>>>> ltt-ext4-filter for ext4, ltt-jbd2-filter for jbd2, ...
> >>>>>
> >>>>> We can select to add filter code into ltt/probes/XXX_trace.c, and call
> >>>>> ltt_module_register on modile_init, or write filter code in a module alone.
> >>>>> Each way is ok, but integrate filter code into ltt/probes/XXX_trace.c seems
> >>>>> more readable.
> >>>>>
> >>>>> Another thing needs consider is call which filter on a event.
> >>>>> 1: Call every filter for one event.
> >>>>>    If one filter say "no", this event is ignore.
> >>>>>    It is simple to write, and each filter can do every thing(flexible).
> >>>>>    But this way is inefficient because it needs more cpu cycles.
> >>>>> 2: Call only filter for current event type.
> >>>>>    Filter register a event type on ltt_module_register, and only used to
> >>>>>    process that type.
> >>>>> I think 2 is better because it uses less CPU.
> >>>>>
> >>>> Yes, filters definitely have to be called only for their own event ID
> >>>> for performance reasons.
> >>>>
> >>>>>> This module would be called by ltt_vtrace and _ltt_specialized_trace
> >>>>>> with this test :
> >>>>>>
> >>>>>>                 if (unlikely(!ltt_run_filter(trace, eID)))
> >>>>>>                         continue;
> >>>>>>
> >>>>>> for each active trace.
> >>>>> I think call filter in tracepoint's callback as probe_jbd2_checkpoint is
> >>>>> more efficient than this way.
> >>>>> Consider that a event which is filtered(don't send to relay), more
> >>>>> unnecessary process is done before ltt_vtrace if we call filter in ltt_vtrace.
> >>>>>
> >>>>> So, in generalization filter:
> >>>>> 1: call filter in ltt_vtrace(_ltt_specialized_trace)
> >>>>> 2: call filter in probe_XXX_checkpoint()
> >>>>> I think 2 is better.
> >>>>>
> >>>>> Which is your opinion about this?
> >>>>>
> >>>> There is a feature of LTTng which would benefit of 1 : lttng can have
> >>>> multiple trace sessions active at once. Therefore, if one trace session
> >>>> needs a subset of events and another trace session need a different set,
> >>>> then they could each have their own filter structure associated with
> >>>> them. I think having this level of flexibility is more important than
> >>>> the few cycles we could save by doing (2). Also, we already have the
> >>>> tracepoints and markers to deactivate the event source at the kernel
> >>>> level when we need to have nearly-zero overhead.
> >>>>
> >>>> Mathieu
> >>>>
> >>>>> B.R.
> >>>>> Zhaolei
> >>>>>
> >>>>>> This filter would receive the trace information and the event ID.
> >>>>>> We may have to add or change some parameters to this to support
> >>>>>> filtering by fields. For the filter called from ltt_vtrace, passing :
> >>>>>>
> >>>>>> const struct marker *mdata
> >>>>>> and
> >>>>>> const char *fmt, va_list *args
> >>>>>>
> >>>>>> Should be more than enough to filter generically by
> >>>>>> - channel name
> >>>>>> - event name
> >>>>>> - field name -> typed field data.
> >>>>>>
> >>>>>> Filtering should be pre-computed as much as possible and be O(1) when
> >>>>>> executed. Creating callbacks for each expected data type to filter will
> >>>>>> probably be requried. A technique similar to what we have done in lttv
> >>>>>> filter.c should be considered.
> >>>>>>
> >>>>>> For specialized probes, it might be more difficult to do this
> >>>>>> generically, because _ltt_specialized_trace has no knowledge of the
> >>>>>> event fields. I guess we would have to create "specialized" filtering
> >>>>>> callbacks for those custom trace points. It's their nature anyway.
> >>>>>>
> >>>>>> Please ask if anything is unclear. Comments/ideas are welcome.
> >>>>>>
> >>>>>> Mathieu
> >>>>>>
> >>>>> _______________________________________________
> >>>>> ltt-dev mailing list
> >>>>> ltt-dev at lists.casi.polymtl.ca
> >>>>> http://lists.casi.polymtl.ca/cgi-bin/mailman/listinfo/ltt-dev
> >>>>>
> > 
> > 
> > 
> > _______________________________________________
> > ltt-dev mailing list
> > ltt-dev at lists.casi.polymtl.ca
> > http://lists.casi.polymtl.ca/cgi-bin/mailman/listinfo/ltt-dev
> > 
> 
> 
> 
> _______________________________________________
> ltt-dev mailing list
> ltt-dev at lists.casi.polymtl.ca
> http://lists.casi.polymtl.ca/cgi-bin/mailman/listinfo/ltt-dev
> 

-- 
Mathieu Desnoyers
OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68