[ltt-dev] [PATCH 0/3] In-kernel filter module

[zhaolei@cn.fujitsu.com (Zhaolei)]

Mathieu Desnoyers Wrote:
> * Zhaolei (zhaolei at cn.fujitsu.com) wrote:
>> Hello, Mathieu
>>
>> Sorry fot trouble you, how about my design of ltt-filter?
>> It no objection, I'd like to starting it.
>>
>> B.R.
>> Zhaolei
>>
>> * Zhaolei Wrote:
>>> * Mathieu Desnoyers wrote:
>>>> * Zhaolei (zhaolei at cn.fujitsu.com) wrote:
>>>>> * Mathieu Desnoyers Wrote:
>>>>>> * Zhaolei (zhaolei at cn.fujitsu.com) wrote:
>>>>>>> * Mathieu Desnoyers Wrote:
>>>>>>>>   - Create an in-kernel event filtering module which connects on
>>>>>>>>     LTTng.
>>>>>>>>
>>>>>>>> Well, I think this last item would be a generalization of the filtering
>>>>>>>> module I created for ext4 and jbd2 recently. The one I did provided
>>>>>>>> basic filtering for inode and device number.
>>>>>>> Hello, Matuieu
>>>>>>>
>>>>>>> I readed filter code of ext4 and jbd2 module,
>>>>>>> and I have some questions of generalization filter implementation.
>>>>>>>
>>>>>>>> The idea would be to extend this and to connect it as a filter callback
>>>>>>>> with 
>>>>>>>> ltt_module_register(LTT_FUNCTION_RUN_FILTER, filter_callback,
>>>>>>>>                     THIS_MODULE);
>>>>>>> Now ltt_module_register can only register one filter_callback.
>>>>>>> But I think when we make one filter for all types of tracepoint, it is hard to
>>>>>>> maintenance.
>>>>>>>
>>>>>> Not exactly. That would be one filter for all types of markers. And it
>>>>>> would iterate on the marker format string to filter by field name. So I
>>>>>> don't see where the maintenant burden is ?
>>>>>>
>>>>> Hello, Mathieu
>>>>>
>>>>> Thanks for your response.
>>>>>
>>>>> Do you means we only need one filter callback function in all lttng source,
>>>>> and this callback is used for every event in one trace?
>>>>>
>>>>> And programmer of ltt/probes/* don't need to consider about filter function
>>>>> as ext4 and jbd2?
>>>>>
>>>> Exactly.
>>>>
>>>> Mathieu
>>>>
>>> Hello, Mathieu
>>>
>>> I'm planing to make a in-kernel-filter for lttng.
>>> This is my image of function and schedule:
>>>
>>> Function:
>>>    Build config dir-struct in debugfs
>>>
>>>    Dir can dynamic updated when add/remove a module with matkers.
>>>
>>>    Data's name is a symlink to __argX, because some data is noname, or some
>>>    data's name is not filename, or some data have samename, for ex:
>>>      Normal: "name1 %d name2 %d" -> __arg1(name1) __arg2(name2)
>>>      No name: "%d %d" -> __arg1() __arg2()
>>>      Not filenane: "* %d" -> __arg1()
>>>      Same name: "name %d name %d" -> __arg1(name) __arg2()
>>>      
>>>    __arg0 is format string, to make user easy to read:
>>>    # echo [debugfs]/ltt/filter/allocate_blocks/__arg0:
>>>      dev %s block %llu flags %u len %u ino %lu logical %llu goal %llu lleft %llu lright %llu pleft %llu pright %llu
>>>
>>>    Dir struct is like this:
>>>    [debugfs]/ltt/filter
>>>        |-- allocate_blocks
>>>        |   |-- __arg0
>>>        |   |-- __arg1
>>>        |   |-- __arg2
>>>        |   |-- __arg3
>>>        |   |-- __arg4
>>>        |   |-- __arg5
>>>        |   |-- __arg6
>>>        |   |-- __arg7
>>>        |   |-- __arg8
>>>        |   |-- __arg9
>>>        |   |-- __arg10
>>>        |   |-- __arg11
>>>        |   |-- dev -> __arg1
>>>        |   |-- block -> __arg2
>>>        |   |-- flags -> __arg3
>>>        |   |-- len -> __arg4
>>>        |   |-- ino -> __arg5
>>>        |   |-- logical -> __arg6
>>>        |   |-- goal -> __arg7
>>>        |   |-- lleft -> __arg8
>>>        |   |-- lright -> __arg9
>>>        |   |-- pleft -> __arg10
>>>        |   `-- pright -> __arg11
>>>        |-- allocate_inode
>>>        |   |-- __arg0
>>>        |   |-- __arg1
>>>        |   |-- __arg2
>>>        |   |-- __arg3
>>>        |   |-- __arg4
>>>        |   |-- dev -> __arg1
>>>        |   |-- dir -> __arg3
>>>        |   |-- ino -> __arg2
>>>        |   `-- mode -> __arg4
>>>        |-- bio_backmerge
>>>        |   `-- ...
>>>        |-- bio_bounce
>>>        |   `-- ...
>>>        |-- bio_complete
>>>        |   `-- ...
>>>        |-- bio_frontmerge
>>>        |   `-- ...
>>>        `-- ...
>>>
>>>    User can config filter by write something to following debugfs files:
>>>    Ex:
>>>      echo "*" > [debugfs]/ltt/filter/allocate_blocks/len # enable all len
>>>      echo ">100" > [debugfs]/ltt/filter/allocate_blocks/len # enable len>100
>>>
>>> Schedule:
>>>   1: Make basic function work
>>>   2: Make support for per-trace-filter
>>>      Make dir struct from:
>>>      [debugfs]/ltt/filter/[marker_name]/[arg_name]
>>>      to
>>>      [debugfs]/ltt/filter/[trace]/[marker_name]/[arg_name]
>>>      Make and use callback when add/delete trace.
>>>      Update filter control dir when add/delete trace.
>>>   3: Otimize for speed(use hlist, rcu, ...)
>>>   4: (Maybe) Support echo "(>0 && <10) || (>50 && <100)" > len
>>>      and echo "eth*" > name
>>>
>>> Do you have some suggestion?
>>>
> 
> This proposal sounds great. I look forward to see the implementation. :)
> The patches from Lai I am currently merging (making marker id permanent
> for the whole trace session) will be useful to you when you try to
> express your filter data structures.
> 
> Mathieu
> 
Hello, Mathieu

I finished a in-kernel filter for lttng.
Its function and usage is same as description in this mail before.
Maybe it is useful for user to filter events to trace.

B.R.
Zhaolei

>>> B.R.
>>> Zhaolei
>>>
>>>>> B.R.
>>>>> Zhaolei
>>>>>
>>>>>>> Maybe it is better to make each filter for each type of event, for ex:
>>>>>>> ltt-ext4-filter for ext4, ltt-jbd2-filter for jbd2, ...
>>>>>>>
>>>>>>> We can select to add filter code into ltt/probes/XXX_trace.c, and call
>>>>>>> ltt_module_register on modile_init, or write filter code in a module alone.
>>>>>>> Each way is ok, but integrate filter code into ltt/probes/XXX_trace.c seems
>>>>>>> more readable.
>>>>>>>
>>>>>>> Another thing needs consider is call which filter on a event.
>>>>>>> 1: Call every filter for one event.
>>>>>>>    If one filter say "no", this event is ignore.
>>>>>>>    It is simple to write, and each filter can do every thing(flexible).
>>>>>>>    But this way is inefficient because it needs more cpu cycles.
>>>>>>> 2: Call only filter for current event type.
>>>>>>>    Filter register a event type on ltt_module_register, and only used to
>>>>>>>    process that type.
>>>>>>> I think 2 is better because it uses less CPU.
>>>>>>>
>>>>>> Yes, filters definitely have to be called only for their own event ID
>>>>>> for performance reasons.
>>>>>>
>>>>>>>> This module would be called by ltt_vtrace and _ltt_specialized_trace
>>>>>>>> with this test :
>>>>>>>>
>>>>>>>>                 if (unlikely(!ltt_run_filter(trace, eID)))
>>>>>>>>                         continue;
>>>>>>>>
>>>>>>>> for each active trace.
>>>>>>> I think call filter in tracepoint's callback as probe_jbd2_checkpoint is
>>>>>>> more efficient than this way.
>>>>>>> Consider that a event which is filtered(don't send to relay), more
>>>>>>> unnecessary process is done before ltt_vtrace if we call filter in ltt_vtrace.
>>>>>>>
>>>>>>> So, in generalization filter:
>>>>>>> 1: call filter in ltt_vtrace(_ltt_specialized_trace)
>>>>>>> 2: call filter in probe_XXX_checkpoint()
>>>>>>> I think 2 is better.
>>>>>>>
>>>>>>> Which is your opinion about this?
>>>>>>>
>>>>>> There is a feature of LTTng which would benefit of 1 : lttng can have
>>>>>> multiple trace sessions active at once. Therefore, if one trace session
>>>>>> needs a subset of events and another trace session need a different set,
>>>>>> then they could each have their own filter structure associated with
>>>>>> them. I think having this level of flexibility is more important than
>>>>>> the few cycles we could save by doing (2). Also, we already have the
>>>>>> tracepoints and markers to deactivate the event source at the kernel
>>>>>> level when we need to have nearly-zero overhead.
>>>>>>
>>>>>> Mathieu
>>>>>>
>>>>>>> B.R.
>>>>>>> Zhaolei
>>>>>>>
>>>>>>>> This filter would receive the trace information and the event ID.
>>>>>>>> We may have to add or change some parameters to this to support
>>>>>>>> filtering by fields. For the filter called from ltt_vtrace, passing :
>>>>>>>>
>>>>>>>> const struct marker *mdata
>>>>>>>> and
>>>>>>>> const char *fmt, va_list *args
>>>>>>>>
>>>>>>>> Should be more than enough to filter generically by
>>>>>>>> - channel name
>>>>>>>> - event name
>>>>>>>> - field name -> typed field data.
>>>>>>>>
>>>>>>>> Filtering should be pre-computed as much as possible and be O(1) when
>>>>>>>> executed. Creating callbacks for each expected data type to filter will
>>>>>>>> probably be requried. A technique similar to what we have done in lttv
>>>>>>>> filter.c should be considered.
>>>>>>>>
>>>>>>>> For specialized probes, it might be more difficult to do this
>>>>>>>> generically, because _ltt_specialized_trace has no knowledge of the
>>>>>>>> event fields. I guess we would have to create "specialized" filtering
>>>>>>>> callbacks for those custom trace points. It's their nature anyway.
>>>>>>>>
>>>>>>>> Please ask if anything is unclear. Comments/ideas are welcome.
>>>>>>>>
>>>>>>>> Mathieu
>>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> ltt-dev mailing list
>>>>>>> ltt-dev at lists.casi.polymtl.ca
>>>>>>> http://lists.casi.polymtl.ca/cgi-bin/mailman/listinfo/ltt-dev
>>>>>>>
>>>
>>>
>>> _______________________________________________
>>> ltt-dev mailing list
>>> ltt-dev at lists.casi.polymtl.ca
>>> http://lists.casi.polymtl.ca/cgi-bin/mailman/listinfo/ltt-dev
>>>
>>
>>
>> _______________________________________________
>> ltt-dev mailing list
>> ltt-dev at lists.casi.polymtl.ca
>> http://lists.casi.polymtl.ca/cgi-bin/mailman/listinfo/ltt-dev
>>
>