[lttng-dev] Human read-writeable format for CTF traces

[gbastien+lttng@versatic.net (Geneviève Bastien)]

By the way, after discussing with Julien, one thing I need to add to 
understand our point of view: In TMF, we use a custom XML format with 
the existing custom XML parser to parse an XML trace, so it's really 
straightforward to use those traces, we don't need to create a new 
parser for it.

Genevi?ve

On 02/04/2014 12:46 PM, J?r?mie Galarneau wrote:
> On Tue, Feb 4, 2014 at 12:35 PM, Genevi?ve Bastien
> <gbastien+lttng at versatic.net> wrote:
>> Hi Jeremie,
>>
>> Thanks for your reply.
>>
>>
>> On 02/04/2014 11:19 AM, J?r?mie Galarneau wrote:
>>> Hi Genevi?ve,
>>>
>>> I wonder if we are not overly complicating things here... The Python
>>> bindings seem to address your immediate testing concerns and will let
>>> you produce test traces with minimal effort.
>> Do you have any information, links, examples how to use what exists in
>> python to generate traces with minimal effort? Our very immediate need is
>> actually to generate traces manually, because we don't need to know how to
>> script in python to do this ;-) and we can manually modify the events. But
>> since I have no idea (yet) what those python bindings do and how they do it,
>> then maybe I miss something here.
>>
> They are part of the current Babeltrace master branch and will be
> included in the next release. An example script is available under
> babeltrace/bindings/python/examples/ctf_writer.py.
> Let me know if you have any questions.
>
> Regards,
> J?r?mie
>
>> Thanks,
>> Genevi?ve
>>
>>> While I agree that it doesn't solve the problem of testing analysis on
>>> Windows, I must ask if this really is a primary concern at this point.
>>>
>>> I would personally start by writing tests in Python using the
>>> CTF-Writer bindings and then, as the analysis feature gains traction,
>>> work with users to determine the best testing strategy. I somehow
>>> doubt that having an external dependancy on Babeltrace would really be
>>> a problem to these power users...
>>>
>>> Sorry for the late reply,
>>> J?r?mie
>>>
>>> On Mon, Feb 3, 2014 at 12:20 PM, Genevi?ve Bastien
>>> <gbastien+lttng at versatic.net> wrote:
>>>> On 02/03/2014 12:16 PM, Mathieu Desnoyers wrote:
>>>>> ----- Original Message -----
>>>>>> From: "Genevi?ve Bastien" <gbastien+lttng@versatic.net>
>>>>>> To: "Michel Dagenais" <michel.dagenais at polymtl.ca>, "Mathieu Desnoyers"
>>>>>> <mathieu.desnoyers at efficios.com>
>>>>>> Cc: lttng-dev at lists.lttng.org
>>>>>> Sent: Monday, February 3, 2014 12:05:38 PM
>>>>>> Subject: Re: [lttng-dev] Human read-writeable format for CTF traces
>>>>>>
>>>>>> Ok, I'll wait for J?r?mie's answer for more details. As I said, my
>>>>>> concern is to have something fully standalone in TMF. But if one has
>>>>>> access to babeltrace and eventual plugins to read-write a CTF trace to
>>>>>> XML, then all the better. We could then import an XML generated by a
>>>>>> python script into TMF, edit it there and then use it to test analyses.
>>>>>>
>>>>>> All we have to settle on is the intermediate format that should be
>>>>>> used.
>>>>>> I'd go for XML because of the possibility to validate it and have
>>>>>> visual
>>>>>> editors.
>>>>> Michel's idea of going for Python seems even better to generate test
>>>>> suites.
>>>>> It would allow importing and combining test "patterns" very easily, thus
>>>>> allowing to create tests by construction without having to copy-paste
>>>>> huge
>>>>> XML files.
>>>>>
>>>>> I don't clearly see why having external dependencies on other tools
>>>>> for a TMF CI test suite would be an issue. What would be the main
>>>>> arguments
>>>>> for having all those tests stand-alone in TMF for the test-suite ?
>>>> It is not just for test suite. XML-defined analysis will need test traces
>>>> as
>>>> well, and that is in main TMF, not in unit tests (one idea of the XML
>>>> analysis is to allow end-user to develop their own analysis without
>>>> writing
>>>> a single line of code or requiring the TMF development environment). And
>>>> the
>>>> user of these analysis and test traces may not have access to babeltrace
>>>> or
>>>> even to a Linux command line.
>>>>
>>>> Thanks,
>>>> Genevi?ve
>>>>
>>>>> Thanks,
>>>>>
>>>>> Mathieu
>>>>>
>>>>>> On 02/03/2014 11:00 AM, Michel Dagenais wrote:
>>>>>>>> I would expect that the ctf writer API recently added to babeltrace
>>>>>>>> (currently in master branch), along with the Python bindings that
>>>>>>>> cover
>>>>>>>> trace read and write APIs, should allow you to implement things like:
>>>>>>>>
>>>>>>>> - A plugin to read a CTF trace, and output it in an intermediate
>>>>>>>> format
>>>>>>>>       to facilitate edits (e.g. XML as you propose),
>>>>>>>> - A plugin to read this XML format and output a CTF trace.
>>>>>>> Yes, this would indeed be extremely helpful, in XML and/or JSON.
>>>>>>>
>>>>>>>> You could also generate the XML trace completely by hand if you like,
>>>>>>>> and
>>>>>>>> then convert it to CTF with the second plugin I'm relating to above.
>>>>>>> The likely scenario is to add a few events by hand.
>>>>>>>
>>>>>>>> Another possibility is that the XML description also allows
>>>>>>>> describing what the trace contains at a slightly higher level. For
>>>>>>>> instance, if you
>>>>>>>> have a periodic event happening for a certain amount of time, it
>>>>>>>> would
>>>>>>>> be described in XML, and then "generated" by the XML-to-CTF
>>>>>>>> converter.
>>>>>>> Do we want to describe this in XML or in Python? We could have "CTF"
>>>>>>> to
>>>>>>> "Python statements" generating XML. Then we could add loops by hand.
>>>>>>> We
>>>>>>> could also have CTF to XML, with hooks to merge Python generated
>>>>>>> events.
>>>>>> Indeed being able to script a trace would be extremely helpful and
>>>>>> convert it either directly to CTF or to the intermediate format. Some
>>>>>> scenarios for unit test would be to script a custom trace then change a
>>>>>> few events for the test purpose, then either import it in TMF or
>>>>>> convert
>>>>>> it to CTF.
>>>>>>
>>>>>> Thanks,
>>>>>> Genevi?ve
>>>>>>> In addition, TMF may also want to offer similar functionality, an XML
>>>>>>> dump
>>>>>>> of events and an XML events reader. Indeed, TMF supports a few formats
>>>>>>> other than CTF.
>>>>>>
>>>> _______________________________________________
>>>> lttng-dev mailing list
>>>> lttng-dev at lists.lttng.org
>>>> http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
>>>
>>>
>
>