[lttng-dev] Machine Interface

[lttng-dev] Machine Interface

[Olivier Cotte]

At first, we wanted thank you for your quick and constructive feedback!

After reflection on the nature of the problem and consideration of your feedback, we've arrived with the following solution.

The technology that will be use is XML. The integration of the machine interface to LTTng will be done in the most modular manner to facilitate the support multiples outputs format. This option will most likely be available at configure time.

For the moment, we will focus our effort into adding the XML feature only. If many outputs languages is still a viable feature, it will be possible to investigate.

Our main goal is of course to promote extensibility, ease of development for new tools and establish a standards for the communication control flow. Here are our primary requirement concerning the machine interface:

   * Fit the tree structure of LTTng commands structures.
   * It can be parse from bash/shell script with no external helpers.
   * The new format must not introduce any dependencies to LTTng.
   * [Optional] Still human readable, but fit to be parse.This is not a
      priority since we are talking about a machine interface.

Here are the reason why we think XML should be used for the machine interface:

   * The expression of LTTng hierarchy really fit a XML tree structure.
   * Since Jgalar add a load-save feature that depend on libxml2, we are
     not introducing any dependencies. Choosing another format would
     most likely introduce new dependency.
   * XML is robust and easily extensible. It will also be easy to
     maintains (add new commands, modify existing commands etc.)
   * There are tools that allow you to evaluate XPath expressions from
     the bash.
   * XML is an established standard

That's why we consider XML format as a suitable solution.

Now for the machine interface syntax:

   * Since we are aware that multiple output could be supported, we want an
     unify flag for all machine interface command.
		The flag --xml would be used to output xml from LTTng (we could add similar
		options like --json, --yaml etc.). This would be similar to svn
		machine interface and speak for itself.
   * Also, to avoid confusion we will use a different primary command for input and
     output machine interface.
		Ex: lttng machine <input>
		Ex: lttng --xml <output>

Any points we are missing?

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lttng.org/pipermail/lttng-dev/attachments/20140213/694aa217/attachment-0001.html>

[lttng-dev] Machine Interface

[Simon Marchi]

It looks good!

Comments below.

On 13 February 2014 13:00, Olivier Cotte <olivier.cotte at polymtl.ca> wrote:
> At first, we wanted thank you for your quick and constructive feedback!
>
> After reflection on the nature of the problem and consideration of your
> feedback, we've arrived with the following solution.
>
> The technology that will be use is XML. The integration of the machine
> interface to LTTng will be done in the most modular manner to facilitate the
> support multiples outputs format. This option will most likely be available
> at configure time.
>
> For the moment, we will focus our effort into adding the XML feature only.
> If many outputs languages is still a viable feature, it will be possible to
> investigate.
>
> Our main goal is of course to promote extensibility, ease of development for
> new tools and establish a standards for the communication control flow. Here
> are our primary requirement concerning the machine interface:
>
>   * Fit the tree structure of LTTng commands structures.
>   * It can be parse from bash/shell script with no external helpers.
>   * The new format must not introduce any dependencies to LTTng.
>   * [Optional] Still human readable, but fit to be parse.This is not a
>      priority since we are talking about a machine interface.
>
> Here are the reason why we think XML should be used for the machine
> interface:
>
>   * The expression of LTTng hierarchy really fit a XML tree structure.
>   * Since Jgalar add a load-save feature that depend on libxml2, we are
>     not introducing any dependencies. Choosing another format would
>     most likely introduce new dependency.
>   * XML is robust and easily extensible. It will also be easy to
>     maintains (add new commands, modify existing commands etc.)
>   * There are tools that allow you to evaluate XPath expressions from
>     the bash.

In the requirements above, you said "It can be parse from bash/shell
script with no external helpers". For XML it's not really the case. I
don't really mind (I am fine withXML), but I just think those are
contradictory statements.

>   * XML is an established standard
>
> That's why we consider XML format as a suitable solution.
>
> Now for the machine interface syntax:
>
>   * Since we are aware that multiple output could be supported, we want an
>     unify flag for all machine interface command.
> The flag --xml would be used to output xml from LTTng (we could add
> similar
> options like --json, --yaml etc.). This would be similar to svn
> machine interface and speak for itself.
>   * Also, to avoid confusion we will use a different primary command for
> input and
>     output machine interface.
> Ex: lttng machine <input>
> Ex: lttng --xml <output>

Can you give an (preliminary) example usage of lttng machine ? The XML
document would be a big string / a single argument ?

Also, could lttng machine also be able to read its input from stdin?
If it does, could it receive multiple commands in a single invocation?
For example, each root-level XML element could represent a command.
This way, a program would have to do a single exec and then push all
the commands one by one to the lttng client.

I have not really thought this through, I am just putting the idea in the wild.

> Any points we are missing?

I don't know how it would look, but as a user/developer, it would be
nice if there was a way for lttng to dump me the XML that corresponds
to a command (e.g. what I need to feed lttng machine to have the same
effect as a command).

Example with syntax I just made up:

lttng --dump-xml-input enable-event -k sched_switch,other_event

would output

<lttng>
    <command name="enable-event" domain="kernel">
        <event name="sched_switch"/>
        <event name="other_event"/>
    </command>
</lttng>

I feel that this feature would make the machine interface more
approachable (less doc to read before you start being efficient). What
do you think?

Simon

[lttng-dev] Machine Interface

[Jonathan Rajotte]

On Thu, Feb 13, 2014 at 01:26:09PM -0500, Simon Marchi wrote:
> It looks good!
> 
> Comments below.
> 
> On 13 February 2014 13:00, Olivier Cotte <olivier.cotte at polymtl.ca> wrote:
> > At first, we wanted thank you for your quick and constructive feedback!
> >
> > After reflection on the nature of the problem and consideration of your
> > feedback, we've arrived with the following solution.
> >
> > The technology that will be use is XML. The integration of the machine
> > interface to LTTng will be done in the most modular manner to facilitate the
> > support multiples outputs format. This option will most likely be available
> > at configure time.
> >
> > For the moment, we will focus our effort into adding the XML feature only.
> > If many outputs languages is still a viable feature, it will be possible to
> > investigate.
> >
> > Our main goal is of course to promote extensibility, ease of development for
> > new tools and establish a standards for the communication control flow. Here
> > are our primary requirement concerning the machine interface:
> >
> >   * Fit the tree structure of LTTng commands structures.
> >   * It can be parse from bash/shell script with no external helpers.
> >   * The new format must not introduce any dependencies to LTTng.
> >   * [Optional] Still human readable, but fit to be parse.This is not a
> >      priority since we are talking about a machine interface.
> >
> > Here are the reason why we think XML should be used for the machine
> > interface:
> >
> >   * The expression of LTTng hierarchy really fit a XML tree structure.
> >   * Since Jgalar add a load-save feature that depend on libxml2, we are
> >     not introducing any dependencies. Choosing another format would
> >     most likely introduce new dependency.
> >   * XML is robust and easily extensible. It will also be easy to
> >     maintains (add new commands, modify existing commands etc.)
> >   * There are tools that allow you to evaluate XPath expressions from
> >     the bash.
> 
> In the requirements above, you said "It can be parse from bash/shell
> script with no external helpers". For XML it's not really the case. I
> don't really mind (I am fine withXML), but I just think those are
> contradictory statements.
True, but the requirement are 'ideal objectives'. The choice of XML do not complies with
this requirement but fits most of the requirements. I think we
misspoke on this.
> 
> >   * XML is an established standard
> >
> > That's why we consider XML format as a suitable solution.
> >
> > Now for the machine interface syntax:
> >
> >   * Since we are aware that multiple output could be supported, we want an
> >     unify flag for all machine interface command.
> > The flag --xml would be used to output xml from LTTng (we could add
> > similar
> > options like --json, --yaml etc.). This would be similar to svn
> > machine interface and speak for itself.
> >   * Also, to avoid confusion we will use a different primary command for
> > input and
> >     output machine interface.
> > Ex: lttng machine <input>
> > Ex: lttng --xml <output>
> 
> Can you give an (preliminary) example usage of lttng machine ? The XML
> document would be a big string / a single argument ?
> 
We are not sure yet... but probably a big string(one-lined XML).
Could be a file but i don't see any advantage.

Another option would be a prompt.. like GDB Machine Interface.
> Also, could lttng machine also be able to read its input from stdin?
> If it does, could it receive multiple commands in a single invocation?
> For example, each root-level XML element could represent a command.
> This way, a program would have to do a single exec and then push all
> the commands one by one to the lttng client.
> 
That was the plan.

To be frank, we will mostly put effort in the output side of MI.If
everything is *functional* and *approved* we will work on input.

> I have not really thought this through, I am just putting the idea in the wild.
> 
> > Any points we are missing?
> 
> I don't know how it would look, but as a user/developer, it would be
> nice if there was a way for lttng to dump me the XML that corresponds
> to a command (e.g. what I need to feed lttng machine to have the same
> effect as a command).
> 
> Example with syntax I just made up:
> 
> lttng --dump-xml-input enable-event -k sched_switch,other_event
> 
> would output
> 
> <lttng>
>     <command name="enable-event" domain="kernel">
>         <event name="sched_switch"/>
>         <event name="other_event"/>
>     </command>
> </lttng>
> 
nice idea

> I feel that this feature would make the machine interface more
> approachable (less doc to read before you start being efficient). What
> do you think?
> 
> Simon
> 
> _______________________________________________
> lttng-dev mailing list
> lttng-dev at lists.lttng.org
> http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

-- 
Jonathan Rajotte Julien
Charg? de laboratoire INF1995
Membre MD6
Polytechnique Montr?al

[lttng-dev] Machine Interface

[Thibault, Daniel]

----------------------------------------------------------------------
Date: Thu, 13 Feb 2014 13:00:17 -0500
From: Olivier Cotte <olivier.cotte@polymtl.ca>

> Now for the machine interface syntax:
>
>   * Also, to avoid confusion we will use a different primary command for input and
>     output machine interface.
>		Ex: lttng machine <input>
>		Ex: lttng --xml <output>

   I have a problem with the broken symmetry here.  If you have a command (e.g. 'machine') for input, why is it you have none for output?  What happens if, for instance, I type 'lttng --xml list -k'?  Am I going to get an xml-formatted list of kernel tracepoints to stdout?

Daniel U. Thibault
Protection des syst?mes et contremesures (PSC) | Systems Protection & Countermeasures (SPC)
Cyber s?curit? pour les missions essentielles (CME) | Mission Critical Cyber Security (MCCS)
R & D pour la d?fense Canada - Valcartier (RDDC Valcartier) | Defence R&D Canada - Valcartier (DRDC Valcartier)
2459 route de la Bravoure
Qu?bec QC  G3J 1X5
CANADA
Vox : (418) 844-4000 x4245
Fax : (418) 844-4538
NAC : 918V QSDJ <http://www.travelgis.com/map.asp?addr=918V%20QSDJ>
Gouvernement du Canada | Government of Canada
<http://www.valcartier.drdc-rddc.gc.ca/>

[lttng-dev] Machine Interface

[Jonathan Rajotte]

> > Now for the machine interface syntax:
> >
> >   * Also, to avoid confusion we will use a different primary command for input and
> >     output machine interface.
> >		Ex: lttng machine <input>
> >		Ex: lttng --xml <output>

This should be :
		for MI input: lttng machine <input>
		for XML output: lttng <command> <command-option> --xml

--xml would be an option for each command
> 
>    I have a problem with the broken symmetry here.  If you have a command (e.g. 'machine') for input, why is it you have none for output?  What happens if, for instance, I type 'lttng --xml list -k'?  Am I going to get an xml-formatted list of kernel tracepoints to stdout?
>
The syntax would be: lttng list -k --xml
and the output would be an xml-formatted list of kernel tracepoints to
stdout.


> Daniel U. Thibault
> Protection des syst?mes et contremesures (PSC) | Systems Protection & Countermeasures (SPC)
> Cyber s?curit? pour les missions essentielles (CME) | Mission Critical Cyber Security (MCCS)
> R & D pour la d?fense Canada - Valcartier (RDDC Valcartier) | Defence R&D Canada - Valcartier (DRDC Valcartier)
> 2459 route de la Bravoure
> Qu?bec QC  G3J 1X5
> CANADA
> Vox : (418) 844-4000 x4245
> Fax : (418) 844-4538
> NAC : 918V QSDJ <http://www.travelgis.com/map.asp?addr=918V%20QSDJ>
> Gouvernement du Canada | Government of Canada
> <http://www.valcartier.drdc-rddc.gc.ca/>
> 
> _______________________________________________
> lttng-dev mailing list
> lttng-dev at lists.lttng.org
> http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

-- 
Jonathan Rajotte Julien
Charg? de laboratoire INF1995
Membre MD6
Polytechnique Montr?al

[lttng-dev] Machine Interface

[Thibault, Daniel]

-----Message d'origine-----
> This should be :
>		for MI input: lttng machine <input>
>		for XML output: lttng <command> <command-option> --xml
>
> --xml would be an option for each command
> The syntax would be: lttng list -k --xml and the output would be an xml-formatted list of kernel tracepoints to stdout.

   Okay, how about an example where one uses ' lttng list -k --xml' to get the list of kernel tracepoints and then pipes this to 'enable-event' in order to enable each event?  As a one-liner, mind you.  The idea is one would insert some sed/grep magic in the pipe to assign events to channels or filter some out.

Daniel U. Thibault
Protection des syst?mes et contremesures (PSC) | Systems Protection & Countermeasures (SPC)
Cyber s?curit? pour les missions essentielles (CME) | Mission Critical Cyber Security (MCCS)
R & D pour la d?fense Canada - Valcartier (RDDC Valcartier) | Defence R&D Canada - Valcartier (DRDC Valcartier)
2459 route de la Bravoure
Qu?bec QC  G3J 1X5
CANADA
Vox : (418) 844-4000 x4245
Fax : (418) 844-4538
NAC : 918V QSDJ <http://www.travelgis.com/map.asp?addr=918V%20QSDJ>
Gouvernement du Canada | Government of Canada
<http://www.valcartier.drdc-rddc.gc.ca/>

[lttng-dev] Machine Interface

[Thibault, Daniel]

----------------------------------------------------------------------
Date: Wed, 5 Feb 2014 21:42:19 -0500
From: Jonathan Rajotte <jonathan.r.julien@gmail.com>

> Machine interface output would be generated only with --mi option and could contain additional information than regular output.
>
> We are focusing on output right now but we may eventually add a 'mi' command for input if necessary.

   I have one recommendation: do not use the same mnemonic for "generate machine interface output" (--mi) and "accept machine interface input" (mi command).  It is bound to create endless user confusion and attendant errors.  Maybe '--machine-interface' and 'machine', respectively, instead?

Daniel U. Thibault
Protection des syst?mes et contremesures (PSC) | Systems Protection & Countermeasures (SPC)
Cyber s?curit? pour les missions essentielles (CME) | Mission Critical Cyber Security (MCCS)
R & D pour la d?fense Canada - Valcartier (RDDC Valcartier) | Defence R&D Canada - Valcartier (DRDC Valcartier)
2459 route de la Bravoure
Qu?bec QC  G3J 1X5
CANADA
Vox : (418) 844-4000 x4245
Fax : (418) 844-4538
NAC : 918V QSDJ <http://www.travelgis.com/map.asp?addr=918V%20QSDJ>
Gouvernement du Canada | Government of Canada
<http://www.valcartier.drdc-rddc.gc.ca/>

[lttng-dev] Machine Interface

[Jonathan Rajotte]

Hello all,

After speaking with Michel Dagenais, Genevi?ve Bastien, folks over at EfficiOs
and Ericsson, a machine interface for lttng-tool would be a nice feature
to have. Olivier Cotte and me will be working on MI for the
next few weeks.

The main objective would be to reduce parsing complexity
in control tools.
ex:Regex used in TMF control service.
(org.eclipse.linuxtools.internal.lttng2.ui.views.control.service.LTTngControlService.java)

Multiples solutions come to mind:
-XML
-JSON
-YAML
-Custom syntax (like GDB Machine interface* as proposed by Alexandre
				Montplaisir)

XML/JSON/YAML promote extensibility and ease of development for new tools
dependant on lttng-tools by being established standards.
On the other hand, they add dependencies to lttng-tools where a custom syntax
would not.

Machine interface output would be generated only with --mi option and
could contain additional information than regular output.

We are focusing on output right now but we may eventually add a 'mi'
command for input if necessary.

What do you think ?

Thanks


*https://sourceware.org/gdb/onlinedocs/gdb/GDB_002fMI.html

-- 
Jonathan Rajotte Julien
Charg? de laboratoire INF1995
Membre MD6
Polytechnique Montr?al

[lttng-dev] Machine Interface

[Woegerer, Paul]

On 02/06/2014 03:42 AM, Jonathan Rajotte wrote:
> Hello all,
>
> After speaking with Michel Dagenais, Genevi?ve Bastien, folks over at EfficiOs
> and Ericsson, a machine interface for lttng-tool would be a nice feature
> to have. Olivier Cotte and me will be working on MI for the
> next few weeks.
>
> The main objective would be to reduce parsing complexity
> in control tools.
> ex:Regex used in TMF control service.
> (org.eclipse.linuxtools.internal.lttng2.ui.views.control.service.LTTngControlService.java)
>
> Multiples solutions come to mind:
> -XML
> -JSON
> -YAML

YAML sounds very attractive in this context.

It's easily parseable even from a shell script. If you need to automate
LTTng tracing tasks on a small embedded target busybox is all you have!

Best,
Paul

> -Custom syntax (like GDB Machine interface* as proposed by Alexandre
> 				Montplaisir)
>
> XML/JSON/YAML promote extensibility and ease of development for new tools
> dependant on lttng-tools by being established standards.
> On the other hand, they add dependencies to lttng-tools where a custom syntax
> would not.
>
> Machine interface output would be generated only with --mi option and
> could contain additional information than regular output.
>
> We are focusing on output right now but we may eventually add a 'mi'
> command for input if necessary.
>
> What do you think ?
>
> Thanks
>
>
> *https://sourceware.org/gdb/onlinedocs/gdb/GDB_002fMI.html
>


-- 
Paul Woegerer, SW Development Engineer
Sourcery Analyzer <http://go.mentor.com/sourceryanalyzer>
Mentor Graphics, Embedded Software Division

[lttng-dev] Machine Interface

[Simon Marchi]

On 6 February 2014 02:41, Woegerer, Paul <Paul_Woegerer at mentor.com> wrote:
> On 02/06/2014 03:42 AM, Jonathan Rajotte wrote:
>> Hello all,
>>
>> After speaking with Michel Dagenais, Genevi?ve Bastien, folks over at EfficiOs
>> and Ericsson, a machine interface for lttng-tool would be a nice feature
>> to have. Olivier Cotte and me will be working on MI for the
>> next few weeks.
>>
>> The main objective would be to reduce parsing complexity
>> in control tools.
>> ex:Regex used in TMF control service.
>> (org.eclipse.linuxtools.internal.lttng2.ui.views.control.service.LTTngControlService.java)
>>
>> Multiples solutions come to mind:
>> -XML
>> -JSON
>> -YAML
>
> YAML sounds very attractive in this context.

I am also fond of the yaml syntax. It is very easy to read/write by a
human. Not sure how much this matters though, since is a machine
interface.

> It's easily parseable even from a shell script. If you need to automate
> LTTng tracing tasks on a small embedded target busybox is all you have!

This point is relevant for me, since I actually intend to use this to
enhance bash completion, which is done with a bash script.

I never thought I would say that one day, but I think that XML could
have an advantage here. Parsing yaml with regexp (such as in a shell
script) is easier than XML, but it's still not bulletproof. Consider
this eventual yaml structure I just made up for the output of the
session list.

sessions:
  - name: my_session
    domain: kernel
    events:
      - name: sched_switch
        filter: prev_pid == 1234
      - name: sched_migrate_task
        otherproperty: "something"
  - name: another_session
    domain: ust
    events:
      - name: event_hello
      - name: event_bonjour
        loglevel: 3

If you want to get the list of sessions, you'll probably want to grep
the "name: " entries, but then event names would match as well. Of
course, names could be different, or you could grep for the
appropriate amount of spaces/tabs at the beginning of the line. But
then it also depends if the name dictionary entry is the first, then
it will have a "-" in front... It's possible, but it will also be very
easy to break scripts.

With XML, there are tools (xmlstarlet, xmllint, probably others) that
allow you to evaluate XPath expressions and thus not depend of the
formatting. For example, an equivalent XML schema could be

<lttng>
  <session name="my_session" domain="kernel">
    <event name="sched_switch" filter="prev_pid == 1234" />
    <event name="sched_migrate_task" otherproperty="something" />
  </session>
  <session name="another_session" domain="ust">
    <event name="event_hello" />
    <event name="event_bonjour" loglevel="3" />
  </session>
</lttng>

To get a plain list of sessions, you could then do

$ lttng list --mi | xmlstarlet sel -T -t -v "/lttng/session/@name" -n
my_session
another_session

Maybe such tools exist for yaml?

Also, I understand your point for embedded devices though, where space
is limited and you don't want to load an XML lib/tool just for that.
If the design of the MI is somewhat modular though, it could be
possible to support more than one output format, with the different
options available at configure time.


> Best,
> Paul
>
>> -Custom syntax (like GDB Machine interface* as proposed by Alexandre
>>                               Montplaisir)

I would suggest not going for a custom syntax... that sounds like more
work for more chances of bugs.

>> XML/JSON/YAML promote extensibility and ease of development for new tools
>> dependant on lttng-tools by being established standards.
>> On the other hand, they add dependencies to lttng-tools where a custom syntax
>> would not.

I would suggest making the dependency optional. If you don't have the
lib, or don't want to depend on the lib, have a --disable-something at
configure time.

>> Machine interface output would be generated only with --mi option and
>> could contain additional information than regular output.
>>
>> We are focusing on output right now but we may eventually add a 'mi'
>> command for input if necessary.
>>
>> What do you think ?
>>
>> Thanks
>>
>>
>> *https://sourceware.org/gdb/onlinedocs/gdb/GDB_002fMI.html
>>
>
>
> --
> Paul Woegerer, SW Development Engineer
> Sourcery Analyzer <http://go.mentor.com/sourceryanalyzer>
> Mentor Graphics, Embedded Software Division
>
>
> _______________________________________________
> lttng-dev mailing list
> lttng-dev at lists.lttng.org
> http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

[lttng-dev] Machine Interface

[Woegerer, Paul]

Hi Simon,

On 02/06/2014 02:32 PM, Simon Marchi wrote:
> script) is easier than XML, but it's still not bulletproof. Consider
> this eventual yaml structure I just made up for the output of the
> session list.
> 
> sessions:
>   - name: my_session
>     domain: kernel
>     events:
>       - name: sched_switch
>         filter: prev_pid == 1234
>       - name: sched_migrate_task
>         otherproperty: "something"
>   - name: another_session
>     domain: ust
>     events:
>       - name: event_hello
>       - name: event_bonjour
>         loglevel: 3
> 
> If you want to get the list of sessions, you'll probably want to grep
> the "name: " entries, but then event names would match as well. Of
> course, names could be different, or you could grep for the
> appropriate amount of spaces/tabs at the beginning of the line. But
> then it also depends if the name dictionary entry is the first, then
> it will have a "-" in front... It's possible, but it will also be very
> easy to break scripts.

The following simple script below would e.g. give you the list of
sessions for you example above. To demonstrate the flexibility it
also generates the event list for session my_session. This is just
a quick prototype but it should make clear that YAML is considerably
easier to parse from a shell script than any other option (and also
reasonably robust). Also notice that no external helpers (like grep)
are needed.

list_of_sessions=''
current_session=''
my_session_events=''

while IFS= read -r; do
	line=$REPLY
	case "${line}" in 
		"      - "*)
			;&
		"        "*)
			L2_key=${line:8}
			L2_key=${L2_key%:*}
			let vpos=2+8+${#L2_key}
			L2_value=${line:${vpos}}
			;;
		"  - "*)
			# New L1 list elem -> reset L2 state
			L2_key=''
			L2_value=''
			;&
		"    "*)
			L1_key=${line:4}
			L1_key=${L1_key%:*}
			let vpos=2+4+${#L1_key}
			L1_value=${line:${vpos}}
			;;
	esac
	echo "($L1_key,$L1_value), ($L2_key,$L2_value)"

	# Collect the session names
	if [ "$L1_key" = "name" ]; then
		current_session="$L1_value"
		list_of_sessions="$list_of_sessions $current_session"
	fi
	# Which events are in session my_session ?
	if [ "$current_session" = "my_session" ] && [ "$L1_key" = "events" ] && [ "$L2_key" = "name" ]; then
		my_session_events="$my_session_events $L2_value"
	fi
done
echo
echo List of sessions: $list_of_sessions
echo my_session events: $my_session_events
echo


Thanks,
Paul

-- 
Paul Woegerer, SW Development Engineer
Sourcery Analyzer <http://go.mentor.com/sourceryanalyzer>
Mentor Graphics, Embedded Software Division

[lttng-dev] Machine Interface

[Simon Marchi]

On 6 February 2014 12:07, Woegerer, Paul <Paul_Woegerer at mentor.com> wrote:
> Hi Simon,
>
> On 02/06/2014 02:32 PM, Simon Marchi wrote:
>> script) is easier than XML, but it's still not bulletproof. Consider
>> this eventual yaml structure I just made up for the output of the
>> session list.
>>
>> sessions:
>>   - name: my_session
>>     domain: kernel
>>     events:
>>       - name: sched_switch
>>         filter: prev_pid == 1234
>>       - name: sched_migrate_task
>>         otherproperty: "something"
>>   - name: another_session
>>     domain: ust
>>     events:
>>       - name: event_hello
>>       - name: event_bonjour
>>         loglevel: 3
>>
>> If you want to get the list of sessions, you'll probably want to grep
>> the "name: " entries, but then event names would match as well. Of
>> course, names could be different, or you could grep for the
>> appropriate amount of spaces/tabs at the beginning of the line. But
>> then it also depends if the name dictionary entry is the first, then
>> it will have a "-" in front... It's possible, but it will also be very
>> easy to break scripts.
>
> The following simple script below would e.g. give you the list of
> sessions for you example above. To demonstrate the flexibility it
> also generates the event list for session my_session. This is just
> a quick prototype but it should make clear that YAML is considerably
> easier to parse from a shell script than any other option (and also
> reasonably robust). Also notice that no external helpers (like grep)
> are needed.

This is impressive scripting! Although I can see this growing a little
bit out of control for more complex yaml trees. I agree with you that
yaml is the easiest of them to parse in a pure shell script since the
formatting is more restricted. But if you're on a machine that can
take it, using a solution based on an XPath tool (or equivalent) will
make it so much more maintainable.

> list_of_sessions=''
> current_session=''
> my_session_events=''
>
> while IFS= read -r; do
>         line=$REPLY
>         case "${line}" in
>                 "      - "*)
>                         ;&
>                 "        "*)
>                         L2_key=${line:8}
>                         L2_key=${L2_key%:*}
>                         let vpos=2+8+${#L2_key}
>                         L2_value=${line:${vpos}}
>                         ;;
>                 "  - "*)
>                         # New L1 list elem -> reset L2 state
>                         L2_key=''
>                         L2_value=''
>                         ;&
>                 "    "*)
>                         L1_key=${line:4}
>                         L1_key=${L1_key%:*}
>                         let vpos=2+4+${#L1_key}
>                         L1_value=${line:${vpos}}
>                         ;;
>         esac
>         echo "($L1_key,$L1_value), ($L2_key,$L2_value)"
>
>         # Collect the session names
>         if [ "$L1_key" = "name" ]; then
>                 current_session="$L1_value"
>                 list_of_sessions="$list_of_sessions $current_session"
>         fi
>         # Which events are in session my_session ?
>         if [ "$current_session" = "my_session" ] && [ "$L1_key" = "events" ] && [ "$L2_key" = "name" ]; then
>                 my_session_events="$my_session_events $L2_value"
>         fi
> done
> echo
> echo List of sessions: $list_of_sessions
> echo my_session events: $my_session_events
> echo
>
>
> Thanks,
> Paul
>
> --
> Paul Woegerer, SW Development Engineer
> Sourcery Analyzer <http://go.mentor.com/sourceryanalyzer>
> Mentor Graphics, Embedded Software Division

[lttng-dev] Machine Interface

[Christian Babeux]

Hi Jonathan,

> Multiples solutions come to mind:
> -XML
> -JSON
> -YAML
> -Custom syntax (like GDB Machine interface* as proposed by Alexandre
>                                 Montplaisir)

I would advocate for the use of XML for the machine interface. Two
reasons come to mind:

1 - The expression of hierarchy (session, events, etc.) is much easier with XML.

2 - In a recent post [1], it was decided that the configuration load
and restore (for sessions) would also use XML. This will introduce a
new dependency on libxml. If we were to choose yet another format for
the machine interface, we would probably need to introduce another
dependency to some library.

> Machine interface output would be generated only with --mi option and
> could contain additional information than regular output.

Perhaps adding --xml option to the relevant command? This would be
similar to svn machine interface [2].

Thanks,

Christian

[1] - http://lists.lttng.org/pipermail/lttng-dev/2013-December/022068.html
[2] - http://svnbook.red-bean.com/en/1.7/svn-book.html#svn.ref.svn.sw

On Wed, Feb 5, 2014 at 9:42 PM, Jonathan Rajotte
<jonathan.r.julien at gmail.com> wrote:
> Hello all,
>
> After speaking with Michel Dagenais, Genevi?ve Bastien, folks over at EfficiOs
> and Ericsson, a machine interface for lttng-tool would be a nice feature
> to have. Olivier Cotte and me will be working on MI for the
> next few weeks.
>
> The main objective would be to reduce parsing complexity
> in control tools.
> ex:Regex used in TMF control service.
> (org.eclipse.linuxtools.internal.lttng2.ui.views.control.service.LTTngControlService.java)
>
> Multiples solutions come to mind:
> -XML
> -JSON
> -YAML
> -Custom syntax (like GDB Machine interface* as proposed by Alexandre
>                                 Montplaisir)
>
> XML/JSON/YAML promote extensibility and ease of development for new tools
> dependant on lttng-tools by being established standards.
> On the other hand, they add dependencies to lttng-tools where a custom syntax
> would not.
>
> Machine interface output would be generated only with --mi option and
> could contain additional information than regular output.
>
> We are focusing on output right now but we may eventually add a 'mi'
> command for input if necessary.
>
> What do you think ?
>
> Thanks
>
>
> *https://sourceware.org/gdb/onlinedocs/gdb/GDB_002fMI.html
>
> --
> Jonathan Rajotte Julien
> Charg? de laboratoire INF1995
> Membre MD6
> Polytechnique Montr?al
>
> _______________________________________________
> lttng-dev mailing list
> lttng-dev at lists.lttng.org
> http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

[lttng-dev] Machine Interface

[Jonathan Rajotte]

On Thu, Feb 06, 2014 at 11:11:39AM -0500, Christian Babeux wrote:
> Hi Jonathan,
> 
> > Multiples solutions come to mind:
> > -XML
> > -JSON
> > -YAML
> > -Custom syntax (like GDB Machine interface* as proposed by Alexandre
> >                                 Montplaisir)
> 
> I would advocate for the use of XML for the machine interface. Two
> reasons come to mind:
> 
> 1 - The expression of hierarchy (session, events, etc.) is much easier with XML.
> 
> 2 - In a recent post [1], it was decided that the configuration load
> and restore (for sessions) would also use XML. This will introduce a
> new dependency on libxml. If we were to choose yet another format for
> the machine interface, we would probably need to introduce another
> dependency to some library.

Good point
+1 for XML

> 
> > Machine interface output would be generated only with --mi option and
> > could contain additional information than regular output.
> 
> Perhaps adding --xml option to the relevant command? This would be
> similar to svn machine interface [2].

How about --machine-interface <output-type>?
By default <output-type> would be the chosen format.

This open opportunities for other MI type and have an unified syntax.

--xml option is clear and efficient.We could
add an option for each type (--json,--yaml) if needed.

> 
> Thanks,
> 
> Christian
> 
> [1] - http://lists.lttng.org/pipermail/lttng-dev/2013-December/022068.html
> [2] - http://svnbook.red-bean.com/en/1.7/svn-book.html#svn.ref.svn.sw
> 
> On Wed, Feb 5, 2014 at 9:42 PM, Jonathan Rajotte
> <jonathan.r.julien at gmail.com> wrote:
> > Hello all,
> >
> > After speaking with Michel Dagenais, Genevi?ve Bastien, folks over at EfficiOs
> > and Ericsson, a machine interface for lttng-tool would be a nice feature
> > to have. Olivier Cotte and me will be working on MI for the
> > next few weeks.
> >
> > The main objective would be to reduce parsing complexity
> > in control tools.
> > ex:Regex used in TMF control service.
> > (org.eclipse.linuxtools.internal.lttng2.ui.views.control.service.LTTngControlService.java)
> >
> > Multiples solutions come to mind:
> > -XML
> > -JSON
> > -YAML
> > -Custom syntax (like GDB Machine interface* as proposed by Alexandre
> >                                 Montplaisir)
> >
> > XML/JSON/YAML promote extensibility and ease of development for new tools
> > dependant on lttng-tools by being established standards.
> > On the other hand, they add dependencies to lttng-tools where a custom syntax
> > would not.
> >
> > Machine interface output would be generated only with --mi option and
> > could contain additional information than regular output.
> >
> > We are focusing on output right now but we may eventually add a 'mi'
> > command for input if necessary.
> >
> > What do you think ?
> >
> > Thanks
> >
> >
> > *https://sourceware.org/gdb/onlinedocs/gdb/GDB_002fMI.html
> >
> > --
> > Jonathan Rajotte Julien
> > Charg? de laboratoire INF1995
> > Membre MD6
> > Polytechnique Montr?al
> >
> > _______________________________________________
> > lttng-dev mailing list
> > lttng-dev at lists.lttng.org
> > http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

-- 
Jonathan Rajotte Juliana
Charg? de laboratoire INF1995
Membre MD6
Polytechnique Montr?al

[lttng-dev] Machine Interface

[Christian Babeux]

> How about --machine-interface <output-type>?
> By default <output-type> would be the chosen format.
>
> This open opportunities for other MI type and have an unified syntax.
>
> --xml option is clear and efficient.We could
> add an option for each type (--json,--yaml) if needed.

IMO, supporting multiple output formats would be a tad bit overkill.
Moreover, the issue of bringing in additional library dependencies to
support multiple output format remains.

Thanks,

Christian

On Thu, Feb 6, 2014 at 12:00 PM, Jonathan Rajotte
<jonathan.r.julien at gmail.com> wrote:
> On Thu, Feb 06, 2014 at 11:11:39AM -0500, Christian Babeux wrote:
>> Hi Jonathan,
>>
>> > Multiples solutions come to mind:
>> > -XML
>> > -JSON
>> > -YAML
>> > -Custom syntax (like GDB Machine interface* as proposed by Alexandre
>> >                                 Montplaisir)
>>
>> I would advocate for the use of XML for the machine interface. Two
>> reasons come to mind:
>>
>> 1 - The expression of hierarchy (session, events, etc.) is much easier with XML.
>>
>> 2 - In a recent post [1], it was decided that the configuration load
>> and restore (for sessions) would also use XML. This will introduce a
>> new dependency on libxml. If we were to choose yet another format for
>> the machine interface, we would probably need to introduce another
>> dependency to some library.
>
> Good point
> +1 for XML
>
>>
>> > Machine interface output would be generated only with --mi option and
>> > could contain additional information than regular output.
>>
>> Perhaps adding --xml option to the relevant command? This would be
>> similar to svn machine interface [2].
>
> How about --machine-interface <output-type>?
> By default <output-type> would be the chosen format.
>
> This open opportunities for other MI type and have an unified syntax.
>
> --xml option is clear and efficient.We could
> add an option for each type (--json,--yaml) if needed.
>
>>
>> Thanks,
>>
>> Christian
>>
>> [1] - http://lists.lttng.org/pipermail/lttng-dev/2013-December/022068.html
>> [2] - http://svnbook.red-bean.com/en/1.7/svn-book.html#svn.ref.svn.sw
>>
>> On Wed, Feb 5, 2014 at 9:42 PM, Jonathan Rajotte
>> <jonathan.r.julien at gmail.com> wrote:
>> > Hello all,
>> >
>> > After speaking with Michel Dagenais, Genevi?ve Bastien, folks over at EfficiOs
>> > and Ericsson, a machine interface for lttng-tool would be a nice feature
>> > to have. Olivier Cotte and me will be working on MI for the
>> > next few weeks.
>> >
>> > The main objective would be to reduce parsing complexity
>> > in control tools.
>> > ex:Regex used in TMF control service.
>> > (org.eclipse.linuxtools.internal.lttng2.ui.views.control.service.LTTngControlService.java)
>> >
>> > Multiples solutions come to mind:
>> > -XML
>> > -JSON
>> > -YAML
>> > -Custom syntax (like GDB Machine interface* as proposed by Alexandre
>> >                                 Montplaisir)
>> >
>> > XML/JSON/YAML promote extensibility and ease of development for new tools
>> > dependant on lttng-tools by being established standards.
>> > On the other hand, they add dependencies to lttng-tools where a custom syntax
>> > would not.
>> >
>> > Machine interface output would be generated only with --mi option and
>> > could contain additional information than regular output.
>> >
>> > We are focusing on output right now but we may eventually add a 'mi'
>> > command for input if necessary.
>> >
>> > What do you think ?
>> >
>> > Thanks
>> >
>> >
>> > *https://sourceware.org/gdb/onlinedocs/gdb/GDB_002fMI.html
>> >
>> > --
>> > Jonathan Rajotte Julien
>> > Charg? de laboratoire INF1995
>> > Membre MD6
>> > Polytechnique Montr?al
>> >
>> > _______________________________________________
>> > lttng-dev mailing list
>> > lttng-dev at lists.lttng.org
>> > http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
>
> --
> Jonathan Rajotte Juliana
> Charg? de laboratoire INF1995
> Membre MD6
> Polytechnique Montr?al