[lttng-dev] Babeltrace ctf-text duplicated error message

[lttng-dev] Babeltrace ctf-text duplicated error message

[Oestman, Fredrik]

Hello,


I've found what appears to be a (very minor) bug in babeltrace, see patch below.


Cheers,

Fredrik ?stman


diff --git a/formats/ctf-text/ctf-text.c b/formats/ctf-text/ctf-text.c
index 273c9b7..fa0cb4d 100644
--- a/formats/ctf-text/ctf-text.c
+++ b/formats/ctf-text/ctf-text.c
@@ -236,8 +236,8 @@ int ctf_text_write_event(struct stream_pos *ppos, struct ctf_stream *stream)
 		return -EINVAL;
 	}
 	event_class = g_ptr_array_index(stream_class->events_by_id, id);
-	if (!event) {
-		fprintf(stderr, "[error] Event id %" PRIu64 " is unknown.\n", id);
+	if (!event_class) {
+		fprintf(stderr, "[error] Event class id %" PRIu64 " is unknown.\n", id);
 		return -EINVAL;
 	}

[lttng-dev] Writing a babeltrace output plugin

[Oestman, Fredrik]

Hello,


I've fairly easily manged to add a new text-based output format to babeltrace by cloning the existing ctf-text code and making changes to it. Thanks to the registration mechanism it was also easy to get the program to recognize the new format on the command line.

Now, the problem I have with my new text format is that it completely lacks metadata.

What would be the intended way to provide an output format with metadata, or from where should I preferably collect metadata when writing a trace log in a new format?

Thanks in advance for any advice!


Cheers,

Fredrik ?stman

[lttng-dev] Writing a babeltrace output plugin

[Mathieu Desnoyers]

* Oestman, Fredrik (Fredrik_Oestman at mentor.com) wrote:
> Hello,
> 
> 
> I've fairly easily manged to add a new text-based output format to babeltrace by cloning the existing ctf-text code and making changes to it. Thanks to the registration mechanism it was also easy to get the program to recognize the new format on the command line.
> 
> Now, the problem I have with my new text format is that it completely lacks metadata.
> 
> What would be the intended way to provide an output format with metadata, or from where should I preferably collect metadata when writing a trace log in a new format?
> 
> Thanks in advance for any advice!

You'll have to either use the public API (see which files are installed
by babeltrace into your system): this is the preferred solution,
although there might be some missing metadata information (it's made to
support a trace reader, not a trace converter that would have to
regenerate the entire metadata).

The second solution is to look at include/babeltrace/ctf-ir/metadata.h
in-tree, and include/babeltrace/types.h (also in-tree). Those files
contain almost all the metadata of the CTF trace read, so you should be
able to extract your own metadata from those.

Hoping this helps,

Best regards,

Mathieu

> 
> 
> Cheers,
> 
> Fredrik ?stman
> 
> _______________________________________________
> lttng-dev mailing list
> lttng-dev at lists.lttng.org
> http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

-- 
Mathieu Desnoyers
Operating System Efficiency R&D Consultant
EfficiOS Inc.
http://www.efficios.com

[lttng-dev] Writing a babeltrace output plugin

[Oestman, Fredrik]

Hallo Mathieu,


Mathieu Desnoyers wrote:
> You'll have to either use the public API (see which files are installed
> by babeltrace into your system): this is the preferred solution,

OK, I found babeltrace.h. Could you give some hints to how to get started with this API?


Will there be a converter API as well in the future?


Cheers,

Fredrik

[lttng-dev] Writing a babeltrace output plugin

[Mathieu Desnoyers]

* Oestman, Fredrik (Fredrik_Oestman at mentor.com) wrote:
> Hallo Mathieu,
> 
> 
> Mathieu Desnoyers wrote:
> > You'll have to either use the public API (see which files are installed
> > by babeltrace into your system): this is the preferred solution,
> 
> OK, I found babeltrace.h. Could you give some hints to how to get
> started with this API?

I have no other clues in mind than looking at the examples already in
there.  Julien, in CC, could help you with some questions you might
have.

> 
> 
> Will there be a converter API as well in the future?

Not currently planned: for now, all converters are expected to be
in-tree.

Thanks,

Mathieu

> 
> 
> Cheers,
> 
> Fredrik
> 

-- 
Mathieu Desnoyers
Operating System Efficiency R&D Consultant
EfficiOS Inc.
http://www.efficios.com

[lttng-dev] Writing a babeltrace output plugin

[Mathieu Desnoyers]

(resent to correct CC email address for Julien)

* Mathieu Desnoyers (mathieu.desnoyers at efficios.com) wrote:
> * Oestman, Fredrik (Fredrik_Oestman at mentor.com) wrote:
> > Hallo Mathieu,
> > 
> > 
> > Mathieu Desnoyers wrote:
> > > You'll have to either use the public API (see which files are installed
> > > by babeltrace into your system): this is the preferred solution,
> > 
> > OK, I found babeltrace.h. Could you give some hints to how to get
> > started with this API?
> 
> I have no other clues in mind than looking at the examples already in
> there.  Julien, in CC, could help you with some questions you might
> have.
> 
> > 
> > 
> > Will there be a converter API as well in the future?
> 
> Not currently planned: for now, all converters are expected to be
> in-tree.
> 
> Thanks,
> 
> Mathieu
> 
> > 
> > 
> > Cheers,
> > 
> > Fredrik
> > 
> 
> -- 
> Mathieu Desnoyers
> Operating System Efficiency R&D Consultant
> EfficiOS Inc.
> http://www.efficios.com
> 
> _______________________________________________
> lttng-dev mailing list
> lttng-dev at lists.lttng.org
> http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

-- 
Mathieu Desnoyers
Operating System Efficiency R&D Consultant
EfficiOS Inc.
http://www.efficios.com

[lttng-dev] Writing a babeltrace output plugin

[Oestman, Fredrik]

Hi Mathieu, hi Julien,


Mathieu wrote:
> I have no other clues in mind [to using the public API] than
> looking at the examples already in there

Where are the examples of using the public API?


> for now, all converters are expected to be in-tree.

Oh. So the way to go to add another output format for conversion from CTF is to
clone the formats/ctf-text directory and make changes as needed?


It would have been interesting to see the CTF output part as an example.
Can you say when such an option will be available?


Cheers,

Fredrik

[lttng-dev] Writing a babeltrace output plugin

[Mathieu Desnoyers]

* Oestman, Fredrik (Fredrik_Oestman at mentor.com) wrote:
> Hi Mathieu, hi Julien,
> 
> 
> Mathieu wrote:
> > I have no other clues in mind [to using the public API] than
> > looking at the examples already in there
> 
> Where are the examples of using the public API?

There is a lttv tree currently under heavy development, and the lttngtop
git tree on git.lttng.org.

> 
> 
> > for now, all converters are expected to be in-tree.
> 
> Oh. So the way to go to add another output format for conversion from CTF is to
> clone the formats/ctf-text directory and make changes as needed?

Yep.

> 
> 
> It would have been interesting to see the CTF output part as an example.
> Can you say when such an option will be available?

We do not use the intermediate representation for CTF generation (see
babeltrace-log.c): we generate the metadata directly as needed.
Generating a CTF output from the CTF intermediate representation is not
planned at this point, as it has very little use for us. LTTng-UST 2.0
and LTTng-modules 2.0 both generate CTF natively.

Thanks,

Mathieu

> 
> 
> Cheers,
> 
> Fredrik
> 

-- 
Mathieu Desnoyers
Operating System Efficiency R&D Consultant
EfficiOS Inc.
http://www.efficios.com

[lttng-dev] Writing a babeltrace output plugin

[Julien Desfossez]

Hi,

>>> I have no other clues in mind [to using the public API] than
>>> looking at the examples already in there
>>
>> Where are the examples of using the public API?
> 
> There is a lttv tree currently under heavy development, and the lttngtop
> git tree on git.lttng.org.

I would recommend looking at this test program :
git://git.dorsal.polymtl.ca/~jdesfossez/babeltrace-api-test

It is a really basic program that uses most of the API.

Julien

> 
>>
>>
>>> for now, all converters are expected to be in-tree.
>>
>> Oh. So the way to go to add another output format for conversion from CTF is to
>> clone the formats/ctf-text directory and make changes as needed?
> 
> Yep.
> 
>>
>>
>> It would have been interesting to see the CTF output part as an example.
>> Can you say when such an option will be available?
> 
> We do not use the intermediate representation for CTF generation (see
> babeltrace-log.c): we generate the metadata directly as needed.
> Generating a CTF output from the CTF intermediate representation is not
> planned at this point, as it has very little use for us. LTTng-UST 2.0
> and LTTng-modules 2.0 both generate CTF natively.
> 
> Thanks,
> 
> Mathieu
> 
>>
>>
>> Cheers,
>>
>> Fredrik
>>
> 

[lttng-dev] Writing a babeltrace output plugin

[Oestman, Fredrik]

Hi Julien,

> I would recommend looking at this test program :
> git://git.dorsal.polymtl.ca/~jdesfossez/babeltrace-api-test
> 
> It is a really basic program that uses most of the API.

This is great advice! Thank you very much. I will take a look at once.

One more question: Is the developer guide you are writing for the API or for the babeltrace tool? Would you consider sending me an intermediate version?


Cheers,

Fredrik

[lttng-dev] Writing a babeltrace output plugin

[Oestman, Fredrik]

Mathieu Desnoyers wrote:
> The second solution is to look at include/babeltrace/ctf-ir/metadata.h
> in-tree, and include/babeltrace/types.h (also in-tree). Those files
> contain almost all the metadata of the CTF trace read, so you should be
> able to extract your own metadata from those.

Is there a special reason that the field names in the metadata are prefixed with an underscore?


Cheers,

Fredrik ?stman

[lttng-dev] Writing a babeltrace output plugin

[Mathieu Desnoyers]

* Oestman, Fredrik (Fredrik_Oestman at mentor.com) wrote:
> Mathieu Desnoyers wrote:
> > The second solution is to look at include/babeltrace/ctf-ir/metadata.h
> > in-tree, and include/babeltrace/types.h (also in-tree). Those files
> > contain almost all the metadata of the CTF trace read, so you should be
> > able to extract your own metadata from those.
> 
> Is there a special reason that the field names in the metadata are
> prefixed with an underscore?

Yes, prefixing the field names is an oddness caused by the need to not
have ambiguity between field names declared within the local scope and
root scope names (event, stream, env, trace).

Lets assume we have a field called "stream" within an event (I am not
saying this should be done, I'm just saying this kind of thing happens
to be done automatically by the tracer, depending on the instrumentation
sources):

  uint32_t stream;

Followed by a sequence that refers to it:

  uint8_t myfield2[stream];

It is ambiguous whether "stream" should refer to the "stream" in the
local scope, or the toplevel "stream" dynamic scope as in:

(7.3.2 Static and Dynamic Scopes from CTF spec)

"The dynamic scope prefixes are thus:

 - Trace Environment: <env. >,
 - Trace Packet Header: <trace.packet.header. >,
 - Stream Packet Context: <stream.packet.context. >,
 - Event Header: <stream.event.header. >,
 - Stream Event Context: <stream.event.context. >,
 - Event Context: <event.context. >,
 - Event Payload: <event.fields. >."

This applies to both sequence and variant types, which both need to
refer to another field. The sequence for the length, the variant for the
"choice" selection (an enumeration).

The ideas I pondered were either to change the grammar late in the game
and prefix a dot (".") before any toplevel dynamic scope, or make the
LTTng tracer preprend an underscore before each event field, which gets
removed by the trace analyzer.

I'm not particularly proud of this hack though. Comments are welcome.

Thanks,

Mathieu

-- 
Mathieu Desnoyers
Operating System Efficiency R&D Consultant
EfficiOS Inc.
http://www.efficios.com

[lttng-dev] Writing a babeltrace output plugin

[Alexandre Montplaisir]

On 12-04-17 11:05 AM, Mathieu Desnoyers wrote:
> [...]
> The ideas I pondered were either to change the grammar late in the
> game and prefix a dot (".") before any toplevel dynamic scope, or make
> the LTTng tracer preprend an underscore before each event field, which
> gets removed by the trace analyzer. I'm not particularly proud of this
> hack though. Comments are welcome. 

Matthew and I have discussed it before, and adding an underscore in
front is fine. However, we think this should be specified in the CTF
spec, and should not be optional. This makes writing a parser much
easier, since you systematically remove the first character, and it
avoids having to parse the first character for every single field name.

For example:
developer adds a tracepoint, with a field called "field1"
tracer adds the underscore when writing the CTF metadata, "_field1"
when reading a trace, the parser removes the "_"
Viewers and readers (Babeltrace, Eclipse, etc.) only see the "field1" part.

That way users never see the underscore itself.

Thoughts?

-- 
Alexandre Montplaisir
DORSAL lab,
?cole Polytechnique de Montr?al

[lttng-dev] Writing a babeltrace output plugin

[Matthew Khouzam]

On 12-04-17 11:30 AM, Alexandre Montplaisir wrote:
> On 12-04-17 11:05 AM, Mathieu Desnoyers wrote:
>> [...]
>> The ideas I pondered were either to change the grammar late in the
>> game and prefix a dot (".") before any toplevel dynamic scope, or make
>> the LTTng tracer preprend an underscore before each event field, which
>> gets removed by the trace analyzer. I'm not particularly proud of this
>> hack though. Comments are welcome. 
> Matthew and I have discussed it before, and adding an underscore in
> front is fine. However, we think this should be specified in the CTF
> spec, and should not be optional. This makes writing a parser much
> easier, since you systematically remove the first character, and it
> avoids having to parse the first character for every single field name.
>
> For example:
> developer adds a tracepoint, with a field called "field1"
> tracer adds the underscore when writing the CTF metadata, "_field1"
> when reading a trace, the parser removes the "_"
> Viewers and readers (Babeltrace, Eclipse, etc.) only see the "field1" part.
>
> That way users never see the underscore itself.
>
> Thoughts?
>
I would like to say that part of the logic behind Alex and I's
perspective of the problem is that this seems very similar to name
mangling in C/C++. It's not up to the compiler to say: I think I'll add
__lcGfield16Fpcpkc_0_ to this function name. It is in the spec. Of
course, ctf needs to be simpler. My concerns are giving an unclear
message to the user: why does seek field and _field work. what if I have
worked at a Redmond company and adds _ all my fields already, then
_field and __field will work.

[lttng-dev] Writing a babeltrace output plugin

[Mathieu Desnoyers]

* Alexandre Montplaisir (alexandre.montplaisir at polymtl.ca) wrote:
> On 12-04-17 11:05 AM, Mathieu Desnoyers wrote:
> > [...]
> > The ideas I pondered were either to change the grammar late in the
> > game and prefix a dot (".") before any toplevel dynamic scope, or make
> > the LTTng tracer preprend an underscore before each event field, which
> > gets removed by the trace analyzer. I'm not particularly proud of this
> > hack though. Comments are welcome. 
> 
> Matthew and I have discussed it before, and adding an underscore in
> front is fine. However, we think this should be specified in the CTF
> spec, and should not be optional. This makes writing a parser much
> easier, since you systematically remove the first character, and it
> avoids having to parse the first character for every single field name.
> 
> For example:
> developer adds a tracepoint, with a field called "field1"
> tracer adds the underscore when writing the CTF metadata, "_field1"
> when reading a trace, the parser removes the "_"
> Viewers and readers (Babeltrace, Eclipse, etc.) only see the "field1" part.
> 
> That way users never see the underscore itself.
> 
> Thoughts?

We also have to consider the case where CTF metadata is generated by
hand for DSP/bare metal tracing, and not dynamically by a tracer. It
would be cumbersome in that case to require the underscore prefix all
the time, given that the person who writes the field names can choose
them so they do not clash with event, traec, stream, env.

Mathieu

-- 
Mathieu Desnoyers
Operating System Efficiency R&D Consultant
EfficiOS Inc.
http://www.efficios.com

[lttng-dev] Babeltrace ctf-text duplicated error message

[Mathieu Desnoyers]

* Oestman, Fredrik (Fredrik_Oestman at mentor.com) wrote:
> Hello,
> 
> 
> I've found what appears to be a (very minor) bug in babeltrace, see patch below.

Merged, thanks!

Mathieu

> 
> 
> Cheers,
> 
> Fredrik ?stman
> 
> 
> diff --git a/formats/ctf-text/ctf-text.c b/formats/ctf-text/ctf-text.c
> index 273c9b7..fa0cb4d 100644
> --- a/formats/ctf-text/ctf-text.c
> +++ b/formats/ctf-text/ctf-text.c
> @@ -236,8 +236,8 @@ int ctf_text_write_event(struct stream_pos *ppos, struct ctf_stream *stream)
>  		return -EINVAL;
>  	}
>  	event_class = g_ptr_array_index(stream_class->events_by_id, id);
> -	if (!event) {
> -		fprintf(stderr, "[error] Event id %" PRIu64 " is unknown.\n", id);
> +	if (!event_class) {
> +		fprintf(stderr, "[error] Event class id %" PRIu64 " is unknown.\n", id);
>  		return -EINVAL;
>  	}
> 
> 
> _______________________________________________
> lttng-dev mailing list
> lttng-dev at lists.lttng.org
> http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

-- 
Mathieu Desnoyers
Operating System Efficiency R&D Consultant
EfficiOS Inc.
http://www.efficios.com