[lttng-dev] sdt.h tracepoints with unicode data and/or structs

[mathieu.desnoyers@efficios.com (Mathieu Desnoyers)]

----- On Sep 22, 2016, at 8:56 AM, Milian Wolff milian.wolff at kdab.com wrote:

> On Wednesday, September 21, 2016 5:22:35 PM CEST Mathieu Desnoyers wrote:
>> ----- On Sep 20, 2016, at 7:22 PM, Philippe Proulx eeppeliteloop at gmail.com
> wrote:
>> >>> >> > Can I pass UTF16 strings? Do they need to be null-terminated?
>> >>> >> 
>> >>> >> You should convert them to UTF8.
>> >>> >> 
>> >>> >> A ctf_string() needs to be null-terminated. A ctf_sequence_text() is
>> >>> >> not required to be null-terminated. See
>> >>> >> http://lttng.org/docs/#doc-liblttng-ust-tp-fields
>> >>> > 
>> >>> > This sounds like tracing would then incur a huge runtime overhead,
>> >>> > when I
>> >>> > need to convert all my UTF-16 strings to UTF-8. How does one deal with
>> >>> > that?
>> >>> 
>> >>> Are you concerned about the runtime overhead when tracing is disabled,
>> >>> or
>> >>> enabled ? It would be good to gather some metrics on the overhead of
>> >>> this
>> >>> conversion.
>> >> 
>> >> I'm mostly concerned about the overhead when tracing is disabled.
>> >> Ideally, I would like to see these tracepoints unconditionally compiled
>> >> into Qt. But if the overhead is noticeable when they are not in use, I
>> >> may have a hard time achieving this goal. Instead, one will then
>> >> probably need to recompile Qt with the tracepoints enabled.
>> > 
>> > As Mathieu wrote, the overhead of disabled LTTng-UST tracepoints is pretty
>> > much unnoticeable. You can use the tracepoint_enabled() and
>> > do_tracepoint()
>> > macros to perform some computation, conversions, and prepare stuff
>> > specifically for the payload of the event.
>> > 
>> >> When in use, the overhead of the tracepoints should also be minimal.
>> >> Allocating memory for a temporary UTF-16 to UTF-8 conversion alone has a
>> >> large overhead, and then converting the data to UTF-8 also adds on top
>> >> of that. Thus, if at all possible, I would like to prevent that.
>> > 
>> > There's no way to support UTF-16 as of the current versions of LTTng and
>> > CTF. Even if you find a way to store the string as is, for example using a
>> > sequence of bytes, the existing CTF viewers and analyzers won't recognize
>> > the field as a string.
>> > 
>> > However we could think about a way to support UTF-16 in the future, and
>> > possibly other Unicode encodings too.
>> > 
>> >> Similarly, I am looking for a way to put URLs into tracepoints, and
>> >> converting a QUrl to string data is far from cheap.
>> > 
>> > Yes, I see that this code is executed:
>> > <https://github.com/qt/qtbase/blob/dev/src/corelib/io/qurl.cpp#L3279>.
>> > 
>> > You could allocate one tracepoint field for each individual attribute of
>> > the QUrl object, for example the scheme, the path, the query string, the
>> > fragment, etc. But I guess you'd still have the UTF-16 issue.
>> > 
>> >> At this point, I don't see a way around only enabling tracepoints
>> >> optionally. Independent of the mechanism in use for the actual
>> >> tracepoints (i.e. lttng-ust or sdt). The conditional to check whether
>> >> tracing is enabled may not be too bad. But then in the conditional it
>> >> looks like $some code will be required to massage the data into a form
>> >> that the tracepoints accept them. This increase in code size negatively
>> >> influences code caches and I don't see any way around that.
>> > 
>> > I leave this part for Mathieu ;-).
>> 
>> This "extra code" can be implemented within the tracepoint provider,
>> which is a cache cold function, not used at all when tracing is disabled.
> 
> This sounds excellent. Can you tell me how? Could you maybe add an example to
> lttng-ust. Also note how http://lttng.org/man/3/lttng-ust/v2.7 says:
> 
>	if (tracepoint_enabled(ust_tests_hello, tptest)) {
>		/* prepare arguments */
>		do_tracepoint(ust_tests_hello, tptest, i, netint, values,
>			text, strlen(text), dbl, flt);
>	}
> 
> If I understood you correctly, then I could do something like
> 
>	if (tracepoint_enabled(ust_tests_hello, tptest)) {
>		/* don't prepare arguments */
>		do_tracepoint(ust_tests_hello, my_complex_data);
>	}
> 
> And then have the "prepare" code somewhere in my TRACEPOINT_EVENT?
> 

Yes. Both approaches can be used.

Note that the second approach you refer to is the same as using a
plain tracepoint() macro and doing the preparation within
the TRACEPOINT_EVENT() macro.

The preparation within TRACEPOINT_EVENT() can currently only be
done as expression evaluation in the TP_FIELDS. LTTng-modules has
more flexibility in that respect, but not lttng-ust yet.

A patch contributing such example to lttng-ust would be welcome :)

Thanks,

Mathieu


> Thanks
> --
> Milian Wolff | milian.wolff at kdab.com | Software Engineer
> KDAB (Deutschland) GmbH&Co KG, a KDAB Group company
> Tel: +49-30-521325470
> KDAB - The Qt Experts

-- 
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com