* [ltt-dev] LTTng specialized probes
2008-10-07 21:28 ` Michael Davidson
@ 2008-10-07 21:28 ` Michael Davidson
2008-10-07 21:28 ` Michael Davidson
2008-10-08 0:07 ` Mathieu Desnoyers
2 siblings, 0 replies; 29+ messages in thread
From: Michael Davidson @ 2008-10-07 21:28 UTC (permalink / raw)
Hi, Mathieu!
Jiaying forwarded this to me and I wanted to try to understand a little
better exactly
what direction you are headed in.
Like Martin, I am (at least) a little confused and since I wasn't involved
in the discussions
in Portland please forgive me if I am going over old ground here.
It seems to me that one of the key issues in getting good performance out of
any kernel
tracing system is that you have to record the data that you are trying to
capture as
quickly as possible. To me that means that during the initial recording of
the data, to
the maximum extent possible, you don't even look at it - you just slam it
straight into
your recording buffer and you are done (this should work well for the
majority of cases
where the data being captured are just scalar values - the more exotic the
data the
more processing it will need).
The internal binary format that is recorded in the kernel buffers is
inherently machine
and kernel specific.
You don't need to even think about putting the data into a canonical format
until you
are reday to export it out of the kernel. (Note that, in one of our likely
use cases we
might run tracing continuously but only extract snapshots of the data from
the kernel
occasionally - so most of the data that we collected would never even need
to be put
into canonical format).
The task of converting the internal binary format to something that can be
exported is
also machine and kernel specific and you need machine and kernel specific
code to
do it (although you can handle a lot of issues by providing some appropriate
meta-data
that gopes along with the trace data and describes its format).
This conversion could be done in the kernel, by a user space program running
on the
same machine, or by a program running elsewhere. While I agree that it is
very
convenient for small scale uses to be able to just do this in the kernel,
once again it
may not be appropriate when it is being used on a large scale with machines
running
real live workloads where CPU cycles are precious. In those cases it is
important that
a tracing solution does not preclude gathering binary trace data from
machines in the
most compact format possible, and doing the conversion to a canonical format
elsewhere where the processing will not impact a live running system.
md
On Tue, Oct 7, 2008 at 11:16 AM, Jiaying Zhang <jiayingz at google.com> wrote:
> Hi Michael,
>
> This email from Mathieu has some interesting performance results
> on the overhead of format passing. It indeed added a lot of overhead.
>
> Jiaying
>
> Forwarded conversation
> Subject: LTTng specialized probes
> ------------------------
>
> From: *Mathieu Desnoyers* <compudj@krystal.dyndns.org>
> Date: Mon, Oct 6, 2008 at 7:11 AM
> To: Michael Rubin <mrubin at google.com>, Jan Blunck <jblunck at suse.de>,
> Jiaying Zhang <jiayingz at google.com>, ltt-dev at lists.casi.polymtl.ca, Martin
> Bligh <mbligh at google.com>
>
>
> Hi,
>
> I'm currently working towards getting LTTng in shape for what is
> required for mainline. I got the "TLB-less" buffers and splice()
> working last week. I then did some performance testing on the flight
> recorder mode and noticed an optimization that's really worth doing :
>
> LTTng "ltt-serialize.c", which parses the format strings and formats
> data into the trace buffers takes a lot of CPU time. I tried only
> keeping the size calculation (first pass on the format string) and
> disabling the real data write and basically got something like :
>
> (default LTTng instrumentation, very approximate numbers)
>
> tbench no tracing : ~1900MB/s
> Markers enabled : ~1800MB/s
> with size calculation : ~1400MB/s
> size calc + data write : ~950MB/s
>
> I then remembered I've done ltt-serialize in such a way that it can be
> easily overridden by per-format string specialized callbacks.
>
> Therefore, it would be worthwhile to create such specialized serializers
> so the common cases can be made much faster. I think it will have a very
> significant impact on performance.
>
> It's simply a matter of creating a new .c kernel module in ltt/ and to
> create structures similar to :
>
> ltt-serialize.c :
>
> struct ltt_available_probe default_probe = {
> .name = "default",
> .format = NULL,
> .probe_func = ltt_vtrace,
> .callbacks[0] = ltt_serialize_data,
> };
>
> Give it a non-null format string (just giving the types expected by the
> callback), a good name, and a callback function, which implements the
> specialized serialization. Note that kernel/marker.c currently expects
> the format string to match exactly the marker format string, including
> the type names, which should be changed. The type verification should
> only check that the %X parameters are the same (and that there are the
> same amount of arguments expected).
>
> That should not be hard, but it's not what I plan to focus on next.
> Anyone is willing to work on this ?
>
> Mathieu
>
> --
> Mathieu Desnoyers
> OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F BA06 3F25 A8FE 3BAE 9A68
>
> ----------
> From: *Martin Bligh* <mbligh@google.com>
> Date: Mon, Oct 6, 2008 at 8:14 AM
> To: Mathieu Desnoyers <compudj at krystal.dyndns.org>
> Cc: Michael Rubin <mrubin at google.com>, Jan Blunck <jblunck at suse.de>,
> Jiaying Zhang <jiayingz at google.com>, ltt-dev at lists.casi.polymtl.ca
>
>
> Question ... It seems that strings are mandatory for markers at the moment.
> I don't see why this is, and it seems significantly less efficient than
> what
> we discussed in Portland recently?
>
> ----------
> From: *Mathieu Desnoyers* <compudj@krystal.dyndns.org>
> Date: Mon, Oct 6, 2008 at 8:26 AM
> To: Martin Bligh <mbligh at google.com>
> Cc: Michael Rubin <mrubin at google.com>, Jan Blunck <jblunck at suse.de>,
> Jiaying Zhang <jiayingz at google.com>, ltt-dev at lists.casi.polymtl.ca
>
>
> The idea is that someone who want to add a new instrumentation site in
> the Linux kernel does not have to write a specialized probe up front.
> The format string parser will take care of writing the typed data into
> the buffers (default behavior), but can still overridden by a
> specialized function which will expect the format string arguments and
> serialize those into the buffers.
>
> About what we discussed in Portland and where Steven is currently going:
> it does not provide any kind of binary standard to export the data
> between different platforms or even from kernel 64-bits kernel to
> 32-bits userland. Steven also cleary states that he doesn't care about
> exporting this data to userspace in binary format. He wants a
> supplementary layer to do this formatting, which I don't think will
> produce the performance results we are looking for. Plus, I think
> feeding the data through the kernel which recorded the information to
> decode it is the wrong approach, especially when the system which
> recorded such information is a small embedded device, where getting the
> data _out_ is already non-trivial. Feeding it back in seems a bit crazy.
>
> Mathieu
> --
>
> ----------
> From: *Martin Bligh* <mbligh@google.com>
> Date: Mon, Oct 6, 2008 at 8:37 AM
> To: Mathieu Desnoyers <compudj at krystal.dyndns.org>
> Cc: Michael Rubin <mrubin at google.com>, Jan Blunck <jblunck at suse.de>,
> Jiaying Zhang <jiayingz at google.com>, ltt-dev at lists.casi.polymtl.ca
>
>
> On Mon, Oct 6, 2008 at 8:26 AM, Mathieu Desnoyers
> OK, it seemed mandatory to me, but if it's not, that's good.
> I know he wants the in-kernel parsing for ease-of-use, and getting things
> upstream ... but it seemed to me that there was nothing in what he was
> doing that made it impossible to get the data in binary form out to
> userspace.
> Exporting the buffers is obviously easy. I was under the impression you
> were
> recording strings in the buffers anyway, in which case I don't see why you
> care, but I might be totally mistaken. Even so, it seems what we'd need
> is just to make sure the buffer headers were exported, plus the decoding
> functions - making C files that will link with both the kernel and into a
> userspace library would be a little tricky, but not impossible?
>
> ----------
> From: *Mathieu Desnoyers* <compudj@krystal.dyndns.org>
> Date: Mon, Oct 6, 2008 at 8:56 AM
> To: Martin Bligh <mbligh at google.com>
> Cc: Michael Rubin <mrubin at google.com>, Jan Blunck <jblunck at suse.de>,
> Jiaying Zhang <jiayingz at google.com>, ltt-dev at lists.casi.polymtl.ca
>
>
> Yes, exporting garbage to userspace is easy too ;) Making sense out of
> it, especially without DWARF info, might be a bit more difficult.
> The LTTng buffer format records those markers format strings
> only once in a "metadata" channel so the mapping
>
> event id <-> marker name <-> format string
>
> can be extracted from the trace. We can therefore encode event size and
> typing in this table and manage to leave that metadata out of the high
> throughput tracing stream. By adding a layer that does not take
> advantage of such indirection, Steven is actually reserving event IDs
> for "internal use" when we could, in many cases, use those bits to put
> the event IDs which map to the marker event table. By separating the
> low-level event header management from the event ID registration
> mechanism, we are aiming at a much less efficient solution.
>
> Also, by limiting the event reservation so events never cross a page
> boundary, we are actually limiting the event size that can be exported
> through such stream to 4kB. To me, 4kB non-contiguous pages should be
> _one_ memory backend to use for the buffers (others being video memory
> which survives hot reboots or linearly addressable buffer allocated at
> boot time), which clearly does not have the same 4kB restrictions. I
> therefore don't see why the higher-level buffer management primitives
> (reserve/commit) should suffer from this specific lower-level buffer
> limitation, especially given we can encapsulate writes so it's easy to
> deal with page-crossing writes (c.f. vmap()-less buffers I posted last
> week).
> Linking 64-bits kernel objects into 32-bits userland executables seems
> messy to me. And this is without considering cross-architecture concerns
> (embedded developers with a small powerpc board but an x86 dev. machine
> might want to look at the trace from a non-ABI compatible architecture).
>
> ----------
> From: *Jiaying Zhang* <jiayingz@google.com>
> Date: Mon, Oct 6, 2008 at 10:55 AM
> To: Mathieu Desnoyers <compudj at krystal.dyndns.org>
> Cc: Michael Rubin <mrubin at google.com>, Jan Blunck <jblunck at suse.de>,
> ltt-dev at lists.casi.polymtl.ca, Martin Bligh <mbligh at google.com>
>
>
>
>
>
> Thanks a lot for sharing these numbers! Looks like we should
> use special probe functions for high-frequency tracing events.
> Also, do you know why enabling markers adds so much overhead?
>
> Jiaying
>
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.casi.polymtl.ca/pipermail/lttng-dev/attachments/20081007/54ef6257/attachment-0002.htm>
^ permalink raw reply [flat|nested] 29+ messages in thread* [ltt-dev] LTTng specialized probes
2008-10-07 21:28 ` Michael Davidson
2008-10-07 21:28 ` Michael Davidson
@ 2008-10-07 21:28 ` Michael Davidson
2008-10-08 0:07 ` Mathieu Desnoyers
2 siblings, 0 replies; 29+ messages in thread
From: Michael Davidson @ 2008-10-07 21:28 UTC (permalink / raw)
Hi, Mathieu!
Jiaying forwarded this to me and I wanted to try to understand a little
better exactly
what direction you are headed in.
Like Martin, I am (at least) a little confused and since I wasn't involved
in the discussions
in Portland please forgive me if I am going over old ground here.
It seems to me that one of the key issues in getting good performance out of
any kernel
tracing system is that you have to record the data that you are trying to
capture as
quickly as possible. To me that means that during the initial recording of
the data, to
the maximum extent possible, you don't even look at it - you just slam it
straight into
your recording buffer and you are done (this should work well for the
majority of cases
where the data being captured are just scalar values - the more exotic the
data the
more processing it will need).
The internal binary format that is recorded in the kernel buffers is
inherently machine
and kernel specific.
You don't need to even think about putting the data into a canonical format
until you
are reday to export it out of the kernel. (Note that, in one of our likely
use cases we
might run tracing continuously but only extract snapshots of the data from
the kernel
occasionally - so most of the data that we collected would never even need
to be put
into canonical format).
The task of converting the internal binary format to something that can be
exported is
also machine and kernel specific and you need machine and kernel specific
code to
do it (although you can handle a lot of issues by providing some appropriate
meta-data
that gopes along with the trace data and describes its format).
This conversion could be done in the kernel, by a user space program running
on the
same machine, or by a program running elsewhere. While I agree that it is
very
convenient for small scale uses to be able to just do this in the kernel,
once again it
may not be appropriate when it is being used on a large scale with machines
running
real live workloads where CPU cycles are precious. In those cases it is
important that
a tracing solution does not preclude gathering binary trace data from
machines in the
most compact format possible, and doing the conversion to a canonical format
elsewhere where the processing will not impact a live running system.
md
On Tue, Oct 7, 2008 at 11:16 AM, Jiaying Zhang <jiayingz at google.com> wrote:
> Hi Michael,
>
> This email from Mathieu has some interesting performance results
> on the overhead of format passing. It indeed added a lot of overhead.
>
> Jiaying
>
> Forwarded conversation
> Subject: LTTng specialized probes
> ------------------------
>
> From: *Mathieu Desnoyers* <compudj@krystal.dyndns.org>
> Date: Mon, Oct 6, 2008 at 7:11 AM
> To: Michael Rubin <mrubin at google.com>, Jan Blunck <jblunck at suse.de>,
> Jiaying Zhang <jiayingz at google.com>, ltt-dev at lists.casi.polymtl.ca, Martin
> Bligh <mbligh at google.com>
>
>
> Hi,
>
> I'm currently working towards getting LTTng in shape for what is
> required for mainline. I got the "TLB-less" buffers and splice()
> working last week. I then did some performance testing on the flight
> recorder mode and noticed an optimization that's really worth doing :
>
> LTTng "ltt-serialize.c", which parses the format strings and formats
> data into the trace buffers takes a lot of CPU time. I tried only
> keeping the size calculation (first pass on the format string) and
> disabling the real data write and basically got something like :
>
> (default LTTng instrumentation, very approximate numbers)
>
> tbench no tracing : ~1900MB/s
> Markers enabled : ~1800MB/s
> with size calculation : ~1400MB/s
> size calc + data write : ~950MB/s
>
> I then remembered I've done ltt-serialize in such a way that it can be
> easily overridden by per-format string specialized callbacks.
>
> Therefore, it would be worthwhile to create such specialized serializers
> so the common cases can be made much faster. I think it will have a very
> significant impact on performance.
>
> It's simply a matter of creating a new .c kernel module in ltt/ and to
> create structures similar to :
>
> ltt-serialize.c :
>
> struct ltt_available_probe default_probe = {
> .name = "default",
> .format = NULL,
> .probe_func = ltt_vtrace,
> .callbacks[0] = ltt_serialize_data,
> };
>
> Give it a non-null format string (just giving the types expected by the
> callback), a good name, and a callback function, which implements the
> specialized serialization. Note that kernel/marker.c currently expects
> the format string to match exactly the marker format string, including
> the type names, which should be changed. The type verification should
> only check that the %X parameters are the same (and that there are the
> same amount of arguments expected).
>
> That should not be hard, but it's not what I plan to focus on next.
> Anyone is willing to work on this ?
>
> Mathieu
>
> --
> Mathieu Desnoyers
> OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F BA06 3F25 A8FE 3BAE 9A68
>
> ----------
> From: *Martin Bligh* <mbligh@google.com>
> Date: Mon, Oct 6, 2008 at 8:14 AM
> To: Mathieu Desnoyers <compudj at krystal.dyndns.org>
> Cc: Michael Rubin <mrubin at google.com>, Jan Blunck <jblunck at suse.de>,
> Jiaying Zhang <jiayingz at google.com>, ltt-dev at lists.casi.polymtl.ca
>
>
> Question ... It seems that strings are mandatory for markers at the moment.
> I don't see why this is, and it seems significantly less efficient than
> what
> we discussed in Portland recently?
>
> ----------
> From: *Mathieu Desnoyers* <compudj@krystal.dyndns.org>
> Date: Mon, Oct 6, 2008 at 8:26 AM
> To: Martin Bligh <mbligh at google.com>
> Cc: Michael Rubin <mrubin at google.com>, Jan Blunck <jblunck at suse.de>,
> Jiaying Zhang <jiayingz at google.com>, ltt-dev at lists.casi.polymtl.ca
>
>
> The idea is that someone who want to add a new instrumentation site in
> the Linux kernel does not have to write a specialized probe up front.
> The format string parser will take care of writing the typed data into
> the buffers (default behavior), but can still overridden by a
> specialized function which will expect the format string arguments and
> serialize those into the buffers.
>
> About what we discussed in Portland and where Steven is currently going:
> it does not provide any kind of binary standard to export the data
> between different platforms or even from kernel 64-bits kernel to
> 32-bits userland. Steven also cleary states that he doesn't care about
> exporting this data to userspace in binary format. He wants a
> supplementary layer to do this formatting, which I don't think will
> produce the performance results we are looking for. Plus, I think
> feeding the data through the kernel which recorded the information to
> decode it is the wrong approach, especially when the system which
> recorded such information is a small embedded device, where getting the
> data _out_ is already non-trivial. Feeding it back in seems a bit crazy.
>
> Mathieu
> --
>
> ----------
> From: *Martin Bligh* <mbligh@google.com>
> Date: Mon, Oct 6, 2008 at 8:37 AM
> To: Mathieu Desnoyers <compudj at krystal.dyndns.org>
> Cc: Michael Rubin <mrubin at google.com>, Jan Blunck <jblunck at suse.de>,
> Jiaying Zhang <jiayingz at google.com>, ltt-dev at lists.casi.polymtl.ca
>
>
> On Mon, Oct 6, 2008 at 8:26 AM, Mathieu Desnoyers
> OK, it seemed mandatory to me, but if it's not, that's good.
> I know he wants the in-kernel parsing for ease-of-use, and getting things
> upstream ... but it seemed to me that there was nothing in what he was
> doing that made it impossible to get the data in binary form out to
> userspace.
> Exporting the buffers is obviously easy. I was under the impression you
> were
> recording strings in the buffers anyway, in which case I don't see why you
> care, but I might be totally mistaken. Even so, it seems what we'd need
> is just to make sure the buffer headers were exported, plus the decoding
> functions - making C files that will link with both the kernel and into a
> userspace library would be a little tricky, but not impossible?
>
> ----------
> From: *Mathieu Desnoyers* <compudj@krystal.dyndns.org>
> Date: Mon, Oct 6, 2008 at 8:56 AM
> To: Martin Bligh <mbligh at google.com>
> Cc: Michael Rubin <mrubin at google.com>, Jan Blunck <jblunck at suse.de>,
> Jiaying Zhang <jiayingz at google.com>, ltt-dev at lists.casi.polymtl.ca
>
>
> Yes, exporting garbage to userspace is easy too ;) Making sense out of
> it, especially without DWARF info, might be a bit more difficult.
> The LTTng buffer format records those markers format strings
> only once in a "metadata" channel so the mapping
>
> event id <-> marker name <-> format string
>
> can be extracted from the trace. We can therefore encode event size and
> typing in this table and manage to leave that metadata out of the high
> throughput tracing stream. By adding a layer that does not take
> advantage of such indirection, Steven is actually reserving event IDs
> for "internal use" when we could, in many cases, use those bits to put
> the event IDs which map to the marker event table. By separating the
> low-level event header management from the event ID registration
> mechanism, we are aiming at a much less efficient solution.
>
> Also, by limiting the event reservation so events never cross a page
> boundary, we are actually limiting the event size that can be exported
> through such stream to 4kB. To me, 4kB non-contiguous pages should be
> _one_ memory backend to use for the buffers (others being video memory
> which survives hot reboots or linearly addressable buffer allocated at
> boot time), which clearly does not have the same 4kB restrictions. I
> therefore don't see why the higher-level buffer management primitives
> (reserve/commit) should suffer from this specific lower-level buffer
> limitation, especially given we can encapsulate writes so it's easy to
> deal with page-crossing writes (c.f. vmap()-less buffers I posted last
> week).
> Linking 64-bits kernel objects into 32-bits userland executables seems
> messy to me. And this is without considering cross-architecture concerns
> (embedded developers with a small powerpc board but an x86 dev. machine
> might want to look at the trace from a non-ABI compatible architecture).
>
> ----------
> From: *Jiaying Zhang* <jiayingz@google.com>
> Date: Mon, Oct 6, 2008 at 10:55 AM
> To: Mathieu Desnoyers <compudj at krystal.dyndns.org>
> Cc: Michael Rubin <mrubin at google.com>, Jan Blunck <jblunck at suse.de>,
> ltt-dev at lists.casi.polymtl.ca, Martin Bligh <mbligh at google.com>
>
>
>
>
>
> Thanks a lot for sharing these numbers! Looks like we should
> use special probe functions for high-frequency tracing events.
> Also, do you know why enabling markers adds so much overhead?
>
> Jiaying
>
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.casi.polymtl.ca/pipermail/ltt-dev/attachments/20081007/54ef6257/attachment.htm>
^ permalink raw reply [flat|nested] 29+ messages in thread* [ltt-dev] LTTng specialized probes
2008-10-07 21:28 ` Michael Davidson
2008-10-07 21:28 ` Michael Davidson
2008-10-07 21:28 ` Michael Davidson
@ 2008-10-08 0:07 ` Mathieu Desnoyers
2008-10-08 15:56 ` Jan Kiszka
2008-10-09 3:04 ` Michael Davidson
2 siblings, 2 replies; 29+ messages in thread
From: Mathieu Desnoyers @ 2008-10-08 0:07 UTC (permalink / raw)
Hi Michael,
* Michael Davidson (md at google.com) wrote:
> Hi, Mathieu!
>
> Jiaying forwarded this to me and I wanted to try to understand a
> little better exactly what direction you are headed in.
>
> Like Martin, I am (at least) a little confused and since I wasn't
> involved in the discussions in Portland please forgive me if I am
> going over old ground here.
>
No problem, I'll try to answer the best I can. Don't hesitate to ask for
clarifications if I am not clear enough.
> It seems to me that one of the key issues in getting good performance
> out of any kernel tracing system is that you have to record the data
> that you are trying to capture as quickly as possible. To me that
> means that during the initial recording of the data, to the maximum
> extent possible, you don't even look at it - you just slam it straight
> into your recording buffer and you are done (this should work well for
> the majority of cases where the data being captured are just scalar
> values - the more exotic the data the more processing it will need).
>
Yes, I agree that the best performance is achieved by having a probe
which already "knows" how much data to save and just "does it", and this
is what we want for high-throughput events.
However, I pursue a supplementary goal in LTTng : I wanted to make it
trivial for kernel developers to add new events in the kernel. This can
be done by declaring a new marker, which has a format string that
permits an automatic vsnprintf-like parsing to save the data into the
trace buffer. The format strings are saved in the trace "metadata"
(only once); they describe the event fields. This provides flexible
and extensible event data. This is by no mean the tracing "fast path".
> The internal binary format that is recorded in the kernel buffers is
> inherently machine and kernel specific.
>
Yes, the binary data I write in the buffers is in the kernel endianness
and follows the machine-specific basic types. However, no compound types
are currently supported.
> You don't need to even think about putting the data into a canonical format
> until you are reday to export it out of the kernel. (Note that, in one
> of our likely use cases we might run tracing continuously but only
> extract snapshots of the data from the kernel occasionally - so most
> of the data that we collected would never even need to be put into
> canonical format).
Yes, one use-case is to keep it within the kernel so it can be later
exported (flight recorder mode). But this mode alone might need to
export the data from an half-crashed kernel where we want to do the
minimum operations to get the data out; e.g. simply copying the data to
a serial link, without any formatting whatsoever. The second use-case
involves saving huge amounts of data to disk ("normal", or continuous,
tracing mode). This is where having a trace buffer representation which
follows the internal machine representation *and* which is
self-described matters, because the most efficient way to get the data
out of the kernel is to write the pages directly to disk, or send them
directly through the network, without any supplementary formatting.
>
> The task of converting the internal binary format to something that
> can be exported is also machine and kernel specific and you need
> machine and kernel specific code to do it (although you can handle a
> lot of issues by providing some appropriate meta-data that gopes along
> with the trace data and describes its format).
By exporting the marker format strings along with the trace in a special
"channel" (this is the metadata), and by exporting the endianness and
type sizes in the trace header, the userspace code which reads the trace
(LTTV) can be (and is) machine-independant. This is very useful in a lot
of use-cases, especially for embedded systems where the traced device
cannot be used for decoding (too little memory, slow link, slow CPU,
half-working kernel...).
Your statement above however applies if one whishes to export C
compound types (structures, unions, arrays) directly into the trace
buffers. Then a specialized decoder would be needed, but hopefully this
will be a specialized corner-case. As long as we leave room for such
decoder, I think nobody will suffer. This specialized decoder does not,
however, need to be kernel or machine-specific. It can expect the data,
e.g. a structure, to follow the C standard regarding alignment.
Therefore, the fields can be accessed in an architecture-independent
manner given we have the endianness and type size information from the
trace header.
>
> This conversion could be done in the kernel, by a user space program
> running on the same machine, or by a program running elsewhere.
You seem to assume that the same ABI the kernel runs in will be
available elsewhere, which might not be true in the embedded field. The
same applies to 64-bits x86 kernel with 32-bits userland. Therefore,
getting data out of the kernel should be done by a standardized ABI;
ideally following the kernel ABI for speed, but more importantly :
self-described. This is what LTTng does.
> While
> I agree that it is very convenient for small scale uses to be able to
> just do this in the kernel, once again it may not be appropriate when
> it is being used on a large scale with machines running real live
> workloads where CPU cycles are precious. In those cases it is
> important that a tracing solution does not preclude gathering binary
> trace data from machines in the most compact format possible, and
> doing the conversion to a canonical format elsewhere where the
> processing will not impact a live running system.
What I propose here is actually an optimization on top of the current
mechanism which writes the data in the traces in a self-described, yet
machine-specific and very compact and efficient binary format. This
optimization will consist of a few specialized callbacks which will
serialize the data of the most common event types in the trace buffers
following the same layout already followed by the fully dynamic (but
slow) serializer. Therefore, the event types which appear in the
high-throughput events will be supported by such high-speed trivial
serializers and we will have both flexibility (it will still be trivial
to add new event types) and speed (if one needs a specific event to be
handled very quickly, he just have to create a new specialized
serializer).
So I think both flexibility and speed are by no way opposed goals and
can be both achieved.
Mathieu
>
> md
>
>
>
> On Tue, Oct 7, 2008 at 11:16 AM, Jiaying Zhang <jiayingz at google.com> wrote:
>
> > Hi Michael,
> >
> > This email from Mathieu has some interesting performance results
> > on the overhead of format passing. It indeed added a lot of overhead.
> >
> > Jiaying
> >
> > Forwarded conversation
> > Subject: LTTng specialized probes
> > ------------------------
> >
> > From: *Mathieu Desnoyers* <compudj@krystal.dyndns.org>
> > Date: Mon, Oct 6, 2008 at 7:11 AM
> > To: Michael Rubin <mrubin at google.com>, Jan Blunck <jblunck at suse.de>,
> > Jiaying Zhang <jiayingz at google.com>, ltt-dev at lists.casi.polymtl.ca, Martin
> > Bligh <mbligh at google.com>
> >
> >
> > Hi,
> >
> > I'm currently working towards getting LTTng in shape for what is
> > required for mainline. I got the "TLB-less" buffers and splice()
> > working last week. I then did some performance testing on the flight
> > recorder mode and noticed an optimization that's really worth doing :
> >
> > LTTng "ltt-serialize.c", which parses the format strings and formats
> > data into the trace buffers takes a lot of CPU time. I tried only
> > keeping the size calculation (first pass on the format string) and
> > disabling the real data write and basically got something like :
> >
> > (default LTTng instrumentation, very approximate numbers)
> >
> > tbench no tracing : ~1900MB/s
> > Markers enabled : ~1800MB/s
> > with size calculation : ~1400MB/s
> > size calc + data write : ~950MB/s
> >
> > I then remembered I've done ltt-serialize in such a way that it can be
> > easily overridden by per-format string specialized callbacks.
> >
> > Therefore, it would be worthwhile to create such specialized serializers
> > so the common cases can be made much faster. I think it will have a very
> > significant impact on performance.
> >
> > It's simply a matter of creating a new .c kernel module in ltt/ and to
> > create structures similar to :
> >
> > ltt-serialize.c :
> >
> > struct ltt_available_probe default_probe = {
> > .name = "default",
> > .format = NULL,
> > .probe_func = ltt_vtrace,
> > .callbacks[0] = ltt_serialize_data,
> > };
> >
> > Give it a non-null format string (just giving the types expected by the
> > callback), a good name, and a callback function, which implements the
> > specialized serialization. Note that kernel/marker.c currently expects
> > the format string to match exactly the marker format string, including
> > the type names, which should be changed. The type verification should
> > only check that the %X parameters are the same (and that there are the
> > same amount of arguments expected).
> >
> > That should not be hard, but it's not what I plan to focus on next.
> > Anyone is willing to work on this ?
> >
> > Mathieu
> >
> > --
> > Mathieu Desnoyers
> > OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F BA06 3F25 A8FE 3BAE 9A68
> >
> > ----------
> > From: *Martin Bligh* <mbligh@google.com>
> > Date: Mon, Oct 6, 2008 at 8:14 AM
> > To: Mathieu Desnoyers <compudj at krystal.dyndns.org>
> > Cc: Michael Rubin <mrubin at google.com>, Jan Blunck <jblunck at suse.de>,
> > Jiaying Zhang <jiayingz at google.com>, ltt-dev at lists.casi.polymtl.ca
> >
> >
> > Question ... It seems that strings are mandatory for markers at the moment.
> > I don't see why this is, and it seems significantly less efficient than
> > what
> > we discussed in Portland recently?
> >
> > ----------
> > From: *Mathieu Desnoyers* <compudj@krystal.dyndns.org>
> > Date: Mon, Oct 6, 2008 at 8:26 AM
> > To: Martin Bligh <mbligh at google.com>
> > Cc: Michael Rubin <mrubin at google.com>, Jan Blunck <jblunck at suse.de>,
> > Jiaying Zhang <jiayingz at google.com>, ltt-dev at lists.casi.polymtl.ca
> >
> >
> > The idea is that someone who want to add a new instrumentation site in
> > the Linux kernel does not have to write a specialized probe up front.
> > The format string parser will take care of writing the typed data into
> > the buffers (default behavior), but can still overridden by a
> > specialized function which will expect the format string arguments and
> > serialize those into the buffers.
> >
> > About what we discussed in Portland and where Steven is currently going:
> > it does not provide any kind of binary standard to export the data
> > between different platforms or even from kernel 64-bits kernel to
> > 32-bits userland. Steven also cleary states that he doesn't care about
> > exporting this data to userspace in binary format. He wants a
> > supplementary layer to do this formatting, which I don't think will
> > produce the performance results we are looking for. Plus, I think
> > feeding the data through the kernel which recorded the information to
> > decode it is the wrong approach, especially when the system which
> > recorded such information is a small embedded device, where getting the
> > data _out_ is already non-trivial. Feeding it back in seems a bit crazy.
> >
> > Mathieu
> > --
> >
> > ----------
> > From: *Martin Bligh* <mbligh@google.com>
> > Date: Mon, Oct 6, 2008 at 8:37 AM
> > To: Mathieu Desnoyers <compudj at krystal.dyndns.org>
> > Cc: Michael Rubin <mrubin at google.com>, Jan Blunck <jblunck at suse.de>,
> > Jiaying Zhang <jiayingz at google.com>, ltt-dev at lists.casi.polymtl.ca
> >
> >
> > On Mon, Oct 6, 2008 at 8:26 AM, Mathieu Desnoyers
> > OK, it seemed mandatory to me, but if it's not, that's good.
> > I know he wants the in-kernel parsing for ease-of-use, and getting things
> > upstream ... but it seemed to me that there was nothing in what he was
> > doing that made it impossible to get the data in binary form out to
> > userspace.
> > Exporting the buffers is obviously easy. I was under the impression you
> > were
> > recording strings in the buffers anyway, in which case I don't see why you
> > care, but I might be totally mistaken. Even so, it seems what we'd need
> > is just to make sure the buffer headers were exported, plus the decoding
> > functions - making C files that will link with both the kernel and into a
> > userspace library would be a little tricky, but not impossible?
> >
> > ----------
> > From: *Mathieu Desnoyers* <compudj@krystal.dyndns.org>
> > Date: Mon, Oct 6, 2008 at 8:56 AM
> > To: Martin Bligh <mbligh at google.com>
> > Cc: Michael Rubin <mrubin at google.com>, Jan Blunck <jblunck at suse.de>,
> > Jiaying Zhang <jiayingz at google.com>, ltt-dev at lists.casi.polymtl.ca
> >
> >
> > Yes, exporting garbage to userspace is easy too ;) Making sense out of
> > it, especially without DWARF info, might be a bit more difficult.
> > The LTTng buffer format records those markers format strings
> > only once in a "metadata" channel so the mapping
> >
> > event id <-> marker name <-> format string
> >
> > can be extracted from the trace. We can therefore encode event size and
> > typing in this table and manage to leave that metadata out of the high
> > throughput tracing stream. By adding a layer that does not take
> > advantage of such indirection, Steven is actually reserving event IDs
> > for "internal use" when we could, in many cases, use those bits to put
> > the event IDs which map to the marker event table. By separating the
> > low-level event header management from the event ID registration
> > mechanism, we are aiming at a much less efficient solution.
> >
> > Also, by limiting the event reservation so events never cross a page
> > boundary, we are actually limiting the event size that can be exported
> > through such stream to 4kB. To me, 4kB non-contiguous pages should be
> > _one_ memory backend to use for the buffers (others being video memory
> > which survives hot reboots or linearly addressable buffer allocated at
> > boot time), which clearly does not have the same 4kB restrictions. I
> > therefore don't see why the higher-level buffer management primitives
> > (reserve/commit) should suffer from this specific lower-level buffer
> > limitation, especially given we can encapsulate writes so it's easy to
> > deal with page-crossing writes (c.f. vmap()-less buffers I posted last
> > week).
> > Linking 64-bits kernel objects into 32-bits userland executables seems
> > messy to me. And this is without considering cross-architecture concerns
> > (embedded developers with a small powerpc board but an x86 dev. machine
> > might want to look at the trace from a non-ABI compatible architecture).
> >
> > ----------
> > From: *Jiaying Zhang* <jiayingz@google.com>
> > Date: Mon, Oct 6, 2008 at 10:55 AM
> > To: Mathieu Desnoyers <compudj at krystal.dyndns.org>
> > Cc: Michael Rubin <mrubin at google.com>, Jan Blunck <jblunck at suse.de>,
> > ltt-dev at lists.casi.polymtl.ca, Martin Bligh <mbligh at google.com>
> >
> >
> >
> >
> >
> > Thanks a lot for sharing these numbers! Looks like we should
> > use special probe functions for high-frequency tracing events.
> > Also, do you know why enabling markers adds so much overhead?
> >
> > Jiaying
> >
> >
> >
> >
> >
--
Mathieu Desnoyers
OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F BA06 3F25 A8FE 3BAE 9A68
^ permalink raw reply [flat|nested] 29+ messages in thread* [ltt-dev] LTTng specialized probes
2008-10-08 0:07 ` Mathieu Desnoyers
@ 2008-10-08 15:56 ` Jan Kiszka
2008-10-08 15:56 ` Jan Kiszka
` (3 more replies)
2008-10-09 3:04 ` Michael Davidson
1 sibling, 4 replies; 29+ messages in thread
From: Jan Kiszka @ 2008-10-08 15:56 UTC (permalink / raw)
Mathieu Desnoyers wrote:
> Hi Michael,
>
> * Michael Davidson (md at google.com) wrote:
>> Hi, Mathieu!
>>
>> Jiaying forwarded this to me and I wanted to try to understand a
>> little better exactly what direction you are headed in.
>>
>> Like Martin, I am (at least) a little confused and since I wasn't
>> involved in the discussions in Portland please forgive me if I am
>> going over old ground here.
>>
>
> No problem, I'll try to answer the best I can. Don't hesitate to ask for
> clarifications if I am not clear enough.
>
>> It seems to me that one of the key issues in getting good performance
>> out of any kernel tracing system is that you have to record the data
>> that you are trying to capture as quickly as possible. To me that
>> means that during the initial recording of the data, to the maximum
>> extent possible, you don't even look at it - you just slam it straight
>> into your recording buffer and you are done (this should work well for
>> the majority of cases where the data being captured are just scalar
>> values - the more exotic the data the more processing it will need).
>>
>
> Yes, I agree that the best performance is achieved by having a probe
> which already "knows" how much data to save and just "does it", and this
> is what we want for high-throughput events.
I only loosely followed the thread and the latest format-string marker
serialization code. So sorry in advance in case I contribute "cold
coffee" now:
Has anyone thought of / tried out some caching mechanism for this task?
I mean, scan the format string once (I don't think it will change during
runtime... :->), save somewhere that it expects n bytes of
to-be-serialized data on the caller's stack and then get away with only
copying those over into the trace buffer on succeeding marker hits?
Just a thought...
Jan
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 257 bytes
Desc: OpenPGP digital signature
URL: <http://lists.casi.polymtl.ca/pipermail/ltt-dev/attachments/20081008/c2cf412d/attachment.pgp>
^ permalink raw reply [flat|nested] 29+ messages in thread* [ltt-dev] LTTng specialized probes
2008-10-08 15:56 ` Jan Kiszka
@ 2008-10-08 15:56 ` Jan Kiszka
2008-10-08 15:56 ` Jan Kiszka
` (2 subsequent siblings)
3 siblings, 0 replies; 29+ messages in thread
From: Jan Kiszka @ 2008-10-08 15:56 UTC (permalink / raw)
Mathieu Desnoyers wrote:
> Hi Michael,
>
> * Michael Davidson (md at google.com) wrote:
>> Hi, Mathieu!
>>
>> Jiaying forwarded this to me and I wanted to try to understand a
>> little better exactly what direction you are headed in.
>>
>> Like Martin, I am (at least) a little confused and since I wasn't
>> involved in the discussions in Portland please forgive me if I am
>> going over old ground here.
>>
>
> No problem, I'll try to answer the best I can. Don't hesitate to ask for
> clarifications if I am not clear enough.
>
>> It seems to me that one of the key issues in getting good performance
>> out of any kernel tracing system is that you have to record the data
>> that you are trying to capture as quickly as possible. To me that
>> means that during the initial recording of the data, to the maximum
>> extent possible, you don't even look at it - you just slam it straight
>> into your recording buffer and you are done (this should work well for
>> the majority of cases where the data being captured are just scalar
>> values - the more exotic the data the more processing it will need).
>>
>
> Yes, I agree that the best performance is achieved by having a probe
> which already "knows" how much data to save and just "does it", and this
> is what we want for high-throughput events.
I only loosely followed the thread and the latest format-string marker
serialization code. So sorry in advance in case I contribute "cold
coffee" now:
Has anyone thought of / tried out some caching mechanism for this task?
I mean, scan the format string once (I don't think it will change during
runtime... :->), save somewhere that it expects n bytes of
to-be-serialized data on the caller's stack and then get away with only
copying those over into the trace buffer on succeeding marker hits?
Just a thought...
Jan
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 257 bytes
Desc: OpenPGP digital signature
URL: <http://lists.casi.polymtl.ca/pipermail/lttng-dev/attachments/20081008/c2cf412d/attachment-0003.pgp>
^ permalink raw reply [flat|nested] 29+ messages in thread* [ltt-dev] LTTng specialized probes
2008-10-08 15:56 ` Jan Kiszka
2008-10-08 15:56 ` Jan Kiszka
@ 2008-10-08 15:56 ` Jan Kiszka
2008-10-08 16:06 ` Martin Bligh
2008-10-09 2:24 ` Mathieu Desnoyers
3 siblings, 0 replies; 29+ messages in thread
From: Jan Kiszka @ 2008-10-08 15:56 UTC (permalink / raw)
Mathieu Desnoyers wrote:
> Hi Michael,
>
> * Michael Davidson (md at google.com) wrote:
>> Hi, Mathieu!
>>
>> Jiaying forwarded this to me and I wanted to try to understand a
>> little better exactly what direction you are headed in.
>>
>> Like Martin, I am (at least) a little confused and since I wasn't
>> involved in the discussions in Portland please forgive me if I am
>> going over old ground here.
>>
>
> No problem, I'll try to answer the best I can. Don't hesitate to ask for
> clarifications if I am not clear enough.
>
>> It seems to me that one of the key issues in getting good performance
>> out of any kernel tracing system is that you have to record the data
>> that you are trying to capture as quickly as possible. To me that
>> means that during the initial recording of the data, to the maximum
>> extent possible, you don't even look at it - you just slam it straight
>> into your recording buffer and you are done (this should work well for
>> the majority of cases where the data being captured are just scalar
>> values - the more exotic the data the more processing it will need).
>>
>
> Yes, I agree that the best performance is achieved by having a probe
> which already "knows" how much data to save and just "does it", and this
> is what we want for high-throughput events.
I only loosely followed the thread and the latest format-string marker
serialization code. So sorry in advance in case I contribute "cold
coffee" now:
Has anyone thought of / tried out some caching mechanism for this task?
I mean, scan the format string once (I don't think it will change during
runtime... :->), save somewhere that it expects n bytes of
to-be-serialized data on the caller's stack and then get away with only
copying those over into the trace buffer on succeeding marker hits?
Just a thought...
Jan
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 257 bytes
Desc: OpenPGP digital signature
URL: <http://lists.casi.polymtl.ca/pipermail/lttng-dev/attachments/20081008/c2cf412d/attachment-0002.pgp>
^ permalink raw reply [flat|nested] 29+ messages in thread
* [ltt-dev] LTTng specialized probes
2008-10-08 15:56 ` Jan Kiszka
2008-10-08 15:56 ` Jan Kiszka
2008-10-08 15:56 ` Jan Kiszka
@ 2008-10-08 16:06 ` Martin Bligh
2008-10-09 2:43 ` Mathieu Desnoyers
2008-10-09 2:24 ` Mathieu Desnoyers
3 siblings, 1 reply; 29+ messages in thread
From: Martin Bligh @ 2008-10-08 16:06 UTC (permalink / raw)
> Has anyone thought of / tried out some caching mechanism for this task?
> I mean, scan the format string once (I don't think it will change during
> runtime... :->), save somewhere that it expects n bytes of
> to-be-serialized data on the caller's stack and then get away with only
> copying those over into the trace buffer on succeeding marker hits?
Is it crazy to think of this as "everyone registers a print function", and
we just provide a default print function that takes a string that people
can use if they are not concerned with efficiency? Possibly with a different
macro header if need be to hide it.
Another random thought ... was talking with Michael and Jiaying here about
the markers, and it seemed one of the performance / complexity issues
was the necessity to call multiple tracers from one marker (disclaimer:
I haven't had the time to look at this in any detail, sorry, so please forgive
if this totally misses something ...). Would it be possible to always just have
one thing attached to the marker - this would be optimal for the common case.
In the event that we do need multiple tracers attached, we could simply make
the one thing attached to the marker a multiplexor that called a list of tracers
(transparently).
I expect this would reduce complexity and speed up the common case, at the
expense of a little performance in the uncommon case. Though all of this
is fairly second-hand, so maybe is not useful ;-( Sorry, buried in
Google-specific
stuff right now.
^ permalink raw reply [flat|nested] 29+ messages in thread
* [ltt-dev] LTTng specialized probes
2008-10-08 16:06 ` Martin Bligh
@ 2008-10-09 2:43 ` Mathieu Desnoyers
0 siblings, 0 replies; 29+ messages in thread
From: Mathieu Desnoyers @ 2008-10-09 2:43 UTC (permalink / raw)
* Martin Bligh (mbligh at google.com) wrote:
> > Has anyone thought of / tried out some caching mechanism for this task?
> > I mean, scan the format string once (I don't think it will change during
> > runtime... :->), save somewhere that it expects n bytes of
> > to-be-serialized data on the caller's stack and then get away with only
> > copying those over into the trace buffer on succeeding marker hits?
>
> Is it crazy to think of this as "everyone registers a print function", and
> we just provide a default print function that takes a string that people
> can use if they are not concerned with efficiency? Possibly with a different
> macro header if need be to hide it.
>
Yes, that's how I see it, just from the opposite end : I first aim to
provide a generic fallback so it's easy to use and leave the ability to
connect a specialized function to serialize the information when
performance matters.
> Another random thought ... was talking with Michael and Jiaying here about
> the markers, and it seemed one of the performance / complexity issues
> was the necessity to call multiple tracers from one marker (disclaimer:
> I haven't had the time to look at this in any detail, sorry, so please forgive
> if this totally misses something ...). Would it be possible to always just have
> one thing attached to the marker - this would be optimal for the common case.
> In the event that we do need multiple tracers attached, we could simply make
> the one thing attached to the marker a multiplexor that called a list of tracers
> (transparently).
This kind of optimization is already in the marker infrastructure : if
there is a single probe connected, a single callback is called and no
iteration is required on the probe function pointer array. This is not
done in the tracepoints however because this involves either another
level of function call (but we don't care in the case of markers because
we already have to do a va_start on the var args, which involves having
a first function level) or to spill mode code at the instrumentation
site. I have tried to keep the tracepoint code very compact at the
kernel instrumentation site by putting small optimized loop which
iterates on every connected marker.
The problem is that as soon as we add special-cases to optimize for
specific single-probe scenario, we have to spill more code at the kernel
instrumentation site, and if we do that, the kernel developers will
strongly oppose.
>
> I expect this would reduce complexity and speed up the common case, at the
> expense of a little performance in the uncommon case. Though all of this
> is fairly second-hand, so maybe is not useful ;-( Sorry, buried in
> Google-specific
> stuff right now.
>
Well, the tradeoff here is mostly to try to keep the instrumentation
that we put in the kernel code as small as possible in terms of
instruction cache hit which is opposed to any kind of tricky
optimization we would like to do to make some common cases faster when
tracing is enabled.
One thing we could do that might help making performance better is this:
We create a DEFINE_MARKER(name, string) and a get_marker_struct(name).
Those will permit to have the exact same thing the current markers give
us (declaring an event name, associate an ID to it and declare its
parameters). However, this would not imply any function call like what
markers currently require. By doing this, we could put the optimized
"type-specific" code which writes the data into the trace buffers
directly in the tracepoint probe and therefore skip two function calls,
which should make performance much better.
Mathieu
--
Mathieu Desnoyers
OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F BA06 3F25 A8FE 3BAE 9A68
^ permalink raw reply [flat|nested] 29+ messages in thread
* [ltt-dev] LTTng specialized probes
2008-10-08 15:56 ` Jan Kiszka
` (2 preceding siblings ...)
2008-10-08 16:06 ` Martin Bligh
@ 2008-10-09 2:24 ` Mathieu Desnoyers
2008-10-09 2:24 ` Mathieu Desnoyers
2008-10-09 2:24 ` Mathieu Desnoyers
3 siblings, 2 replies; 29+ messages in thread
From: Mathieu Desnoyers @ 2008-10-09 2:24 UTC (permalink / raw)
* Jan Kiszka (jan.kiszka at web.de) wrote:
> Mathieu Desnoyers wrote:
> > Hi Michael,
> >
> > * Michael Davidson (md at google.com) wrote:
> >> Hi, Mathieu!
> >>
> >> Jiaying forwarded this to me and I wanted to try to understand a
> >> little better exactly what direction you are headed in.
> >>
> >> Like Martin, I am (at least) a little confused and since I wasn't
> >> involved in the discussions in Portland please forgive me if I am
> >> going over old ground here.
> >>
> >
> > No problem, I'll try to answer the best I can. Don't hesitate to ask for
> > clarifications if I am not clear enough.
> >
> >> It seems to me that one of the key issues in getting good performance
> >> out of any kernel tracing system is that you have to record the data
> >> that you are trying to capture as quickly as possible. To me that
> >> means that during the initial recording of the data, to the maximum
> >> extent possible, you don't even look at it - you just slam it straight
> >> into your recording buffer and you are done (this should work well for
> >> the majority of cases where the data being captured are just scalar
> >> values - the more exotic the data the more processing it will need).
> >>
> >
> > Yes, I agree that the best performance is achieved by having a probe
> > which already "knows" how much data to save and just "does it", and this
> > is what we want for high-throughput events.
>
> I only loosely followed the thread and the latest format-string marker
> serialization code. So sorry in advance in case I contribute "cold
> coffee" now:
>
> Has anyone thought of / tried out some caching mechanism for this task?
> I mean, scan the format string once (I don't think it will change during
> runtime... :->), save somewhere that it expects n bytes of
> to-be-serialized data on the caller's stack and then get away with only
> copying those over into the trace buffer on succeeding marker hits?
>
A few reasons not to do this :
- The data won't always be on the stack in the same layout we want it in
the trace. It is especially true for data smaller than the
architecture pointer size which is aligned on the stack.
- Also, I fear that such caching mechanism will end up being much more
complex, error prone and difficult to maintain than having a simple
format string parser for the slow common-case and specialized code for
the fast case. It's a bit like a JIT, but we use kernel hackers to
generate the code. ;)
Mathieu
> Just a thought...
>
> Jan
>
--
Mathieu Desnoyers
OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F BA06 3F25 A8FE 3BAE 9A68
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
URL: <http://lists.casi.polymtl.ca/pipermail/lttng-dev/attachments/20081008/fae80f7e/attachment-0002.pgp>
^ permalink raw reply [flat|nested] 29+ messages in thread* [ltt-dev] LTTng specialized probes
2008-10-09 2:24 ` Mathieu Desnoyers
@ 2008-10-09 2:24 ` Mathieu Desnoyers
2008-10-09 2:24 ` Mathieu Desnoyers
1 sibling, 0 replies; 29+ messages in thread
From: Mathieu Desnoyers @ 2008-10-09 2:24 UTC (permalink / raw)
* Jan Kiszka (jan.kiszka at web.de) wrote:
> Mathieu Desnoyers wrote:
> > Hi Michael,
> >
> > * Michael Davidson (md at google.com) wrote:
> >> Hi, Mathieu!
> >>
> >> Jiaying forwarded this to me and I wanted to try to understand a
> >> little better exactly what direction you are headed in.
> >>
> >> Like Martin, I am (at least) a little confused and since I wasn't
> >> involved in the discussions in Portland please forgive me if I am
> >> going over old ground here.
> >>
> >
> > No problem, I'll try to answer the best I can. Don't hesitate to ask for
> > clarifications if I am not clear enough.
> >
> >> It seems to me that one of the key issues in getting good performance
> >> out of any kernel tracing system is that you have to record the data
> >> that you are trying to capture as quickly as possible. To me that
> >> means that during the initial recording of the data, to the maximum
> >> extent possible, you don't even look at it - you just slam it straight
> >> into your recording buffer and you are done (this should work well for
> >> the majority of cases where the data being captured are just scalar
> >> values - the more exotic the data the more processing it will need).
> >>
> >
> > Yes, I agree that the best performance is achieved by having a probe
> > which already "knows" how much data to save and just "does it", and this
> > is what we want for high-throughput events.
>
> I only loosely followed the thread and the latest format-string marker
> serialization code. So sorry in advance in case I contribute "cold
> coffee" now:
>
> Has anyone thought of / tried out some caching mechanism for this task?
> I mean, scan the format string once (I don't think it will change during
> runtime... :->), save somewhere that it expects n bytes of
> to-be-serialized data on the caller's stack and then get away with only
> copying those over into the trace buffer on succeeding marker hits?
>
A few reasons not to do this :
- The data won't always be on the stack in the same layout we want it in
the trace. It is especially true for data smaller than the
architecture pointer size which is aligned on the stack.
- Also, I fear that such caching mechanism will end up being much more
complex, error prone and difficult to maintain than having a simple
format string parser for the slow common-case and specialized code for
the fast case. It's a bit like a JIT, but we use kernel hackers to
generate the code. ;)
Mathieu
> Just a thought...
>
> Jan
>
--
Mathieu Desnoyers
OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F BA06 3F25 A8FE 3BAE 9A68
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
URL: <http://lists.casi.polymtl.ca/pipermail/ltt-dev/attachments/20081008/fae80f7e/attachment.pgp>
^ permalink raw reply [flat|nested] 29+ messages in thread
* [ltt-dev] LTTng specialized probes
2008-10-09 2:24 ` Mathieu Desnoyers
2008-10-09 2:24 ` Mathieu Desnoyers
@ 2008-10-09 2:24 ` Mathieu Desnoyers
1 sibling, 0 replies; 29+ messages in thread
From: Mathieu Desnoyers @ 2008-10-09 2:24 UTC (permalink / raw)
* Jan Kiszka (jan.kiszka at web.de) wrote:
> Mathieu Desnoyers wrote:
> > Hi Michael,
> >
> > * Michael Davidson (md at google.com) wrote:
> >> Hi, Mathieu!
> >>
> >> Jiaying forwarded this to me and I wanted to try to understand a
> >> little better exactly what direction you are headed in.
> >>
> >> Like Martin, I am (at least) a little confused and since I wasn't
> >> involved in the discussions in Portland please forgive me if I am
> >> going over old ground here.
> >>
> >
> > No problem, I'll try to answer the best I can. Don't hesitate to ask for
> > clarifications if I am not clear enough.
> >
> >> It seems to me that one of the key issues in getting good performance
> >> out of any kernel tracing system is that you have to record the data
> >> that you are trying to capture as quickly as possible. To me that
> >> means that during the initial recording of the data, to the maximum
> >> extent possible, you don't even look at it - you just slam it straight
> >> into your recording buffer and you are done (this should work well for
> >> the majority of cases where the data being captured are just scalar
> >> values - the more exotic the data the more processing it will need).
> >>
> >
> > Yes, I agree that the best performance is achieved by having a probe
> > which already "knows" how much data to save and just "does it", and this
> > is what we want for high-throughput events.
>
> I only loosely followed the thread and the latest format-string marker
> serialization code. So sorry in advance in case I contribute "cold
> coffee" now:
>
> Has anyone thought of / tried out some caching mechanism for this task?
> I mean, scan the format string once (I don't think it will change during
> runtime... :->), save somewhere that it expects n bytes of
> to-be-serialized data on the caller's stack and then get away with only
> copying those over into the trace buffer on succeeding marker hits?
>
A few reasons not to do this :
- The data won't always be on the stack in the same layout we want it in
the trace. It is especially true for data smaller than the
architecture pointer size which is aligned on the stack.
- Also, I fear that such caching mechanism will end up being much more
complex, error prone and difficult to maintain than having a simple
format string parser for the slow common-case and specialized code for
the fast case. It's a bit like a JIT, but we use kernel hackers to
generate the code. ;)
Mathieu
> Just a thought...
>
> Jan
>
--
Mathieu Desnoyers
OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F BA06 3F25 A8FE 3BAE 9A68
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
URL: <http://lists.casi.polymtl.ca/pipermail/lttng-dev/attachments/20081008/fae80f7e/attachment-0003.pgp>
^ permalink raw reply [flat|nested] 29+ messages in thread
* [ltt-dev] LTTng specialized probes
2008-10-08 0:07 ` Mathieu Desnoyers
2008-10-08 15:56 ` Jan Kiszka
@ 2008-10-09 3:04 ` Michael Davidson
2008-10-09 3:04 ` Michael Davidson
` (2 more replies)
1 sibling, 3 replies; 29+ messages in thread
From: Michael Davidson @ 2008-10-09 3:04 UTC (permalink / raw)
On Tue, Oct 7, 2008 at 5:07 PM, Mathieu Desnoyers <
compudj at krystal.dyndns.org> wrote:
>
> >
> > This conversion could be done in the kernel, by a user space program
> > running on the same machine, or by a program running elsewhere.
>
> You seem to assume that the same ABI the kernel runs in will be
> available elsewhere, which might not be true in the embedded field. The
> same applies to 64-bits x86 kernel with 32-bits userland. Therefore,
> getting data out of the kernel should be done by a standardized ABI;
> ideally following the kernel ABI for speed, but more importantly :
> self-described. This is what LTTng does.
>
I am confused by your response - I think that we are in agreement but may
be using terminology slightly differently.
When I said that the conversion could be done by "a program running
elsewhere"
I was assuming that the metadata which came along with the raw trace data
would be sufficient to make that possible.
In other words:
- the metadata itself must be in a canonical format (ascii perhaps?) or be
sufficiently self describing that it can be interpreted correctly.
- the metadata must completely describe the format of the binary data
including such things as byte order, size and alignment of data types
and the mapping of all binary values into a canonical external format
(ie maps for event numbers to event names, system call numbers to
system call names etc)
Given that information you can write a portable program which can run
anywhere which can do the conversion.
I am not sure what the mechanism for extracting the trace data from
the kernel and getting it into user space has to do with this.
md
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.casi.polymtl.ca/pipermail/ltt-dev/attachments/20081008/34ace8ae/attachment.htm>
^ permalink raw reply [flat|nested] 29+ messages in thread* [ltt-dev] LTTng specialized probes
2008-10-09 3:04 ` Michael Davidson
@ 2008-10-09 3:04 ` Michael Davidson
2008-10-09 3:04 ` Michael Davidson
2008-10-09 15:28 ` Mathieu Desnoyers
2 siblings, 0 replies; 29+ messages in thread
From: Michael Davidson @ 2008-10-09 3:04 UTC (permalink / raw)
On Tue, Oct 7, 2008 at 5:07 PM, Mathieu Desnoyers <
compudj at krystal.dyndns.org> wrote:
>
> >
> > This conversion could be done in the kernel, by a user space program
> > running on the same machine, or by a program running elsewhere.
>
> You seem to assume that the same ABI the kernel runs in will be
> available elsewhere, which might not be true in the embedded field. The
> same applies to 64-bits x86 kernel with 32-bits userland. Therefore,
> getting data out of the kernel should be done by a standardized ABI;
> ideally following the kernel ABI for speed, but more importantly :
> self-described. This is what LTTng does.
>
I am confused by your response - I think that we are in agreement but may
be using terminology slightly differently.
When I said that the conversion could be done by "a program running
elsewhere"
I was assuming that the metadata which came along with the raw trace data
would be sufficient to make that possible.
In other words:
- the metadata itself must be in a canonical format (ascii perhaps?) or be
sufficiently self describing that it can be interpreted correctly.
- the metadata must completely describe the format of the binary data
including such things as byte order, size and alignment of data types
and the mapping of all binary values into a canonical external format
(ie maps for event numbers to event names, system call numbers to
system call names etc)
Given that information you can write a portable program which can run
anywhere which can do the conversion.
I am not sure what the mechanism for extracting the trace data from
the kernel and getting it into user space has to do with this.
md
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.casi.polymtl.ca/pipermail/lttng-dev/attachments/20081008/34ace8ae/attachment-0002.htm>
^ permalink raw reply [flat|nested] 29+ messages in thread
* [ltt-dev] LTTng specialized probes
2008-10-09 3:04 ` Michael Davidson
2008-10-09 3:04 ` Michael Davidson
@ 2008-10-09 3:04 ` Michael Davidson
2008-10-09 15:28 ` Mathieu Desnoyers
2 siblings, 0 replies; 29+ messages in thread
From: Michael Davidson @ 2008-10-09 3:04 UTC (permalink / raw)
On Tue, Oct 7, 2008 at 5:07 PM, Mathieu Desnoyers <
compudj at krystal.dyndns.org> wrote:
>
> >
> > This conversion could be done in the kernel, by a user space program
> > running on the same machine, or by a program running elsewhere.
>
> You seem to assume that the same ABI the kernel runs in will be
> available elsewhere, which might not be true in the embedded field. The
> same applies to 64-bits x86 kernel with 32-bits userland. Therefore,
> getting data out of the kernel should be done by a standardized ABI;
> ideally following the kernel ABI for speed, but more importantly :
> self-described. This is what LTTng does.
>
I am confused by your response - I think that we are in agreement but may
be using terminology slightly differently.
When I said that the conversion could be done by "a program running
elsewhere"
I was assuming that the metadata which came along with the raw trace data
would be sufficient to make that possible.
In other words:
- the metadata itself must be in a canonical format (ascii perhaps?) or be
sufficiently self describing that it can be interpreted correctly.
- the metadata must completely describe the format of the binary data
including such things as byte order, size and alignment of data types
and the mapping of all binary values into a canonical external format
(ie maps for event numbers to event names, system call numbers to
system call names etc)
Given that information you can write a portable program which can run
anywhere which can do the conversion.
I am not sure what the mechanism for extracting the trace data from
the kernel and getting it into user space has to do with this.
md
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.casi.polymtl.ca/pipermail/lttng-dev/attachments/20081008/34ace8ae/attachment-0003.htm>
^ permalink raw reply [flat|nested] 29+ messages in thread
* [ltt-dev] LTTng specialized probes
2008-10-09 3:04 ` Michael Davidson
2008-10-09 3:04 ` Michael Davidson
2008-10-09 3:04 ` Michael Davidson
@ 2008-10-09 15:28 ` Mathieu Desnoyers
2008-10-09 15:39 ` Martin Bligh
2 siblings, 1 reply; 29+ messages in thread
From: Mathieu Desnoyers @ 2008-10-09 15:28 UTC (permalink / raw)
* Michael Davidson (md at google.com) wrote:
> On Tue, Oct 7, 2008 at 5:07 PM, Mathieu Desnoyers <
> compudj at krystal.dyndns.org> wrote:
>
> >
> > >
> > > This conversion could be done in the kernel, by a user space program
> > > running on the same machine, or by a program running elsewhere.
> >
> > You seem to assume that the same ABI the kernel runs in will be
> > available elsewhere, which might not be true in the embedded field. The
> > same applies to 64-bits x86 kernel with 32-bits userland. Therefore,
> > getting data out of the kernel should be done by a standardized ABI;
> > ideally following the kernel ABI for speed, but more importantly :
> > self-described. This is what LTTng does.
> >
>
> I am confused by your response - I think that we are in agreement but may
> be using terminology slightly differently.
>
> When I said that the conversion could be done by "a program running
> elsewhere"
> I was assuming that the metadata which came along with the raw trace data
> would be sufficient to make that possible.
>
> In other words:
>
> - the metadata itself must be in a canonical format (ascii perhaps?) or be
> sufficiently self describing that it can be interpreted correctly.
>
> - the metadata must completely describe the format of the binary data
> including such things as byte order, size and alignment of data types
> and the mapping of all binary values into a canonical external format
> (ie maps for event numbers to event names, system call numbers to
> system call names etc)
>
> Given that information you can write a portable program which can run
> anywhere which can do the conversion.
>
> I am not sure what the mechanism for extracting the trace data from
> the kernel and getting it into user space has to do with this.
>
> md
Those are per-se two distinct topics (getting the data out to userspace
and exporting metadata which permits to parse the output data portably).
Where they become related is :
1 - When we have to consider which data format we use to write the event
data (event-specific payload) into the buffers. Ideally, we would like
that format to be parsable portably and fully described by the metadata.
2 - When we want to consider what is the most compact size for event
records, considering the fact that we have such metadata to describe the
event payload matters. Following discussions on LKML about number of TSC
bits required, and given it would be good to keep the "extended
information" as simple as possible, to me, a header with :
<core event header, aligned on 32 bits>
1-bit opt. ext. TSC (active if a 27-bits TSC overflow is detected)
27-bits TSC
4-bits event ID (ID #7 reserved to specify (optional) extended event ID,
ID #6 reserved to specify both ext. event ID and event
size)
<extended information, optional>
opt. 16-bits event ID
opt. 16-bits event size (if size == 65535 -> has large event size field)
opt. 32-bits large event size (aligned on 32 bits)
opt. 64-bits TSC MSB (aligned on architecture pointer size)
(realign on 32 or 64 bits depending on architecture before the event
payload)
should be enough to look up through the metadata and see what event
payload to expect for a given event ID. Moreover, given we have 6 event
IDs which can fit in the 32-bits header (0 to 5), many uses-cases
(considering per-buffer IDs) will fit within these 6 available numbers.
And that would permit having an optional "event size" field, which can
be enabled on a per-event basis.
However, in the current Unified tracing buffer implementation done by
Steven Rostedt, he reserves those 5-bits for internal event IDs (rather
that putting buffer-specific information in a buffer header) and uses
the bits left for event size (limits the event size to a somewhat low
value). Although I agree that having the event size here is very useful
to cross-check the typing mechanism with the data actually written in
the buffers by the kernel, I think it should be made a "tracer debug"
mode, not a standard required field, because it requires to use bits
that would otherwise be available for event IDs.
Therefore, I think that whether or not there is metadata which describes
the event payload influences the event header and buffer header
decisions. In the implementation currently done by Steven, he ties
the buffering mechanism to the buffer and event headers, but leaves out
the metadata to a "separate" layer, because he only plans to parse the
buffers from within the kernel (he is not interested in exporting them
to userspace). But I think the separation is done at the wrong spot :
event header should be more closely tied to the available metadata and
buffer header should be sufficient to describe the buffer content
without having to reserve any specific event to express buffer-specific
information.
Mathieu
--
Mathieu Desnoyers
OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F BA06 3F25 A8FE 3BAE 9A68
^ permalink raw reply [flat|nested] 29+ messages in thread* [ltt-dev] LTTng specialized probes
2008-10-09 15:28 ` Mathieu Desnoyers
@ 2008-10-09 15:39 ` Martin Bligh
2008-10-09 16:15 ` Mathieu Desnoyers
0 siblings, 1 reply; 29+ messages in thread
From: Martin Bligh @ 2008-10-09 15:39 UTC (permalink / raw)
> <core event header, aligned on 32 bits>
> 1-bit opt. ext. TSC (active if a 27-bits TSC overflow is detected)
> 27-bits TSC
> 4-bits event ID (ID #7 reserved to specify (optional) extended event ID,
> ID #6 reserved to specify both ext. event ID and event
> size)
That's pretty much exactly what ktrace had. You don't need to reserve
a bit for the TSC overflow, that can be an event, which gives you
31 events instead of 16 ?
However, the problem with this is you're no longer separating the
ringbuffer layer (and timing) from the event tracing layer, or at
least not on a byte boundary.
^ permalink raw reply [flat|nested] 29+ messages in thread
* [ltt-dev] LTTng specialized probes
2008-10-09 15:39 ` Martin Bligh
@ 2008-10-09 16:15 ` Mathieu Desnoyers
2008-10-09 16:28 ` Martin Bligh
0 siblings, 1 reply; 29+ messages in thread
From: Mathieu Desnoyers @ 2008-10-09 16:15 UTC (permalink / raw)
* Martin Bligh (mbligh at google.com) wrote:
> > <core event header, aligned on 32 bits>
> > 1-bit opt. ext. TSC (active if a 27-bits TSC overflow is detected)
> > 27-bits TSC
> > 4-bits event ID (ID #7 reserved to specify (optional) extended event ID,
> > ID #6 reserved to specify both ext. event ID and event
> > size)
>
> That's pretty much exactly what ktrace had. You don't need to reserve
> a bit for the TSC overflow, that can be an event, which gives you
> 31 events instead of 16 ?
>
Ah, oops, IDs #7 and #6 should have been IDs #16 and #15 (forgot a bit
here).
> However, the problem with this is you're no longer separating the
> ringbuffer layer (and timing) from the event tracing layer, or at
> least not on a byte boundary.
>
Yes, there is the problem of separation of the two layers, but there is
more than that : if we use a separate event to write the extended TSC
value, we will have to do this when writing an event in a lockless
scheme :
- Do
- read write offset
- Read TSC, compare to per-buffer "last_tsc"
- If 27-bit overflow, write large TSC event
- Do
- read write offset
- read TSC
- compute event size
- while cmpxchg write offset fails
- write per-buffer "last-tsc"
- continue (restart loop)
- compute event size
- while cmpxchg write offset fails
- write per-buffer "last-tsc"
By doing this, we actually have to do 3 TSC reads when we detect an
overflow intead of a single one. Also, we open the window for an
infinite loop if for some weird reason the TSC read becomes slower than
a 27-bits overflow (with virtualization you never know what will bite
you...).
However, I like your idea of using this "ext. TSC" bit as an event bit.
We could do :
(32-bits alignment)
27-bits TSC
5-bits event ID
ID #31 reserved to specify extended event ID
ID #30 reserved to specify both ext. event ID and event size
ID #29 reserved to specify ext. event ID, event size and ext. TSC
<ext.>
16-bits event ID (opt)
16-bits event size (opt) size = 65535 to specify large event size
64-bits TSC (opt) (aligned on sizeof (void *))
32-bits large event size (opt) (aligned on 32-bits)
(event payload aligned on sizeof(void *))
Which lets us put the extended TSC in the same event header by using the
ext. event ID in that case to encode the event ID. This would leave us
29 IDs available (0-28) and would not require any 3-TSC reads algorithm
(which could cause an infinite loop) to deal with overflow.
Mathieu
--
Mathieu Desnoyers
OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F BA06 3F25 A8FE 3BAE 9A68
^ permalink raw reply [flat|nested] 29+ messages in thread* [ltt-dev] LTTng specialized probes
2008-10-09 16:15 ` Mathieu Desnoyers
@ 2008-10-09 16:28 ` Martin Bligh
2008-10-09 16:55 ` Mathieu Desnoyers
0 siblings, 1 reply; 29+ messages in thread
From: Martin Bligh @ 2008-10-09 16:28 UTC (permalink / raw)
> Yes, there is the problem of separation of the two layers, but there is
> more than that : if we use a separate event to write the extended TSC
> value, we will have to do this when writing an event in a lockless
> scheme :
>
> - Do
> - read write offset
> - Read TSC, compare to per-buffer "last_tsc"
> - If 27-bit overflow, write large TSC event
> - Do
> - read write offset
> - read TSC
> - compute event size
> - while cmpxchg write offset fails
> - write per-buffer "last-tsc"
> - continue (restart loop)
> - compute event size
> - while cmpxchg write offset fails
> - write per-buffer "last-tsc"
>
> By doing this, we actually have to do 3 TSC reads when we detect an
> overflow intead of a single one.
OK. TSC read is cheap
> Also, we open the window for an
> infinite loop if for some weird reason the TSC read becomes slower than
> a 27-bits overflow (with virtualization you never know what will bite
> you...).
Umm. if it takes that long to read the TSC, I'd say your machine
is utterly useless. I think you're worrying too much about REALLY
obscure corner cases. Either tracing is not supported on those
platforms, or we shift the TSC eventually by 10 bits right and forgo
any hope of resolution (since your timing is *completely* screwed
at this point anyway.
> However, I like your idea of using this "ext. TSC" bit as an event bit.
> We could do :
>
> (32-bits alignment)
> 27-bits TSC
> 5-bits event ID
> ID #31 reserved to specify extended event ID
> ID #30 reserved to specify both ext. event ID and event size
> ID #29 reserved to specify ext. event ID, event size and ext. TSC
> <ext.>
> 16-bits event ID (opt)
> 16-bits event size (opt) size = 65535 to specify large event size
> 64-bits TSC (opt) (aligned on sizeof (void *))
> 32-bits large event size (opt) (aligned on 32-bits)
> (event payload aligned on sizeof(void *))
>
> Which lets us put the extended TSC in the same event header by using the
> ext. event ID in that case to encode the event ID. This would leave us
> 29 IDs available (0-28) and would not require any 3-TSC reads algorithm
> (which could cause an infinite loop) to deal with overflow.
>
> Mathieu
>
> --
> Mathieu Desnoyers
> OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F BA06 3F25 A8FE 3BAE 9A68
>
^ permalink raw reply [flat|nested] 29+ messages in thread
* [ltt-dev] LTTng specialized probes
2008-10-09 16:28 ` Martin Bligh
@ 2008-10-09 16:55 ` Mathieu Desnoyers
0 siblings, 0 replies; 29+ messages in thread
From: Mathieu Desnoyers @ 2008-10-09 16:55 UTC (permalink / raw)
* Martin Bligh (mbligh at google.com) wrote:
> > Yes, there is the problem of separation of the two layers, but there is
> > more than that : if we use a separate event to write the extended TSC
> > value, we will have to do this when writing an event in a lockless
> > scheme :
> >
> > - Do
> > - read write offset
> > - Read TSC, compare to per-buffer "last_tsc"
> > - If 27-bit overflow, write large TSC event
> > - Do
> > - read write offset
> > - read TSC
> > - compute event size
> > - while cmpxchg write offset fails
> > - write per-buffer "last-tsc"
> > - continue (restart loop)
> > - compute event size
> > - while cmpxchg write offset fails
> > - write per-buffer "last-tsc"
> >
> > By doing this, we actually have to do 3 TSC reads when we detect an
> > overflow intead of a single one.
>
> OK. TSC read is cheap
>
> > Also, we open the window for an
> > infinite loop if for some weird reason the TSC read becomes slower than
> > a 27-bits overflow (with virtualization you never know what will bite
> > you...).
>
> Umm. if it takes that long to read the TSC, I'd say your machine
> is utterly useless. I think you're worrying too much about REALLY
> obscure corner cases. Either tracing is not supported on those
> platforms, or we shift the TSC eventually by 10 bits right and forgo
> any hope of resolution (since your timing is *completely* screwed
> at this point anyway.
>
Ok, if the fact that the algo might loop in some obscure case does not
convince you, this other argument should be considered : by separating
the large TSC event from the actual event which would suffer from TSC
overflow, we are tying the timestamping to space reservation. In a
perfect world, it would never fail, but the world being imperfect, we
might sometimes lose events and have corrupted subbuffers, which implies
that we can actually lose data. It becomes problematic if we lose the
large TSC event but not the event following it. Rather than just having
an event "lost", we end up having completely unreliable timings. This is
why making timestamping depend on space reservation is not such a good
idea.
Putting the large timestamp in the event header itself, like what I
propose below makes sure that if the event which should contain the
large TSC is lost, the following event will have to have a header
containing a large TSC. Note that it does not add any supplementary
space cost to the high-throughput scenario compared to the solution you
propose, because your solution also requires an event ID for the large
timestamp event.
Given it is as compact and more robust, I don't see where the problem
is ?
Mathieu
> > However, I like your idea of using this "ext. TSC" bit as an event bit.
> > We could do :
> >
> > (32-bits alignment)
> > 27-bits TSC
> > 5-bits event ID
> > ID #31 reserved to specify extended event ID
> > ID #30 reserved to specify both ext. event ID and event size
> > ID #29 reserved to specify ext. event ID, event size and ext. TSC
> > <ext.>
> > 16-bits event ID (opt)
> > 16-bits event size (opt) size = 65535 to specify large event size
> > 64-bits TSC (opt) (aligned on sizeof (void *))
> > 32-bits large event size (opt) (aligned on 32-bits)
> > (event payload aligned on sizeof(void *))
> >
> > Which lets us put the extended TSC in the same event header by using the
> > ext. event ID in that case to encode the event ID. This would leave us
> > 29 IDs available (0-28) and would not require any 3-TSC reads algorithm
> > (which could cause an infinite loop) to deal with overflow.
> >
> > Mathieu
> >
> > --
> > Mathieu Desnoyers
> > OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F BA06 3F25 A8FE 3BAE 9A68
> >
>
--
Mathieu Desnoyers
OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F BA06 3F25 A8FE 3BAE 9A68
^ permalink raw reply [flat|nested] 29+ messages in thread