From mboxrd@z Thu Jan 1 00:00:00 1970 From: jiayingz@google.com (Jiaying Zhang) Date: Mon, 6 Oct 2008 10:55:55 -0700 Subject: [ltt-dev] LTTng specialized probes In-Reply-To: <20081006141113.GE1808@Krystal> References: <5df78e1d0809231814i4b9b98eeyfb9746e5dbb9eb72@mail.gmail.com> <20080924072503.GA15570@bolzano.suse.de> <532480950809240032t644448f7lc4fdc0dffca69b9@mail.gmail.com> <20081006141113.GE1808@Krystal> Message-ID: <5df78e1d0810061055q1f02a1f0uac1cf0981fba8f40@mail.gmail.com> On Mon, Oct 6, 2008 at 7:11 AM, Mathieu Desnoyers < compudj at krystal.dyndns.org> wrote: > Hi, > > I'm currently working towards getting LTTng in shape for what is > required for mainline. I got the "TLB-less" buffers and splice() > working last week. I then did some performance testing on the flight > recorder mode and noticed an optimization that's really worth doing : > > LTTng "ltt-serialize.c", which parses the format strings and formats > data into the trace buffers takes a lot of CPU time. I tried only > keeping the size calculation (first pass on the format string) and > disabling the real data write and basically got something like : > > (default LTTng instrumentation, very approximate numbers) > > tbench no tracing : ~1900MB/s > Markers enabled : ~1800MB/s > with size calculation : ~1400MB/s > size calc + data write : ~950MB/s Thanks a lot for sharing these numbers! Looks like we should use special probe functions for high-frequency tracing events. Also, do you know why enabling markers adds so much overhead? Jiaying > > I then remembered I've done ltt-serialize in such a way that it can be > easily overridden by per-format string specialized callbacks. > > Therefore, it would be worthwhile to create such specialized serializers > so the common cases can be made much faster. I think it will have a very > significant impact on performance. > > It's simply a matter of creating a new .c kernel module in ltt/ and to > create structures similar to : > > ltt-serialize.c : > > struct ltt_available_probe default_probe = { > .name = "default", > .format = NULL, > .probe_func = ltt_vtrace, > .callbacks[0] = ltt_serialize_data, > }; > > Give it a non-null format string (just giving the types expected by the > callback), a good name, and a callback function, which implements the > specialized serialization. Note that kernel/marker.c currently expects > the format string to match exactly the marker format string, including > the type names, which should be changed. The type verification should > only check that the %X parameters are the same (and that there are the > same amount of arguments expected). > > That should not be hard, but it's not what I plan to focus on next. > Anyone is willing to work on this ? > > Mathieu > > -- > Mathieu Desnoyers > OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F BA06 3F25 A8FE 3BAE 9A68 > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mboxrd@z Thu Jan 1 00:00:00 1970 From: jiayingz@google.com (Jiaying Zhang) Date: Mon, 6 Oct 2008 10:55:55 -0700 Subject: [ltt-dev] LTTng specialized probes In-Reply-To: <20081006141113.GE1808@Krystal> References: <5df78e1d0809231814i4b9b98eeyfb9746e5dbb9eb72@mail.gmail.com> <20080924072503.GA15570@bolzano.suse.de> <532480950809240032t644448f7lc4fdc0dffca69b9@mail.gmail.com> <20081006141113.GE1808@Krystal> Message-ID: <5df78e1d0810061055q1f02a1f0uac1cf0981fba8f40@mail.gmail.com> Message-ID: <20081006175555.bVOivAbba5MRhDOOlUL7Loeozs0urcHjd8Sl3HYYTTw@z> On Mon, Oct 6, 2008 at 7:11 AM, Mathieu Desnoyers < compudj at krystal.dyndns.org> wrote: > Hi, > > I'm currently working towards getting LTTng in shape for what is > required for mainline. I got the "TLB-less" buffers and splice() > working last week. I then did some performance testing on the flight > recorder mode and noticed an optimization that's really worth doing : > > LTTng "ltt-serialize.c", which parses the format strings and formats > data into the trace buffers takes a lot of CPU time. I tried only > keeping the size calculation (first pass on the format string) and > disabling the real data write and basically got something like : > > (default LTTng instrumentation, very approximate numbers) > > tbench no tracing : ~1900MB/s > Markers enabled : ~1800MB/s > with size calculation : ~1400MB/s > size calc + data write : ~950MB/s Thanks a lot for sharing these numbers! Looks like we should use special probe functions for high-frequency tracing events. Also, do you know why enabling markers adds so much overhead? Jiaying > > I then remembered I've done ltt-serialize in such a way that it can be > easily overridden by per-format string specialized callbacks. > > Therefore, it would be worthwhile to create such specialized serializers > so the common cases can be made much faster. I think it will have a very > significant impact on performance. > > It's simply a matter of creating a new .c kernel module in ltt/ and to > create structures similar to : > > ltt-serialize.c : > > struct ltt_available_probe default_probe = { > .name = "default", > .format = NULL, > .probe_func = ltt_vtrace, > .callbacks[0] = ltt_serialize_data, > }; > > Give it a non-null format string (just giving the types expected by the > callback), a good name, and a callback function, which implements the > specialized serialization. Note that kernel/marker.c currently expects > the format string to match exactly the marker format string, including > the type names, which should be changed. The type verification should > only check that the %X parameters are the same (and that there are the > same amount of arguments expected). > > That should not be hard, but it's not what I plan to focus on next. > Anyone is willing to work on this ? > > Mathieu > > -- > Mathieu Desnoyers > OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F BA06 3F25 A8FE 3BAE 9A68 > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mboxrd@z Thu Jan 1 00:00:00 1970 From: jiayingz@google.com (Jiaying Zhang) Date: Mon, 6 Oct 2008 10:55:55 -0700 Subject: [ltt-dev] LTTng specialized probes In-Reply-To: <20081006141113.GE1808@Krystal> References: <5df78e1d0809231814i4b9b98eeyfb9746e5dbb9eb72@mail.gmail.com> <20080924072503.GA15570@bolzano.suse.de> <532480950809240032t644448f7lc4fdc0dffca69b9@mail.gmail.com> <20081006141113.GE1808@Krystal> Message-ID: <5df78e1d0810061055q1f02a1f0uac1cf0981fba8f40@mail.gmail.com> Message-ID: <20081006175555.io0ED9aXMwV9rVPBCVrZHVHBpQrAe_dfh7QCcysXfbs@z> On Mon, Oct 6, 2008 at 7:11 AM, Mathieu Desnoyers < compudj at krystal.dyndns.org> wrote: > Hi, > > I'm currently working towards getting LTTng in shape for what is > required for mainline. I got the "TLB-less" buffers and splice() > working last week. I then did some performance testing on the flight > recorder mode and noticed an optimization that's really worth doing : > > LTTng "ltt-serialize.c", which parses the format strings and formats > data into the trace buffers takes a lot of CPU time. I tried only > keeping the size calculation (first pass on the format string) and > disabling the real data write and basically got something like : > > (default LTTng instrumentation, very approximate numbers) > > tbench no tracing : ~1900MB/s > Markers enabled : ~1800MB/s > with size calculation : ~1400MB/s > size calc + data write : ~950MB/s Thanks a lot for sharing these numbers! Looks like we should use special probe functions for high-frequency tracing events. Also, do you know why enabling markers adds so much overhead? Jiaying > > I then remembered I've done ltt-serialize in such a way that it can be > easily overridden by per-format string specialized callbacks. > > Therefore, it would be worthwhile to create such specialized serializers > so the common cases can be made much faster. I think it will have a very > significant impact on performance. > > It's simply a matter of creating a new .c kernel module in ltt/ and to > create structures similar to : > > ltt-serialize.c : > > struct ltt_available_probe default_probe = { > .name = "default", > .format = NULL, > .probe_func = ltt_vtrace, > .callbacks[0] = ltt_serialize_data, > }; > > Give it a non-null format string (just giving the types expected by the > callback), a good name, and a callback function, which implements the > specialized serialization. Note that kernel/marker.c currently expects > the format string to match exactly the marker format string, including > the type names, which should be changed. The type verification should > only check that the %X parameters are the same (and that there are the > same amount of arguments expected). > > That should not be hard, but it's not what I plan to focus on next. > Anyone is willing to work on this ? > > Mathieu > > -- > Mathieu Desnoyers > OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F BA06 3F25 A8FE 3BAE 9A68 > -------------- next part -------------- An HTML attachment was scrubbed... URL: