[ltt-dev] LTTng-UST vs SystemTap userspace tracing benchmarks

[fche@redhat.com (Frank Ch. Eigler)]

Julien Desfossez <julien.desfossez at polymtl.ca> writes:

> LTTng-UST vs SystemTap userspace tracing benchmarks

Thank you.

> [...]  For flight recorder tracing, UST is 289 times faster than
> SystemTap on an 8-core system with a LTTng kernel and 279 times with
> a vanilla+utrace kernel.

This is not that surprising, considering how the two tools work.  UST
does its work in userspace, and is therefore focused on an individual
process's activities.  Systemtap does its work in kernelspace, and can
therefore focus on many different processes and the kernel at the same
time.  This entails some ring transitions.

(One may imagine a future version of systemtap where scripts that
happen to independently probe single processes are executed with a
pure userspace backend, but this is not in our immediate roadmap.)

> SystemTap does not scale for multithreaded applications running on
> multi-core systems.  [...]

We know of at least one kernel problem in this area,
<http://sourceware.org/PR5660>, which may be fixable via core or
utrace or uprobes changes.


> This study proves that LTTng-UST and SystemTap are two tools with a
> complementary purpose.  [...]

Strictly speaking, it shows that their performance differs
dramatically in this sort of microbenchmark.  

Thank you for your data gathering.

- FChE