[ltt-dev] [UST] BUG Ubuntu <= Karmic

[ltt-dev] [UST] BUG Ubuntu <= Karmic

[David Goulet]

Hi everyone,

A very _major_ bug, to say the least, was discover this afternoon in 
Ubuntu Karmic and below. The clock_gettime and gettimeofday function are 
syscalls and not VDSO as it suppose to be.

This makes UST go impressively slower because at each tracing event, you 
got a syscall in the fast path. We got at 2 times factor (per event 
speed) on Karmic and 5 times on Hardy.

We are currently waiting for feedback from Debian and Ubuntu to explain 
this changes in the glibc API. (If anybody knows why, please feel free 
to explain it on this list).

Here is a simple test to see if clock_gettime is in fact a syscall on 
your distribution (we've only tested Ubuntu distro) :

$ vim test.c

#include <sys/time.h>
#include <time.h>

int main(int argc, char **argv) {
	struct timespec ts;
	clock_gettime(CLOCK_MONOTONIC, &ts);
	return 0;
}

$ gcc -lrt test.c -o test
$ strace ./test

If you see this line in the strace output :
clock_gettime(CLOCK_MONOTONIC, ...)

it's a problem.

Thanks to all
-- 
David Goulet
LTTng project, DORSAL Lab.

PGP/GPG : 1024D/16BD8563
BE3C 672B 9331 9796 291A  14C6 4AF7 C14B 16BD 8563

[ltt-dev] [UST] BUG Ubuntu <= Karmic

[David Goulet]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

After some talks in the metro and testing it here at home, we found the bad guy
in all this mess... it is .... KVM :D

We have no idea yet why but we will look into it.

Side note to my self : Test on different real host ;)

Thanks
David

On 10-09-16 05:59 PM, David Goulet wrote:
> Hi everyone,
> 
> A very _major_ bug, to say the least, was discover this afternoon in
> Ubuntu Karmic and below. The clock_gettime and gettimeofday function are
> syscalls and not VDSO as it suppose to be.
> 
> This makes UST go impressively slower because at each tracing event, you
> got a syscall in the fast path. We got at 2 times factor (per event
> speed) on Karmic and 5 times on Hardy.
> 
> We are currently waiting for feedback from Debian and Ubuntu to explain
> this changes in the glibc API. (If anybody knows why, please feel free
> to explain it on this list).
> 
> Here is a simple test to see if clock_gettime is in fact a syscall on
> your distribution (we've only tested Ubuntu distro) :
> 
> $ vim test.c
> 
> #include <sys/time.h>
> #include <time.h>
> 
> int main(int argc, char **argv) {
>     struct timespec ts;
>     clock_gettime(CLOCK_MONOTONIC, &ts);
>     return 0;
> }
> 
> $ gcc -lrt test.c -o test
> $ strace ./test
> 
> If you see this line in the strace output :
> clock_gettime(CLOCK_MONOTONIC, ...)
> 
> it's a problem.
> 
> Thanks to all

- -- 
David Goulet
LTTng project, DORSAL Lab.

1024D/16BD8563
BE3C 672B 9331 9796 291A  14C6 4AF7 C14B 16BD 8563
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)

iEYEARECAAYFAkySoR8ACgkQSvfBSxa9hWPSlwCeJpzliMvkiJ/26rCqF7SjJouv
bD4AoIvUCOX+Jqfq/q12lvrPaDrptwV1
=Hwbq
-----END PGP SIGNATURE-----

[ltt-dev] [UST] BUG Ubuntu <= Karmic

[Mathieu Desnoyers]

* David Goulet (david.goulet at polymtl.ca) wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> After some talks in the metro and testing it here at home, we found the bad guy
> in all this mess... it is .... KVM :D

Ah, yes. This is why we need a strategy for a userland trace clock that
takes into account:

- The host kernel
- The guest kernel
- Userspace

Which is transparently available across all three.

Mathieu


> 
> We have no idea yet why but we will look into it.
> 
> Side note to my self : Test on different real host ;)
> 
> Thanks
> David
> 
> On 10-09-16 05:59 PM, David Goulet wrote:
> > Hi everyone,
> > 
> > A very _major_ bug, to say the least, was discover this afternoon in
> > Ubuntu Karmic and below. The clock_gettime and gettimeofday function are
> > syscalls and not VDSO as it suppose to be.
> > 
> > This makes UST go impressively slower because at each tracing event, you
> > got a syscall in the fast path. We got at 2 times factor (per event
> > speed) on Karmic and 5 times on Hardy.
> > 
> > We are currently waiting for feedback from Debian and Ubuntu to explain
> > this changes in the glibc API. (If anybody knows why, please feel free
> > to explain it on this list).
> > 
> > Here is a simple test to see if clock_gettime is in fact a syscall on
> > your distribution (we've only tested Ubuntu distro) :
> > 
> > $ vim test.c
> > 
> > #include <sys/time.h>
> > #include <time.h>
> > 
> > int main(int argc, char **argv) {
> >     struct timespec ts;
> >     clock_gettime(CLOCK_MONOTONIC, &ts);
> >     return 0;
> > }
> > 
> > $ gcc -lrt test.c -o test
> > $ strace ./test
> > 
> > If you see this line in the strace output :
> > clock_gettime(CLOCK_MONOTONIC, ...)
> > 
> > it's a problem.
> > 
> > Thanks to all
> 
> - -- 
> David Goulet
> LTTng project, DORSAL Lab.
> 
> 1024D/16BD8563
> BE3C 672B 9331 9796 291A  14C6 4AF7 C14B 16BD 8563
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.9 (GNU/Linux)
> 
> iEYEARECAAYFAkySoR8ACgkQSvfBSxa9hWPSlwCeJpzliMvkiJ/26rCqF7SjJouv
> bD4AoIvUCOX+Jqfq/q12lvrPaDrptwV1
> =Hwbq
> -----END PGP SIGNATURE-----
> 
> _______________________________________________
> ltt-dev mailing list
> ltt-dev at lists.casi.polymtl.ca
> http://lists.casi.polymtl.ca/cgi-bin/mailman/listinfo/ltt-dev
> 

-- 
Mathieu Desnoyers
Operating System Efficiency R&D Consultant
EfficiOS Inc.
http://www.efficios.com