[ltt-dev] SystemTAP support in tracepoints

[ltt-dev] SystemTAP support in tracepoints

[Mathieu Desnoyers]

Hi Josh,

Following our discussion, I started to think about how we could be
support the systemtap breakpoint approach in UST tracepoints. I created
the following prototype that should let us handle a variable number of
arguments. The "args" I get here from the args... parameter would be
taken from the "TP_ARGS()" declaration in the TRACE_EVENT header. The
code below is really just to see if it is doable at all.

Comments are welcome,

Thanks,

Mathieu


#include <stdio.h>

#define BUILD_BUG_ON_ZERO(e) (sizeof(struct { int:-!!(e); }))
#define BUILD_BUG_ON(condition) ((void)BUILD_BUG_ON_ZERO(condition))

#define __compiler_offsetof(a,b) __builtin_offsetof(a,b)

#undef offsetof
#ifdef __compiler_offsetof
#define offsetof(TYPE,MEMBER) __compiler_offsetof(TYPE,MEMBER)
#else
#define offsetof(TYPE, MEMBER) ((size_t) &((TYPE *)0)->MEMBER)
#endif

/* for x86 64 */
#define _ASM_PTR ".quad "

#define A(first, args...)				\
	"g" (first)
#define B(first, args...)				\
	"g" (first), A(args, 0)
#define C(first, args...)				\
	"g" (first), B(args, 0)
#define D(first, args...)				\
	"g" (first), C(args, 0)
#define E(first, args...)				\
	"g" (first), D(args, 0)

#define NARGS(args...)					\
	(sizeof(struct { unsigned char args; }) / sizeof(unsigned char))

#define _TRACE_GET_IP()					\
	".section __tracepoint_ip\"aw\"\n\t"		\
	_ASM_PTR "(1f)\n\t"				\
	".previous\n\t"					\
	"1:\n\t"

#define trace(name, args...)				\
do {							\
	__builtin_choose_expr((NARGS(args) == 0), 0, 0); \
	__builtin_choose_expr((NARGS(args) == 1), ({ asm volatile (_TRACE_GET_IP() : : A(args, 0) : "memory"); 0; }), 0); \
	__builtin_choose_expr((NARGS(args) == 2), ({ asm volatile (_TRACE_GET_IP() : : B(args, 0) : "memory"); 0; }), 0); \
	__builtin_choose_expr((NARGS(args) == 3), ({ asm volatile (_TRACE_GET_IP() : : C(args, 0) : "memory"); 0; }), 0); \
	__builtin_choose_expr((NARGS(args) == 4), ({ asm volatile (_TRACE_GET_IP() : : D(args, 0) : "memory"); 0; }), 0); \
	__builtin_choose_expr((NARGS(args) == 5), ({ asm volatile (_TRACE_GET_IP() : : E(args, 0) : "memory"); 0; }), 0); \
	BUILD_BUG_ON(NARGS(args) > 5); \
} while (0)

int main(int argc, char **argv)
{
	int one, two, three, four;
	unsigned long long five;

	trace(name, one, two, three, four, five);
	trace(name, one, two, three);
	trace(name, one);

	// error trace(name, one, two, three, four, five, six);
	// error trace(name, one, two, three, four, noexist);

	return 0;
}



-- 
Mathieu Desnoyers
Operating System Efficiency R&D Consultant
EfficiOS Inc.
http://www.efficios.com

[ltt-dev] SystemTAP support in tracepoints

[Josh Stone]

On 04/10/2011 10:44 AM, Mathieu Desnoyers wrote:
> Hi Josh,
> 
> Following our discussion, I started to think about how we could be
> support the systemtap breakpoint approach in UST tracepoints.

Just to set the stage a little more for others, the discussion was about
finding compatibility between our respective developer APIs for
userspace tracing.  LTTng has the UST tracepoints, which are similar in
API to the kernel tracepoints; SystemTap has SDT markers which are
intentionally source-compatible with DTrace SDT.

So while we're waiting for world domination by either API, it would be
nice if our respective tools could make use of the other's userspace
markers.  On the SDT side, the platform coverage is already greater
between SystemTap and DTrace, with Linux, Solaris, FreeBSD, and OSX.
IMO that makes it an easier sell to upstream developers to insert these
markers.  The challenge is how we can adjust the implementation so that
LTTng could link to these, hopefully without sacrificing the current
NOP-like dormant overhead.

Going the other direction, we can try to add metadata to UST such that
SystemTap can find and probe them too, as Mathieu started below...

> I created the following prototype that should let us handle a variable number of
> arguments. The "args" I get here from the args... parameter would be
> taken from the "TP_ARGS()" declaration in the TRACE_EVENT header. The
> code below is really just to see if it is doable at all.

If I understand what you have here, this creates a "__tracepoint_ip"
section which is just an array of tracepoint IPs, as well as asm
clobbers to keep the arguments and memory alive by the compiler.

What's missing is metadata giving a name of some sort to identify the
tracepoint, as well as metadata informing us about the arguments.  We've
already solved these problems in SDT, so I think it would be much better
to just embed an SDT probe in there and call it a day.

AFAIK, the only roadblock is that our probes are argument-numbered
rather than variadic.  I've been thinking for a while that it would be
nice if we had a variadic SDT macro, e.g. STAP_PROBEV, and then UST
could invoke it directly.  Since you and I were brainstorming this,
here's a neat trick I found to count __VA_ARGS__:
  http://groups.google.com/group/comp.std.c/msg/346fc464319b1ee5

I'm going to try to adapt that for SDT, so I'll post if I get it working...

Josh

[ltt-dev] SystemTAP support in tracepoints

[Josh Stone]

On 04/11/2011 02:19 PM, Josh Stone wrote:
> AFAIK, the only roadblock is that our probes are argument-numbered
> rather than variadic.  I've been thinking for a while that it would be
> nice if we had a variadic SDT macro, e.g. STAP_PROBEV, and then UST
> could invoke it directly.  Since you and I were brainstorming this,
> here's a neat trick I found to count __VA_ARGS__:
>   http://groups.google.com/group/comp.std.c/msg/346fc464319b1ee5
> 
> I'm going to try to adapt that for SDT, so I'll post if I get it working...

Here's what I came up with:

#define _SDT_NARG(...) __SDT_NARG(__VA_ARGS__, 10,9,8,7,6,5,4,3,2,1,0)
#define __SDT_NARG(_0,_1,_2,_3,_4,_5,_6,_7,_8,_9,_10, N, ...) N
#define _SDT_PROBE_N(provider, name, N, ...) \
  _SDT_PROBE(provider, name, N, (__VA_ARGS__))
#define STAP_PROBEV(provider, name, ...) \
  _SDT_PROBE_N(provider, name, _SDT_NARG(0,##__VA_ARGS__),##__VA_ARGS__)

_SDT_NARG is actually returning the count minus one, and I call it with
an extra arg, so we can avoid non-standard empty macro args.

Comments?

Josh

[ltt-dev] SystemTAP support in tracepoints

[Frank Ch. Eigler]

jistone wrote:

> [...]
> Here's what I came up with:
> [...]

Cool, ship it.

- FChE

[ltt-dev] SystemTAP support in tracepoints

[Mathieu Desnoyers]

* Josh Stone (jistone at redhat.com) wrote:
> On 04/11/2011 02:19 PM, Josh Stone wrote:
> > AFAIK, the only roadblock is that our probes are argument-numbered
> > rather than variadic.  I've been thinking for a while that it would be
> > nice if we had a variadic SDT macro, e.g. STAP_PROBEV, and then UST
> > could invoke it directly.  Since you and I were brainstorming this,
> > here's a neat trick I found to count __VA_ARGS__:
> >   http://groups.google.com/group/comp.std.c/msg/346fc464319b1ee5
> > 
> > I'm going to try to adapt that for SDT, so I'll post if I get it working...
> 
> Here's what I came up with:
> 
> #define _SDT_NARG(...) __SDT_NARG(__VA_ARGS__, 10,9,8,7,6,5,4,3,2,1,0)
> #define __SDT_NARG(_0,_1,_2,_3,_4,_5,_6,_7,_8,_9,_10, N, ...) N
> #define _SDT_PROBE_N(provider, name, N, ...) \
>   _SDT_PROBE(provider, name, N, (__VA_ARGS__))
> #define STAP_PROBEV(provider, name, ...) \
>   _SDT_PROBE_N(provider, name, _SDT_NARG(0,##__VA_ARGS__),##__VA_ARGS__)
> 
> _SDT_NARG is actually returning the count minus one, and I call it with
> an extra arg, so we can avoid non-standard empty macro args.
> 
> Comments?

Cool! I tweaked it a bit and came up with the following. Comments ?


#include <stdio.h>

#define BUILD_BUG_ON_ZERO(e) (sizeof(struct { int:-!!(e); }))
#define BUILD_BUG_ON(condition) ((void)BUILD_BUG_ON_ZERO(condition))

#define __compiler_offsetof(a,b) __builtin_offsetof(a,b)

#undef offsetof
#ifdef __compiler_offsetof
#define offsetof(TYPE,MEMBER) __compiler_offsetof(TYPE,MEMBER)
#else
#define offsetof(TYPE, MEMBER) ((size_t) &((TYPE *)0)->MEMBER)
#endif

#define __stringify_1(x...)	#x
#define __stringify(x...)	__stringify_1(x)

/* for x86 64 */
#define _ASM_PTR ".quad "

#define _A0()
#define _A1(first)					\
	"g" (first)
#define _A2(first, args...)				\
	"g" (first), _A1(args)
#define _A3(first, args...)				\
	"g" (first), _A2(args)
#define _A4(first, args...)				\
	"g" (first), _A3(args)
#define _A5(first, args...)				\
	"g" (first), _A4(args)
#define _A6(first, args...)				\
	"g" (first), _A5(args)
#define _A7(first, args...)				\
	"g" (first), _A6(args)
#define _A8(first, args...)				\
	"g" (first), _A7(args)
#define _A9(first, args...)				\
	"g" (first), _A8(args)
#define _A10(first, args...)				\
	"g" (first), _A9(args)

#define ___TP_NARGS(_0,_1,_2,_3,_4,_5,_6,_7,_8,_9,_10, N, args...) N
#define __TP_NARGS(args...)				\
	___TP_NARGS(args, 10,9,8,7,6,5,4,3,2,1,0)
/* Handle empty args with added 0. */
#define _TP_NARGS(args...)				\
	__TP_NARGS(0,##args)

/* Create a section with IP and tracepoint name */
#define _TRACEPOINT_ENTRY(name)				\
	".section __tracepoint_strings\"aw\"\n\t"	\
	"0:\n\t"					\
	".string \"" __stringify(name) "\"\n\t"		\
	".previous\n\t"					\
	".section __tracepoint_ip\"aw\"\n\t"		\
	_ASM_PTR "(0b)\n\t"				\
	_ASM_PTR "(1f)\n\t"				\
	".previous\n\t"					\
	"1:\n\t"

/* Create macro name by pasting the result of macro evaluation */
#define __tp_sym(_a, _b)	_a##_b
#define _tp_sym(_a, _b)		__tp_sym(_a, _b)

#define tracepoint(name, args...)			\
do {							\
	asm volatile (_TRACEPOINT_ENTRY(name) : : _tp_sym(_A, _TP_NARGS(args))(args) : "memory");	\
	BUILD_BUG_ON(_TP_NARGS(args) > 10);		\
} while (0)

int main(int argc, char **argv)
{
	int one, two, three, four;
	unsigned long long five, six, seven, eight, nine, ten, eleven;

	//trace(name, one, two, three, four, five);
	//trace(name, one, two, three);
	tracepoint(namet1);
	tracepoint(namet2, one);
	tracepoint(namet3, one, two);
	tracepoint(namet3, one, two, three, four, five);
	tracepoint(namet3, one, two, three, four, five,
		   six, seven, eight, &nine, ten);
	//tracepoint(namet3, one, two, three, four, five,
	//	   six, seven, eight, &nine, ten, eleven);

	// error trace(name, one, two, three, four, five, six);
	// error trace(name, one, two, three, four, noexist);

	return 0;
}



-- 
Mathieu Desnoyers
Operating System Efficiency R&D Consultant
EfficiOS Inc.
http://www.efficios.com

[ltt-dev] SystemTAP support in tracepoints

[Frank Ch. Eigler]

mathieu.desnoyers wrote:

> [...]
> Cool! I tweaked it a bit and came up with the following. Comments ?

How about just:


#define SDT_USE_VARIADIC
#include <sys/sdt.h>

#define tracepoint(name, args...)			\
do {
        STAP_PROBEV(name, name, __VA_ARGS__);           \ 
        /* and whatever ust needs for itself */         \
} while (0)


- FChE

[ltt-dev] SystemTAP support in tracepoints

[Mathieu Desnoyers]

* Frank Ch. Eigler (fche at redhat.com) wrote:
> 
> mathieu.desnoyers wrote:
> 
> > [...]
> > Cool! I tweaked it a bit and came up with the following. Comments ?
> 
> How about just:
> 
> 
> #define SDT_USE_VARIADIC
> #include <sys/sdt.h>

How should support systems that don't have sdt.h ? Is there a define we
could check ?

Mathieu

> 
> #define tracepoint(name, args...)			\
> do {
>         STAP_PROBEV(name, name, __VA_ARGS__);           \ 
>         /* and whatever ust needs for itself */         \
> } while (0)
> 
> 
> - FChE

-- 
Mathieu Desnoyers
Operating System Efficiency R&D Consultant
EfficiOS Inc.
http://www.efficios.com

[ltt-dev] SystemTAP support in tracepoints

[Frank Ch. Eigler]

Hi -

On Tue, Apr 12, 2011 at 02:48:35PM -0400, Mathieu Desnoyers wrote:
> > #define SDT_USE_VARIADIC
> > #include <sys/sdt.h>
> 
> How should support systems that don't have sdt.h ? Is there a define we
> could check ?

If the ust headers are not going to try to replicate all the code from
the systemtap sdt.h file, then they'll need to make a reference to it.
If so, one choice is to make the process manual (have a ust user do
the include before #include <ust/whatever>).  Another choice is for
the ust distribution to autoconf <ust/whatever> based on the existence
of sys/sdt.h header and perhaps #ifdef STAP_PROBEV.

- FChE

[ltt-dev] SystemTAP support in tracepoints

[Mark Wielaard]

On Tue, 2011-04-12 at 14:48 -0400, Mathieu Desnoyers wrote:
> * Frank Ch. Eigler (fche at redhat.com) wrote:
> > How about just:
> > 
> > 
> > #define SDT_USE_VARIADIC
> > #include <sys/sdt.h>
> 
> How should support systems that don't have sdt.h ? Is there a define we
> could check ?

I have just been using the following configure check:

AC_CHECK_HEADER([sys/sdt.h], [SDT_H_FOUND='yes'],
                [SDT_H_FOUND='no';
                   AC_MSG_ERROR([systemtap support needs sys/sdt.h header])])

Which you can adapt to your needs with/without error message, including
an test to see if it fully works for your environment.

[ltt-dev] SystemTAP support in tracepoints

[Mathieu Desnoyers]

* Mark Wielaard (mjw at redhat.com) wrote:
> On Tue, 2011-04-12 at 14:48 -0400, Mathieu Desnoyers wrote:
> > * Frank Ch. Eigler (fche at redhat.com) wrote:
> > > How about just:
> > > 
> > > 
> > > #define SDT_USE_VARIADIC
> > > #include <sys/sdt.h>
> > 
> > How should support systems that don't have sdt.h ? Is there a define we
> > could check ?
> 
> I have just been using the following configure check:
> 
> AC_CHECK_HEADER([sys/sdt.h], [SDT_H_FOUND='yes'],
>                 [SDT_H_FOUND='no';
>                    AC_MSG_ERROR([systemtap support needs sys/sdt.h header])])
> 
> Which you can adapt to your needs with/without error message, including
> an test to see if it fully works for your environment.

Thanks for the hint, but adding this test in the UST build system don't
really make sense, because both systemtap and UST don't have dependency
on each other, and thus the build order between the two packages is not
fixed. So the detection should be done when the UST header is being
included into an application rather than at UST configuration time. And
I don't want to require all applications that want to use tracepoint to
use autoconf neither.

Mathieu


> 
> 
> _______________________________________________
> ltt-dev mailing list
> ltt-dev at lists.casi.polymtl.ca
> http://lists.casi.polymtl.ca/cgi-bin/mailman/listinfo/ltt-dev
> 

-- 
Mathieu Desnoyers
Operating System Efficiency R&D Consultant
EfficiOS Inc.
http://www.efficios.com

[ltt-dev] SystemTAP support in tracepoints

[Mark Wielaard]

On Wed, 2011-04-13 at 11:38 -0400, Mathieu Desnoyers wrote:
> * Mark Wielaard (mjw at redhat.com) wrote:
> > I have just been using the following configure check:
> > 
> > AC_CHECK_HEADER([sys/sdt.h], [SDT_H_FOUND='yes'],
> >                 [SDT_H_FOUND='no';
> >                    AC_MSG_ERROR([systemtap support needs sys/sdt.h header])])
> > 
> > Which you can adapt to your needs with/without error message, including
> > an test to see if it fully works for your environment.
> 
> Thanks for the hint, but adding this test in the UST build system don't
> really make sense, because both systemtap and UST don't have dependency
> on each other, and thus the build order between the two packages is not
> fixed. So the detection should be done when the UST header is being
> included into an application rather than at UST configuration time. And
> I don't want to require all applications that want to use tracepoint to
> use autoconf neither.

Sure autoconf is optional. But if you want UST tracepoints to be
recognized by other consumers that use SDT probe points, then just
depending on the availability of sys/sdt.h in UST seems to make the most
sense. Note that sys/sdt.h is kind of independent from systemtap proper.
It can be installed independently from the systemtap stap translator or
runtime. And it is only a build dependency, not a runtime dependency.
All consumers just parse the notes section to get the addresses to place
their probes and extract the arguments. There is even a binutils bfd
extractor now.

Cheers,

Mark

[ltt-dev] SystemTAP support in tracepoints

[Mathieu Desnoyers]

* Mark Wielaard (mjw at redhat.com) wrote:
> On Wed, 2011-04-13 at 11:38 -0400, Mathieu Desnoyers wrote:
> > * Mark Wielaard (mjw at redhat.com) wrote:
> > > I have just been using the following configure check:
> > > 
> > > AC_CHECK_HEADER([sys/sdt.h], [SDT_H_FOUND='yes'],
> > >                 [SDT_H_FOUND='no';
> > >                    AC_MSG_ERROR([systemtap support needs sys/sdt.h header])])
> > > 
> > > Which you can adapt to your needs with/without error message, including
> > > an test to see if it fully works for your environment.
> > 
> > Thanks for the hint, but adding this test in the UST build system don't
> > really make sense, because both systemtap and UST don't have dependency
> > on each other, and thus the build order between the two packages is not
> > fixed. So the detection should be done when the UST header is being
> > included into an application rather than at UST configuration time. And
> > I don't want to require all applications that want to use tracepoint to
> > use autoconf neither.
> 
> Sure autoconf is optional. But if you want UST tracepoints to be
> recognized by other consumers that use SDT probe points, then just
> depending on the availability of sys/sdt.h in UST seems to make the most
> sense. Note that sys/sdt.h is kind of independent from systemtap proper.
> It can be installed independently from the systemtap stap translator or
> runtime. And it is only a build dependency, not a runtime dependency.
> All consumers just parse the notes section to get the addresses to place
> their probes and extract the arguments. There is even a binutils bfd
> extractor now.

So we seem to have two choices here: either add a dependency on the
package providing sdt.h in UST, or copy sdt.h into UST (it's public
domain, so the licensing permits it).

For the package dependency: I'd need to know:

1 - which package provide sdt.h.
2 - which distributions provide it.
3 - what the dependencies of the package are.
4 - which architectures are supported by this package.

we must keep in mind that UST targets many distributions and many
architectures.

For the sdt.h copy: we could certainly ship it into UST too. It might be
much, much easier. I would put it in ust/sdt.h so we don't overwrite the
SystemTAP header when we install it.  We'd have to be careful about
changes to this header though, because we can end up with two entirely
different versions. The #ifndef _SYS_SDT_H would take care of handling
multiple inclusion of the ust and systemtap sdt.h header incarnations.

One way to handle this versioning problem would be to define a header
"version" at the beginning of sdt.h in both SystemTAP and UST
incarnations:

#define _THIS_SDT_H_VERSION  3
#if (!defined(_SDT_H_VERSION) || _SDT_H_VERSION < _THIS_SDT_H_VERSION)

either

#undef all sdt.h macros
/* rest of the sdt.h header */

or

/* this scheme would require that we only append to the SDT macros,
 * never change the macros per se */
#ifndef each macro name
#define macro
#endif

#endif
#undef _SDT_H_VERSION
#define _SDT_H_VERSION  _THIS_SDT_H_VERSION
#undef _THIS_SDT_H_VERSION

Thoughts ?

Thanks,

Mathieu

> 
> Cheers,
> 
> Mark
> 

-- 
Mathieu Desnoyers
Operating System Efficiency R&D Consultant
EfficiOS Inc.
http://www.efficios.com

[ltt-dev] SystemTAP support in tracepoints

[Josh Stone]

On 04/13/2011 09:58 AM, Mathieu Desnoyers wrote:
> So we seem to have two choices here: either add a dependency on the
> package providing sdt.h in UST, or copy sdt.h into UST (it's public
> domain, so the licensing permits it).

A third choice is to make it a soft dependency, and leave it to the
application's build system to decide whether to enable SDT.  I would
assume that UST consumers will already do some check to see if they have
UST available, and they can do the same for SDT.  That can be signaled
to you either by some define (UST_ENABLE_SDT), or even just require the
app to #include sdt.h themselves before UST, so you can just check
#ifdef STAP_PROBEV.

Josh

[ltt-dev] SystemTAP support in tracepoints

[Mark Wielaard]

On Wed, 2011-04-13 at 12:58 -0400, Mathieu Desnoyers wrote:
> For the package dependency: I'd need to know:
> 
> 1 - which package provide sdt.h.

fedora and opensuse derived distros call it systemtap-sdt-devel,
debian derived distros call it systemtap-sdt-dev.

> 2 - which distributions provide it.

At least those derived from fedora, opensuse and debian.

> 3 - what the dependencies of the package are.

I don't think it has any requirements.

> 4 - which architectures are supported by this package.

I think the package is architecture all since the only architecture
specific thing there possibly could be is whether the NOP instruction is
called NOP. But I know people have been using it at least on amd64,
armel, i386, ia64, powerpc and s390.

> For the sdt.h copy: we could certainly ship it into UST too.

I wouldn't bother, in general sys/sdt.h should be universally available.
And it has some tricks to be more efficient if the assembler contains
the right support for certain directives, so you would then also need to
add the configury for sdt-config.h.in if you are going to install it
yourself.

Cheers,

Mark