Re: Static/dynamic userspace/kernel trace

Re: Static/dynamic userspace/kernel trace

[Frank Ch. Eigler]

Hi -

On Mon, Apr 19, 2010 at 09:16:04AM -0400, Dominique Toupin wrote:

> I had a chat with Tom at the collaboration summit on tracing, he was
> suggesting I send you a mail on a recent GDB improvement.  You might
> not be aware that user space dynamic tracepoint are now available in
> GDB, in process tracing if a few byte space is available to put a
> jump, if that space is not available then a trap between gdbserver
> and the process. With this addition all aspects of tracing seem to
> be covered:
>
>  - static user space: LTTng UST
>  - dynamic user space: GDB dynamic tracepoint
>  - static kernel: kernel static tracepoint/trace event
>  - dynamic kernel: kprobe
>
> and conditional tracing is available for both kernel and user space.

All right.


> Tom was saying uprobes/utrace can bring nice improvements for
> ptrace, for tracing it was not clear to me or Tom why we need
> uprobe/utrace or why systemtap needs it. Do you have a use case
> where the above tracing cannot be use and uprobe/utrace is needed
> instead?

Recall that systemtap goes beyond just tracing with simple local
conditions.  There are dozens of nontrivial systemtap scripts that
involve conditions based on history, searching through live data
structures, taking actions, making state changes.  The above
technologies do not stretch that far, nor are they integrated.

The main motivating dependence by systemtap on utrace has been due to
uprobes.  We shall see to what extent lkml is receptive to a
utrace-free-uprobes implementation being worked on now.  If it is
merged and usable, this would moot one particular dependence.  But
there are other needs: proper process control facility like utrace
enables scripted operations such as clean thread suspend/resume or
signal generation/suppression.

And of course utrace in general has ambitions beyond systemtap.  As
oft repeated on lkml etc., some sort of process control multiplexing
facility is necessary so that a single ptrace-client tool does not
monopolize target processes.  That newfangled gdb tracing stuff does
not work if someone was already strace'ing the same process.  This is
why we prototyped the gdbstub-in-kernel for userspace processes, based
on utrace.  It already demonstrated concurrent gdb+strace on the same
threads.


- FChE

RE: Static/dynamic userspace/kernel trace

[Dominique Toupin]

I agree that GDB could use a better ptrace.

Regarding tracing, for me the main use case is to get the data out of the system very fast and with very low overhead, besides conditional tracing and a few minor things, most of the analysis is done on the host side, either after the fact or in streaming mode. Doing historical analysis on the target or big actions are not an option for me. I guess in the IT industry more cycles are available to do those kinds of analysis directly on the target server.


> -----Original Message-----
> From: Frank Ch. Eigler [mailto:fche@redhat.com] 
> Sent: 19-Apr-10 10:14
> To: Dominique Toupin
> Cc: gdb@sourceware.org; systemtap@sourceware.org
> Subject: Re: Static/dynamic userspace/kernel trace
> 
> Hi -
> 
> On Mon, Apr 19, 2010 at 09:16:04AM -0400, Dominique Toupin wrote:
> 
> > I had a chat with Tom at the collaboration summit on 
> tracing, he was 
> > suggesting I send you a mail on a recent GDB improvement.  
> You might 
> > not be aware that user space dynamic tracepoint are now 
> available in 
> > GDB, in process tracing if a few byte space is available to put a 
> > jump, if that space is not available then a trap between 
> gdbserver and 
> > the process. With this addition all aspects of tracing seem to be 
> > covered:
> >
> >  - static user space: LTTng UST
> >  - dynamic user space: GDB dynamic tracepoint
> >  - static kernel: kernel static tracepoint/trace event
> >  - dynamic kernel: kprobe
> >
> > and conditional tracing is available for both kernel and user space.
> 
> All right.
> 
> 
> > Tom was saying uprobes/utrace can bring nice improvements 
> for ptrace, 
> > for tracing it was not clear to me or Tom why we need 
> uprobe/utrace or 
> > why systemtap needs it. Do you have a use case where the 
> above tracing 
> > cannot be use and uprobe/utrace is needed instead?
> 
> Recall that systemtap goes beyond just tracing with simple 
> local conditions.  There are dozens of nontrivial systemtap 
> scripts that involve conditions based on history, searching 
> through live data structures, taking actions, making state 
> changes.  The above technologies do not stretch that far, nor 
> are they integrated.
> 
> The main motivating dependence by systemtap on utrace has 
> been due to uprobes.  We shall see to what extent lkml is 
> receptive to a utrace-free-uprobes implementation being 
> worked on now.  If it is merged and usable, this would moot 
> one particular dependence.  But there are other needs: proper 
> process control facility like utrace enables scripted 
> operations such as clean thread suspend/resume or signal 
> generation/suppression.
> 
> And of course utrace in general has ambitions beyond 
> systemtap.  As oft repeated on lkml etc., some sort of 
> process control multiplexing facility is necessary so that a 
> single ptrace-client tool does not monopolize target 
> processes.  That newfangled gdb tracing stuff does not work 
> if someone was already strace'ing the same process.  This is 
> why we prototyped the gdbstub-in-kernel for userspace 
> processes, based on utrace.  It already demonstrated 
> concurrent gdb+strace on the same threads.
> 
> 
> - FChE
> 

RE: Static/dynamic userspace/kernel trace

[Mark Wielaard]

On Mon, 2010-04-19 at 15:50 -0400, Dominique Toupin wrote:
> I agree that GDB could use a better ptrace.
> 
> Regarding tracing, for me the main use case is to get the data out of
> the system very fast and with very low overhead, besides conditional
> tracing and a few minor things, most of the analysis is done on the
> host side, either after the fact or in streaming mode. Doing
> historical analysis on the target or big actions are not an option for
> me. I guess in the IT industry more cycles are available to do those
> kinds of analysis directly on the target server.

One of the advantages of pre-filtering your tracing results is that the
overhead will be a lot lower than having to process a stream of
multi-megabytes i/o to disk or off an machines. With SystemTap I am
often not doing any kind of real analysis of the trace data on the
server itself, because it makes it possible to have the trace results in
a compact enough format that the results can be directly interpreted.

Having a very low overhead pre-filter of the trace output using full
expressions based on context variables, keeping statistics through
aggregate state variables and deciding what to push through the trace
output buffer using formatted output and data kept in associative arrays
helps a lot. Since all can be done without incurring extra i/o,
context-switches or external/post filtering it makes
interpreting/analyzing the actual trace data a lot easier and lower
overhead. It might also help in your use case, since you don't have to
push multi-megabytes of trace data off a machine but can tailor the
trace buffers to only have a couple of K of targeted output.

Cheers,

Mark

RE: Static/dynamic userspace/kernel trace

[Dominique Toupin]

 
> Having a very low overhead pre-filter of the trace output 
> using full expressions based on context variables, keeping 
> statistics through aggregate state variables and deciding 
> what to push through the trace output buffer using formatted 
> output and data kept in associative arrays helps a lot. Since 
> all can be done without incurring extra i/o, context-switches 
> or external/post filtering it makes interpreting/analyzing 
> the actual trace data a lot easier and lower overhead. It 
> might also help in your use case, since you don't have to 
> push multi-megabytes of trace data off a machine but can 
> tailor the trace buffers to only have a couple of K of 
> targeted output.

We can use conditional tracing in LTTng/kprobe and GDB tracepoint, it could be good to have a more elaborate conditional tracing. It could be worth it to compare the different Linux conditional tracing options (both user space/kernel and dynamic/static) and see how we can improve some of them.
Our problem is if we do a very fancy condition or live analysis of the data before logging we have too much overhead in CPU cycles.

Re: Static/dynamic userspace/kernel trace

[Frank Ch. Eigler]

Dominique Toupin <dominique.toupin@ericsson.com> writes:

> I agree that GDB could use a better ptrace.

Right.  And since ptrace proper is needed as a backward compatibility
measure, the kernel will in turn need something to multiplex/arbitrate
between the different interfaces.  So this is how we came to utrace
years ago.

> Regarding tracing, for me the main use case is to get the data out
> of the system very fast and with very low overhead [...]

Understood.  That same approach underlies other tracing/performance
tools.  For systemtap, this has been more of an asymptotic case:
something it can do too, with other tools providing some tough-to-beat
performance (and sometimes usability) benchmarks.

- FChE

Re: Static/dynamic userspace/kernel trace

[Srikar Dronamraju]

> 
> > I had a chat with Tom at the collaboration summit on tracing, he was
> > suggesting I send you a mail on a recent GDB improvement.  You might
> > not be aware that user space dynamic tracepoint are now available in
> > GDB, in process tracing if a few byte space is available to put a
> > jump, if that space is not available then a trap between gdbserver
> > and the process. With this addition all aspects of tracing seem to
> > be covered:
> >
> >  - static user space: LTTng UST
> >  - dynamic user space: GDB dynamic tracepoint

I did go thro
http://sources.redhat.com/gdb/current/onlinedocs/gdb.html#Set-Tracepoints
but it says 
 "The tracepoint facility is currently available only for remote targets.
See Targets. In addition, your remote target must know how to collect
trace data. This functionality is implemented in the remote stub;
however, none of the stubs distributed with gdb support tracepoints as
of this writing. "
 
and 

"Some targets may support fast tracepoints, which are inserted in a
different way (such as with a jump instead of a trap), that is faster
but possibly restricted in where they may be installed. "

So it possible to use GDB dynamic tracepoints on regular programs
without using remote protocol? If not do you plans to implement this for
non-remote targets?


> >  - static kernel: kernel static tracepoint/trace event
> >  - dynamic kernel: kprobe
> >
> > and conditional tracing is available for both kernel and user space.
> 
> All right.
> 
> 
> > Tom was saying uprobes/utrace can bring nice improvements for
> > ptrace, for tracing it was not clear to me or Tom why we need
> > uprobe/utrace or why systemtap needs it. Do you have a use case
> > where the above tracing cannot be use and uprobe/utrace is needed
> > instead?

Since gdb tracepoints would fall back on traps, using uprobes could
decrease the overhead on inserting/processing a trap instruction. Also
the uprobes handler could be used to collect the information in the same
process context.

Alternatively gdb could use user_bkpt layer to insert traps and 
collect the information in the tracer's context. However this would
require tracer to do some additional work. Still this should be provide
lesser overhead than ptrace.

Uprobes/user_bkpt implements execution out of line, hence it scales well
with multithreaded applications.

Also as Frank already pointed out, uprobes/utrace were designed for
allowing multiple tracing sessions to connect to the same tracee.

I am ignorant on how gdb dynamic tracepoints was implemented to comment
on how gdb could further use uprobes/utrace. Can you please point me to
some documentation on the same.

--
Thanks and Regards
Srikar

Re: Static/dynamic userspace/kernel trace

[Stan Shebs]

Srikar Dronamraju wrote:
>>> I had a chat with Tom at the collaboration summit on tracing, he was
>>> suggesting I send you a mail on a recent GDB improvement.  You might
>>> not be aware that user space dynamic tracepoint are now available in
>>> GDB, in process tracing if a few byte space is available to put a
>>> jump, if that space is not available then a trap between gdbserver
>>> and the process. With this addition all aspects of tracing seem to
>>> be covered:
>>>
>>>  - static user space: LTTng UST
>>>  - dynamic user space: GDB dynamic tracepoint
>>>       
>
> I did go thro
> http://sources.redhat.com/gdb/current/onlinedocs/gdb.html#Set-Tracepoints
> but it says 
>  "The tracepoint facility is currently available only for remote targets.
> See Targets. In addition, your remote target must know how to collect
> trace data. This functionality is implemented in the remote stub;
> however, none of the stubs distributed with gdb support tracepoints as
> of this writing. "
>   

This is now only accurate in a pedantic sense, since gdbserver does have 
tracepoint support.  The example stubs distributed with GDB have become 
so old and crufty that they offer very little useful guidance about 
writing a stub, and are probably actively misleading people; I'm 
half-inclined to propose their removal.

>  
> and 
>
> "Some targets may support fast tracepoints, which are inserted in a
> different way (such as with a jump instead of a trap), that is faster
> but possibly restricted in where they may be installed. "
>
> So it possible to use GDB dynamic tracepoints on regular programs
> without using remote protocol? If not do you plans to implement this for
> non-remote targets?
>   

No, the gdbserver-based implementation we have at the moment still 
assumes that the agent is speaking remote protocol back to GDB.  
However, the not-yet-contributed fast tracepoint library could be tied 
into a native debugging arrangement, should someone be interested in 
taking that up.

> I am ignorant on how gdb dynamic tracepoints was implemented to comment
> on how gdb could further use uprobes/utrace. Can you please point me to
> some documentation on the same.
>   

There's not a good general description; gdb/doc/agentexpr.texi has some 
material on how the agent expressions fit into the bigger tracing 
picture, but it's also in reference to the EMC target that hasn't been 
used in years.  Now that our first generation of tracepoint revival is 
done, it would be good to go through the docs and update them.

To respond to the larger point, it's worthwhile for us to get some 
experience with the different systems, and start thinking about how they 
can usefully interoperate.  Anybody interested in tracing should at 
least fire up a GDB+GDBserver combo using current sources, and try a 
trace experiment or two using the manual as guide, just to see how it 
works in practice.  Maybe there's not enough functionality overlap to do 
anything, but even if that's the case, we can then make a little writeup 
sending users to the right tracing technology for specific types of 
problems.

Stan