[lttng-dev] using --function/--probe

[lttng-dev] using --function/--probe

[Bill Trost]

Hi,

I'm trying to investigate a problem under Linux 2.6.39.4 involving
kworker threads and am pretty much stymied. I've tried enabling
various events using --function and --probe and it's been
very hit or miss. Very often, "lttng enable-event" comes back
with nothing more specific than "bad ioctl", but every once
in a while I'll find a symbol I can add a probe point for.

So: What does "--probe" mean? What does "--function" mean, and how does
it differ from "--probe"? How do I determine what symbols are valid for
each of these options? And, maybe as a worked example, how do I
trace what work is being enqueued and run by the kworker threads?

Thanks,
Bill

[lttng-dev] using --function/--probe

[Mathieu Desnoyers]

* Bill Trost (trost at cloud.rain.com) wrote:
> Hi,
> 
> I'm trying to investigate a problem under Linux 2.6.39.4 involving
> kworker threads and am pretty much stymied. I've tried enabling
> various events using --function and --probe and it's been
> very hit or miss. Very often, "lttng enable-event" comes back
> with nothing more specific than "bad ioctl", but every once
> in a while I'll find a symbol I can add a probe point for.
> 
> So: What does "--probe" mean?

--probe allows to put a kprobe (breakpoint) at a kernel text address or
symbol.

> What does "--function" mean,

--function allows to put a kretprobes at a function entry site (using
its symbol name), which will trigger events at function entry return.

> and how does
> it differ from "--probe"? How do I determine what symbols are valid for
> each of these options?

It entirely depends on which functions are blacklisted in the kernel
(this is an attribute added to the functions specifically for kprobes).
The keyword is "__kprobes".

> And, maybe as a worked example, how do I
> trace what work is being enqueued and run by the kworker threads?

For this level of details, I think kprobes/kretprobes will not currently
allow you to fetch it. The two options we have are:

- use static tracepoints. Is there a tracepoint that targets the
  information you are looking for ? Try "lttng list -k".
- or extend the dynamic probing to allow fetching variables from
  debug information,

Thanks,

Mathieu

> 
> Thanks,
> Bill
> 
> _______________________________________________
> lttng-dev mailing list
> lttng-dev at lists.lttng.org
> http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

-- 
Mathieu Desnoyers
Operating System Efficiency R&D Consultant
EfficiOS Inc.
http://www.efficios.com

[lttng-dev] using --function/--probe

[Bill Trost]

Mathieu wrote:
    * Bill Trost (trost at cloud.rain.com) wrote:
    > and how does it differ from "--probe"? How do I determine
    > what symbols are valid for each of these options?

    It entirely depends on which functions are blacklisted in the kernel
    (this is an attribute added to the functions specifically for kprobes).
    The keyword is "__kprobes".

Is that the only basis? For example, I can add a dynamic tracepoint (of
either flawor) for queue_work_on(), but not work __queue_work(), yet
neither of those functions have any apparent annotation. Or does
EXPORT_SYMBOL_GPL imply a probe point?

    > ...how do I trace what work is being
    > enqueued and run by the kworker threads?

    For this level of details, I think kprobes/kretprobes will not
    currently allow you to fetch it. The two options we have are:

    - use static tracepoints. Is there a tracepoint that targets the
      information you are looking for ? Try "lttng list -k".

No, I've been capturing all kernel events and haven't seen
anything resembling the kworker tracepoints of lttng 0.x.

    - or extend the dynamic probing to allow fetching variables from
      debug information,

Hmm. Well, I did a bit more poking and got a bit closer -- at least now
I know what dynamic points to use. (eg, "queue_work_on" and friends to
see what is being queued). I take it that LTT can't capture function call
arguments as a form of additional context? That would be an ideal
bit of information in this case.

Thanks again,
Bill