[lttng-dev] Record stacktraces at userspace tracing domain

[lttng-dev] Record stacktraces at userspace tracing domain

[Alexander Krabler via lttng-dev]

Hello,

we want to record stacktraces at specific userspace events like e.g. calls to malloc and free using liblttng-ust-libc-wrapper.so.
There is the callstack-user context to achieve this in general, however, it seems like tracing of userspace stacktraces is only available in the kernel tracing domain.

Is there already a solution to achieve this goal?
If not, what would need to be done to achieve this?

Thanks,
Alexander

Re: [lttng-dev] Record stacktraces at userspace tracing domain

[Christophe Bédard via lttng-dev]

[-- Attachment #1: Type: text/plain, Size: 1443 bytes --]

Hi,

I did the same thing a while ago, i.e., trigger tracepoints on
malloc/free/etc. using liblttng-ust-libc-wrapper and collect userspace
callstack information (so that the indirect calls to malloc/free can be
removed from an application).

There is a userspace callstack context implementation here for lttng-ust
2.10, see the last 3 commits:
https://github.com/tahini/lttng-ust-1/commits/ust-callstack-2.10/. Here's
the corresponding lttng-tools 2.10 branch needed to enable the userspace
callstack context:
https://github.com/tahini/lttng-tools/commits/ust-context-callstack/.

I've rebased it on 2.11 here:
https://github.com/ApexAI/lttng-ust/commits/ust-callstack-2.11/.
lttng-tools:
https://github.com/ApexAI/lttng-tools/commits/ust-callstack-2.11/. It
shouldn't be too hard to rebase it all on a newer version.

Hope that helps,

Christophe

On Mon, Dec 2, 2024 at 8:32 AM Alexander Krabler via lttng-dev <
lttng-dev@lists.lttng.org> wrote:

> Hello,
>
> we want to record stacktraces at specific userspace events like e.g. calls
> to malloc and free using liblttng-ust-libc-wrapper.so.
> There is the callstack-user context to achieve this in general, however,
> it seems like tracing of userspace stacktraces is only available in the
> kernel tracing domain.
>
> Is there already a solution to achieve this goal?
> If not, what would need to be done to achieve this?
>
> Thanks,
> Alexander

[-- Attachment #2: Type: text/html, Size: 2190 bytes --]

Re: [lttng-dev] Record stacktraces at userspace tracing domain

[Kienan Stewart via lttng-dev]

Hi Alexander, Christophe,

On 12/2/24 4:17 PM, Christophe Bédard via lttng-dev wrote:
 > Hi,
 >
 > I did the same thing a while ago, i.e., trigger tracepoints on
 > malloc/free/etc. using liblttng-ust-libc-wrapper and collect userspace
 > callstack information (so that the indirect calls to malloc/free can be
 > removed from an application).
 >
 > There is a userspace callstack context implementation here for lttng-ust
 > 2.10, see the last 3 commits:
 > https://github.com/tahini/lttng-ust-1/commits/ust-callstack-2.10/. Here's
 > the corresponding lttng-tools 2.10 branch needed to enable the userspace
 > callstack context:
 > https://github.com/tahini/lttng-tools/commits/ust-context-callstack/.
 >
 > I've rebased it on 2.11 here:
 > https://github.com/ApexAI/lttng-ust/commits/ust-callstack-2.11/.
 > lttng-tools:
 > https://github.com/ApexAI/lttng-tools/commits/ust-callstack-2.11/. It
 > shouldn't be too hard to rebase it all on a newer version.
 >
 > Hope that helps,
 >
 > Christophe
 >
 > On Mon, Dec 2, 2024 at 8:32 AM Alexander Krabler via lttng-dev <
 > lttng-dev@lists.lttng.org> wrote:
 >
 >> Hello,
 >>
 >> we want to record stacktraces at specific userspace events like e.g. 
calls
 >> to malloc and free using liblttng-ust-libc-wrapper.so.
 >> There is the callstack-user context to achieve this in general, however,
 >> it seems like tracing of userspace stacktraces is only available in the
 >> kernel tracing domain.
 >>
 >> Is there already a solution to achieve this goal?

Depending on the type of information you require, it could be possible 
to use the instruction-pointer (ip) context[1], the statedump[2] and/or 
lttng-ust-dl[3] for base addresses, and the symbol table in-order to 
resolve ip -> symbol name in the offline analysis phase with the help of 
babeltrace's debug-info plugin[4]. While a bit more work for the 
analysis, this process reduces the impact of the tracing on the program 
at run-time.

 >> If not, what would need to be done to achieve this?

The proof of concept suggested by Christophe has some limitations that 
may make it unsuitable for use in a production environment notably lack 
of testing - including corner cases such as emitting callstack traces 
from noexcept regions or signal handlers. Run-time stack unwinding may 
be expensive when frame pointers or hardware mechanisms aren't 
available, and the typical work-around to that is to sample the stack 
and unwind in post processing which is not a desirable solution in the 
case of LTTng-UST. Future work could also include integration with 
sframe[5], which aims to provide the information necessary for callstack 
unwinding with a lower code-size cost than DWARF and without the 
run-time cost of frame pointers.

If your intention is to use the callstack recording in production, it 
could also help to reduce the sampling rate. E.g., only sample every N 
time units or every N calls, or sampling "overly large" allocations - 
whatever that may be in your context.

What is required to get this into upstream UST?

  * Funding to work on making it production grade (robustness, testing, 
performance, and integration with the other LTTng features)

thanks,
kienan

[1]: https://lttng.org/man/3/lttng-ust/v2.13/#doc-_context_information
[2]: https://lttng.org/man/3/lttng-ust/v2.13/#doc-state-dump
[3]: https://lttng.org/man/3/lttng-ust/v2.13/#doc-ust-lib
[4]: 
https://babeltrace.org/docs/v2.0/man7/babeltrace2-filter.lttng-utils.debug-info.7/
[5]: https://www.sourceware.org/binutils/docs/sframe-spec.html

 >>
 >> Thanks,
 >> Alexander
 >

AW: [lttng-dev] Record stacktraces at userspace tracing domain

[Alexander Krabler via lttng-dev]

Hi Christophe and Kienan,


Internal
On 12/2/24 4:17 PM, Christophe Bédard via lttng-dev wrote:
> I did the same thing a while ago, i.e., trigger tracepoints on
> malloc/free/etc. using liblttng-ust-libc-wrapper and collect userspace
> callstack information (so that the indirect calls to malloc/free can be
> removed from an application).
> ...
Thank you for the information and links!
I have looked into it, seems promising.


On 12/10/24 11:41 AM, Kienan Stewart via lttng-dev wrote:
> Depending on the type of information you require, it could be possible
> to use the instruction-pointer (ip) context[1], the statedump[2] and/or
> lttng-ust-dl[3] for base addresses, and the symbol table in-order to
> resolve ip -> symbol name in the offline analysis phase with the help of
> babeltrace's debug-info plugin[4]. While a bit more work for the
> analysis, this process reduces the impact of the tracing on the program
> at run-time.
Thanks for the hints.
This would be possible, but that would give me only the instruction pointer of the top-level stack frame.
If malloc and free calls are wrapped inside of functions (e.g. think about std::string, std::unique/shared_ptr for C++),
I would only see that it's called from that specific type.
But, for me, it's important to see the context, too, where the string/smart pointer is used.
Therefore I want to record the instruction pointer of every stackframe (or at least the top N ones in order to be able to locate the "problem").

As stacktraces are only wanted/needed for specific events, I also consider adding new tracepoints
like lttng_ust_libc:malloc_stacktrace to lttng-ust-libc-wrapper directly
recording the desired information into the trace buffer as an alternative solution.

Best regards,
Alexander

Re: AW: [lttng-dev] Record stacktraces at userspace tracing domain

[Mathieu Desnoyers via lttng-dev]

On 2024-12-11 05:32, Alexander Krabler via lttng-dev wrote:
> Hi Christophe and Kienan,
> 
> 
> Internal
> On 12/2/24 4:17 PM, Christophe Bédard via lttng-dev wrote:
>> I did the same thing a while ago, i.e., trigger tracepoints on
>> malloc/free/etc. using liblttng-ust-libc-wrapper and collect userspace
>> callstack information (so that the indirect calls to malloc/free can be
>> removed from an application).
>> ...
> Thank you for the information and links!
> I have looked into it, seems promising.
> 
> 
> On 12/10/24 11:41 AM, Kienan Stewart via lttng-dev wrote:
>> Depending on the type of information you require, it could be possible
>> to use the instruction-pointer (ip) context[1], the statedump[2] and/or
>> lttng-ust-dl[3] for base addresses, and the symbol table in-order to
>> resolve ip -> symbol name in the offline analysis phase with the help of
>> babeltrace's debug-info plugin[4]. While a bit more work for the
>> analysis, this process reduces the impact of the tracing on the program
>> at run-time.
> Thanks for the hints.
> This would be possible, but that would give me only the instruction pointer of the top-level stack frame.
> If malloc and free calls are wrapped inside of functions (e.g. think about std::string, std::unique/shared_ptr for C++),
> I would only see that it's called from that specific type.
> But, for me, it's important to see the context, too, where the string/smart pointer is used.

Another approach then is to add tracepoints within your C++ standard
library functions. Then you can use the caller ip of _those_ functions
and pass them to the tracepoint as "ip" context override. Here is how
it is done e.g. in src/lib/lttng-ust-libc-wrapper/lttng-ust-malloc.c :

#define LTTNG_UST_TRACEPOINT_DEFINE
#define LTTNG_UST_TRACEPOINT_CREATE_PROBES
#define LTTNG_UST_TP_IP_PARAM ip                        <------ this.
#include "ust_libc.h"

src/lib/lttng-ust-libc-wrapper/ust_libc.h :

LTTNG_UST_TRACEPOINT_EVENT(lttng_ust_libc, malloc,
         LTTNG_UST_TP_ARGS(size_t, size, void *, ptr, void *, ip),
         LTTNG_UST_TP_FIELDS(
                 lttng_ust_field_integer(size_t, size, size)
                 lttng_ust_field_integer_hex(void *, ptr, ptr)
                 lttng_ust_field_unused(ip)
         )
)

You can similarly specify which parameter end up being used to
override the caller ip sampling.

Are those C++ functions something that is under your control ?

Thanks,

Mathieu


> Therefore I want to record the instruction pointer of every stackframe (or at least the top N ones in order to be able to locate the "problem").
> 
> As stacktraces are only wanted/needed for specific events, I also consider adding new tracepoints
> like lttng_ust_libc:malloc_stacktrace to lttng-ust-libc-wrapper directly
> recording the desired information into the trace buffer as an alternative solution.
> 
> Best regards,
> Alexander

-- 
Mathieu Desnoyers
EfficiOS Inc.
https://www.efficios.com

[lttng-dev] Record stacktraces at userspace tracing domain

[Alexander Krabler via lttng-dev]

Hi Mathieu,
 
On 2024-12-11 10:43, Mathieu Desnoyers via lttng-dev wrote:
> Another approach then is to add tracepoints within your C++ standard
> library functions. Then you can use the caller ip of _those_ functions
> and pass them to the tracepoint as "ip" context override.
That's difficult, as this would require to modify very much code locations and may also be difficult due to inlining.
Also it would just add one extra level, but I think that's not enough.

I'm rather thinking of something similar to this:
LTTNG_UST_TRACEPOINT_EVENT(
    ust,
    malloc_callstack,
    LTTNG_UST_TP_ARGS(
        unsigned int, callstack_length,
        int*, callstack
    ),
    LTTNG_UST_TP_FIELDS(
        lttng_ust_field_sequence_hex(int, callstack, callstack, unsigned int, callstack_length)
    )
)

And then use backtrace/libunwind to retrieve the required information and put it into the tracepoint.

Thanks,
Alexander

Re: [lttng-dev] Record stacktraces at userspace tracing domain

[Dirk Eibach via lttng-dev]

[-- Attachment #1: Type: text/plain, Size: 182 bytes --]

For more involved user space tracing I highly recommend
https://github.com/wolfpld/tracy

It is a different approach and probably already has all the bells and
whistles you require.

[-- Attachment #2: Type: text/html, Size: 298 bytes --]