Function tracing not working

Function tracing not working

[David Aldrich via lttng-dev]

Hi,

I am trying to use lttng-ust-cyg-profile.

Our application is large so, so far, I only compile main.cpp with
compiler option '-finstrument-functions' (I don't specify option
'-pg').

Our app must be run as root so I run it, and capture with lttng, like this:

lttng create app_trace
lttng enable-event --userspace 'lttng_ust_tracef:*'
lttng enable-channel --kernel --session app_trace kernel_channel
lttng enable-event --kernel --all --channel kernel_channel --session app_trace
export LD_PRELOAD=liblttng-ust-cyg-profile.so
lttng start
sudo -E ./myapp
lttng stop

I notice that many events get discarded:

Warning: 198522 events were discarded, please refer to the
documentation on channel configuration.

I then import into TraceCompass.  I see the traces in the Resources
View, but when I expand Experiment > Views, all 3 'LTTng-UST
CallStack' entries are striked through so I can't select them. So I
can't visualise the function calls.

Any idea what I am doing wrong please?

Best regards, David

Re: Function tracing not working

[Mathieu Desnoyers via lttng-dev]

On 2025-06-02 10:37, David Aldrich via lttng-dev wrote:
> Hi,
> 
> I am trying to use lttng-ust-cyg-profile.
> 
> Our application is large so, so far, I only compile main.cpp with
> compiler option '-finstrument-functions' (I don't specify option
> '-pg').
> 
> Our app must be run as root so I run it, and capture with lttng, like this:
> 
> lttng create app_trace
> lttng enable-event --userspace 'lttng_ust_tracef:*'
> lttng enable-channel --kernel --session app_trace kernel_channel
> lttng enable-event --kernel --all --channel kernel_channel --session app_trace
> export LD_PRELOAD=liblttng-ust-cyg-profile.so
> lttng start
> sudo -E ./myapp
> lttng stop
> 
> I notice that many events get discarded:
> 
> Warning: 198522 events were discarded, please refer to the
> documentation on channel configuration.
> 
> I then import into TraceCompass.  I see the traces in the Resources
> View, but when I expand Experiment > Views, all 3 'LTTng-UST
> CallStack' entries are striked through so I can't select them. So I
> can't visualise the function calls.
> 
> Any idea what I am doing wrong please?

AFAIK, the "LD_PRELOAD" environment variable is cleared by
sudo when running its child process for security purposes.

Thanks,

Mathieu

> 
> Best regards, David


-- 
Mathieu Desnoyers
EfficiOS Inc.
https://www.efficios.com

Re: Function tracing not working

[David Aldrich via lttng-dev]

Hi Mathieu

> > Any idea what I am doing wrong please?
>
> AFAIK, the "LD_PRELOAD" environment variable is cleared by
> sudo when running its child process for security purposes.

Thanks for your answer, that does help. However, I have now adjusted
the code so that sudo is not required, but when I import the trace the
UST-CallStacks are still strikethrough.

Any thoughts why this is happening?

I don't see any runtime errors whether or not I specify the prefix:

LD_PRELOAD=liblttng-ust-cyg-profile.so

Is that expected?

Best regards, David

Re: Function tracing not working

[David Aldrich via lttng-dev]

Hi Mathieu

I'd like to add some more detail.  I now run our app using this pattern:

LD_PRELOAD=liblttng-ust-cyg-profile.so ./app

On importing the trace into TraceCompass I see:

"null analysis for LTTng-UST CallStack"

Could the reason be the lttng commands I am using to capture UST and
Kernel events:

lttng enable-event --userspace 'lttng_ust_tracef:*'
lttng enable-channel --kernel --session app_trace kernel_channel
lttng enable-event --kernel --all --channel kernel_channel --session app_trace

Best regards
David

On Wed, Jun 4, 2025 at 5:22 PM David Aldrich
<david.aldrich.ntml@gmail.com> wrote:
>
> Hi Mathieu
>
> > > Any idea what I am doing wrong please?
> >
> > AFAIK, the "LD_PRELOAD" environment variable is cleared by
> > sudo when running its child process for security purposes.
>
> Thanks for your answer, that does help. However, I have now adjusted
> the code so that sudo is not required, but when I import the trace the
> UST-CallStacks are still strikethrough.
>
> Any thoughts why this is happening?
>
> I don't see any runtime errors whether or not I specify the prefix:
>
> LD_PRELOAD=liblttng-ust-cyg-profile.so
>
> Is that expected?
>
> Best regards, David

Re: Function tracing not working

[Mathieu Desnoyers via lttng-dev]

On 2025-06-05 06:49, David Aldrich wrote:
> Hi Mathieu
> 
> I'd like to add some more detail.  I now run our app using this pattern:
> 
> LD_PRELOAD=liblttng-ust-cyg-profile.so ./app
> 
> On importing the trace into TraceCompass I see:
> 
> "null analysis for LTTng-UST CallStack"
> 
> Could the reason be the lttng commands I am using to capture UST and
> Kernel events:
> 
> lttng enable-event --userspace 'lttng_ust_tracef:*'
> lttng enable-channel --kernel --session app_trace kernel_channel
> lttng enable-event --kernel --all --channel kernel_channel --session app_trace

Indeed, You will also need to enable something like this:

lttng enable-event --userspace 'lttng_ust_cyg_profile:*'

See "man 3 lttng-ust-cyg-profile" for details.

Thanks,

Mathieu

> 
> Best regards
> David
> 
> On Wed, Jun 4, 2025 at 5:22 PM David Aldrich
> <david.aldrich.ntml@gmail.com> wrote:
>>
>> Hi Mathieu
>>
>>>> Any idea what I am doing wrong please?
>>>
>>> AFAIK, the "LD_PRELOAD" environment variable is cleared by
>>> sudo when running its child process for security purposes.
>>
>> Thanks for your answer, that does help. However, I have now adjusted
>> the code so that sudo is not required, but when I import the trace the
>> UST-CallStacks are still strikethrough.
>>
>> Any thoughts why this is happening?
>>
>> I don't see any runtime errors whether or not I specify the prefix:
>>
>> LD_PRELOAD=liblttng-ust-cyg-profile.so
>>
>> Is that expected?
>>
>> Best regards, David


-- 
Mathieu Desnoyers
EfficiOS Inc.
https://www.efficios.com

Re: Function tracing not working

[David Aldrich via lttng-dev]

Hi Mathieu

> Indeed, You will also need to enable something like this:
>
> lttng enable-event --userspace 'lttng_ust_cyg_profile:*'
>
> See "man 3 lttng-ust-cyg-profile" for details.

Thanks for this. I used your suggested command and can now see the
Flame Chart (Incubator). However, I only see symbol addresses, not the
actual symbolic names.

I compile with:

-O3 -pg -finstrument-functions

Do I need any other compiler/linker options to resolve the symbol names?

Best regards
David

Re: Function tracing not working

[Mathieu Desnoyers via lttng-dev]

On 2025-06-06 10:37, David Aldrich wrote:
> Hi Mathieu
> 
>> Indeed, You will also need to enable something like this:
>>
>> lttng enable-event --userspace 'lttng_ust_cyg_profile:*'
>>
>> See "man 3 lttng-ust-cyg-profile" for details.
> 
> Thanks for this. I used your suggested command and can now see the
> Flame Chart (Incubator). However, I only see symbol addresses, not the
> actual symbolic names.
> 
> I compile with:
> 
> -O3 -pg -finstrument-functions
> 
> Do I need any other compiler/linker options to resolve the symbol names?

Yes. Resolving addresses to symbol names requires to know the
base addresses where the executable mappings are loaded.

See:

man 7 babeltrace2-filter.lttng-utils.debug-info

for a detailed explanation about what needs to be enabled.

Thanks,

Mathieu


> 
> Best regards
> David


-- 
Mathieu Desnoyers
EfficiOS Inc.
https://www.efficios.com