So looking at my lttng-sessiond log I found some concerning issues of things being "not found":

DEBUG1 - 10:53:52.130237000 [9674/9735]: Enable event command for event 'lttng_ust_tracef:*' (in _cmd_enable_event() at ../../../../lttng-tools-2.11.5/src/bin/lttng-sessiond/cmd.c:2036)
DEBUG2 - 10:53:52.130399500 [9674/9735]: Trace UST channel channel0 not found by name (in trace_ust_find_channel_by_name() at ../../../../lttng-tools-2.11.5/src/bin/lttng-sessiond/trace-ust.c:197)
DEBUG1 - 10:53:52.130450125 [9674/9735]: Enabling channel for session auto-20210324-105310 (in cmd_enable_channel() at ../../../../lttng-tools-2.11.5/src/bin/lttng-sessiond/cmd.c:1510)
DEBUG2 - 10:53:52.130496375 [9674/9735]: Trace UST channel channel0 not found by name (in trace_ust_find_channel_by_name() at ../../../../lttng-tools-2.11.5/src/bin/lttng-sessiond/trace-ust.c:197)
DEBUG3 - 10:53:52.130572000 [9674/9735]: Created hashtable size 4 at 0xffff90017170 of type 0 (in lttng_ht_new() at ../../../../lttng-tools-2.11.5/src/common/hashtable/hashtable.c:145)
DEBUG3 - 10:53:52.130619625 [9674/9735]: Created hashtable size 4 at 0xffff900175b0 of type 1 (in lttng_ht_new() at ../../../../lttng-tools-2.11.5/src/common/hashtable/hashtable.c:145)
DEBUG2 - 10:53:52.130666000 [9674/9735]: Trace UST channel channel0 created (in trace_ust_create_channel() at ../../../../lttng-tools-2.11.5/src/bin/lttng-sessiond/trace-ust.c:396)
DEBUG2 - 10:53:52.130713625 [9674/9735]: Channel channel0 is being created for UST with buffer 1 and id 0 (in channel_ust_create() at ../../../../lttng-tools-2.11.5/src/bin/lttng-sessiond/channel.c:461)
DEBUG2 - 10:53:52.130762250 [9674/9735]: Channel channel0 created successfully (in channel_ust_create() at ../../../../lttng-tools-2.11.5/src/bin/lttng-sessiond/channel.c:500)
DEBUG2 - 10:53:52.130807500 [9674/9735]: Trace UST channel channel0 found by name (in trace_ust_find_channel_by_name() at ../../../../lttng-tools-2.11.5/src/bin/lttng-sessiond/trace-ust.c:192)
DEBUG2 - 10:53:52.130851750 [9674/9735]: Trace UST event lttng_ust_tracef:* NOT found (in trace_ust_find_event() at ../../../../lttng-tools-2.11.5/src/bin/lttng-sessiond/trace-ust.c:235)
DEBUG2 - 10:53:52.130896500 [9674/9735]: Trace UST event lttng_ust_tracef:*, loglevel (0,-1) created (in trace_ust_create_event() at ../../../../lttng-tools-2.11.5/src/bin/lttng-sessiond/trace-ust.c:517)
DEBUG1 - 10:53:52.130968375 [9674/9735]: Sending response (size: 20, retcode: Success (10)) (in thread_manage_clients() at ../../../../lttng-tools-2.11.5/src/bin/lttng-sessiond/client.c:2236)
DEBUG1 - 10:53:52.134962250 [9674/9735]: Clean command context structure (in clean_command_ctx() at ../../../../lttng-tools-2.11.5/src/bin/lttng-sessiond/client.c:659)

Not quite sure what to do about that.

Regards,

Brian

On Tue, Apr 25, 2023 at 12:56 AM Brian Hutchinson <b.hutchman@gmail.com> wrote:

So instructions for tracef just say to link in lttng-ust, not dl, I didn't use liblttng-ust.a so maybe I misspoke.

https://lttng.org/docs/v2.11/#doc-tracef

... so that's what I did.

When I compile in a tracepoint with trace provider, I link in lttng-ust and dl. I'm assuming you can mix both tracef and tracepoint calls?

I included a lttng-sessiond -vvv log in original post. Are attachments allowed or should I have posted the context as a plain text post? It's quite big.

When I do lttng start, my app is already running via systemd at startup. Do I need to disable that and start the executable manually after enabling the lttng ust events?

Like I said, kernel trace is working fine, it's UST that appears to not be working for some reason. I have less experience with UST but I've been researching and searching archives and I see others complaining about what looks to be my problem but either don't see a solution or maybe I just don't understand it.

Thanks,

Brian

On Tue, Apr 25, 2023 at 12:13 AM Yitschak, Yehuda <yehuday@amazon.com> wrote:

It might be related to the static linking of lttng-ust
https://lists.lttng.org/pipermail/lttng-dev/2018-January/027705.html

I also went down that path just to figure it’s not supported.

Perhaps lttng documentation should be updated to make a clear note static linking is not supported (or maybe I missed it)

Yehuda

Hi,

I'm on an embedded platform, i.MX8 using Lttng 2.11 with 5.6.69 kernel.

Tracing kernel events with lttng works fine but userspace (tracef and trace provider type tracing) appears to have an issue I can't figure out.

systemd is starting up my application ( a bunch of processes and threads on multiple cpu's) on boot.

I'm statically compiling and linking in lttng-ust. If I use just tracef type tracing, when I do lttng list --userspace I get nothing. If I add in a trace provider and a tracepoint, then I see my trace provider tracepoint and:

     lttng_ust_tracelog:TRACE_DEBUG (loglevel: TRACE_DEBUG (14)) (type: tracepoint)
     lttng_ust_tracelog:TRACE_DEBUG_LINE (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint)
     lttng_ust_tracelog:TRACE_DEBUG_FUNCTION (loglevel: TRACE_DEBUG_FUNCTION (12)) (type: tracepoint)
     lttng_ust_tracelog:TRACE_DEBUG_UNIT (loglevel: TRACE_DEBUG_UNIT (11)) (type: tracepoint)
     lttng_ust_tracelog:TRACE_DEBUG_MODULE (loglevel: TRACE_DEBUG_MODULE (10)) (type: tracepoint)
     lttng_ust_tracelog:TRACE_DEBUG_PROCESS (loglevel: TRACE_DEBUG_PROCESS (9)) (type: tracepoint)
     lttng_ust_tracelog:TRACE_DEBUG_PROGRAM (loglevel: TRACE_DEBUG_PROGRAM (8)) (type: tracepoint)
     lttng_ust_tracelog:TRACE_DEBUG_SYSTEM (loglevel: TRACE_DEBUG_SYSTEM (7)) (type: tracepoint)
     lttng_ust_tracelog:TRACE_INFO (loglevel: TRACE_INFO (6)) (type: tracepoint)
     lttng_ust_tracelog:TRACE_NOTICE (loglevel: TRACE_NOTICE (5)) (type: tracepoint)
     lttng_ust_tracelog:TRACE_WARNING (loglevel: TRACE_WARNING (4)) (type: tracepoint)
     lttng_ust_tracelog:TRACE_ERR (loglevel: TRACE_ERR (3)) (type: tracepoint)
     lttng_ust_tracelog:TRACE_CRIT (loglevel: TRACE_CRIT (2)) (type: tracepoint)
     lttng_ust_tracelog:TRACE_ALERT (loglevel: TRACE_ALERT (1)) (type: tracepoint)
     lttng_ust_tracelog:TRACE_EMERG (loglevel: TRACE_EMERG (0)) (type: tracepoint)
     lttng_ust_tracef:event (loglevel: TRACE_DEBUG (14)) (type: tracepoint)
     lttng_ust_lib:unload (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint)
     lttng_ust_lib:debug_link (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint)
     lttng_ust_lib:build_id (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint)
     lttng_ust_lib:load (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint)
     lttng_ust_statedump:end (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint)
     lttng_ust_statedump:debug_link (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint)
     lttng_ust_statedump:build_id (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint)
     lttng_ust_statedump:bin_info (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint)
     lttng_ust_statedump:start (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint)

When I do lttng start, I get a directory structure in the trace directory but the file sizes do not ever increase. It's just a directory structure with files created and then that's it, no events being written to the files. At first I thought things just weren't linked in right and not working but then discovered how to run lttng-sessiond in -vvv mode and then could see lots of things are going on, tracing appears to be trying to work as there is tons of activity being output by lttng-sessiond, but like I said the trace files aren't being added to.

I looked at the lttng-sessiond log I captured and can't quite make out the problem so attaching it.

I'm probably not doing something right but can't figure out what that is so any pointers appreciated.

Regards,

Brian