* [lttng-dev] I'm still getting empty ust traces using tracef @ 2023-05-11 15:53 Brian Hutchinson via lttng-dev 2023-05-11 16:36 ` Brian Hutchinson via lttng-dev 0 siblings, 1 reply; 15+ messages in thread From: Brian Hutchinson via lttng-dev @ 2023-05-11 15:53 UTC (permalink / raw) To: lttng-dev Hi, I posted a while ago (thread - Using lttng 2.11 and UST doesn't appear to work - getting empty trace files) about this problem I'm having with getting empty trace logs. I've since upgraded to lttng v2.13 and while I can do a simple hello world program with tracef and get events in the log files, my more complicated large multi-threaded app I'm trying to debug is still getting empty log file traces. I can list the user space events in my app. Next I do: lttng enable-event --userspace 'lttng_ust_tracef:*' ... to enable the events, start lttng, start my app, .... and I get a trace directory structure that's empty. I feel like I've read every thread in the archives about people having the same problem. I did try using LD_PRELOAD with various libs thinking that was the problem but so far I'm still getting empty traces. So far I've tried: LD_PRELOAD=liblttng-ust-libc-wrapper.so.1:liblttng-ust-pthread-wrapper.so.1:liblttng-ust-dl.so.1:liblttng-ust-fork.so.1:liblttng-ust-fd.so.1 /usr/local/bin/my_app I guess one question I have is how do I determine which "helper libs" I need to preload? The application I'm working on is made up of a bunch of smaller static libs linked together into one big executable and that is linked with -llttng-ust and -ldl. I'm pretty stuck at the moment. Anyone have any wisdom on what I might be doing wrong or how I can tell why I'm not getting events in the logs? Thanks, Brian _______________________________________________ lttng-dev mailing list lttng-dev@lists.lttng.org https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [lttng-dev] I'm still getting empty ust traces using tracef 2023-05-11 15:53 [lttng-dev] I'm still getting empty ust traces using tracef Brian Hutchinson via lttng-dev @ 2023-05-11 16:36 ` Brian Hutchinson via lttng-dev 2023-05-11 18:13 ` Mathieu Desnoyers via lttng-dev ` (2 more replies) 0 siblings, 3 replies; 15+ messages in thread From: Brian Hutchinson via lttng-dev @ 2023-05-11 16:36 UTC (permalink / raw) To: lttng-dev ... more background. I've always used ltt in the kernel so I don't have much experience with the user side of it and especially multi-threaded, multi-core so I'm probably missing some fundamental concepts that I need to understand. Regards, Brian On Thu, May 11, 2023 at 11:53 AM Brian Hutchinson <b.hutchman@gmail.com> wrote: > > Hi, > > I posted a while ago (thread - Using lttng 2.11 and UST doesn't appear > to work - getting empty trace files) about this problem I'm having > with getting empty trace logs. > > I've since upgraded to lttng v2.13 and while I can do a simple hello > world program with tracef and get events in the log files, my more > complicated large multi-threaded app I'm trying to debug is still > getting empty log file traces. > > I can list the user space events in my app. > > Next I do: > > lttng enable-event --userspace 'lttng_ust_tracef:*' > > ... to enable the events, start lttng, start my app, .... and I get a > trace directory structure that's empty. > > I feel like I've read every thread in the archives about people having > the same problem. > > I did try using LD_PRELOAD with various libs thinking that was the > problem but so far I'm still getting empty traces. > > So far I've tried: > > LD_PRELOAD=liblttng-ust-libc-wrapper.so.1:liblttng-ust-pthread-wrapper.so.1:liblttng-ust-dl.so.1:liblttng-ust-fork.so.1:liblttng-ust-fd.so.1 > /usr/local/bin/my_app > > I guess one question I have is how do I determine which "helper libs" > I need to preload? > > The application I'm working on is made up of a bunch of smaller static > libs linked together into one big executable and that is linked with > -llttng-ust and -ldl. > > I'm pretty stuck at the moment. Anyone have any wisdom on what I > might be doing wrong or how I can tell why I'm not getting events in > the logs? > > Thanks, > > Brian _______________________________________________ lttng-dev mailing list lttng-dev@lists.lttng.org https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [lttng-dev] I'm still getting empty ust traces using tracef 2023-05-11 16:36 ` Brian Hutchinson via lttng-dev @ 2023-05-11 18:13 ` Mathieu Desnoyers via lttng-dev 2023-05-11 18:15 ` Mathieu Desnoyers via lttng-dev 2023-05-11 18:38 ` Brian Hutchinson via lttng-dev 2023-05-11 18:14 ` Michael Jeanson via lttng-dev 2023-05-11 18:57 ` Brian Hutchinson via lttng-dev 2 siblings, 2 replies; 15+ messages in thread From: Mathieu Desnoyers via lttng-dev @ 2023-05-11 18:13 UTC (permalink / raw) To: Brian Hutchinson, lttng-dev On 2023-05-11 12:36, Brian Hutchinson via lttng-dev wrote: > ... more background. I've always used ltt in the kernel so I don't > have much experience with the user side of it and especially > multi-threaded, multi-core so I'm probably missing some fundamental > concepts that I need to understand. Which are the exact versions of LTTng-UST and LTTng-Tools you are using now ? (2.13.N or which git commit ?) Thanks, Mathieu > > Regards, > > Brian > > On Thu, May 11, 2023 at 11:53 AM Brian Hutchinson <b.hutchman@gmail.com> wrote: >> >> Hi, >> >> I posted a while ago (thread - Using lttng 2.11 and UST doesn't appear >> to work - getting empty trace files) about this problem I'm having >> with getting empty trace logs. >> >> I've since upgraded to lttng v2.13 and while I can do a simple hello >> world program with tracef and get events in the log files, my more >> complicated large multi-threaded app I'm trying to debug is still >> getting empty log file traces. >> >> I can list the user space events in my app. >> >> Next I do: >> >> lttng enable-event --userspace 'lttng_ust_tracef:*' >> >> ... to enable the events, start lttng, start my app, .... and I get a >> trace directory structure that's empty. >> >> I feel like I've read every thread in the archives about people having >> the same problem. >> >> I did try using LD_PRELOAD with various libs thinking that was the >> problem but so far I'm still getting empty traces. >> >> So far I've tried: >> >> LD_PRELOAD=liblttng-ust-libc-wrapper.so.1:liblttng-ust-pthread-wrapper.so.1:liblttng-ust-dl.so.1:liblttng-ust-fork.so.1:liblttng-ust-fd.so.1 >> /usr/local/bin/my_app >> >> I guess one question I have is how do I determine which "helper libs" >> I need to preload? >> >> The application I'm working on is made up of a bunch of smaller static >> libs linked together into one big executable and that is linked with >> -llttng-ust and -ldl. >> >> I'm pretty stuck at the moment. Anyone have any wisdom on what I >> might be doing wrong or how I can tell why I'm not getting events in >> the logs? >> >> Thanks, >> >> Brian > _______________________________________________ > lttng-dev mailing list > lttng-dev@lists.lttng.org > https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev -- Mathieu Desnoyers EfficiOS Inc. https://www.efficios.com _______________________________________________ lttng-dev mailing list lttng-dev@lists.lttng.org https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [lttng-dev] I'm still getting empty ust traces using tracef 2023-05-11 18:13 ` Mathieu Desnoyers via lttng-dev @ 2023-05-11 18:15 ` Mathieu Desnoyers via lttng-dev [not found] ` <CAFZh4h9M0eA1SpHhfquoyrqtwRhHV54f0O4hCkkiojwUmPTC9Q@mail.gmail.com> 2023-06-20 22:02 ` Brian Hutchinson via lttng-dev 2023-05-11 18:38 ` Brian Hutchinson via lttng-dev 1 sibling, 2 replies; 15+ messages in thread From: Mathieu Desnoyers via lttng-dev @ 2023-05-11 18:15 UTC (permalink / raw) To: Brian Hutchinson, lttng-dev On 2023-05-11 14:13, Mathieu Desnoyers via lttng-dev wrote: > On 2023-05-11 12:36, Brian Hutchinson via lttng-dev wrote: >> ... more background. I've always used ltt in the kernel so I don't >> have much experience with the user side of it and especially >> multi-threaded, multi-core so I'm probably missing some fundamental >> concepts that I need to understand. > > Which are the exact versions of LTTng-UST and LTTng-Tools you are using > now ? (2.13.N or which git commit ?) > Also, can you try using lttng-ust stable-2.13 branch, which includes the following commit ? commit be2ca8b563bab81be15cbce7b9f52422369f79f7 Author: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Date: Tue Feb 21 14:29:49 2023 -0500 Fix: Reevaluate LTTNG_UST_TRACEPOINT_DEFINE each time tracepoint.h is included Fix issues with missing symbols in use-cases where tracef.h is included before defining LTTNG_UST_TRACEPOINT_DEFINE, e.g.: #include <lttng/tracef.h> #define LTTNG_UST_TRACEPOINT_DEFINE #include <provider.h> It is caused by the fact that tracef.h includes tracepoint.h in a context which has LTTNG_UST_TRACEPOINT_DEFINE undefined, and this is not re-evaluated for the following includes. Fix this by lifting the definition code in tracepoint.h outside of the header include guards, and #undef the old LTTNG_UST__DEFINE_TRACEPOINT before re-defining it to its new semantic. Use a new _LTTNG_UST_TRACEPOINT_DEFINE_ONCE include guard within the LTTNG_UST_TRACEPOINT_DEFINE defined case to ensure symbols are not duplicated. Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Change-Id: I0ef720435003a7ca0bfcf29d7bf27866c5ff8678 Thanks, Mathieu > Thanks, > > Mathieu > >> >> Regards, >> >> Brian >> >> On Thu, May 11, 2023 at 11:53 AM Brian Hutchinson >> <b.hutchman@gmail.com> wrote: >>> >>> Hi, >>> >>> I posted a while ago (thread - Using lttng 2.11 and UST doesn't appear >>> to work - getting empty trace files) about this problem I'm having >>> with getting empty trace logs. >>> >>> I've since upgraded to lttng v2.13 and while I can do a simple hello >>> world program with tracef and get events in the log files, my more >>> complicated large multi-threaded app I'm trying to debug is still >>> getting empty log file traces. >>> >>> I can list the user space events in my app. >>> >>> Next I do: >>> >>> lttng enable-event --userspace 'lttng_ust_tracef:*' >>> >>> ... to enable the events, start lttng, start my app, .... and I get a >>> trace directory structure that's empty. >>> >>> I feel like I've read every thread in the archives about people having >>> the same problem. >>> >>> I did try using LD_PRELOAD with various libs thinking that was the >>> problem but so far I'm still getting empty traces. >>> >>> So far I've tried: >>> >>> LD_PRELOAD=liblttng-ust-libc-wrapper.so.1:liblttng-ust-pthread-wrapper.so.1:liblttng-ust-dl.so.1:liblttng-ust-fork.so.1:liblttng-ust-fd.so.1 >>> /usr/local/bin/my_app >>> >>> I guess one question I have is how do I determine which "helper libs" >>> I need to preload? >>> >>> The application I'm working on is made up of a bunch of smaller static >>> libs linked together into one big executable and that is linked with >>> -llttng-ust and -ldl. >>> >>> I'm pretty stuck at the moment. Anyone have any wisdom on what I >>> might be doing wrong or how I can tell why I'm not getting events in >>> the logs? >>> >>> Thanks, >>> >>> Brian >> _______________________________________________ >> lttng-dev mailing list >> lttng-dev@lists.lttng.org >> https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev > -- Mathieu Desnoyers EfficiOS Inc. https://www.efficios.com _______________________________________________ lttng-dev mailing list lttng-dev@lists.lttng.org https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev ^ permalink raw reply [flat|nested] 15+ messages in thread
[parent not found: <CAFZh4h9M0eA1SpHhfquoyrqtwRhHV54f0O4hCkkiojwUmPTC9Q@mail.gmail.com>]
[parent not found: <fdbcfbd9-742d-dc61-3667-48a00130d6aa@efficios.com>]
* Re: [lttng-dev] I'm still getting empty ust traces using tracef [not found] ` <fdbcfbd9-742d-dc61-3667-48a00130d6aa@efficios.com> @ 2023-05-12 13:34 ` Mathieu Desnoyers via lttng-dev 2023-05-12 14:52 ` Brian Hutchinson via lttng-dev 1 sibling, 0 replies; 15+ messages in thread From: Mathieu Desnoyers via lttng-dev @ 2023-05-12 13:34 UTC (permalink / raw) To: Brian Hutchinson; +Cc: lttng-dev [adding back the mailing list] On 2023-05-12 09:33, Mathieu Desnoyers wrote: > On 2023-05-12 00:10, Brian Hutchinson wrote: >> Hmm, I missed this earlier somehow. >> >> So, I'm not the greatest at updating OE and Yocto recipes. I'm >> currently using this recipe: >> http://cgit.openembedded.org/openembedded-core/tree/meta/recipes-kernel/lttng/lttng-ust_2.13.5.bb?h=master >> >> ... and it looks like the commit you are talking about is newer. >> >> I always think, oh, I'll just update the source URI in the recipe but >> it's never that simple ... and there are patches in the recipe etc. >> >> I've got a sdk (external toolchain) built for my embedded platform. >> Would it be too hard to just download stable-2.13 of everything and >> cross compile it outside of Yocto? >> >> What do you suggest? >> >> And do I need to do anything besides just get 2.13 stable working? I >> was kind of confused if I need to put a #define >> LTTNG_TRACEPOINT_DEFINE somewhere in my code. I'm not using a >> tracepoint provider packages at this point > > Hi Brian, > > You might want to provide a trimmed-down reproducer of your issue: > example .c compile unit instrumented with tracepoints, example .c > compile unit containing the tracepoint probes, and the log of the > console when this application is run with LTTNG_UST_DEBUG=1. > > Thanks, > > Mathieu > -- Mathieu Desnoyers EfficiOS Inc. https://www.efficios.com _______________________________________________ lttng-dev mailing list lttng-dev@lists.lttng.org https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [lttng-dev] I'm still getting empty ust traces using tracef [not found] ` <fdbcfbd9-742d-dc61-3667-48a00130d6aa@efficios.com> 2023-05-12 13:34 ` Mathieu Desnoyers via lttng-dev @ 2023-05-12 14:52 ` Brian Hutchinson via lttng-dev 2023-05-12 14:57 ` Mathieu Desnoyers via lttng-dev 1 sibling, 1 reply; 15+ messages in thread From: Brian Hutchinson via lttng-dev @ 2023-05-12 14:52 UTC (permalink / raw) To: Mathieu Desnoyers; +Cc: lttng-dev Hi Mathieu, On Fri, May 12, 2023 at 9:33 AM Mathieu Desnoyers <mathieu.desnoyers@efficios.com> wrote: > > On 2023-05-12 00:10, Brian Hutchinson wrote: > > Hmm, I missed this earlier somehow. > > > > So, I'm not the greatest at updating OE and Yocto recipes. I'm > > currently using this recipe: > > http://cgit.openembedded.org/openembedded-core/tree/meta/recipes-kernel/lttng/lttng-ust_2.13.5.bb?h=master > > > > ... and it looks like the commit you are talking about is newer. > > > > I always think, oh, I'll just update the source URI in the recipe but > > it's never that simple ... and there are patches in the recipe etc. > > > > I've got a sdk (external toolchain) built for my embedded platform. > > Would it be too hard to just download stable-2.13 of everything and > > cross compile it outside of Yocto? > > > > What do you suggest? > > > > And do I need to do anything besides just get 2.13 stable working? I > > was kind of confused if I need to put a #define > > LTTNG_TRACEPOINT_DEFINE somewhere in my code. I'm not using a > > tracepoint provider packages at this point > > Hi Brian, > > You might want to provide a trimmed-down reproducer of your issue: > example .c compile unit instrumented with tracepoints, example .c > compile unit containing the tracepoint probes, and the log of the > console when this application is run with LTTNG_UST_DEBUG=1. The code has two different areas where I'm trying to use tracef. The way the app is put together, each of these areas end up becoming static libs that all get lumped together to make the final executable (which is then linked with -llttng-ust and -ldl). If I'm reading between the lines correctly with respect to the commit you pointed out (that I'm missing), if I reduce the inclusion of I #include <lttng/tracef.h> to one instance (like with the hello world that worked), I'm thinking the version I have might work. I don't know how I could trim down the large multi threaded app I'm trying to debug to share. Another dynamic I should mention in full disclosure, the app in question has been ported from a different OS and was on a single core cpu. The new host ( imx8) is a quad core A53 and since the app wasn't written for multicore, the cpu's are isolated and systemd is starting the app on cpu 0 but once it's up it switches it's affinity to cpu 1 so I don't know if that's a factor here or not so just mentioning it. I did try with LTTNG_UST_DEBUG=1 last night and it didn't put out much: export LTTNG_UST_DEBUG=1 # systemctl start my_app #lttng create my_tc_trace --output=/tmp/my_tc_trace Spawning a session daemon libringbuffer-clients[711/711] : LTT : ltt ring buffer client "relay-metadata-mmap" init (in lttng_ring_buffer_metadata_client_init() at ../../../lttng-ust-2.13.5/src/common/ringbuffer-clients/metadata-template.h:364) libringbuffer-clients[711/711]: LTT : ltt ring buffer client "relay-overwrite-mmap" init (in lttng_ring_buffer_client_overwrite_init() at ../../../lttng-ust-2.13.5/src/common/ringbuffer-clients/template.h:826) libringbuffer-clients[711/711]: LTT : ltt ring buffer client "relay-overwrite-rt-mmap" init (in lttng_ring_buffer_client_overwrite_rt_init() at ../../../lttng-ust-2.13.5/src/common/ringbuffer-clients/template.h:826) libringbuffer-clients[711/711]: LTT : ltt ring buffer client "relay-discard-mmap" init (in lttng_ring_buffer_client_discard_init() at ../../../lttng-ust-2.13.5/src/common/ringbuffer-clients/template.h:826) libringbuffer-clients[711/711]: LTT : ltt ring buffer client "relay-discard-rt-mmap" init (in lttng_ring_buffer_client_discard_rt_init() at ../../../lttng-ust-2.13.5/src/common/ringbuffer-clients/template.h:826) [ 179.384456] LTTng: Loaded modules v2.13.9 (Nordicit�é) [ 179.390366] LTTng: Experimental bitwise enum enabled. libringbuffer-clients[711/711]: LTT : ltt ring buffer client "relay-discard-rt-mmap" exit (in lttng_ring_buffer_client_discard_rt_exit() at ../../../lttng-ust-2.13.5/src/common/ringbuffer-clients/template.h:833) libringbuffer-clients[711/711]: LTT : ltt ring buffer client "relay-discard-mmap" exit (in lttng_ring_buffer_client_discard_exit() at ../../../lttng-ust-2.13.5/src/common/ringbuffer-clients/template.h:833) libringbuffer-clients[711/711]: LTT : ltt ring buffer client "relay-overwrite-rt-mmap" exit (in lttng_ring_buffer_client_overwrite_rt_exit() at ../../../lttng-ust-2.13.5/src/common/ringbuffer-clients/template.h:833) libringbuffer-clients[711/711]: LTT : ltt ring buffer client "relay-overwrite-mmap" exit (in lttng_ring_buffer_client_overwrite_exit() at ../../../lttng-ust-2.13.5/src/common/ringbuffer-clients/template.h:833) libringbuffer-clients[711/711]: LTT : ltt ring buffer client "relay-metadata-mmap" exit (in lttng_ring_buffer_metadata_client_exit() at ../../../lttng-ust-2.13.5/src/common/ringbuffer-clients/metadata-template.h:371) Session my_tc_trace created. Traces will be output to /tmp/my_tc_trace # lttng enable-event --userspace 'lttng_ust_tracef:*' ust event lttng_ust_tracef:* created in channel channel0 # lttng start Tracing started for session my_tc_trace # lttng destroy Destroying session my_tc_trace.. Session my_tc_trace destroyed root@localhost:/tmp/my_tc_trace/ust/uid/0# cd 64-bit/ root@localhost:/tmp/my_tc_trace/ust/uid/0/64-bit# ls channel0_0 channel0_1 channel0_2 channel0_3 index metadata root@localhost:/tmp/my_tc_trace/ust/uid/0/64-bit# ls -al total 20 drwxrwx--- 3 root root 160 Sep 20 10:48 . drwxrwx--- 3 root root 60 Sep 20 10:48 .. -rw-rw---- 1 root root 4096 Sep 20 10:48 channel0_0 -rw-rw---- 1 root root 4096 Sep 20 10:48 channel0_1 -rw-rw---- 1 root root 4096 Sep 20 10:48 channel0_2 -rw-rw---- 1 root root 4096 Sep 20 10:48 channel0_3 drwxrwx--- 2 root root 120 Sep 20 10:48 index -rw-rw---- 1 root root 4096 Sep 20 10:48 metadata root@localhost:/tmp/my_tc_trace/ust/uid/0/64-bit# cd index/ root@localhost:/tmp/my_tc_trace/ust/uid/0/64-bit/index# ls channel0_0.idx channel0_1.idx channel0_2.idx channel0_3.idx root@localhost:/tmp/my_tc_trace/ust/uid/0/64-bit/index# ls -al total 16 drwxrwx--- 2 root root 120 Sep 20 10:48 . drwxrwx--- 3 root root 160 Sep 20 10:48 .. -rw-rw---- 1 root root 88 Sep 20 10:48 channel0_0.idx -rw-rw---- 1 root root 88 Sep 20 10:48 channel0_1.idx -rw-rw---- 1 root root 88 Sep 20 10:48 channel0_2.idx -rw-rw---- 1 root root 88 Sep 20 10:48 channel0_3.idx No events in the logs, if it were working there would be tons of stuff in there. But it sounds like my main problem is I need stable-2.13 and I need to figure out the best way to do that. Compile from lttng source outside of yocto (probably not a good idea with all the dependencies that would also need to be cross compiled) with my generated sdk for the imx8 or try to update the OE recipe that I'm currently using. Regards, Brian > > Thanks, > > Mathieu > > -- > Mathieu Desnoyers > EfficiOS Inc. > https://www.efficios.com > _______________________________________________ lttng-dev mailing list lttng-dev@lists.lttng.org https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [lttng-dev] I'm still getting empty ust traces using tracef 2023-05-12 14:52 ` Brian Hutchinson via lttng-dev @ 2023-05-12 14:57 ` Mathieu Desnoyers via lttng-dev 0 siblings, 0 replies; 15+ messages in thread From: Mathieu Desnoyers via lttng-dev @ 2023-05-12 14:57 UTC (permalink / raw) To: Brian Hutchinson; +Cc: lttng-dev On 2023-05-12 10:52, Brian Hutchinson wrote: > Hi Mathieu, > > On Fri, May 12, 2023 at 9:33 AM Mathieu Desnoyers > <mathieu.desnoyers@efficios.com> wrote: >> >> On 2023-05-12 00:10, Brian Hutchinson wrote: >>> Hmm, I missed this earlier somehow. >>> >>> So, I'm not the greatest at updating OE and Yocto recipes. I'm >>> currently using this recipe: >>> http://cgit.openembedded.org/openembedded-core/tree/meta/recipes-kernel/lttng/lttng-ust_2.13.5.bb?h=master >>> >>> ... and it looks like the commit you are talking about is newer. >>> >>> I always think, oh, I'll just update the source URI in the recipe but >>> it's never that simple ... and there are patches in the recipe etc. >>> >>> I've got a sdk (external toolchain) built for my embedded platform. >>> Would it be too hard to just download stable-2.13 of everything and >>> cross compile it outside of Yocto? >>> >>> What do you suggest? >>> >>> And do I need to do anything besides just get 2.13 stable working? I >>> was kind of confused if I need to put a #define >>> LTTNG_TRACEPOINT_DEFINE somewhere in my code. I'm not using a >>> tracepoint provider packages at this point >> >> Hi Brian, >> >> You might want to provide a trimmed-down reproducer of your issue: >> example .c compile unit instrumented with tracepoints, example .c >> compile unit containing the tracepoint probes, and the log of the >> console when this application is run with LTTNG_UST_DEBUG=1. > > The code has two different areas where I'm trying to use tracef. The > way the app is put together, each of these areas end up becoming > static libs that all get lumped together to make the final executable > (which is then linked with -llttng-ust and -ldl). > > If I'm reading between the lines correctly with respect to the commit > you pointed out (that I'm missing), if I reduce the inclusion of I > #include <lttng/tracef.h> to one instance (like with the hello world > that worked), I'm thinking the version I have might work. > > I don't know how I could trim down the large multi threaded app I'm > trying to debug to share. > > Another dynamic I should mention in full disclosure, the app in > question has been ported from a different OS and was on a single core > cpu. The new host ( imx8) is a quad core A53 and since the app wasn't > written for multicore, the cpu's are isolated and systemd is starting > the app on cpu 0 but once it's up it switches it's affinity to cpu 1 > so I don't know if that's a factor here or not so just mentioning it. > > I did try with LTTNG_UST_DEBUG=1 last night and it didn't put out much: > > export LTTNG_UST_DEBUG=1 > # systemctl start my_app I suspect that because you run your application under systemctl, we are not seeing the console output from the application. The console output below appears to come from liblttng-ust-ctl.so linked within lttng-sessiond/consumerd, not the application. Can you find a way to run your application and capture the console output ? Thanks, Mathieu > > #lttng create my_tc_trace --output=/tmp/my_tc_trace > Spawning a session daemon > libringbuffer-clients[711/711] > : LTT : ltt ring buffer client > "relay-metadata-mmap" init > (in lttng_ring_buffer_metadata_client_init() at > ../../../lttng-ust-2.13.5/src/common/ringbuffer-clients/metadata-template.h:364) > libringbuffer-clients[711/711]: LTT : ltt ring buffer client > "relay-overwrite-mmap" init > (in lttng_ring_buffer_client_overwrite_init() at > ../../../lttng-ust-2.13.5/src/common/ringbuffer-clients/template.h:826) > libringbuffer-clients[711/711]: LTT : ltt ring buffer client > "relay-overwrite-rt-mmap" init > (in lttng_ring_buffer_client_overwrite_rt_init() at > ../../../lttng-ust-2.13.5/src/common/ringbuffer-clients/template.h:826) > libringbuffer-clients[711/711]: LTT : ltt ring buffer client > "relay-discard-mmap" init > (in lttng_ring_buffer_client_discard_init() at > ../../../lttng-ust-2.13.5/src/common/ringbuffer-clients/template.h:826) > libringbuffer-clients[711/711]: LTT : ltt ring buffer client > "relay-discard-rt-mmap" init > (in lttng_ring_buffer_client_discard_rt_init() at > ../../../lttng-ust-2.13.5/src/common/ringbuffer-clients/template.h:826) > [ 179.384456] LTTng: Loaded modules v2.13.9 (Nordicit�é) > [ 179.390366] LTTng: Experimental bitwise enum enabled. > libringbuffer-clients[711/711]: LTT : ltt ring buffer client > "relay-discard-rt-mmap" exit > (in lttng_ring_buffer_client_discard_rt_exit() at > ../../../lttng-ust-2.13.5/src/common/ringbuffer-clients/template.h:833) > libringbuffer-clients[711/711]: LTT : ltt ring buffer client > "relay-discard-mmap" exit > (in lttng_ring_buffer_client_discard_exit() at > ../../../lttng-ust-2.13.5/src/common/ringbuffer-clients/template.h:833) > libringbuffer-clients[711/711]: LTT : ltt ring buffer client > "relay-overwrite-rt-mmap" exit > (in lttng_ring_buffer_client_overwrite_rt_exit() at > ../../../lttng-ust-2.13.5/src/common/ringbuffer-clients/template.h:833) > libringbuffer-clients[711/711]: LTT : ltt ring buffer client > "relay-overwrite-mmap" exit > (in lttng_ring_buffer_client_overwrite_exit() at > ../../../lttng-ust-2.13.5/src/common/ringbuffer-clients/template.h:833) > libringbuffer-clients[711/711]: LTT : ltt ring buffer client > "relay-metadata-mmap" exit > (in lttng_ring_buffer_metadata_client_exit() at > ../../../lttng-ust-2.13.5/src/common/ringbuffer-clients/metadata-template.h:371) > Session my_tc_trace created. > Traces will be output to /tmp/my_tc_trace > > # lttng enable-event --userspace 'lttng_ust_tracef:*' > ust event lttng_ust_tracef:* created in channel channel0 > > # lttng start > Tracing started for session my_tc_trace > > # lttng destroy > Destroying session my_tc_trace.. > Session my_tc_trace destroyed > > root@localhost:/tmp/my_tc_trace/ust/uid/0# cd 64-bit/ > root@localhost:/tmp/my_tc_trace/ust/uid/0/64-bit# ls > channel0_0 channel0_1 channel0_2 channel0_3 index metadata > root@localhost:/tmp/my_tc_trace/ust/uid/0/64-bit# ls -al > total 20 > drwxrwx--- 3 root root 160 Sep 20 10:48 . > drwxrwx--- 3 root root 60 Sep 20 10:48 .. > -rw-rw---- 1 root root 4096 Sep 20 10:48 channel0_0 > -rw-rw---- 1 root root 4096 Sep 20 10:48 channel0_1 > -rw-rw---- 1 root root 4096 Sep 20 10:48 channel0_2 > -rw-rw---- 1 root root 4096 Sep 20 10:48 channel0_3 > drwxrwx--- 2 root root 120 Sep 20 10:48 index > -rw-rw---- 1 root root 4096 Sep 20 10:48 metadata > root@localhost:/tmp/my_tc_trace/ust/uid/0/64-bit# cd index/ > root@localhost:/tmp/my_tc_trace/ust/uid/0/64-bit/index# ls > channel0_0.idx channel0_1.idx channel0_2.idx channel0_3.idx > root@localhost:/tmp/my_tc_trace/ust/uid/0/64-bit/index# ls -al > total 16 > drwxrwx--- 2 root root 120 Sep 20 10:48 . > drwxrwx--- 3 root root 160 Sep 20 10:48 .. > -rw-rw---- 1 root root 88 Sep 20 10:48 channel0_0.idx > -rw-rw---- 1 root root 88 Sep 20 10:48 channel0_1.idx > -rw-rw---- 1 root root 88 Sep 20 10:48 channel0_2.idx > -rw-rw---- 1 root root 88 Sep 20 10:48 channel0_3.idx > > No events in the logs, if it were working there would be tons of stuff in there. > > But it sounds like my main problem is I need stable-2.13 and I need to > figure out the best way to do that. Compile from lttng source outside > of yocto (probably not a good idea with all the dependencies that > would also need to be cross compiled) with my generated sdk for the > imx8 or try to update the OE recipe that I'm currently using. > > Regards, > > Brian > > > > > >> >> Thanks, >> >> Mathieu >> >> -- >> Mathieu Desnoyers >> EfficiOS Inc. >> https://www.efficios.com >> -- Mathieu Desnoyers EfficiOS Inc. https://www.efficios.com _______________________________________________ lttng-dev mailing list lttng-dev@lists.lttng.org https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [lttng-dev] I'm still getting empty ust traces using tracef 2023-05-11 18:15 ` Mathieu Desnoyers via lttng-dev [not found] ` <CAFZh4h9M0eA1SpHhfquoyrqtwRhHV54f0O4hCkkiojwUmPTC9Q@mail.gmail.com> @ 2023-06-20 22:02 ` Brian Hutchinson via lttng-dev 2023-06-21 20:21 ` Mathieu Desnoyers via lttng-dev 1 sibling, 1 reply; 15+ messages in thread From: Brian Hutchinson via lttng-dev @ 2023-06-20 22:02 UTC (permalink / raw) To: Mathieu Desnoyers; +Cc: lttng-dev On Thu, May 11, 2023 at 2:14 PM Mathieu Desnoyers <mathieu.desnoyers@efficios.com> wrote: > > On 2023-05-11 14:13, Mathieu Desnoyers via lttng-dev wrote: > > On 2023-05-11 12:36, Brian Hutchinson via lttng-dev wrote: > >> ... more background. I've always used ltt in the kernel so I don't > >> have much experience with the user side of it and especially > >> multi-threaded, multi-core so I'm probably missing some fundamental > >> concepts that I need to understand. > > > > Which are the exact versions of LTTng-UST and LTTng-Tools you are using > > now ? (2.13.N or which git commit ?) > > > > Also, can you try using lttng-ust stable-2.13 branch, which includes the following commit ? > > commit be2ca8b563bab81be15cbce7b9f52422369f79f7 > Author: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> > Date: Tue Feb 21 14:29:49 2023 -0500 > > Fix: Reevaluate LTTNG_UST_TRACEPOINT_DEFINE each time tracepoint.h is included > > Fix issues with missing symbols in use-cases where tracef.h is included > before defining LTTNG_UST_TRACEPOINT_DEFINE, e.g.: > > #include <lttng/tracef.h> > #define LTTNG_UST_TRACEPOINT_DEFINE > #include <provider.h> > > It is caused by the fact that tracef.h includes tracepoint.h in a > context which has LTTNG_UST_TRACEPOINT_DEFINE undefined, and this is not > re-evaluated for the following includes. > > Fix this by lifting the definition code in tracepoint.h outside of the > header include guards, and #undef the old LTTNG_UST__DEFINE_TRACEPOINT > before re-defining it to its new semantic. Use a new > _LTTNG_UST_TRACEPOINT_DEFINE_ONCE include guard within the > LTTNG_UST_TRACEPOINT_DEFINE defined case to ensure symbols are not > duplicated. > > Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> > Change-Id: I0ef720435003a7ca0bfcf29d7bf27866c5ff8678 > I applied this patch and if I use "tracef" type calls in our application that is made up of a bunch of static libs ... the UST trace calls work. I verified that traces that were called from several different static libs all worked. But as soon as I include a "tracepoint" style tracepoint (that uses trace provider include files etc.) then doing a "lttng list -u" returns "None" for UST events. Is there some kind of rule that says a file can't use both tracef and tracepoint calls? Is there something special you have to do to use tracef and tracepoints in same file? Doing so appears to have broken everything. Thanks, Brian _______________________________________________ lttng-dev mailing list lttng-dev@lists.lttng.org https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [lttng-dev] I'm still getting empty ust traces using tracef 2023-06-20 22:02 ` Brian Hutchinson via lttng-dev @ 2023-06-21 20:21 ` Mathieu Desnoyers via lttng-dev 2023-06-21 22:02 ` Brian Hutchinson via lttng-dev 0 siblings, 1 reply; 15+ messages in thread From: Mathieu Desnoyers via lttng-dev @ 2023-06-21 20:21 UTC (permalink / raw) To: Brian Hutchinson; +Cc: lttng-dev On 6/20/23 18:02, Brian Hutchinson wrote: > On Thu, May 11, 2023 at 2:14 PM Mathieu Desnoyers > <mathieu.desnoyers@efficios.com> wrote: >> >> On 2023-05-11 14:13, Mathieu Desnoyers via lttng-dev wrote: >>> On 2023-05-11 12:36, Brian Hutchinson via lttng-dev wrote: >>>> ... more background. I've always used ltt in the kernel so I don't >>>> have much experience with the user side of it and especially >>>> multi-threaded, multi-core so I'm probably missing some fundamental >>>> concepts that I need to understand. >>> >>> Which are the exact versions of LTTng-UST and LTTng-Tools you are using >>> now ? (2.13.N or which git commit ?) >>> >> >> Also, can you try using lttng-ust stable-2.13 branch, which includes the following commit ? >> >> commit be2ca8b563bab81be15cbce7b9f52422369f79f7 >> Author: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> >> Date: Tue Feb 21 14:29:49 2023 -0500 >> >> Fix: Reevaluate LTTNG_UST_TRACEPOINT_DEFINE each time tracepoint.h is included >> >> Fix issues with missing symbols in use-cases where tracef.h is included >> before defining LTTNG_UST_TRACEPOINT_DEFINE, e.g.: >> >> #include <lttng/tracef.h> >> #define LTTNG_UST_TRACEPOINT_DEFINE >> #include <provider.h> >> >> It is caused by the fact that tracef.h includes tracepoint.h in a >> context which has LTTNG_UST_TRACEPOINT_DEFINE undefined, and this is not >> re-evaluated for the following includes. >> >> Fix this by lifting the definition code in tracepoint.h outside of the >> header include guards, and #undef the old LTTNG_UST__DEFINE_TRACEPOINT >> before re-defining it to its new semantic. Use a new >> _LTTNG_UST_TRACEPOINT_DEFINE_ONCE include guard within the >> LTTNG_UST_TRACEPOINT_DEFINE defined case to ensure symbols are not >> duplicated. >> >> Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> >> Change-Id: I0ef720435003a7ca0bfcf29d7bf27866c5ff8678 >> > > I applied this patch and if I use "tracef" type calls in our > application that is made up of a bunch of static libs ... the UST > trace calls work. I verified that traces that were called from > several different static libs all worked. > > But as soon as I include a "tracepoint" style tracepoint (that uses > trace provider include files etc.) then doing a "lttng list -u" > returns "None" for UST events. > > Is there some kind of rule that says a file can't use both tracef and > tracepoint calls? Is there something special you have to do to use > tracef and tracepoints in same file? Doing so appears to have broken > everything. It should just work. Can you provide a minimal example of the compile unit having this issue ? Also you mention "static libs". Make sure you do *not* define "LTTNG_UST_TRACEPOINT_PROBE_DYNAMIC_LINKAGE" in this case. See the lttng-ust(3) man page for details (section "Statically linking the tracepoint provider"). Thanks, Mathieu -- Mathieu Desnoyers EfficiOS Inc. https://www.efficios.com _______________________________________________ lttng-dev mailing list lttng-dev@lists.lttng.org https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [lttng-dev] I'm still getting empty ust traces using tracef 2023-06-21 20:21 ` Mathieu Desnoyers via lttng-dev @ 2023-06-21 22:02 ` Brian Hutchinson via lttng-dev 2023-06-28 22:02 ` Brian Hutchinson via lttng-dev 0 siblings, 1 reply; 15+ messages in thread From: Brian Hutchinson via lttng-dev @ 2023-06-21 22:02 UTC (permalink / raw) To: Mathieu Desnoyers; +Cc: lttng-dev On Wed, Jun 21, 2023 at 4:21 PM Mathieu Desnoyers <mathieu.desnoyers@efficios.com> wrote: > > On 6/20/23 18:02, Brian Hutchinson wrote: > > On Thu, May 11, 2023 at 2:14 PM Mathieu Desnoyers > > <mathieu.desnoyers@efficios.com> wrote: > >> > >> On 2023-05-11 14:13, Mathieu Desnoyers via lttng-dev wrote: > >>> On 2023-05-11 12:36, Brian Hutchinson via lttng-dev wrote: > >>>> ... more background. I've always used ltt in the kernel so I don't > >>>> have much experience with the user side of it and especially > >>>> multi-threaded, multi-core so I'm probably missing some fundamental > >>>> concepts that I need to understand. > >>> > >>> Which are the exact versions of LTTng-UST and LTTng-Tools you are using > >>> now ? (2.13.N or which git commit ?) > >>> > >> > >> Also, can you try using lttng-ust stable-2.13 branch, which includes the following commit ? > >> > >> commit be2ca8b563bab81be15cbce7b9f52422369f79f7 > >> Author: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> > >> Date: Tue Feb 21 14:29:49 2023 -0500 > >> > >> Fix: Reevaluate LTTNG_UST_TRACEPOINT_DEFINE each time tracepoint.h is included > >> > >> Fix issues with missing symbols in use-cases where tracef.h is included > >> before defining LTTNG_UST_TRACEPOINT_DEFINE, e.g.: > >> > >> #include <lttng/tracef.h> > >> #define LTTNG_UST_TRACEPOINT_DEFINE > >> #include <provider.h> > >> > >> It is caused by the fact that tracef.h includes tracepoint.h in a > >> context which has LTTNG_UST_TRACEPOINT_DEFINE undefined, and this is not > >> re-evaluated for the following includes. > >> > >> Fix this by lifting the definition code in tracepoint.h outside of the > >> header include guards, and #undef the old LTTNG_UST__DEFINE_TRACEPOINT > >> before re-defining it to its new semantic. Use a new > >> _LTTNG_UST_TRACEPOINT_DEFINE_ONCE include guard within the > >> LTTNG_UST_TRACEPOINT_DEFINE defined case to ensure symbols are not > >> duplicated. > >> > >> Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> > >> Change-Id: I0ef720435003a7ca0bfcf29d7bf27866c5ff8678 > >> > > > > I applied this patch and if I use "tracef" type calls in our > > application that is made up of a bunch of static libs ... the UST > > trace calls work. I verified that traces that were called from > > several different static libs all worked. > > > > But as soon as I include a "tracepoint" style tracepoint (that uses > > trace provider include files etc.) then doing a "lttng list -u" > > returns "None" for UST events. > > > > Is there some kind of rule that says a file can't use both tracef and > > tracepoint calls? Is there something special you have to do to use > > tracef and tracepoints in same file? Doing so appears to have broken > > everything. > > It should just work. > > Can you provide a minimal example of the compile unit having this > issue ? > > Also you mention "static libs". Make sure you do *not* define > "LTTNG_UST_TRACEPOINT_PROBE_DYNAMIC_LINKAGE" in this case. See the > lttng-ust(3) man page for details (section "Statically linking the > tracepoint provider"). About that. It's a big project so all of our components are compiled as static libs and linked into one big executable at the end, I'm not using lttng-ust static libs. I'm linking final executable with -llttng-ust and -ldl I also have another question I just thought of regarding trace provider code. We've got a bunch of C code that is being compiled with g++. I initially had problems getting anything tracepoint related to work (with 2.11) and read somewhere that the trace provider code needed to be compiled with gcc so I built that by hand since our cmake build system is pretty complicated and just copied the resulting C .obj into our build directory structure for cmake to pick up and that worked. I basically copied the gcc .obj over the broken one that cmake/g++ generated. I'm not doing that now and am wrapping the -tp.h TRACEPOINT_EVENT definitions with #ifdef __cplusplus extern "C" {} #endif /* __cplusplus */ I'll have to go back and check my results as I've been on/off working on this for a while but if I stub out tracepoint calls and just do tracef it appears everything works. If I stub out tracef calls and just do tracepoint calls I think that works too. The problem I have is when I try to do both tracef and tracepoint in same file. That's when nothing works. Is there an order to the includes? In the C source file in question I do: #include <lttng/tracef.h> then later ... #define TRACEPOINT_DEFINE #include "my-tp.h" Hmm, now that I pay more attention, I don't see any mention of traceprovider code needing to be compiled with gcc vs g++ in the 2.13 documentation. ... and it looks like the traceprovider header changed to use LTTNG_UST_ in front of everything and I'm still using lttng 2.11 style! So I think my problem is probably associated with needing to re-do my traceprovider code to update to 2.13 style. Don't know if the C++ thing is still an issue. Regards, Brian _______________________________________________ lttng-dev mailing list lttng-dev@lists.lttng.org https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [lttng-dev] I'm still getting empty ust traces using tracef 2023-06-21 22:02 ` Brian Hutchinson via lttng-dev @ 2023-06-28 22:02 ` Brian Hutchinson via lttng-dev 0 siblings, 0 replies; 15+ messages in thread From: Brian Hutchinson via lttng-dev @ 2023-06-28 22:02 UTC (permalink / raw) To: Mathieu Desnoyers; +Cc: lttng-dev On Wed, Jun 21, 2023 at 6:02 PM Brian Hutchinson <b.hutchman@gmail.com> wrote: > > On Wed, Jun 21, 2023 at 4:21 PM Mathieu Desnoyers > <mathieu.desnoyers@efficios.com> wrote: > > > > On 6/20/23 18:02, Brian Hutchinson wrote: > > > On Thu, May 11, 2023 at 2:14 PM Mathieu Desnoyers > > > <mathieu.desnoyers@efficios.com> wrote: > > >> > > >> On 2023-05-11 14:13, Mathieu Desnoyers via lttng-dev wrote: > > >>> On 2023-05-11 12:36, Brian Hutchinson via lttng-dev wrote: > > >>>> ... more background. I've always used ltt in the kernel so I don't > > >>>> have much experience with the user side of it and especially > > >>>> multi-threaded, multi-core so I'm probably missing some fundamental > > >>>> concepts that I need to understand. > > >>> > > >>> Which are the exact versions of LTTng-UST and LTTng-Tools you are using > > >>> now ? (2.13.N or which git commit ?) > > >>> > > >> > > >> Also, can you try using lttng-ust stable-2.13 branch, which includes the following commit ? > > >> > > >> commit be2ca8b563bab81be15cbce7b9f52422369f79f7 > > >> Author: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> > > >> Date: Tue Feb 21 14:29:49 2023 -0500 > > >> > > >> Fix: Reevaluate LTTNG_UST_TRACEPOINT_DEFINE each time tracepoint.h is included > > >> > > >> Fix issues with missing symbols in use-cases where tracef.h is included > > >> before defining LTTNG_UST_TRACEPOINT_DEFINE, e.g.: > > >> > > >> #include <lttng/tracef.h> > > >> #define LTTNG_UST_TRACEPOINT_DEFINE > > >> #include <provider.h> > > >> > > >> It is caused by the fact that tracef.h includes tracepoint.h in a > > >> context which has LTTNG_UST_TRACEPOINT_DEFINE undefined, and this is not > > >> re-evaluated for the following includes. > > >> > > >> Fix this by lifting the definition code in tracepoint.h outside of the > > >> header include guards, and #undef the old LTTNG_UST__DEFINE_TRACEPOINT > > >> before re-defining it to its new semantic. Use a new > > >> _LTTNG_UST_TRACEPOINT_DEFINE_ONCE include guard within the > > >> LTTNG_UST_TRACEPOINT_DEFINE defined case to ensure symbols are not > > >> duplicated. > > >> > > >> Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> > > >> Change-Id: I0ef720435003a7ca0bfcf29d7bf27866c5ff8678 > > >> > > > > > > I applied this patch and if I use "tracef" type calls in our > > > application that is made up of a bunch of static libs ... the UST > > > trace calls work. I verified that traces that were called from > > > several different static libs all worked. > > > > > > But as soon as I include a "tracepoint" style tracepoint (that uses > > > trace provider include files etc.) then doing a "lttng list -u" > > > returns "None" for UST events. > > > > > > Is there some kind of rule that says a file can't use both tracef and > > > tracepoint calls? Is there something special you have to do to use > > > tracef and tracepoints in same file? Doing so appears to have broken > > > everything. > > > > It should just work. > > > > Can you provide a minimal example of the compile unit having this > > issue ? > > > > Also you mention "static libs". Make sure you do *not* define > > "LTTNG_UST_TRACEPOINT_PROBE_DYNAMIC_LINKAGE" in this case. See the > > lttng-ust(3) man page for details (section "Statically linking the > > tracepoint provider"). > > About that. It's a big project so all of our components are compiled > as static libs and linked into one big executable at the end, I'm not > using lttng-ust static libs. I'm linking final executable with > -llttng-ust and -ldl > > I also have another question I just thought of regarding trace provider code. > > We've got a bunch of C code that is being compiled with g++. I > initially had problems getting anything tracepoint related to work > (with 2.11) and read somewhere that the trace provider code needed to > be compiled with gcc so I built that by hand since our cmake build > system is pretty complicated and just copied the resulting C .obj into > our build directory structure for cmake to pick up and that worked. I > basically copied the gcc .obj over the broken one that cmake/g++ > generated. > > I'm not doing that now and am wrapping the -tp.h TRACEPOINT_EVENT > definitions with #ifdef __cplusplus extern "C" {} #endif /* > __cplusplus */ > > I'll have to go back and check my results as I've been on/off working > on this for a while but if I stub out tracepoint calls and just do > tracef it appears everything works. If I stub out tracef calls and > just do tracepoint calls I think that works too. The problem I have > is when I try to do both tracef and tracepoint in same file. That's > when nothing works. > > Is there an order to the includes? > > In the C source file in question I do: > > #include <lttng/tracef.h> > then later ... > #define TRACEPOINT_DEFINE > #include "my-tp.h" > > Hmm, now that I pay more attention, I don't see any mention of > traceprovider code needing to be compiled with gcc vs g++ in the 2.13 > documentation. > > ... and it looks like the traceprovider header changed to use > LTTNG_UST_ in front of everything and I'm still using lttng 2.11 > style! So I think my problem is probably associated with needing to > re-do my traceprovider code to update to 2.13 style. Don't know if > the C++ thing is still an issue./ > I updated my trace provider package I created for lttng 2.11 to 2.13 syntax. I discovered why lttng list -u wasn't returning anything. It's because the application had a seg fault. Here is the debug trace of that: root@localhost:# lttng create my_session --output=/tmp/my_trace --snapshot Spawning a session daemon libringbuffer-clients[41538/41538]: LTT : ltt ring buffer client "relay-metadata-mmap" init (in lttng_ring_buffer_metadata_client_init() at ../../../lttng-ust-2.13.5/src/common/ringbuffer-clients/metadata-template.h:364) libringbuffer-clients[41538/41538]: LTT : ltt ring buffer client "relay-overwrite-mmap" init (in lttng_ring_buffer_client_overwrite_init() at ../../../lttng-ust-2.13.5/src/common/ringbuffer-clients/template.h:826) libringbuffer-clients[41538/41538]: LTT : ltt ring buffer client "relay-overwrite-rt-mmap" init (in lttng_ring_buffer_client_overwrite_rt_init() at ../../../lttng-ust-2.13.5/src/common/ringbuffer-clients/template.h:826) libringbuffer-clients[41538/41538]: LTT : ltt ring buffer client "relay-discard-mmap" init (in lttng_ring_buffer_client_discard_init() at ../../../lttng-ust-2.13.5/src/common/ringbuffer-clients/template.h:826) libringbuffer-clients[41538/41538]: LTT : ltt ring buffer client "relay-discard-rt-mmap" init (in lttng_ring_buffer_client_discard_rt_init() at ../../../lttng-ust-2.13.5/src/common/ringbuffer-clients/template.h:826) [ 2867.702128] LTTng: Loaded modules v2.13.9 (Nordicit�é) [ 2867.711872] LTTng: Experimental bitwise enum enabled. libringbuffer-clients[41538/41538]: LTT : ltt ring buffer client "relay-discard-rt-mmap" exit (in lttng_ring_buffer_client_discard_rt_exit() at ../../../lttng-ust-2.13.5/src/common/ringbuffer-clients/template.h:833) libringbuffer-clients[41538/41538]: LTT : ltt ring buffer client "relay-discard-mmap" exit (in lttng_ring_buffer_client_discard_exit() at ../../../lttng-ust-2.13.5/src/common/ringbuffer-clients/template.h:833) libringbuffer-clients[41538/41538]: LTT : ltt ring buffer client "relay-overwrite-rt-mmap" exit (in lttng_ring_buffer_client_overwrite_rt_exit() at ../../../lttng-ust-2.13.5/src/common/ringbuffer-clients/template.h:833) libringbuffer-clients[41538/41538]: LTT : ltt ring buffer client "relay-overwrite-mmap" exit (in lttng_ring_buffer_client_overwrite_exit() at ../../../lttng-ust-2.13.5/src/common/ringbuffer-clients/template.h:833) libringbuffer-clients[41538/41538]: LTT : ltt ring buffer client "relay-metadata-mmap" exit (in lttng_ring_buffer_metadata_client_exit() at ../../../lttng-ust-2.13.5/src/common/ringbuffer-clients/metadata-template.h:371) Snapshot session my_session created. Default snapshot output set to /tmp/my_trace Every channel enabled for this session will be set to mmap output and default to overwrite mode. root@localhost:# ./my_app liblttng_ust[42358/42358]: dlopened liblttng-ust shared library (liblttng-ust.so.1). (in lttng_ust_ctor() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-ust-comm.c:2217) liblttng_ust_tracepoint[42358/42358]: Your compiler treats weak symbols with hidden visibility for integer objects as SAME address between compile units part of the same module. (in check_weak_hidden() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust-tracepoint/tracepoint.c:10) liblttng_ust_tracepoint[42358/42358]: Your compiler treats weak symbols with hidden visibility for pointer objects as SAME address between compile units part of the same module. (in check_weak_hidden() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust-tracepoint/tracepoint.c:10) liblttng_ust_tracepoint[42358/42358]: Your compiler treats weak symbols with hidden visibility for 24-byte structure objects as SAME address between compile units part of the same module. (in check_weak_hidden() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust-tracepoint/trace) liblttng_ust_tracepoint[42358/42358]: just registered a tracepoints section from 0xffff90b64630 and having 26 tracepoints (in lttng_ust_tracepoint_module_register() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust-tracepoint/tracepoint.c:960) liblttng_ust_tracepoint[42358/42358]: registered tracepoint: "lttng_ust_statedump:end" (in lttng_ust_tracepoint_module_register() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust-tracepoint/tracepoint.c:968) liblttng_ust_tracepoint[42358/42358]: registered tracepoint: "lttng_ust_statedump:procname" (in lttng_ust_tracepoint_module_register() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust-tracepoint/tracepoint.c:968) liblttng_ust_tracepoint[42358/42358]: registered tracepoint: "lttng_ust_statedump:debug_link" (in lttng_ust_tracepoint_module_register() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust-tracepoint/tracepoint.c:968) liblttng_ust_tracepoint[42358/42358]: registered tracepoint: "lttng_ust_statedump:build_id" (in lttng_ust_tracepoint_module_register() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust-tracepoint/tracepoint.c:968) liblttng_ust_tracepoint[42358/42358]: registered tracepoint: "lttng_ust_statedump:bin_info" (in lttng_ust_tracepoint_module_register() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust-tracepoint/tracepoint.c:968) liblttng_ust_tracepoint[42358/42358]: registered tracepoint: "lttng_ust_statedump:start" (in lttng_ust_tracepoint_module_register() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust-tracepoint/tracepoint.c:968) liblttng_ust_tracepoint[42358/42358]: registered tracepoint: "lttng_ust_lib:unload" (in lttng_ust_tracepoint_module_register() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust-tracepoint/tracepoint.c:968) liblttng_ust_tracepoint[42358/42358]: registered tracepoint: "lttng_ust_lib:debug_link" (in lttng_ust_tracepoint_module_register() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust-tracepoint/tracepoint.c:968) liblttng_ust_tracepoint[42358/42358]: registered tracepoint: "lttng_ust_lib:build_id" (in lttng_ust_tracepoint_module_register() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust-tracepoint/tracepoint.c:968) liblttng_ust_tracepoint[42358/42358]: registered tracepoint: "lttng_ust_lib:load" (in lttng_ust_tracepoint_module_register() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust-tracepoint/tracepoint.c:968) liblttng_ust_tracepoint[42358/42358]: registered tracepoint: "lttng_ust_tracef:event" (in lttng_ust_tracepoint_module_register() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust-tracepoint/tracepoint.c:968) liblttng_ust_tracepoint[42358/42358]: registered tracepoint: "lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_DEBUG" (in lttng_ust_tracepoint_module_register() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust-tracepoint/tracepoint.c:968) liblttng_ust_tracepoint[42358/42358]: registered tracepoint: "lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_DEBUG_LINE" (in lttng_ust_tracepoint_module_register() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust-tracepoint/tracepoint.c:968) liblttng_ust_tracepoint[42358/42358]: registered tracepoint: "lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_DEBUG_FUNCTION" (in lttng_ust_tracepoint_module_register() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust-tracepoint/tracepoint.c:968) liblttng_ust_tracepoint[42358/42358]: registered tracepoint: "lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_DEBUG_UNIT" (in lttng_ust_tracepoint_module_register() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust-tracepoint/tracepoint.c:968) liblttng_ust_tracepoint[42358/42358]: registered tracepoint: "lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_DEBUG_MODULE" (in lttng_ust_tracepoint_module_register() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust-tracepoint/tracepoint.c:968) liblttng_ust_tracepoint[42358/42358]: registered tracepoint: "lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_DEBUG_PROCESS" (in lttng_ust_tracepoint_module_register() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust-tracepoint/tracepoint.c:968) liblttng_ust_tracepoint[42358/42358]: registered tracepoint: "lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_DEBUG_PROGRAM" (in lttng_ust_tracepoint_module_register() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust-tracepoint/tracepoint.c:968) liblttng_ust_tracepoint[42358/42358]: registered tracepoint: "lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_DEBUG_SYSTEM" (in lttng_ust_tracepoint_module_register() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust-tracepoint/tracepoint.c:968) liblttng_ust_tracepoint[42358/42358]: registered tracepoint: "lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_INFO" (in lttng_ust_tracepoint_module_register() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust-tracepoint/tracepoint.c:968) liblttng_ust_tracepoint[42358/42358]: registered tracepoint: "lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_NOTICE" (in lttng_ust_tracepoint_module_register() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust-tracepoint/tracepoint.c:968) liblttng_ust_tracepoint[42358/42358]: registered tracepoint: "lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_WARNING" (in lttng_ust_tracepoint_module_register() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust-tracepoint/tracepoint.c:968) liblttng_ust_tracepoint[42358/42358]: registered tracepoint: "lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_ERR" (in lttng_ust_tracepoint_module_register() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust-tracepoint/tracepoint.c:968) liblttng_ust_tracepoint[42358/42358]: registered tracepoint: "lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_CRIT" (in lttng_ust_tracepoint_module_register() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust-tracepoint/tracepoint.c:968) liblttng_ust_tracepoint[42358/42358]: registered tracepoint: "lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_ALERT" (in lttng_ust_tracepoint_module_register() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust-tracepoint/tracepoint.c:968) liblttng_ust_tracepoint[42358/42358]: registered tracepoint: "lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_EMERG" (in lttng_ust_tracepoint_module_register() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust-tracepoint/tracepoint.c:968) liblttng_ust[42358/42358]: Provider "lttng_ust_statedump" accepted, version 3.0 is compatible with LTTng UST provider version 3.0. (in check_provider_version() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-probes.c:50) liblttng_ust[42358/42358]: Provider "lttng_ust_statedump" accepted, version 3.0 is compatible with LTTng UST provider version 3.0. (in check_provider_version() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-probes.c:50) liblttng_ust[42358/42358]: Provider "lttng_ust_statedump" accepted, version 3.0 is compatible with LTTng UST provider version 3.0. (in check_provider_version() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-probes.c:50) liblttng_ust[42358/42358]: Provider "lttng_ust_statedump" accepted, version 3.0 is compatible with LTTng UST provider version 3.0. (in check_provider_version() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-probes.c:50) liblttng_ust[42358/42358]: Provider "lttng_ust_statedump" accepted, version 3.0 is compatible with LTTng UST provider version 3.0. (in check_provider_version() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-probes.c:50) liblttng_ust[42358/42358]: Provider "lttng_ust_statedump" accepted, version 3.0 is compatible with LTTng UST provider version 3.0. (in check_provider_version() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-probes.c:50) liblttng_ust[42358/42358]: Provider "lttng_ust_statedump" accepted, version 3.0 is compatible with LTTng UST provider version 3.0. (in check_provider_version() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-probes.c:50) liblttng_ust[42358/42358]: adding probe lttng_ust_statedump containing 6 events to lazy registration list (in lttng_ust_probe_register() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-probes.c:254) libringbuffer-clients[42358/42358]: LTT : ltt ring buffer client "relay-metadata-mmap" init (in lttng_ring_buffer_metadata_client_init() at ../../../lttng-ust-2.13.5/src/common/ringbuffer-clients/metadata-template.h:364) libringbuffer-clients[42358/42358]: LTT : ltt ring buffer client "relay-overwrite-mmap" init (in lttng_ring_buffer_client_overwrite_init() at ../../../lttng-ust-2.13.5/src/common/ringbuffer-clients/template.h:826) libringbuffer-clients[42358/42358]: LTT : ltt ring buffer client "relay-overwrite-rt-mmap" init (in lttng_ring_buffer_client_overwrite_rt_init() at ../../../lttng-ust-2.13.5/src/common/ringbuffer-clients/template.h:826) libringbuffer-clients[42358/42358]: LTT : ltt ring buffer client "relay-discard-mmap" init (in lttng_ring_buffer_client_discard_init() at ../../../lttng-ust-2.13.5/src/common/ringbuffer-clients/template.h:826) libringbuffer-clients[42358/42358]: LTT : ltt ring buffer client "relay-discard-rt-mmap" init (in lttng_ring_buffer_client_discard_rt_init() at ../../../lttng-ust-2.13.5/src/common/ringbuffer-clients/template.h:826) liblttng_ust[42358/42360]: Info: sessiond not accepting connections to local apps socket (in ust_listener_thread() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-ust-comm.c:1884) liblttng_ust[42358/42360]: Waiting for local apps sessiond (in wait_for_sessiond() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-ust-comm.c:1758) liblttng_ust[42358/42359]: Message Received "Get Tracer Version" (65), Handle "root" (0) (in print_cmd() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-ust-comm.c:476) liblttng_ust[42358/42359]: Return value: 0 (in handle_message() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-ust-comm.c:1412) liblttng_ust[42358/42359]: message successfully sent (in send_reply() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-ust-comm.c:656) liblttng_ust[42358/42359]: Message Received "Create event notifier group" (70), Handle "root" (0) (in print_cmd() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-ust-comm.c:476) liblttng_ust[42358/42359]: message successfully sent (in send_reply() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-ust-comm.c:656) liblttng_ust[42358/42359]: Return value: 1 (in handle_message() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-ust-comm.c:1412) liblttng_ust[42358/42359]: message successfully sent (in send_reply() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-ust-comm.c:656) liblttng_ust[42358/42359]: Message Received "Create Counter" (192), Handle "event_notifier_" (1) (in print_cmd() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-ust-comm.c:476) liblttng_ust[42358/42359]: message successfully sent (in send_reply() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-ust-comm.c:656) liblttng_ust[42358/42359]: counter data received (in handle_message() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-ust-comm.c:1266) liblttng_ust[42358/42359]: Return value: 2 (in handle_message() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-ust-comm.c:1412) liblttng_ust[42358/42359]: message successfully sent (in send_reply() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-ust-comm.c:656) liblttng_ust[42358/42359]: Message Received "Create Counter CPU" (209), Handle "event_notifier " (2) (in print_cmd() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-ust-comm.c:476) liblttng_ust[42358/42359]: message successfully sent (in send_reply() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-ust-comm.c:656) liblttng_ust[42358/42359]: Return value: 0 (in handle_message() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-ust-comm.c:1412) liblttng_ust[42358/42359]: message successfully sent (in send_reply() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-ust-comm.c:656) liblttng_ust[42358/42359]: Message Received "Create Counter CPU" (209), Handle "event_notifier " (2) (in print_cmd() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-ust-comm.c:476) liblttng_ust[42358/42359]: message successfully sent (in send_reply() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-ust-comm.c:656) liblttng_ust[42358/42359]: Return value: 0 (in handle_message() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-ust-comm.c:1412) liblttng_ust[42358/42359]: message successfully sent (in send_reply() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-ust-comm.c:656) liblttng_ust[42358/42359]: Message Received "Create Counter CPU" (209), Handle "event_notifier " (2) (in print_cmd() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-ust-comm.c:476) liblttng_ust[42358/42359]: message successfully sent (in send_reply() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-ust-comm.c:656) liblttng_ust[42358/42359]: Return value: 0 (in handle_message() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-ust-comm.c:1412) liblttng_ust[42358/42359]: message successfully sent (in send_reply() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-ust-comm.c:656) liblttng_ust[42358/42359]: Message Received "Create Counter CPU" (209), Handle "event_notifier " (2) (in print_cmd() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-ust-comm.c:476) liblttng_ust[42358/42359]: message successfully sent (in send_reply() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-ust-comm.c:656) liblttng_ust[42358/42359]: Return value: 0 (in handle_message() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-ust-comm.c:1412) liblttng_ust[42358/42359]: message successfully sent (in send_reply() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-ust-comm.c:656) liblttng_ust[42358/42359]: Message Received "Registration Done" (68), Handle "root" (0) (in print_cmd() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-ust-comm.c:476) liblttng_ust[42358/42358]: Provider "lttng_ust_lib" accepted, version 3.0 is compatible with LTTng UST provider version 3.0. (in check_provider_version() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-probes.c:50) liblttng_ust[42358/42358]: Provider "lttng_ust_lib" accepted, version 3.0 is compatible with LTTng UST provider version 3.0. (in check_provider_version() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-probes.c:50) liblttng_ust[42358/42358]: Provider "lttng_ust_lib" accepted, version 3.0 is compatible with LTTng UST provider version 3.0. (in check_provider_version() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-probes.c:50) liblttng_ust[42358/42358]: Provider "lttng_ust_lib" accepted, version 3.0 is compatible with LTTng UST provider version 3.0. (in check_provider_version() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-probes.c:50) liblttng_ust[42358/42358]: Provider "lttng_ust_lib" accepted, version 3.0 is compatible with LTTng UST provider version 3.0. (in check_provider_version() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-probes.c:50) liblttng_ust[42358/42359]: Return value: 0 (in handle_message() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-ust-comm.c:1412) liblttng_ust[42358/42358]: adding probe lttng_ust_lib containing 4 events to lazy registration list (in lttng_ust_probe_register() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-probes.c:254) liblttng_ust_tracepoint[42358/42358]: Release queue of unregistered tracepoint probes. (in lttng_ust_tp_probe_prune_release_queue() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust-tracepoint/tracepoint.c:787) liblttng_ust[42358/42358]: Provider "lttng_ust_tracef" accepted, version 3.0 is compatible with LTTng UST provider version 3.0. (in check_provider_version() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-probes.c:50) liblttng_ust[42358/42358]: Provider "lttng_ust_tracef" accepted, version 3.0 is compatible with LTTng UST provider version 3.0. (in check_provider_version() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-probes.c:50) liblttng_ust[42358/42358]: adding probe lttng_ust_tracef containing 1 events to lazy registration list (in lttng_ust_probe_register() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-probes.c:254) liblttng_ust_tracepoint[42358/42358]: Release queue of unregistered tracepoint probes. (in lttng_ust_tp_probe_prune_release_queue() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust-tracepoint/tracepoint.c:787) liblttng_ust[42358/42358]: Provider "lttng_ust_tracelog" accepted, version 3.0 is compatible with LTTng UST provider version 3.0. (in check_provider_version() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-probes.c:50) liblttng_ust[42358/42358]: Provider "lttng_ust_tracelog" accepted, version 3.0 is compatible with LTTng UST provider version 3.0. (in check_provider_version() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-probes.c:50) liblttng_ust[42358/42358]: Provider "lttng_ust_tracelog" accepted, version 3.0 is compatible with LTTng UST provider version 3.0. (in check_provider_version() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-probes.c:50) liblttng_ust[42358/42358]: Provider "lttng_ust_tracelog" accepted, version 3.0 is compatible with LTTng UST provider version 3.0. (in check_provider_version() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-probes.c:50) liblttng_ust[42358/42358]: Provider "lttng_ust_tracelog" accepted, version 3.0 is compatible with LTTng UST provider version 3.0. (in check_provider_version() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-probes.c:50) liblttng_ust[42358/42358]: Provider "lttng_ust_tracelog" accepted, version 3.0 is compatible with LTTng UST provider version 3.0. (in check_provider_version() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-probes.c:50) liblttng_ust[42358/42358]: Provider "lttng_ust_tracelog" accepted, version 3.0 is compatible with LTTng UST provider version 3.0. (in check_provider_version() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-probes.c:50) liblttng_ust[42358/42358]: Provider "lttng_ust_tracelog" accepted, version 3.0 is compatible with LTTng UST provider version 3.0. (in check_provider_version() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-probes.c:50) liblttng_ust[42358/42358]: Provider "lttng_ust_tracelog" accepted, version 3.0 is compatible with LTTng UST provider version 3.0. (in check_provider_version() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-probes.c:50) liblttng_ust[42358/42358]: Provider "lttng_ust_tracelog" accepted, version 3.0 is compatible with LTTng UST provider version 3.0. (in check_provider_version() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-probes.c:50) liblttng_ust[42358/42358]: Provider "lttng_ust_tracelog" accepted, version 3.0 is compatible with LTTng UST provider version 3.0. (in check_provider_version() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-probes.c:50) liblttng_ust[42358/42358]: Provider "lttng_ust_tracelog" accepted, version 3.0 is compatible with LTTng UST provider version 3.0. (in check_provider_version() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-probes.c:50) liblttng_ust[42358/42358]: Provider "lttng_ust_tracelog" accepted, version 3.0 is compatible with LTTng UST provider version 3.0. (in check_provider_version() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-probes.c:50) liblttng_ust[42358/42358]: Provider "lttng_ust_tracelog" accepted, version 3.0 is compatible with LTTng UST provider version 3.0. (in check_provider_version() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-probes.c:50) liblttng_ust[42358/42358]: Provider "lttng_ust_tracelog" accepted, version 3.0 is compatible with LTTng UST provider version 3.0. (in check_provider_version() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-probes.c:50) liblttng_ust[42358/42358]: Provider "lttng_ust_tracelog" accepted, version 3.0 is compatible with LTTng UST provider version 3.0. (in check_provider_version() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-probes.c:50) liblttng_ust[42358/42359]: message successfully sent (in send_reply() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-ust-comm.c:656) liblttng_ust[42358/42358]: adding probe lttng_ust_tracelog containing 15 events to lazy registration list (in lttng_ust_probe_register() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-probes.c:254) liblttng_ust_tracepoint[42358/42358]: Release queue of unregistered tracepoint probes. (in lttng_ust_tp_probe_prune_release_queue() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust-tracepoint/tracepoint.c:787) Segmentation fault Not sure exactly what is going on. I did the hello world example stand alone and it worked. Thinking I messed up something in my trace provider package I simply used the hello world one that worked and put the tracepoint "lttng_ust_tracepoint(hello_world, my_first_tracepoint, 23,"hi there!");" in my app and it segfaults. This is using lttng-tools 2.13.9 with patches (from OE recipe): file://0001-tests-do-not-strip-a-helper-library.patch \ file://run-ptest \ file://lttng-sessiond.service \ file://disable-tests.patch \ file://0001-compat-Define-off64_t-as-off_t-on-linux.patch lttng-ust 2.13.5 with patches: file://0001-python-lttngust-Makefile.am-Add-install-lib-to-setup.patch \ file://0001-lttng-ust-common-link-with-liburcu-explicitly.patch \ file://0001-Makefile.am-update-rpath-link.patch And I manually added patch: Fix: Reevaluate LTTNG_UST_TRACEPOINT_DEFINE each time tracepoint.h is included lttng-modules 2.13.9 with patches: file://0009-Rename-genhd-wrapper-to-blkdev.patch \ file://0001-fix-mm-introduce-vma-vm_flags-wrapper-functions-v6.3.patch \ file://0002-fix-uuid-Decouple-guid_t-and-uuid_le-types-and-respe.patch \ file://0003-fix-btrfs-pass-find_free_extent_ctl-to-allocator-tra.patch \ file://0004-fix-net-add-location-to-trace_consume_skb-v6.3.patch Here is log of hello running right afterwards ... it does not seg fault: root@localhost:/home/root# ./hello liblttng_ust[61133/61133]: dlopened liblttng-ust shared library (liblttng-ust.so.1). (in lttng_ust_ctor() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-ust-comm.c:2217) liblttng_ust_tracepoint[61133/61133]: Your compiler treats weak symbols with hidden visibility for integer objects as SAME address between compile units part of the same module. (in check_weak_hidden() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust-tracepoint/tracepoint.c:10) liblttng_ust_tracepoint[61133/61133]: Your compiler treats weak symbols with hidden visibility for pointer objects as SAME address between compile units part of the same module. (in check_weak_hidden() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust-tracepoint/tracepoint.c:10) liblttng_ust_tracepoint[61133/61133]: Your compiler treats weak symbols with hidden visibility for 24-byte structure objects as SAME address between compile units part of the same module. (in check_weak_hidden() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust-tracepoint/trace) liblttng_ust_tracepoint[61133/61133]: just registered a tracepoints section from 0xffffa61d9630 and having 26 tracepoints (in lttng_ust_tracepoint_module_register() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust-tracepoint/tracepoint.c:960) liblttng_ust_tracepoint[61133/61133]: registered tracepoint: "lttng_ust_statedump:end" (in lttng_ust_tracepoint_module_register() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust-tracepoint/tracepoint.c:968) liblttng_ust_tracepoint[61133/61133]: registered tracepoint: "lttng_ust_statedump:procname" (in lttng_ust_tracepoint_module_register() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust-tracepoint/tracepoint.c:968) liblttng_ust_tracepoint[61133/61133]: registered tracepoint: "lttng_ust_statedump:debug_link" (in lttng_ust_tracepoint_module_register() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust-tracepoint/tracepoint.c:968) liblttng_ust_tracepoint[61133/61133]: registered tracepoint: "lttng_ust_statedump:build_id" (in lttng_ust_tracepoint_module_register() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust-tracepoint/tracepoint.c:968) liblttng_ust_tracepoint[61133/61133]: registered tracepoint: "lttng_ust_statedump:bin_info" (in lttng_ust_tracepoint_module_register() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust-tracepoint/tracepoint.c:968) liblttng_ust_tracepoint[61133/61133]: registered tracepoint: "lttng_ust_statedump:start" (in lttng_ust_tracepoint_module_register() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust-tracepoint/tracepoint.c:968) liblttng_ust_tracepoint[61133/61133]: registered tracepoint: "lttng_ust_lib:unload" (in lttng_ust_tracepoint_module_register() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust-tracepoint/tracepoint.c:968) liblttng_ust_tracepoint[61133/61133]: registered tracepoint: "lttng_ust_lib:debug_link" (in lttng_ust_tracepoint_module_register() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust-tracepoint/tracepoint.c:968) liblttng_ust_tracepoint[61133/61133]: registered tracepoint: "lttng_ust_lib:build_id" (in lttng_ust_tracepoint_module_register() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust-tracepoint/tracepoint.c:968) liblttng_ust_tracepoint[61133/61133]: registered tracepoint: "lttng_ust_lib:load" (in lttng_ust_tracepoint_module_register() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust-tracepoint/tracepoint.c:968) liblttng_ust_tracepoint[61133/61133]: registered tracepoint: "lttng_ust_tracef:event" (in lttng_ust_tracepoint_module_register() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust-tracepoint/tracepoint.c:968) liblttng_ust_tracepoint[61133/61133]: registered tracepoint: "lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_DEBUG" (in lttng_ust_tracepoint_module_register() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust-tracepoint/tracepoint.c:968) liblttng_ust_tracepoint[61133/61133]: registered tracepoint: "lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_DEBUG_LINE" (in lttng_ust_tracepoint_module_register() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust-tracepoint/tracepoint.c:968) liblttng_ust_tracepoint[61133/61133]: registered tracepoint: "lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_DEBUG_FUNCTION" (in lttng_ust_tracepoint_module_register() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust-tracepoint/tracepoint.c:968) liblttng_ust_tracepoint[61133/61133]: registered tracepoint: "lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_DEBUG_UNIT" (in lttng_ust_tracepoint_module_register() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust-tracepoint/tracepoint.c:968) liblttng_ust_tracepoint[61133/61133]: registered tracepoint: "lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_DEBUG_MODULE" (in lttng_ust_tracepoint_module_register() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust-tracepoint/tracepoint.c:968) liblttng_ust_tracepoint[61133/61133]: registered tracepoint: "lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_DEBUG_PROCESS" (in lttng_ust_tracepoint_module_register() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust-tracepoint/tracepoint.c:968) liblttng_ust_tracepoint[61133/61133]: registered tracepoint: "lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_DEBUG_PROGRAM" (in lttng_ust_tracepoint_module_register() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust-tracepoint/tracepoint.c:968) liblttng_ust_tracepoint[61133/61133]: registered tracepoint: "lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_DEBUG_SYSTEM" (in lttng_ust_tracepoint_module_register() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust-tracepoint/tracepoint.c:968) liblttng_ust_tracepoint[61133/61133]: registered tracepoint: "lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_INFO" (in lttng_ust_tracepoint_module_register() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust-tracepoint/tracepoint.c:968) liblttng_ust_tracepoint[61133/61133]: registered tracepoint: "lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_NOTICE" (in lttng_ust_tracepoint_module_register() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust-tracepoint/tracepoint.c:968) liblttng_ust_tracepoint[61133/61133]: registered tracepoint: "lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_WARNING" (in lttng_ust_tracepoint_module_register() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust-tracepoint/tracepoint.c:968) liblttng_ust_tracepoint[61133/61133]: registered tracepoint: "lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_ERR" (in lttng_ust_tracepoint_module_register() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust-tracepoint/tracepoint.c:968) liblttng_ust_tracepoint[61133/61133]: registered tracepoint: "lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_CRIT" (in lttng_ust_tracepoint_module_register() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust-tracepoint/tracepoint.c:968) liblttng_ust_tracepoint[61133/61133]: registered tracepoint: "lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_ALERT" (in lttng_ust_tracepoint_module_register() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust-tracepoint/tracepoint.c:968) liblttng_ust_tracepoint[61133/61133]: registered tracepoint: "lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_EMERG" (in lttng_ust_tracepoint_module_register() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust-tracepoint/tracepoint.c:968) liblttng_ust[61133/61133]: Provider "lttng_ust_statedump" accepted, version 3.0 is compatible with LTTng UST provider version 3.0. (in check_provider_version() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-probes.c:50) liblttng_ust[61133/61133]: Provider "lttng_ust_statedump" accepted, version 3.0 is compatible with LTTng UST provider version 3.0. (in check_provider_version() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-probes.c:50) liblttng_ust[61133/61133]: Provider "lttng_ust_statedump" accepted, version 3.0 is compatible with LTTng UST provider version 3.0. (in check_provider_version() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-probes.c:50) liblttng_ust[61133/61133]: Provider "lttng_ust_statedump" accepted, version 3.0 is compatible with LTTng UST provider version 3.0. (in check_provider_version() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-probes.c:50) liblttng_ust[61133/61133]: Provider "lttng_ust_statedump" accepted, version 3.0 is compatible with LTTng UST provider version 3.0. (in check_provider_version() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-probes.c:50) liblttng_ust[61133/61133]: Provider "lttng_ust_statedump" accepted, version 3.0 is compatible with LTTng UST provider version 3.0. (in check_provider_version() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-probes.c:50) liblttng_ust[61133/61133]: Provider "lttng_ust_statedump" accepted, version 3.0 is compatible with LTTng UST provider version 3.0. (in check_provider_version() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-probes.c:50) liblttng_ust[61133/61133]: adding probe lttng_ust_statedump containing 6 events to lazy registration list (in lttng_ust_probe_register() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-probes.c:254) libringbuffer-clients[61133/61133]: LTT : ltt ring buffer client "relay-metadata-mmap" init (in lttng_ring_buffer_metadata_client_init() at ../../../lttng-ust-2.13.5/src/common/ringbuffer-clients/metadata-template.h:364) libringbuffer-clients[61133/61133]: LTT : ltt ring buffer client "relay-overwrite-mmap" init (in lttng_ring_buffer_client_overwrite_init() at ../../../lttng-ust-2.13.5/src/common/ringbuffer-clients/template.h:826) libringbuffer-clients[61133/61133]: LTT : ltt ring buffer client "relay-overwrite-rt-mmap" init (in lttng_ring_buffer_client_overwrite_rt_init() at ../../../lttng-ust-2.13.5/src/common/ringbuffer-clients/template.h:826) libringbuffer-clients[61133/61133]: LTT : ltt ring buffer client "relay-discard-mmap" init (in lttng_ring_buffer_client_discard_init() at ../../../lttng-ust-2.13.5/src/common/ringbuffer-clients/template.h:826) libringbuffer-clients[61133/61133]: LTT : ltt ring buffer client "relay-discard-rt-mmap" init (in lttng_ring_buffer_client_discard_rt_init() at ../../../lttng-ust-2.13.5/src/common/ringbuffer-clients/template.h:826) liblttng_ust[61133/61140]: Info: sessiond not accepting connections to local apps socket (in ust_listener_thread() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-ust-comm.c:1884) liblttng_ust[61133/61140]: Waiting for local apps sessiond (in wait_for_sessiond() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-ust-comm.c:1758) liblttng_ust[61133/61139]: Message Received "Get Tracer Version" (65), Handle "root" (0) (in print_cmd() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-ust-comm.c:476) liblttng_ust[61133/61139]: Return value: 0 (in handle_message() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-ust-comm.c:1412) liblttng_ust[61133/61139]: message successfully sent (in send_reply() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-ust-comm.c:656) liblttng_ust[61133/61139]: Message Received "Create event notifier group" (70), Handle "root" (0) (in print_cmd() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-ust-comm.c:476) liblttng_ust[61133/61139]: message successfully sent (in send_reply() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-ust-comm.c:656) liblttng_ust[61133/61139]: Return value: 1 (in handle_message() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-ust-comm.c:1412) liblttng_ust[61133/61139]: message successfully sent (in send_reply() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-ust-comm.c:656) liblttng_ust[61133/61139]: Message Received "Create Counter" (192), Handle "event_notifier_" (1) (in print_cmd() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-ust-comm.c:476) liblttng_ust[61133/61139]: message successfully sent (in send_reply() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-ust-comm.c:656) liblttng_ust[61133/61139]: counter data received (in handle_message() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-ust-comm.c:1266) liblttng_ust[61133/61139]: Return value: 2 (in handle_message() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-ust-comm.c:1412) liblttng_ust[61133/61139]: message successfully sent (in send_reply() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-ust-comm.c:656) liblttng_ust[61133/61139]: Message Received "Create Counter CPU" (209), Handle "event_notifier " (2) (in print_cmd() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-ust-comm.c:476) liblttng_ust[61133/61139]: message successfully sent (in send_reply() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-ust-comm.c:656) liblttng_ust[61133/61139]: Return value: 0 (in handle_message() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-ust-comm.c:1412) liblttng_ust[61133/61139]: message successfully sent (in send_reply() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-ust-comm.c:656) liblttng_ust[61133/61139]: Message Received "Create Counter CPU" (209), Handle "event_notifier " (2) (in print_cmd() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-ust-comm.c:476) liblttng_ust[61133/61139]: message successfully sent (in send_reply() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-ust-comm.c:656) liblttng_ust[61133/61139]: Return value: 0 (in handle_message() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-ust-comm.c:1412) liblttng_ust[61133/61139]: message successfully sent (in send_reply() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-ust-comm.c:656) liblttng_ust[61133/61139]: Message Received "Create Counter CPU" (209), Handle "event_notifier " (2) (in print_cmd() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-ust-comm.c:476) liblttng_ust[61133/61139]: message successfully sent (in send_reply() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-ust-comm.c:656) liblttng_ust[61133/61139]: Return value: 0 (in handle_message() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-ust-comm.c:1412) liblttng_ust[61133/61139]: message successfully sent (in send_reply() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-ust-comm.c:656) liblttng_ust[61133/61139]: Message Received "Create Counter CPU" (209), Handle "event_notifier " (2) (in print_cmd() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-ust-comm.c:476) liblttng_ust[61133/61139]: message successfully sent (in send_reply() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-ust-comm.c:656) liblttng_ust[61133/61139]: Return value: 0 (in handle_message() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-ust-comm.c:1412) liblttng_ust[61133/61139]: message successfully sent (in send_reply() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-ust-comm.c:656) liblttng_ust[61133/61139]: Message Received "Registration Done" (68), Handle "root" (0) (in print_cmd() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-ust-comm.c:476) liblttng_ust[61133/61133]: Provider "lttng_ust_lib" accepted, version 3.0 is compatible with LTTng UST provider version 3.0. (in check_provider_version() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-probes.c:50) liblttng_ust[61133/61133]: Provider "lttng_ust_lib" accepted, version 3.0 is compatible with LTTng UST provider version 3.0. (in check_provider_version() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-probes.c:50) liblttng_ust[61133/61133]: Provider "lttng_ust_lib" accepted, version 3.0 is compatible with LTTng UST provider version 3.0. (in check_provider_version() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-probes.c:50) liblttng_ust[61133/61133]: Provider "lttng_ust_lib" accepted, version 3.0 is compatible with LTTng UST provider version 3.0. (in check_provider_version() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-probes.c:50) liblttng_ust[61133/61133]: Provider "lttng_ust_lib" accepted, version 3.0 is compatible with LTTng UST provider version 3.0. (in check_provider_version() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-probes.c:50) liblttng_ust[61133/61139]: Return value: 0 (in handle_message() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-ust-comm.c:1412) liblttng_ust[61133/61133]: adding probe lttng_ust_lib containing 4 events to lazy registration list (in lttng_ust_probe_register() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-probes.c:254) liblttng_ust_tracepoint[61133/61133]: Release queue of unregistered tracepoint probes. (in lttng_ust_tp_probe_prune_release_queue() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust-tracepoint/tracepoint.c:787) liblttng_ust[61133/61133]: Provider "lttng_ust_tracef" accepted, version 3.0 is compatible with LTTng UST provider version 3.0. (in check_provider_version() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-probes.c:50) liblttng_ust[61133/61133]: Provider "lttng_ust_tracef" accepted, version 3.0 is compatible with LTTng UST provider version 3.0. (in check_provider_version() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-probes.c:50) liblttng_ust[61133/61133]: adding probe lttng_ust_tracef containing 1 events to lazy registration list (in lttng_ust_probe_register() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-probes.c:254) liblttng_ust_tracepoint[61133/61133]: Release queue of unregistered tracepoint probes. (in lttng_ust_tp_probe_prune_release_queue() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust-tracepoint/tracepoint.c:787) liblttng_ust[61133/61133]: Provider "lttng_ust_tracelog" accepted, version 3.0 is compatible with LTTng UST provider version 3.0. (in check_provider_version() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-probes.c:50) liblttng_ust[61133/61133]: Provider "lttng_ust_tracelog" accepted, version 3.0 is compatible with LTTng UST provider version 3.0. (in check_provider_version() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-probes.c:50) liblttng_ust[61133/61133]: Provider "lttng_ust_tracelog" accepted, version 3.0 is compatible with LTTng UST provider version 3.0. (in check_provider_version() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-probes.c:50) liblttng_ust[61133/61133]: Provider "lttng_ust_tracelog" accepted, version 3.0 is compatible with LTTng UST provider version 3.0. (in check_provider_version() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-probes.c:50) liblttng_ust[61133/61133]: Provider "lttng_ust_tracelog" accepted, version 3.0 is compatible with LTTng UST provider version 3.0. (in check_provider_version() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-probes.c:50) liblttng_ust[61133/61133]: Provider "lttng_ust_tracelog" accepted, version 3.0 is compatible with LTTng UST provider version 3.0. (in check_provider_version() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-probes.c:50) liblttng_ust[61133/61133]: Provider "lttng_ust_tracelog" accepted, version 3.0 is compatible with LTTng UST provider version 3.0. (in check_provider_version() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-probes.c:50) liblttng_ust[61133/61133]: Provider "lttng_ust_tracelog" accepted, version 3.0 is compatible with LTTng UST provider version 3.0. (in check_provider_version() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-probes.c:50) liblttng_ust[61133/61133]: Provider "lttng_ust_tracelog" accepted, version 3.0 is compatible with LTTng UST provider version 3.0. (in check_provider_version() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-probes.c:50) liblttng_ust[61133/61133]: Provider "lttng_ust_tracelog" accepted, version 3.0 is compatible with LTTng UST provider version 3.0. (in check_provider_version() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-probes.c:50) liblttng_ust[61133/61133]: Provider "lttng_ust_tracelog" accepted, version 3.0 is compatible with LTTng UST provider version 3.0. (in check_provider_version() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-probes.c:50) liblttng_ust[61133/61133]: Provider "lttng_ust_tracelog" accepted, version 3.0 is compatible with LTTng UST provider version 3.0. (in check_provider_version() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-probes.c:50) liblttng_ust[61133/61133]: Provider "lttng_ust_tracelog" accepted, version 3.0 is compatible with LTTng UST provider version 3.0. (in check_provider_version() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-probes.c:50) liblttng_ust[61133/61133]: Provider "lttng_ust_tracelog" accepted, version 3.0 is compatible with LTTng UST provider version 3.0. (in check_provider_version() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-probes.c:50) liblttng_ust[61133/61133]: Provider "lttng_ust_tracelog" accepted, version 3.0 is compatible with LTTng UST provider version 3.0. (in check_provider_version() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-probes.c:50) liblttng_ust[61133/61133]: Provider "lttng_ust_tracelog" accepted, version 3.0 is compatible with LTTng UST provider version 3.0. (in check_provider_version() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-probes.c:50) liblttng_ust[61133/61139]: message successfully sent (in send_reply() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-ust-comm.c:656) liblttng_ust[61133/61133]: adding probe lttng_ust_tracelog containing 15 events to lazy registration list (in lttng_ust_probe_register() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-probes.c:254) liblttng_ust_tracepoint[61133/61133]: Release queue of unregistered tracepoint probes. (in lttng_ust_tp_probe_prune_release_queue() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust-tracepoint/tracepoint.c:787) liblttng_ust_tracepoint[61133/61133]: just registered a tracepoints section from 0xaaaaae9c41b8 and having 1 tracepoints (in lttng_ust_tracepoint_module_register() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust-tracepoint/tracepoint.c:960) liblttng_ust_tracepoint[61133/61133]: registered tracepoint: "hello_world:my_first_tracepoint" (in lttng_ust_tracepoint_module_register() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust-tracepoint/tracepoint.c:968) liblttng_ust[61133/61133]: Provider "hello_world" accepted, version 3.0 is compatible with LTTng UST provider version 3.0. (in check_provider_version() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-probes.c:50) liblttng_ust[61133/61133]: Provider "hello_world" accepted, version 3.0 is compatible with LTTng UST provider version 3.0. (in check_provider_version() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-probes.c:50) liblttng_ust[61133/61133]: adding probe hello_world containing 1 events to lazy registration list (in lttng_ust_probe_register() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-probes.c:254) liblttng_ust_tracepoint[61133/61133]: Release queue of unregistered tracepoint probes. (in lttng_ust_tp_probe_prune_release_queue() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust-tracepoint/tracepoint.c:787) Hello, World! Press Enter to continue... Quitting now! liblttng_ust[61133/61133]: Provider "hello_world" accepted, version 3.0 is compatible with LTTng UST provider version 3.0. (in check_provider_version() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-probes.c:50) liblttng_ust_tracepoint[61133/61133]: Release queue of unregistered tracepoint probes. (in lttng_ust_tp_probe_prune_release_queue() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust-tracepoint/tracepoint.c:787) liblttng_ust[61133/61133]: just unregistered probes of provider hello_world (in lttng_ust_probe_unregister() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-probes.c:286) liblttng_ust_tracepoint[61133/61133]: just unregistered a tracepoints section from 0xaaaaae9c41b8 (in lttng_ust_tracepoint_module_unregister() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust-tracepoint/tracepoint.c:995) liblttng_ust[61133/61133]: Provider "lttng_ust_tracelog" accepted, version 3.0 is compatible with LTTng UST provider version 3.0. (in check_provider_version() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-probes.c:50) liblttng_ust_tracepoint[61133/61133]: Release queue of unregistered tracepoint probes. (in lttng_ust_tp_probe_prune_release_queue() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust-tracepoint/tracepoint.c:787) liblttng_ust[61133/61133]: just unregistered probes of provider lttng_ust_tracelog (in lttng_ust_probe_unregister() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-probes.c:286) liblttng_ust[61133/61133]: Provider "lttng_ust_tracef" accepted, version 3.0 is compatible with LTTng UST provider version 3.0. (in check_provider_version() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-probes.c:50) liblttng_ust_tracepoint[61133/61133]: Release queue of unregistered tracepoint probes. (in lttng_ust_tp_probe_prune_release_queue() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust-tracepoint/tracepoint.c:787) liblttng_ust[61133/61133]: just unregistered probes of provider lttng_ust_tracef (in lttng_ust_probe_unregister() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-probes.c:286) liblttng_ust[61133/61133]: Provider "lttng_ust_lib" accepted, version 3.0 is compatible with LTTng UST provider version 3.0. (in check_provider_version() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-probes.c:50) liblttng_ust_tracepoint[61133/61133]: Release queue of unregistered tracepoint probes. (in lttng_ust_tp_probe_prune_release_queue() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust-tracepoint/tracepoint.c:787) liblttng_ust[61133/61133]: just unregistered probes of provider lttng_ust_lib (in lttng_ust_probe_unregister() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-probes.c:286) libringbuffer-clients[61133/61133]: LTT : ltt ring buffer client "relay-discard-rt-mmap" exit (in lttng_ring_buffer_client_discard_rt_exit() at ../../../lttng-ust-2.13.5/src/common/ringbuffer-clients/template.h:833) libringbuffer-clients[61133/61133]: LTT : ltt ring buffer client "relay-discard-mmap" exit (in lttng_ring_buffer_client_discard_exit() at ../../../lttng-ust-2.13.5/src/common/ringbuffer-clients/template.h:833) libringbuffer-clients[61133/61133]: LTT : ltt ring buffer client "relay-overwrite-rt-mmap" exit (in lttng_ring_buffer_client_overwrite_rt_exit() at ../../../lttng-ust-2.13.5/src/common/ringbuffer-clients/template.h:833) libringbuffer-clients[61133/61133]: LTT : ltt ring buffer client "relay-overwrite-mmap" exit (in lttng_ring_buffer_client_overwrite_exit() at ../../../lttng-ust-2.13.5/src/common/ringbuffer-clients/template.h:833) libringbuffer-clients[61133/61133]: LTT : ltt ring buffer client "relay-metadata-mmap" exit (in lttng_ring_buffer_metadata_client_exit() at ../../../lttng-ust-2.13.5/src/common/ringbuffer-clients/metadata-template.h:371) liblttng_ust[61133/61133]: Provider "lttng_ust_statedump" accepted, version 3.0 is compatible with LTTng UST provider version 3.0. (in check_provider_version() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-probes.c:50) liblttng_ust_tracepoint[61133/61133]: Release queue of unregistered tracepoint probes. (in lttng_ust_tp_probe_prune_release_queue() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust-tracepoint/tracepoint.c:787) liblttng_ust[61133/61133]: just unregistered probes of provider lttng_ust_statedump (in lttng_ust_probe_unregister() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-probes.c:286) liblttng_ust_tracepoint[61133/61133]: just unregistered a tracepoints section from 0xffffa61d9630 (in lttng_ust_tracepoint_module_unregister() at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust-tracepoint/tracepoint.c:995) Regards, Brian _______________________________________________ lttng-dev mailing list lttng-dev@lists.lttng.org https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [lttng-dev] I'm still getting empty ust traces using tracef 2023-05-11 18:13 ` Mathieu Desnoyers via lttng-dev 2023-05-11 18:15 ` Mathieu Desnoyers via lttng-dev @ 2023-05-11 18:38 ` Brian Hutchinson via lttng-dev 1 sibling, 0 replies; 15+ messages in thread From: Brian Hutchinson via lttng-dev @ 2023-05-11 18:38 UTC (permalink / raw) To: Mathieu Desnoyers; +Cc: lttng-dev Hi Mathieu, On Thu, May 11, 2023 at 2:12 PM Mathieu Desnoyers <mathieu.desnoyers@efficios.com> wrote: > > On 2023-05-11 12:36, Brian Hutchinson via lttng-dev wrote: > > ... more background. I've always used ltt in the kernel so I don't > > have much experience with the user side of it and especially > > multi-threaded, multi-core so I'm probably missing some fundamental > > concepts that I need to understand. > > Which are the exact versions of LTTng-UST and LTTng-Tools you are using > now ? (2.13.N or which git commit ?) > > Thanks, > > Mathieu Built into my rootfs with Yocto: -rw-r--r-- 4 hutch hutch 977252 May 2 15:32 lttng-tools-2.13.9-r0.aarch64.rpm -rw-r--r-- 4 hutch hutch 11466008 May 2 15:33 lttng-tools-dbg-2.13.9-r0.aarch64.rpm -rw-r--r-- 4 hutch hutch 49104 May 2 15:32 lttng-tools-dev-2.13.9-r0.aarch64.rpm -rw-r--r-- 4 hutch hutch 87452 May 2 15:32 lttng-tools-doc-2.13.9-r0.aarch64.rpm -rw-r--r-- 4 hutch hutch 1602356 May 2 15:32 lttng-tools-ptest-2.13.9-r0.aarch64.rpm -rw-r--r-- 4 hutch hutch 867164 May 2 15:32 lttng-tools-src-2.13.9-r0.aarch64.rpm -rw-r--r-- 4 hutch hutch 219952 May 2 15:31 lttng-ust-2.13.5-r0.aarch64.rpm -rw-r--r-- 4 hutch hutch 9248 May 2 15:31 lttng-ust-bin-2.13.5-r0.aarch64.rpm -rw-r--r-- 4 hutch hutch 970248 May 2 15:31 lttng-ust-dbg-2.13.5-r0.aarch64.rpm -rw-r--r-- 4 hutch hutch 49132 May 2 15:31 lttng-ust-dev-2.13.5-r0.aarch64.rpm -rw-r--r-- 4 hutch hutch 39864 May 2 15:31 lttng-ust-doc-2.13.5-r0.aarch64.rpm -rw-r--r-- 4 hutch hutch 258392 May 2 15:31 lttng-ust-src-2.13.5-r0.aarch64.rpm Regards, Brian > > > > > Regards, > > > > Brian > > > > On Thu, May 11, 2023 at 11:53 AM Brian Hutchinson <b.hutchman@gmail.com> wrote: > >> > >> Hi, > >> > >> I posted a while ago (thread - Using lttng 2.11 and UST doesn't appear > >> to work - getting empty trace files) about this problem I'm having > >> with getting empty trace logs. > >> > >> I've since upgraded to lttng v2.13 and while I can do a simple hello > >> world program with tracef and get events in the log files, my more > >> complicated large multi-threaded app I'm trying to debug is still > >> getting empty log file traces. > >> > >> I can list the user space events in my app. > >> > >> Next I do: > >> > >> lttng enable-event --userspace 'lttng_ust_tracef:*' > >> > >> ... to enable the events, start lttng, start my app, .... and I get a > >> trace directory structure that's empty. > >> > >> I feel like I've read every thread in the archives about people having > >> the same problem. > >> > >> I did try using LD_PRELOAD with various libs thinking that was the > >> problem but so far I'm still getting empty traces. > >> > >> So far I've tried: > >> > >> LD_PRELOAD=liblttng-ust-libc-wrapper.so.1:liblttng-ust-pthread-wrapper.so.1:liblttng-ust-dl.so.1:liblttng-ust-fork.so.1:liblttng-ust-fd.so.1 > >> /usr/local/bin/my_app > >> > >> I guess one question I have is how do I determine which "helper libs" > >> I need to preload? > >> > >> The application I'm working on is made up of a bunch of smaller static > >> libs linked together into one big executable and that is linked with > >> -llttng-ust and -ldl. > >> > >> I'm pretty stuck at the moment. Anyone have any wisdom on what I > >> might be doing wrong or how I can tell why I'm not getting events in > >> the logs? > >> > >> Thanks, > >> > >> Brian > > _______________________________________________ > > lttng-dev mailing list > > lttng-dev@lists.lttng.org > > https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev > > -- > Mathieu Desnoyers > EfficiOS Inc. > https://www.efficios.com > _______________________________________________ lttng-dev mailing list lttng-dev@lists.lttng.org https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [lttng-dev] I'm still getting empty ust traces using tracef 2023-05-11 16:36 ` Brian Hutchinson via lttng-dev 2023-05-11 18:13 ` Mathieu Desnoyers via lttng-dev @ 2023-05-11 18:14 ` Michael Jeanson via lttng-dev 2023-05-11 18:42 ` Brian Hutchinson via lttng-dev 2023-05-11 18:57 ` Brian Hutchinson via lttng-dev 2 siblings, 1 reply; 15+ messages in thread From: Michael Jeanson via lttng-dev @ 2023-05-11 18:14 UTC (permalink / raw) To: lttng-dev On 2023-05-11 12:36, Brian Hutchinson via lttng-dev wrote: > ... more background. I've always used ltt in the kernel so I don't > have much experience with the user side of it and especially > multi-threaded, multi-core so I'm probably missing some fundamental > concepts that I need to understand. > > Regards, > > Brian Hi, If you are static linking 'liblttng-ust' then the library constructor won't be executed and you'll get no traces. While this setup is untested and unsupported, you could call the UST constructor from your application early initialisation but that would require patching lttng-ust to export the 'lttng_ust_ctor' symbol. And the first thing to do to start diagnosing UST tracing issues is to run your application with hte LTTNG_UST_DEBUG environment variable set, see [1] for details. Cheers, Michael [1] https://lttng.org/man/3/lttng-ust/v2.13/#doc-_environment_variables _______________________________________________ lttng-dev mailing list lttng-dev@lists.lttng.org https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [lttng-dev] I'm still getting empty ust traces using tracef 2023-05-11 18:14 ` Michael Jeanson via lttng-dev @ 2023-05-11 18:42 ` Brian Hutchinson via lttng-dev 0 siblings, 0 replies; 15+ messages in thread From: Brian Hutchinson via lttng-dev @ 2023-05-11 18:42 UTC (permalink / raw) To: Michael Jeanson; +Cc: lttng-dev On Thu, May 11, 2023 at 2:14 PM Michael Jeanson <mjeanson@efficios.com> wrote: > > On 2023-05-11 12:36, Brian Hutchinson via lttng-dev wrote: > > ... more background. I've always used ltt in the kernel so I don't > > have much experience with the user side of it and especially > > multi-threaded, multi-core so I'm probably missing some fundamental > > concepts that I need to understand. > > > > Regards, > > > > Brian > > Hi, > > If you are static linking 'liblttng-ust' then the library constructor won't be All of the final executable is a bunch of statically linked libs (all .a's) but lttng-ust isn't statically linked (using .so.1, not .a). I couldn't build/link without linking -llttng-ust and -ldl. > executed and you'll get no traces. While this setup is untested and > unsupported, you could call the UST constructor from your application early > initialisation but that would require patching lttng-ust to export the > 'lttng_ust_ctor' symbol. > > And the first thing to do to start diagnosing UST tracing issues is to run > your application with hte LTTNG_UST_DEBUG environment variable set, see [1] Thanks! I'll check that reference out. Regards, Brian > for details. > > Cheers, > > Michael > > [1] https://lttng.org/man/3/lttng-ust/v2.13/#doc-_environment_variables > _______________________________________________ lttng-dev mailing list lttng-dev@lists.lttng.org https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [lttng-dev] I'm still getting empty ust traces using tracef 2023-05-11 16:36 ` Brian Hutchinson via lttng-dev 2023-05-11 18:13 ` Mathieu Desnoyers via lttng-dev 2023-05-11 18:14 ` Michael Jeanson via lttng-dev @ 2023-05-11 18:57 ` Brian Hutchinson via lttng-dev 2 siblings, 0 replies; 15+ messages in thread From: Brian Hutchinson via lttng-dev @ 2023-05-11 18:57 UTC (permalink / raw) To: lttng-dev On Thu, May 11, 2023 at 12:36 PM Brian Hutchinson <b.hutchman@gmail.com> wrote: > > ... more background. I've always used ltt in the kernel so I don't > have much experience with the user side of it and especially > multi-threaded, multi-core so I'm probably missing some fundamental > concepts that I need to understand. > > Regards, > > Brian I'm currently reading up on the new v2.13 "Recording event rule and event record" features. I came to 2.13 because I want to be able to trigger snapshots on an event so I don't fill up storage on an embedded device. After I start lttng-systemd --daemonize and then start my app, I see this when I do lttng list --userspace: # lttng-sessiond --daemonize [ 85.180717] LTTng: Loaded modules v2.13.9 (Nordicit�é) [ 85.186103] LTTng: Experimental bitwise enum enabled. # lttng list --userspace UST events: ------------- PID: 617 - Name: sh lttng_ust_libc:posix_memalign (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint) lttng_ust_libc:memalign (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint) lttng_ust_libc:realloc (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint) lttng_ust_libc:calloc (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint) lttng_ust_libc:free (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint) lttng_ust_libc:malloc (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint) lttng_ust_pthread:pthread_mutex_unlock (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint) lttng_ust_pthread:pthread_mutex_trylock (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint) lttng_ust_pthread:pthread_mutex_lock_acq (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint) lttng_ust_pthread:pthread_mutex_lock_req (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint) lttng_ust_dl:dlclose (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint) lttng_ust_dl:debug_link (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint) lttng_ust_dl:build_id (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint) lttng_ust_dl:dlmopen (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint) lttng_ust_dl:dlopen (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint) lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_DEBUG (loglevel: TRACE_DEBUG (14)) (type: tracepoint) lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_DEBUG_LINE (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint) lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_DEBUG_FUNCTION (loglevel: TRACE_DEBUG_FUNCTION (12)) (type: tracepoint) lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_DEBUG_UNIT (loglevel: TRACE_DEBUG_UNIT (11)) (type: tracepoint) lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_DEBUG_MODULE (loglevel: TRACE_DEBUG_MODULE (10)) (type: tracepoint) lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_DEBUG_PROCESS (loglevel: TRACE_DEBUG_PROCESS (9)) (type: tracepoint) lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_DEBUG_PROGRAM (loglevel: TRACE_DEBUG_PROGRAM (8)) (type: tracepoint) lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_DEBUG_SYSTEM (loglevel: TRACE_DEBUG_SYSTEM (7)) (type: tracepoint) lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_INFO (loglevel: TRACE_INFO (6)) (type: tracepoint) lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_NOTICE (loglevel: TRACE_NOTICE (5)) (type: tracepoint) lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_WARNING (loglevel: TRACE_WARNING (4)) (type: tracepoint) lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_ERR (loglevel: TRACE_ERR (3)) (type: tracepoint) lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_CRIT (loglevel: TRACE_CRIT (2)) (type: tracepoint) lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_ALERT (loglevel: TRACE_ALERT (1)) (type: tracepoint) lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_EMERG (loglevel: TRACE_EMERG (0)) (type: tracepoint) lttng_ust_tracef:event (loglevel: TRACE_DEBUG (14)) (type: tracepoint) lttng_ust_lib:unload (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint) lttng_ust_lib:debug_link (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint) lttng_ust_lib:build_id (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint) lttng_ust_lib:load (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint) lttng_ust_statedump:end (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint) lttng_ust_statedump:procname (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint) lttng_ust_statedump:debug_link (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint) lttng_ust_statedump:build_id (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint) lttng_ust_statedump:bin_info (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint) lttng_ust_statedump:start (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint) PID: 601 - Name: /usr/loal/bin/my_app lttng_ust_libc:posix_memalign (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint) lttng_ust_libc:memalign (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint) lttng_ust_libc:realloc (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint) lttng_ust_libc:calloc (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint) lttng_ust_libc:free (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint) lttng_ust_libc:malloc (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint) lttng_ust_pthread:pthread_mutex_unlock (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint) lttng_ust_pthread:pthread_mutex_trylock (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint) lttng_ust_pthread:pthread_mutex_lock_acq (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint) lttng_ust_pthread:pthread_mutex_lock_req (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint) lttng_ust_dl:dlclose (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint) lttng_ust_dl:debug_link (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint) lttng_ust_dl:build_id (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint) lttng_ust_dl:dlmopen (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint) lttng_ust_dl:dlopen (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint) lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_DEBUG (loglevel: TRACE_DEBUG (14)) (type: tracepoint) lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_DEBUG_LINE (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint) lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_DEBUG_FUNCTION (loglevel: TRACE_DEBUG_FUNCTION (12)) (type: tracepoint) lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_DEBUG_UNIT (loglevel: TRACE_DEBUG_UNIT (11)) (type: tracepoint) lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_DEBUG_MODULE (loglevel: TRACE_DEBUG_MODULE (10)) (type: tracepoint) lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_DEBUG_PROCESS (loglevel: TRACE_DEBUG_PROCESS (9)) (type: tracepoint) lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_DEBUG_PROGRAM (loglevel: TRACE_DEBUG_PROGRAM (8)) (type: tracepoint) lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_DEBUG_SYSTEM (loglevel: TRACE_DEBUG_SYSTEM (7)) (type: tracepoint) lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_INFO (loglevel: TRACE_INFO (6)) (type: tracepoint) lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_NOTICE (loglevel: TRACE_NOTICE (5)) (type: tracepoint) lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_WARNING (loglevel: TRACE_WARNING (4)) (type: tracepoint) lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_ERR (loglevel: TRACE_ERR (3)) (type: tracepoint) lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_CRIT (loglevel: TRACE_CRIT (2)) (type: tracepoint) lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_ALERT (loglevel: TRACE_ALERT (1)) (type: tracepoint) lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_EMERG (loglevel: TRACE_EMERG (0)) (type: tracepoint) lttng_ust_tracef:event (loglevel: TRACE_DEBUG (14)) (type: tracepoint) lttng_ust_lib:unload (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint) lttng_ust_lib:debug_link (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint) lttng_ust_lib:build_id (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint) lttng_ust_lib:load (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint) lttng_ust_statedump:end (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint) lttng_ust_statedump:procname (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint) lttng_ust_statedump:debug_link (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint) lttng_ust_statedump:build_id (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint) lttng_ust_statedump:bin_info (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint) lttng_ust_statedump:start (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint) PID: 626 - Name: xz lttng_ust_libc:posix_memalign (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint) lttng_ust_libc:memalign (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint) lttng_ust_libc:realloc (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint) lttng_ust_libc:calloc (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint) lttng_ust_libc:free (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint) lttng_ust_libc:malloc (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint) lttng_ust_pthread:pthread_mutex_unlock (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint) lttng_ust_pthread:pthread_mutex_trylock (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint) lttng_ust_pthread:pthread_mutex_lock_acq (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint) lttng_ust_pthread:pthread_mutex_lock_req (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint) lttng_ust_dl:dlclose (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint) lttng_ust_dl:debug_link (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint) lttng_ust_dl:build_id (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint) lttng_ust_dl:dlmopen (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint) lttng_ust_dl:dlopen (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint) lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_DEBUG (loglevel: TRACE_DEBUG (14)) (type: tracepoint) lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_DEBUG_LINE (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint) lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_DEBUG_FUNCTION (loglevel: TRACE_DEBUG_FUNCTION (12)) (type: tracepoint) lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_DEBUG_UNIT (loglevel: TRACE_DEBUG_UNIT (11)) (type: tracepoint) lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_DEBUG_MODULE (loglevel: TRACE_DEBUG_MODULE (10)) (type: tracepoint) lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_DEBUG_PROCESS (loglevel: TRACE_DEBUG_PROCESS (9)) (type: tracepoint) lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_DEBUG_PROGRAM (loglevel: TRACE_DEBUG_PROGRAM (8)) (type: tracepoint) lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_DEBUG_SYSTEM (loglevel: TRACE_DEBUG_SYSTEM (7)) (type: tracepoint) lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_INFO (loglevel: TRACE_INFO (6)) (type: tracepoint) lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_NOTICE (loglevel: TRACE_NOTICE (5)) (type: tracepoint) lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_WARNING (loglevel: TRACE_WARNING (4)) (type: tracepoint) lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_ERR (loglevel: TRACE_ERR (3)) (type: tracepoint) lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_CRIT (loglevel: TRACE_CRIT (2)) (type: tracepoint) lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_ALERT (loglevel: TRACE_ALERT (1)) (type: tracepoint) lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_EMERG (loglevel: TRACE_EMERG (0)) (type: tracepoint) lttng_ust_tracef:event (loglevel: TRACE_DEBUG (14)) (type: tracepoint) lttng_ust_lib:unload (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint) lttng_ust_lib:debug_link (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint) lttng_ust_lib:build_id (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint) lttng_ust_lib:load (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint) lttng_ust_statedump:end (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint) lttng_ust_statedump:procname (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint) lttng_ust_statedump:debug_link (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint) lttng_ust_statedump:build_id (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint) lttng_ust_statedump:bin_info (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint) lttng_ust_statedump:start (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint) I'm not sure why all this showed up 3 times. Could be because I have systemd unit file ExecStart calling a script so that my LD_PRELOAD= line can work. ExecStart= didn't like LD_PRELOAD. So now I'm guessing ... I haven't read the fine documentation far enough yet, that I have to somehow connect my record event rules to PID 601 somehow and I'm getting empty trace records cause I'm not smart enough (yet) to figure out how to do that. Hopefully I'm getting "warmer". Regards, Brian _______________________________________________ lttng-dev mailing list lttng-dev@lists.lttng.org https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev ^ permalink raw reply [flat|nested] 15+ messages in thread
end of thread, other threads:[~2023-06-28 22:02 UTC | newest]
Thread overview: 15+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2023-05-11 15:53 [lttng-dev] I'm still getting empty ust traces using tracef Brian Hutchinson via lttng-dev
2023-05-11 16:36 ` Brian Hutchinson via lttng-dev
2023-05-11 18:13 ` Mathieu Desnoyers via lttng-dev
2023-05-11 18:15 ` Mathieu Desnoyers via lttng-dev
[not found] ` <CAFZh4h9M0eA1SpHhfquoyrqtwRhHV54f0O4hCkkiojwUmPTC9Q@mail.gmail.com>
[not found] ` <fdbcfbd9-742d-dc61-3667-48a00130d6aa@efficios.com>
2023-05-12 13:34 ` Mathieu Desnoyers via lttng-dev
2023-05-12 14:52 ` Brian Hutchinson via lttng-dev
2023-05-12 14:57 ` Mathieu Desnoyers via lttng-dev
2023-06-20 22:02 ` Brian Hutchinson via lttng-dev
2023-06-21 20:21 ` Mathieu Desnoyers via lttng-dev
2023-06-21 22:02 ` Brian Hutchinson via lttng-dev
2023-06-28 22:02 ` Brian Hutchinson via lttng-dev
2023-05-11 18:38 ` Brian Hutchinson via lttng-dev
2023-05-11 18:14 ` Michael Jeanson via lttng-dev
2023-05-11 18:42 ` Brian Hutchinson via lttng-dev
2023-05-11 18:57 ` Brian Hutchinson via lttng-dev
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox