[lttng-dev] I'm still getting empty ust traces using tracef

[lttng-dev] I'm still getting empty ust traces using tracef

[Brian Hutchinson via lttng-dev]

Hi,

I posted a while ago (thread - Using lttng 2.11 and UST doesn't appear
to work - getting empty trace files) about this problem I'm having
with getting empty trace logs.

I've since upgraded to lttng v2.13 and while I can do a simple hello
world program with tracef and get events in the log files, my more
complicated large multi-threaded app I'm trying to debug is still
getting empty log file traces.

I can list the user space events in my app.

Next I do:

lttng enable-event --userspace 'lttng_ust_tracef:*'

... to enable the events, start lttng, start my app, .... and I get a
trace directory structure that's empty.

I feel like I've read every thread in the archives about people having
the same problem.

I did try using LD_PRELOAD with various libs thinking that was the
problem but so far I'm still getting empty traces.

So far I've tried:

LD_PRELOAD=liblttng-ust-libc-wrapper.so.1:liblttng-ust-pthread-wrapper.so.1:liblttng-ust-dl.so.1:liblttng-ust-fork.so.1:liblttng-ust-fd.so.1
/usr/local/bin/my_app

I guess one question I have is how do I determine which "helper libs"
I need to preload?

The application I'm working on is made up of a bunch of smaller static
libs linked together into one big executable and that is linked with
-llttng-ust and -ldl.

I'm pretty stuck at the moment.  Anyone have any wisdom on what I
might be doing wrong or how I can tell why I'm not getting events in
the logs?

Thanks,

Brian
_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [lttng-dev] I'm still getting empty ust traces using tracef

[Brian Hutchinson via lttng-dev]

... more background.  I've always used ltt in the kernel so I don't
have much experience with the user side of it and especially
multi-threaded, multi-core so I'm probably missing some fundamental
concepts that I need to understand.

Regards,

Brian

On Thu, May 11, 2023 at 11:53 AM Brian Hutchinson <b.hutchman@gmail.com> wrote:
>
> Hi,
>
> I posted a while ago (thread - Using lttng 2.11 and UST doesn't appear
> to work - getting empty trace files) about this problem I'm having
> with getting empty trace logs.
>
> I've since upgraded to lttng v2.13 and while I can do a simple hello
> world program with tracef and get events in the log files, my more
> complicated large multi-threaded app I'm trying to debug is still
> getting empty log file traces.
>
> I can list the user space events in my app.
>
> Next I do:
>
> lttng enable-event --userspace 'lttng_ust_tracef:*'
>
> ... to enable the events, start lttng, start my app, .... and I get a
> trace directory structure that's empty.
>
> I feel like I've read every thread in the archives about people having
> the same problem.
>
> I did try using LD_PRELOAD with various libs thinking that was the
> problem but so far I'm still getting empty traces.
>
> So far I've tried:
>
> LD_PRELOAD=liblttng-ust-libc-wrapper.so.1:liblttng-ust-pthread-wrapper.so.1:liblttng-ust-dl.so.1:liblttng-ust-fork.so.1:liblttng-ust-fd.so.1
> /usr/local/bin/my_app
>
> I guess one question I have is how do I determine which "helper libs"
> I need to preload?
>
> The application I'm working on is made up of a bunch of smaller static
> libs linked together into one big executable and that is linked with
> -llttng-ust and -ldl.
>
> I'm pretty stuck at the moment.  Anyone have any wisdom on what I
> might be doing wrong or how I can tell why I'm not getting events in
> the logs?
>
> Thanks,
>
> Brian
_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [lttng-dev] I'm still getting empty ust traces using tracef

[Mathieu Desnoyers via lttng-dev]

On 2023-05-11 12:36, Brian Hutchinson via lttng-dev wrote:
> ... more background.  I've always used ltt in the kernel so I don't
> have much experience with the user side of it and especially
> multi-threaded, multi-core so I'm probably missing some fundamental
> concepts that I need to understand.

Which are the exact versions of LTTng-UST and LTTng-Tools you are using 
now ? (2.13.N or which git commit ?)

Thanks,

Mathieu

> 
> Regards,
> 
> Brian
> 
> On Thu, May 11, 2023 at 11:53 AM Brian Hutchinson <b.hutchman@gmail.com> wrote:
>>
>> Hi,
>>
>> I posted a while ago (thread - Using lttng 2.11 and UST doesn't appear
>> to work - getting empty trace files) about this problem I'm having
>> with getting empty trace logs.
>>
>> I've since upgraded to lttng v2.13 and while I can do a simple hello
>> world program with tracef and get events in the log files, my more
>> complicated large multi-threaded app I'm trying to debug is still
>> getting empty log file traces.
>>
>> I can list the user space events in my app.
>>
>> Next I do:
>>
>> lttng enable-event --userspace 'lttng_ust_tracef:*'
>>
>> ... to enable the events, start lttng, start my app, .... and I get a
>> trace directory structure that's empty.
>>
>> I feel like I've read every thread in the archives about people having
>> the same problem.
>>
>> I did try using LD_PRELOAD with various libs thinking that was the
>> problem but so far I'm still getting empty traces.
>>
>> So far I've tried:
>>
>> LD_PRELOAD=liblttng-ust-libc-wrapper.so.1:liblttng-ust-pthread-wrapper.so.1:liblttng-ust-dl.so.1:liblttng-ust-fork.so.1:liblttng-ust-fd.so.1
>> /usr/local/bin/my_app
>>
>> I guess one question I have is how do I determine which "helper libs"
>> I need to preload?
>>
>> The application I'm working on is made up of a bunch of smaller static
>> libs linked together into one big executable and that is linked with
>> -llttng-ust and -ldl.
>>
>> I'm pretty stuck at the moment.  Anyone have any wisdom on what I
>> might be doing wrong or how I can tell why I'm not getting events in
>> the logs?
>>
>> Thanks,
>>
>> Brian
> _______________________________________________
> lttng-dev mailing list
> lttng-dev@lists.lttng.org
> https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

-- 
Mathieu Desnoyers
EfficiOS Inc.
https://www.efficios.com

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [lttng-dev] I'm still getting empty ust traces using tracef

[Michael Jeanson via lttng-dev]

On 2023-05-11 12:36, Brian Hutchinson via lttng-dev wrote:
> ... more background.  I've always used ltt in the kernel so I don't
> have much experience with the user side of it and especially
> multi-threaded, multi-core so I'm probably missing some fundamental
> concepts that I need to understand.
> 
> Regards,
> 
> Brian

Hi,

If you are static linking 'liblttng-ust' then the library constructor won't be 
executed and you'll get no traces. While this setup is untested and 
unsupported, you could call the UST constructor from your application early 
initialisation but that would require patching lttng-ust to export the 
'lttng_ust_ctor' symbol.

And the first thing to do to start diagnosing UST tracing issues is to run 
your application with hte LTTNG_UST_DEBUG environment variable set, see [1] 
for details.

Cheers,

Michael

[1] https://lttng.org/man/3/lttng-ust/v2.13/#doc-_environment_variables

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [lttng-dev] I'm still getting empty ust traces using tracef

[Mathieu Desnoyers via lttng-dev]

On 2023-05-11 14:13, Mathieu Desnoyers via lttng-dev wrote:
> On 2023-05-11 12:36, Brian Hutchinson via lttng-dev wrote:
>> ... more background.  I've always used ltt in the kernel so I don't
>> have much experience with the user side of it and especially
>> multi-threaded, multi-core so I'm probably missing some fundamental
>> concepts that I need to understand.
> 
> Which are the exact versions of LTTng-UST and LTTng-Tools you are using 
> now ? (2.13.N or which git commit ?)
> 

Also, can you try using lttng-ust stable-2.13 branch, which includes the following commit ?

commit be2ca8b563bab81be15cbce7b9f52422369f79f7
Author: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Date:   Tue Feb 21 14:29:49 2023 -0500

     Fix: Reevaluate LTTNG_UST_TRACEPOINT_DEFINE each time tracepoint.h is included
     
     Fix issues with missing symbols in use-cases where tracef.h is included
     before defining LTTNG_UST_TRACEPOINT_DEFINE, e.g.:
     
      #include <lttng/tracef.h>
      #define LTTNG_UST_TRACEPOINT_DEFINE
      #include <provider.h>
     
     It is caused by the fact that tracef.h includes tracepoint.h in a
     context which has LTTNG_UST_TRACEPOINT_DEFINE undefined, and this is not
     re-evaluated for the following includes.
     
     Fix this by lifting the definition code in tracepoint.h outside of the
     header include guards, and #undef the old LTTNG_UST__DEFINE_TRACEPOINT
     before re-defining it to its new semantic. Use a new
     _LTTNG_UST_TRACEPOINT_DEFINE_ONCE include guard within the
     LTTNG_UST_TRACEPOINT_DEFINE defined case to ensure symbols are not
     duplicated.
     
     Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
     Change-Id: I0ef720435003a7ca0bfcf29d7bf27866c5ff8678

Thanks,

Mathieu


> Thanks,
> 
> Mathieu
> 
>>
>> Regards,
>>
>> Brian
>>
>> On Thu, May 11, 2023 at 11:53 AM Brian Hutchinson 
>> <b.hutchman@gmail.com> wrote:
>>>
>>> Hi,
>>>
>>> I posted a while ago (thread - Using lttng 2.11 and UST doesn't appear
>>> to work - getting empty trace files) about this problem I'm having
>>> with getting empty trace logs.
>>>
>>> I've since upgraded to lttng v2.13 and while I can do a simple hello
>>> world program with tracef and get events in the log files, my more
>>> complicated large multi-threaded app I'm trying to debug is still
>>> getting empty log file traces.
>>>
>>> I can list the user space events in my app.
>>>
>>> Next I do:
>>>
>>> lttng enable-event --userspace 'lttng_ust_tracef:*'
>>>
>>> ... to enable the events, start lttng, start my app, .... and I get a
>>> trace directory structure that's empty.
>>>
>>> I feel like I've read every thread in the archives about people having
>>> the same problem.
>>>
>>> I did try using LD_PRELOAD with various libs thinking that was the
>>> problem but so far I'm still getting empty traces.
>>>
>>> So far I've tried:
>>>
>>> LD_PRELOAD=liblttng-ust-libc-wrapper.so.1:liblttng-ust-pthread-wrapper.so.1:liblttng-ust-dl.so.1:liblttng-ust-fork.so.1:liblttng-ust-fd.so.1
>>> /usr/local/bin/my_app
>>>
>>> I guess one question I have is how do I determine which "helper libs"
>>> I need to preload?
>>>
>>> The application I'm working on is made up of a bunch of smaller static
>>> libs linked together into one big executable and that is linked with
>>> -llttng-ust and -ldl.
>>>
>>> I'm pretty stuck at the moment.  Anyone have any wisdom on what I
>>> might be doing wrong or how I can tell why I'm not getting events in
>>> the logs?
>>>
>>> Thanks,
>>>
>>> Brian
>> _______________________________________________
>> lttng-dev mailing list
>> lttng-dev@lists.lttng.org
>> https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
> 

-- 
Mathieu Desnoyers
EfficiOS Inc.
https://www.efficios.com

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [lttng-dev] I'm still getting empty ust traces using tracef

[Brian Hutchinson via lttng-dev]

Hi Mathieu,


On Thu, May 11, 2023 at 2:12 PM Mathieu Desnoyers
<mathieu.desnoyers@efficios.com> wrote:
>
> On 2023-05-11 12:36, Brian Hutchinson via lttng-dev wrote:
> > ... more background.  I've always used ltt in the kernel so I don't
> > have much experience with the user side of it and especially
> > multi-threaded, multi-core so I'm probably missing some fundamental
> > concepts that I need to understand.
>
> Which are the exact versions of LTTng-UST and LTTng-Tools you are using
> now ? (2.13.N or which git commit ?)
>
> Thanks,
>
> Mathieu

Built into my rootfs with Yocto:

-rw-r--r-- 4 hutch hutch   977252 May  2 15:32 lttng-tools-2.13.9-r0.aarch64.rpm
-rw-r--r-- 4 hutch hutch 11466008 May  2 15:33
lttng-tools-dbg-2.13.9-r0.aarch64.rpm
-rw-r--r-- 4 hutch hutch    49104 May  2 15:32
lttng-tools-dev-2.13.9-r0.aarch64.rpm
-rw-r--r-- 4 hutch hutch    87452 May  2 15:32
lttng-tools-doc-2.13.9-r0.aarch64.rpm
-rw-r--r-- 4 hutch hutch  1602356 May  2 15:32
lttng-tools-ptest-2.13.9-r0.aarch64.rpm
-rw-r--r-- 4 hutch hutch   867164 May  2 15:32
lttng-tools-src-2.13.9-r0.aarch64.rpm
-rw-r--r-- 4 hutch hutch   219952 May  2 15:31 lttng-ust-2.13.5-r0.aarch64.rpm
-rw-r--r-- 4 hutch hutch     9248 May  2 15:31
lttng-ust-bin-2.13.5-r0.aarch64.rpm
-rw-r--r-- 4 hutch hutch   970248 May  2 15:31
lttng-ust-dbg-2.13.5-r0.aarch64.rpm
-rw-r--r-- 4 hutch hutch    49132 May  2 15:31
lttng-ust-dev-2.13.5-r0.aarch64.rpm
-rw-r--r-- 4 hutch hutch    39864 May  2 15:31
lttng-ust-doc-2.13.5-r0.aarch64.rpm
-rw-r--r-- 4 hutch hutch   258392 May  2 15:31
lttng-ust-src-2.13.5-r0.aarch64.rpm

Regards,

Brian

>
> >
> > Regards,
> >
> > Brian
> >
> > On Thu, May 11, 2023 at 11:53 AM Brian Hutchinson <b.hutchman@gmail.com> wrote:
> >>
> >> Hi,
> >>
> >> I posted a while ago (thread - Using lttng 2.11 and UST doesn't appear
> >> to work - getting empty trace files) about this problem I'm having
> >> with getting empty trace logs.
> >>
> >> I've since upgraded to lttng v2.13 and while I can do a simple hello
> >> world program with tracef and get events in the log files, my more
> >> complicated large multi-threaded app I'm trying to debug is still
> >> getting empty log file traces.
> >>
> >> I can list the user space events in my app.
> >>
> >> Next I do:
> >>
> >> lttng enable-event --userspace 'lttng_ust_tracef:*'
> >>
> >> ... to enable the events, start lttng, start my app, .... and I get a
> >> trace directory structure that's empty.
> >>
> >> I feel like I've read every thread in the archives about people having
> >> the same problem.
> >>
> >> I did try using LD_PRELOAD with various libs thinking that was the
> >> problem but so far I'm still getting empty traces.
> >>
> >> So far I've tried:
> >>
> >> LD_PRELOAD=liblttng-ust-libc-wrapper.so.1:liblttng-ust-pthread-wrapper.so.1:liblttng-ust-dl.so.1:liblttng-ust-fork.so.1:liblttng-ust-fd.so.1
> >> /usr/local/bin/my_app
> >>
> >> I guess one question I have is how do I determine which "helper libs"
> >> I need to preload?
> >>
> >> The application I'm working on is made up of a bunch of smaller static
> >> libs linked together into one big executable and that is linked with
> >> -llttng-ust and -ldl.
> >>
> >> I'm pretty stuck at the moment.  Anyone have any wisdom on what I
> >> might be doing wrong or how I can tell why I'm not getting events in
> >> the logs?
> >>
> >> Thanks,
> >>
> >> Brian
> > _______________________________________________
> > lttng-dev mailing list
> > lttng-dev@lists.lttng.org
> > https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
>
> --
> Mathieu Desnoyers
> EfficiOS Inc.
> https://www.efficios.com
>
_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [lttng-dev] I'm still getting empty ust traces using tracef

[Brian Hutchinson via lttng-dev]

On Thu, May 11, 2023 at 2:14 PM Michael Jeanson <mjeanson@efficios.com> wrote:
>
> On 2023-05-11 12:36, Brian Hutchinson via lttng-dev wrote:
> > ... more background.  I've always used ltt in the kernel so I don't
> > have much experience with the user side of it and especially
> > multi-threaded, multi-core so I'm probably missing some fundamental
> > concepts that I need to understand.
> >
> > Regards,
> >
> > Brian
>
> Hi,
>
> If you are static linking 'liblttng-ust' then the library constructor won't be

All of the final executable is a bunch of statically linked libs (all
.a's) but lttng-ust isn't statically linked (using .so.1, not .a).

I couldn't build/link without linking -llttng-ust and -ldl.

> executed and you'll get no traces. While this setup is untested and
> unsupported, you could call the UST constructor from your application early
> initialisation but that would require patching lttng-ust to export the
> 'lttng_ust_ctor' symbol.
>
> And the first thing to do to start diagnosing UST tracing issues is to run
> your application with hte LTTNG_UST_DEBUG environment variable set, see [1]

Thanks!  I'll check that reference out.

Regards,

Brian

> for details.
>
> Cheers,
>
> Michael
>
> [1] https://lttng.org/man/3/lttng-ust/v2.13/#doc-_environment_variables
>
_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [lttng-dev] I'm still getting empty ust traces using tracef

[Brian Hutchinson via lttng-dev]

On Thu, May 11, 2023 at 12:36 PM Brian Hutchinson <b.hutchman@gmail.com> wrote:
>
> ... more background.  I've always used ltt in the kernel so I don't
> have much experience with the user side of it and especially
> multi-threaded, multi-core so I'm probably missing some fundamental
> concepts that I need to understand.
>
> Regards,
>
> Brian

I'm currently reading up on the new v2.13 "Recording event rule and
event record" features.  I came to 2.13 because I want to be able to
trigger snapshots on an event so I don't fill up storage on an
embedded device.

After I start lttng-systemd --daemonize and then start my app, I see
this when I do lttng list --userspace:

# lttng-sessiond --daemonize
[   85.180717] LTTng: Loaded modules v2.13.9 (Nordicit�é)
[   85.186103] LTTng: Experimental bitwise enum enabled.

# lttng list --userspace
UST events:
-------------

PID: 617 - Name: sh
     lttng_ust_libc:posix_memalign (loglevel: TRACE_DEBUG_LINE (13))
(type: tracepoint)
     lttng_ust_libc:memalign (loglevel: TRACE_DEBUG_LINE (13)) (type:
tracepoint)
     lttng_ust_libc:realloc (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint)
     lttng_ust_libc:calloc (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint)
     lttng_ust_libc:free (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint)
     lttng_ust_libc:malloc (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint)
     lttng_ust_pthread:pthread_mutex_unlock (loglevel:
TRACE_DEBUG_LINE (13)) (type: tracepoint)
     lttng_ust_pthread:pthread_mutex_trylock (loglevel:
TRACE_DEBUG_LINE (13)) (type: tracepoint)
     lttng_ust_pthread:pthread_mutex_lock_acq (loglevel:
TRACE_DEBUG_LINE (13)) (type: tracepoint)
     lttng_ust_pthread:pthread_mutex_lock_req (loglevel:
TRACE_DEBUG_LINE (13)) (type: tracepoint)
     lttng_ust_dl:dlclose (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint)
     lttng_ust_dl:debug_link (loglevel: TRACE_DEBUG_LINE (13)) (type:
tracepoint)
     lttng_ust_dl:build_id (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint)
     lttng_ust_dl:dlmopen (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint)
     lttng_ust_dl:dlopen (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint)
     lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_DEBUG (loglevel:
TRACE_DEBUG (14)) (type: tracepoint)
     lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_DEBUG_LINE
(loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint)
     lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_DEBUG_FUNCTION
(loglevel: TRACE_DEBUG_FUNCTION (12)) (type: tracepoint)
     lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_DEBUG_UNIT
(loglevel: TRACE_DEBUG_UNIT (11)) (type: tracepoint)
     lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_DEBUG_MODULE
(loglevel: TRACE_DEBUG_MODULE (10)) (type: tracepoint)
     lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_DEBUG_PROCESS
(loglevel: TRACE_DEBUG_PROCESS (9)) (type: tracepoint)
     lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_DEBUG_PROGRAM
(loglevel: TRACE_DEBUG_PROGRAM (8)) (type: tracepoint)
     lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_DEBUG_SYSTEM
(loglevel: TRACE_DEBUG_SYSTEM (7)) (type: tracepoint)
     lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_INFO (loglevel:
TRACE_INFO (6)) (type: tracepoint)
     lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_NOTICE
(loglevel: TRACE_NOTICE (5)) (type: tracepoint)
     lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_WARNING
(loglevel: TRACE_WARNING (4)) (type: tracepoint)
     lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_ERR (loglevel:
TRACE_ERR (3)) (type: tracepoint)
     lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_CRIT (loglevel:
TRACE_CRIT (2)) (type: tracepoint)
     lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_ALERT (loglevel:
TRACE_ALERT (1)) (type: tracepoint)
     lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_EMERG (loglevel:
TRACE_EMERG (0)) (type: tracepoint)
     lttng_ust_tracef:event (loglevel: TRACE_DEBUG (14)) (type: tracepoint)
     lttng_ust_lib:unload (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint)
     lttng_ust_lib:debug_link (loglevel: TRACE_DEBUG_LINE (13)) (type:
tracepoint)
     lttng_ust_lib:build_id (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint)
     lttng_ust_lib:load (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint)
     lttng_ust_statedump:end (loglevel: TRACE_DEBUG_LINE (13)) (type:
tracepoint)
     lttng_ust_statedump:procname (loglevel: TRACE_DEBUG_LINE (13))
(type: tracepoint)
     lttng_ust_statedump:debug_link (loglevel: TRACE_DEBUG_LINE (13))
(type: tracepoint)
     lttng_ust_statedump:build_id (loglevel: TRACE_DEBUG_LINE (13))
(type: tracepoint)
     lttng_ust_statedump:bin_info (loglevel: TRACE_DEBUG_LINE (13))
(type: tracepoint)
     lttng_ust_statedump:start (loglevel: TRACE_DEBUG_LINE (13))
(type: tracepoint)

PID: 601 - Name: /usr/loal/bin/my_app
     lttng_ust_libc:posix_memalign (loglevel: TRACE_DEBUG_LINE (13))
(type: tracepoint)
     lttng_ust_libc:memalign (loglevel: TRACE_DEBUG_LINE (13)) (type:
tracepoint)
     lttng_ust_libc:realloc (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint)
     lttng_ust_libc:calloc (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint)
     lttng_ust_libc:free (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint)
     lttng_ust_libc:malloc (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint)
     lttng_ust_pthread:pthread_mutex_unlock (loglevel:
TRACE_DEBUG_LINE (13)) (type: tracepoint)
     lttng_ust_pthread:pthread_mutex_trylock (loglevel:
TRACE_DEBUG_LINE (13)) (type: tracepoint)
     lttng_ust_pthread:pthread_mutex_lock_acq (loglevel:
TRACE_DEBUG_LINE (13)) (type: tracepoint)
     lttng_ust_pthread:pthread_mutex_lock_req (loglevel:
TRACE_DEBUG_LINE (13)) (type: tracepoint)
     lttng_ust_dl:dlclose (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint)
     lttng_ust_dl:debug_link (loglevel: TRACE_DEBUG_LINE (13)) (type:
tracepoint)
     lttng_ust_dl:build_id (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint)
     lttng_ust_dl:dlmopen (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint)
     lttng_ust_dl:dlopen (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint)
     lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_DEBUG (loglevel:
TRACE_DEBUG (14)) (type: tracepoint)
     lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_DEBUG_LINE
(loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint)
     lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_DEBUG_FUNCTION
(loglevel: TRACE_DEBUG_FUNCTION (12)) (type: tracepoint)
     lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_DEBUG_UNIT
(loglevel: TRACE_DEBUG_UNIT (11)) (type: tracepoint)
     lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_DEBUG_MODULE
(loglevel: TRACE_DEBUG_MODULE (10)) (type: tracepoint)
     lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_DEBUG_PROCESS
(loglevel: TRACE_DEBUG_PROCESS (9)) (type: tracepoint)
     lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_DEBUG_PROGRAM
(loglevel: TRACE_DEBUG_PROGRAM (8)) (type: tracepoint)
     lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_DEBUG_SYSTEM
(loglevel: TRACE_DEBUG_SYSTEM (7)) (type: tracepoint)
     lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_INFO (loglevel:
TRACE_INFO (6)) (type: tracepoint)
     lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_NOTICE
(loglevel: TRACE_NOTICE (5)) (type: tracepoint)
     lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_WARNING
(loglevel: TRACE_WARNING (4)) (type: tracepoint)
     lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_ERR (loglevel:
TRACE_ERR (3)) (type: tracepoint)
     lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_CRIT (loglevel:
TRACE_CRIT (2)) (type: tracepoint)
     lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_ALERT (loglevel:
TRACE_ALERT (1)) (type: tracepoint)
     lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_EMERG (loglevel:
TRACE_EMERG (0)) (type: tracepoint)
     lttng_ust_tracef:event (loglevel: TRACE_DEBUG (14)) (type: tracepoint)
     lttng_ust_lib:unload (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint)
     lttng_ust_lib:debug_link (loglevel: TRACE_DEBUG_LINE (13)) (type:
tracepoint)
     lttng_ust_lib:build_id (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint)
     lttng_ust_lib:load (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint)
     lttng_ust_statedump:end (loglevel: TRACE_DEBUG_LINE (13)) (type:
tracepoint)
     lttng_ust_statedump:procname (loglevel: TRACE_DEBUG_LINE (13))
(type: tracepoint)
     lttng_ust_statedump:debug_link (loglevel: TRACE_DEBUG_LINE (13))
(type: tracepoint)
     lttng_ust_statedump:build_id (loglevel: TRACE_DEBUG_LINE (13))
(type: tracepoint)
     lttng_ust_statedump:bin_info (loglevel: TRACE_DEBUG_LINE (13))
(type: tracepoint)
     lttng_ust_statedump:start (loglevel: TRACE_DEBUG_LINE (13))
(type: tracepoint)

PID: 626 - Name: xz
     lttng_ust_libc:posix_memalign (loglevel: TRACE_DEBUG_LINE (13))
(type: tracepoint)
     lttng_ust_libc:memalign (loglevel: TRACE_DEBUG_LINE (13)) (type:
tracepoint)
     lttng_ust_libc:realloc (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint)
     lttng_ust_libc:calloc (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint)
     lttng_ust_libc:free (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint)
     lttng_ust_libc:malloc (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint)
     lttng_ust_pthread:pthread_mutex_unlock (loglevel:
TRACE_DEBUG_LINE (13)) (type: tracepoint)
     lttng_ust_pthread:pthread_mutex_trylock (loglevel:
TRACE_DEBUG_LINE (13)) (type: tracepoint)
     lttng_ust_pthread:pthread_mutex_lock_acq (loglevel:
TRACE_DEBUG_LINE (13)) (type: tracepoint)
     lttng_ust_pthread:pthread_mutex_lock_req (loglevel:
TRACE_DEBUG_LINE (13)) (type: tracepoint)
     lttng_ust_dl:dlclose (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint)
     lttng_ust_dl:debug_link (loglevel: TRACE_DEBUG_LINE (13)) (type:
tracepoint)
     lttng_ust_dl:build_id (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint)
     lttng_ust_dl:dlmopen (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint)
     lttng_ust_dl:dlopen (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint)
     lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_DEBUG (loglevel:
TRACE_DEBUG (14)) (type: tracepoint)
     lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_DEBUG_LINE
(loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint)
     lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_DEBUG_FUNCTION
(loglevel: TRACE_DEBUG_FUNCTION (12)) (type: tracepoint)
     lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_DEBUG_UNIT
(loglevel: TRACE_DEBUG_UNIT (11)) (type: tracepoint)
     lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_DEBUG_MODULE
(loglevel: TRACE_DEBUG_MODULE (10)) (type: tracepoint)
     lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_DEBUG_PROCESS
(loglevel: TRACE_DEBUG_PROCESS (9)) (type: tracepoint)
     lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_DEBUG_PROGRAM
(loglevel: TRACE_DEBUG_PROGRAM (8)) (type: tracepoint)
     lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_DEBUG_SYSTEM
(loglevel: TRACE_DEBUG_SYSTEM (7)) (type: tracepoint)
     lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_INFO (loglevel:
TRACE_INFO (6)) (type: tracepoint)
     lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_NOTICE
(loglevel: TRACE_NOTICE (5)) (type: tracepoint)
     lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_WARNING
(loglevel: TRACE_WARNING (4)) (type: tracepoint)
     lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_ERR (loglevel:
TRACE_ERR (3)) (type: tracepoint)
     lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_CRIT (loglevel:
TRACE_CRIT (2)) (type: tracepoint)
     lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_ALERT (loglevel:
TRACE_ALERT (1)) (type: tracepoint)
     lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_EMERG (loglevel:
TRACE_EMERG (0)) (type: tracepoint)
     lttng_ust_tracef:event (loglevel: TRACE_DEBUG (14)) (type: tracepoint)
     lttng_ust_lib:unload (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint)
     lttng_ust_lib:debug_link (loglevel: TRACE_DEBUG_LINE (13)) (type:
tracepoint)
     lttng_ust_lib:build_id (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint)
     lttng_ust_lib:load (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint)
     lttng_ust_statedump:end (loglevel: TRACE_DEBUG_LINE (13)) (type:
tracepoint)
     lttng_ust_statedump:procname (loglevel: TRACE_DEBUG_LINE (13))
(type: tracepoint)
     lttng_ust_statedump:debug_link (loglevel: TRACE_DEBUG_LINE (13))
(type: tracepoint)
     lttng_ust_statedump:build_id (loglevel: TRACE_DEBUG_LINE (13))
(type: tracepoint)
     lttng_ust_statedump:bin_info (loglevel: TRACE_DEBUG_LINE (13))
(type: tracepoint)
     lttng_ust_statedump:start (loglevel: TRACE_DEBUG_LINE (13))
(type: tracepoint)

I'm not sure why all this showed up 3 times.  Could be because I have
systemd unit file ExecStart calling a script so that my LD_PRELOAD=
line can work.  ExecStart= didn't like LD_PRELOAD.

So now I'm guessing ... I haven't read the fine documentation far
enough yet, that I have to somehow connect my record event rules to
PID 601 somehow and I'm getting empty trace records cause I'm not
smart enough (yet) to figure out how to do that.

Hopefully I'm getting "warmer".

Regards,

Brian
_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [lttng-dev] I'm still getting empty ust traces using tracef

[Mathieu Desnoyers via lttng-dev]

[adding back the mailing list]

On 2023-05-12 09:33, Mathieu Desnoyers wrote:
> On 2023-05-12 00:10, Brian Hutchinson wrote:
>> Hmm, I missed this earlier somehow.
>>
>> So, I'm not the greatest at updating OE and Yocto recipes.  I'm
>> currently using this recipe:
>> http://cgit.openembedded.org/openembedded-core/tree/meta/recipes-kernel/lttng/lttng-ust_2.13.5.bb?h=master
>>
>> ... and it looks like the commit you are talking about is newer.
>>
>> I always think, oh, I'll just update the source URI in the recipe but
>> it's never that simple ... and there are patches in the recipe etc.
>>
>> I've got a sdk (external toolchain) built for my embedded platform.
>> Would it be too hard to just download stable-2.13 of everything and
>> cross compile it outside of Yocto?
>>
>> What do you suggest?
>>
>> And do I need to do anything besides just get 2.13 stable working?  I
>> was kind of confused if I need to put a #define
>> LTTNG_TRACEPOINT_DEFINE somewhere in my code.  I'm not using a
>> tracepoint provider packages at this point
> 
> Hi Brian,
> 
> You might want to provide a trimmed-down reproducer of your issue: 
> example .c compile unit instrumented with tracepoints, example .c 
> compile unit containing the tracepoint probes, and the log of the 
> console when this application is run with LTTNG_UST_DEBUG=1.
> 


> Thanks,
> 
> Mathieu
> 

-- 
Mathieu Desnoyers
EfficiOS Inc.
https://www.efficios.com

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [lttng-dev] I'm still getting empty ust traces using tracef

[Brian Hutchinson via lttng-dev]

Hi Mathieu,

On Fri, May 12, 2023 at 9:33 AM Mathieu Desnoyers
<mathieu.desnoyers@efficios.com> wrote:
>
> On 2023-05-12 00:10, Brian Hutchinson wrote:
> > Hmm, I missed this earlier somehow.
> >
> > So, I'm not the greatest at updating OE and Yocto recipes.  I'm
> > currently using this recipe:
> > http://cgit.openembedded.org/openembedded-core/tree/meta/recipes-kernel/lttng/lttng-ust_2.13.5.bb?h=master
> >
> > ... and it looks like the commit you are talking about is newer.
> >
> > I always think, oh, I'll just update the source URI in the recipe but
> > it's never that simple ... and there are patches in the recipe etc.
> >
> > I've got a sdk (external toolchain) built for my embedded platform.
> > Would it be too hard to just download stable-2.13 of everything and
> > cross compile it outside of Yocto?
> >
> > What do you suggest?
> >
> > And do I need to do anything besides just get 2.13 stable working?  I
> > was kind of confused if I need to put a #define
> > LTTNG_TRACEPOINT_DEFINE somewhere in my code.  I'm not using a
> > tracepoint provider packages at this point
>
> Hi Brian,
>
> You might want to provide a trimmed-down reproducer of your issue:
> example .c compile unit instrumented with tracepoints, example .c
> compile unit containing the tracepoint probes, and the log of the
> console when this application is run with LTTNG_UST_DEBUG=1.

The code has two different areas where I'm trying to use tracef.  The
way the app is put together, each of these areas end up becoming
static libs that all get lumped together to make the final executable
(which is then linked with -llttng-ust and -ldl).

If I'm reading between the lines correctly with respect to the commit
you pointed out (that I'm missing), if I reduce the inclusion of I
#include <lttng/tracef.h> to one instance (like with the hello world
that worked), I'm thinking the version I have might work.

I don't know how I could trim down the large multi threaded app I'm
trying to debug to share.

Another dynamic I should mention in full disclosure, the app in
question has been ported from a different OS and was on a single core
cpu.  The new host ( imx8) is a quad core A53 and since the app wasn't
written for multicore, the cpu's are isolated and systemd is starting
the app on cpu 0 but once it's up it switches it's affinity to cpu 1
so I don't know if that's a factor here or not so just mentioning it.

I did try with LTTNG_UST_DEBUG=1 last night and it didn't put out much:

export LTTNG_UST_DEBUG=1
# systemctl start my_app

#lttng create my_tc_trace --output=/tmp/my_tc_trace
Spawning a session daemon
libringbuffer-clients[711/711]
: LTT : ltt ring buffer client
"relay-metadata-mmap" init
(in lttng_ring_buffer_metadata_client_init() at
../../../lttng-ust-2.13.5/src/common/ringbuffer-clients/metadata-template.h:364)
libringbuffer-clients[711/711]: LTT : ltt ring buffer client
"relay-overwrite-mmap" init
(in lttng_ring_buffer_client_overwrite_init() at
../../../lttng-ust-2.13.5/src/common/ringbuffer-clients/template.h:826)
libringbuffer-clients[711/711]: LTT : ltt ring buffer client
"relay-overwrite-rt-mmap" init
(in lttng_ring_buffer_client_overwrite_rt_init() at
../../../lttng-ust-2.13.5/src/common/ringbuffer-clients/template.h:826)
libringbuffer-clients[711/711]: LTT : ltt ring buffer client
"relay-discard-mmap" init
(in lttng_ring_buffer_client_discard_init() at
../../../lttng-ust-2.13.5/src/common/ringbuffer-clients/template.h:826)
libringbuffer-clients[711/711]: LTT : ltt ring buffer client
"relay-discard-rt-mmap" init
(in lttng_ring_buffer_client_discard_rt_init() at
../../../lttng-ust-2.13.5/src/common/ringbuffer-clients/template.h:826)
[  179.384456] LTTng: Loaded modules v2.13.9 (Nordicit�é)
[  179.390366] LTTng: Experimental bitwise enum enabled.
libringbuffer-clients[711/711]: LTT : ltt ring buffer client
"relay-discard-rt-mmap" exit
(in lttng_ring_buffer_client_discard_rt_exit() at
../../../lttng-ust-2.13.5/src/common/ringbuffer-clients/template.h:833)
libringbuffer-clients[711/711]: LTT : ltt ring buffer client
"relay-discard-mmap" exit
(in lttng_ring_buffer_client_discard_exit() at
../../../lttng-ust-2.13.5/src/common/ringbuffer-clients/template.h:833)
libringbuffer-clients[711/711]: LTT : ltt ring buffer client
"relay-overwrite-rt-mmap" exit
(in lttng_ring_buffer_client_overwrite_rt_exit() at
../../../lttng-ust-2.13.5/src/common/ringbuffer-clients/template.h:833)
libringbuffer-clients[711/711]: LTT : ltt ring buffer client
"relay-overwrite-mmap" exit
(in lttng_ring_buffer_client_overwrite_exit() at
../../../lttng-ust-2.13.5/src/common/ringbuffer-clients/template.h:833)
libringbuffer-clients[711/711]: LTT : ltt ring buffer client
"relay-metadata-mmap" exit
(in lttng_ring_buffer_metadata_client_exit() at
../../../lttng-ust-2.13.5/src/common/ringbuffer-clients/metadata-template.h:371)
Session my_tc_trace created.
Traces will be output to /tmp/my_tc_trace

# lttng enable-event --userspace 'lttng_ust_tracef:*'
ust event lttng_ust_tracef:* created in channel channel0

# lttng start
Tracing started for session my_tc_trace

# lttng destroy
Destroying session my_tc_trace..
Session my_tc_trace destroyed

root@localhost:/tmp/my_tc_trace/ust/uid/0# cd 64-bit/
root@localhost:/tmp/my_tc_trace/ust/uid/0/64-bit# ls
channel0_0  channel0_1  channel0_2  channel0_3  index  metadata
root@localhost:/tmp/my_tc_trace/ust/uid/0/64-bit# ls -al
total 20
drwxrwx--- 3 root root  160 Sep 20 10:48 .
drwxrwx--- 3 root root   60 Sep 20 10:48 ..
-rw-rw---- 1 root root 4096 Sep 20 10:48 channel0_0
-rw-rw---- 1 root root 4096 Sep 20 10:48 channel0_1
-rw-rw---- 1 root root 4096 Sep 20 10:48 channel0_2
-rw-rw---- 1 root root 4096 Sep 20 10:48 channel0_3
drwxrwx--- 2 root root  120 Sep 20 10:48 index
-rw-rw---- 1 root root 4096 Sep 20 10:48 metadata
root@localhost:/tmp/my_tc_trace/ust/uid/0/64-bit# cd index/
root@localhost:/tmp/my_tc_trace/ust/uid/0/64-bit/index# ls
channel0_0.idx  channel0_1.idx  channel0_2.idx  channel0_3.idx
root@localhost:/tmp/my_tc_trace/ust/uid/0/64-bit/index# ls -al
total 16
drwxrwx--- 2 root root 120 Sep 20 10:48 .
drwxrwx--- 3 root root 160 Sep 20 10:48 ..
-rw-rw---- 1 root root  88 Sep 20 10:48 channel0_0.idx
-rw-rw---- 1 root root  88 Sep 20 10:48 channel0_1.idx
-rw-rw---- 1 root root  88 Sep 20 10:48 channel0_2.idx
-rw-rw---- 1 root root  88 Sep 20 10:48 channel0_3.idx

No events in the logs, if it were working there would be tons of stuff in there.

But it sounds like my main problem is I need stable-2.13 and I need to
figure out the best way to do that.  Compile from lttng source outside
of yocto (probably not a good idea with all the dependencies that
would also need to be cross compiled) with my generated sdk for the
imx8 or try to update the OE recipe that I'm currently using.

Regards,

Brian





>
> Thanks,
>
> Mathieu
>
> --
> Mathieu Desnoyers
> EfficiOS Inc.
> https://www.efficios.com
>
_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [lttng-dev] I'm still getting empty ust traces using tracef

[Mathieu Desnoyers via lttng-dev]

On 2023-05-12 10:52, Brian Hutchinson wrote:
> Hi Mathieu,
> 
> On Fri, May 12, 2023 at 9:33 AM Mathieu Desnoyers
> <mathieu.desnoyers@efficios.com> wrote:
>>
>> On 2023-05-12 00:10, Brian Hutchinson wrote:
>>> Hmm, I missed this earlier somehow.
>>>
>>> So, I'm not the greatest at updating OE and Yocto recipes.  I'm
>>> currently using this recipe:
>>> http://cgit.openembedded.org/openembedded-core/tree/meta/recipes-kernel/lttng/lttng-ust_2.13.5.bb?h=master
>>>
>>> ... and it looks like the commit you are talking about is newer.
>>>
>>> I always think, oh, I'll just update the source URI in the recipe but
>>> it's never that simple ... and there are patches in the recipe etc.
>>>
>>> I've got a sdk (external toolchain) built for my embedded platform.
>>> Would it be too hard to just download stable-2.13 of everything and
>>> cross compile it outside of Yocto?
>>>
>>> What do you suggest?
>>>
>>> And do I need to do anything besides just get 2.13 stable working?  I
>>> was kind of confused if I need to put a #define
>>> LTTNG_TRACEPOINT_DEFINE somewhere in my code.  I'm not using a
>>> tracepoint provider packages at this point
>>
>> Hi Brian,
>>
>> You might want to provide a trimmed-down reproducer of your issue:
>> example .c compile unit instrumented with tracepoints, example .c
>> compile unit containing the tracepoint probes, and the log of the
>> console when this application is run with LTTNG_UST_DEBUG=1.
> 
> The code has two different areas where I'm trying to use tracef.  The
> way the app is put together, each of these areas end up becoming
> static libs that all get lumped together to make the final executable
> (which is then linked with -llttng-ust and -ldl).
> 
> If I'm reading between the lines correctly with respect to the commit
> you pointed out (that I'm missing), if I reduce the inclusion of I
> #include <lttng/tracef.h> to one instance (like with the hello world
> that worked), I'm thinking the version I have might work.
> 
> I don't know how I could trim down the large multi threaded app I'm
> trying to debug to share.
> 
> Another dynamic I should mention in full disclosure, the app in
> question has been ported from a different OS and was on a single core
> cpu.  The new host ( imx8) is a quad core A53 and since the app wasn't
> written for multicore, the cpu's are isolated and systemd is starting
> the app on cpu 0 but once it's up it switches it's affinity to cpu 1
> so I don't know if that's a factor here or not so just mentioning it.
> 
> I did try with LTTNG_UST_DEBUG=1 last night and it didn't put out much:
> 
> export LTTNG_UST_DEBUG=1
> # systemctl start my_app

I suspect that because you run your application under systemctl, we are 
not seeing the console output from the application.

The console output below appears to come from liblttng-ust-ctl.so linked 
within lttng-sessiond/consumerd, not the application.

Can you find a way to run your application and capture the console output ?

Thanks,

Mathieu


> 
> #lttng create my_tc_trace --output=/tmp/my_tc_trace
> Spawning a session daemon
> libringbuffer-clients[711/711]
> : LTT : ltt ring buffer client
> "relay-metadata-mmap" init
> (in lttng_ring_buffer_metadata_client_init() at
> ../../../lttng-ust-2.13.5/src/common/ringbuffer-clients/metadata-template.h:364)
> libringbuffer-clients[711/711]: LTT : ltt ring buffer client
> "relay-overwrite-mmap" init
> (in lttng_ring_buffer_client_overwrite_init() at
> ../../../lttng-ust-2.13.5/src/common/ringbuffer-clients/template.h:826)
> libringbuffer-clients[711/711]: LTT : ltt ring buffer client
> "relay-overwrite-rt-mmap" init
> (in lttng_ring_buffer_client_overwrite_rt_init() at
> ../../../lttng-ust-2.13.5/src/common/ringbuffer-clients/template.h:826)
> libringbuffer-clients[711/711]: LTT : ltt ring buffer client
> "relay-discard-mmap" init
> (in lttng_ring_buffer_client_discard_init() at
> ../../../lttng-ust-2.13.5/src/common/ringbuffer-clients/template.h:826)
> libringbuffer-clients[711/711]: LTT : ltt ring buffer client
> "relay-discard-rt-mmap" init
> (in lttng_ring_buffer_client_discard_rt_init() at
> ../../../lttng-ust-2.13.5/src/common/ringbuffer-clients/template.h:826)
> [  179.384456] LTTng: Loaded modules v2.13.9 (Nordicit�é)
> [  179.390366] LTTng: Experimental bitwise enum enabled.
> libringbuffer-clients[711/711]: LTT : ltt ring buffer client
> "relay-discard-rt-mmap" exit
> (in lttng_ring_buffer_client_discard_rt_exit() at
> ../../../lttng-ust-2.13.5/src/common/ringbuffer-clients/template.h:833)
> libringbuffer-clients[711/711]: LTT : ltt ring buffer client
> "relay-discard-mmap" exit
> (in lttng_ring_buffer_client_discard_exit() at
> ../../../lttng-ust-2.13.5/src/common/ringbuffer-clients/template.h:833)
> libringbuffer-clients[711/711]: LTT : ltt ring buffer client
> "relay-overwrite-rt-mmap" exit
> (in lttng_ring_buffer_client_overwrite_rt_exit() at
> ../../../lttng-ust-2.13.5/src/common/ringbuffer-clients/template.h:833)
> libringbuffer-clients[711/711]: LTT : ltt ring buffer client
> "relay-overwrite-mmap" exit
> (in lttng_ring_buffer_client_overwrite_exit() at
> ../../../lttng-ust-2.13.5/src/common/ringbuffer-clients/template.h:833)
> libringbuffer-clients[711/711]: LTT : ltt ring buffer client
> "relay-metadata-mmap" exit
> (in lttng_ring_buffer_metadata_client_exit() at
> ../../../lttng-ust-2.13.5/src/common/ringbuffer-clients/metadata-template.h:371)
> Session my_tc_trace created.
> Traces will be output to /tmp/my_tc_trace
> 
> # lttng enable-event --userspace 'lttng_ust_tracef:*'
> ust event lttng_ust_tracef:* created in channel channel0
> 
> # lttng start
> Tracing started for session my_tc_trace
> 
> # lttng destroy
> Destroying session my_tc_trace..
> Session my_tc_trace destroyed
> 
> root@localhost:/tmp/my_tc_trace/ust/uid/0# cd 64-bit/
> root@localhost:/tmp/my_tc_trace/ust/uid/0/64-bit# ls
> channel0_0  channel0_1  channel0_2  channel0_3  index  metadata
> root@localhost:/tmp/my_tc_trace/ust/uid/0/64-bit# ls -al
> total 20
> drwxrwx--- 3 root root  160 Sep 20 10:48 .
> drwxrwx--- 3 root root   60 Sep 20 10:48 ..
> -rw-rw---- 1 root root 4096 Sep 20 10:48 channel0_0
> -rw-rw---- 1 root root 4096 Sep 20 10:48 channel0_1
> -rw-rw---- 1 root root 4096 Sep 20 10:48 channel0_2
> -rw-rw---- 1 root root 4096 Sep 20 10:48 channel0_3
> drwxrwx--- 2 root root  120 Sep 20 10:48 index
> -rw-rw---- 1 root root 4096 Sep 20 10:48 metadata
> root@localhost:/tmp/my_tc_trace/ust/uid/0/64-bit# cd index/
> root@localhost:/tmp/my_tc_trace/ust/uid/0/64-bit/index# ls
> channel0_0.idx  channel0_1.idx  channel0_2.idx  channel0_3.idx
> root@localhost:/tmp/my_tc_trace/ust/uid/0/64-bit/index# ls -al
> total 16
> drwxrwx--- 2 root root 120 Sep 20 10:48 .
> drwxrwx--- 3 root root 160 Sep 20 10:48 ..
> -rw-rw---- 1 root root  88 Sep 20 10:48 channel0_0.idx
> -rw-rw---- 1 root root  88 Sep 20 10:48 channel0_1.idx
> -rw-rw---- 1 root root  88 Sep 20 10:48 channel0_2.idx
> -rw-rw---- 1 root root  88 Sep 20 10:48 channel0_3.idx
> 
> No events in the logs, if it were working there would be tons of stuff in there.
> 
> But it sounds like my main problem is I need stable-2.13 and I need to
> figure out the best way to do that.  Compile from lttng source outside
> of yocto (probably not a good idea with all the dependencies that
> would also need to be cross compiled) with my generated sdk for the
> imx8 or try to update the OE recipe that I'm currently using.
> 
> Regards,
> 
> Brian
> 
> 
> 
> 
> 
>>
>> Thanks,
>>
>> Mathieu
>>
>> --
>> Mathieu Desnoyers
>> EfficiOS Inc.
>> https://www.efficios.com
>>

-- 
Mathieu Desnoyers
EfficiOS Inc.
https://www.efficios.com

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [lttng-dev] I'm still getting empty ust traces using tracef

[Brian Hutchinson via lttng-dev]

On Thu, May 11, 2023 at 2:14 PM Mathieu Desnoyers
<mathieu.desnoyers@efficios.com> wrote:
>
> On 2023-05-11 14:13, Mathieu Desnoyers via lttng-dev wrote:
> > On 2023-05-11 12:36, Brian Hutchinson via lttng-dev wrote:
> >> ... more background.  I've always used ltt in the kernel so I don't
> >> have much experience with the user side of it and especially
> >> multi-threaded, multi-core so I'm probably missing some fundamental
> >> concepts that I need to understand.
> >
> > Which are the exact versions of LTTng-UST and LTTng-Tools you are using
> > now ? (2.13.N or which git commit ?)
> >
>
> Also, can you try using lttng-ust stable-2.13 branch, which includes the following commit ?
>
> commit be2ca8b563bab81be15cbce7b9f52422369f79f7
> Author: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
> Date:   Tue Feb 21 14:29:49 2023 -0500
>
>      Fix: Reevaluate LTTNG_UST_TRACEPOINT_DEFINE each time tracepoint.h is included
>
>      Fix issues with missing symbols in use-cases where tracef.h is included
>      before defining LTTNG_UST_TRACEPOINT_DEFINE, e.g.:
>
>       #include <lttng/tracef.h>
>       #define LTTNG_UST_TRACEPOINT_DEFINE
>       #include <provider.h>
>
>      It is caused by the fact that tracef.h includes tracepoint.h in a
>      context which has LTTNG_UST_TRACEPOINT_DEFINE undefined, and this is not
>      re-evaluated for the following includes.
>
>      Fix this by lifting the definition code in tracepoint.h outside of the
>      header include guards, and #undef the old LTTNG_UST__DEFINE_TRACEPOINT
>      before re-defining it to its new semantic. Use a new
>      _LTTNG_UST_TRACEPOINT_DEFINE_ONCE include guard within the
>      LTTNG_UST_TRACEPOINT_DEFINE defined case to ensure symbols are not
>      duplicated.
>
>      Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
>      Change-Id: I0ef720435003a7ca0bfcf29d7bf27866c5ff8678
>

I applied this patch and if I use "tracef" type calls in our
application that is made up of a bunch of static libs ... the UST
trace calls work.  I verified that traces that were called from
several different static libs all worked.

But as soon as I include a "tracepoint" style tracepoint (that uses
trace provider include files etc.) then doing a "lttng list -u"
returns "None" for UST events.

Is there some kind of rule that says a file can't use both tracef and
tracepoint calls?  Is there something special you have to do to use
tracef and tracepoints in same file?  Doing so appears to have broken
everything.

Thanks,

Brian
_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [lttng-dev] I'm still getting empty ust traces using tracef

[Mathieu Desnoyers via lttng-dev]

On 6/20/23 18:02, Brian Hutchinson wrote:
> On Thu, May 11, 2023 at 2:14 PM Mathieu Desnoyers
> <mathieu.desnoyers@efficios.com> wrote:
>>
>> On 2023-05-11 14:13, Mathieu Desnoyers via lttng-dev wrote:
>>> On 2023-05-11 12:36, Brian Hutchinson via lttng-dev wrote:
>>>> ... more background.  I've always used ltt in the kernel so I don't
>>>> have much experience with the user side of it and especially
>>>> multi-threaded, multi-core so I'm probably missing some fundamental
>>>> concepts that I need to understand.
>>>
>>> Which are the exact versions of LTTng-UST and LTTng-Tools you are using
>>> now ? (2.13.N or which git commit ?)
>>>
>>
>> Also, can you try using lttng-ust stable-2.13 branch, which includes the following commit ?
>>
>> commit be2ca8b563bab81be15cbce7b9f52422369f79f7
>> Author: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
>> Date:   Tue Feb 21 14:29:49 2023 -0500
>>
>>       Fix: Reevaluate LTTNG_UST_TRACEPOINT_DEFINE each time tracepoint.h is included
>>
>>       Fix issues with missing symbols in use-cases where tracef.h is included
>>       before defining LTTNG_UST_TRACEPOINT_DEFINE, e.g.:
>>
>>        #include <lttng/tracef.h>
>>        #define LTTNG_UST_TRACEPOINT_DEFINE
>>        #include <provider.h>
>>
>>       It is caused by the fact that tracef.h includes tracepoint.h in a
>>       context which has LTTNG_UST_TRACEPOINT_DEFINE undefined, and this is not
>>       re-evaluated for the following includes.
>>
>>       Fix this by lifting the definition code in tracepoint.h outside of the
>>       header include guards, and #undef the old LTTNG_UST__DEFINE_TRACEPOINT
>>       before re-defining it to its new semantic. Use a new
>>       _LTTNG_UST_TRACEPOINT_DEFINE_ONCE include guard within the
>>       LTTNG_UST_TRACEPOINT_DEFINE defined case to ensure symbols are not
>>       duplicated.
>>
>>       Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
>>       Change-Id: I0ef720435003a7ca0bfcf29d7bf27866c5ff8678
>>
> 
> I applied this patch and if I use "tracef" type calls in our
> application that is made up of a bunch of static libs ... the UST
> trace calls work.  I verified that traces that were called from
> several different static libs all worked.
> 
> But as soon as I include a "tracepoint" style tracepoint (that uses
> trace provider include files etc.) then doing a "lttng list -u"
> returns "None" for UST events.
> 
> Is there some kind of rule that says a file can't use both tracef and
> tracepoint calls?  Is there something special you have to do to use
> tracef and tracepoints in same file?  Doing so appears to have broken
> everything.

It should just work.

Can you provide a minimal example of the compile unit having this
issue ?

Also you mention "static libs". Make sure you do *not* define 
"LTTNG_UST_TRACEPOINT_PROBE_DYNAMIC_LINKAGE" in this case. See the 
lttng-ust(3) man page for details (section "Statically linking the 
tracepoint provider").

Thanks,

Mathieu

-- 
Mathieu Desnoyers
EfficiOS Inc.
https://www.efficios.com

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [lttng-dev] I'm still getting empty ust traces using tracef

[Brian Hutchinson via lttng-dev]

On Wed, Jun 21, 2023 at 4:21 PM Mathieu Desnoyers
<mathieu.desnoyers@efficios.com> wrote:
>
> On 6/20/23 18:02, Brian Hutchinson wrote:
> > On Thu, May 11, 2023 at 2:14 PM Mathieu Desnoyers
> > <mathieu.desnoyers@efficios.com> wrote:
> >>
> >> On 2023-05-11 14:13, Mathieu Desnoyers via lttng-dev wrote:
> >>> On 2023-05-11 12:36, Brian Hutchinson via lttng-dev wrote:
> >>>> ... more background.  I've always used ltt in the kernel so I don't
> >>>> have much experience with the user side of it and especially
> >>>> multi-threaded, multi-core so I'm probably missing some fundamental
> >>>> concepts that I need to understand.
> >>>
> >>> Which are the exact versions of LTTng-UST and LTTng-Tools you are using
> >>> now ? (2.13.N or which git commit ?)
> >>>
> >>
> >> Also, can you try using lttng-ust stable-2.13 branch, which includes the following commit ?
> >>
> >> commit be2ca8b563bab81be15cbce7b9f52422369f79f7
> >> Author: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
> >> Date:   Tue Feb 21 14:29:49 2023 -0500
> >>
> >>       Fix: Reevaluate LTTNG_UST_TRACEPOINT_DEFINE each time tracepoint.h is included
> >>
> >>       Fix issues with missing symbols in use-cases where tracef.h is included
> >>       before defining LTTNG_UST_TRACEPOINT_DEFINE, e.g.:
> >>
> >>        #include <lttng/tracef.h>
> >>        #define LTTNG_UST_TRACEPOINT_DEFINE
> >>        #include <provider.h>
> >>
> >>       It is caused by the fact that tracef.h includes tracepoint.h in a
> >>       context which has LTTNG_UST_TRACEPOINT_DEFINE undefined, and this is not
> >>       re-evaluated for the following includes.
> >>
> >>       Fix this by lifting the definition code in tracepoint.h outside of the
> >>       header include guards, and #undef the old LTTNG_UST__DEFINE_TRACEPOINT
> >>       before re-defining it to its new semantic. Use a new
> >>       _LTTNG_UST_TRACEPOINT_DEFINE_ONCE include guard within the
> >>       LTTNG_UST_TRACEPOINT_DEFINE defined case to ensure symbols are not
> >>       duplicated.
> >>
> >>       Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
> >>       Change-Id: I0ef720435003a7ca0bfcf29d7bf27866c5ff8678
> >>
> >
> > I applied this patch and if I use "tracef" type calls in our
> > application that is made up of a bunch of static libs ... the UST
> > trace calls work.  I verified that traces that were called from
> > several different static libs all worked.
> >
> > But as soon as I include a "tracepoint" style tracepoint (that uses
> > trace provider include files etc.) then doing a "lttng list -u"
> > returns "None" for UST events.
> >
> > Is there some kind of rule that says a file can't use both tracef and
> > tracepoint calls?  Is there something special you have to do to use
> > tracef and tracepoints in same file?  Doing so appears to have broken
> > everything.
>
> It should just work.
>
> Can you provide a minimal example of the compile unit having this
> issue ?
>
> Also you mention "static libs". Make sure you do *not* define
> "LTTNG_UST_TRACEPOINT_PROBE_DYNAMIC_LINKAGE" in this case. See the
> lttng-ust(3) man page for details (section "Statically linking the
> tracepoint provider").

About that.  It's a big project so all of our components are compiled
as static libs and linked into one big executable at the end, I'm not
using lttng-ust static libs.  I'm linking final executable with
-llttng-ust and -ldl

I also have another question I just thought of regarding trace provider code.

We've got a bunch of C code that is being compiled with g++.  I
initially had problems getting anything tracepoint related to work
(with 2.11) and read somewhere that the trace provider code needed to
be compiled with gcc so I built that by hand since our cmake build
system is pretty complicated and just copied the resulting C .obj into
our build directory structure for cmake to pick up and that worked.  I
basically copied the gcc .obj over the broken one that cmake/g++
generated.

I'm not doing that now and am wrapping the -tp.h TRACEPOINT_EVENT
definitions with #ifdef __cplusplus extern "C" {} #endif /*
__cplusplus */

I'll have to go back and check my results as I've been on/off working
on this for a while but if I stub out tracepoint calls and just do
tracef it appears everything works.  If I stub out tracef calls and
just do tracepoint calls I think that works too.  The problem I have
is when I try to do both tracef and tracepoint in same file.  That's
when nothing works.

Is there an order to the includes?

In the C source file in question I do:

#include <lttng/tracef.h>
then later ...
#define TRACEPOINT_DEFINE
#include "my-tp.h"

Hmm, now that I pay more attention, I don't see any mention of
traceprovider code needing to be compiled with gcc vs g++ in the 2.13
documentation.

... and it looks like the traceprovider header changed to use
LTTNG_UST_ in front of everything and I'm still using lttng 2.11
style!  So I think my problem is probably associated with needing to
re-do my traceprovider code to update to 2.13 style.  Don't know if
the C++ thing is still an issue.

Regards,

Brian
_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [lttng-dev] I'm still getting empty ust traces using tracef

[Brian Hutchinson via lttng-dev]

On Wed, Jun 21, 2023 at 6:02 PM Brian Hutchinson <b.hutchman@gmail.com> wrote:
>
> On Wed, Jun 21, 2023 at 4:21 PM Mathieu Desnoyers
> <mathieu.desnoyers@efficios.com> wrote:
> >
> > On 6/20/23 18:02, Brian Hutchinson wrote:
> > > On Thu, May 11, 2023 at 2:14 PM Mathieu Desnoyers
> > > <mathieu.desnoyers@efficios.com> wrote:
> > >>
> > >> On 2023-05-11 14:13, Mathieu Desnoyers via lttng-dev wrote:
> > >>> On 2023-05-11 12:36, Brian Hutchinson via lttng-dev wrote:
> > >>>> ... more background.  I've always used ltt in the kernel so I don't
> > >>>> have much experience with the user side of it and especially
> > >>>> multi-threaded, multi-core so I'm probably missing some fundamental
> > >>>> concepts that I need to understand.
> > >>>
> > >>> Which are the exact versions of LTTng-UST and LTTng-Tools you are using
> > >>> now ? (2.13.N or which git commit ?)
> > >>>
> > >>
> > >> Also, can you try using lttng-ust stable-2.13 branch, which includes the following commit ?
> > >>
> > >> commit be2ca8b563bab81be15cbce7b9f52422369f79f7
> > >> Author: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
> > >> Date:   Tue Feb 21 14:29:49 2023 -0500
> > >>
> > >>       Fix: Reevaluate LTTNG_UST_TRACEPOINT_DEFINE each time tracepoint.h is included
> > >>
> > >>       Fix issues with missing symbols in use-cases where tracef.h is included
> > >>       before defining LTTNG_UST_TRACEPOINT_DEFINE, e.g.:
> > >>
> > >>        #include <lttng/tracef.h>
> > >>        #define LTTNG_UST_TRACEPOINT_DEFINE
> > >>        #include <provider.h>
> > >>
> > >>       It is caused by the fact that tracef.h includes tracepoint.h in a
> > >>       context which has LTTNG_UST_TRACEPOINT_DEFINE undefined, and this is not
> > >>       re-evaluated for the following includes.
> > >>
> > >>       Fix this by lifting the definition code in tracepoint.h outside of the
> > >>       header include guards, and #undef the old LTTNG_UST__DEFINE_TRACEPOINT
> > >>       before re-defining it to its new semantic. Use a new
> > >>       _LTTNG_UST_TRACEPOINT_DEFINE_ONCE include guard within the
> > >>       LTTNG_UST_TRACEPOINT_DEFINE defined case to ensure symbols are not
> > >>       duplicated.
> > >>
> > >>       Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
> > >>       Change-Id: I0ef720435003a7ca0bfcf29d7bf27866c5ff8678
> > >>
> > >
> > > I applied this patch and if I use "tracef" type calls in our
> > > application that is made up of a bunch of static libs ... the UST
> > > trace calls work.  I verified that traces that were called from
> > > several different static libs all worked.
> > >
> > > But as soon as I include a "tracepoint" style tracepoint (that uses
> > > trace provider include files etc.) then doing a "lttng list -u"
> > > returns "None" for UST events.
> > >
> > > Is there some kind of rule that says a file can't use both tracef and
> > > tracepoint calls?  Is there something special you have to do to use
> > > tracef and tracepoints in same file?  Doing so appears to have broken
> > > everything.
> >
> > It should just work.
> >
> > Can you provide a minimal example of the compile unit having this
> > issue ?
> >
> > Also you mention "static libs". Make sure you do *not* define
> > "LTTNG_UST_TRACEPOINT_PROBE_DYNAMIC_LINKAGE" in this case. See the
> > lttng-ust(3) man page for details (section "Statically linking the
> > tracepoint provider").
>
> About that.  It's a big project so all of our components are compiled
> as static libs and linked into one big executable at the end, I'm not
> using lttng-ust static libs.  I'm linking final executable with
> -llttng-ust and -ldl
>
> I also have another question I just thought of regarding trace provider code.
>
> We've got a bunch of C code that is being compiled with g++.  I
> initially had problems getting anything tracepoint related to work
> (with 2.11) and read somewhere that the trace provider code needed to
> be compiled with gcc so I built that by hand since our cmake build
> system is pretty complicated and just copied the resulting C .obj into
> our build directory structure for cmake to pick up and that worked.  I
> basically copied the gcc .obj over the broken one that cmake/g++
> generated.
>
> I'm not doing that now and am wrapping the -tp.h TRACEPOINT_EVENT
> definitions with #ifdef __cplusplus extern "C" {} #endif /*
> __cplusplus */
>
> I'll have to go back and check my results as I've been on/off working
> on this for a while but if I stub out tracepoint calls and just do
> tracef it appears everything works.  If I stub out tracef calls and
> just do tracepoint calls I think that works too.  The problem I have
> is when I try to do both tracef and tracepoint in same file.  That's
> when nothing works.
>
> Is there an order to the includes?
>
> In the C source file in question I do:
>
> #include <lttng/tracef.h>
> then later ...
> #define TRACEPOINT_DEFINE
> #include "my-tp.h"
>
> Hmm, now that I pay more attention, I don't see any mention of
> traceprovider code needing to be compiled with gcc vs g++ in the 2.13
> documentation.
>
> ... and it looks like the traceprovider header changed to use
> LTTNG_UST_ in front of everything and I'm still using lttng 2.11
> style!  So I think my problem is probably associated with needing to
> re-do my traceprovider code to update to 2.13 style.  Don't know if
> the C++ thing is still an issue./
>

I updated my trace provider package I created for lttng 2.11 to 2.13
syntax.  I discovered why lttng list -u wasn't returning anything.
It's because the application had a seg fault.  Here is the debug trace
of that:

root@localhost:# lttng create my_session --output=/tmp/my_trace --snapshot
Spawning a session daemon
libringbuffer-clients[41538/41538]: LTT : ltt ring buffer client
"relay-metadata-mmap" init
(in lttng_ring_buffer_metadata_client_init() at
../../../lttng-ust-2.13.5/src/common/ringbuffer-clients/metadata-template.h:364)
libringbuffer-clients[41538/41538]: LTT : ltt ring buffer client
"relay-overwrite-mmap" init
(in lttng_ring_buffer_client_overwrite_init() at
../../../lttng-ust-2.13.5/src/common/ringbuffer-clients/template.h:826)
libringbuffer-clients[41538/41538]: LTT : ltt ring buffer client
"relay-overwrite-rt-mmap" init
(in lttng_ring_buffer_client_overwrite_rt_init() at
../../../lttng-ust-2.13.5/src/common/ringbuffer-clients/template.h:826)
libringbuffer-clients[41538/41538]: LTT : ltt ring buffer client
"relay-discard-mmap" init
(in lttng_ring_buffer_client_discard_init() at
../../../lttng-ust-2.13.5/src/common/ringbuffer-clients/template.h:826)
libringbuffer-clients[41538/41538]: LTT : ltt ring buffer client
"relay-discard-rt-mmap" init
(in lttng_ring_buffer_client_discard_rt_init() at
../../../lttng-ust-2.13.5/src/common/ringbuffer-clients/template.h:826)
[ 2867.702128] LTTng: Loaded modules v2.13.9 (Nordicit�é)
[ 2867.711872] LTTng: Experimental bitwise enum enabled.
libringbuffer-clients[41538/41538]: LTT : ltt ring buffer client
"relay-discard-rt-mmap" exit
(in lttng_ring_buffer_client_discard_rt_exit() at
../../../lttng-ust-2.13.5/src/common/ringbuffer-clients/template.h:833)
libringbuffer-clients[41538/41538]: LTT : ltt ring buffer client
"relay-discard-mmap" exit
(in lttng_ring_buffer_client_discard_exit() at
../../../lttng-ust-2.13.5/src/common/ringbuffer-clients/template.h:833)
libringbuffer-clients[41538/41538]: LTT : ltt ring buffer client
"relay-overwrite-rt-mmap" exit
(in lttng_ring_buffer_client_overwrite_rt_exit() at
../../../lttng-ust-2.13.5/src/common/ringbuffer-clients/template.h:833)
libringbuffer-clients[41538/41538]: LTT : ltt ring buffer client
"relay-overwrite-mmap" exit
(in lttng_ring_buffer_client_overwrite_exit() at
../../../lttng-ust-2.13.5/src/common/ringbuffer-clients/template.h:833)
libringbuffer-clients[41538/41538]: LTT : ltt ring buffer client
"relay-metadata-mmap" exit
(in lttng_ring_buffer_metadata_client_exit() at
../../../lttng-ust-2.13.5/src/common/ringbuffer-clients/metadata-template.h:371)
Snapshot session my_session created.
Default snapshot output set to /tmp/my_trace
Every channel enabled for this session will be set to mmap output and
default to overwrite mode.


root@localhost:# ./my_app
liblttng_ust[42358/42358]: dlopened liblttng-ust shared library
(liblttng-ust.so.1). (in lttng_ust_ctor() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-ust-comm.c:2217)
liblttng_ust_tracepoint[42358/42358]: Your compiler treats weak
symbols with hidden visibility for integer objects as SAME address
between compile units part of the same module. (in check_weak_hidden()
at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust-tracepoint/tracepoint.c:10)
liblttng_ust_tracepoint[42358/42358]: Your compiler treats weak
symbols with hidden visibility for pointer objects as SAME address
between compile units part of the same module. (in check_weak_hidden()
at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust-tracepoint/tracepoint.c:10)
liblttng_ust_tracepoint[42358/42358]: Your compiler treats weak
symbols with hidden visibility for 24-byte structure objects as SAME
address between compile units part of the same module. (in
check_weak_hidden() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust-tracepoint/trace)
liblttng_ust_tracepoint[42358/42358]: just registered a tracepoints
section from 0xffff90b64630 and having 26 tracepoints (in
lttng_ust_tracepoint_module_register() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust-tracepoint/tracepoint.c:960)
liblttng_ust_tracepoint[42358/42358]: registered tracepoint:
"lttng_ust_statedump:end" (in lttng_ust_tracepoint_module_register()
at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust-tracepoint/tracepoint.c:968)
liblttng_ust_tracepoint[42358/42358]: registered tracepoint:
"lttng_ust_statedump:procname" (in
lttng_ust_tracepoint_module_register() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust-tracepoint/tracepoint.c:968)
liblttng_ust_tracepoint[42358/42358]: registered tracepoint:
"lttng_ust_statedump:debug_link" (in
lttng_ust_tracepoint_module_register() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust-tracepoint/tracepoint.c:968)
liblttng_ust_tracepoint[42358/42358]: registered tracepoint:
"lttng_ust_statedump:build_id" (in
lttng_ust_tracepoint_module_register() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust-tracepoint/tracepoint.c:968)
liblttng_ust_tracepoint[42358/42358]: registered tracepoint:
"lttng_ust_statedump:bin_info" (in
lttng_ust_tracepoint_module_register() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust-tracepoint/tracepoint.c:968)
liblttng_ust_tracepoint[42358/42358]: registered tracepoint:
"lttng_ust_statedump:start" (in lttng_ust_tracepoint_module_register()
at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust-tracepoint/tracepoint.c:968)
liblttng_ust_tracepoint[42358/42358]: registered tracepoint:
"lttng_ust_lib:unload" (in lttng_ust_tracepoint_module_register() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust-tracepoint/tracepoint.c:968)
liblttng_ust_tracepoint[42358/42358]: registered tracepoint:
"lttng_ust_lib:debug_link" (in lttng_ust_tracepoint_module_register()
at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust-tracepoint/tracepoint.c:968)
liblttng_ust_tracepoint[42358/42358]: registered tracepoint:
"lttng_ust_lib:build_id" (in lttng_ust_tracepoint_module_register() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust-tracepoint/tracepoint.c:968)
liblttng_ust_tracepoint[42358/42358]: registered tracepoint:
"lttng_ust_lib:load" (in lttng_ust_tracepoint_module_register() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust-tracepoint/tracepoint.c:968)
liblttng_ust_tracepoint[42358/42358]: registered tracepoint:
"lttng_ust_tracef:event" (in lttng_ust_tracepoint_module_register() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust-tracepoint/tracepoint.c:968)
liblttng_ust_tracepoint[42358/42358]: registered tracepoint:
"lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_DEBUG" (in
lttng_ust_tracepoint_module_register() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust-tracepoint/tracepoint.c:968)
liblttng_ust_tracepoint[42358/42358]: registered tracepoint:
"lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_DEBUG_LINE" (in
lttng_ust_tracepoint_module_register() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust-tracepoint/tracepoint.c:968)
liblttng_ust_tracepoint[42358/42358]: registered tracepoint:
"lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_DEBUG_FUNCTION" (in
lttng_ust_tracepoint_module_register() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust-tracepoint/tracepoint.c:968)
liblttng_ust_tracepoint[42358/42358]: registered tracepoint:
"lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_DEBUG_UNIT" (in
lttng_ust_tracepoint_module_register() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust-tracepoint/tracepoint.c:968)
liblttng_ust_tracepoint[42358/42358]: registered tracepoint:
"lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_DEBUG_MODULE" (in
lttng_ust_tracepoint_module_register() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust-tracepoint/tracepoint.c:968)
liblttng_ust_tracepoint[42358/42358]: registered tracepoint:
"lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_DEBUG_PROCESS" (in
lttng_ust_tracepoint_module_register() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust-tracepoint/tracepoint.c:968)
liblttng_ust_tracepoint[42358/42358]: registered tracepoint:
"lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_DEBUG_PROGRAM" (in
lttng_ust_tracepoint_module_register() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust-tracepoint/tracepoint.c:968)
liblttng_ust_tracepoint[42358/42358]: registered tracepoint:
"lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_DEBUG_SYSTEM" (in
lttng_ust_tracepoint_module_register() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust-tracepoint/tracepoint.c:968)
liblttng_ust_tracepoint[42358/42358]: registered tracepoint:
"lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_INFO" (in
lttng_ust_tracepoint_module_register() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust-tracepoint/tracepoint.c:968)
liblttng_ust_tracepoint[42358/42358]: registered tracepoint:
"lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_NOTICE" (in
lttng_ust_tracepoint_module_register() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust-tracepoint/tracepoint.c:968)
liblttng_ust_tracepoint[42358/42358]: registered tracepoint:
"lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_WARNING" (in
lttng_ust_tracepoint_module_register() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust-tracepoint/tracepoint.c:968)
liblttng_ust_tracepoint[42358/42358]: registered tracepoint:
"lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_ERR" (in
lttng_ust_tracepoint_module_register() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust-tracepoint/tracepoint.c:968)
liblttng_ust_tracepoint[42358/42358]: registered tracepoint:
"lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_CRIT" (in
lttng_ust_tracepoint_module_register() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust-tracepoint/tracepoint.c:968)
liblttng_ust_tracepoint[42358/42358]: registered tracepoint:
"lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_ALERT" (in
lttng_ust_tracepoint_module_register() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust-tracepoint/tracepoint.c:968)
liblttng_ust_tracepoint[42358/42358]: registered tracepoint:
"lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_EMERG" (in
lttng_ust_tracepoint_module_register() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust-tracepoint/tracepoint.c:968)
liblttng_ust[42358/42358]: Provider "lttng_ust_statedump" accepted,
version 3.0 is compatible with LTTng UST provider version 3.0. (in
check_provider_version() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-probes.c:50)
liblttng_ust[42358/42358]: Provider "lttng_ust_statedump" accepted,
version 3.0 is compatible with LTTng UST provider version 3.0. (in
check_provider_version() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-probes.c:50)
liblttng_ust[42358/42358]: Provider "lttng_ust_statedump" accepted,
version 3.0 is compatible with LTTng UST provider version 3.0. (in
check_provider_version() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-probes.c:50)
liblttng_ust[42358/42358]: Provider "lttng_ust_statedump" accepted,
version 3.0 is compatible with LTTng UST provider version 3.0. (in
check_provider_version() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-probes.c:50)
liblttng_ust[42358/42358]: Provider "lttng_ust_statedump" accepted,
version 3.0 is compatible with LTTng UST provider version 3.0. (in
check_provider_version() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-probes.c:50)
liblttng_ust[42358/42358]: Provider "lttng_ust_statedump" accepted,
version 3.0 is compatible with LTTng UST provider version 3.0. (in
check_provider_version() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-probes.c:50)
liblttng_ust[42358/42358]: Provider "lttng_ust_statedump" accepted,
version 3.0 is compatible with LTTng UST provider version 3.0. (in
check_provider_version() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-probes.c:50)
liblttng_ust[42358/42358]: adding probe lttng_ust_statedump containing
6 events to lazy registration list (in lttng_ust_probe_register() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-probes.c:254)
libringbuffer-clients[42358/42358]: LTT : ltt ring buffer client
"relay-metadata-mmap" init
(in lttng_ring_buffer_metadata_client_init() at
../../../lttng-ust-2.13.5/src/common/ringbuffer-clients/metadata-template.h:364)
libringbuffer-clients[42358/42358]: LTT : ltt ring buffer client
"relay-overwrite-mmap" init
(in lttng_ring_buffer_client_overwrite_init() at
../../../lttng-ust-2.13.5/src/common/ringbuffer-clients/template.h:826)
libringbuffer-clients[42358/42358]: LTT : ltt ring buffer client
"relay-overwrite-rt-mmap" init
(in lttng_ring_buffer_client_overwrite_rt_init() at
../../../lttng-ust-2.13.5/src/common/ringbuffer-clients/template.h:826)
libringbuffer-clients[42358/42358]: LTT : ltt ring buffer client
"relay-discard-mmap" init
(in lttng_ring_buffer_client_discard_init() at
../../../lttng-ust-2.13.5/src/common/ringbuffer-clients/template.h:826)
libringbuffer-clients[42358/42358]: LTT : ltt ring buffer client
"relay-discard-rt-mmap" init
(in lttng_ring_buffer_client_discard_rt_init() at
../../../lttng-ust-2.13.5/src/common/ringbuffer-clients/template.h:826)
liblttng_ust[42358/42360]: Info: sessiond not accepting connections to
local apps socket (in ust_listener_thread() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-ust-comm.c:1884)
liblttng_ust[42358/42360]: Waiting for local apps sessiond (in
wait_for_sessiond() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-ust-comm.c:1758)
liblttng_ust[42358/42359]: Message Received "Get Tracer Version" (65),
Handle "root" (0) (in print_cmd() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-ust-comm.c:476)
liblttng_ust[42358/42359]: Return value: 0 (in handle_message() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-ust-comm.c:1412)
liblttng_ust[42358/42359]: message successfully sent (in send_reply()
at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-ust-comm.c:656)
liblttng_ust[42358/42359]: Message Received "Create event notifier
group" (70), Handle "root" (0) (in print_cmd() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-ust-comm.c:476)
liblttng_ust[42358/42359]: message successfully sent (in send_reply()
at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-ust-comm.c:656)
liblttng_ust[42358/42359]: Return value: 1 (in handle_message() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-ust-comm.c:1412)
liblttng_ust[42358/42359]: message successfully sent (in send_reply()
at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-ust-comm.c:656)
liblttng_ust[42358/42359]: Message Received "Create Counter" (192),
Handle "event_notifier_" (1) (in print_cmd() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-ust-comm.c:476)
liblttng_ust[42358/42359]: message successfully sent (in send_reply()
at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-ust-comm.c:656)
liblttng_ust[42358/42359]: counter data received (in handle_message()
at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-ust-comm.c:1266)
liblttng_ust[42358/42359]: Return value: 2 (in handle_message() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-ust-comm.c:1412)
liblttng_ust[42358/42359]: message successfully sent (in send_reply()
at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-ust-comm.c:656)
liblttng_ust[42358/42359]: Message Received "Create Counter CPU"
(209), Handle "event_notifier " (2) (in print_cmd() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-ust-comm.c:476)
liblttng_ust[42358/42359]: message successfully sent (in send_reply()
at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-ust-comm.c:656)
liblttng_ust[42358/42359]: Return value: 0 (in handle_message() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-ust-comm.c:1412)
liblttng_ust[42358/42359]: message successfully sent (in send_reply()
at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-ust-comm.c:656)
liblttng_ust[42358/42359]: Message Received "Create Counter CPU"
(209), Handle "event_notifier " (2) (in print_cmd() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-ust-comm.c:476)
liblttng_ust[42358/42359]: message successfully sent (in send_reply()
at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-ust-comm.c:656)
liblttng_ust[42358/42359]: Return value: 0 (in handle_message() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-ust-comm.c:1412)
liblttng_ust[42358/42359]: message successfully sent (in send_reply()
at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-ust-comm.c:656)
liblttng_ust[42358/42359]: Message Received "Create Counter CPU"
(209), Handle "event_notifier " (2) (in print_cmd() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-ust-comm.c:476)
liblttng_ust[42358/42359]: message successfully sent (in send_reply()
at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-ust-comm.c:656)
liblttng_ust[42358/42359]: Return value: 0 (in handle_message() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-ust-comm.c:1412)
liblttng_ust[42358/42359]: message successfully sent (in send_reply()
at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-ust-comm.c:656)
liblttng_ust[42358/42359]: Message Received "Create Counter CPU"
(209), Handle "event_notifier " (2) (in print_cmd() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-ust-comm.c:476)
liblttng_ust[42358/42359]: message successfully sent (in send_reply()
at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-ust-comm.c:656)
liblttng_ust[42358/42359]: Return value: 0 (in handle_message() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-ust-comm.c:1412)
liblttng_ust[42358/42359]: message successfully sent (in send_reply()
at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-ust-comm.c:656)
liblttng_ust[42358/42359]: Message Received "Registration Done" (68),
Handle "root" (0) (in print_cmd() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-ust-comm.c:476)
liblttng_ust[42358/42358]: Provider "lttng_ust_lib" accepted, version
3.0 is compatible with LTTng UST provider version 3.0. (in
check_provider_version() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-probes.c:50)
liblttng_ust[42358/42358]: Provider "lttng_ust_lib" accepted, version
3.0 is compatible with LTTng UST provider version 3.0. (in
check_provider_version() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-probes.c:50)
liblttng_ust[42358/42358]: Provider "lttng_ust_lib" accepted, version
3.0 is compatible with LTTng UST provider version 3.0. (in
check_provider_version() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-probes.c:50)
liblttng_ust[42358/42358]: Provider "lttng_ust_lib" accepted, version
3.0 is compatible with LTTng UST provider version 3.0. (in
check_provider_version() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-probes.c:50)
liblttng_ust[42358/42358]: Provider "lttng_ust_lib" accepted, version
3.0 is compatible with LTTng UST provider version 3.0. (in
check_provider_version() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-probes.c:50)
liblttng_ust[42358/42359]: Return value: 0 (in handle_message() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-ust-comm.c:1412)
liblttng_ust[42358/42358]: adding probe lttng_ust_lib containing 4
events to lazy registration list (in lttng_ust_probe_register() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-probes.c:254)
liblttng_ust_tracepoint[42358/42358]: Release queue of unregistered
tracepoint probes. (in lttng_ust_tp_probe_prune_release_queue() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust-tracepoint/tracepoint.c:787)
liblttng_ust[42358/42358]: Provider "lttng_ust_tracef" accepted,
version 3.0 is compatible with LTTng UST provider version 3.0. (in
check_provider_version() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-probes.c:50)
liblttng_ust[42358/42358]: Provider "lttng_ust_tracef" accepted,
version 3.0 is compatible with LTTng UST provider version 3.0. (in
check_provider_version() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-probes.c:50)
liblttng_ust[42358/42358]: adding probe lttng_ust_tracef containing 1
events to lazy registration list (in lttng_ust_probe_register() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-probes.c:254)
liblttng_ust_tracepoint[42358/42358]: Release queue of unregistered
tracepoint probes. (in lttng_ust_tp_probe_prune_release_queue() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust-tracepoint/tracepoint.c:787)
liblttng_ust[42358/42358]: Provider "lttng_ust_tracelog" accepted,
version 3.0 is compatible with LTTng UST provider version 3.0. (in
check_provider_version() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-probes.c:50)
liblttng_ust[42358/42358]: Provider "lttng_ust_tracelog" accepted,
version 3.0 is compatible with LTTng UST provider version 3.0. (in
check_provider_version() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-probes.c:50)
liblttng_ust[42358/42358]: Provider "lttng_ust_tracelog" accepted,
version 3.0 is compatible with LTTng UST provider version 3.0. (in
check_provider_version() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-probes.c:50)
liblttng_ust[42358/42358]: Provider "lttng_ust_tracelog" accepted,
version 3.0 is compatible with LTTng UST provider version 3.0. (in
check_provider_version() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-probes.c:50)
liblttng_ust[42358/42358]: Provider "lttng_ust_tracelog" accepted,
version 3.0 is compatible with LTTng UST provider version 3.0. (in
check_provider_version() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-probes.c:50)
liblttng_ust[42358/42358]: Provider "lttng_ust_tracelog" accepted,
version 3.0 is compatible with LTTng UST provider version 3.0. (in
check_provider_version() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-probes.c:50)
liblttng_ust[42358/42358]: Provider "lttng_ust_tracelog" accepted,
version 3.0 is compatible with LTTng UST provider version 3.0. (in
check_provider_version() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-probes.c:50)
liblttng_ust[42358/42358]: Provider "lttng_ust_tracelog" accepted,
version 3.0 is compatible with LTTng UST provider version 3.0. (in
check_provider_version() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-probes.c:50)
liblttng_ust[42358/42358]: Provider "lttng_ust_tracelog" accepted,
version 3.0 is compatible with LTTng UST provider version 3.0. (in
check_provider_version() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-probes.c:50)
liblttng_ust[42358/42358]: Provider "lttng_ust_tracelog" accepted,
version 3.0 is compatible with LTTng UST provider version 3.0. (in
check_provider_version() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-probes.c:50)
liblttng_ust[42358/42358]: Provider "lttng_ust_tracelog" accepted,
version 3.0 is compatible with LTTng UST provider version 3.0. (in
check_provider_version() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-probes.c:50)
liblttng_ust[42358/42358]: Provider "lttng_ust_tracelog" accepted,
version 3.0 is compatible with LTTng UST provider version 3.0. (in
check_provider_version() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-probes.c:50)
liblttng_ust[42358/42358]: Provider "lttng_ust_tracelog" accepted,
version 3.0 is compatible with LTTng UST provider version 3.0. (in
check_provider_version() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-probes.c:50)
liblttng_ust[42358/42358]: Provider "lttng_ust_tracelog" accepted,
version 3.0 is compatible with LTTng UST provider version 3.0. (in
check_provider_version() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-probes.c:50)
liblttng_ust[42358/42358]: Provider "lttng_ust_tracelog" accepted,
version 3.0 is compatible with LTTng UST provider version 3.0. (in
check_provider_version() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-probes.c:50)
liblttng_ust[42358/42358]: Provider "lttng_ust_tracelog" accepted,
version 3.0 is compatible with LTTng UST provider version 3.0. (in
check_provider_version() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-probes.c:50)
liblttng_ust[42358/42359]: message successfully sent (in send_reply()
at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-ust-comm.c:656)
liblttng_ust[42358/42358]: adding probe lttng_ust_tracelog containing
15 events to lazy registration list (in lttng_ust_probe_register() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-probes.c:254)
liblttng_ust_tracepoint[42358/42358]: Release queue of unregistered
tracepoint probes. (in lttng_ust_tp_probe_prune_release_queue() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust-tracepoint/tracepoint.c:787)
Segmentation fault

Not sure exactly what is going on.  I did the hello world example
stand alone and it worked.  Thinking I messed up something in my trace
provider package I simply used the hello world one that worked and put
the tracepoint "lttng_ust_tracepoint(hello_world, my_first_tracepoint,
23,"hi there!");" in my app and it segfaults.

This is using lttng-tools 2.13.9 with patches (from OE recipe):
          file://0001-tests-do-not-strip-a-helper-library.patch \
          file://run-ptest \
          file://lttng-sessiond.service \
          file://disable-tests.patch \
          file://0001-compat-Define-off64_t-as-off_t-on-linux.patch


lttng-ust 2.13.5 with patches:
          file://0001-python-lttngust-Makefile.am-Add-install-lib-to-setup.patch
\
          file://0001-lttng-ust-common-link-with-liburcu-explicitly.patch \
          file://0001-Makefile.am-update-rpath-link.patch

And I manually added patch:

Fix: Reevaluate LTTNG_UST_TRACEPOINT_DEFINE each time tracepoint.h is included

lttng-modules 2.13.9 with patches:
          file://0009-Rename-genhd-wrapper-to-blkdev.patch \
          file://0001-fix-mm-introduce-vma-vm_flags-wrapper-functions-v6.3.patch
\
          file://0002-fix-uuid-Decouple-guid_t-and-uuid_le-types-and-respe.patch
\
          file://0003-fix-btrfs-pass-find_free_extent_ctl-to-allocator-tra.patch
\
          file://0004-fix-net-add-location-to-trace_consume_skb-v6.3.patch


Here is log of hello running right afterwards ... it does not seg fault:

root@localhost:/home/root# ./hello
liblttng_ust[61133/61133]: dlopened liblttng-ust shared library
(liblttng-ust.so.1). (in lttng_ust_ctor() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-ust-comm.c:2217)
liblttng_ust_tracepoint[61133/61133]: Your compiler treats weak
symbols with hidden visibility for integer objects as SAME address
between compile units part of the same module. (in check_weak_hidden()
at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust-tracepoint/tracepoint.c:10)
liblttng_ust_tracepoint[61133/61133]: Your compiler treats weak
symbols with hidden visibility for pointer objects as SAME address
between compile units part of the same module. (in check_weak_hidden()
at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust-tracepoint/tracepoint.c:10)
liblttng_ust_tracepoint[61133/61133]: Your compiler treats weak
symbols with hidden visibility for 24-byte structure objects as SAME
address between compile units part of the same module. (in
check_weak_hidden() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust-tracepoint/trace)
liblttng_ust_tracepoint[61133/61133]: just registered a tracepoints
section from 0xffffa61d9630 and having 26 tracepoints (in
lttng_ust_tracepoint_module_register() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust-tracepoint/tracepoint.c:960)
liblttng_ust_tracepoint[61133/61133]: registered tracepoint:
"lttng_ust_statedump:end" (in lttng_ust_tracepoint_module_register()
at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust-tracepoint/tracepoint.c:968)
liblttng_ust_tracepoint[61133/61133]: registered tracepoint:
"lttng_ust_statedump:procname" (in
lttng_ust_tracepoint_module_register() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust-tracepoint/tracepoint.c:968)
liblttng_ust_tracepoint[61133/61133]: registered tracepoint:
"lttng_ust_statedump:debug_link" (in
lttng_ust_tracepoint_module_register() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust-tracepoint/tracepoint.c:968)
liblttng_ust_tracepoint[61133/61133]: registered tracepoint:
"lttng_ust_statedump:build_id" (in
lttng_ust_tracepoint_module_register() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust-tracepoint/tracepoint.c:968)
liblttng_ust_tracepoint[61133/61133]: registered tracepoint:
"lttng_ust_statedump:bin_info" (in
lttng_ust_tracepoint_module_register() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust-tracepoint/tracepoint.c:968)
liblttng_ust_tracepoint[61133/61133]: registered tracepoint:
"lttng_ust_statedump:start" (in lttng_ust_tracepoint_module_register()
at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust-tracepoint/tracepoint.c:968)
liblttng_ust_tracepoint[61133/61133]: registered tracepoint:
"lttng_ust_lib:unload" (in lttng_ust_tracepoint_module_register() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust-tracepoint/tracepoint.c:968)
liblttng_ust_tracepoint[61133/61133]: registered tracepoint:
"lttng_ust_lib:debug_link" (in lttng_ust_tracepoint_module_register()
at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust-tracepoint/tracepoint.c:968)
liblttng_ust_tracepoint[61133/61133]: registered tracepoint:
"lttng_ust_lib:build_id" (in lttng_ust_tracepoint_module_register() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust-tracepoint/tracepoint.c:968)
liblttng_ust_tracepoint[61133/61133]: registered tracepoint:
"lttng_ust_lib:load" (in lttng_ust_tracepoint_module_register() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust-tracepoint/tracepoint.c:968)
liblttng_ust_tracepoint[61133/61133]: registered tracepoint:
"lttng_ust_tracef:event" (in lttng_ust_tracepoint_module_register() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust-tracepoint/tracepoint.c:968)
liblttng_ust_tracepoint[61133/61133]: registered tracepoint:
"lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_DEBUG" (in
lttng_ust_tracepoint_module_register() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust-tracepoint/tracepoint.c:968)
liblttng_ust_tracepoint[61133/61133]: registered tracepoint:
"lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_DEBUG_LINE" (in
lttng_ust_tracepoint_module_register() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust-tracepoint/tracepoint.c:968)
liblttng_ust_tracepoint[61133/61133]: registered tracepoint:
"lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_DEBUG_FUNCTION" (in
lttng_ust_tracepoint_module_register() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust-tracepoint/tracepoint.c:968)
liblttng_ust_tracepoint[61133/61133]: registered tracepoint:
"lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_DEBUG_UNIT" (in
lttng_ust_tracepoint_module_register() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust-tracepoint/tracepoint.c:968)
liblttng_ust_tracepoint[61133/61133]: registered tracepoint:
"lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_DEBUG_MODULE" (in
lttng_ust_tracepoint_module_register() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust-tracepoint/tracepoint.c:968)
liblttng_ust_tracepoint[61133/61133]: registered tracepoint:
"lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_DEBUG_PROCESS" (in
lttng_ust_tracepoint_module_register() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust-tracepoint/tracepoint.c:968)
liblttng_ust_tracepoint[61133/61133]: registered tracepoint:
"lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_DEBUG_PROGRAM" (in
lttng_ust_tracepoint_module_register() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust-tracepoint/tracepoint.c:968)
liblttng_ust_tracepoint[61133/61133]: registered tracepoint:
"lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_DEBUG_SYSTEM" (in
lttng_ust_tracepoint_module_register() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust-tracepoint/tracepoint.c:968)
liblttng_ust_tracepoint[61133/61133]: registered tracepoint:
"lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_INFO" (in
lttng_ust_tracepoint_module_register() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust-tracepoint/tracepoint.c:968)
liblttng_ust_tracepoint[61133/61133]: registered tracepoint:
"lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_NOTICE" (in
lttng_ust_tracepoint_module_register() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust-tracepoint/tracepoint.c:968)
liblttng_ust_tracepoint[61133/61133]: registered tracepoint:
"lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_WARNING" (in
lttng_ust_tracepoint_module_register() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust-tracepoint/tracepoint.c:968)
liblttng_ust_tracepoint[61133/61133]: registered tracepoint:
"lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_ERR" (in
lttng_ust_tracepoint_module_register() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust-tracepoint/tracepoint.c:968)
liblttng_ust_tracepoint[61133/61133]: registered tracepoint:
"lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_CRIT" (in
lttng_ust_tracepoint_module_register() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust-tracepoint/tracepoint.c:968)
liblttng_ust_tracepoint[61133/61133]: registered tracepoint:
"lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_ALERT" (in
lttng_ust_tracepoint_module_register() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust-tracepoint/tracepoint.c:968)
liblttng_ust_tracepoint[61133/61133]: registered tracepoint:
"lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_EMERG" (in
lttng_ust_tracepoint_module_register() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust-tracepoint/tracepoint.c:968)
liblttng_ust[61133/61133]: Provider "lttng_ust_statedump" accepted,
version 3.0 is compatible with LTTng UST provider version 3.0. (in
check_provider_version() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-probes.c:50)
liblttng_ust[61133/61133]: Provider "lttng_ust_statedump" accepted,
version 3.0 is compatible with LTTng UST provider version 3.0. (in
check_provider_version() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-probes.c:50)
liblttng_ust[61133/61133]: Provider "lttng_ust_statedump" accepted,
version 3.0 is compatible with LTTng UST provider version 3.0. (in
check_provider_version() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-probes.c:50)
liblttng_ust[61133/61133]: Provider "lttng_ust_statedump" accepted,
version 3.0 is compatible with LTTng UST provider version 3.0. (in
check_provider_version() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-probes.c:50)
liblttng_ust[61133/61133]: Provider "lttng_ust_statedump" accepted,
version 3.0 is compatible with LTTng UST provider version 3.0. (in
check_provider_version() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-probes.c:50)
liblttng_ust[61133/61133]: Provider "lttng_ust_statedump" accepted,
version 3.0 is compatible with LTTng UST provider version 3.0. (in
check_provider_version() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-probes.c:50)
liblttng_ust[61133/61133]: Provider "lttng_ust_statedump" accepted,
version 3.0 is compatible with LTTng UST provider version 3.0. (in
check_provider_version() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-probes.c:50)
liblttng_ust[61133/61133]: adding probe lttng_ust_statedump containing
6 events to lazy registration list (in lttng_ust_probe_register() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-probes.c:254)
libringbuffer-clients[61133/61133]: LTT : ltt ring buffer client
"relay-metadata-mmap" init
(in lttng_ring_buffer_metadata_client_init() at
../../../lttng-ust-2.13.5/src/common/ringbuffer-clients/metadata-template.h:364)
libringbuffer-clients[61133/61133]: LTT : ltt ring buffer client
"relay-overwrite-mmap" init
(in lttng_ring_buffer_client_overwrite_init() at
../../../lttng-ust-2.13.5/src/common/ringbuffer-clients/template.h:826)
libringbuffer-clients[61133/61133]: LTT : ltt ring buffer client
"relay-overwrite-rt-mmap" init
(in lttng_ring_buffer_client_overwrite_rt_init() at
../../../lttng-ust-2.13.5/src/common/ringbuffer-clients/template.h:826)
libringbuffer-clients[61133/61133]: LTT : ltt ring buffer client
"relay-discard-mmap" init
(in lttng_ring_buffer_client_discard_init() at
../../../lttng-ust-2.13.5/src/common/ringbuffer-clients/template.h:826)
libringbuffer-clients[61133/61133]: LTT : ltt ring buffer client
"relay-discard-rt-mmap" init
(in lttng_ring_buffer_client_discard_rt_init() at
../../../lttng-ust-2.13.5/src/common/ringbuffer-clients/template.h:826)
liblttng_ust[61133/61140]: Info: sessiond not accepting connections to
local apps socket (in ust_listener_thread() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-ust-comm.c:1884)
liblttng_ust[61133/61140]: Waiting for local apps sessiond (in
wait_for_sessiond() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-ust-comm.c:1758)
liblttng_ust[61133/61139]: Message Received "Get Tracer Version" (65),
Handle "root" (0) (in print_cmd() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-ust-comm.c:476)
liblttng_ust[61133/61139]: Return value: 0 (in handle_message() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-ust-comm.c:1412)
liblttng_ust[61133/61139]: message successfully sent (in send_reply()
at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-ust-comm.c:656)
liblttng_ust[61133/61139]: Message Received "Create event notifier
group" (70), Handle "root" (0) (in print_cmd() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-ust-comm.c:476)
liblttng_ust[61133/61139]: message successfully sent (in send_reply()
at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-ust-comm.c:656)
liblttng_ust[61133/61139]: Return value: 1 (in handle_message() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-ust-comm.c:1412)
liblttng_ust[61133/61139]: message successfully sent (in send_reply()
at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-ust-comm.c:656)
liblttng_ust[61133/61139]: Message Received "Create Counter" (192),
Handle "event_notifier_" (1) (in print_cmd() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-ust-comm.c:476)
liblttng_ust[61133/61139]: message successfully sent (in send_reply()
at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-ust-comm.c:656)
liblttng_ust[61133/61139]: counter data received (in handle_message()
at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-ust-comm.c:1266)
liblttng_ust[61133/61139]: Return value: 2 (in handle_message() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-ust-comm.c:1412)
liblttng_ust[61133/61139]: message successfully sent (in send_reply()
at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-ust-comm.c:656)
liblttng_ust[61133/61139]: Message Received "Create Counter CPU"
(209), Handle "event_notifier " (2) (in print_cmd() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-ust-comm.c:476)
liblttng_ust[61133/61139]: message successfully sent (in send_reply()
at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-ust-comm.c:656)
liblttng_ust[61133/61139]: Return value: 0 (in handle_message() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-ust-comm.c:1412)
liblttng_ust[61133/61139]: message successfully sent (in send_reply()
at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-ust-comm.c:656)
liblttng_ust[61133/61139]: Message Received "Create Counter CPU"
(209), Handle "event_notifier " (2) (in print_cmd() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-ust-comm.c:476)
liblttng_ust[61133/61139]: message successfully sent (in send_reply()
at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-ust-comm.c:656)
liblttng_ust[61133/61139]: Return value: 0 (in handle_message() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-ust-comm.c:1412)
liblttng_ust[61133/61139]: message successfully sent (in send_reply()
at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-ust-comm.c:656)
liblttng_ust[61133/61139]: Message Received "Create Counter CPU"
(209), Handle "event_notifier " (2) (in print_cmd() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-ust-comm.c:476)
liblttng_ust[61133/61139]: message successfully sent (in send_reply()
at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-ust-comm.c:656)
liblttng_ust[61133/61139]: Return value: 0 (in handle_message() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-ust-comm.c:1412)
liblttng_ust[61133/61139]: message successfully sent (in send_reply()
at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-ust-comm.c:656)
liblttng_ust[61133/61139]: Message Received "Create Counter CPU"
(209), Handle "event_notifier " (2) (in print_cmd() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-ust-comm.c:476)
liblttng_ust[61133/61139]: message successfully sent (in send_reply()
at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-ust-comm.c:656)
liblttng_ust[61133/61139]: Return value: 0 (in handle_message() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-ust-comm.c:1412)
liblttng_ust[61133/61139]: message successfully sent (in send_reply()
at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-ust-comm.c:656)
liblttng_ust[61133/61139]: Message Received "Registration Done" (68),
Handle "root" (0) (in print_cmd() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-ust-comm.c:476)
liblttng_ust[61133/61133]: Provider "lttng_ust_lib" accepted, version
3.0 is compatible with LTTng UST provider version 3.0. (in
check_provider_version() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-probes.c:50)
liblttng_ust[61133/61133]: Provider "lttng_ust_lib" accepted, version
3.0 is compatible with LTTng UST provider version 3.0. (in
check_provider_version() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-probes.c:50)
liblttng_ust[61133/61133]: Provider "lttng_ust_lib" accepted, version
3.0 is compatible with LTTng UST provider version 3.0. (in
check_provider_version() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-probes.c:50)
liblttng_ust[61133/61133]: Provider "lttng_ust_lib" accepted, version
3.0 is compatible with LTTng UST provider version 3.0. (in
check_provider_version() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-probes.c:50)
liblttng_ust[61133/61133]: Provider "lttng_ust_lib" accepted, version
3.0 is compatible with LTTng UST provider version 3.0. (in
check_provider_version() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-probes.c:50)
liblttng_ust[61133/61139]: Return value: 0 (in handle_message() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-ust-comm.c:1412)
liblttng_ust[61133/61133]: adding probe lttng_ust_lib containing 4
events to lazy registration list (in lttng_ust_probe_register() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-probes.c:254)
liblttng_ust_tracepoint[61133/61133]: Release queue of unregistered
tracepoint probes. (in lttng_ust_tp_probe_prune_release_queue() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust-tracepoint/tracepoint.c:787)
liblttng_ust[61133/61133]: Provider "lttng_ust_tracef" accepted,
version 3.0 is compatible with LTTng UST provider version 3.0. (in
check_provider_version() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-probes.c:50)
liblttng_ust[61133/61133]: Provider "lttng_ust_tracef" accepted,
version 3.0 is compatible with LTTng UST provider version 3.0. (in
check_provider_version() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-probes.c:50)
liblttng_ust[61133/61133]: adding probe lttng_ust_tracef containing 1
events to lazy registration list (in lttng_ust_probe_register() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-probes.c:254)
liblttng_ust_tracepoint[61133/61133]: Release queue of unregistered
tracepoint probes. (in lttng_ust_tp_probe_prune_release_queue() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust-tracepoint/tracepoint.c:787)
liblttng_ust[61133/61133]: Provider "lttng_ust_tracelog" accepted,
version 3.0 is compatible with LTTng UST provider version 3.0. (in
check_provider_version() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-probes.c:50)
liblttng_ust[61133/61133]: Provider "lttng_ust_tracelog" accepted,
version 3.0 is compatible with LTTng UST provider version 3.0. (in
check_provider_version() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-probes.c:50)
liblttng_ust[61133/61133]: Provider "lttng_ust_tracelog" accepted,
version 3.0 is compatible with LTTng UST provider version 3.0. (in
check_provider_version() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-probes.c:50)
liblttng_ust[61133/61133]: Provider "lttng_ust_tracelog" accepted,
version 3.0 is compatible with LTTng UST provider version 3.0. (in
check_provider_version() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-probes.c:50)
liblttng_ust[61133/61133]: Provider "lttng_ust_tracelog" accepted,
version 3.0 is compatible with LTTng UST provider version 3.0. (in
check_provider_version() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-probes.c:50)
liblttng_ust[61133/61133]: Provider "lttng_ust_tracelog" accepted,
version 3.0 is compatible with LTTng UST provider version 3.0. (in
check_provider_version() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-probes.c:50)
liblttng_ust[61133/61133]: Provider "lttng_ust_tracelog" accepted,
version 3.0 is compatible with LTTng UST provider version 3.0. (in
check_provider_version() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-probes.c:50)
liblttng_ust[61133/61133]: Provider "lttng_ust_tracelog" accepted,
version 3.0 is compatible with LTTng UST provider version 3.0. (in
check_provider_version() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-probes.c:50)
liblttng_ust[61133/61133]: Provider "lttng_ust_tracelog" accepted,
version 3.0 is compatible with LTTng UST provider version 3.0. (in
check_provider_version() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-probes.c:50)
liblttng_ust[61133/61133]: Provider "lttng_ust_tracelog" accepted,
version 3.0 is compatible with LTTng UST provider version 3.0. (in
check_provider_version() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-probes.c:50)
liblttng_ust[61133/61133]: Provider "lttng_ust_tracelog" accepted,
version 3.0 is compatible with LTTng UST provider version 3.0. (in
check_provider_version() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-probes.c:50)
liblttng_ust[61133/61133]: Provider "lttng_ust_tracelog" accepted,
version 3.0 is compatible with LTTng UST provider version 3.0. (in
check_provider_version() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-probes.c:50)
liblttng_ust[61133/61133]: Provider "lttng_ust_tracelog" accepted,
version 3.0 is compatible with LTTng UST provider version 3.0. (in
check_provider_version() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-probes.c:50)
liblttng_ust[61133/61133]: Provider "lttng_ust_tracelog" accepted,
version 3.0 is compatible with LTTng UST provider version 3.0. (in
check_provider_version() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-probes.c:50)
liblttng_ust[61133/61133]: Provider "lttng_ust_tracelog" accepted,
version 3.0 is compatible with LTTng UST provider version 3.0. (in
check_provider_version() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-probes.c:50)
liblttng_ust[61133/61133]: Provider "lttng_ust_tracelog" accepted,
version 3.0 is compatible with LTTng UST provider version 3.0. (in
check_provider_version() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-probes.c:50)
liblttng_ust[61133/61139]: message successfully sent (in send_reply()
at ../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-ust-comm.c:656)
liblttng_ust[61133/61133]: adding probe lttng_ust_tracelog containing
15 events to lazy registration list (in lttng_ust_probe_register() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-probes.c:254)
liblttng_ust_tracepoint[61133/61133]: Release queue of unregistered
tracepoint probes. (in lttng_ust_tp_probe_prune_release_queue() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust-tracepoint/tracepoint.c:787)
liblttng_ust_tracepoint[61133/61133]: just registered a tracepoints
section from 0xaaaaae9c41b8 and having 1 tracepoints (in
lttng_ust_tracepoint_module_register() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust-tracepoint/tracepoint.c:960)
liblttng_ust_tracepoint[61133/61133]: registered tracepoint:
"hello_world:my_first_tracepoint" (in
lttng_ust_tracepoint_module_register() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust-tracepoint/tracepoint.c:968)
liblttng_ust[61133/61133]: Provider "hello_world" accepted, version
3.0 is compatible with LTTng UST provider version 3.0. (in
check_provider_version() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-probes.c:50)
liblttng_ust[61133/61133]: Provider "hello_world" accepted, version
3.0 is compatible with LTTng UST provider version 3.0. (in
check_provider_version() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-probes.c:50)
liblttng_ust[61133/61133]: adding probe hello_world containing 1
events to lazy registration list (in lttng_ust_probe_register() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-probes.c:254)
liblttng_ust_tracepoint[61133/61133]: Release queue of unregistered
tracepoint probes. (in lttng_ust_tp_probe_prune_release_queue() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust-tracepoint/tracepoint.c:787)
Hello, World!
Press Enter to continue...

Quitting now!
liblttng_ust[61133/61133]: Provider "hello_world" accepted, version
3.0 is compatible with LTTng UST provider version 3.0. (in
check_provider_version() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-probes.c:50)
liblttng_ust_tracepoint[61133/61133]: Release queue of unregistered
tracepoint probes. (in lttng_ust_tp_probe_prune_release_queue() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust-tracepoint/tracepoint.c:787)
liblttng_ust[61133/61133]: just unregistered probes of provider
hello_world (in lttng_ust_probe_unregister() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-probes.c:286)
liblttng_ust_tracepoint[61133/61133]: just unregistered a tracepoints
section from 0xaaaaae9c41b8 (in
lttng_ust_tracepoint_module_unregister() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust-tracepoint/tracepoint.c:995)
liblttng_ust[61133/61133]: Provider "lttng_ust_tracelog" accepted,
version 3.0 is compatible with LTTng UST provider version 3.0. (in
check_provider_version() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-probes.c:50)
liblttng_ust_tracepoint[61133/61133]: Release queue of unregistered
tracepoint probes. (in lttng_ust_tp_probe_prune_release_queue() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust-tracepoint/tracepoint.c:787)
liblttng_ust[61133/61133]: just unregistered probes of provider
lttng_ust_tracelog (in lttng_ust_probe_unregister() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-probes.c:286)
liblttng_ust[61133/61133]: Provider "lttng_ust_tracef" accepted,
version 3.0 is compatible with LTTng UST provider version 3.0. (in
check_provider_version() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-probes.c:50)
liblttng_ust_tracepoint[61133/61133]: Release queue of unregistered
tracepoint probes. (in lttng_ust_tp_probe_prune_release_queue() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust-tracepoint/tracepoint.c:787)
liblttng_ust[61133/61133]: just unregistered probes of provider
lttng_ust_tracef (in lttng_ust_probe_unregister() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-probes.c:286)
liblttng_ust[61133/61133]: Provider "lttng_ust_lib" accepted, version
3.0 is compatible with LTTng UST provider version 3.0. (in
check_provider_version() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-probes.c:50)
liblttng_ust_tracepoint[61133/61133]: Release queue of unregistered
tracepoint probes. (in lttng_ust_tp_probe_prune_release_queue() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust-tracepoint/tracepoint.c:787)
liblttng_ust[61133/61133]: just unregistered probes of provider
lttng_ust_lib (in lttng_ust_probe_unregister() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-probes.c:286)
libringbuffer-clients[61133/61133]: LTT : ltt ring buffer client
"relay-discard-rt-mmap" exit
(in lttng_ring_buffer_client_discard_rt_exit() at
../../../lttng-ust-2.13.5/src/common/ringbuffer-clients/template.h:833)
libringbuffer-clients[61133/61133]: LTT : ltt ring buffer client
"relay-discard-mmap" exit
(in lttng_ring_buffer_client_discard_exit() at
../../../lttng-ust-2.13.5/src/common/ringbuffer-clients/template.h:833)
libringbuffer-clients[61133/61133]: LTT : ltt ring buffer client
"relay-overwrite-rt-mmap" exit
(in lttng_ring_buffer_client_overwrite_rt_exit() at
../../../lttng-ust-2.13.5/src/common/ringbuffer-clients/template.h:833)
libringbuffer-clients[61133/61133]: LTT : ltt ring buffer client
"relay-overwrite-mmap" exit
(in lttng_ring_buffer_client_overwrite_exit() at
../../../lttng-ust-2.13.5/src/common/ringbuffer-clients/template.h:833)
libringbuffer-clients[61133/61133]: LTT : ltt ring buffer client
"relay-metadata-mmap" exit
(in lttng_ring_buffer_metadata_client_exit() at
../../../lttng-ust-2.13.5/src/common/ringbuffer-clients/metadata-template.h:371)
liblttng_ust[61133/61133]: Provider "lttng_ust_statedump" accepted,
version 3.0 is compatible with LTTng UST provider version 3.0. (in
check_provider_version() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-probes.c:50)
liblttng_ust_tracepoint[61133/61133]: Release queue of unregistered
tracepoint probes. (in lttng_ust_tp_probe_prune_release_queue() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust-tracepoint/tracepoint.c:787)
liblttng_ust[61133/61133]: just unregistered probes of provider
lttng_ust_statedump (in lttng_ust_probe_unregister() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust/lttng-probes.c:286)
liblttng_ust_tracepoint[61133/61133]: just unregistered a tracepoints
section from 0xffffa61d9630 (in
lttng_ust_tracepoint_module_unregister() at
../../../../lttng-ust-2.13.5/src/lib/lttng-ust-tracepoint/tracepoint.c:995)


Regards,

Brian
_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev