Mirror of the lttng-dev mailing list
 help / color / mirror / Atom feed
From: Brian Hutchinson via lttng-dev <lttng-dev@lists.lttng.org>
To: lttng-dev@lists.lttng.org
Subject: Re: [lttng-dev] I'm still getting empty ust traces using tracef
Date: Thu, 11 May 2023 14:57:34 -0400	[thread overview]
Message-ID: <CAFZh4h-kf4nWNG4L7T44=-470XVnO9dY4FV4Cwk46oGLUMWDhw@mail.gmail.com> (raw)
In-Reply-To: <CAFZh4h8awEpJgAcygRY=1f7A2qK91b31HBnLBPvvueuRnquOjQ@mail.gmail.com>

On Thu, May 11, 2023 at 12:36 PM Brian Hutchinson <b.hutchman@gmail.com> wrote:
>
> ... more background.  I've always used ltt in the kernel so I don't
> have much experience with the user side of it and especially
> multi-threaded, multi-core so I'm probably missing some fundamental
> concepts that I need to understand.
>
> Regards,
>
> Brian

I'm currently reading up on the new v2.13 "Recording event rule and
event record" features.  I came to 2.13 because I want to be able to
trigger snapshots on an event so I don't fill up storage on an
embedded device.

After I start lttng-systemd --daemonize and then start my app, I see
this when I do lttng list --userspace:

# lttng-sessiond --daemonize
[   85.180717] LTTng: Loaded modules v2.13.9 (Nordicit�é)
[   85.186103] LTTng: Experimental bitwise enum enabled.

# lttng list --userspace
UST events:
-------------

PID: 617 - Name: sh
     lttng_ust_libc:posix_memalign (loglevel: TRACE_DEBUG_LINE (13))
(type: tracepoint)
     lttng_ust_libc:memalign (loglevel: TRACE_DEBUG_LINE (13)) (type:
tracepoint)
     lttng_ust_libc:realloc (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint)
     lttng_ust_libc:calloc (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint)
     lttng_ust_libc:free (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint)
     lttng_ust_libc:malloc (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint)
     lttng_ust_pthread:pthread_mutex_unlock (loglevel:
TRACE_DEBUG_LINE (13)) (type: tracepoint)
     lttng_ust_pthread:pthread_mutex_trylock (loglevel:
TRACE_DEBUG_LINE (13)) (type: tracepoint)
     lttng_ust_pthread:pthread_mutex_lock_acq (loglevel:
TRACE_DEBUG_LINE (13)) (type: tracepoint)
     lttng_ust_pthread:pthread_mutex_lock_req (loglevel:
TRACE_DEBUG_LINE (13)) (type: tracepoint)
     lttng_ust_dl:dlclose (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint)
     lttng_ust_dl:debug_link (loglevel: TRACE_DEBUG_LINE (13)) (type:
tracepoint)
     lttng_ust_dl:build_id (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint)
     lttng_ust_dl:dlmopen (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint)
     lttng_ust_dl:dlopen (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint)
     lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_DEBUG (loglevel:
TRACE_DEBUG (14)) (type: tracepoint)
     lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_DEBUG_LINE
(loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint)
     lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_DEBUG_FUNCTION
(loglevel: TRACE_DEBUG_FUNCTION (12)) (type: tracepoint)
     lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_DEBUG_UNIT
(loglevel: TRACE_DEBUG_UNIT (11)) (type: tracepoint)
     lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_DEBUG_MODULE
(loglevel: TRACE_DEBUG_MODULE (10)) (type: tracepoint)
     lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_DEBUG_PROCESS
(loglevel: TRACE_DEBUG_PROCESS (9)) (type: tracepoint)
     lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_DEBUG_PROGRAM
(loglevel: TRACE_DEBUG_PROGRAM (8)) (type: tracepoint)
     lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_DEBUG_SYSTEM
(loglevel: TRACE_DEBUG_SYSTEM (7)) (type: tracepoint)
     lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_INFO (loglevel:
TRACE_INFO (6)) (type: tracepoint)
     lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_NOTICE
(loglevel: TRACE_NOTICE (5)) (type: tracepoint)
     lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_WARNING
(loglevel: TRACE_WARNING (4)) (type: tracepoint)
     lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_ERR (loglevel:
TRACE_ERR (3)) (type: tracepoint)
     lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_CRIT (loglevel:
TRACE_CRIT (2)) (type: tracepoint)
     lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_ALERT (loglevel:
TRACE_ALERT (1)) (type: tracepoint)
     lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_EMERG (loglevel:
TRACE_EMERG (0)) (type: tracepoint)
     lttng_ust_tracef:event (loglevel: TRACE_DEBUG (14)) (type: tracepoint)
     lttng_ust_lib:unload (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint)
     lttng_ust_lib:debug_link (loglevel: TRACE_DEBUG_LINE (13)) (type:
tracepoint)
     lttng_ust_lib:build_id (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint)
     lttng_ust_lib:load (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint)
     lttng_ust_statedump:end (loglevel: TRACE_DEBUG_LINE (13)) (type:
tracepoint)
     lttng_ust_statedump:procname (loglevel: TRACE_DEBUG_LINE (13))
(type: tracepoint)
     lttng_ust_statedump:debug_link (loglevel: TRACE_DEBUG_LINE (13))
(type: tracepoint)
     lttng_ust_statedump:build_id (loglevel: TRACE_DEBUG_LINE (13))
(type: tracepoint)
     lttng_ust_statedump:bin_info (loglevel: TRACE_DEBUG_LINE (13))
(type: tracepoint)
     lttng_ust_statedump:start (loglevel: TRACE_DEBUG_LINE (13))
(type: tracepoint)

PID: 601 - Name: /usr/loal/bin/my_app
     lttng_ust_libc:posix_memalign (loglevel: TRACE_DEBUG_LINE (13))
(type: tracepoint)
     lttng_ust_libc:memalign (loglevel: TRACE_DEBUG_LINE (13)) (type:
tracepoint)
     lttng_ust_libc:realloc (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint)
     lttng_ust_libc:calloc (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint)
     lttng_ust_libc:free (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint)
     lttng_ust_libc:malloc (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint)
     lttng_ust_pthread:pthread_mutex_unlock (loglevel:
TRACE_DEBUG_LINE (13)) (type: tracepoint)
     lttng_ust_pthread:pthread_mutex_trylock (loglevel:
TRACE_DEBUG_LINE (13)) (type: tracepoint)
     lttng_ust_pthread:pthread_mutex_lock_acq (loglevel:
TRACE_DEBUG_LINE (13)) (type: tracepoint)
     lttng_ust_pthread:pthread_mutex_lock_req (loglevel:
TRACE_DEBUG_LINE (13)) (type: tracepoint)
     lttng_ust_dl:dlclose (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint)
     lttng_ust_dl:debug_link (loglevel: TRACE_DEBUG_LINE (13)) (type:
tracepoint)
     lttng_ust_dl:build_id (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint)
     lttng_ust_dl:dlmopen (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint)
     lttng_ust_dl:dlopen (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint)
     lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_DEBUG (loglevel:
TRACE_DEBUG (14)) (type: tracepoint)
     lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_DEBUG_LINE
(loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint)
     lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_DEBUG_FUNCTION
(loglevel: TRACE_DEBUG_FUNCTION (12)) (type: tracepoint)
     lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_DEBUG_UNIT
(loglevel: TRACE_DEBUG_UNIT (11)) (type: tracepoint)
     lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_DEBUG_MODULE
(loglevel: TRACE_DEBUG_MODULE (10)) (type: tracepoint)
     lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_DEBUG_PROCESS
(loglevel: TRACE_DEBUG_PROCESS (9)) (type: tracepoint)
     lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_DEBUG_PROGRAM
(loglevel: TRACE_DEBUG_PROGRAM (8)) (type: tracepoint)
     lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_DEBUG_SYSTEM
(loglevel: TRACE_DEBUG_SYSTEM (7)) (type: tracepoint)
     lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_INFO (loglevel:
TRACE_INFO (6)) (type: tracepoint)
     lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_NOTICE
(loglevel: TRACE_NOTICE (5)) (type: tracepoint)
     lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_WARNING
(loglevel: TRACE_WARNING (4)) (type: tracepoint)
     lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_ERR (loglevel:
TRACE_ERR (3)) (type: tracepoint)
     lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_CRIT (loglevel:
TRACE_CRIT (2)) (type: tracepoint)
     lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_ALERT (loglevel:
TRACE_ALERT (1)) (type: tracepoint)
     lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_EMERG (loglevel:
TRACE_EMERG (0)) (type: tracepoint)
     lttng_ust_tracef:event (loglevel: TRACE_DEBUG (14)) (type: tracepoint)
     lttng_ust_lib:unload (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint)
     lttng_ust_lib:debug_link (loglevel: TRACE_DEBUG_LINE (13)) (type:
tracepoint)
     lttng_ust_lib:build_id (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint)
     lttng_ust_lib:load (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint)
     lttng_ust_statedump:end (loglevel: TRACE_DEBUG_LINE (13)) (type:
tracepoint)
     lttng_ust_statedump:procname (loglevel: TRACE_DEBUG_LINE (13))
(type: tracepoint)
     lttng_ust_statedump:debug_link (loglevel: TRACE_DEBUG_LINE (13))
(type: tracepoint)
     lttng_ust_statedump:build_id (loglevel: TRACE_DEBUG_LINE (13))
(type: tracepoint)
     lttng_ust_statedump:bin_info (loglevel: TRACE_DEBUG_LINE (13))
(type: tracepoint)
     lttng_ust_statedump:start (loglevel: TRACE_DEBUG_LINE (13))
(type: tracepoint)

PID: 626 - Name: xz
     lttng_ust_libc:posix_memalign (loglevel: TRACE_DEBUG_LINE (13))
(type: tracepoint)
     lttng_ust_libc:memalign (loglevel: TRACE_DEBUG_LINE (13)) (type:
tracepoint)
     lttng_ust_libc:realloc (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint)
     lttng_ust_libc:calloc (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint)
     lttng_ust_libc:free (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint)
     lttng_ust_libc:malloc (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint)
     lttng_ust_pthread:pthread_mutex_unlock (loglevel:
TRACE_DEBUG_LINE (13)) (type: tracepoint)
     lttng_ust_pthread:pthread_mutex_trylock (loglevel:
TRACE_DEBUG_LINE (13)) (type: tracepoint)
     lttng_ust_pthread:pthread_mutex_lock_acq (loglevel:
TRACE_DEBUG_LINE (13)) (type: tracepoint)
     lttng_ust_pthread:pthread_mutex_lock_req (loglevel:
TRACE_DEBUG_LINE (13)) (type: tracepoint)
     lttng_ust_dl:dlclose (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint)
     lttng_ust_dl:debug_link (loglevel: TRACE_DEBUG_LINE (13)) (type:
tracepoint)
     lttng_ust_dl:build_id (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint)
     lttng_ust_dl:dlmopen (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint)
     lttng_ust_dl:dlopen (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint)
     lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_DEBUG (loglevel:
TRACE_DEBUG (14)) (type: tracepoint)
     lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_DEBUG_LINE
(loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint)
     lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_DEBUG_FUNCTION
(loglevel: TRACE_DEBUG_FUNCTION (12)) (type: tracepoint)
     lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_DEBUG_UNIT
(loglevel: TRACE_DEBUG_UNIT (11)) (type: tracepoint)
     lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_DEBUG_MODULE
(loglevel: TRACE_DEBUG_MODULE (10)) (type: tracepoint)
     lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_DEBUG_PROCESS
(loglevel: TRACE_DEBUG_PROCESS (9)) (type: tracepoint)
     lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_DEBUG_PROGRAM
(loglevel: TRACE_DEBUG_PROGRAM (8)) (type: tracepoint)
     lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_DEBUG_SYSTEM
(loglevel: TRACE_DEBUG_SYSTEM (7)) (type: tracepoint)
     lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_INFO (loglevel:
TRACE_INFO (6)) (type: tracepoint)
     lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_NOTICE
(loglevel: TRACE_NOTICE (5)) (type: tracepoint)
     lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_WARNING
(loglevel: TRACE_WARNING (4)) (type: tracepoint)
     lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_ERR (loglevel:
TRACE_ERR (3)) (type: tracepoint)
     lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_CRIT (loglevel:
TRACE_CRIT (2)) (type: tracepoint)
     lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_ALERT (loglevel:
TRACE_ALERT (1)) (type: tracepoint)
     lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_EMERG (loglevel:
TRACE_EMERG (0)) (type: tracepoint)
     lttng_ust_tracef:event (loglevel: TRACE_DEBUG (14)) (type: tracepoint)
     lttng_ust_lib:unload (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint)
     lttng_ust_lib:debug_link (loglevel: TRACE_DEBUG_LINE (13)) (type:
tracepoint)
     lttng_ust_lib:build_id (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint)
     lttng_ust_lib:load (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint)
     lttng_ust_statedump:end (loglevel: TRACE_DEBUG_LINE (13)) (type:
tracepoint)
     lttng_ust_statedump:procname (loglevel: TRACE_DEBUG_LINE (13))
(type: tracepoint)
     lttng_ust_statedump:debug_link (loglevel: TRACE_DEBUG_LINE (13))
(type: tracepoint)
     lttng_ust_statedump:build_id (loglevel: TRACE_DEBUG_LINE (13))
(type: tracepoint)
     lttng_ust_statedump:bin_info (loglevel: TRACE_DEBUG_LINE (13))
(type: tracepoint)
     lttng_ust_statedump:start (loglevel: TRACE_DEBUG_LINE (13))
(type: tracepoint)

I'm not sure why all this showed up 3 times.  Could be because I have
systemd unit file ExecStart calling a script so that my LD_PRELOAD=
line can work.  ExecStart= didn't like LD_PRELOAD.

So now I'm guessing ... I haven't read the fine documentation far
enough yet, that I have to somehow connect my record event rules to
PID 601 somehow and I'm getting empty trace records cause I'm not
smart enough (yet) to figure out how to do that.

Hopefully I'm getting "warmer".

Regards,

Brian
_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

      parent reply	other threads:[~2023-05-11 18:58 UTC|newest]

Thread overview: 15+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2023-05-11 15:53 Brian Hutchinson via lttng-dev
2023-05-11 16:36 ` Brian Hutchinson via lttng-dev
2023-05-11 18:13   ` Mathieu Desnoyers via lttng-dev
2023-05-11 18:15     ` Mathieu Desnoyers via lttng-dev
     [not found]       ` <CAFZh4h9M0eA1SpHhfquoyrqtwRhHV54f0O4hCkkiojwUmPTC9Q@mail.gmail.com>
     [not found]         ` <fdbcfbd9-742d-dc61-3667-48a00130d6aa@efficios.com>
2023-05-12 13:34           ` Mathieu Desnoyers via lttng-dev
2023-05-12 14:52           ` Brian Hutchinson via lttng-dev
2023-05-12 14:57             ` Mathieu Desnoyers via lttng-dev
2023-06-20 22:02       ` Brian Hutchinson via lttng-dev
2023-06-21 20:21         ` Mathieu Desnoyers via lttng-dev
2023-06-21 22:02           ` Brian Hutchinson via lttng-dev
2023-06-28 22:02             ` Brian Hutchinson via lttng-dev
2023-05-11 18:38     ` Brian Hutchinson via lttng-dev
2023-05-11 18:14   ` Michael Jeanson via lttng-dev
2023-05-11 18:42     ` Brian Hutchinson via lttng-dev
2023-05-11 18:57   ` Brian Hutchinson via lttng-dev [this message]

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to='CAFZh4h-kf4nWNG4L7T44=-470XVnO9dY4FV4Cwk46oGLUMWDhw@mail.gmail.com' \
    --to=lttng-dev@lists.lttng.org \
    --cc=b.hutchman@gmail.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body. 
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox