From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from simark.ca by simark.ca with LMTP id OOE0GyHIR2Sl9DUAWB0awg (envelope-from ) for ; Tue, 25 Apr 2023 08:31:29 -0400 Received: by simark.ca (Postfix, from userid 112) id 642781E221; Tue, 25 Apr 2023 08:31:29 -0400 (EDT) Authentication-Results: simark.ca; dkim=pass (2048-bit key; unprotected) header.d=lists.lttng.org header.i=@lists.lttng.org header.a=rsa-sha256 header.s=default header.b=QDYBp/MP; dkim-atps=neutral X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on simark.ca X-Spam-Level: X-Spam-Status: No, score=-6.9 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HTML_MESSAGE,MAILING_LIST_MULTI, MIME_HTML_MOSTLY,MISSING_HEADERS,RCVD_IN_DNSWL_HI,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6 Received: from lists.lttng.org (lists.lttng.org [167.114.26.123]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by simark.ca (Postfix) with ESMTPS id 44C4B1E128 for ; Tue, 25 Apr 2023 08:31:28 -0400 (EDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=lists.lttng.org; s=default; t=1682425887; bh=6EzfFMgVea4aZRTgt3GY8pk+IX87JxUD0tdO6UcFNqQ=; h=References:In-Reply-To:Date:Cc:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To: From; b=QDYBp/MPtr7kas/eX/KjGpZyFqfCGfFEG5CQEHARRt7Tu4x6V6pu9yvRlN1bc8yqo dAepNHcaS6jrzGTqw7p7rpvAfRyNTOC6yZQdwHIYcSzoEm2AS/1DESq4+qmhcujhBW +qKyA4l5Vx3AH0omYVmeojvOY9E6VJUFKbZkQhX5/J9FT1ospnVhQ8Tb9dU3JladY8 h5KU1ieg2YCkfk5IBdm9U7rswIFo0rJw/jpbV4DH09kL+7TX1pKTK2V4XJXkWQfzXt KgfVjOj93inWh7Nui97Wz89YlPm0yNt02HXkKpBWL13wtIS4oJB/hdO+MAynL8rCAB /rnao0hWiEhFg== Received: from lists-lttng01.efficios.com (localhost [IPv6:::1]) by lists.lttng.org (Postfix) with ESMTP id 4Q5Lvz51MnzGpK; Tue, 25 Apr 2023 08:31:27 -0400 (EDT) Received: from mail-ed1-f48.google.com (mail-ed1-f48.google.com [209.85.208.48]) by lists.lttng.org (Postfix) with ESMTPS id 4Q5Lvz22fXzGpJ for ; Tue, 25 Apr 2023 08:31:27 -0400 (EDT) Received: by mail-ed1-f48.google.com with SMTP id 4fb4d7f45d1cf-504efe702d5so8451627a12.3 for ; Tue, 25 Apr 2023 05:31:27 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1682425825; x=1685017825; h=cc:subject:message-id:date:from:in-reply-to:references:mime-version :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=TsN1XXtsUnGh+G55Vc8bmKND4jG9B0P/6zUBJbLj480=; b=MVfe4NbPWiI0zHIa+jF1APJg2/pwEYpWIgmAX/ss32m7oO9OE32Aa6CqaucVcs31IK 3JTIW0oZpHzuDIaHYztmt2JYe1gztfQQDJRp4vjyxrpjFZsWTBoODjXM5M1AgGRQkpnM v5M2ev7+2Ck/12tNtUad6z3vQ1+yIM1ynCq6nRwZf5mytR9ioIVm87M5IHaTBzlIhxd6 RRyx8x/GM6yp9BufZVEPbGz2if8IY7Plcq9WZmzfgl/j4woMlLdTvU1ItYLduRJ8uKBI E/Oevm6cRfhc/AzYYbT6kDoWeJCgRql6wtDzorA4dCaHC7W4vvMs6zoAEcqpbTx9cnGq C9qA== X-Gm-Message-State: AAQBX9fyffcDSaQ2rOVQ4s82FQSjBBhe3Sb098Y7Xbe672W9fUHQ3r4e ipZzg5GeAW+Mw0WoUy1jsL4kmE84qU2tJv9Sbc/N07MFJS4= X-Google-Smtp-Source: AKy350aOCUDfr9JbjQT0vgbd8S40NRmfKwRi13n/xC6ENQQP3Q2iEuVXdiCxigscM0KFKuR/bQUXhRVXv16qBCUaMfk= X-Received: by 2002:aa7:ce06:0:b0:507:4612:12df with SMTP id d6-20020aa7ce06000000b00507461212dfmr15783519edv.14.1682425825333; Tue, 25 Apr 2023 05:30:25 -0700 (PDT) MIME-Version: 1.0 References: <9b162e6a547c41b284327034a9749027@amazon.com> In-Reply-To: Date: Tue, 25 Apr 2023 08:30:14 -0400 Message-ID: Cc: "lttng-dev@lists.lttng.org" Subject: Re: [lttng-dev] Using lttng 2.11 and UST doesn't appear to work - getting empty trace files X-BeenThere: lttng-dev@lists.lttng.org X-Mailman-Version: 2.1.39 Precedence: list List-Id: LTTng development list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , From: Brian Hutchinson via lttng-dev Reply-To: Brian Hutchinson Content-Type: multipart/mixed; boundary="===============7013215603572134866==" Errors-To: lttng-dev-bounces@lists.lttng.org Sender: "lttng-dev" --===============7013215603572134866== Content-Type: multipart/alternative; boundary="000000000000ae172d05fa284932" --000000000000ae172d05fa284932 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable So looking at my lttng-sessiond log I found some concerning issues of things being "not found": DEBUG1 - 10:53:52.130237000 [9674/9735]: Enable event command for event 'lttng_ust_tracef:*' (in _cmd_enable_event() at ../../../../lttng-tools-2.11.5/src/bin/lttng-sessiond/cmd.c:2036) DEBUG2 - 10:53:52.130399500 [9674/9735]: Trace UST channel channel0 not found by name (in trace_ust_find_channel_by_name() at ../../../../lttng-tools-2.11.5/src/bin/lttng-sessiond/trace-ust.c:197) DEBUG1 - 10:53:52.130450125 [9674/9735]: Enabling channel for session auto-20210324-105310 (in cmd_enable_channel() at ../../../../lttng-tools-2.11.5/src/bin/lttng-sessiond/cmd.c:1510) DEBUG2 - 10:53:52.130496375 [9674/9735]: Trace UST channel channel0 not found by name (in trace_ust_find_channel_by_name() at ../../../../lttng-tools-2.11.5/src/bin/lttng-sessiond/trace-ust.c:197) DEBUG3 - 10:53:52.130572000 [9674/9735]: Created hashtable size 4 at 0xffff90017170 of type 0 (in lttng_ht_new() at ../../../../lttng-tools-2.11.5/src/common/hashtable/hashtable.c:145) DEBUG3 - 10:53:52.130619625 [9674/9735]: Created hashtable size 4 at 0xffff900175b0 of type 1 (in lttng_ht_new() at ../../../../lttng-tools-2.11.5/src/common/hashtable/hashtable.c:145) DEBUG2 - 10:53:52.130666000 [9674/9735]: Trace UST channel channel0 created (in trace_ust_create_channel() at ../../../../lttng-tools-2.11.5/src/bin/lttng-sessiond/trace-ust.c:396) DEBUG2 - 10:53:52.130713625 [9674/9735]: Channel channel0 is being created for UST with buffer 1 and id 0 (in channel_ust_create() at ../../../../lttng-tools-2.11.5/src/bin/lttng-sessiond/channel.c:461) DEBUG2 - 10:53:52.130762250 [9674/9735]: Channel channel0 created successfully (in channel_ust_create() at ../../../../lttng-tools-2.11.5/src/bin/lttng-sessiond/channel.c:500) DEBUG2 - 10:53:52.130807500 [9674/9735]: Trace UST channel channel0 found by name (in trace_ust_find_channel_by_name() at ../../../../lttng-tools-2.11.5/src/bin/lttng-sessiond/trace-ust.c:192) DEBUG2 - 10:53:52.130851750 [9674/9735]: Trace UST event lttng_ust_tracef:* NOT found (in trace_ust_find_event() at ../../../../lttng-tools-2.11.5/src/bin/lttng-sessiond/trace-ust.c:235) DEBUG2 - 10:53:52.130896500 [9674/9735]: Trace UST event lttng_ust_tracef:*, loglevel (0,-1) created (in trace_ust_create_event() at ../../../../lttng-tools-2.11.5/src/bin/lttng-sessiond/trace-ust.c:517) DEBUG1 - 10:53:52.130968375 [9674/9735]: Sending response (size: 20, retcode: Success (10)) (in thread_manage_clients() at ../../../../lttng-tools-2.11.5/src/bin/lttng-sessiond/client.c:2236) DEBUG1 - 10:53:52.134962250 [9674/9735]: Clean command context structure (in clean_command_ctx() at ../../../../lttng-tools-2.11.5/src/bin/lttng-sessiond/client.c:659) Not quite sure what to do about that. Regards, Brian On Tue, Apr 25, 2023 at 12:56=E2=80=AFAM Brian Hutchinson wrote: > So instructions for tracef just say to link in lttng-ust, not dl, I didn'= t > use liblttng-ust.a so maybe I misspoke. > > https://lttng.org/docs/v2.11/#doc-tracef > > ... so that's what I did. > > When I compile in a tracepoint with trace provider, I link in lttng-ust > and dl. I'm assuming you can mix both tracef and tracepoint calls? > > I included a lttng-sessiond -vvv log in original post. Are attachments > allowed or should I have posted the context as a plain text post? It's > quite big. > > When I do lttng start, my app is already running via systemd at startup. > Do I need to disable that and start the executable manually after enablin= g > the lttng ust events? > > Like I said, kernel trace is working fine, it's UST that appears to not b= e > working for some reason. I have less experience with UST but I've been > researching and searching archives and I see others complaining about wha= t > looks to be my problem but either don't see a solution or maybe I just > don't understand it. > > Thanks, > > Brian > > On Tue, Apr 25, 2023 at 12:13=E2=80=AFAM Yitschak, Yehuda > wrote: > >> It might be related to the static linking of lttng-ust >> https://lists.lttng.org/pipermail/lttng-dev/2018-January/027705.html >> >> I also went down that path just to figure it=E2=80=99s not supported. >> >> Perhaps lttng documentation should be updated to make a clear note stati= c >> linking is not supported (or maybe I missed it) >> >> Yehuda >> >> Hi, >> >> >> >> I'm on an embedded platform, i.MX8 using Lttng 2.11 with 5.6.69 kernel. >> >> >> >> Tracing kernel events with lttng works fine but userspace (tracef and >> trace provider type tracing) appears to have an issue I can't figure out= . >> >> >> >> systemd is starting up my application ( a bunch of processes and threads >> on multiple cpu's) on boot. >> >> >> >> I'm statically compiling and linking in lttng-ust. If I use just tracef >> type tracing, when I do lttng list --userspace I get nothing. If I add = in >> a trace provider and a tracepoint, then I see my trace provider tracepoi= nt >> and: >> >> >> >> lttng_ust_tracelog:TRACE_DEBUG (loglevel: TRACE_DEBUG (14)) (type: >> tracepoint) >> lttng_ust_tracelog:TRACE_DEBUG_LINE (loglevel: TRACE_DEBUG_LINE >> (13)) (type: tracepoint) >> lttng_ust_tracelog:TRACE_DEBUG_FUNCTION (loglevel: >> TRACE_DEBUG_FUNCTION (12)) (type: tracepoint) >> lttng_ust_tracelog:TRACE_DEBUG_UNIT (loglevel: TRACE_DEBUG_UNIT >> (11)) (type: tracepoint) >> lttng_ust_tracelog:TRACE_DEBUG_MODULE (loglevel: TRACE_DEBUG_MODULE >> (10)) (type: tracepoint) >> lttng_ust_tracelog:TRACE_DEBUG_PROCESS (loglevel: >> TRACE_DEBUG_PROCESS (9)) (type: tracepoint) >> lttng_ust_tracelog:TRACE_DEBUG_PROGRAM (loglevel: >> TRACE_DEBUG_PROGRAM (8)) (type: tracepoint) >> lttng_ust_tracelog:TRACE_DEBUG_SYSTEM (loglevel: TRACE_DEBUG_SYSTEM >> (7)) (type: tracepoint) >> lttng_ust_tracelog:TRACE_INFO (loglevel: TRACE_INFO (6)) (type: >> tracepoint) >> lttng_ust_tracelog:TRACE_NOTICE (loglevel: TRACE_NOTICE (5)) (type: >> tracepoint) >> lttng_ust_tracelog:TRACE_WARNING (loglevel: TRACE_WARNING (4)) >> (type: tracepoint) >> lttng_ust_tracelog:TRACE_ERR (loglevel: TRACE_ERR (3)) (type: >> tracepoint) >> lttng_ust_tracelog:TRACE_CRIT (loglevel: TRACE_CRIT (2)) (type: >> tracepoint) >> lttng_ust_tracelog:TRACE_ALERT (loglevel: TRACE_ALERT (1)) (type: >> tracepoint) >> lttng_ust_tracelog:TRACE_EMERG (loglevel: TRACE_EMERG (0)) (type: >> tracepoint) >> lttng_ust_tracef:event (loglevel: TRACE_DEBUG (14)) (type: >> tracepoint) >> lttng_ust_lib:unload (loglevel: TRACE_DEBUG_LINE (13)) (type: >> tracepoint) >> lttng_ust_lib:debug_link (loglevel: TRACE_DEBUG_LINE (13)) (type: >> tracepoint) >> lttng_ust_lib:build_id (loglevel: TRACE_DEBUG_LINE (13)) (type: >> tracepoint) >> lttng_ust_lib:load (loglevel: TRACE_DEBUG_LINE (13)) (type: >> tracepoint) >> lttng_ust_statedump:end (loglevel: TRACE_DEBUG_LINE (13)) (type: >> tracepoint) >> lttng_ust_statedump:debug_link (loglevel: TRACE_DEBUG_LINE (13)) >> (type: tracepoint) >> lttng_ust_statedump:build_id (loglevel: TRACE_DEBUG_LINE (13)) >> (type: tracepoint) >> lttng_ust_statedump:bin_info (loglevel: TRACE_DEBUG_LINE (13)) >> (type: tracepoint) >> lttng_ust_statedump:start (loglevel: TRACE_DEBUG_LINE (13)) (type: >> tracepoint) >> >> >> >> When I do lttng start, I get a directory structure in the trace director= y >> but the file sizes do not ever increase. It's just a directory structur= e >> with files created and then that's it, no events being written to the >> files. At first I thought things just weren't linked in right and not >> working but then discovered how to run lttng-sessiond in -vvv mode and t= hen >> could see lots of things are going on, tracing appears to be trying to w= ork >> as there is tons of activity being output by lttng-sessiond, but like I >> said the trace files aren't being added to. >> >> >> >> I looked at the lttng-sessiond log I captured and can't quite make out >> the problem so attaching it. >> >> >> >> I'm probably not doing something right but can't figure out what that is >> so any pointers appreciated. >> >> >> >> Regards, >> >> >> >> Brian >> >> >> > --000000000000ae172d05fa284932 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
So looking at my lttng-sessiond log I found some conc= erning issues of things being "not found":

DEBUG1 - 10:53:52.130237000 [9674/9735]: E= nable event command for event 'lttng_ust_tracef:*' (in _cmd_enable_= event() at ../../../../lttng-tools-2.11.5/src/bin/lttng-sessiond/cmd.c:2036= ) =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
DEBUG2 - 10:53:52.130399500 [9674/9735]: Trace UST channel channel0 not= found by name (in trace_ust_find_channel_by_name() at ../../../../lttng-to= ols-2.11.5/src/bin/lttng-sessiond/trace-ust.c:197) =C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0
DEBUG1 - 10:53:52.130450125 [9674/9735]: Enabling cha= nnel =C2=A0for session auto-20210324-105310 (in cmd_enable_channel() at ../= ../../../lttng-tools-2.11.5/src/bin/lttng-sessiond/cmd.c:1510) =C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0
DEBUG2 - 10:53:52.130496375 [9674/9735]: = Trace UST channel channel0 not found by name (in trace_ust_find_channel_by_= name() at ../../../../lttng-tools-2.11.5/src/bin/lttng-sessiond/trace-ust.c= :197) =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0
DEBUG3 - 10:53:52.13057200= 0 [9674/9735]: Created hashtable size 4 at 0xffff90017170 of type 0 (in ltt= ng_ht_new() at ../../../../lttng-tools-2.11.5/src/common/hashtable/hashtabl= e.c:145) =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0
DEBUG3 - 10:53:52= .130619625 [9674/9735]: Created hashtable size 4 at 0xffff900175b0 of type = 1 (in lttng_ht_new() at ../../../../lttng-tools-2.11.5/src/common/hashtable= /hashtable.c:145) =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0
DEBUG2 - 10= :53:52.130666000 [9674/9735]: Trace UST channel channel0 created (in trace_= ust_create_channel() at ../../../../lttng-tools-2.11.5/src/bin/lttng-sessio= nd/trace-ust.c:396) =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0
DEBUG= 2 - 10:53:52.130713625 [9674/9735]: Channel channel0 is being created for U= ST with buffer 1 and id 0 (in channel_ust_create() at ../../../../lttng-too= ls-2.11.5/src/bin/lttng-sessiond/channel.c:461) =C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0
DEBUG2 - 10:53:52.130762250 [9674/9735= ]: Channel channel0 created successfully (in channel_ust_create() at ../../= ../../lttng-tools-2.11.5/src/bin/lttng-sessiond/channel.c:500) =C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0
DEBUG2 - 10= :53:52.130807500 [9674/9735]: Trace UST channel channel0 found by name (in = trace_ust_find_channel_by_name() at ../../../../lttng-tools-2.11.5/src/bin/= lttng-sessiond/trace-ust.c:192) =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0
DEBUG2 - 10:53:52.130851750 [9674/9735]: Trace UST event lt= tng_ust_tracef:* NOT found (in trace_ust_find_event() at ../../../../lttng-= tools-2.11.5/src/bin/lttng-sessiond/trace-ust.c:235) =C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0
DEBUG2 - 10:53:52.130896500 [9674/9735]: Trace UST event lt= tng_ust_tracef:*, loglevel (0,-1) created (in trace_ust_create_event() at .= ./../../../lttng-tools-2.11.5/src/bin/lttng-sessiond/trace-ust.c:517) =C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0
DEBUG1 - 10:53:52.130968375 [9674/9735]: Sending response (size: 20,= retcode: Success (10)) (in thread_manage_clients() at ../../../../lttng-to= ols-2.11.5/src/bin/lttng-sessiond/client.c:2236) =C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0
DEBUG1 - 10:53:52.134962250 [9674/9735]: Clean comman= d context structure (in clean_command_ctx() at ../../../../lttng-tools-2.11= .5/src/bin/lttng-sessiond/client.c:659)

Not quite sure what to do about that.

Regards,

Brian

On Tue, Apr 25, 2023 at 12:56=E2=80=AFAM Brian Hutchinson <b.hutchman@gmail.com> wrote:
<= /div>
So instructions for tracef just say to link in lttng-ust= , not dl, I didn't use liblttng-ust.a so maybe I misspoke.

<= div>... so that's what I did.

When I compile i= n a tracepoint with trace provider, I link in lttng-ust and dl.=C2=A0 I'= ;m assuming you can mix both tracef and tracepoint calls?

I included a lttng-sessiond -vvv log in original post.=C2=A0 Are at= tachments allowed or should I have posted the context as a plain text post?= =C2=A0 It's quite big.

When I do lttng start, = my app is already running via systemd at startup.=C2=A0 Do I need to disabl= e that and start the executable manually after enabling the lttng ust event= s?

Like I said, kernel trace is working fine, it&#= 39;s UST that appears to not be working for some reason.=C2=A0 I have less = experience with UST but I've been researching and searching archives an= d I see others complaining about what looks to be my problem but either don= 't see a solution or maybe I just don't understand it.

Thanks,

Brian

=
On Tue, Ap= r 25, 2023 at 12:13=E2=80=AFAM Yitschak, Yehuda <yehuday@amazon.com> wrote:

It might be related to the static linking of lttng-u= st
https://lists.lttng.org/pipermail/lttng-dev/2018-Ja= nuary/027705.html

I also went down that path just to figure it=E2=80= =99s not supported.

Perhaps lttng documentation should be updated to mak= e a clear note static linking is not supported (or maybe I missed it)

Yehuda

Hi,

=C2=A0

I'm on an embedded platform, i.MX8 using Lttng 2= .11 with 5.6.69 kernel.

=C2=A0

Tracing kernel events with lttng works fine but user= space (tracef and trace provider type tracing) appears to have an issue I c= an't figure out.

=C2=A0

systemd is starting up my application ( a bunch of p= rocesses and threads on multiple cpu's) on boot.

=C2=A0

I'm statically compiling and linking in lttng-us= t.=C2=A0 If I use just tracef type tracing, when I do lttng list --userspac= e I get nothing.=C2=A0 If I add in a trace provider and a tracepoint, then = I see my trace provider tracepoint and:

=C2=A0

=C2=A0=C2=A0=C2=A0=C2=A0 lttng_ust_tracelog:TR= ACE_DEBUG (loglevel: TRACE_DEBUG (14)) (type: tracepoint)
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0lttng_ust_tracelog:TRACE_DEBUG_LINE (loglevel= : TRACE_DEBUG_LINE (13)) (type: tracepoint)
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0lttng_ust_tracelog:TRACE_DEBUG_FUNCTION (logl= evel: TRACE_DEBUG_FUNCTION (12)) (type: tracepoint)
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0lttng_ust_tracelog:TRACE_DEBUG_UNIT (loglevel= : TRACE_DEBUG_UNIT (11)) (type: tracepoint)
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0lttng_ust_tracelog:TRACE_DEBUG_MODULE (loglev= el: TRACE_DEBUG_MODULE (10)) (type: tracepoint)
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0lttng_ust_tracelog:TRACE_DEBUG_PROCESS (logle= vel: TRACE_DEBUG_PROCESS (9)) (type: tracepoint)
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0lttng_ust_tracelog:TRACE_DEBUG_PROGRAM (logle= vel: TRACE_DEBUG_PROGRAM (8)) (type: tracepoint)
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0lttng_ust_tracelog:TRACE_DEBUG_SYSTEM (loglev= el: TRACE_DEBUG_SYSTEM (7)) (type: tracepoint)
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0lttng_ust_tracelog:TRACE_INFO (loglevel: TRAC= E_INFO (6)) (type: tracepoint)
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0lttng_ust_tracelog:TRACE_NOTICE (loglevel: TR= ACE_NOTICE (5)) (type: tracepoint)
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0lttng_ust_tracelog:TRACE_WARNING (loglevel: T= RACE_WARNING (4)) (type: tracepoint)
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0lttng_ust_tracelog:TRACE_ERR (loglevel: TRACE= _ERR (3)) (type: tracepoint)
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0lttng_ust_tracelog:TRACE_CRIT (loglevel: TRAC= E_CRIT (2)) (type: tracepoint)
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0lttng_ust_tracelog:TRACE_ALERT (loglevel: TRA= CE_ALERT (1)) (type: tracepoint)
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0lttng_ust_tracelog:TRACE_EMERG (loglevel: TRA= CE_EMERG (0)) (type: tracepoint)
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0lttng_ust_tracef:event (loglevel: TRACE_DEBUG= (14)) (type: tracepoint)
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0lttng_ust_lib:unload (loglevel: TRACE_DEBUG_L= INE (13)) (type: tracepoint)
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0lttng_ust_lib:debug_link (loglevel: TRACE_DEB= UG_LINE (13)) (type: tracepoint)
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0lttng_ust_lib:build_id (loglevel: TRACE_DEBUG= _LINE (13)) (type: tracepoint)
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0lttng_ust_lib:load (loglevel: TRACE_DEBUG_LIN= E (13)) (type: tracepoint)
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0lttng_ust_statedump:end (loglevel: TRACE_DEBU= G_LINE (13)) (type: tracepoint)
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0lttng_ust_statedump:debug_link (loglevel: TRA= CE_DEBUG_LINE (13)) (type: tracepoint)
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0lttng_ust_statedump:build_id (loglevel: TRACE= _DEBUG_LINE (13)) (type: tracepoint)
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0lttng_ust_statedump:bin_info (loglevel: TRACE= _DEBUG_LINE (13)) (type: tracepoint)
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0lttng_ust_statedump:start (loglevel: TRACE_DE= BUG_LINE (13)) (type: tracepoint)

=C2=A0

= When I do lttng start, I get a directory structure in the trace directory b= ut the file sizes do not ever increase.=C2=A0 It's just a directory str= ucture with files created and then that's it, no events being written to the files.=C2=A0 At first I thought things just weren'= ;t linked in right and not working but then discovered how to run lttng-ses= siond in -vvv mode and then could see lots of things are going on, tracing = appears to be trying to work as there is tons of activity being output by lttng-sessiond, but like I said the trace= files aren't being added to.

=C2=A0

= I looked at the lttng-sessiond log I captured and can't quite make out = the problem so attaching it.

=C2=A0

= I'm probably not doing something right but can't figure out what th= at is so any pointers appreciated.

=C2=A0

= Regards,

=C2=A0

= Brian

=C2=A0

--000000000000ae172d05fa284932-- --===============7013215603572134866== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ lttng-dev mailing list lttng-dev@lists.lttng.org https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev --===============7013215603572134866==--