From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from simark.ca by simark.ca with LMTP id aUEvNEwDXWhM3h8AWB0awg (envelope-from ) for ; Thu, 26 Jun 2025 04:22:36 -0400 Authentication-Results: simark.ca; dkim=pass (2048-bit key; unprotected) header.d=lists.lttng.org header.i=@lists.lttng.org header.a=rsa-sha256 header.s=default header.b=aGVMn5zW; dkim-atps=neutral Received: by simark.ca (Postfix, from userid 112) id C50F81E11E; Thu, 26 Jun 2025 04:22:36 -0400 (EDT) X-Spam-Checker-Version: SpamAssassin 4.0.1 (2024-03-25) on simark.ca X-Spam-Level: X-Spam-Status: No, score=-6.5 required=5.0 tests=ARC_SIGNED,ARC_VALID,BAYES_00, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,FREEMAIL_REPLYTO_END_DIGIT, HTML_MESSAGE,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE, RCVD_IN_VALIDITY_CERTIFIED,RCVD_IN_VALIDITY_RPBL,RCVD_IN_VALIDITY_SAFE autolearn=ham autolearn_force=no version=4.0.1 Received: from lists.lttng.org (lists.lttng.org [158.69.130.29]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (prime256v1) server-digest SHA256) (No client certificate requested) by simark.ca (Postfix) with ESMTPS id A98D71E089 for ; Thu, 26 Jun 2025 04:22:34 -0400 (EDT) ARC-Filter: OpenARC Filter v1.2.1 lists.lttng.org 4bSWrp1PPXz1d1b ARC-Seal: i=1; d=lists.lttng.org; s=arc1; a=rsa-sha256; cv=none; t=1750926154; b=ngyqiarNuuGFPcD6nbyBFZlHddkYpCHshATh76Y3r/jRRgIgOYN+Za59nihQAeZPa+Iu yGOLJKgFjzCGYGX7jTPVD+kg5UkXG86Z4OF3ApzWCi+A6KpuGQni/DQXOTvRVY7jqMZoR jROK8kULeS46o1zMuGfujU9ZrhS4Ch9iPUd/DZxjCIeRWCPYasqoALpNRQy8UK2J9g6+J vNHg6y8rLBV6r0AEaE9U3pO0eZq7Pu6o5hjAVzexXjB4FIEr0771xIvCLutl/OyhFwAEH 0MM+nKfBlgDOcml2QMKCD8rE5WxhJAzPVlrnSDPA2ShrnWbI5Hn18/ta6wdMuYJns5A== ARC-Message-Signature: i=1; d=lists.lttng.org; s=arc1; a=rsa-sha256; c=relaxed/simple; t=1750926154; h=DKIM-Signature:MIME-Version:Date:Message-ID:Subject:To:From:From; bh=PWPqAYBwdfuYtZtlyN58aNNJj9LcVhFoEABN32iIlWg=; b=nQ7RlIAR7pFayFSn2KoVdeEwvER8VyR8OVLW3q25qYt4/pfVMGOUhI1rb1vyFpjIVdJy rqYj2AMWL7MJwIQTPVlOMySLtDBkrFdoGhtwkypXonI/z79lh2Mb92k2vYwz5EM8SMsva lHs/DGETHduWRBOSnUafDWpfyNpNjF0JCTWGsNOfgW4Vw8wuCh3+GjCXeOis6R4rb/zxj GBzxYFMTtO1tblq/bgFQS6K0JSt8qXf+u1a7v++Zl6kXIW2z5V67+8XmijBgyoz2cwB+5 ksRebcDIPSkkkYPuDACOV3ZpjoJAYPBzVWU2Nr4qUpvEX1/yOJBTL/9WSFpyBA/517w== ARC-Authentication-Results: i=1; lists.lttng.org; arc=none smtp.remote-ip="::1" DKIM-Filter: OpenDKIM Filter v2.11.0 lists.lttng.org 4bSWrp1PPXz1d1b DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=lists.lttng.org; s=default; t=1750926154; bh=PWPqAYBwdfuYtZtlyN58aNNJj9LcVhFoEABN32iIlWg=; h=References:In-Reply-To:Date:Subject:To:Cc:List-Id: List-Unsubscribe:List-Archive:List-Post:List-Help:List-Subscribe: From:Reply-To:From; b=aGVMn5zWBTB060ysRCRv9rBYruI30HbJFm9Vs15+EqBOE9Me97Nc78QFZggtmyjhe ukHus4+YbQT9Jd1AD8kAkZJfb7cPcfgGHtbA397WRGLx2/8/rrp8zeNGYwo5EgYXK+ cMaCsUSXsBoLBT0j/e8wReFsK5q+hNcG6Xp+bkuf3hAngzoFJyeuGhMs0/EmKpc0BL y7w6gu/yoCUBYtyeK6wQbgddmJxVCEEeN33+qkSYkAuDcOVHsZkrNE5mSdib+Z2p+R s6esfxdOk1mV25AqyrClrqsIvJE2TtG+wkIRyrJ8tp5zRrS+2Wtw2Fh8VUylFtYSWt 8qwMThBATwsig== Received: from lists-lttng01.efficios.com (localhost [IPv6:::1]) by lists.lttng.org (Postfix) with ESMTP id 4bSWrp1PPXz1d1b; Thu, 26 Jun 2025 04:22:34 -0400 (EDT) ARC-Filter: OpenARC Filter v1.2.1 lists.lttng.org 4bSWrn1MPTz1d1Z DKIM-Filter: OpenDKIM Filter v2.11.0 lists.lttng.org 4bSWrn1MPTz1d1Z Received: from mail-il1-x131.google.com (mail-il1-x131.google.com [IPv6:2607:f8b0:4864:20::131]) by lists.lttng.org (Postfix) with ESMTPS id 4bSWrn1MPTz1d1Z for ; Thu, 26 Jun 2025 04:22:32 -0400 (EDT) Received: by mail-il1-x131.google.com with SMTP id e9e14a558f8ab-3da82c6c5d4so6211705ab.1 for ; Thu, 26 Jun 2025 01:22:32 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1750926151; x=1751530951; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=um8Qkfp329hxxdg1bqMwd+hI1X+uH2PCrBNbKMWergo=; b=mvZ5+Z8LoRh4l6folkvznOm/qavBQ8FBrwQeSjxZdmiccwLxknAkw9F4707DxPk/Rh N3mxntPIIVzPieckUNw7HKna7p9xbJC/aBFTFwgVrBJBfTXiTXpp/UKtnhiNTxdBBYVT TqSPEormNIRQaDRSu3IvJSIabOCwgvIOqw519PIaxEZUuxdBY/5qTqgXzmfNJAIPiV+/ rgDxlrx/wlS58mdvNjSB3dgE3sRUGiDtgQKa+MgRttjGjv5KwbJgmckYXzWp+pHmB/q3 ZUXXAEYUDxwCIOUH76QSlFhjtdh1MOSLS8Y29BjDaYdCCcjyOk4XcAVxS4OzwHQsp0Qw nF/w== X-Gm-Message-State: AOJu0Yx9YbXT2dHe2V04QEaOkjxDj+1daQ7cARzqLiBxB+ghtgmbito4 Vsupj32YxE+xjLr5ZhN5LbW8Fq9YvHvGddGl3oAVHtaxEIRB0EQQLHwaRW8p8kGLXWwgkP+xZ3b LFhGC1rqpA9TBJR/i+33zHjjejo6j6DA= X-Gm-Gg: ASbGncsbrkKnKjOQAAmnxl6xA/nQbUjbTJf3xst3dG05LhHzztuN+iFUuTjtf1R3NEk gkTJq8apsowKTRqicMEYMsfEOvArDq6o/L0mdar20ADE+rOotS/6/tZIVs858lvkEd0chsTD+cT nigt5uOhbPYNlTsvsI5t5upxWQfiqq7j4KMyfDoob97w== X-Google-Smtp-Source: AGHT+IHlLTmaXv7flv8SeJnHyTAnCcZp4DIDCWDjsXw7RBm5HFAqX5NRNwoZ5Dgjk4dMvGQKHBULHZsiyl6XroM2RpM= X-Received: by 2002:a05:6e02:1fc9:b0:3dc:8b57:b750 with SMTP id e9e14a558f8ab-3df32a0af03mr77732615ab.17.1750926151173; Thu, 26 Jun 2025 01:22:31 -0700 (PDT) MIME-Version: 1.0 References: <34912d84-7750-454c-9f96-fb8d865ba8d8@efficios.com> <6914656f-9b7b-43fa-9dd1-9b27506c6d2a@efficios.com> In-Reply-To: Date: Thu, 26 Jun 2025 10:22:18 +0200 X-Gm-Features: Ac12FXxnqUdeX66zkQVimw464XYNffWVqB-BfxCRUDENGXwggRY9WyyFAzaWGDw Message-ID: Subject: Re: Malformed Tracefiles To: Kienan Stewart Cc: lttng-dev@lists.lttng.org Content-Type: multipart/alternative; boundary="00000000000044cd4c0638754320" X-BeenThere: lttng-dev@lists.lttng.org X-Mailman-Version: 2.1.39 Precedence: list List-Id: LTTng development list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , From: Georg Muck via lttng-dev Reply-To: Georg Muck Errors-To: lttng-dev-bounces@lists.lttng.org Sender: "lttng-dev" --00000000000044cd4c0638754320 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hi Kienan, switching to tracing per uid solved the Problem. Huge thanks for your patients! Best regards, Georg Never burn the bridges before you If you've burned the bridges behind Never burn the one that you're walking upon Or you're sure to hit the water in time. *(Savatage, Streets, 1991)* Kienan Stewart schrieb am Mi., 25. Juni 2025, 22:52= : > Hi Georg, > > On 6/25/25 1:27 PM, Georg Muck wrote: > > Hi Kienan, > > > > thanks for your quick response. :) > > > > I replaced "--daemonize" with "--background". > > > > I guess I should briefly describe my goal to avoid confusion: I want to > > trace a ROS2-session online (tried LTTng-live, but it costs too much > > ressources, hence the aggressive rotation of 1s). LTTng gives me the pa= th > > to a newly created trace-file as soon as it is created. This path is > > forwarded to a Thread-Pool that does the rest (analyses each tracefile = by > > itself)=E2=80=A6 . However, the tracefiles are removed in the end, caus= e I'm not > > interested in them, only the program should be interested in them (forg= ot > > to remove it in the example). The problem only occurs when I start to > many > > ROS2 instances at once, meaning gathering too many tracepoints at once > (at > > once meaning in a very short amount of time). It works fine for 1-2 > > instances, but it doesn't work for 16 instances. > > > >> 1. *Before any traced application is run, babeltrace2 has the followin= g > > error output:* Ah ok, thought I would be on the right track. > >> 2. *After a traced application is run whether is produces enabled even= ts > > or not, I see the events recorded*: What do you mean by that? There is = no > > babeltrace2 error if traced instances are running, even tho they are no= t > > creating events? > > This is what I observed with a fresh start: > > before running any traced application: bt2 error with no traced content > > after running a traced application with no events: bt2 no longer > produced an error, but output 0 events (as expected). > > In this case I expect that it is because the consumer started and basic > metadata was available so that was written in the snapshot. If I recall > correctly, the consumer daemon only starts when the first application > connects > > after running a traced application with some events: as usual > >> 3. *When your session script stops, the output folder is deleted*: lik= e > > described, forgot to remove, but it is on purpose > > > > I tried *tp**:tptest*, but there is no output shown by babeltrace2 (is > that > > intended?): > > > > I had the 100 events produced by gen-ust-events binary that invoked. You > ran it but the events weren't recorded? Regardless I think we're on the > same page that the session setup program you are using is fine right now. > > > [georg@KARTOFF8xE-zenbook /tmp/structural_traces]$ babeltrace2 archives > > [georg@KARTOFF8xE-zenbook /tmp/structural_traces]$ > > > > > > However, I figured something out: > > When I trace the "ros:rcl_node_init"-Tracepoint and start 12 ROS2 > > Instances, I receive the following output for the entire > archive-directory: > > [georg@KARTOFF8xE-zenbook /tmp/structural_traces/archives]$ babeltrace2 > . > > > > [19:01:41.474172508] (+?.?????????) KARTOFF8xE-zenbook:listener:(24897) > > ros2:rcl_node_init: { cpu_id =3D 1 }, { procname =3D "listener", vpid = =3D 24897 > > }, { node_handle =3D 0x5CCBD0EFA280, rmw_handle =3D 0x5CCBD107A420, > node_name =3D > > "saturn_listener", namespace =3D "/hallo/saturn" } > > [19:01:41.493613263] (+0.019440755) KARTOFF8xE-zenbook:talker:(24898) > > ros2:rcl_node_init: { cpu_id =3D 4 }, { procname =3D "talker", vpid =3D= 24898 > }, > > { node_handle =3D 0x5A8E10DFC6E0, rmw_handle =3D 0x5A8E10F8AA60, node_n= ame =3D > > "erde_talker", namespace =3D "/hallo/erde" } > > > > All coming from archive-file number 10: > > [georg@KARTOFF8xE-zenbook > > > /tmp/structural_traces/archives/20250625T190141+0200-20250625T190142+0200= -10]$ > > babeltrace2 . > > > > [19:01:41.474172508] (+?.?????????) KARTOFF8xE-zenbook:listener:(*24897= *) > > ros2:rcl_node_init: { cpu_id =3D 1 }, { procname =3D "listener", vpid = =3D 24897 > > }, { node_handle =3D 0x5CCBD0EFA280, rmw_handle =3D 0x5CCBD107A420, > node_name =3D > > "saturn_listener", namespace =3D "/hallo/saturn" } > > [19:01:41.493613263] (+0.019440755) KARTOFF8xE-zenbook:talker:(24898) > > ros2:rcl_node_init: { cpu_id =3D 4 }, { procname =3D "talker", vpid =3D= 24898 > }, > > { node_handle =3D 0x5A8E10DFC6E0, rmw_handle =3D 0x5A8E10F8AA60, node_n= ame =3D > > "erde_talker", namespace =3D "/hallo/erde" } > > > > However, looking deeper into that file I saw that there are all 12 > > instances (+some of ros2 itself) I started within the trace-directory > > number 10! > > [georg@KARTOFF8xE-zenbook > > > /tmp/structural_traces/archives/20250625T190141+0200-20250625T190142+0200= -10/ust/pid]$ > > ls > > > > listener-24897-20250625-190141 listener-24903-20250625-190141 > > ros2-24447-20250625-190131 ros2-24887-20250625-190141 > > talker-24900-20250625-190141 talker-24916-20250625-190141 > > listener-24899-20250625-190141 listener-24914-20250625-190141 > > ros2-24451-20250625-190131 talker-24896-20250625-190141 > > talker-24902-20250625-190141 > > listener-24901-20250625-190141 listener-24917-20250625-190141 > > ros2-24834-20250625-190134 talker-24898-20250625-190141 > > talker-24904-20250625-190141 > > > > BUT: > > I looked into the specific directory of a traced instance (underlined i= t) > > [georg@KARTOFF8xE-zenbook > > > /tmp/structural_traces/archives/20250625T190141+0200-20250625T190142+0200= -10/ust/pid]$ > > ls listener-*24897*-20250625-190141/ > > > > index structuralChannel_0 structuralChannel_2 structuralChannel_4 > > structuralChannel_6 structuralChannel_8 structuralChannel_10 > > *metadata* structuralChannel_1 structuralChannel_3 structuralChannel= _5 > > structuralChannel_7 structuralChannel_9 structuralChannel_11 > > > > and of a non-traced instance: > > [georg@KARTOFF8xE-zenbook > > > /tmp/structural_traces/archives/20250625T190141+0200-20250625T190142+0200= -10/ust/pid]$ > > ls listener-24899-20250625-190141 > > > > index structuralChannel_1 structuralChannel_3 > > structuralChannel_5 structuralChannel_7 structuralChannel_9 > > structuralChannel_11 > > structuralChannel_0 structuralChannel_2 structuralChannel_4 > > structuralChannel_6 structuralChannel_8 structuralChannel_10 > > > > Turns out: There is no metadata file within the second one. Therefore > > babeltrace2 is not able to parse the file. Do you have an idea why? > > > > From your later e-mail with the updated sessiond log which shows the > PERROR for too many open files it sounds like you have a thread to pull o= n. > > You can try using the per-uid buffering scheme to reduce the number of > FDs required when running 12 concurrent instances of ros. > > > Best regards, > > Georg > > > > Never burn the bridges before you > > If you've burned the bridges behind > > Never burn the one that you're walking upon > > Or you're sure to hit the water in time. *(Savatage, Streets, 1991)* > > > > > > Am Mi., 25. Juni 2025 um 17:50 Uhr schrieb Kienan Stewart < > > kstewart@efficios.com>: > > > >> Hi Georg, > >> > >> On 6/25/25 9:36 AM, Georg Muck wrote: > >>> Hi Kienan, > >>> > >>> I did exactly that, without success. > >>> > >>> The log of the lttng-sessiond (lttng.log) did not change after starti= ng > >> the > >>> ROS2 instances. I have also added the output of "lttng list > >>> structuralSession" > >>> I start my tracing instance as follows: > >>> > >>> At first there is a bash-script that sets everything up, it does the > >>> following: > >>> > >>> lttng destroy -a # to destroy all existing sessions > >>> lttng-sessiond --daemonize --verbose-consumer -vvv &> "/tmp/lttng.log= " > # > >> > >> When started with '--daemonize', the file descriptors are closed[1] > >> after forking which is why you don't see any further output to > >> /tmp/lttng.log. Try with '--background' instead. > >> > >>> starting a new session Deamon > >>> ./exemplary_tracing_session # starting my exemplary tracing session > >>> > >>> The exemplary tracing session is written by the help of the c Api > >>> (appendix). > >>> > >> > >> I modified your script to add 'tp:tptest' (a tracepoint from the > >> lttng-tools test suite program gen-ust-events) and did the following: > >> > >> ``` > >> $ lttng-sessiond --background --verbose-consumer -vvv &> sessiond.log > >> $ ./exemplary_tracing_session & > >> $ lttng-tools/./tests/utils/testapp/gen-ust-events/gen-ust-events > >> babeltrace2 /tmp/structural_traces | wc -l > >> 100 > >> ``` > >> > >> It seems to be working fine for me. A couple of notes: > >> > >> 1. Before any traced application is run, babeltrace2 has the following > >> error output: No trace was found based on input `/tmp/structural_trace= s` > >> 2. After a traced application is run whether is produces enabled event= s > >> or not, I see the events recorded. > >> 3. When your session script stops, the output folder is deleted. > >> > >>> I started the the same procedure on another machine. The same fault > >> > >> The fault is that the babeltrace2 has an error, right? Are you willing > >> to share your trace that has the problem and/or show the output of > >> babeltrace2's error? > >> > >>> occurred. I think I need to configure more within the `lttngDomain`. > But > >> I > >>> don't know how, would you have an idea/hint? > >>> > >>> Thanks for your help. > >>> > >>> Best regards, > >>> Georg > >>> > >>> Ps.: I also added the log of my tracing tool (structural.log) > >>> > >>> Never burn the bridges before you > >>> If you've burned the bridges behind > >>> Never burn the one that you're walking upon > >>> Or you're sure to hit the water in time. *(Savatage, Streets, 1991)* > >>> > >>> > >>> Am Mo., 23. Juni 2025 um 16:26 Uhr schrieb Kienan Stewart < > >>> kstewart@efficios.com>: > >>> > >>>> Hi Georg, > >>>> > >>>> a good first step is to run lttng-session with `-vvv > --verbose-consumer` > >>>> and inspect the log file and `LTTNG_UST_DEBUG=3D1` in the environmen= t. > >>>> > >>>> It may also be useful to run your traced applications with > >>>> `LTTNG_UST_DEBUG=3D1` set in the environment. > >>>> > >>>> Do you have steps that reproduce the issue? E.g., all the commands t= o > >>>> set up the tracing session and so forth. > >>>> > >>>> thanks, > >>>> kienan > >>>> > >>>> On 6/19/25 10:58 AM, Georg Muck via lttng-dev wrote: > >>>>> Hey, > >>>>> > >>>>> I am using the LTTng tracer (2.13.11, C-API, rotation=3D1s) to trac= e my > >>>> ROS2 > >>>>> system (userspace only). Everything works just fine within my Docke= r > >>>>> Container (Ubuntu 24.04), but if I start the same procedure on the > host > >>>>> itself (also Ubuntu 24.04), it does not create a TraceFile if to ma= ny > >>>>> tracepoints occur (speaking of roundabout 1500+ per second) > (sometimes > >> it > >>>>> malformed the tracefile, but I was not able to recreate that. If I > >>>> remember > >>>>> correctly, Babeltrace2 threw an error that said that there was no > >>>>> metadata-file). I played around with some configurations (like > >> increasing > >>>>> the buffer size from ~16kB to ~512kB or a less aggressive rotation)= , > >> but > >>>>> the problem stayed. > >>>>> > >>>>> Would be thankful for help. I guess I delivered to less information= , > >>>> please > >>>>> reach out what is interesting at this point, I am not sure about th= at > >> :). > >>>>> > >>>>> Best regards, > >>>>> Georg > >>>>> > >>>>> > >>>>> > >>>>> Never burn the bridges before you > >>>>> If you've burned the bridges behind > >>>>> Never burn the one that you're walking upon > >>>>> Or you're sure to hit the water in time. *(Savatage, Streets, 1991)= * > >>>>> > >>>> > >>>> > >>> > >> > >> thanks, > >> kienan > >> > >> [1]: https://lttng.org/man/8/lttng-sessiond/v2.13/#doc-opt--daemonize > >> > > > > --00000000000044cd4c0638754320 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable

Hi Kienan,

switching to tracing per uid solved the Problem.

Huge thanks for your patients!


Best regards,

Georg=C2=A0


Never burn the bridges before you
If you've burned the= bridges behind
Never burn the one that you're walking upon
Or you're= sure to hit the water in time. (Savatage, Streets, 1991)
=

Kienan= Stewart <kstewart@efficios.com> schrieb am Mi., 25. Juni 2025= , 22:52:
Hi Georg,

On 6/25/25 1:27 PM, Georg Muck wrote:
> Hi Kienan,
>
> thanks for your quick response. :)
>
> I replaced "--daemonize" with "--background".
>
> I guess I should briefly describe my goal to avoid confusion: I want t= o
> trace a ROS2-session online (tried LTTng-live, but it costs too much > ressources, hence the aggressive rotation of 1s). LTTng gives me the p= ath
> to a newly created trace-file as soon as it is created. This path is > forwarded to a Thread-Pool that does the rest (analyses each tracefile= by
> itself)=E2=80=A6 . However, the tracefiles are removed in the end, cau= se I'm not
> interested in them, only the program should be interested in them (for= got
> to remove it in the example). The problem only occurs when I start to = many
> ROS2 instances at once, meaning gathering too many tracepoints at once= (at
> once meaning in a very short amount of time). It works fine for 1-2 > instances, but it doesn't work for 16 instances.
>
>> 1. *Before any traced application is run, babeltrace2 has the foll= owing
> error output:* Ah ok, thought I would be on the right track.
>> 2. *After a traced application is run whether is produces enabled = events
> or not, I see the events recorded*: What do you mean by that? There is= no
> babeltrace2 error if traced instances are running, even tho they are n= ot
> creating events?

This is what I observed with a fresh start:

before running any traced application: bt2 error with no traced content

after running a traced application with no events: bt2 no longer
produced an error, but output 0 events (as expected).

In this case I expect that it is because the consumer started and basic metadata was available so that was written in the snapshot. If I recall correctly, the consumer daemon only starts when the first application
connects

after running a traced application with some events: as usual
>> 3. *When your session script stops, the output folder is deleted*:= like
> described, forgot to remove, but it is on purpose
>
> I tried *tp**:tptest*, but there is no output shown by babeltrace2 (is= that
> intended?):
>

I had the 100 events produced by gen-ust-events binary that invoked. You ran it but the events weren't recorded? Regardless I think we're on= the
same page that the session setup program you are using is fine right now.
> [georg@KARTOFF8xE-zenbook /tmp/structural_traces]$ babeltrace2 archive= s
> [georg@KARTOFF8xE-zenbook /tmp/structural_traces]$
>
>
> However, I figured something out:
> When I trace the "ros:rcl_node_init"-Tracepoint and start 12= ROS2
> Instances, I receive the following output for the entire archive-direc= tory:
> [georg@KARTOFF8xE-zenbook /tmp/structural_traces/archives]$ babeltrace= 2 .
>
> [19:01:41.474172508] (+?.?????????) KARTOFF8xE-zenbook:listener:(24897= )
> ros2:rcl_node_init: { cpu_id =3D 1 }, { procname =3D "listener&qu= ot;, vpid =3D 24897
> }, { node_handle =3D 0x5CCBD0EFA280, rmw_handle =3D 0x5CCBD107A420, no= de_name =3D
> "saturn_listener", namespace =3D "/hallo/saturn" }=
> [19:01:41.493613263] (+0.019440755) KARTOFF8xE-zenbook:talker:(24898)<= br> > ros2:rcl_node_init: { cpu_id =3D 4 }, { procname =3D "talker"= ;, vpid =3D 24898 },
> { node_handle =3D 0x5A8E10DFC6E0, rmw_handle =3D 0x5A8E10F8AA60, node_= name =3D
> "erde_talker", namespace =3D "/hallo/erde" }
>
> All coming from archive-file number 10:
> [georg@KARTOFF8xE-zenbook
> /tmp/structural_traces/archives/20250625T190141+0200-20250625T190142+0= 200-10]$
> babeltrace2 .
>
> [19:01:41.474172508] (+?.?????????) KARTOFF8xE-zenbook:listener:(*2489= 7*)
> ros2:rcl_node_init: { cpu_id =3D 1 }, { procname =3D "listener&qu= ot;, vpid =3D 24897
> }, { node_handle =3D 0x5CCBD0EFA280, rmw_handle =3D 0x5CCBD107A420, no= de_name =3D
> "saturn_listener", namespace =3D "/hallo/saturn" }=
> [19:01:41.493613263] (+0.019440755) KARTOFF8xE-zenbook:talker:(24898)<= br> > ros2:rcl_node_init: { cpu_id =3D 4 }, { procname =3D "talker"= ;, vpid =3D 24898 },
> { node_handle =3D 0x5A8E10DFC6E0, rmw_handle =3D 0x5A8E10F8AA60, node_= name =3D
> "erde_talker", namespace =3D "/hallo/erde" }
>
> However, looking deeper into that file I saw that there are all 12
> instances (+some of ros2 itself) I started within the trace-directory<= br> > number 10!
> [georg@KARTOFF8xE-zenbook
> /tmp/structural_traces/archives/20250625T190141+0200-20250625T190142+0= 200-10/ust/pid]$
> ls
>
> listener-24897-20250625-190141=C2=A0 listener-24903-20250625-190141 >=C2=A0 =C2=A0ros2-24447-20250625-190131=C2=A0 ros2-24887-20250625-19014= 1
>=C2=A0 =C2=A0talker-24900-20250625-190141=C2=A0 talker-24916-20250625-1= 90141
> listener-24899-20250625-190141=C2=A0 listener-24914-20250625-190141 >=C2=A0 =C2=A0ros2-24451-20250625-190131=C2=A0 talker-24896-20250625-190= 141
>=C2=A0 =C2=A0talker-24902-20250625-190141
> listener-24901-20250625-190141=C2=A0 listener-24917-20250625-190141 >=C2=A0 =C2=A0ros2-24834-20250625-190134=C2=A0 talker-24898-20250625-190= 141
>=C2=A0 =C2=A0talker-24904-20250625-190141
>
> BUT:
> I looked into the specific directory of a traced instance (underlined = it)
> [georg@KARTOFF8xE-zenbook
> /tmp/structural_traces/archives/20250625T190141+0200-20250625T190142+0= 200-10/ust/pid]$
> ls listener-*24897*-20250625-190141/
>
> index=C2=A0 =C2=A0 =C2=A0structuralChannel_0=C2=A0 structuralChannel_2= =C2=A0 structuralChannel_4
>=C2=A0 =C2=A0structuralChannel_6=C2=A0 structuralChannel_8=C2=A0 struct= uralChannel_10
> *metadata*=C2=A0 structuralChannel_1=C2=A0 structuralChannel_3=C2=A0 s= tructuralChannel_5
>=C2=A0 =C2=A0structuralChannel_7=C2=A0 structuralChannel_9=C2=A0 struct= uralChannel_11
>
> and of a non-traced instance:
> [georg@KARTOFF8xE-zenbook=C2=A0 =C2=A0 =C2=A0
> /tmp/structural_traces/archives/20250625T190141+0200-20250625T190142+0= 200-10/ust/pid]$
> ls listener-24899-20250625-190141
>
> index=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 structura= lChannel_1=C2=A0 structuralChannel_3
>=C2=A0 =C2=A0structuralChannel_5=C2=A0 structuralChannel_7=C2=A0 struct= uralChannel_9
> structuralChannel_11
> structuralChannel_0=C2=A0 structuralChannel_2=C2=A0 structuralChannel_= 4
>=C2=A0 =C2=A0structuralChannel_6=C2=A0 structuralChannel_8=C2=A0 struct= uralChannel_10
>
> Turns out: There is no metadata file within the second one. Therefore<= br> > babeltrace2 is not able to parse the file. Do you have an idea why? >

=C2=A0From your later e-mail with the updated sessiond log which shows the =
PERROR for too many open files it sounds like you have a thread to pull on.=

You can try using the per-uid buffering scheme to reduce the number of
FDs required when running 12 concurrent instances of ros.

> Best regards,
> Georg
>
> Never burn the bridges before you
> If you've burned the bridges behind
> Never burn the one that you're walking upon
> Or you're sure to hit the water in time. *(Savatage, Streets, 1991= )*
>
>
> Am Mi., 25. Juni 2025 um 17:50 Uhr schrieb Kienan Stewart <
> kstewart@efficios.com>:
>
>> Hi Georg,
>>
>> On 6/25/25 9:36 AM, Georg Muck wrote:
>>> Hi Kienan,
>>>
>>> I did exactly that, without success.
>>>
>>> The log of the lttng-sessiond (lttng.log) did not change after= starting
>> the
>>> ROS2 instances. I have also added the output of "lttng li= st
>>> structuralSession"
>>> I start my tracing instance as follows:
>>>
>>> At first there is a bash-script that sets everything up, it do= es the
>>> following:
>>>
>>> lttng destroy -a # to destroy all existing sessions
>>> lttng-sessiond --daemonize --verbose-consumer -vvv &> &= quot;/tmp/lttng.log" #
>>
>> When started with '--daemonize', the file descriptors are = closed[1]
>> after forking which is why you don't see any further output to=
>> /tmp/lttng.log. Try with '--background' instead.
>>
>>> starting a new session Deamon
>>> ./exemplary_tracing_session # starting my exemplary tracing se= ssion
>>>
>>> The exemplary tracing session is written by the help of the c = Api
>>> (appendix).
>>>
>>
>> I modified your script to add 'tp:tptest' (a tracepoint fr= om the
>> lttng-tools test suite program gen-ust-events) and did the followi= ng:
>>
>> ```
>> $ lttng-sessiond --background --verbose-consumer -vvv &> se= ssiond.log
>> $ ./exemplary_tracing_session &
>> $ lttng-tools/./tests/utils/testapp/gen-ust-events/gen-ust-events<= br> >> babeltrace2 /tmp/structural_traces | wc -l
>> 100
>> ```
>>
>> It seems to be working fine for me. A couple of notes:
>>
>> 1. Before any traced application is run, babeltrace2 has the follo= wing
>> error output: No trace was found based on input `/tmp/structural_t= races`
>> 2. After a traced application is run whether is produces enabled e= vents
>> or not, I see the events recorded.
>> 3. When your session script stops, the output folder is deleted. >>
>>> I started the the same procedure on another machine. The same = fault
>>
>> The fault is that the babeltrace2 has an error, right? Are you wil= ling
>> to share your trace that has the problem and/or show the output of=
>> babeltrace2's error?
>>
>>> occurred. I think I need to configure more within the `lttngDo= main`. But
>> I
>>> don't know how, would you have an idea/hint?
>>>
>>> Thanks for your help.
>>>
>>> Best regards,
>>> Georg
>>>
>>> Ps.: I also added the log of my tracing tool (structural.log)<= br> >>>
>>> Never burn the bridges before you
>>> If you've burned the bridges behind
>>> Never burn the one that you're walking upon
>>> Or you're sure to hit the water in time. *(Savatage, Stree= ts, 1991)*
>>>
>>>
>>> Am Mo., 23. Juni 2025 um 16:26 Uhr schrieb Kienan Stewart <=
>>> kstewart@efficios.com>:
>>>
>>>> Hi Georg,
>>>>
>>>> a good first step is to run lttng-session with `-vvv --ver= bose-consumer`
>>>> and inspect the log file and `LTTNG_UST_DEBUG=3D1` in the = environment.
>>>>
>>>> It may also be useful to run your traced applications with=
>>>> `LTTNG_UST_DEBUG=3D1` set in the environment.
>>>>
>>>> Do you have steps that reproduce the issue? E.g., all the = commands to
>>>> set up the tracing session and so forth.
>>>>
>>>> thanks,
>>>> kienan
>>>>
>>>> On 6/19/25 10:58 AM, Georg Muck via lttng-dev wrote:
>>>>> Hey,
>>>>>
>>>>> I am using the LTTng tracer (2.13.11, C-API, rotation= =3D1s) to trace my
>>>> ROS2
>>>>> system (userspace only). Everything works just fine wi= thin my Docker
>>>>> Container (Ubuntu 24.04), but if I start the same proc= edure on the host
>>>>> itself (also Ubuntu 24.04), it does not create a Trace= File if to many
>>>>> tracepoints occur (speaking of roundabout 1500+ per se= cond) (sometimes
>> it
>>>>> malformed the tracefile, but I was not able to recreat= e that. If I
>>>> remember
>>>>> correctly, Babeltrace2 threw an error that said that t= here was no
>>>>> metadata-file). I played around with some configuratio= ns (like
>> increasing
>>>>> the buffer size from ~16kB to ~512kB or a less aggress= ive rotation),
>> but
>>>>> the problem stayed.
>>>>>
>>>>> Would be thankful for help. I guess I delivered to les= s information,
>>>> please
>>>>> reach out what is interesting at this point, I am not = sure about that
>> :).
>>>>>
>>>>> Best regards,
>>>>> Georg
>>>>>
>>>>>
>>>>>
>>>>> Never burn the bridges before you
>>>>> If you've burned the bridges behind
>>>>> Never burn the one that you're walking upon
>>>>> Or you're sure to hit the water in time. *(Savatag= e, Streets, 1991)*
>>>>>
>>>>
>>>>
>>>
>>
>> thanks,
>> kienan
>>
>> [1]: = https://lttng.org/man/8/lttng-sessiond/v2.13/#doc-opt--daemonize
>>
>

--00000000000044cd4c0638754320--