LTTng kernel tracer writes packetized (binary) metadata instead of plain text on Ubuntu 22.04 5.15.0-140/141-generic with LTTng 2.13.18

LTTng kernel tracer writes packetized (binary) metadata instead of plain text on Ubuntu 22.04 5.15.0-140/141-generic with LTTng 2.13.18

[Shreya Tripathi via lttng-dev]

[-- Attachment #1: Type: text/plain, Size: 3078 bytes --]

Component(s) involved:

  *   LTTng-modules (kernel tracing)

  *   LTTng-tools

________________________________
Component version:

  *   lttng-tools: 2.13.15 (lttng --version)

  *   lttng-modules-dkms: 2.13.18 (modinfo lttng_tracer | grep version)

  *   Babeltrace2: 2.0.4 (babeltrace2 --version)

________________________________
System information:

  *   Architecture: x86_64, 4 CPUs

  *   OS: Ubuntu 22.04.4 LTS

  *   Kernel:

  *   Linux shrtripathi-vm 5.15.0-140-generic #150-Ubuntu SMP Sat Apr 12 06:00:09 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

  *   Also tested: 5.15.0-141-generic

  *   Distribution: Ubuntu 22.04.4 LTS

Expected results: After running a kernel trace session, the file /tmp/my-kernel-trace/kernel/metadata should be plain text, starting with /* CTF 1.8 */, and be readable by Babeltrace2 and the Python bt2 bindings.
________________________________
Actual results:

  *   The metadata file is written in "packetized" (binary) format, as detected by the file command:
text
Apply
  /tmp/my-kernel-trace/kernel/metadata: Common Trace Format (CTF) packetized metadata (LE), v1.8

  *   The first line is binary garbage, not /* CTF 1.8 */.

  *   Babeltrace2 and the Python bt2 bindings cannot parse the trace, reporting errors about invalid metadata.

  *   Example of the first lines of the metadata file:
 W�u�kzM�W{L����^l�typealias integer { size = 8; align = 8; signed = false; } := uint8_t;
  typealias integer { size = 16; align = 8; signed = false; } := uint16_t;
  ...

Logs and context:

  *   Babeltrace2 error:

  At line 146 in metadata stream: Invalid character: char="", val=0x1d
  At line 146 in metadata stream: syntax error, unexpected CTF_ERROR, expecting CTF_TYPEASSIGN or CTF_EQUAL: token=""


  *   All LTTng kernel modules and user-space tools are version 2.13.18.

  *   All modules are loaded from /lib/modules/5.15.0-140-generic/updates/dkms/ and match the running kernel.

  *   No old or mismatched modules are present.

Loaded kernel modules:

lsmod | grep lttng
lttng_clock            16384  0
lttng_kretprobes       16384  0
lttng_kprobes          16384  0
lttng_uprobes          16384  0
lttng_statedump       749568  0
lttng_wrapper          16384  1 lttng_statedump
...

Additional context:

  *   The issue persists across multiple kernel versions (5.15.0-140-generic, 5.15.0-141-generic).

  *   User-space tracing works as expected (metadata is plain text).

  *   The issue is not fixable by editing the file, as the format is fundamentally binary.

________________________________
What should happen:
The kernel tracer should write plain text CTF metadata, starting with /* CTF 1.8 */, so that Babeltrace2 and the Python bt2 bindings can parse the trace.What happens:The kernel tracer writes packetized (binary) metadata, which is not usable for analysis.
________________________________
Thank you for your help! Please let me know if you need any more logs or information.

Regards
Shreya Tripathi

[-- Attachment #2: Type: text/html, Size: 62928 bytes --]

Re: LTTng kernel tracer writes packetized (binary) metadata instead of plain text on Ubuntu 22.04 5.15.0-140/141-generic with LTTng 2.13.18

[Kienan Stewart via lttng-dev]

Hi Shreya,

It's more suspicious that the UST metadata file doesn't have a binary 
header.

It's normal and expected that metadata starts with some binary 
information. In the case of the UST, I believe it's the 
metdata_packet_header as described here[1].

Similarly for the kernel metadata file[2].

Using the same versions you described on Ubuntu jammy I was not able to 
reproduce the issue you had. Both the UST and kernel traces metadata 
files had their binary packet headers intact.

You stated the following:

 >
 >  *   All LTTng kernel modules and user-space tools are version 2.13.18.
 >

There is no release of LTTng-UST 2.13.18. Could you clarify which 
version of LTTng-UST you are using?

Furthermore, commands to replicate the issue would be greatly appreciated.

Finally, can you confirm which part of the trace babeltrace2 is having 
an error with by choosing a different sub-folder. E.g., in my case:

```
$ babeltrace2 auto-20250604-160417/kernel >/dev/null || echo 'Error'
$ babeltrace2 auto-20250604-160417/ust >/dev/null || echo 'Error'
```

Are you certain that it's the kernel trace which is causing babeltrace2 
to throw an error?

thanks,
kienan

[1]: 
https://github.com/lttng/lttng-ust/blob/611ccd9f2f6581faa48b47881520069c30c31daa/src/common/ringbuffer-clients/metadata-template.h#L21
[2]: 
https://github.com/lttng/lttng-modules/blob/d47de02f489686d335ee570cc5060d653757d3cc/src/lttng-ring-buffer-metadata-client.h#L19

On 6/4/25 1:00 AM, Shreya Tripathi via lttng-dev wrote:
> Component(s) involved:
> 
>    *   LTTng-modules (kernel tracing)
> 
>    *   LTTng-tools
> 
> ________________________________
> Component version:
> 
>    *   lttng-tools: 2.13.15 (lttng --version)
> 
>    *   lttng-modules-dkms: 2.13.18 (modinfo lttng_tracer | grep version)
> 
>    *   Babeltrace2: 2.0.4 (babeltrace2 --version)
> 
> ________________________________
> System information:
> 
>    *   Architecture: x86_64, 4 CPUs
> 
>    *   OS: Ubuntu 22.04.4 LTS
> 
>    *   Kernel:
> 
>    *   Linux shrtripathi-vm 5.15.0-140-generic #150-Ubuntu SMP Sat Apr 12 06:00:09 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
> 
>    *   Also tested: 5.15.0-141-generic
> 
>    *   Distribution: Ubuntu 22.04.4 LTS
> 
> Expected results: After running a kernel trace session, the file /tmp/my-kernel-trace/kernel/metadata should be plain text, starting with /* CTF 1.8 */, and be readable by Babeltrace2 and the Python bt2 bindings.
> ________________________________
> Actual results:
> 
>    *   The metadata file is written in "packetized" (binary) format, as detected by the file command:
> text
> Apply
>    /tmp/my-kernel-trace/kernel/metadata: Common Trace Format (CTF) packetized metadata (LE), v1.8
> 
>    *   The first line is binary garbage, not /* CTF 1.8 */.
> 
>    *   Babeltrace2 and the Python bt2 bindings cannot parse the trace, reporting errors about invalid metadata.
> 
>    *   Example of the first lines of the metadata file:
>   W�u�kzM�W{L����^l�typealias integer { size = 8; align = 8; signed = false; } := uint8_t;
>    typealias integer { size = 16; align = 8; signed = false; } := uint16_t;
>    ...
> 
> Logs and context:
> 
>    *   Babeltrace2 error:
> 
>    At line 146 in metadata stream: Invalid character: char="", val=0x1d
>    At line 146 in metadata stream: syntax error, unexpected CTF_ERROR, expecting CTF_TYPEASSIGN or CTF_EQUAL: token=""
> 
> 
>    *   All LTTng kernel modules and user-space tools are version 2.13.18.
> 
>    *   All modules are loaded from /lib/modules/5.15.0-140-generic/updates/dkms/ and match the running kernel.
> 
>    *   No old or mismatched modules are present.
> 
> Loaded kernel modules:
> 
> lsmod | grep lttng
> lttng_clock            16384  0
> lttng_kretprobes       16384  0
> lttng_kprobes          16384  0
> lttng_uprobes          16384  0
> lttng_statedump       749568  0
> lttng_wrapper          16384  1 lttng_statedump
> ...
> 
> Additional context:
> 
>    *   The issue persists across multiple kernel versions (5.15.0-140-generic, 5.15.0-141-generic).
> 
>    *   User-space tracing works as expected (metadata is plain text).
> 
>    *   The issue is not fixable by editing the file, as the format is fundamentally binary.
> 
> ________________________________
> What should happen:
> The kernel tracer should write plain text CTF metadata, starting with /* CTF 1.8 */, so that Babeltrace2 and the Python bt2 bindings can parse the trace.What happens:The kernel tracer writes packetized (binary) metadata, which is not usable for analysis.
> ________________________________
> Thank you for your help! Please let me know if you need any more logs or information.
> 
> Regards
> Shreya Tripathi