Tracing Ceph with LTTng-UST issue

Tracing Ceph with LTTng-UST issue

[Lina SADI via lttng-dev]

[-- Attachment #1: Type: text/plain, Size: 2550 bytes --]

Hello,

We are writing to you regarding an issue we encountered while attempting to
trace Ceph with LTTng-UST. Below are the steps we have followed so far:

   1.

   *Compiling Ceph with LTTng Support*:
   - We modified the debian/rules file to enable LTTng support using the
      following flags: -DWITH_LTTNG=ON and
      -DWITH_OSD_INSTRUMENT_FUNCTIONS=ON.
      - The compilation process took approximately 7 hours on a virtual
      machine running Debian 12 with a kernel version 6.1.0-27-amd64, 8 GB of
      RAM, and 2 CPU cores.
      - The build steps can be found in our GitHub
      <https://github.com/KedadsaChakib/ceph-configuration/blob/main/build/README.md>
       repository.
   2.

   *Deploying a Ceph Cluster*:
   - We deployed the cluster manually (1MON,1MGR,2OSDs). Please find the
      script here
      <https://github.com/KedadsaChakib/ceph-configuration/blob/main/cluster/cluster-dep-script.sh>
      .
   3.

   *Attempting to Trace with LTTng-UST*:
   - We tried listing the tracepoints using the following command :

      # lttng list -u

      - This command did not return any Ceph-related tracepoints:

      UST events:
      -------------
      NONE

      4.

   *Preloading liblttng-ust-fork.so*:
   - We attempted to preload liblttng-ust-fork.so before starting the Ceph
      daemons (MON, OSD, MGR) as follows:

      # systemctl edit ceph-mon@debian

      - We added the following configuration:

      [Service]
      Environment="LD_PRELOAD=/usr/lib/x86_64-linux-gnu/liblttng-ust-fork.so



   - After restarting the daemons, we executed lttng list -u, which
      displayed tracepoints related to the libraries (lttng_ust_lib:*),
      trace logs (lttng_ust_tracelog:*), and state dump events (
      lttng_ust_statedump:*), instead of the expected Ceph-related
      tracepoints.

*Expected Result*:

   UST events:
   -------------
   PID: 100859 - Name: /path/to/ceph-osd
       pg:queue_op (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint)
       osd:do_osd_op_post (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint)
       osd:do_osd_op_pre_unknown (loglevel: TRACE_DEBUG_LINE (13))
(type: tracepoint)
       osd:do_osd_op_pre_copy_from (loglevel: TRACE_DEBUG_LINE (13))
(type: tracepoint)
       osd:do_osd_op_pre_copy_get (loglevel: TRACE_DEBUG_LINE (13))
(type: tracepoint)
       ...


*Actual Result*:

   UST events:
   -------------
   lttng_ust_lib:*
   lttng_ust_tracelog:*
   lttng_ust_statedump:*


We would greatly appreciate any help  you could provide to resolve this problem.

[-- Attachment #2: Type: text/html, Size: 7274 bytes --]

Re: Tracing Ceph with LTTng-UST issue

[Kienan Stewart via lttng-dev]

Hi Lina,

I have a couple of questions about your tracing setup.

On 1/2/25 5:14 PM, Lina SADI via lttng-dev wrote:
> Hello,
> 
> We are writing to you regarding an issue we encountered while attempting to
> trace Ceph with LTTng-UST. Below are the steps we have followed so far:
> 
>     1.
> 
>     *Compiling Ceph with LTTng Support*:
>     - We modified the debian/rules file to enable LTTng support using the
>        following flags: -DWITH_LTTNG=ON and
>        -DWITH_OSD_INSTRUMENT_FUNCTIONS=ON.
>        - The compilation process took approximately 7 hours on a virtual
>        machine running Debian 12 with a kernel version 6.1.0-27-amd64, 8 GB of
>        RAM, and 2 CPU cores.
>        - The build steps can be found in our GitHub
>        <https://github.com/KedadsaChakib/ceph-configuration/blob/main/build/README.md>
>         repository.
>     2.
> 
>     *Deploying a Ceph Cluster*:
>     - We deployed the cluster manually (1MON,1MGR,2OSDs). Please find the
>        script here
>        <https://github.com/KedadsaChakib/ceph-configuration/blob/main/cluster/cluster-dep-script.sh>
>        .

With which user a running lttng-sessiond?

When you run `lttng list -u`, what user are you using?

Is that user a member of the `tracing` group?

If you are running lttng-sessiond with verbose logs and/or verbose 
consumers - do you see the traced application(s) connecting?

>     3.
> 
>     *Attempting to Trace with LTTng-UST*:
>     - We tried listing the tracepoints using the following command :
> 
>        # lttng list -u
> 
>        - This command did not return any Ceph-related tracepoints:
> 
>        UST events:
>        -------------
>        NONE
> 
>        4.
> 
>     *Preloading liblttng-ust-fork.so*:
>     - We attempted to preload liblttng-ust-fork.so before starting the Ceph
>        daemons (MON, OSD, MGR) as follows:
> 
>        # systemctl edit ceph-mon@debian
> 
>        - We added the following configuration:
> 
>        [Service]
>        Environment="LD_PRELOAD=/usr/lib/x86_64-linux-gnu/liblttng-ust-fork.so
> 
> 
> 

Does adding `LTTNG_UST_DEBUG=1` to the environment variables add 
information to the service logs that are relevant?


>     - After restarting the daemons, we executed lttng list -u, which
>        displayed tracepoints related to the libraries (lttng_ust_lib:*),
>        trace logs (lttng_ust_tracelog:*), and state dump events (
>        lttng_ust_statedump:*), instead of the expected Ceph-related
>        tracepoints.
> 
> *Expected Result*:
> 
>     UST events:
>     -------------
>     PID: 100859 - Name: /path/to/ceph-osd
>         pg:queue_op (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint)
>         osd:do_osd_op_post (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint)
>         osd:do_osd_op_pre_unknown (loglevel: TRACE_DEBUG_LINE (13))
> (type: tracepoint)
>         osd:do_osd_op_pre_copy_from (loglevel: TRACE_DEBUG_LINE (13))
> (type: tracepoint)
>         osd:do_osd_op_pre_copy_get (loglevel: TRACE_DEBUG_LINE (13))
> (type: tracepoint)
>         ...
> 
> 
> *Actual Result*:
> 
>     UST events:
>     -------------
>     lttng_ust_lib:*
>     lttng_ust_tracelog:*
>     lttng_ust_statedump:*
> 
> 
> We would greatly appreciate any help  you could provide to resolve this problem.
> 

thanks,
kienan