From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from simark.ca by simark.ca with LMTP id SGZ1ExH5e2daOgUAWB0awg (envelope-from ) for ; Mon, 06 Jan 2025 10:38:57 -0500 Authentication-Results: simark.ca; dkim=pass (2048-bit key; unprotected) header.d=lists.lttng.org header.i=@lists.lttng.org header.a=rsa-sha256 header.s=default header.b=I/JDjHcp; dkim-atps=neutral Received: by simark.ca (Postfix, from userid 112) id 4BB301E097; Mon, 6 Jan 2025 10:38:57 -0500 (EST) X-Spam-Checker-Version: SpamAssassin 4.0.0 (2022-12-13) on simark.ca X-Spam-Level: X-Spam-Status: No, score=-3.0 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HTML_MESSAGE,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham autolearn_force=no version=4.0.0 Received: from lists.lttng.org (lists.lttng.org [158.69.130.29]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (prime256v1) server-digest SHA256) (No client certificate requested) by simark.ca (Postfix) with ESMTPS id BC1C81E05C for ; Mon, 6 Jan 2025 10:38:53 -0500 (EST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=lists.lttng.org; s=default; t=1736177933; bh=ZG5focMBh/o5fDDcGwJkHYDua7L/xD+NmdS1h+l2W2I=; h=Date:Subject:To:List-Id:List-Unsubscribe:List-Archive:List-Post: List-Help:List-Subscribe:From:Reply-To:From; b=I/JDjHcp+YLnAqsCLzx9+MXzF+oTIyMm9zHWQyAFRIuOVrYlqrMyNMVbqfRn0B36X CXA/jdraefMKGdS3yccLGqsjSLSo1othXEiuR6l8ZN7aVZULB03l7K5eW2k2OdwXiQ T5jdznbKrlRQNA/Z/mtxo/e1tNyAwNHHMZrrX3/k6DN99w/1AOSBrmoY8ul+0ID6Zm DUqNeZ9azAgrPIOmqNnIfa+cjwAQG7PdxbArbM5/v+mVZznGlXZsopu3uTMx496ONn d9NC9Ui7XewoS58IGTccgOUf/ZWTfEu51hNanrNwtEOZbXBODRg0lHJbUo2LVLbpwx /GmpVaZTi1Ypw== Received: from lists-lttng01.efficios.com (localhost [IPv6:::1]) by lists.lttng.org (Postfix) with ESMTP id 4YRdd93dZ3z1PF7; Mon, 6 Jan 2025 10:38:53 -0500 (EST) Received: from mail-il1-x135.google.com (mail-il1-x135.google.com [IPv6:2607:f8b0:4864:20::135]) by lists.lttng.org (Postfix) with ESMTPS id 4YPLbY5Gklz1nvn for ; Thu, 2 Jan 2025 17:14:33 -0500 (EST) Received: by mail-il1-x135.google.com with SMTP id e9e14a558f8ab-3ab37d98dd4so44814695ab.3 for ; Thu, 02 Jan 2025 14:14:33 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1735856072; x=1736460872; h=to:subject:message-id:date:from:mime-version:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=9KuZUhJnJQVwGyfy9th2zjHagFL6jhxy7yGGwBdBChM=; b=huOBjnrsxcnKiPZTk6rb6Xr0BOPaDsCxodDOCfVNM1PGFgv9MbNIM0XU20PBz57t4D xlgdWk53rjolipIZ+HWlptzT90iBEMGnOcq6aOGlRaN41x/APo7j63a5YSnVvyt9h0OU oyDGWddqiE+e3p6xet0TcHy5F2RvBU+wykF0XVtcHbwzvpKlI6XFF3eS7fTAkE4B1oFy rlKfAt1Et6UxM7XgnBuGv4oquGxefSaaAuQbsAtY1znXvyTL79PFH9NtjRD4vjfwLga7 U41/B+zJDfdanPuOQ/zUBG1GwPJRErvZ5A03+k3RYddsu0oQrz++Ac6YiUB0IWxBKZB4 6xuQ== X-Gm-Message-State: AOJu0Yz0MFI4dqBuzqeUp7BznA0bvHUgf1krvv62Qge0ahtseKlIr3pm DwRxlb7oxkQA7T/LoDKWCo6a3vVCBnKV8OU7QQ34NNyyw9AbZuUqNN1GkUhxklFoaox+3Xd/Grz jV9VjCIVc4AIxq7EBi5ozbCc6Acb9vZdH4Vn5jwrI+cKlK2p5MDck X-Gm-Gg: ASbGnctALYGdKLXmSkGLoNKfhBhkrvHrB4+BSmoCcG47zsLOuzi2P5jh6WclyZxlQT4 ilG27s9k3v2L5AzISeXiVuOIQUlFX4K+/Xzo7hDtR X-Google-Smtp-Source: AGHT+IEBqE7/mk7obF+ex68SPGMRtB02Yfakd1woe9wkXDxdgiTtkz79XTFgenHeWaZgyRD2R0QhU0X4Y9g7VjTsTpE= X-Received: by 2002:a05:6e02:260b:b0:3a7:819c:5129 with SMTP id e9e14a558f8ab-3c2d48a3256mr403524095ab.18.1735856072432; Thu, 02 Jan 2025 14:14:32 -0800 (PST) MIME-Version: 1.0 Date: Thu, 2 Jan 2025 23:14:16 +0100 Message-ID: Subject: Tracing Ceph with LTTng-UST issue To: lttng-dev@lists.lttng.org Content-Type: multipart/alternative; boundary="0000000000009437c1062ac07c95" X-Mailman-Approved-At: Mon, 06 Jan 2025 10:38:52 -0500 X-BeenThere: lttng-dev@lists.lttng.org X-Mailman-Version: 2.1.39 Precedence: list List-Id: LTTng development list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , From: Lina SADI via lttng-dev Reply-To: Lina SADI Errors-To: lttng-dev-bounces@lists.lttng.org Sender: "lttng-dev" --0000000000009437c1062ac07c95 Content-Type: text/plain; charset="UTF-8" Hello, We are writing to you regarding an issue we encountered while attempting to trace Ceph with LTTng-UST. Below are the steps we have followed so far: 1. *Compiling Ceph with LTTng Support*: - We modified the debian/rules file to enable LTTng support using the following flags: -DWITH_LTTNG=ON and -DWITH_OSD_INSTRUMENT_FUNCTIONS=ON. - The compilation process took approximately 7 hours on a virtual machine running Debian 12 with a kernel version 6.1.0-27-amd64, 8 GB of RAM, and 2 CPU cores. - The build steps can be found in our GitHub repository. 2. *Deploying a Ceph Cluster*: - We deployed the cluster manually (1MON,1MGR,2OSDs). Please find the script here . 3. *Attempting to Trace with LTTng-UST*: - We tried listing the tracepoints using the following command : # lttng list -u - This command did not return any Ceph-related tracepoints: UST events: ------------- NONE 4. *Preloading liblttng-ust-fork.so*: - We attempted to preload liblttng-ust-fork.so before starting the Ceph daemons (MON, OSD, MGR) as follows: # systemctl edit ceph-mon@debian - We added the following configuration: [Service] Environment="LD_PRELOAD=/usr/lib/x86_64-linux-gnu/liblttng-ust-fork.so - After restarting the daemons, we executed lttng list -u, which displayed tracepoints related to the libraries (lttng_ust_lib:*), trace logs (lttng_ust_tracelog:*), and state dump events ( lttng_ust_statedump:*), instead of the expected Ceph-related tracepoints. *Expected Result*: UST events: ------------- PID: 100859 - Name: /path/to/ceph-osd pg:queue_op (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint) osd:do_osd_op_post (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint) osd:do_osd_op_pre_unknown (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint) osd:do_osd_op_pre_copy_from (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint) osd:do_osd_op_pre_copy_get (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint) ... *Actual Result*: UST events: ------------- lttng_ust_lib:* lttng_ust_tracelog:* lttng_ust_statedump:* We would greatly appreciate any help you could provide to resolve this problem. --0000000000009437c1062ac07c95 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable

Hello,=C2=A0

We are writing to y= ou regarding an issue we encountered while attempting to trace Ceph with LT= Tng-UST. Below are the steps we have followed so far:

  1. Compiling Ceph with LTTng Support:

    =
    • We modified the=C2=A0debian/rules<= /code>=C2=A0file to enable LTTng support using the following flags:=C2=A0-DWITH_LTTNG=3DON=C2=A0and=C2=A0-DWITH_OSD_INSTRUMENT_FUNC= TIONS=3DON.
    • The compilation proce= ss took approximately 7 hours on a virtual machine running Debian 12 with a= kernel version 6.1.0-27-amd64, 8 GB of RAM, and 2 CPU cores.
    • The build steps can be found in our=C2=A0GitHub=C2=A0repository.

  2. Deploying a Ceph Clus= ter:

    • We deployed the cluster= manually (1MON,1MGR,2OSDs). Please find the script=C2=A0here.

  3. Attempting to Trace with LTTng-= UST:

    • We tried listing the tr= acepoints using the following command :
      • # lttng =
list -u
    • This c= ommand did not return any Ceph-related tracepoints:
      • UST events:
-------------
NONE

  • Preloading=C2=A0libl= ttng-ust-fork.so:

    • We attempted to preload=C2=A0liblttng-ust-fork.so=C2=A0befor= e starting the Ceph daemons (MON, OSD, MGR) as follows:
      • # systemctl edit ceph-mon@debian
    • We added the following= configuration:
      • [Servi=
ce]
Environment=3D"LD_PRELOAD=3D/usr/lib/x86_64-linux-gnu/liblttng-ust-for=
k.so
      • After restarting the daemons, we executed=C2=A0lttng list -u= , which displayed tracepoints related to the libraries (lttng_= ust_lib:*), trace logs (lttng_ust_tracelog:*), and stat= e dump events (lttng_ust_statedump:*), instead of the expected= Ceph-related tracepoints.

    Expected Result:

      UST events: =20
------------- =20
PID: 100859 - Name: /path/to/ceph-osd =20
    pg:queue_op (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint) =20
    osd:do_osd_op_post (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint)=
 =20
    osd:do_osd_op_pre_unknown (loglevel: TRACE_DEBUG_LINE (13)) (type: trac=
epoint) =20
    osd:do_osd_op_pre_copy_from (loglevel: TRACE_DEBUG_LINE (13)) (type: tr=
acepoint) =20
    osd:do_osd_op_pre_copy_get (loglevel: TRACE_DEBUG_LINE (13)) (type: tra=
cepoint) =20
    ...  
    Actual Result:
      UST events:
-------------
lttng_ust_lib:*
lttng_ust_tracelog:*
lttng_ust_statedump:*
    We would greatly appreciate any help=C2=A0 you =
could provide to resolve this problem.
    3D"" --0000000000009437c1062ac07c95--