From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from simark.ca by simark.ca with LMTP id on78DikS0GeH+QsAWB0awg (envelope-from ) for ; Tue, 11 Mar 2025 06:36:25 -0400 Authentication-Results: simark.ca; dkim=pass (2048-bit key; unprotected) header.d=lists.lttng.org header.i=@lists.lttng.org header.a=rsa-sha256 header.s=default header.b=SiPdHhEb; dkim-atps=neutral Received: by simark.ca (Postfix, from userid 112) id 2C9D91E105; Tue, 11 Mar 2025 06:36:25 -0400 (EDT) X-Spam-Checker-Version: SpamAssassin 4.0.0 (2022-12-13) on simark.ca X-Spam-Level: X-Spam-Status: No, score=-2.7 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_REPLYTO_END_DIGIT,HTML_MESSAGE, MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham autolearn_force=no version=4.0.0 Received: from lists.lttng.org (lists.lttng.org [158.69.130.29]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (prime256v1) server-digest SHA256) (No client certificate requested) by simark.ca (Postfix) with ESMTPS id 6D36E1E08E for ; Tue, 11 Mar 2025 06:36:23 -0400 (EDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=lists.lttng.org; s=default; t=1741689383; bh=iEflrpy6erBLNTSJqzjoLqlJQIPYlnv7S4UNGPp8lM4=; h=Date:Subject:To:List-Id:List-Unsubscribe:List-Archive:List-Post: List-Help:List-Subscribe:From:Reply-To:From; b=SiPdHhEbEUdwIgT3T7r4F5vzLkeXukqmPJvSzqfWRcaSWytosgYwlaS5xMecRqYgC Tdm9eddrjsIsDLxxqV4mcTyfZYTpc57Y7pzAD3MP7lumByq/4r+gl/HaV0Dm+5QlN7 gkFYqBMbzJF0DtLUnTnt/S42pefnWcMGa8NUe2rTlCjX1yyG1/iTCt8qmpZOpCley5 aibRb7xOnuo2zid1AqJaAO00LkKUSW++uGO0rp4b59jutLmPI4n+FQaMXcQDrS8/8i cRYFKvKgFffiOhv+CnwjLGOVKawy5i81xTgB5qf2JvTNw6huc0FLdrUWEUpkDCYJ+2 vVc13szOvspgg== Received: from lists-lttng01.efficios.com (localhost [IPv6:::1]) by lists.lttng.org (Postfix) with ESMTP id 4ZBqtb0Fctz203t; Tue, 11 Mar 2025 06:36:23 -0400 (EDT) Received: from mail-pj1-x1030.google.com (mail-pj1-x1030.google.com [IPv6:2607:f8b0:4864:20::1030]) by lists.lttng.org (Postfix) with ESMTPS id 4ZBqtZ3pkpz206J for ; Tue, 11 Mar 2025 06:36:22 -0400 (EDT) Received: by mail-pj1-x1030.google.com with SMTP id 98e67ed59e1d1-2fec13a4067so8135031a91.2 for ; Tue, 11 Mar 2025 03:36:22 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1741689381; x=1742294181; h=to:subject:message-id:date:from:mime-version:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=I4T/4XddBYrwT7DeSKef2eTKEbd9NWR8Fqj44qe4aRc=; b=JxNVbXKWVStoiVyLIqCH7YQPvK9kUzVZ4249aINd2OfoqXypDOAiPpRWQDJ1ATRjub hZ9e3Mr+dpMZvSe18QmdjUV4VfDNlsz8Tddc27RbyIohtVKxShmBX1rm+2uJMNoICA9+ JogLRn+U4bsmkhNfMAo+/rUPsfYMPk30EBVjtRYCln71aaWqgeN+frdBAZz5m/YP4Btu LRUVQDpefA6KRL036ajn2cg2vhLhlIttfGVGj9z/OutsyeX5lZjaTi7ye1Yad8u3Dkdi YfNRxNR0PRj0ECpo+jz5b6SiKO3FINYY57lWmE8QrdiljyUwiQM6tasM5sM6fCzkbwhK sgDA== X-Gm-Message-State: AOJu0YzH4+78QHJ/9oSiTe8wUtFwjK1Hnhs7sJq5+R8voknI7WJtSRUu LktJNraxPnPsTESlyPGDuL0FevLj1fTqw0yCmm4nMI4H/4G+lJJcOL21VDLk7kkpVi9Z+z3CkJ4 6mebRVMX1J+96wtySRdlfErzQoB39HvyY8dw= X-Gm-Gg: ASbGncvF2MlmcoBJyj+FP2d5JttQqlpRDKfPzkT0YuuZSpMFvHot25Yl0FdSh1Yzukh AFfp6/Uz/OAhB9PA7jl4NUbtF8buy1GePW57r5YyyYj1qmZXJUm6SUxwMXyXBDAxdAlJZx/ay8z zDQAhQ6mBiyVZi+PicYQNaDEBdXg== X-Google-Smtp-Source: AGHT+IH9hsUNLH4zyh+5l5woqoBW5bGOElciG0u1zUNJsDJ9Pcqb4+vPwGmZ8Sg5Ep3zgY+PuW/eqcfWfLP1xXYTsYc= X-Received: by 2002:a17:90a:c886:b0:2fe:68a5:d84b with SMTP id 98e67ed59e1d1-300ff0a4374mr3910898a91.1.1741689381103; Tue, 11 Mar 2025 03:36:21 -0700 (PDT) MIME-Version: 1.0 Date: Tue, 11 Mar 2025 16:05:45 +0530 X-Gm-Features: AQ5f1JqnfLyCeoRAWRhshVH2YnIspTC7cUIiekZpsrTMcrXo40eQY9LGn1vfYAI Message-ID: Subject: Memory Consumption High After Upgrading to 2.13 from 2.10 To: lttng-dev@lists.lttng.org Content-Type: multipart/alternative; boundary="000000000000deb5ad06300ea894" X-BeenThere: lttng-dev@lists.lttng.org X-Mailman-Version: 2.1.39 Precedence: list List-Id: LTTng development list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , From: Gour DEV via lttng-dev Reply-To: Gour DEV Errors-To: lttng-dev-bounces@lists.lttng.org Sender: "lttng-dev" --000000000000deb5ad06300ea894 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hello, I have recently upgraded from lttng 2.10 to 2.13 as part of servers migration from buster to bookworm. lttng 2.10.6 - KeKriek to lttng 2.13.9 - Nordicit=C3=A9 I see an increase in memory consumption upto 11Gig after my recent upgrade. In buster root@localhost:~# top -b -n 1 | grep lttng PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 4296 root 20 0 1003676 19616 4608 S 0.0 0.1 0:00.67 lttng-sessiond 4297 root 20 0 44260 800 0 S 0.0 0.0 0:00.00 lttng-runas 4643 root 20 0 5838512 10396 9204 S 0.0 0.0 0:09.24 lttng-consumerd 4646 root 20 0 48048 544 0 S 0.0 0.0 0:00.00 lttng-runas In bookworm In bookworm. root@edgecore-40XKE-j2-101-32:~# top -b -n 1 | grep lttng PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 4824 root 20 0 1098824 13336 5980 S 11.8 0.0 0:03.46 lttng-s+ 4825 root 20 0 48872 2188 1012 S 0.0 0.0 0:00.00 lttng-r+ 4841 root 20 0 11.5g 11.0g 11.0g S 0.0 35.4 2:33.33 lttng-c+ 4843 root 20 0 3680 1160 816 S 0.0 0.0 0:00.23 lttng-r+ I have a python script which was used to enable channels, these were the parameters I was using large_channels_sz =3D 4000000 medium_channels_sz =3D 200000 small_channels_sz =3D 10000 num_large =3D 4 num_medium =3D 4 num_small =3D 4 def enable_channel(channels, session, subbuf_size, subbuf_num): for c in channels: call(['lttng', 'enable-channel', '-u', c, '-s', session, '--subbuf-size', str(subbuf_size), '--num-subbuf', str(subbuf_num),], stdout=3Ddevnull, stderr=3Dsubprocess.STDOUT) enable_channel(large_channels, def_session, large_channels_sz, num_large) enable_channel(medium_channels, def_session, medium_channels_sz, num_medium= ) enable_channel(small_channels, def_session, small_channels_sz, num_small) there are total 16 large channels and 6 medium channels and no small channels. The total size for all these session cominded turn out to be machine roughly 7GB on my 32 core machine, which is still less that the amount of memory used by the lttng-consumerd. I only have default session configured and nothing else root@edgecore-40XKE-j2-101-32:~# lttng list Available recording sessions: 1) default [active snapshot] I have seen that when only the channel are enable and no process which is using lttng is running then, I see normal memory consumption. root@edgecore-40XKE-j2-101-32:~# top -b -n 1 | grep lttng PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 4824 root 20 0 1098184 12732 5376 S 0.0 0.0 0:03.15 lttng-s+ 4825 root 20 0 48872 2188 1012 S 0.0 0.0 0:00.00 lttng-r+ 4841 root 20 0 536240 4284 2400 S 0.0 0.0 2:09.32 lttng-c+ 4843 root 20 0 3680 1160 816 S 0.0 0.0 0:00.23 lttng-r+ But, as soon as I run even an single process which is linked to lttng, and which is *not* using any lttng channel liblttng-ust.so.1 =3D> /lib/x86_64-linux-gnu/liblttng-ust.so.1 (0x 00007fa867ea3000) liblttng-ust-common.so.1 =3D> /lib/x86_64-linux-gnu/liblttng-ust-common.so.= 1 ( 0x00007fa86752d000) liblttng-ust-tracepoint.so.1 =3D> /lib/x86_64-linux-gnu/liblttng-ust-tracepoint.so.1 (0x00007fa86750b000) I see an very high memory consumption of 5.5g and when i run all the process then i see the total memory consumption reaching uptop 11G. Do you have any hints on why this could be happening, Thank You. --000000000000deb5ad06300ea894 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hello,

I have recently upgraded fro= m lttng 2.10 to 2.13 as part of servers migration from buster to bookworm.<= br>
lttng 2.10.6 - KeKriek to lttng 2.13.9 - Nordicit=C3=A9
I see an increase in memory consumption upto 11Gig after m= y recent upgrade.


In buster

root@localhost:~# top -b -n 1 | grep lttng
<= div style=3D"color:rgb(204,204,204);background-color:rgb(31,31,31);font-fam= ily:"JetBrainsMono Nerd Font",Menlo,Monaco,"Courier New"= ;,monospace;font-weight:normal;font-size:12px;line-height:18px;white-space:= pre-wrap">
PID USER PR N= I VIRT RES SHR S %<= /span>CPU %MEM = TIME+ COMMAND
429= 6 root 20 0 10036= 76 19616 = 4608 S 0.0= 0.1 0:00.67<= span style=3D"color:rgb(204,204,204)"> lttng-sessiond
4297 = root 20 0 44260= 800 0 S 0.0 = 0.0 0:00.00 lttng-runas
4643 root 20 0= 5838512 10= 396 9204 S= 0.0 0.0= 0:= 09.24 lttng-= consumerd
4646 root 20 0 48048 544 0 S 0.0 0.0 0:00.00= lttng-runas=


In bookworm
In bookworm.
= root@edgecore-40XKE-j2-101-= 32:~# top -b = -n 1 | grep lttng
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
4824 root 20 0 1098824 13336 5980 S 11.8 0.0 0:03.46 lt= tng-s+
4825= root 20 0 48872 2188 1= 012 S 0.0 = 0.0 0:00.00 lttng-r+
4841 root = 20 0<= /span> 11.5g= 11.0g 11.0g S 0.0 35.4 = 2:33.33 lttng-c<= span style=3D"color:rgb(212,212,212)">+
4843<= /span> root 20 0 368= 0 1160 = 816 S 0.0<= /span> 0.0 0:00.23 lttng-r+


I have = a python script which was used to enable channels, these were the parameter= s I was using
large_c= hannels_sz =3D 4000000
medium_c= hannels_sz =3D 200000
small_cha= nnels_sz =3D 10000

num_larg= e =3D 4
num_medium =3D 4
<= span style=3D"color:rgb(204,204,204)">num_small =3D 4

def enable_channel(channels, session, subbuf_si= ze, subbuf_num):
for c in channels:
call(['lttng', 'enable-channel'<= span style=3D"color:rgb(204,204,204)">, '-u', c, <= /span>'-s', session, '--subbuf-size',
= str(subbuf_size), '--num-subbuf', = str(subbuf_num),],
= stdout=3Ddevnull, stderr=3Dsubprocess.STDOUT)

enable_channel(large_channe= ls, def_session, large_channels_sz, num_large)
enable_channel(medium_channels, def_session, m= edium_channels_sz, num_medium)
enable_channel(small_channels, def_session, small_channels_sz,= num_small)



there = are total 16 large channels and 6 medium channels and no small channels.


The total size for all these session cominded turn out to be machine=20 roughly 7GB on my 32 core machine, which is still less that the amount=20 of memory used by the lttng-consumerd.

I only have defaul= t session configured and nothing else

root= @edgecore-40= XKE-j2-101-32:~# lttng list
Available recording sessions= :
1) default [active snapshot]



I have seen that when only the channel are enable and no process which is using lttng is running then, I see normal memory consumption.


root@edgecore-40XKE-j2-101-32:~# top -b -n 1 | grep lttng

PID USER PR NI VIRT = RES SHR S %CPU %MEM TIME= + COMMAND
4824 root 20 0 1098184 12732 5376 S 0.0 0.0 0= :03.15 lttng= -s+
4825 root 20 0 48872 2188 1012 S 0.0 0.0 0:00.00 lttng-r+
= 4841 root 20 0= 536240 4= 284 2400 S= 0.0 0.0= 2:= 09.32 lttng-= c+
4843 root = 20 0 3680 1160 816 S 0.0 0.0 0:00.23 lttng-r+


But, as soon as I run even an s= ingle process which is linked to lttng, and which is not using any l= ttng channel

libl= ttng-ust.so.1 =3D> /= lib/x86_64-<= /span>linux-gnu/liblttng-ust.so.1 (0x00007fa= 867ea3000)
<= span style=3D"color:rgb(204,204,204)">liblttng-ust<= span style=3D"color:rgb(212,212,212)">-common.so.1 =3D>= /lib= /x86_64-linux-gnu/liblttng-ust-common.so.1 (0x00007fa86752d000)
liblttng-ust-tracepoint.so.1 =3D> /lib/x86_64-linux-gnu/liblttng-ust-tracep= oint.so.1 (0x00007fa86750b000)

I see an very high memory consumption of 5.5g and when i run all the=20 process then i see the total memory consumption reaching uptop 11G.


Do you have any hints on why this could be= happening,

Thank You.

--000000000000deb5ad06300ea894--