From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from simark.ca by simark.ca with LMTP id GGwuEeD0zmdpEgsAWB0awg (envelope-from ) for ; Mon, 10 Mar 2025 10:19:12 -0400 Authentication-Results: simark.ca; dkim=pass (2048-bit key; unprotected) header.d=lists.lttng.org header.i=@lists.lttng.org header.a=rsa-sha256 header.s=default header.b=CrY2375O; dkim-atps=neutral Received: by simark.ca (Postfix, from userid 112) id 3E0691E105; Mon, 10 Mar 2025 10:19:12 -0400 (EDT) X-Spam-Checker-Version: SpamAssassin 4.0.0 (2022-12-13) on simark.ca X-Spam-Level: X-Spam-Status: No, score=-2.7 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_REPLYTO_END_DIGIT,HTML_MESSAGE, MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham autolearn_force=no version=4.0.0 Received: from lists.lttng.org (lists.lttng.org [158.69.130.29]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (prime256v1)) (No client certificate requested) by simark.ca (Postfix) with ESMTPS id 520031E08E for ; Mon, 10 Mar 2025 10:19:11 -0400 (EDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=lists.lttng.org; s=default; t=1741616351; bh=VZsEXV+WVqgvopF9P56FFB6Wzi/RHO0dPqZBZTSVLi8=; h=Date:Subject:To:List-Id:List-Unsubscribe:List-Archive:List-Post: List-Help:List-Subscribe:From:Reply-To:From; b=CrY2375OC1Q1yRyOWKNG4ZZy9T+hG4QZwx2BVqZcVI1zaBrquC4197SmwQtP1dsgy av6Kr4G9BEACCZvEPclZ2K87aQdF9MxcQ3fUC6nNAa5M4l1UN4CoDsUrKmBJbS4qvP 9UuAbM7WwRLoX2ubANbABhXGiTDKH1inQR5dzQkVRxGf90bWjFSiW0Cz8INDWA6wc2 J+i9xztRbJRbunDs7EdBNPpe4BxWq/YxkHQUCgFB3/Jacl1tX7CvuJ5FKCCj511MRv 5DLR+Zk5SkV93r6LgXWkKM0CNaZN7fkQmRCf+nyjHz43lRA8JXUOs4KB9uimOz5hO7 eRsBBGk/+E7Nw== Received: from lists-lttng01.efficios.com (localhost [IPv6:::1]) by lists.lttng.org (Postfix) with ESMTP id 4ZBJt715K0z1kgJ; Mon, 10 Mar 2025 10:19:11 -0400 (EDT) Received: from mail-pj1-x1034.google.com (mail-pj1-x1034.google.com [IPv6:2607:f8b0:4864:20::1034]) by lists.lttng.org (Postfix) with ESMTPS id 4ZBJWQ5QL7z1SSv for ; Mon, 10 Mar 2025 10:02:58 -0400 (EDT) Received: by mail-pj1-x1034.google.com with SMTP id 98e67ed59e1d1-2ff80290e44so6224889a91.0 for ; Mon, 10 Mar 2025 07:02:58 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1741615377; x=1742220177; h=to:subject:message-id:date:from:mime-version:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=V5GZYP0Yb4ODsBio42Gf0Xj/UF395Ey+HkNYlyv/9YE=; b=YQjwwk4Hve1wwYqRFsWCWg9FXsfFKMGUibJpqzIyAoi9LfNubXY9McAuAE7oO6o72F rcgDOGve48SyPOMDc4pbhgOrjdgTxe5i7LKhhNa1YHfC3d0UQdRH3W3Wd4mu0doZo+x5 lbb5eN9KRJOAlPLY2pxCo7aI2Fmux9/yMLNelCyeh4IA/kyFH/ntLbyLGLu9sl1TQXHu siqJyUQk/gEGRW362QWr1H8xZwr08LEdCkMJGJFf9K9uf7F9VJelZg8Jd0NVnxsRRCye tAe07GcGLnrKQ9LZ+KYCZz41gxW+yRPwoi0iPOKcjWr08GhDRh6zyEv+97XMiK0XgGBn AF1Q== X-Gm-Message-State: AOJu0Yzmr+E2zDhfnNepa/50OILVXDdX2il+J7JnfMsMMqPnKhZW0maN z/SBcPVKDwCF3ZcZ6PHiqpoyEimUL0bpWXHhHgPteTQ2pfGbYg8+LchnHxvmnWXWLXu+OVBNWyo 0XB4mUjQYfsMRGgBk3n/U7IWtlyC/5yNMDIE= X-Gm-Gg: ASbGnctqLRMWr1WVkgRwtVPGCp/vgJI6uiNPhjVnRmax6Ou/PSGAsmtTj5H2ObIaXcJ B1N7ctEkAwNPK4R7IHtm7Mr05JFeT9xK8HDu7Bx0tHRzadV+J3FfGkNGFFLRjmgsSjIdegiGYGm pB5nbbXfaps2Gl3TCGHnINciQ+HsA= X-Google-Smtp-Source: AGHT+IFdhvCCX7yaQePQ1HmDKdHNZZcbPcgf3gYCCMcqjq7vDb2DROjuv1uPEZWMudNiy+HfImXvV5mkSUkZgyg+sfk= X-Received: by 2002:a17:90b:4f8f:b0:2ee:8430:b847 with SMTP id 98e67ed59e1d1-2ff7ce7b4b4mr21115300a91.6.1741615377152; Mon, 10 Mar 2025 07:02:57 -0700 (PDT) MIME-Version: 1.0 Date: Mon, 10 Mar 2025 19:32:18 +0530 X-Gm-Features: AQ5f1JrGcB9xFz8i6-CpDoM6Q58NQPIr--vXRUR_0oGtnDsDKwSv8X0602v8AeM Message-ID: Subject: Memory Consumption High After Upgrading to 2.13 from 2.10 To: lttng-dev@lists.lttng.org Content-Type: multipart/alternative; boundary="000000000000e40382062ffd6d09" X-Mailman-Approved-At: Mon, 10 Mar 2025 10:19:03 -0400 X-BeenThere: lttng-dev@lists.lttng.org X-Mailman-Version: 2.1.39 Precedence: list List-Id: LTTng development list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , From: Gour DEV via lttng-dev Reply-To: Gour DEV Errors-To: lttng-dev-bounces@lists.lttng.org Sender: "lttng-dev" --000000000000e40382062ffd6d09 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hello, I have recently upgraded from lttng 2.10 to 2.13 as part of servers migration from buster to bookworm. lttng 2.10.6 - KeKriek to lttng 2.13.9 - Nordicit=C3=A9 I see an increase in memory consumption upto 11Gig after my recent upgrade. In buster root@localhost:~# top -b -n 1 | grep lttng PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 4296 root 20 0 1003676 19616 4608 S 0.0 0.1 0:00.67 lttng-sessiond 4297 root 20 0 44260 800 0 S 0.0 0.0 0:00.00 lttng-runas 4643 root 20 0 5838512 10396 9204 S 0.0 0.0 0:09.24 lttng-consumerd 4646 root 20 0 48048 544 0 S 0.0 0.0 0:00.00 lttng-runas In bookworm In bookworm. root@edgecore-40XKE-j2-101-32:~# top -b -n 1 | grep lttng PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 4824 root 20 0 1098824 13336 5980 S 11.8 0.0 0:03.46 lttng-s+ 4825 root 20 0 48872 2188 1012 S 0.0 0.0 0:00.00 lttng-r+ 4841 root 20 0 11.5g 11.0g 11.0g S 0.0 35.4 2:33.33 lttng-c+ 4843 root 20 0 3680 1160 816 S 0.0 0.0 0:00.23 lttng-r+ I have a python script which was used to enable channels, these were the parameters I was using large_channels_sz =3D 4000000 medium_channels_sz =3D 200000 small_channels_sz =3D 10000 num_large =3D 4 num_medium =3D 4 num_small =3D 4 def enable_channel(channels, session, subbuf_size, subbuf_num): for c in channels: call(['lttng', 'enable-channel', '-u', c, '-s', session, '--subbuf-size', str(subbuf_size), '--num-subbuf', str(subbuf_num),], stdout=3Ddevnull, stderr=3Dsubprocess.STDOUT) enable_channel(large_channels, def_session, large_channels_sz, num_large) enable_channel(medium_channels, def_session, medium_channels_sz, num_medium= ) enable_channel(small_channels, def_session, small_channels_sz, num_small) there are total 16 large channels and 6 medium channels and no small channels. The total size for all these session cominded turn out to be machine roughly 7GB on my 32 core machine, which is still less that the amount of memory used by the lttng-consumerd. I only have default session configured and nothing else root@edgecore-40XKE-j2-101-32:~# lttng list Available recording sessions: 1) default [active snapshot] I have seen that when only the channel are enable and no process which is using lttng is running then, I see normal memory consumption. root@edgecore-40XKE-j2-101-32:~# top -b -n 1 | grep lttng PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 4824 root 20 0 1098184 12732 5376 S 0.0 0.0 0:03.15 lttng-s+ 4825 root 20 0 48872 2188 1012 S 0.0 0.0 0:00.00 lttng-r+ 4841 root 20 0 536240 4284 2400 S 0.0 0.0 2:09.32 lttng-c+ 4843 root 20 0 3680 1160 816 S 0.0 0.0 0:00.23 lttng-r+ But, as soon as I run even an single process which is linked to lttng, and which is *not* using any lttng channel liblttng-ust.so.1 =3D> /lib/x86_64-linux-gnu/liblttng-ust.so.1 (0x 00007fa867ea3000) liblttng-ust-common.so.1 =3D> /lib/x86_64-linux-gnu/liblttng-ust-common.so.= 1 ( 0x00007fa86752d000) liblttng-ust-tracepoint.so.1 =3D> /lib/x86_64-linux-gnu/liblttng-ust-tracepoint.so.1 (0x00007fa86750b000) I see an very high memory consumption of 5.5g and when i run all the process then i see the total memory consumption reaching uptop 11G. Do you have any hints on why this could be happening, Thank You. --000000000000e40382062ffd6d09 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hello,

I have recently upgraded fro= m lttng 2.10 to 2.13 as part of servers migration from buster to bookworm.<= br>
lttng 2.10.6 - KeKriek to lttng 2.13.9 - Nordicit=C3=A9
I see an increase in memory consumption upto 11Gig after m= y recent upgrade.


In buster

root@localhost:~# top -b -n 1 | grep lttng
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
= 4296 root 20 0 1003676 19616= 4608 S = 0.0 0.1 0:00.67 lttng-sessiond
4297 root 20 <= /span>0 44260 800 0 S 0.0 0.0 0:00.00 lttng-runas
<= div> 4643 root = 20 0 5838512 = 10396 9204= S 0.0 <= span style=3D"color:rgb(181,206,168)">0.0 0:09.24 lttng-consumerd
4646 root = 20 0 48048 544 0 S 0.0 0.0 0:00.00 lttng-runas


<= /div>
In bookworm
In bookworm.
root<= /span>@edgecore-40XKE-j2-101-32:~# top -b -n 1 | grep lttng
PID USER PR NI VIRT RES = SHR S %CPU %MEM TIME+ COMMAND
4824 root 20 0 1098824 13336 5980 S 11.8 0.0 0= :03.46 lttng= -s+
4825 root 20 0 48872 2188 1012 S 0.0 0.0 0:00.00 lttng-r+
= 4841 root 20 0= 11.5g 11.0g 11.0g S 0.0 35.4 2:33.33 lt= tng-c+
4843= root 20 0 3680 1160 816 S 0.0 0.0 0= :00.23 lttng= -r+

<= /div>

I have a python script which was used to enable ch= annels, these were the parameters I was using
large_channels_sz =3D 4000000
medium_channels_sz =3D 200000
small_channels_sz =3D 10000

num_large = =3D 4
num_medium =3D 4
num_small = =3D 4
def enable_channel(channels, session, subbuf_size, subbuf_num):
for c in channels:
call(['lttng', 'enabl= e-channel', '-u', c, '-s&= #39;, session, '--subbuf-size',
str(subbuf_size), '--num-subbuf', str(subbuf_num),],
stdout=3Ddevnull, stderr=3Dsubprocess.STDOUT)<= br>
enable_cha= nnel(large_channels, def_session, large_channels_sz, num_large)
enable_channel(medium_channel= s, def_session, medium_channels_sz, num_medium)
enable_channel(small_channels, def_session, s= mall_channels_sz, num_small)



=
there are total 16 large channels and 6 medium channels and no s= mall channels.


The total size for = all these session cominded turn out to be machine roughly 7GB on my 32 core= machine, which is still less that the amount of memory used by the lttng-c= onsumerd.

I only have default session configured and noth= ing else

root@edgecore-= 40XKE-j2-101-32:~# lttng list
Available recording sessions:
1) default [active snapshot]




I have seen that when = only the channel are enable and no process which is using lttng is running = then, I see normal memory consumption.


root= @edgecore-40= XKE-j2-101-32:~# top -b -n 1 | grep = lttng

PID USER PR NI VIRT RES SHR S %= CPU %MEM TIME+ COMMAND
<= /div>
4824 r= oot 20 0 1098184 = 12732 537= 6 S 0.0 = 0.0 0:03.15 lttng-s+
<= div> 4825 root = 20 0 48872 2188 1012 S 0.0 0.0 0:00.00 lttng-r+
4841 root 20 0 536240 4284 2400 S 0.0 0.0 2:09.32 lttng-c+
4843 root 20 0 3680 1160 816 S 0.0 0.0 0:00.23= lttng-r+


But, as soon as I run even an single process which is linked to lttng= , and which is not using any lttng channel

liblttng-ust.so.1 =3D> /lib/x86_64-linux-gnu/<= /span>liblttng-ust.= so.1 (0x00007fa867ea3000)
libltt= ng-ust-common.so.1 =3D> /lib/x86_64-linux-gnu/liblttng-ust-common.so.1 (0x<= span style=3D"color:rgb(181,206,168)">00007fa86752d000)
liblttng-ust-tracepoint.so.1 =3D> /= lib/x86_64-linux-gnu/liblttng-ust-tracepoint.so.1 (0x00007fa86750b00= 0)

I see an very high memory consumption of 5.5g and when i run all the= process then i see the total memory consumption reaching uptop 11G.
<= div>

Do you have any hints on why this could b= e happening,

Thank You.
--000000000000e40382062ffd6d09--