Re: Memory Consumption High After Upgrading to 2.13 from 2.10

[Kienan Stewart via lttng-dev <lttng-dev@lists.lttng.org>]

Hi Lakshya,

what is the output of the following commands on both systems:

* nproc (32, I assume)
* cat /sys/devices/system/cpu/possible

On 3/10/25 10:02 AM, Gour DEV via lttng-dev wrote:
> Hello,
> 
> I have recently upgraded from lttng 2.10 to 2.13 as part of servers
> migration from buster to bookworm.
> 
> lttng 2.10.6 - KeKriek to lttng 2.13.9 - Nordicité
> 
> I see an increase in memory consumption upto 11Gig after my recent upgrade.
> 
> 
> In buster
> 
> root@localhost:~# top -b -n 1 | grep lttng
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> 4296 root 20 0 1003676 19616 4608 S 0.0 0.1 0:00.67 lttng-sessiond
> 4297 root 20 0 44260 800 0 S 0.0 0.0 0:00.00 lttng-runas
> 4643 root 20 0 5838512 10396 9204 S 0.0 0.0 0:09.24 lttng-consumerd
> 4646 root 20 0 48048 544 0 S 0.0 0.0 0:00.00 lttng-runas
> 
> 
> In bookworm
> In bookworm.
> root@edgecore-40XKE-j2-101-32:~# top -b -n 1 | grep lttng
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> 4824 root 20 0 1098824 13336 5980 S 11.8 0.0 0:03.46 lttng-s+
> 4825 root 20 0 48872 2188 1012 S 0.0 0.0 0:00.00 lttng-r+
> 4841 root 20 0 11.5g 11.0g 11.0g S 0.0 35.4 2:33.33 lttng-c+
> 4843 root 20 0 3680 1160 816 S 0.0 0.0 0:00.23 lttng-r+
> 
> 
> I have a python script which was used to enable channels, these were the
> parameters I was using
> large_channels_sz = 4000000
> medium_channels_sz = 200000
> small_channels_sz = 10000
> 
> num_large = 4
> num_medium = 4
> num_small = 4
> 
> def enable_channel(channels, session, subbuf_size, subbuf_num):
> for c in channels:
> call(['lttng', 'enable-channel', '-u', c, '-s', session, '--subbuf-size',
> str(subbuf_size), '--num-subbuf', str(subbuf_num),],
> stdout=devnull, stderr=subprocess.STDOUT)
> 
> enable_channel(large_channels, def_session, large_channels_sz, num_large)
> enable_channel(medium_channels, def_session, medium_channels_sz, num_medium)
> enable_channel(small_channels, def_session, small_channels_sz, num_small)
> 
> 
> 
> there are total 16 large channels and 6 medium channels and no small
> channels.
> 
> 
> The total size for all these session cominded turn out to be machine
> roughly 7GB on my 32 core machine, which is still less that the amount of
> memory used by the lttng-consumerd.
> 
> I only have default session configured and nothing else
> 
> root@edgecore-40XKE-j2-101-32:~# lttng list
> Available recording sessions:
> 1) default [active snapshot]
> 
> 
> 
> 
> I have seen that when only the channel are enable and no process which is
> using lttng is running then, I see normal memory consumption.
> 
> 
> root@edgecore-40XKE-j2-101-32:~# top -b -n 1 | grep lttng
> 
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> 4824 root 20 0 1098184 12732 5376 S 0.0 0.0 0:03.15 lttng-s+
> 4825 root 20 0 48872 2188 1012 S 0.0 0.0 0:00.00 lttng-r+
> 4841 root 20 0 536240 4284 2400 S 0.0 0.0 2:09.32 lttng-c+
> 4843 root 20 0 3680 1160 816 S 0.0 0.0 0:00.23 lttng-r+
> 
> 
> But, as soon as I run even an single process which is linked to lttng, and
> which is *not* using any lttng channel

With which user(s) are you running the instrumented applications?

> 
> liblttng-ust.so.1 => /lib/x86_64-linux-gnu/liblttng-ust.so.1 (0x
> 00007fa867ea3000)
> liblttng-ust-common.so.1 => /lib/x86_64-linux-gnu/liblttng-ust-common.so.1 (
> 0x00007fa86752d000)
> liblttng-ust-tracepoint.so.1 =>
> /lib/x86_64-linux-gnu/liblttng-ust-tracepoint.so.1
> (0x00007fa86750b000)
> 
> I see an very high memory consumption of 5.5g and when i run all the
> process then i see the total memory consumption reaching uptop 11G.
> 
> 
> Do you have any hints on why this could be happening,
> 
> Thank You.
> 

thanks,
kienan