Memory Consumption High After Upgrading to 2.13 from 2.10

[Gour DEV via lttng-dev <lttng-dev@lists.lttng.org>]

[-- Attachment #1: Type: text/plain, Size: 3299 bytes --]

Hello,

I have recently upgraded from lttng 2.10 to 2.13 as part of servers
migration from buster to bookworm.

lttng 2.10.6 - KeKriek to lttng 2.13.9 - Nordicité

I see an increase in memory consumption upto 11Gig after my recent upgrade.


In buster

root@localhost:~# top -b -n 1 | grep lttng
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
4296 root 20 0 1003676 19616 4608 S 0.0 0.1 0:00.67 lttng-sessiond
4297 root 20 0 44260 800 0 S 0.0 0.0 0:00.00 lttng-runas
4643 root 20 0 5838512 10396 9204 S 0.0 0.0 0:09.24 lttng-consumerd
4646 root 20 0 48048 544 0 S 0.0 0.0 0:00.00 lttng-runas


In bookworm
In bookworm.
root@edgecore-40XKE-j2-101-32:~# top -b -n 1 | grep lttng
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
4824 root 20 0 1098824 13336 5980 S 11.8 0.0 0:03.46 lttng-s+
4825 root 20 0 48872 2188 1012 S 0.0 0.0 0:00.00 lttng-r+
4841 root 20 0 11.5g 11.0g 11.0g S 0.0 35.4 2:33.33 lttng-c+
4843 root 20 0 3680 1160 816 S 0.0 0.0 0:00.23 lttng-r+


I have a python script which was used to enable channels, these were the
parameters I was using
large_channels_sz = 4000000
medium_channels_sz = 200000
small_channels_sz = 10000

num_large = 4
num_medium = 4
num_small = 4

def enable_channel(channels, session, subbuf_size, subbuf_num):
for c in channels:
call(['lttng', 'enable-channel', '-u', c, '-s', session, '--subbuf-size',
str(subbuf_size), '--num-subbuf', str(subbuf_num),],
stdout=devnull, stderr=subprocess.STDOUT)

enable_channel(large_channels, def_session, large_channels_sz, num_large)
enable_channel(medium_channels, def_session, medium_channels_sz, num_medium)
enable_channel(small_channels, def_session, small_channels_sz, num_small)



there are total 16 large channels and 6 medium channels and no small
channels.


The total size for all these session cominded turn out to be machine
roughly 7GB on my 32 core machine, which is still less that the amount of
memory used by the lttng-consumerd.

I only have default session configured and nothing else

root@edgecore-40XKE-j2-101-32:~# lttng list
Available recording sessions:
1) default [active snapshot]




I have seen that when only the channel are enable and no process which is
using lttng is running then, I see normal memory consumption.


root@edgecore-40XKE-j2-101-32:~# top -b -n 1 | grep lttng

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
4824 root 20 0 1098184 12732 5376 S 0.0 0.0 0:03.15 lttng-s+
4825 root 20 0 48872 2188 1012 S 0.0 0.0 0:00.00 lttng-r+
4841 root 20 0 536240 4284 2400 S 0.0 0.0 2:09.32 lttng-c+
4843 root 20 0 3680 1160 816 S 0.0 0.0 0:00.23 lttng-r+


But, as soon as I run even an single process which is linked to lttng, and
which is *not* using any lttng channel

liblttng-ust.so.1 => /lib/x86_64-linux-gnu/liblttng-ust.so.1 (0x
00007fa867ea3000)
liblttng-ust-common.so.1 => /lib/x86_64-linux-gnu/liblttng-ust-common.so.1 (
0x00007fa86752d000)
liblttng-ust-tracepoint.so.1 =>
/lib/x86_64-linux-gnu/liblttng-ust-tracepoint.so.1
(0x00007fa86750b000)

I see an very high memory consumption of 5.5g and when i run all the
process then i see the total memory consumption reaching uptop 11G.


Do you have any hints on why this could be happening,

Thank You.

[-- Attachment #2: Type: text/html, Size: 29023 bytes --]