Memory Consumption High After Upgrading to 2.13 from 2.10

Memory Consumption High After Upgrading to 2.13 from 2.10

[Gour DEV via lttng-dev]

[-- Attachment #1: Type: text/plain, Size: 3299 bytes --]

Hello,

I have recently upgraded from lttng 2.10 to 2.13 as part of servers
migration from buster to bookworm.

lttng 2.10.6 - KeKriek to lttng 2.13.9 - Nordicité

I see an increase in memory consumption upto 11Gig after my recent upgrade.


In buster

root@localhost:~# top -b -n 1 | grep lttng
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
4296 root 20 0 1003676 19616 4608 S 0.0 0.1 0:00.67 lttng-sessiond
4297 root 20 0 44260 800 0 S 0.0 0.0 0:00.00 lttng-runas
4643 root 20 0 5838512 10396 9204 S 0.0 0.0 0:09.24 lttng-consumerd
4646 root 20 0 48048 544 0 S 0.0 0.0 0:00.00 lttng-runas


In bookworm
In bookworm.
root@edgecore-40XKE-j2-101-32:~# top -b -n 1 | grep lttng
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
4824 root 20 0 1098824 13336 5980 S 11.8 0.0 0:03.46 lttng-s+
4825 root 20 0 48872 2188 1012 S 0.0 0.0 0:00.00 lttng-r+
4841 root 20 0 11.5g 11.0g 11.0g S 0.0 35.4 2:33.33 lttng-c+
4843 root 20 0 3680 1160 816 S 0.0 0.0 0:00.23 lttng-r+


I have a python script which was used to enable channels, these were the
parameters I was using
large_channels_sz = 4000000
medium_channels_sz = 200000
small_channels_sz = 10000

num_large = 4
num_medium = 4
num_small = 4

def enable_channel(channels, session, subbuf_size, subbuf_num):
for c in channels:
call(['lttng', 'enable-channel', '-u', c, '-s', session, '--subbuf-size',
str(subbuf_size), '--num-subbuf', str(subbuf_num),],
stdout=devnull, stderr=subprocess.STDOUT)

enable_channel(large_channels, def_session, large_channels_sz, num_large)
enable_channel(medium_channels, def_session, medium_channels_sz, num_medium)
enable_channel(small_channels, def_session, small_channels_sz, num_small)



there are total 16 large channels and 6 medium channels and no small
channels.


The total size for all these session cominded turn out to be machine
roughly 7GB on my 32 core machine, which is still less that the amount of
memory used by the lttng-consumerd.

I only have default session configured and nothing else

root@edgecore-40XKE-j2-101-32:~# lttng list
Available recording sessions:
1) default [active snapshot]




I have seen that when only the channel are enable and no process which is
using lttng is running then, I see normal memory consumption.


root@edgecore-40XKE-j2-101-32:~# top -b -n 1 | grep lttng

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
4824 root 20 0 1098184 12732 5376 S 0.0 0.0 0:03.15 lttng-s+
4825 root 20 0 48872 2188 1012 S 0.0 0.0 0:00.00 lttng-r+
4841 root 20 0 536240 4284 2400 S 0.0 0.0 2:09.32 lttng-c+
4843 root 20 0 3680 1160 816 S 0.0 0.0 0:00.23 lttng-r+


But, as soon as I run even an single process which is linked to lttng, and
which is *not* using any lttng channel

liblttng-ust.so.1 => /lib/x86_64-linux-gnu/liblttng-ust.so.1 (0x
00007fa867ea3000)
liblttng-ust-common.so.1 => /lib/x86_64-linux-gnu/liblttng-ust-common.so.1 (
0x00007fa86752d000)
liblttng-ust-tracepoint.so.1 =>
/lib/x86_64-linux-gnu/liblttng-ust-tracepoint.so.1
(0x00007fa86750b000)

I see an very high memory consumption of 5.5g and when i run all the
process then i see the total memory consumption reaching uptop 11G.


Do you have any hints on why this could be happening,

Thank You.

[-- Attachment #2: Type: text/html, Size: 29023 bytes --]

Re: Memory Consumption High After Upgrading to 2.13 from 2.10

[Kienan Stewart via lttng-dev]

Hi Lakshya,

what is the output of the following commands on both systems:

* nproc (32, I assume)
* cat /sys/devices/system/cpu/possible

also, when you are running instrumented applications which user(s) are 
you running applications as?

thanks,
kienan

On 3/11/25 6:35 AM, Gour DEV via lttng-dev wrote:
> Hello,
> 
> I have recently upgraded from lttng 2.10 to 2.13 as part of servers
> migration from buster to bookworm.
> 
> lttng 2.10.6 - KeKriek to lttng 2.13.9 - Nordicité
> 
> I see an increase in memory consumption upto 11Gig after my recent upgrade.
> 
> 
> In buster
> 
> root@localhost:~# top -b -n 1 | grep lttng
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> 4296 root 20 0 1003676 19616 4608 S 0.0 0.1 0:00.67 lttng-sessiond
> 4297 root 20 0 44260 800 0 S 0.0 0.0 0:00.00 lttng-runas
> 4643 root 20 0 5838512 10396 9204 S 0.0 0.0 0:09.24 lttng-consumerd
> 4646 root 20 0 48048 544 0 S 0.0 0.0 0:00.00 lttng-runas
> 
> 
> In bookworm
> In bookworm.
> root@edgecore-40XKE-j2-101-32:~# top -b -n 1 | grep lttng
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> 4824 root 20 0 1098824 13336 5980 S 11.8 0.0 0:03.46 lttng-s+
> 4825 root 20 0 48872 2188 1012 S 0.0 0.0 0:00.00 lttng-r+
> 4841 root 20 0 11.5g 11.0g 11.0g S 0.0 35.4 2:33.33 lttng-c+
> 4843 root 20 0 3680 1160 816 S 0.0 0.0 0:00.23 lttng-r+
> 
> 
> I have a python script which was used to enable channels, these were the
> parameters I was using
> large_channels_sz = 4000000
> medium_channels_sz = 200000
> small_channels_sz = 10000
> 
> num_large = 4
> num_medium = 4
> num_small = 4
> 
> def enable_channel(channels, session, subbuf_size, subbuf_num):
> for c in channels:
> call(['lttng', 'enable-channel', '-u', c, '-s', session, '--subbuf-size',
> str(subbuf_size), '--num-subbuf', str(subbuf_num),],
> stdout=devnull, stderr=subprocess.STDOUT)
> 
> enable_channel(large_channels, def_session, large_channels_sz, num_large)
> enable_channel(medium_channels, def_session, medium_channels_sz, num_medium)
> enable_channel(small_channels, def_session, small_channels_sz, num_small)
> 
> 
> 
> there are total 16 large channels and 6 medium channels and no small
> channels.
> 
> 
> The total size for all these session cominded turn out to be machine
> roughly 7GB on my 32 core machine, which is still less that the amount of
> memory used by the lttng-consumerd.
> 
> I only have default session configured and nothing else
> 
> root@edgecore-40XKE-j2-101-32:~# lttng list
> Available recording sessions:
> 1) default [active snapshot]
> 
> 
> 
> 
> I have seen that when only the channel are enable and no process which is
> using lttng is running then, I see normal memory consumption.
> 
> 
> root@edgecore-40XKE-j2-101-32:~# top -b -n 1 | grep lttng
> 
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> 4824 root 20 0 1098184 12732 5376 S 0.0 0.0 0:03.15 lttng-s+
> 4825 root 20 0 48872 2188 1012 S 0.0 0.0 0:00.00 lttng-r+
> 4841 root 20 0 536240 4284 2400 S 0.0 0.0 2:09.32 lttng-c+
> 4843 root 20 0 3680 1160 816 S 0.0 0.0 0:00.23 lttng-r+
> 
> 
> But, as soon as I run even an single process which is linked to lttng, and
> which is *not* using any lttng channel
> 
> liblttng-ust.so.1 => /lib/x86_64-linux-gnu/liblttng-ust.so.1 (0x
> 00007fa867ea3000)
> liblttng-ust-common.so.1 => /lib/x86_64-linux-gnu/liblttng-ust-common.so.1 (
> 0x00007fa86752d000)
> liblttng-ust-tracepoint.so.1 =>
> /lib/x86_64-linux-gnu/liblttng-ust-tracepoint.so.1
> (0x00007fa86750b000)
> 
> I see an very high memory consumption of 5.5g and when i run all the
> process then i see the total memory consumption reaching uptop 11G.
> 
> 
> Do you have any hints on why this could be happening,
> 
> Thank You.
> 

Re: Memory Consumption High After Upgrading to 2.13 from 2.10

[Kienan Stewart via lttng-dev]

Hi Lakshya,

On 3/11/25 12:25 PM, Gour DEV wrote:
 > Hi, Kienan
 >
 > here is the requested output
 >
 > root@localhost:~# top -b -n 1 | grep  lttng
 >     4841 root      20   0   11.5g  11.0g  11.0g S   5.9  35.4   8:39.93
 > lttng-c+
 >     4824 root      20   0 1098824  26456   5380 S   0.0   0.1   0:07.25
 > lttng-s+
 >     4825 root      20   0   48872   2188   1012 S   0.0   0.0   0:00.00
 > lttng-r+
 >     4843 root      20   0    3680   1160    816 S   0.0   0.0   0:00.23

This top output for `localhost` seems very different than the output for 
`localhost` in your previous message.


 > lttng-r+
 > root@localhost:~# nrpco
 > bash: nrpco: command not found
 > root@localhost:~# nproc
 > 16
 > root@localhost:~# cat /sys/devices/system/cpu/possible
 > 0-15
 >

You indicated the bookworm machine has 32 cores, this is showing 16. If 
you're comparing a 16 core machine to a 32 core machine, it is very 
normal that the memory usage is higher on the 32 core machine.

 >
 > Most of the process are running as asorcs user but some are running 
as root.

So you have two users with instrumented applications.


Given the discrepancies in the information provided I'm finding it a bit 
hard to understand what you're looking at.


In general, a channel's shared memory footprint can be estimated with[1]:

   (nSubbuf * subbufSize) * (nCPUs + 1 iff snapshot mode is enabled) * 
(nUIDs or nPIDs)

Note that the sub-buffer sizes you are using get rounded to the nearest 
larger power of 2. See [2].

thanks,
kienan

[1]: https://lttng.org/docs/v2.13/#doc-channel-buffering-schemes
[2]: 
https://lttng.org/man/1/lttng-enable-channel/v2.13/#doc-opt--subbuf-size

Re: Memory Consumption High After Upgrading to 2.13 from 2.10

[Gour DEV via lttng-dev]

[-- Attachment #1: Type: text/plain, Size: 3766 bytes --]

Hi, Kienan

I am attaching an screen recording of the behaviour I am seeing in this
mail. The behaviour is same irrespective of the device i use, sorry for
miscommunication in the npocs output (I assumed it was 32), but other than
that all outputs are same (except the hostname as there are multiple
devices with same lttng config but this memory cosumption is seen on all
the devices).

I had few question

1. Does lltng allocated all the memory it needs and mark it as dirty in ram
when any process which links/uses lttng-ust runs? (here i tried with one
process but it is same for any of my process)
2. (nSubbuf * subbufSize) * (nCPUs + 1 iff snapshot mode is enabled) *
(nUIDs or nPIDs)

How do we calculate uid in the system is it all uids in the system? is it
equal to `cat /etc/passwd | wc -l` ?

I will put my calculations according to the above estimate based on all the
channel i am creating

(4194304*4 + 262144*4 + 16384*4) * (16) * (30 if number user are equal to
`cat /etc/passwd | wc -l`)B = 7.998046875 GB approx [this is based on the
start_lttng.py please do correct me if am wrong here.]

But since there are only two users which uses lttng i think the correct
estimate would be
(4194304*4 + 262144*4 + 16384*4) * (16) * (2)B = 546MB

Please do correct me If I am wrong calculations here.

Now, there are a few things here, according to my output lttng is using 11G
which is much more higher than the what is configured.

I am attaching the lttng status and the file which is uses to create the
lttng sessions.



Thank You.


https://drive.google.com/file/d/1tS_ZWEsXDpHZXfWzZHXmWcT0igiIOIaa/view?usp=sharing
-- recording of the behaviour which is seen
https://drive.google.com/file/d/1PrU31oyEw1n9tKETlUtmNGO50s6ywx7p/view?usp=sharing
-- the file which is used to create lttng sessions


On Wed, Mar 12, 2025 at 12:25 AM Kienan Stewart <kstewart@efficios.com>
wrote:

> Hi Lakshya,
>
> On 3/11/25 12:25 PM, Gour DEV wrote:
>  > Hi, Kienan
>  >
>  > here is the requested output
>  >
>  > root@localhost:~# top -b -n 1 | grep  lttng
>  >     4841 root      20   0   11.5g  11.0g  11.0g S   5.9  35.4   8:39.93
>  > lttng-c+
>  >     4824 root      20   0 1098824  26456   5380 S   0.0   0.1   0:07.25
>  > lttng-s+
>  >     4825 root      20   0   48872   2188   1012 S   0.0   0.0   0:00.00
>  > lttng-r+
>  >     4843 root      20   0    3680   1160    816 S   0.0   0.0   0:00.23
>
> This top output for `localhost` seems very different than the output for
> `localhost` in your previous message.
>
>
>  > lttng-r+
>  > root@localhost:~# nrpco
>  > bash: nrpco: command not found
>  > root@localhost:~# nproc
>  > 16
>  > root@localhost:~# cat /sys/devices/system/cpu/possible
>  > 0-15
>  >
>
> You indicated the bookworm machine has 32 cores, this is showing 16. If
> you're comparing a 16 core machine to a 32 core machine, it is very
> normal that the memory usage is higher on the 32 core machine.
>
>  >
>  > Most of the process are running as asorcs user but some are running
> as root.
>
> So you have two users with instrumented applications.
>
>
> Given the discrepancies in the information provided I'm finding it a bit
> hard to understand what you're looking at.
>
>
> In general, a channel's shared memory footprint can be estimated with[1]:
>
>    (nSubbuf * subbufSize) * (nCPUs + 1 iff snapshot mode is enabled) *
> (nUIDs or nPIDs)
>
> Note that the sub-buffer sizes you are using get rounded to the nearest
> larger power of 2. See [2].
>
> thanks,
> kienan
>
> [1]: https://lttng.org/docs/v2.13/#doc-channel-buffering-schemes
> [2]:
> https://lttng.org/man/1/lttng-enable-channel/v2.13/#doc-opt--subbuf-size
>

[-- Attachment #2: Type: text/html, Size: 5365 bytes --]

Re: Memory Consumption High After Upgrading to 2.13 from 2.10

[Kienan Stewart via lttng-dev]

Hi Lakshya,

On 3/12/25 5:03 AM, Gour DEV wrote:
> Hi, Kienan
> 
> I am attaching an screen recording of the behaviour I am seeing in this
> mail. The behaviour is same irrespective of the device i use, sorry for
> miscommunication in the npocs output (I assumed it was 32), but other than
> that all outputs are same (except the hostname as there are multiple
> devices with same lttng config but this memory cosumption is seen on all
> the devices).
> 
> I had few question
> 
> 1. Does lltng allocated all the memory it needs and mark it as dirty in ram
> when any process which links/uses lttng-ust runs? (here i tried with one
> process but it is same for any of my process)

I believe the shared memory for per-CPU data structures is allocated 
when an instrumented application connects. There is no pre-allocation 
for each possible UID on the system.

You can run your instrumented applications with `LTTNG_UST_DEBUG=1` to 
see when the connection happens[1].

> 2. (nSubbuf * subbufSize) * (nCPUs + 1 iff snapshot mode is enabled) *
> (nUIDs or nPIDs)
> 
> How do we calculate uid in the system is it all uids in the system? is it
> equal to `cat /etc/passwd | wc -l` ?

nUIDs is the number of distinct UIDs running instrumented applications.

> 
> I will put my calculations according to the above estimate based on all the
> channel i am creating
> 
> (4194304*4 + 262144*4 + 16384*4) * (16) * (30 if number user are equal to
> `cat /etc/passwd | wc -l`)B = 7.998046875 GB approx [this is based on the
> start_lttng.py please do correct me if am wrong here.]
> 
> But since there are only two users which uses lttng i think the correct
> estimate would be
> (4194304*4 + 262144*4 + 16384*4) * (16) * (2)B = 546MB

The estimate I gave is per-channel.

small channel: (0.015625 MiB * 4) * (16 + 1) = 1.0625 MiB per-channel 
per-UID
medium channel: (0.250 MiB * 4) * (16 + 1) = 17.0 MiB per-channel per-UID
large channel: (4 MiB * 4) * (16 + 1) = 27 2MiB per-channel per-UID

Now, you said you have 0 small channels, 6 medium channels, and 16 large 
channels in your session. (Note: I see your script differs from these 
stated channel counts).

small: 0 * 1.0625 MiB = 0 MiB per-UID
medium: 6 * 17 MiB = 102 MiB per-UID
large: 16 * 272 MiB = 4352 MiB per-UID

And if you're running instrumented applications with 2 users:

small: 0 MiB * 2 = 0 MiB with 2 UIDs
medium: 102 MiB * 2 = 204 MiB with 2 UIDs
large: 4352 MiB * 2 = 8704 MiB with 2 UIDs

Now this is just an estimation for the per-CPU ring buffers only, and 
you numbers aren't hugely off so without analyzing your specific system 
it doesn't seem to be that strange to me.

If I take the number of channels I see in your script, it becomes:

small: 0 MiB with 2 UIDs
medium: 136 MiB with 2 UIDs
large: 7616 MiB with 2 UIDs

total: 7.57 GiB with 2 UIDs

> 
> Please do correct me If I am wrong calculations here.
> 
> Now, there are a few things here, according to my output lttng is using 11G
> which is much more higher than the what is configured.
> 

I have no idea what 'service start spyder' is doing. Maybe it's running 
instrumented applications with an extra user that you didn't expect? I 
can't help you with that aspect of your system.

The above estimated 7.57 GiB with 2 UIDs would be 11.35 GiB with 3 UIDs 
so maybe?

I'd recommend you read your verbose sessiond log so see which 
applications are connecting and with which UIDs.

> I am attaching the lttng status and the file which is uses to create the
> lttng sessions.
> 
> 
> 
> Thank You.
> 

In any case, the information you have given to date hasn't demonstrated 
to me in a tangible manner that you are seeing a difference related to 
the version of LTTng being used.

thanks,
kienan

[1]: https://lttng.org/man/3/lttng-ust/v2.13/#doc-_environment_variables

Memory Consumption High After Upgrading to 2.13 from 2.10

[Gour DEV via lttng-dev]

[-- Attachment #1: Type: text/plain, Size: 3299 bytes --]

Hello,

I have recently upgraded from lttng 2.10 to 2.13 as part of servers
migration from buster to bookworm.

lttng 2.10.6 - KeKriek to lttng 2.13.9 - Nordicité

I see an increase in memory consumption upto 11Gig after my recent upgrade.


In buster

root@localhost:~# top -b -n 1 | grep lttng
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
4296 root 20 0 1003676 19616 4608 S 0.0 0.1 0:00.67 lttng-sessiond
4297 root 20 0 44260 800 0 S 0.0 0.0 0:00.00 lttng-runas
4643 root 20 0 5838512 10396 9204 S 0.0 0.0 0:09.24 lttng-consumerd
4646 root 20 0 48048 544 0 S 0.0 0.0 0:00.00 lttng-runas


In bookworm
In bookworm.
root@edgecore-40XKE-j2-101-32:~# top -b -n 1 | grep lttng
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
4824 root 20 0 1098824 13336 5980 S 11.8 0.0 0:03.46 lttng-s+
4825 root 20 0 48872 2188 1012 S 0.0 0.0 0:00.00 lttng-r+
4841 root 20 0 11.5g 11.0g 11.0g S 0.0 35.4 2:33.33 lttng-c+
4843 root 20 0 3680 1160 816 S 0.0 0.0 0:00.23 lttng-r+


I have a python script which was used to enable channels, these were the
parameters I was using
large_channels_sz = 4000000
medium_channels_sz = 200000
small_channels_sz = 10000

num_large = 4
num_medium = 4
num_small = 4

def enable_channel(channels, session, subbuf_size, subbuf_num):
for c in channels:
call(['lttng', 'enable-channel', '-u', c, '-s', session, '--subbuf-size',
str(subbuf_size), '--num-subbuf', str(subbuf_num),],
stdout=devnull, stderr=subprocess.STDOUT)

enable_channel(large_channels, def_session, large_channels_sz, num_large)
enable_channel(medium_channels, def_session, medium_channels_sz, num_medium)
enable_channel(small_channels, def_session, small_channels_sz, num_small)



there are total 16 large channels and 6 medium channels and no small
channels.


The total size for all these session cominded turn out to be machine
roughly 7GB on my 32 core machine, which is still less that the amount of
memory used by the lttng-consumerd.

I only have default session configured and nothing else

root@edgecore-40XKE-j2-101-32:~# lttng list
Available recording sessions:
1) default [active snapshot]




I have seen that when only the channel are enable and no process which is
using lttng is running then, I see normal memory consumption.


root@edgecore-40XKE-j2-101-32:~# top -b -n 1 | grep lttng

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
4824 root 20 0 1098184 12732 5376 S 0.0 0.0 0:03.15 lttng-s+
4825 root 20 0 48872 2188 1012 S 0.0 0.0 0:00.00 lttng-r+
4841 root 20 0 536240 4284 2400 S 0.0 0.0 2:09.32 lttng-c+
4843 root 20 0 3680 1160 816 S 0.0 0.0 0:00.23 lttng-r+


But, as soon as I run even an single process which is linked to lttng, and
which is *not* using any lttng channel

liblttng-ust.so.1 => /lib/x86_64-linux-gnu/liblttng-ust.so.1 (0x
00007fa867ea3000)
liblttng-ust-common.so.1 => /lib/x86_64-linux-gnu/liblttng-ust-common.so.1 (
0x00007fa86752d000)
liblttng-ust-tracepoint.so.1 =>
/lib/x86_64-linux-gnu/liblttng-ust-tracepoint.so.1
(0x00007fa86750b000)

I see an very high memory consumption of 5.5g and when i run all the
process then i see the total memory consumption reaching uptop 11G.


Do you have any hints on why this could be happening,

Thank You.

[-- Attachment #2: Type: text/html, Size: 28925 bytes --]

Re: Memory Consumption High After Upgrading to 2.13 from 2.10

[Kienan Stewart via lttng-dev]

Hi Lakshya,

what is the output of the following commands on both systems:

* nproc (32, I assume)
* cat /sys/devices/system/cpu/possible

On 3/10/25 10:02 AM, Gour DEV via lttng-dev wrote:
> Hello,
> 
> I have recently upgraded from lttng 2.10 to 2.13 as part of servers
> migration from buster to bookworm.
> 
> lttng 2.10.6 - KeKriek to lttng 2.13.9 - Nordicité
> 
> I see an increase in memory consumption upto 11Gig after my recent upgrade.
> 
> 
> In buster
> 
> root@localhost:~# top -b -n 1 | grep lttng
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> 4296 root 20 0 1003676 19616 4608 S 0.0 0.1 0:00.67 lttng-sessiond
> 4297 root 20 0 44260 800 0 S 0.0 0.0 0:00.00 lttng-runas
> 4643 root 20 0 5838512 10396 9204 S 0.0 0.0 0:09.24 lttng-consumerd
> 4646 root 20 0 48048 544 0 S 0.0 0.0 0:00.00 lttng-runas
> 
> 
> In bookworm
> In bookworm.
> root@edgecore-40XKE-j2-101-32:~# top -b -n 1 | grep lttng
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> 4824 root 20 0 1098824 13336 5980 S 11.8 0.0 0:03.46 lttng-s+
> 4825 root 20 0 48872 2188 1012 S 0.0 0.0 0:00.00 lttng-r+
> 4841 root 20 0 11.5g 11.0g 11.0g S 0.0 35.4 2:33.33 lttng-c+
> 4843 root 20 0 3680 1160 816 S 0.0 0.0 0:00.23 lttng-r+
> 
> 
> I have a python script which was used to enable channels, these were the
> parameters I was using
> large_channels_sz = 4000000
> medium_channels_sz = 200000
> small_channels_sz = 10000
> 
> num_large = 4
> num_medium = 4
> num_small = 4
> 
> def enable_channel(channels, session, subbuf_size, subbuf_num):
> for c in channels:
> call(['lttng', 'enable-channel', '-u', c, '-s', session, '--subbuf-size',
> str(subbuf_size), '--num-subbuf', str(subbuf_num),],
> stdout=devnull, stderr=subprocess.STDOUT)
> 
> enable_channel(large_channels, def_session, large_channels_sz, num_large)
> enable_channel(medium_channels, def_session, medium_channels_sz, num_medium)
> enable_channel(small_channels, def_session, small_channels_sz, num_small)
> 
> 
> 
> there are total 16 large channels and 6 medium channels and no small
> channels.
> 
> 
> The total size for all these session cominded turn out to be machine
> roughly 7GB on my 32 core machine, which is still less that the amount of
> memory used by the lttng-consumerd.
> 
> I only have default session configured and nothing else
> 
> root@edgecore-40XKE-j2-101-32:~# lttng list
> Available recording sessions:
> 1) default [active snapshot]
> 
> 
> 
> 
> I have seen that when only the channel are enable and no process which is
> using lttng is running then, I see normal memory consumption.
> 
> 
> root@edgecore-40XKE-j2-101-32:~# top -b -n 1 | grep lttng
> 
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> 4824 root 20 0 1098184 12732 5376 S 0.0 0.0 0:03.15 lttng-s+
> 4825 root 20 0 48872 2188 1012 S 0.0 0.0 0:00.00 lttng-r+
> 4841 root 20 0 536240 4284 2400 S 0.0 0.0 2:09.32 lttng-c+
> 4843 root 20 0 3680 1160 816 S 0.0 0.0 0:00.23 lttng-r+
> 
> 
> But, as soon as I run even an single process which is linked to lttng, and
> which is *not* using any lttng channel

With which user(s) are you running the instrumented applications?

> 
> liblttng-ust.so.1 => /lib/x86_64-linux-gnu/liblttng-ust.so.1 (0x
> 00007fa867ea3000)
> liblttng-ust-common.so.1 => /lib/x86_64-linux-gnu/liblttng-ust-common.so.1 (
> 0x00007fa86752d000)
> liblttng-ust-tracepoint.so.1 =>
> /lib/x86_64-linux-gnu/liblttng-ust-tracepoint.so.1
> (0x00007fa86750b000)
> 
> I see an very high memory consumption of 5.5g and when i run all the
> process then i see the total memory consumption reaching uptop 11G.
> 
> 
> Do you have any hints on why this could be happening,
> 
> Thank You.
> 

thanks,
kienan