Re: Memory Consumption High After Upgrading to 2.13 from 2.10

[Kienan Stewart via lttng-dev <lttng-dev@lists.lttng.org>]

Hi Lakshya,

On 3/12/25 5:03 AM, Gour DEV wrote:
> Hi, Kienan
> 
> I am attaching an screen recording of the behaviour I am seeing in this
> mail. The behaviour is same irrespective of the device i use, sorry for
> miscommunication in the npocs output (I assumed it was 32), but other than
> that all outputs are same (except the hostname as there are multiple
> devices with same lttng config but this memory cosumption is seen on all
> the devices).
> 
> I had few question
> 
> 1. Does lltng allocated all the memory it needs and mark it as dirty in ram
> when any process which links/uses lttng-ust runs? (here i tried with one
> process but it is same for any of my process)

I believe the shared memory for per-CPU data structures is allocated 
when an instrumented application connects. There is no pre-allocation 
for each possible UID on the system.

You can run your instrumented applications with `LTTNG_UST_DEBUG=1` to 
see when the connection happens[1].

> 2. (nSubbuf * subbufSize) * (nCPUs + 1 iff snapshot mode is enabled) *
> (nUIDs or nPIDs)
> 
> How do we calculate uid in the system is it all uids in the system? is it
> equal to `cat /etc/passwd | wc -l` ?

nUIDs is the number of distinct UIDs running instrumented applications.

> 
> I will put my calculations according to the above estimate based on all the
> channel i am creating
> 
> (4194304*4 + 262144*4 + 16384*4) * (16) * (30 if number user are equal to
> `cat /etc/passwd | wc -l`)B = 7.998046875 GB approx [this is based on the
> start_lttng.py please do correct me if am wrong here.]
> 
> But since there are only two users which uses lttng i think the correct
> estimate would be
> (4194304*4 + 262144*4 + 16384*4) * (16) * (2)B = 546MB

The estimate I gave is per-channel.

small channel: (0.015625 MiB * 4) * (16 + 1) = 1.0625 MiB per-channel 
per-UID
medium channel: (0.250 MiB * 4) * (16 + 1) = 17.0 MiB per-channel per-UID
large channel: (4 MiB * 4) * (16 + 1) = 27 2MiB per-channel per-UID

Now, you said you have 0 small channels, 6 medium channels, and 16 large 
channels in your session. (Note: I see your script differs from these 
stated channel counts).

small: 0 * 1.0625 MiB = 0 MiB per-UID
medium: 6 * 17 MiB = 102 MiB per-UID
large: 16 * 272 MiB = 4352 MiB per-UID

And if you're running instrumented applications with 2 users:

small: 0 MiB * 2 = 0 MiB with 2 UIDs
medium: 102 MiB * 2 = 204 MiB with 2 UIDs
large: 4352 MiB * 2 = 8704 MiB with 2 UIDs

Now this is just an estimation for the per-CPU ring buffers only, and 
you numbers aren't hugely off so without analyzing your specific system 
it doesn't seem to be that strange to me.

If I take the number of channels I see in your script, it becomes:

small: 0 MiB with 2 UIDs
medium: 136 MiB with 2 UIDs
large: 7616 MiB with 2 UIDs

total: 7.57 GiB with 2 UIDs

> 
> Please do correct me If I am wrong calculations here.
> 
> Now, there are a few things here, according to my output lttng is using 11G
> which is much more higher than the what is configured.
> 

I have no idea what 'service start spyder' is doing. Maybe it's running 
instrumented applications with an extra user that you didn't expect? I 
can't help you with that aspect of your system.

The above estimated 7.57 GiB with 2 UIDs would be 11.35 GiB with 3 UIDs 
so maybe?

I'd recommend you read your verbose sessiond log so see which 
applications are connecting and with which UIDs.

> I am attaching the lttng status and the file which is uses to create the
> lttng sessions.
> 
> 
> 
> Thank You.
> 

In any case, the information you have given to date hasn't demonstrated 
to me in a tangible manner that you are seeing a difference related to 
the version of LTTng being used.

thanks,
kienan

[1]: https://lttng.org/man/3/lttng-ust/v2.13/#doc-_environment_variables