[lttng-dev] Seeking Assistance on LTTng for Kubernetes Performance Research

[lttng-dev] Seeking Assistance on LTTng for Kubernetes Performance Research

[Benjamin Saint-Cyr via lttng-dev]

Dear LTTng Developers,

I am currently conducting performance research on Kubernetes for my  
master's thesis and am using LTTng for data collection and Trace  
Compass for analysis. I have a couple of questions and concerns, and I  
would appreciate your guidance to ensure I am using your tools  
effectively.

	1. In the Control Flow view, I can see the CPU throttling due to the  
quota defined in the Cgroup as the pod runs and then waits for the CPU  
periodically. However, it is unclear whether the pod is waiting due to  
a full CPU or because of timeshares. I can easily find this  
information in /sys/fs/cgroup/cpu,cpuacct/kubepods/<burstable or  
besteffort>/pod<UID>/cpu.stat if the pod is being throttled. Is there  
a way to extract this information using LTTng or Trace Compass?

	2. I am attempting to determine which PID is running a specific  
Container/Pod UID. Currently, I can deduce this by observing the  
process named runc[*:INIT], as it creates the cgroup and clones the  
process for the container. However, this approach is fragile and only  
works due to an implementation detail. It also fails if tracing starts  
after deployment. This information is easily available in  
/sys/fs/cgroup/cpu,cpuacct/kubepods/<burstable or  
besteffort>/pod<UID>/<container ID>/cgroup.procs. Is there an existing  
method within LTTng to obtain this information that I may not be aware  
of?

I have been informed that it might be possible to read the information  
in cpu.stat or cgroup.procs using a state dump at the beginning or  
end, but the process remains unclear. Alternatively, I could  
instrument the kernel or the process handling the accounting, but this  
would be quite challenging. Another option would be to use something  
similar to cAdvisor, but its efficiency is questionable since it  
dedicates an entire process to reading a few files.

I would greatly appreciate any guidance or suggestions you may have on  
improving my approach and effectively utilizing LTTng for my research.

Thank you for your time and assistance.

Best regards,

Benjamin Saint-Cyr




_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [lttng-dev] Seeking Assistance on LTTng for Kubernetes Performance Research

[Michael Jeanson via lttng-dev]

On 2023-04-06 15:56, Benjamin Saint-Cyr via lttng-dev wrote:
> Dear LTTng Developers,
> 
> I am currently conducting performance research on Kubernetes for my master's 
> thesis and am using LTTng for data collection and Trace Compass for analysis. 
> I have a couple of questions and concerns, and I would appreciate your 
> guidance to ensure I am using your tools effectively.
> 
>      1. In the Control Flow view, I can see the CPU throttling due to the 
> quota defined in the Cgroup as the pod runs and then waits for the CPU 
> periodically. However, it is unclear whether the pod is waiting due to a full 
> CPU or because of timeshares. I can easily find this information in 
> /sys/fs/cgroup/cpu,cpuacct/kubepods/<burstable or 
> besteffort>/pod<UID>/cpu.stat if the pod is being throttled. Is there a way to 
> extract this information using LTTng or Trace Compass?
> 
>      2. I am attempting to determine which PID is running a specific 
> Container/Pod UID. Currently, I can deduce this by observing the process named 
> runc[*:INIT], as it creates the cgroup and clones the process for the 
> container. However, this approach is fragile and only works due to an 
> implementation detail. It also fails if tracing starts after deployment. This 
> information is easily available in 
> /sys/fs/cgroup/cpu,cpuacct/kubepods/<burstable or 
> besteffort>/pod<UID>/<container ID>/cgroup.procs. Is there an existing method 
> within LTTng to obtain this information that I may not be aware of?

The PID namespace context is probably the closest thing to what you are 
looking for, see 
"https://lttng.org/man/3/lttng-ust/v2.13/#doc-_context_information". The 
hierachy of the namespaces can be recorded in a statedump.

> 
> I have been informed that it might be possible to read the information in 
> cpu.stat or cgroup.procs using a state dump at the beginning or end, but the 
> process remains unclear. Alternatively, I could instrument the kernel or the 
> process handling the accounting, but this would be quite challenging. Another 
> option would be to use something similar to cAdvisor, but its efficiency is 
> questionable since it dedicates an entire process to reading a few files.

I don't think we have anything cgroup related in the statedump at the moment, 
not that it's impossible but it was never implemented.

> 
> I would greatly appreciate any guidance or suggestions you may have on 
> improving my approach and effectively utilizing LTTng for my research.
> 
> Thank you for your time and assistance.
> 
> Best regards,
> 
> Benjamin Saint-Cyr


_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev