[lttng-dev] LTTng and containers.

[lttng-dev] LTTng and containers.

[Maksim Khmelevskiy via lttng-dev]

[-- Attachment #1.1: Type: text/plain, Size: 914 bytes --]

Hi,

I would like to ask regarding the hot topic - container tracing. I've seen
a youtube video <https://www.youtube.com/watch?v=hra-eu6EOpY>, have read a
message <https://lists.lttng.org/pipermail/lttng-dev/2021-May/029952.html>
from LTTng mailing list and tried to google more about this topic but
didn't find much of info. Could you please direct me where should I
continue digging?
My problem:
I would like to have multiple containers where traces are generated by apps
with compiled-in tracepoints. Traces could be stored in these containers as
well.
Besides these containers I would have a trace processor container, a master
container which could address a container(or a trace session) and fetch
traces from it and read with babeltrace or similar tool.
So far, intuitively, remote tracing comes to my mind but before continuing
with the task I would be happy to hear an advice from LTTng devs.

Thank you!

[-- Attachment #1.2: Type: text/html, Size: 1058 bytes --]

[-- Attachment #2: Type: text/plain, Size: 156 bytes --]

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [lttng-dev] LTTng and containers.

[Jérémie Galarneau via lttng-dev]

[-- Attachment #1.1: Type: text/plain, Size: 2454 bytes --]

Hi Maksim,

It's hard to give general advice without knowing more about your constraints.

I can see three main approaches that are easy to reach:

  1.  Store the traces in each container and fetch them,
  2.  Trace to a shared volume (make sure sessions are stopped or rotated before you read them),
  3.  Or stream traces to a "collector" container (perhaps the one doing the analyses) using network streaming

If you want to continuously consume traces, I would suggest you look into "trace rotations". You could setup periodic trace rotations (based on size or time) and ship traces to your processor node as they become available.

Although my demo wasn't running in a container, the basic idea is what I presented at FOSDEM in 2019:

https://archive.fosdem.org/2019/schedule/event/lttng/
[https://fosdem.org/2019/assets/style/logo-gear-7204a6874eb0128932db10ff4030910401ac06f4e907f8b4a40da24ba592b252.png]<https://archive.fosdem.org/2019/schedule/event/lttng/>
FOSDEM 2019 - Fine-grained Distributed Application Monitoring Using LTTng<https://archive.fosdem.org/2019/schedule/event/lttng/>
archive.fosdem.org

I hope that helps a bit,
Jérémie

--
Jérémie Galarneau
EfficiOS Inc.
https://www.efficios.com
________________________________
From: lttng-dev <lttng-dev-bounces@lists.lttng.org> on behalf of Maksim Khmelevskiy via lttng-dev <lttng-dev@lists.lttng.org>
Sent: October 10, 2022 12:12
To: lttng-dev@lists.lttng.org <lttng-dev@lists.lttng.org>
Subject: [lttng-dev] LTTng and containers.

Hi,

I would like to ask regarding the hot topic - container tracing. I've seen a youtube video<https://www.youtube.com/watch?v=hra-eu6EOpY>, have read a message<https://lists.lttng.org/pipermail/lttng-dev/2021-May/029952.html> from LTTng mailing list and tried to google more about this topic but didn't find much of info. Could you please direct me where should I continue digging?
My problem:
I would like to have multiple containers where traces are generated by apps with compiled-in tracepoints. Traces could be stored in these containers as well.
Besides these containers I would have a trace processor container, a master container which could address a container(or a trace session) and fetch traces from it and read with babeltrace or similar tool.
So far, intuitively, remote tracing comes to my mind but before continuing with the task I would be happy to hear an advice from LTTng devs.

Thank you!

[-- Attachment #1.2: Type: text/html, Size: 6276 bytes --]

[-- Attachment #2: Type: text/plain, Size: 156 bytes --]

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [lttng-dev] LTTng and containers.

[Michael Jeanson via lttng-dev]

On 2022-10-10 17:12, Maksim Khmelevskiy via lttng-dev wrote:
> Hi,
> 
> I would like to ask regarding the hot topic - container tracing. I've seen a 
> youtube video <https://www.youtube.com/watch?v=hra-eu6EOpY>, have read a 
> message <https://lists.lttng.org/pipermail/lttng-dev/2021-May/029952.html> 
> from LTTng mailing list and tried to google more about this topic but didn't 
> find much of info. Could you please direct me where should I continue digging?
> My problem:
> I would like to have multiple containers where traces are generated by apps 
> with compiled-in tracepoints. Traces could be stored in these containers as well.
> Besides these containers I would have a trace processor container, a master 
> container which could address a container(or a trace session) and fetch traces 
> from it and read with babeltrace or similar tool.
> So far, intuitively, remote tracing comes to my mind but before continuing 
> with the task I would be happy to hear an advice from LTTng devs.
> 
> Thank you!

Hi,

You can either use lttng-relayd to send the traces over the network to your 
trace processor container, or just trace to disk on shared diectories. You 
should have a look at trace rotation, https://doc.lttng.org is probably the 
best place to start.

Michael

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev