[lttng-dev] relayd with many consumer daemons

[lttng-dev] relayd with many consumer daemons

[Jakub Witczak]

Hi,

I would like to consult on setup where relayd is serving multiple consumer daemons at the same time.

I assume above setup is nothing new and is not a problem as long as each of consumer daemons is having unique tracing session (please confirm).

But what if I would like to have more consumer daemons collecting data from *single session* - is it possible to aggregate data in relayd?

regards,
Kuba

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.lttng.org/pipermail/lttng-dev/attachments/20180227/3c29f4b7/attachment.html>

[lttng-dev] relayd with many consumer daemons

[Jonathan Rajotte-Julien]

Hi Kuba,

On Tue, Feb 27, 2018 at 03:34:59PM +0000, Jakub Witczak wrote:
> Hi,
> 
> I would like to consult on setup where relayd is serving multiple consumer daemons at the same time.

On the same host? On multiple hosts?

In both cases, it is supported.

It is recommended to have unique session id across multiples sessiond/consumer on the same host but not required.

> 
> I assume above setup is nothing new and is not a problem as long as each of consumer daemons is having unique tracing session (please confirm).

As for multiple hosts, relayd also differentiate based on the hostname.

From lttng-relayd man page:

    By default, the relay daemon writes the traces to:

    $LTTNG_HOME/lttng-traces/HOSTNAME/SESSION/DOMAIN

    with:

    HOSTNAME
        Remote hostname.

    SESSION
        Full session name.

    DOMAIN
        Tracing domain.

    You can override the default output directory prefix
    ($LTTNG_HOME/lttng-traces) with the --output option. The other parts
    depend on the remote configuration.

For example, user on host DATA1 create and start a tracing a session (live/streaming) with the name MySession.
Relayd will create its internal state with the hostname, session name, time.
Hence, trace data will be written somewhere like this:
    $LTTNG_HOME/lttng-traces/DATA1/MySession-TIMESTAMP/DOMAIN

At the same time on host DATA2, a user create and start a tracing session
(live/streaming) with the name MySession.
Trace data will be written somewhere like this:
    $LTTNG_HOME/lttng-traces/DATA2/MySession-TIMESTAMP/DOMAIN

> 
> But what if I would like to have more consumer daemons collecting data from *single session* - is it possible to aggregate data in relayd?

Note that Babeltrace can read and merge data from multiple sources:
    babeltrace $LTTNG_HOME/lttng-traces/DATA1/MySession-TIMESTAMP $LTTNG_HOME/lttng-traces/DATA2/MySession-TIMESTAMP/DOMAIN<Paste>

Not sure what you mean by "consumer daemons collecting data from *single session*".

Do you mean having multiple consumer "writing" to the same session on lttng-relayd side?
What would motivate this design choice, is a limitation you encountered?

Is it not sufficient that multiple traces can be merged using babeltrace?

Cheers

-- 
Jonathan Rajotte-Julien
EfficiOS

[lttng-dev] relayd with many consumer daemons

[Jakub Witczak]

Hi,
 
> > I would like to consult on setup where relayd is serving multiple consumer
> daemons at the same time.
> 
> On the same host? On multiple hosts?
> In both cases, it is supported.

It's a setup where traces are transferred over the network to remote relayd.

> It is recommended to have unique session id across multiples
> sessiond/consumer on the same host but not required.

> > I assume above setup is nothing new and is not a problem as long as each
> of consumer daemons is having unique tracing session (please confirm).
> 
> As for multiple hosts, relayd also differentiate based on the hostname.
> 
> From lttng-relayd man page:
> 
>     By default, the relay daemon writes the traces to:
> 
>     $LTTNG_HOME/lttng-traces/HOSTNAME/SESSION/DOMAIN
> 
>     with:
> 
>     HOSTNAME
>         Remote hostname.
> 
>     SESSION
>         Full session name.
> 
>     DOMAIN
>         Tracing domain.
> 
>     You can override the default output directory prefix
>     ($LTTNG_HOME/lttng-traces) with the --output option. The other parts
>     depend on the remote configuration.
> 
> For example, user on host DATA1 create and start a tracing a session
> (live/streaming) with the name MySession.
> Relayd will create its internal state with the hostname, session name, time.
> Hence, trace data will be written somewhere like this:
>     $LTTNG_HOME/lttng-traces/DATA1/MySession-TIMESTAMP/DOMAIN
> 
> At the same time on host DATA2, a user create and start a tracing session
> (live/streaming) with the name MySession.
> Trace data will be written somewhere like this:
>     $LTTNG_HOME/lttng-traces/DATA2/MySession-TIMESTAMP/DOMAIN
> 
> >
> > But what if I would like to have more consumer daemons collecting data
> from *single session* - is it possible to aggregate data in relayd?
> 
> Note that Babeltrace can read and merge data from multiple sources:
>     babeltrace $LTTNG_HOME/lttng-traces/DATA1/MySession-TIMESTAMP
> $LTTNG_HOME/lttng-traces/DATA2/MySession-
> TIMESTAMP/DOMAIN<Paste>
> 
This looks very promising. Can it merge sessions in live mode as well?

> Not sure what you mean by "consumer daemons collecting data from *single
> session*".
> 
> Do you mean having multiple consumer "writing" to the same session on
> lttng-relayd side?
> What would motivate this design choice, is a limitation you encountered?
> 
> Is it not sufficient that multiple traces can be merged using babeltrace?

Yes. I meant multiple consumers (acting as single host but generated elsewhere (traffic NATed from different machine)) writing to single session in relayd.

Babeltrace merge could be enough, but hard to say without testing.

Regards
Kuba

[lttng-dev] relayd with many consumer daemons

[Jonathan Rajotte-Julien]

Hi Kuba,

> > 
> > Note that Babeltrace can read and merge data from multiple sources:
> >     babeltrace $LTTNG_HOME/lttng-traces/DATA1/MySession-TIMESTAMP
> > $LTTNG_HOME/lttng-traces/DATA2/MySession-
> > TIMESTAMP/DOMAIN<Paste>
> > 
> This looks very promising. Can it merge sessions in live mode as well?

Unfortunately Babeltrace 1.x does not support multiple input for lttng-live.

Babeltrace 2 does support reading from multiple sources for live session.
Babeltrace 2 is currently in its pre-release phase and is considered
experimental.

> 
> > Not sure what you mean by "consumer daemons collecting data from *single
> > session*".
> > 
> > Do you mean having multiple consumer "writing" to the same session on
> > lttng-relayd side?
> > What would motivate this design choice, is a limitation you encountered?
> > 
> > Is it not sufficient that multiple traces can be merged using babeltrace?
> 
> Yes. I meant multiple consumers (acting as single host but generated elsewhere (traffic NATed from different machine)) writing to single session in relayd.

lttng-relayd will always append a timestamp to the session name on session
creation on its side. Hence, even if all hosts have the same hostname and the same
session name, the actual trace data will be separated in two different folder on the
lttng-relayd host.

As for the lttng-live merge reading with babeltrace 1.x,
I did a simple experiment locally and this can "work" if and only if all host have
the same hostname and session name. Given that babeltrace 1.x does not support
multiple input for lttng-live it could be a workaround.
**** I do not recommend it and is not supported officially.****

If your use case is streaming traces to relayd and reading it offline you are better
of with using babeltrace for doing the merge without any "hack" regarding
hostname etc. for lttng-live reading.

Note that babeltrace by default will read all traces under the directory you are
passing as argument. This coupled with the --output option of lttng-relayd can
be an easy way to bundle all related traces together and read them in one go.

Cheers

> 
> Babeltrace merge could be enough, but hard to say without testing.
> 
> Regards
> Kuba
> 

-- 
Jonathan Rajotte-Julien
EfficiOS