[lttng-dev] babletrace2 graph performance considerations

[lttng-dev] babletrace2 graph performance considerations

[Rocky Dunlap]

A couple of questions on performance considerations when setting up bt2
processing graphs.

1.  Do parts of the processing graph that can execute concurrently do so?
Does the user have any control over this, e.g., by creating threads in sink
components?

2.  It looks like you cannot connect one output port to multiple inputs.
Is there a way to create a tee component?

Rocky
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.lttng.org/pipermail/lttng-dev/attachments/20200329/69382d00/attachment.htm>

[lttng-dev] babletrace2 graph performance considerations

[Simon Marchi]

On 2020-03-30 12:10 a.m., Rocky Dunlap via lttng-dev wrote:
> A couple of questions on performance considerations when setting up bt2 processing graphs.
> 
> 1.? Do parts of the processing graph that can execute concurrently do so?? Does the user have any control over this, e.g., by creating threads in sink components?

The graph execution in itself is single-threaded at the moment.  We could imaging
a design where different parts of the graph execute concurrently, in different
threads, but it's the the case right now.

You could make your components spawn threads to do some processing on the side,
if that helps, but these other threads should not interact directly with the
graph.

> 2.? It looks like you cannot connect one output port to multiple inputs.? Is there a way to create a tee component?

Yes, we have discussed making a tee component, it is on the roadmap but not really
planned yet.  It should be possible, it's just not as trivial as it may sound.

One easy way to achieve it is to make each iterator that is created on the tee
component create and maintain its own upstream iterator.  If you have a tee with
two outputs, this will effectively make it so you have two graphs executing in
parallel.  If you have a src.ctf.fs source upstream of the tee, then there will
be two iterators created on that source, so the CTF trace will be open and decoded
twice.  We'd like to avoid that.

The other way of doing it is to make the tee buffer messages, and discard a
message once all downstream iterators have consumed it.  This has some more
difficult technical challenges, like what to do when one downstream iterator
consumes, but the other does not (we don't want to buffer an infinite amount
of data).  It also makes seeking a bit tricky.

We could go in more details, if you are interested in starting implementing it
yourself.

Simon

[lttng-dev] babletrace2 graph performance considerations

[Rocky Dunlap]

Simon,

Thanks - see below,

On Mon, Mar 30, 2020 at 7:32 AM Simon Marchi <simark at simark.ca> wrote:

> On 2020-03-30 12:10 a.m., Rocky Dunlap via lttng-dev wrote:
> > A couple of questions on performance considerations when setting up bt2
> processing graphs.
> >
> > 1.  Do parts of the processing graph that can execute concurrently do
> so?  Does the user have any control over this, e.g., by creating threads in
> sink components?
>
> The graph execution in itself is single-threaded at the moment.  We could
> imaging
> a design where different parts of the graph execute concurrently, in
> different
> threads, but it's the the case right now.
>
> You could make your components spawn threads to do some processing on the
> side,
> if that helps, but these other threads should not interact directly with
> the
> graph.
>

In my case I have CTF trace where some analyses can be performed on a
per-stream basis (no need to mux the streams together).  In this case, I
was thinking that it would make sense to thread over the streams.  However,
I think can easily do this at a level above the graph simply by creating
multiple graphs where each one is handling a single stream.  In my case I
am thinking this will be mostly I/O bound, so I'm not sure what kind of
payoff the threads will give.  Overall, I just want to make sure that I am
not doing anything that would, in the long run, preclude
threading/concurrency if it is added to the graph model itself.


>
> > 2.  It looks like you cannot connect one output port to multiple
> inputs.  Is there a way to create a tee component?
>
> Yes, we have discussed making a tee component, it is on the roadmap but
> not really
> planned yet.  It should be possible, it's just not as trivial as it may
> sound.
>
> One easy way to achieve it is to make each iterator that is created on the
> tee
> component create and maintain its own upstream iterator.  If you have a
> tee with
> two outputs, this will effectively make it so you have two graphs
> executing in
> parallel.  If you have a src.ctf.fs source upstream of the tee, then there
> will
> be two iterators created on that source, so the CTF trace will be open and
> decoded
> twice.  We'd like to avoid that.
>
> The other way of doing it is to make the tee buffer messages, and discard a
> message once all downstream iterators have consumed it.  This has some more
> difficult technical challenges, like what to do when one downstream
> iterator
> consumes, but the other does not (we don't want to buffer an infinite
> amount
> of data).  It also makes seeking a bit tricky.
>
> We could go in more details, if you are interested in starting
> implementing it
> yourself.
>

Yea, I can see how this can get tricky.  This is not critical at this very
moment, but I just wondered if there was a precedent for how to do this
kind of thing.


>
> Simon
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.lttng.org/pipermail/lttng-dev/attachments/20200330/e2b64238/attachment.htm>

[lttng-dev] babletrace2 graph performance considerations

[Simon Marchi]

On 2020-03-30 1:30 p.m., Rocky Dunlap wrote:
> In my case I have CTF trace where some analyses can be performed on a per-stream basis (no need to mux the streams together).? In this case, I was thinking that it would make sense to thread over the streams.? However, I think can easily do this at a level above the graph simply by creating multiple graphs where each one is handling a single stream.? In my case I am thinking this will be mostly I/O bound, so I'm not sure what kind of payoff the threads will give.? Overall, I just want to make sure that I am not doing anything that would, in the long run, preclude threading/concurrency if it is added to the graph model itself.

Creating multiple graphs and handling a single stream in each would
work, but each graph would end up reading and parsing the while CTF
trace, which is not great.

If you streams are in a data file of their own, you could move them
to their own directory, each with a copy of the metadata, effectively
creating one trace for each.  Then create one graph for each trace.

I can't really tell you right now how to design your components to work
with a future model where there is concurrency, since that model does
not exist yet :).

Simon

[lttng-dev] babletrace2 graph performance considerations

[Rocky Dunlap]

Each graph would parse the whole stream?  What if I only connect one of the
outputs from the source.ctf.fs?  I would think it would only access that
one stream file.

Rocky

On Mon, Mar 30, 2020 at 4:27 PM Simon Marchi <simark at simark.ca> wrote:

> On 2020-03-30 1:30 p.m., Rocky Dunlap wrote:
> > In my case I have CTF trace where some analyses can be performed on a
> per-stream basis (no need to mux the streams together).  In this case, I
> was thinking that it would make sense to thread over the streams.  However,
> I think can easily do this at a level above the graph simply by creating
> multiple graphs where each one is handling a single stream.  In my case I
> am thinking this will be mostly I/O bound, so I'm not sure what kind of
> payoff the threads will give.  Overall, I just want to make sure that I am
> not doing anything that would, in the long run, preclude
> threading/concurrency if it is added to the graph model itself.
>
> Creating multiple graphs and handling a single stream in each would
> work, but each graph would end up reading and parsing the while CTF
> trace, which is not great.
>
> If you streams are in a data file of their own, you could move them
> to their own directory, each with a copy of the metadata, effectively
> creating one trace for each.  Then create one graph for each trace.
>
> I can't really tell you right now how to design your components to work
> with a future model where there is concurrency, since that model does
> not exist yet :).
>
> Simon
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.lttng.org/pipermail/lttng-dev/attachments/20200330/04375035/attachment.htm>

[lttng-dev] babletrace2 graph performance considerations

[Simon Marchi]

On 2020-03-30 6:34 p.m., Rocky Dunlap via lttng-dev wrote:
> Each graph would parse the whole stream?? What if I only connect one of the outputs from the source.ctf.fs?? I would think it would only access that one stream file.

Hmm yeah perhaps, I didn't think of that.  Give it a try!

Simon