From mboxrd@z Thu Jan 1 00:00:00 1970 From: dunlap@ucar.edu (Rocky Dunlap) Date: Mon, 30 Mar 2020 11:30:58 -0600 Subject: [lttng-dev] babletrace2 graph performance considerations In-Reply-To: <9b2fe91b-4f44-3b89-eaf9-98f945384a33@simark.ca> References: <9b2fe91b-4f44-3b89-eaf9-98f945384a33@simark.ca> Message-ID: Simon, Thanks - see below, On Mon, Mar 30, 2020 at 7:32 AM Simon Marchi wrote: > On 2020-03-30 12:10 a.m., Rocky Dunlap via lttng-dev wrote: > > A couple of questions on performance considerations when setting up bt2 > processing graphs. > > > > 1. Do parts of the processing graph that can execute concurrently do > so? Does the user have any control over this, e.g., by creating threads in > sink components? > > The graph execution in itself is single-threaded at the moment. We could > imaging > a design where different parts of the graph execute concurrently, in > different > threads, but it's the the case right now. > > You could make your components spawn threads to do some processing on the > side, > if that helps, but these other threads should not interact directly with > the > graph. > In my case I have CTF trace where some analyses can be performed on a per-stream basis (no need to mux the streams together). In this case, I was thinking that it would make sense to thread over the streams. However, I think can easily do this at a level above the graph simply by creating multiple graphs where each one is handling a single stream. In my case I am thinking this will be mostly I/O bound, so I'm not sure what kind of payoff the threads will give. Overall, I just want to make sure that I am not doing anything that would, in the long run, preclude threading/concurrency if it is added to the graph model itself. > > > 2. It looks like you cannot connect one output port to multiple > inputs. Is there a way to create a tee component? > > Yes, we have discussed making a tee component, it is on the roadmap but > not really > planned yet. It should be possible, it's just not as trivial as it may > sound. > > One easy way to achieve it is to make each iterator that is created on the > tee > component create and maintain its own upstream iterator. If you have a > tee with > two outputs, this will effectively make it so you have two graphs > executing in > parallel. If you have a src.ctf.fs source upstream of the tee, then there > will > be two iterators created on that source, so the CTF trace will be open and > decoded > twice. We'd like to avoid that. > > The other way of doing it is to make the tee buffer messages, and discard a > message once all downstream iterators have consumed it. This has some more > difficult technical challenges, like what to do when one downstream > iterator > consumes, but the other does not (we don't want to buffer an infinite > amount > of data). It also makes seeking a bit tricky. > > We could go in more details, if you are interested in starting > implementing it > yourself. > Yea, I can see how this can get tricky. This is not critical at this very moment, but I just wondered if there was a precedent for how to do this kind of thing. > > Simon > -------------- next part -------------- An HTML attachment was scrubbed... URL: