[lttng-dev] Configurable tracepath?

[lttng-dev] Configurable tracepath?

[Jesper Derehag]

----------------------------------------
> Date: Mon, 30 Mar 2015 17:34:29 +0000
> From: mathieu.desnoyers@efficios.com
> To: jeremie.galarneau at efficios.com
> CC: jderehag at hotmail.com; lttng-dev at lists.lttng.org
> Subject: Re: [lttng-dev] Configurable tracepath?
>
> ----- Original Message -----
>> On Mon, Mar 30, 2015 at 5:21 AM, Jesper Derehag <jderehag at hotmail.com> wrote:
>>> No thoughts on this one?
>>>
>>> Den 25 mar 2015 16:53 skrev Jesper Derehag <jderehag at hotmail.com>:
>>>
>>> Hi,
>>>
>>> We have run into an issue with unique per-session tracepaths.
>>>
>>> In my particular setup I am using relayd to centralize alot of consumers
>>> (not really important to this discussion, but thought I'd mention it
>>> anyway).
>>> So the problem is that I have very limited quota for %outputpath. So to
>>> manage the quota fairly between channels I am using --tracefile-size to
>>> limit the size of each channel and thus preventing them from starving
>>> eachother out.
>>> BUT, say I would want to restart the session for some reason, in my case it
>>> might be that the card running a specific session reboots, and then relayd
>>> would get a new session & channel, and thus create a new folder for that
>>> data. But since the old data still remains, I would run into the quota
>>> limit..
>>>
>>> Below is how the tracepaths are created in pseudo-code:
>>>
>>> per-pid tracing:
>>> tracepath = %outputpath/%pid/%session_name-%pid-%datetime
>>>
>>> per-uid tracing:
>>> tracepath = %outputpath/uid/%uid/%arch-bit
>>>
>>> So just reading this little snippet I am guessing one could avoid this
>>> particular problem by doing per-uid tracing, but that seems like an
>>> unnecessary workaround for what would seem like a trivial problem?
>>>
>>> Would it instead be feasable to have the channel->path_name configurable,
>>> so
>>> that you essentially could avoid the uniqueness of the path (pid &
>>> datetime)?
>>>
>>> Or is that instead opening pandoras box w.r.t. to tracefile consistency and
>>> contention?
>>
>> I *think* this was done to reasonably ensure the uniqueness of the
>> trace path since the consumer and relay daemons will not deal with
>> existing files gracefully.
>>
>> I can see the use case though... It would probably require making the
>> consumer and relay daemons a bit smarter if we want them to pick-up
>> where they left. Nevertheless, it would be a neat feature.
>>
>> In the meantime, I don't really have a good recommendation for you... :-/
>
> Also, in per-PID tracing, we also need to have separate output files for
> each process that runs concurrently. Per-UID tracing is meant to share
> the ring buffer across processes from the same user, which is not the
> case for per-PID tracing.

Well, yes. This was what I was worring about when talking about concurrency above (allthough its not about CPU concurrency but rather file concurrency).

>
> You might want to try erasing the old trace before starting to record
> the new session.

Yeah thats the obvious solution.?
But that introduces other problems on the other hand. Say you are running flight-recorder, your task crashes (and thus you need to restart session), if we would then erase tracefile we would miss the entire flightrecorder session.
Another issue might be like in our case where we are aiming at replacing syslog (in logrotate fashion), with lttng. So if we would have a reboot (graceful or non-graceful), we would have gaps in the tracefile around the interesting events (and usually most issues do happen during startup/shutdown phases, so in this case we would have gaps in the most interesting region-of-interests).

So yeah, your solution solves the quota issue but also introduces some other maybe-not-so-desirable properties.

I think the takeaway here is that to solve this is really not trivial. So we need to go back to the drawing board a little bit and maybe rethink our storing scheme...
It does not sound like "re-use tracefiles" feature is coming anytime soon.. =)

>
> Thanks,
>
> Mathieu
>
>>
>> J?r?mie
>>
>>>
>>> Regards,
>>> Jesper
>>> _______________________________________________
>>> lttng-dev mailing list
>>> lttng-dev at lists.lttng.org
>>> http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
>>>
>>>
>>> _______________________________________________
>>> lttng-dev mailing list
>>> lttng-dev at lists.lttng.org
>>> http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
>>>
>>
>>
>>
>> --
>> J?r?mie Galarneau
>> EfficiOS Inc.
>> http://www.efficios.com
>>
>> _______________________________________________
>> lttng-dev mailing list
>> lttng-dev at lists.lttng.org
>> http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
>>
>
> --
> Mathieu Desnoyers
> EfficiOS Inc.
> http://www.efficios.com