[lttng-dev] Sessions are disconnected when disk is full on remote host

[lttng-dev] Sessions are disconnected when disk is full on remote host

[Mikael Beckius]

Hello!

I am trying to verify the expected behaviour in the event a machine hosting an lttng relay daemon runs out of disk space.

Currently it appears from testing and inspecting the source code that whenever the lttng relay daemon hits an error caused by the lack of disk space all failing sessions will be disconnected. It also appears that the only way to recover from this situation once disk space has been restored is to recreate all the disconnected sessions.

I have also played around with different combinations of the below lttng enable-channel options:
--discard
--overwrite
--tracefile-count <trace file count>
--tracefile-size <trace file size>

The initial channel configuration was using the options --tracefile-count --tracefile-size but both were abandoned as Babeltrace faced constant streaming errors during heavy live tracing and now the trace files are growing continuously until there is no available disk space.

I assume that --overwrite would not make any difference compared to --discard with regards to a disk full scenario. Even if you specify --tracefile-count and --tracefile-size to limit the amount of trace data from a single session you may still face the same issue with disconnected sessions if you for some reason run out of disk space.

Testing was carried out on the latest 2.6 releases and all sessions were live sessions.

So my questions are:
- Is there any way to keep sessions alive even if you run out of disk space?
- How can you clean up old traces once you have collected them without recreating sessions?

Micke
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lttng.org/pipermail/lttng-dev/attachments/20150923/7d07e1ef/attachment.html>

[lttng-dev] Sessions are disconnected when disk is full on remote host

[Jérémie Galarneau]

On Wed, Sep 23, 2015 at 5:48 AM, Mikael Beckius
<mikael.beckius at ericsson.com> wrote:
> Hello!
>
> I am trying to verify the expected behaviour in the event a machine hosting
> an lttng relay daemon runs out of disk space.
>
> Currently it appears from testing and inspecting the source code that
> whenever the lttng relay daemon hits an error caused by the lack of disk
> space all failing sessions will be disconnected. It also appears that the
> only way to recover from this situation once disk space has been restored is
> to recreate all the disconnected sessions.
>
> I have also played around with different combinations of the below lttng
> enable-channel options:
> --discard
> --overwrite
> --tracefile-count <trace file count>
> --tracefile-size <trace file size>
>
> The initial channel configuration was using the options --tracefile-count
> --tracefile-size but both were abandoned as Babeltrace faced constant
> streaming errors during heavy live tracing and now the trace files are
> growing continuously until there is no available disk space.
>
> I assume that --overwrite would not make any difference compared to
> --discard with regards to a disk full scenario. Even if you specify
> --tracefile-count and --tracefile-size to limit the amount of trace data
> from a single session you may still face the same issue with disconnected
> sessions if you for some reason run out of disk space.
>
> Testing was carried out on the latest 2.6 releases and all sessions were
> live sessions.
>
> So my questions are:
> - Is there any way to keep sessions alive even if you run out of disk space?
> - How can you clean up old traces once you have collected them without
> recreating sessions?

Hi Mikael,

LTTng currently doesn't support these recovery modes.

However, the file rotation issues you mentioned should be resolved in
lttng-tools 2.6.1. Any chance you can give it a try?
The website will be updated later today. In the meantime, here's a
direct link to the release:

https://lttng.org/files/lttng-tools/lttng-tools-2.6.1.tar.bz2

Regards,
J?r?mie

>
> Micke
>
> _______________________________________________
> lttng-dev mailing list
> lttng-dev at lists.lttng.org
> http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
>



-- 
J?r?mie Galarneau
EfficiOS Inc.
http://www.efficios.com

[lttng-dev] Sessions are disconnected when disk is full on remote host

[Mikael Beckius]

________________________________________
Fr?n: jeremie.galarneau at gmail.com [jeremie.galarneau at gmail.com] för J?r?mie Galarneau [jeremie.galarneau at efficios.com]
Skickat: den 25 september 2015 21:50
Till: Mikael Beckius
Kopia: lttng-dev at lists.lttng.org
?mne: Re: [lttng-dev] Sessions are disconnected when disk is full on remote host

On Wed, Sep 23, 2015 at 5:48 AM, Mikael Beckius
<mikael.beckius at ericsson.com> wrote:
> Hello!
>
> I am trying to verify the expected behaviour in the event a machine hosting
> an lttng relay daemon runs out of disk space.
>
> Currently it appears from testing and inspecting the source code that
> whenever the lttng relay daemon hits an error caused by the lack of disk
> space all failing sessions will be disconnected. It also appears that the
> only way to recover from this situation once disk space has been restored is
> to recreate all the disconnected sessions.
>
> I have also played around with different combinations of the below lttng
> enable-channel options:
> --discard
> --overwrite
> --tracefile-count <trace file count>
> --tracefile-size <trace file size>
>
> The initial channel configuration was using the options --tracefile-count
> --tracefile-size but both were abandoned as Babeltrace faced constant
> streaming errors during heavy live tracing and now the trace files are
> growing continuously until there is no available disk space.
>
> I assume that --overwrite would not make any difference compared to
> --discard with regards to a disk full scenario. Even if you specify
> --tracefile-count and --tracefile-size to limit the amount of trace data
> from a single session you may still face the same issue with disconnected
> sessions if you for some reason run out of disk space.
>
> Testing was carried out on the latest 2.6 releases and all sessions were
> live sessions.
>
> So my questions are:
> - Is there any way to keep sessions alive even if you run out of disk space?
> - How can you clean up old traces once you have collected them without
> recreating sessions?

Hi Mikael,

LTTng currently doesn't support these recovery modes.

However, the file rotation issues you mentioned should be resolved in
lttng-tools 2.6.1. Any chance you can give it a try?
The website will be updated later today. In the meantime, here's a
direct link to the release:

https://lttng.org/files/lttng-tools/lttng-tools-2.6.1.tar.bz2

Regards,
J?r?mie

Hello J?r?mie!

Thanks for the hint. I will give the new 2.6.1 and 2.7.0 releases of lttng-tools a try and see if they work better than 2.6.0.

Micke

>
> Micke
>
> _______________________________________________
> lttng-dev mailing list
> lttng-dev at lists.lttng.org
> http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
>



--
J?r?mie Galarneau
EfficiOS Inc.
http://www.efficios.com