From mboxrd@z Thu Jan 1 00:00:00 1970 From: jeremie.galarneau@efficios.com (=?UTF-8?Q?J=C3=A9r=C3=A9mie_Galarneau?=) Date: Fri, 25 Sep 2015 15:50:55 -0400 Subject: [lttng-dev] Sessions are disconnected when disk is full on remote host In-Reply-To: References: Message-ID: On Wed, Sep 23, 2015 at 5:48 AM, Mikael Beckius wrote: > Hello! > > I am trying to verify the expected behaviour in the event a machine hosting > an lttng relay daemon runs out of disk space. > > Currently it appears from testing and inspecting the source code that > whenever the lttng relay daemon hits an error caused by the lack of disk > space all failing sessions will be disconnected. It also appears that the > only way to recover from this situation once disk space has been restored is > to recreate all the disconnected sessions. > > I have also played around with different combinations of the below lttng > enable-channel options: > --discard > --overwrite > --tracefile-count > --tracefile-size > > The initial channel configuration was using the options --tracefile-count > --tracefile-size but both were abandoned as Babeltrace faced constant > streaming errors during heavy live tracing and now the trace files are > growing continuously until there is no available disk space. > > I assume that --overwrite would not make any difference compared to > --discard with regards to a disk full scenario. Even if you specify > --tracefile-count and --tracefile-size to limit the amount of trace data > from a single session you may still face the same issue with disconnected > sessions if you for some reason run out of disk space. > > Testing was carried out on the latest 2.6 releases and all sessions were > live sessions. > > So my questions are: > - Is there any way to keep sessions alive even if you run out of disk space? > - How can you clean up old traces once you have collected them without > recreating sessions? Hi Mikael, LTTng currently doesn't support these recovery modes. However, the file rotation issues you mentioned should be resolved in lttng-tools 2.6.1. Any chance you can give it a try? The website will be updated later today. In the meantime, here's a direct link to the release: https://lttng.org/files/lttng-tools/lttng-tools-2.6.1.tar.bz2 Regards, J?r?mie > > Micke > > _______________________________________________ > lttng-dev mailing list > lttng-dev at lists.lttng.org > http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev > -- J?r?mie Galarneau EfficiOS Inc. http://www.efficios.com