From mboxrd@z Thu Jan 1 00:00:00 1970 From: mikael.beckius@ericsson.com (Mikael Beckius) Date: Tue, 29 Sep 2015 08:06:19 +0000 Subject: [lttng-dev] Sessions are disconnected when disk is full on remote host In-Reply-To: References: , Message-ID: ________________________________________ Fr?n: jeremie.galarneau at gmail.com [jeremie.galarneau at gmail.com] för J?r?mie Galarneau [jeremie.galarneau at efficios.com] Skickat: den 25 september 2015 21:50 Till: Mikael Beckius Kopia: lttng-dev at lists.lttng.org ?mne: Re: [lttng-dev] Sessions are disconnected when disk is full on remote host On Wed, Sep 23, 2015 at 5:48 AM, Mikael Beckius wrote: > Hello! > > I am trying to verify the expected behaviour in the event a machine hosting > an lttng relay daemon runs out of disk space. > > Currently it appears from testing and inspecting the source code that > whenever the lttng relay daemon hits an error caused by the lack of disk > space all failing sessions will be disconnected. It also appears that the > only way to recover from this situation once disk space has been restored is > to recreate all the disconnected sessions. > > I have also played around with different combinations of the below lttng > enable-channel options: > --discard > --overwrite > --tracefile-count > --tracefile-size > > The initial channel configuration was using the options --tracefile-count > --tracefile-size but both were abandoned as Babeltrace faced constant > streaming errors during heavy live tracing and now the trace files are > growing continuously until there is no available disk space. > > I assume that --overwrite would not make any difference compared to > --discard with regards to a disk full scenario. Even if you specify > --tracefile-count and --tracefile-size to limit the amount of trace data > from a single session you may still face the same issue with disconnected > sessions if you for some reason run out of disk space. > > Testing was carried out on the latest 2.6 releases and all sessions were > live sessions. > > So my questions are: > - Is there any way to keep sessions alive even if you run out of disk space? > - How can you clean up old traces once you have collected them without > recreating sessions? Hi Mikael, LTTng currently doesn't support these recovery modes. However, the file rotation issues you mentioned should be resolved in lttng-tools 2.6.1. Any chance you can give it a try? The website will be updated later today. In the meantime, here's a direct link to the release: https://lttng.org/files/lttng-tools/lttng-tools-2.6.1.tar.bz2 Regards, J?r?mie Hello J?r?mie! Thanks for the hint. I will give the new 2.6.1 and 2.7.0 releases of lttng-tools a try and see if they work better than 2.6.0. Micke > > Micke > > _______________________________________________ > lttng-dev mailing list > lttng-dev at lists.lttng.org > http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev > -- J?r?mie Galarneau EfficiOS Inc. http://www.efficios.com