[lttng-dev] Unable to import big LTTng traces

[lttng-dev] Unable to import big LTTng traces

[Matthew Khouzam]

There is a limit to the actual size of the trace being imported
depending on your ram, the parser reads an index and keeps it in memory.
This requires approx 0.004% of the trace size using default settings.
This means if you load a 1 tb trace, you need 400 mb of ram. 400 mb of
ram needs about 16k of ram plus overhead so let's say 1-2 mb. Is it a
kernel trace? if so then you're also building the state system at the
same time and the statistics... this will take a short while. Normally
under 2 minutes (I have a core 2 duo laptop with a 5400 rpm hdd and it
takes 2min for 433 mb with no state system) , so a coffee/tea break,
when you're back, you will have lightning fast views. :)

Maybe we should put a more descriptive please wait message.


On 12-12-13 01:08 PM, Alexandre Montplaisir wrote:
> On 12-12-12 12:59 AM, Radhika Chowdary wrote:
>> Hi
>>
>> I am using linuxtools plugins for importing LTTng traces. I have a trace
>> folder of size 400MB and when I try to import it in the Traces folder it
>> takes for ever to import it. At the bottom left corner I see Indexing...
>> Events view gets populated soon but not others.
> Hi,
>
> It is expected that the indexing phase takes a while for very large
> trace. It could probably be optimized a bit too, since we haven't spent
> much time on performance improvements. Good news is that once indexed,
> it will be faster to reopen the same trace later on.
>
> We have not yet implemented a percentage progression monitor for the
> indexing job. This is not trivial since it has to be implemented
> differently depending on the trace type, but would definitely be nice to
> have.
>
> If you use the latest master branch, the timestamps for the start and
> end of the histogram are now correctly formatted, so it is possible to
> know roughly where the indexing is at, provided you know the end time of
> your trace. I'd suggest trying that out to make sure it's not stopping
> at some point, which would be a bug.
>
>> Is there a limit on the size of traces that it can import?
> There is no theoretical limit on the trace size that can be imported. It
> is capped only by your patience ;)  and the amount of available disk
> space, since the index files can be quite big too.
>
>
> Cheers,
> Alex
>
>
> _______________________________________________
> lttng-dev mailing list
> lttng-dev at lists.lttng.org
> http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev