From mboxrd@z Thu Jan  1 00:00:00 1970
From: matthew.khouzam@ericsson.com (Matthew Khouzam)
Date: Tue, 26 Nov 2013 11:44:48 -0500
Subject: [lttng-dev] Tmf Ctf Parser Scalability issues (2/3)
Message-ID: <5294D000.60505@ericsson.com>

Hello all,
I was looking at the CTF parser in the TMF project of Linux Tools. I
have come up with three points where scalability will be an issue. I am
sending 3 emails, each one describing one of the issues so we can
aggregate them more cohesively. First a primer, CTF is a file format
that has traces written in packets, the packets are parts of files in
streams, the streams are files in a directory.

Issue 2: Packet size
CTF can have an packet with an unlimited size in within a stream. We use
a memory map to access the data. We are limited to 2GB as the size of an
individual packet, as far as I know. If a packet is larger this will
make the trace unreadable. I can imagine hardware tracers that have ~3gb
buffers and dump them to a file in a single packet would be affected by
this.

Proposed solution:
I envision fixing them by having a sliding window of the maximum memory
map size. I see a problem if a single event is say 3GB in size, but I
can't see a short term solution for that, would you have any suggestions
on this front?