From mboxrd@z Thu Jan 1 00:00:00 1970 From: mathieu.desnoyers@efficios.com (Mathieu Desnoyers) Date: Tue, 26 Nov 2013 19:25:50 +0000 (UTC) Subject: [lttng-dev] Tmf Ctf Parser Scalability issues (1/3) In-Reply-To: <5294CFFE.2070001@ericsson.com> References: <5294CFFE.2070001@ericsson.com> Message-ID: <460888779.76255.1385493950838.JavaMail.zimbra@efficios.com> ----- Original Message ----- > From: "Matthew Khouzam" > To: lttng-dev at lists.lttng.org, "Linux Tools developer discussions" > Sent: Tuesday, November 26, 2013 11:44:46 AM > Subject: [lttng-dev] Tmf Ctf Parser Scalability issues (1/3) > > Hello all, > I was looking at the CTF parser in the TMF project of Linux Tools. I > have come up with three points where scalability will be an issue. I am > sending 3 emails, each one describing one of the issues so we can > aggregate them more cohesively. First a primer, CTF is a file format > that has traces written in packets, the packets are parts of files in > streams, the streams are files in a directory. > > Issue 1: Number of open files > CTF can have an unlimited amount of stream files. The parser opens all > of them simultaneously. This means if you have say over 1024 or whatever > your file limit is, files opened simultaneously, you will hit a wall. An > example of this would be the new Xeon Phi boards that have something > like 255 cores, if you have more than 4 channels per core, you're going > to hit this issue. > > Proposed solution: > I envision fixing them by setting up a pool of files and when the limit > is hit, swapping files in and out, incurring a performance hit but still > allowing the program to run. Any objections? No objection. I would think a LRU list of most recently used files would be a good data structure for this. I already described this in a bug against babeltrace: http://bugs.lttng.org/issues/649 Thanks! Mathieu > > _______________________________________________ > lttng-dev mailing list > lttng-dev at lists.lttng.org > http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev > -- Mathieu Desnoyers EfficiOS Inc. http://www.efficios.com