[ltt-dev] limiting LTTng to a certain set of cores

[ltt-dev] limiting LTTng to a certain set of cores

[Andrew McDermott]

Is there an easy way to limit LTTng to trace a specified set of cores
only?  I ask because in some 64+ core systems the data set collected
becomes too large for analysis and the tasks to be analyzed are always
bound to certain cores (using affinity).

I'm anticipating that this would require changes to LTTng - what I would
like to know is how invasive those changes would need to be.

Thanks,
Andy.

[ltt-dev] limiting LTTng to a certain set of cores

[Mathieu Desnoyers]

* Andrew McDermott (andrew.mcdermott at windriver.com) wrote:
> 
> Is there an easy way to limit LTTng to trace a specified set of cores
> only?  I ask because in some 64+ core systems the data set collected
> becomes too large for analysis and the tasks to be analyzed are always
> bound to certain cores (using affinity).
> 
> I'm anticipating that this would require changes to LTTng - what I would
> like to know is how invasive those changes would need to be.
> 

Hi Andrew,

Basically, this could be done on a per tracing-session/per-channel basis
by adding an "active core" bit mask to test within ltt_vtrace and
ltt_serialize_data, in the loop where it iterates on active traces and
calls the "filter" callback. This bitmask would sit within the channel
structure. Setting the active cores should be added to the debugfs
interface :

People would do :
echo tracename > /mnt/debugfs/ltt/setup_trace
echo CPUNR > /mnt/debugfs/ltt/control/tracename/channel/channelname/cpus
cat /mnt/debugfs/ltt/control/tracename/channel/channelname/cpus
...

Maybe we should replace the "enable" test by a cpu mask test with all
cpus active to save some cycles...

And I don't particularly care about the exact debugfs file interface
(file name, the way the cpus are listed). Ideally if we can find
somewhere else in the Linux kernel where it is already done, that would
be a great inspiration source.

Thanks,

Mathieu

> Thanks,
> Andy.
> 
> _______________________________________________
> ltt-dev mailing list
> ltt-dev at lists.casi.polymtl.ca
> http://lists.casi.polymtl.ca/cgi-bin/mailman/listinfo/ltt-dev
> 

-- 
Mathieu Desnoyers
OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68

[ltt-dev] limiting LTTng to a certain set of cores

[Andrew McDermott]

> * Andrew McDermott (andrew.mcdermott at windriver.com) wrote:
>> 
>> Is there an easy way to limit LTTng to trace a specified set of cores
>> only?  I ask because in some 64+ core systems the data set collected
>> becomes too large for analysis and the tasks to be analyzed are always
>> bound to certain cores (using affinity).
>> 
>> I'm anticipating that this would require changes to LTTng - what I would
>> like to know is how invasive those changes would need to be.
>> 
>
> Hi Andrew,
>
> Basically, this could be done on a per tracing-session/per-channel basis
> by adding an "active core" bit mask to test within ltt_vtrace and
> ltt_serialize_data, in the loop where it iterates on active traces and
> calls the "filter" callback. This bitmask would sit within the channel
> structure. Setting the active cores should be added to the debugfs
> interface :

Thanks for the info!

Perhaps it is something to consider for a future release.

One of the issues I'm facing is that the trace is large and by large I
mean I have to transfer the trace in its entirety from a remote
location.  If I could collect on a subset of the cores then I would have
less to transfer (over my very slow link).

>
> People would do :
> echo tracename > /mnt/debugfs/ltt/setup_trace
> echo CPUNR > /mnt/debugfs/ltt/control/tracename/channel/channelname/cpus
> cat /mnt/debugfs/ltt/control/tracename/channel/channelname/cpus
> ...
>
> Maybe we should replace the "enable" test by a cpu mask test with all
> cpus active to save some cycles...
>
> And I don't particularly care about the exact debugfs file interface
> (file name, the way the cpus are listed). Ideally if we can find
> somewhere else in the Linux kernel where it is already done, that would
> be a great inspiration source.
>
> Thanks,
>
> Mathieu
>
>> Thanks,
>> Andy.
>> 
>> _______________________________________________
>> ltt-dev mailing list
>> ltt-dev at lists.casi.polymtl.ca
>> http://lists.casi.polymtl.ca/cgi-bin/mailman/listinfo/ltt-dev
>> 

-- 
andy

[ltt-dev] limiting LTTng to a certain set of cores

[Pierre-Marc Fournier]

Hi Andrew,

Andrew McDermott wrote:
> Is there an easy way to limit LTTng to trace a specified set of cores
> only?  I ask because in some 64+ core systems the data set collected
> becomes too large for analysis and the tasks to be analyzed are always
> bound to certain cores (using affinity).
> 

Out of curiosity, how large is "too large"?

[ltt-dev] limiting LTTng to a certain set of cores

[Andrew McDermott]

Hi,

> Andrew McDermott wrote:
>> Is there an easy way to limit LTTng to trace a specified set of cores
>> only?  I ask because in some 64+ core systems the data set collected
>> becomes too large for analysis and the tasks to be analyzed are always
>> bound to certain cores (using affinity).
>> 
>
> Out of curiosity, how large is "too large"?

The "too large" actually refers to the volume or quantiy of (displayed)
events.  There are a lot of event related to other cores that are of no
interest.  There's no real issue with processing a multi-gigabyte trace
on the host but there are issues with transferring all 64+ channel trace
files across a very slow link.

What I'm trying to do is filter the set of data that a) needs to be
collected and b) transferred half-way around the world.

-- 
andy