* [lttng-dev] scale numbers for LTTNG
@ 2016-05-24 6:01 Vijay Anand
2016-05-26 12:23 ` Vijay Anand
0 siblings, 1 reply; 6+ messages in thread
From: Vijay Anand @ 2016-05-24 6:01 UTC (permalink / raw)
[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain, Size: 1608 bytes --]
Hello Folks,
We have been evaluating LTTNG for use in our production systems for user
space tracing. We have evaluated most of the features supported by LTTNG
and very much see a value add LTTNG brings into our debugging
infrastructure.
- Our current requirement is to trace Userspace programs running on
linux.
- Each of the linux processes define their own tracepoint providers.
- We would like to trace event histories of each process independently.
- We could potentially have 1000s of such processes running
simultaneously.
- We concluded on using a session/channel to trace one tracepoint
provider corresponding to a unique process.
- But I understand we could also create one system wide session and
use channels to trace each of the providers. Either of the
approaches seems
to work for us.
- But upon evaluating I see that we could create only 25 active
process-sessions and not traces from all the processes are logged.
- Please note I have tried increasing the buffer size and the number
of buffers. This doesn't help.
- Each of the process trace 52 events at and interval of 1 second
each.
- I have evaluated this with lttng in session,live and snapshot modes
and I have not been getting favourable.
Could you folks share the scale numbers that LTTNG supports, especially
when it comes to tracing user space programs ?
Regards,
Vijay
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.lttng.org/pipermail/lttng-dev/attachments/20160524/642feecd/attachment.html>
^ permalink raw reply [flat|nested] 6+ messages in thread* [lttng-dev] scale numbers for LTTNG 2016-05-24 6:01 [lttng-dev] scale numbers for LTTNG Vijay Anand @ 2016-05-26 12:23 ` Vijay Anand 2016-05-26 18:02 ` Jonathan Rajotte 2016-05-26 19:46 ` Mathieu Desnoyers 0 siblings, 2 replies; 6+ messages in thread From: Vijay Anand @ 2016-05-26 12:23 UTC (permalink / raw) [-- Warning: decoded text below may be mangled, UTF-8 assumed --] [-- Attachment #1: Type: text/plain, Size: 1827 bytes --] Could anyone please let us know how do we go about this ? Regards, Vijay On Tue, May 24, 2016 at 11:31 AM, Vijay Anand <vjanandr85 at gmail.com> wrote: > Hello Folks, > > We have been evaluating LTTNG for use in our production systems for user > space tracing. We have evaluated most of the features supported by LTTNG > and very much see a value add LTTNG brings into our debugging > infrastructure. > > - Our current requirement is to trace Userspace programs running on > linux. > - Each of the linux processes define their own tracepoint providers. > - We would like to trace event histories of each process independently. > - We could potentially have 1000s of such processes running > simultaneously. > - We concluded on using a session/channel to trace one tracepoint > provider corresponding to a unique process. > - But I understand we could also create one system wide session and > use channels to trace each of the providers. Either of the approaches seems > to work for us. > - But upon evaluating I see that we could create only 25 active > process-sessions and not traces from all the processes are logged. > - Please note I have tried increasing the buffer size and the > number of buffers. This doesn't help. > - Each of the process trace 52 events at and interval of 1 second > each. > - I have evaluated this with lttng in session,live and snapshot modes > and I have not been getting favourable. > > Could you folks share the scale numbers that LTTNG supports, especially > when it comes to tracing user space programs ? > > Regards, > Vijay > > -------------- next part -------------- An HTML attachment was scrubbed... URL: <https://lists.lttng.org/pipermail/lttng-dev/attachments/20160526/556d7889/attachment.html> ^ permalink raw reply [flat|nested] 6+ messages in thread
* [lttng-dev] scale numbers for LTTNG 2016-05-26 12:23 ` Vijay Anand @ 2016-05-26 18:02 ` Jonathan Rajotte 2016-05-26 19:46 ` Mathieu Desnoyers 1 sibling, 0 replies; 6+ messages in thread From: Jonathan Rajotte @ 2016-05-26 18:02 UTC (permalink / raw) [-- Warning: decoded text below may be mangled, UTF-8 assumed --] [-- Attachment #1: Type: text/plain, Size: 2274 bytes --] Hi Vijay, Could you share with us a test program or details on how you evaluated those limitation ? Thanks On Thu, May 26, 2016 at 8:23 AM, Vijay Anand <vjanandr85 at gmail.com> wrote: > Could anyone please let us know how do we go about this ? > > Regards, > Vijay > > On Tue, May 24, 2016 at 11:31 AM, Vijay Anand <vjanandr85 at gmail.com> > wrote: > >> Hello Folks, >> >> We have been evaluating LTTNG for use in our production systems for user >> space tracing. We have evaluated most of the features supported by LTTNG >> and very much see a value add LTTNG brings into our debugging >> infrastructure. >> >> - Our current requirement is to trace Userspace programs running on >> linux. >> - Each of the linux processes define their own tracepoint providers. >> - We would like to trace event histories of each process >> independently. >> - We could potentially have 1000s of such processes running >> simultaneously. >> - We concluded on using a session/channel to trace one tracepoint >> provider corresponding to a unique process. >> - But I understand we could also create one system wide session >> and use channels to trace each of the providers. Either of the approaches >> seems to work for us. >> - But upon evaluating I see that we could create only 25 active >> process-sessions and not traces from all the processes are logged. >> - Please note I have tried increasing the buffer size and the >> number of buffers. This doesn't help. >> - Each of the process trace 52 events at and interval of 1 second >> each. >> - I have evaluated this with lttng in session,live and snapshot modes >> and I have not been getting favourable. >> >> Could you folks share the scale numbers that LTTNG supports, especially >> when it comes to tracing user space programs ? >> >> Regards, >> Vijay >> >> > > > _______________________________________________ > lttng-dev mailing list > lttng-dev at lists.lttng.org > https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev > > -- Jonathan Rajotte Julien -------------- next part -------------- An HTML attachment was scrubbed... URL: <https://lists.lttng.org/pipermail/lttng-dev/attachments/20160526/64747f61/attachment.html> ^ permalink raw reply [flat|nested] 6+ messages in thread
* [lttng-dev] scale numbers for LTTNG 2016-05-26 12:23 ` Vijay Anand 2016-05-26 18:02 ` Jonathan Rajotte @ 2016-05-26 19:46 ` Mathieu Desnoyers 2016-05-30 7:16 ` Vijay Anand 1 sibling, 1 reply; 6+ messages in thread From: Mathieu Desnoyers @ 2016-05-26 19:46 UTC (permalink / raw) [-- Warning: decoded text below may be mangled, UTF-8 assumed --] [-- Attachment #1: Type: text/plain, Size: 2878 bytes --] ----- On May 26, 2016, at 8:23 AM, Vijay Anand <vjanandr85 at gmail.com> wrote: > Could anyone please let us know how do we go about this ? > Regards, > Vijay > On Tue, May 24, 2016 at 11:31 AM, Vijay Anand < vjanandr85 at gmail.com > wrote: >> Hello Folks, >> We have been evaluating LTTNG for use in our production systems for user space >> tracing. We have evaluated most of the features supported by LTTNG and very >> much see a value add LTTNG brings into our debugging infrastructure. >> * Our current requirement is to trace Userspace programs running on linux. >> * Each of the linux processes define their own tracepoint providers. Sounds like a good design. >> * >> * We would like to trace event histories of each process independently. Why ? You can always collect trace data into buffers shared across processes (per-uid buffers) and filter after the fact. >> * >> * We could potentially have 1000s of such processes running simultaneously. Especially with that many processes, having per-process buffers will degrade your cache locality. >> * >> * We concluded on using a session/channel to trace one tracepoint provider >> corresponding to a unique process. >> * But I understand we could also create one system wide session and use channels >> to trace each of the providers. Either of the approaches seems to work for us. Both approach will kill you cache locality with that many process. I don't recommend either of the two approaches you refer to above. You might want to consider sharing your buffers across processes. >> * But upon evaluating I see that we could create only 25 active process-sessions >> and not traces from all the processes are logged. We will need much more details on your tracing configuration setup (exact list of commands you issue to create your tracing sessions). Also please detail what you mean by " not traces from all the processes are logged". What commands do you issue, what is the exact setup, and what is the output you observe (exact listing) which leads you to reach this conclusion. Thanks, Mathieu >> * Please note I have tried increasing the buffer size and the number of buffers. >> This doesn't help. >> * Each of the process trace 52 events at and interval of 1 second each. >> * >> I have evaluated this with lttng in session,live and snapshot modes and I have >> not been getting favourable. >> Could you folks share the scale numbers that LTTNG supports, especially when it >> comes to tracing user space programs ? >> Regards, >> Vijay >> -- Mathieu Desnoyers EfficiOS Inc. http://www.efficios.com -------------- next part -------------- An HTML attachment was scrubbed... URL: <https://lists.lttng.org/pipermail/lttng-dev/attachments/20160526/c7ad1eb7/attachment-0001.html> ^ permalink raw reply [flat|nested] 6+ messages in thread
* [lttng-dev] scale numbers for LTTNG 2016-05-26 19:46 ` Mathieu Desnoyers @ 2016-05-30 7:16 ` Vijay Anand 2016-05-30 13:03 ` Mathieu Desnoyers 0 siblings, 1 reply; 6+ messages in thread From: Vijay Anand @ 2016-05-30 7:16 UTC (permalink / raw) [-- Warning: decoded text below may be mangled, UTF-8 assumed --] [-- Attachment #1: Type: text/plain, Size: 3706 bytes --] Hi Mathieu, Jonathan, I have written python scripts to simulate our requirements. https://github.com/vjanandr/sampleP/tree/master/lttng Please refer to the README file for further details about the scripts. Mathieu, I understand your recommendation to use per UID buffers instead of per PID buffers. But I dont seem to understand your suggestion to use filters... >>>> Why ? You can always collect trace data into buffers shared across processes (per-uid buffers) and filter after the fact. <<<<< Did you mean to say write all traces to a file but filter them while reading based on maybe process ID ? Regards, Vijay On Fri, May 27, 2016 at 1:16 AM, Mathieu Desnoyers < mathieu.desnoyers at efficios.com> wrote: > > > ----- On May 26, 2016, at 8:23 AM, Vijay Anand <vjanandr85 at gmail.com> > wrote: > > Could anyone please let us know how do we go about this ? > > Regards, > Vijay > > On Tue, May 24, 2016 at 11:31 AM, Vijay Anand <vjanandr85 at gmail.com> > wrote: > >> Hello Folks, >> >> We have been evaluating LTTNG for use in our production systems for user >> space tracing. We have evaluated most of the features supported by LTTNG >> and very much see a value add LTTNG brings into our debugging >> infrastructure. >> >> - Our current requirement is to trace Userspace programs running on >> linux. >> - Each of the linux processes define their own tracepoint providers. >> >> > Sounds like a good design. > > >> - >> - We would like to trace event histories of each process >> independently. >> >> > Why ? You can always collect trace data into buffers shared across > processes (per-uid buffers) > and filter after the fact. > > >> - >> - We could potentially have 1000s of such processes running >> simultaneously. >> >> > Especially with that many processes, having per-process buffers will > degrade your cache > locality. > > >> - >> - We concluded on using a session/channel to trace one tracepoint >> provider corresponding to a unique process. >> - But I understand we could also create one system wide session >> and use channels to trace each of the providers. Either of the approaches >> seems to work for us. >> >> Both approach will kill you cache locality with that many process. I > don't recommend either > of the two approaches you refer to above. You might want to consider > sharing your buffers > across processes. > > >> - But upon evaluating I see that we could create only 25 active >> process-sessions and not traces from all the processes are logged. >> >> We will need much more details on your tracing configuration setup (exact > list of commands you > issue to create your tracing sessions). > > Also please detail what you mean by "not traces from all the processes > are logged". > What commands do you issue, what is the exact setup, and what is the output > you observe (exact listing) which leads you to reach this conclusion. > > Thanks, > > Mathieu > > >> - Please note I have tried increasing the buffer size and the number >> of buffers. This doesn't help. >> - Each of the process trace 52 events at and interval of 1 second >> each. >> - I have evaluated this with lttng in session,live and snapshot modes >> and I have not been getting favourable. >> >> Could you folks share the scale numbers that LTTNG supports, especially >> when it comes to tracing user space programs ? >> >> Regards, >> Vijay >> >> > > > -- > Mathieu Desnoyers > EfficiOS Inc. > http://www.efficios.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: <https://lists.lttng.org/pipermail/lttng-dev/attachments/20160530/c87a333f/attachment.html> ^ permalink raw reply [flat|nested] 6+ messages in thread
* [lttng-dev] scale numbers for LTTNG 2016-05-30 7:16 ` Vijay Anand @ 2016-05-30 13:03 ` Mathieu Desnoyers 0 siblings, 0 replies; 6+ messages in thread From: Mathieu Desnoyers @ 2016-05-30 13:03 UTC (permalink / raw) [-- Warning: decoded text below may be mangled, UTF-8 assumed --] [-- Attachment #1: Type: text/plain, Size: 3942 bytes --] ----- On May 30, 2016, at 9:16 AM, Vijay Anand <vjanandr85 at gmail.com> wrote: > Hi Mathieu, Jonathan, > I have written python scripts to simulate our requirements. > https://github.com/vjanandr/sampleP/tree/master/lttng > Please refer to the README file for further details about the scripts. > Mathieu, I understand your recommendation to use per UID buffers instead of per > PID buffers. > But I dont seem to understand your suggestion to use filters... > Why ? You can always collect trace data into buffers shared across processes > (per-uid buffers) > and filter after the fact. > <<<<< > Did you mean to say write all traces to a file but filter them while reading > based on maybe process ID ? Yes, this is what I mean, Thanks, Mathieu > Regards, > Vijay > On Fri, May 27, 2016 at 1:16 AM, Mathieu Desnoyers < > mathieu.desnoyers at efficios.com > wrote: >> ----- On May 26, 2016, at 8:23 AM, Vijay Anand < vjanandr85 at gmail.com > wrote: >>> Could anyone please let us know how do we go about this ? >>> Regards, >>> Vijay >>> On Tue, May 24, 2016 at 11:31 AM, Vijay Anand < vjanandr85 at gmail.com > wrote: >>>> Hello Folks, >>>> We have been evaluating LTTNG for use in our production systems for user space >>>> tracing. We have evaluated most of the features supported by LTTNG and very >>>> much see a value add LTTNG brings into our debugging infrastructure. >>>> * Our current requirement is to trace Userspace programs running on linux. >>>> * Each of the linux processes define their own tracepoint providers. >> Sounds like a good design. >>>> * >>>> * We would like to trace event histories of each process independently. >> Why ? You can always collect trace data into buffers shared across processes >> (per-uid buffers) >> and filter after the fact. >>>> * >>>> * We could potentially have 1000s of such processes running simultaneously. >> Especially with that many processes, having per-process buffers will degrade >> your cache >> locality. >>>> * >>>> * We concluded on using a session/channel to trace one tracepoint provider >>>> corresponding to a unique process. >>>> * But I understand we could also create one system wide session and use channels >>>> to trace each of the providers. Either of the approaches seems to work for us. >> Both approach will kill you cache locality with that many process. I don't >> recommend either >> of the two approaches you refer to above. You might want to consider sharing >> your buffers >> across processes. >>>> * But upon evaluating I see that we could create only 25 active process-sessions >>>> and not traces from all the processes are logged. >> We will need much more details on your tracing configuration setup (exact list >> of commands you >> issue to create your tracing sessions). >> Also please detail what you mean by " not traces from all the processes are >> logged". >> What commands do you issue, what is the exact setup, and what is the output >> you observe (exact listing) which leads you to reach this conclusion. >> Thanks, >> Mathieu >>>> * Please note I have tried increasing the buffer size and the number of buffers. >>>> This doesn't help. >>>> * Each of the process trace 52 events at and interval of 1 second each. >>>> * >>>> I have evaluated this with lttng in session,live and snapshot modes and I have >>>> not been getting favourable. >>>> Could you folks share the scale numbers that LTTNG supports, especially when it >>>> comes to tracing user space programs ? >>>> Regards, >>>> Vijay >>>> >> -- >> Mathieu Desnoyers >> EfficiOS Inc. >> http://www.efficios.com -- Mathieu Desnoyers EfficiOS Inc. http://www.efficios.com -------------- next part -------------- An HTML attachment was scrubbed... URL: <https://lists.lttng.org/pipermail/lttng-dev/attachments/20160530/e5558c8d/attachment.html> ^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2016-05-30 13:03 UTC | newest] Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2016-05-24 6:01 [lttng-dev] scale numbers for LTTNG Vijay Anand 2016-05-26 12:23 ` Vijay Anand 2016-05-26 18:02 ` Jonathan Rajotte 2016-05-26 19:46 ` Mathieu Desnoyers 2016-05-30 7:16 ` Vijay Anand 2016-05-30 13:03 ` Mathieu Desnoyers
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox