[lttng-dev] scale numbers for LTTNG

[lttng-dev] scale numbers for LTTNG

[Vijay Anand]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain, Size: 1608 bytes --]

Hello Folks,

We have been evaluating LTTNG for use in our production systems for user
space tracing. We have evaluated most of the features supported by LTTNG
and very much see a value add LTTNG brings into our debugging
infrastructure.

   - Our current requirement is to trace Userspace programs running on
   linux.
   - Each of the linux processes define their own tracepoint providers.
   - We would like to trace event histories of each process independently.
   - We could potentially have 1000s of such processes running
   simultaneously.
   - We concluded on using a session/channel to trace one tracepoint
   provider corresponding to a unique process.
      - But I understand we could also create one system wide session and
      use channels to trace each of the providers. Either of the
approaches seems
      to work for us.
   - But upon evaluating I see that we could create only 25 active
   process-sessions and not traces from all the processes are logged.
      - Please note I have tried increasing the buffer size and the number
      of buffers. This doesn't help.
      - Each of the process trace 52 events at and interval of 1 second
      each.
   - I have evaluated this with lttng in session,live and snapshot modes
   and I have not been getting favourable.

Could you folks share the scale numbers that LTTNG supports, especially
when it comes to tracing user space programs ?

Regards,
Vijay
​​
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.lttng.org/pipermail/lttng-dev/attachments/20160524/642feecd/attachment.html>

[lttng-dev] scale numbers for LTTNG

[Vijay Anand]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain, Size: 1827 bytes --]

Could anyone please let us know how do we go about this ?

Regards,
Vijay

On Tue, May 24, 2016 at 11:31 AM, Vijay Anand <vjanandr85 at gmail.com> wrote:

> Hello Folks,
>
> We have been evaluating LTTNG for use in our production systems for user
> space tracing. We have evaluated most of the features supported by LTTNG
> and very much see a value add LTTNG brings into our debugging
> infrastructure.
>
>    - Our current requirement is to trace Userspace programs running on
>    linux.
>    - Each of the linux processes define their own tracepoint providers.
>    - We would like to trace event histories of each process independently.
>    - We could potentially have 1000s of such processes running
>    simultaneously.
>    - We concluded on using a session/channel to trace one tracepoint
>    provider corresponding to a unique process.
>       - But I understand we could also create one system wide session and
>       use channels to trace each of the providers. Either of the approaches seems
>       to work for us.
>    - But upon evaluating I see that we could create only 25 active
>    process-sessions and not traces from all the processes are logged.
>       - Please note I have tried increasing the buffer size and the
>       number of buffers. This doesn't help.
>       - Each of the process trace 52 events at and interval of 1 second
>       each.
>    - I have evaluated this with lttng in session,live and snapshot modes
>    and I have not been getting favourable.
>
> Could you folks share the scale numbers that LTTNG supports, especially
> when it comes to tracing user space programs ?
>
> Regards,
> Vijay
> ​​
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.lttng.org/pipermail/lttng-dev/attachments/20160526/556d7889/attachment.html>

[lttng-dev] scale numbers for LTTNG

[Jonathan Rajotte]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain, Size: 2274 bytes --]

Hi Vijay,

Could you share with us a test program or details on how you evaluated
those limitation ?

Thanks

On Thu, May 26, 2016 at 8:23 AM, Vijay Anand <vjanandr85 at gmail.com> wrote:

> Could anyone please let us know how do we go about this ?
>
> Regards,
> Vijay
>
> On Tue, May 24, 2016 at 11:31 AM, Vijay Anand <vjanandr85 at gmail.com>
> wrote:
>
>> Hello Folks,
>>
>> We have been evaluating LTTNG for use in our production systems for user
>> space tracing. We have evaluated most of the features supported by LTTNG
>> and very much see a value add LTTNG brings into our debugging
>> infrastructure.
>>
>>    - Our current requirement is to trace Userspace programs running on
>>    linux.
>>    - Each of the linux processes define their own tracepoint providers.
>>    - We would like to trace event histories of each process
>>    independently.
>>    - We could potentially have 1000s of such processes running
>>    simultaneously.
>>    - We concluded on using a session/channel to trace one tracepoint
>>    provider corresponding to a unique process.
>>       - But I understand we could also create one system wide session
>>       and use channels to trace each of the providers. Either of the approaches
>>       seems to work for us.
>>    - But upon evaluating I see that we could create only 25 active
>>    process-sessions and not traces from all the processes are logged.
>>       - Please note I have tried increasing the buffer size and the
>>       number of buffers. This doesn't help.
>>       - Each of the process trace 52 events at and interval of 1 second
>>       each.
>>    - I have evaluated this with lttng in session,live and snapshot modes
>>    and I have not been getting favourable.
>>
>> Could you folks share the scale numbers that LTTNG supports, especially
>> when it comes to tracing user space programs ?
>>
>> Regards,
>> Vijay
>> ​​
>>
>
>
> _______________________________________________
> lttng-dev mailing list
> lttng-dev at lists.lttng.org
> https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
>
>


-- 
Jonathan Rajotte Julien
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.lttng.org/pipermail/lttng-dev/attachments/20160526/64747f61/attachment.html>

[lttng-dev] scale numbers for LTTNG

[Mathieu Desnoyers]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain, Size: 2878 bytes --]

----- On May 26, 2016, at 8:23 AM, Vijay Anand <vjanandr85 at gmail.com> wrote: 

> Could anyone please let us know how do we go about this ?

> Regards,
> Vijay

> On Tue, May 24, 2016 at 11:31 AM, Vijay Anand < vjanandr85 at gmail.com > wrote:

>> Hello Folks,

>> We have been evaluating LTTNG for use in our production systems for user space
>> tracing. We have evaluated most of the features supported by LTTNG and very
>> much see a value add LTTNG brings into our debugging infrastructure.

>>     * Our current requirement is to trace Userspace programs running on linux.
>>     * Each of the linux processes define their own tracepoint providers.

Sounds like a good design. 

>>     *
>>     * We would like to trace event histories of each process independently.

Why ? You can always collect trace data into buffers shared across processes (per-uid buffers) 
and filter after the fact. 

>>     *
>>     * We could potentially have 1000s of such processes running simultaneously.

Especially with that many processes, having per-process buffers will degrade your cache 
locality. 

>>     *
>>    * We concluded on using a session/channel to trace one tracepoint provider
>>     corresponding to a unique process.

>>        * But I understand we could also create one system wide session and use channels
>>         to trace each of the providers. Either of the approaches seems to work for us.

Both approach will kill you cache locality with that many process. I don't recommend either 
of the two approaches you refer to above. You might want to consider sharing your buffers 
across processes. 

>>    * But upon evaluating I see that we could create only 25 active process-sessions
>>     and not traces from all the processes are logged.

We will need much more details on your tracing configuration setup (exact list of commands you 
issue to create your tracing sessions). 

Also please detail what you mean by " not traces from all the processes are logged". 
What commands do you issue, what is the exact setup, and what is the output 
you observe (exact listing) which leads you to reach this conclusion. 

Thanks, 

Mathieu 

>>        * Please note I have tried increasing the buffer size and the number of buffers.
>>         This doesn't help.
>>         * Each of the process trace 52 events at and interval of 1 second each.
>>     *
>> I have evaluated this with lttng in session,live and snapshot modes and I have
>> not been getting favourable.

>> Could you folks share the scale numbers that LTTNG supports, especially when it
>> comes to tracing user space programs ?

>> Regards,
>> Vijay
>> ​​

-- 
Mathieu Desnoyers 
EfficiOS Inc. 
http://www.efficios.com 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.lttng.org/pipermail/lttng-dev/attachments/20160526/c7ad1eb7/attachment-0001.html>

[lttng-dev] scale numbers for LTTNG

[Vijay Anand]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain, Size: 3706 bytes --]

Hi Mathieu, Jonathan,

I have written python scripts to simulate our requirements.
https://github.com/vjanandr/sampleP/tree/master/lttng

Please refer to the README file for further details about the scripts.

Mathieu, I understand your recommendation to use per UID buffers instead of
per PID buffers.
But I dont seem to understand your suggestion to use filters...

>>>>
Why ? You can always collect trace data into buffers shared across
processes (per-uid buffers)
and filter after the fact.
<<<<<

Did you mean to say write all traces to a file but filter them while
reading based on maybe process ID ?

Regards,
Vijay

On Fri, May 27, 2016 at 1:16 AM, Mathieu Desnoyers <
mathieu.desnoyers at efficios.com> wrote:

>
>
> ----- On May 26, 2016, at 8:23 AM, Vijay Anand <vjanandr85 at gmail.com>
> wrote:
>
> Could anyone please let us know how do we go about this ?
>
> Regards,
> Vijay
>
> On Tue, May 24, 2016 at 11:31 AM, Vijay Anand <vjanandr85 at gmail.com>
> wrote:
>
>> Hello Folks,
>>
>> We have been evaluating LTTNG for use in our production systems for user
>> space tracing. We have evaluated most of the features supported by LTTNG
>> and very much see a value add LTTNG brings into our debugging
>> infrastructure.
>>
>>    - Our current requirement is to trace Userspace programs running on
>>    linux.
>>    - Each of the linux processes define their own tracepoint providers.
>>
>>
> Sounds like a good design.
>
>
>>    -
>>    - We would like to trace event histories of each process
>>    independently.
>>
>>
> Why ? You can always collect trace data into buffers shared across
> processes (per-uid buffers)
> and filter after the fact.
>
>
>>    -
>>    - We could potentially have 1000s of such processes running
>>    simultaneously.
>>
>>
> Especially with that many processes, having per-process buffers will
> degrade your cache
> locality.
>
>
>>    -
>>    - We concluded on using a session/channel to trace one tracepoint
>>    provider corresponding to a unique process.
>>       - But I understand we could also create one system wide session
>>       and use channels to trace each of the providers. Either of the approaches
>>       seems to work for us.
>>
>> Both approach will kill you cache locality with that many process. I
> don't recommend either
> of the two approaches you refer to above. You might want to consider
> sharing your buffers
> across processes.
>
>
>>    - But upon evaluating I see that we could create only 25 active
>>    process-sessions and not traces from all the processes are logged.
>>
>> We will need much more details on your tracing configuration setup (exact
> list of commands you
> issue to create your tracing sessions).
>
> Also please detail what you mean by "not traces from all the processes
> are logged".
> What commands do you issue, what is the exact setup, and what is the output
> you observe (exact listing) which leads you to reach this conclusion.
>
> Thanks,
>
> Mathieu
>
>
>>    - Please note I have tried increasing the buffer size and the number
>>       of buffers. This doesn't help.
>>       - Each of the process trace 52 events at and interval of 1 second
>>       each.
>>    - I have evaluated this with lttng in session,live and snapshot modes
>>    and I have not been getting favourable.
>>
>> Could you folks share the scale numbers that LTTNG supports, especially
>> when it comes to tracing user space programs ?
>>
>> Regards,
>> Vijay
>> ​​
>>
>
>
> --
> Mathieu Desnoyers
> EfficiOS Inc.
> http://www.efficios.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.lttng.org/pipermail/lttng-dev/attachments/20160530/c87a333f/attachment.html>

[lttng-dev] scale numbers for LTTNG

[Mathieu Desnoyers]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain, Size: 3942 bytes --]

----- On May 30, 2016, at 9:16 AM, Vijay Anand <vjanandr85 at gmail.com> wrote: 

> Hi Mathieu, Jonathan,

> I have written python scripts to simulate our requirements.
> https://github.com/vjanandr/sampleP/tree/master/lttng

> Please refer to the README file for further details about the scripts.

> Mathieu, I understand your recommendation to use per UID buffers instead of per
> PID buffers.
> But I dont seem to understand your suggestion to use filters...


> Why ? You can always collect trace data into buffers shared across processes
> (per-uid buffers)
> and filter after the fact.
> <<<<<

> Did you mean to say write all traces to a file but filter them while reading
> based on maybe process ID ?

Yes, this is what I mean, 

Thanks, 

Mathieu 

> Regards,
> Vijay

> On Fri, May 27, 2016 at 1:16 AM, Mathieu Desnoyers <
> mathieu.desnoyers at efficios.com > wrote:

>> ----- On May 26, 2016, at 8:23 AM, Vijay Anand < vjanandr85 at gmail.com > wrote:

>>> Could anyone please let us know how do we go about this ?

>>> Regards,
>>> Vijay

>>> On Tue, May 24, 2016 at 11:31 AM, Vijay Anand < vjanandr85 at gmail.com > wrote:

>>>> Hello Folks,

>>>> We have been evaluating LTTNG for use in our production systems for user space
>>>> tracing. We have evaluated most of the features supported by LTTNG and very
>>>> much see a value add LTTNG brings into our debugging infrastructure.

>>>>     * Our current requirement is to trace Userspace programs running on linux.
>>>>     * Each of the linux processes define their own tracepoint providers.

>> Sounds like a good design.

>>>>     *
>>>>     * We would like to trace event histories of each process independently.

>> Why ? You can always collect trace data into buffers shared across processes
>> (per-uid buffers)
>> and filter after the fact.

>>>>     *
>>>>     * We could potentially have 1000s of such processes running simultaneously.

>> Especially with that many processes, having per-process buffers will degrade
>> your cache
>> locality.

>>>>     *
>>>>    * We concluded on using a session/channel to trace one tracepoint provider
>>>>     corresponding to a unique process.

>>>>        * But I understand we could also create one system wide session and use channels
>>>>         to trace each of the providers. Either of the approaches seems to work for us.

>> Both approach will kill you cache locality with that many process. I don't
>> recommend either
>> of the two approaches you refer to above. You might want to consider sharing
>> your buffers
>> across processes.

>>>>    * But upon evaluating I see that we could create only 25 active process-sessions
>>>>     and not traces from all the processes are logged.

>> We will need much more details on your tracing configuration setup (exact list
>> of commands you
>> issue to create your tracing sessions).

>> Also please detail what you mean by " not traces from all the processes are
>> logged".
>> What commands do you issue, what is the exact setup, and what is the output
>> you observe (exact listing) which leads you to reach this conclusion.

>> Thanks,

>> Mathieu

>>>>        * Please note I have tried increasing the buffer size and the number of buffers.
>>>>         This doesn't help.
>>>>         * Each of the process trace 52 events at and interval of 1 second each.
>>>>     *
>>>> I have evaluated this with lttng in session,live and snapshot modes and I have
>>>> not been getting favourable.

>>>> Could you folks share the scale numbers that LTTNG supports, especially when it
>>>> comes to tracing user space programs ?

>>>> Regards,
>>>> Vijay
>>>> ​​

>> --
>> Mathieu Desnoyers
>> EfficiOS Inc.
>> http://www.efficios.com

-- 
Mathieu Desnoyers 
EfficiOS Inc. 
http://www.efficios.com 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.lttng.org/pipermail/lttng-dev/attachments/20160530/e5558c8d/attachment.html>