[lttng-dev] lttng enable-channel option for blocking

[lttng-dev] lttng enable-channel option for blocking

[Woegerer, Paul]

There are use-cases where loosing an event is not acceptable.

Is there some way to make a userspace application block if the buffer is 
full (instead of discarding the event or overwriting an old event) ?

I'm thinking about something like:

lttng enable-channel myblockingchannel --block

I know I can increase the subbuf-size but sometimes this is not an 
option (embedded targets with less RAM).

A new channel option like --block would be a fine complement for the 
already existing options --discard and --overwrite.

What do you think ?
Thanks,
Paul

-- 
Paul Woegerer | SW Development Engineer
Mentor Embedded(tm) | Prinz Eugen Stra?e 72/2/4, Vienna, 1040 Austria
P 43.1.535991320
Nucleus? | Linux? | Android(tm) | Services | UI | Multi-OS

Android is a trademark of Google Inc. Use of this trademark is subject to Google Permissions.
Linux is the registered trademark of Linus Torvalds in the U.S. and other countries.

[lttng-dev] lttng enable-channel option for blocking

[Mathieu Desnoyers]

* Woegerer, Paul (Paul_Woegerer at mentor.com) wrote:
> There are use-cases where loosing an event is not acceptable.
>
> Is there some way to make a userspace application block if the buffer is  
> full (instead of discarding the event or overwriting an old event) ?
>
> I'm thinking about something like:
>
> lttng enable-channel myblockingchannel --block
>
> I know I can increase the subbuf-size but sometimes this is not an  
> option (embedded targets with less RAM).
>
> A new channel option like --block would be a fine complement for the  
> already existing options --discard and --overwrite.
>
> What do you think ?

I already thought about permitting this, but we currently don't. The
first thing I must say about this is that I prefer to wait a bit before
we add this feature, and think about its impact thoroughly, because
allowing the tracer to block applications gives a lot of power to the
tracer: e.g., if tracing is stopped due to error conditions, or disk
full, or network traffic slowdown, how do we handle the fact that this
might block progress in all traced applications ?

The current modes (discard and overwrite) let the applications continue
even if there is too much data being recorded into the trace buffers --
this is a "safe" approach.

How would you recommend dealing with the possible pitfalls of blocking
traced applications ? We would need a mechanism in place to ensure
gathering a trace cannot make applications unresponsive.

Thanks,

Mathieu


> Thanks,
> Paul
>
> -- 
> Paul Woegerer | SW Development Engineer
> Mentor Embedded(tm) | Prinz Eugen Stra?e 72/2/4, Vienna, 1040 Austria
> P 43.1.535991320
> Nucleus? | Linux? | Android(tm) | Services | UI | Multi-OS
>
> Android is a trademark of Google Inc. Use of this trademark is subject to Google Permissions.
> Linux is the registered trademark of Linus Torvalds in the U.S. and other countries.
>
>
> _______________________________________________
> lttng-dev mailing list
> lttng-dev at lists.lttng.org
> http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

-- 
Mathieu Desnoyers
Operating System Efficiency R&D Consultant
EfficiOS Inc.
http://www.efficios.com

[lttng-dev] lttng enable-channel option for blocking

[Woegerer, Paul]

On 04/26/2012 11:16 PM, Mathieu Desnoyers wrote:
> I already thought about permitting this, but we currently don't. The 
> first thing I must say about this is that I prefer to wait a bit 
> before we add this feature, and think about its impact thoroughly, 
> because allowing the tracer to block applications gives a lot of power 
> to the tracer: e.g., if tracing is stopped due to error conditions, or 
> disk full, or network traffic slowdown, how do we handle the fact that 
> this might block progress in all traced applications ?
Good to know that this is on the agenda.

I agree, it gives a lot of power to the tracer. By messing with channel 
configurations a user could make a tracing application unusable. But the 
user already has ways to make applications unusable (by messing with 
ulimit, for example). There is always enough rope to hang yourself.
> The current modes (discard and overwrite) let the applications continue
> even if there is too much data being recorded into the trace buffers --
> this is a "safe" approach.
>
> How would you recommend dealing with the possible pitfalls of blocking
> traced applications ? We would need a mechanism in place to ensure
> gathering a trace cannot make applications unresponsive.
I guess in case of "lttng enable-channel --block" a user would have to 
expect that a call to tracepoint() might block. He has to deal with the 
fact in his application domain (e.g. implement a watchdog mechanism). 
Nonetheless it might be a good idea to also provide something like 
tracepointnb() ( a non-blocking variant of tracepoint() ) in case the 
user doesn't want to deal with (or simply cannot accept) potential blocking.

--
Thanks,
Paul

-- 
Paul Woegerer | SW Development Engineer
Mentor Embedded(tm) | Prinz Eugen Stra?e 72/2/4, Vienna, 1040 Austria
P 43.1.535991320
Nucleus? | Linux? | Android(tm) | Services | UI | Multi-OS

Android is a trademark of Google Inc. Use of this trademark is subject to Google Permissions.
Linux is the registered trademark of Linus Torvalds in the U.S. and other countries.

[lttng-dev] lttng enable-channel option for blocking

[Mathieu Desnoyers]

* Woegerer, Paul (Paul_Woegerer at mentor.com) wrote:
> On 04/26/2012 11:16 PM, Mathieu Desnoyers wrote:
>> I already thought about permitting this, but we currently don't. The  
>> first thing I must say about this is that I prefer to wait a bit  
>> before we add this feature, and think about its impact thoroughly,  
>> because allowing the tracer to block applications gives a lot of power  
>> to the tracer: e.g., if tracing is stopped due to error conditions, or  
>> disk full, or network traffic slowdown, how do we handle the fact that  
>> this might block progress in all traced applications ?
> Good to know that this is on the agenda.
>
> I agree, it gives a lot of power to the tracer. By messing with channel  
> configurations a user could make a tracing application unusable. But the  
> user already has ways to make applications unusable (by messing with  
> ulimit, for example). There is always enough rope to hang yourself.

A core difference between ulimit and user-space tracing is that ulimit
can only be set within the environment (and access right) of the user
running the application. System-wide tracing sessions can be initiated
by users member of the "tracing" group -- giving them the ability to
potentially DoS an application does not appear to me to be a good
security practice.

Thoughts ?

Thanks,

Mathieu

>> The current modes (discard and overwrite) let the applications continue
>> even if there is too much data being recorded into the trace buffers --
>> this is a "safe" approach.
>>
>> How would you recommend dealing with the possible pitfalls of blocking
>> traced applications ? We would need a mechanism in place to ensure
>> gathering a trace cannot make applications unresponsive.
> I guess in case of "lttng enable-channel --block" a user would have to  
> expect that a call to tracepoint() might block. He has to deal with the  
> fact in his application domain (e.g. implement a watchdog mechanism).  
> Nonetheless it might be a good idea to also provide something like  
> tracepointnb() ( a non-blocking variant of tracepoint() ) in case the  
> user doesn't want to deal with (or simply cannot accept) potential 
> blocking.
>
> --
> Thanks,
> Paul
>
> -- 
> Paul Woegerer | SW Development Engineer
> Mentor Embedded(tm) | Prinz Eugen Stra?e 72/2/4, Vienna, 1040 Austria
> P 43.1.535991320
> Nucleus? | Linux? | Android(tm) | Services | UI | Multi-OS
>
> Android is a trademark of Google Inc. Use of this trademark is subject to Google Permissions.
> Linux is the registered trademark of Linus Torvalds in the U.S. and other countries.
>

-- 
Mathieu Desnoyers
Operating System Efficiency R&D Consultant
EfficiOS Inc.
http://www.efficios.com

[lttng-dev] lttng enable-channel option for blocking

[Woegerer, Paul]

On 04/27/2012 01:33 PM, Mathieu Desnoyers wrote:
> A core difference between ulimit and user-space tracing is that ulimit 
> can only be set within the environment (and access right) of the user 
> running the application. System-wide tracing sessions can be initiated 
> by users member of the "tracing" group -- giving them the ability to 
> potentially DoS an application does not appear to me to be a good 
> security practice. Thoughts ?

Hmm, how would that look in practice ? Lets assume there is the web 
server which was started by an init-script in runlevel 3. How does a 
user that belongs to group tracing hava a chance to DoS the already 
running running web server. As far as I understand the trace session 
concept every tracing user can only see (and affect) the tracing session 
that he initiated. Even if the web server itself runs in a tracing 
session (of user wwwrun) other tracing users wouldn't see it when they 
do a "lttng list", right ?

--
Paul

-- 
Paul Woegerer | SW Development Engineer
Mentor Embedded(tm) | Prinz Eugen Stra?e 72/2/4, Vienna, 1040 Austria
P 43.1.535991320
Nucleus? | Linux? | Android(tm) | Services | UI | Multi-OS

Android is a trademark of Google Inc. Use of this trademark is subject to Google Permissions.
Linux is the registered trademark of Linus Torvalds in the U.S. and other countries.

[lttng-dev] lttng enable-channel option for blocking

[Mathieu Desnoyers]

* Woegerer, Paul (Paul_Woegerer at mentor.com) wrote:
> On 04/27/2012 01:33 PM, Mathieu Desnoyers wrote:
>> A core difference between ulimit and user-space tracing is that ulimit  
>> can only be set within the environment (and access right) of the user  
>> running the application. System-wide tracing sessions can be initiated  
>> by users member of the "tracing" group -- giving them the ability to  
>> potentially DoS an application does not appear to me to be a good  
>> security practice. Thoughts ?
>
> Hmm, how would that look in practice ? Lets assume there is the web  
> server which was started by an init-script in runlevel 3. How does a  
> user that belongs to group tracing hava a chance to DoS the already  
> running running web server. As far as I understand the trace session  
> concept every tracing user can only see (and affect) the tracing session  
> that he initiated. Even if the web server itself runs in a tracing  
> session (of user wwwrun) other tracing users wouldn't see it when they  
> do a "lttng list", right ?

Let me clarify the concept of tracing session in lttng 2.0.

We support launching per-user sessiond, which only interact with the
user's applications. That's all fine with security.

Now, we also support a root system-wide sessiond, which allows kernel
and user-space tracing. The "tracing" group has every right to create a
tracing session and trace the kernel and _all_ applications that were
already or will be running on the system.

So given a tracing session can impact _all_ applications running on the
system, we have to be really careful there.

Thanks,

Mathieu

>
> --
> Paul
>
> -- 
> Paul Woegerer | SW Development Engineer
> Mentor Embedded(tm) | Prinz Eugen Stra?e 72/2/4, Vienna, 1040 Austria
> P 43.1.535991320
> Nucleus? | Linux? | Android(tm) | Services | UI | Multi-OS
>
> Android is a trademark of Google Inc. Use of this trademark is subject to Google Permissions.
> Linux is the registered trademark of Linus Torvalds in the U.S. and other countries.
>

-- 
Mathieu Desnoyers
Operating System Efficiency R&D Consultant
EfficiOS Inc.
http://www.efficios.com

[lttng-dev] lttng enable-channel option for blocking

[Woegerer, Paul]

On 04/27/2012 02:43 PM, Mathieu Desnoyers wrote:
> * Woegerer, Paul (Paul_Woegerer at mentor.com) wrote:
>> On 04/27/2012 01:33 PM, Mathieu Desnoyers wrote:
>>> A core difference between ulimit and user-space tracing is that ulimit
>>> can only be set within the environment (and access right) of the user
>>> running the application. System-wide tracing sessions can be initiated
>>> by users member of the "tracing" group -- giving them the ability to
>>> potentially DoS an application does not appear to me to be a good
>>> security practice. Thoughts ?
>> Hmm, how would that look in practice ? Lets assume there is the web
>> server which was started by an init-script in runlevel 3. How does a
>> user that belongs to group tracing hava a chance to DoS the already
>> running running web server. As far as I understand the trace session
>> concept every tracing user can only see (and affect) the tracing session
>> that he initiated. Even if the web server itself runs in a tracing
>> session (of user wwwrun) other tracing users wouldn't see it when they
>> do a "lttng list", right ?
> Let me clarify the concept of tracing session in lttng 2.0.
>
> We support launching per-user sessiond, which only interact with the
> user's applications. That's all fine with security.
>
> Now, we also support a root system-wide sessiond, which allows kernel
> and user-space tracing. The "tracing" group has every right to create a
> tracing session and trace the kernel and _all_ applications that were
> already or will be running on the system.

Ah, I see. I was not aware of the "... and _all_ applications that were 
already or will be running on the system" aspect of the concept. In that 
case I would rather invert the semantic in the API and instead of having 
a tracepoint() function that potentially blocks I would declare 
tracepoint() to never ever block and additionally provide tracepointb() 
that does potentially block.

--
Paul

-- 
Paul Woegerer | SW Development Engineer
Mentor Embedded(tm) | Prinz Eugen Stra?e 72/2/4, Vienna, 1040 Austria
P 43.1.535991320
Nucleus? | Linux? | Android(tm) | Services | UI | Multi-OS

Android is a trademark of Google Inc. Use of this trademark is subject to Google Permissions.
Linux is the registered trademark of Linus Torvalds in the U.S. and other countries.

[lttng-dev] lttng enable-channel option for blocking

[Mathieu Desnoyers]

* Woegerer, Paul (Paul_Woegerer at mentor.com) wrote:
> On 04/27/2012 02:43 PM, Mathieu Desnoyers wrote:
>> * Woegerer, Paul (Paul_Woegerer at mentor.com) wrote:
>>> On 04/27/2012 01:33 PM, Mathieu Desnoyers wrote:
>>>> A core difference between ulimit and user-space tracing is that ulimit
>>>> can only be set within the environment (and access right) of the user
>>>> running the application. System-wide tracing sessions can be initiated
>>>> by users member of the "tracing" group -- giving them the ability to
>>>> potentially DoS an application does not appear to me to be a good
>>>> security practice. Thoughts ?
>>> Hmm, how would that look in practice ? Lets assume there is the web
>>> server which was started by an init-script in runlevel 3. How does a
>>> user that belongs to group tracing hava a chance to DoS the already
>>> running running web server. As far as I understand the trace session
>>> concept every tracing user can only see (and affect) the tracing session
>>> that he initiated. Even if the web server itself runs in a tracing
>>> session (of user wwwrun) other tracing users wouldn't see it when they
>>> do a "lttng list", right ?
>> Let me clarify the concept of tracing session in lttng 2.0.
>>
>> We support launching per-user sessiond, which only interact with the
>> user's applications. That's all fine with security.
>>
>> Now, we also support a root system-wide sessiond, which allows kernel
>> and user-space tracing. The "tracing" group has every right to create a
>> tracing session and trace the kernel and _all_ applications that were
>> already or will be running on the system.
>
> Ah, I see. I was not aware of the "... and _all_ applications that were  
> already or will be running on the system" aspect of the concept. In that  
> case I would rather invert the semantic in the API and instead of having  
> a tracepoint() function that potentially blocks I would declare  
> tracepoint() to never ever block and additionally provide tracepointb()  
> that does potentially block.

In LTTng, we try very hard to split tracing into separate components and
to keep the number of concepts that cross components to the fewest
number possible, to make both development and end-user understanding
easier.

In this case, the question is: where should the behavior of buffer-full
scenarios should be configured ? Clearly, this is related to the tracer
ring buffer, which is usually configured through commands sent to the
lttng session daemon, and which are kept on a per-session/per-channel,
and possibly per-event basis.

Currently, the tracepoints API does not expose the tracer internals: it
only adds static instrumentation points into applications and libraries.

So keeping this in mind, one way to allow this kind of "blocking" mode
might be to add a special option to the lttng-sessiond, e.g. a
"--developer" mode, which could enable this feature, and possibly other
features that are not "safe" to deploy on production machines.

Thoughts ?

Thanks,

Mathieu

-- 
Mathieu Desnoyers
Operating System Efficiency R&D Consultant
EfficiOS Inc.
http://www.efficios.com