[ltt-dev] [RFC UST and LTTNG] Daemon model proposal v0.3

[ltt-dev] [RFC UST and LTTNG] Daemon model proposal v0.3

[Mathieu Desnoyers]

* David Goulet (david.goulet at polymtl.ca) wrote:
> Everything about a per-user ltt-sessiond will be added to the RFC!
>
> Comments below.
>
> On 11-01-31 12:10 PM, Mathieu Desnoyers wrote:
>> * David Goulet (david.goulet at polymtl.ca) wrote:
>>> This unique identifier SHOULD be a unique string consisting of the username, a
>>> session name and the number of already created session plus one. The last
>>> number makes sure that the ID is unique in the ltt-sessiond context. In the
>>> implementation, a simple number could be use for performance purpose instead of
>>> comparing string at each iteration.
>>>
>>> Ex: Username - "dave", Session name - "mysession", Num. sessions - fifth one
>>>      -->  ID: dave.mysession-5
>>
>> We will run into problems across reboots: session IDs might be reused, thus
>> trying to overwrite older traces. Other ideas, possibly including UUID, would be
>> welcome.
>
> Across reboot... I don't see the problem reusing IDs... The only problem  
> to overwrite older traces is if the session ID is used in the trace path  
> on the disk which is not the case.

Then what do you propose using as directory name to hold the trace on disk ?
Using the session name would seem natural.

>
>>
>> My recommandation would be to allow two schemes:
>>
>> 1 - allow the user to specify the trace session name.
>>      (fails if the trace session name is already taken, or if the target trace is
>>      already there)
>
> Absolutely, session name is user defined unless not specified.
>
>> 2 - automatically assign trace session names with:
>>      username.UUID
>>
>> Thoughts ?
>
> I used the username in order to identified the session on a per user  
> basis. Then, as a unique string for the ID, session name and number of  
> sessions seems to me that no collision will happen... unless I'm missing  
> something.. ?

See remark above for the directory name on disk.

>
>>
>>> This daemon is also responsible for tracing buffer creation. Two main reasons
>>> motivate this design:
>>>
>>>      * The shared memory segment needs to be in the tracing group. We can only
>>>      assume that ltt-sessiond, being root (UID = 0), is always able to do that.
>>
>> As we discussed, I think this argument does not hold: any user can set the file
>> GID to "tracing", right ?
>
> Yeah you are right... Forgot that one. The shm credentials can be set  
> arbitrarily by any user. I will remove this reason.
>
>>
>>>
>>>      * The ltt-sessiond needs to keep track of all the shared memory segments in
>>>      order to be able to give reference to any other consumer since it is this
>>>      daemon that controls the access to all tracing buffers.
>>
>> I'm not convinced that ltt-sessiond needs to _create_ these buffers with this
>> argument, only that it needs some way to know what they are and to be able to
>> pass a reference to them to the consumers.
>
> Well, I think it's an important reason since this is the only way we  
> have to assign already created buffers to another consumer...

Maybe the most straightforward way, but not "the only" way: we could think of a
scheme where the inprocess library creates the buffers, and passes the file
descriptors to the sessiond through sockets. But it's an extra step that would
be good to avoid -- and my argument below supports the
session-registry-allocation of buffers.

>
>>
>> However, the following point might point us towards the requirement for
>> allocating the shared-mem buffers (shm) from the registry: if we ever want to
>> share the tracing buffers between all userspace applications (on a system where
>> security is less of a concern, but where scalability and performance is utterly
>> important), allocating them from the session registry will allow that very
>> easily, but if we allocate them from within the inprocess lib, then we will need
>> to redesign the whole thing.
>
> I'll add this.
>
>>
>>>
>>> A tracing session is associated to a user of the system (UID). Unless a user is
>>> in the tracing group, no other users can access the tracing data for that
>>> session.
>>
>> This is unclear, and does not seem to match our discussion. Access in terms of
>> "write" and "read" accesses to the buffers and control pipes are unclealy
>> specified. I would rather say:
>>
>
> Agree, I will clarify all of this in a Security section to make  
> everything clearer on that aspect.
>
>>> in order to trace the kernel, you are either root (UID=0) or in the tracing
>>> group.
>>
>> Tracing applications should also be allowed from root user (not just tracing
>> group).
>
> Yep but that section is about "ltt-consumerd" only so this is why there  
> is no mention about tracing applications :)

I guess I lost track of the context at this point then. Recalling that this
apply specifically to ltt-consumerd might be good.

>
>>
>> It should also check for its own user's ltt-sessiond named pipe (possibly in
>> /tmp/ltt-sessiond-username or /HOME/.ltt-sessiond ?).
>>
>> Cleanup of the named pipe when the sessiond dies is a very important aspect to
>> test, because we have to keep in mind that if the pipe is left there when there
>> is no sessiond active, the applications might be stucked. I would personally try
>> not to rely on the presence of the named pipe alone to indicate that sessiond is
>> there: if the named pipe exists and has the correct user, we should try to talk
>> to the sessiond (and fail if it does not respond), rather that waiting
>> endlessly. I'm afraid this is the only solid way to deal with improper cleanup
>> of ltt-sessiond (which is something might always happen, and which should not
>> hang all the traced applications).
>>
>> Same thing when we start a ltt-sessiond: if the named pipe already exists, we
>> should check if it is associated with an actual sessiond that responds to
>> commands.
>
> Yes. Very good point. I'll clarify.
>
>>> ltt-sessiond registry daemon for every trace action needed by the user.
>>
>> Actually, we might want to allow various applications to do tracing, drtrace
>> being the main one. If we modularize the pieces of drtrace into various
>> libraries, it will be easier to re-use for others.
>
> Yep.
>
>>> Scenario 2 - Single user tracing already running app_1
>>>
>>> 1) drtrace ask ltt-sessiond for a new session through a Unix socket. If
>>> allowed, ltt-sessiond returns a session ID to the client.
>>>
>>> +-----------+    ops     +--------------+
>>> | drtrace A |<---------->| ltt-sessiond |
>>> +-----------+            +--------------+
>>>                             ^
>>>          +-------+   read   |
>>>          | app_1 |----------+
>>>          +-------+
>>>
>>> NOTE: At this stage, since app_1 is already running, the registration of app_1
>>> to ltt-sessiond has already been done. However, the shared memory segment is
>>> not allocated yet until a trace session is initiated. Having no shared memory,
>>> the inprocess library of app_1 will wait on a name pipe connected to
>>> ltt-sessiond for the reference.
>>
>> I would argue that even if the app has a set of buffers being used by an active
>> tracing session, it should still wait on its named pipe for ltt-sessiond
>> connexions for two reasons:
>>
>> - Connexion may come from the per-user sessiond (local ltt-sessiond)
>> - We might create another concurrent tracing session, which can ask for yet
>>    another set of tracing buffers to be created. It seems more convenient to use
>>    the same communication scheme independently of if there is 0 or 1 active
>>    tracing session.
>>
>
> If I understand you correctly, you are saying that the app_1 should  
> ALWAYS wait on the ltt-sessiond named pipe? If yes, it's the case. I  
> will add some clarification here.

Yes, I would say that the application should always be waiting for new tracing
sessions to come up (from any of the global or per-user ltt-sessiond), and this
should be the exact same mechanism whether there is already a set of buffers
actively tracing or not (no behavior difference between 0 -> 1 session and N ->
N+1 sessions). It will make testing much easier.

> That was a hell of a reply! Cheers! :)

You're welcome!

Mathieu

> David
>
> -- 
> David Goulet
> LTTng project, DORSAL Lab.
>
> PGP/GPG : 1024D/16BD8563
> BE3C 672B 9331 9796 291A  14C6 4AF7 C14B 16BD 8563

-- 
Mathieu Desnoyers
Operating System Efficiency R&D Consultant
EfficiOS Inc.
http://www.efficios.com

[ltt-dev] [RFC UST and LTTNG] Daemon model proposal v0.3

[David Goulet]

On 11-01-31 02:06 PM, Mathieu Desnoyers wrote:
> * David Goulet (david.goulet at polymtl.ca) wrote:
>> Everything about a per-user ltt-sessiond will be added to the RFC!
>>
>> Comments below.
>>
>> On 11-01-31 12:10 PM, Mathieu Desnoyers wrote:
>>> * David Goulet (david.goulet at polymtl.ca) wrote:
>>>> This unique identifier SHOULD be a unique string consisting of the username, a
>>>> session name and the number of already created session plus one. The last
>>>> number makes sure that the ID is unique in the ltt-sessiond context. In the
>>>> implementation, a simple number could be use for performance purpose instead of
>>>> comparing string at each iteration.
>>>>
>>>> Ex: Username - "dave", Session name - "mysession", Num. sessions - fifth one
>>>>       -->   ID: dave.mysession-5
>>>
>>> We will run into problems across reboots: session IDs might be reused, thus
>>> trying to overwrite older traces. Other ideas, possibly including UUID, would be
>>> welcome.
>>
>> Across reboot... I don't see the problem reusing IDs... The only problem
>> to overwrite older traces is if the session ID is used in the trace path
>> on the disk which is not the case.
>
> Then what do you propose using as directory name to hold the trace on disk ?
> Using the session name would seem natural.

Yep, absolutely for the first level directory but the trace file path 
should be quite different. We talked about something like this the other 
day :

$sessionID/
	$appname-$date-$time-$PID

having $sessionID the above proposition : dave.session_name-5

The $time is a bit useless but makes the trace path "unique" since the 
case of having the same date, appname and PID can occur.

I'm actually quite open for better suggestion :)

I'll make sure the rest of that email is taking into account for the 
next version!

Thanks
David

-- 
David Goulet
LTTng project, DORSAL Lab.

PGP/GPG : 1024D/16BD8563
BE3C 672B 9331 9796 291A  14C6 4AF7 C14B 16BD 8563

[ltt-dev] [RFC UST and LTTNG] Daemon model proposal v0.3

[Mathieu Desnoyers]

* David Goulet (david.goulet at polymtl.ca) wrote:
>
>
> On 11-01-31 02:06 PM, Mathieu Desnoyers wrote:
>> * David Goulet (david.goulet at polymtl.ca) wrote:
>>> Everything about a per-user ltt-sessiond will be added to the RFC!
>>>
>>> Comments below.
>>>
>>> On 11-01-31 12:10 PM, Mathieu Desnoyers wrote:
>>>> * David Goulet (david.goulet at polymtl.ca) wrote:
>>>>> This unique identifier SHOULD be a unique string consisting of the username, a
>>>>> session name and the number of already created session plus one. The last
>>>>> number makes sure that the ID is unique in the ltt-sessiond context. In the
>>>>> implementation, a simple number could be use for performance purpose instead of
>>>>> comparing string at each iteration.
>>>>>
>>>>> Ex: Username - "dave", Session name - "mysession", Num. sessions - fifth one
>>>>>       -->   ID: dave.mysession-5
>>>>
>>>> We will run into problems across reboots: session IDs might be reused, thus
>>>> trying to overwrite older traces. Other ideas, possibly including UUID, would be
>>>> welcome.
>>>
>>> Across reboot... I don't see the problem reusing IDs... The only problem
>>> to overwrite older traces is if the session ID is used in the trace path
>>> on the disk which is not the case.
>>
>> Then what do you propose using as directory name to hold the trace on disk ?
>> Using the session name would seem natural.
>
> Yep, absolutely for the first level directory but the trace file path  
> should be quite different. We talked about something like this the other  
> day :
>
> $sessionID/
> 	$appname-$date-$time-$PID
>
> having $sessionID the above proposition : dave.session_name-5

Well, that's where I see a problem:

if we gather a trace on a system from user "dave", with the same session name,
the incremented "unique" number is only unique as long as we don't restart
ltt-sessiond. So if we reboot the system, the same tracing session name will be
reused, rendering it non-unique.

I would really like drtrace to be able to work without requiring the user to
explicitely specify a session name. This is why using a UUID (truly unique ID)
would make sense as default session identifier. When the user specify the
session name, I see no point in trying to fixup the clash with a postfix: we
could simply return an error, saying that there is already a trace in there.

>
> The $time is a bit useless but makes the trace path "unique" since the  
> case of having the same date, appname and PID can occur.

Sure, I agree on this one. Using "$appname-$date-$time-$PID" as subdirectory to
hold the per-application traces makes sense.

Thanks,

Mathieu

> I'm actually quite open for better suggestion :)
>
> I'll make sure the rest of that email is taking into account for the  
> next version!
>
> Thanks
> David
>
> -- 
> David Goulet
> LTTng project, DORSAL Lab.
>
> PGP/GPG : 1024D/16BD8563
> BE3C 672B 9331 9796 291A  14C6 4AF7 C14B 16BD 8563

-- 
Mathieu Desnoyers
Operating System Efficiency R&D Consultant
EfficiOS Inc.
http://www.efficios.com