[lttng-dev] UST app and lttng-tools compatibility

[lttng-dev] UST app and lttng-tools compatibility

[Mathieu Desnoyers]

* Francis Giraldeau (francis.giraldeau at gmail.com) wrote:
> Le 2012-09-29 14:28, Mathieu Desnoyers a ?crit :
> > * Francis Giraldeau (francis.giraldeau at gmail.com) wrote:
> >> Hi,
> >>
> >> I wanted to share my lttng-ust 2.1 update experience, maybe it will save
> >> time for others.
> >>
> >> I updated lttng-ust recently. After this change, the app would not
> >> produce a trace anymore. No error message is displayed by the traced app
> >> to indicate that something is wrong. Even when setting LTTNG_DEBUG_UST
> >> to the app's environment variable, there is no error message. The debug
> >> output suggests that probes are registered and everything is fine, while
> >> it's not.
> >>
> >> By running lttng-sessiond with -vvv --verbose-consumer, I finally got
> >> this message:
> >>
> >> DEBUG2: UST app PID 8112 is not compatible with major version 3
> >> (supporting <= 2) [in ust_app_validate_version() at ust-app.c:2633]
> >>
> >> Updating lttng-tools to 2.1 solved the issue. Seems that it's mandatory
> >> to update lttng-tools to support latest lttng-ust. It may be obvious for
> >> developers, but it should be clear for users that they must upgrade both.
> >>
> >> IMHO, It would be nice if the app side log could tell if the
> >> session/consumer refused the registration.
> > 
> > I agree we should do better.
> > 
> > Regarding lttng-tools, I think changing this DBG2 message to a WARN
> > message would help, so sessiond would show the warning, except in the
> > case where it is started with "-d".
> 
> Excellent idea.

David, can you make this change ?

> 
> > On the application side, this is a bit tricky. It has no way to find out
> > that it has been rejected by the sessiond. The application registers at
> > startup, and then the sessiond keeps the connexion active, but flags it
> > as incompatible internally. The reason we do that is because we don't
> > want the application to retry endlessly.
> 
> Could the registration process block until the sessiond returns some
> status?

The session is not "returning" anything to the application. The
application is registering to the sessiond, and all the sessiond can do
is to keep the socket alive or close it.

Even if we added a "command" that the sessiond could use to tell the
application it has a wrong version, that would not be 2.0 material, and
the application wouldn't know about it. Moreover, given we have changed
the communication protocol, not sure we would like to have this extra
command as entirely fixed for now, as it would be part of the
communication protocol (which is not the case for the initial app
registration, with is part of a lower level protocol).

> I understand that in commercial setup, the application should
> not be prevented to start and run normally if tracing is not available
> or misconfigured.

By default, we only block the application for up to 3 seconds at
registration. I think that if we start the application with
LTTNG_UST_REGISTER_TIMEOUT=-1, the application will, in this case, block
forever (see man lttng-ust).

Thanks!

Mathieu

> But for developers, some env var like
> "LTTNG_ABORT_ON_ERROR" could help to diagnose this king of problem.
> 
> > Moreover, I cannot change the code for the existing 2.0 UST libs, so
> > adding a new message is not possible.
> 
> Of course! ;)
> 
> > One thing we have in mind for 2.2 or 2.3 is to add syslog support within
> > the sessiond. This would provide a nice centralized place to look at
> > those logs.
> 
> I think it would be great. It's a bit less hidden, but certainly address
> concerns for production, and still can be used by developers. An error
> message is better than no one! ;)
> 
> Cheers,
> 
> Francis
> 
> 
> 



-- 
Mathieu Desnoyers
Operating System Efficiency R&D Consultant
EfficiOS Inc.
http://www.efficios.com

[lttng-dev] UST app and lttng-tools compatibility

[David Goulet]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Mathieu Desnoyers:
> * Francis Giraldeau (francis.giraldeau at gmail.com) wrote:
>> Le 2012-09-29 14:28, Mathieu Desnoyers a ?crit :
>>> * Francis Giraldeau (francis.giraldeau at gmail.com) wrote:
>>>> Hi,
>>>> 
>>>> I wanted to share my lttng-ust 2.1 update experience, maybe
>>>> it will save time for others.
>>>> 
>>>> I updated lttng-ust recently. After this change, the app
>>>> would not produce a trace anymore. No error message is
>>>> displayed by the traced app to indicate that something is
>>>> wrong. Even when setting LTTNG_DEBUG_UST to the app's
>>>> environment variable, there is no error message. The debug 
>>>> output suggests that probes are registered and everything is
>>>> fine, while it's not.
>>>> 
>>>> By running lttng-sessiond with -vvv --verbose-consumer, I
>>>> finally got this message:
>>>> 
>>>> DEBUG2: UST app PID 8112 is not compatible with major version
>>>> 3 (supporting <= 2) [in ust_app_validate_version() at
>>>> ust-app.c:2633]
>>>> 
>>>> Updating lttng-tools to 2.1 solved the issue. Seems that it's
>>>> mandatory to update lttng-tools to support latest lttng-ust.
>>>> It may be obvious for developers, but it should be clear for
>>>> users that they must upgrade both.
>>>> 
>>>> IMHO, It would be nice if the app side log could tell if the 
>>>> session/consumer refused the registration.
>>> 
>>> I agree we should do better.
>>> 
>>> Regarding lttng-tools, I think changing this DBG2 message to a
>>> WARN message would help, so sessiond would show the warning,
>>> except in the case where it is started with "-d".
>> 
>> Excellent idea.
> 
> David, can you make this change ?
> 

Yes!

David

>> 
>>> On the application side, this is a bit tricky. It has no way to
>>> find out that it has been rejected by the sessiond. The
>>> application registers at startup, and then the sessiond keeps
>>> the connexion active, but flags it as incompatible internally.
>>> The reason we do that is because we don't want the application
>>> to retry endlessly.
>> 
>> Could the registration process block until the sessiond returns
>> some status?
> 
> The session is not "returning" anything to the application. The 
> application is registering to the sessiond, and all the sessiond
> can do is to keep the socket alive or close it.
> 
> Even if we added a "command" that the sessiond could use to tell
> the application it has a wrong version, that would not be 2.0
> material, and the application wouldn't know about it. Moreover,
> given we have changed the communication protocol, not sure we would
> like to have this extra command as entirely fixed for now, as it
> would be part of the communication protocol (which is not the case
> for the initial app registration, with is part of a lower level
> protocol).
> 
>> I understand that in commercial setup, the application should not
>> be prevented to start and run normally if tracing is not
>> available or misconfigured.
> 
> By default, we only block the application for up to 3 seconds at 
> registration. I think that if we start the application with 
> LTTNG_UST_REGISTER_TIMEOUT=-1, the application will, in this case,
> block forever (see man lttng-ust).
> 
> Thanks!
> 
> Mathieu
> 
>> But for developers, some env var like "LTTNG_ABORT_ON_ERROR"
>> could help to diagnose this king of problem.
>> 
>>> Moreover, I cannot change the code for the existing 2.0 UST
>>> libs, so adding a new message is not possible.
>> 
>> Of course! ;)
>> 
>>> One thing we have in mind for 2.2 or 2.3 is to add syslog
>>> support within the sessiond. This would provide a nice
>>> centralized place to look at those logs.
>> 
>> I think it would be great. It's a bit less hidden, but certainly
>> address concerns for production, and still can be used by
>> developers. An error message is better than no one! ;)
>> 
>> Cheers,
>> 
>> Francis
>> 
>> 
>> 
> 
> 
> 
-----BEGIN PGP SIGNATURE-----

iQEcBAEBCgAGBQJQabGBAAoJEELoaioR9I024c0IAJ8i7hIUthhQdbeESSV6T65L
VgxPYtjUr2aKbOaeInS1FXiW39Gpg5w1Xut9gTaAmlxP+Dsgs1TXLVx0YJFCsqi0
QYrkPLbpdafmCjbXKpWa6G5hNCFa684MorL6hCqhSOQEBXD0N/aKGQmUVz+pxGXp
xR/l/hwHadm4v6VH75N5dr5JvGNHQbcZDYz0gzokemyo1edw0NGa3jQGe0WYcVcp
/K6hybbgRvkZObMB2hhTESkLwAFk/OO4WEP6zTZCSPLCTAdiGLaWNQQ9PdxxkcT+
crT83mDSW8K1yQkZDXOk5U9XSmdBLjBuQW6094EaCF9Nb5K6d+y2U01dii4eIU4=
=6aHd
-----END PGP SIGNATURE-----

[lttng-dev] UST app and lttng-tools compatibility

[Mathieu Desnoyers]

* Mathieu Desnoyers (mathieu.desnoyers at efficios.com) wrote:
> * Francis Giraldeau (francis.giraldeau at gmail.com) wrote:
[...]
> > Could the registration process block until the sessiond returns some
> > status?
> 
> The session is not "returning" anything to the application. The
> application is registering to the sessiond, and all the sessiond can do
> is to keep the socket alive or close it.
> 
> Even if we added a "command" that the sessiond could use to tell the
> application it has a wrong version, that would not be 2.0 material, and
> the application wouldn't know about it. Moreover, given we have changed
> the communication protocol, not sure we would like to have this extra
> command as entirely fixed for now, as it would be part of the
> communication protocol (which is not the case for the initial app
> registration, with is part of a lower level protocol).
> 
> > I understand that in commercial setup, the application should
> > not be prevented to start and run normally if tracing is not available
> > or misconfigured.
> 
> By default, we only block the application for up to 3 seconds at
> registration. I think that if we start the application with
> LTTNG_UST_REGISTER_TIMEOUT=-1, the application will, in this case, block
> forever (see man lttng-ust).

Let me take this last part back. That was a pre-morning-coffee
statement. ;) The application will receive a "registration done" message
when the version is found to be incompatible. Therefore, it will never
wait for the incompatible sessiond.

One thing we could do to make transition smoother between future
versions would be to add a new command to let the sessiond send its own
version info to the application. In this command's handler, the
application could show a warning on stderr. This could not apply to 2.0,
but we could do it starting from 2.1.

Thoughts ?

Thanks,

Mathieu


-- 
Mathieu Desnoyers
Operating System Efficiency R&D Consultant
EfficiOS Inc.
http://www.efficios.com

[lttng-dev] UST app and lttng-tools compatibility

[David Goulet]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512



Mathieu Desnoyers:
> * Mathieu Desnoyers (mathieu.desnoyers at efficios.com) wrote:
>> * Francis Giraldeau (francis.giraldeau at gmail.com) wrote:
> [...]
>>> Could the registration process block until the sessiond returns
>>> some status?
>> 
>> The session is not "returning" anything to the application. The 
>> application is registering to the sessiond, and all the sessiond
>> can do is to keep the socket alive or close it.
>> 
>> Even if we added a "command" that the sessiond could use to tell
>> the application it has a wrong version, that would not be 2.0
>> material, and the application wouldn't know about it. Moreover,
>> given we have changed the communication protocol, not sure we
>> would like to have this extra command as entirely fixed for now,
>> as it would be part of the communication protocol (which is not
>> the case for the initial app registration, with is part of a
>> lower level protocol).
>> 
>>> I understand that in commercial setup, the application should 
>>> not be prevented to start and run normally if tracing is not
>>> available or misconfigured.
>> 
>> By default, we only block the application for up to 3 seconds at 
>> registration. I think that if we start the application with 
>> LTTNG_UST_REGISTER_TIMEOUT=-1, the application will, in this
>> case, block forever (see man lttng-ust).
> 
> Let me take this last part back. That was a pre-morning-coffee 
> statement. ;) The application will receive a "registration done"
> message when the version is found to be incompatible. Therefore, it
> will never wait for the incompatible sessiond.
> 
> One thing we could do to make transition smoother between future 
> versions would be to add a new command to let the sessiond send its
> own version info to the application. In this command's handler,
> the application could show a warning on stderr. This could not
> apply to 2.0, but we could do it starting from 2.1.
> 
> Thoughts ?

Why sending version infos instead of a "incompatible" command ?

David

> 
> Thanks,
> 
> Mathieu
> 
> 
-----BEGIN PGP SIGNATURE-----

iQEcBAEBCgAGBQJQabZVAAoJEELoaioR9I02RYcIALJ7ySGiU6nKDD4F4ffAEIFr
ZSQC/1mhr8x66TfTdB9eGaAxA6qmESEBpg6bjyWSmWCn1dI1i5lOeXhxszM1Z8Oz
q6DKSSlRVRZFrCaZBf3Xy7QNbxTdUkwrcCjd8Y9Ffu1BswXIzzUhsYSHbJRQAHVF
QAuNFy5AeHRlE55H94Gs/ydfiQb3jqVIalVnG5LmeDPyO58sdA+RAWkwaJypgL8e
5m6r3K0IK5ufSIjWWb6G8vmf3HFV3zzdEwGpkEyaVRZjUfDBekRnJ27Bb2XqJUxC
/xe2S519VLnsCWE+j1pfN5BcchI0TKUOeTcaLzzaf/VdkHdlOmZlifSZ6ubRusU=
=OVEY
-----END PGP SIGNATURE-----

[lttng-dev] UST app and lttng-tools compatibility

[Mathieu Desnoyers]

* David Goulet (dgoulet at efficios.com) wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA512
> 
> 
> 
> Mathieu Desnoyers:
> > * Mathieu Desnoyers (mathieu.desnoyers at efficios.com) wrote:
> >> * Francis Giraldeau (francis.giraldeau at gmail.com) wrote:
> > [...]
> >>> Could the registration process block until the sessiond returns
> >>> some status?
> >> 
> >> The session is not "returning" anything to the application. The 
> >> application is registering to the sessiond, and all the sessiond
> >> can do is to keep the socket alive or close it.
> >> 
> >> Even if we added a "command" that the sessiond could use to tell
> >> the application it has a wrong version, that would not be 2.0
> >> material, and the application wouldn't know about it. Moreover,
> >> given we have changed the communication protocol, not sure we
> >> would like to have this extra command as entirely fixed for now,
> >> as it would be part of the communication protocol (which is not
> >> the case for the initial app registration, with is part of a
> >> lower level protocol).
> >> 
> >>> I understand that in commercial setup, the application should 
> >>> not be prevented to start and run normally if tracing is not
> >>> available or misconfigured.
> >> 
> >> By default, we only block the application for up to 3 seconds at 
> >> registration. I think that if we start the application with 
> >> LTTNG_UST_REGISTER_TIMEOUT=-1, the application will, in this
> >> case, block forever (see man lttng-ust).
> > 
> > Let me take this last part back. That was a pre-morning-coffee 
> > statement. ;) The application will receive a "registration done"
> > message when the version is found to be incompatible. Therefore, it
> > will never wait for the incompatible sessiond.
> > 
> > One thing we could do to make transition smoother between future 
> > versions would be to add a new command to let the sessiond send its
> > own version info to the application. In this command's handler,
> > the application could show a warning on stderr. This could not
> > apply to 2.0, but we could do it starting from 2.1.
> > 
> > Thoughts ?
> 
> Why sending version infos instead of a "incompatible" command ?

It could very well be an "incompatible" command, but it would need to
contain the version info, so the application can print the versions.

The main difference between sending version info (all the time) and the
"incompatible" command is that the version info would be sent all the
time. It could provide better debugging logs in the app when using
LTTNG_UST_DEBUG=1, but would require one extra message at app
registration. Not sure which is best.

Thoughts ?

Thanks,

Mathieu

> 
> David
> 
> > 
> > Thanks,
> > 
> > Mathieu
> > 
> > 
> -----BEGIN PGP SIGNATURE-----
> 
> iQEcBAEBCgAGBQJQabZVAAoJEELoaioR9I02RYcIALJ7ySGiU6nKDD4F4ffAEIFr
> ZSQC/1mhr8x66TfTdB9eGaAxA6qmESEBpg6bjyWSmWCn1dI1i5lOeXhxszM1Z8Oz
> q6DKSSlRVRZFrCaZBf3Xy7QNbxTdUkwrcCjd8Y9Ffu1BswXIzzUhsYSHbJRQAHVF
> QAuNFy5AeHRlE55H94Gs/ydfiQb3jqVIalVnG5LmeDPyO58sdA+RAWkwaJypgL8e
> 5m6r3K0IK5ufSIjWWb6G8vmf3HFV3zzdEwGpkEyaVRZjUfDBekRnJ27Bb2XqJUxC
> /xe2S519VLnsCWE+j1pfN5BcchI0TKUOeTcaLzzaf/VdkHdlOmZlifSZ6ubRusU=
> =OVEY
> -----END PGP SIGNATURE-----

-- 
Mathieu Desnoyers
Operating System Efficiency R&D Consultant
EfficiOS Inc.
http://www.efficios.com