[lttng-dev] LTTng 2.9 sessiond misbehaving

[lttng-dev] LTTng 2.9 sessiond misbehaving

[Thibault, Daniel]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain, Size: 2922 bytes --]

System profile:

Ubuntu 12.04.5 LTS precise, kernel 3.9.3
lttng-modules-2.9.0-pre-17-152fe7f
userspace-rcu-0.10-pre-2-d8a4979
lttng-ust-2.9.0-pre-84-4c62d8d
lttng-tools-2.9.0-pre-258-36bc42d

Running a user-local lttng-sessiond from the command line (after shutting down the root service):

$ lttng-sessiond -vvv 2>&1 | tee lttng-sessiond.1b.log

So far so good.  I test the daemon with 'lttng list' from another shell.  But when I hit ctrl-c in the daemon's shell, it doesn't respond and the CPU usage goes crazy (~90%).  Can't sudo kill it either.

dmesg is utterly silent.

What is happening?

The log is available; here are the last few lines:

$ lttng-sessiond -vvv 2>&1 | tee lttng-sessiond.2.log
...
DEBUG3 - 09:53:51.873996 [3186/3189]: [ht-thread] Polling. (in thread_ht_cleanup() at ht-cleanup.c:124)
DEBUG3 - 09:53:51.874189 [3186/3186]: Session daemon client socket 14 and application socket 15 created (in init_daemon_socket() at main.c:5182)
DEBUG3 - 09:53:51.874203 [3186/3186]: Created hashtable size 4 at 0x1635110 of type 2 (in lttng_ht_new() at hashtable.c:145)
DEBUG3 - 09:53:51.874205 [3186/3186]: Global buffer per UID registry initialized (in buffer_reg_init_uid_registry() at buffer-registry.c:99)
...
DEBUG1 - 09:54:10.381606 [3186/3191]: Wait for client response (in thread_manage_clients() at main.c:4455)
DEBUG1 - 09:54:10.382236 [3186/3191]: Receiving data from client ... (in thread_manage_clients() at main.c:4500)
DEBUG1 - 09:54:10.383618 [3186/3191]: Processing client command 13 (in process_client_msg() at main.c:2984)
DEBUG1 - 09:54:10.383812 [3186/3191]: Counting number of available session for UID 1000 GID 1000 (in lttng_sessions_count() at main.c:2953)
DEBUG1 - 09:54:10.383923 [3186/3191]: Getting all available session for UID 1000 GID 1000 (in cmd_list_lttng_sessions() at cmd.c:3098)
DEBUG1 - 09:54:10.384067 [3186/3191]: Sending response (size: 20, retcode: Success (10)) (in thread_manage_clients() at main.c:4550)
DEBUG1 - 09:54:10.384221 [3186/3191]: Clean command context structure (in clean_command_ctx() at main.c:791)
DEBUG1 - 09:54:10.384330 [3186/3191]: Accepting client command ... (in thread_manage_clients() at main.c:4403)
^C

Daniel U. Thibault, M.Sc.²

Informaticien scientifique, CME-PSC, Centre de recherches de Valcartier
Recherche et développement pour la défense Canada / Gouvernement du Canada
daniel.thibault at drdc-rddc.gc.ca / Tél: 418-844-4000x4245 Téléc: 418-844-4538
daniel.thibault at forces.gc.ca / Tél: 418-844-4000x4245 Téléc: 418-844-4538

Computer Scientist, MCCS-SPC, Valcartier Research Centre 
Defence Research and Development Canada / Government of Canada
daniel.thibault at drdc-rddc.gc.ca / Tel: 418-844-4000x4245 Fax: 418-844-4538
daniel.thibault at forces.gc.ca / Tel: 418-844-4000x4245 Fax: 418-844-4538

DRDC is an agency of the Department of National Defence / RDDC est une agence du ministère de la Défense nationale

[lttng-dev] LTTng 2.9 sessiond misbehaving

[Thibault, Daniel]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain, Size: 1390 bytes --]

Interesting.  Rebuilt the entire LTTng set with the stable-1.8 releases (stable-0.9 for userspace-rcu) and I see the same behaviour:
When I hit ctrl-c in the daemon's shell, it doesn't respond and the CPU usage goes crazy (~90%).  Can't sudo kill it (or lttng-runas) either.

What's strange is that if instead of:

$ lttng-sessiond -vvv 2>&1 | tee lttng-sessiond.2.log

I do:

$ lttng-sessiond 2>&1 | tee lttng-sessiond.3.log

Then ctrl-c terminates lttng-sessiond normally.

Using just the first level of verbosity:

$ lttng-sessiond -v 2>&1 | tee lttng-sessiond.4.log

also misbehaves.  Somehow -v is responsible?

At this point, should I open a bug?

Daniel U. Thibault, M.Sc.²

Informaticien scientifique, CME-PSC, Centre de recherches de Valcartier
Recherche et développement pour la défense Canada / Gouvernement du Canada
daniel.thibault at drdc-rddc.gc.ca / Tél: 418-844-4000x4245 Téléc: 418-844-4538
daniel.thibault at forces.gc.ca / Tél: 418-844-4000x4245 Téléc: 418-844-4538

Computer Scientist, MCCS-SPC, Valcartier Research Centre 
Defence Research and Development Canada / Government of Canada
daniel.thibault at drdc-rddc.gc.ca / Tel: 418-844-4000x4245 Fax: 418-844-4538
daniel.thibault at forces.gc.ca / Tel: 418-844-4000x4245 Fax: 418-844-4538

DRDC is an agency of the Department of National Defence / RDDC est une agence du ministère de la Défense nationale

[lttng-dev] LTTng 2.9 sessiond misbehaving

[Jérémie Galarneau]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain, Size: 2031 bytes --]

On Oct 5, 2016 3:52 PM, "Thibault, Daniel" <Daniel.Thibault at drdc-rddc.gc.ca>
wrote:
>
> Interesting.  Rebuilt the entire LTTng set with the stable-1.8 releases
(stable-0.9 for userspace-rcu) and I see the same behaviour:
> When I hit ctrl-c in the daemon's shell, it doesn't respond and the CPU
usage goes crazy (~90%).  Can't sudo kill it (or lttng-runas) either.
>
> What's strange is that if instead of:
>
> $ lttng-sessiond -vvv 2>&1 | tee lttng-sessiond.2.log
>
> I do:
>
> $ lttng-sessiond 2>&1 | tee lttng-sessiond.3.log
>
> Then ctrl-c terminates lttng-sessiond normally.
>
> Using just the first level of verbosity:
>
> $ lttng-sessiond -v 2>&1 | tee lttng-sessiond.4.log
>
> also misbehaves.  Somehow -v is responsible?
>
> At this point, should I open a bug?

I'm assuming you meant 2.8.
Open a bug and we'll try to reproduce it.

Does the problem occur with the other verbosity levels? (-vv, -vvv)

Thanks,
Jérémie

>
> Daniel U. Thibault, M.Sc.²
>
> Informaticien scientifique, CME-PSC, Centre de recherches de Valcartier
> Recherche et développement pour la défense Canada / Gouvernement du Canada
> daniel.thibault at drdc-rddc.gc.ca / Tél: 418-844-4000x4245 Téléc:
418-844-4538
> daniel.thibault at forces.gc.ca / Tél: 418-844-4000x4245 Téléc: 418-844-4538
>
> Computer Scientist, MCCS-SPC, Valcartier Research Centre
> Defence Research and Development Canada / Government of Canada
> daniel.thibault at drdc-rddc.gc.ca / Tel: 418-844-4000x4245 Fax: 418-844-4538
> daniel.thibault at forces.gc.ca / Tel: 418-844-4000x4245 Fax: 418-844-4538
>
> DRDC is an agency of the Department of National Defence / RDDC est une
agence du ministère de la Défense nationale
> _______________________________________________
> lttng-dev mailing list
> lttng-dev at lists.lttng.org
> https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.lttng.org/pipermail/lttng-dev/attachments/20161006/e8d5d4da/attachment.html>

[lttng-dev] LTTng 2.9 sessiond misbehaving

[Mathieu Desnoyers]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain, Size: 1903 bytes --]

----- On Oct 5, 2016, at 3:51 PM, Thibault, Daniel Daniel.Thibault at drdc-rddc.gc.ca wrote:

> Interesting.  Rebuilt the entire LTTng set with the stable-1.8 releases
> (stable-0.9 for userspace-rcu) and I see the same behaviour:
> When I hit ctrl-c in the daemon's shell, it doesn't respond and the CPU usage
> goes crazy (~90%).  Can't sudo kill it (or lttng-runas) either.
> 
> What's strange is that if instead of:
> 
> $ lttng-sessiond -vvv 2>&1 | tee lttng-sessiond.2.log
> 
> I do:
> 
> $ lttng-sessiond 2>&1 | tee lttng-sessiond.3.log
> 
> Then ctrl-c terminates lttng-sessiond normally.
> 
> Using just the first level of verbosity:
> 
> $ lttng-sessiond -v 2>&1 | tee lttng-sessiond.4.log
> 
> also misbehaves.  Somehow -v is responsible?
> 
> At this point, should I open a bug?

Hi Daniel,

I've been able to reproduce, and just sent a fix to
Jérémie:

"Fix: ignore SIGPIPE"

Thanks,

Mathieu

> 
> Daniel U. Thibault, M.Sc.²
> 
> Informaticien scientifique, CME-PSC, Centre de recherches de Valcartier
> Recherche et développement pour la défense Canada / Gouvernement du Canada
> daniel.thibault at drdc-rddc.gc.ca / Tél: 418-844-4000x4245 Téléc: 418-844-4538
> daniel.thibault at forces.gc.ca / Tél: 418-844-4000x4245 Téléc: 418-844-4538
> 
> Computer Scientist, MCCS-SPC, Valcartier Research Centre
> Defence Research and Development Canada / Government of Canada
> daniel.thibault at drdc-rddc.gc.ca / Tel: 418-844-4000x4245 Fax: 418-844-4538
> daniel.thibault at forces.gc.ca / Tel: 418-844-4000x4245 Fax: 418-844-4538
> 
> DRDC is an agency of the Department of National Defence / RDDC est une agence du
> ministère de la Défense nationale
> _______________________________________________
> lttng-dev mailing list
> lttng-dev at lists.lttng.org
> https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

-- 
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com