* [lttng-dev] consumerd crashes (streaming)
@ 2012-10-02 19:04 Bernd Hufmann
2012-10-02 19:09 ` David Goulet
0 siblings, 1 reply; 2+ messages in thread
From: Bernd Hufmann @ 2012-10-02 19:04 UTC (permalink / raw)
Hello
I'm using lttng-tools 2.1 RC4 to stream traces over the network. I
noticed a crash of the consumerd for the following scenario:
host1> lttng-relayd
host2> lttng create mySession -U <network URL>
host2> lttng enable-event -u -a
host2> lttng start
host2> lttng stop
After that, I stopped the relayd on host1. Then, when destroying the
session on host2, the consumerd crashes and remains as a zombi process
(<defunct>). I was able to reproduce this multiple times. However, the
screen print-out varied. See at end of mail for an example of the
output. I hope this output helps to find the problem. Could someone
please have a look?
Thanks
Bernd
PERROR: recvmsg inet: Connection reset by peer [in
lttcomm_recvmsg_inet_sock() at inet.c:227]
PERROR: sendmsg inet: Connection reset by peer [in
lttcomm_sendmsg_inet_sock() at inet.c:271]
PERROR: Error in file write: Bad file descriptor [in
lttng_consumer_on_read_subbuffer_mmap() at consumer.c:1214]
Error: Error writing to tracefile (ret: -1 != len: 4096 != subbuf_size: 56)
PERROR: Error in file write: Bad file descriptor [in
lttng_consumer_on_read_subbuffer_mmap() at consumer.c:1214]
Error: Error writing to tracefile (ret: -1 != len: 4096 != subbuf_size: 37)
*** glibc detected *** lttng-consumerd: free(): invalid pointer:
0xb5c02738 ***
======= Backtrace: =========
/lib/i386-linux-gnu/libc.so.6(+0x73e42)[0x282e42]
lttng-consumerd(lttng_consumer_thread_poll_metadata+0x294)[0x804b6c4]
/lib/i386-linux-gnu/libpthread.so.0(+0x6d4c)[0xd84d4c]
/lib/i386-linux-gnu/libc.so.6(clone+0x5e)[0x2f9ace]
======= Memory map: ========
00110000-0012c000 r-xp 00000000 08:01 524197
/lib/i386-linux-gnu/libgcc_s.so.1
0012c000-0012d000 r--p 0001b000 08:01 524197
/lib/i386-linux-gnu/libgcc_s.so.1
0012d000-0012e000 rw-p 0001c000 08:01 524197
/lib/i386-linux-gnu/libgcc_s.so.1
0020e000-0020f000 r-xp 00000000 00:00 0 [vdso]
0020f000-003ae000 r-xp 00000000 08:01 542235
/lib/i386-linux-gnu/libc-2.15.so
003ae000-003b0000 r--p 0019f000 08:01 542235
/lib/i386-linux-gnu/libc-2.15.so
003b0000-003b1000 rw-p 001a1000 08:01 542235
/lib/i386-linux-gnu/libc-2.15.so
003b1000-003b4000 rw-p 00000000 00:00 0
00694000-0069b000 r-xp 00000000 08:01 542251
/lib/i386-linux-gnu/librt-2.15.so
0069b000-0069c000 r--p 00006000 08:01 542251
/lib/i386-linux-gnu/librt-2.15.so
0069c000-0069d000 rw-p 00007000 08:01 542251
/lib/i386-linux-gnu/librt-2.15.so
0087f000-0089d000 r-xp 00000000 08:01 654290
/usr/local/lib/liblttng-ust-ctl.so.0.0.0
0089d000-0089e000 r--p 0001d000 08:01 654290
/usr/local/lib/liblttng-ust-ctl.so.0.0.0
0089e000-0089f000 rw-p 0001e000 08:01 654290
/usr/local/lib/liblttng-ust-ctl.so.0.0.0
00921000-00925000 r-xp 00000000 08:01 654175
/usr/local/lib/liburcu.so.1.0.0
00925000-00926000 r--p 00003000 08:01 654175
/usr/local/lib/liburcu.so.1.0.0
00926000-00927000 rw-p 00004000 08:01 654175
/usr/local/lib/liburcu.so.1.0.0
00a1c000-00a3c000 r-xp 00000000 08:01 523325
/lib/i386-linux-gnu/ld-2.15.so
00a3c000-00a3d000 r--p 0001f000 08:01 523325
/lib/i386-linux-gnu/ld-2.15.so
00a3d000-00a3e000 rw-p 00020000 08:01 523325
/lib/i386-linux-gnu/ld-2.15.so
00a5e000-00a63000 r-xp 00000000 08:01 654191
/usr/local/lib/liburcu-bp.so.1.0.0
00a63000-00a64000 r--p 00004000 08:01 654191
/usr/local/lib/liburcu-bp.so.1.0.0
00a64000-00a65000 rw-p 00005000 08:01 654191
/usr/local/lib/liburcu-bp.so.1.0.0
00d7e000-00d95000 r-xp 00000000 08:01 542249
/lib/i386-linux-gnu/libpthread-2.15.so
00d95000-00d96000 r--p 00016000 08:01 542249
/lib/i386-linux-gnu/libpthread-2.15.so
00d96000-00d97000 rw-p 00017000 08:01 542249
/lib/i386-linux-gnu/libpthread-2.15.so
00d97000-00d99000 rw-p 00000000 00:00 0
08048000-08062000 r-xp 00000000 08:01 654329
/usr/local/lib/lttng/libexec/lttng-consumerd
08062000-08063000 r--p 0001a000 08:01 654329
/usr/local/lib/lttng/libexec/lttng-consumerd
08063000-08064000 rw-p 0001b000 08:01 654329
/usr/local/lib/lttng/libexec/lttng-consumerd
08064000-08066000 rw-p 00000000 00:00 0
09b9e000-09bbf000 rw-p 00000000 00:00 0 [heap]
b53ff000-b5400000 ---p 00000000 00:00 0
b5400000-b5c00000 rw-p 00000000 00:00 0
b5c00000-b5c21000 rw-p 00000000 00:00 0
b5c21000-b5d00000 ---p 00000000 00:00 0
b5e00000-b5e21000 rw-p 00000000 00:00 0
b5e21000-b5f00000 ---p 00000000 00:00 0
b5fd7000-b5fdd000 rw-s 00000000 00:12 27685
/run/shm/ust-shm-tmp-3bEkF7 (deleted)
b5fdd000-b5fe1000 rw-s 00000000 00:12 27723
/run/shm/ust-shm-tmp-YwqSPR (deleted)
b5fe1000-b5fe2000 ---p 00000000 00:00 0
b5fe2000-b67e2000 rw-p 00000000 00:00 0
b67e2000-b67e3000 ---p 00000000 00:00 0
b67e3000-b6fe3000 rw-p 00000000 00:00 0
b6fe3000-b6fe4000 ---p 00000000 00:00 0
b6fe4000-b77e7000 rw-p 00000000 00:00 0
b77e7000-b77ed000 rw-s 00000000 00:12 27681
/run/shm/ust-shm-tmp-7Lo767 (deleted)
b77ed000-b77f1000 rw-s 00000000 00:12 27719
/run/shm/ust-shm-tmp-uEFKcS (deleted)
b77fa000-b77fd000 rw-p 00000000 00:00 0
bfa07000-bfa28000 rw-p 00000000 00:00 0 [stack]
Error: consumer err socket second poll error
Error: Health error occurred in thread_manage_consumer
--
This Communication is Confidential. We only send and receive email on
the basis of the terms set out at www.ericsson.com/email_disclaimer
^ permalink raw reply [flat|nested] 2+ messages in thread
* [lttng-dev] consumerd crashes (streaming)
2012-10-02 19:04 [lttng-dev] consumerd crashes (streaming) Bernd Hufmann
@ 2012-10-02 19:09 ` David Goulet
0 siblings, 0 replies; 2+ messages in thread
From: David Goulet @ 2012-10-02 19:09 UTC (permalink / raw)
Hi Bernd,
This really looks like a problem that is suppose to be fixed by the
patch on lttng-dev actually under review.
I'll let you know once it's merged.
Thanks!
David
Bernd Hufmann:
> Hello
>
> I'm using lttng-tools 2.1 RC4 to stream traces over the network. I
> noticed a crash of the consumerd for the following scenario:
>
> host1> lttng-relayd
>
> host2> lttng create mySession -U <network URL>
> host2> lttng enable-event -u -a
> host2> lttng start
> host2> lttng stop
>
> After that, I stopped the relayd on host1. Then, when destroying the
> session on host2, the consumerd crashes and remains as a zombi process
> (<defunct>). I was able to reproduce this multiple times. However, the
> screen print-out varied. See at end of mail for an example of the
> output. I hope this output helps to find the problem. Could someone
> please have a look?
>
> Thanks
> Bernd
>
> PERROR: recvmsg inet: Connection reset by peer [in
> lttcomm_recvmsg_inet_sock() at inet.c:227]
> PERROR: sendmsg inet: Connection reset by peer [in
> lttcomm_sendmsg_inet_sock() at inet.c:271]
> PERROR: Error in file write: Bad file descriptor [in
> lttng_consumer_on_read_subbuffer_mmap() at consumer.c:1214]
> Error: Error writing to tracefile (ret: -1 != len: 4096 != subbuf_size: 56)
> PERROR: Error in file write: Bad file descriptor [in
> lttng_consumer_on_read_subbuffer_mmap() at consumer.c:1214]
> Error: Error writing to tracefile (ret: -1 != len: 4096 != subbuf_size: 37)
> *** glibc detected *** lttng-consumerd: free(): invalid pointer:
> 0xb5c02738 ***
> ======= Backtrace: =========
> /lib/i386-linux-gnu/libc.so.6(+0x73e42)[0x282e42]
> lttng-consumerd(lttng_consumer_thread_poll_metadata+0x294)[0x804b6c4]
> /lib/i386-linux-gnu/libpthread.so.0(+0x6d4c)[0xd84d4c]
> /lib/i386-linux-gnu/libc.so.6(clone+0x5e)[0x2f9ace]
> ======= Memory map: ========
> 00110000-0012c000 r-xp 00000000 08:01 524197
> /lib/i386-linux-gnu/libgcc_s.so.1
> 0012c000-0012d000 r--p 0001b000 08:01 524197
> /lib/i386-linux-gnu/libgcc_s.so.1
> 0012d000-0012e000 rw-p 0001c000 08:01 524197
> /lib/i386-linux-gnu/libgcc_s.so.1
> 0020e000-0020f000 r-xp 00000000 00:00 0 [vdso]
> 0020f000-003ae000 r-xp 00000000 08:01 542235
> /lib/i386-linux-gnu/libc-2.15.so
> 003ae000-003b0000 r--p 0019f000 08:01 542235
> /lib/i386-linux-gnu/libc-2.15.so
> 003b0000-003b1000 rw-p 001a1000 08:01 542235
> /lib/i386-linux-gnu/libc-2.15.so
> 003b1000-003b4000 rw-p 00000000 00:00 0
> 00694000-0069b000 r-xp 00000000 08:01 542251
> /lib/i386-linux-gnu/librt-2.15.so
> 0069b000-0069c000 r--p 00006000 08:01 542251
> /lib/i386-linux-gnu/librt-2.15.so
> 0069c000-0069d000 rw-p 00007000 08:01 542251
> /lib/i386-linux-gnu/librt-2.15.so
> 0087f000-0089d000 r-xp 00000000 08:01 654290
> /usr/local/lib/liblttng-ust-ctl.so.0.0.0
> 0089d000-0089e000 r--p 0001d000 08:01 654290
> /usr/local/lib/liblttng-ust-ctl.so.0.0.0
> 0089e000-0089f000 rw-p 0001e000 08:01 654290
> /usr/local/lib/liblttng-ust-ctl.so.0.0.0
> 00921000-00925000 r-xp 00000000 08:01 654175
> /usr/local/lib/liburcu.so.1.0.0
> 00925000-00926000 r--p 00003000 08:01 654175
> /usr/local/lib/liburcu.so.1.0.0
> 00926000-00927000 rw-p 00004000 08:01 654175
> /usr/local/lib/liburcu.so.1.0.0
> 00a1c000-00a3c000 r-xp 00000000 08:01 523325
> /lib/i386-linux-gnu/ld-2.15.so
> 00a3c000-00a3d000 r--p 0001f000 08:01 523325
> /lib/i386-linux-gnu/ld-2.15.so
> 00a3d000-00a3e000 rw-p 00020000 08:01 523325
> /lib/i386-linux-gnu/ld-2.15.so
> 00a5e000-00a63000 r-xp 00000000 08:01 654191
> /usr/local/lib/liburcu-bp.so.1.0.0
> 00a63000-00a64000 r--p 00004000 08:01 654191
> /usr/local/lib/liburcu-bp.so.1.0.0
> 00a64000-00a65000 rw-p 00005000 08:01 654191
> /usr/local/lib/liburcu-bp.so.1.0.0
> 00d7e000-00d95000 r-xp 00000000 08:01 542249
> /lib/i386-linux-gnu/libpthread-2.15.so
> 00d95000-00d96000 r--p 00016000 08:01 542249
> /lib/i386-linux-gnu/libpthread-2.15.so
> 00d96000-00d97000 rw-p 00017000 08:01 542249
> /lib/i386-linux-gnu/libpthread-2.15.so
> 00d97000-00d99000 rw-p 00000000 00:00 0
> 08048000-08062000 r-xp 00000000 08:01 654329
> /usr/local/lib/lttng/libexec/lttng-consumerd
> 08062000-08063000 r--p 0001a000 08:01 654329
> /usr/local/lib/lttng/libexec/lttng-consumerd
> 08063000-08064000 rw-p 0001b000 08:01 654329
> /usr/local/lib/lttng/libexec/lttng-consumerd
> 08064000-08066000 rw-p 00000000 00:00 0
> 09b9e000-09bbf000 rw-p 00000000 00:00 0 [heap]
> b53ff000-b5400000 ---p 00000000 00:00 0
> b5400000-b5c00000 rw-p 00000000 00:00 0
> b5c00000-b5c21000 rw-p 00000000 00:00 0
> b5c21000-b5d00000 ---p 00000000 00:00 0
> b5e00000-b5e21000 rw-p 00000000 00:00 0
> b5e21000-b5f00000 ---p 00000000 00:00 0
> b5fd7000-b5fdd000 rw-s 00000000 00:12 27685
> /run/shm/ust-shm-tmp-3bEkF7 (deleted)
> b5fdd000-b5fe1000 rw-s 00000000 00:12 27723
> /run/shm/ust-shm-tmp-YwqSPR (deleted)
> b5fe1000-b5fe2000 ---p 00000000 00:00 0
> b5fe2000-b67e2000 rw-p 00000000 00:00 0
> b67e2000-b67e3000 ---p 00000000 00:00 0
> b67e3000-b6fe3000 rw-p 00000000 00:00 0
> b6fe3000-b6fe4000 ---p 00000000 00:00 0
> b6fe4000-b77e7000 rw-p 00000000 00:00 0
> b77e7000-b77ed000 rw-s 00000000 00:12 27681
> /run/shm/ust-shm-tmp-7Lo767 (deleted)
> b77ed000-b77f1000 rw-s 00000000 00:12 27719
> /run/shm/ust-shm-tmp-uEFKcS (deleted)
> b77fa000-b77fd000 rw-p 00000000 00:00 0
> bfa07000-bfa28000 rw-p 00000000 00:00 0 [stack]
> Error: consumer err socket second poll error
> Error: Health error occurred in thread_manage_consumer
>
>
>
>
>
>
>
^ permalink raw reply [flat|nested] 2+ messages in thread
end of thread, other threads:[~2012-10-02 19:09 UTC | newest]
Thread overview: 2+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2012-10-02 19:04 [lttng-dev] consumerd crashes (streaming) Bernd Hufmann
2012-10-02 19:09 ` David Goulet
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox