[lttng-dev] LTTng streams not Hung up even session is destroyed

[lttng-dev] LTTng streams not Hung up even session is destroyed

[sai kiran]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain, Size: 2635 bytes --]

Hi,
 I'm using LTTng 2.9.4  version of lttng-tools, lttng-modules, and
lttng-ust. I'm doing tracing in Xilinx Zynq ZC706 evaluation kit.
the sequence of commands used for tracing:
Mode:
live mode.
For relay daemon:
lttng-relayd -L tcp://127.0.0.1:5344--output=/tmp/lttnglogs/
Session commds:
1) lttng create my-session --live --set-url=net://127.0.0.1
2) lttng enable-channel --kernel --subbuf-size=131072 --num-subbuf=8
channel0
3) lttng enable-channel -u --read-timer=2000000 channel0
4) lttng enable-event --kernel
sched_process_fork,sched_process_exit,irq_handler_entry,irq_handler_exit,sched_switch,sched_waking
-c channel0 --session=my-session
5) lttng enable-event -u kernelprofiler:kernelprofiler_task_drop -c
channel0 --session=mysession
6) lttng start
7) lttng stop
8) lttng destroy

Here my application has a tracepoint of
"kernelprofiler:kernelprofiler_task_drop"
event.

Thanks,
T SAI KIRAN.







On Mon, Jul 30, 2018 at 9:31 PM, Mathieu Desnoyers <
mathieu.desnoyers at efficios.com> wrote:

> Also, please add the lttng-dev mailing list in CC, as this is the
> appropriate forum for this
> type of question.
>
> Thanks,
>
> Mathieu
>
> ----- On Jul 30, 2018, at 12:00 PM, Mathieu Desnoyers <
> mathieu.desnoyers at efficios.com> wrote:
>
> Please provide the versions of lttng-tools, lttng-modules and lttng-ust
> you are using.
>
> Also, please provide the exact sequence of commands you use to control
> tracing so we
> can try it out on our end.
>
> Thanks,
>
> Mathieu
>
> ----- On Jul 30, 2018, at 11:46 AM, sai kiran <sai9951775108 at gmail.com>
> wrote:
>
> Hi,
>
> As I’m tracing both domain (kernel and user space) using LTTng with
> following configuration,
>
> Mode:
>
> *Live *mode.
>
> Kernel domain configuration:
>
> Sub buffer size = 131072,
>
> Number of sub buffers = 8,
>
> Switch-timer = 0,
>
> Reader-domain = 2000000.
>
>
>
> User domain configuration:
>
> Default sub buffer size and sub buffer count,
>
> Switch-timer = 0,
>
> Reader-domain = 2000000.
>
> After LTTng session is destroyed, no stream is “*Hung up*”. Continuously
> I’m getting data from “*relay daemon*”. I’ve killed consumer daemon, even
> though tracing data coming from relay daemon. Please help me. One more
> thing application is running infinitely.
>
>
>
> Thanks & Regards,
>
> T Sai Kiran
>
>
> --
> Mathieu Desnoyers
> EfficiOS Inc.
> http://www.efficios.com
>
>
> --
> Mathieu Desnoyers
> EfficiOS Inc.
> http://www.efficios.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.lttng.org/pipermail/lttng-dev/attachments/20180801/57437f29/attachment.html>

[lttng-dev] LTTng streams not Hung up even session is destroyed

[Jonathan Rajotte-Julien]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain, Size: 2565 bytes --]

Hi Sai,

On Wed, Aug 01, 2018 at 10:05:30PM +0530, sai kiran wrote:
> Hi,
>  I'm using LTTng 2.9.4  version of lttng-tools, lttng-modules, and
> lttng-ust. I'm doing tracing in Xilinx Zynq ZC706 evaluation kit.
> the sequence of commands used for tracing:
> Mode:
> live mode.

What is the reason behind using the live feature here since you are running
things locally (lttng-relayd/--set-url 127.0.0.1)?

Are you doing live analysis or performing analysis after the
fact?

In other word, what are you trying to achieve with tracing?

I just want to make sure that you are using the most appropriate way of tracing
to achieve your goal.

> For relay daemon:
> lttng-relayd -L tcp://127.0.0.1:5344--output=/tmp/lttnglogs/
> Session commds:
> 1) lttng create my-session --live --set-url=net://127.0.0.1
> 2) lttng enable-channel --kernel --subbuf-size=131072 --num-subbuf=8
> channel0
> 3) lttng enable-channel -u --read-timer=2000000 channel0

Are you sure you want to use the read-timer option?

Using 2000000 as its value mean that userspace subbuffers status will only be checked
upon each 2 seconds.

> 4) lttng enable-event --kernel
> sched_process_fork,sched_process_exit,irq_handler_entry,irq_handler_exit,sched_switch,sched_waking
> -c channel0 --session=my-session
> 5) lttng enable-event -u kernelprofiler:kernelprofiler_task_drop -c
> channel0 --session=mysession
> 6) lttng start
> 7) lttng stop
> 8) lttng destroy
> 
> Here my application has a tracepoint of
> "kernelprofiler:kernelprofiler_task_drop"
> event.
> 

> > Hi,
> >
> > After LTTng session is destroyed, no stream is “*Hung up*”. Continuously

What do you mean by "Hung up"? Are you using babeltrace and connecting to the
lttng-relayd daemon on localhost?

What is your expectation?

Do you mean that you are still receiving data on babeltrace side even when
performing a lttng stop/destroy command?

> > I’m getting data from “*relay daemon*”. I’ve killed consumer daemon, even
> > though tracing data coming from relay daemon.

This can happen if babeltrace is "catching-up" on the data, keep in mind that
lttng-relayd have the trace data and can continue to send information to a
babeltrace client even if the consumerd/sessiond is killed or the session
stopped/destroyed.

> > Please help me. One more
> > thing application is running infinitely.

Which application? Your application? Babeltrace? lttng-sessiond?

If you are talking about your application, could you provide a backtrace
indicating that ust is the culprit here?

Cheers

-- 
Jonathan Rajotte-Julien
EfficiOS

[lttng-dev] LTTng streams not Hung up even session is destroyed

[sai kiran]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain, Size: 3723 bytes --]

Hi,
The reason for reader timer option is, UST event throughput is very low so
if I don't use this option I'm not at all getting UST trace data. Even
though the difference between "reader-timer" and "switch-timer" is not
clear for me. Correct me if I wrong,

 "Switch-Timer": After switch timer expires, Consumer daemon will take the
current sub-buffer data and new trace data will be written to next
sub-buffer (if available). So switch between sub-buffer will happen.
"Reader-Timer": After Reader-Timer expires, Consumer daemon will check
sub-buffer is full or not. If Full then consumer daemon will take the data
or else consumer daemon won't take trace data.

I'm using my own application.

Thanks,
T SAI KIRAN.

On Wed, Aug 1, 2018 at 10:43 PM, Jonathan Rajotte-Julien <
jonathan.rajotte-julien at efficios.com> wrote:

> Hi Sai,
>
> On Wed, Aug 01, 2018 at 10:05:30PM +0530, sai kiran wrote:
> > Hi,
> >  I'm using LTTng 2.9.4  version of lttng-tools, lttng-modules, and
> > lttng-ust. I'm doing tracing in Xilinx Zynq ZC706 evaluation kit.
> > the sequence of commands used for tracing:
> > Mode:
> > live mode.
>
> What is the reason behind using the live feature here since you are running
> things locally (lttng-relayd/--set-url 127.0.0.1)?
>
> Are you doing live analysis or performing analysis after the
> fact?
>
> In other word, what are you trying to achieve with tracing?
>
> I just want to make sure that you are using the most appropriate way of
> tracing
> to achieve your goal.
>
> > For relay daemon:
> > lttng-relayd -L tcp://127.0.0.1:5344--output=/tmp/lttnglogs/
> > Session commds:
> > 1) lttng create my-session --live --set-url=net://127.0.0.1
> > 2) lttng enable-channel --kernel --subbuf-size=131072 --num-subbuf=8
> > channel0
> > 3) lttng enable-channel -u --read-timer=2000000 channel0
>
> Are you sure you want to use the read-timer option?
>
> Using 2000000 as its value mean that userspace subbuffers status will only
> be checked
> upon each 2 seconds.
>
> > 4) lttng enable-event --kernel
> > sched_process_fork,sched_process_exit,irq_handler_
> entry,irq_handler_exit,sched_switch,sched_waking
> > -c channel0 --session=my-session
> > 5) lttng enable-event -u kernelprofiler:kernelprofiler_task_drop -c
> > channel0 --session=mysession
> > 6) lttng start
> > 7) lttng stop
> > 8) lttng destroy
> >
> > Here my application has a tracepoint of
> > "kernelprofiler:kernelprofiler_task_drop"
> > event.
> >
>
> > > Hi,
> > >
> > > After LTTng session is destroyed, no stream is “*Hung up*”.
> Continuously
>
> What do you mean by "Hung up"? Are you using babeltrace and connecting to
> the
> lttng-relayd daemon on localhost?
>
> What is your expectation?
>
> Do you mean that you are still receiving data on babeltrace side even when
> performing a lttng stop/destroy command?
>
> > > I’m getting data from “*relay daemon*”. I’ve killed consumer daemon,
> even
> > > though tracing data coming from relay daemon.
>
> This can happen if babeltrace is "catching-up" on the data, keep in mind
> that
> lttng-relayd have the trace data and can continue to send information to a
> babeltrace client even if the consumerd/sessiond is killed or the session
> stopped/destroyed.
>
> > > Please help me. One more
> > > thing application is running infinitely.
>
> Which application? Your application? Babeltrace? lttng-sessiond?
>
> If you are talking about your application, could you provide a backtrace
> indicating that ust is the culprit here?
>
> Cheers
>
> --
> Jonathan Rajotte-Julien
> EfficiOS
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.lttng.org/pipermail/lttng-dev/attachments/20180801/c7fec691/attachment-0001.html>

[lttng-dev] LTTng streams not Hung up even session is destroyed

[Jonathan Rajotte-Julien]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain, Size: 4440 bytes --]

Hi Sai,

You did not answer the other questions I asked. Those answers are necessary for us
to understand what is going on.

On Wed, Aug 01, 2018 at 11:11:39PM +0530, sai kiran wrote:
> Hi,
> The reason for reader timer option is, UST event throughput is very low so
> if I don't use this option I'm not at all getting UST trace data. Even
> though the difference between "reader-timer" and "switch-timer" is not
> clear for me. Correct me if I wrong,

Please read the doc/man page for more information regarding those timer [1].

[1] https://lttng.org/man/1/lttng-enable-channel/v2.10/#doc-_switch_timer

"When a channel’s switch timer fires, a sub-buffer switch happens. This timer
may be used to ensure that event data is consumed and committed to trace files
periodically in case of a low event throughput."

This seems to fit your use case. You might be better off using the switch timer.

Cheers
> 
>  "Switch-Timer": After switch timer expires, Consumer daemon will take the
> current sub-buffer data and new trace data will be written to next
> sub-buffer (if available). So switch between sub-buffer will happen.
> "Reader-Timer": After Reader-Timer expires, Consumer daemon will check
> sub-buffer is full or not. If Full then consumer daemon will take the data
> or else consumer daemon won't take trace data.
> 
> I'm using my own application.
> 
> Thanks,
> T SAI KIRAN.
> 
> On Wed, Aug 1, 2018 at 10:43 PM, Jonathan Rajotte-Julien <
> jonathan.rajotte-julien at efficios.com> wrote:
> 
> > Hi Sai,
> >
> > On Wed, Aug 01, 2018 at 10:05:30PM +0530, sai kiran wrote:
> > > Hi,
> > >  I'm using LTTng 2.9.4  version of lttng-tools, lttng-modules, and
> > > lttng-ust. I'm doing tracing in Xilinx Zynq ZC706 evaluation kit.
> > > the sequence of commands used for tracing:
> > > Mode:
> > > live mode.
> >
> > What is the reason behind using the live feature here since you are running
> > things locally (lttng-relayd/--set-url 127.0.0.1)?
> >
> > Are you doing live analysis or performing analysis after the
> > fact?
> >
> > In other word, what are you trying to achieve with tracing?
> >
> > I just want to make sure that you are using the most appropriate way of
> > tracing
> > to achieve your goal.
> >
> > > For relay daemon:
> > > lttng-relayd -L tcp://127.0.0.1:5344--output=/tmp/lttnglogs/
> > > Session commds:
> > > 1) lttng create my-session --live --set-url=net://127.0.0.1
> > > 2) lttng enable-channel --kernel --subbuf-size=131072 --num-subbuf=8
> > > channel0
> > > 3) lttng enable-channel -u --read-timer=2000000 channel0
> >
> > Are you sure you want to use the read-timer option?
> >
> > Using 2000000 as its value mean that userspace subbuffers status will only
> > be checked
> > upon each 2 seconds.
> >
> > > 4) lttng enable-event --kernel
> > > sched_process_fork,sched_process_exit,irq_handler_
> > entry,irq_handler_exit,sched_switch,sched_waking
> > > -c channel0 --session=my-session
> > > 5) lttng enable-event -u kernelprofiler:kernelprofiler_task_drop -c
> > > channel0 --session=mysession
> > > 6) lttng start
> > > 7) lttng stop
> > > 8) lttng destroy
> > >
> > > Here my application has a tracepoint of
> > > "kernelprofiler:kernelprofiler_task_drop"
> > > event.
> > >
> >
> > > > Hi,
> > > >
> > > > After LTTng session is destroyed, no stream is “*Hung up*”.
> > Continuously
> >
> > What do you mean by "Hung up"? Are you using babeltrace and connecting to
> > the
> > lttng-relayd daemon on localhost?
> >
> > What is your expectation?
> >
> > Do you mean that you are still receiving data on babeltrace side even when
> > performing a lttng stop/destroy command?
> >
> > > > I’m getting data from “*relay daemon*”. I’ve killed consumer daemon,
> > even
> > > > though tracing data coming from relay daemon.
> >
> > This can happen if babeltrace is "catching-up" on the data, keep in mind
> > that
> > lttng-relayd have the trace data and can continue to send information to a
> > babeltrace client even if the consumerd/sessiond is killed or the session
> > stopped/destroyed.
> >
> > > > Please help me. One more
> > > > thing application is running infinitely.
> >
> > Which application? Your application? Babeltrace? lttng-sessiond?
> >
> > If you are talking about your application, could you provide a backtrace
> > indicating that ust is the culprit here?
> >
> > Cheers
> >
> > --
> > Jonathan Rajotte-Julien
> > EfficiOS
> >

-- 
Jonathan Rajotte-Julien
EfficiOS