[lttng-dev] Beginner question: how to inspect scheduling of multi-threaded user application?

[lttng-dev] Beginner question: how to inspect scheduling of multi-threaded user application?

[David Aldrich]

Hi

I am new to tracing in Linux and to lttng. I have a multi-threaded user application and I want to see:


1)      When the threads are scheduled to run

2)      Which cores the threads are running on.

I have installed lttng on Ubuntu 14.04 LTS.  I am expecting to visualise the trace using TraceCompass.

I have read the following doc section:

http://lttng.org/docs/#doc-tracing-your-own-user-application

In order to collect my trace, must I define custom tracepoint definitions ( in a tracepoint provider header file ), and insert tracepoints into my user application, or is there a simpler way of achieving my goal?

Best regards

David

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.lttng.org/pipermail/lttng-dev/attachments/20160824/95d16699/attachment-0001.html>

[lttng-dev] Beginner question: how to inspect scheduling of multi-threaded user application?

[Francis Deslauriers]

Hi David,
If you specifically want to trace the scheduling of the threads of your
app, you don't need custom tracepoints.
Enabling the sched_switch kernel event will give you both of cpu id and
thread id. Look at the cpu_id and next_tid fields.

You can enable the sched_switch event using : lttng enable-event -k
sched_switch

Cheers,
Francis

2016-08-24 3:17 GMT-04:00 David Aldrich <David.Aldrich at emea.nec.com>:

> Hi
>
>
>
> I am new to tracing in Linux and to lttng. I have a multi-threaded user
> application and I want to see:
>
>
>
> 1)      When the threads are scheduled to run
>
> 2)      Which cores the threads are running on.
>
>
>
> I have installed lttng on Ubuntu 14.04 LTS.  I am expecting to visualise
> the trace using TraceCompass.
>
>
>
> I have read the following doc section:
>
>
>
> http://lttng.org/docs/#doc-tracing-your-own-user-application
>
>
>
> In order to collect my trace, must I define custom tracepoint definitions ( in
> a tracepoint provider header file ), and insert tracepoints into my user
> application, or is there a simpler way of achieving my goal?
>
>
>
> Best regards
>
>
>
> David
>
>
>
> _______________________________________________
> lttng-dev mailing list
> lttng-dev at lists.lttng.org
> https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.lttng.org/pipermail/lttng-dev/attachments/20160824/d00c3e4c/attachment-0001.html>

[lttng-dev] Beginner question: how to inspect scheduling of multi-threaded user application?

[Jonathan Rajotte]

Hi,

On Aug 24, 2016 12:18 PM, "Francis Deslauriers" <francis.deslauriers at mail.
utoronto.ca> wrote:
>
> Hi David,
> If you specifically want to trace the scheduling of the threads of your
app, you don't need custom tracepoints.
> Enabling the sched_switch kernel event will give you both of cpu id and
thread id. Look at the cpu_id and next_tid fields.
>
> You can enable the sched_switch event using : lttng enable-event -k
sched_switch

In TraceCompass you can inspect this data with the control flow view and
the Ressource view under the Kernel analysis node under the trace node in
the project explorer.

I'm not sure of the base requirement for those views you can use the safe
enable-event:

lttng enable-event -k 'sched*'


You can also use "lttng track" to limi the gathering of event to a certain
pid.


Another way to reduce the scope would be to filter per procname:


lttng create
lttng add-context -k -t procname
lttng enable-event 'sched*' --filter '$ctx.procname == "PROCNAMEHERE"'

>
> Cheers,
> Francis
>
>
> 2016-08-24 3:17 GMT-04:00 David Aldrich <David.Aldrich at emea.nec.com>:
>>
>> Hi
>>
>>
>>
>> I am new to tracing in Linux and to lttng. I have a multi-threaded user
application and I want to see:
>>
>>
>>
>> 1)      When the threads are scheduled to run
>>
>> 2)      Which cores the threads are running on.
>>
>>
>>
>> I have installed lttng on Ubuntu 14.04 LTS.  I am expecting to visualise
the trace using TraceCompass.
>>
>>
>>
>> I have read the following doc section:
>>
>>
>>
>> http://lttng.org/docs/#doc-tracing-your-own-user-application
>>
>>
>>
>> In order to collect my trace, must I define custom tracepoint
definitions ( in a tracepoint provider header file ), and insert
tracepoints into my user application, or is there a simpler way of
achieving my goal?
>>
>>
>>
>> Best regards
>>
>>
>>
>> David
>>
>>
>>
>>
>> _______________________________________________
>> lttng-dev mailing list
>> lttng-dev at lists.lttng.org
>> https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
>>
>
>
> _______________________________________________
> lttng-dev mailing list
> lttng-dev at lists.lttng.org
> https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.lttng.org/pipermail/lttng-dev/attachments/20160824/3938b9ac/attachment.html>

[lttng-dev] Beginner question: how to inspect scheduling of multi-threaded user application?

[Jonathan Rajotte]

Sorry had a sending problem.

Here is the rest.

On Wed, Aug 24, 2016 at 12:35 PM, Jonathan Rajotte <
jonathan.r.julien at gmail.com> wrote:

> Hi,
>
> On Aug 24, 2016 12:18 PM, "Francis Deslauriers" <
> francis.deslauriers at mail.utoronto.ca> wrote:
> >
> > Hi David,
> > If you specifically want to trace the scheduling of the threads of your
> app, you don't need custom tracepoints.
> > Enabling the sched_switch kernel event will give you both of cpu id and
> thread id. Look at the cpu_id and next_tid fields.
> >
> > You can enable the sched_switch event using : lttng enable-event -k
> sched_switch
>
> In TraceCompass you can inspect this data with the control flow view and
> the Ressource view under the Kernel analysis node under the trace node in
> the project explorer.
>
> I'm not sure of the base requirement for those views you can use the safe
> enable-event:
>

replace "safe" by "easiest".


> lttng enable-event -k 'sched*'
>
>
> You can also use "lttng track" to limi the gathering of event to a certain
> pid.
>
>
> Another way to reduce the scope would be to filter per procname:
>
>
> lttng create
> lttng add-context -k -t procname
> lttng enable-event 'sched*' --filter '$ctx.procname == "PROCNAMEHERE"'
>
lttng start


PROCNAMEHERE can contain '*' wildcard. See the man page for more
information.

Cheers



> >
> > Cheers,
> > Francis
> >
> >
> > 2016-08-24 3:17 GMT-04:00 David Aldrich <David.Aldrich at emea.nec.com>:
> >>
> >> Hi
> >>
> >>
> >>
> >> I am new to tracing in Linux and to lttng. I have a multi-threaded user
> application and I want to see:
> >>
> >>
> >>
> >> 1)      When the threads are scheduled to run
> >>
> >> 2)      Which cores the threads are running on.
> >>
> >>
> >>
> >> I have installed lttng on Ubuntu 14.04 LTS.  I am expecting to
> visualise the trace using TraceCompass.
> >>
> >>
> >>
> >> I have read the following doc section:
> >>
> >>
> >>
> >> http://lttng.org/docs/#doc-tracing-your-own-user-application
> >>
> >>
> >>
> >> In order to collect my trace, must I define custom tracepoint
> definitions ( in a tracepoint provider header file ), and insert
> tracepoints into my user application, or is there a simpler way of
> achieving my goal?
> >>
> >>
> >>
> >> Best regards
> >>
> >>
> >>
> >> David
> >>
> >>
> >>
> >>
> >> _______________________________________________
> >> lttng-dev mailing list
> >> lttng-dev at lists.lttng.org
> >> https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
> >>
> >
> >
> > _______________________________________________
> > lttng-dev mailing list
> > lttng-dev at lists.lttng.org
> > https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
> >
>



-- 
Jonathan Rajotte Julien
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.lttng.org/pipermail/lttng-dev/attachments/20160824/15087f99/attachment.html>

[lttng-dev] Beginner question: how to inspect scheduling of multi-threaded user application?

[David Aldrich]

Hi Jonathan and Francis

Thanks for your helpful replies.  I am having a bit of trouble implementing your suggestions. Here is my session:


$ lttng create

Spawning a session daemon

PERROR [2885/2893]: bind inet: Address already in use (in lttcomm_bind_inet_sock() at inet.c:109)

Warning: An other session daemon is using this JUL port. JUL support will be deactivated not interfering with the tracing.

Session auto-20160826-083823 created.

Traces will be written in <snip>/lttng-traces/auto-20160826-083823



$ sudo lttng enable-event -k sched_switch

Error: Event sched_switch: Session name not found (channel channel0, session auto-20160826-083823)

Warning: Some command(s) went wrong

There seem to be two problems here:


1)      How to kill a lttng daemon that is already running?


2)      How to specify a session?


Sorry that these are basic questions. I would be grateful if you can help please.

Best regards

David

From: Jonathan Rajotte [mailto:jonathan.r.julien@gmail.com]
Sent: 24 August 2016 17:39
To: Francis Deslauriers <francis.deslauriers at mail.utoronto.ca>
Cc: David Aldrich <David.Aldrich at EMEA.NEC.COM>; lttng-dev at lists.lttng.org
Subject: Re: [lttng-dev] Beginner question: how to inspect scheduling of multi-threaded user application?

Sorry had a sending problem.

Here is the rest.

On Wed, Aug 24, 2016 at 12:35 PM, Jonathan Rajotte <jonathan.r.julien at gmail.com<mailto:jonathan.r.julien at gmail.com>> wrote:

Hi,

On Aug 24, 2016 12:18 PM, "Francis Deslauriers" <francis.deslauriers at mail.utoronto.ca<mailto:francis.deslauriers at mail.utoronto.ca>> wrote:
>
> Hi David,
> If you specifically want to trace the scheduling of the threads of your app, you don't need custom tracepoints.
> Enabling the sched_switch kernel event will give you both of cpu id and thread id. Look at the cpu_id and next_tid fields.
>
> You can enable the sched_switch event using : lttng enable-event -k sched_switch

In TraceCompass you can inspect this data with the control flow view and the Ressource view under the Kernel analysis node under the trace node in the project explorer.

I'm not sure of the base requirement for those views you can use the safe enable-event:

replace "safe" by "easiest".


lttng enable-event -k 'sched*'



You can also use "lttng track" to limi the gathering of event to a certain pid.



Another way to reduce the scope would be to filter per procname:



lttng create
lttng add-context -k -t procname
lttng enable-event 'sched*' --filter '$ctx.procname == "PROCNAMEHERE"'
lttng start


PROCNAMEHERE can contain '*' wildcard. See the man page for more information.

Cheers



>
> Cheers,
> Francis
>
>
> 2016-08-24 3:17 GMT-04:00 David Aldrich <David.Aldrich at emea.nec.com<mailto:David.Aldrich at emea.nec.com>>:
>>
>> Hi
>>
>>
>>
>> I am new to tracing in Linux and to lttng. I have a multi-threaded user application and I want to see:
>>
>>
>>
>> 1)      When the threads are scheduled to run
>>
>> 2)      Which cores the threads are running on.
>>
>>
>>
>> I have installed lttng on Ubuntu 14.04 LTS.  I am expecting to visualise the trace using TraceCompass.
>>
>>
>>
>> I have read the following doc section:
>>
>>
>>
>> http://lttng.org/docs/#doc-tracing-your-own-user-application
>>
>>
>>
>> In order to collect my trace, must I define custom tracepoint definitions ( in a tracepoint provider header file ), and insert tracepoints into my user application, or is there a simpler way of achieving my goal?
>>
>>
>>
>> Best regards
>>
>>
>>
>> David
>>
>>
>>
>>
>> _______________________________________________
>> lttng-dev mailing list
>> lttng-dev at lists.lttng.org<mailto:lttng-dev at lists.lttng.org>
>> https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
>>
>
>
> _______________________________________________
> lttng-dev mailing list
> lttng-dev at lists.lttng.org<mailto:lttng-dev at lists.lttng.org>
> https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
>



--
Jonathan Rajotte Julien



Click here<https://www.mailcontrol.com/sr/unEwdsSZGlHGX2PQPOmvUmQbZlJUA6MYaHsjpmEwGrBfobipAMW1ZCPMpsLI18FS3tdTuAJyxqmiyla!1rT5xQ==> to report this email as spam.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.lttng.org/pipermail/lttng-dev/attachments/20160826/1a356c3e/attachment-0001.html>

[lttng-dev] Beginner question: how to inspect scheduling of multi-threaded user application?

[Jonathan Rajotte Julien]

Hi David,

> Hi Jonathan and Francis
>
> Thanks for your helpful replies.  I am having a bit of trouble implementing your suggestions. Here is my session:
>
>
> $ lttng create
>
> Spawning a session daemon
>
> PERROR [2885/2893]: bind inet: Address already in use (in lttcomm_bind_inet_sock() at inet.c:109)
>
> Warning: An other session daemon is using this JUL port. JUL support will be deactivated not interfering with the tracing.
>
> Session auto-20160826-083823 created.
>
> Traces will be written in <snip>/lttng-traces/auto-20160826-083823

First let's get a clean state.
Make sure the user is in the "tracing" group if you plan to do kernel 
tracing from the user.

groups UsernameHere

If not make sure to add the user to the 'tracing' group.


Kill all lttng related daemon.

sudo pkill lttng-sessiond
sudo pkill lttng-relayd

If you installed it via apt-get/ppa a lttng-sessiond should spawn 
otherwise start one with sudo:

sudo lttng-sessiond
or
sudo lttng-sessiond -b
or
sudo lttng-sessiond -d

See lttng-sessiond man page for information regarding these options.

>
>
>
> $ sudo lttng enable-event -k sched_switch
>
> Error: Event sched_switch: Session name not found (channel channel0, session auto-20160826-083823)
>
> Warning: Some command(s) went wrong

This is due to the fact that the state (current session) is kept on a 
per-user basis under ~/.lttng. You started the session with another user 
and then used sudo for "lttng enable-event".

If you want to pass the session name most of the commands support the 
'-s SessionNameHere' option.

>
> There seem to be two problems here:
>
>
> 1)      How to kill a lttng daemon that is already running?
>
>
> 2)      How to specify a session?
>
>
> Sorry that these are basic questions. I would be grateful if you can help please.

Np.

Make sure to check http://lttng.org/docs/ if you did not already.

Cheers

>
> Best regards
>
> David
>
> From: Jonathan Rajotte [mailto:jonathan.r.julien@gmail.com]
> Sent: 24 August 2016 17:39
> To: Francis Deslauriers <francis.deslauriers at mail.utoronto.ca>
> Cc: David Aldrich <David.Aldrich at EMEA.NEC.COM>; lttng-dev at lists.lttng.org
> Subject: Re: [lttng-dev] Beginner question: how to inspect scheduling of multi-threaded user application?
>
> Sorry had a sending problem.
>
> Here is the rest.
>
> On Wed, Aug 24, 2016 at 12:35 PM, Jonathan Rajotte <jonathan.r.julien at gmail.com<mailto:jonathan.r.julien at gmail.com>> wrote:
>
> Hi,
>
> On Aug 24, 2016 12:18 PM, "Francis Deslauriers" <francis.deslauriers at mail.utoronto.ca<mailto:francis.deslauriers at mail.utoronto.ca>> wrote:
>> Hi David,
>> If you specifically want to trace the scheduling of the threads of your app, you don't need custom tracepoints.
>> Enabling the sched_switch kernel event will give you both of cpu id and thread id. Look at the cpu_id and next_tid fields.
>>
>> You can enable the sched_switch event using : lttng enable-event -k sched_switch
> In TraceCompass you can inspect this data with the control flow view and the Ressource view under the Kernel analysis node under the trace node in the project explorer.
>
> I'm not sure of the base requirement for those views you can use the safe enable-event:
>
> replace "safe" by "easiest".
>
>
> lttng enable-event -k 'sched*'
>
>
>
> You can also use "lttng track" to limi the gathering of event to a certain pid.
>
>
>
> Another way to reduce the scope would be to filter per procname:
>
>
>
> lttng create
> lttng add-context -k -t procname
> lttng enable-event 'sched*' --filter '$ctx.procname == "PROCNAMEHERE"'
> lttng start
>
>
> PROCNAMEHERE can contain '*' wildcard. See the man page for more information.
>
> Cheers
>
>
>
>> Cheers,
>> Francis
>>
>>
>> 2016-08-24 3:17 GMT-04:00 David Aldrich <David.Aldrich at emea.nec.com<mailto:David.Aldrich at emea.nec.com>>:
>>> Hi
>>>
>>>
>>>
>>> I am new to tracing in Linux and to lttng. I have a multi-threaded user application and I want to see:
>>>
>>>
>>>
>>> 1)      When the threads are scheduled to run
>>>
>>> 2)      Which cores the threads are running on.
>>>
>>>
>>>
>>> I have installed lttng on Ubuntu 14.04 LTS.  I am expecting to visualise the trace using TraceCompass.
>>>
>>>
>>>
>>> I have read the following doc section:
>>>
>>>
>>>
>>> http://lttng.org/docs/#doc-tracing-your-own-user-application
>>>
>>>
>>>
>>> In order to collect my trace, must I define custom tracepoint definitions ( in a tracepoint provider header file ), and insert tracepoints into my user application, or is there a simpler way of achieving my goal?
>>>
>>>
>>>
>>> Best regards
>>>
>>>
>>>
>>> David
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> lttng-dev mailing list
>>> lttng-dev at lists.lttng.org<mailto:lttng-dev at lists.lttng.org>
>>> https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
>>>
>>
>> _______________________________________________
>> lttng-dev mailing list
>> lttng-dev at lists.lttng.org<mailto:lttng-dev at lists.lttng.org>
>> https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
>>
>
>
> --
> Jonathan Rajotte Julien
>
>
>
> Click here<https://www.mailcontrol.com/sr/unEwdsSZGlHGX2PQPOmvUmQbZlJUA6MYaHsjpmEwGrBfobipAMW1ZCPMpsLI18FS3tdTuAJyxqmiyla!1rT5xQ==> to report this email as spam.
>
>
> _______________________________________________
> lttng-dev mailing list
> lttng-dev at lists.lttng.org
> https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

-- 
Jonathan R. Julien
Efficios