[lttng-dev] LTTng user-space callstacks

[lttng-dev] LTTng user-space callstacks

[Valentin Grigorev]

Hello!

Currently, callstack collection in LTTng is only available for kernel-space
events with context fields *callstack-kernel *and *callstack-user*.

Is it expected that callstack collection for LTTng-UST will be added too?
And if it is expected, then how soon?

Best regards,
Valentin Grigorev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.lttng.org/pipermail/lttng-dev/attachments/20200326/67722664/attachment.htm>

[lttng-dev] LTTng user-space callstacks

[Mathieu Desnoyers]

----- On Mar 26, 2020, at 1:39 PM, lttng-dev <lttng-dev at lists.lttng.org> wrote: 

> Hello!

> Currently, callstack collection in LTTng is only available for kernel-space
> events with context fields callstack-kernel and callstack-user .

> Is it expected that callstack collection for LTTng-UST will be added too? And if
> it is expected, then how soon?

Hi Valentin, 

It is something that would be interesting and useful, but a lot of work would 
be needed to have stack-walking this is fast enough and reentrant wrt signal 
handlers. Unfortunately, the backtrace(3) functions do not meet those 
requirements. 

Also, for gathering user-space callstacks from the kernel tracer, it only works if 
all user-space is compiled with frame pointers. This is also a limitation that would 
require a lot of work to overcome neatly. 

None of those features are currently on any roadmap due to lack of customers 
showing interest in getting this done. 

Thanks, 

Mathieu 

-- 
Mathieu Desnoyers 
EfficiOS Inc. 
http://www.efficios.com 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.lttng.org/pipermail/lttng-dev/attachments/20200326/4cb1a12c/attachment.htm>

[lttng-dev] LTTng user-space callstacks

[Milian Wolff]

On Donnerstag, 26. M?rz 2020 20:41:17 CET Mathieu Desnoyers via lttng-dev 
wrote:
> ----- On Mar 26, 2020, at 1:39 PM, lttng-dev <lttng-dev at lists.lttng.org> 
wrote:
> > Hello!
> > 
> > Currently, callstack collection in LTTng is only available for
> > kernel-space
> > events with context fields callstack-kernel and callstack-user .
> > 
> > Is it expected that callstack collection for LTTng-UST will be added too?
> > And if it is expected, then how soon?
> 
> Hi Valentin,
> 
> It is something that would be interesting and useful, but a lot of work
> would be needed to have stack-walking this is fast enough and reentrant wrt
> signal handlers. Unfortunately, the backtrace(3) functions do not meet
> those requirements.
> 
> Also, for gathering user-space callstacks from the kernel tracer, it only
> works if all user-space is compiled with frame pointers. This is also a
> limitation that would require a lot of work to overcome neatly.
> 
> None of those features are currently on any roadmap due to lack of customers
> showing interest in getting this done.

libunwind [1] should meet the requirements you need. It is high performance 
thanks to extensive caching, signal safe and does not rely on frame pointers.

[1]: https://github.com/libunwind/libunwind

-- 
Milian Wolff | milian.wolff at kdab.com | Senior Software Engineer
KDAB (Deutschland) GmbH, a KDAB Group company
Tel: +49-30-521325470
KDAB - The Qt, C++ and OpenGL Experts
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 3826 bytes
Desc: not available
URL: <https://lists.lttng.org/pipermail/lttng-dev/attachments/20200326/9c3b8cf4/attachment.bin>

[lttng-dev] LTTng user-space callstacks

[Valentin Grigorev]

Yeah, as far as I know, perf [1] is successfully using libunwind to provide
callstacks. Probably, LTTng team should think about it too, if there are no
serious obstacles.

[1]: https://perf.wiki.kernel.org/index.php/Main_Page


Best regards,
Valentin Grigorev


On Thu, Mar 26, 2020 at 11:53 PM Milian Wolff <milian.wolff at kdab.com> wrote:

> On Donnerstag, 26. M?rz 2020 20:41:17 CET Mathieu Desnoyers via lttng-dev
> wrote:
> > ----- On Mar 26, 2020, at 1:39 PM, lttng-dev <lttng-dev at lists.lttng.org>
>
> wrote:
> > > Hello!
> > >
> > > Currently, callstack collection in LTTng is only available for
> > > kernel-space
> > > events with context fields callstack-kernel and callstack-user .
> > >
> > > Is it expected that callstack collection for LTTng-UST will be added
> too?
> > > And if it is expected, then how soon?
> >
> > Hi Valentin,
> >
> > It is something that would be interesting and useful, but a lot of work
> > would be needed to have stack-walking this is fast enough and reentrant
> wrt
> > signal handlers. Unfortunately, the backtrace(3) functions do not meet
> > those requirements.
> >
> > Also, for gathering user-space callstacks from the kernel tracer, it only
> > works if all user-space is compiled with frame pointers. This is also a
> > limitation that would require a lot of work to overcome neatly.
> >
> > None of those features are currently on any roadmap due to lack of
> customers
> > showing interest in getting this done.
>
> libunwind [1] should meet the requirements you need. It is high
> performance
> thanks to extensive caching, signal safe and does not rely on frame
> pointers.
>
> [1]: https://github.com/libunwind/libunwind
>
> --
> Milian Wolff | milian.wolff at kdab.com | Senior Software Engineer
> KDAB (Deutschland) GmbH, a KDAB Group company
> Tel: +49-30-521325470
> KDAB - The Qt, C++ and OpenGL Experts
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.lttng.org/pipermail/lttng-dev/attachments/20200327/f902fb81/attachment.htm>

[lttng-dev] LTTng user-space callstacks

[Mathieu Desnoyers]

Last time I checked, the main issue with perf's handling of cases that lack frame pointers 
is that they grab a ~2-4kB worth of stack data into the trace, and then post-process it 
with libunwind. This might be OK for sampling use-cases, but is not something I consider 
appropriate for tracing due to considerably increased use of tracing throughput. 

Has the situation changed on the perf side ? 

Thanks, 

Mathieu 

----- On Mar 27, 2020, at 9:44 AM, Valentin Grigorev <valentin.grigorev at jetbrains.com> wrote: 

> Yeah, as far as I know, perf [1] is successfully using libunwind to provide
> callstacks. Probably, LTTng team should think about it too, if there are no
> serious obstacles.

> [1]: [ https://perf.wiki.kernel.org/index.php/Main_Page |
> https://perf.wiki.kernel.org/index.php/Main_Page ]

> Best regards,
> Valentin Grigorev

> On Thu, Mar 26, 2020 at 11:53 PM Milian Wolff < [ mailto:milian.wolff at kdab.com |
> milian.wolff at kdab.com ] > wrote:

>> On Donnerstag, 26. M?rz 2020 20:41:17 CET Mathieu Desnoyers via lttng-dev
>> wrote:
>>> ----- On Mar 26, 2020, at 1:39 PM, lttng-dev < [
>> > mailto:lttng-dev at lists.lttng.org | lttng-dev at lists.lttng.org ] >
>> wrote:
>> > > Hello!

>> > > Currently, callstack collection in LTTng is only available for
>> > > kernel-space
>> > > events with context fields callstack-kernel and callstack-user .

>> > > Is it expected that callstack collection for LTTng-UST will be added too?
>> > > And if it is expected, then how soon?

>> > Hi Valentin,

>> > It is something that would be interesting and useful, but a lot of work
>> > would be needed to have stack-walking this is fast enough and reentrant wrt
>> > signal handlers. Unfortunately, the backtrace(3) functions do not meet
>> > those requirements.

>> > Also, for gathering user-space callstacks from the kernel tracer, it only
>> > works if all user-space is compiled with frame pointers. This is also a
>> > limitation that would require a lot of work to overcome neatly.

>> > None of those features are currently on any roadmap due to lack of customers
>> > showing interest in getting this done.

>> libunwind [1] should meet the requirements you need. It is high performance
>> thanks to extensive caching, signal safe and does not rely on frame pointers.

>> [1]: [ https://github.com/libunwind/libunwind |
>> https://github.com/libunwind/libunwind ]

>> --
>> Milian Wolff | [ mailto:milian.wolff at kdab.com | milian.wolff at kdab.com ] | Senior
>> Software Engineer
>> KDAB (Deutschland) GmbH, a KDAB Group company
>> Tel: +49-30-521325470
>> KDAB - The Qt, C++ and OpenGL Experts
-- 
Mathieu Desnoyers 
EfficiOS Inc. 
http://www.efficios.com 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.lttng.org/pipermail/lttng-dev/attachments/20200327/3aebbbdb/attachment-0001.htm>

[lttng-dev] LTTng user-space callstacks

[Valentin Grigorev]

Hmm, I don't know unfortunately, it should be checked

Best regards,
Valentin Grigorev


On Fri, Mar 27, 2020 at 4:55 PM Mathieu Desnoyers <
mathieu.desnoyers at efficios.com> wrote:

> Last time I checked, the main issue with perf's handling of cases that
> lack frame pointers
> is that they grab a ~2-4kB worth of stack data into the trace, and then
> post-process it
> with libunwind. This might be OK for sampling use-cases, but is not
> something I consider
> appropriate for tracing due to considerably increased use of tracing
> throughput.
>
> Has the situation changed on the perf side ?
>
> Thanks,
>
> Mathieu
>
> ----- On Mar 27, 2020, at 9:44 AM, Valentin Grigorev <
> valentin.grigorev at jetbrains.com> wrote:
>
> Yeah, as far as I know, perf [1] is successfully using libunwind to
> provide callstacks. Probably, LTTng team should think about it too, if
> there are no serious obstacles.
>
> [1]: https://perf.wiki.kernel.org/index.php/Main_Page
>
>
> Best regards,
> Valentin Grigorev
>
>
> On Thu, Mar 26, 2020 at 11:53 PM Milian Wolff <milian.wolff at kdab.com>
> wrote:
>
>> On Donnerstag, 26. M?rz 2020 20:41:17 CET Mathieu Desnoyers via lttng-dev
>> wrote:
>> > ----- On Mar 26, 2020, at 1:39 PM, lttng-dev <lttng-dev at lists.lttng.org>
>>
>> wrote:
>> > > Hello!
>> > >
>> > > Currently, callstack collection in LTTng is only available for
>> > > kernel-space
>> > > events with context fields callstack-kernel and callstack-user .
>> > >
>> > > Is it expected that callstack collection for LTTng-UST will be added
>> too?
>> > > And if it is expected, then how soon?
>> >
>> > Hi Valentin,
>> >
>> > It is something that would be interesting and useful, but a lot of work
>> > would be needed to have stack-walking this is fast enough and reentrant
>> wrt
>> > signal handlers. Unfortunately, the backtrace(3) functions do not meet
>> > those requirements.
>> >
>> > Also, for gathering user-space callstacks from the kernel tracer, it
>> only
>> > works if all user-space is compiled with frame pointers. This is also a
>> > limitation that would require a lot of work to overcome neatly.
>> >
>> > None of those features are currently on any roadmap due to lack of
>> customers
>> > showing interest in getting this done.
>>
>> libunwind [1] should meet the requirements you need. It is high
>> performance
>> thanks to extensive caching, signal safe and does not rely on frame
>> pointers.
>>
>> [1]: https://github.com/libunwind/libunwind
>>
>> --
>> Milian Wolff | milian.wolff at kdab.com | Senior Software Engineer
>> KDAB (Deutschland) GmbH, a KDAB Group company
>> Tel: +49-30-521325470
>> KDAB - The Qt, C++ and OpenGL Experts
>
>
>
> --
> Mathieu Desnoyers
> EfficiOS Inc.
> http://www.efficios.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.lttng.org/pipermail/lttng-dev/attachments/20200327/7165e681/attachment.htm>

[lttng-dev] LTTng user-space callstacks

[Milian Wolff]

On Freitag, 27. M?rz 2020 14:55:08 CET Mathieu Desnoyers wrote:
> Last time I checked, the main issue with perf's handling of cases that lack
> frame pointers is that they grab a ~2-4kB worth of stack data into the
> trace, and then post-process it with libunwind. This might be OK for
> sampling use-cases, but is not something I consider appropriate for tracing
> due to considerably increased use of tracing throughput.
> 
> Has the situation changed on the perf side ?

Perf still does this. But heaptrack, of which I'm the main author, uses 
libunwind to unwind on every heap allocation and it's plenty fast. Obviously 
the impact is measureable and it's far away from even close to zero overhead. 
But it's super useful. I guess having that capability in LTTng would be 
similarly useful and if it's documented to have an overhead people can use it 
with care.

Cheers

-- 
Milian Wolff | milian.wolff at kdab.com | Senior Software Engineer
KDAB (Deutschland) GmbH, a KDAB Group company
Tel: +49-30-521325470
KDAB - The Qt, C++ and OpenGL Experts
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 3826 bytes
Desc: not available
URL: <https://lists.lttng.org/pipermail/lttng-dev/attachments/20200327/11c0a467/attachment.bin>