[lttng-dev] introduction & usage of LTTNG in machinekit

[lttng-dev] introduction & usage of LTTNG in machinekit

[Michel Dagenais]

How are you progressing to add LTTng instrumentation to the various layers of MachineKit? 

I will have a student this Summer in our lab looking at the use of LTTng on embedded platforms for real-time applications such as MachineKit. He will insure that LTTng is easy to use and deploy on these popular platforms (Raspberry Pi, BeagleBone Black, Parallela), and well documented. He will also look at specific analysis and views that would be useful for such platforms and applications and look at the possibility of tracing the peripheral processors such as the BBB PRU as well. He is in CC and could help you by getting your feedback and proposing adjustments and improvements to the LTTng toolchain as a result.

----- Mail original -----
> Michel,
> 
> > Am 29.03.2015 um 16:31 schrieb Michel Dagenais
> > <michel.dagenais at polymtl.ca>:
> > 
> > 
> >> - is there a complete example out-of-tree kernel module instrumented for
> >> LTTng? I worked through Steve Rostedt's sillymod
> >> (http://lwn.net/Articles/379903 ff) but am fuzzy on "Adding the LTTng
> >> adaptation layer" - is the example from the manual available in toto
> >> somewhere?
> > 
> > A few people in my group have done so. I will check on Monday to get sample
> > code for you.
> 
> on re-reading, I am almost there, but I'd appreciate an example anyway
> 
> > 
> >> - much of the machinekit RT code (HAL library and components) can be
> >> compiled
> >> as userland shared objects (Posix, RT-PREEMPP, Xenomai threads) or kernel
> >> modules (RTAI, and the deprecated Xenomai kernel API), sitting ontop of an
> >> abstracted realtime API ("RTAPI"). Ideally the tracepoints would work
> >> unchanged except for the different context. The manual naturally assumes
> >> an
> >> either-or context. Am I out on a limb here ;-? How would I bring in the
> >> tracepoint definitions in a kernel context - as a separate module maybe or
> >> linked into each using module?
> > 
> > I dont believe that we have examples of code that compile in both kernel
> > and UST contexts. This can certainly be handled with conditional
> > compilation.
> 
> right, I need to wrap my head around the maze of #defines and #includes ;)
> but it looks possible
> 
> > 
> >> - we cannot assume that lttng is available when building machinekit
> >> packages
> >> at least for now, meaning we likely need shim macro definitions and
> >> possibly
> >> fake header files to adjust for the lacking lttng files. Any examples I
> >> can
> >> follow?
> > 
> > MariaDB and others can target different tracers (LTTng, DTrace, none)...
> > Again, I will try to get you good sample code early next week.
> 
> I thought I might not be the first one to do this
> 
> 
> semi-related because I was bitten by it: the Xenomai configs referenced here:
> http://www.xenomai.org/pipermail/xenomai/2013-January/027272.html turn
> CONFIG_FTRACE off, citing performance reasons. I think we'll turn it back on
> in our kernels, cant be that much extra overhead.
> 
> 
> thanks in advance!
> 
> - Michael
> 
> 
> 
> 

[lttng-dev] introduction & usage of LTTNG in machinekit

[Michael Haberler]

Hello Michel, Rafael,

> Am 26.04.2015 um 15:31 schrieb Michel Dagenais <michel.dagenais at polymtl.ca>:
> 
> How are you progressing to add LTTng instrumentation to the various layers of MachineKit? 

I've worked my way through examples to get traces, and I have added autoconf and build support to the machinekit repo so it can be used.

That said, the folks which have timing problems have not caught on yet.. it looks some more guidance, examples and a bit of a machinekit-specific writeup is needed, competing with the other 247 priority projects on my desk ;)

So in principle it's a go as far as the (more important) userland threads flavors (xenomai, rt-preempt) are concerned; RTAI/in-kernel instrumentation would be nice to have to round it out but given the prospects for RTAI's future this might be a bit academic an exercise.
 
> I will have a student this Summer in our lab looking at the use of LTTng on embedded platforms for real-time applications such as MachineKit. He will insure that LTTng is easy to use and deploy on these popular platforms (Raspberry Pi, BeagleBone Black, Parallela), and well documented. He will also look at specific analysis and views that would be useful for such platforms and applications and look at the possibility of tracing the peripheral processors such as the BBB PRU as well. He is in CC and could help you by getting your feedback and proposing adjustments and improvements to the LTTng toolchain as a result.

Well, I would be glad to get Rafael interested in instrumenting machinekit and I am certain we'd be able to coach him through the learning curve so any work actually makes a difference; I would love to see some critical components be instrumented and actually measured on several platforms, and embedded ARM's are becoming more important all the time.

As far as machinekit as a proving ground for LTTNG goes, it is certainly a nontrivial 'customer' - meaning I dont exclude some problems and roadblocks downstream. But the upside for LTTNG I see - if it works here, it should work just about anywhere.


Let me add a question out of the blue: I've read the literature on observing time and distributed systems and am aware of the theoretical issues, as well as clock synchronisation. Still - has any thought been given to merge traces of several systems based on synchronisation points (known causality), well-synchronized  clocks, or other forms of hardware synchronisation?

thanks for caring!

best regards

Michael Haberler

> 
> ----- Mail original -----
>> Michel,
>> 
>>> Am 29.03.2015 um 16:31 schrieb Michel Dagenais
>>> <michel.dagenais at polymtl.ca>:
>>> 
>>> 
>>>> - is there a complete example out-of-tree kernel module instrumented for
>>>> LTTng? I worked through Steve Rostedt's sillymod
>>>> (http://lwn.net/Articles/379903 ff) but am fuzzy on "Adding the LTTng
>>>> adaptation layer" - is the example from the manual available in toto
>>>> somewhere?
>>> 
>>> A few people in my group have done so. I will check on Monday to get sample
>>> code for you.
>> 
>> on re-reading, I am almost there, but I'd appreciate an example anyway
>> 
>>> 
>>>> - much of the machinekit RT code (HAL library and components) can be
>>>> compiled
>>>> as userland shared objects (Posix, RT-PREEMPP, Xenomai threads) or kernel
>>>> modules (RTAI, and the deprecated Xenomai kernel API), sitting ontop of an
>>>> abstracted realtime API ("RTAPI"). Ideally the tracepoints would work
>>>> unchanged except for the different context. The manual naturally assumes
>>>> an
>>>> either-or context. Am I out on a limb here ;-? How would I bring in the
>>>> tracepoint definitions in a kernel context - as a separate module maybe or
>>>> linked into each using module?
>>> 
>>> I dont believe that we have examples of code that compile in both kernel
>>> and UST contexts. This can certainly be handled with conditional
>>> compilation.
>> 
>> right, I need to wrap my head around the maze of #defines and #includes ;)
>> but it looks possible
>> 
>>> 
>>>> - we cannot assume that lttng is available when building machinekit
>>>> packages
>>>> at least for now, meaning we likely need shim macro definitions and
>>>> possibly
>>>> fake header files to adjust for the lacking lttng files. Any examples I
>>>> can
>>>> follow?
>>> 
>>> MariaDB and others can target different tracers (LTTng, DTrace, none)...
>>> Again, I will try to get you good sample code early next week.
>> 
>> I thought I might not be the first one to do this
>> 
>> 
>> semi-related because I was bitten by it: the Xenomai configs referenced here:
>> http://www.xenomai.org/pipermail/xenomai/2013-January/027272.html turn
>> CONFIG_FTRACE off, citing performance reasons. I think we'll turn it back on
>> in our kernels, cant be that much extra overhead.
>> 
>> 
>> thanks in advance!
>> 
>> - Michael
>> 
>> 
>> 
>> 

[lttng-dev] introduction & usage of LTTNG in machinekit

[Michel Dagenais]

> That said, the folks which have timing problems have not caught on yet.. it
> looks some more guidance, examples and a bit of a machinekit-specific
> writeup is needed, competing with the other 247 priority projects on my desk
> ;)

If we can reproduce easily the problems, bad latency on Linux Prempt-RT in a specific configuration, we can take care of tracing and diagnosing the problems. Otherwise, the users will have to activate the tracing themselves on their setup and we can guide them for the diagnosis.

> Let me add a question out of the blue: I've read the literature on observing
> time and distributed systems and am aware of the theoretical issues, as well
> as clock synchronisation. Still - has any thought been given to merge traces
> of several systems based on synchronisation points (known causality),
> well-synchronized  clocks, or other forms of hardware synchronisation?

I suppose that you mean something like this which we offer in TraceCompass:

http://archive.eclipse.org/tracecompass/doc/org.eclipse.tracecompass.doc.user/Trace-synchronization.html

We have two use cases supported already, distributed systems exchanging packets through TCP and physical / virtual machines communicating through vmenter / exit and tracepoints on both sides of those transitions. This framework can easily be extended to other cases. We could in the future make it available declaratively (define event types and unique identifiers that should be used to match causality-related events).

[lttng-dev] introduction & usage of LTTNG in machinekit

[Michael Haberler]

> Am 27.04.2015 um 14:30 schrieb Michel Dagenais <michel.dagenais at polymtl.ca>:
> 
> 
>> That said, the folks which have timing problems have not caught on yet.. it
>> looks some more guidance, examples and a bit of a machinekit-specific
>> writeup is needed, competing with the other 247 priority projects on my desk
>> ;)
> 
> If we can reproduce easily the problems, bad latency on Linux Prempt-RT in a specific configuration, we can take care of tracing and diagnosing the problems. Otherwise, the users will have to activate the tracing themselves on their setup and we can guide them for the diagnosis. 

The problem at hand has these characteristics:

- it happens only on low-end ARM platforms
- the code is part of the trajectory planner, which involves fp and transcendental functions
- the code is executed from a period Xenomai thread and once in a while execution time exceeds the cycle time budget
- no interrupts, system call, or Xenomai API call is involved, so kernel or ipipe tracing will not help to narrow it down

my idea was to insert trace points in the tp code and one triggered on 'time window exceeded' so one could look backwards in history to determine which span of code used excessive time; does this sound reasonable?

Alex (in cc) has a test case which reproducibly triggers the problem

> 
>> Let me add a question out of the blue: I've read the literature on observing
>> time and distributed systems and am aware of the theoretical issues, as well
>> as clock synchronisation. Still - has any thought been given to merge traces
>> of several systems based on synchronisation points (known causality),
>> well-synchronized  clocks, or other forms of hardware synchronisation?
> 
> I suppose that you mean something like this which we offer in TraceCompass:
> 
> http://archive.eclipse.org/tracecompass/doc/org.eclipse.tracecompass.doc.user/Trace-synchronization.html
> 
> We have two use cases supported already, distributed systems exchanging packets through TCP and physical / virtual machines communicating through vmenter / exit and tracepoints on both sides of those transitions. This framework can easily be extended to other cases. We could in the future make it available declaratively (define event types and unique identifiers that should be used to match causality-related events).

That's pretty impressive. I'll try to reproduce an example and think through how this could fit into what we're doing.

Background is - we're getting more into supporting fieldbus-connected components, some of which might run our code, so distributed tracing would become an option

with CANbus in particular, an open question is if driving a remote motion controller is within the range of bus throughput; ballpark - no, but there are some drives which have quite intelligent features and it might work. If not, it would help to know where we loose out, and up to which rates it would still work.

It would good to have such a tool at hand, and the developers aware of it, and how to use it. I could actually see a use case for mixed software tracing and hardware-visible events, like from a logic analyzer.

The other area where I could find use for such a capability is UI/server interaction (could it be there might be much wider interest for this than just me?)

Those is a more 'researchy' applications than the one above though. That one we need to nail.

- Michael

[lttng-dev] introduction & usage of LTTNG in machinekit

[Michel Dagenais]

> The problem at hand has these characteristics:
> 
> - it happens only on low-end ARM platforms
> - the code is part of the trajectory planner, which involves fp and
> transcendental functions
> - the code is executed from a period Xenomai thread and once in a while
> execution time exceeds the cycle time budget
> - no interrupts, system call, or Xenomai API call is involved, so kernel or
> ipipe tracing will not help to narrow it down

I suppose that the time period is in the order of 100us or more?

> my idea was to insert trace points in the tp code and one triggered on 'time
> window exceeded' so one could look backwards in history to determine which
> span of code used excessive time; does this sound reasonable?

In most cases the surprises come from interrupts and competition for resources. If everything is within the tp code, indeed internal tracepoints will help. If you want to be more fancy, there are a few more complicated options. You could trace in "flight recorder" mode, without recording to disk, and trigger a snapshot of the circular buffer content only when a cycle time budget overflow is detected. You could also have a timer, armed for the time window limit, that triggers a stack dump on the interrupted task when fired.

> Alex (in cc) has a test case which reproducibly triggers the problem

Rafa?l will start next week. That would indeed be an interesting problem given that it is reproducible.


> That's pretty impressive. I'll try to reproduce an example and think through
> how this could fit into what we're doing.
> 
> Background is - we're getting more into supporting fieldbus-connected
> components, some of which might run our code, so distributed tracing would
> become an option
...
> The other area where I could find use for such a capability is UI/server
> interaction (could it be there might be much wider interest for this than
> just me?)

Indeed, as long as you can record a trace on each node with local timestamps, and have event pairs with a causal relationship in each direction, it should work. You bring the traces together for analysis and synchronize them using these event pairs.

[lttng-dev] introduction & usage of LTTNG in machinekit

[Michel Dagenais]

> That said, the folks which have timing problems have not caught on yet.. it 
> looks some more guidance, examples and a bit of a machinekit-specific 
> writeup is needed, competing with the other 247 priority projects on my desk 
> ;) 

Many people interested in MachineKit wonder about the control platform to select, from the most inexpensive (Raspberry PI) with more latency issues to the more expensive (Intel with FPGA card), with many interesting alternatives in between. The big question is always the real-time response for a typical challenging target such as 5 axis servo + spindle (3 phase variable frequency). We have the ability with LTTng to diagnose such real-time problems and thus to properly tune and then evaluate these different solutions. At the same time, it provides a challenging testbed for the LTTng toolchain. Your comments and suggestions on the hardware setup to test are most welcome! 

Here is my current list of popular and interesting hardware that we would like to try out and properly assess this Summer: 

- Raspberry PI 2, low end and most inexpensive. 

- BeagleBone Black and eventually BeagleBoard X15, slightly more expensive but very interesting for control applications with PRUs, PWM and eQEP. 

- Combined ARM and FPGA with Xilinx chips like Zedboards. The Adapteva Parallela board comes with a Xilinx Z-7010 or Z-7020. You get a dual-core A9, and FPGA for custom logic for encoder inputs and PWM or step generation. 

- Intel board (Atom or better, suggestions for model welcome) with parallel port I/O or any of the available FPGA I/O cards. 

Indeed, while some of the platforms may work without additional hardware (BeagleBone or Zedboard), an FPGA board provides a robust and high performance solution. I have a MesaNet 5I20 but the PCI bus is becoming legacy. Many newer models are offered using high speed data connections which are very convenient but not normally relied upon for real-time work (USB and Ethernet). However, given their high speed and with a dedicated link, it may be workable. Again an excellent testbed for the LTTng toolchain and some hardware measurements. We could test: 

- Mesanet 6I24 FPGA PCIe card with 72 GPIO (baseline for low latency) 

- Mesanet 7I80HD FPGA Ethernet with 72GPIO 

- Mesanet 7I61 FPGA USB with 96 GPIO 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lttng.org/pipermail/lttng-dev/attachments/20150427/ea505542/attachment.html>