From mboxrd@z Thu Jan 1 00:00:00 1970 From: mwalczak@bbn.com (Mike Walczak) Date: Wed, 06 Apr 2011 11:35:47 -0400 Subject: [ltt-dev] LTTng help Message-ID: <4D9C8853.1000103@bbn.com> Message-ID: <20110406153547.AbQhJTR3VdQFd1m7LgTNcMrl7WiPHcxpgXrqN2QY7ps@z> Hi, I'm a LTTng newbie and would appreciate some hand holding to get me started the right way... I'm having trouble getting traces. I'm trying to debug a bottleneck in a multi-threaded app and have been struggling the last few days to get things started. I initially got a trace but all threads were named the same so I started to instrument the code with thread branding calls. I must have messed something up in the process since the lttv-gui now crashes (bottom of this email) when loading the trace dir. When I tried some of the sample programs from the ltt-usertrace-0.12 I did not get any useful trace info... Again, I'm new to this so please excuse ignorant comments. So basically I'm looking for help in identifying anything that I might have missed in the setup and any possible clues as to why the trace info is not getting generated. Thanks in advance. -Mike Here's the kernel menuconfig: I did not do anything with genevent (was I supposed to?). Are there any other patches for user space tracing ? I'm cross compiling, targetting the Linux kernel 2.6.16.9 (using a target specific toolchain) I downloaded the following components: patch-2.6.16-lttng-0.5.69.tar.bz2. ltt-control-0.10-12062006 ltt-usertrace-0.12 userspace-rcu-0.5.4 ust-0.12 LinuxTraceToolkitViewer-0.8.47-12062006 Is this all or am I missing anything? I took the contents of ltt-control-installed and copied them onto the target into respective dirs. For the sake of completeness, the contents are the below. ltt-control-installed/ ltt-control-installed/usr/ ltt-control-installed/usr/bin/ ltt-control-installed/usr/bin/usttrace ltt-control-installed/usr/bin/ustctl ltt-control-installed/usr/bin/ust-consumerd ltt-control-installed/usr/bin/lttctl ltt-control-installed/usr/bin/lttd ltt-control-installed/usr/share/ ltt-control-installed/usr/share/man/ ltt-control-installed/usr/share/man/man1/ ltt-control-installed/usr/share/man/man1/ust-consumerd.1 ltt-control-installed/usr/share/man/man1/usttrace.1 ltt-control-installed/usr/share/man/man1/ustctl.1 ltt-control-installed/usr/share/info/ ltt-control-installed/usr/share/info/ust.info ltt-control-installed/usr/share/ltt-control/ ltt-control-installed/usr/share/ltt-control/facilities/ ltt-control-installed/usr/share/ltt-control/facilities/stack_arch_i386.xml ltt-control-installed/usr/share/ltt-control/facilities/fs_data.xml ltt-control-installed/usr/share/ltt-control/facilities/user_generic.xml ltt-control-installed/usr/share/ltt-control/facilities/locking.xml ltt-control-installed/usr/share/ltt-control/facilities/kernel_arch_ppc.xml ltt-control-installed/usr/share/ltt-control/facilities/network_ip_interface.xml ltt-control-installed/usr/share/ltt-control/facilities/kernel_arch_powerpc.xml ltt-control-installed/usr/share/ltt-control/facilities/socket.xml ltt-control-installed/usr/share/ltt-control/facilities/memory.xml ltt-control-installed/usr/share/ltt-control/facilities/process.xml ltt-control-installed/usr/share/ltt-control/facilities/statedump.xml ltt-control-installed/usr/share/ltt-control/facilities/ipc.xml ltt-control-installed/usr/share/ltt-control/facilities/network.xml ltt-control-installed/usr/share/ltt-control/facilities/kernel.xml ltt-control-installed/usr/share/ltt-control/facilities/kernel_arch_c2.xml ltt-control-installed/usr/share/ltt-control/facilities/core.xml ltt-control-installed/usr/share/ltt-control/facilities/kernel_arch_arm.xml ltt-control-installed/usr/share/ltt-control/facilities/fs.xml ltt-control-installed/usr/share/ltt-control/facilities/kernel_arch_i386.xml ltt-control-installed/usr/share/ltt-control/facilities/kernel_arch_mips.xml ltt-control-installed/usr/share/ltt-control/facilities/timer.xml ltt-control-installed/usr/include/ ltt-control-installed/usr/include/urcu-defer-static.h ltt-control-installed/usr/include/urcu-pointer.h ltt-control-installed/usr/include/ltt/ ltt-control-installed/usr/include/ltt/ltt-usertrace-fast.h ltt-control-installed/usr/include/ltt/ltt-usertrace.h ltt-control-installed/usr/include/ltt/atomic-ppc.h ltt-control-installed/usr/include/ltt/ltt-facility-id-user_generic.h ltt-control-installed/usr/include/ltt/ppc_asm-ppc.h ltt-control-installed/usr/include/ltt/system-ppc64.h ltt-control-installed/usr/include/ltt/ltt-facility-user_generic.h ltt-control-installed/usr/include/ltt/system-ppc.h ltt-control-installed/usr/include/ltt/timex-ppc.h ltt-control-installed/usr/include/ltt/atomic-ppc64.h ltt-control-installed/usr/include/ltt/ltt-facility-custom-user_generic.h ltt-control-installed/usr/include/ltt/ltt-usertrace-ppc.h ltt-control-installed/usr/include/urcu-bp-static.h ltt-control-installed/usr/include/ust/ ltt-control-installed/usr/include/ust/marker.h ltt-control-installed/usr/include/ust/type-serializer.h ltt-control-installed/usr/include/ust/immediate.h ltt-control-installed/usr/include/ust/kcompat/ ltt-control-installed/usr/include/ust/kcompat/jhash.h ltt-control-installed/usr/include/ust/kcompat/types.h ltt-control-installed/usr/include/ust/kcompat/stringify.h ltt-control-installed/usr/include/ust/kcompat/kcompat.h ltt-control-installed/usr/include/ust/kcompat/compiler.h ltt-control-installed/usr/include/ust/kcompat/simple.h ltt-control-installed/usr/include/ust/define_trace.h ltt-control-installed/usr/include/ust/probe.h ltt-control-installed/usr/include/ust/clock.h ltt-control-installed/usr/include/ust/ust_trace.h ltt-control-installed/usr/include/ust/processor.h ltt-control-installed/usr/include/ust/core.h ltt-control-installed/usr/include/ust/ust.h ltt-control-installed/usr/include/ust/tracectl.h ltt-control-installed/usr/include/ust/tracepoint.h ltt-control-installed/usr/include/ust/ustctl.h ltt-control-installed/usr/include/ust/ustconsumer.h ltt-control-installed/usr/include/urcu-bp.h ltt-control-installed/usr/include/urcu-pointer-static.h ltt-control-installed/usr/include/urcu-defer.h ltt-control-installed/usr/include/urcu-qsbr.h ltt-control-installed/usr/include/urcu/ ltt-control-installed/usr/include/urcu/urcu-futex.h ltt-control-installed/usr/include/urcu/rculist.h ltt-control-installed/usr/include/urcu/rculfqueue.h ltt-control-installed/usr/include/urcu/uatomic_arch.h ltt-control-installed/usr/include/urcu/hlist.h ltt-control-installed/usr/include/urcu/wfstack-static.h ltt-control-installed/usr/include/urcu/urcu_ref.h ltt-control-installed/usr/include/urcu/compiler.h ltt-control-installed/usr/include/urcu/rcuhlist.h ltt-control-installed/usr/include/urcu/rculfstack-static.h ltt-control-installed/usr/include/urcu/wfstack.h ltt-control-installed/usr/include/urcu/arch_generic.h ltt-control-installed/usr/include/urcu/arch.h ltt-control-installed/usr/include/urcu/config.h ltt-control-installed/usr/include/urcu/system.h ltt-control-installed/usr/include/urcu/wfqueue-static.h ltt-control-installed/usr/include/urcu/wfqueue.h ltt-control-installed/usr/include/urcu/uatomic_generic.h ltt-control-installed/usr/include/urcu/list.h ltt-control-installed/usr/include/urcu/rculfstack.h ltt-control-installed/usr/include/urcu/rculfqueue-static.h ltt-control-installed/usr/include/urcu-static.h ltt-control-installed/usr/include/urcu.h ltt-control-installed/usr/include/liblttctl/ ltt-control-installed/usr/include/liblttctl/lttctl.h ltt-control-installed/usr/include/ltt-facility-loader-user_generic.h ltt-control-installed/usr/include/urcu-qsbr-static.h ltt-control-installed/usr/lib/ ltt-control-installed/usr/lib/libustfork.so ltt-control-installed/usr/lib/libust.la ltt-control-installed/usr/lib/libust.so.0 ltt-control-installed/usr/lib/libustctl.so.0 ltt-control-installed/usr/lib/libltt-usertrace-fast.so.0 ltt-control-installed/usr/lib/libtap.so ltt-control-installed/usr/lib/librculfqueue.a ltt-control-installed/usr/lib/liburcu-bp.a ltt-control-installed/usr/lib/libustinstr-malloc.so.0.0.0 ltt-control-installed/usr/lib/libustfork.a ltt-control-installed/usr/lib/liburcu-signal.a ltt-control-installed/usr/lib/libustinstr-malloc.la ltt-control-installed/usr/lib/libwfqueue.la ltt-control-installed/usr/lib/liblttctl.a ltt-control-installed/usr/lib/libustconsumer.so ltt-control-installed/usr/lib/libust.so ltt-control-installed/usr/lib/libltt-instrument-functions.a ltt-control-installed/usr/lib/libustfork.so.0.0.0 ltt-control-installed/usr/lib/libtap.la ltt-control-installed/usr/lib/liburcu-bp.la ltt-control-installed/usr/lib/liblttctl.so.0.0.0 ltt-control-installed/usr/lib/librculfqueue.la ltt-control-installed/usr/lib/liblttctl.so.0 ltt-control-installed/usr/lib/libustctl.la ltt-control-installed/usr/lib/libtap.a ltt-control-installed/usr/lib/libustctl.so ltt-control-installed/usr/lib/libtap.so.0 ltt-control-installed/usr/lib/libltt-instrument-functions.so ltt-control-installed/usr/lib/libwfstack.a ltt-control-installed/usr/lib/libustconsumer.a ltt-control-installed/usr/lib/librculfstack.a ltt-control-installed/usr/lib/libustfork.so.0 ltt-control-installed/usr/lib/libltt-usertrace-fast.so ltt-control-installed/usr/lib/libltt-usertrace-fast.a ltt-control-installed/usr/lib/libustctl.so.0.0.0 ltt-control-installed/usr/lib/libustconsumer.la ltt-control-installed/usr/lib/liblttctl.la ltt-control-installed/usr/lib/libustfork.la ltt-control-installed/usr/lib/libwfqueue.a ltt-control-installed/usr/lib/liburcu-defer.la ltt-control-installed/usr/lib/libust-initializer.o ltt-control-installed/usr/lib/liburcu-qsbr.a ltt-control-installed/usr/lib/liburcu-mb.a ltt-control-installed/usr/lib/liburcu-defer.a ltt-control-installed/usr/lib/libustinstr-malloc.so.0 ltt-control-installed/usr/lib/libtap.so.0.0.0 ltt-control-installed/usr/lib/liburcu-qsbr.la ltt-control-installed/usr/lib/librculfstack.la ltt-control-installed/usr/lib/libustinstr-malloc.a ltt-control-installed/usr/lib/libustctl.a ltt-control-installed/usr/lib/libustinstr-malloc.so ltt-control-installed/usr/lib/liblttctl.so ltt-control-installed/usr/lib/liburcu-mb.la ltt-control-installed/usr/lib/liburcu.a ltt-control-installed/usr/lib/liburcu-signal.la ltt-control-installed/usr/lib/libust.so.0.0.0 ltt-control-installed/usr/lib/liburcu.la ltt-control-installed/usr/lib/libltt-instrument-functions.so.0 ltt-control-installed/usr/lib/libustconsumer.so.0 ltt-control-installed/usr/lib/libwfstack.la ltt-control-installed/usr/lib/libust.a ltt-control-installed/usr/lib/libustconsumer.so.0.0.0 ltt-control-installed/usr/src/ ltt-control-installed/usr/src/sample-thread-brand.c ltt-control-installed/usr/src/sample-instrument-fct.c ltt-control-installed/usr/src/ltt-usertrace-fast.c ltt-control-installed/usr/src/sample-printf.c ltt-control-installed/usr/src/sample-highspeed.c ltt-control-installed/usr/src/ltt-instrument-functions.c ltt-control-installed/usr/src/sample-thread-slow.c ltt-control-installed/usr/src/sample-thread-fast.c ltt-control-installed/usr/src/sample.c ltt-control-installed/usr/src/ltt-facility-loader-user_generic.c ltt-control-installed/usr/test/ ltt-control-installed/usr/test/sample-instrument-fct ltt-control-installed/usr/test/sample-thread-fast ltt-control-installed/usr/test/sample-thread-slow ltt-control-installed/usr/test/sample-thread-brand ltt-control-installed/usr/test/sample-highspeed ltt-control-installed/usr/test/sample-printf First of all, here's how I started things off: (bash) mkdir /mnt/debugfs echo "debugfs /mnt/debugfs debugfs rw 0 0" >> /etc/fstab mount /mnt/debugfs mkdir /mnt/relayfs mount -t relayfs relayfs /mnt/relayfs export LTT_FACILITIES=/usr/share/ltt-control/facilities export LTT_DAEMON=/usr/bin/lttd To start the tracer: lttctl -n trace4 -c -d -t /var/log/trace4 -l /mnt/relayfs/ltt bash-2.04# lttctl -n trace4 -c -d -t /var/log/trace4 -l /mnt/relayfs/ltt/ Linux Trace Toolkit Trace Control Controlling trace : trace4 Linux Trace Toolkit Trace Daemon Reading from relayfs directory : /mnt/relayfs/ltt//trace4 Writing to trace directory : /var/log/trace4 signal 17 received Creating supplementary trace files Appending facility file stack_arch_i386.xml Appending facility file locking.xml Appending facility file process.xml Appending facility file fs_data.xml Appending facility file kernel_arch_mips.xml Appending facility file kernel_arch_ppc.xml Appending facility file timer.xml Appending facility file ipc.xml Appending facility file memory.xml Appending facility file fs.xml Appending facility file kernel_arch_c2.xml Appending facility file kernel.xml Appending facility file network.xml Appending facility file user_generic.xml Appending facility file kernel_arch_i386.xml Appending facility file core.xml Appending facility file socket.xml Appending facility file kernel_arch_powerpc.xml Appending facility file kernel_arch_arm.xml Appending facility file network_ip_interface.xml Appending facility file statedump.xml To stop tracing: bash-2.04# lttctl -n trace4 -R Linux Trace Toolkit Trace Control Controlling trace : trace4 here's lttv-gui crashing when opening the trace [FedoraBox traces]$ lttv-gui The input data file /work/traces/trace4/control/facilities_0 does not contain a trace The input data file /work/traces/trace4/control/interrupts_0 does not contain a trace The input data file /work/traces/trace4/control/network_0 does not contain a trace The input data file /work/traces/trace4/control/processes_0 does not contain a trace The input data file /work/traces/trace4/control/modules_0 does not contain a trace ** ERROR **: Trace /work/traces/trace4 has no facility tracefile aborting... /usr/local/bin/lttv-gui: line 10: 18614 Aborted (core dumped) $LTTV_CMD.real -m guievents -m guifilter -m guicontrolflow -m guistatistics -m guitracecontrol $* [FedoraBox traces]$ -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: clip_image002.jpg Type: image/jpeg Size: 27216 bytes Desc: not available URL: