From mboxrd@z Thu Jan 1 00:00:00 1970 From: bryanhsieh@fihspec.com (Bryan Hsieh) Date: Tue, 4 Aug 2009 16:28:18 +0800 Subject: [ltt-dev] Some LTTng 0.43 issue on ARM In-Reply-To: <20090731154237.GA8847@Krystal> References: <7660C7A486834C48B1487748EA32BCA3503EE618A8@spec-ty-exch.fihspec.com> <345025.22508.qm@web63207.mail.re1.yahoo.com> <7660C7A486834C48B1487748EA32BCA3503EE618AA@spec-ty-exch.fihspec.com> <20090731154237.GA8847@Krystal> Message-ID: <7660C7A486834C48B1487748EA32BCA3503EE618AF@spec-ty-exch.fihspec.com> Hi Mathieu, All Please check my comment which marked by [Bryan], Thanks! ----------------------------------------------------------------------------- [Bryan] All the related LTT functions are build-in not build as modules, Please check attachment- LTTng Kernel Config..txt, I am not sure if Kernel->user space relay support (formerly relayfs) need to build-in. Is there any item I missed? ----------------------------------------------------------------------------- [ ] Kernel->user space relay support (formerly relayfs) [*] Activate tracepoints [*] Activate markers Linux Trace Toolkit ---> <*> Compile lttng tracing probes [*] LTTng fine-grained timestamping [*] Linux Trace Toolkit Instrumentation Support <*> Linux Trace Toolkit High-speed Lockless Data Relay <*> Linux Trace Toolkit Lock-Protected Data Relay [*] Debug check for random access in ltt relay buffers <*> Linux Trace Toolkit Serializer <*> Linux Trace Toolkit Marker Control <*> Linux Trace Toolkit Tracer [*] Align Linux Trace Toolkit Traces [*] Add event size field to LTT events for tracer debugging [*] Support trace extraction from crash dump <*> Linux Trace Toolkit Netlink Controller <*> Linux Trace Toolkit State Dump ----------------------------------------------------------------------------- [Bryan] I am sure that the external utilities are build-in busybox, please check the attachment - " busybox version and command query.txt". Finally, I figure out why the ltt-armall doesn't work correctly in my target system, root cause is the output pipeline will be empty after grep twice, even I use a temp file to avoid connecting two grep commands directly, it's still empty. So I copy the /proc/ltt to my local desktop to do ltt-armall script, then generate the /proc/ltt which been filtered by following command. MARKERS=`cat /proc/ltt|grep -v %k|awk '{print $2}'|sort -u|grep -v ^core_|grep -v ^locking_|grep -v ^lockdep_` I rename it as ltt_filter and copy to my target system path /system/bin/, then modify ltt-armall by following modification and run it. MARKERS=`cat /system/bin/ltt_filter` The execution result on screen: # ltt-armall-1 Connecting all markers Connecting fs_buffer_wait_end Connecting fs_buffer_wait_start Connecting fs_close Connecting fs_exec Connecting fs_ioctl Connecting fs_llseek Connecting fs_lseek Connecting fs_open Connecting fs_pollfd Connecting fs_pread64 Connecting fs_pwrite64 Connecting fs_read Connecting fs_readv Connecting fs_select Connecting fs_write Connecting fs_writev Connecting input_event Connecting ipc_msg_create Connecting ipc_sem_create Connecting ipc_shm_create Connecting kernel_arch_ipc_call Connecting kernel_arch_kthread_create Connecting kernel_arch_syscall_entry Connecting kernel_arch_syscall_exit Connecting kernel_arch_trap_entry Connecting kernel_arch_trap_exit Connecting kernel_irq_entry Connecting kernel_irq_exit Connecting kernel_kthread_stop Connecting kernel_kthread_stop_ret Connecting kernel_module_free Connecting kernel_module_load Connecting kernel_printk Connecting kernel_process_exit Connecting kernel_process_fork Connecting kernel_process_free Connecting kernel_process_wait Connecting kernel_sched_migrate_task Connecting kernel_sched_schedule Connecting kernel_sched_try_wakeup Connecting kernel_sched_wait_task Connecting kernel_sched_wakeup_new_task Connecting kernel_send_signal Connecting kernel_softirq_entry Connecting kernel_softirq_exit Connecting kernel_softirq_raise Connecting kernel_tasklet_high_entry Connecting kernel_tasklet_high_exit Connecting kernel_tasklet_low_entry Connecting kernel_tasklet_low_exit Connecting kernel_timer_itimer_expired Connecting kernel_timer_itimer_set Connecting kernel_timer_set Connecting kernel_timer_timeout Connecting kernel_timer_update_time Connecting kernel_vprintk Connecting list_file_descriptor Connecting list_interrupt Connecting list_module Connecting list_network_ip_interface Connecting list_network_ipv4_interface Connecting list_process_state Connecting list_statedump_end Connecting list_vm_map Connecting mm_handle_fault_entry Connecting mm_handle_fault_exit Connecting mm_huge_page_alloc Connecting mm_huge_page_free Connecting mm_page_alloc Connecting mm_page_free Connecting mm_wait_on_page_end Connecting mm_wait_on_page_start Connecting rcu_classic_callback Connecting rcu_classic_call_rcu Connecting rcu_classic_call_rcu_bh Connecting statedump_softirq_vec Connecting high-rate markers to tap Connecting fs_buffer_wait_end Connecting fs_buffer_wait_start Connecting fs_close Connecting fs_exec Connecting fs_ioctl Connecting fs_llseek Connecting fs_lseek Connecting fs_open Connecting fs_pollfd Connecting fs_pread64 Connecting fs_pwrite64 Connecting fs_read Connecting fs_readv Connecting fs_select Connecting fs_write Connecting fs_writev Connecting input_event Connecting ipc_msg_create Connecting ipc_sem_create Connecting ipc_shm_create Connecting kernel_arch_ipc_call Connecting kernel_arch_kthread_create Connecting kernel_arch_syscall_entry Connecting kernel_arch_syscall_exit Connecting kernel_arch_trap_entry Connecting kernel_arch_trap_exit Connecting kernel_irq_entry Connecting kernel_irq_exit Connecting kernel_kthread_stop Connecting kernel_kthread_stop_ret Connecting kernel_module_free Connecting kernel_module_load Connecting kernel_printk Connecting kernel_process_exit Connecting kernel_process_fork Connecting kernel_process_free Connecting kernel_process_wait Connecting kernel_sched_migrate_task Connecting kernel_sched_schedule Connecting kernel_sched_try_wakeup Connecting kernel_sched_wait_task Connecting kernel_sched_wakeup_new_task Connecting kernel_send_signal Connecting kernel_softirq_entry Connecting kernel_softirq_exit Connecting kernel_softirq_raise Connecting kernel_tasklet_high_entry Connecting kernel_tasklet_high_exit Connecting kernel_tasklet_low_entry Connecting kernel_tasklet_low_exit Connecting kernel_timer_itimer_expired Connecting kernel_timer_itimer_set Connecting kernel_timer_set Connecting kernel_timer_timeout Connecting kernel_timer_update_time Connecting kernel_vprintk Connecting list_file_descriptor Connecting list_interrupt Connecting list_module Connecting list_network_ip_interface Connecting list_network_ipv4_interface Connecting list_process_state Connecting list_statedump_end Connecting list_vm_map Connecting mm_handle_fault_entry Connecting mm_handle_fault_exit Connecting mm_huge_page_alloc Connecting mm_huge_page_free Connecting mm_page_alloc Connecting mm_page_free Connecting mm_wait_on_page_end Connecting mm_wait_on_page_start Connecting rcu_classic_callback Connecting rcu_classic_call_rcu Connecting rcu_classic_call_rcu_bh Connecting statedump_softirq_vec ----------------------------------------------------------------------------- [Bryan] It's that correct result? ----------------------------------------------------------------------------- # cat /proc/ltt marker: kernel_arch_kthread_create format: "pid %ld fn %p" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_arch_syscall_exit format: " " state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_arch_syscall_entry format: "syscall_id %d ip #p%ld" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_arch_ipc_call format: "call %u first %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_arch_trap_exit format: " " state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_arch_trap_entry format: "trap_id %ld ip #p%ld" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: statedump_softirq_vec format: "id %d address %p symbol %s" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: list_module format: "name %s state %d refcount %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: core_marker_format format: "name %s format %s" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: core_marker_format format: "name %s format %s" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: list_statedump_end format: " " state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: list_process_state format: "pid %d parent_pid %d name %s type %d mode %d submode %d status %d tgid %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: list_file_descriptor format: "filename %s pid %d fd %u" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: list_vm_map format: "pid %d start %lu end %lu flags %lu pgoff %lu inode %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: list_interrupt format: "name %s action %s irq_id %u" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: list_network_ip_interface format: "name %s address #4u%lu up %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: list_network_ipv4_interface format: "name %s address #4u%lu up %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: core_marker_id format: "name %s id %hu int #1u%zu long #1u%zu pointer #1u%zu size_t #1u%zu alignment #1u%u" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: core_marker_id format: "name %s id %hu int #1u%zu long #1u%zu pointer #1u%zu size_t #1u%zu alignment #1u%u" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: core_marker_format format: "name %s format %s" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: core_marker_id format: "name %s id %hu int #1u%zu long #1u%zu pointer #1u%zu size_t #1u%zu alignment #1u%u" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: core_marker_id format: "name %s id %hu int #1u%zu long #1u%zu pointer #1u%zu size_t #1u%zu alignment #1u%u" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_module_load format: "name %s" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_module_free format: "name %s" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_vprintk format: "loglevel %c string %s ip %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_printk format: "ip %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_timer_timeout format: "pid %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_timer_update_time format: "jiffies #8u%llu xtime_sec %ld xtime_nsec %ld walltomonotonic_sec %ld walltomonotonic_nsec %ld" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_timer_set format: "expires %lu function %p data %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_timer_itimer_set format: "which %d interval_sec %ld interval_usec %ld value_sec %ld value_usec %ld" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_timer_itimer_expired format: "pid %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_process_fork format: "parent_pid %d child_pid %d child_tgid %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_process_wait format: "pid %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_process_exit format: "pid %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_process_free format: "pid %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_send_signal format: "pid %d signal %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_sched_migrate_task format: "pid %d state %ld dest_cpu %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_sched_schedule format: "prev_pid %d next_pid %d prev_state %ld" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_sched_wakeup_new_task format: "pid %d state %ld cpu_id %u" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_sched_try_wakeup format: "pid %d state %ld cpu_id %u" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_sched_wait_task format: "pid %d state %ld" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_kthread_stop_ret format: "ret %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_kthread_stop format: "pid %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_tasklet_high_exit format: "func %p data %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_tasklet_high_entry format: "func %p data %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_tasklet_low_exit format: "func %p data %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_tasklet_low_entry format: "func %p data %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_softirq_raise format: "softirq_id %u" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_softirq_exit format: "softirq_id %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_softirq_entry format: "softirq_id %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_irq_exit format: "handled #1u%u" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_irq_entry format: "irq_id %u kernel_mode %u ip %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: mm_page_alloc format: "order %u pfn %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: mm_page_free format: "order %u pfn %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: mm_handle_fault_exit format: "res %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: mm_handle_fault_entry format: "address %lu ip #p%ld write_access %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: mm_huge_page_alloc format: "pfn %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: mm_huge_page_free format: "pfn %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: mm_wait_on_page_end format: "pfn %lu bit_nr %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: mm_wait_on_page_start format: "pfn %lu bit_nr %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: fs_pollfd format: "fd %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: fs_select format: "fd %d timeout #8d%lld" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: fs_writev format: "fd %lu vlen %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: fs_readv format: "fd %lu vlen %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: fs_pwrite64 format: "fd %u count %zu pos %llu" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: fs_pread64 format: "fd %u count %zu pos %llu" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: fs_write format: "fd %u count %zu" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: fs_read format: "fd %u count %zu" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: fs_llseek format: "fd %u offset %lld origin %u" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: fs_lseek format: "fd %u offset %ld origin %u" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: fs_close format: "fd %u" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: fs_open format: "fd %d filename %s" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: fs_ioctl format: "fd %u cmd %u arg %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: fs_exec format: "filename %s" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: fs_buffer_wait_end format: "bh %p" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: fs_buffer_wait_start format: "bh %p" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: ipc_shm_create format: "id %ld flags %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: ipc_sem_create format: "id %ld flags %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: ipc_msg_create format: "id %ld flags %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: lockdep_lock_release format: "retaddr 0x%lX lock %p nested %d" state: 0 call: 0xc0093374 probe single : 0xc00920ec marker: lockdep_lock_acquire format: "retaddr 0x%lX subclass %u lock %p trylock %d read %d hardirqs_off %d" state: 0 call: 0xc0093374 probe single : 0xc00920ec marker: lockdep_softirqs_off format: "retaddr 0x%lX" state: 0 call: 0xc0093374 probe single : 0xc00920ec marker: lockdep_softirqs_on format: "retaddr 0x%lX" state: 0 call: 0xc0093374 probe single : 0xc00920ec marker: lockdep_hardirqs_off format: "retaddr 0x%lX" state: 0 call: 0xc0093374 probe single : 0xc00920ec marker: lockdep_hardirqs_on format: "retaddr 0x%lX" state: 0 call: 0xc0093374 probe single : 0xc00920ec marker: rcu_classic_call_rcu_bh format: "func %p ip 0x%lX" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: rcu_classic_call_rcu format: "func %p ip 0x%lX" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: rcu_classic_callback format: "func %p" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: input_event format: "type %u code %u value %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 ----------------------------------------------------------------------------- [Bryan] But the /proc/ltt is still same, it's that correct? What's the expected result? ----------------------------------------------------------------------------- # df /dev: 39564K total, 0K used, 39564K available (block size 4096) /sqlite_stmt_journals: 4096K total, 0K used, 4096K available (block size 4096) /system: 72704K total, 71680K used, 1024K available (block size 4096) /data: 66816K total, 37812K used, 29004K available (block size 4096) /cache: 67584K total, 1156K used, 66428K available (block size 4096) /sdcard: 991708K total, 147040K used, 844668K available (block size 4096) ----------------------------------------------------------------------------- [Bryan] There is 29004K available for trace log file /data/ltt/trace1. Is that enough? ----------------------------------------------------------------------------- # cat /proc/meminfo MemTotal: 79128 kB MemFree: 2440 kB Buffers: 36 kB Cached: 29872 kB SwapCached: 0 kB Active: 61384 kB Inactive: 5052 kB SwapTotal: 0 kB SwapFree: 0 kB Dirty: 0 kB Writeback: 0 kB AnonPages: 36552 kB Mapped: 24888 kB Slab: 3856 kB SReclaimable: 904 kB SUnreclaim: 2952 kB PageTables: 3660 kB NFS_Unstable: 0 kB Bounce: 0 kB WritebackTmp: 0 kB CommitLimit: 39564 kB Committed_AS: 1032216 kB VmallocTotal: 409600 kB VmallocUsed: 52908 kB VmallocChunk: 332796 kB ----------------------------------------------------------------------------- [Bryan] There is 2440 kB free memory. Is that enough? ----------------------------------------------------------------------------- # lttctl -n trace1 -d -l /sys/kernel/debug/ltt -t /data/ltt/trace1 Linux Trace Toolkit Trace Control 0.55-16102008 Controlling trace : trace1 Linux Trace Toolkit Trace Daemon 0.55-16102008 Reading from debugfs directory : /sys/kernel/debug/ltt/trace1 Writing to trace directory : /data/ltt/trace1 ----------------------------------------------------------------------------- [Bryan] do some tests on target device, then check the channel and trace ----------------------------------------------------------------------------- # ls -l /sys/kernel/debug/ltt/trace1 -r-------- root root 0 1980-01-06 00:10 cpu_0 drwxr-xr-x root root 1980-01-06 00:10 control # ls -l /sys/kernel/debug/ltt/trace1/control -r-------- root root 0 1980-01-06 00:10 network_0 -r-------- root root 0 1980-01-06 00:10 modules_0 -r-------- root root 0 1980-01-06 00:10 processes_0 -r-------- root root 0 1980-01-06 00:10 interrupts_0 -r-------- root root 0 1980-01-06 00:10 metadata_0 # ls -l /data/ltt/trace1 drwxrwxrwx root root 1980-01-06 00:10 control -rwxrwxrwx root root 29949952 1980-01-06 00:10 cpu_0 # ls -l /data/ltt/trace1/control -rwxrwxrwx root root 0 1980-01-06 00:10 metadata_0 -rwxrwxrwx root root 0 1980-01-06 00:10 interrupts_0 -rwxrwxrwx root root 0 1980-01-06 00:10 processes_0 -rwxrwxrwx root root 0 1980-01-06 00:10 modules_0 -rwxrwxrwx root root 0 1980-01-06 00:10 network_0 ----------------------------------------------------------------------------- [Bryan] It seems that the capacity has been exhausted by cpu_0, because 29004K available. It's that correct that cpu_0 will grow to so huge (29M)? ----------------------------------------------------------------------------- # lttctl -n trace1 -d -R Linux Trace Toolkit Trace Control 0.55-16102008 Controlling trace : trace1 ^Z[1] + Stopped lttctl -n trace1 -d -R ----------------------------------------------------------------------------- [Bryan] hang in here...I have to break it by Ctrl+Z, What's problem? ----------------------------------------------------------------------------- # ls -l /data/ltt/trace1/ drwxrwxrwx root root 1980-01-06 00:10 control -rwxrwxrwx root root 29949952 1980-01-06 00:10 cpu_0 # ls -l /data/ltt/trace1/control -rwxrwxrwx root root 0 1980-01-06 00:10 metadata_0 -rwxrwxrwx root root 0 1980-01-06 00:10 interrupts_0 -rwxrwxrwx root root 0 1980-01-06 00:10 processes_0 -rwxrwxrwx root root 0 1980-01-06 00:10 modules_0 -rwxrwxrwx root root 0 1980-01-06 00:10 network_0 # ls -l /sys/kernel/debug/ltt/trace1 -r-------- root root 0 1980-01-06 00:10 cpu_0 drwxr-xr-x root root 1980-01-06 00:10 control # ls -l /sys/kernel/debug/ltt/trace1/control -r-------- root root 0 1980-01-06 00:10 network_0 -r-------- root root 0 1980-01-06 00:10 modules_0 -r-------- root root 0 1980-01-06 00:10 processes_0 -r-------- root root 0 1980-01-06 00:10 interrupts_0 -r-------- root root 0 1980-01-06 00:10 metadata_0 # df /dev: 39564K total, 0K used, 39564K available (block size 4096) /sqlite_stmt_journals: 4096K total, 0K used, 4096K available (block size 4096) /system: 72704K total, 71680K used, 1024K available (block size 4096) /data: 66816K total, 66732K used, 84K available (block size 4096) /cache: 67584K total, 1156K used, 66428K available (block size 4096) /sdcard: 991708K total, 147040K used, 844668K available (block size 4096) ----------------------------------------------------------------------------- [Bryan] I think the trace is incomplete. ----------------------------------------------------------------------------- Please help me! Thank you very much! Best regards, Bryan Hsieh SPEC, FIH Co., Ltd. Tel:+886-3-270-0000 #2723 Fax:+886-3-270-0001 -----Original Message----- From: Mathieu Desnoyers [mailto:compudj@krystal.dyndns.org] Sent: Friday, July 31, 2009 11:43 PM To: Bryan Hsieh Cc: 'Ajit R'; 'ltt-dev at lists.casi.polymtl.ca'; Tommy Peng Subject: Re: [ltt-dev] Some LTTng 0.43 issue on ARM * Bryan Hsieh (bryanhsieh at fihspec.com) wrote: > Hi Ajit, > > I appreciate your reply! > But there is a problem here. > > My installation of Busybox is 1.12.4. It sounds like either awk or sed or not built in your busybox installation. Can you provide the output of cat /proc/ltt on your target system ? Check the content of ltt-armall to see which external programs it needs. Also, if you built the lttng probes as modules, you may need to load them explicitely. Mathieu > > I follow your suggestion to test on x86 Desktop ARM simulator, the message said that can't find /proc/ltt. I think it should be, because I didn't patch to ARM simulator Linux kernel. > > [sbox-ARM: ~/ltt-control-0.55-16102008/lttctl] > ltt-armall > Connecting all markers > cat: /proc/ltt: No such file or directory > Connecting high-rate markers to tap > cat: /proc/ltt: No such file or directory > > It should be work on my target device, but it seems not. > I can't find /ltt under /proc > Maybe it's root cause, but why there is no /ltt/proc? And why there is no " cat: /proc/ltt: No such file or directory" message in my target device during running ltt-armall? > > BTW, I mount debugfs in init.c > Mount("debugfs", "/sys/kernel/debug", "debugfs", 0, NULL); > It should not be a problem. > > -----------------------------------------------------------------------------------------------------------Another problem: > I command as following steps. > # lttctl -n trace1 -q > Linux Trace Toolkit Trace Control 0.55-16102008 > Controlling trace : trace1 > > # lttctl -n trace1 -r > Linux Trace Toolkit Trace Control 0.55-16102008 > Controlling trace : trace1 > > >>> Android shell hang in here, I have to press Ctrl+Z to force stopping - lttctl -n trace1 -r .Why? Is anything wrong? > # ls -l ../tmp/trace1 > drwxrwxrwx root root 1980-01-07 06:37 control > -rwxrwxrwx root root 1048576 1980-01-07 06:37 cpu_0 > # ls -l ../tmp/trace1/control > -rwxrwxrwx root root 0 1980-01-07 06:37 metadata_0 > -rwxrwxrwx root root 0 1980-01-07 06:37 interrupts_0 > -rwxrwxrwx root root 10240 1980-01-07 06:37 processes_0 > -rwxrwxrwx root root 65536 1980-01-07 06:37 modules_0 > -rwxrwxrwx root root 65536 1980-01-07 06:37 network_0 > > The Channel in /sys/kernel/debug/ltt/trace1 is clear. Does it cause by destroy channel command? > > The metadata_0 and interrupts_0 are empty, Is that correct? I've done some key press, the interrupts_o shouldn't be empt. > > And those files almost filled by 0 except for the short header I can't make sure, it looks like incomplete. > > I hope you can help me to solve these issues, I am appreciate you so much.. > > Best regards, > Bryan Hsieh > > SPEC, FIH Co., Ltd. > Tel:+886-3-270-0000 #2723 > Fax:+886-3-270-0001 > > > -----Original Message----- > From: Ajit R [mailto:ajitrzid@yahoo.com] > Sent: Friday, July 31, 2009 6:17 PM > To: 'ltt-dev at lists.casi.polymtl.ca'; Bryan Hsieh > Cc: Tommy Peng > Subject: Re: [ltt-dev] Some LTTng 0.43 issue on ARM > > > Hi Bryan, > My replies with the [Ajit] tag. > LTT Gods correct me if I am wrong. > > Thanks > Ajit > > --- On Fri, 7/31/09, Bryan Hsieh wrote: > > > From: Bryan Hsieh > > Subject: [ltt-dev] Some LTTng 0.43 issue on ARM > > To: "'ltt-dev at lists.casi.polymtl.ca'" > > Cc: "Tommy Peng" > > Date: Friday, July 31, 2009, 12:19 PM > > > Hi Mathieu, > > all, > > > > > > > > > > I've ported > > LTTng 0.43 on Android successfully, > > but I am wondering if the result is correct, please help me > > to verify, thank > > you very much! > > > > > > > > > > My Traced > > machine: > > > > ARM11 (ARM1136J(F)-S) / Linux > > 2.6..27 > > (Android 1.5) > > > > My > > visualization machine: > > > > X86 / Sun xVM Virtual Box (Unbuntu > > 8.10) > > > > > > > > LTTng: 0.43 > > > > LTTV: > > 0.11.2 > > > > LTT Control: 0.55 > > > > > > > > > > I > > connect to Android > > shell by ADB tool after Android bring up, then I command as > > following step, the > > command line options are different with LTTng manual, > > because the example in manual > > is for old version. > > > > > > > > > > # ltt-armall > > > [ Ajit ] > ltt-armall is a shell script. It connects + arms the markers in the > kernel. So depending on your kernel + LTT version it would write some > "text" into the /proc/ltt directory or a "1" into files in > the debugfs directory. > Make sure, the binaries for running the script are available in your target environment. For eg. awk, grep, sed etc. Busybox versions might have their limitations. If they do not work, you could do equivalent things on your Desktop and then write a simple script, doing the same on your target. > Once you run "ltt-armall", you should see some debug statements in your "dmesg" like "connect" or "arm" etc. > > > > > Connecting all > > markers > > > > Connecting high-rate > > markers to tap > > > > # lttctl > > -n trace1 -d -l /sys/kernel/debug/ltt -t > > /system/traces/trace1 > > > [Ajit] > Not sure if your "-l" option is right. you have to point it to the place where you mounted your debugfs. > > > > > Linux Trace Toolkit > > Trace Control 0.55-16102008 > > > > > > > > > > Controlling trace : > > trace1 > > > > > > > > > > Linux Trace Toolkit > > Trace Daemon 0.55-16102008 > > > > > > > > > > Reading from debugfs > > directory : /sys/kernel/debug/ltt/trace1 > > > > > > Writing to trace > > directory : /system/traces/trace1 > > > > > > > > > > It > > seems the LTT control > > and daemon work fine. I run some scenarios for 1 minute on > > Android, but the trace > > files are still empty (all files length are > > 0) > > > > What's > > wrong? Is > > there any step I missed? Or the LTT control and daemon > > don't work? > > > > > > > > > > # ls > > -l ../../sys/kernel/debug/ltt/ > > > > > > > > drwxr-xr-x > > root > > root > > 1980-01-07 01:13 trace1 > > > > # ls > > -l > > ../../sys/kernel/debug/ltt/trace1 > > > > > > -r-------- > > root > > root > > 0 > > 1980-01-07 01:13 cpu_0 > > > > [Ajit] > If you have connected and armed it right, you should see the cpu_0 > file increasing in size. After sometime, disconnect by running the > ltt-control -n whatever-name -R > and then you will see files in the control directory (network, process, etc ) grown in size. Tar and move the trace directory to your host and do the needful. > > > > drwxr-xr-x > > root > > root > > 1980-01-07 01:13 control > > > > # ls > > -l > > ../../sys/kernel/debug/ltt/trace1/control > > > > > > -r-------- > > root > > root > > 0 > > 1980-01-07 01:13 network_0 > > > > -r-------- > > root > > root > > 0 > > 1980-01-07 01:13 modules_0 > > > > -r-------- > > root > > root > > 0 > > 1980-01-07 01:13 processes_0 > > > > -r-------- > > root > > root > > 0 > > 1980-01-07 01:13 interrupts_0 > > > > -r-------- > > root > > root > > 0 1980-01-07 01:13 metadata_0 > > > > # ls -l > > .../traces/ > > > > > > > > drwxrwxrwx > > root > > root > > 1980-01-07 01:13 trace1 > > > > # ls -l > > .../traces/trace1 > > > > drwxrwxrwx > > root > > root > > 1980-01-07 01:13 control > > > > -rwxrwxrwx > > root > > root > > 0 > > 1980-01-07 01:13 cpu_0 > > > > # ls -l > > .../traces/trace1/control > > > > > > > > -rwxrwxrwx > > root > > root > > 0 > > 1980-01-07 01:13 metadata_0 > > > > -rwxrwxrwx > > root > > root > > 0 > > 1980-01-07 01:13 interrupts_0 > > > > -rwxrwxrwx > > root > > root > > 0 > > 1980-01-07 01:13 processes_0 > > > > -rwxrwxrwx > > root > > root > > 0 > > 1980-01-07 01:13 modules_0 > > > > -rwxrwxrwx > > root > > root > > 0 > > 1980-01-07 01:13 network_0 > > > > > > > > > > BTW, > > I would like to > > know what is the channel concept for LTTng? What's > > the relationship between > > trace and channel? What's the normal, flight recorder > > and hybrid mode? > > > > Which > > files should I get > > for LTTV on another visualization > > machine? > > > > So far, I > > can't find related document or the > > information isn't enough for me. > > > > > > > > > > Thanks for your > > help! > > > > > > > > > > Best > > regards, > > > > Bryan Hsieh > > > > SPEC, FIH > > Co., Ltd. > > > > Tel:+886-3-270-0000 #2723 > > > > Fax:+886-3-270-0001 > > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > ltt-dev mailing list > > ltt-dev at lists.casi.polymtl.ca > > http://lists.casi.polymtl.ca/cgi-bin/mailman/listinfo/ltt-dev > > > > > > This e-mail message together with any attachments thereto (if any) is confidential, protected under an enforceable non-disclosure agreement, > intended only for the use of the named recipient(s) above and may contain information that is privileged, belonging to professional work products > or exempt from disclosure under applicable laws. Any unauthorized review, use, copying, disclosure, or distribution of any information contained > in or attached to this transmission is STRICTLY PROHIBITED and may be against the laws. If you have received this message in error, or are not > the named recipient(s), please immediately notify the sender by e-mail or telephone at IT support hotline 886-3-2700000 EXT 1333 and delete this > e-mail message and any attached documentation from your computer. Receipt by anyone other than the intended recipient(s) is not a waiver of any > attorney-client or work product privilege. Thank you! > > > _______________________________________________ > ltt-dev mailing list > ltt-dev at lists.casi.polymtl.ca > http://lists.casi.polymtl.ca/cgi-bin/mailman/listinfo/ltt-dev > -- Mathieu Desnoyers OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F BA06 3F25 A8FE 3BAE 9A68 This e-mail message together with any attachments thereto (if any) is confidential, protected under an enforceable non-disclosure agreement, intended only for the use of the named recipient(s) above and may contain information that is privileged, belonging to professional work products or exempt from disclosure under applicable laws. Any unauthorized review, use, copying, disclosure, or distribution of any information contained in or attached to this transmission is STRICTLY PROHIBITED and may be against the laws. If you have received this message in error, or are not the named recipient(s), please immediately notify the sender by e-mail or telephone at IT support hotline 886-3-2700000 EXT 1333 and delete this e-mail message and any attached documentation from your computer. Receipt by anyone other than the intended recipient(s) is not a waiver of any attorney-client or work product privilege. Thank you! -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: busybox version and command query.txt URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: LTTng Execution Log.txt URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: LTTng Kernel Config.txt URL: From mboxrd@z Thu Jan 1 00:00:00 1970 From: bryanhsieh@fihspec.com (Bryan Hsieh) Date: Tue, 4 Aug 2009 16:28:18 +0800 Subject: [ltt-dev] Some LTTng 0.43 issue on ARM In-Reply-To: <20090731154237.GA8847@Krystal> References: <7660C7A486834C48B1487748EA32BCA3503EE618A8@spec-ty-exch.fihspec.com> <345025.22508.qm@web63207.mail.re1.yahoo.com> <7660C7A486834C48B1487748EA32BCA3503EE618AA@spec-ty-exch.fihspec.com> <20090731154237.GA8847@Krystal> Message-ID: <7660C7A486834C48B1487748EA32BCA3503EE618AF@spec-ty-exch.fihspec.com> Message-ID: <20090804082818.n2h0l6IzE-D8vZKCQHHiBvLjHzVZD7VsJH0hzT3_V3c@z> Hi Mathieu, All Please check my comment which marked by [Bryan], Thanks! ----------------------------------------------------------------------------- [Bryan] All the related LTT functions are build-in not build as modules, Please check attachment- LTTng Kernel Config..txt, I am not sure if Kernel->user space relay support (formerly relayfs) need to build-in. Is there any item I missed? ----------------------------------------------------------------------------- [ ] Kernel->user space relay support (formerly relayfs) [*] Activate tracepoints [*] Activate markers Linux Trace Toolkit ---> <*> Compile lttng tracing probes [*] LTTng fine-grained timestamping [*] Linux Trace Toolkit Instrumentation Support <*> Linux Trace Toolkit High-speed Lockless Data Relay <*> Linux Trace Toolkit Lock-Protected Data Relay [*] Debug check for random access in ltt relay buffers <*> Linux Trace Toolkit Serializer <*> Linux Trace Toolkit Marker Control <*> Linux Trace Toolkit Tracer [*] Align Linux Trace Toolkit Traces [*] Add event size field to LTT events for tracer debugging [*] Support trace extraction from crash dump <*> Linux Trace Toolkit Netlink Controller <*> Linux Trace Toolkit State Dump ----------------------------------------------------------------------------- [Bryan] I am sure that the external utilities are build-in busybox, please check the attachment - " busybox version and command query.txt". Finally, I figure out why the ltt-armall doesn't work correctly in my target system, root cause is the output pipeline will be empty after grep twice, even I use a temp file to avoid connecting two grep commands directly, it's still empty. So I copy the /proc/ltt to my local desktop to do ltt-armall script, then generate the /proc/ltt which been filtered by following command. MARKERS=`cat /proc/ltt|grep -v %k|awk '{print $2}'|sort -u|grep -v ^core_|grep -v ^locking_|grep -v ^lockdep_` I rename it as ltt_filter and copy to my target system path /system/bin/, then modify ltt-armall by following modification and run it. MARKERS=`cat /system/bin/ltt_filter` The execution result on screen: # ltt-armall-1 Connecting all markers Connecting fs_buffer_wait_end Connecting fs_buffer_wait_start Connecting fs_close Connecting fs_exec Connecting fs_ioctl Connecting fs_llseek Connecting fs_lseek Connecting fs_open Connecting fs_pollfd Connecting fs_pread64 Connecting fs_pwrite64 Connecting fs_read Connecting fs_readv Connecting fs_select Connecting fs_write Connecting fs_writev Connecting input_event Connecting ipc_msg_create Connecting ipc_sem_create Connecting ipc_shm_create Connecting kernel_arch_ipc_call Connecting kernel_arch_kthread_create Connecting kernel_arch_syscall_entry Connecting kernel_arch_syscall_exit Connecting kernel_arch_trap_entry Connecting kernel_arch_trap_exit Connecting kernel_irq_entry Connecting kernel_irq_exit Connecting kernel_kthread_stop Connecting kernel_kthread_stop_ret Connecting kernel_module_free Connecting kernel_module_load Connecting kernel_printk Connecting kernel_process_exit Connecting kernel_process_fork Connecting kernel_process_free Connecting kernel_process_wait Connecting kernel_sched_migrate_task Connecting kernel_sched_schedule Connecting kernel_sched_try_wakeup Connecting kernel_sched_wait_task Connecting kernel_sched_wakeup_new_task Connecting kernel_send_signal Connecting kernel_softirq_entry Connecting kernel_softirq_exit Connecting kernel_softirq_raise Connecting kernel_tasklet_high_entry Connecting kernel_tasklet_high_exit Connecting kernel_tasklet_low_entry Connecting kernel_tasklet_low_exit Connecting kernel_timer_itimer_expired Connecting kernel_timer_itimer_set Connecting kernel_timer_set Connecting kernel_timer_timeout Connecting kernel_timer_update_time Connecting kernel_vprintk Connecting list_file_descriptor Connecting list_interrupt Connecting list_module Connecting list_network_ip_interface Connecting list_network_ipv4_interface Connecting list_process_state Connecting list_statedump_end Connecting list_vm_map Connecting mm_handle_fault_entry Connecting mm_handle_fault_exit Connecting mm_huge_page_alloc Connecting mm_huge_page_free Connecting mm_page_alloc Connecting mm_page_free Connecting mm_wait_on_page_end Connecting mm_wait_on_page_start Connecting rcu_classic_callback Connecting rcu_classic_call_rcu Connecting rcu_classic_call_rcu_bh Connecting statedump_softirq_vec Connecting high-rate markers to tap Connecting fs_buffer_wait_end Connecting fs_buffer_wait_start Connecting fs_close Connecting fs_exec Connecting fs_ioctl Connecting fs_llseek Connecting fs_lseek Connecting fs_open Connecting fs_pollfd Connecting fs_pread64 Connecting fs_pwrite64 Connecting fs_read Connecting fs_readv Connecting fs_select Connecting fs_write Connecting fs_writev Connecting input_event Connecting ipc_msg_create Connecting ipc_sem_create Connecting ipc_shm_create Connecting kernel_arch_ipc_call Connecting kernel_arch_kthread_create Connecting kernel_arch_syscall_entry Connecting kernel_arch_syscall_exit Connecting kernel_arch_trap_entry Connecting kernel_arch_trap_exit Connecting kernel_irq_entry Connecting kernel_irq_exit Connecting kernel_kthread_stop Connecting kernel_kthread_stop_ret Connecting kernel_module_free Connecting kernel_module_load Connecting kernel_printk Connecting kernel_process_exit Connecting kernel_process_fork Connecting kernel_process_free Connecting kernel_process_wait Connecting kernel_sched_migrate_task Connecting kernel_sched_schedule Connecting kernel_sched_try_wakeup Connecting kernel_sched_wait_task Connecting kernel_sched_wakeup_new_task Connecting kernel_send_signal Connecting kernel_softirq_entry Connecting kernel_softirq_exit Connecting kernel_softirq_raise Connecting kernel_tasklet_high_entry Connecting kernel_tasklet_high_exit Connecting kernel_tasklet_low_entry Connecting kernel_tasklet_low_exit Connecting kernel_timer_itimer_expired Connecting kernel_timer_itimer_set Connecting kernel_timer_set Connecting kernel_timer_timeout Connecting kernel_timer_update_time Connecting kernel_vprintk Connecting list_file_descriptor Connecting list_interrupt Connecting list_module Connecting list_network_ip_interface Connecting list_network_ipv4_interface Connecting list_process_state Connecting list_statedump_end Connecting list_vm_map Connecting mm_handle_fault_entry Connecting mm_handle_fault_exit Connecting mm_huge_page_alloc Connecting mm_huge_page_free Connecting mm_page_alloc Connecting mm_page_free Connecting mm_wait_on_page_end Connecting mm_wait_on_page_start Connecting rcu_classic_callback Connecting rcu_classic_call_rcu Connecting rcu_classic_call_rcu_bh Connecting statedump_softirq_vec ----------------------------------------------------------------------------- [Bryan] It's that correct result? ----------------------------------------------------------------------------- # cat /proc/ltt marker: kernel_arch_kthread_create format: "pid %ld fn %p" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_arch_syscall_exit format: " " state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_arch_syscall_entry format: "syscall_id %d ip #p%ld" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_arch_ipc_call format: "call %u first %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_arch_trap_exit format: " " state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_arch_trap_entry format: "trap_id %ld ip #p%ld" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: statedump_softirq_vec format: "id %d address %p symbol %s" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: list_module format: "name %s state %d refcount %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: core_marker_format format: "name %s format %s" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: core_marker_format format: "name %s format %s" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: list_statedump_end format: " " state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: list_process_state format: "pid %d parent_pid %d name %s type %d mode %d submode %d status %d tgid %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: list_file_descriptor format: "filename %s pid %d fd %u" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: list_vm_map format: "pid %d start %lu end %lu flags %lu pgoff %lu inode %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: list_interrupt format: "name %s action %s irq_id %u" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: list_network_ip_interface format: "name %s address #4u%lu up %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: list_network_ipv4_interface format: "name %s address #4u%lu up %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: core_marker_id format: "name %s id %hu int #1u%zu long #1u%zu pointer #1u%zu size_t #1u%zu alignment #1u%u" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: core_marker_id format: "name %s id %hu int #1u%zu long #1u%zu pointer #1u%zu size_t #1u%zu alignment #1u%u" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: core_marker_format format: "name %s format %s" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: core_marker_id format: "name %s id %hu int #1u%zu long #1u%zu pointer #1u%zu size_t #1u%zu alignment #1u%u" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: core_marker_id format: "name %s id %hu int #1u%zu long #1u%zu pointer #1u%zu size_t #1u%zu alignment #1u%u" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_module_load format: "name %s" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_module_free format: "name %s" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_vprintk format: "loglevel %c string %s ip %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_printk format: "ip %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_timer_timeout format: "pid %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_timer_update_time format: "jiffies #8u%llu xtime_sec %ld xtime_nsec %ld walltomonotonic_sec %ld walltomonotonic_nsec %ld" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_timer_set format: "expires %lu function %p data %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_timer_itimer_set format: "which %d interval_sec %ld interval_usec %ld value_sec %ld value_usec %ld" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_timer_itimer_expired format: "pid %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_process_fork format: "parent_pid %d child_pid %d child_tgid %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_process_wait format: "pid %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_process_exit format: "pid %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_process_free format: "pid %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_send_signal format: "pid %d signal %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_sched_migrate_task format: "pid %d state %ld dest_cpu %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_sched_schedule format: "prev_pid %d next_pid %d prev_state %ld" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_sched_wakeup_new_task format: "pid %d state %ld cpu_id %u" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_sched_try_wakeup format: "pid %d state %ld cpu_id %u" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_sched_wait_task format: "pid %d state %ld" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_kthread_stop_ret format: "ret %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_kthread_stop format: "pid %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_tasklet_high_exit format: "func %p data %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_tasklet_high_entry format: "func %p data %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_tasklet_low_exit format: "func %p data %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_tasklet_low_entry format: "func %p data %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_softirq_raise format: "softirq_id %u" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_softirq_exit format: "softirq_id %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_softirq_entry format: "softirq_id %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_irq_exit format: "handled #1u%u" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_irq_entry format: "irq_id %u kernel_mode %u ip %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: mm_page_alloc format: "order %u pfn %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: mm_page_free format: "order %u pfn %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: mm_handle_fault_exit format: "res %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: mm_handle_fault_entry format: "address %lu ip #p%ld write_access %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: mm_huge_page_alloc format: "pfn %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: mm_huge_page_free format: "pfn %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: mm_wait_on_page_end format: "pfn %lu bit_nr %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: mm_wait_on_page_start format: "pfn %lu bit_nr %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: fs_pollfd format: "fd %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: fs_select format: "fd %d timeout #8d%lld" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: fs_writev format: "fd %lu vlen %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: fs_readv format: "fd %lu vlen %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: fs_pwrite64 format: "fd %u count %zu pos %llu" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: fs_pread64 format: "fd %u count %zu pos %llu" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: fs_write format: "fd %u count %zu" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: fs_read format: "fd %u count %zu" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: fs_llseek format: "fd %u offset %lld origin %u" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: fs_lseek format: "fd %u offset %ld origin %u" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: fs_close format: "fd %u" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: fs_open format: "fd %d filename %s" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: fs_ioctl format: "fd %u cmd %u arg %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: fs_exec format: "filename %s" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: fs_buffer_wait_end format: "bh %p" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: fs_buffer_wait_start format: "bh %p" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: ipc_shm_create format: "id %ld flags %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: ipc_sem_create format: "id %ld flags %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: ipc_msg_create format: "id %ld flags %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: lockdep_lock_release format: "retaddr 0x%lX lock %p nested %d" state: 0 call: 0xc0093374 probe single : 0xc00920ec marker: lockdep_lock_acquire format: "retaddr 0x%lX subclass %u lock %p trylock %d read %d hardirqs_off %d" state: 0 call: 0xc0093374 probe single : 0xc00920ec marker: lockdep_softirqs_off format: "retaddr 0x%lX" state: 0 call: 0xc0093374 probe single : 0xc00920ec marker: lockdep_softirqs_on format: "retaddr 0x%lX" state: 0 call: 0xc0093374 probe single : 0xc00920ec marker: lockdep_hardirqs_off format: "retaddr 0x%lX" state: 0 call: 0xc0093374 probe single : 0xc00920ec marker: lockdep_hardirqs_on format: "retaddr 0x%lX" state: 0 call: 0xc0093374 probe single : 0xc00920ec marker: rcu_classic_call_rcu_bh format: "func %p ip 0x%lX" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: rcu_classic_call_rcu format: "func %p ip 0x%lX" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: rcu_classic_callback format: "func %p" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: input_event format: "type %u code %u value %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 ----------------------------------------------------------------------------- [Bryan] But the /proc/ltt is still same, it's that correct? What's the expected result? ----------------------------------------------------------------------------- # df /dev: 39564K total, 0K used, 39564K available (block size 4096) /sqlite_stmt_journals: 4096K total, 0K used, 4096K available (block size 4096) /system: 72704K total, 71680K used, 1024K available (block size 4096) /data: 66816K total, 37812K used, 29004K available (block size 4096) /cache: 67584K total, 1156K used, 66428K available (block size 4096) /sdcard: 991708K total, 147040K used, 844668K available (block size 4096) ----------------------------------------------------------------------------- [Bryan] There is 29004K available for trace log file /data/ltt/trace1. Is that enough? ----------------------------------------------------------------------------- # cat /proc/meminfo MemTotal: 79128 kB MemFree: 2440 kB Buffers: 36 kB Cached: 29872 kB SwapCached: 0 kB Active: 61384 kB Inactive: 5052 kB SwapTotal: 0 kB SwapFree: 0 kB Dirty: 0 kB Writeback: 0 kB AnonPages: 36552 kB Mapped: 24888 kB Slab: 3856 kB SReclaimable: 904 kB SUnreclaim: 2952 kB PageTables: 3660 kB NFS_Unstable: 0 kB Bounce: 0 kB WritebackTmp: 0 kB CommitLimit: 39564 kB Committed_AS: 1032216 kB VmallocTotal: 409600 kB VmallocUsed: 52908 kB VmallocChunk: 332796 kB ----------------------------------------------------------------------------- [Bryan] There is 2440 kB free memory. Is that enough? ----------------------------------------------------------------------------- # lttctl -n trace1 -d -l /sys/kernel/debug/ltt -t /data/ltt/trace1 Linux Trace Toolkit Trace Control 0.55-16102008 Controlling trace : trace1 Linux Trace Toolkit Trace Daemon 0.55-16102008 Reading from debugfs directory : /sys/kernel/debug/ltt/trace1 Writing to trace directory : /data/ltt/trace1 ----------------------------------------------------------------------------- [Bryan] do some tests on target device, then check the channel and trace ----------------------------------------------------------------------------- # ls -l /sys/kernel/debug/ltt/trace1 -r-------- root root 0 1980-01-06 00:10 cpu_0 drwxr-xr-x root root 1980-01-06 00:10 control # ls -l /sys/kernel/debug/ltt/trace1/control -r-------- root root 0 1980-01-06 00:10 network_0 -r-------- root root 0 1980-01-06 00:10 modules_0 -r-------- root root 0 1980-01-06 00:10 processes_0 -r-------- root root 0 1980-01-06 00:10 interrupts_0 -r-------- root root 0 1980-01-06 00:10 metadata_0 # ls -l /data/ltt/trace1 drwxrwxrwx root root 1980-01-06 00:10 control -rwxrwxrwx root root 29949952 1980-01-06 00:10 cpu_0 # ls -l /data/ltt/trace1/control -rwxrwxrwx root root 0 1980-01-06 00:10 metadata_0 -rwxrwxrwx root root 0 1980-01-06 00:10 interrupts_0 -rwxrwxrwx root root 0 1980-01-06 00:10 processes_0 -rwxrwxrwx root root 0 1980-01-06 00:10 modules_0 -rwxrwxrwx root root 0 1980-01-06 00:10 network_0 ----------------------------------------------------------------------------- [Bryan] It seems that the capacity has been exhausted by cpu_0, because 29004K available. It's that correct that cpu_0 will grow to so huge (29M)? ----------------------------------------------------------------------------- # lttctl -n trace1 -d -R Linux Trace Toolkit Trace Control 0.55-16102008 Controlling trace : trace1 ^Z[1] + Stopped lttctl -n trace1 -d -R ----------------------------------------------------------------------------- [Bryan] hang in here...I have to break it by Ctrl+Z, What's problem? ----------------------------------------------------------------------------- # ls -l /data/ltt/trace1/ drwxrwxrwx root root 1980-01-06 00:10 control -rwxrwxrwx root root 29949952 1980-01-06 00:10 cpu_0 # ls -l /data/ltt/trace1/control -rwxrwxrwx root root 0 1980-01-06 00:10 metadata_0 -rwxrwxrwx root root 0 1980-01-06 00:10 interrupts_0 -rwxrwxrwx root root 0 1980-01-06 00:10 processes_0 -rwxrwxrwx root root 0 1980-01-06 00:10 modules_0 -rwxrwxrwx root root 0 1980-01-06 00:10 network_0 # ls -l /sys/kernel/debug/ltt/trace1 -r-------- root root 0 1980-01-06 00:10 cpu_0 drwxr-xr-x root root 1980-01-06 00:10 control # ls -l /sys/kernel/debug/ltt/trace1/control -r-------- root root 0 1980-01-06 00:10 network_0 -r-------- root root 0 1980-01-06 00:10 modules_0 -r-------- root root 0 1980-01-06 00:10 processes_0 -r-------- root root 0 1980-01-06 00:10 interrupts_0 -r-------- root root 0 1980-01-06 00:10 metadata_0 # df /dev: 39564K total, 0K used, 39564K available (block size 4096) /sqlite_stmt_journals: 4096K total, 0K used, 4096K available (block size 4096) /system: 72704K total, 71680K used, 1024K available (block size 4096) /data: 66816K total, 66732K used, 84K available (block size 4096) /cache: 67584K total, 1156K used, 66428K available (block size 4096) /sdcard: 991708K total, 147040K used, 844668K available (block size 4096) ----------------------------------------------------------------------------- [Bryan] I think the trace is incomplete. ----------------------------------------------------------------------------- Please help me! Thank you very much! Best regards, Bryan Hsieh SPEC, FIH Co., Ltd. Tel:+886-3-270-0000 #2723 Fax:+886-3-270-0001 -----Original Message----- From: Mathieu Desnoyers [mailto:compudj@krystal.dyndns.org] Sent: Friday, July 31, 2009 11:43 PM To: Bryan Hsieh Cc: 'Ajit R'; 'ltt-dev at lists.casi.polymtl.ca'; Tommy Peng Subject: Re: [ltt-dev] Some LTTng 0.43 issue on ARM * Bryan Hsieh (bryanhsieh at fihspec.com) wrote: > Hi Ajit, > > I appreciate your reply! > But there is a problem here. > > My installation of Busybox is 1.12.4. It sounds like either awk or sed or not built in your busybox installation. Can you provide the output of cat /proc/ltt on your target system ? Check the content of ltt-armall to see which external programs it needs. Also, if you built the lttng probes as modules, you may need to load them explicitely. Mathieu > > I follow your suggestion to test on x86 Desktop ARM simulator, the message said that can't find /proc/ltt. I think it should be, because I didn't patch to ARM simulator Linux kernel. > > [sbox-ARM: ~/ltt-control-0.55-16102008/lttctl] > ltt-armall > Connecting all markers > cat: /proc/ltt: No such file or directory > Connecting high-rate markers to tap > cat: /proc/ltt: No such file or directory > > It should be work on my target device, but it seems not. > I can't find /ltt under /proc > Maybe it's root cause, but why there is no /ltt/proc? And why there is no " cat: /proc/ltt: No such file or directory" message in my target device during running ltt-armall? > > BTW, I mount debugfs in init.c > Mount("debugfs", "/sys/kernel/debug", "debugfs", 0, NULL); > It should not be a problem. > > -----------------------------------------------------------------------------------------------------------Another problem: > I command as following steps. > # lttctl -n trace1 -q > Linux Trace Toolkit Trace Control 0.55-16102008 > Controlling trace : trace1 > > # lttctl -n trace1 -r > Linux Trace Toolkit Trace Control 0.55-16102008 > Controlling trace : trace1 > > >>> Android shell hang in here, I have to press Ctrl+Z to force stopping - lttctl -n trace1 -r .Why? Is anything wrong? > # ls -l ../tmp/trace1 > drwxrwxrwx root root 1980-01-07 06:37 control > -rwxrwxrwx root root 1048576 1980-01-07 06:37 cpu_0 > # ls -l ../tmp/trace1/control > -rwxrwxrwx root root 0 1980-01-07 06:37 metadata_0 > -rwxrwxrwx root root 0 1980-01-07 06:37 interrupts_0 > -rwxrwxrwx root root 10240 1980-01-07 06:37 processes_0 > -rwxrwxrwx root root 65536 1980-01-07 06:37 modules_0 > -rwxrwxrwx root root 65536 1980-01-07 06:37 network_0 > > The Channel in /sys/kernel/debug/ltt/trace1 is clear. Does it cause by destroy channel command? > > The metadata_0 and interrupts_0 are empty, Is that correct? I've done some key press, the interrupts_o shouldn't be empt. > > And those files almost filled by 0 except for the short header I can't make sure, it looks like incomplete. > > I hope you can help me to solve these issues, I am appreciate you so much.. > > Best regards, > Bryan Hsieh > > SPEC, FIH Co., Ltd. > Tel:+886-3-270-0000 #2723 > Fax:+886-3-270-0001 > > > -----Original Message----- > From: Ajit R [mailto:ajitrzid@yahoo.com] > Sent: Friday, July 31, 2009 6:17 PM > To: 'ltt-dev at lists.casi.polymtl.ca'; Bryan Hsieh > Cc: Tommy Peng > Subject: Re: [ltt-dev] Some LTTng 0.43 issue on ARM > > > Hi Bryan, > My replies with the [Ajit] tag. > LTT Gods correct me if I am wrong. > > Thanks > Ajit > > --- On Fri, 7/31/09, Bryan Hsieh wrote: > > > From: Bryan Hsieh > > Subject: [ltt-dev] Some LTTng 0.43 issue on ARM > > To: "'ltt-dev at lists.casi.polymtl.ca'" > > Cc: "Tommy Peng" > > Date: Friday, July 31, 2009, 12:19 PM > > > Hi Mathieu, > > all, > > > > > > > > > > I've ported > > LTTng 0.43 on Android successfully, > > but I am wondering if the result is correct, please help me > > to verify, thank > > you very much! > > > > > > > > > > My Traced > > machine: > > > > ARM11 (ARM1136J(F)-S) / Linux > > 2.6..27 > > (Android 1.5) > > > > My > > visualization machine: > > > > X86 / Sun xVM Virtual Box (Unbuntu > > 8.10) > > > > > > > > LTTng: 0.43 > > > > LTTV: > > 0.11.2 > > > > LTT Control: 0.55 > > > > > > > > > > I > > connect to Android > > shell by ADB tool after Android bring up, then I command as > > following step, the > > command line options are different with LTTng manual, > > because the example in manual > > is for old version. > > > > > > > > > > # ltt-armall > > > [ Ajit ] > ltt-armall is a shell script. It connects + arms the markers in the > kernel. So depending on your kernel + LTT version it would write some > "text" into the /proc/ltt directory or a "1" into files in > the debugfs directory. > Make sure, the binaries for running the script are available in your target environment. For eg. awk, grep, sed etc. Busybox versions might have their limitations. If they do not work, you could do equivalent things on your Desktop and then write a simple script, doing the same on your target. > Once you run "ltt-armall", you should see some debug statements in your "dmesg" like "connect" or "arm" etc. > > > > > Connecting all > > markers > > > > Connecting high-rate > > markers to tap > > > > # lttctl > > -n trace1 -d -l /sys/kernel/debug/ltt -t > > /system/traces/trace1 > > > [Ajit] > Not sure if your "-l" option is right. you have to point it to the place where you mounted your debugfs. > > > > > Linux Trace Toolkit > > Trace Control 0.55-16102008 > > > > > > > > > > Controlling trace : > > trace1 > > > > > > > > > > Linux Trace Toolkit > > Trace Daemon 0.55-16102008 > > > > > > > > > > Reading from debugfs > > directory : /sys/kernel/debug/ltt/trace1 > > > > > > Writing to trace > > directory : /system/traces/trace1 > > > > > > > > > > It > > seems the LTT control > > and daemon work fine. I run some scenarios for 1 minute on > > Android, but the trace > > files are still empty (all files length are > > 0) > > > > What's > > wrong? Is > > there any step I missed? Or the LTT control and daemon > > don't work? > > > > > > > > > > # ls > > -l ../../sys/kernel/debug/ltt/ > > > > > > > > drwxr-xr-x > > root > > root > > 1980-01-07 01:13 trace1 > > > > # ls > > -l > > ../../sys/kernel/debug/ltt/trace1 > > > > > > -r-------- > > root > > root > > 0 > > 1980-01-07 01:13 cpu_0 > > > > [Ajit] > If you have connected and armed it right, you should see the cpu_0 > file increasing in size. After sometime, disconnect by running the > ltt-control -n whatever-name -R > and then you will see files in the control directory (network, process, etc ) grown in size. Tar and move the trace directory to your host and do the needful. > > > > drwxr-xr-x > > root > > root > > 1980-01-07 01:13 control > > > > # ls > > -l > > ../../sys/kernel/debug/ltt/trace1/control > > > > > > -r-------- > > root > > root > > 0 > > 1980-01-07 01:13 network_0 > > > > -r-------- > > root > > root > > 0 > > 1980-01-07 01:13 modules_0 > > > > -r-------- > > root > > root > > 0 > > 1980-01-07 01:13 processes_0 > > > > -r-------- > > root > > root > > 0 > > 1980-01-07 01:13 interrupts_0 > > > > -r-------- > > root > > root > > 0 1980-01-07 01:13 metadata_0 > > > > # ls -l > > .../traces/ > > > > > > > > drwxrwxrwx > > root > > root > > 1980-01-07 01:13 trace1 > > > > # ls -l > > .../traces/trace1 > > > > drwxrwxrwx > > root > > root > > 1980-01-07 01:13 control > > > > -rwxrwxrwx > > root > > root > > 0 > > 1980-01-07 01:13 cpu_0 > > > > # ls -l > > .../traces/trace1/control > > > > > > > > -rwxrwxrwx > > root > > root > > 0 > > 1980-01-07 01:13 metadata_0 > > > > -rwxrwxrwx > > root > > root > > 0 > > 1980-01-07 01:13 interrupts_0 > > > > -rwxrwxrwx > > root > > root > > 0 > > 1980-01-07 01:13 processes_0 > > > > -rwxrwxrwx > > root > > root > > 0 > > 1980-01-07 01:13 modules_0 > > > > -rwxrwxrwx > > root > > root > > 0 > > 1980-01-07 01:13 network_0 > > > > > > > > > > BTW, > > I would like to > > know what is the channel concept for LTTng? What's > > the relationship between > > trace and channel? What's the normal, flight recorder > > and hybrid mode? > > > > Which > > files should I get > > for LTTV on another visualization > > machine? > > > > So far, I > > can't find related document or the > > information isn't enough for me. > > > > > > > > > > Thanks for your > > help! > > > > > > > > > > Best > > regards, > > > > Bryan Hsieh > > > > SPEC, FIH > > Co., Ltd. > > > > Tel:+886-3-270-0000 #2723 > > > > Fax:+886-3-270-0001 > > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > ltt-dev mailing list > > ltt-dev at lists.casi.polymtl.ca > > http://lists.casi.polymtl.ca/cgi-bin/mailman/listinfo/ltt-dev > > > > > > This e-mail message together with any attachments thereto (if any) is confidential, protected under an enforceable non-disclosure agreement, > intended only for the use of the named recipient(s) above and may contain information that is privileged, belonging to professional work products > or exempt from disclosure under applicable laws. Any unauthorized review, use, copying, disclosure, or distribution of any information contained > in or attached to this transmission is STRICTLY PROHIBITED and may be against the laws. If you have received this message in error, or are not > the named recipient(s), please immediately notify the sender by e-mail or telephone at IT support hotline 886-3-2700000 EXT 1333 and delete this > e-mail message and any attached documentation from your computer. Receipt by anyone other than the intended recipient(s) is not a waiver of any > attorney-client or work product privilege. Thank you! > > > _______________________________________________ > ltt-dev mailing list > ltt-dev at lists.casi.polymtl.ca > http://lists.casi.polymtl.ca/cgi-bin/mailman/listinfo/ltt-dev > -- Mathieu Desnoyers OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F BA06 3F25 A8FE 3BAE 9A68 This e-mail message together with any attachments thereto (if any) is confidential, protected under an enforceable non-disclosure agreement, intended only for the use of the named recipient(s) above and may contain information that is privileged, belonging to professional work products or exempt from disclosure under applicable laws. Any unauthorized review, use, copying, disclosure, or distribution of any information contained in or attached to this transmission is STRICTLY PROHIBITED and may be against the laws. If you have received this message in error, or are not the named recipient(s), please immediately notify the sender by e-mail or telephone at IT support hotline 886-3-2700000 EXT 1333 and delete this e-mail message and any attached documentation from your computer. Receipt by anyone other than the intended recipient(s) is not a waiver of any attorney-client or work product privilege. Thank you! -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: busybox version and command query.txt URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: LTTng Execution Log.txt URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: LTTng Kernel Config.txt URL: From mboxrd@z Thu Jan 1 00:00:00 1970 From: bryanhsieh@fihspec.com (Bryan Hsieh) Date: Tue, 4 Aug 2009 16:28:18 +0800 Subject: [ltt-dev] Some LTTng 0.43 issue on ARM In-Reply-To: <20090731154237.GA8847@Krystal> References: <7660C7A486834C48B1487748EA32BCA3503EE618A8@spec-ty-exch.fihspec.com> <345025.22508.qm@web63207.mail.re1.yahoo.com> <7660C7A486834C48B1487748EA32BCA3503EE618AA@spec-ty-exch.fihspec.com> <20090731154237.GA8847@Krystal> Message-ID: <7660C7A486834C48B1487748EA32BCA3503EE618AF@spec-ty-exch.fihspec.com> Message-ID: <20090804082818.PKPnFQlH7e5ciYolK0pHqGttOETh8uT5buJ4V4BpbaA@z> Hi Mathieu, All Please check my comment which marked by [Bryan], Thanks! ----------------------------------------------------------------------------- [Bryan] All the related LTT functions are build-in not build as modules, Please check attachment- LTTng Kernel Config..txt, I am not sure if Kernel->user space relay support (formerly relayfs) need to build-in. Is there any item I missed? ----------------------------------------------------------------------------- [ ] Kernel->user space relay support (formerly relayfs) [*] Activate tracepoints [*] Activate markers Linux Trace Toolkit ---> <*> Compile lttng tracing probes [*] LTTng fine-grained timestamping [*] Linux Trace Toolkit Instrumentation Support <*> Linux Trace Toolkit High-speed Lockless Data Relay <*> Linux Trace Toolkit Lock-Protected Data Relay [*] Debug check for random access in ltt relay buffers <*> Linux Trace Toolkit Serializer <*> Linux Trace Toolkit Marker Control <*> Linux Trace Toolkit Tracer [*] Align Linux Trace Toolkit Traces [*] Add event size field to LTT events for tracer debugging [*] Support trace extraction from crash dump <*> Linux Trace Toolkit Netlink Controller <*> Linux Trace Toolkit State Dump ----------------------------------------------------------------------------- [Bryan] I am sure that the external utilities are build-in busybox, please check the attachment - " busybox version and command query.txt". Finally, I figure out why the ltt-armall doesn't work correctly in my target system, root cause is the output pipeline will be empty after grep twice, even I use a temp file to avoid connecting two grep commands directly, it's still empty. So I copy the /proc/ltt to my local desktop to do ltt-armall script, then generate the /proc/ltt which been filtered by following command. MARKERS=`cat /proc/ltt|grep -v %k|awk '{print $2}'|sort -u|grep -v ^core_|grep -v ^locking_|grep -v ^lockdep_` I rename it as ltt_filter and copy to my target system path /system/bin/, then modify ltt-armall by following modification and run it. MARKERS=`cat /system/bin/ltt_filter` The execution result on screen: # ltt-armall-1 Connecting all markers Connecting fs_buffer_wait_end Connecting fs_buffer_wait_start Connecting fs_close Connecting fs_exec Connecting fs_ioctl Connecting fs_llseek Connecting fs_lseek Connecting fs_open Connecting fs_pollfd Connecting fs_pread64 Connecting fs_pwrite64 Connecting fs_read Connecting fs_readv Connecting fs_select Connecting fs_write Connecting fs_writev Connecting input_event Connecting ipc_msg_create Connecting ipc_sem_create Connecting ipc_shm_create Connecting kernel_arch_ipc_call Connecting kernel_arch_kthread_create Connecting kernel_arch_syscall_entry Connecting kernel_arch_syscall_exit Connecting kernel_arch_trap_entry Connecting kernel_arch_trap_exit Connecting kernel_irq_entry Connecting kernel_irq_exit Connecting kernel_kthread_stop Connecting kernel_kthread_stop_ret Connecting kernel_module_free Connecting kernel_module_load Connecting kernel_printk Connecting kernel_process_exit Connecting kernel_process_fork Connecting kernel_process_free Connecting kernel_process_wait Connecting kernel_sched_migrate_task Connecting kernel_sched_schedule Connecting kernel_sched_try_wakeup Connecting kernel_sched_wait_task Connecting kernel_sched_wakeup_new_task Connecting kernel_send_signal Connecting kernel_softirq_entry Connecting kernel_softirq_exit Connecting kernel_softirq_raise Connecting kernel_tasklet_high_entry Connecting kernel_tasklet_high_exit Connecting kernel_tasklet_low_entry Connecting kernel_tasklet_low_exit Connecting kernel_timer_itimer_expired Connecting kernel_timer_itimer_set Connecting kernel_timer_set Connecting kernel_timer_timeout Connecting kernel_timer_update_time Connecting kernel_vprintk Connecting list_file_descriptor Connecting list_interrupt Connecting list_module Connecting list_network_ip_interface Connecting list_network_ipv4_interface Connecting list_process_state Connecting list_statedump_end Connecting list_vm_map Connecting mm_handle_fault_entry Connecting mm_handle_fault_exit Connecting mm_huge_page_alloc Connecting mm_huge_page_free Connecting mm_page_alloc Connecting mm_page_free Connecting mm_wait_on_page_end Connecting mm_wait_on_page_start Connecting rcu_classic_callback Connecting rcu_classic_call_rcu Connecting rcu_classic_call_rcu_bh Connecting statedump_softirq_vec Connecting high-rate markers to tap Connecting fs_buffer_wait_end Connecting fs_buffer_wait_start Connecting fs_close Connecting fs_exec Connecting fs_ioctl Connecting fs_llseek Connecting fs_lseek Connecting fs_open Connecting fs_pollfd Connecting fs_pread64 Connecting fs_pwrite64 Connecting fs_read Connecting fs_readv Connecting fs_select Connecting fs_write Connecting fs_writev Connecting input_event Connecting ipc_msg_create Connecting ipc_sem_create Connecting ipc_shm_create Connecting kernel_arch_ipc_call Connecting kernel_arch_kthread_create Connecting kernel_arch_syscall_entry Connecting kernel_arch_syscall_exit Connecting kernel_arch_trap_entry Connecting kernel_arch_trap_exit Connecting kernel_irq_entry Connecting kernel_irq_exit Connecting kernel_kthread_stop Connecting kernel_kthread_stop_ret Connecting kernel_module_free Connecting kernel_module_load Connecting kernel_printk Connecting kernel_process_exit Connecting kernel_process_fork Connecting kernel_process_free Connecting kernel_process_wait Connecting kernel_sched_migrate_task Connecting kernel_sched_schedule Connecting kernel_sched_try_wakeup Connecting kernel_sched_wait_task Connecting kernel_sched_wakeup_new_task Connecting kernel_send_signal Connecting kernel_softirq_entry Connecting kernel_softirq_exit Connecting kernel_softirq_raise Connecting kernel_tasklet_high_entry Connecting kernel_tasklet_high_exit Connecting kernel_tasklet_low_entry Connecting kernel_tasklet_low_exit Connecting kernel_timer_itimer_expired Connecting kernel_timer_itimer_set Connecting kernel_timer_set Connecting kernel_timer_timeout Connecting kernel_timer_update_time Connecting kernel_vprintk Connecting list_file_descriptor Connecting list_interrupt Connecting list_module Connecting list_network_ip_interface Connecting list_network_ipv4_interface Connecting list_process_state Connecting list_statedump_end Connecting list_vm_map Connecting mm_handle_fault_entry Connecting mm_handle_fault_exit Connecting mm_huge_page_alloc Connecting mm_huge_page_free Connecting mm_page_alloc Connecting mm_page_free Connecting mm_wait_on_page_end Connecting mm_wait_on_page_start Connecting rcu_classic_callback Connecting rcu_classic_call_rcu Connecting rcu_classic_call_rcu_bh Connecting statedump_softirq_vec ----------------------------------------------------------------------------- [Bryan] It's that correct result? ----------------------------------------------------------------------------- # cat /proc/ltt marker: kernel_arch_kthread_create format: "pid %ld fn %p" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_arch_syscall_exit format: " " state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_arch_syscall_entry format: "syscall_id %d ip #p%ld" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_arch_ipc_call format: "call %u first %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_arch_trap_exit format: " " state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_arch_trap_entry format: "trap_id %ld ip #p%ld" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: statedump_softirq_vec format: "id %d address %p symbol %s" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: list_module format: "name %s state %d refcount %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: core_marker_format format: "name %s format %s" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: core_marker_format format: "name %s format %s" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: list_statedump_end format: " " state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: list_process_state format: "pid %d parent_pid %d name %s type %d mode %d submode %d status %d tgid %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: list_file_descriptor format: "filename %s pid %d fd %u" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: list_vm_map format: "pid %d start %lu end %lu flags %lu pgoff %lu inode %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: list_interrupt format: "name %s action %s irq_id %u" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: list_network_ip_interface format: "name %s address #4u%lu up %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: list_network_ipv4_interface format: "name %s address #4u%lu up %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: core_marker_id format: "name %s id %hu int #1u%zu long #1u%zu pointer #1u%zu size_t #1u%zu alignment #1u%u" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: core_marker_id format: "name %s id %hu int #1u%zu long #1u%zu pointer #1u%zu size_t #1u%zu alignment #1u%u" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: core_marker_format format: "name %s format %s" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: core_marker_id format: "name %s id %hu int #1u%zu long #1u%zu pointer #1u%zu size_t #1u%zu alignment #1u%u" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: core_marker_id format: "name %s id %hu int #1u%zu long #1u%zu pointer #1u%zu size_t #1u%zu alignment #1u%u" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_module_load format: "name %s" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_module_free format: "name %s" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_vprintk format: "loglevel %c string %s ip %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_printk format: "ip %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_timer_timeout format: "pid %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_timer_update_time format: "jiffies #8u%llu xtime_sec %ld xtime_nsec %ld walltomonotonic_sec %ld walltomonotonic_nsec %ld" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_timer_set format: "expires %lu function %p data %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_timer_itimer_set format: "which %d interval_sec %ld interval_usec %ld value_sec %ld value_usec %ld" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_timer_itimer_expired format: "pid %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_process_fork format: "parent_pid %d child_pid %d child_tgid %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_process_wait format: "pid %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_process_exit format: "pid %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_process_free format: "pid %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_send_signal format: "pid %d signal %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_sched_migrate_task format: "pid %d state %ld dest_cpu %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_sched_schedule format: "prev_pid %d next_pid %d prev_state %ld" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_sched_wakeup_new_task format: "pid %d state %ld cpu_id %u" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_sched_try_wakeup format: "pid %d state %ld cpu_id %u" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_sched_wait_task format: "pid %d state %ld" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_kthread_stop_ret format: "ret %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_kthread_stop format: "pid %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_tasklet_high_exit format: "func %p data %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_tasklet_high_entry format: "func %p data %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_tasklet_low_exit format: "func %p data %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_tasklet_low_entry format: "func %p data %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_softirq_raise format: "softirq_id %u" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_softirq_exit format: "softirq_id %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_softirq_entry format: "softirq_id %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_irq_exit format: "handled #1u%u" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_irq_entry format: "irq_id %u kernel_mode %u ip %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: mm_page_alloc format: "order %u pfn %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: mm_page_free format: "order %u pfn %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: mm_handle_fault_exit format: "res %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: mm_handle_fault_entry format: "address %lu ip #p%ld write_access %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: mm_huge_page_alloc format: "pfn %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: mm_huge_page_free format: "pfn %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: mm_wait_on_page_end format: "pfn %lu bit_nr %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: mm_wait_on_page_start format: "pfn %lu bit_nr %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: fs_pollfd format: "fd %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: fs_select format: "fd %d timeout #8d%lld" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: fs_writev format: "fd %lu vlen %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: fs_readv format: "fd %lu vlen %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: fs_pwrite64 format: "fd %u count %zu pos %llu" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: fs_pread64 format: "fd %u count %zu pos %llu" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: fs_write format: "fd %u count %zu" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: fs_read format: "fd %u count %zu" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: fs_llseek format: "fd %u offset %lld origin %u" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: fs_lseek format: "fd %u offset %ld origin %u" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: fs_close format: "fd %u" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: fs_open format: "fd %d filename %s" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: fs_ioctl format: "fd %u cmd %u arg %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: fs_exec format: "filename %s" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: fs_buffer_wait_end format: "bh %p" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: fs_buffer_wait_start format: "bh %p" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: ipc_shm_create format: "id %ld flags %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: ipc_sem_create format: "id %ld flags %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: ipc_msg_create format: "id %ld flags %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: lockdep_lock_release format: "retaddr 0x%lX lock %p nested %d" state: 0 call: 0xc0093374 probe single : 0xc00920ec marker: lockdep_lock_acquire format: "retaddr 0x%lX subclass %u lock %p trylock %d read %d hardirqs_off %d" state: 0 call: 0xc0093374 probe single : 0xc00920ec marker: lockdep_softirqs_off format: "retaddr 0x%lX" state: 0 call: 0xc0093374 probe single : 0xc00920ec marker: lockdep_softirqs_on format: "retaddr 0x%lX" state: 0 call: 0xc0093374 probe single : 0xc00920ec marker: lockdep_hardirqs_off format: "retaddr 0x%lX" state: 0 call: 0xc0093374 probe single : 0xc00920ec marker: lockdep_hardirqs_on format: "retaddr 0x%lX" state: 0 call: 0xc0093374 probe single : 0xc00920ec marker: rcu_classic_call_rcu_bh format: "func %p ip 0x%lX" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: rcu_classic_call_rcu format: "func %p ip 0x%lX" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: rcu_classic_callback format: "func %p" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: input_event format: "type %u code %u value %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 ----------------------------------------------------------------------------- [Bryan] But the /proc/ltt is still same, it's that correct? What's the expected result? ----------------------------------------------------------------------------- # df /dev: 39564K total, 0K used, 39564K available (block size 4096) /sqlite_stmt_journals: 4096K total, 0K used, 4096K available (block size 4096) /system: 72704K total, 71680K used, 1024K available (block size 4096) /data: 66816K total, 37812K used, 29004K available (block size 4096) /cache: 67584K total, 1156K used, 66428K available (block size 4096) /sdcard: 991708K total, 147040K used, 844668K available (block size 4096) ----------------------------------------------------------------------------- [Bryan] There is 29004K available for trace log file /data/ltt/trace1. Is that enough? ----------------------------------------------------------------------------- # cat /proc/meminfo MemTotal: 79128 kB MemFree: 2440 kB Buffers: 36 kB Cached: 29872 kB SwapCached: 0 kB Active: 61384 kB Inactive: 5052 kB SwapTotal: 0 kB SwapFree: 0 kB Dirty: 0 kB Writeback: 0 kB AnonPages: 36552 kB Mapped: 24888 kB Slab: 3856 kB SReclaimable: 904 kB SUnreclaim: 2952 kB PageTables: 3660 kB NFS_Unstable: 0 kB Bounce: 0 kB WritebackTmp: 0 kB CommitLimit: 39564 kB Committed_AS: 1032216 kB VmallocTotal: 409600 kB VmallocUsed: 52908 kB VmallocChunk: 332796 kB ----------------------------------------------------------------------------- [Bryan] There is 2440 kB free memory. Is that enough? ----------------------------------------------------------------------------- # lttctl -n trace1 -d -l /sys/kernel/debug/ltt -t /data/ltt/trace1 Linux Trace Toolkit Trace Control 0.55-16102008 Controlling trace : trace1 Linux Trace Toolkit Trace Daemon 0.55-16102008 Reading from debugfs directory : /sys/kernel/debug/ltt/trace1 Writing to trace directory : /data/ltt/trace1 ----------------------------------------------------------------------------- [Bryan] do some tests on target device, then check the channel and trace ----------------------------------------------------------------------------- # ls -l /sys/kernel/debug/ltt/trace1 -r-------- root root 0 1980-01-06 00:10 cpu_0 drwxr-xr-x root root 1980-01-06 00:10 control # ls -l /sys/kernel/debug/ltt/trace1/control -r-------- root root 0 1980-01-06 00:10 network_0 -r-------- root root 0 1980-01-06 00:10 modules_0 -r-------- root root 0 1980-01-06 00:10 processes_0 -r-------- root root 0 1980-01-06 00:10 interrupts_0 -r-------- root root 0 1980-01-06 00:10 metadata_0 # ls -l /data/ltt/trace1 drwxrwxrwx root root 1980-01-06 00:10 control -rwxrwxrwx root root 29949952 1980-01-06 00:10 cpu_0 # ls -l /data/ltt/trace1/control -rwxrwxrwx root root 0 1980-01-06 00:10 metadata_0 -rwxrwxrwx root root 0 1980-01-06 00:10 interrupts_0 -rwxrwxrwx root root 0 1980-01-06 00:10 processes_0 -rwxrwxrwx root root 0 1980-01-06 00:10 modules_0 -rwxrwxrwx root root 0 1980-01-06 00:10 network_0 ----------------------------------------------------------------------------- [Bryan] It seems that the capacity has been exhausted by cpu_0, because 29004K available. It's that correct that cpu_0 will grow to so huge (29M)? ----------------------------------------------------------------------------- # lttctl -n trace1 -d -R Linux Trace Toolkit Trace Control 0.55-16102008 Controlling trace : trace1 ^Z[1] + Stopped lttctl -n trace1 -d -R ----------------------------------------------------------------------------- [Bryan] hang in here...I have to break it by Ctrl+Z, What's problem? ----------------------------------------------------------------------------- # ls -l /data/ltt/trace1/ drwxrwxrwx root root 1980-01-06 00:10 control -rwxrwxrwx root root 29949952 1980-01-06 00:10 cpu_0 # ls -l /data/ltt/trace1/control -rwxrwxrwx root root 0 1980-01-06 00:10 metadata_0 -rwxrwxrwx root root 0 1980-01-06 00:10 interrupts_0 -rwxrwxrwx root root 0 1980-01-06 00:10 processes_0 -rwxrwxrwx root root 0 1980-01-06 00:10 modules_0 -rwxrwxrwx root root 0 1980-01-06 00:10 network_0 # ls -l /sys/kernel/debug/ltt/trace1 -r-------- root root 0 1980-01-06 00:10 cpu_0 drwxr-xr-x root root 1980-01-06 00:10 control # ls -l /sys/kernel/debug/ltt/trace1/control -r-------- root root 0 1980-01-06 00:10 network_0 -r-------- root root 0 1980-01-06 00:10 modules_0 -r-------- root root 0 1980-01-06 00:10 processes_0 -r-------- root root 0 1980-01-06 00:10 interrupts_0 -r-------- root root 0 1980-01-06 00:10 metadata_0 # df /dev: 39564K total, 0K used, 39564K available (block size 4096) /sqlite_stmt_journals: 4096K total, 0K used, 4096K available (block size 4096) /system: 72704K total, 71680K used, 1024K available (block size 4096) /data: 66816K total, 66732K used, 84K available (block size 4096) /cache: 67584K total, 1156K used, 66428K available (block size 4096) /sdcard: 991708K total, 147040K used, 844668K available (block size 4096) ----------------------------------------------------------------------------- [Bryan] I think the trace is incomplete. ----------------------------------------------------------------------------- Please help me! Thank you very much! Best regards, Bryan Hsieh SPEC, FIH Co., Ltd. Tel:+886-3-270-0000 #2723 Fax:+886-3-270-0001 -----Original Message----- From: Mathieu Desnoyers [mailto:compudj@krystal.dyndns.org] Sent: Friday, July 31, 2009 11:43 PM To: Bryan Hsieh Cc: 'Ajit R'; 'ltt-dev at lists.casi.polymtl.ca'; Tommy Peng Subject: Re: [ltt-dev] Some LTTng 0.43 issue on ARM * Bryan Hsieh (bryanhsieh at fihspec.com) wrote: > Hi Ajit, > > I appreciate your reply! > But there is a problem here. > > My installation of Busybox is 1.12.4. It sounds like either awk or sed or not built in your busybox installation. Can you provide the output of cat /proc/ltt on your target system ? Check the content of ltt-armall to see which external programs it needs. Also, if you built the lttng probes as modules, you may need to load them explicitely. Mathieu > > I follow your suggestion to test on x86 Desktop ARM simulator, the message said that can't find /proc/ltt. I think it should be, because I didn't patch to ARM simulator Linux kernel. > > [sbox-ARM: ~/ltt-control-0.55-16102008/lttctl] > ltt-armall > Connecting all markers > cat: /proc/ltt: No such file or directory > Connecting high-rate markers to tap > cat: /proc/ltt: No such file or directory > > It should be work on my target device, but it seems not. > I can't find /ltt under /proc > Maybe it's root cause, but why there is no /ltt/proc? And why there is no " cat: /proc/ltt: No such file or directory" message in my target device during running ltt-armall? > > BTW, I mount debugfs in init.c > Mount("debugfs", "/sys/kernel/debug", "debugfs", 0, NULL); > It should not be a problem. > > -----------------------------------------------------------------------------------------------------------Another problem: > I command as following steps. > # lttctl -n trace1 -q > Linux Trace Toolkit Trace Control 0.55-16102008 > Controlling trace : trace1 > > # lttctl -n trace1 -r > Linux Trace Toolkit Trace Control 0.55-16102008 > Controlling trace : trace1 > > >>> Android shell hang in here, I have to press Ctrl+Z to force stopping - lttctl -n trace1 -r .Why? Is anything wrong? > # ls -l ../tmp/trace1 > drwxrwxrwx root root 1980-01-07 06:37 control > -rwxrwxrwx root root 1048576 1980-01-07 06:37 cpu_0 > # ls -l ../tmp/trace1/control > -rwxrwxrwx root root 0 1980-01-07 06:37 metadata_0 > -rwxrwxrwx root root 0 1980-01-07 06:37 interrupts_0 > -rwxrwxrwx root root 10240 1980-01-07 06:37 processes_0 > -rwxrwxrwx root root 65536 1980-01-07 06:37 modules_0 > -rwxrwxrwx root root 65536 1980-01-07 06:37 network_0 > > The Channel in /sys/kernel/debug/ltt/trace1 is clear. Does it cause by destroy channel command? > > The metadata_0 and interrupts_0 are empty, Is that correct? I've done some key press, the interrupts_o shouldn't be empt. > > And those files almost filled by 0 except for the short header I can't make sure, it looks like incomplete. > > I hope you can help me to solve these issues, I am appreciate you so much.. > > Best regards, > Bryan Hsieh > > SPEC, FIH Co., Ltd. > Tel:+886-3-270-0000 #2723 > Fax:+886-3-270-0001 > > > -----Original Message----- > From: Ajit R [mailto:ajitrzid@yahoo.com] > Sent: Friday, July 31, 2009 6:17 PM > To: 'ltt-dev at lists.casi.polymtl.ca'; Bryan Hsieh > Cc: Tommy Peng > Subject: Re: [ltt-dev] Some LTTng 0.43 issue on ARM > > > Hi Bryan, > My replies with the [Ajit] tag. > LTT Gods correct me if I am wrong. > > Thanks > Ajit > > --- On Fri, 7/31/09, Bryan Hsieh wrote: > > > From: Bryan Hsieh > > Subject: [ltt-dev] Some LTTng 0.43 issue on ARM > > To: "'ltt-dev at lists.casi.polymtl.ca'" > > Cc: "Tommy Peng" > > Date: Friday, July 31, 2009, 12:19 PM > > > Hi Mathieu, > > all, > > > > > > > > > > I've ported > > LTTng 0.43 on Android successfully, > > but I am wondering if the result is correct, please help me > > to verify, thank > > you very much! > > > > > > > > > > My Traced > > machine: > > > > ARM11 (ARM1136J(F)-S) / Linux > > 2.6..27 > > (Android 1.5) > > > > My > > visualization machine: > > > > X86 / Sun xVM Virtual Box (Unbuntu > > 8.10) > > > > > > > > LTTng: 0.43 > > > > LTTV: > > 0.11.2 > > > > LTT Control: 0.55 > > > > > > > > > > I > > connect to Android > > shell by ADB tool after Android bring up, then I command as > > following step, the > > command line options are different with LTTng manual, > > because the example in manual > > is for old version. > > > > > > > > > > # ltt-armall > > > [ Ajit ] > ltt-armall is a shell script. It connects + arms the markers in the > kernel. So depending on your kernel + LTT version it would write some > "text" into the /proc/ltt directory or a "1" into files in > the debugfs directory. > Make sure, the binaries for running the script are available in your target environment. For eg. awk, grep, sed etc. Busybox versions might have their limitations. If they do not work, you could do equivalent things on your Desktop and then write a simple script, doing the same on your target. > Once you run "ltt-armall", you should see some debug statements in your "dmesg" like "connect" or "arm" etc. > > > > > Connecting all > > markers > > > > Connecting high-rate > > markers to tap > > > > # lttctl > > -n trace1 -d -l /sys/kernel/debug/ltt -t > > /system/traces/trace1 > > > [Ajit] > Not sure if your "-l" option is right. you have to point it to the place where you mounted your debugfs. > > > > > Linux Trace Toolkit > > Trace Control 0.55-16102008 > > > > > > > > > > Controlling trace : > > trace1 > > > > > > > > > > Linux Trace Toolkit > > Trace Daemon 0.55-16102008 > > > > > > > > > > Reading from debugfs > > directory : /sys/kernel/debug/ltt/trace1 > > > > > > Writing to trace > > directory : /system/traces/trace1 > > > > > > > > > > It > > seems the LTT control > > and daemon work fine. I run some scenarios for 1 minute on > > Android, but the trace > > files are still empty (all files length are > > 0) > > > > What's > > wrong? Is > > there any step I missed? Or the LTT control and daemon > > don't work? > > > > > > > > > > # ls > > -l ../../sys/kernel/debug/ltt/ > > > > > > > > drwxr-xr-x > > root > > root > > 1980-01-07 01:13 trace1 > > > > # ls > > -l > > ../../sys/kernel/debug/ltt/trace1 > > > > > > -r-------- > > root > > root > > 0 > > 1980-01-07 01:13 cpu_0 > > > > [Ajit] > If you have connected and armed it right, you should see the cpu_0 > file increasing in size. After sometime, disconnect by running the > ltt-control -n whatever-name -R > and then you will see files in the control directory (network, process, etc ) grown in size. Tar and move the trace directory to your host and do the needful. > > > > drwxr-xr-x > > root > > root > > 1980-01-07 01:13 control > > > > # ls > > -l > > ../../sys/kernel/debug/ltt/trace1/control > > > > > > -r-------- > > root > > root > > 0 > > 1980-01-07 01:13 network_0 > > > > -r-------- > > root > > root > > 0 > > 1980-01-07 01:13 modules_0 > > > > -r-------- > > root > > root > > 0 > > 1980-01-07 01:13 processes_0 > > > > -r-------- > > root > > root > > 0 > > 1980-01-07 01:13 interrupts_0 > > > > -r-------- > > root > > root > > 0 1980-01-07 01:13 metadata_0 > > > > # ls -l > > .../traces/ > > > > > > > > drwxrwxrwx > > root > > root > > 1980-01-07 01:13 trace1 > > > > # ls -l > > .../traces/trace1 > > > > drwxrwxrwx > > root > > root > > 1980-01-07 01:13 control > > > > -rwxrwxrwx > > root > > root > > 0 > > 1980-01-07 01:13 cpu_0 > > > > # ls -l > > .../traces/trace1/control > > > > > > > > -rwxrwxrwx > > root > > root > > 0 > > 1980-01-07 01:13 metadata_0 > > > > -rwxrwxrwx > > root > > root > > 0 > > 1980-01-07 01:13 interrupts_0 > > > > -rwxrwxrwx > > root > > root > > 0 > > 1980-01-07 01:13 processes_0 > > > > -rwxrwxrwx > > root > > root > > 0 > > 1980-01-07 01:13 modules_0 > > > > -rwxrwxrwx > > root > > root > > 0 > > 1980-01-07 01:13 network_0 > > > > > > > > > > BTW, > > I would like to > > know what is the channel concept for LTTng? What's > > the relationship between > > trace and channel? What's the normal, flight recorder > > and hybrid mode? > > > > Which > > files should I get > > for LTTV on another visualization > > machine? > > > > So far, I > > can't find related document or the > > information isn't enough for me. > > > > > > > > > > Thanks for your > > help! > > > > > > > > > > Best > > regards, > > > > Bryan Hsieh > > > > SPEC, FIH > > Co., Ltd. > > > > Tel:+886-3-270-0000 #2723 > > > > Fax:+886-3-270-0001 > > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > ltt-dev mailing list > > ltt-dev at lists.casi.polymtl.ca > > http://lists.casi.polymtl.ca/cgi-bin/mailman/listinfo/ltt-dev > > > > > > This e-mail message together with any attachments thereto (if any) is confidential, protected under an enforceable non-disclosure agreement, > intended only for the use of the named recipient(s) above and may contain information that is privileged, belonging to professional work products > or exempt from disclosure under applicable laws. Any unauthorized review, use, copying, disclosure, or distribution of any information contained > in or attached to this transmission is STRICTLY PROHIBITED and may be against the laws. If you have received this message in error, or are not > the named recipient(s), please immediately notify the sender by e-mail or telephone at IT support hotline 886-3-2700000 EXT 1333 and delete this > e-mail message and any attached documentation from your computer. Receipt by anyone other than the intended recipient(s) is not a waiver of any > attorney-client or work product privilege. Thank you! > > > _______________________________________________ > ltt-dev mailing list > ltt-dev at lists.casi.polymtl.ca > http://lists.casi.polymtl.ca/cgi-bin/mailman/listinfo/ltt-dev > -- Mathieu Desnoyers OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F BA06 3F25 A8FE 3BAE 9A68 This e-mail message together with any attachments thereto (if any) is confidential, protected under an enforceable non-disclosure agreement, intended only for the use of the named recipient(s) above and may contain information that is privileged, belonging to professional work products or exempt from disclosure under applicable laws. Any unauthorized review, use, copying, disclosure, or distribution of any information contained in or attached to this transmission is STRICTLY PROHIBITED and may be against the laws. If you have received this message in error, or are not the named recipient(s), please immediately notify the sender by e-mail or telephone at IT support hotline 886-3-2700000 EXT 1333 and delete this e-mail message and any attached documentation from your computer. Receipt by anyone other than the intended recipient(s) is not a waiver of any attorney-client or work product privilege. Thank you! -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: busybox version and command query.txt URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: LTTng Execution Log.txt URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: LTTng Kernel Config.txt URL: