* [ltt-dev] Some LTTng 0.43 issue on ARM [not found] ` <20090731154237.GA8847@Krystal> @ 2009-08-04 8:28 ` Bryan Hsieh 2009-08-04 8:28 ` Bryan Hsieh ` (2 more replies) 0 siblings, 3 replies; 9+ messages in thread From: Bryan Hsieh @ 2009-08-04 8:28 UTC (permalink / raw) Hi Mathieu, All Please check my comment which marked by [Bryan], Thanks! ----------------------------------------------------------------------------- [Bryan] All the related LTT functions are build-in not build as modules, Please check attachment- LTTng Kernel Config..txt, I am not sure if Kernel->user space relay support (formerly relayfs) need to build-in. Is there any item I missed? ----------------------------------------------------------------------------- [ ] Kernel->user space relay support (formerly relayfs) [*] Activate tracepoints [*] Activate markers Linux Trace Toolkit ---> <*> Compile lttng tracing probes [*] LTTng fine-grained timestamping [*] Linux Trace Toolkit Instrumentation Support <*> Linux Trace Toolkit High-speed Lockless Data Relay <*> Linux Trace Toolkit Lock-Protected Data Relay [*] Debug check for random access in ltt relay buffers <*> Linux Trace Toolkit Serializer <*> Linux Trace Toolkit Marker Control <*> Linux Trace Toolkit Tracer [*] Align Linux Trace Toolkit Traces [*] Add event size field to LTT events for tracer debugging [*] Support trace extraction from crash dump <*> Linux Trace Toolkit Netlink Controller <*> Linux Trace Toolkit State Dump ----------------------------------------------------------------------------- [Bryan] I am sure that the external utilities are build-in busybox, please check the attachment - " busybox version and command query.txt". Finally, I figure out why the ltt-armall doesn't work correctly in my target system, root cause is the output pipeline will be empty after grep twice, even I use a temp file to avoid connecting two grep commands directly, it's still empty. So I copy the /proc/ltt to my local desktop to do ltt-armall script, then generate the /proc/ltt which been filtered by following command. MARKERS=`cat /proc/ltt|grep -v %k|awk '{print $2}'|sort -u|grep -v ^core_|grep -v ^locking_|grep -v ^lockdep_` I rename it as ltt_filter and copy to my target system path /system/bin/, then modify ltt-armall by following modification and run it. MARKERS=`cat /system/bin/ltt_filter` The execution result on screen: # ltt-armall-1 Connecting all markers Connecting fs_buffer_wait_end Connecting fs_buffer_wait_start Connecting fs_close Connecting fs_exec Connecting fs_ioctl Connecting fs_llseek Connecting fs_lseek Connecting fs_open Connecting fs_pollfd Connecting fs_pread64 Connecting fs_pwrite64 Connecting fs_read Connecting fs_readv Connecting fs_select Connecting fs_write Connecting fs_writev Connecting input_event Connecting ipc_msg_create Connecting ipc_sem_create Connecting ipc_shm_create Connecting kernel_arch_ipc_call Connecting kernel_arch_kthread_create Connecting kernel_arch_syscall_entry Connecting kernel_arch_syscall_exit Connecting kernel_arch_trap_entry Connecting kernel_arch_trap_exit Connecting kernel_irq_entry Connecting kernel_irq_exit Connecting kernel_kthread_stop Connecting kernel_kthread_stop_ret Connecting kernel_module_free Connecting kernel_module_load Connecting kernel_printk Connecting kernel_process_exit Connecting kernel_process_fork Connecting kernel_process_free Connecting kernel_process_wait Connecting kernel_sched_migrate_task Connecting kernel_sched_schedule Connecting kernel_sched_try_wakeup Connecting kernel_sched_wait_task Connecting kernel_sched_wakeup_new_task Connecting kernel_send_signal Connecting kernel_softirq_entry Connecting kernel_softirq_exit Connecting kernel_softirq_raise Connecting kernel_tasklet_high_entry Connecting kernel_tasklet_high_exit Connecting kernel_tasklet_low_entry Connecting kernel_tasklet_low_exit Connecting kernel_timer_itimer_expired Connecting kernel_timer_itimer_set Connecting kernel_timer_set Connecting kernel_timer_timeout Connecting kernel_timer_update_time Connecting kernel_vprintk Connecting list_file_descriptor Connecting list_interrupt Connecting list_module Connecting list_network_ip_interface Connecting list_network_ipv4_interface Connecting list_process_state Connecting list_statedump_end Connecting list_vm_map Connecting mm_handle_fault_entry Connecting mm_handle_fault_exit Connecting mm_huge_page_alloc Connecting mm_huge_page_free Connecting mm_page_alloc Connecting mm_page_free Connecting mm_wait_on_page_end Connecting mm_wait_on_page_start Connecting rcu_classic_callback Connecting rcu_classic_call_rcu Connecting rcu_classic_call_rcu_bh Connecting statedump_softirq_vec Connecting high-rate markers to tap Connecting fs_buffer_wait_end Connecting fs_buffer_wait_start Connecting fs_close Connecting fs_exec Connecting fs_ioctl Connecting fs_llseek Connecting fs_lseek Connecting fs_open Connecting fs_pollfd Connecting fs_pread64 Connecting fs_pwrite64 Connecting fs_read Connecting fs_readv Connecting fs_select Connecting fs_write Connecting fs_writev Connecting input_event Connecting ipc_msg_create Connecting ipc_sem_create Connecting ipc_shm_create Connecting kernel_arch_ipc_call Connecting kernel_arch_kthread_create Connecting kernel_arch_syscall_entry Connecting kernel_arch_syscall_exit Connecting kernel_arch_trap_entry Connecting kernel_arch_trap_exit Connecting kernel_irq_entry Connecting kernel_irq_exit Connecting kernel_kthread_stop Connecting kernel_kthread_stop_ret Connecting kernel_module_free Connecting kernel_module_load Connecting kernel_printk Connecting kernel_process_exit Connecting kernel_process_fork Connecting kernel_process_free Connecting kernel_process_wait Connecting kernel_sched_migrate_task Connecting kernel_sched_schedule Connecting kernel_sched_try_wakeup Connecting kernel_sched_wait_task Connecting kernel_sched_wakeup_new_task Connecting kernel_send_signal Connecting kernel_softirq_entry Connecting kernel_softirq_exit Connecting kernel_softirq_raise Connecting kernel_tasklet_high_entry Connecting kernel_tasklet_high_exit Connecting kernel_tasklet_low_entry Connecting kernel_tasklet_low_exit Connecting kernel_timer_itimer_expired Connecting kernel_timer_itimer_set Connecting kernel_timer_set Connecting kernel_timer_timeout Connecting kernel_timer_update_time Connecting kernel_vprintk Connecting list_file_descriptor Connecting list_interrupt Connecting list_module Connecting list_network_ip_interface Connecting list_network_ipv4_interface Connecting list_process_state Connecting list_statedump_end Connecting list_vm_map Connecting mm_handle_fault_entry Connecting mm_handle_fault_exit Connecting mm_huge_page_alloc Connecting mm_huge_page_free Connecting mm_page_alloc Connecting mm_page_free Connecting mm_wait_on_page_end Connecting mm_wait_on_page_start Connecting rcu_classic_callback Connecting rcu_classic_call_rcu Connecting rcu_classic_call_rcu_bh Connecting statedump_softirq_vec ----------------------------------------------------------------------------- [Bryan] It's that correct result? ----------------------------------------------------------------------------- # cat /proc/ltt marker: kernel_arch_kthread_create format: "pid %ld fn %p" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_arch_syscall_exit format: " " state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_arch_syscall_entry format: "syscall_id %d ip #p%ld" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_arch_ipc_call format: "call %u first %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_arch_trap_exit format: " " state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_arch_trap_entry format: "trap_id %ld ip #p%ld" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: statedump_softirq_vec format: "id %d address %p symbol %s" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: list_module format: "name %s state %d refcount %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: core_marker_format format: "name %s format %s" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: core_marker_format format: "name %s format %s" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: list_statedump_end format: " " state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: list_process_state format: "pid %d parent_pid %d name %s type %d mode %d submode %d status %d tgid %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: list_file_descriptor format: "filename %s pid %d fd %u" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: list_vm_map format: "pid %d start %lu end %lu flags %lu pgoff %lu inode %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: list_interrupt format: "name %s action %s irq_id %u" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: list_network_ip_interface format: "name %s address #4u%lu up %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: list_network_ipv4_interface format: "name %s address #4u%lu up %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: core_marker_id format: "name %s id %hu int #1u%zu long #1u%zu pointer #1u%zu size_t #1u%zu alignment #1u%u" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: core_marker_id format: "name %s id %hu int #1u%zu long #1u%zu pointer #1u%zu size_t #1u%zu alignment #1u%u" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: core_marker_format format: "name %s format %s" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: core_marker_id format: "name %s id %hu int #1u%zu long #1u%zu pointer #1u%zu size_t #1u%zu alignment #1u%u" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: core_marker_id format: "name %s id %hu int #1u%zu long #1u%zu pointer #1u%zu size_t #1u%zu alignment #1u%u" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_module_load format: "name %s" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_module_free format: "name %s" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_vprintk format: "loglevel %c string %s ip %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_printk format: "ip %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_timer_timeout format: "pid %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_timer_update_time format: "jiffies #8u%llu xtime_sec %ld xtime_nsec %ld walltomonotonic_sec %ld walltomonotonic_nsec %ld" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_timer_set format: "expires %lu function %p data %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_timer_itimer_set format: "which %d interval_sec %ld interval_usec %ld value_sec %ld value_usec %ld" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_timer_itimer_expired format: "pid %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_process_fork format: "parent_pid %d child_pid %d child_tgid %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_process_wait format: "pid %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_process_exit format: "pid %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_process_free format: "pid %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_send_signal format: "pid %d signal %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_sched_migrate_task format: "pid %d state %ld dest_cpu %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_sched_schedule format: "prev_pid %d next_pid %d prev_state %ld" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_sched_wakeup_new_task format: "pid %d state %ld cpu_id %u" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_sched_try_wakeup format: "pid %d state %ld cpu_id %u" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_sched_wait_task format: "pid %d state %ld" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_kthread_stop_ret format: "ret %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_kthread_stop format: "pid %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_tasklet_high_exit format: "func %p data %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_tasklet_high_entry format: "func %p data %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_tasklet_low_exit format: "func %p data %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_tasklet_low_entry format: "func %p data %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_softirq_raise format: "softirq_id %u" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_softirq_exit format: "softirq_id %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_softirq_entry format: "softirq_id %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_irq_exit format: "handled #1u%u" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_irq_entry format: "irq_id %u kernel_mode %u ip %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: mm_page_alloc format: "order %u pfn %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: mm_page_free format: "order %u pfn %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: mm_handle_fault_exit format: "res %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: mm_handle_fault_entry format: "address %lu ip #p%ld write_access %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: mm_huge_page_alloc format: "pfn %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: mm_huge_page_free format: "pfn %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: mm_wait_on_page_end format: "pfn %lu bit_nr %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: mm_wait_on_page_start format: "pfn %lu bit_nr %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: fs_pollfd format: "fd %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: fs_select format: "fd %d timeout #8d%lld" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: fs_writev format: "fd %lu vlen %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: fs_readv format: "fd %lu vlen %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: fs_pwrite64 format: "fd %u count %zu pos %llu" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: fs_pread64 format: "fd %u count %zu pos %llu" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: fs_write format: "fd %u count %zu" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: fs_read format: "fd %u count %zu" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: fs_llseek format: "fd %u offset %lld origin %u" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: fs_lseek format: "fd %u offset %ld origin %u" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: fs_close format: "fd %u" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: fs_open format: "fd %d filename %s" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: fs_ioctl format: "fd %u cmd %u arg %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: fs_exec format: "filename %s" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: fs_buffer_wait_end format: "bh %p" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: fs_buffer_wait_start format: "bh %p" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: ipc_shm_create format: "id %ld flags %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: ipc_sem_create format: "id %ld flags %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: ipc_msg_create format: "id %ld flags %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: lockdep_lock_release format: "retaddr 0x%lX lock %p nested %d" state: 0 call: 0xc0093374 probe single : 0xc00920ec marker: lockdep_lock_acquire format: "retaddr 0x%lX subclass %u lock %p trylock %d read %d hardirqs_off %d" state: 0 call: 0xc0093374 probe single : 0xc00920ec marker: lockdep_softirqs_off format: "retaddr 0x%lX" state: 0 call: 0xc0093374 probe single : 0xc00920ec marker: lockdep_softirqs_on format: "retaddr 0x%lX" state: 0 call: 0xc0093374 probe single : 0xc00920ec marker: lockdep_hardirqs_off format: "retaddr 0x%lX" state: 0 call: 0xc0093374 probe single : 0xc00920ec marker: lockdep_hardirqs_on format: "retaddr 0x%lX" state: 0 call: 0xc0093374 probe single : 0xc00920ec marker: rcu_classic_call_rcu_bh format: "func %p ip 0x%lX" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: rcu_classic_call_rcu format: "func %p ip 0x%lX" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: rcu_classic_callback format: "func %p" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: input_event format: "type %u code %u value %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 ----------------------------------------------------------------------------- [Bryan] But the /proc/ltt is still same, it's that correct? What's the expected result? ----------------------------------------------------------------------------- # df /dev: 39564K total, 0K used, 39564K available (block size 4096) /sqlite_stmt_journals: 4096K total, 0K used, 4096K available (block size 4096) /system: 72704K total, 71680K used, 1024K available (block size 4096) /data: 66816K total, 37812K used, 29004K available (block size 4096) /cache: 67584K total, 1156K used, 66428K available (block size 4096) /sdcard: 991708K total, 147040K used, 844668K available (block size 4096) ----------------------------------------------------------------------------- [Bryan] There is 29004K available for trace log file /data/ltt/trace1. Is that enough? ----------------------------------------------------------------------------- # cat /proc/meminfo MemTotal: 79128 kB MemFree: 2440 kB Buffers: 36 kB Cached: 29872 kB SwapCached: 0 kB Active: 61384 kB Inactive: 5052 kB SwapTotal: 0 kB SwapFree: 0 kB Dirty: 0 kB Writeback: 0 kB AnonPages: 36552 kB Mapped: 24888 kB Slab: 3856 kB SReclaimable: 904 kB SUnreclaim: 2952 kB PageTables: 3660 kB NFS_Unstable: 0 kB Bounce: 0 kB WritebackTmp: 0 kB CommitLimit: 39564 kB Committed_AS: 1032216 kB VmallocTotal: 409600 kB VmallocUsed: 52908 kB VmallocChunk: 332796 kB ----------------------------------------------------------------------------- [Bryan] There is 2440 kB free memory. Is that enough? ----------------------------------------------------------------------------- # lttctl -n trace1 -d -l /sys/kernel/debug/ltt -t /data/ltt/trace1 Linux Trace Toolkit Trace Control 0.55-16102008 Controlling trace : trace1 Linux Trace Toolkit Trace Daemon 0.55-16102008 Reading from debugfs directory : /sys/kernel/debug/ltt/trace1 Writing to trace directory : /data/ltt/trace1 ----------------------------------------------------------------------------- [Bryan] do some tests on target device, then check the channel and trace ----------------------------------------------------------------------------- # ls -l /sys/kernel/debug/ltt/trace1 -r-------- root root 0 1980-01-06 00:10 cpu_0 drwxr-xr-x root root 1980-01-06 00:10 control # ls -l /sys/kernel/debug/ltt/trace1/control -r-------- root root 0 1980-01-06 00:10 network_0 -r-------- root root 0 1980-01-06 00:10 modules_0 -r-------- root root 0 1980-01-06 00:10 processes_0 -r-------- root root 0 1980-01-06 00:10 interrupts_0 -r-------- root root 0 1980-01-06 00:10 metadata_0 # ls -l /data/ltt/trace1 drwxrwxrwx root root 1980-01-06 00:10 control -rwxrwxrwx root root 29949952 1980-01-06 00:10 cpu_0 # ls -l /data/ltt/trace1/control -rwxrwxrwx root root 0 1980-01-06 00:10 metadata_0 -rwxrwxrwx root root 0 1980-01-06 00:10 interrupts_0 -rwxrwxrwx root root 0 1980-01-06 00:10 processes_0 -rwxrwxrwx root root 0 1980-01-06 00:10 modules_0 -rwxrwxrwx root root 0 1980-01-06 00:10 network_0 ----------------------------------------------------------------------------- [Bryan] It seems that the capacity has been exhausted by cpu_0, because 29004K available. It's that correct that cpu_0 will grow to so huge (29M)? ----------------------------------------------------------------------------- # lttctl -n trace1 -d -R Linux Trace Toolkit Trace Control 0.55-16102008 Controlling trace : trace1 ^Z[1] + Stopped lttctl -n trace1 -d -R ----------------------------------------------------------------------------- [Bryan] hang in here...I have to break it by Ctrl+Z, What's problem? ----------------------------------------------------------------------------- # ls -l /data/ltt/trace1/ drwxrwxrwx root root 1980-01-06 00:10 control -rwxrwxrwx root root 29949952 1980-01-06 00:10 cpu_0 # ls -l /data/ltt/trace1/control -rwxrwxrwx root root 0 1980-01-06 00:10 metadata_0 -rwxrwxrwx root root 0 1980-01-06 00:10 interrupts_0 -rwxrwxrwx root root 0 1980-01-06 00:10 processes_0 -rwxrwxrwx root root 0 1980-01-06 00:10 modules_0 -rwxrwxrwx root root 0 1980-01-06 00:10 network_0 # ls -l /sys/kernel/debug/ltt/trace1 -r-------- root root 0 1980-01-06 00:10 cpu_0 drwxr-xr-x root root 1980-01-06 00:10 control # ls -l /sys/kernel/debug/ltt/trace1/control -r-------- root root 0 1980-01-06 00:10 network_0 -r-------- root root 0 1980-01-06 00:10 modules_0 -r-------- root root 0 1980-01-06 00:10 processes_0 -r-------- root root 0 1980-01-06 00:10 interrupts_0 -r-------- root root 0 1980-01-06 00:10 metadata_0 # df /dev: 39564K total, 0K used, 39564K available (block size 4096) /sqlite_stmt_journals: 4096K total, 0K used, 4096K available (block size 4096) /system: 72704K total, 71680K used, 1024K available (block size 4096) /data: 66816K total, 66732K used, 84K available (block size 4096) /cache: 67584K total, 1156K used, 66428K available (block size 4096) /sdcard: 991708K total, 147040K used, 844668K available (block size 4096) ----------------------------------------------------------------------------- [Bryan] I think the trace is incomplete. ----------------------------------------------------------------------------- Please help me! Thank you very much! Best regards, Bryan Hsieh SPEC, FIH Co., Ltd. Tel:+886-3-270-0000 #2723 Fax:+886-3-270-0001 -----Original Message----- From: Mathieu Desnoyers [mailto:compudj@krystal.dyndns.org] Sent: Friday, July 31, 2009 11:43 PM To: Bryan Hsieh Cc: 'Ajit R'; 'ltt-dev at lists.casi.polymtl.ca'; Tommy Peng Subject: Re: [ltt-dev] Some LTTng 0.43 issue on ARM * Bryan Hsieh (bryanhsieh at fihspec.com) wrote: > Hi Ajit, > > I appreciate your reply! > But there is a problem here. > > My installation of Busybox is 1.12.4. It sounds like either awk or sed or not built in your busybox installation. Can you provide the output of cat /proc/ltt on your target system ? Check the content of ltt-armall to see which external programs it needs. Also, if you built the lttng probes as modules, you may need to load them explicitely. Mathieu > > I follow your suggestion to test on x86 Desktop ARM simulator, the message said that can't find /proc/ltt. I think it should be, because I didn't patch to ARM simulator Linux kernel. > > [sbox-ARM: ~/ltt-control-0.55-16102008/lttctl] > ltt-armall > Connecting all markers > cat: /proc/ltt: No such file or directory > Connecting high-rate markers to tap > cat: /proc/ltt: No such file or directory > > It should be work on my target device, but it seems not. > I can't find /ltt under /proc > Maybe it's root cause, but why there is no /ltt/proc? And why there is no " cat: /proc/ltt: No such file or directory" message in my target device during running ltt-armall? > > BTW, I mount debugfs in init.c > Mount("debugfs", "/sys/kernel/debug", "debugfs", 0, NULL); > It should not be a problem. > > -----------------------------------------------------------------------------------------------------------Another problem: > I command as following steps. > # lttctl -n trace1 -q > Linux Trace Toolkit Trace Control 0.55-16102008 > Controlling trace : trace1 > > # lttctl -n trace1 -r > Linux Trace Toolkit Trace Control 0.55-16102008 > Controlling trace : trace1 > > >>> Android shell hang in here, I have to press Ctrl+Z to force stopping - lttctl -n trace1 -r .Why? Is anything wrong? > # ls -l ../tmp/trace1 > drwxrwxrwx root root 1980-01-07 06:37 control > -rwxrwxrwx root root 1048576 1980-01-07 06:37 cpu_0 > # ls -l ../tmp/trace1/control > -rwxrwxrwx root root 0 1980-01-07 06:37 metadata_0 > -rwxrwxrwx root root 0 1980-01-07 06:37 interrupts_0 > -rwxrwxrwx root root 10240 1980-01-07 06:37 processes_0 > -rwxrwxrwx root root 65536 1980-01-07 06:37 modules_0 > -rwxrwxrwx root root 65536 1980-01-07 06:37 network_0 > > The Channel in /sys/kernel/debug/ltt/trace1 is clear. Does it cause by destroy channel command? > > The metadata_0 and interrupts_0 are empty, Is that correct? I've done some key press, the interrupts_o shouldn't be empt. > > And those files almost filled by 0 except for the short header I can't make sure, it looks like incomplete. > > I hope you can help me to solve these issues, I am appreciate you so much.. > > Best regards, > Bryan Hsieh > > SPEC, FIH Co., Ltd. > Tel:+886-3-270-0000 #2723 > Fax:+886-3-270-0001 > > > -----Original Message----- > From: Ajit R [mailto:ajitrzid@yahoo.com] > Sent: Friday, July 31, 2009 6:17 PM > To: 'ltt-dev at lists.casi.polymtl.ca'; Bryan Hsieh > Cc: Tommy Peng > Subject: Re: [ltt-dev] Some LTTng 0.43 issue on ARM > > > Hi Bryan, > My replies with the [Ajit] tag. > LTT Gods correct me if I am wrong. > > Thanks > Ajit > > --- On Fri, 7/31/09, Bryan Hsieh <bryanhsieh at fihspec.com> wrote: > > > From: Bryan Hsieh <bryanhsieh@fihspec.com> > > Subject: [ltt-dev] Some LTTng 0.43 issue on ARM > > To: "'ltt-dev at lists.casi.polymtl.ca'" <ltt-dev at lists.casi.polymtl.ca> > > Cc: "Tommy Peng" <tommypeng at fihspec.com> > > Date: Friday, July 31, 2009, 12:19 PM > > > Hi Mathieu, > > all, > > > > > > > > > > I've ported > > LTTng 0.43 on Android successfully, > > but I am wondering if the result is correct, please help me > > to verify, thank > > you very much! > > > > > > > > > > My Traced > > machine: > > > > ARM11 (ARM1136J(F)-S) / Linux > > 2.6..27 > > (Android 1.5) > > > > My > > visualization machine: > > > > X86 / Sun xVM Virtual Box (Unbuntu > > 8.10) > > > > > > > > LTTng: 0.43 > > > > LTTV: > > 0.11.2 > > > > LTT Control: 0.55 > > > > > > > > > > I > > connect to Android > > shell by ADB tool after Android bring up, then I command as > > following step, the > > command line options are different with LTTng manual, > > because the example in manual > > is for old version. > > > > > > > > > > # ltt-armall > > > [ Ajit ] > ltt-armall is a shell script. It connects + arms the markers in the > kernel. So depending on your kernel + LTT version it would write some > "text" into the /proc/ltt directory or a "1" into files in > the debugfs directory. > Make sure, the binaries for running the script are available in your target environment. For eg. awk, grep, sed etc. Busybox versions might have their limitations. If they do not work, you could do equivalent things on your Desktop and then write a simple script, doing the same on your target. > Once you run "ltt-armall", you should see some debug statements in your "dmesg" like "connect" or "arm" etc. > > > > > Connecting all > > markers > > > > Connecting high-rate > > markers to tap > > > > # lttctl > > -n trace1 -d -l /sys/kernel/debug/ltt -t > > /system/traces/trace1 > > > [Ajit] > Not sure if your "-l" option is right. you have to point it to the place where you mounted your debugfs. > > > > > Linux Trace Toolkit > > Trace Control 0.55-16102008 > > > > > > > > > > Controlling trace : > > trace1 > > > > > > > > > > Linux Trace Toolkit > > Trace Daemon 0.55-16102008 > > > > > > > > > > Reading from debugfs > > directory : /sys/kernel/debug/ltt/trace1 > > > > > > Writing to trace > > directory : /system/traces/trace1 > > > > > > > > > > It > > seems the LTT control > > and daemon work fine. I run some scenarios for 1 minute on > > Android, but the trace > > files are still empty (all files length are > > 0) > > > > What's > > wrong? Is > > there any step I missed? Or the LTT control and daemon > > don't work? > > > > > > > > > > # ls > > -l ../../sys/kernel/debug/ltt/ > > > > > > > > drwxr-xr-x > > root > > root > > 1980-01-07 01:13 trace1 > > > > # ls > > -l > > ../../sys/kernel/debug/ltt/trace1 > > > > > > -r-------- > > root > > root > > 0 > > 1980-01-07 01:13 cpu_0 > > > > [Ajit] > If you have connected and armed it right, you should see the cpu_0 > file increasing in size. After sometime, disconnect by running the > ltt-control -n whatever-name -R > and then you will see files in the control directory (network, process, etc ) grown in size. Tar and move the trace directory to your host and do the needful. > > > > drwxr-xr-x > > root > > root > > 1980-01-07 01:13 control > > > > # ls > > -l > > ../../sys/kernel/debug/ltt/trace1/control > > > > > > -r-------- > > root > > root > > 0 > > 1980-01-07 01:13 network_0 > > > > -r-------- > > root > > root > > 0 > > 1980-01-07 01:13 modules_0 > > > > -r-------- > > root > > root > > 0 > > 1980-01-07 01:13 processes_0 > > > > -r-------- > > root > > root > > 0 > > 1980-01-07 01:13 interrupts_0 > > > > -r-------- > > root > > root > > 0 1980-01-07 01:13 metadata_0 > > > > # ls -l > > .../traces/ > > > > > > > > drwxrwxrwx > > root > > root > > 1980-01-07 01:13 trace1 > > > > # ls -l > > .../traces/trace1 > > > > drwxrwxrwx > > root > > root > > 1980-01-07 01:13 control > > > > -rwxrwxrwx > > root > > root > > 0 > > 1980-01-07 01:13 cpu_0 > > > > # ls -l > > .../traces/trace1/control > > > > > > > > -rwxrwxrwx > > root > > root > > 0 > > 1980-01-07 01:13 metadata_0 > > > > -rwxrwxrwx > > root > > root > > 0 > > 1980-01-07 01:13 interrupts_0 > > > > -rwxrwxrwx > > root > > root > > 0 > > 1980-01-07 01:13 processes_0 > > > > -rwxrwxrwx > > root > > root > > 0 > > 1980-01-07 01:13 modules_0 > > > > -rwxrwxrwx > > root > > root > > 0 > > 1980-01-07 01:13 network_0 > > > > > > > > > > BTW, > > I would like to > > know what is the channel concept for LTTng? What's > > the relationship between > > trace and channel? What's the normal, flight recorder > > and hybrid mode? > > > > Which > > files should I get > > for LTTV on another visualization > > machine? > > > > So far, I > > can't find related document or the > > information isn't enough for me. > > > > > > > > > > Thanks for your > > help! > > > > > > > > > > Best > > regards, > > > > Bryan Hsieh > > > > SPEC, FIH > > Co., Ltd. > > > > Tel:+886-3-270-0000 #2723 > > > > Fax:+886-3-270-0001 > > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > ltt-dev mailing list > > ltt-dev at lists.casi.polymtl.ca > > http://lists.casi.polymtl.ca/cgi-bin/mailman/listinfo/ltt-dev > > > > > > This e-mail message together with any attachments thereto (if any) is confidential, protected under an enforceable non-disclosure agreement, > intended only for the use of the named recipient(s) above and may contain information that is privileged, belonging to professional work products > or exempt from disclosure under applicable laws. Any unauthorized review, use, copying, disclosure, or distribution of any information contained > in or attached to this transmission is STRICTLY PROHIBITED and may be against the laws. If you have received this message in error, or are not > the named recipient(s), please immediately notify the sender by e-mail or telephone at IT support hotline 886-3-2700000 EXT 1333 and delete this > e-mail message and any attached documentation from your computer. Receipt by anyone other than the intended recipient(s) is not a waiver of any > attorney-client or work product privilege. Thank you! > > > _______________________________________________ > ltt-dev mailing list > ltt-dev at lists.casi.polymtl.ca > http://lists.casi.polymtl.ca/cgi-bin/mailman/listinfo/ltt-dev > -- Mathieu Desnoyers OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F BA06 3F25 A8FE 3BAE 9A68 This e-mail message together with any attachments thereto (if any) is confidential, protected under an enforceable non-disclosure agreement, intended only for the use of the named recipient(s) above and may contain information that is privileged, belonging to professional work products or exempt from disclosure under applicable laws. Any unauthorized review, use, copying, disclosure, or distribution of any information contained in or attached to this transmission is STRICTLY PROHIBITED and may be against the laws. If you have received this message in error, or are not the named recipient(s), please immediately notify the sender by e-mail or telephone at IT support hotline 886-3-2700000 EXT 1333 and delete this e-mail message and any attached documentation from your computer. Receipt by anyone other than the intended recipient(s) is not a waiver of any attorney-client or work product privilege. Thank you! -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: busybox version and command query.txt URL: <http://lists.casi.polymtl.ca/pipermail/lttng-dev/attachments/20090804/39917614/attachment-0006.txt> -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: LTTng Execution Log.txt URL: <http://lists.casi.polymtl.ca/pipermail/lttng-dev/attachments/20090804/39917614/attachment-0007.txt> -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: LTTng Kernel Config.txt URL: <http://lists.casi.polymtl.ca/pipermail/lttng-dev/attachments/20090804/39917614/attachment-0008.txt> ^ permalink raw reply [flat|nested] 9+ messages in thread
* [ltt-dev] Some LTTng 0.43 issue on ARM 2009-08-04 8:28 ` [ltt-dev] Some LTTng 0.43 issue on ARM Bryan Hsieh @ 2009-08-04 8:28 ` Bryan Hsieh 2009-08-04 8:28 ` Bryan Hsieh 2009-08-04 13:18 ` Mathieu Desnoyers 2 siblings, 0 replies; 9+ messages in thread From: Bryan Hsieh @ 2009-08-04 8:28 UTC (permalink / raw) Hi Mathieu, All Please check my comment which marked by [Bryan], Thanks! ----------------------------------------------------------------------------- [Bryan] All the related LTT functions are build-in not build as modules, Please check attachment- LTTng Kernel Config..txt, I am not sure if Kernel->user space relay support (formerly relayfs) need to build-in. Is there any item I missed? ----------------------------------------------------------------------------- [ ] Kernel->user space relay support (formerly relayfs) [*] Activate tracepoints [*] Activate markers Linux Trace Toolkit ---> <*> Compile lttng tracing probes [*] LTTng fine-grained timestamping [*] Linux Trace Toolkit Instrumentation Support <*> Linux Trace Toolkit High-speed Lockless Data Relay <*> Linux Trace Toolkit Lock-Protected Data Relay [*] Debug check for random access in ltt relay buffers <*> Linux Trace Toolkit Serializer <*> Linux Trace Toolkit Marker Control <*> Linux Trace Toolkit Tracer [*] Align Linux Trace Toolkit Traces [*] Add event size field to LTT events for tracer debugging [*] Support trace extraction from crash dump <*> Linux Trace Toolkit Netlink Controller <*> Linux Trace Toolkit State Dump ----------------------------------------------------------------------------- [Bryan] I am sure that the external utilities are build-in busybox, please check the attachment - " busybox version and command query.txt". Finally, I figure out why the ltt-armall doesn't work correctly in my target system, root cause is the output pipeline will be empty after grep twice, even I use a temp file to avoid connecting two grep commands directly, it's still empty. So I copy the /proc/ltt to my local desktop to do ltt-armall script, then generate the /proc/ltt which been filtered by following command. MARKERS=`cat /proc/ltt|grep -v %k|awk '{print $2}'|sort -u|grep -v ^core_|grep -v ^locking_|grep -v ^lockdep_` I rename it as ltt_filter and copy to my target system path /system/bin/, then modify ltt-armall by following modification and run it. MARKERS=`cat /system/bin/ltt_filter` The execution result on screen: # ltt-armall-1 Connecting all markers Connecting fs_buffer_wait_end Connecting fs_buffer_wait_start Connecting fs_close Connecting fs_exec Connecting fs_ioctl Connecting fs_llseek Connecting fs_lseek Connecting fs_open Connecting fs_pollfd Connecting fs_pread64 Connecting fs_pwrite64 Connecting fs_read Connecting fs_readv Connecting fs_select Connecting fs_write Connecting fs_writev Connecting input_event Connecting ipc_msg_create Connecting ipc_sem_create Connecting ipc_shm_create Connecting kernel_arch_ipc_call Connecting kernel_arch_kthread_create Connecting kernel_arch_syscall_entry Connecting kernel_arch_syscall_exit Connecting kernel_arch_trap_entry Connecting kernel_arch_trap_exit Connecting kernel_irq_entry Connecting kernel_irq_exit Connecting kernel_kthread_stop Connecting kernel_kthread_stop_ret Connecting kernel_module_free Connecting kernel_module_load Connecting kernel_printk Connecting kernel_process_exit Connecting kernel_process_fork Connecting kernel_process_free Connecting kernel_process_wait Connecting kernel_sched_migrate_task Connecting kernel_sched_schedule Connecting kernel_sched_try_wakeup Connecting kernel_sched_wait_task Connecting kernel_sched_wakeup_new_task Connecting kernel_send_signal Connecting kernel_softirq_entry Connecting kernel_softirq_exit Connecting kernel_softirq_raise Connecting kernel_tasklet_high_entry Connecting kernel_tasklet_high_exit Connecting kernel_tasklet_low_entry Connecting kernel_tasklet_low_exit Connecting kernel_timer_itimer_expired Connecting kernel_timer_itimer_set Connecting kernel_timer_set Connecting kernel_timer_timeout Connecting kernel_timer_update_time Connecting kernel_vprintk Connecting list_file_descriptor Connecting list_interrupt Connecting list_module Connecting list_network_ip_interface Connecting list_network_ipv4_interface Connecting list_process_state Connecting list_statedump_end Connecting list_vm_map Connecting mm_handle_fault_entry Connecting mm_handle_fault_exit Connecting mm_huge_page_alloc Connecting mm_huge_page_free Connecting mm_page_alloc Connecting mm_page_free Connecting mm_wait_on_page_end Connecting mm_wait_on_page_start Connecting rcu_classic_callback Connecting rcu_classic_call_rcu Connecting rcu_classic_call_rcu_bh Connecting statedump_softirq_vec Connecting high-rate markers to tap Connecting fs_buffer_wait_end Connecting fs_buffer_wait_start Connecting fs_close Connecting fs_exec Connecting fs_ioctl Connecting fs_llseek Connecting fs_lseek Connecting fs_open Connecting fs_pollfd Connecting fs_pread64 Connecting fs_pwrite64 Connecting fs_read Connecting fs_readv Connecting fs_select Connecting fs_write Connecting fs_writev Connecting input_event Connecting ipc_msg_create Connecting ipc_sem_create Connecting ipc_shm_create Connecting kernel_arch_ipc_call Connecting kernel_arch_kthread_create Connecting kernel_arch_syscall_entry Connecting kernel_arch_syscall_exit Connecting kernel_arch_trap_entry Connecting kernel_arch_trap_exit Connecting kernel_irq_entry Connecting kernel_irq_exit Connecting kernel_kthread_stop Connecting kernel_kthread_stop_ret Connecting kernel_module_free Connecting kernel_module_load Connecting kernel_printk Connecting kernel_process_exit Connecting kernel_process_fork Connecting kernel_process_free Connecting kernel_process_wait Connecting kernel_sched_migrate_task Connecting kernel_sched_schedule Connecting kernel_sched_try_wakeup Connecting kernel_sched_wait_task Connecting kernel_sched_wakeup_new_task Connecting kernel_send_signal Connecting kernel_softirq_entry Connecting kernel_softirq_exit Connecting kernel_softirq_raise Connecting kernel_tasklet_high_entry Connecting kernel_tasklet_high_exit Connecting kernel_tasklet_low_entry Connecting kernel_tasklet_low_exit Connecting kernel_timer_itimer_expired Connecting kernel_timer_itimer_set Connecting kernel_timer_set Connecting kernel_timer_timeout Connecting kernel_timer_update_time Connecting kernel_vprintk Connecting list_file_descriptor Connecting list_interrupt Connecting list_module Connecting list_network_ip_interface Connecting list_network_ipv4_interface Connecting list_process_state Connecting list_statedump_end Connecting list_vm_map Connecting mm_handle_fault_entry Connecting mm_handle_fault_exit Connecting mm_huge_page_alloc Connecting mm_huge_page_free Connecting mm_page_alloc Connecting mm_page_free Connecting mm_wait_on_page_end Connecting mm_wait_on_page_start Connecting rcu_classic_callback Connecting rcu_classic_call_rcu Connecting rcu_classic_call_rcu_bh Connecting statedump_softirq_vec ----------------------------------------------------------------------------- [Bryan] It's that correct result? ----------------------------------------------------------------------------- # cat /proc/ltt marker: kernel_arch_kthread_create format: "pid %ld fn %p" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_arch_syscall_exit format: " " state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_arch_syscall_entry format: "syscall_id %d ip #p%ld" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_arch_ipc_call format: "call %u first %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_arch_trap_exit format: " " state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_arch_trap_entry format: "trap_id %ld ip #p%ld" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: statedump_softirq_vec format: "id %d address %p symbol %s" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: list_module format: "name %s state %d refcount %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: core_marker_format format: "name %s format %s" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: core_marker_format format: "name %s format %s" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: list_statedump_end format: " " state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: list_process_state format: "pid %d parent_pid %d name %s type %d mode %d submode %d status %d tgid %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: list_file_descriptor format: "filename %s pid %d fd %u" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: list_vm_map format: "pid %d start %lu end %lu flags %lu pgoff %lu inode %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: list_interrupt format: "name %s action %s irq_id %u" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: list_network_ip_interface format: "name %s address #4u%lu up %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: list_network_ipv4_interface format: "name %s address #4u%lu up %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: core_marker_id format: "name %s id %hu int #1u%zu long #1u%zu pointer #1u%zu size_t #1u%zu alignment #1u%u" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: core_marker_id format: "name %s id %hu int #1u%zu long #1u%zu pointer #1u%zu size_t #1u%zu alignment #1u%u" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: core_marker_format format: "name %s format %s" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: core_marker_id format: "name %s id %hu int #1u%zu long #1u%zu pointer #1u%zu size_t #1u%zu alignment #1u%u" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: core_marker_id format: "name %s id %hu int #1u%zu long #1u%zu pointer #1u%zu size_t #1u%zu alignment #1u%u" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_module_load format: "name %s" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_module_free format: "name %s" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_vprintk format: "loglevel %c string %s ip %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_printk format: "ip %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_timer_timeout format: "pid %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_timer_update_time format: "jiffies #8u%llu xtime_sec %ld xtime_nsec %ld walltomonotonic_sec %ld walltomonotonic_nsec %ld" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_timer_set format: "expires %lu function %p data %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_timer_itimer_set format: "which %d interval_sec %ld interval_usec %ld value_sec %ld value_usec %ld" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_timer_itimer_expired format: "pid %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_process_fork format: "parent_pid %d child_pid %d child_tgid %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_process_wait format: "pid %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_process_exit format: "pid %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_process_free format: "pid %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_send_signal format: "pid %d signal %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_sched_migrate_task format: "pid %d state %ld dest_cpu %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_sched_schedule format: "prev_pid %d next_pid %d prev_state %ld" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_sched_wakeup_new_task format: "pid %d state %ld cpu_id %u" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_sched_try_wakeup format: "pid %d state %ld cpu_id %u" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_sched_wait_task format: "pid %d state %ld" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_kthread_stop_ret format: "ret %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_kthread_stop format: "pid %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_tasklet_high_exit format: "func %p data %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_tasklet_high_entry format: "func %p data %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_tasklet_low_exit format: "func %p data %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_tasklet_low_entry format: "func %p data %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_softirq_raise format: "softirq_id %u" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_softirq_exit format: "softirq_id %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_softirq_entry format: "softirq_id %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_irq_exit format: "handled #1u%u" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_irq_entry format: "irq_id %u kernel_mode %u ip %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: mm_page_alloc format: "order %u pfn %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: mm_page_free format: "order %u pfn %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: mm_handle_fault_exit format: "res %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: mm_handle_fault_entry format: "address %lu ip #p%ld write_access %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: mm_huge_page_alloc format: "pfn %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: mm_huge_page_free format: "pfn %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: mm_wait_on_page_end format: "pfn %lu bit_nr %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: mm_wait_on_page_start format: "pfn %lu bit_nr %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: fs_pollfd format: "fd %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: fs_select format: "fd %d timeout #8d%lld" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: fs_writev format: "fd %lu vlen %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: fs_readv format: "fd %lu vlen %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: fs_pwrite64 format: "fd %u count %zu pos %llu" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: fs_pread64 format: "fd %u count %zu pos %llu" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: fs_write format: "fd %u count %zu" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: fs_read format: "fd %u count %zu" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: fs_llseek format: "fd %u offset %lld origin %u" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: fs_lseek format: "fd %u offset %ld origin %u" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: fs_close format: "fd %u" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: fs_open format: "fd %d filename %s" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: fs_ioctl format: "fd %u cmd %u arg %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: fs_exec format: "filename %s" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: fs_buffer_wait_end format: "bh %p" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: fs_buffer_wait_start format: "bh %p" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: ipc_shm_create format: "id %ld flags %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: ipc_sem_create format: "id %ld flags %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: ipc_msg_create format: "id %ld flags %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: lockdep_lock_release format: "retaddr 0x%lX lock %p nested %d" state: 0 call: 0xc0093374 probe single : 0xc00920ec marker: lockdep_lock_acquire format: "retaddr 0x%lX subclass %u lock %p trylock %d read %d hardirqs_off %d" state: 0 call: 0xc0093374 probe single : 0xc00920ec marker: lockdep_softirqs_off format: "retaddr 0x%lX" state: 0 call: 0xc0093374 probe single : 0xc00920ec marker: lockdep_softirqs_on format: "retaddr 0x%lX" state: 0 call: 0xc0093374 probe single : 0xc00920ec marker: lockdep_hardirqs_off format: "retaddr 0x%lX" state: 0 call: 0xc0093374 probe single : 0xc00920ec marker: lockdep_hardirqs_on format: "retaddr 0x%lX" state: 0 call: 0xc0093374 probe single : 0xc00920ec marker: rcu_classic_call_rcu_bh format: "func %p ip 0x%lX" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: rcu_classic_call_rcu format: "func %p ip 0x%lX" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: rcu_classic_callback format: "func %p" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: input_event format: "type %u code %u value %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 ----------------------------------------------------------------------------- [Bryan] But the /proc/ltt is still same, it's that correct? What's the expected result? ----------------------------------------------------------------------------- # df /dev: 39564K total, 0K used, 39564K available (block size 4096) /sqlite_stmt_journals: 4096K total, 0K used, 4096K available (block size 4096) /system: 72704K total, 71680K used, 1024K available (block size 4096) /data: 66816K total, 37812K used, 29004K available (block size 4096) /cache: 67584K total, 1156K used, 66428K available (block size 4096) /sdcard: 991708K total, 147040K used, 844668K available (block size 4096) ----------------------------------------------------------------------------- [Bryan] There is 29004K available for trace log file /data/ltt/trace1. Is that enough? ----------------------------------------------------------------------------- # cat /proc/meminfo MemTotal: 79128 kB MemFree: 2440 kB Buffers: 36 kB Cached: 29872 kB SwapCached: 0 kB Active: 61384 kB Inactive: 5052 kB SwapTotal: 0 kB SwapFree: 0 kB Dirty: 0 kB Writeback: 0 kB AnonPages: 36552 kB Mapped: 24888 kB Slab: 3856 kB SReclaimable: 904 kB SUnreclaim: 2952 kB PageTables: 3660 kB NFS_Unstable: 0 kB Bounce: 0 kB WritebackTmp: 0 kB CommitLimit: 39564 kB Committed_AS: 1032216 kB VmallocTotal: 409600 kB VmallocUsed: 52908 kB VmallocChunk: 332796 kB ----------------------------------------------------------------------------- [Bryan] There is 2440 kB free memory. Is that enough? ----------------------------------------------------------------------------- # lttctl -n trace1 -d -l /sys/kernel/debug/ltt -t /data/ltt/trace1 Linux Trace Toolkit Trace Control 0.55-16102008 Controlling trace : trace1 Linux Trace Toolkit Trace Daemon 0.55-16102008 Reading from debugfs directory : /sys/kernel/debug/ltt/trace1 Writing to trace directory : /data/ltt/trace1 ----------------------------------------------------------------------------- [Bryan] do some tests on target device, then check the channel and trace ----------------------------------------------------------------------------- # ls -l /sys/kernel/debug/ltt/trace1 -r-------- root root 0 1980-01-06 00:10 cpu_0 drwxr-xr-x root root 1980-01-06 00:10 control # ls -l /sys/kernel/debug/ltt/trace1/control -r-------- root root 0 1980-01-06 00:10 network_0 -r-------- root root 0 1980-01-06 00:10 modules_0 -r-------- root root 0 1980-01-06 00:10 processes_0 -r-------- root root 0 1980-01-06 00:10 interrupts_0 -r-------- root root 0 1980-01-06 00:10 metadata_0 # ls -l /data/ltt/trace1 drwxrwxrwx root root 1980-01-06 00:10 control -rwxrwxrwx root root 29949952 1980-01-06 00:10 cpu_0 # ls -l /data/ltt/trace1/control -rwxrwxrwx root root 0 1980-01-06 00:10 metadata_0 -rwxrwxrwx root root 0 1980-01-06 00:10 interrupts_0 -rwxrwxrwx root root 0 1980-01-06 00:10 processes_0 -rwxrwxrwx root root 0 1980-01-06 00:10 modules_0 -rwxrwxrwx root root 0 1980-01-06 00:10 network_0 ----------------------------------------------------------------------------- [Bryan] It seems that the capacity has been exhausted by cpu_0, because 29004K available. It's that correct that cpu_0 will grow to so huge (29M)? ----------------------------------------------------------------------------- # lttctl -n trace1 -d -R Linux Trace Toolkit Trace Control 0.55-16102008 Controlling trace : trace1 ^Z[1] + Stopped lttctl -n trace1 -d -R ----------------------------------------------------------------------------- [Bryan] hang in here...I have to break it by Ctrl+Z, What's problem? ----------------------------------------------------------------------------- # ls -l /data/ltt/trace1/ drwxrwxrwx root root 1980-01-06 00:10 control -rwxrwxrwx root root 29949952 1980-01-06 00:10 cpu_0 # ls -l /data/ltt/trace1/control -rwxrwxrwx root root 0 1980-01-06 00:10 metadata_0 -rwxrwxrwx root root 0 1980-01-06 00:10 interrupts_0 -rwxrwxrwx root root 0 1980-01-06 00:10 processes_0 -rwxrwxrwx root root 0 1980-01-06 00:10 modules_0 -rwxrwxrwx root root 0 1980-01-06 00:10 network_0 # ls -l /sys/kernel/debug/ltt/trace1 -r-------- root root 0 1980-01-06 00:10 cpu_0 drwxr-xr-x root root 1980-01-06 00:10 control # ls -l /sys/kernel/debug/ltt/trace1/control -r-------- root root 0 1980-01-06 00:10 network_0 -r-------- root root 0 1980-01-06 00:10 modules_0 -r-------- root root 0 1980-01-06 00:10 processes_0 -r-------- root root 0 1980-01-06 00:10 interrupts_0 -r-------- root root 0 1980-01-06 00:10 metadata_0 # df /dev: 39564K total, 0K used, 39564K available (block size 4096) /sqlite_stmt_journals: 4096K total, 0K used, 4096K available (block size 4096) /system: 72704K total, 71680K used, 1024K available (block size 4096) /data: 66816K total, 66732K used, 84K available (block size 4096) /cache: 67584K total, 1156K used, 66428K available (block size 4096) /sdcard: 991708K total, 147040K used, 844668K available (block size 4096) ----------------------------------------------------------------------------- [Bryan] I think the trace is incomplete. ----------------------------------------------------------------------------- Please help me! Thank you very much! Best regards, Bryan Hsieh SPEC, FIH Co., Ltd. Tel:+886-3-270-0000 #2723 Fax:+886-3-270-0001 -----Original Message----- From: Mathieu Desnoyers [mailto:compudj@krystal.dyndns.org] Sent: Friday, July 31, 2009 11:43 PM To: Bryan Hsieh Cc: 'Ajit R'; 'ltt-dev at lists.casi.polymtl.ca'; Tommy Peng Subject: Re: [ltt-dev] Some LTTng 0.43 issue on ARM * Bryan Hsieh (bryanhsieh at fihspec.com) wrote: > Hi Ajit, > > I appreciate your reply! > But there is a problem here. > > My installation of Busybox is 1.12.4. It sounds like either awk or sed or not built in your busybox installation. Can you provide the output of cat /proc/ltt on your target system ? Check the content of ltt-armall to see which external programs it needs. Also, if you built the lttng probes as modules, you may need to load them explicitely. Mathieu > > I follow your suggestion to test on x86 Desktop ARM simulator, the message said that can't find /proc/ltt. I think it should be, because I didn't patch to ARM simulator Linux kernel. > > [sbox-ARM: ~/ltt-control-0.55-16102008/lttctl] > ltt-armall > Connecting all markers > cat: /proc/ltt: No such file or directory > Connecting high-rate markers to tap > cat: /proc/ltt: No such file or directory > > It should be work on my target device, but it seems not. > I can't find /ltt under /proc > Maybe it's root cause, but why there is no /ltt/proc? And why there is no " cat: /proc/ltt: No such file or directory" message in my target device during running ltt-armall? > > BTW, I mount debugfs in init.c > Mount("debugfs", "/sys/kernel/debug", "debugfs", 0, NULL); > It should not be a problem. > > -----------------------------------------------------------------------------------------------------------Another problem: > I command as following steps. > # lttctl -n trace1 -q > Linux Trace Toolkit Trace Control 0.55-16102008 > Controlling trace : trace1 > > # lttctl -n trace1 -r > Linux Trace Toolkit Trace Control 0.55-16102008 > Controlling trace : trace1 > > >>> Android shell hang in here, I have to press Ctrl+Z to force stopping - lttctl -n trace1 -r .Why? Is anything wrong? > # ls -l ../tmp/trace1 > drwxrwxrwx root root 1980-01-07 06:37 control > -rwxrwxrwx root root 1048576 1980-01-07 06:37 cpu_0 > # ls -l ../tmp/trace1/control > -rwxrwxrwx root root 0 1980-01-07 06:37 metadata_0 > -rwxrwxrwx root root 0 1980-01-07 06:37 interrupts_0 > -rwxrwxrwx root root 10240 1980-01-07 06:37 processes_0 > -rwxrwxrwx root root 65536 1980-01-07 06:37 modules_0 > -rwxrwxrwx root root 65536 1980-01-07 06:37 network_0 > > The Channel in /sys/kernel/debug/ltt/trace1 is clear. Does it cause by destroy channel command? > > The metadata_0 and interrupts_0 are empty, Is that correct? I've done some key press, the interrupts_o shouldn't be empt. > > And those files almost filled by 0 except for the short header I can't make sure, it looks like incomplete. > > I hope you can help me to solve these issues, I am appreciate you so much.. > > Best regards, > Bryan Hsieh > > SPEC, FIH Co., Ltd. > Tel:+886-3-270-0000 #2723 > Fax:+886-3-270-0001 > > > -----Original Message----- > From: Ajit R [mailto:ajitrzid@yahoo.com] > Sent: Friday, July 31, 2009 6:17 PM > To: 'ltt-dev at lists.casi.polymtl.ca'; Bryan Hsieh > Cc: Tommy Peng > Subject: Re: [ltt-dev] Some LTTng 0.43 issue on ARM > > > Hi Bryan, > My replies with the [Ajit] tag. > LTT Gods correct me if I am wrong. > > Thanks > Ajit > > --- On Fri, 7/31/09, Bryan Hsieh <bryanhsieh at fihspec.com> wrote: > > > From: Bryan Hsieh <bryanhsieh@fihspec.com> > > Subject: [ltt-dev] Some LTTng 0.43 issue on ARM > > To: "'ltt-dev at lists.casi.polymtl.ca'" <ltt-dev at lists.casi.polymtl.ca> > > Cc: "Tommy Peng" <tommypeng at fihspec.com> > > Date: Friday, July 31, 2009, 12:19 PM > > > Hi Mathieu, > > all, > > > > > > > > > > I've ported > > LTTng 0.43 on Android successfully, > > but I am wondering if the result is correct, please help me > > to verify, thank > > you very much! > > > > > > > > > > My Traced > > machine: > > > > ARM11 (ARM1136J(F)-S) / Linux > > 2.6..27 > > (Android 1.5) > > > > My > > visualization machine: > > > > X86 / Sun xVM Virtual Box (Unbuntu > > 8.10) > > > > > > > > LTTng: 0.43 > > > > LTTV: > > 0.11.2 > > > > LTT Control: 0.55 > > > > > > > > > > I > > connect to Android > > shell by ADB tool after Android bring up, then I command as > > following step, the > > command line options are different with LTTng manual, > > because the example in manual > > is for old version. > > > > > > > > > > # ltt-armall > > > [ Ajit ] > ltt-armall is a shell script. It connects + arms the markers in the > kernel. So depending on your kernel + LTT version it would write some > "text" into the /proc/ltt directory or a "1" into files in > the debugfs directory. > Make sure, the binaries for running the script are available in your target environment. For eg. awk, grep, sed etc. Busybox versions might have their limitations. If they do not work, you could do equivalent things on your Desktop and then write a simple script, doing the same on your target. > Once you run "ltt-armall", you should see some debug statements in your "dmesg" like "connect" or "arm" etc. > > > > > Connecting all > > markers > > > > Connecting high-rate > > markers to tap > > > > # lttctl > > -n trace1 -d -l /sys/kernel/debug/ltt -t > > /system/traces/trace1 > > > [Ajit] > Not sure if your "-l" option is right. you have to point it to the place where you mounted your debugfs. > > > > > Linux Trace Toolkit > > Trace Control 0.55-16102008 > > > > > > > > > > Controlling trace : > > trace1 > > > > > > > > > > Linux Trace Toolkit > > Trace Daemon 0.55-16102008 > > > > > > > > > > Reading from debugfs > > directory : /sys/kernel/debug/ltt/trace1 > > > > > > Writing to trace > > directory : /system/traces/trace1 > > > > > > > > > > It > > seems the LTT control > > and daemon work fine. I run some scenarios for 1 minute on > > Android, but the trace > > files are still empty (all files length are > > 0) > > > > What's > > wrong? Is > > there any step I missed? Or the LTT control and daemon > > don't work? > > > > > > > > > > # ls > > -l ../../sys/kernel/debug/ltt/ > > > > > > > > drwxr-xr-x > > root > > root > > 1980-01-07 01:13 trace1 > > > > # ls > > -l > > ../../sys/kernel/debug/ltt/trace1 > > > > > > -r-------- > > root > > root > > 0 > > 1980-01-07 01:13 cpu_0 > > > > [Ajit] > If you have connected and armed it right, you should see the cpu_0 > file increasing in size. After sometime, disconnect by running the > ltt-control -n whatever-name -R > and then you will see files in the control directory (network, process, etc ) grown in size. Tar and move the trace directory to your host and do the needful. > > > > drwxr-xr-x > > root > > root > > 1980-01-07 01:13 control > > > > # ls > > -l > > ../../sys/kernel/debug/ltt/trace1/control > > > > > > -r-------- > > root > > root > > 0 > > 1980-01-07 01:13 network_0 > > > > -r-------- > > root > > root > > 0 > > 1980-01-07 01:13 modules_0 > > > > -r-------- > > root > > root > > 0 > > 1980-01-07 01:13 processes_0 > > > > -r-------- > > root > > root > > 0 > > 1980-01-07 01:13 interrupts_0 > > > > -r-------- > > root > > root > > 0 1980-01-07 01:13 metadata_0 > > > > # ls -l > > .../traces/ > > > > > > > > drwxrwxrwx > > root > > root > > 1980-01-07 01:13 trace1 > > > > # ls -l > > .../traces/trace1 > > > > drwxrwxrwx > > root > > root > > 1980-01-07 01:13 control > > > > -rwxrwxrwx > > root > > root > > 0 > > 1980-01-07 01:13 cpu_0 > > > > # ls -l > > .../traces/trace1/control > > > > > > > > -rwxrwxrwx > > root > > root > > 0 > > 1980-01-07 01:13 metadata_0 > > > > -rwxrwxrwx > > root > > root > > 0 > > 1980-01-07 01:13 interrupts_0 > > > > -rwxrwxrwx > > root > > root > > 0 > > 1980-01-07 01:13 processes_0 > > > > -rwxrwxrwx > > root > > root > > 0 > > 1980-01-07 01:13 modules_0 > > > > -rwxrwxrwx > > root > > root > > 0 > > 1980-01-07 01:13 network_0 > > > > > > > > > > BTW, > > I would like to > > know what is the channel concept for LTTng? What's > > the relationship between > > trace and channel? What's the normal, flight recorder > > and hybrid mode? > > > > Which > > files should I get > > for LTTV on another visualization > > machine? > > > > So far, I > > can't find related document or the > > information isn't enough for me. > > > > > > > > > > Thanks for your > > help! > > > > > > > > > > Best > > regards, > > > > Bryan Hsieh > > > > SPEC, FIH > > Co., Ltd. > > > > Tel:+886-3-270-0000 #2723 > > > > Fax:+886-3-270-0001 > > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > ltt-dev mailing list > > ltt-dev at lists.casi.polymtl.ca > > http://lists.casi.polymtl.ca/cgi-bin/mailman/listinfo/ltt-dev > > > > > > This e-mail message together with any attachments thereto (if any) is confidential, protected under an enforceable non-disclosure agreement, > intended only for the use of the named recipient(s) above and may contain information that is privileged, belonging to professional work products > or exempt from disclosure under applicable laws. Any unauthorized review, use, copying, disclosure, or distribution of any information contained > in or attached to this transmission is STRICTLY PROHIBITED and may be against the laws. If you have received this message in error, or are not > the named recipient(s), please immediately notify the sender by e-mail or telephone at IT support hotline 886-3-2700000 EXT 1333 and delete this > e-mail message and any attached documentation from your computer. Receipt by anyone other than the intended recipient(s) is not a waiver of any > attorney-client or work product privilege. Thank you! > > > _______________________________________________ > ltt-dev mailing list > ltt-dev at lists.casi.polymtl.ca > http://lists.casi.polymtl.ca/cgi-bin/mailman/listinfo/ltt-dev > -- Mathieu Desnoyers OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F BA06 3F25 A8FE 3BAE 9A68 This e-mail message together with any attachments thereto (if any) is confidential, protected under an enforceable non-disclosure agreement, intended only for the use of the named recipient(s) above and may contain information that is privileged, belonging to professional work products or exempt from disclosure under applicable laws. Any unauthorized review, use, copying, disclosure, or distribution of any information contained in or attached to this transmission is STRICTLY PROHIBITED and may be against the laws. If you have received this message in error, or are not the named recipient(s), please immediately notify the sender by e-mail or telephone at IT support hotline 886-3-2700000 EXT 1333 and delete this e-mail message and any attached documentation from your computer. Receipt by anyone other than the intended recipient(s) is not a waiver of any attorney-client or work product privilege. Thank you! -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: busybox version and command query.txt URL: <http://lists.casi.polymtl.ca/pipermail/ltt-dev/attachments/20090804/39917614/attachment-0003.txt> -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: LTTng Execution Log.txt URL: <http://lists.casi.polymtl.ca/pipermail/ltt-dev/attachments/20090804/39917614/attachment-0004.txt> -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: LTTng Kernel Config.txt URL: <http://lists.casi.polymtl.ca/pipermail/ltt-dev/attachments/20090804/39917614/attachment-0005.txt> ^ permalink raw reply [flat|nested] 9+ messages in thread
* [ltt-dev] Some LTTng 0.43 issue on ARM 2009-08-04 8:28 ` [ltt-dev] Some LTTng 0.43 issue on ARM Bryan Hsieh 2009-08-04 8:28 ` Bryan Hsieh @ 2009-08-04 8:28 ` Bryan Hsieh 2009-08-04 13:18 ` Mathieu Desnoyers 2 siblings, 0 replies; 9+ messages in thread From: Bryan Hsieh @ 2009-08-04 8:28 UTC (permalink / raw) Hi Mathieu, All Please check my comment which marked by [Bryan], Thanks! ----------------------------------------------------------------------------- [Bryan] All the related LTT functions are build-in not build as modules, Please check attachment- LTTng Kernel Config..txt, I am not sure if Kernel->user space relay support (formerly relayfs) need to build-in. Is there any item I missed? ----------------------------------------------------------------------------- [ ] Kernel->user space relay support (formerly relayfs) [*] Activate tracepoints [*] Activate markers Linux Trace Toolkit ---> <*> Compile lttng tracing probes [*] LTTng fine-grained timestamping [*] Linux Trace Toolkit Instrumentation Support <*> Linux Trace Toolkit High-speed Lockless Data Relay <*> Linux Trace Toolkit Lock-Protected Data Relay [*] Debug check for random access in ltt relay buffers <*> Linux Trace Toolkit Serializer <*> Linux Trace Toolkit Marker Control <*> Linux Trace Toolkit Tracer [*] Align Linux Trace Toolkit Traces [*] Add event size field to LTT events for tracer debugging [*] Support trace extraction from crash dump <*> Linux Trace Toolkit Netlink Controller <*> Linux Trace Toolkit State Dump ----------------------------------------------------------------------------- [Bryan] I am sure that the external utilities are build-in busybox, please check the attachment - " busybox version and command query.txt". Finally, I figure out why the ltt-armall doesn't work correctly in my target system, root cause is the output pipeline will be empty after grep twice, even I use a temp file to avoid connecting two grep commands directly, it's still empty. So I copy the /proc/ltt to my local desktop to do ltt-armall script, then generate the /proc/ltt which been filtered by following command. MARKERS=`cat /proc/ltt|grep -v %k|awk '{print $2}'|sort -u|grep -v ^core_|grep -v ^locking_|grep -v ^lockdep_` I rename it as ltt_filter and copy to my target system path /system/bin/, then modify ltt-armall by following modification and run it. MARKERS=`cat /system/bin/ltt_filter` The execution result on screen: # ltt-armall-1 Connecting all markers Connecting fs_buffer_wait_end Connecting fs_buffer_wait_start Connecting fs_close Connecting fs_exec Connecting fs_ioctl Connecting fs_llseek Connecting fs_lseek Connecting fs_open Connecting fs_pollfd Connecting fs_pread64 Connecting fs_pwrite64 Connecting fs_read Connecting fs_readv Connecting fs_select Connecting fs_write Connecting fs_writev Connecting input_event Connecting ipc_msg_create Connecting ipc_sem_create Connecting ipc_shm_create Connecting kernel_arch_ipc_call Connecting kernel_arch_kthread_create Connecting kernel_arch_syscall_entry Connecting kernel_arch_syscall_exit Connecting kernel_arch_trap_entry Connecting kernel_arch_trap_exit Connecting kernel_irq_entry Connecting kernel_irq_exit Connecting kernel_kthread_stop Connecting kernel_kthread_stop_ret Connecting kernel_module_free Connecting kernel_module_load Connecting kernel_printk Connecting kernel_process_exit Connecting kernel_process_fork Connecting kernel_process_free Connecting kernel_process_wait Connecting kernel_sched_migrate_task Connecting kernel_sched_schedule Connecting kernel_sched_try_wakeup Connecting kernel_sched_wait_task Connecting kernel_sched_wakeup_new_task Connecting kernel_send_signal Connecting kernel_softirq_entry Connecting kernel_softirq_exit Connecting kernel_softirq_raise Connecting kernel_tasklet_high_entry Connecting kernel_tasklet_high_exit Connecting kernel_tasklet_low_entry Connecting kernel_tasklet_low_exit Connecting kernel_timer_itimer_expired Connecting kernel_timer_itimer_set Connecting kernel_timer_set Connecting kernel_timer_timeout Connecting kernel_timer_update_time Connecting kernel_vprintk Connecting list_file_descriptor Connecting list_interrupt Connecting list_module Connecting list_network_ip_interface Connecting list_network_ipv4_interface Connecting list_process_state Connecting list_statedump_end Connecting list_vm_map Connecting mm_handle_fault_entry Connecting mm_handle_fault_exit Connecting mm_huge_page_alloc Connecting mm_huge_page_free Connecting mm_page_alloc Connecting mm_page_free Connecting mm_wait_on_page_end Connecting mm_wait_on_page_start Connecting rcu_classic_callback Connecting rcu_classic_call_rcu Connecting rcu_classic_call_rcu_bh Connecting statedump_softirq_vec Connecting high-rate markers to tap Connecting fs_buffer_wait_end Connecting fs_buffer_wait_start Connecting fs_close Connecting fs_exec Connecting fs_ioctl Connecting fs_llseek Connecting fs_lseek Connecting fs_open Connecting fs_pollfd Connecting fs_pread64 Connecting fs_pwrite64 Connecting fs_read Connecting fs_readv Connecting fs_select Connecting fs_write Connecting fs_writev Connecting input_event Connecting ipc_msg_create Connecting ipc_sem_create Connecting ipc_shm_create Connecting kernel_arch_ipc_call Connecting kernel_arch_kthread_create Connecting kernel_arch_syscall_entry Connecting kernel_arch_syscall_exit Connecting kernel_arch_trap_entry Connecting kernel_arch_trap_exit Connecting kernel_irq_entry Connecting kernel_irq_exit Connecting kernel_kthread_stop Connecting kernel_kthread_stop_ret Connecting kernel_module_free Connecting kernel_module_load Connecting kernel_printk Connecting kernel_process_exit Connecting kernel_process_fork Connecting kernel_process_free Connecting kernel_process_wait Connecting kernel_sched_migrate_task Connecting kernel_sched_schedule Connecting kernel_sched_try_wakeup Connecting kernel_sched_wait_task Connecting kernel_sched_wakeup_new_task Connecting kernel_send_signal Connecting kernel_softirq_entry Connecting kernel_softirq_exit Connecting kernel_softirq_raise Connecting kernel_tasklet_high_entry Connecting kernel_tasklet_high_exit Connecting kernel_tasklet_low_entry Connecting kernel_tasklet_low_exit Connecting kernel_timer_itimer_expired Connecting kernel_timer_itimer_set Connecting kernel_timer_set Connecting kernel_timer_timeout Connecting kernel_timer_update_time Connecting kernel_vprintk Connecting list_file_descriptor Connecting list_interrupt Connecting list_module Connecting list_network_ip_interface Connecting list_network_ipv4_interface Connecting list_process_state Connecting list_statedump_end Connecting list_vm_map Connecting mm_handle_fault_entry Connecting mm_handle_fault_exit Connecting mm_huge_page_alloc Connecting mm_huge_page_free Connecting mm_page_alloc Connecting mm_page_free Connecting mm_wait_on_page_end Connecting mm_wait_on_page_start Connecting rcu_classic_callback Connecting rcu_classic_call_rcu Connecting rcu_classic_call_rcu_bh Connecting statedump_softirq_vec ----------------------------------------------------------------------------- [Bryan] It's that correct result? ----------------------------------------------------------------------------- # cat /proc/ltt marker: kernel_arch_kthread_create format: "pid %ld fn %p" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_arch_syscall_exit format: " " state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_arch_syscall_entry format: "syscall_id %d ip #p%ld" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_arch_ipc_call format: "call %u first %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_arch_trap_exit format: " " state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_arch_trap_entry format: "trap_id %ld ip #p%ld" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: statedump_softirq_vec format: "id %d address %p symbol %s" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: list_module format: "name %s state %d refcount %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: core_marker_format format: "name %s format %s" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: core_marker_format format: "name %s format %s" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: list_statedump_end format: " " state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: list_process_state format: "pid %d parent_pid %d name %s type %d mode %d submode %d status %d tgid %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: list_file_descriptor format: "filename %s pid %d fd %u" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: list_vm_map format: "pid %d start %lu end %lu flags %lu pgoff %lu inode %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: list_interrupt format: "name %s action %s irq_id %u" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: list_network_ip_interface format: "name %s address #4u%lu up %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: list_network_ipv4_interface format: "name %s address #4u%lu up %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: core_marker_id format: "name %s id %hu int #1u%zu long #1u%zu pointer #1u%zu size_t #1u%zu alignment #1u%u" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: core_marker_id format: "name %s id %hu int #1u%zu long #1u%zu pointer #1u%zu size_t #1u%zu alignment #1u%u" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: core_marker_format format: "name %s format %s" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: core_marker_id format: "name %s id %hu int #1u%zu long #1u%zu pointer #1u%zu size_t #1u%zu alignment #1u%u" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: core_marker_id format: "name %s id %hu int #1u%zu long #1u%zu pointer #1u%zu size_t #1u%zu alignment #1u%u" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_module_load format: "name %s" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_module_free format: "name %s" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_vprintk format: "loglevel %c string %s ip %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_printk format: "ip %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_timer_timeout format: "pid %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_timer_update_time format: "jiffies #8u%llu xtime_sec %ld xtime_nsec %ld walltomonotonic_sec %ld walltomonotonic_nsec %ld" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_timer_set format: "expires %lu function %p data %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_timer_itimer_set format: "which %d interval_sec %ld interval_usec %ld value_sec %ld value_usec %ld" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_timer_itimer_expired format: "pid %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_process_fork format: "parent_pid %d child_pid %d child_tgid %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_process_wait format: "pid %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_process_exit format: "pid %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_process_free format: "pid %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_send_signal format: "pid %d signal %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_sched_migrate_task format: "pid %d state %ld dest_cpu %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_sched_schedule format: "prev_pid %d next_pid %d prev_state %ld" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_sched_wakeup_new_task format: "pid %d state %ld cpu_id %u" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_sched_try_wakeup format: "pid %d state %ld cpu_id %u" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_sched_wait_task format: "pid %d state %ld" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_kthread_stop_ret format: "ret %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_kthread_stop format: "pid %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_tasklet_high_exit format: "func %p data %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_tasklet_high_entry format: "func %p data %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_tasklet_low_exit format: "func %p data %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_tasklet_low_entry format: "func %p data %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_softirq_raise format: "softirq_id %u" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_softirq_exit format: "softirq_id %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_softirq_entry format: "softirq_id %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_irq_exit format: "handled #1u%u" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: kernel_irq_entry format: "irq_id %u kernel_mode %u ip %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: mm_page_alloc format: "order %u pfn %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: mm_page_free format: "order %u pfn %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: mm_handle_fault_exit format: "res %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: mm_handle_fault_entry format: "address %lu ip #p%ld write_access %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: mm_huge_page_alloc format: "pfn %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: mm_huge_page_free format: "pfn %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: mm_wait_on_page_end format: "pfn %lu bit_nr %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: mm_wait_on_page_start format: "pfn %lu bit_nr %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: fs_pollfd format: "fd %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: fs_select format: "fd %d timeout #8d%lld" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: fs_writev format: "fd %lu vlen %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: fs_readv format: "fd %lu vlen %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: fs_pwrite64 format: "fd %u count %zu pos %llu" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: fs_pread64 format: "fd %u count %zu pos %llu" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: fs_write format: "fd %u count %zu" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: fs_read format: "fd %u count %zu" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: fs_llseek format: "fd %u offset %lld origin %u" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: fs_lseek format: "fd %u offset %ld origin %u" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: fs_close format: "fd %u" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: fs_open format: "fd %d filename %s" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: fs_ioctl format: "fd %u cmd %u arg %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: fs_exec format: "filename %s" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: fs_buffer_wait_end format: "bh %p" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: fs_buffer_wait_start format: "bh %p" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: ipc_shm_create format: "id %ld flags %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: ipc_sem_create format: "id %ld flags %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: ipc_msg_create format: "id %ld flags %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: lockdep_lock_release format: "retaddr 0x%lX lock %p nested %d" state: 0 call: 0xc0093374 probe single : 0xc00920ec marker: lockdep_lock_acquire format: "retaddr 0x%lX subclass %u lock %p trylock %d read %d hardirqs_off %d" state: 0 call: 0xc0093374 probe single : 0xc00920ec marker: lockdep_softirqs_off format: "retaddr 0x%lX" state: 0 call: 0xc0093374 probe single : 0xc00920ec marker: lockdep_softirqs_on format: "retaddr 0x%lX" state: 0 call: 0xc0093374 probe single : 0xc00920ec marker: lockdep_hardirqs_off format: "retaddr 0x%lX" state: 0 call: 0xc0093374 probe single : 0xc00920ec marker: lockdep_hardirqs_on format: "retaddr 0x%lX" state: 0 call: 0xc0093374 probe single : 0xc00920ec marker: rcu_classic_call_rcu_bh format: "func %p ip 0x%lX" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: rcu_classic_call_rcu format: "func %p ip 0x%lX" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: rcu_classic_callback format: "func %p" state: 1 call: 0xc0093374 probe single : 0xc015d504 marker: input_event format: "type %u code %u value %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 ----------------------------------------------------------------------------- [Bryan] But the /proc/ltt is still same, it's that correct? What's the expected result? ----------------------------------------------------------------------------- # df /dev: 39564K total, 0K used, 39564K available (block size 4096) /sqlite_stmt_journals: 4096K total, 0K used, 4096K available (block size 4096) /system: 72704K total, 71680K used, 1024K available (block size 4096) /data: 66816K total, 37812K used, 29004K available (block size 4096) /cache: 67584K total, 1156K used, 66428K available (block size 4096) /sdcard: 991708K total, 147040K used, 844668K available (block size 4096) ----------------------------------------------------------------------------- [Bryan] There is 29004K available for trace log file /data/ltt/trace1. Is that enough? ----------------------------------------------------------------------------- # cat /proc/meminfo MemTotal: 79128 kB MemFree: 2440 kB Buffers: 36 kB Cached: 29872 kB SwapCached: 0 kB Active: 61384 kB Inactive: 5052 kB SwapTotal: 0 kB SwapFree: 0 kB Dirty: 0 kB Writeback: 0 kB AnonPages: 36552 kB Mapped: 24888 kB Slab: 3856 kB SReclaimable: 904 kB SUnreclaim: 2952 kB PageTables: 3660 kB NFS_Unstable: 0 kB Bounce: 0 kB WritebackTmp: 0 kB CommitLimit: 39564 kB Committed_AS: 1032216 kB VmallocTotal: 409600 kB VmallocUsed: 52908 kB VmallocChunk: 332796 kB ----------------------------------------------------------------------------- [Bryan] There is 2440 kB free memory. Is that enough? ----------------------------------------------------------------------------- # lttctl -n trace1 -d -l /sys/kernel/debug/ltt -t /data/ltt/trace1 Linux Trace Toolkit Trace Control 0.55-16102008 Controlling trace : trace1 Linux Trace Toolkit Trace Daemon 0.55-16102008 Reading from debugfs directory : /sys/kernel/debug/ltt/trace1 Writing to trace directory : /data/ltt/trace1 ----------------------------------------------------------------------------- [Bryan] do some tests on target device, then check the channel and trace ----------------------------------------------------------------------------- # ls -l /sys/kernel/debug/ltt/trace1 -r-------- root root 0 1980-01-06 00:10 cpu_0 drwxr-xr-x root root 1980-01-06 00:10 control # ls -l /sys/kernel/debug/ltt/trace1/control -r-------- root root 0 1980-01-06 00:10 network_0 -r-------- root root 0 1980-01-06 00:10 modules_0 -r-------- root root 0 1980-01-06 00:10 processes_0 -r-------- root root 0 1980-01-06 00:10 interrupts_0 -r-------- root root 0 1980-01-06 00:10 metadata_0 # ls -l /data/ltt/trace1 drwxrwxrwx root root 1980-01-06 00:10 control -rwxrwxrwx root root 29949952 1980-01-06 00:10 cpu_0 # ls -l /data/ltt/trace1/control -rwxrwxrwx root root 0 1980-01-06 00:10 metadata_0 -rwxrwxrwx root root 0 1980-01-06 00:10 interrupts_0 -rwxrwxrwx root root 0 1980-01-06 00:10 processes_0 -rwxrwxrwx root root 0 1980-01-06 00:10 modules_0 -rwxrwxrwx root root 0 1980-01-06 00:10 network_0 ----------------------------------------------------------------------------- [Bryan] It seems that the capacity has been exhausted by cpu_0, because 29004K available. It's that correct that cpu_0 will grow to so huge (29M)? ----------------------------------------------------------------------------- # lttctl -n trace1 -d -R Linux Trace Toolkit Trace Control 0.55-16102008 Controlling trace : trace1 ^Z[1] + Stopped lttctl -n trace1 -d -R ----------------------------------------------------------------------------- [Bryan] hang in here...I have to break it by Ctrl+Z, What's problem? ----------------------------------------------------------------------------- # ls -l /data/ltt/trace1/ drwxrwxrwx root root 1980-01-06 00:10 control -rwxrwxrwx root root 29949952 1980-01-06 00:10 cpu_0 # ls -l /data/ltt/trace1/control -rwxrwxrwx root root 0 1980-01-06 00:10 metadata_0 -rwxrwxrwx root root 0 1980-01-06 00:10 interrupts_0 -rwxrwxrwx root root 0 1980-01-06 00:10 processes_0 -rwxrwxrwx root root 0 1980-01-06 00:10 modules_0 -rwxrwxrwx root root 0 1980-01-06 00:10 network_0 # ls -l /sys/kernel/debug/ltt/trace1 -r-------- root root 0 1980-01-06 00:10 cpu_0 drwxr-xr-x root root 1980-01-06 00:10 control # ls -l /sys/kernel/debug/ltt/trace1/control -r-------- root root 0 1980-01-06 00:10 network_0 -r-------- root root 0 1980-01-06 00:10 modules_0 -r-------- root root 0 1980-01-06 00:10 processes_0 -r-------- root root 0 1980-01-06 00:10 interrupts_0 -r-------- root root 0 1980-01-06 00:10 metadata_0 # df /dev: 39564K total, 0K used, 39564K available (block size 4096) /sqlite_stmt_journals: 4096K total, 0K used, 4096K available (block size 4096) /system: 72704K total, 71680K used, 1024K available (block size 4096) /data: 66816K total, 66732K used, 84K available (block size 4096) /cache: 67584K total, 1156K used, 66428K available (block size 4096) /sdcard: 991708K total, 147040K used, 844668K available (block size 4096) ----------------------------------------------------------------------------- [Bryan] I think the trace is incomplete. ----------------------------------------------------------------------------- Please help me! Thank you very much! Best regards, Bryan Hsieh SPEC, FIH Co., Ltd. Tel:+886-3-270-0000 #2723 Fax:+886-3-270-0001 -----Original Message----- From: Mathieu Desnoyers [mailto:compudj@krystal.dyndns.org] Sent: Friday, July 31, 2009 11:43 PM To: Bryan Hsieh Cc: 'Ajit R'; 'ltt-dev at lists.casi.polymtl.ca'; Tommy Peng Subject: Re: [ltt-dev] Some LTTng 0.43 issue on ARM * Bryan Hsieh (bryanhsieh at fihspec.com) wrote: > Hi Ajit, > > I appreciate your reply! > But there is a problem here. > > My installation of Busybox is 1.12.4. It sounds like either awk or sed or not built in your busybox installation. Can you provide the output of cat /proc/ltt on your target system ? Check the content of ltt-armall to see which external programs it needs. Also, if you built the lttng probes as modules, you may need to load them explicitely. Mathieu > > I follow your suggestion to test on x86 Desktop ARM simulator, the message said that can't find /proc/ltt. I think it should be, because I didn't patch to ARM simulator Linux kernel. > > [sbox-ARM: ~/ltt-control-0.55-16102008/lttctl] > ltt-armall > Connecting all markers > cat: /proc/ltt: No such file or directory > Connecting high-rate markers to tap > cat: /proc/ltt: No such file or directory > > It should be work on my target device, but it seems not. > I can't find /ltt under /proc > Maybe it's root cause, but why there is no /ltt/proc? And why there is no " cat: /proc/ltt: No such file or directory" message in my target device during running ltt-armall? > > BTW, I mount debugfs in init.c > Mount("debugfs", "/sys/kernel/debug", "debugfs", 0, NULL); > It should not be a problem. > > -----------------------------------------------------------------------------------------------------------Another problem: > I command as following steps. > # lttctl -n trace1 -q > Linux Trace Toolkit Trace Control 0.55-16102008 > Controlling trace : trace1 > > # lttctl -n trace1 -r > Linux Trace Toolkit Trace Control 0.55-16102008 > Controlling trace : trace1 > > >>> Android shell hang in here, I have to press Ctrl+Z to force stopping - lttctl -n trace1 -r .Why? Is anything wrong? > # ls -l ../tmp/trace1 > drwxrwxrwx root root 1980-01-07 06:37 control > -rwxrwxrwx root root 1048576 1980-01-07 06:37 cpu_0 > # ls -l ../tmp/trace1/control > -rwxrwxrwx root root 0 1980-01-07 06:37 metadata_0 > -rwxrwxrwx root root 0 1980-01-07 06:37 interrupts_0 > -rwxrwxrwx root root 10240 1980-01-07 06:37 processes_0 > -rwxrwxrwx root root 65536 1980-01-07 06:37 modules_0 > -rwxrwxrwx root root 65536 1980-01-07 06:37 network_0 > > The Channel in /sys/kernel/debug/ltt/trace1 is clear. Does it cause by destroy channel command? > > The metadata_0 and interrupts_0 are empty, Is that correct? I've done some key press, the interrupts_o shouldn't be empt. > > And those files almost filled by 0 except for the short header I can't make sure, it looks like incomplete. > > I hope you can help me to solve these issues, I am appreciate you so much.. > > Best regards, > Bryan Hsieh > > SPEC, FIH Co., Ltd. > Tel:+886-3-270-0000 #2723 > Fax:+886-3-270-0001 > > > -----Original Message----- > From: Ajit R [mailto:ajitrzid@yahoo.com] > Sent: Friday, July 31, 2009 6:17 PM > To: 'ltt-dev at lists.casi.polymtl.ca'; Bryan Hsieh > Cc: Tommy Peng > Subject: Re: [ltt-dev] Some LTTng 0.43 issue on ARM > > > Hi Bryan, > My replies with the [Ajit] tag. > LTT Gods correct me if I am wrong. > > Thanks > Ajit > > --- On Fri, 7/31/09, Bryan Hsieh <bryanhsieh at fihspec.com> wrote: > > > From: Bryan Hsieh <bryanhsieh@fihspec.com> > > Subject: [ltt-dev] Some LTTng 0.43 issue on ARM > > To: "'ltt-dev at lists.casi.polymtl.ca'" <ltt-dev at lists.casi.polymtl.ca> > > Cc: "Tommy Peng" <tommypeng at fihspec.com> > > Date: Friday, July 31, 2009, 12:19 PM > > > Hi Mathieu, > > all, > > > > > > > > > > I've ported > > LTTng 0.43 on Android successfully, > > but I am wondering if the result is correct, please help me > > to verify, thank > > you very much! > > > > > > > > > > My Traced > > machine: > > > > ARM11 (ARM1136J(F)-S) / Linux > > 2.6..27 > > (Android 1.5) > > > > My > > visualization machine: > > > > X86 / Sun xVM Virtual Box (Unbuntu > > 8.10) > > > > > > > > LTTng: 0.43 > > > > LTTV: > > 0.11.2 > > > > LTT Control: 0.55 > > > > > > > > > > I > > connect to Android > > shell by ADB tool after Android bring up, then I command as > > following step, the > > command line options are different with LTTng manual, > > because the example in manual > > is for old version. > > > > > > > > > > # ltt-armall > > > [ Ajit ] > ltt-armall is a shell script. It connects + arms the markers in the > kernel. So depending on your kernel + LTT version it would write some > "text" into the /proc/ltt directory or a "1" into files in > the debugfs directory. > Make sure, the binaries for running the script are available in your target environment. For eg. awk, grep, sed etc. Busybox versions might have their limitations. If they do not work, you could do equivalent things on your Desktop and then write a simple script, doing the same on your target. > Once you run "ltt-armall", you should see some debug statements in your "dmesg" like "connect" or "arm" etc. > > > > > Connecting all > > markers > > > > Connecting high-rate > > markers to tap > > > > # lttctl > > -n trace1 -d -l /sys/kernel/debug/ltt -t > > /system/traces/trace1 > > > [Ajit] > Not sure if your "-l" option is right. you have to point it to the place where you mounted your debugfs. > > > > > Linux Trace Toolkit > > Trace Control 0.55-16102008 > > > > > > > > > > Controlling trace : > > trace1 > > > > > > > > > > Linux Trace Toolkit > > Trace Daemon 0.55-16102008 > > > > > > > > > > Reading from debugfs > > directory : /sys/kernel/debug/ltt/trace1 > > > > > > Writing to trace > > directory : /system/traces/trace1 > > > > > > > > > > It > > seems the LTT control > > and daemon work fine. I run some scenarios for 1 minute on > > Android, but the trace > > files are still empty (all files length are > > 0) > > > > What's > > wrong? Is > > there any step I missed? Or the LTT control and daemon > > don't work? > > > > > > > > > > # ls > > -l ../../sys/kernel/debug/ltt/ > > > > > > > > drwxr-xr-x > > root > > root > > 1980-01-07 01:13 trace1 > > > > # ls > > -l > > ../../sys/kernel/debug/ltt/trace1 > > > > > > -r-------- > > root > > root > > 0 > > 1980-01-07 01:13 cpu_0 > > > > [Ajit] > If you have connected and armed it right, you should see the cpu_0 > file increasing in size. After sometime, disconnect by running the > ltt-control -n whatever-name -R > and then you will see files in the control directory (network, process, etc ) grown in size. Tar and move the trace directory to your host and do the needful. > > > > drwxr-xr-x > > root > > root > > 1980-01-07 01:13 control > > > > # ls > > -l > > ../../sys/kernel/debug/ltt/trace1/control > > > > > > -r-------- > > root > > root > > 0 > > 1980-01-07 01:13 network_0 > > > > -r-------- > > root > > root > > 0 > > 1980-01-07 01:13 modules_0 > > > > -r-------- > > root > > root > > 0 > > 1980-01-07 01:13 processes_0 > > > > -r-------- > > root > > root > > 0 > > 1980-01-07 01:13 interrupts_0 > > > > -r-------- > > root > > root > > 0 1980-01-07 01:13 metadata_0 > > > > # ls -l > > .../traces/ > > > > > > > > drwxrwxrwx > > root > > root > > 1980-01-07 01:13 trace1 > > > > # ls -l > > .../traces/trace1 > > > > drwxrwxrwx > > root > > root > > 1980-01-07 01:13 control > > > > -rwxrwxrwx > > root > > root > > 0 > > 1980-01-07 01:13 cpu_0 > > > > # ls -l > > .../traces/trace1/control > > > > > > > > -rwxrwxrwx > > root > > root > > 0 > > 1980-01-07 01:13 metadata_0 > > > > -rwxrwxrwx > > root > > root > > 0 > > 1980-01-07 01:13 interrupts_0 > > > > -rwxrwxrwx > > root > > root > > 0 > > 1980-01-07 01:13 processes_0 > > > > -rwxrwxrwx > > root > > root > > 0 > > 1980-01-07 01:13 modules_0 > > > > -rwxrwxrwx > > root > > root > > 0 > > 1980-01-07 01:13 network_0 > > > > > > > > > > BTW, > > I would like to > > know what is the channel concept for LTTng? What's > > the relationship between > > trace and channel? What's the normal, flight recorder > > and hybrid mode? > > > > Which > > files should I get > > for LTTV on another visualization > > machine? > > > > So far, I > > can't find related document or the > > information isn't enough for me. > > > > > > > > > > Thanks for your > > help! > > > > > > > > > > Best > > regards, > > > > Bryan Hsieh > > > > SPEC, FIH > > Co., Ltd. > > > > Tel:+886-3-270-0000 #2723 > > > > Fax:+886-3-270-0001 > > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > ltt-dev mailing list > > ltt-dev at lists.casi.polymtl.ca > > http://lists.casi.polymtl.ca/cgi-bin/mailman/listinfo/ltt-dev > > > > > > This e-mail message together with any attachments thereto (if any) is confidential, protected under an enforceable non-disclosure agreement, > intended only for the use of the named recipient(s) above and may contain information that is privileged, belonging to professional work products > or exempt from disclosure under applicable laws. Any unauthorized review, use, copying, disclosure, or distribution of any information contained > in or attached to this transmission is STRICTLY PROHIBITED and may be against the laws. If you have received this message in error, or are not > the named recipient(s), please immediately notify the sender by e-mail or telephone at IT support hotline 886-3-2700000 EXT 1333 and delete this > e-mail message and any attached documentation from your computer. Receipt by anyone other than the intended recipient(s) is not a waiver of any > attorney-client or work product privilege. Thank you! > > > _______________________________________________ > ltt-dev mailing list > ltt-dev at lists.casi.polymtl.ca > http://lists.casi.polymtl.ca/cgi-bin/mailman/listinfo/ltt-dev > -- Mathieu Desnoyers OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F BA06 3F25 A8FE 3BAE 9A68 This e-mail message together with any attachments thereto (if any) is confidential, protected under an enforceable non-disclosure agreement, intended only for the use of the named recipient(s) above and may contain information that is privileged, belonging to professional work products or exempt from disclosure under applicable laws. Any unauthorized review, use, copying, disclosure, or distribution of any information contained in or attached to this transmission is STRICTLY PROHIBITED and may be against the laws. If you have received this message in error, or are not the named recipient(s), please immediately notify the sender by e-mail or telephone at IT support hotline 886-3-2700000 EXT 1333 and delete this e-mail message and any attached documentation from your computer. Receipt by anyone other than the intended recipient(s) is not a waiver of any attorney-client or work product privilege. Thank you! -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: busybox version and command query.txt URL: <http://lists.casi.polymtl.ca/pipermail/lttng-dev/attachments/20090804/39917614/attachment-0009.txt> -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: LTTng Execution Log.txt URL: <http://lists.casi.polymtl.ca/pipermail/lttng-dev/attachments/20090804/39917614/attachment-0010.txt> -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: LTTng Kernel Config.txt URL: <http://lists.casi.polymtl.ca/pipermail/lttng-dev/attachments/20090804/39917614/attachment-0011.txt> ^ permalink raw reply [flat|nested] 9+ messages in thread
* [ltt-dev] Some LTTng 0.43 issue on ARM 2009-08-04 8:28 ` [ltt-dev] Some LTTng 0.43 issue on ARM Bryan Hsieh 2009-08-04 8:28 ` Bryan Hsieh 2009-08-04 8:28 ` Bryan Hsieh @ 2009-08-04 13:18 ` Mathieu Desnoyers 2009-08-06 2:52 ` Bryan Hsieh 2 siblings, 1 reply; 9+ messages in thread From: Mathieu Desnoyers @ 2009-08-04 13:18 UTC (permalink / raw) * Bryan Hsieh (bryanhsieh at fihspec.com) wrote: > Hi Mathieu, All > > Please check my comment which marked by [Bryan], Thanks! > ----------------------------------------------------------------------------- > [Bryan] > All the related LTT functions are build-in not build as modules, Please check attachment- LTTng Kernel Config..txt, I am not sure if Kernel->user space relay support (formerly relayfs) need to build-in. > Is there any item I missed? > ----------------------------------------------------------------------------- > [ ] Kernel->user space relay support (formerly relayfs) LTTng does not need relayfs anymore. So this is ok. > [*] Activate tracepoints > [*] Activate markers > Linux Trace Toolkit ---> > <*> Compile lttng tracing probes > [*] LTTng fine-grained timestamping > [*] Linux Trace Toolkit Instrumentation Support > <*> Linux Trace Toolkit High-speed Lockless Data Relay > <*> Linux Trace Toolkit Lock-Protected Data Relay > [*] Debug check for random access in ltt relay buffers > <*> Linux Trace Toolkit Serializer > <*> Linux Trace Toolkit Marker Control > <*> Linux Trace Toolkit Tracer > [*] Align Linux Trace Toolkit Traces > [*] Add event size field to LTT events for tracer debugging > [*] Support trace extraction from crash dump > <*> Linux Trace Toolkit Netlink Controller > <*> Linux Trace Toolkit State Dump > ----------------------------------------------------------------------------- > [Bryan] > I am sure that the external utilities are build-in busybox, please check the attachment - " busybox version and command query.txt". > > Finally, I figure out why the ltt-armall doesn't work correctly in my target system, root cause is the output pipeline will be empty after grep twice, even I use a temp file to avoid connecting two grep commands directly, it's still empty. So I copy the /proc/ltt to my local desktop to do ltt-armall script, then generate the /proc/ltt which been filtered by following command. > MARKERS=`cat /proc/ltt|grep -v %k|awk '{print $2}'|sort -u|grep -v ^core_|grep -v ^locking_|grep -v ^lockdep_` > I rename it as ltt_filter and copy to my target system path /system/bin/, then modify ltt-armall by following modification and run it. > MARKERS=`cat /system/bin/ltt_filter` > > The execution result on screen: > # ltt-armall-1 > Connecting all markers > Connecting fs_buffer_wait_end > Connecting fs_buffer_wait_start > Connecting fs_close > Connecting fs_exec > Connecting fs_ioctl > Connecting fs_llseek > Connecting fs_lseek > Connecting fs_open > Connecting fs_pollfd > Connecting fs_pread64 > Connecting fs_pwrite64 > Connecting fs_read > Connecting fs_readv > Connecting fs_select > Connecting fs_write > Connecting fs_writev > Connecting input_event > Connecting ipc_msg_create > Connecting ipc_sem_create > Connecting ipc_shm_create > Connecting kernel_arch_ipc_call > Connecting kernel_arch_kthread_create > Connecting kernel_arch_syscall_entry > Connecting kernel_arch_syscall_exit > Connecting kernel_arch_trap_entry > Connecting kernel_arch_trap_exit > Connecting kernel_irq_entry > Connecting kernel_irq_exit > Connecting kernel_kthread_stop > Connecting kernel_kthread_stop_ret > Connecting kernel_module_free > Connecting kernel_module_load > Connecting kernel_printk > Connecting kernel_process_exit > Connecting kernel_process_fork > Connecting kernel_process_free > Connecting kernel_process_wait > Connecting kernel_sched_migrate_task > Connecting kernel_sched_schedule > Connecting kernel_sched_try_wakeup > Connecting kernel_sched_wait_task > Connecting kernel_sched_wakeup_new_task > Connecting kernel_send_signal > Connecting kernel_softirq_entry > Connecting kernel_softirq_exit > Connecting kernel_softirq_raise > Connecting kernel_tasklet_high_entry > Connecting kernel_tasklet_high_exit > Connecting kernel_tasklet_low_entry > Connecting kernel_tasklet_low_exit > Connecting kernel_timer_itimer_expired > Connecting kernel_timer_itimer_set > Connecting kernel_timer_set > Connecting kernel_timer_timeout > Connecting kernel_timer_update_time > Connecting kernel_vprintk > Connecting list_file_descriptor > Connecting list_interrupt > Connecting list_module > Connecting list_network_ip_interface > Connecting list_network_ipv4_interface > Connecting list_process_state > Connecting list_statedump_end > Connecting list_vm_map > Connecting mm_handle_fault_entry > Connecting mm_handle_fault_exit > Connecting mm_huge_page_alloc > Connecting mm_huge_page_free > Connecting mm_page_alloc > Connecting mm_page_free > Connecting mm_wait_on_page_end > Connecting mm_wait_on_page_start > Connecting rcu_classic_callback > Connecting rcu_classic_call_rcu > Connecting rcu_classic_call_rcu_bh > Connecting statedump_softirq_vec > Connecting high-rate markers to tap > Connecting fs_buffer_wait_end > Connecting fs_buffer_wait_start > Connecting fs_close > Connecting fs_exec > Connecting fs_ioctl > Connecting fs_llseek > Connecting fs_lseek > Connecting fs_open > Connecting fs_pollfd > Connecting fs_pread64 > Connecting fs_pwrite64 > Connecting fs_read > Connecting fs_readv > Connecting fs_select > Connecting fs_write > Connecting fs_writev > Connecting input_event > Connecting ipc_msg_create > Connecting ipc_sem_create > Connecting ipc_shm_create > Connecting kernel_arch_ipc_call > Connecting kernel_arch_kthread_create > Connecting kernel_arch_syscall_entry > Connecting kernel_arch_syscall_exit > Connecting kernel_arch_trap_entry > Connecting kernel_arch_trap_exit > Connecting kernel_irq_entry > Connecting kernel_irq_exit > Connecting kernel_kthread_stop > Connecting kernel_kthread_stop_ret > Connecting kernel_module_free > Connecting kernel_module_load > Connecting kernel_printk > Connecting kernel_process_exit > Connecting kernel_process_fork > Connecting kernel_process_free > Connecting kernel_process_wait > Connecting kernel_sched_migrate_task > Connecting kernel_sched_schedule > Connecting kernel_sched_try_wakeup > Connecting kernel_sched_wait_task > Connecting kernel_sched_wakeup_new_task > Connecting kernel_send_signal > Connecting kernel_softirq_entry > Connecting kernel_softirq_exit > Connecting kernel_softirq_raise > Connecting kernel_tasklet_high_entry > Connecting kernel_tasklet_high_exit > Connecting kernel_tasklet_low_entry > Connecting kernel_tasklet_low_exit > Connecting kernel_timer_itimer_expired > Connecting kernel_timer_itimer_set > Connecting kernel_timer_set > Connecting kernel_timer_timeout > Connecting kernel_timer_update_time > Connecting kernel_vprintk > Connecting list_file_descriptor > Connecting list_interrupt > Connecting list_module > Connecting list_network_ip_interface > Connecting list_network_ipv4_interface > Connecting list_process_state > Connecting list_statedump_end > Connecting list_vm_map > Connecting mm_handle_fault_entry > Connecting mm_handle_fault_exit > Connecting mm_huge_page_alloc > Connecting mm_huge_page_free > Connecting mm_page_alloc > Connecting mm_page_free > Connecting mm_wait_on_page_end > Connecting mm_wait_on_page_start > Connecting rcu_classic_callback > Connecting rcu_classic_call_rcu > Connecting rcu_classic_call_rcu_bh > Connecting statedump_softirq_vec > ----------------------------------------------------------------------------- > [Bryan] It's that correct result? Yes, this output is correct. So you probably need to recompile busybox with awk and/or sed and/or sort support..... > ----------------------------------------------------------------------------- > # cat /proc/ltt > marker: kernel_arch_kthread_create format: "pid %ld fn %p" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: kernel_arch_syscall_exit format: " " state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: kernel_arch_syscall_entry format: "syscall_id %d ip #p%ld" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: kernel_arch_ipc_call format: "call %u first %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: kernel_arch_trap_exit format: " " state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: kernel_arch_trap_entry format: "trap_id %ld ip #p%ld" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: statedump_softirq_vec format: "id %d address %p symbol %s" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: list_module format: "name %s state %d refcount %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: core_marker_format format: "name %s format %s" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: core_marker_format format: "name %s format %s" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: list_statedump_end format: " " state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: list_process_state format: "pid %d parent_pid %d name %s type %d mode %d submode %d status %d tgid %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: list_file_descriptor format: "filename %s pid %d fd %u" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: list_vm_map format: "pid %d start %lu end %lu flags %lu pgoff %lu inode %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: list_interrupt format: "name %s action %s irq_id %u" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: list_network_ip_interface format: "name %s address #4u%lu up %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: list_network_ipv4_interface format: "name %s address #4u%lu up %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: core_marker_id format: "name %s id %hu int #1u%zu long #1u%zu pointer #1u%zu size_t #1u%zu alignment #1u%u" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: core_marker_id format: "name %s id %hu int #1u%zu long #1u%zu pointer #1u%zu size_t #1u%zu alignment #1u%u" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: core_marker_format format: "name %s format %s" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: core_marker_id format: "name %s id %hu int #1u%zu long #1u%zu pointer #1u%zu size_t #1u%zu alignment #1u%u" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: core_marker_id format: "name %s id %hu int #1u%zu long #1u%zu pointer #1u%zu size_t #1u%zu alignment #1u%u" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: kernel_module_load format: "name %s" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: kernel_module_free format: "name %s" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: kernel_vprintk format: "loglevel %c string %s ip %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: kernel_printk format: "ip %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: kernel_timer_timeout format: "pid %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: kernel_timer_update_time format: "jiffies #8u%llu xtime_sec %ld xtime_nsec %ld walltomonotonic_sec %ld walltomonotonic_nsec %ld" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: kernel_timer_set format: "expires %lu function %p data %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: kernel_timer_itimer_set format: "which %d interval_sec %ld interval_usec %ld value_sec %ld value_usec %ld" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: kernel_timer_itimer_expired format: "pid %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: kernel_process_fork format: "parent_pid %d child_pid %d child_tgid %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: kernel_process_wait format: "pid %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: kernel_process_exit format: "pid %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: kernel_process_free format: "pid %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: kernel_send_signal format: "pid %d signal %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: kernel_sched_migrate_task format: "pid %d state %ld dest_cpu %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: kernel_sched_schedule format: "prev_pid %d next_pid %d prev_state %ld" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: kernel_sched_wakeup_new_task format: "pid %d state %ld cpu_id %u" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: kernel_sched_try_wakeup format: "pid %d state %ld cpu_id %u" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: kernel_sched_wait_task format: "pid %d state %ld" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: kernel_kthread_stop_ret format: "ret %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: kernel_kthread_stop format: "pid %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: kernel_tasklet_high_exit format: "func %p data %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: kernel_tasklet_high_entry format: "func %p data %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: kernel_tasklet_low_exit format: "func %p data %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: kernel_tasklet_low_entry format: "func %p data %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: kernel_softirq_raise format: "softirq_id %u" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: kernel_softirq_exit format: "softirq_id %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: kernel_softirq_entry format: "softirq_id %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: kernel_irq_exit format: "handled #1u%u" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: kernel_irq_entry format: "irq_id %u kernel_mode %u ip %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: mm_page_alloc format: "order %u pfn %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: mm_page_free format: "order %u pfn %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: mm_handle_fault_exit format: "res %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: mm_handle_fault_entry format: "address %lu ip #p%ld write_access %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: mm_huge_page_alloc format: "pfn %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: mm_huge_page_free format: "pfn %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: mm_wait_on_page_end format: "pfn %lu bit_nr %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: mm_wait_on_page_start format: "pfn %lu bit_nr %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: fs_pollfd format: "fd %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: fs_select format: "fd %d timeout #8d%lld" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: fs_writev format: "fd %lu vlen %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: fs_readv format: "fd %lu vlen %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: fs_pwrite64 format: "fd %u count %zu pos %llu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: fs_pread64 format: "fd %u count %zu pos %llu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: fs_write format: "fd %u count %zu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: fs_read format: "fd %u count %zu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: fs_llseek format: "fd %u offset %lld origin %u" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: fs_lseek format: "fd %u offset %ld origin %u" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: fs_close format: "fd %u" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: fs_open format: "fd %d filename %s" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: fs_ioctl format: "fd %u cmd %u arg %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: fs_exec format: "filename %s" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: fs_buffer_wait_end format: "bh %p" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: fs_buffer_wait_start format: "bh %p" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: ipc_shm_create format: "id %ld flags %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: ipc_sem_create format: "id %ld flags %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: ipc_msg_create format: "id %ld flags %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: lockdep_lock_release format: "retaddr 0x%lX lock %p nested %d" state: 0 call: 0xc0093374 probe single : 0xc00920ec > marker: lockdep_lock_acquire format: "retaddr 0x%lX subclass %u lock %p trylock %d read %d hardirqs_off %d" state: 0 call: 0xc0093374 probe single : 0xc00920ec > marker: lockdep_softirqs_off format: "retaddr 0x%lX" state: 0 call: 0xc0093374 probe single : 0xc00920ec > marker: lockdep_softirqs_on format: "retaddr 0x%lX" state: 0 call: 0xc0093374 probe single : 0xc00920ec > marker: lockdep_hardirqs_off format: "retaddr 0x%lX" state: 0 call: 0xc0093374 probe single : 0xc00920ec > marker: lockdep_hardirqs_on format: "retaddr 0x%lX" state: 0 call: 0xc0093374 probe single : 0xc00920ec > marker: rcu_classic_call_rcu_bh format: "func %p ip 0x%lX" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: rcu_classic_call_rcu format: "func %p ip 0x%lX" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: rcu_classic_callback format: "func %p" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: input_event format: "type %u code %u value %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > ----------------------------------------------------------------------------- > [Bryan] But the /proc/ltt is still same, it's that correct? What's the expected result? This output is good. busybox problem, as I said above. Try each command used by busybox to test them. > ----------------------------------------------------------------------------- > # df > /dev: 39564K total, 0K used, 39564K available (block size 4096) > /sqlite_stmt_journals: 4096K total, 0K used, 4096K available (block size 4096) > /system: 72704K total, 71680K used, 1024K available (block size 4096) > /data: 66816K total, 37812K used, 29004K available (block size 4096) > /cache: 67584K total, 1156K used, 66428K available (block size 4096) > /sdcard: 991708K total, 147040K used, 844668K available (block size 4096) > ----------------------------------------------------------------------------- > [Bryan] There is 29004K available for trace log file /data/ltt/trace1. Is that enough? No. I would save it to /sdcard. > ----------------------------------------------------------------------------- > # cat /proc/meminfo > MemTotal: 79128 kB > MemFree: 2440 kB > Buffers: 36 kB > Cached: 29872 kB > SwapCached: 0 kB > Active: 61384 kB > Inactive: 5052 kB > SwapTotal: 0 kB > SwapFree: 0 kB > Dirty: 0 kB > Writeback: 0 kB > AnonPages: 36552 kB > Mapped: 24888 kB > Slab: 3856 kB > SReclaimable: 904 kB > SUnreclaim: 2952 kB > PageTables: 3660 kB > NFS_Unstable: 0 kB > Bounce: 0 kB > WritebackTmp: 0 kB > CommitLimit: 39564 kB > Committed_AS: 1032216 kB > VmallocTotal: 409600 kB > VmallocUsed: 52908 kB > VmallocChunk: 332796 kB > ----------------------------------------------------------------------------- > [Bryan] > There is 2440 kB free memory. Is that enough? I think that yes. The page allocator will kick out Cached pages when needed. > ----------------------------------------------------------------------------- > # lttctl -n trace1 -d -l /sys/kernel/debug/ltt -t /data/ltt/trace1 > Linux Trace Toolkit Trace Control 0.55-16102008 > > Controlling trace : trace1 > > Linux Trace Toolkit Trace Daemon 0.55-16102008 > > Reading from debugfs directory : /sys/kernel/debug/ltt/trace1 > Writing to trace directory : /data/ltt/trace1 > ----------------------------------------------------------------------------- > [Bryan] do some tests on target device, then check the channel and trace > ----------------------------------------------------------------------------- > # ls -l /sys/kernel/debug/ltt/trace1 > -r-------- root root 0 1980-01-06 00:10 cpu_0 > drwxr-xr-x root root 1980-01-06 00:10 control > # ls -l /sys/kernel/debug/ltt/trace1/control > -r-------- root root 0 1980-01-06 00:10 network_0 > -r-------- root root 0 1980-01-06 00:10 modules_0 > -r-------- root root 0 1980-01-06 00:10 processes_0 > -r-------- root root 0 1980-01-06 00:10 interrupts_0 > -r-------- root root 0 1980-01-06 00:10 metadata_0 > # ls -l /data/ltt/trace1 > drwxrwxrwx root root 1980-01-06 00:10 control > -rwxrwxrwx root root 29949952 1980-01-06 00:10 cpu_0 > # ls -l /data/ltt/trace1/control > -rwxrwxrwx root root 0 1980-01-06 00:10 metadata_0 > -rwxrwxrwx root root 0 1980-01-06 00:10 interrupts_0 > -rwxrwxrwx root root 0 1980-01-06 00:10 processes_0 > -rwxrwxrwx root root 0 1980-01-06 00:10 modules_0 > -rwxrwxrwx root root 0 1980-01-06 00:10 network_0 > ----------------------------------------------------------------------------- > [Bryan] It seems that the capacity has been exhausted by cpu_0, because 29004K available. > It's that correct that cpu_0 will grow to so huge (29M)? 29M is not huge. It is small :) Especially if the system is very busy. > ----------------------------------------------------------------------------- > # lttctl -n trace1 -d -R > Linux Trace Toolkit Trace Control 0.55-16102008 > > Controlling trace : trace1 > ^Z[1] + Stopped lttctl -n trace1 -d -R > ----------------------------------------------------------------------------- > [Bryan] hang in here...I have to break it by Ctrl+Z, What's problem? It's waiting for lttd to release the trace channels, but I guess that the lack of space is causing errors, so lttd keeps running. > ----------------------------------------------------------------------------- > # ls -l /data/ltt/trace1/ > drwxrwxrwx root root 1980-01-06 00:10 control > -rwxrwxrwx root root 29949952 1980-01-06 00:10 cpu_0 > # ls -l /data/ltt/trace1/control > -rwxrwxrwx root root 0 1980-01-06 00:10 metadata_0 > -rwxrwxrwx root root 0 1980-01-06 00:10 interrupts_0 > -rwxrwxrwx root root 0 1980-01-06 00:10 processes_0 > -rwxrwxrwx root root 0 1980-01-06 00:10 modules_0 > -rwxrwxrwx root root 0 1980-01-06 00:10 network_0 > # ls -l /sys/kernel/debug/ltt/trace1 > -r-------- root root 0 1980-01-06 00:10 cpu_0 > drwxr-xr-x root root 1980-01-06 00:10 control > # ls -l /sys/kernel/debug/ltt/trace1/control > -r-------- root root 0 1980-01-06 00:10 network_0 > -r-------- root root 0 1980-01-06 00:10 modules_0 > -r-------- root root 0 1980-01-06 00:10 processes_0 > -r-------- root root 0 1980-01-06 00:10 interrupts_0 > -r-------- root root 0 1980-01-06 00:10 metadata_0 > # df > /dev: 39564K total, 0K used, 39564K available (block size 4096) > /sqlite_stmt_journals: 4096K total, 0K used, 4096K available (block size 4096) > /system: 72704K total, 71680K used, 1024K available (block size 4096) > /data: 66816K total, 66732K used, 84K available (block size 4096) > /cache: 67584K total, 1156K used, 66428K available (block size 4096) > /sdcard: 991708K total, 147040K used, 844668K available (block size 4096) > ----------------------------------------------------------------------------- > [Bryan] I think the trace is incomplete. Yes, here /data only has 84K avail. It's full. Mathieu > ----------------------------------------------------------------------------- > > Please help me! Thank you very much! > > Best regards, > Bryan Hsieh > > SPEC, FIH Co., Ltd. > Tel:+886-3-270-0000 #2723 > Fax:+886-3-270-0001 > > > -----Original Message----- > From: Mathieu Desnoyers [mailto:compudj@krystal.dyndns.org] > Sent: Friday, July 31, 2009 11:43 PM > To: Bryan Hsieh > Cc: 'Ajit R'; 'ltt-dev at lists.casi.polymtl.ca'; Tommy Peng > Subject: Re: [ltt-dev] Some LTTng 0.43 issue on ARM > > * Bryan Hsieh (bryanhsieh at fihspec.com) wrote: > > Hi Ajit, > > > > I appreciate your reply! > > But there is a problem here. > > > > My installation of Busybox is 1.12.4. > > It sounds like either awk or sed or not built in your busybox > installation. > > Can you provide the output of > > cat /proc/ltt > > on your target system ? > > Check the content of ltt-armall to see which external programs it needs. > > Also, if you built the lttng probes as modules, you may need to load > them explicitely. > > Mathieu > > > > > > I follow your suggestion to test on x86 Desktop ARM simulator, the message said that can't find /proc/ltt. I think it should be, because I didn't patch to ARM simulator Linux kernel. > > > > [sbox-ARM: ~/ltt-control-0.55-16102008/lttctl] > ltt-armall > > Connecting all markers > > cat: /proc/ltt: No such file or directory > > Connecting high-rate markers to tap > > cat: /proc/ltt: No such file or directory > > > > It should be work on my target device, but it seems not. > > I can't find /ltt under /proc > > Maybe it's root cause, but why there is no /ltt/proc? And why there is no " cat: /proc/ltt: No such file or directory" message in my target device during running ltt-armall? > > > > BTW, I mount debugfs in init.c > > Mount("debugfs", "/sys/kernel/debug", "debugfs", 0, NULL); > > It should not be a problem. > > > > -----------------------------------------------------------------------------------------------------------Another problem: > > I command as following steps. > > # lttctl -n trace1 -q > > Linux Trace Toolkit Trace Control 0.55-16102008 > > Controlling trace : trace1 > > > > # lttctl -n trace1 -r > > Linux Trace Toolkit Trace Control 0.55-16102008 > > Controlling trace : trace1 > > > > >>> Android shell hang in here, I have to press Ctrl+Z to force stopping - lttctl -n trace1 -r .Why? Is anything wrong? > > # ls -l ../tmp/trace1 > > drwxrwxrwx root root 1980-01-07 06:37 control > > -rwxrwxrwx root root 1048576 1980-01-07 06:37 cpu_0 > > # ls -l ../tmp/trace1/control > > -rwxrwxrwx root root 0 1980-01-07 06:37 metadata_0 > > -rwxrwxrwx root root 0 1980-01-07 06:37 interrupts_0 > > -rwxrwxrwx root root 10240 1980-01-07 06:37 processes_0 > > -rwxrwxrwx root root 65536 1980-01-07 06:37 modules_0 > > -rwxrwxrwx root root 65536 1980-01-07 06:37 network_0 > > > > The Channel in /sys/kernel/debug/ltt/trace1 is clear. Does it cause by destroy channel command? > > > > The metadata_0 and interrupts_0 are empty, Is that correct? I've done some key press, the interrupts_o shouldn't be empt. > > > > And those files almost filled by 0 except for the short header I can't make sure, it looks like incomplete. > > > > I hope you can help me to solve these issues, I am appreciate you so much.. > > > > Best regards, > > Bryan Hsieh > > > > SPEC, FIH Co., Ltd. > > Tel:+886-3-270-0000 #2723 > > Fax:+886-3-270-0001 > > > > > > -----Original Message----- > > From: Ajit R [mailto:ajitrzid@yahoo.com] > > Sent: Friday, July 31, 2009 6:17 PM > > To: 'ltt-dev at lists.casi.polymtl.ca'; Bryan Hsieh > > Cc: Tommy Peng > > Subject: Re: [ltt-dev] Some LTTng 0.43 issue on ARM > > > > > > Hi Bryan, > > My replies with the [Ajit] tag. > > LTT Gods correct me if I am wrong. > > > > Thanks > > Ajit > > > > --- On Fri, 7/31/09, Bryan Hsieh <bryanhsieh at fihspec.com> wrote: > > > > > From: Bryan Hsieh <bryanhsieh@fihspec.com> > > > Subject: [ltt-dev] Some LTTng 0.43 issue on ARM > > > To: "'ltt-dev at lists.casi.polymtl.ca'" <ltt-dev at lists.casi.polymtl.ca> > > > Cc: "Tommy Peng" <tommypeng at fihspec.com> > > > Date: Friday, July 31, 2009, 12:19 PM > > > > > Hi Mathieu, > > > all, > > > > > > > > > > > > > > > I've ported > > > LTTng 0.43 on Android successfully, > > > but I am wondering if the result is correct, please help me > > > to verify, thank > > > you very much! > > > > > > > > > > > > > > > My Traced > > > machine: > > > > > > ARM11 (ARM1136J(F)-S) / Linux > > > 2.6..27 > > > (Android 1.5) > > > > > > My > > > visualization machine: > > > > > > X86 / Sun xVM Virtual Box (Unbuntu > > > 8.10) > > > > > > > > > > > > LTTng: 0.43 > > > > > > LTTV: > > > 0.11.2 > > > > > > LTT Control: 0.55 > > > > > > > > > > > > > > > I > > > connect to Android > > > shell by ADB tool after Android bring up, then I command as > > > following step, the > > > command line options are different with LTTng manual, > > > because the example in manual > > > is for old version. > > > > > > > > > > > > > > > # ltt-armall > > > > > [ Ajit ] > > ltt-armall is a shell script. It connects + arms the markers in the > > kernel. So depending on your kernel + LTT version it would write some > > "text" into the /proc/ltt directory or a "1" into files in > > the debugfs directory. > > Make sure, the binaries for running the script are available in your target environment. For eg. awk, grep, sed etc. Busybox versions might have their limitations. If they do not work, you could do equivalent things on your Desktop and then write a simple script, doing the same on your target. > > Once you run "ltt-armall", you should see some debug statements in your "dmesg" like "connect" or "arm" etc. > > > > > > > > Connecting all > > > markers > > > > > > Connecting high-rate > > > markers to tap > > > > > > # lttctl > > > -n trace1 -d -l /sys/kernel/debug/ltt -t > > > /system/traces/trace1 > > > > > [Ajit] > > Not sure if your "-l" option is right. you have to point it to the place where you mounted your debugfs. > > > > > > > > Linux Trace Toolkit > > > Trace Control 0.55-16102008 > > > > > > > > > > > > > > > Controlling trace : > > > trace1 > > > > > > > > > > > > > > > Linux Trace Toolkit > > > Trace Daemon 0.55-16102008 > > > > > > > > > > > > > > > Reading from debugfs > > > directory : /sys/kernel/debug/ltt/trace1 > > > > > > > > > Writing to trace > > > directory : /system/traces/trace1 > > > > > > > > > > > > > > > It > > > seems the LTT control > > > and daemon work fine. I run some scenarios for 1 minute on > > > Android, but the trace > > > files are still empty (all files length are > > > 0) > > > > > > What's > > > wrong? Is > > > there any step I missed? Or the LTT control and daemon > > > don't work? > > > > > > > > > > > > > > > # ls > > > -l ../../sys/kernel/debug/ltt/ > > > > > > > > > > > > drwxr-xr-x > > > root > > > root > > > 1980-01-07 01:13 trace1 > > > > > > # ls > > > -l > > > ../../sys/kernel/debug/ltt/trace1 > > > > > > > > > -r-------- > > > root > > > root > > > 0 > > > 1980-01-07 01:13 cpu_0 > > > > > > > [Ajit] > > If you have connected and armed it right, you should see the cpu_0 > > file increasing in size. After sometime, disconnect by running the > > ltt-control -n whatever-name -R > > and then you will see files in the control directory (network, process, etc ) grown in size. Tar and move the trace directory to your host and do the needful. > > > > > > > drwxr-xr-x > > > root > > > root > > > 1980-01-07 01:13 control > > > > > > # ls > > > -l > > > ../../sys/kernel/debug/ltt/trace1/control > > > > > > > > > -r-------- > > > root > > > root > > > 0 > > > 1980-01-07 01:13 network_0 > > > > > > -r-------- > > > root > > > root > > > 0 > > > 1980-01-07 01:13 modules_0 > > > > > > -r-------- > > > root > > > root > > > 0 > > > 1980-01-07 01:13 processes_0 > > > > > > -r-------- > > > root > > > root > > > 0 > > > 1980-01-07 01:13 interrupts_0 > > > > > > -r-------- > > > root > > > root > > > 0 1980-01-07 01:13 metadata_0 > > > > > > # ls -l > > > .../traces/ > > > > > > > > > > > > drwxrwxrwx > > > root > > > root > > > 1980-01-07 01:13 trace1 > > > > > > # ls -l > > > .../traces/trace1 > > > > > > drwxrwxrwx > > > root > > > root > > > 1980-01-07 01:13 control > > > > > > -rwxrwxrwx > > > root > > > root > > > 0 > > > 1980-01-07 01:13 cpu_0 > > > > > > # ls -l > > > .../traces/trace1/control > > > > > > > > > > > > -rwxrwxrwx > > > root > > > root > > > 0 > > > 1980-01-07 01:13 metadata_0 > > > > > > -rwxrwxrwx > > > root > > > root > > > 0 > > > 1980-01-07 01:13 interrupts_0 > > > > > > -rwxrwxrwx > > > root > > > root > > > 0 > > > 1980-01-07 01:13 processes_0 > > > > > > -rwxrwxrwx > > > root > > > root > > > 0 > > > 1980-01-07 01:13 modules_0 > > > > > > -rwxrwxrwx > > > root > > > root > > > 0 > > > 1980-01-07 01:13 network_0 > > > > > > > > > > > > > > > BTW, > > > I would like to > > > know what is the channel concept for LTTng? What's > > > the relationship between > > > trace and channel? What's the normal, flight recorder > > > and hybrid mode? > > > > > > Which > > > files should I get > > > for LTTV on another visualization > > > machine? > > > > > > So far, I > > > can't find related document or the > > > information isn't enough for me. > > > > > > > > > > > > > > > Thanks for your > > > help! > > > > > > > > > > > > > > > Best > > > regards, > > > > > > Bryan Hsieh > > > > > > SPEC, FIH > > > Co., Ltd. > > > > > > Tel:+886-3-270-0000 #2723 > > > > > > Fax:+886-3-270-0001 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > ltt-dev mailing list > > > ltt-dev at lists.casi.polymtl.ca > > > http://lists.casi.polymtl.ca/cgi-bin/mailman/listinfo/ltt-dev > > > > > > > > > > > This e-mail message together with any attachments thereto (if any) is confidential, protected under an enforceable non-disclosure agreement, > > intended only for the use of the named recipient(s) above and may contain information that is privileged, belonging to professional work products > > or exempt from disclosure under applicable laws. Any unauthorized review, use, copying, disclosure, or distribution of any information contained > > in or attached to this transmission is STRICTLY PROHIBITED and may be against the laws. If you have received this message in error, or are not > > the named recipient(s), please immediately notify the sender by e-mail or telephone at IT support hotline 886-3-2700000 EXT 1333 and delete this > > e-mail message and any attached documentation from your computer. Receipt by anyone other than the intended recipient(s) is not a waiver of any > > attorney-client or work product privilege. Thank you! > > > > > > _______________________________________________ > > ltt-dev mailing list > > ltt-dev at lists.casi.polymtl.ca > > http://lists.casi.polymtl.ca/cgi-bin/mailman/listinfo/ltt-dev > > > > -- > Mathieu Desnoyers > OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F BA06 3F25 A8FE 3BAE 9A68 > > This e-mail message together with any attachments thereto (if any) is confidential, protected under an enforceable non-disclosure agreement, > intended only for the use of the named recipient(s) above and may contain information that is privileged, belonging to professional work products > or exempt from disclosure under applicable laws. Any unauthorized review, use, copying, disclosure, or distribution of any information contained > in or attached to this transmission is STRICTLY PROHIBITED and may be against the laws. If you have received this message in error, or are not > the named recipient(s), please immediately notify the sender by e-mail or telephone at IT support hotline 886-3-2700000 EXT 1333 and delete this > e-mail message and any attached documentation from your computer. Receipt by anyone other than the intended recipient(s) is not a waiver of any > attorney-client or work product privilege. Thank you! Content-Description: busybox version and command query.txt > # grep --help > BusyBox v1.12.4 (2009-07-29 23:17:25 CST) multi-call binary > > Usage: grep [-HhrilLnqvsoeFEABC] PATTERN [FILEs...] > > Search for PATTERN in each FILE or standard input > > Options: > -H Prefix output lines with filename where match was found > -h Suppress the prefixing filename on output > -r Recurse subdirectories > -i Ignore case distinctions > -l List names of files that match > -L List names of files that do not match > -n Print line number with output lines > -q Quiet. Return 0 if PATTERN is found, 1 otherwise > -v Select non-matching lines > -s Suppress file open/read error messages > -c Only print count of matching lines > -o Show only the part of a line that matches PATTERN > -m MAX Match up to MAX times per file > -F PATTERN is a set of newline-separated strings > -E PATTERN is an extended regular expression > -e PTRN Pattern to match > -f FILE Read pattern from file > -A Print NUM lines of trailing context > -B Print NUM lines of leading context > -C Print NUM lines of output context > > # awk --help > BusyBox v1.12.4 (2009-07-29 23:17:25 CST) multi-call binary > > Usage: awk [OPTION]... [program-text] [FILE...] > > Options: > -v var=val Set variable > -F sep Use sep as field separator > -f file Read program from file > > # sort --help > BusyBox v1.12.4 (2009-07-29 23:17:25 CST) multi-call binary > > Usage: sort [-nrugMcszbdfimSTokt] [-o FILE] [-k start[.offset][opts][,end[.offset][opts]] [-t CHAR] [FILE]... > > Sort lines of text > > Options: > -b Ignore leading blanks > -c Check whether input is sorted > -d Dictionary order (blank or alphanumeric only) > -f Ignore case > -g General numerical sort > -i Ignore unprintable characters > -k Sort key > -M Sort month > -n Sort numbers > -o Output to file > -k Sort by key > -t CHAR Key separator > -r Reverse sort order > -s Stable (don't sort ties alphabetically) > -u Suppress duplicate lines > -z Lines are terminated by NUL, not newline > -mST Ignored for GNU compatibility > > # cat --help > unknown option -- -usage: cat [-beflnstv] [-] [file ...] > # > Content-Description: LTTng Execution Log.txt > # ltt-armall-1 > Connecting all markers > Connecting fs_buffer_wait_end > Connecting fs_buffer_wait_start > Connecting fs_close > Connecting fs_exec > Connecting fs_ioctl > Connecting fs_llseek > Connecting fs_lseek > Connecting fs_open > Connecting fs_pollfd > Connecting fs_pread64 > Connecting fs_pwrite64 > Connecting fs_read > Connecting fs_readv > Connecting fs_select > Connecting fs_write > Connecting fs_writev > Connecting input_event > Connecting ipc_msg_create > Connecting ipc_sem_create > Connecting ipc_shm_create > Connecting kernel_arch_ipc_call > Connecting kernel_arch_kthread_create > Connecting kernel_arch_syscall_entry > Connecting kernel_arch_syscall_exit > Connecting kernel_arch_trap_entry > Connecting kernel_arch_trap_exit > Connecting kernel_irq_entry > Connecting kernel_irq_exit > Connecting kernel_kthread_stop > Connecting kernel_kthread_stop_ret > Connecting kernel_module_free > Connecting kernel_module_load > Connecting kernel_printk > Connecting kernel_process_exit > Connecting kernel_process_fork > Connecting kernel_process_free > Connecting kernel_process_wait > Connecting kernel_sched_migrate_task > Connecting kernel_sched_schedule > Connecting kernel_sched_try_wakeup > Connecting kernel_sched_wait_task > Connecting kernel_sched_wakeup_new_task > Connecting kernel_send_signal > Connecting kernel_softirq_entry > Connecting kernel_softirq_exit > Connecting kernel_softirq_raise > Connecting kernel_tasklet_high_entry > Connecting kernel_tasklet_high_exit > Connecting kernel_tasklet_low_entry > Connecting kernel_tasklet_low_exit > Connecting kernel_timer_itimer_expired > Connecting kernel_timer_itimer_set > Connecting kernel_timer_set > Connecting kernel_timer_timeout > Connecting kernel_timer_update_time > Connecting kernel_vprintk > Connecting list_file_descriptor > Connecting list_interrupt > Connecting list_module > Connecting list_network_ip_interface > Connecting list_network_ipv4_interface > Connecting list_process_state > Connecting list_statedump_end > Connecting list_vm_map > Connecting mm_handle_fault_entry > Connecting mm_handle_fault_exit > Connecting mm_huge_page_alloc > Connecting mm_huge_page_free > Connecting mm_page_alloc > Connecting mm_page_free > Connecting mm_wait_on_page_end > Connecting mm_wait_on_page_start > Connecting rcu_classic_callback > Connecting rcu_classic_call_rcu > Connecting rcu_classic_call_rcu_bh > Connecting statedump_softirq_vec > Connecting high-rate markers to tap > Connecting fs_buffer_wait_end > Connecting fs_buffer_wait_start > Connecting fs_close > Connecting fs_exec > Connecting fs_ioctl > Connecting fs_llseek > Connecting fs_lseek > Connecting fs_open > Connecting fs_pollfd > Connecting fs_pread64 > Connecting fs_pwrite64 > Connecting fs_read > Connecting fs_readv > Connecting fs_select > Connecting fs_write > Connecting fs_writev > Connecting input_event > Connecting ipc_msg_create > Connecting ipc_sem_create > Connecting ipc_shm_create > Connecting kernel_arch_ipc_call > Connecting kernel_arch_kthread_create > Connecting kernel_arch_syscall_entry > Connecting kernel_arch_syscall_exit > Connecting kernel_arch_trap_entry > Connecting kernel_arch_trap_exit > Connecting kernel_irq_entry > Connecting kernel_irq_exit > Connecting kernel_kthread_stop > Connecting kernel_kthread_stop_ret > Connecting kernel_module_free > Connecting kernel_module_load > Connecting kernel_printk > Connecting kernel_process_exit > Connecting kernel_process_fork > Connecting kernel_process_free > Connecting kernel_process_wait > Connecting kernel_sched_migrate_task > Connecting kernel_sched_schedule > Connecting kernel_sched_try_wakeup > Connecting kernel_sched_wait_task > Connecting kernel_sched_wakeup_new_task > Connecting kernel_send_signal > Connecting kernel_softirq_entry > Connecting kernel_softirq_exit > Connecting kernel_softirq_raise > Connecting kernel_tasklet_high_entry > Connecting kernel_tasklet_high_exit > Connecting kernel_tasklet_low_entry > Connecting kernel_tasklet_low_exit > Connecting kernel_timer_itimer_expired > Connecting kernel_timer_itimer_set > Connecting kernel_timer_set > Connecting kernel_timer_timeout > Connecting kernel_timer_update_time > Connecting kernel_vprintk > Connecting list_file_descriptor > Connecting list_interrupt > Connecting list_module > Connecting list_network_ip_interface > Connecting list_network_ipv4_interface > Connecting list_process_state > Connecting list_statedump_end > Connecting list_vm_map > Connecting mm_handle_fault_entry > Connecting mm_handle_fault_exit > Connecting mm_huge_page_alloc > Connecting mm_huge_page_free > Connecting mm_page_alloc > Connecting mm_page_free > Connecting mm_wait_on_page_end > Connecting mm_wait_on_page_start > Connecting rcu_classic_callback > Connecting rcu_classic_call_rcu > Connecting rcu_classic_call_rcu_bh > Connecting statedump_softirq_vec > > # cat /proc/ltt > marker: kernel_arch_kthread_create format: "pid %ld fn %p" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: kernel_arch_syscall_exit format: " " state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: kernel_arch_syscall_entry format: "syscall_id %d ip #p%ld" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: kernel_arch_ipc_call format: "call %u first %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: kernel_arch_trap_exit format: " " state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: kernel_arch_trap_entry format: "trap_id %ld ip #p%ld" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: statedump_softirq_vec format: "id %d address %p symbol %s" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: list_module format: "name %s state %d refcount %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: core_marker_format format: "name %s format %s" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: core_marker_format format: "name %s format %s" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: list_statedump_end format: " " state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: list_process_state format: "pid %d parent_pid %d name %s type %d mode %d submode %d status %d tgid %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: list_file_descriptor format: "filename %s pid %d fd %u" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: list_vm_map format: "pid %d start %lu end %lu flags %lu pgoff %lu inode %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: list_interrupt format: "name %s action %s irq_id %u" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: list_network_ip_interface format: "name %s address #4u%lu up %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: list_network_ipv4_interface format: "name %s address #4u%lu up %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: core_marker_id format: "name %s id %hu int #1u%zu long #1u%zu pointer #1u%zu size_t #1u%zu alignment #1u%u" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: core_marker_id format: "name %s id %hu int #1u%zu long #1u%zu pointer #1u%zu size_t #1u%zu alignment #1u%u" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: core_marker_format format: "name %s format %s" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: core_marker_id format: "name %s id %hu int #1u%zu long #1u%zu pointer #1u%zu size_t #1u%zu alignment #1u%u" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: core_marker_id format: "name %s id %hu int #1u%zu long #1u%zu pointer #1u%zu size_t #1u%zu alignment #1u%u" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: kernel_module_load format: "name %s" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: kernel_module_free format: "name %s" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: kernel_vprintk format: "loglevel %c string %s ip %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: kernel_printk format: "ip %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: kernel_timer_timeout format: "pid %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: kernel_timer_update_time format: "jiffies #8u%llu xtime_sec %ld xtime_nsec %ld walltomonotonic_sec %ld walltomonotonic_nsec %ld" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: kernel_timer_set format: "expires %lu function %p data %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: kernel_timer_itimer_set format: "which %d interval_sec %ld interval_usec %ld value_sec %ld value_usec %ld" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: kernel_timer_itimer_expired format: "pid %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: kernel_process_fork format: "parent_pid %d child_pid %d child_tgid %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: kernel_process_wait format: "pid %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: kernel_process_exit format: "pid %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: kernel_process_free format: "pid %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: kernel_send_signal format: "pid %d signal %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: kernel_sched_migrate_task format: "pid %d state %ld dest_cpu %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: kernel_sched_schedule format: "prev_pid %d next_pid %d prev_state %ld" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: kernel_sched_wakeup_new_task format: "pid %d state %ld cpu_id %u" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: kernel_sched_try_wakeup format: "pid %d state %ld cpu_id %u" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: kernel_sched_wait_task format: "pid %d state %ld" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: kernel_kthread_stop_ret format: "ret %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: kernel_kthread_stop format: "pid %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: kernel_tasklet_high_exit format: "func %p data %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: kernel_tasklet_high_entry format: "func %p data %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: kernel_tasklet_low_exit format: "func %p data %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: kernel_tasklet_low_entry format: "func %p data %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: kernel_softirq_raise format: "softirq_id %u" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: kernel_softirq_exit format: "softirq_id %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: kernel_softirq_entry format: "softirq_id %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: kernel_irq_exit format: "handled #1u%u" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: kernel_irq_entry format: "irq_id %u kernel_mode %u ip %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: mm_page_alloc format: "order %u pfn %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: mm_page_free format: "order %u pfn %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: mm_handle_fault_exit format: "res %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: mm_handle_fault_entry format: "address %lu ip #p%ld write_access %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: mm_huge_page_alloc format: "pfn %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: mm_huge_page_free format: "pfn %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: mm_wait_on_page_end format: "pfn %lu bit_nr %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: mm_wait_on_page_start format: "pfn %lu bit_nr %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: fs_pollfd format: "fd %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: fs_select format: "fd %d timeout #8d%lld" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: fs_writev format: "fd %lu vlen %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: fs_readv format: "fd %lu vlen %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: fs_pwrite64 format: "fd %u count %zu pos %llu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: fs_pread64 format: "fd %u count %zu pos %llu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: fs_write format: "fd %u count %zu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: fs_read format: "fd %u count %zu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: fs_llseek format: "fd %u offset %lld origin %u" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: fs_lseek format: "fd %u offset %ld origin %u" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: fs_close format: "fd %u" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: fs_open format: "fd %d filename %s" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: fs_ioctl format: "fd %u cmd %u arg %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: fs_exec format: "filename %s" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: fs_buffer_wait_end format: "bh %p" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: fs_buffer_wait_start format: "bh %p" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: ipc_shm_create format: "id %ld flags %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: ipc_sem_create format: "id %ld flags %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: ipc_msg_create format: "id %ld flags %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: lockdep_lock_release format: "retaddr 0x%lX lock %p nested %d" state: 0 call: 0xc0093374 probe single : 0xc00920ec > marker: lockdep_lock_acquire format: "retaddr 0x%lX subclass %u lock %p trylock %d read %d hardirqs_off %d" state: 0 call: 0xc0093374 probe single : 0xc00920ec > marker: lockdep_softirqs_off format: "retaddr 0x%lX" state: 0 call: 0xc0093374 probe single : 0xc00920ec > marker: lockdep_softirqs_on format: "retaddr 0x%lX" state: 0 call: 0xc0093374 probe single : 0xc00920ec > marker: lockdep_hardirqs_off format: "retaddr 0x%lX" state: 0 call: 0xc0093374 probe single : 0xc00920ec > marker: lockdep_hardirqs_on format: "retaddr 0x%lX" state: 0 call: 0xc0093374 probe single : 0xc00920ec > marker: rcu_classic_call_rcu_bh format: "func %p ip 0x%lX" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: rcu_classic_call_rcu format: "func %p ip 0x%lX" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: rcu_classic_callback format: "func %p" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: input_event format: "type %u code %u value %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > [Bryan] Maybe caused by privilege, the the /proc/ltt do not be replaced with content below. It's that will be a problem? > connect statedump_softirq_vec ltt_tap_marker dynami > > # df > /dev: 39564K total, 0K used, 39564K available (block size 4096) > /sqlite_stmt_journals: 4096K total, 0K used, 4096K available (block size 4096) > /system: 72704K total, 71680K used, 1024K available (block size 4096) > /data: 66816K total, 37812K used, 29004K available (block size 4096) > /cache: 67584K total, 1156K used, 66428K available (block size 4096) > /sdcard: 991708K total, 147040K used, 844668K available (block size 4096) > > # lttctl -n trace1 -d -l /sys/kernel/debug/ltt -t /data/ltt/trace1 > Linux Trace Toolkit Trace Control 0.55-16102008 > > Controlling trace : trace1 > > Linux Trace Toolkit Trace Daemon 0.55-16102008 > > Reading from debugfs directory : /sys/kernel/debug/ltt/trace1 > Writing to trace directory : /data/ltt/trace1 > > [Bryan] there is 29004K available for /data/ltt/trace1, is that enough? > [Bryan] do some tests on target device, then check the channel and trace > > # ls -l /sys/kernel/debug/ltt/trace1 > -r-------- root root 0 1980-01-06 00:10 cpu_0 > drwxr-xr-x root root 1980-01-06 00:10 control > # ls -l /sys/kernel/debug/ltt/trace1/control > -r-------- root root 0 1980-01-06 00:10 network_0 > -r-------- root root 0 1980-01-06 00:10 modules_0 > -r-------- root root 0 1980-01-06 00:10 processes_0 > -r-------- root root 0 1980-01-06 00:10 interrupts_0 > -r-------- root root 0 1980-01-06 00:10 metadata_0 > # ls -l /data/ltt/trace1 > drwxrwxrwx root root 1980-01-06 00:10 control > -rwxrwxrwx root root 29949952 1980-01-06 00:10 cpu_0 > # ls -l /data/ltt/trace1/control > -rwxrwxrwx root root 0 1980-01-06 00:10 metadata_0 > -rwxrwxrwx root root 0 1980-01-06 00:10 interrupts_0 > -rwxrwxrwx root root 0 1980-01-06 00:10 processes_0 > -rwxrwxrwx root root 0 1980-01-06 00:10 modules_0 > -rwxrwxrwx root root 0 1980-01-06 00:10 network_0 > > [Bryan] It seems that the space has been exhausted by cpu_0, it's that correct that cpu_0 will grow to so huge(29M)? > How much file space should I prepare for LTTng? > > # lttctl -n trace1 -d -R > Linux Trace Toolkit Trace Control 0.55-16102008 > > Controlling trace : trace1 > > [Bryan] hang in here...I have to break it by Ctrl+Z, What's problem? > > ^Z[1] + Stopped lttctl -n trace1 -d -R > # ls -l /data/ltt/trace1/ > drwxrwxrwx root root 1980-01-06 00:10 control > -rwxrwxrwx root root 29949952 1980-01-06 00:10 cpu_0 > # ls -l /data/ltt/trace1/control > -rwxrwxrwx root root 0 1980-01-06 00:10 metadata_0 > -rwxrwxrwx root root 0 1980-01-06 00:10 interrupts_0 > -rwxrwxrwx root root 0 1980-01-06 00:10 processes_0 > -rwxrwxrwx root root 0 1980-01-06 00:10 modules_0 > -rwxrwxrwx root root 0 1980-01-06 00:10 network_0 > # ls -l /sys/kernel/debug/ltt/trace1 > -r-------- root root 0 1980-01-06 00:10 cpu_0 > drwxr-xr-x root root 1980-01-06 00:10 control > # ls -l /sys/kernel/debug/ltt/trace1/control > -r-------- root root 0 1980-01-06 00:10 network_0 > -r-------- root root 0 1980-01-06 00:10 modules_0 > -r-------- root root 0 1980-01-06 00:10 processes_0 > -r-------- root root 0 1980-01-06 00:10 interrupts_0 > -r-------- root root 0 1980-01-06 00:10 metadata_0 > # df > /dev: 39564K total, 0K used, 39564K available (block size 4096) > /sqlite_stmt_journals: 4096K total, 0K used, 4096K available (block size 4096) > /system: 72704K total, 71680K used, 1024K available (block size 4096) > /data: 66816K total, 66732K used, 84K available (block size 4096) > /cache: 67584K total, 1156K used, 66428K available (block size 4096) > /sdcard: 991708K total, 147040K used, 844668K available (block size 4096) > > > Content-Description: LTTng Kernel Config.txt > > [ ] Kernel->user space relay support (formerly relayfs) > [*] Activate tracepoints x x > [*] Activate markers x x > Linux Trace Toolkit ---> > <*> Compile lttng tracing probes x x > [*] LTTng fine-grained timestamping x x > [*] Linux Trace Toolkit Instrumentation Support x x > <*> Linux Trace Toolkit High-speed Lockless Data Relay x x > <*> Linux Trace Toolkit Lock-Protected Data Relay x x > [*] Debug check for random access in ltt relay buffers x x > <*> Linux Trace Toolkit Serializer x x > <*> Linux Trace Toolkit Marker Control x x > <*> Linux Trace Toolkit Tracer x x > [*] Align Linux Trace Toolkit Traces x x > [*] Add event size field to LTT events for tracer debugging x x > [*] Support trace extraction from crash dump x x > <*> Linux Trace Toolkit Netlink Controller x x > <*> Linux Trace Toolkit State Dump -- Mathieu Desnoyers OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F BA06 3F25 A8FE 3BAE 9A68 ^ permalink raw reply [flat|nested] 9+ messages in thread
* [ltt-dev] Some LTTng 0.43 issue on ARM 2009-08-04 13:18 ` Mathieu Desnoyers @ 2009-08-06 2:52 ` Bryan Hsieh 2009-08-06 17:20 ` Mathieu Desnoyers 2009-08-27 8:56 ` [ltt-dev] Change process name to be command line Bryan Hsieh 0 siblings, 2 replies; 9+ messages in thread From: Bryan Hsieh @ 2009-08-06 2:52 UTC (permalink / raw) Hi Mathieu, All [Bryan] I modified install path and re-compile, then install to my target system, even I replace the original toolbox with new busybox, it still doesn't work, I try to use older version busybox and install to my target system, finally It's work! I watch the space change and stop trace before the capacity been exhausted, the lttd can stop successfully. Thank LTT Gods! # ls -l /data/ltt/trace1 drwxrwxrwx 1 0 0 2048 Jan 6 01:17 control -rwxrwxrwx 1 0 0 25165824 Jan 6 01:17 cpu_0 # ls -l /data/ltt/trace1/control -rwxrwxrwx 1 0 0 65536 Jan 6 01:17 interrupts_0 -rwxrwxrwx 1 0 0 65536 Jan 6 01:17 metadata_0 -rwxrwxrwx 1 0 0 65536 Jan 6 01:17 modules_0 -rwxrwxrwx 1 0 0 65536 Jan 6 01:17 network_0 -rwxrwxrwx 1 0 0 262144 Jan 6 01:17 processes_0 [Bryan] I have another problem, I can store trace in /data or /cache directory by -t option, but I still can't store trace in /sdcard directory, the files size are empty in /sdcard/ltt/trace1, what's wrong? # mount rootfs on / type rootfs (ro) tmpfs on /dev type tmpfs (rw,mode=755) devpts on /dev/pts type devpts (rw,mode=600) proc on /proc type proc (rw) sysfs on /sys type sysfs (rw) debugfs on /sys/kernel/debug type debugfs (rw) tmpfs on /sqlite_stmt_journals type tmpfs (rw,size=4096k) /dev/block/mtdblock3 on /system type yaffs2 (rw) /dev/block/mtdblock6 on /data type yaffs2 (rw,nosuid,nodev) /dev/block/mtdblock4 on /cache type yaffs2 (rw,nosuid,nodev) /dev/block//vold/179:1 on /sdcard type vfat (rw,dirsync,nosuid,nodev,noexec,uid=1000,gid=1000,fmask=0711,dmask=0700,allow_utime=0022,codepage=cp437,iocharset=iso8859-1,shortname=mixed,utf8) Best regards, Bryan Hsieh SPEC, FIH Co., Ltd. Tel:+886-3-270-0000 #2723 Fax:+886-3-270-0001 -----Original Message----- From: Mathieu Desnoyers [mailto:compudj@krystal.dyndns.org] Sent: Tuesday, August 04, 2009 9:18 PM To: Bryan Hsieh Cc: 'Ajit R'; 'ltt-dev at lists.casi.polymtl.ca'; Tommy Peng Subject: Re: [ltt-dev] Some LTTng 0.43 issue on ARM * Bryan Hsieh (bryanhsieh at fihspec.com) wrote: > Hi Mathieu, All > > Please check my comment which marked by [Bryan], Thanks! > ----------------------------------------------------------------------------- > [Bryan] > All the related LTT functions are build-in not build as modules, Please check attachment- LTTng Kernel Config..txt, I am not sure if Kernel->user space relay support (formerly relayfs) need to build-in. > Is there any item I missed? > ----------------------------------------------------------------------------- > [ ] Kernel->user space relay support (formerly relayfs) LTTng does not need relayfs anymore. So this is ok. > [*] Activate tracepoints > [*] Activate markers > Linux Trace Toolkit ---> > <*> Compile lttng tracing probes > [*] LTTng fine-grained timestamping > [*] Linux Trace Toolkit Instrumentation Support > <*> Linux Trace Toolkit High-speed Lockless Data Relay > <*> Linux Trace Toolkit Lock-Protected Data Relay > [*] Debug check for random access in ltt relay buffers > <*> Linux Trace Toolkit Serializer > <*> Linux Trace Toolkit Marker Control > <*> Linux Trace Toolkit Tracer > [*] Align Linux Trace Toolkit Traces > [*] Add event size field to LTT events for tracer debugging > [*] Support trace extraction from crash dump > <*> Linux Trace Toolkit Netlink Controller > <*> Linux Trace Toolkit State Dump > ----------------------------------------------------------------------------- > [Bryan] > I am sure that the external utilities are build-in busybox, please check the attachment - " busybox version and command query.txt". > > Finally, I figure out why the ltt-armall doesn't work correctly in my target system, root cause is the output pipeline will be empty after grep twice, even I use a temp file to avoid connecting two grep commands directly, it's still empty. So I copy the /proc/ltt to my local desktop to do ltt-armall script, then generate the /proc/ltt which been filtered by following command. > MARKERS=`cat /proc/ltt|grep -v %k|awk '{print $2}'|sort -u|grep -v ^core_|grep -v ^locking_|grep -v ^lockdep_` > I rename it as ltt_filter and copy to my target system path /system/bin/, then modify ltt-armall by following modification and run it. > MARKERS=`cat /system/bin/ltt_filter` > > The execution result on screen: > # ltt-armall-1 > Connecting all markers > Connecting fs_buffer_wait_end > Connecting fs_buffer_wait_start > Connecting fs_close > Connecting fs_exec > Connecting fs_ioctl > Connecting fs_llseek > Connecting fs_lseek > Connecting fs_open > Connecting fs_pollfd > Connecting fs_pread64 > Connecting fs_pwrite64 > Connecting fs_read > Connecting fs_readv > Connecting fs_select > Connecting fs_write > Connecting fs_writev > Connecting input_event > Connecting ipc_msg_create > Connecting ipc_sem_create > Connecting ipc_shm_create > Connecting kernel_arch_ipc_call > Connecting kernel_arch_kthread_create > Connecting kernel_arch_syscall_entry > Connecting kernel_arch_syscall_exit > Connecting kernel_arch_trap_entry > Connecting kernel_arch_trap_exit > Connecting kernel_irq_entry > Connecting kernel_irq_exit > Connecting kernel_kthread_stop > Connecting kernel_kthread_stop_ret > Connecting kernel_module_free > Connecting kernel_module_load > Connecting kernel_printk > Connecting kernel_process_exit > Connecting kernel_process_fork > Connecting kernel_process_free > Connecting kernel_process_wait > Connecting kernel_sched_migrate_task > Connecting kernel_sched_schedule > Connecting kernel_sched_try_wakeup > Connecting kernel_sched_wait_task > Connecting kernel_sched_wakeup_new_task > Connecting kernel_send_signal > Connecting kernel_softirq_entry > Connecting kernel_softirq_exit > Connecting kernel_softirq_raise > Connecting kernel_tasklet_high_entry > Connecting kernel_tasklet_high_exit > Connecting kernel_tasklet_low_entry > Connecting kernel_tasklet_low_exit > Connecting kernel_timer_itimer_expired > Connecting kernel_timer_itimer_set > Connecting kernel_timer_set > Connecting kernel_timer_timeout > Connecting kernel_timer_update_time > Connecting kernel_vprintk > Connecting list_file_descriptor > Connecting list_interrupt > Connecting list_module > Connecting list_network_ip_interface > Connecting list_network_ipv4_interface > Connecting list_process_state > Connecting list_statedump_end > Connecting list_vm_map > Connecting mm_handle_fault_entry > Connecting mm_handle_fault_exit > Connecting mm_huge_page_alloc > Connecting mm_huge_page_free > Connecting mm_page_alloc > Connecting mm_page_free > Connecting mm_wait_on_page_end > Connecting mm_wait_on_page_start > Connecting rcu_classic_callback > Connecting rcu_classic_call_rcu > Connecting rcu_classic_call_rcu_bh > Connecting statedump_softirq_vec > Connecting high-rate markers to tap > Connecting fs_buffer_wait_end > Connecting fs_buffer_wait_start > Connecting fs_close > Connecting fs_exec > Connecting fs_ioctl > Connecting fs_llseek > Connecting fs_lseek > Connecting fs_open > Connecting fs_pollfd > Connecting fs_pread64 > Connecting fs_pwrite64 > Connecting fs_read > Connecting fs_readv > Connecting fs_select > Connecting fs_write > Connecting fs_writev > Connecting input_event > Connecting ipc_msg_create > Connecting ipc_sem_create > Connecting ipc_shm_create > Connecting kernel_arch_ipc_call > Connecting kernel_arch_kthread_create > Connecting kernel_arch_syscall_entry > Connecting kernel_arch_syscall_exit > Connecting kernel_arch_trap_entry > Connecting kernel_arch_trap_exit > Connecting kernel_irq_entry > Connecting kernel_irq_exit > Connecting kernel_kthread_stop > Connecting kernel_kthread_stop_ret > Connecting kernel_module_free > Connecting kernel_module_load > Connecting kernel_printk > Connecting kernel_process_exit > Connecting kernel_process_fork > Connecting kernel_process_free > Connecting kernel_process_wait > Connecting kernel_sched_migrate_task > Connecting kernel_sched_schedule > Connecting kernel_sched_try_wakeup > Connecting kernel_sched_wait_task > Connecting kernel_sched_wakeup_new_task > Connecting kernel_send_signal > Connecting kernel_softirq_entry > Connecting kernel_softirq_exit > Connecting kernel_softirq_raise > Connecting kernel_tasklet_high_entry > Connecting kernel_tasklet_high_exit > Connecting kernel_tasklet_low_entry > Connecting kernel_tasklet_low_exit > Connecting kernel_timer_itimer_expired > Connecting kernel_timer_itimer_set > Connecting kernel_timer_set > Connecting kernel_timer_timeout > Connecting kernel_timer_update_time > Connecting kernel_vprintk > Connecting list_file_descriptor > Connecting list_interrupt > Connecting list_module > Connecting list_network_ip_interface > Connecting list_network_ipv4_interface > Connecting list_process_state > Connecting list_statedump_end > Connecting list_vm_map > Connecting mm_handle_fault_entry > Connecting mm_handle_fault_exit > Connecting mm_huge_page_alloc > Connecting mm_huge_page_free > Connecting mm_page_alloc > Connecting mm_page_free > Connecting mm_wait_on_page_end > Connecting mm_wait_on_page_start > Connecting rcu_classic_callback > Connecting rcu_classic_call_rcu > Connecting rcu_classic_call_rcu_bh > Connecting statedump_softirq_vec > ----------------------------------------------------------------------------- > [Bryan] It's that correct result? Yes, this output is correct. So you probably need to recompile busybox with awk and/or sed and/or sort support..... > ----------------------------------------------------------------------------- > # cat /proc/ltt > marker: kernel_arch_kthread_create format: "pid %ld fn %p" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: kernel_arch_syscall_exit format: " " state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: kernel_arch_syscall_entry format: "syscall_id %d ip #p%ld" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: kernel_arch_ipc_call format: "call %u first %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: kernel_arch_trap_exit format: " " state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: kernel_arch_trap_entry format: "trap_id %ld ip #p%ld" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: statedump_softirq_vec format: "id %d address %p symbol %s" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: list_module format: "name %s state %d refcount %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: core_marker_format format: "name %s format %s" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: core_marker_format format: "name %s format %s" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: list_statedump_end format: " " state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: list_process_state format: "pid %d parent_pid %d name %s type %d mode %d submode %d status %d tgid %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: list_file_descriptor format: "filename %s pid %d fd %u" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: list_vm_map format: "pid %d start %lu end %lu flags %lu pgoff %lu inode %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: list_interrupt format: "name %s action %s irq_id %u" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: list_network_ip_interface format: "name %s address #4u%lu up %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: list_network_ipv4_interface format: "name %s address #4u%lu up %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: core_marker_id format: "name %s id %hu int #1u%zu long #1u%zu pointer #1u%zu size_t #1u%zu alignment #1u%u" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: core_marker_id format: "name %s id %hu int #1u%zu long #1u%zu pointer #1u%zu size_t #1u%zu alignment #1u%u" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: core_marker_format format: "name %s format %s" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: core_marker_id format: "name %s id %hu int #1u%zu long #1u%zu pointer #1u%zu size_t #1u%zu alignment #1u%u" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: core_marker_id format: "name %s id %hu int #1u%zu long #1u%zu pointer #1u%zu size_t #1u%zu alignment #1u%u" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: kernel_module_load format: "name %s" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: kernel_module_free format: "name %s" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: kernel_vprintk format: "loglevel %c string %s ip %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: kernel_printk format: "ip %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: kernel_timer_timeout format: "pid %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: kernel_timer_update_time format: "jiffies #8u%llu xtime_sec %ld xtime_nsec %ld walltomonotonic_sec %ld walltomonotonic_nsec %ld" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: kernel_timer_set format: "expires %lu function %p data %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: kernel_timer_itimer_set format: "which %d interval_sec %ld interval_usec %ld value_sec %ld value_usec %ld" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: kernel_timer_itimer_expired format: "pid %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: kernel_process_fork format: "parent_pid %d child_pid %d child_tgid %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: kernel_process_wait format: "pid %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: kernel_process_exit format: "pid %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: kernel_process_free format: "pid %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: kernel_send_signal format: "pid %d signal %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: kernel_sched_migrate_task format: "pid %d state %ld dest_cpu %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: kernel_sched_schedule format: "prev_pid %d next_pid %d prev_state %ld" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: kernel_sched_wakeup_new_task format: "pid %d state %ld cpu_id %u" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: kernel_sched_try_wakeup format: "pid %d state %ld cpu_id %u" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: kernel_sched_wait_task format: "pid %d state %ld" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: kernel_kthread_stop_ret format: "ret %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: kernel_kthread_stop format: "pid %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: kernel_tasklet_high_exit format: "func %p data %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: kernel_tasklet_high_entry format: "func %p data %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: kernel_tasklet_low_exit format: "func %p data %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: kernel_tasklet_low_entry format: "func %p data %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: kernel_softirq_raise format: "softirq_id %u" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: kernel_softirq_exit format: "softirq_id %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: kernel_softirq_entry format: "softirq_id %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: kernel_irq_exit format: "handled #1u%u" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: kernel_irq_entry format: "irq_id %u kernel_mode %u ip %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: mm_page_alloc format: "order %u pfn %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: mm_page_free format: "order %u pfn %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: mm_handle_fault_exit format: "res %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: mm_handle_fault_entry format: "address %lu ip #p%ld write_access %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: mm_huge_page_alloc format: "pfn %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: mm_huge_page_free format: "pfn %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: mm_wait_on_page_end format: "pfn %lu bit_nr %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: mm_wait_on_page_start format: "pfn %lu bit_nr %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: fs_pollfd format: "fd %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: fs_select format: "fd %d timeout #8d%lld" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: fs_writev format: "fd %lu vlen %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: fs_readv format: "fd %lu vlen %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: fs_pwrite64 format: "fd %u count %zu pos %llu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: fs_pread64 format: "fd %u count %zu pos %llu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: fs_write format: "fd %u count %zu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: fs_read format: "fd %u count %zu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: fs_llseek format: "fd %u offset %lld origin %u" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: fs_lseek format: "fd %u offset %ld origin %u" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: fs_close format: "fd %u" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: fs_open format: "fd %d filename %s" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: fs_ioctl format: "fd %u cmd %u arg %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: fs_exec format: "filename %s" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: fs_buffer_wait_end format: "bh %p" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: fs_buffer_wait_start format: "bh %p" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: ipc_shm_create format: "id %ld flags %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: ipc_sem_create format: "id %ld flags %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: ipc_msg_create format: "id %ld flags %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: lockdep_lock_release format: "retaddr 0x%lX lock %p nested %d" state: 0 call: 0xc0093374 probe single : 0xc00920ec > marker: lockdep_lock_acquire format: "retaddr 0x%lX subclass %u lock %p trylock %d read %d hardirqs_off %d" state: 0 call: 0xc0093374 probe single : 0xc00920ec > marker: lockdep_softirqs_off format: "retaddr 0x%lX" state: 0 call: 0xc0093374 probe single : 0xc00920ec > marker: lockdep_softirqs_on format: "retaddr 0x%lX" state: 0 call: 0xc0093374 probe single : 0xc00920ec > marker: lockdep_hardirqs_off format: "retaddr 0x%lX" state: 0 call: 0xc0093374 probe single : 0xc00920ec > marker: lockdep_hardirqs_on format: "retaddr 0x%lX" state: 0 call: 0xc0093374 probe single : 0xc00920ec > marker: rcu_classic_call_rcu_bh format: "func %p ip 0x%lX" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: rcu_classic_call_rcu format: "func %p ip 0x%lX" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: rcu_classic_callback format: "func %p" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: input_event format: "type %u code %u value %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > ----------------------------------------------------------------------------- > [Bryan] But the /proc/ltt is still same, it's that correct? What's the expected result? This output is good. busybox problem, as I said above. Try each command used by busybox to test them. > ----------------------------------------------------------------------------- > # df > /dev: 39564K total, 0K used, 39564K available (block size 4096) > /sqlite_stmt_journals: 4096K total, 0K used, 4096K available (block size 4096) > /system: 72704K total, 71680K used, 1024K available (block size 4096) > /data: 66816K total, 37812K used, 29004K available (block size 4096) > /cache: 67584K total, 1156K used, 66428K available (block size 4096) > /sdcard: 991708K total, 147040K used, 844668K available (block size 4096) > ----------------------------------------------------------------------------- > [Bryan] There is 29004K available for trace log file /data/ltt/trace1. Is that enough? No. I would save it to /sdcard. > ----------------------------------------------------------------------------- > # cat /proc/meminfo > MemTotal: 79128 kB > MemFree: 2440 kB > Buffers: 36 kB > Cached: 29872 kB > SwapCached: 0 kB > Active: 61384 kB > Inactive: 5052 kB > SwapTotal: 0 kB > SwapFree: 0 kB > Dirty: 0 kB > Writeback: 0 kB > AnonPages: 36552 kB > Mapped: 24888 kB > Slab: 3856 kB > SReclaimable: 904 kB > SUnreclaim: 2952 kB > PageTables: 3660 kB > NFS_Unstable: 0 kB > Bounce: 0 kB > WritebackTmp: 0 kB > CommitLimit: 39564 kB > Committed_AS: 1032216 kB > VmallocTotal: 409600 kB > VmallocUsed: 52908 kB > VmallocChunk: 332796 kB > ----------------------------------------------------------------------------- > [Bryan] > There is 2440 kB free memory. Is that enough? I think that yes. The page allocator will kick out Cached pages when needed. > ----------------------------------------------------------------------------- > # lttctl -n trace1 -d -l /sys/kernel/debug/ltt -t /data/ltt/trace1 > Linux Trace Toolkit Trace Control 0.55-16102008 > > Controlling trace : trace1 > > Linux Trace Toolkit Trace Daemon 0.55-16102008 > > Reading from debugfs directory : /sys/kernel/debug/ltt/trace1 > Writing to trace directory : /data/ltt/trace1 > ----------------------------------------------------------------------------- > [Bryan] do some tests on target device, then check the channel and trace > ----------------------------------------------------------------------------- > # ls -l /sys/kernel/debug/ltt/trace1 > -r-------- root root 0 1980-01-06 00:10 cpu_0 > drwxr-xr-x root root 1980-01-06 00:10 control > # ls -l /sys/kernel/debug/ltt/trace1/control > -r-------- root root 0 1980-01-06 00:10 network_0 > -r-------- root root 0 1980-01-06 00:10 modules_0 > -r-------- root root 0 1980-01-06 00:10 processes_0 > -r-------- root root 0 1980-01-06 00:10 interrupts_0 > -r-------- root root 0 1980-01-06 00:10 metadata_0 > # ls -l /data/ltt/trace1 > drwxrwxrwx root root 1980-01-06 00:10 control > -rwxrwxrwx root root 29949952 1980-01-06 00:10 cpu_0 > # ls -l /data/ltt/trace1/control > -rwxrwxrwx root root 0 1980-01-06 00:10 metadata_0 > -rwxrwxrwx root root 0 1980-01-06 00:10 interrupts_0 > -rwxrwxrwx root root 0 1980-01-06 00:10 processes_0 > -rwxrwxrwx root root 0 1980-01-06 00:10 modules_0 > -rwxrwxrwx root root 0 1980-01-06 00:10 network_0 > ----------------------------------------------------------------------------- > [Bryan] It seems that the capacity has been exhausted by cpu_0, because 29004K available. > It's that correct that cpu_0 will grow to so huge (29M)? 29M is not huge. It is small :) Especially if the system is very busy. > ----------------------------------------------------------------------------- > # lttctl -n trace1 -d -R > Linux Trace Toolkit Trace Control 0.55-16102008 > > Controlling trace : trace1 > ^Z[1] + Stopped lttctl -n trace1 -d -R > ----------------------------------------------------------------------------- > [Bryan] hang in here...I have to break it by Ctrl+Z, What's problem? It's waiting for lttd to release the trace channels, but I guess that the lack of space is causing errors, so lttd keeps running. > ----------------------------------------------------------------------------- > # ls -l /data/ltt/trace1/ > drwxrwxrwx root root 1980-01-06 00:10 control > -rwxrwxrwx root root 29949952 1980-01-06 00:10 cpu_0 > # ls -l /data/ltt/trace1/control > -rwxrwxrwx root root 0 1980-01-06 00:10 metadata_0 > -rwxrwxrwx root root 0 1980-01-06 00:10 interrupts_0 > -rwxrwxrwx root root 0 1980-01-06 00:10 processes_0 > -rwxrwxrwx root root 0 1980-01-06 00:10 modules_0 > -rwxrwxrwx root root 0 1980-01-06 00:10 network_0 > # ls -l /sys/kernel/debug/ltt/trace1 > -r-------- root root 0 1980-01-06 00:10 cpu_0 > drwxr-xr-x root root 1980-01-06 00:10 control > # ls -l /sys/kernel/debug/ltt/trace1/control > -r-------- root root 0 1980-01-06 00:10 network_0 > -r-------- root root 0 1980-01-06 00:10 modules_0 > -r-------- root root 0 1980-01-06 00:10 processes_0 > -r-------- root root 0 1980-01-06 00:10 interrupts_0 > -r-------- root root 0 1980-01-06 00:10 metadata_0 > # df > /dev: 39564K total, 0K used, 39564K available (block size 4096) > /sqlite_stmt_journals: 4096K total, 0K used, 4096K available (block size 4096) > /system: 72704K total, 71680K used, 1024K available (block size 4096) > /data: 66816K total, 66732K used, 84K available (block size 4096) > /cache: 67584K total, 1156K used, 66428K available (block size 4096) > /sdcard: 991708K total, 147040K used, 844668K available (block size 4096) > ----------------------------------------------------------------------------- > [Bryan] I think the trace is incomplete. Yes, here /data only has 84K avail. It's full. Mathieu > ----------------------------------------------------------------------------- > > Please help me! Thank you very much! > > Best regards, > Bryan Hsieh > > SPEC, FIH Co., Ltd. > Tel:+886-3-270-0000 #2723 > Fax:+886-3-270-0001 > > > -----Original Message----- > From: Mathieu Desnoyers [mailto:compudj@krystal.dyndns.org] > Sent: Friday, July 31, 2009 11:43 PM > To: Bryan Hsieh > Cc: 'Ajit R'; 'ltt-dev at lists.casi.polymtl.ca'; Tommy Peng > Subject: Re: [ltt-dev] Some LTTng 0.43 issue on ARM > > * Bryan Hsieh (bryanhsieh at fihspec.com) wrote: > > Hi Ajit, > > > > I appreciate your reply! > > But there is a problem here. > > > > My installation of Busybox is 1.12.4. > > It sounds like either awk or sed or not built in your busybox > installation. > > Can you provide the output of > > cat /proc/ltt > > on your target system ? > > Check the content of ltt-armall to see which external programs it needs. > > Also, if you built the lttng probes as modules, you may need to load > them explicitely. > > Mathieu > > > > > > I follow your suggestion to test on x86 Desktop ARM simulator, the message said that can't find /proc/ltt. I think it should be, because I didn't patch to ARM simulator Linux kernel. > > > > [sbox-ARM: ~/ltt-control-0.55-16102008/lttctl] > ltt-armall > > Connecting all markers > > cat: /proc/ltt: No such file or directory > > Connecting high-rate markers to tap > > cat: /proc/ltt: No such file or directory > > > > It should be work on my target device, but it seems not. > > I can't find /ltt under /proc > > Maybe it's root cause, but why there is no /ltt/proc? And why there is no " cat: /proc/ltt: No such file or directory" message in my target device during running ltt-armall? > > > > BTW, I mount debugfs in init.c > > Mount("debugfs", "/sys/kernel/debug", "debugfs", 0, NULL); > > It should not be a problem. > > > > -----------------------------------------------------------------------------------------------------------Another problem: > > I command as following steps. > > # lttctl -n trace1 -q > > Linux Trace Toolkit Trace Control 0.55-16102008 > > Controlling trace : trace1 > > > > # lttctl -n trace1 -r > > Linux Trace Toolkit Trace Control 0.55-16102008 > > Controlling trace : trace1 > > > > >>> Android shell hang in here, I have to press Ctrl+Z to force stopping - lttctl -n trace1 -r .Why? Is anything wrong? > > # ls -l ../tmp/trace1 > > drwxrwxrwx root root 1980-01-07 06:37 control > > -rwxrwxrwx root root 1048576 1980-01-07 06:37 cpu_0 > > # ls -l ../tmp/trace1/control > > -rwxrwxrwx root root 0 1980-01-07 06:37 metadata_0 > > -rwxrwxrwx root root 0 1980-01-07 06:37 interrupts_0 > > -rwxrwxrwx root root 10240 1980-01-07 06:37 processes_0 > > -rwxrwxrwx root root 65536 1980-01-07 06:37 modules_0 > > -rwxrwxrwx root root 65536 1980-01-07 06:37 network_0 > > > > The Channel in /sys/kernel/debug/ltt/trace1 is clear. Does it cause by destroy channel command? > > > > The metadata_0 and interrupts_0 are empty, Is that correct? I've done some key press, the interrupts_o shouldn't be empt. > > > > And those files almost filled by 0 except for the short header I can't make sure, it looks like incomplete. > > > > I hope you can help me to solve these issues, I am appreciate you so much.. > > > > Best regards, > > Bryan Hsieh > > > > SPEC, FIH Co., Ltd. > > Tel:+886-3-270-0000 #2723 > > Fax:+886-3-270-0001 > > > > > > -----Original Message----- > > From: Ajit R [mailto:ajitrzid@yahoo.com] > > Sent: Friday, July 31, 2009 6:17 PM > > To: 'ltt-dev at lists.casi.polymtl.ca'; Bryan Hsieh > > Cc: Tommy Peng > > Subject: Re: [ltt-dev] Some LTTng 0.43 issue on ARM > > > > > > Hi Bryan, > > My replies with the [Ajit] tag. > > LTT Gods correct me if I am wrong. > > > > Thanks > > Ajit > > > > --- On Fri, 7/31/09, Bryan Hsieh <bryanhsieh at fihspec.com> wrote: > > > > > From: Bryan Hsieh <bryanhsieh@fihspec.com> > > > Subject: [ltt-dev] Some LTTng 0.43 issue on ARM > > > To: "'ltt-dev at lists.casi.polymtl.ca'" <ltt-dev at lists.casi.polymtl.ca> > > > Cc: "Tommy Peng" <tommypeng at fihspec.com> > > > Date: Friday, July 31, 2009, 12:19 PM > > > > > Hi Mathieu, > > > all, > > > > > > > > > > > > > > > I've ported > > > LTTng 0.43 on Android successfully, > > > but I am wondering if the result is correct, please help me > > > to verify, thank > > > you very much! > > > > > > > > > > > > > > > My Traced > > > machine: > > > > > > ARM11 (ARM1136J(F)-S) / Linux > > > 2.6..27 > > > (Android 1.5) > > > > > > My > > > visualization machine: > > > > > > X86 / Sun xVM Virtual Box (Unbuntu > > > 8.10) > > > > > > > > > > > > LTTng: 0.43 > > > > > > LTTV: > > > 0.11.2 > > > > > > LTT Control: 0.55 > > > > > > > > > > > > > > > I > > > connect to Android > > > shell by ADB tool after Android bring up, then I command as > > > following step, the > > > command line options are different with LTTng manual, > > > because the example in manual > > > is for old version. > > > > > > > > > > > > > > > # ltt-armall > > > > > [ Ajit ] > > ltt-armall is a shell script. It connects + arms the markers in the > > kernel. So depending on your kernel + LTT version it would write some > > "text" into the /proc/ltt directory or a "1" into files in > > the debugfs directory. > > Make sure, the binaries for running the script are available in your target environment. For eg. awk, grep, sed etc. Busybox versions might have their limitations. If they do not work, you could do equivalent things on your Desktop and then write a simple script, doing the same on your target. > > Once you run "ltt-armall", you should see some debug statements in your "dmesg" like "connect" or "arm" etc. > > > > > > > > Connecting all > > > markers > > > > > > Connecting high-rate > > > markers to tap > > > > > > # lttctl > > > -n trace1 -d -l /sys/kernel/debug/ltt -t > > > /system/traces/trace1 > > > > > [Ajit] > > Not sure if your "-l" option is right. you have to point it to the place where you mounted your debugfs. > > > > > > > > Linux Trace Toolkit > > > Trace Control 0.55-16102008 > > > > > > > > > > > > > > > Controlling trace : > > > trace1 > > > > > > > > > > > > > > > Linux Trace Toolkit > > > Trace Daemon 0.55-16102008 > > > > > > > > > > > > > > > Reading from debugfs > > > directory : /sys/kernel/debug/ltt/trace1 > > > > > > > > > Writing to trace > > > directory : /system/traces/trace1 > > > > > > > > > > > > > > > It > > > seems the LTT control > > > and daemon work fine. I run some scenarios for 1 minute on > > > Android, but the trace > > > files are still empty (all files length are > > > 0) > > > > > > What's > > > wrong? Is > > > there any step I missed? Or the LTT control and daemon > > > don't work? > > > > > > > > > > > > > > > # ls > > > -l ../../sys/kernel/debug/ltt/ > > > > > > > > > > > > drwxr-xr-x > > > root > > > root > > > 1980-01-07 01:13 trace1 > > > > > > # ls > > > -l > > > ../../sys/kernel/debug/ltt/trace1 > > > > > > > > > -r-------- > > > root > > > root > > > 0 > > > 1980-01-07 01:13 cpu_0 > > > > > > > [Ajit] > > If you have connected and armed it right, you should see the cpu_0 > > file increasing in size. After sometime, disconnect by running the > > ltt-control -n whatever-name -R > > and then you will see files in the control directory (network, process, etc ) grown in size. Tar and move the trace directory to your host and do the needful. > > > > > > > drwxr-xr-x > > > root > > > root > > > 1980-01-07 01:13 control > > > > > > # ls > > > -l > > > ../../sys/kernel/debug/ltt/trace1/control > > > > > > > > > -r-------- > > > root > > > root > > > 0 > > > 1980-01-07 01:13 network_0 > > > > > > -r-------- > > > root > > > root > > > 0 > > > 1980-01-07 01:13 modules_0 > > > > > > -r-------- > > > root > > > root > > > 0 > > > 1980-01-07 01:13 processes_0 > > > > > > -r-------- > > > root > > > root > > > 0 > > > 1980-01-07 01:13 interrupts_0 > > > > > > -r-------- > > > root > > > root > > > 0 1980-01-07 01:13 metadata_0 > > > > > > # ls -l > > > .../traces/ > > > > > > > > > > > > drwxrwxrwx > > > root > > > root > > > 1980-01-07 01:13 trace1 > > > > > > # ls -l > > > .../traces/trace1 > > > > > > drwxrwxrwx > > > root > > > root > > > 1980-01-07 01:13 control > > > > > > -rwxrwxrwx > > > root > > > root > > > 0 > > > 1980-01-07 01:13 cpu_0 > > > > > > # ls -l > > > .../traces/trace1/control > > > > > > > > > > > > -rwxrwxrwx > > > root > > > root > > > 0 > > > 1980-01-07 01:13 metadata_0 > > > > > > -rwxrwxrwx > > > root > > > root > > > 0 > > > 1980-01-07 01:13 interrupts_0 > > > > > > -rwxrwxrwx > > > root > > > root > > > 0 > > > 1980-01-07 01:13 processes_0 > > > > > > -rwxrwxrwx > > > root > > > root > > > 0 > > > 1980-01-07 01:13 modules_0 > > > > > > -rwxrwxrwx > > > root > > > root > > > 0 > > > 1980-01-07 01:13 network_0 > > > > > > > > > > > > > > > BTW, > > > I would like to > > > know what is the channel concept for LTTng? What's > > > the relationship between > > > trace and channel? What's the normal, flight recorder > > > and hybrid mode? > > > > > > Which > > > files should I get > > > for LTTV on another visualization > > > machine? > > > > > > So far, I > > > can't find related document or the > > > information isn't enough for me. > > > > > > > > > > > > > > > Thanks for your > > > help! > > > > > > > > > > > > > > > Best > > > regards, > > > > > > Bryan Hsieh > > > > > > SPEC, FIH > > > Co., Ltd. > > > > > > Tel:+886-3-270-0000 #2723 > > > > > > Fax:+886-3-270-0001 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > ltt-dev mailing list > > > ltt-dev at lists.casi.polymtl.ca > > > http://lists.casi.polymtl.ca/cgi-bin/mailman/listinfo/ltt-dev > > > > > > > > > > > This e-mail message together with any attachments thereto (if any) is confidential, protected under an enforceable non-disclosure agreement, > > intended only for the use of the named recipient(s) above and may contain information that is privileged, belonging to professional work products > > or exempt from disclosure under applicable laws. Any unauthorized review, use, copying, disclosure, or distribution of any information contained > > in or attached to this transmission is STRICTLY PROHIBITED and may be against the laws. If you have received this message in error, or are not > > the named recipient(s), please immediately notify the sender by e-mail or telephone at IT support hotline 886-3-2700000 EXT 1333 and delete this > > e-mail message and any attached documentation from your computer. Receipt by anyone other than the intended recipient(s) is not a waiver of any > > attorney-client or work product privilege. Thank you! > > > > > > _______________________________________________ > > ltt-dev mailing list > > ltt-dev at lists.casi.polymtl.ca > > http://lists.casi.polymtl.ca/cgi-bin/mailman/listinfo/ltt-dev > > > > -- > Mathieu Desnoyers > OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F BA06 3F25 A8FE 3BAE 9A68 > > This e-mail message together with any attachments thereto (if any) is confidential, protected under an enforceable non-disclosure agreement, > intended only for the use of the named recipient(s) above and may contain information that is privileged, belonging to professional work products > or exempt from disclosure under applicable laws. Any unauthorized review, use, copying, disclosure, or distribution of any information contained > in or attached to this transmission is STRICTLY PROHIBITED and may be against the laws. If you have received this message in error, or are not > the named recipient(s), please immediately notify the sender by e-mail or telephone at IT support hotline 886-3-2700000 EXT 1333 and delete this > e-mail message and any attached documentation from your computer. Receipt by anyone other than the intended recipient(s) is not a waiver of any > attorney-client or work product privilege. Thank you! Content-Description: busybox version and command query.txt > # grep --help > BusyBox v1.12.4 (2009-07-29 23:17:25 CST) multi-call binary > > Usage: grep [-HhrilLnqvsoeFEABC] PATTERN [FILEs...] > > Search for PATTERN in each FILE or standard input > > Options: > -H Prefix output lines with filename where match was found > -h Suppress the prefixing filename on output > -r Recurse subdirectories > -i Ignore case distinctions > -l List names of files that match > -L List names of files that do not match > -n Print line number with output lines > -q Quiet. Return 0 if PATTERN is found, 1 otherwise > -v Select non-matching lines > -s Suppress file open/read error messages > -c Only print count of matching lines > -o Show only the part of a line that matches PATTERN > -m MAX Match up to MAX times per file > -F PATTERN is a set of newline-separated strings > -E PATTERN is an extended regular expression > -e PTRN Pattern to match > -f FILE Read pattern from file > -A Print NUM lines of trailing context > -B Print NUM lines of leading context > -C Print NUM lines of output context > > # awk --help > BusyBox v1.12.4 (2009-07-29 23:17:25 CST) multi-call binary > > Usage: awk [OPTION]... [program-text] [FILE...] > > Options: > -v var=val Set variable > -F sep Use sep as field separator > -f file Read program from file > > # sort --help > BusyBox v1.12.4 (2009-07-29 23:17:25 CST) multi-call binary > > Usage: sort [-nrugMcszbdfimSTokt] [-o FILE] [-k start[.offset][opts][,end[.offset][opts]] [-t CHAR] [FILE]... > > Sort lines of text > > Options: > -b Ignore leading blanks > -c Check whether input is sorted > -d Dictionary order (blank or alphanumeric only) > -f Ignore case > -g General numerical sort > -i Ignore unprintable characters > -k Sort key > -M Sort month > -n Sort numbers > -o Output to file > -k Sort by key > -t CHAR Key separator > -r Reverse sort order > -s Stable (don't sort ties alphabetically) > -u Suppress duplicate lines > -z Lines are terminated by NUL, not newline > -mST Ignored for GNU compatibility > > # cat --help > unknown option -- -usage: cat [-beflnstv] [-] [file ...] > # > Content-Description: LTTng Execution Log.txt > # ltt-armall-1 > Connecting all markers > Connecting fs_buffer_wait_end > Connecting fs_buffer_wait_start > Connecting fs_close > Connecting fs_exec > Connecting fs_ioctl > Connecting fs_llseek > Connecting fs_lseek > Connecting fs_open > Connecting fs_pollfd > Connecting fs_pread64 > Connecting fs_pwrite64 > Connecting fs_read > Connecting fs_readv > Connecting fs_select > Connecting fs_write > Connecting fs_writev > Connecting input_event > Connecting ipc_msg_create > Connecting ipc_sem_create > Connecting ipc_shm_create > Connecting kernel_arch_ipc_call > Connecting kernel_arch_kthread_create > Connecting kernel_arch_syscall_entry > Connecting kernel_arch_syscall_exit > Connecting kernel_arch_trap_entry > Connecting kernel_arch_trap_exit > Connecting kernel_irq_entry > Connecting kernel_irq_exit > Connecting kernel_kthread_stop > Connecting kernel_kthread_stop_ret > Connecting kernel_module_free > Connecting kernel_module_load > Connecting kernel_printk > Connecting kernel_process_exit > Connecting kernel_process_fork > Connecting kernel_process_free > Connecting kernel_process_wait > Connecting kernel_sched_migrate_task > Connecting kernel_sched_schedule > Connecting kernel_sched_try_wakeup > Connecting kernel_sched_wait_task > Connecting kernel_sched_wakeup_new_task > Connecting kernel_send_signal > Connecting kernel_softirq_entry > Connecting kernel_softirq_exit > Connecting kernel_softirq_raise > Connecting kernel_tasklet_high_entry > Connecting kernel_tasklet_high_exit > Connecting kernel_tasklet_low_entry > Connecting kernel_tasklet_low_exit > Connecting kernel_timer_itimer_expired > Connecting kernel_timer_itimer_set > Connecting kernel_timer_set > Connecting kernel_timer_timeout > Connecting kernel_timer_update_time > Connecting kernel_vprintk > Connecting list_file_descriptor > Connecting list_interrupt > Connecting list_module > Connecting list_network_ip_interface > Connecting list_network_ipv4_interface > Connecting list_process_state > Connecting list_statedump_end > Connecting list_vm_map > Connecting mm_handle_fault_entry > Connecting mm_handle_fault_exit > Connecting mm_huge_page_alloc > Connecting mm_huge_page_free > Connecting mm_page_alloc > Connecting mm_page_free > Connecting mm_wait_on_page_end > Connecting mm_wait_on_page_start > Connecting rcu_classic_callback > Connecting rcu_classic_call_rcu > Connecting rcu_classic_call_rcu_bh > Connecting statedump_softirq_vec > Connecting high-rate markers to tap > Connecting fs_buffer_wait_end > Connecting fs_buffer_wait_start > Connecting fs_close > Connecting fs_exec > Connecting fs_ioctl > Connecting fs_llseek > Connecting fs_lseek > Connecting fs_open > Connecting fs_pollfd > Connecting fs_pread64 > Connecting fs_pwrite64 > Connecting fs_read > Connecting fs_readv > Connecting fs_select > Connecting fs_write > Connecting fs_writev > Connecting input_event > Connecting ipc_msg_create > Connecting ipc_sem_create > Connecting ipc_shm_create > Connecting kernel_arch_ipc_call > Connecting kernel_arch_kthread_create > Connecting kernel_arch_syscall_entry > Connecting kernel_arch_syscall_exit > Connecting kernel_arch_trap_entry > Connecting kernel_arch_trap_exit > Connecting kernel_irq_entry > Connecting kernel_irq_exit > Connecting kernel_kthread_stop > Connecting kernel_kthread_stop_ret > Connecting kernel_module_free > Connecting kernel_module_load > Connecting kernel_printk > Connecting kernel_process_exit > Connecting kernel_process_fork > Connecting kernel_process_free > Connecting kernel_process_wait > Connecting kernel_sched_migrate_task > Connecting kernel_sched_schedule > Connecting kernel_sched_try_wakeup > Connecting kernel_sched_wait_task > Connecting kernel_sched_wakeup_new_task > Connecting kernel_send_signal > Connecting kernel_softirq_entry > Connecting kernel_softirq_exit > Connecting kernel_softirq_raise > Connecting kernel_tasklet_high_entry > Connecting kernel_tasklet_high_exit > Connecting kernel_tasklet_low_entry > Connecting kernel_tasklet_low_exit > Connecting kernel_timer_itimer_expired > Connecting kernel_timer_itimer_set > Connecting kernel_timer_set > Connecting kernel_timer_timeout > Connecting kernel_timer_update_time > Connecting kernel_vprintk > Connecting list_file_descriptor > Connecting list_interrupt > Connecting list_module > Connecting list_network_ip_interface > Connecting list_network_ipv4_interface > Connecting list_process_state > Connecting list_statedump_end > Connecting list_vm_map > Connecting mm_handle_fault_entry > Connecting mm_handle_fault_exit > Connecting mm_huge_page_alloc > Connecting mm_huge_page_free > Connecting mm_page_alloc > Connecting mm_page_free > Connecting mm_wait_on_page_end > Connecting mm_wait_on_page_start > Connecting rcu_classic_callback > Connecting rcu_classic_call_rcu > Connecting rcu_classic_call_rcu_bh > Connecting statedump_softirq_vec > > # cat /proc/ltt > marker: kernel_arch_kthread_create format: "pid %ld fn %p" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: kernel_arch_syscall_exit format: " " state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: kernel_arch_syscall_entry format: "syscall_id %d ip #p%ld" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: kernel_arch_ipc_call format: "call %u first %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: kernel_arch_trap_exit format: " " state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: kernel_arch_trap_entry format: "trap_id %ld ip #p%ld" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: statedump_softirq_vec format: "id %d address %p symbol %s" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: list_module format: "name %s state %d refcount %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: core_marker_format format: "name %s format %s" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: core_marker_format format: "name %s format %s" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: list_statedump_end format: " " state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: list_process_state format: "pid %d parent_pid %d name %s type %d mode %d submode %d status %d tgid %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: list_file_descriptor format: "filename %s pid %d fd %u" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: list_vm_map format: "pid %d start %lu end %lu flags %lu pgoff %lu inode %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: list_interrupt format: "name %s action %s irq_id %u" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: list_network_ip_interface format: "name %s address #4u%lu up %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: list_network_ipv4_interface format: "name %s address #4u%lu up %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: core_marker_id format: "name %s id %hu int #1u%zu long #1u%zu pointer #1u%zu size_t #1u%zu alignment #1u%u" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: core_marker_id format: "name %s id %hu int #1u%zu long #1u%zu pointer #1u%zu size_t #1u%zu alignment #1u%u" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: core_marker_format format: "name %s format %s" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: core_marker_id format: "name %s id %hu int #1u%zu long #1u%zu pointer #1u%zu size_t #1u%zu alignment #1u%u" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: core_marker_id format: "name %s id %hu int #1u%zu long #1u%zu pointer #1u%zu size_t #1u%zu alignment #1u%u" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: kernel_module_load format: "name %s" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: kernel_module_free format: "name %s" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: kernel_vprintk format: "loglevel %c string %s ip %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: kernel_printk format: "ip %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: kernel_timer_timeout format: "pid %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: kernel_timer_update_time format: "jiffies #8u%llu xtime_sec %ld xtime_nsec %ld walltomonotonic_sec %ld walltomonotonic_nsec %ld" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: kernel_timer_set format: "expires %lu function %p data %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: kernel_timer_itimer_set format: "which %d interval_sec %ld interval_usec %ld value_sec %ld value_usec %ld" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: kernel_timer_itimer_expired format: "pid %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: kernel_process_fork format: "parent_pid %d child_pid %d child_tgid %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: kernel_process_wait format: "pid %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: kernel_process_exit format: "pid %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: kernel_process_free format: "pid %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: kernel_send_signal format: "pid %d signal %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: kernel_sched_migrate_task format: "pid %d state %ld dest_cpu %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: kernel_sched_schedule format: "prev_pid %d next_pid %d prev_state %ld" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: kernel_sched_wakeup_new_task format: "pid %d state %ld cpu_id %u" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: kernel_sched_try_wakeup format: "pid %d state %ld cpu_id %u" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: kernel_sched_wait_task format: "pid %d state %ld" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: kernel_kthread_stop_ret format: "ret %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: kernel_kthread_stop format: "pid %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: kernel_tasklet_high_exit format: "func %p data %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: kernel_tasklet_high_entry format: "func %p data %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: kernel_tasklet_low_exit format: "func %p data %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: kernel_tasklet_low_entry format: "func %p data %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: kernel_softirq_raise format: "softirq_id %u" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: kernel_softirq_exit format: "softirq_id %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: kernel_softirq_entry format: "softirq_id %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: kernel_irq_exit format: "handled #1u%u" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: kernel_irq_entry format: "irq_id %u kernel_mode %u ip %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: mm_page_alloc format: "order %u pfn %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: mm_page_free format: "order %u pfn %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: mm_handle_fault_exit format: "res %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: mm_handle_fault_entry format: "address %lu ip #p%ld write_access %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: mm_huge_page_alloc format: "pfn %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: mm_huge_page_free format: "pfn %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: mm_wait_on_page_end format: "pfn %lu bit_nr %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: mm_wait_on_page_start format: "pfn %lu bit_nr %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: fs_pollfd format: "fd %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: fs_select format: "fd %d timeout #8d%lld" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: fs_writev format: "fd %lu vlen %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: fs_readv format: "fd %lu vlen %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: fs_pwrite64 format: "fd %u count %zu pos %llu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: fs_pread64 format: "fd %u count %zu pos %llu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: fs_write format: "fd %u count %zu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: fs_read format: "fd %u count %zu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: fs_llseek format: "fd %u offset %lld origin %u" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: fs_lseek format: "fd %u offset %ld origin %u" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: fs_close format: "fd %u" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: fs_open format: "fd %d filename %s" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: fs_ioctl format: "fd %u cmd %u arg %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: fs_exec format: "filename %s" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: fs_buffer_wait_end format: "bh %p" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: fs_buffer_wait_start format: "bh %p" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: ipc_shm_create format: "id %ld flags %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: ipc_sem_create format: "id %ld flags %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: ipc_msg_create format: "id %ld flags %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: lockdep_lock_release format: "retaddr 0x%lX lock %p nested %d" state: 0 call: 0xc0093374 probe single : 0xc00920ec > marker: lockdep_lock_acquire format: "retaddr 0x%lX subclass %u lock %p trylock %d read %d hardirqs_off %d" state: 0 call: 0xc0093374 probe single : 0xc00920ec > marker: lockdep_softirqs_off format: "retaddr 0x%lX" state: 0 call: 0xc0093374 probe single : 0xc00920ec > marker: lockdep_softirqs_on format: "retaddr 0x%lX" state: 0 call: 0xc0093374 probe single : 0xc00920ec > marker: lockdep_hardirqs_off format: "retaddr 0x%lX" state: 0 call: 0xc0093374 probe single : 0xc00920ec > marker: lockdep_hardirqs_on format: "retaddr 0x%lX" state: 0 call: 0xc0093374 probe single : 0xc00920ec > marker: rcu_classic_call_rcu_bh format: "func %p ip 0x%lX" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: rcu_classic_call_rcu format: "func %p ip 0x%lX" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: rcu_classic_callback format: "func %p" state: 1 call: 0xc0093374 probe single : 0xc015d504 > marker: input_event format: "type %u code %u value %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > [Bryan] Maybe caused by privilege, the the /proc/ltt do not be replaced with content below. It's that will be a problem? > connect statedump_softirq_vec ltt_tap_marker dynami > > # df > /dev: 39564K total, 0K used, 39564K available (block size 4096) > /sqlite_stmt_journals: 4096K total, 0K used, 4096K available (block size 4096) > /system: 72704K total, 71680K used, 1024K available (block size 4096) > /data: 66816K total, 37812K used, 29004K available (block size 4096) > /cache: 67584K total, 1156K used, 66428K available (block size 4096) > /sdcard: 991708K total, 147040K used, 844668K available (block size 4096) > > # lttctl -n trace1 -d -l /sys/kernel/debug/ltt -t /data/ltt/trace1 > Linux Trace Toolkit Trace Control 0.55-16102008 > > Controlling trace : trace1 > > Linux Trace Toolkit Trace Daemon 0.55-16102008 > > Reading from debugfs directory : /sys/kernel/debug/ltt/trace1 > Writing to trace directory : /data/ltt/trace1 > > [Bryan] there is 29004K available for /data/ltt/trace1, is that enough? > [Bryan] do some tests on target device, then check the channel and trace > > # ls -l /sys/kernel/debug/ltt/trace1 > -r-------- root root 0 1980-01-06 00:10 cpu_0 > drwxr-xr-x root root 1980-01-06 00:10 control > # ls -l /sys/kernel/debug/ltt/trace1/control > -r-------- root root 0 1980-01-06 00:10 network_0 > -r-------- root root 0 1980-01-06 00:10 modules_0 > -r-------- root root 0 1980-01-06 00:10 processes_0 > -r-------- root root 0 1980-01-06 00:10 interrupts_0 > -r-------- root root 0 1980-01-06 00:10 metadata_0 > # ls -l /data/ltt/trace1 > drwxrwxrwx root root 1980-01-06 00:10 control > -rwxrwxrwx root root 29949952 1980-01-06 00:10 cpu_0 > # ls -l /data/ltt/trace1/control > -rwxrwxrwx root root 0 1980-01-06 00:10 metadata_0 > -rwxrwxrwx root root 0 1980-01-06 00:10 interrupts_0 > -rwxrwxrwx root root 0 1980-01-06 00:10 processes_0 > -rwxrwxrwx root root 0 1980-01-06 00:10 modules_0 > -rwxrwxrwx root root 0 1980-01-06 00:10 network_0 > > [Bryan] It seems that the space has been exhausted by cpu_0, it's that correct that cpu_0 will grow to so huge(29M)? > How much file space should I prepare for LTTng? > > # lttctl -n trace1 -d -R > Linux Trace Toolkit Trace Control 0.55-16102008 > > Controlling trace : trace1 > > [Bryan] hang in here...I have to break it by Ctrl+Z, What's problem? > > ^Z[1] + Stopped lttctl -n trace1 -d -R > # ls -l /data/ltt/trace1/ > drwxrwxrwx root root 1980-01-06 00:10 control > -rwxrwxrwx root root 29949952 1980-01-06 00:10 cpu_0 > # ls -l /data/ltt/trace1/control > -rwxrwxrwx root root 0 1980-01-06 00:10 metadata_0 > -rwxrwxrwx root root 0 1980-01-06 00:10 interrupts_0 > -rwxrwxrwx root root 0 1980-01-06 00:10 processes_0 > -rwxrwxrwx root root 0 1980-01-06 00:10 modules_0 > -rwxrwxrwx root root 0 1980-01-06 00:10 network_0 > # ls -l /sys/kernel/debug/ltt/trace1 > -r-------- root root 0 1980-01-06 00:10 cpu_0 > drwxr-xr-x root root 1980-01-06 00:10 control > # ls -l /sys/kernel/debug/ltt/trace1/control > -r-------- root root 0 1980-01-06 00:10 network_0 > -r-------- root root 0 1980-01-06 00:10 modules_0 > -r-------- root root 0 1980-01-06 00:10 processes_0 > -r-------- root root 0 1980-01-06 00:10 interrupts_0 > -r-------- root root 0 1980-01-06 00:10 metadata_0 > # df > /dev: 39564K total, 0K used, 39564K available (block size 4096) > /sqlite_stmt_journals: 4096K total, 0K used, 4096K available (block size 4096) > /system: 72704K total, 71680K used, 1024K available (block size 4096) > /data: 66816K total, 66732K used, 84K available (block size 4096) > /cache: 67584K total, 1156K used, 66428K available (block size 4096) > /sdcard: 991708K total, 147040K used, 844668K available (block size 4096) > > > Content-Description: LTTng Kernel Config.txt > > [ ] Kernel->user space relay support (formerly relayfs) > [*] Activate tracepoints x x > [*] Activate markers x x > Linux Trace Toolkit ---> > <*> Compile lttng tracing probes x x > [*] LTTng fine-grained timestamping x x > [*] Linux Trace Toolkit Instrumentation Support x x > <*> Linux Trace Toolkit High-speed Lockless Data Relay x x > <*> Linux Trace Toolkit Lock-Protected Data Relay x x > [*] Debug check for random access in ltt relay buffers x x > <*> Linux Trace Toolkit Serializer x x > <*> Linux Trace Toolkit Marker Control x x > <*> Linux Trace Toolkit Tracer x x > [*] Align Linux Trace Toolkit Traces x x > [*] Add event size field to LTT events for tracer debugging x x > [*] Support trace extraction from crash dump x x > <*> Linux Trace Toolkit Netlink Controller x x > <*> Linux Trace Toolkit State Dump -- Mathieu Desnoyers OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F BA06 3F25 A8FE 3BAE 9A68 This e-mail message together with any attachments thereto (if any) is confidential, protected under an enforceable non-disclosure agreement, intended only for the use of the named recipient(s) above and may contain information that is privileged, belonging to professional work products or exempt from disclosure under applicable laws. Any unauthorized review, use, copying, disclosure, or distribution of any information contained in or attached to this transmission is STRICTLY PROHIBITED and may be against the laws. If you have received this message in error, or are not the named recipient(s), please immediately notify the sender by e-mail or telephone at IT support hotline 886-3-2700000 EXT 1333 and delete this e-mail message and any attached documentation from your computer. Receipt by anyone other than the intended recipient(s) is not a waiver of any attorney-client or work product privilege. Thank you! ^ permalink raw reply [flat|nested] 9+ messages in thread
* [ltt-dev] Some LTTng 0.43 issue on ARM 2009-08-06 2:52 ` Bryan Hsieh @ 2009-08-06 17:20 ` Mathieu Desnoyers 2009-08-06 17:24 ` Mathieu Desnoyers 2009-08-27 8:56 ` [ltt-dev] Change process name to be command line Bryan Hsieh 1 sibling, 1 reply; 9+ messages in thread From: Mathieu Desnoyers @ 2009-08-06 17:20 UTC (permalink / raw) * Bryan Hsieh (bryanhsieh at fihspec.com) wrote: > Hi Mathieu, All > > [Bryan] > I modified install path and re-compile, then install to my target system, even I replace the original toolbox with new busybox, it still doesn't work, I try to use older version busybox and install to my target system, finally It's work! I watch the space change and stop trace before the capacity been exhausted, the lttd can stop successfully. Thank LTT Gods! > # ls -l /data/ltt/trace1 > drwxrwxrwx 1 0 0 2048 Jan 6 01:17 control > -rwxrwxrwx 1 0 0 25165824 Jan 6 01:17 cpu_0 > # ls -l /data/ltt/trace1/control > -rwxrwxrwx 1 0 0 65536 Jan 6 01:17 interrupts_0 > -rwxrwxrwx 1 0 0 65536 Jan 6 01:17 metadata_0 > -rwxrwxrwx 1 0 0 65536 Jan 6 01:17 modules_0 > -rwxrwxrwx 1 0 0 65536 Jan 6 01:17 network_0 > -rwxrwxrwx 1 0 0 262144 Jan 6 01:17 processes_0 > > [Bryan] > I have another problem, I can store trace in /data or /cache directory by -t option, but I still can't store trace in /sdcard directory, the files size are empty in /sdcard/ltt/trace1, what's wrong? > > # mount > rootfs on / type rootfs (ro) > tmpfs on /dev type tmpfs (rw,mode=755) > devpts on /dev/pts type devpts (rw,mode=600) > proc on /proc type proc (rw) > sysfs on /sys type sysfs (rw) > debugfs on /sys/kernel/debug type debugfs (rw) > tmpfs on /sqlite_stmt_journals type tmpfs (rw,size=4096k) > /dev/block/mtdblock3 on /system type yaffs2 (rw) > /dev/block/mtdblock6 on /data type yaffs2 (rw,nosuid,nodev) > /dev/block/mtdblock4 on /cache type yaffs2 (rw,nosuid,nodev) > /dev/block//vold/179:1 on /sdcard type vfat (rw,dirsync,nosuid,nodev,noexec,uid=1000,gid=1000,fmask=0711,dmask=0700,allow_utime=0022,codepage=cp437,iocharset=iso8859-1,shortname=mixed,utf8) Hrm, I guess it's a problem with vfat splice support. You try to write it to a NFS mounted drive instead. Mathieu > > > Best regards, > Bryan Hsieh > > SPEC, FIH Co., Ltd. > Tel:+886-3-270-0000 #2723 > Fax:+886-3-270-0001 > > > -----Original Message----- > From: Mathieu Desnoyers [mailto:compudj@krystal.dyndns.org] > Sent: Tuesday, August 04, 2009 9:18 PM > To: Bryan Hsieh > Cc: 'Ajit R'; 'ltt-dev at lists.casi.polymtl.ca'; Tommy Peng > Subject: Re: [ltt-dev] Some LTTng 0.43 issue on ARM > > * Bryan Hsieh (bryanhsieh at fihspec.com) wrote: > > Hi Mathieu, All > > > > Please check my comment which marked by [Bryan], Thanks! > > ----------------------------------------------------------------------------- > > [Bryan] > > All the related LTT functions are build-in not build as modules, Please check attachment- LTTng Kernel Config..txt, I am not sure if Kernel->user space relay support (formerly relayfs) need to build-in. > > Is there any item I missed? > > ----------------------------------------------------------------------------- > > [ ] Kernel->user space relay support (formerly relayfs) > > LTTng does not need relayfs anymore. So this is ok. > > > [*] Activate tracepoints > > [*] Activate markers > > Linux Trace Toolkit ---> > > <*> Compile lttng tracing probes > > [*] LTTng fine-grained timestamping > > [*] Linux Trace Toolkit Instrumentation Support > > <*> Linux Trace Toolkit High-speed Lockless Data Relay > > <*> Linux Trace Toolkit Lock-Protected Data Relay > > [*] Debug check for random access in ltt relay buffers > > <*> Linux Trace Toolkit Serializer > > <*> Linux Trace Toolkit Marker Control > > <*> Linux Trace Toolkit Tracer > > [*] Align Linux Trace Toolkit Traces > > [*] Add event size field to LTT events for tracer debugging > > [*] Support trace extraction from crash dump > > <*> Linux Trace Toolkit Netlink Controller > > <*> Linux Trace Toolkit State Dump > > ----------------------------------------------------------------------------- > > [Bryan] > > I am sure that the external utilities are build-in busybox, please check the attachment - " busybox version and command query.txt". > > > > Finally, I figure out why the ltt-armall doesn't work correctly in my target system, root cause is the output pipeline will be empty after grep twice, even I use a temp file to avoid connecting two grep commands directly, it's still empty. So I copy the /proc/ltt to my local desktop to do ltt-armall script, then generate the /proc/ltt which been filtered by following command. > > MARKERS=`cat /proc/ltt|grep -v %k|awk '{print $2}'|sort -u|grep -v ^core_|grep -v ^locking_|grep -v ^lockdep_` > > I rename it as ltt_filter and copy to my target system path /system/bin/, then modify ltt-armall by following modification and run it. > > MARKERS=`cat /system/bin/ltt_filter` > > > > The execution result on screen: > > # ltt-armall-1 > > Connecting all markers > > Connecting fs_buffer_wait_end > > Connecting fs_buffer_wait_start > > Connecting fs_close > > Connecting fs_exec > > Connecting fs_ioctl > > Connecting fs_llseek > > Connecting fs_lseek > > Connecting fs_open > > Connecting fs_pollfd > > Connecting fs_pread64 > > Connecting fs_pwrite64 > > Connecting fs_read > > Connecting fs_readv > > Connecting fs_select > > Connecting fs_write > > Connecting fs_writev > > Connecting input_event > > Connecting ipc_msg_create > > Connecting ipc_sem_create > > Connecting ipc_shm_create > > Connecting kernel_arch_ipc_call > > Connecting kernel_arch_kthread_create > > Connecting kernel_arch_syscall_entry > > Connecting kernel_arch_syscall_exit > > Connecting kernel_arch_trap_entry > > Connecting kernel_arch_trap_exit > > Connecting kernel_irq_entry > > Connecting kernel_irq_exit > > Connecting kernel_kthread_stop > > Connecting kernel_kthread_stop_ret > > Connecting kernel_module_free > > Connecting kernel_module_load > > Connecting kernel_printk > > Connecting kernel_process_exit > > Connecting kernel_process_fork > > Connecting kernel_process_free > > Connecting kernel_process_wait > > Connecting kernel_sched_migrate_task > > Connecting kernel_sched_schedule > > Connecting kernel_sched_try_wakeup > > Connecting kernel_sched_wait_task > > Connecting kernel_sched_wakeup_new_task > > Connecting kernel_send_signal > > Connecting kernel_softirq_entry > > Connecting kernel_softirq_exit > > Connecting kernel_softirq_raise > > Connecting kernel_tasklet_high_entry > > Connecting kernel_tasklet_high_exit > > Connecting kernel_tasklet_low_entry > > Connecting kernel_tasklet_low_exit > > Connecting kernel_timer_itimer_expired > > Connecting kernel_timer_itimer_set > > Connecting kernel_timer_set > > Connecting kernel_timer_timeout > > Connecting kernel_timer_update_time > > Connecting kernel_vprintk > > Connecting list_file_descriptor > > Connecting list_interrupt > > Connecting list_module > > Connecting list_network_ip_interface > > Connecting list_network_ipv4_interface > > Connecting list_process_state > > Connecting list_statedump_end > > Connecting list_vm_map > > Connecting mm_handle_fault_entry > > Connecting mm_handle_fault_exit > > Connecting mm_huge_page_alloc > > Connecting mm_huge_page_free > > Connecting mm_page_alloc > > Connecting mm_page_free > > Connecting mm_wait_on_page_end > > Connecting mm_wait_on_page_start > > Connecting rcu_classic_callback > > Connecting rcu_classic_call_rcu > > Connecting rcu_classic_call_rcu_bh > > Connecting statedump_softirq_vec > > Connecting high-rate markers to tap > > Connecting fs_buffer_wait_end > > Connecting fs_buffer_wait_start > > Connecting fs_close > > Connecting fs_exec > > Connecting fs_ioctl > > Connecting fs_llseek > > Connecting fs_lseek > > Connecting fs_open > > Connecting fs_pollfd > > Connecting fs_pread64 > > Connecting fs_pwrite64 > > Connecting fs_read > > Connecting fs_readv > > Connecting fs_select > > Connecting fs_write > > Connecting fs_writev > > Connecting input_event > > Connecting ipc_msg_create > > Connecting ipc_sem_create > > Connecting ipc_shm_create > > Connecting kernel_arch_ipc_call > > Connecting kernel_arch_kthread_create > > Connecting kernel_arch_syscall_entry > > Connecting kernel_arch_syscall_exit > > Connecting kernel_arch_trap_entry > > Connecting kernel_arch_trap_exit > > Connecting kernel_irq_entry > > Connecting kernel_irq_exit > > Connecting kernel_kthread_stop > > Connecting kernel_kthread_stop_ret > > Connecting kernel_module_free > > Connecting kernel_module_load > > Connecting kernel_printk > > Connecting kernel_process_exit > > Connecting kernel_process_fork > > Connecting kernel_process_free > > Connecting kernel_process_wait > > Connecting kernel_sched_migrate_task > > Connecting kernel_sched_schedule > > Connecting kernel_sched_try_wakeup > > Connecting kernel_sched_wait_task > > Connecting kernel_sched_wakeup_new_task > > Connecting kernel_send_signal > > Connecting kernel_softirq_entry > > Connecting kernel_softirq_exit > > Connecting kernel_softirq_raise > > Connecting kernel_tasklet_high_entry > > Connecting kernel_tasklet_high_exit > > Connecting kernel_tasklet_low_entry > > Connecting kernel_tasklet_low_exit > > Connecting kernel_timer_itimer_expired > > Connecting kernel_timer_itimer_set > > Connecting kernel_timer_set > > Connecting kernel_timer_timeout > > Connecting kernel_timer_update_time > > Connecting kernel_vprintk > > Connecting list_file_descriptor > > Connecting list_interrupt > > Connecting list_module > > Connecting list_network_ip_interface > > Connecting list_network_ipv4_interface > > Connecting list_process_state > > Connecting list_statedump_end > > Connecting list_vm_map > > Connecting mm_handle_fault_entry > > Connecting mm_handle_fault_exit > > Connecting mm_huge_page_alloc > > Connecting mm_huge_page_free > > Connecting mm_page_alloc > > Connecting mm_page_free > > Connecting mm_wait_on_page_end > > Connecting mm_wait_on_page_start > > Connecting rcu_classic_callback > > Connecting rcu_classic_call_rcu > > Connecting rcu_classic_call_rcu_bh > > Connecting statedump_softirq_vec > > ----------------------------------------------------------------------------- > > [Bryan] It's that correct result? > > Yes, this output is correct. So you probably need to recompile busybox > with awk and/or sed and/or sort support..... > > > ----------------------------------------------------------------------------- > > # cat /proc/ltt > > marker: kernel_arch_kthread_create format: "pid %ld fn %p" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: kernel_arch_syscall_exit format: " " state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: kernel_arch_syscall_entry format: "syscall_id %d ip #p%ld" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: kernel_arch_ipc_call format: "call %u first %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: kernel_arch_trap_exit format: " " state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: kernel_arch_trap_entry format: "trap_id %ld ip #p%ld" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: statedump_softirq_vec format: "id %d address %p symbol %s" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: list_module format: "name %s state %d refcount %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: core_marker_format format: "name %s format %s" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: core_marker_format format: "name %s format %s" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: list_statedump_end format: " " state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: list_process_state format: "pid %d parent_pid %d name %s type %d mode %d submode %d status %d tgid %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: list_file_descriptor format: "filename %s pid %d fd %u" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: list_vm_map format: "pid %d start %lu end %lu flags %lu pgoff %lu inode %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: list_interrupt format: "name %s action %s irq_id %u" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: list_network_ip_interface format: "name %s address #4u%lu up %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: list_network_ipv4_interface format: "name %s address #4u%lu up %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: core_marker_id format: "name %s id %hu int #1u%zu long #1u%zu pointer #1u%zu size_t #1u%zu alignment #1u%u" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: core_marker_id format: "name %s id %hu int #1u%zu long #1u%zu pointer #1u%zu size_t #1u%zu alignment #1u%u" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: core_marker_format format: "name %s format %s" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: core_marker_id format: "name %s id %hu int #1u%zu long #1u%zu pointer #1u%zu size_t #1u%zu alignment #1u%u" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: core_marker_id format: "name %s id %hu int #1u%zu long #1u%zu pointer #1u%zu size_t #1u%zu alignment #1u%u" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: kernel_module_load format: "name %s" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: kernel_module_free format: "name %s" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: kernel_vprintk format: "loglevel %c string %s ip %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: kernel_printk format: "ip %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: kernel_timer_timeout format: "pid %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: kernel_timer_update_time format: "jiffies #8u%llu xtime_sec %ld xtime_nsec %ld walltomonotonic_sec %ld walltomonotonic_nsec %ld" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: kernel_timer_set format: "expires %lu function %p data %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: kernel_timer_itimer_set format: "which %d interval_sec %ld interval_usec %ld value_sec %ld value_usec %ld" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: kernel_timer_itimer_expired format: "pid %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: kernel_process_fork format: "parent_pid %d child_pid %d child_tgid %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: kernel_process_wait format: "pid %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: kernel_process_exit format: "pid %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: kernel_process_free format: "pid %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: kernel_send_signal format: "pid %d signal %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: kernel_sched_migrate_task format: "pid %d state %ld dest_cpu %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: kernel_sched_schedule format: "prev_pid %d next_pid %d prev_state %ld" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: kernel_sched_wakeup_new_task format: "pid %d state %ld cpu_id %u" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: kernel_sched_try_wakeup format: "pid %d state %ld cpu_id %u" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: kernel_sched_wait_task format: "pid %d state %ld" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: kernel_kthread_stop_ret format: "ret %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: kernel_kthread_stop format: "pid %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: kernel_tasklet_high_exit format: "func %p data %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: kernel_tasklet_high_entry format: "func %p data %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: kernel_tasklet_low_exit format: "func %p data %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: kernel_tasklet_low_entry format: "func %p data %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: kernel_softirq_raise format: "softirq_id %u" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: kernel_softirq_exit format: "softirq_id %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: kernel_softirq_entry format: "softirq_id %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: kernel_irq_exit format: "handled #1u%u" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: kernel_irq_entry format: "irq_id %u kernel_mode %u ip %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: mm_page_alloc format: "order %u pfn %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: mm_page_free format: "order %u pfn %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: mm_handle_fault_exit format: "res %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: mm_handle_fault_entry format: "address %lu ip #p%ld write_access %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: mm_huge_page_alloc format: "pfn %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: mm_huge_page_free format: "pfn %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: mm_wait_on_page_end format: "pfn %lu bit_nr %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: mm_wait_on_page_start format: "pfn %lu bit_nr %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: fs_pollfd format: "fd %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: fs_select format: "fd %d timeout #8d%lld" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: fs_writev format: "fd %lu vlen %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: fs_readv format: "fd %lu vlen %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: fs_pwrite64 format: "fd %u count %zu pos %llu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: fs_pread64 format: "fd %u count %zu pos %llu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: fs_write format: "fd %u count %zu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: fs_read format: "fd %u count %zu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: fs_llseek format: "fd %u offset %lld origin %u" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: fs_lseek format: "fd %u offset %ld origin %u" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: fs_close format: "fd %u" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: fs_open format: "fd %d filename %s" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: fs_ioctl format: "fd %u cmd %u arg %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: fs_exec format: "filename %s" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: fs_buffer_wait_end format: "bh %p" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: fs_buffer_wait_start format: "bh %p" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: ipc_shm_create format: "id %ld flags %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: ipc_sem_create format: "id %ld flags %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: ipc_msg_create format: "id %ld flags %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: lockdep_lock_release format: "retaddr 0x%lX lock %p nested %d" state: 0 call: 0xc0093374 probe single : 0xc00920ec > > marker: lockdep_lock_acquire format: "retaddr 0x%lX subclass %u lock %p trylock %d read %d hardirqs_off %d" state: 0 call: 0xc0093374 probe single : 0xc00920ec > > marker: lockdep_softirqs_off format: "retaddr 0x%lX" state: 0 call: 0xc0093374 probe single : 0xc00920ec > > marker: lockdep_softirqs_on format: "retaddr 0x%lX" state: 0 call: 0xc0093374 probe single : 0xc00920ec > > marker: lockdep_hardirqs_off format: "retaddr 0x%lX" state: 0 call: 0xc0093374 probe single : 0xc00920ec > > marker: lockdep_hardirqs_on format: "retaddr 0x%lX" state: 0 call: 0xc0093374 probe single : 0xc00920ec > > marker: rcu_classic_call_rcu_bh format: "func %p ip 0x%lX" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: rcu_classic_call_rcu format: "func %p ip 0x%lX" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: rcu_classic_callback format: "func %p" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: input_event format: "type %u code %u value %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > ----------------------------------------------------------------------------- > > [Bryan] But the /proc/ltt is still same, it's that correct? What's the expected result? > > This output is good. busybox problem, as I said above. Try each command > used by busybox to test them. > > > ----------------------------------------------------------------------------- > > # df > > /dev: 39564K total, 0K used, 39564K available (block size 4096) > > /sqlite_stmt_journals: 4096K total, 0K used, 4096K available (block size 4096) > > /system: 72704K total, 71680K used, 1024K available (block size 4096) > > /data: 66816K total, 37812K used, 29004K available (block size 4096) > > /cache: 67584K total, 1156K used, 66428K available (block size 4096) > > /sdcard: 991708K total, 147040K used, 844668K available (block size 4096) > > ----------------------------------------------------------------------------- > > [Bryan] There is 29004K available for trace log file /data/ltt/trace1. Is that enough? > > No. I would save it to /sdcard. > > > ----------------------------------------------------------------------------- > > # cat /proc/meminfo > > MemTotal: 79128 kB > > MemFree: 2440 kB > > Buffers: 36 kB > > Cached: 29872 kB > > SwapCached: 0 kB > > Active: 61384 kB > > Inactive: 5052 kB > > SwapTotal: 0 kB > > SwapFree: 0 kB > > Dirty: 0 kB > > Writeback: 0 kB > > AnonPages: 36552 kB > > Mapped: 24888 kB > > Slab: 3856 kB > > SReclaimable: 904 kB > > SUnreclaim: 2952 kB > > PageTables: 3660 kB > > NFS_Unstable: 0 kB > > Bounce: 0 kB > > WritebackTmp: 0 kB > > CommitLimit: 39564 kB > > Committed_AS: 1032216 kB > > VmallocTotal: 409600 kB > > VmallocUsed: 52908 kB > > VmallocChunk: 332796 kB > > ----------------------------------------------------------------------------- > > [Bryan] > > There is 2440 kB free memory. Is that enough? > > I think that yes. The page allocator will kick out Cached pages when > needed. > > > ----------------------------------------------------------------------------- > > # lttctl -n trace1 -d -l /sys/kernel/debug/ltt -t /data/ltt/trace1 > > Linux Trace Toolkit Trace Control 0.55-16102008 > > > > Controlling trace : trace1 > > > > Linux Trace Toolkit Trace Daemon 0.55-16102008 > > > > Reading from debugfs directory : /sys/kernel/debug/ltt/trace1 > > Writing to trace directory : /data/ltt/trace1 > > ----------------------------------------------------------------------------- > > [Bryan] do some tests on target device, then check the channel and trace > > ----------------------------------------------------------------------------- > > # ls -l /sys/kernel/debug/ltt/trace1 > > -r-------- root root 0 1980-01-06 00:10 cpu_0 > > drwxr-xr-x root root 1980-01-06 00:10 control > > # ls -l /sys/kernel/debug/ltt/trace1/control > > -r-------- root root 0 1980-01-06 00:10 network_0 > > -r-------- root root 0 1980-01-06 00:10 modules_0 > > -r-------- root root 0 1980-01-06 00:10 processes_0 > > -r-------- root root 0 1980-01-06 00:10 interrupts_0 > > -r-------- root root 0 1980-01-06 00:10 metadata_0 > > # ls -l /data/ltt/trace1 > > drwxrwxrwx root root 1980-01-06 00:10 control > > -rwxrwxrwx root root 29949952 1980-01-06 00:10 cpu_0 > > # ls -l /data/ltt/trace1/control > > -rwxrwxrwx root root 0 1980-01-06 00:10 metadata_0 > > -rwxrwxrwx root root 0 1980-01-06 00:10 interrupts_0 > > -rwxrwxrwx root root 0 1980-01-06 00:10 processes_0 > > -rwxrwxrwx root root 0 1980-01-06 00:10 modules_0 > > -rwxrwxrwx root root 0 1980-01-06 00:10 network_0 > > ----------------------------------------------------------------------------- > > [Bryan] It seems that the capacity has been exhausted by cpu_0, because 29004K available. > > It's that correct that cpu_0 will grow to so huge (29M)? > > 29M is not huge. It is small :) Especially if the system is very busy. > > > ----------------------------------------------------------------------------- > > # lttctl -n trace1 -d -R > > Linux Trace Toolkit Trace Control 0.55-16102008 > > > > Controlling trace : trace1 > > ^Z[1] + Stopped lttctl -n trace1 -d -R > > ----------------------------------------------------------------------------- > > [Bryan] hang in here...I have to break it by Ctrl+Z, What's problem? > > It's waiting for lttd to release the trace channels, but I guess that > the lack of space is causing errors, so lttd keeps running. > > > ----------------------------------------------------------------------------- > > # ls -l /data/ltt/trace1/ > > drwxrwxrwx root root 1980-01-06 00:10 control > > -rwxrwxrwx root root 29949952 1980-01-06 00:10 cpu_0 > > # ls -l /data/ltt/trace1/control > > -rwxrwxrwx root root 0 1980-01-06 00:10 metadata_0 > > -rwxrwxrwx root root 0 1980-01-06 00:10 interrupts_0 > > -rwxrwxrwx root root 0 1980-01-06 00:10 processes_0 > > -rwxrwxrwx root root 0 1980-01-06 00:10 modules_0 > > -rwxrwxrwx root root 0 1980-01-06 00:10 network_0 > > # ls -l /sys/kernel/debug/ltt/trace1 > > -r-------- root root 0 1980-01-06 00:10 cpu_0 > > drwxr-xr-x root root 1980-01-06 00:10 control > > # ls -l /sys/kernel/debug/ltt/trace1/control > > -r-------- root root 0 1980-01-06 00:10 network_0 > > -r-------- root root 0 1980-01-06 00:10 modules_0 > > -r-------- root root 0 1980-01-06 00:10 processes_0 > > -r-------- root root 0 1980-01-06 00:10 interrupts_0 > > -r-------- root root 0 1980-01-06 00:10 metadata_0 > > # df > > /dev: 39564K total, 0K used, 39564K available (block size 4096) > > /sqlite_stmt_journals: 4096K total, 0K used, 4096K available (block size 4096) > > /system: 72704K total, 71680K used, 1024K available (block size 4096) > > /data: 66816K total, 66732K used, 84K available (block size 4096) > > /cache: 67584K total, 1156K used, 66428K available (block size 4096) > > /sdcard: 991708K total, 147040K used, 844668K available (block size 4096) > > ----------------------------------------------------------------------------- > > [Bryan] I think the trace is incomplete. > > Yes, here /data only has 84K avail. It's full. > > Mathieu > > > ----------------------------------------------------------------------------- > > > > Please help me! Thank you very much! > > > > Best regards, > > Bryan Hsieh > > > > SPEC, FIH Co., Ltd. > > Tel:+886-3-270-0000 #2723 > > Fax:+886-3-270-0001 > > > > > > -----Original Message----- > > From: Mathieu Desnoyers [mailto:compudj@krystal.dyndns.org] > > Sent: Friday, July 31, 2009 11:43 PM > > To: Bryan Hsieh > > Cc: 'Ajit R'; 'ltt-dev at lists.casi.polymtl.ca'; Tommy Peng > > Subject: Re: [ltt-dev] Some LTTng 0.43 issue on ARM > > > > * Bryan Hsieh (bryanhsieh at fihspec.com) wrote: > > > Hi Ajit, > > > > > > I appreciate your reply! > > > But there is a problem here. > > > > > > My installation of Busybox is 1.12.4. > > > > It sounds like either awk or sed or not built in your busybox > > installation. > > > > Can you provide the output of > > > > cat /proc/ltt > > > > on your target system ? > > > > Check the content of ltt-armall to see which external programs it needs. > > > > Also, if you built the lttng probes as modules, you may need to load > > them explicitely. > > > > Mathieu > > > > > > > > > > I follow your suggestion to test on x86 Desktop ARM simulator, the message said that can't find /proc/ltt. I think it should be, because I didn't patch to ARM simulator Linux kernel. > > > > > > [sbox-ARM: ~/ltt-control-0.55-16102008/lttctl] > ltt-armall > > > Connecting all markers > > > cat: /proc/ltt: No such file or directory > > > Connecting high-rate markers to tap > > > cat: /proc/ltt: No such file or directory > > > > > > It should be work on my target device, but it seems not. > > > I can't find /ltt under /proc > > > Maybe it's root cause, but why there is no /ltt/proc? And why there is no " cat: /proc/ltt: No such file or directory" message in my target device during running ltt-armall? > > > > > > BTW, I mount debugfs in init.c > > > Mount("debugfs", "/sys/kernel/debug", "debugfs", 0, NULL); > > > It should not be a problem. > > > > > > -----------------------------------------------------------------------------------------------------------Another problem: > > > I command as following steps. > > > # lttctl -n trace1 -q > > > Linux Trace Toolkit Trace Control 0.55-16102008 > > > Controlling trace : trace1 > > > > > > # lttctl -n trace1 -r > > > Linux Trace Toolkit Trace Control 0.55-16102008 > > > Controlling trace : trace1 > > > > > > >>> Android shell hang in here, I have to press Ctrl+Z to force stopping - lttctl -n trace1 -r .Why? Is anything wrong? > > > # ls -l ../tmp/trace1 > > > drwxrwxrwx root root 1980-01-07 06:37 control > > > -rwxrwxrwx root root 1048576 1980-01-07 06:37 cpu_0 > > > # ls -l ../tmp/trace1/control > > > -rwxrwxrwx root root 0 1980-01-07 06:37 metadata_0 > > > -rwxrwxrwx root root 0 1980-01-07 06:37 interrupts_0 > > > -rwxrwxrwx root root 10240 1980-01-07 06:37 processes_0 > > > -rwxrwxrwx root root 65536 1980-01-07 06:37 modules_0 > > > -rwxrwxrwx root root 65536 1980-01-07 06:37 network_0 > > > > > > The Channel in /sys/kernel/debug/ltt/trace1 is clear. Does it cause by destroy channel command? > > > > > > The metadata_0 and interrupts_0 are empty, Is that correct? I've done some key press, the interrupts_o shouldn't be empt. > > > > > > And those files almost filled by 0 except for the short header I can't make sure, it looks like incomplete. > > > > > > I hope you can help me to solve these issues, I am appreciate you so much.. > > > > > > Best regards, > > > Bryan Hsieh > > > > > > SPEC, FIH Co., Ltd. > > > Tel:+886-3-270-0000 #2723 > > > Fax:+886-3-270-0001 > > > > > > > > > -----Original Message----- > > > From: Ajit R [mailto:ajitrzid@yahoo.com] > > > Sent: Friday, July 31, 2009 6:17 PM > > > To: 'ltt-dev at lists.casi.polymtl.ca'; Bryan Hsieh > > > Cc: Tommy Peng > > > Subject: Re: [ltt-dev] Some LTTng 0.43 issue on ARM > > > > > > > > > Hi Bryan, > > > My replies with the [Ajit] tag. > > > LTT Gods correct me if I am wrong. > > > > > > Thanks > > > Ajit > > > > > > --- On Fri, 7/31/09, Bryan Hsieh <bryanhsieh at fihspec.com> wrote: > > > > > > > From: Bryan Hsieh <bryanhsieh@fihspec.com> > > > > Subject: [ltt-dev] Some LTTng 0.43 issue on ARM > > > > To: "'ltt-dev at lists.casi.polymtl.ca'" <ltt-dev at lists.casi.polymtl.ca> > > > > Cc: "Tommy Peng" <tommypeng at fihspec.com> > > > > Date: Friday, July 31, 2009, 12:19 PM > > > > > > > Hi Mathieu, > > > > all, > > > > > > > > > > > > > > > > > > > > I've ported > > > > LTTng 0.43 on Android successfully, > > > > but I am wondering if the result is correct, please help me > > > > to verify, thank > > > > you very much! > > > > > > > > > > > > > > > > > > > > My Traced > > > > machine: > > > > > > > > ARM11 (ARM1136J(F)-S) / Linux > > > > 2.6..27 > > > > (Android 1.5) > > > > > > > > My > > > > visualization machine: > > > > > > > > X86 / Sun xVM Virtual Box (Unbuntu > > > > 8.10) > > > > > > > > > > > > > > > > LTTng: 0.43 > > > > > > > > LTTV: > > > > 0.11.2 > > > > > > > > LTT Control: 0.55 > > > > > > > > > > > > > > > > > > > > I > > > > connect to Android > > > > shell by ADB tool after Android bring up, then I command as > > > > following step, the > > > > command line options are different with LTTng manual, > > > > because the example in manual > > > > is for old version. > > > > > > > > > > > > > > > > > > > > # ltt-armall > > > > > > > [ Ajit ] > > > ltt-armall is a shell script. It connects + arms the markers in the > > > kernel. So depending on your kernel + LTT version it would write some > > > "text" into the /proc/ltt directory or a "1" into files in > > > the debugfs directory. > > > Make sure, the binaries for running the script are available in your target environment. For eg. awk, grep, sed etc. Busybox versions might have their limitations. If they do not work, you could do equivalent things on your Desktop and then write a simple script, doing the same on your target. > > > Once you run "ltt-armall", you should see some debug statements in your "dmesg" like "connect" or "arm" etc. > > > > > > > > > > > Connecting all > > > > markers > > > > > > > > Connecting high-rate > > > > markers to tap > > > > > > > > # lttctl > > > > -n trace1 -d -l /sys/kernel/debug/ltt -t > > > > /system/traces/trace1 > > > > > > > [Ajit] > > > Not sure if your "-l" option is right. you have to point it to the place where you mounted your debugfs. > > > > > > > > > > > Linux Trace Toolkit > > > > Trace Control 0.55-16102008 > > > > > > > > > > > > > > > > > > > > Controlling trace : > > > > trace1 > > > > > > > > > > > > > > > > > > > > Linux Trace Toolkit > > > > Trace Daemon 0.55-16102008 > > > > > > > > > > > > > > > > > > > > Reading from debugfs > > > > directory : /sys/kernel/debug/ltt/trace1 > > > > > > > > > > > > Writing to trace > > > > directory : /system/traces/trace1 > > > > > > > > > > > > > > > > > > > > It > > > > seems the LTT control > > > > and daemon work fine. I run some scenarios for 1 minute on > > > > Android, but the trace > > > > files are still empty (all files length are > > > > 0) > > > > > > > > What's > > > > wrong? Is > > > > there any step I missed? Or the LTT control and daemon > > > > don't work? > > > > > > > > > > > > > > > > > > > > # ls > > > > -l ../../sys/kernel/debug/ltt/ > > > > > > > > > > > > > > > > drwxr-xr-x > > > > root > > > > root > > > > 1980-01-07 01:13 trace1 > > > > > > > > # ls > > > > -l > > > > ../../sys/kernel/debug/ltt/trace1 > > > > > > > > > > > > -r-------- > > > > root > > > > root > > > > 0 > > > > 1980-01-07 01:13 cpu_0 > > > > > > > > > > [Ajit] > > > If you have connected and armed it right, you should see the cpu_0 > > > file increasing in size. After sometime, disconnect by running the > > > ltt-control -n whatever-name -R > > > and then you will see files in the control directory (network, process, etc ) grown in size. Tar and move the trace directory to your host and do the needful. > > > > > > > > > > drwxr-xr-x > > > > root > > > > root > > > > 1980-01-07 01:13 control > > > > > > > > # ls > > > > -l > > > > ../../sys/kernel/debug/ltt/trace1/control > > > > > > > > > > > > -r-------- > > > > root > > > > root > > > > 0 > > > > 1980-01-07 01:13 network_0 > > > > > > > > -r-------- > > > > root > > > > root > > > > 0 > > > > 1980-01-07 01:13 modules_0 > > > > > > > > -r-------- > > > > root > > > > root > > > > 0 > > > > 1980-01-07 01:13 processes_0 > > > > > > > > -r-------- > > > > root > > > > root > > > > 0 > > > > 1980-01-07 01:13 interrupts_0 > > > > > > > > -r-------- > > > > root > > > > root > > > > 0 1980-01-07 01:13 metadata_0 > > > > > > > > # ls -l > > > > .../traces/ > > > > > > > > > > > > > > > > drwxrwxrwx > > > > root > > > > root > > > > 1980-01-07 01:13 trace1 > > > > > > > > # ls -l > > > > .../traces/trace1 > > > > > > > > drwxrwxrwx > > > > root > > > > root > > > > 1980-01-07 01:13 control > > > > > > > > -rwxrwxrwx > > > > root > > > > root > > > > 0 > > > > 1980-01-07 01:13 cpu_0 > > > > > > > > # ls -l > > > > .../traces/trace1/control > > > > > > > > > > > > > > > > -rwxrwxrwx > > > > root > > > > root > > > > 0 > > > > 1980-01-07 01:13 metadata_0 > > > > > > > > -rwxrwxrwx > > > > root > > > > root > > > > 0 > > > > 1980-01-07 01:13 interrupts_0 > > > > > > > > -rwxrwxrwx > > > > root > > > > root > > > > 0 > > > > 1980-01-07 01:13 processes_0 > > > > > > > > -rwxrwxrwx > > > > root > > > > root > > > > 0 > > > > 1980-01-07 01:13 modules_0 > > > > > > > > -rwxrwxrwx > > > > root > > > > root > > > > 0 > > > > 1980-01-07 01:13 network_0 > > > > > > > > > > > > > > > > > > > > BTW, > > > > I would like to > > > > know what is the channel concept for LTTng? What's > > > > the relationship between > > > > trace and channel? What's the normal, flight recorder > > > > and hybrid mode? > > > > > > > > Which > > > > files should I get > > > > for LTTV on another visualization > > > > machine? > > > > > > > > So far, I > > > > can't find related document or the > > > > information isn't enough for me. > > > > > > > > > > > > > > > > > > > > Thanks for your > > > > help! > > > > > > > > > > > > > > > > > > > > Best > > > > regards, > > > > > > > > Bryan Hsieh > > > > > > > > SPEC, FIH > > > > Co., Ltd. > > > > > > > > Tel:+886-3-270-0000 #2723 > > > > > > > > Fax:+886-3-270-0001 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > > ltt-dev mailing list > > > > ltt-dev at lists.casi.polymtl.ca > > > > http://lists.casi.polymtl.ca/cgi-bin/mailman/listinfo/ltt-dev > > > > > > > > > > > > > > > > This e-mail message together with any attachments thereto (if any) is confidential, protected under an enforceable non-disclosure agreement, > > > intended only for the use of the named recipient(s) above and may contain information that is privileged, belonging to professional work products > > > or exempt from disclosure under applicable laws. Any unauthorized review, use, copying, disclosure, or distribution of any information contained > > > in or attached to this transmission is STRICTLY PROHIBITED and may be against the laws. If you have received this message in error, or are not > > > the named recipient(s), please immediately notify the sender by e-mail or telephone at IT support hotline 886-3-2700000 EXT 1333 and delete this > > > e-mail message and any attached documentation from your computer. Receipt by anyone other than the intended recipient(s) is not a waiver of any > > > attorney-client or work product privilege. Thank you! > > > > > > > > > _______________________________________________ > > > ltt-dev mailing list > > > ltt-dev at lists.casi.polymtl.ca > > > http://lists.casi.polymtl.ca/cgi-bin/mailman/listinfo/ltt-dev > > > > > > > -- > > Mathieu Desnoyers > > OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F BA06 3F25 A8FE 3BAE 9A68 > > > > This e-mail message together with any attachments thereto (if any) is confidential, protected under an enforceable non-disclosure agreement, > > intended only for the use of the named recipient(s) above and may contain information that is privileged, belonging to professional work products > > or exempt from disclosure under applicable laws. Any unauthorized review, use, copying, disclosure, or distribution of any information contained > > in or attached to this transmission is STRICTLY PROHIBITED and may be against the laws. If you have received this message in error, or are not > > the named recipient(s), please immediately notify the sender by e-mail or telephone at IT support hotline 886-3-2700000 EXT 1333 and delete this > > e-mail message and any attached documentation from your computer. Receipt by anyone other than the intended recipient(s) is not a waiver of any > > attorney-client or work product privilege. Thank you! > > Content-Description: busybox version and command query.txt > > # grep --help > > BusyBox v1.12.4 (2009-07-29 23:17:25 CST) multi-call binary > > > > Usage: grep [-HhrilLnqvsoeFEABC] PATTERN [FILEs...] > > > > Search for PATTERN in each FILE or standard input > > > > Options: > > -H Prefix output lines with filename where match was found > > -h Suppress the prefixing filename on output > > -r Recurse subdirectories > > -i Ignore case distinctions > > -l List names of files that match > > -L List names of files that do not match > > -n Print line number with output lines > > -q Quiet. Return 0 if PATTERN is found, 1 otherwise > > -v Select non-matching lines > > -s Suppress file open/read error messages > > -c Only print count of matching lines > > -o Show only the part of a line that matches PATTERN > > -m MAX Match up to MAX times per file > > -F PATTERN is a set of newline-separated strings > > -E PATTERN is an extended regular expression > > -e PTRN Pattern to match > > -f FILE Read pattern from file > > -A Print NUM lines of trailing context > > -B Print NUM lines of leading context > > -C Print NUM lines of output context > > > > # awk --help > > BusyBox v1.12.4 (2009-07-29 23:17:25 CST) multi-call binary > > > > Usage: awk [OPTION]... [program-text] [FILE...] > > > > Options: > > -v var=val Set variable > > -F sep Use sep as field separator > > -f file Read program from file > > > > # sort --help > > BusyBox v1.12.4 (2009-07-29 23:17:25 CST) multi-call binary > > > > Usage: sort [-nrugMcszbdfimSTokt] [-o FILE] [-k start[.offset][opts][,end[.offset][opts]] [-t CHAR] [FILE]... > > > > Sort lines of text > > > > Options: > > -b Ignore leading blanks > > -c Check whether input is sorted > > -d Dictionary order (blank or alphanumeric only) > > -f Ignore case > > -g General numerical sort > > -i Ignore unprintable characters > > -k Sort key > > -M Sort month > > -n Sort numbers > > -o Output to file > > -k Sort by key > > -t CHAR Key separator > > -r Reverse sort order > > -s Stable (don't sort ties alphabetically) > > -u Suppress duplicate lines > > -z Lines are terminated by NUL, not newline > > -mST Ignored for GNU compatibility > > > > # cat --help > > unknown option -- -usage: cat [-beflnstv] [-] [file ...] > > # > > > > Content-Description: LTTng Execution Log.txt > > # ltt-armall-1 > > Connecting all markers > > Connecting fs_buffer_wait_end > > Connecting fs_buffer_wait_start > > Connecting fs_close > > Connecting fs_exec > > Connecting fs_ioctl > > Connecting fs_llseek > > Connecting fs_lseek > > Connecting fs_open > > Connecting fs_pollfd > > Connecting fs_pread64 > > Connecting fs_pwrite64 > > Connecting fs_read > > Connecting fs_readv > > Connecting fs_select > > Connecting fs_write > > Connecting fs_writev > > Connecting input_event > > Connecting ipc_msg_create > > Connecting ipc_sem_create > > Connecting ipc_shm_create > > Connecting kernel_arch_ipc_call > > Connecting kernel_arch_kthread_create > > Connecting kernel_arch_syscall_entry > > Connecting kernel_arch_syscall_exit > > Connecting kernel_arch_trap_entry > > Connecting kernel_arch_trap_exit > > Connecting kernel_irq_entry > > Connecting kernel_irq_exit > > Connecting kernel_kthread_stop > > Connecting kernel_kthread_stop_ret > > Connecting kernel_module_free > > Connecting kernel_module_load > > Connecting kernel_printk > > Connecting kernel_process_exit > > Connecting kernel_process_fork > > Connecting kernel_process_free > > Connecting kernel_process_wait > > Connecting kernel_sched_migrate_task > > Connecting kernel_sched_schedule > > Connecting kernel_sched_try_wakeup > > Connecting kernel_sched_wait_task > > Connecting kernel_sched_wakeup_new_task > > Connecting kernel_send_signal > > Connecting kernel_softirq_entry > > Connecting kernel_softirq_exit > > Connecting kernel_softirq_raise > > Connecting kernel_tasklet_high_entry > > Connecting kernel_tasklet_high_exit > > Connecting kernel_tasklet_low_entry > > Connecting kernel_tasklet_low_exit > > Connecting kernel_timer_itimer_expired > > Connecting kernel_timer_itimer_set > > Connecting kernel_timer_set > > Connecting kernel_timer_timeout > > Connecting kernel_timer_update_time > > Connecting kernel_vprintk > > Connecting list_file_descriptor > > Connecting list_interrupt > > Connecting list_module > > Connecting list_network_ip_interface > > Connecting list_network_ipv4_interface > > Connecting list_process_state > > Connecting list_statedump_end > > Connecting list_vm_map > > Connecting mm_handle_fault_entry > > Connecting mm_handle_fault_exit > > Connecting mm_huge_page_alloc > > Connecting mm_huge_page_free > > Connecting mm_page_alloc > > Connecting mm_page_free > > Connecting mm_wait_on_page_end > > Connecting mm_wait_on_page_start > > Connecting rcu_classic_callback > > Connecting rcu_classic_call_rcu > > Connecting rcu_classic_call_rcu_bh > > Connecting statedump_softirq_vec > > Connecting high-rate markers to tap > > Connecting fs_buffer_wait_end > > Connecting fs_buffer_wait_start > > Connecting fs_close > > Connecting fs_exec > > Connecting fs_ioctl > > Connecting fs_llseek > > Connecting fs_lseek > > Connecting fs_open > > Connecting fs_pollfd > > Connecting fs_pread64 > > Connecting fs_pwrite64 > > Connecting fs_read > > Connecting fs_readv > > Connecting fs_select > > Connecting fs_write > > Connecting fs_writev > > Connecting input_event > > Connecting ipc_msg_create > > Connecting ipc_sem_create > > Connecting ipc_shm_create > > Connecting kernel_arch_ipc_call > > Connecting kernel_arch_kthread_create > > Connecting kernel_arch_syscall_entry > > Connecting kernel_arch_syscall_exit > > Connecting kernel_arch_trap_entry > > Connecting kernel_arch_trap_exit > > Connecting kernel_irq_entry > > Connecting kernel_irq_exit > > Connecting kernel_kthread_stop > > Connecting kernel_kthread_stop_ret > > Connecting kernel_module_free > > Connecting kernel_module_load > > Connecting kernel_printk > > Connecting kernel_process_exit > > Connecting kernel_process_fork > > Connecting kernel_process_free > > Connecting kernel_process_wait > > Connecting kernel_sched_migrate_task > > Connecting kernel_sched_schedule > > Connecting kernel_sched_try_wakeup > > Connecting kernel_sched_wait_task > > Connecting kernel_sched_wakeup_new_task > > Connecting kernel_send_signal > > Connecting kernel_softirq_entry > > Connecting kernel_softirq_exit > > Connecting kernel_softirq_raise > > Connecting kernel_tasklet_high_entry > > Connecting kernel_tasklet_high_exit > > Connecting kernel_tasklet_low_entry > > Connecting kernel_tasklet_low_exit > > Connecting kernel_timer_itimer_expired > > Connecting kernel_timer_itimer_set > > Connecting kernel_timer_set > > Connecting kernel_timer_timeout > > Connecting kernel_timer_update_time > > Connecting kernel_vprintk > > Connecting list_file_descriptor > > Connecting list_interrupt > > Connecting list_module > > Connecting list_network_ip_interface > > Connecting list_network_ipv4_interface > > Connecting list_process_state > > Connecting list_statedump_end > > Connecting list_vm_map > > Connecting mm_handle_fault_entry > > Connecting mm_handle_fault_exit > > Connecting mm_huge_page_alloc > > Connecting mm_huge_page_free > > Connecting mm_page_alloc > > Connecting mm_page_free > > Connecting mm_wait_on_page_end > > Connecting mm_wait_on_page_start > > Connecting rcu_classic_callback > > Connecting rcu_classic_call_rcu > > Connecting rcu_classic_call_rcu_bh > > Connecting statedump_softirq_vec > > > > # cat /proc/ltt > > marker: kernel_arch_kthread_create format: "pid %ld fn %p" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: kernel_arch_syscall_exit format: " " state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: kernel_arch_syscall_entry format: "syscall_id %d ip #p%ld" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: kernel_arch_ipc_call format: "call %u first %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: kernel_arch_trap_exit format: " " state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: kernel_arch_trap_entry format: "trap_id %ld ip #p%ld" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: statedump_softirq_vec format: "id %d address %p symbol %s" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: list_module format: "name %s state %d refcount %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: core_marker_format format: "name %s format %s" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: core_marker_format format: "name %s format %s" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: list_statedump_end format: " " state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: list_process_state format: "pid %d parent_pid %d name %s type %d mode %d submode %d status %d tgid %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: list_file_descriptor format: "filename %s pid %d fd %u" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: list_vm_map format: "pid %d start %lu end %lu flags %lu pgoff %lu inode %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: list_interrupt format: "name %s action %s irq_id %u" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: list_network_ip_interface format: "name %s address #4u%lu up %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: list_network_ipv4_interface format: "name %s address #4u%lu up %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: core_marker_id format: "name %s id %hu int #1u%zu long #1u%zu pointer #1u%zu size_t #1u%zu alignment #1u%u" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: core_marker_id format: "name %s id %hu int #1u%zu long #1u%zu pointer #1u%zu size_t #1u%zu alignment #1u%u" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: core_marker_format format: "name %s format %s" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: core_marker_id format: "name %s id %hu int #1u%zu long #1u%zu pointer #1u%zu size_t #1u%zu alignment #1u%u" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: core_marker_id format: "name %s id %hu int #1u%zu long #1u%zu pointer #1u%zu size_t #1u%zu alignment #1u%u" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: kernel_module_load format: "name %s" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: kernel_module_free format: "name %s" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: kernel_vprintk format: "loglevel %c string %s ip %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: kernel_printk format: "ip %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: kernel_timer_timeout format: "pid %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: kernel_timer_update_time format: "jiffies #8u%llu xtime_sec %ld xtime_nsec %ld walltomonotonic_sec %ld walltomonotonic_nsec %ld" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: kernel_timer_set format: "expires %lu function %p data %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: kernel_timer_itimer_set format: "which %d interval_sec %ld interval_usec %ld value_sec %ld value_usec %ld" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: kernel_timer_itimer_expired format: "pid %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: kernel_process_fork format: "parent_pid %d child_pid %d child_tgid %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: kernel_process_wait format: "pid %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: kernel_process_exit format: "pid %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: kernel_process_free format: "pid %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: kernel_send_signal format: "pid %d signal %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: kernel_sched_migrate_task format: "pid %d state %ld dest_cpu %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: kernel_sched_schedule format: "prev_pid %d next_pid %d prev_state %ld" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: kernel_sched_wakeup_new_task format: "pid %d state %ld cpu_id %u" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: kernel_sched_try_wakeup format: "pid %d state %ld cpu_id %u" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: kernel_sched_wait_task format: "pid %d state %ld" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: kernel_kthread_stop_ret format: "ret %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: kernel_kthread_stop format: "pid %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: kernel_tasklet_high_exit format: "func %p data %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: kernel_tasklet_high_entry format: "func %p data %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: kernel_tasklet_low_exit format: "func %p data %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: kernel_tasklet_low_entry format: "func %p data %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: kernel_softirq_raise format: "softirq_id %u" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: kernel_softirq_exit format: "softirq_id %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: kernel_softirq_entry format: "softirq_id %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: kernel_irq_exit format: "handled #1u%u" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: kernel_irq_entry format: "irq_id %u kernel_mode %u ip %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: mm_page_alloc format: "order %u pfn %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: mm_page_free format: "order %u pfn %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: mm_handle_fault_exit format: "res %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: mm_handle_fault_entry format: "address %lu ip #p%ld write_access %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: mm_huge_page_alloc format: "pfn %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: mm_huge_page_free format: "pfn %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: mm_wait_on_page_end format: "pfn %lu bit_nr %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: mm_wait_on_page_start format: "pfn %lu bit_nr %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: fs_pollfd format: "fd %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: fs_select format: "fd %d timeout #8d%lld" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: fs_writev format: "fd %lu vlen %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: fs_readv format: "fd %lu vlen %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: fs_pwrite64 format: "fd %u count %zu pos %llu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: fs_pread64 format: "fd %u count %zu pos %llu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: fs_write format: "fd %u count %zu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: fs_read format: "fd %u count %zu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: fs_llseek format: "fd %u offset %lld origin %u" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: fs_lseek format: "fd %u offset %ld origin %u" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: fs_close format: "fd %u" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: fs_open format: "fd %d filename %s" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: fs_ioctl format: "fd %u cmd %u arg %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: fs_exec format: "filename %s" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: fs_buffer_wait_end format: "bh %p" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: fs_buffer_wait_start format: "bh %p" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: ipc_shm_create format: "id %ld flags %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: ipc_sem_create format: "id %ld flags %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: ipc_msg_create format: "id %ld flags %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: lockdep_lock_release format: "retaddr 0x%lX lock %p nested %d" state: 0 call: 0xc0093374 probe single : 0xc00920ec > > marker: lockdep_lock_acquire format: "retaddr 0x%lX subclass %u lock %p trylock %d read %d hardirqs_off %d" state: 0 call: 0xc0093374 probe single : 0xc00920ec > > marker: lockdep_softirqs_off format: "retaddr 0x%lX" state: 0 call: 0xc0093374 probe single : 0xc00920ec > > marker: lockdep_softirqs_on format: "retaddr 0x%lX" state: 0 call: 0xc0093374 probe single : 0xc00920ec > > marker: lockdep_hardirqs_off format: "retaddr 0x%lX" state: 0 call: 0xc0093374 probe single : 0xc00920ec > > marker: lockdep_hardirqs_on format: "retaddr 0x%lX" state: 0 call: 0xc0093374 probe single : 0xc00920ec > > marker: rcu_classic_call_rcu_bh format: "func %p ip 0x%lX" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: rcu_classic_call_rcu format: "func %p ip 0x%lX" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: rcu_classic_callback format: "func %p" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > marker: input_event format: "type %u code %u value %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > > [Bryan] Maybe caused by privilege, the the /proc/ltt do not be replaced with content below. It's that will be a problem? > > connect statedump_softirq_vec ltt_tap_marker dynami > > > > # df > > /dev: 39564K total, 0K used, 39564K available (block size 4096) > > /sqlite_stmt_journals: 4096K total, 0K used, 4096K available (block size 4096) > > /system: 72704K total, 71680K used, 1024K available (block size 4096) > > /data: 66816K total, 37812K used, 29004K available (block size 4096) > > /cache: 67584K total, 1156K used, 66428K available (block size 4096) > > /sdcard: 991708K total, 147040K used, 844668K available (block size 4096) > > > > # lttctl -n trace1 -d -l /sys/kernel/debug/ltt -t /data/ltt/trace1 > > Linux Trace Toolkit Trace Control 0.55-16102008 > > > > Controlling trace : trace1 > > > > Linux Trace Toolkit Trace Daemon 0.55-16102008 > > > > Reading from debugfs directory : /sys/kernel/debug/ltt/trace1 > > Writing to trace directory : /data/ltt/trace1 > > > > [Bryan] there is 29004K available for /data/ltt/trace1, is that enough? > > [Bryan] do some tests on target device, then check the channel and trace > > > > # ls -l /sys/kernel/debug/ltt/trace1 > > -r-------- root root 0 1980-01-06 00:10 cpu_0 > > drwxr-xr-x root root 1980-01-06 00:10 control > > # ls -l /sys/kernel/debug/ltt/trace1/control > > -r-------- root root 0 1980-01-06 00:10 network_0 > > -r-------- root root 0 1980-01-06 00:10 modules_0 > > -r-------- root root 0 1980-01-06 00:10 processes_0 > > -r-------- root root 0 1980-01-06 00:10 interrupts_0 > > -r-------- root root 0 1980-01-06 00:10 metadata_0 > > # ls -l /data/ltt/trace1 > > drwxrwxrwx root root 1980-01-06 00:10 control > > -rwxrwxrwx root root 29949952 1980-01-06 00:10 cpu_0 > > # ls -l /data/ltt/trace1/control > > -rwxrwxrwx root root 0 1980-01-06 00:10 metadata_0 > > -rwxrwxrwx root root 0 1980-01-06 00:10 interrupts_0 > > -rwxrwxrwx root root 0 1980-01-06 00:10 processes_0 > > -rwxrwxrwx root root 0 1980-01-06 00:10 modules_0 > > -rwxrwxrwx root root 0 1980-01-06 00:10 network_0 > > > > [Bryan] It seems that the space has been exhausted by cpu_0, it's that correct that cpu_0 will grow to so huge(29M)? > > How much file space should I prepare for LTTng? > > > > # lttctl -n trace1 -d -R > > Linux Trace Toolkit Trace Control 0.55-16102008 > > > > Controlling trace : trace1 > > > > [Bryan] hang in here...I have to break it by Ctrl+Z, What's problem? > > > > ^Z[1] + Stopped lttctl -n trace1 -d -R > > # ls -l /data/ltt/trace1/ > > drwxrwxrwx root root 1980-01-06 00:10 control > > -rwxrwxrwx root root 29949952 1980-01-06 00:10 cpu_0 > > # ls -l /data/ltt/trace1/control > > -rwxrwxrwx root root 0 1980-01-06 00:10 metadata_0 > > -rwxrwxrwx root root 0 1980-01-06 00:10 interrupts_0 > > -rwxrwxrwx root root 0 1980-01-06 00:10 processes_0 > > -rwxrwxrwx root root 0 1980-01-06 00:10 modules_0 > > -rwxrwxrwx root root 0 1980-01-06 00:10 network_0 > > # ls -l /sys/kernel/debug/ltt/trace1 > > -r-------- root root 0 1980-01-06 00:10 cpu_0 > > drwxr-xr-x root root 1980-01-06 00:10 control > > # ls -l /sys/kernel/debug/ltt/trace1/control > > -r-------- root root 0 1980-01-06 00:10 network_0 > > -r-------- root root 0 1980-01-06 00:10 modules_0 > > -r-------- root root 0 1980-01-06 00:10 processes_0 > > -r-------- root root 0 1980-01-06 00:10 interrupts_0 > > -r-------- root root 0 1980-01-06 00:10 metadata_0 > > # df > > /dev: 39564K total, 0K used, 39564K available (block size 4096) > > /sqlite_stmt_journals: 4096K total, 0K used, 4096K available (block size 4096) > > /system: 72704K total, 71680K used, 1024K available (block size 4096) > > /data: 66816K total, 66732K used, 84K available (block size 4096) > > /cache: 67584K total, 1156K used, 66428K available (block size 4096) > > /sdcard: 991708K total, 147040K used, 844668K available (block size 4096) > > > > > > > > Content-Description: LTTng Kernel Config.txt > > > > [ ] Kernel->user space relay support (formerly relayfs) > > [*] Activate tracepoints x x > > [*] Activate markers x x > > Linux Trace Toolkit ---> > > <*> Compile lttng tracing probes x x > > [*] LTTng fine-grained timestamping x x > > [*] Linux Trace Toolkit Instrumentation Support x x > > <*> Linux Trace Toolkit High-speed Lockless Data Relay x x > > <*> Linux Trace Toolkit Lock-Protected Data Relay x x > > [*] Debug check for random access in ltt relay buffers x x > > <*> Linux Trace Toolkit Serializer x x > > <*> Linux Trace Toolkit Marker Control x x > > <*> Linux Trace Toolkit Tracer x x > > [*] Align Linux Trace Toolkit Traces x x > > [*] Add event size field to LTT events for tracer debugging x x > > [*] Support trace extraction from crash dump x x > > <*> Linux Trace Toolkit Netlink Controller x x > > <*> Linux Trace Toolkit State Dump > > -- > Mathieu Desnoyers > OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F BA06 3F25 A8FE 3BAE 9A68 > This e-mail message together with any attachments thereto (if any) is confidential, protected under an enforceable non-disclosure agreement, > intended only for the use of the named recipient(s) above and may contain information that is privileged, belonging to professional work products > or exempt from disclosure under applicable laws. Any unauthorized review, use, copying, disclosure, or distribution of any information contained > in or attached to this transmission is STRICTLY PROHIBITED and may be against the laws. If you have received this message in error, or are not > the named recipient(s), please immediately notify the sender by e-mail or telephone at IT support hotline 886-3-2700000 EXT 1333 and delete this > e-mail message and any attached documentation from your computer. Receipt by anyone other than the intended recipient(s) is not a waiver of any > attorney-client or work product privilege. Thank you! > -- Mathieu Desnoyers OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F BA06 3F25 A8FE 3BAE 9A68 ^ permalink raw reply [flat|nested] 9+ messages in thread
* [ltt-dev] Some LTTng 0.43 issue on ARM 2009-08-06 17:20 ` Mathieu Desnoyers @ 2009-08-06 17:24 ` Mathieu Desnoyers 0 siblings, 0 replies; 9+ messages in thread From: Mathieu Desnoyers @ 2009-08-06 17:24 UTC (permalink / raw) * Mathieu Desnoyers (compudj at krystal.dyndns.org) wrote: > * Bryan Hsieh (bryanhsieh at fihspec.com) wrote: > > Hi Mathieu, All > > > > [Bryan] > > I modified install path and re-compile, then install to my target system, even I replace the original toolbox with new busybox, it still doesn't work, I try to use older version busybox and install to my target system, finally It's work! I watch the space change and stop trace before the capacity been exhausted, the lttd can stop successfully. Thank LTT Gods! > > # ls -l /data/ltt/trace1 > > drwxrwxrwx 1 0 0 2048 Jan 6 01:17 control > > -rwxrwxrwx 1 0 0 25165824 Jan 6 01:17 cpu_0 > > # ls -l /data/ltt/trace1/control > > -rwxrwxrwx 1 0 0 65536 Jan 6 01:17 interrupts_0 > > -rwxrwxrwx 1 0 0 65536 Jan 6 01:17 metadata_0 > > -rwxrwxrwx 1 0 0 65536 Jan 6 01:17 modules_0 > > -rwxrwxrwx 1 0 0 65536 Jan 6 01:17 network_0 > > -rwxrwxrwx 1 0 0 262144 Jan 6 01:17 processes_0 > > > > [Bryan] > > I have another problem, I can store trace in /data or /cache directory by -t option, but I still can't store trace in /sdcard directory, the files size are empty in /sdcard/ltt/trace1, what's wrong? > > > > # mount > > rootfs on / type rootfs (ro) > > tmpfs on /dev type tmpfs (rw,mode=755) > > devpts on /dev/pts type devpts (rw,mode=600) > > proc on /proc type proc (rw) > > sysfs on /sys type sysfs (rw) > > debugfs on /sys/kernel/debug type debugfs (rw) > > tmpfs on /sqlite_stmt_journals type tmpfs (rw,size=4096k) > > /dev/block/mtdblock3 on /system type yaffs2 (rw) > > /dev/block/mtdblock6 on /data type yaffs2 (rw,nosuid,nodev) > > /dev/block/mtdblock4 on /cache type yaffs2 (rw,nosuid,nodev) > > /dev/block//vold/179:1 on /sdcard type vfat (rw,dirsync,nosuid,nodev,noexec,uid=1000,gid=1000,fmask=0711,dmask=0700,allow_utime=0022,codepage=cp437,iocharset=iso8859-1,shortname=mixed,utf8) > > Hrm, I guess it's a problem with vfat splice support. You try to write > it to a NFS mounted drive instead. > Another solution would be to format an SD card with ext2/3. Or you can look at the vfat code and try to figure out what is missing for splice support. Mathieu > Mathieu > > > > > > > Best regards, > > Bryan Hsieh > > > > SPEC, FIH Co., Ltd. > > Tel:+886-3-270-0000 #2723 > > Fax:+886-3-270-0001 > > > > > > -----Original Message----- > > From: Mathieu Desnoyers [mailto:compudj@krystal.dyndns.org] > > Sent: Tuesday, August 04, 2009 9:18 PM > > To: Bryan Hsieh > > Cc: 'Ajit R'; 'ltt-dev at lists.casi.polymtl.ca'; Tommy Peng > > Subject: Re: [ltt-dev] Some LTTng 0.43 issue on ARM > > > > * Bryan Hsieh (bryanhsieh at fihspec.com) wrote: > > > Hi Mathieu, All > > > > > > Please check my comment which marked by [Bryan], Thanks! > > > ----------------------------------------------------------------------------- > > > [Bryan] > > > All the related LTT functions are build-in not build as modules, Please check attachment- LTTng Kernel Config..txt, I am not sure if Kernel->user space relay support (formerly relayfs) need to build-in. > > > Is there any item I missed? > > > ----------------------------------------------------------------------------- > > > [ ] Kernel->user space relay support (formerly relayfs) > > > > LTTng does not need relayfs anymore. So this is ok. > > > > > [*] Activate tracepoints > > > [*] Activate markers > > > Linux Trace Toolkit ---> > > > <*> Compile lttng tracing probes > > > [*] LTTng fine-grained timestamping > > > [*] Linux Trace Toolkit Instrumentation Support > > > <*> Linux Trace Toolkit High-speed Lockless Data Relay > > > <*> Linux Trace Toolkit Lock-Protected Data Relay > > > [*] Debug check for random access in ltt relay buffers > > > <*> Linux Trace Toolkit Serializer > > > <*> Linux Trace Toolkit Marker Control > > > <*> Linux Trace Toolkit Tracer > > > [*] Align Linux Trace Toolkit Traces > > > [*] Add event size field to LTT events for tracer debugging > > > [*] Support trace extraction from crash dump > > > <*> Linux Trace Toolkit Netlink Controller > > > <*> Linux Trace Toolkit State Dump > > > ----------------------------------------------------------------------------- > > > [Bryan] > > > I am sure that the external utilities are build-in busybox, please check the attachment - " busybox version and command query.txt". > > > > > > Finally, I figure out why the ltt-armall doesn't work correctly in my target system, root cause is the output pipeline will be empty after grep twice, even I use a temp file to avoid connecting two grep commands directly, it's still empty. So I copy the /proc/ltt to my local desktop to do ltt-armall script, then generate the /proc/ltt which been filtered by following command. > > > MARKERS=`cat /proc/ltt|grep -v %k|awk '{print $2}'|sort -u|grep -v ^core_|grep -v ^locking_|grep -v ^lockdep_` > > > I rename it as ltt_filter and copy to my target system path /system/bin/, then modify ltt-armall by following modification and run it. > > > MARKERS=`cat /system/bin/ltt_filter` > > > > > > The execution result on screen: > > > # ltt-armall-1 > > > Connecting all markers > > > Connecting fs_buffer_wait_end > > > Connecting fs_buffer_wait_start > > > Connecting fs_close > > > Connecting fs_exec > > > Connecting fs_ioctl > > > Connecting fs_llseek > > > Connecting fs_lseek > > > Connecting fs_open > > > Connecting fs_pollfd > > > Connecting fs_pread64 > > > Connecting fs_pwrite64 > > > Connecting fs_read > > > Connecting fs_readv > > > Connecting fs_select > > > Connecting fs_write > > > Connecting fs_writev > > > Connecting input_event > > > Connecting ipc_msg_create > > > Connecting ipc_sem_create > > > Connecting ipc_shm_create > > > Connecting kernel_arch_ipc_call > > > Connecting kernel_arch_kthread_create > > > Connecting kernel_arch_syscall_entry > > > Connecting kernel_arch_syscall_exit > > > Connecting kernel_arch_trap_entry > > > Connecting kernel_arch_trap_exit > > > Connecting kernel_irq_entry > > > Connecting kernel_irq_exit > > > Connecting kernel_kthread_stop > > > Connecting kernel_kthread_stop_ret > > > Connecting kernel_module_free > > > Connecting kernel_module_load > > > Connecting kernel_printk > > > Connecting kernel_process_exit > > > Connecting kernel_process_fork > > > Connecting kernel_process_free > > > Connecting kernel_process_wait > > > Connecting kernel_sched_migrate_task > > > Connecting kernel_sched_schedule > > > Connecting kernel_sched_try_wakeup > > > Connecting kernel_sched_wait_task > > > Connecting kernel_sched_wakeup_new_task > > > Connecting kernel_send_signal > > > Connecting kernel_softirq_entry > > > Connecting kernel_softirq_exit > > > Connecting kernel_softirq_raise > > > Connecting kernel_tasklet_high_entry > > > Connecting kernel_tasklet_high_exit > > > Connecting kernel_tasklet_low_entry > > > Connecting kernel_tasklet_low_exit > > > Connecting kernel_timer_itimer_expired > > > Connecting kernel_timer_itimer_set > > > Connecting kernel_timer_set > > > Connecting kernel_timer_timeout > > > Connecting kernel_timer_update_time > > > Connecting kernel_vprintk > > > Connecting list_file_descriptor > > > Connecting list_interrupt > > > Connecting list_module > > > Connecting list_network_ip_interface > > > Connecting list_network_ipv4_interface > > > Connecting list_process_state > > > Connecting list_statedump_end > > > Connecting list_vm_map > > > Connecting mm_handle_fault_entry > > > Connecting mm_handle_fault_exit > > > Connecting mm_huge_page_alloc > > > Connecting mm_huge_page_free > > > Connecting mm_page_alloc > > > Connecting mm_page_free > > > Connecting mm_wait_on_page_end > > > Connecting mm_wait_on_page_start > > > Connecting rcu_classic_callback > > > Connecting rcu_classic_call_rcu > > > Connecting rcu_classic_call_rcu_bh > > > Connecting statedump_softirq_vec > > > Connecting high-rate markers to tap > > > Connecting fs_buffer_wait_end > > > Connecting fs_buffer_wait_start > > > Connecting fs_close > > > Connecting fs_exec > > > Connecting fs_ioctl > > > Connecting fs_llseek > > > Connecting fs_lseek > > > Connecting fs_open > > > Connecting fs_pollfd > > > Connecting fs_pread64 > > > Connecting fs_pwrite64 > > > Connecting fs_read > > > Connecting fs_readv > > > Connecting fs_select > > > Connecting fs_write > > > Connecting fs_writev > > > Connecting input_event > > > Connecting ipc_msg_create > > > Connecting ipc_sem_create > > > Connecting ipc_shm_create > > > Connecting kernel_arch_ipc_call > > > Connecting kernel_arch_kthread_create > > > Connecting kernel_arch_syscall_entry > > > Connecting kernel_arch_syscall_exit > > > Connecting kernel_arch_trap_entry > > > Connecting kernel_arch_trap_exit > > > Connecting kernel_irq_entry > > > Connecting kernel_irq_exit > > > Connecting kernel_kthread_stop > > > Connecting kernel_kthread_stop_ret > > > Connecting kernel_module_free > > > Connecting kernel_module_load > > > Connecting kernel_printk > > > Connecting kernel_process_exit > > > Connecting kernel_process_fork > > > Connecting kernel_process_free > > > Connecting kernel_process_wait > > > Connecting kernel_sched_migrate_task > > > Connecting kernel_sched_schedule > > > Connecting kernel_sched_try_wakeup > > > Connecting kernel_sched_wait_task > > > Connecting kernel_sched_wakeup_new_task > > > Connecting kernel_send_signal > > > Connecting kernel_softirq_entry > > > Connecting kernel_softirq_exit > > > Connecting kernel_softirq_raise > > > Connecting kernel_tasklet_high_entry > > > Connecting kernel_tasklet_high_exit > > > Connecting kernel_tasklet_low_entry > > > Connecting kernel_tasklet_low_exit > > > Connecting kernel_timer_itimer_expired > > > Connecting kernel_timer_itimer_set > > > Connecting kernel_timer_set > > > Connecting kernel_timer_timeout > > > Connecting kernel_timer_update_time > > > Connecting kernel_vprintk > > > Connecting list_file_descriptor > > > Connecting list_interrupt > > > Connecting list_module > > > Connecting list_network_ip_interface > > > Connecting list_network_ipv4_interface > > > Connecting list_process_state > > > Connecting list_statedump_end > > > Connecting list_vm_map > > > Connecting mm_handle_fault_entry > > > Connecting mm_handle_fault_exit > > > Connecting mm_huge_page_alloc > > > Connecting mm_huge_page_free > > > Connecting mm_page_alloc > > > Connecting mm_page_free > > > Connecting mm_wait_on_page_end > > > Connecting mm_wait_on_page_start > > > Connecting rcu_classic_callback > > > Connecting rcu_classic_call_rcu > > > Connecting rcu_classic_call_rcu_bh > > > Connecting statedump_softirq_vec > > > ----------------------------------------------------------------------------- > > > [Bryan] It's that correct result? > > > > Yes, this output is correct. So you probably need to recompile busybox > > with awk and/or sed and/or sort support..... > > > > > ----------------------------------------------------------------------------- > > > # cat /proc/ltt > > > marker: kernel_arch_kthread_create format: "pid %ld fn %p" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: kernel_arch_syscall_exit format: " " state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: kernel_arch_syscall_entry format: "syscall_id %d ip #p%ld" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: kernel_arch_ipc_call format: "call %u first %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: kernel_arch_trap_exit format: " " state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: kernel_arch_trap_entry format: "trap_id %ld ip #p%ld" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: statedump_softirq_vec format: "id %d address %p symbol %s" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: list_module format: "name %s state %d refcount %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: core_marker_format format: "name %s format %s" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: core_marker_format format: "name %s format %s" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: list_statedump_end format: " " state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: list_process_state format: "pid %d parent_pid %d name %s type %d mode %d submode %d status %d tgid %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: list_file_descriptor format: "filename %s pid %d fd %u" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: list_vm_map format: "pid %d start %lu end %lu flags %lu pgoff %lu inode %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: list_interrupt format: "name %s action %s irq_id %u" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: list_network_ip_interface format: "name %s address #4u%lu up %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: list_network_ipv4_interface format: "name %s address #4u%lu up %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: core_marker_id format: "name %s id %hu int #1u%zu long #1u%zu pointer #1u%zu size_t #1u%zu alignment #1u%u" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: core_marker_id format: "name %s id %hu int #1u%zu long #1u%zu pointer #1u%zu size_t #1u%zu alignment #1u%u" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: core_marker_format format: "name %s format %s" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: core_marker_id format: "name %s id %hu int #1u%zu long #1u%zu pointer #1u%zu size_t #1u%zu alignment #1u%u" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: core_marker_id format: "name %s id %hu int #1u%zu long #1u%zu pointer #1u%zu size_t #1u%zu alignment #1u%u" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: kernel_module_load format: "name %s" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: kernel_module_free format: "name %s" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: kernel_vprintk format: "loglevel %c string %s ip %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: kernel_printk format: "ip %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: kernel_timer_timeout format: "pid %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: kernel_timer_update_time format: "jiffies #8u%llu xtime_sec %ld xtime_nsec %ld walltomonotonic_sec %ld walltomonotonic_nsec %ld" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: kernel_timer_set format: "expires %lu function %p data %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: kernel_timer_itimer_set format: "which %d interval_sec %ld interval_usec %ld value_sec %ld value_usec %ld" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: kernel_timer_itimer_expired format: "pid %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: kernel_process_fork format: "parent_pid %d child_pid %d child_tgid %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: kernel_process_wait format: "pid %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: kernel_process_exit format: "pid %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: kernel_process_free format: "pid %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: kernel_send_signal format: "pid %d signal %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: kernel_sched_migrate_task format: "pid %d state %ld dest_cpu %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: kernel_sched_schedule format: "prev_pid %d next_pid %d prev_state %ld" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: kernel_sched_wakeup_new_task format: "pid %d state %ld cpu_id %u" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: kernel_sched_try_wakeup format: "pid %d state %ld cpu_id %u" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: kernel_sched_wait_task format: "pid %d state %ld" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: kernel_kthread_stop_ret format: "ret %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: kernel_kthread_stop format: "pid %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: kernel_tasklet_high_exit format: "func %p data %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: kernel_tasklet_high_entry format: "func %p data %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: kernel_tasklet_low_exit format: "func %p data %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: kernel_tasklet_low_entry format: "func %p data %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: kernel_softirq_raise format: "softirq_id %u" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: kernel_softirq_exit format: "softirq_id %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: kernel_softirq_entry format: "softirq_id %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: kernel_irq_exit format: "handled #1u%u" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: kernel_irq_entry format: "irq_id %u kernel_mode %u ip %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: mm_page_alloc format: "order %u pfn %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: mm_page_free format: "order %u pfn %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: mm_handle_fault_exit format: "res %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: mm_handle_fault_entry format: "address %lu ip #p%ld write_access %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: mm_huge_page_alloc format: "pfn %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: mm_huge_page_free format: "pfn %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: mm_wait_on_page_end format: "pfn %lu bit_nr %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: mm_wait_on_page_start format: "pfn %lu bit_nr %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: fs_pollfd format: "fd %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: fs_select format: "fd %d timeout #8d%lld" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: fs_writev format: "fd %lu vlen %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: fs_readv format: "fd %lu vlen %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: fs_pwrite64 format: "fd %u count %zu pos %llu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: fs_pread64 format: "fd %u count %zu pos %llu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: fs_write format: "fd %u count %zu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: fs_read format: "fd %u count %zu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: fs_llseek format: "fd %u offset %lld origin %u" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: fs_lseek format: "fd %u offset %ld origin %u" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: fs_close format: "fd %u" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: fs_open format: "fd %d filename %s" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: fs_ioctl format: "fd %u cmd %u arg %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: fs_exec format: "filename %s" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: fs_buffer_wait_end format: "bh %p" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: fs_buffer_wait_start format: "bh %p" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: ipc_shm_create format: "id %ld flags %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: ipc_sem_create format: "id %ld flags %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: ipc_msg_create format: "id %ld flags %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: lockdep_lock_release format: "retaddr 0x%lX lock %p nested %d" state: 0 call: 0xc0093374 probe single : 0xc00920ec > > > marker: lockdep_lock_acquire format: "retaddr 0x%lX subclass %u lock %p trylock %d read %d hardirqs_off %d" state: 0 call: 0xc0093374 probe single : 0xc00920ec > > > marker: lockdep_softirqs_off format: "retaddr 0x%lX" state: 0 call: 0xc0093374 probe single : 0xc00920ec > > > marker: lockdep_softirqs_on format: "retaddr 0x%lX" state: 0 call: 0xc0093374 probe single : 0xc00920ec > > > marker: lockdep_hardirqs_off format: "retaddr 0x%lX" state: 0 call: 0xc0093374 probe single : 0xc00920ec > > > marker: lockdep_hardirqs_on format: "retaddr 0x%lX" state: 0 call: 0xc0093374 probe single : 0xc00920ec > > > marker: rcu_classic_call_rcu_bh format: "func %p ip 0x%lX" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: rcu_classic_call_rcu format: "func %p ip 0x%lX" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: rcu_classic_callback format: "func %p" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: input_event format: "type %u code %u value %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > ----------------------------------------------------------------------------- > > > [Bryan] But the /proc/ltt is still same, it's that correct? What's the expected result? > > > > This output is good. busybox problem, as I said above. Try each command > > used by busybox to test them. > > > > > ----------------------------------------------------------------------------- > > > # df > > > /dev: 39564K total, 0K used, 39564K available (block size 4096) > > > /sqlite_stmt_journals: 4096K total, 0K used, 4096K available (block size 4096) > > > /system: 72704K total, 71680K used, 1024K available (block size 4096) > > > /data: 66816K total, 37812K used, 29004K available (block size 4096) > > > /cache: 67584K total, 1156K used, 66428K available (block size 4096) > > > /sdcard: 991708K total, 147040K used, 844668K available (block size 4096) > > > ----------------------------------------------------------------------------- > > > [Bryan] There is 29004K available for trace log file /data/ltt/trace1. Is that enough? > > > > No. I would save it to /sdcard. > > > > > ----------------------------------------------------------------------------- > > > # cat /proc/meminfo > > > MemTotal: 79128 kB > > > MemFree: 2440 kB > > > Buffers: 36 kB > > > Cached: 29872 kB > > > SwapCached: 0 kB > > > Active: 61384 kB > > > Inactive: 5052 kB > > > SwapTotal: 0 kB > > > SwapFree: 0 kB > > > Dirty: 0 kB > > > Writeback: 0 kB > > > AnonPages: 36552 kB > > > Mapped: 24888 kB > > > Slab: 3856 kB > > > SReclaimable: 904 kB > > > SUnreclaim: 2952 kB > > > PageTables: 3660 kB > > > NFS_Unstable: 0 kB > > > Bounce: 0 kB > > > WritebackTmp: 0 kB > > > CommitLimit: 39564 kB > > > Committed_AS: 1032216 kB > > > VmallocTotal: 409600 kB > > > VmallocUsed: 52908 kB > > > VmallocChunk: 332796 kB > > > ----------------------------------------------------------------------------- > > > [Bryan] > > > There is 2440 kB free memory. Is that enough? > > > > I think that yes. The page allocator will kick out Cached pages when > > needed. > > > > > ----------------------------------------------------------------------------- > > > # lttctl -n trace1 -d -l /sys/kernel/debug/ltt -t /data/ltt/trace1 > > > Linux Trace Toolkit Trace Control 0.55-16102008 > > > > > > Controlling trace : trace1 > > > > > > Linux Trace Toolkit Trace Daemon 0.55-16102008 > > > > > > Reading from debugfs directory : /sys/kernel/debug/ltt/trace1 > > > Writing to trace directory : /data/ltt/trace1 > > > ----------------------------------------------------------------------------- > > > [Bryan] do some tests on target device, then check the channel and trace > > > ----------------------------------------------------------------------------- > > > # ls -l /sys/kernel/debug/ltt/trace1 > > > -r-------- root root 0 1980-01-06 00:10 cpu_0 > > > drwxr-xr-x root root 1980-01-06 00:10 control > > > # ls -l /sys/kernel/debug/ltt/trace1/control > > > -r-------- root root 0 1980-01-06 00:10 network_0 > > > -r-------- root root 0 1980-01-06 00:10 modules_0 > > > -r-------- root root 0 1980-01-06 00:10 processes_0 > > > -r-------- root root 0 1980-01-06 00:10 interrupts_0 > > > -r-------- root root 0 1980-01-06 00:10 metadata_0 > > > # ls -l /data/ltt/trace1 > > > drwxrwxrwx root root 1980-01-06 00:10 control > > > -rwxrwxrwx root root 29949952 1980-01-06 00:10 cpu_0 > > > # ls -l /data/ltt/trace1/control > > > -rwxrwxrwx root root 0 1980-01-06 00:10 metadata_0 > > > -rwxrwxrwx root root 0 1980-01-06 00:10 interrupts_0 > > > -rwxrwxrwx root root 0 1980-01-06 00:10 processes_0 > > > -rwxrwxrwx root root 0 1980-01-06 00:10 modules_0 > > > -rwxrwxrwx root root 0 1980-01-06 00:10 network_0 > > > ----------------------------------------------------------------------------- > > > [Bryan] It seems that the capacity has been exhausted by cpu_0, because 29004K available. > > > It's that correct that cpu_0 will grow to so huge (29M)? > > > > 29M is not huge. It is small :) Especially if the system is very busy. > > > > > ----------------------------------------------------------------------------- > > > # lttctl -n trace1 -d -R > > > Linux Trace Toolkit Trace Control 0.55-16102008 > > > > > > Controlling trace : trace1 > > > ^Z[1] + Stopped lttctl -n trace1 -d -R > > > ----------------------------------------------------------------------------- > > > [Bryan] hang in here...I have to break it by Ctrl+Z, What's problem? > > > > It's waiting for lttd to release the trace channels, but I guess that > > the lack of space is causing errors, so lttd keeps running. > > > > > ----------------------------------------------------------------------------- > > > # ls -l /data/ltt/trace1/ > > > drwxrwxrwx root root 1980-01-06 00:10 control > > > -rwxrwxrwx root root 29949952 1980-01-06 00:10 cpu_0 > > > # ls -l /data/ltt/trace1/control > > > -rwxrwxrwx root root 0 1980-01-06 00:10 metadata_0 > > > -rwxrwxrwx root root 0 1980-01-06 00:10 interrupts_0 > > > -rwxrwxrwx root root 0 1980-01-06 00:10 processes_0 > > > -rwxrwxrwx root root 0 1980-01-06 00:10 modules_0 > > > -rwxrwxrwx root root 0 1980-01-06 00:10 network_0 > > > # ls -l /sys/kernel/debug/ltt/trace1 > > > -r-------- root root 0 1980-01-06 00:10 cpu_0 > > > drwxr-xr-x root root 1980-01-06 00:10 control > > > # ls -l /sys/kernel/debug/ltt/trace1/control > > > -r-------- root root 0 1980-01-06 00:10 network_0 > > > -r-------- root root 0 1980-01-06 00:10 modules_0 > > > -r-------- root root 0 1980-01-06 00:10 processes_0 > > > -r-------- root root 0 1980-01-06 00:10 interrupts_0 > > > -r-------- root root 0 1980-01-06 00:10 metadata_0 > > > # df > > > /dev: 39564K total, 0K used, 39564K available (block size 4096) > > > /sqlite_stmt_journals: 4096K total, 0K used, 4096K available (block size 4096) > > > /system: 72704K total, 71680K used, 1024K available (block size 4096) > > > /data: 66816K total, 66732K used, 84K available (block size 4096) > > > /cache: 67584K total, 1156K used, 66428K available (block size 4096) > > > /sdcard: 991708K total, 147040K used, 844668K available (block size 4096) > > > ----------------------------------------------------------------------------- > > > [Bryan] I think the trace is incomplete. > > > > Yes, here /data only has 84K avail. It's full. > > > > Mathieu > > > > > ----------------------------------------------------------------------------- > > > > > > Please help me! Thank you very much! > > > > > > Best regards, > > > Bryan Hsieh > > > > > > SPEC, FIH Co., Ltd. > > > Tel:+886-3-270-0000 #2723 > > > Fax:+886-3-270-0001 > > > > > > > > > -----Original Message----- > > > From: Mathieu Desnoyers [mailto:compudj@krystal.dyndns.org] > > > Sent: Friday, July 31, 2009 11:43 PM > > > To: Bryan Hsieh > > > Cc: 'Ajit R'; 'ltt-dev at lists.casi.polymtl.ca'; Tommy Peng > > > Subject: Re: [ltt-dev] Some LTTng 0.43 issue on ARM > > > > > > * Bryan Hsieh (bryanhsieh at fihspec.com) wrote: > > > > Hi Ajit, > > > > > > > > I appreciate your reply! > > > > But there is a problem here. > > > > > > > > My installation of Busybox is 1.12.4. > > > > > > It sounds like either awk or sed or not built in your busybox > > > installation. > > > > > > Can you provide the output of > > > > > > cat /proc/ltt > > > > > > on your target system ? > > > > > > Check the content of ltt-armall to see which external programs it needs. > > > > > > Also, if you built the lttng probes as modules, you may need to load > > > them explicitely. > > > > > > Mathieu > > > > > > > > > > > > > > I follow your suggestion to test on x86 Desktop ARM simulator, the message said that can't find /proc/ltt. I think it should be, because I didn't patch to ARM simulator Linux kernel. > > > > > > > > [sbox-ARM: ~/ltt-control-0.55-16102008/lttctl] > ltt-armall > > > > Connecting all markers > > > > cat: /proc/ltt: No such file or directory > > > > Connecting high-rate markers to tap > > > > cat: /proc/ltt: No such file or directory > > > > > > > > It should be work on my target device, but it seems not. > > > > I can't find /ltt under /proc > > > > Maybe it's root cause, but why there is no /ltt/proc? And why there is no " cat: /proc/ltt: No such file or directory" message in my target device during running ltt-armall? > > > > > > > > BTW, I mount debugfs in init.c > > > > Mount("debugfs", "/sys/kernel/debug", "debugfs", 0, NULL); > > > > It should not be a problem. > > > > > > > > -----------------------------------------------------------------------------------------------------------Another problem: > > > > I command as following steps. > > > > # lttctl -n trace1 -q > > > > Linux Trace Toolkit Trace Control 0.55-16102008 > > > > Controlling trace : trace1 > > > > > > > > # lttctl -n trace1 -r > > > > Linux Trace Toolkit Trace Control 0.55-16102008 > > > > Controlling trace : trace1 > > > > > > > > >>> Android shell hang in here, I have to press Ctrl+Z to force stopping - lttctl -n trace1 -r .Why? Is anything wrong? > > > > # ls -l ../tmp/trace1 > > > > drwxrwxrwx root root 1980-01-07 06:37 control > > > > -rwxrwxrwx root root 1048576 1980-01-07 06:37 cpu_0 > > > > # ls -l ../tmp/trace1/control > > > > -rwxrwxrwx root root 0 1980-01-07 06:37 metadata_0 > > > > -rwxrwxrwx root root 0 1980-01-07 06:37 interrupts_0 > > > > -rwxrwxrwx root root 10240 1980-01-07 06:37 processes_0 > > > > -rwxrwxrwx root root 65536 1980-01-07 06:37 modules_0 > > > > -rwxrwxrwx root root 65536 1980-01-07 06:37 network_0 > > > > > > > > The Channel in /sys/kernel/debug/ltt/trace1 is clear. Does it cause by destroy channel command? > > > > > > > > The metadata_0 and interrupts_0 are empty, Is that correct? I've done some key press, the interrupts_o shouldn't be empt. > > > > > > > > And those files almost filled by 0 except for the short header I can't make sure, it looks like incomplete. > > > > > > > > I hope you can help me to solve these issues, I am appreciate you so much.. > > > > > > > > Best regards, > > > > Bryan Hsieh > > > > > > > > SPEC, FIH Co., Ltd. > > > > Tel:+886-3-270-0000 #2723 > > > > Fax:+886-3-270-0001 > > > > > > > > > > > > -----Original Message----- > > > > From: Ajit R [mailto:ajitrzid@yahoo.com] > > > > Sent: Friday, July 31, 2009 6:17 PM > > > > To: 'ltt-dev at lists.casi.polymtl.ca'; Bryan Hsieh > > > > Cc: Tommy Peng > > > > Subject: Re: [ltt-dev] Some LTTng 0.43 issue on ARM > > > > > > > > > > > > Hi Bryan, > > > > My replies with the [Ajit] tag. > > > > LTT Gods correct me if I am wrong. > > > > > > > > Thanks > > > > Ajit > > > > > > > > --- On Fri, 7/31/09, Bryan Hsieh <bryanhsieh at fihspec.com> wrote: > > > > > > > > > From: Bryan Hsieh <bryanhsieh@fihspec.com> > > > > > Subject: [ltt-dev] Some LTTng 0.43 issue on ARM > > > > > To: "'ltt-dev at lists.casi.polymtl.ca'" <ltt-dev at lists.casi.polymtl.ca> > > > > > Cc: "Tommy Peng" <tommypeng at fihspec.com> > > > > > Date: Friday, July 31, 2009, 12:19 PM > > > > > > > > > Hi Mathieu, > > > > > all, > > > > > > > > > > > > > > > > > > > > > > > > > I've ported > > > > > LTTng 0.43 on Android successfully, > > > > > but I am wondering if the result is correct, please help me > > > > > to verify, thank > > > > > you very much! > > > > > > > > > > > > > > > > > > > > > > > > > My Traced > > > > > machine: > > > > > > > > > > ARM11 (ARM1136J(F)-S) / Linux > > > > > 2.6..27 > > > > > (Android 1.5) > > > > > > > > > > My > > > > > visualization machine: > > > > > > > > > > X86 / Sun xVM Virtual Box (Unbuntu > > > > > 8.10) > > > > > > > > > > > > > > > > > > > > LTTng: 0.43 > > > > > > > > > > LTTV: > > > > > 0.11.2 > > > > > > > > > > LTT Control: 0.55 > > > > > > > > > > > > > > > > > > > > > > > > > I > > > > > connect to Android > > > > > shell by ADB tool after Android bring up, then I command as > > > > > following step, the > > > > > command line options are different with LTTng manual, > > > > > because the example in manual > > > > > is for old version. > > > > > > > > > > > > > > > > > > > > > > > > > # ltt-armall > > > > > > > > > [ Ajit ] > > > > ltt-armall is a shell script. It connects + arms the markers in the > > > > kernel. So depending on your kernel + LTT version it would write some > > > > "text" into the /proc/ltt directory or a "1" into files in > > > > the debugfs directory. > > > > Make sure, the binaries for running the script are available in your target environment. For eg. awk, grep, sed etc. Busybox versions might have their limitations. If they do not work, you could do equivalent things on your Desktop and then write a simple script, doing the same on your target. > > > > Once you run "ltt-armall", you should see some debug statements in your "dmesg" like "connect" or "arm" etc. > > > > > > > > > > > > > > Connecting all > > > > > markers > > > > > > > > > > Connecting high-rate > > > > > markers to tap > > > > > > > > > > # lttctl > > > > > -n trace1 -d -l /sys/kernel/debug/ltt -t > > > > > /system/traces/trace1 > > > > > > > > > [Ajit] > > > > Not sure if your "-l" option is right. you have to point it to the place where you mounted your debugfs. > > > > > > > > > > > > > > Linux Trace Toolkit > > > > > Trace Control 0.55-16102008 > > > > > > > > > > > > > > > > > > > > > > > > > Controlling trace : > > > > > trace1 > > > > > > > > > > > > > > > > > > > > > > > > > Linux Trace Toolkit > > > > > Trace Daemon 0.55-16102008 > > > > > > > > > > > > > > > > > > > > > > > > > Reading from debugfs > > > > > directory : /sys/kernel/debug/ltt/trace1 > > > > > > > > > > > > > > > Writing to trace > > > > > directory : /system/traces/trace1 > > > > > > > > > > > > > > > > > > > > > > > > > It > > > > > seems the LTT control > > > > > and daemon work fine. I run some scenarios for 1 minute on > > > > > Android, but the trace > > > > > files are still empty (all files length are > > > > > 0) > > > > > > > > > > What's > > > > > wrong? Is > > > > > there any step I missed? Or the LTT control and daemon > > > > > don't work? > > > > > > > > > > > > > > > > > > > > > > > > > # ls > > > > > -l ../../sys/kernel/debug/ltt/ > > > > > > > > > > > > > > > > > > > > drwxr-xr-x > > > > > root > > > > > root > > > > > 1980-01-07 01:13 trace1 > > > > > > > > > > # ls > > > > > -l > > > > > ../../sys/kernel/debug/ltt/trace1 > > > > > > > > > > > > > > > -r-------- > > > > > root > > > > > root > > > > > 0 > > > > > 1980-01-07 01:13 cpu_0 > > > > > > > > > > > > > [Ajit] > > > > If you have connected and armed it right, you should see the cpu_0 > > > > file increasing in size. After sometime, disconnect by running the > > > > ltt-control -n whatever-name -R > > > > and then you will see files in the control directory (network, process, etc ) grown in size. Tar and move the trace directory to your host and do the needful. > > > > > > > > > > > > > drwxr-xr-x > > > > > root > > > > > root > > > > > 1980-01-07 01:13 control > > > > > > > > > > # ls > > > > > -l > > > > > ../../sys/kernel/debug/ltt/trace1/control > > > > > > > > > > > > > > > -r-------- > > > > > root > > > > > root > > > > > 0 > > > > > 1980-01-07 01:13 network_0 > > > > > > > > > > -r-------- > > > > > root > > > > > root > > > > > 0 > > > > > 1980-01-07 01:13 modules_0 > > > > > > > > > > -r-------- > > > > > root > > > > > root > > > > > 0 > > > > > 1980-01-07 01:13 processes_0 > > > > > > > > > > -r-------- > > > > > root > > > > > root > > > > > 0 > > > > > 1980-01-07 01:13 interrupts_0 > > > > > > > > > > -r-------- > > > > > root > > > > > root > > > > > 0 1980-01-07 01:13 metadata_0 > > > > > > > > > > # ls -l > > > > > .../traces/ > > > > > > > > > > > > > > > > > > > > drwxrwxrwx > > > > > root > > > > > root > > > > > 1980-01-07 01:13 trace1 > > > > > > > > > > # ls -l > > > > > .../traces/trace1 > > > > > > > > > > drwxrwxrwx > > > > > root > > > > > root > > > > > 1980-01-07 01:13 control > > > > > > > > > > -rwxrwxrwx > > > > > root > > > > > root > > > > > 0 > > > > > 1980-01-07 01:13 cpu_0 > > > > > > > > > > # ls -l > > > > > .../traces/trace1/control > > > > > > > > > > > > > > > > > > > > -rwxrwxrwx > > > > > root > > > > > root > > > > > 0 > > > > > 1980-01-07 01:13 metadata_0 > > > > > > > > > > -rwxrwxrwx > > > > > root > > > > > root > > > > > 0 > > > > > 1980-01-07 01:13 interrupts_0 > > > > > > > > > > -rwxrwxrwx > > > > > root > > > > > root > > > > > 0 > > > > > 1980-01-07 01:13 processes_0 > > > > > > > > > > -rwxrwxrwx > > > > > root > > > > > root > > > > > 0 > > > > > 1980-01-07 01:13 modules_0 > > > > > > > > > > -rwxrwxrwx > > > > > root > > > > > root > > > > > 0 > > > > > 1980-01-07 01:13 network_0 > > > > > > > > > > > > > > > > > > > > > > > > > BTW, > > > > > I would like to > > > > > know what is the channel concept for LTTng? What's > > > > > the relationship between > > > > > trace and channel? What's the normal, flight recorder > > > > > and hybrid mode? > > > > > > > > > > Which > > > > > files should I get > > > > > for LTTV on another visualization > > > > > machine? > > > > > > > > > > So far, I > > > > > can't find related document or the > > > > > information isn't enough for me. > > > > > > > > > > > > > > > > > > > > > > > > > Thanks for your > > > > > help! > > > > > > > > > > > > > > > > > > > > > > > > > Best > > > > > regards, > > > > > > > > > > Bryan Hsieh > > > > > > > > > > SPEC, FIH > > > > > Co., Ltd. > > > > > > > > > > Tel:+886-3-270-0000 #2723 > > > > > > > > > > Fax:+886-3-270-0001 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > > > ltt-dev mailing list > > > > > ltt-dev at lists.casi.polymtl.ca > > > > > http://lists.casi.polymtl.ca/cgi-bin/mailman/listinfo/ltt-dev > > > > > > > > > > > > > > > > > > > > > This e-mail message together with any attachments thereto (if any) is confidential, protected under an enforceable non-disclosure agreement, > > > > intended only for the use of the named recipient(s) above and may contain information that is privileged, belonging to professional work products > > > > or exempt from disclosure under applicable laws. Any unauthorized review, use, copying, disclosure, or distribution of any information contained > > > > in or attached to this transmission is STRICTLY PROHIBITED and may be against the laws. If you have received this message in error, or are not > > > > the named recipient(s), please immediately notify the sender by e-mail or telephone at IT support hotline 886-3-2700000 EXT 1333 and delete this > > > > e-mail message and any attached documentation from your computer. Receipt by anyone other than the intended recipient(s) is not a waiver of any > > > > attorney-client or work product privilege. Thank you! > > > > > > > > > > > > _______________________________________________ > > > > ltt-dev mailing list > > > > ltt-dev at lists.casi.polymtl.ca > > > > http://lists.casi.polymtl.ca/cgi-bin/mailman/listinfo/ltt-dev > > > > > > > > > > -- > > > Mathieu Desnoyers > > > OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F BA06 3F25 A8FE 3BAE 9A68 > > > > > > This e-mail message together with any attachments thereto (if any) is confidential, protected under an enforceable non-disclosure agreement, > > > intended only for the use of the named recipient(s) above and may contain information that is privileged, belonging to professional work products > > > or exempt from disclosure under applicable laws. Any unauthorized review, use, copying, disclosure, or distribution of any information contained > > > in or attached to this transmission is STRICTLY PROHIBITED and may be against the laws. If you have received this message in error, or are not > > > the named recipient(s), please immediately notify the sender by e-mail or telephone at IT support hotline 886-3-2700000 EXT 1333 and delete this > > > e-mail message and any attached documentation from your computer. Receipt by anyone other than the intended recipient(s) is not a waiver of any > > > attorney-client or work product privilege. Thank you! > > > > Content-Description: busybox version and command query.txt > > > # grep --help > > > BusyBox v1.12.4 (2009-07-29 23:17:25 CST) multi-call binary > > > > > > Usage: grep [-HhrilLnqvsoeFEABC] PATTERN [FILEs...] > > > > > > Search for PATTERN in each FILE or standard input > > > > > > Options: > > > -H Prefix output lines with filename where match was found > > > -h Suppress the prefixing filename on output > > > -r Recurse subdirectories > > > -i Ignore case distinctions > > > -l List names of files that match > > > -L List names of files that do not match > > > -n Print line number with output lines > > > -q Quiet. Return 0 if PATTERN is found, 1 otherwise > > > -v Select non-matching lines > > > -s Suppress file open/read error messages > > > -c Only print count of matching lines > > > -o Show only the part of a line that matches PATTERN > > > -m MAX Match up to MAX times per file > > > -F PATTERN is a set of newline-separated strings > > > -E PATTERN is an extended regular expression > > > -e PTRN Pattern to match > > > -f FILE Read pattern from file > > > -A Print NUM lines of trailing context > > > -B Print NUM lines of leading context > > > -C Print NUM lines of output context > > > > > > # awk --help > > > BusyBox v1.12.4 (2009-07-29 23:17:25 CST) multi-call binary > > > > > > Usage: awk [OPTION]... [program-text] [FILE...] > > > > > > Options: > > > -v var=val Set variable > > > -F sep Use sep as field separator > > > -f file Read program from file > > > > > > # sort --help > > > BusyBox v1.12.4 (2009-07-29 23:17:25 CST) multi-call binary > > > > > > Usage: sort [-nrugMcszbdfimSTokt] [-o FILE] [-k start[.offset][opts][,end[.offset][opts]] [-t CHAR] [FILE]... > > > > > > Sort lines of text > > > > > > Options: > > > -b Ignore leading blanks > > > -c Check whether input is sorted > > > -d Dictionary order (blank or alphanumeric only) > > > -f Ignore case > > > -g General numerical sort > > > -i Ignore unprintable characters > > > -k Sort key > > > -M Sort month > > > -n Sort numbers > > > -o Output to file > > > -k Sort by key > > > -t CHAR Key separator > > > -r Reverse sort order > > > -s Stable (don't sort ties alphabetically) > > > -u Suppress duplicate lines > > > -z Lines are terminated by NUL, not newline > > > -mST Ignored for GNU compatibility > > > > > > # cat --help > > > unknown option -- -usage: cat [-beflnstv] [-] [file ...] > > > # > > > > > > > Content-Description: LTTng Execution Log.txt > > > # ltt-armall-1 > > > Connecting all markers > > > Connecting fs_buffer_wait_end > > > Connecting fs_buffer_wait_start > > > Connecting fs_close > > > Connecting fs_exec > > > Connecting fs_ioctl > > > Connecting fs_llseek > > > Connecting fs_lseek > > > Connecting fs_open > > > Connecting fs_pollfd > > > Connecting fs_pread64 > > > Connecting fs_pwrite64 > > > Connecting fs_read > > > Connecting fs_readv > > > Connecting fs_select > > > Connecting fs_write > > > Connecting fs_writev > > > Connecting input_event > > > Connecting ipc_msg_create > > > Connecting ipc_sem_create > > > Connecting ipc_shm_create > > > Connecting kernel_arch_ipc_call > > > Connecting kernel_arch_kthread_create > > > Connecting kernel_arch_syscall_entry > > > Connecting kernel_arch_syscall_exit > > > Connecting kernel_arch_trap_entry > > > Connecting kernel_arch_trap_exit > > > Connecting kernel_irq_entry > > > Connecting kernel_irq_exit > > > Connecting kernel_kthread_stop > > > Connecting kernel_kthread_stop_ret > > > Connecting kernel_module_free > > > Connecting kernel_module_load > > > Connecting kernel_printk > > > Connecting kernel_process_exit > > > Connecting kernel_process_fork > > > Connecting kernel_process_free > > > Connecting kernel_process_wait > > > Connecting kernel_sched_migrate_task > > > Connecting kernel_sched_schedule > > > Connecting kernel_sched_try_wakeup > > > Connecting kernel_sched_wait_task > > > Connecting kernel_sched_wakeup_new_task > > > Connecting kernel_send_signal > > > Connecting kernel_softirq_entry > > > Connecting kernel_softirq_exit > > > Connecting kernel_softirq_raise > > > Connecting kernel_tasklet_high_entry > > > Connecting kernel_tasklet_high_exit > > > Connecting kernel_tasklet_low_entry > > > Connecting kernel_tasklet_low_exit > > > Connecting kernel_timer_itimer_expired > > > Connecting kernel_timer_itimer_set > > > Connecting kernel_timer_set > > > Connecting kernel_timer_timeout > > > Connecting kernel_timer_update_time > > > Connecting kernel_vprintk > > > Connecting list_file_descriptor > > > Connecting list_interrupt > > > Connecting list_module > > > Connecting list_network_ip_interface > > > Connecting list_network_ipv4_interface > > > Connecting list_process_state > > > Connecting list_statedump_end > > > Connecting list_vm_map > > > Connecting mm_handle_fault_entry > > > Connecting mm_handle_fault_exit > > > Connecting mm_huge_page_alloc > > > Connecting mm_huge_page_free > > > Connecting mm_page_alloc > > > Connecting mm_page_free > > > Connecting mm_wait_on_page_end > > > Connecting mm_wait_on_page_start > > > Connecting rcu_classic_callback > > > Connecting rcu_classic_call_rcu > > > Connecting rcu_classic_call_rcu_bh > > > Connecting statedump_softirq_vec > > > Connecting high-rate markers to tap > > > Connecting fs_buffer_wait_end > > > Connecting fs_buffer_wait_start > > > Connecting fs_close > > > Connecting fs_exec > > > Connecting fs_ioctl > > > Connecting fs_llseek > > > Connecting fs_lseek > > > Connecting fs_open > > > Connecting fs_pollfd > > > Connecting fs_pread64 > > > Connecting fs_pwrite64 > > > Connecting fs_read > > > Connecting fs_readv > > > Connecting fs_select > > > Connecting fs_write > > > Connecting fs_writev > > > Connecting input_event > > > Connecting ipc_msg_create > > > Connecting ipc_sem_create > > > Connecting ipc_shm_create > > > Connecting kernel_arch_ipc_call > > > Connecting kernel_arch_kthread_create > > > Connecting kernel_arch_syscall_entry > > > Connecting kernel_arch_syscall_exit > > > Connecting kernel_arch_trap_entry > > > Connecting kernel_arch_trap_exit > > > Connecting kernel_irq_entry > > > Connecting kernel_irq_exit > > > Connecting kernel_kthread_stop > > > Connecting kernel_kthread_stop_ret > > > Connecting kernel_module_free > > > Connecting kernel_module_load > > > Connecting kernel_printk > > > Connecting kernel_process_exit > > > Connecting kernel_process_fork > > > Connecting kernel_process_free > > > Connecting kernel_process_wait > > > Connecting kernel_sched_migrate_task > > > Connecting kernel_sched_schedule > > > Connecting kernel_sched_try_wakeup > > > Connecting kernel_sched_wait_task > > > Connecting kernel_sched_wakeup_new_task > > > Connecting kernel_send_signal > > > Connecting kernel_softirq_entry > > > Connecting kernel_softirq_exit > > > Connecting kernel_softirq_raise > > > Connecting kernel_tasklet_high_entry > > > Connecting kernel_tasklet_high_exit > > > Connecting kernel_tasklet_low_entry > > > Connecting kernel_tasklet_low_exit > > > Connecting kernel_timer_itimer_expired > > > Connecting kernel_timer_itimer_set > > > Connecting kernel_timer_set > > > Connecting kernel_timer_timeout > > > Connecting kernel_timer_update_time > > > Connecting kernel_vprintk > > > Connecting list_file_descriptor > > > Connecting list_interrupt > > > Connecting list_module > > > Connecting list_network_ip_interface > > > Connecting list_network_ipv4_interface > > > Connecting list_process_state > > > Connecting list_statedump_end > > > Connecting list_vm_map > > > Connecting mm_handle_fault_entry > > > Connecting mm_handle_fault_exit > > > Connecting mm_huge_page_alloc > > > Connecting mm_huge_page_free > > > Connecting mm_page_alloc > > > Connecting mm_page_free > > > Connecting mm_wait_on_page_end > > > Connecting mm_wait_on_page_start > > > Connecting rcu_classic_callback > > > Connecting rcu_classic_call_rcu > > > Connecting rcu_classic_call_rcu_bh > > > Connecting statedump_softirq_vec > > > > > > # cat /proc/ltt > > > marker: kernel_arch_kthread_create format: "pid %ld fn %p" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: kernel_arch_syscall_exit format: " " state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: kernel_arch_syscall_entry format: "syscall_id %d ip #p%ld" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: kernel_arch_ipc_call format: "call %u first %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: kernel_arch_trap_exit format: " " state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: kernel_arch_trap_entry format: "trap_id %ld ip #p%ld" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: statedump_softirq_vec format: "id %d address %p symbol %s" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: list_module format: "name %s state %d refcount %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: core_marker_format format: "name %s format %s" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: core_marker_format format: "name %s format %s" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: list_statedump_end format: " " state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: list_process_state format: "pid %d parent_pid %d name %s type %d mode %d submode %d status %d tgid %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: list_file_descriptor format: "filename %s pid %d fd %u" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: list_vm_map format: "pid %d start %lu end %lu flags %lu pgoff %lu inode %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: list_interrupt format: "name %s action %s irq_id %u" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: list_network_ip_interface format: "name %s address #4u%lu up %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: list_network_ipv4_interface format: "name %s address #4u%lu up %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: core_marker_id format: "name %s id %hu int #1u%zu long #1u%zu pointer #1u%zu size_t #1u%zu alignment #1u%u" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: core_marker_id format: "name %s id %hu int #1u%zu long #1u%zu pointer #1u%zu size_t #1u%zu alignment #1u%u" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: core_marker_format format: "name %s format %s" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: core_marker_id format: "name %s id %hu int #1u%zu long #1u%zu pointer #1u%zu size_t #1u%zu alignment #1u%u" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: core_marker_id format: "name %s id %hu int #1u%zu long #1u%zu pointer #1u%zu size_t #1u%zu alignment #1u%u" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: kernel_module_load format: "name %s" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: kernel_module_free format: "name %s" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: kernel_vprintk format: "loglevel %c string %s ip %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: kernel_printk format: "ip %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: kernel_timer_timeout format: "pid %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: kernel_timer_update_time format: "jiffies #8u%llu xtime_sec %ld xtime_nsec %ld walltomonotonic_sec %ld walltomonotonic_nsec %ld" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: kernel_timer_set format: "expires %lu function %p data %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: kernel_timer_itimer_set format: "which %d interval_sec %ld interval_usec %ld value_sec %ld value_usec %ld" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: kernel_timer_itimer_expired format: "pid %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: kernel_process_fork format: "parent_pid %d child_pid %d child_tgid %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: kernel_process_wait format: "pid %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: kernel_process_exit format: "pid %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: kernel_process_free format: "pid %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: kernel_send_signal format: "pid %d signal %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: kernel_sched_migrate_task format: "pid %d state %ld dest_cpu %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: kernel_sched_schedule format: "prev_pid %d next_pid %d prev_state %ld" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: kernel_sched_wakeup_new_task format: "pid %d state %ld cpu_id %u" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: kernel_sched_try_wakeup format: "pid %d state %ld cpu_id %u" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: kernel_sched_wait_task format: "pid %d state %ld" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: kernel_kthread_stop_ret format: "ret %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: kernel_kthread_stop format: "pid %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: kernel_tasklet_high_exit format: "func %p data %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: kernel_tasklet_high_entry format: "func %p data %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: kernel_tasklet_low_exit format: "func %p data %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: kernel_tasklet_low_entry format: "func %p data %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: kernel_softirq_raise format: "softirq_id %u" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: kernel_softirq_exit format: "softirq_id %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: kernel_softirq_entry format: "softirq_id %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: kernel_irq_exit format: "handled #1u%u" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: kernel_irq_entry format: "irq_id %u kernel_mode %u ip %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: mm_page_alloc format: "order %u pfn %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: mm_page_free format: "order %u pfn %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: mm_handle_fault_exit format: "res %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: mm_handle_fault_entry format: "address %lu ip #p%ld write_access %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: mm_huge_page_alloc format: "pfn %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: mm_huge_page_free format: "pfn %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: mm_wait_on_page_end format: "pfn %lu bit_nr %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: mm_wait_on_page_start format: "pfn %lu bit_nr %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: fs_pollfd format: "fd %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: fs_select format: "fd %d timeout #8d%lld" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: fs_writev format: "fd %lu vlen %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: fs_readv format: "fd %lu vlen %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: fs_pwrite64 format: "fd %u count %zu pos %llu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: fs_pread64 format: "fd %u count %zu pos %llu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: fs_write format: "fd %u count %zu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: fs_read format: "fd %u count %zu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: fs_llseek format: "fd %u offset %lld origin %u" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: fs_lseek format: "fd %u offset %ld origin %u" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: fs_close format: "fd %u" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: fs_open format: "fd %d filename %s" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: fs_ioctl format: "fd %u cmd %u arg %lu" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: fs_exec format: "filename %s" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: fs_buffer_wait_end format: "bh %p" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: fs_buffer_wait_start format: "bh %p" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: ipc_shm_create format: "id %ld flags %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: ipc_sem_create format: "id %ld flags %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: ipc_msg_create format: "id %ld flags %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: lockdep_lock_release format: "retaddr 0x%lX lock %p nested %d" state: 0 call: 0xc0093374 probe single : 0xc00920ec > > > marker: lockdep_lock_acquire format: "retaddr 0x%lX subclass %u lock %p trylock %d read %d hardirqs_off %d" state: 0 call: 0xc0093374 probe single : 0xc00920ec > > > marker: lockdep_softirqs_off format: "retaddr 0x%lX" state: 0 call: 0xc0093374 probe single : 0xc00920ec > > > marker: lockdep_softirqs_on format: "retaddr 0x%lX" state: 0 call: 0xc0093374 probe single : 0xc00920ec > > > marker: lockdep_hardirqs_off format: "retaddr 0x%lX" state: 0 call: 0xc0093374 probe single : 0xc00920ec > > > marker: lockdep_hardirqs_on format: "retaddr 0x%lX" state: 0 call: 0xc0093374 probe single : 0xc00920ec > > > marker: rcu_classic_call_rcu_bh format: "func %p ip 0x%lX" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: rcu_classic_call_rcu format: "func %p ip 0x%lX" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: rcu_classic_callback format: "func %p" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > marker: input_event format: "type %u code %u value %d" state: 1 call: 0xc0093374 probe single : 0xc015d504 > > > > > > [Bryan] Maybe caused by privilege, the the /proc/ltt do not be replaced with content below. It's that will be a problem? > > > connect statedump_softirq_vec ltt_tap_marker dynami > > > > > > # df > > > /dev: 39564K total, 0K used, 39564K available (block size 4096) > > > /sqlite_stmt_journals: 4096K total, 0K used, 4096K available (block size 4096) > > > /system: 72704K total, 71680K used, 1024K available (block size 4096) > > > /data: 66816K total, 37812K used, 29004K available (block size 4096) > > > /cache: 67584K total, 1156K used, 66428K available (block size 4096) > > > /sdcard: 991708K total, 147040K used, 844668K available (block size 4096) > > > > > > # lttctl -n trace1 -d -l /sys/kernel/debug/ltt -t /data/ltt/trace1 > > > Linux Trace Toolkit Trace Control 0.55-16102008 > > > > > > Controlling trace : trace1 > > > > > > Linux Trace Toolkit Trace Daemon 0.55-16102008 > > > > > > Reading from debugfs directory : /sys/kernel/debug/ltt/trace1 > > > Writing to trace directory : /data/ltt/trace1 > > > > > > [Bryan] there is 29004K available for /data/ltt/trace1, is that enough? > > > [Bryan] do some tests on target device, then check the channel and trace > > > > > > # ls -l /sys/kernel/debug/ltt/trace1 > > > -r-------- root root 0 1980-01-06 00:10 cpu_0 > > > drwxr-xr-x root root 1980-01-06 00:10 control > > > # ls -l /sys/kernel/debug/ltt/trace1/control > > > -r-------- root root 0 1980-01-06 00:10 network_0 > > > -r-------- root root 0 1980-01-06 00:10 modules_0 > > > -r-------- root root 0 1980-01-06 00:10 processes_0 > > > -r-------- root root 0 1980-01-06 00:10 interrupts_0 > > > -r-------- root root 0 1980-01-06 00:10 metadata_0 > > > # ls -l /data/ltt/trace1 > > > drwxrwxrwx root root 1980-01-06 00:10 control > > > -rwxrwxrwx root root 29949952 1980-01-06 00:10 cpu_0 > > > # ls -l /data/ltt/trace1/control > > > -rwxrwxrwx root root 0 1980-01-06 00:10 metadata_0 > > > -rwxrwxrwx root root 0 1980-01-06 00:10 interrupts_0 > > > -rwxrwxrwx root root 0 1980-01-06 00:10 processes_0 > > > -rwxrwxrwx root root 0 1980-01-06 00:10 modules_0 > > > -rwxrwxrwx root root 0 1980-01-06 00:10 network_0 > > > > > > [Bryan] It seems that the space has been exhausted by cpu_0, it's that correct that cpu_0 will grow to so huge(29M)? > > > How much file space should I prepare for LTTng? > > > > > > # lttctl -n trace1 -d -R > > > Linux Trace Toolkit Trace Control 0.55-16102008 > > > > > > Controlling trace : trace1 > > > > > > [Bryan] hang in here...I have to break it by Ctrl+Z, What's problem? > > > > > > ^Z[1] + Stopped lttctl -n trace1 -d -R > > > # ls -l /data/ltt/trace1/ > > > drwxrwxrwx root root 1980-01-06 00:10 control > > > -rwxrwxrwx root root 29949952 1980-01-06 00:10 cpu_0 > > > # ls -l /data/ltt/trace1/control > > > -rwxrwxrwx root root 0 1980-01-06 00:10 metadata_0 > > > -rwxrwxrwx root root 0 1980-01-06 00:10 interrupts_0 > > > -rwxrwxrwx root root 0 1980-01-06 00:10 processes_0 > > > -rwxrwxrwx root root 0 1980-01-06 00:10 modules_0 > > > -rwxrwxrwx root root 0 1980-01-06 00:10 network_0 > > > # ls -l /sys/kernel/debug/ltt/trace1 > > > -r-------- root root 0 1980-01-06 00:10 cpu_0 > > > drwxr-xr-x root root 1980-01-06 00:10 control > > > # ls -l /sys/kernel/debug/ltt/trace1/control > > > -r-------- root root 0 1980-01-06 00:10 network_0 > > > -r-------- root root 0 1980-01-06 00:10 modules_0 > > > -r-------- root root 0 1980-01-06 00:10 processes_0 > > > -r-------- root root 0 1980-01-06 00:10 interrupts_0 > > > -r-------- root root 0 1980-01-06 00:10 metadata_0 > > > # df > > > /dev: 39564K total, 0K used, 39564K available (block size 4096) > > > /sqlite_stmt_journals: 4096K total, 0K used, 4096K available (block size 4096) > > > /system: 72704K total, 71680K used, 1024K available (block size 4096) > > > /data: 66816K total, 66732K used, 84K available (block size 4096) > > > /cache: 67584K total, 1156K used, 66428K available (block size 4096) > > > /sdcard: 991708K total, 147040K used, 844668K available (block size 4096) > > > > > > > > > > > > > Content-Description: LTTng Kernel Config.txt > > > > > > [ ] Kernel->user space relay support (formerly relayfs) > > > [*] Activate tracepoints x x > > > [*] Activate markers x x > > > Linux Trace Toolkit ---> > > > <*> Compile lttng tracing probes x x > > > [*] LTTng fine-grained timestamping x x > > > [*] Linux Trace Toolkit Instrumentation Support x x > > > <*> Linux Trace Toolkit High-speed Lockless Data Relay x x > > > <*> Linux Trace Toolkit Lock-Protected Data Relay x x > > > [*] Debug check for random access in ltt relay buffers x x > > > <*> Linux Trace Toolkit Serializer x x > > > <*> Linux Trace Toolkit Marker Control x x > > > <*> Linux Trace Toolkit Tracer x x > > > [*] Align Linux Trace Toolkit Traces x x > > > [*] Add event size field to LTT events for tracer debugging x x > > > [*] Support trace extraction from crash dump x x > > > <*> Linux Trace Toolkit Netlink Controller x x > > > <*> Linux Trace Toolkit State Dump > > > > -- > > Mathieu Desnoyers > > OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F BA06 3F25 A8FE 3BAE 9A68 > > This e-mail message together with any attachments thereto (if any) is confidential, protected under an enforceable non-disclosure agreement, > > intended only for the use of the named recipient(s) above and may contain information that is privileged, belonging to professional work products > > or exempt from disclosure under applicable laws. Any unauthorized review, use, copying, disclosure, or distribution of any information contained > > in or attached to this transmission is STRICTLY PROHIBITED and may be against the laws. If you have received this message in error, or are not > > the named recipient(s), please immediately notify the sender by e-mail or telephone at IT support hotline 886-3-2700000 EXT 1333 and delete this > > e-mail message and any attached documentation from your computer. Receipt by anyone other than the intended recipient(s) is not a waiver of any > > attorney-client or work product privilege. Thank you! > > > > -- > Mathieu Desnoyers > OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F BA06 3F25 A8FE 3BAE 9A68 > > _______________________________________________ > ltt-dev mailing list > ltt-dev at lists.casi.polymtl.ca > http://lists.casi.polymtl.ca/cgi-bin/mailman/listinfo/ltt-dev > -- Mathieu Desnoyers OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F BA06 3F25 A8FE 3BAE 9A68 ^ permalink raw reply [flat|nested] 9+ messages in thread
* [ltt-dev] Change process name to be command line 2009-08-06 2:52 ` Bryan Hsieh 2009-08-06 17:20 ` Mathieu Desnoyers @ 2009-08-27 8:56 ` Bryan Hsieh 2009-08-27 10:31 ` Ajit R 1 sibling, 1 reply; 9+ messages in thread From: Bryan Hsieh @ 2009-08-27 8:56 UTC (permalink / raw) Hi All, My user space process name are all the same - "app_process" in LTTV, I'd like to distinguish with them, it seems LTT cat from /proc/[number]/stat. I'd like to change to be command line from /proc/[number]/cmdline when the process name is specific name or just like what the busybox implement in PS, where can I modify in LTT patch? Thank you very much! Best regards, Bryan Hsieh This e-mail message together with any attachments thereto (if any) is confidential, protected under an enforceable non-disclosure agreement, intended only for the use of the named recipient(s) above and may contain information that is privileged, belonging to professional work products or exempt from disclosure under applicable laws. Any unauthorized review, use, copying, disclosure, or distribution of any information contained in or attached to this transmission is STRICTLY PROHIBITED and may be against the laws. If you have received this message in error, or are not the named recipient(s), please immediately notify the sender by e-mail or telephone at IT support hotline 886-3-2700000 EXT 1333 and delete this e-mail message and any attached documentation from your computer. Receipt by anyone other than the intended recipient(s) is not a waiver of any attorney-client or work product privilege. Thank you! ^ permalink raw reply [flat|nested] 9+ messages in thread
* [ltt-dev] Change process name to be command line 2009-08-27 8:56 ` [ltt-dev] Change process name to be command line Bryan Hsieh @ 2009-08-27 10:31 ` Ajit R 0 siblings, 0 replies; 9+ messages in thread From: Ajit R @ 2009-08-27 10:31 UTC (permalink / raw) Hi --- On Thu, 8/27/09, Bryan Hsieh <bryanhsieh at fihspec.com> wrote: > From: Bryan Hsieh <bryanhsieh@fihspec.com> > Subject: [ltt-dev] Change process name to be command line > To: "'ltt-dev at lists.casi.polymtl.ca'" <ltt-dev at lists.casi.polymtl.ca> > Date: Thursday, August 27, 2009, 2:26 PM > Hi All, > > My user space process name are all the same - "app_process" > in LTTV, I'd like to distinguish with them, it seems LTT cat > from /proc/[number]/stat. I'd like to change to be command > line from /proc/[number]/cmdline when the process name is > specific name or just like what the busybox implement in PS, > where can I modify in LTT patch? > Thank you very much! > > Best regards, > Bryan Hsieh _______________________________________________ > ltt-dev mailing list > ltt-dev at lists.casi.polymtl.ca > http://lists.casi.polymtl.ca/cgi-bin/mailman/listinfo/ltt-dev > You could change the app names by using the prctl system call. This works for threads too. Refer to this discussion on the ltt-dev about how you can get thread/process names and can be seen by ltt: http://lists.casi.polymtl.ca/pipermail/ltt-dev/2009-August/001859.html Also if you have an older ltt implementation you might have to use the marker differently, in "set_task_comm" add this line trace_mark(fs_exec, "filename %s", tsk->comm). In Busybox too you would have to enable code to use the task structure's "comm" field printing. Regards Ajit ^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2009-08-27 10:31 UTC | newest]
Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
[not found] <7660C7A486834C48B1487748EA32BCA3503EE618A8@spec-ty-exch.fihspec.com>
[not found] ` <345025.22508.qm@web63207.mail.re1.yahoo.com>
[not found] ` <7660C7A486834C48B1487748EA32BCA3503EE618AA@spec-ty-exch.fihspec.com>
[not found] ` <20090731154237.GA8847@Krystal>
2009-08-04 8:28 ` [ltt-dev] Some LTTng 0.43 issue on ARM Bryan Hsieh
2009-08-04 8:28 ` Bryan Hsieh
2009-08-04 8:28 ` Bryan Hsieh
2009-08-04 13:18 ` Mathieu Desnoyers
2009-08-06 2:52 ` Bryan Hsieh
2009-08-06 17:20 ` Mathieu Desnoyers
2009-08-06 17:24 ` Mathieu Desnoyers
2009-08-27 8:56 ` [ltt-dev] Change process name to be command line Bryan Hsieh
2009-08-27 10:31 ` Ajit R
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox