From mboxrd@z Thu Jan 1 00:00:00 1970 From: compudj@krystal.dyndns.org (Mathieu Desnoyers) Date: Wed, 10 Nov 2010 10:14:37 -0500 Subject: [ltt-dev] "Kernel/LTTV event size differs for event udpv4_rcv_extended: kernel 32, LTTV 28" error when view trace In-Reply-To: <20101110140821.GA8878@Krystal> References: <4CD3A32D.6030706@windriver.com> <4CD75636.1020305@polymtl.ca> <4CD788AE.60301@windriver.com> <20101108095545.GA17190@Krystal> <4CD8F840.2080704@windriver.com> <20101109114500.GA18027@Krystal> <20101109115743.GA19744@Krystal> <4CDA477B.7010508@windriver.com> <20101110140821.GA8878@Krystal> Message-ID: <20101110151437.GA14585@Krystal> * Mathieu Desnoyers (compudj at krystal.dyndns.org) wrote: > * xufeng zhang (xufeng.zhang at windriver.com) wrote: > > On 11/09/2010 07:57 PM, Mathieu Desnoyers wrote: > >> * Mathieu Desnoyers (compudj at krystal.dyndns.org) wrote: > >> > >>> * xufeng zhang (xufeng.zhang at windriver.com) wrote: > >>> > >>>> On 11/08/2010 05:55 PM, Mathieu Desnoyers wrote: > >>>> > >>>>> * xufeng zhang (xufeng.zhang at windriver.com) wrote: > >>>>> > >>>>> > >>>>>> On 11/08/2010 09:45 AM, Benjamin Poirier wrote: > >>>>>> > >>>>>> > >>>>>>> On 05/11/10 02:24 AM, xufeng zhang wrote: > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>>> Hi, > >>>>>>>> I'm using ltt-control-0.86 and lttv-0.12.31 on ARM to trace the kernel, > >>>>>>>> however, > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>> Hello, > >>>>>>> > >>>>>>> Which version of the kernel tracer patches are you using? > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>> I have query the developers who merged LTTng into kernel, they said > >>>>>> there is no incompatibility > >>>>>> between LTTV and LTTng, so does there any other reasons which lead to > >>>>>> this error? > >>>>>> > >>>>>> Thanks, > >>>>>> Xufeng Zhang > >>>>>> > >>>>>> > >>>>>>> -Ben > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>>> I cannot view the trace after I have successfully generated, here is the > >>>>>>>> error > >>>>>>>> when I run 'lttv-gui -t trace1' command: > >>>>>>>> ** Message: statistics viewer : background computation data ready. > >>>>>>>> > >>>>>>>> ** (lttv.real:6559): WARNING **: Cannot find pin_in in schedchange 671 > >>>>>>>> > >>>>>>>> ** ERROR **: Kernel/LTTV event size differs for event > >>>>>>>> udpv4_rcv_extended: kernel 32, LTTV 28 > >>>>>>>> > >>>>>>>> > >>>>> Please compare the /ltt/probes/net-extended-trace.c file from your > >>>>> distro kernel tree to the: > >>>>> > >>>>> lttng-modules package, file: /probes/net-extended-trace.c > >>>>> > >>>>> Also compare the /ltt/ltt-type-serializer.h (or was it > >>>>> include/linux/ltt-type-serializer.h ?) from your distro kernel with the > >>>>> lttng-modules package: > >>>>> > >>>>> /ltt-type-serializer.h > >>>>> > >>>>> I remember having fixed this problem in the past months. Look closely at > >>>>> the structure layout for "struct serialize_l214421224411111", "struct > >>>>> serialize_l4412228" and "struct serialize_l4421224411111" vs the types > >>>>> passed to the DEFINE_MARKER_TP() in net-extended-trace.c. > >>>>> > >>>>> > >>>> I have compare my kernel tree to lttng-modules-0.19.2, there is no > >>>> different > >>>> in ltt-type-serializer.h file, and below is the diff info for > >>>> net-extended-trace.c: > >>>> > >>>> --- net-extended-trace.c.kernel 2010-11-09 15:09:24.000000000 +0800 > >>>> +++ net-extended-trace.c 2010-11-09 15:12:47.000000000 +0800 > >>>> @@ -20,14 +20,15 @@ > >>>> > >>>> #include > >>>> #include > >>>> -#include > >>>> #include > >>>> #include > >>>> #include > >>>> #include > >>>> #include > >>>> > >>>> -void probe_net_dev_xmit_extended(struct sk_buff *skb); > >>>> +#include "../ltt-type-serializer.h" > >>>> + > >>>> +void probe_net_dev_xmit_extended(void *_data, struct sk_buff *skb); > >>>> > >>>> DEFINE_MARKER_TP(net, dev_xmit_extended, net_dev_xmit, > >>>> probe_net_dev_xmit_extended, "skb 0x%lX network_protocol #n2u%hu " > >>>> @@ -35,7 +36,7 @@ > >>>> "tot_len #n2u%hu ihl #1u%u source #n2u%hu dest #n2u%hu seq > >>>> #n4u%lu " > >>>> "ack_seq #n4u%lu doff #1u%u ack #1u%u rst #1u%u syn #1u%u fin > >>>> #1u%u"); > >>>> > >>>> -notrace void probe_net_dev_xmit_extended(struct sk_buff *skb) > >>>> +notrace void probe_net_dev_xmit_extended(void *_data, struct sk_buff *skb) > >>>> { > >>>> struct marker *marker; > >>>> struct serialize_l214421224411111 data; > >>>> @@ -70,14 +71,14 @@ > >>>> &data, serialize_sizeof(data), sizeof(long)); > >>>> } > >>>> > >>>> -void probe_tcpv4_rcv_extended(struct sk_buff *skb); > >>>> +void probe_tcpv4_rcv_extended(void *_data, struct sk_buff *skb); > >>>> > >>>> DEFINE_MARKER_TP(net, tcpv4_rcv_extended, net_tcpv4_rcv, > >>>> probe_tcpv4_rcv_extended, "skb 0x%lX saddr #n4u%lu daddr #n4u%lu " > >>>> "tot_len #n2u%hu ihl #1u%u source #n2u%hu dest #n2u%hu seq > >>>> #n4u%lu " > >>>> "ack_seq #n4u%lu doff #1u%u ack #1u%u rst #1u%u syn #1u%u fin > >>>> #1u%u"); > >>>> > >>>> -notrace void probe_tcpv4_rcv_extended(struct sk_buff *skb) > >>>> +notrace void probe_tcpv4_rcv_extended(void *_data, struct sk_buff *skb) > >>>> { > >>>> struct marker *marker; > >>>> struct serialize_l4421224411111 data; > >>>> @@ -104,14 +105,14 @@ > >>>> &data, serialize_sizeof(data), sizeof(long)); > >>>> } > >>>> > >>>> -void probe_udpv4_rcv_extended(struct sk_buff *skb); > >>>> +void probe_udpv4_rcv_extended(void *_data, struct sk_buff *skb); > >>>> > >>>> DEFINE_MARKER_TP(net, udpv4_rcv_extended, net_udpv4_rcv, > >>>> probe_udpv4_rcv_extended, "skb 0x%lX saddr #n4u%lu daddr #n4u%lu " > >>>> "unicast #1u%u ulen #n2u%hu source #n2u%hu dest #n2u%hu " > >>>> "data_start #8u%lx"); > >>>> > >>>> -notrace void probe_udpv4_rcv_extended(struct sk_buff *skb) > >>>> +notrace void probe_udpv4_rcv_extended(void *_data, struct sk_buff *skb) > >>>> { > >>>> struct marker *marker; > >>>> struct serialize_l4412228 data; > >>>> > >>>> > >>>> I don't think the difference in net-extended-trace.c will make any impact. > >>>> This problem has troubled me for weeks. > >>>> > >>> Can you try to reproduce with the latest LTTng version ? It's very hard > >>> for us to help you if we don't know the LTTng version you are using, > >>> especially for a bug I am aware to have fixed recently. > >>> > >> Ah, I remember it now. It had something to do with alignment of 64-bit > >> fields on 32-bit machines. Older LTTng versions did align these fields > >> on the next 64-bit multiple. New LTTng versions do the same as gcc and > >> align these on the next 32-bit boundary. > >> > >> Check the diff of your include/linux/ltt-core.h ltt_align() function > >> between a recent lttng and yours. > >> > > Could you provide the content of this patch, I think my problem > > is almost the same. Thank you very much. > > Then you may be facing the following problem: your kernel LTTng version > is too recent for your LTTV version. I had to change both LTTV and LTTng > at the same time to support this different alignment style. > > So your LTTV is very probably incompatible with the LTTng version you > are using. Please compare yout "ltt_align" macro in lttv : ltt/ltt-private.h between a more recent (and also older) lttv and yours. Thanks, Mathieu > > Mathieu > > > > > Thanks, > > Xufeng Zhang > >> Thanks, > >> > >> Mathieu > >> > >> > >>> Also, you might simply try to unload the net-extended-trace module (or > >>> disable its markers) to see if it helps getting a parseable trace. > >>> > >>> Thanks, > >>> > >>> Mathieu > >>> > >>> > >>>> Thanks, > >>>> Xufeng Zhang > >>>> > >>>>> Thanks, > >>>>> > >>>>> Mathieu > >>>>> > >>>>> > >>>>> > >>>>> > >>>>>>>> aborting... > >>>>>>>> /usr/local/bin/lttv-gui: line 10: 6559 Aborted > >>>>>>>> $LTTV_CMD.real -m lttvwindow -m guievents -m guifilter -m guicontrolflow > >>>>>>>> -m resourceview -m guistat > >>>>>>>> > >>>>>>>> > >>>>>>>> And I found another people met the same problem on MIPS32, a patch have > >>>>>>>> also been provided for MIPS32. > >>>>>>>> Here is the similar code in ARM trace-clock.c: > >>>>>>>> linux/arch/arm/mach-omap2/trace-clock.c line 537 > >>>>>>>> void get_trace_clock(void) > >>>>>>>> { > >>>>>>>> spin_lock(&trace_clock_lock); > >>>>>>>> if (trace_clock_refcount++) > >>>>>>>> goto end; > >>>>>>>> _start_trace_clock(); > >>>>>>>> end: > >>>>>>>> spin_unlock(&trace_clock_lock); > >>>>>>>> } > >>>>>>>> EXPORT_SYMBOL_GPL(get_trace_clock); > >>>>>>>> > >>>>>>>> void put_trace_clock(void) > >>>>>>>> { > >>>>>>>> spin_lock(&trace_clock_lock); > >>>>>>>> WARN_ON(trace_clock_refcount<= 0); > >>>>>>>> if (trace_clock_refcount != 1) > >>>>>>>> goto end; > >>>>>>>> _stop_trace_clock(); > >>>>>>>> end: > >>>>>>>> trace_clock_refcount--; > >>>>>>>> spin_unlock(&trace_clock_lock); > >>>>>>>> } > >>>>>>>> EXPORT_SYMBOL_GPL(put_trace_clock); > >>>>>>>> > >>>>>>>> Could anybody figure out what's wrong with the kernel code? Any > >>>>>>>> suggestion would be appreciated. > >>>>>>>> > >>>>>>>> > >>>>>>>> Thanks, > >>>>>>>> Xufeng Zhang > >>>>>>>> > >>>>>>>> > >>>>>>>> _______________________________________________ > >>>>>>>> ltt-dev mailing list > >>>>>>>> ltt-dev at lists.casi.polymtl.ca > >>>>>>>> http://lists.casi.polymtl.ca/cgi-bin/mailman/listinfo/ltt-dev > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>> > >>>>>>> > >>>>>> _______________________________________________ > >>>>>> ltt-dev mailing list > >>>>>> ltt-dev at lists.casi.polymtl.ca > >>>>>> http://lists.casi.polymtl.ca/cgi-bin/mailman/listinfo/ltt-dev > >>>>>> > >>>>>> > >>>>>> > >>>>> > >>>>> > >>>> > >>> -- > >>> Mathieu Desnoyers > >>> Operating System Efficiency R&D Consultant > >>> EfficiOS Inc. > >>> http://www.efficios.com > >>> > >>> _______________________________________________ > >>> ltt-dev mailing list > >>> ltt-dev at lists.casi.polymtl.ca > >>> http://lists.casi.polymtl.ca/cgi-bin/mailman/listinfo/ltt-dev > >>> > >>> > >> > > > > -- > Mathieu Desnoyers > Operating System Efficiency R&D Consultant > EfficiOS Inc. > http://www.efficios.com > > _______________________________________________ > ltt-dev mailing list > ltt-dev at lists.casi.polymtl.ca > http://lists.casi.polymtl.ca/cgi-bin/mailman/listinfo/ltt-dev > -- Mathieu Desnoyers Operating System Efficiency R&D Consultant EfficiOS Inc. http://www.efficios.com