From mboxrd@z Thu Jan 1 00:00:00 1970 From: benjamin.poirier@polymtl.ca (Benjamin Poirier) Date: Tue, 09 Feb 2010 13:16:22 -0500 Subject: [ltt-dev] net extended trace openin error with LTTV (without attachment) In-Reply-To: <20100209175945.GA20711@Krystal> References: <20100208153634.GA9978@Krystal> <4B706EEC.30207@polymtl.ca> <20100208203209.GA3634@Krystal> <4B708283.7060105@polymtl.ca> <20100209170345.GA15485@Krystal> <20100209175945.GA20711@Krystal> Message-ID: <4B71A676.3020101@polymtl.ca> Mathieu Desnoyers wrote: > * Mathieu Desnoyers (compudj at krystal.dyndns.org) wrote: > >> * Benjamin Poirier (benjamin.poirier at polymtl.ca) wrote: >> >>> Mathieu Desnoyers wrote: >>> >>>> * Benjamin Poirier (benjamin.poirier at polymtl.ca) wrote: >>>> >>>> >>>>> paljak at mit.bme.hu wrote: >>>>> >>>>> >>>>>> -----Mathieu Desnoyers wrote: ----- >>>>>> >>>>>> To: paljak at mit.bme.hu >>>>>> From: Mathieu Desnoyers >>>>>> Date: 2010/02/08 16:36 >>>>>> cc: benjamin.poirier at polymtl.ca, ltt-dev at lists.casi.polymtl.ca >>>>>> Subject: Re: [ltt-dev] net extended trace openin error with LTTV >>>>>> (without attachment) >>>>>> >>>>>> * paljak at mit.bme.hu (paljak at mit.bme.hu) wrote: >>>>>> > Hi Ben, >>>>>> > >>>>>> > I have been working with LTTng lately and the more I know it, the >>>>>> more I >>>>>> > like it! >>>>>> > >>>>>> > I've also been able to open and examine traces, and it looks very >>>>>> > interesting. >>>>>> > >>>>>> > I've also tried to create my own, but I've ran into an error, >>>>>> could you >>>>>> > take a look? >>>>>> > >>>>>> > Basically, I receive an error >>>>>> > (ERROR:tracefile.c:1582:ltt_update_event_size: assertion failed: >>>>>> (info != >>>>>> > NULL)) every time i try to open a net_extended trace with LTTV. >>>>>> > >>>>>> > Here's what i've done: >>>>>> > >>>>>> > 1. Downloaded the patched kernel: git clone >>>>>> > >>>>>> git://git.kernel.org/pub/scm/linux/kernel/git/compudj/linux-2.6-lttng.git >>>>>> > 2. Compiled the kernel with these options (.config): >>>>>> > # >>>>>> > # GCOV-based kernel profiling >>>>>> > # >>>>>> > # CONFIG_GCOV_KERNEL is not set >>>>>> > CONFIG_SLOW_WORK=y >>>>>> > # CONFIG_SLOW_WORK_DEBUG is not set >>>>>> > CONFIG_LTT=y >>>>>> > CONFIG_LTT_FILTER=m >>>>>> > CONFIG_HAVE_LTT_DUMP_TABLES=y >>>>>> > CONFIG_LTT_RELAY=y >>>>>> > CONFIG_LTT_RELAY_LOCKLESS=y >>>>>> > # CONFIG_LTT_RELAY_IRQOFF is not set >>>>>> > # CONFIG_LTT_RELAY_LOCKED is not set >>>>>> > CONFIG_LTT_SERIALIZE=y >>>>>> > CONFIG_LTT_FAST_SERIALIZE=y >>>>>> > CONFIG_LTT_TRACEPROBES=m >>>>>> > CONFIG_LTT_TRACE_CONTROL=y >>>>>> > CONFIG_LTT_TRACER=y >>>>>> > CONFIG_LTT_ALIGNMENT=y >>>>>> > CONFIG_LTT_CHECK_ARCH_EFFICIENT_UNALIGNED_ACCESS=y >>>>>> > # CONFIG_LTT_DEBUG_EVENT_SIZE is not set >>>>>> > CONFIG_LTT_USERSPACE_EVENT=m >>>>>> > CONFIG_LTT_VMCORE=y >>>>>> > CONFIG_LTT_STATEDUMP=m >>>>>> > CONFIG_LTT_KPROBES=m >>>>>> > # CONFIG_LTT_ASCII is not set >>>>>> > 3. Install the kernel and boot it up >>>>>> > 4. Load all the modules >>>>>> > modprobe -v ltt-trace-control >>>>>> > modprobe -v ltt-marker-control >>>>>> > modprobe -v ltt-tracer >>>>>> > modprobe -v ltt-serialize >>>>>> > modprobe -v ltt-relay >>>>>> > modprobe -v ipc-trace >>>>>> > modprobe -v kernel-trace >>>>>> > modprobe -v mm-trace >>>>>> > modprobe -v net-trace >>>>>> > modprobe -v fs-trace >>>>>> > modprobe -v jbd2-trace >>>>>> > modprobe -v ext4-trace >>>>>> > modprobe -v syscall-trace >>>>>> > modprobe -v trap-trace >>>>>> > modprobe -v ltt-statedump >>>>>> > #extended trace >>>>>> > modprobe -v net-extended-trace >>>>>> > 5. ltt-armall -n >>>>>> > 6. lttctl -C -w /tmp/tr1 tr1 >>>>>> > 7. wget http://google.com >>>>>> > 8. lttctl -D tr1 >>>>>> > 9. lttv -m textDump -t tr1 >>>>>> > ** >>>>>> > ERROR:tracefile.c:1582:ltt_update_event_size: assertion failed: >>>>>> (info != >>>>>> > NULL) >>>>>> > /usr/local/bin/lttv: line 15: 2083 Aborted $0.real $* >>>>>> > 10. lttv -m textDump -t tr1 --verbose --debug --edebug >>>>>> > net.udpv4_rcv_extended: 1003.743082325 (/tmp/tr1/net_0), 1878, 1878, >>>>>> > gnome-terminal, , 1, 0x0, SOFTIRQ { skb = 0xC4BBC240, saddr = >>>>>> 3576624693, >>>>>> > daddr = 3232235780, unicast = 1, ulen = 110, source = 53, dest = >>>>>> 39087, >>>>>> > data_start = a299884901000100 } >>>>>> > Event header (tracefile /tmp/tr1/net_0 offset bb8): >>>>>> > bb8 40 C2 BB C4 >>>>>> > ** >>>>>> > ERROR:tracefile.c:1582:ltt_update_event_size: assertion failed: >>>>>> (info != >>>>>> > NULL) >>>>>> > /usr/local/bin/lttv: line 15: 2104 Aborted $0.real $* >>>>>> > >>>>>> > The installed versions are: >>>>>> > Linux Trace Toolkit Visualizer 0.12.29-02022010 >>>>>> > Linux Trace Toolkit Trace Control 0.79-01022010 >>>>>> > >>>>>> > Here's the created trace for reference: >>>>>> > http://www.mit.bme.hu/~paljak/trace.tgz >>>>>> >>>>>> > >>>>>> > If I don't use net extended, than it works just fine. And I've >>>>>> also had >>>>>> > success open your example trace. >>>>>> > >>>>>> > Do you have any idea why do I get this error and how could I >>>>>> correct it? >>>>>> >>>>>> That looks like a field type error in the net extended event. Can you >>>>>> tell us what LTTng version you use ? (git branch should tell you) >>>>>> >>>>>> Is your traced machine 32 or 64 bits ? And what about the machine >>>>>> running lttv ? >>>>>> >>>>>> Reviewing ltt/probes/net-extended-trace.c might help. >>>>>> >>>>>> Thanks, >>>>>> >>>>>> Mathieu >>>>>> >>>>>> I use lttng-0.188. And I run lttng and lttv on the same, 32 bit machine. >>>>>> >>>>>> I'm checking net-extended-trace.c but reason I could think of to cause >>>>>> this error... >>>>>> >>>>>> >>>>>> >>>>> I can't reproduce the problem, but looking at the traces, I notice the alignment is different: >>>>> >>>>> your trace: >>>>> metadata.core_marker_id: 994.751622140 (/tmp/trace/tr1/metadata_0), 0, 0, , , 0, 0x0, MODE_UNKNOWN { channel = "net", name = "udpv4_rcv_extended", event_id = 2, int = 4, long = 4, pointer = 4, size_t = 4, alignment = 4 } >>>>> >>>>> my trace: >>>>> metadata.core_marker_id: 193.282545152 (/tmp/tr1/metadata_0), 0, 0, , , 0, 0x0, MODE_UNKNOWN { channel = "net", name = "udpv4_rcv_extended", event_id = 2, int = 4, long = 4, pointer = 4, size_t = 4, alignment = 0 } >>>>> >>>>> Could you apply the following patch over the git kernel, build and test? Thanks. >>>>> >>>>> >>>> Seem like my LTTng patch: >>>> >>>> lttng-trace-format-allow-large-alignment.patch >>>> "lttng trace format allow large alignment >>>> >>>> Do not limit alignment on architecture size anymore, because uint64_t >>>> types are >>>> aligned on 64-bit even on 32-bit archs. It's a waste of space for 32-bit >>>> architectures, but makes our format compatible with gcc." >>>> >>>> does not do everything it needs to do. Or maybe LTTV still do an >>>> alignment on min(arch size, data size) somewhere. >>>> >>>> A way to "just amke it work" is to disable alignment in LTTng. But it >>>> would be good to find the culprit. >>>> >>>> >>> Indeed, I have CONFIG_LTT_ALIGNMENT=n in my kernel, lucky me! If I >>> activate it I can reproduce the problem and the patch I suggested >>> doesn't change it, so don't bother. Disable alignment in the meantime as >>> suggested by Mathieu. >>> >>> >> I found the culprit: >> >> include/linux/ltt-type-serializer.h: >> >> /* >> * Statically check that 0 < largest_align < sizeof(void *) to make sure it is >> * dumb-proof. It will make sure 0 is changed into 1 and unsigned long long is >> * changed into sizeof(void *) on 32-bit architectures. >> */ >> static inline void ltt_specialized_trace(const struct marker *mdata, >> void *probe_data, >> void *serialize_private, unsigned int data_size, >> unsigned int largest_align) >> { >> largest_align = min_t(unsigned int, largest_align, sizeof(void *)); >> largest_align = max_t(unsigned int, largest_align, 1); >> _ltt_specialized_trace(mdata, probe_data, serialize_private, data_size, >> largest_align); >> } >> >> It caps the alignment to the architecture size. I'll remove the min_t() >> line and it should be allright for the next release. Can you test it >> meanwhile ? >> > > LTTng 0.190 should fix this alignment problem. Feedback would be welcome. > > Thanks, > Nope, same problem I believe: metadata.core_marker_id: 40.870000627 (/tmp/trace3/metadata_0), 0, 0, , , 0, 0x0, MODE_UNKNOWN { channel = "net", name = "udpv4_rcv_extended", event_id = 2, int = 4, long = 4, pointer = 4, size_t = 4, alignment = 4 } metadata.core_marker_format: 40.870002196 (/tmp/trace3/metadata_0), 0, 0, , , 0, 0x0, MODE_UNKNOWN { channel = "net", name = "udpv4_rcv_extended", format = "skb 0x%lX saddr #n4u%lu daddr #n4u%lu unicast #1u%u ulen #n2u%hu source #n2u%hu dest #n2u%hu data_start #8u%lx" } [...] net.udpv4_rcv_extended: 44.175529809 (/tmp/trace3/net_0), 0, 0, swapper, , 0, 0x0, SOFTIRQ { skb = 0xDA225A80, saddr = 2228195330, daddr = 2228200116, unicast = 1, ulen = 188, source = 53, dest = 45569, data_start = a2068db001000100 } ** ERROR:tracefile.c:1582:ltt_update_event_size: assertion failed: (info != NULL) > Mathieu > > >> Thanks, >> >> Mathieu >> >> >> >>> -Ben >>> >>>> Thanks, >>>> >>>> Mathieu >>>> >>>> >>>> >>>> >>>>> diff --git a/ltt/probes/net-extended-trace.c b/ltt/probes/net-extended-trace.c >>>>> index 074bfec..88410f8 100644 >>>>> --- a/ltt/probes/net-extended-trace.c >>>>> +++ b/ltt/probes/net-extended-trace.c >>>>> @@ -130,14 +130,14 @@ notrace void probe_udpv4_rcv_extended(struct sk_buff *skb) >>>>> * bytes of UDP data if they are not in a fragment*/ >>>>> data.f8 = 0; >>>>> if (skb_headlen(skb) >= sizeof(struct udphdr) + 8) >>>>> - data.f8 = *(unsigned long long *)(skb->data + sizeof(*uh)); >>>>> + data.f8 = *(uint64_t *)(skb->data + sizeof(*uh)); >>>>> else if (skb_headlen(skb) >= sizeof(struct udphdr)) >>>>> memcpy(&data.f8, skb->data + sizeof(struct udphdr), >>>>> skb_headlen(skb) - sizeof(struct udphdr)); >>>>> >>>>> marker = &GET_MARKER(net, udpv4_rcv_extended); >>>>> ltt_specialized_trace(marker, marker->single.probe_private, >>>>> - &data, serialize_sizeof(data), sizeof(unsigned long long)); >>>>> + &data, serialize_sizeof(data), sizeof(uint64_t)); >>>>> } >>>>> >>>>> MODULE_LICENSE("GPL and additional rights"); >>>>> >>>>> >>>>> >>>>>> Thanks, >>>>>> Gergely >>>>>> >>>>>> >>>>>> ------------------------------------------------------------------------ >>>>>> >>>>>> _______________________________________________ >>>>>> ltt-dev mailing list >>>>>> ltt-dev at lists.casi.polymtl.ca >>>>>> http://lists.casi.polymtl.ca/cgi-bin/mailman/listinfo/ltt-dev >>>>>> >>>>>> >>>>> _______________________________________________ >>>>> ltt-dev mailing list >>>>> ltt-dev at lists.casi.polymtl.ca >>>>> http://lists.casi.polymtl.ca/cgi-bin/mailman/listinfo/ltt-dev >>>>> >>>>> >>>>> >>>> >>>> >> -- >> Mathieu Desnoyers >> OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F BA06 3F25 A8FE 3BAE 9A68 >> >> _______________________________________________ >> ltt-dev mailing list >> ltt-dev at lists.casi.polymtl.ca >> http://lists.casi.polymtl.ca/cgi-bin/mailman/listinfo/ltt-dev >> >> > >