From mboxrd@z Thu Jan 1 00:00:00 1970 From: compudj@krystal.dyndns.org (Mathieu Desnoyers) Date: Tue, 9 Feb 2010 12:59:45 -0500 Subject: [ltt-dev] net extended trace openin error with LTTV (without attachment) In-Reply-To: <20100209170345.GA15485@Krystal> References: <20100208153634.GA9978@Krystal> <4B706EEC.30207@polymtl.ca> <20100208203209.GA3634@Krystal> <4B708283.7060105@polymtl.ca> <20100209170345.GA15485@Krystal> Message-ID: <20100209175945.GA20711@Krystal> * Mathieu Desnoyers (compudj at krystal.dyndns.org) wrote: > * Benjamin Poirier (benjamin.poirier at polymtl.ca) wrote: > > Mathieu Desnoyers wrote: > > > * Benjamin Poirier (benjamin.poirier at polymtl.ca) wrote: > > > > > >> paljak at mit.bme.hu wrote: > > >> > > >>> -----Mathieu Desnoyers wrote: ----- > > >>> > > >>> To: paljak at mit.bme.hu > > >>> From: Mathieu Desnoyers > > >>> Date: 2010/02/08 16:36 > > >>> cc: benjamin.poirier at polymtl.ca, ltt-dev at lists.casi.polymtl.ca > > >>> Subject: Re: [ltt-dev] net extended trace openin error with LTTV > > >>> (without attachment) > > >>> > > >>> * paljak at mit.bme.hu (paljak at mit.bme.hu) wrote: > > >>> > Hi Ben, > > >>> > > > >>> > I have been working with LTTng lately and the more I know it, the > > >>> more I > > >>> > like it! > > >>> > > > >>> > I've also been able to open and examine traces, and it looks very > > >>> > interesting. > > >>> > > > >>> > I've also tried to create my own, but I've ran into an error, > > >>> could you > > >>> > take a look? > > >>> > > > >>> > Basically, I receive an error > > >>> > (ERROR:tracefile.c:1582:ltt_update_event_size: assertion failed: > > >>> (info != > > >>> > NULL)) every time i try to open a net_extended trace with LTTV. > > >>> > > > >>> > Here's what i've done: > > >>> > > > >>> > 1. Downloaded the patched kernel: git clone > > >>> > > > >>> git://git.kernel.org/pub/scm/linux/kernel/git/compudj/linux-2.6-lttng.git > > >>> > 2. Compiled the kernel with these options (.config): > > >>> > # > > >>> > # GCOV-based kernel profiling > > >>> > # > > >>> > # CONFIG_GCOV_KERNEL is not set > > >>> > CONFIG_SLOW_WORK=y > > >>> > # CONFIG_SLOW_WORK_DEBUG is not set > > >>> > CONFIG_LTT=y > > >>> > CONFIG_LTT_FILTER=m > > >>> > CONFIG_HAVE_LTT_DUMP_TABLES=y > > >>> > CONFIG_LTT_RELAY=y > > >>> > CONFIG_LTT_RELAY_LOCKLESS=y > > >>> > # CONFIG_LTT_RELAY_IRQOFF is not set > > >>> > # CONFIG_LTT_RELAY_LOCKED is not set > > >>> > CONFIG_LTT_SERIALIZE=y > > >>> > CONFIG_LTT_FAST_SERIALIZE=y > > >>> > CONFIG_LTT_TRACEPROBES=m > > >>> > CONFIG_LTT_TRACE_CONTROL=y > > >>> > CONFIG_LTT_TRACER=y > > >>> > CONFIG_LTT_ALIGNMENT=y > > >>> > CONFIG_LTT_CHECK_ARCH_EFFICIENT_UNALIGNED_ACCESS=y > > >>> > # CONFIG_LTT_DEBUG_EVENT_SIZE is not set > > >>> > CONFIG_LTT_USERSPACE_EVENT=m > > >>> > CONFIG_LTT_VMCORE=y > > >>> > CONFIG_LTT_STATEDUMP=m > > >>> > CONFIG_LTT_KPROBES=m > > >>> > # CONFIG_LTT_ASCII is not set > > >>> > 3. Install the kernel and boot it up > > >>> > 4. Load all the modules > > >>> > modprobe -v ltt-trace-control > > >>> > modprobe -v ltt-marker-control > > >>> > modprobe -v ltt-tracer > > >>> > modprobe -v ltt-serialize > > >>> > modprobe -v ltt-relay > > >>> > modprobe -v ipc-trace > > >>> > modprobe -v kernel-trace > > >>> > modprobe -v mm-trace > > >>> > modprobe -v net-trace > > >>> > modprobe -v fs-trace > > >>> > modprobe -v jbd2-trace > > >>> > modprobe -v ext4-trace > > >>> > modprobe -v syscall-trace > > >>> > modprobe -v trap-trace > > >>> > modprobe -v ltt-statedump > > >>> > #extended trace > > >>> > modprobe -v net-extended-trace > > >>> > 5. ltt-armall -n > > >>> > 6. lttctl -C -w /tmp/tr1 tr1 > > >>> > 7. wget http://google.com > > >>> > 8. lttctl -D tr1 > > >>> > 9. lttv -m textDump -t tr1 > > >>> > ** > > >>> > ERROR:tracefile.c:1582:ltt_update_event_size: assertion failed: > > >>> (info != > > >>> > NULL) > > >>> > /usr/local/bin/lttv: line 15: 2083 Aborted $0.real $* > > >>> > 10. lttv -m textDump -t tr1 --verbose --debug --edebug > > >>> > net.udpv4_rcv_extended: 1003.743082325 (/tmp/tr1/net_0), 1878, 1878, > > >>> > gnome-terminal, , 1, 0x0, SOFTIRQ { skb = 0xC4BBC240, saddr = > > >>> 3576624693, > > >>> > daddr = 3232235780, unicast = 1, ulen = 110, source = 53, dest = > > >>> 39087, > > >>> > data_start = a299884901000100 } > > >>> > Event header (tracefile /tmp/tr1/net_0 offset bb8): > > >>> > bb8 40 C2 BB C4 > > >>> > ** > > >>> > ERROR:tracefile.c:1582:ltt_update_event_size: assertion failed: > > >>> (info != > > >>> > NULL) > > >>> > /usr/local/bin/lttv: line 15: 2104 Aborted $0.real $* > > >>> > > > >>> > The installed versions are: > > >>> > Linux Trace Toolkit Visualizer 0.12.29-02022010 > > >>> > Linux Trace Toolkit Trace Control 0.79-01022010 > > >>> > > > >>> > Here's the created trace for reference: > > >>> > http://www.mit.bme.hu/~paljak/trace.tgz > > >>> > > >>> > > > >>> > If I don't use net extended, than it works just fine. And I've > > >>> also had > > >>> > success open your example trace. > > >>> > > > >>> > Do you have any idea why do I get this error and how could I > > >>> correct it? > > >>> > > >>> That looks like a field type error in the net extended event. Can you > > >>> tell us what LTTng version you use ? (git branch should tell you) > > >>> > > >>> Is your traced machine 32 or 64 bits ? And what about the machine > > >>> running lttv ? > > >>> > > >>> Reviewing ltt/probes/net-extended-trace.c might help. > > >>> > > >>> Thanks, > > >>> > > >>> Mathieu > > >>> > > >>> I use lttng-0.188. And I run lttng and lttv on the same, 32 bit machine. > > >>> > > >>> I'm checking net-extended-trace.c but reason I could think of to cause > > >>> this error... > > >>> > > >>> > > >> I can't reproduce the problem, but looking at the traces, I notice the alignment is different: > > >> > > >> your trace: > > >> metadata.core_marker_id: 994.751622140 (/tmp/trace/tr1/metadata_0), 0, 0, , , 0, 0x0, MODE_UNKNOWN { channel = "net", name = "udpv4_rcv_extended", event_id = 2, int = 4, long = 4, pointer = 4, size_t = 4, alignment = 4 } > > >> > > >> my trace: > > >> metadata.core_marker_id: 193.282545152 (/tmp/tr1/metadata_0), 0, 0, , , 0, 0x0, MODE_UNKNOWN { channel = "net", name = "udpv4_rcv_extended", event_id = 2, int = 4, long = 4, pointer = 4, size_t = 4, alignment = 0 } > > >> > > >> Could you apply the following patch over the git kernel, build and test? Thanks. > > >> > > > > > > Seem like my LTTng patch: > > > > > > lttng-trace-format-allow-large-alignment.patch > > > "lttng trace format allow large alignment > > > > > > Do not limit alignment on architecture size anymore, because uint64_t > > > types are > > > aligned on 64-bit even on 32-bit archs. It's a waste of space for 32-bit > > > architectures, but makes our format compatible with gcc." > > > > > > does not do everything it needs to do. Or maybe LTTV still do an > > > alignment on min(arch size, data size) somewhere. > > > > > > A way to "just amke it work" is to disable alignment in LTTng. But it > > > would be good to find the culprit. > > > > > Indeed, I have CONFIG_LTT_ALIGNMENT=n in my kernel, lucky me! If I > > activate it I can reproduce the problem and the patch I suggested > > doesn't change it, so don't bother. Disable alignment in the meantime as > > suggested by Mathieu. > > > > I found the culprit: > > include/linux/ltt-type-serializer.h: > > /* > * Statically check that 0 < largest_align < sizeof(void *) to make sure it is > * dumb-proof. It will make sure 0 is changed into 1 and unsigned long long is > * changed into sizeof(void *) on 32-bit architectures. > */ > static inline void ltt_specialized_trace(const struct marker *mdata, > void *probe_data, > void *serialize_private, unsigned int data_size, > unsigned int largest_align) > { > largest_align = min_t(unsigned int, largest_align, sizeof(void *)); > largest_align = max_t(unsigned int, largest_align, 1); > _ltt_specialized_trace(mdata, probe_data, serialize_private, data_size, > largest_align); > } > > It caps the alignment to the architecture size. I'll remove the min_t() > line and it should be allright for the next release. Can you test it > meanwhile ? LTTng 0.190 should fix this alignment problem. Feedback would be welcome. Thanks, Mathieu > > Thanks, > > Mathieu > > > > -Ben > > > Thanks, > > > > > > Mathieu > > > > > > > > > > > >> diff --git a/ltt/probes/net-extended-trace.c b/ltt/probes/net-extended-trace.c > > >> index 074bfec..88410f8 100644 > > >> --- a/ltt/probes/net-extended-trace.c > > >> +++ b/ltt/probes/net-extended-trace.c > > >> @@ -130,14 +130,14 @@ notrace void probe_udpv4_rcv_extended(struct sk_buff *skb) > > >> * bytes of UDP data if they are not in a fragment*/ > > >> data.f8 = 0; > > >> if (skb_headlen(skb) >= sizeof(struct udphdr) + 8) > > >> - data.f8 = *(unsigned long long *)(skb->data + sizeof(*uh)); > > >> + data.f8 = *(uint64_t *)(skb->data + sizeof(*uh)); > > >> else if (skb_headlen(skb) >= sizeof(struct udphdr)) > > >> memcpy(&data.f8, skb->data + sizeof(struct udphdr), > > >> skb_headlen(skb) - sizeof(struct udphdr)); > > >> > > >> marker = &GET_MARKER(net, udpv4_rcv_extended); > > >> ltt_specialized_trace(marker, marker->single.probe_private, > > >> - &data, serialize_sizeof(data), sizeof(unsigned long long)); > > >> + &data, serialize_sizeof(data), sizeof(uint64_t)); > > >> } > > >> > > >> MODULE_LICENSE("GPL and additional rights"); > > >> > > >> > > >>> Thanks, > > >>> Gergely > > >>> > > >>> > > >>> ------------------------------------------------------------------------ > > >>> > > >>> _______________________________________________ > > >>> ltt-dev mailing list > > >>> ltt-dev at lists.casi.polymtl.ca > > >>> http://lists.casi.polymtl.ca/cgi-bin/mailman/listinfo/ltt-dev > > >>> > > >> _______________________________________________ > > >> ltt-dev mailing list > > >> ltt-dev at lists.casi.polymtl.ca > > >> http://lists.casi.polymtl.ca/cgi-bin/mailman/listinfo/ltt-dev > > >> > > >> > > > > > > > > > > -- > Mathieu Desnoyers > OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F BA06 3F25 A8FE 3BAE 9A68 > > _______________________________________________ > ltt-dev mailing list > ltt-dev at lists.casi.polymtl.ca > http://lists.casi.polymtl.ca/cgi-bin/mailman/listinfo/ltt-dev > -- Mathieu Desnoyers OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F BA06 3F25 A8FE 3BAE 9A68