From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (qmail 16536 invoked by alias); 4 Aug 2011 22:39:56 -0000 Received: (qmail 16330 invoked by uid 22791); 4 Aug 2011 22:39:53 -0000 X-SWARE-Spam-Status: No, hits=-6.9 required=5.0 tests=AWL,BAYES_00,RCVD_IN_DNSWL_HI,RP_MATCHES_RCVD,SPF_HELO_PASS X-Spam-Check-By: sourceware.org Received: from mx1.redhat.com (HELO mx1.redhat.com) (209.132.183.28) by sourceware.org (qpsmtpd/0.43rc1) with ESMTP; Thu, 04 Aug 2011 22:39:38 +0000 Received: from int-mx10.intmail.prod.int.phx2.redhat.com (int-mx10.intmail.prod.int.phx2.redhat.com [10.5.11.23]) by mx1.redhat.com (8.14.4/8.14.4) with ESMTP id p74MdbjD024322 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK) for ; Thu, 4 Aug 2011 18:39:37 -0400 Received: from host1.jankratochvil.net (ovpn-116-26.ams2.redhat.com [10.36.116.26]) by int-mx10.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id p74MdXmc002497 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Thu, 4 Aug 2011 18:39:36 -0400 Received: from host1.jankratochvil.net (localhost [127.0.0.1]) by host1.jankratochvil.net (8.14.4/8.14.4) with ESMTP id p74MdQKj029848; Fri, 5 Aug 2011 00:39:26 +0200 Received: (from jkratoch@localhost) by host1.jankratochvil.net (8.14.4/8.14.4/Submit) id p74MdPlU029842; Fri, 5 Aug 2011 00:39:25 +0200 Date: Thu, 04 Aug 2011 22:39:00 -0000 From: Jan Kratochvil To: Tom Tromey Cc: Marc Khouzam , "'gdb@sourceware.org'" Subject: Re: How to properly check for regressions? Message-ID: <20110804223924.GA29692@host1.jankratochvil.net> References: MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="OXfL5xGRrasGEqWY" Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.21 (2010-09-15) X-IsSubscribed: yes Mailing-List: contact gdb-help@sourceware.org; run by ezmlm Precedence: bulk List-Id: List-Subscribe: List-Archive: List-Post: List-Help: , Sender: gdb-owner@sourceware.org X-SW-Source: 2011-08/txt/msg00019.txt.bz2 --OXfL5xGRrasGEqWY Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-length: 697 On Thu, 04 Aug 2011 22:51:10 +0200, Tom Tromey wrote: > Our buildbot automates all this -- but I suspect somewhat poorly, > because Jan is constantly finding regressions I didn't see. I just use `diff -u' on gdb.sum files. > Marc> And how do I deal with tests that sometimes pass and sometimes fail. > > The most common approach is to learn what they are and ignore them. > This isn't totally satisfactory; fixing them or removing them would be > nicer. I have a list of them for automated ignoring but I use the automated ignoring only in some cases, ignore list (+attaching its current state). http://git.jankratochvil.net/?p=nethome.git;a=blob_plain;f=bin/diffgdb;hb=master Thanks, Jan --OXfL5xGRrasGEqWY Content-Type: text/plain; charset=us-ascii Content-Disposition: inline; filename=diffgdb Content-length: 6810 #! /usr/bin/perl use strict; use warnings; my @ignore=( qr{^.PASS: }o, qr{^.KPASS: }o, qr{^-FAIL: }o, qr{^-KFAIL: }o, qr{^-XFAIL: }o, qr{^.Test Run By }o, qr{^.gnatbind }o, qr{^.gnatlink }o, qr{^.gnatlink: }o, qr{^.gnatmake: }o, qr{^.gcc }o, qr{^./usr/bin/ld: .* architecture of input file .* is incompatible with .* output\n$}o, qr{^.collect2: }o, #qr{^.UNSUPPORTED: }o, #qr{^.UNTESTED: }o, #qr{^.UNRESOLVED: }o, #qr{^.ERROR: }o, #qr{^.WARNING: }o, qr{^.[^:]*[.]ad[sb]:[0-9]}o, qr{^.(?:PASS|FAIL): gdb.server/ext-run.exp: get process list\n$}o, qr{^.(?:PASS|FAIL): gdb.base/checkpoint.exp: info checkpoints with at least 600 checkpoints\n$}o, qr{^.(?:PASS|FAIL): gdb.base/checkpoint.exp: break2 with many checkpoints\n$}o, qr{^.(?:PASS|FAIL): gdb.base/interrupt.exp: }o, qr{^.(?:PASS|FAIL): gdb.base/gdb1250.exp: }o, qr{^.KFAIL: gdb.threads/watchthreads2.exp: gdb can drop watchpoints in multithreaded app \(PRMS: gdb/10116\)\n$}o, qr{^.PASS: gdb.threads/watchthreads2.exp: all threads incremented x\n$}o, qr{^.FAIL: gdb.base/sigbpt.exp: Verify that SIGSEGV occurs at the last STEPI insn \(none 0x[a-f0-9]*\)\n$}o, qr{^.PASS: gdb.threads/threadcrash.exp: core file: }o, qr{^.PASS: gdb.threads/watchthreads.exp: disable [0-9]*\n$}o, qr{^.FAIL: gdb.mi/mi-nsmoribund.exp: unexpected stop\n$}o, qr{^.PASS: gdb.mi/mi-nsmoribund.exp: resume all, program exited normally\n$}o, qr{^.gdb compile failed, }o, qr{^.gdb compile failed, /usr/bin/ld: /tmp/[a-zA-Z0-9]*.o: relocation R_X86_64_32S against `a local symbol. can not be used when making a shared object; recompile with -fPIC\n$}o, qr{^./tmp/[a-zA-Z0-9]*.o: could not read symbols: Bad value\n$}o, qr{^./tmp/[a-zA-Z0-9]*.s:\d+: Error: unrecognized symbol type "gnu_indirect_function"\n$}o, qr{^./tmp/[a-zA-Z0-9]*.o: In function `\S+':\n$}o, qr{^./tmp/[a-zA-Z0-9]*.o:.*: undefined reference to `.*'\n$}o, qr{^./tmp/[a-zA-Z0-9]*.s:\d+: Error: syntax error; found `.' but expected `,'\n$}o, qr{^./tmp/[a-zA-Z0-9]*.s:\d+: Error: junk at end of line: `.(?:plt|local)'\n$}o, qr{^./usr/bin/ld: /tmp/[a-zA-Z0-9]*.o\(\.text\+0x[0-9a-f]+\): unresolvable R_PPC64_REL24 relocation against symbol `\S+'\n$}o, qr{^.# of }o, qr{^./home/}o, qr{^.PASS: gdb.arch/i386-biarch-core.exp: core-file /home/.*/i386-biarch-core.core\n$}o, qr{^.\.\./as-new [0-9.]*\n$}o, qr{^.ld/ld-new [0-9.]*\n$}o, qr{^.(?:PASS|FAIL): gdb.mi/mi-break.exp: test hitting breakpoint with commands\n$}o, qr{^.(?:PASS|FAIL): gdb.threads/attach-stopped.exp: threaded: attach2, exit leaves process sleeping\n$}o, qr{^.(?:PASS|FAIL): gdb.threads/attachstop-mt.exp: attach3, exit leaves process stopped\n$}o, qr{^.(?:PASS|FAIL): gdb.threads/attachstop-mt.exp: attach4 stop by interrupt(?: \(timeout\))?\n$}o, qr{^.(?:PASS|FAIL): gdb.threads/attachstop-mt.exp: attach4, exit leaves process sleeping\n$}o, qr{^.(?:PASS|FAIL): gdb.threads/manythreads.exp: info threads\n$}o, qr{gdb version \d\S+ -nw -nx \n$}o, qr{^.(?:PASS|FAIL): gdb.threads/attach-stopped.exp: threaded: attach1, exit leaves process stopped\n$}o, qr{^.(?:PASS|FAIL): gdb.threads/local-watch-wrong-thread.exp: local watchpoint automatically deleted(?: \(timeout\))?\n$}o, qr{^.(?:PASS|FAIL): gdb.server/server-run.exp: continue to main\n$}o, qr{^.(?:PASS|FAIL): gdb.mi/mi-nonstop.exp: w1,i0 stop(?: \(timeout\))?\n$}o, qr{^.(?:PASS|FAIL): gdb.reverse/i387-env-reverse.exp: set reverse direction(?: \(timeout\))?\n$}o, qr{^.(?:PASS|FAIL): gdb.reverse/i387-env-reverse.exp: record to end of main(?: \(timeout\))?\n$}o, qr{^.(?:PASS|FAIL): gdb.mi/mi-file-transfer.exp: compare intermediate binary file\n$}o, qr{^.(?:PASS|FAIL): gdb.mi/mi-file-transfer.exp: compare binary file\n$}o, qr{^.(?:PASS|FAIL): gdb.mi/mi-file-transfer.exp: deleted binary file\n$}o, qr{^.(?:PASS|FAIL): gdb.mi/mi-simplerun.exp: continue to end(?: \(failed to resume\))?\n$}o, qr{^.(?:PASS|FAIL): gdb.mi/mi-simplerun.exp: continue to end(?: \(unknown output after running\))?\n$}o, qr{^.(?:PASS|FAIL): gdb.server/file-transfer.exp: verified deleted binary file\n$}o, qr{^.(?:PASS|FAIL): gdb.server/file-transfer.exp: deleted binary file\n$}o, qr{^.(?:PASS|FAIL): gdb.threads/attachstop-mt.exp: attach1, post-gdb sanity check of the sleeping state - Red Hat BZ 197584\n$}o, qr{^."uplevel #0 source .*gdb/testsuite/gdb.threads/watchthreads-thr(?:eaded\.exp|.*\.\.\.)"\n$}o, qr{^.(?:PASS|FAIL): gdb.cp/ref-types.exp: print value of rL\n$}o, qr{^.FAIL: gdb.threads/watchthreads-reorder.exp: reorder[01]: (?:continue [ab]|rwatch unused1_rwatch)(?: \((?:timeout|the program exited|the program is no longer running)\))?\n$}o, qr{^.FAIL: gdb.base/sigbpt.exp: Verify that SIGSEGV occurs at the last STEPI insn \(none 0x[0-9a-f]+\)\n$}o, qr{^.(?:KFAIL|FAIL): gdb.base/sigbpt.exp: stepi bp at segv; stepi out of handler(?: \(corrupt pc\) \(PRMS: gdb/1702\))?\n$}o, qr{^.(?:KFAIL|FAIL): gdb.base/sigbpt.exp: stepi bp before and at segv; stepi out of handler(?: \(corrupt pc\) \(PRMS: gdb/1702\))?\n$}o, qr{^.FAIL: \S+\.exp: valgrind check \d+\n$}o, ); my $ignore=shift @ignore; $ignore=qr/$ignore|$_/ for @ignore; my $opt_new; while (1) { if ($ARGV[0] eq "-r") { $ignore=undef; shift; next; } if ($ARGV[0] eq "-N") { $opt_new=1; shift; next; } last; } my $last; my $lastfile; local *DIFF; my(@diff)=("diff","-rU-1",@ARGV); open DIFF,"-|",@diff or die join(" ",@diff).": $!"; while () { next if /^diff /; my($thisname)=m{^.(?:Running (?:.*/)?|[A-Z]+: )(gdb[.]\w+/\S+[.]exp)(?: \Q...\E|: .*)$}; $thisname||=(m{^.\t\t=== gdb (Summary) ===\n$})[0]; $thisname||=(m{^(Only in) })[0]; my $this; if (!$thisname || ($last && $thisname eq $last->{"name"})) { #if (!$thisname) { print "X: !thisname\n"; } else { print "X: \"".$last->{"name"}."\" -> \"$thisname\"\n"; } $this=$last; } else { #print "X: new \"$thisname\"\n"; $last=$this={"name"=>$thisname,"+Running"=>(/^[+]/||0)}; } next if /^@@ .* @@\n$/; if ($this->{"print"}) { print; next; } if (/^Only in /) { $lastfile=$_; } if (/^--- /) { $lastfile=$_; next; } if (/^\Q+++\E /) { $lastfile.=$_; next; } $this->{"buffer"}.=$_; next if /^ /; next if /^[+]Running / && $thisname && !$opt_new; next if $ignore && /$ignore/o; next if /^[+]FAIL: / && $this->{"+Running"}; next if /^[+]KFAIL: / && $this->{"+Running"}; next if /^[+]XFAIL: / && $this->{"+Running"}; next if /^[+]UNTESTED: / && $this->{"+Running"}; next if /^[+]gdb compile failed, / && $this->{"+Running"}; next if /^[+].*: Error: bad register name `%.*'\n$/ && $this->{"+Running"}; next if /^[+].*: Error: `.*' is only supported in 64-bit mode\n$/ && $this->{"+Running"}; $this->{"print"}=1; print $lastfile||""; $lastfile=undef(); print "\n#".($this->{"name"}||"???")."\n"; print $this->{"buffer"}; $this->{"buffer"}=undef(); next; } --OXfL5xGRrasGEqWY--