How to properly check for regressions?

How to properly check for regressions?

[Marc Khouzam]

Hi,

I just realized that I might have been checking for regressions wrong!

The doc says 
(http://sourceware.org/gdb/current/onlinedocs/gdbint/Testsuite.html#Testsuite)

"When the testsuite is finished, you'll get a summary that looks like this:"

So, I've always looked at that summary, at the end of the console printouts,
and made sure it was the same before and after my patches. (note that
with -j4, that last summary showed thousands of tests)

But now I found the testsuite/gdb.sum file which shows a larger
set of tests in the summary that what is shown in the console.
Doing a diff of that file before and after my patch showed failures
I hadn't seen from the console summary.

What is the right way to make sure my patch does not cause regressions?

And how do I deal with tests that sometimes pass and sometimes fail.
The gdb.sum file does not seem to be constant, event without code changes.

Thanks

Marc

Re: How to properly check for regressions?

[Tom Tromey]

>>>>> "Marc" == Marc Khouzam <marc.khouzam@ericsson.com> writes:

Marc> What is the right way to make sure my patch does not cause regressions?

What I do is have a script that extracts the relevant lines from gdb.sum
and normalizes them; then I compare the normalized files.

Our buildbot automates all this -- but I suspect somewhat poorly,
because Jan is constantly finding regressions I didn't see.
Probably we should all use his script.  I think he's sent it to one of
the lists before; or maybe it is on the wiki.

Marc> And how do I deal with tests that sometimes pass and sometimes fail.

The most common approach is to learn what they are and ignore them.
This isn't totally satisfactory; fixing them or removing them would be
nicer.

Tom

Re: How to properly check for regressions?

[Jan Kratochvil]

[-- Attachment #1: Type: text/plain, Size: 697 bytes --]

On Thu, 04 Aug 2011 22:51:10 +0200, Tom Tromey wrote:
> Our buildbot automates all this -- but I suspect somewhat poorly,
> because Jan is constantly finding regressions I didn't see.

I just use `diff -u' on gdb.sum files.


> Marc> And how do I deal with tests that sometimes pass and sometimes fail.
> 
> The most common approach is to learn what they are and ignore them.
> This isn't totally satisfactory; fixing them or removing them would be
> nicer.

I have a list of them for automated ignoring but I use the automated ignoring
only in some cases, ignore list (+attaching its current state).
	http://git.jankratochvil.net/?p=nethome.git;a=blob_plain;f=bin/diffgdb;hb=master


Thanks,
Jan

[-- Attachment #2: diffgdb --]
[-- Type: text/plain, Size: 6810 bytes --]

#! /usr/bin/perl
use strict;
use warnings;

my @ignore=(
qr{^.PASS: }o,
qr{^.KPASS: }o,
qr{^-FAIL: }o,
qr{^-KFAIL: }o,
qr{^-XFAIL: }o,
qr{^.Test Run By }o,
qr{^.gnatbind }o,
qr{^.gnatlink }o,
qr{^.gnatlink: }o,
qr{^.gnatmake: }o,
qr{^.gcc }o,
qr{^./usr/bin/ld: .* architecture of input file .* is incompatible with .* output\n$}o,
qr{^.collect2: }o,
#qr{^.UNSUPPORTED: }o,
#qr{^.UNTESTED: }o,
#qr{^.UNRESOLVED: }o,
#qr{^.ERROR: }o,
#qr{^.WARNING: }o,
qr{^.[^:]*[.]ad[sb]:[0-9]}o,
qr{^.(?:PASS|FAIL): gdb.server/ext-run.exp: get process list\n$}o,
qr{^.(?:PASS|FAIL): gdb.base/checkpoint.exp: info checkpoints with at least 600 checkpoints\n$}o,
qr{^.(?:PASS|FAIL): gdb.base/checkpoint.exp: break2 with many checkpoints\n$}o,
qr{^.(?:PASS|FAIL): gdb.base/interrupt.exp: }o,
qr{^.(?:PASS|FAIL): gdb.base/gdb1250.exp: }o,
qr{^.KFAIL: gdb.threads/watchthreads2.exp: gdb can drop watchpoints in multithreaded app \(PRMS: gdb/10116\)\n$}o,
qr{^.PASS: gdb.threads/watchthreads2.exp: all threads incremented x\n$}o,
qr{^.FAIL: gdb.base/sigbpt.exp: Verify that SIGSEGV occurs at the last STEPI insn \(none 0x[a-f0-9]*\)\n$}o,
qr{^.PASS: gdb.threads/threadcrash.exp: core file: }o,
qr{^.PASS: gdb.threads/watchthreads.exp: disable [0-9]*\n$}o,
qr{^.FAIL: gdb.mi/mi-nsmoribund.exp: unexpected stop\n$}o,
qr{^.PASS: gdb.mi/mi-nsmoribund.exp: resume all, program exited normally\n$}o,
qr{^.gdb compile failed, }o,
qr{^.gdb compile failed, /usr/bin/ld: /tmp/[a-zA-Z0-9]*.o: relocation R_X86_64_32S against `a local symbol. can not be used when making a shared object; recompile with -fPIC\n$}o,
qr{^./tmp/[a-zA-Z0-9]*.o: could not read symbols: Bad value\n$}o,
qr{^./tmp/[a-zA-Z0-9]*.s:\d+: Error: unrecognized symbol type "gnu_indirect_function"\n$}o,
qr{^./tmp/[a-zA-Z0-9]*.o: In function `\S+':\n$}o,
qr{^./tmp/[a-zA-Z0-9]*.o:.*: undefined reference to `.*'\n$}o,
qr{^./tmp/[a-zA-Z0-9]*.s:\d+: Error: syntax error; found `.' but expected `,'\n$}o,
qr{^./tmp/[a-zA-Z0-9]*.s:\d+: Error: junk at end of line: `.(?:plt|local)'\n$}o,
qr{^./usr/bin/ld: /tmp/[a-zA-Z0-9]*.o\(\.text\+0x[0-9a-f]+\): unresolvable R_PPC64_REL24 relocation against symbol `\S+'\n$}o,
qr{^.# of }o,
qr{^./home/}o,
qr{^.PASS: gdb.arch/i386-biarch-core.exp: core-file /home/.*/i386-biarch-core.core\n$}o,
qr{^.\.\./as-new [0-9.]*\n$}o,
qr{^.ld/ld-new [0-9.]*\n$}o,
qr{^.(?:PASS|FAIL): gdb.mi/mi-break.exp: test hitting breakpoint with commands\n$}o,
qr{^.(?:PASS|FAIL): gdb.threads/attach-stopped.exp: threaded: attach2, exit leaves process sleeping\n$}o,
qr{^.(?:PASS|FAIL): gdb.threads/attachstop-mt.exp: attach3, exit leaves process stopped\n$}o,
qr{^.(?:PASS|FAIL): gdb.threads/attachstop-mt.exp: attach4 stop by interrupt(?: \(timeout\))?\n$}o,
qr{^.(?:PASS|FAIL): gdb.threads/attachstop-mt.exp: attach4, exit leaves process sleeping\n$}o,
qr{^.(?:PASS|FAIL): gdb.threads/manythreads.exp: info threads\n$}o,
qr{gdb version  \d\S+ -nw -nx \n$}o,
qr{^.(?:PASS|FAIL): gdb.threads/attach-stopped.exp: threaded: attach1, exit leaves process stopped\n$}o,
qr{^.(?:PASS|FAIL): gdb.threads/local-watch-wrong-thread.exp: local watchpoint automatically deleted(?: \(timeout\))?\n$}o,
qr{^.(?:PASS|FAIL): gdb.server/server-run.exp: continue to main\n$}o,
qr{^.(?:PASS|FAIL): gdb.mi/mi-nonstop.exp: w1,i0 stop(?: \(timeout\))?\n$}o,
qr{^.(?:PASS|FAIL): gdb.reverse/i387-env-reverse.exp: set reverse direction(?: \(timeout\))?\n$}o,
qr{^.(?:PASS|FAIL): gdb.reverse/i387-env-reverse.exp: record to end of main(?: \(timeout\))?\n$}o,
qr{^.(?:PASS|FAIL): gdb.mi/mi-file-transfer.exp: compare intermediate binary file\n$}o,
qr{^.(?:PASS|FAIL): gdb.mi/mi-file-transfer.exp: compare binary file\n$}o,
qr{^.(?:PASS|FAIL): gdb.mi/mi-file-transfer.exp: deleted binary file\n$}o,
qr{^.(?:PASS|FAIL): gdb.mi/mi-simplerun.exp: continue to end(?: \(failed to resume\))?\n$}o,
qr{^.(?:PASS|FAIL): gdb.mi/mi-simplerun.exp: continue to end(?: \(unknown output after running\))?\n$}o,
qr{^.(?:PASS|FAIL): gdb.server/file-transfer.exp: verified deleted binary file\n$}o,
qr{^.(?:PASS|FAIL): gdb.server/file-transfer.exp: deleted binary file\n$}o,
qr{^.(?:PASS|FAIL): gdb.threads/attachstop-mt.exp: attach1, post-gdb sanity check of the sleeping state - Red Hat BZ 197584\n$}o,
qr{^."uplevel #0 source .*gdb/testsuite/gdb.threads/watchthreads-thr(?:eaded\.exp|.*\.\.\.)"\n$}o,
qr{^.(?:PASS|FAIL): gdb.cp/ref-types.exp: print value of rL\n$}o,
qr{^.FAIL: gdb.threads/watchthreads-reorder.exp: reorder[01]: (?:continue [ab]|rwatch unused1_rwatch)(?: \((?:timeout|the program exited|the program is no longer running)\))?\n$}o,
qr{^.FAIL: gdb.base/sigbpt.exp: Verify that SIGSEGV occurs at the last STEPI insn \(none 0x[0-9a-f]+\)\n$}o,
qr{^.(?:KFAIL|FAIL): gdb.base/sigbpt.exp: stepi bp at segv; stepi out of handler(?: \(corrupt pc\) \(PRMS: gdb/1702\))?\n$}o,
qr{^.(?:KFAIL|FAIL): gdb.base/sigbpt.exp: stepi bp before and at segv; stepi out of handler(?: \(corrupt pc\) \(PRMS: gdb/1702\))?\n$}o,
qr{^.FAIL: \S+\.exp: valgrind check \d+\n$}o,
);
my $ignore=shift @ignore;
$ignore=qr/$ignore|$_/ for @ignore;

my $opt_new;
while (1) {
  if ($ARGV[0] eq "-r") {
    $ignore=undef;
    shift;
    next;
  }
  if ($ARGV[0] eq "-N") {
    $opt_new=1;
    shift;
    next;
  }
  last;
}

my $last;
my $lastfile;
local *DIFF;
my(@diff)=("diff","-rU-1",@ARGV);
open DIFF,"-|",@diff or die join(" ",@diff).": $!";
while (<DIFF>) {
  next if /^diff /;
  my($thisname)=m{^.(?:Running (?:.*/)?|[A-Z]+: )(gdb[.]\w+/\S+[.]exp)(?: \Q...\E|: .*)$};
  $thisname||=(m{^.\t\t=== gdb (Summary) ===\n$})[0];
  $thisname||=(m{^(Only in) })[0];
  my $this;
  if (!$thisname || ($last && $thisname eq $last->{"name"})) {
#if (!$thisname) { print "X: !thisname\n"; } else { print "X: \"".$last->{"name"}."\" -> \"$thisname\"\n"; }
    $this=$last;
  } else {
#print "X: new \"$thisname\"\n";
    $last=$this={"name"=>$thisname,"+Running"=>(/^[+]/||0)};
  }
  next if /^@@ .* @@\n$/;
  if ($this->{"print"}) {
    print;
    next;
  }
  if (/^Only in /) {
    $lastfile=$_;
  }
  if (/^--- /) {
    $lastfile=$_;
    next;
  }
  if (/^\Q+++\E /) {
    $lastfile.=$_;
    next;
  }
  $this->{"buffer"}.=$_;
  next if /^ /;
  next if /^[+]Running / && $thisname && !$opt_new;
  next if $ignore && /$ignore/o;
  next if /^[+]FAIL: / && $this->{"+Running"};
  next if /^[+]KFAIL: / && $this->{"+Running"};
  next if /^[+]XFAIL: / && $this->{"+Running"};
  next if /^[+]UNTESTED: / && $this->{"+Running"};
  next if /^[+]gdb compile failed, / && $this->{"+Running"};
  next if /^[+].*: Error: bad register name `%.*'\n$/ && $this->{"+Running"};
  next if /^[+].*: Error: `.*' is only supported in 64-bit mode\n$/ && $this->{"+Running"};

  $this->{"print"}=1;
  print $lastfile||""; $lastfile=undef();
  print "\n#".($this->{"name"}||"???")."\n";
  print $this->{"buffer"}; $this->{"buffer"}=undef();
  next;
}

Re: How to properly check for regressions?

[Andreas Schwab]

Jan Kratochvil <jan.kratochvil@redhat.com> writes:

> On Thu, 04 Aug 2011 22:51:10 +0200, Tom Tromey wrote:
>> Our buildbot automates all this -- but I suspect somewhat poorly,
>> because Jan is constantly finding regressions I didn't see.
>
> I just use `diff -u' on gdb.sum files.

See also contrib/compare_tests in the gcc repo.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: How to properly check for regressions?

[Sergio Durigan Junior]

Tom Tromey <tromey@redhat.com> writes:

>>>>>> "Marc" == Marc Khouzam <marc.khouzam@ericsson.com> writes:
>
> Marc> And how do I deal with tests that sometimes pass and sometimes fail.
>
> The most common approach is to learn what they are and ignore them.
> This isn't totally satisfactory; fixing them or removing them would be
> nicer.

At IBM I remember they had a list of "non-deterministic tests" in a wiki
page.  Maybe Thiago can confirm if this list still exists?  It would
probably be useful if we could create a similar list in our wiki?