Re: RFC: KFAIL DejaGnu patch

[Michael Elizabeth Chastain <mec@shout.net>]

More comments ...

gdb.c++/cplusfuncs.exp has a good place for some KFAIL's.  Grep for
hairyfunc5, hairyfunc6, hairyfunc7.  They are associated with PR gdb/19.
I added some setup_kfail's to that and it appears to work okay,
except that the bug id is not printed on xpass.

Okay, here's some proofreading comments.

Michael C

=== Distinction between XFAIL and KFAIL

An XFAIL is:

  A test is expected to fail in some environment(s) due to some
  bug in the environment ...

And a KFAIL is:

  A test is known to fail in some environment(s) due to a known bug
  in the tool being tested (identified by a bug id string).

I like this distinction.  The "K" letter means that it is a known problem
inside the tool, and the "X" letter means that it is an expected problem
outside the tool.

That makes it weird for a KFAIL to turn into an XPASS.  I've got test
results here with XPASS for the gcc v2 compilers and KFAIL for the gcc
v3 compilers.  I really want KFAIL/KPASS, not KFAIL/XPASS.

A specific note: proc record_test has this code:

  XPASS {
      set exit_status 1
      if { $xfail_prms != 0 } {
	  set message [concat $message "\t(PRMS $xfail_prms)"]
      }
  }

In fact that code could use a little refactoring, because
UNRESOLVED/UNSUPPORTD/UNTESTED all have the same code to pick up
kfail_prms and xfail_prms.

So an XPASS that comes from setup_xfail will have a PRMS id,
but an XPASS that comes from setup_kfail will not.

=== KFAIL as a way of hiding problems

Later on in the documentation:

  @item KFAIL
  ...
  This exists so that, after a bug is identified and properly registered
  in a bug tracking database (Gnats, for instance), the count of failures
  can be kept as zero.  Having zero has a baseline in all platforms allow
  the tool developers to immediately detect regressions caused by changes
  (which may affect some platforms and not others).

Conceptually, there is a disconnect here.  To me, KFAIL means: "there
is a problem report associated with this test".  The problem could be
minor, or it could be a showstopper.  But you say that KFAIL means:
"tool developers can ignore this test when they look for regressions".

I think this is fundamentally wrong.  The right way to look for
regressions is to compare before-and-after test runs.  gdb.sum files
are already pretty good for this; you can just diff them.

I see two practical problems that come out of this wrongness:

(1) If I find a showstopper regression, such as PR gdb/379, and I mark
    it with setup_kfail (as I should), then someone who is using the
    "# of FAILs" metric is not going to see the regression.

(2) If the test suite has a significant number of setup_kfail (and
    it should), then a regression bug may manifest as a transition from
    PASS -> KFAIL.  I will see that because my regression reports look
    at all transitions.  But someone looking at "# of FAILs" will not see
    this transition, so they won't see the regression.

I believe we already have this problem with XFAIL.  People conflate
the idea of "this test fails due to an external problem" with the idea
"this failure is not important enough to care about at this time".

=== ChangeLog entry

DejaGnu 1.4.2 has a ChangeLog, so the patch needs a ChangeLog entry.

=== Tests

testsuite/ needs some tests for the new KFAIL feature.

=== lib/dg.exp

lib/dg.exp has a bunch of XFAIL stuff, so it needs KFAIL stuff.