Re: Print KFAIL's in dejagnu summary?

[David Carlton <carlton@kealia.com>]

On Thu, 25 Sep 2003 17:44:27 -0400, Andrew Cagney <ac131313@redhat.com> said:

> At present KFAILs are supressed from the summary output (the stuff
> on the terminal from "make check").  I'd like to change this so that
> KFAILs, just like FAILs, are included in the summary.  A KFAIL, just
> like a FAIL, indicates a bug in the system under test, and hence
> should be included in the summary.

I have a mild preference for the current behavior.  Mostly I use the
summary output to get a feel for whether or not a change of mine has
obviously gone wrong; the noisier the summary output is, the harder it
is to use it this way.  Of course, I always search the entire gdb.sum
for regressions, just to make sure, so it won't make a big practical
difference to me one way or another.

I also think that, right now, a bigger issue is to get the testsuite
and GNATS to agree about what the bugs are in GDB.  So I'm more
interesting in removing the undiagnosed XFAILs and in making some
effort to diagnose existing FAILs than I am in making the KFAILs more
prominent.  (I'd also like to see some effort towards making sure that
open bugs in GNATS are reflected via KFAILs in the test suite.)  It
would also be nice if we had a feel for what bugs are low-hanging
fruit.

The part of the test suite that I pay the most attention to is, of
course, gdb.cp; I haven't noticed that the addition of KFAILs there
has had much of an effect on the rate of bug fixing.  (There are a lot
of old bugs that are still present, but they'd been present for a
while.)  Of course, a lot of the KFAIL additions are from tests that
used to be XFAILs, which didn't change the visibility level of the
bugs.

Some more random musings about the possible roles of the testsuite:

* Its main role is to catch regressions.  Here, I think that making
  FAILs more prominent is good: a FAIL is more likely to be a
  regression than a KFAIL (at least if people delete the appropriate
  setup_kfails from the test suite when they fix bugs).

* It's also useful for people trying to fix bugs: if you've decided to
  fix bug X and if the testsuite contains a KFAIL associated to X,
  then that makes it a lot easier to check if you've fixed the bug
  (and to try to figure out what's causing the bug, for that matter).
  Here, too, there's no reason for KFAILs to be prominent.

* Another role is to remind people of the existence of bugs, to exhort
  people to fix them; that's important to you, but a little less
  important to me than the other two roles.

David Carlton
carlton@kealia.com