Re: [rfa:testsuite} Overhaul sizeof.exp

Michael Elizabeth Chastain
2002-02-20T17:17:50Z

Hmmm, let's start by looking at the alternatives. Suppose someone writes
a new test that revels an unexpected bug in gdb (hi Andrew).

[0] Stqtus quo: we reject the test.
[1] FAIL it: we accept the test and let it FAIL.
[2] XFAIL it: we accept the test and mark it with XFAIL.
[3] KFAIL it: we accept the test and mark it with KFAIL.

I really want that test in the test suite. If the test is in the test
suite, it gets run regularly in a lot of configurations. Anyone can
check the status of the test just by reading the latest Sunday Project
errors-warnings-fails report. Someone who tackles the bug has the saves
time when they start and when they finish. And so on.

My preference order is [2] > [1] > [3] > [0]. All but [0] are
acceptable to me.

I understand that [1] has a problem because it looks similar to a
regression. I don't see that in my tables because they distinguish
between "old test that PASSed before and FAILs now" and "new test with
no result before and FAILs now". But I acknowledge that other people
oppose [1].

[3] has a problem because KFAIL does not exist. If the senior maintainers
decide to approve [3] then I can add it to my analysis tables right away,
but someone will have to add it to dejagnu.

I agree with Daniel Jacobowitz; I think [2] is the right thing to do.
It's also practical because we can start today.

@item XFAIL
@kindex XFAIL
@cindex expected failure
@cindex failing test, expected
A test failed, but it was expected to fail. This result indicates no
change in a known bug. If a test fails because the operating system
where the test runs lacks some facility required by the test, the
outcome is @code{UNSUPPORTED} instead.

@item UNSUPPORTED
@kindex UNSUPPORTED
@cindex unsupported test
@cindex test, unsupported
A test depends on a conditionally available feature that does not exist
(in the configured testing environment). For example, you can use this
outcome to report on a test case that does not work on a particular
target because its operating system support does not include a required
subroutine.
@end table

Michael C