From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (qmail 28134 invoked by alias); 20 Feb 2002 17:17:50 -0000 Mailing-List: contact gdb-patches-help@sources.redhat.com; run by ezmlm Precedence: bulk List-Subscribe: List-Archive: List-Post: List-Help: , Sender: gdb-patches-owner@sources.redhat.com Received: (qmail 27907 invoked from network); 20 Feb 2002 17:17:46 -0000 Received: from unknown (HELO duracef.shout.net) (204.253.184.12) by sources.redhat.com with SMTP; 20 Feb 2002 17:17:46 -0000 Received: (from mec@localhost) by duracef.shout.net (8.11.6/8.11.6) id g1KHHYO30993; Wed, 20 Feb 2002 11:17:34 -0600 Date: Wed, 20 Feb 2002 09:17:00 -0000 From: Michael Elizabeth Chastain Message-Id: <200202201717.g1KHHYO30993@duracef.shout.net> To: drow@mvista.com, fnasser@redhat.com Subject: Re: [rfa:testsuite} Overhaul sizeof.exp Cc: ac131313@cygnus.com, gdb-patches@sources.redhat.com X-SW-Source: 2002-02/txt/msg00547.txt.bz2 Hmmm, let's start by looking at the alternatives. Suppose someone writes a new test that revels an unexpected bug in gdb (hi Andrew). [0] Stqtus quo: we reject the test. [1] FAIL it: we accept the test and let it FAIL. [2] XFAIL it: we accept the test and mark it with XFAIL. [3] KFAIL it: we accept the test and mark it with KFAIL. I really want that test in the test suite. If the test is in the test suite, it gets run regularly in a lot of configurations. Anyone can check the status of the test just by reading the latest Sunday Project errors-warnings-fails report. Someone who tackles the bug has the saves time when they start and when they finish. And so on. My preference order is [2] > [1] > [3] > [0]. All but [0] are acceptable to me. I understand that [1] has a problem because it looks similar to a regression. I don't see that in my tables because they distinguish between "old test that PASSed before and FAILs now" and "new test with no result before and FAILs now". But I acknowledge that other people oppose [1]. [3] has a problem because KFAIL does not exist. If the senior maintainers decide to approve [3] then I can add it to my analysis tables right away, but someone will have to add it to dejagnu. I agree with Daniel Jacobowitz; I think [2] is the right thing to do. It's also practical because we can start today. @item XFAIL @kindex XFAIL @cindex expected failure @cindex failing test, expected A test failed, but it was expected to fail. This result indicates no change in a known bug. If a test fails because the operating system where the test runs lacks some facility required by the test, the outcome is @code{UNSUPPORTED} instead. @item UNSUPPORTED @kindex UNSUPPORTED @cindex unsupported test @cindex test, unsupported A test depends on a conditionally available feature that does not exist (in the configured testing environment). For example, you can use this outcome to report on a test case that does not work on a particular target because its operating system support does not include a required subroutine. @end table Michael C