From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (qmail 21090 invoked by alias); 25 Sep 2003 22:13:55 -0000 Mailing-List: contact gdb-help@sources.redhat.com; run by ezmlm Precedence: bulk List-Subscribe: List-Archive: List-Post: List-Help: , Sender: gdb-owner@sources.redhat.com Received: (qmail 21083 invoked from network); 25 Sep 2003 22:13:54 -0000 Received: from unknown (HELO hawaii.kealia.com) (209.3.10.89) by sources.redhat.com with SMTP; 25 Sep 2003 22:13:54 -0000 Received: by hawaii.kealia.com (Postfix, from userid 2049) id 9017ACB31; Thu, 25 Sep 2003 15:13:53 -0700 (PDT) To: Andrew Cagney Cc: dejagnu@gnu.org, gdb@sources.redhat.com, Fernando Nasser Subject: Re: Print KFAIL's in dejagnu summary? References: <3F7361BB.1000706@redhat.com> From: David Carlton Date: Fri, 26 Sep 2003 00:03:00 -0000 In-Reply-To: <3F7361BB.1000706@redhat.com> (Andrew Cagney's message of "Thu, 25 Sep 2003 17:44:27 -0400") Message-ID: User-Agent: Gnus/5.1002 (Gnus v5.10.2) XEmacs/21.4 (Rational FORTRAN, linux) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-SW-Source: 2003-09/txt/msg00319.txt.bz2 On Thu, 25 Sep 2003 17:44:27 -0400, Andrew Cagney said: > At present KFAILs are supressed from the summary output (the stuff > on the terminal from "make check"). I'd like to change this so that > KFAILs, just like FAILs, are included in the summary. A KFAIL, just > like a FAIL, indicates a bug in the system under test, and hence > should be included in the summary. I have a mild preference for the current behavior. Mostly I use the summary output to get a feel for whether or not a change of mine has obviously gone wrong; the noisier the summary output is, the harder it is to use it this way. Of course, I always search the entire gdb.sum for regressions, just to make sure, so it won't make a big practical difference to me one way or another. I also think that, right now, a bigger issue is to get the testsuite and GNATS to agree about what the bugs are in GDB. So I'm more interesting in removing the undiagnosed XFAILs and in making some effort to diagnose existing FAILs than I am in making the KFAILs more prominent. (I'd also like to see some effort towards making sure that open bugs in GNATS are reflected via KFAILs in the test suite.) It would also be nice if we had a feel for what bugs are low-hanging fruit. The part of the test suite that I pay the most attention to is, of course, gdb.cp; I haven't noticed that the addition of KFAILs there has had much of an effect on the rate of bug fixing. (There are a lot of old bugs that are still present, but they'd been present for a while.) Of course, a lot of the KFAIL additions are from tests that used to be XFAILs, which didn't change the visibility level of the bugs. Some more random musings about the possible roles of the testsuite: * Its main role is to catch regressions. Here, I think that making FAILs more prominent is good: a FAIL is more likely to be a regression than a KFAIL (at least if people delete the appropriate setup_kfails from the test suite when they fix bugs). * It's also useful for people trying to fix bugs: if you've decided to fix bug X and if the testsuite contains a KFAIL associated to X, then that makes it a lot easier to check if you've fixed the bug (and to try to figure out what's causing the bug, for that matter). Here, too, there's no reason for KFAILs to be prominent. * Another role is to remind people of the existence of bugs, to exhort people to fix them; that's important to you, but a little less important to me than the other two roles. David Carlton carlton@kealia.com