From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (qmail 14802 invoked by alias); 18 Sep 2003 01:20:45 -0000 Mailing-List: contact gdb-patches-help@sources.redhat.com; run by ezmlm Precedence: bulk List-Subscribe: List-Archive: List-Post: List-Help: , Sender: gdb-patches-owner@sources.redhat.com Received: (qmail 14790 invoked from network); 18 Sep 2003 01:20:44 -0000 Received: from unknown (HELO hawaii.kealia.com) (209.3.10.89) by sources.redhat.com with SMTP; 18 Sep 2003 01:20:44 -0000 Received: by hawaii.kealia.com (Postfix, from userid 2049) id 3EB91CB2D; Wed, 17 Sep 2003 18:20:44 -0700 (PDT) To: gdb-patches@sources.redhat.com Subject: Re: [testsuite] add gdb.cp/gdb1355.exp References: <200309180053.h8I0rvWc012998@duracef.shout.net> <20030918005612.GA2546@nevyn.them.org> From: David Carlton Date: Thu, 18 Sep 2003 01:20:00 -0000 In-Reply-To: <20030918005612.GA2546@nevyn.them.org> (Daniel Jacobowitz's message of "Wed, 17 Sep 2003 20:56:12 -0400") Message-ID: User-Agent: Gnus/5.1002 (Gnus v5.10.2) XEmacs/21.4 (Rational FORTRAN, linux) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-SW-Source: 2003-09/txt/msg00389.txt.bz2 On Wed, 17 Sep 2003 20:56:12 -0400, Daniel Jacobowitz said: > On Wed, Sep 17, 2003 at 08:53:57PM -0400, Michael Elizabeth Chastain wrote: >>> For one thing, it would be an XFAIL, because it's a GCC bug, >>> not a GDB bug. >> The test script, gdb.cp/gdb1355.exp, refers to PR gdb/1355. >> gdb/1355 is an external PR and it refers to PR gcc/12066. >> So there is a gdb PR in there. >>> For another thing, though, the bug in question has been fixed, >>> so we don't expect it to fail: if it does, it should show up as a FAIL. >> This has been a controversy in the past, too. >> My view is that "KFAIL" means "Known FAIL", which basically means >> there is a PR for it (the PR is the locus of knowledge). > I don't think that was the consensus. KFAILs are known failures of > the tool under test, i.e. bugs in it. This is a problem in GDB's > input. That makes it an xfail. That's my understanding as well. See, for example, : it says, among other things, "An XFAIL means that we have found a bug in software external to gdb" and "KFAIL means that the failure is due to a known bug in gdb". A short thread followed that message; nobody disagreed with those statements. Also, in the thread entitled "GDB `cannotfix' pr state, require PR with xfail `moving forward'.", it seems to me that there was consensus that new xfails would be identified with a PR, and that that PR should come from GDB's bug database. (See, e.g., .) So that, too, argues against "GDB bug database implies KFAIL". >>> I would leave in the new test, with branches and comments as is, >>> but I would change all the occurrences of kfail to fail. >> >> I prefer gdb1355.exp the way it is but I would be okay with that change >> if other people want it that way. I can't find a mailing list reference for this, so I'll repeat my argument for changing KFAIL to FAIL. Let's say that we have an annoying GDB bug. We add a test to the suite reflecting the bug; initially, we've KFAILed the bug. But then somebody fixes the bug. If the bug is left as a KFAIL, instead of changing to FAIL, then we have problems in these two scenarios: Scenario 1: The bug is fixed. A year later, the bug in question regresses. But, if you just run 'make check', this isn't so obvious: no message gets printed in the output, and while the numbers at the end of the output will change, those numbers fluctuate for lots of reasons. So it will only be noticed by somebody who actually looks at gdb.sum (or does the moral equivalent); ideally that will happen, but I'd rather not count on it (especially if the bug is on an obscure platform). Scenario 2: The fix works on platform A, but not on platform B. And nobody using platform B has been paying close enough attention to the discussion of the bug to realize that the test is supposed to start passing. Then, nobody might ever realize that something's wrong: platform A users think that everything is fine (which is the case for them), and platform B have no indication from the testsuite that something is wrong and that the patch didn't work as indicated. I'm especially worried about Scenario 2, but I don't like Scenario 1 either. And I don't see any correspondingly strong scenarios that argue in the other direction. About the only problem with changing it from a KFAIL to a FAIL is that, if the bug reoccurs, you won't learn from make check (or gdb.sum or gdb.log) that the bug in question is one that has been seen earlier. But that's not a big deal, since you know which test failed, so as long as that test has a comment in it saying "this was PR gdb/123, but it's been fixed now", then you'll have access to the information you need. Which is why, in the situation at question, I like it that you have the explicit failure branches, containing an informative comment as to when the bug occurred - I just want the letter 'k' removed in a few places. :-) David Carlton carlton@kealia.com