From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (qmail 8698 invoked by alias); 3 Jan 2003 22:14:40 -0000 Mailing-List: contact gdb-patches-help@sources.redhat.com; run by ezmlm Precedence: bulk List-Subscribe: List-Archive: List-Post: List-Help: , Sender: gdb-patches-owner@sources.redhat.com Received: (qmail 8691 invoked from network); 3 Jan 2003 22:14:39 -0000 Received: from unknown (HELO jackfruit.Stanford.EDU) (171.64.38.136) by 209.249.29.67 with SMTP; 3 Jan 2003 22:14:39 -0000 Received: (from carlton@localhost) by jackfruit.Stanford.EDU (8.11.6/8.11.6) id h03MEPm31823; Fri, 3 Jan 2003 14:14:25 -0800 X-Authentication-Warning: jackfruit.Stanford.EDU: carlton set sender to carlton@math.stanford.edu using -f To: Daniel Jacobowitz Cc: gdb-patches@sources.redhat.com, Michael Elizabeth Chastain Subject: Re: [patch/rfc] KFAIL gdb.c++/annota2.exp watch triggered on a.x References: <20030103213920.GA21687@nevyn.them.org> <20030103215134.GB9980@nevyn.them.org> From: David Carlton Date: Fri, 03 Jan 2003 22:14:00 -0000 In-Reply-To: <20030103215134.GB9980@nevyn.them.org> Message-ID: User-Agent: Gnus/5.0808 (Gnus v5.8.8) XEmacs/21.4 (Common Lisp) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-SW-Source: 2003-01/txt/msg00104.txt.bz2 On Fri, 3 Jan 2003 16:51:34 -0500, Daniel Jacobowitz said: > How do you envision them updating the testsuite? Certainly not by > removing the KFAIL's pattern; that defeats the point of having a > regression test. That's actually exactly how I expect them to update the testsuite (though they might want to keep the pattern around in a comment somewhere, or even leave the pattern intact but replace the kfail by fail plus a comment). If a bug is claimed to be fixed but isn't actually fixed, then that bug isn't a known failure any more, so it should be FAILed until the PR is reopened. Consider this scenario: we have a bug, with a test for it; that test has a PASS pattern (either because people don't like KPASS or because the bug is intermittent) and a KFAIL pattern. Then programmer A, a habitual user of platform A, fixes the bug, closes the PR, happily notices that the test has changed from KFAIL to PASS, but leaves the KFAIL branch intact. Except that it turns out that programmer A's fix was more specific to platform A than he or she realized. The bug is still present on platform B; programmer B on platform B didn't notice that the bug was allegedly fixed (maybe programmer B wasn't even working on GDB at the time that the bug was fixed), but programmer B does notice that there's a KFAILed test case corresponding to the bug. So programmer B assumes that the bug is, in fact, known, whereas from the point of view of both programmer A and the GDB bug database, the bug is supposed to have gone away. On the other hand, if programmer A had deleted the KFAIL pattern, then programmer B would see the test start FAILing, would say "hey, the test suite says that GDB has an unknown bug here", and would then have a reason to investigate the situation. Removing the test entirely would the point of having a regression test. But removing patterns that handle casses that we don't expect to occur should make the test more effective rather than less effective: if we get surprising output from GDB, we want that to be flagged as prominently as possible. At least, that's my reasoning. David Carlton carlton@math.stanford.edu