From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (qmail 28092 invoked by alias); 8 Apr 2002 01:41:10 -0000 Mailing-List: contact gdb-patches-help@sources.redhat.com; run by ezmlm Precedence: bulk List-Subscribe: List-Archive: List-Post: List-Help: , Sender: gdb-patches-owner@sources.redhat.com Received: (qmail 27802 invoked from network); 8 Apr 2002 01:41:07 -0000 Received: from unknown (HELO duracef.shout.net) (204.253.184.12) by sources.redhat.com with SMTP; 8 Apr 2002 01:41:07 -0000 Received: (from mec@localhost) by duracef.shout.net (8.11.6/8.11.6) id g381ehg18770; Sun, 7 Apr 2002 20:40:43 -0500 Date: Sun, 07 Apr 2002 18:41:00 -0000 From: Michael Elizabeth Chastain Message-Id: <200204080140.g381ehg18770@duracef.shout.net> To: fnasser@redhat.com Subject: Re: RFC: KFAIL DejaGnu patch Cc: ac131313@cygnus.com, drow@mvista.com, gdb-patches@sources.redhat.com, rob@welcomehome.org X-SW-Source: 2002-04/txt/msg00286.txt.bz2 More comments ... gdb.c++/cplusfuncs.exp has a good place for some KFAIL's. Grep for hairyfunc5, hairyfunc6, hairyfunc7. They are associated with PR gdb/19. I added some setup_kfail's to that and it appears to work okay, except that the bug id is not printed on xpass. Okay, here's some proofreading comments. Michael C === Distinction between XFAIL and KFAIL An XFAIL is: A test is expected to fail in some environment(s) due to some bug in the environment ... And a KFAIL is: A test is known to fail in some environment(s) due to a known bug in the tool being tested (identified by a bug id string). I like this distinction. The "K" letter means that it is a known problem inside the tool, and the "X" letter means that it is an expected problem outside the tool. That makes it weird for a KFAIL to turn into an XPASS. I've got test results here with XPASS for the gcc v2 compilers and KFAIL for the gcc v3 compilers. I really want KFAIL/KPASS, not KFAIL/XPASS. A specific note: proc record_test has this code: XPASS { set exit_status 1 if { $xfail_prms != 0 } { set message [concat $message "\t(PRMS $xfail_prms)"] } } In fact that code could use a little refactoring, because UNRESOLVED/UNSUPPORTD/UNTESTED all have the same code to pick up kfail_prms and xfail_prms. So an XPASS that comes from setup_xfail will have a PRMS id, but an XPASS that comes from setup_kfail will not. === KFAIL as a way of hiding problems Later on in the documentation: @item KFAIL ... This exists so that, after a bug is identified and properly registered in a bug tracking database (Gnats, for instance), the count of failures can be kept as zero. Having zero has a baseline in all platforms allow the tool developers to immediately detect regressions caused by changes (which may affect some platforms and not others). Conceptually, there is a disconnect here. To me, KFAIL means: "there is a problem report associated with this test". The problem could be minor, or it could be a showstopper. But you say that KFAIL means: "tool developers can ignore this test when they look for regressions". I think this is fundamentally wrong. The right way to look for regressions is to compare before-and-after test runs. gdb.sum files are already pretty good for this; you can just diff them. I see two practical problems that come out of this wrongness: (1) If I find a showstopper regression, such as PR gdb/379, and I mark it with setup_kfail (as I should), then someone who is using the "# of FAILs" metric is not going to see the regression. (2) If the test suite has a significant number of setup_kfail (and it should), then a regression bug may manifest as a transition from PASS -> KFAIL. I will see that because my regression reports look at all transitions. But someone looking at "# of FAILs" will not see this transition, so they won't see the regression. I believe we already have this problem with XFAIL. People conflate the idea of "this test fails due to an external problem" with the idea "this failure is not important enough to care about at this time". === ChangeLog entry DejaGnu 1.4.2 has a ChangeLog, so the patch needs a ChangeLog entry. === Tests testsuite/ needs some tests for the new KFAIL feature. === lib/dg.exp lib/dg.exp has a bunch of XFAIL stuff, so it needs KFAIL stuff.