RFC: KFAILs [Was: [RFA/mi-testsuite] XFAIL mi*-console.exp]

[Fernando Nasser <fnasser@redhat.com>]

Michael Elizabeth Chastain wrote:
> 
> What's the status of KFAIL?
> 

I have it here in my sandbox, but I will have to clean it up a
little bit this Sunday. I have a few questions and comments:


1) All current Dejagnu messages have the bug identification at
the end (between parenthesis).  I want to keep things consistent.
Is that OK?

2) Also, to keep things consistent in Dejagnu, the bug identification
for the kfail would be printed as "(PRMS xxxx)" as it is done
for XFAIL etc., to distinguish from bug_id which is printed as
"(BUG xxxx)".  Is that a problem?

So, a KFAIL would be printed as:

KFAIL: could not run to marker1 (PRMS gdb/999)

Would that make the scripts happy?


3) For KFAILs, the bug identification is mandatory.  In this case
I would like to change the syntax a little bit and have it as the
first argument and issue an error if it is not there.  I would 
still keep the current Dejagnu convention that bug identifications
are strings without a '-' in it.  I am doing this just to be able
to detect that someone forgot to specify it.  I know that there are
other syntax possibilities (like command switches), but I am trying
to be consistent with what is already there.

So, here is an example of a kfail use:

setup_kfail "gdb/999" *-*-*


4) Note that, when a test that was expected to fail due to a known
bug suddenly starts to pass, it becomes a KPASS (as XFAILs do).
It is then time to go and check if the bug was fixed and someone
forgot to remove the setup_kfail or if something changed in the
environment and the bug is not applicable anymore (or the test
is not catching it anymore).  More or less as we should do with
XPASSes that appear instead of XFAILs.  Also, sometimes the target
specification is a little bit too general (the problem does not
happen in all the specified conditions).


5) As soon as we have KFAILs, we will have to re-examine the FAILs
we have, register a bug for what is a gdb bug.  Also, we will be able
to have zero FAILs, so in any platform a non-zero result will mean that
we broke something.  We will also have to re-check everything that was
marked as XFAIL and see if they are actually known GDB bugs.



6) I would like to write a script that, as part of the release, goes 
through the KFAILs, access the Gnats database with the Bug Id and 
fills the BUGS part of the man page (can also generate a report and
put it in a file -- or append it to a Release Notes").  
We can also have a script to compare the new known bugs report with
the previous release one and generate a list of "Bugs fixed in this
release" (this can help people decide if they should upgrade, or stop
people to post just to hear: "I suggest you upgrade").

I will do it in Perl (I still don't know how to programmatically access
the Gnats database though).  But I have very little spare time, so I
will
not mind if someone that can do it sooner volunteers to do this.
 


-- 
Fernando Nasser
Red Hat Canada Ltd.                     E-Mail:  fnasser@redhat.com
2323 Yonge Street, Suite #300
Toronto, Ontario   M4P 2C9