[RFA:] sim-defs.exp: support xfail

[RFA:] sim-defs.exp: support xfail

[Hans-Peter Nilsson]

Directly taken from the same option in the ld run_dump_test.

Ok to commit?

2004-11-10  Hans-Peter Nilsson  <hp@axis.com>

	* lib/sim-defs.exp (run_sim_test): Support "xfail" option.


Index: sim-defs.exp
===================================================================
RCS file: /cvs/src/src/sim/testsuite/lib/sim-defs.exp,v
retrieving revision 1.8
diff -c -p -r1.8 sim-defs.exp
*** sim-defs.exp	26 Oct 2004 08:07:59 -0000	1.8
--- sim-defs.exp	10 Nov 2004 09:49:44 -0000
*************** proc sim_run { prog sim_opts prog_opts r
*** 163,168 ****
--- 163,169 ----
  # sim[(mach-list)]: <simulator options>
  # output: program output pattern to match with string-match
  # xerror: program is expected to return with a "failure" exit code
+ # xfail: <target-triple-where-test-fails>
  # If `output' is not specified, the program must output "pass" if !xerror or
  # "fail" if xerror.
  # The parens in "optname()" are optional if the specification is for all machs.
*************** proc run_sim_test { name requested_machs
*** 196,201 ****
--- 197,203 ----
      set opts(mach) ""
      set opts(timeout) ""
      set opts(xerror) "no"
+     set opts(xfail) ""
  
      if ![info exists global_as_options] {
          set global_as_options ""
*************** proc run_sim_test { name requested_machs
*** 262,267 ****
--- 268,278 ----
  
  	verbose -log "Testing $name on machine $mach."
  
+ 	# Time to setup xfailures.
+ 	foreach targ $opts(xfail) {
+ 	    setup_xfail $targ
+ 	}
+ 
  	if ![info exists opts(as,$mach)] {
  	    set opts(as,$mach) $opts(as)
  	}

brgds, H-P

Re: [RFA:] sim-defs.exp: support xfail

[Andrew Cagney]

Hans-Peter Nilsson wrote:
> Directly taken from the same option in the ld run_dump_test.
> 
> Ok to commit?
> 
> 2004-11-10  Hans-Peter Nilsson  <hp@axis.com>
> 
> 	* lib/sim-defs.exp (run_sim_test): Support "xfail" option.

What do you mean by "xfail"? (Hint, check a current dejagnu document 
where it describes kfail :-)

Andrew

> Index: sim-defs.exp
> ===================================================================
> RCS file: /cvs/src/src/sim/testsuite/lib/sim-defs.exp,v
> retrieving revision 1.8
> diff -c -p -r1.8 sim-defs.exp
> *** sim-defs.exp	26 Oct 2004 08:07:59 -0000	1.8
> --- sim-defs.exp	10 Nov 2004 09:49:44 -0000
> *************** proc sim_run { prog sim_opts prog_opts r
> *** 163,168 ****
> --- 163,169 ----
>   # sim[(mach-list)]: <simulator options>
>   # output: program output pattern to match with string-match
>   # xerror: program is expected to return with a "failure" exit code
> + # xfail: <target-triple-where-test-fails>
>   # If `output' is not specified, the program must output "pass" if !xerror or
>   # "fail" if xerror.
>   # The parens in "optname()" are optional if the specification is for all machs.
> *************** proc run_sim_test { name requested_machs
> *** 196,201 ****
> --- 197,203 ----
>       set opts(mach) ""
>       set opts(timeout) ""
>       set opts(xerror) "no"
> +     set opts(xfail) ""
>   
>       if ![info exists global_as_options] {
>           set global_as_options ""
> *************** proc run_sim_test { name requested_machs
> *** 262,267 ****
> --- 268,278 ----
>   
>   	verbose -log "Testing $name on machine $mach."
>   
> + 	# Time to setup xfailures.
> + 	foreach targ $opts(xfail) {
> + 	    setup_xfail $targ
> + 	}
> + 
>   	if ![info exists opts(as,$mach)] {
>   	    set opts(as,$mach) $opts(as)
>   	}
> 
> brgds, H-P
> 

Re: [RFA:] sim-defs.exp: support xfail

[Hans-Peter Nilsson]

> Date: Wed, 10 Nov 2004 09:01:56 -0500
> From: Andrew Cagney <cagney@gnu.org>

> Hans-Peter Nilsson wrote:
> > Directly taken from the same option in the ld run_dump_test.
> > Ok to commit?
> > 2004-11-10  Hans-Peter Nilsson  <hp@axis.com>
> > 
> > 	* lib/sim-defs.exp (run_sim_test): Support "xfail" option.
> 
> What do you mean by "xfail"? (Hint, check a current dejagnu document 
> where it describes kfail :-)

I know about kfail, but I'm agnostic.  I just mean what happens
for the ld run_dump_test "xfail" option; a simple setup_xfail.
The purpose is to be able to keep failing tests around but not
get the total results marked as partially failing.  Reasons for
the failure would be unspecified, matching both xfail and kfail
usage.

To me, it doesn't really matter whether it's kfail or xfail
(vivid descriptions of kfail vs. xfail to /dev/null).  In the
sim testsuite I believe it'll be used very sparingly, perhaps
not even for things that are checked in.  But it'd be Nice to
Have.

Is the change acceptable by naming the option "kfail" and
calling "setup_kfail"?  Or do you want both?

brgds, H-P
PS.  BTW, the dejagnu documentation on
<URL:http://www.gnu.org/software/dejagnu/manual/> is not
current; KFAIL isn't mentioned there.

Re: [RFA:] sim-defs.exp: support xfail

[Andrew Cagney]

Hans-Peter Nilsson wrote:
>>Date: Wed, 10 Nov 2004 09:01:56 -0500
>>From: Andrew Cagney <cagney@gnu.org>
> 
> 
>>Hans-Peter Nilsson wrote:
>>
>>>Directly taken from the same option in the ld run_dump_test.
>>>Ok to commit?
>>>2004-11-10  Hans-Peter Nilsson  <hp@axis.com>
>>>
>>>	* lib/sim-defs.exp (run_sim_test): Support "xfail" option.
>>
>>What do you mean by "xfail"? (Hint, check a current dejagnu document 
>>where it describes kfail :-)
> 
> 
> I know about kfail, but I'm agnostic.  I just mean what happens
> for the ld run_dump_test "xfail" option; a simple setup_xfail.
> The purpose is to be able to keep failing tests around but not
> get the total results marked as partially failing.  Reasons for
> the failure would be unspecified, matching both xfail and kfail
> usage.
> 
> To me, it doesn't really matter whether it's kfail or xfail
> (vivid descriptions of kfail vs. xfail to /dev/null).  In the
> sim testsuite I believe it'll be used very sparingly, perhaps
> not even for things that are checked in.  But it'd be Nice to
> Have.
> 
> Is the change acceptable by naming the option "kfail" and
> calling "setup_kfail"?  Or do you want both?
> 
> brgds, H-P
> PS.  BTW, the dejagnu documentation on
> <URL:http://www.gnu.org/software/dejagnu/manual/> is not
> current; KFAIL isn't mentioned there.

Ok, really weird, the date on that page says 2004 but the manual is 
missing (from dejagnu 1.4.4):

@item KFAIL
A test is known to fail in some environment(s) due to a known bug
in the tool being tested (identified by a bug id string).  This
exists so that, after a bug is identified and properly registered
in a bug tracking database (Gnats, for instance), the count of
failures can be kept as zero.  Having zero as a baseline in all
platforms allow the tool developers to immediately detect regressions
caused by changes (which may affect some platforms and not others).
The connection with a bug tracking database allows for automatic
generation of the BUGS section of man pages or Release Notes, as
well as a "Bugs Fixed this Release" section (by comparing to a
previous release set of known failures).
The procedure @code{setup_kfail} is used to indicate a failure is
known to exist.

@cindex KFAIL, avoiding for POSIX
As with @code{XFAIL}, @sc{posix} tests must return @code{FAIL} rather
than @code{KFAIL} even if a failure was due to a known bug.

I can't think of a single reason for a simulator test that failing not 
being for a known bug.

However, for consistency with dejagnu, we might as well support both. 
Feel free to add the necessary glue to cover all of xfail and kfail and 
kpass.

Andrew

Re: [RFA:] sim-defs.exp: support xfail

[Hans-Peter Nilsson]

> Date: Wed, 10 Nov 2004 11:38:47 -0500
> From: Andrew Cagney <cagney@gnu.org>

> I can't think of a single reason for a simulator test that failing not 
> being for a known bug.
> 
> However, for consistency with dejagnu, we might as well support both. 
> Feel free to add the necessary glue to cover all of xfail and kfail and 
> kpass.

I felt free to the following, tested with dejagnu-1.4.4.
There's no separate kpass support (and no need for any, for the
same reason you don't need to add separate xpass support. :-)

	* lib/sim-defs.exp (run_sim_test): Support "xfail" and "kfail".
	In head comment, mention that output specifications concatenate.

Index: sim-defs.exp
===================================================================
RCS file: /cvs/src/src/sim/testsuite/lib/sim-defs.exp,v
retrieving revision 1.9
diff -c -p -r1.9 sim-defs.exp
*** sim-defs.exp	16 Nov 2004 07:06:17 -0000	1.9
--- sim-defs.exp	16 Nov 2004 16:12:10 -0000
*************** proc sim_run { prog sim_opts prog_opts r
*** 163,171 ****
--- 163,177 ----
  # sim[(mach-list)]: <simulator options>
  # output: program output pattern to match with string-match
  # xerror: program is expected to return with a "failure" exit code
+ # xfail: <PRMS-opt> <target-triplets-where-test-fails>
+ # kfail: <PRMS> <target-triplets-where-test-fails>
  # If `output' is not specified, the program must output "pass" if !xerror or
  # "fail" if xerror.
  # The parens in "optname()" are optional if the specification is for all machs.
+ # Multiple "output", "xfail" and "kfail" options concatenate.
+ # The xfail and kfail arguments are space-separated target triplets and PRIDs.
+ # There must be a PRMS (bug report ID) specified for kfail, while it's
+ # optional for xfail.
  
  proc run_sim_test { name requested_machs } {
      global subdir srcdir
*************** proc run_sim_test { name requested_machs
*** 196,201 ****
--- 202,209 ----
      set opts(mach) ""
      set opts(timeout) ""
      set opts(xerror) "no"
+     set opts(xfail) ""
+     set opts(kfail) ""
  
      if ![info exists global_as_options] {
          set global_as_options ""
*************** proc run_sim_test { name requested_machs
*** 233,238 ****
--- 241,251 ----
  	if { $opt_name == "output" } {
  	    set opt_val "$opts(output)$opt_val"
  	}
+ 	# Similar with "xfail" and "kfail", but arguments are space-separated.
+ 	if { $opt_name == "xfail" || $opt_name == "kfail" } {
+ 	    set opt_val "$opts($opt_name) $opt_val"
+ 	}
+ 
  	foreach m $opt_machs {
  	    set opts($opt_name,$m) $opt_val
  	}
*************** proc run_sim_test { name requested_machs
*** 266,271 ****
--- 279,296 ----
  
  	verbose -log "Testing $name on machine $mach."
  
+ 	# Time to setup xfailures and kfailures.
+ 	if { "$opts(xfail)" != "" } {
+ 	    verbose -log "xfail: $opts(xfail)"
+ 	    # Using eval to make $opts(xfail) appear as individual
+ 	    # arguments.
+ 	    eval setup_xfail $opts(xfail)
+ 	}
+ 	if { "$opts(kfail)" != "" } {
+ 	    verbose -log "kfail: $opts(kfail)"
+ 	    eval setup_kfail $opts(kfail)
+ 	}
+ 
  	if ![info exists opts(as,$mach)] {
  	    set opts(as,$mach) $opts(as)
  	}

brgds, H-P