[rfa/testsuite/PING] asm-source.exp: use UNTESTED

[rfa/testsuite/PING] asm-source.exp: use UNTESTED

[Michael Elizabeth Chastain]

[Originally submitted 2003-12-18]

I'm walking through the gdb.sum file for native hppa2.0w-hp-hpux11.11,
cleaning up crap in the test suite so that I can see the condition of
gdb better.  So here's the first patch.

This patch changes asm-source.exp for architectures that aren't
implemented yet.  The existing code calls gdb_suppress_entire_file, a
nasty function that doesn't actually suppress any tests: it just forces
a lot of test results to FAIL.  So I get: 4 ERROR, 5 WARNING, 28 FAIL, 1
UNRESOLVED.

My patch just reports UNTESTED and then returns.

I think that UNTESTED is the right test result here.  The dejagnu doco
says:

  @item UNTESTED
  @kindex UNTESTED
  @cindex untested properties
  A test case is not yet complete, and in particular cannot yet produce a
  @code{PASS} or @code{FAIL}.  You can also use this outcome in dummy
  ``tests'' that note explicitly the absence of a real test case
  for a particular property.

If you want the output to be WARNING and then UNTESTED, I wouldn't
object.  Or if you want a FAIL in there too so that people who ignore
everything but FAIL would see it.  I think UNTESTED is completely right
for this situation, but I'm flexible about happens.

But the call to gdb_suppress_entire_file really must die.  I can't deal
with 4 ERROR, 5 WARNING, 28 FAIL, and 1 UNRESOLVED just because no one
has written the assembly language test yet.  And I'm not going to write
it yet because I have to deal with testing HP's compilers and assemblers
first; it can't be just another hunk of gnu assembly code with a few
opcodes changed.

I tested this on native hppa2.0w-hp-hpux11.11, with a result
of UNTESTED.

Okay to commit?

Michael C

2003-12-17  Michael Chastain  <mec.gnu@mindspring.com>

	* gdb.asm/asm-source.exp: Return UNTESTED for platforms that
	have not implemented the assembly source test.

*** ORIGINAL-asm-source.exp	2003-12-18 01:15:58.000000000 -0500
--- asm-source.exp	2003-12-18 01:16:07.000000000 -0500
*************** switch -glob -- [istarget] {
*** 115,121 ****
  }
  
  if { "${asm-arch}" == "" } {
!     gdb_suppress_entire_file "Assembly source test -- not implemented for this target."
  }
  
  # On FreeBSD and NetBSD, crt1.o the final link will fail because of
--- 115,122 ----
  }
  
  if { "${asm-arch}" == "" } {
!     untested "assembly source test not implemented for this target"
!     return
  }
  
  # On FreeBSD and NetBSD, crt1.o the final link will fail because of

Re: [rfa/testsuite/PING] asm-source.exp: use UNTESTED

[Andrew Cagney]

> [Originally submitted 2003-12-18]
> 
> I'm walking through the gdb.sum file for native hppa2.0w-hp-hpux11.11,
> cleaning up crap in the test suite so that I can see the condition of
> gdb better.  So here's the first patch.
> 
> This patch changes asm-source.exp for architectures that aren't
> implemented yet.  The existing code calls gdb_suppress_entire_file, a
> nasty function that doesn't actually suppress any tests: it just forces
> a lot of test results to FAIL.  So I get: 4 ERROR, 5 WARNING, 28 FAIL, 1
> UNRESOLVED.
> 
> My patch just reports UNTESTED and then returns.
> 
> I think that UNTESTED is the right test result here.  The dejagnu doco
> says:
> 
>   @item UNTESTED
>   @kindex UNTESTED
>   @cindex untested properties
>   A test case is not yet complete, and in particular cannot yet produce a
>   @code{PASS} or @code{FAIL}.  You can also use this outcome in dummy
>   ``tests'' that note explicitly the absence of a real test case
>   for a particular property.

Technically yes.

However, remember why this test was originally changed to fail messy - 
the test was being skipped and, as a demonstratable consequence, 
everyone chose to ignore it rather than fix the testcase :-(

What about UNTESTED, and then KFAIL everything?  Knowing my luck that 
will be much harder than it seems ...

The other solution is to require people to post test results (so that 
they are recorded) when submitting a new architecture.

> If you want the output to be WARNING and then UNTESTED, I wouldn't
> object.  Or if you want a FAIL in there too so that people who ignore
> everything but FAIL would see it.  I think UNTESTED is completely right
> for this situation, but I'm flexible about happens.
> 
> But the call to gdb_suppress_entire_file really must die.  I can't deal
> with 4 ERROR, 5 WARNING, 28 FAIL, and 1 UNRESOLVED just because no one
> has written the assembly language test yet.  And I'm not going to write
> it yet because I have to deal with testing HP's compilers and assemblers
> first; it can't be just another hunk of gnu assembly code with a few
> opcodes changed.
> 
> I tested this on native hppa2.0w-hp-hpux11.11, with a result
> of UNTESTED.
> 
> Okay to commit?
> 
> Michael C
> 
> 2003-12-17  Michael Chastain  <mec.gnu@mindspring.com>
> 
> 	* gdb.asm/asm-source.exp: Return UNTESTED for platforms that
> 	have not implemented the assembly source test.
> 
> *** ORIGINAL-asm-source.exp	2003-12-18 01:15:58.000000000 -0500
> --- asm-source.exp	2003-12-18 01:16:07.000000000 -0500
> *************** switch -glob -- [istarget] {
> *** 115,121 ****
>   }
>   
>   if { "${asm-arch}" == "" } {
> !     gdb_suppress_entire_file "Assembly source test -- not implemented for this target."
>   }
>   
>   # On FreeBSD and NetBSD, crt1.o the final link will fail because of
> --- 115,122 ----
>   }
>   
>   if { "${asm-arch}" == "" } {
> !     untested "assembly source test not implemented for this target"
> !     return
>   }
>   
>   # On FreeBSD and NetBSD, crt1.o the final link will fail because of
> 

Re: [rfa/testsuite/PING] asm-source.exp: use UNTESTED

[Michael Elizabeth Chastain]

ac> However, remember why this test was originally changed to fail messy - 
ac> the test was being skipped and, as a demonstratable consequence, 
ac> everyone chose to ignore it rather than fix the testcase :-(

Well, I'm choosing to ignore it for native hppa2.0w-hp-hpux11.11,
until after function calls work, and C++ works, and the internal
hp_aCC_compiler variable works, and the other 200 ERRORs and
2000 FAILs have gotten some attention.

It's just annoying to open up gdb.log and the first 4 ERRORs and 5
WARNINGs are things that I know I won't fix this month.  I keep having
to look past this to get to more important problems.

ac> What about UNTESTED, and then KFAIL everything?  Knowing my luck that 
ac> will be much harder than it seems ...

That's what gdb_suppress_entire_file is supposed to do now, but it
doesn't work.  It sets a little state variable that is supposed to FAIL
every test in the current file.  But I still get a slow of
ERROR/WARNING on top of the 28 FAILs.

If I can't change asm-source.exp, I guess I'll just keep ignoring
the results by hand.  At least it's not incurring timeouts.

Michael C

Re: [rfa/testsuite/PING] asm-source.exp: use UNTESTED

[Andrew Cagney]

> ac> However, remember why this test was originally changed to fail messy - 
> ac> the test was being skipped and, as a demonstratable consequence, 
> ac> everyone chose to ignore it rather than fix the testcase :-(
> 
> Well, I'm choosing to ignore it for native hppa2.0w-hp-hpux11.11,
> until after function calls work, and C++ works, and the internal
> hp_aCC_compiler variable works, and the other 200 ERRORs and
> 2000 FAILs have gotten some attention.
> 
> It's just annoying to open up gdb.log and the first 4 ERRORs and 5
> WARNINGs are things that I know I won't fix this month.  I keep having
> to look past this to get to more important problems.
> 
> ac> What about UNTESTED, and then KFAIL everything?  Knowing my luck that 
> ac> will be much harder than it seems ...

Prior to "gdb_supress_entire_file", what was this test reporting when 
the code was missing?

> That's what gdb_suppress_entire_file is supposed to do now, but it
> doesn't work.  It sets a little state variable that is supposed to FAIL
> every test in the current file.  But I still get a slow of
> ERROR/WARNING on top of the 28 FAILs.
> 
> If I can't change asm-source.exp, I guess I'll just keep ignoring
> the results by hand.  At least it's not incurring timeouts.

Andrew