[PATCH] use "verbose" for notification of not running a test

[PATCH] use "verbose" for notification of not running a test

[Fred Fish]

Other places in the testsuite where tests are not run based on the
target or it's capabilities use "verbose" to notify the user.  The
suppression mechanisms are generally reserved for short circuiting
future tests when a failure of some kind is observed when attempting a
test.


2003-11-29  Fred Fish  <fnf@redhat.com>

	* gdb.asm/asm-source.exp: Use "verbose" when not running
	assembly tests or not supporting multilib.


Index: gdb.asm/asm-source.exp
===================================================================
RCS file: /cvs/src/src/gdb/testsuite/gdb.asm/asm-source.exp,v
retrieving revision 1.44
diff -c -p -r1.44 asm-source.exp
*** gdb.asm/asm-source.exp	26 Nov 2003 01:17:16 -0000	1.44
--- gdb.asm/asm-source.exp	29 Nov 2003 22:22:01 -0000
*************** switch -glob -- [istarget] {
*** 111,117 ****
  }
  
  if { "${asm-arch}" == "" } {
!     gdb_suppress_entire_file "Assembly source test -- not implemented for this target."
  }
  
  # On FreeBSD and NetBSD, crt1.o the final link will fail because of
--- 111,118 ----
  }
  
  if { "${asm-arch}" == "" } {
!     verbose "Assembly source test -- not implemented for this target."
!     return;
  }
  
  # On FreeBSD and NetBSD, crt1.o the final link will fail because of
*************** set dest [target_info name]
*** 134,140 ****
  if [board_info $dest exists multilib_flags] {
  	set multilib_flags [board_info $dest multilib_flags]
  	if { "${multilib_flags}" != "" } {
! 	   gdb_suppress_entire_file "Assembly source test -- multilibs not supported by this test."
  	   return;
  	 }
  }
--- 135,141 ----
  if [board_info $dest exists multilib_flags] {
  	set multilib_flags [board_info $dest multilib_flags]
  	if { "${multilib_flags}" != "" } {
! 	   verbose "Assembly source test -- multilibs not supported by this test."
  	   return;
  	 }
  }

Re: [PATCH] use "verbose" for notification of not running a test

[Daniel Jacobowitz]

On Sat, Nov 29, 2003 at 03:24:54PM -0700, Fred Fish wrote:
> Other places in the testsuite where tests are not run based on the
> target or it's capabilities use "verbose" to notify the user.  The
> suppression mechanisms are generally reserved for short circuiting
> future tests when a failure of some kind is observed when attempting a
> test.
> 
> 
> 2003-11-29  Fred Fish  <fnf@redhat.com>
> 
> 	* gdb.asm/asm-source.exp: Use "verbose" when not running
> 	assembly tests or not supporting multilib.

Hmm... I'd rather not.  In general, I'm all in favor of destroying
calls to gdb_suppress_entire_file.  However, most other tests are
skipped because of some serious lack in GDB or the available OS
services, not because the test needs a quick port to that architecture. 
The asm-source.exp suppression was left messy to encourage people to
write the appropriate wrappers; it's really not hard to do.

-- 
Daniel Jacobowitz
MontaVista Software                         Debian GNU/Linux Developer

Re: [PATCH] use "verbose" for notification of not running a test

[Fred Fish]

> The asm-source.exp suppression was left messy to encourage people to
> write the appropriate wrappers; it's really not hard to do.

OK, I understand the reasoning for the first suppression.

The second part of the patch is to shut up the testsuite when it can't
handle multilibs, which from the comment in the test file looks like
it would take significant work to support and isn't just a "per
target" issue.  Perhaps in this case it is more appropriate to just
silently skip the test.

-Fred

Re: [PATCH] use "verbose" for notification of not running a test

[Andrew Cagney]

>> The asm-source.exp suppression was left messy to encourage people to
>> write the appropriate wrappers; it's really not hard to do.
> 
> 
> OK, I understand the reasoning for the first suppression.
> 
> The second part of the patch is to shut up the testsuite when it can't
> handle multilibs, which from the comment in the test file looks like
> it would take significant work to support and isn't just a "per
> target" issue.  Perhaps in this case it is more appropriate to just
> silently skip the test.

At present it prints:

WARNING: Assembly source test -- multilibs not supported by this test.

then skips everything.  That way the fact that the test was 
intentionally skipped is recorded in the test logs - makes comparing 
test results easier.

Andrew

(I just wish that the message was prefixed with the test case that it 
applied to)

Re: [PATCH] use "verbose" for notification of not running a test

[Fred Fish]

> At present it prints:
> 
> WARNING: Assembly source test -- multilibs not supported by this test.
> 
> then skips everything.  That way the fact that the test was 
> intentionally skipped is recorded in the test logs - makes comparing 
> test results easier.

I just thought it was inconsistent with all the other places that skip
tests due to something other than a failure in a prior test, compilation
failure, or some other failing event.  Here are some examples that have
"skip" in their message:

gdb.arch/altivec-abi.exp:    verbose "Skipping altivec abi tests."
gdb.arch/altivec-regs.exp:    verbose "Skipping altivec register tests."
gdb.arch/e500-abi.exp:    verbose "Skipping e500 abi tests."
gdb.arch/e500-regs.exp:    verbose "Skipping e500 register tests."
gdb.arch/gdb1291.exp:    verbose "Skipping SH backtrace tests."
gdb.arch/gdb1431.exp:    verbose "Skipping SH backtrace tests."
gdb.arch/i386-prologue.exp:    verbose "Skipping i386 prologue tests."
gdb.arch/i386-unwind.exp:    verbose "Skipping i386 unwinder tests."
gdb.base/a2-run.exp:    verbose "Skipping a2-run.exp because of noargs."
gdb.base/commands.exp:        verbose "Skipping progvar_simple_if_test because of noargs."
gdb.base/commands.exp:        verbose "Skipping progvar_simple_while_test because of noargs."
gdb.base/commands.exp:        verbose "Skipping progvar_simple_if_while_test because of noargs."
gdb.base/commands.exp:        verbose "Skipping if_while_breakpoint_command_test because of noargs."
gdb.base/commands.exp:        verbose "Skipping infrun_breakpoint_command_test because of noargs."
gdb.base/commands.exp:        verbose "Skipping breakpoint_command_test because of noargs."
gdb.base/commands.exp:        verbose "Skipping watchpoint_command_test because of noargs."
gdb.base/commands.exp:        verbose "Skipping test_command_prompt_position because of noargs."
gdb.base/commands.exp:        verbose "Skipping bp_deleted_in_command_test because of noargs."
gdb.base/commands.exp:        verbose "Skipping temporary_breakpoint_commands because of noargs."
gdb.base/fileio.exp:    verbose "Skipping fileio.exp because of no fileio capabilities."
gdb.base/interrupt.exp:    verbose "Skipping interrupt.exp because of nointerrupts."
gdb.base/interrupt.exp:    verbose "Skipping interrupt.exp because of noinferiorio."
gdb.base/overlays.exp:    verbose "Skipping overlay test -- not implemented for this target."
gdb.base/shreloc.exp:    verbose "test skipped - shared object files not supported by this target."
gdb.base/sigall.exp:    verbose "Skipping sigall.exp because of nosignals."
gdb.base/signals.exp:    verbose "Skipping signals.exp because of nosignals."
gdb.gdbtk/cpp_variable.exp:    verbose "No C++ support -- skipping test"
gdb.mi/mi2-syn-frame.exp:    verbose "Skipping mi-syn-frame.exp because of nosignals."
gdb.mi/mi-syn-frame.exp:    verbose "Skipping mi-syn-frame.exp because of nosignals."

-Fred

Re: [PATCH] use "verbose" for notification of not running a test

[Andrew Cagney]

>> At present it prints:
>> 
>> WARNING: Assembly source test -- multilibs not supported by this test.
>> 
>> then skips everything.  That way the fact that the test was 
>> intentionally skipped is recorded in the test logs - makes comparing 
>> test results easier.
> 
> 
> I just thought it was inconsistent with all the other places that skip
> tests due to something other than a failure in a prior test, compilation
> failure, or some other failing event.  Here are some examples that have
> "skip" in their message:
> 
> gdb.arch/altivec-abi.exp:    verbose "Skipping altivec abi tests."
> gdb.arch/altivec-regs.exp:    verbose "Skipping altivec register tests."
> gdb.arch/e500-abi.exp:    verbose "Skipping e500 abi tests."
> gdb.arch/e500-regs.exp:    verbose "Skipping e500 register tests."
> gdb.arch/gdb1291.exp:    verbose "Skipping SH backtrace tests."
> gdb.arch/gdb1431.exp:    verbose "Skipping SH backtrace tests."
> gdb.arch/i386-prologue.exp:    verbose "Skipping i386 prologue tests."
> gdb.arch/i386-unwind.exp:    verbose "Skipping i386 unwinder tests."

Not to worried by the above :-)

> gdb.base/a2-run.exp:    verbose "Skipping a2-run.exp because of noargs."

Michael, should these be "unsupported"?

Andrew

Re: [PATCH] use "verbose" for notification of not running a test

[Andrew Cagney]

>> gdb.arch/altivec-abi.exp:    verbose "Skipping altivec abi tests."
>> gdb.arch/altivec-regs.exp:    verbose "Skipping altivec register tests."
>> gdb.arch/e500-abi.exp:    verbose "Skipping e500 abi tests."
>> gdb.arch/e500-regs.exp:    verbose "Skipping e500 register tests."
>> gdb.arch/gdb1291.exp:    verbose "Skipping SH backtrace tests."
>> gdb.arch/gdb1431.exp:    verbose "Skipping SH backtrace tests."
>> gdb.arch/i386-prologue.exp:    verbose "Skipping i386 prologue tests."
>> gdb.arch/i386-unwind.exp:    verbose "Skipping i386 unwinder tests."
> 
> 
> Hmmm.  These are all conditioned on target architecture.
> 
> My first reaction is that it's good to have arch-specific tests.
> It would be silly to pretend that gdb has the same features on all
> architectures.  It's a debugger, not a web server or a text editor!
> 
> My second reaction is that when I run an sh-*-* test on native
> i686-pc-linux-gnu, I don't need to see an UNSUPPORTED about it.
> 
> So the existing "verbose ... / return" works for me.
> 
> 
>> gdb.base/a2-run.exp:    verbose "Skipping a2-run.exp because of noargs."
> 
> 
> Yes, I think this is a good place for UNSUPPORTED.
> 
> "noargs" is either a limitation of the test environment or of the
> toolchain.  If it's a limitation of the test environment, I think
> that UNSUPPORTED matches it.  If it's a limitation of the toolchain,
> then it's actually a bit of a cheat to set "noargs", but UNSUPPORTED
> would not be too bad.
> 
> A casual look at dejagnu/baseboards.exp and "noargs" makes me think
> that "noargs" is generally a property of the test board, not a
> property of the toolchain.
> 
> The dejagnu doco says:
> 
>   UNSUPPORTED
>   A test depends on a conditionally available feature that does not
>   exist (in the configured testing environment).  For example, you can
>   use this outcome to report on a test case that does not work on a
>   particular target because its operating system support does not
>   include a required subroutine.
> 
> To me, the things in gdb.arch are not "conditionally available" on
> non-matching targets; they are just plain unavailable.
> 
> And if the limit is in the toolchain rather than the testing
> environment, it is really a FAIL, not an UNSUPPORTED.  UNIMPLEMENTED
> would be more accurate but there is no such word, so I'm not too
> hurt by fudging UNSUPPORTED to mean "unsupported by the toolchain"
> as well as "unsupported by the test environment" from time to time.

Unimplemented is, in a sense, equivalent to KFAIL.  It fails because 
someone hasn't fixed it.

Here, though, its fuzzy.

For some baseboards it really can't be implemented - cases where the 
"osabi" (I use the term loosely given its a baseboard) doesn't define 
the the mechanism for passing argv to the starting program for others it 
may be possible, just that no one is motivated to do it.

However, UNSUPPORTED may be sufficient.  What ever :-)

Andrew

> What do other people think?
> 
> Michael C
> 

Re: [PATCH] use "verbose" for notification of not running a test

[Michael Elizabeth Chastain]

> gdb.arch/altivec-abi.exp:    verbose "Skipping altivec abi tests."
> gdb.arch/altivec-regs.exp:    verbose "Skipping altivec register tests."
> gdb.arch/e500-abi.exp:    verbose "Skipping e500 abi tests."
> gdb.arch/e500-regs.exp:    verbose "Skipping e500 register tests."
> gdb.arch/gdb1291.exp:    verbose "Skipping SH backtrace tests."
> gdb.arch/gdb1431.exp:    verbose "Skipping SH backtrace tests."
> gdb.arch/i386-prologue.exp:    verbose "Skipping i386 prologue tests."
> gdb.arch/i386-unwind.exp:    verbose "Skipping i386 unwinder tests."

Hmmm.  These are all conditioned on target architecture.

My first reaction is that it's good to have arch-specific tests.
It would be silly to pretend that gdb has the same features on all
architectures.  It's a debugger, not a web server or a text editor!

My second reaction is that when I run an sh-*-* test on native
i686-pc-linux-gnu, I don't need to see an UNSUPPORTED about it.

So the existing "verbose ... / return" works for me.

> gdb.base/a2-run.exp:    verbose "Skipping a2-run.exp because of noargs."

Yes, I think this is a good place for UNSUPPORTED.

"noargs" is either a limitation of the test environment or of the
toolchain.  If it's a limitation of the test environment, I think
that UNSUPPORTED matches it.  If it's a limitation of the toolchain,
then it's actually a bit of a cheat to set "noargs", but UNSUPPORTED
would not be too bad.

A casual look at dejagnu/baseboards.exp and "noargs" makes me think
that "noargs" is generally a property of the test board, not a
property of the toolchain.

The dejagnu doco says:

  UNSUPPORTED
  A test depends on a conditionally available feature that does not
  exist (in the configured testing environment).  For example, you can
  use this outcome to report on a test case that does not work on a
  particular target because its operating system support does not
  include a required subroutine.

To me, the things in gdb.arch are not "conditionally available" on
non-matching targets; they are just plain unavailable.

And if the limit is in the toolchain rather than the testing
environment, it is really a FAIL, not an UNSUPPORTED.  UNIMPLEMENTED
would be more accurate but there is no such word, so I'm not too
hurt by fudging UNSUPPORTED to mean "unsupported by the toolchain"
as well as "unsupported by the test environment" from time to time.

What do other people think?

Michael C

Re: [PATCH] use "verbose" for notification of not running a test

[Michael Elizabeth Chastain]

mec> How about making this an UNSUPPORTED?
ac> It is supported, it's just not being tested ... :-(

Whoops, you caught me!

UNSUPPORTED should always mean "unsupported in gdb",
but I was attempting to use it to mean "unsupported in the test suite".

gdb and the test suite are not symmetric at all; in fact they are
adverserial.

Michael C

Re: [PATCH] use "verbose" for notification of not running a test

[Andrew Cagney]

> ac> At present it prints:
> ac> 
> ac> WARNING: Assembly source test -- multilibs not supported by this test.
> ac> 
> ac> then skips everything.  That way the fact that the test was 
> ac> intentionally skipped is recorded in the test logs - makes comparing 
> ac> test results easier.
> 
> Except that dejagnu doesn't print ERROR and WARNING and NOTE counts
> in the totals at the end, so lots of people don't notice if they get
> one of those.  I wish these people would change, and I wish dejagnu
> would change, too.
> 
> How about making this an UNSUPPORTED?

It is supported, it's just not being tested ... :-(

Andrew

Re: [PATCH] use "verbose" for notification of not running a test

[Michael Elizabeth Chastain]

ac> At present it prints:
ac> 
ac> WARNING: Assembly source test -- multilibs not supported by this test.
ac> 
ac> then skips everything.  That way the fact that the test was 
ac> intentionally skipped is recorded in the test logs - makes comparing 
ac> test results easier.

Except that dejagnu doesn't print ERROR and WARNING and NOTE counts
in the totals at the end, so lots of people don't notice if they get
one of those.  I wish these people would change, and I wish dejagnu
would change, too.

How about making this an UNSUPPORTED?

Michael C