[RFA/mi-testsuite] XFAIL mi*-console.exp

[RFA/mi-testsuite] XFAIL mi*-console.exp

[Daniel Jacobowitz]

These tests are testing for a feature that exists either nowhere or just in
simulators and some remote stubs: that the inferior's output goes through
GDB and is properly encoded by the MI layer.  Since support isn't there for
many remote debugging stubs or for native, I think these two tests should be
XFAIL'd.  Does that make sense, Andrew?  If so, OK to commit this?

-- 
Daniel Jacobowitz                           Carnegie Mellon University
MontaVista Software                         Debian GNU/Linux Developer

2002-04-02  Daniel Jacobowitz  <drow@mvista.com>

	* gdb.mi/mi-console.exp: Accept lack of MI-console output as an
	XFAIL.
	* gdb.mi/mi0-console.exp: Likewise.

Index: testsuite/gdb.mi/mi-console.exp
===================================================================
RCS file: /cvs/src/src/gdb/testsuite/gdb.mi/mi-console.exp,v
retrieving revision 1.7
diff -u -p -r1.7 mi-console.exp
--- mi-console.exp	2001/08/19 01:23:43	1.7
+++ mi-console.exp	2002/04/03 00:38:09
@@ -92,7 +92,10 @@ gdb_expect {
 	# multiple event sources to channel the output back through the
 	# MI.
 
-	fail "Hello message (known bug)"
+	xfail "Hello message (known limitation)"
+    }
+    -notransfer -re "\r\n$mi_gdb_prompt" {
+	xfail "Hello message (no output)"
     }
     timeout {
 	fail "Hello message (timeout)"
Index: testsuite/gdb.mi/mi0-console.exp
===================================================================
RCS file: /cvs/src/src/gdb/testsuite/gdb.mi/mi0-console.exp,v
retrieving revision 1.5
diff -u -p -r1.5 mi0-console.exp
--- mi0-console.exp	2001/08/19 01:23:43	1.5
+++ mi0-console.exp	2002/04/03 00:38:09
@@ -92,7 +92,10 @@ gdb_expect {
 	# multiple event sources to channel the output back through the
 	# MI.
 
-	fail "Hello message (known bug)"
+	xfail "Hello message (known limitation)"
+    }
+    -notransfer -re "\r\n$mi_gdb_prompt" {
+	xfail "Hello message (no output)"
     }
     timeout {
 	fail "Hello message (timeout)"

Re: [RFA/mi-testsuite] XFAIL mi*-console.exp

[Michael Snyder]

Daniel Jacobowitz wrote:
> 
> These tests are testing for a feature that exists either nowhere or just in
> simulators and some remote stubs: that the inferior's output goes through
> GDB and is properly encoded by the MI layer.  Since support isn't there for
> many remote debugging stubs or for native, I think these two tests should be
> XFAIL'd.  Does that make sense, Andrew?  If so, OK to commit this?

There is a dejagnu variable that you can use to see
whether this is supported... lemme see...

Ah -- here you go.  You want to do something like the following:

	if { ![gdb_skip_stdio_test "Hello message"] } then {
		do the "hello message" test...


> 
> --
> Daniel Jacobowitz                           Carnegie Mellon University
> MontaVista Software                         Debian GNU/Linux Developer
> 
> 2002-04-02  Daniel Jacobowitz  <drow@mvista.com>
> 
>         * gdb.mi/mi-console.exp: Accept lack of MI-console output as an
>         XFAIL.
>         * gdb.mi/mi0-console.exp: Likewise.
> 
> Index: testsuite/gdb.mi/mi-console.exp
> ===================================================================
> RCS file: /cvs/src/src/gdb/testsuite/gdb.mi/mi-console.exp,v
> retrieving revision 1.7
> diff -u -p -r1.7 mi-console.exp
> --- mi-console.exp      2001/08/19 01:23:43     1.7
> +++ mi-console.exp      2002/04/03 00:38:09
> @@ -92,7 +92,10 @@ gdb_expect {
>         # multiple event sources to channel the output back through the
>         # MI.
> 
> -       fail "Hello message (known bug)"
> +       xfail "Hello message (known limitation)"
> +    }
> +    -notransfer -re "\r\n$mi_gdb_prompt" {
> +       xfail "Hello message (no output)"
>      }
>      timeout {
>         fail "Hello message (timeout)"
> Index: testsuite/gdb.mi/mi0-console.exp
> ===================================================================
> RCS file: /cvs/src/src/gdb/testsuite/gdb.mi/mi0-console.exp,v
> retrieving revision 1.5
> diff -u -p -r1.5 mi0-console.exp
> --- mi0-console.exp     2001/08/19 01:23:43     1.5
> +++ mi0-console.exp     2002/04/03 00:38:09
> @@ -92,7 +92,10 @@ gdb_expect {
>         # multiple event sources to channel the output back through the
>         # MI.
> 
> -       fail "Hello message (known bug)"
> +       xfail "Hello message (known limitation)"
> +    }
> +    -notransfer -re "\r\n$mi_gdb_prompt" {
> +       xfail "Hello message (no output)"
>      }
>      timeout {
>         fail "Hello message (timeout)"

Re: [RFA/mi-testsuite] XFAIL mi*-console.exp

[Daniel Jacobowitz]

On Tue, Apr 02, 2002 at 04:40:38PM -0800, Michael Snyder wrote:
> Daniel Jacobowitz wrote:
> > 
> > These tests are testing for a feature that exists either nowhere or just in
> > simulators and some remote stubs: that the inferior's output goes through
> > GDB and is properly encoded by the MI layer.  Since support isn't there for
> > many remote debugging stubs or for native, I think these two tests should be
> > XFAIL'd.  Does that make sense, Andrew?  If so, OK to commit this?
> 
> There is a dejagnu variable that you can use to see
> whether this is supported... lemme see...
> 
> Ah -- here you go.  You want to do something like the following:
> 
> 	if { ![gdb_skip_stdio_test "Hello message"] } then {
> 		do the "hello message" test...

Isn't gdb_skip_stdio_test for things where there's no way at all to see
the output?  grep, grep,...
    if [target_info exists gdb,noinferiorio] {

That's not quite the same thing.  You can run stdio test on a native
target, and (while GDB never sees the output) DejaGNU will.  I could
do this instead, though... revised patch attached.

-- 
Daniel Jacobowitz                           Carnegie Mellon University
MontaVista Software                         Debian GNU/Linux Developer

2002-04-02  Daniel Jacobowitz  <drow@mvista.com>

        * gdb.mi/mi-console.exp: Accept native console output as an
        XFAIL.  Don't run the test if we don't have inferior IO.
        * gdb.mi/mi0-console.exp: Likewise.

Index: testsuite/gdb.mi/mi-console.exp
===================================================================
RCS file: /cvs/src/src/gdb/testsuite/gdb.mi/mi-console.exp,v
retrieving revision 1.7
diff -u -p -r1.7 mi-console.exp
--- mi-console.exp	2001/08/19 01:23:43	1.7
+++ mi-console.exp	2002/04/03 01:00:34
@@ -79,7 +79,8 @@ gdb_expect {
     }
 }
 
-gdb_expect {
+if {![gdb_skip_stdio_test "Hello message"]} {
+  gdb_expect {
     -re "@\"H\"\r\n.*@\"e\"\r\n.*@\"l\"\r\n.*@\"l\"\r\n.*@\"o\"\r\n.*@\" \"\r\n.*@\"\\\\\\\\\"\r\n.*@\"\\\\\"\"\r\n.*@\"!\"\r\n.*@\"\\\\r\"\r\n.*@\"\\\\n\"\r\n" {
 	pass "Hello message"
     }
@@ -92,11 +93,12 @@ gdb_expect {
 	# multiple event sources to channel the output back through the
 	# MI.
 
-	fail "Hello message (known bug)"
+	xfail "Hello message (known limitation)"
     }
     timeout {
 	fail "Hello message (timeout)"
     }
+  }
 }
     
 gdb_expect {
Index: testsuite/gdb.mi/mi0-console.exp
===================================================================
RCS file: /cvs/src/src/gdb/testsuite/gdb.mi/mi0-console.exp,v
retrieving revision 1.5
diff -u -p -r1.5 mi0-console.exp
--- mi0-console.exp	2001/08/19 01:23:43	1.5
+++ mi0-console.exp	2002/04/03 01:00:34
@@ -79,7 +79,8 @@ gdb_expect {
     }
 }
 
-gdb_expect {
+if {![gdb_skip_stdio_test "Hello message"]} {
+  gdb_expect {
     -re "@\"H\"\r\n.*@\"e\"\r\n.*@\"l\"\r\n.*@\"l\"\r\n.*@\"o\"\r\n.*@\" \"\r\n.*@\"\\\\\\\\\"\r\n.*@\"\\\\\"\"\r\n.*@\"!\"\r\n.*@\"\\\\r\"\r\n.*@\"\\\\n\"\r\n" {
 	pass "Hello message"
     }
@@ -92,11 +93,12 @@ gdb_expect {
 	# multiple event sources to channel the output back through the
 	# MI.
 
-	fail "Hello message (known bug)"
+	xfail "Hello message (known limitation)"
     }
     timeout {
 	fail "Hello message (timeout)"
     }
+  }
 }
     
 gdb_expect {

Re: [RFA/mi-testsuite] XFAIL mi*-console.exp

[Michael Snyder]

Daniel Jacobowitz wrote:
> 
> On Tue, Apr 02, 2002 at 04:40:38PM -0800, Michael Snyder wrote:
> > Daniel Jacobowitz wrote:
> > >
> > > These tests are testing for a feature that exists either nowhere or just in
> > > simulators and some remote stubs: that the inferior's output goes through
> > > GDB and is properly encoded by the MI layer.  Since support isn't there for
> > > many remote debugging stubs or for native, I think these two tests should be
> > > XFAIL'd.  Does that make sense, Andrew?  If so, OK to commit this?
> >
> > There is a dejagnu variable that you can use to see
> > whether this is supported... lemme see...
> >
> > Ah -- here you go.  You want to do something like the following:
> >
> >       if { ![gdb_skip_stdio_test "Hello message"] } then {
> >               do the "hello message" test...
> 
> Isn't gdb_skip_stdio_test for things where there's no way at all to see
> the output?  grep, grep,...
>     if [target_info exists gdb,noinferiorio] {

Well, yes... and that's only going to be true for remote stubs
and simulators that don't support sending target output thru the
gdb console.  Isn't that what you want?  I'm not familiar with
these tests.

Do you also need to exclude it from running on natives?
How about
	if { ![isnative && ![gdb_skip_stdio_test ...


> 
> That's not quite the same thing.  You can run stdio test on a native
> target, and (while GDB never sees the output) DejaGNU will.  I could
> do this instead, though... revised patch attached.
> 
> --
> Daniel Jacobowitz                           Carnegie Mellon University
> MontaVista Software                         Debian GNU/Linux Developer
> 
> 2002-04-02  Daniel Jacobowitz  <drow@mvista.com>
> 
>         * gdb.mi/mi-console.exp: Accept native console output as an
>         XFAIL.  Don't run the test if we don't have inferior IO.
>         * gdb.mi/mi0-console.exp: Likewise.
> 
> Index: testsuite/gdb.mi/mi-console.exp
> ===================================================================
> RCS file: /cvs/src/src/gdb/testsuite/gdb.mi/mi-console.exp,v
> retrieving revision 1.7
> diff -u -p -r1.7 mi-console.exp
> --- mi-console.exp      2001/08/19 01:23:43     1.7
> +++ mi-console.exp      2002/04/03 01:00:34
> @@ -79,7 +79,8 @@ gdb_expect {
>      }
>  }
> 
> -gdb_expect {
> +if {![gdb_skip_stdio_test "Hello message"]} {
> +  gdb_expect {
>      -re "@\"H\"\r\n.*@\"e\"\r\n.*@\"l\"\r\n.*@\"l\"\r\n.*@\"o\"\r\n.*@\" \"\r\n.*@\"\\\\\\\\\"\r\n.*@\"\\\\\"\"\r\n.*@\"!\"\r\n.*@\"\\\\r\"\r\n.*@\"\\\\n\"\r\n" {
>         pass "Hello message"
>      }
> @@ -92,11 +93,12 @@ gdb_expect {
>         # multiple event sources to channel the output back through the
>         # MI.
> 
> -       fail "Hello message (known bug)"
> +       xfail "Hello message (known limitation)"
>      }
>      timeout {
>         fail "Hello message (timeout)"
>      }
> +  }
>  }
> 
>  gdb_expect {
> Index: testsuite/gdb.mi/mi0-console.exp
> ===================================================================
> RCS file: /cvs/src/src/gdb/testsuite/gdb.mi/mi0-console.exp,v
> retrieving revision 1.5
> diff -u -p -r1.5 mi0-console.exp
> --- mi0-console.exp     2001/08/19 01:23:43     1.5
> +++ mi0-console.exp     2002/04/03 01:00:34
> @@ -79,7 +79,8 @@ gdb_expect {
>      }
>  }
> 
> -gdb_expect {
> +if {![gdb_skip_stdio_test "Hello message"]} {
> +  gdb_expect {
>      -re "@\"H\"\r\n.*@\"e\"\r\n.*@\"l\"\r\n.*@\"l\"\r\n.*@\"o\"\r\n.*@\" \"\r\n.*@\"\\\\\\\\\"\r\n.*@\"\\\\\"\"\r\n.*@\"!\"\r\n.*@\"\\\\r\"\r\n.*@\"\\\\n\"\r\n" {
>         pass "Hello message"
>      }
> @@ -92,11 +93,12 @@ gdb_expect {
>         # multiple event sources to channel the output back through the
>         # MI.
> 
> -       fail "Hello message (known bug)"
> +       xfail "Hello message (known limitation)"
>      }
>      timeout {
>         fail "Hello message (timeout)"
>      }
> +  }
>  }
> 
>  gdb_expect {

Re: [RFA/mi-testsuite] XFAIL mi*-console.exp

[Daniel Jacobowitz]

On Tue, Apr 02, 2002 at 05:10:49PM -0800, Michael Snyder wrote:
> Daniel Jacobowitz wrote:
> > 
> > On Tue, Apr 02, 2002 at 04:40:38PM -0800, Michael Snyder wrote:
> > > Daniel Jacobowitz wrote:
> > > >
> > > > These tests are testing for a feature that exists either nowhere or just in
> > > > simulators and some remote stubs: that the inferior's output goes through
> > > > GDB and is properly encoded by the MI layer.  Since support isn't there for
> > > > many remote debugging stubs or for native, I think these two tests should be
> > > > XFAIL'd.  Does that make sense, Andrew?  If so, OK to commit this?
> > >
> > > There is a dejagnu variable that you can use to see
> > > whether this is supported... lemme see...
> > >
> > > Ah -- here you go.  You want to do something like the following:
> > >
> > >       if { ![gdb_skip_stdio_test "Hello message"] } then {
> > >               do the "hello message" test...
> > 
> > Isn't gdb_skip_stdio_test for things where there's no way at all to see
> > the output?  grep, grep,...
> >     if [target_info exists gdb,noinferiorio] {
> 
> Well, yes... and that's only going to be true for remote stubs
> and simulators that don't support sending target output thru the
> gdb console.  Isn't that what you want?  I'm not familiar with
> these tests.
> 
> Do you also need to exclude it from running on natives?
> How about
> 	if { ![isnative && ![gdb_skip_stdio_test ...

The test is checking that output passes through GDB on its way to the
viewer's eye, basically - checks that MI has the chance to get at it
and quote it up all properly.

Remote stubs that pass output through GDB support this, but native
programs are run on gdb's stdout directly at present.  There's no
reason it shouldn't work, though - it just doesn't right now.  That's
why I'd rather XFAIL than skip the tests.

-- 
Daniel Jacobowitz                           Carnegie Mellon University
MontaVista Software                         Debian GNU/Linux Developer

Re: [RFA/mi-testsuite] XFAIL mi*-console.exp

[Fernando Nasser]

Daniel Jacobowitz wrote:
> 
> That's
> why I'd rather XFAIL than skip the tests.
> 

It would be "unsupported" (which means that some feature is not
supported in this platform and so we can't run the test).

Cheers,
Fernando


-- 
Fernando Nasser
Red Hat Canada Ltd.                     E-Mail:  fnasser@redhat.com
2323 Yonge Street, Suite #300
Toronto, Ontario   M4P 2C9

Re: [RFA/mi-testsuite] XFAIL mi*-console.exp

[Andrew Cagney]

> These tests are testing for a feature that exists either nowhere or just in
> simulators and some remote stubs: that the inferior's output goes through
> GDB and is properly encoded by the MI layer.  Since support isn't there for
> many remote debugging stubs or for native, I think these two tests should be
> XFAIL'd.  Does that make sense, Andrew?  If so, OK to commit this?

I believe GDB's rule for XFAIL is something that can't work (due to an 
external constraint) rather than doesn't work (due to a lack of code).

Hence it was marked as a known bug rather than a limitation.

Andrew

Re: [RFA/mi-testsuite] XFAIL mi*-console.exp

[Daniel Jacobowitz]

On Wed, Apr 03, 2002 at 11:27:13PM -0500, Andrew Cagney wrote:
> >These tests are testing for a feature that exists either nowhere or just in
> >simulators and some remote stubs: that the inferior's output goes through
> >GDB and is properly encoded by the MI layer.  Since support isn't there for
> >many remote debugging stubs or for native, I think these two tests should 
> >be
> >XFAIL'd.  Does that make sense, Andrew?  If so, OK to commit this?
> 
> I believe GDB's rule for XFAIL is something that can't work (due to an 
> external constraint) rather than doesn't work (due to a lack of code).
> 
> Hence it was marked as a known bug rather than a limitation.

OK, so it isn't an XFAIL.  I don't think FAIL is really appropriate
either; tests which test a not-yet-implemented feature (and one that I
think is a bad idea, for native targets, to be honest) don't add any
information by failing.  UNSUPPORTED perhaps?  Or just not running the
test in native setups, for now?

-- 
Daniel Jacobowitz                           Carnegie Mellon University
MontaVista Software                         Debian GNU/Linux Developer

Re: [RFA/mi-testsuite] XFAIL mi*-console.exp

[Fernando Nasser]

I think you all missed this one...

Fernando Nasser wrote:
> 
> Daniel Jacobowitz wrote:
> >
> > That's
> > why I'd rather XFAIL than skip the tests.
> >
> 
> It would be "unsupported" (which means that some feature is not
> supported in this platform and so we can't run the test).
> 
> Cheers,
> Fernando
> 

-- 
Fernando Nasser
Red Hat Canada Ltd.                     E-Mail:  fnasser@redhat.com
2323 Yonge Street, Suite #300
Toronto, Ontario   M4P 2C9

Re: [RFA/mi-testsuite] XFAIL mi*-console.exp

[Daniel Jacobowitz]

On Thu, Apr 04, 2002 at 10:12:41AM -0500, Fernando Nasser wrote:
> I think you all missed this one...
> 
> Fernando Nasser wrote:
> > 
> > Daniel Jacobowitz wrote:
> > >
> > > That's
> > > why I'd rather XFAIL than skip the tests.
> > >
> > 
> > It would be "unsupported" (which means that some feature is not
> > supported in this platform and so we can't run the test).

Probably because we've had this conversation a half-dozen times and I
can never remember what we agreed on.  This seems unintuitive to me
because:
`UNSUPPORTED'
     There is no support for the tested case.  This may mean that a
     conditional feature of an operating system, or of a compiler, is
     not implemented.  DejaGnu also uses this message when a testing
     environment (often a "bare board" target) lacks basic support for
     compiling or running the test case.  For example, a test for the
     system subroutine `gethostname' would never work on a target board
     running only a boot monitor.

This is not a case of the test being inherently unsupported by the
system; it's a case of the code not being there to do it.  Compare to:

`XFAIL'
     A test failed, but it was expected to fail.  This result indicates
     no change in a known bug.  If a test fails because the operating
     system where the test runs lacks some facility required by the
     test, the outcome is `UNSUPPORTED' instead.


That's not how GDB is using XFAIL, although it does seem to be how
we're using UNSUPPORTED (more or less, sometimes less).

In any case, how's this?  It doesn't mark it unsupported and skip the
test; instead it detects the failure which means "no gdb code to
support this" and returns UNSUPPORTED.

-- 
Daniel Jacobowitz                           Carnegie Mellon University
MontaVista Software                         Debian GNU/Linux Developer

2002-04-04  Daniel Jacobowitz  <drow@mvista.com>

	* gdb.mi/mi-console.exp: Accept native console output as an
	XFAIL.  Don't run the test if we don't have inferior IO.
	* gdb.mi/mi0-console.exp: Likewise.

Index: testsuite/gdb.mi/mi-console.exp
===================================================================
RCS file: /cvs/src/src/gdb/testsuite/gdb.mi/mi-console.exp,v
retrieving revision 1.7
diff -u -p -r1.7 mi-console.exp
--- mi-console.exp	2001/08/19 01:23:43	1.7
+++ mi-console.exp	2002/04/03 01:00:34
@@ -79,7 +79,8 @@ gdb_expect {
     }
 }
 
-gdb_expect {
+if {![gdb_skip_stdio_test "Hello message"]} {
+  gdb_expect {
     -re "@\"H\"\r\n.*@\"e\"\r\n.*@\"l\"\r\n.*@\"l\"\r\n.*@\"o\"\r\n.*@\" \"\r\n.*@\"\\\\\\\\\"\r\n.*@\"\\\\\"\"\r\n.*@\"!\"\r\n.*@\"\\\\r\"\r\n.*@\"\\\\n\"\r\n" {
 	pass "Hello message"
     }
@@ -92,11 +93,12 @@ gdb_expect {
 	# multiple event sources to channel the output back through the
 	# MI.
 
-	fail "Hello message (known bug)"
+	unsupported "Hello message (known limitation)"
     }
     timeout {
 	fail "Hello message (timeout)"
     }
+  }
 }
     
 gdb_expect {
Index: testsuite/gdb.mi/mi0-console.exp
===================================================================
RCS file: /cvs/src/src/gdb/testsuite/gdb.mi/mi0-console.exp,v
retrieving revision 1.5
diff -u -p -r1.5 mi0-console.exp
--- mi0-console.exp	2001/08/19 01:23:43	1.5
+++ mi0-console.exp	2002/04/03 01:00:34
@@ -79,7 +79,8 @@ gdb_expect {
     }
 }
 
-gdb_expect {
+if {![gdb_skip_stdio_test "Hello message"]} {
+  gdb_expect {
     -re "@\"H\"\r\n.*@\"e\"\r\n.*@\"l\"\r\n.*@\"l\"\r\n.*@\"o\"\r\n.*@\" \"\r\n.*@\"\\\\\\\\\"\r\n.*@\"\\\\\"\"\r\n.*@\"!\"\r\n.*@\"\\\\r\"\r\n.*@\"\\\\n\"\r\n" {
 	pass "Hello message"
     }
@@ -92,11 +93,12 @@ gdb_expect {
 	# multiple event sources to channel the output back through the
 	# MI.
 
-	fail "Hello message (known bug)"
+	unsupported "Hello message (known limitation)"
     }
     timeout {
 	fail "Hello message (timeout)"
     }
+  }
 }
     
 gdb_expect {

Re: [RFA/mi-testsuite] XFAIL mi*-console.exp

[Andrew Cagney]

> I think you all missed this one...
> 
> Fernando Nasser wrote:
> 
>> 
>> Daniel Jacobowitz wrote:
> 
>> >
>> > That's
>> > why I'd rather XFAIL than skip the tests.
>> >
> 
>> 
>> It would be "unsupported" (which means that some feature is not
>> supported in this platform and so we can't run the test).

Ah, here it is "unimplemented" :-)  The platform does have the 
mechanisms needed to implement this, just not the code.

--

Aside, fernando and I had a brief discussion about xfail vs unsupported 
and came up with the following concrete example.

Attach/detach:

FreeBSD has a bug in its detach, since at present it doesn't work but 
did in the past, and will again in the next release it will work, it 
gets marked as ``xfail'.  Next release it will mysteriously ``xpass'' 
and can be adjusted accordingly.

Cygwin, due to limitations in the underlying OS, simply wasn't able to 
support detach, it should be marked as ``unsupported''.  (As a foot 
note, recent versions of the underlying OS, did fix this limitation).

Andrew

Re: [RFA/mi-testsuite] XFAIL mi*-console.exp

[Andrew Cagney]

> On Wed, Apr 03, 2002 at 11:27:13PM -0500, Andrew Cagney wrote:
> 
>> >These tests are testing for a feature that exists either nowhere or just in
>> >simulators and some remote stubs: that the inferior's output goes through
>> >GDB and is properly encoded by the MI layer.  Since support isn't there for
>> >many remote debugging stubs or for native, I think these two tests should 
>> >be
>> >XFAIL'd.  Does that make sense, Andrew?  If so, OK to commit this?
> 
>> 
>> I believe GDB's rule for XFAIL is something that can't work (due to an 
>> external constraint) rather than doesn't work (due to a lack of code).
>> 
>> Hence it was marked as a known bug rather than a limitation.
> 
> 
> OK, so it isn't an XFAIL.  I don't think FAIL is really appropriate
> either; tests which test a not-yet-implemented feature (and one that I
> think is a bad idea, for native targets, to be honest) don't add any
> information by failing.  UNSUPPORTED perhaps?  Or just not running the
> test in native setups, for now?
Er, actually, XFAIL might be closer to the truth than UNSUPPORTED. 
Although neither indicate UNIMPLEMENTED.

Andrew

> Aside, fernando and I had a brief discussion about xfail vs unsupported and came up with the following concrete example.
> 
> Attach/detach:
> 
> FreeBSD has a bug in its detach, since at present it doesn't work but did in the past, and will again in the next release it will work, it gets marked as ``xfail'.  Next release it will mysteriously ``xpass'' and can be adjusted accordingly.
> 
> Cygwin, due to limitations in the underlying OS, simply wasn't able to support detach, it should be marked as ``unsupported''.  (As a foot note, recent versions of the underlying OS, did fix this limitation).

Re: [RFA/mi-testsuite] XFAIL mi*-console.exp

[Daniel Jacobowitz]

On Thu, Apr 04, 2002 at 04:49:54PM -0500, Andrew Cagney wrote:
> >OK, so it isn't an XFAIL.  I don't think FAIL is really appropriate
> >either; tests which test a not-yet-implemented feature (and one that I
> >think is a bad idea, for native targets, to be honest) don't add any
> >information by failing.  UNSUPPORTED perhaps?  Or just not running the
> >test in native setups, for now?
> Er, actually, XFAIL might be closer to the truth than UNSUPPORTED. 
> Although neither indicate UNIMPLEMENTED.

I'm just going to sit on this patch until we can decide what the result
should be.  I've posted both XFAIL and UNSUPPORTED versions for your
viewing pleasure...

> 
> Andrew
> 
> >Aside, fernando and I had a brief discussion about xfail vs unsupported and 
> >came up with the following concrete example.
> >
> >Attach/detach:
> >
> >FreeBSD has a bug in its detach, since at present it doesn't work but did 
> >in the past, and will again in the next release it will work, it gets 
> >marked as ``xfail'.  Next release it will mysteriously ``xpass'' and can be 
> >adjusted accordingly.
> >
> >Cygwin, due to limitations in the underlying OS, simply wasn't able to 
> >support detach, it should be marked as ``unsupported''.  (As a foot note, 
> >recent versions of the underlying OS, did fix this limitation).
> 
> 
> 

-- 
Daniel Jacobowitz                           Carnegie Mellon University
MontaVista Software                         Debian GNU/Linux Developer

Re: [RFA/mi-testsuite] XFAIL mi*-console.exp

[Fernando Nasser]

Andrew Cagney wrote:
> 
> > On Wed, Apr 03, 2002 at 11:27:13PM -0500, Andrew Cagney wrote:
> >
> >> >These tests are testing for a feature that exists either nowhere or just in
> >> >simulators and some remote stubs: that the inferior's output goes through
> >> >GDB and is properly encoded by the MI layer.  Since support isn't there for
> >> >many remote debugging stubs or for native, I think these two tests should
> >> >be
> >> >XFAIL'd.  Does that make sense, Andrew?  If so, OK to commit this?
> >
> >>
> >> I believe GDB's rule for XFAIL is something that can't work (due to an
> >> external constraint) rather than doesn't work (due to a lack of code).
> >>
> >> Hence it was marked as a known bug rather than a limitation.
> >
> >
> > OK, so it isn't an XFAIL.  I don't think FAIL is really appropriate
> > either; tests which test a not-yet-implemented feature (and one that I
> > think is a bad idea, for native targets, to be honest) don't add any
> > information by failing.  UNSUPPORTED perhaps?  Or just not running the
> > test in native setups, for now?
> Er, actually, XFAIL might be closer to the truth than UNSUPPORTED.
> Although neither indicate UNIMPLEMENTED.
> 

I see. It is not UNSUPPORTED because the environment have it, just some
gsb stubs (and native) do not handle it (i.e., our fault).

Well, as we don't have UNIMPLEMENTED, we are left with only 2 options:

I guess we would use XFAIL if we decide that this is a feature that
we should have and not having it is a failure that should be fixed.

Otherwise we can just skip the test where it can't run.

 

-- 
Fernando Nasser
Red Hat Canada Ltd.                     E-Mail:  fnasser@redhat.com
2323 Yonge Street, Suite #300
Toronto, Ontario   M4P 2C9

Re: [RFA/mi-testsuite] XFAIL mi*-console.exp

[Andrew Cagney]

> Andrew Cagney wrote:
> 
>> 
> 
>> > On Wed, Apr 03, 2002 at 11:27:13PM -0500, Andrew Cagney wrote:
>> >
> 
>> >> >These tests are testing for a feature that exists either nowhere or just in
>> >> >simulators and some remote stubs: that the inferior's output goes through
>> >> >GDB and is properly encoded by the MI layer.  Since support isn't there for
>> >> >many remote debugging stubs or for native, I think these two tests should
>> >> >be
>> >> >XFAIL'd.  Does that make sense, Andrew?  If so, OK to commit this?
> 
>> >
> 
>> >>
>> >> I believe GDB's rule for XFAIL is something that can't work (due to an
>> >> external constraint) rather than doesn't work (due to a lack of code).
>> >>
>> >> Hence it was marked as a known bug rather than a limitation.
> 
>> >
>> >
>> > OK, so it isn't an XFAIL.  I don't think FAIL is really appropriate
>> > either; tests which test a not-yet-implemented feature (and one that I
>> > think is a bad idea, for native targets, to be honest) don't add any
>> > information by failing.  UNSUPPORTED perhaps?  Or just not running the
>> > test in native setups, for now?
> 
>> Er, actually, XFAIL might be closer to the truth than UNSUPPORTED.
>> Although neither indicate UNIMPLEMENTED.
>> 
> 
> 
> I see. It is not UNSUPPORTED because the environment have it, just some
> gsb stubs (and native) do not handle it (i.e., our fault).
> 
> Well, as we don't have UNIMPLEMENTED, we are left with only 2 options:
> 
> I guess we would use XFAIL if we decide that this is a feature that
> we should have and not having it is a failure that should be fixed.

> Otherwise we can just skip the test where it can't run.

I don't think this one is an option.  The gdb.asm test was being skipped 
and as a consequence everyone ignored it.  Now that it fails, people are 
fixing it.

Given it is a GDB ``bug'', is the correct approach:

	set prms_id <i-can't-find-it>
	FAIL ...

Andrew

Re: [RFA/mi-testsuite] XFAIL mi*-console.exp

[Fernando Nasser]

Andrew Cagney wrote:
> 
> >
> > I see. It is not UNSUPPORTED because the environment have it, just some
> > gsb stubs (and native) do not handle it (i.e., our fault).
> >
> > Well, as we don't have UNIMPLEMENTED, we are left with only 2 options:
> >
> > I guess we would use XFAIL if we decide that this is a feature that
> > we should have and not having it is a failure that should be fixed.
> 
> > Otherwise we can just skip the test where it can't run.
> 
> I don't think this one is an option.  The gdb.asm test was being skipped
> and as a consequence everyone ignored it.  Now that it fails, people are
> fixing it.
> 
> Given it is a GDB ``bug'', is the correct approach:
> 
>         set prms_id <i-can't-find-it>
>         FAIL ...
> 

Lets do what Daniel suggested then, and XFAIL it.  We will soon be able
to make it into a KFAIL.  It would e important to have a bug database
entry for this though (do we have one already?).

-- 
Fernando Nasser
Red Hat Canada Ltd.                     E-Mail:  fnasser@redhat.com
2323 Yonge Street, Suite #300
Toronto, Ontario   M4P 2C9

Re: [RFA/mi-testsuite] XFAIL mi*-console.exp

[Andrew Cagney]

> On Thu, Apr 04, 2002 at 04:49:54PM -0500, Andrew Cagney wrote:
> 
>> >OK, so it isn't an XFAIL.  I don't think FAIL is really appropriate
>> >either; tests which test a not-yet-implemented feature (and one that I
>> >think is a bad idea, for native targets, to be honest) don't add any
>> >information by failing.  UNSUPPORTED perhaps?  Or just not running the
>> >test in native setups, for now?
> 
>> Er, actually, XFAIL might be closer to the truth than UNSUPPORTED. 
>> Although neither indicate UNIMPLEMENTED.
> 
> 
> I'm just going to sit on this patch until we can decide what the result
> should be.  I've posted both XFAIL and UNSUPPORTED versions for your
> viewing pleasure...

Suggest a bug report so it isn't forgotten.

Andrew

Re: [RFA/mi-testsuite] XFAIL mi*-console.exp

[Michael Elizabeth Chastain]

What's the status of KFAIL?

Michael C