Re: [RFA/mi-testsuite] XFAIL mi*-console.exp

Re: [RFA/mi-testsuite] XFAIL mi*-console.exp

[Michael Elizabeth Chastain]

What's the status of KFAIL?

Michael C

RFC: KFAILs [Was: [RFA/mi-testsuite] XFAIL mi*-console.exp]

[Fernando Nasser]

Michael Elizabeth Chastain wrote:
> 
> What's the status of KFAIL?
> 

I have it here in my sandbox, but I will have to clean it up a
little bit this Sunday. I have a few questions and comments:


1) All current Dejagnu messages have the bug identification at
the end (between parenthesis).  I want to keep things consistent.
Is that OK?

2) Also, to keep things consistent in Dejagnu, the bug identification
for the kfail would be printed as "(PRMS xxxx)" as it is done
for XFAIL etc., to distinguish from bug_id which is printed as
"(BUG xxxx)".  Is that a problem?

So, a KFAIL would be printed as:

KFAIL: could not run to marker1 (PRMS gdb/999)

Would that make the scripts happy?


3) For KFAILs, the bug identification is mandatory.  In this case
I would like to change the syntax a little bit and have it as the
first argument and issue an error if it is not there.  I would 
still keep the current Dejagnu convention that bug identifications
are strings without a '-' in it.  I am doing this just to be able
to detect that someone forgot to specify it.  I know that there are
other syntax possibilities (like command switches), but I am trying
to be consistent with what is already there.

So, here is an example of a kfail use:

setup_kfail "gdb/999" *-*-*


4) Note that, when a test that was expected to fail due to a known
bug suddenly starts to pass, it becomes a KPASS (as XFAILs do).
It is then time to go and check if the bug was fixed and someone
forgot to remove the setup_kfail or if something changed in the
environment and the bug is not applicable anymore (or the test
is not catching it anymore).  More or less as we should do with
XPASSes that appear instead of XFAILs.  Also, sometimes the target
specification is a little bit too general (the problem does not
happen in all the specified conditions).


5) As soon as we have KFAILs, we will have to re-examine the FAILs
we have, register a bug for what is a gdb bug.  Also, we will be able
to have zero FAILs, so in any platform a non-zero result will mean that
we broke something.  We will also have to re-check everything that was
marked as XFAIL and see if they are actually known GDB bugs.



6) I would like to write a script that, as part of the release, goes 
through the KFAILs, access the Gnats database with the Bug Id and 
fills the BUGS part of the man page (can also generate a report and
put it in a file -- or append it to a Release Notes").  
We can also have a script to compare the new known bugs report with
the previous release one and generate a list of "Bugs fixed in this
release" (this can help people decide if they should upgrade, or stop
people to post just to hear: "I suggest you upgrade").

I will do it in Perl (I still don't know how to programmatically access
the Gnats database though).  But I have very little spare time, so I
will
not mind if someone that can do it sooner volunteers to do this.
 


-- 
Fernando Nasser
Red Hat Canada Ltd.                     E-Mail:  fnasser@redhat.com
2323 Yonge Street, Suite #300
Toronto, Ontario   M4P 2C9

[RFA/mi-testsuite] XFAIL mi*-console.exp

[Daniel Jacobowitz]

These tests are testing for a feature that exists either nowhere or just in
simulators and some remote stubs: that the inferior's output goes through
GDB and is properly encoded by the MI layer.  Since support isn't there for
many remote debugging stubs or for native, I think these two tests should be
XFAIL'd.  Does that make sense, Andrew?  If so, OK to commit this?

-- 
Daniel Jacobowitz                           Carnegie Mellon University
MontaVista Software                         Debian GNU/Linux Developer

2002-04-02  Daniel Jacobowitz  <drow@mvista.com>

	* gdb.mi/mi-console.exp: Accept lack of MI-console output as an
	XFAIL.
	* gdb.mi/mi0-console.exp: Likewise.

Index: testsuite/gdb.mi/mi-console.exp
===================================================================
RCS file: /cvs/src/src/gdb/testsuite/gdb.mi/mi-console.exp,v
retrieving revision 1.7
diff -u -p -r1.7 mi-console.exp
--- mi-console.exp	2001/08/19 01:23:43	1.7
+++ mi-console.exp	2002/04/03 00:38:09
@@ -92,7 +92,10 @@ gdb_expect {
 	# multiple event sources to channel the output back through the
 	# MI.
 
-	fail "Hello message (known bug)"
+	xfail "Hello message (known limitation)"
+    }
+    -notransfer -re "\r\n$mi_gdb_prompt" {
+	xfail "Hello message (no output)"
     }
     timeout {
 	fail "Hello message (timeout)"
Index: testsuite/gdb.mi/mi0-console.exp
===================================================================
RCS file: /cvs/src/src/gdb/testsuite/gdb.mi/mi0-console.exp,v
retrieving revision 1.5
diff -u -p -r1.5 mi0-console.exp
--- mi0-console.exp	2001/08/19 01:23:43	1.5
+++ mi0-console.exp	2002/04/03 00:38:09
@@ -92,7 +92,10 @@ gdb_expect {
 	# multiple event sources to channel the output back through the
 	# MI.
 
-	fail "Hello message (known bug)"
+	xfail "Hello message (known limitation)"
+    }
+    -notransfer -re "\r\n$mi_gdb_prompt" {
+	xfail "Hello message (no output)"
     }
     timeout {
 	fail "Hello message (timeout)"

Re: [RFA/mi-testsuite] XFAIL mi*-console.exp

[Michael Snyder]

Daniel Jacobowitz wrote:
> 
> These tests are testing for a feature that exists either nowhere or just in
> simulators and some remote stubs: that the inferior's output goes through
> GDB and is properly encoded by the MI layer.  Since support isn't there for
> many remote debugging stubs or for native, I think these two tests should be
> XFAIL'd.  Does that make sense, Andrew?  If so, OK to commit this?

There is a dejagnu variable that you can use to see
whether this is supported... lemme see...

Ah -- here you go.  You want to do something like the following:

	if { ![gdb_skip_stdio_test "Hello message"] } then {
		do the "hello message" test...


> 
> --
> Daniel Jacobowitz                           Carnegie Mellon University
> MontaVista Software                         Debian GNU/Linux Developer
> 
> 2002-04-02  Daniel Jacobowitz  <drow@mvista.com>
> 
>         * gdb.mi/mi-console.exp: Accept lack of MI-console output as an
>         XFAIL.
>         * gdb.mi/mi0-console.exp: Likewise.
> 
> Index: testsuite/gdb.mi/mi-console.exp
> ===================================================================
> RCS file: /cvs/src/src/gdb/testsuite/gdb.mi/mi-console.exp,v
> retrieving revision 1.7
> diff -u -p -r1.7 mi-console.exp
> --- mi-console.exp      2001/08/19 01:23:43     1.7
> +++ mi-console.exp      2002/04/03 00:38:09
> @@ -92,7 +92,10 @@ gdb_expect {
>         # multiple event sources to channel the output back through the
>         # MI.
> 
> -       fail "Hello message (known bug)"
> +       xfail "Hello message (known limitation)"
> +    }
> +    -notransfer -re "\r\n$mi_gdb_prompt" {
> +       xfail "Hello message (no output)"
>      }
>      timeout {
>         fail "Hello message (timeout)"
> Index: testsuite/gdb.mi/mi0-console.exp
> ===================================================================
> RCS file: /cvs/src/src/gdb/testsuite/gdb.mi/mi0-console.exp,v
> retrieving revision 1.5
> diff -u -p -r1.5 mi0-console.exp
> --- mi0-console.exp     2001/08/19 01:23:43     1.5
> +++ mi0-console.exp     2002/04/03 00:38:09
> @@ -92,7 +92,10 @@ gdb_expect {
>         # multiple event sources to channel the output back through the
>         # MI.
> 
> -       fail "Hello message (known bug)"
> +       xfail "Hello message (known limitation)"
> +    }
> +    -notransfer -re "\r\n$mi_gdb_prompt" {
> +       xfail "Hello message (no output)"
>      }
>      timeout {
>         fail "Hello message (timeout)"

Re: [RFA/mi-testsuite] XFAIL mi*-console.exp

[Daniel Jacobowitz]

On Tue, Apr 02, 2002 at 04:40:38PM -0800, Michael Snyder wrote:
> Daniel Jacobowitz wrote:
> > 
> > These tests are testing for a feature that exists either nowhere or just in
> > simulators and some remote stubs: that the inferior's output goes through
> > GDB and is properly encoded by the MI layer.  Since support isn't there for
> > many remote debugging stubs or for native, I think these two tests should be
> > XFAIL'd.  Does that make sense, Andrew?  If so, OK to commit this?
> 
> There is a dejagnu variable that you can use to see
> whether this is supported... lemme see...
> 
> Ah -- here you go.  You want to do something like the following:
> 
> 	if { ![gdb_skip_stdio_test "Hello message"] } then {
> 		do the "hello message" test...

Isn't gdb_skip_stdio_test for things where there's no way at all to see
the output?  grep, grep,...
    if [target_info exists gdb,noinferiorio] {

That's not quite the same thing.  You can run stdio test on a native
target, and (while GDB never sees the output) DejaGNU will.  I could
do this instead, though... revised patch attached.

-- 
Daniel Jacobowitz                           Carnegie Mellon University
MontaVista Software                         Debian GNU/Linux Developer

2002-04-02  Daniel Jacobowitz  <drow@mvista.com>

        * gdb.mi/mi-console.exp: Accept native console output as an
        XFAIL.  Don't run the test if we don't have inferior IO.
        * gdb.mi/mi0-console.exp: Likewise.

Index: testsuite/gdb.mi/mi-console.exp
===================================================================
RCS file: /cvs/src/src/gdb/testsuite/gdb.mi/mi-console.exp,v
retrieving revision 1.7
diff -u -p -r1.7 mi-console.exp
--- mi-console.exp	2001/08/19 01:23:43	1.7
+++ mi-console.exp	2002/04/03 01:00:34
@@ -79,7 +79,8 @@ gdb_expect {
     }
 }
 
-gdb_expect {
+if {![gdb_skip_stdio_test "Hello message"]} {
+  gdb_expect {
     -re "@\"H\"\r\n.*@\"e\"\r\n.*@\"l\"\r\n.*@\"l\"\r\n.*@\"o\"\r\n.*@\" \"\r\n.*@\"\\\\\\\\\"\r\n.*@\"\\\\\"\"\r\n.*@\"!\"\r\n.*@\"\\\\r\"\r\n.*@\"\\\\n\"\r\n" {
 	pass "Hello message"
     }
@@ -92,11 +93,12 @@ gdb_expect {
 	# multiple event sources to channel the output back through the
 	# MI.
 
-	fail "Hello message (known bug)"
+	xfail "Hello message (known limitation)"
     }
     timeout {
 	fail "Hello message (timeout)"
     }
+  }
 }
     
 gdb_expect {
Index: testsuite/gdb.mi/mi0-console.exp
===================================================================
RCS file: /cvs/src/src/gdb/testsuite/gdb.mi/mi0-console.exp,v
retrieving revision 1.5
diff -u -p -r1.5 mi0-console.exp
--- mi0-console.exp	2001/08/19 01:23:43	1.5
+++ mi0-console.exp	2002/04/03 01:00:34
@@ -79,7 +79,8 @@ gdb_expect {
     }
 }
 
-gdb_expect {
+if {![gdb_skip_stdio_test "Hello message"]} {
+  gdb_expect {
     -re "@\"H\"\r\n.*@\"e\"\r\n.*@\"l\"\r\n.*@\"l\"\r\n.*@\"o\"\r\n.*@\" \"\r\n.*@\"\\\\\\\\\"\r\n.*@\"\\\\\"\"\r\n.*@\"!\"\r\n.*@\"\\\\r\"\r\n.*@\"\\\\n\"\r\n" {
 	pass "Hello message"
     }
@@ -92,11 +93,12 @@ gdb_expect {
 	# multiple event sources to channel the output back through the
 	# MI.
 
-	fail "Hello message (known bug)"
+	xfail "Hello message (known limitation)"
     }
     timeout {
 	fail "Hello message (timeout)"
     }
+  }
 }
     
 gdb_expect {

Re: [RFA/mi-testsuite] XFAIL mi*-console.exp

[Michael Snyder]

Daniel Jacobowitz wrote:
> 
> On Tue, Apr 02, 2002 at 04:40:38PM -0800, Michael Snyder wrote:
> > Daniel Jacobowitz wrote:
> > >
> > > These tests are testing for a feature that exists either nowhere or just in
> > > simulators and some remote stubs: that the inferior's output goes through
> > > GDB and is properly encoded by the MI layer.  Since support isn't there for
> > > many remote debugging stubs or for native, I think these two tests should be
> > > XFAIL'd.  Does that make sense, Andrew?  If so, OK to commit this?
> >
> > There is a dejagnu variable that you can use to see
> > whether this is supported... lemme see...
> >
> > Ah -- here you go.  You want to do something like the following:
> >
> >       if { ![gdb_skip_stdio_test "Hello message"] } then {
> >               do the "hello message" test...
> 
> Isn't gdb_skip_stdio_test for things where there's no way at all to see
> the output?  grep, grep,...
>     if [target_info exists gdb,noinferiorio] {

Well, yes... and that's only going to be true for remote stubs
and simulators that don't support sending target output thru the
gdb console.  Isn't that what you want?  I'm not familiar with
these tests.

Do you also need to exclude it from running on natives?
How about
	if { ![isnative && ![gdb_skip_stdio_test ...


> 
> That's not quite the same thing.  You can run stdio test on a native
> target, and (while GDB never sees the output) DejaGNU will.  I could
> do this instead, though... revised patch attached.
> 
> --
> Daniel Jacobowitz                           Carnegie Mellon University
> MontaVista Software                         Debian GNU/Linux Developer
> 
> 2002-04-02  Daniel Jacobowitz  <drow@mvista.com>
> 
>         * gdb.mi/mi-console.exp: Accept native console output as an
>         XFAIL.  Don't run the test if we don't have inferior IO.
>         * gdb.mi/mi0-console.exp: Likewise.
> 
> Index: testsuite/gdb.mi/mi-console.exp
> ===================================================================
> RCS file: /cvs/src/src/gdb/testsuite/gdb.mi/mi-console.exp,v
> retrieving revision 1.7
> diff -u -p -r1.7 mi-console.exp
> --- mi-console.exp      2001/08/19 01:23:43     1.7
> +++ mi-console.exp      2002/04/03 01:00:34
> @@ -79,7 +79,8 @@ gdb_expect {
>      }
>  }
> 
> -gdb_expect {
> +if {![gdb_skip_stdio_test "Hello message"]} {
> +  gdb_expect {
>      -re "@\"H\"\r\n.*@\"e\"\r\n.*@\"l\"\r\n.*@\"l\"\r\n.*@\"o\"\r\n.*@\" \"\r\n.*@\"\\\\\\\\\"\r\n.*@\"\\\\\"\"\r\n.*@\"!\"\r\n.*@\"\\\\r\"\r\n.*@\"\\\\n\"\r\n" {
>         pass "Hello message"
>      }
> @@ -92,11 +93,12 @@ gdb_expect {
>         # multiple event sources to channel the output back through the
>         # MI.
> 
> -       fail "Hello message (known bug)"
> +       xfail "Hello message (known limitation)"
>      }
>      timeout {
>         fail "Hello message (timeout)"
>      }
> +  }
>  }
> 
>  gdb_expect {
> Index: testsuite/gdb.mi/mi0-console.exp
> ===================================================================
> RCS file: /cvs/src/src/gdb/testsuite/gdb.mi/mi0-console.exp,v
> retrieving revision 1.5
> diff -u -p -r1.5 mi0-console.exp
> --- mi0-console.exp     2001/08/19 01:23:43     1.5
> +++ mi0-console.exp     2002/04/03 01:00:34
> @@ -79,7 +79,8 @@ gdb_expect {
>      }
>  }
> 
> -gdb_expect {
> +if {![gdb_skip_stdio_test "Hello message"]} {
> +  gdb_expect {
>      -re "@\"H\"\r\n.*@\"e\"\r\n.*@\"l\"\r\n.*@\"l\"\r\n.*@\"o\"\r\n.*@\" \"\r\n.*@\"\\\\\\\\\"\r\n.*@\"\\\\\"\"\r\n.*@\"!\"\r\n.*@\"\\\\r\"\r\n.*@\"\\\\n\"\r\n" {
>         pass "Hello message"
>      }
> @@ -92,11 +93,12 @@ gdb_expect {
>         # multiple event sources to channel the output back through the
>         # MI.
> 
> -       fail "Hello message (known bug)"
> +       xfail "Hello message (known limitation)"
>      }
>      timeout {
>         fail "Hello message (timeout)"
>      }
> +  }
>  }
> 
>  gdb_expect {

Re: [RFA/mi-testsuite] XFAIL mi*-console.exp

[Daniel Jacobowitz]

On Tue, Apr 02, 2002 at 05:10:49PM -0800, Michael Snyder wrote:
> Daniel Jacobowitz wrote:
> > 
> > On Tue, Apr 02, 2002 at 04:40:38PM -0800, Michael Snyder wrote:
> > > Daniel Jacobowitz wrote:
> > > >
> > > > These tests are testing for a feature that exists either nowhere or just in
> > > > simulators and some remote stubs: that the inferior's output goes through
> > > > GDB and is properly encoded by the MI layer.  Since support isn't there for
> > > > many remote debugging stubs or for native, I think these two tests should be
> > > > XFAIL'd.  Does that make sense, Andrew?  If so, OK to commit this?
> > >
> > > There is a dejagnu variable that you can use to see
> > > whether this is supported... lemme see...
> > >
> > > Ah -- here you go.  You want to do something like the following:
> > >
> > >       if { ![gdb_skip_stdio_test "Hello message"] } then {
> > >               do the "hello message" test...
> > 
> > Isn't gdb_skip_stdio_test for things where there's no way at all to see
> > the output?  grep, grep,...
> >     if [target_info exists gdb,noinferiorio] {
> 
> Well, yes... and that's only going to be true for remote stubs
> and simulators that don't support sending target output thru the
> gdb console.  Isn't that what you want?  I'm not familiar with
> these tests.
> 
> Do you also need to exclude it from running on natives?
> How about
> 	if { ![isnative && ![gdb_skip_stdio_test ...

The test is checking that output passes through GDB on its way to the
viewer's eye, basically - checks that MI has the chance to get at it
and quote it up all properly.

Remote stubs that pass output through GDB support this, but native
programs are run on gdb's stdout directly at present.  There's no
reason it shouldn't work, though - it just doesn't right now.  That's
why I'd rather XFAIL than skip the tests.

-- 
Daniel Jacobowitz                           Carnegie Mellon University
MontaVista Software                         Debian GNU/Linux Developer

Re: [RFA/mi-testsuite] XFAIL mi*-console.exp

[Fernando Nasser]

Daniel Jacobowitz wrote:
> 
> That's
> why I'd rather XFAIL than skip the tests.
> 

It would be "unsupported" (which means that some feature is not
supported in this platform and so we can't run the test).

Cheers,
Fernando


-- 
Fernando Nasser
Red Hat Canada Ltd.                     E-Mail:  fnasser@redhat.com
2323 Yonge Street, Suite #300
Toronto, Ontario   M4P 2C9

Re: [RFA/mi-testsuite] XFAIL mi*-console.exp

[Fernando Nasser]

I think you all missed this one...

Fernando Nasser wrote:
> 
> Daniel Jacobowitz wrote:
> >
> > That's
> > why I'd rather XFAIL than skip the tests.
> >
> 
> It would be "unsupported" (which means that some feature is not
> supported in this platform and so we can't run the test).
> 
> Cheers,
> Fernando
> 

-- 
Fernando Nasser
Red Hat Canada Ltd.                     E-Mail:  fnasser@redhat.com
2323 Yonge Street, Suite #300
Toronto, Ontario   M4P 2C9

Re: [RFA/mi-testsuite] XFAIL mi*-console.exp

[Daniel Jacobowitz]

On Thu, Apr 04, 2002 at 10:12:41AM -0500, Fernando Nasser wrote:
> I think you all missed this one...
> 
> Fernando Nasser wrote:
> > 
> > Daniel Jacobowitz wrote:
> > >
> > > That's
> > > why I'd rather XFAIL than skip the tests.
> > >
> > 
> > It would be "unsupported" (which means that some feature is not
> > supported in this platform and so we can't run the test).

Probably because we've had this conversation a half-dozen times and I
can never remember what we agreed on.  This seems unintuitive to me
because:
`UNSUPPORTED'
     There is no support for the tested case.  This may mean that a
     conditional feature of an operating system, or of a compiler, is
     not implemented.  DejaGnu also uses this message when a testing
     environment (often a "bare board" target) lacks basic support for
     compiling or running the test case.  For example, a test for the
     system subroutine `gethostname' would never work on a target board
     running only a boot monitor.

This is not a case of the test being inherently unsupported by the
system; it's a case of the code not being there to do it.  Compare to:

`XFAIL'
     A test failed, but it was expected to fail.  This result indicates
     no change in a known bug.  If a test fails because the operating
     system where the test runs lacks some facility required by the
     test, the outcome is `UNSUPPORTED' instead.


That's not how GDB is using XFAIL, although it does seem to be how
we're using UNSUPPORTED (more or less, sometimes less).

In any case, how's this?  It doesn't mark it unsupported and skip the
test; instead it detects the failure which means "no gdb code to
support this" and returns UNSUPPORTED.

-- 
Daniel Jacobowitz                           Carnegie Mellon University
MontaVista Software                         Debian GNU/Linux Developer

2002-04-04  Daniel Jacobowitz  <drow@mvista.com>

	* gdb.mi/mi-console.exp: Accept native console output as an
	XFAIL.  Don't run the test if we don't have inferior IO.
	* gdb.mi/mi0-console.exp: Likewise.

Index: testsuite/gdb.mi/mi-console.exp
===================================================================
RCS file: /cvs/src/src/gdb/testsuite/gdb.mi/mi-console.exp,v
retrieving revision 1.7
diff -u -p -r1.7 mi-console.exp
--- mi-console.exp	2001/08/19 01:23:43	1.7
+++ mi-console.exp	2002/04/03 01:00:34
@@ -79,7 +79,8 @@ gdb_expect {
     }
 }
 
-gdb_expect {
+if {![gdb_skip_stdio_test "Hello message"]} {
+  gdb_expect {
     -re "@\"H\"\r\n.*@\"e\"\r\n.*@\"l\"\r\n.*@\"l\"\r\n.*@\"o\"\r\n.*@\" \"\r\n.*@\"\\\\\\\\\"\r\n.*@\"\\\\\"\"\r\n.*@\"!\"\r\n.*@\"\\\\r\"\r\n.*@\"\\\\n\"\r\n" {
 	pass "Hello message"
     }
@@ -92,11 +93,12 @@ gdb_expect {
 	# multiple event sources to channel the output back through the
 	# MI.
 
-	fail "Hello message (known bug)"
+	unsupported "Hello message (known limitation)"
     }
     timeout {
 	fail "Hello message (timeout)"
     }
+  }
 }
     
 gdb_expect {
Index: testsuite/gdb.mi/mi0-console.exp
===================================================================
RCS file: /cvs/src/src/gdb/testsuite/gdb.mi/mi0-console.exp,v
retrieving revision 1.5
diff -u -p -r1.5 mi0-console.exp
--- mi0-console.exp	2001/08/19 01:23:43	1.5
+++ mi0-console.exp	2002/04/03 01:00:34
@@ -79,7 +79,8 @@ gdb_expect {
     }
 }
 
-gdb_expect {
+if {![gdb_skip_stdio_test "Hello message"]} {
+  gdb_expect {
     -re "@\"H\"\r\n.*@\"e\"\r\n.*@\"l\"\r\n.*@\"l\"\r\n.*@\"o\"\r\n.*@\" \"\r\n.*@\"\\\\\\\\\"\r\n.*@\"\\\\\"\"\r\n.*@\"!\"\r\n.*@\"\\\\r\"\r\n.*@\"\\\\n\"\r\n" {
 	pass "Hello message"
     }
@@ -92,11 +93,12 @@ gdb_expect {
 	# multiple event sources to channel the output back through the
 	# MI.
 
-	fail "Hello message (known bug)"
+	unsupported "Hello message (known limitation)"
     }
     timeout {
 	fail "Hello message (timeout)"
     }
+  }
 }
     
 gdb_expect {

Re: [RFA/mi-testsuite] XFAIL mi*-console.exp

[Andrew Cagney]

> I think you all missed this one...
> 
> Fernando Nasser wrote:
> 
>> 
>> Daniel Jacobowitz wrote:
> 
>> >
>> > That's
>> > why I'd rather XFAIL than skip the tests.
>> >
> 
>> 
>> It would be "unsupported" (which means that some feature is not
>> supported in this platform and so we can't run the test).

Ah, here it is "unimplemented" :-)  The platform does have the 
mechanisms needed to implement this, just not the code.

--

Aside, fernando and I had a brief discussion about xfail vs unsupported 
and came up with the following concrete example.

Attach/detach:

FreeBSD has a bug in its detach, since at present it doesn't work but 
did in the past, and will again in the next release it will work, it 
gets marked as ``xfail'.  Next release it will mysteriously ``xpass'' 
and can be adjusted accordingly.

Cygwin, due to limitations in the underlying OS, simply wasn't able to 
support detach, it should be marked as ``unsupported''.  (As a foot 
note, recent versions of the underlying OS, did fix this limitation).

Andrew

Re: [RFA/mi-testsuite] XFAIL mi*-console.exp

[Andrew Cagney]

> These tests are testing for a feature that exists either nowhere or just in
> simulators and some remote stubs: that the inferior's output goes through
> GDB and is properly encoded by the MI layer.  Since support isn't there for
> many remote debugging stubs or for native, I think these two tests should be
> XFAIL'd.  Does that make sense, Andrew?  If so, OK to commit this?

I believe GDB's rule for XFAIL is something that can't work (due to an 
external constraint) rather than doesn't work (due to a lack of code).

Hence it was marked as a known bug rather than a limitation.

Andrew

Re: [RFA/mi-testsuite] XFAIL mi*-console.exp

[Daniel Jacobowitz]

On Wed, Apr 03, 2002 at 11:27:13PM -0500, Andrew Cagney wrote:
> >These tests are testing for a feature that exists either nowhere or just in
> >simulators and some remote stubs: that the inferior's output goes through
> >GDB and is properly encoded by the MI layer.  Since support isn't there for
> >many remote debugging stubs or for native, I think these two tests should 
> >be
> >XFAIL'd.  Does that make sense, Andrew?  If so, OK to commit this?
> 
> I believe GDB's rule for XFAIL is something that can't work (due to an 
> external constraint) rather than doesn't work (due to a lack of code).
> 
> Hence it was marked as a known bug rather than a limitation.

OK, so it isn't an XFAIL.  I don't think FAIL is really appropriate
either; tests which test a not-yet-implemented feature (and one that I
think is a bad idea, for native targets, to be honest) don't add any
information by failing.  UNSUPPORTED perhaps?  Or just not running the
test in native setups, for now?

-- 
Daniel Jacobowitz                           Carnegie Mellon University
MontaVista Software                         Debian GNU/Linux Developer

Re: [RFA/mi-testsuite] XFAIL mi*-console.exp

[Andrew Cagney]

> On Wed, Apr 03, 2002 at 11:27:13PM -0500, Andrew Cagney wrote:
> 
>> >These tests are testing for a feature that exists either nowhere or just in
>> >simulators and some remote stubs: that the inferior's output goes through
>> >GDB and is properly encoded by the MI layer.  Since support isn't there for
>> >many remote debugging stubs or for native, I think these two tests should 
>> >be
>> >XFAIL'd.  Does that make sense, Andrew?  If so, OK to commit this?
> 
>> 
>> I believe GDB's rule for XFAIL is something that can't work (due to an 
>> external constraint) rather than doesn't work (due to a lack of code).
>> 
>> Hence it was marked as a known bug rather than a limitation.
> 
> 
> OK, so it isn't an XFAIL.  I don't think FAIL is really appropriate
> either; tests which test a not-yet-implemented feature (and one that I
> think is a bad idea, for native targets, to be honest) don't add any
> information by failing.  UNSUPPORTED perhaps?  Or just not running the
> test in native setups, for now?
Er, actually, XFAIL might be closer to the truth than UNSUPPORTED. 
Although neither indicate UNIMPLEMENTED.

Andrew

> Aside, fernando and I had a brief discussion about xfail vs unsupported and came up with the following concrete example.
> 
> Attach/detach:
> 
> FreeBSD has a bug in its detach, since at present it doesn't work but did in the past, and will again in the next release it will work, it gets marked as ``xfail'.  Next release it will mysteriously ``xpass'' and can be adjusted accordingly.
> 
> Cygwin, due to limitations in the underlying OS, simply wasn't able to support detach, it should be marked as ``unsupported''.  (As a foot note, recent versions of the underlying OS, did fix this limitation).

Re: [RFA/mi-testsuite] XFAIL mi*-console.exp

[Daniel Jacobowitz]

On Thu, Apr 04, 2002 at 04:49:54PM -0500, Andrew Cagney wrote:
> >OK, so it isn't an XFAIL.  I don't think FAIL is really appropriate
> >either; tests which test a not-yet-implemented feature (and one that I
> >think is a bad idea, for native targets, to be honest) don't add any
> >information by failing.  UNSUPPORTED perhaps?  Or just not running the
> >test in native setups, for now?
> Er, actually, XFAIL might be closer to the truth than UNSUPPORTED. 
> Although neither indicate UNIMPLEMENTED.

I'm just going to sit on this patch until we can decide what the result
should be.  I've posted both XFAIL and UNSUPPORTED versions for your
viewing pleasure...

> 
> Andrew
> 
> >Aside, fernando and I had a brief discussion about xfail vs unsupported and 
> >came up with the following concrete example.
> >
> >Attach/detach:
> >
> >FreeBSD has a bug in its detach, since at present it doesn't work but did 
> >in the past, and will again in the next release it will work, it gets 
> >marked as ``xfail'.  Next release it will mysteriously ``xpass'' and can be 
> >adjusted accordingly.
> >
> >Cygwin, due to limitations in the underlying OS, simply wasn't able to 
> >support detach, it should be marked as ``unsupported''.  (As a foot note, 
> >recent versions of the underlying OS, did fix this limitation).
> 
> 
> 

-- 
Daniel Jacobowitz                           Carnegie Mellon University
MontaVista Software                         Debian GNU/Linux Developer

Re: [RFA/mi-testsuite] XFAIL mi*-console.exp

[Andrew Cagney]

> On Thu, Apr 04, 2002 at 04:49:54PM -0500, Andrew Cagney wrote:
> 
>> >OK, so it isn't an XFAIL.  I don't think FAIL is really appropriate
>> >either; tests which test a not-yet-implemented feature (and one that I
>> >think is a bad idea, for native targets, to be honest) don't add any
>> >information by failing.  UNSUPPORTED perhaps?  Or just not running the
>> >test in native setups, for now?
> 
>> Er, actually, XFAIL might be closer to the truth than UNSUPPORTED. 
>> Although neither indicate UNIMPLEMENTED.
> 
> 
> I'm just going to sit on this patch until we can decide what the result
> should be.  I've posted both XFAIL and UNSUPPORTED versions for your
> viewing pleasure...

Suggest a bug report so it isn't forgotten.

Andrew

Re: [RFA/mi-testsuite] XFAIL mi*-console.exp

[Fernando Nasser]

Andrew Cagney wrote:
> 
> > On Wed, Apr 03, 2002 at 11:27:13PM -0500, Andrew Cagney wrote:
> >
> >> >These tests are testing for a feature that exists either nowhere or just in
> >> >simulators and some remote stubs: that the inferior's output goes through
> >> >GDB and is properly encoded by the MI layer.  Since support isn't there for
> >> >many remote debugging stubs or for native, I think these two tests should
> >> >be
> >> >XFAIL'd.  Does that make sense, Andrew?  If so, OK to commit this?
> >
> >>
> >> I believe GDB's rule for XFAIL is something that can't work (due to an
> >> external constraint) rather than doesn't work (due to a lack of code).
> >>
> >> Hence it was marked as a known bug rather than a limitation.
> >
> >
> > OK, so it isn't an XFAIL.  I don't think FAIL is really appropriate
> > either; tests which test a not-yet-implemented feature (and one that I
> > think is a bad idea, for native targets, to be honest) don't add any
> > information by failing.  UNSUPPORTED perhaps?  Or just not running the
> > test in native setups, for now?
> Er, actually, XFAIL might be closer to the truth than UNSUPPORTED.
> Although neither indicate UNIMPLEMENTED.
> 

I see. It is not UNSUPPORTED because the environment have it, just some
gsb stubs (and native) do not handle it (i.e., our fault).

Well, as we don't have UNIMPLEMENTED, we are left with only 2 options:

I guess we would use XFAIL if we decide that this is a feature that
we should have and not having it is a failure that should be fixed.

Otherwise we can just skip the test where it can't run.

 

-- 
Fernando Nasser
Red Hat Canada Ltd.                     E-Mail:  fnasser@redhat.com
2323 Yonge Street, Suite #300
Toronto, Ontario   M4P 2C9

Re: [RFA/mi-testsuite] XFAIL mi*-console.exp

[Andrew Cagney]

> Andrew Cagney wrote:
> 
>> 
> 
>> > On Wed, Apr 03, 2002 at 11:27:13PM -0500, Andrew Cagney wrote:
>> >
> 
>> >> >These tests are testing for a feature that exists either nowhere or just in
>> >> >simulators and some remote stubs: that the inferior's output goes through
>> >> >GDB and is properly encoded by the MI layer.  Since support isn't there for
>> >> >many remote debugging stubs or for native, I think these two tests should
>> >> >be
>> >> >XFAIL'd.  Does that make sense, Andrew?  If so, OK to commit this?
> 
>> >
> 
>> >>
>> >> I believe GDB's rule for XFAIL is something that can't work (due to an
>> >> external constraint) rather than doesn't work (due to a lack of code).
>> >>
>> >> Hence it was marked as a known bug rather than a limitation.
> 
>> >
>> >
>> > OK, so it isn't an XFAIL.  I don't think FAIL is really appropriate
>> > either; tests which test a not-yet-implemented feature (and one that I
>> > think is a bad idea, for native targets, to be honest) don't add any
>> > information by failing.  UNSUPPORTED perhaps?  Or just not running the
>> > test in native setups, for now?
> 
>> Er, actually, XFAIL might be closer to the truth than UNSUPPORTED.
>> Although neither indicate UNIMPLEMENTED.
>> 
> 
> 
> I see. It is not UNSUPPORTED because the environment have it, just some
> gsb stubs (and native) do not handle it (i.e., our fault).
> 
> Well, as we don't have UNIMPLEMENTED, we are left with only 2 options:
> 
> I guess we would use XFAIL if we decide that this is a feature that
> we should have and not having it is a failure that should be fixed.

> Otherwise we can just skip the test where it can't run.

I don't think this one is an option.  The gdb.asm test was being skipped 
and as a consequence everyone ignored it.  Now that it fails, people are 
fixing it.

Given it is a GDB ``bug'', is the correct approach:

	set prms_id <i-can't-find-it>
	FAIL ...

Andrew

Re: [RFA/mi-testsuite] XFAIL mi*-console.exp

[Fernando Nasser]

Andrew Cagney wrote:
> 
> >
> > I see. It is not UNSUPPORTED because the environment have it, just some
> > gsb stubs (and native) do not handle it (i.e., our fault).
> >
> > Well, as we don't have UNIMPLEMENTED, we are left with only 2 options:
> >
> > I guess we would use XFAIL if we decide that this is a feature that
> > we should have and not having it is a failure that should be fixed.
> 
> > Otherwise we can just skip the test where it can't run.
> 
> I don't think this one is an option.  The gdb.asm test was being skipped
> and as a consequence everyone ignored it.  Now that it fails, people are
> fixing it.
> 
> Given it is a GDB ``bug'', is the correct approach:
> 
>         set prms_id <i-can't-find-it>
>         FAIL ...
> 

Lets do what Daniel suggested then, and XFAIL it.  We will soon be able
to make it into a KFAIL.  It would e important to have a bug database
entry for this though (do we have one already?).

-- 
Fernando Nasser
Red Hat Canada Ltd.                     E-Mail:  fnasser@redhat.com
2323 Yonge Street, Suite #300
Toronto, Ontario   M4P 2C9