Re: [patch/rfc] Remove all setup_xfail's from testsuite/gdb.mi/

Re: [patch/rfc] Remove all setup_xfail's from testsuite/gdb.mi/

[Michael Elizabeth Chastain]

Andrew C writes:

> See: http://sources.redhat.com/ml/gdb/2003-01/msg00296.html

Ah, okay, I misunderstood.  Now I understand that you are talking
about 's/CLLbsNNNNN//', not 'g/CLLbsNNNNN/d'.

I still think that's not useful in light of the progress we are
making but I would not be opposed to it happening.

Michael C

Re: [patch/rfc] Remove all setup_xfail's from testsuite/gdb.mi/

[Michael Elizabeth Chastain]

David C writes:
> I've noticed that there are an awful lot of DWARF 1 ones, which I
> assume actually are legit.

Here is what I want to do about gdb DWARF-1 C++ support:

  make a policy decision whether gdb supports DWARF-1 with C++

  if yes:
    remove all the setup_xfail_format DWARF-1 calls
    add regular DWARF-1 testing
    file about 50 new PR's for all the new FAIL's
    file PR's against gcc for the external bugs (and they will hate us)

  if no:
    document that gdb supports C++ with DWARF-2 or stabs+ but not DWARF-1
    add a DWARF-1 check to "skip_cplus_tests"
    remove all the setup_xfail_format DWARF-1 calls

Michael C

Re: [patch/rfc] Remove all setup_xfail's from testsuite/gdb.mi/

[Daniel Jacobowitz]

On Fri, Jan 17, 2003 at 01:28:08PM -0600, Michael Elizabeth Chastain wrote:
> David C writes:
> > I've noticed that there are an awful lot of DWARF 1 ones, which I
> > assume actually are legit.
> 
> Here is what I want to do about gdb DWARF-1 C++ support:
> 
>   make a policy decision whether gdb supports DWARF-1 with C++
> 
>   if yes:
>     remove all the setup_xfail_format DWARF-1 calls
>     add regular DWARF-1 testing
>     file about 50 new PR's for all the new FAIL's
>     file PR's against gcc for the external bugs (and they will hate us)
> 
>   if no:
>     document that gdb supports C++ with DWARF-2 or stabs+ but not DWARF-1
>     add a DWARF-1 check to "skip_cplus_tests"
>     remove all the setup_xfail_format DWARF-1 calls

My inclination is "no"...

-- 
Daniel Jacobowitz
MontaVista Software                         Debian GNU/Linux Developer

Re: [patch/rfc] Remove all setup_xfail's from testsuite/gdb.mi/

[Michael Elizabeth Chastain]

Good morning xfail crew,

There are a lot of xfail's in the corpus, but the number is shrinking
steadily with time.

  % cd testsuite
  % grep -r setup_xfail gdb.* | wc

    5.2.1               649
    5.3                 577
    HEAD%20030115       517

After sleeping on the problem, I like Daniel J's path for a while.
Just keep banging on the test suite and reducing that "517" number with
no more machinery and no Andrew C Big Rip.  After we kill a bunch more
of them then we can do some kind of Big Rip or new visual markings for
the legitimate ones.

> Just one note: there are cases where the XFAIL is produced by a 'xfail'
> command inside a '-re' clause.

I count 26 of those in HEAD%20030115.

> The case for debugger format and compiler version is so common that I
> would feel tempted to have a special function for that.

The trouble is that you wind up with various boolean combinations,
such as a test that xfail's with either gcc v2 stabs or icc+blah.

I think the real issue is that we have to specify and document which
compilers that the gdb test suite supports.

Michael C

Re: [patch/rfc] Remove all setup_xfail's from testsuite/gdb.mi/

[David Carlton]

On Fri, 17 Jan 2003 13:00:03 -0600, Michael Elizabeth Chastain <mec@shout.net> said:

> Good morning xfail crew,
> There are a lot of xfail's in the corpus, but the number is shrinking
> steadily with time.

>   % cd testsuite
>   % grep -r setup_xfail gdb.* | wc

>     5.2.1               649
>     5.3                 577
>     HEAD%20030115       517

> After sleeping on the problem, I like Daniel J's path for a while.
> Just keep banging on the test suite and reducing that "517" number with
> no more machinery and no Andrew C Big Rip.

Personally, I basically prefer that approach, too.  I'm gradually
going through the current non-{PASS,KFAIL} results that I see on my
test runs in gdb.c++, and investigating them; I'm not distinguishing
between XFAIL and FAIL (and XPASS, for that matter) when doing so.
Once I'm done with that, I'll move on to the xfail's that I don't see;
I've noticed that there are an awful lot of DWARF 1 ones, which I
assume actually are legit.  (And I also want to bring in bugs from the
database into the testsuite, and add some new bugs that I've seen in
my own C++ debugging, but that's a separate issue.)

On the other hand, while by now it's clear that Andrew would meet
considerable resistance in a wholescale ripping out of XFAIL's in
gdb.c++, what Andrew was actually proposing (if memory serves me well)
is starting by ripping them out of gdb.mi.  Andrew's one of the
maintainers there; if he thinks that ripping them out would make the
results of test runs in gdb.mi more accurate, then doing so sounds
entirely reasonable to me.  Elena's the other maintainer of the mi
tests; I plan to keep her quite busy enough reviewing my various
forthcoming symtab patches that she won't want have time to audit the
mi tests, either.

I'm not sure that I see the urgency of ripping them out of the
testsuite as a whole, given that there is actual concrete progress
being made in cleaning up the testsuite right now.

David Carlton
carlton@math.stanford.edu

Re: [patch/rfc] Remove all setup_xfail's from testsuite/gdb.mi/

[David Carlton]

On 17 Jan 2003 11:16:48 -0800, David Carlton <carlton@math.Stanford.EDU> said:

> I'm gradually going through the current non-{PASS,KFAIL} results
> that I see on my test runs in gdb.c++, and investigating them; I'm
> not distinguishing between XFAIL and FAIL (and XPASS, for that
> matter) when doing so.  Once I'm done with that, I'll move on to the
> xfail's that I don't see; I've noticed that there are an awful lot
> of DWARF 1 ones, which I assume actually are legit.

Also, one thing I've noticed is that the string "FIXME" occurs a fair
number of times in gdb.c++, with a similar meaning to XFAIL (and
typically used in combination with XFAIL).  That's another thing that
has to be audited.

David Carlton
carlton@math.stanford.edu

Re: [patch/rfc] Remove all setup_xfail's from testsuite/gdb.mi/

[Daniel Jacobowitz]

On Fri, Jan 17, 2003 at 11:20:30AM -0800, David Carlton wrote:
> On 17 Jan 2003 11:16:48 -0800, David Carlton <carlton@math.Stanford.EDU> said:
> 
> > I'm gradually going through the current non-{PASS,KFAIL} results
> > that I see on my test runs in gdb.c++, and investigating them; I'm
> > not distinguishing between XFAIL and FAIL (and XPASS, for that
> > matter) when doing so.  Once I'm done with that, I'll move on to the
> > xfail's that I don't see; I've noticed that there are an awful lot
> > of DWARF 1 ones, which I assume actually are legit.
> 
> Also, one thing I've noticed is that the string "FIXME" occurs a fair
> number of times in gdb.c++, with a similar meaning to XFAIL (and
> typically used in combination with XFAIL).  That's another thing that
> has to be audited.

Yes.  In particular, all the vtable related ones should be fixed.  I
just haven't figured out what the right output should look like, yet...

-- 
Daniel Jacobowitz
MontaVista Software                         Debian GNU/Linux Developer

Re: [patch/rfc] Remove all setup_xfail's from testsuite/gdb.mi/

[Andrew Cagney]

> Good morning xfail crew,
> 
> There are a lot of xfail's in the corpus, but the number is shrinking
> steadily with time.
> 
>   % cd testsuite
>   % grep -r setup_xfail gdb.* | wc
> 
>     5.2.1               649
>     5.3                 577
>     HEAD%20030115       517
> 
> After sleeping on the problem, I like Daniel J's path for a while.
> Just keep banging on the test suite and reducing that "517" number with
> no more machinery and no Andrew C Big Rip.  After we kill a bunch more
> of them then we can do some kind of Big Rip or new visual markings for
> the legitimate ones.

See: http://sources.redhat.com/ml/gdb/2003-01/msg00296.html

Andrew

Re: [patch/rfc] Remove all setup_xfail's from testsuite/gdb.mi/

[Michael Elizabeth Chastain]

My two cents ...

Daniel J suggests that we keep making improvements:
  > XFAIL->KFAIL
  > random XFAIL->analyzed XFAIL
  > XFAIL->PASS

The problem is that, in the source code, "setup_xfail" looks the same
for both our crap legacy XFAIL's and the nice new analyzed xfail's.

Perhaps a little mechanism like "gdb_mark_external_fail" would help.
Then "grep xfail" would find only the shrinking pool of old stuff.

> This I definitely like.  "Cantfix"?

I propose "external".

I find "cantfix" to be a bit arrogant and a bit negative.  And it doesn't
distinguish between "I can't fix this because I don't have the resources"
versus "I can't fix this because I can show you that binutils is feeding
gdb incorrect / incomplete information".

Michael C

Re: [patch/rfc] Remove all setup_xfail's from testsuite/gdb.mi/

[Daniel Jacobowitz]

On Thu, Jan 16, 2003 at 02:06:05PM -0600, Michael Elizabeth Chastain wrote:
> My two cents ...
> 
> Daniel J suggests that we keep making improvements:
>   > XFAIL->KFAIL
>   > random XFAIL->analyzed XFAIL
>   > XFAIL->PASS
> 
> The problem is that, in the source code, "setup_xfail" looks the same
> for both our crap legacy XFAIL's and the nice new analyzed xfail's.
> 
> Perhaps a little mechanism like "gdb_mark_external_fail" would help.
> Then "grep xfail" would find only the shrinking pool of old stuff.

I was going to mark individual files, but the concept is the same...
then we can kill the markers when we're done.

> > This I definitely like.  "Cantfix"?
> 
> I propose "external".
> 
> I find "cantfix" to be a bit arrogant and a bit negative.  And it doesn't
> distinguish between "I can't fix this because I don't have the resources"
> versus "I can't fix this because I can show you that binutils is feeding
> gdb incorrect / incomplete information".

I like "external".

-- 
Daniel Jacobowitz
MontaVista Software                         Debian GNU/Linux Developer

Re: [patch/rfc] Remove all setup_xfail's from testsuite/gdb.mi/

[Fernando Nasser]

Daniel Jacobowitz wrote:
> 
> I like "external".
> 

Mee too.


-- 
Fernando Nasser
Red Hat - Toronto                       E-Mail:  fnasser@redhat.com
2323 Yonge Street, Suite #300
Toronto, Ontario   M4P 2C9

Re: [patch/rfc] Remove all setup_xfail's from testsuite/gdb.mi/

[Andrew Cagney]

> Daniel Jacobowitz wrote:
> 
> I like "external".
> 
> 
> Mee too.

BTW, I think `external' works as a category but not as a state.  That 
does still work for GDB though.  Put it into the external category and 
either suspend or close it.

Andrew

Re: [patch/rfc] Remove all setup_xfail's from testsuite/gdb.mi/

[Fernando Nasser]

Michael Elizabeth Chastain wrote:> My two cents ...
> 
> Daniel J suggests that we keep making improvements:
>   > XFAIL->KFAIL
>   > random XFAIL->analyzed XFAIL
>   > XFAIL->PASS
> 
> The problem is that, in the source code, "setup_xfail" looks the same
> for both our crap legacy XFAIL's and the nice new analyzed xfail's.
> 
> Perhaps a little mechanism like "gdb_mark_external_fail" would help.
> Then "grep xfail" would find only the shrinking pool of old stuff.
> 

You are right, we need some way to distinguish what have been verified and what 
has not yet been.

But if we do what Andrew proposes and get rid of all old PR and HP bug number 
markings we can make the distinction visually.  setup_xfail lines without a bug 
number have not yet been verified.  There aren't that many that would make it so 
hard, and the poorly marked ones related to stabs or old compilers do show up in 
your reports very clearly (just diff with a dwarf2 on a new compiler results). 
And as the reports point out, there are only 71 that are not that case.

Just one note: there are cases where the XFAIL is produced by a 'xfail' command 
inside a '-re' clause.


One variation of your idea:

The case for debugger format and compiler version is so common that I would feel 
tempted to have a special function for that.  Something like

gdb_compiler_related_xfail <format> <version spec> <target>+ <bug id (external)>

<format> could be "stabs", "dwarf2", "any"

<version spec>  not sure.  A list? An expression? Suggestions?

<target>+ one or more target specs, as currently (we may even make it optional 
meaning *-*-*)

<bug id>  a bug id, which, of course, will be categorized as "external"



> 
>>This I definitely like.  "Cantfix"?
> 
> 
> I propose "external".
> 
> I find "cantfix" to be a bit arrogant and a bit negative.  And it doesn't
> distinguish between "I can't fix this because I don't have the resources"
> versus "I can't fix this because I can show you that binutils is feeding
> gdb incorrect / incomplete information".
> 

Agreed.



-- 
Fernando Nasser
Red Hat - Toronto                       E-Mail:  fnasser@redhat.com
2323 Yonge Street, Suite #300
Toronto, Ontario   M4P 2C9

Re: [patch/rfc] Remove all setup_xfail's from testsuite/gdb.mi/

[Michael Elizabeth Chastain]

Daniel writes:
> Associating a PR with them is a different issue.  Just because we
> associate a PR doesn't mean we have to use KFAIL.

Fernando writes:
> That is correct.  The last argument of a setup_xfail, if it does not
> contain '-' (Argh!  Don't blame me, it was already there since immemorial
> times) is the PR number.  It can easily be a gdb/NNN bug id.  We cannot
> enforce the syntax, but we can enforce it as a police.

Okay, I can live with that.  As long as there is a *gdb* bug id on it;
that is the important part to me.

For the mechanics, we can set 'class=suspended', or maybe we can add a
new class to gnats.

> Why doing this?  A script can go through the KFAILs (for each platform)
> and, by reading the Gnats database, automatically create a KNOWN BUGS
> man page section, or a section of a Release Notes document.

Right now, I do this for native i686-pc-linux-gnu for every release
candidate.  So it helps a lot when all KFAIL's and XFAIL's point to
a place where we can write analysis notes.

> P.S.: Does someone know how to programatically access the Gnats database?

I don't know.  You could do screen scraping with URL's like the ones
that I post in the sunday project reports.

Michael C

Re: [patch/rfc] Remove all setup_xfail's from testsuite/gdb.mi/

[Michael Elizabeth Chastain]

Fernando writes:

> That is a true XFAIL, but it should be conditional to version 2 compilers
> and stabs, so that it does not show as XPASS everywhere else.

(regarding constvars.exp)
I agree.  In fact the mechanics are really easy, I think it's a 1-line
patch.  'gcc_info' contains more information than we thought, it is
not just a boolean, it's an integer with the gcc major version number.

> Please see my reply to Daniel's message.

Okay I'll take my thoughts over there.

Michael C

Re: [patch/rfc] Remove all setup_xfail's from testsuite/gdb.mi/

[Michael Elizabeth Chastain]

Fernando Nasser writes:
> As I said, this is just a brainstorm.  Please comment.

Ooh!  Ooh!

First I think that we are making great progress, especially Daniel J.
There is no hurry on the policy issue because the test suite corpus is
getting a lot better.  In particular, I feel no hurry to rip out
every setup_xfail en masse, because we are making progress with them
individually.

I agree with Fernando that it's okay to tie every XFAIL to a gdb PR
and turn it into a KFAIL (at least, I think he is saying that).

Let's take a specific case.  gdb.base/constvars.exp has a lot of
tests such as "const char * foo ; ptype foo".  With gcc 2.95.3/stabs+,
the "ptype" prints "char * foo".  This happens because gcc 2.95.3/stabs+
does not put any const information in the stabs.  This is probably never
going to get fixed in the gcc 2 series.

Now the real problem comes to light.  'K' and 'X' are really orthogonal.
'K' means that we know about the problem, and 'X' means that it is
a problem in an external tool, and these two things are separate.
But we made them an either/or, so we have to choose.

I'd rather have this become a KFAIL with reference to a gdb PR.  Then
the gdb PR can say that this incorrect behavior happens, but it's not
gdb's fault.  The gdb PR should refer to a gcc PR or other external PR.
And then we can't close the gdb PR until gcc revives gcc 2.X development
or gdb drop supports for gcc 2.X.

We could add another PR state for these kind of PR's, or we could
just use the 'suspended' state.

From the gdb user's point of view, a bug is a bug.  A gdb user can do
the same thing as the test suite and then file a PR: 'gdb fails to
print 'const' for const types'.

Michael C

Re: [patch/rfc] Remove all setup_xfail's from testsuite/gdb.mi/

[Daniel Jacobowitz]

On Wed, Jan 15, 2003 at 11:44:06AM -0600, Michael Elizabeth Chastain wrote:
> Now the real problem comes to light.  'K' and 'X' are really orthogonal.
> 'K' means that we know about the problem, and 'X' means that it is
> a problem in an external tool, and these two things are separate.
> But we made them an either/or, so we have to choose.
> 
> I'd rather have this become a KFAIL with reference to a gdb PR.  Then
> the gdb PR can say that this incorrect behavior happens, but it's not
> gdb's fault.  The gdb PR should refer to a gcc PR or other external PR.
> And then we can't close the gdb PR until gcc revives gcc 2.X development
> or gdb drop supports for gcc 2.X.
> 
> We could add another PR state for these kind of PR's, or we could
> just use the 'suspended' state.
> 
> >From the gdb user's point of view, a bug is a bug.  A gdb user can do
> the same thing as the test suite and then file a PR: 'gdb fails to
> print 'const' for const types'.

I'm not sure I agree with this.  My point of view was that either it's
an expected bug (environment) or it is a "known bug in the tool being
tested".  I don't like calling environment bugs KFAILs.  Do that, and
we'll just end up with no XFAILs...

Associating a PR with them is a different issue.  Just because we
associate a PR doesn't mean we have to use KFAIL.

-- 
Daniel Jacobowitz
MontaVista Software                         Debian GNU/Linux Developer

Re: [patch/rfc] Remove all setup_xfail's from testsuite/gdb.mi/

[Fernando Nasser]

Daniel Jacobowitz wrote:
>>
>>>From the gdb user's point of view, a bug is a bug.  A gdb user can do
>>the same thing as the test suite and then file a PR: 'gdb fails to
>>print 'const' for const types'.
> 
> 
> I'm not sure I agree with this.  My point of view was that either it's
> an expected bug (environment) or it is a "known bug in the tool being
> tested".  I don't like calling environment bugs KFAILs.  Do that, and
> we'll just end up with no XFAILs...
> 
> Associating a PR with them is a different issue.  Just because we
> associate a PR doesn't mean we have to use KFAIL.
> 

That is correct.  The last argument of a setup_xfail, if it does not contain '-' 
(Argh!  Don't blame me, it was already there since immemorial times) is the PR 
number.  It can easily be a gdb/NNN bug id.  We cannot enforce the syntax, but 
we can enforce it as a police.

So, to address Michael's concerns, we could open a WONTFIX bug report (is there 
such category?  This is the Bugzilla one) saying that it is a GCC or whatever 
bug and outside of our control and add the id to the setup_xfail.

Why doing this?  A script can go through the KFAILs (for each platform) and, by 
reading the Gnats database, automatically create a KNOWN BUGS man page section, 
or a section of a Release Notes document.  Using the same principle, it can go 
through the XFAILs and generate a section of known limitations _on a specific 
environment_ (the one where the test results were obtained).

Regards,
Fernando

P.S.: Does someone know how to programatically access the Gnats database?


-- 
Fernando Nasser
Red Hat - Toronto                       E-Mail:  fnasser@redhat.com
2323 Yonge Street, Suite #300
Toronto, Ontario   M4P 2C9

Re: [patch/rfc] Remove all setup_xfail's from testsuite/gdb.mi/

[Daniel Jacobowitz]

On Thu, Jan 16, 2003 at 09:27:12AM -0500, Fernando Nasser wrote:
> Daniel Jacobowitz wrote:
> >>
> >>>From the gdb user's point of view, a bug is a bug.  A gdb user can do
> >>the same thing as the test suite and then file a PR: 'gdb fails to
> >>print 'const' for const types'.
> >
> >
> >I'm not sure I agree with this.  My point of view was that either it's
> >an expected bug (environment) or it is a "known bug in the tool being
> >tested".  I don't like calling environment bugs KFAILs.  Do that, and
> >we'll just end up with no XFAILs...
> >
> >Associating a PR with them is a different issue.  Just because we
> >associate a PR doesn't mean we have to use KFAIL.
> >
> 
> That is correct.  The last argument of a setup_xfail, if it does not 
> contain '-' (Argh!  Don't blame me, it was already there since immemorial 
> times) is the PR number.  It can easily be a gdb/NNN bug id.  We cannot 
> enforce the syntax, but we can enforce it as a police.

[Last or first?  Oh, the answer is "either" according to framework.exp.]

> So, to address Michael's concerns, we could open a WONTFIX bug report (is 
> there such category?  This is the Bugzilla one) saying that it is a GCC or 
> whatever bug and outside of our control and add the id to the setup_xfail.

We could use the Suspended state for this, and we could just assign
them to a new environment/ category...

> Why doing this?  A script can go through the KFAILs (for each platform) 
> and, by reading the Gnats database, automatically create a KNOWN BUGS man 
> page section, or a section of a Release Notes document.  Using the same 
> principle, it can go through the XFAILs and generate a section of known 
> limitations _on a specific environment_ (the one where the test results 
> were obtained).
> 
> Regards,
> Fernando
> 
> P.S.: Does someone know how to programatically access the Gnats database?

Yes; the GNATS software package includes a number of remote tools,
particularly nquery-pr.  You have to find a copy of GNATS 3.x; the 4.x
tools don't interoperate with the version on sourceware.

-- 
Daniel Jacobowitz
MontaVista Software                         Debian GNU/Linux Developer

Re: [patch/rfc] Remove all setup_xfail's from testsuite/gdb.mi/

[Fernando Nasser]

Daniel Jacobowitz wrote:
>>That is correct.  The last argument of a setup_xfail, if it does not 
>>contain '-' (Argh!  Don't blame me, it was already there since immemorial 
>>times) is the PR number.  It can easily be a gdb/NNN bug id.  We cannot 
>>enforce the syntax, but we can enforce it as a police.
> 
> 
> [Last or first?  Oh, the answer is "either" according to framework.exp.]
> 

I was referring to the way it has been used in our tests.  The distinction is 
the absence of '-'s.


>>P.S.: Does someone know how to programatically access the Gnats database?
> 
> 
> Yes; the GNATS software package includes a number of remote tools,
> particularly nquery-pr.  You have to find a copy of GNATS 3.x; the 4.x
> tools don't interoperate with the version on sourceware.
> 

Ah, but that is a command, I was thinking of an API.  Oh well, we can always run 
the thing and capture the output I guess...


-- 
Fernando Nasser
Red Hat - Toronto                       E-Mail:  fnasser@redhat.com
2323 Yonge Street, Suite #300
Toronto, Ontario   M4P 2C9

Re: [patch/rfc] Remove all setup_xfail's from testsuite/gdb.mi/

[Daniel Jacobowitz]

On Thu, Jan 16, 2003 at 09:46:33AM -0500, Fernando Nasser wrote:
> Daniel Jacobowitz wrote:
> >>That is correct.  The last argument of a setup_xfail, if it does not 
> >>contain '-' (Argh!  Don't blame me, it was already there since immemorial 
> >>times) is the PR number.  It can easily be a gdb/NNN bug id.  We cannot 
> >>enforce the syntax, but we can enforce it as a police.
> >
> >
> >[Last or first?  Oh, the answer is "either" according to framework.exp.]
> >
> 
> I was referring to the way it has been used in our tests.  The distinction 
> is the absence of '-'s.
> 
> 
> >>P.S.: Does someone know how to programatically access the Gnats database?
> >
> >
> >Yes; the GNATS software package includes a number of remote tools,
> >particularly nquery-pr.  You have to find a copy of GNATS 3.x; the 4.x
> >tools don't interoperate with the version on sourceware.
> >
> 
> Ah, but that is a command, I was thinking of an API.  Oh well, we can 
> always run the thing and capture the output I guess...

It's got a pretty simple network protocol; nquery-pr is just a
command-line wrapper for it.

-- 
Daniel Jacobowitz
MontaVista Software                         Debian GNU/Linux Developer

Re: [patch/rfc] Remove all setup_xfail's from testsuite/gdb.mi/

[Andrew Cagney]

> That is correct.  The last argument of a setup_xfail, if it does not contain '-' (Argh!  Don't blame me, it was already there since immemorial times) is the PR number.  It can easily be a gdb/NNN bug id.  We cannot enforce the syntax, but we can enforce it as a police.


> So, to address Michael's concerns, we could open a WONTFIX bug report (is there such category?  This is the Bugzilla one) saying that it is a GCC or whatever bug and outside of our control and add the id to the setup_xfail.

There is suspended and closed.  I'm pretty sure I can add other states 
(e.g., cannotfix?).

Andrew

Re: [patch/rfc] Remove all setup_xfail's from testsuite/gdb.mi/

[Fernando Nasser]

Michael Elizabeth Chastain wrote:
> 
> I agree with Fernando that it's okay to tie every XFAIL to a gdb PR
> and turn it into a KFAIL (at least, I think he is saying that).
> 

Right.

> Let's take a specific case.  gdb.base/constvars.exp has a lot of
> tests such as "const char * foo ; ptype foo".  With gcc 2.95.3/stabs+,
> the "ptype" prints "char * foo".  This happens because gcc 2.95.3/stabs+
> does not put any const information in the stabs.  This is probably never
> going to get fixed in the gcc 2 series.
> 

That is a true XFAIL, but it should be conditional to version 2 compilers and 
stabs, so that it does not show as XPASS everywhere else.


> Now the real problem comes to light.  'K' and 'X' are really orthogonal.
> 'K' means that we know about the problem, and 'X' means that it is
> a problem in an external tool, and these two things are separate.
> But we made them an either/or, so we have to choose.
> 
> I'd rather have this become a KFAIL with reference to a gdb PR.  Then
> the gdb PR can say that this incorrect behavior happens, but it's not
> gdb's fault.  The gdb PR should refer to a gcc PR or other external PR.
> And then we can't close the gdb PR until gcc revives gcc 2.X development
> or gdb drop supports for gcc 2.X.
> 
> We could add another PR state for these kind of PR's, or we could
> just use the 'suspended' state.
> 

The Dejagnu community will kill me if I come out with yet another type of 
failure ;-)  The problem is that the international standard don't even recognize 
these extensions we've added (XFAIL and KFAIL), although I think they are really 
important and your weekely reports prove that.

>>From the gdb user's point of view, a bug is a bug.  A gdb user can do
> the same thing as the test suite and then file a PR: 'gdb fails to
> print 'const' for const types'.
> 

Yes, but we have little control over what we call XFAIL cases.  But you have a 
good point that they deserve some user documentation as much as the KFAILs do.

Please see my reply to Daniel's message.



-- 
Fernando Nasser
Red Hat - Toronto                       E-Mail:  fnasser@redhat.com
2323 Yonge Street, Suite #300
Toronto, Ontario   M4P 2C9

[patch/rfc] Remove all setup_xfail's from testsuite/gdb.mi/

[Andrew Cagney]

[-- Attachment #1: Type: text/plain, Size: 1108 bytes --]

Hello,

GDB's testsuite is known to be full of xfails that are really kfails or 
testsuite bugs.  Rather than try to audit each xfail in turn, the 
proposal as been to rip out all the xfails (creating a clean slate) and 
start marking up the tests from scratch - two steps forward but first 
one step back.

I figure I might as well try to get the ball rolling on this and find 
out just how much real resistance there is going to be to a change like 
this.  To that end, this removes all xfail's from the gdb.mi testsuite. 
  Similar tests, for the other directories, would follow.

On a GNU/Linux i386 system (Red Hat 7.2 I think) I see:

Before:

                 === gdb Summary ===

# of expected passes            8298
# of unexpected failures        60
# of unexpected successes       10
# of expected failures          172
# of unsupported tests          2

After:

                 === gdb Summary ===

# of expected passes            8298
# of unexpected failures        88
# of unexpected successes       10
# of expected failures          144
# of unsupported tests          2

Comments?

Andrew

[-- Attachment #2: diffs --]
[-- Type: text/plain, Size: 13873 bytes --]

2002-10-24  Andrew Cagney  <cagney@redhat.com>

	* mi-var-block.exp, mi1-var-block.exp: Remove all uses of
	setup_xfail and clear_xfail.
	* mi-var-cmd.exp, mi1-var-cmd.exp: Ditto.
	* mi1-var-display.exp, mi-var-display.exp: Ditto.
	* mi-var-child.exp, mi1-var-child.exp: Ditto.
	* mi-break.exp, mi1-break.exp: Remove all uses of setup_xfail.
	Update copyright.

Index: gdb.mi/mi-break.exp
===================================================================
RCS file: /cvs/src/src/gdb/testsuite/gdb.mi/mi-break.exp,v
retrieving revision 1.5
diff -u -r1.5 mi-break.exp
--- gdb.mi/mi-break.exp	27 Jun 2001 17:27:07 -0000	1.5
+++ gdb.mi/mi-break.exp	24 Oct 2002 18:07:01 -0000
@@ -1,4 +1,4 @@
-#   Copyright 1999 Free Software Foundation, Inc.
+# Copyright 1999, 2002 Free Software Foundation, Inc.
 
 # This program is free software; you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by
@@ -101,27 +101,22 @@
     # -break-insert -r .*llee
     # -break-list
 
-    setup_xfail "*-*-*"
     mi_gdb_test "122-break-insert -r main" \
              "122\\^done,bkpt=\{number=\"5\",addr=\"$hex\",file=\".*basics.c\",line=\"32\"\}" \
              "break-insert -r operation"
 
-    setup_xfail "*-*-*"
     mi_gdb_test "133-break-insert -r callee2" \
              "133\\^done,bkpt=\{number=\"6\",addr=\"$hex\",file=\".*basics.c\",line=\"22\"\}" \
              "insert breakpoint with regexp callee2"
 
-    setup_xfail "*-*-*"
     mi_gdb_test "144-break-insert -r callee" \
 	    "144\\^done,bkpt=\{number=\"7\",addr=\"$hex\",file=\".*basics.c\",line=\"27\"\},bkpt=\{number=\"8\",addr=\"$hex\",file=\".*basics.c\",line=\"22\"\},bkpt=\{number=\"9\",addr=\"$hex\",file=\".*basics.c\",line=\"17\"\},bkpt=\{number=\"10\",addr=\"$hex\",file=\".*basics.c\",line=\"8\"\}" \
              "insert breakpoint with regexp callee"
 
-    setup_xfail "*-*-*"
     mi_gdb_test "155-break-insert -r \.\*llee" \
 	    "155\\^done,bkpt=\{number=\"11\",addr=\"$hex\",file=\".*basics.c\",line=\"27\"\},bkpt=\{number=\"12\",addr=\"$hex\",file=\".*basics.c\",line=\"22\"\},bkpt=\{number=\"13\",addr=\"$hex\",file=\".*basics.c\",line=\"17\"\},bkpt=\{number=\"14\",addr=\"$hex\",file=\".*basics.c\",line=\"8\"\}" \
              "insert breakpoint with regexp .*llee"
 
-    setup_xfail "*-*-*"
     mi_gdb_test "166-break-list" \
 	    "1\\\^done,BreakpointTable=\{nr_rows=\".\",nr_cols=\".\",hdr=\\\[\{width=\".*\",alignment=\".*\",col_name=\"number\",colhdr=\"Num\"\}.*colhdr=\"Type\".*colhdr=\"Disp\".*colhdr=\"Enb\".*colhdr=\"Address\".*colhdr=\"What\".*\\\],body=\\\[bkpt=\{number=\"5\",type=\"breakpoint\",disp=\"keep\",enabled=\"y\",addr=\"$hex\",func=\"main\",file=\".*basics.c\",line=\"32\",times=\"0\"\},.*\}\\\]\}" \
                 "list of breakpoints"
Index: gdb.mi/mi-var-block.exp
===================================================================
RCS file: /cvs/src/src/gdb/testsuite/gdb.mi/mi-var-block.exp,v
retrieving revision 1.8
diff -u -r1.8 mi-var-block.exp
--- gdb.mi/mi-var-block.exp	3 Oct 2002 20:03:55 -0000	1.8
+++ gdb.mi/mi-var-block.exp	24 Oct 2002 18:07:01 -0000
@@ -142,17 +142,13 @@
 # Test: c_variable-3.7
 # Desc: check that outer foo in scope and inner foo out of scope
 # Note: also a known gdb problem
-setup_xfail *-*-*
 mi_gdb_test "-var-update inner_foo" \
 	"\\^done,changelist=\{FIXME\}" \
 	"update inner_foo: should be out of scope: KNOWN PROBLEM"
-clear_xfail *-*-*
 
-setup_xfail *-*-*
 mi_gdb_test "-var-evaluate-expression inner_foo" \
 	"\\^done,value=\{FIXME\}" \
 	"evaluate inner_foo: should be out of scope: KNOWN PROBLEM"
-clear_xfail *-*-*
 
 mi_gdb_test "-var-update foo" \
 	"\\^done,changelist=\\\[\\\]" \
@@ -173,11 +169,9 @@
 
 # Test: c_variable-3.8
 # Desc: check that foo2 out of scope (known gdb problem)
-setup_xfail *-*-*
 mi_gdb_test "-var-update foo2" \
 	"\\^done,changelist=\{FIXME\}" \
 	"update foo2: should be out of scope: KNOWN PROBLEM"
-clear_xfail *-*-*
 
 # step to "cb = 21;"
 send_gdb "-exec-step\n"
@@ -191,16 +185,12 @@
 
 # Test: c_variable-3.9
 # Desc: check that only cb is in scope (known gdb problem)
-setup_xfail *-*-*
 mi_gdb_test "-var-update foo2" \
 	"\\^done,changelist=\\\[FIXME\\\]" \
 	"update foo2 should be out of scope: KNOWN PROBLEM"
-clear_xfail *-*-*
-setup_xfail *-*-*
 mi_gdb_test "-var-update foo" \
 	"\\^done,changelist=\{FIXME\}" \
 	"update foo should be out of scope: KNOWN PROBLEM"
-clear_xfail *-*-*
 mi_gdb_test "-var-update cb" \
 	"\\^done,changelist=\\\[\\\]" \
 	"update cb"
Index: gdb.mi/mi-var-child.exp
===================================================================
RCS file: /cvs/src/src/gdb/testsuite/gdb.mi/mi-var-child.exp,v
retrieving revision 1.12
diff -u -r1.12 mi-var-child.exp
--- gdb.mi/mi-var-child.exp	3 Oct 2002 20:03:55 -0000	1.12
+++ gdb.mi/mi-var-child.exp	24 Oct 2002 18:07:01 -0000
@@ -1176,11 +1176,9 @@
 # Test: c_variable-5.51
 # Desc: check that psnp->next->long_ptr (and [1].long_ptr) changed
 # Why does this have a FIXME?
-setup_xfail *-*-*
 mi_gdb_test "-var-update *" \
 	"FIXME\\^done,changelist=\\\[\{name=\"psnp->ptrs.0.next.long_ptr\",in_scope=\"true\",type_changed=\"false\"\}\\\]" \
 	"update all vars psnp->next->long_ptr (and 1.long_ptr) changed"
-clear_xfail *-*-*
 
 # This command produces this error message:
 # &"warning: varobj_list: assertion failed - mycount <> 0\n"
Index: gdb.mi/mi-var-cmd.exp
===================================================================
RCS file: /cvs/src/src/gdb/testsuite/gdb.mi/mi-var-cmd.exp,v
retrieving revision 1.14
diff -u -r1.14 mi-var-cmd.exp
--- gdb.mi/mi-var-cmd.exp	23 Oct 2002 23:57:28 -0000	1.14
+++ gdb.mi/mi-var-cmd.exp	24 Oct 2002 18:07:02 -0000
@@ -429,11 +429,9 @@
 #       special case that, since it is not what a human expects to
 #       see.
 
-setup_xfail *-*-*
 mi_gdb_test "-var-update *" \
 	"\\^done,changelist=\{FIXME: WHAT IS CORRECT HERE\}" \
 	"update all vars: changed FIXME"
-clear_xfail *-*-*
 
 mi_step_to "subroutine1" "\{name=\"i\",value=\".*\"\},\{name=\"l\",value=\".*\"\}" \
 	"var-cmd.c" "148" "step at subroutine1 (2)"
Index: gdb.mi/mi-var-display.exp
===================================================================
RCS file: /cvs/src/src/gdb/testsuite/gdb.mi/mi-var-display.exp,v
retrieving revision 1.8
diff -u -r1.8 mi-var-display.exp
--- gdb.mi/mi-var-display.exp	17 Sep 2002 20:30:24 -0000	1.8
+++ gdb.mi/mi-var-display.exp	24 Oct 2002 18:07:02 -0000
@@ -491,13 +491,11 @@
 	"\\^done,name=\"e\",numchild=\"0\",type=\"enum foo\"" \
 	"create local variable e"
 
-setup_xfail "*-*-*"
 # Test: c_variable-7.51
 # Desc: value of e
 mi_gdb_test "-var-evaluate-expression e" \
 	"\\^done,value=\"FIXME\"" \
 	"eval variable e"
-clear_xfail "*-*-*"
 
 # Test: c_variable-7.52
 # Desc: type of e
@@ -529,13 +527,11 @@
 	"\\^done,name=\"anone\",numchild=\"0\",type=\"enum \{\\.\\.\\.\}\"" \
 	"create local variable anone"
 
-setup_xfail "*-*-*"
 # Test: c_variable-7.61
 # Desc: value of anone
 mi_gdb_test "-var-evaluate-expression anone" \
 	"\\^done,value=\"A\"" \
 	"eval variable anone"
-clear_xfail "*-*-*"
 
 
 # Test: c_variable-7.70
Index: gdb.mi/mi1-break.exp
===================================================================
RCS file: /cvs/src/src/gdb/testsuite/gdb.mi/mi1-break.exp,v
retrieving revision 1.2
diff -u -r1.2 mi1-break.exp
--- gdb.mi/mi1-break.exp	1 Oct 2002 16:01:37 -0000	1.2
+++ gdb.mi/mi1-break.exp	24 Oct 2002 18:07:02 -0000
@@ -1,4 +1,4 @@
-#   Copyright 1999 Free Software Foundation, Inc.
+# Copyright 1999, 2002 Free Software Foundation, Inc.
 
 # This program is free software; you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by
@@ -101,27 +101,22 @@
     # -break-insert -r .*llee
     # -break-list
 
-    setup_xfail "*-*-*"
     mi_gdb_test "122-break-insert -r main" \
              "122\\^done,bkpt=\{number=\"5\",addr=\"$hex\",file=\".*basics.c\",line=\"32\"\}" \
              "break-insert -r operation"
 
-    setup_xfail "*-*-*"
     mi_gdb_test "133-break-insert -r callee2" \
              "133\\^done,bkpt=\{number=\"6\",addr=\"$hex\",file=\".*basics.c\",line=\"22\"\}" \
              "insert breakpoint with regexp callee2"
 
-    setup_xfail "*-*-*"
     mi_gdb_test "144-break-insert -r callee" \
 	    "144\\^done,bkpt=\{number=\"7\",addr=\"$hex\",file=\".*basics.c\",line=\"27\"\},bkpt=\{number=\"8\",addr=\"$hex\",file=\".*basics.c\",line=\"22\"\},bkpt=\{number=\"9\",addr=\"$hex\",file=\".*basics.c\",line=\"17\"\},bkpt=\{number=\"10\",addr=\"$hex\",file=\".*basics.c\",line=\"8\"\}" \
              "insert breakpoint with regexp callee"
 
-    setup_xfail "*-*-*"
     mi_gdb_test "155-break-insert -r \.\*llee" \
 	    "155\\^done,bkpt=\{number=\"11\",addr=\"$hex\",file=\".*basics.c\",line=\"27\"\},bkpt=\{number=\"12\",addr=\"$hex\",file=\".*basics.c\",line=\"22\"\},bkpt=\{number=\"13\",addr=\"$hex\",file=\".*basics.c\",line=\"17\"\},bkpt=\{number=\"14\",addr=\"$hex\",file=\".*basics.c\",line=\"8\"\}" \
              "insert breakpoint with regexp .*llee"
 
-    setup_xfail "*-*-*"
     mi_gdb_test "166-break-list" \
 	    "1\\\^done,BreakpointTable=\{nr_rows=\".\",nr_cols=\".\",hdr=\\\[\{width=\".*\",alignment=\".*\",col_name=\"number\",colhdr=\"Num\"\}.*colhdr=\"Type\".*colhdr=\"Disp\".*colhdr=\"Enb\".*colhdr=\"Address\".*colhdr=\"What\".*\\\],body=\\\[bkpt=\{number=\"5\",type=\"breakpoint\",disp=\"keep\",enabled=\"y\",addr=\"$hex\",func=\"main\",file=\".*basics.c\",line=\"32\",times=\"0\"\},.*\}\\\]\}" \
                 "list of breakpoints"
Index: gdb.mi/mi1-var-block.exp
===================================================================
RCS file: /cvs/src/src/gdb/testsuite/gdb.mi/mi1-var-block.exp,v
retrieving revision 1.2
diff -u -r1.2 mi1-var-block.exp
--- gdb.mi/mi1-var-block.exp	1 Oct 2002 16:01:37 -0000	1.2
+++ gdb.mi/mi1-var-block.exp	24 Oct 2002 18:07:02 -0000
@@ -142,17 +142,13 @@
 # Test: c_variable-3.7
 # Desc: check that outer foo in scope and inner foo out of scope
 # Note: also a known gdb problem
-setup_xfail *-*-*
 mi_gdb_test "-var-update inner_foo" \
 	"\\^done,changelist=\{FIXME\}" \
 	"update inner_foo: should be out of scope: KNOWN PROBLEM"
-clear_xfail *-*-*
 
-setup_xfail *-*-*
 mi_gdb_test "-var-evaluate-expression inner_foo" \
 	"\\^done,value=\{FIXME\}" \
 	"evaluate inner_foo: should be out of scope: KNOWN PROBLEM"
-clear_xfail *-*-*
 
 mi_gdb_test "-var-update foo" \
 	"\\^done,changelist=\{\}" \
@@ -173,11 +169,9 @@
 
 # Test: c_variable-3.8
 # Desc: check that foo2 out of scope (known gdb problem)
-setup_xfail *-*-*
 mi_gdb_test "-var-update foo2" \
 	"\\^done,changelist=\{FIXME\}" \
 	"update foo2: should be out of scope: KNOWN PROBLEM"
-clear_xfail *-*-*
 
 # step to "cb = 21;"
 send_gdb "-exec-step\n"
@@ -191,16 +185,12 @@
 
 # Test: c_variable-3.9
 # Desc: check that only cb is in scope (known gdb problem)
-setup_xfail *-*-*
 mi_gdb_test "-var-update foo2" \
 	"\\^done,changelist=\{FIXME\}" \
 	"update foo2 should be out of scope: KNOWN PROBLEM"
-clear_xfail *-*-*
-setup_xfail *-*-*
 mi_gdb_test "-var-update foo" \
 	"\\^done,changelist=\{FIXME\}" \
 	"update foo should be out of scope: KNOWN PROBLEM"
-clear_xfail *-*-*
 mi_gdb_test "-var-update cb" \
 	"\\^done,changelist=\{\}" \
 	"update cb"
Index: gdb.mi/mi1-var-child.exp
===================================================================
RCS file: /cvs/src/src/gdb/testsuite/gdb.mi/mi1-var-child.exp,v
retrieving revision 1.2
diff -u -r1.2 mi1-var-child.exp
--- gdb.mi/mi1-var-child.exp	1 Oct 2002 16:01:37 -0000	1.2
+++ gdb.mi/mi1-var-child.exp	24 Oct 2002 18:07:03 -0000
@@ -1176,11 +1176,9 @@
 # Test: c_variable-5.51
 # Desc: check that psnp->next->long_ptr (and [1].long_ptr) changed
 # Why does this have a FIXME?
-setup_xfail *-*-*
 mi_gdb_test "-var-update *" \
 	"FIXME\\^done,changelist=\{name=\"psnp->ptrs.0.next.long_ptr\",in_scope=\"true\",type_changed=\"false\"\}" \
 	"update all vars psnp->next->long_ptr (and 1.long_ptr) changed"
-clear_xfail *-*-*
 
 # This command produces this error message:
 # &"warning: varobj_list: assertion failed - mycount <> 0\n"
Index: gdb.mi/mi1-var-cmd.exp
===================================================================
RCS file: /cvs/src/src/gdb/testsuite/gdb.mi/mi1-var-cmd.exp,v
retrieving revision 1.3
diff -u -r1.3 mi1-var-cmd.exp
--- gdb.mi/mi1-var-cmd.exp	23 Oct 2002 23:57:28 -0000	1.3
+++ gdb.mi/mi1-var-cmd.exp	24 Oct 2002 18:07:03 -0000
@@ -429,11 +429,9 @@
 #       special case that, since it is not what a human expects to
 #       see.
 
-setup_xfail *-*-*
 mi_gdb_test "-var-update *" \
 	"\\^done,changelist=\{FIXME: WHAT IS CORRECT HERE\}" \
 	"update all vars: changed FIXME"
-clear_xfail *-*-*
 
 mi_step_to "subroutine1" "\{name=\"i\",value=\".*\"\},\{name=\"l\",value=\".*\"\}" \
 	"var-cmd.c" "148" "step at subroutine1 (2)"
Index: gdb.mi/mi1-var-display.exp
===================================================================
RCS file: /cvs/src/src/gdb/testsuite/gdb.mi/mi1-var-display.exp,v
retrieving revision 1.2
diff -u -r1.2 mi1-var-display.exp
--- gdb.mi/mi1-var-display.exp	1 Oct 2002 16:01:37 -0000	1.2
+++ gdb.mi/mi1-var-display.exp	24 Oct 2002 18:07:03 -0000
@@ -491,13 +491,11 @@
 	"\\^done,name=\"e\",numchild=\"0\",type=\"enum foo\"" \
 	"create local variable e"
 
-setup_xfail "*-*-*"
 # Test: c_variable-7.51
 # Desc: value of e
 mi_gdb_test "-var-evaluate-expression e" \
 	"\\^done,value=\"FIXME\"" \
 	"eval variable e"
-clear_xfail "*-*-*"
 
 # Test: c_variable-7.52
 # Desc: type of e
@@ -529,13 +527,11 @@
 	"\\^done,name=\"anone\",numchild=\"0\",type=\"enum \{\\.\\.\\.\}\"" \
 	"create local variable anone"
 
-setup_xfail "*-*-*"
 # Test: c_variable-7.61
 # Desc: value of anone
 mi_gdb_test "-var-evaluate-expression anone" \
 	"\\^done,value=\"A\"" \
 	"eval variable anone"
-clear_xfail "*-*-*"
 
 
 # Test: c_variable-7.70

Re: [patch/rfc] Remove all setup_xfail's from testsuite/gdb.mi/

[Daniel Jacobowitz]

On Thu, Oct 24, 2002 at 02:41:05PM -0400, Andrew Cagney wrote:
> Hello,
> 
> GDB's testsuite is known to be full of xfails that are really kfails or 
> testsuite bugs.  Rather than try to audit each xfail in turn, the 
> proposal as been to rip out all the xfails (creating a clean slate) and 
> start marking up the tests from scratch - two steps forward but first 
> one step back.
> 
> I figure I might as well try to get the ball rolling on this and find 
> out just how much real resistance there is going to be to a change like 
> this.  To that end, this removes all xfail's from the gdb.mi testsuite. 
>  Similar tests, for the other directories, would follow.

I'm known to be a testsuite nazi - I really, really dislike the current
failure levels, and people aren't doing much about it.  I'm all in
favor of getting the ball rolling.  But are you planning to do the
marking promptly, or just make us stare at even more MI failures for a
while?  I've been staring at the mi-console one for a year...

I know it's not practical to do them completely without getting the
testsuite run on a lot of systems, by a lot of people.  But you can do
a fair first-order approximation without that.

-- 
Daniel Jacobowitz
MontaVista Software                         Debian GNU/Linux Developer

Re: [patch/rfc] Remove all setup_xfail's from testsuite/gdb.mi/

[Andrew Cagney]

> On Thu, Oct 24, 2002 at 02:41:05PM -0400, Andrew Cagney wrote:
> 
>> Hello,
>> 
>> GDB's testsuite is known to be full of xfails that are really kfails or 
>> testsuite bugs.  Rather than try to audit each xfail in turn, the 
>> proposal as been to rip out all the xfails (creating a clean slate) and 
>> start marking up the tests from scratch - two steps forward but first 
>> one step back.
>> 
>> I figure I might as well try to get the ball rolling on this and find 
>> out just how much real resistance there is going to be to a change like 
>> this.  To that end, this removes all xfail's from the gdb.mi testsuite. 
>>  Similar tests, for the other directories, would follow.
> 
> 
> I'm known to be a testsuite nazi - I really, really dislike the current
> failure levels, and people aren't doing much about it.  I'm all in
> favor of getting the ball rolling.  But are you planning to do the
> marking promptly, or just make us stare at even more MI failures for a
> while?  I've been staring at the mi-console one for a year...

I believe that you're free to start kfailing things :-)

As for me doing the kfail's, the chances of me getting back to that 
short term are, lets say, pretty remote.  I'm just trying to get the 
process started - eliminate the task that's going to cop the most flack :-)

> I know it's not practical to do them completely without getting the
> testsuite run on a lot of systems, by a lot of people.  But you can do
> a fair first-order approximation without that.

Andrew

Re: [patch/rfc] Remove all setup_xfail's from testsuite/gdb.mi/

[Daniel Jacobowitz]

On Thu, Oct 24, 2002 at 03:29:56PM -0400, Andrew Cagney wrote:
> >On Thu, Oct 24, 2002 at 02:41:05PM -0400, Andrew Cagney wrote:
> >
> >>Hello,
> >>
> >>GDB's testsuite is known to be full of xfails that are really kfails or 
> >>testsuite bugs.  Rather than try to audit each xfail in turn, the 
> >>proposal as been to rip out all the xfails (creating a clean slate) and 
> >>start marking up the tests from scratch - two steps forward but first 
> >>one step back.
> >>
> >>I figure I might as well try to get the ball rolling on this and find 
> >>out just how much real resistance there is going to be to a change like 
> >>this.  To that end, this removes all xfail's from the gdb.mi testsuite. 
> >> Similar tests, for the other directories, would follow.
> >
> >
> >I'm known to be a testsuite nazi - I really, really dislike the current
> >failure levels, and people aren't doing much about it.  I'm all in
> >favor of getting the ball rolling.  But are you planning to do the
> >marking promptly, or just make us stare at even more MI failures for a
> >while?  I've been staring at the mi-console one for a year...
> 
> I believe that you're free to start kfailing things :-)
> 
> As for me doing the kfail's, the chances of me getting back to that 
> short term are, lets say, pretty remote.  I'm just trying to get the 
> process started - eliminate the task that's going to cop the most flack :-)

In that case, I'd ask you not to commit this.  There's no point in
adding to the FAILs if it doesn't gain us anything.  Would it be
hard to file PRs for all the failures you see and mark them KFAIL?

-- 
Daniel Jacobowitz
MontaVista Software                         Debian GNU/Linux Developer

Re: [patch/rfc] Remove all setup_xfail's from testsuite/gdb.mi/

[Andrew Cagney]

> On Thu, Oct 24, 2002 at 03:29:56PM -0400, Andrew Cagney wrote:
> 
>> >On Thu, Oct 24, 2002 at 02:41:05PM -0400, Andrew Cagney wrote:
>> >
> 
>> >>Hello,
>> >>
>> >>GDB's testsuite is known to be full of xfails that are really kfails or 
>> >>testsuite bugs.  Rather than try to audit each xfail in turn, the 
>> >>proposal as been to rip out all the xfails (creating a clean slate) and 
>> >>start marking up the tests from scratch - two steps forward but first 
>> >>one step back.
>> >>
>> >>I figure I might as well try to get the ball rolling on this and find 
>> >>out just how much real resistance there is going to be to a change like 
>> >>this.  To that end, this removes all xfail's from the gdb.mi testsuite. 
>> >> Similar tests, for the other directories, would follow.
> 
>> >
>> >
>> >I'm known to be a testsuite nazi - I really, really dislike the current
>> >failure levels, and people aren't doing much about it.  I'm all in
>> >favor of getting the ball rolling.  But are you planning to do the
>> >marking promptly, or just make us stare at even more MI failures for a
>> >while?  I've been staring at the mi-console one for a year...
> 
>> 
>> I believe that you're free to start kfailing things :-)
>> 
>> As for me doing the kfail's, the chances of me getting back to that 
>> short term are, lets say, pretty remote.  I'm just trying to get the 
>> process started - eliminate the task that's going to cop the most flack :-)

Like I said, the task that would cop the most flack ... :-)

> In that case, I'd ask you not to commit this.  There's no point in
> adding to the FAILs if it doesn't gain us anything.

I think the patch, regardless of KFAIL, is still technically correct. It 
fixes a bug: the XFAILs are all wrong so removing them changes the 
testsuite so that the numbers it reports better reflect reality.  It's 
just unfortunate that part of the reality is a jump in testsuite 
failures.  Remember, the XFAILs were originally added to artifically 
deflate the test failure rate.

 >  Would it be
 > hard to file PRs for all the failures you see and mark them KFAIL?

I think that would be a step backwards as all it would do is fill the 
bug database with reports like ``test failed''.

At least this does move things forward - it puts the tesuite in a state 
where everyone and everyone can incrementally do the marking.

Andrew

Re: [patch/rfc] Remove all setup_xfail's from testsuite/gdb.mi/

[Daniel Jacobowitz]

On Thu, Oct 24, 2002 at 05:22:42PM -0400, Andrew Cagney wrote:
> I think the patch, regardless of KFAIL, is still technically correct. It 
> fixes a bug: the XFAILs are all wrong so removing them changes the 
> testsuite so that the numbers it reports better reflect reality.  It's 
> just unfortunate that part of the reality is a jump in testsuite 
> failures.  Remember, the XFAILs were originally added to artifically 
> deflate the test failure rate.

As you wish.  Michael's already said he just ignores gdb.mi; if it
picks up this many new failures, probably so will I.  I don't agree
that it's technically correct; the XFAILs were being used for a
slightly suboptimal meaning since KFAIL wasn't available.  They aren't
real failures no matter which way I look at it.

> >  Would it be
> > hard to file PRs for all the failures you see and mark them KFAIL?
> 
> I think that would be a step backwards as all it would do is fill the 
> bug database with reports like ``test failed''.

What do you want in the database then?

> At least this does move things forward - it puts the tesuite in a state 
> where everyone and everyone can incrementally do the marking.

But nobody will...

-- 
Daniel Jacobowitz
MontaVista Software                         Debian GNU/Linux Developer

Re: [patch/rfc] Remove all setup_xfail's from testsuite/gdb.mi/

[Michael Snyder]

Daniel Jacobowitz wrote:
> 
> On Thu, Oct 24, 2002 at 05:22:42PM -0400, Andrew Cagney wrote:
> > I think the patch, regardless of KFAIL, is still technically correct. It
> > fixes a bug: the XFAILs are all wrong so removing them changes the
> > testsuite so that the numbers it reports better reflect reality.  It's
> > just unfortunate that part of the reality is a jump in testsuite
> > failures.  Remember, the XFAILs were originally added to artifically
> > deflate the test failure rate.
> 
> As you wish.  Michael's already said he just ignores gdb.mi; if it
> picks up this many new failures, probably so will I.  I don't agree
> that it's technically correct; the XFAILs were being used for a
> slightly suboptimal meaning since KFAIL wasn't available.  They aren't
> real failures no matter which way I look at it.
> 
> > >  Would it be
> > > hard to file PRs for all the failures you see and mark them KFAIL?
> >
> > I think that would be a step backwards as all it would do is fill the
> > bug database with reports like ``test failed''.
> 
> What do you want in the database then?
> 
> > At least this does move things forward - it puts the tesuite in a state
> > where everyone and everyone can incrementally do the marking.
> 
> But nobody will...

Removing the XFAILS without replacing them with KFAILS is not fixing a bug.

Re: [patch/rfc] Remove all setup_xfail's from testsuite/gdb.mi/

[Andrew Cagney]

> At least this does move things forward - it puts the tesuite in a state
>> > where everyone and everyone can incrementally do the marking.
> 
>> 
>> But nobody will...
> 
> 
> Removing the XFAILS without replacing them with KFAILS is not fixing a bug.

Why?

While not immediatly installing KFAILs does increase the number of 
FAILs, GDB's failure rate has, for the last >2 years been artifically 
deflated to the tune of ~200!  All I'm doing is removing that artifical 
adjustment, putting things back to where they should have been >2 years ago.

Those XFAILs should have been pulled years ago but were not.  Instead, 
as a political compromise, the move/debate was delayed until after 
Fernando invested their own time in getting KFAIL added.  Are we now 
going to delay things by another N years while we all sit around and 
wait for someone to magically volenteer to do, in a single hit, all that 
audit work?

Andrew

Re: [patch/rfc] Remove all setup_xfail's from testsuite/gdb.mi/

[Michael Snyder]

Andrew Cagney wrote:
> 
> > At least this does move things forward - it puts the tesuite in a state
> >> > where everyone and everyone can incrementally do the marking.
> >
> >>
> >> But nobody will...
> >
> >
> > Removing the XFAILS without replacing them with KFAILS is not fixing a bug.
> 
> Why?
> 
> While not immediatly installing KFAILs does increase the number of
> FAILs, GDB's failure rate has, for the last >2 years been artifically
> deflated to the tune of ~200!  All I'm doing is removing that artifical
> adjustment, putting things back to where they should have been >2 years ago.
> 
> Those XFAILs should have been pulled years ago but were not.  Instead,
> as a political compromise, the move/debate was delayed until after
> Fernando invested their own time in getting KFAIL added.  Are we now
> going to delay things by another N years while we all sit around and
> wait for someone to magically volenteer to do, in a single hit, all that
> audit work?

Calm down, your points are not under debate, and no one made the
suggestion you imply.  Particularly the "magically" part.

Re: [patch/rfc] Remove all setup_xfail's from testsuite/gdb.mi/

[Andrew Cagney]

> On Thu, Oct 24, 2002 at 05:22:42PM -0400, Andrew Cagney wrote:
> 
>> I think the patch, regardless of KFAIL, is still technically correct. It 
>> fixes a bug: the XFAILs are all wrong so removing them changes the 
>> testsuite so that the numbers it reports better reflect reality.  It's 
>> just unfortunate that part of the reality is a jump in testsuite 
>> failures.  Remember, the XFAILs were originally added to artifically 
>> deflate the test failure rate.
> 
> 
> As you wish.  Michael's already said he just ignores gdb.mi; if it
> picks up this many new failures, probably so will I.

So ..., what will happen when I submit an equivalent patch for one of 
the other directories?

 >  I don't agree
 > that it's technically correct; the XFAILs were being used for a
 > slightly suboptimal meaning since KFAIL wasn't available.  They aren't
 > real failures no matter which way I look at it.

The ones I know about were real failures that reflected real bugs.  They 
were XFAILed to supress a bug that wasn't going to be fixed.  Grab an 
old GDB and check the comments that go with the a29k XFAILs.  That is 
very different to XFAILing something because it isn't possible to fix.

>> >  Would it be
>> > hard to file PRs for all the failures you see and mark them KFAIL?
> 
>> 
>> I think that would be a step backwards as all it would do is fill the 
>> bug database with reports like ``test failed''.
> 
> 
> What do you want in the database then?

An analysis of the bug.

>> At least this does move things forward - it puts the tesuite in a state 
>> where everyone and everyone can incrementally do the marking.
> 
> 
> But nobody will...

In one hit, or here and there?  I know I will.  I just won't be spending 
a solid week reviewing all of them.

Andrew

Re: [patch/rfc] Remove all setup_xfail's from testsuite/gdb.mi/

[Michael Snyder]

Andrew Cagney wrote:
> 
> > On Thu, Oct 24, 2002 at 05:22:42PM -0400, Andrew Cagney wrote:
> >
> >> I think the patch, regardless of KFAIL, is still technically correct. It
> >> fixes a bug: the XFAILs are all wrong so removing them changes the
> >> testsuite so that the numbers it reports better reflect reality.  It's
> >> just unfortunate that part of the reality is a jump in testsuite
> >> failures.  Remember, the XFAILs were originally added to artifically
> >> deflate the test failure rate.
> >
> >
> > As you wish.  Michael's already said he just ignores gdb.mi; if it
> > picks up this many new failures, probably so will I.
> 
> So ..., what will happen when I submit an equivalent patch for one of
> the other directories?
> 
>  >  I don't agree
>  > that it's technically correct; the XFAILs were being used for a
>  > slightly suboptimal meaning since KFAIL wasn't available.  They aren't
>  > real failures no matter which way I look at it.
> 
> The ones I know about were real failures that reflected real bugs.  They
> were XFAILed to supress a bug that wasn't going to be fixed.  Grab an
> old GDB and check the comments that go with the a29k XFAILs.  That is
> very different to XFAILing something because it isn't possible to fix.
> 
> >> >  Would it be
> >> > hard to file PRs for all the failures you see and mark them KFAIL?
> >
> >>
> >> I think that would be a step backwards as all it would do is fill the
> >> bug database with reports like ``test failed''.
> >
> >
> > What do you want in the database then?
> 
> An analysis of the bug.
> 
> >> At least this does move things forward - it puts the tesuite in a state
> >> where everyone and everyone can incrementally do the marking.
> >
> >
> > But nobody will...
> 
> In one hit, or here and there?  I know I will.  I just won't be spending
> a solid week reviewing all of them.

Well, so far we've got two nays.  Why don't Daniel and I shut up, 
and wait to see if there are any yays?

Re: [patch/rfc] Remove all setup_xfail's from testsuite/gdb.mi/

[Ben Elliston]

>>>>> "Michael" == Michael Snyder <msnyder@redhat.com> writes:

  >> > But nobody will...

  >> In one hit, or here and there?  I know I will.  I just won't be
  >> spending a solid week reviewing all of them.

  Michael> Well, so far we've got two nays.  Why don't Daniel and I
  Michael> shut up, and wait to see if there are any yays?

I think Andrew's idea is a sound one.  The GDB testsuite has needed a
lot of attention in recent times.  I wonder if many of the XFAILs in
the testsuite are there without good reason and a forced audit is
probably not a bad idea.

Ben

Re: [patch/rfc] Remove all setup_xfail's from testsuite/gdb.mi/

[Andrew Cagney]

> But nobody will...
> 
>> 
>> In one hit, or here and there?  I know I will.  I just won't be spending
>> a solid week reviewing all of them.
> 
> 
> Well, so far we've got two nays.  Why don't Daniel and I shut up, 
> and wait to see if there are any yays?

Michael, it might still be helpful if you could summarise the technical 
grounds for your position.  This isn't a question of yay/nay but rather 
a technical and strategic problem that needs to be resolved.

Andrew

Re: [patch/rfc] Remove all setup_xfail's from testsuite/gdb.mi/

[Michael Snyder]

Andrew Cagney wrote:
> 
> > But nobody will...
> >
> >>
> >> In one hit, or here and there?  I know I will.  I just won't be spending
> >> a solid week reviewing all of them.
> >
> >
> > Well, so far we've got two nays.  Why don't Daniel and I shut up,
> > and wait to see if there are any yays?
> 
> Michael, it might still be helpful if you could summarise the technical
> grounds for your position.  This isn't a question of yay/nay but rather
> a technical and strategic problem that needs to be resolved.


I feel like I've said all there is to be said about it.
Relatively few people work on mi or mi tests.
It seems to me more productive if we start on a small subset
of tests that a large number of people may actually be
motivated to hack on.  Including me.

You might say, I want to play, and choosing mi excludes me from the game.

Re: [patch/rfc] Remove all setup_xfail's from testsuite/gdb.mi/

[Andrew Cagney]

> Andrew Cagney wrote:
> 
>> 
> 
>> > But nobody will...
>> >
> 
>> >>
>> >> In one hit, or here and there?  I know I will.  I just won't be spending
>> >> a solid week reviewing all of them.
> 
>> >
>> >
>> > Well, so far we've got two nays.  Why don't Daniel and I shut up,
>> > and wait to see if there are any yays?
> 
>> 
>> Michael, it might still be helpful if you could summarise the technical
>> grounds for your position.  This isn't a question of yay/nay but rather
>> a technical and strategic problem that needs to be resolved.
> 
> 
> 
> I feel like I've said all there is to be said about it.
> Relatively few people work on mi or mi tests.

BTW, at present it looks like Keith, JeffJ and ElenaZ hacking on MI. 
Add in QNX people reporting bugs and Apple people with their patches and 
this is getting scary!

> It seems to me more productive if we start on a small subset
> of tests that a large number of people may actually be
> motivated to hack on.  Including me.

> You might say, I want to play, and choosing mi excludes me from the game.

The intent is to remove the XFAILs from all directories in quick 
succession.  You won't have long to wait before a directory you're 
interested in becomes available.

Hmm, I also think that trying to do a small subsets will make things 
more confusing and less productive.   Assuming that someone can be found 
to co-ordinate this, and drive it through in a very timely manner, it 
will lead to a situtation where the the test results slowly decay over 
time (as each subset is attacked).

I think it is better to just purge the problem with a single flush. 
That way, from that point on, we'll know that the results can only get 
better.

Given this problem, I think I'll revise my patch so that it does flush 
everything in a single hit.

Andrew

Re: [patch/rfc] Remove all setup_xfail's from testsuite/gdb.mi/

[Michael Snyder]

Andrew Cagney wrote:
> 
> Hello,
> 
> GDB's testsuite is known to be full of xfails that are really kfails or
> testsuite bugs.  Rather than try to audit each xfail in turn, the
> proposal as been to rip out all the xfails (creating a clean slate) and
> start marking up the tests from scratch - two steps forward but first
> one step back.
> 
> I figure I might as well try to get the ball rolling on this and find
> out just how much real resistance there is going to be to a change like
> this.  To that end, this removes all xfail's from the gdb.mi testsuite.
>   Similar tests, for the other directories, would follow.

I don't know if MI is the best choice to start with.
Who's going to fix them up?  Certainly not me.

Why not pick one or two mainstream tests, from gdb.base, that have a lot
of xfails in them.  Let us get a feel for the pain threshold.  Changes
to gdb.mi don't tell me a thing, because I generally ignore it.

Re: [patch/rfc] Remove all setup_xfail's from testsuite/gdb.mi/

[Andrew Cagney]

> Andrew Cagney wrote:
> 
>> 
>> Hello,
>> 
>> GDB's testsuite is known to be full of xfails that are really kfails or
>> testsuite bugs.  Rather than try to audit each xfail in turn, the
>> proposal as been to rip out all the xfails (creating a clean slate) and
>> start marking up the tests from scratch - two steps forward but first
>> one step back.
>> 
>> I figure I might as well try to get the ball rolling on this and find
>> out just how much real resistance there is going to be to a change like
>> this.  To that end, this removes all xfail's from the gdb.mi testsuite.
>>   Similar tests, for the other directories, would follow.
> 
> 
> I don't know if MI is the best choice to start with.
> Who's going to fix them up?  Certainly not me.
> 
> Why not pick one or two mainstream tests, from gdb.base, that have a lot
> of xfails in them.  Let us get a feel for the pain threshold.  Changes
> to gdb.mi don't tell me a thing, because I generally ignore it.

True.  For the MI though, I know all the existing XFAILs are bogus so 
I'm in a stronger position to argue that they should be removed anyway. 
  Besides, it appears MI is being worked on at present.

Anyway, it will look like this:

> Before:
> 
>                 === gdb Summary ===
> 
> # of expected passes            8298
> # of unexpected failures        60
> # of unexpected successes       10
> # of expected failures          172
> # of unsupported tests          2
> 
> After:
> 
>                 === gdb Summary ===
> 
> # of expected passes            8308
> # of unexpected failures        232
> # of unsupported tests          2

Andrew.

Re: [patch/rfc] Remove all setup_xfail's from testsuite/gdb.mi/

[David Carlton]

On Thu, 24 Oct 2002 14:41:05 -0400, Andrew Cagney <ac131313@redhat.com> said:

> GDB's testsuite is known to be full of xfails that are really kfails
> or testsuite bugs.  Rather than try to audit each xfail in turn, the
> proposal as been to rip out all the xfails (creating a clean slate)
> and start marking up the tests from scratch - two steps forward but
> first one step back.

Can you give me a little guidance here?  Elena recently made the
suggestion that I should add tests to the testsuite for namespace
stuff, even before I've modified GDB to handle that.  That sounded
sensible to me, so I added that to a branch, and marked them all as
xfail.

I suspect I was wrong about that, though I'm not sure about the
subtleties of what xfail is actually supposed to mean.  I was thinking
I should go and change them to kfail, but now I'm not confident that I
know the intended semantics of that, either.  Is kfail only allowed
for tests with a PR associated to them?  Admittedly, in a branch,
xfail and kfail mean whatever I want them to mean, I suppose, and I'm
not going to try to get those tests added to the mainline unless I can
bring along much of the code that cause them to pass instead of fail.

I guess I don't see the point in removing xfails from the testsuite:
it's useful information, it doesn't make regression testing any harder
(there, the main culprit is the !@#%# schedlock test), so why throw
that away?  If xfail has the wrong meaning, then change it to kfail;
if kfail also has the wrong meaning, then change the meaning of kfail.

David Carlton
carlton@math.stanford.edu

Re: [patch/rfc] Remove all setup_xfail's from testsuite/gdb.mi/

[Andrew Cagney]

> On Thu, 24 Oct 2002 14:41:05 -0400, Andrew Cagney <ac131313@redhat.com> said:
> 
> 
>> GDB's testsuite is known to be full of xfails that are really kfails
>> or testsuite bugs.  Rather than try to audit each xfail in turn, the
>> proposal as been to rip out all the xfails (creating a clean slate)
>> and start marking up the tests from scratch - two steps forward but
>> first one step back.
> 
> 
> Can you give me a little guidance here?  Elena recently made the
> suggestion that I should add tests to the testsuite for namespace
> stuff, even before I've modified GDB to handle that.  That sounded
> sensible to me, so I added that to a branch, and marked them all as
> xfail.

I think Elena mentioned KFAIL.  Any way, that is want you want to use - 
you'll need to bug report any failures though.

> I suspect I was wrong about that, though I'm not sure about the
> subtleties of what xfail is actually supposed to mean.  I was thinking
> I should go and change them to kfail, but now I'm not confident that I
> know the intended semantics of that, either.  Is kfail only allowed
> for tests with a PR associated to them?  Admittedly, in a branch,
> xfail and kfail mean whatever I want them to mean, I suppose, and I'm
> not going to try to get those tests added to the mainline unless I can
> bring along much of the code that cause them to pass instead of fail.

A fairly good definition is:

KFAIL == bug, in GDB, something to fix.
XFAIL == bug, not in GDB (kernel, debug info, linker, ...), something to 
ignore.

> I guess I don't see the point in removing xfails from the testsuite:
> it's useful information, it doesn't make regression testing any harder
> (there, the main culprit is the !@#%# schedlock test), so why throw
> that away?  If xfail has the wrong meaning, then change it to kfail;
> if kfail also has the wrong meaning, then change the meaning of kfail.

People have been XFAILing bugs in GDB.  That is simply wrong.

Andrew

Re: [patch/rfc] Remove all setup_xfail's from testsuite/gdb.mi/

[Michael Snyder]

Andrew Cagney wrote:
> 
> > On Thu, 24 Oct 2002 14:41:05 -0400, Andrew Cagney <ac131313@redhat.com> said:
> >
> >
> >> GDB's testsuite is known to be full of xfails that are really kfails
> >> or testsuite bugs.  Rather than try to audit each xfail in turn, the
> >> proposal as been to rip out all the xfails (creating a clean slate)
> >> and start marking up the tests from scratch - two steps forward but
> >> first one step back.
> >
> >
> > Can you give me a little guidance here?  Elena recently made the
> > suggestion that I should add tests to the testsuite for namespace
> > stuff, even before I've modified GDB to handle that.  That sounded
> > sensible to me, so I added that to a branch, and marked them all as
> > xfail.
> 
> I think Elena mentioned KFAIL.  Any way, that is want you want to use -
> you'll need to bug report any failures though.
> 
> > I suspect I was wrong about that, though I'm not sure about the
> > subtleties of what xfail is actually supposed to mean.  I was thinking
> > I should go and change them to kfail, but now I'm not confident that I
> > know the intended semantics of that, either.  Is kfail only allowed
> > for tests with a PR associated to them?  Admittedly, in a branch,
> > xfail and kfail mean whatever I want them to mean, I suppose, and I'm
> > not going to try to get those tests added to the mainline unless I can
> > bring along much of the code that cause them to pass instead of fail.
> 
> A fairly good definition is:
> 
> KFAIL == bug, in GDB, something to fix.
> XFAIL == bug, not in GDB (kernel, debug info, linker, ...), something to
> ignore.
> 
> > I guess I don't see the point in removing xfails from the testsuite:
> > it's useful information, it doesn't make regression testing any harder
> > (there, the main culprit is the !@#%# schedlock test), so why throw
> > that away?  If xfail has the wrong meaning, then change it to kfail;
> > if kfail also has the wrong meaning, then change the meaning of kfail.
> 
> People have been XFAILing bugs in GDB.  That is simply wrong.

We all agree about that.

Re: [patch/rfc] Remove all setup_xfail's from testsuite/gdb.mi/

[David Carlton]

On Thu, 24 Oct 2002 17:57:10 -0400, Andrew Cagney <ac131313@redhat.com> said:

> I think Elena mentioned KFAIL.

Yeah, she did, I just glossed over that the first time I read her
message.

> Any way, that is want you want to use - you'll need to bug report
> any failures though.

Will do.

> A fairly good definition is:

> KFAIL == bug, in GDB, something to fix.
> XFAIL == bug, not in GDB (kernel, debug info, linker, ...), something
> to ignore.

Ah, I see, thanks.  And then FAIL == regression in GDB?  Then
doubtless XFAIL is being overused, so they should be changed to KFAIL
or FAIL.

David Carlton
carlton@math.stanford.edu

Re: [patch/rfc] Remove all setup_xfail's from testsuite/gdb.mi/

[Andrew Cagney]

> On Thu, 24 Oct 2002 17:57:10 -0400, Andrew Cagney <ac131313@redhat.com> said:
> 
> 
>> I think Elena mentioned KFAIL.
> 
> 
> Yeah, she did, I just glossed over that the first time I read her
> message.
> 
> 
>> Any way, that is want you want to use - you'll need to bug report
>> any failures though.
> 
> 
> Will do.
> 
> 
>> A fairly good definition is:
> 
> 
>> KFAIL == bug, in GDB, something to fix.
>> XFAIL == bug, not in GDB (kernel, debug info, linker, ...), something
>> to ignore.
> 
> 
> Ah, I see, thanks.  And then FAIL == regression in GDB?  Then
> doubtless XFAIL is being overused, so they should be changed to KFAIL
> or FAIL.

Right.  They should be changed back to FAILs (I don't see anyone 
complaining about how the removal of XFAILs also the passes to increase :-).

Then people can start auding all the FAILs (not just the XFAILs) and 
mark up any bugs that are not going to be immediatly fixed.

Andrew

Re: [patch/rfc] Remove all setup_xfail's from testsuite/gdb.mi/

[Andrew Cagney]

[-- Attachment #1: Type: text/plain, Size: 215 bytes --]

Fernando,

What is the resolution of this thread?
http://sources.redhat.com/ml/gdb-patches/2002-10/msg00508.html
(you'll probably want to read all of it :-)
(I'm now going back and emptying my mail box :-()

Andrew

[-- Attachment #2: mailbox-message://ac131313@movemail/fsf/gdb/patches#24629765 --]
[-- Type: message/rfc822, Size: 18092 bytes --]

[-- Attachment #2.1.1: Type: text/plain, Size: 1108 bytes --]

Hello,

GDB's testsuite is known to be full of xfails that are really kfails or 
testsuite bugs.  Rather than try to audit each xfail in turn, the 
proposal as been to rip out all the xfails (creating a clean slate) and 
start marking up the tests from scratch - two steps forward but first 
one step back.

I figure I might as well try to get the ball rolling on this and find 
out just how much real resistance there is going to be to a change like 
this.  To that end, this removes all xfail's from the gdb.mi testsuite. 
  Similar tests, for the other directories, would follow.

On a GNU/Linux i386 system (Red Hat 7.2 I think) I see:

Before:

                 === gdb Summary ===

# of expected passes            8298
# of unexpected failures        60
# of unexpected successes       10
# of expected failures          172
# of unsupported tests          2

After:

                 === gdb Summary ===

# of expected passes            8298
# of unexpected failures        88
# of unexpected successes       10
# of expected failures          144
# of unsupported tests          2

Comments?

Andrew

[-- Attachment #2.1.2: diffs --]
[-- Type: text/plain, Size: 13873 bytes --]

2002-10-24  Andrew Cagney  <cagney@redhat.com>

	* mi-var-block.exp, mi1-var-block.exp: Remove all uses of
	setup_xfail and clear_xfail.
	* mi-var-cmd.exp, mi1-var-cmd.exp: Ditto.
	* mi1-var-display.exp, mi-var-display.exp: Ditto.
	* mi-var-child.exp, mi1-var-child.exp: Ditto.
	* mi-break.exp, mi1-break.exp: Remove all uses of setup_xfail.
	Update copyright.

Index: gdb.mi/mi-break.exp
===================================================================
RCS file: /cvs/src/src/gdb/testsuite/gdb.mi/mi-break.exp,v
retrieving revision 1.5
diff -u -r1.5 mi-break.exp
--- gdb.mi/mi-break.exp	27 Jun 2001 17:27:07 -0000	1.5
+++ gdb.mi/mi-break.exp	24 Oct 2002 18:07:01 -0000
@@ -1,4 +1,4 @@
-#   Copyright 1999 Free Software Foundation, Inc.
+# Copyright 1999, 2002 Free Software Foundation, Inc.
 
 # This program is free software; you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by
@@ -101,27 +101,22 @@
     # -break-insert -r .*llee
     # -break-list
 
-    setup_xfail "*-*-*"
     mi_gdb_test "122-break-insert -r main" \
              "122\\^done,bkpt=\{number=\"5\",addr=\"$hex\",file=\".*basics.c\",line=\"32\"\}" \
              "break-insert -r operation"
 
-    setup_xfail "*-*-*"
     mi_gdb_test "133-break-insert -r callee2" \
              "133\\^done,bkpt=\{number=\"6\",addr=\"$hex\",file=\".*basics.c\",line=\"22\"\}" \
              "insert breakpoint with regexp callee2"
 
-    setup_xfail "*-*-*"
     mi_gdb_test "144-break-insert -r callee" \
 	    "144\\^done,bkpt=\{number=\"7\",addr=\"$hex\",file=\".*basics.c\",line=\"27\"\},bkpt=\{number=\"8\",addr=\"$hex\",file=\".*basics.c\",line=\"22\"\},bkpt=\{number=\"9\",addr=\"$hex\",file=\".*basics.c\",line=\"17\"\},bkpt=\{number=\"10\",addr=\"$hex\",file=\".*basics.c\",line=\"8\"\}" \
              "insert breakpoint with regexp callee"
 
-    setup_xfail "*-*-*"
     mi_gdb_test "155-break-insert -r \.\*llee" \
 	    "155\\^done,bkpt=\{number=\"11\",addr=\"$hex\",file=\".*basics.c\",line=\"27\"\},bkpt=\{number=\"12\",addr=\"$hex\",file=\".*basics.c\",line=\"22\"\},bkpt=\{number=\"13\",addr=\"$hex\",file=\".*basics.c\",line=\"17\"\},bkpt=\{number=\"14\",addr=\"$hex\",file=\".*basics.c\",line=\"8\"\}" \
              "insert breakpoint with regexp .*llee"
 
-    setup_xfail "*-*-*"
     mi_gdb_test "166-break-list" \
 	    "1\\\^done,BreakpointTable=\{nr_rows=\".\",nr_cols=\".\",hdr=\\\[\{width=\".*\",alignment=\".*\",col_name=\"number\",colhdr=\"Num\"\}.*colhdr=\"Type\".*colhdr=\"Disp\".*colhdr=\"Enb\".*colhdr=\"Address\".*colhdr=\"What\".*\\\],body=\\\[bkpt=\{number=\"5\",type=\"breakpoint\",disp=\"keep\",enabled=\"y\",addr=\"$hex\",func=\"main\",file=\".*basics.c\",line=\"32\",times=\"0\"\},.*\}\\\]\}" \
                 "list of breakpoints"
Index: gdb.mi/mi-var-block.exp
===================================================================
RCS file: /cvs/src/src/gdb/testsuite/gdb.mi/mi-var-block.exp,v
retrieving revision 1.8
diff -u -r1.8 mi-var-block.exp
--- gdb.mi/mi-var-block.exp	3 Oct 2002 20:03:55 -0000	1.8
+++ gdb.mi/mi-var-block.exp	24 Oct 2002 18:07:01 -0000
@@ -142,17 +142,13 @@
 # Test: c_variable-3.7
 # Desc: check that outer foo in scope and inner foo out of scope
 # Note: also a known gdb problem
-setup_xfail *-*-*
 mi_gdb_test "-var-update inner_foo" \
 	"\\^done,changelist=\{FIXME\}" \
 	"update inner_foo: should be out of scope: KNOWN PROBLEM"
-clear_xfail *-*-*
 
-setup_xfail *-*-*
 mi_gdb_test "-var-evaluate-expression inner_foo" \
 	"\\^done,value=\{FIXME\}" \
 	"evaluate inner_foo: should be out of scope: KNOWN PROBLEM"
-clear_xfail *-*-*
 
 mi_gdb_test "-var-update foo" \
 	"\\^done,changelist=\\\[\\\]" \
@@ -173,11 +169,9 @@
 
 # Test: c_variable-3.8
 # Desc: check that foo2 out of scope (known gdb problem)
-setup_xfail *-*-*
 mi_gdb_test "-var-update foo2" \
 	"\\^done,changelist=\{FIXME\}" \
 	"update foo2: should be out of scope: KNOWN PROBLEM"
-clear_xfail *-*-*
 
 # step to "cb = 21;"
 send_gdb "-exec-step\n"
@@ -191,16 +185,12 @@
 
 # Test: c_variable-3.9
 # Desc: check that only cb is in scope (known gdb problem)
-setup_xfail *-*-*
 mi_gdb_test "-var-update foo2" \
 	"\\^done,changelist=\\\[FIXME\\\]" \
 	"update foo2 should be out of scope: KNOWN PROBLEM"
-clear_xfail *-*-*
-setup_xfail *-*-*
 mi_gdb_test "-var-update foo" \
 	"\\^done,changelist=\{FIXME\}" \
 	"update foo should be out of scope: KNOWN PROBLEM"
-clear_xfail *-*-*
 mi_gdb_test "-var-update cb" \
 	"\\^done,changelist=\\\[\\\]" \
 	"update cb"
Index: gdb.mi/mi-var-child.exp
===================================================================
RCS file: /cvs/src/src/gdb/testsuite/gdb.mi/mi-var-child.exp,v
retrieving revision 1.12
diff -u -r1.12 mi-var-child.exp
--- gdb.mi/mi-var-child.exp	3 Oct 2002 20:03:55 -0000	1.12
+++ gdb.mi/mi-var-child.exp	24 Oct 2002 18:07:01 -0000
@@ -1176,11 +1176,9 @@
 # Test: c_variable-5.51
 # Desc: check that psnp->next->long_ptr (and [1].long_ptr) changed
 # Why does this have a FIXME?
-setup_xfail *-*-*
 mi_gdb_test "-var-update *" \
 	"FIXME\\^done,changelist=\\\[\{name=\"psnp->ptrs.0.next.long_ptr\",in_scope=\"true\",type_changed=\"false\"\}\\\]" \
 	"update all vars psnp->next->long_ptr (and 1.long_ptr) changed"
-clear_xfail *-*-*
 
 # This command produces this error message:
 # &"warning: varobj_list: assertion failed - mycount <> 0\n"
Index: gdb.mi/mi-var-cmd.exp
===================================================================
RCS file: /cvs/src/src/gdb/testsuite/gdb.mi/mi-var-cmd.exp,v
retrieving revision 1.14
diff -u -r1.14 mi-var-cmd.exp
--- gdb.mi/mi-var-cmd.exp	23 Oct 2002 23:57:28 -0000	1.14
+++ gdb.mi/mi-var-cmd.exp	24 Oct 2002 18:07:02 -0000
@@ -429,11 +429,9 @@
 #       special case that, since it is not what a human expects to
 #       see.
 
-setup_xfail *-*-*
 mi_gdb_test "-var-update *" \
 	"\\^done,changelist=\{FIXME: WHAT IS CORRECT HERE\}" \
 	"update all vars: changed FIXME"
-clear_xfail *-*-*
 
 mi_step_to "subroutine1" "\{name=\"i\",value=\".*\"\},\{name=\"l\",value=\".*\"\}" \
 	"var-cmd.c" "148" "step at subroutine1 (2)"
Index: gdb.mi/mi-var-display.exp
===================================================================
RCS file: /cvs/src/src/gdb/testsuite/gdb.mi/mi-var-display.exp,v
retrieving revision 1.8
diff -u -r1.8 mi-var-display.exp
--- gdb.mi/mi-var-display.exp	17 Sep 2002 20:30:24 -0000	1.8
+++ gdb.mi/mi-var-display.exp	24 Oct 2002 18:07:02 -0000
@@ -491,13 +491,11 @@
 	"\\^done,name=\"e\",numchild=\"0\",type=\"enum foo\"" \
 	"create local variable e"
 
-setup_xfail "*-*-*"
 # Test: c_variable-7.51
 # Desc: value of e
 mi_gdb_test "-var-evaluate-expression e" \
 	"\\^done,value=\"FIXME\"" \
 	"eval variable e"
-clear_xfail "*-*-*"
 
 # Test: c_variable-7.52
 # Desc: type of e
@@ -529,13 +527,11 @@
 	"\\^done,name=\"anone\",numchild=\"0\",type=\"enum \{\\.\\.\\.\}\"" \
 	"create local variable anone"
 
-setup_xfail "*-*-*"
 # Test: c_variable-7.61
 # Desc: value of anone
 mi_gdb_test "-var-evaluate-expression anone" \
 	"\\^done,value=\"A\"" \
 	"eval variable anone"
-clear_xfail "*-*-*"
 
 
 # Test: c_variable-7.70
Index: gdb.mi/mi1-break.exp
===================================================================
RCS file: /cvs/src/src/gdb/testsuite/gdb.mi/mi1-break.exp,v
retrieving revision 1.2
diff -u -r1.2 mi1-break.exp
--- gdb.mi/mi1-break.exp	1 Oct 2002 16:01:37 -0000	1.2
+++ gdb.mi/mi1-break.exp	24 Oct 2002 18:07:02 -0000
@@ -1,4 +1,4 @@
-#   Copyright 1999 Free Software Foundation, Inc.
+# Copyright 1999, 2002 Free Software Foundation, Inc.
 
 # This program is free software; you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by
@@ -101,27 +101,22 @@
     # -break-insert -r .*llee
     # -break-list
 
-    setup_xfail "*-*-*"
     mi_gdb_test "122-break-insert -r main" \
              "122\\^done,bkpt=\{number=\"5\",addr=\"$hex\",file=\".*basics.c\",line=\"32\"\}" \
              "break-insert -r operation"
 
-    setup_xfail "*-*-*"
     mi_gdb_test "133-break-insert -r callee2" \
              "133\\^done,bkpt=\{number=\"6\",addr=\"$hex\",file=\".*basics.c\",line=\"22\"\}" \
              "insert breakpoint with regexp callee2"
 
-    setup_xfail "*-*-*"
     mi_gdb_test "144-break-insert -r callee" \
 	    "144\\^done,bkpt=\{number=\"7\",addr=\"$hex\",file=\".*basics.c\",line=\"27\"\},bkpt=\{number=\"8\",addr=\"$hex\",file=\".*basics.c\",line=\"22\"\},bkpt=\{number=\"9\",addr=\"$hex\",file=\".*basics.c\",line=\"17\"\},bkpt=\{number=\"10\",addr=\"$hex\",file=\".*basics.c\",line=\"8\"\}" \
              "insert breakpoint with regexp callee"
 
-    setup_xfail "*-*-*"
     mi_gdb_test "155-break-insert -r \.\*llee" \
 	    "155\\^done,bkpt=\{number=\"11\",addr=\"$hex\",file=\".*basics.c\",line=\"27\"\},bkpt=\{number=\"12\",addr=\"$hex\",file=\".*basics.c\",line=\"22\"\},bkpt=\{number=\"13\",addr=\"$hex\",file=\".*basics.c\",line=\"17\"\},bkpt=\{number=\"14\",addr=\"$hex\",file=\".*basics.c\",line=\"8\"\}" \
              "insert breakpoint with regexp .*llee"
 
-    setup_xfail "*-*-*"
     mi_gdb_test "166-break-list" \
 	    "1\\\^done,BreakpointTable=\{nr_rows=\".\",nr_cols=\".\",hdr=\\\[\{width=\".*\",alignment=\".*\",col_name=\"number\",colhdr=\"Num\"\}.*colhdr=\"Type\".*colhdr=\"Disp\".*colhdr=\"Enb\".*colhdr=\"Address\".*colhdr=\"What\".*\\\],body=\\\[bkpt=\{number=\"5\",type=\"breakpoint\",disp=\"keep\",enabled=\"y\",addr=\"$hex\",func=\"main\",file=\".*basics.c\",line=\"32\",times=\"0\"\},.*\}\\\]\}" \
                 "list of breakpoints"
Index: gdb.mi/mi1-var-block.exp
===================================================================
RCS file: /cvs/src/src/gdb/testsuite/gdb.mi/mi1-var-block.exp,v
retrieving revision 1.2
diff -u -r1.2 mi1-var-block.exp
--- gdb.mi/mi1-var-block.exp	1 Oct 2002 16:01:37 -0000	1.2
+++ gdb.mi/mi1-var-block.exp	24 Oct 2002 18:07:02 -0000
@@ -142,17 +142,13 @@
 # Test: c_variable-3.7
 # Desc: check that outer foo in scope and inner foo out of scope
 # Note: also a known gdb problem
-setup_xfail *-*-*
 mi_gdb_test "-var-update inner_foo" \
 	"\\^done,changelist=\{FIXME\}" \
 	"update inner_foo: should be out of scope: KNOWN PROBLEM"
-clear_xfail *-*-*
 
-setup_xfail *-*-*
 mi_gdb_test "-var-evaluate-expression inner_foo" \
 	"\\^done,value=\{FIXME\}" \
 	"evaluate inner_foo: should be out of scope: KNOWN PROBLEM"
-clear_xfail *-*-*
 
 mi_gdb_test "-var-update foo" \
 	"\\^done,changelist=\{\}" \
@@ -173,11 +169,9 @@
 
 # Test: c_variable-3.8
 # Desc: check that foo2 out of scope (known gdb problem)
-setup_xfail *-*-*
 mi_gdb_test "-var-update foo2" \
 	"\\^done,changelist=\{FIXME\}" \
 	"update foo2: should be out of scope: KNOWN PROBLEM"
-clear_xfail *-*-*
 
 # step to "cb = 21;"
 send_gdb "-exec-step\n"
@@ -191,16 +185,12 @@
 
 # Test: c_variable-3.9
 # Desc: check that only cb is in scope (known gdb problem)
-setup_xfail *-*-*
 mi_gdb_test "-var-update foo2" \
 	"\\^done,changelist=\{FIXME\}" \
 	"update foo2 should be out of scope: KNOWN PROBLEM"
-clear_xfail *-*-*
-setup_xfail *-*-*
 mi_gdb_test "-var-update foo" \
 	"\\^done,changelist=\{FIXME\}" \
 	"update foo should be out of scope: KNOWN PROBLEM"
-clear_xfail *-*-*
 mi_gdb_test "-var-update cb" \
 	"\\^done,changelist=\{\}" \
 	"update cb"
Index: gdb.mi/mi1-var-child.exp
===================================================================
RCS file: /cvs/src/src/gdb/testsuite/gdb.mi/mi1-var-child.exp,v
retrieving revision 1.2
diff -u -r1.2 mi1-var-child.exp
--- gdb.mi/mi1-var-child.exp	1 Oct 2002 16:01:37 -0000	1.2
+++ gdb.mi/mi1-var-child.exp	24 Oct 2002 18:07:03 -0000
@@ -1176,11 +1176,9 @@
 # Test: c_variable-5.51
 # Desc: check that psnp->next->long_ptr (and [1].long_ptr) changed
 # Why does this have a FIXME?
-setup_xfail *-*-*
 mi_gdb_test "-var-update *" \
 	"FIXME\\^done,changelist=\{name=\"psnp->ptrs.0.next.long_ptr\",in_scope=\"true\",type_changed=\"false\"\}" \
 	"update all vars psnp->next->long_ptr (and 1.long_ptr) changed"
-clear_xfail *-*-*
 
 # This command produces this error message:
 # &"warning: varobj_list: assertion failed - mycount <> 0\n"
Index: gdb.mi/mi1-var-cmd.exp
===================================================================
RCS file: /cvs/src/src/gdb/testsuite/gdb.mi/mi1-var-cmd.exp,v
retrieving revision 1.3
diff -u -r1.3 mi1-var-cmd.exp
--- gdb.mi/mi1-var-cmd.exp	23 Oct 2002 23:57:28 -0000	1.3
+++ gdb.mi/mi1-var-cmd.exp	24 Oct 2002 18:07:03 -0000
@@ -429,11 +429,9 @@
 #       special case that, since it is not what a human expects to
 #       see.
 
-setup_xfail *-*-*
 mi_gdb_test "-var-update *" \
 	"\\^done,changelist=\{FIXME: WHAT IS CORRECT HERE\}" \
 	"update all vars: changed FIXME"
-clear_xfail *-*-*
 
 mi_step_to "subroutine1" "\{name=\"i\",value=\".*\"\},\{name=\"l\",value=\".*\"\}" \
 	"var-cmd.c" "148" "step at subroutine1 (2)"
Index: gdb.mi/mi1-var-display.exp
===================================================================
RCS file: /cvs/src/src/gdb/testsuite/gdb.mi/mi1-var-display.exp,v
retrieving revision 1.2
diff -u -r1.2 mi1-var-display.exp
--- gdb.mi/mi1-var-display.exp	1 Oct 2002 16:01:37 -0000	1.2
+++ gdb.mi/mi1-var-display.exp	24 Oct 2002 18:07:03 -0000
@@ -491,13 +491,11 @@
 	"\\^done,name=\"e\",numchild=\"0\",type=\"enum foo\"" \
 	"create local variable e"
 
-setup_xfail "*-*-*"
 # Test: c_variable-7.51
 # Desc: value of e
 mi_gdb_test "-var-evaluate-expression e" \
 	"\\^done,value=\"FIXME\"" \
 	"eval variable e"
-clear_xfail "*-*-*"
 
 # Test: c_variable-7.52
 # Desc: type of e
@@ -529,13 +527,11 @@
 	"\\^done,name=\"anone\",numchild=\"0\",type=\"enum \{\\.\\.\\.\}\"" \
 	"create local variable anone"
 
-setup_xfail "*-*-*"
 # Test: c_variable-7.61
 # Desc: value of anone
 mi_gdb_test "-var-evaluate-expression anone" \
 	"\\^done,value=\"A\"" \
 	"eval variable anone"
-clear_xfail "*-*-*"
 
 
 # Test: c_variable-7.70

Re: [patch/rfc] Remove all setup_xfail's from testsuite/gdb.mi/

[Fernando Nasser]

Andrew Cagney wrote:
> Fernando,
> 
> What is the resolution of this thread?
> http://sources.redhat.com/ml/gdb-patches/2002-10/msg00508.html
> (you'll probably want to read all of it :-)
> (I'm now going back and emptying my mail box :-()
> 
> Andrew
> 

Hi Andrew,

I am just starting to think about this, so bear with me while I think aloud a 
little bit :-)  There are lots of good points made on that discussion and I 
don't think this is a clear cut yet or no, lets vote, or anything like that.  I 
am trying to see if I can collect all good suggestions and propose a course of 
action.

It seems that a widespread concern is not to through the baby with the water. 
On the other hand there is the feeling that we do not act now nothing will ever 
happen.  And we all agree that many xfails were added incorrectly.

Independently of the "big plan", one thing is clear.  If you know that a xfail 
is incorrect (I mean, know for good), then it should be ripped off. But read on...

Maybe we may not be in full agreement here, but I think it would be nice to 
enter a bug report even if does not include a full analysis -- that should come 
eventually.  Something like "something is wrong with the '--xxx-yyy' command 
output" may eventually cause someone to look into it.  And you can make the 
xfail a kfail in that case, which is much better.  But I know it is much more 
work than just removing the xfail and letting the test fail -- you would have to 
enter 28 bug reports just for the MI tests.  Still sounds like a better approach 
to me, but there is no consensus on it.

W.r.t. the non-MI xfails, looking at Michael's reports, more than half of them 
seem to be related to stabs problems in different gcc versions.  They should be 
conditional to stabs and the compiler version and are not, but otherwise they 
are valid xfails (by definition).  There are a couple of XPASSes that are always 
there, so we have at least two obsolete XFAILs that we can get rid off (just 
look at a dwarf-2 test result -- those 2).

The remaining ones are 71 (take or add one or two).  We can use the gcc 3.2.1 
results on RHL 8.0 to determine which ones they are.  If seven of us (the 
magnificent seven? :-) agree to look into 10 of those each and enter even a 
basic bug report we can kfail them all and use the bug database to track the 
resolution.

As I said, this is just a brainstorm.  Please comment.

Regards to all,
Fernando

P.S.: Michael's table is very revealing.  See, for instance,
    http://www.shout.net/~mec/sunday/2003-01-09/Counts.html




-- 
Fernando Nasser
Red Hat - Toronto                       E-Mail:  fnasser@redhat.com
2323 Yonge Street, Suite #300
Toronto, Ontario   M4P 2C9

Re: [patch/rfc] Remove all setup_xfail's from testsuite/gdb.mi/

[Andrew Cagney]

> Andrew Cagney wrote:
> Fernando,
> 
> What is the resolution of this thread?
> http://sources.redhat.com/ml/gdb-patches/2002-10/msg00508.html
> (you'll probably want to read all of it :-)
> (I'm now going back and emptying my mail box :-()
> 
> Andrew
> 
> 
> Hi Andrew,
> 
> I am just starting to think about this, so bear with me while I think aloud a little bit :-)  There are lots of good points made on that discussion and I don't think this is a clear cut yet or no, lets vote, or anything like that.  I am trying to see if I can collect all good suggestions and propose a course of action.
> 
> It seems that a widespread concern is not to through the baby with the water. On the other hand there is the feeling that we do not act now nothing will ever happen.  And we all agree that many xfails were added incorrectly.
> 
> Independently of the "big plan", one thing is clear.  If you know that a xfail is incorrect (I mean, know for good), then it should be ripped off. But read on...
> 
> Maybe we may not be in full agreement here, but I think it would be nice to enter a bug report even if does not include a full analysis -- that should come eventually.  Something like "something is wrong with the '--xxx-yyy' command output" may eventually cause someone to look into it.  And you can make the xfail a kfail in that case, which is much better.  But I know it is much more work than just removing the xfail and letting the test fail -- you would have to enter 28 bug reports just for the MI tests.  Still sounds like a better approach to me, but there is no consensus on it.

Some things I think worth noting.

In this process, I don't think it is reasonable make it a requirement 
that people fully kfail/analize old badly specified xfails.  Rather I 
think it is only reasonable to ask people to ensure that new tests, or 
changes to existing tests, meet the new requirement (there must be a bug 
report).

An initial bar is established, and then, over time, that bar is raised.

The difference is important.  It avoids the problem of developement 
stalling because of the impossible requirement fixing all the old code 
before the new code can be approved.

As a specific example, the i386 has an apparently low failure rate. 
That rate is badly misleading and the real number of failures is much 
higher :-(  It's just that those failures have been [intentionally] 
camoflaged using xfail.  It would be unfortunate if people, for the 
i386, tried to use that false result (almost zero fails) when initally 
setting the bar.

This is also why I think the xfail's should simply be yanked.  It acts 
as a one time reset of gdb's test results, restoring them to their true 
values.   While this may cause the bar to start out lower than some 
would like,  I think that is far better and far more realistic than 
trying to start with a bar falsely set too high.

Andrew

Re: [patch/rfc] Remove all setup_xfail's from testsuite/gdb.mi/

[Daniel Jacobowitz]

On Thu, Jan 16, 2003 at 11:53:29AM -0500, Andrew Cagney wrote:
> >Andrew Cagney wrote:
> >Fernando,
> >
> >What is the resolution of this thread?
> >http://sources.redhat.com/ml/gdb-patches/2002-10/msg00508.html
> >(you'll probably want to read all of it :-)
> >(I'm now going back and emptying my mail box :-()
> >
> >Andrew
> >
> >
> >Hi Andrew,
> >
> >I am just starting to think about this, so bear with me while I think 
> >aloud a little bit :-)  There are lots of good points made on that 
> >discussion and I don't think this is a clear cut yet or no, lets vote, or 
> >anything like that.  I am trying to see if I can collect all good 
> >suggestions and propose a course of action.
> >
> >It seems that a widespread concern is not to through the baby with the 
> >water. On the other hand there is the feeling that we do not act now 
> >nothing will ever happen.  And we all agree that many xfails were added 
> >incorrectly.
> >
> >Independently of the "big plan", one thing is clear.  If you know that a 
> >xfail is incorrect (I mean, know for good), then it should be ripped off. 
> >But read on...
> >
> >Maybe we may not be in full agreement here, but I think it would be nice 
> >to enter a bug report even if does not include a full analysis -- that 
> >should come eventually.  Something like "something is wrong with the 
> >'--xxx-yyy' command output" may eventually cause someone to look into it.  
> >And you can make the xfail a kfail in that case, which is much better.  
> >But I know it is much more work than just removing the xfail and letting 
> >the test fail -- you would have to enter 28 bug reports just for the MI 
> >tests.  Still sounds like a better approach to me, but there is no 
> >consensus on it.
> 
> Some things I think worth noting.
> 
> In this process, I don't think it is reasonable make it a requirement 
> that people fully kfail/analize old badly specified xfails.  Rather I 
> think it is only reasonable to ask people to ensure that new tests, or 
> changes to existing tests, meet the new requirement (there must be a bug 
> report).
> 
> An initial bar is established, and then, over time, that bar is raised.
> 
> The difference is important.  It avoids the problem of developement 
> stalling because of the impossible requirement fixing all the old code 
> before the new code can be approved.

I don't think making it a requirement that go out and analyze all the
existing XFAILs is reasonable, although it is patently something we
need to do.  That's not the same as ripping them out and introducing
failures in the test results without addressing those failures.

> As a specific example, the i386 has an apparently low failure rate. 
> That rate is badly misleading and the real number of failures is much 
> higher :-(  It's just that those failures have been [intentionally] 
> camoflaged using xfail.  It would be unfortunate if people, for the 
> i386, tried to use that false result (almost zero fails) when initally 
> setting the bar.

Have you reviewed the list of XFAILs?  None of them are related to the
i386.  One, in signals.exp, is either related to GDB's handling of
signals or to a longstanding limitation in most operating system
kernels, depending how you look at it.  The rest are pretty much
platform independent.

> This is also why I think the xfail's should simply be yanked.  It acts 
> as a one time reset of gdb's test results, restoring them to their true 
> values.   While this may cause the bar to start out lower than some 
> would like,  I think that is far better and far more realistic than 
> trying to start with a bar falsely set too high.

This is a _regression_ testsuite.  I've been trying for months to get
it down to zero failures without compromising its integrity, and I've
just about done it for one target, by judicious use of KFAILs (and
fixing bugs!).  The existing XFAILs all look to me like either
legitimate XFAILs or things that should be KFAILed.  If you're going
to rip up my test results, please sort them accordingly first.

It doesn't need to be done all at once.  We can put markers in .exp
files saying "xfails audited".  But I think that we should audit
individual files, not yank madly.

Am I the only one who considers well-categorized results important?  If
you introduce seventy failures, then that's another couple of weeks I
can't just look at the results, see "oh, two failures in threads and
that's it, I didn't break anything".

-- 
Daniel Jacobowitz
MontaVista Software                         Debian GNU/Linux Developer

Re: [patch/rfc] Remove all setup_xfail's from testsuite/gdb.mi/

[Andrew Cagney]

> I don't think making it a requirement that go out and analyze all the
> existing XFAILs is reasonable, although it is patently something we
> need to do.  That's not the same as ripping them out and introducing
> failures in the test results without addressing those failures.



>> As a specific example, the i386 has an apparently low failure rate. 
>> That rate is badly misleading and the real number of failures is much 
>> higher :-(  It's just that those failures have been [intentionally] 
>> camoflaged using xfail.  It would be unfortunate if people, for the 
>> i386, tried to use that false result (almost zero fails) when initally 
>> setting the bar.
> 
> 
> Have you reviewed the list of XFAILs?  None of them are related to the
> i386.  One, in signals.exp, is either related to GDB's handling of
> signals or to a longstanding limitation in most operating system
> kernels, depending how you look at it.  The rest are pretty much
> platform independent.

I've been through the files and looked at the actual xfail markings. 
They are dominated by what look like cpu specific cases (rs6000 and HP 
are especially bad at this).

I've also noticed cases where simply hanking the xfail doesn't make 
sense - when the failure has already been analized (easy to spot since 
they are conditional on the debug info or compiler version).

>> This is also why I think the xfail's should simply be yanked.  It acts 
>> as a one time reset of gdb's test results, restoring them to their true 
>> values.   While this may cause the bar to start out lower than some 
>> would like,  I think that is far better and far more realistic than 
>> trying to start with a bar falsely set too high.
> 
> 
> This is a _regression_ testsuite.  I've been trying for months to get
> it down to zero failures without compromising its integrity, and I've
> just about done it for one target, by judicious use of KFAILs (and
> fixing bugs!).  The existing XFAILs all look to me like either
> legitimate XFAILs or things that should be KFAILed.  If you're going
> to rip up my test results, please sort them accordingly first.

No one is ripping up your individual and personal test results.

Several years ago some maintainers were intentionally xfailing many of 
the bugs that they had no intention of fixing.  That was wrong, and that 
needs to be fixed.

An unfortunate consequence of that action is that the zero you've been 
shooting for is really only a local minimum.  The real zero is further 
out, that zero was a mirage :-(

> It doesn't need to be done all at once.  We can put markers in .exp
> files saying "xfails audited".  But I think that we should audit
> individual files, not yank madly.

(which reminds me, the existing xfail reference to bug reports need to 
be ripped out - they refer to Red Hat and HP bug databases :-().

> Am I the only one who considers well-categorized results important?

Of course not.  All the good developers on this list take the test 
results, and their analysis, very seriously.

>  If
> you introduce seventy failures, then that's another couple of weeks I
> can't just look at the results, see "oh, two failures in threads and
> that's it, I didn't break anything".

People doing proper test analysis should be comparing the summary files 
and not the final numbers.  A summary analysis would show 70 XFAIL->FAIL 
changes, but no real regressions.

Anyway,

If the eixsting (bogus) xfail PR numbers are _all_ ripped out, and then 
the requirement for all new xfail's to include a corresponding bug 
report, I think there is a way forward.

Andrew

Re: [patch/rfc] Remove all setup_xfail's from testsuite/gdb.mi/

[Daniel Jacobowitz]

Once again, I feel the need to apologize for my tone.  I'm being too
sensitive about this.  Sorry... let's try this again.

On Thu, Jan 16, 2003 at 02:02:50PM -0500, Andrew Cagney wrote:
> 
> >I don't think making it a requirement that go out and analyze all the
> >existing XFAILs is reasonable, although it is patently something we
> >need to do.  That's not the same as ripping them out and introducing
> >failures in the test results without addressing those failures.
> 
> 
> 
> >>As a specific example, the i386 has an apparently low failure rate. 
> >>That rate is badly misleading and the real number of failures is much 
> >>higher :-(  It's just that those failures have been [intentionally] 
> >>camoflaged using xfail.  It would be unfortunate if people, for the 
> >>i386, tried to use that false result (almost zero fails) when initally 
> >>setting the bar.
> >
> >
> >Have you reviewed the list of XFAILs?  None of them are related to the
> >i386.  One, in signals.exp, is either related to GDB's handling of
> >signals or to a longstanding limitation in most operating system
> >kernels, depending how you look at it.  The rest are pretty much
> >platform independent.
> 
> I've been through the files and looked at the actual xfail markings. 
> They are dominated by what look like cpu specific cases (rs6000 and HP 
> are especially bad at this).

Most of these which are *-*-* are actually generic, even when they have
HP markings, in my experience.

> I've also noticed cases where simply hanking the xfail doesn't make 
> sense - when the failure has already been analized (easy to spot since 
> they are conditional on the debug info or compiler version).

Definitely.  On the other hand, the particular choice of xfail
conditions is often really bogus.

> >>This is also why I think the xfail's should simply be yanked.  It acts 
> >>as a one time reset of gdb's test results, restoring them to their true 
> >>values.   While this may cause the bar to start out lower than some 
> >>would like,  I think that is far better and far more realistic than 
> >>trying to start with a bar falsely set too high.
> >
> >
> >This is a _regression_ testsuite.  I've been trying for months to get
> >it down to zero failures without compromising its integrity, and I've
> >just about done it for one target, by judicious use of KFAILs (and
> >fixing bugs!).  The existing XFAILs all look to me like either
> >legitimate XFAILs or things that should be KFAILed.  If you're going
> >to rip up my test results, please sort them accordingly first.
> 
> No one is ripping up your individual and personal test results.
> 
> Several years ago some maintainers were intentionally xfailing many of 
> the bugs that they had no intention of fixing.  That was wrong, and that 
> needs to be fixed.
> 
> An unfortunate consequence of that action is that the zero you've been 
> shooting for is really only a local minimum.  The real zero is further 
> out, that zero was a mirage :-(

Close, close... what I'm trying to avoid is a local minimum.  The zero
I've been shooting for should be a local _plateau_.  Then we continue
going down as XFAIL/KFAILs are fixed/analyzed/recategorized/everything
else that happens to bugs when they go to bug heaven.

> >It doesn't need to be done all at once.  We can put markers in .exp
> >files saying "xfails audited".  But I think that we should audit
> >individual files, not yank madly.
> 
> (which reminds me, the existing xfail reference to bug reports need to 
> be ripped out - they refer to Red Hat and HP bug databases :-().

Ayup.

> > If
> >you introduce seventy failures, then that's another couple of weeks I
> >can't just look at the results, see "oh, two failures in threads and
> >that's it, I didn't break anything".
> 
> People doing proper test analysis should be comparing the summary files 
> and not the final numbers.  A summary analysis would show 70 XFAIL->FAIL 
> changes, but no real regressions.

I do, but it's exceedingly convenient for, e.g., automated testing
purposes to have the actual number of FAILs come out as zero and each
bug to be otherwise accounted for.  What I would like to do is get to
that point, and then recategorize directly from:
  XFAIL->KFAIL
  random XFAIL->analyzed XFAIL
  XFAIL->PASS
etc. on a case-by-case basis.  I don't see any benefit from ripping out
the XFAILs wholesale and then analyzing them as we find the time; why
not (rip out and analyze) as we find the time?

> Anyway,
> 
> If the eixsting (bogus) xfail PR numbers are _all_ ripped out, and then 
> the requirement for all new xfail's to include a corresponding bug 
> report, I think there is a way forward.

This I definitely like.  "Cantfix"?

-- 
Daniel Jacobowitz
MontaVista Software                         Debian GNU/Linux Developer