Re: [testsuite] add gdb.cp/gdb1355.exp

[Michael Elizabeth Chastain <mec@shout.net>]

Okay, I'm willing to make it an XFAIL rather than a KFAIL.
I remember those threads.  The XFAIL feels odd but it does stand
for "external" and this bug is indeed a textbook external bug.

As far as making it a FAIL goes, I think David's argument comes down to:
people pay attention to FAIL, but not to XFAIL or KFAIL.  I really don't
like that situation.  I see the question as: whether to fight the
situation and use more accurate XFAIL/KFAIL rather than generic FAIL, or
to acquiesce to situation and give people what they expect to see.
I favor the former.

> Scenario 1: The bug is fixed.  A year later, the bug in question
> regresses.  But, if you just run 'make check', this isn't so obvious:
> no message gets printed in the output, and while the numbers at the
> end of the output will change, those numbers fluctuate for lots of
> reasons.  So it will only be noticed by somebody who actually looks at
> gdb.sum (or does the moral equivalent); ideally that will happen, but
> I'd rather not count on it (especially if the bug is on an obscure
> platform).

I'm willing to count on it.  Because if they just grep for '^FAIL'
then they will miss any tests that barf out completely with an
ERROR at the beginning and abort.  I have little sympathy for
people who analyze gdb.sum that way.

dc> Scenario 2: The fix works on platform A, but not on platform B.  And
dc> nobody using platform B has been paying close enough attention to the
dc> discussion of the bug to realize that the test is supposed to start
dc> passing.  Then, nobody might ever realize that something's wrong:
dc> platform A users think that everything is fine (which is the case for
dc> them), and platform B have no indication from the testsuite that
dc> something is wrong and that the patch didn't work as indicated.

Ah, this is the old scenario "change the KFAIL/XFAIL to a FAIL"
makes people on platform B aware that something has regressed
(I always investigate transitions from KFAIL -> FAIL).
I have some sympathy for this.

dc> Which is why, in the situation at question, I like it that you have the
dc> explicit failure branches, containing an informative comment as to when
dc> the bug occurred - I just want the letter 'k' removed in a few places.
dc> :-)

So we have two proposals here: I'd like to make this an XFAIL,
and David would like to make it a FAIL.

Daniel, or anyone else, do you have a preference?

Michael C