Re: RFC: KFAIL DejaGnu patch

Re: RFC: KFAIL DejaGnu patch

[Michael Elizabeth Chastain]

Fernando Nasser writes:
> Michael Chastain: Will you be willing to help me test this?
> I've tried it and it seems to work.  Yours scripts must also
> be happy with it.

You got it.  I use dejagnu 1.4.2, so I'll be working from that base.
I'm willing to switch to sourceware dejagnu but I'd rather stay close
to the fsf dejagnu if I can.

> I was trying to find some test to try the setup_kfail on.

Try PR gdb/460.  It mentions one failure in the bug report.  You can
download my testsuite directory tarballs off the bug report and diff
the gdb.log files to pick up the regressions in gdb.base/condbreak.exp
and gdb.base/ena-dis-br.exp.

I have 2.0 gigabytes of test directories now.  I'm still experimenting
how to communicate stuff so that people can understand it.  The testsuite
directory tarballs are just another experiment, let me know how it works.

It's kinda depressing that the test suite and the bug database are
so disjoint.  I guess we're here to fix that.

Michael C

Re: RFC: KFAIL DejaGnu patch

[Michael Elizabeth Chastain]

Hi Rob,

I'm not looking for any changes in 1.4.3.  Think of this more as one
user's feedback.  If you get any good ideas from it, that's fine.

mec> ERRORs and WARNINGs: it would help if all ERRORs and WARNINGs ...
rob> This is probably a good idea.

Now that I know Dejagnu is actively maintained, maybe I'll pick this
one up myself.  I would have to learn more TCL but Dejagnu is not
that large.

mec> Duplicate test names: ...
rob> Um, I don't think it's the responsibility of DejaGnu to work around a
rob> poor programming practice.

Okay, that's fine.  I admit it's a poor practice.  It would have been
nice to move the anti-duplication code into a common place (that isn't
in my scripts, grin).

mec> TIMEOUT: ...
rob> If GDB has crashed, this is supposed to be UNRESOLVED, according to POSIX.
rob> UNRESOLVED is a test case that can't be finished, and needs a human to look
rob> at it.  UNTESTED is when there isn't support in the underlying OS or
rob> hardware to run a test case.

If gdb has crashed, indeed a human has to look at it.  But the tone of
the documentation is that the human is deciding whether the test passed
or not.  Actually a human is needed here to fix the tool, or perhaps fix
the testing environment.  I think TIMEOUT would be more accurate than
using UNRESOLVED for this.

Right now we use FAIL "... (timeout)" for this, so in principle, we're
getting the information we need.  So I can live without this if we really
need it.
  
mec> Split gdb.log file: when I look at gdb.log, I am usually interested in
mec> just one test script.  So I'd like to have a directory of gdb.log files,
mec> one per each test script.  I don't need the giant log.
rob> Hum... I might try that. Mind you, the log files were more for debugging
rob> purposes, than analysing them for test results.

Here is the situation: each week, I run the test suite on a few dozen
configurations.  I have a Perl script that looks for changes in the
results since the last test run.  When I find a negative change, then
I file a bug report.

The problem is what to put in that bug report so that an independent
person can use it effectively.  I'd like to include a copy of the gdb.log
section for that test script, so that people perusing the bug report
can see what's going on without downloading a big tarball from me first.

Another way to look at it: as projects get larger, the roles separate.
I'm planning to publish my scripts so that lots of people can run gdb
tests and mail in useful information.  My end goal is an Internet scale
"gnutest@home" facility, where developers can inject proposed patches and
get some kind of differential analysis over a diverse group of test beds.

rob> Any changes I make have to work generically, since the gdb team is a 
rob> only part of the DejaGnu user community.

Sure, I understand.

Thanks for being here and doing this.

Michael C

Re: RFC: KFAIL DejaGnu patch

[Rob Savoye]

On Tue, Apr 09, 2002 at 11:13:18AM -0500, Michael Elizabeth Chastain wrote:

> I'm not looking for any changes in 1.4.3.  Think of this more as one
> user's feedback.  If you get any good ideas from it, that's fine.
 
  I'm always looking for constructive comments from end users.

> Now that I know Dejagnu is actively maintained, maybe I'll pick this
> one up myself.  I would have to learn more TCL but Dejagnu is not
> that large.

  DejaGnu has been mostly actively maintained except for a period 5-6 years
ago where I needed a break... plus occasionally income generating projects
get in the way... Gotta pay the bills...

> the documentation is that the human is deciding whether the test passed
> or not.  Actually a human is needed here to fix the tool, or perhaps fix
> the testing environment.  I think TIMEOUT would be more accurate than
> using UNRESOLVED for this.
 
  According to POSIX though, UNRESOLVED is the test state, if GDB crashed
while executing a test case. If GDB crashed between test cases, then TIMEOUT,
would be appropriate. Maybe between tests, GDB should be checked to see if
it's crashed...

> Right now we use FAIL "... (timeout)" for this, so in principle, we're
> getting the information we need.  So I can live without this if we really

  In a sense FAIL is appropriate, since a test case fails if GDB crashes. :-)
the FAIL should automatically become an UNRESOLVED, unless somewhere in the
GDB tests the threshold is changed to turn off this feature. Mind you, there
is no real need to have the GDB test suite be POSIX conforming in behavior, so
whatever you do that works is fine.

> Another way to look at it: as projects get larger, the roles separate.
> I'm planning to publish my scripts so that lots of people can run gdb
> tests and mail in useful information.  My end goal is an Internet scale

  Maybe I should include your scripts in dejagnu/contrib. I used to have the
ancient ones we used at Cygnus there till they've kindof fallen out of use...

  Just as a note, DejaGnu now has some support for unit testing. You basically
include a dejagnu.h file, and your test case (typically on an embedded system)
spits out standard test states, and DejaGnu then parses those to produce the
final results. This is probably more useful for GCC or libstdc++ testing, but
I've been using this feature alot lately for customers. (which is why I wrote it)
I rarely write any new code these days without an associated test case.

	- rob -

Re: RFC: KFAIL DejaGnu patch

[Michael Elizabeth Chastain]

Daniel Jacobowitz writes:
> That would only be useful if we always marked all tests - which we're
> awful about.  continue {2} might be any number of different continue
> statements in the test.

Let me explain in more detail.

Right now there are tests with output like this:

  gdb.base/foo.exp: PASS: continue
  gdb.base/foo.exp: PASS: continue
  gdb.base/foo.exp: PASS: continue
  gdb.base/foo.exp: PASS: continue

I have to do something to make the test names unique.  So I behave
as if the input is this:

  gdb.base/foo.exp: PASS: continue
  gdb.base/foo.exp: PASS: continue {2}
  gdb.base/foo.exp: PASS: continue {3}
  gdb.base/foo.exp: PASS: continue {4}

This is flawed, because if someone adds or subtracts sections from
the test, the sequence numbers will get re-numbered, and I lose the
ability to compare across many runs.  As you point out, "continue {2}"
might be in different places depending on conditional execution and
so on.

But I have to do *something*.  If I just do "$hash{$name} = $result",
then the totals don't even add up correctly.

Michael C

Re: RFC: KFAIL DejaGnu patch

[Michael Elizabeth Chastain]

Rob Savoye writes:
> btw - Matt says he'll add the extra fields for ERROR and WARNING to the
> XML output, which he's hoping will be done as soon as he firms up the DTD.
> Anything else you need to have that be useful ?

Now there is an open ended question.  I'll ramble on it for a while.

ERRORs and WARNINGs: it would help if all ERRORs and WARNINGs went
through report_test and got treated like the other results, including
printing the name of the current test script.  Right now I pick this up
when I am lexing the gdb.sum file.

Duplicate test names: we have as many as 30 tests in the same file
with the same name (often named "continue").  Right now I add sequence
numbers to this when I am lexing the gdb.sum file:

  gdb.base/foo.exp: continue
  gdb.base/foo.exp: continue {2}
  gdb.base/foo.exp: continue {3}

It would be nice if DejaGnu did this automatically.

TIMEOUT: I would like to have a new test result of TIMEOUT for a test
that times out.  A TIMEOUT often indicates that gdb has crashed, and is
much more serious than an ordinary FAIL.

Split gdb.log file: when I look at gdb.log, I am usually interested in
just one test script.  So I'd like to have a directory of gdb.log files,
one per each test script.  I don't need the giant log.

Clean up the bug id stuff: the name "PRMS" is hard wired into Dejagnu!

I also have a lot of vague stuff related to scaling up the testing process.
Right now I run 30 configurations per week.  I use external scripts to
manage my archive of test results.  I don't want dejagnu to be in the
business of managing 100's of test scripts, but eventually I will want some
features to add more stuff to the log file (like a uuid, the email
address of the person who ran the test, the version of binutils and
gcc used when testing gdb, and lots of stuff like that).

All of this stuff is just me; the gdb group hasn't talked this over
and reached any kind of consensus yet.

Michael C

Re: RFC: KFAIL DejaGnu patch

[Daniel Jacobowitz]

On Mon, Apr 08, 2002 at 12:33:54PM -0500, Michael Elizabeth Chastain wrote:
> Rob Savoye writes:
> > btw - Matt says he'll add the extra fields for ERROR and WARNING to the
> > XML output, which he's hoping will be done as soon as he firms up the DTD.
> > Anything else you need to have that be useful ?
> 
> Now there is an open ended question.  I'll ramble on it for a while.
> 
> ERRORs and WARNINGs: it would help if all ERRORs and WARNINGs went
> through report_test and got treated like the other results, including
> printing the name of the current test script.  Right now I pick this up
> when I am lexing the gdb.sum file.
> 
> Duplicate test names: we have as many as 30 tests in the same file
> with the same name (often named "continue").  Right now I add sequence
> numbers to this when I am lexing the gdb.sum file:
> 
>   gdb.base/foo.exp: continue
>   gdb.base/foo.exp: continue {2}
>   gdb.base/foo.exp: continue {3}
> 
> It would be nice if DejaGnu did this automatically.

That would only be useful if we always marked all tests - which we're
awful about.  continue {2} might be any number of different continue
statements in the test.

-- 
Daniel Jacobowitz                           Carnegie Mellon University
MontaVista Software                         Debian GNU/Linux Developer

Re: RFC: KFAIL DejaGnu patch

[Rob Savoye]

On Mon, Apr 08, 2002 at 12:33:54PM -0500, Michael Elizabeth Chastain wrote:

> Now there is an open ended question.  I'll ramble on it for a while.
 
  Really, I might have stepped in a deep one there... :-) I was thinking of
changes the XML format, but these are mostly good suggestions.

> ERRORs and WARNINGs: it would help if all ERRORs and WARNINGs went
> through report_test and got treated like the other results, including
> printing the name of the current test script.  Right now I pick this up
> when I am lexing the gdb.sum file.

  This is probably a good idea.

> Duplicate test names: we have as many as 30 tests in the same file
> with the same name (often named "continue").  Right now I add sequence
> numbers to this when I am lexing the gdb.sum file:
> It would be nice if DejaGnu did this automatically.
 
  Um, I don't think it's the responsibility of DejaGnu to work around a
poor programming practice.

> TIMEOUT: I would like to have a new test result of TIMEOUT for a test
> that times out.  A TIMEOUT often indicates that gdb has crashed, and is
> much more serious than an ordinary FAIL.
 
  If GDB has crashed, this is supposed to be UNRESOLVED, according to POSIX.
UNRESOLVED is a test case that can't be finished, and needs a human to look
at it. UNTESTED is when there isn't support in the underlying OS or
hardware to run a test case.
  
> Split gdb.log file: when I look at gdb.log, I am usually interested in
> just one test script.  So I'd like to have a directory of gdb.log files,
> one per each test script.  I don't need the giant log.

  Hum... I might try that. Mind you, the log files were more for debugging
purposes, than analysing them for test results.

> features to add more stuff to the log file (like a uuid, the email
> address of the person who ran the test, the version of binutils and
> gcc used when testing gdb, and lots of stuff like that).

  Most of this is GDB testing specific, but I'll see what I can do. I can
probably add an "output_hook", that lets test suites add their own info
to the log.

> All of this stuff is just me; the gdb group hasn't talked this over
> and reached any kind of consensus yet.

  Any changes I make have to work generically, since the gdb team is a 
only part of the DejaGnu user community.

	- rob -

Re: RFC: KFAIL DejaGnu patch

[Michael Elizabeth Chastain]

Rob Savoye writes:
> Oops, I spaced that part. I think we'll need the kpass/kfail patch first,
> to them make those output routines spit out XML if the flag is set.

record_test puts out a <$type>$message</$type> for all types,
so it will automatically put out kpass/kfail when they are available.

The xml output section in log_summary has an explicit list
of "PASS FAIL XPASS XFAIL ..." so the xml patch needs to co-ordinate with
the kpass/kfail patch somehow.

Michael C

Re: RFC: KFAIL DejaGnu patch

[Michael Elizabeth Chastain]

More comments ...

gdb.c++/cplusfuncs.exp has a good place for some KFAIL's.  Grep for
hairyfunc5, hairyfunc6, hairyfunc7.  They are associated with PR gdb/19.
I added some setup_kfail's to that and it appears to work okay,
except that the bug id is not printed on xpass.

Okay, here's some proofreading comments.

Michael C

=== Distinction between XFAIL and KFAIL

An XFAIL is:

  A test is expected to fail in some environment(s) due to some
  bug in the environment ...

And a KFAIL is:

  A test is known to fail in some environment(s) due to a known bug
  in the tool being tested (identified by a bug id string).

I like this distinction.  The "K" letter means that it is a known problem
inside the tool, and the "X" letter means that it is an expected problem
outside the tool.

That makes it weird for a KFAIL to turn into an XPASS.  I've got test
results here with XPASS for the gcc v2 compilers and KFAIL for the gcc
v3 compilers.  I really want KFAIL/KPASS, not KFAIL/XPASS.

A specific note: proc record_test has this code:

  XPASS {
      set exit_status 1
      if { $xfail_prms != 0 } {
	  set message [concat $message "\t(PRMS $xfail_prms)"]
      }
  }

In fact that code could use a little refactoring, because
UNRESOLVED/UNSUPPORTD/UNTESTED all have the same code to pick up
kfail_prms and xfail_prms.

So an XPASS that comes from setup_xfail will have a PRMS id,
but an XPASS that comes from setup_kfail will not.

=== KFAIL as a way of hiding problems

Later on in the documentation:

  @item KFAIL
  ...
  This exists so that, after a bug is identified and properly registered
  in a bug tracking database (Gnats, for instance), the count of failures
  can be kept as zero.  Having zero has a baseline in all platforms allow
  the tool developers to immediately detect regressions caused by changes
  (which may affect some platforms and not others).

Conceptually, there is a disconnect here.  To me, KFAIL means: "there
is a problem report associated with this test".  The problem could be
minor, or it could be a showstopper.  But you say that KFAIL means:
"tool developers can ignore this test when they look for regressions".

I think this is fundamentally wrong.  The right way to look for
regressions is to compare before-and-after test runs.  gdb.sum files
are already pretty good for this; you can just diff them.

I see two practical problems that come out of this wrongness:

(1) If I find a showstopper regression, such as PR gdb/379, and I mark
    it with setup_kfail (as I should), then someone who is using the
    "# of FAILs" metric is not going to see the regression.

(2) If the test suite has a significant number of setup_kfail (and
    it should), then a regression bug may manifest as a transition from
    PASS -> KFAIL.  I will see that because my regression reports look
    at all transitions.  But someone looking at "# of FAILs" will not see
    this transition, so they won't see the regression.

I believe we already have this problem with XFAIL.  People conflate
the idea of "this test fails due to an external problem" with the idea
"this failure is not important enough to care about at this time".

=== ChangeLog entry

DejaGnu 1.4.2 has a ChangeLog, so the patch needs a ChangeLog entry.

=== Tests

testsuite/ needs some tests for the new KFAIL feature.

=== lib/dg.exp

lib/dg.exp has a bunch of XFAIL stuff, so it needs KFAIL stuff.

Re: RFC: KFAIL DejaGnu patch

[Michael Elizabeth Chastain]

Okay, I checked two gdb.log files chosen at random, and did
before-and-after diffs.  The only differences are the usual differences
in process id's and addresses and times and stuff (3000 lines per file!)

I'll proofread the code and documents tonight.

Michael C

Re: RFC: KFAIL DejaGnu patch

[Michael Elizabeth Chastain]

I have preliminary results.  On my full test bed (30 configurations,
native i686-pc-linux-gnu), there is no significant difference in
the gdb.sum files produced.

The "before" set is tcl 8.3.4 + expect 5.33 + dejagnu 1.4.2.
The "after"  set is tcl 8.3.4 + expect 5.33 + dejagnu 1.4.2 + fn kfail patch.

Later this weekend, I will pick two or three of the configurations at
random and look carefully at every difference in gdb.log.  There is a lot
of noise to wade through (machine addresses and process id's different
from run to run).

Also I will actually proofread the code.  So far I just skimmed it.

Michael C

Re: RFC: KFAIL DejaGnu patch

[Michael Elizabeth Chastain]

ac> To be honest, I think just removing all but the most recent (i.e. in 
ac> last two years) xfails might be for the best.

I'm okay with that.

I just changed my report generator so that everything except PASS is an
attention line.  So starting with the next report, the attention reports
are going to include KPASS, KFAIL, XPASS, XFAIL.  That increases the
attention table for gdb HEAD from 86 lines to 271 lines.  I think that
is manageable.

The difference tables have always been agnostic to the type of result,
and those are what I use most of the time.

Michael C

Re: RFC: KFAILs [Was: [RFA/mi-testsuite] XFAIL mi*-console.exp]

[Michael Elizabeth Chastain]

Fernando Nasser writes:

fna> KFAIL: could not run to marker1 (PRMS gdb/999)
fna> Would that make the scripts happy?

Err, I'm not sure if you mean the dejagnu scripts, or my scripts.
My scripts are happy with this format.

A lot of tests use "(...)" for various things, so the "PRMS"
and "gdb/NNN" bits need to be mandatory in order to pick out this
information from the noise.

fna> setup_kfail "gdb/999" *-*-*

Fine with me.  setup_kfail *-*-* "gdb/999" is fine with me as well.

fna> 4) Note that, when a test that was expected to fail due to a known
fna> bug suddenly starts to pass, it becomes a KPASS (as XFAILs do).

Okay, I added a KPASS column to my tables.

fna> I will do it in Perl (I still don't know how to programmatically access
fna> the Gnats database though).  But I have very little spare time, so I
fna> will not mind if someone that can do it sooner volunteers to do this.

You can access the Gnats database through a URL:

  http://sources.redhat.com/cgi-bin/gnatsweb.pl?database=gdb&cmd=view&pr=460

For programmatic access, there may already be a more suitable "cmd" than
"cmd=view", or someone may need to update gnatsweb.pl.

I volunteer to write Perl analysis scripts.  My test bed is almost all Perl,
and I am planning to release it.

Michael C

RFC: KFAIL DejaGnu patch

[Fernando Nasser]

[-- Attachment #1: Type: text/plain, Size: 913 bytes --]

Well, I am attaching the DejaGnu changes for KFAILs.

Rob: can you please take a look, specially in the documentation
changes and see if they are fit for the master repository?

Michael Chastain: Will you be willing to help me test this?
I've tried it and it seems to work.  Yours scripts must also
be happy with it.

Best regards,
Fernnado


P.S.: I was trying to find some test to try the setup_kfail on.  I went
looking for the setup_xfail *-*-* calls but most don't have any comment
why they are marked like that.  Then I've tried looking for the FAILs
and see if I could find a bug entry for those -- but it is not a
trivial task.  I guess we will need everybody's help and memories to
gradually have things properly documented and related to the bug
entries.




-- 
Fernando Nasser
Red Hat Canada Ltd.                     E-Mail:  fnasser@redhat.com
2323 Yonge Street, Suite #300
Toronto, Ontario   M4P 2C9

[-- Attachment #2: KFAIL.PATCH --]
[-- Type: text/plain, Size: 25409 bytes --]

Index: dejagnu/runtest.exp
===================================================================
RCS file: /cvs/src/src/dejagnu/runtest.exp,v
retrieving revision 1.7
diff -c -p -r1.7 runtest.exp
*** runtest.exp	2001/01/21 22:47:21	1.7
--- runtest.exp	2002/04/06 00:19:24
*************** set psum_file   "latest"	;# file name of
*** 49,56 ****
  
  set exit_status	0		;# exit code returned by this program
  
! set xfail_flag  0
! set xfail_prms	0
  set sum_file	""		;# name of the file that contains the summary log
  set base_dir	""		;# the current working directory
  set logname     ""		;# the users login name
--- 49,58 ----
  
  set exit_status	0		;# exit code returned by this program
  
! set xfail_flag  0		;# indicates that a failure is expected
! set xfail_prms	0		;# GNATS prms id number for this expected failure
! set kfail_flag  0		;# indicates that it is a known failure
! set kfail_prms	0		;# bug id for the description of the known failure 
  set sum_file	""		;# name of the file that contains the summary log
  set base_dir	""		;# the current working directory
  set logname     ""		;# the users login name
Index: dejagnu/doc/dejagnu.texi
===================================================================
RCS file: /cvs/src/src/dejagnu/doc/dejagnu.texi,v
retrieving revision 1.2
diff -c -p -r1.2 dejagnu.texi
*** dejagnu.texi	2001/07/17 16:37:27	1.2
--- dejagnu.texi	2002/04/06 00:19:28
*************** case:
*** 453,467 ****
  @item PASS
  A test has succeeded.  That is, it demonstrated that the assertion is true.
  
- @cindex XFAIL, avoiding for POSIX
- @item XFAIL
- @sc{posix} 1003.3 does not incorporate the notion of expected failures,
- so @code{PASS}, instead of @code{XPASS}, must also be returned for test
- cases which were expected to fail and did not.  This means that
- @code{PASS} is in some sense more ambiguous than if @code{XPASS} is also
- used.  For information on @code{XPASS} and @code{XFAIL}, see
- @ref{Invoking runtest,,Using @code{runtest}}.
- 
  @item FAIL
  @cindex failure, POSIX definition
  A test @emph{has} produced the bug it was intended to capture.  That is,
--- 453,458 ----
*************** it has demonstrated that the assertion i
*** 469,477 ****
  message is based on the test case only.  Other messages are used to
  indicate a failure of the framework.
  
- As with @code{PASS}, @sc{posix} tests must return @code{FAIL} rather
- than @code{XFAIL} even if a failure was expected.
- 
  @item UNRESOLVED
  @cindex ambiguity, required for POSIX
  A test produced indeterminate results.  Usually, this means the test
--- 460,465 ----
*************** real test case yet.
*** 516,523 ****
  @end ftable
  
  @noindent
! The only remaining output message left is intended to test features that
! are specified by the applicable @sc{posix} standard as conditional:
  
  @ftable @code
  @item UNSUPPORTED
--- 504,512 ----
  @end ftable
  
  @noindent
! The only remaining @sc{posix} output message left is intended to test
! features that are specified by the applicable @sc{posix} standard as
! conditional:
  
  @ftable @code
  @item UNSUPPORTED
*************** running the test case.  For example, a t
*** 529,543 ****
  @code{gethostname} would never work on a target board running only a
  boot monitor.
  @end ftable
!   
  DejaGnu uses the same output procedures to produce these messages for
  all test suites, and these procedures are already known to conform to
  @sc{posix} 1003.3.  For a DejaGnu test suite to conform to @sc{posix}
! 1003.3, you must avoid the @code{setup_xfail} procedure as described in
! the @code{PASS} section above, and you must be careful to return
  @code{UNRESOLVED} where appropriate, as described in the
  @code{UNRESOLVED} section above.
  
  @node Future Directions
  @section Future directions
  @cindex future directions
--- 518,594 ----
  @code{gethostname} would never work on a target board running only a
  boot monitor.
  @end ftable
! 
  DejaGnu uses the same output procedures to produce these messages for
  all test suites, and these procedures are already known to conform to
  @sc{posix} 1003.3.  For a DejaGnu test suite to conform to @sc{posix}
! 1003.3, you must avoid the @code{setup_xfail} and @code{setup_kfail}
! procedures (see below), and you must be careful to return
  @code{UNRESOLVED} where appropriate, as described in the
  @code{UNRESOLVED} section above.
+   
+ Besides the @sc{posix} messages, DejaGnu provides for variations of the
+ PASS and FAIL messages that can be helpful for the tool maintainers.
+ It must be noted, however, that this feature is not @sc{posix} 1003.3
+ compliant, so it should be avoided if compliance is necessary.
+ 
+ The additional messages are:
+ 
+ @ftable @code
+ 
+ @item XFAIL
+ A test is expected to fail in some environment(s) due to some bug
+ in the environment that we hope is fixed someday (but we can't do
+ nothing about as it is not a bug in the tool that we are testing).
+ The procedure @code{setup_xfail} is used to indicate that a failure
+ is expected.
+ 
+ @cindex XFAIL, avoiding for POSIX
+ @sc{posix} 1003.3 does not incorporate the notion of expected failures,
+ so @sc{posix} tests must return @code{FAIL} rather
+ than @code{XFAIL} even if a failure was expected.
+ 
+ @item KFAIL
+ A test is known to fail in some environment(s) due to a known bug
+ in the tool being tested (identified by a bug id string).  This 
+ exists so that, after a bug is identified and properly registered
+ in a bug tracking database (Gnats, for instance), the count of 
+ failures can be kept as zero.  Having zero as a baseline in all
+ platforms allow the tool developers to immediately detect regressions
+ caused by changes (which may affect some platforms and not others).
+ The connection with a bug tracking database allows for automatic
+ generation of the BUGS section of man pages or Release Notes, as
+ well as a "Bugs Fixed this Release" section (by comparing to a
+ previous release set of known failures).
+ The procedure @code{setup_kfail} is used to indicate a failure is
+ known to exist.
+ 
+ @cindex KFAIL, avoiding for POSIX
+ As with @code{XFAIL}, @sc{posix} tests must return @code{FAIL} rather
+ than @code{KFAIL} even if a failure was due to a known bug.
+ 
  
+ @item XPASS
+ A test was expected to fail with either @code{XFAIL} or @code{KFAIL}
+ but passed instead.  Someone may have fixed the bug and failed to
+ unmark the test, or whatever problem that used to exist in the 
+ environment was corrected (the test may also be failing to detect the
+ failure due to some environment or output changes, so this must be
+ investigated as well).
+ 
+ @code{PASS}, instead of @code{XPASS}, must also be returned for test
+ cases which were expected to fail and did not.  This means that
+ @code{PASS} is in some sense more ambiguous than if @code{XPASS} is also
+ used.  
+ 
+ @end ftable
+ 
+ See also @ref{Invoking runtest,,Using @code{runtest}}.
+ For information on how to mark tests as expected/known to fail by using
+ @code{setup_xfail} and @code{setup_kfail}, see
+ @ref{framework.exp,,Core Internal Procedures}.
+ 
+ 
  @node Future Directions
  @section Future directions
  @cindex future directions
*************** succeed.
*** 612,618 ****
  @kindex XPASS
  @cindex successful test, unexpected
  @cindex unexpected success
! A pleasant kind of failure: a test was expected to fail, but succeeded.
  This may indicate progress; inspect the test case to determine whether
  you should amend it to stop expecting failure.
  
--- 663,669 ----
  @kindex XPASS
  @cindex successful test, unexpected
  @cindex unexpected success
! A pleasant kind of failure: a test was expected/known to fail, but succeeded.
  This may indicate progress; inspect the test case to determine whether
  you should amend it to stop expecting failure.
  
*************** regress; inspect the test case and the f
*** 628,636 ****
  @cindex expected failure
  @cindex failing test, expected
  A test failed, but it was expected to fail.  This result indicates no
! change in a known bug.  If a test fails because the operating system
! where the test runs lacks some facility required by the test, the
! outcome is @code{UNSUPPORTED} instead.
  
  @item UNRESOLVED
  @kindex UNRESOLVED
--- 679,697 ----
  @cindex expected failure
  @cindex failing test, expected
  A test failed, but it was expected to fail.  This result indicates no
! change in a known environment bug.  If a test fails because the operating
! system where the test runs lacks some facility required by the test
! (i.e. failure is due to the lack of a feature, not the existence of a bug),
! the outcome is @code{UNSUPPORTED} instead.
! 
! @item KFAIL
! @kindex KFAIL
! @cindex known failure
! @cindex failing test, known
! A test failed, but it was known to fail.  This result indicates no
! change in a known bug.  If a test fails because of a problem in the
! environment, not in the tool being tested, that is expected to be
! fixed one day, the outcome is @code{XFAIL} instead.
  
  @item UNRESOLVED
  @kindex UNRESOLVED
*************** recorded by your configuration's choice 
*** 844,851 ****
  change how anything is actually configured unless --build is also
  specified; it affects @emph{only} DejaGnu procedures that compare the
  host string with particular values.  The procedures @code{ishost},
! @code{istarget}, @code{isnative}, and @code{setup_xfail} are affected by
! @samp{--host}. In this usage, @code{host} refers to the machine that the
  tests are to be run on, which may not be the same as the @code{build}
  machine. If @code{--build} is also specified, then @code{--host} refers
  to the machine that the tests wil, be run on, not the machine DejaGnu is
--- 905,913 ----
  change how anything is actually configured unless --build is also
  specified; it affects @emph{only} DejaGnu procedures that compare the
  host string with particular values.  The procedures @code{ishost},
! @code{istarget}, @code{isnative}, @code{setup_xfail} and
! @code{setup_kfail} are affected by @samp{--host}.
! In this usage, @code{host} refers to the machine that the
  tests are to be run on, which may not be the same as the @code{build}
  machine. If @code{--build} is also specified, then @code{--host} refers
  to the machine that the tests wil, be run on, not the machine DejaGnu is
*************** common shell wildcard characters to spec
*** 1860,1877 ****
  output; use it as a link to a bug-tracking system such as @sc{gnats}
  (@pxref{Overview,, Overview, gnats.info, Tracking Bugs With GNATS}).
  
  @cindex @code{XFAIL}, producing
  @cindex @code{XPASS}, producing
! Once you use @code{setup_xfail}, the @code{fail} and @code{pass}
! procedures produce the messages @samp{XFAIL} and @samp{XPASS}
! respectively, allowing you to distinguish expected failures (and
! unexpected success!) from other test outcomes.
! 
! @emph{Warning:} you must clear the expected failure after using
! @code{setup_xfail} in a test case.  Any call to @code{pass} or
! @code{fail} clears the expected failure implicitly; if the test has some
! other outcome, e.g. an error, you can call @code{clear_xfail} to clear
! the expected failure explicitly.  Otherwise, the expected-failure
  declaration applies to whatever test runs next, leading to surprising
  results.
  
--- 1922,1964 ----
  output; use it as a link to a bug-tracking system such as @sc{gnats}
  (@pxref{Overview,, Overview, gnats.info, Tracking Bugs With GNATS}).
  
+ See notes under setup_kfail (below).
+  
+ @item setup_kfail "@var{config}  @r{[}@var{bugid}@r{]}"
+ @c two spaces above to make it absolutely clear there's whitespace---a
+ @c crude sort of italic correction!
+ @cindex test case, known failure
+ @cindex failure, known
+ @cindex known failure
+ Declares that the test is known to fail on a particular set of
+ configurations.  The @var{config} argument must be a list of full
+ three-part @code{configure} target name; in particular, you may not use
+ the shorter nicknames supported by @code{configure} (but you can use the
+ common shell wildcard characters to specify sets of names).  The
+ @var{bugid} argument is mandatory, and used only in the logging file
+ output; use it as a link to a bug-tracking system such as @sc{gnats}
+ (@pxref{Overview,, Overview, gnats.info, Tracking Bugs With GNATS}).
+ 
  @cindex @code{XFAIL}, producing
+ @cindex @code{KFAIL}, producing
  @cindex @code{XPASS}, producing
! Once you use @code{setup_xfail} or @code{setup_kfail}, the @code{fail}
! and @code{pass} procedures produce the messages @samp{XFAIL} or @samp{KFAIL}
! and @samp{XPASS} respectively, allowing you to distinguish expected/known
! failures (and unexpected success!) from other test outcomes.
! 
! If a test is marked as both expected to fail and known to fail for a
! certain configuration, a @samp{KFAIL} message will be generated.
! As @samp{KFAIL} messages are expected to draw more attention than
! the @samp{XFAIL} ones this will hopefuly ensure the test result is not
! overlooked.
! 
! @emph{Warning:} you must clear the expected/known failure after using
! @code{setup_xfail} or @code{setup_kfail} in a test case.  Any call to 
! @code{pass} or @code{fail} clears the expectedknown failure implicitly;
! if the test has some other outcome, e.g. an error, you can call
! @code{clear_xfail} to clear the expected failure or @code{clear_kfail}
! to clear the known failure explicitly.  Otherwise, the expected-failure
  declaration applies to whatever test runs next, leading to surprising
  results.
  
*************** for a particular set of configurations. 
*** 1951,1956 ****
--- 2038,2052 ----
  list of configuration target names.  It is only necessary to call
  @code{clear_xfail} if a test case ends without calling either
  @code{pass} or @code{fail}, after calling @code{setup_xfail}.
+ 
+ @item clear_kfail @var{config}
+ @cindex cancelling known failure
+ @cindex known failure, cancelling
+ Cancel a known failure (previously declared with @code{setup_kfail})
+ for a particular set of configurations.  The @var{config} argument is a
+ list of configuration target names.  It is only necessary to call
+ @code{clear_kfail} if a test case ends without calling either
+ @code{pass} or @code{fail}, after calling @code{setup_kfail}.
  
  @item verbose @r{[}-log@r{]} @r{[}-n@r{]} @r{[}--@r{]} "@var{string}" @var{number}
  @cindex @code{verbose} builtin function
Index: dejagnu/lib/framework.exp
===================================================================
RCS file: /cvs/src/src/dejagnu/lib/framework.exp,v
retrieving revision 1.6
diff -c -p -r1.6 framework.exp
*** framework.exp	2001/01/15 08:12:07	1.6
--- framework.exp	2002/04/06 00:19:28
*************** proc unknown { args } {
*** 252,258 ****
  # Without this, all messages that start with a keyword are written only to the
  # detail log file.  All messages that go to the screen will also appear in the
  # detail log.  This should only be used by the framework itself using pass,
! # fail, xpass, xfail, warning, perror, note, untested, unresolved, or
  # unsupported procedures.
  #
  proc clone_output { message } {
--- 252,258 ----
  # Without this, all messages that start with a keyword are written only to the
  # detail log file.  All messages that go to the screen will also appear in the
  # detail log.  This should only be used by the framework itself using pass,
! # fail, xpass, xfail, kfail, warning, perror, note, untested, unresolved, or
  # unsupported procedures.
  #
  proc clone_output { message } {
*************** proc clone_output { message } {
*** 265,271 ****
  
      regsub "^\[ \t\]*(\[^ \t\]+).*$" "$message" "\\1" firstword;
      case "$firstword" in {
! 	{"PASS:" "XFAIL:" "UNRESOLVED:" "UNSUPPORTED:" "UNTESTED:"} {
  	    if $all_flag {
  		send_user "$message\n"
  		return "$message"
--- 265,271 ----
  
      regsub "^\[ \t\]*(\[^ \t\]+).*$" "$message" "\\1" firstword;
      case "$firstword" in {
! 	{"PASS:" "XFAIL:" "KFAIL:" "UNRESOLVED:" "UNSUPPORTED:" "UNTESTED:"} {
  	    if $all_flag {
  		send_user "$message\n"
  		return "$message"
*************** proc log_summary { args } {
*** 365,371 ****
  	if { $testcnt > 0 } {
  	    set totlcnt 0;
  	    # total all the testcases reported
! 	    foreach x { FAIL PASS XFAIL XPASS UNTESTED UNRESOLVED UNSUPPORTED } {
  		incr totlcnt test_counts($x,$which);
  	    }
  	    set testcnt test_counts(total,$which);
--- 365,371 ----
  	if { $testcnt > 0 } {
  	    set totlcnt 0;
  	    # total all the testcases reported
! 	    foreach x { FAIL PASS XFAIL KFAIL XPASS UNTESTED UNRESOLVED UNSUPPORTED } {
  		incr totlcnt test_counts($x,$which);
  	    }
  	    set testcnt test_counts(total,$which);
*************** proc log_summary { args } {
*** 389,395 ****
  	    }
  	}
      }
!     foreach x { PASS FAIL XPASS XFAIL UNRESOLVED UNTESTED UNSUPPORTED } {
  	set val $test_counts($x,$which);
  	if { $val > 0 } {
  	    set mess "# of $test_counts($x,name)";
--- 389,395 ----
  	    }
  	}
      }
!     foreach x { PASS FAIL XPASS XFAIL KFAIL UNRESOLVED UNTESTED UNSUPPORTED } {
  	set val $test_counts($x,$which);
  	if { $val > 0 } {
  	    set mess "# of $test_counts($x,name)";
*************** proc setup_xfail { args } {
*** 442,447 ****
--- 442,484 ----
  }
  
  
+ #
+ # Setup a flag to control whether it is a known failure
+ #
+ # A bug report ID _MUST_ be specified, and is the first argument.
+ # It still must be a string without '-' so we can be sure someone
+ # did not just forget it and we end-up using a taget triple as
+ # bug id.
+ #
+ # Multiple target triplet patterns can be specified for targets
+ # for which the test is known to fail.  
+ #
+ #
+ proc setup_kfail { args } {
+     global kfail_flag
+     global kfail_prms
+     
+     set kfail_prms 0
+     set argc [ llength $args ]
+     for { set i 0 } { $i < $argc } { incr i } {
+ 	set sub_arg [ lindex $args $i ]
+ 	# is a prms number. we assume this is a string with no '-' characters
+ 	if [regexp "^\[^\-\]+$" $sub_arg] { 
+ 	    set kfail_prms $sub_arg
+ 	    continue
+ 	}
+ 	if [istarget $sub_arg] {
+ 	    set kfail_flag 1
+ 	    continue
+ 	}
+     }
+ 
+     if {$kfail_prms == 0} {
+ 	perror "Attempt to set a kfail without specifying bug tracking id"
+     }
+ }
+ 
+ 
  # check to see if a conditional xfail is triggered
  #	message {targets} {include} {exclude}
  #              
*************** proc clear_xfail { args } {
*** 558,563 ****
--- 595,622 ----
  }
  
  #
+ # Clear the kfail flag for a particular target
+ #
+ proc clear_kfail { args } {
+     global kfail_flag
+     global kfail_prms
+     
+     set argc [ llength $args ]
+     for { set i 0 } { $i < $argc } { incr i } {
+ 	set sub_arg [ lindex $args $i ]
+ 	case $sub_arg in {
+ 	    "*-*-*" {			# is a configuration triplet
+ 		if [istarget $sub_arg] {
+ 		    set kfail_flag 0
+ 		    set kfail_prms 0
+ 		}
+ 		continue
+ 	    }
+ 	}
+     }
+ }
+ 
+ #
  # Record that a test has passed or failed (perhaps unexpectedly)
  #
  # This is an internal procedure, only used in this file.
*************** proc record_test { type message args } {
*** 566,571 ****
--- 625,631 ----
      global exit_status
      global prms_id bug_id
      global xfail_flag xfail_prms
+     global kfail_flag kfail_prms
      global errcnt warncnt
      global warning_threshold perror_threshold
      global pf_prefix
*************** proc record_test { type message args } {
*** 613,622 ****
  		set message [concat $message "\t(PRMS $xfail_prms)"]
  	    }
  	}
  	UNTESTED {
! 	    # The only reason we look at the xfail stuff is to pick up
  	    # `xfail_prms'.
! 	    if { $xfail_flag && $xfail_prms != 0 } {
  		set message [concat $message "\t(PRMS $xfail_prms)"]
  	    } elseif $prms_id {
  		set message [concat $message "\t(PRMS $prms_id)"]
--- 673,689 ----
  		set message [concat $message "\t(PRMS $xfail_prms)"]
  	    }
  	}
+ 	KFAIL {
+ 	    if { $kfail_prms != 0 } {
+ 		set message [concat $message "\t(PRMS: $kfail_prms)"]
+ 	    }
+ 	}
  	UNTESTED {
! 	    # The only reason we look at the xfail/kfail stuff is to pick up
  	    # `xfail_prms'.
! 	    if { $kfail_flag && $kfail_prms != 0 } {
! 		set message [concat $message "\t(PRMS $kfail_prms)"]
! 	    } elseif { $xfail_flag && $xfail_prms != 0 } {
  		set message [concat $message "\t(PRMS $xfail_prms)"]
  	    } elseif $prms_id {
  		set message [concat $message "\t(PRMS $prms_id)"]
*************** proc record_test { type message args } {
*** 624,641 ****
  	}
  	UNRESOLVED {
  	    set exit_status 1
! 	    # The only reason we look at the xfail stuff is to pick up
  	    # `xfail_prms'.
! 	    if { $xfail_flag && $xfail_prms != 0 } {
  		set message [concat $message "\t(PRMS $xfail_prms)"]
  	    } elseif $prms_id {
  		set message [concat $message "\t(PRMS $prms_id)"]
  	    }
  	}
  	UNSUPPORTED {
! 	    # The only reason we look at the xfail stuff is to pick up
  	    # `xfail_prms'.
! 	    if { $xfail_flag && $xfail_prms != 0 } {
  		set message [concat $message "\t(PRMS $xfail_prms)"]
  	    } elseif $prms_id {
  		set message [concat $message "\t(PRMS $prms_id)"]
--- 691,712 ----
  	}
  	UNRESOLVED {
  	    set exit_status 1
! 	    # The only reason we look at the xfail/kfail stuff is to pick up
  	    # `xfail_prms'.
! 	    if { $kfail_flag && $kfail_prms != 0 } {
! 		set message [concat $message "\t(PRMS $kfail_prms)"]
! 	    } elseif { $xfail_flag && $xfail_prms != 0 } {
  		set message [concat $message "\t(PRMS $xfail_prms)"]
  	    } elseif $prms_id {
  		set message [concat $message "\t(PRMS $prms_id)"]
  	    }
  	}
  	UNSUPPORTED {
! 	    # The only reason we look at the xfail/kfail stuff is to pick up
  	    # `xfail_prms'.
! 	    if { $kfail_flag && $kfail_prms != 0 } {
! 		set message [concat $message "\t(PRMS $kfail_prms)"]
! 	    } elseif { $xfail_flag && $xfail_prms != 0 } {
  		set message [concat $message "\t(PRMS $xfail_prms)"]
  	    } elseif $prms_id {
  		set message [concat $message "\t(PRMS $prms_id)"]
*************** proc record_test { type message args } {
*** 676,689 ****
      set warncnt 0
      set errcnt 0
      set xfail_flag 0
      set xfail_prms 0
  }
  
  #
  # Record that a test has passed
  #
  proc pass { message } {
!     global xfail_flag compiler_conditional_xfail_data
  
      # if we have a conditional xfail setup, then see if our compiler flags match
      if [ info exists compiler_conditional_xfail_data ] {
--- 747,762 ----
      set warncnt 0
      set errcnt 0
      set xfail_flag 0
+     set kfail_flag 0
      set xfail_prms 0
+     set kfail_prms 0
  }
  
  #
  # Record that a test has passed
  #
  proc pass { message } {
!     global xfail_flag kfail_flag compiler_conditional_xfail_data
  
      # if we have a conditional xfail setup, then see if our compiler flags match
      if [ info exists compiler_conditional_xfail_data ] {
*************** proc pass { message } {
*** 693,699 ****
  	unset compiler_conditional_xfail_data
      }
      
!     if $xfail_flag {
  	record_test XPASS $message
      } else {
  	record_test PASS $message
--- 766,772 ----
  	unset compiler_conditional_xfail_data
      }
      
!     if {$xfail_flag || $kfail_flag} {
  	record_test XPASS $message
      } else {
  	record_test PASS $message
*************** proc pass { message } {
*** 704,710 ****
  # Record that a test has failed
  #
  proc fail { message } {
!     global xfail_flag compiler_conditional_xfail_data
  
      # if we have a conditional xfail setup, then see if our compiler flags match
      if [ info exists compiler_conditional_xfail_data ] {
--- 777,783 ----
  # Record that a test has failed
  #
  proc fail { message } {
!     global xfail_flag kfail_flag compiler_conditional_xfail_data
  
      # if we have a conditional xfail setup, then see if our compiler flags match
      if [ info exists compiler_conditional_xfail_data ] {
*************** proc fail { message } {
*** 714,720 ****
  	unset compiler_conditional_xfail_data
      }
  
!     if $xfail_flag {
  	record_test XFAIL $message
      } else {
  	record_test FAIL $message
--- 787,795 ----
  	unset compiler_conditional_xfail_data
      }
  
!     if $kfail_flag {
! 	record_test KFAIL $message
!     } elseif $xfail_flag {
  	record_test XFAIL $message
      } else {
  	record_test FAIL $message
*************** proc init_testcounts { } {
*** 845,850 ****
--- 920,926 ----
      set test_counts(PASS,name) "expected passes"
      set test_counts(FAIL,name) "unexpected failures"
      set test_counts(XFAIL,name) "expected failures"
+     set test_counts(KFAIL,name) "known failures"
      set test_counts(XPASS,name) "unexpected successes"
      set test_counts(WARNING,name) "warnings"
      set test_counts(ERROR,name) "errors"
Index: dejagnu/testsuite/lib/libsup.exp
===================================================================
RCS file: /cvs/src/src/dejagnu/testsuite/lib/libsup.exp,v
retrieving revision 1.1.1.1
diff -c -p -r1.1.1.1 libsup.exp
*** libsup.exp	1999/11/09 01:28:42	1.1.1.1
--- libsup.exp	2002/04/06 00:19:28
*************** proc make_defaults_file { defs } {
*** 64,69 ****
--- 64,71 ----
      puts ${fd} "set unsupportedcnt 0"
      puts ${fd} "set xfail_flag 0"
      puts ${fd} "set xfail_prms 0"
+     puts ${fd} "set kfail_flag 0"
+     puts ${fd} "set kfail_prms 0"
      puts ${fd} "set mail_logs 0"
      puts ${fd} "set multipass_name 0"
      catch "close $fd"

Re: RFC: KFAIL DejaGnu patch

[Andrew Cagney]

> P.S.: I was trying to find some test to try the setup_kfail on.  I went
> looking for the setup_xfail *-*-* calls but most don't have any comment
> why they are marked like that.  Then I've tried looking for the FAILs
> and see if I could find a bug entry for those -- but it is not a
> trivial task.  I guess we will need everybody's help and memories to
> gradually have things properly documented and related to the bug
> entries.

To be honest, I think just removing all but the most recent (i.e. in 
last two years) xfails might be for the best.

Andrew

Re: RFC: KFAIL DejaGnu patch

[Rob Savoye]

On Fri, Apr 05, 2002 at 08:05:44PM -0500, Andrew Cagney wrote:

> To be honest, I think just removing all but the most recent (i.e. in 
> last two years) xfails might be for the best.

  Tottally... One of the biggest worries I had when the whole XFAIL/XPASS
feature was originally implemented was that old bugs would get marked as XFAIL,
and then be left that way forever. Cleaning out the very oldest ones would
be a great idea.

	- rob -

Re: RFC: KFAIL DejaGnu patch

[Daniel Jacobowitz]

On Fri, Apr 05, 2002 at 07:29:54PM -0500, Fernando Nasser wrote:
> Well, I am attaching the DejaGnu changes for KFAILs.
> 
> Rob: can you please take a look, specially in the documentation
> changes and see if they are fit for the master repository?
> 
> Michael Chastain: Will you be willing to help me test this?
> I've tried it and it seems to work.  Yours scripts must also
> be happy with it.
> 
> Best regards,
> Fernnado
> 
> 
> P.S.: I was trying to find some test to try the setup_kfail on.  I went
> looking for the setup_xfail *-*-* calls but most don't have any comment
> why they are marked like that.  Then I've tried looking for the FAILs
> and see if I could find a bug entry for those -- but it is not a
> trivial task.  I guess we will need everybody's help and memories to
> gradually have things properly documented and related to the bug
> entries.

Looks good to me.  Fernando, could I persuade you to write up a
GDB-specific list of what each of the DejaGNU result codes is meant to
cover?  We could stick it in testsuite/ somewhere.

-- 
Daniel Jacobowitz                           Carnegie Mellon University
MontaVista Software                         Debian GNU/Linux Developer

Re: RFC: KFAIL DejaGnu patch

[Andrew Cagney]

> On Fri, Apr 05, 2002 at 07:29:54PM -0500, Fernando Nasser wrote:
> 
>> Well, I am attaching the DejaGnu changes for KFAILs.
>> 
>> Rob: can you please take a look, specially in the documentation
>> changes and see if they are fit for the master repository?
>> 
>> Michael Chastain: Will you be willing to help me test this?
>> I've tried it and it seems to work.  Yours scripts must also
>> be happy with it.
>> 
>> Best regards,
>> Fernnado
>> 
>> 
>> P.S.: I was trying to find some test to try the setup_kfail on.  I went
>> looking for the setup_xfail *-*-* calls but most don't have any comment
>> why they are marked like that.  Then I've tried looking for the FAILs
>> and see if I could find a bug entry for those -- but it is not a
>> trivial task.  I guess we will need everybody's help and memories to
>> gradually have things properly documented and related to the bug
>> entries.
> 
> 
> Looks good to me.  Fernando, could I persuade you to write up a
> GDB-specific list of what each of the DejaGNU result codes is meant to
> cover?  We could stick it in testsuite/ somewhere.

I had a similar suggestion in mind but not ``in testsuite/ somewhere''. 
  See:
http://sources.redhat.com/gdb/onlinedocs/gdbint_16.html#SEC154

(just making a note that testsuite/gdb.gdb/ isn't documented)
Andrew

Re: RFC: KFAIL DejaGnu patch

[Fernando Nasser]

Daniel Jacobowitz wrote:
> 
> Looks good to me.  Fernando, could I persuade you to write up a
> GDB-specific list of what each of the DejaGNU result codes is meant to
> cover?  We could stick it in testsuite/ somewhere.
> 

Sure.  I will stick it to the Internals as Andrew suggested.
I will try and do it, if not this weekend, the next.

-- 
Fernando Nasser
Red Hat Canada Ltd.                     E-Mail:  fnasser@redhat.com
2323 Yonge Street, Suite #300
Toronto, Ontario   M4P 2C9

Re: RFC: KFAIL DejaGnu patch

[Rob Savoye]

On Fri, Apr 05, 2002 at 07:29:54PM -0500, Fernando Nasser wrote:

> Rob: can you please take a look, specially in the documentation
> changes and see if they are fit for the master repository?
 
  The texi document is mostly out of date, as I rewrote that whole
manual in SGML about 3 years ago. Would it be possible to get these
same editing changes for the SGML/DocBook version of the docs ?

	- rob -

Re: RFC: KFAIL DejaGnu patch

[Eli Zaretskii]

On Sun, 7 Apr 2002, Rob Savoye wrote:

> On Fri, Apr 05, 2002 at 07:29:54PM -0500, Fernando Nasser wrote:
> 
> > Rob: can you please take a look, specially in the documentation
> > changes and see if they are fit for the master repository?
>  
>   The texi document is mostly out of date, as I rewrote that whole
> manual in SGML about 3 years ago. Would it be possible to get these
> same editing changes for the SGML/DocBook version of the docs ?

Are there any reason not to continue maintaining the document in 
Texinfo?  Especially since the latest makeinfo supports DocBook (as well 
as HTML and XML)?

Re: RFC: KFAIL DejaGnu patch

[Rob Savoye]

On Mon, Apr 08, 2002 at 08:58:52AM +0300, Eli Zaretskii wrote:
 
> Are there any reason not to continue maintaining the document in 
> Texinfo?  Especially since the latest makeinfo supports DocBook (as well 
> as HTML and XML)?

  The main reason is that texinfo formatted version is way out of date,
because I stopped maintaining it when I converted the whole thing by hand
to SGML/DocBook several years ago. I guess the main problem is that DocBook
doesn't spit out an info page, which would be kindof nice... But there has
been alot added to the sgml formatted version that would all have to be
merged back into the texinfo document.

  btw - Matt says he'll add the extra fields for ERROR and WARNING to the
XML output, which he's hoping will be done as soon as he firms up the DTD.
Anything else you need to have that be useful ?

	- rob -

Re: RFC: KFAIL DejaGnu patch

[Fernando Nasser]

Rob Savoye wrote:
> 
> On Mon, Apr 08, 2002 at 08:58:52AM +0300, Eli Zaretskii wrote:
> 
> > Are there any reason not to continue maintaining the document in
> > Texinfo?  Especially since the latest makeinfo supports DocBook (as well
> > as HTML and XML)?
> 
>   The main reason is that texinfo formatted version is way out of date,
> because I stopped maintaining it when I converted the whole thing by hand
> to SGML/DocBook several years ago. I guess the main problem is that DocBook
> doesn't spit out an info page, which would be kindof nice... But there has
> been alot added to the sgml formatted version that would all have to be
> merged back into the texinfo document.
> 
>   btw - Matt says he'll add the extra fields for ERROR and WARNING to the
> XML output, which he's hoping will be done as soon as he firms up the DTD.
> Anything else you need to have that be useful ?
> 

The KFAIL/KPASS will be there, right?

-- 
Fernando Nasser
Red Hat Canada Ltd.                     E-Mail:  fnasser@redhat.com
2323 Yonge Street, Suite #300
Toronto, Ontario   M4P 2C9

Re: RFC: KFAIL DejaGnu patch

[Rob Savoye]

On Mon, Apr 08, 2002 at 11:30:12AM -0400, Fernando Nasser wrote:

> The KFAIL/KPASS will be there, right?

  Oops, I spaced that part. I think we'll need the kpass/kfail patch first,
to them make those output routines spit out XML if the flag is set.

	- rob -

Re: RFC: KFAIL DejaGnu patch

[Fernando Nasser]

Rob Savoye wrote:
> 
> On Fri, Apr 05, 2002 at 07:29:54PM -0500, Fernando Nasser wrote:
> 
> > Rob: can you please take a look, specially in the documentation
> > changes and see if they are fit for the master repository?
> 
>   The texi document is mostly out of date, as I rewrote that whole
> manual in SGML about 3 years ago. Would it be possible to get these
> same editing changes for the SGML/DocBook version of the docs ?
> 

I don't know DocBook, but I can certainly try.  I hope to be able to
do it next weekend.

-- 
Fernando Nasser
Red Hat Canada Ltd.                     E-Mail:  fnasser@redhat.com
2323 Yonge Street, Suite #300
Toronto, Ontario   M4P 2C9

Re: RFC: KFAIL DejaGnu patch

[Rob Savoye]

On Mon, Apr 08, 2002 at 11:31:47AM -0400, Fernando Nasser wrote:

> I don't know DocBook, but I can certainly try.  I hope to be able to
> do it next weekend.

  I found docbook2texi, which (if it worked) let me generate texinfo -> info
from Docbook source. Once I get that to work, I'll probably drop the old
texinfo doc completely to avoid confusion.

	- rob -

Re: RFC: KFAIL DejaGnu patch

[Andrew Cagney]

> On Mon, Apr 08, 2002 at 11:31:47AM -0400, Fernando Nasser wrote:
> 
> 
>> I don't know DocBook, but I can certainly try.  I hope to be able to
>> do it next weekend.
> 
> 
> I found docbook2texi, which (if it worked) let me generate texinfo -> info
> from Docbook source. Once I get that to work, I'll probably drop the old
> texinfo doc completely to avoid confusion.

(Put on flame proof jacket :-).

Um, I really have no desire to have to learn yet another documentation 
language (Ask Eli how painfull it has been having me learn the current 
one).  Especially when all other GNU projects I'm involved in use texinfo.

Is there any absolutly compelling reason to not convert the current 
docbook back to texinfo and, instead, stick with that?

Andrew

Re: RFC: KFAIL DejaGnu patch

[Rob Savoye]

On Mon, Apr 08, 2002 at 04:55:24PM -0400, Andrew Cagney wrote:

> Um, I really have no desire to have to learn yet another documentation 
> language (Ask Eli how painfull it has been having me learn the current 
> one).  Especially when all other GNU projects I'm involved in use texinfo.

  SGML and texinfo are similar enough, I didn't find it that hard to learn
yet another documentation language. Funny enough, I switched to Docbook at
Cygnus, after all the gripes from the doc department. :-) We used it for
eCOS, cause it would generate RTF output, which was a requirement of our
customer.

> Is there any absolutly compelling reason to not convert the current 
> docbook back to texinfo and, instead, stick with that?

  Yes. DocBook is way better than Texinfo at representing technical documents,
than texinfo. Texinfo is great for glorified man pages, but SGML is better
for technical manuals. While most older GNU projects use texinfo, I see that
many newer GNU/Linux projects use DocBook.

  Although I haven't tried it, Abiword supposedly lets you do WYSIWYG
editing of DocBook files. I just use psgml mode for Emacs now.

	- rob -

Re: RFC: KFAIL DejaGnu patch

[Andrew Cagney]

> 
> Yes. DocBook is way better than Texinfo at representing technical documents,
> than texinfo. Texinfo is great for glorified man pages, but SGML is better
> for technical manuals.

Why?  Is there a posting somewhere explaining the rationale for this?

 > While most older GNU projects use texinfo, I see that
> many newer GNU/Linux projects use DocBook.

None of the ones that I'm interested in - gcc, binutils, gdb - do.  It 
is a shame that DejaGnu does as that is the only other tool I really 
depend on.

enjoy,
Andrew

Re: RFC: KFAIL DejaGnu patch

[Rob Savoye]

On Mon, Apr 08, 2002 at 07:34:25PM -0400, Andrew Cagney wrote:

> Why?  Is there a posting somewhere explaining the rationale for this?
 
  I'll have to look around. Mark Gallassi at that time had alot of good
info on this, as he was Cygnus's DocTools maintainer. (btw, the page on
sourceware is way out of date) Briefly though, the main reasons we chose
it for eCOS, and the Linux Doc project choose it is Docbook is an 
international used standard for technical documentation. Framemaker
suports it, CISCO, and many other companies use it for technical
documentation. At the time several years ago when I rewrote the manual,
using SGML gave me much better HTML, PDF, PS, RTF, and dvi output. As the
DocBook SGML tools are included in most modern Linux releases (RedHat, 7.x,
etc...), they are now an accepted format for free software project
documentation. I commonly come across free software projects with Docbook
format manuals.

> None of the ones that I'm interested in - gcc, binutils, gdb - do.  It 
> is a shame that DejaGnu does as that is the only other tool I really 
> depend on.

  Well, these tools have been around for quite a while, and already have
reasonably complete manuals. Luckily most people rarely ever had anything
to the DejaGnu manual, so I don't think it's a big problem. Adding a few
paragraphs here and there is just a cut and paste job anyway, whether texinfo
or docbook is used.

  At this point though, I have no interest in porting the manual *again* to
switch to texinfo. Too much work, for too little gain.

	- rob -

Re: RFC: KFAIL DejaGnu patch

[Andrew Cagney]

> None of the ones that I'm interested in - gcc, binutils, gdb - do.  It 
>> is a shame that DejaGnu does as that is the only other tool I really 
>> depend on.
> 
> 
> Well, these tools have been around for quite a while, and already have
> reasonably complete manuals. Luckily most people rarely ever had anything
> to the DejaGnu manual, so I don't think it's a big problem. Adding a few
> paragraphs here and there is just a cut and paste job anyway, whether texinfo
> or docbook is used.
> 
>   At this point though, I have no interest in porting the manual *again* to
> switch to texinfo. Too much work, for too little gain.

That is a judgement call.  The ammount of work required by an individual 
to switch vs the advantages (and disadvantages) to the entire dejagnu 
community.

enjoy,
Andrew

Re: RFC: KFAIL DejaGnu patch

[Rob Savoye]

On Mon, Apr 08, 2002 at 07:58:33PM -0400, Andrew Cagney wrote:

> That is a judgement call.  The ammount of work required by an individual 
> to switch vs the advantages (and disadvantages) to the entire dejagnu 
> community.

  Thanks for volunteering! :-) The entire DejaGnu community is way larger than
just the GCC/GDB/Binutils team... and some of those other developers also
know Docbook... While I think the various GCC/GDB people spread around the
world are likely the most active DejaGnu users, as long as they can *read*
the documentation in whatever output format they prefer, they're happy. So
few of us (mostly usually just me) ever change the actual DejaGnu manual that
I don't think the format really matters. It's not really that different
to edit an SGML file, than a texinfo one. They're both just text, with
ugly verbose formatting commands.

	- rob -

Re: RFC: KFAIL DejaGnu patch

[Eli Zaretskii]

On Mon, 8 Apr 2002, Rob Savoye wrote:

> The entire DejaGnu community is way larger than
> just the GCC/GDB/Binutils team... and some of those other developers also
> know Docbook... While I think the various GCC/GDB people spread around the
> world are likely the most active DejaGnu users, as long as they can *read*
> the documentation in whatever output format they prefer, they're happy.

What good is it to have documentation one is unable to modify?

It took (and still takes) a lot of effort to help developers master 
Texinfo (try searching this list for comments to various docs patches, if 
you want examples).  I hope that slowly people here are beginning to 
realize that accompaining code changes with corresponding doco changes is 
not such a hard requirement after all.  This is a good development, I 
think.  If we add yet another language to what people need to know, we 
might simply ruin it all.

So please reconsider the possibility of going back to Texinfo.  Since I 
understand there's a tool available to make the conversion, it sounds 
like the amount of work is not large.

Re: RFC: KFAIL DejaGnu patch

[Rob Savoye]

On Tue, Apr 09, 2002 at 10:56:21AM +0300, Eli Zaretskii wrote:

> What good is it to have documentation one is unable to modify?
 
  "Unable to modify" ? I think you mean unwilling. People modify the DejaGnu
manual without problem all the time. Docbook looks alot like HTML... It's
easy to modify, produces better output, and used by many GNU projects.
I realize texinfo is the standard format for the GNU project, but then
again, I'm not supposed to be using Tcl either. :-) Seriously, considering
the quality of most of the engineers I've known on the GDB team, learning
what little one would know to update a docbook manual is trivial. 

> So please reconsider the possibility of going back to Texinfo.  Since I 

  As an engineer, I've gone through several documentation formats over the
last 24 years. Nroff, man pages, texinfo, and now docbook. I'm far from
an expert on documentation formats, but I only switched after many,
many meetings about this back when we started eCOS.  Even Cygnus's
own doc team prefered Docbook. (I don't know about RedHats's) Anyway,
I see no reason to go backwards. Sorry. If you have to, send me manual
updates as text, and I'll merge them in. 99& of the time, nobody ever
updates the doc but me anyway. I prefer Docbook, once I got used to it.

  I haven't been able to get docbook2texi to fully work yet anyway...

	- rob -

Re: RFC: KFAIL DejaGnu patch

[Eli Zaretskii]

On Mon, 8 Apr 2002, Andrew Cagney  wrote:

> > Yes. DocBook is way better than Texinfo at representing technical documents,
> > than texinfo. Texinfo is great for glorified man pages, but SGML is better
> > for technical manuals.
> 
> Why?  Is there a posting somewhere explaining the rationale for this?
> 
>  > While most older GNU projects use texinfo, I see that
> > many newer GNU/Linux projects use DocBook.
> 
> None of the ones that I'm interested in - gcc, binutils, gdb - do.  It 
> is a shame that DejaGnu does as that is the only other tool I really 
> depend on.

I have to agree with Andrew here.  Texinfo is good enough for what we 
need, especially with the new features introduced in the latest release 
4.2, which also supports XML and DocBook output (in addition to HTML, 
Info, and plain text).

On top of that, since Texinfo is the GNU standard documentation system,
a GNU package, especially an important GNU package such as GDB, should be 
honest and use the GNU tools for its documentation.  Otherwise the GNU 
project could be rightfully accused of hypocrisy.

Re: RFC: KFAIL DejaGnu patch

[Daniel Jacobowitz]

On Tue, Apr 09, 2002 at 10:51:32AM +0300, Eli Zaretskii wrote:
> 
> On Mon, 8 Apr 2002, Andrew Cagney  wrote:
> 
> > > Yes. DocBook is way better than Texinfo at representing technical documents,
> > > than texinfo. Texinfo is great for glorified man pages, but SGML is better
> > > for technical manuals.
> > 
> > Why?  Is there a posting somewhere explaining the rationale for this?
> > 
> >  > While most older GNU projects use texinfo, I see that
> > > many newer GNU/Linux projects use DocBook.
> > 
> > None of the ones that I'm interested in - gcc, binutils, gdb - do.  It 
> > is a shame that DejaGnu does as that is the only other tool I really 
> > depend on.
> 
> I have to agree with Andrew here.  Texinfo is good enough for what we 
> need, especially with the new features introduced in the latest release 
> 4.2, which also supports XML and DocBook output (in addition to HTML, 
> Info, and plain text).
> 
> On top of that, since Texinfo is the GNU standard documentation system,
> a GNU package, especially an important GNU package such as GDB, should be 
> honest and use the GNU tools for its documentation.  Otherwise the GNU 
> project could be rightfully accused of hypocrisy.

"Good enough for what we need" seems like an awfully bad reason to
disagree with another developer's choice of documentation formats.  Has
either of you ever looked at DocBook source?  If not, I recommend
casually browsing some - the DejaGNU manual perhaps:

http://savannah.gnu.org/cgi-bin/viewcvs/*checkout*/dejagnu/dejagnu/doc/overview.sgml?rev=1.6&content-type=text/plain

It is drastically simpler than Texinfo, easy to learn, easy to modify,
and only a little harder to write from scratch.


And where did GDB using anything but Texinfo come in to this
conversation?  We're talking about the DejaGNU documentation!

-- 
Daniel Jacobowitz                           Carnegie Mellon University
MontaVista Software                         Debian GNU/Linux Developer