Re: exit status of 'make check'

Re: exit status of 'make check'

[Michael Elizabeth Chastain]

I agree with Daniel here, but from a different point of view.

It would be nice if the exit status was, for example:

  0  no results are ERROR, WARNING, or FAIL
  1  no results are ERROR or WARNING; some results are FAIL
  2  some results are ERROR or WARNING
  3  the test harness did not run well enough to generate results
     (e.g. runtest not found).

(or something like that)

To achieve this result, someone would have to: design a set of exit statuses,
build consensus for it among several user communities, and submit patches to
tools like dejagnu.

That's a lot of work.  For the same amount of work, one could implement other
useful features in the test machinery and in the test suite itself.  For
instance, I would like to have ERRORs and WARNINGs reported with the test
script filename in them, just like FAILs are.

I do sympathize that the exit status is imprecise and that this can
interfere with large-scale automation.

Michael C

Re: exit status of 'make check'

[George France]

On Monday 26 November 2001 00:10, Michael Elizabeth Chastain wrote:
> I agree with Daniel here, but from a different point of view.
>
> It would be nice if the exit status was, for example:
>
>   0  no results are ERROR, WARNING, or FAIL
>   1  no results are ERROR or WARNING; some results are FAIL
>   2  some results are ERROR or WARNING
>   3  the test harness did not run well enough to generate results
>      (e.g. runtest not found).
>
> (or something like that)

This makes good sense to me. :-) 

>
> To achieve this result, someone would have to: design a set of exit
> statuses, build consensus for it among several user communities, and submit
> patches to tools like dejagnu.
>
> That's a lot of work.

Agreed.

>  For the same amount of work, one could implement
> other useful features in the test machinery and in the test suite itself. 
> For instance, I would like to have ERRORs and WARNINGs reported with the
> test script filename in them, just like FAILs are.

I would like to see both implemented. I just need about 3 more hours in each 
day to work on such a project.  I hope to have some time next week to look 
into this matter further.

> I do sympathize that the exit status is imprecise and that this can
> interfere with large-scale automation.
>

Thank you.


Best Regards,



--George

Re: exit status of 'make check'

[George France]

On Monday 26 November 2001 00:10, Michael Elizabeth Chastain wrote:
> I agree with Daniel here, but from a different point of view.
>
> It would be nice if the exit status was, for example:
>
>   0  no results are ERROR, WARNING, or FAIL
>   1  no results are ERROR or WARNING; some results are FAIL
>   2  some results are ERROR or WARNING
>   3  the test harness did not run well enough to generate results
>      (e.g. runtest not found).
>
> (or something like that)

This makes good sense to me. :-) 

>
> To achieve this result, someone would have to: design a set of exit
> statuses, build consensus for it among several user communities, and submit
> patches to tools like dejagnu.
>
> That's a lot of work.

Agreed.

>  For the same amount of work, one could implement
> other useful features in the test machinery and in the test suite itself. 
> For instance, I would like to have ERRORs and WARNINGs reported with the
> test script filename in them, just like FAILs are.

I would like to see both implemented. I just need about 3 more hours in each 
day to work on such a project.  I hope to have some time next week to look 
into this matter further.

> I do sympathize that the exit status is imprecise and that this can
> interfere with large-scale automation.
>

Thank you.


Best Regards,



--George

Re: exit status of 'make check'

[Michael Elizabeth Chastain]

I agree with Daniel here, but from a different point of view.

It would be nice if the exit status was, for example:

  0  no results are ERROR, WARNING, or FAIL
  1  no results are ERROR or WARNING; some results are FAIL
  2  some results are ERROR or WARNING
  3  the test harness did not run well enough to generate results
     (e.g. runtest not found).

(or something like that)

To achieve this result, someone would have to: design a set of exit statuses,
build consensus for it among several user communities, and submit patches to
tools like dejagnu.

That's a lot of work.  For the same amount of work, one could implement other
useful features in the test machinery and in the test suite itself.  For
instance, I would like to have ERRORs and WARNINGs reported with the test
script filename in them, just like FAILs are.

I do sympathize that the exit status is imprecise and that this can
interfere with large-scale automation.

Michael C

Re: exit status of 'make check'

[Michael Elizabeth Chastain]

From gdb-return-6857-mec=duracef.shout.net@sources.redhat.com  Wed Nov 21 15:05:34 2001
Mailing-List: contact gdb-help@sourceware.cygnus.com; run by ezmlm
List-Unsubscribe: <mailto:gdb-unsubscribe-mec=duracef.shout.net@sources.redhat.com>
List-Subscribe: <mailto:gdb-subscribe@sources.redhat.com>
List-Archive: <http://sources.redhat.com/ml/gdb/>
List-Post: <mailto:gdb@sources.redhat.com>
List-Help: <mailto:gdb-help@sources.redhat.com>, <http://sources.redhat.com/ml/#faqs>
Sender: gdb-owner@sources.redhat.com
From: George France <france@handhelds.org>
To: gdb@sources.redhat.com
Subject: exit status of 'make check'
Date: Wed, 21 Nov 2001 12:57:55 -0500

Greetings,

When I ever I do a 'make check' on gdb using the  cvs HEAD tree on either the 
intel or Alpha architecture. It always returnes a non-zero exit code. The 
testsuites execute and produce a good gdb.log and gdb.sum files.  Has anybody 
Hi George,

> When I ever I do a 'make check' on gdb using the  cvs HEAD tree on either the 
> intel or Alpha architecture. It always returnes a non-zero exit code. The 
> testsuites execute and produce a good gdb.log and gdb.sum files.  Has anybody 
> seen this before??

Yes, this is normal.  I just do:

  make check || true

Michael C

Re: exit status of 'make check'

[George France]

Hello Michael,

It has been a while since we chatted. I hope you are doing well.

> > When I ever I do a 'make check' on gdb using the  cvs HEAD tree on either
> > the intel or Alpha architecture. It always returnes a non-zero exit code.
> > The testsuites execute and produce a good gdb.log and gdb.sum files.  Has
> > anybody seen this before??
>
> Yes, this is normal.  I just do:
>
>   make check || true

This was not exactly the solution that I was hoping for. :-)  We need to fix 
this problem.  It would be really useful to know if 'make check' did or did 
not have an anomaly.

Any Ideas before I go mucking about?

Best Regards,


--George

Re: exit status of 'make check'

[Daniel Jacobowitz]

On Thu, Nov 22, 2001 at 08:08:25AM -0500, George France wrote:
> Hello Michael,
> 
> It has been a while since we chatted. I hope you are doing well.
> 
> > > When I ever I do a 'make check' on gdb using the  cvs HEAD tree on either
> > > the intel or Alpha architecture. It always returnes a non-zero exit code.
> > > The testsuites execute and produce a good gdb.log and gdb.sum files.  Has
> > > anybody seen this before??
> >
> > Yes, this is normal.  I just do:
> >
> >   make check || true
> 
> This was not exactly the solution that I was hoping for. :-)  We need to fix 
> this problem.  It would be really useful to know if 'make check' did or did 
> not have an anomaly.
> 
> Any Ideas before I go mucking about?
> 
> Best Regards,

That pretty much means that it did.  You can grep for ^FAIL: to see how
many there are.  I'm hoping to see the GDB testsuite pass entirely for
some target, someday...

-- 
Daniel Jacobowitz                           Carnegie Mellon University
MontaVista Software                         Debian GNU/Linux Developer

Re: exit status of 'make check'

[George France]

> That pretty much means that it did.  You can grep for ^FAIL: to see how
> many there are.  I'm hoping to see the GDB testsuite pass entirely for
> some target, someday...

Greetings,

Even if all the testsuites have a FAILed status, 'make check' should not 
return a non-zero exit code unless there is a failure in the 'make check' 
programs. The programs that comprise 'make check' should succeed.  

For example, if the "gdb.base/annota1.exp: breakpoint info" testsuite fails, 
it is a failure of that testsuite, not 'make check'.  Not finding 'runtest' 
would be an example of a failure in 'make check', that should return a 
non-zero exit code.

Currently 'make check' always returns a non-zero exit code indicating the 
testsuite results are unreliable due to a failure in 'make check'. 

I hope this makes things clearer.

Best Regards,


--George

Re: exit status of 'make check'

[Daniel Jacobowitz]

On Sun, Nov 25, 2001 at 12:37:42PM -0500, George France wrote:
> > That pretty much means that it did.  You can grep for ^FAIL: to see how
> > many there are.  I'm hoping to see the GDB testsuite pass entirely for
> > some target, someday...
> 
> Greetings,
> 
> Even if all the testsuites have a FAILed status, 'make check' should not 
> return a non-zero exit code unless there is a failure in the 'make check' 
> programs. The programs that comprise 'make check' should succeed.  
> 
> For example, if the "gdb.base/annota1.exp: breakpoint info" testsuite fails, 
> it is a failure of that testsuite, not 'make check'.  Not finding 'runtest' 
> would be an example of a failure in 'make check', that should return a 
> non-zero exit code.
> 
> Currently 'make check' always returns a non-zero exit code indicating the 
> testsuite results are unreliable due to a failure in 'make check'. 
> 
> I hope this makes things clearer.

I really can't agree.  I don't care (in an idealized world in which
we've finished fixing the GDB testsuite, which we're working on...)
what went wrong.  I'd consider make check to have failed if a test
failed - for use in, say, unattended builds.

From runtest's man page:

EXIT CODES
       runtest sets the exit code to 1 if any of the tests failed, or
       sets it to 0 if all the
       tests passed.

DejaGNU seems to agree with me on this one.

-- 
Daniel Jacobowitz                           Carnegie Mellon University
MontaVista Software                         Debian GNU/Linux Developer

Re: exit status of 'make check'

[George France]

[-- Attachment #1: Type: text/plain, Size: 537 bytes --]

We are not discussing, the results of 'runtest'. We are discussing the 
results of 'make check'.  Here is another simple example, On 2001-11-22, I 
built the gcc_3.0.2 HEAD branch.  'make check' returned a zero for an exit 
code, but many of the testsuites FAILed.


--George

I have attached a copy of the results.

>
> From runtest's man page:
>
> EXIT CODES
>        runtest sets the exit code to 1 if any of the tests failed, or
>        sets it to 0 if all the
>        tests passed.
>
> DejaGNU seems to agree with me on this one.

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: X-check.log --]
[-- Type: text/english; name="X-check.log", Size: 7427 bytes --]

make[1]: Entering directory `/home/buildsys/build/intel1/bootstrap-gcc_3.0.2/native/gcc-head/gcc'
Making a new config file...
echo "set tmpdir /home/buildsys/build/intel1/bootstrap-gcc_3.0.2/native/gcc-head/gcc/testsuite" >> ./tmp0
test -d testsuite || mkdir testsuite
rm -f testsuite/site.exp
sed '/set tmpdir/ s|testsuite|testsuite|' < site.exp > testsuite/site.exp
(rootme=`pwd`; export rootme; \
srcdir=`cd /home/buildsys/source/gcc-head/gcc; pwd` ; export srcdir ; \
cd testsuite; \
EXPECT=expect ; export EXPECT ; \
if [ -f ${rootme}/../expect/expect ] ; then  \
   TCL_LIBRARY=`cd .. ; cd /home/buildsys/source/gcc-head/gcc/../tcl/library ; pwd` ; \
   export TCL_LIBRARY ; fi ; \
runtest --tool gcc )
WARNING: global config file /home/buildsys/site.exp not found.
Test Run By france on Thu Nov 22 13:51:44 2001
Native configuration is i686-pc-linux-gnu

		=== gcc tests ===

Schedule of variations:
    unix

Running target unix
Using /mnt/u2/toolchain/baseline/install/intel1/bootstrap-gcc_3.0.2/native/dejagnu-1.4.2/share/dejagnu/baseboards/unix.exp as board description file for target.
Using /mnt/u2/toolchain/baseline/install/intel1/bootstrap-gcc_3.0.2/native/dejagnu-1.4.2/share/dejagnu/config/unix.exp as generic interface file for target.
Using /home/buildsys/source/gcc-head/gcc/testsuite/config/default.exp as tool-and-target-specific interface file.
Running /home/buildsys/source/gcc-head/gcc/testsuite/gcc.c-torture/compile/compile.exp ...
FAIL: gcc.c-torture/compile/20011114-1.c,  -O0  
FAIL: gcc.c-torture/compile/20011114-1.c,  -O1  
FAIL: gcc.c-torture/compile/20011114-1.c,  -O2  
FAIL: gcc.c-torture/compile/20011114-1.c,  -O3 -fomit-frame-pointer  
FAIL: gcc.c-torture/compile/20011114-1.c,  -O3 -g  
FAIL: gcc.c-torture/compile/20011114-1.c,  -Os  
Running /home/buildsys/source/gcc-head/gcc/testsuite/gcc.c-torture/execute/execute.exp ...
FAIL: gcc.c-torture/execute/20011114-1.c execution,  -O3 -fomit-frame-pointer 
FAIL: gcc.c-torture/execute/20011114-1.c execution,  -O3 -g 
Running /home/buildsys/source/gcc-head/gcc/testsuite/gcc.c-torture/execute/ieee/ieee.exp ...
Running /home/buildsys/source/gcc-head/gcc/testsuite/gcc.c-torture/execute/memcheck/memcheck.exp ...
Running /home/buildsys/source/gcc-head/gcc/testsuite/gcc.c-torture/unsorted/unsorted.exp ...
Running /home/buildsys/source/gcc-head/gcc/testsuite/gcc.dg/cpp/cpp.exp ...
Running /home/buildsys/source/gcc-head/gcc/testsuite/gcc.dg/dg.exp ...
FAIL: gcc.dg/c99-intconst-1.c (test for excess errors)
Running /home/buildsys/source/gcc-head/gcc/testsuite/gcc.dg/format/format.exp ...
Running /home/buildsys/source/gcc-head/gcc/testsuite/gcc.dg/noncompile/noncompile.exp ...
Running /home/buildsys/source/gcc-head/gcc/testsuite/gcc.dg/special/ecos.exp ...
Running /home/buildsys/source/gcc-head/gcc/testsuite/gcc.dg/special/special.exp ...
Running /home/buildsys/source/gcc-head/gcc/testsuite/gcc.misc-tests/acker1.exp ...
Running /home/buildsys/source/gcc-head/gcc/testsuite/gcc.misc-tests/bprob.exp ...
Running /home/buildsys/source/gcc-head/gcc/testsuite/gcc.misc-tests/dg-test.exp ...
Running /home/buildsys/source/gcc-head/gcc/testsuite/gcc.misc-tests/dhry.exp ...
Running /home/buildsys/source/gcc-head/gcc/testsuite/gcc.misc-tests/gcov.exp ...
Running /home/buildsys/source/gcc-head/gcc/testsuite/gcc.misc-tests/linkage.exp ...
Running /home/buildsys/source/gcc-head/gcc/testsuite/gcc.misc-tests/matrix1.exp ...
Running /home/buildsys/source/gcc-head/gcc/testsuite/gcc.misc-tests/mg.exp ...
Running /home/buildsys/source/gcc-head/gcc/testsuite/gcc.misc-tests/sieve.exp ...
Running /home/buildsys/source/gcc-head/gcc/testsuite/gcc.misc-tests/sort2.exp ...

		=== gcc Summary ===

# of expected passes		16336
# of unexpected failures	9
# of expected failures		61
# of unsupported tests		25
/home/buildsys/build/intel1/bootstrap-gcc_3.0.2/native/gcc-head/gcc/xgcc version 3.1 20011122 (experimental)


make[1]: [check-gcc] Error 1 (ignored)
(rootme=`pwd`; export rootme; \
srcdir=`cd /home/buildsys/source/gcc-head/gcc; pwd` ; export srcdir ; \
cd testsuite; \
EXPECT=expect ; export EXPECT ; \
if [ -f ${rootme}/../expect/expect ] ; then  \
   TCL_LIBRARY=`cd .. ; cd /home/buildsys/source/gcc-head/gcc/../tcl/library ; pwd` ; \
    export TCL_LIBRARY ; fi ; \
runtest --tool g++ )
WARNING: global config file /home/buildsys/site.exp not found.
Test Run By france on Thu Nov 22 14:10:15 2001
Native configuration is i686-pc-linux-gnu

		=== g++ tests ===

Schedule of variations:
    unix

Running target unix
Using /mnt/u2/toolchain/baseline/install/intel1/bootstrap-gcc_3.0.2/native/dejagnu-1.4.2/share/dejagnu/baseboards/unix.exp as board description file for target.
Using /mnt/u2/toolchain/baseline/install/intel1/bootstrap-gcc_3.0.2/native/dejagnu-1.4.2/share/dejagnu/config/unix.exp as generic interface file for target.
Using /home/buildsys/source/gcc-head/gcc/testsuite/config/default.exp as tool-and-target-specific interface file.
Running /home/buildsys/source/gcc-head/gcc/testsuite/g++.dg/dg.exp ...
Running /home/buildsys/source/gcc-head/gcc/testsuite/g++.dg/special/ecos.exp ...
Running /home/buildsys/source/gcc-head/gcc/testsuite/g++.old-deja/old-deja.exp ...
FAIL: g++.other/enum5.C (test for excess errors)
XPASS: g++.pt/decl2.C ICE - (test for errors, line 8)
XPASS: g++.robertl/eb17.C  Execution test

		=== g++ Summary ===

# of expected passes		6726
# of unexpected failures	1
# of unexpected successes	2
# of expected failures		94
# of untested testcases		9
/home/buildsys/build/intel1/bootstrap-gcc_3.0.2/native/gcc-head/gcc/testsuite/../g++ version 3.1 20011122 (experimental)


make[1]: [check-g++] Error 1 (ignored)
(rootme=`pwd`; export rootme; \
srcdir=`cd /home/buildsys/source/gcc-head/gcc; pwd` ; export srcdir ; \
cd testsuite; \
EXPECT=expect ; export EXPECT ; \
if [ -f ${rootme}/../expect/expect ] ; then  \
   TCL_LIBRARY=`cd .. ; cd /home/buildsys/source/gcc-head/gcc/../tcl/library ; pwd` ; \
   export TCL_LIBRARY ; fi ; \
runtest --tool g77 )
WARNING: global config file /home/buildsys/site.exp not found.
Test Run By france on Thu Nov 22 14:22:04 2001
Native configuration is i686-pc-linux-gnu

		=== g77 tests ===

Schedule of variations:
    unix

Running target unix
Using /mnt/u2/toolchain/baseline/install/intel1/bootstrap-gcc_3.0.2/native/dejagnu-1.4.2/share/dejagnu/baseboards/unix.exp as board description file for target.
Using /mnt/u2/toolchain/baseline/install/intel1/bootstrap-gcc_3.0.2/native/dejagnu-1.4.2/share/dejagnu/config/unix.exp as generic interface file for target.
Using /home/buildsys/source/gcc-head/gcc/testsuite/config/default.exp as tool-and-target-specific interface file.
Running /home/buildsys/source/gcc-head/gcc/testsuite/g77.dg/bprob/bprob.exp ...
Running /home/buildsys/source/gcc-head/gcc/testsuite/g77.dg/dg.exp ...
Running /home/buildsys/source/gcc-head/gcc/testsuite/g77.dg/gcov/gcov.exp ...
Running /home/buildsys/source/gcc-head/gcc/testsuite/g77.f-torture/compile/compile.exp ...
Running /home/buildsys/source/gcc-head/gcc/testsuite/g77.f-torture/execute/execute.exp ...
Running /home/buildsys/source/gcc-head/gcc/testsuite/g77.f-torture/noncompile/noncompile.exp ...

		=== g77 Summary ===

# of expected passes		1360
/home/buildsys/build/intel1/bootstrap-gcc_3.0.2/native/gcc-head/gcc/testsuite/../g77 version 3.1 20011122 (experimental)


make[1]: Leaving directory `/home/buildsys/build/intel1/bootstrap-gcc_3.0.2/native/gcc-head/gcc'

Re: exit status of 'make check'

[Daniel Jacobowitz]

On Sun, Nov 25, 2001 at 01:40:09PM -0500, George France wrote:
> We are not discussing, the results of 'runtest'. We are discussing the 
> results of 'make check'.  Here is another simple example, On 2001-11-22, I 
> built the gcc_3.0.2 HEAD branch.  'make check' returned a zero for an exit 
> code, but many of the testsuites FAILed.

GCC does this by completely ignoring the exit code of runtest; in other
words, make check will always succeed.  I don't think that's an
acceptable step.  Do you?

We are discussing the result of runtest, because the DejaGNU
maintainers are the appropriate people to set policy for this sort of
thing.

-- 
Daniel Jacobowitz                           Carnegie Mellon University
MontaVista Software                         Debian GNU/Linux Developer

Re: exit status of 'make check'

[George France]

On Sunday 25 November 2001 13:54, Daniel Jacobowitz wrote:
> On Sun, Nov 25, 2001 at 01:40:09PM -0500, George France wrote:
> > We are not discussing, the results of 'runtest'. We are discussing the
> > results of 'make check'.  Here is another simple example, On 2001-11-22,
> > I built the gcc_3.0.2 HEAD branch.  'make check' returned a zero for an
> > exit code, but many of the testsuites FAILed.
>
> GCC does this by completely ignoring the exit code of runtest; in other
> words, make check will always succeed.  I don't think that's an
> acceptable step.  Do you?
>

I looked at the Makefiles (gdb & gcc).  You are correct.  It is equally 
unacceptable to always succeed.  <sigh>

> We are discussing the result of runtest, because the DejaGNU
> maintainers are the appropriate people to set policy for this sort of
> thing.

I am disappointed that there is not a more consistent GNU policy in regards 
to the meaning of exit code for 'make check'.  For the auto-build system that 
I use, it makes much more sense to be able to separate errors from 'make 
check' into two categories:

A) errors in the check system - Example, 'runtest' not being available.
B) testsuite results either PASS / FAIL - Which are the result of a 
successful run of the check system.

I can see that this is not possible under the current policy.  I will have to 
research this matter.  Any further suggestions would be greatly appreciated.

Thank you for your help.

Best Regards,


--George 

Re: exit status of 'make check'

[George France]

On Sunday 25 November 2001 13:54, Daniel Jacobowitz wrote:
> On Sun, Nov 25, 2001 at 01:40:09PM -0500, George France wrote:
> > We are not discussing, the results of 'runtest'. We are discussing the
> > results of 'make check'.  Here is another simple example, On 2001-11-22,
> > I built the gcc_3.0.2 HEAD branch.  'make check' returned a zero for an
> > exit code, but many of the testsuites FAILed.
>
> GCC does this by completely ignoring the exit code of runtest; in other
> words, make check will always succeed.  I don't think that's an
> acceptable step.  Do you?
>

I looked at the Makefiles (gdb & gcc).  You are correct.  It is equally 
unacceptable to always succeed.  <sigh>

> We are discussing the result of runtest, because the DejaGNU
> maintainers are the appropriate people to set policy for this sort of
> thing.

I am disappointed that there is not a more consistent GNU policy in regards 
to the meaning of exit code for 'make check'.  For the auto-build system that 
I use, it makes much more sense to be able to separate errors from 'make 
check' into two categories:

A) errors in the check system - Example, 'runtest' not being available.
B) testsuite results either PASS / FAIL - Which are the result of a 
successful run of the check system.

I can see that this is not possible under the current policy.  I will have to 
research this matter.  Any further suggestions would be greatly appreciated.

Thank you for your help.

Best Regards,


--George 

exit status of 'make check'

[George France]

Greetings,

When I ever I do a 'make check' on gdb using the  cvs HEAD tree on either the 
intel or Alpha architecture. It always returnes a non-zero exit code. The 
testsuites execute and produce a good gdb.log and gdb.sum files.  Has anybody 
seen this before??

Any Ideas??

Best Regards,


--George