Re: How to abort a test?

[Pedro Alves <palves@redhat.com>]

On 01/14/2016 05:13 PM, Simon Marchi wrote:
> On 16-01-14 11:25 AM, Pedro Alves wrote:
>>> The test will be aborted, runtest will output a detailed error, but the test will still
>>> pass.  Intuitively, I would think that a test that throws an error should automatically
>>> be failed or unresolved, since something unexpected happened.
>>
>> IIUC, you wouldn't want pass/fail to even be reached, so I don't understand
>> what test would you want to fail.
> 
> I think unresolved would be appropriate in that case, since it's not the program under test
> failed, but the test, or its setup/environment.

OK.  I'd say "testcase" to disambiguate with a particular pass/fail invocation.

> 
> According to the Dejagnu doc, unresolved: "... usually means the test did not execute as
> expected, and a human being must go over results to determine if it passed or failed (and
> to improve the test case). ".  I think that applies here.
> 
> So when there is an uncaught exception coming from the test, Dejagnu could issue a:
> 
>   unresolved "Uncaught error from test " # needs a better message
> 
> The important part is that the runtest command returns a failure return code, so that
> automated builds or scripts consider the run as failed.  To me, a test that ends with
> an exception is not a test that ran successfully.
> 
>>>
>>> The only option I see right now would be to fix the whole return chain and add proper
>>> error handling everywhere, to exit early when an error happens.  However, that means
>>> changing tens (hundreds?) of callsites through the testsuite, which is why I'm
>>> looking for alternative solutions first.
>>
>> Test usually return early if running to main fails (if ![runto_main]), so maybe it
>> won't be that many places.  Maybe some judiciously placed "return -code return"
>> hacks saves you a good chunk.  I don't have any other idea.
> 
> Many tests use gdb_run_cmd directly without checking the result:
> 
> gdb/testsuite $ grep -rin '^gdb_run_cmd$' gdb.* | wc -l
> 73
> 
> So they would need to be changed:
> 
> -gdb_run_cmd
> +if ![gdb_run_cmd] {
> +    fail "Failed to run"
> +}
> 

That's where the "return -code return" hack would come in.  You'd
do that inside gdb_run_cmd so that would return directly to gdb_run_cmd's
caller.  But it's an ugly hack.

>> For the particular case of gdbserver not being present in the target,
>> probably the easiest would be to check that earlier, likely before
>> runtest, even.  Not ideal, since the testsuite can mix native and gdbserver
>> tests, for instance, but...
> 
> And it's a bit hard to check in the case of a remote target, given that it's runtest
> that knows how to "compute" the remote hostname, username, expected gdbserver path,
> etc, from the --target_board.

I meant, before runtest, the dejagnu procedure, not before invoking the
runtest binary.  E.g., straight from the target board file, say.

E.g., with:

--- c/gdb/testsuite/boards/gdbserver-base.exp
+++ w/gdb/testsuite/boards/gdbserver-base.exp
@@ -33,6 +33,8 @@ set_board_info gdb,predefined_tsv "\\\$trace_timestamp"

 set GDBFLAGS "${GDBFLAGS} -ex \"set auto-connect-native-target off\""

+error "gdbserver not present on target!"
+
 proc ${board}_file { dest op args } {
     if { $op == "delete" } {
        return 0

then I get:

$ time make check RUNTESTFLAGS="--target_board=native-gdbserver"
make[1]: Entering directory `/home/pedro/gdb/mygit/build/gdb/testsuite'
Nothing to be done for all...
make check-single
make[2]: Entering directory `/home/pedro/gdb/mygit/build/gdb/testsuite'
rootme=`pwd`; export rootme; srcdir=/home/pedro/gdb/mygit/src/gdb/testsuite ; export srcdir ; EXPECT=`if [ "${READ1}" != "" ] ; then echo ${rootme}/expect-read1; elif [ -f ${rootme}/../../expect/expect ] ; then echo ${rootme}/../../expect/expect ; else echo expect ; fi` ; export EXPECT ; EXEEXT= ; export EXEEXT ; LD_LIBRARY_PATH=$rootme/../../expect:$rootme/../../libstdc++:$rootme/../../tk/unix:$rootme/../../tcl/unix:$rootme/../../bfd:$rootme/../../opcodes:$LD_LIBRARY_PATH; export LD_LIBRARY_PATH; if [ -f ${rootme}/../../expect/expect ] ; then TCL_LIBRARY=${srcdir}/../../tcl/library ; export TCL_LIBRARY ; fi ; runtest --target_board=native-gdbserver
Test Run By pedro on Thu Jan 14 17:24:37 2016
Native configuration is x86_64-pc-linux-gnu

                === gdb tests ===

Schedule of variations:
    native-gdbserver

Running target native-gdbserver
Using /home/pedro/gdb/mygit/src/gdb/testsuite/boards/../boards/native-gdbserver.exp as board description file for target.
Using /home/pedro/gdb/mygit/src/gdb/testsuite/boards/../boards/gdbserver-base.exp as board description file for target.
ERROR: tcl error sourcing board description file for target /home/pedro/gdb/mygit/src/gdb/testsuite/boards/../boards/gdbserver-base.exp.
gdbserver not present on target!
gdbserver not present on target!
    while executing
"error "gdbserver not present on target!""
    (file "/home/pedro/gdb/mygit/src/gdb/testsuite/boards/../boards/gdbserver-base.exp" line 36)
    invoked from within
"source /home/pedro/gdb/mygit/src/gdb/testsuite/boards/../boards/gdbserver-base.exp"
    ("uplevel" body line 1)
    invoked from within
"uplevel #0 source /home/pedro/gdb/mygit/src/gdb/testsuite/boards/../boards/gdbserver-base.exp"
    invoked from within
"catch "uplevel #0 source ${dir}/${initfile}" error"
make[2]: *** [check-single] Error 1
make[2]: Leaving directory `/home/pedro/gdb/mygit/build/gdb/testsuite'
make[1]: *** [check] Error 2
make[1]: Leaving directory `/home/pedro/gdb/mygit/build/gdb/testsuite'
make: *** [check] Error 2

real    0m0.404s
user    0m0.336s
sys     0m0.060s

Obviously not ideal; I'm just pointing out the likely easiest.

Thanks,
Pedro Alves