From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (qmail 128055 invoked by alias); 14 Jan 2016 17:28:31 -0000 Mailing-List: contact gdb-patches-help@sourceware.org; run by ezmlm Precedence: bulk List-Id: List-Subscribe: List-Archive: List-Post: List-Help: , Sender: gdb-patches-owner@sourceware.org Received: (qmail 128039 invoked by uid 89); 14 Jan 2016 17:28:30 -0000 Authentication-Results: sourceware.org; auth=none X-Virus-Found: No X-Spam-SWARE-Status: No, score=-0.9 required=5.0 tests=BAYES_00,KAM_LAZY_DOMAIN_SECURITY,RP_MATCHES_RCVD,SPF_HELO_PASS autolearn=no version=3.3.2 spammy=160114, sourcing, op, disambiguate X-HELO: mx1.redhat.com Received: from mx1.redhat.com (HELO mx1.redhat.com) (209.132.183.28) by sourceware.org (qpsmtpd/0.93/v0.84-503-g423c35a) with (AES256-GCM-SHA384 encrypted) ESMTPS; Thu, 14 Jan 2016 17:28:29 +0000 Received: from int-mx09.intmail.prod.int.phx2.redhat.com (int-mx09.intmail.prod.int.phx2.redhat.com [10.5.11.22]) by mx1.redhat.com (Postfix) with ESMTPS id 6CB8218B322; Thu, 14 Jan 2016 17:28:28 +0000 (UTC) Received: from [127.0.0.1] (ovpn01.gateway.prod.ext.phx2.redhat.com [10.5.9.1]) by int-mx09.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u0EHSR5G003226; Thu, 14 Jan 2016 12:28:27 -0500 Message-ID: <5697DABA.8010008@redhat.com> Date: Thu, 14 Jan 2016 17:28:00 -0000 From: Pedro Alves User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.7.0 MIME-Version: 1.0 To: Simon Marchi , dejagnu@gnu.org, gdb-patches Subject: Re: How to abort a test? References: <56958359.8070708@ericsson.com> <5697CC09.8010306@redhat.com> <5697D721.1000305@ericsson.com> In-Reply-To: <5697D721.1000305@ericsson.com> Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-SW-Source: 2016-01/txt/msg00312.txt.bz2 On 01/14/2016 05:13 PM, Simon Marchi wrote: > On 16-01-14 11:25 AM, Pedro Alves wrote: >>> The test will be aborted, runtest will output a detailed error, but the test will still >>> pass. Intuitively, I would think that a test that throws an error should automatically >>> be failed or unresolved, since something unexpected happened. >> >> IIUC, you wouldn't want pass/fail to even be reached, so I don't understand >> what test would you want to fail. > > I think unresolved would be appropriate in that case, since it's not the program under test > failed, but the test, or its setup/environment. OK. I'd say "testcase" to disambiguate with a particular pass/fail invocation. > > According to the Dejagnu doc, unresolved: "... usually means the test did not execute as > expected, and a human being must go over results to determine if it passed or failed (and > to improve the test case). ". I think that applies here. > > So when there is an uncaught exception coming from the test, Dejagnu could issue a: > > unresolved "Uncaught error from test " # needs a better message > > The important part is that the runtest command returns a failure return code, so that > automated builds or scripts consider the run as failed. To me, a test that ends with > an exception is not a test that ran successfully. > >>> >>> The only option I see right now would be to fix the whole return chain and add proper >>> error handling everywhere, to exit early when an error happens. However, that means >>> changing tens (hundreds?) of callsites through the testsuite, which is why I'm >>> looking for alternative solutions first. >> >> Test usually return early if running to main fails (if ![runto_main]), so maybe it >> won't be that many places. Maybe some judiciously placed "return -code return" >> hacks saves you a good chunk. I don't have any other idea. > > Many tests use gdb_run_cmd directly without checking the result: > > gdb/testsuite $ grep -rin '^gdb_run_cmd$' gdb.* | wc -l > 73 > > So they would need to be changed: > > -gdb_run_cmd > +if ![gdb_run_cmd] { > + fail "Failed to run" > +} > That's where the "return -code return" hack would come in. You'd do that inside gdb_run_cmd so that would return directly to gdb_run_cmd's caller. But it's an ugly hack. >> For the particular case of gdbserver not being present in the target, >> probably the easiest would be to check that earlier, likely before >> runtest, even. Not ideal, since the testsuite can mix native and gdbserver >> tests, for instance, but... > > And it's a bit hard to check in the case of a remote target, given that it's runtest > that knows how to "compute" the remote hostname, username, expected gdbserver path, > etc, from the --target_board. I meant, before runtest, the dejagnu procedure, not before invoking the runtest binary. E.g., straight from the target board file, say. E.g., with: --- c/gdb/testsuite/boards/gdbserver-base.exp +++ w/gdb/testsuite/boards/gdbserver-base.exp @@ -33,6 +33,8 @@ set_board_info gdb,predefined_tsv "\\\$trace_timestamp" set GDBFLAGS "${GDBFLAGS} -ex \"set auto-connect-native-target off\"" +error "gdbserver not present on target!" + proc ${board}_file { dest op args } { if { $op == "delete" } { return 0 then I get: $ time make check RUNTESTFLAGS="--target_board=native-gdbserver" make[1]: Entering directory `/home/pedro/gdb/mygit/build/gdb/testsuite' Nothing to be done for all... make check-single make[2]: Entering directory `/home/pedro/gdb/mygit/build/gdb/testsuite' rootme=`pwd`; export rootme; srcdir=/home/pedro/gdb/mygit/src/gdb/testsuite ; export srcdir ; EXPECT=`if [ "${READ1}" != "" ] ; then echo ${rootme}/expect-read1; elif [ -f ${rootme}/../../expect/expect ] ; then echo ${rootme}/../../expect/expect ; else echo expect ; fi` ; export EXPECT ; EXEEXT= ; export EXEEXT ; LD_LIBRARY_PATH=$rootme/../../expect:$rootme/../../libstdc++:$rootme/../../tk/unix:$rootme/../../tcl/unix:$rootme/../../bfd:$rootme/../../opcodes:$LD_LIBRARY_PATH; export LD_LIBRARY_PATH; if [ -f ${rootme}/../../expect/expect ] ; then TCL_LIBRARY=${srcdir}/../../tcl/library ; export TCL_LIBRARY ; fi ; runtest --target_board=native-gdbserver Test Run By pedro on Thu Jan 14 17:24:37 2016 Native configuration is x86_64-pc-linux-gnu === gdb tests === Schedule of variations: native-gdbserver Running target native-gdbserver Using /home/pedro/gdb/mygit/src/gdb/testsuite/boards/../boards/native-gdbserver.exp as board description file for target. Using /home/pedro/gdb/mygit/src/gdb/testsuite/boards/../boards/gdbserver-base.exp as board description file for target. ERROR: tcl error sourcing board description file for target /home/pedro/gdb/mygit/src/gdb/testsuite/boards/../boards/gdbserver-base.exp. gdbserver not present on target! gdbserver not present on target! while executing "error "gdbserver not present on target!"" (file "/home/pedro/gdb/mygit/src/gdb/testsuite/boards/../boards/gdbserver-base.exp" line 36) invoked from within "source /home/pedro/gdb/mygit/src/gdb/testsuite/boards/../boards/gdbserver-base.exp" ("uplevel" body line 1) invoked from within "uplevel #0 source /home/pedro/gdb/mygit/src/gdb/testsuite/boards/../boards/gdbserver-base.exp" invoked from within "catch "uplevel #0 source ${dir}/${initfile}" error" make[2]: *** [check-single] Error 1 make[2]: Leaving directory `/home/pedro/gdb/mygit/build/gdb/testsuite' make[1]: *** [check] Error 2 make[1]: Leaving directory `/home/pedro/gdb/mygit/build/gdb/testsuite' make: *** [check] Error 2 real 0m0.404s user 0m0.336s sys 0m0.060s Obviously not ideal; I'm just pointing out the likely easiest. Thanks, Pedro Alves