[RFA] bailing out of tests with multiple errors or timeouts

[RFA] bailing out of tests with multiple errors or timeouts

[Janis Johnson]

[-- Attachment #1: Type: text/plain, Size: 491 bytes --]

Several gdb tests get multiple ERRORS and/or timeouts for some multilibs
of some embedded targets, with a few tests timing out a dozen times.  I
would like to modify these tests to look for the first expected ERROR or
timeout and then bail out with an indication that the remainder of the
test has been skipped.  I haven't found a standard way to do this; any
suggestions?  I've attached a couple of examples as a starting point for
discussion.

Janis Johnson
CodeSourcery / Mentor Graphics

[-- Attachment #2: gdb-20110504-2 --]
[-- Type: text/plain, Size: 1597 bytes --]

2011-05-04  Janis Johnson  <janisjo@codesourcery.com>

	* gdb.threads/manythreads.exp: Bail out after an error that
	indicates the test might have further errors.
	* gdb.threads/staticthreads.exp: Bail out after a timeout
	that indicates the test might have further timeouts.

Index: testsuite/gdb.threads/manythreads.exp
===================================================================
RCS file: /cvs/src/src/gdb/testsuite/gdb.threads/manythreads.exp,v
retrieving revision 1.22
diff -u -p -r1.22 manythreads.exp
--- testsuite/gdb.threads/manythreads.exp	19 Jan 2011 17:21:39 -0000	1.22
+++ testsuite/gdb.threads/manythreads.exp	4 May 2011 17:34:16 -0000
@@ -97,6 +97,11 @@ gdb_test_multiple $cmd $cmd {
 	    fail $cmd
 	}
     }
+    eof {
+	fail "$cmd (Process no longer exists)"
+	fail "Remainder of test"
+	return
+    }
 }
 
 gdb_test_no_output "thread name zardoz" "give a name to the thread"
Index: testsuite/gdb.threads/staticthreads.exp
===================================================================
RCS file: /cvs/src/src/gdb/testsuite/gdb.threads/staticthreads.exp,v
retrieving revision 1.13
diff -u -p -r1.13 staticthreads.exp
--- testsuite/gdb.threads/staticthreads.exp	1 Jan 2011 15:33:50 -0000	1.13
+++ testsuite/gdb.threads/staticthreads.exp	4 May 2011 17:34:16 -0000
@@ -54,6 +54,12 @@ gdb_test_multiple "continue" "$test" {
     -re "Program received signal .*$gdb_prompt " {
 	kfail gdb/1328 "$test"
     }
+    timeout {
+	# A timeout here often indicates more timeouts later in the test.
+	fail "$test (timeout)"
+	fail "Remainder of test"
+	return
+    }
 }
     
 

Re: [RFA] bailing out of tests with multiple errors or timeouts

[Mark Kettenis]

> Date: Wed, 04 May 2011 10:46:50 -0700
> From: Janis Johnson <janisjo@codesourcery.com>

> +    timeout {
> +	# A timeout here often indicates more timeouts later in the test.
> +	fail "$test (timeout)"
> +	fail "Remainder of test"
> +	return
> +    }

Often, so not always.  Really the best way to get rid of these is to
fix the underlying problem.

Re: [RFA] bailing out of tests with multiple errors or timeouts

[Janis Johnson]

On 05/04/2011 11:10 AM, Mark Kettenis wrote:
>> Date: Wed, 04 May 2011 10:46:50 -0700
>> From: Janis Johnson <janisjo@codesourcery.com>
> 
>> +    timeout {
>> +	# A timeout here often indicates more timeouts later in the test.
>> +	fail "$test (timeout)"
>> +	fail "Remainder of test"
>> +	return
>> +    }
> 
> Often, so not always.  Really the best way to get rid of these is to
> fix the underlying problem.

I agree, the best way to get rid of timeouts and ERRORs in gdb tests is
to fix gdb.  Unfortunately, that's not an easy task for a target whose
support is buggy or incomplete whether in gdb, simulator, hardware, or
operating system.  It's painful to test on such targets when a few tests
time out multiple times.

What I'm looking for is a way to bail out of tests that are known to
time out multiple times on some targets, or have a cascade of ERRORs.
That might involve changes to specific tests like the examples I posted;
it might involve setting a magic variable somewhere to bail out of any
test that times out or runs into an error; it might involve having a
list of tests that bail out after a timeout or error.  Ideally there
would be an easily-recognizable failure message for those tests to
indicate it's more than a normal failure; maybe UNRESOLVED.

I'm not familiar enough with the workings of the gdb test infrastructure
or the general conventions of the gdb testsuite to know what might be
appropriate, but presumably others have run into this problem and have
ideas.

Janis Johnson
CodeSourcery / Mentor Graphics

Re: [RFA] bailing out of tests with multiple errors or timeouts

[Joel Brobecker]

There was a discussion in the past about this, and we thought that
the benefit of continuing after detecting a TIMEOUT was, overall,
usually nul. A timeout is often a source of loss of synchronization
between what you're sending and the associated output, so continuing
in that situation isn't going to be all that useful. I'm sure we'll
find some counter examples, but the ration between benefits vs time
wasted is probably going to remain very small.

So the conclusion the two of us who discussed this reached was that
we should just abort the testcase at the first TIMEOUT.  But rather
than do it in an ad hoc way as is done here, do it systematically.
The problem is, I haven't found a way to do this, yet. I thought
about raising an exception, but I don't think there is a way for
the testsuite to trap it before dejagnu sees it...

-- 
Joel