Re: How to run multiple target boards

[Pedro Alves <palves@redhat.com>]

On 5/26/20 9:49 AM, Tom de Vries wrote:
> [ was: Re: [committed][gdb/testsuite] Fix incorrect string concat in
> jit-elf.exp ]
> 
> On 13-05-2020 15:45, Simon Marchi wrote:
>> FWIW, I just learned it's possible to pass a list of target boards to --target_board:
>>
>>   --target_board='unix native-gdbserver native-extended-gdbserver'
>>
>> And it gives a combined result at the end.  So I'll make myself an alias to test on all
>> the commonly used boards, probably:
>>
>> - cc-with-debug-names.exp
>> - cc-with-dwz.exp
>> - cc-with-dwz-m.exp
>> - cc-with-gdb-index.exp
>> - debug-types.exp
>> - dwarf4-gdb-index.exp
>> - fission-dwp.exp
>> - fission.exp
>> - native-extended-gdbserver.exp
>> - native-gdbserver.exp
>> - native-stdio-gdbserver.exp
>> - readnow.exp

What are the advantages of doing this compared to a script that runs the
testsuite for each of the boards, separately?

>>
>> When working on a single test, it's usually not to long to run them all.  Of course, it's
>> not really practical to run the complete testsuite twice (before and after) for each board
>> for every change we do...
> 
> Hi Simon,
> 
> I just tried this out.
> 
> My test scripts use make check, but I didn't manage to make that work
> yet, so I tried out using runtest directly (and I may be missing
> something obvious, given that I haven't used this before):
> ...
> $ cd build/gdb/testsuite
> $ runtest gdb.base/gold-gdb-index.exp --target_board='cc-with-gdb-index
>  unix'
> ...

One issue this with that I've run into in the past, is with global
variables leaking from one board to the other.

And I've just tried a quick test, and lucky me I seem to have run into
something like that immediately:

 $ make check RUNTESTFLAGS="--target_board='native-gdbserver unix' break.exp"
 ...
 Schedule of variations:
     native-gdbserver
     unix
 ...
 FAIL: gdb.base/break.exp: run until function breakpoint
 FAIL: gdb.base/break.exp: run until breakpoint set at a line number (the program is no longer running)
 FAIL: gdb.base/break.exp: run until file:function(6) breakpoint (the program is no longer running)
 FAIL: gdb.base/break.exp: run until file:function(5) breakpoint (the program is no longer running)
 (snip a bunch more)

(Running break.exp for each board in isolation passes cleanly, of course.)

Another thing is that the test messages in gdb.sum don't indicate which
board issued the PASS/FAIL (on each result), making it more difficult
to analyze.

> 
> It seems however that the global settings (CC_FOR_TARGET etc) from the
> first board stay active in the unix board:

Yeah, things like that.

Also, the tests for all boards end up under the same testsuite/outputs/ directory,
so whatever the last board was, wins.  I'm not even sure how the testsuite
caching works when you run more than one board in parallel mode.

It would seem to me that if you're looking at running tests against different
boards, that you would want the output directories to be separate, so you can
analyze the results afterwards, avoid cache issues, etc.

Maybe it would be better to come up with our own way to do this, without
relying on dejagnu directly.  Much like we have the TESTS variable,
we could have a BOARDS variable:

 make check TESTS="gdb.base/break.exp" BOARDS="native-gdbserver unix"

and then the outputs foreach board would be in
"testsuite/outputs.native-gdbserver/" and "testsuite/outputs.unix/"
respectively.

Thanks,
Pedro Alves