Re: How to run multiple target boards

[Tom de Vries <tdevries@suse.de>]

On 26-05-2020 12:39, Pedro Alves wrote:
> On 5/26/20 9:49 AM, Tom de Vries wrote:
>> [ was: Re: [committed][gdb/testsuite] Fix incorrect string concat in
>> jit-elf.exp ]
>>
>> On 13-05-2020 15:45, Simon Marchi wrote:
>>> FWIW, I just learned it's possible to pass a list of target boards to --target_board:
>>>
>>>   --target_board='unix native-gdbserver native-extended-gdbserver'
>>>
>>> And it gives a combined result at the end.  So I'll make myself an alias to test on all
>>> the commonly used boards, probably:
>>>
>>> - cc-with-debug-names.exp
>>> - cc-with-dwz.exp
>>> - cc-with-dwz-m.exp
>>> - cc-with-gdb-index.exp
>>> - debug-types.exp
>>> - dwarf4-gdb-index.exp
>>> - fission-dwp.exp
>>> - fission.exp
>>> - native-extended-gdbserver.exp
>>> - native-gdbserver.exp
>>> - native-stdio-gdbserver.exp
>>> - readnow.exp
> 
> What are the advantages of doing this compared to a script that runs the
> testsuite for each of the boards, separately?
> 

I'd say:
- the target boards not overwriting the .log and .sum files of other
  target boards
- the joint summary.

Thanks,
- Tom

>>>
>>> When working on a single test, it's usually not to long to run them all.  Of course, it's
>>> not really practical to run the complete testsuite twice (before and after) for each board
>>> for every change we do...
>>
>> Hi Simon,
>>
>> I just tried this out.
>>
>> My test scripts use make check, but I didn't manage to make that work
>> yet, so I tried out using runtest directly (and I may be missing
>> something obvious, given that I haven't used this before):
>> ...
>> $ cd build/gdb/testsuite
>> $ runtest gdb.base/gold-gdb-index.exp --target_board='cc-with-gdb-index
>>  unix'
>> ...
> 
> One issue this with that I've run into in the past, is with global
> variables leaking from one board to the other.
> 
> And I've just tried a quick test, and lucky me I seem to have run into
> something like that immediately:
> 
>  $ make check RUNTESTFLAGS="--target_board='native-gdbserver unix' break.exp"
>  ...
>  Schedule of variations:
>      native-gdbserver
>      unix
>  ...
>  FAIL: gdb.base/break.exp: run until function breakpoint
>  FAIL: gdb.base/break.exp: run until breakpoint set at a line number (the program is no longer running)
>  FAIL: gdb.base/break.exp: run until file:function(6) breakpoint (the program is no longer running)
>  FAIL: gdb.base/break.exp: run until file:function(5) breakpoint (the program is no longer running)
>  (snip a bunch more)
> 
> (Running break.exp for each board in isolation passes cleanly, of course.)
> 
> Another thing is that the test messages in gdb.sum don't indicate which
> board issued the PASS/FAIL (on each result), making it more difficult
> to analyze.
> 
>>
>> It seems however that the global settings (CC_FOR_TARGET etc) from the
>> first board stay active in the unix board:
> 
> Yeah, things like that.
> 
> Also, the tests for all boards end up under the same testsuite/outputs/ directory,
> so whatever the last board was, wins.  I'm not even sure how the testsuite
> caching works when you run more than one board in parallel mode.
> 
> It would seem to me that if you're looking at running tests against different
> boards, that you would want the output directories to be separate, so you can
> analyze the results afterwards, avoid cache issues, etc.
> 
> Maybe it would be better to come up with our own way to do this, without
> relying on dejagnu directly.  Much like we have the TESTS variable,
> we could have a BOARDS variable:
> 
>  make check TESTS="gdb.base/break.exp" BOARDS="native-gdbserver unix"
> 
> and then the outputs foreach board would be in
> "testsuite/outputs.native-gdbserver/" and "testsuite/outputs.unix/"
> respectively.
> 
> Thanks,
> Pedro Alves
>