From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mx2.suse.de (mx2.suse.de [195.135.220.15]) by sourceware.org (Postfix) with ESMTPS id 156BC386F829 for ; Tue, 26 May 2020 10:56:50 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.3.2 sourceware.org 156BC386F829 Authentication-Results: sourceware.org; dmarc=none (p=none dis=none) header.from=suse.de Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=tdevries@suse.de X-Virus-Scanned: by amavisd-new at test-mx.suse.de Received: from relay2.suse.de (unknown [195.135.220.254]) by mx2.suse.de (Postfix) with ESMTP id DAD21AC22; Tue, 26 May 2020 10:56:51 +0000 (UTC) Subject: Re: How to run multiple target boards To: Pedro Alves , Simon Marchi , "gdb-patches@sourceware.org" References: <20200512130002.GA3290@delia> <73a5e27b-a709-8861-1f88-a07a0a2a71f9@polymtl.ca> <256a0db7-98c2-7769-7991-981f59106ee4@suse.de> <82395a9a-1595-e16c-8977-5f929f9041cc@polymtl.ca> <85005cf2-783d-7048-1cfa-8b648854c981@suse.de> <3319bd1f-a39f-1bf0-f291-e166edb2ab97@redhat.com> From: Tom de Vries Autocrypt: addr=tdevries@suse.de; keydata= xsBNBF0ltCcBCADDhsUnMMdEXiHFfqJdXeRvgqSEUxLCy/pHek88ALuFnPTICTwkf4g7uSR7 HvOFUoUyu8oP5mNb4VZHy3Xy8KRZGaQuaOHNhZAT1xaVo6kxjswUi3vYgGJhFMiLuIHdApoc u5f7UbV+egYVxmkvVLSqsVD4pUgHeSoAcIlm3blZ1sDKviJCwaHxDQkVmSsGXImaAU+ViJ5l CwkvyiiIifWD2SoOuFexZyZ7RUddLosgsO0npVUYbl6dEMq2a5ijGF6/rBs1m3nAoIgpXk6P TCKlSWVW6OCneTaKM5C387972qREtiArTakRQIpvDJuiR2soGfdeJ6igGA1FZjU+IsM5ABEB AAHNH1RvbSBkZSBWcmllcyA8dGRldnJpZXNAc3VzZS5kZT7CwKsEEwEIAD4WIQSsnSe5hKbL MK1mGmjuhV2rbOJEoAUCXSW0JwIbAwUJA8JnAAULCQgHAgYVCgkICwIEFgIDAQIeAQIXgAAh CRDuhV2rbOJEoBYhBKydJ7mEpsswrWYaaO6FXats4kSgc48H/Ra2lq5p3dHsrlQLqM7N68Fo eRDf3PMevXyMlrCYDGLVncQwMw3O/AkousktXKQ42DPJh65zoXB22yUt8m0g12xkLax98KFJ 5NyUloa6HflLl+wQL/uZjIdNUQaHQLw3HKwRMVi4l0/Jh/TygYG1Dtm8I4o708JS4y8GQxoQ UL0z1OM9hyM3gI2WVTTyprsBHy2EjMOu/2Xpod95pF8f90zBLajy6qXEnxlcsqreMaqmkzKn 3KTZpWRxNAS/IH3FbGQ+3RpWkNGSJpwfEMVCeyK5a1n7yt1podd1ajY5mA1jcaUmGppqx827 8TqyteNe1B/pbiUt2L/WhnTgW1NC1QDOwE0EXSW0JwEIAM99H34Bu4MKM7HDJVt864MXbx7B 1M93wVlpJ7Uq+XDFD0A0hIal028j+h6jA6bhzWto4RUfDl/9mn1StngNVFovvwtfzbamp6+W pKHZm9X5YvlIwCx131kTxCNDcF+/adRW4n8CU3pZWYmNVqhMUiPLxElA6QhXTtVBh1RkjCZQ Kmbd1szvcOfaD8s+tJABJzNZsmO2hVuFwkDrRN8Jgrh92a+yHQPd9+RybW2l7sJv26nkUH5Z 5s84P6894ebgimcprJdAkjJTgprl1nhgvptU5M9Uv85Pferoh2groQEAtRPlCGrZ2/2qVNe9 XJfSYbiyedvApWcJs5DOByTaKkcAEQEAAcLAkwQYAQgAJhYhBKydJ7mEpsswrWYaaO6FXats 4kSgBQJdJbQnAhsMBQkDwmcAACEJEO6FXats4kSgFiEErJ0nuYSmyzCtZhpo7oVdq2ziRKD3 twf7BAQBZ8TqR812zKAD7biOnWIJ0McV72PFBxmLIHp24UVe0ZogtYMxSWKLg3csh0yLVwc7 H3vldzJ9AoK3Qxp0Q6K/rDOeUy3HMqewQGcqrsRRh0NXDIQk5CgSrZslPe47qIbe3O7ik/MC q31FNIAQJPmKXX25B115MMzkSKlv4udfx7KdyxHrTSkwWZArLQiEZj5KG4cCKhIoMygPTA3U yGaIvI/BGOtHZ7bEBVUCFDFfOWJ26IOCoPnSVUvKPEOH9dv+sNy7jyBsP5QxeTqwxC/1ZtNS DUCSFQjqA6bEGwM22dP8OUY6SC94x1G81A9/xbtm9LQxKm0EiDH8KBMLfQ== Message-ID: Date: Tue, 26 May 2020 12:56:43 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Thunderbird/68.5.0 MIME-Version: 1.0 In-Reply-To: <3319bd1f-a39f-1bf0-f291-e166edb2ab97@redhat.com> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit X-Spam-Status: No, score=-12.5 required=5.0 tests=BAYES_00, KAM_DMARC_STATUS, RCVD_IN_MSPIKE_H3, RCVD_IN_MSPIKE_WL, SPF_HELO_NONE, SPF_PASS, TXREP autolearn=ham autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on server2.sourceware.org X-BeenThere: gdb-patches@sourceware.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Gdb-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 26 May 2020 10:56:52 -0000 On 26-05-2020 12:39, Pedro Alves wrote: > On 5/26/20 9:49 AM, Tom de Vries wrote: >> [ was: Re: [committed][gdb/testsuite] Fix incorrect string concat in >> jit-elf.exp ] >> >> On 13-05-2020 15:45, Simon Marchi wrote: >>> FWIW, I just learned it's possible to pass a list of target boards to --target_board: >>> >>> --target_board='unix native-gdbserver native-extended-gdbserver' >>> >>> And it gives a combined result at the end. So I'll make myself an alias to test on all >>> the commonly used boards, probably: >>> >>> - cc-with-debug-names.exp >>> - cc-with-dwz.exp >>> - cc-with-dwz-m.exp >>> - cc-with-gdb-index.exp >>> - debug-types.exp >>> - dwarf4-gdb-index.exp >>> - fission-dwp.exp >>> - fission.exp >>> - native-extended-gdbserver.exp >>> - native-gdbserver.exp >>> - native-stdio-gdbserver.exp >>> - readnow.exp > > What are the advantages of doing this compared to a script that runs the > testsuite for each of the boards, separately? > I'd say: - the target boards not overwriting the .log and .sum files of other target boards - the joint summary. Thanks, - Tom >>> >>> When working on a single test, it's usually not to long to run them all. Of course, it's >>> not really practical to run the complete testsuite twice (before and after) for each board >>> for every change we do... >> >> Hi Simon, >> >> I just tried this out. >> >> My test scripts use make check, but I didn't manage to make that work >> yet, so I tried out using runtest directly (and I may be missing >> something obvious, given that I haven't used this before): >> ... >> $ cd build/gdb/testsuite >> $ runtest gdb.base/gold-gdb-index.exp --target_board='cc-with-gdb-index >> unix' >> ... > > One issue this with that I've run into in the past, is with global > variables leaking from one board to the other. > > And I've just tried a quick test, and lucky me I seem to have run into > something like that immediately: > > $ make check RUNTESTFLAGS="--target_board='native-gdbserver unix' break.exp" > ... > Schedule of variations: > native-gdbserver > unix > ... > FAIL: gdb.base/break.exp: run until function breakpoint > FAIL: gdb.base/break.exp: run until breakpoint set at a line number (the program is no longer running) > FAIL: gdb.base/break.exp: run until file:function(6) breakpoint (the program is no longer running) > FAIL: gdb.base/break.exp: run until file:function(5) breakpoint (the program is no longer running) > (snip a bunch more) > > (Running break.exp for each board in isolation passes cleanly, of course.) > > Another thing is that the test messages in gdb.sum don't indicate which > board issued the PASS/FAIL (on each result), making it more difficult > to analyze. > >> >> It seems however that the global settings (CC_FOR_TARGET etc) from the >> first board stay active in the unix board: > > Yeah, things like that. > > Also, the tests for all boards end up under the same testsuite/outputs/ directory, > so whatever the last board was, wins. I'm not even sure how the testsuite > caching works when you run more than one board in parallel mode. > > It would seem to me that if you're looking at running tests against different > boards, that you would want the output directories to be separate, so you can > analyze the results afterwards, avoid cache issues, etc. > > Maybe it would be better to come up with our own way to do this, without > relying on dejagnu directly. Much like we have the TESTS variable, > we could have a BOARDS variable: > > make check TESTS="gdb.base/break.exp" BOARDS="native-gdbserver unix" > > and then the outputs foreach board would be in > "testsuite/outputs.native-gdbserver/" and "testsuite/outputs.unix/" > respectively. > > Thanks, > Pedro Alves >