Re: [RFC/PATCH] Don't disable selftests in a non-development build

[Sergio Durigan Junior <sergiodj@redhat.com>]

On Tuesday, August 14 2018, Philippe Waroquiers wrote:

> On Tue, 2018-08-14 at 19:08 +0100, Pedro Alves wrote:
>> Another approach to addressing this issue here:
>> 
>> > Due to the many racy testcases and random failures we see when running
>> > the GDB testsuite, it is unfortunately not possible to perform a full
>> > test when one is building a downstream package.  As the Fedora GDB
>> > maintainer and one of the Debian GDB uploaders, I feel like this
>> > situation could be improved by, at least, executing our selftests
>> > after the package has been built.  However, we currently (for some
>> > reason that is not clear by reading the archives, but see more below)
>> > disable selftests on non-development builds.  Therefore, this patch
>> > aims to leave them enabled all the time, for everyone (including the
>> > end users).
>> 
>> ... is to come up with some small set of stable testcases that
>> are considered the "smoke tests" and add a mechanism to run them.
>> Could be just a list of testcases in a file that is passed to
>>   make check TESTS="list of basic tests here"
>> or some make target like "make check-smoke", or something
>> else even.
>
> Instead of (or in addition to) some list of tests that are known to be
> non-racy/alwayd ok,
> maybe it would be nice to have a list of tests that are known to
> be racy/sometimes wrong ?
> We e.g. could mark these racy tests as part of the test itself,
> with some pseudo dejagnu code like :
>
> if [do_not_run_racy_or_sometimes_wrong_tests] {
>   return
> }
>
> # or, if the test is only racy on some platform
> if [do_not_run_racy_or_sometimes_wrong_tests && istarget x86_64-*-*] {
>   return
> }

Yes, this is a good idea.  Pedro suggested something very similar (if
not equal) a while ago, when I was doing some work on detecting racy
tests for our BuildBot.  The idea is to mark the known racy tests using
something a la "setup_kfail".  I haven't gotten the time to work on this
yet, but it's on my TODO list.

> The above would also help the gdb developers to interpret
> the results of a test run.
> I am always wondering if a failure I see can (or cannot) be
> explained by the change I just did.

It's a PITA, indeed.  If it helps, the BuildBot automatically refreshes
the list of known racy tests every week (or every day, for x86*
builders).  You can have a look at the lists here:

  https://git.sergiodj.net/gdb-xfails.git/tree/xfails

So, for a list of known racy tests for Fedora-x86-64-m64:

  https://git.sergiodj.net/gdb-xfails.git/tree/xfails/Fedora-x86_64-m64/xfails/master/xfail.table

Thanks,

-- 
Sergio
GPG key ID: 237A 54B1 0287 28BF 00EF  31F4 D0EB 7628 65FC 5E36
Please send encrypted e-mail if possible
http://sergiodj.net/