Re: [RFC/PATCH] Don't disable selftests in a non-development build

[Philippe Waroquiers <philippe.waroquiers@skynet.be>]

On Tue, 2018-08-14 at 19:08 +0100, Pedro Alves wrote:
> Another approach to addressing this issue here:
> 
> > Due to the many racy testcases and random failures we see when running
> > the GDB testsuite, it is unfortunately not possible to perform a full
> > test when one is building a downstream package.  As the Fedora GDB
> > maintainer and one of the Debian GDB uploaders, I feel like this
> > situation could be improved by, at least, executing our selftests
> > after the package has been built.  However, we currently (for some
> > reason that is not clear by reading the archives, but see more below)
> > disable selftests on non-development builds.  Therefore, this patch
> > aims to leave them enabled all the time, for everyone (including the
> > end users).
> 
> ... is to come up with some small set of stable testcases that
> are considered the "smoke tests" and add a mechanism to run them.
> Could be just a list of testcases in a file that is passed to
>   make check TESTS="list of basic tests here"
> or some make target like "make check-smoke", or something
> else even.

Instead of (or in addition to) some list of tests that are known to be
non-racy/alwayd ok,
maybe it would be nice to have a list of tests that are known to
be racy/sometimes wrong ?
We e.g. could mark these racy tests as part of the test itself,
with some pseudo dejagnu code like :

if [do_not_run_racy_or_sometimes_wrong_tests] {
  return
}

# or, if the test is only racy on some platform
if [do_not_run_racy_or_sometimes_wrong_tests && istarget x86_64-*-*] {
  return
}

The above would also help the gdb developers to interpret
the results of a test run.
I am always wondering if a failure I see can (or cannot) be
explained by the change I just did.

Philippe