Re: [PATCH] gdb: fix an issue with thread list corruption

[Thiago Jung Bauermann <thiago.bauermann@linaro.org>]

"Metzger, Markus T" <markus.t.metzger@intel.com> writes:

> And what am I supposed to do about such sporadically failing tests when
> I want to test a patch series to ensure that I'm not introducing regressions?
>
> I created two empty patches and tested upstream master (of a few days ago)
> and my two empty patches, then diff'ed the corresponding gdb.sum files
> against the respective parent commit [1].
>
> The noise caused by sporadically failing tests makes it difficult to judge
> whether a given patch is OK.  Even if a patch would trigger a new fail in a
> test that sporadically passes, it would not be noticed in all that noise.

Ideally over time we'll fix the tests that fail sporadically for no good
reason.

In addition to that, there's suspicion of a kernel bug which causes
Expect to miss some bytes in the pty:

https://sourceware.org/bugzilla/show_bug.cgi?id=28561

A GCC engineer also noticed that sometimes GCC tests spuriously fail
because the output seen by Expect is missing a \n that by all accounts
should have been there (I don't have a bugzilla link for that one though).

I recently wrote¹ a suggestion about how to deal with the flaky tests
while these things aren't sorted out. I'll reproduce it here for
convenience:

The Linaro CI keeps an "expected failures" file containing the baseline
failures and automatically detected flaky failures. For example:

http://54.172.246.49:9090/jobs/tcwg_gdb_check--master-aarch64-build/builds/10102/archive/artifacts/sumfiles/xfails.xfail

(We're currently doing some server migration so the URL above is likely
 to change at some point. CI notification emails will have the current
 one.)

You can use it with GCC's
contrib/testsuite-management/validate_failures.py script to filter the
noise:

$ validate_failures.py --manifest=xfails.xfail --results=gdb.sum > failures.txt

Note that strictly speaking, the baseline failures are specific of the
machine the CI jobs run on. If you want to be precise you can replace
the "known failures" section of the file with the failures from your
testsuite run of the patch's base commit.

-- 
Thiago

¹ https://inbox.sourceware.org/gdb-patches/87zf2sc47o.fsf@linaro.org/