Re: [PATCH] Improve analysis of racy testcases

[Antoine Tremblay <antoine.tremblay@ericsson.com>]

Sergio Durigan Junior writes:

> On Thursday, February 25 2016, Antoine Tremblay wrote:
>
>> Sergio Durigan Junior writes:
>>
>>> This patch is a proposal to introduce some mechanisms to identify racy
>>> testcases present in our testsuite.  As can be seen in previous
>>> discussions, racy tests are really bothersome and cause our BuildBot to
>>> pollute the gdb-testers mailing list with hundreds of false-positives
>>> messages every month.  Hopefully, by identifying these racy tests in
>>> advance (and automatically) will contribute to the reduction of noise
>>> traffic to gdb-testers, maybe to the point where we will be able to send
>>> the failure messages directly to the authors of the commits.
>>> [...]
>> Thanks for this ! This was quite a problem for me while testing on arm.
>> I'm testing it now...
>
> Thanks!  Please let me know if you find anything wrong with the script.

Just to let know you that the script worked as expected during my
testing and it's quite nice to see. The tests however were more flacky
then expected so I can't use the output directly guess I'll need a never
ignore this test list... (like base/break.exp). Still it's very helpful.

>
>> One note maybe it would be nice output the list of unracy tests too to
>> be able to auto-build a list of tests to run out of this since I'm not
>> sure you can set an exclusion list ?
>
> Hm, it would be possible to output the non-racy tests, but only to a
> different file (instead of outputting to stdout, as I'm doing with the
> racy tests).  Perhaps this could be a separate option to the script?
> I'm not sure if the users would always want this information...
>
> As for the exclusion list you mentioned, DejaGNU's runtest allows one to
> specify a --ignore flag with the names of the tests you don't want to
> run.  It should be possible to pass this via RUNTESTFLAGS, but I haven't
> tried.  I'll give it a go tomorrow.

Great, I did not know that, I'll give it a try and that works no need to
output the non-racy tests...

Thanks,
Antoine