Re: GDB 7.9.90 available for testing

Pedro Alves
2015-07-10T14:33:47Z

On 07/10/2015 03:04 PM, David Edelsohn wrote:
> On Thu, Jul 9, 2015 at 11:42 PM, Joel Brobecker <brobecker@adacore.com> wrote:
>>> I'm not certain if the baselines truly are accurate for all
>>> buildslaves, but it seems strange to create a release when the
>>> buildbot testsuite results show patches causing new failures.
>>
>> To me, you are saying the same thing, and I don't disagree with you.
>> I said I didn't know that the buildBots were showing regressions.
>> Of course I would have held the creation of the branch if I had
>> known about this. But I didn't, and so here we are. Now we all know,
>> and the only way forward is to look at those regressions, and decide
>> what to do. We can and will delay the release if we have to.
>
> Joel,
>
> We are agreeing. I was trying to provide some additional information
> about interpretation of the buildbot status.
>
> I am note two things about the buildbots:
>
> 1) Their color-coded "regression status" apparently is a comparison of
> the testsuite between a "base" run and the current run. This is due to
> few or no targets have completely clean testsuite runs to consider
> "green". Because there has been some adjustment and tweaking while
> buildbots were added, the first run was not necessarily the ideal one
> to choose as the "base" run, i.e., "regressions" may be due to changes
> in the measurements after the first "base" run, not new failing tests.

There's no single "base" run, actually. The baseline is dynamically
adjusted at each build; it's a moving baseline, and it's per
test (single PASS/FAIL, not file). As soon as a test PASSes, it's
added to the baseline. That means that if some test is racy, it'll
sometimes FAIL, and then a few builds later it'll PASS, at which point
the PASS is recorded in the baseline, and then a few builds again
later, the test FAIL again, and so the buildbot email report mentions
the regression against the baseline. In sum, if a test goes
FAIL -> PASS -> FAIL -> PASS on and on over builds, you'll constantly
get reports of regressions against the baseline for that racy test.

For each build, you can find the baseline file in the corresponding
git commit pointed at in the email report. E.g., see the "baseline"
file here:

http://gdb-build.sergiodj.net/cgit/AIX-POWER7-plain/.git/tree/?h=master&id=42b08c842d422ae995d244efeb1a85aa8a082e7b

The gdb.thread/ FAILs you see on AIX seem to fall in that category.
From the results, it looks to me that those are caused by the AIX port
not implementing schedlock correctly. Is anyone from IBM available
to look at these?

The gdb.cp/var-tag.exp FAILs currently reported on AIX are not really
regressions, but new FAILs. And they are really a test problem, not
a GDB bug. They actually depend on compiler or debug info format
used, not system.

>
> 2) Separate from the "regression" status, a quick inspection of some
> testsuite output in the buildbots show the introduction of new errors
> with recent commits. Even if the overall regression status is not an
> accurate measure of the state of GDB on those targets, the change in
> regression status that is not monotonic reduction in regressions in
> preparation for a release is disappointing.
>
> I'm not demanding STOP SHIP. GDB may not necessarily be in a bad
> state for a release. I don't know how this compares to the regression
> status of previous releases.
>
> I hope that GDB developers will become more aware of the effects of
> their patches on multiple targets.

This is just a misunderstanding.

Thanks,
Pedro Alves