GDB 7.9.90 available for testing

GDB 7.9.90 available for testing

[Joel Brobecker]

Hello,

I have just finished creating the gdb-7.9.90 pre-release.
It is available for download at the following location:

    ftp://sourceware.org/pub/gdb/snapshots/branch/gdb-7.9.90.tar.xz

A gzip'ed version is also available: gdb-7.9.90.tar.gz.

Please give it a test if you can and report any problems you might find.

On behalf of all the GDB contributors, thank you!
-- 
Joel Brobecker

Re: GDB 7.9.90 available for testing

[David Edelsohn]

Why should GDB 7.10 be considered ready for release with the large
number of regressions shown by the GDB Buildbot?  The buildbot only
recently was created and shows numerous regressions on almost all
architectures relative to when the buildslave started running.

Thanks, David

Re: GDB 7.9.90 available for testing

[Joel Brobecker]

> Why should GDB 7.10 be considered ready for release with the large
> number of regressions shown by the GDB Buildbot?  The buildbot only
> recently was created and shows numerous regressions on almost all
> architectures relative to when the buildslave started running.

I don't follow the Buildbot day to day, unfortunately, and there are
many configurations; so I rely on everyone with regular emails on
this list to help me determine whether there might be issues blocking
for (1) cutting the branch, and (2) making the official release.
We keep the list on a wiki page that helps us track identified issues
for each release.  For instance, for 7.10, we have:
    https://sourceware.org/gdb/wiki/GDB_7.10_Release
As you can see, very little has been reported.

I am happy holding the creation of the official release up for
however long it would take to analyze the buildBot failures,
and to fix all blocking issues. Honestly, given the traffic seen
here derived from issues found by the buildBot, I thought that
people were already keeping an eye on the results.

-- 
Joel

Re: GDB 7.9.90 available for testing

[David Edelsohn]

On Thu, Jul 9, 2015 at 7:21 PM, Joel Brobecker <brobecker@adacore.com> wrote:
>> Why should GDB 7.10 be considered ready for release with the large
>> number of regressions shown by the GDB Buildbot?  The buildbot only
>> recently was created and shows numerous regressions on almost all
>> architectures relative to when the buildslave started running.
>
> I don't follow the Buildbot day to day, unfortunately, and there are
> many configurations; so I rely on everyone with regular emails on
> this list to help me determine whether there might be issues blocking
> for (1) cutting the branch, and (2) making the official release.
> We keep the list on a wiki page that helps us track identified issues
> for each release.  For instance, for 7.10, we have:
>     https://sourceware.org/gdb/wiki/GDB_7.10_Release
> As you can see, very little has been reported.
>
> I am happy holding the creation of the official release up for
> however long it would take to analyze the buildBot failures,
> and to fix all blocking issues. Honestly, given the traffic seen
> here derived from issues found by the buildBot, I thought that
> people were already keeping an eye on the results.

I'm not certain if the baselines truly are accurate for all
buildslaves, but it seems strange to create a release when the
buildbot testsuite results show patches causing new failures.

- David

Re: GDB 7.9.90 available for testing

[Joel Brobecker]

> I'm not certain if the baselines truly are accurate for all
> buildslaves, but it seems strange to create a release when the
> buildbot testsuite results show patches causing new failures.

To me, you are saying the same thing, and I don't disagree with you.
I said I didn't know that the buildBots were showing regressions.
Of course I would have held the creation of the branch if I had
known about this. But I didn't, and so here we are. Now we all know,
and the only way forward is to look at those regressions, and decide
what to do. We can and will delay the release if we have to.

Now, if you are implying that I didn't do enough as Release
Manager to prepare for this release, then that's an entirely
different discussion.

-- 
Joel

Re: GDB 7.9.90 available for testing

[David Edelsohn]

On Thu, Jul 9, 2015 at 11:42 PM, Joel Brobecker <brobecker@adacore.com> wrote:
>> I'm not certain if the baselines truly are accurate for all
>> buildslaves, but it seems strange to create a release when the
>> buildbot testsuite results show patches causing new failures.
>
> To me, you are saying the same thing, and I don't disagree with you.
> I said I didn't know that the buildBots were showing regressions.
> Of course I would have held the creation of the branch if I had
> known about this. But I didn't, and so here we are. Now we all know,
> and the only way forward is to look at those regressions, and decide
> what to do. We can and will delay the release if we have to.

Joel,

We are agreeing.  I was trying to provide some additional information
about interpretation of the buildbot status.

I am note two things about the buildbots:

1) Their color-coded "regression status" apparently is a comparison of
the testsuite between a "base" run and the current run. This is due to
few or no targets have completely clean testsuite runs to consider
"green".  Because there has been some adjustment and tweaking while
buildbots were added, the first run was not necessarily the ideal one
to choose as the "base" run, i.e., "regressions" may be due to changes
in the measurements after the first "base" run, not new failing tests.

2) Separate from the "regression" status, a quick inspection of some
testsuite output in the buildbots show the introduction of new errors
with recent commits.  Even if the overall regression status is not an
accurate measure of the state of GDB on those targets, the change in
regression status that is not monotonic reduction in regressions in
preparation for a release is disappointing.

I'm not demanding STOP SHIP.  GDB may not necessarily be in a bad
state for a release.  I don't know how this compares to the regression
status of previous releases.

I hope that GDB developers will become more aware of the effects of
their patches on multiple targets.

Thanks, David

Re: GDB 7.9.90 available for testing

[Pedro Alves]

On 07/10/2015 03:04 PM, David Edelsohn wrote:
> On Thu, Jul 9, 2015 at 11:42 PM, Joel Brobecker <brobecker@adacore.com> wrote:
>>> I'm not certain if the baselines truly are accurate for all
>>> buildslaves, but it seems strange to create a release when the
>>> buildbot testsuite results show patches causing new failures.
>>
>> To me, you are saying the same thing, and I don't disagree with you.
>> I said I didn't know that the buildBots were showing regressions.
>> Of course I would have held the creation of the branch if I had
>> known about this. But I didn't, and so here we are. Now we all know,
>> and the only way forward is to look at those regressions, and decide
>> what to do. We can and will delay the release if we have to.
> 
> Joel,
> 
> We are agreeing.  I was trying to provide some additional information
> about interpretation of the buildbot status.
> 
> I am note two things about the buildbots:
> 
> 1) Their color-coded "regression status" apparently is a comparison of
> the testsuite between a "base" run and the current run. This is due to
> few or no targets have completely clean testsuite runs to consider
> "green".  Because there has been some adjustment and tweaking while
> buildbots were added, the first run was not necessarily the ideal one
> to choose as the "base" run, i.e., "regressions" may be due to changes
> in the measurements after the first "base" run, not new failing tests.

There's no single "base" run, actually.  The baseline is dynamically
adjusted at each build; it's a moving baseline, and it's per
test (single PASS/FAIL, not file).  As soon as a test PASSes, it's
added to the baseline.  That means that if some test is racy, it'll
sometimes FAIL, and then a few builds later it'll PASS, at which point
the PASS is recorded in the baseline, and then a few builds again
later, the test FAIL again, and so the buildbot email report mentions
the regression against the baseline.  In sum, if a test goes
FAIL -> PASS -> FAIL -> PASS on and on over builds, you'll constantly
get reports of regressions against the baseline for that racy test.

For each build, you can find the baseline file in the corresponding
git commit pointed at in the email report.  E.g., see the "baseline"
file here:

  http://gdb-build.sergiodj.net/cgit/AIX-POWER7-plain/.git/tree/?h=master&id=42b08c842d422ae995d244efeb1a85aa8a082e7b

The gdb.thread/ FAILs you see on AIX seem to fall in that category.
From the results, it looks to me that those are caused by the AIX port
not implementing schedlock correctly.  Is anyone from IBM available
to look at these?

The gdb.cp/var-tag.exp FAILs currently reported on AIX are not really
regressions, but new FAILs.  And they are really a test problem, not
a GDB bug.  They actually depend on compiler or debug info format
used, not system.

> 
> 2) Separate from the "regression" status, a quick inspection of some
> testsuite output in the buildbots show the introduction of new errors
> with recent commits.  Even if the overall regression status is not an
> accurate measure of the state of GDB on those targets, the change in
> regression status that is not monotonic reduction in regressions in
> preparation for a release is disappointing.
> 
> I'm not demanding STOP SHIP.  GDB may not necessarily be in a bad
> state for a release.  I don't know how this compares to the regression
> status of previous releases.
> 
> I hope that GDB developers will become more aware of the effects of
> their patches on multiple targets.

This is just a misunderstanding.

Thanks,
Pedro Alves

Re: GDB 7.9.90 available for testing

[David Edelsohn]

On Fri, Jul 10, 2015 at 10:33 AM, Pedro Alves <palves@redhat.com> wrote:
> On 07/10/2015 03:04 PM, David Edelsohn wrote:
>> On Thu, Jul 9, 2015 at 11:42 PM, Joel Brobecker <brobecker@adacore.com> wrote:
>>>> I'm not certain if the baselines truly are accurate for all
>>>> buildslaves, but it seems strange to create a release when the
>>>> buildbot testsuite results show patches causing new failures.
>>>
>>> To me, you are saying the same thing, and I don't disagree with you.
>>> I said I didn't know that the buildBots were showing regressions.
>>> Of course I would have held the creation of the branch if I had
>>> known about this. But I didn't, and so here we are. Now we all know,
>>> and the only way forward is to look at those regressions, and decide
>>> what to do. We can and will delay the release if we have to.
>>
>> Joel,
>>
>> We are agreeing.  I was trying to provide some additional information
>> about interpretation of the buildbot status.
>>
>> I am note two things about the buildbots:
>>
>> 1) Their color-coded "regression status" apparently is a comparison of
>> the testsuite between a "base" run and the current run. This is due to
>> few or no targets have completely clean testsuite runs to consider
>> "green".  Because there has been some adjustment and tweaking while
>> buildbots were added, the first run was not necessarily the ideal one
>> to choose as the "base" run, i.e., "regressions" may be due to changes
>> in the measurements after the first "base" run, not new failing tests.
>
> There's no single "base" run, actually.  The baseline is dynamically
> adjusted at each build; it's a moving baseline, and it's per
> test (single PASS/FAIL, not file).  As soon as a test PASSes, it's
> added to the baseline.  That means that if some test is racy, it'll
> sometimes FAIL, and then a few builds later it'll PASS, at which point
> the PASS is recorded in the baseline, and then a few builds again
> later, the test FAIL again, and so the buildbot email report mentions
> the regression against the baseline.  In sum, if a test goes
> FAIL -> PASS -> FAIL -> PASS on and on over builds, you'll constantly
> get reports of regressions against the baseline for that racy test.

Thanks for the clarification.

>
> For each build, you can find the baseline file in the corresponding
> git commit pointed at in the email report.  E.g., see the "baseline"
> file here:
>
>   http://gdb-build.sergiodj.net/cgit/AIX-POWER7-plain/.git/tree/?h=master&id=42b08c842d422ae995d244efeb1a85aa8a082e7b
>
> The gdb.thread/ FAILs you see on AIX seem to fall in that category.
> From the results, it looks to me that those are caused by the AIX port
> not implementing schedlock correctly.  Is anyone from IBM available
> to look at these?
>
> The gdb.cp/var-tag.exp FAILs currently reported on AIX are not really
> regressions, but new FAILs.  And they are really a test problem, not
> a GDB bug.  They actually depend on compiler or debug info format
> used, not system.

My concern is more about GDB on Linux on z Systems and even GDB on
x86-64, not AIX.  AIX is weird.

Shouldn't the buildbots for z Series and x86-64 be green before a release?

Thanks, David

Re: GDB 7.9.90 available for testing

[Pedro Alves]

On 07/10/2015 03:56 PM, David Edelsohn wrote:

> My concern is more about GDB on Linux on z Systems and even GDB on
> x86-64, not AIX.  AIX is weird.
> 
> Shouldn't the buildbots for z Series and x86-64 be green before a release?

Ideally yes, but until the racy tests issue is fixed/kfailed and another
set of old known failures is kfail/xfailed, that can't happen.
I don't think any buildbot slave has ever been stably green yet; they
weren't green to start with.  It used to be much worse a few months ago,
we're getting there, but it requires effort, and we could use all
the help we can get.

The buildbots are quite useful, but we can't rely on greenness
alone to determine release-readyness at the moment.

Thanks,
Pedro Alves

Re: GDB 7.9.90 available for testing

[David Edelsohn]

On Fri, Jul 10, 2015 at 11:08 AM, Pedro Alves <palves@redhat.com> wrote:
> On 07/10/2015 03:56 PM, David Edelsohn wrote:
>
>> My concern is more about GDB on Linux on z Systems and even GDB on
>> x86-64, not AIX.  AIX is weird.
>>
>> Shouldn't the buildbots for z Series and x86-64 be green before a release?
>
> Ideally yes, but until the racy tests issue is fixed/kfailed and another
> set of old known failures is kfail/xfailed, that can't happen.
> I don't think any buildbot slave has ever been stably green yet; they
> weren't green to start with.  It used to be much worse a few months ago,
> we're getting there, but it requires effort, and we could use all
> the help we can get.
>
> The buildbots are quite useful, but we can't rely on greenness
> alone to determine release-readyness at the moment.

Yes, I was not requesting "green".  Because we do have the buildbots
available, I thought that it was important to understand the variable
failures or instability before a release.  If all of the failures on
primary platforms are due to race conditions, then they're understood.

Thanks, David

Re: GDB 7.9.90 available for testing

[Pedro Alves]

On 07/10/2015 04:25 PM, David Edelsohn wrote:

> Yes, I was not requesting "green".  Because we do have the buildbots
> available, I thought that it was important to understand the variable
> failures or instability before a release.  If all of the failures on
> primary platforms are due to race conditions, then they're understood.

Agreed, but what makes you think people aren't doing that?

We have a wiki page to track issues that need to be fixed before
the release, and I'm sure Joel will not release without first
asking the community if there are other issues people know
should be addressed.

Reminder: anyone, if there are issues that need fixing that are
not on the wiki page, please list them there so we don't forget:

 https://sourceware.org/gdb/wiki/GDB_7.10_Release

Thanks,
Pedro Alves

Re: GDB 7.9.90 available for testing

[Doug Evans]

On Fri, Jul 10, 2015 at 9:33 AM, Pedro Alves <palves@redhat.com> wrote:
> There's no single "base" run, actually.  The baseline is dynamically
> adjusted at each build; it's a moving baseline, and it's per
> test (single PASS/FAIL, not file).  As soon as a test PASSes, it's
> added to the baseline.  That means that if some test is racy, it'll
> sometimes FAIL, and then a few builds later it'll PASS, at which point
> the PASS is recorded in the baseline, and then a few builds again
> later, the test FAIL again, and so the buildbot email report mentions
> the regression against the baseline.  In sum, if a test goes
> FAIL -> PASS -> FAIL -> PASS on and on over builds, you'll constantly
> get reports of regressions against the baseline for that racy test.

Time for another plug to change how we manage racy tests?

E.g., if a test fails, run it again a few times.
I can think of various things to do after that.
E.g. if any of the additional runs of the test record a PASS then flag
the test as RACY, and remember this state for the next run, and rerun
the same test multiple times in the next run. If the next time all N
runs pass (or all N runs fail) then switch its state to PASS/FAIL.
That's not perfect, it's hard to be perfect with racy tests. One can
build on that, but there's a pragmatic tradeoff here between being too
complex and not doing anything at all.
I think we should do something. The above keeps the baseline
machine-generated and does minimal work to manage racy tests. A lot of
racy tests get exposed during these additional runs for me because I
don't run them in parallel and thus the system is under less load, and
it's system load that triggers a lot of the racyness.

The ultimate goal is of course to remove racy tests, but first we need
to be more systematic in identifying them, which is one of the goals
of this process.