Re: GDB 7.9.90 available for testing

Re: GDB 7.9.90 available for testing

[David Edelsohn]

Why should GDB 7.10 be considered ready for release with the large
number of regressions shown by the GDB Buildbot?  The buildbot only
recently was created and shows numerous regressions on almost all
architectures relative to when the buildslave started running.

Thanks, David

Re: GDB 7.9.90 available for testing

[Joel Brobecker]

> Why should GDB 7.10 be considered ready for release with the large
> number of regressions shown by the GDB Buildbot?  The buildbot only
> recently was created and shows numerous regressions on almost all
> architectures relative to when the buildslave started running.

I don't follow the Buildbot day to day, unfortunately, and there are
many configurations; so I rely on everyone with regular emails on
this list to help me determine whether there might be issues blocking
for (1) cutting the branch, and (2) making the official release.
We keep the list on a wiki page that helps us track identified issues
for each release.  For instance, for 7.10, we have:
    https://sourceware.org/gdb/wiki/GDB_7.10_Release
As you can see, very little has been reported.

I am happy holding the creation of the official release up for
however long it would take to analyze the buildBot failures,
and to fix all blocking issues. Honestly, given the traffic seen
here derived from issues found by the buildBot, I thought that
people were already keeping an eye on the results.

-- 
Joel

Re: GDB 7.9.90 available for testing

[David Edelsohn]

On Thu, Jul 9, 2015 at 7:21 PM, Joel Brobecker <brobecker@adacore.com> wrote:
>> Why should GDB 7.10 be considered ready for release with the large
>> number of regressions shown by the GDB Buildbot?  The buildbot only
>> recently was created and shows numerous regressions on almost all
>> architectures relative to when the buildslave started running.
>
> I don't follow the Buildbot day to day, unfortunately, and there are
> many configurations; so I rely on everyone with regular emails on
> this list to help me determine whether there might be issues blocking
> for (1) cutting the branch, and (2) making the official release.
> We keep the list on a wiki page that helps us track identified issues
> for each release.  For instance, for 7.10, we have:
>     https://sourceware.org/gdb/wiki/GDB_7.10_Release
> As you can see, very little has been reported.
>
> I am happy holding the creation of the official release up for
> however long it would take to analyze the buildBot failures,
> and to fix all blocking issues. Honestly, given the traffic seen
> here derived from issues found by the buildBot, I thought that
> people were already keeping an eye on the results.

I'm not certain if the baselines truly are accurate for all
buildslaves, but it seems strange to create a release when the
buildbot testsuite results show patches causing new failures.

- David

Re: GDB 7.9.90 available for testing

[Joel Brobecker]

> I'm not certain if the baselines truly are accurate for all
> buildslaves, but it seems strange to create a release when the
> buildbot testsuite results show patches causing new failures.

To me, you are saying the same thing, and I don't disagree with you.
I said I didn't know that the buildBots were showing regressions.
Of course I would have held the creation of the branch if I had
known about this. But I didn't, and so here we are. Now we all know,
and the only way forward is to look at those regressions, and decide
what to do. We can and will delay the release if we have to.

Now, if you are implying that I didn't do enough as Release
Manager to prepare for this release, then that's an entirely
different discussion.

-- 
Joel

Re: GDB 7.9.90 available for testing

[David Edelsohn]

On Thu, Jul 9, 2015 at 11:42 PM, Joel Brobecker <brobecker@adacore.com> wrote:
>> I'm not certain if the baselines truly are accurate for all
>> buildslaves, but it seems strange to create a release when the
>> buildbot testsuite results show patches causing new failures.
>
> To me, you are saying the same thing, and I don't disagree with you.
> I said I didn't know that the buildBots were showing regressions.
> Of course I would have held the creation of the branch if I had
> known about this. But I didn't, and so here we are. Now we all know,
> and the only way forward is to look at those regressions, and decide
> what to do. We can and will delay the release if we have to.

Joel,

We are agreeing.  I was trying to provide some additional information
about interpretation of the buildbot status.

I am note two things about the buildbots:

1) Their color-coded "regression status" apparently is a comparison of
the testsuite between a "base" run and the current run. This is due to
few or no targets have completely clean testsuite runs to consider
"green".  Because there has been some adjustment and tweaking while
buildbots were added, the first run was not necessarily the ideal one
to choose as the "base" run, i.e., "regressions" may be due to changes
in the measurements after the first "base" run, not new failing tests.

2) Separate from the "regression" status, a quick inspection of some
testsuite output in the buildbots show the introduction of new errors
with recent commits.  Even if the overall regression status is not an
accurate measure of the state of GDB on those targets, the change in
regression status that is not monotonic reduction in regressions in
preparation for a release is disappointing.

I'm not demanding STOP SHIP.  GDB may not necessarily be in a bad
state for a release.  I don't know how this compares to the regression
status of previous releases.

I hope that GDB developers will become more aware of the effects of
their patches on multiple targets.

Thanks, David

Re: GDB 7.9.90 available for testing

[Pedro Alves]

On 07/10/2015 03:04 PM, David Edelsohn wrote:
> On Thu, Jul 9, 2015 at 11:42 PM, Joel Brobecker <brobecker@adacore.com> wrote:
>>> I'm not certain if the baselines truly are accurate for all
>>> buildslaves, but it seems strange to create a release when the
>>> buildbot testsuite results show patches causing new failures.
>>
>> To me, you are saying the same thing, and I don't disagree with you.
>> I said I didn't know that the buildBots were showing regressions.
>> Of course I would have held the creation of the branch if I had
>> known about this. But I didn't, and so here we are. Now we all know,
>> and the only way forward is to look at those regressions, and decide
>> what to do. We can and will delay the release if we have to.
> 
> Joel,
> 
> We are agreeing.  I was trying to provide some additional information
> about interpretation of the buildbot status.
> 
> I am note two things about the buildbots:
> 
> 1) Their color-coded "regression status" apparently is a comparison of
> the testsuite between a "base" run and the current run. This is due to
> few or no targets have completely clean testsuite runs to consider
> "green".  Because there has been some adjustment and tweaking while
> buildbots were added, the first run was not necessarily the ideal one
> to choose as the "base" run, i.e., "regressions" may be due to changes
> in the measurements after the first "base" run, not new failing tests.

There's no single "base" run, actually.  The baseline is dynamically
adjusted at each build; it's a moving baseline, and it's per
test (single PASS/FAIL, not file).  As soon as a test PASSes, it's
added to the baseline.  That means that if some test is racy, it'll
sometimes FAIL, and then a few builds later it'll PASS, at which point
the PASS is recorded in the baseline, and then a few builds again
later, the test FAIL again, and so the buildbot email report mentions
the regression against the baseline.  In sum, if a test goes
FAIL -> PASS -> FAIL -> PASS on and on over builds, you'll constantly
get reports of regressions against the baseline for that racy test.

For each build, you can find the baseline file in the corresponding
git commit pointed at in the email report.  E.g., see the "baseline"
file here:

  http://gdb-build.sergiodj.net/cgit/AIX-POWER7-plain/.git/tree/?h=master&id=42b08c842d422ae995d244efeb1a85aa8a082e7b

The gdb.thread/ FAILs you see on AIX seem to fall in that category.
From the results, it looks to me that those are caused by the AIX port
not implementing schedlock correctly.  Is anyone from IBM available
to look at these?

The gdb.cp/var-tag.exp FAILs currently reported on AIX are not really
regressions, but new FAILs.  And they are really a test problem, not
a GDB bug.  They actually depend on compiler or debug info format
used, not system.

> 
> 2) Separate from the "regression" status, a quick inspection of some
> testsuite output in the buildbots show the introduction of new errors
> with recent commits.  Even if the overall regression status is not an
> accurate measure of the state of GDB on those targets, the change in
> regression status that is not monotonic reduction in regressions in
> preparation for a release is disappointing.
> 
> I'm not demanding STOP SHIP.  GDB may not necessarily be in a bad
> state for a release.  I don't know how this compares to the regression
> status of previous releases.
> 
> I hope that GDB developers will become more aware of the effects of
> their patches on multiple targets.

This is just a misunderstanding.

Thanks,
Pedro Alves

Re: GDB 7.9.90 available for testing

[David Edelsohn]

On Fri, Jul 10, 2015 at 10:33 AM, Pedro Alves <palves@redhat.com> wrote:
> On 07/10/2015 03:04 PM, David Edelsohn wrote:
>> On Thu, Jul 9, 2015 at 11:42 PM, Joel Brobecker <brobecker@adacore.com> wrote:
>>>> I'm not certain if the baselines truly are accurate for all
>>>> buildslaves, but it seems strange to create a release when the
>>>> buildbot testsuite results show patches causing new failures.
>>>
>>> To me, you are saying the same thing, and I don't disagree with you.
>>> I said I didn't know that the buildBots were showing regressions.
>>> Of course I would have held the creation of the branch if I had
>>> known about this. But I didn't, and so here we are. Now we all know,
>>> and the only way forward is to look at those regressions, and decide
>>> what to do. We can and will delay the release if we have to.
>>
>> Joel,
>>
>> We are agreeing.  I was trying to provide some additional information
>> about interpretation of the buildbot status.
>>
>> I am note two things about the buildbots:
>>
>> 1) Their color-coded "regression status" apparently is a comparison of
>> the testsuite between a "base" run and the current run. This is due to
>> few or no targets have completely clean testsuite runs to consider
>> "green".  Because there has been some adjustment and tweaking while
>> buildbots were added, the first run was not necessarily the ideal one
>> to choose as the "base" run, i.e., "regressions" may be due to changes
>> in the measurements after the first "base" run, not new failing tests.
>
> There's no single "base" run, actually.  The baseline is dynamically
> adjusted at each build; it's a moving baseline, and it's per
> test (single PASS/FAIL, not file).  As soon as a test PASSes, it's
> added to the baseline.  That means that if some test is racy, it'll
> sometimes FAIL, and then a few builds later it'll PASS, at which point
> the PASS is recorded in the baseline, and then a few builds again
> later, the test FAIL again, and so the buildbot email report mentions
> the regression against the baseline.  In sum, if a test goes
> FAIL -> PASS -> FAIL -> PASS on and on over builds, you'll constantly
> get reports of regressions against the baseline for that racy test.

Thanks for the clarification.

>
> For each build, you can find the baseline file in the corresponding
> git commit pointed at in the email report.  E.g., see the "baseline"
> file here:
>
>   http://gdb-build.sergiodj.net/cgit/AIX-POWER7-plain/.git/tree/?h=master&id=42b08c842d422ae995d244efeb1a85aa8a082e7b
>
> The gdb.thread/ FAILs you see on AIX seem to fall in that category.
> From the results, it looks to me that those are caused by the AIX port
> not implementing schedlock correctly.  Is anyone from IBM available
> to look at these?
>
> The gdb.cp/var-tag.exp FAILs currently reported on AIX are not really
> regressions, but new FAILs.  And they are really a test problem, not
> a GDB bug.  They actually depend on compiler or debug info format
> used, not system.

My concern is more about GDB on Linux on z Systems and even GDB on
x86-64, not AIX.  AIX is weird.

Shouldn't the buildbots for z Series and x86-64 be green before a release?

Thanks, David

Re: GDB 7.9.90 available for testing

[Pedro Alves]

On 07/10/2015 03:56 PM, David Edelsohn wrote:

> My concern is more about GDB on Linux on z Systems and even GDB on
> x86-64, not AIX.  AIX is weird.
> 
> Shouldn't the buildbots for z Series and x86-64 be green before a release?

Ideally yes, but until the racy tests issue is fixed/kfailed and another
set of old known failures is kfail/xfailed, that can't happen.
I don't think any buildbot slave has ever been stably green yet; they
weren't green to start with.  It used to be much worse a few months ago,
we're getting there, but it requires effort, and we could use all
the help we can get.

The buildbots are quite useful, but we can't rely on greenness
alone to determine release-readyness at the moment.

Thanks,
Pedro Alves

Re: GDB 7.9.90 available for testing

[David Edelsohn]

On Fri, Jul 10, 2015 at 11:08 AM, Pedro Alves <palves@redhat.com> wrote:
> On 07/10/2015 03:56 PM, David Edelsohn wrote:
>
>> My concern is more about GDB on Linux on z Systems and even GDB on
>> x86-64, not AIX.  AIX is weird.
>>
>> Shouldn't the buildbots for z Series and x86-64 be green before a release?
>
> Ideally yes, but until the racy tests issue is fixed/kfailed and another
> set of old known failures is kfail/xfailed, that can't happen.
> I don't think any buildbot slave has ever been stably green yet; they
> weren't green to start with.  It used to be much worse a few months ago,
> we're getting there, but it requires effort, and we could use all
> the help we can get.
>
> The buildbots are quite useful, but we can't rely on greenness
> alone to determine release-readyness at the moment.

Yes, I was not requesting "green".  Because we do have the buildbots
available, I thought that it was important to understand the variable
failures or instability before a release.  If all of the failures on
primary platforms are due to race conditions, then they're understood.

Thanks, David

Re: GDB 7.9.90 available for testing

[Pedro Alves]

On 07/10/2015 04:25 PM, David Edelsohn wrote:

> Yes, I was not requesting "green".  Because we do have the buildbots
> available, I thought that it was important to understand the variable
> failures or instability before a release.  If all of the failures on
> primary platforms are due to race conditions, then they're understood.

Agreed, but what makes you think people aren't doing that?

We have a wiki page to track issues that need to be fixed before
the release, and I'm sure Joel will not release without first
asking the community if there are other issues people know
should be addressed.

Reminder: anyone, if there are issues that need fixing that are
not on the wiki page, please list them there so we don't forget:

 https://sourceware.org/gdb/wiki/GDB_7.10_Release

Thanks,
Pedro Alves

Re: GDB 7.9.90 available for testing

[Doug Evans]

On Fri, Jul 10, 2015 at 9:33 AM, Pedro Alves <palves@redhat.com> wrote:
> There's no single "base" run, actually.  The baseline is dynamically
> adjusted at each build; it's a moving baseline, and it's per
> test (single PASS/FAIL, not file).  As soon as a test PASSes, it's
> added to the baseline.  That means that if some test is racy, it'll
> sometimes FAIL, and then a few builds later it'll PASS, at which point
> the PASS is recorded in the baseline, and then a few builds again
> later, the test FAIL again, and so the buildbot email report mentions
> the regression against the baseline.  In sum, if a test goes
> FAIL -> PASS -> FAIL -> PASS on and on over builds, you'll constantly
> get reports of regressions against the baseline for that racy test.

Time for another plug to change how we manage racy tests?

E.g., if a test fails, run it again a few times.
I can think of various things to do after that.
E.g. if any of the additional runs of the test record a PASS then flag
the test as RACY, and remember this state for the next run, and rerun
the same test multiple times in the next run. If the next time all N
runs pass (or all N runs fail) then switch its state to PASS/FAIL.
That's not perfect, it's hard to be perfect with racy tests. One can
build on that, but there's a pragmatic tradeoff here between being too
complex and not doing anything at all.
I think we should do something. The above keeps the baseline
machine-generated and does minimal work to manage racy tests. A lot of
racy tests get exposed during these additional runs for me because I
don't run them in parallel and thus the system is under less load, and
it's system load that triggers a lot of the racyness.

The ultimate goal is of course to remove racy tests, but first we need
to be more systematic in identifying them, which is one of the goals
of this process.

racy tests

[Pedro Alves]

On 07/11/2015 02:58 PM, Doug Evans wrote:
> On Fri, Jul 10, 2015 at 9:33 AM, Pedro Alves <palves@redhat.com> wrote:
>> There's no single "base" run, actually.  The baseline is dynamically
>> adjusted at each build; it's a moving baseline, and it's per
>> test (single PASS/FAIL, not file).  As soon as a test PASSes, it's
>> added to the baseline.  That means that if some test is racy, it'll
>> sometimes FAIL, and then a few builds later it'll PASS, at which point
>> the PASS is recorded in the baseline, and then a few builds again
>> later, the test FAIL again, and so the buildbot email report mentions
>> the regression against the baseline.  In sum, if a test goes
>> FAIL -> PASS -> FAIL -> PASS on and on over builds, you'll constantly
>> get reports of regressions against the baseline for that racy test.
> 
> Time for another plug to change how we manage racy tests?

I'm all for something more structured.

> E.g., if a test fails, run it again a few times.

Agreed.

> I can think of various things to do after that.
> E.g. if any of the additional runs of the test record a PASS then flag
> the test as RACY, and remember this state for the next run, and rerun
> the same test multiple times in the next run. If the next time all N
> runs pass (or all N runs fail) then switch its state to PASS/FAIL.
> That's not perfect, it's hard to be perfect with racy tests. One can
> build on that, but there's a pragmatic tradeoff here between being too
> complex and not doing anything at all.
> I think we should do something. The above keeps the baseline
> machine-generated and does minimal work to manage racy tests. A lot of
> racy tests get exposed during these additional runs for me because I
> don't run them in parallel and thus the system is under less load, and
> it's system load that triggers a lot of the racyness.
> 

One thing that I'd like is for this to be part of the testsuite
itself, rather than separate machinery the buildbot uses.  That way,
everyone benefits from it, and so that we all maintain/evolve it.
I think this is important, because people are often confused that
they do a test run before patch, apply patch, run test, and see
confusing new FAILs their patch can't explain.

E.g., we could have the testsuite machinery itself run the tests
multiple times, iff they failed.  May all tests would be eligible for
this, or maybe we'd run apply this to those which are explicitly
marked racy somehow, but that's separate policy from the framework
that actually re-runs tests.  On a parallel test run, we run
each .exp under its own separate runtest invocation, driven from
the testsuite's Makefile; we could wrap each of those invocation and
check whether it failed, and if so, rerun that exp a few times.

That may mean that only parallel mode supports this, but I'd be
myself fine with that, because we can always do

  make check -j1 FORCE_PARALLEL="1"

or some convenience for that, to get the benefits.

Maybe it's possible to restart the same .exp test in a
sequential run too, from gdb_finish, say; I haven't thought much
about that.

> The ultimate goal is of course to remove racy tests, but first we need
> to be more systematic in identifying them, which is one of the goals
> of this process.

Agreed.

Thanks,
Pedro Alves

Re: racy tests

[Doug Evans]

On Sat, Jul 11, 2015 at 11:54 AM, Pedro Alves <palves@redhat.com> wrote:
> One thing that I'd like is for this to be part of the testsuite
> itself, rather than separate machinery the buildbot uses.  That way,
> everyone benefits from it, and so that we all maintain/evolve it.
> I think this is important, because people are often confused that
> they do a test run before patch, apply patch, run test, and see
> confusing new FAILs their patch can't explain.

No disagreement there.
I would build it on top of what's there now.
[I'd rather build this up in layers, and not have
overly complicated lower layers.]

A next question that arises is maintaining history.
E.g., how does one diff the results of the current run
with the current "gold standard"?

The way I do it here is to have separate files that augment the
XFAIL/KFAIL markers in the test (it's far easier to maintain a few
files than editing each test's .exp file)
but  I'm not sure it scales well.
[E.g., I need to keep separate files for different compilers,
though there is a #include mechanism for common stuff.]

Alternatively,
If a test run could take as input the gdb.sum file from a baseline
run (e.g., from an unpatched trunk) then that could work.
Buildbot could use the previous run, and Joe-Developer
could either use as input a buildbot run's output file
or run the testsuite twice (e.g., with/without the patch-under-test).
[I wouldn't use gdb.sum specifically, I'm just using it here
for illustration's sake.]

Re: racy tests

[Sergio Durigan Junior]

On Saturday, July 11 2015, Pedro Alves wrote:

>> I can think of various things to do after that.
>> E.g. if any of the additional runs of the test record a PASS then flag
>> the test as RACY, and remember this state for the next run, and rerun
>> the same test multiple times in the next run. If the next time all N
>> runs pass (or all N runs fail) then switch its state to PASS/FAIL.
>> That's not perfect, it's hard to be perfect with racy tests. One can
>> build on that, but there's a pragmatic tradeoff here between being too
>> complex and not doing anything at all.
>> I think we should do something. The above keeps the baseline
>> machine-generated and does minimal work to manage racy tests. A lot of
>> racy tests get exposed during these additional runs for me because I
>> don't run them in parallel and thus the system is under less load, and
>> it's system load that triggers a lot of the racyness.
>> 
>
> One thing that I'd like is for this to be part of the testsuite
> itself, rather than separate machinery the buildbot uses.  That way,
> everyone benefits from it, and so that we all maintain/evolve it.
> I think this is important, because people are often confused that
> they do a test run before patch, apply patch, run test, and see
> confusing new FAILs their patch can't explain.

I have something implemented for BuildBot that would address the issue
of racy tests, but I agree that having this as part of the official
testsuite is the best approach.  I'm willing to work on this, BTW.  I
will see what I can do this week during my spare time.

> E.g., we could have the testsuite machinery itself run the tests
> multiple times, iff they failed.

I would expand this to "have the testsuite machinery itself run the
tests multiple times".  Racy tests will also PASS in the first run,
which means that we'd miss some of them if the testsuite only re-ran
those that failed.  This would have side-effects, of course: the
testsuite run would take much longer, because we would be running all
tests several times...

BuildBot would then be able to use this to update its own xfail files.
I think running this "special test mode" once a week would be enough for
BuildBot to keep things up-to-dated.

> May all tests would be eligible for
> this, or maybe we'd run apply this to those which are explicitly
> marked racy somehow, but that's separate policy from the framework
> that actually re-runs tests.  On a parallel test run, we run
> each .exp under its own separate runtest invocation, driven from
> the testsuite's Makefile; we could wrap each of those invocation and
> check whether it failed, and if so, rerun that exp a few times.
>
> That may mean that only parallel mode supports this, but I'd be
> myself fine with that, because we can always do
>
>   make check -j1 FORCE_PARALLEL="1"
>
> or some convenience for that, to get the benefits.

I glanced over the testsuite's Makefile, and I thought about having an
explicit "FORCE_RACY_PARALLEL" (with its do-check-racy-parallel
counterpart).

This would be a "special mode", that would take longer to complete (as
explained above), but that would also generate real results.  I think
we'd have to modify dg-extract-results.sh as well; not sure.

-- 
Sergio
GPG key ID: 237A 54B1 0287 28BF 00EF  31F4 D0EB 7628 65FC 5E36
Please send encrypted e-mail if possible
http://sergiodj.net/