Re: gcc development schedule [Re: sharing libcpp between GDB and GCC]

Re: gcc development schedule [Re: sharing libcpp between GDB and GCC]

[Kaveh R. Ghazi]

 > From: Zack Weinberg <zack@codesourcery.com> 
 > 
 > I'll add that this is an argument for a better version control
 > system; most of the changes on the queue inside my head, I would
 > have done on a branch if CVS didn't make branches such a pain.
 > (E.g. it takes about 10x longer to do "cvs update" on the 3.0
 > branch than the trunk.)
 > zw

Yeah, what's up with that?  (I thought it was just me.)

--
Kaveh R. Ghazi			Director of Systems Architecture
ghazi@caip.rutgers.edu		Qwest Global Services

Re: gcc development schedule [Re: sharing libcpp between GDB and GCC]

[Zack Weinberg]

On Wed, Mar 27, 2002 at 10:17:03PM -0500, Kaveh R. Ghazi wrote:
>  > (E.g. it takes about 10x longer to do "cvs update" on the 3.0
>  > branch than the trunk.)
> 
> Yeah, what's up with that?  (I thought it was just me.)

The way RCS stores branches makes the cost of calculating diffs from a
branch tip proportional to the number of versions on the branch *and*
the distance between the branchpoint and top-of-trunk.  I imagine
update has to do diffs for some reason.

That's the problem I know about; there may be others.

zw

Re: gcc development schedule [Re: sharing libcpp between GDB and GCC]

[Gerald Pfeifer]

[ moved to overseers ]

On Wed, 27 Mar 2002, Zack Weinberg wrote:
>>> (E.g. it takes about 10x longer to do "cvs update" on the 3.0
>>> branch than the trunk.)
>> Yeah, what's up with that?  (I thought it was just me.)

Same here.

On the gcc-3.1 branch, I'm approaching the situation where cvs update
takes longer than bootstrapping the compiler (with C and C++ frontends).

> The way RCS stores branches makes the cost of calculating diffs from a
> branch tip proportional to the number of versions on the branch *and*
> the distance between the branchpoint and top-of-trunk.  I imagine
> update has to do diffs for some reason.
>
> That's the problem I know about; there may be others.

Overseers, is there anything we can do?

I noticed that we have more anoncvs processes running than authenticated
users; is there any way we could find out which CVS modules these
anonymous users currently access? (If it's gcc, we could see whether
we can simply disable anoncvs access to the gcc module.)

Gerald
-- 
Gerald "Jerry" pfeifer@dbai.tuwien.ac.at http://www.dbai.tuwien.ac.at/~pfeifer/

Re: gcc development schedule [Re: sharing libcpp between GDB and GCC]

[Jason Molenda]

On Thu, Mar 28, 2002 at 10:22:49AM +0100, Gerald Pfeifer wrote:

> On Wed, 27 Mar 2002, Zack Weinberg wrote:
> >>> (E.g. it takes about 10x longer to do "cvs update" on the 3.0
> >>> branch than the trunk.)
> >> Yeah, what's up with that?  (I thought it was just me.)

The cvs server on sourceware uses an optimization (a patch written
by Ian Lance Taylor) to cache the information needed for a cvs
update in a single file per directory.  These are the CVS/fileattr
files--you know, the ones that get out of date once every few months
and need to be blown away.  One of them in the gcc dir looks like

F.cvsignore     _head=1.9;_expand=
F.gdbinit       _head=1.6;_attic=;_expand=
FABOUT-GCC-NLS  _head=1.5;_expand=
FABOUT-NLS      _head=1.3;_expand=
FCOPYING        _head=1.4;_expand=
FCOPYING.LIB    _head=1.4;_expand=
FChangeLog      _head=1.13542;_expand=
FChangeLog.0    _head=1.12;_expand=
FChangeLog.1    _head=1.6;_expand=

So the cvs server doesn't have to open every RCS file in the
directory to find its head revision.  A do-nothing cvs update goes
much, much faster.

For the cvs server to update on a branch, it has to read each RCS
file to find the latest version on that branch.  Even worse, it
looks like it'll  need to read several blocks into each file for
frequently modified files.  CVS shouldn't have to compute any diffs
for most 'cvs update' operations - chances are that only a handful
of files were modified on the branch so only a few of these expensive
diffs are computed.  It's the get-the-head-revision operation that
is killing it.


I did make a neato little mechanism for Apple's benefit a few months
back.  It logs every cvs commit as it happens to a plain text file,
and provides a web interface to get the list of files that were
modified since the last time the client contacted the server.  (the
scripts on user's systems would just use wget with a pre-set URL
to get this list of files to update.)  It was a neato little thing,
but for trunk updates I found that doing a cvs update for the gcc
repository consistently took less than two minutes, so it wasn't
really all that useful.  I could revisit it and finish it up if
people would use it - it would work particularly well for things
like a branch where a global cvs update becomes more expensive.

> > That's the problem I know about; there may be others.
> 
> Overseers, is there anything we can do?

We could always disable the trunk cvs update optimization.  The
cvs updates on branches would still be slower than trunk cvs updates,
but the difference wouldn't be so stark.

OK, kidding aside, Ian's patch could be modified to keep the head
revisions of branches in the fileattr cache file in addition to
the trunk.  But someone would have to do that.  Goodness knows I
don't have the time.

The sourceware system is under a bit of memory pressure these days,
which doesn't help; if these blocks could be cached for a longer
period of time, this RCS file reading/parsing will go faster.

(incidentally, Ian's patch is one of the reason we're still running
cvs 1.10.  Stock cvs performance regressed between 1.11 and 1.10,
and no one has gotten Ian's patch to work fully with 1.11, so moving
to 1.11 from 1.10+Ian's patch would be a big lose.  To be fair, except
for one weekend that Chris Faylor spent on Ian's patch, no one has
really put any effort into making it work with 1.11 fully.)

> I noticed that we have more anoncvs processes running than authenticated
> users; is there any way we could find out which CVS modules these
> anonymous users currently access? (If it's gcc, we could see whether
> we can simply disable anoncvs access to the gcc module.)

None that I know of.  cvs logging bites.  We do track the # of
bytes being sent/received from different hosts and the frequency
with which hosts connect.  The host that downloaded the most number
of bytes last week by anoncvs?  A purdue.edu site.  Two redhat.com
and one suse.de sites make the top six; an IP# and a cable modem
(rogers.com) finish out the top six.  Not very useful without some
idea what repository or what sorts of operations we're talking
about.


I haven't followed the discussion that led up to this note, so I
apologize if I'm repeating things already said.

Jason

Re: gcc development schedule [Re: sharing libcpp between GDB and GCC]

[Jason Molenda]

On Thu, Mar 28, 2002 at 01:53:46AM -0800, Jason Molenda wrote:

> None that I know of.  cvs logging bites.  We do track the # of
> bytes being sent/received from different hosts and the frequency
> with which hosts connect.  The host that downloaded the most number
> of bytes last week by anoncvs?  A purdue.edu site.  Two redhat.com
> and one suse.de sites make the top six; an IP# and a cable modem
> (rogers.com) finish out the top six.  Not very useful without some
> idea what repository or what sorts of operations we're talking
> about.

Ahem, this is how it looks when sorted by connections, but it's
rather different when sorted by bytes downloaded.  For last week,
it looks more like this:

698 358 959  cwi.nl
523 131 718  cpe.net.cable.rogers.com
316 859 474  dsl.snfc21.pacbell.net
302 256 065  sympatico.ca
220 184 438  austin.ibm.com
208 933 189  redhat.com
175 256 836  mypointsinc.com
144 845 712  dip.t-dialin.net

The first field is bytes transferred over the week; the second is
the domains for these sites.

It's easy to think these sites are doing something Evil, but chances
are they're just doing some automated cvs updates on multiple
repositories once or twice a day, or they had a single big checkout
of gcc/src and a periodic cvs update or what have you.  Don't read
too much into it without looking closer - I'm just providing an
example of what we do track and what we can report.

Jason

Re: gcc development schedule [Re: sharing libcpp between GDB and GCC]

[Christopher Faylor]

On Thu, Mar 28, 2002 at 02:01:02AM -0800, Jason Molenda wrote:
>On Thu, Mar 28, 2002 at 01:53:46AM -0800, Jason Molenda wrote:
>
>> None that I know of.  cvs logging bites.  We do track the # of
>> bytes being sent/received from different hosts and the frequency
>> with which hosts connect.  The host that downloaded the most number
>> of bytes last week by anoncvs?  A purdue.edu site.  Two redhat.com
>> and one suse.de sites make the top six; an IP# and a cable modem
>> (rogers.com) finish out the top six.  Not very useful without some
>> idea what repository or what sorts of operations we're talking
>> about.
>
>Ahem, this is how it looks when sorted by connections, but it's
>rather different when sorted by bytes downloaded.  For last week,
>it looks more like this:
>
>698 358 959  cwi.nl
>523 131 718  cpe.net.cable.rogers.com
>316 859 474  dsl.snfc21.pacbell.net
>302 256 065  sympatico.ca
>220 184 438  austin.ibm.com
>208 933 189  redhat.com
>175 256 836  mypointsinc.com
>144 845 712  dip.t-dialin.net
>
>The first field is bytes transferred over the week; the second is
>the domains for these sites.
>
>It's easy to think these sites are doing something Evil, but chances
>are they're just doing some automated cvs updates on multiple
>repositories once or twice a day, or they had a single big checkout
>of gcc/src and a periodic cvs update or what have you.  Don't read
>too much into it without looking closer - I'm just providing an
>example of what we do track and what we can report.

Btw, just as a data point, I did make a change wrt anoncvs access
last week sometime.  I upped the load average block so that anoncvs
would still keep working under high loads.  There is still a maximum
of 10 anoncvs processes possible, though.  I can lower that but then
the anoncvs people will be complaining.

FWIW, I don't know what's special about me but I'm not seeing a problem
with cvs update.  Maybe I'm just hitting it at the right time.

cgf

Re: gcc development schedule [Re: sharing libcpp between GDB and GCC]

[David O'Brien]

On Thu, Mar 28, 2002 at 10:22:49AM +0100, Gerald Pfeifer wrote:
> On Wed, 27 Mar 2002, Zack Weinberg wrote:
> >>> (E.g. it takes about 10x longer to do "cvs update" on the 3.0
> >>> branch than the trunk.)
> >> Yeah, what's up with that?  (I thought it was just me.)
> 
> Same here.
> 
> On the gcc-3.1 branch, I'm approaching the situation where cvs update
> takes longer than bootstrapping the compiler (with C and C++ frontends).

That is only because one must go across the network to do 'cvs up'.
When are public CVSup servers going to be made available?  A bootstrap on
a fast machine takes 30 minutes.  It does not take anywhere near that
long to 'cvs up' GCC from a local repository.

For those not wanting a local copy of the repository, CVSup in "check out
mode" (gives same results as 'cvs co' or 'cvs up' is 4x faster than
using CVS directly).

-- 
-- David  (obrien@FreeBSD.org)

Re: gcc development schedule [Re: sharing libcpp between GDB and GCC]

[David O'Brien]

On Thu, Mar 28, 2002 at 01:53:46AM -0800, Jason Molenda wrote:
> > On Wed, 27 Mar 2002, Zack Weinberg wrote:
> > >>> (E.g. it takes about 10x longer to do "cvs update" on the 3.0
> > >>> branch than the trunk.)
> > >> Yeah, what's up with that?  (I thought it was just me.)
> 
> The cvs server on sourceware uses an optimization (a patch written
> by Ian Lance Taylor) to cache the information needed for a cvs
> update in a single file per directory.

How about a *REAL* optimization, such as not locking ,v files for anoncvs
users??

-- 
-- David  (obrien@FreeBSD.org)

Re: gcc development schedule [Re: sharing libcpp between GDB and GCC]

[Phil Edwards]

On Wed, Mar 27, 2002 at 10:17:03PM -0500, Kaveh R. Ghazi wrote:
>  > From: Zack Weinberg <zack@codesourcery.com> 
>  > 
>  > I'll add that this is an argument for a better version control
>  > system; most of the changes on the queue inside my head, I would
>  > have done on a branch if CVS didn't make branches such a pain.
>  > (E.g. it takes about 10x longer to do "cvs update" on the 3.0
>  > branch than the trunk.)
>  > zw
> 
> Yeah, what's up with that?  (I thought it was just me.)

I will point out that for doing more than a trivial number of cvs operations,
I saw a huge win after I started mirroring the CVS repository, and then
doing checkouts/updates/diffs from the local mirror.  The only time I need
to explicitly talk to the server is for checkins.


Phil

-- 
If ye love wealth greater than liberty, the tranquility of servitude greater
than the animating contest for freedom, go home and leave us in peace.  We seek
not your counsel, nor your arms.  Crouch down and lick the hand that feeds you;
and may posterity forget that ye were our countrymen.            - Samuel Adams

Re: gcc development schedule [Re: sharing libcpp between GDB and GCC]

[Christopher Faylor]

On Thu, Mar 28, 2002 at 09:00:29AM -0800, David O'Brien wrote:
>On Thu, Mar 28, 2002 at 01:53:46AM -0800, Jason Molenda wrote:
>>>On Wed, 27 Mar 2002, Zack Weinberg wrote:
>>>>>>(E.g.  it takes about 10x longer to do "cvs update" on the 3.0 branch
>>>>>>than the trunk.)
>>>>>Yeah, what's up with that?  (I thought it was just me.)
>>
>>The cvs server on sourceware uses an optimization (a patch written by
>>Ian Lance Taylor) to cache the information needed for a cvs update in a
>>single file per directory.
>
>How about a *REAL* optimization, such as not locking ,v files for
>anoncvs users??

The last time you suggested this, I offered to send you the source code
for the version of cvs that is being run on sources.redhat.com.  And, I
suggested that if you would like to rebuild cvs, verify that it works,
and be available for bug reports for your changes, I'd be willing to
install your new cvs binary on the system.

Why haven't you taken me up on my offer?

cgf

Re: gcc development schedule [Re: sharing libcpp between GDB and GCC]

[Jim Blandy]

Zack Weinberg <zack@codesourcery.com> writes:
> On Wed, Mar 27, 2002 at 10:17:03PM -0500, Kaveh R. Ghazi wrote:
> >  > (E.g. it takes about 10x longer to do "cvs update" on the 3.0
> >  > branch than the trunk.)
> > 
> > Yeah, what's up with that?  (I thought it was just me.)
> 
> The way RCS stores branches makes the cost of calculating diffs from a
> branch tip proportional to the number of versions on the branch *and*
> the distance between the branchpoint and top-of-trunk.  I imagine
> update has to do diffs for some reason.
> 
> That's the problem I know about; there may be others.

The Subversion version control system (http://subversion.tigris.org)
stores the tip of each branch in full text, not as deltas --- for
files modified on that branch, at least.  It should also be very cheap
to create branches; the cost isn't proportional to the size of the
tree being branched.

Or at least, that's the way it's *supposed* to be.  :) Subversion is
still a ways from a 1.0 release.  But the developers are using it to
manage its own source tree.

Something to keep in mind in the future.

Re: gcc development schedule [Re: sharing libcpp between GDB and GCC]

[Phil Edwards]

On Wed, Apr 03, 2002 at 05:19:04PM -0500, Jim Blandy wrote:
> 
> Or at least, that's the way it's *supposed* to be.  :) Subversion is
> still a ways from a 1.0 release.  But the developers are using it to
> manage its own source tree.
> 
> Something to keep in mind in the future.

I think the expectation has always been to eventually replace CVS with
Subversion for the gcc repo (and presumably the src repo); it seems to
be a matter of "when" rather than "if" -- and "when" is "as soon as it's
stable enough".


Phil

-- 
If ye love wealth greater than liberty, the tranquility of servitude greater
than the animating contest for freedom, go home and leave us in peace.  We seek
not your counsel, nor your arms.  Crouch down and lick the hand that feeds you;
and may posterity forget that ye were our countrymen.            - Samuel Adams

Re: gcc development schedule [Re: sharing libcpp between GDB and GCC]

[Neil Booth]

law@redhat.com wrote:-

> and eventually they turn into a Richard Henderson, Mark Mitchell,
> Jeff Law, etc.

That sounds quite frightening.  Surely there's another way?

Neil.

Re: gcc development schedule [Re: sharing libcpp between GDB and GCC]

[law]

 In message <200203271517.HAA22671@kankakee.wrs.com>, mike stump writes:
 > flow out of their, well, uhm, because they are god, rather, because we
 > have seen them make the hard choices before, we have experience with
 > how they make them, and we have in the past agreed with their manner
 > and style.
Right.  And in a similar manner, we support those developers who we see
consistently providing high quality designs, code, etc.  When we see that
happening we give those developers more and more freedom and eventually
they turn into a Richard Henderson, Mark Mitchell, Jeff Law, etc.

 > To overcome this obstacle, you will need to either come up with ideas
 > that are obviously better to us that we agree with, or to put in
 > enough face time for us to know how you make choices.
Right.  Ultimately it comes down to building a certain amount of trust between
the new developer and the older developers.  Building that trust usually
happens as the new developer submits quality analysis, designs and code.

It may seem odd, but there was a time when each and every one of the key
developers today had to go through the same process of building the trust
of the older developers.   In some cases that process actually happened
nearly a decade before we even had a steering committee.



 > Also, ideas are free...  Or put another way, ideas are a dime a dozen.
 > We can all come up with many ideas, the problem isn't one of coming up
 > with ideas, it is finding the resources to implement them.
Amen!  From my personal experience, that is really the kicker.  And in
fact, changes over the last few years in my personal and professional
life have severely limited how much I can be involved in the day to day
development of GCC.  I contribute when I can, but the torch has largely
been passed along to others.

One of my key goals when we started the EGCS project 5 years ago was to
build a self-sustaining project.  It's been a joy to watch folks like
Mark, Richard, Bernd, Benjamin, Gerald, David, Alex, Jan and a host of
others pick up where I left off and continue to move the project forward
as I stepped away from the daily grind.  I'm sure that some of the key
folks today will move on in the future, but hopefully we've put a system
in place by which new developers are continually brought into the fold.

 > It has been that, it now that, and will always be that.  If you increased 
the
 > value of your contributions to include the resources to implement
 > them, and yes, you can implement any idea you want...  If you want a
 > SC committee that collects money from companies, you can always create
 > one, using anybody you want, using any rules you want.  Just do it.
 > You don't need anybodies permission or acceptance.
Right.  One of the core ideas when we started EGCS was that folks contribute
based on their abilities.  For some that means they work on the internals
of GCC, others deal with web pages, other deal with political issues, others
deal with documentation, testsuites, etc etc.

If someone's skill is finding money and managing dispersal of that money to
foster development of key technologies, then, well, we'd welcome that.  But
we're not necessarily going to re-focus the steering committee or the
developers to perform those tasks.

jeff

Re: gcc development schedule [Re: sharing libcpp between GDB and GCC]

[mike stump]

> Date: Tue, 26 Mar 2002 22:44:40 -0800 (PST)
> From: Tom Lord <lord@emf.net>
> To: eliz@is.elta.co.il

> 1) It's no secret that I'm not a core maintainer of GCC.  I have done
>    some GCC hacking and ran into some practical obstacles while
>    looking for ways to see that work wind up in GCC distributions.

My take on how the gcc group runs...  We take contributions, both of
code and in design from those that we have seen the most that we agree
with their choices.  We back rth, or law, or Mark, not because gold
flow out of their, well, uhm, because they are god, rather, because we
have seen them make the hard choices before, we have experience with
how they make them, and we have in the past agreed with their manner
and style.

To overcome this obstacle, you will need to either come up with ideas
that are obviously better to us that we agree with, or to put in
enough face time for us to know how you make choices.

Also, ideas are free...  Or put another way, ideas are a dime a dozen.
We can all come up with many ideas, the problem isn't one of coming up
with ideas, it is finding the resources to implement them.  It has
been that, it now that, and will always be that.  If you increased the
value of your contributions to include the resources to implement
them, and yes, you can implement any idea you want...  If you want a
SC committee that collects money from companies, you can always create
one, using anybody you want, using any rules you want.  Just do it.
You don't need anybodies permission or acceptance.

If you want an automated system that manages branches, and removes
code that doesn't meet the testsuite standard, create it.  If you want
a system that tries out people patches before they check them in,
create it.

If anything you create is good and has value, to any one of us, then
that one of us will endorse it and use it, to the extent they can.  A
better contribution is one that is valued and used by more and more
people.  If enough people buy into it, it becomes the standard,
whether or not rms or the FSF likes it (witness egcs, cygwin, Linux),
whether or not the official SC likes it, whether or not any single or
small group of contributors like it.  This is the path before you.

If you are unwilling to do the time, we have a solution for that.
Your ideas and suggestions will just be archived in the list, and if
no one else values them high enough to implement them, then the slowly
fade away in the archive, eventually to either be lost, or if someone
wakes up and smells the roses, to be implemented by someone that does
believe in them.

>    Some of the invective I get in private mail (or in Eli's message)
>    seems to suggest that unless you've done 10 ports, have write
>    access, or otherwise have a suitable GCC-testosterone certificate
>    that, well, you're a valid target for complete disregard or worse.

Yes.  And above I try and explain why that is.  It isn't about being
disregarded, it is that you are a beggar and we just have no money for
you.  Don't beg.  Give.

> 2) I do have a pretty decent amount of experience in software tools,
>    process automation, and related software engineering issues.  I
>    have a pretty decent amount experience in the Free Software and
>    open source worlds.  These are areas I think a lot about and build
>    tools for.  I'm really not talking through my hat here.

This isn't enough for most of us.  In fact, it isn't enough for any of
us, if no one steps forward and implements any of your ideas.

>    They've remarked that some of my ideas seem like good ones,

Even the ones that seem to object the most to your ideas will
internally agree with your best ideas.  The problem is anytime you
present two orthogonal ideas, and they argree with one of them, some
can only shoot down the one.

>    though overall there's a lot of hesitancy to embrace any idea

You mean, you can witness that money and real resources don't just
follow you around and people just throw money at you...  Well, did you
expect any different?  If people don't throw money at you, maybe it is
because you haven't eared it?  You can either reset your expectation,
or only suggest ideas that people will throw money at, or pay people
to throw money at you, the choice is yours alone.

>    that people don't see immediately how to pay for.

No there isn't.  Lots of people implement lots of things.  Witness
gcc-patches.  Witness libcpp.

>    I have detected in SC member comments, both privately and on the
>    list, insidious conflict-of-interest issues at work:

And as I've said before the SC is irrelevant.  The will break ties,
but you have to be tied first.  You're not tied.  They will keep us
from doing things that are incredibly bad.  They cannot keep up from
doing things that are just bad.  Not all your ideas are incredibly
bad, or put another way, the SC cannot stop you from implementing most
of your ideas.

Re: gcc development schedule [Re: sharing libcpp between GDB and GCC]

[Gianni Mariani]

But, does it have a Windows client ?

Andrew Cagney wrote:

> [big long thread about revision control systems deleted]
>
> Hmm, Aegis (http://aegis.sourceforge.net).  (Is this the Goodwin key 
> word of software configuration management systems? :-)
>
> Andrew

Re: gcc development schedule [Re: sharing libcpp between GDB and GCC]

[Andrew Cagney]

[big long thread about revision control systems deleted]

Hmm, Aegis (http://aegis.sourceforge.net).  (Is this the Goodwin key 
word of software configuration management systems? :-)

Andrew

Re: gcc development schedule [Re: sharing libcpp between GDB and GCC]

[Robert Dewar]

<<(and I suspect that there are, in fact, several more than two); a
better approach would be to collect those automated testing
infrastructures under a common (automated) interface and make them
available to lots of branches (even branches not maintained by people
with write access to the main repository).
>>

In the case of GNAT, our primary test suite is composed almost entirely
of proprietary code from customers, and cannot be made available. I would
be surprised if that is not the case in other situations.

Re: gcc development schedule [Re: sharing libcpp between GDB and GCC]

[Fergus Henderson]

On 26-Mar-2002, Tom Lord <lord@emf.net> wrote:
> Regarding revision control, there can be a queue from branches
> (including branches outside the main repository) to the trunk.  For
> most change sets, propagation along that queue, passing through the
> testing barrier, can be automated.

Resolving merge conflicts can never be automated.

> >  	In addition, it assumes there is no overhead associated with
> >	maintaining a branch
> >     Sorry, but nobody is going to merge a complete rewrite of the
> >     C++ parser, for instance, to the head, without *any*
> >     difficulties whatsover.
> 
> Using _any_ modern system, the merge can be worked out on a branch and
> the transition to the trunk automated.

That doesn't solve the problem.  If changes stay on branches for longer,
rather than being incorporated in the trunk, there will be more conflicts.
This leads to increased overhead resolving the merge conflicts.

-- 
Fergus Henderson <fjh@cs.mu.oz.au>  |  "I have always known that the pursuit
The University of Melbourne         |  of excellence is a lethal habit"
WWW: <http://www.cs.mu.oz.au/~fjh>  |     -- the last words of T. S. Garp.

Re: gcc development schedule [Re: sharing libcpp between GDB and GCC]

[Tom Lord]

	   Dan:

	   Because it causes things to stagnate on branches, while
	   attempting to meet the "high barrier" on the trunk.

Not necessarily.  It depends on your tools.  For this issue, the
relevant tools are the testing infrastructure and revision control
infrastructure.

Regarding testing, there seems to be two shortfalls.  One is in the
completeness of the existing test suite: at least one vendor finds it
necessary to supplement that test suite with tests they buy a license
for in order to obtain better coverage; beefing up the test suite
would give more developers more direct access to more thorough
testing.  The other issue is automation.  There are at least two
instances of vendors implementing their own infrastructures to grab
the head from the trunk and test that periodically in various ways
(and I suspect that there are, in fact, several more than two); a
better approach would be to collect those automated testing
infrastructures under a common (automated) interface and make them
available to lots of branches (even branches not maintained by people
with write access to the main repository).

Regarding revision control, there can be a queue from branches
(including branches outside the main repository) to the trunk.  For
most change sets, propagation along that queue, passing through the
testing barrier, can be automated.  BitKeeper seems to be up to that
already.  arch and Subversion are both close behind.  Even where you
might want to inject a human conducted review into the queue process,
tools infrastructure can help simplify that process and reduce the
reviewer's action from "check in these changes" to "click the
thumbs-up/down button".

	In addition, it assumes there is no overhead associated with
	maintaining a branch (and let's not get into a discussion
	about arch or anything

Actually, I've been trying to avoid the topic of merging tools
precisely because the issue is bigger than arch.  Yes, I think arch
can be a big help here -- but so, perhaps, can Subversion or
BitKeeper.  Heck, there's even a GCC maintainer who contributes to
Subversion.  If the discussion ever reaches the stage of contemplating
choices of revision control system, I'll be happy to help make the
case for arch.  At this stage, in this context, it's probably not
important.


       Sorry, but nobody is going to merge a complete rewrite of the
       C++ parser, for instance, to the head, without *any*
       difficulties whatsover.

Using _any_ modern system, the merge can be worked out on a branch and
the transition to the trunk automated.

One advantage of "hierarchical software management" is that it allows
difficult merge instances to be distributed down the hierarchy --
making it easier for more than just maintainers with write access to
work on them.  Core maintainer bandwidth can be _increased_ by
effective branching, in contrast to the expectation expressed
there and elsewhere in your message.

-t

Re: gcc development schedule [Re: sharing libcpp between GDB and GCC]

[Daniel Berlin]

On Wed, 27 Mar 2002, Eli Zaretskii wrote:

> 
> On Tue, 26 Mar 2002, Tim Hollebeek wrote:
> 
> > > The approach I am suggesting is an evolutionary step beyond 
> > > the current practices and is quite consistent with the SC 
> > > development goals.
> > 
> > The SC can and will be the judge of that.  Until then, please make
> > sure that your contributions to gcc meet or exceed your contributions
> > to this list.
> 
> Sigh.  Another attempt to shut up discenting opinions instead of dealing 
> with them in a civilized and technical manner.
> 
> If Tom's arguments are as grossly invalid as you indicate, there should 
> be no problem to show that with simple technical arguments.  Could you 
> please do that?

Sure.
Tom has not shown how his approach differs substantially from the one that 
has been previously been attempted, or for that matter, the one that was 
in use that eventually led to the creation of EGCS.

In fact, it seems exactly like the situation that led to the creation of 
EGCS, minus any political overtones/interpersonal relationships/etc that 
were incidental.

This has been shown to not work.  Why?
Because it causes things to stagnate on branches, while attempting 
to meet the "high barrier" on the trunk.  If you move the barrier 
lower, however, you also let in some of the changes that really 
weren't ready.  
In addition, it assumes there is no overhead associated with 
maintaining a branch (and let's not get into a discussion about arch 
or anything, which i have a weird feeling he's trying to drag this 
into, no offense to Tom or anything), and that the branches will be as 
instantly up-to-date as the trunk, such that merges are not difficult, 
and one can effectively insure that merging a branch will cause no 
problems.
Sorry, but nobody is going to merge a complete rewrite of the C++ parser, 
for instance, to the head, without *any* difficulties whatsover.
It would require so much Q&A it's not even funny, and you would still 
likely have problems, even with this massive amount of Q&A.
Furthermore, raising the barrier effectively pushes away young and new 
contributors, because they'll never be able to get simple patches 
accepted without a large amount of work (which is needed to ensure the 
trunk is high quality).  It'll likely turn them off to trying to do 
anything larger.  Simple patches that haven't been heavily tested do not 
necessarily affect trunk quality in a meaningful way. If some minimal 
level of testing was performed on them, it is generally sufficient.
Certainly, I would not have started working on gcc in an environment that 
Tom suggests.
  While I could probably keep my patches at the level he's 
asking for now, I could not when i first started out, and if it took 5000 
revisions and regression tests before it would even be *accepted*, i would 
have just stopped caring and moved on.
Lastly, It's also very difficult to have a single level barrier that 
always exists, that simultaenously allows   possibly intrusive changes to 
be made, without causing the quality of the trunk to be lower.
The current plan attempts to alleviate this by progressively raising the 
barrier over time.
You'll note that most people doing fixes right now on the 3.1 branch are 
experienced gcc developers.  People just starting out rarely sit for hours debugging 
single problems in an effort to get a high quality release.

Theoretically, the model you want to use is that which simulatenously:
1. maximizes the productivity of your scarecest resource, while at the same time, enables others to 
evolve into the role your scarecest resource currently fulfills. (i.e. 
lets experienced developers get work done and new features done, while at 
the same time, not stopping people just starting out from getting patches 
in, learning, and growing as part of the community).
2. maximizes the quality of your releases, in order to give the users 
something to use that works.

In Tom's model, 2 is more important than 1.
The current model tries to balance it more.

Trying to make a model that perfectly balances this is almost impossible 
in practice, since all the variables change over time due to external 
factors.

One of my biggest peeves about most software management models and 
theories is that few times, if ever, has anyone knowledgeable about 
software development just sat and observed the process, day after day, 
without being a part of it.
In effect, studied software developers like they study animals, or other 
groups of people.
I've never seen a posting on a bulletin board requesting software 
developers for any sort of study.
Maybe it's just me.

This is all rambling at 2am in the morning, so if it makes no sense, 
ignore it.
:)
--Dan

Re: gcc development schedule [Re: sharing libcpp between GDB and GCC]

[Tom Lord]

       Further enhancing the testing infrastructure is one objective that I'd
       like to see and where I've worked on already,

Yes.  The GCC project has _very_noticably_ improved in many process
areas since roughly the time the SC came on-line.

I come from a very "software tools" oriented background.  I think
there's a lot of leverage to be obtained not by waiting for volunteers
to contribute things that happen to be useful, but by making an
aggressive, worked-out plan and then asking how it can be implemented
via funded projects.  The SC mission statement contains lofty goals,
but it isn't an aggressive, actionable plan for the most part.  The
list discussions contain many good ideas and initiatives, but overall,
that's a scattershot rather than an especially organized approach to
the problem.

And it's an issue that's both larger than GCC, but to which GCC can
make a very useful contribution.  Well designed tools for improving
the GCC process will work for other projects as well.  At the same
time, GCC, more than most projects, exemplifies both the strengths and
challenges of open source processes -- an emerging software
engineering discipline of exponentially increasing commercial and
social importance.

-t

Re: gcc development schedule [Re: sharing libcpp between GDB and GCC]

[Tom Lord]

   From: Eli Zaretskii <eliz@is.elta.co.il>
   On Tue, 26 Mar 2002, Tim Hollebeek wrote:

   > > The approach I am suggesting is an evolutionary step beyond 
   > > the current practices and is quite consistent with the SC 
   > > development goals.
   > 
   > The SC can and will be the judge of that.  Until then, please make
   > sure that your contributions to gcc meet or exceed your contributions
   > to this list.

   Sigh.  Another attempt to shut up discenting opinions instead of dealing 
   with them in a civilized and technical manner.


Thanks.  May I add a few points?

1) It's no secret that I'm not a core maintainer of GCC.  I have done
   some GCC hacking and ran into some practical obstacles while
   looking for ways to see that work wind up in GCC distributions.

   Some of the invective I get in private mail (or in Eli's message)
   seems to suggest that unless you've done 10 ports, have write
   access, or otherwise have a suitable GCC-testosterone certificate
   that, well, you're a valid target for complete disregard or worse.


2) I do have a pretty decent amount of experience in software tools,
   process automation, and related software engineering issues.  I
   have a pretty decent amount experience in the Free Software and
   open source worlds.  These are areas I think a lot about and build
   tools for.  I'm really not talking through my hat here.


3) I've had some chats with a few maintainers and SC members via
   email.  They've corrected me on a few of my perceptions of the
   current processes (as I asked them to).  They've remarked that some
   of my ideas seem like good ones, though overall there's a lot of
   hesitancy to embrace any idea that people don't see immediately how
   to pay for.  I have detected in SC member comments, both privately
   and on the list, insidious conflict-of-interest issues at work: 
   it's not where I want to focus on this list because I think
   people's intentions are rightly oriented -- but there _is_ a
   structural problem with how the project is run, having both tool
   and political components.


4) There is a history, in the GCC/egcs schizm, of people identifying
   practical needs of the project and that turning into improvements
   in project governance, process, infrastructure, and relation to
   commercial activities.  So it really isn't even naive to raise some
   pretty well understood process issues in hopes that the engineers
   can lead the market in this area.


5) The GCC project is larger now than just a few years ago, and vastly
   more important to the entire Free Software and open source worlds.
   This is a critical structural element.  It deserves careful
   attention and input from multiple perspectives.


-t

(And I'll leave gdb on the CC line since these issues are broader in
scope than just GCC.)

Re: gcc development schedule [Re: sharing libcpp between GDB and GCC]

[Tom Lord]

   From: Eli Zaretskii <eliz@is.elta.co.il>
   On Tue, 26 Mar 2002, Tim Hollebeek wrote:

   > > The approach I am suggesting is an evolutionary step beyond 
   > > the current practices and is quite consistent with the SC 
   > > development goals.
   > 
   > The SC can and will be the judge of that.  Until then, please make
   > sure that your contributions to gcc meet or exceed your contributions
   > to this list.

   Sigh.  Another attempt to shut up discenting opinions instead of dealing 
   with them in a civilized and technical manner.


Thanks.  May I add a few points?

1) It's no secret that I'm not a core maintainer of GCC.  I have done
   some GCC hacking and ran into some practical obstacles while
   looking for ways to see that work wind up in GCC distributions.

   Some of the invective I get in private mail (or in Eli's message)
   seems to suggest that unless you've done 10 ports, have write
   access, or otherwise have a suitable GCC-testosterone certificate
   that, well, you're a valid target for complete disregard or worse.


2) I do have a pretty decent amount of experience in software tools,
   process automation, and related software engineering issues.  I
   have a pretty decent amount experience in the Free Software and
   open source worlds.  These are areas I think a lot about and build
   tools for.  I'm really not talking through my hat here.


3) I've had some chats with a few maintainers and SC members via
   email.  They've corrected me on a few of my perceptions of the
   current processes (as I asked them to).  They've remarked that some
   of my ideas seem like good ones, though overall there's a lot of
   hesitancy to embrace any idea that people don't see immediately how
   to pay for.  I have detected in SC member comments, both privately
   and on the list, insidious conflict-of-interest issues at work: 
   it's not where I want to focus on this list because I think
   people's intentions are rightly oriented -- but there _is_ a
   structural problem with how the project is run, having both tool
   and political components.


4) There is a history, in the GCC/egcs schizm, of people identifying
   practical needs of the project and that turning into improvements
   in project governance, process, infrastructure, and relation to
   commercial activities.  So it really isn't even naive to raise some
   pretty well understood process issues in hopes that the engineers
   can lead the market in this area.


5) The GCC project is larger now than just a few years ago, and vastly
   more important to the entire Free Software and open source worlds.
   This is a critical structural element.  It deserves careful
   attention and input from multiple perspectives.


-t

(And I'll leave gdb on the CC line since these issues are broader in
scope than just GCC.)

Re: gcc development schedule [Re: sharing libcpp between GDB and GCC]

[Andreas Jaeger]

Tom Lord <lord@emf.net> writes:

>        I agree.  I think the releases should be an 8-month cycle rather
>        than a 6-month cycle.  In other words, 4 months for destabilizing
>        changes.
>
> Given the distributed and opportunistic nature of development,
> wouldn't a phaseless approach be worth considering?  Ultimately
> lower cost for all participants?  Certainly put GCC in the position
> of being better able to make near-instant "emergency releases" to correct
> defects that escape up-front testing?  Certainly avoid snafus like
> Red Hat experienced a little while back?
>
> By "phaseless" I mean an approach in which there is a permanently,
> continuously QA'd trunk with a high barrier for changes.  Presumably
> along with that, a collection of advanced trunks on which related
> destabilizing changes are collected and worked-out.  This is sometimes
> called "hierarchical software management" (where the hierarchy is of
> lines-of-development, not people).
>
> The path from here to there would seem to be one of simply
> beefing up the infrastructure with better automation, and 
> better testing tools.

I think GCC is going already slowly in this direction, the barrier is
higher than it was, a year or two ago:

- we have automatic testers and benchmarks so that the trunk is build
  and tested on a daily basis on different platforms
- we have a policy of patch reversion for patches that are too broken
- we have a policy that patches should have testcases with them for
  regression testing

I agree that it's better to analyze the effects of a patch directly
after it went in (or even better before) instead of some months
later.  

Further enhancing the testing infrastructure is one objective that I'd
like to see and where I've worked on already,

Andreas
-- 
 Andreas Jaeger
  SuSE Labs aj@suse.de
   private aj@arthur.inka.de
    http://www.suse.de/~aj

Re: gcc development schedule [Re: sharing libcpp between GDB and GCC]

[Eli Zaretskii]

On Tue, 26 Mar 2002, Tim Hollebeek wrote:

> > The approach I am suggesting is an evolutionary step beyond 
> > the current practices and is quite consistent with the SC 
> > development goals.
> 
> The SC can and will be the judge of that.  Until then, please make
> sure that your contributions to gcc meet or exceed your contributions
> to this list.

Sigh.  Another attempt to shut up discenting opinions instead of dealing 
with them in a civilized and technical manner.

If Tom's arguments are as grossly invalid as you indicate, there should 
be no problem to show that with simple technical arguments.  Could you 
please do that?  I, for one, would like to hear those arguments.

And while at that, is it too much to ask for some minimal politeness, 
please?

Re: gcc development schedule [Re: sharing libcpp between GDB and GCC]

[Andreas Jaeger]

David Edelsohn <dje@watson.ibm.com> writes:

>>>>>> Richard Henderson writes:
>
> Richard> I think this is a mistake.  It is completely unreasonable to have
> Richard> the entire "major structural changes" time period concurrent with
> Richard> "please work on making a release happen".
>
> Richard> I've been working under the assumption that the current schedule can
> Richard> be changed, so that I can help Diego with the language independent
> Richard> tree-ish IL after the 3.1 release.
>
> 	I agree.  We do not have enough developer bandwidth to work on
> major, new changes concurrently with the release process.  Even after GCC
> 3.1.0 is released, we will need to assess how much effort will be required
> for the GCC 3.1.1 bug-fix release the following quarter.

<AOL> I agree, too</AOL>.  

Even if development is done on branches, it takes time to bring
changes back into mainline in a clean way: test them properly on
various platforms, split them in small pieces, get them reviewed,...
2 months is not enough for this - and then you should be working on
the same time on getting a good release out.  An 8 or 10 month release
cycle seems more approbriate with the GCC man power.

Andreas
-- 
 Andreas Jaeger
  SuSE Labs aj@suse.de
   private aj@arthur.inka.de
    http://www.suse.de/~aj

Re: gcc development schedule [Re: sharing libcpp between GDB and GCC]

[Tim Hollebeek]

> The approach I am suggesting is an evolutionary step beyond 
> the current practices and is quite consistent with the SC 
> development goals.

The SC can and will be the judge of that.  Until then, please make
sure that your contributions to gcc meet or exceed your contributions
to this list.

-Tim

Re: gcc development schedule [Re: sharing libcpp between GDB and GCC]

[Tom Lord]

   From: David Edelsohn <dje@watson.ibm.com>

	   Because many users (both inviduals and companies) want a schedule
   so that they can plan releases.

That is an orthogonal issue.


   GCC has operated [under a continuous release management,
   hierarchical software management model] and it was not very
   effective.

You are quite mistaken.  The GCC project has never operated that way.

The approach I am suggesting is an evolutionary step beyond 
the current practices and is quite consistent with the SC 
development goals.

-t

Re: gcc development schedule [Re: sharing libcpp between GDB and GCC]

[David Edelsohn]

>>>>> Tom Lord writes:

Tom> Given the distributed and opportunistic nature of development,
Tom> wouldn't a phaseless approach be worth considering?  Ultimately
Tom> lower cost for all participants?  Certainly put GCC in the position
Tom> of being better able to make near-instant "emergency releases" to correct
Tom> defects that escape up-front testing?  Certainly avoid snafus like
Tom> Red Hat experienced a little while back?

	Because many users (both inviduals and companies) want a schedule
so that they can plan releases.

	Tom, may I suggest that you take some time to learn how the GCC
Development Plan came into existance.  As with the GCC SC issue, you are
making suggestions based on invalid premises.  GCC has operated the other
way and it was not very effective.

Thanks, David

Re: gcc development schedule [Re: sharing libcpp between GDB and GCC]

[Zack Weinberg]

On Tue, Mar 26, 2002 at 11:17:30PM +0000, Neil Booth wrote:
> Richard Henderson wrote:-
> > I think this is a mistake.  It is completely unreasonable to have
> > the entire "major structural changes" time period concurrent with
> > "please work on making a release happen".
> 
> I agree.  I think the releases should be an 8-month cycle rather
> than a 6-month cycle.  In other words, 4 months for destabilizing
> changes.

m3 t00.

I'll add that this is an argument for a better version control system;
most of the changes on the queue inside my head, I would have done on
a branch if CVS didn't make branches such a pain.  (E.g. it takes
about 10x longer to do "cvs update" on the 3.0 branch than the trunk.)

zw

Re: gcc development schedule [Re: sharing libcpp between GDB and GCC]

[Tom Lord]

       I agree.  I think the releases should be an 8-month cycle rather
       than a 6-month cycle.  In other words, 4 months for destabilizing
       changes.

Given the distributed and opportunistic nature of development,
wouldn't a phaseless approach be worth considering?  Ultimately
lower cost for all participants?  Certainly put GCC in the position
of being better able to make near-instant "emergency releases" to correct
defects that escape up-front testing?  Certainly avoid snafus like
Red Hat experienced a little while back?

By "phaseless" I mean an approach in which there is a permanently,
continuously QA'd trunk with a high barrier for changes.  Presumably
along with that, a collection of advanced trunks on which related
destabilizing changes are collected and worked-out.  This is sometimes
called "hierarchical software management" (where the hierarchy is of
lines-of-development, not people).

The path from here to there would seem to be one of simply
beefing up the infrastructure with better automation, and 
better testing tools.

-t

Re: gcc development schedule [Re: sharing libcpp between GDB and GCC]

[Neil Booth]

Richard Henderson wrote:-

> I think this is a mistake.  It is completely unreasonable to have
> the entire "major structural changes" time period concurrent with
> "please work on making a release happen".

I agree.  I think the releases should be an 8-month cycle rather
than a 6-month cycle.  In other words, 4 months for destabilizing
changes.

Neil.

Re: gcc development schedule [Re: sharing libcpp between GDB and GCC]

[David Edelsohn]

>>>>> Richard Henderson writes:

Richard> I think this is a mistake.  It is completely unreasonable to have
Richard> the entire "major structural changes" time period concurrent with
Richard> "please work on making a release happen".

Richard> I've been working under the assumption that the current schedule can
Richard> be changed, so that I can help Diego with the language independent
Richard> tree-ish IL after the 3.1 release.

	I agree.  We do not have enough developer bandwidth to work on
major, new changes concurrently with the release process.  Even after GCC
3.1.0 is released, we will need to assess how much effort will be required
for the GCC 3.1.1 bug-fix release the following quarter.

David

gcc development schedule [Re: sharing libcpp between GDB and GCC]

[Richard Henderson]

On Mon, Mar 25, 2002 at 08:07:35PM -0800, Zack Weinberg wrote:
> I like this plan, and am provisionally willing to do the work, with
> the warning that we're coming up on the "no more major structural
> changes" freeze deadline for GCC 3.2 ...

I think this is a mistake.  It is completely unreasonable to have
the entire "major structural changes" time period concurrent with
"please work on making a release happen".

I've been working under the assumption that the current schedule can
be changed, so that I can help Diego with the language independent
tree-ish IL after the 3.1 release.


r~