Re: [maint] The GDB maintenance process

Daniel Jacobowitz
2003-02-19T01:49:11Z

Thanks to everyone who responded; I'll try to address everything.

I want to share a piece of perspective on why I raise this issue now.
I'm finding the GDB development process to be too slow to be workable -
patches take a month or more routinely, and I don't have the time to
juggle them for that long. I'm facing that I may need to back down my
level of involvement as a consequence of how awkward the development
model is for getting any forward progress.

Maybe that means that I just don't have the time and the patience to be
a useful contributor to GDB. Me, I think that it means that we need to
make the process less painful for contributors.

On Mon, Feb 17, 2003 at 09:35:53PM -0500, Andrew Cagney wrote:
> > I've discussed this general situation with a (small) sampling of other GDB
> > developers and contributors - enough to know that I'm not alone in my
> > concerns. These aren't entirely my own words, either. I'll let other
> > people take credit/blame for them if they want to, and if I've represented
> > their opinions accurately.
>
> I recall you rasing this previously with me but I don't recall
> agreeing with your opinions (my concerns run along similar lines to
> what Elena posted).

Hmm, I don't remember actually discussing this with you before. Must
be going soft. We've certainly butted heads over related topics.

> Normally political issues get discussed with the global maintainers
> (so that the mailing lists can focus on technical discussion).
> However, if you feel that it is better discussed here, then so be it.

If you feel that's more appropriate, just say so. I've no particular
preference for where the conversation happens, as long as it happens.

On Mon, Feb 17, 2003 at 04:05:40PM -0500, Elena Zannoni wrote:
> Daniel Jacobowitz writes:
> > I get the feeling I'm already pretty unpopular for some of my opinions on
> > how GDB maintenane should work. This isn't going to make it any better, but
> > I feel it should be said.
> >
>
> OK, I'll take the bait. No offense intended. Just read until the end.
>
> > I believe that our current process has some problems. Let me try to
> > explain. First, to make sure we're all on the same page...
> >
> > What does it mean to be a Global Maintainer, in practice?
> > - A certain amount of autonomy in the areas of GDB that no one wants to
> > take specific responsibility for. There's no specific maintainer for
> > things like the frame support or the type/value systems.
> > - A little more freedom in approving patches to other people's areas of
> > GDB - not a lot, but it's definitely there.
> > [In practice, this depends on:
> > o How much risk you're willing to take of annoying people.
> > o How likely other maintainers are to shout at you about it.]
>
> The above point is not part of the global maintainer description. I
> take you want it to be? I don't agree. It just makes it possible for
> particularly 'active' individuals to dominate the list and removes
> the more egalitarian feel that this community has. You can say that
> the linux kernel community is like that, and that gcc is like that,
> etc, but we don't have to be.

I don't especially want it to be; I want something different, farther
down. This is my vision of how we work right now. The above is
informal but in my experience a pretty accurate summary of the way
things really are.

> I have seen sometimes very quick turnarounds on patches during
> holidays, and maybe some of such patches should have been thought
> through more carefully. If you don't give time to the appropriate
> maintainers to chime in, the entropy can become way too high, with
> patches, and reverted patches going around.

I guess I just don't see this to be as much of a problem as others do.
For one thing, with the higher entropy level, more development actually
happens.

> > Another thing to think about: because of the layout of the above, there is
> > frequently no one who has the _right_ to approve a patch. They require
> > buy-in from a number of additional maintainers. In addition our volunteers
> > are often too busy to find time to respond to patches. This impacts patches
> > from other maintainers (frequently, but generally a small impact) and from
> > outside contributors (happens less frequently, but larger impact - most of
> > these never get approved at all, from what I've seen).
> >
>
> Wait, lots of external contributor's patches never make it in because
> of the copyright assignment problems. Also I see external people
> dropping patches, not because the are not reviewed, but because they
> *are reviewed*. I.e. a patch is submitted, I ask for some changes, and
> the person never comes back with a new patch.
>
> There would be also something to say about championing your patches,
> ie, push a bit. For instance this is what David is doing with the
> namespaces stuff, he is keeping me posted, and he makes sure I don't
> fall asleep at the wheel.
>
> Just saying, things are not as black and white as they are portraited.

Sure. But my portrayal isn't entirely inaccurate, either. One example
- inter-compilation-unit references in DWARF-2. It's been posted at
least twice, by different contributors. When asked for changes,
they've come back with changes. It's still not supported, over a year
later.

On the subject of David, infinite kudos to him for his
single-mindedness and patience. When I look at the amount of forward
progress he's made compared to the time he's invested in making it, I
become even more convinced that we have a problem.

> > Some other GNU projects have a similar setup and don't have this problem.
> > GCC and binutils are my usual examples. How do they avoid it? They have a
> > different definition of global maintainer. That's what ours used to be
> > called - Blanket Write Privileges. The system works a little differently:
> > - Maintainers for specific areas of the compiler can commit/approve
> > patches to the areas they maintain without buy-in from a blanket
> > maintainer.
>
> I think you are misunderstanding. Gdb is like that. If one maintains
> one area, that's it, period. His/her approval is enough. Even if the
> maintainer is not global, the authority is the same *in that specific
> area*.

I know that. That's no different between GCC and GDB; I'm not
discussing a radical 100% turnaround here.

> > - Blanket maintainers can commit/approve patches anywhere without buy-in
> > from a specific area maintainer.
> >
> > [I hope Richard will forgive me for using him as an example and for putting
> > words in his mouth...] This doesn't replace common sense - you generally
> > won't find Richard Henderson approving patches to the C++ frontend, because:
> > - He knows he isn't familiar with it
> > - He knows it has an active set of maintainers at all times
> >
> > Similarly, just because he can check in patches to any target backend, that
> > doesn't mean he won't ask a target maintainer to look over it first. If
> > someone objects to a patch in their area, he would generally not just check
> > it in anyway. If they object to it after he checks it in, the two will
> > discuss the problem like reasonable people and come to some agreement.
> >
> >
> > Some noticeable differences between these two models:
> > - In the GCC model, more people are able/likely to check in patches which
> > break things.
> > - But in the GCC model, more people are able/likely to check in patches to
> > fix it afterwards.
> > - Because more people have the privilege of approving a given patch,
> > and fewer people's approvals are needed for any particular patch,
> > patches (usually) get approved more quickly.
> > - Development can happen more quickly, and does not get slowed to a
> > standstill when (say) one of us is pulled off of community GDB work for
> > an urgent customer project. This happens all the time - I've never seen
> > all the GDB maintainers with time for GDB at the same time.
> >
>
> Yes, we have day jobs, that are not 'work on FSF gdb'. If you loosen
> the rules, then it becomes possible for a small, fully funded, elite to
> take control.
>
> And how long would you wait before stepping in? That's a hard problem.
> 1 hour, one day, one working day, one week?

Are we talking about the same thing? I don't think I understand you.

First of all, the idea of a small, fully funded elite taking control of
the project doesn't make any sense to me. Either their changes are
worthwhile - the existing maintainers will cooperate with them - or
they overstep their boundaries - they are asked to cease. We don't
hand out maintainership to all comers.

It's not about control of GDB, it's about the growth and quality of
GDB! I don't care who controls it. I'll mostly resist making a
comment here about who currently controls GDB - more than half the
global maintainers work at the same place - because it's an entirely
benign and open control.

Second, what do you mean by "step in"? Step in in response to what? A
broken patch? A violation of authority? The latter and partly the
former are what GCC has a formal steering committee for. We could do
the same thing, or use a vote of the global maintainers. In practice
it's rarely necessary.

> > Right now, we use stricter policies to prevent problems which cause
> > breakage. I think these policies are stifling us. Loosening them (and
> > maybe adding a formal patch reversion policy) would let more people fix
> > problems more easily, as they arise, without slowing development.
> >
>
> I really think that having the bug database track patches is a very
> big improvement. We should try to push its use a bit more before
> giving up on the current rules.

I don't like it, but that's just my general anti-GNATS rancor. Maybe
Bugzilla will be better.

> > If there are people on our Global Maintainer list that we don't think should
> > be trusted with the extra responsibility of the above, then perhaps we need
> > to rethink who belongs in that list. I'm not pointing any fingers - I don't
> > have anyone in mind, and I've been quite happy working with absolutely all
> > of the current team. Just putting the idea out.
> >
> >
> > I've discussed this general situation with a (small) sampling of other GDB
> > developers and contributors - enough to know that I'm not alone in my
> > concerns. These aren't entirely my own words, either. I'll let other
> > people take credit/blame for them if they want to, and if I've represented
> > their opinions accurately.
>
> Hmm, I am not too happy with this behind the scenes discontent, if
> there is something wrong, let's hear it.

That's what I'm doing! I assure you, I don't enjoy generating this
much rancor.

On Mon, Feb 17, 2003 at 11:28:46PM -0500, Andrew Cagney wrote:
>
> > What does it mean to be a Global Maintainer, in practice?
> > - A certain amount of autonomy in the areas of GDB that no one wants to
> > take specific responsibility for. There's no specific maintainer for
> > things like the frame support or the type/value systems.
> > - A little more freedom in approving patches to other people's areas of
> > GDB - not a lot, but it's definitely there.
> > [In practice, this depends on:
> > o How much risk you're willing to take of annoying people.
> > o How likely other maintainers are to shout at you about it.]
> > - Authority to approve patches covering general debugger issues.
> >
> > What does it mean to be a maintainer for a specific host/target/subsystem,
> > in practice?
> > - The authority to approve patches and apply your own patches to that area
> > of the debugger.
> >
> > I'd like everyone to notice one thing missing from the above list. No one
> > has the _responsibility_ for approving patches. This is a volunteer
> > project, and anyone who's watched it in action for a little while will see
> > that the volunteers are often busy and distracted. There's no one who
> > can handle or should have to handle the responsibilities of patch approval.
>
> This isn't correct. The MAINTAINERS file reads:
>
> Various Maintainers
>
> Note individuals who maintain parts of the debugger need approval to
> check in changes outside of the immediate domain that they maintain.
>
> If there is no maintainer for a given domain then the responsibility
> falls to a global maintainer.
>
> If there are several maintainers for a given domain then
> responsibility falls to the first maintainer. The first maintainer is
> free to devolve that responsibility among the other maintainers.
>
> so it makes definite references to `responsibility'. Put simply, `the
> buck stops'. Perhaps you should ask a different question `is the buck
> ignored?' and what should be done if/when it is.
>
> Even though GDB is developed by volunteers, it is entirely reasonable
> to expect the volunteers to take their duties seriously. Monitoring
> the bug data base; approving patches; where helpful, contributing to
> discussions. Now that I typically file patches into the bug database,
> this should be much easier.
>
> However, what is unreasonable is to place un-realistic expectations on
> the said maintainers. The assumption that a volunteer is working 24x7
> and living in north america isn't, for instance, reasonable. A
> significant number of GDB maintainers have real jobs and real
> timezones.

[North America doesn't have real jobs and real timezones? They're just
different...] OK, I withdraw my comment about no one having
responsibility. Let me try from another side.

The volunteers do take their duty seriously, but with the density of
responsibility involved [i.e., when a particular person or people are
responsible for a decision, there tend to be just one or two of them]
it's very hard to make forward progress when people are busy. And
we're all busy people. Significant patches almost always take three
weeks or more if they can't be approved by their author; some take
months, even with repeated prodding.

We don't just have a bottleneck, we have a bottleneck per maintainer.

> > Some other GNU projects have a similar setup and don't have this problem.
> > GCC and binutils are my usual examples. How do they avoid it? They have a
> > different definition of global maintainer. That's what ours used to be
> > called - Blanket Write Privileges. The system works a little differently:
> > - Maintainers for specific areas of the compiler can commit/approve
> > patches to the areas they maintain without buy-in from a blanket
> > maintainer.
> > - Blanket maintainers can commit/approve patches anywhere without buy-in
> > from a specific area maintainer.
>
> `write' was a simple typo. From the outset (i.e., from when I got
> handed this job) it was made very clear that neither I, nor any other
> global maintainer, could override the decision of a specific
> maintainer.
>
> This was, and still is, very important. It ensures that the area has
> a certain level of continuity, and it ensures that I (or another
> global maintainer) doesn't ride gunshot over a specific maintainer.

Which raises two thoughts in my mind.
- What are the benefits of this continuity, and do they really
outweigh the disadvantages (slow progress, primarily)? My position is
that they do _not_.
- Why do we need rules to prevent global maintainers riding gunshot
over area maintainers? If people are willing (AND ABLE - people's
circumstances, however regrettable, aren't relevant to the GDB project
as a whole) to maintain their area responsively, and if the global
maintainers are reasonable people, there are few problems and problems
can be easily resolved.

> > [I hope Richard will forgive me for using him as an example and for putting
> > words in his mouth...] This doesn't replace common sense - you generally
> > won't find Richard Henderson approving patches to the C++ frontend, because:
> > - He knows he isn't familiar with it
> > - He knows it has an active set of maintainers at all times
> >
> > Similarly, just because he can check in patches to any target backend, that
> > doesn't mean he won't ask a target maintainer to look over it first. If
> > someone objects to a patch in their area, he would generally not just check
> > it in anyway. If they object to it after he checks it in, the two will
> > discuss the problem like reasonable people and come to some agreement.
>
> The interaction dynamics of a committed patch are very different to
> those of an unapproved patch. Once a patch is committed, the person
> with objections is, unequestionably, put in the back seat.

I disagree. Especially in the presence of a reversion policy.

> > Some noticeable differences between these two models:
> > - In the GCC model, more people are able/likely to check in patches which
> > break things.
> > - But in the GCC model, more people are able/likely to check in patches to
> > fix it afterwards.
>
> (ROFL.)
>
> The GCC model involves a number of development phases and the above
> comments would only relate to one of those phases. At other times
> increasingly strict controls are placed on what can be
> committed/approved. The GCC group spend a significant (out of
> control?) amount of their time trying to re-stablize GCC for their
> releases.

That's not accurate. First of all, the comments relate to all three of
the phases in the GCC development process, although the effect is
mostly damped down in the third phase by the release manager's hand.
Even in stage 3, other maintainers are free to exercise their
judgement.

Also, the GCC tree spends most of its time in a more useful state than
the above really portrays. Things tend to work. The restabilization
is 75% a function of less-frequently-tested platforms - and you can't
realistically claim that the stability of GDB development keeps THEM
working, since we have evidence to the contrary for each of the last
several releases.

> For GDB, on the other hand, interesting development can and does get
> approved/committed at any time. GDB snaps are of such quality that we
> can confidently refer someone to current sources for fixes (except
> when I have a bad day like today :-). Further, instead of using
> official releases (and as you yourself have done) arbitrary snaps can
> even make their way into a distro.

[Red Hat does this too, by the way.] Having done it, I've resolved to
avoid it in the future. Costs outweighed benefits.

> > - Because more people have the privilege of approving a given patch,
> > and fewer people's approvals are needed for any particular patch,
> > patches (usually) get approved more quickly.
> > - Development can happen more quickly, and does not get slowed to a
> > standstill when (say) one of us is pulled off of community GDB work for
> > an urgent customer project. This happens all the time - I've never seen
> > all the GDB maintainers with time for GDB at the same time.
>
> The only way I've got my patches into GCC is by acting as a champion
> for their cause. Exactly like Elena described. Having worked on
> other open source projects I'd have to conclude the same also applied
> to them.

Let me put it this way. When I initially became a GDB developer, I was
in the process of merging back MontaVista patches to the community for
all of the toolchain packages. For GCC and binutils, I ended up with
(eventually) write-after-approval, which means exactly nothing.
Depending on the patch, I have to nag people about them for a day or
several. The only way I could get patches into GDB efficiently enough
for it to be worth my time was to invest a great deal of effort into
the GDB project; to become one of the maintainers in question. Should
that really be necessary?

--
Daniel Jacobowitz
MontaVista Software Debian GNU/Linux Developer