A new strategy for internals documentation

A new strategy for internals documentation

[Stan Shebs]

Although GDB has had an internals manual for many years, nobody has
been especially happy with it.  It is perennially out of date,
sometimes majorly so, and it is quite incomplete.  With the passage of
time, the whole idea of an internals manual has come to seem like a
throwback to the old days, when the prospective GDB hacker's
information had to come from either printed books, or tape archives
downloaded from an FTP site.

Various ideas for how to fix have been floating around, generally with
the goal of improving the manual somehow, but nothing has developed
into action.

I propose a two-pronged strategy, first migrating the internals manual
to the wiki, and then introducing Doxygen for source code
documentation.

*** Migration of gdbint.texinfo to wiki

The first proposed step is to pull the manual into the wiki and retire
it from the sources.  gdbint.texinfo includes a variety of classes of
info, and not counting the obsolete parts, they generally fall into
these groups:

* General description of architecture and algorithms

Each of these sections will work well as an individual wiki article.

* How-tos and tutorials, such as for language and target ports

Likewise, these sections will make good wiki articles.

* Procedures and standards, such as for releases and end of year

These evolve as the release manager and other owners see fit, and the
wiki's dynamic style is a better fit for keeping them up to date.

* Cross-references to more info

Much of this is redundant with information already on the GDB website
or in the wiki.

* Official description of internal API

This material has a different character, in that it is closely tied to
the current state of the sources.  In theory, this should be getting
updated in sync with source changes, but in practice, updates are
rare.  Worse, proposals to auto-generate any of it from the (GPLed)
sources run afoul of the incompatible GFDL that governs the wiki and
the manuals.  I expect that this category will not do any better as
wiki pages, and will eventually be deleted, or made into links to the
sources or Doxygen pages.

* Migration mechanics

At the mechanical level, I suggest a mostly-automatic process in which
each node of gdbint.texinfo becomes a separate wiki page, with the
navigation links preserved as wiki links.  Subsequently wiki editors
can remove redundant or obsolete material, and the wiki change history
will reflect that process.

*** Doxygen

As a second step, we should adopt Doxygen for the sources, and use it
to generate material for a new area of the website, which will be the
detailed documentation of GDB internals.

While Doxygen is not commonly used by FSF projects, it is widespread
in the open-source world.  GNU Radio and libg++ are two well-known FSF
projects using Doxygen now.

The initial policy for Doxygen usage should be to allow but not
require it for new checkins, while at the same time developing the
infrastructure that builds online documentation.  Later policy changes
could make it required for particular cases, such as for cross-file or
"official" internal API.

*** Upsides

Having the narrative and descriptive material as a pile of wiki page
will make it easier to tinker with incrementally.  Changes won't go
through a formal patch review process, which is OK because the pages
are more informal and tutorial rather than authoritative.
(Knowledgeable GDBers should still monitor edits and fix any serious
errors that creep in.)

Conversely, Doxygen in the sources will bring additional rigor and
detail to a chronically weak aspect of GDB's code, namely, how the
different parts are supposed to work with each other.

A more subtle advantage of the dual strategy is that it allows us room
to experiment with approaches.  If the wiki proves popular, then we
could end up with a sizeable and useful "GDBedia" of articles
describing different parts of the debugger in detail.  Similarly, if
Doxygen proves popular, then we might see the Doxygen section grow to
include not just the bare documentation, but a collection of
instructions and howtos.  (And if neither prove popular, we're not
any worse off than we are now. :-) )

*** Downsides

Maybe somebody will be bummed that gdbint.texinfo is gone.

As a texinfo document, it has always been possible to produce printed
versions, info files, PDF files, etc, and we will lose that
capability.  However, while this is important for the main GDB manual,
there is no evidence that any GDB hacker uses the internals manual in
these formats as guidance when working on GDB, nor do I know of
anybody selling a printed internals manual.

The licensing requires us to be careful about migrating pieces of
text; material from gdbint.texinfo can be copied to the wiki, but not
into the sources.  This is not purely hypothetical, as for instance
the gdbint.texinfo description of i386_insert_watchpoint is more
detailed than the comments currently in the source code.  In such a
case, it is allowable to refer to the gdbint description when
expanding on the comments in the source code, but not to copy
verbatim.  However, I believe the amount of material that would need
to be rewritten in this way is small, perhaps a dozen pages or so,
requiring only a few hours of work.

The existence of two places to record information may become a dilemma
for contributors.  Strictly speaking, the dilemma exists now, but in
the absence of a internals documentation strategy, we have not
insisted on much beyond a few comments in source code.  If there is
much confusion, we may need to lay out a more precise policy about
where specific kinds of information should go.

Stan
stan@codesourcery.com

Re: A new strategy for internals documentation

[Eli Zaretskii]

> Date: Tue, 06 Aug 2013 15:26:34 -0700
> From: Stan Shebs <stanshebs@earthlink.net>
> 
> Although GDB has had an internals manual for many years, nobody has
> been especially happy with it.  It is perennially out of date,
> sometimes majorly so, and it is quite incomplete.  With the passage of
> time, the whole idea of an internals manual has come to seem like a
> throwback to the old days, when the prospective GDB hacker's
> information had to come from either printed books, or tape archives
> downloaded from an FTP site.
> 
> Various ideas for how to fix have been floating around, generally with
> the goal of improving the manual somehow, but nothing has developed
> into action.
> 
> I propose a two-pronged strategy, first migrating the internals manual
> to the wiki, and then introducing Doxygen for source code
> documentation.

It is customary to save the important issues to the end, but I'd like
to break that custom and ask this now: do we even want to document the
internals?  This is a serious question: over the years, I've had my
share of trying to convince the main contributors to improve the
internals manual, and the result was always the same: people are
generally happy with the commentary in the code and don't feel any
need to go any further.

So how just declaring gdbint.texinfo dead and deleting it altogether?
Why should we waste any breath arguing about a creature that is not
loved by anyone?

Having said that (and I do suggest to discuss this main issue first),
here are some comments about specific points of the proposal.

> *** Migration of gdbint.texinfo to wiki

Wiki is a bad idea, because there's virtually no control on the
quality of the information there.

> * Procedures and standards, such as for releases and end of year
> 
> These evolve as the release manager and other owners see fit, and the
> wiki's dynamic style is a better fit for keeping them up to date.

That is a strange opinion, given the ease of use of today's VCSs.
What exactly makes it hard to keep this part up to date now?

> * Cross-references to more info
> 
> Much of this is redundant with information already on the GDB website
> or in the wiki.

If you renounce cross-references, you in effect are saying that
hyperlinks, which IMO are one of the most important inventions in
on-line docs, are useless and should not be considered important at
all.  That is a strange opinion, indeed.

> proposals to auto-generate any of [internal API docs] from the (GPLed)
> sources run afoul of the incompatible GFDL that governs the wiki and
> the manuals.

That's not correct.  Generation of a Texinfo manual from program
sources is well established, and used in libiberty, binutils, and
elsewhere.  The licensing issue is AFAIU a red herring.

> *** Doxygen
> 
> As a second step, we should adopt Doxygen for the sources, and use it
> to generate material for a new area of the website, which will be the
> detailed documentation of GDB internals.

AFAIK, Doxygen cannot generate an Info manual, only HTML.  If so, this
would be a serious downside that you don't mention.

If we want to keep the documentation in the sources, why not use the
same system as libiberty?

> *** Upsides
> 
> Having the narrative and descriptive material as a pile of wiki page
> will make it easier to tinker with incrementally.  Changes won't go
> through a formal patch review process, which is OK because the pages
> are more informal and tutorial rather than authoritative.

Not having a review process for documentation is an "upside"
because?...

> (Knowledgeable GDBers should still monitor edits and fix any serious
> errors that creep in.)

Why should "Knowledgeable GDBers" do the job of whoever submits the
patch?  Do we have a lot of free time on our hands to do that?  Patch
review is a well established process in GDB maintenance and
development, and AFAICS it works reasonably well, certainly in the
area of the documentation.  Why would we consider getting rid of it?

> Conversely, Doxygen in the sources will bring additional rigor and
> detail to a chronically weak aspect of GDB's code, namely, how the
> different parts are supposed to work with each other.

This is precisely the problem with documenting in comments, but (see
above) people seem to be happy with it.  So I don't see how Doxygen
will change any of that.

> As a texinfo document, it has always been possible to produce printed
> versions, info files, PDF files, etc, and we will lose that
> capability.  However, while this is important for the main GDB manual,
> there is no evidence that any GDB hacker uses the internals manual in
> these formats as guidance when working on GDB, nor do I know of
> anybody selling a printed internals manual.

That's tautology: since the manual is terribly outdated, why would
someone use it?

> The licensing requires us to be careful about migrating pieces of
> text; material from gdbint.texinfo can be copied to the wiki, but not
> into the sources.  This is not purely hypothetical, as for instance
> the gdbint.texinfo description of i386_insert_watchpoint is more
> detailed than the comments currently in the source code.

Because the author for some strange reason believed that the
documentation should be in the manual rather than in the sources.  But
that author is still among us, so you could ask him politely whether
he would like to consider relicensing the stuff.

> In such a case, it is allowable to refer to the gdbint description
> when expanding on the comments in the source code, but not to copy
> verbatim.  However, I believe the amount of material that would need
> to be rewritten in this way is small, perhaps a dozen pages or so,
> requiring only a few hours of work.

Most of the authors are still available, you can ask them about
relicensing.

> The existence of two places to record information may become a dilemma
> for contributors.

No one will want to update two places.  Heck, even with one we have
difficulties.

Re: A new strategy for internals documentation

[Stan Shebs]

On 8/6/13 9:28 PM, Eli Zaretskii wrote:
>> Date: Tue, 06 Aug 2013 15:26:34 -0700
>> From: Stan Shebs <stanshebs@earthlink.net>
>>
>>
>> I propose a two-pronged strategy, first migrating the internals manual
>> to the wiki, and then introducing Doxygen for source code
>> documentation.
> 
> It is customary to save the important issues to the end, but I'd like
> to break that custom and ask this now: do we even want to document the
> internals?  This is a serious question: over the years, I've had my
> share of trying to convince the main contributors to improve the
> internals manual, and the result was always the same: people are
> generally happy with the commentary in the code and don't feel any
> need to go any further.
> 
> So how just declaring gdbint.texinfo dead and deleting it altogether?

Isn't that going to be the end state of what I'm proposing? :-) I just
added a path to conserve any bits that seem useful.

People do want internals documentation, we hear grumbles about it on a
weekly basis.  But, having made a couple runs at updating the internals
manual myself, it's a daunting prospect, and always unclear what its
scope and detail level should be.  I'm not ready to give up on the
idea of internals documentation altogether, since we do have other
options we can try.

>> *** Migration of gdbint.texinfo to wiki
> 
> Wiki is a bad idea, because there's virtually no control on the
> quality of the information there.

That's true now, but I don't think I've heard of anybody being misled by
poor information on the GDB wiki.

>> * Procedures and standards, such as for releases and end of year
>>
>> These evolve as the release manager and other owners see fit, and the
>> wiki's dynamic style is a better fit for keeping them up to date.
> 
> That is a strange opinion, given the ease of use of today's VCSs.
> What exactly makes it hard to keep this part up to date now?

If nothing else, ensuring that "make info" doesn't break because
you forgot an @-sign. :-)

>> * Cross-references to more info
>>
>> Much of this is redundant with information already on the GDB website
>> or in the wiki.
> 
> If you renounce cross-references, you in effect are saying that
> hyperlinks, which IMO are one of the most important inventions in
> on-line docs, are useless and should not be considered important at
> all.  That is a strange opinion, indeed.

I just meant that there are sections of the internals manual saying "go
here for GNU coding standards", "go here to post a bug report", that
sort ofthing.  We have multiple copies of all that verbiage elsewhere now.

>> proposals to auto-generate any of [internal API docs] from the (GPLed)
>> sources run afoul of the incompatible GFDL that governs the wiki and
>> the manuals.
> 
> That's not correct.  Generation of a Texinfo manual from program
> sources is well established, and used in libiberty, binutils, and
> elsewhere.  The licensing issue is AFAIU a red herring.

True, if you're not mixing anything.  In the discussion that started
with http://www.sourceware.org/ml/gdb/2009-10/msg00208.html , there
seemed to be valid concerns about how things could be intermixed, and I
note that the libg++ documentation has a specific disclaimer that the
auto-generated parts of its docs are GPL and separate from the GFDL parts.

>> *** Doxygen
>>
>> As a second step, we should adopt Doxygen for the sources, and use it
>> to generate material for a new area of the website, which will be the
>> detailed documentation of GDB internals.
> 
> AFAIK, Doxygen cannot generate an Info manual, only HTML.  If so, this
> would be a serious downside that you don't mention.

Hmm, I'm perhaps prejudiced since I haven't looked at any info file in
probably a decade.  Are many people using them still?

> If we want to keep the documentation in the sources, why not use the
> same system as libiberty?

It's plausible, but idiosyncratic.  How much development does it get,
vs Doxygen?

> 
>> *** Upsides
>>
>> Having the narrative and descriptive material as a pile of wiki page
>> will make it easier to tinker with incrementally.  Changes won't go
>> through a formal patch review process, which is OK because the pages
>> are more informal and tutorial rather than authoritative.
> 
> Not having a review process for documentation is an "upside"
> because?...

I think it's time-consuming overkill for tutorials and howtos.  An
underlying implicit goal here is to encourage contribution, not think of
ways to chase people away.  Sure, for the official user manual, we need
to be careful about what it says, but a "hey, here's how I like to debug
signal handlers" shouldn't need anybody else to decide whether it's
allowed or not.

>> Conversely, Doxygen in the sources will bring additional rigor and
>> detail to a chronically weak aspect of GDB's code, namely, how the
>> different parts are supposed to work with each other.
> 
> This is precisely the problem with documenting in comments, but (see
> above) people seem to be happy with it.  So I don't see how Doxygen
> will change any of that.

It might not; it would be an experiment.  Lots of hackers like it
though, so there's at least some empirical evidence in its favor.

>> The licensing requires us to be careful about migrating pieces of
>> text; material from gdbint.texinfo can be copied to the wiki, but not
>> into the sources.  This is not purely hypothetical, as for instance
>> the gdbint.texinfo description of i386_insert_watchpoint is more
>> detailed than the comments currently in the source code.
> 
> Because the author for some strange reason believed that the
> documentation should be in the manual rather than in the sources.  But
> that author is still among us, so you could ask him politely whether
> he would like to consider relicensing the stuff.

Yes, in theory I could re-contribute my bits, but I really really don't
want to spend hours of debate on whether the licenses, or FSF ownership,
even allow for that, or if there if some obscure obstacle.
There are some bits of value in the old manual, but not enough to
justify sucking everybody into contemplation of arcane legal points.
I'd rather just leave it at "don't go there". :-)

Stan
stan@codesourcery.com

Re: A new strategy for internals documentation

[Yao Qi]

On 08/07/2013 12:28 PM, Eli Zaretskii wrote:
> internals manual, and the result was always the same: people are
> generally happy with the commentary in the code and don't feel any
> need to go any further.

My personal feeling is that it is harder to edit the internal manual
than to change the .c code.  I feel competent to edit c files to fix a
bug or add a new feature, with some comments in the code and rationale
description in the mail.  However, when it comes to the internal doc, I
don't know how/where to edit because I don't have a global view on both
the internal doc and the source code.  Unfortunately, only few
contributors has such global view.  That may be the reason why some
contributors complain about the internal manual, but fail to post
patches to improve it.

I like the idea that using wiki, which is not very formal, to encourage
contributions on the internal doc, tutorials, howtos, etc.  I had slides 
"Port GDB To A New Processor Architecture: TI C6X" on the GNU Cauldron 
this year, and John Gilmore suggests that "It would be lovely if you 
could improve the Internals manual in the spots where you learned things 
that were not well documented." [1].  It is a very good idea, and I 
checked the internal manual, and tried to improve it.  Finally, I gave 
up, because I can see something is missing here and something is unclear 
there, but I was unable to extract some necessary bits from my slides, 
and add them to the internal doc, it is too hard for me.  Then I'd like 
to convert my slides to several blog posts or wiki pages, which is 
informal, and people also can get benefits from them.

The current Internals is like a book, most of people can't write a book 
or revise a book, but people can write a lot of useful blog or wiki pages.

LLVM is famous for its documentation, go through its web site 
http://llvm.org/docs/, most of them are tutorials and howtos.  Probably 
there is a "LLVM Programmer’s Manual" [2], which is equivalent to GDB 
Internals.  It is quite general, doesn't include much details. 
Contributors and developers can read tutorials and howtos which are 
related to their tasks, and get their works started.

Experienced hackers are too familiar with the code to rely much on the 
internal doc, but newbie contributors need them.  Usually, documentation 
from newbie contributors usually fits the needs of other contributors. 
We need the contributions to the internal doc, and wiki is a good way to 
go, IMO.

-- 
Yao (齐尧)

[1] http://sourceware.org/ml/gdb/2013-07/msg00095.html
[2] http://llvm.org/docs/ProgrammersManual.html

Re: A new strategy for internals documentation

[Eli Zaretskii]

> Date: Wed, 07 Aug 2013 12:58:14 -0700
> From: Stan Shebs <stanshebs@earthlink.net>
> CC: gdb@sourceware.org
> 
> > So how just declaring gdbint.texinfo dead and deleting it altogether?
> 
> Isn't that going to be the end state of what I'm proposing? :-) I just
> added a path to conserve any bits that seem useful.

My way is faster and easier.

> People do want internals documentation, we hear grumbles about it on a
> weekly basis.

The grumbles come from people other than those who can provide the
documentation.  And the latter don't think we have a problem in the
first place.

> >> *** Migration of gdbint.texinfo to wiki
> > 
> > Wiki is a bad idea, because there's virtually no control on the
> > quality of the information there.
> 
> That's true now, but I don't think I've heard of anybody being misled by
> poor information on the GDB wiki.

Again, if we don't care about the documentation, then of course we
shouldn't care about poor information.  If we do care, then wiki is a
way to waste resources at best.

> > What exactly makes it hard to keep this part up to date now?
> 
> If nothing else, ensuring that "make info" doesn't break because
> you forgot an @-sign. :-)

How is this different from a typo in the code?

> >> proposals to auto-generate any of [internal API docs] from the (GPLed)
> >> sources run afoul of the incompatible GFDL that governs the wiki and
> >> the manuals.
> > 
> > That's not correct.  Generation of a Texinfo manual from program
> > sources is well established, and used in libiberty, binutils, and
> > elsewhere.  The licensing issue is AFAIU a red herring.
> 
> True, if you're not mixing anything.  In the discussion that started
> with http://www.sourceware.org/ml/gdb/2009-10/msg00208.html , there
> seemed to be valid concerns about how things could be intermixed, and I
> note that the libg++ documentation has a specific disclaimer that the
> auto-generated parts of its docs are GPL and separate from the GFDL parts.

Red herring.  I asked RMS about this once, and the answer was there
was no problem.

> > AFAIK, Doxygen cannot generate an Info manual, only HTML.  If so, this
> > would be a serious downside that you don't mention.
> 
> Hmm, I'm perhaps prejudiced since I haven't looked at any info file in
> probably a decade.  Are many people using them still?

I do, quite a lot.  Besides, Info is the official format of GNU
documentation, you will have hard time convincing the FSF to demote an
existing manual.

> > If we want to keep the documentation in the sources, why not use the
> > same system as libiberty?
> 
> It's plausible, but idiosyncratic.  How much development does it get,
> vs Doxygen?

Why do you need development for comments?

> > Not having a review process for documentation is an "upside"
> > because?...
> 
> I think it's time-consuming overkill for tutorials and howtos.  An
> underlying implicit goal here is to encourage contribution, not think of
> ways to chase people away.

The net result will be that the documentation will be unreadable.  Not
everybody who writes good code can write good documentation.

> >> The licensing requires us to be careful about migrating pieces of
> >> text; material from gdbint.texinfo can be copied to the wiki, but not
> >> into the sources.  This is not purely hypothetical, as for instance
> >> the gdbint.texinfo description of i386_insert_watchpoint is more
> >> detailed than the comments currently in the source code.
> > 
> > Because the author for some strange reason believed that the
> > documentation should be in the manual rather than in the sources.  But
> > that author is still among us, so you could ask him politely whether
> > he would like to consider relicensing the stuff.
> 
> Yes, in theory I could re-contribute my bits, but I really really don't
> want to spend hours of debate on whether the licenses, or FSF ownership,
> even allow for that, or if there if some obscure obstacle.

Another red herring, AFAIK.  I can decide to distribute the text that
I wrote under any license I want, as long as I don't preclude the FSF
to distribute that text in the GNU manuals under GFDL.  The fact that
I assigned the copyright to the FSF for the manual just means that the
FSF is free to distribute that text in that manual.  But it doesn't in
any way preclude me to distribute the same text in other ways.  IOW,
assignment of copyright doesn't mean the FSF now owns the text.

Re: A new strategy for internals documentation

[Eli Zaretskii]

> Date: Thu, 8 Aug 2013 11:44:52 +0800
> From: Yao Qi <yao@codesourcery.com>
> CC: Stan Shebs <stanshebs@earthlink.net>, <gdb@sourceware.org>
> 
> The current Internals is like a book, most of people can't write a book 
> or revise a book, but people can write a lot of useful blog or wiki pages.

I'm sorry, but the result will surely be an unorganized heap of useful
tips, inconsistent in style, certainly contradicting in some places,
and impossible to read more than one tip at a time.

If this is the kind of documentation we want, let's just delete
gdbint.texinfo and declare that we no longer have internals docs as
part of the GDB project.  Why disguise this by talking about wiki?

Re: A new strategy for internals documentation

[Tom Tromey]

>>>>> "Stan" == Stan Shebs <stanshebs@earthlink.net> writes:

Stan> Although GDB has had an internals manual for many years, nobody has
Stan> been especially happy with it.  It is perennially out of date,
Stan> sometimes majorly so, and it is quite incomplete.  With the passage of
Stan> time, the whole idea of an internals manual has come to seem like a
Stan> throwback to the old days, when the prospective GDB hacker's
Stan> information had to come from either printed books, or tape archives
Stan> downloaded from an FTP site.

Stan> Various ideas for how to fix have been floating around, generally with
Stan> the goal of improving the manual somehow, but nothing has developed
Stan> into action.

Stan> I propose a two-pronged strategy, first migrating the internals manual
Stan> to the wiki, and then introducing Doxygen for source code
Stan> documentation.

It all sounds good to me.  Please get started :-)

Tom

Re: A new strategy for internals documentation

[Tom Tromey]

Eli> So how just declaring gdbint.texinfo dead and deleting it altogether?
Eli> Why should we waste any breath arguing about a creature that is not
Eli> loved by anyone?

Good idea.  Let's do it.

Tom

Re: A new strategy for internals documentation

[Doug Evans]

On Tue, Aug 6, 2013 at 9:28 PM, Eli Zaretskii <eliz@gnu.org> wrote:
>> Date: Tue, 06 Aug 2013 15:26:34 -0700
>> From: Stan Shebs <stanshebs@earthlink.net>
>>
>> Although GDB has had an internals manual for many years, nobody has
>> been especially happy with it.  It is perennially out of date,
>> sometimes majorly so, and it is quite incomplete.  With the passage of
>> time, the whole idea of an internals manual has come to seem like a
>> throwback to the old days, when the prospective GDB hacker's
>> information had to come from either printed books, or tape archives
>> downloaded from an FTP site.
>>
>> Various ideas for how to fix have been floating around, generally with
>> the goal of improving the manual somehow, but nothing has developed
>> into action.
>>
>> I propose a two-pronged strategy, first migrating the internals manual
>> to the wiki, and then introducing Doxygen for source code
>> documentation.
>
> It is customary to save the important issues to the end, but I'd like
> to break that custom and ask this now: do we even want to document the
> internals?  This is a serious question: over the years, I've had my
> share of trying to convince the main contributors to improve the
> internals manual, and the result was always the same: people are
> generally happy with the commentary in the code and don't feel any
> need to go any further.
>
> So how just declaring gdbint.texinfo dead and deleting it altogether?
> Why should we waste any breath arguing about a creature that is not
> loved by anyone?

Well, there's deleting gdbint.texinfo and replacing it.
[just stating that for clarity's sake]

> Having said that (and I do suggest to discuss this main issue first),
> here are some comments about specific points of the proposal.
>
>> *** Migration of gdbint.texinfo to wiki
>
> Wiki is a bad idea, because there's virtually no control on the
> quality of the information there.

I don't see this as a deciding issue - the quality can be good enough.

>> * Cross-references to more info
>>
>> Much of this is redundant with information already on the GDB website
>> or in the wiki.
>
> If you renounce cross-references, you in effect are saying that
> hyperlinks, which IMO are one of the most important inventions in
> on-line docs, are useless and should not be considered important at
> all.  That is a strange opinion, indeed.

Are cross-references being renounced in their entirety?
A good question would be: what sufficiently important cross-references
will no longer be possible?

>> *** Doxygen
>>
>> As a second step, we should adopt Doxygen for the sources, and use it
>> to generate material for a new area of the website, which will be the
>> detailed documentation of GDB internals.
>
> AFAIK, Doxygen cannot generate an Info manual, only HTML.  If so, this
> would be a serious downside that you don't mention.
>
> If we want to keep the documentation in the sources, why not use the
> same system as libiberty?

Does any developer like that system? [honest question]

>> *** Upsides
>>
>> Having the narrative and descriptive material as a pile of wiki page
>> will make it easier to tinker with incrementally.  Changes won't go
>> through a formal patch review process, which is OK because the pages
>> are more informal and tutorial rather than authoritative.
>
> Not having a review process for documentation is an "upside"
> because?...
>
>> (Knowledgeable GDBers should still monitor edits and fix any serious
>> errors that creep in.)
>
> Why should "Knowledgeable GDBers" do the job of whoever submits the
> patch?  Do we have a lot of free time on our hands to do that?  Patch
> review is a well established process in GDB maintenance and
> development, and AFAICS it works reasonably well, certainly in the
> area of the documentation.  Why would we consider getting rid of it?

A good question to ask here is given that knowledgeable GDBers will
either review the doc and exchange several emails until it can be
checked in, or fix it after it's committed (e.g. to wiki), which is an
overall better use of time?

>> Conversely, Doxygen in the sources will bring additional rigor and
>> detail to a chronically weak aspect of GDB's code, namely, how the
>> different parts are supposed to work with each other.
>
> This is precisely the problem with documenting in comments, but (see
> above) people seem to be happy with it.  So I don't see how Doxygen
> will change any of that.

One real problem we have (IMHO) is knowing what the various "modules"
(for lack of a better word) of gdb are and what their *public* APIs
are.  Can Doxygen help with that?  E.g. can part of the generated
output be a chapter (or some such) on what the modules are and their
APIs?
[honest question]

Re: A new strategy for internals documentation

[Doug Evans]

On Thu, Aug 8, 2013 at 10:26 AM, Eli Zaretskii <eliz@gnu.org> wrote:
>> Date: Wed, 07 Aug 2013 12:58:14 -0700
>> From: Stan Shebs <stanshebs@earthlink.net>
>> CC: gdb@sourceware.org
>>
>> > So how just declaring gdbint.texinfo dead and deleting it altogether?
>>
>> Isn't that going to be the end state of what I'm proposing? :-) I just
>> added a path to conserve any bits that seem useful.
>
> My way is faster and easier.

Possibly alright, but what final result do we want?
[What are we working towards?]

>> People do want internals documentation, we hear grumbles about it on a
>> weekly basis.
>
> The grumbles come from people other than those who can provide the
> documentation.  And the latter don't think we have a problem in the
> first place.

If the latter includes me I disagree.
[I think the latter includes me, IIUC, but I could be wrong.]

>> >> *** Migration of gdbint.texinfo to wiki
>> >
>> > Wiki is a bad idea, because there's virtually no control on the
>> > quality of the information there.
>>
>> That's true now, but I don't think I've heard of anybody being misled by
>> poor information on the GDB wiki.
>
> Again, if we don't care about the documentation, then of course we
> shouldn't care about poor information.  If we do care, then wiki is a
> way to waste resources at best.

I disagree (that the wiki is a way to waste resources at best).

>> > AFAIK, Doxygen cannot generate an Info manual, only HTML.  If so, this
>> > would be a serious downside that you don't mention.
>>
>> Hmm, I'm perhaps prejudiced since I haven't looked at any info file in
>> probably a decade.  Are many people using them still?
>
> I do, quite a lot.  Besides, Info is the official format of GNU
> documentation, you will have hard time convincing the FSF to demote an
> existing manual.
>
>> > If we want to keep the documentation in the sources, why not use the
>> > same system as libiberty?
>>
>> It's plausible, but idiosyncratic.  How much development does it get,
>> vs Doxygen?
>
> Why do you need development for comments?

He's referring to development of the comment->doc generator.
[right?]

>> > Not having a review process for documentation is an "upside"
>> > because?...
>>
>> I think it's time-consuming overkill for tutorials and howtos.  An
>> underlying implicit goal here is to encourage contribution, not think of
>> ways to chase people away.
>
> The net result will be that the documentation will be unreadable.  Not
> everybody who writes good code can write good documentation.

OTOH, It's easier to improve documentation over time.
[it's more important to get code right sooner than later]

Re: A new strategy for internals documentation

[Eli Zaretskii]

> Date: Thu, 8 Aug 2013 14:07:51 -0700
> From: Doug Evans <dje@google.com>
> Cc: Stan Shebs <stanshebs@earthlink.net>, gdb <gdb@sourceware.org>
> 
> On Thu, Aug 8, 2013 at 10:26 AM, Eli Zaretskii <eliz@gnu.org> wrote:
> >> Date: Wed, 07 Aug 2013 12:58:14 -0700
> >> From: Stan Shebs <stanshebs@earthlink.net>
> >> CC: gdb@sourceware.org
> >>
> >> > So how just declaring gdbint.texinfo dead and deleting it altogether?
> >>
> >> Isn't that going to be the end state of what I'm proposing? :-) I just
> >> added a path to conserve any bits that seem useful.
> >
> > My way is faster and easier.
> 
> Possibly alright, but what final result do we want?

The same as now: the internals manual is useless.

> > The grumbles come from people other than those who can provide the
> > documentation.  And the latter don't think we have a problem in the
> > first place.
> 
> If the latter includes me I disagree.

Disagree with what, and why?

> > Again, if we don't care about the documentation, then of course we
> > shouldn't care about poor information.  If we do care, then wiki is a
> > way to waste resources at best.
> 
> I disagree (that the wiki is a way to waste resources at best).

It is a waste because nothing good will ever come out of it.  It will
be a heap of notes various people at various times thought it would be
a good idea to share.  You cannot create a coherent document that way.

> > Why do you need development for comments?
> 
> He's referring to development of the comment->doc generator.

Why do we need that developed, if it already does the job?

> > The net result will be that the documentation will be unreadable.  Not
> > everybody who writes good code can write good documentation.
> 
> OTOH, It's easier to improve documentation over time.

Who will do that, and why?  Again, the core developers think that what
we have in the comments is enough, and if it is not enough, the
comments should be improved/expanded.  Why would someone invest
efforts in another resource?

Re: A new strategy for internals documentation

[Stan Shebs]

I'm not sure what we're arguing at this point, so let me see if I can
summarize your views on specific actions:

1. Delete gdbint.texinfo.

We're all agreed on this, right?

2. Add pieces of the manual to the wiki.

You doubt the value of this.  Should we forbid people from adding wiki
pages?

3. Introduce an annotation/structured scheme in source code comments.

You're skeptical, but not opposed to this, right?

4. Use Doxygen.

Are you for or against, or indifferent?

(For me Doxygen gets the nod by elimination, if nothing else.  In the
rather lengthy

http://en.wikipedia.org/wiki/Comparison_of_documentation_generators

there are not a lot of options that are portable, GPL, etc.  LLVM's use
of Doxygen, http://llvm.org/doxygen/index.html , seems pretty useful.)

Stan
stan@codesourcery.com

Re: A new strategy for internals documentation

[Doug Evans]

On Thu, Aug 8, 2013 at 2:55 PM, Eli Zaretskii <eliz@gnu.org> wrote:
>> Date: Thu, 8 Aug 2013 14:07:51 -0700
>> From: Doug Evans <dje@google.com>
>> Cc: Stan Shebs <stanshebs@earthlink.net>, gdb <gdb@sourceware.org>
> [...]
>> > The grumbles come from people other than those who can provide the
>> > documentation.  And the latter don't think we have a problem in the
>> > first place.
>>
>> If the latter includes me I disagree.
>
> Disagree with what, and why?

I disagree with the statement "the latter don't think we have a problem".
We do have a problem: I think our internals documentation needs improving.

>> > Again, if we don't care about the documentation, then of course we
>> > shouldn't care about poor information.  If we do care, then wiki is a
>> > way to waste resources at best.
>>
>> I disagree (that the wiki is a way to waste resources at best).
>
> It is a waste because nothing good will ever come out of it.  It will
> be a heap of notes various people at various times thought it would be
> a good idea to share.  You cannot create a coherent document that way.

I think we'll have to agree to disagree that nothing good will come out of it.
For example, gdb's current wiki pages are useful, and are edited far
more often than gdbint.texinfo (AFAICT).

>> > Why do you need development for comments?
>>
>> He's referring to development of the comment->doc generator.
>
> Why do we need that developed, if it already does the job?

Assuming it doesn't have latent bugs that no one has tripped over yet,
and assuming it does everything we want, now and tomorrow.

>> > The net result will be that the documentation will be unreadable.  Not
>> > everybody who writes good code can write good documentation.
>>
>> OTOH, It's easier to improve documentation over time.
>
> Who will do that, and why?  Again, the core developers think that what
> we have in the comments is enough, and if it is not enough, the
> comments should be improved/expanded.  Why would someone invest
> efforts in another resource?

I'm one that thinks that there is not enough, and that expanding the
comments is not enough.  For one there's a higher level / descriptive
view that's missing with that approach.  Plus the S/N ratio when faced
with reading all the source code is much lower than when able to
browse something generated from the comments in the code.

Re: A new strategy for internals documentation

[John Gilmore]

The real issue is not about where the information is stored.  The real
issue is that people are evolving the code base without evolving the
matching documentation.  This is accepted to be a sin with regard to
the "Debugging with GDB" manual, which IS required to be complete,
definitive, tutorial, and readable.  Could I gently suggest that
contributors who make patches that evolve e.g. the symbol table handling,
but who don't document their improvements in Gdbint, be gently reminded
to improve and resubmit their patch by including a patch to gdbint as well?  

I started the GDB Internals manual.  Why?  Because we had no place to
record textual explanations about why and how GDB is internally
structured.  For example, there was no obvious place to put a list of
"how to add a new target type to GDB".  You had to edit half a dozen
files, in sync, and none of those files was the right place to put the
overview information.  For another example, we had nothing that
described an overview of how symbol table processing works -- how it
gets read in from files, how it gets searched, what jobs the various kinds
of symbol tables in GDB are designed to do (and designed specifically NOT
to do).  I had to maintain that code but had no idea how psymtabs versus
symtabs were supposed to work.  (I could figure out what the code did,
but was that what it was supposed to be doing?)

So as I gradually learned (or created) these things, I put whatever
documentation I had, into the GDB Internals manual.  Frequently a
contribution was the sort that might be sent as an email to the
maintainers list, like "I just added a new host type -- here are all
the places I had to touch."

Gdbint was never intended to be definitive -- because there has seldom
been enough time, or writing ability, to make it definitive.  And it
was never intended to be complete.  It was designed as a net to catch
useful short bits of prose so that future maintainers would be able to
find them.  But it was far better than having nothing.  And it is part
of the source code, so it can evolve along with the source code, and
the version that ships in, and matches, each source release of GDB can
be easily found.

It seems to me that the main arguments for change are:

  *  It's fragmented and inconsistent.

Any replacement for gdbint will also be fragmented and inconsistent, 
unless the team decides it's worth spending the resources to write
a complete and consistent internals manual, which seems unlikely.
Therefore this is not an argument for change.

  *  It's too hard to edit texifo.

Texinfo markup is not significantly easier or harder than Wiki markup.
They do the same basic things.  And anyone who maintains GDB will need
to understand how to edit texinfo anyway, since anyone who adds or removes
a feature has to fix its documentation in the GDB Manual.  So this is
not an argument for change either.

  *  Wikis have a more "dynamic style"

Gdbint is just as dynamic as the GDB source code.  It's stored in a
widely distributed, branch oriented source code control system.  Anyone
who can usefully edit a GDB source file can usefully edit gdbint.texinfo.
On the contrary, to edit the Wiki requires a separate login and permission.

  *  Changes to Gdbint should not go through approval

If the team agrees with this, it's easy enough for any GDB maintainer
to merely approve gdbint changes that are submitted as patches.
(Presumably if they come in along with a substantive code change,
you'd only insert the gdbint change when the code change is also
approved.  Having half the change in the source code and half in a
wiki would complicate that process.)

My suggestion is that if somebody (Stan?) actually wants to put many
hours into improving the situation with respect to GDB internal
documentation, spend your time making the text better, strategically,
rather than wasting time moving info from one format (texinfo) to
another (Wiki markup).  As a senior maintainer, you know what parts
of it are obsolete and why -- insert a FIXME paragraph explaining
that that part is obsolete and why, and encouraging anyone who has
the time and information to update it to match the code.

My experience with Doxygen comments has been that they quickly become
formulaic.  This provides sometimes-useful material about the
arguments of individual functions -- which is best read directly in
the source code rather than in a separate manual -- but seldom
provides an overview of what's actually going on.  It produces an
endless forest of detailed information, when what the reader wants is a few
paragraphs of overview to know where to focus their attention to handle
the job in front of them.

That's what the GDB Internals manual was originally created to do.
If it isn't doing that job, don't just change its format; change the
process of updating the information, so that the information becomes
relevant again.

	John

Re: A new strategy for internals documentation

[John Gilmore]

> 1. Delete gdbint.texinfo.
> 
> We're all agreed on this, right?

I'm not agreed on this, but I hardly count as an active maintainer.
More like a ghost from GDB's past...

	John

PS: I think if y'all have time to mess around converting perfectly
good working C to C++, then surely some time can be spared for making
the doc match the code.  The issue is probably more like "everyone on
the team thinks of writing English prose as work, whereas writing C++
code is fun".

Re: A new strategy for internals documentation

[Eli Zaretskii]

> Date: Thu, 8 Aug 2013 16:04:49 -0700
> From: Doug Evans <dje@google.com>
> Cc: Stan Shebs <stanshebs@earthlink.net>, gdb <gdb@sourceware.org>
> 
> On Thu, Aug 8, 2013 at 2:55 PM, Eli Zaretskii <eliz@gnu.org> wrote:
> >> Date: Thu, 8 Aug 2013 14:07:51 -0700
> >> From: Doug Evans <dje@google.com>
> >> Cc: Stan Shebs <stanshebs@earthlink.net>, gdb <gdb@sourceware.org>
> > [...]
> >> > The grumbles come from people other than those who can provide the
> >> > documentation.  And the latter don't think we have a problem in the
> >> > first place.
> >>
> >> If the latter includes me I disagree.
> >
> > Disagree with what, and why?
> 
> I disagree with the statement "the latter don't think we have a problem".
> We do have a problem: I think our internals documentation needs improving.

Then you seem to belong to the same minority as I do.

> >> > Why do you need development for comments?
> >>
> >> He's referring to development of the comment->doc generator.
> >
> > Why do we need that developed, if it already does the job?
> 
> Assuming it doesn't have latent bugs that no one has tripped over yet,
> and assuming it does everything we want, now and tomorrow.

What is good enough for libiberty and binutils ought to be good enough
for us.

> I'm one that thinks that there is not enough, and that expanding the
> comments is not enough.  For one there's a higher level / descriptive
> view that's missing with that approach.  Plus the S/N ratio when faced
> with reading all the source code is much lower than when able to
> browse something generated from the comments in the code.

I think the same, but others don't, as was demonstrated numerous times
in past discussions.

Re: A new strategy for internals documentation

[Eli Zaretskii]

> cc: Yao Qi <yao@codesourcery.com>, stanshebs@earthlink.net, gdb@sourceware.org
> Comments: In-reply-to Eli Zaretskii <eliz@gnu.org>
>    message dated "Thu, 08 Aug 2013 20:30:24 +0300."
> Date: Thu, 08 Aug 2013 18:29:58 -0700
> From: John Gilmore <gnu@toad.com>
> 
> The real issue is not about where the information is stored.  The real
> issue is that people are evolving the code base without evolving the
> matching documentation.

Maybe it was so back when you started with this manual.  It is no
longer the case.  The real issue with this manual now is that it is
profoundly incomplete, so much so that some of the most important
parts of the GDB internals are not described at all, or their
description is just a listing of APIs without any glue.

By contrast, what _is_ there is being updated as GDB is developed, as
part of the development, and changes in GDB that invalidate what's in
the manual are accompanied by suitable changes to the manual.

>   *  Changes to Gdbint should not go through approval
> 
> If the team agrees with this, it's easy enough for any GDB maintainer
> to merely approve gdbint changes that are submitted as patches.

The approval part is a non-issue: I usually review patches to
documentation within hours of the RFA.  Note that people who mentioned
this issue didn't talk about approval, they talked about the writing
process itself.

> If [the internals manual] isn't doing that job, don't just change
> its format; change the process of updating the information, so that
> the information becomes relevant again.

People want _zero_ process on its behalf.  I tried to convince them
otherwise, offering help in many ways, including those you mentioned,
but you cannot convince someone to invest any effort when they want to
invest none.  So I gave up.  After all, a project in general, and its
documentation in particular, cannot be better than what the project's
community wants it to be.

Once again, the main problem (and some will say it's not a problem at
all) is that the majority of GDB developers don't see any need in this
manual's existence, or in any documentation of the internals besides
what's in the comments.  None whatsoever.  Until this changes, there's
no hope in improving the internals' documentation, and no need to
invest any additional effort in any replacements.

Re: A new strategy for internals documentation

[Eli Zaretskii]

> Date: Thu, 08 Aug 2013 16:02:58 -0700
> From: Stan Shebs <stanshebs@earthlink.net>
> CC: Doug Evans <dje@google.com>, gdb@sourceware.org
> 
> I'm not sure what we're arguing at this point

I'm done arguing.  I regret I responded to your original message,
because all that did is cause me aggravation and open old wounds.
(Yes, it hurts to try to contribute to a community only to see the
attempts rejected time and again.)

> so let me see if I can summarize your views on specific actions:

Let me do it for you:

 . delete gdbint.texinfo (and the Makefile rules to go with it)

 . don't ever bother me again with any documentation of the GDB
   internals, because, following the 1st step above, I will resign
   from my responsibilities regarding (mis)documentation of the
   internals

 . you want to have a wiki or Doxygen comments or whatever, you don't
   need my approval anymore, see above; I was never asked to review
   code comments until now anyway

One last bit is that changing the format of a GNU manual might need to
be communicated to the FSF, I think.  Or maybe a majority vote by the
SC will be enough, I don't know (and don't care).

OK?

Re: A new strategy for internals documentation

[Mark Kettenis]

> Date: Thu, 08 Aug 2013 16:02:58 -0700
> From: Stan Shebs <stanshebs@earthlink.net>
> 
> 3. Introduce an annotation/structured scheme in source code comments.
> 
> You're skeptical, but not opposed to this, right?

It's not going to make the source code comments more readable.  Will
probably make people focus on getting the sybtax of the annotations
right instead of the actual context.

> 4. Use Doxygen.
> 
> Are you for or against, or indifferent?
> 
> (For me Doxygen gets the nod by elimination, if nothing else.  In the
> rather lengthy
> 
> http://en.wikipedia.org/wiki/Comparison_of_documentation_generators
> 
> there are not a lot of options that are portable, GPL, etc.  LLVM's use
> of Doxygen, http://llvm.org/doxygen/index.html , seems pretty useful.)

Yeah, that's a typical example of doxygen-generated documentation.
Lots of function prototypes, a few inheritance diagrams, and barely
any actual content.  Not my defenition of useful.  In fact I'm pretty
much conditioned such that my response to seeing doxygen generated
pages is to not ever bother reading it.

Stan, I fear you're proposing a technical solution for a social
probleem.

Re: A new strategy for internals documentation

[Eli Zaretskii]

> cc: Eli Zaretskii <eliz@gnu.org>, Doug Evans <dje@google.com>,
>    gdb@sourceware.org
> Date: Fri, 09 Aug 2013 01:07:56 -0700
> From: John Gilmore <gnu@toad.com>
> 
> PS: I think if y'all have time to mess around converting perfectly
> good working C to C++, then surely some time can be spared for making
> the doc match the code.

Time is not the problem; will (or, rather, lack thereof) is.

Re: A new strategy for internals documentation

[Tom Tromey]

>>>>> "Eli" == Eli Zaretskii <eliz@gnu.org> writes:

Eli> Once again, the main problem (and some will say it's not a problem at
Eli> all) is that the majority of GDB developers don't see any need in this
Eli> manual's existence, or in any documentation of the internals besides
Eli> what's in the comments.

I just don't think this is true.
I don't know of anybody who believes this.
The disagreement, I think, is about the best way to achieve the goal.

Tom

Re: A new strategy for internals documentation

[Eli Zaretskii]

> From: Tom Tromey <tromey@redhat.com>
> Cc: John Gilmore <gnu@toad.com>, yao@codesourcery.com, stanshebs@earthlink.net,
>         gdb@sourceware.org
> Date: Fri, 09 Aug 2013 12:15:48 -0600
> 
> >>>>> "Eli" == Eli Zaretskii <eliz@gnu.org> writes:
> 
> Eli> Once again, the main problem (and some will say it's not a problem at
> Eli> all) is that the majority of GDB developers don't see any need in this
> Eli> manual's existence, or in any documentation of the internals besides
> Eli> what's in the comments.
> 
> I just don't think this is true.

That is my conclusion from the discussions on this matter.  I can show
citations, if needed.

If I'm wrong, please tell what form do you want the documentation of
the GDB internals to have.

Re: A new strategy for internals documentation

[Stan Shebs]

On 8/8/13 6:29 PM, John Gilmore wrote:
> The real issue is not about where the information is stored.  The real
> issue is that people are evolving the code base without evolving the
> matching documentation.  This is accepted to be a sin with regard to
> the "Debugging with GDB" manual, which IS required to be complete,
> definitive, tutorial, and readable.  Could I gently suggest that
> contributors who make patches that evolve e.g. the symbol table handling,
> but who don't document their improvements in Gdbint, be gently reminded
> to improve and resubmit their patch by including a patch to gdbint as well?  

In theory that's always been the case, but once the manual fell out of
sync, it was easy to reply with "my change refers to material not in the
manual" or "I don't have the time nor understanding to get the internals
manual back up to date".  We also have never had consensus among the
maintainers whether this was something to enforce, and it
only take one maintainer approving patches (or committing their own)
that change the internals, and the manual immediately stop reflecting
reality.

> I started the GDB Internals manual.  Why?  Because we had no place to
> record textual explanations about why and how GDB is internally
> structured.

The ironic thing is that no one working on a new free software project
today would react to that situation by saying "we need an internals
manual".  The project would add a wiki page, send an email, or maybe a
blog posting, or a long comment block in the source code.  If there were
a half-dozen files to edit in sync, these days there is more likely to
be intense pressure to refactor that code and bring it back down to one
place to edit, which would then be the place for the documentation about it.

While your detailed points are logically valid, we have twenty years of
empirical experience that says reality is not conforming to what we
originally imagined.  So I've suggested a Plan B where we try a couple
alternatives that are known to have a good track record in other
projects; they do not entail massive commitments or even very much
change initially.  If neither work out, then we shrug, drop them, and
think of a Plan C to try instead.

Stan
stan@codesourcery.com

Re: A new strategy for internals documentation

[Stan Shebs]

On 8/9/13 2:35 AM, Eli Zaretskii wrote:
>> Date: Thu, 08 Aug 2013 16:02:58 -0700
>> From: Stan Shebs <stanshebs@earthlink.net>
>> CC: Doug Evans <dje@google.com>, gdb@sourceware.org
>>
>> I'm not sure what we're arguing at this point
> 
> I'm done arguing.  I regret I responded to your original message,
> because all that did is cause me aggravation and open old wounds.
> (Yes, it hurts to try to contribute to a community only to see the
> attempts rejected time and again.)

Indeed - I've probably put more hours into fixing up gdbint.texinfo than
any other person, and it's disappointing to see that it's pretty much
all gone to waste.

> One last bit is that changing the format of a GNU manual might need to
> be communicated to the FSF, I think.  Or maybe a majority vote by the
> SC will be enough, I don't know (and don't care).

That's a good point.  I was tending to assume that any new
auto-generated thing would just be web pages.

Stan
stan@codesourcery.com

Re: A new strategy for internals documentation

[Stan Shebs]

On 8/9/13 2:49 AM, Mark Kettenis wrote:
>> Date: Thu, 08 Aug 2013 16:02:58 -0700
>> From: Stan Shebs <stanshebs@earthlink.net>
>>

>> 4. Use Doxygen.
>>
>> Are you for or against, or indifferent?
>>
>> (For me Doxygen gets the nod by elimination, if nothing else.  In the
>> rather lengthy
>>
>> http://en.wikipedia.org/wiki/Comparison_of_documentation_generators
>>
>> there are not a lot of options that are portable, GPL, etc.  LLVM's use
>> of Doxygen, http://llvm.org/doxygen/index.html , seems pretty useful.)
> 
> Yeah, that's a typical example of doxygen-generated documentation.
> Lots of function prototypes, a few inheritance diagrams, and barely
> any actual content.  Not my defenition of useful.  In fact I'm pretty
> much conditioned such that my response to seeing doxygen generated
> pages is to not ever bother reading it.
> 
> Stan, I fear you're proposing a technical solution for a social
> probleem.

It does look that way :-) , but I'm not under any illusion that it will
somehow magically change what people do.  It does address a couple of
the extant complaints, by expanding on the source-code commenting that
is a well-established habit now, and by having good support for API
specification.

On the general subject of technical solutions changing social behavior,
I will risk embarrassing myself by noting that I was long against moving
GDB to a public repository, because I didn't think it was going to
result in any more patches being contributed - after all, it was the
same sources and the same approval process, so what difference did it
make?  I think I've been decisively proven wrong about that one! :-)

Stan
stan@codesourcery.com

Re: A new strategy for internals documentation

[Matt Rice]

On Fri, Aug 9, 2013 at 3:31 PM, Stan Shebs <stanshebs@earthlink.net> wrote:

>  If there were
> a half-dozen files to edit in sync, these days there is more likely to
> be intense pressure to refactor that code and bring it back down to one
> place to edit

to me this is the most important thing, that anything that belongs in
the source code, shouldn't need to be duplicated in the manual, and
have to be kept in sync.  I personally don't care if that manual is
wiki/texinfo but by removing all the stuff thats already in/belongs in
the source code, the manual will be smaller and easier to keep in
sync.

as I said I don't care much about wiki/texinfo argument but it would
be nice if the manual could link directly to the doxygen
documentation, in those places where there is currently duplicate
information.  The wiki makes that very easy with hyperlinks,  If its
easy with texinfo thats fine with me too...

I don't think doxygen is the greatest thing ever, but if it gives the
internals manual an iota of a chance at staying relevent by drawing a
line in the sand the manual does not cross and reducing the churn, it
will be worth its while.

Re: A new strategy for internals documentation

[Yao Qi]

On 08/09/2013 07:04 AM, Doug Evans wrote:
> I'm one that thinks that there is not enough, and that expanding the
> comments is not enough.  For one there's a higher level / descriptive
> view that's missing with that approach.  Plus the S/N ratio when faced
> with reading all the source code is much lower than when able to
> browse something generated from the comments in the code.

Code comments can be about high-level view too.  Existing comments in
the beginning of event-loop.h and remote-notif.c are about high-level
view.

Document is generated from the comments, and we need some special 
annotations or markups to identify these comments are descriptive views 
for a certain module or components.  Doxygen or other documentation 
generators are able to do that.

-- 
Yao (齐尧)

Re: A new strategy for internals documentation

[John Gilmore]

> The ironic thing is that no one working on a new free software project
> today would react to that situation by saying "we need an internals
> manual".  The project would add a wiki page, send an email, or maybe a
> blog posting, or a long comment block in the source code.  

Or a README file in the source code.  But the info in gdbint is too
voluminous for that.  Or for an email or a blog posting or a long 
comment block.  Because it aggregates info from a lot of emails
and blog postings and comment blocks.

I mean, we could paste the info about symbol tables from gdbint into
symtab.h, as a long multipage comment block.  I doubt that that would
encourage people to update it any more than they do now.  Maybe more
people would notice it because they would find it in grep results,
since it'd be in the main source directory rather than in the doc
directory.

I was a long term volunteer for One Laptop Per Child.  OLPC used a
Wiki for its documentation.  You can see that the older stuff got well
documented, partly by staff and partly by volunteers like me.  It fell
rapidly out of date as soon as the staff shrunk and most volunteers
went on to other things.  You can see it today at
http://wiki.laptop.org .  It doesn't even mention the latest OLPC
hardware or software (the XO Tablet on sale at Walmart)!

And if you want the Wiki pages that match your OLPC hardware (they
had four different vintages), operating system release (they've gone
through about 13), etc, you are SOL -- you have to find them manually,
possibly digging through the revision history of particular pages.

And when wiki.laptop.org goes down, which it will one day, then
only the Internet Archive will have a copy -- if we're lucky.  At
least the GDB documentation is in the source code, so if GDB survives,
the doc also survives.

> If there were a half-dozen files to edit in sync, these days there
> is more likely to be intense pressure to refactor that code and
> bring it back down to one place to edit, which would then be the
> place for the documentation about it.

That's a lovely concept.  But I don't think even the amazing all-powerful
C++ has managed to eliminate the requirement to add things to both header
files and source files.  And if you change the classic way from:

  *  Add definitions for a new something to a struct or enum in category.h
  *  Add code to use that new something in category.c
  *  Add a new source module something.c that implements the something
  *  Add something.c to the Makefile

to:

  *  Add something.c to the directory, the Makefile builds and links in
     every .c automatically, initializers build a dynamic data
     structure that enumerates all things of this category, etc...

... then you have scattered the info about "things of this category"
into X number of files, rather than having a central place to ask,
"How many things fall into this category?  Which are they?"

The classic problem with object-oriented code like this is that the
information that would let you wrap your head around a concept (like a
"target" or an "architecture" or a "symtab") is scattered among dozens
of source files, and even merely instantiating one of them results in
code running in a dozen files that inherit from each other in obscure
ways.

	John

Re: A new strategy for internals documentation

[Steinar Bang]

>>>>> Stan Shebs <stanshebs@earthlink.net>:

>>> *** Doxygen

>>> As a second step, we should adopt Doxygen for the sources, and use it
>>> to generate material for a new area of the website, which will be the
>>> detailed documentation of GDB internals.

>> AFAIK, Doxygen cannot generate an Info manual, only HTML.  If so, this
>> would be a serious downside that you don't mention.

> Hmm, I'm perhaps prejudiced since I haven't looked at any info file in
> probably a decade.  Are many people using them still?

Doxygen can generate output in Docbook XML.  A quick google found me
this tool that can convert Docbook to texinfo:
 http://docbook2x.sourceforge.net/

How well that works I have no idea.

The Docbook output can be converted to a printable manual, and so can
Doxygen's LaTeX output.

Re: A new strategy for internals documentation

[Eli Zaretskii]

> From: Steinar Bang <sb@dod.no>
> Date: Wed, 21 Aug 2013 20:09:18 +0200
> 
> Doxygen can generate output in Docbook XML.  A quick google found me
> this tool that can convert Docbook to texinfo:
>  http://docbook2x.sourceforge.net/

Texinfo's makeinfo can generate Docbook for quite some time (since
2002, actually).

Re: A new strategy for internals documentation

[Steinar Bang]

>>>>> Eli Zaretskii <eliz@gnu.org>:

> Texinfo's makeinfo can generate Docbook for quite some time (since
> 2002, actually).

Yes, but I was looking for a toolchain to make texinfo from Doxygen.

I actually use texinfo results in the emacs info reader.  I find it fast
and convenient compared to a web browser.

(admittedly gdb internals hasn't been a texinfo/info document I have
read frequently...)