Automatic dependency tracking

Automatic dependency tracking

[Daniel Jacobowitz]

On Mon, Jun 23, 2008 at 11:00:22AM -0600, Tom Tromey wrote:
> I can send out my unfinished Makefile.am for gdb if you really want
> it.  That would solve this problem.
> 
> "Unfinished" means it is like 30% done :-(
> 
> If we didn't mind requiring GNU make we could probably implement
> dependency tracking more simply.  I suspect that is still a no-go ...

Let's talk about this... I'd be very happy if we could remove the
manual dependency lists.

Option one: we could switch to automake.  That would bring dependency
tracking for free.  GDB already has a dependency on automake, so aside
from the work of the transition I don't think this would be a big
deal.  I'm not too fond of automake, but it does work.

Option two: we could require GNU make.  GCC has done this for years,
but binutils does not; it may be premature.

Option three: we could manually list dependencies on generated files,
and support dependencies on source files only on systems with GNU
make.  This effectively means GNU make is a requirement if you are
hacking on GDB, but not to build GDB from a clean tree.

Option four: We could borrow logic from BFD to generate dependency
lists.  What's going on with this code anyway?  Why does BFD need
dependency lists in Makefile.am if it uses automake?

Have I missed anything?  Any comments on these options?

-- 
Daniel Jacobowitz
CodeSourcery

Re: Automatic dependency tracking

[Andreas Schwab]

Daniel Jacobowitz <drow@false.org> writes:

> Option four: We could borrow logic from BFD to generate dependency
> lists.  What's going on with this code anyway?  Why does BFD need
> dependency lists in Makefile.am if it uses automake?

It's a side effect of using the cygnus option.

Andreas.

-- 
Andreas Schwab, SuSE Labs, schwab@suse.de
SuSE Linux Products GmbH, Maxfeldstraße 5, 90409 Nürnberg, Germany
PGP key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: Automatic dependency tracking

[Stan Shebs]

Daniel Jacobowitz wrote:
> Option two: we could require GNU make.  GCC has done this for years,
> but binutils does not; it may be premature.
>   
I think requiring GNU make is pretty safe at this point; it's 
preinstalled on most (all?) modern systems, we've scrubbed out a lot of 
ancient hosts, and if people really want to build on something old, 
getting GNU make is going to be among the easier of their tasks.

Stan

Re: Automatic dependency tracking

[Tom Tromey]

>>>>> "Daniel" == Daniel Jacobowitz <drow@false.org> writes:

Daniel> Option two: we could require GNU make.

FWIW it is pretty easy to implement automake-style dependency tracking
with GNU make.  This failed for GCC, due to make bugs, but GCC's
Makefiles are weird.

GNU make is really much nicer than the other alternatives.

Daniel> Option four: We could borrow logic from BFD to generate dependency
Daniel> lists.  What's going on with this code anyway?  Why does BFD need
Daniel> dependency lists in Makefile.am if it uses automake?

The BFD code predates Automake and, as I recall, was deemed to work
well enough.

Tom

Re: Automatic dependency tracking

[Joel Brobecker]

> Option two: we could require GNU make.  GCC has done this for years,
> but binutils does not; it may be premature.

I think it's reasonable to require GNU make to build GDB. On all
the platforms that we have here at AdaCore, we've never had any
problem getting our hands on a recent version of GNU make. Hopefully,
that's representative of the availability of that tool.

-- 
Joel

Re: Automatic dependency tracking

[Andrew STUBBS]

Daniel Jacobowitz wrote:
> Option two: we could require GNU make.  GCC has done this for years,
> but binutils does not; it may be premature.

> Option three: we could manually list dependencies on generated files,
> and support dependencies on source files only on systems with GNU
> make.  This effectively means GNU make is a requirement if you are
> hacking on GDB, but not to build GDB from a clean tree.

As I understand it, dependencies on generated files must always be hard 
coded - otherwise the autogeneration build would fail (potentially, with 
parallel make being worse), so these two options are the same.

There is no need to require GNU make for build purposes. It would only 
be required for dependency tracking. The configure script can be made to 
disable dependency tracking on incompatible make variants.

BTW, which are the GNU make specific features in question?

Andrew

Re: Automatic dependency tracking

[Tom Tromey]

>>>>> "Andrew" == Andrew STUBBS <andrew.stubbs@st.com> writes:

>> Daniel Jacobowitz wrote:
>> Option two: we could require GNU make.  GCC has done this for years,
>> but binutils does not; it may be premature.

>> Option three: we could manually list dependencies on generated files,
>> and support dependencies on source files only on systems with GNU
>> make.  This effectively means GNU make is a requirement if you are
>> hacking on GDB, but not to build GDB from a clean tree.

Andrew> As I understand it, dependencies on generated files must always be
Andrew> hard coded - otherwise the autogeneration build would fail
Andrew> (potentially, with parallel make being worse), so these two options
Andrew> are the same.

Yeah, this is basically true -- you always have to explicitly mention
the generated headers somewhere.

However, with order-only dependencies, you don't actually need to
maintain these by hand, in the sense of figuring out which objects
depend on which generated headers.  Instead you just require that the
generated files all be built before anything else.

Andrew> There is no need to require GNU make for build purposes. It would only
Andrew> be required for dependency tracking. The configure script can be made
Andrew> to disable dependency tracking on incompatible make variants.

You can do this dynamically by creating a GNUMakefile in the build
tree :-).  Well, if you don't mind the hackiness of that.

Andrew> BTW, which are the GNU make specific features in question?

'-include', and probably order-only dependencies.
'ifeq' makes it a bit friendlier too, and sometimes new-style pattern
rules (depending on details in the Makefile that I don't remember
right now).

I've appended the relevant code from libcpp's Makefile.  libcpp does
not use automake but does implement the dependency tracking bits.
Hmm, reading it now, I see it looks like goo :)

libcpp also has this:

    # Dependencies on generated headers have to be explicit.
    init.o: localedir.h

But with order-only dependencies I would just write:

    $(all_objects): | $(generated_headers)

This does the right thing.

Tom


# Dependency rule.
COMPILE.base = $(CC) $(DEFS) $(INCLUDES) $(CPPFLAGS) $(ALL_CFLAGS) -c
ifeq ($(DEPMODE),depmode=gcc3)
# Note that we put the dependencies into a .Tpo file, then move them
# into place if the compile succeeds.  We need this because gcc does
# not atomically write the dependency output file.
COMPILE = $(COMPILE.base) -o $@ -MT $@ -MMD -MP -MF $(DEPDIR)/$*.Tpo
POSTCOMPILE = @mv $(DEPDIR)/$*.Tpo $(DEPDIR)/$*.Po
else
COMPILE = source='$<' object='$@' libtool=no DEPDIR=$(DEPDIR) $(DEPMODE) \
	  $(depcomp) $(COMPILE.base)
# depcomp handles atomicity for us, so we don't need a postcompile
# step.
POSTCOMPILE =
endif

# Implicit rules and I18N

.c.o:
	$(COMPILE) $<
	$(POSTCOMPILE)

# Dependencies
-include $(patsubst %.o, $(DEPDIR)/%.Po, $(libcpp_a_OBJS) $(makedepend_OBJS))

Re: Automatic dependency tracking

[Andrew STUBBS]

Tom Tromey wrote:
> '-include', and probably order-only dependencies.
> 'ifeq' makes it a bit friendlier too, and sometimes new-style pattern
> rules (depending on details in the Makefile that I don't remember
> right now).

...

> But with order-only dependencies I would just write:

>     $(all_objects): | $(generated_headers)

OK, I understand now. Presumably it would still be possible to retain 
non-GNU make compatibility for build-only purposes by also providing a 
regular dependency - basically just removing the bar from the above - 
and then preprocessing the Makefile, like AM_CONDITIONAL?

I don't know if it's worth it though. Like Daniel said, GNU make isn't 
hard to build, and other projects already require it.

Auto dependency tracking would be very nice to have though. :)

Andrew

Re: Automatic dependency tracking

[Daniel Jacobowitz]

On Fri, Jun 27, 2008 at 11:22:27AM +0100, Andrew STUBBS wrote:
> OK, I understand now. Presumably it would still be possible to retain  
> non-GNU make compatibility for build-only purposes by also providing a  
> regular dependency - basically just removing the bar from the above - and 
> then preprocessing the Makefile, like AM_CONDITIONAL?

Sure - e.g. AC_SUBST(ORDER, ${bar_if_gnu_make}).

> I don't know if it's worth it though. Like Daniel said, GNU make isn't  
> hard to build, and other projects already require it.

Actually, Stan and Joel said it :-)

Mark, do you have any comments about requiring GNU make for correct
dependencies?  I may be misremembering, but I believe you were
concerned about GNU make dependency the last time this came up.
Even if not, you work on more host platforms than any other GDB
maintainer.

Is supporting clean builds (actually, also modifying a source file and
rebuilding - everything but modifying a header file) without GNU make
sufficient?

-- 
Daniel Jacobowitz
CodeSourcery

Re: Automatic dependency tracking

[Andrew STUBBS]

Daniel Jacobowitz wrote:
>> I don't know if it's worth it though. Like Daniel said, GNU make isn't  
>> hard to build, and other projects already require it.

> Actually, Stan and Joel said it :-)

Sorry guys. :)

Re: Automatic dependency tracking

[Tom Tromey]

[-- Attachment #1: Type: text/plain, Size: 1076 bytes --]

[ I'm re-sending this, this time w/ an attachment, since sourceware
rejected the original note as too big... Andrew, sorry if you get this
twice. ]

>>>>> "Andrew" == Andrew STUBBS <andrew.stubbs@st.com> writes:

Andrew> OK, I understand now. Presumably it would still be possible to retain
Andrew> non-GNU make compatibility for build-only purposes by also providing a
Andrew> regular dependency - basically just removing the bar from the above - 
Andrew> and then preprocessing the Makefile, like AM_CONDITIONAL?

Yeah, probably.  Or someone could redo the Makefile in Automake, which
automates all this.

I can't emphasize enough how much nicer it is to assume GNU make than
to try to be portable.

Andrew> Auto dependency tracking would be very nice to have though. :)

I've appended a patch to do it.  I made this against the python
repository on gitorious -- but aside from the extra code for the
python objects, it would be the same for the trunk.  The patch is
nearly all deletions.

Give it a whirl & let me know what you think.  No ChangeLog entry yet,
sorry.

Tom


[-- Attachment #2: the patch --]
[-- Type: application/x-gzip, Size: 20587 bytes --]

Re: Automatic dependency tracking

[Mark Kettenis]

> Date: Wed, 25 Jun 2008 14:28:58 -0400
> From: Daniel Jacobowitz <drow@false.org>
> 
> Option two: we could require GNU make.  GCC has done this for years,
> but binutils does not; it may be premature.

I'd like to avoid this if we can.  On OpenBSD we include GDB in the
base system, but we don't have GNU make.  This'll mean I'll have to do
extensive makefile hackary whenever I import a new version of GDB in
the OpenBSD tree.  I'd rather spend that time on hacking more useful
stuff.

> Option three: we could manually list dependencies on generated files,
> and support dependencies on source files only on systems with GNU
> make.  This effectively means GNU make is a requirement if you are
> hacking on GDB, but not to build GDB from a clean tree.

Given the way CVS handles time stamps on checkouts, this probably gets
me in the same situation as above.

Re: Automatic dependency tracking

[Tom Tromey]

>>>>> "Mark" == Mark Kettenis <mark.kettenis@xs4all.nl> writes:

Mark> I'd like to avoid this if we can.  On OpenBSD we include GDB in the
Mark> base system, but we don't have GNU make.  This'll mean I'll have to do
Mark> extensive makefile hackary whenever I import a new version of GDB in
Mark> the OpenBSD tree.  I'd rather spend that time on hacking more useful
Mark> stuff.

How does OpenBSD handle this for GCC?
Or does the OpenBSD gcc maintainer also rewrite the Makefiles?

Ugh.

FWIW, rewriting the patch I sent to avoid GNU make is likely to be
pretty fiddly & painful.

Tom

Re: Automatic dependency tracking

[Daniel Jacobowitz]

On Sat, Jun 28, 2008 at 01:30:46PM -0600, Tom Tromey wrote:
> How does OpenBSD handle this for GCC?
> Or does the OpenBSD gcc maintainer also rewrite the Makefiles?
> 
> Ugh.

I believe OpenBSD is enough versions back on GCC that it is not yet an
issue; I see GCC 3.3.5 in their cvsweb.  GCC documented the
requirement in August 2003; GCC 3.3.0 was released in May 2003.

-- 
Daniel Jacobowitz
CodeSourcery

Re: Automatic dependency tracking

[Tom Tromey]

>>>>> "Tom" == Tom Tromey <tromey@redhat.com> writes:

Tom> FWIW, rewriting the patch I sent to avoid GNU make is likely to
Tom> be pretty fiddly & painful.

So much for guessing.  Now I think it isn't too bad, if we do not mind
having dependencies only in the GNU make case.

Daniel> So timestamps shouldn't matter; the only new failure mode is
Daniel> that if you modify a header, the source files including it
Daniel> might not be automatically rebuilt.

I am happy to implement this, but it will be a major regression for
folks not using GNU make.

Tom

Re: Automatic dependency tracking

[Tom Tromey]

[-- Attachment #1: Type: text/plain, Size: 969 bytes --]

>>>>> "Tom" == Tom Tromey <tromey@redhat.com> writes:

Tom> I am happy to implement this, but it will be a major regression for
Tom> folks not using GNU make.

Here is a new patch.

Again, I made this against the Python branch.  Sorry about that, but I
have uncommitted patches in my trunk tree and didn't want to mess with
it.

This one works by overriding selected variables and adding new rules
using a GNUmakefile.  While somewhat more obscure than AM_CONDITIONAL,
this one has the nice property that the user can switch between GNU
make and other makes without needing to re-run configure.

IMO this patch is uglier than the previous.  For one thing I had to
recreate explicit rules for all the objects whose source files are in
subdirs.

I only have GNU make.  So, I don't actually know that this works for
non-GNU makes.  Mark, maybe you could try it out?

Let me know what you think of this.  If it looks ok-ish I will clean
it up and submit it for real.

Tom


[-- Attachment #2: patch --]
[-- Type: application/x-gzip, Size: 22077 bytes --]

Re: Automatic dependency tracking

[Daniel Jacobowitz]

On Sat, Jun 28, 2008 at 09:20:10PM +0200, Mark Kettenis wrote:
> > Option three: we could manually list dependencies on generated files,
> > and support dependencies on source files only on systems with GNU
> > make.  This effectively means GNU make is a requirement if you are
> > hacking on GDB, but not to build GDB from a clean tree.
> 
> Given the way CVS handles time stamps on checkouts, this probably gets
> me in the same situation as above.

Could you elaborate?  I don't think I understand the problem.

What I had in mind was having configure detect GNU Make and
enable/disable (equivalent of AM_CONDITIONAL) the dependency
tracking code in the generated Makefile.  So timestamps shouldn't
matter; the only new failure mode is that if you modify a header,
the source files including it might not be automatically rebuilt.

-- 
Daniel Jacobowitz
CodeSourcery