RFA/RFC: Pass --cache-file=/dev/null on to subconfigures

RFA/RFC: Pass --cache-file=/dev/null on to subconfigures

[Nick Clifton]

Hi Guys,

  I am regularly encountering problems rebuilding my toolchains
  because the config.cache file in one or more of the sub-directories
  is inconsistent:

configure: error: run `make distclean' and/or `rm ./config.cache' and start over
  
  This is annoying because it stops the automated build process and I
  have to manually remove the cache file before I can resume it.
  Since I am not worried about the time it takes to rerun the
  configure script in these sub-directories I would like to be able to
  build the entire toolchain without using any caches.

  In theory I can specify "--cache-file=/dev/null" when configuring
  the toolchain.  Unfortunately the top level configure file strips
  out all --cache-file arguments before invoking configure in any of
  the sub-directories.  Although this might be reasonable for a
  configure cache that is specified via an absolute path, it is
  unreasonable when the user is trying to disable caches altogether.

  So I have created the patch below.  It simply allows
  --cache-file=/dev/null to be passed on to the configure scripts in
  the sub-directories.  I have run a gcc bootstrap without problems
  and so I would like to apply the patch.

  Any comments or reasons why the patch should not be applied ?

Cheers
  Nick

./ChangeLog
2009-09-25  Nick Clifton  <nickc@redhat.com>

	* configure.ac: Pass any --cache-file=/dev/null option on to
	subconfigures.
	* configure: Regenerate.

Index: configure.ac
===================================================================
--- configure.ac	(revision 152111)
+++ configure.ac	(working copy)
@@ -2632,9 +2632,11 @@
 
 # Base args.  Strip norecursion, cache-file, srcdir, host, build,
 # target, nonopt, and variable assignments.  These are the ones we
-# might not want to pass down to subconfigures.  Also strip
-# program-prefix, program-suffix, and program-transform-name, so that
-# we can pass down a consistent program-transform-name.
+# might not want to pass down to subconfigures.  The exception being
+# --cache-file=/dev/null, which is used to turn off the use of cache
+# files altogether, and which should be passed on to subconfigures.
+# Also strip program-prefix, program-suffix, and program-transform-name,
+# so that we can pass down a consistent program-transform-name.
 baseargs=
 keep_next=no
 skip_next=no
@@ -2673,6 +2675,13 @@
   esac
 
   case "$ac_arg" in
+    --cache-file=/dev/null | \
+    -cache-file=/dev/null )
+      # Handled here to avoid the test to skip args below.
+      baseargs="$baseargs '$ac_arg'"
+      # Assert: $separate_arg should always be no.
+      keep_next=$separate_arg
+      ;;
     --no*)
       continue
       ;;

Re: RFA/RFC: Pass --cache-file=/dev/null on to subconfigures

[Ralf Wildenhues]

Hi Nick,

* Nick Clifton wrote on Fri, Sep 25, 2009 at 12:23:48PM CEST:
>   I am regularly encountering problems rebuilding my toolchains
>   because the config.cache file in one or more of the sub-directories
>   is inconsistent:
> 
> configure: error: run `make distclean' and/or `rm ./config.cache' and start over

>   Any comments or reasons why the patch should not be applied ?

Yes.  Unless you have actively changed any precious variables, they
should not be inconsistent and the cache should not be invalidated.
Can you please provide me with a recipe to reproduce this (this might
be difficult to do) or otherwise circumstances when it happens, and post
the full error message?  If you could bisect this back to the change
when this started to occur first that would be helpful too (I assume the
autotools update is relevant).

There might be more than one such issue lurking.  Developers, please
open bugs for them and Cc: me on them.

Thanks,
Ralf

Re: RFA/RFC: Pass --cache-file=/dev/null on to subconfigures

[Charles Wilson]

Ralf Wildenhues wrote:
> Yes.  Unless you have actively changed any precious variables, they
> should not be inconsistent and the cache should not be invalidated.
> Can you please provide me with a recipe to reproduce this (this might
> be difficult to do) or otherwise circumstances when it happens, and post
> the full error message?

I've seen it happen -- building gdb/insight which is part of the src
tree if not gcc specfically -- when a precious variable contained a
leading space. I think it was because the top-level configure
automatically augmented it as follows, when the var was originally empty:

  X="$X --some-new-flag"

For some reason, the sub-configure had the variable without the leading
space, so it complained of a mismatch.

Sorry I don't have more details, but I don't have the build log anymore
so I'm going by memory here. (I wasn't overriding the --cache-val in any
way).

--
Chuck

Re: RFA/RFC: Pass --cache-file=/dev/null on to subconfigures

[Nick Clifton]

Hi Ralf,

>>   Any comments or reasons why the patch should not be applied ?
> 
> Yes.  Unless you have actively changed any precious variables, they
> should not be inconsistent and the cache should not be invalidated.

Ok, first of all though, do you agree that it should be possible to 
build a toolchain without using config caches ?  Ie should a 
"--cache-file=/dev/null" specified at the top level be passed on to the 
configure scripts in sub-directories ?

Also, as an aside, what about a "--config-file=<foo>" top-level 
configure switch where <foo> is a non-absolute filename.  Eg:

   --config-file=foo.config.cache

Should a switch like this be allowed to change the name of the config 
cache files used in the sub-directories ?  Currently this will not work, 
and my patch would not fix this, but I was wondering if it was expected 
to work.

> Can you please provide me with a recipe to reproduce this

OK, here is what works for me.  Using the latest gcc and sourceware 
mainline sources, with symlinks from the gcc source directory to the 
sourceware source directory, on a Fedora 11 x86 host, in an empty 
directory, run the following commands:

   % <path-to-top-level-gcc-sources>/configure \
     --quiet \
     --target=arm-eabi \
     --enable-languages=c

   [The configure options are not essential, but they do make it faster 
to reproduce the problem.  I suspect that you need to choose a target 
that uses multilibs].

   % make all-target-newlib

   % touch <path-to-sourceware-source>/newlib/configure

   % make all-target-newlib

Result:

   configure: error: changes in the environment can compromise the build
   configure: error: run `make distclean' and/or `rm ./config.cache' and 
start over
   make[1]: *** [Makefile] Error 1


Note that the configure script at the top level of the newlib source 
directory was built with autoconf v2.59.  This is doubtless part of the 
problem.  But the issue is that if the gcc sources are part of an 
integrated source tree where some of the configure files have not been 
created with the latest version of autoconf (2.64 ?) then rebuilds of 
the toolchain can fail.

Requiring that all configure files be updated to the latest autoconf 
version might be the technically correct way to solve the problem, but 
is not really a very practical solution.  Allowing toolchains to be 
configured without config caches is a workaround, but one that is much 
simpler to adopt.

Cheers
   Nick

Re: RFA/RFC: Pass --cache-file=/dev/null on to subconfigures

[Andreas Schwab]

Charles Wilson <cygwin@cwilson.fastmail.fm> writes:

> I've seen it happen -- building gdb/insight which is part of the src
> tree if not gcc specfically -- when a precious variable contained a
> leading space. I think it was because the top-level configure
> automatically augmented it as follows, when the var was originally empty:
>
>   X="$X --some-new-flag"
>
> For some reason, the sub-configure had the variable without the leading
> space, so it complained of a mismatch.

That should no longer be an issue with the switch to autoconf 2.64.
Such whitespace differences are now ignored.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: RFA/RFC: Pass --cache-file=/dev/null on to subconfigures

[Ralf Wildenhues]

* Nick Clifton wrote on Fri, Sep 25, 2009 at 04:50:06PM CEST:
> >>  Any comments or reasons why the patch should not be applied ?
> >
> >Yes.  Unless you have actively changed any precious variables, they
> >should not be inconsistent and the cache should not be invalidated.
> 
> Ok, first of all though, do you agree that it should be possible to
> build a toolchain without using config caches ?  Ie should a
> "--cache-file=/dev/null" specified at the top level be passed on to
> the configure scripts in sub-directories ?

Please do commit it, since I guess it is necessary in practice.
However, this necessity is always a sign of a bug in the GCC/src
build system.

> Also, as an aside, what about a "--config-file=<foo>" top-level
> configure switch where <foo> is a non-absolute filename.  Eg:
> 
>   --config-file=foo.config.cache
> 
> Should a switch like this be allowed to change the name of the
> config cache files used in the sub-directories ?  Currently this
> will not work, and my patch would not fix this, but I was wondering
> if it was expected to work.

Well, with upstream Autoconf, you only ever have one cache file anyway,
so it works there.  GCC/src could easily let such a switch cause the
use of different cache file names in the sub-directories.  I'm not sure
what the benefit would be, though.

> >Can you please provide me with a recipe to reproduce this
> 
> OK, here is what works for me.

Thanks.  I will investigate it.  A couple of notes:

[...]
>   % make all-target-newlib
>   % touch <path-to-sourceware-source>/newlib/configure
>   % make all-target-newlib
> 
> Result:
> 
>   configure: error: changes in the environment can compromise the build
>   configure: error: run `make distclean' and/or `rm ./config.cache'
> and start over
>   make[1]: *** [Makefile] Error 1

There should be more information in the error message, namely the
variable(s) that changed and how they changed.  This information
can be helpful to find the bug.

> Note that the configure script at the top level of the newlib source
> directory was built with autoconf v2.59.

That is not a problem *per se*: a couple of caching bugs were fixed in
2.60, and 2.62, respectively, but the toplevel config/override.m4 file
has overrides for these fixes that *should* work for any released
Autoconf between 2.59 and current.  The only crucial bit is that
override.m4 is used for creating newlib/configure (and all other
configure scripts, for that matter).  If aclocal is used, then adding
'-I ../config' or similar to its flags (ACLOCAL_AMFLAGS) should be
sufficient, otherwise adding sinclude([../config/override.m4]) early in
configure.ac (before AC_INIT) works.

The current newlib uses confsubdir.m4 which fixes one of the problems,
but not the other.

* Charles Wilson wrote on Fri, Sep 25, 2009 at 04:48:56PM CEST:
> I've seen it happen -- building gdb/insight which is part of the src
> tree if not gcc specfically -- when a precious variable contained a
> leading space.

This is probably one of the bugs fixed by override.m4, and the insight
part is one that the autotools update hasn't touched.

Cheers,
Ralf

Re: RFA/RFC: Pass --cache-file=/dev/null on to subconfigures

[DJ Delorie]

> Well, with upstream Autoconf, you only ever have one cache file anyway,

Does it properly handle parallel builds simultaneously trying to
update the cache, yet?  I've been adding a .NOTPARALLEL: to the
toplevel Makefiles of all my builds for some time now, to avoid cache
collision problems.

Re: RFA/RFC: Pass --cache-file=/dev/null on to subconfigures

[Ralf Wildenhues]

* DJ Delorie wrote on Fri, Sep 25, 2009 at 07:29:21PM CEST:
> 
> > Well, with upstream Autoconf, you only ever have one cache file anyway,
> 
> Does it properly handle parallel builds simultaneously trying to
> update the cache, yet?

No.  I wasn't aware of this need.  I'll fix Autoconf, and override.m4.

> I've been adding a .NOTPARALLEL: to the toplevel Makefiles of all my
> builds for some time now, to avoid cache collision problems.

So far I had been under the impression that all configure scripts
invoked by the toplevel Makefile would be distinct anyway.  I see now
that that is not the case for subdirs that contain libraries for the
$build system.  I never noticed --enable-serial-configure before; and
there is some more dead code dealing with cache files in toplevel
configure.ac, that should be cleaned up once src has completed the move
to newer Autoconf.  Esp. the host dirs should then be able to once again
share a cache file, I guess.

Thanks,
Ralf

Re: RFA/RFC: Pass --cache-file=/dev/null on to subconfigures

[Charles Wilson]

Andreas Schwab wrote:
> Charles Wilson <cygwin@cwilson.fastmail.fm> writes:
> 
>> I've seen it happen -- building gdb/insight which is part of the src
>> tree if not gcc specfically -- when a precious variable contained a
>> leading space. I think it was because the top-level configure
>> automatically augmented it as follows, when the var was originally empty:
>>
>>   X="$X --some-new-flag"
>>
>> For some reason, the sub-configure had the variable without the leading
>> space, so it complained of a mismatch.
> 
> That should no longer be an issue with the switch to autoconf 2.64.
> Such whitespace differences are now ignored.

Ah:
$ find . -name "configure"|\
	xargs grep 'Generated by GNU Autoconf' | grep -v 2.64

./gdb/gdbtk/configure:# Generated by GNU Autoconf 2.59.
./gdb/gdbtk/plugins/configure:# Generated by GNU Autoconf 2.59.
./gdb/testsuite/gdb.cell/configure:# Generated by GNU Autoconf 2.59.
./gdb/testsuite/gdb.gdbtk/configure:# Generated by GNU Autoconf 2.59.
./libgui/configure:# Generated by GNU Autoconf 2.59.

Looks like some stuff missed regeneration?

--
Chuck

Re: RFA/RFC: Pass --cache-file=/dev/null on to subconfigures

[Charles Wilson]

Ralf Wildenhues wrote:
> * Charles Wilson wrote on Fri, Sep 25, 2009 at 04:48:56PM CEST:
>> I've seen it happen -- building gdb/insight which is part of the src
>> tree if not gcc specfically -- when a precious variable contained a
>> leading space.
> 
> This is probably one of the bugs fixed by override.m4, and the insight
> part is one that the autotools update hasn't touched.

Looks that way (gdbtk & libgui):
./gdb/gdbtk/configure:# Generated by GNU Autoconf 2.59.
./gdb/gdbtk/plugins/configure:# Generated by GNU Autoconf 2.59.
./gdb/testsuite/gdb.cell/configure:# Generated by GNU Autoconf 2.59.
./gdb/testsuite/gdb.gdbtk/configure:# Generated by GNU Autoconf 2.59.
./libgui/configure:# Generated by GNU Autoconf 2.59.

--
Chuck

Re: RFA/RFC: Pass --cache-file=/dev/null on to subconfigures

[Nick Clifton]

Hi Ralf,

> Please do commit it, since I guess it is necessary in practice.

Done.

> However, this necessity is always a sign of a bug in the GCC/src
> build system.

True.

> Well, with upstream Autoconf, you only ever have one cache file anyway,
> so it works there.

Really ?  I did not realise that.

> GCC/src could easily let such a switch cause the
> use of different cache file names in the sub-directories.  I'm not sure
> what the benefit would be, though.

Nor do I.  It was more of a theoretical situation than anything I might 
actually expect to happen in real life.

Cheers
   Nick