From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from simark.ca by simark.ca with LMTP id HwRWLWTpMGY75AAAWB0awg (envelope-from ) for ; Tue, 30 Apr 2024 08:51:48 -0400 Authentication-Results: simark.ca; dkim=pass (1024-bit key; secure) header.d=sourceware.org header.i=@sourceware.org header.a=rsa-sha256 header.s=default header.b=txTsWmHa; dkim-atps=neutral Received: by simark.ca (Postfix, from userid 112) id 95FFA1E0C1; Tue, 30 Apr 2024 08:51:48 -0400 (EDT) Received: from server2.sourceware.org (server2.sourceware.org [8.43.85.97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (prime256v1) server-digest SHA256) (No client certificate requested) by simark.ca (Postfix) with ESMTPS id 6944A1E092 for ; Tue, 30 Apr 2024 08:51:46 -0400 (EDT) Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id C02AC3858429 for ; Tue, 30 Apr 2024 12:51:45 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org C02AC3858429 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sourceware.org; s=default; t=1714481505; bh=YAk0C/JRJ3GVQmPY5EMCo/FFaYvBnLBFhiHkrh1newc=; h=References:In-Reply-To:Date:Subject:To:Cc:List-Id: List-Unsubscribe:List-Archive:List-Post:List-Help:List-Subscribe: From:Reply-To:From; b=txTsWmHaCe/0mb/FZBpeek9/YDDQE3+s4JOdmvXYs38whW5CJI/Hp717WcrDJGa5q x+xvh2nDbz3DU8ZBTqAfgO8IZ0OdRpu2xSc7MAv3rFIoOKrL4CjOwlTrtrfefiSyKh lE73YT1nu/9SKOtmcaV7s0N05QdUY90HL/tKXCwY= Received: from mail-lj1-x235.google.com (mail-lj1-x235.google.com [IPv6:2a00:1450:4864:20::235]) by sourceware.org (Postfix) with ESMTPS id D93523858D20 for ; Tue, 30 Apr 2024 12:50:56 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org D93523858D20 ARC-Filter: OpenARC Filter v1.0.0 sourceware.org D93523858D20 ARC-Seal: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1714481459; cv=none; b=MYb0lif+zxuZNhMX5afM/L5WBd84i2ETGfohP8oXA+c6dW+QiuXVnE3vFSQFPPWtJqQ2Roqq9BQ2FTDT14B0cJDaW7Ri6bEv6lxK3xt1ny8N6fN0NJeJxTcv4d4OK2RxU/OMPji5ENp9+QHXtIrv1Al+y+2Ah5/E+p11SIMDyMU= ARC-Message-Signature: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1714481459; c=relaxed/simple; bh=+ywJelLPlgwJacDxoI6l3pULB76Y+9snSbIhmuonx/o=; h=DKIM-Signature:MIME-Version:From:Date:Message-ID:Subject:To; b=t1fa5luN9zQaPlYJqENdibc3a25jLlTVXjAWljIbGNwKvQgDAY2wspAz8AAKETIcbox08wlNo31FPE+2aa39oPyts3aM7IOs2x2DPwBA4uBrqQpBImk8QWb0LHZc9GidrbWluSyMd1b0598XNDW1tYZo7Ghb7xCxHRU5AoBBn98= ARC-Authentication-Results: i=1; server2.sourceware.org Received: by mail-lj1-x235.google.com with SMTP id 38308e7fff4ca-2def8e58471so89053061fa.0 for ; Tue, 30 Apr 2024 05:50:56 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1714481455; x=1715086255; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=YAk0C/JRJ3GVQmPY5EMCo/FFaYvBnLBFhiHkrh1newc=; b=DPMt7mli7rdOQkaXOnq+PFowhDTlFFJoZN8T3o75mBixCcolIYUHqiAG2deEVsJLHq QmGmZdxy18XGjS3NgxCXbLAVrLjsRNujXkJQ1gfozJJ29j2MX4KirSLJJRsuQu6eHby+ avj51VvjueZoJ+ZQL5eEOfbIis60drJ6WH5Scbxy5DkprVgZigKYu6ZRxoVkarkJdHJl qZW1qbT3AR90BU1Tw16+v/o2InZfriH17Fe9M8Fu7KtMvKZHBlzzFXY+VUEmxNzUQhRe dvws3jl3Z5A3ZWn7rY0KjPfybHkbvid70SOZLzWbb633kKUzQPMVPehuJNCHAZSTGtuK DODA== X-Gm-Message-State: AOJu0Yz5GRqCEhu8pi5sTrj3xa1d3vQL8akUn9FfDynTw7pomAf5nfig ccbLhKRrK64/43XiEEtc6MSMWLsj3fy9e8eR8O3lHKf2mlHT+ymo+CNahIber4VA50AoAozBDxe ZN/c2tVWtpcOOb/zIU2msZFxP5WJJx2NGEeqfiw== X-Google-Smtp-Source: AGHT+IGpX6dJVzM6NVHi7iJU86WlfIzrgi+edt49H36kKOl/+ivA4gC1vTBs2OWxYrh4tWrc6kLZvS+DKh+EKmCP2NE= X-Received: by 2002:a2e:2203:0:b0:2e0:3ad2:b371 with SMTP id i3-20020a2e2203000000b002e03ad2b371mr5710214lji.25.1714481453023; Tue, 30 Apr 2024 05:50:53 -0700 (PDT) MIME-Version: 1.0 References: <20240419090903.2080043-1-christophe.lyon@linaro.org> <87sezh558o.fsf@gentoo.org> In-Reply-To: <87sezh558o.fsf@gentoo.org> Date: Tue, 30 Apr 2024 14:50:42 +0200 Message-ID: Subject: Re: [PATCH] contrib: Add autoregen.py To: Sam James Cc: gdb@sourceware.org Content-Type: text/plain; charset="UTF-8" X-Spam-Status: No, score=-9.7 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, GIT_PATCH_0, RCVD_IN_DNSWL_NONE, SPF_HELO_NONE, SPF_PASS, TXREP autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: gdb@sourceware.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Gdb mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , From: Christophe Lyon via Gdb Reply-To: Christophe Lyon Errors-To: gdb-bounces+public-inbox=simark.ca@sourceware.org Sender: "Gdb" Hi, On Sat, 20 Apr 2024 at 00:59, Sam James wrote: > > Christophe Lyon via Gdb writes: > > > This script is a copy of the current script used by Sourceware's > > autoregen buildbots. > > Please include a link to where it can be found and the commit > you're importing it from. I'd also consider CCing the builder ML. Good points, I agree. > It may be nice (not sure) to include `git log` or `git shortlog` > in the commit message to preserve some history too. > > > > > It is intended as a helper to regenerate files managed by autotools > > (autoconf, automake, aclocal, ....), as well as the toplevel > > Makefile.in which is created by autogen. > > > > Other files can be updated when using maintainer-mode, but this is not > > covered by this script. > > Who will own contrib/autoregen.py? Do we need to get all changes > reviewed across gdb, binutils, and gcc? The shared files tend to be a > huge pain for this. Yes, that's a concern I have. My motivation for adding it to contrib/ is to make it more visible (I don't think many contributors know where to find the sources of the autoregen buildbot, maybe most don't know it exists) It would also mean that the buildbot could do the equivalent of cd $project #project is one of gcc, binutils-gdb if [ -f .contrib/autoregen.py ]; then ./contrib/autoregen.py fi but that doesn't solve the maintenance pain indeed. Or maybe just drop this idea, and somehow make the existence of the buildbot more widely known? (how?) Thanks, Christophe > > > > > 2024-04-19 Christophe Lyon > > > > contrib/ > > * autoregen.py: New script. > > --- > > contrib/autoregen.py | 221 +++++++++++++++++++++++++++++++++++++++++++ > > 1 file changed, 221 insertions(+) > > create mode 100755 contrib/autoregen.py > > > > diff --git a/contrib/autoregen.py b/contrib/autoregen.py > > new file mode 100755 > > index 00000000000..faffc88c5bd > > --- /dev/null > > +++ b/contrib/autoregen.py > > @@ -0,0 +1,221 @@ > > +#!/usr/bin/env python3 > > + > > +# This script helps to regenerate files managed by autotools and > > +# autogen in binutils-gdb and gcc repositories. > > + > > +# It can be used by buildbots to check that the current repository > > +# contents has been updated correctly, and by developers to update > > +# such files as expected. > > + > > +import os > > +import shutil > > +import subprocess > > +from pathlib import Path > > + > > + > > +# On Gentoo, vanilla unpatched autotools are packaged separately. > > +# We place the vanilla names first as we want to prefer those if both exist. > > +AUTOCONF_NAMES = ["autoconf-vanilla-2.69", "autoconf-2.69", "autoconf"] > > +AUTOMAKE_NAMES = ["automake-vanilla-1.15", "automake-1.15.1", "automake"] > > +ACLOCAL_NAMES = ["aclocal-vanilla-1.15", "aclocal-1.15.1", "aclocal"] > > +AUTOHEADER_NAMES = ["autoheader-vanilla-2.69", "autoheader-2.69", "autoheader"] > > +AUTORECONF_NAMES = ["autoreconf-vanilla-2.69", "autoreconf-2.69", "autoreconf"] > > + > > +# Pick the first for each list that exists on this system. > > +AUTOCONF_BIN = next(name for name in AUTOCONF_NAMES if shutil.which(name)) > > +AUTOMAKE_BIN = next(name for name in AUTOMAKE_NAMES if shutil.which(name)) > > +ACLOCAL_BIN = next(name for name in ACLOCAL_NAMES if shutil.which(name)) > > +AUTOHEADER_BIN = next(name for name in AUTOHEADER_NAMES if shutil.which(name)) > > +AUTORECONF_BIN = next(name for name in AUTORECONF_NAMES if shutil.which(name)) > > + > > +AUTOGEN_BIN = "autogen" > > + > > +# autoconf-wrapper and automake-wrapper from Gentoo look at this environment variable. > > +# It's harmless to set it on other systems though. > > +EXTRA_ENV = { > > + "WANT_AUTOCONF": AUTOCONF_BIN.split("-", 1)[1] if "-" in AUTOCONF_BIN else "", > > + "WANT_AUTOMAKE": AUTOMAKE_BIN.split("-", 1)[1] if "-" in AUTOMAKE_BIN else "", > > + "AUTOCONF": AUTOCONF_BIN, > > + "ACLOCAL": ACLOCAL_BIN, > > + "AUTOMAKE": AUTOMAKE_BIN, > > + "AUTOGEN": AUTOGEN_BIN, > > +} > > +ENV = os.environ.copy() > > +ENV.update(EXTRA_ENV) > > + > > + > > +# Directories we should skip entirely because they're vendored or have different > > +# autotools versions. > > +SKIP_DIRS = [ > > + # readline and minizip are maintained with different autotools versions > > + "readline", > > + "minizip", > > +] > > + > > +# these directories are known to be re-generatable with a simple autoreconf > > +# without special -I flags > > +# Entries commented out (and directories not listed) are handled by > > +# regenerate_manually(). > > +AUTORECONF_DIRS = [ > > + # subdirs common to binutils-gdb and gcc > > + "libbacktrace", > > + "libdecnumber", # No Makefile.am > > + "libiberty", # No Makefile.am > > + "zlib", > > + > > + # binutils-gdb subdirs > > + "bfd", > > + "binutils", > > + "etc", > > + "gas", > > + "gdb", > > + "gdbserver", > > + "gdbsupport", > > + "gnulib", > > + "gold", > > + "gprof", > > + "gprofng", > > + "gprofng/libcollector", > > + "ld", > > + "libctf", > > + "libsframe", > > + "opcodes", > > + "sim", > > + > > + # gcc subdirs > > + "c++tools", # No aclocal.m4 > > + "gcc", # No Makefile.am > > + #"fixincludes", # autoreconf complains about GCC_AC_FUNC_MMAP_BLACKLIST > > + "gnattools", # No aclocal.m4 > > + "gotools", > > + "libada", # No aclocal.m4 > > + "libatomic", > > + "libcc1", > > + "libcody", # No aclocal.m4 > > + "libcpp", # No Makefile.am > > + "libffi", > > + "libgcc", # No aclocal.m4 > > + "libgfortran", > > + # Hack: ACLOCAL_AMFLAGS = -I .. -I ../config in Makefile.in but we > > + # apply -I../config -I.. otherwise we do not match the current > > + # contents > > + #"libgm2", > > + "libgo", > > + "libgomp", > > + "libgrust", > > + "libitm", > > + "libobjc", # No Makefile.am > > + "libphobos", > > + "libquadmath", > > + "libsanitizer", > > + "libssp", > > + "libstdc++-v3", > > + # This does not cover libvtv/testsuite/other-tests/Makefile.in > > + "libvtv", > > + "lto-plugin", > > +] > > + > > + > > +# Run the shell command CMD. > > +# > > +# Print the command on stdout prior to running it. > > +def run_shell(cmd: str): > > + print(f"+ {cmd}", flush=True) > > + res = subprocess.run( > > + f"{cmd}", > > + shell=True, > > + encoding="utf8", > > + env=ENV, > > + ) > > + res.check_returncode() > > + > > + > > +def regenerate_with_autoreconf(): > > + run_shell(f"{AUTORECONF_BIN} -f") > > + > > +def regenerate_with_autogen(): > > + run_shell(f"{AUTOGEN_BIN} Makefile.def") > > + > > +def regenerate_manually(): > > + configure_lines = open("configure.ac").read().splitlines() > > + if folder.stem == "fixincludes" or folder.stem == "libgm2" or any( > > + True for line in configure_lines if line.startswith("AC_CONFIG_MACRO_DIR") > > + ): > > + include_arg = "" > > + include_arg2 = "" > > + if (folder / ".." / "config").is_dir(): > > + include_arg = "-I../config" > > + > > + # this is really a hack just for binutils-gdb/gprofng/libcollector > > + # make sure that the order of includes is done as --enable-maintainer-mode > > + if (folder / ".." / ".." / "config").is_dir(): > > + include_arg = "-I../.." > > + include_arg2 = "-I../../config" > > + > > + if folder.stem == "fixincludes": > > + include_arg = "-I.." > > + include_arg2 = "-I../config" > > + > > + if folder.stem == "libgm2": > > + include_arg = "-I../config" > > + include_arg2 = "-I.." > > + > > + # aclocal does not support the -f short option for force > > + run_shell(f"{ACLOCAL_BIN} --force {include_arg} {include_arg2}") > > + > > + if (folder / "config.in").is_file() or any( > > + True for line in configure_lines if line.startswith("AC_CONFIG_HEADERS") > > + ): > > + run_shell(f"{AUTOHEADER_BIN} -f") > > + > > + # The few lines below do not regenerate the exact same content as > > + # currently in the repo. Disable them for now. > > + # if (folder / "gm2-libs").is_dir(): > > + # run_shell(f"{AUTOCONF_BIN} -f gm2-libs/config-host.in > gm2-libs/config-host") > > + # run_shell(f"{AUTOHEADER_BIN} gm2-libs/config-host.in") > > + > > + > > + # apparently automake is somehow unstable -> skip it for gotools > > + if any( > > + True for line in configure_lines if line.startswith("AM_INIT_AUTOMAKE") > > + ) and not str(folder).endswith("gotools"): > > + run_shell(f"{AUTOMAKE_BIN} -f") > > + > > + run_shell(f"{AUTOCONF_BIN} -f") > > + > > + > > +run_shell(f"{AUTOCONF_BIN} --version") > > +run_shell(f"{AUTOMAKE_BIN} --version") > > +run_shell(f"{ACLOCAL_BIN} --version") > > +run_shell(f"{AUTOHEADER_BIN} --version") > > + > > +print(f"Extra environment: {EXTRA_ENV}", flush=True) > > + > > +config_folders: list[Path] = [] > > +autogen_folders: list[Path] = [] > > +repo_root = Path.cwd() > > + > > +for root, _, files in os.walk("."): > > + for file in files: > > + if file == "configure.ac": > > + config_folders.append(Path(root).resolve()) > > + if file == "Makefile.tpl": > > + autogen_folders.append(Path(root).resolve()) > > + > > +for folder in sorted(autogen_folders): > > + print(f"Entering directory {folder}", flush=True) > > + os.chdir(folder) > > + regenerate_with_autogen() > > + > > +for folder in sorted(config_folders): > > + if folder.stem in SKIP_DIRS: > > + print(f"Skipping directory {folder}", flush=True) > > + continue > > + > > + print(f"Entering directory {folder}", flush=True) > > + os.chdir(folder) > > + > > + if str(folder.relative_to(repo_root)) in AUTORECONF_DIRS: > > + regenerate_with_autoreconf() > > + else: > > + regenerate_manually()