From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from simark.ca by simark.ca with LMTP id +ZeQIObCmWpCJCYAWB0awg (envelope-from ) for ; Thu, 03 Sep 2026 14:56:38 -0400 Authentication-Results: simark.ca; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=palves.net header.i=@palves.net header.a=rsa-sha256 header.s=dreamhost header.b=i/5PPNrT; dkim-atps=neutral Received: by simark.ca (Postfix, from userid 112) id 808441E166; Thu, 03 Sep 2026 14:56:38 -0400 (EDT) X-Spam-Checker-Version: SpamAssassin 4.0.1 (2024-03-25) on simark.ca X-Spam-Level: X-Spam-Status: No, score=-2.1 required=5.0 tests=ARC_SIGNED,ARC_VALID,BAYES_00, DKIM_INVALID,DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_DNSWL_MED, RCVD_IN_VALIDITY_CERTIFIED_BLOCKED,RCVD_IN_VALIDITY_RPBL_BLOCKED, RCVD_IN_VALIDITY_SAFE_BLOCKED autolearn=ham autolearn_force=no version=4.0.1 Received: from vm01.sourceware.org (vm01.sourceware.org [38.145.34.32]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519 server-signature ECDSA (prime256v1) server-digest SHA256) (No client certificate requested) by simark.ca (Postfix) with ESMTPS id 0FF7E1E033 for ; Thu, 03 Sep 2026 14:56:37 -0400 (EDT) Received: from vm01.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id A898C4BB1C27 for ; Thu, 3 Sep 2026 18:56:36 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org A898C4BB1C27 Authentication-Results: sourceware.org; dkim=fail reason="signature verification failed" (2048-bit key, unprotected) header.d=palves.net header.i=@palves.net header.a=rsa-sha256 header.s=dreamhost header.b=i/5PPNrT Received: from lightblue.cherry.relay.mailchannels.net (lightblue.cherry.relay.mailchannels.net [23.83.223.104]) by sourceware.org (Postfix) with ESMTPS id A09E04BB1C1A for ; Thu, 3 Sep 2026 18:56:05 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org A09E04BB1C1A Authentication-Results: sourceware.org; dmarc=none (p=none dis=none) header.from=palves.net Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=palves.net ARC-Filter: OpenARC Filter v1.0.0 sourceware.org A09E04BB1C1A Authentication-Results: sourceware.org; arc=none smtp.remote-ip=23.83.223.104 ARC-Seal: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1788461765; cv=none; b=xPHwLyfQlcYBOP9/C6V3JZXLqkBs/3IXjdi6Q/M9w8rNfxljQvXwRZlqitN4+WCJ6/z4UsT6HIb2+8898xDoAJ5TymYfq1kh0gYWzp65B/6ZuwYwIhGPLLbeH6pe9zMHpLj84T3BYkMTB3LKQKg3oas0RnwOyyYjahAKp2EVvDg= ARC-Message-Signature: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1788461765; c=relaxed/simple; bh=fiA8FUp0LOa1pvfJkigVvRWqfUjE12HJM0TW3PeoX9g=; h=DKIM-Signature:From:To:Subject:Date:Message-ID:MIME-Version; b=p/sP8NZ1kkXeKSHej6r5PqFpvmmPqr7m+UJtBhs77WXCPbjn8h90YluyyFAN1rZXkOY2LglaLrMzPBnq2eub8yadKRbno+WfXXUDuJeD10PuPC6q6iZfhBdteckQKLOa5ZMqCLkv9B/MgtujOnMU/E2QD3ETahJVVqNc4LKl7Eo= ARC-Authentication-Results: i=1; sourceware.org; dkim=pass (2048-bit key, unprotected) header.d=palves.net header.i=@palves.net header.a=rsa-sha256 header.s=dreamhost header.b=i/5PPNrT DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org A09E04BB1C1A X-Sender-Id: dreamhost|x-authsender|pedro@palves.net Received: from relay.mailchannels.net (localhost [127.0.0.1]) by relay.mailchannels.net (Postfix) with ESMTP id BE91294013B; Thu, 03 Sep 2026 18:56:04 +0000 (UTC) Received: from pdx1-sub0-mail-a240.dreamhost.com (trex-green-1.trex.outbound.svc.cluster.local [100.97.105.60]) (Authenticated sender: dreamhost) by relay.mailchannels.net (Postfix) with ESMTPA id 88549943312; Thu, 03 Sep 2026 18:56:04 +0000 (UTC) X-Sender-Id: dreamhost|x-authsender|pedro@palves.net X-MC-Relay: Good X-MailChannels-SenderId: dreamhost|x-authsender|pedro@palves.net X-MailChannels-Auth-Id: dreamhost X-Little-Bubble: 7013d03020c20c8b_1788461764630_3761071475 X-MC-Loop-Signature: 1788461764630:873029750 X-MC-Ingress-Time: 1788461764630 Received: from pdx1-sub0-mail-a240.dreamhost.com (pop.dreamhost.com [64.90.62.162]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384) by 100.97.105.60 (trex/8.0.2); Thu, 03 Sep 2026 18:56:04 +0000 Received: from localhost (bl22-81-37.dsl.telepac.pt [2.83.81.37]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-256) server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) (Authenticated sender: pedro@palves.net) by pdx1-sub0-mail-a240.dreamhost.com (Postfix) with ESMTPSA id 4hbTMR6S8kz105H; Thu, 3 Sep 2026 11:56:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=palves.net; s=dreamhost; t=1788461764; bh=09RlXZzL0CgdIw1oxWFaFoO7sIAohEMDtn0UD5wXiJs=; h=From:To:Cc:Subject:Date:Content-Transfer-Encoding; b=i/5PPNrTf1bxSFnfF18ag5Z9hsQTrElNSLU2Z4SEwbEKGI/ULE+ryZaU+YP3OC2ad IDbVmIeX/0JP2Q0oskDcL/gG1z+RhlYm9ofR2ZCEgN0X5RQxpTbrMU5Hc+fShcAGlZ WMEzRyQUmBIZqJfOQ6j7IkrEZStX960f23RI9m1OuEfW+Nk3cAxFIgRSNyhd4gx/7c UmfhP996swY9nE7geypAyAnjIsf5rUczSPpFon1ta3kF4S84K+TcD7T8r+css2aUUb EwJhpzL08Eyht3shxzAFAn8mEKfaOySWKNYxBrRl+6KHdAwo7jX8MbVnLpwSqKGIC3 qprJFcN3wP7yQ== From: Pedro Alves To: gcc-patches@gcc.gnu.org Cc: gdb-patches@sourceware.org Subject: [PATCH 1/1] dg-extract-results.{sh, py}: Optionally read file list from stdin/file Date: Thu, 3 Sep 2026 19:55:28 +0100 Message-ID: <20260903185528.2542431-2-pedro@palves.net> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260903185528.2542431-1-pedro@palves.net> References: <20260903185528.2542431-1-pedro@palves.net> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-BeenThere: gdb-patches@sourceware.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Gdb-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: gdb-patches-bounces~public-inbox=simark.ca@sourceware.org dg-extract-results.sh and dg-extract-results.py take the summary/log files to combine as command-line arguments, one argument per file. That is fine for GCC, whose parallel harness produces a handful of files. GDB uses these scripts too, but its parallel harness writes one .sum (and one .log) per test script, of which there are 2657 at the moment. On Windows, with native Windows python, passing that many sum or log files to dg-extract-results overflows the command-line length limit (~32 KB), and the combine step dies. The .sh wrapper re-execs the faster Python implementation, forwarding the file list, so the overflow can also surface at the re-exec. E.g.: $ make check -j$(nproc) ... make[1]: Leaving directory '/c/msys2/home/alves/gdb/build-testsuite-windows' /c/gdb/src/gdb/testsuite/../../contrib/dg-extract-results.sh: line 39: /ucrt64/bin/python3: Argument list too long /c/gdb/src/gdb/testsuite/../../contrib/dg-extract-results.sh: line 39: /ucrt64/bin/python3: Argument list too long make: *** [Makefile:274: check-parallel] Error 2 ... $ find outputs/ -name gdb.sum | wc -l 2657 Fix it by teaching both dg-extract-results.sh and dg-extract-results.py a new -f option that reads the list of files to process, one per line, from a named file, or from standard input when the argument is "-". The option may be given more than once and mixed with file arguments. A caller can then pipe the file list straight in via "-f -" instead of expanding it onto the command line, so the length limit no longer applies regardless of host or Python flavor. Existing callers can continue working as they were, as passing the file list as one file per argument is still supported. contrib/ChangeLog: * dg-extract-results.py: Handle new "-f list-file" option. * dg-extract-results.sh: Ditto. Change-Id: Iaced5ba228972ce267f355de65fe5ad8a1e38c68 --- contrib/dg-extract-results.py | 30 ++++++++++++++++++++++++++---- contrib/dg-extract-results.sh | 17 +++++++++++++++-- 2 files changed, 41 insertions(+), 6 deletions(-) diff --git a/contrib/dg-extract-results.py b/contrib/dg-extract-results.py index 98b0f4989c96..b5d890f64063 100644 --- a/contrib/dg-extract-results.py +++ b/contrib/dg-extract-results.py @@ -164,7 +164,7 @@ class Prog: def usage (self): name = sys.argv[0] sys.stderr.write ('Usage: ' + name - + ''' [-t tool] [-l variant-list] [-L] log-or-sum-file ... + + ''' [-t tool] [-l variant-list] [-L] [-f list-file] log-or-sum-file ... tool The tool (e.g. g++, libffi) for which to create a new test summary file. If not specified then output @@ -174,6 +174,12 @@ class Prog: variants in the files for . sum-file A test summary file with the format of those created by runtest from DejaGnu. + list-file A file listing the log-or-sum files to process, one + per line. Use "-" to read the list from standard + input. This avoids the command-line length limit + when combining very many files. May be given more + than once, and may be mixed with log-or-sum-file + arguments. If -L is used, merge *.log files instead of *.sum. In this mode the exact order of lines may not be preserved, just different Running *.exp chunks should be in correct order. @@ -189,19 +195,35 @@ class Prog: # Parse the command-line arguments. def parse_cmdline (self): try: - (options, self.files) = getopt.getopt (sys.argv[1:], 'l:t:L') - if len (self.files) == 0: - self.usage() + (options, self.files) = getopt.getopt (sys.argv[1:], 'l:t:Lf:') for (option, value) in options: if option == '-l': self.variations.append (value) elif option == '-t': self.tools.append (value) + elif option == '-f': + self.read_file_list (value) else: self.do_sum = False + if len (self.files) == 0: + self.usage() except getopt.GetoptError as e: self.fatal (None, e.msg) + # Append the files listed in FILENAME, one per line, to self.files. + # FILENAME of "-" means read the list from standard input. Blank + # lines are ignored. + def read_file_list (self, filename): + f = sys.stdin if filename == '-' else open (filename, 'r') + try: + for line in f: + line = line.strip() + if line: + self.files.append (line) + finally: + if f is not sys.stdin: + f.close() + # Try to parse time string TIME, returning an arbitrary time on failure. # Getting this right is just a nice-to-have so failures should be silent. def parse_time (self, time): diff --git a/contrib/dg-extract-results.sh b/contrib/dg-extract-results.sh index 6ea897033b57..f3093f94582e 100755 --- a/contrib/dg-extract-results.sh +++ b/contrib/dg-extract-results.sh @@ -42,7 +42,7 @@ done usage() { cat <&2 -Usage: $PROGNAME [-t tool] [-l variant-list] [-L] sum-file ... +Usage: $PROGNAME [-t tool] [-l variant-list] [-L] [-f list-file] sum-file ... tool The tool (e.g. g++, libffi) for which to create a new test summary file. If not specified then all @@ -52,6 +52,11 @@ Usage: $PROGNAME [-t tool] [-l variant-list] [-L] sum-file ... variants in the files for . sum-file A test summary file with the format of those created by runtest from DejaGnu. + list-file A file listing the sum-files to process, one per line. + Use "-" to read the list from standard input. This + avoids the command-line length limit when combining + very many files. May be given more than once, and + may be mixed with sum-file arguments. If -L is used, merge *.log files instead of *.sum. In this mode the exact order of lines may not be preserved, just different Running *.exp chunks should be in correct order. @@ -69,18 +74,26 @@ msg() { VARIANTS="" TOOL="" MODE="sum" +LIST_FILES="" -while getopts "l:t:L" ARG; do +while getopts "l:t:Lf:" ARG; do case $ARG in l) VARIANTS="${VARIANTS} ${OPTARG}";; t) test -z "$TOOL" || (msg "${PROGNAME}: only one tool can be specified"; exit 1); TOOL="${OPTARG}";; L) MODE="log";; + f) if test "${OPTARG}" = "-" ; then + LIST_FILES="${LIST_FILES} `cat`" + else + LIST_FILES="${LIST_FILES} `cat "${OPTARG}"`" + fi;; \?) usage; exit 0;; esac done shift `expr ${OPTIND} - 1` +set -- ${LIST_FILES} "$@" + if test $# -lt 1 ; then usage exit 1 -- 2.54.0