RFC: parallelize "make check"

RFC: parallelize "make check"

[Tom Tromey]

This patch adds parallelization for "make check", if you are using GNU
make.  If you are not using GNU make, nothing changes.

It mostly runs each testsuite subdirectory in parallel, but it also
splits gdb.base into two parts (because gdb.base took longer than any
other subdir).

On the compile farm, with -j6, the test suite time drops from about 15
minutes to about 5.  This was good enough for me, so I didn't try to
make it any faster.

After running the tests in parallel, it combines them using a script
from gcc's contrib directory.  I just copied this into gdb/testsuite/,
but that might not be best.

Finally, if you pass any arguments via RUNTESTFLAGS, by default
parallelization is disabled.  You can set FORCE_PARALLEL to work
around this -- but this also will force all the tests to be run, so it
can't meaningfully be used in conjunction with "--directory" or
something like that.

Let me know what you think.  Barring objection I will commit this next
week.

Tom

2009-06-24  Tom Tromey  <tromey@redhat.com>

	* dg-extract-results.sh: New file.
	* Makefile.in (FORCE_PARALLEL): New variable.
	(CHECK_TARGET): New conditional variable.
	(check): Use CHECK_TARGET.
	(DO_RUNTEST): New variable.
	(check-single): New target.
	(TEST_DIRS): New variable.
	(TEST_TARGETS): Likewise.
	(check-parallel): New target.
	(check-gdb.%): New pattern.
	(BASE1_FILES): New variable.
	(BASE2_FILES): Likewise.
	(check-gdb.base%): New pattern.
	(%/.dir): New pattern.
	* configure: Rebuild.
	* aclocal.m4 (AM_CONDITIONAL): New defun.
	* configure.ac: Check whether user is using GNU make.
	(GMAKE): New conditional.

diff --git a/gdb/testsuite/Makefile.in b/gdb/testsuite/Makefile.in
index 92f1a67..5344418 100644
--- a/gdb/testsuite/Makefile.in
+++ b/gdb/testsuite/Makefile.in
@@ -48,6 +48,8 @@ RUNTEST = $(RUNTEST_FOR_TARGET)
 
 RUNTESTFLAGS =
 
+FORCE_PARALLEL =
+
 RUNTEST_FOR_TARGET = `\
   if [ -f $${srcdir}/../../dejagnu/runtest ]; then \
     echo $${srcdir}/../../dejagnu/runtest; \
@@ -122,7 +124,18 @@ site.exp: ./config.status Makefile
 
 installcheck:
 
-check: all site.exp
+# For GNU make, try to run the tests in parallel.  If RUNTESTFLAGS is
+# not empty, then by default the tests will be serialized.  This can
+# be overridden by setting FORCE_PARALLEL to any non-empty value.
+# For a non-GNU make, do not parallelize.
+@GMAKE_TRUE@CHECK_TARGET = $(if $(FORCE_PARALLEL),check-parallel,$(if $(RUNTESTFLAGS),check-single,check-parallel))
+@GMAKE_FALSE@CHECK_TARGET = check-single
+
+check: $(CHECK_TARGET)
+
+# All the hair to invoke dejagnu.  A given invocation can just append
+# $(RUNTESTFLAGS)
+DO_RUNTEST = \
 	rootme=`pwd`; export rootme; \
 	srcdir=${srcdir} ; export srcdir ; \
 	EXPECT=${EXPECT} ; export EXPECT ; \
@@ -132,7 +145,46 @@ check: all site.exp
 	if [ -f $${rootme}/../../expect/expect ] ; then  \
 	  TCL_LIBRARY=$${srcdir}/../../tcl/library ; \
 	  export TCL_LIBRARY ; fi ; \
-	$(RUNTEST) $(RUNTESTFLAGS) 
+	$(RUNTEST)
+
+check-single: all site.exp
+	$(DO_RUNTEST) $(RUNTESTFLAGS)
+
+# A list of all directories named "gdb.*" which also hold a .exp file.
+# We filter out gdb.base and add fake entries, because that directory
+# takes the longest to process, and so we split it in half.
+TEST_DIRS = gdb.base1 gdb.base2 $(filter-out gdb.base,$(sort $(notdir $(patsubst %/,%,$(dir $(wildcard $(srcdir)/gdb.*/*.exp))))))
+
+TEST_TARGETS = $(addprefix check-,$(TEST_DIRS))
+
+# We explicitly re-invoke make here for two reasons.  First, it lets
+# us add a -k option, which makes the parallel check mimic the
+# behavior of the serial check; and second, it means that we can still
+# regenerate the sum and log files even if a sub-make fails -- which
+# it usually does because dejagnu exits with an error if any test
+# fails.
+check-parallel:
+	$(MAKE) -k $(TEST_TARGETS); \
+	$(SHELL) $(srcdir)/dg-extract-results.sh \
+	  $(addsuffix /gdb.sum,$(TEST_DIRS)) > gdb.sum; \
+	$(SHELL) $(srcdir)/dg-extract-results.sh -L \
+	  $(addsuffix /gdb.log,$(TEST_DIRS)) > gdb.log
+
+$(filter-out check-gdb.base%,$(TEST_TARGETS)): check-gdb.%: all site.exp gdb.%/.dir
+	$(DO_RUNTEST) --directory=gdb.$* --outdir=gdb.$* $(RUNTESTFLAGS)
+
+# Each half (roughly) of the .exp files from gdb.base.
+BASE1_FILES = $(patsubst $(srcdir)/%,%,$(wildcard $(srcdir)/gdb.base/[a-m]*.exp))
+BASE2_FILES = $(patsubst $(srcdir)/%,%,$(wildcard $(srcdir)/gdb.base/[n-z]*.exp))
+
+# Handle each half of gdb.base.
+check-gdb.base%: all site.exp gdb.base%/.dir
+	@if test ! -d gdb.base$*; then mkdir gdb.base$*; fi
+	$(DO_RUNTEST) $(BASE$*_FILES) --outdir gdb.base$* $(RUNTESTFLAGS)
+
+%/.dir:
+	@-if test ! -d $*; then mkdir $*; fi
+	@echo > $@
 
 subdir_do: force
 	@for i in $(DODIRS); do \
diff --git a/gdb/testsuite/aclocal.m4 b/gdb/testsuite/aclocal.m4
index f58abae..2934db2 100644
--- a/gdb/testsuite/aclocal.m4
+++ b/gdb/testsuite/aclocal.m4
@@ -1,2 +1,35 @@
 sinclude(../../config/acx.m4)
 sinclude(../../config/override.m4)
+
+# AM_CONDITIONAL                                            -*- Autoconf -*-
+
+# Copyright (C) 1997, 2000, 2001, 2003, 2004, 2005
+# Free Software Foundation, Inc.
+#
+# This file is free software; the Free Software Foundation
+# gives unlimited permission to copy and/or distribute it,
+# with or without modifications, as long as this notice is preserved.
+
+# serial 7
+
+# AM_CONDITIONAL(NAME, SHELL-CONDITION)
+# -------------------------------------
+# Define a conditional.
+AC_DEFUN([AM_CONDITIONAL],
+[AC_PREREQ(2.52)dnl
+ ifelse([$1], [TRUE],  [AC_FATAL([$0: invalid condition: $1])],
+	[$1], [FALSE], [AC_FATAL([$0: invalid condition: $1])])dnl
+AC_SUBST([$1_TRUE])
+AC_SUBST([$1_FALSE])
+if $2; then
+  $1_TRUE=
+  $1_FALSE='#'
+else
+  $1_TRUE='#'
+  $1_FALSE=
+fi
+AC_CONFIG_COMMANDS_PRE(
+[if test -z "${$1_TRUE}" && test -z "${$1_FALSE}"; then
+  AC_MSG_ERROR([[conditional "$1" was never defined.
+Usually this means the macro was only invoked conditionally.]])
+fi])])
diff --git a/gdb/testsuite/configure.ac b/gdb/testsuite/configure.ac
index 3d8fae4..1a0f590 100644
--- a/gdb/testsuite/configure.ac
+++ b/gdb/testsuite/configure.ac
@@ -1,7 +1,7 @@
 #                                                       -*- Autoconf -*-
 # Process this file with autoconf to produce a configure script.
 
-# Copyright 2002, 2003, 2004, 2005
+# Copyright 2002, 2003, 2004, 2005, 2009
 # Free Software Foundation, Inc.
 #
 # This program is free software; you can redistribute it and/or modify
@@ -26,6 +26,17 @@ AC_CANONICAL_TARGET
 
 ACX_NONCANONICAL_TARGET
 
+# Check for the 'make' the user wants to use.
+AC_CHECK_PROGS(MAKE, make)
+MAKE_IS_GNU=
+case "`$MAKE --version 2>&1 | sed 1q`" in
+  *GNU*)
+    MAKE_IS_GNU=yes
+    ;;
+esac
+AM_CONDITIONAL(GMAKE, test "$MAKE_IS_GNU" = yes)
+AC_PROG_MAKE_SET
+
 # Add HP-specific tests when appropriate.
 case $target in
   hppa*-*-hpux*)
diff --git a/gdb/testsuite/dg-extract-results.sh b/gdb/testsuite/dg-extract-results.sh
new file mode 100755
index 0000000..518c19f
--- /dev/null
+++ b/gdb/testsuite/dg-extract-results.sh
@@ -0,0 +1,423 @@
+#! /bin/sh
+
+# For a specified tool and optional list of test variants, extract
+# test results from one or more test summary (.sum) files and combine
+# the results into a new test summary file, sent to the standard output.
+# The resulting file can be used with test result comparison scripts for
+# results from tests that were run in parallel.  See usage() below.
+
+# Copyright (C) 2008, 2009 Free Software Foundation
+# Contributed by Janis Johnson <janis187@us.ibm.com>
+#
+# This file is part of GCC.
+#
+# GCC is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3, or (at your option)
+# any later version.
+#
+# GCC is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with GCC; see the file COPYING.  If not, write to
+# the Free Software Foundation, 51 Franklin Street, Fifth Floor,
+# Boston, MA 02110-1301, USA.
+
+PROGNAME=dg-extract-results.sh
+
+usage() {
+  cat <<EOF >&2
+Usage: $PROGNAME [-t tool] [-l variant-list] [-L] sum-file ...
+
+    tool           The tool (e.g. g++, libffi) for which to create a
+                   new test summary file.  If not specified then all
+                   specified sum files must be for the same tool.
+    variant-list   One or more test variant names.  If the list is
+                   not specified then one is constructed from all
+                   variants in the files for <tool>.
+    sum-file       A test summary file with the format of those
+                   created by runtest from DejaGnu.
+    If -L is used, merge *.log files instead of *.sum.  In this
+    mode the exact order of lines may not be preserved, just different
+    Running *.exp chunks should be in correct order.
+EOF
+}
+
+# Write a message to the standard error.
+
+msg() {
+  echo "$@" >&2
+}
+
+# Parse the command-line options.
+
+VARIANTS=""
+TOOL=""
+MODE="sum"
+
+while getopts "l:t:L" ARG; do
+  case $ARG in
+  l)  VARIANTS="${VARIANTS} ${OPTARG}";;
+  t)  test -z "$TOOL" || (msg "${PROGNAME}: only one tool can be specified"; exit 1);
+      TOOL="${OPTARG}";;
+  L)  MODE="log";;
+  \?) usage; exit 0;;
+  esac
+done
+shift `expr ${OPTIND} - 1`
+
+if test $# -lt 1 ; then
+  usage
+  exit 1
+fi
+
+TMPDIR=${TMPDIR-/tmp}
+SUM_FILES="$@"
+FIRST_SUM=$1
+TMP=
+trap 'EXIT_STATUS=$?; rm -rf $TMP && exit $EXIT_STATUS' 0
+# Create a (secure) tmp directory for tmp files.
+{
+  TMP=`(umask 077 && mktemp -d -q "${TMPDIR}/dg-combine-results-$$-XXXXXX") 2>/dev/null` &&
+  test -n "$TMP" && test -d "$TMP"
+} ||
+{
+  TMP=${TMPDIR}/dg-combine-results-$$-$RANDOM
+  (umask 077 && mkdir $TMP)
+} ||
+{
+  msg "${PROGNAME}: cannot create a temporary directory"
+  { (exit 1); exit 1; }
+}
+
+# Find a good awk.
+
+if test -z "$AWK" ; then
+  for AWK in gawk nawk awk
+  do
+    if type $AWK 2>&1 | grep 'not found' > /dev/null 2>&1 ; then
+      :
+    else
+      break
+    fi
+  done
+fi
+
+# Verify that the specified summary files exist.
+
+ERROR=0
+for FILE in $SUM_FILES
+do
+  if ! test -f $FILE ; then
+    msg "${PROGNAME}: file $FILE does not exist."
+    ERROR=1
+  fi
+done
+test $ERROR -eq 0 || exit 1
+
+if [ -z "$TOOL" ]; then
+  # If no tool was specified, all specified summary files must be for
+  # the same tool.
+
+  CNT=`grep '=== .* tests ===' $SUM_FILES | $AWK '{ print $3 }' | sort -u | wc -l`
+  if [ $CNT -eq 1 ]; then
+    TOOL=`grep '=== .* tests ===' $FIRST_SUM | $AWK '{ print $2 }'`
+  else
+    msg "${PROGNAME}: sum files are for multiple tools, specify a tool"
+    msg ""
+    usage
+    exit 1
+  fi
+else
+  # Ignore the specified summary files that are not for this tool.  This
+  # should keep the relevant files in the same order.
+
+  SUM_FILES=`grep -l "=== $TOOL" $SUM_FILES`
+  if test -z "$SUM_FILES" ; then
+    msg "${PROGNAME}: none of the specified files are results for $TOOL"
+    exit 1
+  fi
+fi
+
+if [ "$TOOL" = acats ]; then
+  # Acats *.sum or *.log files aren't dejagnu generated, and they have
+  # somewhat different format.
+  ACATS_AWK=${TMP}/acats.awk
+  cat <<EOF > $ACATS_AWK
+BEGIN {
+  print_prologue=1; curfile=""; insummary=0
+  passcnt=0; failcnt=0; unsupcnt=0; failures=""
+}
+/^[ \t]*=== acats configuration ===/ {
+  insummary=0
+  if (print_prologue) print
+  next
+}
+/^[ \t]*=== acats tests ===/ {
+  if (print_prologue) print
+  print_prologue=0
+  next
+}
+/^Running chapter / {
+  if (curfile) close (curfile)
+  curfile="${TMP}/chapter-"\$3
+  print >> curfile
+  next
+}
+/^[ \t]*=== acats Summary ===/ {
+  if (curfile) close (curfile)
+  curfile=""
+  insummary=1
+  next
+}
+/^# of expected passes/		{ if (insummary == 1) passcnt += \$5; next; }
+/^# of unexpected failures/	{ if (insummary == 1) failcnt += \$5; next; }
+/^# of unsupported tests/	{ if (insummary == 1) unsupcnt += \$5; next; }
+/^\*\*\* FAILURES: / {
+  if (insummary == 1) {
+    if (failures) sub(/^\*\*\* FAILURES:/,"")
+    failures=failures""\$0
+  }
+}
+{
+  if (print_prologue) { print; next }
+  if (curfile) print >> curfile
+}
+END {
+  system ("cat ${TMP}/chapter-*")
+  print "		=== acats Summary ==="
+  print "# of expected passes		" passcnt
+  print "# of unexpected failures	" failcnt
+  if (unsupcnt) print "# of unsupported tests		" unsupcnt
+  if (failures) print failures
+}
+EOF
+
+  rm -f ${TMP}/chapter-*
+  $AWK -f $ACATS_AWK $SUM_FILES
+  exit 0
+fi
+
+# If no variants were specified, find all variants in the remaining
+# summary files.  Otherwise, ignore specified variants that aren't in
+# any of those summary files.
+
+if test -z "$VARIANTS" ; then
+  VAR_AWK=${TMP}/variants.awk
+  cat <<EOF > $VAR_AWK
+/^Schedule of variations:/      { in_vars=1; next }
+/^$/                            { in_vars=0 }
+/^Running target/               { exit }
+{ if (in_vars==1) print \$1; else next }
+EOF
+
+  touch ${TMP}/varlist
+  for FILE in $SUM_FILES; do
+    $AWK -f $VAR_AWK $FILE >> ${TMP}/varlist
+  done
+  VARIANTS="`sort -u ${TMP}/varlist`"
+else
+  VARS="$VARIANTS"
+  VARIANTS=""
+  for VAR in $VARS
+  do
+    grep -q "Running target $VAR" $SUM_FILES && VARIANTS="$VARIANTS $VAR"
+  done
+fi
+
+# Find out if we have more than one variant, or any at all.
+
+VARIANT_COUNT=0
+for VAR in $VARIANTS
+do
+  VARIANT_COUNT=`expr $VARIANT_COUNT + 1`
+done
+
+if test $VARIANT_COUNT -eq 0 ; then
+  msg "${PROGNAME}: no file for $TOOL has results for the specified variants"
+  exit 1
+fi
+
+cat $SUM_FILES \
+  | $AWK '/^Running/ { if ($2 != "target" && $3 == "...") print "EXPFILE: "$2 } ' \
+  | sort -u > ${TMP}/expfiles
+
+# Write the begining of the combined summary file.
+
+head -n 2 $FIRST_SUM
+echo
+echo "		=== $TOOL tests ==="
+echo
+echo "Schedule of variations:"
+for VAR in $VARIANTS
+do
+  echo "    $VAR"
+done
+echo
+
+# For each test variant for the tool, copy test reports from each of the
+# summary files.  Set up two awk scripts from within the loop to
+# initialize VAR and TOOL with the script, rather than assuming that the
+# available version of awk can pass variables from the command line.
+
+for VAR in $VARIANTS
+do
+  GUTS_AWK=${TMP}/guts.awk
+  cat << EOF > $GUTS_AWK
+BEGIN {
+  variant="$VAR"
+  firstvar=1
+  expfileno=1
+  cnt=0
+  print_using=0
+  need_close=0
+}
+/^EXPFILE: / {
+  expfiles[expfileno] = \$2
+  expfilesr[\$2] = expfileno
+  expfileno = expfileno + 1
+}
+/^Running target / {
+  curvar = \$3
+  if (variant == curvar && firstvar == 1) { print; print_using=1; firstvar = 0 }
+  next
+}
+/^Using / {
+  if (variant == curvar && print_using) { print; next }
+}
+/^Running / {
+  print_using=0
+  if (variant == curvar) {
+    if (need_close) close(curfile)
+    curfile="${TMP}/list"expfilesr[\$2]
+    expfileseen[\$2]=expfileseen[\$2] + 1
+    need_close=0
+    testname="00"
+    next
+  }
+}
+/\===/ { curvar = ""; next }
+/^(PASS|XPASS|FAIL|XFAIL|UNRESOLVED|WARNING|ERROR|UNSUPPORTED|UNTESTED|KFAIL):/ {
+  testname=\$2
+  # Ugly hack for gfortran.dg/dg.exp
+  if ("$TOOL" == "gfortran" && testname ~ /^gfortran.dg\/g77\//)
+    testname="h"testname
+}
+/^$/ { if ("$MODE" == "sum") next }
+{ if (variant == curvar && curfile) {
+    if ("$MODE" == "sum") {
+      printf "%s %08d|", testname, cnt >> curfile
+      cnt = cnt + 1
+    }
+    filewritten[curfile]=1
+    need_close=1
+    print >> curfile
+  } else
+    next
+}
+END {
+  n=1
+  while (n < expfileno) {
+    if (expfileseen[expfiles[n]]) {
+      print "Running "expfiles[n]" ..."
+      if (filewritten["${TMP}/list"n]) {
+	if (expfileseen[expfiles[n]] == 1)
+	  cmd="cat"
+	else
+	  cmd="LC_ALL=C sort"
+	if ("$MODE" == "sum")
+	  system (cmd" ${TMP}/list"n" | sed -n 's/^[^ ]* [^ |]*|//p'")
+	else
+	  system ("cat ${TMP}/list"n)
+      }
+    }
+    n = n + 1
+  }
+}
+EOF
+
+  SUMS_AWK=${TMP}/sums.awk
+  rm -f $SUMS_AWK
+  cat << EOF > $SUMS_AWK
+BEGIN {
+  variant="$VAR"
+  tool="$TOOL"
+  passcnt=0; failcnt=0; untstcnt=0; xpasscnt=0; xfailcnt=0; unsupcnt=0; unrescnt=0;
+  curvar=""; insummary=0
+}
+/^Running target /		{ curvar = \$3; next }
+/^# of /			{ if (variant == curvar) insummary = 1 }
+/^# of expected passes/		{ if (insummary == 1) passcnt += \$5; next; }
+/^# of unexpected successes/	{ if (insummary == 1) xpasscnt += \$5; next; }
+/^# of unexpected failures/	{ if (insummary == 1) failcnt += \$5; next; }
+/^# of expected failures/	{ if (insummary == 1) xfailcnt += \$5; next; }
+/^# of untested testcases/	{ if (insummary == 1) untstcnt += \$5; next; }
+/^# of unresolved testcases/	{ if (insummary == 1) unrescnt += \$5; next; }
+/^# of unsupported tests/	{ if (insummary == 1) unsupcnt += \$5; next; }
+/^$/				{ if (insummary == 1)
+				    { insummary = 0; curvar = "" }
+				  next
+				}
+{ next }
+END {
+  printf ("\t\t=== %s Summary for %s ===\n\n", tool, variant)
+  if (passcnt != 0) printf ("# of expected passes\t\t%d\n", passcnt)
+  if (xpasscnt != 0) printf ("# of unexpected successes\t%d\n", xpasscnt)
+  if (failcnt != 0) printf ("# of unexpected failures\t%d\n", failcnt)
+  if (xfailcnt != 0) printf ("# of expected failures\t\t%d\n", xfailcnt)
+  if (untstcnt != 0) printf ("# of untested testcases\t\t%d\n", untstcnt)
+  if (unrescnt != 0) printf ("# of unresolved testcases\t%d\n", unrescnt)
+  if (unsupcnt != 0) printf ("# of unsupported tests\t\t%d\n", unsupcnt)
+}
+EOF
+
+  PVAR=`echo $VAR | sed 's,/,.,g'`
+  TMPFILE=${TMP}/var-$PVAR
+  rm -f $TMPFILE
+  rm -f ${TMP}/list*
+  cat ${TMP}/expfiles $SUM_FILES | $AWK -f $GUTS_AWK
+  cat $SUM_FILES | $AWK -f $SUMS_AWK > $TMPFILE
+  # If there are multiple variants, output the counts for this one;
+  # otherwise there will just be the final counts at the end.
+  test $VARIANT_COUNT -eq 1 || cat $TMPFILE
+done
+
+# Set up an awk script to get the combined summary counts for the tool.
+
+TOTAL_AWK=${TMP}/total.awk
+cat << EOF > $TOTAL_AWK
+BEGIN {
+  tool="$TOOL"
+  passcnt=0; failcnt=0; untstcnt=0; xpasscnt=0; xfailcnt=0; unsupcnt=0; unrescnt=0
+}
+/^# of expected passes/		{ passcnt += \$5 }
+/^# of unexpected failures/	{ failcnt += \$5 }
+/^# of unexpected successes/	{ xpasscnt += \$5 }
+/^# of expected failures/	{ xfailcnt += \$5 }
+/^# of untested testcases/	{ untstcnt += \$5 }
+/^# of unresolved testcases/	{ unrescnt += \$5 }
+/^# of unsupported tests/	{ unsupcnt += \$5 }
+END {
+  printf ("\n\t\t=== %s Summary ===\n\n", tool)
+  if (passcnt != 0) printf ("# of expected passes\t\t%d\n", passcnt)
+  if (failcnt != 0) printf ("# of unexpected failures\t%d\n", failcnt)
+  if (xpasscnt != 0) printf ("# of unexpected successes\t%d\n", xpasscnt)
+  if (xfailcnt != 0) printf ("# of expected failures\t\t%d\n", xfailcnt)
+  if (untstcnt != 0) printf ("# of untested testcases\t\t%d\n", untstcnt)
+  if (unrescnt != 0) printf ("# of unresolved testcases\t%d\n", unrescnt)
+  if (unsupcnt != 0) printf ("# of unsupported tests\t\t%d\n", unsupcnt)
+}
+EOF
+
+# Find the total summaries for the tool and add to the end of the output.
+cat ${TMP}/var-* | $AWK -f $TOTAL_AWK
+
+# This is ugly, but if there's version output from the compiler under test
+# at the end of the file, we want it.  The other thing that might be there
+# is the final summary counts.
+tail -n 2 $FIRST_SUM | grep -q '^#' || tail -n 2 $FIRST_SUM
+
+exit 0

Re: RFC: parallelize "make check"

[Eli Zaretskii]

> From: Tom Tromey <tromey@redhat.com>
> Date: Wed, 24 Jun 2009 13:27:41 -0600
> 
> This patch adds parallelization for "make check", if you are using GNU
> make.  If you are not using GNU make, nothing changes.

It would be good to mention this possibility in README, IMO.

Re: RFC: parallelize "make check"

[Tom Tromey]

>>>>> "Eli" == Eli Zaretskii <eliz@gnu.org> writes:

Eli> It would be good to mention this possibility in README, IMO.

Please review.

Tom

2009-06-25  Tom Tromey  <tromey@redhat.com>

	* README: Mention parallel check.

diff --git a/gdb/README b/gdb/README
index 7116227..7feff8a 100644
--- a/gdb/README
+++ b/gdb/README
@@ -646,6 +646,15 @@ The last method gives you slightly more control in case of problems
 with building one or more test executables or if you are using the
 testsuite `standalone', without it being part of the GDB source tree.
 
+The first two methods will let you run different parts of the
+testsuite in parallel, if you use GNU make and its `-j' option.  In
+this case, if you set `RUNTESTFLAGS' then, by default, the tests will
+be run serially.  You can override this and force a parallel run by
+setting the `make' variable `FORCE_PARALLEL' to any non-empty value.
+Note that the parallel `make check' assumes that you want to run the
+entire testsuite, so it is not compatible with some dejagnu options,
+like `--directory'.
+
 See the DejaGNU documentation for further details.
 
 \f

Re: RFC: parallelize "make check"

[Eli Zaretskii]

> Cc: gdb-patches@sourceware.org
> From: Tom Tromey <tromey@redhat.com>
> Date: Thu, 25 Jun 2009 14:21:43 -0600
> 
> >>>>> "Eli" == Eli Zaretskii <eliz@gnu.org> writes:
> 
> Eli> It would be good to mention this possibility in README, IMO.
> 
> Please review.

Thanks.  A couple of comments:

 . You mention RUNTESTFLAGS that is not otherwise documented in
   README.  I understand that you mention it (and the solution via
   setting RUNTESTFLAGS) as ``gotcha'' for those who use this variable
   for reasons unrelated to -j.  If so, I would suggest to make it an
   explicit note, something like this (the parens are deliberate):

    (Note that setting `RUNTESTFLAGS' then, by default, the tests will
    be run serially even under -j.  You can override this and force a
    parallel run ...)

 . I think we should say that using -j significantly cuts the time for
   running the test suite.  Otherwise, the reader would not
   necessarily understand why she would like to use -j.

Re: RFC: parallelize "make check"

[Tom Tromey]

>>>>> "Eli" == Eli Zaretskii <eliz@gnu.org> writes:

Eli>  . You mention RUNTESTFLAGS that is not otherwise documented in
Eli>    README.

It seemed better to document this.

Tom


diff --git a/gdb/README b/gdb/README
index 7116227..57f600a 100644
--- a/gdb/README
+++ b/gdb/README
@@ -642,6 +642,20 @@ or
 	make site.exp	(builds the site specific file)
 	runtest -tool gdb GDB=../gdb    (or GDB=<somepath> as appropriate)
 
+When using a `make'-based method, you can use the Makefile variable
+`RUNTESTFLAGS' to pass flags to `runtest', e.g.:
+
+	make RUNTESTFLAGS=--directory=gdb.cp check
+
+If you use GNU make, you can use its `-j' option to run the testsuite
+in parallel.  This can greatly reduce the amount of time it takes for
+the testsuite to run.  In this case, if you set `RUNTESTFLAGS' then,
+by default, the tests will be run serially even under `-j'.  You can
+override this and force a parallel run by setting the `make' variable
+`FORCE_PARALLEL' to any non-empty value.  Note that the parallel `make
+check' assumes that you want to run the entire testsuite, so it is not
+compatible with some dejagnu options, like `--directory'.
+
 The last method gives you slightly more control in case of problems
 with building one or more test executables or if you are using the
 testsuite `standalone', without it being part of the GDB source tree.

Re: RFC: parallelize "make check"

[Eli Zaretskii]

> Cc: gdb-patches@sourceware.org
> From: Tom Tromey <tromey@redhat.com>
> Date: Fri, 26 Jun 2009 09:56:06 -0600
> 
> >>>>> "Eli" == Eli Zaretskii <eliz@gnu.org> writes:
> 
> Eli>  . You mention RUNTESTFLAGS that is not otherwise documented in
> Eli>    README.
> 
> It seemed better to document this.

Thanks, this version is fine.

Re: RFC: parallelize "make check"

[Joel Brobecker]

Hi Tom,

> 2009-06-24  Tom Tromey  <tromey@redhat.com>
> 
> 	* dg-extract-results.sh: New file.
> 	* Makefile.in (FORCE_PARALLEL): New variable.
> 	(CHECK_TARGET): New conditional variable.
> 	(check): Use CHECK_TARGET.
> 	(DO_RUNTEST): New variable.
> 	(check-single): New target.
> 	(TEST_DIRS): New variable.
> 	(TEST_TARGETS): Likewise.
> 	(check-parallel): New target.
> 	(check-gdb.%): New pattern.
> 	(BASE1_FILES): New variable.
> 	(BASE2_FILES): Likewise.
> 	(check-gdb.base%): New pattern.
> 	(%/.dir): New pattern.
> 	* configure: Rebuild.
> 	* aclocal.m4 (AM_CONDITIONAL): New defun.
> 	* configure.ac: Check whether user is using GNU make.
> 	(GMAKE): New conditional.

This looks awesome. If I understand your patch correctly,
parallelization is active by default when one does "make -j6 check"
as you split the "check" target into multiple sub targets.
If one does a regular "make check" without the -j, then the sub
targets will simply be run in sequence.

Thanks for doing this. At this rate, I don't think you'll be able
to remain sober at the next GCC Summit :). (Editor's note: Daniel
and now myself promised Tom a drink for various very useful patches
that he wrote).

-- 
Joel

Re: RFC: parallelize "make check"

[Tom Tromey]

>>>>> "Joel" == Joel Brobecker <brobecker@adacore.com> writes:

Joel> This looks awesome. If I understand your patch correctly,
Joel> parallelization is active by default when one does "make -j6 check"
Joel> as you split the "check" target into multiple sub targets.
Joel> If one does a regular "make check" without the -j, then the sub
Joel> targets will simply be run in sequence.

Yeah, that's right.

Joel> Thanks for doing this. At this rate, I don't think you'll be able
Joel> to remain sober at the next GCC Summit :). (Editor's note: Daniel
Joel> and now myself promised Tom a drink for various very useful patches
Joel> that he wrote).

:)

Though in this case I took the idea, if not the implementation, from
Jakub's similar patch to GCC.

Tom

Re: RFC: parallelize "make check"

[Pedro Alves]

On Wednesday 24 June 2009 20:27:41, Tom Tromey wrote:
> After running the tests in parallel, it combines them using a script
> from gcc's contrib directory.  I just copied this into gdb/testsuite/,
> but that might not be best.

Of course, the best solution is to put gcc/contrib in its own git
repository and share it with gdb.  :-P

[[[ seriously, am I the only one that would prefer that gdb and
the whole of src move into the same repository as gcc, which
would allow easier sharing of code between the projects, and
more easily spring up new shared libraries?  Think reusing
bits of gcc in gdb :-) (other than libiberty).  Wasn't svn gaining
support for modules a-la cvs? ]]]

Okay, sorry for hijacking.  Please ignore.  Great stuff, Tom!

-- 
Pedro Alves

Re: RFC: parallelize "make check"

[Daniel Jacobowitz]

On Thu, Jun 25, 2009 at 11:59:02PM +0100, Pedro Alves wrote:
> bits of gcc in gdb :-) (other than libiberty).  Wasn't svn gaining
> support for modules a-la cvs? ]]]

Not adequately.

-- 
Daniel Jacobowitz
CodeSourcery

Re: RFC: parallelize "make check"

[Tom Tromey]

>>>>> "Pedro" == Pedro Alves <pedro@codesourcery.com> writes:

Pedro> [[[ seriously, am I the only one that would prefer that gdb and
Pedro> the whole of src move into the same repository as gcc, which
Pedro> would allow easier sharing of code between the projects, and
Pedro> more easily spring up new shared libraries?  Think reusing
Pedro> bits of gcc in gdb :-) (other than libiberty).

I daydreamed a bit about putting parts of the front ends into gdb...
correct expression parsing!  A C interpreter!  We could get statement
expressions to work!

In the end I figured that the programs are different enough that the
work would be massive, and the payoff not large enough.

I don't know what else from gcc I'd want to use.

Tom

Re: RFC: parallelize "make check"

[Joseph S. Myers]

On Fri, 26 Jun 2009, Tom Tromey wrote:

> I don't know what else from gcc I'd want to use.

Constant folding support for arithmetic on target types would be nice to 
share, rather than the GDB kludges of using host floating point types as a 
bad approximation to target types and duplicating floating point format 
handling code for GCC and GDB.  But it would also be desirable to make 
more use of MPFR in GCC for this in place of its own code, so sharing with 
GDB would introduce a probably unwanted MPFR dependency to GDB (and in due 
course an MPC dependency for complex arithmetic as well), and any move of 
GCC code to C++ would also make sharing harder unless GDB also moves.

-- 
Joseph S. Myers
joseph@codesourcery.com

Re: RFC: parallelize "make check"

[Joel Brobecker]

> In the end I figured that the programs are different enough that the
> work would be massive, and the payoff not large enough.

Another aspect that we've considered as well for Ada is to be
more relaxed about the expressions we accept in GDB, compared to
what the language actually permits.

-- 
Joel

Re: RFC: parallelize "make check"

[Tom Tromey]

>>>>> "Tom" == Tom Tromey <tromey@redhat.com> writes:

Tom> This patch adds parallelization for "make check", if you are using GNU
Tom> make.  If you are not using GNU make, nothing changes.

I checked this in.

Just FYI, Jan mentioned on irc that this made some existing test suite
instability worse.

Tom

Re: RFC: parallelize "make check"

[Daniel Jacobowitz]

On Mon, Jun 29, 2009 at 10:43:30AM -0600, Tom Tromey wrote:
> I checked this in.

Great!

> Just FYI, Jan mentioned on irc that this made some existing test suite
> instability worse.

I've been having a lot of trouble with foll-fork.exp printing output
after prompts.

-- 
Daniel Jacobowitz
CodeSourcery

Re: RFC: parallelize "make check"

[Tom Tromey]

>>>>> "Daniel" == Daniel Jacobowitz <drow@false.org> writes:

Tom> Just FYI, Jan mentioned on irc that this made some existing test suite
Tom> instability worse.

Daniel> I've been having a lot of trouble with foll-fork.exp printing output
Daniel> after prompts.

I hadn't heard about that one... Jan?

What he did mention to me is:

    gdb.mi/mi-file-transfer.exp, gdb.server/file-transfer.exp,
    gdb.base/focus-cmd-prev.exp, gdb.xml/tdesc-regs.exp,
    gdb.base/watchpoint.exp and some others already filtered out randomly
    FAIL when run in parallel

Tom

Re: RFC: parallelize "make check"

[Daniel Jacobowitz]

On Mon, Jun 29, 2009 at 12:05:10PM -0600, Tom Tromey wrote:
> What he did mention to me is:
> 
>     gdb.mi/mi-file-transfer.exp, gdb.server/file-transfer.exp,
>     gdb.base/focus-cmd-prev.exp, gdb.xml/tdesc-regs.exp,
>     gdb.base/watchpoint.exp and some others already filtered out randomly
>     FAIL when run in parallel

Interesting, I was not aware of intermittent failures in these files.
I've never seen them that I can remember.

-- 
Daniel Jacobowitz
CodeSourcery

Re: RFC: parallelize "make check"

[Jan Kratochvil]

On Mon, 29 Jun 2009 20:05:10 +0200, Tom Tromey wrote:
> Daniel> I've been having a lot of trouble with foll-fork.exp printing output
> Daniel> after prompts.
> 
> I hadn't heard about that one... Jan?

This one is also a common false difference.

> What he did mention to me is:
> 
>     gdb.mi/mi-file-transfer.exp, gdb.server/file-transfer.exp,
>     gdb.base/focus-cmd-prev.exp, gdb.xml/tdesc-regs.exp,
>     gdb.base/watchpoint.exp and some others already filtered out randomly
>     FAIL when run in parallel

Tried now the new parallel make check with no instabilities on CVS HEAD.

Still getting instable results on gdb-6.8.50.20090302 with the Fedora patchset,
using the diff -I options below to make it a bit comparable.

-ERROR: Process no longer exists
-UNRESOLVED: gdb.base/focus-cmd-prev.exp: focus cmd
-ERROR: Couldn't send focus prev to GDB.
-UNRESOLVED: gdb.base/focus-cmd-prev.exp: focus prev
+PASS: gdb.base/focus-cmd-prev.exp: focus cmd
+PASS: gdb.base/focus-cmd-prev.exp: focus prev

 Running ../../../gdb/testsuite/gdb.server/file-transfer.exp ...
-PASS: gdb.server/file-transfer.exp: put binary file
-PASS: gdb.server/file-transfer.exp: get binary file
-PASS: gdb.server/file-transfer.exp: compare intermediate binary file
-PASS: gdb.server/file-transfer.exp: compare binary file
-PASS: gdb.server/file-transfer.exp: deleted binary file
-PASS: gdb.server/file-transfer.exp: verified deleted binary file
-PASS: gdb.server/file-transfer.exp: put text file
-PASS: gdb.server/file-transfer.exp: get text file
-PASS: gdb.server/file-transfer.exp: compare intermediate text file
-PASS: gdb.server/file-transfer.exp: compare text file
-PASS: gdb.server/file-transfer.exp: deleted text file
-PASS: gdb.server/file-transfer.exp: verified deleted text file
+ERROR: tcl error sourcing ../../../gdb/testsuite/gdb.server/file-transfer.exp.
+ERROR: : spawn id exp18 not open
+    while executing
+"expect_background -nobrace -i exp18 full_buffer { } eof {
+           # The spawn ID is already closed now (but not yet waited for).
+           wait -i $expect_out..."
+    invoked from within
+"expect_background {
+       -i $server_spawn_id
+       full_buffer { }
+       eof {
+           # The spawn ID is already closed now (but not yet waited for).
+           wait -i $exp..."
+    (procedure "gdbserver_start" line 54)
+    invoked from within
+"gdbserver_start "" $arguments"
+    (procedure "gdbserver_spawn" line 11)
+    invoked from within
+"gdbserver_spawn $child_args"
+    (procedure "gdbserver_run" line 20)
+    invoked from within
+"gdbserver_run """
+    (file "../../../gdb/testsuite/gdb.server/file-transfer.exp" line 39)
+    invoked from within
+"source ../../../gdb/testsuite/gdb.server/file-transfer.exp"
+    ("uplevel" body line 1)
+    invoked from within
+"uplevel #0 source ../../../gdb/testsuite/gdb.server/file-transfer.exp"
+    invoked from within
+"catch "uplevel #0 source $test_file_name""

etc.


I do not know more now, it is a bit difficult to catch to debug it.


Regards,
Jan


#! /bin/sh
exec diff \
	-I '^Test Run By ' \
	-I '^gnatbind ' \
	-I '^gnatlink ' \
	-I '^gnatmake: ' \
	-I '^gcc ' \
	-I '^UNSUPPORTED: ' \
	-I '^UNTESTED: ' \
	-I '^UNRESOLVED: ' \
	-I '^ERROR: ' \
	-I '^WARNING: ' \
	-I '^\(PASS\|FAIL\): gdb.server/ext-run.exp: get process list$' \
	-I '^\(PASS\|FAIL\): gdb.base/checkpoint.exp: info checkpoints with at least 600 checkpoints$' \
	-I '^\(PASS\|FAIL\): gdb.base/checkpoint.exp: break2 with many checkpoints$' \
	-I '^\(PASS\|FAIL\): gdb.base/interrupt.exp: ' \
	-I '^\(PASS\|FAIL\): gdb.base/gdb1250.exp: ' \
	-I '^KFAIL: gdb.threads/watchthreads2.exp: gdb can drop watchpoints in multithreaded app (PRMS: gdb/10116)$' \
	-I '^PASS: gdb.threads/watchthreads2.exp: all threads incremented x$' \
	-I '^FAIL: gdb.base/sigbpt.exp: Verify that SIGSEGV occurs at the last STEPI insn (none 0x[a-f0-9]*)$' \
	-I '^PASS: gdb.threads/threadcrash.exp: core file: ' \
	-I '^PASS: gdb.threads/watchthreads.exp: disable [0-9]*$' \
	-I '^FAIL: gdb.mi/mi-nsmoribund.exp: unexpected stop$' \
	-I '^PASS: gdb.mi/mi-nsmoribund.exp: resume all, program exited normally$' \
	-I '^gdb compile failed, ' \
	-I '^gdb compile failed, /usr/bin/ld: /tmp/[a-zA-Z0-9]*.o: relocation R_X86_64_32S against `a local symbol. can not be used when making a shared object; recompile with -fPIC$' \
	-I '^/tmp/[a-zA-Z0-9]*.o: could not read symbols: Bad value$' \
	-I '^# of ' \
	-I '^/home/' \
	-I '^PASS: gdb.arch/i386-biarch-core.exp: core-file /home/.*/i386-biarch-core.core$' \
-ru "$@" | grep -v '^diff '

[patch] testsuite: Fix multiple runs in parallel on a single host [Re: RFC: parallelize "make check"]

[Jan Kratochvil]

On Mon, 29 Jun 2009 23:16:57 +0200, Jan Kratochvil wrote:
> On Mon, 29 Jun 2009 20:05:10 +0200, Tom Tromey wrote:
> >     gdb.base/watchpoint.exp
(This is a FSF GDB HEAD problem watchpoints are not distributed across threads.)


> >     gdb.mi/mi-file-transfer.exp, gdb.server/file-transfer.exp,

These + others get fixed by the patch below.

>  Running ../../../gdb/testsuite/gdb.server/file-transfer.exp ...
> -PASS: gdb.server/file-transfer.exp: put binary file
> -PASS: gdb.server/file-transfer.exp: get binary file
...
> +ERROR: tcl error sourcing ../../../gdb/testsuite/gdb.server/file-transfer.exp.
> +ERROR: : spawn id exp18 not open
> +    while executing
> +"expect_background -nobrace -i exp18 full_buffer { } eof {
...

It normally mostly always happens with multiple parallel testsuite runs on the
same host.  Now with the parallel check there is also a (low probability) it
would bite even during a single testsuite parallel run.

Regression tested on FSF GDB HEAD on {x86_64,i686}-fedora-linux-gnu.


Thanks,
Jan


gdb/testsuite/
2009-07-01  Jan Kratochvil  <jan.kratochvil@redhat.com>

	* lib/gdbserver-support.exp (gdbserver_start): Loop spawning
	gdbserver increasing $portnum if "Can't bind address" has been seen.

--- a/gdb/testsuite/lib/gdbserver-support.exp
+++ b/gdb/testsuite/lib/gdbserver-support.exp
@@ -209,26 +209,39 @@ proc gdbserver_start { options arguments } {
 
     set gdbserver [find_gdbserver]
 
-    # Export the host:port pair.
-    set gdbport $debughost$portnum
-
-    # Fire off the debug agent.
-    set gdbserver_command "$gdbserver"
-    if { $options != "" } {
-	append gdbserver_command " $options"
-    }
-    append gdbserver_command " :$portnum"
-    if { $arguments != "" } {
-	append gdbserver_command " $arguments"
-    }
-
-    set server_spawn_id [remote_spawn target $gdbserver_command]
+    # Loop till we find a free port.
+    while 1 {
+	# Export the host:port pair.
+	set gdbport $debughost$portnum
+
+	# Fire off the debug agent.
+	set gdbserver_command "$gdbserver"
+	if { $options != "" } {
+	    append gdbserver_command " $options"
+	}
+	append gdbserver_command " :$portnum"
+	if { $arguments != "" } {
+	    append gdbserver_command " $arguments"
+	}
 
-    # Wait for the server to open its TCP socket, so that GDB can connect.
-    expect {
-	-i $server_spawn_id
-	-notransfer
-	-re "Listening on" { }
+	set server_spawn_id [remote_spawn target $gdbserver_command]
+
+	# Wait for the server to open its TCP socket, so that GDB can connect.
+	expect {
+	    -i $server_spawn_id
+	    -notransfer
+	    -re "Listening on" { }
+	    -re "Can't bind address: Address already in use\\.\r\n" {
+		verbose -log "Port $portnum is already in use."
+		if ![target_info exists gdb,socketport] {
+		    # Bump the port number to avoid the conflict.
+		    wait -i $expect_out(spawn_id)
+		    incr portnum
+		    continue
+		}
+	    }
+	}
+	break
     }
 
     # We can't just call close, because if gdbserver is local then that means

Re: [patch] testsuite: Fix multiple runs in parallel on a single host [Re: RFC: parallelize "make check"]

[Tom Tromey]

>>>>> "Jan" == Jan Kratochvil <jan.kratochvil@redhat.com> writes:

Jan> 2009-07-01  Jan Kratochvil  <jan.kratochvil@redhat.com>
Jan> 	* lib/gdbserver-support.exp (gdbserver_start): Loop spawning
Jan> 	gdbserver increasing $portnum if "Can't bind address" has been seen.

This looks reasonable, if unfortunate, to me.
Ok.

Tom

Re: [patch] testsuite: Fix multiple runs in parallel on a single host

[Jan Kratochvil]

On Mon, 06 Jul 2009 20:38:47 +0200, Tom Tromey wrote:
> >>>>> "Jan" == Jan Kratochvil <jan.kratochvil@redhat.com> writes:
> 
> Jan> 2009-07-01  Jan Kratochvil  <jan.kratochvil@redhat.com>
> Jan> 	* lib/gdbserver-support.exp (gdbserver_start): Loop spawning
> Jan> 	gdbserver increasing $portnum if "Can't bind address" has been seen.
> 
> This looks reasonable, if unfortunate, to me.
> Ok.

Checked-in:
	http://sourceware.org/ml/gdb-cvs/2009-07/msg00051.html


Thanks,
Jan