Re: [PATCH 1/1] [gdb/contrib] Add make-check-all.sh

[Tom de Vries via Gdb-patches <gdb-patches@sourceware.org>]

[-- Attachment #1: Type: text/plain, Size: 2860 bytes --]

On 4/4/23 13:35, Andrew Burgess wrote:
> Tom de Vries via Gdb-patches <gdb-patches@sourceware.org> writes:
> 
>> Add script gdb/contrib/make-check-all.sh, that's intended to function as a
>> drop-in replacement of make check, but excercising all host/target boards in
>> gdb/testsuite/boards.
>>
>> Shell-checked and tested on x86_64-linux.
> 
> Hi Tom,
> 
> Thanks for putting this together, I think this could be really useful.
> 

That's great to hear :)

> I'm not a fan of the way you've split the patch description into email
> 0/1 and not included it with this commit.  I think there's lots of
> useful information in there, and I'd much rather have the whole
> description included in the commit message -- it's much easier to find
> then rather than having to hunt on the mailing list in the future.
> 

Ack, I've reworked this into a patch rather than patch series.

>> ---
>>   gdb/contrib/make-check-all.sh | 255 ++++++++++++++++++++++++++++++++++
>>   1 file changed, 255 insertions(+)
>>   create mode 100755 gdb/contrib/make-check-all.sh
>>
>> diff --git a/gdb/contrib/make-check-all.sh b/gdb/contrib/make-check-all.sh
>> new file mode 100755
>> index 00000000000..1befe418b81
>> --- /dev/null
>> +++ b/gdb/contrib/make-check-all.sh
>> @@ -0,0 +1,255 @@
>> +#!/bin/bash
>> +
>> +# Copyright (C) 2023 Free Software Foundation, Inc.
>> +# This program is free software; you can redistribute it and/or modify
>> +# it under the terms of the GNU General Public License as published by
>> +# the Free Software Foundation; either version 3 of the License, or
>> +# (at your option) any later version.
>> +#
>> +# This program is distributed in the hope that it will be useful,
>> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
>> +# GNU General Public License for more details.
>> +#
>> +# You should have received a copy of the GNU General Public License
>> +# along with this program.  If not, see <http://www.gnu.org/licenses/>.
>> +
>> +# Run make check with all boards from gdb/testsuite/boards.
> 
> I think you need some kind of usage test here too.  Imagine a new
> developer finds this file and looks inside.  Ideally, I think they
> should know how to use it, and what it will do for them without having
> to read any of the actual script content.
> 

Added.

> On implementation, I wonder if it would be useful to provide a mechanism
> by which the gdb.sum and gdb.log files for each test run could be
> preserved?
> 

Done, I've renamed --keep to --keep-tmp, and added this new 
functionality under --keep.

I've also added "set -e" to make sure ^C aborts the script rather than 
just one make check invocation, and moved the tmpdir removal to a 
cleanup function that's executed on exit, to make sure ^C doesn't leave 
tmpdirs behind.

Thanks,
- Tom


[-- Attachment #2: 0001-gdb-contrib-Add-make-check-all.sh.patch --]
[-- Type: text/x-patch, Size: 7218 bytes --]

From 8647ef0c3dd076ee18d1060882552a20a016b2d2 Mon Sep 17 00:00:00 2001
From: Tom de Vries <tdevries@suse.de>
Date: Mon, 3 Apr 2023 03:47:46 +0200
Subject: [PATCH] [gdb/contrib] Add make-check-all.sh

Directory gdb/testsuite/boards contains a number of host/target boards, which
run a test-case (or test-cases) in a different way.

The benefits of using these boards are:
- improving test coverage of gdb,
- making the testsuite more robust, and
- making sure the test-cases work for non-native and remote setups, if
  possible.

Each board is slightly different, and developers need to learn how to use each
one, what parameters to pass and how, and which ones can be using in
combination with each other.  This is a threshold to start using them.

And then there quite a few, so I suppose typically only a few will be used by
each developer.

Add script gdb/contrib/make-check-all.sh, that's intended to function as a
drop-in replacement of make check, while excercising all host/target boards in
gdb/testsuite/boards.

An an example of make-check-all.sh for one test-case is:
...
 $  ~/gdb/src/gdb/contrib/make-check-all.sh gdb.base/advance.exp
 LOCAL:
 # of expected passes            8
 TARGET BOARD: cc-with-gdb-index
 # of expected passes            8
   ...
 HOST BOARD: local-remote-host-notty, TARGET BOARD: remote-stdio-gdbserver
 # of expected passes            8
 HOST/TARGET BOARD: local-remote-host-native
 # of expected passes            8
...

Shell-checked and tested on x86_64-linux.
---
 gdb/contrib/make-check-all.sh | 299 ++++++++++++++++++++++++++++++++++
 1 file changed, 299 insertions(+)
 create mode 100755 gdb/contrib/make-check-all.sh

diff --git a/gdb/contrib/make-check-all.sh b/gdb/contrib/make-check-all.sh
new file mode 100755
index 00000000000..0d9f983302f
--- /dev/null
+++ b/gdb/contrib/make-check-all.sh
@@ -0,0 +1,299 @@
+#!/bin/bash
+
+# Copyright (C) 2023 Free Software Foundation, Inc.
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+# Run make check with all boards from gdb/testsuite/boards.
+
+# Note that running with the --host-user and --target-user options is
+# recommended, because:
+# - remote host/target boards will use $HOME and leave (potentially lots of)
+#   files behind, and
+# - it enabled more strict checking of build/host/target file manipulations.
+#
+# Recommended usage example:
+#
+# bash$ cd $objdir/gdb/testsuite
+# bash$ $srcdir/contrib/gdb/make-check-all.sh --host-user remote-host \
+#           --target-user remote-target gdb.base/advance.exp
+
+set -e
+
+host_boards=(
+    local-remote-host
+    local-remote-host-notty
+)
+
+gdbserver_boards=(
+    native-extended-gdbserver
+    native-gdbserver
+    native-stdio-gdbserver
+)
+
+remote_gdbserver_boards=(
+    remote-gdbserver-on-localhost
+    remote-stdio-gdbserver
+)
+
+host_target_boards=(
+    local-remote-host-native
+)
+
+target_boards=(
+    cc-with-gdb-index
+    cc-with-debug-names
+    cc-with-dwz
+    cc-with-dwz-m
+    cc-with-gnu-debuglink
+    debug-types
+    dwarf4-gdb-index
+    dwarf64
+    fission
+    fission-dwp
+    gold
+    gold-gdb-index
+    readnow
+    stabs
+)
+
+rtf_for_board ()
+{
+    local b
+    b="$1"
+
+    case $b in
+	local-remote-host-native)
+	    mkdir -p "$tmpdir/$b"
+	    rtf=(
+		"${rtf[@]}"
+		"HOST_DIR=$tmpdir/$b"
+	    )
+	    ;;
+	remote-stdio-gdbserver)
+	    rtf=(
+		"${rtf[@]}"
+		"REMOTE_HOSTNAME=localhost"
+	    )
+	    if [ "$target_user" != "" ]; then
+		rtf=(
+		    "${rtf[@]}"
+		    "REMOTE_USERNAME=$target_user"
+		)
+	    else
+		rtf=(
+		    "${rtf[@]}"
+		    "REMOTE_USERNAME=$USER"
+		)
+	    fi
+	    ;;
+	remote-gdbserver-on-localhost)
+	    if [ "$target_user" != "" ]; then
+		rtf=(
+		    "${rtf[@]}"
+		    "REMOTE_TARGET_USERNAME=$target_user"
+		)
+	    fi
+	    ;;
+	local-remote-host|local-remote-host-notty)
+	    if [ "$host_user" != "" ]; then
+		rtf=(
+		    "${rtf[@]}"
+		    "REMOTE_HOST_USERNAME=$host_user"
+		)
+	    else
+		rtf=(
+		    "${rtf[@]}"
+		    "REMOTE_HOST_USERNAME=$USER"
+		)
+	    fi
+	    ;;
+	*)
+	    ;;
+    esac
+}
+
+summary ()
+{
+    if $verbose; then
+	cat
+    else
+	grep -E "^(#|FAIL:|ERROR:|WARNING:)" \
+	    | sort -u
+    fi
+}
+
+do_tests ()
+{
+    if $debug; then
+	echo "RTF: ${rtf[*]}"
+    fi
+
+    if $dry_run; then
+	return
+    fi
+
+    make check \
+	 RUNTESTFLAGS="${rtf[*]} ${tests[*]}" \
+	 2>&1 \
+	| summary
+
+    if [ "$h" = "" ]; then
+	if [ "$b" = "" ]; then
+	    cfg=local
+	else
+	    cfg=$b
+	fi
+    else
+	cfg=$h-$b
+    fi
+
+    local dir
+    dir="check-all/$cfg"
+
+    if $keep; then
+	mkdir -p "$dir"
+	cp gdb.sum gdb.log "$dir"
+    fi
+}
+
+parse_args ()
+{
+    debug=false
+    keep=false
+    keep_tmp=false
+    verbose=false
+    dry_run=false
+
+    host_user=""
+    target_user=""
+
+    while [ $# -gt 0 ]; do
+	case "$1" in
+	    --debug)
+		debug=true
+		;;
+	    --keep)
+		keep=true
+		;;
+	    --keep-tmp)
+		keep_tmp=true
+		;;
+	    --verbose)
+		verbose=true
+		;;
+	    --dry-run)
+		dry_run=true
+		;;
+	    --host-user)
+		shift
+		host_user="$1"
+		;;
+	    --target-user)
+		shift
+		target_user="$1"
+		;;
+	    *)
+		break
+		;;
+	esac
+	shift
+    done
+
+    tests=("$@")
+}
+
+cleanup ()
+{
+    if [ "$tmpdir" != "" ]; then
+        if $keep_tmp; then
+	    echo "keeping tmp dir $tmpdir"
+	else	
+	    rm -Rf "$tmpdir"
+	fi
+    fi
+}
+
+main ()
+{
+    parse_args "$@"
+
+    tmpdir=""
+    trap cleanup EXIT
+    tmpdir=$(mktemp -d)
+
+    if $debug; then
+	echo "TESTS: ${tests[*]}"
+    fi
+
+    h=""
+    b=""
+
+    if true; then
+	echo "LOCAL:"
+	rtf=()
+	do_tests
+    fi
+
+    if true; then
+	for b in "${target_boards[@]}"; do
+	    echo "TARGET BOARD: $b"
+	    rtf=(
+		--target_board="$b"
+	    )
+	    rtf_for_board "$b"
+	    do_tests
+	done
+    fi
+
+    if true; then
+	for b in "${gdbserver_boards[@]}" "${remote_gdbserver_boards[@]}"; do
+	    echo "TARGET BOARD: $b"
+	    rtf=(
+		--target_board="$b"
+	    )
+	    rtf_for_board "$b"
+	    do_tests
+	done
+    fi
+
+    if true; then
+	for h in "${host_boards[@]}"; do
+	    for b in "${remote_gdbserver_boards[@]}"; do
+		echo "HOST BOARD: $h, TARGET BOARD: $b"
+		rtf=(
+		    --host_board="$h"
+		    --target_board="$b"
+		)
+		rtf_for_board "$h"
+		rtf_for_board "$b"
+		do_tests
+	    done
+	done
+	h=""
+    fi
+
+    if true; then
+	for b in "${host_target_boards[@]}"; do
+	    echo "HOST/TARGET BOARD: $b"
+	    rtf=(
+		--host_board="$b"
+		--target_board="$b"
+	    )
+	    rtf_for_board "$b"
+	    do_tests
+	done
+    fi
+}
+
+main "$@"

base-commit: 9278aa6171edd477326f1a73a112d98b8b785954
-- 
2.35.3