[PATCH v2] gdb/testsuite: add support for running tests using QEMU

[Jan Vrany <jan@vrany.io>]

>  I think that is really cool, I am in favor of adding it.  Naming nit:
>  perhaps name it qemu-user, I could see us also adding a qemu-system
>  board to test bare metal.  For example, I saw that qemu now has an AVR
>  port, I would like to make a board based on that.

OK, I renamed it. I just tried to save me some typing :-)

>
>  Do you pass a -L to the qemu user binary, or set QEMU_LD_PREFIX?  I
>  found it necessary for the right dynamic linker to be found:
>
>      $ qemu-aarch64 ./a.out
>      qemu-aarch64: Could not open '/lib/ld-linux-aarch64.so.1': No such file or directory                                                                                                                          /0.0s
>      $ qemu-aarch64 -L /usr/aarch64-linux-gnu  ./a.out
>      Hello!
>
>  If it's always going to be required to pass one or the other, could the
>  board require that the user specify it, or could it try to guess it
>  (e.g. "/usr/$triplet")?

AFAIK no. If no QEMU_LD_PREFIX is set nor -L passed, QEMU uses default,
compiled-in path (if it exists). The value of that default depends, for
example in recent Debian it is /usr/gnemul/qemu-aarch64 (/usr/gnemul/qemu-riscv64, ...).
On my system I have things setup so the "minimal sysroot" is in that it
"just works".

If one wants to use different prefix, he/she can either set QEMU_LD_PREFIX
or QEMU_FLAGS accordingly.

In v2 (see below) I have added comment in the qemu-user.exp to clarify
this. I have also added a way to force static compilation - this still
allows running large set of tests without need to for dynamic linker.

>
>  I ran your patch through Claude, most of the important comments were
>  about robustnessof the "non-happy" paths, which in my experience is very
>  important when you're trying to use the board and figuring things out.
>  I would suggest giving all the failure modes a try to see if the error
>  reported is sensible and the user experience is good.
>

Thanks. I hope I addressed all issues in v2 except following:

>
>   - gdb/testsuite/boards/qemu.exp:179: `exit 0` terminates the runtest
>     process, not just the current test.  In a serial run, all the
>     remaining tests are skipped silently.  Use `unsupported "..."` with a
>     message and `return -1` instead; gdb_run_cmd already handles a
>     non-zero return (lib/gdb.exp:549).

This is intentional. My understanding is that in GDB testsuite there's
no distinction between test "setup" and tests itself in a sense that if
setup fails, then the tests are meaningless. So in one fail to connect
to the target, the failure is noted and testing proceeds, leading to
inevitable timeouts and tests are recorded as failures. The "exit 0" here
is to avoid these timeouts for tests that cannot work.

It may cause some tests that could work to be silently skipped but seems
to me as an unfortunate consequence of lack of setup-test-teardown
structure in GDB tests.

If simply returning a failure and let the rest of tests to timeout is
preferable, I have no problem.

Thanks!
Jan

-- 8< --
gdb/testsuite: add support for running tests using QEMU

While debugging failing RISC-V tests, I found myself once again in a
situation where running tests easily using QEMU usermode emulation
would be useful. Since this is something that resurfaces every now and
then [1], I decided to have another stab at it.

This commit adds a new "board" file - qemu-user.exp - that allows running
tests using QEMU usermode emulation. The main goal is to make it easier
to test changes on multiple different architectures. However using QEMU
to run tests is no substitute for running tests on real hardware (or
in a VM). See limitations below.

The implementation based on simavr.exp and on an earlier patch by
Zeck S [2] and somewhat simplified. Unlike previous attempts, test
programs are no compiled statically by default in order to support
broader set of testcases. A static compilation can be forced by setting
CFLAGS_FOR_TARGET=-static. It also supports more targets (new targets
should be easy to add).

Not all test that pass on real hardware pass when run under usermode
QEMU. Some failures stem from limitations of QEMU's GDB stub - it does
not support non-stop mode, multiple threads, hardware watchpoints. In
some other cases GDB crashes, perhaps due to a bug in either GDB or QEMU.
I have made no attempt to investigate these failures in detail.

Running all tests in gdb.base on x86_64 Linux using qemu-aarch64 I got:

    # of unexpected core files	1
    # of expected passes	66182
    # of unexpected failures	558
    # of expected failures	20
    # of known failures		29
    # of untested testcases	14
    # of unresolved testcases	4
    # of unsupported tests	146
    # of paths in test names	7
    # of duplicate test names	3

Running same set of tests using qemu-x86_64 I got:

    # of unexpected core files	1
    # of expected passes	64988
    # of unexpected failures	2527
    # of expected failures	20
    # of known failures		29
    # of untested testcases	8
    # of unresolved testcases	5
    # of unsupported tests	174
    # of paths in test names	7
    # of duplicate test names	5

[1]: https://inbox.sourceware.org/gdb/938d2b5574b94d9584b758eee3c0c064@ti.com/
[2]: https://inbox.sourceware.org/gdb-patches/20231204024634.419445-3-zeck654321@gmail.com/

Bug: https://sourceware.org/bugzilla/show_bug.cgi?id=33590
---
 gdb/testsuite/boards/qemu-user.exp | 235 +++++++++++++++++++++++++++++
 1 file changed, 235 insertions(+)
 create mode 100644 gdb/testsuite/boards/qemu-user.exp

diff --git a/gdb/testsuite/boards/qemu-user.exp b/gdb/testsuite/boards/qemu-user.exp
new file mode 100644
index 00000000000..286940f16a4
--- /dev/null
+++ b/gdb/testsuite/boards/qemu-user.exp
@@ -0,0 +1,235 @@
+# Copyright 2026 Free Software Foundation, Inc.
+
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+# This file is a dejagnu "board file" and is used to run the testsuite
+# using QEMU's usermode emulation. Use --target to specify desired target.
+#
+# To use specific QEMU, pass custom options to QEMU, target toolchain to use or
+# to pass custom options to the compiler, set environment variable QEMU,
+# QEMU_FLAGS, CROSS or CFLAGS_FOR_TARGET (see examples below).
+#
+# Note that to execute dynamically linked programs with qemu-user, a minimal
+# sysroot is required. It contains the target system's dynamic linker and the
+# shared libraries needed by the program. If it is not located at the default
+# path, either set QEMU_LD_PREFIX to the sysroot path or set QEMU_FLAGS to
+# "-L /path/to/sysroot".
+#
+# To avoid need for minimal sysroot altogether, force static compilation by
+# setting CFLAGS_FOR_TARGET to "-static" (see example below). Note however
+# that some tests need dynamic linking (for example, gdb.base/solib-vanish.exp)
+#
+# Example usage:
+#
+#     make check RUNTESTFLAGS='--target=riscv64-unknown-linux-gnu --target_board=qemu-user'
+#
+#     make check QEMU=qemu-arm CROSS=/opt/gcc-8-armhf/bin/arm-linux-gnueabihf- \
+#        RUNTESTFLAGS='--target=arm-unknown-linux-gnueabihf --target_board=qemu-user'
+#
+#     make check CFLAGS_FOR_TARGET=-static \
+#        RUNTESTFLAGS='--target=aarch64-unknown-linux-gnu --target_board=qemu-user'
+#
+
+# Load this for "gdb_target_cmd_ext"
+load_lib gdbserver-support.exp
+
+#
+# Setup
+#
+set qemu ""
+set qemu_flags ""
+set cross ""
+
+switch -glob $target_triplet {
+    "x86_64-*-linux-gnu" {
+	set qemu "qemu-x86_64"
+	set cross "x86_64-linux-gnu-"
+    }
+    "riscv64-*-linux-gnu" {
+	set qemu "qemu-riscv64"
+	set cross "riscv64-linux-gnu-"
+    }
+    "aarch64-*-linux-gnu" {
+	set qemu "qemu-aarch64"
+	set cross "aarch64-linux-gnu-"
+    }
+    "arm-*-linux-gnueabihf" {
+	set qemu "qemu-arm"
+	set cross "arm-linux-gnueabihf-"
+    }
+    "powerpc-*-linux-gnu" {
+	set qemu "qemu-ppc"
+	set cross "powerpc-linux-gnu-"
+    }
+    "powerpc64le-*-linux-gnu" {
+	set qemu "qemu-ppc64le"
+	set cross "powerpc64le-linux-gnu-"
+    }
+    #
+    # Add more here if needed.
+    #
+    default {
+	error "Target $target_triplet not yet supported by board"
+    }
+}
+
+#
+# Allow user to specify QEMU binary and/or custom options by setting
+# QEMU and/or QEMU_FLAGS environment variables.
+#
+if [info exists ::env(QEMU)] {
+    set qemu $::env(QEMU)
+}
+if [info exists ::env(QEMU_FLAGS)] {
+    set qemu_flags $::env(QEMU_FLAGS)
+}
+
+#
+# Allow user to specify target toolchain by setting CROSS environment variable
+#
+if [info exists ::env(CROSS)] {
+    set cross $::env(CROSS)
+}
+
+#
+# Allow user to pass custom flags to the compiler.
+#
+if [info exists ::env(CFLAGS_FOR_TARGET)] {
+    set CFLAGS_FOR_TARGET $::env(CFLAGS_FOR_TARGET)
+}
+
+set qemu_spawn_id ""
+
+set_board_info compiler "${cross}gcc"
+set_board_info c++compiler "${cross}g++"
+
+set_board_info use_gdb_stub 1
+set_board_info gdb_protocol "remote"
+set_board_info gdb,do_reload_on_run 1
+set_board_info gdb,nofileio 1
+set_board_info exit_is_reliable 1
+# QEMU does not support HW watchpoints
+set_board_info gdb,no_hardware_watchpoints 1
+
+#
+# Helper function to run QEMU. Returns the port number on which QEMU
+# stub listens or 0 on failure.
+#
+proc qemu_spawn { file inferior_args } {
+    global qemu_spawn_id
+    global qemu
+    global qemu_flags
+
+    #
+    # Close any previous QEMU instance.
+    #
+    if { $qemu_spawn_id != "" } {
+	verbose -log "qemu: closing previous spawn id $qemu_spawn_id"
+	if [catch { close -i $qemu_spawn_id } != 0] {
+	    warning "qemu: failed to close connection to previous qemu instance"
+	}
+
+	# some tests get QEMU/its gdb stub
+	# into a state where the process doesn't want to die
+	# so -nowait to not hang the entire test run
+	wait -nowait -i $qemu_spawn_id
+	set qemu_spawn_id ""
+    }
+
+
+    #
+    # Run QEMU.
+    #
+    while 1 {
+	set portnum [expr {1024+round(rand()*(65535-1024))}]
+	set cmd "spawn -noecho $qemu $qemu_flags -g $portnum $file $inferior_args"
+
+	verbose -log "Spawning qemu: $cmd"
+
+	if {[catch { eval $cmd } err ]} {
+	    verbose -log "Failed to run QEMU: $err"
+	    return 0
+	}
+	set qemu_spawn_id $spawn_id
+
+	# QEMU stub doesn't do inferior I/O through GDB.  But we can
+	# talk to the program using QEMU's tty instead.
+	global inferior_spawn_id
+	set inferior_spawn_id $spawn_id
+
+	expect {
+	    -i $qemu_spawn_id
+	    -timeout 1
+	    -re ".*Address already in use.*" {
+		# Port already in use. Cleanup and retry.
+		verbose -log "Port $portnum is already in use."
+		if [catch { close -i $qemu_spawn_id } != 0] {
+		    warning "failed to terminate QEMU instance"
+		}
+		wait -i $qemu_spawn_id
+		continue
+	    }
+	    eof {
+		verbose -log "QEMU terminated prematurely."
+		return 0
+	    }
+
+	}
+
+	verbose -log "QEMU spawned with spawn id $qemu_spawn_id, pid [exp_pid $qemu_spawn_id], stub listening at $portnum"
+	break
+    }
+
+    return $portnum
+}
+
+
+#
+# gdb_reload -- load a file into the target.
+#
+proc gdb_reload { {inferior_args {}} } {
+    global last_loaded_file
+    set portnum [qemu_spawn $last_loaded_file $inferior_args]
+
+    if { $portnum == 0 } {
+	return 1
+    }
+
+    set res [gdb_target_cmd_ext "remote" ":$portnum"]
+
+    if { $res == 2 } {
+	unsupported "Cannot connect to QEMU stub (likely because QEMU does not support non-stop mode)."
+	# QEMU does not support non-stop mode. Exit here to avoid subsequent
+	# failures / timeouts.
+	exit 0
+    }
+    return [expr {$res == 0 ? 0 : 1}]
+}
+
+proc mi_gdb_target_load { } {
+    global last_loaded_file
+    set portnum [qemu_spawn $last_loaded_file ""]
+
+    if { $portnum == 0 } {
+	return -1
+    }
+
+    if { [mi_gdb_target_cmd "remote" ":$portnum"] != 0 } {
+	return -1
+    }
+
+    return 0
+}
+
+set GDBFLAGS "${GDBFLAGS} -iex \"set auto-connect-native-target off\""
-- 
2.53.0