[PATCH] gdb/testsuite: add support for running tests using QEMU

[PATCH] gdb/testsuite: add support for running tests using QEMU

[Jan Vrany]

While debugging failing RISC-V tests, I found myself once again in a
situation where running tests easily using QEMU usermode emulation
would be useful. Since this is something that resurfaces every now and
then [1], I decided to have another stab at it.

This commit adds a new "board" file - qemu.exp - that allows running
tests using QEMU usermode emulation. The main goal is to make it easier
to test changes on multiple different architectures. However using QEMU
to run tests is no substitute for running tests on real hardware (or
in a VM). See limitations below.

The implementation based on simavr.exp and on an earlier patch by
Zeck S [2] and somewhat simplified. Unlike previous attempts, test
programs are no compiled statically in order to support broader set of
testcases. It also supports more targets (new targets should be easy to
add).

Not all test that pass on real hardware pass when run under usermode
QEMU. Some failures stem from limitations of QEMU's GDB stub - it does
not support non-stop mode, multiple threads, hardware watchpoints. In
some other cases GDB crashes, perhaps due to a bug in either GDB or QEMU.
I have made no attempt to investigate these failures in detail.

Running all tests in gdb.base on x86_64 Linux using qemu-aarch64 I got:

    # of unexpected core files	1
    # of expected passes	66088
    # of unexpected failures	1431
    # of expected failures	14
    # of known failures		25
    # of untested testcases	14
    # of unresolved testcases	25
    # of unsupported tests	227
    # of paths in test names	38
    # of duplicate test names	5

Running same set of tests using qemu-x86_64 I got:

    # of unexpected core files	1
    # of expected passes	64988
    # of unexpected failures	2527
    # of expected failures	20
    # of known failures		29
    # of untested testcases	8
    # of unresolved testcases	5
    # of unsupported tests	174
    # of paths in test names	7
    # of duplicate test names	5

[1]: https://inbox.sourceware.org/gdb/938d2b5574b94d9584b758eee3c0c064@ti.com/
[2]: https://inbox.sourceware.org/gdb-patches/20231204024634.419445-3-zeck654321@gmail.com/

Bug: https://sourceware.org/bugzilla/show_bug.cgi?id=33590
---
 gdb/testsuite/boards/qemu.exp | 195 ++++++++++++++++++++++++++++++++++
 1 file changed, 195 insertions(+)
 create mode 100644 gdb/testsuite/boards/qemu.exp

diff --git a/gdb/testsuite/boards/qemu.exp b/gdb/testsuite/boards/qemu.exp
new file mode 100644
index 00000000000..2d15294d369
--- /dev/null
+++ b/gdb/testsuite/boards/qemu.exp
@@ -0,0 +1,195 @@
+# Copyright 2026 Free Software Foundation, Inc.
+
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+# This file is a dejagnu "board file" and is used to run the testsuite
+# using QEMU's usermode emulation. Use --target to specify desired target.
+#
+# To use specific QEMU, pass custom options to QEMU or target toolchain to use,
+# set environment variable QEMU, QEMU_FLAGS or CROSS (see examples below).
+#
+# Example usage:
+#
+#     make check RUNTESTFLAGS='--target=riscv64-unknown-linux-gnu --target_board=qemu'
+#
+#     make check QEMU=qemu-arm CROSS=/opt/gcc-8-armhf/bin/arm-linux-gnueabihf- \
+#        RUNTESTFLAGS='--target=arm-unknown-linux-gnueabihf --target_board=qemu'
+#
+
+# Load this for "gdb_target_cmd_ext"
+load_lib gdbserver-support.exp
+
+#
+# Setup
+#
+set qemu ""
+set qemu_flags ""
+set cross ""
+
+switch -glob $target_triplet {
+    "x86_64-*-linux-gnu" {
+	set qemu "qemu-x86_64"
+	set cross "x86_64-linux-gnu-"
+    }
+    "riscv64-*-linux-gnu" {
+	set qemu "qemu-riscv64"
+	set cross "riscv64-linux-gnu-"
+    }
+    "aarch64-*-linux-gnu" {
+	set qemu "qemu-aarch64"
+	set cross "aarch64-linux-gnu-"
+    }
+    "arm-*-linux-gnueabihf" {
+	set qemu "qemu-arm"
+	set cross "arm-linux-gnueabihf-"
+    }
+    "powerpc-*-linux-gnu" {
+	set qemu "qemu-ppc"
+	set cross "powerpc-linux-gnu-"
+    }
+    "powerpc64le-*-linux-gnu" {
+	set qemu "qemu-ppc64le"
+	set cross "powerpc64le-linux-gnu-"
+    }
+    #
+    # Add more here if needed.
+    #
+    default {
+	error "Target $target_triplet not yet supported by board"
+    }
+}
+
+#
+# Allow user to specify QEMU binary and/or custom options by setting
+# QEMU and/or QEMU_FLAGS environment variables.
+#
+if [info exists ::env(QEMU)] {
+    set qemu $::env(QEMU)
+}
+if [info exists ::env(QEMU_FLAGS)] {
+    set qemu_flags $::env(QEMU_FLAGS)
+}
+
+#
+# Allow user to specify target toolchain by setting CROSS environment variable
+#
+if [info exists ::env(CROSS)] {
+    set cross $::env(CROSS)
+}
+
+set qemu_spawn_id ""
+
+set_board_info compiler "${cross}gcc"
+set_board_info c++compiler "${cross}g++"
+
+set_board_info use_gdb_stub 1
+set_board_info gdb_protocol "remote"
+set_board_info gdb,do_reload_on_run 1
+set_board_info gdb,nofileio 1
+set_board_info exit_is_reliable 1
+# QEMU does not support HW watchpoints
+set_board_info gdb,no_hardware_watchpoints 1
+
+#
+# Helper function to run QEMU. Returns the port number on which QEMU
+# stub listens.
+#
+proc qemu_spawn { file inferior_args } {
+    global qemu_spawn_id
+    global qemu
+    global qemu_flags
+
+    #
+    # Close any previous QEMU instance.
+    #
+    if { $qemu_spawn_id != "" } {
+	verbose -log "qemu: closing previous spawn id $qemu_spawn_id"
+	if [catch { close -i $qemu_spawn_id } != 0] {
+	    warning "qemu: failed to close connection to previous qemu instance"
+	}
+
+	# some tests get QEMU/its gdb stub
+	# into a state where the process doesn't want to die
+	# so -nowait to not hang the entire test run
+	wait -nowait -i $qemu_spawn_id
+	set qemu_spawn_id ""
+    }
+
+
+    #
+    # Run QEMU.
+    #
+    while 1 {
+	set portnum [expr {1024+round(rand()*(65535-1024))}]
+	set cmd "spawn -noecho $qemu $qemu_flags -g $portnum $file $inferior_args"
+
+	verbose -log "Spawning qemu: $cmd"
+	eval $cmd
+	set qemu_spawn_id $spawn_id
+
+	# QEMU stub doesn't do inferior I/O through GDB.  But we can
+	# talk to the program using QEMU's tty instead.
+	global inferior_spawn_id
+	set inferior_spawn_id $spawn_id
+
+	expect {
+	    -i $qemu_spawn_id
+	    -timeout 1
+	    -re "$qemu:.*Address already in use.*" {
+		# Port already in use. Cleanup and retry.
+		verbose -log "Port $portnum is already in use."
+		if [catch { close -i $qemu_spawn_id } != 0] {
+		    warning "failed to terminate QEMU instance"
+		}
+		wait -i $qemu_spawn_id
+		continue
+	    }
+	}
+
+	verbose -log "QEMU spawned with spawn id $qemu_spawn_id, pid [exp_pid $qemu_spawn_id], stub listening at $portnum"
+	break
+    }
+
+    return $portnum
+}
+
+
+#
+# gdb_reload -- load a file into the target.
+#
+proc gdb_reload { {inferior_args {}} } {
+    global last_loaded_file
+    set portnum [qemu_spawn $last_loaded_file $inferior_args]
+
+    if  { [gdb_target_cmd_ext "remote" ":$portnum"] == 2 } {
+	unsupported
+	# QEMU does not support non-stop mode. Exit here to avoid subseqn=uent
+	# failts / timeouts.
+	exit 0
+    }
+    return 0
+}
+
+proc mi_gdb_target_load { } {
+    global last_loaded_file
+    set portnum [qemu_spawn $last_loaded_file ""]
+
+    if { [mi_gdb_target_cmd "remote" ":$portnum"] != 0 } {
+	return -1
+    }
+
+    return 0
+}
+
+set GDBFLAGS "${GDBFLAGS} -iex \"set auto-connect-native-target off\""
-- 
2.53.0

Re: [PATCH] gdb/testsuite: add support for running tests using QEMU

[Simon Marchi]

On 10/26/26 6:02 AM, Jan Vrany wrote:
> While debugging failing RISC-V tests, I found myself once again in a
> situation where running tests easily using QEMU usermode emulation
> would be useful. Since this is something that resurfaces every now and
> then [1], I decided to have another stab at it.
> 
> This commit adds a new "board" file - qemu.exp - that allows running
> tests using QEMU usermode emulation. The main goal is to make it easier
> to test changes on multiple different architectures. However using QEMU
> to run tests is no substitute for running tests on real hardware (or
> in a VM). See limitations below.
> 
> The implementation based on simavr.exp and on an earlier patch by
> Zeck S [2] and somewhat simplified. Unlike previous attempts, test
> programs are no compiled statically in order to support broader set of
> testcases. It also supports more targets (new targets should be easy to
> add).
> 
> Not all test that pass on real hardware pass when run under usermode
> QEMU. Some failures stem from limitations of QEMU's GDB stub - it does
> not support non-stop mode, multiple threads, hardware watchpoints. In
> some other cases GDB crashes, perhaps due to a bug in either GDB or QEMU.
> I have made no attempt to investigate these failures in detail.
> 
> Running all tests in gdb.base on x86_64 Linux using qemu-aarch64 I got:
> 
>     # of unexpected core files	1
>     # of expected passes	66088
>     # of unexpected failures	1431
>     # of expected failures	14
>     # of known failures		25
>     # of untested testcases	14
>     # of unresolved testcases	25
>     # of unsupported tests	227
>     # of paths in test names	38
>     # of duplicate test names	5
> 
> Running same set of tests using qemu-x86_64 I got:
> 
>     # of unexpected core files	1
>     # of expected passes	64988
>     # of unexpected failures	2527
>     # of expected failures	20
>     # of known failures		29
>     # of untested testcases	8
>     # of unresolved testcases	5
>     # of unsupported tests	174
>     # of paths in test names	7
>     # of duplicate test names	5
> 
> [1]: https://inbox.sourceware.org/gdb/938d2b5574b94d9584b758eee3c0c064@ti.com/
> [2]: https://inbox.sourceware.org/gdb-patches/20231204024634.419445-3-zeck654321@gmail.com/
> 
> Bug: https://sourceware.org/bugzilla/show_bug.cgi?id=33590

I think that is really cool, I am in favor of adding it.  Naming nit:
perhaps name it qemu-user, I could see us also adding a qemu-system
board to test bare metal.  For example, I saw that qemu now has an AVR
port, I would like to make a board based on that.

Do you pass a -L to the qemu user binary, or set QEMU_LD_PREFIX?  I
found it necessary for the right dynamic linker to be found:

    $ qemu-aarch64 ./a.out
    qemu-aarch64: Could not open '/lib/ld-linux-aarch64.so.1': No such file or directory                                                                                                                          /0.0s
    $ qemu-aarch64 -L /usr/aarch64-linux-gnu  ./a.out
    Hello!

If it's always going to be required to pass one or the other, could the
board require that the user specify it, or could it try to guess it
(e.g. "/usr/$triplet")?

I ran your patch through Claude, most of the important comments were
about robustnessof the "non-happy" paths, which in my experience is very
important when you're trying to use the board and figuring things out.
I would suggest giving all the failure modes a try to see if the error
reported is sensible and the user experience is good.

 - gdb/testsuite/boards/qemu.exp:176: `unsupported` takes one argument
   (proc unsupported { message } in DejaGnu's framework.exp).  Called
   with none, it raises a Tcl error, so the test ends with an ERROR and
   `exit 0` is never reached.

 - gdb/testsuite/boards/qemu.exp:179: `exit 0` terminates the runtest
   process, not just the current test.  In a serial run, all the
   remaining tests are skipped silently.  Use `unsupported "..."` with a
   message and `return -1` instead; gdb_run_cmd already handles a
   non-zero return (lib/gdb.exp:549).

   Also, gdb_target_cmd_ext returns 2 for "Network is unreachable" as well
   as for the non-stop case (lib/gdbserver-support.exp:113), so the
   message shouldn't assume non-stop.

 - gdb/testsuite/boards/qemu.exp:175-181: when gdb_target_cmd_ext returns
   1 (connection failure), gdb_reload still returns 0, and callers then
   send "continue" to an unconnected GDB.  Return non-zero on any result
   other than 0.

 - gdb/testsuite/boards/qemu.exp:146-158: the expect has no eof clause, so
   if QEMU exits right away (bad binary, bad QEMU_FLAGS, missing sysroot),
   qemu_spawn still returns the port as if QEMU were listening.  GDB then
   gets "Connection refused", which gdb_target_cmd_ext retries three
   times with a 30-second sleep each (lib/gdbserver-support.exp:100-104),
   then the failure is swallowed (see above).  Add an eof clause that
   reports the failure.

 - gdb/testsuite/boards/qemu.exp:149: the "$qemu:.*Address already in
   use" pattern is fragile.  With QEMU 11.1, both "qemu-aarch64" and
   "/usr/bin/qemu-aarch64" print:

     qemu-aarch64: Failed to bind socket: Address already in use

   So with QEMU set to a full path, the pattern never matches, the port
   collision isn't retried, and the test runs against a dead QEMU.  I
   believe older QEMU versions printed "bind: Address already in use"
   followed by "qemu: could not open gdbserver on PORT", which doesn't
   match either.  Matching on "Address already in use" alone would be
   more robust.

Thanks,

Simon