Re: [PATCH] gdb/testsuite: add support for running tests using QEMU

[Simon Marchi <simark@simark.ca>]

On 10/26/26 6:02 AM, Jan Vrany wrote:
> While debugging failing RISC-V tests, I found myself once again in a
> situation where running tests easily using QEMU usermode emulation
> would be useful. Since this is something that resurfaces every now and
> then [1], I decided to have another stab at it.
> 
> This commit adds a new "board" file - qemu.exp - that allows running
> tests using QEMU usermode emulation. The main goal is to make it easier
> to test changes on multiple different architectures. However using QEMU
> to run tests is no substitute for running tests on real hardware (or
> in a VM). See limitations below.
> 
> The implementation based on simavr.exp and on an earlier patch by
> Zeck S [2] and somewhat simplified. Unlike previous attempts, test
> programs are no compiled statically in order to support broader set of
> testcases. It also supports more targets (new targets should be easy to
> add).
> 
> Not all test that pass on real hardware pass when run under usermode
> QEMU. Some failures stem from limitations of QEMU's GDB stub - it does
> not support non-stop mode, multiple threads, hardware watchpoints. In
> some other cases GDB crashes, perhaps due to a bug in either GDB or QEMU.
> I have made no attempt to investigate these failures in detail.
> 
> Running all tests in gdb.base on x86_64 Linux using qemu-aarch64 I got:
> 
>     # of unexpected core files	1
>     # of expected passes	66088
>     # of unexpected failures	1431
>     # of expected failures	14
>     # of known failures		25
>     # of untested testcases	14
>     # of unresolved testcases	25
>     # of unsupported tests	227
>     # of paths in test names	38
>     # of duplicate test names	5
> 
> Running same set of tests using qemu-x86_64 I got:
> 
>     # of unexpected core files	1
>     # of expected passes	64988
>     # of unexpected failures	2527
>     # of expected failures	20
>     # of known failures		29
>     # of untested testcases	8
>     # of unresolved testcases	5
>     # of unsupported tests	174
>     # of paths in test names	7
>     # of duplicate test names	5
> 
> [1]: https://inbox.sourceware.org/gdb/938d2b5574b94d9584b758eee3c0c064@ti.com/
> [2]: https://inbox.sourceware.org/gdb-patches/20231204024634.419445-3-zeck654321@gmail.com/
> 
> Bug: https://sourceware.org/bugzilla/show_bug.cgi?id=33590

I think that is really cool, I am in favor of adding it.  Naming nit:
perhaps name it qemu-user, I could see us also adding a qemu-system
board to test bare metal.  For example, I saw that qemu now has an AVR
port, I would like to make a board based on that.

Do you pass a -L to the qemu user binary, or set QEMU_LD_PREFIX?  I
found it necessary for the right dynamic linker to be found:

    $ qemu-aarch64 ./a.out
    qemu-aarch64: Could not open '/lib/ld-linux-aarch64.so.1': No such file or directory                                                                                                                          /0.0s
    $ qemu-aarch64 -L /usr/aarch64-linux-gnu  ./a.out
    Hello!

If it's always going to be required to pass one or the other, could the
board require that the user specify it, or could it try to guess it
(e.g. "/usr/$triplet")?

I ran your patch through Claude, most of the important comments were
about robustnessof the "non-happy" paths, which in my experience is very
important when you're trying to use the board and figuring things out.
I would suggest giving all the failure modes a try to see if the error
reported is sensible and the user experience is good.

 - gdb/testsuite/boards/qemu.exp:176: `unsupported` takes one argument
   (proc unsupported { message } in DejaGnu's framework.exp).  Called
   with none, it raises a Tcl error, so the test ends with an ERROR and
   `exit 0` is never reached.

 - gdb/testsuite/boards/qemu.exp:179: `exit 0` terminates the runtest
   process, not just the current test.  In a serial run, all the
   remaining tests are skipped silently.  Use `unsupported "..."` with a
   message and `return -1` instead; gdb_run_cmd already handles a
   non-zero return (lib/gdb.exp:549).

   Also, gdb_target_cmd_ext returns 2 for "Network is unreachable" as well
   as for the non-stop case (lib/gdbserver-support.exp:113), so the
   message shouldn't assume non-stop.

 - gdb/testsuite/boards/qemu.exp:175-181: when gdb_target_cmd_ext returns
   1 (connection failure), gdb_reload still returns 0, and callers then
   send "continue" to an unconnected GDB.  Return non-zero on any result
   other than 0.

 - gdb/testsuite/boards/qemu.exp:146-158: the expect has no eof clause, so
   if QEMU exits right away (bad binary, bad QEMU_FLAGS, missing sysroot),
   qemu_spawn still returns the port as if QEMU were listening.  GDB then
   gets "Connection refused", which gdb_target_cmd_ext retries three
   times with a 30-second sleep each (lib/gdbserver-support.exp:100-104),
   then the failure is swallowed (see above).  Add an eof clause that
   reports the failure.

 - gdb/testsuite/boards/qemu.exp:149: the "$qemu:.*Address already in
   use" pattern is fragile.  With QEMU 11.1, both "qemu-aarch64" and
   "/usr/bin/qemu-aarch64" print:

     qemu-aarch64: Failed to bind socket: Address already in use

   So with QEMU set to a full path, the pattern never matches, the port
   collision isn't retried, and the test runs against a dead QEMU.  I
   believe older QEMU versions printed "bind: Address already in use"
   followed by "qemu: could not open gdbserver on PORT", which doesn't
   match either.  Matching on "Address already in use" alone would be
   more robust.

Thanks,

Simon