From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from simark.ca by simark.ca with LMTP id viRSN75w32okmBwAWB0awg (envelope-from ) for ; Mon, 26 Oct 2026 11:24:46 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=simark.ca; s=mail; t=1793028286; bh=14Nx1b94TJ2vmDh2BPHz7urabMLD3BxGxZBVur/xApI=; h=Date:Subject:To:References:From:In-Reply-To:List-Id: List-Unsubscribe:List-Archive:List-Post:List-Help:List-Subscribe: From; b=hLzjNBtYGdNU0azp+htK+4EFyl4Ptg2mYkqcOJJDFRB4fhPPFQ2/o9w9E+ajCuBZN XxYjnDmIh6jNQisLmGEMA1NYGbqsMGUzlEen5NOAX2BekOPTACEa2nFq32AbEyIglo NPa3MmBFAXG0AQe7DIlCcIV618uL/diVW4Z+0wic= Received: by simark.ca (Postfix, from userid 112) id CE1F41E023; Mon, 26 Oct 2026 11:24:46 -0400 (EDT) X-Spam-Checker-Version: SpamAssassin 4.0.1 (2024-03-25) on simark.ca X-Spam-Level: X-Spam-Status: No, score=-2.4 required=5.0 tests=ARC_SIGNED,ARC_VALID,BAYES_00, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI, RCVD_IN_DNSWL_MED,RCVD_IN_VALIDITY_CERTIFIED_BLOCKED, RCVD_IN_VALIDITY_RPBL_BLOCKED,RCVD_IN_VALIDITY_SAFE_BLOCKED autolearn=ham autolearn_force=no version=4.0.1 Authentication-Results: simark.ca; dkim=pass (1024-bit key; unprotected) header.d=simark.ca header.i=@simark.ca header.a=rsa-sha256 header.s=mail header.b=YTOBtc9m; dkim-atps=neutral Received: from vm01.sourceware.org (vm01.sourceware.org [38.145.34.32]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519 server-signature ECDSA (prime256v1) server-digest SHA256) (No client certificate requested) by simark.ca (Postfix) with ESMTPS id EFE771E023 for ; Mon, 26 Oct 2026 11:24:45 -0400 (EDT) Received: from vm01.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id B93D24BA23CA for ; Mon, 26 Oct 2026 15:24:38 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org B93D24BA23CA Authentication-Results: sourceware.org; dkim=pass (1024-bit key, unprotected) header.d=simark.ca header.i=@simark.ca header.a=rsa-sha256 header.s=mail header.b=YTOBtc9m Received: from simark.ca (simark.ca [158.69.221.121]) by sourceware.org (Postfix) with ESMTPS id 273DC4BA2E28 for ; Mon, 26 Oct 2026 15:24:12 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 273DC4BA2E28 Authentication-Results: sourceware.org; dmarc=pass (p=none dis=none) header.from=simark.ca Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=simark.ca ARC-Filter: OpenARC Filter v1.0.0 sourceware.org 273DC4BA2E28 Authentication-Results: sourceware.org; arc=none smtp.remote-ip=158.69.221.121 ARC-Seal: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1793028252; cv=none; b=S3M8P4A/wlyekG+oHYYe9y4DugaV7r549LslixBH3RtldEMvEdZveNPsPlNcslf7Njxix74YimfiXYPFV2Cn5F/3ljA+Kfhq9On9QM8hY18Q7/+IdWeUHLlrhKRqjefdQTgdPPkYdzJ3Cu1AHdlnXaLLIU36kr9KAe9akSdObSk= ARC-Message-Signature: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1793028252; c=relaxed/simple; bh=14Nx1b94TJ2vmDh2BPHz7urabMLD3BxGxZBVur/xApI=; h=DKIM-Signature:Message-ID:Date:MIME-Version:Subject:To:From; b=RDRBdQJOVGzxfSaYwIQc9YDFFYK8FwgLtxsqKGB4SbtunlxBGtk9EI1GPxVHTb6TvSPOrYaQi9DzfgbOyy1spOfAS5n9YQm2HZpwfsUkloRabQaDxRYiTsaFhiCo9WPiVKKnuhWddFUqatOZL2f5hXNBOo0HeAxJ64fjXjBmebE= ARC-Authentication-Results: i=1; sourceware.org; dkim=pass (1024-bit key, unprotected) header.d=simark.ca header.i=@simark.ca header.a=rsa-sha256 header.s=mail header.b=YTOBtc9m DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 273DC4BA2E28 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=simark.ca; s=mail; t=1793028249; bh=14Nx1b94TJ2vmDh2BPHz7urabMLD3BxGxZBVur/xApI=; h=Date:Subject:To:References:From:In-Reply-To:From; b=YTOBtc9mOwQl/2rmJWJx3cxbXJVKG02mhK/FLELUTTxK1lTH9OVA4o3JN85/BHreE 4UHdkhlgjTr7pezwWMA5Qqw5eFYkWL5r2FtUR8ZCpfdHLo+SCKTAFnobzEU3bXH0OZ g+UbeQXwKd9fLUJYXomyUtTpLEp759GpU8tFJTMk= Received: by simark.ca (Postfix) id 2AD121E023; Mon, 26 Oct 2026 11:24:08 -0400 (EDT) Message-ID: <32977d50-72bd-4e79-900a-d8d6a80988e5@simark.ca> Date: Mon, 26 Oct 2026 11:24:08 -0400 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH] gdb/testsuite: add support for running tests using QEMU To: Jan Vrany , gdb-patches@sourceware.org References: <20261026100153.1789950-1-jan@vrany.io> Content-Language: fr From: Simon Marchi In-Reply-To: <20261026100153.1789950-1-jan@vrany.io> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-BeenThere: gdb-patches@sourceware.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Gdb-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: gdb-patches-bounces~public-inbox=simark.ca@sourceware.org On 10/26/26 6:02 AM, Jan Vrany wrote: > While debugging failing RISC-V tests, I found myself once again in a > situation where running tests easily using QEMU usermode emulation > would be useful. Since this is something that resurfaces every now and > then [1], I decided to have another stab at it. > > This commit adds a new "board" file - qemu.exp - that allows running > tests using QEMU usermode emulation. The main goal is to make it easier > to test changes on multiple different architectures. However using QEMU > to run tests is no substitute for running tests on real hardware (or > in a VM). See limitations below. > > The implementation based on simavr.exp and on an earlier patch by > Zeck S [2] and somewhat simplified. Unlike previous attempts, test > programs are no compiled statically in order to support broader set of > testcases. It also supports more targets (new targets should be easy to > add). > > Not all test that pass on real hardware pass when run under usermode > QEMU. Some failures stem from limitations of QEMU's GDB stub - it does > not support non-stop mode, multiple threads, hardware watchpoints. In > some other cases GDB crashes, perhaps due to a bug in either GDB or QEMU. > I have made no attempt to investigate these failures in detail. > > Running all tests in gdb.base on x86_64 Linux using qemu-aarch64 I got: > > # of unexpected core files 1 > # of expected passes 66088 > # of unexpected failures 1431 > # of expected failures 14 > # of known failures 25 > # of untested testcases 14 > # of unresolved testcases 25 > # of unsupported tests 227 > # of paths in test names 38 > # of duplicate test names 5 > > Running same set of tests using qemu-x86_64 I got: > > # of unexpected core files 1 > # of expected passes 64988 > # of unexpected failures 2527 > # of expected failures 20 > # of known failures 29 > # of untested testcases 8 > # of unresolved testcases 5 > # of unsupported tests 174 > # of paths in test names 7 > # of duplicate test names 5 > > [1]: https://inbox.sourceware.org/gdb/938d2b5574b94d9584b758eee3c0c064@ti.com/ > [2]: https://inbox.sourceware.org/gdb-patches/20231204024634.419445-3-zeck654321@gmail.com/ > > Bug: https://sourceware.org/bugzilla/show_bug.cgi?id=33590 I think that is really cool, I am in favor of adding it. Naming nit: perhaps name it qemu-user, I could see us also adding a qemu-system board to test bare metal. For example, I saw that qemu now has an AVR port, I would like to make a board based on that. Do you pass a -L to the qemu user binary, or set QEMU_LD_PREFIX? I found it necessary for the right dynamic linker to be found: $ qemu-aarch64 ./a.out qemu-aarch64: Could not open '/lib/ld-linux-aarch64.so.1': No such file or directory /0.0s $ qemu-aarch64 -L /usr/aarch64-linux-gnu ./a.out Hello! If it's always going to be required to pass one or the other, could the board require that the user specify it, or could it try to guess it (e.g. "/usr/$triplet")? I ran your patch through Claude, most of the important comments were about robustnessof the "non-happy" paths, which in my experience is very important when you're trying to use the board and figuring things out. I would suggest giving all the failure modes a try to see if the error reported is sensible and the user experience is good. - gdb/testsuite/boards/qemu.exp:176: `unsupported` takes one argument (proc unsupported { message } in DejaGnu's framework.exp). Called with none, it raises a Tcl error, so the test ends with an ERROR and `exit 0` is never reached. - gdb/testsuite/boards/qemu.exp:179: `exit 0` terminates the runtest process, not just the current test. In a serial run, all the remaining tests are skipped silently. Use `unsupported "..."` with a message and `return -1` instead; gdb_run_cmd already handles a non-zero return (lib/gdb.exp:549). Also, gdb_target_cmd_ext returns 2 for "Network is unreachable" as well as for the non-stop case (lib/gdbserver-support.exp:113), so the message shouldn't assume non-stop. - gdb/testsuite/boards/qemu.exp:175-181: when gdb_target_cmd_ext returns 1 (connection failure), gdb_reload still returns 0, and callers then send "continue" to an unconnected GDB. Return non-zero on any result other than 0. - gdb/testsuite/boards/qemu.exp:146-158: the expect has no eof clause, so if QEMU exits right away (bad binary, bad QEMU_FLAGS, missing sysroot), qemu_spawn still returns the port as if QEMU were listening. GDB then gets "Connection refused", which gdb_target_cmd_ext retries three times with a 30-second sleep each (lib/gdbserver-support.exp:100-104), then the failure is swallowed (see above). Add an eof clause that reports the failure. - gdb/testsuite/boards/qemu.exp:149: the "$qemu:.*Address already in use" pattern is fragile. With QEMU 11.1, both "qemu-aarch64" and "/usr/bin/qemu-aarch64" print: qemu-aarch64: Failed to bind socket: Address already in use So with QEMU set to a full path, the pattern never matches, the port collision isn't retried, and the test runs against a dead QEMU. I believe older QEMU versions printed "bind: Address already in use" followed by "qemu: could not open gdbserver on PORT", which doesn't match either. Matching on "Address already in use" alone would be more robust. Thanks, Simon