From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from simark.ca by simark.ca with LMTP id HSMMNjWkw2fRTwIAWB0awg (envelope-from ) for ; Sat, 01 Mar 2025 19:20:05 -0500 Received: by simark.ca (Postfix, from userid 112) id CE53F1E105; Sat, 1 Mar 2025 19:20:05 -0500 (EST) X-Spam-Checker-Version: SpamAssassin 4.0.0 (2022-12-13) on simark.ca X-Spam-Level: X-Spam-Status: No, score=-5.3 required=5.0 tests=ARC_SIGNED,ARC_VALID,BAYES_00, MAILING_LIST_MULTI,RCVD_IN_DNSWL_MED autolearn=unavailable autolearn_force=no version=4.0.0 Received: from server2.sourceware.org (server2.sourceware.org [8.43.85.97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (prime256v1) server-digest SHA256) (No client certificate requested) by simark.ca (Postfix) with ESMTPS id ABECA1E05C for ; Sat, 1 Mar 2025 19:20:04 -0500 (EST) Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 2BB193858C50 for ; Sun, 2 Mar 2025 00:20:04 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 2BB193858C50 Received: from angie.orcam.me.uk (angie.orcam.me.uk [IPv6:2001:4190:8020::34]) by sourceware.org (Postfix) with ESMTP id 21DD23858D1E for ; Sun, 2 Mar 2025 00:17:33 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 21DD23858D1E Authentication-Results: sourceware.org; dmarc=none (p=none dis=none) header.from=orcam.me.uk Authentication-Results: sourceware.org; spf=none smtp.mailfrom=orcam.me.uk ARC-Filter: OpenARC Filter v1.0.0 sourceware.org 21DD23858D1E Authentication-Results: server2.sourceware.org; arc=none smtp.remote-ip=2001:4190:8020::34 ARC-Seal: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1740874653; cv=none; b=sbtSOUcgIELuqEGUNC9X+mygtd9MpAX+12wM6FuhwFMhro6K8rlhJYhx2hltgwV0fPpJu0hBhzf/GLnS8uOs1RzlUPQF8X6I1sOLoBanqey/o9iWmB7+A/5vsr0QZIraUtUbk4Eb8WzME5Aqkc4vEmcD4wCTvRIZlEIahj+8w5I= ARC-Message-Signature: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1740874653; c=relaxed/simple; bh=UJFerqcOPjT87e1gAta0PUSrYyQAWmKbk+LsN7CDWqs=; h=Date:From:To:Subject:Message-ID:MIME-Version; b=iUVdLCWZjPGFTzJtog+tkA6LtfXmqAF+KnfqT+X/Puyd0ZUABvbXRE2fQCGi41NiASeapaz4vq2UhW36sgCc3C5NHlsK/pWx9rTIQ4Ah4lBROJ0PWlRCynJD9Vd6uYncRyBNMsR7nNXd5qS0D19Y+idRJCPFEBb1hl78M+VgrqE= ARC-Authentication-Results: i=1; server2.sourceware.org DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 21DD23858D1E Received: by angie.orcam.me.uk (Postfix, from userid 500) id 75B9F92009E; Sun, 2 Mar 2025 01:17:32 +0100 (CET) Received: from localhost (localhost [127.0.0.1]) by angie.orcam.me.uk (Postfix) with ESMTP id 6E6CB92009C; Sun, 2 Mar 2025 00:17:32 +0000 (GMT) Date: Sun, 2 Mar 2025 00:17:32 +0000 (GMT) From: "Maciej W. Rozycki" To: Milica Matic cc: Djordje Todorovic , Milos Kalicanin , Simon Marchi , Chao-ying Fu , Andrew Burgess , Kevin Buettner , Thomas Bogendoerfer , gdb-patches@sourceware.org Subject: Re: [PATCH v12 2/2] gdb: mips: Add MIPSR6 support In-Reply-To: Message-ID: References: <20250219130125.356900-1-milica.matic@htecgroup.com> <20250219130125.356900-4-milica.matic@htecgroup.com> User-Agent: Alpine 2.21 (DEB 202 2017-01-01) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII X-BeenThere: gdb-patches@sourceware.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Gdb-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: gdb-patches-bounces~public-inbox=simark.ca@sourceware.org On Tue, 25 Feb 2025, Maciej W. Rozycki wrote: > > Introduce new instruction encodings from Release 6 of the MIPS > > architecture [1]. Support breakpoints and single stepping with > > compact branches, forbidden slots, new branch instruction and > > new atomic load-store instruction encodings. > > I have now skimmed over your change and obviously branch decoding is the > huge update and the most problematic part. At high level I find new code > a little bit hairy and hard to follow in some places, and will go over it > in details double-checking all is in order, and maybe trying to untangle > it a little. I won't be able to do it before this coming weekend though > at which point I'm going on holiday for a fortnight. Please be a little > patient meanwhile. > > Also you may have seen my response to an earlier version where I reported > how I tried to do verification against my 74K Malta only to see it run > unreliably. Sadly the 5Kc core card I previously used broke a couple of > years ago and the replacement newer module does not seem to run equally > well (also it's 32-bit only while the 5Kc was 64-bit). > > I suspect it just wasn't debugged as thoroughly as the older one, there > are issues with PCI handling both in Linux and even in the YAMON firmware, > and I think this instability in running GDB verification can be attributed > to cache coherency maintenance issues in Linux, which can be hairy. I may > not have the resources available at hand though to get it sorted quickly. > > It doesn't appear to me that the 74K core card is at fault as I have two > of them, slightly different and carrying a 74Kc and a 74Kf each, so if it > was owing to a faulty card, then I imagine they wouldn't behave exactly > the same. Also glibc and GCC verification seems more reproducible. The > old 5Kc core card was rock-solid, I used it for 20 years until it failed. > > So instead I made an attempt the other day to use alternate test machines > I have in my remote lab, and that also hit obstacles. One of the systems > refused to respond on the serial console. I've seen it before and it is > likely owing to its lithium coin cell powering the configuration memory > having run out of energy, making the system revert to the video console, > which I have no remote access to. So I cannot make use of that machine > for the time being, until the cell has been replaced and configuration > restored. > > The other system booted just fine and completed reference GDB testing. > On second iteration however it broke with uncorrectable multiple-bit ECC > errors reported by DRAM making the system unusable. The errors persisted > after a reboot and firmware testing pointed at just one problematic DRAM > module (out of eight installed total). I have seen this before as well > and cleaning contacts and/or reseating the module should clear the issue. > At worst the module can be removed, reducing system RAM from 256MiB down > to 224MiB, which should still be fine for the purpose of this testing. > > I have a maintenance visit scheduled to my lab tomorrow, so I'm going to > address these issues. I'll report back and, fingers crossed, I should at > least be able to verify the lack of regressions from your change for older > systems, possibly by the end of this week. Both systems now fixed and results are as follows: 1. The R4400 system suffers from exactly the same testing instability that the 74Kf Malta does. Tests time out intermittently that otherwise run successfully almost instantaneously, so it's not that they are on the verge of timing out and just either miss it or hit. The two systems are different enough from each other a hardware fault can be surely ruled out. However they do share the grand MIPS R4k cache architecture and given that both are overall stable with other loads I'm even more confident this has something to do with cache coherency in the context of ptrace(2) calls. Example result fluctuations with 74Kf Malta: === gdb Summary === -# of expected passes 102278 -# of unexpected failures 4060 -# of expected failures 80 +# of expected passes 100275 +# of unexpected failures 1697 +# of expected failures 79 # of unknown successes 5 # of known failures 125 -# of unresolved testcases 190 +# of unresolved testcases 145 # of untested testcases 45 -# of unsupported tests 849 -# of duplicate test names 34 +# of unsupported tests 850 +# of duplicate test names 33 Example result fluctuations with R4400SC DECstation: === gdb Summary === -# of expected passes 94006 -# of unexpected failures 4236 -# of expected failures 77 -# of known failures 111 -# of unresolved testcases 688 -# of untested testcases 51 +# of expected passes 91166 +# of unexpected failures 3140 +# of expected failures 75 +# of unknown successes 4 +# of known failures 91 +# of unresolved testcases 280 +# of untested testcases 50 # of unsupported tests 879 -# of duplicate test names 9 +# of duplicate test names 29 2. Conversely the R3400 system, which was one I meant to use originally, but could not owing to the flat coin call, provides stable results with almost no intermittent failures except for the gdb.threads/ subset, which has never been stable for MIPS/Linux, even 20 years ago, and is likely a deficiency of implementation outside GDB. This system has the unique simpler R3k write-through cache architecture, but is otherwise the same as the R4400 system referred to above, i.e. the difference between them is just the CPU daughtercard, which further suggests we have an issue with R4k cache handling. Example result fluctuations with R3400 DECstation: === gdb Summary === -# of expected passes 106361 -# of unexpected failures 923 +# of expected passes 106248 +# of unexpected failures 920 # of expected failures 84 # of unknown successes 4 -# of known failures 125 +# of known failures 124 # of unresolved testcases 3 # of untested testcases 47 # of unsupported tests 879 Note the best scores overall despite the system being the most obsolete of the three. It's not completely fine though, as with the `call-ar-st' and `varargs' tests corrupt kernel oopses reliably trigger that follow this pattern: FP exception in kernel code[#1]: CPU: 0 PID: 1301 Comm: call-ar-st Not tainted 5.18.0-rc2-00254-gfb649bda6f56-dirty #6 $ 0 : 00000000 00000000 00000000 00000000 $ 4 : 00000000 00000000 00000000 00000000 $ 8 : 00000000 00000000 00000000 00000000 $12 : 00000000 00000000 00000000 00000000 $16 : 00000000 00000000 00000000 00000000 $20 : 00000000 00000000 00000000 00000000 $24 : 00000000 00000000 $28 : 00000000 00000000 00000000 00000000 Hi : 00000000 Lo : 00000000 epc : 00000000 0x0 ra : 00000000 0x0 Status: 00000000 Cause : 00000000 (ExcCode 00) PrId : 00000230 (R3000A) Modules linked in: Process call-ar-st (pid: 1301, threadinfo=649d947d, task=88fb543d, tls=77ffd580) Stack : Call Trace: Code: (Bad address in epc) ---[ end trace 0000000000000000 ]--- Reserved instruction in kernel code[#2]: CPU: 0 PID: 1301 Comm: call-ar-st Tainted: G D 5.18.0-rc2-00254-gfb649bda6f56-dirty #6 $ 0 : 00000000 1000ec01 00000080 00000000 $ 4 : 81880660 80de4c40 1000ec00 ffff00fe $ 8 : 344c6906 0000ec00 0000022b ffffffff $12 : 00000000 00395b10 00000000 656e7265 $16 : 00000001 00010001 00000000 00000000 $20 : 81831668 00000000 00000000 00400000 $24 : 00000000 80f3ded0 $28 : 80e30000 80c0bce0 819a96e0 81a52000 Hi : 005509b3 Lo : 4102d5e4 epc : 81a52084 0x81a52084 ra : 81a52000 0x81a52000 Status: 1008ec04 IEp Cause : 10000028 (ExcCode 0a) PrId : 00000230 (R3000A) Modules linked in: Process call-ar-st (pid: 1301, threadinfo=649d947d, task=88fb543d, tls=77ffd580) Stack : 80c01300 8042b188 1000ec01 80c7d430 81a53f00 8018a374 00001142 00000000 00000d06 00000000 00000000 80730000 00000001 00000000 80c7d430 00000020 81a5222c 8042b188 00000000 80c04a00 80c7d430 1000ec01 804e35e0 2abe260a 00000020 8018993c 81a5222c 2abe260a 00000020 8042fdf0 2abe260a 818f7010 80c7d430 00000020 81a5222c 804e35e0 807027c8 80786f60 00011606 80786f60 ... Call Trace: [<8042b188>] skb_release_data+0x1a8/0x1ec [<8018a374>] kfree+0xac/0x168 [<8042b188>] skb_release_data+0x1a8/0x1ec [<804e35e0>] tcp_rcv_established+0x214/0x8ac [<8018993c>] kmem_cache_free.part.0+0x4c/0xac [<8042fdf0>] __kfree_skb+0x2c/0x40 [<804e35e0>] tcp_rcv_established+0x214/0x8ac [<804f2d68>] tcp_v4_do_rcv+0x190/0x250 [<804f5d38>] tcp_v4_rcv+0xf3c/0xfd4 [<800b0654>] timekeeping_update+0x158/0x2dc [<80138a78>] __pagevec_lru_add+0x3bc/0x3f8 [<8041d888>] sock_pfree+0x0/0x4c [<80740000>] mitigations_parse_cmdline+0x90/0x98 [<804f4d2c>] tcp_v4_early_demux+0xe4/0x1b4 [<804bc2b0>] ip_protocol_deliver_rcu+0x54/0x2ec [<804bb70c>] ip_rcv_finish_core.constprop.0+0x18c/0x538 [<800a7b20>] dma_map_page_attrs+0x21c/0x324 [<804bc5fc>] ip_local_deliver+0xb4/0x108 [<80447c90>] __netif_receive_skb_one_core+0x4c/0x5c [<80447e34>] process_backlog+0x94/0x1bc [<80448b18>] __napi_poll+0x40/0x1d0 [<80740000>] mitigations_parse_cmdline+0x90/0x98 [<80740000>] mitigations_parse_cmdline+0x90/0x98 [<80448ebc>] net_rx_action+0xf4/0x25c [<80071a00>] queue_work_on+0x7c/0x84 [<803bdf68>] add_interrupt_randomness+0x24/0x210 [<806032e4>] __do_softirq+0x17c/0x34c [<8009e494>] handle_level_irq+0xc8/0x208 [<80049180>] die+0x110/0x128 [<80740000>] mitigations_parse_cmdline+0x90/0x98 [<80041cf0>] dec_irq_dispatch+0x10/0x20 [<80043754>] handle_int+0x14c/0x158 Code: 00000000 00000000 00000000 <54002820> 936d6000 00089891 f831262b 0003aaaa 00080000 ---[ end trace 0000000000000000 ]--- Fixing recursive fault but reboot is needed! (mind that the R3k uses an ordinary interrupt to report FP exceptions). Given the obvious all-zero corruption I suspect bloody `.set noreorder' piece of code code somewhere missing a load delay slot that has no valid justification to schedule instructions by hand in the first place. Needless to say both tests score a lot of failures as a result. The R3k issue is a bohrbug though and I have a pair of solid reproducers, so I'll take care of it. I don't have the resources however to chase the R4k cache coherency issue, so instead I've cc'd Thomas, the MIPS/Linux kernel maintainer, to raise his awareness of the problem. This all also means I can use the R3400 system to reliably verify GDB changes and will proceed with your submission once I'm back from my holiday in a fortnight's time. Meanwhile I do hope this analysis helps. Maciej