[PATCH] gdb/remote: fix assertions when attaching in non-stop mode

[PATCH] gdb/remote: fix assertions when attaching in non-stop mode

[Martin KOCH]

When connecting to a remote target in non-stop mode against a
multi-threaded inferior stopped at raise(SIGSTOP), three internal-error
assertions can fire in sequence:

  remote.c:546: mark_async_event_handler: Assertion 'this->is_async_p ()' failed.
  thread.c:429: set_pending_waitstatus: Assertion 'this->internal_state () ==
                THREAD_INT_STOPPED || ...' failed.
  thread.c:426: set_pending_waitstatus: Assertion '!this->has_pending_waitstatus ()' failed.

The first one is what PR 30630 reports and is fixed with the PR proposed
by Mikhail Terekhov, but with the fix the two other assertions surface:

 * thread.c:429: addressed by reordering set_internal_state / set_state
   to run before set_pending_waitstatus.
 * thread.c:426: addressed by clearing any existing pending wait
   status before installing the new one, when gdbserver delivers
   multiple events for the same thread.

[1] https://sourceware.org/pipermail/gdb-patches/2023-October/202937.html

Bug: https://sourceware.org/bugzilla/show_bug.cgi?id=30630
Signed-off-by: Martin KOCH <Martin.KOCH@bachmann.info>
---
 gdb/remote.c | 14 +++++++++-----
 1 file changed, 9 insertions(+), 5 deletions(-)

diff --git a/gdb/remote.c b/gdb/remote.c
index 2961664cf33..4148301bc6f 100644
--- a/gdb/remote.c
+++ b/gdb/remote.c
@@ -5237,13 +5237,17 @@ remote_target::process_initial_stop_replies (int from_tty)
 	  ws.set_stopped (sig);
 	}
 
-      if (ws.kind () != TARGET_WAITKIND_STOPPED
-	  || ws.sig () != GDB_SIGNAL_0)
-	evthread->set_pending_waitstatus (ws);
-
       set_internal_state (this, event_ptid, THREAD_INT_STOPPED);
       set_state (this, event_ptid, THREAD_STOPPED);
       get_remote_thread_info (evthread)->set_not_resumed ();
+
+      if (ws.kind () != TARGET_WAITKIND_STOPPED
+	  || ws.sig () != GDB_SIGNAL_0)
+	{
+	  if (evthread->has_pending_waitstatus ())
+	    evthread->clear_pending_waitstatus ();
+	  evthread->set_pending_waitstatus (ws);
+	}
     }
 
   /* "Notice" the new inferiors before anything related to
@@ -8384,7 +8388,7 @@ remote_target::queued_stop_reply (ptid_t ptid)
   remote_state *rs = get_remote_state ();
   stop_reply_up r = remote_notif_remove_queued_reply (ptid);
 
-  if (!rs->stop_reply_queue.empty () && target_can_async_p ())
+  if (!rs->stop_reply_queue.empty () && target_is_async_p ())
     {
       /* There's still at least an event left.  */
       rs->mark_async_event_handler ();
-- 
2.43.0

Re: [PATCH] gdb/remote: fix assertions when attaching in non-stop mode

[Simon Marchi]

On 6/11/26 1:36 AM, Martin KOCH wrote:
> When connecting to a remote target in non-stop mode against a
> multi-threaded inferior stopped at raise(SIGSTOP), three internal-error
> assertions can fire in sequence:
> 
>   remote.c:546: mark_async_event_handler: Assertion 'this->is_async_p ()' failed.
>   thread.c:429: set_pending_waitstatus: Assertion 'this->internal_state () ==
>                 THREAD_INT_STOPPED || ...' failed.
>   thread.c:426: set_pending_waitstatus: Assertion '!this->has_pending_waitstatus ()' failed.
> 
> The first one is what PR 30630 reports and is fixed with the PR proposed
> by Mikhail Terekhov, but with the fix the two other assertions surface:
> 
>  * thread.c:429: addressed by reordering set_internal_state / set_state
>    to run before set_pending_waitstatus.
>  * thread.c:426: addressed by clearing any existing pending wait
>    status before installing the new one, when gdbserver delivers
>    multiple events for the same thread.
> 
> [1] https://sourceware.org/pipermail/gdb-patches/2023-October/202937.html
> 
> Bug: https://sourceware.org/bugzilla/show_bug.cgi?id=30630
> Signed-off-by: Martin KOCH <Martin.KOCH@bachmann.info>

Hi Martin,

We will definitely want a test for this.

I played with the reproducer a little bit, I think that the
target_is_async_p and the reorder set_internal_state vs
set_pending_waitstatus fixes make sense.

For the "clear duplicate waitstatus" one, I'd like to understand the
root cause a bit better.  The fix is not necessarily bad, in order to
protect against misbehaving remotes, but I'm thinking that if GDB really
sees duplicate stop replies for a given thread, then perhaps gdbserver
is doing something wrong that should also be fixed.

I looked at the RSP logs, and saw this:

  [remote] Sending packet: $?#3f
  [remote] Packet received: T1206:e0419624fd7f0000;07:b8419624fd7f0000;10:520aaa1ba67f0000;thread:p194d4.194d4;core:a;
  [remote] Sending packet: $vStopped#55
  [remote] Packet received: T1206:c0ed9f1ba67f0000;07:98ed9f1ba67f0000;10:520aaa1ba67f0000;thread:p194d4.194d5;core:d;
  [remote] Sending packet: $vStopped#55
  [remote] Packet received: OK

Seems fine, one event for each thread.  But I scrolled up and saw:

  [remote] Sending packet: $QNonStop:1#8d
  [remote] Packet received: OK
  [remote] Sending packet: $qXfer:threads:read::0,1000#92
  [remote]   Notification received: Stop:T1206:e0419624fd7f0000;07:b8419624fd7f0000;10:520aaa1ba67f0000;thread:p194d4.194d4;core:a;
  [remote] Packet received: l<threads>\n<thread id="p194d4.194d4" core="10" name="test"/>\n<thread id="p194d4.194d5" core="13" name="test"/>\n</threads>\n
  [remote] Sending packet: $vStopped#55
  [remote] Packet received: OK

That's probably it: early on, GDB sent an asynchronous stop reply for
one of the threads.  So here, I end up with two stop replies for thread
194d4.

On page

  https://sourceware.org/gdb/current/onlinedocs/gdb.html/Remote-Non_002dStop.html#Remote-Non_002dStop

we read:

  In non-stop mode, the target shall respond to the ‘?’ packet as
  follows. First, any incomplete stop reply notification/‘vStopped’
  sequence in progress is abandoned. The target must begin a new
  sequence reporting stop events for all stopped threads, whether or not
  it has previously reported those events to GDB. The first stop reply
  is sent as a synchronous reply to the ‘?’ packet, and subsequent stop
  replies are sent as responses to ‘vStopped’ packets using the
  mechanism described above. The target must not send asynchronous stop
  reply notifications until the sequence is complete. If all threads are
  running when the target receives the ‘?’ packet, or if the target is
  not attached to any process, it shall respond ‘OK’.

I could be wrong, but my interpretation of the "target must not send
asynchronous stop reply notifications until the sequence is complete"
part is that gdbserver should refrain from sending async stop replies
until GDB has sent packet ? and the whole dialog for packet ? is over.
So that would be a gdbserver bug.

Another way to interpret the sentence could be that the target must not
send async stop replies only during the window that starts when
receiving the ? packet and ends when sending the final OK that ends the
? dialog.  But that would be surprising.

If we agree it's a gdbserver bug, then ideally we would have:

 - patch that fixes the gdbserver bug
 - patch that fixes the other two asserts, with a test (it should work
   fine at this point)
 - optionally, patch that adds handling for duplicate stop replies /
   misbehaving targets

To test the last one, it is always possible to add knobs to gdbserver to
make it misbehave on purpose.  Or, there are tests that run the program
once to record the RSP communication (for use with gdbreplay), modify
the log to insert something wrong (like here, you could make the target
send back two stop replies for the same thread), and replay using
gdbreplay.  But this is starting to sound like some scope creep, if we
just had the first two I'd be happy.

Also, I just saw that patches 2 and 3 in Mohamed's series here are
basically the same fixes as you.

https://inbox.sourceware.org/gdb-patches/20260518183316.127043-1-mohamed.bouhaouel@intel.com/M

Either way, I think it would be good to have a test based on the
reproducer in bug 30630.

Thanks,

Simon

[PATCH v2] gdb/remote: fix assertions when attaching in non-stop mode

[Martin KOCH]

When connecting to a remote target in non-stop mode against a
multi-threaded inferior stopped at raise(SIGSTOP), three internal-error
assertions can fire in sequence:

  remote.c:546: mark_async_event_handler: Assertion 'this->is_async_p ()' failed.
  thread.c:429: set_pending_waitstatus: Assertion 'this->internal_state () ==
                THREAD_INT_STOPPED || ...' failed.
  thread.c:426: set_pending_waitstatus: Assertion '!this->has_pending_waitstatus ()' failed.

The first one is what PR 30630 reports and is fixed with the PR proposed
by Mikhail Terekhov, but with the fix the two other assertions surface:

 * thread.c:429: addressed by reordering set_internal_state / set_state
   to run before set_pending_waitstatus.
 * thread.c:426: addressed by clearing any existing pending wait
   status before installing the new one, when gdbserver delivers
   multiple events for the same thread.

These assertions are reached through
remote.c:process_initial_stop_replies, which only runs on the initial
connection to a remote target, not via GDB's own "attach" command.  The
new test gdb.threads/attach-non-stop-stopped.exp reproduces the issue:
it starts gdbserver attached to a multi-threaded inferior that is
already stopped via a pending SIGSTOP, then connects GDB to it in
non-stop mode.  Without this fix the connection trips the assertions
above; with it, the connection succeeds.

[1] https://sourceware.org/pipermail/gdb-patches/2023-October/202937.html

Bug: https://sourceware.org/bugzilla/show_bug.cgi?id=30630
Signed-off-by: Martin KOCH <Martin.KOCH@bachmann.info>
---
 gdb/remote.c                                  |  14 ++-
 .../gdb.threads/attach-non-stop-stopped.c     |  78 +++++++++++++
 .../gdb.threads/attach-non-stop-stopped.exp   | 104 ++++++++++++++++++
 3 files changed, 191 insertions(+), 5 deletions(-)
 create mode 100644 gdb/testsuite/gdb.threads/attach-non-stop-stopped.c
 create mode 100644 gdb/testsuite/gdb.threads/attach-non-stop-stopped.exp

diff --git a/gdb/remote.c b/gdb/remote.c
index 2961664cf33..4148301bc6f 100644
--- a/gdb/remote.c
+++ b/gdb/remote.c
@@ -5237,13 +5237,17 @@ remote_target::process_initial_stop_replies (int from_tty)
 	  ws.set_stopped (sig);
 	}
 
-      if (ws.kind () != TARGET_WAITKIND_STOPPED
-	  || ws.sig () != GDB_SIGNAL_0)
-	evthread->set_pending_waitstatus (ws);
-
       set_internal_state (this, event_ptid, THREAD_INT_STOPPED);
       set_state (this, event_ptid, THREAD_STOPPED);
       get_remote_thread_info (evthread)->set_not_resumed ();
+
+      if (ws.kind () != TARGET_WAITKIND_STOPPED
+	  || ws.sig () != GDB_SIGNAL_0)
+	{
+	  if (evthread->has_pending_waitstatus ())
+	    evthread->clear_pending_waitstatus ();
+	  evthread->set_pending_waitstatus (ws);
+	}
     }
 
   /* "Notice" the new inferiors before anything related to
@@ -8384,7 +8388,7 @@ remote_target::queued_stop_reply (ptid_t ptid)
   remote_state *rs = get_remote_state ();
   stop_reply_up r = remote_notif_remove_queued_reply (ptid);
 
-  if (!rs->stop_reply_queue.empty () && target_can_async_p ())
+  if (!rs->stop_reply_queue.empty () && target_is_async_p ())
     {
       /* There's still at least an event left.  */
       rs->mark_async_event_handler ();
diff --git a/gdb/testsuite/gdb.threads/attach-non-stop-stopped.c b/gdb/testsuite/gdb.threads/attach-non-stop-stopped.c
new file mode 100644
index 00000000000..c58562892e9
--- /dev/null
+++ b/gdb/testsuite/gdb.threads/attach-non-stop-stopped.c
@@ -0,0 +1,78 @@
+/* This testcase is part of GDB, the GNU debugger.
+
+   Copyright 2026 Free Software Foundation, Inc.
+
+   This program is free software; you can redistribute it and/or modify
+   it under the terms of the GNU General Public License as published by
+   the Free Software Foundation; either version 3 of the License, or
+   (at your option) any later version.
+
+   This program is distributed in the hope that it will be useful,
+   but WITHOUT ANY WARRANTY; without even the implied warranty of
+   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+   GNU General Public License for more details.
+
+   You should have received a copy of the GNU General Public License
+   along with this program.  If not, see <http://www.gnu.org/licenses/>.  */
+
+/* This program creates a few threads and then stops the whole process
+   with a pending SIGSTOP, leaving it group-stopped.  It is intended to
+   be started outside of GDB and then attached to.  See
+   attach-non-stop-stopped.exp.  */
+
+#include <pthread.h>
+#include <unistd.h>
+#include <signal.h>
+#include <assert.h>
+
+/* Number of threads we'll create, in addition to the main thread.  */
+#define N_THREADS 4
+
+/* Used to make sure all threads are running before we stop the
+   process.  */
+static pthread_barrier_t barrier;
+
+/* Entry point for threads.  Loops forever.  */
+
+static void *
+thread_func (void *arg)
+{
+  pthread_barrier_wait (&barrier);
+
+  while (1)
+    sleep (1);
+
+  return NULL;
+}
+
+int
+main (void)
+{
+  pthread_t threads[N_THREADS];
+  int i;
+
+  /* Safety net, in case the process is left running.  */
+  alarm (300);
+
+  pthread_barrier_init (&barrier, NULL, N_THREADS + 1);
+
+  for (i = 0; i < N_THREADS; ++i)
+    {
+      int rc = pthread_create (&threads[i], NULL, thread_func, NULL);
+      assert (rc == 0);
+    }
+
+  /* Wait until all threads are up and running.  */
+  pthread_barrier_wait (&barrier);
+
+  /* Stop the whole process (all threads) with a real, pending SIGSTOP.
+     When GDB later attaches in non-stop mode against a remote target,
+     each thread is reported with this pending signal.  That is what
+     used to trigger the assertions described in PR gdb/30630.  */
+  raise (SIGSTOP);
+
+  while (1)
+    sleep (1);
+
+  return 0;
+}
diff --git a/gdb/testsuite/gdb.threads/attach-non-stop-stopped.exp b/gdb/testsuite/gdb.threads/attach-non-stop-stopped.exp
new file mode 100644
index 00000000000..af68f8f781a
--- /dev/null
+++ b/gdb/testsuite/gdb.threads/attach-non-stop-stopped.exp
@@ -0,0 +1,104 @@
+# Copyright 2026 Free Software Foundation, Inc.
+
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+# Regression test for PR gdb/30630.
+#
+# Start gdbserver attached to a multi-threaded inferior that is already
+# stopped (here, via a pending SIGSTOP), then connect GDB to it in
+# non-stop mode.
+
+load_lib gdbserver-support.exp
+
+require allow_gdbserver_tests
+require can_spawn_for_attach
+
+standard_testfile
+
+if {[build_executable "failed to prepare" $testfile $srcfile \
+	 {debug pthreads}] == -1} {
+    return
+}
+
+# Number of threads the test program creates (see N_THREADS in .c file)
+set n_threads 4
+set total [expr {$n_threads + 1}]
+
+# Spawn the program.  It creates its threads and then stops the whole
+# process with a pending SIGSTOP.
+set test_spawn_id [spawn_wait_for_attach $binfile]
+set testpid [spawn_id_get_pid $test_spawn_id]
+
+save_vars { GDBFLAGS } {
+    append GDBFLAGS " -ex \"set non-stop on\""
+
+    if { ![is_remote host] && ![is_remote target] } {
+	append GDBFLAGS " -ex \"set sysroot\""
+    }
+
+    clean_restart $testfile
+}
+
+gdb_test "disconnect" ".*"
+
+# Start gdbserver already attached to the stopped process.
+set res [gdbserver_start "--attach" $testpid]
+if { [llength $res] == 0 } {
+    kill_wait_spawned_process $test_spawn_id
+    unsupported "gdbserver --attach failed"
+    return
+}
+set gdbserver_gdbport [lindex $res 1]
+
+# Connect to gdbserver in non-stop mode.  Without the fix for PR 30630,
+# GDB hits an internal error here.
+set connected 0
+gdb_test_multiple "target remote $gdbserver_gdbport" \
+    "connect to gdbserver in non-stop mode" {
+    -re "Non-stop mode requested, but remote does not support non-stop.*$gdb_prompt $" {
+	unsupported $gdb_test_name
+    }
+    -re "Remote debugging using .*$gdb_prompt $" {
+	pass $gdb_test_name
+	set connected 1
+    }
+}
+
+if { !$connected } {
+    kill_wait_spawned_process $test_spawn_id
+    return
+}
+
+# GDB survived the connection.  As a sanity check, confirm it is
+# responsive and that all threads are present.
+for {set attempt 0} {$attempt < 10} {incr attempt} {
+    set thread_count 0
+    gdb_test_multiple "info threads" "" {
+	-re "\r\n\[ *\]+$decimal\[ \t\]+(Thread|LWP|process)\[^\r\n\]*" {
+	    incr thread_count
+	    exp_continue
+	}
+	-re "$gdb_prompt " {
+	}
+    }
+    if {$thread_count >= $total} {
+	break
+    }
+    sleep 1
+}
+gdb_assert {$thread_count == $total} "all threads present"
+
+gdb_test "detach" "Detaching from.*"
+
+kill_wait_spawned_process $test_spawn_id

base-commit: ddc575edf104eb477018a32b31074beffb179034
-- 
2.43.0

Re: [PATCH v2] gdb/remote: fix assertions when attaching in non-stop mode

[Simon Marchi]

On 6/19/26 5:25 AM, Martin KOCH wrote:
> When connecting to a remote target in non-stop mode against a
> multi-threaded inferior stopped at raise(SIGSTOP), three internal-error
> assertions can fire in sequence:
> 
>   remote.c:546: mark_async_event_handler: Assertion 'this->is_async_p ()' failed.
>   thread.c:429: set_pending_waitstatus: Assertion 'this->internal_state () ==
>                 THREAD_INT_STOPPED || ...' failed.
>   thread.c:426: set_pending_waitstatus: Assertion '!this->has_pending_waitstatus ()' failed.
> 
> The first one is what PR 30630 reports and is fixed with the PR proposed
> by Mikhail Terekhov, but with the fix the two other assertions surface:
> 
>  * thread.c:429: addressed by reordering set_internal_state / set_state
>    to run before set_pending_waitstatus.
>  * thread.c:426: addressed by clearing any existing pending wait
>    status before installing the new one, when gdbserver delivers
>    multiple events for the same thread.

As mentioned in me previous message, the root cause of this is likely a
bug in gdbserver.  It's good to make gdb resilient against bugs on the
remote side, but I wouldn't want to merge this in gdb before fixing the
gdbserver bug.  Otherwise, it will get forgotten about and it will never
get fixed.

> These assertions are reached through
> remote.c:process_initial_stop_replies, which only runs on the initial
> connection to a remote target, not via GDB's own "attach" command.  The
> new test gdb.threads/attach-non-stop-stopped.exp reproduces the issue:
> it starts gdbserver attached to a multi-threaded inferior that is
> already stopped via a pending SIGSTOP, then connects GDB to it in
> non-stop mode.  Without this fix the connection trips the assertions
> above; with it, the connection succeeds.
> 
> [1] https://sourceware.org/pipermail/gdb-patches/2023-October/202937.html
> 
> Bug: https://sourceware.org/bugzilla/show_bug.cgi?id=30630
> Signed-off-by: Martin KOCH <Martin.KOCH@bachmann.info>
> ---
>  gdb/remote.c                                  |  14 ++-
>  .../gdb.threads/attach-non-stop-stopped.c     |  78 +++++++++++++
>  .../gdb.threads/attach-non-stop-stopped.exp   | 104 ++++++++++++++++++
>  3 files changed, 191 insertions(+), 5 deletions(-)
>  create mode 100644 gdb/testsuite/gdb.threads/attach-non-stop-stopped.c
>  create mode 100644 gdb/testsuite/gdb.threads/attach-non-stop-stopped.exp
> 
> diff --git a/gdb/remote.c b/gdb/remote.c
> index 2961664cf33..4148301bc6f 100644
> --- a/gdb/remote.c
> +++ b/gdb/remote.c
> @@ -5237,13 +5237,17 @@ remote_target::process_initial_stop_replies (int from_tty)
>  	  ws.set_stopped (sig);
>  	}
>  
> -      if (ws.kind () != TARGET_WAITKIND_STOPPED
> -	  || ws.sig () != GDB_SIGNAL_0)
> -	evthread->set_pending_waitstatus (ws);
> -
>        set_internal_state (this, event_ptid, THREAD_INT_STOPPED);
>        set_state (this, event_ptid, THREAD_STOPPED);
>        get_remote_thread_info (evthread)->set_not_resumed ();
> +
> +      if (ws.kind () != TARGET_WAITKIND_STOPPED
> +	  || ws.sig () != GDB_SIGNAL_0)
> +	{

Here, add a comment like:

/* This should not happen, but a misbehaving remote could send duplicate
   stop replies for a given thread.  We arbitrarily keep the last one.  */

> +	  if (evthread->has_pending_waitstatus ())
> +	    evthread->clear_pending_waitstatus ();
> +	  evthread->set_pending_waitstatus (ws);

Newline after the if.

> +# GDB survived the connection.  As a sanity check, confirm it is
> +# responsive and that all threads are present.
> +for {set attempt 0} {$attempt < 10} {incr attempt} {
> +    set thread_count 0
> +    gdb_test_multiple "info threads" "" {
> +	-re "\r\n\[ *\]+$decimal\[ \t\]+(Thread|LWP|process)\[^\r\n\]*" {
> +	    incr thread_count
> +	    exp_continue
> +	}
> +	-re "$gdb_prompt " {
> +	}
> +    }
> +    if {$thread_count >= $total} {
> +	break
> +    }
> +    sleep 1
> +}

What is the reason for the "attempt" loop and the sleep?  Shouldn't all
threads be visible from the start?

Simon