[PATCH v2 44/47] Add gdb.threads/leader-exit-schedlock.exp

[Pedro Alves <pedro@palves.net>]

This adds a new test for letting the main thread exit the process with
scheduler-locking on, while there are other threads live.

On Linux, when the main thread exits without causing a whole-process
exit (e.g., via the main thread doing pthread_exit), the main thread
becomes zombie but does not report a thread exit event.  When
eventually all other threads of the process exit, the main thread is
unblocked out of its zombie state and reports its exit which we
interpret as the whole-process exit.

If the main-thread-exit causes a whole-process exit (e.g., via the
exit syscall), the process is the same, except that the exit syscall
makes the kernel force-close all threads immediately.

Importantly, the main thread on Linux is always the last thread that
reports the exit event.

On Windows, the main thread exiting is not special at all.  When the
main thread causes a process exit (e.g., for ExitProcess or by
returning from main), the debugger sees a normal thread exit event for
the main thread.  All other threads will follow up with a thread-exit
event too, except whichever thread happens to be the last one.  That
last one is the one that reports a whole-process-exit event instead of
an exit-thread event.  So, since programs are typically multi-threaded
on Windows (because the OS/runtime spawns some threads), when the main
thread just returns from main(), it is very typically _not_ the main
thread that reports the whole-process exit.

As a result, stepping the main thread with schedlock on Windows
results in the main thread exiting and the continue aborting due to
no-resumed-threads left instead of a whole-process exit as seen on
Linux:

(gdb) info threads
  Id   Target Id                                    Frame
* 1    Thread 11768.0x1bc "leader-exit-schedlock"   main () at .../gdb.threads/leader-exit-schedlock.c:55
  2    Thread 11768.0x31e0 (in kernel)              0x00007ffbb23dfc77 in ntdll!ZwWaitForWorkViaWorkerFactory () from C:/WINDOWS/SYSTEM32/ntdll.dll
  3    Thread 11768.0x2dec "sig" (in kernel)        0x00007ffbb23dc087 in ntdll!ZwReadFile () from C:/WINDOWS/SYSTEM32/ntdll.dll
  4    Thread 11768.0x2530 (in kernel)              0x00007ffbb23dfc77 in ntdll!ZwWaitForWorkViaWorkerFactory () from C:/WINDOWS/SYSTEM32/ntdll.dll
  5    Thread 11768.0x3384 "leader-exit-schedlock"  0x00007ffbb23dcb17 in ntdll!ZwWaitForMultipleObjects () from C:/WINDOWS/SYSTEM32/ntdll.dll
  6    Thread 11768.0x3198 "leader-exit-schedlock"  0x00007ffbb23dcb17 in ntdll!ZwWaitForMultipleObjects () from C:/WINDOWS/SYSTEM32/ntdll.dll
  7    Thread 11768.0x1ab8 "leader-exit-schedlock"  0x00007ffbb23dcb17 in ntdll!ZwWaitForMultipleObjects () from C:/WINDOWS/SYSTEM32/ntdll.dll
  8    Thread 11768.0x3fe4 "leader-exit-schedlock"  0x00007ffbb23dcb17 in ntdll!ZwWaitForMultipleObjects () from C:/WINDOWS/SYSTEM32/ntdll.dll
  9    Thread 11768.0x3b5c "leader-exit-schedlock"  0x00007ffbb23dcb17 in ntdll!ZwWaitForMultipleObjects () from C:/WINDOWS/SYSTEM32/ntdll.dll
  10   Thread 11768.0x45c "leader-exit-schedlock"   0x00007ffbb23dcb17 in ntdll!ZwWaitForMultipleObjects () from C:/WINDOWS/SYSTEM32/ntdll.dll
  11   Thread 11768.0x3724 "leader-exit-schedlock"  0x00007ffbb23dcb17 in ntdll!ZwWaitForMultipleObjects () from C:/WINDOWS/SYSTEM32/ntdll.dll
  12   Thread 11768.0x1e44 "leader-exit-schedlock"  0x00007ffbb23dcb17 in ntdll!ZwWaitForMultipleObjects () from C:/WINDOWS/SYSTEM32/ntdll.dll
  13   Thread 11768.0x23f0 "leader-exit-schedlock"  0x00007ffbb23dcb17 in ntdll!ZwWaitForMultipleObjects () from C:/WINDOWS/SYSTEM32/ntdll.dll
  14   Thread 11768.0x3b80 "leader-exit-schedlock"  0x00007ffbb23dcb17 in ntdll!ZwWaitForMultipleObjects () from C:/WINDOWS/SYSTEM32/ntdll.dll
(gdb) set scheduler-locking on
(gdb) c
Continuing.
[Thread 11768.0x1bc exited]
No unwaited-for children left.
(gdb) info threads
  Id   Target Id                                                     Frame
  2    Thread 11768.0x31e0 (exiting)                                 0x00007ffbb23dfc77 in ntdll!ZwWaitForWorkViaWorkerFactory () from C:/WINDOWS/SYSTEM32/ntdll.dll
  3    Thread 11768.0x2dec "sig" (exiting)                           0x00007ffbb23dc087 in ntdll!ZwReadFile () from C:/WINDOWS/SYSTEM32/ntdll.dll
  4    Thread 11768.0x2530 (exiting)                                 0x00007ffbb23dfc77 in ntdll!ZwWaitForWorkViaWorkerFactory () from C:/WINDOWS/SYSTEM32/ntdll.dll
  5    Thread 11768.0x3384 "leader-exit-schedlock" (exiting)         0x00007ffbb23dcb17 in ntdll!ZwWaitForMultipleObjects () from C:/WINDOWS/SYSTEM32/ntdll.dll
  6    Thread 11768.0x3198 "leader-exit-schedlock" (exiting)         0x00007ffbb23dcb17 in ntdll!ZwWaitForMultipleObjects () from C:/WINDOWS/SYSTEM32/ntdll.dll
  7    Thread 11768.0x1ab8 "leader-exit-schedlock" (exiting)         0x00007ffbb23dcb17 in ntdll!ZwWaitForMultipleObjects () from C:/WINDOWS/SYSTEM32/ntdll.dll
  8    Thread 11768.0x3fe4 "leader-exit-schedlock" (exiting)         0x00007ffbb23dcb17 in ntdll!ZwWaitForMultipleObjects () from C:/WINDOWS/SYSTEM32/ntdll.dll
  9    Thread 11768.0x3b5c "leader-exit-schedlock" (exiting)         0x00007ffbb23dcb17 in ntdll!ZwWaitForMultipleObjects () from C:/WINDOWS/SYSTEM32/ntdll.dll
  10   Thread 11768.0x45c "leader-exit-schedlock" (exiting)          0x00007ffbb23dcb17 in ntdll!ZwWaitForMultipleObjects () from C:/WINDOWS/SYSTEM32/ntdll.dll
  11   Thread 11768.0x3724 "leader-exit-schedlock" (exiting)         0x00007ffbb23dcb17 in ntdll!ZwWaitForMultipleObjects () from C:/WINDOWS/SYSTEM32/ntdll.dll
  12   Thread 11768.0x1e44 "leader-exit-schedlock" (exiting)         0x00007ffbb23dcb17 in ntdll!ZwWaitForMultipleObjects () from C:/WINDOWS/SYSTEM32/ntdll.dll
  13   Thread 11768.0x23f0 "leader-exit-schedlock" (exiting)         0x00007ffbb23dcb17 in ntdll!ZwWaitForMultipleObjects () from C:/WINDOWS/SYSTEM32/ntdll.dll
  14   Thread 11768.0x3b80 "leader-exit-schedlock" (exiting process) 0x00007ffbb23dcb17 in ntdll!ZwWaitForMultipleObjects () from C:/WINDOWS/SYSTEM32/ntdll.dll

The current thread <Thread ID 1> has terminated.  See `help thread'.
(gdb)

The "(exiting)" and "(exiting process)" threads are threads for which
the kernel already reported their exit to GDB's Windows backend (via
WaitForDebugEvent), but the Windows backend hasn't yet reported the
event to infrun.  The events are still pending in windows-nat.c.

The "(exiting process)" thread above (thread 14) is the one that won
the process-exit event lottery on the Windows kernel side (because it
was the last to exit).  Continuing the (exiting) threads with
schedlock enabled should result in the Windows backend reporting that
thread's pending exit to infrun.  While continuing thread 14 should
result in the inferior exiting.  Vis:

 (gdb) c
 Continuing.
 [Thread 11768.0x31e0 exited]
 No unwaited-for children left.
 (gdb) t 14
 [Switching to thread 14 (Thread 11768.0x3b80)]
 #0  0x00007ffbb23dcb17 in ntdll!ZwWaitForMultipleObjects () from C:/WINDOWS/SYSTEM32/ntdll.dll
 (gdb) c
 Continuing.
 [Inferior 1 (process 11768) exited normally]

The testcase continues all the (exiting) threads, one by one, and then
finally continues the (exiting process) one, expecting an inferior
exit.

The testcase also tries a similar scenario: instead immediately
continue the (exiting process) thread without continuing the others.
That should result in the inferior exiting immediately.

It is actually not guaranteed that the Windows backend will consume
all the thread and process exit events out of the kernel before the
first thread exit event is processed by infrun.  So often we will see
for example, instead:

(gdb) info threads
  Id   Target Id                                                     Frame
  2    Thread 11768.0x31e0 (exiting)                                 0x00007ffbb23dfc77 in ntdll!ZwWaitForWorkViaWorkerFactory () from C:/WINDOWS/SYSTEM32/ntdll.dll
  3    Thread 11768.0x2dec "sig" (exiting)                           0x00007ffbb23dc087 in ntdll!ZwReadFile () from C:/WINDOWS/SYSTEM32/ntdll.dll
  4    Thread 11768.0x2530 (exiting)                                 0x00007ffbb23dfc77 in ntdll!ZwWaitForWorkViaWorkerFactory () from C:/WINDOWS/SYSTEM32/ntdll.dll
  5    Thread 11768.0x3384 "leader-exit-schedlock" (exiting)         0x00007ffbb23dcb17 in ntdll!ZwWaitForMultipleObjects () from C:/WINDOWS/SYSTEM32/ntdll.dll
  6    Thread 11768.0x3198 "leader-exit-schedlock" (exiting)         0x00007ffbb23dcb17 in ntdll!ZwWaitForMultipleObjects () from C:/WINDOWS/SYSTEM32/ntdll.dll
  7    Thread 11768.0x1ab8 "leader-exit-schedlock" (exiting)         0x00007ffbb23dcb17 in ntdll!ZwWaitForMultipleObjects () from C:/WINDOWS/SYSTEM32/ntdll.dll
  8    Thread 11768.0x3fe4 "leader-exit-schedlock" (exiting)         0x00007ffbb23dcb17 in ntdll!ZwWaitForMultipleObjects () from C:/WINDOWS/SYSTEM32/ntdll.dll
  9    Thread 11768.0x3b5c "leader-exit-schedlock" (exiting)         0x00007ffbb23dcb17 in ntdll!ZwWaitForMultipleObjects () from C:/WINDOWS/SYSTEM32/ntdll.dll
  10   Thread 11768.0x45c "leader-exit-schedlock"                    0x00007ffbb23dcb17 in ntdll!ZwWaitForMultipleObjects () from C:/WINDOWS/SYSTEM32/ntdll.dll
  11   Thread 11768.0x3724 "leader-exit-schedlock"                   0x00007ffbb23dcb17 in ntdll!ZwWaitForMultipleObjects () from C:/WINDOWS/SYSTEM32/ntdll.dll
  12   Thread 11768.0x1e44 "leader-exit-schedlock"                   0x00007ffbb23dcb17 in ntdll!ZwWaitForMultipleObjects () from C:/WINDOWS/SYSTEM32/ntdll.dll
  13   Thread 11768.0x23f0 "leader-exit-schedlock" (exiting)         0x00007ffbb23dcb17 in ntdll!ZwWaitForMultipleObjects () from C:/WINDOWS/SYSTEM32/ntdll.dll
  14   Thread 11768.0x3b80 "leader-exit-schedlock" (exiting)         0x00007ffbb23dcb17 in ntdll!ZwWaitForMultipleObjects () from C:/WINDOWS/SYSTEM32/ntdll.dll

Above, we can't tell which thread will get the exit-process event,
there is no "(exiting process)" thread.  We do know it'll be one of
threads 10, 11, and 12, because those do not have "(exiting)".  The
Windows kernel has already decided which one it is at this point, we
just haven't seen the exit-process event yet.

This is actually what we _always_ see with "maint set target-non-stop
off" too, because in all-stop, the Windows backend only processes one
Windows debug event at a time.

So when the the test first continues all the (exiting) threads, one by
one, and then when there are no more "(exiting)" threads, if there is
no "(exiting process)" thread, it tries to exit the remaining threads,
(in the above case threads 10, 11 and 12), expecting that one of those
continues may cause an inferior exit.

On systems other than Windows, the testcase expects that continuing
the main thread results in an inferior exit.  If we find out that
isn't correct for some system, we can adjust the testcase then.

Change-Id: I52fb8de5e72bc12195ffb8bedd1d8070464332d3
---
 .../gdb.threads/leader-exit-schedlock.c       |  56 +++++
 .../gdb.threads/leader-exit-schedlock.exp     | 215 ++++++++++++++++++
 2 files changed, 271 insertions(+)
 create mode 100644 gdb/testsuite/gdb.threads/leader-exit-schedlock.c
 create mode 100644 gdb/testsuite/gdb.threads/leader-exit-schedlock.exp

diff --git a/gdb/testsuite/gdb.threads/leader-exit-schedlock.c b/gdb/testsuite/gdb.threads/leader-exit-schedlock.c
new file mode 100644
index 00000000000..25492f6c0a7
--- /dev/null
+++ b/gdb/testsuite/gdb.threads/leader-exit-schedlock.c
@@ -0,0 +1,56 @@
+/* This testcase is part of GDB, the GNU debugger.
+
+   Copyright 2025 Free Software Foundation, Inc.
+
+   This program is free software; you can redistribute it and/or modify
+   it under the terms of the GNU General Public License as published by
+   the Free Software Foundation; either version 3 of the License, or
+   (at your option) any later version.
+
+   This program is distributed in the hope that it will be useful,
+   but WITHOUT ANY WARRANTY; without even the implied warranty of
+   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+   GNU General Public License for more details.
+
+   You should have received a copy of the GNU General Public License
+   along with this program.  If not, see <http://www.gnu.org/licenses/>.  */
+
+#include <pthread.h>
+#include <assert.h>
+#include <unistd.h>
+
+static pthread_barrier_t threads_started_barrier;
+
+static void *
+start (void *arg)
+{
+  pthread_barrier_wait (&threads_started_barrier);
+
+  while (1)
+    sleep (1);
+
+  return NULL;
+}
+
+#define NTHREADS 10
+
+int
+main (void)
+{
+  int i;
+
+  pthread_barrier_init (&threads_started_barrier, NULL, NTHREADS + 1);
+
+  for (i = 0; i < NTHREADS; i++)
+    {
+      pthread_t thread;
+      int res;
+
+      res = pthread_create (&thread, NULL, start, NULL);
+      assert (res == 0);
+    }
+
+  pthread_barrier_wait (&threads_started_barrier);
+
+  return 0; /* break-here */
+}
diff --git a/gdb/testsuite/gdb.threads/leader-exit-schedlock.exp b/gdb/testsuite/gdb.threads/leader-exit-schedlock.exp
new file mode 100644
index 00000000000..a2745a22613
--- /dev/null
+++ b/gdb/testsuite/gdb.threads/leader-exit-schedlock.exp
@@ -0,0 +1,215 @@
+# Copyright (C) 2025 Free Software Foundation, Inc.
+
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+# On Linux, exiting the main thread with scheduler locking on results
+# in an inferior exit immediately.  On Windows, however, it results in
+# a normal thread exit of the main thread, with the other threads
+# staying listed.  Test this, and then test iterating over all the
+# other threads and continuing then too, one by one, with
+# scheduler-locking on.  Also test schedlock off, for completeness.
+
+standard_testfile
+
+if {[build_executable "failed to prepare" $testfile $srcfile {debug pthreads}]} {
+    return -1
+}
+
+# Run "info threads" and return a list of various information:
+#
+#  Element 0 - the highest numbered thread, zero if no thread is found.
+#  Element 1 - the "(exiting process)" thread, zero if not found.
+#  Element 2 - a list of "(exiting)" threads.
+#  Element 3 - a list of the other threads.
+
+proc info_threads {} {
+    set highest_thread 0
+    set exit_process_thread 0
+    set exit_thread_threads {}
+    set other_threads {}
+    set any "\[^\r\n\]*"
+    set ws "\[ \t\]*"
+    set eol "(?=\r\n)"
+    set common_prefix "^\r\n(?:\\*)?${ws}($::decimal)\[ \t\]*${::tdlabel_re}${any}"
+    gdb_test_multiple "info threads" "" -lbl {
+	-re "${common_prefix}\\(exiting process\\)${any}${eol}" {
+	    set highest_thread $expect_out(1,string)
+	    verbose -log "\nhighest_thread=$highest_thread, exiting process\n"
+	    set exit_process_thread $highest_thread
+	    exp_continue
+	}
+	-re "${common_prefix}\\(exiting\\)${any}${eol}" {
+	    set highest_thread $expect_out(1,string)
+	    verbose -log "\nhighest_thread=$highest_thread, exiting thread\n"
+	    lappend exit_thread_threads $highest_thread
+	    exp_continue
+	}
+	-re "${common_prefix}${eol}" {
+	    set highest_thread $expect_out(1,string)
+	    verbose -log "\nhighest_thread=$highest_thread, other thread\n"
+	    lappend other_threads $highest_thread
+	    exp_continue
+	}
+	-re "^\r\n$::gdb_prompt $" {
+	    verbose -log "\nhighest_thread=$highest_thread, prompt\n"
+	    gdb_assert {$highest_thread > 0} $gdb_test_name
+	}
+    }
+    verbose -log "info_threads: highest_thread=$highest_thread"
+    verbose -log "info_threads: exit_thread_threads=$exit_thread_threads"
+    verbose -log "info_threads: other_threads=$other_threads"
+    verbose -log "info_threads: exit_process_thread=$exit_process_thread"
+
+    return [list \
+		$highest_thread \
+		$exit_process_thread \
+		$exit_thread_threads \
+		$other_threads]
+}
+
+# If EXIT-THREADS-FIRST is true, continues all threads which have a
+# pending exit-thread event first, before continuing the thread with
+# the pending exit-process event.
+proc test {target-non-stop exit-threads-first schedlock} {
+    save_vars ::GDBFLAGS {
+	append ::GDBFLAGS " -ex \"maintenance set target-non-stop ${target-non-stop}\""
+	clean_restart $::binfile
+    }
+
+    set is_windows [expr [istarget *-*-mingw*] || [istarget *-*-cygwin*]]
+    set is_linux [istarget *-*-linux*]
+
+    if {!$is_windows && ${exit-threads-first}} {
+	# No point in exercising this combination because we will
+	# return before we reach the point where it is tested.
+	return
+    }
+
+    if ![runto_main] {
+	return
+    }
+
+    gdb_breakpoint [gdb_get_line_number "break-here"]
+    gdb_continue_to_breakpoint "break-here" ".* break-here .*"
+
+    gdb_test_no_output "set scheduler-locking $schedlock"
+
+    # Continuing the main thread on Linux makes the whole process
+    # exit.  This makes GDB report all threads exits immediately, and
+    # then the inferior exit.  The thread exits don't stay pending
+    # because Linux supports per-thread thread-exit control, while
+    # Windows is per-target.
+    if {!$is_windows || $schedlock == off} {
+	gdb_test_multiple "c" "continue exit-process thread to exit" {
+	    -re -wrap "Inferior.*exited normally.*" {
+		pass $gdb_test_name
+	    }
+	    -re -wrap "No unwaited-for children left.*" {
+		# On Linux, GDB may briefly see the main thread turn
+		# zombie before seeing its exit event.
+		gdb_assert $is_linux $gdb_test_name
+	    }
+	}
+
+	return
+    }
+
+    # On Windows, continuing the thread that calls TerminateProcess
+    # (the main thread when it returns from main in our case) with
+    # scheduler-locking enabled exits the whole process, but core of
+    # GDB won't see the exit process event right away.  Windows only
+    # reports it to the last thread that exits, whichever that is.
+    # Due to scheduler locking, that won't happen until we resume all
+    # other threads.  The TerminateProcess-caller thread gets a plain
+    # thread exit event.
+    gdb_test "c" "No unwaited-for children left\\." "continue main thread"
+
+    if {${target-non-stop} == "on"} {
+	# With non-stop. GDB issues ContinueDebugEvent as soon as it
+	# seens a debug event, so after a bit, the windows backend
+	# will have seen all the thread and process exit events, even
+	# while the user has the prompt.  Give it a bit of time for
+	# that to happen, so we can tell which threads have exited by
+	# looking for (exiting) and "(exiting process) in "info
+	# threads" output.
+	sleep 2
+    }
+
+    with_test_prefix "initial threads info" {
+	lassign [info_threads] \
+	    highest_thread \
+	    exit_process_thread \
+	    exit_thread_threads \
+	    other_threads
+
+	gdb_assert {$highest_thread > 0}
+    }
+
+    # Continue one thread at a time, collecting the exit status.
+    set thread_count $highest_thread
+    for {set i 2} {$i <= $thread_count} {incr i} {
+	with_test_prefix "thread $i" {
+	    lassign [info_threads] \
+		highest_thread \
+		exit_process_thread \
+		exit_thread_threads \
+		other_threads
+
+	    # Default to a value that forces FAILs below.
+	    set thr 0
+	    # Whether we expect to find a thread with "(exiting process)":
+	    #   0 - not expected - it's a failure if we see one.
+	    #   1 - possible - we may or may not see one.
+	    #   2 - required - it's a failure if we don't see one.
+	    set process_exit_expected 0
+
+	    if {$i == $thread_count} {
+		set thr $highest_thread
+		set process_exit_expected 2
+		gdb_test "p/d \$_inferior_thread_count == 1" " = 1" "one thread left"
+	    } else {
+		if {${exit-threads-first} && [llength $exit_thread_threads] != 0} {
+		    set thr [lindex $exit_thread_threads 0]
+		} elseif {$exit_process_thread > 0} {
+		    set thr $exit_process_thread
+		    set process_exit_expected 2
+		} elseif {[llength $other_threads] != 0} {
+		    set thr [lindex $other_threads 0]
+		    set process_exit_expected 1
+		}
+	    }
+
+	    gdb_test "thread $thr" \
+		"Switching to .*" \
+		"switch to thread"
+	    gdb_test_multiple "c" "continue thread to exit" {
+		-re -wrap "No unwaited-for children left\\." {
+		    gdb_assert {$process_exit_expected != 2} $gdb_test_name
+		}
+		-re -wrap "Inferior.*exited normally.*" {
+		    gdb_assert {$process_exit_expected != 0} $gdb_test_name
+		    return
+		}
+	    }
+	}
+    }
+}
+
+foreach_with_prefix target-non-stop {off on} {
+    foreach_with_prefix exit-threads-first {0 1} {
+	foreach_with_prefix schedlock {off on} {
+	    test ${target-non-stop} ${exit-threads-first} $schedlock
+	}
+    }
+}
-- 
2.49.0