[RFA] win32-nat.c: Real detaching from processes under Windows XP

[Corinna Vinschen <vinschen@redhat.com>]

Hi,

Windows XP introduces an exciting new feature to the Windows debugger
world.  Under XP a native debugger actually can _detach_ from processes.

I added that feature to win32-nat.c so that the needed functions
are loaded dynamically.  If that fails, gdb assumes that we are not
running under XP and behaves as before.  Otherwise it will now behave
as any other system allowing detaching from processes.

It works fine except for a special case:

If GDB has started the process by itself (in contrast to attaching
to a running process) the detaching might crash the inferior process
when the user has just stepped into a function or gdb just stopped on
a breakpoint right at the functions entry point.  If at that point
the user steps further one or two steps, everything's fine but if the
user detaches at that point, the inferior suffers from a SEGV due
to an apparently corrupted stack.

Then I found the following comment in the GDB source, right above
infcmd.c, detach_command():

/*
 * detach_command --
 * takes a program previously attached to and detaches it.
 * The program resumes execution and will no longer stop
 * on signals, etc.  We better not have left any breakpoints
 * in the program or it'll die when it hits one.  For this
 * to work, it may be necessary for the process to have been
 * previously attached.  It *might* work if the program was
 * started via the normal ptrace (PTRACE_TRACEME).
 */

So, AFAICS, WinXP doesn't behave that different from other systems.

Corinna

2001-10-24  Corinna Vinschen  <vinschen@redhat.com>

	* win32-nat.c (DebugSetProcessKillOnExit): New static function
	pointer to Windows' DebugSetProcessKillOnExit() function.
	(DebugActiveProcessStop): Ditto for DebugActiveProcessStop().
	(has_detach_ability): New function.
	(child_attach): If system has detach ability, enable it.
	(child_detach): If system has detach ability, actually
	detach from process.
	Change tty output to Linux format.

Index: win32-nat.c
===================================================================
RCS file: /cvs/src/src/gdb/win32-nat.c,v
retrieving revision 1.32
diff -u -p -r1.32 win32-nat.c
--- win32-nat.c	2001/10/20 05:11:41	1.32
+++ win32-nat.c	2001/10/24 09:49:32
@@ -1022,8 +1022,36 @@ do_initial_child_stuff (DWORD pid)
   return;
 }
 
-/* Attach to process PID, then initialize for debugging it.  */
+/* Since Windows XP, detaching from a process is supported by Windows.
+   The following code tries loading the appropriate functions dynamically.
+   If loading these functions succeeds use them to actually detach from
+   the inferior process, otherwise behave as usual, pretending that
+   detach has worked. */
+static BOOL WINAPI (*DebugSetProcessKillOnExit)(BOOL);
+static BOOL WINAPI (*DebugActiveProcessStop)(DWORD);
+
+static int
+has_detach_ability ()
+{
+  static HMODULE kernel32 = NULL;
+
+  if (!kernel32)
+    kernel32 = LoadLibrary ("kernel32.dll");
+  if (kernel32)
+    {
+      if (!DebugSetProcessKillOnExit)
+	DebugSetProcessKillOnExit = GetProcAddress (kernel32,
+						 "DebugSetProcessKillOnExit");
+      if (!DebugActiveProcessStop)
+	DebugActiveProcessStop = GetProcAddress (kernel32,
+						 "DebugActiveProcessStop");
+      if (DebugSetProcessKillOnExit && DebugActiveProcessStop)
+	return 1;
+    }
+  return 0;
+}
 
+/* Attach to process PID, then initialize for debugging it.  */
 static void
 child_attach (char *args, int from_tty)
 {
@@ -1039,6 +1067,12 @@ child_attach (char *args, int from_tty)
   if (!ok)
     error ("Can't attach to process.");
 
+  if (has_detach_ability ())
+    {
+      attach_flag = 1;
+      DebugSetProcessKillOnExit (FALSE);
+    }
+
   if (from_tty)
     {
       char *exec_file = (char *) get_exec_file (0);
@@ -1060,13 +1094,27 @@ child_attach (char *args, int from_tty)
 static void
 child_detach (char *args ATTRIBUTE_UNUSED, int from_tty)
 {
-  if (from_tty)
+  int detached = 1;
+
+  if (has_detach_ability ())
+    {
+      delete_command (NULL, 0);
+      child_continue (DBG_CONTINUE, -1);
+      if (!DebugActiveProcessStop (current_event.dwProcessId))
+        {
+	  error ("Can't detach process %lu (error %lu)",
+		 current_event.dwProcessId, GetLastError ());
+	  detached = 0;
+        }
+      DebugSetProcessKillOnExit (FALSE);
+    }
+  if (detached && from_tty)
     {
       char *exec_file = get_exec_file (0);
       if (exec_file == 0)
 	exec_file = "";
-      printf_unfiltered ("Detaching from program: %s %s\n", exec_file,
-			 target_pid_to_str (inferior_ptid));
+      printf_unfiltered ("Detaching from program: %s, Pid %lu\n", exec_file,
+			 current_event.dwProcessId);
       gdb_flush (gdb_stdout);
     }
   inferior_ptid = null_ptid;