[PATCH 0/3] [gdb/tui] Allow second tui enable if first fails

[PATCH 0/3] [gdb/tui] Allow second tui enable if first fails

[Tom de Vries]

Commit fb23d7ba4a2 ("[gdb/tui] Handle error in tui_enable") handles a
particular error during tui_enable, but doesn't allow trying to enable TUI
again.

This patch series fixes that.

The first two patches do some refactoring.

The third patch contains the fix.

Tom de Vries (3):
  [gdb/tui] Factor out require_tui_terminal
  [gdb/tui] Factor out init_ncurses
  [gdb/tui] Allow second tui enable if first fails

 gdb/tui/tui.c | 104 +++++++++++++++++++++++++++++++++-----------------
 1 file changed, 68 insertions(+), 36 deletions(-)


base-commit: 8a5e138e6276ce7644ab858751440cb59f35623b
-- 
2.51.0

[PATCH 1/3] [gdb/tui] Factor out require_tui_terminal

[Tom de Vries]

I noticed this bit of code in tui_enable:
...
      /* Check required terminal capabilities.  The MinGW port of
	 ncurses does have them, but doesn't expose them through "cup".  */
      cap = tigetstr ((char *) "cup");
      if (cap == NULL || cap == (char *) -1 || *cap == '\0')
	{
	  endwin ();
	  delscreen (s);
	  error (_("Cannot enable the TUI: "
		   "terminal doesn't support cursor addressing [TERM=%s]"),
		 gdb_getenv_term ());
	}
...

At this point in tui_enable, we need endwin and delscreen to clean up after
newterm, but the check can be done before newterm.

Fix this by factoring out a new function require_tui_terminal, and moving the
code there.
---
 gdb/tui/tui.c | 45 ++++++++++++++++++++++++---------------------
 1 file changed, 24 insertions(+), 21 deletions(-)

diff --git a/gdb/tui/tui.c b/gdb/tui/tui.c
index ba4f6f68769..0cafd361878 100644
--- a/gdb/tui/tui.c
+++ b/gdb/tui/tui.c
@@ -406,6 +406,28 @@ require_tui_interpreter ()
 	   interp);
 }
 
+/* Error out if the terminal doesn't support TUI.  */
+
+static void
+require_tui_terminal ()
+{
+  /* Don't try to setup curses (and print funny control
+     characters) if we're not outputting to a terminal.  */
+  if (!gdb_stderr->isatty ())
+    error (_("Cannot enable the TUI when output is not a terminal"));
+
+  /* Check required terminal capabilities.  The MinGW port of
+     ncurses does have them, but doesn't expose them through "cup".  */
+#ifndef __MINGW32__
+  const char *cap = tigetstr ((char *) "cup");
+  const char *not_a_string_capability = (char *) -1;
+  if (cap == nullptr || cap == not_a_string_capability || *cap == '\0')
+    error (_("Cannot enable the TUI: "
+	     "terminal doesn't support cursor addressing [TERM=%s]"),
+	   gdb_getenv_term ());
+#endif
+}
+
 /* Enter in the tui mode (curses).
    When in normal mode, it installs the tui hooks in gdb, redirects
    the gdb output, configures the readline to work in tui mode.
@@ -433,18 +455,13 @@ tui_enable (void)
     {
       WINDOW *w;
       SCREEN *s;
-#ifndef __MINGW32__
-       const char *cap;
-#endif
 
       /* If the top level interpreter is not the console/tui (e.g.,
 	 MI), enabling curses will certainly lose.  */
       require_tui_interpreter ();
 
-      /* Don't try to setup curses (and print funny control
-	 characters) if we're not outputting to a terminal.  */
-      if (!gdb_stderr->isatty ())
-	error (_("Cannot enable the TUI when output is not a terminal"));
+      /* Require a terminal that supports TUI.  */
+      require_tui_terminal ();
 
       /* Don't try initialization again.  */
       tui_finish_init = TRIBOOL_UNKNOWN;
@@ -472,20 +489,6 @@ tui_enable (void)
 	  start_color ();
 	}
 
-      /* Check required terminal capabilities.  The MinGW port of
-	 ncurses does have them, but doesn't expose them through "cup".  */
-#ifndef __MINGW32__
-      cap = tigetstr ((char *) "cup");
-      if (cap == NULL || cap == (char *) -1 || *cap == '\0')
-	{
-	  endwin ();
-	  delscreen (s);
-	  error (_("Cannot enable the TUI: "
-		   "terminal doesn't support cursor addressing [TERM=%s]"),
-		 gdb_getenv_term ());
-	}
-#endif
-
       /* We must mark the tui sub-system active before trying to setup the
 	 current layout as tui windows defined by an extension language
 	 rely on this flag being true in order to know that the window
-- 
2.51.0

[PATCH 2/3] [gdb/tui] Factor out init_ncurses

[Tom de Vries]

Factor out init_ncurses out of tui_enable, and make sure calling it twice is
harmless.
---
 gdb/tui/tui.c | 43 ++++++++++++++++++++++++++++++-------------
 1 file changed, 30 insertions(+), 13 deletions(-)

diff --git a/gdb/tui/tui.c b/gdb/tui/tui.c
index 0cafd361878..ea01448f087 100644
--- a/gdb/tui/tui.c
+++ b/gdb/tui/tui.c
@@ -428,6 +428,34 @@ require_tui_terminal ()
 #endif
 }
 
+/* Initialize ncurses, if necessary.  */
+
+static SCREEN *
+init_ncurses ()
+{
+  static SCREEN *tui_screen = nullptr;
+  if (tui_screen != nullptr)
+    {
+      /* Init ncurses only once.  */
+      return tui_screen;
+    }
+
+  tui_screen = newterm (nullptr, stdout, stdin);
+
+#ifdef __MINGW32__
+  /* The MinGW port of ncurses requires $TERM to be unset in order
+     to activate the Windows console driver.  */
+  if (tui_screen == nullptr)
+    tui_screen = newterm ((char *) "unknown", stdout, stdin);
+#endif
+
+  if (tui_screen == nullptr)
+    error (_("Cannot enable the TUI: error opening terminal [TERM=%s]"),
+	   gdb_getenv_term ());
+
+  return tui_screen;
+}
+
 /* Enter in the tui mode (curses).
    When in normal mode, it installs the tui hooks in gdb, redirects
    the gdb output, configures the readline to work in tui mode.
@@ -454,7 +482,6 @@ tui_enable (void)
   else if (tui_finish_init == TRIBOOL_TRUE)
     {
       WINDOW *w;
-      SCREEN *s;
 
       /* If the top level interpreter is not the console/tui (e.g.,
 	 MI), enabling curses will certainly lose.  */
@@ -466,19 +493,9 @@ tui_enable (void)
       /* Don't try initialization again.  */
       tui_finish_init = TRIBOOL_UNKNOWN;
 
-      s = newterm (NULL, stdout, stdin);
-#ifdef __MINGW32__
-      /* The MinGW port of ncurses requires $TERM to be unset in order
-	 to activate the Windows console driver.  */
-      if (s == NULL)
-	s = newterm ((char *) "unknown", stdout, stdin);
-#endif
-      if (s == NULL)
-	{
-	  error (_("Cannot enable the TUI: error opening terminal [TERM=%s]"),
-		 gdb_getenv_term ());
-	}
+      SCREEN *s = init_ncurses ();
       w = stdscr;
+
       if (has_colors ())
 	{
 #ifdef HAVE_USE_DEFAULT_COLORS
-- 
2.51.0

[PATCH 3/3] [gdb/tui] Allow second tui enable if first fails

[Tom de Vries]

Commit fb23d7ba4a2 ("[gdb/tui] Handle error in tui_enable") handles a
particular error during tui_enable, but doesn't allow trying to enable TUI
again.

I tried to get this to work, but didn't manage because I didn't understand the
interaction between endwin and the "tui_batch_rendering defer" destructor:
- first endwin restores the shell terminal mode
- then the defer destructor calls doupdate, restoring program shell mode.

Fix this by using reset_shell_mode instead.
---
 gdb/tui/tui.c | 18 +++++++++++++++---
 1 file changed, 15 insertions(+), 3 deletions(-)

diff --git a/gdb/tui/tui.c b/gdb/tui/tui.c
index ea01448f087..ee967e1e38b 100644
--- a/gdb/tui/tui.c
+++ b/gdb/tui/tui.c
@@ -493,7 +493,7 @@ tui_enable (void)
       /* Don't try initialization again.  */
       tui_finish_init = TRIBOOL_UNKNOWN;
 
-      SCREEN *s = init_ncurses ();
+      init_ncurses ();
       w = stdscr;
 
       if (has_colors ())
@@ -530,9 +530,21 @@ tui_enable (void)
 	}
       catch (const gdb_exception &)
 	{
-	  endwin ();
-	  delscreen (s);
+	  /* Clean up screen.  */
+	  clear ();
+	  refresh ();
+
+	  /* Reset terminal to shell terminal mode.  We can't use endwin here,
+	     because afterwards the defer destructor might call doupdate,
+	     restoring program terminal mode.  */
+	  reset_shell_mode ();
+
+	  /* Initialization failed, so TUI is not active.  */
 	  tui_active = false;
+
+	  /* Allow trying to initialize TUI again.  */
+	  tui_finish_init = TRIBOOL_TRUE;
+
 	  throw;
 	}
 
-- 
2.51.0

Re: [PATCH 3/3] [gdb/tui] Allow second tui enable if first fails

[Tom Tromey]

>>>>> "Tom" == Tom de Vries <tdevries@suse.de> writes:

Tom> Commit fb23d7ba4a2 ("[gdb/tui] Handle error in tui_enable") handles a
Tom> particular error during tui_enable, but doesn't allow trying to enable TUI
Tom> again.

Tom> I tried to get this to work, but didn't manage because I didn't understand the
Tom> interaction between endwin and the "tui_batch_rendering defer" destructor:
Tom> - first endwin restores the shell terminal mode
Tom> - then the defer destructor calls doupdate, restoring program shell mode.

This is probably just an oversight in tui_batch_rendering and perhaps
the destructor should check some other flag as well before calling doupdate.

Tom

Re: [PATCH 3/3] [gdb/tui] Allow second tui enable if first fails

[Tom de Vries]

On 5/1/26 9:06 PM, Tom Tromey wrote:
>>>>>> "Tom" == Tom de Vries <tdevries@suse.de> writes:
> 
> Tom> Commit fb23d7ba4a2 ("[gdb/tui] Handle error in tui_enable") handles a
> Tom> particular error during tui_enable, but doesn't allow trying to enable TUI
> Tom> again.
> 
> Tom> I tried to get this to work, but didn't manage because I didn't understand the
> Tom> interaction between endwin and the "tui_batch_rendering defer" destructor:
> Tom> - first endwin restores the shell terminal mode
> Tom> - then the defer destructor calls doupdate, restoring program shell mode.
> 
> This is probably just an oversight in tui_batch_rendering and perhaps
> the destructor should check some other flag as well before calling doupdate.

Thanks for the suggestion, I've submitted a v2 that checks for 
tui_active ( 
https://sourceware.org/pipermail/gdb-patches/2026-May/227293.html ).

Thanks,
- Tom