[PATCH v2 0/3] [gdb/tui] Allow second tui enable if first fails

[PATCH v2 0/3] [gdb/tui] Allow second tui enable if first fails

[Tom de Vries]

Commit fb23d7ba4a2 ("[gdb/tui] Handle error in tui_enable") handles a
particular error during tui_enable, but doesn't allow trying to enable TUI
again.

This patch series fixes that.

The first two patches do some refactoring.

The third patch contains the fix.

Changes in v2:
- change fix approach: check tui_active in ~tui_batch_rendering, instead of
  replacing endwin with reset_shell_mode

Versions:
- v1: https://sourceware.org/pipermail/gdb-patches/2026-April/226966.html

Tom de Vries (3):
  [gdb/tui] Factor out require_tui_terminal
  [gdb/tui] Factor out init_ncurses
  [gdb/tui] Allow second tui enable if first fails

 gdb/tui/tui-wingeneral.c |  2 +-
 gdb/tui/tui.c            | 95 +++++++++++++++++++++++++---------------
 2 files changed, 61 insertions(+), 36 deletions(-)


base-commit: a8f34ab5485c9a16e7c8853eeacaca5dfb946461
-- 
2.51.0

[PATCH v2 1/3] [gdb/tui] Factor out require_tui_terminal

[Tom de Vries]

I noticed this bit of code in tui_enable:
...
      /* Check required terminal capabilities.  The MinGW port of
	 ncurses does have them, but doesn't expose them through "cup".  */
      cap = tigetstr ((char *) "cup");
      if (cap == NULL || cap == (char *) -1 || *cap == '\0')
	{
	  endwin ();
	  delscreen (s);
	  error (_("Cannot enable the TUI: "
		   "terminal doesn't support cursor addressing [TERM=%s]"),
		 gdb_getenv_term ());
	}
...

At this point in tui_enable, we need endwin and delscreen to clean up after
newterm, but the check can be done before newterm.

Fix this by factoring out a new function require_tui_terminal, and moving the
code there.
---
 gdb/tui/tui.c | 45 ++++++++++++++++++++++++---------------------
 1 file changed, 24 insertions(+), 21 deletions(-)

diff --git a/gdb/tui/tui.c b/gdb/tui/tui.c
index de2faa36af7..c1457e148a2 100644
--- a/gdb/tui/tui.c
+++ b/gdb/tui/tui.c
@@ -406,6 +406,28 @@ require_tui_interpreter ()
 	   interp);
 }
 
+/* Error out if the terminal doesn't support TUI.  */
+
+static void
+require_tui_terminal ()
+{
+  /* Don't try to setup curses (and print funny control
+     characters) if we're not outputting to a terminal.  */
+  if (!gdb_stderr->isatty ())
+    error (_("Cannot enable the TUI when output is not a terminal"));
+
+  /* Check required terminal capabilities.  The MinGW port of
+     ncurses does have them, but doesn't expose them through "cup".  */
+#ifndef __MINGW32__
+  const char *cap = tigetstr ((char *) "cup");
+  const char *not_a_string_capability = (char *) -1;
+  if (cap == nullptr || cap == not_a_string_capability || *cap == '\0')
+    error (_("Cannot enable the TUI: "
+	     "terminal doesn't support cursor addressing [TERM=%s]"),
+	   gdb_getenv_term ());
+#endif
+}
+
 /* Enter in the tui mode (curses).
    When in normal mode, it installs the tui hooks in gdb, redirects
    the gdb output, configures the readline to work in tui mode.
@@ -434,18 +456,13 @@ tui_enable (void)
     {
       WINDOW *w;
       SCREEN *s;
-#ifndef __MINGW32__
-       const char *cap;
-#endif
 
       /* If the top level interpreter is not the console/tui (e.g.,
 	 MI), enabling curses will certainly lose.  */
       require_tui_interpreter ();
 
-      /* Don't try to setup curses (and print funny control
-	 characters) if we're not outputting to a terminal.  */
-      if (!gdb_stderr->isatty ())
-	error (_("Cannot enable the TUI when output is not a terminal"));
+      /* Require a terminal that supports TUI.  */
+      require_tui_terminal ();
 
       /* Don't try initialization again.  */
       tui_finish_init = TRIBOOL_UNKNOWN;
@@ -473,20 +490,6 @@ tui_enable (void)
 	  start_color ();
 	}
 
-      /* Check required terminal capabilities.  The MinGW port of
-	 ncurses does have them, but doesn't expose them through "cup".  */
-#ifndef __MINGW32__
-      cap = tigetstr ((char *) "cup");
-      if (cap == NULL || cap == (char *) -1 || *cap == '\0')
-	{
-	  endwin ();
-	  delscreen (s);
-	  error (_("Cannot enable the TUI: "
-		   "terminal doesn't support cursor addressing [TERM=%s]"),
-		 gdb_getenv_term ());
-	}
-#endif
-
       /* We must mark the tui sub-system active before trying to setup the
 	 current layout as tui windows defined by an extension language
 	 rely on this flag being true in order to know that the window
-- 
2.51.0

[PATCH v2 2/3] [gdb/tui] Factor out init_ncurses

[Tom de Vries]

Factor out init_ncurses out of tui_enable, and make sure calling it twice is
harmless.
---
 gdb/tui/tui.c | 43 ++++++++++++++++++++++++++++++-------------
 1 file changed, 30 insertions(+), 13 deletions(-)

diff --git a/gdb/tui/tui.c b/gdb/tui/tui.c
index c1457e148a2..11bcba813bc 100644
--- a/gdb/tui/tui.c
+++ b/gdb/tui/tui.c
@@ -428,6 +428,34 @@ require_tui_terminal ()
 #endif
 }
 
+/* Initialize ncurses, if necessary.  */
+
+static SCREEN *
+init_ncurses ()
+{
+  static SCREEN *tui_screen = nullptr;
+  if (tui_screen != nullptr)
+    {
+      /* Init ncurses only once.  */
+      return tui_screen;
+    }
+
+  tui_screen = newterm (nullptr, stdout, stdin);
+
+#ifdef __MINGW32__
+  /* The MinGW port of ncurses requires $TERM to be unset in order
+     to activate the Windows console driver.  */
+  if (tui_screen == nullptr)
+    tui_screen = newterm ((char *) "unknown", stdout, stdin);
+#endif
+
+  if (tui_screen == nullptr)
+    error (_("Cannot enable the TUI: error opening terminal [TERM=%s]"),
+	   gdb_getenv_term ());
+
+  return tui_screen;
+}
+
 /* Enter in the tui mode (curses).
    When in normal mode, it installs the tui hooks in gdb, redirects
    the gdb output, configures the readline to work in tui mode.
@@ -455,7 +483,6 @@ tui_enable (void)
   if (tui_finish_init == TRIBOOL_TRUE)
     {
       WINDOW *w;
-      SCREEN *s;
 
       /* If the top level interpreter is not the console/tui (e.g.,
 	 MI), enabling curses will certainly lose.  */
@@ -467,19 +494,9 @@ tui_enable (void)
       /* Don't try initialization again.  */
       tui_finish_init = TRIBOOL_UNKNOWN;
 
-      s = newterm (NULL, stdout, stdin);
-#ifdef __MINGW32__
-      /* The MinGW port of ncurses requires $TERM to be unset in order
-	 to activate the Windows console driver.  */
-      if (s == NULL)
-	s = newterm ((char *) "unknown", stdout, stdin);
-#endif
-      if (s == NULL)
-	{
-	  error (_("Cannot enable the TUI: error opening terminal [TERM=%s]"),
-		 gdb_getenv_term ());
-	}
+      SCREEN *s = init_ncurses ();
       w = stdscr;
+
       if (has_colors ())
 	{
 #ifdef HAVE_USE_DEFAULT_COLORS
-- 
2.51.0

[PATCH v2 3/3] [gdb/tui] Allow second tui enable if first fails

[Tom de Vries]

Commit fb23d7ba4a2 ("[gdb/tui] Handle error in tui_enable") handles a
particular error during tui_enable, but doesn't allow trying to enable TUI
again.

I tried to get this to work, but didn't manage because I didn't understand the
interaction between endwin and the "tui_batch_rendering defer" destructor:
- first endwin restores the shell terminal mode
- then the defer destructor calls doupdate, restoring program terminal mode.

Fix this by checking for tui_active in ~tui_batch_rendering.
---
 gdb/tui/tui-wingeneral.c | 2 +-
 gdb/tui/tui.c            | 9 +++++++--
 2 files changed, 8 insertions(+), 3 deletions(-)

diff --git a/gdb/tui/tui-wingeneral.c b/gdb/tui/tui-wingeneral.c
index 86f13d275b4..149ca4efc6e 100644
--- a/gdb/tui/tui-wingeneral.c
+++ b/gdb/tui/tui-wingeneral.c
@@ -46,7 +46,7 @@ tui_batch_rendering::tui_batch_rendering ()
 tui_batch_rendering::~tui_batch_rendering ()
 {
   suppress_output = m_saved_suppress;
-  if (!suppress_output)
+  if (!suppress_output && tui_active)
     doupdate ();
 }
 
diff --git a/gdb/tui/tui.c b/gdb/tui/tui.c
index 11bcba813bc..958fc47b7b6 100644
--- a/gdb/tui/tui.c
+++ b/gdb/tui/tui.c
@@ -494,7 +494,7 @@ tui_enable (void)
       /* Don't try initialization again.  */
       tui_finish_init = TRIBOOL_UNKNOWN;
 
-      SCREEN *s = init_ncurses ();
+      init_ncurses ();
       w = stdscr;
 
       if (has_colors ())
@@ -532,8 +532,13 @@ tui_enable (void)
       catch (const gdb_exception &)
 	{
 	  endwin ();
-	  delscreen (s);
+
+	  /* Initialization failed, so TUI is not active.  */
 	  tui_active = false;
+
+	  /* Allow trying to initialize TUI again.  */
+	  tui_finish_init = TRIBOOL_TRUE;
+
 	  throw;
 	}
 
-- 
2.51.0

Re: [PATCH v2 0/3] [gdb/tui] Allow second tui enable if first fails

[Tom Tromey]

>>>>> "Tom" == Tom de Vries <tdevries@suse.de> writes:

Tom> Changes in v2:
Tom> - change fix approach: check tui_active in ~tui_batch_rendering, instead of
Tom>   replacing endwin with reset_shell_mode

Looks good to me, thanks.
Approved-By: Tom Tromey <tom@tromey.com>

Tom