[PATCH] [gdb/tui] Handle error in tui_enable

[PATCH] [gdb/tui] Handle error in tui_enable

[Tom de Vries]

Say we simulate an error:
...
static void error_once () {
  static int v = 0;
  if (v == 1)
    return;

  v = 1;
  error (_("Oh no!!!"));
}
...
in the call to tui_set_initial_layout in tui_enable:
...
       tui_show_frame_info (deprecated_safe_get_selected_frame ());
+      error_once ();
       tui_set_initial_layout ();
...

After doing "tui enable"
...
$ gdb
(gdb) tui enable
...
the screen is cleared, and we have:
...
❌️ Oh no!!!
           (gdb) <blinking cursor>
...

After typing "apropos tui" (which is not echoed) and pressing enter, I run into
a segmentation violation here in tui_inject_newline_into_command_window:
...
    at /data/vries/gdb/leap-16-0/build/../../src/gdb/tui/tui-io.c:1084
1084	  WINDOW *w = tui_cmd_win ()->handle.get ();
...
because:
...
$2 = (tui_cmd_window *) 0x0
(gdb) p tui_cmd_win ()
...

The problem is that tui_active is true, and so
tui_inject_newline_into_command_window get called:
...
static void
tui_command_line_handler (gdb::unique_xmalloc_ptr<char> &&rl)
{
 ...
  if (tui_active)
    tui_inject_newline_into_command_window ();
 ...
}
...

Fix this by catching the error in tui_enable, and resetting tui_active back to
false.

While this fixes the segmentation fault, and does allow "apropos tui" to run,
still the command is not echoed, and the output of the command is garbled by
runaway indentation.

Fix this by using endwin / delscreen, as borrowed from earlier in tui_enable:
...
      if (cap == NULL || cap == (char *) -1 || *cap == '\0')
        {
          endwin ();
          delscreen (s);
          error (_("Cannot enable the TUI: "
                   "terminal doesn't support cursor addressing [TERM=%s]"),
                 gdb_getenv_term ());
        }
...

Note that this doesn't allow a second attempt:
...
$ gdb -q
(gdb) tui enable
❌️ Oh no!!!
(gdb) tui enable
❌️ Cannot enable the TUI
(gdb)
...
because tui_finish_init is stuck at TRIBOOL_UNKNOWN.

IWBN to fix this in a way that allows the second "tui enable" to succeed.  I
tried this for a bit, but didn't get it to work.

Tested on x86_64-linux.

Suggested-By: Tom Tromey <tom@tromey.com> [1]

Bug: https://sourceware.org/bugzilla/show_bug.cgi?id=34100

[1] https://sourceware.org/pipermail/gdb-patches/2025-May/217660.html
---
 gdb/tui/tui.c | 39 ++++++++++++++++++++++++---------------
 1 file changed, 24 insertions(+), 15 deletions(-)

diff --git a/gdb/tui/tui.c b/gdb/tui/tui.c
index 5e39eef71c6..9fef7feffe5 100644
--- a/gdb/tui/tui.c
+++ b/gdb/tui/tui.c
@@ -490,22 +490,31 @@ tui_enable (void)
 	 rely on this flag being true in order to know that the window
 	 they are creating is currently valid.  */
       tui_active = true;
+      try
+	{
+	  cbreak ();
+	  noecho ();
+	  /* timeout (1); */
+	  nodelay (w, FALSE);
+	  nl ();
+	  keypad (w, TRUE);
+	  tui_set_term_height_to (LINES);
+	  tui_set_term_width_to (COLS);
+	  def_prog_mode ();
+	  tui_show_frame_info (deprecated_safe_get_selected_frame ());
+	  tui_set_initial_layout ();
+	  tui_set_win_focus_to (tui_src_win ());
+	  keypad (tui_cmd_win ()->handle.get (), TRUE);
+	  wrefresh (tui_cmd_win ()->handle.get ());
+	}
+      catch (const gdb_exception &)
+	{
+	  endwin ();
+	  delscreen (s);
+	  tui_active = false;
+	  throw;
+	}
 
-      cbreak ();
-      noecho ();
-      /* timeout (1); */
-      nodelay(w, FALSE);
-      nl();
-      keypad (w, TRUE);
-      tui_set_term_height_to (LINES);
-      tui_set_term_width_to (COLS);
-      def_prog_mode ();
-
-      tui_show_frame_info (deprecated_safe_get_selected_frame ());
-      tui_set_initial_layout ();
-      tui_set_win_focus_to (tui_src_win ());
-      keypad (tui_cmd_win ()->handle.get (), TRUE);
-      wrefresh (tui_cmd_win ()->handle.get ());
       tui_finish_init = TRIBOOL_FALSE;
     }
   else

base-commit: 4308969628943757f972e766c70352c4828cb4d2
-- 
2.51.0

Re: [PATCH] [gdb/tui] Handle error in tui_enable

[Tom Tromey]

>>>>> "Tom" == Tom de Vries <tdevries@suse.de> writes:

Tom> Say we simulate an error:
Tom> ...

Is there a way that this can happen without simulation?

I'm asking because:

Tom> Note that this doesn't allow a second attempt:
Tom> ...
Tom> $ gdb -q
Tom> (gdb) tui enable
Tom> ❌️ Oh no!!!
Tom> (gdb) tui enable
Tom> ❌️ Cannot enable the TUI
Tom> (gdb)
Tom> ...
Tom> because tui_finish_init is stuck at TRIBOOL_UNKNOWN.

Tom> IWBN to fix this in a way that allows the second "tui enable" to succeed.  I
Tom> tried this for a bit, but didn't get it to work.

If this can occur due to some user problem (say a bad layout or bad
window implementation) then gdb ideally should be more robust somehow.

But OTOH if this is something that's really obscure, what you have seems
totally fine.

Anyway I think the patch itself is ok.
Approved-By: Tom Tromey <tom@tromey.com>

Tom

Re: [PATCH] [gdb/tui] Handle error in tui_enable

[Tom de Vries]

On 4/28/26 4:10 PM, Tom Tromey wrote:
> Tom> IWBN to fix this in a way that allows the second "tui enable" to succeed.  I
> Tom> tried this for a bit, but didn't get it to work.
> 
> If this can occur due to some user problem (say a bad layout or bad
> window implementation) then gdb ideally should be more robust somehow.

OK, I've got a theory why I had difficulty getting this to work.

The problem is probably that after endwin "Refreshing a window resumes 
program mode" and I suppose that got triggered by "tui_batch_rendering 
defer".

So using an explicit reset_shell_mode instead of endwin makes this work 
for me.

Thanks,
- Tom

Re: [PATCH] [gdb/tui] Handle error in tui_enable

[Tom de Vries]

On 4/28/26 11:42 PM, Tom de Vries wrote:
> On 4/28/26 4:10 PM, Tom Tromey wrote:
>> Tom> IWBN to fix this in a way that allows the second "tui enable" to 
>> succeed.  I
>> Tom> tried this for a bit, but didn't get it to work.
>>
>> If this can occur due to some user problem (say a bad layout or bad
>> window implementation) then gdb ideally should be more robust somehow.
> 
> OK, I've got a theory why I had difficulty getting this to work.
> 
> The problem is probably that after endwin "Refreshing a window resumes 
> program mode" and I suppose that got triggered by "tui_batch_rendering 
> defer".
> 
> So using an explicit reset_shell_mode instead of endwin makes this work 
> for me.
> 

I've submitted a patch series that fixes this ( 
https://sourceware.org/pipermail/gdb-patches/2026-April/226966.html ).

Thanks,
- Tom