[PATCH] gdb/tui: use init_extended_color where possible

[PATCH] gdb/tui: use init_extended_color where possible

[Andrew Burgess]

After commit:

  commit fbe7f20a0f098ca03913452b29f50f0dc8568f77
  Date:   Sat May 9 23:27:43 2026 +0200

    gdb/tui: fix unexpected reuse of color pairs

which converted GDB to use init_extended_pair where possible, I
realised we could also make use of init_extended_color.

The motivation for using init_extended_color is slightly less than
init_extended_pair.  Assuming the terminal supports it the standard
init_color API supports up to SHRT_MAX (32767) different colors,
switching to init_extended_color removes the SHRT_MAX limit on color
indices, allowing us to support the full range of COLORS.

But the cost of making this change is minimal, we already track the
color indices as an `int` within the global COLOR_MAP, so it's mostly
just a case of calling init_extended_color where needed.

We only use init_extended_color when both that function and
init_extended_pair is available.  The fallback to init_extended_pair
is init_pair, which expects the color indices to be shorts.  If we are
using the init_pair fallback then using init_extended_color is
pointless.

In reality init_extended_pair and init_extended_color were both added
in ncurses 6.1, so should both be available together.

There is one additional change in here.  Assuming that a terminal does
support more than SHRT_MAX colours, but for some reason GDB is
compiled with a version of the curses library that doesn't support
init_extended_color, then it is possible that in `get_color` the value
of NEXT could end up above SHRT_MAX, in which case the `init_color`
call will truncate the value of NEXT to a short and we will end up
redefining an earlier color index.  To avoid this unlikely case I've
added a compare against SHRT_MAX.

The init_extended_color path doesn't have this risk as COLORS is an
`int` and NEXT is passed as an `int` on this path so there is no risk
of truncation.
---
 gdb/config.in    |  3 +++
 gdb/configure    |  1 +
 gdb/configure.ac |  1 +
 gdb/tui/tui-io.c | 20 +++++++++++++++++---
 4 files changed, 22 insertions(+), 3 deletions(-)

diff --git a/gdb/config.in b/gdb/config.in
index 1ef5dc5c2f4..74745665d4d 100644
--- a/gdb/config.in
+++ b/gdb/config.in
@@ -250,6 +250,9 @@
 /* Define to 1 if you have the `iconvlist' function. */
 #undef HAVE_ICONVLIST
 
+/* Define to 1 if you have the `init_extended_color' function. */
+#undef HAVE_INIT_EXTENDED_COLOR
+
 /* Define to 1 if you have the `init_extended_pair' function. */
 #undef HAVE_INIT_EXTENDED_PAIR
 
diff --git a/gdb/configure b/gdb/configure
index 303d6ea011c..fe58ba47e73 100755
--- a/gdb/configure
+++ b/gdb/configure
@@ -30238,6 +30238,7 @@ for ac_func in  \
   getrlimit \
   getuid \
   iconvlist \
+  init_extended_color \
   init_extended_pair \
   libiconvlist \
   posix_madvise \
diff --git a/gdb/configure.ac b/gdb/configure.ac
index e55a733fba7..3d8847df9d8 100644
--- a/gdb/configure.ac
+++ b/gdb/configure.ac
@@ -1520,6 +1520,7 @@ AC_CHECK_FUNCS([ \
   getrlimit \
   getuid \
   iconvlist \
+  init_extended_color \
   init_extended_pair \
   libiconvlist \
   posix_madvise \
diff --git a/gdb/tui/tui-io.c b/gdb/tui/tui-io.c
index 896d00f44c8..1c430f2ea45 100644
--- a/gdb/tui/tui-io.c
+++ b/gdb/tui/tui-io.c
@@ -252,10 +252,24 @@ get_color (const ui_file_style::color &color, int *result)
 	  if (next >= COLORS)
 	    return false;
 	  rgb_color rgb = color.get_rgb ();
-	  /* We store RGB as 0..255, but curses wants 0..1000.  */
-	  if (init_color (next, rgb[0] * 1000 / 255, rgb[1] * 1000 / 255,
-			  rgb[2] * 1000 / 255) == ERR)
+	  /* We store RGB as 0..255, but curses wants 0..1000.  There's no
+	     point using init_extended_color if init_extended_pair is not
+	     available as the fallback init_pair can only handle short
+	     color IDs.  */
+#if defined HAVE_INIT_EXTENDED_COLOR && defined HAVE_INIT_EXTENDED_PAIR
+	  if (init_extended_color (next, rgb[0] * 1000 / 255,
+				   rgb[1] * 1000 / 255,
+				   rgb[2] * 1000 / 255) == ERR)
 	    return false;
+#else
+	  /* NEXT is an int, but is passed as a short.  If COLORS is
+	     more than SHRT_MAX then NEXT will be truncated and end up
+	     redefining a color entry that we don't expect.  */
+	  if (next > SHRT_MAX
+	      || init_color (next, rgb[0] * 1000 / 255, rgb[1] * 1000 / 255,
+			     rgb[2] * 1000 / 255) == ERR)
+	    return false;
+#endif
 	  color_map[color] = next;
 	  *result = next;
 	}

base-commit: fbe7f20a0f098ca03913452b29f50f0dc8568f77
-- 
2.25.4