[PATCH 11/13] gdb: handle hexadecimal escapes in C++ name parser

[Simon Marchi <simon.marchi@efficios.com>]

When non-printable character values are involved, clang produces
template names with character literals using hexadecimal escapes.  For
instance, this:

    template <unsigned char c> struct UChar {};
    UChar<5> u;

produces a type with name:

    UChar<(unsigned char)'\x05'>

The C++ name parser doesn't currently handle the hex escapes in the
character literals.  These names therefore fail to canonicalize and are
kept as-is in the index.  This spelling does not match the demangler's
spelling of the same type:

    UChar<(unsigned char)5>

One visible consequence of this is that GDB fails to find the runtime
type of polymorphic objects when such types are involved.  Given a
pointer to an object, GDB finds its runtime type like this:

 1. Read the vtable pointer stored in the object, and adjust it to get
    the address of the vtable.

 2. Find the minimal symbol at that address.

 3. Demangle it, giving

      vtable for UChar<(unsigned char)5>

    and extract the class name.

 4. Look up that class name in the debug info.

Step 4 fails, because the cooked index contains the non-canonical name

  UChar<(unsigned char)'\x05'>

GDB then gives up with:

  warning: RTTI symbol not found for class 'UChar<(unsigned char)5>'

Fix it by handling hexadecimal escapes (\x followed by one or more hex
digits) in cp_parse_escape.  Make cp_parse_escape return an optional,
where an empty optional means "error".

For testing, add:

 - some canonicalization selftests

 - some parser selftests, checking that \x without hex digits doesn't
   parse

 - gdb.dwarf2/nontype-template-lookup.exp, which uses the DWARF
   assembler to create types named like clang names them, and looks
   them up using the demangler's spelling.  Following patches add more
   cases to it.

 - gdb.cp/nontype-template-rtti.exp, which exercises the RTTI case with
   a real compiler.  It uses a single class template,
   "template <typename T, T N> struct S", that following patches
   instantiate with more template arguments.  The test checks the value
   of a member initialized from N, to make sure that GDB resolves each
   object to the right instantiation.  It fails with gcc, which names
   the types "S<unsigned char, 200>" and "S<signed char, 5>" (missing
   the casts), so it is marked as a known failure with gcc.

Change-Id: I3183ec87db182b1c64fa69d909febef6221272b6
---
 gdb/cp-name-parser.c                          | 50 ++++++++++++-
 gdb/testsuite/gdb.cp/nontype-template-rtti.cc | 36 ++++++++++
 .../gdb.cp/nontype-template-rtti.exp          | 72 +++++++++++++++++++
 .../gdb.dwarf2/nontype-template-lookup.exp    | 61 ++++++++++++++++
 4 files changed, 216 insertions(+), 3 deletions(-)
 create mode 100644 gdb/testsuite/gdb.cp/nontype-template-rtti.cc
 create mode 100644 gdb/testsuite/gdb.cp/nontype-template-rtti.exp
 create mode 100644 gdb/testsuite/gdb.dwarf2/nontype-template-lookup.exp

diff --git a/gdb/cp-name-parser.c b/gdb/cp-name-parser.c
index 38ab258a3cfc..b8007187b35b 100644
--- a/gdb/cp-name-parser.c
+++ b/gdb/cp-name-parser.c
@@ -478,9 +478,11 @@ c_parse_backslash (int host_char, int *target_char)
    string pointer pointing at the null character.
 
    If \ is followed by 000, we return 0 and leave the string pointer
-   after the zeros.  A value of 0 does not mean end of string.  */
+   after the zeros.  A value of 0 does not mean end of string.
 
-static int
+   If the escape sequence is invalid, return an empty optional.  */
+
+static std::optional<int>
 cp_parse_escape (const char **string_ptr)
 {
   int target_char;
@@ -497,6 +499,27 @@ cp_parse_escape (const char **string_ptr)
   else
     switch (c)
       {
+      case 'x':
+	{
+	  /* Hexadecimal escape (\x followed by one or more hex digits).
+
+	     An escape with more than 8 digits will silently wrap around,
+	     that's ok.  */
+	  unsigned int value = 0;
+	  int count = 0;
+
+	  while (c_isxdigit (**string_ptr))
+	    {
+	      value = value * 16 + fromhex (*(*string_ptr)++);
+	      count++;
+	    }
+
+	  if (count == 0)
+	    return {};
+
+	  return value;
+	}
+
       case '0':
       case '1':
       case '2':
@@ -552,7 +575,16 @@ lex_char_literal (cp_name_parser_YYSTYPE *lvalp,
 
   int c = *state->lexptr++;
   if (c == '\\')
-    c = cp_parse_escape (&state->lexptr);
+    {
+      std::optional<int> escaped = cp_parse_escape (&state->lexptr);
+      if (!escaped.has_value ())
+	{
+	  cpname_yyerror (state, _("invalid escape sequence"));
+	  return ERROR;
+	}
+
+      c = *escaped;
+    }
   else if (c == '\'')
     {
       cpname_yyerror (state, _("empty character constant"));
@@ -1080,6 +1112,13 @@ canonicalize_tests ()
 
   should_canonicalize_to ("x::y::z<0xff'ff>", "x::y::z<65535>");
 
+  /* Hexadecimal escapes.  */
+  should_canonicalize_to ("C<'\\xa'>", "C<(char)10>");
+  should_canonicalize_to ("C<(char) 10>", "C<(char)10>");
+  should_canonicalize_to ("C<(unsigned char)'\\xC8'>", "C<(unsigned char)200>");
+  should_canonicalize_to ("C<(unsigned char) 200>", "C<(unsigned char)200>");
+  should_canonicalize_to ("C<(signed char)'\\x05'>", "C<(signed char)5>");
+
   should_canonicalize_to ("something<  void()  >", "something<void()>");
   should_canonicalize_to ("something<void (void)>", "something<void()>");
 
@@ -1107,6 +1146,11 @@ parser_tests ()
   should_not_parse ("C<'", "unterminated character constant");
   should_not_parse ("C<'a", "unterminated character constant");
   should_not_parse ("C<'\\", "unterminated character constant");
+
+  should_not_parse ("C<'\\x'>", "invalid escape sequence");
+  should_not_parse ("C<'\\x", "invalid escape sequence");
+  should_not_parse ("C<'\\xg'>", "invalid escape sequence");
+  should_not_parse ("C<'\\xabg'>", "invalid character constant");
 }
 
 #endif
diff --git a/gdb/testsuite/gdb.cp/nontype-template-rtti.cc b/gdb/testsuite/gdb.cp/nontype-template-rtti.cc
new file mode 100644
index 000000000000..c12203fbee39
--- /dev/null
+++ b/gdb/testsuite/gdb.cp/nontype-template-rtti.cc
@@ -0,0 +1,36 @@
+/* This testcase is part of GDB, the GNU debugger.
+
+   Copyright 2026 Free Software Foundation, Inc.
+
+   This program is free software; you can redistribute it and/or modify
+   it under the terms of the GNU General Public License as published by
+   the Free Software Foundation; either version 3 of the License, or
+   (at your option) any later version.
+
+   This program is distributed in the hope that it will be useful,
+   but WITHOUT ANY WARRANTY; without even the implied warranty of
+   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+   GNU General Public License for more details.
+
+   You should have received a copy of the GNU General Public License
+   along with this program.  If not, see <http://www.gnu.org/licenses/>.  */
+
+struct base
+{
+  virtual ~base () {}
+};
+
+template <typename T, T N>
+struct S : base
+{
+  T value = N;
+};
+
+base *uchar_200 = new S<unsigned char, 200>;
+base *schar_5 = new S<signed char, 5>;
+
+int
+main ()
+{
+  return 0;	/* break-here */
+}
diff --git a/gdb/testsuite/gdb.cp/nontype-template-rtti.exp b/gdb/testsuite/gdb.cp/nontype-template-rtti.exp
new file mode 100644
index 000000000000..b67736d37697
--- /dev/null
+++ b/gdb/testsuite/gdb.cp/nontype-template-rtti.exp
@@ -0,0 +1,72 @@
+# Copyright 2026 Free Software Foundation, Inc.
+
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+# Check that GDB can resolve the dynamic (RTTI) type of a polymorphic
+# object whose class is a template instantiation with a non-type
+# template argument.  GDB finds the dynamic type by demangling the
+# object's vtable symbol and looking the name up.  Producers spell
+# non-type template arguments in DW_AT_name differently from the
+# demangler, which can make the lookup fail.  We need to make sure
+# that GDB can match those up.
+
+require allow_cplus_tests
+
+standard_testfile .cc
+
+if {[prepare_for_testing "failed to prepare" $testfile $srcfile {debug c++}]} {
+    return
+}
+
+if {![runto [gdb_get_line_number "break-here"]]} {
+    return
+}
+
+# Enable dynamic type resolution.
+gdb_test_no_output "set print object on"
+
+# clang 21 and later emit an artificial static member describing the
+# vtable.  Disable printing static members so that the output is easier
+# to match.
+gdb_test_no_output "set print static-members off"
+
+# The objects to check, with the type of their template argument and the
+# expected value of their "value" member.
+set objects {
+    {uchar_200 "unsigned char" 0xc8}
+    {schar_5 "signed char" 0x5}
+}
+
+foreach obj $objects {
+    lassign $obj var type value
+
+    # gcc spells the template arguments of these objects in a way GDB
+    # doesn't match yet.
+    if {[test_compiler_info gcc-* c++] && $var in {uchar_200 schar_5}} {
+	setup_kfail "gdb/19320" *-*-*
+    }
+
+    # If GDB successfully determines the runtime type of the object, it
+    # prints it, as well as the "<base>" subobject.  The spelling of the
+    # value template argument can vary based on the compiler, so we
+    # don't match it, but the type template argument does not, so we
+    # match it.
+    #
+    # The value of the "value" member tells us that GDB found the right
+    # instantiation, in case there are multiple for the same type.
+    # Print in hexadecimal, so that the output is in a predictable form.
+    set type_re "S<[string_to_regexp $type], \[^>]+>"
+    gdb_test "print/x *$var" \
+	"= \\($type_re\\) \\{<base> = \\{.*\\}, value = $value\\}"
+}
diff --git a/gdb/testsuite/gdb.dwarf2/nontype-template-lookup.exp b/gdb/testsuite/gdb.dwarf2/nontype-template-lookup.exp
new file mode 100644
index 000000000000..464766c7c3fb
--- /dev/null
+++ b/gdb/testsuite/gdb.dwarf2/nontype-template-lookup.exp
@@ -0,0 +1,61 @@
+# Copyright 2026 Free Software Foundation, Inc.
+
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+# Check that GDB can look up C++ types whose names have non-type template
+# arguments.  Producers emit those types with different DW_AT_names, this
+# test verifies that GDB is able to parse and canonicalize them.
+
+load_lib dwarf.exp
+
+require dwarf2_support allow_cplus_tests
+
+standard_testfile main.c -dw.S
+
+set asm_file [standard_output_file $srcfile2]
+Dwarf::assemble $asm_file {
+    cu {} {
+	compile_unit {
+	    DW_AT_language @DW_LANG_C_plus_plus
+	    DW_AT_name "nontype-template-lookup.c"
+	} {
+	    # This is how clang encodes an unsigned char value.
+	    structure_type {
+		DW_AT_name {UCharHexEscape<(unsigned char)'\\xc8'>}
+		DW_AT_byte_size 1 DW_FORM_sdata
+	    }
+
+	    # This is how clang encodes a signed char value.
+	    structure_type {
+		DW_AT_name {SCharHexEscape<(signed char)'\\x05'>}
+		DW_AT_byte_size 1 DW_FORM_sdata
+	    }
+	}
+    }
+}
+
+if { [prepare_for_testing "failed to prepare" $testfile \
+	  [list $srcfile $asm_file] {nodebug}] } {
+    return
+}
+
+# Look up each type using the spelling of the demangler, which is the
+# canonical spelling.  GDB also prints the type name using that spelling.
+foreach name [list \
+    {UCharHexEscape<(unsigned char)200>} \
+    {SCharHexEscape<(signed char)5>} \
+] {
+    gdb_test "ptype $name" \
+	"type = struct [string_to_regexp $name] \{.*\}"
+}
-- 
2.56.0