[PATCH v5 0/1] gdb: Add python support for demangling register unwind values

[Craig Blackmore <craig.blackmore@embecosm.com>]

This version addresses review comments from Andrew Burgess and Eli
Zaretskii from:

  https://sourceware.org/pipermail/gdb-patches/2026-May/227549.html
  https://sourceware.org/pipermail/gdb-patches/2026-May/227536.html

Changes since v4:

* Change `demangle` method to `read`.

* Refer to transforming instead of demangling in the code and
  documentation.

* Fix long line in python.texi.

* Rebase.

Changes since v3:

* Change lval_type to lvalue type in NEWS and python documentation.

* Remove line about future support for multiple transformers or
  different loci from python documentation.

Changes since v2:

* Removed RFC from subject.

* Changed `gdb.unwinder.UnwindRegisterValueDemangler` to
  `gdb.unwinder.UnwindRegisterValueTransformer` so that other transform
  methods could be supported in future rather than being restricted to
  demangling only.  Demangling is now done by calling the `demangle`
  method instead of `__call__`.

* Made transformer registration similar to unwinder registration to
  allow future support for registering multiple transformers and
  registering to other loci.

* Call `invalidate_cached_frames` after registering a transformer.

* Made `gdb.unwinder.UnwindRegisterValueTransformer.name` read-only
  and added type check.

* Pass `gdb.Frame` to demanglers instead of `frame_type`.

* Added global `python_transformer_enabled` to `py-unwind.c` which
  python sets when any transformer is registered.  This allows us to
  skip entering the python environment when demangling and no
  transformers are registered.

* Replaced `gdb.Value.is_lval_register` and `gdb.Value.is_lval_memory`
  read-only boolean attributes with `gdb.Value.lval_type` read-only
  type attribute.

* Removed redundant `return` after call to `error`.

* Replaced `NULL` with `nullptr`.

* Added tests for invalid registration and invalid return value from
  demangler.

* Updated tests and documentation to match the above changes.

* Rebased.

Craig Blackmore (1):
  gdb: Add python support for transforming register unwind values

 gdb/NEWS                                      |  17 ++
 gdb/doc/python.texi                           |  96 ++++++++++
 gdb/extension-priv.h                          |   8 +
 gdb/extension.c                               |  35 ++++
 gdb/extension.h                               |   6 +
 gdb/frame.c                                   |  10 ++
 gdb/guile/guile.c                             |   2 +
 gdb/python/lib/gdb/__init__.py                |  21 +++
 gdb/python/lib/gdb/unwinder.py                |  93 +++++++++-
 gdb/python/py-registers.c                     |   2 +-
 gdb/python/py-unwind.c                        | 169 ++++++++++++++++++
 gdb/python/py-value.c                         |  21 +++
 gdb/python/python-internal.h                  |   3 +
 gdb/python/python.c                           |   4 +-
 .../gdb.python/py-unwind-transformer-error.py |  33 ++++
 .../gdb.python/py-unwind-transformer.c        |  80 +++++++++
 .../gdb.python/py-unwind-transformer.exp      | 115 ++++++++++++
 .../gdb.python/py-unwind-transformer.py       |  48 +++++
 gdb/testsuite/gdb.python/py-value.exp         |   8 +
 19 files changed, 768 insertions(+), 3 deletions(-)
 create mode 100644 gdb/testsuite/gdb.python/py-unwind-transformer-error.py
 create mode 100644 gdb/testsuite/gdb.python/py-unwind-transformer.c
 create mode 100644 gdb/testsuite/gdb.python/py-unwind-transformer.exp
 create mode 100644 gdb/testsuite/gdb.python/py-unwind-transformer.py

-- 
2.43.0