Re: [RFC] Allowing GDB to use a more recent version of Python at runtime than it was compiled with

[Andrew Burgess via Gdb <gdb@sourceware.org>]

Tom Tromey <tom@tromey.com> writes:

>>>>>> "Matthieu" == Matthieu Longo <matthieu.longo@arm.com> writes:
>
> Matthieu> ## End goal
> Matthieu> Allowing GDB to use a more recent version of Python at runtime than it
> Matthieu> was compiled with.
>
> FWIW I've looked at this several times over the years.  See:
>
> https://sourceware.org/bugzilla/show_bug.cgi?id=23830
>
> Matthieu> I got errors about Py_buffer and PyBuffer_Release (part of the Stable
> Matthieu> ABI (including all members) since version 3.11, so changed
> Matthieu> Py_LIMITED_API to 0x030b0000):
> Matthieu> https://docs.python.org/3/c-api/buffer.html#c.Py_buffer
>
> This is new since the last time I looked.
>
> Note that gdb still supports a very old Python by default -- like 3.4 I
> think.  This is bad but it is hard to change.
>
> Matthieu> There might be more issues as the build process stopped and didn't go
> Matthieu> over all the files in gdb/python.
>
> Yeah, there's the readline stuff mentioned in the bug.  Maybe this could
> be disabled.  One idea might be to have a configure flag requesting the
> stable API and then just drop this module in this case.

The specific issue mentioned in bug 23830:

  ../../binutils-gdb/gdb/python/py-gdb-readline.c:58:29: error: ‘_PyOS_ReadlineTState’ was not declared in this scope

Was fixed by this commit:

  commit 8170efad364aa8e062cba3b722b81aca9eda8cf5
  Author: Andrew Burgess <aburgess@redhat.com>
  Date:   Wed Dec 6 16:57:11 2023 +0100
  
      gdb/python: avoid use of _PyOS_ReadlineTState

You're welcome :)

I tried compiling py-gdb-readline with Py_LIMITED_API defined.  With the
old (3.7) version of Python I have installed on my machine I'm seeing
problems with the following things:

  Py_buffer type -- added to the stable API in 3.11
  PyBuffer_Release function -- likewise added in 3.11
  PyMem_RawMalloc function -- added to stable API in 3.13
  PyRun_SimpleString function -- This ones harder.

So, just for the readline module, the real issue is the
PyRun_SimpleString function.  At a quick glance, I would guess we need
to rewrite this using primitives that are part of the stable API.

>
> This idea might 'interface' with the 3.4 requirement above.  Like, we
> can have a fallback mode in the source for people building against older
> versions of Python.
>
> Matthieu> Making the usage of PyTypeObject opaque would consist in transforming
> Matthieu> the declaration of those PyTypeObjects to a static PyType_Slot and
> Matthieu> PyType_Spec equivalent, and then creating a PyObject* by calling
> Matthieu> PyType_FromSpec() instead of PyModule_AddObject(), which is, by the
> Matthieu> way, "soft deprecated" since Python 3.13.
>
> Yes, converting to PyType_FromSpec would be fine.

Agreed, and also this is likely the lowest risk change as it's going to
be pretty formulaic, and I would expect any issues to show up pretty quickly.

>
> Matthieu> Following the logic of PEP-384, linking against the Python limited C
> Matthieu> API means linking against libpython3.so, not libpython3.x.so
>
> Matthieu> With in the current state of Python packaging, this creates issues at
> Matthieu> different stages:
>
> I don't know what to do about this one.  But I guess one idea would be
> that gdb could adopt the changes that make sense, and then you could
> handle the build-time madness for your own builds.  Then this could be
> improved in gdb as the upstream Python/distro situation develops.

Based on what I wrote above, with some things we use not being part of
the stable API until 3.13, and given our desire to support older
versions of Python.  I think even the Py_LIMITED_API define would want
(at least initially) to be optional.

So we can adjust the code as much as possible (without raising the
current minimum version) to only use stable API calls, that's fine to do
for everyone.

Then we can have a configure switch to say "enforce stable API", which
would define Py_LIMITED_API, and (eventually) would change which Python
library we link against 3.x.so vs 3.so.

>
> Matthieu> - testing. I have had a look at the test coverage of the Python GDB
> Matthieu>   API, and it seems ok at a first glance. Please let me know if you
> Matthieu>   can see any difficulty with the current coverage.
>
> How did you test coverage?
>
> I used to routinely to --coverage builds and test these, but they
> haven't been reliable for me for a couple of years now :-(
>
> Matthieu> - any performance impact caused by this transition that I should be
> Matthieu>   aware of. And how did you measure it ? How critical are performances
> Matthieu>   regarding the Python limited C API ?
>
> I would not worry about this at all.

Agreed.

Thanks,
Andrew