Re: Finding GDB's Python Libraries

[Joel Brobecker <brobecker@adacore.com>]

[-- Attachment #1: Type: text/plain, Size: 7052 bytes --]

Hi Andrew,

On Mon, Jul 06, 2020 at 06:33:46PM +0100, Andrew Burgess wrote:
> I have been thinking about how we tell GDB where to find the Python
> libraries, and I'm currently going around in circles trying to figure
> out what the right thing to do is.
> 
> Some background reading, some original work to tell GDB about where to
> find Python libraries:
> 
>   https://sourceware.org/legacy-ml/gdb-patches/2010-05/msg00434.html
> 
> This allowed GDB to figure out where the Python it was configured with
> was located, and from that Python can figure out where its libraries are.
> 
> Then there was this:
> 
>   https://sourceware.org/pipermail/gdb-patches/2020-February/165389.html
> 
> which built on the above, but was more flexible in some cross
> compiling situations, not requiring Python to actually be installed
> into the GDB install tree (just the libraries can be installed).
> 
> I recently ran into a situation where I needed to cross-build GDB for
> a MinGW target, including Python support.
> 
> By writing a support script as described here:
> 
>   https://sourceware.org/gdb/wiki/CrossCompilingWithPythonSupport
> 
> I was able to successfully build a MinGW GDB on Linux.  I configured
> this GDB something like this:
> 
>   /project/gdb/src/configure --target=.....  --host=i686-w64-mingw32 \
>                              --prefix=/project/gdb/install
>                              --with-python=/project/gdb/x-compile-script.sh
>                              --with-python-libdir=/project/gdb/install/lib/
> 
> After building and installing GDB I then copied the Python libraries
> and the Python DLL from an unpacked i686-w64-mingw32 Python package
> into the GDB install directory. With this done I was able to zip the
> GDB install directory and distribute it as needed, it all worked fine,
> except....
> 
> ... I ran into a case where a user had PYTHONHOME set in their
> environment, and the Python libraries that this path pointed to were
> not compatible with the MinGW Python version that GDB was linking
> against.
> 
> The motivation for originally setting PYTHONHOME in the environment
> have been lost to the mists of time, but this started me thinking.
> 
> If we have configured GDB to use a specific set of Python libraries,
> even going so far (in this case) to ship the libraries along with GDB,
> should we allow PYTHONHOME to override that library selection?
> 
> My initial thought was sure, that makes sense, that's what PYTHONHOME
> is for.  But, what if the user really needed PYTHONHOME to be set in
> order to correctly use their installed (nothing to do with GDB) Python
> interpreter?  In that case they now have a situation, through no fault
> of their own, where GDB doesn't "just work".
> 
> 
> A quick digression on --with-python and --with-python-libdir:
> Currently, even when the --with-python-libdir flag is not passed at
> configure time, if configure chooses to use Python, then a setting for
> this flag is created within the configure script.  As far as GDB is
> concerned it is impossible to tell if the user specified
> --with-python-libdir, or if configure filled this in for us.  I think
> this might be important shortly.
> 
> 
> Currently within GDB we call Py_SetProgramName.  This call tells the
> Python library where the Python executable is, and from this path the
> Python library figures out where the libraries are.  Funnily the value
> we pass to Py_SetProgramName is actually built within GDB based on the
> location where the libraries are.  So,
> 
>   --with-python-libdir=/usr/lib
> 
> Causes us to call:
> 
>   Py_SetProgramName ("/usr/bin/python")
> 
> >From which the Python library figures out that the libraries are
> located at:
> 
>   /usr/lib
> 
> It doesn't actually matter to the Python library if there isn't a
> python at /usr/bin/python, so long as the libraries are where it
> expects them all is good.  This is why we can do:
> 
>   --with-python=/usr/bin/python
>   --with-python-libdir=/project/gdb/install/lib
> 
> Even if there is never a /project/gdb/install/bin/python, our the
> Python library will still find the libraries correctly.
> 
> Setting the program name in this way still allows for PYTHONHOME to
> override the location of the libraries.  However, we could switch to
> using Py_SetPythonHome, which also tells the Python library where to
> finds its library files, but doesn't honour PYTHONHOME.
> 
> In some cases using Py_SetPythonHome seems like a better choice, but
> in other cases not listening to PYTHONHOME feels like a mistake, and
> Py_SetProgramName feels like a better choice.  The problem I'm
> struggling with is what would be a good set of rules for choosing
> between these two options?
> 
> I initially thought that maybe if the user has specified a
> --with-python-libdir that is under the install prefix of GDB then we
> should not honour PYTHONPATH (so use Py_SetPythonHome), however, that
> doesn't feel strong enough.
> 
> What if the user does:
> 
>   --with-python=/opt/my-special-python/bin/python
>   --with-python-libdir=/opt/my-special-python/lib
> 
> Should they not expect GDB to pick this up in preference to any other
> Python?
> 
> So then I wondered if we could detect the case where configure has
> found the "default" Python on the machine, so, if the user configures
> like:
> 
>   --with-python=yes
> 
> Then they will get whatever python configure can find, maybe in this
> case, and this case only we should listen to PYTHONHOME, but if the
> user has specifically specified a particular version of python in any
> location, then we should force GDB to use that python above all
> others?
> 
> If we did go down this route (or make any use of Py_SetPythonHome)
> then I would suggest we have GDB look for a new environment variable
> GDB_PYTHONHOME, which would be just like PYTHONHOME, but only for GDB,
> and would override the value from --with-python-libdir.
> 
> I'd be interested to hear if anyone has any thoughts on this issue.

At AdaCore, we ship a version that overrides any value that PYTHONHOME
might have, for exactly the reason that you explained. You hit that issue
on Windows, but from our perspective, this isn't just OS-specific either.

I found the following reference to a discussion where we proposed
the approach:
https://sourceware.org/pipermail/gdb-patches/2010-November/078220.html
https://sourceware.org/pipermail/gdb-patches/2010-December/078618.html
https://sourceware.org/pipermail/gdb-patches/2011-January/079287.html

It looks like the discussed died because I gave up due to lack of
support, that Doug clarified by saying we should use honor
GDB_PYTHONHOME and disregard PYTHONHOME instead. My answer was
not very clear, but it sounds like I was thinking "someone else"
can take care of it (I'm wondering how I could have thought that,
but anyways).

Attached is a local patch that we have internally at AdaCore.
I think you could build on it to add support for GDB_PYTHONHOME
and submit it, if others agree that this is going in the right
direction.

-- 
Joel

[-- Attachment #2: python-home-adacore.diff --]
[-- Type: text/x-diff, Size: 2088 bytes --]

diff --git a/gdb/python/python.c b/gdb/python/python.c
index 67f362b852d..f5ace2ff978 100644
--- a/gdb/python/python.c
+++ b/gdb/python/python.c
@@ -1644,6 +1644,8 @@ do_start_initialization ()
      program_name, and respect the requirement of Py_SetProgramName
      for Python versions that do not duplicate program_name.  */
   static wchar_t *progname_copy;
+  /* Similar treatment for the Python home directory.  */
+  static wchar_t *home_copy;
 #endif
 
 #ifdef WITH_PYTHON_PATH
@@ -1667,6 +1669,18 @@ do_start_initialization ()
       fprintf (stderr, "Could not convert python path to string\n");
       return false;
     }
+
+  std::string homedir = ldirname (python_libdir.c_str ());
+  home_copy = XNEWVEC (wchar_t, homedir.size () + 1);
+  count = mbstowcs (home_copy, homedir.c_str (), homedir.size () + 1);
+  if (count == (size_t) -1)
+    {
+      fprintf (stderr, "Could not convert python home to wide string\n");
+      return false;
+    }
+  /* Note that, as with the program name, we don't free HOME_COPY.  */
+  Py_SetPythonHome (home_copy);
+
   setlocale (LC_ALL, oldloc.c_str ());
 
   /* Note that Py_SetProgramName expects the string it is passed to
@@ -1678,8 +1692,22 @@ do_start_initialization ()
   PyImport_AppendInittab ("_gdb", init__gdb_module);
 #else
   Py_SetProgramName (progname.release ());
-#endif
-#endif
+
+  /* We override any value that the PYTHONHOME might have, as we want
+     to make sure that we use the Python library that comes with GDB.
+
+     Note that the Python 2.7.17 documentation for Py_SetPythonHome
+     says that the "home" argument should be:
+
+         [...] a zero-terminated character string in static storage
+	 whose contents will not change for the duration of the program’s
+	 execution. No code in the Python interpreter will change
+	 the contents of this storage.
+
+     That's why we have to xstrdup the dirname.  */
+  Py_SetPythonHome (xstrdup (ldirname (python_libdir.c_str ()).c_str ()));
+#endif	/* IS_PY3K */
+#endif	/* WITH_PYTHON_PATH */
 
   Py_Initialize ();
 #if PY_VERSION_HEX < 0x03090000