Re: [PATCH] Fix python-interactive with Python 3.6

[Simon Marchi <simon.marchi@polymtl.ca>]

On 2017-01-20 10:15, Simon Marchi wrote:
> Since Python 3.4, the callback installed in 
> PyOS_ReadlineFunctionPointer
> should return a value allocated with PyMem_RawMalloc instead of
> PyMem_Malloc.  The reason is that PyMem_Malloc must be called with the
> Python Global Interpreter Lock (GIL) held, which is not the case in the
> context where this function is called.  PyMem_RawMalloc was introduced
> for cases like this.
> 
> In Python 3.6, it looks like they added an assert to verify that
> PyMem_Malloc was not called without the GIL.  The consequence is that
> typing anything in the python-interactive mode of gdb crashes the
> process.  The same behavior was observed with the official package on
> Arch Linux as well as with a manual Python build on Ubuntu 14.04.
> 
> This is what is shown with a debug build of Python 3.6 (the error with 
> a
> non-debug build is far less clear):
> 
>   (gdb) pi
>   >>> print(1)
>   Fatal Python error: Python memory allocator called without holding 
> the GIL
> 
>   Current thread 0x00007f1459af8780 (most recent call first):
>   [1]    21326 abort      ./gdb
> 
> and the backtrace:
> 
>   #0  0x00007ffff618bc37 in raise () from 
> /lib/x86_64-linux-gnu/libc.so.6
>   #1  0x00007ffff618f028 in abort () from 
> /lib/x86_64-linux-gnu/libc.so.6
>   #2  0x00007ffff6b104d6 in Py_FatalError
> (msg=msg@entry=0x7ffff6ba15b8 "Python memory allocator called without
> holding the GIL") at Python/pylifecycle.c:1457
>   #3  0x00007ffff6a37a68 in _PyMem_DebugCheckGIL () at 
> Objects/obmalloc.c:1972
>   #4  0x00007ffff6a3804e in _PyMem_DebugFree (ctx=0x7ffff6e65290
> <_PyMem_Debug+48>, ptr=0x24f8830) at Objects/obmalloc.c:1994
>   #5  0x00007ffff6a38e1d in PyMem_Free (ptr=<optimized out>) at
> Objects/obmalloc.c:442
>   #6  0x00007ffff6b866c6 in _PyFaulthandler_Fini () at
> ./Modules/faulthandler.c:1369
>   #7  0x00007ffff6b104bd in Py_FatalError
> (msg=msg@entry=0x7ffff6ba15b8 "Python memory allocator called without
> holding the GIL") at Python/pylifecycle.c:1431
>   #8  0x00007ffff6a37a68 in _PyMem_DebugCheckGIL () at 
> Objects/obmalloc.c:1972
>   #9  0x00007ffff6a37aa3 in _PyMem_DebugMalloc (ctx=0x7ffff6e65290
> <_PyMem_Debug+48>, nbytes=5) at Objects/obmalloc.c:1980
>   #10 0x00007ffff6a38d91 in PyMem_Malloc (size=<optimized out>) at
> Objects/obmalloc.c:418
>   #11 0x000000000064dbe2 in gdbpy_readline_wrapper
> (sys_stdin=0x7ffff6514640 <_IO_2_1_stdin_>, sys_stdout=0x7ffff6514400
> <_IO_2_1_stdout_>, prompt=0x7ffff4d4f7d0 ">>> ")
>     at /home/emaisin/src/binutils-gdb/gdb/python/py-gdb-readline.c:75
> 
> The documentation is very clear about it [1] and it was also mentioned
> in the "What's New In Python 3.4" page [2].
> 
> [1] 
> https://docs.python.org/3/c-api/veryhigh.html#c.PyOS_ReadlineFunctionPointer
> [2] https://docs.python.org/3/whatsnew/3.4.html#changes-in-the-c-api
> 
> gdb/ChangeLog:
> 
> 	* python/py-gdb-readline.c (PyOS_ReadlineFunctionPointer_Malloc):
> 	Define.
> 	(gdbpy_readline_wrapper): Use it.
> ---
>  gdb/python/py-gdb-readline.c | 14 ++++++++++++--
>  1 file changed, 12 insertions(+), 2 deletions(-)
> 
> diff --git a/gdb/python/py-gdb-readline.c 
> b/gdb/python/py-gdb-readline.c
> index 8b396db443..1d02b03f50 100644
> --- a/gdb/python/py-gdb-readline.c
> +++ b/gdb/python/py-gdb-readline.c
> @@ -21,6 +21,16 @@
>  #include "python-internal.h"
>  #include "top.h"
>  #include "cli/cli-utils.h"
> +
> +/* Starting from Python 3.4, the result of the 
> PyOS_ReadlineFunctionPointer
> +   callback must be allocated with PyMem_RawMalloc rather than
> PyMem_Malloc.  */
> +
> +#if PY_MAJOR_VERSION == 3 && PY_MINOR_VERSION >= 4
> +#define PyOS_ReadlineFunctionPointer_Malloc PyMem_RawMalloc
> +#else
> +#define PyOS_ReadlineFunctionPointer_Malloc PyMem_Malloc
> +#endif
> +
>  /* Readline function suitable for PyOS_ReadlineFunctionPointer, which
>     is used for Python's interactive parser and raw_input.  In both
>     cases, sys_stdin and sys_stdout are always stdin and stdout
> @@ -63,7 +73,7 @@ gdbpy_readline_wrapper (FILE *sys_stdin, FILE 
> *sys_stdout,
>    /* Detect EOF (Ctrl-D).  */
>    if (p == NULL)
>      {
> -      q = (char *) PyMem_Malloc (1);
> +      q = (char *) PyOS_ReadlineFunctionPointer_Malloc (1);
>        if (q != NULL)
>  	q[0] = '\0';
>        return q;
> @@ -72,7 +82,7 @@ gdbpy_readline_wrapper (FILE *sys_stdin, FILE 
> *sys_stdout,
>    n = strlen (p);
> 
>    /* Copy the line to Python and return.  */
> -  q = (char *) PyMem_Malloc (n + 2);
> +  q = (char *) PyOS_ReadlineFunctionPointer_Malloc (n + 2);
>    if (q != NULL)
>      {
>        strncpy (q, p, n);

Hmm, running the testsuite, it seems like there are a ton of other 
issues similar to this one...  For example it blows up when an object 
gets freed because of a decref.  That'll be fun to fix :).