* [PATCH] Fix python-interactive with Python 3.6
@ 2017-01-20 15:19 Simon Marchi
2017-01-20 15:38 ` Simon Marchi
2017-01-20 16:37 ` Pedro Alves
0 siblings, 2 replies; 5+ messages in thread
From: Simon Marchi @ 2017-01-20 15:19 UTC (permalink / raw)
To: gdb-patches; +Cc: Simon Marchi
Since Python 3.4, the callback installed in PyOS_ReadlineFunctionPointer
should return a value allocated with PyMem_RawMalloc instead of
PyMem_Malloc. The reason is that PyMem_Malloc must be called with the
Python Global Interpreter Lock (GIL) held, which is not the case in the
context where this function is called. PyMem_RawMalloc was introduced
for cases like this.
In Python 3.6, it looks like they added an assert to verify that
PyMem_Malloc was not called without the GIL. The consequence is that
typing anything in the python-interactive mode of gdb crashes the
process. The same behavior was observed with the official package on
Arch Linux as well as with a manual Python build on Ubuntu 14.04.
This is what is shown with a debug build of Python 3.6 (the error with a
non-debug build is far less clear):
(gdb) pi
>>> print(1)
Fatal Python error: Python memory allocator called without holding the GIL
Current thread 0x00007f1459af8780 (most recent call first):
[1] 21326 abort ./gdb
and the backtrace:
#0 0x00007ffff618bc37 in raise () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007ffff618f028 in abort () from /lib/x86_64-linux-gnu/libc.so.6
#2 0x00007ffff6b104d6 in Py_FatalError (msg=msg@entry=0x7ffff6ba15b8 "Python memory allocator called without holding the GIL") at Python/pylifecycle.c:1457
#3 0x00007ffff6a37a68 in _PyMem_DebugCheckGIL () at Objects/obmalloc.c:1972
#4 0x00007ffff6a3804e in _PyMem_DebugFree (ctx=0x7ffff6e65290 <_PyMem_Debug+48>, ptr=0x24f8830) at Objects/obmalloc.c:1994
#5 0x00007ffff6a38e1d in PyMem_Free (ptr=<optimized out>) at Objects/obmalloc.c:442
#6 0x00007ffff6b866c6 in _PyFaulthandler_Fini () at ./Modules/faulthandler.c:1369
#7 0x00007ffff6b104bd in Py_FatalError (msg=msg@entry=0x7ffff6ba15b8 "Python memory allocator called without holding the GIL") at Python/pylifecycle.c:1431
#8 0x00007ffff6a37a68 in _PyMem_DebugCheckGIL () at Objects/obmalloc.c:1972
#9 0x00007ffff6a37aa3 in _PyMem_DebugMalloc (ctx=0x7ffff6e65290 <_PyMem_Debug+48>, nbytes=5) at Objects/obmalloc.c:1980
#10 0x00007ffff6a38d91 in PyMem_Malloc (size=<optimized out>) at Objects/obmalloc.c:418
#11 0x000000000064dbe2 in gdbpy_readline_wrapper (sys_stdin=0x7ffff6514640 <_IO_2_1_stdin_>, sys_stdout=0x7ffff6514400 <_IO_2_1_stdout_>, prompt=0x7ffff4d4f7d0 ">>> ")
at /home/emaisin/src/binutils-gdb/gdb/python/py-gdb-readline.c:75
The documentation is very clear about it [1] and it was also mentioned
in the "What's New In Python 3.4" page [2].
[1] https://docs.python.org/3/c-api/veryhigh.html#c.PyOS_ReadlineFunctionPointer
[2] https://docs.python.org/3/whatsnew/3.4.html#changes-in-the-c-api
gdb/ChangeLog:
* python/py-gdb-readline.c (PyOS_ReadlineFunctionPointer_Malloc):
Define.
(gdbpy_readline_wrapper): Use it.
---
gdb/python/py-gdb-readline.c | 14 ++++++++++++--
1 file changed, 12 insertions(+), 2 deletions(-)
diff --git a/gdb/python/py-gdb-readline.c b/gdb/python/py-gdb-readline.c
index 8b396db443..1d02b03f50 100644
--- a/gdb/python/py-gdb-readline.c
+++ b/gdb/python/py-gdb-readline.c
@@ -21,6 +21,16 @@
#include "python-internal.h"
#include "top.h"
#include "cli/cli-utils.h"
+
+/* Starting from Python 3.4, the result of the PyOS_ReadlineFunctionPointer
+ callback must be allocated with PyMem_RawMalloc rather than PyMem_Malloc. */
+
+#if PY_MAJOR_VERSION == 3 && PY_MINOR_VERSION >= 4
+#define PyOS_ReadlineFunctionPointer_Malloc PyMem_RawMalloc
+#else
+#define PyOS_ReadlineFunctionPointer_Malloc PyMem_Malloc
+#endif
+
/* Readline function suitable for PyOS_ReadlineFunctionPointer, which
is used for Python's interactive parser and raw_input. In both
cases, sys_stdin and sys_stdout are always stdin and stdout
@@ -63,7 +73,7 @@ gdbpy_readline_wrapper (FILE *sys_stdin, FILE *sys_stdout,
/* Detect EOF (Ctrl-D). */
if (p == NULL)
{
- q = (char *) PyMem_Malloc (1);
+ q = (char *) PyOS_ReadlineFunctionPointer_Malloc (1);
if (q != NULL)
q[0] = '\0';
return q;
@@ -72,7 +82,7 @@ gdbpy_readline_wrapper (FILE *sys_stdin, FILE *sys_stdout,
n = strlen (p);
/* Copy the line to Python and return. */
- q = (char *) PyMem_Malloc (n + 2);
+ q = (char *) PyOS_ReadlineFunctionPointer_Malloc (n + 2);
if (q != NULL)
{
strncpy (q, p, n);
--
2.11.0
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [PATCH] Fix python-interactive with Python 3.6
2017-01-20 15:19 [PATCH] Fix python-interactive with Python 3.6 Simon Marchi
@ 2017-01-20 15:38 ` Simon Marchi
2017-01-20 16:37 ` Pedro Alves
1 sibling, 0 replies; 5+ messages in thread
From: Simon Marchi @ 2017-01-20 15:38 UTC (permalink / raw)
To: Simon Marchi; +Cc: gdb-patches
On 2017-01-20 10:15, Simon Marchi wrote:
> Since Python 3.4, the callback installed in
> PyOS_ReadlineFunctionPointer
> should return a value allocated with PyMem_RawMalloc instead of
> PyMem_Malloc. The reason is that PyMem_Malloc must be called with the
> Python Global Interpreter Lock (GIL) held, which is not the case in the
> context where this function is called. PyMem_RawMalloc was introduced
> for cases like this.
>
> In Python 3.6, it looks like they added an assert to verify that
> PyMem_Malloc was not called without the GIL. The consequence is that
> typing anything in the python-interactive mode of gdb crashes the
> process. The same behavior was observed with the official package on
> Arch Linux as well as with a manual Python build on Ubuntu 14.04.
>
> This is what is shown with a debug build of Python 3.6 (the error with
> a
> non-debug build is far less clear):
>
> (gdb) pi
> >>> print(1)
> Fatal Python error: Python memory allocator called without holding
> the GIL
>
> Current thread 0x00007f1459af8780 (most recent call first):
> [1] 21326 abort ./gdb
>
> and the backtrace:
>
> #0 0x00007ffff618bc37 in raise () from
> /lib/x86_64-linux-gnu/libc.so.6
> #1 0x00007ffff618f028 in abort () from
> /lib/x86_64-linux-gnu/libc.so.6
> #2 0x00007ffff6b104d6 in Py_FatalError
> (msg=msg@entry=0x7ffff6ba15b8 "Python memory allocator called without
> holding the GIL") at Python/pylifecycle.c:1457
> #3 0x00007ffff6a37a68 in _PyMem_DebugCheckGIL () at
> Objects/obmalloc.c:1972
> #4 0x00007ffff6a3804e in _PyMem_DebugFree (ctx=0x7ffff6e65290
> <_PyMem_Debug+48>, ptr=0x24f8830) at Objects/obmalloc.c:1994
> #5 0x00007ffff6a38e1d in PyMem_Free (ptr=<optimized out>) at
> Objects/obmalloc.c:442
> #6 0x00007ffff6b866c6 in _PyFaulthandler_Fini () at
> ./Modules/faulthandler.c:1369
> #7 0x00007ffff6b104bd in Py_FatalError
> (msg=msg@entry=0x7ffff6ba15b8 "Python memory allocator called without
> holding the GIL") at Python/pylifecycle.c:1431
> #8 0x00007ffff6a37a68 in _PyMem_DebugCheckGIL () at
> Objects/obmalloc.c:1972
> #9 0x00007ffff6a37aa3 in _PyMem_DebugMalloc (ctx=0x7ffff6e65290
> <_PyMem_Debug+48>, nbytes=5) at Objects/obmalloc.c:1980
> #10 0x00007ffff6a38d91 in PyMem_Malloc (size=<optimized out>) at
> Objects/obmalloc.c:418
> #11 0x000000000064dbe2 in gdbpy_readline_wrapper
> (sys_stdin=0x7ffff6514640 <_IO_2_1_stdin_>, sys_stdout=0x7ffff6514400
> <_IO_2_1_stdout_>, prompt=0x7ffff4d4f7d0 ">>> ")
> at /home/emaisin/src/binutils-gdb/gdb/python/py-gdb-readline.c:75
>
> The documentation is very clear about it [1] and it was also mentioned
> in the "What's New In Python 3.4" page [2].
>
> [1]
> https://docs.python.org/3/c-api/veryhigh.html#c.PyOS_ReadlineFunctionPointer
> [2] https://docs.python.org/3/whatsnew/3.4.html#changes-in-the-c-api
>
> gdb/ChangeLog:
>
> * python/py-gdb-readline.c (PyOS_ReadlineFunctionPointer_Malloc):
> Define.
> (gdbpy_readline_wrapper): Use it.
> ---
> gdb/python/py-gdb-readline.c | 14 ++++++++++++--
> 1 file changed, 12 insertions(+), 2 deletions(-)
>
> diff --git a/gdb/python/py-gdb-readline.c
> b/gdb/python/py-gdb-readline.c
> index 8b396db443..1d02b03f50 100644
> --- a/gdb/python/py-gdb-readline.c
> +++ b/gdb/python/py-gdb-readline.c
> @@ -21,6 +21,16 @@
> #include "python-internal.h"
> #include "top.h"
> #include "cli/cli-utils.h"
> +
> +/* Starting from Python 3.4, the result of the
> PyOS_ReadlineFunctionPointer
> + callback must be allocated with PyMem_RawMalloc rather than
> PyMem_Malloc. */
> +
> +#if PY_MAJOR_VERSION == 3 && PY_MINOR_VERSION >= 4
> +#define PyOS_ReadlineFunctionPointer_Malloc PyMem_RawMalloc
> +#else
> +#define PyOS_ReadlineFunctionPointer_Malloc PyMem_Malloc
> +#endif
> +
> /* Readline function suitable for PyOS_ReadlineFunctionPointer, which
> is used for Python's interactive parser and raw_input. In both
> cases, sys_stdin and sys_stdout are always stdin and stdout
> @@ -63,7 +73,7 @@ gdbpy_readline_wrapper (FILE *sys_stdin, FILE
> *sys_stdout,
> /* Detect EOF (Ctrl-D). */
> if (p == NULL)
> {
> - q = (char *) PyMem_Malloc (1);
> + q = (char *) PyOS_ReadlineFunctionPointer_Malloc (1);
> if (q != NULL)
> q[0] = '\0';
> return q;
> @@ -72,7 +82,7 @@ gdbpy_readline_wrapper (FILE *sys_stdin, FILE
> *sys_stdout,
> n = strlen (p);
>
> /* Copy the line to Python and return. */
> - q = (char *) PyMem_Malloc (n + 2);
> + q = (char *) PyOS_ReadlineFunctionPointer_Malloc (n + 2);
> if (q != NULL)
> {
> strncpy (q, p, n);
Hmm, running the testsuite, it seems like there are a ton of other
issues similar to this one... For example it blows up when an object
gets freed because of a decref. That'll be fun to fix :).
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [PATCH] Fix python-interactive with Python 3.6
2017-01-20 15:19 [PATCH] Fix python-interactive with Python 3.6 Simon Marchi
2017-01-20 15:38 ` Simon Marchi
@ 2017-01-20 16:37 ` Pedro Alves
2017-01-20 16:49 ` Simon Marchi
1 sibling, 1 reply; 5+ messages in thread
From: Pedro Alves @ 2017-01-20 16:37 UTC (permalink / raw)
To: Simon Marchi, gdb-patches
On 01/20/2017 03:15 PM, Simon Marchi wrote:
> Since Python 3.4, the callback installed in PyOS_ReadlineFunctionPointer
> should return a value allocated with PyMem_RawMalloc instead of
> PyMem_Malloc. The reason is that PyMem_Malloc must be called with the
> Python Global Interpreter Lock (GIL) held, which is not the case in the
> context where this function is called. PyMem_RawMalloc was introduced
> for cases like this.
>
> +/* Starting from Python 3.4, the result of the PyOS_ReadlineFunctionPointer
> + callback must be allocated with PyMem_RawMalloc rather than PyMem_Malloc. */
> +
> +#if PY_MAJOR_VERSION == 3 && PY_MINOR_VERSION >= 4
This will break with PY_MAJOR_VERSION 4 (at some point :-) ).
Write:
#if PY_MAJOR_VERSION > 3
|| (PY_MAJOR_VERSION == 3 && PY_MINOR_VERSION >= 4)
?
Or add some macro that makes it a bit easier to write,
like "#if HAVE_PY_AT_LEAST(3, 4)" or
"#if GDB_PY_VERSION >= 3004" (like GCC_VERSION).
Or use Python's PY_VERSION_HEX, like
"#if PY_VERSION_HEX >= 0x03040000".
> +#define PyOS_ReadlineFunctionPointer_Malloc PyMem_RawMalloc
> +#else
> +#define PyOS_ReadlineFunctionPointer_Malloc PyMem_Malloc
> +#endif
It sounds like we'll find other cases that will need
to call PyMem_RawMalloc going forward? How about handling
this similarly to how we handle fixing up other missing
Python bits, in python-internal.h. Like:
#if python < 3.4
static inline void *
gdb_PyMem_RawMalloc (size_t n)
{
return gdb_PyMem_Malloc (n);
}
#define PyMem_RawMalloc(n) gdb_PyMem_RawMalloc (n)
#endif
> +
> /* Readline function suitable for PyOS_ReadlineFunctionPointer, which
> is used for Python's interactive parser and raw_input. In both
> cases, sys_stdin and sys_stdout are always stdin and stdout
> @@ -63,7 +73,7 @@ gdbpy_readline_wrapper (FILE *sys_stdin, FILE *sys_stdout,
> /* Detect EOF (Ctrl-D). */
> if (p == NULL)
> {
> - q = (char *) PyMem_Malloc (1);
> + q = (char *) PyOS_ReadlineFunctionPointer_Malloc (1);
and then write PyMem_RawMalloc here.
Thanks,
Pedro Alves
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [PATCH] Fix python-interactive with Python 3.6
2017-01-20 16:37 ` Pedro Alves
@ 2017-01-20 16:49 ` Simon Marchi
2017-01-20 16:51 ` Pedro Alves
0 siblings, 1 reply; 5+ messages in thread
From: Simon Marchi @ 2017-01-20 16:49 UTC (permalink / raw)
To: Pedro Alves; +Cc: Simon Marchi, gdb-patches
On 2017-01-20 11:37, Pedro Alves wrote:
> On 01/20/2017 03:15 PM, Simon Marchi wrote:
>> Since Python 3.4, the callback installed in
>> PyOS_ReadlineFunctionPointer
>> should return a value allocated with PyMem_RawMalloc instead of
>> PyMem_Malloc. The reason is that PyMem_Malloc must be called with the
>> Python Global Interpreter Lock (GIL) held, which is not the case in
>> the
>> context where this function is called. PyMem_RawMalloc was introduced
>> for cases like this.
>>
>
>> +/* Starting from Python 3.4, the result of the
>> PyOS_ReadlineFunctionPointer
>> + callback must be allocated with PyMem_RawMalloc rather than
>> PyMem_Malloc. */
>> +
>> +#if PY_MAJOR_VERSION == 3 && PY_MINOR_VERSION >= 4
>
> This will break with PY_MAJOR_VERSION 4 (at some point :-) ).
Yeah, I thought about that and concluded it would be a problem for my
grandchildren ;).
> Write:
>
> #if PY_MAJOR_VERSION > 3
> || (PY_MAJOR_VERSION == 3 && PY_MINOR_VERSION >= 4)
>
> ?
Ok.
> Or add some macro that makes it a bit easier to write,
> like "#if HAVE_PY_AT_LEAST(3, 4)" or
> "#if GDB_PY_VERSION >= 3004" (like GCC_VERSION).
> Or use Python's PY_VERSION_HEX, like
> "#if PY_VERSION_HEX >= 0x03040000".
I think that's better.
>> +#define PyOS_ReadlineFunctionPointer_Malloc PyMem_RawMalloc
>> +#else
>> +#define PyOS_ReadlineFunctionPointer_Malloc PyMem_Malloc
>> +#endif
>
> It sounds like we'll find other cases that will need
> to call PyMem_RawMalloc going forward? How about handling
> this similarly to how we handle fixing up other missing
> Python bits, in python-internal.h. Like:
For now it looks like this is the only place where the Python API
requires using PyMem_RawMalloc. In the other cases, we are holding the
GIL, thanks to gdbpy_enter, so using the regular PyMem functions is ok.
It doesn't mean there won't be other cases added in the future though...
> #if python < 3.4
>
> static inline void *
> gdb_PyMem_RawMalloc (size_t n)
> {
> return gdb_PyMem_Malloc (n);
> }
> #define PyMem_RawMalloc(n) gdb_PyMem_RawMalloc (n)
>
> #endif
... and this is cleaner anyway. But why not just
#if python < 3.4
#define PyMem_RawMalloc PyMem_Malloc
#endif
?
>> +
>> /* Readline function suitable for PyOS_ReadlineFunctionPointer, which
>> is used for Python's interactive parser and raw_input. In both
>> cases, sys_stdin and sys_stdout are always stdin and stdout
>> @@ -63,7 +73,7 @@ gdbpy_readline_wrapper (FILE *sys_stdin, FILE
>> *sys_stdout,
>> /* Detect EOF (Ctrl-D). */
>> if (p == NULL)
>> {
>> - q = (char *) PyMem_Malloc (1);
>> + q = (char *) PyOS_ReadlineFunctionPointer_Malloc (1);
>
> and then write PyMem_RawMalloc here.
>
> Thanks,
> Pedro Alves
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [PATCH] Fix python-interactive with Python 3.6
2017-01-20 16:49 ` Simon Marchi
@ 2017-01-20 16:51 ` Pedro Alves
0 siblings, 0 replies; 5+ messages in thread
From: Pedro Alves @ 2017-01-20 16:51 UTC (permalink / raw)
To: Simon Marchi; +Cc: Simon Marchi, gdb-patches
On 01/20/2017 04:48 PM, Simon Marchi wrote:
> Yeah, I thought about that and concluded it would be a problem for my
> grandchildren ;).
>
;-)
> ... and this is cleaner anyway. But why not just
>
> #if python < 3.4
> #define PyMem_RawMalloc PyMem_Malloc
> #endif
>
> ?
Because I was just copy/pasting from python-internal.h
and didn't really think. :-) The other instances in
python-internal.h do it like that because they need to
redefine a symbol, not just define something missing.
Thanks,
Pedro Alves
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2017-01-20 16:51 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2017-01-20 15:19 [PATCH] Fix python-interactive with Python 3.6 Simon Marchi
2017-01-20 15:38 ` Simon Marchi
2017-01-20 16:37 ` Pedro Alves
2017-01-20 16:49 ` Simon Marchi
2017-01-20 16:51 ` Pedro Alves
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox