[commit/powerpc] crash trying to allocate memory in inferior

Joel Brobecker
2011-10-24T16:53:41Z

Our testsuite noticed a crash when trying to call a function which
requires GDB to allocate memory in the inferior. Typically, this
happens when one of the parameters is a string. For instance, our
testcase tries:

(gdb) call debug.trace (me, "You")
[1] 32737 segmentation fault /path/to/gdb

What happens is that GDB sees the string, and thus tries to allocate
memory for it in the inferior:

> /* Allocate NBYTES of space in the inferior using the inferior's
> malloc and return a value that is a pointer to the allocated
> space. */
>
> struct value *
> value_allocate_space_in_inferior (int len)
> {
> struct objfile *objf;
> struct value *val = find_function_in_inferior ("malloc", &objf);

And find_function_in_inferior first searches the symtab in case
we have debug info. But, in our case (bareboard powerpc), we don't,
so it gets "malloc"'s address from the minimal symbols, and builds
a value whose type is a TYPE_CODE_PTR, not a TYPE_CODE_FUNC.

As a result, when we later try to make the call to malloc, we end up
inside the powerpc tdep code that has:

> do_ppc_sysv_return_value (struct gdbarch *gdbarch, struct type *func_type,
[...]
> if (func_type
> && TYPE_CALLING_CONVENTION (func_type) == DW_CC_GDB_IBM_OpenCL)

The problem is that func_type is not a TYPE_CODE_FUNC, and thus
the type-specific kind is not TYPE_SPECIFIC_FUNC, and so we do
TYPE_CALLING_CONVENTION is an invalid access.

Interestingly, the other call to TYPE_CALLING_CONVENTION is correctly
preceded by a check of the type's TYPE_CODE (making sure that it is
TYPE_CODE_FUNC).

gdb/ChangeLog:

* ppc-sysv-tdep.c (do_ppc_sysv_return_value): Do not check
FUNC_TYPE's calling convention if FUNC_TYPE is not a function.

tested on powerpc-elf. Checked in.

---
gdb/ChangeLog | 5 +++++
gdb/ppc-sysv-tdep.c | 1 +
2 files changed, 6 insertions(+), 0 deletions(-)

diff --git a/gdb/ChangeLog b/gdb/ChangeLog
index 30cf144..f5cdd45 100644
--- a/gdb/ChangeLog
+++ b/gdb/ChangeLog
@@ -1,3 +1,8 @@
+2011-10-24 Joel Brobecker <brobecker@adacore.com>
+
+ * ppc-sysv-tdep.c (do_ppc_sysv_return_value): Do not check
+ FUNC_TYPE's calling convention if FUNC_TYPE is not a function.
+
2011-10-24 Pedro Alves <pedro@codesourcery.com>

* linux-nat.c (linux_handle_extended_wait): When handling a clone
diff --git a/gdb/ppc-sysv-tdep.c b/gdb/ppc-sysv-tdep.c
index e431363..bda4544 100644
--- a/gdb/ppc-sysv-tdep.c
+++ b/gdb/ppc-sysv-tdep.c
@@ -692,6 +692,7 @@ do_ppc_sysv_return_value (struct gdbarch *gdbarch, struct type *func_type,
int opencl_abi = 0;

if (func_type
+ && TYPE_CODE (func_type) == TYPE_CODE_FUNC
&& TYPE_CALLING_CONVENTION (func_type) == DW_CC_GDB_IBM_OpenCL)
opencl_abi = 1;

--
1.7.1