Re: [RFA] Skip the "red zone" on AMD64

[Michal Ludvig <mludvig@suse.cz>]

Andrew Cagney told me that:

> The PowerOpen (a.k.a. AIX) and (as I only just discovered) PPC64 ABIs 
> make use of the stack beyond the top-of-stack (SP) address.  A very long 
> standing bug is that GDB needs to skip that area before allocating stack 
> space.
> 
> This architecture's `red zone' sounds like the same [mis-]feature.  If 
> it is then the posted change won't fix the problem :-(  The return 
> structure, already allocated on the stack, will smash that stack area.

I think it's something different - while on AIX or PPC64 the called 
function may clobber the stack above it's frame's top-of-stack address, 
on AMD64 it's the calling function that could have used the space below 
bottom-of-stack.

Functions on AMD64 sometimes don't allocate space for their local 
variables (i.e. don't decrement %rsp before pushing something below it), 
which is often the case for leaf functions that don't call anything else.

Imagine a simple leaf function:

int add (int a, int b)
{
	return a + b;
}

If depending on the red zone, it gets compiled to:

add:
         movl    %edi, -4(%rsp)
         movl    %esi, -8(%rsp)
         movl    -8(%rsp), %eax
         addl    -4(%rsp), %eax
         ret

But if the use of the red zone is prohibited (gcc's flag -mno-red-zone), 
it must allocate the space for storing temporary local variables:

add:
         subq    $8, %rsp	# Allocate
         movl    %edi, 4(%rsp)
         movl    %esi, (%rsp)
         movl    (%rsp), %eax
         addl    4(%rsp), %eax
         addq    $8, %rsp	# Deallocate
         ret

The problem is that in this case GDB couldn't know how much space below 
%rsp was used by the function. ABI says it can be up to 128 Bytes of 
"unregistered" space, that is not delimited by %rsp.

So this red zone is not used by the called function but could have been 
used by the calling (or "interrupted" ?) function.

Or is it the same case that AIX does?

> If this is the case, can you please revert the patch and post something 
> more along the above lines.  Is there code in the testsuite that 
> fails/passes with the fix?

It fixed about 10 failures and didn't break anything.

Michal Ludvig
-- 
* SuSE CR, s.r.o     * mludvig@suse.cz
* (+420) 296.545.373 * http://www.suse.cz