ARM signal trampolines

ARM signal trampolines

[Matt Fischer]

I recently had a situation where I needed to attach to a program, and
examine a backtrace from a thread which was currently handling a
signal.  I know GDB has infrastructure in place for dealing with the
stack trampolines that get put in place during a signal emission, but
it didn't appear to be working--it hit the frame with the trampoline
and basically gave up.  In trying to track down why this wasn't
working, I ran into what I think are two separate problems.  I was
wondering if someone more knowledgeable in this code can confirm that
I'm understanding this correctly.

1.  When delivering a signal, ARM Linux constructs a stack frame with
a return address pointing to a trampoline in the vector table
(0xffff0500-0xffff0514), which in turn issues a syscall back into the
kernel.  GDB has unwinders for this, but it looks like they have a
couple problems.
   a.  There is no unwinder for a return to Thumb code
(SWI_THUMB_SIGRETURN in arch/arm/kernel/signal.c.)
   b.  The normal EABI unwinder (arm_eabi_linux_sigreturn_tramp_frame)
seems to be incorrect as well.  It expects to see 0xe3a07077
0xef000000 (ARM_SET_R7_SIGRETURN followed by ARM_EABI_SYSCALL.)
However the kernel's vector table tries to be cute, and overlaps the
EABI return sequence with the OABI syscall instruction, meaning that
what's actually in the vector table is 0xe3a07077 0xef900077.
Therefore, it doesn't match correctly.

2.  A bigger problem, though, is that GDB can't seem to read the
vector table area at all.  Any attempt to examine that memory area
results in garbage values.  Indeed, I can't even seem to get ptrace()
to read that memory area in any process--any attempt to do so just
returns -1.  This makes it seem as though the kernel just doesn't let
you read that region via ptrace() at all.

Are these assessments correct?  If they are, I guess I don't
understand how this ever worked.  The first two are fairly trivial
problems, and I'd be happy to provide patches.  However, if the latter
is true, it doesn't seem like it's even going to be possible to do
this.  Does anybody have any thoughts on what's supposed to be going
on here?

Thanks,
Matt

Re: ARM signal trampolines

[Daniel Jacobowitz]

On Fri, Jan 15, 2010 at 04:46:52PM -0600, Matt Fischer wrote:
> 1.  When delivering a signal, ARM Linux constructs a stack frame with
> a return address pointing to a trampoline in the vector table
> (0xffff0500-0xffff0514), which in turn issues a syscall back into the
> kernel.  GDB has unwinders for this, but it looks like they have a
> couple problems.
>    a.  There is no unwinder for a return to Thumb code
> (SWI_THUMB_SIGRETURN in arch/arm/kernel/signal.c.)

Looks that way.  This code doesn't matter though...  GLIBC always sets
SA_RESTORER.  What C library are you using that could trigger this
at all?

>    b.  The normal EABI unwinder (arm_eabi_linux_sigreturn_tramp_frame)
> seems to be incorrect as well.  It expects to see 0xe3a07077
> 0xef000000 (ARM_SET_R7_SIGRETURN followed by ARM_EABI_SYSCALL.)
> However the kernel's vector table tries to be cute, and overlaps the
> EABI return sequence with the OABI syscall instruction, meaning that
> what's actually in the vector table is 0xe3a07077 0xef900077.
> Therefore, it doesn't match correctly.

None of this code is for the vector area trampolines which are brand
new.  Just a few months old, I believe.  It is for the SA_RESTORER
functions in glibc.

> 2.  A bigger problem, though, is that GDB can't seem to read the
> vector table area at all.  Any attempt to examine that memory area
> results in garbage values.  Indeed, I can't even seem to get ptrace()
> to read that memory area in any process--any attempt to do so just
> returns -1.  This makes it seem as though the kernel just doesn't let
> you read that region via ptrace() at all.

That's correct.  There's a workaround for single-stepping through the
atomic operations on the same vector page, for this reason.  The
correct solution depends on whether the signal trampolines are at an
ABI-fixed address.  If they are, we could hardcode them.  Otherwise,
someone has to stop putting off making the page ptrace readable.

-- 
Daniel Jacobowitz
CodeSourcery

Re: ARM signal trampolines

[Matt Fischer]

On Fri, Jan 15, 2010 at 5:17 PM, Daniel Jacobowitz <dan@codesourcery.com> wrote:

> Looks that way.  This code doesn't matter though...  GLIBC always sets
> SA_RESTORER.  What C library are you using that could trigger this
> at all?

I'm working on an Android system.  Bionic doesn't seem to bother with
SA_RESTORER--it just wires sigaction() straight up to the syscall, so
unless you specify your own, the kernel sets up the stack frame with a
return address pointing to the trampolines on the vector page.

> None of this code is for the vector area trampolines which are brand
> new.  Just a few months old, I believe.  It is for the SA_RESTORER
> functions in glibc.

I guess I'm confused--the code I'm looking at appears to have been in
the kernel since about 2.6.13--it's the vector of return codes called
sigreturn_codes[] in arch/arm/kernel/signal.c, which gets copied to
the vector page by trap_init() in arch/arm/kernel/traps.c.  Is there
some other change which has been made to this mechanism in more recent
kernels?

>> 2.  A bigger problem, though, is that GDB can't seem to read the
>> vector table area at all.  Any attempt to examine that memory area
>> results in garbage values.  Indeed, I can't even seem to get ptrace()
>> to read that memory area in any process--any attempt to do so just
>> returns -1.  This makes it seem as though the kernel just doesn't let
>> you read that region via ptrace() at all.
>
> That's correct.  There's a workaround for single-stepping through the
> atomic operations on the same vector page, for this reason.  The
> correct solution depends on whether the signal trampolines are at an
> ABI-fixed address.  If they are, we could hardcode them.  Otherwise,
> someone has to stop putting off making the page ptrace readable.

Given what you've said, the easiest thing to do for my purposes is
probably just to patch Bionic to use SA_RESTORER.  Then I can just
ensure the trampoline is constructed to match what's already in there
for glibc, and things should all work out.  I don't know if I could
get it accepted upstream or not, but it should at least allow my local
testing to work out.

Long term, though, it would certainly be nice if gdb could see the
vector page--I've run into a couple situations where I've needed to
see what was in there, and gdb wasn't able to help.  It seems like the
kernel patch to do this wouldn't be overly complicated--is there some
reason that this isn't a desirable feature, or is it just that
nobody's had a pressing enough need for it so far?

--Matt

Re: ARM signal trampolines

[Daniel Jacobowitz]

On Mon, Jan 18, 2010 at 04:27:23PM -0600, Matt Fischer wrote:
> > None of this code is for the vector area trampolines which are brand
> > new. ?Just a few months old, I believe. ?It is for the SA_RESTORER
> > functions in glibc.
> 
> I guess I'm confused--the code I'm looking at appears to have been in
> the kernel since about 2.6.13--it's the vector of return codes called
> sigreturn_codes[] in arch/arm/kernel/signal.c, which gets copied to
> the vector page by trap_init() in arch/arm/kernel/traps.c.  Is there
> some other change which has been made to this mechanism in more recent
> kernels?

I may be confused.  I thought it previously copied code to the stack,
and only recently started putting it on the vector page.

> Given what you've said, the easiest thing to do for my purposes is
> probably just to patch Bionic to use SA_RESTORER.  Then I can just
> ensure the trampoline is constructed to match what's already in there
> for glibc, and things should all work out.  I don't know if I could
> get it accepted upstream or not, but it should at least allow my local
> testing to work out.

Yes, that will be easy and should work.

> Long term, though, it would certainly be nice if gdb could see the
> vector page--I've run into a couple situations where I've needed to
> see what was in there, and gdb wasn't able to help.  It seems like the
> kernel patch to do this wouldn't be overly complicated--is there some
> reason that this isn't a desirable feature, or is it just that
> nobody's had a pressing enough need for it so far?

I think it's just not been needed.

-- 
Daniel Jacobowitz
CodeSourcery

Re: ARM signal trampolines

[Matt Fischer]

On Mon, Jan 18, 2010 at 10:03 PM, Daniel Jacobowitz
<dan@codesourcery.com> wrote:
> On Mon, Jan 18, 2010 at 04:27:23PM -0600, Matt Fischer wrote:
>> > None of this code is for the vector area trampolines which are brand
>> > new. ?Just a few months old, I believe. ?It is for the SA_RESTORER
>> > functions in glibc.
>>
>> I guess I'm confused--the code I'm looking at appears to have been in
>> the kernel since about 2.6.13--it's the vector of return codes called
>> sigreturn_codes[] in arch/arm/kernel/signal.c, which gets copied to
>> the vector page by trap_init() in arch/arm/kernel/traps.c.  Is there
>> some other change which has been made to this mechanism in more recent
>> kernels?
>
> I may be confused.  I thought it previously copied code to the stack,
> and only recently started putting it on the vector page.
>
It looks like it also copies it to the stack, but as long as we're
running in 32-bit mode, it uses the vector page, so that it doesn't
need to do an i-cache flush on the stack (and to reduce the need for
an executable stack.)  This looks to have been added around 2005.
Glibc appears to set a restorer for exactly the same reason, but it
predates this kernel logic by several years, so I imagine it was added
to work around this problem back in the days of ARM2 or something,
before the kernel had the snazzy high memory page.  It looks like
there is probably not a need for it anymore, since the kernel now does
exactly the same thing anyway.

>> Given what you've said, the easiest thing to do for my purposes is
>> probably just to patch Bionic to use SA_RESTORER.  Then I can just
>> ensure the trampoline is constructed to match what's already in there
>> for glibc, and things should all work out.  I don't know if I could
>> get it accepted upstream or not, but it should at least allow my local
>> testing to work out.
>
> Yes, that will be easy and should work.
>
I've implemented this and it seems to work great.  Thanks for all your help.

--Matt