Problem reading corefiles on ARM

Problem reading corefiles on ARM

[Sergei Poselenov]

Hello,

I'm trying to test the GDB ability to analyze ARM coredumps:

-bash-3.2# gdb t core.967
GNU gdb Red Hat Linux (6.7-1rh)
Copyright (C) 2007 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later 
<http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show 
copying"
and "show warranty" for details.
This GDB was configured as "arm-linux"...
Using host libthread_db library "/lib/libthread_db.so.1".
Reading symbols from /lib/libc.so.6...done.
Loaded symbols for /lib/libc.so.6
Reading symbols from /lib/ld-linux.so.3...done.
Loaded symbols for /lib/ld-linux.so.3
Core was generated by `./t'.
Program terminated with signal 6, Aborted.
#0  0x4004ec0c in raise () from /lib/libc.so.6
(gdb) bt
#0  0x4004ec0c in raise () from /lib/libc.so.6
#1  0x40050234 in abort () from /lib/libc.so.6
Backtrace stopped: frame did not save the PC
(gdb) l
1       foo (int a, int b)
2       {
3               int c=40;
4               abort();
5               return a+b;
6       }
7       main()
8       {
9               printf("hello\n");
10              foo(10, 20);
(gdb)
11      }
(gdb)


My environment: ARM native GDB-6.7, toolchain is gcc-4.2.2,
binutils-2.17.50.0.12, glibc-2.6, built as cross-tools, 
arm-linux-eabi. Target running kernel 2.6.22.6.

Is this ARM GDB bug? I tried GDB 6.3 with non-eabi ARM toolchain
(gcc-4.0.0) running the same kernel - all the same.

Thanks for any help.

Regards,
Sergei

Re: Problem reading corefiles on ARM

[Daniel Jacobowitz]

On Wed, Aug 06, 2008 at 07:19:26PM +0400, Sergei Poselenov wrote:
> (gdb) bt
> #0  0x4004ec0c in raise () from /lib/libc.so.6
> #1  0x40050234 in abort () from /lib/libc.so.6
> Backtrace stopped: frame did not save the PC

Your implementation of abort does not save a return address, so GDB
can't display it.  I believe tehis is a known limitation of the ARM
GCC port.


-- 
Daniel Jacobowitz
CodeSourcery

Re: Problem reading corefiles on ARM

[Mark Kettenis]

> Date: Wed, 6 Aug 2008 11:27:36 -0400
> From: Daniel Jacobowitz <drow@false.org>
> 
> On Wed, Aug 06, 2008 at 07:19:26PM +0400, Sergei Poselenov wrote:
> > (gdb) bt
> > #0  0x4004ec0c in raise () from /lib/libc.so.6
> > #1  0x40050234 in abort () from /lib/libc.so.6
> > Backtrace stopped: frame did not save the PC
> 
> Your implementation of abort does not save a return address, so GDB
> can't display it.  I believe tehis is a known limitation of the ARM
> GCC port.

GCC should really not do this.  People are almost guaranteed to want
to be able to see a backtrace from abort(3).

I suppose it optimizes away the instructions to save the return
address, because abort() is marked with __attribute__(noreturn).  But
that means there is very little point in actually doing that
optimization since __attribute__(noreturn) implies that the function
will only be called once!  I suppose there are some space savings but
are they really significant?

Re: Problem reading corefiles on ARM

[Daniel Jacobowitz]

On Wed, Aug 06, 2008 at 05:42:48PM +0200, Mark Kettenis wrote:
> GCC should really not do this.  People are almost guaranteed to want
> to be able to see a backtrace from abort(3).

Yes, it's come up for discussion several times... I don't like the
current behavior either.  The current behavior was added many years
ago for an unspecified application :-(

  /* Decide if the current function is volatile.  Such functions
     never return, and many memory cycles can be saved by not storing
     register values that will never be needed again.  This optimization
     was added to speed up context switching in a kernel application.  */

IMO that shouldn't be written in C, then...

-- 
Daniel Jacobowitz
CodeSourcery

Re: Problem reading corefiles on ARM

[Paul Koning]

>>>>> "Mark" == Mark Kettenis <mark.kettenis@xs4all.nl> writes:

 >> Date: Wed, 6 Aug 2008 11:27:36 -0400 From: Daniel Jacobowitz
 >> <drow@false.org>
 >> 
 >> On Wed, Aug 06, 2008 at 07:19:26PM +0400, Sergei Poselenov wrote:
 >> > (gdb) bt > #0 0x4004ec0c in raise () from /lib/libc.so.6 > #1
 >> 0x40050234 in abort () from /lib/libc.so.6 > Backtrace stopped:
 >> frame did not save the PC
 >> 
 >> Your implementation of abort does not save a return address, so
 >> GDB can't display it.  I believe tehis is a known limitation of
 >> the ARM GCC port.

 Mark> GCC should really not do this.  People are almost guaranteed to
 Mark> want to be able to see a backtrace from abort(3).

Absolutely.

 Mark> I suppose it optimizes away the instructions to save the return
 Mark> address, because abort() is marked with
 Mark> __attribute__(noreturn).  But that means there is very little
 Mark> point in actually doing that optimization since
 Mark> __attribute__(noreturn) implies that the function will only be
 Mark> called once!  I suppose there are some space savings but are
 Mark> they really significant?

I think the space savings in "noreturn" come from not having to save
caller-saved registers in the calling function.  That savings can add
up if the noreturn function is called from many places.

Clearly the return address needs to be saved in the case of functions
like "abort".  Come to think of it, probably all the usual registers
should be saved, so you can examine variables in the function that
called abort and not get nonsense.

It sounds to me like the "noreturn" attribute should be removed from
"abort".

     paul

Re: Problem reading corefiles on ARM

[Joe Buck]

On Wed, Aug 06, 2008 at 11:54:42AM -0400, Paul Koning wrote:
> I think the space savings in "noreturn" come from not having to save
> caller-saved registers in the calling function.  That savings can add
> up if the noreturn function is called from many places.
> 
> Clearly the return address needs to be saved in the case of functions
> like "abort".  Come to think of it, probably all the usual registers
> should be saved, so you can examine variables in the function that
> called abort and not get nonsense.
> 
> It sounds to me like the "noreturn" attribute should be removed from
> "abort".

I don't think that this is the right way to go.

There are several effects from "noreturn".  We would want some of these
effects for "abort", but not others, to get debuggable code without
degrading compile-time warnings.

For the function

#include <stdlib.h>

int func(int arg) {
  switch (arg) {
  case 0:  return 23;
  case 1:  return 42;
  default: abort();
  }
}

getting rid of the "noreturn" attribute on abort() would make -Wall
give the warning

foo.c: In function `func':
foo.c:9: warning: control reaches end of non-void function

So, to produce debuggable code, instead of getting rid of the attribute,
we should instead keep the attribute, as well as enough information to
debug.

Some users will want the maximum optimization even if it impedes
debugging.  We already have -fomit-frame-pointer, which leaves out
information on stack frames.  Perhaps that flag could also enable
(or in its absence, disable) the omission of information about
calls to noreturn functions.  An alternative would be a separate flag.

Re: Problem reading corefiles on ARM

[Paul Koning]

>>>>> "Joe" == Joe Buck <Joe.Buck@synopsys.COM> writes:

 Joe> On Wed, Aug 06, 2008 at 11:54:42AM -0400, Paul Koning wrote:
 >> I think the space savings in "noreturn" come from not having to
 >> save caller-saved registers in the calling function.  That savings
 >> can add up if the noreturn function is called from many places.
 >> 
 >> Clearly the return address needs to be saved in the case of
 >> functions like "abort".  Come to think of it, probably all the
 >> usual registers should be saved, so you can examine variables in
 >> the function that called abort and not get nonsense.
 >> 
 >> It sounds to me like the "noreturn" attribute should be removed
 >> from "abort".

 Joe> I don't think that this is the right way to go.

 Joe> There are several effects from "noreturn".  We would want some
 Joe> of these effects for "abort", but not others, to get debuggable
 Joe> code without degrading compile-time warnings.

Good point.

So the issue is that two unrelated features are currently combined in
a single attribute:

1. This function doesn't return, do the right thing with warnings in 
   the caller of this function.

2. Don't bother saving registers when calling this function because it
   won't return so the registers aren't needed afterwards.

The issue is that #2 doesn't apply to "abort" because the registers
ARE needed afterwards -- at debug time.

One possibility is to have "noreturn" mean #1 only, and invent a new
flag for #2.  The alternative is to have "noreturn" mean both (as
today) and invent a new flag to cancel #2.

I'd suggest the former because that's the more likely case.  It's hard
to think of examples where #2 is needed (or at least, where it is
important). 

	    paul

Re: Problem reading corefiles on ARM

[Joe Buck]

I wrote:
>  Joe> There are several effects from "noreturn".  We would want some
>  Joe> of these effects for "abort", but not others, to get debuggable
>  Joe> code without degrading compile-time warnings.

On Wed, Aug 06, 2008 at 01:37:51PM -0400, Paul Koning wrote:
> So the issue is that two unrelated features are currently combined in
> a single attribute:
> 
> 1. This function doesn't return, do the right thing with warnings in 
>    the caller of this function.
> 
> 2. Don't bother saving registers when calling this function because it
>    won't return so the registers aren't needed afterwards.
> 
> The issue is that #2 doesn't apply to "abort" because the registers
> ARE needed afterwards -- at debug time.

But not necessarily all of them (depending on platform).  That is,
the caller-saved registers don't have to be saved because the function
isn't returning, but there has to be enough of a stack frame so that
a debugger can set a breakpoint on the abort and determine who the
caller was.

> One possibility is to have "noreturn" mean #1 only, and invent a new
> flag for #2.  The alternative is to have "noreturn" mean both (as
> today) and invent a new flag to cancel #2.
> 
> I'd suggest the former because that's the more likely case.  It's hard
> to think of examples where #2 is needed (or at least, where it is
> important). 

Evidently there was a motivating application where #2 was an important
optimization.  Still, I think you're right that it would be better to
do it only if the optimization is asked for (via a new flag).

Re: Problem reading corefiles on ARM

[Paul Koning]

>>>>> "Joe" == Joe Buck <Joe.Buck@synopsys.COM> writes:

 Joe> I wrote: There are several effects from "noreturn".  We would
 Joe> want some of these effects for "abort", but not others, to get
 Joe> debuggable code without degrading compile-time warnings.

 Joe> On Wed, Aug 06, 2008 at 01:37:51PM -0400, Paul Koning wrote:
 >> So the issue is that two unrelated features are currently combined
 >> in a single attribute:
 >> 
 >> 1. This function doesn't return, do the right thing with warnings
 >> in the caller of this function.
 >> 
 >> 2. Don't bother saving registers when calling this function
 >> because it won't return so the registers aren't needed afterwards.
 >> 
 >> The issue is that #2 doesn't apply to "abort" because the
 >> registers ARE needed afterwards -- at debug time.

 Joe> But not necessarily all of them (depending on platform).  That
 Joe> is, the caller-saved registers don't have to be saved because
 Joe> the function isn't returning, but there has to be enough of a
 Joe> stack frame so that a debugger can set a breakpoint on the abort
 Joe> and determine who the caller was.

That's sufficient for live debugging but not for corefiles.  In that
case you do want caller-saved registers, because they may contain
local variable values that don't live in memory at the time of the
abort call.

      paul

Re: Problem reading corefiles on ARM

[Andreas Schwab]

Paul Koning <Paul_Koning@dell.com> writes:

> That's sufficient for live debugging but not for corefiles.  In that
> case you do want caller-saved registers, because they may contain
> local variable values that don't live in memory at the time of the
> abort call.

In an optimized build you can't expect any local variable to survive,
since it may just be dead before the call, or its value may be
unavailable for any other reason.  The use of the noreturn attribute
only adds little to this.

Andreas.

-- 
Andreas Schwab, SuSE Labs, schwab@suse.de
SuSE Linux Products GmbH, Maxfeldstraße 5, 90409 Nürnberg, Germany
PGP key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: Problem reading corefiles on ARM

[Joe Buck]

Paul Koning <Paul_Koning@dell.com> writes:
> > That's sufficient for live debugging but not for corefiles.  In that
> > case you do want caller-saved registers, because they may contain
> > local variable values that don't live in memory at the time of the
> > abort call.

On Wed, Aug 06, 2008 at 11:38:14PM +0200, Andreas Schwab wrote:
> In an optimized build you can't expect any local variable to survive,
> since it may just be dead before the call, or its value may be
> unavailable for any other reason.  The use of the noreturn attribute
> only adds little to this.

Agreed.  I think that the objective should be that if there is an abort
call, then from either a core dump or when gdb'ing a live process it
should be possible to determine the call site of the abort() call, even
with -O2/-Os.  But beyond that, the optimizer should be free, just as
in other cases, to discard values in registers that will never be used
again.

After all, if we have

int func1(int);
void func2(int);
void ordinary_function(void);

void func(int arg) {
   int v1 = func1(arg);
   func2(v1);
   ordinary_function();
}

and there's a segfault in ordinary_function, in general v1 isn't
going to be kept around for inspection in the core dump.  So why
should we impose a stricter requirement if ordinary_function is
replaced by abort() ?  It seems Paul thinks we should be required
to save v1 if there's an abort call, unless I'm misunderstanding.

Re: Problem reading corefiles on ARM

[Paul Koning]

>>>>> "Joe" == Joe Buck <Joe.Buck@synopsys.COM> writes:

 Joe> Paul Koning <Paul_Koning@dell.com> writes:
 >> > That's sufficient for live debugging but not for corefiles.  In
 >> that > case you do want caller-saved registers, because they may
 >> contain > local variable values that don't live in memory at the
 >> time of the > abort call.

 Joe> On Wed, Aug 06, 2008 at 11:38:14PM +0200, Andreas Schwab wrote:
 >> In an optimized build you can't expect any local variable to
 >> survive, since it may just be dead before the call, or its value
 >> may be unavailable for any other reason.  The use of the noreturn
 >> attribute only adds little to this.

 Joe> Agreed.  I think that the objective should be that if there is
 Joe> an abort call, then from either a core dump or when gdb'ing a
 Joe> live process it should be possible to determine the call site of
 Joe> the abort() call, even with -O2/-Os.  But beyond that, the
 Joe> optimizer should be free, just as in other cases, to discard
 Joe> values in registers that will never be used again.

 Joe> After all, if we have

 Joe> int func1(int); void func2(int); void ordinary_function(void);

 Joe> void func(int arg) { int v1 = func1(arg); func2(v1);
 Joe> ordinary_function(); }

 Joe> and there's a segfault in ordinary_function, in general v1 isn't
 Joe> going to be kept around for inspection in the core dump.  So why
 Joe> should we impose a stricter requirement if ordinary_function is
 Joe> replaced by abort() ?  It seems Paul thinks we should be
 Joe> required to save v1 if there's an abort call, unless I'm
 Joe> misunderstanding.

My view is that abort() should be like other (returning) functions --
no special treatment for variable liveness.  Yes, that means in the
optimized case you may lose, some of the time.  People chose -O
settings with that issue in mind.  I'm not asking for more to be saved
for abort() than for regular functions -- but also no less.

    paul

Re: Problem reading corefiles on ARM

[Joe Buck]

Joe> After all, if we have

int func1(int);
void func2(int);
void ordinary_function(void);
 
void func(int arg) {
      int v1 = func1(arg);
      func2(v1);
      ordinary_function();
}

>  Joe> and there's a segfault in ordinary_function, in general v1 isn't
>  Joe> going to be kept around for inspection in the core dump.  So why
>  Joe> should we impose a stricter requirement if ordinary_function is
>  Joe> replaced by abort() ?  It seems Paul thinks we should be
>  Joe> required to save v1 if there's an abort call, unless I'm
>  Joe> misunderstanding.

On Thu, Aug 07, 2008 at 09:28:30AM -0400, Paul Koning wrote:
> My view is that abort() should be like other (returning) functions --
> no special treatment for variable liveness.  Yes, that means in the
> optimized case you may lose, some of the time.  People chose -O
> settings with that issue in mind.  I'm not asking for more to be saved
> for abort() than for regular functions -- but also no less.

I'm not sure if anyone besides Paul and me care any more, but I'll just
do one more.

OK, consider this case:

int func1(int);
void func2(int);
bool test(void);
void func3(int);

void func(int arg) {
      int v1 = func1(arg);
      func2(v1);
      if (test()) {
          func3(v1);
      }
}

Here if we put v1 in a register, we obviously have to save it across
the call to test(), unless we know that test() will never return true,
in which case we don't need to save v1.

But what about replacing the "if" by

      if (!test())
          abort();
      func3(v1);

Now, if I read you right, we'd have so save v1 even if we know that
test() returns false.

Re: Problem reading corefiles on ARM

[Paul Koning]

>>>>> "Joe" == Joe Buck <Joe.Buck@synopsys.COM> writes:

 Joe> ...OK, consider this case:

 Joe> int func1(int); void func2(int); bool test(void); void
 Joe> func3(int);

 Joe> void func(int arg) { int v1 = func1(arg); func2(v1); if (test())
 Joe> { func3(v1); } }

 Joe> Here if we put v1 in a register, we obviously have to save it
 Joe> across the call to test(), unless we know that test() will never
 Joe> return true, in which case we don't need to save v1.

 Joe> But what about replacing the "if" by

 Joe> if (!test()) abort(); func3(v1);

 Joe> Now, if I read you right, we'd have so save v1 even if we know
 Joe> that test() returns false.

All I meant is "treat abort() like a regular function that returns,
from the point of view of what state is saved.  If in this case it
means that normal GCC processing would mean "save v1" that's what it
means.  No special handling to save more than the usual -- but no
saving less than the usual either.

       paul

Re: Problem reading corefiles on ARM

[Sergei Poselenov]

Hello Daniel, all,


Daniel Jacobowitz wrote:
> On Wed, Aug 06, 2008 at 07:19:26PM +0400, Sergei Poselenov wrote:
>> (gdb) bt
>> #0  0x4004ec0c in raise () from /lib/libc.so.6
>> #1  0x40050234 in abort () from /lib/libc.so.6
>> Backtrace stopped: frame did not save the PC
> 
> Your implementation of abort does not save a return address, so GDB
> can't display it.  I believe tehis is a known limitation of the ARM
> GCC port.
> 

Thanks, I've got the point.
Indeed, changing abort() to invalid memory reference makes backtrace 
work.

How about threaded applications?

-bash-3.2# gdb linux-dp core.969
GNU gdb Red Hat Linux (6.7-1rh)
Copyright (C) 2007 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later 
<http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show 
copying"
and "show warranty" for details.
This GDB was configured as "arm-linux"...
Using host libthread_db library "/lib/libthread_db.so.1".
Reading symbols from /lib/libpthread.so.0...done.
Loaded symbols for /lib/libpthread.so.0
Reading symbols from /lib/libc.so.6...done.
Loaded symbols for /lib/libc.so.6
Reading symbols from /lib/ld-linux.so.3...done.
Loaded symbols for /lib/ld-linux.so.3
Core was generated by `./linux-dp'.
Program terminated with signal 11, Segmentation fault.
#0  0x400ce8e4 in nanosleep () from /lib/libc.so.6
(gdb) bt
#0  0x400ce8e4 in nanosleep () from /lib/libc.so.6
#1  0x400ce8d4 in nanosleep () from /lib/libc.so.6
Backtrace stopped: previous frame identical to this frame (corrupt 
stack?)
(gdb) info thr
   6 process 970  0x00000000 in ?? ()
   5 process 971  0x00000000 in ?? ()
   4 process 972  0x00000000 in ?? ()
   3 process 973  0x00000000 in ?? ()
   2 process 974  0x00000000 in ?? ()
* 1 process 969  0x400ce8e4 in nanosleep () from /lib/libc.so.6
(gdb) thr 3
[Switching to thread 3 (process 973)]#0  0x00000000 in ?? ()
(gdb) bt
#0  0x00000000 in ?? ()
#1  0x00000000 in ?? ()
(gdb)

This time, the core was generated by "kill -SIGSEGV".

Could it be a kernel bug?

Regards,
Sergei

Re: Problem reading corefiles on ARM

[Daniel Jacobowitz]

On Thu, Aug 07, 2008 at 01:29:34PM +0400, Sergei Poselenov wrote:
> This time, the core was generated by "kill -SIGSEGV".
>
> Could it be a kernel bug?

Yes, that's certainly a kernel bug.  I believe it is fixed in current
kernel.org kernels, but I'm not sure; it breaks periodically.

-- 
Daniel Jacobowitz
CodeSourcery