Debugging glibc crashes with GDB

Debugging glibc crashes with GDB

[Rich Rattanni]

Below you will find the GDB output related to my question.  My problem
is that when I try to debug a program after receiving a glibc abort
error (MALLOC_CHECK = 2) I cannot backtrace the program to find out
what piece of my application caused the problem.  I can see the call
to raise and abort, but then I get the error message "Backtrace
stopped:
frame did not save the PC".  Could this be due to code compiled with
the -fomit-frame-pointers options.  I am really pulling my hair out
over this one, any guidance or suggests would be vastly appreciated.
If further forensic info is required just ask.  Thanks in advance.
Note: I can switch the thread I am in and view other stack backtraces
with no problem.
-------------------------------------------------------------------------------------------
*** glibc detected *** /lifecor/shell: double free or corruption
(!prev): 0x000e8368 ***

Program received signal SIGABRT, Aborted.
[Switching to Thread 1224766608 (LWP 3149)]
0x4039aa24 in raise () from /lib/libc.so.6
(gdb) bt
#0  0x4039aa24 in raise () from /lib/libc.so.6
#1  0x4039bd5c in abort () from /lib/libc.so.6 Backtrace stopped:
frame did not save the PC
(gdb) info threads
 353 Thread 1241867408 (LWP 3482)  0x4035fed4 in
pthread_cond_timedwait@@GLIBC_2.4 () from /lib/libpthread.so.0
* 20 Thread 1224766608 (LWP 3149)  0x4039aa24 in raise () from
/lib/libc.so.6
 18 Thread 1206998160 (LWP 3147)  0x40362c54 in accept () from
/lib/libpthread.so.0
 17 Thread 1198609552 (LWP 3146)  0x4035fbdc in
pthread_cond_wait@@GLIBC_2.4 () from /lib/libpthread.so.0
 16 Thread 1189414032 (LWP 3145)  0x403629a4 in read () from
/lib/libpthread.so.0
 15 Thread 1180988560 (LWP 3142)  0x403629a4 in read () from
/lib/libpthread.so.0
 14 Thread 1172599952 (LWP 3141)  0x403629a4 in read () from
/lib/libpthread.so.0
 13 Thread 1163916432 (LWP 3140)  0x4035fbdc in
pthread_cond_wait@@GLIBC_2.4 () from /lib/libpthread.so.0
 12 Thread 1155527824 (LWP 3135)  0x403ee074 in nanosleep () from
/lib/libc.so.6
 11 Thread 1137185936 (LWP 3134)  0x4035fbdc in
pthread_cond_wait@@GLIBC_2.4 () from /lib/libpthread.so.0
 10 Thread 1145574544 (LWP 3133)  0x403ee074 in nanosleep () from
/lib/libc.so.6
 7 Thread 1128797328 (LWP 3123)  0x40404d7c in sched_yield () from
/lib/libc.so.6
 6 Thread 1120408720 (LWP 3122)  0x403ee074 in nanosleep () from
/lib/libc.so.6
 5 Thread 1112020112 (LWP 3121)  0x4035fed4 in
pthread_cond_timedwait@@GLIBC_2.4 () from /lib/libpthread.so.0
 4 Thread 1103631504 (LWP 3120)  0x4035fbdc in
pthread_cond_wait@@GLIBC_2.4 () from /lib/libpthread.so.0
 3 Thread 1095242896 (LWP 3119)  0x403629a4 in read () from
/lib/libpthread.so.0
 2 Thread 1086854288 (LWP 3118)  0x40363e90 in do_sigwait () from
/lib/libpthread.so.0
 1 Thread 1073871200 (LWP 3117)  0x403ee074 in nanosleep () from
/lib/libc.so.6
(gdb) quit

Re: Debugging glibc crashes with GDB

[Mark Kettenis]

> Date: Sun, 11 Mar 2007 14:04:41 -0400
> From: "Rich Rattanni" <rattanni@gmail.com>
> 
> Below you will find the GDB output related to my question.  My problem
> is that when I try to debug a program after receiving a glibc abort
> error (MALLOC_CHECK = 2) I cannot backtrace the program to find out
> what piece of my application caused the problem.  I can see the call
> to raise and abort, but then I get the error message "Backtrace
> stopped:
> frame did not save the PC".  Could this be due to code compiled with
> the -fomit-frame-pointers options.  I am really pulling my hair out
> over this one, any guidance or suggests would be vastly appreciated.
> If further forensic info is required just ask.  Thanks in advance.
> Note: I can switch the thread I am in and view other stack backtraces
> with no problem.

We need more information to help you.  What version of gdb are you
using?  On what machine architecture?  What version of glibc is this?
What flavour of Linux are you using?

Re: Debugging glibc crashes with GDB

[shanevolpe]

Mark,
gdb version: 6.6
glibc version: 2.5
ARCH: PXA270 (xscale)
Linux Version: 2.6.20 (Distro: OpenEmbedded)
Regards,
Shane (Co-employee with Rich)

On 3/11/07, Mark Kettenis <mark.kettenis@xs4all.nl> wrote:
> > Date: Sun, 11 Mar 2007 14:04:41 -0400
> > From: "Rich Rattanni" <rattanni@gmail.com>
> >
> > Below you will find the GDB output related to my question.  My problem
> > is that when I try to debug a program after receiving a glibc abort
> > error (MALLOC_CHECK = 2) I cannot backtrace the program to find out
> > what piece of my application caused the problem.  I can see the call
> > to raise and abort, but then I get the error message "Backtrace
> > stopped:
> > frame did not save the PC".  Could this be due to code compiled with
> > the -fomit-frame-pointers options.  I am really pulling my hair out
> > over this one, any guidance or suggests would be vastly appreciated.
> > If further forensic info is required just ask.  Thanks in advance.
> > Note: I can switch the thread I am in and view other stack backtraces
> > with no problem.
>
> We need more information to help you.  What version of gdb are you
> using?  On what machine architecture?  What version of glibc is this?
> What flavour of Linux are you using?
>


-- 
Registered Linux User: #293401

Re: Debugging glibc crashes with GDB

[Rich Rattanni]

Jan:
    The debug files for glibc are installed under /lib/.debug and when
I execute gdb I make sure to issue the following command, 'set
debug-file-directory /lib/.debug'.

On 3/11/07, shanevolpe@gmail.com <shanevolpe@gmail.com> wrote:
> Mark,
> gdb version: 6.6
> glibc version: 2.5
> ARCH: PXA270 (xscale)
> Linux Version: 2.6.20 (Distro: OpenEmbedded)
> Regards,
> Shane (Co-employee with Rich)
>
> On 3/11/07, Mark Kettenis <mark.kettenis@xs4all.nl> wrote:
> > > Date: Sun, 11 Mar 2007 14:04:41 -0400
> > > From: "Rich Rattanni" <rattanni@gmail.com>
> > >
> > > Below you will find the GDB output related to my question.  My problem
> > > is that when I try to debug a program after receiving a glibc abort
> > > error (MALLOC_CHECK = 2) I cannot backtrace the program to find out
> > > what piece of my application caused the problem.  I can see the call
> > > to raise and abort, but then I get the error message "Backtrace
> > > stopped:
> > > frame did not save the PC".  Could this be due to code compiled with
> > > the -fomit-frame-pointers options.  I am really pulling my hair out
> > > over this one, any guidance or suggests would be vastly appreciated.
> > > If further forensic info is required just ask.  Thanks in advance.
> > > Note: I can switch the thread I am in and view other stack backtraces
> > > with no problem.
> >
> > We need more information to help you.  What version of gdb are you
> > using?  On what machine architecture?  What version of glibc is this?
> > What flavour of Linux are you using?
> >
>
>
> --
> Registered Linux User: #293401
>

Re: Debugging glibc crashes with GDB

[Daniel Jacobowitz]

On Sun, Mar 11, 2007 at 02:04:41PM -0400, Rich Rattanni wrote:
> Below you will find the GDB output related to my question.  My problem
> is that when I try to debug a program after receiving a glibc abort
> error (MALLOC_CHECK = 2) I cannot backtrace the program to find out
> what piece of my application caused the problem.  I can see the call
> to raise and abort, but then I get the error message "Backtrace
> stopped:
> frame did not save the PC".

I believe this is a known problem with the ARM port of GCC.  What GCC
version are you using to build glibc?

It doesn't save the return address anywhere when it reaches abort;
this is a somewhat limited value optimization.

-- 
Daniel Jacobowitz
CodeSourcery

Re: Debugging glibc crashes with GDB

[Mark Kettenis]

> Date: Sun, 11 Mar 2007 21:15:02 -0400
> From: Daniel Jacobowitz <drow@false.org>
> 
> On Sun, Mar 11, 2007 at 02:04:41PM -0400, Rich Rattanni wrote:
> > Below you will find the GDB output related to my question.  My problem
> > is that when I try to debug a program after receiving a glibc abort
> > error (MALLOC_CHECK = 2) I cannot backtrace the program to find out
> > what piece of my application caused the problem.  I can see the call
> > to raise and abort, but then I get the error message "Backtrace
> > stopped:
> > frame did not save the PC".
> 
> I believe this is a known problem with the ARM port of GCC.  What GCC
> version are you using to build glibc?
> 
> It doesn't save the return address anywhere when it reaches abort;
> this is a somewhat limited value optimization.

It's a completely stupid optimization.  The only reason to call
abort() is to make your program crash.  And in almost any case you
want to look at the core dump afterwards.

Someone who cares enough about this should talk to the GCC and/or
glibc people to make sure backtraces through abort() work.

Mark

Re: Debugging glibc crashes with GDB

[Daniel Jacobowitz]

On Mon, Mar 12, 2007 at 09:29:40AM +0100, Mark Kettenis wrote:
> Someone who cares enough about this should talk to the GCC and/or
> glibc people to make sure backtraces through abort() work.

Working on it.

-- 
Daniel Jacobowitz
CodeSourcery

Re: Debugging glibc crashes with GDB

[Michael Snyder]

On Mon, 2007-03-12 at 07:03 -0400, Daniel Jacobowitz wrote:
> On Mon, Mar 12, 2007 at 09:29:40AM +0100, Mark Kettenis wrote:
> > Someone who cares enough about this should talk to the GCC and/or
> > glibc people to make sure backtraces through abort() work.
> 
> Working on it.

Might a person work around this, in the mean time, 
by supplying his own version of abort?

Re: Debugging glibc crashes with GDB

[Daniel Jacobowitz]

On Mon, Mar 12, 2007 at 04:59:20PM -0700, Michael Snyder wrote:
> On Mon, 2007-03-12 at 07:03 -0400, Daniel Jacobowitz wrote:
> > On Mon, Mar 12, 2007 at 09:29:40AM +0100, Mark Kettenis wrote:
> > > Someone who cares enough about this should talk to the GCC and/or
> > > glibc people to make sure backtraces through abort() work.
> > 
> > Working on it.
> 
> Might a person work around this, in the mean time, 
> by supplying his own version of abort?

Yes, if you can build one that GDB is happy with.  GCC detects that it
is declared noreturn, and/or doesn't return.

-- 
Daniel Jacobowitz
CodeSourcery