stack corruption?

stack corruption?

[Louis LeBlanc]

This might be a bit of a trivial question, but I am having trouble
with stack corruption.

I started seeing this after I began using gcc 3.3 with the -ggdb -g3
flags to try to provide more detailed info for gdb.  Can anyone tell
me how these flags change the executable?

Without the -ggdb switch, I don't get any debugging symbols at all.

Anyone have any ideas what else I should be looking at?

TIA
Lou
-- 
Louis LeBlanc                                     dev@keyslapper.net
Fully Funded Hobbyist,                   KeySlapper Extrordinaire :þ
http://www.keyslapper.net                                       Ô¿Ô¬
Key fingerprint = C5E7 4762 F071 CE3B ED51  4FB8 AF85 A2FE 80C8 D9A2

Ginsburg's Law:
  At the precise moment you take off your shoe in a shoe store, your
  big toe will pop out of your sock to see what's going on.

Re: stack corruption?

[Daniel Jacobowitz]

On Mon, Jun 13, 2005 at 09:45:20PM -0400, Louis LeBlanc wrote:
> This might be a bit of a trivial question, but I am having trouble
> with stack corruption.
> 
> I started seeing this after I began using gcc 3.3 with the -ggdb -g3
> flags to try to provide more detailed info for gdb.  Can anyone tell
> me how these flags change the executable?
> 
> Without the -ggdb switch, I don't get any debugging symbols at all.
> 
> Anyone have any ideas what else I should be looking at?

Stack corruption in your program, or in GDB?  Neither should change the
generated code at all.  -g3 adds macro debugging information.

-- 
Daniel Jacobowitz
CodeSourcery, LLC

Re: stack corruption?

[Louis LeBlanc]

On 06/13/05 10:01 PM, Daniel Jacobowitz sat at the `puter and typed:
> On Mon, Jun 13, 2005 at 09:45:20PM -0400, Louis LeBlanc wrote:
> > This might be a bit of a trivial question, but I am having trouble
> > with stack corruption.
> > 
> > I started seeing this after I began using gcc 3.3 with the -ggdb -g3
> > flags to try to provide more detailed info for gdb.  Can anyone tell
> > me how these flags change the executable?
> > 
> > Without the -ggdb switch, I don't get any debugging symbols at all.
> > 
> > Anyone have any ideas what else I should be looking at?
> 
> Stack corruption in your program, or in GDB?  Neither should change the
> generated code at all.  -g3 adds macro debugging information.

Stack corruption shows up anytime I open the program in gdb.  I get an
occasional core dump (SEGV and BUS) from my program, but when I can
make heads or tails of it, it appears to be deep in a host lookup
(system libs, Solaris 9) or Oracle 8.1.7 database calls (which could
also be a host lookup).

Thanks.

Lou
-- 
Louis LeBlanc                                     dev@keyslapper.net
Fully Funded Hobbyist,                   KeySlapper Extrordinaire :þ
http://www.keyslapper.net                                       Ô¿Ô¬
Key fingerprint = C5E7 4762 F071 CE3B ED51  4FB8 AF85 A2FE 80C8 D9A2

Arbitrary systems, pl.n.:
  Systems about which nothing general can be said, save "nothing
  general can be said."

Re: stack corruption?

[Jason Molenda]

On Jun 13, 2005, at 8:32 PM, Louis LeBlanc wrote:

> Stack corruption shows up anytime I open the program in gdb.  I get an
> occasional core dump (SEGV and BUS) from my program, but when I can
> make heads or tails of it, it appears to be deep in a host lookup
> (system libs, Solaris 9) or Oracle 8.1.7 database calls (which could
> also be a host lookup).


Louis, I think you're going to need to be a lot more specific.   
You're using gdb on Solaris 9 with a SPARC CPU?  What version of  
gdb?  Your paragraph above sounds like there's a bug in your program  
causing it to crash, and you cannot understand the stack backtrace --  
are you asking for help in using gdb to find your bug, or are you  
saying that gdb is not behaving properly?

If it's the latter, some example output of the problem would do  
wonders to clear things up.

Jason

Re: stack corruption?

[Louis LeBlanc]

On 06/14/05 01:35 PM, Jason Molenda sat at the `puter and typed:
> 
> On Jun 13, 2005, at 8:32 PM, Louis LeBlanc wrote:
> 
> > Stack corruption shows up anytime I open the program in gdb.  I get an
> > occasional core dump (SEGV and BUS) from my program, but when I can
> > make heads or tails of it, it appears to be deep in a host lookup
> > (system libs, Solaris 9) or Oracle 8.1.7 database calls (which could
> > also be a host lookup).
> 
> 
> Louis, I think you're going to need to be a lot more specific.   
> You're using gdb on Solaris 9 with a SPARC CPU?  What version of  
> gdb?  Your paragraph above sounds like there's a bug in your program  
> causing it to crash, and you cannot understand the stack backtrace --  
> are you asking for help in using gdb to find your bug, or are you  
> saying that gdb is not behaving properly?
> 
> If it's the latter, some example output of the problem would do  
> wonders to clear things up.

Sorry for being too general.  I am on a SPARC, running Solaris 8 and 9
on different machines.  Whether there's a bug in my code, well, I'm
sure there are minor bugs, but I'm not convinced yet that there's a
serious one.  This process runs at high network volumes for days and
weeks at a time without problem, then suddenly it dumps.  When I try
to examine the core, I get stack corruption messages.

Basically, I'm not entirely sure.  The codebase in question has never
had this problem in the past, but now (with a newer version of gdb,
and after much work on my code) I always get the corrupt stack notice
when I open the process in gdb.

The gdb version I am using is:
$ gdb -v
GNU gdb 6.3.0.20050516-cvs

I have been planning to get a later snapshot, but haven't gotten
around to it yet.

BTW, I've seen problems in several applications while performing
network lookups, not just my code, and not just C programs.  I got a
couple created by Perl scripts a couple weeks ago and they were also
in the middle of network lookups.  This makes me skeptical of the
system libs, not my code and not gcc.  The real problem is that gdb
tells me nothing useful in these cases, unless the corrupt stack
message is a real issue with my code.

BTW, I have always used the standard pstack utility on all cores I've
had to examine on Solaris - they don't always match up directly with
the gdb backtrace, but they're usually close.  Thread numbers and LWP
ids are sometimes way off too.

Lou
-- 
Louis LeBlanc                                     dev@keyslapper.net
Fully Funded Hobbyist,                   KeySlapper Extrordinaire :þ
http://www.keyslapper.net                                       Ô¿Ô¬
Key fingerprint = C5E7 4762 F071 CE3B ED51  4FB8 AF85 A2FE 80C8 D9A2

Facts are stubborn, but statistics are more pliable.

Re: stack corruption?

[Mark Kettenis]

   Date: Tue, 14 Jun 2005 16:52:12 -0400
   From: Louis LeBlanc <leblanc@keyslapper.net>
   BTW, I've seen problems in several applications while performing
   network lookups, not just my code, and not just C programs.  I got a
   couple created by Perl scripts a couple weeks ago and they were also
   in the middle of network lookups.  This makes me skeptical of the
   system libs, not my code and not gcc.  The real problem is that gdb
   tells me nothing useful in these cases, unless the corrupt stack
   message is a real issue with my code.

But unless you're going to send us a transcript of your debug session
with the *exact* messages, we won't be able to help you.

Mark

Re: stack corruption?

[Louis LeBlanc]

On 06/14/05 11:24 PM, Mark Kettenis sat at the `puter and typed:
>    Date: Tue, 14 Jun 2005 16:52:12 -0400
>    From: Louis LeBlanc <leblanc@keyslapper.net>
>    BTW, I've seen problems in several applications while performing
>    network lookups, not just my code, and not just C programs.  I got a
>    couple created by Perl scripts a couple weeks ago and they were also
>    in the middle of network lookups.  This makes me skeptical of the
>    system libs, not my code and not gcc.  The real problem is that gdb
>    tells me nothing useful in these cases, unless the corrupt stack
>    message is a real issue with my code.
> 
> But unless you're going to send us a transcript of your debug session
> with the *exact* messages, we won't be able to help you.


Ok, here we go.

$ gdb myprog cores/myprog.core.22367.1117114569
GNU gdb 6.3.0.20050516-cvs
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "sparc-sun-solaris2.8"...
Core was generated by `myprog -c /opt/progHome/conf/myprog.conf'.
Program terminated with signal 11, Segmentation fault.
Reading symbols from /opt/oracle/8.1.7/lib/libclntsh.so.8.0...done.
Loaded symbols for /opt/oracle/8.1.7/lib/libclntsh.so.8.0
Reading symbols from /usr/lib/libnsl.so.1...done.
Loaded symbols for /usr/lib/libnsl.so.1
Reading symbols from /usr/lib/libsocket.so.1...done.
Loaded symbols for /usr/lib/libsocket.so.1
Reading symbols from /usr/lib/lwp/libthread.so.1...done.
Loaded symbols for /usr/lib/lwp/libthread.so.1
Reading symbols from /usr/lib/librt.so.1...done.
Loaded symbols for /usr/lib/librt.so.1
Reading symbols from /usr/lib/libresolv.so.2...done.
Loaded symbols for /usr/lib/libresolv.so.2
Reading symbols from /usr/lib/libc.so.1...done.
Loaded symbols for /usr/lib/libc.so.1
Reading symbols from /opt/oracle/8.1.7/lib/libwtc8.so...done.
Loaded symbols for /opt/oracle/8.1.7/lib/libwtc8.so
Reading symbols from /usr/lib/libgen.so.1...done.
Loaded symbols for /usr/lib/libgen.so.1
Reading symbols from /usr/lib/libdl.so.1...done.
Loaded symbols for /usr/lib/libdl.so.1
Reading symbols from /usr/lib/libsched.so.1...done.
Loaded symbols for /usr/lib/libsched.so.1
Reading symbols from /usr/lib/libaio.so.1...done.
Loaded symbols for /usr/lib/libaio.so.1
Reading symbols from /usr/lib/libm.so.1...done.
Loaded symbols for /usr/lib/libm.so.1
Reading symbols from /usr/lib/libmp.so.2...done.
Loaded symbols for /usr/lib/libmp.so.2
Reading symbols from /usr/platform/sun4u/lib/libc_psr.so.1...done.
Loaded symbols for /usr/platform/SUNW,UltraSPARC-IIi-cEngine/lib/libc_psr.so.1
Reading symbols from /usr/lib/nss_files.so.1...done.
Loaded symbols for /usr/lib/nss_files.so.1
Reading symbols from /usr/lib/nss_dns.so.1...done.
Loaded symbols for /usr/lib/nss_dns.so.1
#0  0xff024044 in ttcpip () from /opt/oracle/8.1.7/lib/libclntsh.so.8.0
(gdb) bt
#0  0xff024044 in ttcpip () from /opt/oracle/8.1.7/lib/libclntsh.so.8.0
#1  0xff000470 in ttcdrv () from /opt/oracle/8.1.7/lib/libclntsh.so.8.0
#2  0xfeeb81e0 in nioqwa () from /opt/oracle/8.1.7/lib/libclntsh.so.8.0
#3  0xfed3b5cc in upirtrc () from /opt/oracle/8.1.7/lib/libclntsh.so.8.0
#4  0xfed8b64c in kpurcsc () from /opt/oracle/8.1.7/lib/libclntsh.so.8.0
#5  0xfed5b104 in kpuexecv8 () from /opt/oracle/8.1.7/lib/libclntsh.so.8.0
#6  0xfed5d2d4 in kpuexec () from /opt/oracle/8.1.7/lib/libclntsh.so.8.0
#7  0xfed9a318 in OCIStmtExecute () from /opt/oracle/8.1.7/lib/libclntsh.so.8.0
#8  0x000e5234 in execute_statement (connID=1649752, stmntID=1, args=0xf827a650, dbTimingRec=0xf8279b80, inOpCode=19541) at dbase.c:873
#9  0x0002e9f4 in handle_check (thread=0x759b70) at myprog.c:904
#10 0x000e3134 in spawn (arg=0x759b70) at mythread.c:195
#11 0xff354978 in _lwp_start () from /usr/lib/lwp/libthread.so.1
#12 0xff354978 in _lwp_start () from /usr/lib/lwp/libthread.so.1
Previous frame identical to this frame (corrupt stack?)
(gdb) 

I have changed identifying names to protect the guilty, but this is
pretty much typical.

Thank you
Lou
-- 
Louis LeBlanc                                     dev@keyslapper.net
Fully Funded Hobbyist,                   KeySlapper Extrordinaire :þ
http://www.keyslapper.net                                       Ô¿Ô¬
Key fingerprint = C5E7 4762 F071 CE3B ED51  4FB8 AF85 A2FE 80C8 D9A2

If they can make penicillin out of moldy bread, they can sure make
something out of you.
    -- Muhammad Ali

Re: stack corruption?

[Jason Molenda]

On Jun 14, 2005, at 2:31 PM, Louis LeBlanc wrote:

> #9  0x0002e9f4 in handle_check (thread=0x759b70) at myprog.c:904
> #10 0x000e3134 in spawn (arg=0x759b70) at mythread.c:195
> #11 0xff354978 in _lwp_start () from /usr/lib/lwp/libthread.so.1
> #12 0xff354978 in _lwp_start () from /usr/lib/lwp/libthread.so.1
> Previous frame identical to this frame (corrupt stack?)
> (gdb)
>
> I have changed identifying names to protect the guilty, but this is
> pretty much typical.


In this particular example, the problem is that gdb didn't correctly  
recognize when to stop the backtrace.  While lame, were you expecting  
something to be after _lwp_start in your stack backtrace?


Jason

Re: stack corruption?

[Mark Kettenis]

   Date: Tue, 14 Jun 2005 17:31:42 -0400
   From: Louis LeBlanc <dev@keyslapper.net>

   On 06/14/05 11:24 PM, Mark Kettenis sat at the `puter and typed:
   >    Date: Tue, 14 Jun 2005 16:52:12 -0400
   >    From: Louis LeBlanc <leblanc@keyslapper.net>
   >    BTW, I've seen problems in several applications while performing
   >    network lookups, not just my code, and not just C programs.  I got a
   >    couple created by Perl scripts a couple weeks ago and they were also
   >    in the middle of network lookups.  This makes me skeptical of the
   >    system libs, not my code and not gcc.  The real problem is that gdb
   >    tells me nothing useful in these cases, unless the corrupt stack
   >    message is a real issue with my code.
   > 
   > But unless you're going to send us a transcript of your debug session
   > with the *exact* messages, we won't be able to help you.


   Ok, here we go.

   $ gdb myprog cores/myprog.core.22367.1117114569
   GNU gdb 6.3.0.20050516-cvs
   Copyright 2004 Free Software Foundation, Inc.
   GDB is free software, covered by the GNU General Public License, and you are
   welcome to change it and/or distribute copies of it under certain conditions.
   Type "show copying" to see the conditions.
   There is absolutely no warranty for GDB.  Type "show warranty" for details.
   This GDB was configured as "sparc-sun-solaris2.8"...
   Core was generated by `myprog -c /opt/progHome/conf/myprog.conf'.
   Program terminated with signal 11, Segmentation fault.
   Reading symbols from /opt/oracle/8.1.7/lib/libclntsh.so.8.0...done.
   Loaded symbols for /opt/oracle/8.1.7/lib/libclntsh.so.8.0
   Reading symbols from /usr/lib/libnsl.so.1...done.
   Loaded symbols for /usr/lib/libnsl.so.1
   Reading symbols from /usr/lib/libsocket.so.1...done.
   Loaded symbols for /usr/lib/libsocket.so.1
   Reading symbols from /usr/lib/lwp/libthread.so.1...done.
   Loaded symbols for /usr/lib/lwp/libthread.so.1
   Reading symbols from /usr/lib/librt.so.1...done.
   Loaded symbols for /usr/lib/librt.so.1
   Reading symbols from /usr/lib/libresolv.so.2...done.
   Loaded symbols for /usr/lib/libresolv.so.2
   Reading symbols from /usr/lib/libc.so.1...done.
   Loaded symbols for /usr/lib/libc.so.1
   Reading symbols from /opt/oracle/8.1.7/lib/libwtc8.so...done.
   Loaded symbols for /opt/oracle/8.1.7/lib/libwtc8.so
   Reading symbols from /usr/lib/libgen.so.1...done.
   Loaded symbols for /usr/lib/libgen.so.1
   Reading symbols from /usr/lib/libdl.so.1...done.
   Loaded symbols for /usr/lib/libdl.so.1
   Reading symbols from /usr/lib/libsched.so.1...done.
   Loaded symbols for /usr/lib/libsched.so.1
   Reading symbols from /usr/lib/libaio.so.1...done.
   Loaded symbols for /usr/lib/libaio.so.1
   Reading symbols from /usr/lib/libm.so.1...done.
   Loaded symbols for /usr/lib/libm.so.1
   Reading symbols from /usr/lib/libmp.so.2...done.
   Loaded symbols for /usr/lib/libmp.so.2
   Reading symbols from /usr/platform/sun4u/lib/libc_psr.so.1...done.
   Loaded symbols for /usr/platform/SUNW,UltraSPARC-IIi-cEngine/lib/libc_psr.so.1
   Reading symbols from /usr/lib/nss_files.so.1...done.
   Loaded symbols for /usr/lib/nss_files.so.1
   Reading symbols from /usr/lib/nss_dns.so.1...done.
   Loaded symbols for /usr/lib/nss_dns.so.1
   #0  0xff024044 in ttcpip () from /opt/oracle/8.1.7/lib/libclntsh.so.8.0
   (gdb) bt
   #0  0xff024044 in ttcpip () from /opt/oracle/8.1.7/lib/libclntsh.so.8.0
   #1  0xff000470 in ttcdrv () from /opt/oracle/8.1.7/lib/libclntsh.so.8.0
   #2  0xfeeb81e0 in nioqwa () from /opt/oracle/8.1.7/lib/libclntsh.so.8.0
   #3  0xfed3b5cc in upirtrc () from /opt/oracle/8.1.7/lib/libclntsh.so.8.0
   #4  0xfed8b64c in kpurcsc () from /opt/oracle/8.1.7/lib/libclntsh.so.8.0
   #5  0xfed5b104 in kpuexecv8 () from /opt/oracle/8.1.7/lib/libclntsh.so.8.0
   #6  0xfed5d2d4 in kpuexec () from /opt/oracle/8.1.7/lib/libclntsh.so.8.0
   #7  0xfed9a318 in OCIStmtExecute () from /opt/oracle/8.1.7/lib/libclntsh.so.8.0
   #8  0x000e5234 in execute_statement (connID=1649752, stmntID=1, args=0xf827a650, dbTimingRec=0xf8279b80, inOpCode=19541) at dbase.c:873
   #9  0x0002e9f4 in handle_check (thread=0x759b70) at myprog.c:904
   #10 0x000e3134 in spawn (arg=0x759b70) at mythread.c:195
   #11 0xff354978 in _lwp_start () from /usr/lib/lwp/libthread.so.1
   #12 0xff354978 in _lwp_start () from /usr/lib/lwp/libthread.so.1
   Previous frame identical to this frame (corrupt stack?)
   (gdb) 

   I have changed identifying names to protect the guilty, but this is
   pretty much typical.

Oh, that's pretty harmless.  It's gdb falling off the end of the
stack.  Ideally gdb would now that it reached the end of the stack,
and not print the message.  But I haven't found a reliable way to do
so for Solaris SPARC. But then I haven't really tried very hard.

Mark

Re: stack corruption?

[Louis LeBlanc]

On 06/14/05 11:44 PM, Mark Kettenis sat at the `puter and typed:
> <SNIP>
>    #0  0xff024044 in ttcpip () from /opt/oracle/8.1.7/lib/libclntsh.so.8.0
>    (gdb) bt
>    #0  0xff024044 in ttcpip () from /opt/oracle/8.1.7/lib/libclntsh.so.8.0
>    #1  0xff000470 in ttcdrv () from /opt/oracle/8.1.7/lib/libclntsh.so.8.0
>    #2  0xfeeb81e0 in nioqwa () from /opt/oracle/8.1.7/lib/libclntsh.so.8.0
>    #3  0xfed3b5cc in upirtrc () from /opt/oracle/8.1.7/lib/libclntsh.so.8.0
>    #4  0xfed8b64c in kpurcsc () from /opt/oracle/8.1.7/lib/libclntsh.so.8.0
>    #5  0xfed5b104 in kpuexecv8 () from /opt/oracle/8.1.7/lib/libclntsh.so.8.0
>    #6  0xfed5d2d4 in kpuexec () from /opt/oracle/8.1.7/lib/libclntsh.so.8.0
>    #7  0xfed9a318 in OCIStmtExecute () from /opt/oracle/8.1.7/lib/libclntsh.so.8.0
>    #8  0x000e5234 in execute_statement (connID=1649752, stmntID=1, args=0xf827a650, dbTimingRec=0xf8279b80, inOpCode=19541) at dbase.c:873
>    #9  0x0002e9f4 in handle_check (thread=0x759b70) at myprog.c:904
>    #10 0x000e3134 in spawn (arg=0x759b70) at mythread.c:195
>    #11 0xff354978 in _lwp_start () from /usr/lib/lwp/libthread.so.1
>    #12 0xff354978 in _lwp_start () from /usr/lib/lwp/libthread.so.1
>    Previous frame identical to this frame (corrupt stack?)
>    (gdb) 
> 
>    I have changed identifying names to protect the guilty, but this is
>    pretty much typical.
> 
> Oh, that's pretty harmless.  It's gdb falling off the end of the
> stack.  Ideally gdb would now that it reached the end of the stack,
> and not print the message.  But I haven't found a reliable way to do
> so for Solaris SPARC. But then I haven't really tried very hard.

Ah.  Well,  I'm glad this isn't something wrong with my code.  Problem
is it makes applying some commands to all threads difficult.

Thanks for the clarification.

Lou
-- 
Louis LeBlanc                                     dev@keyslapper.net
Fully Funded Hobbyist,                   KeySlapper Extrordinaire :þ
http://www.keyslapper.net                                       Ô¿Ô¬
Key fingerprint = C5E7 4762 F071 CE3B ED51  4FB8 AF85 A2FE 80C8 D9A2

Boren's Laws:
  (1) When in charge, ponder.
  (2) When in trouble, delegate.
  (3) When in doubt, mumble.

Re: stack corruption?

[Daniel Jacobowitz]

On Tue, Jun 14, 2005 at 06:09:19PM -0400, Louis LeBlanc wrote:
> Ah.  Well,  I'm glad this isn't something wrong with my code.  Problem
> is it makes applying some commands to all threads difficult.

In the CVS version of GDB this won't be a problem; backtrace errors are
non-fatal.

-- 
Daniel Jacobowitz
CodeSourcery, LLC

Re: stack corruption?

[Jason Molenda]

On Jun 14, 2005, at 1:52 PM, Louis LeBlanc wrote:

> Basically, I'm not entirely sure.  The codebase in question has never
> had this problem in the past, but now (with a newer version of gdb,
> and after much work on my code) I always get the corrupt stack notice
> when I open the process in gdb.

You mean you're doing a backtrace and mid-way through you get an  
error message like

Previous frame identical to this frame (corrupt stack?)

and the backtrace stops at that point?  These days, the majority of  
the time, this is gdb getting confused.  Your program don't  
necessarily have a corrupt stack.

> The gdb version I am using is:
> $ gdb -v
> GNU gdb 6.3.0.20050516-cvs
>
> I have been planning to get a later snapshot, but haven't gotten
> around to it yet.

In addition, I'd recommend grabbing an older gdb and give that a try,  
like 6.0.  There was some work done in the frame handling code that  
changed the way stack backtraces are done, and it gives up too  
easily.  I don't remember when the new frame code went in, but  
gdb-6.0 is probably early enough.  Downloading & building gdb doesn't  
take much time; it's worth doing this as a sanity check.

Jason