backtrace when corrupt stack

backtrace when corrupt stack

[Cristian Zamfir]

Hi, 

I am trying to debug a bug that leads to a corrupt stack and I do not get an accurate backtrace with gdb. 

Is it possible to get an accurate backtrace in this case?

There is  "Backtrace stopped: previous frame inner to this frame (corrupt stack?)" indicates that the backtrace stopped later than the function I am interested in. 

I would like to get the function name and the arguments from frame 5 (see output below) --- the one called from the function Log()). The backtrace suggests this is __stack_chk_fail () from libc but that is not correct.

I tried this with gdb 7.0 and I also compiled gdb 7.0.1 but I get the same result.

This is the output:

*** stack smashing detected ***: /home/zamf/code/pin/source/tools/ddb/klee/examples/ghttpd-1.4-4/ghttpd terminated
======= Backtrace: =========
/lib/tls/i686/cmov/libc.so.6(__fortify_fail+0x48)[0x651de8]
/lib/tls/i686/cmov/libc.so.6(__fortify_fail+0x0)[0x651da0]
/home/zamf/code/pin/source/tools/ddb/klee/examples/ghttpd-1.4-4/ghttpd[0x804b4da]
[0x61616161]
======= Memory map: ========
00320000-0032a000 r-xp 00000000 08:41 131689     /lib/tls/i686/cmov/libnss_files-2.10.1.so
0032a000-0032b000 r--p 00009000 08:41 131689     /lib/tls/i686/cmov/libnss_files-2.10.1.so
0032b000-0032c000 rw-p 0000a000 08:41 131689     /lib/tls/i686/cmov/libnss_files-2.10.1.so
00571000-006af000 r-xp 00000000 08:41 131672     /lib/tls/i686/cmov/libc-2.10.1.so
006af000-006b1000 r--p 0013e000 08:41 131672     /lib/tls/i686/cmov/libc-2.10.1.so
006b1000-006b2000 rw-p 00140000 08:41 131672     /lib/tls/i686/cmov/libc-2.10.1.so
006b2000-006b5000 rw-p 00000000 00:00 0 
007ad000-007c0000 r-xp 00000000 08:41 131683     /lib/tls/i686/cmov/libnsl-2.10.1.so
007c0000-007c1000 r--p 00012000 08:41 131683     /lib/tls/i686/cmov/libnsl-2.10.1.so
007c1000-007c2000 rw-p 00013000 08:41 131683     /lib/tls/i686/cmov/libnsl-2.10.1.so
007c2000-007c4000 rw-p 00000000 00:00 0 
00895000-0089e000 r-xp 00000000 08:41 131693     /lib/tls/i686/cmov/libnss_nis-2.10.1.so
0089e000-0089f000 r--p 00008000 08:41 131693     /lib/tls/i686/cmov/libnss_nis-2.10.1.so
0089f000-008a0000 rw-p 00009000 08:41 131693     /lib/tls/i686/cmov/libnss_nis-2.10.1.so
008b1000-008cc000 r-xp 00000000 08:41 521        /lib/ld-2.10.1.so
008cc000-008cd000 r--p 0001a000 08:41 521        /lib/ld-2.10.1.so
008cd000-008ce000 rw-p 0001b000 08:41 521        /lib/ld-2.10.1.so
00a5c000-00a62000 r-xp 00000000 08:41 131685     /lib/tls/i686/cmov/libnss_compat-2.10.1.so
00a62000-00a63000 r--p 00005000 08:41 131685     /lib/tls/i686/cmov/libnss_compat-2.10.1.so
00a63000-00a64000 rw-p 00006000 08:41 131685     /lib/tls/i686/cmov/libnss_compat-2.10.1.so
00be9000-00c05000 r-xp 00000000 08:41 571        /lib/libgcc_s.so.1
00c05000-00c06000 r--p 0001b000 08:41 571        /lib/libgcc_s.so.1
00c06000-00c07000 rw-p 0001c000 08:41 571        /lib/libgcc_s.so.1
00c93000-00c94000 r-xp 00000000 00:00 0          [vdso]
08048000-0804c000 r-xp 00000000 08:43 657039     /home/zamf/code/pin/source/tools/ddb/klee/examples/ghttpd-1.4-4/ghttpd
0804c000-0804d000 r--p 00003000 08:43 657039     /home/zamf/code/pin/source/tools/ddb/klee/examples/ghttpd-1.4-4/ghttpd
0804d000-0804f000 rw-p 00004000 08:43 657039     /home/zamf/code/pin/source/tools/ddb/klee/examples/ghttpd-1.4-4/ghttpd
0804f000-08071000 rw-p 00000000 00:00 0          [heap]
b7feb000-b7fec000 rw-p 00000000 00:00 0 
b7ffd000-b8000000 rw-p 00000000 00:00 0 
bffeb000-c0000000 rw-p 00000000 00:00 0          [stack]

Program received signal SIGABRT, Aborted.
0x00c93422 in __kernel_vsyscall ()
(gdb) bt
#0  0x00c93422 in __kernel_vsyscall ()
#1  0x0059b4d1 in *__GI_raise (sig=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64
#2  0x0059e932 in *__GI_abort () at abort.c:92
#3  0x005d1ee5 in __libc_message (do_abort=2, fmt=0x69359d "*** %s ***: %s terminated\n") at ../sysdeps/unix/sysv/linux/libc_fatal.c:189
#4  0x00651de8 in *__GI___fortify_fail (msg=0x693585 "stack smashing detected") at fortify_fail.c:32
#5  0x00651da0 in __stack_chk_fail () at stack_chk_fail.c:29
#6  0x0804b4da in Log (format=0x804b8bc "Connection from %s, request = \"GET %s\"") at util.c:274
#7  0x61616161 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)


Thank you,
Cristi

Re: backtrace when corrupt stack

[Robert Dewar]

Cristian Zamfir wrote:
> Hi, 
> 
> I am trying to debug a bug that leads to a corrupt stack and I do not get an accurate backtrace with gdb. 
> 
> Is it possible to get an accurate backtrace in this case?

Not in general! The back trace works by interpreting the stack, if you 
clobber the stack, then you clobber the trace back. How badly? Well it
depends on how badly you clobber the stack!!!
> 
> There is  "Backtrace stopped: previous frame inner to this frame (corrupt stack?)" indicates that the backtrace stopped later than the function I am interested in. 
> 
> I would like to get the function name and the arguments from frame 5 (see output below) --- the one called from the function Log()). The backtrace suggests this is __stack_chk_fail () from libc but that is not correct.
> 
> I tried this with gdb 7.0 and I also compiled gdb 7.0.1 but I get the same result.
> 
> This is the output:
> 
> *** stack smashing detected ***: /home/zamf/code/pin/source/tools/ddb/klee/examples/ghttpd-1.4-4/ghttpd terminated
> ======= Backtrace: =========
> /lib/tls/i686/cmov/libc.so.6(__fortify_fail+0x48)[0x651de8]
> /lib/tls/i686/cmov/libc.so.6(__fortify_fail+0x0)[0x651da0]
> /home/zamf/code/pin/source/tools/ddb/klee/examples/ghttpd-1.4-4/ghttpd[0x804b4da]
> [0x61616161]
> ======= Memory map: ========
> 00320000-0032a000 r-xp 00000000 08:41 131689     /lib/tls/i686/cmov/libnss_files-2.10.1.so
> 0032a000-0032b000 r--p 00009000 08:41 131689     /lib/tls/i686/cmov/libnss_files-2.10.1.so
> 0032b000-0032c000 rw-p 0000a000 08:41 131689     /lib/tls/i686/cmov/libnss_files-2.10.1.so
> 00571000-006af000 r-xp 00000000 08:41 131672     /lib/tls/i686/cmov/libc-2.10.1.so
> 006af000-006b1000 r--p 0013e000 08:41 131672     /lib/tls/i686/cmov/libc-2.10.1.so
> 006b1000-006b2000 rw-p 00140000 08:41 131672     /lib/tls/i686/cmov/libc-2.10.1.so
> 006b2000-006b5000 rw-p 00000000 00:00 0 
> 007ad000-007c0000 r-xp 00000000 08:41 131683     /lib/tls/i686/cmov/libnsl-2.10.1.so
> 007c0000-007c1000 r--p 00012000 08:41 131683     /lib/tls/i686/cmov/libnsl-2.10.1.so
> 007c1000-007c2000 rw-p 00013000 08:41 131683     /lib/tls/i686/cmov/libnsl-2.10.1.so
> 007c2000-007c4000 rw-p 00000000 00:00 0 
> 00895000-0089e000 r-xp 00000000 08:41 131693     /lib/tls/i686/cmov/libnss_nis-2.10.1.so
> 0089e000-0089f000 r--p 00008000 08:41 131693     /lib/tls/i686/cmov/libnss_nis-2.10.1.so
> 0089f000-008a0000 rw-p 00009000 08:41 131693     /lib/tls/i686/cmov/libnss_nis-2.10.1.so
> 008b1000-008cc000 r-xp 00000000 08:41 521        /lib/ld-2.10.1.so
> 008cc000-008cd000 r--p 0001a000 08:41 521        /lib/ld-2.10.1.so
> 008cd000-008ce000 rw-p 0001b000 08:41 521        /lib/ld-2.10.1.so
> 00a5c000-00a62000 r-xp 00000000 08:41 131685     /lib/tls/i686/cmov/libnss_compat-2.10.1.so
> 00a62000-00a63000 r--p 00005000 08:41 131685     /lib/tls/i686/cmov/libnss_compat-2.10.1.so
> 00a63000-00a64000 rw-p 00006000 08:41 131685     /lib/tls/i686/cmov/libnss_compat-2.10.1.so
> 00be9000-00c05000 r-xp 00000000 08:41 571        /lib/libgcc_s.so.1
> 00c05000-00c06000 r--p 0001b000 08:41 571        /lib/libgcc_s.so.1
> 00c06000-00c07000 rw-p 0001c000 08:41 571        /lib/libgcc_s.so.1
> 00c93000-00c94000 r-xp 00000000 00:00 0          [vdso]
> 08048000-0804c000 r-xp 00000000 08:43 657039     /home/zamf/code/pin/source/tools/ddb/klee/examples/ghttpd-1.4-4/ghttpd
> 0804c000-0804d000 r--p 00003000 08:43 657039     /home/zamf/code/pin/source/tools/ddb/klee/examples/ghttpd-1.4-4/ghttpd
> 0804d000-0804f000 rw-p 00004000 08:43 657039     /home/zamf/code/pin/source/tools/ddb/klee/examples/ghttpd-1.4-4/ghttpd
> 0804f000-08071000 rw-p 00000000 00:00 0          [heap]
> b7feb000-b7fec000 rw-p 00000000 00:00 0 
> b7ffd000-b8000000 rw-p 00000000 00:00 0 
> bffeb000-c0000000 rw-p 00000000 00:00 0          [stack]
> 
> Program received signal SIGABRT, Aborted.
> 0x00c93422 in __kernel_vsyscall ()
> (gdb) bt
> #0  0x00c93422 in __kernel_vsyscall ()
> #1  0x0059b4d1 in *__GI_raise (sig=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64
> #2  0x0059e932 in *__GI_abort () at abort.c:92
> #3  0x005d1ee5 in __libc_message (do_abort=2, fmt=0x69359d "*** %s ***: %s terminated\n") at ../sysdeps/unix/sysv/linux/libc_fatal.c:189
> #4  0x00651de8 in *__GI___fortify_fail (msg=0x693585 "stack smashing detected") at fortify_fail.c:32
> #5  0x00651da0 in __stack_chk_fail () at stack_chk_fail.c:29
> #6  0x0804b4da in Log (format=0x804b8bc "Connection from %s, request = \"GET %s\"") at util.c:274
> #7  0x61616161 in ?? ()
> Backtrace stopped: previous frame inner to this frame (corrupt stack?)
> 
> 
> Thank you,
> Cristi

Re: backtrace when corrupt stack

[Tom Tromey]

>>>>> "Cristian" == Cristian Zamfir <cristian.zamfir@epfl.ch> writes:

Cristian> Is it possible to get an accurate backtrace in this case?

In addition to what Robert said, you may want to try valgrind instead of
gdb in a case like this.  That is usually the tool I try first when I
have this kind of bug.

Tom

Re: backtrace when corrupt stack

[Andreas Schwab]

Cristian Zamfir <cristian.zamfir@epfl.ch> writes:

> I would like to get the function name and the arguments from frame 5 (see output below) --- the one called from the function Log()). The backtrace suggests this is __stack_chk_fail () from libc but that is not correct.

Why do you think so?  That frame looks pretty reasonable.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: backtrace when corrupt stack

[Dave Korn]

On 05/02/2010 15:24, Cristian Zamfir wrote:

> #6  0x0804b4da in Log (format=0x804b8bc "Connection from %s, request = \"GET %s\"") at util.c:274
> #7  0x61616161 in ?? ()

  thx 4 ur 0-day ;-)

    cheers,
      DaveK

Re: backtrace when corrupt stack

[Cristian Zamfir]

On Feb 5, 2010, at 8:05 PM, Andreas Schwab wrote:
>> I would like to get the function name and the arguments from frame 5 (see output below) --- the one called from the function Log()). The backtrace suggests this is __stack_chk_fail () from libc but that is not correct.
> 
> Why do you think so?  That frame looks pretty reasonable.
> 
> Andreas.

I am pretty sure that the function in frame #5 should be vsprintf() which is passed a buffer with insufficiently allocated space. But on the stack I see __stack_chk_fail () instead of vsprintf (). 

My guess is that glibc detects a stack smashing and in the backtrace I see the abort handler instead of the call to vsprintf. Both vsprintf and Log have variable length arguments, but I am not sure that explains anything.  

Thanks,
Cristi  

Re: backtrace when corrupt stack

[Cristian Zamfir]

>> #6  0x0804b4da in Log (format=0x804b8bc "Connection from %s, request = \"GET %s\"") at util.c:274
>> #7  0x61616161 in ?? ()
> 
>  thx 4 ur 0-day ;-)

Welcome ;) It is in an older version of a web server and it is a known exploit, so I don't think it is too useful to anyone ;) --- I need to debug it for research purposes. 

Cristi

Re: backtrace when corrupt stack

[Andreas Schwab]

Cristian Zamfir <cristian.zamfir@epfl.ch> writes:

> I am pretty sure that the function in frame #5 should be vsprintf() which is passed a buffer with insufficiently allocated space. But on the stack I see __stack_chk_fail () instead of vsprintf (). 

vsprintf is expanded inline.  The real vsprintf is not called when the
fortify check fails.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: backtrace when corrupt stack

[Dave Korn]

On 05/02/2010 20:03, Cristian Zamfir wrote:
>>> #6  0x0804b4da in Log (format=0x804b8bc "Connection from %s, request =
>>> \"GET %s\"") at util.c:274 #7  0x61616161 in ?? ()
>> thx 4 ur 0-day ;-)
> 
> Welcome ;) It is in an older version of a web server and it is a known
> exploit, so I don't think it is too useful to anyone ;) --- I need to debug
> it for research purposes.

  Yeh, I checked, it's more like a 2673-day ;) anyway happy hacking!

    cheers,
      DaveK