Re: gdb stack trace unreadable for a pthreads application

[x x <bapluda@yahoo.com.br>]

Paul,

Thank you for your reply.
I am building and running mfsrv exactly from the same machine and same directory. And I am analyzing the core also from the same machine and same user account. I am running on a chrooted system. Could this be the cause of the path mismatch? How can I view which libraries it is using when running, and make sure that gdb uses the same?

Ian



> De: Paul Pluzhnikov <ppluzhnikov@google.com>
> Assunto: Re: gdb stack trace unreadable for a pthreads application
> Para: "x x" <bapluda@yahoo.com.br>
> Cc: gdb@sourceware.org
> Data: Segunda-feira, 27 de Abril de 2009, 1:25
> On Sat, Apr 25, 2009 at 6:56 AM, x x
> <bapluda@yahoo.com.br>
> wrote:
> 
> > I have this server app that creates a pool of 24
> threads. I had a core dump the other day and tried to
> investigate the cause.
> 
> You didn't say whether SIGSEGV happened on the same host on
> which you
> are trying to analyze the core, but all clues indicate that
> it didn't.
> 
> > I can view the stack trace of the thread that caused
> the segmentation fault, but I cannot view the stack trace of
> the other threads.
> 
> This is a typical symptom of libc.so.6 mismatch between
> when core was
> produced (on target?) and when it is being analyzed (on
> host).
> 
> > The stack trace of the other threads is important for
> me to determine the cause of the crash. Except for thread 2
> these other threads are blocked inside a pthread_mutex_lock
> call. But the stack trace doesn't make any sense.
> > Please view the gdb output in the bottom.
> > Should I upgrade my gdb version? Can you help?
> >
> > Thanks
> >
> > Ian
> >
> >
> >
> > $ uname -a
> > Linux ds5632 2.6.22-8-server #1 SMP Thu Jul 12
> 16:28:57 GMT 2007 i686 GNU/Linux
> >
> > $lsb_release -a
> > No LSB modules are available.
> > Distributor ID: Ubuntu
> > Description:    Ubuntu 6.06 LTS
> > Release:        6.06
> > Codename:       dapper
> >
> > $gdb mfsrv core.31589
> >
> > GNU gdb 6.4-debian
> > Copyright 2005 Free Software Foundation, Inc.
> > GDB is free software, covered by the GNU General
> Public License, and you are
> > welcome to change it and/or distribute copies of it
> under certain conditions.
> > Type "show copying" to see the conditions.
> > There is absolutely no warranty for GDB.  Type "show
> warranty" for details.
> > This GDB was configured as "i486-linux-gnu"...Using
> host libthread_db library
> "/lib/tls/i686/cmov/libthread_db.so.1".
> >
> 
> So is this:
> 
> > Failed to read a valid object file image from memory.
> > Core was generated by `./mfsrv mf MundialFoot 1'.
> > Program terminated with signal 11, Segmentation
> fault.
> >
> 
> And this:
> 
> > warning: Can't read pathname for load map:
> Input/output error.
> > Reading symbols from /usr/lib/libstdc++.so.6...done.
> > Loaded symbols for /usr/lib/libstdc++.so.6
> > Reading symbols from
> /lib/tls/i686/cmov/libpthread.so.0...done.
> > Loaded symbols for /lib/tls/i686/cmov/libpthread.so.0
> > Reading symbols from
> /usr/lib/libmysqlclient_r.so.15...done.
> > Loaded symbols for /usr/lib/libmysqlclient_r.so.15
> > Reading symbols from /usr/lib/libz.so.1...done.
> > Loaded symbols for /usr/lib/libz.so.1
> > Reading symbols from
> /lib/tls/i686/cmov/libcrypt.so.1...done.
> > Loaded symbols for /lib/tls/i686/cmov/libcrypt.so.1
> > Reading symbols from
> /lib/tls/i686/cmov/libnsl.so.1...done.
> > Loaded symbols for /lib/tls/i686/cmov/libnsl.so.1
> > Reading symbols from
> /lib/tls/i686/cmov/libm.so.6...done.
> > Loaded symbols for /lib/tls/i686/cmov/libm.so.6
> > Reading symbols from /lib/libgcc_s.so.1...done.
> > Loaded symbols for /lib/libgcc_s.so.1
> > Reading symbols from
> /lib/tls/i686/cmov/libc.so.6...done.
> > Loaded symbols for /lib/tls/i686/cmov/libc.so.6
> > Reading symbols from /lib/ld-linux.so.2...done.
> > Loaded symbols for /lib/ld-linux.so.2
> > #0  0xb7b9c443 in memmove () from
> /lib/tls/i686/cmov/libc.so.6
> > (gdb) info threads
> 
> Threads 2 through 24 are blocked in a system call (in
> kernel VDSO page):
> 
> >  24 process 31589  0xffffe410 in ?? ()
> >  23 process 31591  0xffffe410 in ?? ()
> >  22 process 31592  0xffffe410 in ?? ()
> >  21 process 31593  0xffffe410 in ?? ()
> >  20 process 31594  0xffffe410 in ?? ()
> >  19 process 31595  0xffffe410 in ?? ()
> >  18 process 31600  0xffffe410 in ?? ()
> >  17 process 31601  0xffffe410 in ?? ()
> >  16 process 31602  0xffffe410 in ?? ()
> >  15 process 31603  0xffffe410 in ?? ()
> >  14 process 31608  0xffffe410 in ?? ()
> >  13 process 31609  0xffffe410 in ?? ()
> >  12 process 31610  0xffffe410 in ?? ()
> >  11 process 31611  0xffffe410 in ?? ()
> >  10 process 31616  0xffffe410 in ?? ()
> >  9 process 31617  0xffffe410 in ?? ()
> >  8 process 31618  0xffffe410 in ?? ()
> >  7 process 31619  0xffffe410 in ?? ()
> >  6 process 31624  0xffffe410 in ?? ()
> >  5 process 31625  0xffffe410 in ?? ()
> >  4 process 31626  0xffffe410 in ?? ()
> >  3 process 31632  0xffffe410 in ?? ()
> >  2 process 31633  0xffffe410 in ?? ()
> > * 1 process 31627  0xb7b9c443 in memmove () from
> /lib/tls/i686/cmov/libc.so.6
> > (gdb) bt 12
> > #0  0xb7b9c443 in memmove () from
> /lib/tls/i686/cmov/libc.so.6
> > #1  0x08066b62 in IE_Sock::Recv (this=0x8098268,
> pch=0x8098264 "", cbToRecv=4, iSecs=-1) at IE_Sock.cc:284
> > #2  0x08066d51 in IE_Sock::RecvInt (this=0x8098268,
> pi=0x8098264, iSecs=-1) at IE_Sock.cc:257
> > #3  0x08070b3f in Thread::InitConnection
> (this=0x8097dfc) at commands.cc:45
> > #4  0x08062d73 in Thread::Start (this=0x8097dfc) at
> SMthreads.cc:195
> > #5  0x08062ec7 in Thread::Thread_Start (pv=0x8097dfc)
> at SMthreads.cc:37
> > #6  0xb7e3f341 in start_thread () from
> /lib/tls/i686/cmov/libpthread.so.0
> > #7  0xb7bfa4ee in clone () from
> /lib/tls/i686/cmov/libc.so.6
> > (gdb) thread 2
> > [Switching to thread 2 (process 31633)]#0  0xffffe410
> in ?? ()
> > (gdb) bt 12
> 
> Typical stack trace when libc.so.6 and libpthread.so.0 are
> different
> at core dump and at analysis time:
> 
> > #0  0xffffe410 in ?? ()
> > #1  0xacaf8258 in ?? ()
> > #2  0xb7e493b4 in ?? () from
> /lib/tls/i686/cmov/libpthread.so.0
> > #3  0xacaf8220 in ?? ()
> > #4  0xb7e44778 in accept () from
> /lib/tls/i686/cmov/libpthread.so.0
> > #5  0x080661fd in IE_SockSrv::Accept (this=0x8099530,
> psock=0x8099100, sz=0x0, sl=0) at IE_Sock.cc:74
> > #6  0x0808039c in admin_thread::thread_start (pv=0x0)
> at admin_thread.cc:44
> > #7  0xb7e3f341 in start_thread () from
> /lib/tls/i686/cmov/libpthread.so.0
> > #8  0xb7bfa4ee in clone () from
> /lib/tls/i686/cmov/libc.so.6
> > (gdb) thread 3
> > [Switching to thread 3 (process 31632)]#0  0xffffe410
> in ?? ()
> > (gdb) bt 12
> > #0  0xffffe410 in ?? ()
> > #1  0xad2f940c in ?? ()
> > #2  0x00000021 in ?? ()
> > #3  0x00000000 in ?? ()
> >
> > and all remaining the threads (4 to 24) have the same
> screwed up stack
> >
> >
> >
> >      Veja quais são os assuntos do momento no
> Yahoo! +Buscados
> > http://br.maisbuscados.yahoo.com
> >
> 
> Assuming my guess of host/target mismatch is correct, the
> solution is:
> 
> gdb mfsrv
> (gdb) set solib-absolute-prefix /path/to/target/root
>     # libc.so.6, libpthread.so.0 should be in
> /path/to/target/root/lib/tls/i686/cmov
> (gdb) core core.31589
> 
> Cheers,
> -- 
> Paul Pluzhnikov
> 


      Veja quais são os assuntos do momento no Yahoo! +Buscados
http://br.maisbuscados.yahoo.com