From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (qmail 21287 invoked by alias); 27 Apr 2009 00:26:07 -0000 Received: (qmail 21275 invoked by uid 22791); 27 Apr 2009 00:26:03 -0000 X-SWARE-Spam-Status: No, hits=-1.8 required=5.0 tests=AWL,BAYES_00,SARE_MSGID_LONG40,SPF_PASS,WEIRD_PORT X-Spam-Check-By: sourceware.org Received: from smtp-out.google.com (HELO smtp-out.google.com) (216.239.45.13) by sourceware.org (qpsmtpd/0.43rc1) with ESMTP; Mon, 27 Apr 2009 00:25:57 +0000 Received: from spaceape13.eur.corp.google.com (spaceape13.eur.corp.google.com [172.28.16.147]) by smtp-out.google.com with ESMTP id n3R0PrsX031159 for ; Sun, 26 Apr 2009 17:25:54 -0700 Received: from qw-out-1920.google.com (qwj9.prod.google.com [10.241.195.73]) by spaceape13.eur.corp.google.com with ESMTP id n3R0PpdW028535 for ; Sun, 26 Apr 2009 17:25:51 -0700 Received: by qw-out-1920.google.com with SMTP id 9so1636677qwj.62 for ; Sun, 26 Apr 2009 17:25:51 -0700 (PDT) MIME-Version: 1.0 Received: by 10.229.74.71 with SMTP id t7mr2086223qcj.47.1240791951180; Sun, 26 Apr 2009 17:25:51 -0700 (PDT) In-Reply-To: <786195.79367.qm@web38107.mail.mud.yahoo.com> References: <786195.79367.qm@web38107.mail.mud.yahoo.com> Date: Mon, 27 Apr 2009 00:43:00 -0000 Message-ID: <8ac60eac0904261725r2983ff86j8360cd06e8cb45ad@mail.gmail.com> Subject: Re: gdb stack trace unreadable for a pthreads application From: Paul Pluzhnikov To: x x Cc: gdb@sourceware.org Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable X-System-Of-Record: true X-IsSubscribed: yes Mailing-List: contact gdb-help@sourceware.org; run by ezmlm Precedence: bulk List-Id: List-Subscribe: List-Archive: List-Post: List-Help: , Sender: gdb-owner@sourceware.org X-SW-Source: 2009-04/txt/msg00196.txt.bz2 On Sat, Apr 25, 2009 at 6:56 AM, x x wrote: > I have this server app that creates a pool of 24 threads. I had a core du= mp the other day and tried to investigate the cause. You didn't say whether SIGSEGV happened on the same host on which you are trying to analyze the core, but all clues indicate that it didn't. > I can view the stack trace of the thread that caused the segmentation fau= lt, but I cannot view the stack trace of the other threads. This is a typical symptom of libc.so.6 mismatch between when core was produced (on target?) and when it is being analyzed (on host). > The stack trace of the other threads is important for me to determine the= cause of the crash. Except for thread 2 these other threads are blocked in= side a pthread_mutex_lock call. But the stack trace doesn't make any sense. > Please view the gdb output in the bottom. > Should I upgrade my gdb version? Can you help? > > Thanks > > Ian > > > > $ uname -a > Linux ds5632 2.6.22-8-server #1 SMP Thu Jul 12 16:28:57 GMT 2007 i686 GNU= /Linux > > $lsb_release -a > No LSB modules are available. > Distributor ID: Ubuntu > Description: =A0 =A0Ubuntu 6.06 LTS > Release: =A0 =A0 =A0 =A06.06 > Codename: =A0 =A0 =A0 dapper > > $gdb mfsrv core.31589 > > GNU gdb 6.4-debian > Copyright 2005 Free Software Foundation, Inc. > GDB is free software, covered by the GNU General Public License, and you = are > welcome to change it and/or distribute copies of it under certain conditi= ons. > Type "show copying" to see the conditions. > There is absolutely no warranty for GDB. =A0Type "show warranty" for deta= ils. > This GDB was configured as "i486-linux-gnu"...Using host libthread_db lib= rary "/lib/tls/i686/cmov/libthread_db.so.1". > So is this: > Failed to read a valid object file image from memory. > Core was generated by `./mfsrv mf MundialFoot 1'. > Program terminated with signal 11, Segmentation fault. > And this: > warning: Can't read pathname for load map: Input/output error. > Reading symbols from /usr/lib/libstdc++.so.6...done. > Loaded symbols for /usr/lib/libstdc++.so.6 > Reading symbols from /lib/tls/i686/cmov/libpthread.so.0...done. > Loaded symbols for /lib/tls/i686/cmov/libpthread.so.0 > Reading symbols from /usr/lib/libmysqlclient_r.so.15...done. > Loaded symbols for /usr/lib/libmysqlclient_r.so.15 > Reading symbols from /usr/lib/libz.so.1...done. > Loaded symbols for /usr/lib/libz.so.1 > Reading symbols from /lib/tls/i686/cmov/libcrypt.so.1...done. > Loaded symbols for /lib/tls/i686/cmov/libcrypt.so.1 > Reading symbols from /lib/tls/i686/cmov/libnsl.so.1...done. > Loaded symbols for /lib/tls/i686/cmov/libnsl.so.1 > Reading symbols from /lib/tls/i686/cmov/libm.so.6...done. > Loaded symbols for /lib/tls/i686/cmov/libm.so.6 > Reading symbols from /lib/libgcc_s.so.1...done. > Loaded symbols for /lib/libgcc_s.so.1 > Reading symbols from /lib/tls/i686/cmov/libc.so.6...done. > Loaded symbols for /lib/tls/i686/cmov/libc.so.6 > Reading symbols from /lib/ld-linux.so.2...done. > Loaded symbols for /lib/ld-linux.so.2 > #0 =A00xb7b9c443 in memmove () from /lib/tls/i686/cmov/libc.so.6 > (gdb) info threads Threads 2 through 24 are blocked in a system call (in kernel VDSO page): > =A024 process 31589 =A00xffffe410 in ?? () > =A023 process 31591 =A00xffffe410 in ?? () > =A022 process 31592 =A00xffffe410 in ?? () > =A021 process 31593 =A00xffffe410 in ?? () > =A020 process 31594 =A00xffffe410 in ?? () > =A019 process 31595 =A00xffffe410 in ?? () > =A018 process 31600 =A00xffffe410 in ?? () > =A017 process 31601 =A00xffffe410 in ?? () > =A016 process 31602 =A00xffffe410 in ?? () > =A015 process 31603 =A00xffffe410 in ?? () > =A014 process 31608 =A00xffffe410 in ?? () > =A013 process 31609 =A00xffffe410 in ?? () > =A012 process 31610 =A00xffffe410 in ?? () > =A011 process 31611 =A00xffffe410 in ?? () > =A010 process 31616 =A00xffffe410 in ?? () > =A09 process 31617 =A00xffffe410 in ?? () > =A08 process 31618 =A00xffffe410 in ?? () > =A07 process 31619 =A00xffffe410 in ?? () > =A06 process 31624 =A00xffffe410 in ?? () > =A05 process 31625 =A00xffffe410 in ?? () > =A04 process 31626 =A00xffffe410 in ?? () > =A03 process 31632 =A00xffffe410 in ?? () > =A02 process 31633 =A00xffffe410 in ?? () > * 1 process 31627 =A00xb7b9c443 in memmove () from /lib/tls/i686/cmov/lib= c.so.6 > (gdb) bt 12 > #0 =A00xb7b9c443 in memmove () from /lib/tls/i686/cmov/libc.so.6 > #1 =A00x08066b62 in IE_Sock::Recv (this=3D0x8098268, pch=3D0x8098264 "", = cbToRecv=3D4, iSecs=3D-1) at IE_Sock.cc:284 > #2 =A00x08066d51 in IE_Sock::RecvInt (this=3D0x8098268, pi=3D0x8098264, i= Secs=3D-1) at IE_Sock.cc:257 > #3 =A00x08070b3f in Thread::InitConnection (this=3D0x8097dfc) at commands= .cc:45 > #4 =A00x08062d73 in Thread::Start (this=3D0x8097dfc) at SMthreads.cc:195 > #5 =A00x08062ec7 in Thread::Thread_Start (pv=3D0x8097dfc) at SMthreads.cc= :37 > #6 =A00xb7e3f341 in start_thread () from /lib/tls/i686/cmov/libpthread.so= .0 > #7 =A00xb7bfa4ee in clone () from /lib/tls/i686/cmov/libc.so.6 > (gdb) thread 2 > [Switching to thread 2 (process 31633)]#0 =A00xffffe410 in ?? () > (gdb) bt 12 Typical stack trace when libc.so.6 and libpthread.so.0 are different at core dump and at analysis time: > #0 =A00xffffe410 in ?? () > #1 =A00xacaf8258 in ?? () > #2 =A00xb7e493b4 in ?? () from /lib/tls/i686/cmov/libpthread.so.0 > #3 =A00xacaf8220 in ?? () > #4 =A00xb7e44778 in accept () from /lib/tls/i686/cmov/libpthread.so.0 > #5 =A00x080661fd in IE_SockSrv::Accept (this=3D0x8099530, psock=3D0x80991= 00, sz=3D0x0, sl=3D0) at IE_Sock.cc:74 > #6 =A00x0808039c in admin_thread::thread_start (pv=3D0x0) at admin_thread= .cc:44 > #7 =A00xb7e3f341 in start_thread () from /lib/tls/i686/cmov/libpthread.so= .0 > #8 =A00xb7bfa4ee in clone () from /lib/tls/i686/cmov/libc.so.6 > (gdb) thread 3 > [Switching to thread 3 (process 31632)]#0 =A00xffffe410 in ?? () > (gdb) bt 12 > #0 =A00xffffe410 in ?? () > #1 =A00xad2f940c in ?? () > #2 =A00x00000021 in ?? () > #3 =A00x00000000 in ?? () > > and all remaining the threads (4 to 24) have the same screwed up stack > > > > =A0 =A0 =A0Veja quais s=E3o os assuntos do momento no Yahoo! +Buscados > http://br.maisbuscados.yahoo.com > Assuming my guess of host/target mismatch is correct, the solution is: gdb mfsrv (gdb) set solib-absolute-prefix /path/to/target/root # libc.so.6, libpthread.so.0 should be in /path/to/target/root/lib/tls/i686/cmov (gdb) core core.31589 Cheers, --=20 Paul Pluzhnikov