Problem debugging core dump in gdb 7.1 on Solaris x86_64

Problem debugging core dump in gdb 7.1 on Solaris x86_64

[Max Kaehn]

On a Solaris x86_64 machine:

tempest-solaris% uname -a
SunOS tempest-solaris 5.10 Generic_141445-09 i86pc i386 i86pc Solaris

with a gdb 7.1 configured thusly:

./configure  --build=x86_64-pc-solaris2.10 --host=x86_64-pc-solaris2.10 
--target=x86_64-pc-solaris2.10 
--prefix=/net/tools/gnu/gdb/7.1/i686_SunOS_64.5.10 
--with-mpfr=/net/tools/mpfr/2.4.1/i686_SunOS_64.5.10 
--with-gmp=/net/tools/gmp/4.3.1/i686_SunOS_64.5.10 
--with-tcl=/net/tools/tcl/8.6b1/i686_SunOS_64.5.10/lib 
--with-tk=/net/tools/tcl/8.6b1/i686_SunOS_64.5.10/lib --with-expat 
--with-libexpat-prefix=/net/tools/expat/2.0.1/i686_SunOS_64.5.10 
--with-curses --with-python=no --enable-tui  --enable-64bit 
--enable-64-bit-bfd 
--with-libiconv-prefix=/net/tools/libiconv/1.13.1/i686_SunOS_64.5.10 
--with-libintl-prefix=/net/tools/gnu/gettext/0.17/i686_SunOS_64.5.10 
--enable-targets=i686-pc-solaris2.10,x86_64-pc-solaris2.10

and built using gcc 4.3.4 configured for i686-pc-solaris2.10 (which 
generates functioning amd64 binaries when I build with -m64), I can get 
a reasonable stack trace from the core dump from a simple program 
("hello, world" + core dump):

GNU gdb (GDB) 7.1
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later 
<http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-pc-solaris2.10".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /home/slothman/Projects/goodbye...done.
[New LWP 1]
Reading symbols from 
/net/tools/util/i686_SunOS.5.10/lib/amd64/libstdc++.so.6...done.
Loaded symbols for /net/tools/util/i686_SunOS.5.10/lib/amd64/libstdc++.so.6
Reading symbols from /lib/amd64/libm.so.2...(no debugging symbols 
found)...done.
Loaded symbols for /lib/amd64/libm.so.2
Reading symbols from 
/net/tools/util/i686_SunOS.5.10/lib/amd64/libgcc_s.so.1...done.
Loaded symbols for /net/tools/util/i686_SunOS.5.10/lib/amd64/libgcc_s.so.1
Reading symbols from /lib/amd64/libc.so.1...(no debugging symbols 
found)...done.
[Thread debugging using libthread_db enabled]
[New Thread 1 (LWP 1)]
Loaded symbols for /lib/amd64/libc.so.1
Reading symbols from /lib/amd64/ld.so.1...(no debugging symbols 
found)...done.
Loaded symbols for /lib/amd64/ld.so.1
Core was generated by `./goodbye'.
Program terminated with signal 11, Segmentation fault.
#0  0x0000000000400d17 in main (argc=1, argv=0xfffffd7fffdff848)
     at goodbye.cpp:7
7           *p = 0;

but I've been unable to get a sensible stack trace out of a core dump 
from a more complex program:

tempest-solaris% /net/tools/gnu/gdb/7.1/i686_SunOS_64.5.10/bin/gdb 
/net/chronic2nas/emake-slothman-main/out/i686_SunOS_64.5.10/ecloud/agent/ecagent 
/net/chronic2nas/emake-slothman-main/logs-201006211902-solx2-ea2/core
GNU gdb (GDB) 7.1
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later 
<http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-pc-solaris2.10".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from 
/net/chronic2nas/emake-slothman-main-201006211520/out/i686_SunOS_64.5.10/ecloud/agent/ecagent...done.
[New LWP 1]
[New LWP 2]
[New LWP 3]
[New LWP 4]
[New LWP 5]
Reading symbols from /usr/lib/amd64/ld.so.1...(no debugging symbols 
found)...done.
Loaded symbols for /usr/lib/amd64/ld.so.1
Core was generated by `/opt/ecloud/i686_SunOS.5.10/bin/ecagent 
/opt/ecloud/i686_SunOS.5.10/bin/runagen'.
Program terminated with signal 11, Segmentation fault.
#0  0xfffffd7ffeac431c in ?? ()
(gdb) bt
#0  0xfffffd7ffeac431c in ?? ()
Cannot access memory at address 0xfffffd7fffdfed30
(gdb) thread 2
[Switching to thread 2 (LWP 2)]#0  0xfffffd7ffeb2c8fa in ?? ()
(gdb) bt
#0  0xfffffd7ffeb2c8fa in ?? ()
Cannot access memory at address 0xfffffd7ffe1f8dd8
(gdb) thread 3
[Switching to thread 3 (LWP 3)]#0  0xfffffd7ffeb27527 in ?? ()
(gdb) bt
#0  0xfffffd7ffeb27527 in ?? ()
Cannot access memory at address 0xfffffd7ffdfffd68
(gdb) thread 4
[Switching to thread 4 (LWP 4)]#0  0xfffffd7ffeb27527 in ?? ()
(gdb) bt
#0  0xfffffd7ffeb27527 in ?? ()
Cannot access memory at address 0xfffffd7ffde00e78
(gdb) thread 5
[Switching to thread 5 (LWP 5)]#0  0xfffffd7ffeb2c8fa in ?? ()
(gdb) bt
#0  0xfffffd7ffeb2c8fa in ?? ()
Cannot access memory at address 0xfffffd7ffdbffc58
(gdb) info sharedlibrary
From                To                  Syms Read   Shared Object Library
0xfffffd7fff3c1010  0xfffffd7fff3e614e  Yes (*)     /usr/lib/amd64/ld.so.1
(*): Shared library is missing debugging information.

ldd reports this binary pulls in about 30 shared libraries.  I can load 
ecagent, type "break main", get a line number for it, and step through 
the code when it's live, but I still haven't managed to get gdb to 
figure out the core dump.

Is anyone familiar with the arcana of Solaris x86?

Thanks,
Max

Re: Problem debugging core dump in gdb 7.1 on Solaris x86_64

[Michael Snyder]

Max Kaehn wrote:
> On a Solaris x86_64 machine:
> 
> tempest-solaris% uname -a
> SunOS tempest-solaris 5.10 Generic_141445-09 i86pc i386 i86pc Solaris
> 
> with a gdb 7.1 configured thusly:
> 
> ./configure  --build=x86_64-pc-solaris2.10 --host=x86_64-pc-solaris2.10 
> --target=x86_64-pc-solaris2.10 
> --prefix=/net/tools/gnu/gdb/7.1/i686_SunOS_64.5.10 
> --with-mpfr=/net/tools/mpfr/2.4.1/i686_SunOS_64.5.10 
> --with-gmp=/net/tools/gmp/4.3.1/i686_SunOS_64.5.10 
> --with-tcl=/net/tools/tcl/8.6b1/i686_SunOS_64.5.10/lib 
> --with-tk=/net/tools/tcl/8.6b1/i686_SunOS_64.5.10/lib --with-expat 
> --with-libexpat-prefix=/net/tools/expat/2.0.1/i686_SunOS_64.5.10 
> --with-curses --with-python=no --enable-tui  --enable-64bit 
> --enable-64-bit-bfd 
> --with-libiconv-prefix=/net/tools/libiconv/1.13.1/i686_SunOS_64.5.10 
> --with-libintl-prefix=/net/tools/gnu/gettext/0.17/i686_SunOS_64.5.10 
> --enable-targets=i686-pc-solaris2.10,x86_64-pc-solaris2.10
> 
> and built using gcc 4.3.4 configured for i686-pc-solaris2.10 (which 
> generates functioning amd64 binaries when I build with -m64), I can get 
> a reasonable stack trace from the core dump from a simple program 
> ("hello, world" + core dump):
> 
> GNU gdb (GDB) 7.1
> Copyright (C) 2010 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later 
> <http://gnu.org/licenses/gpl.html>
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
> and "show warranty" for details.
> This GDB was configured as "x86_64-pc-solaris2.10".
> For bug reporting instructions, please see:
> <http://www.gnu.org/software/gdb/bugs/>...
> Reading symbols from /home/slothman/Projects/goodbye...done.
> [New LWP 1]
> Reading symbols from 
> /net/tools/util/i686_SunOS.5.10/lib/amd64/libstdc++.so.6...done.
> Loaded symbols for /net/tools/util/i686_SunOS.5.10/lib/amd64/libstdc++.so.6
> Reading symbols from /lib/amd64/libm.so.2...(no debugging symbols 
> found)...done.
> Loaded symbols for /lib/amd64/libm.so.2
> Reading symbols from 
> /net/tools/util/i686_SunOS.5.10/lib/amd64/libgcc_s.so.1...done.
> Loaded symbols for /net/tools/util/i686_SunOS.5.10/lib/amd64/libgcc_s.so.1
> Reading symbols from /lib/amd64/libc.so.1...(no debugging symbols 
> found)...done.
> [Thread debugging using libthread_db enabled]
> [New Thread 1 (LWP 1)]
> Loaded symbols for /lib/amd64/libc.so.1
> Reading symbols from /lib/amd64/ld.so.1...(no debugging symbols 
> found)...done.
> Loaded symbols for /lib/amd64/ld.so.1
> Core was generated by `./goodbye'.
> Program terminated with signal 11, Segmentation fault.
> #0  0x0000000000400d17 in main (argc=1, argv=0xfffffd7fffdff848)
>      at goodbye.cpp:7
> 7           *p = 0;
> 
> but I've been unable to get a sensible stack trace out of a core dump 
> from a more complex program:
> 
> tempest-solaris% /net/tools/gnu/gdb/7.1/i686_SunOS_64.5.10/bin/gdb 
> /net/chronic2nas/emake-slothman-main/out/i686_SunOS_64.5.10/ecloud/agent/ecagent 
> /net/chronic2nas/emake-slothman-main/logs-201006211902-solx2-ea2/core
> GNU gdb (GDB) 7.1
> Copyright (C) 2010 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later 
> <http://gnu.org/licenses/gpl.html>
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
> and "show warranty" for details.
> This GDB was configured as "x86_64-pc-solaris2.10".
> For bug reporting instructions, please see:
> <http://www.gnu.org/software/gdb/bugs/>...
> Reading symbols from 
> /net/chronic2nas/emake-slothman-main-201006211520/out/i686_SunOS_64.5.10/ecloud/agent/ecagent...done.
> [New LWP 1]
> [New LWP 2]
> [New LWP 3]
> [New LWP 4]
> [New LWP 5]
> Reading symbols from /usr/lib/amd64/ld.so.1...(no debugging symbols 
> found)...done.
> Loaded symbols for /usr/lib/amd64/ld.so.1

Seems like the only shared library that got loaded was ld.so.

Should more shared libs (eg. libc) be loaded?

Could there be an issue with finding the needed shared libs?



> Core was generated by `/opt/ecloud/i686_SunOS.5.10/bin/ecagent 
> /opt/ecloud/i686_SunOS.5.10/bin/runagen'.
> Program terminated with signal 11, Segmentation fault.
> #0  0xfffffd7ffeac431c in ?? ()
> (gdb) bt
> #0  0xfffffd7ffeac431c in ?? ()
> Cannot access memory at address 0xfffffd7fffdfed30
> (gdb) thread 2
> [Switching to thread 2 (LWP 2)]#0  0xfffffd7ffeb2c8fa in ?? ()
> (gdb) bt
> #0  0xfffffd7ffeb2c8fa in ?? ()
> Cannot access memory at address 0xfffffd7ffe1f8dd8
> (gdb) thread 3
> [Switching to thread 3 (LWP 3)]#0  0xfffffd7ffeb27527 in ?? ()
> (gdb) bt
> #0  0xfffffd7ffeb27527 in ?? ()
> Cannot access memory at address 0xfffffd7ffdfffd68
> (gdb) thread 4
> [Switching to thread 4 (LWP 4)]#0  0xfffffd7ffeb27527 in ?? ()
> (gdb) bt
> #0  0xfffffd7ffeb27527 in ?? ()
> Cannot access memory at address 0xfffffd7ffde00e78
> (gdb) thread 5
> [Switching to thread 5 (LWP 5)]#0  0xfffffd7ffeb2c8fa in ?? ()
> (gdb) bt
> #0  0xfffffd7ffeb2c8fa in ?? ()
> Cannot access memory at address 0xfffffd7ffdbffc58
> (gdb) info sharedlibrary
> From                To                  Syms Read   Shared Object Library
> 0xfffffd7fff3c1010  0xfffffd7fff3e614e  Yes (*)     /usr/lib/amd64/ld.so.1
> (*): Shared library is missing debugging information.
> 
> ldd reports this binary pulls in about 30 shared libraries.  I can load 
> ecagent, type "break main", get a line number for it, and step through 
> the code when it's live, but I still haven't managed to get gdb to 
> figure out the core dump.
> 
> Is anyone familiar with the arcana of Solaris x86?
> 
> Thanks,
> Max
> 

Re: Problem debugging core dump in gdb 7.1 on Solaris x86_64

[Max Kaehn]

Even dbx has trouble with it; this may be more of a Solaris issue than a 
gdb one.  This is running the debugger on the same machine as the core 
dump, and it still has issues:

solx2-ea2% /net/tools/solstudio/12.2/i686_SunOS.5.10/bin/dbx 
/net/chronic2nas/emake-slothman-main/out/i686_SunOS_64.5.10/ecloud/agent/ecagent 
/net/chronic2nas/emake-slothman-main/logs-201006211902-solx2-ea2/core
Reading ecagent
dbx: internal warning: writable memory segment 0x597000[28672] of size 0 
in core
dbx: internal warning: writable memory segment 0x59e000[3600384] of size 
0 in core
dbx: internal warning: writable memory segment 0xfffffd7ffd405000[4096] 
of size 0 in core
dbx: internal warning: writable memory segment 0xfffffd7ffd602000[12288] 
of size 0 in core
dbx: internal warning: writable memory segment 0xfffffd7ffd801000[12288] 
of size 0 in core
dbx: internal warning: writable memory segment 0xfffffd7ffda00000[12288] 
of size 0 in core
dbx: internal warning: writable memory segment 0xfffffd7ffdbff000[12288] 
of size 0 in core
dbx: internal warning: writable memory segment 0xfffffd7ffde00000[4096] 
of size 0 in core
dbx: internal warning: writable memory segment 0xfffffd7ffdfff000[4096] 
of size 0 in core
dbx: internal warning: writable memory segment 0xfffffd7ffe1f8000[28672] 
of size 0 in core
dbx: internal warning: writable memory segment 0xfffffd7ffe431000[8192] 
of size 0 in core
dbx: internal warning: writable memory segment 0xfffffd7ffe450000[65536] 
of size 0 in core
dbx: internal warning: writable memory segment 0xfffffd7ffe470000[65536] 
of size 0 in core
dbx: internal warning: writable memory segment 0xfffffd7ffe490000[65536] 
of size 0 in core
dbx: internal warning: writable memory segment 0xfffffd7ffe4b0000[65536] 
of size 0 in core
dbx: internal warning: writable memory segment 0xfffffd7ffe4f1000[8192] 
of size 0 in core
dbx: internal warning: writable memory segment 0xfffffd7ffe500000[65536] 
of size 0 in core
dbx: internal warning: writable memory segment 0xfffffd7ffe58a000[16384] 
of size 0 in core
dbx: internal warning: writable memory segment 0xfffffd7ffe58e000[4096] 
of size 0 in core
dbx: internal warning: writable memory segment 0xfffffd7ffe5a3000[4096] 
of size 0 in core
dbx: internal warning: writable memory segment 0xfffffd7ffe5e3000[24576] 
of size 0 in core
dbx: internal warning: writable memory segment 0xfffffd7ffe5e9000[4096] 
of size 0 in core
dbx: internal warning: writable memory segment 0xfffffd7ffe7e2000[49152] 
of size 0 in core
dbx: internal warning: writable memory segment 0xfffffd7ffe7ee000[4096] 
of size 0 in core
dbx: internal warning: writable memory segment 0xfffffd7ffe803000[8192] 
of size 0 in core
dbx: internal warning: writable memory segment 0xfffffd7ffe830000[4096] 
of size 0 in core
dbx: internal warning: writable memory segment 0xfffffd7ffe835000[4096] 
of size 0 in core
dbx: internal warning: writable memory segment 0xfffffd7ffe850000[4096] 
of size 0 in core
dbx: internal warning: writable memory segment 0xfffffd7ffe857000[4096] 
of size 0 in core
dbx: internal warning: writable memory segment 0xfffffd7ffe870000[4096] 
of size 0 in core
dbx: internal warning: writable memory segment 0xfffffd7ffe879000[4096] 
of size 0 in core
dbx: internal warning: writable memory segment 0xfffffd7ffe890000[4096] 
of size 0 in core
dbx: internal warning: writable memory segment 0xfffffd7ffe892000[4096] 
of size 0 in core
dbx: internal warning: writable memory segment 0xfffffd7ffe8c0000[4096] 
of size 0 in core
dbx: internal warning: writable memory segment 0xfffffd7ffe8c9000[4096] 
of size 0 in core
dbx: internal warning: writable memory segment 0xfffffd7ffe8e0000[4096] 
of size 0 in core
dbx: internal warning: writable memory segment 0xfffffd7ffe8e4000[4096] 
of size 0 in core
dbx: internal warning: writable memory segment 0xfffffd7ffe950000[4096] 
of size 0 in core
dbx: internal warning: writable memory segment 0xfffffd7ffe95e000[12288] 
of size 0 in core
dbx: internal warning: writable memory segment 0xfffffd7ffe961000[8192] 
of size 0 in core
dbx: internal warning: writable memory segment 0xfffffd7ffe970000[65536] 
of size 0 in core
dbx: internal warning: writable memory segment 0xfffffd7ffe9ac000[4096] 
of size 0 in core
dbx: internal warning: writable memory segment 0xfffffd7ffe9c0000[4096] 
of size 0 in core
dbx: internal warning: writable memory segment 0xfffffd7ffe9c8000[4096] 
of size 0 in core
dbx: internal warning: writable memory segment 0xfffffd7ffe9c9000[4096] 
of size 0 in core
dbx: internal warning: writable memory segment 0xfffffd7ffe9e1000[4096] 
of size 0 in core
dbx: internal warning: writable memory segment 0xfffffd7ffea40000[4096] 
of size 0 in core
dbx: internal warning: writable memory segment 0xfffffd7ffea43000[16384] 
of size 0 in core
dbx: internal warning: writable memory segment 0xfffffd7ffeb90000[24576] 
of size 0 in core
dbx: internal warning: writable memory segment 0xfffffd7ffeb9a000[36864] 
of size 0 in core
dbx: internal warning: writable memory segment 0xfffffd7ffeba3000[16384] 
of size 0 in core
dbx: internal warning: writable memory segment 0xfffffd7ffebc0000[4096] 
of size 0 in core
dbx: internal warning: writable memory segment 0xfffffd7ffebcd000[4096] 
of size 0 in core
dbx: internal warning: writable memory segment 0xfffffd7ffec30000[4096] 
of size 0 in core
dbx: internal warning: writable memory segment 0xfffffd7ffec36000[24576] 
of size 0 in core
dbx: internal warning: writable memory segment 0xfffffd7ffed51000[40960] 
of size 0 in core
dbx: internal warning: writable memory segment 0xfffffd7ffed5b000[77824] 
of size 0 in core
dbx: internal warning: writable memory segment 0xfffffd7ffee60000[4096] 
of size 0 in core
dbx: internal warning: writable memory segment 0xfffffd7ffee6b000[8192] 
of size 0 in core
dbx: internal warning: writable memory segment 0xfffffd7ffee80000[4096] 
of size 0 in core
dbx: internal warning: writable memory segment 0xfffffd7ffee88000[4096] 
of size 0 in core
dbx: internal warning: writable memory segment 0xfffffd7ffef40000[4096] 
of size 0 in core
dbx: internal warning: writable memory segment 0xfffffd7ffef46000[16384] 
of size 0 in core
dbx: internal warning: writable memory segment 0xfffffd7ffef4a000[36864] 
of size 0 in core
dbx: internal warning: writable memory segment 0xfffffd7ffef70000[4096] 
of size 0 in core
dbx: internal warning: writable memory segment 0xfffffd7ffef7f000[4096] 
of size 0 in core
dbx: internal warning: writable memory segment 0xfffffd7ffefb0000[4096] 
of size 0 in core
dbx: internal warning: writable memory segment 0xfffffd7ffefb4000[8192] 
of size 0 in core
dbx: internal warning: writable memory segment 0xfffffd7fff100000[4096] 
of size 0 in core
dbx: internal warning: writable memory segment 0xfffffd7fff10c000[57344] 
of size 0 in core
dbx: internal warning: writable memory segment 0xfffffd7fff11a000[4096] 
of size 0 in core
dbx: internal warning: writable memory segment 0xfffffd7fff290000[4096] 
of size 0 in core
dbx: internal warning: writable memory segment 
0xfffffd7fff29a000[151552] of size 0 in core
dbx: internal warning: writable memory segment 0xfffffd7fff2bf000[8192] 
of size 0 in core
dbx: internal warning: writable memory segment 0xfffffd7fff320000[4096] 
of size 0 in core
dbx: internal warning: writable memory segment 0xfffffd7fff329000[28672] 
of size 0 in core
dbx: internal warning: writable memory segment 0xfffffd7fff390000[4096] 
of size 0 in core
dbx: internal warning: writable memory segment 0xfffffd7fff39c000[12288] 
of size 0 in core
dbx: internal warning: writable memory segment 0xfffffd7fff3b0000[4096] 
of size 0 in core
dbx: internal warning: writable memory segment 0xfffffd7fff3ed000[4096] 
of size 0 in core
dbx: internal warning: writable memory segment 0xfffffd7fff3f0000[4096] 
of size 0 in core
dbx: internal warning: writable memory segment 0xfffffd7fff3fb000[8192] 
of size 0 in core
dbx: internal warning: writable memory segment 0xfffffd7fff3fd000[8192] 
of size 0 in core
dbx: internal warning: writable memory segment 0xfffffd7fffdfa000[24576] 
of size 0 in core
core file header read successfully
Reading ld.so.1
dbx: core file read error: address 0xfffffd7fff3fb000 not available
dbx: core file read error: address 0xfffffd7fff3fbae0 not available
dbx: core file read error: address 0x598ff0 not available
dbx: warning: Dbx could not initialize rtld_db
Make sure this is the same version of Solaris where the core dump 
originated.
Use `help core mismatch' for more info.
(l@1) terminated by signal SEGV (no mapping at the fault address)
0xffffffffffffffff: <bad address 0xffffffffffffffff>
(dbx) where
   [1] 0xfffffd7ffeac431c(0x0, 0x0, 0x784120, 0x170, 0x7, 0xffffffff), 
at 0xfffffd7ffeac431c
(dbx) threads
dbx: thread related commands not available

Re: Problem debugging core dump in gdb 7.1 on Solaris x86_64

[Max Kaehn]

On 06/22/2010 12:21 PM, Michael Snyder wrote:
> Seems like the only shared library that got loaded was ld.so.
>
> Should more shared libs (eg. libc) be loaded?
>
> Could there be an issue with finding the needed shared libs?
>    
"ldd" seems to be able to find them; I would hope gdb would be able to 
use the same mechanisms.

solx2-ea2% ldd 
/net/chronic2nas/emake-slothman-main/out/i686_SunOS_64.5.10/ecloud/agent/ecagent
         libcurl.so.4 =>  
/net/tools/util/i686_SunOS_64.5.10/lib/libcurl.so.4
         libssl.so.0.9.8 =>       
/net/tools/util/i686_SunOS_64.5.10/lib/libssl.so.0.9.8
         libcrypto.so.0.9.8 =>    
/net/tools/util/i686_SunOS_64.5.10/lib/libcrypto.so.0.9.8
         libpthread.so.1 =>       /lib/amd64/libpthread.so.1
         libtcl8.6.so =>  
/net/tools/util/i686_SunOS_64.5.10/lib/libtcl8.6.so
         libz.so.1 =>     /net/tools/util/i686_SunOS_64.5.10/lib/libz.so.1
         libsocket.so.1 =>        /lib/amd64/libsocket.so.1
         libnsl.so.1 =>   /lib/amd64/libnsl.so.1
         librt.so.1 =>    /lib/amd64/librt.so.1
         libiconv.so.2 =>         
/net/tools/util/i686_SunOS_64.5.10/lib/libiconv.so.2
         libstdc++.so.6 =>        
/net/tools/util/i686_SunOS.5.10/lib/amd64/libstdc++.so.6
         libm.so.2 =>     /lib/amd64/libm.so.2
         libgcc_s.so.1 =>         
/net/tools/util/i686_SunOS.5.10/lib/amd64/libgcc_s.so.1
         libc.so.1 =>     /lib/amd64/libc.so.1
         libldap.so.5 =>  /usr/lib/amd64/libldap.so.5
         libdl.so.1 =>    /lib/amd64/libdl.so.1
         libz.so.1 (SUNW_1.1) =>  (version not found)
         libz.so.1 (SUNW_1.2) =>  (version not found)
         libmp.so.2 =>    /lib/amd64/libmp.so.2
         libmd.so.1 =>    /lib/amd64/libmd.so.1
         libscf.so.1 =>   /lib/amd64/libscf.so.1
         libaio.so.1 =>   /lib/amd64/libaio.so.1
         libsasl.so.1 =>  /usr/lib/amd64/libsasl.so.1
         libnspr4.so =>   /usr/lib/mps/64/libnspr4.so
         libplc4.so =>    /usr/lib/mps/64/libplc4.so
         libnss3.so =>    /usr/lib/mps/64/libnss3.so
         libssl3.so =>    /usr/lib/mps/64/libssl3.so
         libdoor.so.1 =>  /lib/amd64/libdoor.so.1
         libuutil.so.1 =>         /lib/amd64/libuutil.so.1
         libgen.so.1 =>   /lib/amd64/libgen.so.1
         libnssutil3.so =>        /usr/lib/mps/amd64/libnssutil3.so
         libplds4.so =>   /usr/lib/mps/amd64/libplds4.so
         libthread.so.1 =>        /lib/amd64/libthread.so.1

Re: Problem debugging core dump in gdb 7.1 on Solaris x86_64

[Michael Snyder]

Max Kaehn wrote:
> On 06/22/2010 12:21 PM, Michael Snyder wrote:
>> Seems like the only shared library that got loaded was ld.so.
>>
>> Should more shared libs (eg. libc) be loaded?
>>
>> Could there be an issue with finding the needed shared libs?
>>    
> "ldd" seems to be able to find them; I would hope gdb would be able to 
> use the same mechanisms.

Hmmm, well I don't know if those are the standard locations, but
you might try using "set solib-search-path" before loading the
corefile.

> 
> solx2-ea2% ldd 
> /net/chronic2nas/emake-slothman-main/out/i686_SunOS_64.5.10/ecloud/agent/ecagent
>          libcurl.so.4 =>  
> /net/tools/util/i686_SunOS_64.5.10/lib/libcurl.so.4
>          libssl.so.0.9.8 =>       
> /net/tools/util/i686_SunOS_64.5.10/lib/libssl.so.0.9.8
>          libcrypto.so.0.9.8 =>    
> /net/tools/util/i686_SunOS_64.5.10/lib/libcrypto.so.0.9.8
>          libpthread.so.1 =>       /lib/amd64/libpthread.so.1
>          libtcl8.6.so =>  
> /net/tools/util/i686_SunOS_64.5.10/lib/libtcl8.6.so
>          libz.so.1 =>     /net/tools/util/i686_SunOS_64.5.10/lib/libz.so.1
>          libsocket.so.1 =>        /lib/amd64/libsocket.so.1
>          libnsl.so.1 =>   /lib/amd64/libnsl.so.1
>          librt.so.1 =>    /lib/amd64/librt.so.1
>          libiconv.so.2 =>         
> /net/tools/util/i686_SunOS_64.5.10/lib/libiconv.so.2
>          libstdc++.so.6 =>        
> /net/tools/util/i686_SunOS.5.10/lib/amd64/libstdc++.so.6
>          libm.so.2 =>     /lib/amd64/libm.so.2
>          libgcc_s.so.1 =>         
> /net/tools/util/i686_SunOS.5.10/lib/amd64/libgcc_s.so.1
>          libc.so.1 =>     /lib/amd64/libc.so.1
>          libldap.so.5 =>  /usr/lib/amd64/libldap.so.5
>          libdl.so.1 =>    /lib/amd64/libdl.so.1
>          libz.so.1 (SUNW_1.1) =>  (version not found)
>          libz.so.1 (SUNW_1.2) =>  (version not found)
>          libmp.so.2 =>    /lib/amd64/libmp.so.2
>          libmd.so.1 =>    /lib/amd64/libmd.so.1
>          libscf.so.1 =>   /lib/amd64/libscf.so.1
>          libaio.so.1 =>   /lib/amd64/libaio.so.1
>          libsasl.so.1 =>  /usr/lib/amd64/libsasl.so.1
>          libnspr4.so =>   /usr/lib/mps/64/libnspr4.so
>          libplc4.so =>    /usr/lib/mps/64/libplc4.so
>          libnss3.so =>    /usr/lib/mps/64/libnss3.so
>          libssl3.so =>    /usr/lib/mps/64/libssl3.so
>          libdoor.so.1 =>  /lib/amd64/libdoor.so.1
>          libuutil.so.1 =>         /lib/amd64/libuutil.so.1
>          libgen.so.1 =>   /lib/amd64/libgen.so.1
>          libnssutil3.so =>        /usr/lib/mps/amd64/libnssutil3.so
>          libplds4.so =>   /usr/lib/mps/amd64/libplds4.so
>          libthread.so.1 =>        /lib/amd64/libthread.so.1
> 

Re: Problem debugging core dump in gdb 7.1 on Solaris x86_64

[Max Kaehn]

On 06/22/2010 04:18 PM, Michael Snyder wrote:
> Hmmm, well I don't know if those are the standard locations, but
> you might try using "set solib-search-path" before loading the
> corefile.
>    
Certainly worth a try...  sadly, same result:

(gdb) set solib-search-path 
/lib:/usr/lib:/lib/amd64:/usr/lib/amd64:/net/tools/util/i686_SunOS_64.5.10/lib:/net/tools/util/i686_SunOS.5.10/lib/amd64
(gdb) core-file 
/net/chronic2nas/emake-slothman-main-201006211520/logs-201006211902-solx2-ea2/core
[New LWP 1]
[New LWP 2]
[New LWP 3]
[New LWP 4]
[New LWP 5]
Reading symbols from /usr/lib/amd64/ld.so.1...(no debugging symbols 
found)...done.
Loaded symbols for /usr/lib/amd64/ld.so.1
Core was generated by `/opt/ecloud/i686_SunOS.5.10/bin/ecagent 
/opt/ecloud/i686_SunOS.5.10/bin/runagen'.
Program terminated with signal 11, Segmentation fault.
#0  0xfffffd7ffeac431c in ?? ()
(gdb) bt
#0  0xfffffd7ffeac431c in ?? ()
Cannot access memory at address 0xfffffd7fffdfed30
(gdb) info sharedlibrary
From                To                  Syms Read   Shared Object Library
0xfffffd7fff3c1010  0xfffffd7fff3e614e  Yes (*)     /usr/lib/amd64/ld.so.1
(*): Shared library is missing debugging information.

Thanks for the suggestion, though.

Re: Problem debugging core dump in gdb 7.1 on Solaris x86_64

[Max Kaehn]

It turns out that the problem was that the default Solaris setting for 
core dumps wasn't giving all information.  Running "coreadm -G all -I 
all" caused the system to start generating core dumps with readable call 
stacks.