lin-lwp.c assertion failures

lin-lwp.c assertion failures

[Christian Jaeger]

Hello

I'm having a hard time trying to use gdb on proftpd (and also afpd) 
server childs that have LD_PRELOAD'ed a shared library. The shared 
lib intercepts glibc/system calls (mainly filesystem calls), and also 
loads a perl interpreter - it serves filesystem calls to a part of 
the file tree (virtual filesystem), the basis is the AVFS-preload 
library from Frederik Eaton.

Gdb gives an internal error when the server child segfaults, or also 
when I interrupt the child from gdb.

In detail: I can attach to the child process i.e. while it is waiting 
for user input from authentification, and it (at least gvd) is also 
showing the correct position in the virtual select() code; I give 
'continue', gdb waits; I enter the authentification data over the 
wire, now the server child segfaults probably because of some bug in 
my code. gdb gives an internal error message, and even when I say Y 
to continue debugging, it doesn't seem to work anymore.
Below you find the output from this session.
I've also appended part of the strace -p `childpid` output from 
another identical session without an attached gdb.

(This is gdb packaged for Debian Woody or unstable, running on kernel 2.4.18)

Are these problems coming from "too much different code in the same 
process", or is it because I'm trapping glibc calls, or is it simply 
a gdb bug? What can I do?

The bad thing is that I also can't get core dumps since proftpd 
changes uid and linux won't drop any core after that.

Thanks for any help or ideas
Christian.



root@pflanze root# ps aux|grep proftpd
nobody   23341  0.0  4.4 17668 11264 ?       S    00:10   0:00 
proftpd (accepting connections)
nobody   23437  0.1  4.5 17784 11616 ?       S    00:39   0:00 
proftpd: connected: chris (192.168.1.8)

root@pflanze root# gdb /usr/sbin/proftpd 23437
GNU gdb 2002-04-01-cvs
Copyright 2002 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i386-linux"...(no debugging symbols found)...
Attaching to program: /usr/sbin/proftpd, process 23437
Reading symbols from /home/chris/avfs-0.9.0-fe6/preload/avfs_preload.so...done.
Loaded symbols for /home/chris/avfs-0.9.0-fe6/preload/avfs_preload.so
Reading symbols from /lib/libcrypt.so.1...done.
Loaded symbols for /lib/libcrypt.so.1
Reading symbols from /usr/lib/libssl.so.0.9.6...done.
Loaded symbols for /usr/lib/libssl.so.0.9.6
...
...
Reading symbols from /usr/lib/libperl.so.5.6...done.
...
Reading symbols from /usr/lib/perl/5.6.1/auto/POSIX/POSIX.so...done.
...
Loaded symbols for /usr/lib/perl5/auto/Image/Magick/Magick.so
Reading symbols from /usr/lib/libMagick.so.5...done.
Loaded symbols for /usr/lib/libMagick.so.5
...
Reading symbols from /lib/libpthread.so.0...done.
[New Thread 1024 (LWP 23337)]
Error while reading shared library symbols:
Can't attach LWP 23337: No such process
Reading symbols from /usr/lib/libwmf-0.2.so.2...done.
...
Loaded symbols for /lib/libresolv.so.2
0x4024ce1e in select () from /lib/libc.so.6
(gdb) continue
Continuing.
../../gdb/lin-lwp.c:1105: gdb-internal-error: lin_lwp_wait: Assertion 
`WIFSTOPPED (status) && WSTOPSIG (status) == SIGSTOP' failed.
An internal GDB error was detected.  This may make further
debugging unreliable.  Quit this debugging session? (y or n) n

Create a core file containing the current state of GDB? (y or n) y
(gdb) bt
#0  0x4024ce1e in select () from /lib/libc.so.6
#1  0x0877515c in ?? ()
#2  0x08051685 in strcpy ()
Cannot access memory at address 0x7
(gdb)



root@pflanze root# strace -p 23713
select(1, [0], NULL, NULL, {40, 930000}) = 1 (in [0], left {38, 850000})
read(0, "PASS s9i2fhsu1d\r\n", 1022)   = 18
alarm(0)                                = 277
alarm(277)                              = 0
close(4)                                = 0
munmap(0x404c4000, 4096)                = 0
close(5)                                = 0
munmap(0x404c5000, 4096)                = 0
...
...(quite much stuff)
...
time([1022369441])                      = 1022369441
getpid()                                = 23713
geteuid32()                             = 1024
getegid32()                             = 107
flock(9, LOCK_EX)                       = 0
lseek(9, 644, SEEK_SET)                 = 644
read(9, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 644) = 644
read(9, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 644) = 644
read(9, "", 644)                        = 0
lseek(9, 644, SEEK_END)                 = 2576
lseek(9, 644, SEEK_SET)                 = 644
write(9, "\241\\\0\0\0\4\0\0k\0\0\0\0\0\0\0004\10\0\0\241\36\360"..., 
644) = 644
flock(9, LOCK_UN)                       = 0
alarm(0)                                = 277
alarm(577)                              = 0
--- SIGSEGV (Segmentation fault) ---
rt_sigaction(SIGSEGV, {SIG_DFL}, {0x804f0d0, [SEGV], 
SA_RESTART|0x4000000}, 8) = 0
brk(0x8794000)                          = 0x8794000
time([1022369441])                      = 1022369441
getpid()                                = 23713
rt_sigaction(SIGPIPE, {0x4024f578, [], 0x4000000}, {SIG_IGN}, 8) = 0
send(2, "<27>May 26 01:30:41 proftpd[2371"..., 99, 0) = 99
rt_sigaction(SIGPIPE, {SIG_IGN}, NULL, 8) = 0
rt_sigprocmask(SIG_BLOCK, [HUP INT QUIT BUS USR1 ALRM TERM CHLD IO], 
NULL, 8) = 0
getpid()                                = 23713
write(6, "CDEBUG(23713): VIRT_SETEUID call"..., 35) = 35
setresuid32(0xffffffff, 0, 0xffffffff)  = 0
getuid32()                              = 0
getuid32()                              = 0
time([1022369441])                      = 1022369441
getpid()                                = 23713
rt_sigaction(SIGPIPE, {0x4024f578, [], 0x4000000}, {SIG_IGN}, 8) = 0
send(2, "<38>May 26 01:30:41 PAM_unix[237"..., 81, 0) = 81
rt_sigaction(SIGPIPE, {SIG_IGN}, NULL, 8) = 0
close(2)                                = 0
munmap(0x4050f000, 11788)               = 0
munmap(0x40a30000, 94144)               = 0
geteuid32()                             = 0
getpid()                                = 23713
write(6, "CDEBUG(23713): VIRT_SETEUID call"..., 35) = 35
setresuid32(0xffffffff, 0x400, 0xffffffff) = 0
rt_sigprocmask(SIG_UNBLOCK, [HUP INT QUIT BUS USR1 ALRM TERM CHLD 
IO], NULL, 8) = 0
getpid()                                = 23713
getpid()                                = 23713
write(6, "CDEBUG(23713): VIRT___FXSTAT64: "..., 62) = 62
fstat64(7, {st_mode=S_IFREG|0664, st_size=259200, ...}) = 0
getpid()                                = 23713
time([1022369441])                      = 1022369441
write(7, "\10\0\0\0\241\\\0\0ftpd23713\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 
384) = 384
getpid()                                = 23713
geteuid32()                             = 1024
getegid32()                             = 107
flock(9, LOCK_EX)                       = 0
lseek(9, 644, SEEK_SET)                 = 644
read(9, "\241\\\0\0\0\4\0\0k\0\0\0\0\0\0\0004\10\0\0\241\36\360"..., 644) = 644
lseek(9, 644, SEEK_SET)                 = 644
write(9, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 
644) = 644
flock(9, LOCK_UN)                       = 0
close(9)                                = 0
_exit(1)                                = ?

-- 
Christian Jaeger  Programmer & System Engineer  +41 1 430 45 26
ETHLife CMS Project - www.ethlife.ethz.ch/newcms - www.ethlife.ethz.ch

Re: lin-lwp.c assertion failures

[Christian Jaeger]

Following up [on] my own post:

I've done the following now to try to track down the failure causes: 
(I'm on an AMD Duron running Debian Woody)
- removed gcc-2.95 and installed gcc-3.1, compiled my library with 
this compiler and tried to attach & continue still with gdb 
5.2.cvs20020401-6 (from Debian).
- removed gcc-3.1 again and reinstalled 2.95.4, recompiled library 
and tried to attach & continue still with same gdb version.
- fetched gdb 5.2 source and ./configure && make && make install, 
used this gdb to attach & continue.

Same thing with all the above.

For gdb 5.2 I've told it to drop a core and got a bt which I'm 
pasting below. Tell me if you need anything else.

Yours,
Christian Jaeger


root@pflanze root# gdb proftpd 18553
GNU gdb 5.2
Copyright 2002 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i686-pc-linux-gnu"...(no debugging 
symbols found)...
Attaching to program: /usr/sbin/proftpd, process 18553
Reading symbols from /home/chris/avfs-0.9.0-fe6/preload/avfs_preload.so...done.
...
Reading symbols from /lib/libpthread.so.0...done.
[New Thread 1024 (LWP 10218)]
Error while reading shared library symbols:
Can't attach LWP 10218: No such process
Reading symbols from /usr/lib/libwmf-0.2.so.2...done.
...
Loaded symbols for /lib/libnss_compat.so.2
0x4024ce1e in select () from /lib/libc.so.6
(gdb) continue
Continuing.
lin-lwp.c:1105: gdb-internal-error: lin_lwp_wait: Assertion 
`WIFSTOPPED (status) && WSTOPSIG (status) == SIGSTOP' failed.
An internal GDB error was detected.  This may make further
debugging unreliable.  Quit this debugging session? (y or n) y

Create a core file containing the current state of GDB? (y or n) y
Aborted (core dumped)

root@pflanze root# gdb /usr/local/bin/gdb core
GNU gdb 5.2
Copyright 2002 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i686-pc-linux-gnu"...
Core was generated by `gdb proftpd 18553'.
Program terminated with signal 6, Aborted.
...
(gdb) bt
#0  0x400b5911 in kill () from /lib/libc.so.6
#1  0x400b55f4 in raise () from /lib/libc.so.6
#2  0x400b6d71 in abort () from /lib/libc.so.6
#3  0x080d16bd in internal_verror (file=0x81ae49d "lin-lwp.c", line=1105,
     fmt=0x81ae482 "%s%sAssertion `%s' failed.", ap=0xbffff060) at utils.c:728
#4  0x080d1710 in safe_strerror (errnum=135980189) at utils.c:751
#5  0x080add30 in lin_lwp_wait (ptid={pid = -1, lwp = 0, tid = 0}, 
ourstatus=0xbffff1cc)
     at lin-lwp.c:1105
#6  0x080e8f1d in thread_db_wait (ptid={pid = -1, lwp = 0, tid = 0}, 
ourstatus=0xbffff1cc)
     at thread-db.c:725
#7  0x08092944 in wait_for_inferior () at infrun.c:1238
#8  0x08092782 in proceed (addr=4294967295, 
siggnal=TARGET_SIGNAL_DEFAULT, step=0) at infrun.c:1037
#9  0x0808ffb3 in continue_command (proc_count_exp=0x0, from_tty=1) 
at infcmd.c:528
#10 0x08126ac7 in do_cfunc (c=0x8226cb8, args=0x0, from_tty=1) at 
./cli/cli-decode.c:50
#11 0x080cf403 in execute_command (p=0x821cc51 "", from_tty=1) at top.c:715
#12 0x0809b816 in command_handler (command=0x821cc48 "continue ") at 
event-top.c:504
#13 0x0809be83 in command_line_handler (rl=0x821cc28 "continue ") at 
event-top.c:800
#14 0x0817b68a in rl_callback_read_char () at callback.c:114
#15 0x0809b17b in rl_callback_read_char_wrapper (client_data=0x0) at 
event-top.c:168
#16 0x0809b6ed in stdin_event_handler (error=0, client_data=0x0) at 
event-top.c:418
#17 0x080e16f6 in handle_file_event (event_file_desc=0) at event-loop.c:714
#18 0x080e1206 in process_event () at event-loop.c:335
#19 0x080e1245 in gdb_do_one_event (data=0x0) at event-loop.c:372
#20 0x080cf07b in do_catch_errors (uiout=0x823a198, data=0xbffff5a4) 
at top.c:491
#21 0x080cef45 in catcher (func=0x80cf068 <do_catch_errors>, 
func_uiout=0x823a198,
     func_args=0xbffff5a4, func_val=0xbffff59c, 
func_caught=0xbffff5a0, errstring=0x81c1e60 "",
     mask=6) at top.c:423
#22 0x080cf0bb in catch_errors (func=0x80e121c <gdb_do_one_event>, 
func_args=0x0,
     errstring=0x81c1e60 "", mask=6) at top.c:503
#23 0x080e1283 in start_event_loop () at event-loop.c:416
#24 0x0809b262 in cli_command_loop () at event-top.c:200
#25 0x080710f8 in captured_command_loop (data=0x0) at main.c:94
#26 0x080cf07b in do_catch_errors (uiout=0x823a198, data=0xbffff754) 
at top.c:491
#27 0x080cef45 in catcher (func=0x80cf068 <do_catch_errors>, 
func_uiout=0x823a198,
     func_args=0xbffff754, func_val=0xbffff74c, 
func_caught=0xbffff750, errstring=0x8192cad "",
     mask=6) at top.c:423
#28 0x080cf0bb in catch_errors (func=0x80710e0 
<captured_command_loop>, func_args=0x0,
     errstring=0x8192cad "", mask=6) at top.c:503
#29 0x08071b73 in captured_main (data=0xbffffa24) at main.c:723
#30 0x080cf07b in do_catch_errors (uiout=0x81fe6c0, data=0xbffff9f4) 
at top.c:491
#31 0x080cef45 in catcher (func=0x80cf068 <do_catch_errors>, 
func_uiout=0x81fe6c0,
     func_args=0xbffff9f4, func_val=0xbffff9ec, 
func_caught=0xbffff9f0, errstring=0x8192cad "",
     mask=6) at top.c:423
#32 0x080cf0bb in catch_errors (func=0x807112c <captured_main>, 
func_args=0xbffffa24,
     errstring=0x8192cad "", mask=6) at top.c:503
#33 0x08071b9f in main (argc=3, argv=0xbffffa94) at main.c:734
#34 0x400a565f in __libc_start_main () from /lib/libc.so.6
(gdb)