how can i stop a running process to do a backtrace?

how can i stop a running process to do a backtrace?

[Jelle de Jong]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hello,

My name is Jelle and i am new to this list, so thanks you for all the
hard work everybody is doing on the gdb debugger.

I have a problem with freezing sshfs mounts on my systems, so I wanted
to debug the sshfs system.

I contacted the fuse-devel mailinglist and ask them how I could help to
create some usefull bug reports, they told me to do some backtrace with
gdb: (the archive does not seem to be working on this moment)
Message-ID: <487C568C.4010504@powercraft.nl>
https://lists.sourceforge.net/lists/listinfo/fuse-devel

I use debian sid and there were no sshfs-dbg packages, so I created a
bug-report and attached my debdiff and new deb packages:
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=491210

I now wanted to run my gdb in the following way:

pidof sshfs
1948

gdb 2>&1 | tee gdb-sshfs.txt
(gdb) handle SIG33 pass nostop noprint
(gdb) set pagination 0
(gdb) attach 1948
(gdb) continue
# ok how to stop the continue command so i can do a backtrace?
# kill 1948 wil stop the process but leaves no traceabale info
(gdb) backtrace full
(gdb) info registers
(gdb) thread apply all backtrace
(gdb) quit

But when the gdb is in "Continue" mode and the sshfs is freezing i want
to go back to gdb and do a backtrace its still in "Continue" mode and i
cant do anything. If i kill sshfs then there is no backtrace information
left. So how can i get debug information out of my frozen sshfs program?

I asked this question today on the IRC channel with an exception of hlep
from "jeremybennett" nobody was there. So i decided to send this mail.

I hope somebody can help me?

Best regards,

Jelle
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iJwEAQECAAYFAkiCAMoACgkQ1WclBW9j5HnhegQAjKoIl1zAAtR+bPFYJlinZLK1
KOKXCBvceuk1ri6neb3aGK/RsWtKf63hP4VLrdT8Gh6Aif5GWAR6ojhe7HI5yrKl
1roeSf4/0rPSpgY7coIopcsgcKBLAqMxpXdD/sbw6clLMOcE5pIn3+5BbvjW1Z0V
V7o+q7CXqgJUg3Gy0KY=
=Poet
-----END PGP SIGNATURE-----

Re: how can i stop a running process to do a backtrace?

[Daniel Jacobowitz]

On Sat, Jul 19, 2008 at 04:57:16PM +0200, Jelle de Jong wrote:
> (gdb) continue
> # ok how to stop the continue command so i can do a backtrace?
> # kill 1948 wil stop the process but leaves no traceabale info

What does it show in GDB after you do this?

It sounds like you're out of luck.  WHen programs, like fuse, go in
and out of the kernel it is sometimes impossible for GDB to regain
control.

-- 
Daniel Jacobowitz
CodeSourcery

Re: how can i stop a running process to do a backtrace?

[Jelle de Jong]

[-- Attachment #1: Type: text/plain, Size: 622 bytes --]

Daniel Jacobowitz wrote:
> On Sat, Jul 19, 2008 at 04:57:16PM +0200, Jelle de Jong wrote:
>> (gdb) continue
>> # ok how to stop the continue command so i can do a backtrace?
>> # kill 1948 wil stop the process but leaves no traceabale info
> 
> What does it show in GDB after you do this?
> 
> It sounds like you're out of luck.  WHen programs, like fuse, go in
> and out of the kernel it is sometimes impossible for GDB to regain
> control.

Thank you for your fast response, I attached my logfile i created
yesterday.

After the sshfs froze i did a killall -9 sshfs and run the next gdb
commands.

Best regards,

Jelle


[-- Attachment #2: gdb-sshfs-v0,0.1.txt --]
[-- Type: text/plain, Size: 2047 bytes --]

GNU gdb 6.8-debian
Copyright (C) 2008 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "i486-linux-gnu".
(gdb) handle SIG33 pass nostop noprint
Signal        Stop	Print	Pass to program	Description
SIG33         No	No	Yes		Real-time event 33
(gdb) set pagination 0
(gdb) attach 2312
Attaching to process 2312
Reading symbols from /usr/bin/sshfs...Reading symbols from /usr/lib/debug/usr/bin/sshfs...done.
done.
Reading symbols from /usr/lib/libfuse.so.2...done.
Loaded symbols for /usr/lib/libfuse.so.2
Reading symbols from /lib/i686/cmov/libdl.so.2...done.
Loaded symbols for /lib/i686/cmov/libdl.so.2
Reading symbols from /usr/lib/libgthread-2.0.so.0...done.
Loaded symbols for /usr/lib/libgthread-2.0.so.0
Reading symbols from /lib/i686/cmov/librt.so.1...done.
Loaded symbols for /lib/i686/cmov/librt.so.1
Reading symbols from /usr/lib/libglib-2.0.so.0...done.
Loaded symbols for /usr/lib/libglib-2.0.so.0
Reading symbols from /lib/i686/cmov/libpthread.so.0...done.
[Thread debugging using libthread_db enabled]
[New Thread 0xb7d36960 (LWP 2312)]
[New Thread 0xb7d13b90 (LWP 2313)]
Loaded symbols for /lib/i686/cmov/libpthread.so.0
Reading symbols from /lib/i686/cmov/libc.so.6...done.
Loaded symbols for /lib/i686/cmov/libc.so.6
Reading symbols from /lib/ld-linux.so.2...done.
Loaded symbols for /lib/ld-linux.so.2
Reading symbols from /lib/libselinux.so.1...done.
Loaded symbols for /lib/libselinux.so.1
Reading symbols from /usr/lib/libpcre.so.3...done.
Loaded symbols for /usr/lib/libpcre.so.3
0xb7fde424 in __kernel_vsyscall ()
(gdb) continue
Continuing.

Program terminated with signal SIGKILL, Killed.
The program no longer exists.
(gdb) backtrace full
No stack.
(gdb) info registers
The program has no registers now.
(gdb) thread apply all backtrace
No registers.
(gdb) quit

Re: how can i stop a running process to do a backtrace?

[Paul Pluzhnikov]

On Sat, Jul 19, 2008 at 8:03 AM, Jelle de Jong
<jelledejong@powercraft.nl> wrote:

> After the sshfs froze i did a killall -9 sshfs

Well, don't do *that* (it causes the kernel to obliterate the process
completely).
Try "killall sshfs", or "killall -SIGABRT sshfs", or "killall -SIGQUIT sshfs".

If one of the above wakes sshfs up, then you'll have a chance to debug it.

Cheers,
-- 
Paul Pluzhnikov

Re: how can i stop a running process to do a backtrace?

[Jelle de Jong]

[-- Attachment #1: Type: text/plain, Size: 713 bytes --]

Paul Pluzhnikov wrote:
> On Sat, Jul 19, 2008 at 8:03 AM, Jelle de Jong
> <jelledejong@powercraft.nl> wrote:
> 
>> After the sshfs froze i did a killall -9 sshfs
> 
> Well, don't do *that* (it causes the kernel to obliterate the process
> completely).
> Try "killall sshfs", or "killall -SIGABRT sshfs", or "killall -SIGQUIT sshfs".
> 
> If one of the above wakes sshfs up, then you'll have a chance to debug it.

Thank you all for the good information.

I managed to create three backtraces after the sshfs program froze. I 
attached one of them as example. I something is obvious wrong with the 
report please tell me.

I took the reports upstream to the sshfs developers.

Thank you all,

Kind regards,

Jelle

[-- Attachment #2: gdb-sshfs-v0.1.4.txt --]
[-- Type: text/plain, Size: 5817 bytes --]

GNU gdb 6.8-debian
Copyright (C) 2008 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "i486-linux-gnu".
(gdb) handle SIG33 pass nostop noprint
Signal        Stop  Print   Pass to program Description
SIG33         No    No  Yes     Real-time event 33
(gdb) set pagination 0
(gdb) attach 1973
Attaching to process 1973
Reading symbols from /usr/bin/sshfs...Reading symbols from /usr/lib/debug/usr/bin/sshfs...done.
done.
Reading symbols from /usr/lib/libfuse.so.2...done.
Loaded symbols for /usr/lib/libfuse.so.2
Reading symbols from /lib/i686/cmov/libdl.so.2...done.
Loaded symbols for /lib/i686/cmov/libdl.so.2
Reading symbols from /usr/lib/libgthread-2.0.so.0...done.
Loaded symbols for /usr/lib/libgthread-2.0.so.0
Reading symbols from /lib/i686/cmov/librt.so.1...done.
Loaded symbols for /lib/i686/cmov/librt.so.1
Reading symbols from /usr/lib/libglib-2.0.so.0...done.
Loaded symbols for /usr/lib/libglib-2.0.so.0
Reading symbols from /lib/i686/cmov/libpthread.so.0...done.
[Thread debugging using libthread_db enabled]
[New Thread 0xb7c72960 (LWP 1973)]
[New Thread 0xb7c4fb90 (LWP 1974)]
Loaded symbols for /lib/i686/cmov/libpthread.so.0
Reading symbols from /lib/i686/cmov/libc.so.6...done.
Loaded symbols for /lib/i686/cmov/libc.so.6
Reading symbols from /lib/ld-linux.so.2...done.
Loaded symbols for /lib/ld-linux.so.2
Reading symbols from /lib/libselinux.so.1...done.
Loaded symbols for /lib/libselinux.so.1
Reading symbols from /usr/lib/libpcre.so.3...done.
Loaded symbols for /usr/lib/libpcre.so.3
0xb7f1a424 in __kernel_vsyscall ()
(gdb) continue
Continuing.

Program received signal SIGTERM, Terminated.
[Switching to Thread 0xb7c72960 (LWP 1973)]
0xb7f1a424 in __kernel_vsyscall ()
(gdb) backtrace full
#0  0xb7f1a424 in __kernel_vsyscall ()
No symbol table info available.
#1  0xb7e1d3a5 in sem_wait@@GLIBC_2.1 () from /lib/i686/cmov/libpthread.so.0
No symbol table info available.
#2  0x0804b2c8 in sftp_request_wait (req=0xfffffe00, type=5 '\005', expect_type=103 'g', outbuf=0xbfb36488) at sshfs.c:1616
    err = <value optimized out>
#3  0x0804c28a in sftp_request_iov (type=<value optimized out>, iov=<value optimized out>, count=<value optimized out>, expect_type=103 'g', outbuf=0xbfb36488) at sshfs.c:1735
    req = (struct request *) 0xb7306f90
#4  0x0804c2cf in sftp_request (type=0 '\0', buf=<value optimized out>, expect_type=<value optimized out>, outbuf=0xbfb36488) at sshfs.c:1744
    iov = {iov_base = 0xb730acb0, iov_len = 20}
#5  0x0804f9ef in sshfs_read (path=0xb730b0d0 "/home/user/media/series/DVD/DVD_S05/DVD_S05D05B.iso", rbuf=0xb731f6f8 "", size=65536, offset=6602395648, fi=0xbfb365bc) at sshfs.c:2268
    sf = (struct sshfs_file *) 0xb73012e0
#6  0xb7ef826e in fuse_fs_read () from /usr/lib/libfuse.so.2
No symbol table info available.
#7  0xb7efca13 in ?? () from /usr/lib/libfuse.so.2
No symbol table info available.
#8  0x098485a0 in ?? ()
No symbol table info available.
#9  0xb730b0d0 in ?? ()
No symbol table info available.
#10 0xb731f6f8 in ?? ()
No symbol table info available.
#11 0x00010000 in ?? ()
No symbol table info available.
#12 0x89889000 in ?? ()
No symbol table info available.
#13 0x00000001 in ?? ()
No symbol table info available.
#14 0xbfb365bc in ?? ()
No symbol table info available.
#15 0x00000000 in ?? ()
No symbol table info available.
(gdb) info registers
eax            0xfffffe00   -512
ecx            0x80 128
edx            0x0  0
ebx            0xb7306f94   -1221562476
esp            0xbfb36390   0xbfb36390
ebp            0xbfb363e8   0xbfb363e8
esi            0x0  0
edi            0x67 103
eip            0xb7f1a424   0xb7f1a424 <__kernel_vsyscall+16>
eflags         0x246    [ PF ZF IF ]
cs             0x73 115
ss             0x7b 123
ds             0x7b 123
es             0x7b 123
fs             0x0  0
gs             0x33 51
(gdb) thread apply all backtrace

Thread 2 (Thread 0xb7c4fb90 (LWP 1974)):
#0  0xb7f1a424 in __kernel_vsyscall ()
#1  0xb7e1e09b in read () from /lib/i686/cmov/libpthread.so.0
#2  0x08049fa9 in do_read (buf=<value optimized out>) at sshfs.c:1111
#3  0x0804a9a2 in sftp_read (type=0xb7c4f367 "g", buf=0xb7c4f34c) at sshfs.c:1142
#4  0x0804aac1 in process_requests (data_=0x0) at sshfs.c:1202
#5  0xb7e174b0 in start_thread () from /lib/i686/cmov/libpthread.so.0
#6  0xb7d9638e in clone () from /lib/i686/cmov/libc.so.6

Thread 1 (Thread 0xb7c72960 (LWP 1973)):
#0  0xb7f1a424 in __kernel_vsyscall ()
#1  0xb7e1d3a5 in sem_wait@@GLIBC_2.1 () from /lib/i686/cmov/libpthread.so.0
#2  0x0804b2c8 in sftp_request_wait (req=0xfffffe00, type=5 '\005', expect_type=103 'g', outbuf=0xbfb36488) at sshfs.c:1616
#3  0x0804c28a in sftp_request_iov (type=<value optimized out>, iov=<value optimized out>, count=<value optimized out>, expect_type=103 'g', outbuf=0xbfb36488) at sshfs.c:1735
#4  0x0804c2cf in sftp_request (type=0 '\0', buf=<value optimized out>, expect_type=<value optimized out>, outbuf=0xbfb36488) at sshfs.c:1744
#5  0x0804f9ef in sshfs_read (path=0xb730b0d0 "/home/user/download/usenet/series/DVD/DVD_S05/DVD_S05D05B.iso", rbuf=0xb731f6f8 "", size=65536, offset=6602395648, fi=0xbfb365bc) at sshfs.c:2268
#6  0xb7ef826e in fuse_fs_read () from /usr/lib/libfuse.so.2
#7  0xb7efca13 in ?? () from /usr/lib/libfuse.so.2
#8  0x098485a0 in ?? ()
#9  0xb730b0d0 in ?? ()
#10 0xb731f6f8 in ?? ()
#11 0x00010000 in ?? ()
#12 0x89889000 in ?? ()
#13 0x00000001 in ?? ()
#14 0xbfb365bc in ?? ()
#15 0x00000000 in ?? ()
(gdb) quit
The program is running.  Quit anyway (and detach it)? (y or n) Detaching from program: /usr/bin/sshfs, process 1973

Re: how can i stop a running process to do a backtrace?

[Paul Pluzhnikov]

On Sun, Jul 20, 2008 at 11:05 AM, Jelle de Jong
<jelledejong@powercraft.nl> wrote:

> I managed to create three backtraces after the sshfs program froze. I
> attached one of them as example. I something is obvious wrong with the
> report please tell me.

I am no expert to FUSE, and this doesn't belong on this list,
but I think your stack trace simply means that sshfs is blocked
waiting for 'sftp' to finish whatever it is doing.

As such, it will likely be of very little use to sshfs developers;
you'll need to see whether the are any sftp or ssh processes, and if
so what *they* are blocked on. And if they aren't blocked, then
maybe you just need to be more patient -- after all, sshfs can't
be any faster than the underlying transport, can it?


-- 
Paul Pluzhnikov