libSegFault and just in time debugging

libSegFault and just in time debugging

[Michael Snyder]

Got an idea to pass in front of you...

If you don't know how libSegFault works, it is a dynamic library that
comes packaged with glibc.  To use it, you get the runtime loader to
preload it and link it to your programs, using either the environment
variable LD_PRELOAD or the file /etc/ld.so.preload.  Upon loading,
it takes over the signal handling vectors for half a dozen fatal signals
(SEGV, BUS, ILL, FPU...), but it does so *before* the main user program
is invoked -- so there's no interferance with programs that want to use
those vectors themselves.  If the program *doesn't* handle the signal,
then libSegFault catches it and prints some useful debugging info to
the system console or syslog (before passing the signal back to the
kernel so that a core file can be generated).

What if, instead of doing that, libSegFault (or another similar library)
were to open a socket to a daemon and say "I caught a crash -- what
do you want me to do?".  And then wait for a reply.  All that can be
done with async-signal-safe function calls.

The daemon, because it is NOT in a signal handler, can do anything
it wants, eg. open a GUI window and say "Program XYZ has crashed:
would you like to debug it?"  User clicks a button, daemon fires up
gdb, attaches to crashing program, then responds to the signal handler
library with a message saying "get out of the way, please...".  The
library removes itself from the signal vector, re-raises the signal,
and returns.  And gdb finds itself at the point where the signal was
originally raised.

If the user doesn't want to debug it, the daemon can tell the library
to just let it crash, or to log the information a la libSegFault, or
any number of other things.

Interesting?

Re: libSegFault and just in time debugging

[Daniel Jacobowitz]

On Fri, Jun 29, 2007 at 12:10:32PM -0700, Michael Snyder wrote:
> What if, instead of doing that, libSegFault (or another similar library)
> were to open a socket to a daemon and say "I caught a crash -- what
> do you want me to do?".  And then wait for a reply.  All that can be
> done with async-signal-safe function calls.

It's a brilliant idea.  Ubuntu did it :-)  It uses the Linux kernel's
core handling support, and is called apport.

  https://wiki.ubuntu.com/Apport

You'll notice a bit of a way down that there's GDB output; in this
case it comes from a core dump, but I'm pretty sure the same kernel
hooks can be used to take control before the core is dumped.  I'd have
to check.

-- 
Daniel Jacobowitz
CodeSourcery

Re: libSegFault and just in time debugging

[Michael Snyder]

> On Fri, Jun 29, 2007 at 12:10:32PM -0700, Michael Snyder wrote:
> > What if, instead of doing that, libSegFault (or another similar library)
> > were to open a socket to a daemon and say "I caught a crash -- what
> > do you want me to do?".  And then wait for a reply.  All that can be
> > done with async-signal-safe function calls.
>
> It's a brilliant idea.  Ubuntu did it :-)  It uses the Linux kernel's
> core handling support, and is called apport.

Yes, apport is slick, but it relies on kernel mods.
This doesn't.  In fact, it isn't even peculiar to Linux, it would
work on any glibc system, and in principle even on systems
that don't use glibc.  Probably any unix, and even cygwin.

Re: libSegFault and just in time debugging

[Daniel Jacobowitz]

On Fri, Jun 29, 2007 at 12:52:42PM -0700, Michael Snyder wrote:
> Yes, apport is slick, but it relies on kernel mods.
> This doesn't.  In fact, it isn't even peculiar to Linux, it would
> work on any glibc system, and in principle even on systems
> that don't use glibc.  Probably any unix, and even cygwin.

FYI, cygwin already does this, too; it can invoke dumper (to generate core
dumps) or GDB.  I imagine you could invoke apport from a preloaded
library easily; after all, you can hook into it from Python exceptions.

-- 
Daniel Jacobowitz
CodeSourcery

Re: libSegFault and just in time debugging

[Jim Ingham]

We did this with Mac OS X for a while (the way Mach Exceptions work  
it's a little easier to implement, you don't need a special library or  
anything like that).  It's pretty neat.  We were doing it  
automatically with all processes on the system, which is even handier,  
but you do have to be careful - if you are attaching with gdb rather  
than dumping core
On Jun 29, 2007, at 1:00 PM, Daniel Jacobowitz wrote:

> On Fri, Jun 29, 2007 at 12:52:42PM -0700, Michael Snyder wrote:
>> Yes, apport is slick, but it relies on kernel mods.
>> This doesn't.  In fact, it isn't even peculiar to Linux, it would
>> work on any glibc system, and in principle even on systems
>> that don't use glibc.  Probably any unix, and even cygwin.
>
> FYI, cygwin already does this, too; it can invoke dumper (to  
> generate core
> dumps) or GDB.  I imagine you could invoke apport from a preloaded
> library easily; after all, you can hook into it from Python  
> exceptions.
>
> -- 
> Daniel Jacobowitz
> CodeSourcery

Re: libSegFault and just in time debugging

[Jim Ingham]

Oops, hit send too soon...

Meant to say if you are automatically attaching to everything on the  
system, you have to be careful about auto-launched daemons, because  
they won't die all the way while waiting for the connection to the  
debugger, and that can cause them not to get restarted.  Not a problem  
in general with released systems, where hopefully the daemons aren't  
crashing all that often.  But with development systems it could cause  
a problem...

Jim

On Jun 29, 2007, at 1:12 PM, Jim Ingham wrote:

> We did this with Mac OS X for a while (the way Mach Exceptions work  
> it's a little easier to implement, you don't need a special library  
> or anything like that).  It's pretty neat.  We were doing it  
> automatically with all processes on the system, which is even  
> handier, but you do have to be careful - if you are attaching with  
> gdb rather than dumping core
> On Jun 29, 2007, at 1:00 PM, Daniel Jacobowitz wrote:
>
>> On Fri, Jun 29, 2007 at 12:52:42PM -0700, Michael Snyder wrote:
>>> Yes, apport is slick, but it relies on kernel mods.
>>> This doesn't.  In fact, it isn't even peculiar to Linux, it would
>>> work on any glibc system, and in principle even on systems
>>> that don't use glibc.  Probably any unix, and even cygwin.
>>
>> FYI, cygwin already does this, too; it can invoke dumper (to  
>> generate core
>> dumps) or GDB.  I imagine you could invoke apport from a preloaded
>> library easily; after all, you can hook into it from Python  
>> exceptions.
>>
>> -- 
>> Daniel Jacobowitz
>> CodeSourcery
>

Re: libSegFault and just in time debugging

[Michael Snyder]

[Daniel Jacobowitz]
> FYI, cygwin already does this, too; it can invoke dumper (to
> generate core dumps) or GDB.

That's right -- I'd forgotten about that.

[Jim Ingham:]

> We did this with Mac OS X for a while (the way Mach Exceptions work
> it's a little easier to implement, you don't need a special library or
> anything like that).  It's pretty neat.  We were doing it
> automatically with all processes on the system, which is even handier,
> but you do have to be careful - if you are attaching with gdb rather
> than dumping core.

Of course the old Mac model ("macsbug"?) is the classic instance
of just in time debugging.  And then (surprise), Windows had it too!

I'm just wondering if linux/unix users might not like to have the same
thing.

Michael

Re: libSegFault and just in time debugging

[Michael Snyder]

> Oops, hit send too soon...
>
> Meant to say if you are automatically attaching to everything on the
> system, you have to be careful about auto-launched daemons, because
> they won't die all the way while waiting for the connection to the
> debugger, and that can cause them not to get restarted.  Not a problem
> in general with released systems, where hopefully the daemons aren't
> crashing all that often.  But with development systems it could cause
> a problem...

Right.  But there's considerable flexibility in, eg. at what point in your
boot scripts do you introduce the LD_PRELOAD variable or the
/etc/ld.so.preload file.  You could start your system daemons first,
for instance, and then they wouldn't get the preload library.  And the
env variable would only apply to children of a specific shell.  Could
even be turned on by a user, thus not affecting any root or other-user
processes.

Or, you could introduce the /etc file before boot, and get everything,
thus enabling you to debug crashing system daemons etc.

Re: libSegFault and just in time debugging

[Eli Zaretskii]

> Date: Fri, 29 Jun 2007 16:00:00 -0400
> From: Daniel Jacobowitz <drow@false.org>
> Cc: gdb@sourceware.org
> 
> FYI, cygwin already does this, too; it can invoke dumper (to generate core
> dumps) or GDB.

By Cygwin does this using a Windows-specific technique, doesn't it?

Re: libSegFault and just in time debugging

[Eli Zaretskii]

> Date: Sat, 30 Jun 2007 13:52:35 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> CC: msnyder@sonic.net, gdb@sourceware.org
> 
> By Cygwin does this using a Windows-specific technique, doesn't it?
  ^^

Meant to say "But", of course.

Re: libSegFault and just in time debugging

[Daniel Jacobowitz]

On Sat, Jun 30, 2007 at 01:52:35PM +0300, Eli Zaretskii wrote:
> > Date: Fri, 29 Jun 2007 16:00:00 -0400
> > From: Daniel Jacobowitz <drow@false.org>
> > Cc: gdb@sourceware.org
> > 
> > FYI, cygwin already does this, too; it can invoke dumper (to generate core
> > dumps) or GDB.
> 
> By Cygwin does this using a Windows-specific technique, doesn't it?

Sure.  But LD_PRELOAD is a Unix-specific technique.  I don't think
there's any way around it - when you want to deal with program crashes
and exceptions, you do something at least a bit platform specific.

-- 
Daniel Jacobowitz
CodeSourcery

Re: libSegFault and just in time debugging

[Michael Snyder]

> > > FYI, cygwin already does this, too; it can invoke dumper (to generate
core
> > > dumps) or GDB.
> >
> > By Cygwin does this using a Windows-specific technique, doesn't it?
>
> Sure.  But LD_PRELOAD is a Unix-specific technique.  I don't think
> there's any way around it - when you want to deal with program crashes
> and exceptions, you do something at least a bit platform specific.

Either way, right now we don't have a JIT debugging technique
for Unix.  The question is, is it worth having one?

Re: libSegFault and just in time debugging

[Mark Kettenis]

> From: "Michael Snyder" <msnyder@sonic.net>
> Cc: <gdb@sourceware.org>
> 
> > > > FYI, cygwin already does this, too; it can invoke dumper (to generate
> core
> > > > dumps) or GDB.
> > >
> > > By Cygwin does this using a Windows-specific technique, doesn't it?
> >
> > Sure.  But LD_PRELOAD is a Unix-specific technique.  I don't think
> > there's any way around it - when you want to deal with program crashes
> > and exceptions, you do something at least a bit platform specific.

LD_PRELOAD is actually ELF-specific.

> Either way, right now we don't have a JIT debugging technique
> for Unix.  The question is, is it worth having one?

Given that the question pops up every now and then on the mailing
list, I'd say it is (even if I've never felt the desire for it
myself).

Mark

RE: libSegFault and just in time debugging

[Dave Korn]

On 29 June 2007 20:11, Michael Snyder wrote:

> What if, instead of doing that, libSegFault (or another similar library)
> were to open a socket to a daemon and say "I caught a crash -- what
> do you want me to do?".  And then wait for a reply.  All that can be
> done with async-signal-safe function calls.
> 
> The daemon, because it is NOT in a signal handler, can do anything
> it wants, eg. open a GUI window and say "Program XYZ has crashed:
> would you like to debug it?"  User clicks a button, daemon fires up
> gdb, attaches to crashing program, then responds to the signal handler
> library with a message saying "get out of the way, please...".  The
> library removes itself from the signal vector, re-raises the signal,
> and returns.  And gdb finds itself at the point where the signal was
> originally raised.

  Implementation-wise, wouldn't it be easier to just build a mini-gdbserver[*]
into libsegfault itself?  It would save having to do the get-out-of-the-way
dance.


    cheers,
      DaveK

[*] - sorry, vague terminology here - i mean a remote protocol talker, like
the embedded targets use, not the big i'm-a-separate-process gdbserver.
-- 
Can't think of a witty .sigline today....

Re: libSegFault and just in time debugging

[Eli Zaretskii]

> Date: Sat, 30 Jun 2007 11:49:04 -0400
> From: Daniel Jacobowitz <drow@false.org>
> Cc: msnyder@sonic.net, gdb@sourceware.org
> 
> > By Cygwin does this using a Windows-specific technique, doesn't it?
> 
> Sure.  But LD_PRELOAD is a Unix-specific technique.

There's a world of difference between preloading a library and hooking
into the Windows JIT debug machinery.

Re: libSegFault and just in time debugging

[Eli Zaretskii]

> From: "Michael Snyder" <msnyder@sonic.net>
> Cc: <gdb@sourceware.org>
> Date: Sat, 30 Jun 2007 09:03:30 -0700
> 
> Either way, right now we don't have a JIT debugging technique
> for Unix.  The question is, is it worth having one?

Yes, it is.

Re: libSegFault and just in time debugging

[Michael Snyder]

----- Original Message ----- 
From: "Dave Korn" <dave.korn@artimi.com>
To: "'Michael Snyder'" <msnyder@sonic.net>; <gdb@sourceware.org>
Sent: Saturday, June 30, 2007 10:27 AM
Subject: RE: libSegFault and just in time debugging


> On 29 June 2007 20:11, Michael Snyder wrote:
>
> > What if, instead of doing that, libSegFault (or another similar library)
> > were to open a socket to a daemon and say "I caught a crash -- what
> > do you want me to do?".  And then wait for a reply.  All that can be
> > done with async-signal-safe function calls.
> >
> > The daemon, because it is NOT in a signal handler, can do anything
> > it wants, eg. open a GUI window and say "Program XYZ has crashed:
> > would you like to debug it?"  User clicks a button, daemon fires up
> > gdb, attaches to crashing program, then responds to the signal handler
> > library with a message saying "get out of the way, please...".  The
> > library removes itself from the signal vector, re-raises the signal,
> > and returns.  And gdb finds itself at the point where the signal was
> > originally raised.
>
>   Implementation-wise, wouldn't it be easier to just build a
mini-gdbserver[*]
> into libsegfault itself?

Maybe -- but how mini?  We're in a signal handler, so there are limits
to what we can do.  On the other hand, with the "real" gdbserver, or
gdb, we should be able to get the process back to virtually the state
that it was in when the signal occurred, and debug it as if it had been
running under the debugger all along.  Better than a core file...

On the other hand, I have been thinking that maybe gdbserver could
be the daemon...  then it could just fork itself and attach to the crashed
process.