Remote core file debugging

Remote core file debugging

[Eli Zaretskii]

Is gdbserver supposed to support remote post-mortem debugging using a
core file on a remote machine?

I have a situation where producing a backtrace from a dead program's
core file by running GDB on the machine where the application crashed
is impractical.  This is because the program has huge amounts of debug
info, so starting GDB eats up all memory, which is a no-no, since the
application which restarted itself after a crash must be up at all
times, and eating up too much memory causes the OS to kill
applications.  So I'm looking for ways of producing the equivalent of
"bt full", but without firing up GDB on the target system (a Red Hat
box).

If gdbserver does not support this, what alternatives do I have?

TIA

Re: Remote core file debugging

[David Daney]

Eli Zaretskii wrote:
> Is gdbserver supposed to support remote post-mortem debugging using a
> core file on a remote machine?
> 
> I have a situation where producing a backtrace from a dead program's
> core file by running GDB on the machine where the application crashed
> is impractical.  This is because the program has huge amounts of debug
> info, so starting GDB eats up all memory, which is a no-no, since the
> application which restarted itself after a crash must be up at all
> times, and eating up too much memory causes the OS to kill
> applications.  So I'm looking for ways of producing the equivalent of
> "bt full", but without firing up GDB on the target system (a Red Hat
> box).
> 
> If gdbserver does not support this, what alternatives do I have?
> 

I must be missing something, but why can't you move the core file to the 
debugging host and debug it there with gdb?

David Daney

Re: Remote core file debugging

[Eli Zaretskii]

> Date: Thu, 30 Apr 2009 12:08:02 -0700
> From: David Daney <ddaney@caviumnetworks.com>
> CC: gdb@sourceware.org
> 
> I must be missing something, but why can't you move the core file to the 
> debugging host and debug it there with gdb?

Sorry, I should have said that in the first place: because it's huge,
and there are quite a few of them.  It's not a single target system,
either: there are a few dozen of them, so copying huge core files from
all of them would be at least time consuming if not impractical.

Re: Remote core file debugging

[Michael Snyder]

Eli Zaretskii wrote:
>> Date: Thu, 30 Apr 2009 12:08:02 -0700
>> From: David Daney <ddaney@caviumnetworks.com>
>> CC: gdb@sourceware.org
>>
>> I must be missing something, but why can't you move the core file to the 
>> debugging host and debug it there with gdb?
> 
> Sorry, I should have said that in the first place: because it's huge,
> and there are quite a few of them.  It's not a single target system,
> either: there are a few dozen of them, so copying huge core files from
> all of them would be at least time consuming if not impractical.

GDBserver cannot currently do what you want.

Options include
1) Teach gdbserver to understand core files, and
2) Write a corefile server that understands gdb serial protocol.

#2 might be a bit easier, at least if you start with libbfd
which already knows how to read core files.  Your server would
only need to understand a reasonably small subset of commands
(memory read, register read, possibly threads if your system
includes them).

Re: Remote core file debugging

[Hui Zhu]

On Fri, May 1, 2009 at 03:23, Michael Snyder <msnyder@vmware.com> wrote:
> Eli Zaretskii wrote:
>>>
>>> Date: Thu, 30 Apr 2009 12:08:02 -0700
>>> From: David Daney <ddaney@caviumnetworks.com>
>>> CC: gdb@sourceware.org
>>>
>>> I must be missing something, but why can't you move the core file to the
>>> debugging host and debug it there with gdb?
>>
>> Sorry, I should have said that in the first place: because it's huge,
>> and there are quite a few of them.  It's not a single target system,
>> either: there are a few dozen of them, so copying huge core files from
>> all of them would be at least time consuming if not impractical.
>
> GDBserver cannot currently do what you want.
>
> Options include
> 1) Teach gdbserver to understand core files, and
> 2) Write a corefile server that understands gdb serial protocol.
>
> #2 might be a bit easier, at least if you start with libbfd
> which already knows how to read core files.  Your server would
> only need to understand a reasonably small subset of commands
> (memory read, register read, possibly threads if your system
> includes them).

#2 sounds cool.  Maybe it can be optional function of gdbserver.

Re: Remote core file debugging

[Daniel Jacobowitz]

On Thu, Apr 30, 2009 at 10:03:24PM +0300, Eli Zaretskii wrote:
> Is gdbserver supposed to support remote post-mortem debugging using a
> core file on a remote machine?

No.  Can you move the core file to the remote machine?  You also need
a copy of the target application and libraries on the remote machine,
as with live remote debugging.

-- 
Daniel Jacobowitz
CodeSourcery

Re: Remote core file debugging

[Eli Zaretskii]

> Date: Thu, 30 Apr 2009 15:11:01 -0400
> From: Daniel Jacobowitz <drow@false.org>
> Cc: gdb@sourceware.org
> 
> Can you move the core file to the remote machine?

See my response to David: I'd like to avoid that, if possible.

> You also need a copy of the target application and libraries on the
> remote machine, as with live remote debugging.

These are already available on the remote: the application is built
there and then pushed to the targets.

Re: Remote core file debugging

[Mark Kettenis]

> Date: Thu, 30 Apr 2009 22:03:24 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> 
> Is gdbserver supposed to support remote post-mortem debugging using a
> core file on a remote machine?

I don't think it is.

> I have a situation where producing a backtrace from a dead program's
> core file by running GDB on the machine where the application crashed
> is impractical.  This is because the program has huge amounts of debug
> info, so starting GDB eats up all memory, which is a no-no, since the
> application which restarted itself after a crash must be up at all
> times, and eating up too much memory causes the OS to kill
> applications.  So I'm looking for ways of producing the equivalent of
> "bt full", but without firing up GDB on the target system (a Red Hat
> box).
> 
> If gdbserver does not support this, what alternatives do I have?

Copy the core file and binaries to a somewhat more powerful machine
and use a cross-gdb (the same you'd probably use with gdbserver)?

Re: Remote core file debugging

[Eli Zaretskii]

> Date: Thu, 30 Apr 2009 23:37:05 +0200 (CEST)
> From: Mark Kettenis <mark.kettenis@xs4all.nl>
> CC: gdb@sourceware.org
> 
> > If gdbserver does not support this, what alternatives do I have?
> 
> Copy the core file and binaries to a somewhat more powerful machine
> and use a cross-gdb (the same you'd probably use with gdbserver)?

No, I don't need a cross-gdb, as both systems are Red Hat.

Thanks.