Backtrace extraction ONLY gdb

Backtrace extraction ONLY gdb

[Maucci, Cyrille]

Hello dear gdb community,


Have you guys ever heared things like having gdb installed on a mission critical production system is prohibited for security reasons ?
('Debuggers' being part of the hackers swiss army knife...)

So, that is the kind of barrier I need to work around because for support reasons, I dramatically need a single feature of gdb on such systems which is the backtrace extraction feature.

Is there any possibility to 'recompile' gdb where only the 'bt' (or 'threads apply all bt') feature would be enabled?
Such a gdb would not be a debugger but just a backtrace extraction tool... and therefore would not be characterized as dangerous.

Or do you know any tool that would do this?


Thank you very much in advance
++Cyrille

Re: Backtrace extraction ONLY gdb

[Eli Zaretskii]

> From: "Maucci, Cyrille" <cyrille.maucci@hp.com>
> CC: "Maucci, Cyrille" <cyrille.maucci@hp.com>
> Date: Mon, 21 Mar 2011 17:13:02 +0000
> 
> So, that is the kind of barrier I need to work around because for support reasons, I dramatically need a single feature of gdb on such systems which is the backtrace extraction feature.
> 
> Is there any possibility to 'recompile' gdb where only the 'bt' (or 'threads apply all bt') feature would be enabled?

If that system is GNU/Linux (or anything else glibc-based), then you
should know that there's a glibc function that can produce a
backtrace.

RE: Backtrace extraction ONLY gdb

[Maucci, Cyrille]

Hi Eli, All,

Yes that is a glibc based system.
I imagine you are referring to http://www.gnu.org/s/libc/manual/html_node/Backtraces.html

From what I read it gives a very 'raw' output, very far from the beautiful gdb backtrace with function names and function arguments values.
But that is a beginning.

Thanks a lot
++Cyrille

-----Original Message-----
From: Eli Zaretskii [mailto:eliz@gnu.org] 
Sent: Monday, March 21, 2011 7:49 PM
To: Maucci, Cyrille
Cc: gdb@sourceware.org
Subject: Re: Backtrace extraction ONLY gdb

> From: "Maucci, Cyrille" <cyrille.maucci@hp.com>
> CC: "Maucci, Cyrille" <cyrille.maucci@hp.com>
> Date: Mon, 21 Mar 2011 17:13:02 +0000
> 
> So, that is the kind of barrier I need to work around because for support reasons, I dramatically need a single feature of gdb on such systems which is the backtrace extraction feature.
> 
> Is there any possibility to 'recompile' gdb where only the 'bt' (or 'threads apply all bt') feature would be enabled?

If that system is GNU/Linux (or anything else glibc-based), then you should know that there's a glibc function that can produce a backtrace.

Re: Backtrace extraction ONLY gdb

[Joel Brobecker]

> >From what I read it gives a very 'raw' output, very far from the
> >beautiful gdb backtrace with function names and function arguments
> >values.

You would be indeed missing the function argument values.  But
getting the function names from the addresses can be done later on,
using a tool such as addr2line.

The best option, IMO, is to get your process to dump a core file.
You'd then debug on a separate host using that core file. A few
companies I know do that routinely.

-- 
Joel

RE: Backtrace extraction ONLY gdb

[Maucci, Cyrille]

Hi Joel

>> You'd then debug on a separate host using that core file. A few companies I know do that routinely.
On the principle I agree.
However, it gets complex when:
	. the core files are multi GB large and the 'separate' host is not located on the customer network.
	. you not only need the core files but the libs to debug the core. Indeed, you could argue to setup a 'replica' system to read the core file, but when you've got dozens of customers with dozens of different software versions, this becomes the nightmare.

This is why the method we had put in place years ago is a kind of automated gdb backtrace extraction upon core dump detection (ends up with a very small file a few KB at most) with more or less an automatic 'sending' of that file from the customer to the support team by mail.

If the backtrace is a 'known' backtrace or if the backtrace is enough for us to understand the problem, then no need to get the full core.
Upon rare circumstances, we need the full core (that represents maybe less than 5% of the cases).

Again, that is part of an efficient support line.

Thanks anyway
++Cyrille
 

-----Original Message-----
From: Joel Brobecker [mailto:brobecker@adacore.com] 
Sent: Wednesday, March 23, 2011 11:44 PM
To: Maucci, Cyrille
Cc: Eli Zaretskii; gdb@sourceware.org
Subject: Re: Backtrace extraction ONLY gdb

> >From what I read it gives a very 'raw' output, very far from the 
> >beautiful gdb backtrace with function names and function arguments 
> >values.

You would be indeed missing the function argument values.  But getting the function names from the addresses can be done later on, using a tool such as addr2line.

The best option, IMO, is to get your process to dump a core file.
You'd then debug on a separate host using that core file. A few companies I know do that routinely.

--
Joel

Re: Backtrace extraction ONLY gdb

[Jan Kratochvil]

On Wed, 23 Mar 2011 23:51:42 +0100, Maucci, Cyrille wrote:
> 	. the core files are multi GB large and the 'separate' host is not located on the customer network.
> 	. you not only need the core files but the libs to debug the core.
> 	Indeed, you could argue to setup a 'replica' system to read the core
> 	file, but when you've got dozens of customers with dozens of different
> 	software versions, this becomes the nightmare.

This is solved by the ABRT project https://fedorahosted.org/abrt/wiki
automatically getting the right binaries and debug info versions by the
build-id present in any binary and core file (with properly configured GCC).

It is now deploying a http://fedoraproject.org/wiki/Features/RetraceServer .
If there are security concerns about running GDB locally (as ABRT has been
doing before RetraceServer so far) isn't there located a host for backtraces
in LAN where multi-GB transfers are not an issue?


Regards,
Jan

RE: Backtrace extraction ONLY gdb

[Maucci, Cyrille]

Hey Jan,

Thanks for the pointers.
The ABRT project is very interesting.

Concerning the presence of a host on the LAN where multi-GB xfers are not an issue, it is very rarely the case.

++Cyrille

-----Original Message-----
From: Jan Kratochvil [mailto:jan.kratochvil@redhat.com] 
Sent: Thursday, March 24, 2011 6:25 AM
To: Maucci, Cyrille
Cc: Joel Brobecker; Eli Zaretskii; gdb@sourceware.org
Subject: Re: Backtrace extraction ONLY gdb

On Wed, 23 Mar 2011 23:51:42 +0100, Maucci, Cyrille wrote:
> 	. the core files are multi GB large and the 'separate' host is not located on the customer network.
> 	. you not only need the core files but the libs to debug the core.
> 	Indeed, you could argue to setup a 'replica' system to read the core
> 	file, but when you've got dozens of customers with dozens of different
> 	software versions, this becomes the nightmare.

This is solved by the ABRT project https://fedorahosted.org/abrt/wiki
automatically getting the right binaries and debug info versions by the build-id present in any binary and core file (with properly configured GCC).

It is now deploying a http://fedoraproject.org/wiki/Features/RetraceServer .
If there are security concerns about running GDB locally (as ABRT has been doing before RetraceServer so far) isn't there located a host for backtraces in LAN where multi-GB transfers are not an issue?


Regards,
Jan

RE: Backtrace extraction ONLY gdb

[Maucci, Cyrille]

Hi All,

I've just found http://sourceware.org/frysk/manpages/fstack.1.html.

This looks like what I need.

++Cyrille 

-----Original Message-----
From: Maucci, Cyrille 
Sent: Sunday, March 27, 2011 10:25 PM
To: 'Jan Kratochvil'
Cc: Joel Brobecker; Eli Zaretskii; gdb@sourceware.org
Subject: RE: Backtrace extraction ONLY gdb

Hey Jan,

Thanks for the pointers.
The ABRT project is very interesting.

Concerning the presence of a host on the LAN where multi-GB xfers are not an issue, it is very rarely the case.

++Cyrille

-----Original Message-----
From: Jan Kratochvil [mailto:jan.kratochvil@redhat.com]
Sent: Thursday, March 24, 2011 6:25 AM
To: Maucci, Cyrille
Cc: Joel Brobecker; Eli Zaretskii; gdb@sourceware.org
Subject: Re: Backtrace extraction ONLY gdb

On Wed, 23 Mar 2011 23:51:42 +0100, Maucci, Cyrille wrote:
> 	. the core files are multi GB large and the 'separate' host is not located on the customer network.
> 	. you not only need the core files but the libs to debug the core.
> 	Indeed, you could argue to setup a 'replica' system to read the core
> 	file, but when you've got dozens of customers with dozens of different
> 	software versions, this becomes the nightmare.

This is solved by the ABRT project https://fedorahosted.org/abrt/wiki
automatically getting the right binaries and debug info versions by the build-id present in any binary and core file (with properly configured GCC).

It is now deploying a http://fedoraproject.org/wiki/Features/RetraceServer .
If there are security concerns about running GDB locally (as ABRT has been doing before RetraceServer so far) isn't there located a host for backtraces in LAN where multi-GB transfers are not an issue?


Regards,
Jan

Re: Backtrace extraction ONLY gdb

[Jan Kratochvil]

On Fri, 22 Apr 2011 22:50:10 +0200, Maucci, Cyrille wrote:
> I've just found http://sourceware.org/frysk/manpages/fstack.1.html.

FYI Frysk is no longer maintained for some years, compatibility with existing
systems may be reduced.


Regards,
Jan

Re: Backtrace extraction ONLY gdb

[Pedro Alves]

On Wednesday 23 March 2011 22:51:42, Maucci, Cyrille wrote:
>         . you not only need the core files but the libs to debug the core. Indeed, you could argue to setup a 'replica' system to read the core file, but when you've got dozens of customers with dozens of different software versions, this becomes the nightmare.
> 

FYI, you don't need a full replica system to read the core file.  You only
need copies of all the libraries that where loaded at the time of
the crash, and point gdb at them with "set sysroot".

-- 
Pedro Alves

RE: Backtrace extraction ONLY gdb

[Maucci, Cyrille]

Hi Alves,

Yes I know that.

++Cyrille

-----Original Message-----
From: Pedro Alves [mailto:pedro@codesourcery.com] 
Sent: Thursday, March 24, 2011 11:25 AM
To: gdb@sourceware.org
Cc: Maucci, Cyrille; Joel Brobecker; Eli Zaretskii
Subject: Re: Backtrace extraction ONLY gdb

On Wednesday 23 March 2011 22:51:42, Maucci, Cyrille wrote:
>         . you not only need the core files but the libs to debug the core. Indeed, you could argue to setup a 'replica' system to read the core file, but when you've got dozens of customers with dozens of different software versions, this becomes the nightmare.
> 

FYI, you don't need a full replica system to read the core file.  You only need copies of all the libraries that where loaded at the time of the crash, and point gdb at them with "set sysroot".

--
Pedro Alves

Re: Backtrace extraction ONLY gdb

[Steffen Dettmer]

On Wed, Mar 23, 2011 at 11:43 PM, Joel Brobecker <brobecker@adacore.com> wrote:
> The best option, IMO, is to get your process to dump a core file.

If this isn't prohibited on the mission critical production
system for security reasons...  From the core file, potentially
all sorts of sensitive information could be revealed I think.

oki,

Steffen

Re: Backtrace extraction ONLY gdb

[Jan Kratochvil]

On Thu, 24 Mar 2011 11:28:40 +0100, Steffen Dettmer wrote:
> If this isn't prohibited on the mission critical production
> system for security reasons...  From the core file, potentially
> all sorts of sensitive information could be revealed I think.

The same security concerns apply to backtraces containing parameters - being
discussed here as the goal.  Sometimes the parameter is a cleartext password,
for example.


Regards,
Jan

Re: Backtrace extraction ONLY gdb

[Tom Tromey]

>>>>> "Cyrille" == Maucci, Cyrille <cyrille.maucci@hp.com> writes:

Cyrille> Is there any possibility to 'recompile' gdb where only the 'bt'
Cyrille> (or 'threads apply all bt') feature would be enabled?

Maybe with a lot of hacking.

If the process you need to attach to is always owned by a particular
user, you can play games with suid to restrict gdb to doing backtraces.
(Install gdb so that it is only executable by that user.  Then, make a
suid wrapper owned by that user that invokes `gdb -batch -p PID -ex
"thread apply all bt full"'.)

Alternatively, core files are good for this kind of use.  You can make
them on the production machine and then analyze them elsewhere.

Tom

RE: Backtrace extraction ONLY gdb

[Maucci, Cyrille]

Hey Tom,

Thanks for your reply.
The suid trick may do it... Though isn't there some story about suid scripts not being allowed?

>> You can make them on the production machine and then analyze them elsewhere.
Well if analyze them elsewhere is 'looking at the backtrace', yes.
If analyze them elsewhere is fully browsing the core file, it requires the exact same exe/libs versions as on the production system.
HP's gdb adds a great feature called packcore to ease the packing of everything needed for offline browsing with one command.

++Cyrille

 

-----Original Message-----
From: Tom Tromey [mailto:tromey@redhat.com] 
Sent: Tuesday, March 22, 2011 5:47 PM
To: Maucci, Cyrille
Cc: gdb@sourceware.org
Subject: Re: Backtrace extraction ONLY gdb

>>>>> "Cyrille" == Maucci, Cyrille <cyrille.maucci@hp.com> writes:

Cyrille> Is there any possibility to 'recompile' gdb where only the 'bt'
Cyrille> (or 'threads apply all bt') feature would be enabled?

Maybe with a lot of hacking.

If the process you need to attach to is always owned by a particular user, you can play games with suid to restrict gdb to doing backtraces.
(Install gdb so that it is only executable by that user.  Then, make a suid wrapper owned by that user that invokes `gdb -batch -p PID -ex "thread apply all bt full"'.)

Alternatively, core files are good for this kind of use.  You can make them on the production machine and then analyze them elsewhere.

Tom

Re: Backtrace extraction ONLY gdb

[Tom Tromey]

>>>>> "Cyrille" == Maucci, Cyrille <cyrille.maucci@hp.com> writes:

Cyrille> The suid trick may do it... Though isn't there some story about
Cyrille> suid scripts not being allowed?

Yeah.  Write a little C program instead.

Tom> You can make them on the production machine and then analyze them
Tom> elsewhere.

Cyrille> Well if analyze them elsewhere is 'looking at the backtrace',
Cyrille> yes.  If analyze them elsewhere is fully browsing the core
Cyrille> file, it requires the exact same exe/libs versions as on the
Cyrille> production system.

Yes.  You can just copy those back, of course.  Or, one thing I think
people do is keep one copy of the production build "locally", with full
debuginfo (separate or not), and then put a stripped copy on the
production machine.

Cyrille> HP's gdb adds a great feature called packcore to ease the
Cyrille> packing of everything needed for offline browsing with one
Cyrille> command.

Maybe this could be done from Python.  I don't know offhand.

Tom

Re: Backtrace extraction ONLY gdb

[Jan Kratochvil]

On Wed, 23 Mar 2011 23:22:04 +0100, Maucci, Cyrille wrote:
> HP's gdb adds a great feature called packcore to ease the packing of
> everything needed for offline browsing with one command.

$ eu-unstrip -n --core=core.1659|awk '{print $3;print $4}'|grep ^/|sort -u|tar cJT - >packcore.tar.xz
$ tar tJf packcore.tar.xz 
bin/sleep
lib64/ld-2.13.so
lib64/libc-2.13.so
usr/lib/debug/bin/sleep.debug
usr/lib/debug/lib/modules/2.6.35.11-83.fc14.x86_64/vdso/vdso.so.debug
usr/lib/debug/lib64/ld-2.13.so.debug
usr/lib/debug/lib64/libc-2.13.so.debug


Regards,
Jan

RE: Backtrace extraction ONLY gdb

[Maucci, Cyrille]

Hi Jan,

Thanks. My point was not that I had no way to do it. But only that one single command in gdb is much cleaner.
My own method was much dirtier than yours.

tar cvzf /var/tmp/packcore.tgz ${exe} ${corefile} $(echo "info sharedlibrary" |\
  gdb -batch -x /dev/stdin ${exe} ${corefile} | grep '^0x' | awk '{print $NF}');

++Cyrille

-----Original Message-----
From: Jan Kratochvil [mailto:jan.kratochvil@redhat.com] 
Sent: Thursday, March 24, 2011 6:21 PM
To: Maucci, Cyrille
Cc: Tom Tromey; gdb@sourceware.org
Subject: Re: Backtrace extraction ONLY gdb

On Wed, 23 Mar 2011 23:22:04 +0100, Maucci, Cyrille wrote:
> HP's gdb adds a great feature called packcore to ease the packing of 
> everything needed for offline browsing with one command.

$ eu-unstrip -n --core=core.1659|awk '{print $3;print $4}'|grep ^/|sort -u|tar cJT - >packcore.tar.xz $ tar tJf packcore.tar.xz bin/sleep lib64/ld-2.13.so lib64/libc-2.13.so usr/lib/debug/bin/sleep.debug usr/lib/debug/lib/modules/2.6.35.11-83.fc14.x86_64/vdso/vdso.so.debug
usr/lib/debug/lib64/ld-2.13.so.debug
usr/lib/debug/lib64/libc-2.13.so.debug


Regards,
Jan