breakpoints in shared libraries

breakpoints in shared libraries

[Florian Hackenberger]

Dear fellows,

This question pops up from time to time, but while skimming through the 
archives I was not able to find any suggestions. The problem I have is that I 
am not able to stop at breakpoints within some shared libraries. The layout 
of the program I'm trying to debug is as following (arrows indicate dynamic 
linkage against some library)

testHoap2Collision ->libhoap2.so -> libroboop.so, libnewmat.so, 
libController.so, libCGAL.so

The project is built using libtool and the autotools and the following flags: 
configure: --enable-debug=full
g++: -O0 -g3
All libraries apart from libController.so and libCGAL.so are built using the 
same autotools environment (one ./configure for all).

What I'm trying to do is to set a breakpoint within libroboop.so within the 
following method:

Quaternion::Quaternion(const Real angle, const ColumnVector & axis)
//! @brief Constructor.
{
   if(axis.Nrows() != 3)
   {
      cerr << "Quaternion::Quaternion, size of axis != 3" << endl;
      exit(1);
   }

   // make sure axis is a unit vector
   Real norm_axis = sqrt(DotProduct(axis, axis));

   if(norm_axis != 1)
   {
      cerr << "Quaternion::Quaternion(angle, axis), axis is not unit." << 
endl;
      cerr << "Make the axis unit." << endl;
      v_ = sin(angle/2) * axis/norm_axis;
   }
   else
      v_ = sin(angle/2) * axis;

   s_ = cos(angle/2);
}

It does not matter where the breakpoint is set within this method, but I would 
like to set it on the line:
cerr << "Make the axis unit." << endl;

Here is the declaration of the method:
Quaternion(const Real angle_in_rad, const ColumnVector & axis);
So, it's not inline or anything fancy, just a plain constructor.

The following happens:
After starting gdb (libtool --mode=execute gdb testHoap2Collision) I try the 
following:

(gdb) break quaternion.cpp:100
No source file named quaternion.cpp.
Make breakpoint pending on future shared library load? (y or [n]) y
Breakpoint 1 (quaternion.cpp:100) pending.

Ok, that's fine, now let's see if the breakpoint is set:
(gdb) run
Starting program: 
msl/debug/libhoap2/examples/testCollision/.libs/lt-testHoap2Collision
[Thread debugging using libthread_db enabled]
[New Thread -1216158016 (LWP 16481)]
Breakpoint 3 at 0xb7e73147: file msl/libroboop/source/quaternion.cpp, line 
100.
Pending breakpoint "quaternion.cpp:100" resolved
Quaternion::Quaternion(angle, axis), axis is not unit.
Make the axis unit.

That tells me:
1. The breakpoint was set.
2. The breakpoint did not work, as I can see the output "Make the axis unit." 
and that's the line where my breakpoint was supposed to interrupt the 
execution.

So that's the problem. Can anyone help me?

Here's some info about my environment:
GNU gdb 6.3
g++ (GCC) 3.4.3-20050110
A "typical call to g++":
g++ -DHAVE_CONFIG_H -I. 
-I/home/hacki/Documents/sandbox/master_project/msl-2.0_automake/msl/libhoap2/examples/testCollision 
-I../../.. 
-I/home/hacki/Documents/sandbox/master_project/msl-2.0_automake/msl/libhoap2 
-I/home/hacki/Documents/sandbox/master_project/msl-2.0_automake/msl/libnewmat 
-I/home/hacki/Documents/sandbox/master_project/msl-2.0_automake/msl/libroboop/source 
-I/usr/qt/3/include -I/usr/local/webots/include 
-I/home/hacki/Documents/sandbox/master_project/msl-2.0_automake/msl/external_include 
-I/home/hacki/Documents/sandbox/master_project/msl-2.0_automake/msl/../CGAL-3.1/include/CGAL/config/i686_Linux-2.6.15-suspend2-r3_g++-3.4.3 
-I/home/hacki/Documents/sandbox/master_project/msl-2.0_automake/msl/../CGAL-3.1/include    
-O0 -g3 -MT testHoap2Collision.o -MD -MP -MF ".deps/testHoap2Collision.Tpo" 
-c -o 
testHoap2Collision.o /home/hacki/Documents/sandbox/master_project/msl-2.0_automake/msl/libhoap2/examples/testCollision/testHoap2Collision.cpp

The call to link the executable:
g++ -O0 -g3 -o .libs/testHoap2Collision 
testHoap2Collision.o  ../../../libhoap2/.libs/libhoap2.so 
-L/home/hacki/Documents/sandbox/master_project/msl-2.0_automake/msl/../CGAL-3.1/lib/i686_Linux-2.6.15-suspend2-r3_g++-3.4.3 
-L/usr/local/webots/lib -L/usr/i686-pc-linux-gnu/bin 
-L/usr/i686-pc-linux-gnu/lib /home/hacki/Documents/sandbox/master_project/msl-2.0_automake/msl/debug/libroboop/source/.libs/libroboop.so /home/hacki/Documents/sandbox/master_project/msl-2.0_automake/msl/debug/libnewmat/.libs/libnewmat.so 
-lController 
-lCGAL //usr/lib/gcc/i686-pc-linux-gnu/3.4.3-20050110/libstdc++.so 
-Wl,--rpath -Wl,/usr/local/lib -Wl,--rpath 
-Wl,//usr/lib/gcc/i686-pc-linux-gnu/3.4.3-20050110

Sincerely Yours,
	Florian

-- 
Florian Hackenberger
student @
University of Technology
Graz, Austria
florian@hackenberger.at
www.hackenberger.at

Re: breakpoints in shared libraries

[Daniel Jacobowitz]

On Tue, Feb 14, 2006 at 05:29:09PM +0100, Florian Hackenberger wrote:
> Pending breakpoint "quaternion.cpp:100" resolved
> Quaternion::Quaternion(angle, axis), axis is not unit.
> Make the axis unit.
> 
> That tells me:
> 1. The breakpoint was set.
> 2. The breakpoint did not work, as I can see the output "Make the axis unit." 
> and that's the line where my breakpoint was supposed to interrupt the 
> execution.
> 
> So that's the problem. Can anyone help me?

This has nothing to do with shared libraries; if you search the
archives for breakpoints in constructors, you'll learn lots more about
the (still unsolved) problem.


-- 
Daniel Jacobowitz
CodeSourcery

Re: breakpoints in shared libraries

[Mark Kettenis]

> From: Florian Hackenberger <f.hackenberger@chello.at>
> Date: Tue, 14 Feb 2006 17:29:09 +0100
> 
> That tells me:
> 1. The breakpoint was set.
> 2. The breakpoint did not work, as I can see the output "Make the axis unit." 
> and that's the line where my breakpoint was supposed to interrupt the 
> execution.
> 
> So that's the problem. Can anyone help me?

So the breakpoint was set at the wrong location, either because the
line number information generated by the compiler is wrong or GDB
doesn't interpret it right.  Can you post the output of

$ readline -w LIB

where LIB is the shared library containing the code you're trying to
debug?

Mark

Re: breakpoints in shared libraries

[Florian Hackenberger]

On Tuesday 14 February 2006 18:00, Daniel Jacobowitz wrote:
> This has nothing to do with shared libraries; if you search the
> archives for breakpoints in constructors, you'll learn lots more about
> the (still unsolved) problem.
Thank you very much for the quick answer. Is the problem resolved with gcc 
>=4.0?

Regards,
	Florian

-- 
Florian Hackenberger
student @
University of Technology
Graz, Austria
florian@hackenberger.at
www.hackenberger.at

Re: breakpoints in shared libraries

[Daniel Jacobowitz]

On Tue, Feb 14, 2006 at 06:10:21PM +0100, Florian Hackenberger wrote:
> On Tuesday 14 February 2006 18:00, Daniel Jacobowitz wrote:
> > This has nothing to do with shared libraries; if you search the
> > archives for breakpoints in constructors, you'll learn lots more about
> > the (still unsolved) problem.
> Thank you very much for the quick answer. Is the problem resolved with gcc 
> >=4.0?

No, there's no resolution today.

-- 
Daniel Jacobowitz
CodeSourcery

Re: breakpoints in shared libraries

[Florian Hackenberger]

On Tuesday 14 February 2006 18:08, Mark Kettenis wrote:
> So the breakpoint was set at the wrong location, either because the
> line number information generated by the compiler is wrong or GDB
> doesn't interpret it right.  Can you post the output of
Thank you for trying to help me, but the problem seems to be, that gdb is not 
able to break within constructors, as Daniel stated.

Thank you mates!
-- 
Florian Hackenberger
student @
University of Technology
Graz, Austria
florian@hackenberger.at
www.hackenberger.at

Re: breakpoints in shared libraries

[Mark Kettenis]

> X-Spam-Checker-Version: SpamAssassin 3.1.0 (2005-09-13) on 
> 	elgar.sibelius.xs4all.nl
> X-Spam-Level: 
> X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=no 
> 	version=3.1.0
> X-From_: gdb-return-24270-m.m.kettenis=alumnus.utwente.nl@sourceware.org  Tue Feb 14 18:00:38 2006
> Date: Tue, 14 Feb 2006 12:00:31 -0500
> From: Daniel Jacobowitz <drow@false.org>
> Cc: gdb@sourceware.org
> Mail-Followup-To: Florian Hackenberger <f.hackenberger@chello.at>, 	gdb@sourceware.org
> Content-Disposition: inline
> X-IsSubscribed: yes
> Mailing-List: contact gdb-help@sourceware.org; run by ezmlm
> Sender: gdb-owner@sourceware.org
> X-UTwente-MailScanner-Information: Scanned by MailScanner. Contact helpdesk@ITBE.utwente.nl for more information.
> X-UTwente-MailScanner: Found to be clean
> X-MailScanner-From: gdb-return-24270-m.m.kettenis=alumnus.utwente.nl@sourceware.org
> 
> On Tue, Feb 14, 2006 at 05:29:09PM +0100, Florian Hackenberger wrote:
> > Pending breakpoint "quaternion.cpp:100" resolved
> > Quaternion::Quaternion(angle, axis), axis is not unit.
> > Make the axis unit.
> > 
> > That tells me:
> > 1. The breakpoint was set.
> > 2. The breakpoint did not work, as I can see the output "Make the axis unit." 
> > and that's the line where my breakpoint was supposed to interrupt the 
> > execution.
> > 
> > So that's the problem. Can anyone help me?
> 
> This has nothing to do with shared libraries; if you search the
> archives for breakpoints in constructors, you'll learn lots more about
> the (still unsolved) problem.

But Florian is setting a breakpoint using FILENAME:LINE syntax and not
using FUNCTIONNAME syntax.  That should work isn't it?  At least as
long as GCC is not inlining the constructor such that there are
actually multiple copies in the code and GDB sets the breakpoint in a
different copy than the one being executed.

Mark

Re: breakpoints in shared libraries

[Bob Rossi]

On Tue, Feb 14, 2006 at 06:21:07PM +0100, Mark Kettenis wrote:
> > X-Spam-Checker-Version: SpamAssassin 3.1.0 (2005-09-13) on 
> > 	elgar.sibelius.xs4all.nl
> > X-Spam-Level: 
> > X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=no 
> > 	version=3.1.0
> > X-From_: gdb-return-24270-m.m.kettenis=alumnus.utwente.nl@sourceware.org  Tue Feb 14 18:00:38 2006
> > Date: Tue, 14 Feb 2006 12:00:31 -0500
> > From: Daniel Jacobowitz <drow@false.org>
> > Cc: gdb@sourceware.org
> > Mail-Followup-To: Florian Hackenberger <f.hackenberger@chello.at>, 	gdb@sourceware.org
> > Content-Disposition: inline
> > X-IsSubscribed: yes
> > Mailing-List: contact gdb-help@sourceware.org; run by ezmlm
> > Sender: gdb-owner@sourceware.org
> > X-UTwente-MailScanner-Information: Scanned by MailScanner. Contact helpdesk@ITBE.utwente.nl for more information.
> > X-UTwente-MailScanner: Found to be clean
> > X-MailScanner-From: gdb-return-24270-m.m.kettenis=alumnus.utwente.nl@sourceware.org
> > 
> > On Tue, Feb 14, 2006 at 05:29:09PM +0100, Florian Hackenberger wrote:
> > > Pending breakpoint "quaternion.cpp:100" resolved
> > > Quaternion::Quaternion(angle, axis), axis is not unit.
> > > Make the axis unit.
> > > 
> > > That tells me:
> > > 1. The breakpoint was set.
> > > 2. The breakpoint did not work, as I can see the output "Make the axis unit." 
> > > and that's the line where my breakpoint was supposed to interrupt the 
> > > execution.
> > > 
> > > So that's the problem. Can anyone help me?
> > 
> > This has nothing to do with shared libraries; if you search the
> > archives for breakpoints in constructors, you'll learn lots more about
> > the (still unsolved) problem.
> 
> But Florian is setting a breakpoint using FILENAME:LINE syntax and not
> using FUNCTIONNAME syntax.  That should work isn't it?  At least as
> long as GCC is not inlining the constructor such that there are
> actually multiple copies in the code and GDB sets the breakpoint in a
> different copy than the one being executed.

I know this is probably obvious, but if you are desperate, you could
have your constructor call a new function that you write and then set a
breakpoint in the new function.

Bob Rossi

Re: breakpoints in shared libraries

[Daniel Jacobowitz]

On Tue, Feb 14, 2006 at 06:21:07PM +0100, Mark Kettenis wrote:
> But Florian is setting a breakpoint using FILENAME:LINE syntax and not
> using FUNCTIONNAME syntax.  That should work isn't it?  At least as
> long as GCC is not inlining the constructor such that there are
> actually multiple copies in the code and GDB sets the breakpoint in a
> different copy than the one being executed.

Nope.  I don't have a handy copy of the explanation, so here it is.

Take a look at the "Itanium C++ ABI", which has some Itanium-specific
language in it but is actually widely adopted for non-Itanium platforms
by multiple compilers, GNU and otherwise.  Here it is:

  http://www.codesourcery.com/cxx-abi/abi.html

It's got this to say about constructors:

  <ctor-dtor-name> ::= C1	# complete object constructor
		   ::= C2	# base object constructor
		   ::= C3	# complete object allocating constructor
		   ::= D0	# deleting destructor
		   ::= D1	# complete object destructor
		   ::= D2	# base object destructor

The relevant section is 2.6.  When constructing a virtual base class,
the C2 constructor is called instead of the C1 constructor.

Now, in theory, a compiler can generate one of these as a tiny
trampoline that jumps to the other, and only emit debug information for
the real one, and things will work.  But GCC doesn't do this and
attempts to get it to do so have, so far, met with failure.  So there's
two functions for every single constructor, and GDB tends to breakpoint
one of them more or less at random.

I maintain that the two right things to do here are (A) fix GCC to
support functions with multiple entry points, or at least something
that can give a good impression of it, and (B) set breakpoints by
function name or line number at all copies.

An astute observer will notice that both (A) and (B) are hard :-)
I've been working on (B) as I can find the time, and some cleanups
for it have already been done, and I posted an early prototype.
But I never got back to it.  The Ada folks expressed some interested,
and I think Fred did too, but so far nothing's come of it.

-- 
Daniel Jacobowitz
CodeSourcery

Re: breakpoints in shared libraries

[Mark Kettenis]

> Date: Tue, 14 Feb 2006 12:31:07 -0500
> From: Daniel Jacobowitz <drow@false.org>
> 
> Take a look at the "Itanium C++ ABI", which has some Itanium-specific
> language in it but is actually widely adopted for non-Itanium platforms
> by multiple compilers, GNU and otherwise.  Here it is:
> 
>   http://www.codesourcery.com/cxx-abi/abi.html
> 
> It's got this to say about constructors:
> 
>   <ctor-dtor-name> ::= C1	# complete object constructor
> 		   ::= C2	# base object constructor
> 		   ::= C3	# complete object allocating constructor
> 		   ::= D0	# deleting destructor
> 		   ::= D1	# complete object destructor
> 		   ::= D2	# base object destructor
> 
> The relevant section is 2.6.  When constructing a virtual base class,
> the C2 constructor is called instead of the C1 constructor.
> 
> Now, in theory, a compiler can generate one of these as a tiny
> trampoline that jumps to the other, and only emit debug information for
> the real one, and things will work.  But GCC doesn't do this and
> attempts to get it to do so have, so far, met with failure.  So there's
> two functions for every single constructor, and GDB tends to breakpoint
> one of them more or less at random.
> 
> I maintain that the two right things to do here are (A) fix GCC to
> support functions with multiple entry points, or at least something
> that can give a good impression of it, and (B) set breakpoints by
> function name or line number at all copies.
> 
> An astute observer will notice that both (A) and (B) are hard :-)
> I've been working on (B) as I can find the time, and some cleanups
> for it have already been done, and I posted an early prototype.
> But I never got back to it.  The Ada folks expressed some interested,
> and I think Fred did too, but so far nothing's come of it.

Thanks for the explanation Daniel.  The complexity of C++ continues to
amaze me.

Mark

Re: breakpoints in shared libraries

[Paul Koning]

>>>>> "Daniel" == Daniel Jacobowitz <drow@false.org> writes:

 Daniel> On Tue, Feb 14, 2006 at 06:10:21PM +0100, Florian
 Daniel> Hackenberger wrote:
 >> On Tuesday 14 February 2006 18:00, Daniel Jacobowitz wrote: > This
 >> has nothing to do with shared libraries; if you search the >
 >> archives for breakpoints in constructors, you'll learn lots more
 >> about > the (still unsolved) problem.  Thank you very much for the
 >> quick answer. Is the problem resolved with gcc >=4.0?

 Daniel> No, there's no resolution today.

I did a hack solution as a local tweak to GDB 5.3 -- a few lines worth
of change to set the "verbose" flag in the demangler calls.  That way,
the several flavors of constructor and destructor end up with
different names.  Odd ones, admittedly, but it allows them to be
referenced individually.

I wouldn't hold it up as a real solution, but it's a workaround, if
you care to use it.

	   paul

Re: breakpoints in shared libraries

[Florian Hackenberger]

On Tuesday 14 February 2006 18:52, you wrote:
> I did a hack solution as a local tweak to GDB 5.3 -- a few lines worth
> of change to set the "verbose" flag in the demangler calls.  That way,
> the several flavors of constructor and destructor end up with
> different names.  Odd ones, admittedly, but it allows them to be
> referenced individually.
Thank you very much indeed, but I just set breakpoints on every constructor 
call and resolved the underlying problem. Just for the sake of being able to 
set these breakpoints would not be worth the effort for patching and 
recompiling gdb.

Thanks anyway,
	Florian
-- 
Florian Hackenberger
student @
University of Technology
Graz, Austria
florian@hackenberger.at
www.hackenberger.at

Re: breakpoints in shared libraries

[Per Bothner]

Tom Tromey wrote:

>Suppose I set a breakpoint in a shared library.  I find on my Linux
>box I can only do this after the inferior has been started.  So I use
>`b main', then `run', then set the breakpoints I really care about.
>(This gets old fast, btw.)
>
Yes. It should be fixed.  I seem to remember that at one point Gilmore
wrote code so that gdb could set a break point on a shared Solaris library
that had not been loaded.  There really is no excuse for gdb not being able
to set a breakpoint on a shared library specified on the linker command
line.  (Shared libraries loaded with an explicit call to dlopen are of
course a different matter.)

It should not be very difficult, but it requires knowing how the linker
does things.  Somewhere in the executable there has to be a data structure
that lists the libraries that will be loaded.  I seem to remember there 
actually
is some Linux program that will list the shared libraries requried by an
excutable, which should make it easy..  Gdb just has to be able to read
that data structure, and pre-load the symbol tables for the shared 
libraries.
You can then set breakpoints in those shared libraries.  When the library
actually gets loaded, we don't need to read the symbol tables, but we
need to relocate them.

Let's fix the real bug instead of kludge to make the bug slightly less
obnoxious.

    --Per

Re: breakpoints in shared libraries

[Tom Tromey]

>>>>> "Daniel" == Daniel Jacobowitz <drow@mvista.com> writes:

>> How can this be fixed?  I don't know the answer.  One idea I had is to
>> annotate each breakpoint with information about what shared library it
>> comes from.  Then, defer processing for that breakpoint until the
>> shared library is actually loaded by the inferior.

Daniel> I think the best approach for this is to expose an interface
Daniel> to create shlib_disabled breakpoints.  Then they'll be
Daniel> automatically re-enabled when the library containing that
Daniel> symbol is loaded.

One thing to think about is how Insight will determine that it should
save a breakpoint as shlib_disabled as opposed to the ordinary kind.
At the time Insight is saving the session, all libraries may be loaded
and all the breakpoints activated.

Tom

Re: breakpoints in shared libraries

[Daniel Jacobowitz]

On Sun, Nov 25, 2001 at 11:30:21AM -0700, Tom Tromey wrote:
> A while back I changed Insight so that it would save and restore
> breakpoints.  This is really handy, but it has a problem that I think
> requires changes to gdb itself.
> 
> Suppose I set a breakpoint in a shared library.  I find on my Linux
> box I can only do this after the inferior has been started.  So I use
> `b main', then `run', then set the breakpoints I really care about.
> (This gets old fast, btw.)
> 
> When Insight saves breakpoints, it doesn't know about this.  How could
> it?  So it saves everything.  When I restart Insight and load my old
> session, the breakpoints that are interesting to me don't load,
> because gdb doesn't yet know about the shared libraries.
> 
> How can this be fixed?  I don't know the answer.  One idea I had is to
> annotate each breakpoint with information about what shared library it
> comes from.  Then, defer processing for that breakpoint until the
> shared library is actually loaded by the inferior.
> 
> Maybe there would be other ways to accomplish the same thing.  It
> would be nice if I didn't ever have to do the whole `b main; r' thing
> in the first place.  But as it stands this problem is a real
> impediment to my typical uses for gdb.

I think the best approach for this is to expose an interface to create
shlib_disabled breakpoints.  Then they'll be automatically re-enabled
when the library containing that symbol is loaded.

-- 
Daniel Jacobowitz                           Carnegie Mellon University
MontaVista Software                         Debian GNU/Linux Developer

breakpoints in shared libraries

[Tom Tromey]

A while back I changed Insight so that it would save and restore
breakpoints.  This is really handy, but it has a problem that I think
requires changes to gdb itself.

Suppose I set a breakpoint in a shared library.  I find on my Linux
box I can only do this after the inferior has been started.  So I use
`b main', then `run', then set the breakpoints I really care about.
(This gets old fast, btw.)

When Insight saves breakpoints, it doesn't know about this.  How could
it?  So it saves everything.  When I restart Insight and load my old
session, the breakpoints that are interesting to me don't load,
because gdb doesn't yet know about the shared libraries.

How can this be fixed?  I don't know the answer.  One idea I had is to
annotate each breakpoint with information about what shared library it
comes from.  Then, defer processing for that breakpoint until the
shared library is actually loaded by the inferior.

Maybe there would be other ways to accomplish the same thing.  It
would be nice if I didn't ever have to do the whole `b main; r' thing
in the first place.  But as it stands this problem is a real
impediment to my typical uses for gdb.

Tom