gdb symbol lookup very slow

gdb symbol lookup very slow

[Antonio Cavallo]

Hi,
I'm having hard time debugging a (very large) C++ library under gdb (gdb 
7.7.1, gcc 4.8, binutils 2.22).

The main issue is the time it takes to reach a breakpoint: gdb takes an 
insane amount of time (order of 2mins) vs vs2012 (a couple of seconds).

I've profiled gdb and the top functions called during the debugging are 
(more than 90% is spent in these):

   strcmp_iw
   find_pc_sect_psymtab
   symbol_get_demangled_name
   symbol_search_name

I suspect gdb doesn't cache the symbols: is there any way to speedup 
this lookup? Is there any other explanation for why gdb is so much 
slower than visual studio?

I've repeated the run with lldb (from svn), and it is at least two times 
faster than gdb.

Any suggestion where to look for?

Thanks

Re: gdb symbol lookup very slow

[Andrew Burgess]

On 28/05/2014 8:59 AM, Antonio Cavallo wrote:
> I'm having hard time debugging a (very large) C++ library under gdb (gdb
> 7.7.1, gcc 4.8, binutils 2.22).

Details of your target architecture / operating system /etc would be
helpful.

> 
> The main issue is the time it takes to reach a breakpoint: gdb takes an
> insane amount of time (order of 2mins) vs vs2012 (a couple of seconds).
> 
> I've profiled gdb and the top functions called during the debugging are
> (more than 90% is spent in these):
> 
>   strcmp_iw
>   find_pc_sect_psymtab
>   symbol_get_demangled_name
>   symbol_search_name
> 
> I suspect gdb doesn't cache the symbols: is there any way to speedup
> this lookup? Is there any other explanation for why gdb is so much
> slower than visual studio?

I suspect you've not mentioned something important :) In "normal"
circumstances, once a breakpoint has been placed, and the program being
debugged has been resumed, then there should be no time spent inside gdb.

Are you by chance using conditional breakpoints?  Or stepping, or using
some other commands?

The best thing would be to post a small example session, an annotate
which commands you feel are taking longer than expected.  Also, you
mention that you profiled gdb, assuming that it's not too large, making
the profile data available might allow some more insight.

Thanks for reporting this issue, and taking the time to investigate it.

Andrew

Re: gdb symbol lookup very slow

[Antonio Cavallo]

Hi,

thank you for the reply.


Andrew Burgess wrote:
> On 28/05/2014 8:59 AM, Antonio Cavallo wrote:
>> I'm having hard time debugging a (very large) C++ library under gdb (gdb
>> 7.7.1, gcc 4.8, binutils 2.22).
>
> Details of your target architecture / operating system /etc would be
> helpful.

the target is x86_64, it's centos (red hat 6.5) system;


>
> I suspect you've not mentioned something important :) In "normal"
> circumstances, once a breakpoint has been placed, and the program being
> debugged has been resumed, then there should be no time spent inside gdb.

The break point is "pending" because the library to be debugged is not 
loaded at the main debuggee startup.

> Are you by chance using conditional breakpoints?  Or stepping, or using
> some other commands?

After the gdb starts (gdb --args <myprogram> <arguments>) I simply set 
the breakpoint (pending) and run the application: at that point after 
2mins it reaches the breakpoint.

As test I've run the gdb --args <myprogram> <arguments> without the 
breakpoint and the app completes in under few seconds. So the 
performance hit might be due to the breakpoint (and the libraries loaded).


> The best thing would be to post a small example session, an annotate
> which commands you feel are taking longer than expected.  Also, you
> mention that you profiled gdb, assuming that it's not too large, making
> the profile data available might allow some more insight.

I'll grab all the info I can get out of it. In the meanwhile is there 
anything I can try to check what might cause that?

> Thanks for reporting this issue, and taking the time to investigate it.
>
> Andrew
>

Thanks
Antonio

Re: gdb symbol lookup very slow

[Jan Kratochvil]

On Wed, 28 May 2014 09:59:02 +0200, Antonio Cavallo wrote:
> Hi,
> I'm having hard time debugging a (very large) C++ library under gdb (gdb
> 7.7.1, gcc 4.8, binutils 2.22).
> 
> The main issue is the time it takes to reach a breakpoint: gdb takes an
> insane amount of time (order of 2mins) vs vs2012 (a couple of seconds).
> 
> I've profiled gdb and the top functions called during the debugging are
> (more than 90% is spent in these):

Try to use gdb-add-index (gdb/contrib/gdb-add-index.sh or
/usr/bin/gdb-add-index on some systems) with gdb-7.7+.

That the index is present in a binary you can verify with:
$ readelf -WS /usr/lib/debug/bin/bash.debug|grep -w gdb_index
  [34] .gdb_index        PROGBITS        0000000000000000 21bc44 0252fa 00      0   0  1

Or with recent enough readelf even with:
	readelf --debug-dump=gdb_index /usr/lib/debug/bin/bash.debug
or with elfutils:
	eu-readelf --debug-dump=gdb_index /usr/lib/debug/bin/bash.debug

You should check you use .gdb_index version 8, older versions had various
issues and they can get ignored by recent GDBs.

There is also an unrelated GDB performance Bug which happens with binaries
optimized by 'dwz' but I do not think that is your described case.
	Bug 16405 - backtrace takes GBs and minutes with dwz -m
	https://sourceware.org/bugzilla/show_bug.cgi?id=16405



Jan