very very slow symbol searches

very very slow symbol searches

[Martin M. Hunt]

Anyone have any idea why search_symbols in symtab.c is now a few
thousand times slower than it used to be?  At least it is on Linux x86.

Insight calls search_symbols twice on startup.  Sometime 2-4 weeks ago,
(builds from CVS on sources) Insight started taking a minute or two to
load when debugging itself.  It is very fast when debugging small
programs, but large things like insight or gdb take forever.  At first
glance the problem is that search_symbols() is taking 20-30 seconds to
return.

Everything is fine on Windows which starts up much faster than Linux,
even before this latest slowness.

Martin

Re: very very slow symbol searches

[Elena Zannoni]

Martin M. Hunt writes:
 > Anyone have any idea why search_symbols in symtab.c is now a few
 > thousand times slower than it used to be?  At least it is on Linux x86.
 > 
 > Insight calls search_symbols twice on startup.  Sometime 2-4 weeks ago,
 > (builds from CVS on sources) Insight started taking a minute or two to
 > load when debugging itself.  It is very fast when debugging small
 > programs, but large things like insight or gdb take forever.  At first
 > glance the problem is that search_symbols() is taking 20-30 seconds to
 > return.

there have been some changes to that function on 12-23.  Could you try
backing those out? This was a change we convinced ourselves it was ok
to do, but maybe it is problematic.

Another change was to find_pc_sect_line, not sure if that would have
an impact.

Elena


 > 
 > Everything is fine on Windows which starts up much faster than Linux,
 > even before this latest slowness.
 > 
 > Martin
 > 
 > 
 > 

Re: very very slow symbol searches

[David Carlton]

On Wed, 22 Jan 2003 10:07:10 -0500, Elena Zannoni <ezannoni@redhat.com> said:
> Martin M. Hunt writes:

>> Anyone have any idea why search_symbols in symtab.c is now a few
>> thousand times slower than it used to be?  At least it is on Linux
>> x86.

>> Insight calls search_symbols twice on startup.  Sometime 2-4 weeks
>> ago, (builds from CVS on sources) Insight started taking a minute
>> or two to load when debugging itself.  It is very fast when
>> debugging small programs, but large things like insight or gdb take
>> forever.  At first glance the problem is that search_symbols() is
>> taking 20-30 seconds to return.

> there have been some changes to that function on 12-23.  Could you
> try backing those out? This was a change we convinced ourselves it
> was ok to do, but maybe it is problematic.

Yeah, I actually submitted a patch to revert the search_symbols part
of my 12-23 change as part of my patch to demangle partial symbols:
now that I understand lookup_symbol_aux_minimal better, I'm sure
that's not the right thing to call in that situation.  I'd been
waiting on pushing that reversion until we'd made a final decision as
to whether or not to demangle partial symbols, but probably reverting
it now would be a good idea.

Of course, even if the search_symbols part of that patch is reverted,
it may be the case that the change to lookup_symbol_aux_minimal also
causes problems.  I hope not, but it's certainly not out of the
question.  I'm including the patch that Elena referred to after my
signature: Martin, could you test if reverting just the search_symbols
part helps, or if it's necessary to revert the whole thing?  (Or we
might be barking up the wrong tree entirely, I suppose.)

David Carlton
carlton@math.stanford.edu

Index: symtab.c
===================================================================
RCS file: /cvs/src/src/gdb/symtab.c,v
retrieving revision 1.82
diff -u -p -r1.82 symtab.c
--- symtab.c	17 Dec 2002 18:34:06 -0000	1.82
+++ symtab.c	19 Dec 2002 23:02:35 -0000
@@ -117,8 +117,7 @@ struct symbol *lookup_symbol_aux_minsyms
 					  const char *mangled_name,
 					  const namespace_enum namespace,
 					  int *is_a_field_of_this,
-					  struct symtab **symtab,
-					  int *force_return);
+					  struct symtab **symtab);
 
 static struct symbol *find_active_alias (struct symbol *sym, CORE_ADDR addr);
 
@@ -805,14 +804,6 @@ lookup_symbol_aux (const char *name, con
   struct symbol *sym;
   const struct block *static_block;
 
-  /* FIXME: carlton/2002-11-05: This variable is here so that
-     lookup_symbol_aux will sometimes return NULL after receiving a
-     NULL return value from lookup_symbol_aux_minsyms, without
-     proceeding on to the partial symtab and static variable tests.  I
-     suspect that that's a bad idea.  */
-  
-  int force_return;
-
   /* Search specified block and its superiors.  Don't search
      STATIC_BLOCK or GLOBAL_BLOCK.  */
 
@@ -931,13 +922,11 @@ lookup_symbol_aux (const char *name, con
      a mangled variable that is stored in one of the minimal symbol tables.
      Eventually, all global symbols might be resolved in this way.  */
 
-  force_return = 0;
-
   sym = lookup_symbol_aux_minsyms (name, mangled_name,
 				   namespace, is_a_field_of_this,
-				   symtab, &force_return);
+				   symtab);
   
-  if (sym != NULL || force_return == 1)
+  if (sym != NULL)
     return sym;
 
 #endif
@@ -981,13 +970,11 @@ lookup_symbol_aux (const char *name, con
    */
 
 
-  force_return = 0;
-
   sym = lookup_symbol_aux_minsyms (name, mangled_name,
 				   namespace, is_a_field_of_this,
-				   symtab, &force_return);
+				   symtab);
   
-  if (sym != NULL || force_return == 1)
+  if (sym != NULL)
     return sym;
 
 #endif
@@ -1172,13 +1159,20 @@ lookup_symbol_aux_psymtabs (int block_in
    tables.  Eventually, all global symbols might be resolved in this
    way.  */
 
+/* NOTE: carlton/2002-12-05: At one point, this function was part of
+   lookup_symbol_aux, and what are now 'return' statements within
+   lookup_symbol_aux_minsyms returned from lookup_symbol_aux, even if
+   sym was NULL.  As far as I can tell, this was basically accidental;
+   it didn't happen every time that msymbol was non-NULL, but only if
+   some additional conditions held as well, and it caused problems
+   with HP-generated symbol tables.  */
+
 static struct symbol *
 lookup_symbol_aux_minsyms (const char *name,
 			   const char *mangled_name,
 			   const namespace_enum namespace,
 			   int *is_a_field_of_this,
-			   struct symtab **symtab,
-			   int *force_return)
+			   struct symtab **symtab)
 {
   struct symbol *sym;
   struct blockvector *bv;
@@ -1271,7 +1265,6 @@ lookup_symbol_aux_minsyms (const char *n
 
 	      if (symtab != NULL && sym != NULL)
 		*symtab = s;
-	      *force_return = 1;
 	      return fixup_symbol_section (sym, s->objfile);
 	    }
 	  else if (MSYMBOL_TYPE (msymbol) != mst_text
@@ -1280,7 +1273,6 @@ lookup_symbol_aux_minsyms (const char *n
 	    {
 	      /* This is a mangled variable, look it up by its
 	         mangled name.  */
-	      *force_return = 1;
 	      return lookup_symbol_aux (SYMBOL_NAME (msymbol), mangled_name,
 					NULL, namespace, is_a_field_of_this,
 					symtab);
@@ -2904,12 +2896,31 @@ search_symbols (char *regexp, namespace_
 	      {
 		if (0 == find_pc_symtab (SYMBOL_VALUE_ADDRESS (msymbol)))
 		  {
-		    if (kind == FUNCTIONS_NAMESPACE
-			|| lookup_symbol (SYMBOL_NAME (msymbol),
-					  (struct block *) NULL,
-					  VAR_NAMESPACE,
-					0, (struct symtab **) NULL) == NULL)
-		      found_misc = 1;
+		    if (kind == FUNCTIONS_NAMESPACE)
+		      {
+			found_misc = 1;
+		      }
+		    else
+		      {
+			struct symbol *sym;
+
+			if (SYMBOL_DEMANGLED_NAME (msymbol) != NULL)
+			  sym
+			    = lookup_symbol_aux_minsyms (SYMBOL_DEMANGLED_NAME
+							 (msymbol),
+							 SYMBOL_NAME (msymbol),
+							 VAR_NAMESPACE,
+							 NULL, NULL);
+			else
+			  sym
+			    = lookup_symbol_aux_minsyms (SYMBOL_NAME (msymbol),
+							 NULL,
+							 VAR_NAMESPACE,
+							 NULL, NULL);
+
+			if (sym == NULL)
+			  found_misc = 1;
+		      }
 		  }
 	      }
 	  }

Re: very very slow symbol searches

[David Carlton]

On 22 Jan 2003 04:00:16 -0800, "Martin M. Hunt" <hunt@redhat.com> said:

> Insight calls search_symbols twice on startup.  Sometime 2-4 weeks ago,
> (builds from CVS on sources) Insight started taking a minute or two to
> load when debugging itself.  It is very fast when debugging small
> programs, but large things like insight or gdb take forever.  At first
> glance the problem is that search_symbols() is taking 20-30 seconds to
> return.

Could you give some commands to reproduce this problem?  (Preferably
using GDB to debug itself.)  I tried running GDB on itself and then
typing 'info functions', and I don't see any significant time
difference on current mainline GDB versus GDB 5.2 or 5.3.  (In both
cases, the version of GDB being debugged current mainline GDB.)  I
also tried running to main first, to pull in dynamic libraries (which
shouldn't have debug info on my system, I think).  Without running to
main, mainline GDB was actually the fastest, albeit only slightly so;
with running to main, mainline GDB was slightly slower than 5.3 but
dramatically faster than 5.2.

But search_symbols is kind of weird (e.g. in its efforts to get info
from minimal symbols if there isn't enough debug info), so I could
easily believe that I'm not managing to tickle the bug that you're
seeing.  Could you give more details about what you're doing?

David Carlton
carlton@math.stanford.edu

Re: very very slow symbol searches

[David Carlton]

On 22 Jan 2003 10:26:08 -0800, David Carlton <carlton@math.Stanford.EDU> said:
> On 22 Jan 2003 04:00:16 -0800, "Martin M. Hunt" <hunt@redhat.com> said:

>> Insight calls search_symbols twice on startup.  Sometime 2-4 weeks
>> ago, (builds from CVS on sources) Insight started taking a minute
>> or two to load when debugging itself.  It is very fast when
>> debugging small programs, but large things like insight or gdb take
>> forever.  At first glance the problem is that search_symbols() is
>> taking 20-30 seconds to return.

> Could you give some commands to reproduce this problem?  (Preferably
> using GDB to debug itself.)  I tried running GDB on itself and then
> typing 'info functions', and I don't see any significant time
> difference on current mainline GDB versus GDB 5.2 or 5.3.  (In both
> cases, the version of GDB being debugged current mainline GDB.)  I
> also tried running to main first, to pull in dynamic libraries
> (which shouldn't have debug info on my system, I think).  Without
> running to main, mainline GDB was actually the fastest, albeit only
> slightly so; with running to main, mainline GDB was slightly slower
> than 5.3 but dramatically faster than 5.2.

Of course, my patch would have affected 'info variables' more than
'info functions'.  So I did the same tests with 'info variables';
here, 5.2 is really slow, 5.3 and mainline are better, and actually
mainline is faster than 5.3 on the version where I run to main first.

At any rate, obviously I'm not managing to tickle this bug correctly.
It is nice to see the speedup since 5.2: somebody obviously did
something good there.  (Is it an effect of hashing symtabs, maybe, or
did 5.2 already do that?)

David Carlton
carlton@math.stanford.edu

Re: very very slow symbol searches

[Andrew Cagney]

> to whether or not to demangle partial symbols, but probably reverting
> it now would be a good idea.

David, suggest seriously beefing up the comments that go with this 
`reversion' - revert the code but add more comments.  Over to Elena.

Andrew

Re: very very slow symbol searches

[Martin M. Hunt]

On Wed, 2003-01-22 at 07:07, Elena Zannoni wrote:
> 
> there have been some changes to that function on 12-23.  Could you try
> backing those out? This was a change we convinced ourselves it was ok
> to do, but maybe it is problematic.

I'm using "./insight insight" as a test case, measuring the time it
takes for a source window to be displayed.

1 Jan 03 snapshot takes 8 seconds.  This is normal.
8 Jan 03 snapshot takes 70 seconds.

I'll try to refine this further and come up with a better test case.

Martin

Re: very very slow symbol searches

[Martin M. Hunt]

More details.  The slowdown is caused by using the glibc regex.  Why
would that make it so much slower?  Don't know yet.  Anyway forcing GDB
to use the included regex in libiberty fixes the problem.

2003-01-02  Mark Kettenis  <kettenis@gnu.org>

        * configure.in: Rewrite check for GNU regex and the
        --without-included regex option, and move it into the "Checks
for
        library functions" section.  This makes us use the system regex
        again by default on systems with version 2 of the GNU C library.
        This was apparently broken.
        * gdb_regex.h [!USE_INCLUDED_REGEX] (_REGEX_RE_COMP): Define.
        * acconfig.h (USE_INCLUDED_REGEX): Remove.
        * config.in, configure: Regenerated.


On Wed, 2003-01-22 at 12:48, Martin M. Hunt wrote:
> On Wed, 2003-01-22 at 07:07, Elena Zannoni wrote:
> > 
> > there have been some changes to that function on 12-23.  Could you try
> > backing those out? This was a change we convinced ourselves it was ok
> > to do, but maybe it is problematic.
> 
> I'm using "./insight insight" as a test case, measuring the time it
> takes for a source window to be displayed.
> 
> 1 Jan 03 snapshot takes 8 seconds.  This is normal.
> 8 Jan 03 snapshot takes 70 seconds.
> 
> I'll try to refine this further and come up with a better test case.
> 
> Martin
> 
> 
> 
> 

Re: very very slow symbol searches

[Daniel Jacobowitz]

On Wed, Jan 22, 2003 at 05:16:29PM -0800, Martin M. Hunt wrote:
> More details.  The slowdown is caused by using the glibc regex.  Why
> would that make it so much slower?  Don't know yet.  Anyway forcing GDB
> to use the included regex in libiberty fixes the problem.
> 
> 2003-01-02  Mark Kettenis  <kettenis@gnu.org>
> 
>         * configure.in: Rewrite check for GNU regex and the
>         --without-included regex option, and move it into the "Checks
> for
>         library functions" section.  This makes us use the system regex
>         again by default on systems with version 2 of the GNU C library.
>         This was apparently broken.
>         * gdb_regex.h [!USE_INCLUDED_REGEX] (_REGEX_RE_COMP): Define.
>         * acconfig.h (USE_INCLUDED_REGEX): Remove.
>         * config.in, configure: Regenerated.

Sigh, let me gues, is your host glibc version 2.3?  The regex engine
was rewritten, not always for the better.

Some performance issues were fixed recently in the glibc CVS.

-- 
Daniel Jacobowitz
MontaVista Software                         Debian GNU/Linux Developer

Re: very very slow symbol searches

[Martin M. Hunt]

On Wed, 2003-01-22 at 17:45, Daniel Jacobowitz wrote:
> On Wed, Jan 22, 2003 at 05:16:29PM -0800, Martin M. Hunt wrote:
> > More details.  The slowdown is caused by using the glibc regex.  Why
> > would that make it so much slower?  Don't know yet.  Anyway forcing GDB
> > to use the included regex in libiberty fixes the problem.
> > 
> > 2003-01-02  Mark Kettenis  <kettenis@gnu.org>
> > 
> >         * configure.in: Rewrite check for GNU regex and the
> >         --without-included regex option, and move it into the "Checks
> > for
> >         library functions" section.  This makes us use the system regex
> >         again by default on systems with version 2 of the GNU C library.
> >         This was apparently broken.
> >         * gdb_regex.h [!USE_INCLUDED_REGEX] (_REGEX_RE_COMP): Define.
> >         * acconfig.h (USE_INCLUDED_REGEX): Remove.
> >         * config.in, configure: Regenerated.
> 
> Sigh, let me gues, is your host glibc version 2.3?  The regex engine
> was rewritten, not always for the better.

I'm running unmodified RH8.0
~> rpm -q glibc
glibc-2.2.93-5

> Some performance issues were fixed recently in the glibc CVS.

That doesn't help much.  What was wrong with having GDB on all platforms
using the standard libiberty regex?

Martin

Re: very very slow symbol searches

[Eli Zaretskii]

On 22 Jan 2003, Martin M. Hunt wrote:

> What was wrong with having GDB on all platforms using the standard 
> libiberty regex?

Perhaps on some platforms, the system regex is better.

But whatever the reasons, having an option to use the system regex is a 
Good Thing, IMHO.  Maybe we just should not make this the default, or 
have specialized tests for certain versions of glibc.

Re: very very slow symbol searches

[Daniel Jacobowitz]

On Wed, Jan 22, 2003 at 06:45:51PM -0800, Martin M. Hunt wrote:
> > Sigh, let me gues, is your host glibc version 2.3?  The regex engine
> > was rewritten, not always for the better.
> 
> I'm running unmodified RH8.0
> ~> rpm -q glibc
> glibc-2.2.93-5

Yes, that has the new regex engine.  Glibc 2.3.1 and 2.3-cvs have some
substantial performance improvements.  What's the regex in question
anyway?

> > Some performance issues were fixed recently in the glibc CVS.
> 
> That doesn't help much.  What was wrong with having GDB on all platforms
> using the standard libiberty regex?

Libiberty is there to augment deficiencies in the system libraries, not
to replace them; all the normal advantages of using shared libraries
apply to using libiberty only when necessary.  For instance, you will
probably be able to upgrade to 8.1 eventually and have the regex issues
automatically fixed.

-- 
Daniel Jacobowitz
MontaVista Software                         Debian GNU/Linux Developer

Re: very very slow symbol searches

[Andrew Cagney]

>> Some performance issues were fixed recently in the glibc CVS.
> 
> 
> That doesn't help much.  What was wrong with having GDB on all platforms
> using the standard libiberty regex?

It is like readline and bfd.  While a distro can choose to mix 'n' match 
GDB in with a random regx and/or readline and/or bfd, I think GDB 
should, by default, link itself against the regx it was bundled with.

After all, we know that the bundled regex works with GDB.

enjoy,
Andrew

Re: very very slow symbol searches

[Martin M. Hunt]

> Could you give some commands to reproduce this problem?  (Preferably
> using GDB to debug itself.)  I tried running GDB on itself and then
> typing 'info functions', and I don't see any significant time
> difference on current mainline GDB versus GDB 5.2 or 5.3.  

Interesting.  Maybe the problem is limited to certain glibcs?

> ./gdb gdb
> info func main*

01Jan03 build: less than 1 second pause before output.
Current build: about 10 seconds pause

This is on a standard RH8.0 Linux system, glibc-2.2.93-5

Martin

Re: very very slow symbol searches

[Andrew Cagney]

Martin, can you please bug report this (mark it as high).  It is 
something that needs to be resolved before the next release.

Of course, if you manage to resolve it ...

Andrew