improving psymtab generation memory usage

improving psymtab generation memory usage

[Jim Blandy]

I think you mentioned a while back that it was possible to greatly
reduce the amount of memory GDB used generating partial symbol tables
from Dwarf 2 debug info.  (Or is that just confabulation on my part?)

Can you explain again the approach you suggested?

Re: improving psymtab generation memory usage

[Daniel Berlin]

Jim Blandy <jimb@zwingli.cygnus.com> writes:

> I think you mentioned a while back that it was possible to greatly
> reduce the amount of memory GDB used generating partial symbol tables
> from Dwarf 2 debug info.  (Or is that just confabulation on my
> part?)
Yup.
I also did it for symtabs.

> Can you explain again the approach you suggested?

Sure.
I gave up on playing language specific games, and now just MD5 the
partial (and full die) attributes (for partial, we md5 the tag, name,
lowpc, and highpc), we keep a splay tree of md5 checksums, and skip
any dies (and their children) at the top level if it's md5 is in the
tree.

You'd have to have something at the same name, same tag, same
namespace, and same location in memory for the partial die to match.

I don't think that's possible unless they are the same thing.
Certainly, GDB wouldn't know what to do with two things with the same
address and name, and same namespace.
That's all our partial symbols have.

For full die's, we process all attributes but the sibling attribute.

So they'd have to have the same cu relative offsets for the
attributes location attributes (I.E. The DW_AT_type for a variable would
have to be at the same offset in the CU for both cases) , etc, in
order to be the  
same.  Luckily, because gcc defers output, they end up with the same
cu relative offsets if they processed the include files in the same
order.
So while you'd think we'd never get any duplicates this way, we
eliminate just about all of them.
:)

MD5 can be done at over 30 meg a second on a pentium 90, so speed is
not a concern. It doesn't show up on profiles.

--Dan

-- 
"I was once arrested for walking in someone else's sleep.
"-Steven Wright

Re: improving psymtab generation memory usage

[Jim Blandy]

Daniel Berlin <dan@cgsoftware.com> writes:
> I gave up on playing language specific games, and now just MD5 the
> partial (and full die) attributes (for partial, we md5 the tag, name,
> lowpc, and highpc), we keep a splay tree of md5 checksums, and skip
> any dies (and their children) at the top level if it's md5 is in the
> tree.
> 
> You'd have to have something at the same name, same tag, same
> namespace, and same location in memory for the partial die to match.

Okay, so, just considering partial symbol table generation:

Dwarf 2 partial symbol table production is driven by a nested loop.

- The outer loop is in dwarf2_build_psymtabs_hard, handling one
  compilation unit per iteration.  If the CU die has any children,
  then we call scan_partial_symbols to read them.

- scan_partial_symbols contains the inner loop, with one iteration per
  die (although we sometimes skip dies using sibling pointers, if
  they're present).  This inner loop calls read_partial_die to read in
  a die and some of its attributes.  If the die represents something
  that needs a partial symbol table entry, it calls add_partial_symbol
  on the die.

- add_partial_symbol looks at the die and makes the appropriate call
  to add_psymbol_to_list.

- add_psymbol_to_list (in symfile.c now) bcaches the symbol name, and
  then bcaches the `struct partial_symbol' object referring to the
  name.  So all the data the psymtab needs has been copied out of the
  objects dwarf2read.c created.

If I'm following the code correctly, this means that, after every
iteration of the inner loop, there are no references to the die data
we just read.  This had better be true, because all the psymtab's
memory needs to be owned by the objfile, and not disappear when
dwarf2_build_psymtabs_hard returns.

While we're building the partial symbol table, all the attribute
values are allocated in dwarf2_tmp_obstack.  In fact, except for the
abbrev table, all the memory consumed is in dwarf2_tmp_obstack.  (And
the abbrev table could go there too, it seems.)

This means that we could free dwarf2_tmp_obstack after every iteration
in scan_partial_symbols.  (At least, we could free it back to the
point it was at when we entered the function.)  If the die's were
small, this wouldn't even cause us to call free.  The maximum memory
consumption by the Dwarf 2 reader at any one time would be two dies:
the CU, and the current die.

Now, if I see a duplicate die, I'm still going to process it.  But the
bcache should be eliminating duplicate psymtab entries anyway ---
you'll get another entry in the global_psymbols or static_psymbols
array, but that's just a pointer, not a whole object.

This approach isn't sensitive to changes in attribute offsets
(although you point out that that isn't as big a deal as one would
expect).

So what I'm saying is, we should be able to cut partial symbol table
generation memory usage to a minimum by adding one call to
obstack_free in scan_partial_symbols.

I know I'm punting on the full symbol stuff, but for partial symbols,
does this sound right?

Re: improving psymtab generation memory usage

[Daniel Berlin]

Jim Blandy <jimb@zwingli.cygnus.com> writes:

> Daniel Berlin <dan@cgsoftware.com> writes:
>> I gave up on playing language specific games, and now just MD5 the
>> partial (and full die) attributes (for partial, we md5 the tag, name,
>> lowpc, and highpc), we keep a splay tree of md5 checksums, and skip
>> any dies (and their children) at the top level if it's md5 is in the
>> tree.
>> 
>> You'd have to have something at the same name, same tag, same
>> namespace, and same location in memory for the partial die to match.
> 
> Okay, so, just considering partial symbol table generation:
> 
> Dwarf 2 partial symbol table production is driven by a nested loop.
> 
> - The outer loop is in dwarf2_build_psymtabs_hard, handling one
>   compilation unit per iteration.  If the CU die has any children,
>   then we call scan_partial_symbols to read them.
> 
> - scan_partial_symbols contains the inner loop, with one iteration per
>   die (although we sometimes skip dies using sibling pointers, if
>   they're present).  This inner loop calls read_partial_die to read in
>   a die and some of its attributes.  If the die represents something
>   that needs a partial symbol table entry, it calls add_partial_symbol
>   on the die.
> 
> - add_partial_symbol looks at the die and makes the appropriate call
>   to add_psymbol_to_list.
> 
> - add_psymbol_to_list (in symfile.c now) bcaches the symbol name, and
>   then bcaches the `struct partial_symbol' object referring to the
>   name.  So all the data the psymtab needs has been copied out of the
>   objects dwarf2read.c created.
> 
> If I'm following the code correctly, this means that, after every
> iteration of the inner loop, there are no references to the die data
> we just read.  This had better be true, because all the psymtab's
> memory needs to be owned by the objfile, and not disappear when
> dwarf2_build_psymtabs_hard returns.
> 
> While we're building the partial symbol table, all the attribute
> values are allocated in dwarf2_tmp_obstack.  In fact, except for the
> abbrev table, all the memory consumed is in dwarf2_tmp_obstack.  (And
> the abbrev table could go there too, it seems.)
> 
> This means that we could free dwarf2_tmp_obstack after every iteration
> in scan_partial_symbols.  (At least, we could free it back to the
> point it was at when we entered the function.)  If the die's were
> small, this wouldn't even cause us to call free.  The maximum memory
> consumption by the Dwarf 2 reader at any one time would be two dies:
> the CU, and the current die.
> 
> Now, if I see a duplicate die, I'm still going to process it.  But the
> bcache should be eliminating duplicate psymtab entries anyway ---
> you'll get another entry in the global_psymbols or static_psymbols
> array, but that's just a pointer, not a whole object.
> 
> This approach isn't sensitive to changes in attribute offsets
> (although you point out that that isn't as big a deal as one would
> expect).
> 
> So what I'm saying is, we should be able to cut partial symbol table
> generation memory usage to a minimum by adding one call to
> obstack_free in scan_partial_symbols.

Sure, for psymbols, this will work fine.
For full symbols, it won't.
I did the full symbol elimination first, then psymbols, so i just used
the full symbol approach scaled down.


> 
> I know I'm punting on the full symbol stuff, but for partial symbols,
> does this sound right?

-- 
"In Vegas, I got into a long argument with the man at the
roulette wheel over what I considered to be an odd number.
"-Steven Wright

Re: improving psymtab generation memory usage

[Jim Blandy]

Daniel Berlin <dan@cgsoftware.com> writes:
> > So what I'm saying is, we should be able to cut partial symbol table
> > generation memory usage to a minimum by adding one call to
> > obstack_free in scan_partial_symbols.
> 
> Sure, for psymbols, this will work fine.
> For full symbols, it won't.
> I did the full symbol elimination first, then psymbols, so i just used
> the full symbol approach scaled down.

Sure, and that seems reasonable.  I'm just checking my understanding
with someone who's been down the path before.

I'm going to make my change, simply because it's small, and will at
least help somewhat until we can do more.