Re: improving psymtab generation memory usage

[Daniel Berlin <dan@cgsoftware.com>]

Jim Blandy <jimb@zwingli.cygnus.com> writes:

> Daniel Berlin <dan@cgsoftware.com> writes:
>> I gave up on playing language specific games, and now just MD5 the
>> partial (and full die) attributes (for partial, we md5 the tag, name,
>> lowpc, and highpc), we keep a splay tree of md5 checksums, and skip
>> any dies (and their children) at the top level if it's md5 is in the
>> tree.
>> 
>> You'd have to have something at the same name, same tag, same
>> namespace, and same location in memory for the partial die to match.
> 
> Okay, so, just considering partial symbol table generation:
> 
> Dwarf 2 partial symbol table production is driven by a nested loop.
> 
> - The outer loop is in dwarf2_build_psymtabs_hard, handling one
>   compilation unit per iteration.  If the CU die has any children,
>   then we call scan_partial_symbols to read them.
> 
> - scan_partial_symbols contains the inner loop, with one iteration per
>   die (although we sometimes skip dies using sibling pointers, if
>   they're present).  This inner loop calls read_partial_die to read in
>   a die and some of its attributes.  If the die represents something
>   that needs a partial symbol table entry, it calls add_partial_symbol
>   on the die.
> 
> - add_partial_symbol looks at the die and makes the appropriate call
>   to add_psymbol_to_list.
> 
> - add_psymbol_to_list (in symfile.c now) bcaches the symbol name, and
>   then bcaches the `struct partial_symbol' object referring to the
>   name.  So all the data the psymtab needs has been copied out of the
>   objects dwarf2read.c created.
> 
> If I'm following the code correctly, this means that, after every
> iteration of the inner loop, there are no references to the die data
> we just read.  This had better be true, because all the psymtab's
> memory needs to be owned by the objfile, and not disappear when
> dwarf2_build_psymtabs_hard returns.
> 
> While we're building the partial symbol table, all the attribute
> values are allocated in dwarf2_tmp_obstack.  In fact, except for the
> abbrev table, all the memory consumed is in dwarf2_tmp_obstack.  (And
> the abbrev table could go there too, it seems.)
> 
> This means that we could free dwarf2_tmp_obstack after every iteration
> in scan_partial_symbols.  (At least, we could free it back to the
> point it was at when we entered the function.)  If the die's were
> small, this wouldn't even cause us to call free.  The maximum memory
> consumption by the Dwarf 2 reader at any one time would be two dies:
> the CU, and the current die.
> 
> Now, if I see a duplicate die, I'm still going to process it.  But the
> bcache should be eliminating duplicate psymtab entries anyway ---
> you'll get another entry in the global_psymbols or static_psymbols
> array, but that's just a pointer, not a whole object.
> 
> This approach isn't sensitive to changes in attribute offsets
> (although you point out that that isn't as big a deal as one would
> expect).
> 
> So what I'm saying is, we should be able to cut partial symbol table
> generation memory usage to a minimum by adding one call to
> obstack_free in scan_partial_symbols.

Sure, for psymbols, this will work fine.
For full symbols, it won't.
I did the full symbol elimination first, then psymbols, so i just used
the full symbol approach scaled down.


> 
> I know I'm punting on the full symbol stuff, but for partial symbols,
> does this sound right?

-- 
"In Vegas, I got into a long argument with the man at the
roulette wheel over what I considered to be an odd number.
"-Steven Wright