Re: [RFC][gdb/symtab] Lazy expansion of full symbol table

[Tom de Vries <tdevries@suse.de>]

On 6/14/21 10:54 PM, Tom Tromey wrote:
>>>>>> "Tom" == Tom de Vries <tdevries@suse.de> writes:
> 
> Tom> [ I'm not posting the experimental patch series as such for now.  Available
> Tom> here ( https://github.com/vries/gdb/commits/lazy-full-symtab ). ]
> 
> Tom> In PR23710, the stated problem is that gdb is slow and memory hungry when
> Tom> consuming debug information generated by GCC with LTO.
> 
> Tom> | Minimal symbols | 0.18           |
> Tom> | Partial symbols | 2.34           |
> Tom> | Full symbols    | 3.34           |
> 
> I don't have this executable

Uploaded to https://ftp.suse.com/pub/people/tdevries/gdb/pr23710/cc1

> but FWIW my scanner rewrite is ~10x faster
> than the current psymtab reader.
> 

Interesting, that's
https://github.com/tromey/gdb/commits/submit/no-more-psym ?

I've tried that branch with the cc1 example, and ran into:
...
DW_FORM_strp pointing outside of .debug_str section [in module cc1]
...

> Tom> A way to fix this is to do processing of the full symbols in a lazy fashion.
> 
> Tom> This patch series implements a first attempt at this, for now intended not to
> Tom> be functionally correct, but to assess the kind of performance improvement we
> Tom> get from this.
> 
> Tom> With current trunk (commit 987610f2d68), we get 3.44, instead of the 6.44
> Tom> without this patch series.
> 
> That's about in line with the preliminary results I saw as well.
> You can see that branch at tromey/lazily-read-function-bodies, but it's
> probably unusable since I last rebased it in 2017.
> 

Ack, thanks for the pointer.

> Tom> The problem is that we need a way to do this gradually instead:
> Tom> - expand a few symbols
> Tom> - get the correspoding symbol table
> Tom> - expand a few more symbols
> Tom> - get the updated symbol table contain all expanded symbols
> 
> Tom> I'm not sure what is the smartest way to do that.
> 
> Tom> My current idea is to try to keep the builder around rather than destroy it,
> Tom> and have it generate an updated symbol table each time.
> 
> Tom> Is this a good idea?
> 
> Tom> Any other comments?
> 
> I think it's a good idea to do something.  Historically I didn't look
> into speeding up CU expansion on the theory that it didn't really matter
> -- most CUs are small -- but LTO upends that.
> 
> I never finished the earlier patch that I referenced, but the idea there
> was to skip function bodies when reading a CU.  Then, the functions
> could be fully read when needed.  I got this to work for by-name
> lookups, but I never finished the part where a by-address lookup would
> resize the blockvector, etc.
> 
> This approach could be extended to also not read type bodies until
> needed.  This could work by inflating the type during check_typedef.
> This can be a little dangerous because we do miss these calls from time
> to time; but maybe nearly as good would be to only lazily read
> "complicated" types, and use the accessor methods on those types to
> ensure they are inflated properly.
> 
> 
> Another deeper approach would be to abstract away symtabs (and I guess
> the blockvector) the way that we did for psymtabs: hide the details
> behind the "quick" methods and let each debug reader decide for itself
> how to store and look up symbols.  This is probably more work, but would
> allow even more laziness.  For example with the new indexer the symbols
> could simply hang off the entries in the index.
> 

OK, I'll try to take a look at the branch and digest your comments.

Thanks,
- Tom