Re: gdb and dlopen

[Kimball Thurston <kimball@sgrail.com>]

At Tue, 16 Oct 2001 22:03:53 -0400,
Daniel Jacobowitz wrote:
> 
> On Tue, Oct 16, 2001 at 09:32:52PM -0400, Daniel Jacobowitz wrote:
> > On Tue, Oct 16, 2001 at 06:23:39PM -0700, Kimball Thurston wrote:
> > > It wasn't symbol reading inefficiency - or at least not directly. I
> > > thought that at first, but I grabbed the snapshot from Oct 5th - I
> > > haven't tried the latest yet, compiled it up with profiling info to
> > > find where gdb is spending it's time. The majority of the time is
> > > spent in child_xfer_memory - like 56% of the time (and most of that is
> > > spent calling ptrace to copy bytes around) - the child_xfer_memory
> > > seems to end up being called as a result of resetting breakpoints via
> > > a chain of other things. I don't know why (ignorance). I've attached a
> > > bzip of the profile data from the Oct 5th snapshot. Unfortunately, I
> > > don't know about the internals of gdb to know what memory it's
> > > transferring between processes. I tweaked on child_xfer_memory to not
> > > call ptid_get_pid quite so much, but that obviously had only a
> > > marginal improvement - it's all in ptrace and system calls - you can
> > > see the system calls hit pretty hard from a cpuload application.
> > > 
> > > The plugins are very small (minus debug code info) - they should have
> > > only 3 exported functions, a few static functions, and their local
> > > data block has ~ 1K of data in it or so. Right now, there are about 50
> > > of them.
> 
> Can you give me more information, or a testcase?  My suspicion that the
> link map was responsible seems to be wrong, since I can write a dummy
> application that loads 50 DSOs and debug it without any noticeable
> stalls at all.
> 
> Is this application multithreaded, by chance, or at least linked to
> libpthread?  The overhead in stopping and starting via the LinuxThreads
> debugging package, even without multiple threads, is so ridiculous that
> the times go through the roof.  I think there's something which can be
> done about that.  I see a staggering amount of time spent in
> svr4_current_sos, because target_read_string percolates down to
> thread_db_xfer_memory and thus to thread_db_thread_alive.  We should be
> able to do some intelligent caching here.
> 

Aha! Yes, I was just about to send more info, when I got your
reply. Our application is multi-threaded (pthreads). We have not
started any threads when we are loading the dsos - just the main
thread is active, but I guess that doesn't matter... If you need any
more info, let me know...

I put a timer in our code, so I start the timer before I call the
first dlopen, and stop it right after the last one:

no gdb:
   0.164334 seconds for 54 plugins

with gdb 5.0.90-cvs from 07-Oct-2001, it takes ~ 30 seconds or so to
enter main, and then:
  151.355 seconds for 54 plugins, 144.033 to unload
  73.3301 seconds for 31 plugins, 68.953 to unload
  50.4318 seconds for 23 plugins, 47.6721 to unload
  31.2207 seconds for 15 plugins, 29.0286 to unload
  19.7027 seconds for 10 plugins, 18.4323 to unload
  9.372 seconds for 5 plugins, 8.70661 to unload
  1.83179 seconds for 1 plugins, 1.67519 to unload

gdb 5.1-branch snapshot from sources.redhat.com from 16-Oct-2001:
  151.919 seconds for 54 plugins, 144.9 to unload

so 5.1 appears to be approximately the same as 5. This definitely
doesn't happen in 4.18, although 4.18 has other, worse problems (like
about half the time, it isn't useful for debugging - loses the call
stack or ability to look at variables...) The scary thing is that it
looks like the numbers are starting to grow non-linearly - it starts
off at just under 2 seconds per plugin, then grows to almost 3 when we
get up to 54, but I guess that is just the link map growing, so
whatever...

I am more than willing to do the leg work / coding on this, just need
to know what direction to head down...

thanks,
Kimball