From mboxrd@z Thu Jan 1 00:00:00 1970 From: Kimball Thurston To: Kimball Thurston , gdb@sources.redhat.com Subject: Re: gdb and dlopen Date: Tue, 16 Oct 2001 20:04:00 -0000 Message-id: References: <20011016161525.A1241@nevyn.them.org> <20011016213252.A8694@nevyn.them.org> <20011016220353.A9538@nevyn.them.org> X-SW-Source: 2001-10/msg00164.html At Tue, 16 Oct 2001 22:03:53 -0400, Daniel Jacobowitz wrote: > > On Tue, Oct 16, 2001 at 09:32:52PM -0400, Daniel Jacobowitz wrote: > > On Tue, Oct 16, 2001 at 06:23:39PM -0700, Kimball Thurston wrote: > > > It wasn't symbol reading inefficiency - or at least not directly. I > > > thought that at first, but I grabbed the snapshot from Oct 5th - I > > > haven't tried the latest yet, compiled it up with profiling info to > > > find where gdb is spending it's time. The majority of the time is > > > spent in child_xfer_memory - like 56% of the time (and most of that is > > > spent calling ptrace to copy bytes around) - the child_xfer_memory > > > seems to end up being called as a result of resetting breakpoints via > > > a chain of other things. I don't know why (ignorance). I've attached a > > > bzip of the profile data from the Oct 5th snapshot. Unfortunately, I > > > don't know about the internals of gdb to know what memory it's > > > transferring between processes. I tweaked on child_xfer_memory to not > > > call ptid_get_pid quite so much, but that obviously had only a > > > marginal improvement - it's all in ptrace and system calls - you can > > > see the system calls hit pretty hard from a cpuload application. > > > > > > The plugins are very small (minus debug code info) - they should have > > > only 3 exported functions, a few static functions, and their local > > > data block has ~ 1K of data in it or so. Right now, there are about 50 > > > of them. > > Can you give me more information, or a testcase? My suspicion that the > link map was responsible seems to be wrong, since I can write a dummy > application that loads 50 DSOs and debug it without any noticeable > stalls at all. > > Is this application multithreaded, by chance, or at least linked to > libpthread? The overhead in stopping and starting via the LinuxThreads > debugging package, even without multiple threads, is so ridiculous that > the times go through the roof. I think there's something which can be > done about that. I see a staggering amount of time spent in > svr4_current_sos, because target_read_string percolates down to > thread_db_xfer_memory and thus to thread_db_thread_alive. We should be > able to do some intelligent caching here. > Aha! Yes, I was just about to send more info, when I got your reply. Our application is multi-threaded (pthreads). We have not started any threads when we are loading the dsos - just the main thread is active, but I guess that doesn't matter... If you need any more info, let me know... I put a timer in our code, so I start the timer before I call the first dlopen, and stop it right after the last one: no gdb: 0.164334 seconds for 54 plugins with gdb 5.0.90-cvs from 07-Oct-2001, it takes ~ 30 seconds or so to enter main, and then: 151.355 seconds for 54 plugins, 144.033 to unload 73.3301 seconds for 31 plugins, 68.953 to unload 50.4318 seconds for 23 plugins, 47.6721 to unload 31.2207 seconds for 15 plugins, 29.0286 to unload 19.7027 seconds for 10 plugins, 18.4323 to unload 9.372 seconds for 5 plugins, 8.70661 to unload 1.83179 seconds for 1 plugins, 1.67519 to unload gdb 5.1-branch snapshot from sources.redhat.com from 16-Oct-2001: 151.919 seconds for 54 plugins, 144.9 to unload so 5.1 appears to be approximately the same as 5. This definitely doesn't happen in 4.18, although 4.18 has other, worse problems (like about half the time, it isn't useful for debugging - loses the call stack or ability to look at variables...) The scary thing is that it looks like the numbers are starting to grow non-linearly - it starts off at just under 2 seconds per plugin, then grows to almost 3 when we get up to 54, but I guess that is just the link map growing, so whatever... I am more than willing to do the leg work / coding on this, just need to know what direction to head down... thanks, Kimball