Bad performance in updating JIT debug symbols

Fredrik Tolf
2016-09-17T17:25:08Z

Dear list,

I'm currently developing a library that uses LLVM to JIT-compile code into
the running process every now and then. It happens fairly often, but in
completely independent batches, so I'm racking up a fair number of LLVM
modules, usually in the thousands, but at the very least in the hundreds.

Whenever a new module is compiled, LLVM uses GDB's JIT-compiler interface
to register the debug symbols for the module, which is pretty nice, but as
the modules pile up, this is becoming very slow. In the end, it starts
taking multiples of tens of milliseconds per new module, and since they
usually come in several at once, GDB starts causing delays of a second or
longer, which is pretty annoying when debugging.

Debugging GDB in the process, it appears that it's stuck in the sorting
phase of update_section_map(), and the place where it's spending all the
time is the part of qsort_cmp() which, according to the comments, "could
be slow (since we iterate over all objfiles in each call to qsort_cmp),
but this shouldn't happen very often (GDB is already in a confused state;
one hopes this doesn't happen at all)."

Which obviously raises the question why it happens at all. I haven't
decoded enough of GDB's source code yet to tell what it's really trying to
do, but the reason for the slowness appears to be related to how the code
and debug sections are organized together, so I'm wondering if this should
be considered a bug in LLVM for organizing them incorrectly.

Thoughts?

--
Fredrik Tolf