Re: performance of multithreading gets gradually worse under gdb

Michael Snyder
2011-02-02T21:43:58Z

Markus Alber wrote:
> Thank you for the quick response.

You're welcome, but let's keep the discussion on the public list, so
that other maintainers may jump in if so inclined.

> It allocates about 100kB per iteration.

Hmmm, and that's for roughly 36 thread start/stops, so it could be
losing roughly 3000 bytes per thread. That's much bigger than my
first guess would have been (a "struct thread_info" is only 336 bytes).

> One interesting finding might also be:
>
> I terminated the process when an iteration took about 3 min (instead of
> 1 sec)
> and gdb had about 115MB allocated.

I assume that at this point, your system still had plenty of ram to
spare? It wasn't simply swapping?

> On starting the application again, it ran alright for a while and the
> gdb memory allocation
> stayed constant. When it finally started to grow again, the application
> slowed down, and became
> slower with every iteration - the usual picture.
>
> I attached a sample file from the application where the computation
> bifurcates into
> the worker threads. This is one of three instances per iteration, but
> they all follow the
> same pattern.

I was really hoping for a stripped-down sample that we could compile and
run.

> The machine has 2x6 cores x 3 instances per iteration =
> 36 worker threads per iteration.

x86 architecture?

> On another note, I tried to compile gdb-6.5 on my machine (because it
> was the release I
> used to work with before, without problems) and configure comes back
> with an error that it cannot find a termcap lib. There is none on the
> SuSE. Which package would I need to install?

That would be libncurses, I think.

> On Wed, 02 Feb 2011 12:27:58 -0800, Michael Snyder wrote:
>> Markus Alber wrote:
>>> Hello,
>>> I have experienced the following problem:
>>> I'm debugging a number-crunching application which spawns a lot (36)
>>> little
>>> worker threads per iteration. The system does typically OoM 200
>>> iterations.
>>> Although each of them should take about the same amount of time,
>>> the performance
>>> gets worse with every iteration and becomes excruciatingly slow.
>>> A system monitor reveals that gdb allocates more memory with every
>>> iteration,
>>> i.e. with every 36 threads started and finished. The CPU load of
>>> GDB goes up, too.
>>> The CPU usage of the application goes down. Compared to the solo
>>> performance, it
>>> gets slower by a factor 20 and more, if run long enough.
>>> The application behaves perfectly when run by itself. The
>>> multi-threaded part is not
>>> debugging compiled when this behaviour occurs.
>>> The distribution is SuSE 11.3 / gdb 7.1.
>>> Is there anything I can change about this behaviour, any options of
>>> gdb that need to
>>> be set in these circumstances?
>> Interesting.
>>
>> By how much does gdb's memory allocation increase?
>> In total or, if possible, per iteration? This might
>> give is a clue as to where to look.
>>
>> Do you think you could write a simple sample program that
>> allocates threads in a manner similar to your application?
>>
>> Thanks,