Re: performance of multithreading gets gradually worse under gdb

[Michael Snyder <msnyder@vmware.com>]

Markus Alber wrote:
>  Thank you for the quick response.

You're welcome, but let's keep the discussion on the public list, so 
that other maintainers may jump in if so inclined.


>  It allocates about 100kB per iteration.

Hmmm, and that's for roughly 36 thread start/stops, so it could be
losing roughly 3000 bytes per thread.  That's much bigger than my
first guess would have been (a "struct thread_info" is only 336 bytes).

>  One interesting finding might also be:
> 
>  I terminated the process when an iteration took about 3 min (instead of 
>  1 sec)
>  and gdb had about 115MB allocated.

I assume that at this point, your system still had plenty of ram to 
spare?  It wasn't simply swapping?

>  On starting the application again, it ran alright for a while and the 
>  gdb memory allocation
>  stayed constant. When it finally started to grow again, the application 
>  slowed down, and became
>  slower with every iteration - the usual picture.
> 
>  I attached a sample file from the application where the computation 
>  bifurcates into
>  the worker threads. This is one of three instances per iteration, but 
>  they all follow the
>  same pattern.

I was really hoping for a stripped-down sample that we could compile and 
run.


> The machine has 2x6 cores x 3 instances per iteration = 
>  36 worker threads per iteration.

x86 architecture?

>  On another note, I tried to compile gdb-6.5 on my machine (because it 
>  was the release I
>  used to work with before, without problems) and configure comes back 
>  with an error that it cannot find a termcap lib. There is none on the 
>  SuSE. Which package would I need to install?

That would be libncurses, I think.


>  On Wed, 02 Feb 2011 12:27:58 -0800, Michael Snyder wrote:
>> Markus Alber wrote:
>>>  Hello,
>>> I have experienced the following problem:
>>> I'm debugging a number-crunching application which spawns a lot (36) 
>>> little
>>>  worker threads per iteration. The system does typically OoM 200 
>>> iterations.
>>>  Although each of them should take about the same amount of time, 
>>> the performance
>>>  gets worse with every iteration and becomes excruciatingly slow.
>>> A system monitor reveals that gdb allocates more memory with every 
>>> iteration,
>>>  i.e. with every 36 threads started and finished. The CPU load of 
>>> GDB goes up, too.
>>>  The CPU usage of the application goes down. Compared to the solo 
>>> performance, it
>>>  gets slower by a factor 20 and more, if run long enough.
>>> The application behaves perfectly when run by itself. The 
>>> multi-threaded part is not
>>>  debugging compiled when this behaviour occurs.
>>> The distribution is SuSE 11.3 / gdb 7.1.
>>> Is there anything I can change about this behaviour, any options of 
>>> gdb that need to
>>>  be set in these circumstances?
>> Interesting.
>>
>> By how much does gdb's memory allocation increase?
>> In total or, if possible, per iteration?  This might
>> give is a clue as to where to look.
>>
>> Do you think you could write a simple sample program that
>> allocates threads in a manner similar to your application?
>>
>> Thanks,