Re: performance of multithreading gets gradually worse under gdb

[Markus Alber <markus@hyperion-imrt.org>]

[-- Attachment #1: Type: text/plain, Size: 3355 bytes --]

 On Wed, 02 Feb 2011 13:43:50 -0800, Michael Snyder wrote:

>>  It allocates about 100kB per iteration.
>
> Hmmm, and that's for roughly 36 thread start/stops, so it could be
> losing roughly 3000 bytes per thread.  That's much bigger than my
> first guess would have been (a "struct thread_info" is only 336 
> bytes).
>
>>  One interesting finding might also be:
>> I terminated the process when an iteration took about 3 min (instead 
>> of 1 sec)
>>  and gdb had about 115MB allocated.
>
> I assume that at this point, your system still had plenty of ram to
> spare?  It wasn't simply swapping?

 The system had about 20 GB to spare.
 
>>  On starting the application again, it ran alright for a while and 
>> the gdb memory allocation
>>  stayed constant. When it finally started to grow again, the 
>> application slowed down, and became
>>  slower with every iteration - the usual picture.
>> I attached a sample file from the application where the computation 
>> bifurcates into
>>  the worker threads. This is one of three instances per iteration, 
>> but they all follow the
>>  same pattern.
>
> I was really hoping for a stripped-down sample that we could compile 
> and run.

 See the attached file. It shows a similar behaviour, although it only 
 allocates 8kB per iteration.
 You have to wait some time before this happens.

>> The machine has 2x6 cores x 3 instances per iteration = 36 worker 
>> threads per iteration.
>
> x86 architecture?
>
>>  On another note, I tried to compile gdb-6.5 on my machine (because 
>> it was the release I
>>  used to work with before, without problems) and configure comes 
>> back with an error that it cannot find a termcap lib. There is none on 
>> the SuSE. Which package would I need to install?
>
> That would be libncurses, I think.


 I compiled gdb-6.5 alright and it performs well as usual, without this 
 problem.


>
>
>>  On Wed, 02 Feb 2011 12:27:58 -0800, Michael Snyder wrote:
>>> Markus Alber wrote:
>>>>  Hello,
>>>> I have experienced the following problem:
>>>> I'm debugging a number-crunching application which spawns a lot 
>>>> (36) little
>>>>  worker threads per iteration. The system does typically OoM 200 
>>>> iterations.
>>>>  Although each of them should take about the same amount of time, 
>>>> the performance
>>>>  gets worse with every iteration and becomes excruciatingly slow.
>>>> A system monitor reveals that gdb allocates more memory with every 
>>>> iteration,
>>>>  i.e. with every 36 threads started and finished. The CPU load of 
>>>> GDB goes up, too.
>>>>  The CPU usage of the application goes down. Compared to the solo 
>>>> performance, it
>>>>  gets slower by a factor 20 and more, if run long enough.
>>>> The application behaves perfectly when run by itself. The 
>>>> multi-threaded part is not
>>>>  debugging compiled when this behaviour occurs.
>>>> The distribution is SuSE 11.3 / gdb 7.1.
>>>> Is there anything I can change about this behaviour, any options 
>>>> of gdb that need to
>>>>  be set in these circumstances?
>>> Interesting.
>>>
>>> By how much does gdb's memory allocation increase?
>>> In total or, if possible, per iteration?  This might
>>> give is a clue as to where to look.
>>>
>>> Do you think you could write a simple sample program that
>>> allocates threads in a manner similar to your application?
>>>
>>> Thanks,

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: mt_test.cpp --]
[-- Type: text/x-c++src; name=mt_test.cpp, Size: 1555 bytes --]

#include <stdlib.h>
#include <iostream>
#include <math.h>

#include <vector>

#include <boost/thread/thread.hpp>
#include <boost/bind.hpp>

using namespace std;

// Compile with: g++ -g -lboost_thread mt_test.cpp

struct ThreadData {
  ThreadData(const vector<double>& vOne,
	     const double factor) :
    m_vOne(vOne),
    m_factor(factor),
    m_vTwo(vOne.size(), m_factor)
    {
      // empty
    }
  
  const vector<double>&   m_vOne;
  double                  m_factor;
  vector<double>          m_vTwo;
};
  
static void* doAdd(void* data)
{
  ThreadData* pData = static_cast<ThreadData*>(data);

  for(unsigned int nAt = 0; nAt < pData->m_vOne.size(); ++nAt)
    pData->m_vTwo[nAt] += pData->m_factor * pData->m_vOne[nAt]; 

  return NULL;
}


int main(void) {

  const int nThreads = 12;
  const int nRepetitions = 200;
  
  vector<double> vOne(1<<24, 1.0);
  
  for(int nAtRepetition = 0; nAtRepetition < nRepetitions; ++nAtRepetition) {

    cout << "Repetition no. " << nAtRepetition << endl;

    vector<ThreadData*> vpData(nThreads, static_cast<ThreadData*>(NULL));  
    boost::thread_group threads;
    
    for (int nAtThread = 0; nAtThread < nThreads; nAtThread++) {
      vpData[nAtThread] = new ThreadData(vOne, static_cast<double>(nAtThread+1) );
      if( threads.create_thread( boost::bind(&doAdd, static_cast<void*> (vpData[nAtThread])) ) == 0 )
   	throw std::exception();
    }
    
    threads.join_all();
    
    for (int nAtThread = 0; nAtThread < nThreads; nAtThread++) 
      delete vpData[nAtThread];
  }

  return 0;
}