From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (qmail 17882 invoked by alias); 3 Feb 2011 07:03:09 -0000 Received: (qmail 17870 invoked by uid 22791); 3 Feb 2011 07:03:07 -0000 X-SWARE-Spam-Status: No, hits=-1.9 required=5.0 tests=BAYES_00 X-Spam-Check-By: sourceware.org Received: from zwopi.com (HELO daffy.zwopi.com) (85.214.124.126) by sourceware.org (qpsmtpd/0.43rc1) with ESMTP; Thu, 03 Feb 2011 07:03:01 +0000 Received: by daffy.zwopi.com (Postfix, from userid 80) id 6980385; Thu, 3 Feb 2011 08:02:57 +0100 (CET) To: Michael Snyder Subject: Re: performance of multithreading gets gradually worse under gdb X-PHP-Originating-Script: 0:func.inc MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="=_8271c57821829415752e7d7ced895e25" Date: Thu, 03 Feb 2011 07:03:00 -0000 From: Markus Alber Cc: In-Reply-To: <4D49D016.7000607@vmware.com> References: <8b62d2819c94a232987155aa99e01983@hyperion-imrt.org> <4D49BE4E.9000009@vmware.com> <76bccf1875854ebc69b6a892fb84a976@hyperion-imrt.org> <4D49D016.7000607@vmware.com> Message-ID: <0e69c90ce85e34a24a5bcef1ce391aae@hyperion-imrt.org> X-Sender: markus@hyperion-imrt.org User-Agent: Roundcube Webmail/0.5 Mailing-List: contact gdb-help@sourceware.org; run by ezmlm Precedence: bulk List-Id: List-Subscribe: List-Archive: List-Post: List-Help: , Sender: gdb-owner@sourceware.org X-SW-Source: 2011-02/txt/msg00014.txt.bz2 --=_8271c57821829415752e7d7ced895e25 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=UTF-8; format=flowed Content-length: 3355 On Wed, 02 Feb 2011 13:43:50 -0800, Michael Snyder wrote: >> It allocates about 100kB per iteration. > > Hmmm, and that's for roughly 36 thread start/stops, so it could be > losing roughly 3000 bytes per thread. That's much bigger than my > first guess would have been (a "struct thread_info" is only 336 > bytes). > >> One interesting finding might also be: >> I terminated the process when an iteration took about 3 min (instead >> of 1 sec) >> and gdb had about 115MB allocated. > > I assume that at this point, your system still had plenty of ram to > spare? It wasn't simply swapping? The system had about 20 GB to spare. >> On starting the application again, it ran alright for a while and >> the gdb memory allocation >> stayed constant. When it finally started to grow again, the >> application slowed down, and became >> slower with every iteration - the usual picture. >> I attached a sample file from the application where the computation >> bifurcates into >> the worker threads. This is one of three instances per iteration, >> but they all follow the >> same pattern. > > I was really hoping for a stripped-down sample that we could compile > and run. See the attached file. It shows a similar behaviour, although it only allocates 8kB per iteration. You have to wait some time before this happens. >> The machine has 2x6 cores x 3 instances per iteration = 36 worker >> threads per iteration. > > x86 architecture? > >> On another note, I tried to compile gdb-6.5 on my machine (because >> it was the release I >> used to work with before, without problems) and configure comes >> back with an error that it cannot find a termcap lib. There is none on >> the SuSE. Which package would I need to install? > > That would be libncurses, I think. I compiled gdb-6.5 alright and it performs well as usual, without this problem. > > >> On Wed, 02 Feb 2011 12:27:58 -0800, Michael Snyder wrote: >>> Markus Alber wrote: >>>> Hello, >>>> I have experienced the following problem: >>>> I'm debugging a number-crunching application which spawns a lot >>>> (36) little >>>> worker threads per iteration. The system does typically OoM 200 >>>> iterations. >>>> Although each of them should take about the same amount of time, >>>> the performance >>>> gets worse with every iteration and becomes excruciatingly slow. >>>> A system monitor reveals that gdb allocates more memory with every >>>> iteration, >>>> i.e. with every 36 threads started and finished. The CPU load of >>>> GDB goes up, too. >>>> The CPU usage of the application goes down. Compared to the solo >>>> performance, it >>>> gets slower by a factor 20 and more, if run long enough. >>>> The application behaves perfectly when run by itself. The >>>> multi-threaded part is not >>>> debugging compiled when this behaviour occurs. >>>> The distribution is SuSE 11.3 / gdb 7.1. >>>> Is there anything I can change about this behaviour, any options >>>> of gdb that need to >>>> be set in these circumstances? >>> Interesting. >>> >>> By how much does gdb's memory allocation increase? >>> In total or, if possible, per iteration? This might >>> give is a clue as to where to look. >>> >>> Do you think you could write a simple sample program that >>> allocates threads in a manner similar to your application? >>> >>> Thanks, --=_8271c57821829415752e7d7ced895e25 Content-Transfer-Encoding: base64 Content-Type: text/x-c++src; name=mt_test.cpp Content-Disposition: attachment; filename=mt_test.cpp Content-length: 2111 I2luY2x1ZGUgPHN0ZGxpYi5oPgojaW5jbHVkZSA8aW9zdHJlYW0+CiNpbmNs dWRlIDxtYXRoLmg+CgojaW5jbHVkZSA8dmVjdG9yPgoKI2luY2x1ZGUgPGJv b3N0L3RocmVhZC90aHJlYWQuaHBwPgojaW5jbHVkZSA8Ym9vc3QvYmluZC5o cHA+Cgp1c2luZyBuYW1lc3BhY2Ugc3RkOwoKLy8gQ29tcGlsZSB3aXRoOiBn KysgLWcgLWxib29zdF90aHJlYWQgbXRfdGVzdC5jcHAKCnN0cnVjdCBUaHJl YWREYXRhIHsKICBUaHJlYWREYXRhKGNvbnN0IHZlY3Rvcjxkb3VibGU+JiB2 T25lLAoJICAgICBjb25zdCBkb3VibGUgZmFjdG9yKSA6CiAgICBtX3ZPbmUo dk9uZSksCiAgICBtX2ZhY3RvcihmYWN0b3IpLAogICAgbV92VHdvKHZPbmUu c2l6ZSgpLCBtX2ZhY3RvcikKICAgIHsKICAgICAgLy8gZW1wdHkKICAgIH0K ICAKICBjb25zdCB2ZWN0b3I8ZG91YmxlPiYgICBtX3ZPbmU7CiAgZG91Ymxl ICAgICAgICAgICAgICAgICAgbV9mYWN0b3I7CiAgdmVjdG9yPGRvdWJsZT4g ICAgICAgICAgbV92VHdvOwp9OwogIApzdGF0aWMgdm9pZCogZG9BZGQodm9p ZCogZGF0YSkKewogIFRocmVhZERhdGEqIHBEYXRhID0gc3RhdGljX2Nhc3Q8 VGhyZWFkRGF0YSo+KGRhdGEpOwoKICBmb3IodW5zaWduZWQgaW50IG5BdCA9 IDA7IG5BdCA8IHBEYXRhLT5tX3ZPbmUuc2l6ZSgpOyArK25BdCkKICAgIHBE YXRhLT5tX3ZUd29bbkF0XSArPSBwRGF0YS0+bV9mYWN0b3IgKiBwRGF0YS0+ bV92T25lW25BdF07IAoKICByZXR1cm4gTlVMTDsKfQoKCmludCBtYWluKHZv aWQpIHsKCiAgY29uc3QgaW50IG5UaHJlYWRzID0gMTI7CiAgY29uc3QgaW50 IG5SZXBldGl0aW9ucyA9IDIwMDsKICAKICB2ZWN0b3I8ZG91YmxlPiB2T25l KDE8PDI0LCAxLjApOwogIAogIGZvcihpbnQgbkF0UmVwZXRpdGlvbiA9IDA7 IG5BdFJlcGV0aXRpb24gPCBuUmVwZXRpdGlvbnM7ICsrbkF0UmVwZXRpdGlv bikgewoKICAgIGNvdXQgPDwgIlJlcGV0aXRpb24gbm8uICIgPDwgbkF0UmVw ZXRpdGlvbiA8PCBlbmRsOwoKICAgIHZlY3RvcjxUaHJlYWREYXRhKj4gdnBE YXRhKG5UaHJlYWRzLCBzdGF0aWNfY2FzdDxUaHJlYWREYXRhKj4oTlVMTCkp OyAgCiAgICBib29zdDo6dGhyZWFkX2dyb3VwIHRocmVhZHM7CiAgICAKICAg IGZvciAoaW50IG5BdFRocmVhZCA9IDA7IG5BdFRocmVhZCA8IG5UaHJlYWRz OyBuQXRUaHJlYWQrKykgewogICAgICB2cERhdGFbbkF0VGhyZWFkXSA9IG5l dyBUaHJlYWREYXRhKHZPbmUsIHN0YXRpY19jYXN0PGRvdWJsZT4obkF0VGhy ZWFkKzEpICk7CiAgICAgIGlmKCB0aHJlYWRzLmNyZWF0ZV90aHJlYWQoIGJv b3N0OjpiaW5kKCZkb0FkZCwgc3RhdGljX2Nhc3Q8dm9pZCo+ICh2cERhdGFb bkF0VGhyZWFkXSkpICkgPT0gMCApCiAgIAl0aHJvdyBzdGQ6OmV4Y2VwdGlv bigpOwogICAgfQogICAgCiAgICB0aHJlYWRzLmpvaW5fYWxsKCk7CiAgICAK ICAgIGZvciAoaW50IG5BdFRocmVhZCA9IDA7IG5BdFRocmVhZCA8IG5UaHJl YWRzOyBuQXRUaHJlYWQrKykgCiAgICAgIGRlbGV0ZSB2cERhdGFbbkF0VGhy ZWFkXTsKICB9CgogIHJldHVybiAwOwp9Cg== --=_8271c57821829415752e7d7ced895e25--