From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (qmail 27497 invoked by alias); 2 Feb 2011 21:43:58 -0000 Received: (qmail 27487 invoked by uid 22791); 2 Feb 2011 21:43:57 -0000 X-SWARE-Spam-Status: No, hits=-4.8 required=5.0 tests=AWL,BAYES_00,RCVD_IN_DNSWL_HI,T_RP_MATCHES_RCVD X-Spam-Check-By: sourceware.org Received: from smtp-outbound-2.vmware.com (HELO smtp-outbound-2.vmware.com) (65.115.85.73) by sourceware.org (qpsmtpd/0.43rc1) with ESMTP; Wed, 02 Feb 2011 21:43:52 +0000 Received: from mailhost2.vmware.com (mailhost2.vmware.com [10.16.67.167]) by smtp-outbound-2.vmware.com (Postfix) with ESMTP id D53904D006; Wed, 2 Feb 2011 13:43:50 -0800 (PST) Received: from msnyder-server.eng.vmware.com (promd-2s-dhcp138.eng.vmware.com [10.20.124.138]) by mailhost2.vmware.com (Postfix) with ESMTP id C5F288EE29; Wed, 2 Feb 2011 13:43:50 -0800 (PST) Message-ID: <4D49D016.7000607@vmware.com> Date: Wed, 02 Feb 2011 21:43:00 -0000 From: Michael Snyder User-Agent: Thunderbird 2.0.0.24 (X11/20101201) MIME-Version: 1.0 To: Markus Alber , "gdb@sourceware.org" Subject: Re: performance of multithreading gets gradually worse under gdb References: <8b62d2819c94a232987155aa99e01983@hyperion-imrt.org> <4D49BE4E.9000009@vmware.com> <76bccf1875854ebc69b6a892fb84a976@hyperion-imrt.org> In-Reply-To: <76bccf1875854ebc69b6a892fb84a976@hyperion-imrt.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-IsSubscribed: yes Mailing-List: contact gdb-help@sourceware.org; run by ezmlm Precedence: bulk List-Id: List-Subscribe: List-Archive: List-Post: List-Help: , Sender: gdb-owner@sourceware.org X-SW-Source: 2011-02/txt/msg00012.txt.bz2 Markus Alber wrote: > Thank you for the quick response. You're welcome, but let's keep the discussion on the public list, so that other maintainers may jump in if so inclined. > It allocates about 100kB per iteration. Hmmm, and that's for roughly 36 thread start/stops, so it could be losing roughly 3000 bytes per thread. That's much bigger than my first guess would have been (a "struct thread_info" is only 336 bytes). > One interesting finding might also be: > > I terminated the process when an iteration took about 3 min (instead of > 1 sec) > and gdb had about 115MB allocated. I assume that at this point, your system still had plenty of ram to spare? It wasn't simply swapping? > On starting the application again, it ran alright for a while and the > gdb memory allocation > stayed constant. When it finally started to grow again, the application > slowed down, and became > slower with every iteration - the usual picture. > > I attached a sample file from the application where the computation > bifurcates into > the worker threads. This is one of three instances per iteration, but > they all follow the > same pattern. I was really hoping for a stripped-down sample that we could compile and run. > The machine has 2x6 cores x 3 instances per iteration = > 36 worker threads per iteration. x86 architecture? > On another note, I tried to compile gdb-6.5 on my machine (because it > was the release I > used to work with before, without problems) and configure comes back > with an error that it cannot find a termcap lib. There is none on the > SuSE. Which package would I need to install? That would be libncurses, I think. > On Wed, 02 Feb 2011 12:27:58 -0800, Michael Snyder wrote: >> Markus Alber wrote: >>> Hello, >>> I have experienced the following problem: >>> I'm debugging a number-crunching application which spawns a lot (36) >>> little >>> worker threads per iteration. The system does typically OoM 200 >>> iterations. >>> Although each of them should take about the same amount of time, >>> the performance >>> gets worse with every iteration and becomes excruciatingly slow. >>> A system monitor reveals that gdb allocates more memory with every >>> iteration, >>> i.e. with every 36 threads started and finished. The CPU load of >>> GDB goes up, too. >>> The CPU usage of the application goes down. Compared to the solo >>> performance, it >>> gets slower by a factor 20 and more, if run long enough. >>> The application behaves perfectly when run by itself. The >>> multi-threaded part is not >>> debugging compiled when this behaviour occurs. >>> The distribution is SuSE 11.3 / gdb 7.1. >>> Is there anything I can change about this behaviour, any options of >>> gdb that need to >>> be set in these circumstances? >> Interesting. >> >> By how much does gdb's memory allocation increase? >> In total or, if possible, per iteration? This might >> give is a clue as to where to look. >> >> Do you think you could write a simple sample program that >> allocates threads in a manner similar to your application? >> >> Thanks,