Trying to spot memory corruption with core dump

Trying to spot memory corruption with core dump

[Pavel Shevaev]

Folks, could you please share any tips on spotting possible memory
corruption by examining the core dump?

My application seg. faults every several hours under moderate load and
the core dump points to malloc/calloc which I believe is a sign I have
a serious memory corruption somewhere in my application. The problem
is the fact I can not trace the problematic place in the code and I
wonder if the core dump can give me some more clues.

Somebody told me I could examine +-200 bytes of the memory where the
last exception happened but I don't really understand what I can find
there. For me it's just the raw amount of bytes...

Thanks in advance.

-- 
Best regards, Pavel

Re: Trying to spot memory corruption with core dump

[Paul Pluzhnikov]

On Mon, Jul 27, 2009 at 11:44 PM, Pavel Shevaev<pacha.shevaev@gmail.com> wrote:

> My application seg. faults every several hours under moderate load and
> the core dump points to malloc/calloc which I believe is a sign I have
> a serious memory corruption somewhere in my application.

Indeed.

> The problem
> is the fact I can not trace the problematic place in the code and I
> wonder if the core dump can give me some more clues.

Depending on which platform you are on, you may have tools available
which will get you to the problem *much* faster (or at least the
computer will be looking for the problem while you can do something
else :-)

On Linux/ix86 and x86_64, nothing beats Valgrind (for finding heap corruption).
On Solaris, try libumem. Most other UNIX platforms have various heap
checking libraries as well.

Cheers,
-- 
Paul Pluzhnikov

Re: Trying to spot memory corruption with core dump

[Pavel Shevaev]

> On Linux/ix86 and x86_64, nothing beats Valgrind (for finding heap corruption).

Oh, forgot to mention this on the very first post. I'm using linux
x86_64 and I'm aware of Valgrind and I even tried it....the problem is
it doesn't detect any memory corruption at all and what's even more
amazing my application doesn't seg.fault under Valgrind even after
running the whole day :( I think it's because Valgrind slows down the
application considerably and some how the rare edge case when memory
corruption occurs simply doesn't happen.

-- 
Best regards, Pavel

Re: Trying to spot memory corruption with core dump

[Paul Pluzhnikov]

On Tue, Jul 28, 2009 at 12:11 AM, Pavel Shevaev<pacha.shevaev@gmail.com> wrote:
>> On Linux/ix86 and x86_64, nothing beats Valgrind (for finding heap corruption).
>
> Oh, forgot to mention this on the very first post. I'm using linux
> x86_64 and I'm aware of Valgrind and I even tried it....the problem is
> it doesn't detect any memory corruption at all and what's even more
> amazing my application doesn't seg.fault under Valgrind even after
> running the whole day :(

With typical VG slow-downs (50x to 100x), you need to run it longer: 3
hours without VG => 50*3/24.0 == 1 week under VG.

> I think it's because Valgrind slows down the
> application considerably and some how the rare edge case when memory
> corruption occurs simply doesn't happen.

Is the application multithreaded?
Does it process a "random" event source?

VG could change timing and make bugs hide; that's not unheard of.
You could try setting MALLOC_CHECK_=2 in the environment (see 'info
libc'), or linking the application with '-lmcheck'.

Finally, does your application handle async signals?
If so, does signal handler call anything async-signal unsafe? That is
one common source of "random" heap corruption.

Cheers,
-- 
Paul Pluzhnikov

Re: Trying to spot memory corruption with core dump

[Pavel Shevaev]

> With typical VG slow-downs (50x to 100x), you need to run it longer: 3
> hours without VG => 50*3/24.0 == 1 week under VG.

Unfortunately, I can't afford it :(

> Is the application multithreaded?

Yes. And yes, I tried using helgrind as well and it discovered nothing...

> Does it process a "random" event source?

Well, it uses async handlers of boost::asio which in my case used as a
nice demultiplexing wrapper around epoll

> VG could change timing and make bugs hide; that's not unheard of.
> You could try setting MALLOC_CHECK_=2 in the environment (see 'info
> libc'), or linking the application with '-lmcheck'.

Thanks for the MALLOC_CHECK_ tip! I'm going to stress test my app using it...

> Finally, does your application handle async signals?
> If so, does signal handler call anything async-signal unsafe? That is
> one common source of "random" heap corruption.

Oh, no, nothing like that is used.

P.S. thanks again for being helpful, I've been trying to spot this
memory corruption for a week, I think I'm gradually going mad, so,
_any_ help is highly appreciated

-- 
Best regards, Pavel

Re: Trying to spot memory corruption with core dump

[Michael Snyder]

Pavel Shevaev wrote:

> P.S. thanks again for being helpful, I've been trying to spot this
> memory corruption for a week, I think I'm gradually going mad, so,
> _any_ help is highly appreciated

Don't worry, this is a typical symptom of trying to analyze
heap corruption problems.  ;-)

Re: Trying to spot memory corruption with core dump

[Samuel Bronson]

At Tue, 28 Jul 2009 20:55:40 +0400,
Pavel Shevaev wrote:
> > Is the application multithreaded?
> 
> Yes. And yes, I tried using helgrind as well and it discovered nothing...

Have you also tried DRD
<http://valgrind.org/docs/manual/drd-manual.html>? I'm not sure how
likely that is to help, but...