program does not crash when attached to gdbserver

program does not crash when attached to gdbserver

[Dr. Rolf Jansen]

My setup is as follows:

gdb/gdbserver built from CVS trunk from today (6.8.50.20090612-cvs)

The source were configured and compiled unchanged:

gdb was configured as "--host=powerpc-apple-darwin9.7.0 --target=i386- 
mingw32msvc".
gdbserver was configured as "--host=i386-mingw32msvc --target=i386- 
mingw32msvc".

gdb has been compiled by gcc-4.2 on Mac OS X 10.5.7 with the Xcode  
3.1.2 toolchain
gdbserver has been compiled with a gcc-4.3.1 cross compiler (-- 
host=powerpc-apple-darwin9.7.0 --target=i386-mingw32msvc)

The situation is, that I am trying to debug a huge GUI application for  
Win32. It has been entirely compiled with said cross-compiler using  
the options -gdwarf-2, -O0. I can step through each part, so the  
symbols are there.

The problem is that the application crashes consistently after a  
certain sequence of user interactions if it runs by its own. However,  
the same binary does not crash, once it is attached to gdbserver, and  
it does not crash even when continuing  with a couple of stress tests  
beyound the point at which it would have crashed without gdbserver.

It would help so much to find the bug if the program would crash into  
gdb and if gdb could show me the related source code. This usually  
works quite well, for example when writing to memory at NULL.

Perhaps somebody has an idea about what type of bug might cause the  
behaviour described above.

There was a debugger for Mac OS Classic called MacsBug, that had a  
setting for scrambling the memory, so that accessing released memory  
would immediately result into a crash. I cannot seem to find a similar  
feature in gdb. Does gdb have any settings, that I can try?

Many thanks in advance for any help.

Best regards

Rolf Jansen

Re: program does not crash when attached to gdbserver

[Jonas Maebe]

On 13 Jun 2009, at 00:55, Dr. Rolf Jansen wrote:

> The problem is that the application crashes consistently after a  
> certain sequence of user interactions if it runs by its own.  
> However, the same binary does not crash, once it is attached to  
> gdbserver, and it does not crash even when continuing  with a couple  
> of stress tests beyound the point at which it would have crashed  
> without gdbserver.
>
> It would help so much to find the bug if the program would crash  
> into gdb and if gdb could show me the related source code. This  
> usually works quite well, for example when writing to memory at NULL.
>
> Perhaps somebody has an idea about what type of bug might cause the  
> behaviour described above.

As you surmise below: probably using uninitialised and/or freed memory.

> There was a debugger for Mac OS Classic called MacsBug, that had a  
> setting for scrambling the memory, so that accessing released memory  
> would immediately result into a crash. I cannot seem to find a  
> similar feature in gdb. Does gdb have any settings, that I can try?

In general, this is a feature of the compiler and/or run time, rather  
than of the debugger (the debugger cannot know how the memory manager  
of your run time works, so unless you exclusively use OS or OS- 
supplied library functions, it cannot scramble anything). E.g., in  
case of the Free Pascal Compiler, there are the -gttt (scramble all  
local variables on function entry) and -gh (use the heaptrc unit,  
which, a.o., scrambles all freed memory) options.

For GCC, you can have a look which of these work on your target  
platform: http://en.wikipedia.org/wiki/Memory_debugger

Regarding scrambling local variables on function entry, I believe that  
recent GCC's support doing that as well, but I don't know the command  
line option by heart.


Jonas

Re: program does not crash when attached to gdbserver

[Dr. Rolf Jansen]

Jonas,

many thanks for your reply!

Am 13.06.2009 um 05:50 schrieb Jonas Maebe:

> On 13 Jun 2009, at 00:55, Dr. Rolf Jansen wrote:
>
>> The problem is that the application crashes consistently after a  
>> certain sequence of user interactions if it runs by its own.  
>> However, the same binary does not crash, once it is attached to  
>> gdbserver, and it does not crash even when continuing  with a  
>> couple of stress tests beyound the point at which it would have  
>> crashed without gdbserver.
>>
>> It would help so much to find the bug if the program would crash  
>> into gdb and if gdb could show me the related source code. This  
>> usually works quite well, for example when writing to memory at NULL.
>>
>> Perhaps somebody has an idea about what type of bug might cause the  
>> behaviour described above.
>
> As you surmise below: probably using uninitialised and/or freed  
> memory.
>
>> There was a debugger for Mac OS Classic called MacsBug, that had a  
>> setting for scrambling the memory, so that accessing released  
>> memory would immediately result into a crash. I cannot seem to find  
>> a similar feature in gdb. Does gdb have any settings, that I can try?
>
> In general, this is a feature of the compiler and/or run time,  
> rather than of the debugger (the debugger cannot know how the memory  
> manager of your run time works, so unless you exclusively use OS or  
> OS-supplied library functions, it cannot scramble anything).

I can understand this, and as a matter of fact I would have expected  
that the debugger does not interact with the memory management, my  
problem is that it acts somehow on mm and the runtime. It would  
already be of help for me if somebody could point me to some possible  
areas of interaction, which could make up for the above mentioned  
different behaviour of running the same binary with and without  
gdbserver being attached.

> E.g., in case of the Free Pascal Compiler, there are the -gttt  
> (scramble all local variables on function entry) and -gh (use the  
> heaptrc unit, which, a.o., scrambles all freed memory) options.
>
> For GCC, you can have a look which of these work on your target  
> platform: http://en.wikipedia.org/wiki/Memory_debugger

I experimented a bit with memwatch, to no more avail than finding some  
freeing-NULL occasions. I would not have bothered you with my case, if  
I would still have own ideas to try. Anyway, I will have a look to the  
other options mentioned at the wiki page.

> Regarding scrambling local variables on function entry, I believe  
> that recent GCC's support doing that as well, but I don't know the  
> command line option by heart.

For several reasons (which would make up for a story of its own), it  
would surprise me, if this is problem with local variables, but I will  
have a look in that, anyway.

 From memwatch I know that there are more than 70000 memory  
allocations from which half of it are already free'd until the crash  
occurs. My program is written in objective-c and my part consist of  
30000 lines of code and the frameworks that are in use are about  
150000 lines, which makes this problem not a trivial one. My part and  
the frameworks are compiled with complete debugging symbols, and I can  
step through every single part of the whole program. So, for me it  
would already be perfect if gdb/gdbserver could simply stop the  
program where without gdbserver being attached it would crash.

Rolf Jansen

Re: program does not crash when attached to gdbserver

[Jonas Maebe]

On 13 Jun 2009, at 15:53, Dr. Rolf Jansen wrote:

> Am 13.06.2009 um 05:50 schrieb Jonas Maebe:
>
>>
>> In general, this is a feature of the compiler and/or run time,  
>> rather than of the debugger (the debugger cannot know how the  
>> memory manager of your run time works, so unless you exclusively  
>> use OS or OS-supplied library functions, it cannot scramble  
>> anything).
>
> I can understand this, and as a matter of fact I would have expected  
> that the debugger does not interact with the memory management, my  
> problem is that it acts somehow on mm and the runtime. It would  
> already be of help for me if somebody could point me to some  
> possible areas of interaction, which could make up for the above  
> mentioned different behaviour of running the same binary with and  
> without gdbserver being attached.

I was only pointing out why gdb does not include memory scrambling  
functionality similar to macsbug.

It is true that running a program under a debugger sometimes changes  
its behaviour, particularly when uninitialised memory is involved.  
Such problems are often called "heisenbugs".

>> E.g., in case of the Free Pascal Compiler, there are the -gttt  
>> (scramble all local variables on function entry) and -gh (use the  
>> heaptrc unit, which, a.o., scrambles all freed memory) options.
>>
>> For GCC, you can have a look which of these work on your target  
>> platform: http://en.wikipedia.org/wiki/Memory_debugger
>
> I experimented a bit with memwatch, to no more avail than finding  
> some freeing-NULL occasions. I would not have bothered you with my  
> case, if I would still have own ideas to try. Anyway, I will have a  
> look to the other options mentioned at the wiki page.

At least some of those should have options to fill freed and allocated  
memory with random data. You may also want to try this one: http://support.microsoft.com/kb/286470


Jonas

Re: program does not crash when attached to gdbserver

[Dr. Rolf Jansen]

Jonas!

Many thanks again for your helpful hints. I will try more of the other  
"Memory Debuggers" and I will have a close look on the MS Pageheap.exe.

Best regards

Rolf

Am 13.06.2009 um 12:26 schrieb Jonas Maebe:

> On 13 Jun 2009, at 15:53, Dr. Rolf Jansen wrote:
>
>> Am 13.06.2009 um 05:50 schrieb Jonas Maebe:
>>
>>
>>> For GCC, you can have a look which of these work on your target  
>>> platform: http://en.wikipedia.org/wiki/Memory_debugger
>>
>> I experimented a bit with memwatch, to no more avail than finding  
>> some freeing-NULL occasions. I would not have bothered you with my  
>> case, if I would still have own ideas to try. Anyway, I will have a  
>> look to the other options mentioned at the wiki page.
>
> At least some of those should have options to fill freed and  
> allocated memory with random data. You may also want to try this  
> one: http://support.microsoft.com/kb/286470