debugging link_map corruption

debugging link_map corruption

[Maule Mark]

Tried this on libc-help, no response, so trying gdb ...

I have a heavily threaded program linked against many (~50) shared libraries which occasionally experiences memory corruption such that the link_map list gets trashed rendering the core mostly undebuggable.  I'm looking for ways to debug these sorts of problems.  My operating environment is Linux.

One idea I am experimenting with is to create an audit library which saves the publicly available link_map (the one exposed through /usr/include/link.h on Linux) list to a write-protected area upon receipt of a LA_ACT_CONSISTENT activity callback.  The thinking is that if gdb can't follow the link_map from the core, at least I would be able to manually load the .so's at their correct addresses from gdb when debugging the 'corrupt' core.  In my current implementation, the audit library has a simple 8k buffer which it uses to store the public link_map structs in.

The problem I'm having with the above, is that I can't figure out how to expose information about the address of the audit library's link_map buffer to gdb when debugging the core.  I could issue a fprintf from my audit library to save that information in a file, but it would be much better if I could just figure that out with gdb.  I assume the issue is that the audit library symbols are in a separate namespace.

Anyway, I'm looking for guidance on how to gain access to audit library symbols from gdb when examining a program core.  Additionally, if there are better ideas for how to attack the problem of corrupt link_map lists in general, I'd appreciate those as well.

Thanks
Mark Maule

Re: debugging link_map corruption

[Philippe Waroquiers]

On Wed, 2014-06-04 at 06:36 -0700, Maule Mark wrote:
> Tried this on libc-help, no response, so trying gdb ...
> 
> I have a heavily threaded program linked against many (~50) shared libraries which occasionally experiences memory corruption
>  such that the link_map list gets trashed rendering the core mostly undebuggable.  I'm looking for ways to debug these sorts
>  of problems.  My operating environment is Linux.
Already tried Valgrind ?

Philippe

Re: debugging link_map corruption

[Maule Mark]

Unfortunately, no.  We run in a very limited diskless software environment, so there is not much opportunity for external profiling.  I'm not confident that our sw stack would run under valgrind.



> On Wednesday, June 4, 2014 3:21 PM, Philippe Waroquiers <philippe.waroquiers@skynet.be> wrote:
> > On Wed, 2014-06-04 at 06:36 -0700, Maule Mark wrote:
> 
>>  Tried this on libc-help, no response, so trying gdb ...
>> 
>>  I have a heavily threaded program linked against many (~50) shared 
> libraries which occasionally experiences memory corruption
>>   such that the link_map list gets trashed rendering the core mostly 
> undebuggable.  I'm looking for ways to debug these sorts
>>   of problems.  My operating environment is Linux.
> Already tried Valgrind ?
> 
> Philippe
>

Re: debugging link_map corruption

[Philippe Waroquiers]

On Wed, 2014-06-04 at 13:46 -0700, Maule Mark wrote:
> Unfortunately, no.  We run in a very limited diskless software environment, so there is not much opportunity for external profiling.
>   I'm not confident that our sw stack would run under valgrind.
Valgrind has been used on quite "small" environments (e.g. on Android
ARM based phones), so maybe it is not *that* desperate.

If your sw stack does not run under valgrind, then that is a Valgrind
bug, which I am sure valgrind developers will be more than happy to
fix :).

Philippe