Trouble debugging a Java Virtual Machine on Linux

[Johan Walles <johan.walles@appeal.se>]

Hi!

We are having trouble debugging our Java Virtual Machine on Linux.  The 
reason is that:

JRockit (our virtual machine) uses ptrace() for controlling threads. 
Gdb uses ptrace() for controlling processes / threads.  Linux does not 
allow two processes (like both gdb and JRockit) at a time to ptrace() 
one target process (in our case, a Java thread).

Thus, as both gdb and JRockit tries to PTRACE_ATTACH, they clash.

We have discussed a number of solutions to this problem, and we are 
currently leaning towards making some sort of (non-JRockit specific) 
modifications to gdb so that a debuggee can provide alternative ways for 
gdb to manage processes.

Now, I'd like very much to get some input from somebody knowledgable 
about gdb.  First, is there already some simple way of interfacing with 
gdb to tell it to use our functions for managing processes and not its 
own?  If not, does patching gdb seem like a reasonable solution?  What 
would have to happen for these patches to get accepted into the official 
gdb?

   Regards, Johan Walles, Appeal Virtual Machines