Re: [RFC-v2] Enhance backtrace for microsoft system DLL calls

[Pedro Alves <pedro_alves@portugalmail.pt>]

Pierre Muller wrote:
>   I wrote a i386_skip_noop function.

>   Tested on cygwin target, no regressions found.
> The patch allows to get the backtrace of the main thread of gdb 
> to come up to the  functions that called the systems DLL.
> If I use ./gdb ./gdb with 'set new-console on'
> and use Ctrl-C on the debuggee gdb window.
> Without the patch, the backtrace only shows
> 3 levels in ntdll.dll and kernel32.dll
> 

[ I forgot to say before:
   This is great, thanks for doing this! ]

> Questions:
>  1) Is the 'nop' test useful or should it be removed?
> 
>  2) Should we add other possible no-ops?

It is my opinion that it isn't needed, and the function that
detects the mov %edi,%edi should be called i386_skip_msft_hotpatch,
or i386_skip_hotpatch.  The nop is a 2 byte op for a reason,
plus, I don't see the point of detecting a lot of patterns if
we know they're never emitted.  Plus, if we ever need to augment
this hot-patching support to another different form, of detect the
5 bytes slack before, this is the natural place to do it.
Plus not detecting for a 1 byte nop, you can read 2 bytes at once.
(ok, that is going extreme :-) )

> 3) this call is used for all i386 targets, but it
> is probably useless for all operating systems but Microsoft Windows,
> so should it be called only for that OS, and if yes, how should
> we code this?
> 

The way to do it would be to put a flag in i386's gdbarch_tdep,
but Daniel didn't think we need to keep this MSFT specific.
(adding a flag would at least prevent a needless memory read
  on non Windows platforms)

 > 4) Any suggestions to make the comment clearer will be
 > most appreciated.

Grabbing my own comment from upthread, how about:

/* Some Microsoft's system dll functions start with a
    `mov %edi,%edi' instruction, which is effectively a two byte `nop'.
    This instruction is used for hot patching support, together with 5
    bytes of slack before the function.  Later, when hot-patching, the 2
    byte op can be replaced with a relative jump to 5 bytes back.  The 5
    bytes slack is large enough to hold a jump into anywhere in
    the 32-bit address space.  */

If you find it interesting, you can add:

/* A two byte nop is used to be sure that no thread is executing
    the instruction at byte 1 of the function, so the patching can be
    performed atomically.  */

-- 
Pedro Alves