PATCH: 0/6 [2nd try]: Add AVX support

H.J. Lu
2010-03-06T22:16:39Z

AVX registers are saved and restored via the XSAVE extended state. The
extended control register 0 (the XFEATURE_ENABLED_MASK register), XCR0,
is used to determine which states, x87, SSE, AVX, ... are supported
in the XSAVE extended state. XCR0 can be read with the new "xgetbv"
instruction. The xstate_bv field at byte offset 512 in the XSAVE
extended state indicates what states the current process is in. If
the feature bit is cleared, the corresponding registers should be read as
0. If we update a register, we should set the corresponding feature
bit in the xstate_bv field.

We added PTRACE_GETREGSET and PTRACE_SETREGSET to Linux kernel to
fetch and store AVX registers with ptrace. Linux kernel also stores
XCR0 at the first 8 bytes of the software usable bytes, starting at
byte offset 464.

There are total 6 patches to add AVX support for Linux. They support:

1. The upper 128bit YMM registers are added for AVX support. The upper
128bit YMM registers are hidden from users. Gdb combines XMM register,
%xmmX, with 128bit YMM register, %ymmXh, and present the whole 256bit
YMM register, %ymmX, as pseudo register to users.
2. Backward compatible. If AVX isn't supported, SSE will be used.
3. Forward compatible. If new state beyond AVX is supported in
the XSAVE extended state, only AVX state will be used.
4. Remote gdb protocol extension. GDB will send "x86=xml" in qSupported
request packet to indicate that GDB supports x86 XML target desciption.
The gdb stub will send x86 XML target desciption if it sees "x86=xml"
in qSupported request packet.

One advantage of this approach is YMM registers are actually stored as
XMM registers and upper YMM registers in the XSAVE extended state. It
is easy and natural to access them as %xmmX and %ymmXh internally. We
just need to hide %ymmXh from users.

To support AVX on other OSes, the following changes are needed:

1. Kernel support to get/set the XSAVE extended state.
2. Handle 8/16 upper YMM registers.
3. Provide target to_read_description to return SSE or AVX target
description.
4. Update gdbarch_core_read_description to return SSE or AVX target
description based on contents of core dump.

H.J.