[PATCH v2 00/47] A new target to debug Intel GPUs

Tankut Baris Aktemur
2024-12-13T16:14:27Z

Hello,

We (Intel) would like to submit patches to enable fundamental debug
support for Intel GPU devices. In the future, we plan to add more
patches that improve the performance and the user experience.
Those patches are already available in the downstream "Intel
Distribution for GDB" debugger at

https://github.com/intel/gdb

The v1 of the submission is available at

https://sourceware.org/pipermail/gdb-patches/2024-July/210264.html

This revision (v2) makes the following changes:

- The comments that have been received so far are addressed.

- Patches are rebased on the master branch.

- A new patch is added:
"gdbserver: allow configuring for a heterogeneous target"

- The patch titled "gdb: load solibs even when exec_bfd does not
exist" has been revised and is now titled "gdb: load solibs if the
target does not have the notion of an exec file".

- The testsuite now includes a proc named 'sycl_with_intelgt_lock'
similar to 'with_rocm_gpu_lock'.

GPU threads operate in a SIMD (single instruction multiple data)
manner: they are vectorized, where each lane (also known as "execution
channel") executes the same instruction but using different data
values. Lanes of the same thread execute in a lock-step movement.
Displaying the value of a source program variable therefore requires
not only a thread context but also a lane context. GDB currently does
not have this knowledge built-in. Furthermore, some DWARF extensions
are necessary to express data locations in a lane-relative way, which
are currently under discussion of or to be submitted to the DWARF
committee. Hence, with this submission, variables may appear with an
error like "<error reading variable: Unhandled dwarf expression opcode
0xed>". Similar restrictions apply also to the AMD ROCm (AMDGPU)
target in the upstream GDB for the same reasons. The downstream
"Intel Distribution for GDB" debugger implements lane support as well
as DWARF extensions and hence is able to print lane-relative values
properly. Lane support is a future GDB topic; see a BoF hosted by
Intel and AMD in GNU Tools Cauldron 2024 for more details if
interested.

We provide a gdbserver low target definition. The target uses the
Level-Zero debug API:

https://spec.oneapi.io/level-zero/latest/tools/PROG.html#program-debug
https://spec.oneapi.io/level-zero/latest/tools/api.html#debug

The user-space implementation of the Level-Zero Debug API comes from
"Intel(R) Graphics Compute Runtime for oneAPI Level Zero and
OpenCL(TM) Driver":

https://github.com/intel/compute-runtime

The kernel-space implementation of the Level-Zero Debug API, i.e. the
'eudebug' feature of the "Xe Intel graphics driver", is under submission:

https://lists.freedesktop.org/archives/intel-xe/2024-December/061476.html (v3)
https://lists.freedesktop.org/archives/intel-xe/2024-October/052260.html (v2)
https://lists.freedesktop.org/archives/intel-xe/2024-July/043605.html (v1)

For Level-Zero based devices, we model hardware threads. There is one
GDB thread for each hardware thread on the device. We opted for this
model for the following reasons:

- Programs that use GPUs to accelerate computation typically offload
many computation kernels. Hence, software threads in GPUs have
much shorter lives than threads in multi-threaded CPU programs.
For real-world cases, the data processed by the GPU is typically
large, causing the number of software threads to be usually higher
than the number of available hardware threads. Therefore, dealing
with software threads may cause proliferation of threads.
Modeling hardware threads, on the other hand, means that they
would be created once at the beginning of the debug session and
then the list of threads stays stable.

- As of today, Intel GPUs do not switch context for threads. That
is, once a software thread is assigned to run on a particular
hardware thread, it always runs on that hardware thread until
termination. Therefore, focusing on a hardware thread does not
create context switch confusion for the user that would otherwise
be experienced with e.g. CPU threads.

Hardware threads may be idle inbetween computation kernel executions
or when a kernel does not utilize the GPU fully. They may also be
used by applications other than the one currently under debug. During
these times, those hardware threads cannot be interacted with
(e.g. cannot be interrupted) by the current debug user and appear as
unavailable. To handle this case, we introduce an UNAVAILABLE wait
kind and also model it as a thread execution state. In particular,
UNAVAILABLE means that we have tried to stop the thread and failed.

The Intel GPU target can be used in combination with a native target,
relying on GDB's multi-target feature, to debug the GPU and the host
application in the same debug session. For this, bring the native app
(e.g. a SYCL [https://www.khronos.org/sycl/] program) to a state
where the Level-Zero backend for the GPU has been initialized (e.g.
after the first queue has been created in SYCL), then create a
gdbserver instance and connect to it from a second inferior.
At GNU Tools Cauldron 2024, we gave a talk presenting the approach.
We'd welcome interested parties to watch the recording.

Below is a sample session that shows how to set up inferiors and
targets manually. In the downstream debugger, a Python script is used
to take these steps in an automated manner for better user experience.

$ gdb demo
...
(gdb) maintenance set target-non-stop on
(gdb) tbreak 60
Temporary breakpoint 1 at 0x4049c8: file demo.cpp, line 60.
(gdb) run
...
[SYCL] Using device: [Intel(R) Graphics ...] from [Intel(R) oneAPI Unified Runtime over Level-Zero]

Thread 1 "demo" hit Temporary breakpoint 1, main (argc=1, argv=0x7fffffffd9b8) at demo.cpp:60
60 range data_range{length};
(gdb)

# Connect the Intel GT gdbserver by specifying the host inferior PID.

(gdb) add-inferior -no-connection
[New inferior 2]
Added inferior 2
(gdb) inferior 2
[Switching to inferior 2 [<null>] (<noexec>)]
(gdb) info inferiors
Num Description Connection Executable
1 process 16458 1 (native) /temp/demo
* 2 <null>
(gdb) target remote | gdbserver-intelgt --attach - 16458
Remote debugging using | gdbserver-intelgt --attach - 16458
Attached; given pid = 16458, updated to 1
Remote debugging using stdio
<unavailable> in ?? ()
(gdb)

We also include patches for the testsuite, where we introduce the
infrastructure and a number of test cases using SYCL.

For convenience, the patches in this series are available at

https://github.com/intel/gdb/tree/upstream/intelgt-mvp

To those who may want to try the debugger, we also provide

https://github.com/intel/gdb/tree/upstream/intelgt-mvp-plus

with a number of additional patches (not yet upstreamed) that bring
(1) SIMD lane support, (2) ability to make GPU threads do inferior
calls for expression evaluation, (3) a minimal Python script that
starts and connects gdbserver-intelgt automatically for more
convenience. Submission of these additional features (and more) is
planned for future after the fundamental debug support is accepted.

Best regards,
Baris

---
Albertano Caruso (2):
gdb, intelgt: add disassemble feature for the Intel GT architecture.
testsuite, arch, intelgt: add a disassembly test

Klaus Gerlicher (1):
gdb, ze: on a whole process stop, mark all threads as not_resumed

Markus Metzger (15):
gdb, arch, intelgt: add intelgt arch definitions
gdbsupport, filestuff, ze: temporary files
gdb, gdbserver, ze: in-memory libraries
gdb, gdbserver, rsp, ze: acknowledge libraries
gdb, solib, ze: solib_bfd_open_from_target_memory
gdb, remote, ze: fix "$Hc-1#09...Packet received: E01" during startup
gdb, infrun, ze: allow saving process events
gdb, ze: add TARGET_WAITKIND_UNAVAILABLE
gdb, infrun, ze: handle stopping unavailable threads
gdb, infrun, ze: allow resuming unavailable threads
gdb, gdbserver, ze: add U stop reply
gdb, gdbserver, ze: add library notification to U stop reply
gdbserver: wait for stopped threads in queue_stop_reply_callback
gdb, dwarf, ze: add DW_OP_INTEL_regval_bits
gdbserver, ze, intelgt: introduce ze-low and intel-ze-low targets

Natalia Saiapova (2):
bfd: add intelgt target to BFD
gdb: do not create a thread after a process event.

Nils-Christian Kempke (1):
gdb, gdbserver, gdbsupport: add 'device' tag to XML target description

Tankut Baris Aktemur (26):
gdb, intelgt: add intelgt as a basic machine
ld: add intelgt as a target configuration
opcodes: add intelgt as a configuration
gdb, intelgt: add the target-dependent definitions for the Intel GT architecture
gdbserver, ze: report TARGET_WAITKIND_UNAVAILABLE events
gdb, ze: handle TARGET_WAITKIND_UNAVAILABLE in stop_all_threads
gdb, remote: handle thread unavailability in print_one_stopped_thread
gdb, remote: do 'remote_add_inferior' in 'remote_notice_new_inferior' earlier
gdb, remote: handle a generic process PID in remote_notice_new_inferior
gdb, remote: handle a generic process PID in process_stop_reply
gdb: use the pid from inferior in setup_inferior
gdb: revise the pid_to_exec_file target op
gdb: load solibs if the target does not have the notion of an exec file
gdbserver: import AC_LIB_HAVE_LINKFLAGS macro into the autoconf script
gdbserver: add a pointer to the owner thread in regcache
gdbserver: dump 'xx...x' in collect_register_as_string for unavailable register
gdbserver: adjust pid after the target attaches
gdbserver: allow configuring for a heterogeneous target
testsuite, sycl: add SYCL support
testsuite, sycl: add test for backtracing inside a kernel
testsuite, sycl: add test for 'info locals' and 'info args'
testsuite, sycl: add tests for stepping and accessing data elements
testsuite, sycl: add test for 1-D and 2-D parallel_for kernels
testsuite, sycl: add test for scheduler-locking
testsuite, arch, intelgt: add intelgt-program-bp.exp
testsuite, sycl: test canceling a stepping flow

bfd/Makefile.am | 2 +
bfd/Makefile.in | 4 +
bfd/archures.c | 4 +
bfd/bfd-in2.h | 6 +
bfd/config.bfd | 13 +-
bfd/configure | 1 +
bfd/configure.ac | 1 +
bfd/cpu-intelgt.c | 57 +
bfd/elf64-intelgt.c | 195 ++
bfd/libbfd.h | 2 +
bfd/reloc.c | 7 +
bfd/targets.c | 2 +
binutils/dwarf.c | 6 +
binutils/readelf.c | 9 +
config.sub | 5 +-
gdb/Makefile.in | 8 +-
gdb/NEWS | 27 +
gdb/arch/intelgt.c | 77 +
gdb/arch/intelgt.h | 175 ++
gdb/config.in | 3 +
gdb/configure | 537 +++-
gdb/configure.ac | 40 +
gdb/configure.tgt | 5 +
gdb/disasm-selftests.c | 4 +
gdb/doc/gdb.texinfo | 153 +-
gdb/dwarf2/expr.c | 36 +
gdb/dwarf2/expr.h | 5 +
gdb/dwarf2/loc.c | 2 +
gdb/exec.c | 6 +
gdb/features/gdb-target.dtd | 19 +-
gdb/features/library-list.dtd | 12 +-
gdb/fork-child.c | 10 +-
gdb/gdbthread.h | 12 +-
gdb/infcmd.c | 14 +-
gdb/inferior.h | 4 +
gdb/infrun.c | 124 +-
gdb/intelgt-tdep.c | 1093 +++++++++
gdb/nat/fork-inferior.c | 10 +
gdb/regcache.c | 6 +-
gdb/remote.c | 196 +-
gdb/solib-target.c | 142 +-
gdb/solib.c | 103 +-
gdb/solist.h | 25 +-
gdb/target-delegates-gen.c | 50 +
gdb/target-descriptions.c | 19 +
gdb/target.c | 16 +
gdb/target.h | 24 +
gdb/target/waitstatus.c | 1 +
gdb/target/waitstatus.h | 22 +
gdb/testsuite/README | 9 +
gdb/testsuite/boards/intel-offload.exp | 36 +
gdb/testsuite/gdb.arch/intelgt-disassemble.exp | 82 +
gdb/testsuite/gdb.arch/intelgt-program-bp.exp | 83 +
gdb/testsuite/gdb.arch/sycl-simple.cpp | 42 +
gdb/testsuite/gdb.sycl/break.exp | 63 +
gdb/testsuite/gdb.sycl/break2.exp | 66 +
gdb/testsuite/gdb.sycl/call-stack.cpp | 92 +
gdb/testsuite/gdb.sycl/call-stack.exp | 179 ++
gdb/testsuite/gdb.sycl/info-locals-and-args.exp | 78 +
gdb/testsuite/gdb.sycl/parallel-for-1D.cpp | 72 +
gdb/testsuite/gdb.sycl/parallel-for-1D.exp | 55 +
gdb/testsuite/gdb.sycl/parallel-for-2D.cpp | 73 +
gdb/testsuite/gdb.sycl/parallel-for-2D.exp | 55 +
gdb/testsuite/gdb.sycl/scheduler-locking.exp | 67 +
gdb/testsuite/gdb.sycl/single-task.cpp | 50 +
gdb/testsuite/gdb.sycl/step-canceled.exp | 86 +
gdb/testsuite/gdb.sycl/step-into-function.exp | 47 +
gdb/testsuite/gdb.sycl/step-parallel-for.exp | 63 +
gdb/testsuite/gdb.sycl/step.exp | 51 +
gdb/testsuite/gdb.threads/killed-outside.exp | 4 +
gdb/testsuite/lib/gdb.exp | 17 +-
gdb/testsuite/lib/intelgt-utils.exp | 43 +
gdb/testsuite/lib/sycl-devices.cpp | 107 +
gdb/testsuite/lib/sycl-hello.cpp | 43 +
gdb/testsuite/lib/sycl-util.cpp | 135 +
gdb/testsuite/lib/sycl.exp | 410 ++++
gdb/thread.c | 2 +-
gdb/top.c | 10 +
gdb/xml-tdesc.c | 76 +
gdbserver/Makefile.in | 4 +-
gdbserver/acinclude.m4 | 5 +
gdbserver/config.in | 6 +
gdbserver/configure | 500 ++++
gdbserver/configure.ac | 18 +
gdbserver/configure.srv | 15 +-
gdbserver/dll.cc | 155 +-
gdbserver/dll.h | 36 +-
gdbserver/gdbthread.h | 2 +-
gdbserver/intelgt-ze-low.cc | 1041 ++++++++
gdbserver/linux-low.cc | 6 +-
gdbserver/linux-low.h | 2 +-
gdbserver/netbsd-low.cc | 2 +-
gdbserver/netbsd-low.h | 2 +-
gdbserver/regcache.cc | 26 +-
gdbserver/regcache.h | 3 +
gdbserver/remote-utils.cc | 21 +
gdbserver/server.cc | 254 +-
gdbserver/server.h | 7 +
gdbserver/target.cc | 14 +
gdbserver/target.h | 31 +-
gdbserver/tdesc.cc | 16 +
gdbserver/tdesc.h | 3 +
gdbserver/win32-low.cc | 4 +-
gdbserver/win32-low.h | 2 +-
gdbserver/ze-low.cc | 2995 +++++++++++++++++++++++
gdbserver/ze-low.h | 492 ++++
gdbsupport/filestuff.cc | 53 +
gdbsupport/filestuff.h | 6 +
gdbsupport/tdesc.cc | 40 +
gdbsupport/tdesc.h | 90 +
include/dwarf2.def | 4 +
include/elf/intelgt.h | 39 +
ld/configure.tgt | 2 +
opcodes/configure | 1 +
opcodes/configure.ac | 1 +
115 files changed, 11245 insertions(+), 143 deletions(-)
---
base-commit: 038590b067bafb19ceec62f73fe0b41ec37e8236
change-id: 20241213-upstream-intelgt-mvp-684d5f2f6730

Best regards,
--
Tankut Baris Aktemur <tankut.baris.aktemur@intel.com>