how GDB use ptrace to return from a function

[Yubin Ruan <ablacktshirt@gmail.com>]

Hi GDB developer ;-)

I am writing a toy debugger and currently looking into the GDB source
because I want to know: after setting the tracee's registers and
trying to let it execute a function with ptrace(PTRACE_CONT, ...), how
can the tracee return to the tracer?

Currently I manipulate the tracee's stack and place a NULL return
address there (I am on X86), so that after ptrace(PTRACE_CONT, ...),
the tracee will execute a function and return, at which point a SIGSEV
is generated (because the return address is NULL), so tracee will be
caught by the tracer again.

I don't know whether GDB is using this kind of technique. If anyone
know that, can you enlighten me, and probably point me to the source?

Yubin