Re: [RFC PATCH v1] Support inspecting green threads

["Botond Dénes" <bdenes@scylladb.com>]

On Mon, 2020-08-17 at 15:34 -0600, Tom Tromey wrote:
> > > > > > "Botond" == Botond Dénes <bdenes@scylladb.com> writes:
> 
> Botond> Some applications use green threads that have their own
> stacks. One such
> Botond> example is Scylla [1] which is using the seastar framework
> [2], which
> Botond> provides green threads in the form of seastar::thread [3].
> These threads
> Botond> are created with `setcontext()` and later we switch in/out
> using
> Botond> `setjmp()`/`longjmp()`
> 
> Thanks for bringing this up.  We've talked about adding a facility
> like
> this a couple of times -- maybe it's finally time.
> 
> Botond> This patch aims to solve this problem by providing a family
> of commands
> Botond> which allow inspecting green threads.
> Botond> * fiber view - allows inspecting a green thread. This command
> takes a
> Botond>   list of registers as its arguments. More on these later.
> Botond> * fiber reset - reset the viewed fiber to the stack proper.
> 
> I like the name.


Glad to hear it. :)

> 
> Botond> TODO:
> Botond> * Better interface for the command: instead of a fixed set of
> registers
> Botond>   (those that happen to be in glibc's jmpbuf on amd64), allow
> the user
> Botond>   to pass any amount of named registers to override.
> Botond> * More documentation.
> Botond> * More testing.
> Botond> * Code style.
> 
> Botond> I apologize for the mixed code style, but I wanted to probe
> my approach
> Botond> first, before spending more time on polishing.
> 
> Yeah, don't worry about that yet.
> 
> It may interest you to know that there's already an implementation of
> something like this in gdb -- the ravenscar thread code.  Ravenscar
> is
> an Ada runtime that implements green threads.  (It is a bit weird
> though
> in that these threads never exit.)
> 
> gdb handles this by making "fake" threads corresponding to each green
> thread.  If you "next" or the like, it uses the underlying real
> thread
> when communicating with the remote.
> 
> One thought I had was that maybe we could generalize this code.  In
> conjunction with a Python API, we could make it so that 3rd parties
> (the
> green thread library) could provide the knowledge needed to hook gdb
> to
> the threads.
> 
> A consequence of taking this approach would be that fibers could be
> integrated with threads, with the upside being that fibers would
> automatically start working in gdb GUIs.  (But maybe there are
> downsides
> as well -- for instance if a program uses setcontext in a
> non-thread-like way.  This is worth discussing.)
> 
> I think we'd have to add some kind of "fiber" flag to threads and
> then
> have a check on operations like "step" to make sure that an attempt
> to
> step a paused thread is rejected.  I don't know for sure but maybe
> we'd
> need some kind of tweak to ptid as well.


This would indeed make these green threads first class citizens in gdb.
Now I started thinking whether we could extend this support to
stackless fibers. In seastar there are two kinds of fibers: stackful
(we call these `seastar::thread` -- these are the green threads) and
stackless. The latter is future-promise based and we have experimental
glue code to enable C++20 coroutines based on this.

These stackless fibers are made up of continuation objects, which form
an intrusive forward linked-list. Each continuation object is a
callable and a pointer to the next continuation that should be executed
next, with the value computed by this continuation.

These fibers are an absolute pain to debug in gdb. Backtracing of
course just takes you back to the event loop. As the continuations are
classes templated on lambdas, their type name (even if known based on
the vptr of the continuation object) is unreadable.

I'm wondering if we could piece together these continuations into a
pseudo backtrace, making at least lambda captures (those the optimizer
forgot to destroy) inspectable. AFAIK gdb already has a notion of
pseudo frames that it uses to render inline frames into backtraces.
We should of course allow this mode to be disabled because sometimes
you do want to look at the event loop frames.

>   Finally, we'd need something
> like the "random thread switch" patch that I've been wanting to get
> in
> for Ravenscar :-)
> 
> A typical way for this to work would be to have some Python code that
> sets hidden breakpoints in the thread-start and thread-exit code in a
> green thread library.  Then, when a thread is started, register it
> with
> gdb core as one of these "fiber" threads.  The Python code would also
> be
> responsible for supplying registers on request.
> 
> One other thing I'm not sure of in this approach is how non-stop
> would
> work.  I suppose the Python code would need to help with this as
> well;
> but that seems odd because it implies maybe adding otherwise
> undesirable
> facilities to green threads libraries.  So, perhaps we'd need to
> change
> how this works in gdb instead.  (I think this isn't an issue for
> Ravenscar at present because, IIUC, Ravenscar statically assigns
> threads
> to CPUs; but also because presumably nobody has ever tried the
> combination.)
> 
> Tom