Re: Stack unwinding for green threads

["Botond Dénes" <bdenes@scylladb.com>]

On Fri, 2020-07-03 at 11:16 +0100, Andrew Burgess wrote:
> * Botond Dénes <bdenes@scylladb.com> [2020-07-03 12:50:54 +0300]:
> 
> > Hi,
> > 
> > I'm working on Scylla [1], an application which is built using the
> > seastar framework [2]. This framework provides green threads [3]
> > that
> > have their own stacks. These threads are created with
> > `setcontext()`
> > and later we switch in/out using `setjmp()`/`longjmp()`.
> > 
> > We have a collection of python scripts [4] to help debug Scylla,
> > among
> > these we have a utility command which allows switching in/out of
> > these
> > green threads in gdb. This command basically (tries) to emulate
> > `setjmp()`/`longjmp()` in python, saving and restoring registers.
> > There
> > are several problems with this method. For starters it crashes gdb
> > for
> > some time now, and also it doesn't work in coredumps, where gdb
> > refuses
> > to write to registers (even after `set write on`). So for these
> > reasons
> > I started to look for alternative ways to unwind the stacks of our
> > green threads. However after going through the gdb documentation
> > [5],
> > reading all I could find about threads, stacks, frames and the
> > Python
> > API I haven't found an entry point to call which would unwind a
> > stack
> > located at a custom address (not the current $rsp). Is there such
> > an
> > API that I haven't found? Is there another way to achive what I
> > want?
> > Alternatively, how hard would it be to implement such an API?
> 
> No, there's currently nothing in GDB for having a look at a different
> stack.
> 
> There is 'frame view ...' [1] which I'm sure someone might mention,
> as
> it almost, sort-of, kind-of feels like this should solve your
> problem,
> but it wont.
> 
> I did do some work on this area, years ago now, and at the time my
> plan was to add a set of commands that solved exactly this problem,
> my
> intention was to have some kind of 'stack create' command that would
> allow you to supply an entire register set (if you wanted) and then
> GDB would unwind assuming that this was the "current" register
> values.  Of course, the only "required" registers would be $sp, and
> probably $pc too, but you would have been able to provide others if
> you wanted.  Anyway I digress, my need for this work disappeared and
> this kind of fell of my radar, though I would like to find time to
> revisit this ... one day.
> 
> However, thinking about the above kind of gave me an idea, it's not
> perfect, but it might work.  How about writing a custom Python
> Unwinder[2]?
> 
> Here's how I think this might help you:
> 
>   (1) By default the custom unwinder is "off", it doesn't claim any
>   frames, your stack unwinds in the "normal" way.
> 
>   (2) Add a command 'magic-stack-unwind <address-of-register-set>',
>   this turns "on" (sets a flag) the unwinder, and stores the address
>   of the register set you want to unwind with.
> 
>   (3) The unwinder, now "on" would always claim frames at depth 0,
> and
>   would return all of the registers from your stored register set.
> 
> I think that this would present you with your alternative stack, but
> offset by 1 frame, so I think you'd still get the original "current"
> frame, but everything below that would be from your alternative
> register stack.  Not ideal maybe, but it might work.


I tried this, but gdb doesn't seem to like the fact that I start
returing heap addresses (where customs stacks are located) as $rsp, I
get:

Backtrace stopped: previous frame inner to this frame (corrupt stack?)

> 
> It might even be possible to make use of frame filters[3] to hide
> frame 0, though again, I've never tried hiding frame 0, only even
> frames > 0, so I don't know how that would work for you.
> 
> Good luck,
> Andrew
> 
> 
> 
> [1] 
> https://sourceware.org/gdb/current/onlinedocs/gdb/Selection.html#Selection
> [2] 
> https://sourceware.org/gdb/current/onlinedocs/gdb/Unwinding-Frames-in-Python.html#Unwinding-Frames-in-Python
> [3] 
> https://sourceware.org/gdb/current/onlinedocs/gdb/Frame-Filter-API.html#Frame-Filter-API
> > Thanks,
> > Botond
> > 
> > [1] https://github.com/scylladb/scylla
> > [2] https://github.com/scylladb/seastar
> > [3] http://docs.seastar.io/master/group__thread-module.html
> > [4] https://github.com/scylladb/scylla/blob/master/scylla-gdb.py
> > [5] https://sourceware.org/gdb/onlinedocs/gdb/index.html
> >