GDB Linux Awareness revisited

GDB Linux Awareness revisited

[Kieran Bingham]

Hi Yao, et al,

I am currently working through the GDB Linux Awareness project, started
by Peter Griffin and originally discussed in the gdb{,-patches}
mailinglist in June 2015
 [https://sourceware.org/ml/gdb-patches/2015-06/msg00040.html]

The main points that I took from that discussion were that it was a
valuable contribution to make, and that the main focus should be on
writing the support using the Python Extensions, so my hope is to try to
extend the python interfaces where possible to maximise the kernel
specific code living in the kernel.

Since Peter's original investigation, Jan Kiszka's work has been
integrated into the Kernel - paving the way for this work to follow on
and I have started taking functionality from ST's LKD implementation and
re-implementing the helper commands into linux.git/scripts/gdb/

The Linux Awareness support provided by STLinux can be (very roughly)
divided into the following categories:
 * Information inspection helper commands
   - /proc/{version,cmdline,mounts,iomem,ioports,meminfo,interrupts}
   - These are currently being implemented and will be sent for review
to LKML/Jan

 * Module Symbol loading and break pointing support
   - Fairly well supported already by the existing linux.git/scripts/gdb
work

 * Kernel thread awareness
   - thread list integration, and thread switching support
   - kernel frame-unwinder
   - syscall frame-unwinder
   - breakpoint filtering hooks

 * User space awareness support
   - process information
   - MMU control, and switching (yes, this is a big topic)
   - userspace frame-unwinder and symbol loading where available


The first topic, for which I hope to establish a bit more discussion and
(even encourage) bike-shedding for the moment is the Kernel Thread
Awareness.
Support for the frame-unwinders are already exposed via the Python
API's, however access to the gdb thread_list is not yet available, and
this is where I will be hacking next.


Briefly chatting on IRC, Pedro mentioned that exposing this support
could help for other runtimes such as Go subroutines, so my hope is that
this work will be re-usable in other places too.


Previous discussions on this have mentioned defining a consistent
location in the kernel to allow catching any new threads which are
created while running, however an important feature of LKD is to be able
to attach to an existing (either running, or crashed) target, so we need
to be able to enumerate threads at connection time as well.

If we implement a breakpointing method to add new threads as they are
created, will this provide us any benefit over iterating them on-demand
(breakpoint, target-stop)?
I would have argued that breakpointing on new process creation could
have a performance impact on a running target, but I suspect it will not
be too much of an impact.


Following this method as option A:

To implement this I could see that we would need to:
  Export the thread_list as an object through the python interface:
   - gdb.inferior.threads{.clearall, add, remove, iterator... }
   - Add some means of updating the registers for each thread

  On load (gdb.linux.__init__())
   - gdb.inferior.threads.clearall()
   - for thread in linux.inferior.threads():
        gdb.inferior.threads.add(thread)

  Establish constant points in the kernel to break point on for new
   process creation and destruction
   - Creation at the end of wake_up_new_task()
   - Destruction (near/at) the start of do_exit()


As an alternative option B:

STLinux LKD handles threads by repopulating the thread_list through the
to_update_thread_list() op, when the target is connected, or stopped.
This means that it doesn't rely on knowing when threads are created, and
simply walks the process list updating the thread list before the
(debug)user interacts with it.

Doing this however, would require the ability to 'hook' in gdb from
python to update the thread list, and this hook and more is already
supported through the target-ops layers.

Just to throw it into the mix for debate's sake; what do you think to
the idea of implementing a python object to represent a high-stratum
level target ops, which would allow us to (incrementally) implement
hooks in python and provide linux (or other application) specific
operation overrides. Any operations not implemented would simply
delegate down to the layer below.

This would allow us to almost mirror much more of the implementation
from ST in the kernel - but potentially be more complicated, so I would
like to hear peoples thoughts on it (especially if it's "You're crazy"
or "This is a minefield" and "Thar be dragons")



I feel like having the ability to access more hooks through establishing
a target layer would provide more extensibility as we move forwards, but
I also realise that it may not be a suitable method for generic thread
level functionality for other simpler applications such as the Go
subroutines, or other language implementations.


I look forward to any replies, thoughts, or suggestions

Regards
--
Kieran Bingham

Re: GDB Linux Awareness revisited

[Andreas Arnez]

On Mon, Jan 18 2016, Kieran Bingham wrote:

> Hi Yao, et al,
>
> I am currently working through the GDB Linux Awareness project, started
> by Peter Griffin and originally discussed in the gdb{,-patches}
> mailinglist in June 2015
>  [https://sourceware.org/ml/gdb-patches/2015-06/msg00040.html]
>
> The main points that I took from that discussion were that it was a
> valuable contribution to make, and that the main focus should be on
> writing the support using the Python Extensions, so my hope is to try to
> extend the python interfaces where possible to maximise the kernel
> specific code living in the kernel.

I am sure everybody agrees that "Linux awareness" would be a valuable
contribution to GDB.  And I doubt that you will encounter much
opposition against adding kernel specific code to GDB, unless it is
actually kernel *version* specific.  IMO, then it depends on the
expected amount of change over time.

Just in case you haven't seen, I presented this slide deck at the last
GNU Tools Cauldron:

  https://gcc.gnu.org/wiki/cauldron2015?action=AttachFile&do=get&target=Andreas+Arnez_+Debugging+Linux+kernel+dumps+with+GDB.pdf

These slides were mainly intended to give food for thought and get the
discussion going.  There were some follow-ups on the GDB mailing list,
e.g.:

  * About the gdb-kdump project:
    https://sourceware.org/ml/gdb/2015-09/threads.html#00014

  * About Linux Awareness and LKD:
    https://sourceware.org/ml/gdb/2015-10/threads.html#00000

[...]

>  * Module Symbol loading and break pointing support
>    - Fairly well supported already by the existing linux.git/scripts/gdb
> work

This only works if the debug target uses *virtual* rather than physical
kernel addresses, right?  Is this typical for the scenarios you want to
support?

[...]

> Support for the frame-unwinders are already exposed via the Python
> API's, however access to the gdb thread_list is not yet available, and
> this is where I will be hacking next.
>
>
> Briefly chatting on IRC, Pedro mentioned that exposing this support
> could help for other runtimes such as Go subroutines, so my hope is that
> this work will be re-usable in other places too.

Right, such a Python interface may be usable for Goroutines.  Here's the
Golang community's current approach to this:

  https://golang.org/src/runtime/runtime-gdb.py

Due to the Python interface's limitations, that script defines a command
"info goroutines" instead of supporting "info threads".  It also offers
a command prefix "goroutine <id>", rather than allowing the user to
switch contexts with "thread <id>".

Also note that gccgo has a different runtime and is not supported by
the script above.

--
Andreas

Re: GDB Linux Awareness revisited

[Kieran Bingham]

On 29/01/16 19:09, Andreas Arnez wrote:
> On Mon, Jan 18 2016, Kieran Bingham wrote:
>> Hi Yao, et al,
>>
>> I am currently working through the GDB Linux Awareness project, started
>> by Peter Griffin and originally discussed in the gdb{,-patches}
>> mailinglist in June 2015
>>  [https://sourceware.org/ml/gdb-patches/2015-06/msg00040.html]
>>
>> The main points that I took from that discussion were that it was a
>> valuable contribution to make, and that the main focus should be on
>> writing the support using the Python Extensions, so my hope is to try to
>> extend the python interfaces where possible to maximise the kernel
>> specific code living in the kernel.
> I am sure everybody agrees that "Linux awareness" would be a valuable
> contribution to GDB.  And I doubt that you will encounter much
> opposition against adding kernel specific code to GDB, unless it is
> actually kernel *version* specific.  IMO, then it depends on the
> expected amount of change over time.
The code from ST, which has been used since Linux 2.6.32 has already
been littered with various conditional paths for versions. This is
partly why I am trying to get as much of the kernel specific code to
function correctly in the python layer, so that it can live side by side
in the correct versions.

Backporting to earlier versions can become much cleaner, with simply
making the adjustments on that branch, rather than a wave of conditionals.

By ensuring that kernel specific code lives in the kernel (therefore as
Python) GDB will not be tied to supporting only a select set of kernel
versions.

> Just in case you haven't seen, I presented this slide deck at the last
> GNU Tools Cauldron:
>
>   https://gcc.gnu.org/wiki/cauldron2015?action=AttachFile&do=get&target=Andreas+Arnez_+Debugging+Linux+kernel+dumps+with+GDB.pdf
>
> These slides were mainly intended to give food for thought and get the
> discussion going.  There were some follow-ups on the GDB mailing list,
> e.g.:
>
>   * About the gdb-kdump project:
>     https://sourceware.org/ml/gdb/2015-09/threads.html#00014
>
>   * About Linux Awareness and LKD:
>     https://sourceware.org/ml/gdb/2015-10/threads.html#00000
>
> [...]
>
>>  * Module Symbol loading and break pointing support
>>    - Fairly well supported already by the existing linux.git/scripts/gdb
>> work
> This only works if the debug target uses *virtual* rather than physical
> kernel addresses, right?  Is this typical for the scenarios you want to
> support?

Yes, for the most part.

STLinuxLKD does in fact implement various memory functions including
virtual/physical address translations.
It provided the ability to control the MMU and adjust accordingly. My
discussions with Jan at the weekend suggested that this may not be a
desired route forward so it will need some more consideration.

And of course these were ARMv7 specific of course, and we would have to
look at how things get layered to consider performing mappings in the
future.

> [...]
>
>> Support for the frame-unwinders are already exposed via the Python
>> API's, however access to the gdb thread_list is not yet available, and
>> this is where I will be hacking next.
>>
>>
>> Briefly chatting on IRC, Pedro mentioned that exposing this support
>> could help for other runtimes such as Go subroutines, so my hope is that
>> this work will be re-usable in other places too.
> Right, such a Python interface may be usable for Goroutines.  Here's the
> Golang community's current approach to this:
>
>   https://golang.org/src/runtime/runtime-gdb.py

Thankyou, I had not seen this.

However, it does look very close of course to how they tasks are
currently listed in linux/scripts/gdb/ so perhaps we could work towards
a common way to implement this.

Have you seen my RFC regarding the gdb.Target layer object
representation? A python class here could iterate the tasks, and parse
the information for the info threads through hooks on the
.to_update_thread_list target operation and it's pals ...
> Due to the Python interface's limitations, that script defines a command
> "info goroutines" instead of supporting "info threads".  It also offers
> a command prefix "goroutine <id>", rather than allowing the user to
> switch contexts with "thread <id>".
Ok - so perhaps we could fix this with GoSubroutineAwareness(gdb.Target)

> Also note that gccgo has a different runtime and is not supported by
> the script above.
And perhaps a companion GccGoSubroutineAwareness(gdb.Target) could be
written to support this. :)
> --
> Andreas

Thankyou for your input, I see you have been following these topics for
a while, and you appear to be providing some of the glue between the
separate parties involved!

Regards

Kieran