Re: [RFC] Target Layer Python Interface

[Kieran Bingham <kieran.bingham@linaro.org>]

Hi Phil,

On 31/01/16 15:21, Phil Muldoon wrote:
> On 29/01/16 12:33, Kieran Bingham wrote:
>> While investigating how to get Python to define the threads presented by > the application layer (in my instance the linux kernel) I found that I > could create threads with a call from Python, (by writing a hook through > the inferior object)
> 
> A snippet would be cool to see what you are doing here, if possible.

My initial test was simply a method to generate PID's and add them to
test what would happen.

Commit viewable @ :
https://git.linaro.org/people/kieran.bingham/binutils-gdb.git/commitdiff/0e2692cdc6d195810ed61ac96ada91a715f94990


In testing this, I quickly discovered that the thread list was updated
as soon as I then called "info threads" which cleared down the list.


Thus, I then went to look at creating a gdb.Target object to mirror ST's
implementations, and provide the ability to call into the kernels python
iterators to generate the list instead.


So far I have created a gdb.Target python object representation which
has a call to register the layer from a python subclass (following the
implementation style of gdb.Command, where calling super.__init__
performs the registration)

An early version of (work in progress) gdb.Target is available here:

https://git.linaro.org/people/kieran.bingham/binutils-gdb.git/commitdiff/77ca621b2d2fcdc9376221833c9dbae557881090

And of course usual caveats of this branch will be rebased very soon :)

> 
>> However, whilst these threads were added, they were immediately removed > by the next call to (target).to_update_thread_list()
> 
> I presume you mean GDB's accounting of them, not the actual threads?

Aha, of course :) - Only GDB's 'view' of the threads are removed. Not
actual Kernel threads.


This brings me on to needing to implement .to_update_thread_list() et al.

I believe if we can expose this functionality through to Python, the
implementation will live in the kernel - and be common to the targets,
and maintain kernel specific versions in the correct place. The kernel.

> 
>> This has led me to believe I should create a Python Target layer > interface, which I have started, and have been able to register a > target, and provide Python bindings so that (currently _to_thread_name) > calls into the python layer object, and returns a new name. > > My intention is that this can be extended to provide the whole set of > target operations allowing me to implement much of the Linux Kernel > Debugger interface in Python.
> 
> At this point, there's just not enough information to form an opinion
> of this being a good thing or a bad thing. Do you have an interface in
> mind? An API?


The API, I would expect to match that of the Target API operations. I
would expect a one-to-one mapping of (required) operation names to
perform functions.

For instance, to support thread listings, I would expect something along
the lines of:

.to_update_thread_list
.to_pid_to_str
.to_extra_thread_info
.to_thread_alive
.to_fetch_registers
...

And having seen Jeff's work today, we could utilise Jeff's py-regcache
object quite effectively



> 
>>> Through this layer, we can even tackle some of the issues with memory > address spaces, by adding appropriate hooks to adjust MMU context, or > interpret page tables through the relevant target_ops hooks.
> 
> Interesting!
> 

It appears, that the guys at SUSE are already looking at SLAB table
walks, and I wonder if we could introduce this into our layers and
provide a common architecture for both kdump/core files, and live targets.

The STLinuxLKD implementation for interacting with Userspace is to
manipulate the MMU, which Jan has noted has issues as it changes the
state of the target you are debugging.

However if we can identify the data locations to read by walking tables,
then this could be a route to providing a much more enriched data retrieval




>> Some of the interface can even hopefully be autogenerated by a > make-target-delegate equivalent
> 
> Presume you mean the Python C code/interface?


Yes, I suspect some of the functionality to implement will be very
repeatable throughout each of the operation call implementations.


However, the more I look into it - the more I see each function is
likely to need very specific bindings, as it is not simple passing from
c function to c function.

Perhaps we can factor out commonality as we go - and try to keep as DRY
as possible, but I suspect it will be an iterative implementation process.




>> Before I head too much further, I wanted to ask the opinions of the > community as to whether this is an acceptable interface to implement, or > if it opens up too much of the GDB internals, to external dependencies.
> 
> This is something that is always a balancing act with Python
> GDB. There are, as you allude too, some parts of GDB that are not
> suitable to be extended to the Python layer, and some parts that while
> theoretically OK, need a little work to make safe and accessible in a
> Pythonic way.

I was discussing this with Jan this yesterday morning. We don't want a
fragile plugin implementation causing undesirable effects in the rest of
the GDB program space.

The layers would have to be able to gracefully handle failures in the
python functions, without catastrophic crashing of gdb. The layer would
also need to be able to enable/disable which is perhaps as simple as
poping it off the target stack..., or perhaps more likely having an
enable flag in the gdb.Target object, which would effectively make the
entire layer delegate to beneath.




> I can't comment without more details though. My initial reaction
> though is yeah, this sounds useful and exciting.

Perfect :)



> 
>> We could of course tackle this by providing a versioned API which would > allow adaptation in the Python objects, but I thought it was a big > enough question that it should be posed to a wider audience (wider than > just me!)
> 
> The versioned API would also need some documenting first.

Of course.




> I ask a few questions here that might provide you with some
> additional work! Sorry ;) But I find this an interesting proposal.

Ask away! I'm keen to have engagement. I'm considering putting
submitting a proposal for a presentation on the topic at ELC to follow
on from Peter's presentation in Dublin, and hopefully find out what use
cases people would *actually* expect from this.


I think we already can see that we will need to make this layer work
with a large group of users.

I hope that it will be common across remote(jtag/serial) targets,
including QEmu/KVM stubs, and with the work from the guys at SUSE we
should be able to cater this for kdump cores too.

I think the importance is to make sure everything works well together.




> 
> Cheers
> 
> Phil
> 
> 

Regards

Kieran