[RFC] Target Layer Python Interface

[RFC] Target Layer Python Interface

[Kieran Bingham]

While investigating how to get Python to define the threads presented by
the application layer (in my instance the linux kernel) I found that I
could create threads with a call from Python, (by writing a hook through
the inferior object)

However, whilst these threads were added, they were immediately removed
by the next call to (target).to_update_thread_list()


This has led me to believe I should create a Python Target layer
interface, which I have started, and have been able to register a
target, and provide Python bindings so that (currently _to_thread_name)
calls into the python layer object, and returns a new name.


My intention is that this can be extended to provide the whole set of
target operations allowing me to implement much of the Linux Kernel
Debugger interface in Python.


Through this layer, we can even tackle some of the issues with memory
address spaces, by adding appropriate hooks to adjust MMU context, or
interpret page tables through the relevant target_ops hooks.


Some of the interface can even hopefully be autogenerated by a
make-target-delegate equivalent

This layer interface, will mean that I can provide a Linux Thread
Awareness layer to implement a thread_stratum layer, and can even go
further to provide extra layers for other functionality later.

With a example implementation looking like the start of this:

======================================================================
class LxAwarenessTarget(gdb.Target):
    """ Provide a Linux Awareness Target Layer

This will provide hooks for our Thread implementation, and later
extra architecture specific target layers to handle memory """

    def __init__(self):
        self.shortname = "LxThreads"
        self.longname = "Linux Thread Integration Layer"
        super(LxAwarenessTarget, self).__init__(gdb.Stratums.thread)

    def to_thread_name(self, gdbthread):
        return "A thread"

    ...

(gdb) maintenance print target-stack
The current target stack is:
  - LxThreads (Linux Thread Integration Layer)
  - remote (Remote serial target in gdb-specific protocol)
  - exec (Local exec file)
  - None (None)

(gdb) info threads
  Id   Target Id         Frame
  2    Thread 2 "A thread" (CPU#1 [halted ]) default_idle ()
    at /home/linaro/sources/linux/arch/x86/kernel/process.c:305
* 1    Thread 1 "A thread" (CPU#0 [halted ]) default_idle ()
    at /home/linaro/sources/linux/arch/x86/kernel/process.c:305
======================================================================



By implementing this as a layer, it will also allow other
implementations from OS's or targets to define their specific
implementation detail in their source trees


Before I head too much further, I wanted to ask the opinions of the
community as to whether this is an acceptable interface to implement, or
if it opens up too much of the GDB internals, to external dependencies.

Whilst the target layer may be defined as 'fairly' stable - It has
undergone changes, and this may cause issues if exposed to an external
interface.

We could of course tackle this by providing a versioned API which would
allow adaptation in the Python objects, but I thought it was a big
enough question that it should be posed to a wider audience (wider than
just me!)

Regards
--
Kieran

Re: [RFC] Target Layer Python Interface

[Phil Muldoon]

On 29/01/16 12:33, Kieran Bingham wrote:
> While investigating how to get Python to define the threads presented by > the application layer (in my instance the linux kernel) I found that I > could create threads with a call from Python, (by writing a hook through > the inferior object)

A snippet would be cool to see what you are doing here, if possible.

> However, whilst these threads were added, they were immediately removed > by the next call to (target).to_update_thread_list()

I presume you mean GDB's accounting of them, not the actual threads?


> This has led me to believe I should create a Python Target layer > interface, which I have started, and have been able to register a > target, and provide Python bindings so that (currently _to_thread_name) > calls into the python layer object, and returns a new name. > > My intention is that this can be extended to provide the whole set of > target operations allowing me to implement much of the Linux Kernel > Debugger interface in Python.

At this point, there's just not enough information to form an opinion
of this being a good thing or a bad thing. Do you have an interface in
mind? An API?

> > Through this layer, we can even tackle some of the issues with memory > address spaces, by adding appropriate hooks to adjust MMU context, or > interpret page tables through the relevant target_ops hooks.

Interesting!

> Some of the interface can even hopefully be autogenerated by a > make-target-delegate equivalent

Presume you mean the Python C code/interface?


> Before I head too much further, I wanted to ask the opinions of the > community as to whether this is an acceptable interface to implement, or > if it opens up too much of the GDB internals, to external dependencies.

This is something that is always a balancing act with Python
GDB. There are, as you allude too, some parts of GDB that are not
suitable to be extended to the Python layer, and some parts that while
theoretically OK, need a little work to make safe and accessible in a
Pythonic way.

I can't comment without more details though. My initial reaction
though is yeah, this sounds useful and exciting.


> We could of course tackle this by providing a versioned API which would > allow adaptation in the Python objects, but I thought it was a big > enough question that it should be posed to a wider audience (wider than > just me!)

The versioned API would also need some documenting first.

I ask a few questions here that might provide you with some
additional work! Sorry ;) But I find this an interesting proposal.

Cheers

Phil

Re: [RFC] Target Layer Python Interface

[Kieran Bingham]

Hi Phil,

On 31/01/16 15:21, Phil Muldoon wrote:
> On 29/01/16 12:33, Kieran Bingham wrote:
>> While investigating how to get Python to define the threads presented by > the application layer (in my instance the linux kernel) I found that I > could create threads with a call from Python, (by writing a hook through > the inferior object)
> 
> A snippet would be cool to see what you are doing here, if possible.

My initial test was simply a method to generate PID's and add them to
test what would happen.

Commit viewable @ :
https://git.linaro.org/people/kieran.bingham/binutils-gdb.git/commitdiff/0e2692cdc6d195810ed61ac96ada91a715f94990


In testing this, I quickly discovered that the thread list was updated
as soon as I then called "info threads" which cleared down the list.


Thus, I then went to look at creating a gdb.Target object to mirror ST's
implementations, and provide the ability to call into the kernels python
iterators to generate the list instead.


So far I have created a gdb.Target python object representation which
has a call to register the layer from a python subclass (following the
implementation style of gdb.Command, where calling super.__init__
performs the registration)

An early version of (work in progress) gdb.Target is available here:

https://git.linaro.org/people/kieran.bingham/binutils-gdb.git/commitdiff/77ca621b2d2fcdc9376221833c9dbae557881090

And of course usual caveats of this branch will be rebased very soon :)

> 
>> However, whilst these threads were added, they were immediately removed > by the next call to (target).to_update_thread_list()
> 
> I presume you mean GDB's accounting of them, not the actual threads?

Aha, of course :) - Only GDB's 'view' of the threads are removed. Not
actual Kernel threads.


This brings me on to needing to implement .to_update_thread_list() et al.

I believe if we can expose this functionality through to Python, the
implementation will live in the kernel - and be common to the targets,
and maintain kernel specific versions in the correct place. The kernel.

> 
>> This has led me to believe I should create a Python Target layer > interface, which I have started, and have been able to register a > target, and provide Python bindings so that (currently _to_thread_name) > calls into the python layer object, and returns a new name. > > My intention is that this can be extended to provide the whole set of > target operations allowing me to implement much of the Linux Kernel > Debugger interface in Python.
> 
> At this point, there's just not enough information to form an opinion
> of this being a good thing or a bad thing. Do you have an interface in
> mind? An API?


The API, I would expect to match that of the Target API operations. I
would expect a one-to-one mapping of (required) operation names to
perform functions.

For instance, to support thread listings, I would expect something along
the lines of:

.to_update_thread_list
.to_pid_to_str
.to_extra_thread_info
.to_thread_alive
.to_fetch_registers
...

And having seen Jeff's work today, we could utilise Jeff's py-regcache
object quite effectively



> 
>>> Through this layer, we can even tackle some of the issues with memory > address spaces, by adding appropriate hooks to adjust MMU context, or > interpret page tables through the relevant target_ops hooks.
> 
> Interesting!
> 

It appears, that the guys at SUSE are already looking at SLAB table
walks, and I wonder if we could introduce this into our layers and
provide a common architecture for both kdump/core files, and live targets.

The STLinuxLKD implementation for interacting with Userspace is to
manipulate the MMU, which Jan has noted has issues as it changes the
state of the target you are debugging.

However if we can identify the data locations to read by walking tables,
then this could be a route to providing a much more enriched data retrieval




>> Some of the interface can even hopefully be autogenerated by a > make-target-delegate equivalent
> 
> Presume you mean the Python C code/interface?


Yes, I suspect some of the functionality to implement will be very
repeatable throughout each of the operation call implementations.


However, the more I look into it - the more I see each function is
likely to need very specific bindings, as it is not simple passing from
c function to c function.

Perhaps we can factor out commonality as we go - and try to keep as DRY
as possible, but I suspect it will be an iterative implementation process.




>> Before I head too much further, I wanted to ask the opinions of the > community as to whether this is an acceptable interface to implement, or > if it opens up too much of the GDB internals, to external dependencies.
> 
> This is something that is always a balancing act with Python
> GDB. There are, as you allude too, some parts of GDB that are not
> suitable to be extended to the Python layer, and some parts that while
> theoretically OK, need a little work to make safe and accessible in a
> Pythonic way.

I was discussing this with Jan this yesterday morning. We don't want a
fragile plugin implementation causing undesirable effects in the rest of
the GDB program space.

The layers would have to be able to gracefully handle failures in the
python functions, without catastrophic crashing of gdb. The layer would
also need to be able to enable/disable which is perhaps as simple as
poping it off the target stack..., or perhaps more likely having an
enable flag in the gdb.Target object, which would effectively make the
entire layer delegate to beneath.




> I can't comment without more details though. My initial reaction
> though is yeah, this sounds useful and exciting.

Perfect :)



> 
>> We could of course tackle this by providing a versioned API which would > allow adaptation in the Python objects, but I thought it was a big > enough question that it should be posed to a wider audience (wider than > just me!)
> 
> The versioned API would also need some documenting first.

Of course.




> I ask a few questions here that might provide you with some
> additional work! Sorry ;) But I find this an interesting proposal.

Ask away! I'm keen to have engagement. I'm considering putting
submitting a proposal for a presentation on the topic at ELC to follow
on from Peter's presentation in Dublin, and hopefully find out what use
cases people would *actually* expect from this.


I think we already can see that we will need to make this layer work
with a large group of users.

I hope that it will be common across remote(jtag/serial) targets,
including QEmu/KVM stubs, and with the work from the guys at SUSE we
should be able to cater this for kdump cores too.

I think the importance is to make sure everything works well together.




> 
> Cheers
> 
> Phil
> 
> 

Regards

Kieran

Re: [RFC] Target Layer Python Interface

[Ales Novak]

Hello,

On 2016-2-1 19:19, Kieran Bingham wrote:
> [...]
> The API, I would expect to match that of the Target API operations. I
> would expect a one-to-one mapping of (required) operation names to
> perform functions.
>
> For instance, to support thread listings, I would expect something along
> the lines of:
>
> .to_update_thread_list
> .to_pid_to_str
> .to_extra_thread_info
> .to_thread_alive
> .to_fetch_registers
> ...

FTR I've slightly tweaked your gdb.Target to process "to_xfer_partial", 
the respective commit is:

https://github.com/alesax/gdb-kdump/commit/efba160691273ef3c1547112554584088b5dba75

(and the respective branch is "gdb-target")

Then the target code which is accessing virtual (!) memory of the kernel 
dump on the disk (using libkdumpfile library) is as small as:

===
from gdb import Target
from _kdumpfile import kdumpfile

class MyTarget(Target):
     def __init__(self, fil):
         self.kdump = kdumpfile(fil)
         self.kdump.symbol_func = \
             lambda nam: long(gdb.lookup_minimal_symbol(nam).value())
         self.kdump.vtop_init()
         super(MyTarget, self).__init__()
     def to_xfer_partial(self, obj, annex, readbuf, writebuf, offset, ln):
         if obj == self.TARGET_OBJECT_MEMORY:
             r = self.kdump.read (self.kdump.KDUMP_KVADDR, offset, ln)
             readbuf[:] = r
         return ln

MyTarget(file("/tmp/vmcore"))
===

which is really nice, I'd say. Now it would be interesting


> And having seen Jeff's work today, we could utilise Jeff's py-regcache
> object quite effectively
>
> [...]
>
> Yes, I suspect some of the functionality to implement will be very
> repeatable throughout each of the operation call implementations.
>
>
> However, the more I look into it - the more I see each function is
> likely to need very specific bindings, as it is not simple passing from
> c function to c function.

Yes, the mentioned to_xfer_partial being a good example (of not simple 
passing).

> Perhaps we can factor out commonality as we go - and try to keep as DRY
> as possible, but I suspect it will be an iterative implementation process.
>
>
>> I can't comment without more details though. My initial reaction
>> though is yeah, this sounds useful and exciting.
>
> Perfect :)

Yes, this definitely is worth pursuing.

-- 
Ales Novak
SUSE L3 Team

Re: [RFC] Target Layer Python Interface

[Kieran Bingham]

On 04/02/16 22:16, Ales Novak wrote:
> Hello,
> 
> On 2016-2-1 19:19, Kieran Bingham wrote:
>> [...]
>> The API, I would expect to match that of the Target API operations. I
>> would expect a one-to-one mapping of (required) operation names to
>> perform functions.
>>
>> For instance, to support thread listings, I would expect something along
>> the lines of:
>>
>> .to_update_thread_list
>> .to_pid_to_str
>> .to_extra_thread_info
>> .to_thread_alive
>> .to_fetch_registers
>> ...
> 
> FTR I've slightly tweaked your gdb.Target to process "to_xfer_partial",
> the respective commit is:
> 
> https://github.com/alesax/gdb-kdump/commit/efba160691273ef3c1547112554584088b5dba75
> 
> 
> (and the respective branch is "gdb-target")
> 
> Then the target code which is accessing virtual (!) memory of the kernel
> dump on the disk (using libkdumpfile library) is as small as:
> 
> ===
> from gdb import Target
> from _kdumpfile import kdumpfile
> 
> class MyTarget(Target):
>     def __init__(self, fil):
>         self.kdump = kdumpfile(fil)
>         self.kdump.symbol_func = \
>             lambda nam: long(gdb.lookup_minimal_symbol(nam).value())
>         self.kdump.vtop_init()
>         super(MyTarget, self).__init__()
>     def to_xfer_partial(self, obj, annex, readbuf, writebuf, offset, ln):
>         if obj == self.TARGET_OBJECT_MEMORY:
>             r = self.kdump.read (self.kdump.KDUMP_KVADDR, offset, ln)
>             readbuf[:] = r
>         return ln
> 
> MyTarget(file("/tmp/vmcore"))
> ===
> 
> which is really nice, I'd say. Now it would be interesting

Were you going to say something else here? ... looks like it got chopped!


Pulling the const's through is pretty neat, and that does make an
effective way to implement the read overrides to a file!


By the way, I'd started to add thread support - but I'm on holiday now.

Adding an add_thread(pid,lwp,tid) method to the inferior allows

    def to_update_thread_list(self):
        gdb.write("LX.to_update_thread_list\n")
        inferior = gdb.selected_inferior()
        threads = inferior.threads()
        for task in tasks.task_lists():
            # Build ptid_t ... class object better here still
            ptid = (inferior.pid, 0, task['pid'])  # (pid, lwp, tid)
            if ptid not in threads:
                gdb.write("- New Task [{} {}]\n"
                          .format(task['pid'], task['comm'].string()))
                inferior.add_thread(ptid)


The 'if ptid not in tasks' is not working yet. That was going to be next
on my list.

I think the comparison function is in the wrong place, it should be
implementing __contains__ instead of compare I think.

Then it's just a matter of wiring up Jeff's Regcache ...

If you're interested: My latest patches are at:

http://git.linaro.org/people/kieran.bingham/binutils-gdb.git lkd-python

And the Kernel Awareness object is at
http://git.linaro.org/people/kieran.bingham/linux.git lkd-python

Feel free to have a go at wiring up while I'm away if it's useful to you.


> 
> 
>> And having seen Jeff's work today, we could utilise Jeff's py-regcache
>> object quite effectively
>>
>> [...]
>>
>> Yes, I suspect some of the functionality to implement will be very
>> repeatable throughout each of the operation call implementations.
>>
>>
>> However, the more I look into it - the more I see each function is
>> likely to need very specific bindings, as it is not simple passing from
>> c function to c function.
> 
> Yes, the mentioned to_xfer_partial being a good example (of not simple
> passing).

Indeed - but probably not too many hooks to implement to get thread
integration through python.

> 
>> Perhaps we can factor out commonality as we go - and try to keep as DRY
>> as possible, but I suspect it will be an iterative implementation
>> process.
>>
>>
>>> I can't comment without more details though. My initial reaction
>>> though is yeah, this sounds useful and exciting.
>>
>> Perfect :)
> 
> Yes, this definitely is worth pursuing.
> 

I'm glad you like the concept.
I think it can work well with the recent code Jeff has written.

Although I may be slightly diverted for a bit when I get back from
holiday - so if it can go somewhere for you guys ... do have a go until
I return. (And let me know how it goes!)

Regards

Kieran

Re: [RFC] Target Layer Python Interface

[Jeff Mahoney]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 2/5/16 11:36 AM, Kieran Bingham wrote:
> On 04/02/16 22:16, Ales Novak wrote:
>> Hello,
>> 
>> On 2016-2-1 19:19, Kieran Bingham wrote:
>>> [...] The API, I would expect to match that of the Target API
>>> operations. I would expect a one-to-one mapping of (required)
>>> operation names to perform functions.
>>> 
>>> For instance, to support thread listings, I would expect
>>> something along the lines of:
>>> 
>>> .to_update_thread_list .to_pid_to_str .to_extra_thread_info 
>>> .to_thread_alive .to_fetch_registers ...
>> 
>> FTR I've slightly tweaked your gdb.Target to process
>> "to_xfer_partial", the respective commit is:
>> 
>> https://github.com/alesax/gdb-kdump/commit/efba160691273ef3c154711255
4584088b5dba75
>>
>>
>>
>> 
(and the respective branch is "gdb-target")
>> 
>> Then the target code which is accessing virtual (!) memory of the
>> kernel dump on the disk (using libkdumpfile library) is as small
>> as:
>> 
>> === from gdb import Target from _kdumpfile import kdumpfile
>> 
>> class MyTarget(Target): def __init__(self, fil): self.kdump =
>> kdumpfile(fil) self.kdump.symbol_func = \ lambda nam:
>> long(gdb.lookup_minimal_symbol(nam).value()) 
>> self.kdump.vtop_init() super(MyTarget, self).__init__() def
>> to_xfer_partial(self, obj, annex, readbuf, writebuf, offset,
>> ln): if obj == self.TARGET_OBJECT_MEMORY: r = self.kdump.read
>> (self.kdump.KDUMP_KVADDR, offset, ln) readbuf[:] = r return ln
>> 
>> MyTarget(file("/tmp/vmcore")) ===
>> 
>> which is really nice, I'd say. Now it would be interesting
> 
> Were you going to say something else here? ... looks like it got
> chopped!
> 
> 
> Pulling the const's through is pretty neat, and that does make an 
> effective way to implement the read overrides to a file!
> 
> 
> By the way, I'd started to add thread support - but I'm on holiday
> now.
> 
> Adding an add_thread(pid,lwp,tid) method to the inferior allows

Ok, I have code doing that too.  There's a big messy commit at the top
of my repo last night that I'm going to refactor but I got most of
this working.

- -Jeff

> def to_update_thread_list(self): 
> gdb.write("LX.to_update_thread_list\n") inferior =
> gdb.selected_inferior() threads = inferior.threads() for task in
> tasks.task_lists(): # Build ptid_t ... class object better here
> still ptid = (inferior.pid, 0, task['pid'])  # (pid, lwp, tid) if
> ptid not in threads: gdb.write("- New Task [{} {}]\n" 
> .format(task['pid'], task['comm'].string())) 
> inferior.add_thread(ptid)
> 
> 
> The 'if ptid not in tasks' is not working yet. That was going to be
> next on my list.
> 
> I think the comparison function is in the wrong place, it should
> be implementing __contains__ instead of compare I think.
> 
> Then it's just a matter of wiring up Jeff's Regcache ...
> 
> If you're interested: My latest patches are at:
> 
> http://git.linaro.org/people/kieran.bingham/binutils-gdb.git
> lkd-python
> 
> And the Kernel Awareness object is at 
> http://git.linaro.org/people/kieran.bingham/linux.git lkd-python
> 
> Feel free to have a go at wiring up while I'm away if it's useful
> to you.
> 
> 
>> 
>> 
>>> And having seen Jeff's work today, we could utilise Jeff's
>>> py-regcache object quite effectively
>>> 
>>> [...]
>>> 
>>> Yes, I suspect some of the functionality to implement will be
>>> very repeatable throughout each of the operation call
>>> implementations.
>>> 
>>> 
>>> However, the more I look into it - the more I see each function
>>> is likely to need very specific bindings, as it is not simple
>>> passing from c function to c function.
>> 
>> Yes, the mentioned to_xfer_partial being a good example (of not
>> simple passing).
> 
> Indeed - but probably not too many hooks to implement to get
> thread integration through python.
> 
>> 
>>> Perhaps we can factor out commonality as we go - and try to
>>> keep as DRY as possible, but I suspect it will be an iterative
>>> implementation process.
>>> 
>>> 
>>>> I can't comment without more details though. My initial
>>>> reaction though is yeah, this sounds useful and exciting.
>>> 
>>> Perfect :)
>> 
>> Yes, this definitely is worth pursuing.
>> 
> 
> I'm glad you like the concept. I think it can work well with the
> recent code Jeff has written.
> 
> Although I may be slightly diverted for a bit when I get back from 
> holiday - so if it can go somewhere for you guys ... do have a go
> until I return. (And let me know how it goes!)
> 
> Regards
> 
> Kieran
> 


- -- 
Jeff Mahoney
SUSE Labs
-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.19 (Darwin)
Comment: GPGTools - http://gpgtools.org

iQIcBAEBAgAGBQJWtNAKAAoJEB57S2MheeWySjwP/RyE9xWolrCjFXo7f8yVFW8C
aF/AP/zYa7A56fc9PtkdpDjqRzCnhoHK/9iBi62yN+F+gBdHOE13EytRLsdlimwi
M+QNRqJf658oEho4s11G+WznCw5E8TLSBmx426nnOnfHDJ3umSo++az05WY3GX9e
PH+UdUJRfEBLz7i0+Y2ZAv9b5oYUpLGwzXJJVwEHKYE6x30BWUUA/0NDUnaSRJpo
+2b2xWLHUWGKis9po0H20zRyO5srAHPlwlNyA+p087GD1oRHttDExrSuYrHeikx1
6wfp7GVjFDl6D/iMa1f9Gf9QzoqI7zkwaGxX5C6ex7AmdNofRSVk/WaaZdzWuEgm
4xV9EDsy5BebU7rzDBE1dByW1Jo/3h8EQARS+RvuYA7xml8quqosKevH+iXDWFyZ
aJG4bhhkTpV1+50AHQluJh7N8GxoUTBQTZtlgpbpSw2+0B/NyYlImZiLdQDesmkR
TptueFkScOt7l5NZ3EkjglRaNURD1OyG+KP/t3Faf5EiyQrnAk/wkhtKX71R44Mk
/F8PnlJRYDbPM9+VE326gH9VmXCLDDxqj2UMcAG0dNEYWlCeaYBZB90gLcQuxAvL
ch28FAdI8qAJ85pMfXKEPrG9aEa55swTdLjmeIxcOsQMF8t0jfBigXfQ47TIhb/R
Lt58vNw8Yu25Eh+/GujZ
=UeOP
-----END PGP SIGNATURE-----

Re: [RFC] Target Layer Python Interface

[Kieran Bingham]

On 05/02/16 16:38, Jeff Mahoney wrote:
> On 2/5/16 11:36 AM, Kieran Bingham wrote:
>> On 04/02/16 22:16, Ales Novak wrote:
>>> Hello,
>>>
>>> On 2016-2-1 19:19, Kieran Bingham wrote:
>>>> [...] The API, I would expect to match that of the Target API
>>>> operations. I would expect a one-to-one mapping of (required)
>>>> operation names to perform functions.
>>>>
>>>> For instance, to support thread listings, I would expect
>>>> something along the lines of:
>>>>
>>>> .to_update_thread_list .to_pid_to_str .to_extra_thread_info 
>>>> .to_thread_alive .to_fetch_registers ...
>>>
>>> FTR I've slightly tweaked your gdb.Target to process
>>> "to_xfer_partial", the respective commit is:
>>>
>>> https://github.com/alesax/gdb-kdump/commit/efba160691273ef3c154711255
> 4584088b5dba75
>>>
>>>
>>>
>>>
> (and the respective branch is "gdb-target")
>>>
>>> Then the target code which is accessing virtual (!) memory of the
>>> kernel dump on the disk (using libkdumpfile library) is as small
>>> as:
>>>
>>> === from gdb import Target from _kdumpfile import kdumpfile
>>>
>>> class MyTarget(Target): def __init__(self, fil): self.kdump =
>>> kdumpfile(fil) self.kdump.symbol_func = \ lambda nam:
>>> long(gdb.lookup_minimal_symbol(nam).value()) 
>>> self.kdump.vtop_init() super(MyTarget, self).__init__() def
>>> to_xfer_partial(self, obj, annex, readbuf, writebuf, offset,
>>> ln): if obj == self.TARGET_OBJECT_MEMORY: r = self.kdump.read
>>> (self.kdump.KDUMP_KVADDR, offset, ln) readbuf[:] = r return ln
>>>
>>> MyTarget(file("/tmp/vmcore")) ===
>>>
>>> which is really nice, I'd say. Now it would be interesting
> 
>> Were you going to say something else here? ... looks like it got
>> chopped!
> 
> 
>> Pulling the const's through is pretty neat, and that does make an 
>> effective way to implement the read overrides to a file!
> 
> 
>> By the way, I'd started to add thread support - but I'm on holiday
>> now.
> 
>> Adding an add_thread(pid,lwp,tid) method to the inferior allows
> 
> Ok, I have code doing that too.  There's a big messy commit at the top
> of my repo last night that I'm going to refactor but I got most of
> this working.

Fantastic, you've probably spent more time on it than me so I'll bet
your version works before I'm back from holiday!

I'll go have a quick look now. :D

I felt a bit like we need a bit better encapsulation on the ptid_t
object. Might need a small object to wrap it up for passing around
instead of just throwing Tuples everywhere.

--
Kieran

> 
> -Jeff
> 
>> def to_update_thread_list(self): 
>> gdb.write("LX.to_update_thread_list\n") inferior =
>> gdb.selected_inferior() threads = inferior.threads() for task in
>> tasks.task_lists(): # Build ptid_t ... class object better here
>> still ptid = (inferior.pid, 0, task['pid'])  # (pid, lwp, tid) if
>> ptid not in threads: gdb.write("- New Task [{} {}]\n" 
>> .format(task['pid'], task['comm'].string())) 
>> inferior.add_thread(ptid)
> 
> 
>> The 'if ptid not in tasks' is not working yet. That was going to be
>> next on my list.
> 
>> I think the comparison function is in the wrong place, it should
>> be implementing __contains__ instead of compare I think.
> 
>> Then it's just a matter of wiring up Jeff's Regcache ...
> 
>> If you're interested: My latest patches are at:
> 
>> http://git.linaro.org/people/kieran.bingham/binutils-gdb.git
>> lkd-python
> 
>> And the Kernel Awareness object is at 
>> http://git.linaro.org/people/kieran.bingham/linux.git lkd-python
> 
>> Feel free to have a go at wiring up while I'm away if it's useful
>> to you.
> 
> 
>>>
>>>
>>>> And having seen Jeff's work today, we could utilise Jeff's
>>>> py-regcache object quite effectively
>>>>
>>>> [...]
>>>>
>>>> Yes, I suspect some of the functionality to implement will be
>>>> very repeatable throughout each of the operation call
>>>> implementations.
>>>>
>>>>
>>>> However, the more I look into it - the more I see each function
>>>> is likely to need very specific bindings, as it is not simple
>>>> passing from c function to c function.
>>>
>>> Yes, the mentioned to_xfer_partial being a good example (of not
>>> simple passing).
> 
>> Indeed - but probably not too many hooks to implement to get
>> thread integration through python.
> 
>>>
>>>> Perhaps we can factor out commonality as we go - and try to
>>>> keep as DRY as possible, but I suspect it will be an iterative
>>>> implementation process.
>>>>
>>>>
>>>>> I can't comment without more details though. My initial
>>>>> reaction though is yeah, this sounds useful and exciting.
>>>>
>>>> Perfect :)
>>>
>>> Yes, this definitely is worth pursuing.
>>>
> 
>> I'm glad you like the concept. I think it can work well with the
>> recent code Jeff has written.
> 
>> Although I may be slightly diverted for a bit when I get back from 
>> holiday - so if it can go somewhere for you guys ... do have a go
>> until I return. (And let me know how it goes!)
> 
>> Regards
> 
>> Kieran
> 
> 
> 
> 

Re: [RFC] Target Layer Python Interface

[Kieran Bingham]

On 05/02/16 16:47, Kieran Bingham wrote:
> 
> 
> On 05/02/16 16:38, Jeff Mahoney wrote:
>> On 2/5/16 11:36 AM, Kieran Bingham wrote:
>>> On 04/02/16 22:16, Ales Novak wrote:
>>>> Hello,
>>>>
>>>> On 2016-2-1 19:19, Kieran Bingham wrote:
>>>>> [...] The API, I would expect to match that of the Target API
>>>>> operations. I would expect a one-to-one mapping of (required)
>>>>> operation names to perform functions.
>>>>>
>>>>> For instance, to support thread listings, I would expect
>>>>> something along the lines of:
>>>>>
>>>>> .to_update_thread_list .to_pid_to_str .to_extra_thread_info 
>>>>> .to_thread_alive .to_fetch_registers ...
>>>>
>>>> FTR I've slightly tweaked your gdb.Target to process
>>>> "to_xfer_partial", the respective commit is:
>>>>
>>>> https://github.com/alesax/gdb-kdump/commit/efba160691273ef3c154711255
>> 4584088b5dba75
>>>>
>>>>
>>>>
>>>>
>> (and the respective branch is "gdb-target")
>>>>
>>>> Then the target code which is accessing virtual (!) memory of the
>>>> kernel dump on the disk (using libkdumpfile library) is as small
>>>> as:
>>>>
>>>> === from gdb import Target from _kdumpfile import kdumpfile
>>>>
>>>> class MyTarget(Target): def __init__(self, fil): self.kdump =
>>>> kdumpfile(fil) self.kdump.symbol_func = \ lambda nam:
>>>> long(gdb.lookup_minimal_symbol(nam).value()) 
>>>> self.kdump.vtop_init() super(MyTarget, self).__init__() def
>>>> to_xfer_partial(self, obj, annex, readbuf, writebuf, offset,
>>>> ln): if obj == self.TARGET_OBJECT_MEMORY: r = self.kdump.read
>>>> (self.kdump.KDUMP_KVADDR, offset, ln) readbuf[:] = r return ln
>>>>
>>>> MyTarget(file("/tmp/vmcore")) ===
>>>>
>>>> which is really nice, I'd say. Now it would be interesting
>>
>>> Were you going to say something else here? ... looks like it got
>>> chopped!
>>
>>
>>> Pulling the const's through is pretty neat, and that does make an 
>>> effective way to implement the read overrides to a file!
>>
>>
>>> By the way, I'd started to add thread support - but I'm on holiday
>>> now.
>>
>>> Adding an add_thread(pid,lwp,tid) method to the inferior allows
>>
>> Ok, I have code doing that too.  There's a big messy commit at the top
>> of my repo last night that I'm going to refactor but I got most of
>> this working.
> 
> Fantastic, you've probably spent more time on it than me so I'll bet
> your version works before I'm back from holiday!
> 
> I'll go have a quick look now. :D
> 
> I felt a bit like we need a bit better encapsulation on the ptid_t
> object. Might need a small object to wrap it up for passing around
> instead of just throwing Tuples everywhere.



+    if (!result)
+      error("Callback failed.\n");
+
+    ret = python_string_to_host_string (result);
     do_cleanups(cleanup);
+    return ret;
 }

Watch out for the memory leak here ...
It's one of the annoyances of this layer. No way to free the pointers
when we pass them back up to the caller:

In my implementation of pid_to_str, I have a scratch buffer to copy the
string into and then free the object.

It's probably worth putting one scratch buffer in the layer to share
across all functions that need it. We should be can_async_p = 0 anyway I
think.

https://git.linaro.org/people/kieran.bingham/binutils-gdb.git/commitdiff/ecd824755584f5c3f2083f7d9ee4b5e25eb357f3

Ha - I say that - and then I see I didn't free the char array from
python_string_to_host_string (result); in my version! That'll have to be
a fixup later :)

--
Kieran

> 
> --
> Kieran
> 
>>
>> -Jeff
>>
>>> def to_update_thread_list(self): 
>>> gdb.write("LX.to_update_thread_list\n") inferior =
>>> gdb.selected_inferior() threads = inferior.threads() for task in
>>> tasks.task_lists(): # Build ptid_t ... class object better here
>>> still ptid = (inferior.pid, 0, task['pid'])  # (pid, lwp, tid) if
>>> ptid not in threads: gdb.write("- New Task [{} {}]\n" 
>>> .format(task['pid'], task['comm'].string())) 
>>> inferior.add_thread(ptid)
>>
>>
>>> The 'if ptid not in tasks' is not working yet. That was going to be
>>> next on my list.
>>
>>> I think the comparison function is in the wrong place, it should
>>> be implementing __contains__ instead of compare I think.
>>
>>> Then it's just a matter of wiring up Jeff's Regcache ...
>>
>>> If you're interested: My latest patches are at:
>>
>>> http://git.linaro.org/people/kieran.bingham/binutils-gdb.git
>>> lkd-python
>>
>>> And the Kernel Awareness object is at 
>>> http://git.linaro.org/people/kieran.bingham/linux.git lkd-python
>>
>>> Feel free to have a go at wiring up while I'm away if it's useful
>>> to you.
>>
>>
>>>>
>>>>
>>>>> And having seen Jeff's work today, we could utilise Jeff's
>>>>> py-regcache object quite effectively
>>>>>
>>>>> [...]
>>>>>
>>>>> Yes, I suspect some of the functionality to implement will be
>>>>> very repeatable throughout each of the operation call
>>>>> implementations.
>>>>>
>>>>>
>>>>> However, the more I look into it - the more I see each function
>>>>> is likely to need very specific bindings, as it is not simple
>>>>> passing from c function to c function.
>>>>
>>>> Yes, the mentioned to_xfer_partial being a good example (of not
>>>> simple passing).
>>
>>> Indeed - but probably not too many hooks to implement to get
>>> thread integration through python.
>>
>>>>
>>>>> Perhaps we can factor out commonality as we go - and try to
>>>>> keep as DRY as possible, but I suspect it will be an iterative
>>>>> implementation process.
>>>>>
>>>>>
>>>>>> I can't comment without more details though. My initial
>>>>>> reaction though is yeah, this sounds useful and exciting.
>>>>>
>>>>> Perfect :)
>>>>
>>>> Yes, this definitely is worth pursuing.
>>>>
>>
>>> I'm glad you like the concept. I think it can work well with the
>>> recent code Jeff has written.
>>
>>> Although I may be slightly diverted for a bit when I get back from 
>>> holiday - so if it can go somewhere for you guys ... do have a go
>>> until I return. (And let me know how it goes!)
>>
>>> Regards
>>
>>> Kieran
>>
>>
>>
>>