Python API plans

Python API plans

[Tom Tromey]

Phil asked me to write up a list of the various Python projects that I
think we should pursue in gdb.  So, here it is; I welcome feedback,
additions, etc.  Don't expect many details here, it is really just the
result of making notes whenever I missed something over the last couple
of years.

* Rebase and polish Oguz's inferior event work from the 2009 SoC.  I
  think Oguz has started looking into this again.  This functionality
  blocks a number of interesting Python uses; I think the event registry
  work is needed for some other (smaller) projects too.

* Fix all open bugs.

* Polish some of the existing commands and functions from the
  archer-tromey-python branch.  Sub-tasks:

  * Finish the command-loading code.  Time eager loading to see how
    expensive it is; if too expensive, write some lazy-loading
    wrappers.  (We definitely do not want the "require" command from the
    branch.)

  * (See below for info on frame wrapper, new-backtrace, etc.)

* Write Python wrappers for all MI commands, see:

    http://sourceware.org/bugzilla/show_bug.cgi?id=11688

  This is an easy (-ish) way to provide a simple API to most of the
  relevant parts of GDB.

* Generic command improvements.

  * Sometimes it would be nice to write a command with the same name as
    an existing command, but then delegate to the existing command (say,
    via a super call).

  * Likewise it would be nice if you could easily uninstall an
    overriding command.

  These considerations are why "new-backtrace" isn't just called
  "backtrace" -- if it has a bug the user is just stuck with it.  Maybe
  just a "rename" facility is enough (or maybe just the existing
  alias.py plus deprecation -- worth looking at).

* Breakpoint improvements:

  * Make it possible to create an internal breakpoint from Python.

  * Add a bit to breakpoints to make the breakpoint "maybe non-stopping"
    -- in the sense that when the breakpoint is hit, it will call an
    attached Python function, and if the function returns True, the
    breakpoint will not stop.  This would be different from "commands"
    including "cont" in that it would not interfere with step or next.
    (You can fake this right now by doing work via a convenience
    function attached to the breakpoint condition -- but that is gross
    and user-visible.)

  * Use these two new features to write a command to create a "return"
    breakpoint.  This is a breakpoint that fires when a function
    returns.  (I think you can do this by putting a breakpoint at the
    function's entry, then going up and looking at the outer frame's
    PC, or something along those lines.  Bonus points for interacting
    properly with longjmp and throw...)

* Minimal systemtap integration.  In particular I've been thinking it
  would be nice if gdb could read and use systemtap probe points.  This
  doesn't have to be written in Python, but it seems like a nice
  experiment.  Some sub-tasks:

  * Hooking into linespecs so users can write "probe:name" to refer to a
    systemtap probe.  This would also mean dealing with completion,
    maybe some ugly linespec refactoring.

  * Let Python code access the contents of a section in an objfile.

* Make a new Expression object that represents a parsed expression.
  Add a new gdb.parse method to create these, something like:

    gdb.parse (EXPR, [BLOCK], [STOP_AT_COMMA]) -> gdb.Expression

  This functionality is handy for writing certain kinds of commands.

* Frame and "new-backtrace" cleanup

  * Clean up the frame wrapper and iteration code to solve problems
    reported on the Archer list; look at the various use cases posted
    there to make sure they can all be handled.  This may mean adding
    more methods to Frame so we can avoid having a FrameWrapper at all.

  * Add a feature to new-backtrace to print the objfile name for a
    frame.  (This might be a request in bugzilla somewhere.)

  * Rewrite the "upto" family of commands.  They are ok but there were
    some ideas on the Archer list about how to improve them.

  * Give frame wrappers, or their replacement, some notion of being
    synthetic and having "original" sub-frames.  The idea here is to
    provide multiple views of the stack, as a start toward debugging
    scripting languages.  Provide versions of up/down/etc that work on
    the filtered view or the raw view, enhance new-backtrace to display
    sub-frames indented or something like that.  Figure out how to
    present this via MI.

* True scripting language debugging.  This is a big goal that requires a
  lot of infrastructure, some already listed above.

  * Hook into linespecs so that "break x" can call into the scripting
    support code.

  * Hook into "watch" so the python code can do the right thing.

  * Hook into expression parsing.  Maybe this could be done by letting
    people write new languages in Python.

  * Frame-based operations (up/down/etc) -- see above.

  * Hook into step/next/finish as needed.

* New commands and functions I've wanted:

  * "log-expression".  Like a combination between "break" and "printf",
    would trace expressions at some point in the code, without
    interfering with "next".  (Needs at least the expression parsing
    stuff and a breakpoint enhancement.)

  * Finish "info mutex".  The one posted to the list is a little dumb;
    it could be made more robust by using the systemtap probes in the
    mutex functions.  With Oguz's work we would also have a way to look
    for deadlocks.

  * "declare $var : type" - make $var a reference to
    *(type*)malloc(sizeof(type)).  I'm not sure this one is super
    useful, but it might make scripting gdb a little easier sometimes.

  * PR 8320, recursive version of "directory" command.

  * New function "$memoize".  It would memoize strings in the inferior,
    so that they aren't constantly re-allocated.  Useful when using a
    string in a breakpoint condition.  This would need Oguz's work since
    the cache would have to be invalidated when the inferior is
    restarted.

Tom

Re: Python API plans

[André Pönitz]

On Tuesday 24 August 2010 03:06:40 ext Tom Tromey wrote:
> Phil asked me to write up a list of the various Python projects that I
> think we should pursue in gdb.  So, here it is;

Puh, that's quite a bit.

> I welcome feedback,
> additions, etc.  Don't expect many details here, it is really just the
> result of making notes whenever I missed something over the last couple
> of years.
> 
> * Rebase and polish Oguz's inferior event work from the 2009 SoC.  I
>   think Oguz has started looking into this again.  This functionality
>   blocks a number of interesting Python uses; I think the event registry
>   work is needed for some other (smaller) projects too.

I think inferior interaction (run/stop stuff) and breakpoint access are the 
big "missing" areas in Python support. From a frontend's point of view
the missing "run control" is actually not that bad as it is sufficiently 
well supported by MI (in contrast to breakpoints), but from a CLI/Python
user's point of view the priorities are probably inverted.

> * Fix all open bugs.

I am not sure whether e.g.

   http://sourceware.org/bugzilla/show_bug.cgi?id=10790

really deserves a "Pythonic fix" as long as there are real problems
around that don't have obvious (if any) workarounds.
 
> * Polish some of the existing commands and functions from the
>   archer-tromey-python branch.  Sub-tasks:
> 
>   * Finish the command-loading code.  Time eager loading to see how
>     expensive it is; if too expensive, write some lazy-loading
>     wrappers.  (We definitely do not want the "require" command from the
>     branch.)
> 
>   * (See below for info on frame wrapper, new-backtrace, etc.)
> 
> * Write Python wrappers for all MI commands, see:
> 
>     http://sourceware.org/bugzilla/show_bug.cgi?id=11688
> 
>   This is an easy (-ish) way to provide a simple API to most of the
>   relevant parts of GDB.

Similar here. The effective gain seems limited, especially since quite
a bit of the MI commands are not optimal themselves. As a really awful
example: "-data-read-memory" produces a lot of unneeded "information"
yet is so slow that a "dump binary memory"/read-from-file combination
easily beats it by a factor of two or three. That behaviour should not 
be wrapped in Python.

I could imagine handling that item on a case-by-case base and spending
an extra thought or two on each individual API might make more sense.

> * Generic command improvements.
> 
>   * Sometimes it would be nice to write a command with the same name as
>     an existing command, but then delegate to the existing command (say,
>     via a super call).
> 
>   * Likewise it would be nice if you could easily uninstall an
>     overriding command.
> 
>   These considerations are why "new-backtrace" isn't just called
>   "backtrace" -- if it has a bug the user is just stuck with it.  Maybe
>   just a "rename" facility is enough (or maybe just the existing
>   alias.py plus deprecation -- worth looking at).

I have no opinion on these.

> * Breakpoint improvements:
> 
>   * Make it possible to create an internal breakpoint from Python.
> 
>   * Add a bit to breakpoints to make the breakpoint "maybe non-stopping"
>     -- in the sense that when the breakpoint is hit, it will call an
>     attached Python function, and if the function returns True, the
>     breakpoint will not stop.  This would be different from "commands"
>     including "cont" in that it would not interfere with step or next.
>     (You can fake this right now by doing work via a convenience
>     function attached to the breakpoint condition -- but that is gross
>     and user-visible.)

I haven't looked at Python breakpoint handling for a while, MI is
certainly missing feedback about multiple breakpoints, and gdb as
a whole seems to be missing some user visible feedback when
pending breakpoints.

>   * Use these two new features to write a command to create a "return"
>     breakpoint.  This is a breakpoint that fires when a function
>     returns.  (I think you can do this by putting a breakpoint at the
>     function's entry, then going up and looking at the outer frame's
>     PC, or something along those lines.  Bonus points for interacting
>     properly with longjmp and throw...)

Maybe the issue discussed in http://cygwin.ru/ml/gdb/2010-03/msg00116.html
could be resolved in the way "everybody" expects, too? ;-)

> [snip] 
>   * Let Python code access the contents of a section in an objfile.

That would be nice indeed.
 
> * Make a new Expression object that represents a parsed expression.
>   Add a new gdb.parse method to create these, something like:
> 
>     gdb.parse (EXPR, [BLOCK], [STOP_AT_COMMA]) -> gdb.Expression
> 
>   This functionality is handy for writing certain kinds of commands.

What's the use case? Keeping "anything" on the inferior side?

> [snip]
>   * "declare $var : type" - make $var a reference to
>     *(type*)malloc(sizeof(type)).  I'm not sure this one is super
>     useful, but it might make scripting gdb a little easier sometimes.

I have a similar construct five times in my code (creating an initialized
gdb.Value, but that 'declare' is the first step) I think having it handled
in gdb would be nice, especially if it relieves me from figuring out
when to add 'class ' to the type, and when to quote it etc to 'survive'
gdb command line parsing.

Another open issue in the area is to provide a way to keep (structure)
return values of inferior calls alive in the inferior. Right now the only way 
I am aware of to chain or nest inferior calls is to poke data back manually.

From a quick glance,

  http://sourceware.org/bugzilla/show_bug.cgi?id=11914
  http://sourceware.org/bugzilla/show_bug.cgi?id=10344

seem to be related.

>   * PR 8320, recursive version of "directory" command.

That's not solvable using "set substitute-path"?

But anyway, it would be nice if gdb was able to tell the user that it did
not find a file, or that it had to guess and is not sure the guess is right,
and ideally even provide some means to query for user advice in that
situation.

Looks like you have a lot of work ahead.

Andre'

Re: Python API plans

[Tom Tromey]

>>>>> "André" == André Pönitz <andre.poenitz@nokia.com> writes:

André> I think inferior interaction (run/stop stuff) and breakpoint
André> access are the big "missing" areas in Python support. From a
André> frontend's point of view the missing "run control" is actually
André> not that bad as it is sufficiently well supported by MI (in
André> contrast to breakpoints), but from a CLI/Python user's point of
André> view the priorities are probably inverted.

Yeah, I think so.  I am not as much an MI expert as I would like to
be... feedback like this is very valuable.

Tom> * Write Python wrappers for all MI commands, see:
Tom> http://sourceware.org/bugzilla/show_bug.cgi?id=11688

André> Similar here. The effective gain seems limited, especially since quite
André> a bit of the MI commands are not optimal themselves. As a really awful
André> example: "-data-read-memory" produces a lot of unneeded "information"
André> yet is so slow that a "dump binary memory"/read-from-file combination
André> easily beats it by a factor of two or three. That behaviour should not 
André> be wrapped in Python.

André> I could imagine handling that item on a case-by-case base and spending
André> an extra thought or two on each individual API might make more sense.

There are definitely MI features that, IMO, are not useful to Python.
Varobj comes to mind.

The idea behind this is to reuse all the existing ui-out machinery to
mostly automate Python wrappers to internals.  It isn't as nice as the
hand-written approach.

Given your comment I think a good first step would be to review the MI
docs to get an idea of what is missing from the Python API.  If we're
close enough to full coverage then we don't really need this.

Tom> * Make a new Expression object that represents a parsed expression.
Tom> Add a new gdb.parse method to create these, something like:
Tom> gdb.parse (EXPR, [BLOCK], [STOP_AT_COMMA]) -> gdb.Expression
Tom> This functionality is handy for writing certain kinds of commands.

André> What's the use case? Keeping "anything" on the inferior side?

It can be handy to separate parsing from evaluation.
For example if you wanted to write the "log a variable" command, you
could use this to parse the expression, then keep the parsed
representation around so that evaluating it at the breakpoint would be
more efficient.

This reminds me, another possible breakpoint-related cleanup is pushing
more things into breakpoint_ops and then exposing this to Python.
It would be nice for various breakpoint-like commands in Python to have
more control over how their mechanics are presented to the user.

Tom> * "declare $var : type" - make $var a reference to

André> I have a similar construct five times in my code (creating an
André> initialized gdb.Value, but that 'declare' is the first step) I
André> think having it handled in gdb would be nice, especially if it
André> relieves me from figuring out when to add 'class ' to the type,
André> and when to quote it etc to 'survive' gdb command line parsing.

Please file bugs for problems like this.  I think we've made big strides
in eliminating random quoting; we definitely want to know about
remaining bugs.

André> Another open issue in the area is to provide a way to keep
André> (structure) return values of inferior calls alive in the
André> inferior. Right now the only way I am aware of to chain or nest
André> inferior calls is to poke data back manually.

Yeah, we plan to fix this, it just isn't specifically a Python task.

Tom

Re: Python API plans

[Phil Muldoon]

On 24/08/10 02:06, Tom Tromey wrote:
> Phil asked me to write up a list of the various Python projects that I
> think we should pursue in gdb.  So, here it is; I welcome feedback,
> additions, etc.  Don't expect many details here, it is really just the
> result of making notes whenever I missed something over the last couple
> of years.
> 
> * Rebase and polish Oguz's inferior event work from the 2009 SoC.  I
>   think Oguz has started looking into this again.  This functionality
>   blocks a number of interesting Python uses; I think the event registry
>   work is needed for some other (smaller) projects too.

There are two areas I would like to see improved beyond the initial
patch.  I have no opinion whether these can be incrementally improved
after submission, or if we should hack on them first.

* Events should report rich data when triggered.  So when a
  breakpoint event occurs it should also have an object  payload
  describing the breakpoint event. (IE, a gdb.Breakpoint object that
  describes the breakpoint that was triggered).  Each event should have
  this object payload.

* Event classes should be able to be written in Python as well as
  C. Hopefully this would eliminate some boilerplate code and make it
  more accessible to the Python developer.  This probably means
  writing an API wrapper around the various observer-* routines that
  GDB already exports for various inferior events.  Also, the existing
  observer patterns may not offer full coverage to the amount of
  events we can respond to in Python, so we might need to add more.
  Or figure/adapt how MI reports on various inferior events.

> 
> * Fix all open bugs.
> 
> * Polish some of the existing commands and functions from the
>   archer-tromey-python branch.  Sub-tasks:
> 
>   * Finish the command-loading code.  Time eager loading to see how
>     expensive it is; if too expensive, write some lazy-loading
>     wrappers.  (We definitely do not want the "require" command from the
>     branch.)
> 
>   * (See below for info on frame wrapper, new-backtrace, etc.)


I agree about the eager loading.  It might amount to nothing at all.
If it does turn out to be expensive, what about having commands
register themselves against a global Python list? This would be
similar to how pretty-printers work.  We could defer loading until the
command is needed, but still (with some alteration to the completion
code) allow the user to see the command-list in 'help 'and make the
commands available for tab-completion?


> * Write Python wrappers for all MI commands, see:
> 
>     http://sourceware.org/bugzilla/show_bug.cgi?id=11688
> 
>   This is an easy (-ish) way to provide a simple API to most of the
>   relevant parts of GDB.

Andre picked up my thoughts here, so I won't embellish.

> 
> * Generic command improvements.
> 
>   * Sometimes it would be nice to write a command with the same name as
>     an existing command, but then delegate to the existing command (say,
>     via a super call).

Do you mean purely Python commands overriding other Python commands, or
in-built commands as well?
 
>   * Likewise it would be nice if you could easily uninstall an
>     overriding command.


Or any command after loading.

 
> * Breakpoint improvements:
> 
>   * Make it possible to create an internal breakpoint from Python.
> 
>   * Add a bit to breakpoints to make the breakpoint "maybe non-stopping"
>     -- in the sense that when the breakpoint is hit, it will call an
>     attached Python function, and if the function returns True, the
>     breakpoint will not stop.  This would be different from "commands"
>     including "cont" in that it would not interfere with step or next.
>     (You can fake this right now by doing work via a convenience
>     function attached to the breakpoint condition -- but that is gross
>     and user-visible.)


Well you can already make a Python breakpoint conditional, so you
could just predicate the breakpoint stopping the inferior by setting the
appropriate condition..  I'm not sure if you can run arbitrary
Python code in a condition though (i.e., .condition("python if
somePythonFunction() == True").  It would be nice to do it more
formally anyway (as you note, I think we are violently agreeing ;).  


>   * Use these two new features to write a command to create a "return"
>     breakpoint.  This is a breakpoint that fires when a function
>     returns.  (I think you can do this by putting a breakpoint at the
>     function's entry, then going up and looking at the outer frame's
>     PC, or something along those lines.  Bonus points for interacting
>     properly with longjmp and throw...)

Nice!


> * Minimal systemtap integration.  In particular I've been thinking it
>   would be nice if gdb could read and use systemtap probe points.  This
>   doesn't have to be written in Python, but it seems like a nice
>   experiment.  Some sub-tasks:
> 
>   * Hooking into linespecs so users can write "probe:name" to refer to a
>     systemtap probe.  This would also mean dealing with completion,
>     maybe some ugly linespec refactoring.
> 
>   * Let Python code access the contents of a section in an objfile.


It would be nice to be able to set mnemonic breakpoints on System Tap probe
points as well.  And somehow export the probe-points via debug-info (or however) so
the user can tab-complete them (and make all of this available through
the Python API as well).


> * Make a new Expression object that represents a parsed expression.
>   Add a new gdb.parse method to create these, something like:
> 
>     gdb.parse (EXPR, [BLOCK], [STOP_AT_COMMA]) -> gdb.Expression
> 
>   This functionality is handy for writing certain kinds of commands.
> 
> * Frame and "new-backtrace" cleanup
> 
>   * Clean up the frame wrapper and iteration code to solve problems
>     reported on the Archer list; look at the various use cases posted
>     there to make sure they can all be handled.  This may mean adding
>     more methods to Frame so we can avoid having a FrameWrapper at all.
> 
>   * Add a feature to new-backtrace to print the objfile name for a
>     frame.  (This might be a request in bugzilla somewhere.)
> 
>   * Rewrite the "upto" family of commands.  They are ok but there were
>     some ideas on the Archer list about how to improve them.
> 
>   * Give frame wrappers, or their replacement, some notion of being
>     synthetic and having "original" sub-frames.  The idea here is to
>     provide multiple views of the stack, as a start toward debugging
>     scripting languages.  Provide versions of up/down/etc that work on
>     the filtered view or the raw view, enhance new-backtrace to display
>     sub-frames indented or something like that.  Figure out how to
>     present this via MI.
> 

These are all good suggestions, thanks.

Ideally I'd like the Python API to be fully aware of the state of the
inferior. On the flip-side I'd also to be able to control the
inferior in a more atomic way than issuing gdb.execute("step")
commands and guessing if the inferior did as expected.

There are few small others things. Some of these are in the wiki.

* Python breakpoints to support 'catch' breakpoints (especially the
  system-call variety).

* Make it so that tab completion knows about Python API
  completions. So (gdb) gdb.<TAB> would list all of the available
  Python bits associated with the GDB module.

* Python representation of a target.

* Allow Python scripts to control start-up of GDB so it allows the
  equivalent of gcore like commands.

I talked with Oguz about the event inferior code.  I'll write up the
conclusion to that email soon.

Cheers

Phil

Re: Python API plans

[Tom Tromey]

>>>>> "Phil" == Phil Muldoon <pmuldoon@redhat.com> writes:

Phil> There are two areas I would like to see improved beyond the initial
Phil> patch.  I have no opinion whether these can be incrementally improved
Phil> after submission, or if we should hack on them first.

These look good.  I think it is a good idea for the first commit of a
few feature to be as "API-complete" as we can make it: that is, we
should try to reduce the number of future changes to which a user must
adapt.  E.g., changing the return type of a method is hard to adapt to,
but adding a new method to a class is easy to adapt to.  In this
particular case, I think it is best to try to make the events as
expressive as possible first; but the details of how events are
implemented (Python or C) are not important provided they are
future-proof.

Phil> I agree about the eager loading.  It might amount to nothing at all.
Phil> If it does turn out to be expensive, what about having commands
Phil> register themselves against a global Python list? This would be
Phil> similar to how pretty-printers work.  We could defer loading until the
Phil> command is needed, but still (with some alteration to the completion
Phil> code) allow the user to see the command-list in 'help 'and make the
Phil> commands available for tab-completion?

I think there is a simpler way that we could try first:

    http://sourceware.org/ml/gdb-patches/2010-07/msg00201.html

This wouldn't require any changes to the gdb core.

Tom> * Sometimes it would be nice to write a command with the same name as
Tom> an existing command, but then delegate to the existing command (say,
Tom> via a super call).

Phil> Do you mean purely Python commands overriding other Python commands, or
Phil> in-built commands as well?
 
I meant built-in commands, but being generic would not be bad.

Phil> I'm not sure if you can run arbitrary Python code in a condition
Phil> though (i.e., .condition("python if somePythonFunction() ==
Phil> True").

Yeah, you can via a convenience function.  That is pretty ugly though.

Phil> Ideally I'd like the Python API to be fully aware of the state of the
Phil> inferior. On the flip-side I'd also to be able to control the
Phil> inferior in a more atomic way than issuing gdb.execute("step")
Phil> commands and guessing if the inferior did as expected.

The dreaded MI wrappers would achieve this ;-)

I'm mildly concerned about writing a bunch of wrappers for these
commands that work either synchronously or asynchronously, depending on
the non-stop setting.  It seems like it would be preferable to make them
always async, with some kind of completion hook.  But, I think that may
require a big gdb change, like always enabling non-stop or something
along those lines.

So, I think we should defer explicit support for step/next/etc until we
have a bit more clarity about long term goals here.  (I don't really
consider the MI wrapper project as "explicit" -- since that is more
like, do one change and get a bunch of stuff "for free", even if some of
what you get is useless or doesn't really make sense.)

Phil> * Python breakpoints to support 'catch' breakpoints (especially the
Phil>   system-call variety).

Sounds good.

Phil> * Make it so that tab completion knows about Python API
Phil>   completions. So (gdb) gdb.<TAB> would list all of the available
Phil>   Python bits associated with the GDB module.

Make this lower priority.  It might be nice to have but I think basic
functionality is more important.

Phil> * Python representation of a target.

I think the note on the wiki was about letting people write new targets
in Python.  This might be interesting but I think it is a very low
priority.

If there are parts of the target or gdbarch that are needed for generic
scripting stuff, then we should export those via some read-only
interface.

Phil> * Allow Python scripts to control start-up of GDB so it allows the
Phil>   equivalent of gcore like commands.

I pushed this kind of scripting/tracing idea for a while.  And, I do
still like it.  But I also think that, for us, systemtap fills the
non-interactive tracing role quite well.

So, I think we should focus more on gdb's interactive functionality.

I'd like to do some deeper systemtap integration with gdb, too, but that
isn't on this list since it isn't related to Python.

Tom