repo to work on python scripting support

repo to work on python scripting support

[Thiago Jung Bauermann]

Hi folks,

Since Tromey announced this in another thread, it might have been
missed by some people: Tromey set up a git repo to host our work of
integrating python into GDB. To play with it:

$ git clone git://git.gitorious.org/gdb-python/mainline.git
$ cd mainline
$ git remote add origin git@gitorious.org:gdb-python/mainline.git
$ git checkout --track -b python-revisited origin/python-revisited

The current code has some rudimentary support for exposing value
structs as python objects, some initial frame querying from python,
evaluating python expressions on the GDB command line (or in a script),
evaluating GDB commands from python, and some varobj trickery which is
explained here:

http://sourceware.org/ml/gdb/2008-02/msg00140.html

There's also some reference to all this on the wiki:

http://sourceware.org/gdb/wiki/OngoingWork

And quoting Tromey:

"If you are interested in helping out, and you have an FSF assignment
on file, I can set you up with write
access."
-- 
[]'s
Thiago Jung Bauermann
Software Engineer
IBM Linux Technology Center

Re: repo to work on python scripting support

[Tom Tromey]

I did some hacking on the Python integration over the last couple of
days, enough to warrant a status update.

You can evaluate python, either as a block, as a one-liner, or as an
expression.  I cleaned up the code so that it works properly in other
block structures, like commands or loops.  The expression evaluation
code makes an attempt to convert python values to gdb values as
appropriate (strings, ints, and floating-point types).  Some contrived
examples...

Block form:

    python
    print 23
    end

One liner:

    python print 23

Expression:

    break foo if $(23 > 5)


There is minimal hook-up to frames.  You can get the name of the
function at any frame depth and the number of frames.  This area needs
an OO rewrite.

I haven't tried out the value support at all yet (sorry Thiago and
Volodya).


Breakpoints are now visible as objects and some of their properties
are inspectable.  Some properties can be set from Python as well.
Enough works that I was able to write a simple "save my breakpoints"
script -- it is appended.

Breakpoints need some more plumbing.  I want to make it so you can
write: new gdb.Breakpoint('location') and have it do the right thing.
(This looks pretty easy.)


You can set Python hooks on a few events but this is not documented
and not tested yet.  There's also a Python command to pass any string
to the gdb CLI, and another one to get the value of any "set/show"
variable.


As you can see it is pretty rough still, but I think it is coming
along rather well.  It is not very hard to add new functionality.
There are a few things I don't understand about hooking this code to
gdb, I'll ask in a separate thread.

Comments, constructive criticism, etc, welcome.

Tom

# Set up a python function to do the work.
python
from __future__ import with_statement
import os
def gdb_save_breakpoints(filename):
    with open(filename, 'w') as file:
        for bp in gdb.breakpoints():
    	    if not bp:
	       continue
	    if not bp.is_valid():
	       continue
	    print >> file, "break", bp.get_location(),
    	    cond = bp.get_condition()
	    if cond:
	       print >> file, " if", cond,
	    print >> file
	    commands = bp.get_commands()
	    if commands:
	       print >> file, "commands"
	       print >> file, commands,
	       print >> file, "end"
	    if not bp.is_enabled():
	       print >> file, "disable $bpnum"
end

# And now the gdb user command.
# Try:  save_breakpoints /tmp/out
define save_breakpoints
dont-repeat
# Silly quoting games...
python gdb_save_breakpoints('$arg0')
end

Re: repo to work on python scripting support

[Thiago Jung Bauermann]

Hi everybody,

My turn at the status update. :-)

I did the OO rewrite for the frames support, it has a few useful methods
already (equals, is_inner_than, get_name, get_pc, get_prev, get_next,
find_sal). You can also call gdb.frames(), which will return a tuple
containing all frames in the stack (or throw an exception if there's a
problem with the unwinding).

I also exposed a little bit of struct symtab and struct symtab_and_line
(which is returned by Frame.find_sal), just enough to be able to
implement a mostly complete backtrace command in Python (it's only
missing ability to print function arguments, which I plan to work on
later).

Regarding the values work that Tromey mentioned, I didn't add
functionality to the code that Volodya implemented, just converted it to
OO form and worked on lifetime of struct value vs lifetime of Python
Value object (I believe I is correct now).

Here's the Python implementation of an "rbt" command which prints the
backtrace in reverse (or right :-P) order:

python
import sys

def gdb_rbacktrace ():
  frames = gdb.frames ()
  for i in reversed (range (len (frames))):
    sal = frames[i].find_sal ()
    pc = frames[i].get_pc ()

    sys.stdout.write ("#%-2d" % i)
    if pc != sal.get_pc () or not sal.symtab:
      sys.stdout.write (" 0x%08x in" % pc)
    sys.stdout.write (" " + frames[i].get_name ())
    if sal.symtab and sal.symtab.get_filename ():
      sys.stdout.write (" at " + sal.symtab.get_filename ())
      sys.stdout.write (":" + str (sal.get_line ()))
    if not frames[i].get_name () or (not sal.symtab or not
sal.symtab.get_filename ()):
       lib = gdb.solib_address (pc)
       if lib:
         sys.stdout.write (" from " + lib)
    sys.stdout.write ("\n")
end

define rbt
dont-repeat
python gdb_rbacktrace ()
end

Here's sample output:

(gdb) rbt
#2  0x080483bb in main at ../../src/examples/funcs.c:15
#1  0x08048391 in f1 at ../../src/examples/funcs.c:10
#0  f2 at ../../src/examples/funcs.c:5
(gdb)
-- 
[]'s
Thiago Jung Bauermann
Software Engineer
IBM Linux Technology Center

Re: repo to work on python scripting support

[Tom Tromey]

>>>>> "Thiago" == Thiago Jung Bauermann <bauerman@br.ibm.com> writes:

Thiago> My turn at the status update. :-)

Thanks for sending this.  Funnily enough I was just thinking about
asking you to write up an example :)


I've been thinking... maybe we do want $(...) to allow things other
than Python commands.  Or, more precisely, maybe we want to let Python
code register a function by name for use in $(...).  The Python
function would get the uninterpreted string argument for processing.

My reason is that after playing a bit, I'm not over-fond of the
verboseness and extra quoting required by the Python bits.  It is more
gdb-ish, I think, not to do this.

Any opinions?

The canonical example is a function to match the calling function at a
breakpoint.  gdb-ish:

    break inner if $(caller-is outer)

Pythonic:

    break inner if $(gdb.caller_is ('outer'))


BTW other suggestions for fun demos are not only accepted, they are
actually helpful.  I tend to drive my gdb-python hacking by picking
target functionality.  My last hack was making it so I could override
the (barely useful IMO) 'edit' command to instead emit an annotation
that Emacs will pick up :-)

Tom

Re: repo to work on python scripting support

[Daniel Jacobowitz]

On Mon, Mar 24, 2008 at 11:32:16PM -0600, Tom Tromey wrote:
> I've been thinking... maybe we do want $(...) to allow things other
> than Python commands.  Or, more precisely, maybe we want to let Python
> code register a function by name for use in $(...).  The Python
> function would get the uninterpreted string argument for processing.

I was thinking about this unrelatedly last night and I agree, we
should register the arguments independently.  I'm not so sure about
quoting decisions for the arguments yet, though.  What I don't want to
do is end up with the GDB CLI again, where everyone who adds a command
picks a different way to quote the arguments.

Maybe we can handle this the same way that I have been speculating
about handling CLI commands: register a "prototype" of the function?
For example, you could register:
  $(strcmp expression, "foo")
as "EXPRESSION, EXPRESSION".  Other functions might not have the comma
if the preceeding argument was not an EXPRESSION.

-- 
Daniel Jacobowitz
CodeSourcery

Re: repo to work on python scripting support

[Jim Blandy]

It's worth putting some investment into the way arguments get handled,
to make writing commands as easy as possible.  For example, in Emacs
Lisp, any function you write in Lisp can be turned into a command just
by adding an "interactive spec" to it, which says how to prompt the
user for the function's arguments.  It's still just an ordinary
function that you can call from Lisp code, but if you invoke it as an
editing command, Emacs knows how to prompt for arguments for it.

The interactive spec is either a bit of arbitrary lisp code that
returns a list of arguments to pass, or just a string with magic
character codes that are shorthand for the most common cases.

For example, if I have a function that writes the currently selected
region to a file, it might take three arguments:

   (start end filename)

and its interactive spec might be:

  (interactive "rFFile to write to: ")

where the 'r' passes the start and end of the currently selected
region as 'start' and 'end', and the 'F' means "filename", to be
prompted for with the following text.  With 'F', emacs will do
filename completion; with 'f', it'll do completion, and also insist
that the file exists.  A function that was going to read the file
would use 'f'.

The point here is that it becomes dirt-easy to write most of the
interactive functions you'd care to.  You get the interactivity out of
the way and focus on the meat of your command.

I don't know the most Pythonish way of doing things, but it seems to
me that Python could register not functions, but objects, to be
callable with $().  The object could have an 'interactive' member to
specify how to call it (say, "e" might mean, evaluate an expression
and pass its value to me, and "s" might mean, just hand it to me as a
string), and then some other method to actually do the call.

class Strcmp (GdbCommand):
  interactive = 'ee'
  def doit (expr1, expr2):
    // compare expr1 and expr2 as strings

strcmp = Strcmp ();

or something.  Surely someone here is more fluent in Python than I am.

Re: repo to work on python scripting support

[Daniel Jacobowitz]

On Tue, Mar 25, 2008 at 11:18:12AM -0700, Jim Blandy wrote:
> I don't know the most Pythonish way of doing things, but it seems to
> me that Python could register not functions, but objects, to be
> callable with $().  The object could have an 'interactive' member to
> specify how to call it (say, "e" might mean, evaluate an expression
> and pass its value to me, and "s" might mean, just hand it to me as a
> string), and then some other method to actually do the call.

Hmm, I've worked with some packages that did similar things using
mandatory docstrings.

def strcmp (expr1, expr2):
  """strcmp: EXPR, EXPR

  Compare expr1 and expr2 as strings."""
  doit

Dunno if that's wise.

-- 
Daniel Jacobowitz
CodeSourcery

Re: repo to work on python scripting support

[Jim Blandy]

On Tue, Mar 25, 2008 at 11:30 AM, Daniel Jacobowitz >  Hmm, I've
worked with some packages that did similar things using
>  mandatory docstrings.
>
>  def strcmp (expr1, expr2):
>   """strcmp: EXPR, EXPR
>
>   Compare expr1 and expr2 as strings."""
>   doit
>
>  Dunno if that's wise.

Eww.  Emacs Lisp distinguishes docstrings from interactive specs.

In the example you give here, there's no nice way to break out and
allow arbitrary Python for computing the arguments; the 'arbitrary
lisp' escape hatch is necessary in Emacs from time to time.  With the
approach I suggested, it'd be pretty easy to have 'interactive' be
either a string or a method, and to have the GDB invocation code do
some reflection to decide how to interpret things.

Re: repo to work on python scripting support

[Tom Tromey]

>>>>> "Daniel" == Daniel Jacobowitz <drow@false.org> writes:

Jim> On Tue, Mar 25, 2008 at 11:18:12AM -0700, Jim Blandy wrote:
Jim> I don't know the most Pythonish way of doing things, but it seems to
Jim> me that Python could register not functions, but objects, to be
Jim> callable with $().

Daniel> Hmm, I've worked with some packages that did similar things using
Daniel> mandatory docstrings.

We could use decorators.

    http://wiki.python.org/moin/PythonDecorators

Well, assuming they exist.  I'm not really that up on the current
state of Python.


For this particular thing I would be just as happy with some explicit
constructor thing:

class Strcmp(GdbFunction):
  def __init__(self):
    Strcmp.__init__(self, 'strcmp', 'ee')

  def invoke(expr1, expr2):
    lalala


You could also use a real data structure and named constants:

    Strcmp.__init__(self, 'strcmp', (gdb.EXPRESSION, gdb.EXPRESSION))

.. or what have you.

Aside from the interactive spec, the current gdb-python support for
command objects already works this way.  A command subclass has to
pass in the user-visible command name at construction time.

FWIW here's the code I wrote to override "edit" and turn it into an
annotation-based command:

class Edit(gdb.Command):
    def __init__(self):
        gdb.Command.__init__(self, "edit", gdb.COMMAND_NONE)

    def invoke(self, arg, fromtty):
        if gdb.show('annotate') > -1:
            loc = gdb.decode_line(arg)
            if loc:
                loc = loc[0]
                if loc.symtab.to_fullname():
                    print "\n\x1a\x1asource %s:%d:0:beg:0x0" \
                        % (loc.symtab.to_fullname(), loc.get_line())

# Install it.
Edit()



Just to be clear, I don't really want to consider anything in this
area as fixed.  Advice and input about how everything should fit
together is really helpful.

Tom

Re: repo to work on python scripting support

[Jim Blandy]

Python decorators seem to be in the latest python, at least:
http://docs.python.org/ref/function.html

The discussion Tom linked to was back in 2004.

Decorators would be perfect for this, I think:

@Gdb.interactive (Gdb.Expr, Gdb.Expr)
def strcmp (str1, str2):
    // compare strings

Easy squeezy.

Re: repo to work on python scripting support

[Paul Koning]

>>>>> "Jim" == Jim Blandy <jimb@red-bean.com> writes:

 Jim> Python decorators seem to be in the latest python, at least:
 Jim> http://docs.python.org/ref/function.html

 Jim> The discussion Tom linked to was back in 2004.

 Jim> Decorators would be perfect for this, I think:

 Jim> @Gdb.interactive (Gdb.Expr, Gdb.Expr) def strcmp (str1, str2):
 Jim> // compare strings

 Jim> Easy squeezy.

Decorators showed up in Python 2.4, which was released in November
2004.  And yes, that syntax or something like it can do what you
described. 

I have the impression that decorators are a fairly obscure part of the
language.  That impression may be wrong...

I'm still not convinced that adding Elisp-style "interactive"
decoration to Python is a good idea.  (If it's a good idea, maybe it
should be a Python PEP, added to the base language, as opposed to a
GDB-specific extension.)  Yes, if you have programmed in Elisp (I
have) then it will be familiar, and it might feel like a handy
convenience.  If you're not an Elisp programmer, the likely reaction
is "I don't recognize this style of Python".

Clearly, Python embedded in/with GDB needs to have hooks into GDB.
The obvious style that will be familiar to programmers in any language
is by way of funtion calls -- for example "gdb.eval_expr(string)".

A less obvious way but part of existing Python -- albeit a part
probably not all that familiar to the community -- is decorators.
Those avoid the need to do gdb.xxx function calls at function entry
(though you still need them, for use at spots other than function
entry).

The least desirable approach is to have a new non-Python way of
describing argument processing -- things like the doc string parsing
or other similar things that have been talked about.  I think those
are a bad idea because you end up with something that isn't Python, is
GDBython... :-)

I still think just gdb.xxx() functions are sufficient and intuitive.

But if you want to do the function entry stuff, then decorators look
like the right way to do it.  They can be implemented with
semi-straightforward Python code and the gdb.xxx() functions that you
need anyway (a nice exercise for the reader).

	paul

Re: repo to work on python scripting support

[Paul Koning]

>>>>> "Jim" == Jim Blandy <jimb@red-bean.com> writes:

 Jim> It's worth putting some investment into the way arguments get
 Jim> handled, to make writing commands as easy as possible. 

 Jim> I don't know the most Pythonish way of doing things, but it
 Jim> seems to me that Python could register not functions, but
 Jim> objects, to be callable with $(). 

You can make objects callable, but I don't know if that's the answer.
Calling a class object is how you construct instances, by the way...

An obvious Pythonic way of doing this is with default arguments, which
allows a call to omit some of the arguments defined for the function.

For example:

    def foo (arg1, arg2=None):
	if arg2 is None:
	   arg2 = raw_input ("please supply arg2: ")

   paul

Re: repo to work on python scripting support

[Tom Tromey]

>>>>> "Paul" == Paul Koning <Paul_Koning@Dell.com> writes:

Paul> An obvious Pythonic way of doing this is with default arguments, which
Paul> allows a call to omit some of the arguments defined for the function.
Paul> For example:
Paul>     def foo (arg1, arg2=None):
Paul> 	if arg2 is None:
Paul> 	   arg2 = raw_input ("please supply arg2: ")

I think default arguments are part of the solution, but not the whole
solution.

The basic problem is that we have a syntax for embedding a python call
in an expression that looks like $(stuff).

Now, internally to gdb, "stuff" is just a string.  But, most of the
time, the implementation of this function, whatever it is, won't want
just a string -- it will want an expression, or a file name, or
something.

So, what Jim and Daniel want, I think, is a declarative way for the
Python code (which implements the given function) to tell gdb's core
how to parse this string.

I think the hope here is that we avoid the mess of gdb commands, which
really do get just a plain string and then proceed to be inconsistent
with each other and just generally hard to modify.


It would be interesting to see if we could enumerate the core
possibilities for argument types.

Tom

Re: repo to work on python scripting support

[Paul Koning]

>>>>> "Tom" == Tom Tromey <tromey@redhat.com> writes:

 Tom> The basic problem is that we have a syntax for embedding a
 Tom> python call in an expression that looks like $(stuff).

 Tom> Now, internally to gdb, "stuff" is just a string.  But, most of
 Tom> the time, the implementation of this function, whatever it is,
 Tom> won't want just a string -- it will want an expression, or a
 Tom> file name, or something.

 Tom> So, what Jim and Daniel want, I think, is a declarative way for
 Tom> the Python code (which implements the given function) to tell
 Tom> gdb's core how to parse this string.

Something akin to the way that C extension modules inside Python tell
the Python execution machinery what data type it wants might serve.

The notion of asking for a particular type is a bit foreign to Python;
arguments have no fixed type.

Another possibility is to pass strings but then have standard
conversion routines (things callable by Python and supplied by gdb).
For example parse_and_eval_address.  And the target functions.  And so
on.

def walklist (head):
    addr = parse_and_eval_address (head)
    while addr:
	  print "list item at", addr
	  addr = target_read_memory (addr, 4)

   paul

Re: repo to work on python scripting support

[Tom Tromey]

>>>>> "Paul" == Paul Koning <Paul_Koning@Dell.com> writes:

Paul> Something akin to the way that C extension modules inside Python tell
Paul> the Python execution machinery what data type it wants might serve.

Yeah.

Paul> The notion of asking for a particular type is a bit foreign to Python;
Paul> arguments have no fixed type.

"Type" here was a confusing choice of word.  It is really just about
having a declarative way to decide how to parse the argument string.

Paul> Another possibility is to pass strings but then have standard
Paul> conversion routines (things callable by Python and supplied by gdb).
Paul> For example parse_and_eval_address.  And the target functions.  And so
Paul> on.

One nice property of the declarative approach is that the core code
can automatically handle completion as well as parsing.

One idea is to let a Python class define a special method which is
used to parse the string.  If there is no such method, we would look
for declarative properties.

Tom

Re: repo to work on python scripting support

[Jim Blandy]

On Tue, Mar 25, 2008 at 12:31 PM, Paul Koning <Paul_Koning@dell.com> wrote:
>  The notion of asking for a particular type is a bit foreign to Python;
>  arguments have no fixed type.

It's not really about type.  For example, Emacs has about a dozen
interactive spec code letters that compute string arguments for the
function.  Some are filenames, some are buffer names, some are bits of
text...  It's about how to prompt for the arguments.  Or, in the case
of $() functions in GDB, how to parse the arguments.

>  Another possibility is to pass strings but then have standard
>  conversion routines (things callable by Python and supplied by gdb).
>  For example parse_and_eval_address.  And the target functions.  And so
>  on.
>
>  def walklist (head):
>     addr = parse_and_eval_address (head)
>     while addr:
>           print "list item at", addr
>           addr = target_read_memory (addr, 4)

Using "None" this way means that the function's arguments are (say)
"address, or None if I should prompt interactively."

The nice thing about interactive specs is that the function itself has
a single contract to follow: each of its arguments always means the
same thing, whether the function is invoked from code or interactively
by the user.

You may also get interactive prompting by accident, if you pass the
wrong arguments to the function.

Re: repo to work on python scripting support

[Paul Koning]

>>>>> "Jim" == Jim Blandy <jimb@red-bean.com> writes:

 Jim> On Tue, Mar 25, 2008 at 12:31 PM, Paul Koning
 Jim> <Paul_Koning@dell.com> wrote:
 >> The notion of asking for a particular type is a bit foreign to
 >> Python; arguments have no fixed type.

 Jim> It's not really about type.  For example, Emacs has about a
 Jim> dozen interactive spec code letters that compute string
 Jim> arguments for the function.  Some are filenames, some are buffer
 Jim> names, some are bits of text...  It's about how to prompt for
 Jim> the arguments.  Or, in the case of $() functions in GDB, how to
 Jim> parse the arguments.

 >> Another possibility is to pass strings but then have standard
 >> conversion routines (things callable by Python and supplied by
 >> gdb).  For example parse_and_eval_address.  And the target
 >> functions.  And so on.
 >> 
 >> def walklist (head): addr = parse_and_eval_address (head) while
 >> addr: print "list item at", addr addr = target_read_memory (addr,
 >> 4)

 Jim> Using "None" this way means that the function's arguments are
 Jim> (say) "address, or None if I should prompt interactively."

 Jim> The nice thing about interactive specs is that the function
 Jim> itself has a single contract to follow: each of its arguments
 Jim> always means the same thing, whether the function is invoked
 Jim> from code or interactively by the user.

 Jim> You may also get interactive prompting by accident, if you pass
 Jim> the wrong arguments to the function.

It seems to me that there's an attempt here to treat Python as if it
were Elisp.  But it isn't, and it isn't similar.

The goal of supporting Python scripting should be to provide an
environment that is familiar to Python programmers.  That means
passing arguments in the way Python normally does.  That is not the
way Elisp does it.

If you're going to merge Elisp and Python, you're creating a new
programming language.  There may be good reasons for inventing a new
scripting language, but if that's the plan, it shouldn't be descriped
as supporting Python for scripting.

In Python, the caller picks the type of the arguments (implicitly, by
picking the variables and expressions in the call).  The called
function then either examines what it received and converts as needed,
or assumes that what it got was what it wanted and uses exceptions to
convert as needed, etc.

If you add decorations of some sort, or magical methods in classes, to
do Elisp-style argument mapping, a Python programmer is going to look
at that and say "what language is this?"

   paul

Re: repo to work on python scripting support

[Daniel Jacobowitz]

On Wed, Mar 26, 2008 at 10:15:22AM -0400, Paul Koning wrote:
> The goal of supporting Python scripting should be to provide an
> environment that is familiar to Python programmers.

No, I don't think so.  The goal of supporting Python scripting is
to make it easy to implement useful and usable GDB commands in a
way accessible to Python programmers.

These are functions which will be directly shown to the user.  We
need _consistency_ in how their arguments are passed; experience has
shown that inconsistency is a serious problem.  I don't think any
sort of polymorphism is going to be necessary or desirable in
practice.

Don't confuse this with typing.  It's just going to say EXPRESSION
and if you want to handle strings and ints, that's fine.  The "caller"
is the human user typing some characters.

Can you give me an example of where it would be more useful not to
do it this way?  Regardless of the details of how we pre-specify
arguments.

-- 
Daniel Jacobowitz
CodeSourcery

Re: repo to work on python scripting support

[Tom Tromey]

Paul> The goal of supporting Python scripting should be to provide an
Paul> environment that is familiar to Python programmers.  That means
Paul> passing arguments in the way Python normally does.

I think we're misunderstanding each other somehow.  This particular
sub-discussion is just about how to parse arguments passed from the
gdb CLI into Python.

The Python code people write and plug into gdb will just be plain old
Python.  Fetching data from gdb will look Pythonic.  Longer term, I'd
like to be able to fully control gdb from Python.

Tom

Re: repo to work on python scripting support

[Paul Koning]

>>>>> "Tom" == Tom Tromey <tromey@redhat.com> writes:

 Paul> The goal of supporting Python scripting should be to provide an
 Paul> environment that is familiar to Python programmers.  That means
 Paul> passing arguments in the way Python normally does.

 Tom> I think we're misunderstanding each other somehow.  This
 Tom> particular sub-discussion is just about how to parse arguments
 Tom> passed from the gdb CLI into Python.

I interpreted it that way.  My point is that I'd say "a call is a
call" -- this case is no different from any other Python call.  And
that fits the Python model, where interactions with the interpreter
(at the >>> prompt) look just like program code.  In particular, there
is no difference in the way call arguments are handled.

 Tom> The Python code people write and plug into gdb will just be
 Tom> plain old Python.  Fetching data from gdb will look Pythonic.
 Tom> Longer term, I'd like to be able to fully control gdb from
 Tom> Python.

Sounds good.

On the earlier note:

 Daniel> print $(py "this is a python \"program\"")

 Tom> I was thinking some functions could ask for an uninterpreted
 Tom> string.

 Tom> print $(py do something in python)

I like that.  Quoting in strings would be a major hassle, especially
if the Python code contains raw strings or block (triple quote)
strings.  Parentheses should work fine given that those have to
match.  The only difficulty is that you have to know at least some of
Python's syntax to find the matching close paren.  (Basically, the
scanner would have to know about comments and about the various
flavors of Python strings.)

	paul

Re: repo to work on python scripting support

[Jim Blandy]

On Wed, Mar 26, 2008 at 7:15 AM, Paul Koning <Paul_Koning@dell.com> wrote:
>  It seems to me that there's an attempt here to treat Python as if it
>  were Elisp.  But it isn't, and it isn't similar.

You're right, Python and Emacs Lisp are not the same, and we shouldn't
thoughtlessly adapt ideas from Emacs Lisp to GDB Python.

However, Emacs does have a similar situation to the one we have
here, in that Emacs commands are simultaneously:
- ordinary Emacs Lisp functions callable from Lisp, and
- interactive commands that can be bound to keys or invoked by name,
  and when this happens need to prompt or otherwise gather their
  arguments appropriately.

This is directly analogous to what we're doing with GDB.  We want user
commands and functions to be simultaneously:
- Python functions (or at least objects) that can be used in the
  ordinary fashion in Python code, and
- either commands that can be typed at the (gdb) prompt, or functions
  that can be used in GDB expressions using the $(FUNC ARGS) syntax.

So I think it's a sound analogy that rests on the novel uses to which
we're putting functions, not on anything specific to Emacs Lisp.

> In Python, the caller picks the type of the arguments (implicitly, by
> picking the variables and expressions in the call).  The called
> function then either examines what it received and converts as needed,
> or assumes that what it got was what it wanted and uses exceptions to
> convert as needed, etc.

I'm proposing we continue to use Python in exactly this way.

> If you add decorations of some sort, or magical methods in classes, to
> do Elisp-style argument mapping, a Python programmer is going to look
> at that and say "what language is this?"

Have you written code in Emacs Lisp?  It sounds like you've gotten the
impression that what's going on is much more magic and involved than
it really is.

It's possible to write Emacs Lisp functions that work like the Python
code you posted that checks arguments against None and prompts for
them if needed.  They would work fine, and interactive users wouldn't
be able to distinguish them from ordinary Emacs commands.

But Emacs Lisp coders don't write their functions this way:

- They would lose error checking when calling their functions from
  Lisp: if they forget an argument to the function, the function will
  prompt interactively instead of raising an error.

- They would lose the ability to use optional arguments in the way one
  usually does --- when there's an obvious or useful default, or for
  backwards compatibility.  Instead of using default values, the
  function would have to prompt for their values interactively.

- Interactive specs are often terser, and because they're an idiom
  they're a little easier to read.

Re: repo to work on python scripting support

[Doug Evans]

On Mon, Mar 24, 2008 at 10:32 PM, Tom Tromey <tromey@redhat.com> wrote:
>  I've been thinking... maybe we do want $(...) to allow things other
>  than Python commands.  Or, more precisely, maybe we want to let Python
>  code register a function by name for use in $(...).  The Python
>  function would get the uninterpreted string argument for processing.

OOC, Would there be an intent to not disallow dynamically loaded C/C++
instead of Python for $(foo)? [Emphasis on OOC.]

>  My reason is that after playing a bit, I'm not over-fond of the
>  verboseness and extra quoting required by the Python bits.  It is more
>  gdb-ish, I think, not to do this.
>
>  Any opinions?
>
>  The canonical example is a function to match the calling function at a
>  breakpoint.  gdb-ish:
>
>     break inner if $(caller-is outer)
>
>  Pythonic:
>
>     break inner if $(gdb.caller_is ('outer'))

Parsing two separate languages in one line is problematic too (I'm not
sure how far the Pythonic version was intended to be interpreted).

Re: repo to work on python scripting support

[Daniel Jacobowitz]

On Wed, Mar 26, 2008 at 10:28:53AM -0700, Doug Evans wrote:
> On Mon, Mar 24, 2008 at 10:32 PM, Tom Tromey <tromey@redhat.com> wrote:
> >  I've been thinking... maybe we do want $(...) to allow things other
> >  than Python commands.  Or, more precisely, maybe we want to let Python
> >  code register a function by name for use in $(...).  The Python
> >  function would get the uninterpreted string argument for processing.
> 
> OOC, Would there be an intent to not disallow dynamically loaded C/C++
> instead of Python for $(foo)? [Emphasis on OOC.]

My plan on this, to date, has been to insist on use of the Python
interface for marshalling and unmarshalling.  This keeps the interface
explicit.  Of course, Python can load C modules itself and there'd be
nothing to stop that if you needed to do e.g. some more intensive
computation in the C side; and with a little forethought we could
probably keep likely bulky operations (like memory reading) as
zero-copy.

> Parsing two separate languages in one line is problematic too (I'm not
> sure how far the Pythonic version was intended to be interpreted).

Yes, I think we'll not want to allow raw Python in most GDB
expressions; if it's useful in a convenience expression the
outer layer of quoting should probably belong to GDB.

print $(py "this is a python \"program\"")

-- 
Daniel Jacobowitz
CodeSourcery

Re: repo to work on python scripting support

[Tom Tromey]

>>>>> "Daniel" == Daniel Jacobowitz <drow@false.org> writes:

>> OOC, Would there be an intent to not disallow dynamically loaded C/C++
>> instead of Python for $(foo)? [Emphasis on OOC.]

Daniel> My plan on this, to date, has been to insist on use of the Python
Daniel> interface for marshalling and unmarshalling.  This keeps the interface
Daniel> explicit.

Interesting -- since I just said the opposite :-)

Are you concerned about the parsing problem for C-defined functions?
We could push the parsing-style declarations down to the C level to
solve that.

I did want to say to Doug, but forgot -- is this something you're
actually interested in?  AFAIK you can't load gdb commands written in
C or C++, so it would be a bit odd to add that feature just for these
functions.

Not that this matters hugely.  And dynamically loading commands might
also be useful.

Daniel> print $(py "this is a python \"program\"")

I was thinking some functions could ask for an uninterpreted string.

  print $(py do something in python)

I'm not wedded to it.

Tom

Re: repo to work on python scripting support

[Daniel Jacobowitz]

On Wed, Mar 26, 2008 at 11:16:37AM -0600, Tom Tromey wrote:
> Are you concerned about the parsing problem for C-defined functions?
> We could push the parsing-style declarations down to the C level to
> solve that.

No, I'm concerned about what the C-defined functions would include
or call to interpret the details.  I don't want them including any
of GDB's normal headers.

> Daniel> print $(py "this is a python \"program\"")
> 
> I was thinking some functions could ask for an uninterpreted string.
> 
>   print $(py do something in python)

But what if there's quotes in there?  Or parentheses, or both?

-- 
Daniel Jacobowitz
CodeSourcery

Re: repo to work on python scripting support

[Tom Tromey]

>>>>> "Daniel" == Daniel Jacobowitz <drow@false.org> writes:

Daniel> No, I'm concerned about what the C-defined functions would include
Daniel> or call to interpret the details.  I don't want them including any
Daniel> of GDB's normal headers.

Gotcha, that makes sense to me.

Long-term compatibility in the Python API is something we haven't
discussed.  I'm really not sure what to do about this -- we really are
exposing a decent amount of gdb internals to the world this way.

>> print $(py do something in python)

Daniel> But what if there's quotes in there?  Or parentheses, or both?

Yeah.  They have to match and if gdb does any processing (I didn't
look into this), you have to know how to trick it.  Talk about
unsatisfactory...  OTOH double-quoting the python code is also a bit
yucky.  But, I suppose this will be a relatively unusual thing to want
to do.

I'll try to implement this soon.

So far my list of declarative argument "kinds" is:

* expression
* linespec (or whatever the thing that "break" takes is called)
* filename (do we need it?)

I was thinking that the declaration could be a tuple using named
constants; string elements in the tuple would be looked for
explicitly, like: (EXPRESSION "," EXPRESSION) would parse an
expression, a comma (ignoring whitespace around it), and another
expression.

I guess we need a syntax for optional arguments as well, perhaps just
a named 'OPTIONAL' constant in the tuple.

Maybe we could do the same kind of parsing for commands.

Tom

Re: repo to work on python scripting support

[Daniel Jacobowitz]

On Wed, Mar 26, 2008 at 11:44:39AM -0600, Tom Tromey wrote:
> Maybe we could do the same kind of parsing for commands.

Yep.

The only part of this I'm concerned about is separators.  Basically,
an expression can be followed only by a comma or the end of arguments.
If you want just a number or address, though, you can use spaces.

Maybe we should always use commas to simplify.

-- 
Daniel Jacobowitz
CodeSourcery

Re: repo to work on python scripting support

[Thiago Jung Bauermann]

On Wed, 2008-03-26 at 11:44 -0600, Tom Tromey wrote:
> Long-term compatibility in the Python API is something we haven't
> discussed.  I'm really not sure what to do about this -- we really are
> exposing a decent amount of gdb internals to the world this way.

We are, but at least up until now we've been exposing things which are
there in GDB for a long time now (struct breakpoint, struct frame_info,
struct value), and attributes which are fairly intrinsic to them, like
frame.get_pc(). I expect this stuff to be stable between releases.

I also expect that most (if not all) of what is exposed is along these
lines, that is, providing a "view" of the general GDB internal
components, as oposed to exposing exoteric internal GDB code which is
likely to change often.

Since the C <-> Python glue code provide some decoupling, it's possible
to accomodate for some changes in the GDB internals.

All in all, my hope is that compatibility of python scripts accross GDB
versions won't be a frequent issue. We could provide a method which
returns the GDB versions so that scripts can accomodate for changes, if
the need arises?
-- 
[]'s
Thiago Jung Bauermann
Software Engineer
IBM Linux Technology Center

Re: repo to work on python scripting support

[Jim Blandy]

What kind of functions for parsing Python does the Python API export?
Is there anything that would, say, parse an expression as far as it
can, and let us know where the expression ends?  If so, we could use
that for $(py <python code here>), and things like embedded quotes and
parens would just work.

Re: repo to work on python scripting support

[Paul Koning]

>>>>> "Jim" == Jim Blandy <jimb@red-bean.com> writes:

 Jim> What kind of functions for parsing Python does the Python API
 Jim> export?  Is there anything that would, say, parse an expression
 Jim> as far as it can, and let us know where the expression ends?  If
 Jim> so, we could use that for $(py <python code here>), and things
 Jim> like embedded quotes and parens would just work.

I don't see anything to do this in http://docs.python.org/ext/ext.html
There are APIs to pass in the Python text, as a string, or as a file
-- but it appears that the text is expected to be valid Python code,
not Python code optionally followed by something else.

Lexically speaking Python is very simple; looking for the closing
paren in the proposed syntax wouldn't be a big deal.

      paul

Re: repo to work on python scripting support

[Tom Tromey]

>>>>> "Doug" == Doug Evans <dje@google.com> writes:

>> I've been thinking... maybe we do want $(...) to allow things other
>> than Python commands.  Or, more precisely, maybe we want to let Python
>> code register a function by name for use in $(...).  The Python
>> function would get the uninterpreted string argument for processing.

Doug> OOC, Would there be an intent to not disallow dynamically loaded C/C++
Doug> instead of Python for $(foo)? [Emphasis on OOC.]

I think once we've decided to do some extra parsing of $(...), instead
of just passing it verbatim to Python for interpretation, we can do
whatever we like.  I wouldn't be opposed to being able to register new
functions implemented in C.

>> break inner if $(gdb.caller_is ('outer'))

Doug> Parsing two separate languages in one line is problematic too (I'm not
Doug> sure how far the Pythonic version was intended to be interpreted).

In the current implementation the `...' in $(...) is just passed to
Python for evaluation.  So, it has to follow Python quoting rules, for
instance.

This is simple to implement :), but I think yields a not-very-nice
user experience.

Tom