gdb + perl

gdb + perl

[Kip Macy]

In order to increase the amount of triage that could be fully automated
by developers I recently added support to my company's in-house GDB
tree for  writing macros using external scripting languages. Because
perl is the most popular scripting language in-house, that is the
language I've gone with for the initial implementation. To the base GDB
command set I've added three commands: runperl, sourceperl, and
resetperl. "runperl" runs a perl script from gdb using a non-persistent
interpreter. "sourceperl" runs a perl script from gdb that uses
a persistent interpreter. This allows one to register callbacks with
GDB to add to the command set, much like "define" for gdb macro
language. "resetperl" destroys and rebuilds the persistent interpreter.
All communication between perl and gdb (except for callback
registration) goes through the MI interface, so the changes to GDB are
quite minimal. I've written a perl parser for demarshalling the MI
output and a library on top of that that users can use to define
GDB scripts. I've also converted a number of in-house GDB macros
to perl to demonstrate to others how it is done.

I'm posting this to the list to ask if this is something that people
would like to see incorporated into the base GDB.



				-Kip

Re: gdb + perl

[Kip Macy]

I know it is bad form to follow-up on one's own messages, but since I
took the time to pull the changes into 6.0 I thought I'd make them
available to anyone who might be curious about it.

Documentation:
http://www.fsmware.com/gdb/gdb_ffi.html
Tarball:
http://www.fsmware.com/gdb/gdb-6.0-perl.tgz

perl support can be turned on by passing --enable-ffiperl to configure.


Even if it isn't deemed appropriate for mainline inclusion, I'm eager to
hear any feedback.

Thanks.

				-Kip


On Fri, 30 Jan 2004, Kip Macy wrote:

> In order to increase the amount of triage that could be fully automated
> by developers I recently added support to my company's in-house GDB
> tree for  writing macros using external scripting languages. Because
> perl is the most popular scripting language in-house, that is the
> language I've gone with for the initial implementation. To the base GDB
> command set I've added three commands: runperl, sourceperl, and
> resetperl. "runperl" runs a perl script from gdb using a non-persistent
> interpreter. "sourceperl" runs a perl script from gdb that uses
> a persistent interpreter. This allows one to register callbacks with
> GDB to add to the command set, much like "define" for gdb macro
> language. "resetperl" destroys and rebuilds the persistent interpreter.
> All communication between perl and gdb (except for callback
> registration) goes through the MI interface, so the changes to GDB are
> quite minimal. I've written a perl parser for demarshalling the MI
> output and a library on top of that that users can use to define
> GDB scripts. I've also converted a number of in-house GDB macros
> to perl to demonstrate to others how it is done.
>
> I'm posting this to the list to ask if this is something that people
> would like to see incorporated into the base GDB.
>
>
>
> 				-Kip
>

Re: gdb + perl

[Kevin Buettner]

On Fri, 30 Jan 2004 19:42:30 -0800 (PST)
Kip Macy <kmacy@fsmware.com> wrote:

> I know it is bad form to follow-up on one's own messages, but since I
> took the time to pull the changes into 6.0 I thought I'd make them
> available to anyone who might be curious about it.
> 
> Documentation:
> http://www.fsmware.com/gdb/gdb_ffi.html
> Tarball:
> http://www.fsmware.com/gdb/gdb-6.0-perl.tgz
> 
> perl support can be turned on by passing --enable-ffiperl to configure.
> 
> Even if it isn't deemed appropriate for mainline inclusion, I'm eager to
> hear any feedback.

Thanks for making your changes available.

I happen to like the idea of providing access to perl from GDB, but then
I also happen to like perl.

My hunch is that you'll have a hard time getting these changes into
mainline GDB.  The reason for this hunch is as follows:

  1) I don't think that perl would be the first choice for an
     extension language for very many of the GDB maintainers.  That
     said, if we were to take a vote, I have no idea what the first
     choice would be.

  2) GDB is a GNU project and as such would probably use guile as its
     primary extension language.  (Of course, there could be others.)

I haven't looked at your work at all yet.  Do you think it would be
possible to develop an extension language API that could be used by
perl as well as other extension languages?  That way, it'd be possible
to do extension language plugins, of which your work would be one. 
It'd also be possible (and easier) to maintain the code you've written
independent of mainline GDB.

Kevin

Re: gdb + perl

[Kip Macy]

> Thanks for making your changes available.

Thanks for replying.

>
> I happen to like the idea of providing access to perl from GDB, but then
> I also happen to like perl.

I happen to really *not like* perl. However, this is targeted at the
developers at my company who predominantly do like perl.  I don't know
what would be the ideal language. I've actually started ocaml support.
As far as mainstream scripting languages go, I would've chosen python.
I've structured the such that the only real work is writing a parser
for MI output in the target language. I export the MI functionality and
callback mechanism through a language independent interface.


>   2) GDB is a GNU project and as such would probably use guile as its
>      primary extension language.  (Of course, there could be others.)

I would add support for guile if doing so would get the FFI code
incorporated into mainline GDB.


>
> I haven't looked at your work at all yet.  Do you think it would be
> possible to develop an extension language API that could be used by
> perl as well as other extension languages?

That wouldn't be a giant leap.

> That way, it'd be possible
> to do extension language plugins, of which your work would be one.
> It'd also be possible (and easier) to maintain the code you've written
> independent of mainline GDB.

On a more general note I'd like to see loadable module support added to
GDB. This would allow people to maintain GDB extensions independently of
GDB. There are a number of things that I see adding to GDB that are only
interesting if you have a very large complex system and hence would
never be interesting for the majority of GDB users.


			-Kip

Re: gdb + perl

[Daniel Jacobowitz]

Throwing in my two cents.

On Tue, Feb 03, 2004 at 07:30:59PM -0800, Kip Macy wrote:
> I happen to really *not like* perl. However, this is targeted at the
> developers at my company who predominantly do like perl.  I don't know
> what would be the ideal language. I've actually started ocaml support.
> As far as mainstream scripting languages go, I would've chosen python.
> I've structured the such that the only real work is writing a parser
> for MI output in the target language. I export the MI functionality and
> callback mechanism through a language independent interface.

Great...

> I would add support for guile if doing so would get the FFI code
> incorporated into mainline GDB.

Greater....

> > I haven't looked at your work at all yet.  Do you think it would be
> > possible to develop an extension language API that could be used by
> > perl as well as other extension languages?
> 
> That wouldn't be a giant leap.

And really awesome.  Kip, have you / could you file the copyright
assignment paperwork for GDB?  That's a necessary first step to
including any code.

> > That way, it'd be possible
> > to do extension language plugins, of which your work would be one.
> > It'd also be possible (and easier) to maintain the code you've written
> > independent of mainline GDB.
> 
> On a more general note I'd like to see loadable module support added to
> GDB. This would allow people to maintain GDB extensions independently of
> GDB. There are a number of things that I see adding to GDB that are only
> interesting if you have a very large complex system and hence would
> never be interesting for the majority of GDB users.

We've talked about this before.  There are a couple of problems; the
biggest is that the only advantage of a loadable plugin over a source
patch is if you think you can use it to play games with the GPL.  The
biggest disadvantages of plugins are that you have to develop an API
that lets the plugins do what they want to do.  GDB really isn't laid
out that way.

Now, if you happen to have a brilliant idea on how to make it work...
:)

The most frequent need for "plugins" that I encounter is custom module
loaders; the Linux kernel's, XFree86's, et cetera.  This could be done
with a sufficiently well-integrated scripting language, and that's
a safer and simpler way to do it.

I took a look at your documentation, but not your code.  Basically, it
looks like you are creating a Perl binding to the MI interface - right?

In recent versions of GDB (6.0, but not 5.3), there is a console
command "interpreter-exec" that can be used to have a little dialog
with with another interpreter; either the MI or console interpreters. 
That may simplify your changes.  It would be really neat if you could
expose MI directly and then implement the rest on the Perl side as
functions which issue and parse MI; then you wouldn't need to add to
each binding when new MI commands are added.  That may be what you do
already for all I know :)

-- 
Daniel Jacobowitz
MontaVista Software                         Debian GNU/Linux Developer

Re: gdb + perl

[Kip Macy]

> And really awesome.  Kip, have you / could you file the copyright
> assignment paperwork for GDB?  That's a necessary first step to
> including any code.

I did that a couple years ago hoping to get in some fixes that I did for
thread support on FreeBSD. Does it still apply, or does one have to
renew it every year?

> > On a more general note I'd like to see loadable module support added to
> > GDB. This would allow people to maintain GDB extensions independently of
> > GDB. There are a number of things that I see adding to GDB that are only
> > interesting if you have a very large complex system and hence would
> > never be interesting for the majority of GDB users.
>
> We've talked about this before.  There are a couple of problems; the
> biggest is that the only advantage of a loadable plugin over a source
> patch is if you think you can use it to play games with the GPL.  The
> biggest disadvantages of plugins are that you have to develop an API
> that lets the plugins do what they want to do.  GDB really isn't laid
> out that way.

The code seems pretty well abstracted with various subsystem
initializers. It has something resembling object hierarchies, but
written in C. If anything there is periodically too much layering.
Neither I, nor my employer, have any issue with the GPL when it
comes to areas that are not part of their value add. Their toolchain,
although critical to their functioning, is not part of their value add.

> Now, if you happen to have a brilliant idea on how to make it work...
> :)

I'm in no rush. However, I'm happy that you are at least receptive to
the idea.

> The most frequent need for "plugins" that I encounter is custom module
> loaders; the Linux kernel's, XFree86's, et cetera.  This could be done
> with a sufficiently well-integrated scripting language, and that's
> a safer and simpler way to do it.

The motivation for me would be similar to that of mod_{perl, python,
ocaml} for apache. It allows to have a stock system GDB but load in
custom functionality on the fly. I could see something mainstream like
perl or python being accepted in the stock version, but as much I like
ocaml it is fairly obscure. There is a lot of functionality that could
be added to the debugger when one gets beyond the strict notion of
symbol lookups. I readily admit that the "ROI" on the complexity
of adding dynamic loading might not be adequate. Nonetheless, if the
loader could just call the appropriate "_initialize" functions in the
module just the way gdb does at startup for its subsystems, that might
be all you need.

>
> I took a look at your documentation, but not your code.  Basically, it
> looks like you are creating a Perl binding to the MI interface - right?

That is where most of the work is, however, that is not all. This is the
gross part, I actually link in libperl. This isn't the "right" approach
from engineering perspective. However, I'm looking at it from a social
engineering perspective. I wanted to not alter the way GDB was used at
all and yet allow perl to be used in a fashion similar to gdb macro
scripts. I export 2 functions to perl

1) "ext_mi_command" which takes an mi command plus args as a string
and returns what the MI would output to the console.

2) "register_callback" which takes the name of function, its help
string, the number of args, and array containing each of the types.

The drawback to this synchronous approach is it is only good for triage.
It doesn't handle the case of breakpoints being hit. In my defense
triage seems like something much easier to automate. In addition this is
not not be the final implementation. I'm trying to move developers over
to using perl. This is initially more an effort of social engineering
than of technical engineering. In 6 months if I've been successful and
developers feel a need to automated breakpoint debugging I envision a
new architecture, where instead of:

|---------------------------|
|            _________      |
|            |       |      |
| GDB       <-> Perl |      |
|            |_______|      |
|___________________________|



We could have:


|--------------------|    |---------------------|
|       GDB          |    |      Perl           |
|                  <------->                    |
|                    |    |                     |
|                    |    |                     |
|--------------------|    |---------------------|

This will:
1) allow us to avoid the symbol and header collision
garbage that I have to deal with when linking in perl

2) support live debugging automation for triaging problems
on systems to wedged to dump core - but still up enough to
use breakpoints

3) Eliminate any language specific stubs

>
> In recent versions of GDB (6.0, but not 5.3),

Although I made my changes available on 6.0, our in-house GDB is 5.3
for the moment.

> there is a console
> command "interpreter-exec" that can be used to have a little dialog
> with with another interpreter; either the MI or console interpreters.
> That may simplify your changes.

I'm not familiar with that - I hope I can use that, but I'd almost
certainly have to slice things up somewhat differently. This might ease
my path towards the second architecture. However, it isn't obvious how
that would support a callback mechanism necessary for adding to the
actual command set as define does.

> It would be really neat if you could
> expose MI directly and then implement the rest on the Perl side as
> functions which issue and parse MI; then you wouldn't need to add to
> each binding when new MI commands are added.  That may be what you do
> already for all I know :)

It is, modulo some "1.0" issues.

Thanks for your support. I appreciate the thoughtful consideration.

			-Kip

Re: gdb + perl

[Eli Zaretskii]

> Date: Tue, 3 Feb 2004 21:44:21 -0800 (PST)
> From: Kip Macy <kmacy@fsmware.com>
> 
> > And really awesome.  Kip, have you / could you file the copyright
> > assignment paperwork for GDB?  That's a necessary first step to
> > including any code.
> 
> I did that a couple years ago hoping to get in some fixes that I did for
> thread support on FreeBSD. Does it still apply, or does one have to
> renew it every year?

I just checked: your assignment for GDB is still on file in the FSF
records, so you don't need to file a new one.

Re: gdb + perl

[Andrew Cagney]

Kip,

This comes up often.  The theory is roughly as follows:

- At present MI is implemented using a hybrid of internal grubbing and 
relatively clean interfaces.  It has warts and all:

-- the breakpoint code is a mess (and further changes will just make it 
worse)
-- the disassembler is relatively clean
-- the varobj code is relatively clean (but needs a frame ID overhaul)
-- the stop code and event handling is a mess
-- the lack async in targets hurts
-- it doesn't use observers

- The MI interface is tested (relative to the CLI it's very well tested)

- The intent is for the MI to split into a thin vineer (the MI cli) and 
a "libgdb" like interface.

If anyone is going to look to binding GDB to their favorite interpreter 
they will also need to work on MI and its internal interfaces.  I think 
they would reasonably be expected to contribute the binding code to the FSF.

(Before you ask, GDB's licence won't be changed from GPL :-)

If you're looking for an example of how to not do things, study Red 
Hat's Insight.  It grubs around with all sorts of GDB internals, it 
pokes fingers where they should never go.  Something aproaching a clean 
integration would be a significant task (I'm working on the legal hurdle).

Andrew

Re: gdb + perl

[Kip Macy]

It was clear just by looking at the dispatch table in mi-cmd.c that the
MI implementation wasn't even as complete as the documentation. However,
fleshing out the MI as the need arises seemed much cleaner than any of
the alternatives. I am actually quite happy with the MI infrastructure
itself. I'd be happy to flesh out needed bits of the implementation over
time.

			-Kip




On Wed, 4 Feb 2004, Andrew Cagney wrote:

> Kip,
>
> This comes up often.  The theory is roughly as follows:
>
> - At present MI is implemented using a hybrid of internal grubbing and
> relatively clean interfaces.  It has warts and all:
>
> -- the breakpoint code is a mess (and further changes will just make it
> worse)
> -- the disassembler is relatively clean
> -- the varobj code is relatively clean (but needs a frame ID overhaul)
> -- the stop code and event handling is a mess
> -- the lack async in targets hurts
> -- it doesn't use observers
>
> - The MI interface is tested (relative to the CLI it's very well tested)
>
> - The intent is for the MI to split into a thin vineer (the MI cli) and
> a "libgdb" like interface.
>
> If anyone is going to look to binding GDB to their favorite interpreter
> they will also need to work on MI and its internal interfaces.  I think
> they would reasonably be expected to contribute the binding code to the FSF.
>
> (Before you ask, GDB's licence won't be changed from GPL :-)
>
> If you're looking for an example of how to not do things, study Red
> Hat's Insight.  It grubs around with all sorts of GDB internals, it
> pokes fingers where they should never go.  Something aproaching a clean
> integration would be a significant task (I'm working on the legal hurdle).
>
> Andrew
>
>

Re: gdb + perl

[Bob Rossi]

> - The intent is for the MI to split into a thin vineer (the MI cli) and 
> a "libgdb" like interface.

Who is working on the libgdb like interface?
I have been working on libtgdb for a while now. It is beginning to
support mi and it already supports annotate 2. It's a C library that
does all the I/O between GDB and returns data structures. 

It is at least functional in a trivial front end to GDB.

It is far from complete, but I am willing to add all the MI commands to
it as the need arises.

What do you think about making that the C binding to the MI?

Bob Rossi