plugin interface for GDB

plugin interface for GDB

[Thiago Jung Bauermann]

Hi folks,

I am interested in developing a plugin interface for GDB, and would
like to know what are your thoughts about it. Would you find such
feature useful?  Would it be integrated into GDB?

This is useful to me, so I'll probably do it even if there's no plan
to integrate the plugin interface upstream (in fact, I have a very
basic working prototype already). But if there is community interest in
such thing, I'd like to do it in a way that will increase its chance of
integration and shape it so that more people could benefit from it.

I saw some discussion on this subject in the following thread:

http://sourceware.org/ml/gdb/2002-04/msg00371.html

But it didn't come to a conclusion, so I'd like to revisit the idea.

Motivation for having a plugin interface:

- enable seamless debugging of programs written in multiple languages
  (Java + C, Python + C, etc) (in fact, this is the reason I need the
  plugin interface.)

- useful for specialized debugging (specific problem domain)

- lessens the need to maintain custom GDBs. You just use a standard GDB
  and write a plugin (or use an existing one). If you decide to upgrade
  GDB, you won't need to port your code and not even recompile the
  plugin (ideally, if we manage to design a resilient interface).

- allow people outside of the core GDB team to implement distribute and
  test their plugins without modifying GDB.
  
- lowers the bar to creating modifications to GDB (exposes a well
  defined interface, so there's no need to learn the ins and outs of
  GDB to write the plugin you need/want)

Scott Moser's idea was to have very basic plugin support (loading and
unloading), and leave to the plugin writer the work of searching which
internal GDB functions he'd need to use and directly call them from the
plugin code. I am more inclined to go into a different direction, which
is to provide an abstraction layer which would expose functionality
which is potentially useful for a plugin.

Each approach has advantages and disadvantages, but some of the
advantages I pointed out above only hold if you have such abstraction
layer.
-- 
[]'s
Thiago Jung Bauermann
Software Engineer
IBM Linux Technology Center

Re: plugin interface for GDB

[Brian Dessent]

Thiago Jung Bauermann wrote:

> Scott Moser's idea was to have very basic plugin support (loading and
> unloading), and leave to the plugin writer the work of searching which
> internal GDB functions he'd need to use and directly call them from the
> plugin code. I am more inclined to go into a different direction, which
> is to provide an abstraction layer which would expose functionality
> which is potentially useful for a plugin.

You should read the recent threads about adding scripting support to
gdb.  The one from several months ago
<http://sourceware.org/ml/gdb/2007-01/threads.html#00126> seemed to
mostly converge on Python, with tcl and guile as runners up.

To me it seems like it would be a lot easier to go in that direction
than to actually allow for plugins in the sense of dynamically loading a
shared module of compiled code.  In either case, you have to implement
some kind of API that exposes the internals of gdb to the plugin or the
script, and in either case you can extend the functionality of gdb in
pretty much arbitrary ways.  But in the case of scripting you don't have
ABI headaches to worry about, and  it increases accessilbility since you
can express relatively high level concepts easily in just a short amount
of python script, compared to having to code the same amount of
functionality with a C plugin.

Brian

Re: plugin interface for GDB

[Daniel Jacobowitz]

On Fri, Apr 20, 2007 at 04:43:48PM -0300, Thiago Jung Bauermann wrote:
> I am interested in developing a plugin interface for GDB, and would
> like to know what are your thoughts about it. Would you find such
> feature useful?  Would it be integrated into GDB?

I don't think it very likely that we would add a C plugin interface.
We've discussed the disadvantages many times before.  But at the same
time, I'm very interested in a "plugin"-like system using a scripting
language.  You can find a discussion about using Python earlier in
this year's archives.

In my opinion the main difference is the character of the interface;
using a scripting language has plenty of inherent advantages, plus it
makes it clear that you do not get to grub around in any other bit of
GDB, period, no matter how clever you think you are.

> - enable seamless debugging of programs written in multiple languages
>   (Java + C, Python + C, etc) (in fact, this is the reason I need the
>   plugin interface.)

What does this have to do with a plugin?  That is, why is a plugin
useful for it, rather than additional support in GDB proper?  It's a
very vague problem statement - an interesting one, though.

> - lessens the need to maintain custom GDBs. You just use a standard GDB
>   and write a plugin (or use an existing one). If you decide to upgrade
>   GDB, you won't need to port your code and not even recompile the
>   plugin (ideally, if we manage to design a resilient interface).
> 
> - allow people outside of the core GDB team to implement distribute and
>   test their plugins without modifying GDB.
>   
> - lowers the bar to creating modifications to GDB (exposes a well
>   defined interface, so there's no need to learn the ins and outs of
>   GDB to write the plugin you need/want)

Remember, we want GDB contributors!  That's the whole point - getting
people to contribute.  So for things that would be generally useful,
which all of the above are, helping people do them without
contributing to GDB is entirely the wrong way round.

I do not believe that you can come up with an abstraction layer big
enough to be useful that is not a significant maintenance burden for
the GDB developers.  I've been surprised before, though!

-- 
Daniel Jacobowitz
CodeSourcery

Re: plugin interface for GDB

[Thiago Jung Bauermann]

On Fri, 2007-04-20 at 16:00 -0400, Daniel Jacobowitz wrote:
> On Fri, Apr 20, 2007 at 04:43:48PM -0300, Thiago Jung Bauermann wrote:
> > I am interested in developing a plugin interface for GDB, and would
> > like to know what are your thoughts about it. Would you find such
> > feature useful?  Would it be integrated into GDB?
> 
> I don't think it very likely that we would add a C plugin interface.
> We've discussed the disadvantages many times before.  But at the same
> time, I'm very interested in a "plugin"-like system using a scripting
> language.  You can find a discussion about using Python earlier in
> this year's archives.

Yes, I saw those. Thanks for pointing them out, though. I think it's
very interesting and useful as well.

> In my opinion the main difference is the character of the interface;
> using a scripting language has plenty of inherent advantages, plus it
> makes it clear that you do not get to grub around in any other bit of
> GDB, period, no matter how clever you think you are.

I agree that a plugin interface and a scripting language overlap a lot
in terms of functionality. I'm not sure it would cut it in my specific
case... I'm looking at something which will potentially need to examine
10's of GB of inferior memory in some cases and I'm a bit worried about
the performance. If there was already scripting language support in GDB,
I could do some benchmarks. It's quite possible that the overhead of the
scripting language would be low or acceptable. But right now I don't
want to work (probably a lot) on developing such support only to find
out it won't meet my performance needs... :-)

> > - enable seamless debugging of programs written in multiple languages
> >   (Java + C, Python + C, etc) (in fact, this is the reason I need the
> >   plugin interface.)
> 
> What does this have to do with a plugin?  That is, why is a plmaybe ugin
> useful for it, rather than additional support in GDB proper?

We want to add support in GDB to debug both IBM's JVM and programs
running on top of it at the same time. It could go into GDB proper,
except that we need to keep our modifications internal and I want to
minimize the burden of porting the code to new GDB releases, so I don't
want to mess too much with the guts of GDB. (objectionable motivation, I
know. But these are the constraints I need to work with).

As I said, a plugin interface is a good answer to that need, and I'll
probably do such interface even if only to keep it internal.

I believe there are other people in the same situation as mine. I've
seen messages here stating something like "I'm using an old version of
GDB because I customized it and still didn't port the code to a newer
version". A plugin interface would help those people.

Granted, a scripting language would possibly solve their needs, but
maybe not for all of them. Knowing more reasons which lead to folks
getting into the trouble of maintaining an internal GDB version would
probably help this discussion. I can only speculate...

>   It's a vague problem statement - an interesting one, though.

Think Java code which calls native methods via JNI (very common), or the
other way around: a C or C++ program which fires up a JVM. It's
cumbersome to debug the combination today. You can't have a backtrace
which shows both the interpreted language and the native language
frames, you can't step into a function which is written in the other
language.. You can't define some breakpoints in the interpreted language
part and others in the C part. And so on.

I agree, it's a very interesting problem, and it mostly doesn't need a
plugin interface to be implemented. But there might be a few little
known or unpopular interpreted languages for which someone might have an
interest in writing such support, but then wouldn't be accepted upstream
for one reason or another (e.g., because GDB maintainers could think
it's not worth the effort to keep it in mainline (hypotetical situation,
of course! :-) )).

> > - lessens the need to maintain custom GDBs. You just use a standard GDB
> >   and write a plugin (or use an existing one). If you decide to upgrade
> >   GDB, you won't need to port your code and not even recompile the
> >   plugin (ideally, if we manage to design a resilient interface).
> > 
> > - allow people outside of the core GDB team to implement distribute and
> >   test their plugins without modifying GDB.
> >   
> > - lowers the bar to creating modifications to GDB (exposes a well
> >   defined interface, so there's no need to learn the ins and outs of
> >   GDB to write the plugin you need/want)
> 
> Remember, we want GDB contributors!  That's the whole point - getting
> people to contribute.  So for things that would be generally useful,
> which all of the above are, helping people do them without
> contributing to GDB is entirely the wrong way round.

Agreed. But like in the example I gave above, there might be some
features which don't interest the GDB community for one reason or
another, but other people need or want.

I don't think this would turn people away from contributing to GDB,
because things which make sense to be in GDB proper would still go
there. It's less work to have a patch accepted into GDB mainline than
have to maintain one's own plugin. Besides, the integrated patch will
reach more people (all GDB users, as opposed to users of the plugin).
But for the things which can't go into GDB for one reason or another,
it's a good option.

In fact, this could even attract developers because of the "lower the
bar" point. Developing a plugin would help people get familiar with GDB
and also instill courage to start hacking on it.

> I do not believe that you can come up with an abstraction layer big
> enough to be useful that is not a significant maintenance burden for
> the GDB developers.  I've been surprised before, though!

I think you'd be surprised. :-)

I was. In a very short time we were able to come up with a working
prototype which provides very little functionality (basically, just
interaction with the user and peek inferior memory) but at the same time
is enough to meet some of our needs.

Of course there's a long way to go from there, and code will start
getting more complex. But if it's useful enough, the maintenance burden
is bearable.

BTW, Apple did write a plugin interface for GDB. It's available from
their Darwin project website, if you're curious:

http://www.opensource.apple.com/darwinsource/DevToolsAug2006/gdb-563/macsbug/gdb_plugin_support/

-- 
[]'s
Thiago Jung Bauermann
Software Engineer
IBM Linux Technology Center

Re: plugin interface for GDB

[Thiago Jung Bauermann]

On Fri, 2007-04-20 at 12:57 -0700, Brian Dessent wrote:
> > Scott Moser's idea was to have very basic plugin support (loading and
> > unloading), and leave to the plugin writer the work of searching which
> > internal GDB functions he'd need to use and directly call them from the
> > plugin code. I am more inclined to go into a different direction, which
> > is to provide an abstraction layer which would expose functionality
> > which is potentially useful for a plugin.
> 
> You should read the recent threads about adding scripting support to
> gdb.  The one from several months ago
> <http://sourceware.org/ml/gdb/2007-01/threads.html#00126> seemed to
> mostly converge on Python, with tcl and guile as runners up.
> 
> To me it seems like it would be a lot easier to go in that direction
> than to actually allow for plugins in the sense of dynamically loading a
> shared module of compiled code.

I saw that thread. It's certainly very desirable functionality. My
problem is that maybe it would meet my needs, maybe not. Like I just
mentioned in a message to Daniel, we'll be looking at lots of data. I'm
not sure what would be the performance of that...

Also, we have a lot of C code which we'll integrate with GDB, so a layer
of interpreted language is unnecessary to us and just adds complexity.

>   In either case, you have to implement
> some kind of API that exposes the internals of gdb to the plugin or the
> script, and in either case you can extend the functionality of gdb in
> pretty much arbitrary ways.

Agreed. Both features overlap a lot in terms of functionality provided
to users, but I'm afraid they are not completely equivalent. I think
this discussion is a good opportunity to try to determine this.

>   But in the case of scripting you don't have
> ABI headaches to worry about,

You mean keeping binary compatibility with plugins accross different GDB
versions? That's a nice goal, but not worth worrying about too much, I
think.

>  and  it increases accessilbility since you
> can express relatively high level concepts easily in just a short amount
> of python script, compared to having to code the same amount of
> functionality with a C plugin.

Agreed.
-- 
[]'s
Thiago Jung Bauermann
Software Engineer
IBM Linux Technology Center

Re: plugin interface for GDB

[Daniel Jacobowitz]

On Fri, Apr 20, 2007 at 06:21:11PM -0300, Thiago Jung Bauermann wrote:
> I agree that a plugin interface and a scripting language overlap a lot
> in terms of functionality. I'm not sure it would cut it in my specific
> case... I'm looking at something which will potentially need to examine
> 10's of GB of inferior memory in some cases and I'm a bit worried about
> the performance. If there was already scripting language support in GDB,
> I could do some benchmarks. It's quite possible that the overhead of the
> scripting language would be low or acceptable. But right now I don't
> want to work (probably a lot) on developing such support only to find
> out it won't meet my performance needs... :-)

Well, it's going to get developed eventually in either case.  If you
can't write things fast enough in Python, Python itself lets you load
C modules to do the real work; the question is how fast GDB can
provide memory access to the Python layer.  And I think the answer is
roughly as fast as GDB could get at it directly.

> We want to add support in GDB to debug both IBM's JVM and programs
> running on top of it at the same time. It could go into GDB proper,
> except that we need to keep our modifications internal and I want to
> minimize the burden of porting the code to new GDB releases, so I don't
> want to mess too much with the guts of GDB. (objectionable motivation, I
> know. But these are the constraints I need to work with).

Not surprising constraints.  The trick, for you and for us, is to
minimize the amount of it is local to IBM and specific to the JVM.

> As I said, a plugin interface is a good answer to that need, and I'll
> probably do such interface even if only to keep it internal.
> 
> I believe there are other people in the same situation as mine. I've
> seen messages here stating something like "I'm using an old version of
> GDB because I customized it and still didn't port the code to a newer
> version". A plugin interface would help those people.

This, you see, is the part I don't believe.  I think that a _perfect_
plugin interface would help.  You get to draw your own conclusions on
how likely that is :-)

> > I do not believe that you can come up with an abstraction layer big
> > enough to be useful that is not a significant maintenance burden for
> > the GDB developers.  I've been surprised before, though!
> 
> I think you'd be surprised. :-)
> 
> I was. In a very short time we were able to come up with a working
> prototype which provides very little functionality (basically, just
> interaction with the user and peek inferior memory) but at the same time
> is enough to meet some of our needs.
> 
> Of course there's a long way to go from there, and code will start
> getting more complex. But if it's useful enough, the maintenance burden
> is bearable.

The question is what you need, and how easy it is to expose in another
way.  I did a prototype of Guile integration in just a couple of days
that did basically the same amount you've got here; and if I'd done it
in Python (being a language I can actually write things in, unlike the
staggering around I do in Guile), it could even have been useful.

-- 
Daniel Jacobowitz
CodeSourcery

Re: plugin interface for GDB

[Paul Koning]

>>>>> "Daniel" == Daniel Jacobowitz <drow@false.org> writes:

 Daniel> On Fri, Apr 20, 2007 at 06:21:11PM -0300, Thiago Jung
 Daniel> Bauermann wrote:
 >> I agree that a plugin interface and a scripting language overlap a
 >> lot in terms of functionality. I'm not sure it would cut it in my
 >> specific case... I'm looking at something which will potentially
 >> need to examine 10's of GB of inferior memory in some cases and
 >> I'm a bit worried about the performance. If there was already
 >> scripting language support in GDB, I could do some
 >> benchmarks. It's quite possible that the overhead of the scripting
 >> language would be low or acceptable. But right now I don't want to
 >> work (probably a lot) on developing such support only to find out
 >> it won't meet my performance needs... :-)

 Daniel> Well, it's going to get developed eventually in either case.
 Daniel> If you can't write things fast enough in Python, Python
 Daniel> itself lets you load C modules to do the real work; the
 Daniel> question is how fast GDB can provide memory access to the
 Daniel> Python layer.  And I think the answer is roughly as fast as
 Daniel> GDB could get at it directly.

Very likely.  "Scripting language" often makes people expect low
performance, and for some that may be true (shell scripts?  Tcl
perhaps).

Python is very fast.  It compiles to bytecode, and the execution
machinery seems to work quite well.  I have an application that runs
through very large files, doing some simple string matching.  It runs
at 50 MB/s or better, on a PC.

So I don't think speed will be a concern.

   paul

Re: plugin interface for GDB

[Robert Dewar]

Paul Koning wrote:

> Python is very fast.  It compiles to bytecode, and the execution
> machinery seems to work quite well.  I have an application that runs
> through very large files, doing some simple string matching.  It runs
> at 50 MB/s or better, on a PC.

And in fact Python could be a lot faster. In a class project we did
a proof of concept project to build a Python using the same approach
as was used for SPITBOL, compiling to real executable code. Python
programs ran 5-50 times faster than current interpretors. Maybe one
day :-)