Re: [RFC] plugin/extension interface

[Andrew STUBBS <andrew.stubbs@st.com>]

Thank you for your opinions. I did not expect quite so much controversy. 
Rather than reply to each part of the conversation separately I shall 
attempt to follow up all in one message. I would have responded sooner 
but for the time difference.

 > Such suggestions (not only for GDB, for other GNU projects as well)
 > are usually rejected by Richard Stallman and the FSF, because they
 > make it possible to add to GDB significant new features with
 > proprietary (i.e. non-free software) shared libraries, thus avoiding
 > the need to release any parts of GDB under GPL.

The GPL issues surprise me. I was under the impression that the GPL was 
harder to avoid than this. Is there anything you can point me at (an FSF 
judgement or whatever) that will prove that a dynamic library is not 
forced to use the GPL if it is linked to a GPL program (other than your 
assurances that this is the case)? The GPL itself mentions that system 
libraries are exempt, but I thought all others had to be GPL or 
compatible. This is an issue I would really like to understand.

 > It's true that plugin interfaces weaken the incentives the GPL tries
 > to create, but while I think that was very important even five years
 > ago, I don't think that's such a big deal any more.

I don't really want to comment on the politics, but I think it is worth 
pointing out that the alternative - implementing a GDB server - does not 
force use of the GPL either.

 > GDB is a hugely complicated program, with many independent areas; it's
 > not clear to me which of those areas you're trying to make pluggable.
 > It looks to me like it is specifically the target interface - just a
 > very small piece of the pie.  Also, new CLI commands.

As you say, GDB has many interfaces. However, I cannot see any practical 
use for exporting many of them - GDB already supports most of what the 
rest of the GNU toolchain can throw at it. Perhaps some people would 
like it to support other toolchains, but they probably have their own 
debuggers. The user interface may be extended through MI, and indeed 
many projects do this including DDD and Eclipse, so there's not real 
need to do anything there (although it might be more efficient). In most 
respects GDB has anticipated everything it can anticipate. Of course, 
there are always exceptions.

As I see it the thing that needs to be extendible is the target 
interface. Targets are one of the few areas that GDB cannot (and should 
not attempt to) anticipate.

 > If that's right, the traditional solution to the target interface is a
 > third party program which acts as a conversion tool between the GDB
 > remote protocol and your proprietary target protocol, with whatever
 > glue logic you would otherwise put into the plugin.  The remote
 > protocol serves as the boundary interface in the same way that the DLL
 > boundary would serve as an interface.

The remote protocol is very good at connecting to already-running 
programs, such as kgdb, uboot/redboot/etc, and gdb server. As you say, 
it permits custom features by means of a monitor command, but these are 
uni-directional and only work once connected to a target.

What I propose is more of a 'local' (but not 'exec') target interface. 
It does not require a gdb server running already and it allows the 
monitor commands to be present before connecting to the target, thus 
allowing configuration in .gdbinit and such like.

I suppose one way to look at it is this: an implementation of the remote 
protocol with direct function call instead of serial/ethernet and, 
because there is no need to arbitrarily impose the limitations of the 
remote protocol, a slightly friendlier monitor interface and, perhaps, 
access to a bit more detail.

The problems I have with the remote protocol as it stands are these:
   a) it is inefficient when the target interface software is not 
actually remote;
   b) it is built around the idea of a ROM monitor which is not suitable 
for many target types - many do not exist (conceptually or actually) 
until the debugger causes a process to start somehow;
   c) the monitor commands cannot feed data or commands back into the 
debugger;
   d) the remote target cannot adjust the debugger to its configuration;
   e) the remote gdb server cannot access the binary, symbol table or 
other bits and pieces that an advanced target interface supporting 
profiling/trace/etc. requires.

Of these a) can be fixed by having another comms flavour (such as direct 
function call or RPC); b) could be fixed by adding the ability to launch 
a process and hook up a pipe or something; and c) and d) can be fixed by 
extending the protocol. The other is more tricky.

 > My initial reaction (without reading the rest of your mail) would be
 > "Not over my dead body!".  In my experience plugin/extension
 > interfaces add a lot of bloat, are difficult to maintain, and
 > difficult to support.

I had planned to add one extra file to GDB (I'm not sure that 
constitutes bloat). This would contain all the code to load the DLL and 
the implementation of the target_ops functions. Looking at it again we 
could unify the latter part with the remote.c code by abstracting away 
the actual communications somehow. I'm not convinced that nailing them 
together would not create more trouble than it is worth though - 
divergent feature sets might introduce a lot of conditional code.

 > All the other bits you wanted to export
 > as methods have less clear semantics, and some of them are guaranteed
 > to change over time.

I certainly do not suggest that we expose any internal GDB structures. 
Not only would doing so make it hard to maintain plugins, they could not 
be compatible with multiple GDB versions.

I propose that a function is added for each piece of data or action that 
a plugin is interested in, and no more. Each of these would be 
independent of the internal GDB implementation. We might have 
'find_symbolname_at_address()' on the GDB side and 
'objfile_has_changed()' on the plugin side. As plugin developers need 
access to other details they can add new methods along similar lines and 
their plugin (or a specific feature of it) will only work with the new 
version of GDB or later. If GDB no longer supports one of these (or they 
are thought ugly or broken) then they can be deprecated (without 
changing the name though) and removed as normal - plugins relying on 
this one feature will need fixing, but not for every new release. New 
ways of doing old things may be added in parallel without trouble. 
Obviously, all new such interfaces would go through the review process, 
just like any other patch.

 > GDB is burdened already with support for a huge
 > number of targets using the _one_ interface; I don't want to deal with
 > backwards compatibility.

I'm not sure what you mean by this. I am not suggesting removing or 
changing the remote target in any way. Of course, to keep the number of 
internal interfaces down there's no reason why it couldn't be 
implemented as a plugin :) I'm not seriously suggesting this.

 > For new CLI commands, the right answer is probably to implement them
 > in-core in some scripting language, and optionally pass things back to
 > your remote target via "monitor" commands.

A new scripting language would, perhaps, make this somewhat less of an 
issue. It is important that there is some way to configure, control and 
query targets with more advanced features. For example, ours supports 
various performance measuring, tracing and direct JTAG wiggling among 
other things. I would be in favour of a more integrated approach. It 
isn't like adding and removing CLI commands is at all difficult.

 > Why can't you just compile your remote protocol support code into GDB.
 > That's the way it's been always done.  It's simple, and works, and I
 > think that'll actually be the easiest to maintain.  Plugin/extension
 > interfaces only make sense for third parties that want to distribute
 > binary stuff.  That doesn't make sense for an open source debugger.

This is indeed the case here. There are certain features of the targets 
which we use under NDA and so those parts of the code must not be open 
sourced. There would be no problem with the rest of the code, beyond 
that we prefer to keep it as a separate project with a separate code base.

 > I do think we will
 > want plugins eventually, but I haven't thought much about whether real
 > dlopen'd plugins are worthwhile, or e.g. python.

Python plugins could certainly provide UIs or user toys, but will they 
ever really be able to connect GDB to a target? Would you want them to? 
I can't see them easily utilising many of GDB's other interfaces either.

 > Short term it'd only increase the load since we
 > have to build the plug-in interface.  Long term, we still have to
 > maintain that interface.

I am intending to build the initial plug-in interface myself, so that 
won't cost you anything beyond review and discussion. It will be one 
self-contained patch which touches very little of the rest of the 
debugger. Most of what it needs is already available for the use of the 
existing target interface and the rest of GDB in general. A few things, 
such as the compare section command, may need some work, but this will 
be quite limited. I do not intend to expose every possible aspect of 
GDB, nor do I think it should. If somebody needs something more later 
then they extend it themselves, also at no great cost to you.

Maintainance is an issue of course, but such is life.

 > We'll also get lots of support questions
 > where the problems are really with the third party plugins, but where
 > the bug reporter conveniently forgets to mention that he's using a
 > third party plugin.  Let's keep people out of our address space if
 > they don't want to commit themselves to providing sources and manpower
 > to maintain those sources.

GDB should report errors that occur within plugins prefixed with the 
plugins name. That should head off most usage related issues. Of course, 
idiots are nothing if not cunning and will always find new ways to waste 
you time so I see your point. Hopefully the command names they quote 
would ring the alarm bells.

 > In other words, if the plug-ins had enough hooks to customize GDB's
 > behavior (printing values, or decoding the stack, say), I think you'd
 > be really surprised what people would do with it.  They'd do tons of
 > stuff that we'd never want to include in the core distribution, but
 > which would add enormous value to GDB for their problems.  And of
 > course, there would be a few that we'd want to incorporate.

I don't have time to go that far in the initial version, but these 
things evolve and I'd be happy to make the start and do targets and 
commands. As I hinted above, there's no point implementing something, in 
an open source project, on the off chance someone will use it. You put 
in the core bits and let people extend it as they need to.

As I think somebody else said, the frontend stuff can be done other 
ways. I'm more interested in backend stuff (which has a frontend presence).

 > I'm already in the long process of creating a level on top of GDB
 > using MI that allows another process to do anything it wants with GDB.

Can it add target interfaces? Efficiently?

 > Oh, I figured we'd just let the plugin code #include GDB's headers
 > directly, so they could get at whatever they wanted.

No, that would be just nasty.

 > ... okay, that does sound like a nightmare.

Agreed, if you tried to take it too far.

 > I did it to replace the current hack of having an extra program on the
 > pc that converts between the default gdb target protocol, and this
 > ICE-specific protocol.

 > The conversion hack was extremely slow, clumsy, fault-intolerant, and 
 > error-prone.

 > With the specific interface in gdb, the performance was wonderful, 
and > new gdb commands can be easily registered for this ICE, that are only
 > accessible after the command "target avrjtagice -d /dev/ttyS2" is
 > performed (like an emacs minor mode or whatever).

So there's at least two (groups) of us have gone through exactly the 
same process. How many others I wonder?

I think the rest of the discussion was mostly about GDB internals and 
documentation. If I have missed any important points that needed an 
answer then I apologise.

Andrew Stubbs