Registering pretty-printers

Registering pretty-printers

[Vladimir Prus]

Hello,
I have a concern about the way Python pretty-printers seem to work now.
To register a pretty-printer, one is supposed to do this
(according to http://permalink.gmane.org/gmane.comp.debugging.archer/1352)

	import sys
	sys.path.insert(0, 'XXX')
	from libstdcxx.v6.printers import register_libstdcxx_printers
	register_libstdcxx_printers (None)

This seem to contain quite a lot of information that is specific
for given pretty printer -- like the package name, and the name
of function to register pretty-printers. Some other project
might have package named boost and function called register_boost_printers,
and some other package might use something else. As result, it is not
possible to IDE user to just point at directory and have pretty-printers
loaded. Can we maybe require that the __init__ module does registration?

I am not quite sure how to avoid specifying this random (for user)
libstdcxx.v6 part -- any ideas?

- Volodya

Re: Registering pretty-printers

[Tom Tromey]

>>>>> "Vladimir" == Vladimir Prus <vladimir@codesourcery.com> writes:

Vladimir> I have a concern about the way Python pretty-printers seem
Vladimir> to work now.  To register a pretty-printer, one is supposed
Vladimir> to do this (according to
Vladimir> http://permalink.gmane.org/gmane.comp.debugging.archer/1352)

Vladimir> 	import sys
Vladimir> 	sys.path.insert(0, 'XXX')
Vladimir> 	from libstdcxx.v6.printers import register_libstdcxx_printers
Vladimir> 	register_libstdcxx_printers (None)

Just to be clear -- the context here is a user checking out just the
python printers from the gcc repository.  The gyrations are needed
because the usual hook file is processed by configure, and so isn't
available if you do a limited checkout like this.  So, you get to
reproduce it by hand somehow.

Vladimir> This seem to contain quite a lot of information that is
Vladimir> specific for given pretty printer -- like the package name,
Vladimir> and the name of function to register pretty-printers. Some
Vladimir> other project might have package named boost and function
Vladimir> called register_boost_printers, and some other package might
Vladimir> use something else. As result, it is not possible to IDE
Vladimir> user to just point at directory and have pretty-printers
Vladimir> loaded. Can we maybe require that the __init__ module does
Vladimir> registration?

I don't understand how this would work.


My thinking behind the current plan is that it is ok for gdb to fix
file names, but not Python module names.  I thought it was acceptable
to require a specific hook file name computed from an objfile's name,
but it was not acceptable to compute the Python module name from the
objfile's name.  Partly that is because objfile names cannot clash.

The hook file is re-read every time the objfile is created.  This is
needed because an object may be loaded and unloaded several times,
creating a new objfile each time.

Finally, we recommend that the hook file import the actual printer
implementations and install them on the objfile.  Furthermore, we
recommend putting some kind of version number into the namespace name
(the "v6" in the libstdc++ example).  The reason for these things is
future-proofing printers for multi-process -- the import means that
the printers are not re-created over and over each time the objfile is
created, and the versioning means that we can properly handle multiple
inferiors, each using a different version of a given library.


I realize this is all very shared-library- (aka distro-) centric.  It
works very well when all the bits are packaged nicely: python hook
files installed alongside libraries, etc.

It works less well in a couple of reasonably common scenarios:

1. Static linking.  Lookups are done by objfile, but with static
linking the distinctions are erased.

2. Users who have an old version of a library but who want to tack on
the pretty-printers.


I'm not really sure what to do in these cases.

For case #1, one idea is to punt the problem to the IDE.  An IDE could
presumably arrange to invoke the hook files for the static libraries
used by a given link.

I don't find this a particularly satisfactory solution, but I don't
have other ideas.  I think someone who uses static linking regularly
(I do not) would probably be better suited to come up with a solution.


For case #2 ... to me this seems like a short term problem, so I have
not been worrying about it much.  We could put a HOWTO on the wiki,
for example.  That would suffice for many users.

Tom

Re: Registering pretty-printers

[Vladimir Prus]

On Wednesday 10 June 2009 Tom Tromey wrote:

> >>>>> "Vladimir" == Vladimir Prus <vladimir@codesourcery.com> writes:
> 
> Vladimir> I have a concern about the way Python pretty-printers seem
> Vladimir> to work now.  To register a pretty-printer, one is supposed
> Vladimir> to do this (according to
> Vladimir> http://permalink.gmane.org/gmane.comp.debugging.archer/1352)
> 
> Vladimir> 	import sys
> Vladimir> 	sys.path.insert(0, 'XXX')
> Vladimir> 	from libstdcxx.v6.printers import register_libstdcxx_printers
> Vladimir> 	register_libstdcxx_printers (None)
> 
> Just to be clear -- the context here is a user checking out just the
> python printers from the gcc repository.  The gyrations are needed
> because the usual hook file is processed by configure, and so isn't
> available if you do a limited checkout like this.  So, you get to
> reproduce it by hand somehow.

Oh, I actually did not know about this hook (objname-gdb.py, right?).
Now that I've read about it, I will be able to better explain my
concerns.

> Vladimir> This seem to contain quite a lot of information that is
> Vladimir> specific for given pretty printer -- like the package name,
> Vladimir> and the name of function to register pretty-printers. Some
> Vladimir> other project might have package named boost and function
> Vladimir> called register_boost_printers, and some other package might
> Vladimir> use something else. As result, it is not possible to IDE
> Vladimir> user to just point at directory and have pretty-printers
> Vladimir> loaded. Can we maybe require that the __init__ module does
> Vladimir> registration?
> 
> I don't understand how this would work.
> 
> 
> My thinking behind the current plan is that it is ok for gdb to fix
> file names, but not Python module names.  I thought it was acceptable
> to require a specific hook file name computed from an objfile's name,
> but it was not acceptable to compute the Python module name from the
> objfile's name.  Partly that is because objfile names cannot clash.
> 
> The hook file is re-read every time the objfile is created.  This is
> needed because an object may be loaded and unloaded several times,
> creating a new objfile each time.
> 
> Finally, we recommend that the hook file import the actual printer
> implementations and install them on the objfile.  Furthermore, we
> recommend putting some kind of version number into the namespace name
> (the "v6" in the libstdc++ example).  The reason for these things is
> future-proofing printers for multi-process -- the import means that
> the printers are not re-created over and over each time the objfile is
> created, and the versioning means that we can properly handle multiple
> inferiors, each using a different version of a given library.
> 
> 
> I realize this is all very shared-library- (aka distro-) centric.  It
> works very well when all the bits are packaged nicely: python hook
> files installed alongside libraries, etc.
> 
> It works less well in a couple of reasonably common scenarios:
> 
> 1. Static linking.  Lookups are done by objfile, but with static
> linking the distinctions are erased.
> 
> 2. Users who have an old version of a library but who want to tack on
> the pretty-printers.

To begin with, this works only if there's linking. Say, I want to
provide pretty-printers for the Boost C++ Libraries. There's about 90
of individual libraries, and only 10 or so are actually compiled. A
user application might not use any of those 10, so there's no objfile
at all, ever. 

> I'm not really sure what to do in these cases.
> 
> For case #1, one idea is to punt the problem to the IDE.  An IDE could
> presumably arrange to invoke the hook files for the static libraries
> used by a given link.
> 
> I don't find this a particularly satisfactory solution, but I don't
> have other ideas.  I think someone who uses static linking regularly
> (I do not) would probably be better suited to come up with a solution.

I think it's generally wrong to assume that pretty-printers are either:
1. Always distributed with the library they provide pretty-printing for
2. Are associated with any separately compiled code.

Therefore, we should make up some conventions how a pretty-printer can
be packaged separately, preferably as a single file, and how IDE,
given name of the file and nothing else, can enable pretty printing.

Say, how about tar.gz, which is unpacked by IDE, whose top-level
directory is added to PYTHONPATH, and which should have top-level
file called init.py, with a function 'init_pretty_printers'?

> For case #2 ... to me this seems like a short term problem, so I have
> not been worrying about it much.  We could put a HOWTO on the wiki,
> for example.  That would suffice for many users.

I suspect you approach this from a bit of Fedora-centric approach. You
can easily transition important libraries to using pretty-printers,
or provide updates should pretty-printers be modified. IDE developers
have no luxury to assume that libraries users care about will be updated
in any time frame.

- Volodya


> 
> Tom
> 

Re: Registering pretty-printers

[Paul Pluzhnikov]

On Thu, Jun 11, 2009 at 1:29 AM, Vladimir Prus<vladimir@codesourcery.com> wrote:

> I think it's generally wrong to assume that pretty-printers are either:
> 1. Always distributed with the library they provide pretty-printing for
> 2. Are associated with any separately compiled code.

Perhaps my proposal for ".gdb_py" section here:
  http://sourceware.org/ml/archer/2008-q4/msg00162.html
deserves another look?

Maybe we don't even need a separate section; just a convention that any
variable linked into executable, named "gdb_python_.*_source_me" and
containing ASCII be "python sourced" by GDB?

The Boost code could declare:

  const char gdb_python_Boost_Array_source_me[] = "...python code here...";

in the header, and voila: GDB knows how to print boost::array.

This could also be easily distributed separately from the library, and
the user will just have to link in gdb_boost_array_printer.c into his exe.

-- 
Paul Pluzhnikov

Re: Registering pretty-printers

[Daniel Jacobowitz]

On Thu, Jun 11, 2009 at 10:14:32AM -0700, Paul Pluzhnikov wrote:
> Perhaps my proposal for ".gdb_py" section here:
>   http://sourceware.org/ml/archer/2008-q4/msg00162.html
> deserves another look?
> 
> Maybe we don't even need a separate section; just a convention that any
> variable linked into executable, named "gdb_python_.*_source_me" and
> containing ASCII be "python sourced" by GDB?
> 
> The Boost code could declare:
> 
>   const char gdb_python_Boost_Array_source_me[] = "...python code here...";
> 
> in the header, and voila: GDB knows how to print boost::array.
> 
> This could also be easily distributed separately from the library, and
> the user will just have to link in gdb_boost_array_printer.c into his exe.

In my opinion, anything that increases the size of the executable is a
non-starter.  I don't think there's any reliable way to create a
non-allocatable section, and it would have other problems, like
duplicate elimination.

-- 
Daniel Jacobowitz
CodeSourcery

Re: Registering pretty-printers

[Vladimir Prus]

Daniel Jacobowitz wrote:

> On Thu, Jun 11, 2009 at 10:14:32AM -0700, Paul Pluzhnikov wrote:
>> Perhaps my proposal for ".gdb_py" section here:
>>   http://sourceware.org/ml/archer/2008-q4/msg00162.html
>> deserves another look?
>> 
>> Maybe we don't even need a separate section; just a convention that any
>> variable linked into executable, named "gdb_python_.*_source_me" and
>> containing ASCII be "python sourced" by GDB?
>> 
>> The Boost code could declare:
>> 
>>   const char gdb_python_Boost_Array_source_me[] = "...python code here...";
>> 
>> in the header, and voila: GDB knows how to print boost::array.
>> 
>> This could also be easily distributed separately from the library, and
>> the user will just have to link in gdb_boost_array_printer.c into his exe.
> 
> In my opinion, anything that increases the size of the executable is a
> non-starter.  I don't think there's any reliable way to create a
> non-allocatable section, and it would have other problems, like
> duplicate elimination.

Personally, I don't care about executable size much, but I find creating
custom source file and linking it into executable a very indirect method.
It is hard to implement from IDE side, too, because IDE can be used to
debug projects that are build by JRHackerBuildSystem, and be unable to
automatically add source files to build. My ultimate goal is allowing IDE
user to just specify 10 pretty-printer packages, and have them work on
every single project he's working on.

- Volodya

Re: Registering pretty-printers

[Doug Evans]

On Thu, Jun 11, 2009 at 8:51 PM, Daniel Jacobowitz<drow@false.org> wrote:
> In my opinion, anything that increases the size of the executable is a
> non-starter.  I don't think there's any reliable way to create a
> non-allocatable section, and it would have other problems, like
> duplicate elimination.

Reliable in what sense?  [I realize the term is pretty unambiguous.
I'm guessing I'm missing something as it doesn't seem to be
excessively hard for many important targets.]  Or did you mean
portable?

As for duplicate elimination,
I'm reminded of the new comdat types support in DWARF 4:  Type debug
info is put in comdat sections with a signature computed from the
type.  The linker removes all duplicates and the result ends up in the
.debug_types section.  Employing a similar thing here would solve that
it seems.  It might even be cool to use the same signature.  It avoids
wheel re-invention, and uses something documented and well-vetted.
And maybe the pretty-printer registration could be based on the type
signature.  OTOH the signature wasn't really intended for anything
else so hijacking it may be problematic.
[Just thinking out loud ...]

Not saying there aren't other problems though.
OTOH, I wouldn't mind seeing a couple of paradigms for registering
pretty printers if it means targets that can handle better solutions
aren't left to suffer to the lowest common denominator.

Re: Registering pretty-printers

[Daniel Jacobowitz]

On Fri, Jun 12, 2009 at 12:43:12PM -0400, Doug Evans wrote:
> On Thu, Jun 11, 2009 at 8:51 PM, Daniel Jacobowitz<drow@false.org> wrote:
> > In my opinion, anything that increases the size of the executable is a
> > non-starter.  I don't think there's any reliable way to create a
> > non-allocatable section, and it would have other problems, like
> > duplicate elimination.
> 
> Reliable in what sense?  [I realize the term is pretty unambiguous.
> I'm guessing I'm missing something as it doesn't seem to be
> excessively hard for many important targets.]  Or did you mean
> portable?

Poor choice of word.  Same thing Vladimir was talking about, though,
in essence.  I don't want the only decent solution to be one that
requires pushing new tools out.  Fedora can do that; maybe Google can
too, I don't know anything about your development environments.  But
plenty of other places can't.

I think Vladimir's got a solid point - we need a way to just point the
debugger at files and have them go.

-- 
Daniel Jacobowitz
CodeSourcery

Re: Registering pretty-printers

[Tom Tromey]

>>>>> "Daniel" == Daniel Jacobowitz <drow@false.org> writes:

Paul> Perhaps my proposal for ".gdb_py" section here:
Paul> http://sourceware.org/ml/archer/2008-q4/msg00162.html
Paul> deserves another look?

Daniel> In my opinion, anything that increases the size of the
Daniel> executable is a non-starter.

In addition to this, I think there are a few other problems with this
approach.

First, this requires a gcc update in order to work.  Also, it means a
vendor-specific compiler extension in order for the feature to work.
This makes one of Vladimir's problems worse.

Some languages (Java at least) don't have pragmas.  (This problem is
not as severe, since you could mix in a C++ CU.)

This approach means recompiling your source whenever you want to
update the printers.  This seems to defeat one of the nicest qualities
of using a scripting language.

Tom

Re: Registering pretty-printers

[Doug Evans]

On Fri, Jun 12, 2009 at 10:51 AM, Daniel Jacobowitz<drow@false.org> wrote:
> I think Vladimir's got a solid point - we need a way to just point the
> debugger at files and have them go.

Thanks.  I agree that it's important to be able to do this.

Re: Registering pretty-printers

[Tom Tromey]

>>>>> "Vladimir" == Vladimir Prus <vladimir@codesourcery.com> writes:

Vladimir> To begin with, this works only if there's linking.

Yeah.

Vladimir> I think it's generally wrong to assume that pretty-printers
Vladimir> are either:
Vladimir> 1. Always distributed with the library they provide
Vladimir> pretty-printing for
Vladimir> 2. Are associated with any separately compiled code.

What we actually have is a setup that makes it very easy to auto-load
pretty-printers in this situation, which I believe will be the most
common one in the long run.

This setup is constructed by gluing together a couple generic pieces:
objfile hook auto-loading and per-objfile pretty-printer lookup
functions.

Because this functionality is generic, we can actually load
pretty-printers any other way.  When doing development I just "python
execfile" them into my gdb session; in the printers I write I use the
convention that if the current objfile is None, then the code
registers the printers globally.  Well, in theory -- I see I broke
this for libstdc++.

(I think we ought do document this convention btw.)

I would like to solve the remaining problems.  I just don't know how
to do it, in a way that preserves the qualities I care about.  What I
care about is, more or less, embedded in what I implemented: make it
super easy for printers to be developed alongside applications and
libraries, and be made available automatically without user
intervention.

Vladimir> Therefore, we should make up some conventions how a
Vladimir> pretty-printer can be packaged separately, preferably as a
Vladimir> single file, and how IDE, given name of the file and nothing
Vladimir> else, can enable pretty printing.

That would be fine by me.

Vladimir> Say, how about tar.gz, which is unpacked by IDE, whose top-level
Vladimir> directory is added to PYTHONPATH, and which should have top-level
Vladimir> file called init.py, with a function 'init_pretty_printers'?

I don't see how it is better than the hook file approach we already
have.  Your IDE could easily send "python execfile" to load any given
hook file.  Defining a function in that file doesn't add anything that
I can see.

It seems to me that you could make a .tar holding all the files from
libstdc++/python, then have your IDE unpack these somewhere, update
sys.path, and execfile the file(s) in the topmost directory.  Assuming
these files follow the "None convention", it will work fine.

Packaging as a .tar or whatever would have to be IDE-specific, I think.
I don't think it would work for my situation.

Another thing I've contemplated is having a generic event that is
fired when an objfile is loaded.  Then you could avoid the
file-name-based auto-loading scheme and roll your own in Python.  I
don't know if this would help you or not.

Tom> For case #2 ... to me this seems like a short term problem, so I have
Tom> not been worrying about it much.  We could put a HOWTO on the wiki,
Tom> for example.  That would suffice for many users.

Vladimir> I suspect you approach this from a bit of Fedora-centric
Vladimir> approach.

I'm sure you're right.

Tom

Re: Registering pretty-printers

[Vladimir Prus]

On Friday 12 June 2009 Tom Tromey wrote:

> I would like to solve the remaining problems.  I just don't know how
> to do it, in a way that preserves the qualities I care about.  What I
> care about is, more or less, embedded in what I implemented: make it
> super easy for printers to be developed alongside applications and
> libraries, and be made available automatically without user
> intervention.

Probably the cases I have described are too tricky to allow for automatic
setup of pretty-printers. E.g. header-only-library case is pretty much
impossible to handle.

> Vladimir> Therefore, we should make up some conventions how a
> Vladimir> pretty-printer can be packaged separately, preferably as a
> Vladimir> single file, and how IDE, given name of the file and nothing
> Vladimir> else, can enable pretty printing.
> 
> That would be fine by me.
> 
> Vladimir> Say, how about tar.gz, which is unpacked by IDE, whose top-level
> Vladimir> directory is added to PYTHONPATH, and which should have top-level
> Vladimir> file called init.py, with a function 'init_pretty_printers'?
> 
> I don't see how it is better than the hook file approach we already
> have.  Your IDE could easily send "python execfile" to load any given
> hook file.  Defining a function in that file doesn't add anything that
> I can see.

There are two important points I propose:

1. Having a file at top-level, as opposed in some subdir which name
differs.
2. Having a file with fixed name. 

I am probably wrong, but neither of this is true with the current
recommended approach.

> It seems to me that you could make a .tar holding all the files from
> libstdc++/python, then have your IDE unpack these somewhere, update
> sys.path, and execfile the file(s) in the topmost directory.  Assuming
> these files follow the "None convention", it will work fine.

I did not notice any files on top-level. Have I missed something?

- Volodya

Re: Registering pretty-printers

[Tom Tromey]

>>>>> "Vladimir" == Vladimir Prus <vladimir@codesourcery.com> writes:

Vladimir> There are two important points I propose:

Vladimir> 1. Having a file at top-level, as opposed in some subdir which name
Vladimir> differs.
Vladimir> 2. Having a file with fixed name. 

Vladimir> I am probably wrong, but neither of this is true with the current
Vladimir> recommended approach.

Yeah.  But isn't this just a matter of documenting "this is how we
recommend you distribute stand-alone printers"?  This whole structure
can only really work for the "standalone package" case anyway, because
in the distro case there is no identifiable top-level; the hook file
and the printers are probably stored in entirely different places.

For #1, if you don't care what the file is called, then you can often
just use the existing hook file.  Just copy it to the right place
before making the tar.

Tom> It seems to me that you could make a .tar holding all the files from
Tom> libstdc++/python, then have your IDE unpack these somewhere, update
Tom> sys.path, and execfile the file(s) in the topmost directory.  Assuming
Tom> these files follow the "None convention", it will work fine.

Vladimir> I did not notice any files on top-level. Have I missed something?

Nope, but you are probably looking at the libstdc++ source tree.  The
hook file is created by configure; libstdc++ requires an abnormally
weird hook file due to relocatability requirements of gcc.

If you were going to actually package this up for independent release,
you would make the top-level file "somehow", maybe by running sed on
hook.in.

Tom

Re: Registering pretty-printers

[Tom Tromey]

Vladimir> There are two important points I propose:
Vladimir> 1. Having a file at top-level, as opposed in some subdir which name
Vladimir> differs.
Vladimir> 2. Having a file with fixed name. 
Vladimir> I am probably wrong, but neither of this is true with the current
Vladimir> recommended approach.

Tom> Yeah.  But isn't this just a matter of documenting "this is how we
Tom> recommend you distribute stand-alone printers"?

I have been thinking more about this problem.  I am now wondering if
we could have a solution based on Python Eggs.  Background reading:

http://peak.telecommunity.com/DevCenter/PythonEggs

My rough idea would be that a suite of pretty-printers would be
packaged as an Egg.  We would define a specific resource which would
name the registration function.

Hook files could continue to work more or less as they do now.

An IDE could upload Python code to gdb to invoke the registration
function manually.

I haven't tried to experiment with this, but based on my reading so
far, it seems like it ought to work.  What do you think?

Tom

Re: Registering pretty-printers

[Vladimir Prus]

On Saturday 27 June 2009 Tom Tromey wrote:

> Vladimir> There are two important points I propose:
> Vladimir> 1. Having a file at top-level, as opposed in some subdir which name
> Vladimir> differs.
> Vladimir> 2. Having a file with fixed name. 
> Vladimir> I am probably wrong, but neither of this is true with the current
> Vladimir> recommended approach.
> 
> Tom> Yeah.  But isn't this just a matter of documenting "this is how we
> Tom> recommend you distribute stand-alone printers"?
> 
> I have been thinking more about this problem.  I am now wondering if
> we could have a solution based on Python Eggs.  Background reading:
> 
> http://peak.telecommunity.com/DevCenter/PythonEggs
> 
> My rough idea would be that a suite of pretty-printers would be
> packaged as an Egg.  We would define a specific resource which would
> name the registration function.
> 
> Hook files could continue to work more or less as they do now.
> 
> An IDE could upload Python code to gdb to invoke the registration
> function manually.
> 
> I haven't tried to experiment with this, but based on my reading so
> far, it seems like it ought to work.  What do you think?

I did not tried either, nor did I worked through design, but it looks like
python eggs is the de-facto standard way to publish anything python and
should work.

- Volodya