[RFC] Make python/lib/gdb and submodules proper Python modules

Khoo Yit Phang
2012-09-03T03:44:32Z

[-- Attachment #1: Type: text/plain, Size: 1292 bytes --]

Hi,

I found it strange that python/lib/gdb and its submodules aren't truly Python modules (e.g., the python/lib/gdb/__init__.py had to explicitly refer to the gdb module), so I've put together a patch that makes them proper Python modules.

This patch works by splitting out the gdb module into _gdb for the native code extensions (as recommended by PEP8), setting up sys.path in finish_python_initialization and finally loading the gdb module, as well as switching from using exec to reload/__import__ (which updates/creates Python modules). Also, I've made a few other related changes: I removed gdb.PYTHONDIR which isn't necessary anymore since the same information is available via gdb.__file__, and moved the Python code from finish_python_initialization to python/lib/gdb/__init__.py.

One caveat about this patch is that the data-directory must contain the gdb module, otherwise Python will not be fully initialized (with a warning) and Python support will be very limited (only the _gdb module will be available).

I've ran the testsuite on Ubuntu 11.04 with Python 2.7.1 and Python 2.4.6 and found no regressions (you may need the patch I recently posted that adds -nx to a few tests to avoid spurious results).

What do you think?

Thanks,

Yit
September 2, 2012

[-- Attachment #2: python-gdb-native-module.txt --]
[-- Type: text/plain, Size: 13848 bytes --]

Refactor Python "gdb" module into a proper Python package, by introducing a new "_gdb" module for code implemented in C.

gdb/ChangeLog:

2012-09-02 Khoo Yit Phang <khooyp@cs.umd.edu>

Refactor Python "gdb" module into a proper Python package, by
introducing a new "_gdb" module for code implemented in C, and
using reload/__import__ instead of exec.
* python/lib/gdb/__init__.py: Import * from _gdb.
(GdbOutputFile, sys.stdout, GdbOutputErrorFile, sys.stderr,
prompt_hook, sys.argv): Moved from finish_python_initialization.
(pretty_printers): Moved from _initialize_python.
(packages, auto_load_packages): New list and function replacing
module_dict and auto-loading code, using __file__ instead of
gdb.PYTHONDIR and reload/__import__ instead of exec.
(GdbSetPythonDirectory): Replacing function of the same name
from finish_python_initialization, using __file__ instead of
gdb.PYTHONDIR and reload/__import__ instead of exec,
as well as call auto_load_packages.
* python/py-prettyprint.c (find_pretty_printer_from_gdb): Check
gdb_python_module and not gdb_module.
* python/python-internal.h (gdb_python_module): Declare.
* python/python.c (gdb_python_module): New global.
(before_prompt_hook): Check gdb_python_module and not gdb_module.
(_initialize_python): Rename gdb module to _gdb.
Remove gdb.PYTHONDIR.
Move gdb.pretty_printer to lib/gdb/__init__.py.
(finish_python_initialization): Move Python code to
lib/gdb/__init__.py; instead, set up sys.path and import gdb into
__main__.

gdb/testsuite/ChangeLog:

2012-09-02 Khoo Yit Phang <khooyp@cs.umd.edu>

Refactor Python "gdb" module into a proper Python package, by
introducing a new "_gdb" module for code implemented in C.
* testsuite/gdb.python/python.exp (Test stderr location): Update
module location of GDB-specific sys.stderr.
(Test stdout location): Ditto for sys.stdout.

diff --git a/gdb/python/lib/gdb/__init__.py b/gdb/python/lib/gdb/__init__.py
--- a/gdb/python/lib/gdb/__init__.py
+++ b/gdb/python/lib/gdb/__init__.py
@@ -14,28 +14,110 @@
# along with this program. If not, see <http://www.gnu.org/licenses/>.

import traceback
+import os
+import sys
+import _gdb
+
+from _gdb import *
+
+class GdbOutputFile:
+ def close(self):
+ # Do nothing.
+ return None
+
+ def isatty(self):
+ return False
+
+ def write(self, s):
+ write(s, stream=STDOUT)
+
+ def writelines(self, iterable):
+ for line in iterable:
+ self.write(line)
+
+ def flush(self):
+ flush()
+
+sys.stdout = GdbOutputFile()
+
+class GdbOutputErrorFile:
+ def close(self):
+ # Do nothing.
+ return None
+
+ def isatty(self):
+ return False
+
+ def write(self, s):
+ write(s, stream=STDERR)
+
+ def writelines(self, iterable):
+ for line in iterable:
+ self.write(line)
+
+ def flush(self):
+ flush()
+
+sys.stderr = GdbOutputErrorFile()
+
+# Default prompt hook does nothing.
+prompt_hook = None
+
+# Ensure that sys.argv is set to something.
+# We do not use PySys_SetArgvEx because it did not appear until 2.6.6.
+sys.argv = ['']
+
+# Initial pretty printers.
+pretty_printers = []

# Auto-load all functions/commands.

-# Modules to auto-load, and the paths where those modules exist.
+# Packages to auto-load.

-module_dict = {
- 'gdb.function': os.path.join(gdb.PYTHONDIR, 'gdb', 'function'),
- 'gdb.command': os.path.join(gdb.PYTHONDIR, 'gdb', 'command')
-}
+packages = [
+ 'function',
+ 'command'
+]

-# Iterate the dictionary, collating the Python files in each module
+# pkgutil.iter_modules is not available prior to Python 2.6. Instead,
+# manually iterate the list, collating the Python files in each module
# path. Construct the module name, and import.

-for module, location in module_dict.iteritems():
- if os.path.exists(location):
- py_files = filter(lambda x: x.endswith('.py') and x != '__init__.py',
- os.listdir(location))
+def auto_load_packages():
+ for package in packages:
+ location = os.path.join(os.path.dirname(__file__), package)
+ if os.path.exists(location):
+ py_files = filter(lambda x: x.endswith('.py') and x != '__init__.py',
+ os.listdir(location))

- for py_file in py_files:
- # Construct from foo.py, gdb.module.foo
- py_file = module + '.' + py_file[:-3]
- try:
- exec('import ' + py_file)
- except:
- print >> sys.stderr, traceback.format_exc()
+ for py_file in py_files:
+ # Construct from foo.py, gdb.module.foo
+ modname = "%s.%s.%s" % ( __name__, package, py_file[:-3] )
+ try:
+ if modname in sys.modules:
+ # reload modules with duplicate names
+ reload(__import__(modname))
+ else:
+ __import__(modname)
+ except:
+ print >> sys.stderr, traceback.format_exc()
+
+auto_load_packages()
+
+def GdbSetPythonDirectory(dir):
+ """Update sys.path, reload gdb and auto-load packages."""
+
+ # Packages are identified by __init__.{py,pyc,pyd,pyo}.
+ if os.path.basename(__file__).startswith('__init__.py'):
+ old_dir = os.path.dirname(os.path.dirname(__file__))
+ else:
+ old_dir = os.path.dirname(__file__)
+
+ if old_dir in sys.path:
+ sys.path.remove(old_dir)
+ sys.path.insert(0, dir)
+
+ # note that reload overwrites the gdb module without deleting existing
+ # attributes
+ reload(__import__(__name__))
+ auto_load_packages()
diff --git a/gdb/python/py-prettyprint.c b/gdb/python/py-prettyprint.c
--- a/gdb/python/py-prettyprint.c
+++ b/gdb/python/py-prettyprint.c
@@ -162,9 +162,10 @@
PyObject *function;

/* Fetch the global pretty printer list. */
- if (! PyObject_HasAttrString (gdb_module, "pretty_printers"))
+ if (gdb_python_module == NULL
+ || ! PyObject_HasAttrString (gdb_python_module, "pretty_printers"))
Py_RETURN_NONE;
- pp_list = PyObject_GetAttrString (gdb_module, "pretty_printers");
+ pp_list = PyObject_GetAttrString (gdb_python_module, "pretty_printers");
if (pp_list == NULL || ! PyList_Check (pp_list))
{
Py_XDECREF (pp_list);
diff --git a/gdb/python/python-internal.h b/gdb/python/python-internal.h
--- a/gdb/python/python-internal.h
+++ b/gdb/python/python-internal.h
@@ -114,6 +114,7 @@
struct inferior;

extern PyObject *gdb_module;
+extern PyObject *gdb_python_module;
extern PyTypeObject value_object_type;
extern PyTypeObject block_object_type;
extern PyTypeObject symbol_object_type;
diff --git a/gdb/python/python.c b/gdb/python/python.c
--- a/gdb/python/python.c
+++ b/gdb/python/python.c
@@ -74,6 +74,7 @@
static PyMethodDef GdbMethods[];

PyObject *gdb_module;
+PyObject *gdb_python_module;

/* Some string constants we may wish to use. */
PyObject *gdbpy_to_string_cst;
@@ -879,11 +880,11 @@

cleanup = ensure_python_env (get_current_arch (), current_language);

- if (PyObject_HasAttrString (gdb_module, "prompt_hook"))
+ if (gdb_python_module && PyObject_HasAttrString (gdb_python_module, "prompt_hook"))
{
PyObject *hook;

- hook = PyObject_GetAttrString (gdb_module, "prompt_hook");
+ hook = PyObject_GetAttrString (gdb_python_module, "prompt_hook");
if (hook == NULL)
goto fail;

@@ -1358,7 +1359,7 @@
Py_Initialize ();
PyEval_InitThreads ();

- gdb_module = Py_InitModule ("gdb", GdbMethods);
+ gdb_module = Py_InitModule ("_gdb", GdbMethods);

/* The casts to (char*) are for python 2.4. */
PyModule_AddStringConstant (gdb_module, "VERSION", (char*) version);
@@ -1370,17 +1371,6 @@
PyModule_AddIntConstant (gdb_module, "STDOUT", 0);
PyModule_AddIntConstant (gdb_module, "STDERR", 1);
PyModule_AddIntConstant (gdb_module, "STDLOG", 2);
-
- /* gdb.parameter ("data-directory") doesn't necessarily exist when the python
- script below is run (depending on order of _initialize_* functions).
- Define the initial value of gdb.PYTHONDIR here. */
- {
- char *gdb_pythondir;
-
- gdb_pythondir = concat (gdb_datadir, SLASH_STRING, "python", NULL);
- PyModule_AddStringConstant (gdb_module, "PYTHONDIR", gdb_pythondir);
- xfree (gdb_pythondir);
- }

gdbpy_gdb_error = PyErr_NewException ("gdb.error", PyExc_RuntimeError, NULL);
PyModule_AddObject (gdb_module, "error", gdbpy_gdb_error);
@@ -1425,9 +1415,6 @@

observer_attach_before_prompt (before_prompt_hook);

- PyRun_SimpleString ("import gdb");
- PyRun_SimpleString ("gdb.pretty_printers = []");
-
gdbpy_to_string_cst = PyString_FromString ("to_string");
gdbpy_children_cst = PyString_FromString ("children");
gdbpy_display_hint_cst = PyString_FromString ("display_hint");
@@ -1452,88 +1439,67 @@
void
finish_python_initialization (void)
{
+ PyObject *m;
+ char *gdb_pythondir;
+ PyObject *sys_path;
struct cleanup *cleanup;

cleanup = ensure_python_env (get_current_arch (), current_language);

- PyRun_SimpleString ("\
-import os\n\
-import sys\n\
-\n\
-class GdbOutputFile:\n\
- def close(self):\n\
- # Do nothing.\n\
- return None\n\
-\n\
- def isatty(self):\n\
- return False\n\
-\n\
- def write(self, s):\n\
- gdb.write(s, stream=gdb.STDOUT)\n \
-\n\
- def writelines(self, iterable):\n\
- for line in iterable:\n\
- self.write(line)\n\
-\n\
- def flush(self):\n\
- gdb.flush()\n\
-\n\
-sys.stdout = GdbOutputFile()\n\
-\n\
-class GdbOutputErrorFile:\n\
- def close(self):\n\
- # Do nothing.\n\
- return None\n\
-\n\
- def isatty(self):\n\
- return False\n\
-\n\
- def write(self, s):\n\
- gdb.write(s, stream=gdb.STDERR)\n \
-\n\
- def writelines(self, iterable):\n\
- for line in iterable:\n\
- self.write(line)\n \
-\n\
- def flush(self):\n\
- gdb.flush()\n\
-\n\
-sys.stderr = GdbOutputErrorFile()\n\
-\n\
-# Ideally this would live in the gdb module, but it's intentionally written\n\
-# in python, and we need this to bootstrap the gdb module.\n\
-\n\
-def GdbSetPythonDirectory (dir):\n\
- \"Set gdb.PYTHONDIR and update sys.path,etc.\"\n\
- old_dir = gdb.PYTHONDIR\n\
- gdb.PYTHONDIR = dir\n\
- # GDB's python scripts are stored inside gdb.PYTHONDIR. So insert\n\
- # that directory name at the start of sys.path to allow the Python\n\
- # interpreter to find them.\n\
- if old_dir in sys.path:\n\
- sys.path.remove (old_dir)\n\
- sys.path.insert (0, gdb.PYTHONDIR)\n\
-\n\
- # Tell python where to find submodules of gdb.\n\
- gdb.__path__ = [os.path.join (gdb.PYTHONDIR, 'gdb')]\n\
-\n\
- # The gdb module is implemented in C rather than in Python. As a result,\n\
- # the associated __init.py__ script is not not executed by default when\n\
- # the gdb module gets imported. Execute that script manually if it\n\
- # exists.\n\
- ipy = os.path.join (gdb.PYTHONDIR, 'gdb', '__init__.py')\n\
- if os.path.exists (ipy):\n\
- execfile (ipy)\n\
-\n\
-# Install the default gdb.PYTHONDIR.\n\
-GdbSetPythonDirectory (gdb.PYTHONDIR)\n\
-# Default prompt hook does nothing.\n\
-prompt_hook = None\n\
-# Ensure that sys.argv is set to something.\n\
-# We do not use PySys_SetArgvEx because it did not appear until 2.6.6.\n\
-sys.argv = ['']\n\
-");
+ /* Add the initial data-directory to sys.path. */

+ gdb_pythondir = concat (gdb_datadir, SLASH_STRING, "python", NULL);
+ make_cleanup (xfree, gdb_pythondir);
+
+ sys_path = PySys_GetObject ("path");
+
+ if (sys_path && PyList_Check (sys_path))
+ {
+ PyObject *pythondir;
+ int err;
+
+ pythondir = PyString_FromString (gdb_pythondir);
+ if (pythondir == NULL)
+ goto fail;
+
+ err = PyList_Insert (sys_path, 0, pythondir);
+ if (err)
+ goto fail;
+
+ Py_DECREF (pythondir);
+ }
+ else
+ PySys_SetPath (gdb_pythondir);
+
+ /* Import the gdb module to finish the initialization, and
+ add it to __main__ for convenience. */
+ m = PyImport_AddModule ("__main__");
+ if (m == NULL)
+ goto fail;
+
+ gdb_python_module = PyImport_ImportModule ("gdb");
+ if (gdb_python_module == NULL)
+ {
+ gdbpy_print_stack ();
+ warning (_("Could not load the Python gdb module from `%s'."),
+ gdb_pythondir);
+ warning (_("Limited Python support is available from the _gdb module."));
+ do_cleanups (cleanup);
+ return;
+ }
+
+ if (PyModule_AddObject (m, "gdb", gdb_python_module))
+ goto fail;
+
+ /* Keep the reference to gdb_python_module since it is in a global
+ variable. */
+
+ do_cleanups (cleanup);
+ return;
+
+ fail:
+ gdbpy_print_stack ();
+ warning (_("internal error: Unhandled Python exception"));
do_cleanups (cleanup);
}

diff --git a/gdb/testsuite/gdb.python/python.exp b/gdb/testsuite/gdb.python/python.exp
--- a/gdb/testsuite/gdb.python/python.exp
+++ b/gdb/testsuite/gdb.python/python.exp
@@ -183,8 +183,8 @@
gdb_test {python print symtab[0]} ",func2" "stop at comma in linespec"

# gdb.write
-gdb_test "python print sys.stderr" ".*__main__.GdbOutputErrorFile instance at.*" "Test stderr location"
-gdb_test "python print sys.stdout" ".*__main__.GdbOutputFile instance at.*" "Test stdout location"
+gdb_test "python print sys.stderr" ".*gdb.GdbOutputErrorFile instance at.*" "Test stderr location"
+gdb_test "python print sys.stdout" ".*gdb.GdbOutputFile instance at.*" "Test stdout location"
gdb_test "python gdb.write(\"Foo\\n\")" "Foo" "Test default write"
gdb_test "python gdb.write(\"Error stream\\n\", stream=gdb.STDERR)" "Error stream" "Test stderr write"
gdb_test "python gdb.write(\"Normal stream\\n\", stream=gdb.STDOUT)" "Normal stream" "Test stdout write"