[RFC] Make python/lib/gdb and submodules proper Python modules

[RFC] Make python/lib/gdb and submodules proper Python modules

[Khoo Yit Phang]

[-- Attachment #1: Type: text/plain, Size: 1292 bytes --]

Hi,

I found it strange that python/lib/gdb and its submodules aren't truly Python modules (e.g., the python/lib/gdb/__init__.py had to explicitly refer to the gdb module), so I've put together a patch that makes them proper Python modules.

This patch works by splitting out the gdb module into _gdb for the native code extensions (as recommended by PEP8), setting up sys.path in finish_python_initialization and finally loading the gdb module, as well as switching from using exec to reload/__import__ (which updates/creates Python modules). Also, I've made a few other related changes: I removed gdb.PYTHONDIR which isn't necessary anymore since the same information is available via gdb.__file__, and moved the Python code from finish_python_initialization to python/lib/gdb/__init__.py.

One caveat about this patch is that the data-directory must contain the gdb module, otherwise Python will not be fully initialized (with a warning) and Python support will be very limited (only the _gdb module will be available).

I've ran the testsuite on Ubuntu 11.04 with Python 2.7.1 and Python 2.4.6 and found no regressions (you may need the patch I recently posted that adds -nx to a few tests to avoid spurious results).

What do you think?

Thanks,

Yit
September 2, 2012


[-- Attachment #2: python-gdb-native-module.txt --]
[-- Type: text/plain, Size: 13848 bytes --]

Refactor Python "gdb" module into a proper Python package, by introducing a new "_gdb" module for code implemented in C.

gdb/ChangeLog:

2012-09-02  Khoo Yit Phang  <khooyp@cs.umd.edu>

	Refactor Python "gdb" module into a proper Python package, by
	introducing a new "_gdb" module for code implemented in C, and
	using reload/__import__ instead of exec.
	* python/lib/gdb/__init__.py: Import * from _gdb.
	(GdbOutputFile, sys.stdout, GdbOutputErrorFile, sys.stderr,
	prompt_hook, sys.argv): Moved from finish_python_initialization.
	(pretty_printers): Moved from _initialize_python.
	(packages, auto_load_packages): New list and function replacing
	module_dict and auto-loading code, using __file__ instead of
	gdb.PYTHONDIR and reload/__import__ instead of exec.
	(GdbSetPythonDirectory): Replacing function of the same name
	from finish_python_initialization, using __file__ instead of
	gdb.PYTHONDIR and reload/__import__ instead of exec,
	as well as call auto_load_packages.
	* python/py-prettyprint.c (find_pretty_printer_from_gdb): Check
	gdb_python_module and not gdb_module.
	* python/python-internal.h (gdb_python_module): Declare.
	* python/python.c (gdb_python_module): New global.
	(before_prompt_hook): Check gdb_python_module and not gdb_module.
	(_initialize_python): Rename gdb module to _gdb.
	Remove gdb.PYTHONDIR.
	Move gdb.pretty_printer to lib/gdb/__init__.py.
	(finish_python_initialization): Move Python code to
	lib/gdb/__init__.py; instead, set up sys.path and import gdb into
	__main__.

gdb/testsuite/ChangeLog:

2012-09-02  Khoo Yit Phang  <khooyp@cs.umd.edu>

	Refactor Python "gdb" module into a proper Python package, by
	introducing a new "_gdb" module for code implemented in C.
	* testsuite/gdb.python/python.exp (Test stderr location): Update
	module location of GDB-specific sys.stderr.
	(Test stdout location): Ditto for sys.stdout.


diff --git a/gdb/python/lib/gdb/__init__.py b/gdb/python/lib/gdb/__init__.py
--- a/gdb/python/lib/gdb/__init__.py
+++ b/gdb/python/lib/gdb/__init__.py
@@ -14,28 +14,110 @@
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 
 import traceback
+import os
+import sys
+import _gdb
+
+from _gdb import *
+
+class GdbOutputFile:
+  def close(self):
+    # Do nothing.
+    return None
+
+  def isatty(self):
+    return False
+
+  def write(self, s):
+    write(s, stream=STDOUT)
+
+  def writelines(self, iterable):
+    for line in iterable:
+      self.write(line)
+
+  def flush(self):
+    flush()
+
+sys.stdout = GdbOutputFile()
+
+class GdbOutputErrorFile:
+  def close(self):
+    # Do nothing.
+    return None
+
+  def isatty(self):
+    return False
+
+  def write(self, s):
+    write(s, stream=STDERR)
+
+  def writelines(self, iterable):
+    for line in iterable:
+      self.write(line)
+
+  def flush(self):
+    flush()
+
+sys.stderr = GdbOutputErrorFile()
+
+# Default prompt hook does nothing.
+prompt_hook = None
+
+# Ensure that sys.argv is set to something.
+# We do not use PySys_SetArgvEx because it did not appear until 2.6.6.
+sys.argv = ['']
+
+# Initial pretty printers.
+pretty_printers = []
 
 # Auto-load all functions/commands.
 
-# Modules to auto-load, and the paths where those modules exist.
+# Packages to auto-load.
 
-module_dict = {
-  'gdb.function': os.path.join(gdb.PYTHONDIR, 'gdb', 'function'),
-  'gdb.command': os.path.join(gdb.PYTHONDIR, 'gdb', 'command')
-}
+packages = [
+  'function',
+  'command'
+]
 
-# Iterate the dictionary, collating the Python files in each module
+# pkgutil.iter_modules is not available prior to Python 2.6.  Instead,
+# manually iterate the list, collating the Python files in each module
 # path.  Construct the module name, and import.
 
-for module, location in module_dict.iteritems():
-  if os.path.exists(location):
-     py_files = filter(lambda x: x.endswith('.py') and x != '__init__.py',
-                       os.listdir(location))
+def auto_load_packages():
+  for package in packages:
+    location = os.path.join(os.path.dirname(__file__), package)
+    if os.path.exists(location):
+      py_files = filter(lambda x: x.endswith('.py') and x != '__init__.py',
+			os.listdir(location))
 
-     for py_file in py_files:
-       # Construct from foo.py, gdb.module.foo
-       py_file = module + '.' + py_file[:-3]
-       try:
-         exec('import ' + py_file)
-       except:
-         print >> sys.stderr, traceback.format_exc()
+      for py_file in py_files:
+	# Construct from foo.py, gdb.module.foo
+	modname = "%s.%s.%s" % ( __name__, package, py_file[:-3] )
+	try:
+	  if modname in sys.modules:
+	    # reload modules with duplicate names
+	    reload(__import__(modname))
+	  else:
+	    __import__(modname)
+	except:
+	  print >> sys.stderr, traceback.format_exc()
+
+auto_load_packages()
+
+def GdbSetPythonDirectory(dir):
+  """Update sys.path, reload gdb and auto-load packages."""
+
+  # Packages are identified by __init__.{py,pyc,pyd,pyo}.
+  if os.path.basename(__file__).startswith('__init__.py'):
+    old_dir = os.path.dirname(os.path.dirname(__file__))
+  else:
+    old_dir = os.path.dirname(__file__)
+
+  if old_dir in sys.path:
+    sys.path.remove(old_dir)
+  sys.path.insert(0, dir)
+
+  # note that reload overwrites the gdb module without deleting existing
+  # attributes
+  reload(__import__(__name__))
+  auto_load_packages()
diff --git a/gdb/python/py-prettyprint.c b/gdb/python/py-prettyprint.c
--- a/gdb/python/py-prettyprint.c
+++ b/gdb/python/py-prettyprint.c
@@ -162,9 +162,10 @@
   PyObject *function;
 
   /* Fetch the global pretty printer list.  */
-  if (! PyObject_HasAttrString (gdb_module, "pretty_printers"))
+  if (gdb_python_module == NULL
+      || ! PyObject_HasAttrString (gdb_python_module, "pretty_printers"))
     Py_RETURN_NONE;
-  pp_list = PyObject_GetAttrString (gdb_module, "pretty_printers");
+  pp_list = PyObject_GetAttrString (gdb_python_module, "pretty_printers");
   if (pp_list == NULL || ! PyList_Check (pp_list))
     {
       Py_XDECREF (pp_list);
diff --git a/gdb/python/python-internal.h b/gdb/python/python-internal.h
--- a/gdb/python/python-internal.h
+++ b/gdb/python/python-internal.h
@@ -114,6 +114,7 @@
 struct inferior;
 
 extern PyObject *gdb_module;
+extern PyObject *gdb_python_module;
 extern PyTypeObject value_object_type;
 extern PyTypeObject block_object_type;
 extern PyTypeObject symbol_object_type;
diff --git a/gdb/python/python.c b/gdb/python/python.c
--- a/gdb/python/python.c
+++ b/gdb/python/python.c
@@ -74,6 +74,7 @@
 static PyMethodDef GdbMethods[];
 
 PyObject *gdb_module;
+PyObject *gdb_python_module;
 
 /* Some string constants we may wish to use.  */
 PyObject *gdbpy_to_string_cst;
@@ -879,11 +880,11 @@
 
   cleanup = ensure_python_env (get_current_arch (), current_language);
 
-  if (PyObject_HasAttrString (gdb_module, "prompt_hook"))
+  if (gdb_python_module && PyObject_HasAttrString (gdb_python_module, "prompt_hook"))
     {
       PyObject *hook;
 
-      hook = PyObject_GetAttrString (gdb_module, "prompt_hook");
+      hook = PyObject_GetAttrString (gdb_python_module, "prompt_hook");
       if (hook == NULL)
 	goto fail;
 
@@ -1358,7 +1359,7 @@
   Py_Initialize ();
   PyEval_InitThreads ();
 
-  gdb_module = Py_InitModule ("gdb", GdbMethods);
+  gdb_module = Py_InitModule ("_gdb", GdbMethods);
 
   /* The casts to (char*) are for python 2.4.  */
   PyModule_AddStringConstant (gdb_module, "VERSION", (char*) version);
@@ -1370,17 +1371,6 @@
   PyModule_AddIntConstant (gdb_module, "STDOUT", 0);
   PyModule_AddIntConstant (gdb_module, "STDERR", 1);
   PyModule_AddIntConstant (gdb_module, "STDLOG", 2);
-  
-  /* gdb.parameter ("data-directory") doesn't necessarily exist when the python
-     script below is run (depending on order of _initialize_* functions).
-     Define the initial value of gdb.PYTHONDIR here.  */
-  {
-    char *gdb_pythondir;
-
-    gdb_pythondir = concat (gdb_datadir, SLASH_STRING, "python", NULL);
-    PyModule_AddStringConstant (gdb_module, "PYTHONDIR", gdb_pythondir);
-    xfree (gdb_pythondir);
-  }
 
   gdbpy_gdb_error = PyErr_NewException ("gdb.error", PyExc_RuntimeError, NULL);
   PyModule_AddObject (gdb_module, "error", gdbpy_gdb_error);
@@ -1425,9 +1415,6 @@
 
   observer_attach_before_prompt (before_prompt_hook);
 
-  PyRun_SimpleString ("import gdb");
-  PyRun_SimpleString ("gdb.pretty_printers = []");
-
   gdbpy_to_string_cst = PyString_FromString ("to_string");
   gdbpy_children_cst = PyString_FromString ("children");
   gdbpy_display_hint_cst = PyString_FromString ("display_hint");
@@ -1452,88 +1439,67 @@
 void
 finish_python_initialization (void)
 {
+  PyObject *m;
+  char *gdb_pythondir;
+  PyObject *sys_path;
   struct cleanup *cleanup;
 
   cleanup = ensure_python_env (get_current_arch (), current_language);
 
-  PyRun_SimpleString ("\
-import os\n\
-import sys\n\
-\n\
-class GdbOutputFile:\n\
-  def close(self):\n\
-    # Do nothing.\n\
-    return None\n\
-\n\
-  def isatty(self):\n\
-    return False\n\
-\n\
-  def write(self, s):\n\
-    gdb.write(s, stream=gdb.STDOUT)\n   \
-\n\
-  def writelines(self, iterable):\n\
-    for line in iterable:\n\
-      self.write(line)\n\
-\n\
-  def flush(self):\n\
-    gdb.flush()\n\
-\n\
-sys.stdout = GdbOutputFile()\n\
-\n\
-class GdbOutputErrorFile:\n\
-  def close(self):\n\
-    # Do nothing.\n\
-    return None\n\
-\n\
-  def isatty(self):\n\
-    return False\n\
-\n\
-  def write(self, s):\n\
-    gdb.write(s, stream=gdb.STDERR)\n		\
-\n\
-  def writelines(self, iterable):\n\
-    for line in iterable:\n\
-      self.write(line)\n \
-\n\
-  def flush(self):\n\
-    gdb.flush()\n\
-\n\
-sys.stderr = GdbOutputErrorFile()\n\
-\n\
-# Ideally this would live in the gdb module, but it's intentionally written\n\
-# in python, and we need this to bootstrap the gdb module.\n\
-\n\
-def GdbSetPythonDirectory (dir):\n\
-  \"Set gdb.PYTHONDIR and update sys.path,etc.\"\n\
-  old_dir = gdb.PYTHONDIR\n\
-  gdb.PYTHONDIR = dir\n\
-  # GDB's python scripts are stored inside gdb.PYTHONDIR.  So insert\n\
-  # that directory name at the start of sys.path to allow the Python\n\
-  # interpreter to find them.\n\
-  if old_dir in sys.path:\n\
-    sys.path.remove (old_dir)\n\
-  sys.path.insert (0, gdb.PYTHONDIR)\n\
-\n\
-  # Tell python where to find submodules of gdb.\n\
-  gdb.__path__ = [os.path.join (gdb.PYTHONDIR, 'gdb')]\n\
-\n\
-  # The gdb module is implemented in C rather than in Python.  As a result,\n\
-  # the associated __init.py__ script is not not executed by default when\n\
-  # the gdb module gets imported.  Execute that script manually if it\n\
-  # exists.\n\
-  ipy = os.path.join (gdb.PYTHONDIR, 'gdb', '__init__.py')\n\
-  if os.path.exists (ipy):\n\
-    execfile (ipy)\n\
-\n\
-# Install the default gdb.PYTHONDIR.\n\
-GdbSetPythonDirectory (gdb.PYTHONDIR)\n\
-# Default prompt hook does nothing.\n\
-prompt_hook = None\n\
-# Ensure that sys.argv is set to something.\n\
-# We do not use PySys_SetArgvEx because it did not appear until 2.6.6.\n\
-sys.argv = ['']\n\
-");
+  /* Add the initial data-directory to sys.path.  */
 
+  gdb_pythondir = concat (gdb_datadir, SLASH_STRING, "python", NULL);
+  make_cleanup (xfree, gdb_pythondir);
+
+  sys_path = PySys_GetObject ("path");
+
+  if (sys_path && PyList_Check (sys_path))
+    {
+      PyObject *pythondir;
+      int err;
+
+      pythondir = PyString_FromString (gdb_pythondir);
+      if (pythondir == NULL)
+	goto fail;
+
+      err = PyList_Insert (sys_path, 0, pythondir);
+      if (err)
+	goto fail;
+
+      Py_DECREF (pythondir);
+    }
+  else
+    PySys_SetPath (gdb_pythondir);
+
+  /* Import the gdb module to finish the initialization, and
+     add it to __main__ for convenience.  */
+  m = PyImport_AddModule ("__main__");
+  if (m == NULL)
+    goto fail;
+
+  gdb_python_module = PyImport_ImportModule ("gdb");
+  if (gdb_python_module == NULL)
+    {
+      gdbpy_print_stack ();
+      warning (_("Could not load the Python gdb module from `%s'."),
+		 gdb_pythondir);
+      warning (_("Limited Python support is available from the _gdb module."));
+      do_cleanups (cleanup);
+      return;
+    }
+
+  if (PyModule_AddObject (m, "gdb", gdb_python_module))
+    goto fail;
+
+  /* Keep the reference to gdb_python_module since it is in a global
+     variable.  */
+
+  do_cleanups (cleanup);
+  return;
+
+ fail:
+  gdbpy_print_stack ();
+  warning (_("internal error: Unhandled Python exception"));
   do_cleanups (cleanup);
 }
 
diff --git a/gdb/testsuite/gdb.python/python.exp b/gdb/testsuite/gdb.python/python.exp
--- a/gdb/testsuite/gdb.python/python.exp
+++ b/gdb/testsuite/gdb.python/python.exp
@@ -183,8 +183,8 @@
 gdb_test {python print symtab[0]} ",func2" "stop at comma in linespec"
 
 # gdb.write
-gdb_test "python print sys.stderr" ".*__main__.GdbOutputErrorFile instance at.*" "Test stderr location"
-gdb_test "python print sys.stdout" ".*__main__.GdbOutputFile instance at.*" "Test stdout location"
+gdb_test "python print sys.stderr" ".*gdb.GdbOutputErrorFile instance at.*" "Test stderr location"
+gdb_test "python print sys.stdout" ".*gdb.GdbOutputFile instance at.*" "Test stdout location"
 gdb_test "python gdb.write(\"Foo\\n\")" "Foo" "Test default write"
 gdb_test "python gdb.write(\"Error stream\\n\", stream=gdb.STDERR)" "Error stream" "Test stderr write"
 gdb_test "python gdb.write(\"Normal stream\\n\", stream=gdb.STDOUT)" "Normal stream" "Test stdout write"

Re: [RFC] Make python/lib/gdb and submodules proper Python modules

[Doug Evans]

Oh bother, sorry for the resend.

On Sun, Sep 2, 2012 at 8:44 PM, Khoo Yit Phang <khooyp@cs.umd.edu> wrote:
>
> Hi,
>
> I found it strange that python/lib/gdb and its submodules aren't truly
> Python modules (e.g., the python/lib/gdb/__init__.py had to explicitly refer
> to the gdb module), so I've put together a patch that makes them proper
> Python modules.
>
> This patch works by splitting out the gdb module into _gdb for the native
> code extensions (as recommended by PEP8), setting up sys.path in
> finish_python_initialization and finally loading the gdb module, as well as
> switching from using exec to reload/__import__ (which updates/creates Python
> modules). Also, I've made a few other related changes: I removed
> gdb.PYTHONDIR which isn't necessary anymore since the same information is
> available via gdb.__file__, and moved the Python code from
> finish_python_initialization to python/lib/gdb/__init__.py.
>
> One caveat about this patch is that the data-directory must contain the
> gdb module, otherwise Python will not be fully initialized (with a warning)
> and Python support will be very limited (only the _gdb module will be
> available).
>
> I've ran the testsuite on Ubuntu 11.04 with Python 2.7.1 and Python 2.4.6
> and found no regressions (you may need the patch I recently posted that adds
> -nx to a few tests to avoid spurious results).

Hi.
One thing that comes to mind is that, regardless of implementation
details, we can't easily get rid of gdb.PYTHONDIR because it's part of
the API gdb provides.
[grep PYTHONDIR doc/gdb.texinfo]

Re: [RFC] Make python/lib/gdb and submodules proper Python modules

[Khoo Yit Phang]

Hi,

(Sorry for the resend again!)

On Sep 7, 2012, at 2:18 AM, Doug Evans wrote:

> On Sun, Sep 2, 2012 at 8:44 PM, Khoo Yit Phang <khooyp@cs.umd.edu> wrote:
>> 
>> One caveat about this patch is that the data-directory must contain the
>> gdb module, otherwise Python will not be fully initialized (with a warning)
>> and Python support will be very limited (only the _gdb module will be
>> available).
> 
> One thing that comes to mind is that, regardless of implementation
> details, we can't easily get rid of gdb.PYTHONDIR because it's part of
> the API gdb provides.
> [grep PYTHONDIR doc/gdb.texinfo]

It's pretty easy to restore PYTHONDIR if necessary, by computing it from gdb.__file__. I'm more concerned about the caveat about requiring the initial data-directory for Python. Is it common to change the data-directory, other than for testing GDB?

Yit
September 7, 2012

Re: [RFC] Make python/lib/gdb and submodules proper Python modules

[Tom Tromey]

>>>>> "Yit" == Khoo Yit Phang <khooyp@cs.umd.edu> writes:

Yit> I found it strange that python/lib/gdb and its submodules aren't truly
Yit> Python modules (e.g., the python/lib/gdb/__init__.py had to explicitly
Yit> refer to the gdb module), so I've put together a patch that makes them
Yit> proper Python modules.

Seems reasonable to me.  Thanks.

Yit> Also, I've made a few other related
Yit> changes: I removed gdb.PYTHONDIR which isn't necessary anymore since
Yit> the same information is available via gdb.__file__, and moved the
Yit> Python code from finish_python_initialization to
Yit> python/lib/gdb/__init__.py.

I think it would be good to restore this, as Doug said.
I realize Python programs can adapt, but it is friendlier not to make
them adapt.

Yit> One caveat about this patch is that the data-directory must contain
Yit> the gdb module, otherwise Python will not be fully initialized (with a
Yit> warning) and Python support will be very limited (only the _gdb module
Yit> will be available).

I think this is probably ok.

Yit> +  if (gdb_python_module && PyObject_HasAttrString (gdb_python_module, "prompt_hook"))

This line should probably split before the "&&".

Tom

Re: [RFC] Make python/lib/gdb and submodules proper Python modules

[Khoo Yit Phang]

[-- Attachment #1: Type: text/plain, Size: 411 bytes --]

Hi Tom,

Thanks for the review. I've attached an updated patch that restores PYTHONDIR and fixes the formatting. Also, is there a particular preferred style for the Python code? Most of the code under gdb/python/lib/gdb seem to conform to PEP 8 (no tabs, 4-space indents, no space before parentheses) rather than the GNU style. So I've reformatted the "gdb" module to PEP 8 too.

Yit
September 12, 2012


[-- Attachment #2: python-gdb-native-module.txt --]
[-- Type: text/plain, Size: 14228 bytes --]

2012-09-12  Khoo Yit Phang  <khooyp@cs.umd.edu>

	Refactor Python "gdb" module into a proper Python package, by
	introducing a new "_gdb" module for code implemented in C, and
	using reload/__import__ instead of exec.
	* python/lib/gdb/__init__.py: Import * from _gdb.
	(GdbOutputFile, sys.stdout, GdbOutputErrorFile, sys.stderr,
	prompt_hook, sys.argv): Moved from finish_python_initialization.
	(pretty_printers, PYTHONDIR): Moved from _initialize_python.
	(packages, auto_load_packages): New list and function replacing
	module_dict and auto-loading code, using __file__ instead of
	gdb.PYTHONDIR and reload/__import__ instead of exec.
	(GdbSetPythonDirectory): Replacing function of the same name
	from finish_python_initialization, using reload/__import__ instead
	of exec, as well as call auto_load_packages.
	* python/py-prettyprint.c (find_pretty_printer_from_gdb): Check
	gdb_python_module and not gdb_module.
	* python/python-internal.h (gdb_python_module): Declare.
	* python/python.c (gdb_python_module): New global.
	(before_prompt_hook): Check gdb_python_module and not gdb_module.
	(_initialize_python): Rename gdb module to _gdb.
	Move gdb.PYTHONDIR and gdb.pretty_printer to lib/gdb/__init__.py.
	(finish_python_initialization): Move Python code to
	lib/gdb/__init__.py; instead, set up sys.path and import gdb into
	__main__.

diff --git a/gdb/python/lib/gdb/__init__.py b/gdb/python/lib/gdb/__init__.py
--- a/gdb/python/lib/gdb/__init__.py
+++ b/gdb/python/lib/gdb/__init__.py
@@ -14,28 +14,113 @@
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 
 import traceback
+import os
+import sys
+import _gdb
+
+from _gdb import *
+
+class GdbOutputFile:
+    def close(self):
+        # Do nothing.
+        return None
+
+    def isatty(self):
+        return False
+
+    def write(self, s):
+        write(s, stream=STDOUT)
+
+    def writelines(self, iterable):
+        for line in iterable:
+            self.write(line)
+
+    def flush(self):
+        flush()
+
+sys.stdout = GdbOutputFile()
+
+class GdbOutputErrorFile:
+    def close(self):
+        # Do nothing.
+        return None
+
+    def isatty(self):
+        return False
+
+    def write(self, s):
+        write(s, stream=STDERR)
+
+    def writelines(self, iterable):
+        for line in iterable:
+            self.write(line)
+
+    def flush(self):
+        flush()
+
+sys.stderr = GdbOutputErrorFile()
+
+# Default prompt hook does nothing.
+prompt_hook = None
+
+# Ensure that sys.argv is set to something.
+# We do not use PySys_SetArgvEx because it did not appear until 2.6.6.
+sys.argv = ['']
+
+# Initial pretty printers.
+pretty_printers = []
+
+# Convenience variable to GDB's python directory
+PYTHONDIR = os.path.dirname(os.path.dirname(__file__))
 
 # Auto-load all functions/commands.
 
-# Modules to auto-load, and the paths where those modules exist.
+# Packages to auto-load.
 
-module_dict = {
-  'gdb.function': os.path.join(gdb.PYTHONDIR, 'gdb', 'function'),
-  'gdb.command': os.path.join(gdb.PYTHONDIR, 'gdb', 'command')
-}
+packages = [
+    'function',
+    'command'
+]
 
-# Iterate the dictionary, collating the Python files in each module
+# pkgutil.iter_modules is not available prior to Python 2.6.  Instead,
+# manually iterate the list, collating the Python files in each module
 # path.  Construct the module name, and import.
 
-for module, location in module_dict.iteritems():
-  if os.path.exists(location):
-     py_files = filter(lambda x: x.endswith('.py') and x != '__init__.py',
-                       os.listdir(location))
+def auto_load_packages():
+    for package in packages:
+        location = os.path.join(os.path.dirname(__file__), package)
+        if os.path.exists(location):
+            py_files = filter(lambda x: x.endswith('.py')
+                                        and x != '__init__.py',
+                              os.listdir(location))
 
-     for py_file in py_files:
-       # Construct from foo.py, gdb.module.foo
-       py_file = module + '.' + py_file[:-3]
-       try:
-         exec('import ' + py_file)
-       except:
-         print >> sys.stderr, traceback.format_exc()
+            for py_file in py_files:
+                # Construct from foo.py, gdb.module.foo
+                modname = "%s.%s.%s" % ( __name__, package, py_file[:-3] )
+                try:
+                    if modname in sys.modules:
+                        # reload modules with duplicate names
+                        reload(__import__(modname))
+                    else:
+                        __import__(modname)
+                except:
+                    print >> sys.stderr, traceback.format_exc()
+
+auto_load_packages()
+
+def GdbSetPythonDirectory(dir):
+    """Update sys.path, reload gdb and auto-load packages."""
+    global PYTHONDIR
+
+    try:
+        sys.path.remove(PYTHONDIR)
+    except ValueError:
+        pass
+    sys.path.insert(0, dir)
+
+    PYTHONDIR = dir
+
+    # note that reload overwrites the gdb module without deleting existing
+    # attributes
+    reload(__import__(__name__))
+    auto_load_packages()
diff --git a/gdb/python/py-prettyprint.c b/gdb/python/py-prettyprint.c
--- a/gdb/python/py-prettyprint.c
+++ b/gdb/python/py-prettyprint.c
@@ -162,9 +162,10 @@
   PyObject *function;
 
   /* Fetch the global pretty printer list.  */
-  if (! PyObject_HasAttrString (gdb_module, "pretty_printers"))
+  if (gdb_python_module == NULL
+      || ! PyObject_HasAttrString (gdb_python_module, "pretty_printers"))
     Py_RETURN_NONE;
-  pp_list = PyObject_GetAttrString (gdb_module, "pretty_printers");
+  pp_list = PyObject_GetAttrString (gdb_python_module, "pretty_printers");
   if (pp_list == NULL || ! PyList_Check (pp_list))
     {
       Py_XDECREF (pp_list);
diff --git a/gdb/python/python-internal.h b/gdb/python/python-internal.h
--- a/gdb/python/python-internal.h
+++ b/gdb/python/python-internal.h
@@ -114,6 +114,7 @@
 struct inferior;
 
 extern PyObject *gdb_module;
+extern PyObject *gdb_python_module;
 extern PyTypeObject value_object_type;
 extern PyTypeObject block_object_type;
 extern PyTypeObject symbol_object_type;
diff --git a/gdb/python/python.c b/gdb/python/python.c
--- a/gdb/python/python.c
+++ b/gdb/python/python.c
@@ -74,6 +74,7 @@
 static PyMethodDef GdbMethods[];
 
 PyObject *gdb_module;
+PyObject *gdb_python_module;
 
 /* Some string constants we may wish to use.  */
 PyObject *gdbpy_to_string_cst;
@@ -879,11 +880,12 @@
 
   cleanup = ensure_python_env (get_current_arch (), current_language);
 
-  if (PyObject_HasAttrString (gdb_module, "prompt_hook"))
+  if (gdb_python_module
+      && PyObject_HasAttrString (gdb_python_module, "prompt_hook"))
     {
       PyObject *hook;
 
-      hook = PyObject_GetAttrString (gdb_module, "prompt_hook");
+      hook = PyObject_GetAttrString (gdb_python_module, "prompt_hook");
       if (hook == NULL)
 	goto fail;
 
@@ -1358,7 +1360,7 @@
   Py_Initialize ();
   PyEval_InitThreads ();
 
-  gdb_module = Py_InitModule ("gdb", GdbMethods);
+  gdb_module = Py_InitModule ("_gdb", GdbMethods);
 
   /* The casts to (char*) are for python 2.4.  */
   PyModule_AddStringConstant (gdb_module, "VERSION", (char*) version);
@@ -1370,17 +1372,6 @@
   PyModule_AddIntConstant (gdb_module, "STDOUT", 0);
   PyModule_AddIntConstant (gdb_module, "STDERR", 1);
   PyModule_AddIntConstant (gdb_module, "STDLOG", 2);
-  
-  /* gdb.parameter ("data-directory") doesn't necessarily exist when the python
-     script below is run (depending on order of _initialize_* functions).
-     Define the initial value of gdb.PYTHONDIR here.  */
-  {
-    char *gdb_pythondir;
-
-    gdb_pythondir = concat (gdb_datadir, SLASH_STRING, "python", NULL);
-    PyModule_AddStringConstant (gdb_module, "PYTHONDIR", gdb_pythondir);
-    xfree (gdb_pythondir);
-  }
 
   gdbpy_gdb_error = PyErr_NewException ("gdb.error", PyExc_RuntimeError, NULL);
   PyModule_AddObject (gdb_module, "error", gdbpy_gdb_error);
@@ -1425,9 +1416,6 @@
 
   observer_attach_before_prompt (before_prompt_hook);
 
-  PyRun_SimpleString ("import gdb");
-  PyRun_SimpleString ("gdb.pretty_printers = []");
-
   gdbpy_to_string_cst = PyString_FromString ("to_string");
   gdbpy_children_cst = PyString_FromString ("children");
   gdbpy_display_hint_cst = PyString_FromString ("display_hint");
@@ -1452,88 +1440,67 @@
 void
 finish_python_initialization (void)
 {
+  PyObject *m;
+  char *gdb_pythondir;
+  PyObject *sys_path;
   struct cleanup *cleanup;
 
   cleanup = ensure_python_env (get_current_arch (), current_language);
 
-  PyRun_SimpleString ("\
-import os\n\
-import sys\n\
-\n\
-class GdbOutputFile:\n\
-  def close(self):\n\
-    # Do nothing.\n\
-    return None\n\
-\n\
-  def isatty(self):\n\
-    return False\n\
-\n\
-  def write(self, s):\n\
-    gdb.write(s, stream=gdb.STDOUT)\n   \
-\n\
-  def writelines(self, iterable):\n\
-    for line in iterable:\n\
-      self.write(line)\n\
-\n\
-  def flush(self):\n\
-    gdb.flush()\n\
-\n\
-sys.stdout = GdbOutputFile()\n\
-\n\
-class GdbOutputErrorFile:\n\
-  def close(self):\n\
-    # Do nothing.\n\
-    return None\n\
-\n\
-  def isatty(self):\n\
-    return False\n\
-\n\
-  def write(self, s):\n\
-    gdb.write(s, stream=gdb.STDERR)\n		\
-\n\
-  def writelines(self, iterable):\n\
-    for line in iterable:\n\
-      self.write(line)\n \
-\n\
-  def flush(self):\n\
-    gdb.flush()\n\
-\n\
-sys.stderr = GdbOutputErrorFile()\n\
-\n\
-# Ideally this would live in the gdb module, but it's intentionally written\n\
-# in python, and we need this to bootstrap the gdb module.\n\
-\n\
-def GdbSetPythonDirectory (dir):\n\
-  \"Set gdb.PYTHONDIR and update sys.path,etc.\"\n\
-  old_dir = gdb.PYTHONDIR\n\
-  gdb.PYTHONDIR = dir\n\
-  # GDB's python scripts are stored inside gdb.PYTHONDIR.  So insert\n\
-  # that directory name at the start of sys.path to allow the Python\n\
-  # interpreter to find them.\n\
-  if old_dir in sys.path:\n\
-    sys.path.remove (old_dir)\n\
-  sys.path.insert (0, gdb.PYTHONDIR)\n\
-\n\
-  # Tell python where to find submodules of gdb.\n\
-  gdb.__path__ = [os.path.join (gdb.PYTHONDIR, 'gdb')]\n\
-\n\
-  # The gdb module is implemented in C rather than in Python.  As a result,\n\
-  # the associated __init.py__ script is not not executed by default when\n\
-  # the gdb module gets imported.  Execute that script manually if it\n\
-  # exists.\n\
-  ipy = os.path.join (gdb.PYTHONDIR, 'gdb', '__init__.py')\n\
-  if os.path.exists (ipy):\n\
-    execfile (ipy)\n\
-\n\
-# Install the default gdb.PYTHONDIR.\n\
-GdbSetPythonDirectory (gdb.PYTHONDIR)\n\
-# Default prompt hook does nothing.\n\
-prompt_hook = None\n\
-# Ensure that sys.argv is set to something.\n\
-# We do not use PySys_SetArgvEx because it did not appear until 2.6.6.\n\
-sys.argv = ['']\n\
-");
+  /* Add the initial data-directory to sys.path.  */
 
+  gdb_pythondir = concat (gdb_datadir, SLASH_STRING, "python", NULL);
+  make_cleanup (xfree, gdb_pythondir);
+
+  sys_path = PySys_GetObject ("path");
+
+  if (sys_path && PyList_Check (sys_path))
+    {
+      PyObject *pythondir;
+      int err;
+
+      pythondir = PyString_FromString (gdb_pythondir);
+      if (pythondir == NULL)
+	goto fail;
+
+      err = PyList_Insert (sys_path, 0, pythondir);
+      if (err)
+	goto fail;
+
+      Py_DECREF (pythondir);
+    }
+  else
+    PySys_SetPath (gdb_pythondir);
+
+  /* Import the gdb module to finish the initialization, and
+     add it to __main__ for convenience.  */
+  m = PyImport_AddModule ("__main__");
+  if (m == NULL)
+    goto fail;
+
+  gdb_python_module = PyImport_ImportModule ("gdb");
+  if (gdb_python_module == NULL)
+    {
+      gdbpy_print_stack ();
+      warning (_("Could not load the Python gdb module from `%s'."),
+		 gdb_pythondir);
+      warning (_("Limited Python support is available from the _gdb module."));
+      do_cleanups (cleanup);
+      return;
+    }
+
+  if (PyModule_AddObject (m, "gdb", gdb_python_module))
+    goto fail;
+
+  /* Keep the reference to gdb_python_module since it is in a global
+     variable.  */
+
+  do_cleanups (cleanup);
+  return;
+
+ fail:
+  gdbpy_print_stack ();
+  warning (_("internal error: Unhandled Python exception"));
   do_cleanups (cleanup);
 }
 
diff --git a/gdb/testsuite/ChangeLog b/gdb/testsuite/ChangeLog
--- a/gdb/testsuite/ChangeLog
+++ b/gdb/testsuite/ChangeLog
@@ -51,6 +51,14 @@
 
 	* gdb.base/info-macros.c: Fix whitespace.
 
+2012-09-02  Khoo Yit Phang  <khooyp@cs.umd.edu>
+
+	Refactor Python "gdb" module into a proper Python package, by
+	introducing a new "_gdb" module for code implemented in C.
+	* testsuite/gdb.python/python.exp (Test stderr location): Update
+	module location of GDB-specific sys.stderr.
+	(Test stdout location): Ditto for sys.stdout.
+
 2012-08-28  Pedro Alves  <palves@redhat.com>
 
 	PR gdb/14428
diff --git a/gdb/testsuite/gdb.python/python.exp b/gdb/testsuite/gdb.python/python.exp
--- a/gdb/testsuite/gdb.python/python.exp
+++ b/gdb/testsuite/gdb.python/python.exp
@@ -183,8 +183,8 @@
 gdb_test {python print symtab[0]} ",func2" "stop at comma in linespec"
 
 # gdb.write
-gdb_test "python print sys.stderr" ".*__main__.GdbOutputErrorFile instance at.*" "Test stderr location"
-gdb_test "python print sys.stdout" ".*__main__.GdbOutputFile instance at.*" "Test stdout location"
+gdb_test "python print sys.stderr" ".*gdb.GdbOutputErrorFile instance at.*" "Test stderr location"
+gdb_test "python print sys.stdout" ".*gdb.GdbOutputFile instance at.*" "Test stdout location"
 gdb_test "python gdb.write(\"Foo\\n\")" "Foo" "Test default write"
 gdb_test "python gdb.write(\"Error stream\\n\", stream=gdb.STDERR)" "Error stream" "Test stderr write"
 gdb_test "python gdb.write(\"Normal stream\\n\", stream=gdb.STDOUT)" "Normal stream" "Test stdout write"

Re: [RFC] Make python/lib/gdb and submodules proper Python modules

[Tom Tromey]

>>>>> "Yit" == Khoo Yit Phang <khooyp@cs.umd.edu> writes:

Yit> Thanks for the review. I've attached an updated patch that restores
Yit> PYTHONDIR and fixes the formatting. Also, is there a particular
Yit> preferred style for the Python code? Most of the code under
Yit> gdb/python/lib/gdb seem to conform to PEP 8 (no tabs, 4-space indents,
Yit> no space before parentheses) rather than the GNU style. So I've
Yit> reformatted the "gdb" module to PEP 8 too.

Yeah, I believe we agreed on following PEP 8 for the Python code.  Maybe
we did this after writing a chunk of the code, though, so it may not all
be consistent.

Your patch is ok.  Thanks for doing this.

Tom

Re: [RFC] Make python/lib/gdb and submodules proper Python modules

[Khoo Yit Phang]

Hi Tom,

On Sep 13, 2012, at 4:56 PM, Tom Tromey wrote:

>>>>>> "Yit" == Khoo Yit Phang <khooyp@cs.umd.edu> writes:
> 
> Yit> Thanks for the review. I've attached an updated patch that restores
> Yit> PYTHONDIR and fixes the formatting. Also, is there a particular
> Yit> preferred style for the Python code? Most of the code under
> Yit> gdb/python/lib/gdb seem to conform to PEP 8 (no tabs, 4-space indents,
> Yit> no space before parentheses) rather than the GNU style. So I've
> Yit> reformatted the "gdb" module to PEP 8 too.
> 
> Yeah, I believe we agreed on following PEP 8 for the Python code.  Maybe
> we did this after writing a chunk of the code, though, so it may not all
> be consistent.
> 
> Your patch is ok.  Thanks for doing this.

Thanks, I've checked in the patch.

Yit
September 13, 2012

Regression for --batch [Re: [RFC] Make python/lib/gdb and submodules proper Python modules]

[Jan Kratochvil]

Hi Yit,

On Thu, 13 Sep 2012 23:51:08 +0200, Khoo Yit Phang wrote:
> Thanks, I've checked in the patch.

$ ./gdb --batch
Python Exception <type 'exceptions.NameError'> name 'os' is not defined: 

warning: Could not load the Python gdb module from `/usr/share/gdb/python'.

warning: Limited Python support is available from the _gdb module.



Regards,
Jan

Re: Regression for --batch [Re: [RFC] Make python/lib/gdb and submodules proper Python modules]

[Khoo Yit Phang]

Hi Jan,

On Sep 14, 2012, at 1:47 AM, Jan Kratochvil wrote:

> Hi Yit,
> 
> On Thu, 13 Sep 2012 23:51:08 +0200, Khoo Yit Phang wrote:
>> Thanks, I've checked in the patch.
> 
> $ ./gdb --batch
> Python Exception <type 'exceptions.NameError'> name 'os' is not defined: 
> 
> warning: Could not load the Python gdb module from `/usr/share/gdb/python'.
> 
> warning: Limited Python support is available from the _gdb module.

Is ./gdb from a GDB build directory? If so, the error is expected; you'd need to run with -data-directory build-directory/gdb/data-directory, since much of the Python initialization has been moved into data-directory/python/gdb/__init__.py, and the old __init__.py is incompatible.

Yit
September 14, 2012

Re: Regression for --batch [Re: [RFC] Make python/lib/gdb and submodules proper Python modules]

[Jan Kratochvil]

On Fri, 14 Sep 2012 07:59:46 +0200, Khoo Yit Phang wrote:
> Is ./gdb from a GDB build directory? If so, the error is expected; you'd
> need to run with -data-directory build-directory/gdb/data-directory, since
> much of the Python initialization has been moved into
> data-directory/python/gdb/__init__.py, and the old __init__.py is
> incompatible.

In such case there is a need to update gdb/contrib/cc-with-tweaks.sh .

runtest CC_FOR_TARGET=/bin/sh\ $PWD/../contrib/cc-with-tweaks.sh\ -i\ gcc gdb.base/display.exp 
[...]
Running ./gdb.base/display.exp ...
gdb compile failed, Python Exception <type 'exceptions.NameError'> name 'os' is not defined: 
warning: Could not load the Python gdb module from `/usr/share/gdb/python'.
warning: Limited Python support is available from the _gdb module.
		=== gdb Summary ===
# of untested testcases		1


Regards,
Jan

Re: Regression for --batch [Re: [RFC] Make python/lib/gdb and submodules proper Python modules]

[Khoo Yit Phang]

Hi Jan,

On Sep 14, 2012, at 2:15 AM, Jan Kratochvil wrote:

> On Fri, 14 Sep 2012 07:59:46 +0200, Khoo Yit Phang wrote:
>> Is ./gdb from a GDB build directory? If so, the error is expected; you'd
>> need to run with -data-directory build-directory/gdb/data-directory, since
>> much of the Python initialization has been moved into
>> data-directory/python/gdb/__init__.py, and the old __init__.py is
>> incompatible.
> 
> In such case there is a need to update gdb/contrib/cc-with-tweaks.sh .
> 
> runtest CC_FOR_TARGET=/bin/sh\ $PWD/../contrib/cc-with-tweaks.sh\ -i\ gcc gdb.base/display.exp 
> [...]
> Running ./gdb.base/display.exp ...
> gdb compile failed, Python Exception <type 'exceptions.NameError'> name 'os' is not defined: 
> warning: Could not load the Python gdb module from `/usr/share/gdb/python'.
> warning: Limited Python support is available from the _gdb module.
> 		=== gdb Summary ===
> # of untested testcases		1

How do you run this? I can't seem to figure it out:

% cd build-directory/gdb/testsuite
% runtest CC_FOR_TARGET=/bin/sh\ $PWD/../../../gdb/contrib/cc-with-tweaks.sh\ -i\ gcc gdb.base/display.exp
WARNING: No tool specified
Test Run By khooyp on Fri Sep 14 02:24:12 2012
Native configuration is i686-pc-linux-gnu

		===  tests ===

Schedule of variations:
    unix

Running target unix
Using /usr/share/dejagnu/baseboards/unix.exp as board description file for target.
Using /usr/share/dejagnu/config/unix.exp as generic interface file for target.
WARNING: Couldn't find tool config file for unix, using default.

		===  Summary ===

% cd source-directory/gdb/testsuite
% runtest CC_FOR_TARGET=/bin/sh\ $PWD/../contrib/cc-with-tweaks.sh\ -i\ gcc gdb.base/display.exp
WARNING: No tool specified
Test Run By khooyp on Fri Sep 14 02:22:07 2012
Native configuration is i686-pc-linux-gnu

		===  tests ===

Schedule of variations:
    unix

Running target unix
Using /usr/share/dejagnu/baseboards/unix.exp as board description file for target.
Using /usr/share/dejagnu/config/unix.exp as generic interface file for target.
Using ./config/unix.exp as tool-and-target-specific interface file.
`site.exp' not found, run `make site.exp'!
% make site.exp
remake: *** No rule to make target `site.exp'.  Stop.


Yit
September 14, 2012

Re: Regression for --batch [Re: [RFC] Make python/lib/gdb and submodules proper Python modules]

[Jan Kratochvil]

Hi Yit,

On Fri, 14 Sep 2012 08:26:32 +0200, Khoo Yit Phang wrote:
> How do you run this? I can't seem to figure it out:

cd gdb-test-ootree/
mkdir build
cd build
../configure ;make
cd gdb/testsuite/
make site.exp
runtest CC_FOR_TARGET=/bin/sh\ $PWD/../../../gdb/contrib/cc-with-tweaks.sh\ -i\ gcc gdb.base/display.exp 



Thanks,
Jan

Re: Regression for --batch [Re: [RFC] Make python/lib/gdb and submodules proper Python modules]

[Khoo Yit Phang]

[-- Attachment #1: Type: text/plain, Size: 129 bytes --]

Hi,

Here's a patch that points gdb/contrib/cc-with-tweaks.sh to the appropriate data-directory.

Yit
September 14, 2012


[-- Attachment #2: fix-cc-with-tweaks.txt --]
[-- Type: text/plain, Size: 744 bytes --]

2012-09-14  Khoo Yit Phang  <khooyp@cs.umd.edu>

	Point contrib/cc-with-tweaks.sh to the build-local data-directory.
	* contrib/cc-with-tweaks.sh (GDB): Add -data-directory
	data-directory as appropriate.

diff --git a/gdb/contrib/cc-with-tweaks.sh b/gdb/contrib/cc-with-tweaks.sh
--- a/gdb/contrib/cc-with-tweaks.sh
+++ b/gdb/contrib/cc-with-tweaks.sh
@@ -46,13 +46,13 @@
 then
     if [ -f ./gdb ]
     then
-	GDB="./gdb"
+	GDB="./gdb -data-directory data-directory"
     elif [ -f ../gdb ]
     then
-	GDB="../gdb"
+	GDB="../gdb -data-directory ../data-directory"
     elif [ -f ../../gdb ]
     then
-	GDB="../../gdb"
+	GDB="../../gdb -data-directory ../../data-directory"
     else
 	echo "$myname: unable to find usable gdb" >&2
 	exit 1

Re: Regression for --batch [Re: [RFC] Make python/lib/gdb and submodules proper Python modules]

[Jan Kratochvil]

Hello Yit,

On Fri, 14 Sep 2012 08:52:20 +0200, Khoo Yit Phang wrote:
> Here's a patch that points gdb/contrib/cc-with-tweaks.sh to the appropriate
> data-directory.

Update also the comment above:
	# They may be overridden by setting environment variables GDB and OBJCOPY
	# respectively.

OK for check-in with that change.


Thanks,
Jan