[RFC - Python Scripting] New method Objfile.symtabs () - docs included

[RFC - Python Scripting] New method Objfile.symtabs () - docs included

[Siva Chandra]

[-- Attachment #1: Type: text/plain, Size: 1552 bytes --]

Hi all,

Attached is a patch which adds a new method Objfile.symtabs to
gdb.Objfile.  This new method returns a list of gdb.Symtab objects
associated with the underlying object file.

Code ChangeLog:

2012-04-02  Siva Chandra Reddy  <sivachandra@google.com>

	Python Scripting: New method Objfile.symtabs which returns
	the list of obj.Symtab objects associated with the underlying
	objfile.
	* NEWS: Add entery under 'Python scripting' about the new
	Objfile.symtabs method.
	* python/py-objfile.c (stpy_iterator_object): New iterator
	object type to iterate over a list of obj.Symtab objects.
	(stpy_node): New struct to aid the iterator.
	(stpy_iterator_free_symtab_list, new_stpy_iterator,
	 objfpy_symtabs, stpy_iterator_dealloc, stpy_iterator_iter,
	 stpy_iterator_iternext): New functions which manage the new
	iterator object type.
	(objfpy_dealloc, objfpy_new, objfile_to_objfile_object,
	 gdbpy_initialize_objfile): Add initializations and reference
	management for the new iterator object and type.

Docs ChangeLog:

2012-04-02  Siva Chandra Reddy  <sivachandra@google.com>

	* gdb.texinfo (Python API/Objfiles In Python): Add description
	about the new method Objfile.symtabs on gdb.Objfile objects.

Testsuite ChangeLog:

2012-04-02  Siva Chandra Reddy  <sivachandra@google.com>

	* gdb.python/py-objfile.exp: Add tests to test the method
	Objfile.symtabs.
	* gdb.python/py-objfile1.c: New file to help test the new
	method Objfile.symtabs.
	* gdb.python/py-objfile2.c: Likewise
	* gdb.python/py-objfile3.c: Likewise

Thanks,
Siva Chandra

[-- Attachment #2: objfile_symtabs_patch_v1.txt --]
[-- Type: text/plain, Size: 16415 bytes --]

Index: python/py-objfile.c
===================================================================
RCS file: /cvs/src/src/gdb/python/py-objfile.c,v
retrieving revision 1.11
diff -c -p -r1.11 py-objfile.c
*** python/py-objfile.c	4 Jan 2012 08:17:25 -0000	1.11
--- python/py-objfile.c	2 Apr 2012 06:09:17 -0000
***************
*** 22,27 ****
--- 22,28 ----
  #include "charset.h"
  #include "objfiles.h"
  #include "language.h"
+ #include "psympriv.h"
  
  typedef struct
  {
*************** typedef struct
*** 32,45 ****
--- 33,172 ----
  
    /* The pretty-printer list of functions.  */
    PyObject *printers;
+ 
+   /* The iterator object to iterate over the list of symtabs associated
+      with the corresponding objfile.  */
+   PyObject *symtab_iterator;
  } objfile_object;
  
+ typedef struct stpy_node_i
+ {
+   PyObject *symtab_object;
+   struct stpy_node_i *next;
+ } stpy_node;
+ 
+ /* The iterator object to iterate over the list of symtabs associated
+    with an Objfile.  */
+ 
+ typedef struct
+ {
+   PyObject_HEAD
+ 
+   stpy_node *symtab_list;
+   stpy_node *current_node;
+ } stpy_iterator_object;
+ 
  static PyTypeObject objfile_object_type;
+ static PyTypeObject stpy_iterator_object_type;
  
  static const struct objfile_data *objfpy_objfile_data_key;
  
  \f
  
+ /* Free the symtab list in ITER_OBJECT.  Decrement the references to
+    the individual Symtab objects in the list.  */
+ 
+ static void
+ stpy_iterator_free_symtab_list (stpy_iterator_object *iter_object)
+ {
+   stpy_node *node;
+ 
+   if (! iter_object || ! iter_object->symtab_list)
+     return;
+ 
+   node = iter_object->symtab_list;
+   while (node)
+     {
+       stpy_node *temp = node;
+ 
+       Py_XDECREF (node->symtab_object);
+       node = node->next;
+ 
+       xfree (temp);
+     }
+ 
+   iter_object->symtab_list = NULL;
+   iter_object->current_node = NULL; 
+ }
+ 
+ /* Return a new stpy_iterator_object given an OBJFILE.  The caller holds
+    the reference to the new object.  */
+ 
+ static stpy_iterator_object *
+ new_stpy_iterator (struct objfile *objfile)
+ {
+   stpy_iterator_object *iter_object;
+   struct symtab *symtab;
+   struct partial_symtab *psymtab;
+   stpy_node *last_node = NULL;
+   int first = 1;
+   int invalid = 0;
+ 
+   iter_object = PyObject_New (stpy_iterator_object,
+                               &stpy_iterator_object_type);
+ 
+   if (! iter_object)
+     return NULL;
+ 
+   iter_object->symtab_list = NULL;
+   if (! objfile)
+     {
+       /* If the objfile is NULL, return an iter with no elements.  */
+       iter_object->current_node = NULL;
+       return iter_object;
+     }
+ 
+   /* If the symtabs are not yet read, then read them.  */
+   ALL_OBJFILE_PSYMTABS (objfile, psymtab)
+     {
+       if (psymtab->readin)
+         continue;
+ 
+       if (psymtab->read_symtab)
+         psymtab->read_symtab (psymtab);
+     }
+ 
+   ALL_OBJFILE_SYMTABS (objfile, symtab)
+     {
+       PyObject *symtab_object;
+ 
+       symtab_object = symtab_to_symtab_object (symtab);
+       if (symtab_object)
+         {
+           stpy_node *new_node;
+ 
+           new_node = (stpy_node *) xmalloc (sizeof (stpy_node));
+           new_node->symtab_object = symtab_object;
+           new_node->next = NULL;
+           if (first)
+             {
+               iter_object->symtab_list = new_node;
+               first = 0;
+             }
+           else
+             {
+               last_node->next = new_node;
+             }
+ 
+           last_node = new_node;
+         }
+       else
+         {
+           /* Give up if it is not possible to get the symtab_object
+              for a symtab.  */
+           invalid = 1;
+           break;
+         }
+     }
+ 
+   iter_object->current_node = iter_object->symtab_list;
+ 
+   if (invalid)
+     stpy_iterator_free_symtab_list (iter_object);
+ 
+   return iter_object;
+ }
+ 
  /* An Objfile method which returns the objfile's file name, or None.  */
  static PyObject *
  objfpy_get_filename (PyObject *self, void *closure)
*************** objfpy_dealloc (PyObject *o)
*** 58,63 ****
--- 185,191 ----
    objfile_object *self = (objfile_object *) o;
  
    Py_XDECREF (self->printers);
+   Py_XDECREF (self->symtab_iterator);
    self->ob_type->tp_free ((PyObject *) self);
  }
  
*************** objfpy_new (PyTypeObject *type, PyObject
*** 69,74 ****
--- 197,203 ----
    if (self)
      {
        self->objfile = NULL;
+       self->symtab_iterator = NULL;
  
        self->printers = PyList_New (0);
        if (!self->printers)
*************** objfpy_set_printers (PyObject *o, PyObje
*** 118,123 ****
--- 247,281 ----
    return 0;
  }
  
+ /* Return a sequence of symtab objects associated with the corresponding
+    objfile.  Each element in the sequence is a gdb.Symtab object.  */
+ 
+ static PyObject *
+ objfpy_symtabs (PyObject *self, PyObject *args)
+ {
+   objfile_object *obj = (objfile_object *) self;
+   stpy_iterator_object *iter;
+ 
+   if (! obj->objfile)
+     return NULL;
+ 
+   if (obj->symtab_iterator)
+     {
+       iter = (stpy_iterator_object *) obj->symtab_iterator;
+       iter->current_node = iter->symtab_list;
+     }
+   else
+     {
+       iter = new_stpy_iterator (obj->objfile);
+       if (! iter)
+         return NULL;
+ 
+       obj->symtab_iterator = (PyObject *) iter;
+     }
+ 
+   return PySequence_List ((PyObject *) iter);
+ }
+ 
  /* Implementation of gdb.Objfile.is_valid (self) -> Boolean.
     Returns True if this object file still exists in GDB.  */
  
*************** objfile_to_objfile_object (struct objfil
*** 164,169 ****
--- 322,328 ----
        if (object)
  	{
  	  object->objfile = objfile;
+           object->symtab_iterator = NULL;
  
  	  object->printers = PyList_New (0);
  	  if (!object->printers)
*************** objfile_to_objfile_object (struct objfil
*** 179,184 ****
--- 338,380 ----
    return (PyObject *) object;
  }
  
+ /* 'tp_dealloc' function for stpy_iterator_object_type.  */
+ 
+ static void
+ stpy_iterator_dealloc (PyObject *o)
+ {
+   stpy_iterator_free_symtab_list ((stpy_iterator_object *) o);
+ }
+ 
+ /* 'tp_iter' function for stpy_iterator_object_type.  */
+ 
+ static PyObject *
+ stpy_iterator_iter (PyObject *self)
+ {
+   Py_INCREF (self);
+   return self;
+ }
+ 
+ /* 'tp_iternext' function for stpy_iterator_object_type.  */
+ 
+ static PyObject *
+ stpy_iterator_iternext (PyObject *self)
+ {
+   stpy_iterator_object *iter = (stpy_iterator_object *) self;
+   stpy_node *node;
+   PyObject *elem;
+ 
+   if (! iter || ! iter->current_node)
+     return NULL;
+ 
+   node = iter->current_node;
+   iter->current_node = node->next;
+ 
+   elem = node->symtab_object;
+   Py_INCREF ((PyObject *) elem);
+   return elem;
+ }
+ 
  void
  gdbpy_initialize_objfile (void)
  {
*************** gdbpy_initialize_objfile (void)
*** 188,196 ****
--- 384,399 ----
    if (PyType_Ready (&objfile_object_type) < 0)
      return;
  
+   if (PyType_Ready (&stpy_iterator_object_type) < 0)
+     return;
+ 
    Py_INCREF (&objfile_object_type);
    PyModule_AddObject (gdb_module, "Objfile",
  		      (PyObject *) &objfile_object_type);
+ 
+   Py_INCREF (&stpy_iterator_object_type);
+   PyModule_AddObject (gdb_module, "SymtabIterator",
+                       (PyObject *) &stpy_iterator_object_type);
  }
  
  \f
*************** static PyMethodDef objfile_object_method
*** 200,205 ****
--- 403,412 ----
    { "is_valid", objfpy_is_valid, METH_NOARGS,
      "is_valid () -> Boolean.\n\
  Return true if this object file is valid, false if not." },
+   { "symtabs", objfpy_symtabs, METH_NOARGS,
+     "symtabs () -> list.\n\
+ Return the list of gdb.Symtab objects associated with the underlying\n\
+ object file."},
  
    { NULL }
  };
*************** static PyTypeObject objfile_object_type 
*** 255,257 ****
--- 462,496 ----
    0,				  /* tp_alloc */
    objfpy_new,			  /* tp_new */
  };
+ 
+ static PyTypeObject stpy_iterator_object_type =
+ {
+   PyObject_HEAD_INIT (NULL)
+   0,				  /*ob_size*/
+   "gdb.SymtabIterator",		  /*tp_name*/
+   sizeof (stpy_iterator_object),  /*tp_basicsize*/
+   0,				  /*tp_itemsize*/
+   stpy_iterator_dealloc,	  /*tp_dealloc*/
+   0,				  /*tp_print*/
+   0,				  /*tp_getattr*/
+   0,				  /*tp_setattr*/
+   0,				  /*tp_compare*/
+   0,				  /*tp_repr*/
+   0,				  /*tp_as_number*/
+   0,				  /*tp_as_sequence*/
+   0,				  /*tp_as_mapping*/
+   0,				  /*tp_hash */
+   0,				  /*tp_call*/
+   0,				  /*tp_str*/
+   0,				  /*tp_getattro*/
+   0,				  /*tp_setattro*/
+   0,				  /*tp_as_buffer*/
+   Py_TPFLAGS_DEFAULT | Py_TPFLAGS_HAVE_ITER,  /*tp_flags*/
+   "GDB symtab iterator object",	  /* tp_doc */
+   0,				  /* tp_traverse */
+   0,				  /* tp_clear */
+   0,				  /* tp_richcompare */
+   0,				  /* tp_weaklistoffset */
+   stpy_iterator_iter,		  /* tp_iter */
+   stpy_iterator_iternext,	  /* tp_iternext */
+ };
Index: doc/gdb.texinfo
===================================================================
RCS file: /cvs/src/src/gdb/doc/gdb.texinfo,v
retrieving revision 1.939
diff -c -p -r1.939 gdb.texinfo
*** doc/gdb.texinfo	28 Mar 2012 21:31:46 -0000	1.939
--- doc/gdb.texinfo	2 Apr 2012 06:09:24 -0000
*************** longer.  All other @code{gdb.Objfile} me
*** 23812,23817 ****
--- 23812,23823 ----
  if it is invalid at the time the method is called.
  @end defun
  
+ @defun Objfile.symtabs ()
+ Returns the list of @code{gdb.Symtab} objects associated with the
+ underlying object file referred by the @code{gdb.Objfile} object.
+ @xref{Symbol Tables In Python}.
+ @end defun
+ 
  @node Frames In Python
  @subsubsection Accessing inferior stack frames from Python.
  
Index: testsuite/gdb.python/py-objfile.exp
===================================================================
RCS file: /cvs/src/src/gdb/testsuite/gdb.python/py-objfile.exp,v
retrieving revision 1.3
diff -c -p -r1.3 py-objfile.exp
*** testsuite/gdb.python/py-objfile.exp	16 Jan 2012 16:21:52 -0000	1.3
--- testsuite/gdb.python/py-objfile.exp	2 Apr 2012 06:09:24 -0000
***************
*** 19,25 ****
  load_lib gdb-python.exp
  
  set testfile "py-objfile"
! set srcfile ${testfile}.c
  
  if { [prepare_for_testing ${testfile}.exp ${testfile} ${srcfile}] } {
      return -1
--- 19,26 ----
  load_lib gdb-python.exp
  
  set testfile "py-objfile"
! set srcfile "${testfile}.c ${testfile}1.c ${testfile}2.c ${testfile}3.c"
! set fnameprefix "${srcdir}/${subdir}/"
  
  if { [prepare_for_testing ${testfile}.exp ${testfile} ${srcfile}] } {
      return -1
*************** gdb_py_test_silent_cmd "python sym = gdb
*** 37,47 ****
--- 38,67 ----
      "Find a symbol in objfile" 1
  gdb_py_test_silent_cmd "python objfile = sym\[0\].symtab.objfile" \
      "Get backing object file" 1
+ gdb_py_test_silent_cmd "python symtabs = objfile.symtabs()" \
+     "Get the symtabs associated with the objfile" 1
+ gdb_py_test_silent_cmd "python filenames = \['${fnameprefix}${testfile}.c', \
+                                              '${fnameprefix}${testfile}1.c', \
+                                              '${fnameprefix}${testfile}2.c', \
+                                              '${fnameprefix}${testfile}3.c'\]" \
+     "Create a list of source file names" 1
  
  gdb_test "python print objfile.filename" ".*py-objfile.*" \
    "Get objfile validity"
  gdb_test "python print objfile.is_valid()" "True" \
    "Get objfile validity"
+ 
+ gdb_test "python print len(symtabs)" "4" \
+   "Get the number of symtabs associated with the objfile."
+ gdb_test "python print symtabs\[0\].filename in filenames" "True" \
+   "Check if symtab file is present in the list of filenames."
+ gdb_test "python print symtabs\[1\].filename in filenames" "True" \
+   "Check if symtab file is present in the list of filenames."
+ gdb_test "python print symtabs\[2\].filename in filenames" "True" \
+   "Check if symtab file is present in the list of filenames."
+ gdb_test "python print symtabs\[3\].filename in filenames" "True" \
+   "Check if symtab file is present in the list of filenames."
+ 
  gdb_unload
  gdb_test "python print objfile.is_valid()" "False" \
    "Get objfile validity after unload"
Index: testsuite/gdb.python/py-objfile1.c
===================================================================
RCS file: testsuite/gdb.python/py-objfile1.c
diff -N testsuite/gdb.python/py-objfile1.c
*** /dev/null	1 Jan 1970 00:00:00 -0000
--- testsuite/gdb.python/py-objfile1.c	2 Apr 2012 06:09:24 -0000
***************
*** 0 ****
--- 1,22 ----
+ /* This testcase is part of GDB, the GNU debugger.
+ 
+    Copyright 2012 Free Software Foundation, Inc.
+ 
+    This program is free software; you can redistribute it and/or modify
+    it under the terms of the GNU General Public License as published by
+    the Free Software Foundation; either version 3 of the License, or
+    (at your option) any later version.
+ 
+    This program is distributed in the hope that it will be useful,
+    but WITHOUT ANY WARRANTY; without even the implied warranty of
+    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+    GNU General Public License for more details.
+ 
+    You should have received a copy of the GNU General Public License
+    along with this program.  If not, see <http://www.gnu.org/licenses/>.  */
+ 
+ static int
+ add (int a, int b)
+ {
+   return a + b;
+ }
Index: testsuite/gdb.python/py-objfile2.c
===================================================================
RCS file: testsuite/gdb.python/py-objfile2.c
diff -N testsuite/gdb.python/py-objfile2.c
*** /dev/null	1 Jan 1970 00:00:00 -0000
--- testsuite/gdb.python/py-objfile2.c	2 Apr 2012 06:09:24 -0000
***************
*** 0 ****
--- 1,23 ----
+ /* This testcase is part of GDB, the GNU debugger.
+ 
+    Copyright 2012 Free Software Foundation, Inc.
+ 
+    This program is free software; you can redistribute it and/or modify
+    it under the terms of the GNU General Public License as published by
+    the Free Software Foundation; either version 3 of the License, or
+    (at your option) any later version.
+ 
+    This program is distributed in the hope that it will be useful,
+    but WITHOUT ANY WARRANTY; without even the implied warranty of
+    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+    GNU General Public License for more details.
+ 
+    You should have received a copy of the GNU General Public License
+    along with this program.  If not, see <http://www.gnu.org/licenses/>.  */
+ 
+ static int
+ sub (int a, int b)
+ {
+   return a - b;
+ }
+ 
Index: testsuite/gdb.python/py-objfile3.c
===================================================================
RCS file: testsuite/gdb.python/py-objfile3.c
diff -N testsuite/gdb.python/py-objfile3.c
*** /dev/null	1 Jan 1970 00:00:00 -0000
--- testsuite/gdb.python/py-objfile3.c	2 Apr 2012 06:09:24 -0000
***************
*** 0 ****
--- 1,23 ----
+ /* This testcase is part of GDB, the GNU debugger.
+ 
+    Copyright 2012 Free Software Foundation, Inc.
+ 
+    This program is free software; you can redistribute it and/or modify
+    it under the terms of the GNU General Public License as published by
+    the Free Software Foundation; either version 3 of the License, or
+    (at your option) any later version.
+ 
+    This program is distributed in the hope that it will be useful,
+    but WITHOUT ANY WARRANTY; without even the implied warranty of
+    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+    GNU General Public License for more details.
+ 
+    You should have received a copy of the GNU General Public License
+    along with this program.  If not, see <http://www.gnu.org/licenses/>.  */
+ 
+ static int
+ mul (int a, int b)
+ {
+   return a * b;
+ }
+ 
Index: NEWS
===================================================================
RCS file: /cvs/src/src/gdb/NEWS,v
retrieving revision 1.505
diff -c -p -r1.505 NEWS
*** NEWS	28 Mar 2012 21:31:56 -0000	1.505
--- NEWS	2 Apr 2012 06:09:24 -0000
***************
*** 32,37 ****
--- 32,41 ----
    ** A new method 'referenced_value' on gdb.Value objects which can
       dereference pointer as well as C++ reference values.
  
+   ** A new method 'symtabs' on gdb.Objfile objects which returns the
+      list of gdb.Symtab objects associated with the underlying object
+      files.
+ 
  * GDBserver now supports stdio connections.
    E.g. (gdb) target remote | ssh myhost gdbserver - hello
  

Re: [RFC - Python Scripting] New method Objfile.symtabs () - docs included

[Eli Zaretskii]

> Date: Mon, 2 Apr 2012 11:47:31 +0530
> From: Siva Chandra <sivachandra@google.com>
> 
> Attached is a patch which adds a new method Objfile.symtabs to
> gdb.Objfile.  This new method returns a list of gdb.Symtab objects
> associated with the underlying object file.

Thanks.

> *** doc/gdb.texinfo	28 Mar 2012 21:31:46 -0000	1.939
> --- doc/gdb.texinfo	2 Apr 2012 06:09:24 -0000
> *************** longer.  All other @code{gdb.Objfile} me
> *** 23812,23817 ****
> --- 23812,23823 ----
>   if it is invalid at the time the method is called.
>   @end defun
>   
> + @defun Objfile.symtabs ()
> + Returns the list of @code{gdb.Symtab} objects associated with the
> + underlying object file referred by the @code{gdb.Objfile} object.
                           ^^^^^^^^^^^
"referred to by ..."

> +   ** A new method 'symtabs' on gdb.Objfile objects which returns the
> +      list of gdb.Symtab objects associated with the underlying object
> +      files.

"file", I think, in single.

OK with those changes.

Re: [RFC - Python Scripting] New method Objfile.symtabs () - docs included

[Doug Evans]

On Sun, Apr 1, 2012 at 11:17 PM, Siva Chandra <sivachandra@google.com> wrote:
> Hi all,
>
> Attached is a patch which adds a new method Objfile.symtabs to
> gdb.Objfile.  This new method returns a list of gdb.Symtab objects
> associated with the underlying object file.
>
> Code ChangeLog:
>
> 2012-04-02  Siva Chandra Reddy  <sivachandra@google.com>
>
>        Python Scripting: New method Objfile.symtabs which returns
>        the list of obj.Symtab objects associated with the underlying
>        objfile.
>        * NEWS: Add entery under 'Python scripting' about the new

spelling: "entry"

>        Objfile.symtabs method.
>        * python/py-objfile.c (stpy_iterator_object): New iterator
>        object type to iterate over a list of obj.Symtab objects.
>        (stpy_node): New struct to aid the iterator.
>        (stpy_iterator_free_symtab_list, new_stpy_iterator,
>         objfpy_symtabs, stpy_iterator_dealloc, stpy_iterator_iter,
>         stpy_iterator_iternext): New functions which manage the new
>        iterator object type.
>        (objfpy_dealloc, objfpy_new, objfile_to_objfile_object,
>         gdbpy_initialize_objfile): Add initializations and reference
>        management for the new iterator object and type.

Hi.
You're touching on one of the more troublesome areas of gdb (symbol
handling), I salute your courage. :-)

I didn't read the entire patch but one thing stood out:

+   /* If the symtabs are not yet read, then read them.  */
+   ALL_OBJFILE_PSYMTABS (objfile, psymtab)
+     {
+       if (psymtab->readin)
+         continue;
+
+       if (psymtab->read_symtab)
+         psymtab->read_symtab (psymtab);
+     }

This code is doing something we want to try really hard to avoid.
In a large program this will push gdb's memory usage far beyond
anything the user will find acceptable (not to mention taking a long
time).

We want to do the expansion as lazily as possible.
[We also, IMO, want to expose as little of the symbol table
implementation details in the Python API as possible, at least
initially.]
The current code may not make that easy (or even possible, absent
changes), but if a simple and innocent looking my_objfile.symtabs() in
Python can push gdb's memory usage to several 10s of gigabytes
(assuming you have enough memory and swap :-)), then I think we need
to avoid that.

Assuming we want to provide the ability to iterate over all the symbol
tables, we want to be able to do that without first expanding them.
So I think the first question is what will the user want to do with
this feature?

Re: [RFC - Python Scripting] New method Objfile.symtabs () - docs included

[Siva Chandra]

On Tue, Apr 3, 2012 at 5:34 AM, Doug Evans <dje@google.com> wrote:
>
> You're touching on one of the more troublesome areas of gdb (symbol
> handling), I salute your courage. :-)

I seem courageous as I do not know what awaits me. :-)

> Assuming we want to provide the ability to iterate over all the symbol
> tables, we want to be able to do that without first expanding them.
> So I think the first question is what will the user want to do with
> this feature?

Thanks for taking a look and starting a discussion on this.  An
example usage of this feature can be to implement something like run
time code path tracer.  This can probably be done in many other ways,
but I am using this as an example.  In this code path tracer, a user
might want to trace the function calls when between state A and state
B of program.  A way to do this from Python would be to get all the
function names and set internal breakpoints at these functions.  But
the user should first have a way to get the list of functions.  This
is where I thought the Objfile.symtabs becomes useful - The
exploration of symbols to look for functions can happen in this order:
gdb.objfiles () => Objfile.symtabs () => Symtab.blocks () =>
Block.symbol => Symbol.name.  This patch adding Objfile.symtabs () was
first of my patches towards completing this path.

Looking at it more generically, I thought that having the exploratory
path that I mention above available in the Python API completes a
'view' of GDB from a user's point.

For the memory problem you point out, do you think something like
Objfile.symtabs (regex, [symbol_type]) would be good?  REGEX is a file
name matcher to specify that only those symtabs whose source file
names match the REGEX should be loaded, and SYMBOL_TYPE is to specify
that only symbols of this kind should be loaded.  This can still be
abused to match everything and load everything, but it is at the
user's own peril.

Thanks,
Siva Chandra

RE: [RFC - Python Scripting] New method Objfile.symtabs () - docs included

[Paul_Koning]

> Assuming we want to provide the ability to iterate over all the symbol 
> tables, we want to be able to do that without first expanding them.
> So I think the first question is what will the user want to do with 
> this feature?

Something else I'm looking for, and this seems to be a step in that direction, is a way to iterate over all the defined types.  For example, I want to do some checking similar to what "pahole" does.  The command line utility by that name processes all the types but its output is hard to handle.  The Python machinery to do the equivalent on a single type is already in place, but how do I get all the types defined in my program?  Those presumably live in a symtab but I didn't see a gdb.Symtab method to get them.  (Is that the right approach?  I can work on creating the machinery if this is the place for them, or if someone can point me to where in the gdb.* class hierarchy to fit this.)

	paul

Re: [RFC - Python Scripting] New method Objfile.symtabs () - docs included

[Tom Tromey]

>>>>> "Paul" ==   <Paul_Koning@Dell.com> writes:

Paul> Those presumably live in a symtab but I didn't
Paul> see a gdb.Symtab method to get them.  (Is that the right approach?  I
Paul> can work on creating the machinery if this is the place for them, or
Paul> if someone can point me to where in the gdb.* class hierarchy to fit
Paul> this.)

Yes, I think adding the ability to iterate over symbols in a symtab is
the way to go.

Actually, what Siva said sounds good, aside from the lazy CU
instantiation issue:

Siva> The exploration of symbols to look for functions can happen in
Siva> this order: gdb.objfiles () => Objfile.symtabs () => Symtab.blocks
Siva> () => Block.symbol => Symbol.name.

Of course for your use, you won't care about lazy instantiation so much,
since you will be expanding them all anyway.

An extension to the earlier idea would be to make it possible to have
the iterator take a symbol name to search for as well.  This would let
you instantiate many fewer CUs.  This is more or less how the "quick"
symbol API works.

Tom

Re: [RFC - Python Scripting] New method Objfile.symtabs () - docs included

[Tom Tromey]

>>>>> "Siva" == Siva Chandra <sivachandra@google.com> writes:

Siva> For the memory problem you point out, do you think something like
Siva> Objfile.symtabs (regex, [symbol_type]) would be good?  REGEX is a file
Siva> name matcher to specify that only those symtabs whose source file
Siva> names match the REGEX should be loaded, and SYMBOL_TYPE is to specify
Siva> that only symbols of this kind should be loaded.  This can still be
Siva> abused to match everything and load everything, but it is at the
Siva> user's own peril.

I think this proposed API mixes symtab- and symbol-level operations.
The file name is a property of a symtab, but the symbol type is a
property of the symbol.

There's presently no way in gdb to load just part of a symtab.
CUs are always fully instantiated.  This could be changed -- I think it
is a good idea with more than one concrete benefit -- but it doesn't
exist now.

How about just the plain Objfile.iterator(REGEX)?
It would return an iterator that lazily instantiates symtabs.
(Or, at least conceptually lazily instantiates them, as it isn't clear
that this is efficiently implementable with the current
quick_symbol_functions API.)

Tom

RE: [RFC - Python Scripting] New method Objfile.symtabs () - docs included

[Paul_Koning]

In this patch, you have an iterator that walks over the symtabs for an objfile, but that is only used internally in the symtabs() method which is written to return a list.

How about having symtabs() return the iterator instead?  An iterator is usually just as good as a list (and more efficient); in the rare cases where you actually need the list, simply doing list(objfile.symtabs()) will do that.

	paul

Re: [RFC - Python Scripting] New method Objfile.symtabs () - docs included

[Doug Evans]

On Thu, Apr 5, 2012 at 12:56 PM,  <Paul_Koning@dell.com> wrote:
> In this patch, you have an iterator that walks over the symtabs for an objfile, but that is only used internally in the symtabs() method which is written to return a list.
>
> How about having symtabs() return the iterator instead?  An iterator is usually just as good as a list (and more efficient); in the rare cases where you actually need the list, simply doing list(objfile.symtabs()) will do that.

There is a lower fundamental problem that needs to be addressed.

In a nutshell: If "list (objfile.symtabs())" can increase gdb's memory
usage by gigabytes, then it's a non-starter (IMO).
Before we provide anything along these lines, we need to fix the
underlying problem.

I can think of a simple(quick) solution: create the symtab object when
the psymtab object is created (suitably modified to clean up anything
obvious), but not expand it.  [btw, Psymtabs at the moment aren't as
much of an implementation detail as we want them to be (IMO).] It's
not perfect: Depending on what you want from the symtab you may
ultimately end up expanding everything anyway.  But it's a step.  One
way to go would be to build into gdb the ability to discard the
expansion say when memory gets tight.  I think there's room for
improvement before we get to that though.

Re: [RFC - Python Scripting] New method Objfile.symtabs () - docs included

[Siva Chandra]

Tom> How about just the plain Objfile.iterator(REGEX)?
Tom> It would return an iterator that lazily instantiates symtabs.
Tom> (Or, at least conceptually lazily instantiates them, as it isn't clear
Tom> that this is efficiently implementable with the current
Tom> quick_symbol_functions API.)

Is this equivalent to what Paul Koning suggests and has the same
problem which Doug Evans points out (source file matching with a REGEX
aside)?

Paul> How about having symtabs() return the iterator instead?

Doug> In a nutshell: If "list (objfile.symtabs())" can increase gdb's memory
Doug> usage by gigabytes, then it's a non-starter (IMO).
Doug> Before we provide anything along these lines, we need to fix the
Doug> underlying problem.

Doug> I can think of a simple(quick) solution: create the symtab object when
Doug> the psymtab object is created (suitably modified to clean up anything
Doug> obvious), but not expand it.  [btw, Psymtabs at the moment aren't as
Doug> much of an implementation detail as we want them to be (IMO).] It's
Doug> not perfect: Depending on what you want from the symtab you may
Doug> ultimately end up expanding everything anyway.  But it's a step.  One
Doug> way to go would be to build into gdb the ability to discard the
Doug> expansion say when memory gets tight.  I think there's room for
Doug> improvement before we get to that though.

I am OK to think through this proposal.  I will need some time to
understand and come up with something.

Thanks,
Siva Chandra

Re: [RFC - Python Scripting] New method Objfile.symtabs () - docs included

[Tom Tromey]

>>>>> "Siva" == Siva Chandra <sivachandra@google.com> writes:

Tom> How about just the plain Objfile.iterator(REGEX)?
Tom> It would return an iterator that lazily instantiates symtabs.
Tom> (Or, at least conceptually lazily instantiates them, as it isn't clear
Tom> that this is efficiently implementable with the current
Tom> quick_symbol_functions API.)

Siva> Is this equivalent to what Paul Koning suggests and has the same
Siva> problem which Doug Evans points out (source file matching with a REGEX
Siva> aside)?

More or less.  The key thing is to pass a kind of selector to the
iterator so that it can pick which symtabs to expand.  This lets you
keep a cap on memory use.

Ultimately if a script asks for all the CUs, memory and CPU are going to
go through the roof.  I think it is good to warn about this and provide
ways to work around it, but in the end I don't have a problem with it.
If you want to look at all the symbols in a big program, well, that's
just a lot of work.

Doug> I can think of a simple(quick) solution: create the symtab object when
Doug> the psymtab object is created (suitably modified to clean up anything
Doug> obvious), but not expand it.  [btw, Psymtabs at the moment aren't as
Doug> much of an implementation detail as we want them to be (IMO).] It's
Doug> not perfect: Depending on what you want from the symtab you may
Doug> ultimately end up expanding everything anyway.  But it's a step.  One
Doug> way to go would be to build into gdb the ability to discard the
Doug> expansion say when memory gets tight.  I think there's room for
Doug> improvement before we get to that though.

FWIW, I have a 2 part plan for lazy CU expansion that, I think, would
help this situation quite a bit, for DWARF anyway.


The first part is to change the DWARF reader not to fully read symbols,
then change symtab.c to finish filling in a function symbol just before
it is returned, and change SYMBOL_TYPE to lazily read the type.

This is not extremely hard to do, and it both speeds up CU expansion
(just skipping function bodies is worth ~30%) and reduces memory use in
the common case where most symbols in a CU are not needed.

I have a starter patch for the function body part if anybody is curious.


Part 2 is more involved -- change partial symtabs to also record the DIE
offsets, and then change the psymtab -> symtab expander to just copy
this information over.  The idea here is that we then don't need to
re-scan .debug_info at all, we're just shuffling some bits around.

Part of this could also be to introduce some memory savings by letting
(some) symbols point to their corresponding partial symbol, instead of
effectively having a copy of the data.


I like the idea of being able to discard symtabs when memory is tight.

Tom

Re: [RFC - Python Scripting] New method Objfile.symtabs () - docs included

[Doug Evans]

On Mon, Apr 9, 2012 at 10:55 AM, Tom Tromey <tromey@redhat.com> wrote:
> Part of this could also be to introduce some memory savings by letting
> (some) symbols point to their corresponding partial symbol, instead of
> effectively having a copy of the data.

Indeed.
I'd like to push this further as well.