RFC: Add a mechanism to stop backtraces using dwarf2 frame information

[Daniel Jacobowitz <drow@false.org>]

I'd like opinions on this patch.

There are some times when it's just not possible to backtrace through
hand-written assembly.  This can be true of anything which will never
return, and is often true of functions which are called or return in
a "unique" manner - the specific case that prompted me to write this was a
processor exception handler.  In this instance there is a theoretical return
path, but it will almost always lead out of the binary into some other
running code, so backtracing through it isn't useful.  Trying to apply
normal unwinding just produces garbage.

I picked an idiom which GDB currently doesn't handle to mean "no backtrace
information is available": DW_CFA_undefined in the return address column.
Seems a plausible interpretation to me.  This idiom implies that not only
is no DWARF unwinding data available, but also that more conventional means
of unwinding are unlikely to succeed.  Obviously, if GDB has an earlier
sniffer which recognizes the particular location, we can continue
backtracing.  This just stops us from falling back to the prologue
analyzers.

What do you think of the idea?  The patch?  If both seem OK, I'll propose
the idiom to the DWARF working group.  It doesn't require any changes to the
standard, but it might be nice to document it explicitly.

-- 
Daniel Jacobowitz
CodeSourcery, LLC

2005-03-02  Daniel Jacobowitz  <dan@codesourcery.com>

	* dwarf2-frame.c (struct dwarf2_frame_cache): New field
	undefined_retaddr.
	(dwarf2_frame_cache): Initialize undefined_retaddr.
	(dwarf2_frame_this_id): Return an invalid frame ID if
	undefined_retaddr.

Index: dwarf2-frame.c
===================================================================
RCS file: /big/fsf/rsync/src-cvs/src/gdb/dwarf2-frame.c,v
retrieving revision 1.48
diff -u -p -r1.48 dwarf2-frame.c
--- dwarf2-frame.c	11 Feb 2005 18:13:49 -0000	1.48
+++ dwarf2-frame.c	2 Mar 2005 22:04:58 -0000
@@ -1,6 +1,6 @@
 /* Frame unwinder for frames with DWARF Call Frame Information.
 
-   Copyright 2003, 2004 Free Software Foundation, Inc.
+   Copyright 2003, 2004, 2005 Free Software Foundation, Inc.
 
    Contributed by Mark Kettenis.
 
@@ -585,6 +585,9 @@ struct dwarf2_frame_cache
   /* DWARF Call Frame Address.  */
   CORE_ADDR cfa;
 
+  /* Set if the return address column was marked as undefined.  */
+  int undefined_retaddr;
+
   /* Saved registers, indexed by GDB register number, not by DWARF
      register number.  */
   struct dwarf2_frame_state_reg *reg;
@@ -748,6 +751,10 @@ incomplete CFI data; unspecified registe
       }
   }
 
+  if (fs->retaddr_column < fs->regs.num_regs
+      && fs->regs.reg[fs->retaddr_column].how == DWARF2_FRAME_REG_UNDEFINED)
+    cache->undefined_retaddr = 1;
+
   do_cleanups (old_chain);
 
   *this_cache = cache;
@@ -761,6 +768,9 @@ dwarf2_frame_this_id (struct frame_info 
   struct dwarf2_frame_cache *cache =
     dwarf2_frame_cache (next_frame, this_cache);
 
+  if (cache->undefined_retaddr)
+    return;
+
   (*this_id) = frame_id_build (cache->cfa, frame_func_unwind (next_frame));
 }