How to tell gdb about dlls using remote protocol

How to tell gdb about dlls using remote protocol

[Wiljan Derks]

Hello,

I have build my own code that can reside in our applications and that allows 
a connection to gdb.
It allows remote debugging on Windows using gdb and also allows us to keep 
specific threads running
because this is a realtime control application.
Thus I am using the "gdb remote" protocol and not the win32 debug api.

The problem that I have run into, is that I can find no way to let gdb know 
what dll's are inside the
debugee using the remote protocol.

Does anyone know if there is a way to do this ?

Thanks,

Wiljan 

Re: How to tell gdb about dlls using remote protocol

[Daniel Jacobowitz]

On Wed, Jan 31, 2007 at 10:09:13PM +0100, Wiljan Derks wrote:
> The problem that I have run into, is that I can find no way to let gdb know 
> what dll's are inside the
> debugee using the remote protocol.
> 
> Does anyone know if there is a way to do this ?

There's no way to do this yet.  If you look at the list archives for
the last several months, you'll see a patch (in the "GDB solib
interface" thread) that implements something which might help.  But it
hasn't been finalized or committed yet (sorry Stephen - I just haven't
had time).

-- 
Daniel Jacobowitz
CodeSourcery

Re: How to tell gdb about dlls using remote protocol

[Joel Brobecker]

[-- Attachment #1: Type: text/plain, Size: 1143 bytes --]

Hi Daniel,

> > The problem that I have run into, is that I can find no way to let gdb know 
> > what dll's are inside the
> > debugee using the remote protocol.
> > 
> > Does anyone know if there is a way to do this ?
> 
> There's no way to do this yet.  If you look at the list archives for
> the last several months, you'll see a patch (in the "GDB solib
> interface" thread) that implements something which might help.  But it
> hasn't been finalized or committed yet (sorry Stephen - I just haven't
> had time).

This makes me wonder how well the debugger can work in certain
situations like when backtracing from DLL code. If the debugger
doesn't know where it is, then it's probably let to prologue analysis
to do the unwinding. Except that it cannot determine where the prologue
is... In that case, I see that the i386 unwinder assumes that the frame
base can be deduced from the SP and the SP offset. Unfortunately,
this SP offset can only be deduced from prologue analysis. Catch 22?

Mark,

What do you think of this (untested) patch? We we couldn't find
the function start address, the safest seems to be relying on ebp.

-- 
Joel

[-- Attachment #2: i386-tdep.c.diff --]
[-- Type: text/plain, Size: 717 bytes --]

Index: i386-tdep.c
===================================================================
RCS file: /cvs/src/src/gdb/i386-tdep.c,v
retrieving revision 1.230
diff -u -p -r1.230 i386-tdep.c
--- i386-tdep.c	29 Jan 2007 17:31:06 -0000	1.230
+++ i386-tdep.c	1 Feb 2007 17:50:51 -0000
@@ -979,6 +979,12 @@ i386_frame_cache (struct frame_info *nex
 	  /* This will be added back below.  */
 	  cache->saved_regs[I386_EIP_REGNUM] -= cache->base;
 	}
+      else if (cache->pc == 0)
+        {
+          /* We couldn't determine the function start address, bla
+             bla bla.  */
+          cache->saved_regs[I386_EBP_REGNUM] = 0;
+        }
       else
 	{
 	  frame_unwind_register (next_frame, I386_ESP_REGNUM, buf);

Re: How to tell gdb about dlls using remote protocol

[Daniel Jacobowitz]

On Thu, Feb 01, 2007 at 09:53:11AM -0800, Joel Brobecker wrote:
> This makes me wonder how well the debugger can work in certain
> situations like when backtracing from DLL code. If the debugger
> doesn't know where it is, then it's probably let to prologue analysis
> to do the unwinding. Except that it cannot determine where the prologue
> is... In that case, I see that the i386 unwinder assumes that the frame
> base can be deduced from the SP and the SP offset. Unfortunately,
> this SP offset can only be deduced from prologue analysis. Catch 22?

In general, if we can not find the start of the current function,
we have absolutely no chance of getting out of it.

This is a standard problem e.g. with the Windows system DLLs, since
we have inadequate means to recover symbol information from them.

-- 
Daniel Jacobowitz
CodeSourcery

Re: How to tell gdb about dlls using remote protocol

[Joel Brobecker]

> In general, if we can not find the start of the current function,
> we have absolutely no chance of getting out of it.
> 
> This is a standard problem e.g. with the Windows system DLLs, since
> we have inadequate means to recover symbol information from them.

At least on Windows, I think the change I proposed should help
increase the odds in our favor in this situation. If I read the code
correctly, the current approach will almost always fail, whereas
using %ebp should get us somewhere sensible unless %ebp is used
as a scratch register...

-- 
Joel

Re: How to tell gdb about dlls using remote protocol

[Daniel Jacobowitz]

On Thu, Feb 01, 2007 at 03:03:01PM -0800, Joel Brobecker wrote:
> > In general, if we can not find the start of the current function,
> > we have absolutely no chance of getting out of it.
> > 
> > This is a standard problem e.g. with the Windows system DLLs, since
> > we have inadequate means to recover symbol information from them.
> 
> At least on Windows, I think the change I proposed should help
> increase the odds in our favor in this situation. If I read the code
> correctly, the current approach will almost always fail, whereas
> using %ebp should get us somewhere sensible unless %ebp is used
> as a scratch register...

Except it's pretty standard to compile without a frame pointer on
Windows, from what I've encountered - and more importantly, all of the
Microsoft DLLs seem to do so.  I still think we need to come up with
some way to take advantage of the MS symbol info, though it's not a
small project (so I haven't really tried to do it).

-- 
Daniel Jacobowitz
CodeSourcery

Re: How to tell gdb about dlls using remote protocol

[Robert Dewar]

Daniel Jacobowitz wrote:

> Except it's pretty standard to compile without a frame pointer on
> Windows, from what I've encountered - and more importantly, all of the
> Microsoft DLLs seem to do so.  I still think we need to come up with
> some way to take advantage of the MS symbol info, though it's not a
> small project (so I haven't really tried to do it).

True, but it is relatively unlikely to be interesting to debug through
Microsoft DLL's, much more likely and useful to debug through user
written DLL's, and people can learn that if they want to debug such
DLL's they should not suppress the frame pointer.

I agree it would be nice to take advantage of the MS symbol info if
this is possible (technically and legally), but as you say it is a
big project, and we should not let best be the enemy of better.

Re: How to tell gdb about dlls using remote protocol

[Daniel Jacobowitz]

On Fri, Feb 02, 2007 at 01:19:58AM -0500, Robert Dewar wrote:
> True, but it is relatively unlikely to be interesting to debug through
> Microsoft DLL's, much more likely and useful to debug through user
> written DLL's, and people can learn that if they want to debug such
> DLL's they should not suppress the frame pointer.

Sorry, I have to disagree from my own experience.  Pretty much any time
you stop a running program it's in those DLLs.  And this is doubly true
for any threaded program - there's always a couple of threads that I
have no idea what the heck they're doing.

> I agree it would be nice to take advantage of the MS symbol info if
> this is possible (technically and legally), but as you say it is a
> big project, and we should not let best be the enemy of better.

I'm hoping Mark will respond to the patch.  We stopped doing it
deliberately in the past; I don't know if we want to revisit or not.

-- 
Daniel Jacobowitz
CodeSourcery

Re: How to tell gdb about dlls using remote protocol

[Joel Brobecker]

> Sorry, I have to disagree from my own experience.  Pretty much any time
> you stop a running program it's in those DLLs.  And this is doubly true
> for any threaded program - there's always a couple of threads that I
> have no idea what the heck they're doing.

In fact, we have a local modification in our tree (different from the
one I recently suggested) where we default to using %ebp when inside a
frameless function, and inside a DLL.  This is a heuristic way to handle
all those highly optimized functions for which prologue analysis cannot
be used. Short of implementing support for MS symbol info, this is the
only way we could get the backtrace of most threads. This is a hack I
didn't submit because it's a bit crude, and it only exchanges certain
failures for others - but we have found in practice that this was the
right choice for us. I can certainly post it too, if you are interested.

-- 
Joel

Re: How to tell gdb about dlls using remote protocol

[Daniel Jacobowitz]

On Fri, Feb 02, 2007 at 08:51:55AM -0800, Joel Brobecker wrote:
> > Sorry, I have to disagree from my own experience.  Pretty much any time
> > you stop a running program it's in those DLLs.  And this is doubly true
> > for any threaded program - there's always a couple of threads that I
> > have no idea what the heck they're doing.
> 
> In fact, we have a local modification in our tree (different from the
> one I recently suggested) where we default to using %ebp when inside a
> frameless function, and inside a DLL.  This is a heuristic way to handle
> all those highly optimized functions for which prologue analysis cannot
> be used. Short of implementing support for MS symbol info, this is the
> only way we could get the backtrace of most threads. This is a hack I
> didn't submit because it's a bit crude, and it only exchanges certain
> failures for others - but we have found in practice that this was the
> right choice for us. I can certainly post it too, if you are interested.

Hmm, perhaps it successfully gets you out and only misses frames.  I'm
pretty sure I remember that when debugging a Windows build of GDB, the
select helper threads live somewhere in NTDLL without a valid frame.

I'd be curious to see it, at least, but I'm not sure what we can do.

-- 
Daniel Jacobowitz
CodeSourcery

Re: How to tell gdb about dlls using remote protocol

[Joel Brobecker]

[-- Attachment #1: Type: text/plain, Size: 592 bytes --]

> Hmm, perhaps it successfully gets you out and only misses frames.  I'm
> pretty sure I remember that when debugging a Windows build of GDB, the
> select helper threads live somewhere in NTDLL without a valid frame.

I am not sure about that particular thread. But for the other ones,
we are indeed counting on the fact that all we lose is skipping one
frame.

> I'd be curious to see it, at least, but I'm not sure what we can do.

Here it is. The diff is probably malformed, because I had to remove
a couple of patches we backported from head, but that should give
you the idea.

-- 
Joel

[-- Attachment #2: ebp.diff --]
[-- Type: text/plain, Size: 3459 bytes --]

+/* Return non-zero if the function starting at START_PC has a prologue
+   that sets up a standard frame.  */
+
+static int
+i386_function_has_frame (CORE_ADDR start_pc)
+{
+  struct i386_frame_cache cache;
+
+  cache.locals = -1;
+  i386_analyze_prologue (start_pc, 0xffffffff, &cache);
+
+  return (cache.locals >= 0);
+}
+
+/* Return non-zero if PC is inside one of the inferior's DLLs.  */
+
+static int
+i386_in_dll (CORE_ADDR pc)
+{
+   char *so_name = solib_address (pc);
+   int len;
+
+   if (so_name == NULL)
+     return 0;
+
+   len = strlen (so_name);
+   if (len < 5)
+     return 0;
+
+   return ((so_name[len - 1] == 'l' || so_name[len - 1] == 'L')
+           && (so_name[len - 2] == 'l' || so_name[len - 2] == 'L')
+           && (so_name[len - 3] == 'd' || so_name[len - 3] == 'D')
+           && so_name[len - 4] == '.');
+}
+
 /* Normal frames.  */
 
 static struct i386_frame_cache *
@@ -954,20 +1159,49 @@ i386_frame_cache (struct frame_info *nex
 	 frame by looking at the stack pointer.  For truly "frameless"
 	 functions this might work too.  */
 
-      if (cache->stack_align)
-	{
-	  /* We're halfway aligning the stack.  */
-	  cache->base = ((cache->saved_sp - 4) & 0xfffffff0) - 4;
-	  cache->saved_regs[I386_EIP_REGNUM] = cache->saved_sp - 4;
+      if (i386_in_dll (cache->pc)
+          && !i386_function_has_frame (cache->pc))
+        {
+          /* Functions in DLL for which do not seem to create a standard
+             frame are unwound using %ebp.  This is actually the caller's
+             frame base instead of our own, but there are some functions
+             such as WaitForSingleObjectEx in one of the Windows system
+             DLLs for which the frame base cannot possibly be determined
+             from the stack pointer.  As a consequence, our caller will be
+             missing from the backtrace, but this is better than having
+             an aborted backtrace due to a bogus frame base.
+             
+             We use this approach only for functions in DLLs because
+             this is the only place where we have seen the type of
+             highly optimized code that cause us trouble.  In other
+             cases, we expect the code to come with frame debugging
+             information, making prologue scanning unnecessary.
+             
+             We also avoid blindly following %ebp if we are midway through
+             setting up a standard frame.  In that case, we know how to
+             determine the frame base using the stack pointer.  */
 
-	  /* This will be added back below.  */
-	  cache->saved_regs[I386_EIP_REGNUM] -= cache->base;
-	}
+          cache->saved_regs[I386_EBP_REGNUM] = 0;
+        }
       else
-	{
-	  frame_unwind_register (next_frame, I386_ESP_REGNUM, buf);
-	  cache->base = extract_unsigned_integer (buf, 4) + cache->sp_offset;
-	}
+        {
+	  if (cache->stack_align)
+	    {
+	      /* We're halfway aligning the stack.  */
+	      cache->base = ((cache->saved_sp - 4) & 0xfffffff0) - 4;
+	      cache->saved_regs[I386_EIP_REGNUM] = cache->saved_sp - 4;
+
+	      /* This will be added back below.  */
+	      cache->saved_regs[I386_EIP_REGNUM] -= cache->base;
+	    }
+	  else
+	    {
+	      frame_unwind_register (next_frame, I386_ESP_REGNUM, buf);
+	      cache->base = extract_unsigned_integer (buf, 4) + cache->sp_offset;
+	    }
+        }
     }
 
   /* Now that we have the base address for the stack frame we can

Re: How to tell gdb about dlls using remote protocol

[Wiljan Derks]

I totoally agree with Joel that some practical approach is needed to make 
tracebacks
from code of which no information is available since usually most of the 
treads are somewhere
in a windows dll when stopping things.
The patch of Joel is based on a procedure that checks if code is in a known 
dll.
This is exactly what got me into problems when trying to make tracebacks 
using the gdb remote
interface on windows.
Because the gdb remote protocol does not allow to inform gdb about the dll's 
in de debugee,
this patch does not work in that case.
For that reason, I modified the code so that I check if the pc is inside one 
of the sections of the program.
If it is not, I use the same unwinding as used in Joels patch when the pc is 
detected to be in some dll.
This avoids having knowledge about the dlls in gdb and still gives the same 
tracebacks.
I guess the result is the same as for Joels patch, except that now also the 
remote protocol works
because gdb does not need to have knowledge on the dlls.

This patch made things work for me using the remote protocol of gdb.

What do you think of this approach ?

Wiljan

Re: How to tell gdb about dlls using remote protocol

[Joel Brobecker]

Wiljan,

> The patch of Joel is based on a procedure that checks if code is in a
> known dll.

I think you are slightly confused. The little patch that I suggested
(and labeled untested) is not based on any previous patch. Have you
tried it without making the modifications you suggest below?

> For that reason, I modified the code so that I check if the pc is
> inside one of the sections of the program.

My other patch that you are refering to is the one that forces GDB
to trust %ebp in frameless functions that live in DLLs. This change
was to work-around a problem that is totally unrelated from the one
you are trying to solve.

To re-explain the logic behind the suggestion I posted here, when
we don't find the function start address (as in your case), we don't
parse the prologue. In this case, the current code will fallback on
using the default value for cache->sp_offset, which is unlikely to
be right. My suggestion instead is to trust the %ebp register, for
which we have found that we get better results (on Windows).

-- 
Joel

Re: How to tell gdb about dlls using remote protocol

[Mark Kettenis]

> Date: Fri, 2 Feb 2007 09:34:56 -0800
> From: Joel Brobecker <brobecker@adacore.com>
> 
> > Hmm, perhaps it successfully gets you out and only misses frames.  I'm
> > pretty sure I remember that when debugging a Windows build of GDB, the
> > select helper threads live somewhere in NTDLL without a valid frame.
> 
> I am not sure about that particular thread. But for the other ones,
> we are indeed counting on the fact that all we lose is skipping one
> frame.
> 
> > I'd be curious to see it, at least, but I'm not sure what we can do.
> 
> Here it is. The diff is probably malformed, because I had to remove
> a couple of patches we backported from head, but that should give
> you the idea.

I wouldn't be too thrilled to have this Windows-specific code in the
generic i386-tdep.c.

Re: How to tell gdb about dlls using remote protocol

[Mark Kettenis]

> X-Spam-Check-By: sourceware.org
> Date: Thu, 1 Feb 2007 09:53:11 -0800
> From: Joel Brobecker <brobecker@adacore.com>
> Content-Disposition: inline
> Mailing-List: contact gdb-help@sourceware.org; run by ezmlm
> Sender: gdb-owner@sourceware.org
> X-UTwente-MailScanner-Information: Scanned by MailScanner. Contact helpdesk@ITBE.utwente.nl for more information.
> X-UTwente-MailScanner: Found to be clean
> X-UTwente-MailScanner-From: gdb-return-27623-m.m.kettenis=alumnus.utwente.nl@sourceware.org
> X-Spam-Status: No
> X-XS4ALL-DNSBL-Checked: mxdrop39.xs4all.nl checked 130.89.2.13 against DNS blacklists
> X-Virus-Scanned: by XS4ALL Virus Scanner
> X-XS4ALL-Spam-Score: -2.0 () DK_POLICY_SIGNSOME,RCVD_IN_NLWHITE
> X-XS4ALL-Spam: NO
> Envelope-To: mark.kettenis@xs4all.nl
> X-UIDL: 1170352362._smtp.mxdrop39.85044,S=5181
> 
> 
> --vkogqOf2sHV7VnPd
> Content-Type: text/plain; charset=us-ascii
> Content-Disposition: inline
> 
> Hi Daniel,
> 
> > > The problem that I have run into, is that I can find no way to let gdb know 
> > > what dll's are inside the
> > > debugee using the remote protocol.
> > > 
> > > Does anyone know if there is a way to do this ?
> > 
> > There's no way to do this yet.  If you look at the list archives for
> > the last several months, you'll see a patch (in the "GDB solib
> > interface" thread) that implements something which might help.  But it
> > hasn't been finalized or committed yet (sorry Stephen - I just haven't
> > had time).
> 
> This makes me wonder how well the debugger can work in certain
> situations like when backtracing from DLL code. If the debugger
> doesn't know where it is, then it's probably let to prologue analysis
> to do the unwinding. Except that it cannot determine where the prologue
> is... In that case, I see that the i386 unwinder assumes that the frame
> base can be deduced from the SP and the SP offset. Unfortunately,
> this SP offset can only be deduced from prologue analysis. Catch 22?
> 
> Mark,
> 
> What do you think of this (untested) patch? We we couldn't find
> the function start address, the safest seems to be relying on ebp.

I think this diff makes sense.  However, I'm pretty sure there are
Linux systems out there where this will make things worse :(.  In
particular, on kernels with a vsyscall page buit without the stub
shared library for that page, this change will systematically skip a
frame.  And that frame is quite crucial since it is the frame for the
libc system call stub, so it will be hard for a user to find out in
what system call the program is blocked on.

I have no idea though how many people are still runing those kernels.
That number might be very low enough for us not to care.

Mark

Re: How to tell gdb about dlls using remote protocol

[Daniel Jacobowitz]

On Wed, Feb 07, 2007 at 11:14:27PM +0100, Mark Kettenis wrote:
> I think this diff makes sense.  However, I'm pretty sure there are
> Linux systems out there where this will make things worse :(.  In
> particular, on kernels with a vsyscall page buit without the stub
> shared library for that page, this change will systematically skip a
> frame.  And that frame is quite crucial since it is the frame for the
> libc system call stub, so it will be hard for a user to find out in
> what system call the program is blocked on.
> 
> I have no idea though how many people are still runing those kernels.
> That number might be very low enough for us not to care.

For what it's worth, I think it is.  And, if it isn't, it would be
straightforward to add a custom frame sniffer to i386-linux-tdep.c
to recognize that case.  I don't know what the affected versions
are, though.

-- 
Daniel Jacobowitz
CodeSourcery

Re: How to tell gdb about dlls using remote protocol

[Mark Kettenis]

> Date: Wed, 7 Feb 2007 17:16:44 -0500
> From: Daniel Jacobowitz <drow@false.org>
> 
> On Wed, Feb 07, 2007 at 11:14:27PM +0100, Mark Kettenis wrote:
> > I think this diff makes sense.  However, I'm pretty sure there are
> > Linux systems out there where this will make things worse :(.  In
> > particular, on kernels with a vsyscall page buit without the stub
> > shared library for that page, this change will systematically skip a
> > frame.  And that frame is quite crucial since it is the frame for the
> > libc system call stub, so it will be hard for a user to find out in
> > what system call the program is blocked on.
> > 
> > I have no idea though how many people are still runing those kernels.
> > That number might be very low enough for us not to care.
> 
> For what it's worth, I think it is.  And, if it isn't, it would be
> straightforward to add a custom frame sniffer to i386-linux-tdep.c
> to recognize that case.  I don't know what the affected versions
> are, though.

Me neither.  However, Joel's diff has a problem: it makes the
signull.exp tests fail.  They explicitly test calling a null pointer,
and that case is now caught by Joel's check.  Skipping a frame in that
case is not acceptable to me.

I'm currently testing chaning Joel's original:

      else if (cache->pc == 0)

into:

      else if (cache->pc == 0 && frame_pc_unwind (next_frame) != 0)

What do you think of that?

Mark

Re: How to tell gdb about dlls using remote protocol

[Daniel Jacobowitz]

On Thu, Feb 08, 2007 at 10:14:37PM +0100, Mark Kettenis wrote:
> Me neither.  However, Joel's diff has a problem: it makes the
> signull.exp tests fail.  They explicitly test calling a null pointer,
> and that case is now caught by Joel's check.  Skipping a frame in that
> case is not acceptable to me.
> 
> I'm currently testing chaning Joel's original:
> 
>       else if (cache->pc == 0)
> 
> into:
> 
>       else if (cache->pc == 0 && frame_pc_unwind (next_frame) != 0)
> 
> What do you think of that?

Can we actually check for a failure to find the start address?  Using
pc == 0 is unfortunate - we keep encountering people who link code at
zero.

Oh, I guess there's no way; we already use the pc != 0 check in the
same function.

-- 
Daniel Jacobowitz
CodeSourcery

Re: How to tell gdb about dlls using remote protocol

[Joel Brobecker]

> >       else if (cache->pc == 0 && frame_pc_unwind (next_frame) != 0)
> > 
> > What do you think of that?
> 
> Can we actually check for a failure to find the start address?  Using
> pc == 0 is unfortunate - we keep encountering people who link code at
> zero.

The problem is in the way we designed frame_func_unwind: It returns
zero if we couldn't find the function start, which cannot be
differentiated from the case where the function actually starts at
address zero (which I didn't think I would ever encounter).

I think in the long term, we might want to adjust frame_func_unwind
to return a be something like:

   int frame_func_unwind (struct frame_info *fi, CORE_ADDR *addr)

The return value would be non-zero only if the function could be
found.

In the short-term, Mark's proposal seems a good way to make progress
for little cost.

> Oh, I guess there's no way; we already use the pc != 0 check in the
> same function.

Yep, same issue.

-- 
Joel