problem remote debugging

problem remote debugging

[Brendan Miller]

I'm having a problem with remote debugging where when debugging
locally I will launch fine, but when remotely debugging my program
will fail to open a certain text file, then segfault.

Both the host and client are running x86 RHEL4. The gdb version is
6.3.0.0-1.153.el4rh.

In the example below, I am trying to remote debug a fairly complicated
service that has it's own user "a_user".

On the remote machine I start gdbserver with:

sudo -u a_user gdbserver host:4000 my_binary

On the local machine I run:
gdb my_binary

Then within gdb I type:

target remote my_remote_machine:4000
run

Then I get the "failed to open file error" followed by a segfault.

On the other hand if I run this on the remote machine:
sudo -u a_user gdb my_binary
run

everything is fine.

Is one of my steps for remote debugging wrong? Would permissions be
different under remote debugging for some reason, and that would
explain why I can't open that file under remote debugging?

The service I"m trying to debug is heavily multithreaded, if that
makes a difference.

Re: problem remote debugging

[teawater]

On Tue, Feb 24, 2009 at 08:59, Brendan Miller <catphive@catphive.net> wrote:
> I'm having a problem with remote debugging where when debugging
> locally I will launch fine, but when remotely debugging my program
> will fail to open a certain text file, then segfault.
>
> Both the host and client are running x86 RHEL4. The gdb version is
> 6.3.0.0-1.153.el4rh.
>
> In the example below, I am trying to remote debug a fairly complicated
> service that has it's own user "a_user".
>
> On the remote machine I start gdbserver with:
>
> sudo -u a_user gdbserver host:4000 my_binary
>
> On the local machine I run:
> gdb my_binary
>
> Then within gdb I type:
>
> target remote my_remote_machine:4000

Are you sure connect to my_remote_machine:4000 is OK?


> run
>
> Then I get the "failed to open file error" followed by a segfault.
>
> On the other hand if I run this on the remote machine:
> sudo -u a_user gdb my_binary
> run
>
> everything is fine.
>
> Is one of my steps for remote debugging wrong? Would permissions be
> different under remote debugging for some reason, and that would
> explain why I can't open that file under remote debugging?
>
> The service I"m trying to debug is heavily multithreaded, if that
> makes a difference.
>

Re: problem remote debugging

[Daniel Jacobowitz]

On Mon, Feb 23, 2009 at 04:59:48PM -0800, Brendan Miller wrote:
> I'm having a problem with remote debugging where when debugging
> locally I will launch fine, but when remotely debugging my program
> will fail to open a certain text file, then segfault.
> 
> Both the host and client are running x86 RHEL4. The gdb version is
> 6.3.0.0-1.153.el4rh.

A frequent cause of this problem is different patch levels of
libraries installed on the two systems.  If they don't exactly match,
you need a copy on the host of the target's libraries, and to use set
sysroot.  Otherwise GDB may place internal breakpoints in the wrong
locations, leading to mayhem.

Also, this is a very old GDB - I always recommend trying the latest
(GDB and gdbserver).

-- 
Daniel Jacobowitz
CodeSourcery

Re: problem remote debugging

[Brendan Miller]

On Mon, Feb 23, 2009 at 6:32 PM, Daniel Jacobowitz <drow@false.org> wrote:
> On Mon, Feb 23, 2009 at 04:59:48PM -0800, Brendan Miller wrote:
>> I'm having a problem with remote debugging where when debugging
>> locally I will launch fine, but when remotely debugging my program
>> will fail to open a certain text file, then segfault.
>>
>> Both the host and client are running x86 RHEL4. The gdb version is
>> 6.3.0.0-1.153.el4rh.
>
> A frequent cause of this problem is different patch levels of
> libraries installed on the two systems.  If they don't exactly match,
> you need a copy on the host of the target's libraries, and to use set
> sysroot.  Otherwise GDB may place internal breakpoints in the wrong
> locations, leading to mayhem.

I'm not sure if that makes sense. What happened was a file failed to
be read, which to me implies different behavior given the same
binaries executed on the same machine, depending on if in local or
remote debugging mode. This was followed by a segfault which may or
may not be related. I can see how the gdb client would crash if I
didn't have the proper debugging symbols on the client, but could that
really effect other behavior on the remote machine?

I did originally copy over all .so's related to the service directly
and included their path in LD_LIBRARY_PATH, although I may have a
*slightly* different glibc or something.

>
> Also, this is a very old GDB - I always recommend trying the latest
> (GDB and gdbserver).

Is there a known bug that was fixed that would resolve this? That
would be a lot of work, given that I have no newer RPM's and there
isn't a compiler on my remote machine. I'll try it if there's a reason
to think it would work, or if I run out of other ideas.

This really looks to me like executable behavior is different under
local and remote debugging, so I was hoping someone knowledgeable
about GDB could comment on if that is possible, or if there are known
bugs here. Things like mismatched symbols don't seem like they should
effect runtime behavior on the remote machine.

Re: problem remote debugging

[Daniel Jacobowitz]

On Tue, Feb 24, 2009 at 09:01:11AM -0800, Brendan Miller wrote:
> I'm not sure if that makes sense. What happened was a file failed to
> be read, which to me implies different behavior given the same
> binaries executed on the same machine, depending on if in local or
> remote debugging mode. This was followed by a segfault which may or
> may not be related. I can see how the gdb client would crash if I
> didn't have the proper debugging symbols on the client, but could that
> really effect other behavior on the remote machine?

Yes, really.  I've just tried to explain this... GDB sets breakpoints
in the target automatically, and reads debug data from the target.  If
symbols do not match, then it will set the breakpoints at the wrong
location (which might even be data rather than code), and it will read
incorrect debug information and make decisions based on that.

> > Also, this is a very old GDB - I always recommend trying the latest
> > (GDB and gdbserver).
> 
> Is there a known bug that was fixed that would resolve this?

Well, for one thing it'll try to warn you if your libraries don't
match.

-- 
Daniel Jacobowitz
CodeSourcery

Re: problem remote debugging

[Brendan Miller]

On Tue, Feb 24, 2009 at 9:08 AM, Daniel Jacobowitz <drow@false.org> wrote:
> On Tue, Feb 24, 2009 at 09:01:11AM -0800, Brendan Miller wrote:
> Yes, really.  I've just tried to explain this... GDB sets breakpoints
> in the target automatically, and reads debug data from the target.  If
> symbols do not match, then it will set the breakpoints at the wrong
> location (which might even be data rather than code), and it will read
> incorrect debug information and make decisions based on that.

Intuitively that seemed wrong to me because if a breakpoint is in the
wrong place, or outside the text segment, then I'd just expect it to
break in a different place, or not at all. I wouldn't expect runtime
execution of the code to take a different path. However, it sounds
like you understand what's going on in the internals a lot better than
I do, so I'll take your word for it.

>
>> > Also, this is a very old GDB - I always recommend trying the latest
>> > (GDB and gdbserver).
>>
>> Is there a known bug that was fixed that would resolve this?
>
> Well, for one thing it'll try to warn you if your libraries don't
> match.

Ok... I'll give that a shot. That would actually be pretty useful
because I'm dealing with a lot of hard to track version skew in my
development environment.

Re: problem remote debugging

[Paul Pluzhnikov]

On Tue, Feb 24, 2009 at 11:51 AM, Brendan Miller <catphive@catphive.net> wrote:

> Intuitively that seemed wrong to me because if a breakpoint is in the
> wrong place, or outside the text segment, then I'd just expect it to
> break in a different place, or not at all.

What if a "breakpoint" is set in read-only data or .bss of the program?
Imagine that an arbitrary data byte in your program is replaced with 0xCC.
Will your program surivive any such replacement?

For example:

  size_t foo()
  {
    static char *p = NULL;
    if (!p) { p = strdup("Hello"); }
    return strlen(p);
  }

What will this routine return if "p" is overwritten by 0x000000CC
by GDB before the execution starts?

Cheers,
-- 
Paul Pluzhnikov