gdbserver ptrace not doing waitpid after attaching to threads?

gdbserver ptrace not doing waitpid after attaching to threads?

[Doug Evans]

Hi.

There's something about gdbserver's use of ptrace that I don't understand
and I'm hoping someone can help.

AIUI, after attaching to a thread (or process) one must wait until
the thread has stopped before one can, for example, read regs.
[Well, one can cheat and assume enough time passes before trying to read
regs, but that's cheating. :-)]
If one tries to read regs too soon ptrace will return with ESRCH.

This is the behaviour I'm seeing in a version of gdbserver I'm hacking on -
my register reads are intermittently failing with ESRCH.  If I add code to
wait for SIGSTOP after attaching to all the threads then my register reads
start working.

However, gdb+gdbserver works just fine (obviously), and in fact there is
code in gdbserver to explicitly skip/ignore the initial SIGSTOP in
linux-low.c:linux_attach_lwp:

  /* The next time we wait for this LWP we'll see a SIGSTOP as PTRACE_ATTACH
     brings it to a halt.  We should ignore that SIGSTOP and resume the process
                              ^^^^^^^^^^^^^^^^^^^^^^^^^^
     (unless this is the first process, in which case the flag will be cleared
     in linux_attach).

     On the other hand, if we are currently trying to stop all threads, we
     should treat the new thread as if we had sent it a SIGSTOP.  This works
     because we are guaranteed that add_process added us to the end of the
     list, and so the new thread has not yet reached wait_for_sigstop (but
     will).  */
  if (! stopping_threads)
    new_process->stop_expected = 1;

and there is code linux_attach to undo this because for main thread
we *do* want to wait for the initial SIGSTOP after attaching to the process:

  /* Don't ignore the initial SIGSTOP if we just attached to this process.
     It will be collected by wait shortly.  */
  process = (struct process_info *) find_inferior_id (&all_processes, pid);
  process->stop_expected = 0;

Anyone know what's going on here?
Why the different treatment for the main thread vs the others?
Why isn't gdbserver waiting for a SIGSTOP after attaching to threads?

I've run gdbserver under strace and see that gdbserver is essentially doing
ptrace (ATTACH), ptrace (SETOPTIONS), ptrace (GETREGS) on threads
with no intervening waitpid.

I wonder if gdb is working just because of luck: maybe enough packets go back
and forth before gdb tries to read regs that all threads have stopped
by the time gdb needs them to have stopped.
In my hacked gdbserver I'm doing a register read almost immediately
after attaching.

Re: gdbserver ptrace not doing waitpid after attaching to threads?

[Daniel Jacobowitz]

On Fri, Dec 12, 2008 at 01:13:15PM -0800, Doug Evans wrote:
> Why isn't gdbserver waiting for a SIGSTOP after attaching to threads?

It's supposed to be.  At one point it definitely was, though it might
have broken recently and been saved by luck.  We attached the thread
but did not mark it stopped 

> I've run gdbserver under strace and see that gdbserver is essentially doing
> ptrace (ATTACH), ptrace (SETOPTIONS), ptrace (GETREGS) on threads
> with no intervening waitpid.

Is this current source?  A vaguely current kernel?  It shouldn't be
doing any ATTACH; instead it should be trusting SETOPTIONS to deliver
new threads to it via wait.

-- 
Daniel Jacobowitz
CodeSourcery

Re: gdbserver ptrace not doing waitpid after attaching to threads?

[Doug Evans]

On Fri, Dec 12, 2008 at 1:28 PM, Daniel Jacobowitz <drow@false.org> wrote:
>
> On Fri, Dec 12, 2008 at 01:13:15PM -0800, Doug Evans wrote:
> > Why isn't gdbserver waiting for a SIGSTOP after attaching to threads?
>
> It's supposed to be.  At one point it definitely was, though it might
> have broken recently and been saved by luck.  We attached the thread
> but did not mark it stopped
>
> > I've run gdbserver under strace and see that gdbserver is essentially doing
> > ptrace (ATTACH), ptrace (SETOPTIONS), ptrace (GETREGS) on threads
> > with no intervening waitpid.
>
> Is this current source?  A vaguely current kernel?  It shouldn't be
> doing any ATTACH; instead it should be trusting SETOPTIONS to deliver
> new threads to it via wait.

source = cvs head
kernel = 2.6.18-ish and 2.6.27-ish

I mightn't have been clear enough btw.  I'm observing these things
from doing an attach in gdb.  Example below.

Heh.  For reference sake, with cvs head in fc10 (2.6.27-ish) I just saw this:

ptrace(PTRACE_ATTACH, 12293, 0, 0)      = 0
ptrace(0x4200 /* PTRACE_??? */, 12293, 0, 0x8) = -1 ESRCH (No such process)

This is the first attach that gdbserver does, the one for the process.
Might it be possible that one has to wait for SIGSTOP even before
ptrace (SETOPTIONS)? :-)

Observed with:

bash1$ ./foo.x & # simple program that creates 4 threads that just burn cpu
12293
bash1$ strace gdbserver --multi :1234

bash2$ gdb foo.x
(gdb) tar extended-remote :1234
(gdb) attach 12293

The relevent output of strace is (from running the example on 2.6.27):

ptrace(PTRACE_ATTACH, 12293, 0, 0)      = 0
ptrace(0x4200 /* PTRACE_??? */, 12293, 0, 0x8) = -1 ESRCH (No such process)
[...]
waitpid(-1, [{WIFSTOPPED(s) && WSTOPSIG(s) == SIGSTOP}], WNOHANG) = 12293
ptrace(PTRACE_GETREGS, 12293, 0, 0x9c36248) = 0
ptrace(PTRACE_GETFPXREGS, 12293, 0, 0x9c36290) = 0
ptrace(PTRACE_GETFPREGS, 12293, 0, 0x9c36498) = 0
[...]
ptrace(PTRACE_ATTACH, 12297, 0, 0)      = 0
ptrace(0x4200 /* PTRACE_??? */, 12297, 0, 0x8) = -1 ESRCH (No such process)
[...]
ptrace(PTRACE_ATTACH, 12296, 0, 0)      = 0
ptrace(0x4200 /* PTRACE_??? */, 12296, 0, 0x8) = -1 ESRCH (No such process)
[...]
ptrace(PTRACE_ATTACH, 12295, 0, 0)      = 0
ptrace(0x4200 /* PTRACE_??? */, 12295, 0, 0x8) = 0
[...]
ptrace(PTRACE_ATTACH, 12294, 0, 0)      = 0
ptrace(0x4200 /* PTRACE_??? */, 12294, 0, 0x8) = -1 ESRCH (No such process)
[...]
ptrace(PTRACE_GETREGS, 12294, 0, 0x9c37c50) = 0
ptrace(PTRACE_GETFPXREGS, 12294, 0, 0x9c37c98) = 0
ptrace(PTRACE_GETFPREGS, 12294, 0, 0x9c37ea0) = 0

There are no waitpid calls after the one you see above.
I may be missing something, but it seems the registers for 12294 are
being read before waiting for SIGSTOP.
A lot happens between them so presumably that's why gdb is getting away with it.
Note that one ptrace (SETOPTIONS) did work, the one for thread 12295,
the other's didn't.

For clarification, your comment:

> It shouldn't be
> doing any ATTACH; instead it should be trusting SETOPTIONS to deliver
> new threads to it via wait.

was for the case where gdbserver starts the program, and not for the
case where gdbserver attaches to the program, right?  In the latter
case it seems like gdbserver does indeed need to attach to all the
threads when gdb connects.  That's what it's doing, it's just not
waiting for SIGSTOP (IIUC).

Re: gdbserver ptrace not doing waitpid after attaching to threads?

[Daniel Jacobowitz]

On Fri, Dec 12, 2008 at 03:06:44PM -0800, Doug Evans wrote:
> I mightn't have been clear enough btw.  I'm observing these things
> from doing an attach in gdb.  Example below.

Oh!  Well then.  Sounds like it's gotten broken.  I know we've fixed
this at least once before.

> This is the first attach that gdbserver does, the one for the process.
> Might it be possible that one has to wait for SIGSTOP even before
> ptrace (SETOPTIONS)? :-)

Yes, that's correct.  A bit silly, but correct.

> For clarification, your comment:
> 
> > It shouldn't be
> > doing any ATTACH; instead it should be trusting SETOPTIONS to deliver
> > new threads to it via wait.
> 
> was for the case where gdbserver starts the program, and not for the
> case where gdbserver attaches to the program, right?

Well, it's for any thread created once the program is running under
gdbserver control.  Which amounts to the same thing here.

-- 
Daniel Jacobowitz
CodeSourcery

Re: gdbserver ptrace not doing waitpid after attaching to threads?

[Doug Evans]

On Fri, Dec 12, 2008 at 3:16 PM, Daniel Jacobowitz <drow@false.org> wrote:
>> For clarification, your comment:
>>
>> > It shouldn't be
>> > doing any ATTACH; instead it should be trusting SETOPTIONS to deliver
>> > new threads to it via wait.
>>
>> was for the case where gdbserver starts the program, and not for the
>> case where gdbserver attaches to the program, right?
>
> Well, it's for any thread created once the program is running under
> gdbserver control.  Which amounts to the same thing here.

Right, new threads could be created after the SETOPTIONS goes into
affect.  But I'm still unclear whether SETOPTIONS will deliver threads
that have already been created and exist when gdbserver attaches.

Re: gdbserver ptrace not doing waitpid after attaching to threads?

[Daniel Jacobowitz]

On Fri, Dec 12, 2008 at 03:21:36PM -0800, Doug Evans wrote:
> Right, new threads could be created after the SETOPTIONS goes into
> affect.  But I'm still unclear whether SETOPTIONS will deliver threads
> that have already been created and exist when gdbserver attaches.

Nope.  You need a different mechanism for that; either libthread_db as
today, or /proc/pid/task/.

-- 
Daniel Jacobowitz
CodeSourcery