Does HEAD support non-stop with 'gdbserver --multi' on Linux?

Does HEAD support non-stop with 'gdbserver --multi' on Linux?

[Marc Khouzam]

Hi,

In the last couple months there has been a set of patches about
non-stop/multi-process 
for gdbserver and GDB for Linux from Pedro.  Sadly, I wasn't able to
keep track of what went
in and all the functionality they supported.

I'm trying HEAD with:
- gdbserver --multi
- non-stop for a multi-threaded app
- on Linux

Is that supported yet?
Because I'm not getting the correct list of threads (only in the
non-stop case).

Thanks

Marc

Re: Does HEAD support non-stop with 'gdbserver --multi' on Linux?

[Pedro Alves]

Hi Marc,

On Thursday 30 April 2009 20:39:47, Marc Khouzam wrote:

> In the last couple months there has been a set of patches about
> non-stop/multi-process 
> for gdbserver and GDB for Linux from Pedro.  Sadly, I wasn't able to
> keep track of what went
> in and all the functionality they supported.
> 
> I'm trying HEAD with:
> - gdbserver --multi

Note that --multi is not about multi-process in the same sense you're
talking about.  It is a preexisting option that means that gdbserver 
will stay persistent even when the inferior exits.  It also enables
starting up gdbserver without giving it a process to start with.
You'll need to connect with "target extended-remote" in that case.

The gdbserver multiprocess support is in head already, but needs
more GDB side work to be useable.

> - non-stop for a multi-threaded app
> - on Linux
> 
> Is that supported yet?
> Because I'm not getting the correct list of threads (only in the
> non-stop case).

What exactly are you seeing?  I just run a few non-stop test
(mi-nonstop.exp, mi-nsintrall.exp and ns-nsmoribund.exp tests)
against linux x86-64 gdbserver head, and they passed cleanly for
me, so *something* is working.  :-)

-- 
Pedro Alves

RE: Does HEAD support non-stop with 'gdbserver --multi' on Linux?

[Marc Khouzam]

 

> -----Original Message-----
> From: Pedro Alves [mailto:pedro@codesourcery.com] 
> Sent: Thursday, April 30, 2009 3:50 PM
> To: gdb@sourceware.org
> Cc: Marc Khouzam
> Subject: Re: Does HEAD support non-stop with 'gdbserver 
> --multi' on Linux?
> 
> Hi Marc,
> 
> On Thursday 30 April 2009 20:39:47, Marc Khouzam wrote:
> 
> > In the last couple months there has been a set of patches about
> > non-stop/multi-process 
> > for gdbserver and GDB for Linux from Pedro.  Sadly, I wasn't able to
> > keep track of what went
> > in and all the functionality they supported.
> > 
> > I'm trying HEAD with:
> > - gdbserver --multi
> 
> Note that --multi is not about multi-process in the same sense you're
> talking about.  It is a preexisting option that means that gdbserver 
> will stay persistent even when the inferior exits.  It also enables
> starting up gdbserver without giving it a process to start with.
> You'll need to connect with "target extended-remote" in that case.

Yep.  I use that to do a Remote Attach debug session.


> The gdbserver multiprocess support is in head already, but needs
> more GDB side work to be useable.

I don't want to waste your time if it is not ready yet.
But I put the details below, just in case.

> > - non-stop for a multi-threaded app
> > - on Linux
> > 
> > Is that supported yet?
> > Because I'm not getting the correct list of threads (only in the
> > non-stop case).
> 
> What exactly are you seeing?  I just run a few non-stop test
> (mi-nonstop.exp, mi-nsintrall.exp and ns-nsmoribund.exp tests)
> against linux x86-64 gdbserver head, and they passed cleanly for
> me, so *something* is working.  :-)

It seems no new thread is listed by GDB.

Thanks

Marc

Details
=======

My program has the main thread start a second thread which prints every
second.
The main thread then does a pthread_join
I let both threads run and ask GDB for the list of threads using 'info
thread'.
Only the main thread is reported as you can see below.

== I first start gdbserver like this: ==
> gdbserver --multi :10008&

Then I start GDB:
> gdb
GNU gdb (GDB) 6.8.50.20090430-cvs
Copyright (C) 2009 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
<http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show
copying"
and "show warranty" for details.
This GDB was configured as "i686-pc-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
(gdb)  file a.out
Reading symbols from /local/home/lmckhou/testing/a.out...done.
(gdb) l 1
1       #include <stdio.h>
2       #include <stdlib.h>
3       #include <pthread.h>
4       #include <unistd.h>
5
6       void *thread_exec(void *ptr)
7       {
8           for (int i=0;i<30;i++) {
9               printf("thread 2 prints: %d\n" , i);
10              sleep(1);
(gdb) l
11          }
12      }
13
14      int main()
15      {
16
17          sleep(15);
18
19          pthread_t thread2;
20          char *message2 = "Thread 2";
(gdb) l
21
22          pthread_create(&thread2, NULL, thread_exec, (void*)
message2);
23
24          pthread_join(thread2, NULL);
25
26          printf("Thread 2 finished\n");
27
28          return 0;
29      }
(gdb) b 22
Breakpoint 1 at 0x80485b8: file MultiThread.cc, line 22.
(gdb) set target-async on
(gdb) set pagination off
(gdb) set non-stop on
(gdb) target extended-remote localhost:10008
Remote debugging using localhost:10008

=== Here I start the a.out program in another ==
===  window and it gets pid 30668.            ==

(gdb)  attach 30668
Attached to process 30668
[New Thread 30668.30668]
Reading symbols from /lib/libpthread.so.0...done.
Loaded symbols for /lib/libpthread.so.0
Reading symbols from /usr/lib/libstdc++.so.6...done.
Loaded symbols for /usr/lib/libstdc++.so.6
Reading symbols from /lib/libm.so.6...done.
Loaded symbols for /lib/libm.so.6
Reading symbols from /lib/libgcc_s.so.1...done.
Loaded symbols for /lib/libgcc_s.so.1
Reading symbols from /lib/libc.so.6...done.
Loaded symbols for /lib/libc.so.6
Reading symbols from /lib/ld-linux.so.2...done.
Loaded symbols for /lib/ld-linux.so.2
0xffffe410 in __kernel_vsyscall ()
(gdb) c&
Continuing.
(gdb) 
Breakpoint 1, main () at MultiThread.cc:22
22          pthread_create(&thread2, NULL, thread_exec, (void*)
message2);
info th
* 1 Thread 30668.30668  main () at MultiThread.cc:22
(gdb) c&
Continuing.

== This continues past the creation of the second thread ==
== I actually see the printouts of the second thread     ==
== in the other window, every second                     ==

(gdb) info th
* 1 Thread 30668.30668  (running)

== Thread 2 is not shown ==
== This also happens if I have thread 1 stopped at line 24 ==

RE: Does HEAD support non-stop with 'gdbserver --multi' on Linux?

[Marc Khouzam]

 

> -----Original Message-----
> From: Pedro Alves [mailto:pedro@codesourcery.com] 
> Sent: Thursday, April 30, 2009 5:45 PM
> To: Marc Khouzam
> Cc: gdb@sourceware.org
> Subject: Re: Does HEAD support non-stop with 'gdbserver 
> --multi' on Linux?
> 
> On Thursday 30 April 2009 21:18:58, Marc Khouzam wrote:
> 
> > > What exactly are you seeing?  I just run a few non-stop test
> > > (mi-nonstop.exp, mi-nsintrall.exp and ns-nsmoribund.exp tests)
> > > against linux x86-64 gdbserver head, and they passed cleanly for
> > > me, so *something* is working.  :-)
> > 
> > It seems no new thread is listed by GDB.
> 
> Hmmm, that should work.  This looks like another manifestation
> of PR threads/10048.  Does this make a difference?

Yes, this fixed the problem.
My Eclipse is still mis-behaving a bit, but that is probably
my code.  The 'info thread' does show all threads.

Heads up on a breakpoint mis-behavior coming in a separate mail :-)

Thanks!

> 
> -- 
> Pedro Alves
> 
> 2009-04-30  Pedro Alves  <pedro@codesourcery.com>
> 
> 	* linux-low.c (must_set_ptrace_flags): Delete.
> 	(linux_create_inferior): Set 
> `lwp->must_set_ptrace_flags' instead
> 	of the global.
> 	(linux_attach_lwp_1): Don't set PTRACE_SETOPTIONS here.  Set
> 	`lwp->must_set_ptrace_flags' instead.
> 	(linux_wait_for_event_1): Set ptrace options here.
> 	(linux_wait_1): ... not here.
> 
> ---
>  gdb/gdbserver/linux-low.c |   27 ++++++++++++++-------------
>  gdb/gdbserver/linux-low.h |    4 ++++
>  2 files changed, 18 insertions(+), 13 deletions(-)
> 
> Index: src/gdb/gdbserver/linux-low.c
> ===================================================================
> --- src.orig/gdb/gdbserver/linux-low.c	2009-04-12 
> 22:44:01.000000000 +0100
> +++ src/gdb/gdbserver/linux-low.c	2009-04-30 
> 22:38:20.000000000 +0100
> @@ -109,8 +109,6 @@ int stopping_threads;
>  /* FIXME make into a target method?  */
>  int using_threads = 1;
>  
> -static int must_set_ptrace_flags;
> -
>  /* This flag is true iff we've just created or attached to our first
>     inferior but it has not stopped yet.  As soon as it does, we need
>     to call the low target's arch_setup callback.  Doing this only on
> @@ -309,7 +307,7 @@ add_lwp (ptid_t ptid)
>  static int
>  linux_create_inferior (char *program, char **allargs)
>  {
> -  void *new_lwp;
> +  struct lwp_info *new_lwp;
>    int pid;
>    ptid_t ptid;
>  
> @@ -344,7 +342,7 @@ linux_create_inferior (char *program, ch
>    ptid = ptid_build (pid, pid, 0);
>    new_lwp = add_lwp (ptid);
>    add_thread (ptid, new_lwp);
> -  must_set_ptrace_flags = 1;
> +  new_lwp->must_set_ptrace_flags = 1;
>  
>    return pid;
>  }
> @@ -373,10 +371,6 @@ linux_attach_lwp_1 (unsigned long lwpid,
>  	       strerror (errno), errno);
>      }
>  
> -  /* FIXME: This intermittently fails.
> -     We need to wait for SIGSTOP first.  */
> -  ptrace (PTRACE_SETOPTIONS, lwpid, 0, PTRACE_O_TRACECLONE);
> -
>    if (initial)
>      /* NOTE/FIXME: This lwp might have not been the tgid.  */
>      ptid = ptid_build (lwpid, lwpid, 0);
> @@ -392,6 +386,11 @@ linux_attach_lwp_1 (unsigned long lwpid,
>    new_lwp = (struct lwp_info *) add_lwp (ptid);
>    add_thread (ptid, new_lwp);
>  
> +
> +  /* We need to wait for SIGSTOP before being able to make the next
> +     ptrace call on this LWP.  */
> +  new_lwp->must_set_ptrace_flags = 1;
> +
>    /* The next time we wait for this LWP we'll see a SIGSTOP 
> as PTRACE_ATTACH
>       brings it to a halt.
>  
> @@ -986,6 +985,13 @@ linux_wait_for_event_1 (ptid_t ptid, int
>  	  continue;
>  	}
>  
> +      if (event_child->must_set_ptrace_flags)
> +	{
> +	  ptrace (PTRACE_SETOPTIONS, lwpid_of (event_child),
> +		  0, PTRACE_O_TRACECLONE);
> +	  event_child->must_set_ptrace_flags = 0;
> +	}
> +
>        if (WIFSTOPPED (*wstat)
>  	  && WSTOPSIG (*wstat) == SIGSTOP
>  	  && event_child->stop_expected)
> @@ -1248,11 +1254,6 @@ retry:
>  
>    lwp = get_thread_lwp (current_inferior);
>  
> -  if (must_set_ptrace_flags)
> -    {
> -      ptrace (PTRACE_SETOPTIONS, lwpid_of (lwp), 0, 
> PTRACE_O_TRACECLONE);
> -      must_set_ptrace_flags = 0;
> -    }
>    /* If we are waiting for a particular child, and it exited,
>       linux_wait_for_event will return its exit status.  Similarly if
>       the last child exited.  If this is not the last child, however,
> Index: src/gdb/gdbserver/linux-low.h
> ===================================================================
> --- src.orig/gdb/gdbserver/linux-low.h	2009-04-12 
> 22:44:01.000000000 +0100
> +++ src/gdb/gdbserver/linux-low.h	2009-04-30 
> 22:33:09.000000000 +0100
> @@ -146,6 +146,10 @@ struct lwp_info
>       was a single-step.  */
>    int stepping;
>  
> +  /* If this flag is set, we need to set the event request flags the
> +     next time we see this LWP stop.  */
> +  int must_set_ptrace_flags;
> +
>    /* If this is non-zero, it points to a chain of signals 
> which need to
>       be delivered to this process.  */
>    struct pending_signals *pending_signals;
> 
> 

RE: Does HEAD support non-stop with 'gdbserver --multi' on Linux?

[Marc Khouzam]

 

> -----Original Message-----
> From: gdb-owner@sourceware.org 
> [mailto:gdb-owner@sourceware.org] On Behalf Of Marc Khouzam
> Sent: Friday, May 01, 2009 2:40 PM
> To: Pedro Alves
> Cc: gdb@sourceware.org
> Subject: RE: Does HEAD support non-stop with 'gdbserver 
> --multi' on Linux?
> 
>  
> 
> > -----Original Message-----
> > From: Pedro Alves [mailto:pedro@codesourcery.com] 
> > Sent: Thursday, April 30, 2009 5:45 PM
> > To: Marc Khouzam
> > Cc: gdb@sourceware.org
> > Subject: Re: Does HEAD support non-stop with 'gdbserver 
> > --multi' on Linux?
> > 
> > On Thursday 30 April 2009 21:18:58, Marc Khouzam wrote:
> > 
> > > > What exactly are you seeing?  I just run a few non-stop test
> > > > (mi-nonstop.exp, mi-nsintrall.exp and ns-nsmoribund.exp tests)
> > > > against linux x86-64 gdbserver head, and they passed cleanly for
> > > > me, so *something* is working.  :-)
> > > 
> > > It seems no new thread is listed by GDB.
> > 
> > Hmmm, that should work.  This looks like another manifestation
> > of PR threads/10048.  Does this make a difference?
> 
> Yes, this fixed the problem.
> My Eclipse is still mis-behaving a bit, but that is probably
> my code.  The 'info thread' does show all threads.

I just noticed that after this fix, although the threads are properly
listed, the creation events only happen when the info thread
command is given.

I saw that with non-stop, when doing native linux debugging, there
is a conscious decision to trigger thread events as soon as possible
(from a comment of linux-nat.c:linux_handle_extended_wait()).
However, this does not happen when using gdbserver.

Is this something that is not ready yet, or a bug, or ... ?

Thanks

Marc



 
> Thanks!
> 
> > 
> > -- 
> > Pedro Alves
> > 
> > 2009-04-30  Pedro Alves  <pedro@codesourcery.com>
> > 
> > 	* linux-low.c (must_set_ptrace_flags): Delete.
> > 	(linux_create_inferior): Set 
> > `lwp->must_set_ptrace_flags' instead
> > 	of the global.
> > 	(linux_attach_lwp_1): Don't set PTRACE_SETOPTIONS here.  Set
> > 	`lwp->must_set_ptrace_flags' instead.
> > 	(linux_wait_for_event_1): Set ptrace options here.
> > 	(linux_wait_1): ... not here.
> > 
> > ---
> >  gdb/gdbserver/linux-low.c |   27 ++++++++++++++-------------
> >  gdb/gdbserver/linux-low.h |    4 ++++
> >  2 files changed, 18 insertions(+), 13 deletions(-)
> > 
> > Index: src/gdb/gdbserver/linux-low.c
> > ===================================================================
> > --- src.orig/gdb/gdbserver/linux-low.c	2009-04-12 
> > 22:44:01.000000000 +0100
> > +++ src/gdb/gdbserver/linux-low.c	2009-04-30 
> > 22:38:20.000000000 +0100
> > @@ -109,8 +109,6 @@ int stopping_threads;
> >  /* FIXME make into a target method?  */
> >  int using_threads = 1;
> >  
> > -static int must_set_ptrace_flags;
> > -
> >  /* This flag is true iff we've just created or attached to 
> our first
> >     inferior but it has not stopped yet.  As soon as it 
> does, we need
> >     to call the low target's arch_setup callback.  Doing 
> this only on
> > @@ -309,7 +307,7 @@ add_lwp (ptid_t ptid)
> >  static int
> >  linux_create_inferior (char *program, char **allargs)
> >  {
> > -  void *new_lwp;
> > +  struct lwp_info *new_lwp;
> >    int pid;
> >    ptid_t ptid;
> >  
> > @@ -344,7 +342,7 @@ linux_create_inferior (char *program, ch
> >    ptid = ptid_build (pid, pid, 0);
> >    new_lwp = add_lwp (ptid);
> >    add_thread (ptid, new_lwp);
> > -  must_set_ptrace_flags = 1;
> > +  new_lwp->must_set_ptrace_flags = 1;
> >  
> >    return pid;
> >  }
> > @@ -373,10 +371,6 @@ linux_attach_lwp_1 (unsigned long lwpid,
> >  	       strerror (errno), errno);
> >      }
> >  
> > -  /* FIXME: This intermittently fails.
> > -     We need to wait for SIGSTOP first.  */
> > -  ptrace (PTRACE_SETOPTIONS, lwpid, 0, PTRACE_O_TRACECLONE);
> > -
> >    if (initial)
> >      /* NOTE/FIXME: This lwp might have not been the tgid.  */
> >      ptid = ptid_build (lwpid, lwpid, 0);
> > @@ -392,6 +386,11 @@ linux_attach_lwp_1 (unsigned long lwpid,
> >    new_lwp = (struct lwp_info *) add_lwp (ptid);
> >    add_thread (ptid, new_lwp);
> >  
> > +
> > +  /* We need to wait for SIGSTOP before being able to make the next
> > +     ptrace call on this LWP.  */
> > +  new_lwp->must_set_ptrace_flags = 1;
> > +
> >    /* The next time we wait for this LWP we'll see a SIGSTOP 
> > as PTRACE_ATTACH
> >       brings it to a halt.
> >  
> > @@ -986,6 +985,13 @@ linux_wait_for_event_1 (ptid_t ptid, int
> >  	  continue;
> >  	}
> >  
> > +      if (event_child->must_set_ptrace_flags)
> > +	{
> > +	  ptrace (PTRACE_SETOPTIONS, lwpid_of (event_child),
> > +		  0, PTRACE_O_TRACECLONE);
> > +	  event_child->must_set_ptrace_flags = 0;
> > +	}
> > +
> >        if (WIFSTOPPED (*wstat)
> >  	  && WSTOPSIG (*wstat) == SIGSTOP
> >  	  && event_child->stop_expected)
> > @@ -1248,11 +1254,6 @@ retry:
> >  
> >    lwp = get_thread_lwp (current_inferior);
> >  
> > -  if (must_set_ptrace_flags)
> > -    {
> > -      ptrace (PTRACE_SETOPTIONS, lwpid_of (lwp), 0, 
> > PTRACE_O_TRACECLONE);
> > -      must_set_ptrace_flags = 0;
> > -    }
> >    /* If we are waiting for a particular child, and it exited,
> >       linux_wait_for_event will return its exit status.  
> Similarly if
> >       the last child exited.  If this is not the last 
> child, however,
> > Index: src/gdb/gdbserver/linux-low.h
> > ===================================================================
> > --- src.orig/gdb/gdbserver/linux-low.h	2009-04-12 
> > 22:44:01.000000000 +0100
> > +++ src/gdb/gdbserver/linux-low.h	2009-04-30 
> > 22:33:09.000000000 +0100
> > @@ -146,6 +146,10 @@ struct lwp_info
> >       was a single-step.  */
> >    int stepping;
> >  
> > +  /* If this flag is set, we need to set the event request 
> flags the
> > +     next time we see this LWP stop.  */
> > +  int must_set_ptrace_flags;
> > +
> >    /* If this is non-zero, it points to a chain of signals 
> > which need to
> >       be delivered to this process.  */
> >    struct pending_signals *pending_signals;
> > 
> > 
> 

Re: Does HEAD support non-stop with 'gdbserver --multi' on Linux?

[Pedro Alves]

On Tuesday 12 May 2009 16:21:48, Marc Khouzam wrote:
> I saw that with non-stop, when doing native linux debugging, there
> is a conscious decision to trigger thread events as soon as possible
> (from a comment of linux-nat.c:linux_handle_extended_wait()).

At the time I added that to linux-nat.c it seemed like a
good idea, but in hindsight, I'm not sure it was.  This has
the potential to generate *a lot* of create/exit event pairs
for no apparent good reason --- consider a program that spawns
a lot of short lived worker threads.  Against remote
targets, this would be even worse, due to extra slowness.

> However, this does not happen when using gdbserver.
> 
> Is this something that is not ready yet, or a bug, or ... ?

This is a remote protocol issue.  The remote protocol does not have
any support for new thread notifications.  GDB only learns about
new threads when they report an event (say one hits a breakpoints),
or when the thread list is explicitly requested.

OOC, how does the eclipse/java debugger behave on such
scenario?  Are short lived threads added/removed from the GUI
even if they don't report a stop event?

-- 
Pedro Alves

RE: Does HEAD support non-stop with 'gdbserver --multi' on Linux?

[Marc Khouzam]

> -----Original Message-----
> From: gdb-owner@sourceware.org 
> [mailto:gdb-owner@sourceware.org] On Behalf Of Pedro Alves
> Sent: Tuesday, May 12, 2009 11:58 AM
> To: Marc Khouzam
> Cc: gdb@sourceware.org
> Subject: Re: Does HEAD support non-stop with 'gdbserver 
> --multi' on Linux?
> 
> On Tuesday 12 May 2009 16:21:48, Marc Khouzam wrote:
> > I saw that with non-stop, when doing native linux debugging, there
> > is a conscious decision to trigger thread events as soon as possible
> > (from a comment of linux-nat.c:linux_handle_extended_wait()).
> 
> At the time I added that to linux-nat.c it seemed like a
> good idea, but in hindsight, I'm not sure it was.  This has
> the potential to generate *a lot* of create/exit event pairs
> for no apparent good reason --- consider a program that spawns
> a lot of short lived worker threads.  Against remote
> targets, this would be even worse, due to extra slowness.

Yes, this was bothering me too, from the frontend point-of-view.

> 
> > However, this does not happen when using gdbserver.
> > 
> > Is this something that is not ready yet, or a bug, or ... ?
> 
> This is a remote protocol issue.  The remote protocol does not have
> any support for new thread notifications.  GDB only learns about
> new threads when they report an event (say one hits a breakpoints),
> or when the thread list is explicitly requested.
> 
> OOC, how does the eclipse/java debugger behave on such
> scenario?  Are short lived threads added/removed from the GUI
> even if they don't report a stop event?

Good question.
I just ran some test to figure this out.  It turns out that
threads are added and removed from the GUI without any stop
event.  Furthermore, was creating and destroying a thread
every 200 milisec, and the UI was flickering to show each 
and every change!

But that may not be the way we want to go... I'm not sure.
This behavior could be a user preference.
Right now, in DSF-GDB for GDB7.0, any =thread-created/exited
event will immediately be shown to the user.
I was postponing addressing this risk until someone
brought this up as a problem...

Currently, I'm leaning towards a periodic refresh scheme.
Say every half second, the list of threads would be refreshed
if needed.  But I have to think about this further.
Such a scheme could be used in GDB also.

Marc