Remote debugging of multithreaded programs using gdbserver

Remote debugging of multithreaded programs using gdbserver

[Subhashini Nagarajan Rao]

[-- Attachment #1.1: Type: text/plain, Size: 485 bytes --]

Hi All,
 
Remote debugging of multithreaded programs using gdbserver was
implemented with GDB-5.1.1 as the code base for LINUX platform
IA32 architecture.

Daniel Jacobowitz had earlier pointed out the importance
of resolving copyright issues before we could post our patch.
We have requested help from Jim Blandy towards resolving
this issue. In the meanwhile we would like to get some
feedback on our approach - kindly refer to the attachment.

Thanks & Regards,
Subha

[-- Attachment #1.2: Type: text/html, Size: 885 bytes --]

[-- Attachment #2: Rem_Debug_App.txt --]
[-- Type: text/plain, Size: 16775 bytes --]


  
  Approach document for remote multi-threaded program
  debuggability feature in gdb
  
  * The scope of the project involves:
  ----------------------------------
  
  Providing remote multi-threaded program debuggability feature
  extensions in gdb on Pentium architectures.
  
  The target platform for the scope mentioned above is Linux.
  
  * Detailed Description of Software Components and Assumptions Made
  ----------------------------------------------------------------
  
  Gdbserver is a component in the client-server gdb. The gdbserver resides
  in the target system (ie. the system on which the program to be debugged will
  be executed). As of today, multi-threaded program debuggability features in
  gdbserver does not exist. The objective of this project is to provide this
  feature.
  
  The following items will be developed for this multi-threaded extension:
  
  * Providing support to handle notification and tracking creation and
  deletion of threads in the target program.
  * Providing thread related information for all threads or for a specific
  thread based on thread id. The thread related information could be
  a) thread id, b) thread alive detection, c) thread name,
  d) Detailed information on threads.
  * Support for applying different commands on specific threads. The
  commands could be back trace, procuring register content information, etc.
  
  To provide multi-threaded extension support, the components that would
  require modifications are:
  Gdb Client-Server Protocol Enhancements
  Gdb Client related changes
  Gdb Server related changes
  
  Description of each of the items to be implemented
  --------------------------------------------------
  
  1. Handle notification and tracking of thread creation and deletion in the
  target program.
  
  Thread creation is notified to the client through breakpoint mechanism
  much in the same lines as native gdb. Essentially breakpoints
  are inserted at the `notified' address, address to which execution is
  guarenteed if thread creation event is enabled by libthread_db.
  The client needs to update its breakpoint list with this address
  to notify gdbserver on a thread creation event. Since libthread_db interfaces
  are used towards procurement of these addresses, the proc_service
  routines used by libthread_db needs to be implemented in the gdbserver.
  Since event reporting facility is broken for TD_DEATH events in glibc 2.1.3
  and the present implementation uses this version of glibc, thread deaths are
  are handled differently. While an immediate reporting of thread creation
  to gdbserver is mandatory to ensure that the gdbserver traces the newly
  created threads with immediate effect, thread death updation at the client
  needs could be procrastinated to the time when one of the following
  events takes place:
  (i) info threads command is fired at the client by the user.
  (ii) context switch happens from one thread to another during execution.
  It is crucial that a context switch does not happen from or to a dead
  thread. This is ensured by making a call to prune_threads which
  clobbers thread ids of dead threads from the thread list.
  
  For both these cases, the client thread updates its threadlist by
  sending the T(thread alive packet ). gdbserver however clobbers
  dead threads from its thread list as soon as a thread dies as it can
  easily track dead threads through the waitpid status flag.
  
  
  Gdb Client related and protocol enhancements towards the same are:
  
  The client sends a qSymbol:: packet to notify the target that
  it is prepared to service symbol look-up requests, typically when new
  objects are loaded. On the initial receipt of a qSymbol:: ,
  the server enables thread event reporting and sends the thread
  creation notification address through qSymbol! packet for the client to add
  breakpoint at that address. qSymbol! packet essentially sticks to the the
  following format:
  
  qSymbol!<thread creation notification address>:<thread death notification address>
  
  Since the present implementation does not rely on libthread_db
  for thread death notification, qSymbol! sets 0 towards thread death
  notification so that client ignores the same.
  
  On subsequent receipts of qSymbol::, the server checks if it has been
  stopped because of a thread creation event, through libthread_db interfaces,
  in which case it adds the new threadid to its locally maintained list of
  threads that it traces.
  
  Functions taking care of this - remote_check_symbols(), (to get the breakpoint
     address), set_td_create_bp_addr(), set_td_death_bp_addr()
  
  Functions set_td_create_bp_addr & set_td_death_bp_addr are inturn
  modified to call create_thread_event_breakpoint for the
  insertion of the breakpoint in the breakpoint chain.
  
  Gdb Server related changes are :
  
  Once the qSymbol packet is sent -
  
  *) Open a connection to libthread_db to allocate a thread agent. This
  needs to be done only if the thread agent was not previously allocated.
  
  function taking care of this - thread_db_new_agent().
  
  *) Set the process wide event masks and obtain the thread creation
  notification address from the thread library using libthread_db
  interfaces and also enable event reporting.
  
  function taking care of this - enable_thread_event_reporting().
  
  *) check_event() checks for if the threads have been stopped
  because of a thread creation event/breakpoint in which case, it calls
  attach_thread() which does a PT_ATTACH to the new thread so that the
  created thread can also be traced.
  
  The proc_service routine ps_pglobal_lookup in the server side is
  responsible for getting the symbol values required from the client.
  
  
  2. Maintenance of list of threads in the target program.
  
  The Server maintains various lists of thread ids for the following
  purpose.
  
  *) threadid_list = a linked list to store the information of the active
  threads that are traced.
  *) stopped_pid_list = a linked list of thread ids of the stopped threads.
  
  The following functions are used to work on the lists of thread ids.
  
  *) in_threadid_list = to check if a thread is present in the
  threadid_list.
  *) add_thread = add a thread id to the threadid_list.
  *) delete_thread = delete a thread id from the threadid_list.
  *) in_stopped_pid_list = to check if a thread is present in the
  stopped_pid_list.
  *) insert_stopped_pid_list = insert a stopped thread's id into the
  stopped_pid_list.
  *) delete_stopped_pid_list = delete from the stopped_pid_list the
  thread's id which resumed its execution from the stopped state.
  
  3. Provide information on threads.
  
  For 'info threads' command it is required to first find all the live
  threads in the target program. Gdb Client sends a qfThreadInfo packet to
  the server to get the thread ids. If all the threads are not sent 
  due to packet size limitations, a qsThreadInfo packet is sent for
  subsequent threads id retrievals until it receives an 'l'.
  To get the extra information on the thread like, running or blocked
  statuses, qThreadExtraInfo packet is sent. These changes are presently
  inherited from gdb5.1.1 original code from the client.
  
  Handling of the Protocol packets :
  
  *) qfThreadInfo : When this packet is received libthread's thread
  iterator function is called to collect all the active thread
  ids. The collected thread ids are maintained in a
  linked list, thread_list. These thread ids are then
  copied from this list to the buffer to be sent to the
  client. The purpose behind storing the thread ids
  rather than writing directly into the buffer is to
  overcome the limitation posed by the buffer size.
  
  *) qsThreadInfo : Copy from the next thread id from the thread_list
  to the buffer if any more are to be copied else send 'l' as the reply. 
  Also free the thread_list once done.
  
  *) qThreadExtraInfo : This provides the additional thread information
  like the state of the thread. For this purpose libthread_db interfaces 
  are used.  Functions used to implement this are
  i386_get_thread_info, mem2hstr, thr_state_string.
  The thread iterator function is used to obtain the state of the threads.
  
  4. Adding breakpoints in the threads.
  
  The list of breakpoints in the program is maintained as a linked list
  of breakpoint addresses. The breakpoint list is a doubly linked list
  having the breakpoint address and a 2 character array('cc' for IA32).
  A free pool of these nodes are maintained from where the breakpoint list
  draws out nodes as and when the need arises.
  
  Whenever a breakpoint is to be inserted or deleted the client sends
  'Z' or 'z' packet. If it is a 'Z' packet, the new breakpoint address
  is added to the breakpoint list if it is not already added and if it is
  a 'z' packet the specified breakpoint address is deleted from the
  breakpoint list.
  
  The functions parse_Z and parse_z takes care of inserting or deleting
  the breakpoint from the breakpoint list.
  
  Note - some portions of the code like breakpoint handling in the server
         (parseZ() function) was taken from a patch posted earlier on the 
         mailing list for gdb-5.0

  5. Procedure of waiting for the inferior process to return the status.
  
  In the Server, mywait() handles this.
  *) To denote the thread of interest the variables used are 
  o general_thread - this thread's id is needed for all info retrieval
  like register set details.
  o thread_to_cont - this is the `main' thread that needs to be
  subsequently resumed( besides the threads that are anyway stopped by gdb )
  o cont_thread - thread which needs to continue.
  o inferior_pid - thread corresponding to main().
  
  *) 'waitpid' is used to wait for the inferior process and know the status.
  
  WCLONE option needs to be passed to wait on clonned threads. Hence
  waipid() needs to be called with WCLONE and WNOHANG(to prevent
  hangs ) till a valid pid is returned by waitpid(). To ensure
  that such a framework works fine for non-threaded programs, the WCLONE
  option is toggled everytime.
  
  When waitpid() returns because of a thread death, the locally maintained
  threadlist is updated to reflect the same to prevent this threadid
  from subsequently being traced.
  
  If a thread has stopped due to TARGET_SIGNAL_STOP,
  TARGET_SIGNAL_REALTIME_32/33 or a TARGET_SIGNAL_SIGCHLD - these are
  inherited from the client code which essentially does not report
  all signals to the native gdb which is not of its interest,  a
  ptrace(CONT) is done and the above steps are performed again. If the 
  thread is stopped by any other signal other than the above it is inserted 
  in the stopped_pid_list. The signals mentioned above are stored as an array
  with the values denoting the action to be taken. It is stored as three
  arrays wherein we specify whether the signal should be handled by the
  debugger or ignored. This is a copy of the signals array which the client
  stores. Presently the array is initialized to required values to denote
  the present action exhibited by the client over these signals.
  Any change in the client side will require a change in the server also.
  
  When one thread is stopped all the other threads are also stopped so that all
  are in sync and in control under the debugger. This is
  done by calling stop_all_threads(). If the thread stopped is single-stepped
  then preference is given to this thread and allowed to resume next. 
  When none of the threads are single-stepped and more than 1 
  thread has received a SIGTRAP, randomnly 1 thread is chosen among the threads
  which received the trap signal and allowed to resume next 
  If any of the other threads has got a SIGTRAP because of a gdb 
  inserted breakpoint, the pc values have to be re-adjusted to ensure that 
  these threads re-execute the breakpoint instruction. The selection of the 
  thread to resume next is handled by select_event_pid(). 
  
  For every continue or next the server replies with the thread id. The
  client reads this threadid and if this is not in its maintained
  threadid list, it updates the same.
  
  6. Procedure of resuming the threads from the stopped state.
  
  In the Client side,
  
  *) remote_resume() calls set_thread() to specify the thread to be resumed.
  In case of resume all continue_thread is set to -1 to indicate this.
  
  In the Server side, myresume() handles this.
  
  *) If the cont_thread is -1 it means all the threads need to be resumed.
  Now, the thread_to_cont holds the thread which is selected as the event thread
  by select_event_pid. This thread is subsequently stepped or continued
  as per the user's request and other threads are continued. For every thread 
  in the threadid_list there is a field in its data structure to store the step 
  information. When a thread is continued this field is reset. For the 
  thread corresponding to the thread_to_cont, the step information field is
  updated with 1 or 0 depending on whether a step/cont is given to this 
  thread.  
  *) If cont_thread is 0 - which essentially means that this program is 
  non-threaded - a step/cont is given to inferior_pid and this thread is 
  deleted from the stopped_pid_list. Otherwise the step/cont is given to the 
  respective thread and deleted from the stopped_pid_list. The step information
  for this thread is also updated in the threadid_list. 
  
  7. Other relevant thread related pkts changes:
  
  *) action for T pkt from client - check in the threadid_list to know
  whether the req thread is alive or not.
  *) action for Hc pkt from client - the id sent along with this is set to
  cont_thread.
  If it is not a -1 then update the thread_to_cont also to this value
  immediately. In resume_all_stopped_threads thread ids are compared against 
  thread_to_cont where the assumption is thread_to_cont is the thread of 
  interest to be resumed next. So, this needs to be updated to reflect this.
  *) action for Hg pkt from client - first need to chk if the specified
  thread is in stopped state. If so respond immediately by sending the register
  contents. If not first stop this thread, insert it into the stopped_pid_list 
  and then proceed.
  
  8. Miscellaneous changes
  
  When the user fires a 'thread <id>' command , switch to the specified
  thread but a step after this was not able to tell the server that the
  thread to be stepped is the new thread and not the previous one. Hence,
  there is a need for the client, to remotely specify the current thread
  in switch_to_thread(). However since switch_to_thread() is written
  for native gdb, and our present requirement is to call set_thread() which
  is specific to remote debugging, we cannot call set_thread directly.
  The proposed solution is to have a hook function(remote_set_thread_hook)
  which is set in remote.c during remote_open and switch_to_thread()
  calls this function when the hook is non-zero (essentially remote-debugging)
  and does not call it when the hook is NULL(essentially native mode).
  Also in the server for the 'Hc' packet we set update the thread_to_cont to
  this value so that a resume following this can compare the thread ids
  against the correct thread_to_cont and fires does cont/step on this
  thread as is requested by the user accordingly.
  
  * Assumptions made :
  ----------------
  
  For the development of this project, following Assumptions are
  considered:
  
  Current Scope of gdbserver enhancements is for debugging User Space
  applications
  Gdb base to be used for development of the project would be gdb
  version 5.1.1.
  

 * Functionalities yet to be implemented :
   -------------------------------------

   i) Handling of ^C -  This is presently not tested exhaustively. 

  ii) Handling of signals if they are dynamically changed by the user as the
      actions to be done for a particular signal is maintained as a static 
      array in the gdbserver. 

  iii) whenever a new thread comes up gdb prints "[New Thread tid]". 
       In this implementation presently lwpid gets printed instead of tid. This 
       needs to be changed. 


 * Testing through DejaGNU :
   -----------------------

      pthreads.c/pthreads.exp testsuite was tested and verified with this fix.

 ******************************************************************************

[-- Attachment #3: Wipro_Disclaimer.txt --]
[-- Type: text/plain, Size: 477 bytes --]

**************************Disclaimer************************************
Information contained in this E-MAIL being proprietary to Wipro Limited
is 'privileged' and 'confidential' and intended for use only by the
individual or entity to which it is addressed. You are notified that any
use, copying or dissemination of the information contained in the E-MAIL
in any manner whatsoever is strictly prohibited.
********************************************************************

Re: Remote debugging of multithreaded programs using gdbserver

[Daniel Jacobowitz]

First of all, gdbserver has been essentially rewritten between 5.1.1 and
5.2.  I recommend you look at the current CVS.  I have a great deal of
(thread-related) work pending, also.

Second - do you have details on the "broken" death event reporting in
glibc 2.1.3?  Does it make the death events unreliable, or outright
harmful?

On Thu, Apr 18, 2002 at 10:29:21AM +0530, Subhashini Nagarajan Rao wrote:
>   Since event reporting facility is broken for TD_DEATH events in glibc 2.1.3
>   and the present implementation uses this version of glibc, thread deaths are
>   are handled differently. While an immediate reporting of thread creation
>   to gdbserver is mandatory to ensure that the gdbserver traces the newly
>   created threads with immediate effect, thread death updation at the client
>   needs could be procrastinated to the time when one of the following
>   events takes place:
>   (i) info threads command is fired at the client by the user.
>   (ii) context switch happens from one thread to another during execution.

"context switch" is misleading.  The context switches of the process
are irrelevant.  I assume you mean whenever the client GDB requests a
change of thread...

>   Gdb Client related and protocol enhancements towards the same are:
>   
>   The client sends a qSymbol:: packet to notify the target that
>   it is prepared to service symbol look-up requests, typically when new
>   objects are loaded. On the initial receipt of a qSymbol:: ,
>   the server enables thread event reporting and sends the thread
>   creation notification address through qSymbol! packet for the client to add
>   breakpoint at that address. qSymbol! packet essentially sticks to the the
>   following format:
>   
>   qSymbol!<thread creation notification address>:<thread death notification address>

I don't think this is the right approach.  The client GDB should not be
aware of anything involving thread_db.  It's none of its business.

OTOH, the bugs which cause thread_db to be pushed on the target stack
when connected to a remote target definitely need to be fixed.  I'm
not quite decided on how to do this yet.

Why do you bother with this if you have obviously given the server the
ability to place breakpoints?

>   Handling of the Protocol packets :
>   
>   *) qfThreadInfo : When this packet is received libthread's thread
>   iterator function is called to collect all the active thread
>   ids. The collected thread ids are maintained in a
>   linked list, thread_list. These thread ids are then
>   copied from this list to the buffer to be sent to the
>   client. The purpose behind storing the thread ids
>   rather than writing directly into the buffer is to
>   overcome the limitation posed by the buffer size.
>   
>   *) qsThreadInfo : Copy from the next thread id from the thread_list
>   to the buffer if any more are to be copied else send 'l' as the reply. 
>   Also free the thread_list once done.

This information should be cached, there's no point in looking it up
from scratch every time you are queried.

>   4. Adding breakpoints in the threads.
>   
>   The list of breakpoints in the program is maintained as a linked list
>   of breakpoint addresses. The breakpoint list is a doubly linked list
>   having the breakpoint address and a 2 character array('cc' for IA32).
>   A free pool of these nodes are maintained from where the breakpoint list
>   draws out nodes as and when the need arises.

"A two character array"?  0xCC is the code for a one-byte i386
breakpoint, but I can't imagine what else you mean by this. 
Breakpoints can be a word, or possibly more.

>   When one thread is stopped all the other threads are also stopped so that all
>   are in sync and in control under the debugger. This is
>   done by calling stop_all_threads(). If the thread stopped is single-stepped
>   then preference is given to this thread and allowed to resume next. 
>   When none of the threads are single-stepped and more than 1 
>   thread has received a SIGTRAP, randomnly 1 thread is chosen among the threads
>   which received the trap signal and allowed to resume next 
>   If any of the other threads has got a SIGTRAP because of a gdb 
>   inserted breakpoint, the pc values have to be re-adjusted to ensure that 
>   these threads re-execute the breakpoint instruction. The selection of the 
>   thread to resume next is handled by select_event_pid(). 

All of this appears to be taken straight from GDB's native thread debug
package.  None of it is clearly thought out in this context, IMHO. 
Limitations in GDB make it hard to fix these properly there (I spent a
full day trying recently...) but gdbserver has plenty of room.  For
instance, I'd wager you stop all threads on a thread creation event. 
Also, there's no reason to do the fancy PC-adjustment.  You should be
able to save that stop event and re-report it later if nothing disturbs
it.

>   8. Miscellaneous changes
>   
>   When the user fires a 'thread <id>' command , switch to the specified
>   thread but a step after this was not able to tell the server that the
>   thread to be stepped is the new thread and not the previous one. Hence,
>   there is a need for the client, to remotely specify the current thread
>   in switch_to_thread(). However since switch_to_thread() is written

I don't see anything here that requires information gdbserver does not
already have.  It knows the last thread stepped and the thread
requested.

-- 
Daniel Jacobowitz                           Carnegie Mellon University
MontaVista Software                         Debian GNU/Linux Developer

profiling with GDB

[abhijit ray]

I am not sure but I think with the simulator for ARM , PPC that comes with gdb you can get profiling informations too
But I don't know hwo .. I have configured and installed simulators for arm-elf , powerpc-eabi and sh ..and am using the simulator quite well but i am not able to get the profiler output 

can anyone help ?
Thanks ..
abhijit ray 

_________________________________________________________
Do You Yahoo!?
Get your free @yahoo.com address at http://mail.yahoo.com

Re: profiling with GDB

[Andrew Cagney]

> I am not sure but I think with the simulator for ARM , PPC that comes with gdb you can get profiling informations too
> But I don't know hwo .. I have configured and installed simulators for arm-elf , powerpc-eabi and sh ..and am using the simulator quite well but i am not able to get the profiler output 

Features such as capturing profile information are simulator dependant. 
The MIPS, for instance, supports this.

Andrew

Re: Remote debugging of multithreaded programs using gdbserver

[Daniel Jacobowitz]

On Tue, Apr 23, 2002 at 12:12:12PM +0530, Subhashini Nagarajan Rao wrote:
> Hi Daniel,
> 
>  As you stated I was able to reproduce it with the pthreads.c program and with backing 
> off of breakpoints the bogus SIGTRAP doesnt come. If you dont mind could you please
> tell me what 'SMP' stands for in your mail. 

SMP is Symmetric Multi-Processing.  If you aren't testing multithreaded
programs on a machine with more than one physical processor, you won't
see entire classes of bugs.

-- 
Daniel Jacobowitz                           Carnegie Mellon University
MontaVista Software                         Debian GNU/Linux Developer

Re: Remote debugging of multithreaded programs using gdbserver

[Subhashini Nagarajan Rao]

[-- Attachment #1.1: Type: text/plain, Size: 255 bytes --]

Hi Daniel,

 As you stated I was able to reproduce it with the pthreads.c program and with backing 
off of breakpoints the bogus SIGTRAP doesnt come. If you dont mind could you please
tell me what 'SMP' stands for in your mail. 

Thanks, 
Subha


[-- Attachment #1.2: Type: text/html, Size: 869 bytes --]

Re: Remote debugging of multithreaded programs using gdbserver

[Daniel Jacobowitz]

On Fri, Apr 19, 2002 at 06:32:17PM +0530, Subhashini Nagarajan Rao wrote:
> Hi Daniel, 
> 
>   With reference to your comment on the PC-adjustment, we tried to avoid it and 
> the design for that is given below. I had also tested with this change and things
> are working fine. Requesting your valuable comments on this. 
> 
> In order to avoid the PC-adjustment for the threads other than the thread of 
> interest (thread_to_cont), we can remember the stop_event for the other threads.
> which may require the following, 
>  i) A boolean, has_pending, in the threadid_info structure to denote that that 
>    particular thread has got a trap event (due to hitting a breakpoint) pending. This 
>    should be initialised to 0. 
> 
> (Note - in the details that follow, thread_to_cont refers to the thread of interest, 
>                                                'tp' refers to a thread in the list of current threads in 
>                                                 the program maintained in 'threadid_list' )
> 
> The changes to be made are, 
>  i) In stop_all_threads(), if it is found that a thread has hit a breakpoint 
>     then set the boolean tp->has_pending. Remove the code corresponding 
>     to cancelling of breakpoints. 
> ii) Do not reset tp->status for the threads which has tp->has_pending set as 
>     it will have the trap event recorded. (This will be automatically taken care of
>     if the code corresponding to cancellation of breakpoints is removed). 

I reached the point in my implementation where I needed this code
yesterday.  I got it to work, but in the process gained a good
understanding for why it is done the other way in core GDB - because
otherwise, if a breakpoint is removed after two threads have hit it,
one of them will resume with the wrong PC and a bogus SIGTRAP.

You should be able to reproduce this with pthreads.c in the GDB
testsuite, by setting a breakpoint on common_function if hits >= 15,
continuing to it, disabling the breakpoint, and continuing again.  That
most likely requires SMP in order to see it; I highly recommend testing
all thread-debugging work on SMP.

I think that this approach is still valuable, because it simplifies the
arbitration logic, but backing off of breakpoints may still be
necessary.

> **************************Disclaimer************************************
> Information contained in this E-MAIL being proprietary to Wipro Limited
> is 'privileged' and 'confidential' and intended for use only by the
> individual or entity to which it is addressed. You are notified that any
> use, copying or dissemination of the information contained in the E-MAIL
> in any manner whatsoever is strictly prohibited.
> ********************************************************************

This form of disclaimer violates the accepted policy for most mailing
lists; the GCC lists page explicitly says:

|Please do not include or reference confidentiality notices, like:
|
|  The referring document contains privileged and confidential
|  information. If you are not the intended recipient you must not
|  disseminate, copy or take any action in reliance on it, and we request
|  that you notify companyname immediately.
|
|Such disclaimers are inappropriate for mail sent to public lists. If
|your company automatically adds something like this to outgoing mail,
|and you can't convince them to stop, you might consider using a free
|web-based e-mail account.

-- 
Daniel Jacobowitz                           Carnegie Mellon University
MontaVista Software                         Debian GNU/Linux Developer

Remote debugging of multithreaded programs using gdbserver

[Subhashini Nagarajan Rao]

[-- Attachment #1.1: Type: text/plain, Size: 2610 bytes --]

Hi Daniel, 

  With reference to your comment on the PC-adjustment, we tried to avoid it and 
the design for that is given below. I had also tested with this change and things
are working fine. Requesting your valuable comments on this. 

In order to avoid the PC-adjustment for the threads other than the thread of 
interest (thread_to_cont), we can remember the stop_event for the other threads.
which may require the following, 
 i) A boolean, has_pending, in the threadid_info structure to denote that that 
   particular thread has got a trap event (due to hitting a breakpoint) pending. This 
   should be initialised to 0. 

(Note - in the details that follow, thread_to_cont refers to the thread of interest, 
                                               'tp' refers to a thread in the list of current threads in 
                                                the program maintained in 'threadid_list' )

The changes to be made are, 
 i) In stop_all_threads(), if it is found that a thread has hit a breakpoint 
    then set the boolean tp->has_pending. Remove the code corresponding 
    to cancelling of breakpoints. 
ii) Do not reset tp->status for the threads which has tp->has_pending set as 
    it will have the trap event recorded. (This will be automatically taken care of
    if the code corresponding to cancellation of breakpoints is removed). 

If a 's' packet is sent, it indicates that the current thread is single-stepped 
and hence the resume logic will be kept the same except that those threads 
having a pending event will not get resumed. 

If a 'c' packet is sent, it means the previous hit breakpoint is cleared and 
now we can look into the threads having a pending event. Select the first 
thread in the threadid_list having a tp->has_pending equal to 1 and update the 
thread_to_cont and the general_thread (so that we fetch the register info of the 
correct thread) to this thread id. Reset tp->has_pending to indicate its selection. 
As all the other threads are already stopped the prepare_resume_reply should be 
called with this updated thread_to_cont with status set to 'T'. 

An 's' or 'c' packet now will be similar to what is stated above. 

As the pending trap events are getting serviced whenever a 'c' packet is sent , this 
eliminates the need to have the arbitration logic of selecting the next thread to 
continue in the function select_event_pid(). [The arbitration logic is explained in the
approach document under the subtitle - Procedure of waiting for the inferior process to return the status.]

- Subha


[-- Attachment #1.2: Type: text/html, Size: 4576 bytes --]

[-- Attachment #2: Wipro_Disclaimer.txt --]
[-- Type: text/plain, Size: 477 bytes --]

**************************Disclaimer************************************
Information contained in this E-MAIL being proprietary to Wipro Limited
is 'privileged' and 'confidential' and intended for use only by the
individual or entity to which it is addressed. You are notified that any
use, copying or dissemination of the information contained in the E-MAIL
in any manner whatsoever is strictly prohibited.
********************************************************************

Re: Remote debugging of multithreaded programs using gdbserver

[Daniel Jacobowitz]

Comments about the rest later, but:

On Thu, Apr 18, 2002 at 06:55:56PM +0530, Subhashini Nagarajan Rao wrote:
> Hi Daniel,
> 
> As you had stated there are instances of inheritance of the
> concepts in the implementation of the gdbserver from the native gdb. This
> is due to the fact that we want to get the code functionally correct and
> running
> and later revisions can be done to it. Reply to the comments follow.

You had a very thorough design document, and a design document ought to
consider the eventual goal and not the current step.  Otherwise you
just waste a lot of work.

> > >   8. Miscellaneous changes
> > >
> > >   When the user fires a 'thread <id>' command , switch to the specified
> > >   thread but a step after this was not able to tell the server that the
> > >   thread to be stepped is the new thread and not the previous one.
> Hence,
> > >   there is a need for the client, to remotely specify the current thread
> > >   in switch_to_thread(). However since switch_to_thread() is written
> >
> > I don't see anything here that requires information gdbserver does not
> > already have.  It knows the last thread stepped and the thread
> > requested.
> >
> When we switch from one thread to another as a result of 'thread' command
> the inferior_ptid gets updated to the new thread. Now, when I do a `next',
> in the native gdb,  all the threads are resumed and in this case the step
> signal will be given to the updated inferior_ptid. However, in the case
> remote
> debugging if I resume all the threads after the `thread' command by doing a
> next on the new thread I will be sending a Hc-1 packet and the server is not
> notified about the thread switch. To mention the current thread I need to
> send
> a Hc<id> packet so that the server will be informed about the current thread
> of
> execution and it will do a step on the correct thread. This is done by
> calling set_thread().

OK, I believe I see the problem.  It looks like a bug in core GDB; we
resume all threads so remote_resume does not communicate which one to
step.  I'll think about this for a bit.


-- 
Daniel Jacobowitz                           Carnegie Mellon University
MontaVista Software                         Debian GNU/Linux Developer

Re: Remote debugging of multithreaded programs using gdbserver

[Subhashini Nagarajan Rao]

[-- Attachment #1.1: Type: text/plain, Size: 9647 bytes --]

Hi Daniel,

As you had stated there are instances of inheritance of the
concepts in the implementation of the gdbserver from the native gdb. This
is due to the fact that we want to get the code functionally correct and
running
and later revisions can be done to it. Reply to the comments follow.

Thanks & Regards,
Subha

----- Original Message -----
From: "Daniel Jacobowitz" <drow@mvista.com>
To: "Subhashini Nagarajan Rao" <subhashini.nagarajan@wipro.com>
Cc: <gdb@sources.redhat.com>; "Dan Kegel" <dank@kegel.com>; "Anantha
Subramanian" <anantha.subramanian@wipro.com>; "Ganeshan Kailasam"
<ganeshan.kailasam@wipro.com>; <sanjay.nair@wipro.com>
Sent: Thursday, April 18, 2002 11:02 AM
Subject: Re: Remote debugging of multithreaded programs using gdbserver


> First of all, gdbserver has been essentially rewritten between 5.1.1 and
> 5.2.  I recommend you look at the current CVS.  I have a great deal of
> (thread-related) work pending, also.
>
Fine
> Second - do you have details on the "broken" death event reporting in
> glibc 2.1.3?  Does it make the death events unreliable, or outright
> harmful?
>
Gdb source says event reporting for thread death is broken for glibc 2.1.3.
As we are using this libc we are not presently enabling the thread death
events. Related comment from the native Gdb source code, (from thread-db.c)

  /* FIXME: kettenis/2000-04-23: The event reporting facility is
     broken for TD_DEATH events in glibc 2.1.3, so don't enable it for
     now.  */

> On Thu, Apr 18, 2002 at 10:29:21AM +0530, Subhashini Nagarajan Rao wrote:
> >   Since event reporting facility is broken for TD_DEATH events in glibc
2.1.3
> >   and the present implementation uses this version of glibc, thread
deaths are
> >   are handled differently. While an immediate reporting of thread
creation
> >   to gdbserver is mandatory to ensure that the gdbserver traces the
newly
> >   created threads with immediate effect, thread death updation at the
client
> >   needs could be procrastinated to the time when one of the following
> >   events takes place:
> >   (i) info threads command is fired at the client by the user.
> >   (ii) context switch happens from one thread to another during
execution.
>
> "context switch" is misleading.  The context switches of the process
> are irrelevant.  I assume you mean whenever the client GDB requests a
> change of thread...
>
The thread list maintained in the client and the server are not in sync as
the thread death events are presently not enabled. So if there is a
context switch (changing of thread) to a dead thread then we will be having
problems hence requiring pruning of threads.

> >   Gdb Client related and protocol enhancements towards the same are:
> >
> >   The client sends a qSymbol:: packet to notify the target that
> >   it is prepared to service symbol look-up requests, typically when new
> >   objects are loaded. On the initial receipt of a qSymbol:: ,
> >   the server enables thread event reporting and sends the thread
> >   creation notification address through qSymbol! packet for the client
to add
> >   breakpoint at that address. qSymbol! packet essentially sticks to the
the
> >   following format:
> >
> >   qSymbol!<thread creation notification address>:<thread death
notification address>
>
> I don't think this is the right approach.  The client GDB should not be
> aware of anything involving thread_db.  It's none of its business.
>
> OTOH, the bugs which cause thread_db to be pushed on the target stack
> when connected to a remote target definitely need to be fixed.  I'm
> not quite decided on how to do this yet.
>
The server actually does not maintain a breakpoint list as done in the case
of gdb. It only has a mirror image of the breakpoints to be inserted which
it gets as a "Z0, <address>" packet whenver there is a call to
target_insert_breakpoints. This was initially required for the
'fancy PC_adjustment' but was found to be useful in other cases also.

> Why do you bother with this if you have obviously given the server the
> ability to place breakpoints?
>
> >   Handling of the Protocol packets :
> >
> >   *) qfThreadInfo : When this packet is received libthread's thread
> >   iterator function is called to collect all the active thread
> >   ids. The collected thread ids are maintained in a
> >   linked list, thread_list. These thread ids are then
> >   copied from this list to the buffer to be sent to the
> >   client. The purpose behind storing the thread ids
> >   rather than writing directly into the buffer is to
> >   overcome the limitation posed by the buffer size.
> >
> >   *) qsThreadInfo : Copy from the next thread id from the thread_list
> >   to the buffer if any more are to be copied else send 'l' as the reply.
> >   Also free the thread_list once done.
>
> This information should be cached, there's no point in looking it up
> from scratch every time you are queried.
>
This is a performance issue which can be looked into once we are through
with resolving the design issues. Also if there is enabling of thread death
events it would be more helpful in caching of the thread ids.

> >   4. Adding breakpoints in the threads.
> >
> >   The list of breakpoints in the program is maintained as a linked list
> >   of breakpoint addresses. The breakpoint list is a doubly linked list
> >   having the breakpoint address and a 2 character array('cc' for IA32).
> >   A free pool of these nodes are maintained from where the breakpoint
list
> >   draws out nodes as and when the need arises.
>
> "A two character array"?  0xCC is the code for a one-byte i386
> breakpoint, but I can't imagine what else you mean by this.
> Breakpoints can be a word, or possibly more.
>
The 'cc' mentioned here is the string which gets converted to the byte for
the purpose of adding breakpoints. Hence it was stated as a character array.

> >   When one thread is stopped all the other threads are also stopped so
that all
> >   are in sync and in control under the debugger. This is
> >   done by calling stop_all_threads(). If the thread stopped is
single-stepped
> >   then preference is given to this thread and allowed to resume next.
> >   When none of the threads are single-stepped and more than 1
> >   thread has received a SIGTRAP, randomnly 1 thread is chosen among the
threads
> >   which received the trap signal and allowed to resume next
> >   If any of the other threads has got a SIGTRAP because of a gdb
> >   inserted breakpoint, the pc values have to be re-adjusted to ensure
that
> >   these threads re-execute the breakpoint instruction. The selection of
the
> >   thread to resume next is handled by select_event_pid().
>
> All of this appears to be taken straight from GDB's native thread debug
> package.  None of it is clearly thought out in this context, IMHO.
> Limitations in GDB make it hard to fix these properly there (I spent a
> full day trying recently...) but gdbserver has plenty of room.  For
> instance, I'd wager you stop all threads on a thread creation event.
> Also, there's no reason to do the fancy PC-adjustment.  You should be
> able to save that stop event and re-report it later if nothing disturbs
> it.
>
Native Gdb implementation has been accepted as standardised and correct
functionally and this explains for code inheritance in the gdbserver.
We also did not understand your specific concern for stopping all the
threads.
Do you mean to say for non-thread creation breakpoints we are justified in
allowing other threads to continue? How would that handle cases like one of
the threads doing a `pthread_cond_timedwait'. However, we give the following
comment in the native code,

  /* Don't report signals that GDB isn't interested in, such as
     signals that are neither printed nor stopped upon.  Stopping all
     threads can be a bit time-consuming so if we want decent
     performance with heavily multi-threaded programs, especially when
     they're using a high frequency timer, we'd better avoid it if we
     can.  */

> >   8. Miscellaneous changes
> >
> >   When the user fires a 'thread <id>' command , switch to the specified
> >   thread but a step after this was not able to tell the server that the
> >   thread to be stepped is the new thread and not the previous one.
Hence,
> >   there is a need for the client, to remotely specify the current thread
> >   in switch_to_thread(). However since switch_to_thread() is written
>
> I don't see anything here that requires information gdbserver does not
> already have.  It knows the last thread stepped and the thread
> requested.
>
When we switch from one thread to another as a result of 'thread' command
the inferior_ptid gets updated to the new thread. Now, when I do a `next',
in the native gdb,  all the threads are resumed and in this case the step
signal will be given to the updated inferior_ptid. However, in the case
remote
debugging if I resume all the threads after the `thread' command by doing a
next on the new thread I will be sending a Hc-1 packet and the server is not
notified about the thread switch. To mention the current thread I need to
send
a Hc<id> packet so that the server will be informed about the current thread
of
execution and it will do a step on the correct thread. This is done by
calling set_thread().

> --
> Daniel Jacobowitz                           Carnegie Mellon University
> MontaVista Software                         Debian GNU/Linux Developer


[-- Attachment #1.2: Type: text/html, Size: 12597 bytes --]

[-- Attachment #2: Wipro_Disclaimer.txt --]
[-- Type: text/plain, Size: 477 bytes --]

**************************Disclaimer************************************
Information contained in this E-MAIL being proprietary to Wipro Limited
is 'privileged' and 'confidential' and intended for use only by the
individual or entity to which it is addressed. You are notified that any
use, copying or dissemination of the information contained in the E-MAIL
in any manner whatsoever is strictly prohibited.
********************************************************************