Remote debugging of multithreaded programs using gdbserver

Subhashini Nagarajan Rao
2002-04-18T04:53:47Z

[-- Attachment #1.1: Type: text/plain, Size: 485 bytes --]

Hi All,

Remote debugging of multithreaded programs using gdbserver was
implemented with GDB-5.1.1 as the code base for LINUX platform
IA32 architecture.

Daniel Jacobowitz had earlier pointed out the importance
of resolving copyright issues before we could post our patch.
We have requested help from Jim Blandy towards resolving
this issue. In the meanwhile we would like to get some
feedback on our approach - kindly refer to the attachment.

Thanks & Regards,
Subha

[-- Attachment #1.2: Type: text/html, Size: 885 bytes --]

[-- Attachment #2: Rem_Debug_App.txt --]
[-- Type: text/plain, Size: 16775 bytes --]

Approach document for remote multi-threaded program
debuggability feature in gdb

* The scope of the project involves:
----------------------------------

Providing remote multi-threaded program debuggability feature
extensions in gdb on Pentium architectures.

The target platform for the scope mentioned above is Linux.

* Detailed Description of Software Components and Assumptions Made
----------------------------------------------------------------

Gdbserver is a component in the client-server gdb. The gdbserver resides
in the target system (ie. the system on which the program to be debugged will
be executed). As of today, multi-threaded program debuggability features in
gdbserver does not exist. The objective of this project is to provide this
feature.

The following items will be developed for this multi-threaded extension:

* Providing support to handle notification and tracking creation and
deletion of threads in the target program.
* Providing thread related information for all threads or for a specific
thread based on thread id. The thread related information could be
a) thread id, b) thread alive detection, c) thread name,
d) Detailed information on threads.
* Support for applying different commands on specific threads. The
commands could be back trace, procuring register content information, etc.

To provide multi-threaded extension support, the components that would
require modifications are:
Gdb Client-Server Protocol Enhancements
Gdb Client related changes
Gdb Server related changes

Description of each of the items to be implemented
--------------------------------------------------

1. Handle notification and tracking of thread creation and deletion in the
target program.

Thread creation is notified to the client through breakpoint mechanism
much in the same lines as native gdb. Essentially breakpoints
are inserted at the `notified' address, address to which execution is
guarenteed if thread creation event is enabled by libthread_db.
The client needs to update its breakpoint list with this address
to notify gdbserver on a thread creation event. Since libthread_db interfaces
are used towards procurement of these addresses, the proc_service
routines used by libthread_db needs to be implemented in the gdbserver.
Since event reporting facility is broken for TD_DEATH events in glibc 2.1.3
and the present implementation uses this version of glibc, thread deaths are
are handled differently. While an immediate reporting of thread creation
to gdbserver is mandatory to ensure that the gdbserver traces the newly
created threads with immediate effect, thread death updation at the client
needs could be procrastinated to the time when one of the following
events takes place:
(i) info threads command is fired at the client by the user.
(ii) context switch happens from one thread to another during execution.
It is crucial that a context switch does not happen from or to a dead
thread. This is ensured by making a call to prune_threads which
clobbers thread ids of dead threads from the thread list.

For both these cases, the client thread updates its threadlist by
sending the T(thread alive packet ). gdbserver however clobbers
dead threads from its thread list as soon as a thread dies as it can
easily track dead threads through the waitpid status flag.

Gdb Client related and protocol enhancements towards the same are:

The client sends a qSymbol:: packet to notify the target that
it is prepared to service symbol look-up requests, typically when new
objects are loaded. On the initial receipt of a qSymbol:: ,
the server enables thread event reporting and sends the thread
creation notification address through qSymbol! packet for the client to add
breakpoint at that address. qSymbol! packet essentially sticks to the the
following format:

qSymbol!<thread creation notification address>:<thread death notification address>

Since the present implementation does not rely on libthread_db
for thread death notification, qSymbol! sets 0 towards thread death
notification so that client ignores the same.

On subsequent receipts of qSymbol::, the server checks if it has been
stopped because of a thread creation event, through libthread_db interfaces,
in which case it adds the new threadid to its locally maintained list of
threads that it traces.

Functions taking care of this - remote_check_symbols(), (to get the breakpoint
address), set_td_create_bp_addr(), set_td_death_bp_addr()

Functions set_td_create_bp_addr & set_td_death_bp_addr are inturn
modified to call create_thread_event_breakpoint for the
insertion of the breakpoint in the breakpoint chain.

Gdb Server related changes are :

Once the qSymbol packet is sent -

*) Open a connection to libthread_db to allocate a thread agent. This
needs to be done only if the thread agent was not previously allocated.

function taking care of this - thread_db_new_agent().

*) Set the process wide event masks and obtain the thread creation
notification address from the thread library using libthread_db
interfaces and also enable event reporting.

function taking care of this - enable_thread_event_reporting().

*) check_event() checks for if the threads have been stopped
because of a thread creation event/breakpoint in which case, it calls
attach_thread() which does a PT_ATTACH to the new thread so that the
created thread can also be traced.

The proc_service routine ps_pglobal_lookup in the server side is
responsible for getting the symbol values required from the client.

2. Maintenance of list of threads in the target program.

The Server maintains various lists of thread ids for the following
purpose.

*) threadid_list = a linked list to store the information of the active
threads that are traced.
*) stopped_pid_list = a linked list of thread ids of the stopped threads.

The following functions are used to work on the lists of thread ids.

*) in_threadid_list = to check if a thread is present in the
threadid_list.
*) add_thread = add a thread id to the threadid_list.
*) delete_thread = delete a thread id from the threadid_list.
*) in_stopped_pid_list = to check if a thread is present in the
stopped_pid_list.
*) insert_stopped_pid_list = insert a stopped thread's id into the
stopped_pid_list.
*) delete_stopped_pid_list = delete from the stopped_pid_list the
thread's id which resumed its execution from the stopped state.

3. Provide information on threads.

For 'info threads' command it is required to first find all the live
threads in the target program. Gdb Client sends a qfThreadInfo packet to
the server to get the thread ids. If all the threads are not sent
due to packet size limitations, a qsThreadInfo packet is sent for
subsequent threads id retrievals until it receives an 'l'.
To get the extra information on the thread like, running or blocked
statuses, qThreadExtraInfo packet is sent. These changes are presently
inherited from gdb5.1.1 original code from the client.

Handling of the Protocol packets :

*) qfThreadInfo : When this packet is received libthread's thread
iterator function is called to collect all the active thread
ids. The collected thread ids are maintained in a
linked list, thread_list. These thread ids are then
copied from this list to the buffer to be sent to the
client. The purpose behind storing the thread ids
rather than writing directly into the buffer is to
overcome the limitation posed by the buffer size.

*) qsThreadInfo : Copy from the next thread id from the thread_list
to the buffer if any more are to be copied else send 'l' as the reply.
Also free the thread_list once done.

*) qThreadExtraInfo : This provides the additional thread information
like the state of the thread. For this purpose libthread_db interfaces
are used. Functions used to implement this are
i386_get_thread_info, mem2hstr, thr_state_string.
The thread iterator function is used to obtain the state of the threads.

4. Adding breakpoints in the threads.

The list of breakpoints in the program is maintained as a linked list
of breakpoint addresses. The breakpoint list is a doubly linked list
having the breakpoint address and a 2 character array('cc' for IA32).
A free pool of these nodes are maintained from where the breakpoint list
draws out nodes as and when the need arises.

Whenever a breakpoint is to be inserted or deleted the client sends
'Z' or 'z' packet. If it is a 'Z' packet, the new breakpoint address
is added to the breakpoint list if it is not already added and if it is
a 'z' packet the specified breakpoint address is deleted from the
breakpoint list.

The functions parse_Z and parse_z takes care of inserting or deleting
the breakpoint from the breakpoint list.

Note - some portions of the code like breakpoint handling in the server
(parseZ() function) was taken from a patch posted earlier on the
mailing list for gdb-5.0

5. Procedure of waiting for the inferior process to return the status.

In the Server, mywait() handles this.
*) To denote the thread of interest the variables used are
o general_thread - this thread's id is needed for all info retrieval
like register set details.
o thread_to_cont - this is the `main' thread that needs to be
subsequently resumed( besides the threads that are anyway stopped by gdb )
o cont_thread - thread which needs to continue.
o inferior_pid - thread corresponding to main().

*) 'waitpid' is used to wait for the inferior process and know the status.

WCLONE option needs to be passed to wait on clonned threads. Hence
waipid() needs to be called with WCLONE and WNOHANG(to prevent
hangs ) till a valid pid is returned by waitpid(). To ensure
that such a framework works fine for non-threaded programs, the WCLONE
option is toggled everytime.

When waitpid() returns because of a thread death, the locally maintained
threadlist is updated to reflect the same to prevent this threadid
from subsequently being traced.

If a thread has stopped due to TARGET_SIGNAL_STOP,
TARGET_SIGNAL_REALTIME_32/33 or a TARGET_SIGNAL_SIGCHLD - these are
inherited from the client code which essentially does not report
all signals to the native gdb which is not of its interest, a
ptrace(CONT) is done and the above steps are performed again. If the
thread is stopped by any other signal other than the above it is inserted
in the stopped_pid_list. The signals mentioned above are stored as an array
with the values denoting the action to be taken. It is stored as three
arrays wherein we specify whether the signal should be handled by the
debugger or ignored. This is a copy of the signals array which the client
stores. Presently the array is initialized to required values to denote
the present action exhibited by the client over these signals.
Any change in the client side will require a change in the server also.

When one thread is stopped all the other threads are also stopped so that all
are in sync and in control under the debugger. This is
done by calling stop_all_threads(). If the thread stopped is single-stepped
then preference is given to this thread and allowed to resume next.
When none of the threads are single-stepped and more than 1
thread has received a SIGTRAP, randomnly 1 thread is chosen among the threads
which received the trap signal and allowed to resume next
If any of the other threads has got a SIGTRAP because of a gdb
inserted breakpoint, the pc values have to be re-adjusted to ensure that
these threads re-execute the breakpoint instruction. The selection of the
thread to resume next is handled by select_event_pid().

For every continue or next the server replies with the thread id. The
client reads this threadid and if this is not in its maintained
threadid list, it updates the same.

6. Procedure of resuming the threads from the stopped state.

In the Client side,

*) remote_resume() calls set_thread() to specify the thread to be resumed.
In case of resume all continue_thread is set to -1 to indicate this.

In the Server side, myresume() handles this.

*) If the cont_thread is -1 it means all the threads need to be resumed.
Now, the thread_to_cont holds the thread which is selected as the event thread
by select_event_pid. This thread is subsequently stepped or continued
as per the user's request and other threads are continued. For every thread
in the threadid_list there is a field in its data structure to store the step
information. When a thread is continued this field is reset. For the
thread corresponding to the thread_to_cont, the step information field is
updated with 1 or 0 depending on whether a step/cont is given to this
thread.
*) If cont_thread is 0 - which essentially means that this program is
non-threaded - a step/cont is given to inferior_pid and this thread is
deleted from the stopped_pid_list. Otherwise the step/cont is given to the
respective thread and deleted from the stopped_pid_list. The step information
for this thread is also updated in the threadid_list.

7. Other relevant thread related pkts changes:

*) action for T pkt from client - check in the threadid_list to know
whether the req thread is alive or not.
*) action for Hc pkt from client - the id sent along with this is set to
cont_thread.
If it is not a -1 then update the thread_to_cont also to this value
immediately. In resume_all_stopped_threads thread ids are compared against
thread_to_cont where the assumption is thread_to_cont is the thread of
interest to be resumed next. So, this needs to be updated to reflect this.
*) action for Hg pkt from client - first need to chk if the specified
thread is in stopped state. If so respond immediately by sending the register
contents. If not first stop this thread, insert it into the stopped_pid_list
and then proceed.

8. Miscellaneous changes

When the user fires a 'thread <id>' command , switch to the specified
thread but a step after this was not able to tell the server that the
thread to be stepped is the new thread and not the previous one. Hence,
there is a need for the client, to remotely specify the current thread
in switch_to_thread(). However since switch_to_thread() is written
for native gdb, and our present requirement is to call set_thread() which
is specific to remote debugging, we cannot call set_thread directly.
The proposed solution is to have a hook function(remote_set_thread_hook)
which is set in remote.c during remote_open and switch_to_thread()
calls this function when the hook is non-zero (essentially remote-debugging)
and does not call it when the hook is NULL(essentially native mode).
Also in the server for the 'Hc' packet we set update the thread_to_cont to
this value so that a resume following this can compare the thread ids
against the correct thread_to_cont and fires does cont/step on this
thread as is requested by the user accordingly.

* Assumptions made :
----------------

For the development of this project, following Assumptions are
considered:

Current Scope of gdbserver enhancements is for debugging User Space
applications
Gdb base to be used for development of the project would be gdb
version 5.1.1.

* Functionalities yet to be implemented :
-------------------------------------

i) Handling of ^C - This is presently not tested exhaustively.

ii) Handling of signals if they are dynamically changed by the user as the
actions to be done for a particular signal is maintained as a static
array in the gdbserver.

iii) whenever a new thread comes up gdb prints "[New Thread tid]".
In this implementation presently lwpid gets printed instead of tid. This
needs to be changed.

* Testing through DejaGNU :
-----------------------

pthreads.c/pthreads.exp testsuite was tested and verified with this fix.

******************************************************************************

[-- Attachment #3: Wipro_Disclaimer.txt --]
[-- Type: text/plain, Size: 477 bytes --]

**************************Disclaimer************************************
Information contained in this E-MAIL being proprietary to Wipro Limited
is 'privileged' and 'confidential' and intended for use only by the
individual or entity to which it is addressed. You are notified that any
use, copying or dissemination of the information contained in the E-MAIL
in any manner whatsoever is strictly prohibited.
********************************************************************