Re: Remote debugging of multithreaded programs using gdbserver

Subhashini Nagarajan Rao
2002-04-18T13:20:20Z

[-- Attachment #1.1: Type: text/plain, Size: 9647 bytes --]

Hi Daniel,

As you had stated there are instances of inheritance of the
concepts in the implementation of the gdbserver from the native gdb. This
is due to the fact that we want to get the code functionally correct and
running
and later revisions can be done to it. Reply to the comments follow.

Thanks & Regards,
Subha

----- Original Message -----
From: "Daniel Jacobowitz" <drow@mvista.com>
To: "Subhashini Nagarajan Rao" <subhashini.nagarajan@wipro.com>
Cc: <gdb@sources.redhat.com>; "Dan Kegel" <dank@kegel.com>; "Anantha
Subramanian" <anantha.subramanian@wipro.com>; "Ganeshan Kailasam"
<ganeshan.kailasam@wipro.com>; <sanjay.nair@wipro.com>
Sent: Thursday, April 18, 2002 11:02 AM
Subject: Re: Remote debugging of multithreaded programs using gdbserver

> First of all, gdbserver has been essentially rewritten between 5.1.1 and
> 5.2. I recommend you look at the current CVS. I have a great deal of
> (thread-related) work pending, also.
>
Fine
> Second - do you have details on the "broken" death event reporting in
> glibc 2.1.3? Does it make the death events unreliable, or outright
> harmful?
>
Gdb source says event reporting for thread death is broken for glibc 2.1.3.
As we are using this libc we are not presently enabling the thread death
events. Related comment from the native Gdb source code, (from thread-db.c)

/* FIXME: kettenis/2000-04-23: The event reporting facility is
broken for TD_DEATH events in glibc 2.1.3, so don't enable it for
now. */

> On Thu, Apr 18, 2002 at 10:29:21AM +0530, Subhashini Nagarajan Rao wrote:
> > Since event reporting facility is broken for TD_DEATH events in glibc
2.1.3
> > and the present implementation uses this version of glibc, thread
deaths are
> > are handled differently. While an immediate reporting of thread
creation
> > to gdbserver is mandatory to ensure that the gdbserver traces the
newly
> > created threads with immediate effect, thread death updation at the
client
> > needs could be procrastinated to the time when one of the following
> > events takes place:
> > (i) info threads command is fired at the client by the user.
> > (ii) context switch happens from one thread to another during
execution.
>
> "context switch" is misleading. The context switches of the process
> are irrelevant. I assume you mean whenever the client GDB requests a
> change of thread...
>
The thread list maintained in the client and the server are not in sync as
the thread death events are presently not enabled. So if there is a
context switch (changing of thread) to a dead thread then we will be having
problems hence requiring pruning of threads.

> > Gdb Client related and protocol enhancements towards the same are:
> >
> > The client sends a qSymbol:: packet to notify the target that
> > it is prepared to service symbol look-up requests, typically when new
> > objects are loaded. On the initial receipt of a qSymbol:: ,
> > the server enables thread event reporting and sends the thread
> > creation notification address through qSymbol! packet for the client
to add
> > breakpoint at that address. qSymbol! packet essentially sticks to the
the
> > following format:
> >
> > qSymbol!<thread creation notification address>:<thread death
notification address>
>
> I don't think this is the right approach. The client GDB should not be
> aware of anything involving thread_db. It's none of its business.
>
> OTOH, the bugs which cause thread_db to be pushed on the target stack
> when connected to a remote target definitely need to be fixed. I'm
> not quite decided on how to do this yet.
>
The server actually does not maintain a breakpoint list as done in the case
of gdb. It only has a mirror image of the breakpoints to be inserted which
it gets as a "Z0, " packet whenver there is a call to
target_insert_breakpoints. This was initially required for the
'fancy PC_adjustment' but was found to be useful in other cases also.

> Why do you bother with this if you have obviously given the server the
> ability to place breakpoints?
>
> > Handling of the Protocol packets :
> >
> > *) qfThreadInfo : When this packet is received libthread's thread
> > iterator function is called to collect all the active thread
> > ids. The collected thread ids are maintained in a
> > linked list, thread_list. These thread ids are then
> > copied from this list to the buffer to be sent to the
> > client. The purpose behind storing the thread ids
> > rather than writing directly into the buffer is to
> > overcome the limitation posed by the buffer size.
> >
> > *) qsThreadInfo : Copy from the next thread id from the thread_list
> > to the buffer if any more are to be copied else send 'l' as the reply.
> > Also free the thread_list once done.
>
> This information should be cached, there's no point in looking it up
> from scratch every time you are queried.
>
This is a performance issue which can be looked into once we are through
with resolving the design issues. Also if there is enabling of thread death
events it would be more helpful in caching of the thread ids.

> > 4. Adding breakpoints in the threads.
> >
> > The list of breakpoints in the program is maintained as a linked list
> > of breakpoint addresses. The breakpoint list is a doubly linked list
> > having the breakpoint address and a 2 character array('cc' for IA32).
> > A free pool of these nodes are maintained from where the breakpoint
list
> > draws out nodes as and when the need arises.
>
> "A two character array"? 0xCC is the code for a one-byte i386
> breakpoint, but I can't imagine what else you mean by this.
> Breakpoints can be a word, or possibly more.
>
The 'cc' mentioned here is the string which gets converted to the byte for
the purpose of adding breakpoints. Hence it was stated as a character array.

> > When one thread is stopped all the other threads are also stopped so
that all
> > are in sync and in control under the debugger. This is
> > done by calling stop_all_threads(). If the thread stopped is
single-stepped
> > then preference is given to this thread and allowed to resume next.
> > When none of the threads are single-stepped and more than 1
> > thread has received a SIGTRAP, randomnly 1 thread is chosen among the
threads
> > which received the trap signal and allowed to resume next
> > If any of the other threads has got a SIGTRAP because of a gdb
> > inserted breakpoint, the pc values have to be re-adjusted to ensure
that
> > these threads re-execute the breakpoint instruction. The selection of
the
> > thread to resume next is handled by select_event_pid().
>
> All of this appears to be taken straight from GDB's native thread debug
> package. None of it is clearly thought out in this context, IMHO.
> Limitations in GDB make it hard to fix these properly there (I spent a
> full day trying recently...) but gdbserver has plenty of room. For
> instance, I'd wager you stop all threads on a thread creation event.
> Also, there's no reason to do the fancy PC-adjustment. You should be
> able to save that stop event and re-report it later if nothing disturbs
> it.
>
Native Gdb implementation has been accepted as standardised and correct
functionally and this explains for code inheritance in the gdbserver.
We also did not understand your specific concern for stopping all the
threads.
Do you mean to say for non-thread creation breakpoints we are justified in
allowing other threads to continue? How would that handle cases like one of
the threads doing a `pthread_cond_timedwait'. However, we give the following
comment in the native code,

/* Don't report signals that GDB isn't interested in, such as
signals that are neither printed nor stopped upon. Stopping all
threads can be a bit time-consuming so if we want decent
performance with heavily multi-threaded programs, especially when
they're using a high frequency timer, we'd better avoid it if we
can. */

> > 8. Miscellaneous changes
> >
> > When the user fires a 'thread <id>' command , switch to the specified
> > thread but a step after this was not able to tell the server that the
> > thread to be stepped is the new thread and not the previous one.
Hence,
> > there is a need for the client, to remotely specify the current thread
> > in switch_to_thread(). However since switch_to_thread() is written
>
> I don't see anything here that requires information gdbserver does not
> already have. It knows the last thread stepped and the thread
> requested.
>
When we switch from one thread to another as a result of 'thread' command
the inferior_ptid gets updated to the new thread. Now, when I do a `next',
in the native gdb, all the threads are resumed and in this case the step
signal will be given to the updated inferior_ptid. However, in the case
remote
debugging if I resume all the threads after the `thread' command by doing a
next on the new thread I will be sending a Hc-1 packet and the server is not
notified about the thread switch. To mention the current thread I need to
send
a Hc<id> packet so that the server will be informed about the current thread
of
execution and it will do a step on the correct thread. This is done by
calling set_thread().

> --
> Daniel Jacobowitz Carnegie Mellon University
> MontaVista Software Debian GNU/Linux Developer

[-- Attachment #1.2: Type: text/html, Size: 12597 bytes --]

[-- Attachment #2: Wipro_Disclaimer.txt --]
[-- Type: text/plain, Size: 477 bytes --]

**************************Disclaimer************************************
Information contained in this E-MAIL being proprietary to Wipro Limited
is 'privileged' and 'confidential' and intended for use only by the
individual or entity to which it is addressed. You are notified that any
use, copying or dissemination of the information contained in the E-MAIL
in any manner whatsoever is strictly prohibited.
********************************************************************