gdb and cloned process

gdb and cloned process

[Lukasz Lempart]

I am using gdb 6.8 and running linux 2.6.9 on a 64bit AMD Opteron.

I am working with a multi-threaded application (using glibc's pthread
implementation). One of the threads, clones another process using the
following flags:

CLONE_VM | CLONE_UNTRACED | CLONE_DETACHED | CLONE_PARENT | CLONE_FILES

The cloned process also calls setsid creating its own process group.
The idea here is to peek at the memory of the original application
without stopping it, by attaching gdb to the cloned process.

However, when I attach for the first time, the cloned process as well
as the original and all of its threads are stopped under ptrace.
Detaching, and then reattaching results in the correct behavior of
only the cloned process being stopped under ptrace. The "info threads"
command, once again places the original process and all its threads
under ptrace. stracing gdb I can see an explicit ptrace attach with
the pid of the original process.

Attaching to the original process never causes the cloned process to
be put under ptrace even when "info threads" command is issued.
However, "info threads" causes duplicates of all threads to be
displayed.

Everything seems fine w.r.t. to the system. Varieties of top show the
two running as separate processes. /proc/<pid>/status shows the
correct number of threads (1 for the cloned, multiple for the
original). The process group ids and thread group ids are different.

My questions are as follows:
How does gdb (through libthread_db) figure out what threads belong to a process?
Does anyone have any idea what is causing this behavior?
Is there currently a way to disable thread debugging in gdb?

Thank you for any help you can afford.

Lukasz

Re: gdb and cloned process

[Daniel Jacobowitz]

On Wed, Oct 22, 2008 at 05:09:32PM -0700, Lukasz Lempart wrote:
> How does gdb (through libthread_db) figure out what threads belong to a process?

The thread library maintains an internal list of threads.  If you've
cloned the process, without telling the C library about that, you're
going to end up with the same list of threads; so the behavior you
describe is not surprising.

> Is there currently a way to disable thread debugging in gdb?

Not really.  You might be able to preload a dummy libthread_db.so.1
that always failed to detect new threads.

-- 
Daniel Jacobowitz
CodeSourcery

Re: gdb and cloned process

[Lukasz Lempart]

On Wed, Oct 22, 2008 at 8:05 PM, Daniel Jacobowitz <drow@false.org> wrote:
> On Wed, Oct 22, 2008 at 05:09:32PM -0700, Lukasz Lempart wrote:
>> How does gdb (through libthread_db) figure out what threads belong to a process?
>
> The thread library maintains an internal list of threads.  If you've
> cloned the process, without telling the C library about that, you're
> going to end up with the same list of threads; so the behavior you
> describe is not surprising.
>
Is there a way to do this? I can't seem to find anything in
libthread_db that would allow me to do this. Furthermore, the two
processes share their entire address space, so any change made to the
internal list of threads for one process would have impact on the
other one.

Re: gdb and cloned process

[Daniel Jacobowitz]

On Thu, Oct 23, 2008 at 10:13:11AM -0700, Lukasz Lempart wrote:
> On Wed, Oct 22, 2008 at 8:05 PM, Daniel Jacobowitz <drow@false.org> wrote:
> > On Wed, Oct 22, 2008 at 05:09:32PM -0700, Lukasz Lempart wrote:
> >> How does gdb (through libthread_db) figure out what threads belong to a process?
> >
> > The thread library maintains an internal list of threads.  If you've
> > cloned the process, without telling the C library about that, you're
> > going to end up with the same list of threads; so the behavior you
> > describe is not surprising.
> >
> Is there a way to do this? I can't seem to find anything in
> libthread_db that would allow me to do this. Furthermore, the two
> processes share their entire address space, so any change made to the
> internal list of threads for one process would have impact on the
> other one.

Not that I can think of.  You'd have to hack up gdb a bit.

-- 
Daniel Jacobowitz
CodeSourcery

Re: gdb and cloned process

[Michael Snyder]

Daniel Jacobowitz wrote:
> On Wed, Oct 22, 2008 at 05:09:32PM -0700, Lukasz Lempart wrote:
>> How does gdb (through libthread_db) figure out what threads belong to a process?
> 
> The thread library maintains an internal list of threads.  If you've
> cloned the process, without telling the C library about that, you're
> going to end up with the same list of threads; so the behavior you
> describe is not surprising.
> 
>> Is there currently a way to disable thread debugging in gdb?
> 
> Not really.  You might be able to preload a dummy libthread_db.so.1
> that always failed to detect new threads.

What if you strip libthread.so?  Isn't that supposed to
cause thread debugging to fail?

Re: gdb and cloned process

[Lukasz Lempart]

On Thu, Oct 23, 2008 at 11:26 AM, Michael Snyder <msnyder@vmware.com> wrote:
> Daniel Jacobowitz wrote:
>>
>> On Wed, Oct 22, 2008 at 05:09:32PM -0700, Lukasz Lempart wrote:
>>>
>>> How does gdb (through libthread_db) figure out what threads belong to a
>>> process?
>>
>> The thread library maintains an internal list of threads.  If you've
>> cloned the process, without telling the C library about that, you're
>> going to end up with the same list of threads; so the behavior you
>> describe is not surprising.
>>
>>> Is there currently a way to disable thread debugging in gdb?
>>
>> Not really.  You might be able to preload a dummy libthread_db.so.1
>> that always failed to detect new threads.
>
> What if you strip libthread.so?  Isn't that supposed to
> cause thread debugging to fail?
>

Stripping libthread_db.so seems to do the trick. Thanks for the
suggestion. Keeping both version of the library around and just
changing LD_LIBRARY_PATH to point to the one I want seems to be the
most portable way to handle debugging both the original and the cloned
process. A gdb command to turn on/off thread debugging would be very
nice to have though.

Re: gdb and cloned process

[Michael Snyder]

Lukasz Lempart wrote:
> On Thu, Oct 23, 2008 at 11:26 AM, Michael Snyder <msnyder@vmware.com> wrote:
>> Daniel Jacobowitz wrote:
>>> On Wed, Oct 22, 2008 at 05:09:32PM -0700, Lukasz Lempart wrote:
>>>> How does gdb (through libthread_db) figure out what threads belong to a
>>>> process?
>>> The thread library maintains an internal list of threads.  If you've
>>> cloned the process, without telling the C library about that, you're
>>> going to end up with the same list of threads; so the behavior you
>>> describe is not surprising.
>>>
>>>> Is there currently a way to disable thread debugging in gdb?
>>> Not really.  You might be able to preload a dummy libthread_db.so.1
>>> that always failed to detect new threads.
>> What if you strip libthread.so?  Isn't that supposed to
>> cause thread debugging to fail?
>>
> 
> Stripping libthread_db.so seems to do the trick. Thanks for the
> suggestion. Keeping both version of the library around and just
> changing LD_LIBRARY_PATH to point to the one I want seems to be the
> most portable way to handle debugging both the original and the cloned
> process. A gdb command to turn on/off thread debugging would be very
> nice to have though.

Cool... so tell us a little bit about your application, please.

What sorts of data are you monitoring in the running process,
how are you monitoring the data, how often are you updating it,
and how useful has the technique proven to be?

And have you thought ahead as far as actually modifying
the data of the running process?

Re: gdb and cloned process

[Michael Snyder]

Lukasz Lempart wrote:
> I am using gdb 6.8 and running linux 2.6.9 on a 64bit AMD Opteron.
> 
> I am working with a multi-threaded application (using glibc's pthread
> implementation). One of the threads, clones another process using the
> following flags:
> 
> CLONE_VM | CLONE_UNTRACED | CLONE_DETACHED | CLONE_PARENT | CLONE_FILES
> 
> The cloned process also calls setsid creating its own process group.
> The idea here is to peek at the memory of the original application
> without stopping it, by attaching gdb to the cloned process.

I think your idea is interesting, but because of the intimate
relationship between linux clone and threads, the idea might not
be workable with multi-threaded programs.

Linux threads are built on top of clone.  It's hard to
separate them, and I'm not sure that we have ever really
tried to, since use of both in the same program is probably
quite rare.

The only suggestion I can think to offer is to build a
special gdb with threads disabled, and use that one to
view the clone.