howto debug libthread_db

howto debug libthread_db

[Carmelo Amoroso]

Hi All,
I'm trying to debug a simple multithread application on a remote target,
and I like to debug the libthread_db itself... is it possible to do it? 
is it possible to set some breakpoints and stepping through?
Is there anyone already played with it?

I tried to use symbol-file to gdb client, and I successfully added a 
breakpoint, but I cannot reach it after I connected to the target and 
issued 'continue'.

Any help will be appreciated

Thanks,
Carmelo

Re: howto debug libthread_db

[Daniel Jacobowitz]

On Tue, Sep 26, 2006 at 04:06:57PM +0200, Carmelo Amoroso wrote:
> Hi All,
> I'm trying to debug a simple multithread application on a remote target,
> and I like to debug the libthread_db itself... is it possible to do it? 
> is it possible to set some breakpoints and stepping through?
> Is there anyone already played with it?
> 
> I tried to use symbol-file to gdb client, and I successfully added a 
> breakpoint, but I cannot reach it after I connected to the target and 
> issued 'continue'.
> 
> Any help will be appreciated

Sorry, but you need to be more precise.  What exactly did you do - the
entire GDB session?  What didn't work?  Do you really mean
libthread_db, or do you mean libpthread?

-- 
Daniel Jacobowitz
CodeSourcery

Re: howto debug libthread_db

[Carmelo Amoroso]

Daniel Jacobowitz wrote:
> On Tue, Sep 26, 2006 at 04:06:57PM +0200, Carmelo Amoroso wrote:
>> Hi All,
>> I'm trying to debug a simple multithread application on a remote target,
>> and I like to debug the libthread_db itself... is it possible to do it? 
>> is it possible to set some breakpoints and stepping through?
>> Is there anyone already played with it?
>>
>> I tried to use symbol-file to gdb client, and I successfully added a 
>> breakpoint, but I cannot reach it after I connected to the target and 
>> issued 'continue'.
>>
>> Any help will be appreciated
> 
> Sorry, but you need to be more precise.  What exactly did you do - the
> entire GDB session?  What didn't work?  Do you really mean
> libthread_db, or do you mean libpthread?
> 
Yes, libthread_db exactly... I got a SIGSEV due to a unaligned access
just after exiting from td_ta_map_lwp2thr function, and I never return 
to the caller (iterate_thread_list).
I added a lot of printf into my own libthread_db and I can track all the
flow until the end td_ta_map_lwp2thr... and then nothing

The thread_db library belongs to the uClibc-nptl port on SH.
gdbserver has been linked against uClibc-nptl port and works
fine for non multithread example

Carmelo

Re: howto debug libthread_db

[Michael Snyder]

On Tue, 2006-09-26 at 16:06 +0200, Carmelo Amoroso wrote:
> Hi All,
> I'm trying to debug a simple multithread application on a remote target,
> and I like to debug the libthread_db itself... is it possible to do it? 
> is it possible to set some breakpoints and stepping through?
> Is there anyone already played with it?
> 
> I tried to use symbol-file to gdb client, and I successfully added a 
> breakpoint, but I cannot reach it after I connected to the target and 
> issued 'continue'.
> 
> Any help will be appreciated

Interesting.  I haven't done it (or not much anyway).

The first thing you must bear in mind is that libthread_db is not
part of the target application -- it is part of gdb (or in the
remote debugging context, part of gdbserver).  So to debug it, 
you must debug gdb, or in your case gdbserver.

It might be easier if you try this out first using a native gdb, 
debugging itself.  

When you debug gdb with gdb, it is easier if you change the prompt.
I usually do this in the 'parent' gdb, so I can tell it apart from
the 'child' gdb:

	(gdb) set prompt (GDB) 
	(GDB) 

Now, in the parent GDB, you should be able to set your 
breakpoints in libthread_db, then start the child gdb, 
load the multi-threaded program, and begin debugging it.

When you get ready to try this with gdbserver, you will
have to start one gdbserver and attach to it with another
gdbserver.  Then you'll again have two gdbs running on the
host side, which you can again think of as parent and child, 
only now the parent is debugging the child's gdbserver, not
the child gdb itself.

Re: howto debug libthread_db

[Andreas Schwab]

Michael Snyder <Michael.Snyder@palmsource.com> writes:

> When you debug gdb with gdb, it is easier if you change the prompt.
> I usually do this in the 'parent' gdb, so I can tell it apart from
> the 'child' gdb:

Or just run it from the gdb build directory, which contains a .gdbinit
that does more useful things.

Andreas.

-- 
Andreas Schwab, SuSE Labs, schwab@suse.de
SuSE Linux Products GmbH, Maxfeldstraße 5, 90409 Nürnberg, Germany
PGP key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: howto debug libthread_db

[Carmelo Amoroso]

Andreas Schwab wrote:
> Michael Snyder <Michael.Snyder@palmsource.com> writes:
> 
>> When you debug gdb with gdb, it is easier if you change the prompt.
>> I usually do this in the 'parent' gdb, so I can tell it apart from
>> the 'child' gdb:
> 
> Or just run it from the gdb build directory, which contains a .gdbinit
> that does more useful things.
> 
Hi,
my one is a cross build, is it ok copying the .gdbinit from the host 
build directory to the target platform ?
> Andreas.
> 
Carmelo

Re: howto debug libthread_db

[Michael Snyder]

On Wed, 2006-09-27 at 08:23 +0200, Carmelo Amoroso wrote:
> Andreas Schwab wrote:
> > Michael Snyder <Michael.Snyder@palmsource.com> writes:
> > 
> >> When you debug gdb with gdb, it is easier if you change the prompt.
> >> I usually do this in the 'parent' gdb, so I can tell it apart from
> >> the 'child' gdb:
> > 
> > Or just run it from the gdb build directory, which contains a .gdbinit
> > that does more useful things.
> > 
> Hi,
> my one is a cross build, is it ok copying the .gdbinit from the host 
> build directory to the target platform ?

Won't do any good -- gdbserver does not use that file.
It has to be visible to gdb.

Re: howto debug libthread_db

[Carmelo Amoroso]

Michael Snyder wrote:
> On Tue, 2006-09-26 at 16:06 +0200, Carmelo Amoroso wrote:
>> Hi All,
>> I'm trying to debug a simple multithread application on a remote target,
>> and I like to debug the libthread_db itself... is it possible to do it? 
>> is it possible to set some breakpoints and stepping through?
>> Is there anyone already played with it?
>>
>> I tried to use symbol-file to gdb client, and I successfully added a 
>> breakpoint, but I cannot reach it after I connected to the target and 
>> issued 'continue'.
>>
>> Any help will be appreciated
> 
> Interesting.  I haven't done it (or not much anyway).
> 
> The first thing you must bear in mind is that libthread_db is not
> part of the target application -- it is part of gdb (or in the
> remote debugging context, part of gdbserver).  So to debug it, 
> you must debug gdb, or in your case gdbserver.
> 
Yes. it's what I want to do
> It might be easier if you try this out first using a native gdb, 
> debugging itself.  
> 
> When you debug gdb with gdb, it is easier if you change the prompt.
> I usually do this in the 'parent' gdb, so I can tell it apart from
> the 'child' gdb:
> 
> 	(gdb) set prompt (GDB) 
> 	(GDB) 
> 
> Now, in the parent GDB, you should be able to set your 
> breakpoints in libthread_db, then start the child gdb, 
> load the multi-threaded program, and begin debugging it.
> 
Does the native gdb use the thread_db as the gdbserver does?
I was thinking to run gdb --args my-gdbserver <application>
ont the target, and connect to the my-gdbserver from the host
just to trigger the thread_db initialization process.
Comments?

> When you get ready to try this with gdbserver, you will
> have to start one gdbserver and attach to it with another
> gdbserver.  Then you'll again have two gdbs running on the
> host side, which you can again think of as parent and child, 
> only now the parent is debugging the child's gdbserver, not
> the child gdb itself.
> 

Well, just before reading your reply, I tried with this:

target:
gdbserver localhost:999 my-gdbserver localhost:998 <applcation>

host:
xxx-gdb my-gdbserver (connecting to port 999)

xxx-gdb application (connecting to port 998)

Wit the first gdb session I was able to set a breakpoint into thread_db,
but not able to stepping through the code.

Is this approach as well as the gdb native session.

Btw, I'll try again (both the native and the remote technique)

Thanks a lot

Carmelo

Re: howto debug libthread_db

[Michael Snyder]

On Wed, 2006-09-27 at 08:23 +0200, Carmelo Amoroso wrote:
> Michael Snyder wrote:
> > On Tue, 2006-09-26 at 16:06 +0200, Carmelo Amoroso wrote:
> >> Hi All,
> >> I'm trying to debug a simple multithread application on a remote target,
> >> and I like to debug the libthread_db itself... is it possible to do it? 
> >> is it possible to set some breakpoints and stepping through?
> >> Is there anyone already played with it?
> >>
> >> I tried to use symbol-file to gdb client, and I successfully added a 
> >> breakpoint, but I cannot reach it after I connected to the target and 
> >> issued 'continue'.
> >>
> >> Any help will be appreciated
> > 
> > Interesting.  I haven't done it (or not much anyway).
> > 
> > The first thing you must bear in mind is that libthread_db is not
> > part of the target application -- it is part of gdb (or in the
> > remote debugging context, part of gdbserver).  So to debug it, 
> > you must debug gdb, or in your case gdbserver.
> > 
> Yes. it's what I want to do

> > It might be easier if you try this out first using a native gdb, 
> > debugging itself.  
> > 
> > When you debug gdb with gdb, it is easier if you change the prompt.
> > I usually do this in the 'parent' gdb, so I can tell it apart from
> > the 'child' gdb:
> > 
> > 	(gdb) set prompt (GDB) 
> > 	(GDB) 
> > 
> > Now, in the parent GDB, you should be able to set your 
> > breakpoints in libthread_db, then start the child gdb, 
> > load the multi-threaded program, and begin debugging it.
> > 
> Does the native gdb use the thread_db as the gdbserver does?

Yes.

> I was thinking to run gdb --args my-gdbserver <application>
> on the target, and connect to the my-gdbserver from the host
> just to trigger the thread_db initialization process.
> Comments?

Oh, so you can actually run gdb on the target?
Yes, that would make things simpler, and what you
suggest above is pretty much what I would recommend.


> > When you get ready to try this with gdbserver, you will
> > have to start one gdbserver and attach to it with another
> > gdbserver.  Then you'll again have two gdbs running on the
> > host side, which you can again think of as parent and child, 
> > only now the parent is debugging the child's gdbserver, not
> > the child gdb itself.
> > 
> 
> Well, just before reading your reply, I tried with this:
> 
> target:
> gdbserver localhost:999 my-gdbserver localhost:998 <applcation>
> 
> host:
> xxx-gdb my-gdbserver (connecting to port 999)
> 
> xxx-gdb application (connecting to port 998)
> 
> With the first gdb session I was able to set a breakpoint into
> thread_db, but not able to stepping through the code.
> 
> Is this approach as well as the gdb native session.

Hmmm, that's pretty much what I would have tried.
What happened when you tried to step?  How did it fail?

Re: howto debug libthread_db

[Carmelo Amoroso]

Michael Snyder wrote:
> On Wed, 2006-09-27 at 08:23 +0200, Carmelo Amoroso wrote:
>> Michael Snyder wrote:
>>> On Tue, 2006-09-26 at 16:06 +0200, Carmelo Amoroso wrote:
>>>> Hi All,
>>>> I'm trying to debug a simple multithread application on a remote target,
>>>> and I like to debug the libthread_db itself... is it possible to do it? 
>>>> is it possible to set some breakpoints and stepping through?
>>>> Is there anyone already played with it?
>>>>
>>>> I tried to use symbol-file to gdb client, and I successfully added a 
>>>> breakpoint, but I cannot reach it after I connected to the target and 
>>>> issued 'continue'.
>>>>
>>>> Any help will be appreciated
>>> Interesting.  I haven't done it (or not much anyway).
>>>
>>> The first thing you must bear in mind is that libthread_db is not
>>> part of the target application -- it is part of gdb (or in the
>>> remote debugging context, part of gdbserver).  So to debug it, 
>>> you must debug gdb, or in your case gdbserver.
>>>
>> Yes. it's what I want to do
> 
>>> It might be easier if you try this out first using a native gdb, 
>>> debugging itself.  
>>>
>>> When you debug gdb with gdb, it is easier if you change the prompt.
>>> I usually do this in the 'parent' gdb, so I can tell it apart from
>>> the 'child' gdb:
>>>
>>> 	(gdb) set prompt (GDB) 
>>> 	(GDB) 
>>>
>>> Now, in the parent GDB, you should be able to set your 
>>> breakpoints in libthread_db, then start the child gdb, 
>>> load the multi-threaded program, and begin debugging it.
>>>
>> Does the native gdb use the thread_db as the gdbserver does?
> 
> Yes.
> 
>> I was thinking to run gdb --args my-gdbserver <application>
>> on the target, and connect to the my-gdbserver from the host
>> just to trigger the thread_db initialization process.
>> Comments?
> 
> Oh, so you can actually run gdb on the target?
Yes, I'm using it after I read your reply. So I'm running
a working gdb (glibc/sh4 based) to debug a half-working gdbserver 
(uClibc-nptl/sh4 based).
I was able to set breakpoint on gdbserver itself 
(thread_db_find_new_threads function for example) and to step into
my thread_db library. But up to now I could not reach the 
td_ta_map_lp2thr function due to some failures in packets transmission
between gdb remote client and target gdbserver, so the tbread_db
initializaition is failing before entering in that function.
I'm still working on it.

I'd like to describe the problem I have in case some one may have an idea:
The stack is as follows:

td_ta_map_lwp2thr
iterate_list_thread
td_ta_thr_iter
thread_db_find_new_threads
thread_db_init

The td_ta_map_lwp2thr funtcion reach its end successfully
(performing any symbol lookup; I also printed out the thread pointer
and its value is just the same as printed by the pthread_self function
from the libpthread), but when returns to the caller it produces a 
SIGSEV due to a misaligned access (as reported by dmesg command).
Any printf after the function's invocation is never executed!

The thread_db code is he same as the glibc, so I cannot understand
where it's failing. The gdbserver version is 6.4 (built using a
sh4-linux-gcc configured for uClibc) and it is the same code used to 
build the glibc based version.


> Yes, that would make things simpler, and what you
> suggest above is pretty much what I would recommend.
> 
> 
>>> When you get ready to try this with gdbserver, you will
>>> have to start one gdbserver and attach to it with another
>>> gdbserver.  Then you'll again have two gdbs running on the
>>> host side, which you can again think of as parent and child, 
>>> only now the parent is debugging the child's gdbserver, not
>>> the child gdb itself.
>>>
>> Well, just before reading your reply, I tried with this:
>>
>> target:
>> gdbserver localhost:999 my-gdbserver localhost:998 <applcation>
>>
>> host:
>> xxx-gdb my-gdbserver (connecting to port 999)
>>
>> xxx-gdb application (connecting to port 998)
>>
>> With the first gdb session I was able to set a breakpoint into
>> thread_db, but not able to stepping through the code.
>>
>> Is this approach as well as the gdb native session.
> 
> Hmmm, that's pretty much what I would have tried.
> What happened when you tried to step?  How did it fail?
> 

Difficult to describe; I'm trying to follow the native approach for now,
but I'd like trying this again later; I'll report any failure or success
with more details.

Thanks a lot again
Carmelo

Re: howto debug libthread_db

[Carmelo Amoroso]

Well,
after a few days of debugging my gdbserver (with uClibc-nptl on sh4),
I found the bug into uClibc,
that was causing a stack frame corruption due to a buffer overrun.
This was the reason for which the td_ta_map_lwp2thr was never
returning to the caller
(the stack location where the PR had been saved was corupted).

It was a very interesting debugging session. I successfully used a
native gdb (glibc base)on target
to debug a gdbserver (uClinc-nptl based) I connect to with a gdb
client from a linux host.

At the end I did not need to use a double remote gdb session.

Thanks to Michael for his very usefull tips.

Carmelo


On 9/27/06, Carmelo Amoroso <carmelo73@gmail.com> wrote:
> Michael Snyder wrote:
> > On Wed, 2006-09-27 at 08:23 +0200, Carmelo Amoroso wrote:
> >> Michael Snyder wrote:
> >>> On Tue, 2006-09-26 at 16:06 +0200, Carmelo Amoroso wrote:
> >>>> Hi All,
> >>>> I'm trying to debug a simple multithread application on a remote target,
> >>>> and I like to debug the libthread_db itself... is it possible to do it?
> >>>> is it possible to set some breakpoints and stepping through?
> >>>> Is there anyone already played with it?
> >>>>
> >>>> I tried to use symbol-file to gdb client, and I successfully added a
> >>>> breakpoint, but I cannot reach it after I connected to the target and
> >>>> issued 'continue'.
> >>>>
> >>>> Any help will be appreciated
> >>> Interesting.  I haven't done it (or not much anyway).
> >>>
> >>> The first thing you must bear in mind is that libthread_db is not
> >>> part of the target application -- it is part of gdb (or in the
> >>> remote debugging context, part of gdbserver).  So to debug it,
> >>> you must debug gdb, or in your case gdbserver.
> >>>
> >> Yes. it's what I want to do
> >
> >>> It might be easier if you try this out first using a native gdb,
> >>> debugging itself.
> >>>
> >>> When you debug gdb with gdb, it is easier if you change the prompt.
> >>> I usually do this in the 'parent' gdb, so I can tell it apart from
> >>> the 'child' gdb:
> >>>
> >>>     (gdb) set prompt (GDB)
> >>>     (GDB)
> >>>
> >>> Now, in the parent GDB, you should be able to set your
> >>> breakpoints in libthread_db, then start the child gdb,
> >>> load the multi-threaded program, and begin debugging it.
> >>>
> >> Does the native gdb use the thread_db as the gdbserver does?
> >
> > Yes.
> >
> >> I was thinking to run gdb --args my-gdbserver <application>
> >> on the target, and connect to the my-gdbserver from the host
> >> just to trigger the thread_db initialization process.
> >> Comments?
> >
> > Oh, so you can actually run gdb on the target?
> Yes, I'm using it after I read your reply. So I'm running
> a working gdb (glibc/sh4 based) to debug a half-working gdbserver
> (uClibc-nptl/sh4 based).
> I was able to set breakpoint on gdbserver itself
> (thread_db_find_new_threads function for example) and to step into
> my thread_db library. But up to now I could not reach the
> td_ta_map_lp2thr function due to some failures in packets transmission
> between gdb remote client and target gdbserver, so the tbread_db
> initializaition is failing before entering in that function.
> I'm still working on it.
>
> I'd like to describe the problem I have in case some one may have an idea:
> The stack is as follows:
>
> td_ta_map_lwp2thr
> iterate_list_thread
> td_ta_thr_iter
> thread_db_find_new_threads
> thread_db_init
>
> The td_ta_map_lwp2thr funtcion reach its end successfully
> (performing any symbol lookup; I also printed out the thread pointer
> and its value is just the same as printed by the pthread_self function
> from the libpthread), but when returns to the caller it produces a
> SIGSEV due to a misaligned access (as reported by dmesg command).
> Any printf after the function's invocation is never executed!
>
> The thread_db code is he same as the glibc, so I cannot understand
> where it's failing. The gdbserver version is 6.4 (built using a
> sh4-linux-gcc configured for uClibc) and it is the same code used to
> build the glibc based version.
>
>
> > Yes, that would make things simpler, and what you
> > suggest above is pretty much what I would recommend.
> >
> >
> >>> When you get ready to try this with gdbserver, you will
> >>> have to start one gdbserver and attach to it with another
> >>> gdbserver.  Then you'll again have two gdbs running on the
> >>> host side, which you can again think of as parent and child,
> >>> only now the parent is debugging the child's gdbserver, not
> >>> the child gdb itself.
> >>>
> >> Well, just before reading your reply, I tried with this:
> >>
> >> target:
> >> gdbserver localhost:999 my-gdbserver localhost:998 <applcation>
> >>
> >> host:
> >> xxx-gdb my-gdbserver (connecting to port 999)
> >>
> >> xxx-gdb application (connecting to port 998)
> >>
> >> With the first gdb session I was able to set a breakpoint into
> >> thread_db, but not able to stepping through the code.
> >>
> >> Is this approach as well as the gdb native session.
> >
> > Hmmm, that's pretty much what I would have tried.
> > What happened when you tried to step?  How did it fail?
> >
>
> Difficult to describe; I'm trying to follow the native approach for now,
> but I'd like trying this again later; I'll report any failure or success
> with more details.
>
> Thanks a lot again
> Carmelo
>

Re: howto debug libthread_db

[Michael Snyder]

On Fri, 2006-09-29 at 11:08 +0200, Carmelo Amoroso wrote:
> Well,
> after a few days of debugging my gdbserver (with uClibc-nptl on sh4),
> I found the bug into uClibc,
> that was causing a stack frame corruption due to a buffer overrun.
> This was the reason for which the td_ta_map_lwp2thr was never
> returning to the caller
> (the stack location where the PR had been saved was corupted).
> 
> It was a very interesting debugging session. I successfully used a
> native gdb (glibc base)on target
> to debug a gdbserver (uClinc-nptl based) I connect to with a gdb
> client from a linux host.
> 
> At the end I did not need to use a double remote gdb session.
> 
> Thanks to Michael for his very usefull tips.


So, does that mean that there was no actual problem with gdb, 
with respect to stepping in libthread_db?  This works now?

Thanks,
Michael


> 
> On 9/27/06, Carmelo Amoroso <carmelo73@gmail.com> wrote:
> > Michael Snyder wrote:
> > > On Wed, 2006-09-27 at 08:23 +0200, Carmelo Amoroso wrote:
> > >> Michael Snyder wrote:
> > >>> On Tue, 2006-09-26 at 16:06 +0200, Carmelo Amoroso wrote:
> > >>>> Hi All,
> > >>>> I'm trying to debug a simple multithread application on a remote target,
> > >>>> and I like to debug the libthread_db itself... is it possible to do it?
> > >>>> is it possible to set some breakpoints and stepping through?
> > >>>> Is there anyone already played with it?
> > >>>>
> > >>>> I tried to use symbol-file to gdb client, and I successfully added a
> > >>>> breakpoint, but I cannot reach it after I connected to the target and
> > >>>> issued 'continue'.
> > >>>>
> > >>>> Any help will be appreciated
> > >>> Interesting.  I haven't done it (or not much anyway).
> > >>>
> > >>> The first thing you must bear in mind is that libthread_db is not
> > >>> part of the target application -- it is part of gdb (or in the
> > >>> remote debugging context, part of gdbserver).  So to debug it,
> > >>> you must debug gdb, or in your case gdbserver.
> > >>>
> > >> Yes. it's what I want to do
> > >
> > >>> It might be easier if you try this out first using a native gdb,
> > >>> debugging itself.
> > >>>
> > >>> When you debug gdb with gdb, it is easier if you change the prompt.
> > >>> I usually do this in the 'parent' gdb, so I can tell it apart from
> > >>> the 'child' gdb:
> > >>>
> > >>>     (gdb) set prompt (GDB)
> > >>>     (GDB)
> > >>>
> > >>> Now, in the parent GDB, you should be able to set your
> > >>> breakpoints in libthread_db, then start the child gdb,
> > >>> load the multi-threaded program, and begin debugging it.
> > >>>
> > >> Does the native gdb use the thread_db as the gdbserver does?
> > >
> > > Yes.
> > >
> > >> I was thinking to run gdb --args my-gdbserver <application>
> > >> on the target, and connect to the my-gdbserver from the host
> > >> just to trigger the thread_db initialization process.
> > >> Comments?
> > >
> > > Oh, so you can actually run gdb on the target?
> > Yes, I'm using it after I read your reply. So I'm running
> > a working gdb (glibc/sh4 based) to debug a half-working gdbserver
> > (uClibc-nptl/sh4 based).
> > I was able to set breakpoint on gdbserver itself
> > (thread_db_find_new_threads function for example) and to step into
> > my thread_db library. But up to now I could not reach the
> > td_ta_map_lp2thr function due to some failures in packets transmission
> > between gdb remote client and target gdbserver, so the tbread_db
> > initializaition is failing before entering in that function.
> > I'm still working on it.
> >
> > I'd like to describe the problem I have in case some one may have an idea:
> > The stack is as follows:
> >
> > td_ta_map_lwp2thr
> > iterate_list_thread
> > td_ta_thr_iter
> > thread_db_find_new_threads
> > thread_db_init
> >
> > The td_ta_map_lwp2thr funtcion reach its end successfully
> > (performing any symbol lookup; I also printed out the thread pointer
> > and its value is just the same as printed by the pthread_self function
> > from the libpthread), but when returns to the caller it produces a
> > SIGSEV due to a misaligned access (as reported by dmesg command).
> > Any printf after the function's invocation is never executed!
> >
> > The thread_db code is he same as the glibc, so I cannot understand
> > where it's failing. The gdbserver version is 6.4 (built using a
> > sh4-linux-gcc configured for uClibc) and it is the same code used to
> > build the glibc based version.
> >
> >
> > > Yes, that would make things simpler, and what you
> > > suggest above is pretty much what I would recommend.
> > >
> > >
> > >>> When you get ready to try this with gdbserver, you will
> > >>> have to start one gdbserver and attach to it with another
> > >>> gdbserver.  Then you'll again have two gdbs running on the
> > >>> host side, which you can again think of as parent and child,
> > >>> only now the parent is debugging the child's gdbserver, not
> > >>> the child gdb itself.
> > >>>
> > >> Well, just before reading your reply, I tried with this:
> > >>
> > >> target:
> > >> gdbserver localhost:999 my-gdbserver localhost:998 <applcation>
> > >>
> > >> host:
> > >> xxx-gdb my-gdbserver (connecting to port 999)
> > >>
> > >> xxx-gdb application (connecting to port 998)
> > >>
> > >> With the first gdb session I was able to set a breakpoint into
> > >> thread_db, but not able to stepping through the code.
> > >>
> > >> Is this approach as well as the gdb native session.
> > >
> > > Hmmm, that's pretty much what I would have tried.
> > > What happened when you tried to step?  How did it fail?
> > >
> >
> > Difficult to describe; I'm trying to follow the native approach for now,
> > but I'd like trying this again later; I'll report any failure or success
> > with more details.
> >
> > Thanks a lot again
> > Carmelo
> >

Re: howto debug libthread_db

[Carmelo Amoroso]

Michael Snyder wrote:
> On Fri, 2006-09-29 at 11:08 +0200, Carmelo Amoroso wrote:
>> Well,
>> after a few days of debugging my gdbserver (with uClibc-nptl on sh4),
>> I found the bug into uClibc,
>> that was causing a stack frame corruption due to a buffer overrun.
>> This was the reason for which the td_ta_map_lwp2thr was never
>> returning to the caller
>> (the stack location where the PR had been saved was corupted).
>>
>> It was a very interesting debugging session. I successfully used a
>> native gdb (glibc base)on target
>> to debug a gdbserver (uClinc-nptl based) I connect to with a gdb
>> client from a linux host.
>>
>> At the end I did not need to use a double remote gdb session.
>>
>> Thanks to Michael for his very usefull tips.
> 
> 
> So, does that mean that there was no actual problem with gdb, 
> with respect to stepping in libthread_db?  
Yes, it was my thread_db that was failing killing the gdbserver
This works now?
Yes, it works perfectly, even with multithreadrd applications.

Carmelo
> 
> Thanks,
> Michael
> 
> 
>> On 9/27/06, Carmelo Amoroso <carmelo73@gmail.com> wrote:
>>> Michael Snyder wrote:
>>>> On Wed, 2006-09-27 at 08:23 +0200, Carmelo Amoroso wrote:
>>>>> Michael Snyder wrote:
>>>>>> On Tue, 2006-09-26 at 16:06 +0200, Carmelo Amoroso wrote:
>>>>>>> Hi All,
>>>>>>> I'm trying to debug a simple multithread application on a remote target,
>>>>>>> and I like to debug the libthread_db itself... is it possible to do it?
>>>>>>> is it possible to set some breakpoints and stepping through?
>>>>>>> Is there anyone already played with it?
>>>>>>>
>>>>>>> I tried to use symbol-file to gdb client, and I successfully added a
>>>>>>> breakpoint, but I cannot reach it after I connected to the target and
>>>>>>> issued 'continue'.
>>>>>>>
>>>>>>> Any help will be appreciated
>>>>>> Interesting.  I haven't done it (or not much anyway).
>>>>>>
>>>>>> The first thing you must bear in mind is that libthread_db is not
>>>>>> part of the target application -- it is part of gdb (or in the
>>>>>> remote debugging context, part of gdbserver).  So to debug it,
>>>>>> you must debug gdb, or in your case gdbserver.
>>>>>>
>>>>> Yes. it's what I want to do
>>>>>> It might be easier if you try this out first using a native gdb,
>>>>>> debugging itself.
>>>>>>
>>>>>> When you debug gdb with gdb, it is easier if you change the prompt.
>>>>>> I usually do this in the 'parent' gdb, so I can tell it apart from
>>>>>> the 'child' gdb:
>>>>>>
>>>>>>     (gdb) set prompt (GDB)
>>>>>>     (GDB)
>>>>>>
>>>>>> Now, in the parent GDB, you should be able to set your
>>>>>> breakpoints in libthread_db, then start the child gdb,
>>>>>> load the multi-threaded program, and begin debugging it.
>>>>>>
>>>>> Does the native gdb use the thread_db as the gdbserver does?
>>>> Yes.
>>>>
>>>>> I was thinking to run gdb --args my-gdbserver <application>
>>>>> on the target, and connect to the my-gdbserver from the host
>>>>> just to trigger the thread_db initialization process.
>>>>> Comments?
>>>> Oh, so you can actually run gdb on the target?
>>> Yes, I'm using it after I read your reply. So I'm running
>>> a working gdb (glibc/sh4 based) to debug a half-working gdbserver
>>> (uClibc-nptl/sh4 based).
>>> I was able to set breakpoint on gdbserver itself
>>> (thread_db_find_new_threads function for example) and to step into
>>> my thread_db library. But up to now I could not reach the
>>> td_ta_map_lp2thr function due to some failures in packets transmission
>>> between gdb remote client and target gdbserver, so the tbread_db
>>> initializaition is failing before entering in that function.
>>> I'm still working on it.
>>>
>>> I'd like to describe the problem I have in case some one may have an idea:
>>> The stack is as follows:
>>>
>>> td_ta_map_lwp2thr
>>> iterate_list_thread
>>> td_ta_thr_iter
>>> thread_db_find_new_threads
>>> thread_db_init
>>>
>>> The td_ta_map_lwp2thr funtcion reach its end successfully
>>> (performing any symbol lookup; I also printed out the thread pointer
>>> and its value is just the same as printed by the pthread_self function
>>> from the libpthread), but when returns to the caller it produces a
>>> SIGSEV due to a misaligned access (as reported by dmesg command).
>>> Any printf after the function's invocation is never executed!
>>>
>>> The thread_db code is he same as the glibc, so I cannot understand
>>> where it's failing. The gdbserver version is 6.4 (built using a
>>> sh4-linux-gcc configured for uClibc) and it is the same code used to
>>> build the glibc based version.
>>>
>>>
>>>> Yes, that would make things simpler, and what you
>>>> suggest above is pretty much what I would recommend.
>>>>
>>>>
>>>>>> When you get ready to try this with gdbserver, you will
>>>>>> have to start one gdbserver and attach to it with another
>>>>>> gdbserver.  Then you'll again have two gdbs running on the
>>>>>> host side, which you can again think of as parent and child,
>>>>>> only now the parent is debugging the child's gdbserver, not
>>>>>> the child gdb itself.
>>>>>>
>>>>> Well, just before reading your reply, I tried with this:
>>>>>
>>>>> target:
>>>>> gdbserver localhost:999 my-gdbserver localhost:998 <applcation>
>>>>>
>>>>> host:
>>>>> xxx-gdb my-gdbserver (connecting to port 999)
>>>>>
>>>>> xxx-gdb application (connecting to port 998)
>>>>>
>>>>> With the first gdb session I was able to set a breakpoint into
>>>>> thread_db, but not able to stepping through the code.
>>>>>
>>>>> Is this approach as well as the gdb native session.
>>>> Hmmm, that's pretty much what I would have tried.
>>>> What happened when you tried to step?  How did it fail?
>>>>
>>> Difficult to describe; I'm trying to follow the native approach for now,
>>> but I'd like trying this again later; I'll report any failure or success
>>> with more details.
>>>
>>> Thanks a lot again
>>> Carmelo
>>>
> 

Re: howto debug libthread_db

[Michael Snyder]

On Sat, 2006-09-30 at 18:48 +0200, Carmelo Amoroso wrote:
> Michael Snyder wrote:
> > 
> > 
> > So, does that mean that there was no actual problem with gdb, 
> > with respect to stepping in libthread_db?  
> Yes, it was my thread_db that was failing killing the gdbserver
> This works now?
> Yes, it works perfectly, even with multithreaded applications.

Excellent.