Debugging pthread in ARM.

Debugging pthread in ARM.

[gomrioje]

I am not able to see the point of Daniel (mea culpa).
I have readen documentation as recommended by Daniel and William.
Maybe trying again...

The application uses: pthread_create.
For that I am using as library (-lpthread):
libpthread.so.0->libpthread-0.10.so

For debugging purposes, I am using gdbserver in the target (AT91RM9200).
gdbserver must use library: libthread_db.so.1->libthread_db-1.0.so

Now I describe the problem:

1. When I set a breakpoint inside a thread code, I get a SIGTRAP.
2. After creating several threads (POSIX: pthread_create),
command: info threads, result in: 1 main...............
3. Sometimes after running a while, I get a message in the gdbserver
console:
Gdb: error initalizing thread_db library: version mismatch between
libthread_db and libpthread

Both of the libraries come from the same Debian distribution, for ARM
architecture. 

Daniel mentions problem pointing to libraries. For such situation,
the config is:
- For the gdbserver:
export LD_LIBRARY_PATH=/lib:/usr/lib/:/app/arm/debug/:$LD_LIBRARY_PATH

- For the gdb (host):
set solib-absolute-prefix /dev/null/
set
solib-search-path /app/arm/debug/:/app/arm/bitbaketmp/staging/arm-linux/lib/

I could miss something. Some ideas ?

Regards,

> Thanks for the link.
> But I guess my problem is not described there.
> I have another message:
> Gdb: error initalizing thread_db library: version mismatch between libthread_db and libpthread
> 
> I have read in the mailing list that both should have the same glibc.
> (what do you mean there?).
> 
> In another hand, I am already pointing solib-absolute-prefix to the directory libraries.

If you're doing that, and getting the above error, then you are not
pointing to the correct libraries.

-- 
Jesus Gomez Rio
R&D Technician
Design & Manufacturing Division
gomez.j@ikusi.com
www.ikusi.es
---

Re: Debugging pthread in ARM.

[Daniel Jacobowitz]

On Tue, May 15, 2007 at 02:49:04PM +0200, gomrioje wrote:
> - For the gdb (host):
> set solib-absolute-prefix /dev/null/
> set
> solib-search-path /app/arm/debug/:/app/arm/bitbaketmp/staging/arm-linux/lib/

Don't do that.  Whatever told you to do it was wrong.

You need to set solib-absolute-prefix to point at something that looks
like your target's file system, that is, you should be able to add
/lib/ld-linux.so.2 (or however your platform spells it) and find the
dynamic linker at that path on the host.

-- 
Daniel Jacobowitz
CodeSourcery

Re: Debugging pthread in ARM.

[gomrioje]

Thanks Daniel, I point solib-absolute, right now my libraries (info
sharedlibrary) are like that:


From        To          Syms Read   Shared Object Library
0x400284d8  0x400859a4
No          /app/arm/bitbaketmp/staging/arm-linux/lib/libsqlite.so.0
0x4009b748  0x4009bdf8
No          /app/arm/sc0504/Software/src/libarm9hardpin/debug/src/library/libarm9hardpin.so.0
0x400a5d60  0x400aa2f0
No          /app/arm/bitbaketmp/staging/arm-linux/lib/librt.so.1
0x400c1f70  0x400cbff4
No          /app/arm/bitbaketmp/staging/arm-linux/lib/libpthread.so.0
0x40153278  0x401dc63c
No          /app/arm/bitbaketmp/cross/arm-linux/lib/libstdc++.so.6
0x401ef390  0x40259ae8
No          /app/arm/bitbaketmp/staging/arm-linux/lib/libm.so.6
0x4029a9f8  0x402a0f18
No          /app/arm/bitbaketmp/cross/arm-linux/lib/libgcc_s.so.1
0x402bebb0  0x40393564
No          /app/arm/bitbaketmp/staging/arm-linux/lib/libc.so.6
0x40000740  0x4001491c
No          /app/arm/bitbaketmp/staging/arm-linux/lib/ld-linux.so.2

(for other people reading:
http://sourceware.org/ml/gdb-patches/2007-01/msg00198.html)

Nevertheless, after that I have the same problem.
The response to: info threads is: 1 main....

Something that looks bizarre to me is that gdb converts in the console,
"info threads" to "info thread".
I have checked the threads are running. They have different PIDs, could
be a problem for gdb? 

Regards,





 




On Tue, 2007-05-15 at 08:58 -0400, Daniel Jacobowitz wrote:
> On Tue, May 15, 2007 at 02:49:04PM +0200, gomrioje wrote:
> > - For the gdb (host):
> > set solib-absolute-prefix /dev/null/
> > set
> > solib-search-path /app/arm/debug/:/app/arm/bitbaketmp/staging/arm-linux/lib/
> 
> Don't do that.  Whatever told you to do it was wrong.
> 
> You need to set solib-absolute-prefix to point at something that looks
> like your target's file system, that is, you should be able to add
> /lib/ld-linux.so.2 (or however your platform spells it) and find the
> dynamic linker at that path on the host.
> 
-- 
Jesus Gomez Rio
R&D Technician
Design & Manufacturing Division
gomez.j@ikusi.com
www.ikusi.es
---

Re: Debugging pthread in ARM.

[Daniel Jacobowitz]

On Tue, May 15, 2007 at 07:13:58PM +0200, gomrioje wrote:
> Thanks Daniel, I point solib-absolute, right now my libraries (info
> sharedlibrary) are like that:
> 
> 
> >From        To          Syms Read   Shared Object Library
> 0x400284d8  0x400859a4
> No          /app/arm/bitbaketmp/staging/arm-linux/lib/libsqlite.so.0

Something is still wrong, since if GDB had found the libraries it
would have read in symbols automatically.  Are you getting any
warnings?  Are you sure they are exactly the same versions?

ld-linux.so.2 should be under your solib-absolute-prefix.


-- 
Daniel Jacobowitz
CodeSourcery

RE: Debugging pthread in ARM.

[Williams, Allen]

I am getting multiple copies of these emails.  Am I the only one on the
list this is happening to?  This is about the sixth copy of this one
that I've received, and I've received multiple copies of others, too.


-----Original Message-----
From: gdb-owner@sourceware.org [mailto:gdb-owner@sourceware.org] On
Behalf Of gomrioje
Sent: Tuesday, May 15, 2007 8:49 AM
To: gdb@sourceware.org
Subject: Debugging pthread in ARM.

I am not able to see the point of Daniel (mea culpa).
I have readen documentation as recommended by Daniel and William.
Maybe trying again...

The application uses: pthread_create.
For that I am using as library (-lpthread):
libpthread.so.0->libpthread-0.10.so

For debugging purposes, I am using gdbserver in the target (AT91RM9200).
gdbserver must use library: libthread_db.so.1->libthread_db-1.0.so

Now I describe the problem:

1. When I set a breakpoint inside a thread code, I get a SIGTRAP.
2. After creating several threads (POSIX: pthread_create),
command: info threads, result in: 1 main...............
3. Sometimes after running a while, I get a message in the gdbserver
console:
Gdb: error initalizing thread_db library: version mismatch between
libthread_db and libpthread

Both of the libraries come from the same Debian distribution, for ARM
architecture. 

Daniel mentions problem pointing to libraries. For such situation,
the config is:
- For the gdbserver:
export LD_LIBRARY_PATH=/lib:/usr/lib/:/app/arm/debug/:$LD_LIBRARY_PATH

- For the gdb (host):
set solib-absolute-prefix /dev/null/
set
solib-search-path
/app/arm/debug/:/app/arm/bitbaketmp/staging/arm-linux/lib/

I could miss something. Some ideas ?

Regards,

> Thanks for the link.
> But I guess my problem is not described there.
> I have another message:
> Gdb: error initalizing thread_db library: version mismatch between
libthread_db and libpthread
> 
> I have read in the mailing list that both should have the same glibc.
> (what do you mean there?).
> 
> In another hand, I am already pointing solib-absolute-prefix to the
directory libraries.

If you're doing that, and getting the above error, then you are not
pointing to the correct libraries.

-- 
Jesus Gomez Rio
R&D Technician
Design & Manufacturing Division
gomez.j@ikusi.com
www.ikusi.es
---

Re: Debugging pthread in ARM.

[gomrioje]

Running the aplication in the PC, I get with command backtrace:
(gdb) [Thread debugging using libthread_db enabled]

I don't get this message in the cross-debugging with arm.

Both the gdb and the gdbserver are:

GNU gdbserver 6.6-debian
Copyright (C) 2006 Free Software Foundation, Inc.
gdbserver is free software, covered by the GNU General Public License.
This gdbserver was configured as "arm-linux-gnu"

GNU gdb 6.4.90
Copyright (C) 2006 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you
are
welcome to change it and/or distribute copies of it under certain
conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for
details.
This GDB was configured as "--host=i686-linux --target=arm-linux".


I guess the libraries for the ARM have no symbols charged. Do I need
them to debug my application?

Regards,



On Tue, 2007-05-15 at 13:21 -0400, Daniel Jacobowitz wrote:
> On Tue, May 15, 2007 at 07:13:58PM +0200, gomrioje wrote:
> > Thanks Daniel, I point solib-absolute, right now my libraries (info
> > sharedlibrary) are like that:
> > 
> > 
> > >From        To          Syms Read   Shared Object Library
> > 0x400284d8  0x400859a4
> > No          /app/arm/bitbaketmp/staging/arm-linux/lib/libsqlite.so.0
> 
> Something is still wrong, since if GDB had found the libraries it
> would have read in symbols automatically.  Are you getting any
> warnings?  Are you sure they are exactly the same versions?
> 
> ld-linux.so.2 should be under your solib-absolute-prefix.
> 
> 
-- 
Jesus Gomez Rio
R&D Technician
Design & Manufacturing Division
gomez.j@ikusi.com
www.ikusi.es
---

Re: Debugging pthread in ARM.

[Daniel Jacobowitz]

On Wed, May 16, 2007 at 05:27:54PM +0200, gomrioje wrote:
> 
> Running the aplication in the PC, I get with command backtrace:
> (gdb) [Thread debugging using libthread_db enabled]

That's normal.  Only native GDB prints this message.

> I guess the libraries for the ARM have no symbols charged. Do I need
> them to debug my application?

I believe you need symbols for ld.so and libpthread.so.

-- 
Daniel Jacobowitz
CodeSourcery

Re: Debugging pthread in ARM.

[gomrioje]

Just typing sharedlibrary I get all my libraries with symbols:
From        To          Syms Read   Shared Object Library
0x40024f70  0x4002eff4
Yes         /app/arm/bitbaketmp/staging/arm-linux/lib/libpthread.so.0
0x4007cd60  0x400812f0
Yes         /app/arm/bitbaketmp/staging/arm-linux/lib/librt.so.1
0x400d0278  0x4015963c
Yes         /app/arm/bitbaketmp/cross/arm-linux/lib/libstdc++.so.6
0x4016c390  0x401d6ae8
Yes         /app/arm/bitbaketmp/staging/arm-linux/lib/libm.so.6
0x402179f8  0x4021df18
Yes         /app/arm/bitbaketmp/cross/arm-linux/lib/libgcc_s.so.1
0x4023bbb0  0x40310564
Yes         /app/arm/bitbaketmp/staging/arm-linux/lib/libc.so.6
0x40000740  0x4001491c
Yes         /app/arm/bitbaketmp/staging/arm-linux/lib/ld-linux.so.2


Nevertheless GDB is receiving a SIGNALKILL when I try to set a
breakpoint inside a thread.

You ask me for warnings, there are 2 things:

1. At Host side (gdb console):
(gdb) source prepare_gdb.gdb
0x40000740 in ?? ()

prepare_gdb.gdb is a script:

set $connected=0
shell sleep 1       
target remote 192.168.1.55:1234
set $connected=1
set solib-absolute-prefix /app/arm/bitbaketmp/staging/arm-linux/lib/
set
solib-search-path  /app/arm/bitbaketmp/staging/arm-linux/lib/:/app/arm/sc0504/Software/src/libarm9hardpin/debug/src/library/:/app/arm/bitbaketmp/cross/arm-linux/lib/:/app/arm/sc0504/Software/src/3rdParty/nana-2.5/arm/usr/lib/
continue 

2. At Target side (gdbserver)
gdb: error initializing thread_db library: version mismatch between
libthread_db and libpthread

Regards





> > 
> > Running the aplication in the PC, I get with command backtrace:
> > (gdb) [Thread debugging using libthread_db enabled]
> 
> That's normal.  Only native GDB prints this message.
> 
> > I guess the libraries for the ARM have no symbols charged. Do I need
> > them to debug my application?
> 
> I believe you need symbols for ld.so and libpthread.so.
> 
-- 
Jesus Gomez Rio

Re: Debugging pthread in ARM.

[Daniel Jacobowitz]

On Wed, May 16, 2007 at 07:33:53PM +0200, gomrioje wrote:
> set solib-absolute-prefix /app/arm/bitbaketmp/staging/arm-linux/lib/

I'm not sure how many times I can try to explain this... this is the
wrong path.  You should be able to append "/lib/ld-linux.so.2" to that
path and find your dynamic loader.  Try removing "/lib/" from the end
of your setting.

-- 
Daniel Jacobowitz
CodeSourcery

Re: Debugging pthread in ARM.

[gomrioje]

I have tried, and I get the same result.

Could it be link to:
http://sources.redhat.com/ml/gdb/2003-06/msg00535.html ?

It was about single-step in gdbserver for pthread_create?


2003-06-28  Daniel Jacobowitz  <drow@mvista.com>

	* linux-arm-low.c (arm_reinsert_addr): New function.
	(the_low_target): Add arm_reinsert_addr.

Index: linux-arm-low.c
===================================================================
RCS file: /cvs/src/src/gdb/gdbserver/linux-arm-low.c,v
retrieving revision 1.5
diff -u -p -r1.5 linux-arm-low.c
--- linux-arm-low.c	18 Nov 2002 00:37:50 -0000	1.5
+++ linux-arm-low.c	28 Jun 2003 23:53:49 -0000
@@ -80,6 +80,17 @@ arm_breakpoint_at (CORE_ADDR where)
   return 0;
 }
 
+/* We only place breakpoints in empty marker functions, and thread locking
+   is outside of the function.  So rather than importing software single-step,
+   we can just run until exit.  */
+static CORE_ADDR
+arm_reinsert_addr ()
+{
+  unsigned long pc;
+  collect_register_by_name ("lr", &pc);
+  return pc;
+}
+
 struct linux_target_ops the_low_target = {
   arm_num_regs,
   arm_regmap,
@@ -89,7 +100,7 @@ struct linux_target_ops the_low_target =
   arm_set_pc,
   (const char *) &arm_breakpoint,
   arm_breakpoint_len,
-  NULL,
+  arm_reinsert_addr,
   0,
   arm_breakpoint_at,
 };


Regards,

> On Wed, May 16, 2007 at 07:33:53PM +0200, gomrioje wrote:
> > set solib-absolute-prefix /app/arm/bitbaketmp/staging/arm-linux/lib/
> 
> I'm not sure how many times I can try to explain this... this is the
> wrong path.  You should be able to append "/lib/ld-linux.so.2" to that
> path and find your dynamic loader.  Try removing "/lib/" from the end
> of your setting.
> 
-

Re: Debugging pthread in ARM.

[Daniel Jacobowitz]

On Wed, May 16, 2007 at 07:47:00PM +0200, gomrioje wrote:
> I have tried, and I get the same result.
> 
> Could it be link to:
> http://sources.redhat.com/ml/gdb/2003-06/msg00535.html ?

No, that has nothing to do with it.

Try looking at the libpthread.so GDB opens on your host with
"strings".  Search for its version number in the output.  Then
check the version number on your target by running /lib/libc.so.6 as a
program and looking at its output.

-- 
Daniel Jacobowitz
CodeSourcery

Re: Debugging pthread in ARM.

[gomrioje]

Ok, then:
Target side:
# /lib/libc.so.6
GNU C Library development release version 2.3.90, by Roland McGrath et
al.
Compiled by GNU CC version 3.4.4 (release) (CodeSourcery ARM Q1cvs
2005).
Compiled on a Linux 2.6.11 system on 2005-11-25.


Host size:

$ strings /app/arm/bitbaketmp/staging/arm-linux/lib/libpthread-0.10.so
...
libc.so.6
ld-linux.so.2
libpthread.so.0
GLIBC_2.0
GLIBC_2.1
GLIBC_2.1.1
GLIBC_2.1.2
GLIBC_2.2
GLIBC_2.2.3
GLIBC_2.2.6
GLIBC_2.3.2
GLIBC_PRIVATE
GLIBC_2.1.3
GLIBC_2.3
/proc/self/maps
%x-%x
2.3.90
/proc/sys/kernel/version
__fcntl_nocancel
do_pread
do_pwrite
/lib/ld-linux.so.2


Regards,


> > I have tried, and I get the same result.
> > 
> > Could it be link to:
> > http://sources.redhat.com/ml/gdb/2003-06/msg00535.html ?
> 
> No, that has nothing to do with it.
> 
> Try looking at the libpthread.so GDB opens on your host with
> "strings".  Search for its version number in the output.  Then
> check the version number on your target by running /lib/libc.so.6 as a
> program and looking at its output.
> 

Re: Debugging pthread in ARM.

[Daniel Jacobowitz]

I'm out of ideas.  I recommend you debug gdbserver to see why it
thinks there is a mismatch.

-- 
Daniel Jacobowitz
CodeSourcery

Re: Debugging pthread in ARM.

[gomrioje]

Not able to debug gdbserver.
Maybe that is the problem...
I get the gdbserver from Debian distribution for ARM architecture.
file gdbserver
gdbserver: ELF 32-bit LSB executable, ARM, version 1 (ARM), for
GNU/Linux 2.4.1, dynamically linked (uses shared libs), for GNU/Linux
2.4.1, stripped

It makes sense to you?

On Wed, 2007-05-16 at 14:10 -0400, Daniel Jacobowitz wrote:
> I'm out of ideas.  I recommend you debug gdbserver to see why it
> thinks there is a mismatch.
> 

Re: Debugging pthread in ARM.

[Baurzhan Ismagulov]

Hello Gómez,

On Thu, May 17, 2007 at 10:11:41AM +0200, gomrioje wrote:
> Not able to debug gdbserver.

What does "not able" mean? Are you getting an error message?

With kind regards,
-- 
Baurzhan Ismagulov
http://www.kz-easy.com/

Re: Debugging pthread in ARM.

[gomrioje]

Thanks all, specially Daniel.
Changing the library libpthread to the right version, 
let me to set breakpoints in the threads.

But still something is not working:
In a simple application, I get an infinite backtrace in the gdb console,
when it stops in a breakpoint inside a thread.
(always as:
#149 0x400c3f78 in pthread_start_thread ()
from /app/arm/bitbaketmp/staging/arm-linux/lib/libpthread.so.0
#150 0x400c3f78 in pthread_start_thread ()
from /app/arm/bitbaketmp/staging/arm-linux/lib/libpthread.so.0
#151 0x400c3f78 in pthread_start_thread ()
from /app/arm/bitbaketmp/staging/arm-linux/lib/libpthread.so.0
..........)




On Wed, 2007-05-16 at 14:10 -0400, Daniel Jacobowitz wrote:
> I'm out of ideas.  I recommend you debug gdbserver to see why it
> thinks there is a mismatch.
> 

Re: Debugging pthread in ARM.

[Daniel Jacobowitz]

On Thu, May 17, 2007 at 02:34:12PM +0200, gomrioje wrote:
> Thanks all, specially Daniel.
> Changing the library libpthread to the right version, 
> let me to set breakpoints in the threads.
> 
> But still something is not working:
> In a simple application, I get an infinite backtrace in the gdb console,
> when it stops in a breakpoint inside a thread.
> (always as:
> #149 0x400c3f78 in pthread_start_thread ()
> from /app/arm/bitbaketmp/staging/arm-linux/lib/libpthread.so.0

I believe a newer GDB will do something more sensible with this.

-- 
Daniel Jacobowitz
CodeSourcery