Restarting gdbserver

Restarting gdbserver

[Leon Pollak]

Hello, all.

I searched Google and mail list for hours - very strange, but found 
nothing...:-(

I debug remotely gdb <-> gdbserver - totally standard situation...:-)
The target's root FS is NFS on my PC.

To start one debug session I go to the target (via telnet) and start the 
gdbserver manually, then in my IDE start the gdb session. When a change is 
required, I exit the gdb, this kills the gdbserver, make a change, recompile, 
go to telnet, restart gdbserver, connect in IDE, etc.
Very annoying....

I tried to automate the gdbserver restart process by running script which 
loops it, but this is not very reliable, as this means that I am allowed to 
exit the gdb only AFTER I have recompiled the application. Otherwise, 
gdbserver takes the old executable.

All this seems to be VERY standard situation which should be solved for years, 
but how...:-(

Sorry to ask something so primitive.

Thanks a lot ahead.
-- 
Leon

Re: Restarting gdbserver

[Jan Kratochvil]

On Thu, 20 Jan 2011 10:29:49 +0100, Leon Pollak wrote:
> gdbserver manually, then in my IDE start the gdb session. When a change is 
> required, I exit the gdb, this kills the gdbserver, make a change, recompile, 
> go to telnet, restart gdbserver, connect in IDE, etc.

Either some:
	while true;do gdbserver COMM PROG; done

or also:
	gdbserver --multi COMM
	using `target extended-remote ...' instead.


Regards,
Jan

Re: Restarting gdbserver

[Leon Pollak]

On Thursday January 20 2011, Jan Kratochvil wrote:
> or also:
> 	gdbserver --multi COMM
> 	using `target extended-remote ...' instead.
WOW!!!!!!!!
How could I not find this!!??
1001 of thanks!
Works fine. I start thegdb with:

set sysroot /opt/FS/
set remote exec-file /common/csu
target extended-remote 192.168.50.4:44000
---
The only one problem left for me is that when stopped/broken, the stack frame 
is not presented as it usually is:

(gdb) frame
#0  0x400314bc in pthread_cond_wait@@GLIBC_2.4 () from 
/opt/FS/lib/libpthread.so.0

And nothing else...:-(

Thousands of thanks ahead for the help...
-- 
Leon

Re: Restarting gdbserver

[Jan Kratochvil]

On Thu, 20 Jan 2011 15:11:37 +0100, Leon Pollak wrote:
> set sysroot /opt/FS/
> set remote exec-file /common/csu
> target extended-remote 192.168.50.4:44000
> ---
> The only one problem left for me is that when stopped/broken, the stack frame 
> is not presented as it usually is:
> 
> (gdb) frame
> #0  0x400314bc in pthread_cond_wait@@GLIBC_2.4 () from /opt/FS/lib/libpthread.so.0
> 
> And nothing else...:-(

Have you tried to play with `set debug-file-directory'?  Which way is the
debug info provided for /opt/FS?


Regards,
Jan

Re: Restarting gdbserver

[Leon Pollak]

> Have you tried to play with `set debug-file-directory'?  
> Which way is the debug info provided for /opt/FS?

Sorry, I probably do not know how to answer...:-)

I read the docs about `set debug-file-directory' - it says that this is 
applicable in the case of the SEPARATE debug info, but I suppose that in my 
case it is inside the ELF module, no?

Regarding the second question - I feel that the problem is here, but where 
exactly?

I have the target's root FS on my PC in /opt/FS directory mounted via NFS. 
The development is done in another separate place - /Projects/CSU/...
The compiled&linked executable module is hard-linked to the file in the 
/opt/FS/common/csu and I start the gdb saying:
set sysroot /opt/FS/
set remote exec-file /common/csu

Is this insufficient? 
I can put break points and step through the code. The problem seems to be just 
with frame/backtrace only.

Thank you very much for your help.
-- 
Leon

Re: Restarting gdbserver

[Jan Kratochvil]

On Thu, 20 Jan 2011 16:00:07 +0100, Leon Pollak wrote:
> I read the docs about `set debug-file-directory' - it says that this is 
> applicable in the case of the SEPARATE debug info, but I suppose that in my 
> case it is inside the ELF module, no?

According to your environment description I agree you do not need
`set debug-file-directory'.


> I have the target's root FS on my PC in /opt/FS directory mounted via NFS. 
> The development is done in another separate place - /Projects/CSU/...
> The compiled&linked executable module is hard-linked to the file in the 
> /opt/FS/common/csu and I start the gdb saying:
> set sysroot /opt/FS/
> set remote exec-file /common/csu
> 
> Is this insufficient? 
> I can put break points and step through the code. The problem seems to be just 
> with frame/backtrace only.

`set remote exec-file' only instructs remote gdbserver in --multi mode.
It does not provide symbols to local GDB in any way.

Tested two cases, run on machine A, with remote host host1s which is also
mounted at /host1:

./gdb -nx -ex 'set sysroot /host1' -ex 'set debug-file-directory /host1/usr/lib/debug' -ex 'file /host1/home/jkratoch/t/1' -ex 'target remote host1s:1234' -ex 'b main' -ex c
./gdbserver :1234 /home/jkratoch/t/1

or:

./gdb -nx -ex 'file /host1/home/jkratoch/t/1' -ex 'target extended-remote host1s:1234' -ex 'set sysroot /host1' -ex 'set remote exec-file /home/jkratoch/t/1' -ex start
./gdbserver --multi :1234 

The `file' command provides the symbols for local GDB.


Regards,
Jan

Re: Restarting gdbserver

[Leon Pollak]

Thank you, Jan, for your efforts.
Although I did not understand exactly what your meant by your last big text 
fragment, I guess you meant that I need to add the 'file' command.
I did this, but this did not help, no change.
Can you be so kind to give me some hint what may be the cause of the problem?
Thank you again.


On Thursday January 20 2011, Jan Kratochvil wrote:
> On Thu, 20 Jan 2011 16:00:07 +0100, Leon Pollak wrote:
> > I read the docs about `set debug-file-directory' - it says that this is
> > applicable in the case of the SEPARATE debug info, but I suppose that in
> > my case it is inside the ELF module, no?
> 
> According to your environment description I agree you do not need
> `set debug-file-directory'.
> 
> > I have the target's root FS on my PC in /opt/FS directory mounted via
> > NFS. The development is done in another separate place -
> > /Projects/CSU/... The compiled&linked executable module is hard-linked
> > to the file in the /opt/FS/common/csu and I start the gdb saying:
> > set sysroot /opt/FS/
> > set remote exec-file /common/csu
> > 
> > Is this insufficient?
> > I can put break points and step through the code. The problem seems to be
> > just with frame/backtrace only.
> 
> `set remote exec-file' only instructs remote gdbserver in --multi mode.
> It does not provide symbols to local GDB in any way.
> 
> Tested two cases, run on machine A, with remote host host1s which is also
> mounted at /host1:
> 
> ./gdb -nx -ex 'set sysroot /host1' -ex 'set debug-file-directory
> /host1/usr/lib/debug' -ex 'file /host1/home/jkratoch/t/1' -ex 'target
> remote host1s:1234' -ex 'b main' -ex c ./gdbserver :1234
> /home/jkratoch/t/1
> 
> or:
> 
> ./gdb -nx -ex 'file /host1/home/jkratoch/t/1' -ex 'target extended-remote
> host1s:1234' -ex 'set sysroot /host1' -ex 'set remote exec-file
> /home/jkratoch/t/1' -ex start ./gdbserver --multi :1234
> 
> The `file' command provides the symbols for local GDB.
> 
> 
> Regards,
> Jan


-- 
Leon

Re: Restarting gdbserver

[Jan Kratochvil]

On Fri, 21 Jan 2011 23:52:00 +0100, Leon Pollak wrote:
> Although I did not understand exactly what your meant by your last big text
> fragment,

There were two examples of gdb-gdbserver commands to make reproducible remote
debugging with working symbols resolving.

Your bugreport does not provide a complete reproducer of your problem so
I could not say anything specific.  Therefore providing a complete reproducer
of my working sample command(s).  If you replace `host1' by your hostname,
`/host1' by `/opt/FS/' and `/home/jkratoch/t/1' by `/common/csu' the commands
of mine should be reproducible on your machine(s). You should be then be able
to find out by trial-and-error which specific different will make your setup
working.


> I guess you meant that I need to add the 'file' command.

In general yes.


> I did this, but this did not help, no change.
> Can you be so kind to give me some hint what may be the cause of the problem?

You have not provided an exact reproducer of the problem so I cannot say
anything more.


Thanks,
jan

Re: Restarting gdbserver

[Leon Pollak]

Thank you, Jan.

I hope I understood and did what you said:

bash-4.1$ ./gdb -nx -ex file /opt/FS/common/csu -ex target extended-remote 
192.168.50.4:44000 -ex set sysroot /opt/FS -ex set remote exec-file 
/common/csu -ex start
GNU gdb (GDB) 7.2
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "--host=x86_64-unknown-linux-gnu --target=armv5tl-
montavista-linux-gnueabi".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Reading symbols from /opt/FS/common/csu...done.
Remote debugging using 192.168.50.4:44000
Temporary breakpoint 1 at 0x20374: file /Projects/CSU/src/main.cpp, line 16.
Starting program: /opt/FS/common/csu 

Temporary breakpoint 1, main () at /Projects/CSU/src/main.cpp:16
16              MsgLog("OFP Version: %s dated %s", OFP_VERSION, 
__TIMESTAMP__);
(gdb) c
Continuing.
^C
Program received signal SIGINT, Interrupt.
0x400314bc in pthread_cond_wait@@GLIBC_2.4 () from /opt/FS/lib/libpthread.so.0
(gdb) bt
#0  0x400314bc in pthread_cond_wait@@GLIBC_2.4 () from 
/opt/FS/lib/libpthread.so.0
#1  0x00000000 in ?? ()
(gdb) 
----------------------------------------------------------------------------
God knows why it decided to stop at main... I did not set break points here 
and '.gdbinit' file was deleted for the test.

Is it what you meant?

Thanks again!
-- 
Leon

Re: Restarting gdbserver

[Jan Kratochvil]

On Sun, 23 Jan 2011 17:43:44 +0100, Leon Pollak wrote:
> Thank you, Jan.
> 
> I hope I understood and did what you said:
> 
> bash-4.1$ ./gdb -nx -ex file /opt/FS/common/csu -ex target extended-remote 
> 192.168.50.4:44000 -ex set sysroot /opt/FS -ex set remote exec-file 
> /common/csu -ex start
> GNU gdb (GDB) 7.2
> Copyright (C) 2010 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
> and "show warranty" for details.
> This GDB was configured as "--host=x86_64-unknown-linux-gnu --target=armv5tl-
> montavista-linux-gnueabi".
> For bug reporting instructions, please see:
> <http://www.gnu.org/software/gdb/bugs/>.
> Reading symbols from /opt/FS/common/csu...done.
> Remote debugging using 192.168.50.4:44000
> Temporary breakpoint 1 at 0x20374: file /Projects/CSU/src/main.cpp, line 16.
> Starting program: /opt/FS/common/csu 
> 
> Temporary breakpoint 1, main () at /Projects/CSU/src/main.cpp:16
> 16              MsgLog("OFP Version: %s dated %s", OFP_VERSION, 
> __TIMESTAMP__);
> (gdb) c
> Continuing.
> ^C
> Program received signal SIGINT, Interrupt.
> 0x400314bc in pthread_cond_wait@@GLIBC_2.4 () from /opt/FS/lib/libpthread.so.0
> (gdb) bt
> #0  0x400314bc in pthread_cond_wait@@GLIBC_2.4 () from 
> /opt/FS/lib/libpthread.so.0
> #1  0x00000000 in ?? ()
> (gdb) 
> ----------------------------------------------------------------------------
> God knows why it decided to stop at main...

This is done intentionally by the GDB command `start'.


> Is it what you meant?

So symbols from /opt/FS/common/csu work as can be seen in the main function.

Symbols from remote `/lib/libpthread.so.0' do not work. I guess it uses
separate debug info.  You can try:
readelf -wi /opt/FS/lib/libpthread.so.0
readelf -wi /opt/FS/usr/lib/debug/opt/FS/lib/libpthread.so.0
readelf -wi /opt/FS/usr/lib/debug/opt/FS/lib/libpthread.so.0.debug
# There are just guesses, it depends on the montavista distro from which comes
# /opt/FS/lib/libpthread.so.0 and I do not know montavista. readelf should
# come from the montavista cross binutils package.

I guess you should really use also the part of the reproducer from me:
  -ex 'set debug-file-directory /host1/usr/lib/debug
that is in your case:
  -ex 'set debug-file-directory /opt/FS/usr/lib/debug

And you should also verify the separate debug info files from montavista are
installed under /opt/FS/usr/lib/debug (for example *-debuginfo.rpm packages in
Fedora).


Regards,
Jan

Re: Restarting gdbserver

[Leon Pollak]

Jan, I start to understand...:-)

You say that the libpthread shared library does not have debug info in it. 

OK, to check myself and understand your explanation, I compiled the same 
application in the "normal" Fedora 12 - it does show the backtrace.

I ran 'i shared' command in both cases. 
Fedora case - exactly what I need:
From        To          Syms Read   Shared Object Library
0x0065a830  0x006720bf  Yes (*)     /lib/ld-linux.so.2
0x00114170  0x00122b28  Yes (*)     /lib/libnsl.so.1
0x009a3880  0x009a7678  Yes (*)     /lib/librt.so.1
0x00809a60  0x0080aa88  Yes (*)     /lib/libdl.so.2
0x008145e0  0x00820518  Yes (*)     /lib/libpthread.so.0
0x0016eeb0  0x001e1838  Yes (*)     /usr/lib/libstdc++.so.6
0x00830470  0x0084a968  Yes (*)     /lib/libm.so.6
0x00218fa0  0x002313c8  Yes (*)     /lib/libgcc_s.so.1
0x00692aa0  0x007b11f4  Yes (*)     /lib/libc.so.6
(*): Shared library is missing debugging information.
(gdb) bt
#0  0x00110430 in __kernel_vsyscall ()
#1  0x0081a22c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
#2  0x08078757 in ucc::Conditional::wait() ()
#3  0x0807c15b in ucc::queue::fifo (this=0x80a07e0, timeout=4294967295) at 
/Projects/ucommon.3.0.5/common/thread.cpp:2115
#4  0x0806efd2 in cDispatcher::Processor (this=0x80a08a4) at 
/Projects/CSU/src/Dispatcher.cpp:94
#5  0x0806d323 in main () at /Projects/CSU/src/main.cpp:27
=============================

Cross-linux case:
(gdb) i shared
From        To          Syms Read   Shared Object Library
0x400007e0  0x40018ab4  Yes (*)     /opt/FS/lib/ld-linux.so.3
0x40029ce0  0x400383e8  Yes (*)     /opt/FS/lib/libpthread.so.0
0x4005d9f8  0x400a505c  Yes         /opt/FS/lib/libucommon.so.3
0x400f8664  0x4017358c  Yes (*)     /opt/FS/usr/lib/libstdc++.so.6
0x40199234  0x40201264  Yes (*)     /opt/FS/lib/libm.so.6
0x40243248  0x4024b2c8  Yes (*)     /opt/FS/lib/libgcc_s.so.1
0x40268990  0x4034e710  Yes (*)     /opt/FS/lib/libc.so.6
0x4037a640  0x4037dc6c  Yes (*)     /opt/FS/lib/librt.so.1
0x40388974  0x403896f0  Yes (*)     /opt/FS/lib/libdl.so.2
(*): Shared library is missing debugging information.
(gdb) bt
#0  0x400314bc in pthread_cond_wait@@GLIBC_2.4 () from 
/opt/FS/lib/libpthread.so.0
#1  0x00000000 in ?? ()
=======================================

In both cases gdb says "Shared library is missing debugging information"!
And I searched my Fedora installatoin - no *debug*.so at all.

So, how all this works?


Thank you very much for your lessons...:-)

On Sunday January 23 2011, Jan Kratochvil wrote:
> On Sun, 23 Jan 2011 17:43:44 +0100, Leon Pollak wrote:
> > Thank you, Jan.
> > 
> > I hope I understood and did what you said:
> > 
> > bash-4.1$ ./gdb -nx -ex file /opt/FS/common/csu -ex target
> > extended-remote 192.168.50.4:44000 -ex set sysroot /opt/FS -ex set
> > remote exec-file /common/csu -ex start
> > GNU gdb (GDB) 7.2
> > Copyright (C) 2010 Free Software Foundation, Inc.
> > License GPLv3+: GNU GPL version 3 or later
> > <http://gnu.org/licenses/gpl.html> This is free software: you are free
> > to change and redistribute it. There is NO WARRANTY, to the extent
> > permitted by law.  Type "show copying" and "show warranty" for details.
> > This GDB was configured as "--host=x86_64-unknown-linux-gnu
> > --target=armv5tl- montavista-linux-gnueabi".
> > For bug reporting instructions, please see:
> > <http://www.gnu.org/software/gdb/bugs/>.
> > Reading symbols from /opt/FS/common/csu...done.
> > Remote debugging using 192.168.50.4:44000
> > Temporary breakpoint 1 at 0x20374: file /Projects/CSU/src/main.cpp, line
> > 16. Starting program: /opt/FS/common/csu
> > 
> > Temporary breakpoint 1, main () at /Projects/CSU/src/main.cpp:16
> > 16              MsgLog("OFP Version: %s dated %s", OFP_VERSION,
> > __TIMESTAMP__);
> > (gdb) c
> > Continuing.
> > ^C
> > Program received signal SIGINT, Interrupt.
> > 0x400314bc in pthread_cond_wait@@GLIBC_2.4 () from
> > /opt/FS/lib/libpthread.so.0 (gdb) bt
> > #0  0x400314bc in pthread_cond_wait@@GLIBC_2.4 () from
> > /opt/FS/lib/libpthread.so.0
> > #1  0x00000000 in ?? ()
> > (gdb)
> > -------------------------------------------------------------------------
> > --- God knows why it decided to stop at main...
> 
> This is done intentionally by the GDB command `start'.
> 
> > Is it what you meant?
> 
> So symbols from /opt/FS/common/csu work as can be seen in the main
> function.
> 
> Symbols from remote `/lib/libpthread.so.0' do not work. I guess it uses
> separate debug info.  You can try:
> readelf -wi /opt/FS/lib/libpthread.so.0
> readelf -wi /opt/FS/usr/lib/debug/opt/FS/lib/libpthread.so.0
> readelf -wi /opt/FS/usr/lib/debug/opt/FS/lib/libpthread.so.0.debug
> # There are just guesses, it depends on the montavista distro from which
> comes # /opt/FS/lib/libpthread.so.0 and I do not know montavista. readelf
> should # come from the montavista cross binutils package.
> 
> I guess you should really use also the part of the reproducer from me:
>   -ex 'set debug-file-directory /host1/usr/lib/debug
> that is in your case:
>   -ex 'set debug-file-directory /opt/FS/usr/lib/debug
> 
> And you should also verify the separate debug info files from montavista
> are installed under /opt/FS/usr/lib/debug (for example *-debuginfo.rpm
> packages in Fedora).
> 
> 
> Regards,
> Jan


-- 

Re: Restarting gdbserver

[Jan Kratochvil]

On Sun, 23 Jan 2011 19:05:42 +0100, Leon Pollak wrote:
> OK, to check myself and understand your explanation, I compiled the same 
> application in the "normal" Fedora 12 - it does show the backtrace.

OK, so you do not have debug info in either case.

But on x86* Fedora you can backtrace any system code even if you do not have
the debug info installed.  (This is not true for any x86* code but Fedora is
using -fasynchronous-unwind-tables so it can backtrace even
-fomit-frame-pointer code without debug info.)

On arm I guess - I do not know arm - you need debug info for the backtrace.
But this is very montavista distro specific question and in fact offtopic here.


> From        To          Syms Read   Shared Object Library
[...]
> 0x008145e0  0x00820518  Yes (*)     /lib/libpthread.so.0
[...]
> (*): Shared library is missing debugging information.
> (gdb) bt
> #0  0x00110430 in __kernel_vsyscall ()
> #1  0x0081a22c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0

This record shows the debug info is missing.  You can try
`debuginfo-install glibc' on Fedora to get the debug info.  Still it does not
affect the capability of backtracing on x86*.


> And I searched my Fedora installatoin - no *debug*.so at all.

You would need to install it first:

# repoquery -q --enablerepo='fedora*-debuginfo' glibc-debuginfo
glibc-debuginfo-0:2.12.90-17.i686
# debuginfo-install glibc
# ls -lL /usr/lib/debug/lib/libpthread.so.0.debug 
-r--r--r-- 1 root root 708656 Dec 14 17:53 /usr/lib/debug/lib/libpthread.so.0.debug
(Adjusted from x86_64 as you seem to run 32bit.)



Regards,
Jan

Re: Restarting gdbserver

[Andreas Schwab]

Leon Pollak <leonp@plris.com> writes:

> OK, to check myself and understand your explanation, I compiled the same 
> application in the "normal" Fedora 12 - it does show the backtrace.

Probably due to -fasynchronous-unwind-tables.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: Restarting gdbserver

[Leon Pollak]

SORRY PLEASE!!!!!
I forgot the backtrace output:
(gdb) bt
#0  0x400314bc in pthread_cond_wait@@GLIBC_2.4 () from 
/opt/FS/lib/libpthread.so.0
#1  0x00000000 in ?? ()



On Thursday January 20 2011, Jan Kratochvil wrote:
> or also:
> 	gdbserver --multi COMM
> 	using `target extended-remote ...' instead.
WOW!!!!!!!!
How could I not find this!!??
1001 of thanks!
Works fine. I start thegdb with:

set sysroot /opt/FS/
set remote exec-file /common/csu
target extended-remote 192.168.50.4:44000
---
The only one problem left for me is that when stopped/broken, the stack frame 
is not presented as it usually is:

(gdb) frame
#0  0x400314bc in pthread_cond_wait@@GLIBC_2.4 () from 
/opt/FS/lib/libpthread.so.0

And nothing else...:-(

Thousands of thanks ahead for the help...
-- 
Leon