Setting breakpoint misbehaving with all threads running in Non-Stop on Linux

Setting breakpoint misbehaving with all threads running in Non-Stop on Linux

[Marc Khouzam]

Hi again,

I'm using HEAD (from yesterday) with Non-Stop locally on Linux.
I notice that when all my threads are running, setting a breakpoint
is misbehaving.

First, should I be able to set a breakpoint when all threads
are running (on Linux)?

Either way though, setting a bp reports an error -with-
a breakpoint id, and then 'info break' shows the breakpoint
as being set.  However, the breakpoint does not actually hit.

See below for the session.

Thanks

> gdb -i mi
~"GNU gdb (GDB) 6.8.50.20090430-cvs\n"
~"Copyright (C) 2009 Free Software Foundation, Inc.\n"
~"License GPLv3+: GNU GPL version 3 or later
<http://gnu.org/licenses/gpl.html>\n"
~"This is free software: you are free to change and redistribute it.\n"
~"There is NO WARRANTY, to the extent permitted by law.  Type \"show
copying\"\n"
~"and \"show warranty\" for details.\n"
~"This GDB was configured as \"i686-pc-linux-gnu\".\n"
~"For bug reporting instructions, please see:\n"
~"<http://www.gnu.org/software/gdb/bugs/>.\n"
(gdb) 
file a.out
&"file a.out\n"
~"Reading symbols from /local/home/lmckhou/testing/a.out..."
~"done.\n"
^done
(gdb) 
l 1
&"l 1\n"
~"1\t#include <stdio.h>\n"
~"2\t#include <stdlib.h>\n"
~"3\t#include <pthread.h>\n"
~"4\t#include <unistd.h>\n"
~"5\t\n"
~"6\tvoid *thread_exec(void *ptr)\n"
~"7\t{\n"
~"8\t    int j = 0;\n"
~"9\t    for (int i=0;i<30;i++) {\n"
~"10\t        j++;\n"
^done
(gdb) 
l
&"l\n"
~"11\t        sleep(1);\n"
~"12\t    }\n"
~"13\t}\n"
~"14\t\n"
~"15\tint main()\n"
~"16\t{\n"
~"17\t    pthread_t thread2;\n"
~"18\t    char *message2 = \"Thread 2\";\n"
~"19\t\n"
~"20\t    pthread_create(&thread2, NULL, thread_exec, (void*)
message2);\n"
^done
(gdb) 
l
&"l\n"
~"21\t\n"
~"22\t    sleep(15);\n"
~"23\t\n"
~"24\t    pthread_join(thread2, NULL);\n"
~"25\t\n"
~"26\t    printf(\"Thread 2 finished\\n\");\n"
~"27\t\n"
~"28\t    return 0;\n"
~"29\t}\n"
^done
(gdb) 
set target-async on
&"set target-async on\n"
^done
(gdb) 
set pagination off
&"set pagination off\n"
^done
(gdb) 
set non-stop on
&"set non-stop on\n"
^done
(gdb) 
b 24
&"b 24\n"
~"Breakpoint 1 at 0x80485dc: file MultiThread.cc, line 24.\n"
^done
(gdb) 
run&
&"run&\n"
~"Starting program: /local/home/lmckhou/testing/a.out \n"
=thread-group-created,id="23591"
=thread-created,id="1",group-id="23591"
^running
*running,thread-id="1"
=library-loaded,id="/lib/ld-linux.so.2",target-name="/lib/ld-linux.so.2"
,host-name="/lib/ld-linux.so.2",symbols-loaded="0"
(gdb) 
=library-loaded,id="/lib/libpthread.so.0",target-name="/lib/libpthread.s
o.0",host-name="/lib/libpthread.so.0",symbols-loaded="0"
=library-loaded,id="/usr/lib/libstdc++.so.6",target-name="/usr/lib/libst
dc++.so.6",host-name="/usr/lib/libstdc++.so.6",symbols-loaded="0"
=library-loaded,id="/lib/libm.so.6",target-name="/lib/libm.so.6",host-na
me="/lib/libm.so.6",symbols-loaded="0"
=library-loaded,id="/lib/libgcc_s.so.1",target-name="/lib/libgcc_s.so.1"
,host-name="/lib/libgcc_s.so.1",symbols-loaded="0"
=library-loaded,id="/lib/libc.so.6",target-name="/lib/libc.so.6",host-na
me="/lib/libc.so.6",symbols-loaded="0"
~"[Thread debugging using libthread_db enabled]\n"
=thread-created,id="2",group-id="23591"
~"[New Thread 0xb7d6cba0 (LWP 23598)]\n"
*running,thread-id="2"
info th
&"info th\n"
~"  2 Thread 0xb7d6cba0 (LWP 23598)  (running)\n"
~"* 1 Thread 0xb7d6d6b0 (LWP 23591)  (running)\n"
^done
(gdb) 

== Here all threads are running ==

b 10
&"b 10\n"
~"Breakpoint 2 at 0x804857a: file MultiThread.cc, line 10.\n"
&"Warning:\n"
&"Cannot insert breakpoint 2.\n"
&"Error accessing memory address 0x804857a: Input/output error.\n"
&"\n"
^error,msg="Warning:\nCannot insert breakpoint 2.\nError accessing
memory address 0x804857a: Input/output error.\n"

== error but bp is shown in 'info break' (see below) ==
(gdb) 
b 11
&"b 11\n"
~"Breakpoint 3 at 0x804857e: file MultiThread.cc, line 11.\n"
&"Warning:\n"
&"Cannot insert breakpoint 2.\n"
&"Error accessing memory address 0x804857a: Input/output error.\n"
&"Cannot insert breakpoint 3.\n"
&"Error accessing memory address 0x804857e: Input/output error.\n"
&"\n"
^error,msg="Warning:\nCannot insert breakpoint 2.\nError accessing
memory address 0x804857a: Input/output error.\nCannot insert breakpoint
3.\nError accessing memory address 0x804857e: Input/output error.\n"

== Cummulative error above ==

(gdb) 
info b
&"info b\n"
~"Num     Type           Disp Enb Address    What\n"
~"1       breakpoint     keep y   0x080485dc in main at
MultiThread.cc:24\n"
~"2       breakpoint     keep y   0x0804857a in thread_exec(void*) at
MultiThread.cc:10\n"
~"3       breakpoint     keep y   0x0804857e in thread_exec(void*) at
MultiThread.cc:11\n"
^done

== Both 'failed' breakpoints show as installed, but they ==
== don't actually stop the thread.                       ==

(gdb) 
*stopped,reason="breakpoint-hit",disp="keep",bkptno="1",frame={addr="0x0
80485dc",func="main",args=[],file="MultiThread.cc",fullname="/local/home
/lmckhou/testing/MultiThread.cc",line="24"},thread-id="1",stopped-thread
s=["1"]

Re: Setting breakpoint misbehaving with all threads running in Non-Stop on Linux

[Pedro Alves]

On Friday 01 May 2009 20:31:21, Marc Khouzam wrote:

> I'm using HEAD (from yesterday) with Non-Stop locally on Linux.
> I notice that when all my threads are running, setting a breakpoint
> is misbehaving.
> 
> First, should I be able to set a breakpoint when all threads
> are running (on Linux)?

I've worked with non-stop mode in a few targets other than linux
already, and so far, only linux has this issue, and it
is *really* a nuisance.  I've been thinking we should make it
possible on linux to insert breakpoints when threads are running
as well.  The user experience is just bad otherwise.

> Either way though, setting a bp reports an error -with-
> a breakpoint id, and then 'info break' shows the breakpoint
> as being set.  However, the breakpoint does not actually hit.
> 
> See below for the session.


> 
> (gdb) 
> info b
> &"info b\n"
> ~"Num     Type           Disp Enb Address    What\n"
> ~"1       breakpoint     keep y   0x080485dc in main at
> MultiThread.cc:24\n"
> ~"2       breakpoint     keep y   0x0804857a in thread_exec(void*) at
> MultiThread.cc:10\n"
> ~"3       breakpoint     keep y   0x0804857e in thread_exec(void*) at
> MultiThread.cc:11\n"
> ^done
> 
> == Both 'failed' breakpoints show as installed, but they ==
> == don't actually stop the thread.                       ==

I don't think this output indicates the breakpoints are installed or not. 
"Enb" is a high level user setting; The "Address" field showing a resolved address
means that GDB is sure of the breakpoint address and believes those addresses
belong to currently mapped memory area --- if the breakpoints that failed
to insert pointed at code in a shared library, GDB would have made the
breakpoint locations pending (<PENDING>), because it would assume that
the reason the insertion failed was due to the shared library having
been unloaded.  But, GDB doesn't do that for breakpoints set in
the main executable -- only in shlibs.

What would people think of adding a new column in "info breakpoints" showing
the "inserted" status of the breakpoint?  It should be used to
show 'inserted', 'not inserted', 'not inserted due to error' state,
and perhaps more states.

E.g.:

 Num     Type           Disp Enb Ins Address    What
 1       breakpoint     keep y   y   0x080485dc in main at MultiThread.cc:24
 2       breakpoint     keep y   n   <PENDING>  at foo
 3       breakpoint     keep y   E   0xaaff8f45 at foo2

?

-- 
Pedro Alves

RE: Setting breakpoint misbehaving with all threads running in Non-Stop on Linux

[Marc Khouzam]

> > I'm using HEAD (from yesterday) with Non-Stop locally on Linux.
> > I notice that when all my threads are running, setting a breakpoint
> > is misbehaving.
> >
> > First, should I be able to set a breakpoint when all threads
> > are running (on Linux)?
> 
> I've worked with non-stop mode in a few targets other than linux
> already, and so far, only linux has this issue, and it
> is *really* a nuisance.  I've been thinking we should make it
> possible on linux to insert breakpoints when threads are running
> as well.  The user experience is just bad otherwise.

I very much agree with you.  
 
And that is really what we need
on the frontend side.  I'll probably have to force such a behavior
(by suspending a thread, planting the bp, and resuming) 
if GDB does not provide it directly.

> > Either way though, setting a bp reports an error -with-
> > a breakpoint id, and then 'info break' shows the breakpoint
> > as being set.  However, the breakpoint does not actually hit.
> >
> > See below for the session.
> >
> > (gdb)
> > info b
> > &"info b\n"
> > ~"Num     Type           Disp Enb Address    What\n"
> > ~"1       breakpoint     keep y   0x080485dc in main at
> > MultiThread.cc:24\n"
> > ~"2       breakpoint     keep y   0x0804857a in thread_exec(void*) at
> > MultiThread.cc:10\n"
> > ~"3       breakpoint     keep y   0x0804857e in thread_exec(void*) at
> > MultiThread.cc:11\n"
> > ^done
> >
> > == Both 'failed' breakpoints show as installed, but they ==
> > == don't actually stop the thread.                       ==
> 
> I don't think this output indicates the breakpoints are installed or not.
> "Enb" is a high level user setting; The "Address" field showing a resolved address
> means that GDB is sure of the breakpoint address and believes those addresses
> belong to currently mapped memory area --- if the breakpoints that failed
> to insert pointed at code in a shared library, GDB would have made the
> breakpoint locations pending (<PENDING>), because it would assume that
> the reason the insertion failed was due to the shared library having
> been unloaded.  But, GDB doesn't do that for breakpoints set in
> the main executable -- only in shlibs.

I'm just surprised that GDB returns an ^error, and still shows the breakpoint
in the list.  If the bp was <PENDING> I don't believe GDB would have returned
an ^error.

> What would people think of adding a new column in "info breakpoints" showing
> the "inserted" status of the breakpoint?  It should be used to
> show 'inserted', 'not inserted', 'not inserted due to error' state,
> and perhaps more states.
> 
> E.g.:
> 
>  Num     Type           Disp Enb Ins Address    What
> 1       breakpoint     keep y   y   0x080485dc in main at MultiThread.cc:24
 > 2       breakpoint     keep y   n   <PENDING>  at foo
>  3       breakpoint     keep y   E   0xaaff8f45 at foo2

What is the value of have a bp shown with an Error?  Will that bp ever
work?  Why should it be kept around?
What I noticed is that even after the main thread stops, in my test,
the failed bp still does not hit.  So, I'm wondering of it usefulness.
 
Thanks
 
Marc
 

Re: Setting breakpoint misbehaving with all threads running in Non-Stop on Linux

[Pedro Alves]

On Sunday 03 May 2009 03:30:09, Marc Khouzam wrote:

> I'm just surprised that GDB returns an ^error, and still shows the breakpoint
> in the list.  If the bp was <PENDING> I don't believe GDB would have returned
> an ^error.

Yes, I see what you mean.  Hmmm, to understand the issue here, you
have to realise that breakpoint handling is split in roughly two
layers --- the logical/user/high-level breakpoints, and the
physical/low-level breakpoint locations (the distinction isn't that
pure in GDB's implementation, but still, it's there).

When in stock all-stop mode, you issue "break foo", only the
high-level part is handled --- roughly, the breakpoint address is
resolved, and the breakpoint is added to the breakpoint list.  The command
is completed at this point, and ends successfully.  The breakpoints are
only physically inserted in the target the next time you try to
resume it, say with a "continue".  At this point, in all-stop mode, if
the insertion fails, you'll see those errors in the console, which
will cancel the "continue" command (remember that errors are like java/c++
exceptions in GDB) and leave the inferior stopped.  So, when these kinds of
errors happen, the 'break foo' command as long since been
executed successfully.

In non-stop mode, however, (or with all-stop + 'set breakpoint
always-inserted on'), the "breakpoint" command behaves a bit
differently.  As soon as the logical high-level breakpoint
is added to the breakpoint list, GDB tries to insert the physical
breakpoints into the target, still before the "b foo" command finishes.
So, any error thrown while inserting the breakpoint in the target
ends up as an "^error" return to the 'break command'.  Maybe this
behaviour isn't correct, but it may be tricky to do otherwise.

Since in non-stop mode, this exception isn't going to stop the inferior
from continuing to execute, perhaps the insertion errors
should be downgraded to warnings (in non-stop mode only), and GDB should
return success indicating that the logical breakpoint was created anyway?

> > What would people think of adding a new column in "info breakpoints" showing
> > the "inserted" status of the breakpoint?  It should be used to
> > show 'inserted', 'not inserted', 'not inserted due to error' state,
> > and perhaps more states.
> > 
> > E.g.:
> > 
> >  Num     Type           Disp Enb Ins Address    What
> > 1       breakpoint     keep y   y   0x080485dc in main at MultiThread.cc:24
>  > 2       breakpoint     keep y   n   <PENDING>  at foo
> >  3       breakpoint     keep y   E   0xaaff8f45 at foo2
> 
> What is the value of have a bp shown with an Error?  Will that bp ever
> work?  Why should it be kept around?
> What I noticed is that even after the main thread stops, in my test,
> the failed bp still does not hit.  So, I'm wondering of it usefulness.

I bet that if you *resume* the thread again after that stop, the breakpoint
would finally insert successfully.

I've worked with some remote targets that can refuse to insert
breakpoints in some locations for whatever reasons.  If it happens that
the breakpoint is in a shared library, and because GDB currently assumes
that any error while inserting such breakpoints mean that the corresponding
shared library was unloaded, so it suppresses the insert failed error message,
and the "Address" field is displayed as "<PENDING>".
I've had customers report that as confusing, since the address location
isn't pending at all, per the PENDING definition --- GDB knows well
where it should be inserted, and, the shared library *is* mapped
in.  If the breakpoint was set in the main code, as in your example, 
than GDB is noisy when the user tries to insert such breakpoints, but
the error out is transient and gets lost in the console log.
Having an extra column would allow showing those breakpoint's insertion
states.  Another example is the case of the target *temporarily* refusing
to install breakpoints, telling GDB "not now, but at a later time they
can be inserted".  I know Ulrich proposed adding such support for the
multi-arch Cell debugger.  I'm thinking that such new column
could "Ins" -> "nT" for "not inserted, target is refusing such
breakpoints temporarily".

Hope I've made some sense.

-- 
Pedro Alves

RE: Setting breakpoint misbehaving with all threads running in Non-Stop on Linux

[Marc Khouzam]

> -----Original Message-----
> From: Pedro Alves [mailto:pedro@codesourcery.com] 
> Sent: Tuesday, May 05, 2009 1:42 PM
> To: Marc Khouzam
> Cc: gdb@sourceware.org
> Subject: Re: Setting breakpoint misbehaving with all threads 
> running in Non-Stop on Linux
> 
> On Sunday 03 May 2009 03:30:09, Marc Khouzam wrote:
> 
> > I'm just surprised that GDB returns an ^error, and still 
> shows the breakpoint
> > in the list.  If the bp was <PENDING> I don't believe GDB 
> would have returned
> > an ^error.
> 
> Yes, I see what you mean.  Hmmm, to understand the issue here, you
> have to realise that breakpoint handling is split in roughly two
> layers --- the logical/user/high-level breakpoints, and the
> physical/low-level breakpoint locations (the distinction isn't that
> pure in GDB's implementation, but still, it's there).
> 
> When in stock all-stop mode, you issue "break foo", only the
> high-level part is handled --- roughly, the breakpoint address is
> resolved, and the breakpoint is added to the breakpoint list. 
>  The command
> is completed at this point, and ends successfully.  The 
> breakpoints are
> only physically inserted in the target the next time you try to
> resume it, say with a "continue".  At this point, in all-stop mode, if
> the insertion fails, you'll see those errors in the console, which
> will cancel the "continue" command (remember that errors are 
> like java/c++
> exceptions in GDB) and leave the inferior stopped.  So, when 
> these kinds of
> errors happen, the 'break foo' command as long since been
> executed successfully.
> 
> In non-stop mode, however, (or with all-stop + 'set breakpoint
> always-inserted on'), the "breakpoint" command behaves a bit
> differently.  As soon as the logical high-level breakpoint
> is added to the breakpoint list, GDB tries to insert the physical
> breakpoints into the target, still before the "b foo" command 
> finishes.
> So, any error thrown while inserting the breakpoint in the target
> ends up as an "^error" return to the 'break command'.  Maybe this
> behaviour isn't correct, but it may be tricky to do otherwise.
> 

Thanks for this explanation, it made things much clearer for me.

Eclipse has a similar mechanism, where a blue dot is shown where
a breakpoint is set, and if the bp is properly installed, a check
mark is added.

The problem I have, if I understood your explanation properly,
is that GDB does not clearly tell the frontend if the breakpoint
is installed.  What I mean is, that if Eclipse shows a breakpoint
as not installed then it should never hit, and the reverse should
be true too.

But from what you wrote, it seems that GDB can report a failed
installation (in non-stop) and yet the bp will hit at some point
in the future, when installation succeeds.
From what I understand, the ^error or ^done following a -break-insert
does not really indicate if the breakpoint has been/will be installed
successfully.  A little like <PENDING>.

I guess this comes back to the need for a =breapoint-inserted/installed
event.  We need a way for the frontend to properly indicate to the user
if a bp should be expected to hit or not.

We have this same problem for <PENDING> because we don't have such an
event notifying the frontend that the breapoint is no longer Pending,
but has been installed.

I think my best bet is to show a bp that failed, as Uninstalled, and
wait for it to hit.  At that point, I can mark it as Installed.
This is a bit what we are planning to do for <PENDING> bp, except
that in that case, we can use the new =library-loaded events
as an indicator that some Pending bp might have just been installed
and update the UI.

> Since in non-stop mode, this exception isn't going to stop 
> the inferior
> from continuing to execute, perhaps the insertion errors
> should be downgraded to warnings (in non-stop mode only), and 
> GDB should
> return success indicating that the logical breakpoint was 
> created anyway?

At this point, I think this would be an improvement for frontends.
What I think we need is that
^error would mean bp failed permanently, while a successful answer
would need to differentiate between 'already installed' and
'may install in the future' which comes down to PENDING.
-break-insert already has an indicator for PENDING it is reply and
this situation could be treated the same, I think.

> > > What would people think of adding a new column in "info 
> breakpoints" showing
> > > the "inserted" status of the breakpoint?  It should be used to
> > > show 'inserted', 'not inserted', 'not inserted due to 
> error' state,
> > > and perhaps more states.
> > > 
> > > E.g.:
> > > 
> > >  Num     Type           Disp Enb Ins Address    What
> > > 1       breakpoint     keep y   y   0x080485dc in main at 
> MultiThread.cc:24
> >  > 2       breakpoint     keep y   n   <PENDING>  at foo
> > >  3       breakpoint     keep y   E   0xaaff8f45 at foo2
> > 
> > What is the value of have a bp shown with an Error?  Will 
> that bp ever
> > work?  Why should it be kept around?
> > What I noticed is that even after the main thread stops, in my test,
> > the failed bp still does not hit.  So, I'm wondering of it 
> usefulness.
> 
> I bet that if you *resume* the thread again after that stop, 
> the breakpoint
> would finally insert successfully.
> 
> I've worked with some remote targets that can refuse to insert
> breakpoints in some locations for whatever reasons.  If it 
> happens that
> the breakpoint is in a shared library, and because GDB 
> currently assumes
> that any error while inserting such breakpoints mean that the 
> corresponding
> shared library was unloaded, so it suppresses the insert 
> failed error message,
> and the "Address" field is displayed as "<PENDING>".
> I've had customers report that as confusing, since the 
> address location
> isn't pending at all, per the PENDING definition --- GDB knows well
> where it should be inserted, and, the shared library *is* mapped
> in.  If the breakpoint was set in the main code, as in your example, 
> than GDB is noisy when the user tries to insert such breakpoints, but
> the error out is transient and gets lost in the console log.
> Having an extra column would allow showing those breakpoint's 
> insertion
> states.  Another example is the case of the target 
> *temporarily* refusing
> to install breakpoints, telling GDB "not now, but at a later time they
> can be inserted".  I know Ulrich proposed adding such support for the
> multi-arch Cell debugger.  I'm thinking that such new column
> could "Ins" -> "nT" for "not inserted, target is refusing such
> breakpoints temporarily".
> 
> Hope I've made some sense.

Yes, thank you.
We don't really use the 'info break' output, as we have tried to rely
on MI and MI events only, so I have no preference about the suggested 
new column from a DSF-GDB point-of-view.

Thanks again.

Marc