[mi] watchpoint-scope exec async command

[mi] watchpoint-scope exec async command

[Bob Rossi]

Hi,

I've been trying to understand and document the exec-async commands that
GDB sends to the FE. So far, I think I've understood them all, and 
reproduced them. However, the "watchpoint-scope" one is giving me a hard time.

Here's the sample program, 

int wpscope ( int param ) {
    return param+1;
}

int main(int argc, char **argv){
    return wpscope ( argc );
}

$ ../gdb -i=mi ./wpscope
(gdb)
-break-insert wpscope
^done,bkpt={number="1",type="breakpoint",disp="keep",enabled="y",addr="0x08048357",func="wpscope",file="wpscope.c",line="2",times="0"}
(gdb)
-exec-run
^running
(gdb)
*stopped,reason="breakpoint-hit",bkptno="1",thread-id="0",frame={addr="0x08048357",func="wpscope",args=[{name="param",value="1"}],file="wpscope.c",line="2"}
(gdb)
-break-watch param
^done,wpt={number="2",exp="param"}
(gdb)
-exec-continue
^running
(gdb)
~"Hardware watchpoint 2 deleted because the program has left the block \n"
~"in which its expression is valid.\n"
*stopped,reason="exited",exit-code="02"
(gdb)

Is it just a bug that there is no 'watchpoint-scope' returned?
The 'break-watch' doco in the manual leads me to believe that I should
get back the "watchpoint-scope" when the hardware watchpoint was
deleted. However, I don't really have any clue if this is a bug or
desired behavior.

Any help would be appreciated, since hopefully I can better document the
reasons why such events would occur.

Thanks,
Bob Rossi

gdbserver question

[james osburn]

I am trying to remote debug a program using gdb server and i am
having the following problems.

first i have started my program on the remote target as follows:

gdbserver 192.168.0.10:1234 ./stubs

(the name of the executeable is "stubs")

i then start gdb on the host side as follows:
$> gdb
GNU gdb 5.3
Copyright 2002 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain 
conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i686-pc-linux-gnu".
(gdb) target extended-remote 192.168.0.10:1234
Remote debugging using 192.168.0.10:1234
0x400009c0 in ?? ()
(gdb) break main
No symbol table is loaded.  Use the "file" command.

what do i need to do?
thanks.
jim

Re: gdbserver question

[Daniel Jacobowitz]

On Fri, Mar 25, 2005 at 04:25:17PM +0000, james osburn wrote:
> I am trying to remote debug a program using gdb server and i am
> having the following problems.

Please don't reply to unrelated messages to post to the list; just send
mail directly.

> first i have started my program on the remote target as follows:
> 
> gdbserver 192.168.0.10:1234 ./stubs
> 
> (the name of the executeable is "stubs")
> 
> i then start gdb on the host side as follows:
> $> gdb
> GNU gdb 5.3
> Copyright 2002 Free Software Foundation, Inc.
> GDB is free software, covered by the GNU General Public License, and you are
> welcome to change it and/or distribute copies of it under certain 
> conditions.
> Type "show copying" to see the conditions.
> There is absolutely no warranty for GDB.  Type "show warranty" for details.
> This GDB was configured as "i686-pc-linux-gnu".
> (gdb) target extended-remote 192.168.0.10:1234
> Remote debugging using 192.168.0.10:1234
> 0x400009c0 in ?? ()
> (gdb) break main
> No symbol table is loaded.  Use the "file" command.
> 
> what do i need to do?

Have you read the error message, and tried what it suggested?  Use the
file command, before connecting to your remote target.

You may want to read the GDB manual; there's documentation of the
"file" command, and also examples for GDB and gdbserver.

-- 
Daniel Jacobowitz
CodeSourcery, LLC

Re: [mi] watchpoint-scope exec async command

[Eli Zaretskii]

> Date: Fri, 25 Mar 2005 11:12:39 -0500
> From: Bob Rossi <bob@brasko.net>
> 
> (gdb)
> -break-watch param
> ^done,wpt={number="2",exp="param"}
> (gdb)
> -exec-continue
> ^running
> (gdb)
> ~"Hardware watchpoint 2 deleted because the program has left the block \n"
> ~"in which its expression is valid.\n"
> *stopped,reason="exited",exit-code="02"
> (gdb)
> 
> Is it just a bug that there is no 'watchpoint-scope' returned?

I think it's a bug, yes: there's one instance in breakpoint.c
(specifically, in the function insert_bp_location) that uses
printf_filtered instead of the ui_* functions to produce the warning
about a watchpoint that went out of scope.  I think it should use the
same code that is used by watchpoint_check to produce a similar
warning.

Re: [mi] watchpoint-scope exec async command

[Bob Rossi]

On Sat, Mar 26, 2005 at 03:24:10PM +0200, Eli Zaretskii wrote:
> > Date: Fri, 25 Mar 2005 11:12:39 -0500
> > From: Bob Rossi <bob@brasko.net>
> > 
> > (gdb)
> > -break-watch param
> > ^done,wpt={number="2",exp="param"}
> > (gdb)
> > -exec-continue
> > ^running
> > (gdb)
> > ~"Hardware watchpoint 2 deleted because the program has left the block \n"
> > ~"in which its expression is valid.\n"
> > *stopped,reason="exited",exit-code="02"
> > (gdb)
> > 
> > Is it just a bug that there is no 'watchpoint-scope' returned?
> 
> I think it's a bug, yes: there's one instance in breakpoint.c
> (specifically, in the function insert_bp_location) that uses
> printf_filtered instead of the ui_* functions to produce the warning
> about a watchpoint that went out of scope.  I think it should use the
> same code that is used by watchpoint_check to produce a similar
> warning.

OK, I'll look into fixing this.

Bob Rossi

Re: [mi] watchpoint-scope exec async command

[Bob Rossi]

On Sat, Mar 26, 2005 at 03:24:10PM +0200, Eli Zaretskii wrote:
> > Date: Fri, 25 Mar 2005 11:12:39 -0500
> > From: Bob Rossi <bob@brasko.net>
> > 
> > (gdb)
> > -break-watch param
> > ^done,wpt={number="2",exp="param"}
> > (gdb)
> > -exec-continue
> > ^running
> > (gdb)
> > ~"Hardware watchpoint 2 deleted because the program has left the block \n"
> > ~"in which its expression is valid.\n"
> > *stopped,reason="exited",exit-code="02"
> > (gdb)
> > 
> > Is it just a bug that there is no 'watchpoint-scope' returned?
> 
> I think it's a bug, yes: there's one instance in breakpoint.c
> (specifically, in the function insert_bp_location) that uses
> printf_filtered instead of the ui_* functions to produce the warning
> about a watchpoint that went out of scope.  I think it should use the
> same code that is used by watchpoint_check to produce a similar
> warning.

Sorry, long email, please scan quickly.

Unfortunately, this is going to much harder than I originally thought. In
fact, it looks like the whole thing is broken.

I think part of the problem is the fact that the way the MI output is
generated is hackish. I think it's kind of bad that the MI output is
basically concatenated strings from the rest of GDB.  For instance, 
I think it would be much better if there were data structures available, 
and the breakpoint code set up structures with the information that the 
MI needed. Then later on, the MI could traverse the structure and print 
what it wanted to. In that way, all of the information needed could be 
used to print a message at once.

It seems as if the watchpoint-scope has 2 problems that I can find. The
first is that the manual says,
   Setting a watchpoint on a variable local to a function. GDB will stop
   the program execution twice: first for the variable changing value,
   then for the watchpoint going out of scope.

   ...
   -exec-continue
   ^running
   ^done,reason="watchpoint-scope",wpnum="5",
   frame={func="callee3",args=[{name="strarg",
   value="0x11940 \"A string argument.\""}],
   file="../../../devo/gdb/testsuite/gdb.mi/basics.c",line="18"}
   (gdb)

In the above example, things work fine. However, if you happen to use
-exec-next or -exec-finish, something that by itself produces an
exec-async-output command, then instead of having 2 async output
commands, you get 1, with the data combined. Two example are below, 1
with -exec-continue, and one with -exec-next

   (gdb) 
   -exec-continue
   ^running
   (gdb) 
   ~"Hardware watchpoint 2 deleted because the program has left the block \n"
   ~"in which its expression is valid.\n"
   *stopped,reason="watchpoint-scope",reason="exited",exit-code="02"
   (gdb) 

   (gdb) 
   -exec-next
   ^running
   (gdb) 
   ~"Single stepping until exit from function __libc_start_main, \n"
   ~"which has no line number information.\n"
   ~"Hardware watchpoint 2 deleted because the program has left the block \n"
   ~"in which its expression is valid.\n"
   *stopped,reason="watchpoint-scope",reason="exited-normally"
   (gdb) 

Both of the examples above give 2 reasons for the async command.
So, the MI syntax is correct, but the semantics are completely wrong. I
think it's currently impossible to return 2 async records in 1 MI output
command. This is because in mi-main.c the *stopped is hard coded, as if
there could only be 1 async reason per command. 

There will probably have to be some fundamental changes in order to get
this to work. I think the rest of GDB needs to be able to tell MI
through the uiout that a new async record is ready to be sent, instead
of having the MI hard code it.

Thanks,
Bob Rossi

Re: [mi] watchpoint-scope exec async command

[Bob Rossi]

On Sat, Mar 26, 2005 at 03:24:10PM +0200, Eli Zaretskii wrote:
> > Date: Fri, 25 Mar 2005 11:12:39 -0500
> > From: Bob Rossi <bob@brasko.net>
> > 
> > (gdb)
> > -break-watch param
> > ^done,wpt={number="2",exp="param"}
> > (gdb)
> > -exec-continue
> > ^running
> > (gdb)
> > ~"Hardware watchpoint 2 deleted because the program has left the block \n"
> > ~"in which its expression is valid.\n"
> > *stopped,reason="exited",exit-code="02"
> > (gdb)
> > 
> > Is it just a bug that there is no 'watchpoint-scope' returned?
> 
> I think it's a bug, yes: there's one instance in breakpoint.c
> (specifically, in the function insert_bp_location) that uses
> printf_filtered instead of the ui_* functions to produce the warning
> about a watchpoint that went out of scope.  I think it should use the
> same code that is used by watchpoint_check to produce a similar
> warning.

OK, things have gotten much much worse. I've been studying watchpoints,
and there behavior in GDB. This way, I could understand what the
appropriate MI response should be. I came across this example,

wpscope.c:
   int wpscope ( int param ) {
       return param+1;
   }

   int main(int argc, char **argv){
       return wpscope ( argc );
   }

and ran these commands,

   $ ../gdb ./wpscope
   (gdb) b wpscope
   Breakpoint 1 at 0x8048357: file wpscope.c, line 2.
   (gdb) r
   Starting program: /home/bob/cvs/gdb/original/builddir/gdb/tmp/wpscope

   Breakpoint 1, wpscope (param=1) at wpscope.c:2
   2           return param+1;
   (gdb) watch param
   Hardware watchpoint 2: param
   (gdb) fin
   Run till exit from #0  wpscope (param=1) at wpscope.c:2
   main (argc=1, argv=0xbffffce4) at wpscope.c:7
   7       }
   Value returned is $1 = 2
   (gdb) n
   Hardware watchpoint 2 deleted because the program has left the block
   in which its expression is valid.
   Segmentation fault

The crash only happens sometimes. Although when I run it over and over,
I will eventually get it.

Here is the problem I found,

   (gdb) n
   Hardware watchpoint 2 deleted because the program has left the block
   in which its expression is valid.
   ==26644== Invalid write of size 4
   ==26644==    at 0x80D9B32: insert_bp_location (breakpoint.c:1022)
   ==26644==    by 0x80D9EDA: insert_breakpoints (breakpoint.c:1151)
   ==26644==    by 0x8119620: proceed (infrun.c:774)
   ==26644==    by 0x81165A7: step_1 (infcmd.c:698)
   ==26644==    by 0x8116310: next_command (infcmd.c:595)
   ==26644==    by 0x80BE17A: do_cfunc (cli-decode.c:57)
   ==26644==    by 0x80C0675: cmd_func (cli-decode.c:1636)
   ==26644==    by 0x80863D4: execute_command (top.c:442)
   ==26644==    by 0x812A3EE: command_handler (event-top.c:508)
   ==26644==    by 0x812ABA3: command_line_handler (event-top.c:793)
   ==26644==    by 0x81E8E0B: rl_callback_read_char (callback.c:123)
   ==26644==    by 0x8129B46: rl_callback_read_char_wrapper (event-top.c:174)
   ==26644==    by 0x812A2B0: stdin_event_handler (event-top.c:424)
   ==26644==    by 0x8129280: handle_file_event (event-loop.c:722)
   ==26644==    by 0x8128B54: process_event (event-loop.c:335)
   ==26644==    by 0x8128B9D: gdb_do_one_event (event-loop.c:372)
   ==26644==    by 0x8125E40: catch_errors (exceptions.c:515)
   ==26644==    by 0x80CE73B: tui_command_loop (tui-interp.c:151)
   ==26644==    by 0x812635B: current_interp_command_loop (interps.c:278)
   ==26644==    by 0x807BE62: captured_command_loop (main.c:92)

So, basically, I can't figure out why the breakpoint field
'related_breakpoint' became in valid. I believe it was valid when the
breakpoint was created, since the memset is there. However, at some
point, it must have become invalid ...

Any help would be appreciated.

Thanks,
Bob Rossi

Re: [mi] watchpoint-scope exec async command

[Daniel Jacobowitz]

On Mon, Mar 28, 2005 at 05:56:19PM -0500, Bob Rossi wrote:
>    (gdb) n
>    Hardware watchpoint 2 deleted because the program has left the block
>    in which its expression is valid.
>    Segmentation fault
> 
> The crash only happens sometimes. Although when I run it over and over,
> I will eventually get it.
> 
> Here is the problem I found,
> 
>    (gdb) n
>    Hardware watchpoint 2 deleted because the program has left the block
>    in which its expression is valid.
>    ==26644== Invalid write of size 4
>    ==26644==    at 0x80D9B32: insert_bp_location (breakpoint.c:1022)
>    ==26644==    by 0x80D9EDA: insert_breakpoints (breakpoint.c:1151)
>    ==26644==    by 0x8119620: proceed (infrun.c:774)

I envy you.  I can't get valgrind to work with GDB; once I try to run a
child process, it falls over.

> So, basically, I can't figure out why the breakpoint field
> 'related_breakpoint' became in valid. I believe it was valid when the
> breakpoint was created, since the memset is there. However, at some
> point, it must have become invalid ...

The related breakpoint is supposed to be the scope breakpoint; I can't
see how they could become unpaired.  You may want to run a debugger on
GDB, using a watchpoint to see what changes it?

-- 
Daniel Jacobowitz
CodeSourcery, LLC

Re: [mi] watchpoint-scope exec async command

[Bob Rossi]

On Mon, Mar 28, 2005 at 05:41:01PM -0500, Daniel Jacobowitz wrote:
> On Mon, Mar 28, 2005 at 05:56:19PM -0500, Bob Rossi wrote:
> >    (gdb) n
> >    Hardware watchpoint 2 deleted because the program has left the block
> >    in which its expression is valid.
> >    Segmentation fault
> > 
> > The crash only happens sometimes. Although when I run it over and over,
> > I will eventually get it.
> > 
> > Here is the problem I found,
> > 
> >    (gdb) n
> >    Hardware watchpoint 2 deleted because the program has left the block
> >    in which its expression is valid.
> >    ==26644== Invalid write of size 4
> >    ==26644==    at 0x80D9B32: insert_bp_location (breakpoint.c:1022)
> >    ==26644==    by 0x80D9EDA: insert_breakpoints (breakpoint.c:1151)
> >    ==26644==    by 0x8119620: proceed (infrun.c:774)
> 
> I envy you.  I can't get valgrind to work with GDB; once I try to run a
> child process, it falls over.

Yeah, I was thinking I could script valgrind through the GDB testsuite,
and report any memory leaks/memory corruption. This would probably be a
big task though ...

> > So, basically, I can't figure out why the breakpoint field
> > 'related_breakpoint' became in valid. I believe it was valid when the
> > breakpoint was created, since the memset is there. However, at some
> > point, it must have become invalid ...
> 
> The related breakpoint is supposed to be the scope breakpoint; I can't
> see how they could become unpaired.  You may want to run a debugger on
> GDB, using a watchpoint to see what changes it?

I've tried this, GDB didn't seem to think anyone access'd
b->related_breakpoint between the time it was set in watch_command_1 to
the time that valgrind gives the error in insert_bp_location.

My hunch is that b->related_breakpoint's memory was free'd and never set
to NULL. Is this possible? I don't think a watchpoint would pick that
up, would it?

Any other ideas? I've been debugging, but since I don't have a high
level overview of the data structures, I'm a little handicapped.

Bob Rossi

Re: [mi] watchpoint-scope exec async command

[Daniel Jacobowitz]

On Mon, Mar 28, 2005 at 06:53:10PM -0500, Bob Rossi wrote:
> On Mon, Mar 28, 2005 at 05:41:01PM -0500, Daniel Jacobowitz wrote:
> > The related breakpoint is supposed to be the scope breakpoint; I can't
> > see how they could become unpaired.  You may want to run a debugger on
> > GDB, using a watchpoint to see what changes it?
> 
> I've tried this, GDB didn't seem to think anyone access'd
> b->related_breakpoint between the time it was set in watch_command_1 to
> the time that valgrind gives the error in insert_bp_location.

Check manually if the value has changed?  Does it look like a real
breakpoint?  Have its contents been corrupted?  Etc.

> My hunch is that b->related_breakpoint's memory was free'd and never set
> to NULL. Is this possible? I don't think a watchpoint would pick that
> up, would it?

No, but valgrind would.  Anyway, a breakpoint on delete_breakpoint
would probably catch this also.

I can't imagine how that would happen though.

-- 
Daniel Jacobowitz
CodeSourcery, LLC

Re: [mi] watchpoint-scope exec async command

[Bob Rossi]

> > My hunch is that b->related_breakpoint's memory was free'd and never set
> > to NULL. Is this possible? I don't think a watchpoint would pick that
> > up, would it?
> 
> No, but valgrind would.  Anyway, a breakpoint on delete_breakpoint
> would probably catch this also.
> 
> I can't imagine how that would happen though.

Yeah, this appears to be what is happening. With a little help, we could
probably squash this bug.

breakpoint.c:5761 is where the related_breakpoint is allocated
breakpoint.c:6721 is where the related_breakpoint is deleted
breakpoint.c:1022 is where the problem occurs (just the next sucker to 
read/write the free'd related_breakpoint field)

So, at breakpoint.c:5761 I do,
   (tgdb) p b
   $1 = (struct breakpoint *) 0x83b4878
   (tgdb) p b->related_breakpoint
   $2 = (struct breakpoint *) 0x83b49d0

Then at breakpoint.c:6721, I print the breakpoint to be deleted
   (tgdb) p bpt
   $3 = (struct breakpoint *) 0x83b49d0

This is the related_breakpoint!

at the end of breakpoint_delete I do
   (tgdb) p breakpoint_chain->next->next->next->next->next->next
   $30 = (struct breakpoint *) 0x83b4878

   (tgdb) p breakpoint_chain->next->next->next->next->next->next->related_breakpoint
   $32 = (struct breakpoint *) 0x83b49d0

So the related_breakpoint was deleted, why other breakpoints still point
to it.  So, this shows exactly why GDB is crashing. The questions I have now is,
Should the watchpoint be deleted when it is? Do we have to scan all the 
breakpoints to look at bp->related_breakpoint everytime a breakpoint is deleted?

The backtrace is given below to explain why we are in delete_breakpoint.

Thanks,
Bob Rossi

#0  delete_breakpoint (bpt=0x83b49d0) at ../../src/gdb/breakpoint.c:6895
#1  0x080e2888 in breakpoint_auto_delete (bs=0x8380a50) at ../../src/gdb/breakpoint.c:6702
#2  0x0811c8e3 in normal_stop () at ../../src/gdb/infrun.c:2999
#3  0x081196c2 in proceed (addr=4294967295, siggnal=TARGET_SIGNAL_DEFAULT, step=0) at ../../src/gdb/in
frun.c:827
#4  0x0811741d in finish_command (arg=0x0, from_tty=1) at ../../src/gdb/infcmd.c:1280
#5  0x080be17b in do_cfunc (c=0x8337d60, args=0x0, from_tty=1) at ../../src/gdb/cli/cli-decode.c:57
#6  0x080c0676 in cmd_func (cmd=0x8337d60, args=0x0, from_tty=1) at ../../src/gdb/cli/cli-decode.c:163
6
#7  0x080863d5 in execute_command (p=0x831ff76 "", from_tty=1) at ../../src/gdb/top.c:442
#8  0x0812a3ef in command_handler (command=0x831ff70 "finish") at ../../src/gdb/event-top.c:508
#9  0x0812aba4 in command_line_handler (rl=0x8378f00 "x\2177\b") at ../../src/gdb/event-top.c:793
#10 0x081e8e0c in rl_callback_read_char () at ../../src/readline/callback.c:123
#11 0x08129b47 in rl_callback_read_char_wrapper (client_data=0x0) at ../../src/gdb/event-top.c:174
#12 0x0812a2b1 in stdin_event_handler (error=0, client_data=0x0) at ../../src/gdb/event-top.c:424
#13 0x08129281 in handle_file_event (event_file_desc=0) at ../../src/gdb/event-loop.c:722
#14 0x08128b55 in process_event () at ../../src/gdb/event-loop.c:335
#15 0x08128b9e in gdb_do_one_event (data=0x0) at ../../src/gdb/event-loop.c:372
#16 0x08125e41 in catch_errors (func=0x8128b6a <gdb_do_one_event>, func_args=0x0, errstring=0x8286e7f
"", mask=6) at ../../src/gdb/exceptions.c:515
#17 0x080ce73c in tui_command_loop (data=0x0) at ../../src/gdb/tui/tui-interp.c:151
#18 0x0812635c in current_interp_command_loop () at ../../src/gdb/interps.c:278
#19 0x0807be63 in captured_command_loop (data=0x0) at ../../src/gdb/main.c:92
#20 0x08125e41 in catch_errors (func=0x807be58 <captured_command_loop>, func_args=0x0, errstring=0x826
c4dc "", mask=6) at ../../src/gdb/exceptions.c:515
#21 0x0807cd24 in captured_main (data=0xbffffae0) at ../../src/gdb/main.c:800
#22 0x08125e41 in catch_errors (func=0x807be97 <captured_main>, func_args=0xbffffae0, errstring=0x826c
4dc "", mask=6) at ../../src/gdb/exceptions.c:515
#23 0x0807cd5a in gdb_main (args=0xbffffae0) at ../../src/gdb/main.c:809
#24 0x0807be54 in main (argc=2, argv=0xbffffb54) at ../../src/gdb/gdb.c:35

Re: [mi] watchpoint-scope exec async command

[Daniel Jacobowitz]

On Mon, Mar 28, 2005 at 08:42:03PM -0500, Bob Rossi wrote:
> > > My hunch is that b->related_breakpoint's memory was free'd and never set
> > > to NULL. Is this possible? I don't think a watchpoint would pick that
> > > up, would it?
> > 
> > No, but valgrind would.  Anyway, a breakpoint on delete_breakpoint
> > would probably catch this also.
> > 
> > I can't imagine how that would happen though.
> 
> Yeah, this appears to be what is happening. With a little help, we could
> probably squash this bug.

You need to follow some of these pointers to figure out what's going
on.

> breakpoint.c:5761 is where the related_breakpoint is allocated
> breakpoint.c:6721 is where the related_breakpoint is deleted
> breakpoint.c:1022 is where the problem occurs (just the next sucker to 
> read/write the free'd related_breakpoint field)
> 
> So, at breakpoint.c:5761 I do,
>    (tgdb) p b
>    $1 = (struct breakpoint *) 0x83b4878
>    (tgdb) p b->related_breakpoint
>    $2 = (struct breakpoint *) 0x83b49d0
> 
> Then at breakpoint.c:6721, I print the breakpoint to be deleted
>    (tgdb) p bpt
>    $3 = (struct breakpoint *) 0x83b49d0
> 
> This is the related_breakpoint!
> 
> at the end of breakpoint_delete I do
>    (tgdb) p breakpoint_chain->next->next->next->next->next->next
>    $30 = (struct breakpoint *) 0x83b4878
> 
>    (tgdb) p breakpoint_chain->next->next->next->next->next->next->related_breakpoint
>    $32 = (struct breakpoint *) 0x83b49d0

So far, actually, so good - but is this still true when
breakpoint_auto_delete returns?  Why isn't the other one deleted too?
What's its "disposition" at this point?


-- 
Daniel Jacobowitz
CodeSourcery, LLC

Re: [mi] watchpoint-scope exec async command

[Bob Rossi]

> > breakpoint.c:5761 is where the related_breakpoint is allocated
> > breakpoint.c:6721 is where the related_breakpoint is deleted
> > breakpoint.c:1022 is where the problem occurs (just the next sucker to 
> > read/write the free'd related_breakpoint field)
> > 
> > So, at breakpoint.c:5761 I do,
> >    (tgdb) p b
> >    $1 = (struct breakpoint *) 0x83b4878
> >    (tgdb) p b->related_breakpoint
> >    $2 = (struct breakpoint *) 0x83b49d0
> > 
> > Then at breakpoint.c:6721, I print the breakpoint to be deleted
> >    (tgdb) p bpt
> >    $3 = (struct breakpoint *) 0x83b49d0
> > 
> > This is the related_breakpoint!
> > 
> > at the end of breakpoint_delete I do
> >    (tgdb) p breakpoint_chain->next->next->next->next->next->next
> >    $30 = (struct breakpoint *) 0x83b4878
> > 
> >    (tgdb) p breakpoint_chain->next->next->next->next->next->next->related_breakpoint
> >    $32 = (struct breakpoint *) 0x83b49d0
> 
> So far, actually, so good - but is this still true when
> breakpoint_auto_delete returns?  Why isn't the other one deleted too?
> What's its "disposition" at this point?

Yes, 

(tgdb) p breakpoint_chain->next->next->next->next->next->next
$47 = (struct breakpoint *) 0x83b4878
(tgdb) p breakpoint_chain->next->next->next->next->next->next->disposition
$51 = disp_donttouch
(tgdb) p breakpoint_chain->next->next->next->next->next->next->related_breakpoint
$48 = (struct breakpoint *) 0x83b49d0
(tgdb) p breakpoint_chain->next->next->next->next->next->next->related_breakpoint->disposition
$49 = disp_del

I don't know why this seems OK to you. I must be missing something.
breakpoint_chain->next->next->next->next->next->next->related_breakpoint
was a pointer to a breakpoint that was deleted. Thus, the pointer is
invalid. The reference to the deleted breakpoint (related_breakpoint
field) must be set to NULL somehow. 

It seems that whenever a breakpoint is deleted, all other
breakpoints related_breakpoint field needs to be checked, to see if they
pointed to the data that was just deleted.

Please look at the comments at breakpoint.c:6723 and breakpoint.c:1325.

Thanks,
Bob Rossi

Re: [mi] watchpoint-scope exec async command

[Daniel Jacobowitz]

On Mon, Mar 28, 2005 at 09:49:45PM -0500, Bob Rossi wrote:
> > > breakpoint.c:5761 is where the related_breakpoint is allocated
> > > breakpoint.c:6721 is where the related_breakpoint is deleted
> > > breakpoint.c:1022 is where the problem occurs (just the next sucker to 
> > > read/write the free'd related_breakpoint field)
> > > 
> > > So, at breakpoint.c:5761 I do,
> > >    (tgdb) p b
> > >    $1 = (struct breakpoint *) 0x83b4878
> > >    (tgdb) p b->related_breakpoint
> > >    $2 = (struct breakpoint *) 0x83b49d0
> > > 
> > > Then at breakpoint.c:6721, I print the breakpoint to be deleted
> > >    (tgdb) p bpt
> > >    $3 = (struct breakpoint *) 0x83b49d0
> > > 
> > > This is the related_breakpoint!
> > > 
> > > at the end of breakpoint_delete I do
> > >    (tgdb) p breakpoint_chain->next->next->next->next->next->next
> > >    $30 = (struct breakpoint *) 0x83b4878
> > > 
> > >    (tgdb) p breakpoint_chain->next->next->next->next->next->next->related_breakpoint
> > >    $32 = (struct breakpoint *) 0x83b49d0
> > 
> > So far, actually, so good - but is this still true when
> > breakpoint_auto_delete returns?  Why isn't the other one deleted too?
> > What's its "disposition" at this point?
> 
> Yes, 
> 
> (tgdb) p breakpoint_chain->next->next->next->next->next->next
> $47 = (struct breakpoint *) 0x83b4878
> (tgdb) p breakpoint_chain->next->next->next->next->next->next->disposition
> $51 = disp_donttouch
> (tgdb) p breakpoint_chain->next->next->next->next->next->next->related_breakpoint
> $48 = (struct breakpoint *) 0x83b49d0
> (tgdb) p breakpoint_chain->next->next->next->next->next->next->related_breakpoint->disposition
> $49 = disp_del
> 
> I don't know why this seems OK to you. I must be missing something.
> breakpoint_chain->next->next->next->next->next->next->related_breakpoint
> was a pointer to a breakpoint that was deleted. Thus, the pointer is
> invalid. The reference to the deleted breakpoint (related_breakpoint
> field) must be set to NULL somehow. 

Both of them should have the same disposition at this point.  That's
why it's OK.  They should always be deleted together; they should have
had disp_del_at_next_stop.  We are deleting the scope breakpoint
because the watchpoint has gone out of scope; how did we fail to delete
the watchpoint too?

The answer seems to be that we use disp_del_at_next_stop if we hit the
_watchpoint_, but not if we hit the related breakpoint.  When we delete
it we ought to be deleting its related breakpoint too (they point to
each other).  But we don't.  The only things we ever do with
related_breakpoints are set their dispositions.

That's presumably the bug.  Why it never hit before, I have no clue.
Eli, you know a lot about the watchpoint mechanism; do you know how
this is supposed to work?

> It seems that whenever a breakpoint is deleted, all other
> breakpoints related_breakpoint field needs to be checked, to see if they
> pointed to the data that was just deleted.
> 
> Please look at the comments at breakpoint.c:6723 and breakpoint.c:1325.

The comment at 6723 doesn't apply to this case, because that's bpstats
dangling a pointer; the breakpoint chain should never do this.

-- 
Daniel Jacobowitz
CodeSourcery, LLC

Re: [mi] watchpoint-scope exec async command

[Eli Zaretskii]

[Sorry for a late response: I needed to refresh my memory of how the
related code works.]

> Date: Mon, 28 Mar 2005 21:01:23 -0500
> From: Daniel Jacobowitz <drow@false.org>
> Cc: Eli Zaretskii <eliz@gnu.org>
> 
> We are deleting the scope breakpoint because the watchpoint has gone
> out of scope

Are you sure?  Bob didn't show enough backtraces for me to be sure,
but the one backtrace he did show seems to tell that we are deleting
the scope breakpoint because it's marked either disp_del or
disp_del_at_next_stop (and I'd like to know which of these two
possibilities actually happens there), not because the watchpoint has
gone out of scope (or at least not directly because of that).

Bob, could you please show the backtraces at every place where you
until now only posted the result of printing variables?  Also, please
put a breakpoint where GDB says a watchpoint has gone out of scope,
and show a transcript of a session where we could see the sequence of
the events: the watchpoint and/or scope breakpoint is hit, marked to
be deleted ASAP, then actually deleted.

> how did we fail to delete the watchpoint too?
> 
> The answer seems to be that we use disp_del_at_next_stop if we hit the
> _watchpoint_, but not if we hit the related breakpoint.  When we delete
> it we ought to be deleting its related breakpoint too (they point to
> each other).  But we don't.  The only things we ever do with
> related_breakpoints are set their dispositions.

Sorry, I'm not sure I understand correctly what you are saying,
because of the ambiguous use of "it" and "its".  This sentence is the
most ambiguous for me:

> When we delete it we ought to be deleting its related breakpoint too
> (they point to each other).

Who is "it" here?  If it's the watchpoint, then I don't understand
your reasoning, because if the scope breakpoint rather than the
watchpoint itself triggered, we don't delete anything.

> That's presumably the bug.

Probably, but I'd like to see more information (from Bob) to be sure.

Let's not forget that software watchpoints also share some (most?) of
the code used for hardware data-write watchpoints.  I'm quite sure
that the scope breakpoint was invented for software watchpoints,
because hardware watchpoints simply don't need it: they incur no
overhead if they are not deleted, as long as they are not hit; and
once they are hit and we find them to be out of scope, we do TRT: mark
both the watchpoint and its related breakpoint for deletion at next
stop.  (I suspect that we don't delete them right away due to some
peculiarity of software watchpoints that are implemented by
single-stepping the inferior.)

Bob, could you also try the same program with a software watchpoint,
and see if you hit the same bug or not?  If the behavior is different,
please show where the GDB execution thread goes a different way and
how it avoids the crash.  (Forcing software watchpoints might not be
easy.  One way, on a x86 machine, is to watch a region that is more
than 16 bytes long.  Another way is to build a hacked version of GDB
that uses only software watchpoints.)

> > Please look at the comments at breakpoint.c:6723 and breakpoint.c:1325.
> 
> The comment at 6723 doesn't apply to this case, because that's bpstats
> dangling a pointer; the breakpoint chain should never do this.

??? My interpretation of that comment is that a breakpoint/watchpoint
that was already deleted should have its type set to bp_none, and in
that case we don't delete it again.  What am I missing?

Re: [mi] watchpoint-scope exec async command

[Mark Kettenis]

   Date: Tue, 29 Mar 2005 23:29:48 +0200
   From: "Eli Zaretskii" <eliz@gnu.org>

   Bob, could you also try the same program with a software watchpoint,
   and see if you hit the same bug or not?  If the behavior is different,
   please show where the GDB execution thread goes a different way and
   how it avoids the crash.  (Forcing software watchpoints might not be
   easy.  One way, on a x86 machine, is to watch a region that is more
   than 16 bytes long.  Another way is to build a hacked version of GDB
   that uses only software watchpoints.)

How about "set can-use-hw-watchpoints 0"? ;-)

Mark

Re: [mi] watchpoint-scope exec async command

[Bob Rossi]

On Tue, Mar 29, 2005 at 11:39:44PM +0200, Mark Kettenis wrote:
>    Date: Tue, 29 Mar 2005 23:29:48 +0200
>    From: "Eli Zaretskii" <eliz@gnu.org>
> 
>    Bob, could you also try the same program with a software watchpoint,
>    and see if you hit the same bug or not?  If the behavior is different,
>    please show where the GDB execution thread goes a different way and
>    how it avoids the crash.  (Forcing software watchpoints might not be
>    easy.  One way, on a x86 machine, is to watch a region that is more
>    than 16 bytes long.  Another way is to build a hacked version of GDB
>    that uses only software watchpoints.)
> 
> How about "set can-use-hw-watchpoints 0"? ;-)

Seriously, thanks! I was just digesting that paragraph.

Bob Rossi

Re: [mi] watchpoint-scope exec async command

[Eli Zaretskii]

> Date: Tue, 29 Mar 2005 23:39:44 +0200 (CEST)
> From: Mark Kettenis <mark.kettenis@xs4all.nl>
> CC: gdb@sources.redhat.com
> 
> How about "set can-use-hw-watchpoints 0"? ;-)

Undocumented commands don't exist for me. ;-)

Ts-ts-ts, this command was introduced in 1998, and was not mentioned
in the manual since then.  Sigh.

Re: [mi] watchpoint-scope exec async command

[Daniel Jacobowitz]

On Tue, Mar 29, 2005 at 11:29:48PM +0200, Eli Zaretskii wrote:
> > how did we fail to delete the watchpoint too?
> > 
> > The answer seems to be that we use disp_del_at_next_stop if we hit the
> > _watchpoint_, but not if we hit the related breakpoint.  When we delete
> > it we ought to be deleting its related breakpoint too (they point to
> > each other).  But we don't.  The only things we ever do with
> > related_breakpoints are set their dispositions.
> 
> Sorry, I'm not sure I understand correctly what you are saying,
> because of the ambiguous use of "it" and "its".  This sentence is the
> most ambiguous for me:
> 
> > When we delete it we ought to be deleting its related breakpoint too
> > (they point to each other).
> 
> Who is "it" here?  If it's the watchpoint, then I don't understand
> your reasoning, because if the scope breakpoint rather than the
> watchpoint itself triggered, we don't delete anything.

"it" was intended to be the scope breakpoint.  Sorry, I had to read
that three times to work out what I meant, and it was only yesterday.

If the scope breakpoint triggers, we delete it.  From watch_command_1:
          /* Automatically delete the breakpoint when it hits.  */
          scope_breakpoint->disposition = disp_del;

That's what's happening in this case.  Then, shortly thereafter, the
watchpoint triggers.  That's when we detect that it has gone out of
scope, and set it to delete at next stop; and we crash, because we
already deleted the scope breakpoint when it was hit.

> > > Please look at the comments at breakpoint.c:6723 and breakpoint.c:1325.
> > 
> > The comment at 6723 doesn't apply to this case, because that's bpstats
> > dangling a pointer; the breakpoint chain should never do this.
> 
> ??? My interpretation of that comment is that a breakpoint/watchpoint
> that was already deleted should have its type set to bp_none, and in
> that case we don't delete it again.  What am I missing?

Generally, when we delete a breakpoint, we free it; so it doesn't
matter what its type gets set to.

  /* On the chance that someone will soon try again to delete this same
     bp, we mark it as deleted before freeing its storage. */
  bpt->type = bp_none;

  xfree (bpt->loc);
  xfree (bpt);

All the code surrounding bp_none is horribly bogus; we're walking freed
memory if we hit it.  I don't know if it still triggers today; probably
it does in some cases, and no one's been brave enough to fix them.

-- 
Daniel Jacobowitz
CodeSourcery, LLC

Re: [mi] watchpoint-scope exec async command

[Eli Zaretskii]

> Date: Tue, 29 Mar 2005 16:44:14 -0500
> From: Daniel Jacobowitz <drow@false.org>
> Cc: GDB <gdb@sources.redhat.com>
> 
> If the scope breakpoint triggers, we delete it.  From watch_command_1:
>           /* Automatically delete the breakpoint when it hits.  */
>           scope_breakpoint->disposition = disp_del;
> 
> That's what's happening in this case.  Then, shortly thereafter, the
> watchpoint triggers.  That's when we detect that it has gone out of
> scope, and set it to delete at next stop; and we crash, because we
> already deleted the scope breakpoint when it was hit.

I hoped to see this from Bob's tracebacks, but I only saw the first
part of what you describe: that the scope breakpoint is being deleted
after it triggers (not _when_, _after_: it is deleted by
breakpoint_auto_delete).

Assuming that the watchpoint triggers after that, it is marked as
disp_del_at_next_stop, so it would be slated for deletion by the same
breakpoint_auto_delete function when it is called shortly after.  This
is the part that I didn't see in Bob's session.  I will assume that
things indeed happen like you say: that when we try to delete that
watchpoint, we crash when we access its scope breakpoint, which was
already deleted and freed.

I think we have the following alternatives to fix this.  First, we
could stop using scope breakpoints for hardware-assisted watchpoints.
(The scope breakpoint is not needed in this case, since they don't
slow down the executable, and because we have an independent facility
to detect that a hardware watchpoint went out of scope: that is the
code run by insert_bp_location and watchpoint_check which prints a
warning about the fact that the watchpoint went out of scope.)
Software watchpoints do need the scope breakpoint (to stop
single-stepping the inferior once the watchpoint goes out of scope),
and in that case Bob's testing demonstrates that the scope breakpoint
machinery works correctly.  So we need to continue using scope
breakpoints for software watchpoints alone.

If we don't arrange a scope breakpoint for a hardware watchpoint, we
won't hit the problem Bob reported.

The second alternative is to treat scope breakpoints specially in
breakpoint_auto_delete: when we see a scope breakpoint that is marked
for deletion, we will have to find its watchpoint, and if that
watchpoint is a hardware watchpoint, we will have to delete that
watchpoint as well.

I like the first alternative better, since it seems cleaner.

As an aside, I'd ask Bob to run the same test program, but this time
use awatch instead of watch command.  I'd be interested to hear if the
same problems (i.e. memory write into a freed block reported by
valgrind and an occasional crash) happen in that case as well.  The
reason that I'm asking this is that we handle watch and rwatch/awatch
slightly differently, since the code that handles watch is run for
both software and hardware watchpoints.

> All the code surrounding bp_none is horribly bogus; we're walking freed
> memory if we hit it.

That's true, it sounds as if someone was not sure what our code does,
so he/she went over-defensive.

> I don't know if it still triggers today

I think Bob's testing shows that it does, for the software
watchpoints.

Did I help resolving this issue?

Re: [mi] watchpoint-scope exec async command

[Bob Rossi]

On Wed, Mar 30, 2005 at 10:06:33PM +0200, Eli Zaretskii wrote:
> > Date: Tue, 29 Mar 2005 16:44:14 -0500
> > From: Daniel Jacobowitz <drow@false.org>
> > Cc: GDB <gdb@sources.redhat.com>
> > 
> > If the scope breakpoint triggers, we delete it.  From watch_command_1:
> >           /* Automatically delete the breakpoint when it hits.  */
> >           scope_breakpoint->disposition = disp_del;
> > 
> > That's what's happening in this case.  Then, shortly thereafter, the
> > watchpoint triggers.  That's when we detect that it has gone out of
> > scope, and set it to delete at next stop; and we crash, because we
> > already deleted the scope breakpoint when it was hit.
> 
> I hoped to see this from Bob's tracebacks, but I only saw the first
> part of what you describe: that the scope breakpoint is being deleted
> after it triggers (not _when_, _after_: it is deleted by
> breakpoint_auto_delete).
> 
> Assuming that the watchpoint triggers after that, it is marked as
> disp_del_at_next_stop, so it would be slated for deletion by the same
> breakpoint_auto_delete function when it is called shortly after.  This
> is the part that I didn't see in Bob's session.  I will assume that
> things indeed happen like you say: that when we try to delete that
> watchpoint, we crash when we access its scope breakpoint, which was
> already deleted and freed.
> 
> I think we have the following alternatives to fix this.  First, we
> could stop using scope breakpoints for hardware-assisted watchpoints.
> (The scope breakpoint is not needed in this case, since they don't
> slow down the executable, and because we have an independent facility
> to detect that a hardware watchpoint went out of scope: that is the
> code run by insert_bp_location and watchpoint_check which prints a
> warning about the fact that the watchpoint went out of scope.)
> Software watchpoints do need the scope breakpoint (to stop
> single-stepping the inferior once the watchpoint goes out of scope),
> and in that case Bob's testing demonstrates that the scope breakpoint
> machinery works correctly.  So we need to continue using scope
> breakpoints for software watchpoints alone.
> 
> If we don't arrange a scope breakpoint for a hardware watchpoint, we
> won't hit the problem Bob reported.
> 
> The second alternative is to treat scope breakpoints specially in
> breakpoint_auto_delete: when we see a scope breakpoint that is marked
> for deletion, we will have to find its watchpoint, and if that
> watchpoint is a hardware watchpoint, we will have to delete that
> watchpoint as well.
> 
> I like the first alternative better, since it seems cleaner.

The second alternative was my initial idea. However, I was just
guessing, since I really know nothing about the code. The first 
approach seems good, I was just wondering if that would slow things
down? Aren't hardware watchpoints must faster than software? 

If desired, I'd be interested in looking into either of these 2 fixes.
However, I'll need a small amount of hand holding, so it might be faster
for someone else to do it ...

> As an aside, I'd ask Bob to run the same test program, but this time
> use awatch instead of watch command.  I'd be interested to hear if the
> same problems (i.e. memory write into a freed block reported by
> valgrind and an occasional crash) happen in that case as well.  The
> reason that I'm asking this is that we handle watch and rwatch/awatch
> slightly differently, since the code that handles watch is run for
> both software and hardware watchpoints.

Funny you ask. When using hardware watchpoints, both rwatch and awatch
result in the same bad behavior. However, when using software
watchpoints,

(gdb) rwatch param
Expression cannot be implemented with read/access watchpoint.
(gdb) awatch param
Expression cannot be implemented with read/access watchpoint.
(gdb) watch param
Watchpoint 2: param
(gdb) 

both rwatch and awatch are refused by GDB.

> > I don't know if it still triggers today
> 
> I think Bob's testing shows that it does, for the software
> watchpoints.
> 
> Did I help resolving this issue?

But of course!

Thanks for all the help,
Bob Rossi

Re: [mi] watchpoint-scope exec async command

[Eli Zaretskii]

> Date: Wed, 30 Mar 2005 20:47:49 -0500
> From: Bob Rossi <bob@brasko.net>
> Cc: GDB <gdb@sources.redhat.com>
> 
> The first approach seems good, I was just wondering if that would
> slow things down? Aren't hardware watchpoints must faster than
> software?

With hardware watchpoints. the inferior runs at its normal speed.  So
how would it slow down things if we leave the watchpoint in place in
that case?

> If desired, I'd be interested in looking into either of these 2 fixes.
> However, I'll need a small amount of hand holding, so it might be faster
> for someone else to do it ...

If no one does, feel free to ask questions.

> Funny you ask. When using hardware watchpoints, both rwatch and awatch
> result in the same bad behavior. However, when using software
> watchpoints,
> 
> (gdb) rwatch param
> Expression cannot be implemented with read/access watchpoint.
> (gdb) awatch param
> Expression cannot be implemented with read/access watchpoint.
> (gdb) watch param
> Watchpoint 2: param
> (gdb) 
> 
> both rwatch and awatch are refused by GDB.

rwatch and awatch can only be set with hardware assistance, so the
refusal is expected.

Re: [mi] watchpoint-scope exec async command

[Bob Rossi]

On Thu, Mar 31, 2005 at 06:39:23AM +0200, Eli Zaretskii wrote:
> > Date: Wed, 30 Mar 2005 20:47:49 -0500
> > From: Bob Rossi <bob@brasko.net>
> > Cc: GDB <gdb@sources.redhat.com>
> > 
> > The first approach seems good, I was just wondering if that would
> > slow things down? Aren't hardware watchpoints must faster than
> > software?
> 
> With hardware watchpoints. the inferior runs at its normal speed.  So
> how would it slow down things if we leave the watchpoint in place in
> that case?

Sorry, I re-read what you wrote and understand it a little better now.

I do have another question though. If GDB has another mecanism to
determine when hardware watchpoints go out of scope, why does it add the
scope watchpoint at all? Is it added and then never used? or does  it
not serve a function at all in this case?

Thanks,
Bob Rossi

Re: [mi] watchpoint-scope exec async command

[Eli Zaretskii]

> Date: Thu, 31 Mar 2005 15:58:26 -0500
> From: Bob Rossi <bob@brasko.net>
> Cc: GDB <gdb@sources.redhat.com>
> 
> I do have another question though. If GDB has another mecanism to
> determine when hardware watchpoints go out of scope, why does it add the
> scope watchpoint at all?

I think that the scope breakpoint was introduced when software
watchpoints were coded.  Software watchpoints do need the scope
breakpoint, and as you demonstrated, there's no problem in that case.
That is why I think we should simply not use the scope breakpoint for
hardware watchpoints.

> Is it added and then never used?

Well, your research shows that they _are_ used.  We have two separate
mechanisms that serve the same purpose.

> or does it not serve a function at all in this case?

You mean, except for crashing GDB? ;-)

Anyway, note that the warning about the watchpoint going out of scope
comes from the code that independently detects this for hardware
watchpoints, with no help from the scope breakpoint.  Which perhaps
means that we need to add a similar warning to the code that handles
the case that the scope breakpoint was hit, so that software
watchpoints will also produce such a warning.

Re: [mi] watchpoint-scope exec async command

[Daniel Jacobowitz]

On Fri, Apr 01, 2005 at 11:06:05AM +0200, Eli Zaretskii wrote:
> > Date: Thu, 31 Mar 2005 15:58:26 -0500
> > From: Bob Rossi <bob@brasko.net>
> > Cc: GDB <gdb@sources.redhat.com>
> > 
> > I do have another question though. If GDB has another mecanism to
> > determine when hardware watchpoints go out of scope, why does it add the
> > scope watchpoint at all?
> 
> I think that the scope breakpoint was introduced when software
> watchpoints were coded.  Software watchpoints do need the scope
> breakpoint, and as you demonstrated, there's no problem in that case.
> That is why I think we should simply not use the scope breakpoint for
> hardware watchpoints.

Actually, I don't think software watchpoints need it at all.  A
software watchpoint is implemented primarily by single-stepping the
inferior, right?  Well, after every single-step we know whether or not
the breakpoint is still in scope...

-- 
Daniel Jacobowitz
CodeSourcery, LLC

Re: [mi] watchpoint-scope exec async command

[Eli Zaretskii]

> Date: Fri, 1 Apr 2005 09:11:05 -0500
> From: Daniel Jacobowitz <drow@false.org>
> Cc: GDB <gdb@sources.redhat.com>
> 
> Actually, I don't think software watchpoints need it at all.  A
> software watchpoint is implemented primarily by single-stepping the
> inferior, right?  Well, after every single-step we know whether or not
> the breakpoint is still in scope...

That's true, but running the code that checks whether the watchpoint
is still in scope after each instruction would slow down GDB even
more, while the scope breakpoint doesn't add any slowdown.

Of course, this is all based on speculative arguments, at least from
my side, so it could be 100% wrong.  If someone who reads this knows
for a fact why scope breakpoints were introduced, please speak up.

Re: [mi] watchpoint-scope exec async command

[Bob Rossi]

On Sat, Apr 02, 2005 at 12:50:14PM +0300, Eli Zaretskii wrote:
> > Date: Fri, 1 Apr 2005 09:11:05 -0500
> > From: Daniel Jacobowitz <drow@false.org>
> > Cc: GDB <gdb@sources.redhat.com>
> > 
> > Actually, I don't think software watchpoints need it at all.  A
> > software watchpoint is implemented primarily by single-stepping the
> > inferior, right?  Well, after every single-step we know whether or not
> > the breakpoint is still in scope...
> 
> That's true, but running the code that checks whether the watchpoint
> is still in scope after each instruction would slow down GDB even
> more, while the scope breakpoint doesn't add any slowdown.
> 
> Of course, this is all based on speculative arguments, at least from
> my side, so it could be 100% wrong.  If someone who reads this knows
> for a fact why scope breakpoints were introduced, please speak up.

Please forgive me for not being able to contribute to the solution to
this problem. 

Eli or Daneil, if either of you can decide which solution is correct,
I'll look into fixing the problem. It just so happens that this bug
doesn't really effect me in any way. I just happened to notice it when
trying to enumerate many of the possibilities in GDB/MI.

Would it be good if I entered a bug report? or should i just forget
about this?

Bob Rossi

Re: [mi] watchpoint-scope exec async command

[Eli Zaretskii]

> Date: Tue, 5 Apr 2005 22:12:59 -0400
> From: Bob Rossi <bob@brasko.net>
> Cc: GDB <gdb@sources.redhat.com>
> 
> Eli or Daneil, if either of you can decide which solution is correct,
> I'll look into fixing the problem. It just so happens that this bug
> doesn't really effect me in any way. I just happened to notice it when
> trying to enumerate many of the possibilities in GDB/MI.
> 
> Would it be good if I entered a bug report? or should i just forget
> about this?

It's best if you picked one of the two alternatives and implemented
it.  I can approve changes to breakpoint.c, so if the patch is good,
it will go in.

TIA

Re: [mi] watchpoint-scope exec async command

[Daniel Jacobowitz]

On Wed, Mar 30, 2005 at 10:06:33PM +0200, Eli Zaretskii wrote:
> > Date: Tue, 29 Mar 2005 16:44:14 -0500
> > From: Daniel Jacobowitz <drow@false.org>
> > Cc: GDB <gdb@sources.redhat.com>
> > 
> > If the scope breakpoint triggers, we delete it.  From watch_command_1:
> >           /* Automatically delete the breakpoint when it hits.  */
> >           scope_breakpoint->disposition = disp_del;
> > 
> > That's what's happening in this case.  Then, shortly thereafter, the
> > watchpoint triggers.  That's when we detect that it has gone out of
> > scope, and set it to delete at next stop; and we crash, because we
> > already deleted the scope breakpoint when it was hit.
> 
> I hoped to see this from Bob's tracebacks, but I only saw the first
> part of what you describe: that the scope breakpoint is being deleted
> after it triggers (not _when_, _after_: it is deleted by
> breakpoint_auto_delete).

Immediately after, though.  We're in the first loop in
breakpoint_auto_delete, for breakpoints in the current bpstate.

> Assuming that the watchpoint triggers after that, it is marked as
> disp_del_at_next_stop, so it would be slated for deletion by the same
> breakpoint_auto_delete function when it is called shortly after.  This
> is the part that I didn't see in Bob's session.  I will assume that
> things indeed happen like you say: that when we try to delete that
> watchpoint, we crash when we access its scope breakpoint, which was
> already deleted and freed.

No, we're crashing earlier than that.  This was in one of Bob's earlier
messages; we crash here:

   1021           if (bpt->owner->related_breakpoint)
   1022             bpt->owner->related_breakpoint->disposition = disp_del_at_next_stop;
   1023           bpt->owner->disposition = disp_del_at_next_stop;

Since the related breakpoint has already been freed... well, accessing
it isn't so clever.

> I think we have the following alternatives to fix this.  First, we
> could stop using scope breakpoints for hardware-assisted watchpoints.
> (The scope breakpoint is not needed in this case, since they don't
> slow down the executable, and because we have an independent facility
> to detect that a hardware watchpoint went out of scope: that is the
> code run by insert_bp_location and watchpoint_check which prints a
> warning about the fact that the watchpoint went out of scope.)
> Software watchpoints do need the scope breakpoint (to stop
> single-stepping the inferior once the watchpoint goes out of scope),
> and in that case Bob's testing demonstrates that the scope breakpoint
> machinery works correctly.  So we need to continue using scope
> breakpoints for software watchpoints alone.
> 
> If we don't arrange a scope breakpoint for a hardware watchpoint, we
> won't hit the problem Bob reported.

I think this would be pretty tricky.  We would have to recognize that
if the next thing to trigger is the watchpoint, it doesn't "count".
There's machinery for this, but it's pretty tricky.

> The second alternative is to treat scope breakpoints specially in
> breakpoint_auto_delete: when we see a scope breakpoint that is marked
> for deletion, we will have to find its watchpoint, and if that
> watchpoint is a hardware watchpoint, we will have to delete that
> watchpoint as well.
> 
> I like the first alternative better, since it seems cleaner.

The second seems marginally cleaner to me.  Even better would be
deleting the software watchpoint at the same time, i.e. decreasing
divergence between the two; do you see any reason why this wouldn't
work out?

-- 
Daniel Jacobowitz
CodeSourcery, LLC

Re: [mi] watchpoint-scope exec async command

[Eli Zaretskii]

> Date: Wed, 30 Mar 2005 21:33:07 -0500
> From: Daniel Jacobowitz <drow@false.org>
> Cc: GDB <gdb@sources.redhat.com>
> 
> No, we're crashing earlier than that.  This was in one of Bob's earlier
> messages; we crash here:
> 
>    1021           if (bpt->owner->related_breakpoint)
>    1022             bpt->owner->related_breakpoint->disposition = disp_del_at_next_stop;
>    1023           bpt->owner->disposition = disp_del_at_next_stop;

Right.

> > If we don't arrange a scope breakpoint for a hardware watchpoint, we
> > won't hit the problem Bob reported.
> 
> I think this would be pretty tricky.  We would have to recognize that
> if the next thing to trigger is the watchpoint, it doesn't "count".

Sorry, I'm not following.  What I meant was this: when we _create_ the
watchpoint, if it's a hardware-assisted watchpoint, we should simply
not arrange a scope breakpoint for it.  How is that tricky, and why
would we need to know that the next thing to trigger is the
watchpoint?

> The second seems marginally cleaner to me.

I wouldn't mind the second alternative too much.

> Even better would be deleting the software watchpoint at the same
> time

How is this different from what I said?  I said:

> The second alternative is to treat scope breakpoints specially in
> breakpoint_auto_delete: when we see a scope breakpoint that is marked
> for deletion, we will have to find its watchpoint, and if that
> watchpoint is a hardware watchpoint, we will have to delete that
> watchpoint as well.

Re: [mi] watchpoint-scope exec async command

[Daniel Jacobowitz]

On Thu, Mar 31, 2005 at 06:45:06AM +0200, Eli Zaretskii wrote:
> > Date: Wed, 30 Mar 2005 21:33:07 -0500
> > From: Daniel Jacobowitz <drow@false.org>
> > Cc: GDB <gdb@sources.redhat.com>
> > 
> > No, we're crashing earlier than that.  This was in one of Bob's earlier
> > messages; we crash here:
> > 
> >    1021           if (bpt->owner->related_breakpoint)
> >    1022             bpt->owner->related_breakpoint->disposition = disp_del_at_next_stop;
> >    1023           bpt->owner->disposition = disp_del_at_next_stop;
> 
> Right.
> 
> > > If we don't arrange a scope breakpoint for a hardware watchpoint, we
> > > won't hit the problem Bob reported.
> > 
> > I think this would be pretty tricky.  We would have to recognize that
> > if the next thing to trigger is the watchpoint, it doesn't "count".
> 
> Sorry, I'm not following.  What I meant was this: when we _create_ the
> watchpoint, if it's a hardware-assisted watchpoint, we should simply
> not arrange a scope breakpoint for it.  How is that tricky, and why
> would we need to know that the next thing to trigger is the
> watchpoint?

Creating it is easy.  Handling it when it goes out of scope, however,
is harder - if the next thing to trigger a stop is the watchpoint, and
we discover there that it has gone out of scope.  I don't know, this
may just work.  But it feels more complicated to me, for whatever
that's worth.  I like the idea of keeping the multiple watchpoint types
as similar as possible.

> > Even better would be deleting the software watchpoint at the same
> > time
> 
> How is this different from what I said?  I said:
> 
> > The second alternative is to treat scope breakpoints specially in
> > breakpoint_auto_delete: when we see a scope breakpoint that is marked
> > for deletion, we will have to find its watchpoint, and if that
> > watchpoint is a hardware watchpoint, we will have to delete that
> > watchpoint as well.

You said "if that watchpoint is a hardware watchpoint".  I'm just
suggesting treating software watchpoints the same way.

-- 
Daniel Jacobowitz
CodeSourcery, LLC

Re: [mi] watchpoint-scope exec async command

[Eli Zaretskii]

> Date: Wed, 30 Mar 2005 23:54:26 -0500
> From: Daniel Jacobowitz <drow@false.org>
> Cc: GDB <gdb@sources.redhat.com>
> 
> Creating it is easy.  Handling it when it goes out of scope, however,
> is harder - if the next thing to trigger a stop is the watchpoint, and
> we discover there that it has gone out of scope.  I don't know, this
> may just work.

I think it already works -- see watchpoint_check and insert_bp_location.

> But it feels more complicated to me, for whatever that's worth.

I don't mind using the second alternative.

> > > The second alternative is to treat scope breakpoints specially in
> > > breakpoint_auto_delete: when we see a scope breakpoint that is marked
> > > for deletion, we will have to find its watchpoint, and if that
> > > watchpoint is a hardware watchpoint, we will have to delete that
> > > watchpoint as well.
> 
> You said "if that watchpoint is a hardware watchpoint".  I'm just
> suggesting treating software watchpoints the same way.

For that, someone will have either to explain why software watchpoints
don't cause the crash with the current code, or make very sure that
making that change for software watchpoints doesn't break them.

Re: [mi] watchpoint-scope exec async command

[Bob Rossi]

On Tue, Mar 29, 2005 at 11:29:48PM +0200, Eli Zaretskii wrote:
> [Sorry for a late response: I needed to refresh my memory of how the
> related code works.]

Are you serious? That was like the speed of light compared to what I'm
used to.
> Let's not forget that software watchpoints also share some (most?) of
> the code used for hardware data-write watchpoints.  I'm quite sure
> that the scope breakpoint was invented for software watchpoints,
> because hardware watchpoints simply don't need it: they incur no
> overhead if they are not deleted, as long as they are not hit; and
> once they are hit and we find them to be out of scope, we do TRT: mark
> both the watchpoint and its related breakpoint for deletion at next
> stop.  (I suspect that we don't delete them right away due to some
> peculiarity of software watchpoints that are implemented by
> single-stepping the inferior.)
> 
> Bob, could you also try the same program with a software watchpoint,
> and see if you hit the same bug or not?  If the behavior is different,
> please show where the GDB execution thread goes a different way and
> how it avoids the crash.  (Forcing software watchpoints might not be
> easy.  One way, on a x86 machine, is to watch a region that is more
> than 16 bytes long.  Another way is to build a hacked version of GDB
> that uses only software watchpoints.)

It'll be a little bit before I can figure out how to present all the
information that you need. However, in case it clear's things up for
you, here's some more information. (this info makes me even *more*
confused.)

When running with software breakpoints, the crash does not happen, and
valgrind is happy. Now, what confuses me is that the memory for the
related_breakpoint is still deleted in delete_breakpoint and when we 
set the disposition of the deleted "related_breakpoint" at 
breakpoint.c:1022(insert_bp_location), things seem to work OK.

I don't understand this, but i will present you with the information you
asked for as soon as possible.

Bob Rossi

Re: [mi] watchpoint-scope exec async command

[Eli Zaretskii]

> Date: Tue, 29 Mar 2005 19:27:48 -0500
> From: Bob Rossi <bob@brasko.net>
> Cc: GDB <gdb@sources.redhat.com>
> 
> On Tue, Mar 29, 2005 at 11:29:48PM +0200, Eli Zaretskii wrote:
> > [Sorry for a late response: I needed to refresh my memory of how the
> > related code works.]
> 
> Are you serious? That was like the speed of light compared to what I'm
> used to.

I have different standards.

> When running with software breakpoints, the crash does not happen, and
> valgrind is happy. Now, what confuses me is that the memory for the
> related_breakpoint is still deleted in delete_breakpoint and when we 
> set the disposition of the deleted "related_breakpoint" at 
> breakpoint.c:1022(insert_bp_location), things seem to work OK.

I suspect that, since software watchpoints are imlpemented by
single-stepping the inferior, the order of deletion of the watchpoint
and its associated scope breakpoint is different (because the inferior
stops after each instruction).

More in the evening, after I read through the other info you sent.

Thanks.

Re: [mi] watchpoint-scope exec async command

[Bob Rossi]

> > We are deleting the scope breakpoint because the watchpoint has gone
> > out of scope
> 
> Are you sure?  Bob didn't show enough backtraces for me to be sure,
> but the one backtrace he did show seems to tell that we are deleting
> the scope breakpoint because it's marked either disp_del or
> disp_del_at_next_stop (and I'd like to know which of these two
> possibilities actually happens there), not because the watchpoint has
> gone out of scope (or at least not directly because of that).
> 
> Bob, could you please show the backtraces at every place where you
> until now only posted the result of printing variables?  Also, please
> put a breakpoint where GDB says a watchpoint has gone out of scope,
> and show a transcript of a session where we could see the sequence of
> the events: the watchpoint and/or scope breakpoint is hit, marked to
> be deleted ASAP, then actually deleted.

OK, this is going to be confusing. To recap, this is the source code
with the problem,

Just to recap, the bug happens like this,

    wpscope.c: 
       int wpscope ( int param ) {
           return param+1;
       }

       int main(int argc, char **argv){
           return wpscope ( argc );
       }

    and ran these commands,

       $ ../gdb ./wpscope
       (gdb) b wpscope
       Breakpoint 1 at 0x8048357: file wpscope.c, line 2.
       (gdb) r
       Starting program: /home/bob/cvs/gdb/original/builddir/gdb/tmp/wpscope

       Breakpoint 1, wpscope (param=1) at wpscope.c:2
       2           return param+1;
       (gdb) watch param
       Hardware watchpoint 2: param
       (gdb) fin
       Run till exit from #0  wpscope (param=1) at wpscope.c:2
       main (argc=1, argv=0xbffffce4) at wpscope.c:7
       7       }
       Value returned is $1 = 2
       (gdb) n
       Hardware watchpoint 2 deleted because the program has left the block
       in which its expression is valid.
       Segmentation fault

Now, I've run gdb twice, once with hardware watchpoints and then again
with software watchpoints.

Most of the transcript between the hardware and software is identical.
So I'm going to try to print this in a way that makes things simple.
If you see '>>' at the beggining of a line, that is data I typed into
the debugged gdb.(the one that is crashing)

When you see h> at the beggining thats a hardware watchpoint and when
you see s> that's a software watchpoint.

>> b wpscope
>> r
>> watch program

Breakpoint 1, watch_command_1 (arg=0x831ff7b "", accessflag=0, from_tty=1) at ../../src/gdb/breakpoint.c:5761
(tgdb) p b
$1 = (struct breakpoint *) 0x83b4878
(tgdb) p b->disposition
$2 = disp_donttouch
(tgdb) p b->related_breakpoint
$3 = (struct breakpoint *) 0x83b49d0
(tgdb) p b->related_breakpoint->disposition
$4 = disp_del

(tgdb) bt
#0  watch_command_1 (arg=0x831ff7b "", accessflag=0, from_tty=1) at ../../src/gdb/breakpoint.c:5764
#1  0x080e131f in watch_command (arg=0x831ff76 "param", from_tty=1) at ../../src/gdb/breakpoint.c:5859
#2  0x080be17b in do_cfunc (c=0x8331b48, args=0x831ff76 "param", from_tty=1) at ../../src/gdb/cli/cli-decode.c:57
#3  0x080c0676 in cmd_func (cmd=0x8331b48, args=0x831ff76 "param", from_tty=1) at ../../src/gdb/cli/cli-decode.c:1636
#4  0x080863d5 in execute_command (p=0x831ff7a "m", from_tty=1) at ../../src/gdb/top.c:442
#5  0x0812a3ef in command_handler (command=0x831ff70 "watch param") at ../../src/gdb/event-top.c:508
#6  0x0812aba4 in command_line_handler (rl=0x8378dd8 "\230\2157\b") at ../../src/gdb/event-top.c:793
#7  0x081e8e0c in rl_callback_read_char () at ../../src/readline/callback.c:123
#8  0x08129b47 in rl_callback_read_char_wrapper (client_data=0x0) at ../../src/gdb/event-top.c:174
#9  0x0812a2b1 in stdin_event_handler (error=0, client_data=0x0) at ../../src/gdb/event-top.c:424
#10 0x08129281 in handle_file_event (event_file_desc=0) at ../../src/gdb/event-loop.c:722
#11 0x08128b55 in process_event () at ../../src/gdb/event-loop.c:335
#12 0x08128b9e in gdb_do_one_event (data=0x0) at ../../src/gdb/event-loop.c:372
#13 0x08125e41 in catch_errors (func=0x8128b6a <gdb_do_one_event>, func_args=0x0, errstring=0x8286e7f"", mask=6) at ../../src/gdb/exceptions.c:515
#14 0x080ce73c in tui_command_loop (data=0x0) at ../../src/gdb/tui/tui-interp.c:151
#15 0x0812635c in current_interp_command_loop () at ../../src/gdb/interps.c:278
#16 0x0807be63 in captured_command_loop (data=0x0) at ../../src/gdb/main.c:92
#17 0x08125e41 in catch_errors (func=0x807be58 <captured_command_loop>, func_args=0x0, errstring=0x826c4dc "", mask=6) at ../../src/gdb/exceptions.c:515
#18 0x0807cd24 in captured_main (data=0xbffffae0) at ../../src/gdb/main.c:800
#19 0x08125e41 in catch_errors (func=0x807be97 <captured_main>, func_args=0xbffffae0, errstring=0x826c4dc "", mask=6) at ../../src/gdb/exceptions.c:515
#20 0x0807cd5a in gdb_main (args=0xbffffae0) at ../../src/gdb/main.c:809
#21 0x0807be54 in main (argc=2, argv=0xbffffb54) at ../../src/gdb/gdb.c:35

>> finish

Breakpoint 2, delete_breakpoint (bpt=0x83b49d0) at ../../src/gdb/breakpoint.c:6721
(tgdb) p bpt
$5 = (struct breakpoint *) 0x83b49d0
(tgdb) p bpt->disposition
$6 = disp_del
(tgdb) p breakpoint_chain->next->next->next->next->next->next
$7 = (struct breakpoint *) 0x83b4878
(tgdb) p breakpoint_chain->next->next->next->next->next->next->related_breakpoint
$8 = (struct breakpoint *) 0x83b49d0

#0  delete_breakpoint (bpt=0x83b49d0) at ../../src/gdb/breakpoint.c:6721
#1  0x080e2888 in breakpoint_auto_delete (bs=0x8380a50) at ../../src/gdb/breakpoint.c:6702
#2  0x0811c8e3 in normal_stop () at ../../src/gdb/infrun.c:2999
#3  0x081196c2 in proceed (addr=4294967295, siggnal=TARGET_SIGNAL_DEFAULT, step=0) at ../../src/gdb/infrun.c:827
#4  0x0811741d in finish_command (arg=0x0, from_tty=1) at ../../src/gdb/infcmd.c:1280
#5  0x080be17b in do_cfunc (c=0x8337d60, args=0x0, from_tty=1) at ../../src/gdb/cli/cli-decode.c:57
#6  0x080c0676 in cmd_func (cmd=0x8337d60, args=0x0, from_tty=1) at ../../src/gdb/cli/cli-decode.c:1636
#7  0x080863d5 in execute_command (p=0x831ff76 "", from_tty=1) at ../../src/gdb/top.c:442
#8  0x0812a3ef in command_handler (command=0x831ff70 "finish") at ../../src/gdb/event-top.c:508
#9  0x0812aba4 in command_line_handler (rl=0x8378f00 "x\2177\b") at ../../src/gdb/event-top.c:793
#10 0x081e8e0c in rl_callback_read_char () at ../../src/readline/callback.c:123
#11 0x08129b47 in rl_callback_read_char_wrapper (client_data=0x0) at ../../src/gdb/event-top.c:174
#12 0x0812a2b1 in stdin_event_handler (error=0, client_data=0x0) at ../../src/gdb/event-top.c:424
#13 0x08129281 in handle_file_event (event_file_desc=0) at ../../src/gdb/event-loop.c:722
#14 0x08128b55 in process_event () at ../../src/gdb/event-loop.c:335
#15 0x08128b9e in gdb_do_one_event (data=0x0) at ../../src/gdb/event-loop.c:372
#16 0x08125e41 in catch_errors (func=0x8128b6a <gdb_do_one_event>, func_args=0x0, errstring=0x8286e7f"", mask=6) at ../../src/gdb/exceptions.c:515
#17 0x080ce73c in tui_command_loop (data=0x0) at ../../src/gdb/tui/tui-interp.c:151
#18 0x0812635c in current_interp_command_loop () at ../../src/gdb/interps.c:278
#19 0x0807be63 in captured_command_loop (data=0x0) at ../../src/gdb/main.c:92
#20 0x08125e41 in catch_errors (func=0x807be58 <captured_command_loop>, func_args=0x0, errstring=0x826c4dc "", mask=6) at ../../src/gdb/exceptions.c:515
#21 0x0807cd24 in captured_main (data=0xbffffae0) at ../../src/gdb/main.c:800
#22 0x08125e41 in catch_errors (func=0x807be97 <captured_main>, func_args=0xbffffae0, errstring=0x826c4dc "", mask=6) at ../../src/gdb/exceptions.c:515
#23 0x0807cd5a in gdb_main (args=0xbffffae0) at ../../src/gdb/main.c:809
#24 0x0807be54 in main (argc=2, argv=0xbffffb54) at ../../src/gdb/gdb.c:35

NOTE: Then I type 'finish' twice because Daniel asked me to print the same data after the 
breakpoint_auto_delete function was done,

Run till exit from #0  breakpoint_auto_delete (bs=0x8380a50) at ../../src/gdb/breakpoint.c:6699
normal_stop () at ../../src/gdb/infrun.c:3004


(tgdb) p breakpoint_chain->next->next->next->next->next->next
$9 = (struct breakpoint *) 0x83b4878
(tgdb) p breakpoint_chain->next->next->next->next->next->next->disposition 
$10 = disp_donttouch
(tgdb) p breakpoint_chain->next->next->next->next->next->next->related_breakpoint
$11 = (struct breakpoint *) 0x83b49d0
(tgdb) p breakpoint_chain->next->next->next->next->next->next->related_breakpoint->disposition
$12 = disp_del

NOTE: At this point, there is a difference between hardware and software
version.

h> (tgdb) p *breakpoint_chain->next->next->next->next->next->next->related_breakpoint
h> $13 = {
h>   next = 0x401ba0bc,
h>   type = 1075552444,
h>   enable_state = bp_enabled,
h>   disposition = disp_del,
h>   number = -11,
h>   loc = 0x83b4a78,
h>   line_number = 0,
h>   source_file = 0x0,
h>   silent = 0 '\0',
h>   ignore_count = 0,
h>   commands = 0x0,
h>   frame_id = {
h>     stack_addr = 3221224176,
h>    code_addr = 134513501,
h>    special_addr = 0,
h>    stack_addr_p = 1,
h>    code_addr_p = 1,
h>    special_addr_p = 0
h>},
h>  cond = 0x0,
h>  addr_string = 0x0,
h>  language = language_c,
h>  input_radix = 10,
h>  cond_string = 0x0,
h>  exp_string = 0x0,
h>  exp = 0x0,
h>  exp_valid_block = 0x0,
h>  val = 0x0,
h>  val_chain = 0x0,
h>  related_breakpoint = 0x0,
h>  watchpoint_frame = {
h>    stack_addr = 0,
h>    code_addr = 0,
h>    special_addr = 0,
h>    stack_addr_p = 0,
h>    code_addr_p = 0,
h>    special_addr_p = 0
h>  },
h>  thread = -1,
h>  hit_count = 1,
h>  dll_pathname = 0x0,
h>  triggered_dll_pathname = 0x0,
h>  forked_inferior_pid = 0,
h>  exec_pathname = 0x0,
h>  ops = 0x0,
h>  from_tty = 0,
h>  flag = 0,
h>  pending = 0
h>}
h>

s> $14 = {
s>   next = 0x83b4918,
s>   type = 1075552444,
s>   enable_state = bp_enabled,
s>   disposition = disp_del,
s>   number = -11,
s>   loc = 0x83b4a78,
s>   line_number = 0,
s>   source_file = 0x0,
s>   silent = 0 '\0',
s>   ignore_count = 0,
s>   commands = 0x0,
s>   frame_id = {
s>     stack_addr = 3221224176,
s>     code_addr = 134513501,
s>     special_addr = 0,
s>     stack_addr_p = 1,
s>     code_addr_p = 1,
s>     special_addr_p = 0
s>   },
s>   cond = 0x0,
s>   addr_string = 0x0,
s>   language = language_c,
s>   input_radix = 10,
s>   cond_string = 0x0,
s>   exp_string = 0x0,
s>   exp = 0x0,
s>   exp_valid_block = 0x0,
s>   val = 0x0,
s>   val_chain = 0x0,
s>   related_breakpoint = 0x0,
s>   watchpoint_frame = {
s>     stack_addr = 0,
s>     code_addr = 0,
s>     special_addr = 0,
s>     stack_addr_p = 0,
s>     code_addr_p = 0,
s>     special_addr_p = 0
s>   },
s>   thread = -1,
s>   hit_count = 1,
s>   dll_pathname = 0x0,
s>   triggered_dll_pathname = 0x0,
s>   forked_inferior_pid = 0,
s>   exec_pathname = 0x0,
s>   ops = 0x0,
s>   from_tty = 0,
s>   flag = 0,
s>   pending = 0
s> }

>> next
Breakpoint 3, insert_bp_location (bpt=0x83a9048, tmp_error_stream=0x837b390, disabled_breaks=0xbffff574, process_warning=0xbffff56c, hw_breakpoint_error=0xbffff570) at ../../src/gdb/breakpoint.c:1022

(tgdb) p bpt->owner
$16 = (struct breakpoint *) 0x83b4878
(tgdb) p bpt->owner->disposition
$17 = disp_donttouch
(tgdb) p bpt->owner->related_breakpoint
$18 = (struct breakpoint *) 0x83b49d0

NOTE: Another spot where hardware/software are different,

h>(tgdb) p bpt->owner->related_breakpoint->disposition
h>$19 = 49
h>
h>(tgdb) p *breakpoint_chain->next->next->next->next->next->next->related_breakpoint
h>$20 = {
h>  next = 0x0,
h>  type = 1650878054,
h>  enable_state = 134231092,
h>  disposition = 49,
h>  number = 0,
h>  loc = 0x401ba0bc,
h>  line_number = 0,
h>  source_file = 0x0,
h>  silent = 0 '\0',
h>  ignore_count = 0,
h>  commands = 0x0,
h>  frame_id = {
h>    stack_addr = 3221224176,
h>    code_addr = 134513501,
h>    special_addr = 0,
h>    stack_addr_p = 1,
h>    code_addr_p = 1,
h>    special_addr_p = 0
h>  },
h>  cond = 0x29,
h>  addr_string = 0x401ba0dc "\033@\033@\bJ;\b\bJ;\b\033@\033@\033@\033@\033@\033@\033@\033@\004\033@\00
h>4\033@\f\033@\f\033@>\b>\b\034\033@\034\033@$\033@$\033@,\033@,\033@4\033@4\033@<\033@<\033@D\033@D\03
h>3@L\033@L\033@T\033@T\033@\\\033@\\\033@d\033@d\033@K;\bK;\bt\033@t\033@|\033@|\033@\204\033@\204\033@
h>\214\033@\214\033@\224\033@\224\033@"...,
h>  language = 1075552476,
h>  input_radix = 10,  
h>  cond_string = 0x0,
h>  exp_string = 0x0,
h>  exp = 0x0,
h>  exp_valid_block = 0x0,
h>  val = 0x0,
h>  val_chain = 0x28,
h>  related_breakpoint = 0x10,
h>  watchpoint_frame = {
h>    stack_addr = 137890056,
h>    code_addr = 134777731,
h>    special_addr = 137890064,
h>    stack_addr_p = 1,
h>    code_addr_p = 0,
h>    special_addr_p = 0
h>  },
h>  thread = 138103256,
h>  hit_count = 134789552,
h>  dll_pathname = 0x808be88 "U\211S\203\024\213E\b\211\004$\211E\213E\2018Xa0\bt$\004$$'\b\210\211D$\bD
h>$\004\201\001",
h>  triggered_dll_pathname = 0x808ba20 "U\211\203\030\213E\f\201x\b\b\bu\002$\213E\b\211\004$\213U\f\211
h>D$\b\213E\b\211D$\004\213E\f\211\004$\213B\bU\211]U\211\203\030\213E\b\20181\bt$\004$#'\b\211D$\bD$\00
h>4",
h>  forked_inferior_pid = 134789640,
h>  exec_pathname = 0x808bd58 "U\211\203\030\213E\b\211\004$\211E\213E\2018Xa0\bt$\004$",
h>  ops = 0x808b99c,
h>  from_tty = 134790604,
h>  flag = 134790683,
h>  pending = 137890080
h>}

s>(tgdb) p bpt->owner->related_breakpoint->disposition
s> $19 = disp_del
s> 
s> (tgdb) p *breakpoint_chain->next->next->next->next->next->next->related_breakpoint
s> 
s> $20 = {
s>   next = 0x0,
s>   type = bp_none,
s>   enable_state = bp_disabled,
s>   disposition = disp_del,
s>   number = 0,
s>   loc = 0x0,
s>   line_number = 0,
s>   source_file = 0x0,
s>   silent = 0 '\0',
s>   ignore_count = 0,
s>   commands = 0x837a52c,
s>   frame_id = {
s>     stack_addr = 137864492,
s>     code_addr = 0,
s>     special_addr = 0,
s>     stack_addr_p = 0,
s>     code_addr_p = 0,
s>     special_addr_p = 0
s>   },
s>   cond = 0xffff,
s>   addr_string = 0x2 <Address 0x2 out of bounds>,
s>   language = language_unknown,
s>   input_radix = 0,
s>   cond_string = 0x0,
s>   exp_string = 0x0,
s>   exp = 0x51,
s>   exp_valid_block = 0x401ba104,
s>   val = 0x401ba104,
s>   val_chain = 0x0,
s>   related_breakpoint = 0x41,
s>   watchpoint_frame = {
s>     stack_addr = 1075552444,
s>     code_addr = 1075552444,
s>     special_addr = 0,
s>     stack_addr_p = 0,
s>     code_addr_p = 0,
s>     special_addr_p = 0
s>   },
s>   thread = -1,
s>   hit_count = 1,
s>   dll_pathname = 0x0,
s>   triggered_dll_pathname = 0x0,
s>   forked_inferior_pid = 0,
s>   exec_pathname = 0x0,
s>   ops = 0x0,
s>   from_tty = 0,
s>   flag = 0,
s>   pending = 0
s> }

NOTE: Back to normal

(tgdb) p breakpoint_chain->next->next->next->next->next->next
$21 = (struct breakpoint *) 0x83b4878
(tgdb) p breakpoint_chain->next->next->next->next->next->next->disposition
$22 = disp_donttouch
(tgdb) p breakpoint_chain->next->next->next->next->next->next->related_breakpoint
$23 = (struct breakpoint *) 0x83b49d0


h>(tgdb) p breakpoint_chain->next->next->next->next->next->next->related_breakpoint->disposition
h>$24 = 49
h>
h>(tgdb) p *breakpoint_chain->next->next->next->next->next->next->related_breakpoint
h>$25 = {
h>  next = 0x0,
h>  type = 1650878054,
h>  enable_state = 134231092,
h>  disposition = 49,
h>  number = 0,
h>  loc = 0x401ba0bc,
h>  line_number = 0,
h>  source_file = 0x0,
h>  silent = 0 '\0',
h>  ignore_count = 0,
h>  commands = 0x0,
h>  frame_id = {
h>    stack_addr = 3221224176,
h>    code_addr = 134513501,
h>    special_addr = 0,
h>    stack_addr_p = 1,
h>    code_addr_p = 1,
h>    special_addr_p = 0
h>  },
h>  cond = 0x29,
h>  addr_string = 0x401ba0dc "\033@\033@\bJ;\b\bJ;\b\033@\033@\033@\033@\033@\033@\033@\033@\004\033@\00
h>4\033@\f\033@\f\033@>\b>\b\034\033@\034\033@$\033@$\033@,\033@,\033@4\033@4\033@<\033@<\033@D\033@D\03
h>3@L\033@L\033@T\033@T\033@\\\033@\\\033@d\033@d\033@K;\bK;\bt\033@t\033@|\033@|\033@\204\033@\204\033@
h>\214\033@\214\033@\224\033@\224\033@"...,
h>  language = 1075552476,
h>  input_radix = 10,
h>  cond_string = 0x0,
h>  exp_string = 0x0,
h>  exp = 0x0,
h>  exp_valid_block = 0x0,
h>  val = 0x0,
h>  val_chain = 0x28,
h>  related_breakpoint = 0x10,
h>  watchpoint_frame = {
h>    stack_addr = 137890056,
h>    code_addr = 134777731,
h>    special_addr = 137890064,
h>    stack_addr_p = 1,
h>    code_addr_p = 0,
h>    special_addr_p = 0
h>  },
h>  thread = 138103256,
h>  hit_count = 134789552,
h>  dll_pathname = 0x808be88 "U\211S\203\024\213E\b\211\004$\211E\213E\2018Xa0\bt$\004$$'\b\210\211D$\bD
h>$\004\201\001",
h>  triggered_dll_pathname = 0x808ba20 "U\211\203\030\213E\f\201x\b\b\bu\002$\213E\b\211\004$\213U\f\211
h>D$\b\213E\b\211D$\004\213E\f\211\004$\213B\bU\211]U\211\203\030\213E\b\20181\bt$\004$#'\b\211D$\bD$\00
h>4",
h>  forked_inferior_pid = 134789640,
h>  exec_pathname = 0x808bd58 "U\211\203\030\213E\b\211\004$\211E\213E\2018Xa0\bt$\004$",
h>  ops = 0x808b99c,
h>  from_tty = 134790604,
h>  flag = 134790683,
h>  pending = 137890080
h>}

s> (tgdb) p breakpoint_chain->next->next->next->next->next->next->related_breakpoint->disposition
s> $24 = disp_del
s> (tgdb) p *breakpoint_chain->next->next->next->next->next->next->related_breakpoint
s> 
s> $25 = {
s>   next = 0x0,
s>   type = bp_none,
s>   enable_state = bp_disabled,
s>   disposition = disp_del,
s>   number = 0,
s>   loc = 0x0,
s>   line_number = 0,
s>   source_file = 0x0,
s>   silent = 0 '\0',
s>   ignore_count = 0,
s>   commands = 0x837a52c,
s>   frame_id = {
s>     stack_addr = 137864492,
s>     code_addr = 0,
s>     special_addr = 0,
s>     stack_addr_p = 0,
s>     code_addr_p = 0,
s>     special_addr_p = 0
s>   },
s>   cond = 0xffff,
s>   addr_string = 0x2 <Address 0x2 out of bounds>,
s>   language = language_unknown,
s>   input_radix = 0,
s>   cond_string = 0x0,
s>   exp_string = 0x0,
s>   exp = 0x51,
s>   exp_valid_block = 0x401ba104,
s>   val = 0x401ba104,
s>   val_chain = 0x0,
s>   related_breakpoint = 0x41,
s>   watchpoint_frame = {
s>     stack_addr = 1075552444,
s>     code_addr = 1075552444,
s>     special_addr = 0,
s>     stack_addr_p = 0,
s>     code_addr_p = 0,
s>     special_addr_p = 0
s>   },
s>   thread = -1,
s>   hit_count = 1,
s>   dll_pathname = 0x0,
s>   triggered_dll_pathname = 0x0,
s>   forked_inferior_pid = 0,
s>   exec_pathname = 0x0,
s>   ops = 0x0,
s>   from_tty = 0,
s>   flag = 0,
s>   pending = 0
s> }

#0  insert_bp_location (bpt=0x83a9048, tmp_error_stream=0x837b390, disabled_breaks=0xbffff574, process_warning=0xbffff56c, hw_breakpoint_error=0xbffff570) at ../../src/gdb/breakpoint.c:1022
#1  0x080d9edb in insert_breakpoints () at ../../src/gdb/breakpoint.c:1151
#2  0x08119621 in proceed (addr=4294967295, siggnal=TARGET_SIGNAL_DEFAULT, step=1) at ../../src/gdb/infrun.c:774
#3  0x081165a8 in step_1 (skip_subroutines=1, single_inst=0, count_string=0x0) at ../../src/gdb/infcmd.c:698
#4  0x08116311 in next_command (count_string=0x0, from_tty=1) at ../../src/gdb/infcmd.c:595
#5  0x080be17b in do_cfunc (c=0x8337dd0, args=0x0, from_tty=1) at ../../src/gdb/cli/cli-decode.c:57
#6  0x080c0676 in cmd_func (cmd=0x8337dd0, args=0x0, from_tty=1) at ../../src/gdb/cli/cli-decode.c:1636
#7  0x080863d5 in execute_command (p=0x831ff74 "", from_tty=1) at ../../src/gdb/top.c:442
#8  0x0812a3ef in command_handler (command=0x831ff70 "next") at ../../src/gdb/event-top.c:508
#9  0x0812aba4 in command_line_handler (rl=0x837b270 "@\n8\b") at ../../src/gdb/event-top.c:793
#10 0x081e8e0c in rl_callback_read_char () at ../../src/readline/callback.c:123
#11 0x08129b47 in rl_callback_read_char_wrapper (client_data=0x0) at ../../src/gdb/event-top.c:174
#12 0x0812a2b1 in stdin_event_handler (error=0, client_data=0x0) at ../../src/gdb/event-top.c:424
#13 0x08129281 in handle_file_event (event_file_desc=0) at ../../src/gdb/event-loop.c:722
#14 0x08128b55 in process_event () at ../../src/gdb/event-loop.c:335
#15 0x08128b9e in gdb_do_one_event (data=0x0) at ../../src/gdb/event-loop.c:372
#16 0x08125e41 in catch_errors (func=0x8128b6a <gdb_do_one_event>, func_args=0x0, errstring=0x8286e7f"", mask=6) at ../../src/gdb/exceptions.c:515
#17 0x080ce73c in tui_command_loop (data=0x0) at ../../src/gdb/tui/tui-interp.c:151
#18 0x0812635c in current_interp_command_loop () at ../../src/gdb/interps.c:278
#19 0x0807be63 in captured_command_loop (data=0x0) at ../../src/gdb/main.c:92
#20 0x08125e41 in catch_errors (func=0x807be58 <captured_command_loop>, func_args=0x0, errstring=0x826c4dc "", mask=6) at ../../src/gdb/exceptions.c:515
#21 0x0807cd24 in captured_main (data=0xbffffae0) at ../../src/gdb/main.c:800
#22 0x08125e41 in catch_errors (func=0x807be97 <captured_main>, func_args=0xbffffae0, errstring=0x826c4dc "", mask=6) at ../../src/gdb/exceptions.c:515
#23 0x0807cd5a in gdb_main (args=0xbffffae0) at ../../src/gdb/main.c:809
#24 0x0807be54 in main (argc=2, argv=0xbffffb54) at ../../src/gdb/gdb.c:35

Bob Rossi