watchpoints inside 'commands'

watchpoints inside 'commands'

[Edward Peschko]

On Thu, Apr 05, 2001 at 08:00:28PM -0700, Edward Peschko wrote:
hey,

I was wondering how people might tackle the following problem:

I'm trying to track down a memory corruption bug in my C++ code, and am finding
it very difficult. it comes in a function that creates several items... 

Anyways, I was thinking of putting in a watchpoint, but I would have to put 
in several watchpoints because the bug is intermittant:

Key *Object::getItem(Key key)
{
	return (Object::getItem(&key));  bug here.
}

So. I tried the following:

b Object.cpp:12
commands 1
> silent
> watch key._data[0]
> continue

Unfortunately, this doesn't seem to work because, when the watchpoint is 
eliminated, the program auto halts. Why?

And can you set an 'intelligent' watchpoint, one that watches the value of a 
variable *name* (not a variable instance) between point 'a' and point 'b' in 
your code? This would be far more useful than the current behaviour - 
currently, tracing one instance of a variable is useless if you've got a 
function which creates and destroys tons of them...

ed

(ps -- this brings up another thing.. if you've got a heisenbug, how do you go
about tracking it down? Say that another piece of your code (in another thread)
is trashing your thread via an array bounds write (or some such thing) How can
you track this down as being the cause? And how do you fix it?)

Re: watchpoints inside 'commands'

[Eli Zaretskii]

> Date: Thu, 5 Apr 2001 20:05:25 -0700
> From: Edward Peschko <edwardp@excitehome.net>
> 
> Key *Object::getItem(Key key)
> {
>	return (Object::getItem(&key));  bug here.
> }
> 
> So. I tried the following:
> 
> b Object.cpp:12
> commands 1
> > silent
> > watch key._data[0]
> > continue
> 
> Unfortunately, this doesn't seem to work because, when the watchpoint is 
> eliminated, the program auto halts. Why?

What exactly do you mean by ``when the watchpoint is eliminated, the
program auto halts''?  Can you tell what commands do you type and what
does GDB print in response?

> And can you set an 'intelligent' watchpoint, one that watches the value of a 
> variable *name* (not a variable instance) between point 'a' and point 'b' in 
> your code? This would be far more useful than the current behaviour - 
> currently, tracing one instance of a variable is useless if you've got a 
> function which creates and destroys tons of them...

I'm not sure I understand what you want, but it sounds like watching
the variable by its address instead of by its name should do the
trick.

> (ps -- this brings up another thing.. if you've got a heisenbug, how
> do you go about tracking it down? Say that another piece of your
> code (in another thread) is trashing your thread via an array bounds
> write (or some such thing) How can you track this down as being the
> cause?

I usually do that with hardware-assisted watchpoints on the memory
region that is being trashed.

Re: watchpoints inside 'commands'

[Edward Peschko]

On Fri, Apr 06, 2001 at 05:09:18AM -0400, Eli Zaretskii wrote:
> > Date: Thu, 5 Apr 2001 20:05:25 -0700
> > From: Edward Peschko <edwardp@excitehome.net>
> > 
> > Key *Object::getItem(Key key)
> > {
> >	return (Object::getItem(&key));  bug here.
> > }
> > 
> > So. I tried the following:
> > 
> > b Object.cpp:12
> > commands 1
> > > silent
> > > watch key._data[0]
> > > continue
> > 
> > Unfortunately, this doesn't seem to work because, when the watchpoint is 
> > eliminated, the program auto halts. Why?
> 
> What exactly do you mean by ``when the watchpoint is eliminated, the
> program auto halts''?  Can you tell what commands do you type and what
> does GDB print in response?

Basically, the program halts (stops) after the watchpoint is hit.

> > And can you set an 'intelligent' watchpoint, one that watches the value of a 
> > variable *name* (not a variable instance) between point 'a' and point 'b' in 
> > your code? This would be far more useful than the current behaviour - 
> > currently, tracing one instance of a variable is useless if you've got a 
> > function which creates and destroys tons of them...
> 
> I'm not sure I understand what you want, but it sounds like watching
> the variable by its address instead of by its name should do the
> trick.

No, it won't - the point is, when you have code that looks like 

int function(char *string)
{
	# begin (line 15)
	... (string shouldn't change but it does)
	...
	# end (line 40)
}
	
then 'string' will have a different address every single time function is 
called. I'd like the ability to track 'string' from line 15 to line 40, put 
a hardware assisted watchpoint on it when line 15 is being reached, and letting
go the watchpoint when line 40 is being reached.


As it stands, if I say:

b 15
commands
> silent
> watch string
> continue


as soon as the scope changes inside of 'function', the program breaks... It says
'watchpoint being deleted as the variable goes out of scope'.


> > (ps -- this brings up another thing.. if you've got a heisenbug, how
> > do you go about tracking it down? Say that another piece of your
> > code (in another thread) is trashing your thread via an array bounds
> > write (or some such thing) How can you track this down as being the
> > cause?
> 
> I usually do that with hardware-assisted watchpoints on the memory
> region that is being trashed.

but the memory region that is being trashed varies from time to time...

Ed

Re: watchpoints inside 'commands'

[Eli Zaretskii]

> Date: Fri, 6 Apr 2001 11:19:45 -0700
> From: Edward Peschko <edwardp@excitehome.net>
> 
> On Fri, Apr 06, 2001 at 05:09:18AM -0400, Eli Zaretskii wrote:
> > > Date: Thu, 5 Apr 2001 20:05:25 -0700
> > > From: Edward Peschko <edwardp@excitehome.net>
> > > 
> > > Key *Object::getItem(Key key)
> > > {
> > >	return (Object::getItem(&key));  bug here.
> > > }
> > > 
> > > So. I tried the following:
> > > 
> > > b Object.cpp:12
> > > commands 1
> > > > silent
> > > > watch key._data[0]
> > > > continue
> > > 
> > > Unfortunately, this doesn't seem to work because, when the watchpoint is 
> > > eliminated, the program auto halts. Why?
> > 
> > What exactly do you mean by ``when the watchpoint is eliminated, the
> > program auto halts''?  Can you tell what commands do you type and what
> > does GDB print in response?
> 
> Basically, the program halts (stops) after the watchpoint is hit.

Sorry, I'm still missing something: the program _should_ stop when the
watchpoint is hit, shouldn't it?  That's what a watchpoint is for: it
stops the execution of the program being debugged when memory watched
by a watchpoint is accessed.  What am I missing here?

> int function(char *string)
> {
>	# begin (line 15)
>	... (string shouldn't change but it does)
>	...
>	# end (line 40)
> }
>	
> then 'string' will have a different address every single time function is 
> called. I'd like the ability to track 'string' from line 15 to line 40, put 
> a hardware assisted watchpoint on it when line 15 is being reached,
> and letting go the watchpoint when line 40 is being reached.
> 
> 
> As it stands, if I say:
> 
> b 15
> commands
> > silent
> > watch string
> > continue
> 
> 
> as soon as the scope changes inside of 'function', the program
> breaks... It says 'watchpoint being deleted as the variable goes out
> of scope'.

But this is exactly what you want: GDB just warns you about the fact
that it deleted the watchpoint because it watches a variable on the
stack that is no longer in use.  Why is that a problem?  The message
doesn't mean that the program is broken, you should be able to
continue it as usual.

> > > (ps -- this brings up another thing.. if you've got a heisenbug, how
> > > do you go about tracking it down? Say that another piece of your
> > > code (in another thread) is trashing your thread via an array bounds
> > > write (or some such thing) How can you track this down as being the
> > > cause?
> > 
> > I usually do that with hardware-assisted watchpoints on the memory
> > region that is being trashed.
> 
> but the memory region that is being trashed varies from time to time...

You will have to find a repeatable scenario in which the same memory
region is trashed, and then watch that memory region.

How do you find out where's the trashed region, if it changes from
time to time?  Does it change because you modify your program,
perhaps?

Re: watchpoints inside 'commands'

[Edward Peschko]

> Sorry, I'm still missing something: the program _should_ stop when the
> watchpoint is hit, shouldn't it?  That's what a watchpoint is for: it
> stops the execution of the program being debugged when memory watched
> by a watchpoint is accessed.  What am I missing here?

But the point is that its not stopping when the watchpoint is hit; its stopping
WHEN THE WATCHPOINT IS DELETED BECAUSE IT GOES OUT OF SCOPE. Which is different,
and annoying. The proper behaviour should be that the watchpoint deletes itself
and the program continues running.

> > as soon as the scope changes inside of 'function', the program
> > breaks... It says 'watchpoint being deleted as the variable goes out
> > of scope'.
> 
> But this is exactly what you want: GDB just warns you about the fact
> that it deleted the watchpoint because it watches a variable on the
> stack that is no longer in use.  Why is that a problem?  The message
> doesn't mean that the program is broken, you should be able to
> continue it as usual.

But it *doesn't* continue as usual. It warns (which it shouldn't if I tell it
to be silent ) and then *stops*. It is transparent to the continue.

Say I'm running a program and 39,999 out of 40,000 times the variable key 
is allocated and deallocated successfully.

Now, I don't want to have the watchpoint stop all of those 40,000 times; I only
want it to stop if something else trounces the memory that I don't know about.

> You will have to find a repeatable scenario in which the same memory
> region is trashed, and then watch that memory region.
> 
> How do you find out where's the trashed region, if it changes from
> time to time?  Does it change because you modify your program,
> perhaps?

No, it changes because the program is threaded, and hence is asynchronous, and 
hence is not deterministic as to where it crashes. In addition, if I set up a 
watchpoint on a specific piece of memory, there are three points that the
watchpoint will break at:

	1) when the memory is initially allocated.
	2) when the memory is cleared/deleted.
	3) when the memory is trounced over.

Now, if it stops at #1, then the conditions of the program will have changed, 
and in all probability #3 will not occur at the same address because it has 
changed the state of the program before #3 occurs.

Ed

Re: watchpoints inside 'commands'

[Eli Zaretskii]

> Date: Sat, 7 Apr 2001 17:33:41 -0700
> From: Edward Peschko <edwardp@excitehome.net>
> 
> > > as soon as the scope changes inside of 'function', the program
> > > breaks... It says 'watchpoint being deleted as the variable goes out
> > > of scope'.
> > 
> > But this is exactly what you want: GDB just warns you about the fact
> > that it deleted the watchpoint because it watches a variable on the
> > stack that is no longer in use.  Why is that a problem?  The message
> > doesn't mean that the program is broken, you should be able to
> > continue it as usual.
> 
> But it *doesn't* continue as usual. It warns (which it shouldn't if I tell it
> to be silent ) and then *stops*. It is transparent to the continue.
> 
> Say I'm running a program and 39,999 out of 40,000 times the variable key 
> is allocated and deallocated successfully.
> 
> Now, I don't want to have the watchpoint stop all of those 40,000
> times; I only want it to stop if something else trounces the memory
> that I don't know about.

Ah, okay, then how about setting a breakpoint near the exit from the
scope where the watchpoint is defined, and setting up the commands of
that breakpoint to silently delete the watchpoint and continue?  Would
that do what you want?

> > You will have to find a repeatable scenario in which the same memory
> > region is trashed, and then watch that memory region.
> > 
> > How do you find out where's the trashed region, if it changes from
> > time to time?  Does it change because you modify your program,
> > perhaps?
> 
> No, it changes because the program is threaded, and hence is
> asynchronous, and hence is not deterministic as to where it
> crashes.

Can you get the trashing if some of the threads don't run?  It's
probably best to have the minimum number of threads that still
reproduce the problem.

Anyway, how many possible memory regions did you find to be trashed
until now?  If it is a relatively small number, watchpoints could
still help (you could watch all or most of them).

> In addition, if I set up a watchpoint on a specific piece
> of memory, there are three points that the watchpoint will break at:
> 
>	1) when the memory is initially allocated.
>	2) when the memory is cleared/deleted.
>	3) when the memory is trounced over.

That's true, but you could handle this with watchpoint commands: they
could simply print their usual stuff and then continue.  Later you
could examine the session script and decide which of the breaks were
false alarms and which not.

> Now, if it stops at #1, then the conditions of the program will have
> changed, and in all probability #3 will not occur at the same
> address because it has changed the state of the program before #3
> occurs.

What you describe sounds like uninitialized pointer somewhere, or some
other source of random addresses being poked.  (Are the regions that
get trashed really random, or do you have only a small set of possible
victims?)  If the address is really random, I'd suggest to think what
could be the source of such randomness.  A random address is not easy
to get by in most programs.

Re: watchpoints inside 'commands'

[Edward Peschko]

> Ah, okay, then how about setting a breakpoint near the exit from the
> scope where the watchpoint is defined, and setting up the commands of
> that breakpoint to silently delete the watchpoint and continue?  Would
> that do what you want?

Ok, but there is no command - as far as I can tell - to 'silently delete the 
watchpoint and continue'. Take the following code:

---
#include <stdio.h>

void a();

int main()
{
	int xx;
	for (xx = 0; xx < 100; xx++)
	{
		function();
	}
}

int yy = 0;

void a()
{

	char data[10]  = "hello";
	if (yy++%10 == 0) 
		data[0] = 'i';

	
}

if you say something like:

b 20
Breakpoint 1 at 0x10924: file a.c, line 20.
commands 1
> silent
> watch data[0];
> continue

then, gdb will set up a watchpoint (which I assume is a breakpoint) at 20 when 
line #20 is hit. So far so good.

But the watchpoint has a different *number* each time it comes up (ie: watch
data[0] is 'watchpoint 2' on the first way round, watch data[0] is 'watchpoint
3' the second time round, etc. etc.

So this does not work. And in any case, wouldn't it be easier to say -

watch data[0] 22 25

meaning 'watch the memory space represented by data[0] from line 22 to line 25'?

Ed

> Can you get the trashing if some of the threads don't run?  It's
> probably best to have the minimum number of threads that still
> reproduce the problem.

If we turn down the threads, its pretty stable (ie: it still takes time to 
crash, but a longer period) And it does not crash if we take threading out
completely.

> Anyway, how many possible memory regions did you find to be trashed
> until now?  If it is a relatively small number, watchpoints could
> still help (you could watch all or most of them).

its different every time.

> That's true, but you could handle this with watchpoint commands: they
> could simply print their usual stuff and then continue.  Later you
> could examine the session script and decide which of the breaks were
> false alarms and which not.

That would be fine, if watchpoints didn't change their number each time you 
created and deleted them...

> What you describe sounds like uninitialized pointer somewhere, or some
> other source of random addresses being poked.  (Are the regions that
> get trashed really random, or do you have only a small set of possible
> victims?)  If the address is really random, I'd suggest to think what
> could be the source of such randomness.  A random address is not easy
> to get by in most programs.

which is why I think the interface into gcc should change to accomodate this.
Its hell to track down problems like this in program right now, it wouldn't be
(as much hell) if we had an interface like this.

Ed

Re: watchpoints inside 'commands'

[Daniel Berlin]

Edward Peschko <edwardp@excitehome.net> writes:

> > Ah, okay, then how about setting a breakpoint near the exit from the
> > scope where the watchpoint is defined, and setting up the commands of
> > that breakpoint to silently delete the watchpoint and continue?  Would
> > that do what you want?
> 
> Ok, but there is no command - as far as I can tell - to 'silently delete the 
> watchpoint and continue'. Take the following code:
> 
> ---
> #include <stdio.h>
> 
> void a();
> 
> int main()
> {
> 	int xx;
> 	for (xx = 0; xx < 100; xx++)
> 	{
> 		function();
> 	}
> }
> 
> int yy = 0;
> 
> void a()
> {
> 
> 	char data[10]  = "hello";
> 	if (yy++%10 == 0) 
> 		data[0] = 'i';
> 
> 	
> }
> 
> if you say something like:
> 
> b 20
> Breakpoint 1 at 0x10924: file a.c, line 20.
> commands 1
> > silent
> > watch data[0];
> > continue
> 
> then, gdb will set up a watchpoint (which I assume is a breakpoint) at 20 when 
> line #20 is hit. So far so good.
> 
> But the watchpoint has a different *number* each time it comes up (ie: watch
> data[0] is 'watchpoint 2' on the first way round, watch data[0] is 'watchpoint
> 3' the second time round, etc. etc.

$bpnum will give you the number of the last breakpoint set (which
include watchpoints)

HTH,
Dan

-- 
I went to a general store.  They wouldn't let me buy anything
specifically.

Re: watchpoints inside 'commands'

[Eli Zaretskii]

> Date: Sun, 8 Apr 2001 16:58:52 -0700
> From: Edward Peschko <edwardp@excitehome.net>
> 
> if you say something like:
> 
> b 20
> Breakpoint 1 at 0x10924: file a.c, line 20.
> commands 1
> > silent
> > watch data[0];
> > continue
> 
> then, gdb will set up a watchpoint (which I assume is a breakpoint)
> at 20 when line #20 is hit. So far so good.
> 
> But the watchpoint has a different *number* each time it comes up (ie: watch
> data[0] is 'watchpoint 2' on the first way round, watch data[0] is
> 'watchpoint 3' the second time round, etc. etc.

The command `commands' can be used with no argument, in which case it
refers to the last breakpoint or watchpoint set.  So this (UNTESTED):

 b 20
 Breakpoint 1 at 0x10924: file a.c, line 20.
 commands 1
 > silent
 > watch data[0]
 > set $wpnum = $bpnum
 > tbreak 25
 > commands
 > > silent
 > > delete $wpnum
 > > continue
 > > end
 > continue
 > end

should do the trick.

> And in any case, wouldn't it be easier to say -
> 
> watch data[0] 22 25
> 
> meaning 'watch the memory space represented by data[0] from line 22
> to line 25'?

I think the above just about implements this ;-).  You can define a
new command which does that, save it to your .gdbinit, and then use it
at will.

> > That's true, but you could handle this with watchpoint commands: they
> > could simply print their usual stuff and then continue.  Later you
> > could examine the session script and decide which of the breaks were
> > false alarms and which not.
> 
> That would be fine, if watchpoints didn't change their number each time you 
> created and deleted them...

I think you can solve this with $bpnum and `commands' without an
argument.

Re: watchpoints inside 'commands'

[Andrew Cagney]

> > Now, I don't want to have the watchpoint stop all of those 40,000
> > times; I only want it to stop if something else trounces the memory
> > that I don't know about.
> 
> Ah, okay, then how about setting a breakpoint near the exit from the
> scope where the watchpoint is defined, and setting up the commands of
> that breakpoint to silently delete the watchpoint and continue?  Would
> that do what you want?

Just FYI, I've this memory of GDB (or is it insight) doing something for
some cases already.  GDB quietly sets a breakpoint on the function's
exit code so that it can zap certain tempoary breakpoints when the
function exits.

There may well be a case for making the behavour consistent (however
standard disclaimer about this being my vague memory).

	Andrew