dynamic breakpoints/watchpoints

dynamic breakpoints/watchpoints

[Ed S. Peschko]

All,

I'm not sure how feasible this is for gdb, but I've written something like 
this for perl and it does wonders for your productivity, especially in 
dealing with unfamiliar code.

What I'm looking for is something akin to grep integrated with gdb. Commands 
like:

	bc /pattern_in_code/

would stop at the first place where 'pattern_in_code' was seen after continuing 
execution, and

	bv /pattern_in_variable/

would monitor the data that *any* variable is using, and stop as soon as an 
assignment of that pattern is made to that variable.


Of course, it might be difficult to do this with c, with differences between 
floats && ints && ascii data, but it's not impossible. Maybe, 'bs' for break string,
or bi for break integer would do the trick..


Thoughts? I'd think that 'bc' would be easier to implement but I'm not sure...

Ed

Re: dynamic breakpoints/watchpoints

[Eli Zaretskii]

> Date: Tue, 13 Mar 2007 17:22:42 -0700
> From: "Ed S. Peschko" <esp5@pge.com>
> 
> What I'm looking for is something akin to grep integrated with gdb. Commands 
> like:
> 
> 	bc /pattern_in_code/
> 
> would stop at the first place where 'pattern_in_code' was seen after continuing 
> execution

Is `rbreak' (q.v.) what you are looking for?

> 	bv /pattern_in_variable/
> 
> would monitor the data that *any* variable is using, and stop as soon as an 
> assignment of that pattern is made to that variable.

I think this one would be impractical, at least on x86, where the
number of watchpoints is severely limited by the available number of
debug registers.

Re: dynamic breakpoints/watchpoints

[Ed S. Peschko]

On Wed, Mar 14, 2007 at 06:20:51AM +0200, Eli Zaretskii wrote:
> > Date: Tue, 13 Mar 2007 17:22:42 -0700
> > From: "Ed S. Peschko" <esp5@pge.com>
> > 
> > What I'm looking for is something akin to grep integrated with gdb. Commands 
> > like:
> > 
> > 	bc /pattern_in_code/
> > 
> > would stop at the first place where 'pattern_in_code' was seen after continuing 
> > execution
> 
> Is `rbreak' (q.v.) what you are looking for?


Well, sort of, but I don't see why it has to be limited by subroutines. 
And I don't see why it has to be static.

For example, 

int main()
{
	==>	int stuff;
	aha();
	aha2();
}


int aha()
{
	fprintf(stderr, "HERE1");
}

int aha2()
{
	fprintf(stderr, "HERE");
}
	
Suppose that I'm at the '==>'. if I say bc /HERE/", I would break in the 
function 'aha', at the fprintf statement HERE1'. If I said bc /HERE/ at THAT
point, then I'd break at 'HERE2'. The breakpoint changes as you go along.


The main point of bc /HERE/ is that you forget the name of a variable, a function,
or what-have-you but you know that a statement is going to contain that pattern
at some point when you reach it in the future. Say for instance, a logging message. 

The way I use 'bc' (at least in perl) adds a simple method to get to ANY piece of 
code that I recognize.


> > 	bv /pattern_in_variable/
> > 
> > would monitor the data that *any* variable is using, and stop as soon as an 
> > assignment of that pattern is made to that variable.
> 
> I think this one would be impractical, at least on x86, where the
> number of watchpoints is severely limited by the available number of
> debug registers.

Well, I guess I'm not talking about a hardware watchpoint but a software one. 
And yes, that would mean examining each chunk of memory before it gets written 
to a variable, which would probably require a gcc-hook for gdb to plug into.

But this particular feature is WELL worth it - as I've used it in perl. 
Its a real GODSEND. Again, instead of needing to track down the program flow by several
runs to get the point where you need to break, you simply look at the data coming
into the program to see where to break, and THEN figure out the program flow 
by going backwards via trace.

A good example - I was trying to track down how curl wrote data to a certain socket
the other day. I knew that it consisted of writing a tag 'filename=' in a file
somewhere, but for the life of me I couldn't figure out where that string was 
coming about in the curl source code.

If I had 'bv' , I could have simply used 'bv /filename=/', ran the program, found out
where the code picked up this tag, and used trace to figure out what made it get
to that point.

It would have taken a 4 hour excruciating hunt and peck operation, and turned it
into a 5 minute task.

Ed

Re: dynamic breakpoints/watchpoints

[Michael Veksler]

Ed S. Peschko wrote:
> For example, 
>
> int main()
> {
> 	==>	int stuff;
> 	aha();
> 	aha2();
> }
>
>
> int aha()
> {
> 	fprintf(stderr, "HERE1");
> }
>
> int aha2()
> {
> 	fprintf(stderr, "HERE");
> }
> 	
> Suppose that I'm at the '==>'. if I say bc /HERE/", I would break in the 
> function 'aha', at the fprintf statement HERE1'. If I said bc /HERE/ at THAT
> point, then I'd break at 'HERE2'. The breakpoint changes as you go along.
>   
What I usually do is use emacs' tags system to search for the string, 
and then manually put breakpoints at the relevant locations. This works 
well when there is only couple of matching lines.
My normal use case is:

   1. An error text is printed, or some activity is logged into a file.
   2. I want to stop right at the place that caused that behavior. So I
      perform the following:
         1. Try to guess how the relevant code looks like
         2. Look for all occurrences of such code using grep/tags
         3. If search result is too wide or too narrow go to step i.
         4. Break at all relevant locations

This process is quite long and prone to errors. It would be much better 
if it were integrated into gdb+libc+libstdc++ in the following way:

   1. libc+libstdc++ will have hooks that gdb will be able to use to
      intercept FILE/stream writes.
   2. A slower alternative to 1 is for for GDB to intercept write system
      calls, and check their contents. To improve granularity in C,
      buffering should be set to line granularity. In C++, buffering
      should be eliminated (unless people use std::endl instead of the
      much better "\n").
   3. GDB will stop at each write, and look for the requested text. If
      the text is encountered - break.

This technique can be implemented to a limited extend by a conditional 
breakpoint on write() which will wait for memmem(name_of_buffer_arg, 
size_of_buffer, your_text, strlen(your_text) ). But that is ugly, 
imperfect and is not something you want to require from average users to 
implement. I was never desperate enough to implement this in my code. 
This should be implemented in gdb+glibc+libstdc++.

My proposed feature seems easier to implement and more powerful in the 
cases that interest me (and you?).

Even the watchpoints of std::string can be implemented by some hooks 
provided by the libstdc++ team. They could add an strcmp looking for the 
requested string, every time the string is mutated, and then GDB will be 
able to break on a match. Other data-types are more complicated, and 
most of them are impractical candidates for dynamic watchpoints (unless 
GCC is heavily modified to insert hooks for all updates to, for example, 
int variables - this will be slow as hell).

---
Michael

Re: dynamic breakpoints/watchpoints

[Ed S. Peschko]

On Wed, Mar 14, 2007 at 11:20:18AM +0200, Michael Veksler wrote:
> Ed S. Peschko wrote:
> >For example, 
> >
> >int main()
> >{
> >	==>	int stuff;
> >	aha();
> >	aha2();
> >}
> >
> >
> >int aha()
> >{
> >	fprintf(stderr, "HERE1");
> >}
> >
> >int aha2()
> >{
> >	fprintf(stderr, "HERE");
> >}
> >	
> >Suppose that I'm at the '==>'. if I say bc /HERE/", I would break in the 
> >function 'aha', at the fprintf statement HERE1'. If I said bc /HERE/ at 
> >THAT
> >point, then I'd break at 'HERE2'. The breakpoint changes as you go along.
> >  
> What I usually do is use emacs' tags system to search for the string, 
> and then manually put breakpoints at the relevant locations. This works 
> well when there is only couple of matching lines.

Well, yeah, of course there are workarounds, but they are IMO messy and
hard to deal with. Instead of setting up tags, or grepping through code,
doing a code match in the debugger is far easier. That's why I wrote
it for perl in the first place, I got sick of doing these workarounds.

Plus, I'm not sure if the tags solution is complete. Tags normally only
handles function calls, declarations, etc. as I understand it. How do you 
make breakpoints for statements inside function calls using it without
grepping through the code and manufacturing breakpoints using some sort of 
scripting?

> My normal use case is:
> 
>   1. An error text is printed, or some activity is logged into a file.
>   2. I want to stop right at the place that caused that behavior. So I
>      perform the following:
>         1. Try to guess how the relevant code looks like
>         2. Look for all occurrences of such code using grep/tags
>         3. If search result is too wide or too narrow go to step i.
>         4. Break at all relevant locations
> 
> This process is quite long and prone to errors. It would be much better 
> if it were integrated into gdb+libc+libstdc++ in the following way:
> 
>   1. libc+libstdc++ will have hooks that gdb will be able to use to
>      intercept FILE/stream writes.
>   2. A slower alternative to 1 is for for GDB to intercept write system
>      calls, and check their contents. To improve granularity in C,
>      buffering should be set to line granularity. In C++, buffering
>      should be eliminated (unless people use std::endl instead of the
>      much better "\n").
>   3. GDB will stop at each write, and look for the requested text. If
>      the text is encountered - break.

look, I agree.. having some searching ability is better than none at 
all, and if it is easy to implement this then maybe that is the way to go.

But it's incredibly comforting in perl to have a universal search (in
perl this is of course integrated with regular expressions). I know that 
no matter what I search for, I'll find it as soon as the data is
matched. 

And true, its slow as hell because everything is interpreted, but it would
be far slower for me to track it down manually. 

For gcc and gdb, I'd be willing to build my own special compiler 
instance to carry this functionality, and avoid the overhead for
my regular compiles by having ifdefs in the gcc code .

Another point - unless this functionality is to be linux only, 
I'd be hesitant to put the change into libstdc++ and/or glibc. 
glibc is NOTORIOUSLY unportable, and is far from universal.

> provided by the libstdc++ team. They could add an strcmp looking for the 
> requested string, every time the string is mutated, and then GDB will be 
> able to break on a match. Other data-types are more complicated, and 
> most of them are impractical candidates for dynamic watchpoints (unless 
> GCC is heavily modified to insert hooks for all updates to, for example, 
> int variables - this will be slow as hell).

yeah, but as I indicated above, I think that patching gcc would be
probably the only way to get widespread portability.

Ed

Re: dynamic breakpoints/watchpoints

[Eli Zaretskii]

> Date: Wed, 14 Mar 2007 11:03:17 -0700
> From: "Ed S. Peschko" <esp5@pge.com>
> Cc: Eli Zaretskii <eliz@gnu.org>, gdb@sourceware.org
> 
> Tags normally only handles function calls, declarations, etc. as I
> understand it.

Not that it matters for this discussion, but the above is incorrect:
the `etags' command can tag almost _anything_, if you use its --regex
option.

Re: dynamic breakpoints/watchpoints

[Ed S. Peschko]

> I still don't see what you mean. From your use case it is not clear why
> you would like to search the source code, rather than put a break-point
> on the FILE or stream.
> I always hated the perl debugger (maybe it is now better!), so any
> change to its old state should have had a radical productivity
> improvement. This does not mean that such a modification to GDB
> will have the same effect (it might, but I don't understand it from
> your example, can you give a better example?)
> 

Ok, just to be clear and to recap. I had two use cases in mind, one for 
breaking via searching data, one for breaking via searching code.

For searching code, I think that we are pretty much talking about the same thing.

    bc /hello there/ 

would break at the next point where the string 'hello there' was in the code; or 
set multiple breakpoints the same way that rbreak does (although I'm really skeptical
how it could do this for the case of, say libraries loaded at runtime and that have
not been loaded yet, which you could do with a runtime implementation.)


For searching data, I think we are close. I think we just differ in the details. 
I'm talking about a universal data searching mechanism, such that if you say:

    int main()
    {
        string a = "hello";

        a.append(" there");
        cout << a <<endl;
    }

    (gdb) bd /hello there/


it would break at the point in the c++ code where a became 'hello there'.

I think it should be implemented as hooks in the gcc compiler that talk to gdb,
you think that it should be implemented as part as libstdc++ and glibc.


My reason for preferring gcc over glibc is because of portability and universality. 
libstdc++ might be highly portable, but 99% of the open source packages are 
implemented in c, and glibc is highly linux specific. 

I would think that gcc could be tweaked to make a function call after any statement 
that does a data write, and that gdb could be tweaked to break at any point that the 
desired pattern is found, but I didn't develop gcc, so that's probably a question for them.



Anyways, as far as the perl debugger is concerned, I don't want to start a flame war, 
but IMO its more powerful than gdb as it stands, whether or not these features are 
implemented: for it allows you to inject any interpreted code you want at runtime at
any point in the execution stream, and have that code interact with variables inside
the running program.

I ended up implementing the code and data searching because I found myself swamped 
with code that I didn't write and still needed to maintain, and because it was far 
easier to look at what the code *produced* to deduce what it was doing rather than 
the code *itself*.  That's how I primarily use this - as a quick way to understand 
what is going on in a program without lots of code scrutiny.


Ed

(
    ps - I don't know gdb as much as the perl debugger, so I don't know - 
         maybe it is possible to do the same thing with gdb. I know you 
         can set variables in gdb, but I don't know if you can insert 
         whole blocks of c-code before or after a certain point..  
         I'd love to be able to do this, too..
)