Re: Bug with watchpoints on Linux

Re: Bug with watchpoints on Linux

[Pierre Muller]

At 19:25 21/11/2001 , Eli Zaretskii a écrit:
> > Date: Wed, 21 Nov 2001 13:01:59 +0100
> > From: Pierre Muller <muller@cerbere.u-strasbg.fr>
> > 
> >    But at the second run, the program is never stopped because
> > of the changes to this global variable.
> > 
> >    It seems like there is a problem with the hardware watchpoint
> > resetting.
> > 
> >    I tested this on only one Linux machine,
> > but both the main and the 5.1 branches show this problem.
>
>The DJGPP port doesn't have this problem.

   Nevertheless, according to my debug,
I expect that most targets using 
i386 hardware watchpoints and 
not calling i386_cleanup_dregs
will show this problem.

   I think that this is a really nasty bug for 
the i386 linux target for 5.1 release....

    What do we do about this?
I proposed to add a call to i386_cleanup_dregs
for all targets using the standard 
i386 hardware watchpoints, but 
Eli seems still not convinced by my reports....





Pierre Muller
Institut Charles Sadron
6,rue Boussingault
F 67083 STRASBOURG CEDEX (France)
mailto:muller@ics.u-strasbg.fr
Phone : (33)-3-88-41-40-07  Fax : (33)-3-88-41-40-99

Re: Bug with watchpoints on Linux

[Eli Zaretskii]

> Date: Fri, 30 Nov 2001 16:17:27 +0100
> From: Pierre Muller <muller@cerbere.u-strasbg.fr>
> 
>     What do we do about this?
> I proposed to add a call to i386_cleanup_dregs
> for all targets using the standard 
> i386 hardware watchpoints, but 
> Eli seems still not convinced by my reports....

You don't need to convince me (the port I'm responsible for _does_
use i386_cleanup_dregs, after all ;-).  You need to convince Mark
Kettenis.  I'm sure that if you post a test program and a full recipe
to reproduce the problem, and then show the patch which makes the
problem go away, Mark will happily accept it.

Btw, please use "diff -c" or 'diff -u" when you send diffs.  The
format you were using is very hard to read, and also is prone to
failures when applied with Patch, if the sources changed a bit since
the version you used to produce the diffs.

[RFC] Possible bug with i386 watchpoints on several targets.

[Pierre Muller]

At 16:45 30/11/2001 , Eli Zaretskii a écrit:
> > Date: Fri, 30 Nov 2001 16:17:27 +0100
> > From: Pierre Muller <muller@cerbere.u-strasbg.fr>
> > 
> >     What do we do about this?
> > I proposed to add a call to i386_cleanup_dregs
> > for all targets using the standard 
> > i386 hardware watchpoints, but 
> > Eli seems still not convinced by my reports....
>
>You don't need to convince me (the port I'm responsible for _does_
>use i386_cleanup_dregs, after all ;-).  You need to convince Mark
>Kettenis.  I'm sure that if you post a test program and a full recipe
>to reproduce the problem, and then show the patch which makes the
>problem go away, Mark will happily accept it.

   The way to check if the bug exist is easy:

   take gdb (5.1 or current snapshot)
compiled with debug info and 
run
   ./gdb ./gdb
on the newly generated executable.
(preferabily in the compilation dir,
which will ensure that you get (top-gdb) prompt
for the debugger and (gdb) for the debuggee).

use the following command
(top-gdb) watch gdb_stdout
(top-gdb) r
Here you should get a stop due to the setting of gdb_stdout value.
(top-gdb) cont
you should now reach the debuggee command prompt,
simply quit.
(gdb) quit
and rerun the same executable without any watchpoint modification.
(top-gdb) run
if the bug that I found on both win32 (without i386_cleanup_deregs call)
and on current (a few days old) CVS for linux.
You won't get any stop on the second run.

The fix is not that obvious, because I am still a bit unclear
about wher it should be put:
   the i386_cleanup_dregs should be called (in my opinion)
both when a program is killed and when a program
terminates normally.

   But linux does seem to use generic_mourn_inferior,
so for this target it would probably require that we 
change this into a target specific function.

PS: Eli, did you check the go32v2 target if you kill the program after
the watchpoint is reached, sorry, I still can't get GDB to compile for djgpp
on my win2000.


Pierre Muller
Institut Charles Sadron
6,rue Boussingault
F 67083 STRASBOURG CEDEX (France)
mailto:muller@ics.u-strasbg.fr
Phone : (33)-3-88-41-40-07  Fax : (33)-3-88-41-40-99

Re: [RFC] Possible bug with i386 watchpoints on several targets.

[Eli Zaretskii]

> Date: Fri, 30 Nov 2001 17:04:58 +0100
> From: Pierre Muller <muller@cerbere.u-strasbg.fr>
> 
> use the following command
> (top-gdb) watch gdb_stdout
> (top-gdb) r
> Here you should get a stop due to the setting of gdb_stdout value.
> (top-gdb) cont
> you should now reach the debuggee command prompt,
> simply quit.
> (gdb) quit
> and rerun the same executable without any watchpoint modification.
> (top-gdb) run
> if the bug that I found on both win32 (without i386_cleanup_deregs call)
> and on current (a few days old) CVS for linux.
> You won't get any stop on the second run.

This has nothing to do with i386_cleanup_dregs or, indeed, the x86
watchpoint support.  Type "maintenance show-debug-regs", and you will
see that the watchpoint does trigger on the second run, but GDB
ignores it.  It ignores the watchpoint because the old and the new
values compare equal on the second run, so GDB thinks it's a false
alarm.

The reason that hardware watchpoints are only considered to fire when
the watched value changes is that hardware watchpoints are treated teh
same as software watchpoints, and software watchpoints obviously
cannot fire unless the watched value changes.

I think it is fundamentally wrong to treat hardware and software
watchpoints in a similar way.  I think hardware watchpoints should be
treated like read and access watchpoints, not like software
watchpoints.  If others (mainly Michael Snyder) agree, I will submit a
patch that will make that change, and will also solve this particular
problem raised by Pierre.

Re: Bug with watchpoints on Linux

[Pierre Muller]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain, Size: 1136 bytes --]

At 19:25 21/11/2001 , Eli Zaretskii a écrit:
> > Date: Wed, 21 Nov 2001 13:01:59 +0100
> > From: Pierre Muller <muller@cerbere.u-strasbg.fr>
> > 
> >    But at the second run, the program is never stopped because
> > of the changes to this global variable.
> > 
> >    It seems like there is a problem with the hardware watchpoint
> > resetting.
> > 
> >    I tested this on only one Linux machine,
> > but both the main and the 5.1 branches show this problem.
>
>The DJGPP port doesn't have this problem.

   Nevertheless, according to my debug,
I expect that most targets using 
i386 hardware watchpoints and 
not calling i386_cleanup_dregs
will show this problem.

   I think that this is a really nasty bug for 
the i386 linux target for 5.1 release....

    What do we do about this?
I proposed to add a call to i386_cleanup_dregs
for all targets using the standard 
i386 hardware watchpoints, but 
Eli seems still not convinced by my reports....





Pierre Muller
Institut Charles Sadron
6,rue Boussingault
F 67083 STRASBOURG CEDEX (France)
mailto:muller@ics.u-strasbg.fr
Phone : (33)-3-88-41-40-07  Fax : (33)-3-88-41-40-99

Re: Bug with watchpoints on Linux

[Eli Zaretskii]

> Date: Fri, 30 Nov 2001 16:17:27 +0100
> From: Pierre Muller <muller@cerbere.u-strasbg.fr>
> 
>     What do we do about this?
> I proposed to add a call to i386_cleanup_dregs
> for all targets using the standard 
> i386 hardware watchpoints, but 
> Eli seems still not convinced by my reports....

You don't need to convince me (the port I'm responsible for _does_
use i386_cleanup_dregs, after all ;-).  You need to convince Mark
Kettenis.  I'm sure that if you post a test program and a full recipe
to reproduce the problem, and then show the patch which makes the
problem go away, Mark will happily accept it.

Btw, please use "diff -c" or 'diff -u" when you send diffs.  The
format you were using is very hard to read, and also is prone to
failures when applied with Patch, if the sources changed a bit since
the version you used to produce the diffs.

[RFC] Possible bug with i386 watchpoints on several targets.

[Pierre Muller]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain, Size: 2167 bytes --]

At 16:45 30/11/2001 , Eli Zaretskii a écrit:
> > Date: Fri, 30 Nov 2001 16:17:27 +0100
> > From: Pierre Muller <muller@cerbere.u-strasbg.fr>
> > 
> >     What do we do about this?
> > I proposed to add a call to i386_cleanup_dregs
> > for all targets using the standard 
> > i386 hardware watchpoints, but 
> > Eli seems still not convinced by my reports....
>
>You don't need to convince me (the port I'm responsible for _does_
>use i386_cleanup_dregs, after all ;-).  You need to convince Mark
>Kettenis.  I'm sure that if you post a test program and a full recipe
>to reproduce the problem, and then show the patch which makes the
>problem go away, Mark will happily accept it.

   The way to check if the bug exist is easy:

   take gdb (5.1 or current snapshot)
compiled with debug info and 
run
   ./gdb ./gdb
on the newly generated executable.
(preferabily in the compilation dir,
which will ensure that you get (top-gdb) prompt
for the debugger and (gdb) for the debuggee).

use the following command
(top-gdb) watch gdb_stdout
(top-gdb) r
Here you should get a stop due to the setting of gdb_stdout value.
(top-gdb) cont
you should now reach the debuggee command prompt,
simply quit.
(gdb) quit
and rerun the same executable without any watchpoint modification.
(top-gdb) run
if the bug that I found on both win32 (without i386_cleanup_deregs call)
and on current (a few days old) CVS for linux.
You won't get any stop on the second run.

The fix is not that obvious, because I am still a bit unclear
about wher it should be put:
   the i386_cleanup_dregs should be called (in my opinion)
both when a program is killed and when a program
terminates normally.

   But linux does seem to use generic_mourn_inferior,
so for this target it would probably require that we 
change this into a target specific function.

PS: Eli, did you check the go32v2 target if you kill the program after
the watchpoint is reached, sorry, I still can't get GDB to compile for djgpp
on my win2000.


Pierre Muller
Institut Charles Sadron
6,rue Boussingault
F 67083 STRASBOURG CEDEX (France)
mailto:muller@ics.u-strasbg.fr
Phone : (33)-3-88-41-40-07  Fax : (33)-3-88-41-40-99

Re: [RFC] Possible bug with i386 watchpoints on several targets.

[Eli Zaretskii]

> Date: Fri, 30 Nov 2001 17:04:58 +0100
> From: Pierre Muller <muller@cerbere.u-strasbg.fr>
> 
> use the following command
> (top-gdb) watch gdb_stdout
> (top-gdb) r
> Here you should get a stop due to the setting of gdb_stdout value.
> (top-gdb) cont
> you should now reach the debuggee command prompt,
> simply quit.
> (gdb) quit
> and rerun the same executable without any watchpoint modification.
> (top-gdb) run
> if the bug that I found on both win32 (without i386_cleanup_deregs call)
> and on current (a few days old) CVS for linux.
> You won't get any stop on the second run.

This has nothing to do with i386_cleanup_dregs or, indeed, the x86
watchpoint support.  Type "maintenance show-debug-regs", and you will
see that the watchpoint does trigger on the second run, but GDB
ignores it.  It ignores the watchpoint because the old and the new
values compare equal on the second run, so GDB thinks it's a false
alarm.

The reason that hardware watchpoints are only considered to fire when
the watched value changes is that hardware watchpoints are treated teh
same as software watchpoints, and software watchpoints obviously
cannot fire unless the watched value changes.

I think it is fundamentally wrong to treat hardware and software
watchpoints in a similar way.  I think hardware watchpoints should be
treated like read and access watchpoints, not like software
watchpoints.  If others (mainly Michael Snyder) agree, I will submit a
patch that will make that change, and will also solve this particular
problem raised by Pierre.

Re: [RFC] Possible bug with i386 watchpoints on several targets.

[Pierre Muller]

At 20:48 30/11/2001 , Eli Zaretskii a écrit:
> > Date: Fri, 30 Nov 2001 17:04:58 +0100
> > From: Pierre Muller <muller@cerbere.u-strasbg.fr>
> > 
> > use the following command
> > (top-gdb) watch gdb_stdout
> > (top-gdb) r
> > Here you should get a stop due to the setting of gdb_stdout value.
> > (top-gdb) cont
> > you should now reach the debuggee command prompt,
> > simply quit.
> > (gdb) quit
> > and rerun the same executable without any watchpoint modification.
> > (top-gdb) run
> > if the bug that I found on both win32 (without i386_cleanup_deregs call)
> > and on current (a few days old) CVS for linux.
> > You won't get any stop on the second run.
>
>This has nothing to do with i386_cleanup_dregs or, indeed, the x86
>watchpoint support.  Type "maintenance show-debug-regs", and you will
>see that the watchpoint does trigger on the second run, but GDB
>ignores it.  It ignores the watchpoint because the old and the new
>values compare equal on the second run, so GDB thinks it's a false
>alarm.

No, sorry, but I don't agree here:
I retested with today CVS compiled for i386 linux .

   Of course i386_stopped_data_address gets called, but if you 
set a breakpoint on the end of this function and add the condition
' cond X ret != 0'
Then you will clearly see that the behavior is wrong:
only on the first run after setting a hardware watchpoint you will get
a non zero return value for this watchpoint.

   The reason is that on the second run
i386_linux_drset is not called again on the second run.
And its not called again because in i386_insert_aligned_watchpoint
the macros 
   I386_DR_LOW_SET_ADDR (i, addr);
   I386_DR_LOW_SET_CONTROL (dr_control_mirror);
are only called if the the debug register is used for the first time.

   As the i386_cleanup_dregs is never called for linux target,
the internal state of the dr_mirror and dr_ref_count
is wrong on the second start.
dr_mirror[0] still contains the hardware watchpoint address
and dr_ref_count contains one.

  Thus the enabling of the hardware watchpoint only 
does increment dr_ref_count[0] (line 334)
and no linux specifc functions are called.

   This is exactly the same behaviour as 
what I got before calling i386_cleanup_dregs 
for the cygwin target.


>The reason that hardware watchpoints are only considered to fire when
>the watched value changes is that hardware watchpoints are treated teh
>same as software watchpoints, and software watchpoints obviously
>cannot fire unless the watched value changes.
>
>I think it is fundamentally wrong to treat hardware and software
>watchpoints in a similar way.  I think hardware watchpoints should be
>treated like read and access watchpoints, not like software
>watchpoints.  If others (mainly Michael Snyder) agree, I will submit a
>patch that will make that change, and will also solve this particular
>problem raised by Pierre.

There may be different solutions to the problem above, but I
think that in any case, we need to reset dr_mirror and dr_ref_count 
array at each new start.

I believe that this would be enough to solve all current troubles.

PS: A propos unwanted output of the cygwin target
using debug registers, I find the same sort of unwanted output for the linux target,
probably due to the removal/reinsertion of hardware watchpoints
when loading dynamic libraries.




Pierre Muller
Institut Charles Sadron
6,rue Boussingault
F 67083 STRASBOURG CEDEX (France)
mailto:muller@ics.u-strasbg.fr
Phone : (33)-3-88-41-40-07  Fax : (33)-3-88-41-40-99

Re: [RFC] Possible bug with i386 watchpoints on several targets.

[Eli Zaretskii]

On Mon, 3 Dec 2001, Pierre Muller wrote:

>    Of course i386_stopped_data_address gets called, but if you 
> set a breakpoint on the end of this function and add the condition
> ' cond X ret != 0'
> Then you will clearly see that the behavior is wrong:
> only on the first run after setting a hardware watchpoint you will get
> a non zero return value for this watchpoint.

i386_stopped_data_address has no role when a write-data hardware 
watchpoint is hit, only when read or access watchpoints are hit.  At 
least that's my reading of the code and what I see tracing through 
execution.  See breakpoint.c:bpstat_stop_status, you will see there that 
hardware watchpoints and rwatch/awatch watchpoints are treated 
differently, and i386_stopped_data_address (called via the 
target_stopped_data_address macro) is only called for awatch/rwatch.

The important function for hardware watchpoints is watchpoint_check, not 
i386_stopped_data_address.

>    As the i386_cleanup_dregs is never called for linux target,
> the internal state of the dr_mirror and dr_ref_count
> is wrong on the second start.
> dr_mirror[0] still contains the hardware watchpoint address
> and dr_ref_count contains one.

Then please explain why does the DJGPP port exhibits the same behavior as 
the Linux port with the test program you posted.  go32-nat.c _does_ call 
i386_cleanup_dregs, but the watchpoint doesn't trigger on the second run.

Re: [RFC] Possible bug with i386 watchpoints on several targets.

[Pierre Muller]

At 12:09 03/12/2001 , Eli Zaretskii a écrit:

>On Mon, 3 Dec 2001, Pierre Muller wrote:
>
> >    Of course i386_stopped_data_address gets called, but if you 
> > set a breakpoint on the end of this function and add the condition
> > ' cond X ret != 0'
> > Then you will clearly see that the behavior is wrong:
> > only on the first run after setting a hardware watchpoint you will get
> > a non zero return value for this watchpoint.
>
>i386_stopped_data_address has no role when a write-data hardware 
>watchpoint is hit, only when read or access watchpoints are hit.  At 
>least that's my reading of the code and what I see tracing through 
>execution.  See breakpoint.c:bpstat_stop_status, you will see there that 
>hardware watchpoints and rwatch/awatch watchpoints are treated 
>differently, and i386_stopped_data_address (called via the 
>target_stopped_data_address macro) is only called for awatch/rwatch.

That is not totally true, because
i386_stopped_by_data_address is also used in STOPPED_BY_WATCHPOINT
macro, which explains why I see that the code uses that function for
normal watchpoints.
But I agree that its called but not really used.

>The important function for hardware watchpoints is watchpoint_check, not 
>i386_stopped_data_address.
>
> >    As the i386_cleanup_dregs is never called for linux target,
> > the internal state of the dr_mirror and dr_ref_count
> > is wrong on the second start.
> > dr_mirror[0] still contains the hardware watchpoint address
> > and dr_ref_count contains one.
>
>Then please explain why does the DJGPP port exhibits the same behavior as 
>the Linux port with the test program you posted.  go32-nat.c _does_ call 
>i386_cleanup_dregs, but the watchpoint doesn't trigger on the second run.


   I now also managed to compile gdb for djgpp
and understood why I do get different results for cygwin and djgpp
targets:
    do to DLL loadings, the watchpoints are removed and reinserted at program startup
(same goes for linux programs that use shared libs)
and thus the correct starting state of the watchpoints are 
loaded again.

   On the contrary, this does never occur in djgpp target,
so that this target keeps the wrong value of the end of the previous run.

   So I finally agree with Eli to say that there a two problems here.
One relative to the fact that the watchpoint values are not correctly resetted
when the program restarts. But also a second problem if 
i386_cleanup_dregs is not called, because the target is not supposed to reuse
the debug register values of the previous run, and with the current implementation
not calling the i386_cleanup_dregs will result in
not explicitly setting the values of the debug registers
in the next run...



Pierre Muller
Institut Charles Sadron
6,rue Boussingault
F 67083 STRASBOURG CEDEX (France)
mailto:muller@ics.u-strasbg.fr
Phone : (33)-3-88-41-40-07  Fax : (33)-3-88-41-40-99

Re: [RFC] Possible bug with i386 watchpoints on several targets.

[Eli Zaretskii]

On Fri, 30 Nov 2001, I wrote:

> The reason that hardware watchpoints are only considered to fire when
> the watched value changes is that hardware watchpoints are treated teh
> same as software watchpoints, and software watchpoints obviously
> cannot fire unless the watched value changes.
> 
> I think it is fundamentally wrong to treat hardware and software
> watchpoints in a similar way.  I think hardware watchpoints should be
> treated like read and access watchpoints, not like software
> watchpoints.  If others (mainly Michael Snyder) agree, I will submit a
> patch that will make that change, and will also solve this particular
> problem raised by Pierre.

Michael, did you read this part of my message?  If so, do you agree with 
the principle?  If you do, I'd like to write a patch that will fix this.

Re: [RFC] Possible bug with i386 watchpoints on several targets.

[Pierre Muller]

At 08:49 05/12/2001 , Eli Zaretskii a écrit:

>On Fri, 30 Nov 2001, I wrote:
>
> > The reason that hardware watchpoints are only considered to fire when
> > the watched value changes is that hardware watchpoints are treated teh
> > same as software watchpoints, and software watchpoints obviously
> > cannot fire unless the watched value changes.
> > 
> > I think it is fundamentally wrong to treat hardware and software
> > watchpoints in a similar way.  I think hardware watchpoints should be
> > treated like read and access watchpoints, not like software
> > watchpoints.  If others (mainly Michael Snyder) agree, I will submit a
> > patch that will make that change, and will also solve this particular
> > problem raised by Pierre.
>
>Michael, did you read this part of my message?  If so, do you agree with 
>the principle?  If you do, I'd like to write a patch that will fix this.

Eli, you didn't answer if you agree that 
the call to i386_cleanup_dregs is required for all targets using
standard i386 hardware watchpoints.

I still think that the linux problem is related to that.
Your bug exists also there but its hidden by the fact
that loading shared libs at startup does set a correct value to
the value of the watched expression at the start of a program.

Which means that adding i386_cleanup_dregs call is not enough
to solve the complete problem, and that your fix is also needed.




Pierre Muller
Institut Charles Sadron
6,rue Boussingault
F 67083 STRASBOURG CEDEX (France)
mailto:muller@ics.u-strasbg.fr
Phone : (33)-3-88-41-40-07  Fax : (33)-3-88-41-40-99

Re: [RFC] Possible bug with i386 watchpoints on several targets.

[Eli Zaretskii]

On Wed, 5 Dec 2001, Pierre Muller wrote:

> Eli, you didn't answer if you agree that 
> the call to i386_cleanup_dregs is required for all targets using
> standard i386 hardware watchpoints.

I don't know; theoretically, GDB should remove all the watchpoints from 
the debuggee on the way to exit.  If all watchpoints are removed, the 
dr_* variables are cleaned up automatically.

But an explicit call to i386_cleanup_dregs when the debuggee dies cannot 
possibly do any harm, so there should be no reason not to do that, if it 
is sometimes needed.

Re: [RFC] Possible bug with i386 watchpoints on several targets.

[Pierre Muller]

At 12:22 05/12/2001 , vous avez écrit:

>On Wed, 5 Dec 2001, Pierre Muller wrote:
>
> > Eli, you didn't answer if you agree that 
> > the call to i386_cleanup_dregs is required for all targets using
> > standard i386 hardware watchpoints.
>
>I don't know; theoretically, GDB should remove all the watchpoints from 
>the debuggee on the way to exit.  If all watchpoints are removed, the 
>dr_* variables are cleaned up automatically.

But that is not the case for now...
I agree that this also should solve the problem,
but I don't know exactly how to do this, and if
there are reasons NOT to do so.

I think that for instance removing a normal breakpoint
will fail after program termination, if the breakpoint
is set by changing the .text section,
as the address will most probably not be accessible anymore.
  Processors different from ix86 might also
need access to the memory location to remove a watchpoint successfully.

We could use function remove_hw_watchpoints,
but hardware breakpoints should probably also be removed, no ?

I don't know exactly at which location this call should be made...

>But an explicit call to i386_cleanup_dregs when the debuggee dies cannot 
>possibly do any harm, so there should be no reason not to do that, if it 
>is sometimes needed.

    If your patch also does explicitly remove all hardware watchpoints, 
the it will indeed become unneeded  (probably also for go32-nat target).


Pierre Muller
Institut Charles Sadron
6,rue Boussingault
F 67083 STRASBOURG CEDEX (France)
mailto:muller@ics.u-strasbg.fr
Phone : (33)-3-88-41-40-07  Fax : (33)-3-88-41-40-99