Question about solaris CANNOT_STEP_HW_WATCHPOINTS macro

Question about solaris CANNOT_STEP_HW_WATCHPOINTS macro

[Pierre Muller]

   The macro
CANNOT_STEP_HW_WATCHPOINTS
is currently only defined at one location 
in the whole gdb sources:
gdb/config/i386/nm-i386sol2.h
with the following comment:
/* Solaris x86 2.6 and 2.7 targets have a kernel bug when stepping
   over an instruction that causes a page fault without triggering
   a hardware watchpoint. The kernel properly notices that it shouldn't
   stop, because the hardware watchpoint is not triggered, but it forgets
   the step request and continues the program normally.
   Work around the problem by removing hardware watchpoints if a step is
   requested, GDB will check for a hardware watchpoint trigger after the
   step anyway.  */

  I would like to know if any of the solaris gurus
around knows if this bug is still present in newer version of the solaris
kernel or at which version it has been fixed.

  I looked into OpenSolaris sources,
but watchpoint.c (which is where I suspect 
hardware watchpoints are handled)
was not updated since OpenSolaris launch in June 2005.

  On the other hand, the macro 
seems to have been introduced into the configuration file
back in April 2000 (revision 1.3 of current CVS tree).
  So this leaves about 5 years for a possible fix inside
the Solaris kernel.

  If this bug has been solve before 2.9 version of Solaris,
would it be reasonable to just drop that code out?

  Pieter, did you consider that issue when writing the
gdbserver port for Solaris?


Pierre Muller
Pascal language support maintainer for GDB

Re: Question about solaris CANNOT_STEP_HW_WATCHPOINTS macro

[Joel Brobecker]

>    The macro CANNOT_STEP_HW_WATCHPOINTS
[...]
> If this bug has been solve before 2.9 version of Solaris,
> would it be reasonable to just drop that code out?

I think we should still support version 2.8 if we can.

I'd be happy to do any testing with some of our machines running Solaris,
but I need a reproducer that I can run by hand.  I am still under strict
orders to *NOT* run the dejagnu testsuite on our solaris machines, as it
causes them to crash pretty badly :-(.

-- 
Joel

RE: Question about solaris CANNOT_STEP_HW_WATCHPOINTS macro

[Pierre Muller]

> -----Message d'origine-----
> De : gdb-owner@sourceware.org [mailto:gdb-owner@sourceware.org] De la
> part de Joel Brobecker
> Envoyé : Thursday, April 22, 2010 3:57 PM
> À : Pierre Muller
> Cc : gdb@sourceware.org; 'Pieter Maljaars'; 'Joseph S. Myers'; 'Pedro
> Alves'
> Objet : Re: Question about solaris CANNOT_STEP_HW_WATCHPOINTS macro
> 
> >    The macro CANNOT_STEP_HW_WATCHPOINTS
> [...]
> > If this bug has been solve before 2.9 version of Solaris,
> > would it be reasonable to just drop that code out?
> 
> I think we should still support version 2.8 if we can.

  I have no objections
 
> I'd be happy to do any testing with some of our machines running
> Solaris,
> but I need a reproducer that I can run by hand.  I am still under
> strict
> orders to *NOT* run the dejagnu testsuite on our solaris machines, as
> it
> causes them to crash pretty badly :-(.
 The main problem is that the comment inside
infrun.c is not that clear...

  The patch was discussed in 2000:
http://sourceware.org/ml/gdb-patches/2000-q1/msg00664.html
submitted by Peter Schauer.
  But I don't know if this person still is reachable.
I suspect that the problem appears if
 1) you are stepping
 2) you have a watchpoint on some memory location
 3) you modify some other location within the same 
protected page.
  The kernel stops because of the page protection,
sees that the accessed address is not the one you asked to 
watch, thus restarts the debuggee, forgetting that
stepping was required.

  This is at least how I understood it.

The problem is that I have no access
to a machine for which we can be sure that this bug 
still exists ...

With this source
struct rec
{
 int x;
 double y;
 int z;
};

static struct rec myrec;

int
main ()
{
  myrec.x = 5;
  myrec.y = 3.4;
  myrec.z = 56;
  myrec.x = 78;
  return myrec.x;
}

setting a watchpoint on myrec.x and
stepping should expose the bug if you
remove the CANNOT_STEP_HW_WATCHPOINT from nm-i386sol2.h


Pierre

Re: Question about solaris CANNOT_STEP_HW_WATCHPOINTS macro

[Peter Schauer]

Thanks Pierre, you are describing the problem perfectly well.

According to my notes the problem caused

FAIL: gdb.base/watchpoint.exp: until to ival1 assignment
FAIL: gdb.base/watchpoint.exp: until to ival3 assignment
FAIL: gdb.base/watchpoint.exp: until out of loop
FAIL: gdb.base/watchpoint.exp: step to ival2 assignment

regressions, but your example should trigger the problem as well.
IIRC, it happened on x86 Solaris 2.6 and early 2.7 systems, and was
fixed in 2.8 systems.

Unfortunately I have no longer access to any Solaris system, so
I can't be of any help here.

Sorry,

> > -----Message d'origine-----
> > De : gdb-owner@sourceware.org [mailto:gdb-owner@sourceware.org] De la
> > part de Joel Brobecker
> > Envoyé : Thursday, April 22, 2010 3:57 PM
> > À : Pierre Muller
> > Cc : gdb@sourceware.org; 'Pieter Maljaars'; 'Joseph S. Myers'; 'Pedro
> > Alves'
> > Objet : Re: Question about solaris CANNOT_STEP_HW_WATCHPOINTS macro
> > 
> > >    The macro CANNOT_STEP_HW_WATCHPOINTS
> > [...]
> > > If this bug has been solve before 2.9 version of Solaris,
> > > would it be reasonable to just drop that code out?
> > 
> > I think we should still support version 2.8 if we can.
> 
>   I have no objections
>  
> > I'd be happy to do any testing with some of our machines running
> > Solaris,
> > but I need a reproducer that I can run by hand.  I am still under
> > strict
> > orders to *NOT* run the dejagnu testsuite on our solaris machines, as
> > it
> > causes them to crash pretty badly :-(.
>  The main problem is that the comment inside
> infrun.c is not that clear...
> 
>   The patch was discussed in 2000:
> http://sourceware.org/ml/gdb-patches/2000-q1/msg00664.html
> submitted by Peter Schauer.
>   But I don't know if this person still is reachable.
> I suspect that the problem appears if
>  1) you are stepping
>  2) you have a watchpoint on some memory location
>  3) you modify some other location within the same 
> protected page.
>   The kernel stops because of the page protection,
> sees that the accessed address is not the one you asked to 
> watch, thus restarts the debuggee, forgetting that
> stepping was required.
> 
>   This is at least how I understood it.
> 
> The problem is that I have no access
> to a machine for which we can be sure that this bug 
> still exists ...
> 
> With this source
> struct rec
> {
>  int x;
>  double y;
>  int z;
> };
> 
> static struct rec myrec;
> 
> int
> main ()
> {
>   myrec.x = 5;
>   myrec.y = 3.4;
>   myrec.z = 56;
>   myrec.x = 78;
>   return myrec.x;
> }
> 
> setting a watchpoint on myrec.x and
> stepping should expose the bug if you
> remove the CANNOT_STEP_HW_WATCHPOINT from nm-i386sol2.h
> 
> 
> Pierre

-- 
Peter Schauer			Peter.Schauer@mytum.de

Re: Question about solaris CANNOT_STEP_HW_WATCHPOINTS macro

[Joel Brobecker]

> setting a watchpoint on myrec.x and
> stepping should expose the bug if you
> remove the CANNOT_STEP_HW_WATCHPOINT from nm-i386sol2.h

Looks like a different bug is now occurring:

    (gdb) start
    Temporary breakpoint 1 at 0x805067a: file foo.c, line 13.
    Starting program: [...]/foo 
    
    Temporary breakpoint 1, main () at foo.c:13
    13        myrec.x = 5;
    (gdb) print myrec.x
    $1 = 0
    (gdb) watch myrec.x
    Hardware watchpoint 2: myrec.x
    (gdb) s
    14        myrec.y = 3.4;

In other words, the program did not "continue" during the step, but
the watchpoint did not trigger either. Later on, during the same run:

    (gdb) s
    16        myrec.x = 78;
    (gdb) s
    17        return myrec.x;

(no trigger of the watchpoint either).

However, when doing a "continue" instead of a step, we do get the
watchpoint hit:

    (gdb) start
    Temporary breakpoint 1 at 0x805067a: file foo.c, line 13.
    Starting program: [...]/foo 
    
    Temporary breakpoint 1, main () at foo.c:13
    13        myrec.x = 5;
    (gdb) watch myrec.x
    Hardware watchpoint 2: myrec.x
    (gdb) cont
    Continuing.
    Hardware watchpoint 2: myrec.x
    
    Old value = 0
    New value = 5
    main () at foo.c:14
    14        myrec.y = 3.4;

I compared the behavior with the same program, but on x86-linux.
We get the expected behavior:

    (gdb) watch myrec.x
    Hardware watchpoint 2: myrec.x
    (gdb) s
    Hardware watchpoint 2: myrec.x
    
    Old value = 0
    New value = 5
    main () at foo.c:14
    14        myrec.y = 3.4;

Looking at the infrun debug output:

    (gdb) set debug infrun 1
    (gdb) s   
    infrun: clear_proceed_status_thread (LWP 1)
    infrun: proceed (addr=0xffffffff, signal=144, step=1)
    infrun: resume (step=1, signal=0), trap_expected=0
    infrun: wait_for_inferior (treat_exec_as_sigtrap=0)
    infrun: target_wait (-1, status) =
    infrun:   3497 [LWP 1],
    infrun:   status->kind = stopped, signal = SIGTRAP
    infrun: infwait_normal_state
    infrun: TARGET_WAITKIND_STOPPED
    infrun: stop_pc = 0x8050684
    infrun: stepped to a different line
    infrun: stop_stepping
    14        myrec.y = 3.4;

So we failed to notice that the watchpoint triggered - we should probably
look in the area of procfs_stopped_by_watchpoint. Maybe another kernel
issue???

If I use the "continue" command instead of a step, the infrun debug
output looks like this:

    (gdb) cont
    Continuing.
    infrun: clear_proceed_status_thread (LWP 1)
    infrun: proceed (addr=0xffffffff, signal=144, step=0)
    infrun: resume (step=0, signal=0), trap_expected=0
    infrun: wait_for_inferior (treat_exec_as_sigtrap=0)
    infrun: target_wait (-1, status) =
    infrun:   3524 [LWP 1],
    infrun:   status->kind = stopped, signal = SIGTRAP
    infrun: infwait_normal_state
    infrun: TARGET_WAITKIND_STOPPED
    infrun: stop_pc = 0x8050684
    infrun: stopped by watchpoint    <<<<<---------
    infrun: (no data address available)
    infrun: BPSTAT_WHAT_STOP_NOISY
    infrun: stop_stepping
    Hardware watchpoint 2: myrec.x
    
    Old value = 0
    New value = 5
    main () at foo.c:14
    14        myrec.y = 3.4;

I ran watchpoint.exp alone and the testcase passes without any problem.

One last thing: It does not make any difference whether the
CANNOT_STEP_HW_WATCHPOINT macro is defined or not.  So, I think that,
starting with version 2.8, it's safe to not have it defined.

-- 
Joel

Re: Question about solaris CANNOT_STEP_HW_WATCHPOINTS macro

[Pedro Alves]

On Friday 23 April 2010 00:59:56, Joel Brobecker wrote:
> > setting a watchpoint on myrec.x and
> > stepping should expose the bug if you
> > remove the CANNOT_STEP_HW_WATCHPOINT from nm-i386sol2.h
> 
> Looks like a different bug is now occurring:

...

> Looking at the infrun debug output:
> 
>     (gdb) set debug infrun 1
>     (gdb) s   
>     infrun: clear_proceed_status_thread (LWP 1)
>     infrun: proceed (addr=0xffffffff, signal=144, step=1)
>     infrun: resume (step=1, signal=0), trap_expected=0
>     infrun: wait_for_inferior (treat_exec_as_sigtrap=0)
>     infrun: target_wait (-1, status) =
>     infrun:   3497 [LWP 1],
>     infrun:   status->kind = stopped, signal = SIGTRAP
>     infrun: infwait_normal_state
>     infrun: TARGET_WAITKIND_STOPPED
>     infrun: stop_pc = 0x8050684
>     infrun: stepped to a different line
>     infrun: stop_stepping
>     14        myrec.y = 3.4;
> 
> So we failed to notice that the watchpoint triggered - we should probably
> look in the area of procfs_stopped_by_watchpoint. Maybe another kernel
> issue???

I think older GDBs handled this because they always checked for
watchpoint hits at every SIGTRAP.  Notice the comment in infrun.c:

>  /* Some targets (e.g. Solaris x86) have a kernel bug when stepping
>     over an instruction that causes a page fault without triggering
>     a hardware watchpoint. The kernel properly notices that it shouldn't
>     stop, because the hardware watchpoint is not triggered, but it forgets
>     the step request and continues the program normally.
>     Work around the problem by removing hardware watchpoints if a step is
>     requested, GDB will check for a hardware watchpoint trigger after the
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>     step anyway.  */
      ^^^^^^^^^^^

That underlined part of the comment is no longer true nowadays.

See PR9633 <http://sourceware.org/bugzilla/show_bug.cgi?id=9633>, which
points at some other target relying on it.

> If I use the "continue" command instead of a step, the infrun debug
> output looks like this:
> 
>     (gdb) cont
>     Continuing.
>     infrun: clear_proceed_status_thread (LWP 1)
>     infrun: proceed (addr=0xffffffff, signal=144, step=0)
>     infrun: resume (step=0, signal=0), trap_expected=0
>     infrun: wait_for_inferior (treat_exec_as_sigtrap=0)
>     infrun: target_wait (-1, status) =
>     infrun:   3524 [LWP 1],
>     infrun:   status->kind = stopped, signal = SIGTRAP
>     infrun: infwait_normal_state
>     infrun: TARGET_WAITKIND_STOPPED
>     infrun: stop_pc = 0x8050684
>     infrun: stopped by watchpoint    <<<<<---------
>     infrun: (no data address available)
>     infrun: BPSTAT_WHAT_STOP_NOISY
>     infrun: stop_stepping
>     Hardware watchpoint 2: myrec.x
>     
>     Old value = 0
>     New value = 5
>     main () at foo.c:14
>     14        myrec.y = 3.4;
> 
> I ran watchpoint.exp alone and the testcase passes without any problem.
> 
> One last thing: It does not make any difference whether the
> CANNOT_STEP_HW_WATCHPOINT macro is defined or not.  So, I think that,
> starting with version 2.8, it's safe to not have it defined.

Do we still care for <= 2.7 at this point?  (and before someone
spontaneously says "yes"; does GDB still work okay on such old
Solaris systems?)

-- 
Pedro Alves

Re: Question about solaris CANNOT_STEP_HW_WATCHPOINTS macro

[Joel Brobecker]

> Do we still care for <= 2.7 at this point?  (and before someone
> spontaneously says "yes"; does GDB still work okay on such old
> Solaris systems?)

To care and support 2.6 or 2.7, we need a contributor who can run and
fix bugs on those platforms.  AdaCore unfortunately cannot help since
the older version we have is 2.8, and so far, I am under the impression
that none of the usual contributors do.

We can send an email to gdb-announce if helps making the decision,
but at this point, I'm quite comfortable dropping 2.7 and older.

-- 
Joel

Re: Question about solaris CANNOT_STEP_HW_WATCHPOINTS macro

[Pieter Maljaars]

> Looks like a different bug is now occurring:
> 
>     (gdb) start
>     Temporary breakpoint 1 at 0x805067a: file foo.c, line 13.
>     Starting program: [...]/foo 
>     
>     Temporary breakpoint 1, main () at foo.c:13
>     13        myrec.x = 5;
>     (gdb) print myrec.x
>     $1 = 0
>     (gdb) watch myrec.x
>     Hardware watchpoint 2: myrec.x
>     (gdb) s
>     14        myrec.y = 3.4;

I have tested this with Solaris 2.10 and GDB-7.1. I have removed 
the CANNOT_STEP_HW_WATCHPOINT define.  Here this bug does not occur.

    (gdb) start
    Temporary breakpoint 1 at 0x8050681: file watchp.c, line 13.
    Starting program: [...]/watchp

    Temporary breakpoint 1, main () at watchp.c:13
    13        myrec.x = 5;
    (gdb) print myrec.x
    $1 = 0
    (gdb) watch myrec.x
    Hardware watchpoint 2: myrec.x
    (gdb) s
    Hardware watchpoint 2: myrec.x
    
    Old value = 0
    New value = 5
    main () at watchp.c:14
    14        myrec.y = 3.4;


Here it works as expected.  GDB stops also at the other watchpoint.
So with "step" and "continue" both watchpoints are triggered, like
it behaves on x86-linux.

Pieter Maljaars

RE: Question about solaris CANNOT_STEP_HW_WATCHPOINTS macro

[Pierre Muller]

  I reproduced Pieter's results and filed
a bug report:
http://sourceware.org/bugzilla/show_bug.cgi?id=11531

  This changes the scenery completely:
it becomes important that CANNOT_STEP_HW_WATCHPOINTS macro
does not get set.
  If, as Pedro said, GDB does not check for a hardware watchpoint 
trigger after the step anyway, the whole code of this macro
becomes useless and only creates problems.

  I will try to submit a testcase for this bug report.

Pierre Muller

> -----Message d'origine-----
> De : gdb-owner@sourceware.org [mailto:gdb-owner@sourceware.org] De la
> part de Pieter Maljaars
> Envoyé : Friday, April 23, 2010 9:42 AM
> À : Joel Brobecker; Pierre Muller
> Cc : Peter.Schauer@regent.e-technik.tu-muenchen.de; gdb@sourceware.org;
> 'Joseph S. Myers'; 'Pedro Alves'
> Objet : Re: Question about solaris CANNOT_STEP_HW_WATCHPOINTS macro
> 
> > Looks like a different bug is now occurring:
> >
> >     (gdb) start
> >     Temporary breakpoint 1 at 0x805067a: file foo.c, line 13.
> >     Starting program: [...]/foo
> >
> >     Temporary breakpoint 1, main () at foo.c:13
> >     13        myrec.x = 5;
> >     (gdb) print myrec.x
> >     $1 = 0
> >     (gdb) watch myrec.x
> >     Hardware watchpoint 2: myrec.x
> >     (gdb) s
> >     14        myrec.y = 3.4;
> 
> I have tested this with Solaris 2.10 and GDB-7.1. I have removed
> the CANNOT_STEP_HW_WATCHPOINT define.  Here this bug does not occur.
> 
>     (gdb) start
>     Temporary breakpoint 1 at 0x8050681: file watchp.c, line 13.
>     Starting program: [...]/watchp
> 
>     Temporary breakpoint 1, main () at watchp.c:13
>     13        myrec.x = 5;
>     (gdb) print myrec.x
>     $1 = 0
>     (gdb) watch myrec.x
>     Hardware watchpoint 2: myrec.x
>     (gdb) s
>     Hardware watchpoint 2: myrec.x
> 
>     Old value = 0
>     New value = 5
>     main () at watchp.c:14
>     14        myrec.y = 3.4;
> 
> 
> Here it works as expected.  GDB stops also at the other watchpoint.
> So with "step" and "continue" both watchpoints are triggered, like
> it behaves on x86-linux.
> 
> Pieter Maljaars

Re: Question about solaris CANNOT_STEP_HW_WATCHPOINTS macro

[Joel Brobecker]

>   I reproduced Pieter's results and filed
> a bug report:
> http://sourceware.org/bugzilla/show_bug.cgi?id=11531

Just a small suggestion when filing bug reports, if you can - it makes
it easier to figure out the problem being reported is when showing what
is happening and what should have happened. Also, confirming the command
used to build the program is also useful...

Note that, on my end, it looked like the CANNOT_STEP_HW_WATCHPOINTS
macro had no effect, so I am not sure I understand the PR.  But I
did the testing on x86 as opposed to x86_64, so perhaps that makes
a difference. Or maybe the OS version does - I tested on version 8.
I don't know if we have a machine that runs version 10.

-- 
Joel

Re: Question about solaris CANNOT_STEP_HW_WATCHPOINTS macro

[Pedro Alves]

On Friday 23 April 2010 08:41:50, Pieter Maljaars wrote:
> I have tested this with Solaris 2.10 and GDB-7.1. I have removed 
> the CANNOT_STEP_HW_WATCHPOINT define.  Here this bug does not occur.
> 
>     (gdb) start
>     Temporary breakpoint 1 at 0x8050681: file watchp.c, line 13.
>     Starting program: [...]/watchp
> 
>     Temporary breakpoint 1, main () at watchp.c:13
>     13        myrec.x = 5;
>     (gdb) print myrec.x
>     $1 = 0
>     (gdb) watch myrec.x
>     Hardware watchpoint 2: myrec.x
>     (gdb) s
>     Hardware watchpoint 2: myrec.x
>     
>     Old value = 0
>     New value = 5
>     main () at watchp.c:14
>     14        myrec.y = 3.4;
> 
> 
> Here it works as expected.  GDB stops also at the other watchpoint.
> So with "step" and "continue" both watchpoints are triggered, like
> it behaves on x86-linux.

(I assume this was x86 or x86-64 solaris, not sparc.)  If you 
_don't_ remove the CANNOT_STEP_HW_WATCHPOINT, then I suppose
you'll see watchpoints being missed when you step (since
the workaround works by removing watchpoints
when single-stepping) on Solaris 10 as well.

-- 
Pedro Alves

Re: Question about solaris CANNOT_STEP_HW_WATCHPOINTS macro

[Joel Brobecker]

> (I assume this was x86 or x86-64 solaris, not sparc.)  If you 
> _don't_ remove the CANNOT_STEP_HW_WATCHPOINT, then I suppose
> you'll see watchpoints being missed when you step (since
> the workaround works by removing watchpoints
> when single-stepping) on Solaris 10 as well.

Just for the record, I just finished testing on x86-solaris 10,
and same results as on 8...

-- 
Joel

RE: Question about solaris CANNOT_STEP_HW_WATCHPOINTS macro

[Pierre Muller]

> -----Message d'origine-----
> De : gdb-owner@sourceware.org [mailto:gdb-owner@sourceware.org] De la
> part de Joel Brobecker
> Envoyé : Friday, April 23, 2010 4:08 PM
> À : Pedro Alves
> Cc : pieter.maljaars@altenpts.nl; Pierre Muller;
> Peter.Schauer@regent.e-technik.tu-muenchen.de; gdb@sourceware.org;
> 'Joseph S. Myers'
> Objet : Re: Question about solaris CANNOT_STEP_HW_WATCHPOINTS macro
> 
> > (I assume this was x86 or x86-64 solaris, not sparc.)  If you
> > _don't_ remove the CANNOT_STEP_HW_WATCHPOINT, then I suppose
> > you'll see watchpoints being missed when you step (since
> > the workaround works by removing watchpoints
> > when single-stepping) on Solaris 10 as well.
> 
> Just for the record, I just finished testing on x86-solaris 10,
> and same results as on 8...

  We are getting troubles, but I
think that I understood what is going on:
  the problem is that it all depends on how you start your
program:
  If you use 'break main' followed by 'run',
you will have a breakpoint that is at the exact instruction
that should trigger the watchpoint hit.
  But currently GDB disables all breakpoints (including watchpoints)
to step over the breakpoint on main without hitting again the 'int 3'
instruction.
  If you use the 'start' command, a temporary breakpoint at the same
location
is created, but the breakpoint is removed by GDB.
  Thus the 'next' does not need to disable the breakpoints.
In that configuration you do get a failure with current GDB
due to CANNOT_STEP_OVER_HW_WATCHPOINTS macro that gets
resolved by removing that macro.

  I will send a RFA for this test to gdb-patches.

Pierre

Re: Question about solaris CANNOT_STEP_HW_WATCHPOINTS macro

[Pieter Maljaars]

> (I assume this was x86 or x86-64 solaris, not sparc.)  

Yes, it's x86. I forgot to mention it.

> _don't_ remove the CANNOT_STEP_HW_WATCHPOINT, then I suppose
> you'll see watchpoints being missed when you step (since
> the workaround works by removing watchpoints
> when single-stepping) on Solaris 10 as well.

That's correct.  The watchpoints are missed when stepping.  If 
I use "continue", GDB stops at the watchpoints.

Pieter