Re: RFC: gdb.c++/main-falloff.exp (a new KFAIL)

Re: RFC: gdb.c++/main-falloff.exp (a new KFAIL)

[Michael Elizabeth Chastain]

Daniel J wrote:
> Secondly, I really dislike this form.  Adding gdb_expect's all over is
> bad, because gdb_test has a much more thorough list of things to expect
> indicating various errors.  Better would be to solve this problem with
> a little TCL.  What do you think of:
>   gdb_test_multiple "info locals" \
> 	{pass "(i|j|k) = (101|102|103)\r\n(i|j|k) = (101|102|103)\r\n(i|j|k) = (101|102|103)"
> 	 kfail "gdb/900" "No locals."} \
> 	"testing locals"

David C replies:
> It would be nice if the branches could execute arbitrary code, like
> gdb_expect does, though, so that the xfails/kfails could be conditional
> on the operating system, debug format, or whatever.

I'm changing my mind about the gdb_test_multiple approach.  I'm not
opposed to gdb_test_multiple, but I don't want KFAIL activity to
wait for it.

My original goals, back around April 2002, were:

(1) provide a way to add new tests which show bugs in gdb.

For example, look at PR gdb/186, "gdb have problems with C++ casting".

  http://sources.redhat.com/cgi-bin/gnatsweb.pl?cmd=view&database=gdb&pr=186

I have test code for this.  It's not even a new test case; it is
more test code for gdb.c++/casts.exp (which does not cover classes
that have virtual functions).

My understanding is that it's forbidden to add new tests which FAIL,
but acceptable to add new tests which KFAIL.

I would like to commit my new tests and have them KFAIL with reference
to PR gdb/186.  We talked about problems like this 9 months ago and
KFAIL is the solution that Fernando picked.

(2) connect existing FAILs to the PR database.

We have dozens of tests that already FAIL due to known reasons.
I think everybody wants to start marking those with KFAIL.

I'm getting dismayed by the new turn of events where KFAIL deployment
is sprouting a dependency on new syntax in lib/gdb.exp which needs
to be designed and implemented.

I would rather do these things in parallel.  There are already plenty
of tests which use send_gdb/gdb_expect.  If someone wants to implement a
better facility than send_gdb/gdb_expect, go for it, I will support such
an effort.  As soon as it's available then I will convert gdb.c++/*.exp
to use it.  But I no longer want to hold off on KFAIL activity to wait
for gdb_test_multiple.

Michael C

Re: RFC: gdb.c++/main-falloff.exp (a new KFAIL)

[David Carlton]

On Fri, 3 Jan 2003 15:16:43 -0600, Michael Elizabeth Chastain <mec@shout.net> said:

> I'm changing my mind about the gdb_test_multiple approach.  I'm not
> opposed to gdb_test_multiple, but I don't want KFAIL activity to
> wait for it.

I completely agree with this.  It seems to me that, for now, we should
just add KFAIL's using either gdb_expect or using setup_kfail+gdb_test
(with both being acceptable, depending on the test writer's
preferences).  If somebody eventually writes a spiffy
gdb_test_multiple which unifies both of their virtues, then we can go
back and convert those new KFAILs to use that format (along with
existing gdb_expects).  My message was only meant to brainstorm on the
possible design of such a gdb_test_multiple.

David Carlton
carlton@math.stanford.edu

Re: RFC: gdb.c++/main-falloff.exp (a new KFAIL)

[Daniel Jacobowitz]

On Fri, Jan 03, 2003 at 03:16:43PM -0600, Michael Elizabeth Chastain wrote:
> Daniel J wrote:
> > Secondly, I really dislike this form.  Adding gdb_expect's all over is
> > bad, because gdb_test has a much more thorough list of things to expect
> > indicating various errors.  Better would be to solve this problem with
> > a little TCL.  What do you think of:
> >   gdb_test_multiple "info locals" \
> > 	{pass "(i|j|k) = (101|102|103)\r\n(i|j|k) = (101|102|103)\r\n(i|j|k) = (101|102|103)"
> > 	 kfail "gdb/900" "No locals."} \
> > 	"testing locals"
> 
> David C replies:
> > It would be nice if the branches could execute arbitrary code, like
> > gdb_expect does, though, so that the xfails/kfails could be conditional
> > on the operating system, debug format, or whatever.
> 
> I'm changing my mind about the gdb_test_multiple approach.  I'm not
> opposed to gdb_test_multiple, but I don't want KFAIL activity to
> wait for it.
> 
> My original goals, back around April 2002, were:
> 
> (1) provide a way to add new tests which show bugs in gdb.
> 
> For example, look at PR gdb/186, "gdb have problems with C++ casting".
> 
>   http://sources.redhat.com/cgi-bin/gnatsweb.pl?cmd=view&database=gdb&pr=186
> 
> I have test code for this.  It's not even a new test case; it is
> more test code for gdb.c++/casts.exp (which does not cover classes
> that have virtual functions).
> 
> My understanding is that it's forbidden to add new tests which FAIL,
> but acceptable to add new tests which KFAIL.
> 
> I would like to commit my new tests and have them KFAIL with reference
> to PR gdb/186.  We talked about problems like this 9 months ago and
> KFAIL is the solution that Fernando picked.
> 
> (2) connect existing FAILs to the PR database.
> 
> We have dozens of tests that already FAIL due to known reasons.
> I think everybody wants to start marking those with KFAIL.
> 
> I'm getting dismayed by the new turn of events where KFAIL deployment
> is sprouting a dependency on new syntax in lib/gdb.exp which needs
> to be designed and implemented.
> 
> I would rather do these things in parallel.  There are already plenty
> of tests which use send_gdb/gdb_expect.  If someone wants to implement a
> better facility than send_gdb/gdb_expect, go for it, I will support such
> an effort.  As soon as it's available then I will convert gdb.c++/*.exp
> to use it.  But I no longer want to hold off on KFAIL activity to wait
> for gdb_test_multiple.

TCL is not as fierce as everyone seems to think it is!  Really. 
gdb_test_multiple will only take me an hour tops to put together; I
just wanted to get at least a little feedback on the syntax first.

If it's bugging you that bad I'll do it or something similar tomorrow morning.

-- 
Daniel Jacobowitz
MontaVista Software                         Debian GNU/Linux Developer

Re: RFC: gdb.c++/main-falloff.exp (a new KFAIL)

[Michael Elizabeth Chastain]

Daniel J rebuffs me with:
> No, please wait until tomorrow so that I can give you something less
> error prone.

Okay.  Heck, for you: Monday.

Monday, I wanna start the KFAIL bacchanalia with whatever syntax
is available at that time.  Deal?

Michael C

Re: RFC: gdb.c++/main-falloff.exp (a new KFAIL)

[Michael Elizabeth Chastain]

As David C mentioned, gdb_test_multiple cannot replace send_gdb/gdb_expect,
because there is no room in the syntax to add TCL code blocks
which some tests use to check compiler version, target version.

> If it's bugging you that bad I'll do it or something similar tomorrow
> morning.

That would be great.

But: is it okay with you if we de-couple these things and proceed
with the KFAIL bacchanal?

Michael C
pushy new yorker

Re: RFC: gdb.c++/main-falloff.exp (a new KFAIL)

[Daniel Jacobowitz]

On Fri, Jan 03, 2003 at 03:45:03PM -0600, Michael Elizabeth Chastain wrote:
> As David C mentioned, gdb_test_multiple cannot replace send_gdb/gdb_expect,
> because there is no room in the syntax to add TCL code blocks
> which some tests use to check compiler version, target version.

That's why we don't have a fixed syntax yet.  I have some better ideas.

> > If it's bugging you that bad I'll do it or something similar tomorrow
> > morning.
> 
> That would be great.
> 
> But: is it okay with you if we de-couple these things and proceed
> with the KFAIL bacchanal?
> 
> Michael C
> pushy new yorker

No, please wait until tomorrow so that I can give you something less
error prone.

Daniel J
pushy pittsburgher

Re: RFC: gdb.c++/main-falloff.exp (a new KFAIL)

[Michael Elizabeth Chastain]

Hi Daniel!

> First of all, KFAIL is (in my opinion) for things that we have analyzed
> and established to be known bugs _in the tool under test_.  That's what
> differentiates them from XFAILs; I thought that was the consensus.

You are right.  I am mixing two issues here.  I agree with you,
but my code doesn't.  :-(

I would like to file a gcc bug on this and then it would be an xfail.
It's only a kfail because I am so conservative about marking bugs
as "gdb bugs" until proven otherwise.  In the past we have been
lax about blaming things as xfail prematurely.

> I.E. the "return 0" is outside of the lexical block for main.  That's
> not necessarily wrong.  We have to decide if it is wrong - whether the
> test case should be updated or a GCC bug report filed.  My inclination
> is that it's a GCC bug.

Me too.  How about if I file it as such, and then make this an XFAIL?

> GDB is behaving exactly as expected given its inputs; ergo, this is not
> a KFAIL at all.

POW.  Ya got me.

> What do you think of:
>   gdb_test_multiple "info locals" \
> 	{pass "(i|j|k) = (101|102|103)\r\n(i|j|k) = (101|102|103)\r\n(i|j|k) = (101|102|103)"
> 	 kfail "gdb/900" "No locals."} \
> 	"testing locals"

I am open to new syntax.  I do prefer gdb_test to send_gdb/gdb_expect.
I never thought of extending the gdb_test idea but it's a good idea.

So if you're cool with me filing a gcc bug report, I can s/kfail/xfail/,
close PR gdb/900 as "not a gdb bug -- see PR gcc/9NNN", and we can
wrangle about gdb_test_multiple.

I will definitely suspend committing this for a while.

Michael C

Re: RFC: gdb.c++/main-falloff.exp (a new KFAIL)

[Daniel Jacobowitz]

On Mon, Dec 30, 2002 at 10:04:54AM -0600, Michael Elizabeth Chastain wrote:
> Hi Daniel!
> 
> > First of all, KFAIL is (in my opinion) for things that we have analyzed
> > and established to be known bugs _in the tool under test_.  That's what
> > differentiates them from XFAILs; I thought that was the consensus.
> 
> You are right.  I am mixing two issues here.  I agree with you,
> but my code doesn't.  :-(
> 
> I would like to file a gcc bug on this and then it would be an xfail.
> It's only a kfail because I am so conservative about marking bugs
> as "gdb bugs" until proven otherwise.  In the past we have been
> lax about blaming things as xfail prematurely.
> 
> > I.E. the "return 0" is outside of the lexical block for main.  That's
> > not necessarily wrong.  We have to decide if it is wrong - whether the
> > test case should be updated or a GCC bug report filed.  My inclination
> > is that it's a GCC bug.
> 
> Me too.  How about if I file it as such, and then make this an XFAIL?
> 
> > GDB is behaving exactly as expected given its inputs; ergo, this is not
> > a KFAIL at all.
> 
> POW.  Ya got me.
> 
> > What do you think of:
> >   gdb_test_multiple "info locals" \
> > 	{pass "(i|j|k) = (101|102|103)\r\n(i|j|k) = (101|102|103)\r\n(i|j|k) = (101|102|103)"
> > 	 kfail "gdb/900" "No locals."} \
> > 	"testing locals"
> 
> I am open to new syntax.  I do prefer gdb_test to send_gdb/gdb_expect.
> I never thought of extending the gdb_test idea but it's a good idea.
> 
> So if you're cool with me filing a gcc bug report, I can s/kfail/xfail/,
> close PR gdb/900 as "not a gdb bug -- see PR gcc/9NNN", and we can
> wrangle about gdb_test_multiple.
> 
> I will definitely suspend committing this for a while.

Sounds good to me.  Give me about an hour first; I'm looking at the GCC
bug.

-- 
Daniel Jacobowitz
MontaVista Software                         Debian GNU/Linux Developer

RFC: gdb.c++/main-falloff.exp (a new KFAIL)

[Michael Elizabeth Chastain]

This is an RFC for main-falloff.exp, a new test script with a single
KFAIL.

This script tests a C++ main() function which has return type int but does
not return a value.  I don't like it, but this is legal C++.  I noticed
this happening with anon-union.exp on 2002-12-18.  anon-union.exp is
exposing another gcc bug so I am writing a separate test just for the
the fall-off-main behavior.

I'd like to get comments on the KFAIL aspect.  main-falloff.exp has just
one useful test, which currently KFAIL's with gcc HEAD%20021224 -gdwarf-2.
I have written the KFAIL code the way that I would like to use KFAIL.

. Is this an acceptable way to use KFAIL?
. Is this a good way to use KFAIL?
. Should we ask/require everyone to do KFAIL like this?

I know I'm probably being a little pedantic about this, I just want to
have a discussion before we find ourselves with four different KFAIL
philosophies in our test suite.  I won't mind at all doing it whatever
way that FernandoN and AndrewC think is best.

Testing: tested on my native i686-pc-linux-gnu test bed with gcc v2
and v3, dwarf-2 and stabs+.  It PASSes with all stabs+ configurations
and with dwarf-2 in 2.95.3, 3.2.1, gcc-3_2-branch, and gcc-3_3-branch.
It KFAILs with gcc HEAD -gdwarf-2.

Michael C

== main-falloff.cc

int main ()
{
  int i, j, k;

  i = 101;
  j = 102;
  k = 103;
}

=== main-falloff.exp

# Copyright 2002 Free Software Foundation, Inc.
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
# 
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
# 
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.  
#
# Please email any bugs, comments, and/or additions to this file to:
# bug-gdb@prep.ai.mit.edu
#
# Written by Michael Elizabeth Chastain (mec@shout.net)
# This file is part of the gdb testsuite.
#
# Stroustrup, "The C++ Programming Language", 3rd edition, section 3.2, says:
#
#   "Every C++ program must have a function named main().
#    The program starts by executing that function.
#    The int value returned by main(), if any, is the
#    program's return value to ``the system'.''
#    If no value is returned, the system will receive a value
#    indicating successful completion.  A nonzero return
#    value from main() indicates failure."
#
# gcc has special code to handle this wart where main() can legally fall
# off the end.  I have seen bugs in the special code, hence this test case.

if $tracelevel then {
   strace $tracelevel
}

if { [skip_cplus_tests] } { continue }

set prms_id 0
set bug_id 0

set testfile "main-falloff"
set srcfile ${testfile}.cc
set binfile ${objdir}/${subdir}/${testfile}

if { [gdb_compile "${srcdir}/${subdir}/${srcfile}" "${binfile}" executable {debug c++}] != "" } {
  gdb_suppress_entire_file "Testcase compile failed, so all tests in this file will automatically fail."
}

if [get_compiler_info ${binfile} "c++"] {
  return -1
}

gdb_exit
gdb_start
gdb_reinitialize_dir $srcdir/$subdir
gdb_load ${binfile}

if ![runto_main] then {
  perror "couldn't run to breakpoint"
  continue
}

# next on through to the closing brace.

gdb_test "next 3" "8.*" "next 3"

# print the locals

send_gdb "info locals\n"
gdb_expect {
  -re "(i|j|k) = (101|102|103)\r\n(i|j|k) = (101|102|103)\r\n(i|j|k) = (101|102|103)\r\n$gdb_prompt $" {
    pass "info locals"
  }
  -re "No locals.\r\n$gdb_prompt $" {
    # The locals have disappeared just before the end of the function.
    # This happened with gcc HEAD%20021224 -gdwarf-2.
    # -- chastain 2002-12-29
    kfail "gdb/900" "info locals"
  }
  -re ".*$gdb_prompt $" {
    fail "info locals"
  }
  timeout {
    fail "info locals (timeout)"
  }
}

gdb_exit
return 0

Re: RFC: gdb.c++/main-falloff.exp (a new KFAIL)

[Daniel Jacobowitz]

On Mon, Dec 30, 2002 at 12:12:53AM -0600, Michael Elizabeth Chastain wrote:
> This is an RFC for main-falloff.exp, a new test script with a single
> KFAIL.
> 
> This script tests a C++ main() function which has return type int but does
> not return a value.  I don't like it, but this is legal C++.  I noticed
> this happening with anon-union.exp on 2002-12-18.  anon-union.exp is
> exposing another gcc bug so I am writing a separate test just for the
> the fall-off-main behavior.
> 
> I'd like to get comments on the KFAIL aspect.  main-falloff.exp has just
> one useful test, which currently KFAIL's with gcc HEAD%20021224 -gdwarf-2.
> I have written the KFAIL code the way that I would like to use KFAIL.
> 
> . Is this an acceptable way to use KFAIL?
> . Is this a good way to use KFAIL?
> . Should we ask/require everyone to do KFAIL like this?
> 
> I know I'm probably being a little pedantic about this, I just want to
> have a discussion before we find ourselves with four different KFAIL
> philosophies in our test suite.  I won't mind at all doing it whatever
> way that FernandoN and AndrewC think is best.

I've got two separate complaints.

First of all, KFAIL is (in my opinion) for things that we have analyzed
and established to be known bugs _in the tool under test_.  That's what
differentiates them from XFAILs; I thought that was the consensus.  I
don't see that kind of analysis having happened yet; and in fact I see
reason to believe that it isn't the case.

The bug in this case is apparently:

  10:   c7 45 fc 65 00 00 00    movl   $0x65,0xfffffffc(%ebp)
  17:   c7 45 f8 66 00 00 00    movl   $0x66,0xfffffff8(%ebp)
  1e:   c7 45 f4 67 00 00 00    movl   $0x67,0xfffffff4(%ebp)
  25:   b8 00 00 00 00          mov    $0x0,%eax

 <2><3f>: Abbrev Number: 3 (DW_TAG_lexical_block)
     DW_AT_low_pc      : 0x10 16
     DW_AT_high_pc     : 0x25 37

I.E. the "return 0" is outside of the lexical block for main.  That's
not necessarily wrong.  We have to decide if it is wrong - whether the
test case should be updated or a GCC bug report filed.  My inclination
is that it's a GCC bug.

GDB is behaving exactly as expected given its inputs; ergo, this is not
a KFAIL at all.

> send_gdb "info locals\n"
> gdb_expect {
>   -re "(i|j|k) = (101|102|103)\r\n(i|j|k) = (101|102|103)\r\n(i|j|k) = (101|102|103)\r\n$gdb_prompt $" {
>     pass "info locals"
>   }
>   -re "No locals.\r\n$gdb_prompt $" {
>     # The locals have disappeared just before the end of the function.
>     # This happened with gcc HEAD%20021224 -gdwarf-2.
>     # -- chastain 2002-12-29
>     kfail "gdb/900" "info locals"
>   }
>   -re ".*$gdb_prompt $" {
>     fail "info locals"
>   }
>   timeout {
>     fail "info locals (timeout)"
>   }
> }

Secondly, I really dislike this form.  Adding gdb_expect's all over is
bad, because gdb_test has a much more thorough list of things to expect
indicating various errors.  Better would be to solve this problem with
a little TCL.  What do you think of:
  gdb_test_multiple "info locals" \
	{pass "(i|j|k) = (101|102|103)\r\n(i|j|k) = (101|102|103)\r\n(i|j|k) = (101|102|103)"
	 kfail "gdb/900" "No locals."} \
	"testing locals"

-- 
Daniel Jacobowitz
MontaVista Software                         Debian GNU/Linux Developer

Re: RFC: gdb.c++/main-falloff.exp (a new KFAIL)

[David Carlton]

On Mon, 30 Dec 2002 10:46:41 -0500, Daniel Jacobowitz <drow@mvista.com> said:

> Secondly, I really dislike this form.  Adding gdb_expect's all over is
> bad, because gdb_test has a much more thorough list of things to expect
> indicating various errors.  Better would be to solve this problem with
> a little TCL.  What do you think of:
>   gdb_test_multiple "info locals" \
> 	{pass "(i|j|k) = (101|102|103)\r\n(i|j|k) = (101|102|103)\r\n(i|j|k) = (101|102|103)"
> 	 kfail "gdb/900" "No locals."} \
> 	"testing locals"

Something like this seems reasonable to me.  It would be nice if the
branches could execute arbitrary code, like gdb_expect does, though,
so that the xfails/kfails could be conditional on the operating
system, debug format, or whatever.  Though I do like the fact that
your version means that you only have to type the message once for all
branches (and presumably the message would be optional if the message
is the same as the command).

David Carlton
carlton@math.stanford.edu