Re: RFC: KFAILs [Was: [RFA/mi-testsuite] XFAIL mi*-console.exp]

Re: RFC: KFAILs [Was: [RFA/mi-testsuite] XFAIL mi*-console.exp]

[Michael Elizabeth Chastain]

I'm running dejagnu with the kfail patch all the time now, just for
the experience.  I put in some kfail's in gdb.c++/cplusfuncs.exp
but I can't commit them.

Another feature request ...

lib/framework.exp has a "proc xfail" which directly generates an xfail,
instead of calling "setup_xfail ... ; fail ... ".  There is also a
"proc xpass".

gdb uses "proc xfail".  So, could you add "proc kfail" and "proc kpass"?

Michael C

Re: RFC: KFAILs [Was: [RFA/mi-testsuite] XFAIL mi*-console.exp]

[Fernando Nasser]

Michael Elizabeth Chastain wrote:
> 
> I'm running dejagnu with the kfail patch all the time now, just for
> the experience.  I put in some kfail's in gdb.c++/cplusfuncs.exp
> but I can't commit them.
> 
> Another feature request ...
> 
> lib/framework.exp has a "proc xfail" which directly generates an xfail,
> instead of calling "setup_xfail ... ; fail ... ".  There is also a
> "proc xpass".
> 
> gdb uses "proc xfail".  So, could you add "proc kfail" and "proc kpass"?
> 

Sure.  Will do.


-- 
Fernando Nasser
Red Hat Canada Ltd.                     E-Mail:  fnasser@redhat.com
2323 Yonge Street, Suite #300
Toronto, Ontario   M4P 2C9

Re: RFC: KFAILs [Was: [RFA/mi-testsuite] XFAIL mi*-console.exp]

[Fernando Nasser]

Rob Savoye wrote:
> 
> On Fri, Apr 05, 2002 at 01:09:09PM -0600, Michael Elizabeth Chastain wrote:
> > Actually I prefer the pair "kpass/kfail" to "xpass/kfail" and "xpass/xfail"!
> > "xpass/kfail" looks weird and lets in some ambiguity.  Can you implement it
> > as "kpass/kfail"?
> 
>   I prefer kpass/kfail too.
> 

OK, I will make the changes (maybe tomorrow night) and repost.

-- 
Fernando Nasser
Red Hat Canada Ltd.                     E-Mail:  fnasser@redhat.com
2323 Yonge Street, Suite #300
Toronto, Ontario   M4P 2C9

Re: RFC: KFAILs [Was: [RFA/mi-testsuite] XFAIL mi*-console.exp]

[Rob Savoye]

On Sun, Apr 07, 2002 at 07:30:43PM -0500, Michael Elizabeth Chastain wrote:
> Surfety-surf-surf ...  I found something from Matt Bemis.  Is this it?

  Yep. 

> For me, this is incomplete, because it omits the ERROR and WARNING lines.
> I don't how what the experience is for other tools, but I get several
> ERRORs and WARNINGs almost every week when I test gdb.  If they get left
> out of gdb.xml then I will keep parsing gdb.sum.

  That can always be added. I just emailed Matt to see if he'll do it while
he's finishing the other changes.

	- rob -

Re: RFC: KFAILs [Was: [RFA/mi-testsuite] XFAIL mi*-console.exp]

[Michael Elizabeth Chastain]

Surfety-surf-surf ...  I found something from Matt Bemis.  Is this it?

  http://mail.gnu.org/pipermail/dejagnu/2002-March/000195.html

This has two parts.  First, every PASS, FAIL, XPASS, XFAIL, and so on,
gets an XML record.  Second, there is a summary section.  So it's a
pretty straightforward encoding of the foo.sum file into XML.

For me, this is incomplete, because it omits the ERROR and WARNING lines.
I don't how what the experience is for other tools, but I get several
ERRORs and WARNINGs almost every week when I test gdb.  If they get left
out of gdb.xml then I will keep parsing gdb.sum.

I would really like ERROR and WARNING to be treated the same as other
test results, with the script name printed and everything.

Michael C

Re: RFC: KFAILs [Was: [RFA/mi-testsuite] XFAIL mi*-console.exp]

[Rob Savoye]

On Sun, Apr 07, 2002 at 06:52:41PM -0500, Michael Elizabeth Chastain wrote:
> Rob Savoye writes:
> > I'm not sure if this will help in the bug tracking, but somebody just
> > added an option to make DejaGnu spit out all the testing output in XML.
> > That way the results can be sucked into a database for analysis.
> 
> That is interesting.  Is the DTD available somewhere?

  Hum... I'm not sure. I'll ping them and ask. They're tweaking the patch right
now. Originally it worked as more of a post-processor, but we all decided it'd
work better if it was merged into the output procedures. They're doing it for
cross GDB/GCC testing on their embedded targets, so it should be in good shape
for you to use.

	- rob -

Re: RFC: KFAILs [Was: [RFA/mi-testsuite] XFAIL mi*-console.exp]

[Michael Elizabeth Chastain]

Rob Savoye writes:
> I'm not sure if this will help in the bug tracking, but somebody just
> added an option to make DejaGnu spit out all the testing output in XML.
> That way the results can be sucked into a database for analysis.

That is interesting.  Is the DTD available somewhere?

Michael C

Re: RFC: KFAILs [Was: [RFA/mi-testsuite] XFAIL mi*-console.exp]

[Rob Savoye]

On Fri, Apr 05, 2002 at 01:14:32PM -0600, Michael Elizabeth Chastain wrote:

> My weekly reports already have clickable bug links in them, but I won't
> have to make them by hand.  On the other hand, several people will have
> to crawl through the existing attention lines and create bug reports for
> lots of things.

  I'm not sure if this will help in the bug tracking, but somebody just
added an option to make DejaGnu spit out all the testing output in XML.
That way the results can be sucked into a database for analysis. I should have
that merged in shortly. I've been working on a direct interface from DejaGnu
to MySQL as well for the same purpose.

  Once I have that tested, and the kpass/kfail done, I'll push out a 1.4.3
release.

	- rob -

Re: RFC: KFAILs [Was: [RFA/mi-testsuite] XFAIL mi*-console.exp]

[Rob Savoye]

On Fri, Apr 05, 2002 at 01:09:09PM -0600, Michael Elizabeth Chastain wrote:
> Actually I prefer the pair "kpass/kfail" to "xpass/kfail" and "xpass/xfail"!
> "xpass/kfail" looks weird and lets in some ambiguity.  Can you implement it
> as "kpass/kfail"?

  I prefer kpass/kfail too.

	- rob -

Re: RFC: KFAILs [Was: [RFA/mi-testsuite] XFAIL mi*-console.exp]

[Daniel Jacobowitz]

On Fri, Apr 05, 2002 at 04:45:41PM -0500, Andrew Cagney wrote:
> >You can access the Gnats database through a URL:
> >>
> >>http://sources.redhat.com/cgi-bin/gnatsweb.pl?database=gdb&cmd=view&pr=460
> >>
> >>For programmatic access, there may already be a more suitable "cmd" than
> >>"cmd=view", or someone may need to update gnatsweb.pl.
> >
> >
> >Don't use gnatsweb for this sort of thing at all.  If you look in the
> >gnats package (Debian: gnats-user) you find all sorts of good tools for
> >this.  /usr/bin/edit-pr and /usr/bin/query-pr, in particular.
> 
> How does one query a pr using e-mail?  I need to add it to the PROBLEMS 
> file.

I've no idea, but the GNATS documentation is doubtless on the web
somewhere.

-- 
Daniel Jacobowitz                           Carnegie Mellon University
MontaVista Software                         Debian GNU/Linux Developer

Re: RFC: KFAILs [Was: [RFA/mi-testsuite] XFAIL mi*-console.exp]

[Andrew Cagney]

> You can access the Gnats database through a URL:
>> 
>> http://sources.redhat.com/cgi-bin/gnatsweb.pl?database=gdb&cmd=view&pr=460
>> 
>> For programmatic access, there may already be a more suitable "cmd" than
>> "cmd=view", or someone may need to update gnatsweb.pl.
> 
> 
> Don't use gnatsweb for this sort of thing at all.  If you look in the
> gnats package (Debian: gnats-user) you find all sorts of good tools for
> this.  /usr/bin/edit-pr and /usr/bin/query-pr, in particular.

How does one query a pr using e-mail?  I need to add it to the PROBLEMS 
file.

Andrew

Re: RFC: KFAILs [Was: [RFA/mi-testsuite] XFAIL mi*-console.exp]

[Fernando Nasser]

Michael Elizabeth Chastain wrote:
> 
> Actually I prefer the pair "kpass/kfail" to "xpass/kfail" and "xpass/xfail"!
> "xpass/kfail" looks weird and lets in some ambiguity.  Can you implement it
> as "kpass/kfail"?
> 
> fna> I can change things to accept the second form like setup_xfail does
> fna> and just make sure that one with no '-' in it was found (the bug id)
> fna> and error out if none was found.  Just let me know if you prefer
> fna> this instead of the positional first argument (I am now having
> fna> second thoughts about that).
> 
> I don't have a strong preference.  (It sounds like you don't either).
> 

Yes, I don't mind one way or another.  But I am more inclined to keep
things as similar as the ones that exist for xfail to avoid confusion.
So, I will only make the bug identification mandatory and keep the
rest the same as for setup_xfail (unless someone has a good argument 
for us to do differently -- I could not think of a good one).

-- 
Fernando Nasser
Red Hat Canada Ltd.                     E-Mail:  fnasser@redhat.com
2323 Yonge Street, Suite #300
Toronto, Ontario   M4P 2C9

Re: RFC: KFAILs [Was: [RFA/mi-testsuite] XFAIL mi*-console.exp]

[Michael Elizabeth Chastain]

Daniel Jacobowitz writes:
> Don't use gnatsweb for this sort of thing at all.  If you look in the
> this.  /usr/bin/edit-pr and /usr/bin/query-pr, in particular.

Thanks.  I don't even know if the underlying database is web-accessible
for query-pr, though.  And I don't want to do research right now, because
this isn't on the 5.2 path.

> Don't get too hooked on these though.  I'd much rather see us move to
> Bugzilla, since it should be available on sources.redhat.com in a
> couple of weeks.

I figure I will have to have a modular "take this bug id and dereference it"
function anyways.

My weekly reports already have clickable bug links in them, but I won't
have to make them by hand.  On the other hand, several people will have
to crawl through the existing attention lines and create bug reports for
lots of things.

Michael C

Re: RFC: KFAILs [Was: [RFA/mi-testsuite] XFAIL mi*-console.exp]

[Michael Elizabeth Chastain]

Actually I prefer the pair "kpass/kfail" to "xpass/kfail" and "xpass/xfail"!
"xpass/kfail" looks weird and lets in some ambiguity.  Can you implement it
as "kpass/kfail"?

fna> I can change things to accept the second form like setup_xfail does
fna> and just make sure that one with no '-' in it was found (the bug id)
fna> and error out if none was found.  Just let me know if you prefer
fna> this instead of the positional first argument (I am now having
fna> second thoughts about that).

I don't have a strong preference.  (It sounds like you don't either).

Michael C

Re: RFC: KFAILs [Was: [RFA/mi-testsuite] XFAIL mi*-console.exp]

[Fernando Nasser]

Michael Elizabeth Chastain wrote:
> 
> Fernando Nasser writes:
> 
> fna> KFAIL: could not run to marker1 (PRMS gdb/999)
> fna> Would that make the scripts happy?
> 
> Err, I'm not sure if you mean the dejagnu scripts, or my scripts.
> My scripts are happy with this format.
> 

Your scripts.


> A lot of tests use "(...)" for various things, so the "PRMS"
> and "gdb/NNN" bits need to be mandatory in order to pick out this
> information from the noise.
> 
> fna> setup_kfail "gdb/999" *-*-*
> 
> Fine with me.  setup_kfail *-*-* "gdb/999" is fine with me as well.
> 

I can change things to accept the second form like setup_xfail does
and just make sure that one with no '-' in it was found (the bug id)
and error out if none was found.  Just let me know if you prefer
this instead of the positional first argument (I am now having
second thoughts about that).


> fna> 4) Note that, when a test that was expected to fail due to a known
> fna> bug suddenly starts to pass, it becomes a KPASS (as XFAILs do).
> 
> Okay, I added a KPASS column to my tables.
> 

Ouch!  That was a typo!!!

Sorry Michael, I meant "XPASS (as XFAILs do)".  There is no reason for
us to have a different category for that (I think).

My apologies.



> fna> I will do it in Perl (I still don't know how to programmatically access
> fna> the Gnats database though).  But I have very little spare time, so I
> fna> will not mind if someone that can do it sooner volunteers to do this.
> 
> You can access the Gnats database through a URL:
> 
>   http://sources.redhat.com/cgi-bin/gnatsweb.pl?database=gdb&cmd=view&pr=460
> 
> For programmatic access, there may already be a more suitable "cmd" than
> "cmd=view", or someone may need to update gnatsweb.pl.
> 
> I volunteer to write Perl analysis scripts.  My test bed is almost all Perl,
> and I am planning to release it.
> 

That would be great!  Thanks!

-- 
Fernando Nasser
Red Hat Canada Ltd.                     E-Mail:  fnasser@redhat.com
2323 Yonge Street, Suite #300
Toronto, Ontario   M4P 2C9

Re: RFC: KFAILs [Was: [RFA/mi-testsuite] XFAIL mi*-console.exp]

[Daniel Jacobowitz]

On Fri, Apr 05, 2002 at 11:53:30AM -0600, Michael Elizabeth Chastain wrote:
> Fernando Nasser writes:
> 
> fna> KFAIL: could not run to marker1 (PRMS gdb/999)
> fna> Would that make the scripts happy?
> 
> Err, I'm not sure if you mean the dejagnu scripts, or my scripts.
> My scripts are happy with this format.
> 
> A lot of tests use "(...)" for various things, so the "PRMS"
> and "gdb/NNN" bits need to be mandatory in order to pick out this
> information from the noise.
> 
> fna> setup_kfail "gdb/999" *-*-*
> 
> Fine with me.  setup_kfail *-*-* "gdb/999" is fine with me as well.
> 
> fna> 4) Note that, when a test that was expected to fail due to a known
> fna> bug suddenly starts to pass, it becomes a KPASS (as XFAILs do).
> 
> Okay, I added a KPASS column to my tables.
> 
> fna> I will do it in Perl (I still don't know how to programmatically access
> fna> the Gnats database though).  But I have very little spare time, so I
> fna> will not mind if someone that can do it sooner volunteers to do this.
> 
> You can access the Gnats database through a URL:
> 
>   http://sources.redhat.com/cgi-bin/gnatsweb.pl?database=gdb&cmd=view&pr=460
> 
> For programmatic access, there may already be a more suitable "cmd" than
> "cmd=view", or someone may need to update gnatsweb.pl.

Don't use gnatsweb for this sort of thing at all.  If you look in the
gnats package (Debian: gnats-user) you find all sorts of good tools for
this.  /usr/bin/edit-pr and /usr/bin/query-pr, in particular.

Don't get too hooked on these though.  I'd much rather see us move to
Bugzilla, since it should be available on sources.redhat.com in a
couple of weeks.


-- 
Daniel Jacobowitz                           Carnegie Mellon University
MontaVista Software                         Debian GNU/Linux Developer

Re: RFC: KFAILs [Was: [RFA/mi-testsuite] XFAIL mi*-console.exp]

[Michael Elizabeth Chastain]

Fernando Nasser writes:

fna> KFAIL: could not run to marker1 (PRMS gdb/999)
fna> Would that make the scripts happy?

Err, I'm not sure if you mean the dejagnu scripts, or my scripts.
My scripts are happy with this format.

A lot of tests use "(...)" for various things, so the "PRMS"
and "gdb/NNN" bits need to be mandatory in order to pick out this
information from the noise.

fna> setup_kfail "gdb/999" *-*-*

Fine with me.  setup_kfail *-*-* "gdb/999" is fine with me as well.

fna> 4) Note that, when a test that was expected to fail due to a known
fna> bug suddenly starts to pass, it becomes a KPASS (as XFAILs do).

Okay, I added a KPASS column to my tables.

fna> I will do it in Perl (I still don't know how to programmatically access
fna> the Gnats database though).  But I have very little spare time, so I
fna> will not mind if someone that can do it sooner volunteers to do this.

You can access the Gnats database through a URL:

  http://sources.redhat.com/cgi-bin/gnatsweb.pl?database=gdb&cmd=view&pr=460

For programmatic access, there may already be a more suitable "cmd" than
"cmd=view", or someone may need to update gnatsweb.pl.

I volunteer to write Perl analysis scripts.  My test bed is almost all Perl,
and I am planning to release it.

Michael C

RFC: KFAILs [Was: [RFA/mi-testsuite] XFAIL mi*-console.exp]

[Fernando Nasser]

Michael Elizabeth Chastain wrote:
> 
> What's the status of KFAIL?
> 

I have it here in my sandbox, but I will have to clean it up a
little bit this Sunday. I have a few questions and comments:


1) All current Dejagnu messages have the bug identification at
the end (between parenthesis).  I want to keep things consistent.
Is that OK?

2) Also, to keep things consistent in Dejagnu, the bug identification
for the kfail would be printed as "(PRMS xxxx)" as it is done
for XFAIL etc., to distinguish from bug_id which is printed as
"(BUG xxxx)".  Is that a problem?

So, a KFAIL would be printed as:

KFAIL: could not run to marker1 (PRMS gdb/999)

Would that make the scripts happy?


3) For KFAILs, the bug identification is mandatory.  In this case
I would like to change the syntax a little bit and have it as the
first argument and issue an error if it is not there.  I would 
still keep the current Dejagnu convention that bug identifications
are strings without a '-' in it.  I am doing this just to be able
to detect that someone forgot to specify it.  I know that there are
other syntax possibilities (like command switches), but I am trying
to be consistent with what is already there.

So, here is an example of a kfail use:

setup_kfail "gdb/999" *-*-*


4) Note that, when a test that was expected to fail due to a known
bug suddenly starts to pass, it becomes a KPASS (as XFAILs do).
It is then time to go and check if the bug was fixed and someone
forgot to remove the setup_kfail or if something changed in the
environment and the bug is not applicable anymore (or the test
is not catching it anymore).  More or less as we should do with
XPASSes that appear instead of XFAILs.  Also, sometimes the target
specification is a little bit too general (the problem does not
happen in all the specified conditions).


5) As soon as we have KFAILs, we will have to re-examine the FAILs
we have, register a bug for what is a gdb bug.  Also, we will be able
to have zero FAILs, so in any platform a non-zero result will mean that
we broke something.  We will also have to re-check everything that was
marked as XFAIL and see if they are actually known GDB bugs.



6) I would like to write a script that, as part of the release, goes 
through the KFAILs, access the Gnats database with the Bug Id and 
fills the BUGS part of the man page (can also generate a report and
put it in a file -- or append it to a Release Notes").  
We can also have a script to compare the new known bugs report with
the previous release one and generate a list of "Bugs fixed in this
release" (this can help people decide if they should upgrade, or stop
people to post just to hear: "I suggest you upgrade").

I will do it in Perl (I still don't know how to programmatically access
the Gnats database though).  But I have very little spare time, so I
will
not mind if someone that can do it sooner volunteers to do this.
 


-- 
Fernando Nasser
Red Hat Canada Ltd.                     E-Mail:  fnasser@redhat.com
2323 Yonge Street, Suite #300
Toronto, Ontario   M4P 2C9