[rfa/testsuite] Don't display values in output of pc-fp.exp

[rfa/testsuite] Don't display values in output of pc-fp.exp

[David Carlton]

The tests in pc-fp.exp display the actual values of $pc and $fp in
their PASS messages.  That makes regression testing a bit more
annoying for me: the value of $fp changes with every change I make to
GDB.  Is it okay to tweak the PASS messages not to display those
values?

David Carlton
carlton@math.stanford.edu

2002-11-05  David Carlton  <carlton@math.stanford.edu>

	* gdb.base/pc-fp.exp (get_valueofx): Don't display 'val' in PASS
	message.

Index: pc-fp.exp
===================================================================
RCS file: /cvs/src/src/gdb/testsuite/gdb.base/pc-fp.exp,v
retrieving revision 1.1
diff -u -p -r1.1 pc-fp.exp
--- pc-fp.exp	25 Sep 2002 20:30:38 -0000	1.1
+++ pc-fp.exp	5 Nov 2002 22:09:51 -0000
@@ -62,7 +62,7 @@ proc get_valueofx { fmt exp default } {
     gdb_expect {
 	-re "\\$\[0-9\]* = (0x\[0-9a-zA-Z\]+).*$gdb_prompt $" {
 	    set val $expect_out(1,string)
-	    pass "get value of ${exp} ($val)"
+	    pass "get value of ${exp}"
 	}
 	timeout {
 	    set size ${default}

Re: [rfa/testsuite] Don't display values in output of pc-fp.exp

[Michael Snyder]

David Carlton wrote:
> 
> The tests in pc-fp.exp display the actual values of $pc and $fp in
> their PASS messages.  That makes regression testing a bit more
> annoying for me: the value of $fp changes with every change I make to
> GDB.  Is it okay to tweak the PASS messages not to display those
> values?
> 
> David Carlton

I'm not the maintainer, but the suggestion seems sound.

> carlton@math.stanford.edu
> 
> 2002-11-05  David Carlton  <carlton@math.stanford.edu>
> 
>         * gdb.base/pc-fp.exp (get_valueofx): Don't display 'val' in PASS
>         message.
> 
> Index: pc-fp.exp
> ===================================================================
> RCS file: /cvs/src/src/gdb/testsuite/gdb.base/pc-fp.exp,v
> retrieving revision 1.1
> diff -u -p -r1.1 pc-fp.exp
> --- pc-fp.exp   25 Sep 2002 20:30:38 -0000      1.1
> +++ pc-fp.exp   5 Nov 2002 22:09:51 -0000
> @@ -62,7 +62,7 @@ proc get_valueofx { fmt exp default } {
>      gdb_expect {
>         -re "\\$\[0-9\]* = (0x\[0-9a-zA-Z\]+).*$gdb_prompt $" {
>             set val $expect_out(1,string)
> -           pass "get value of ${exp} ($val)"
> +           pass "get value of ${exp}"
>         }
>         timeout {
>             set size ${default}

Re: [rfa/testsuite] Don't display values in output of pc-fp.exp

[Andrew Cagney]

[-- Attachment #1: Type: text/plain, Size: 1317 bytes --]

> The tests in pc-fp.exp display the actual values of $pc and $fp in
> their PASS messages.  That makes regression testing a bit more
> annoying for me: the value of $fp changes with every change I make to
> GDB.  Is it okay to tweak the PASS messages not to display those
> values?


As far as I know, anything in trailing paren should be ignored when 
comparing test results.  You might want to tweak your script (I've 
attached mine) to do this.

Also, why is FP/PC changing?  Your GDB changes shouldn't affect the 
behavior of the target program's $fp / $pc.

Andrew



> 2002-11-05  David Carlton  <carlton@math.stanford.edu>
> 
> 	* gdb.base/pc-fp.exp (get_valueofx): Don't display 'val' in PASS
> 	message.
> 
> Index: pc-fp.exp
> ===================================================================
> RCS file: /cvs/src/src/gdb/testsuite/gdb.base/pc-fp.exp,v
> retrieving revision 1.1
> diff -u -p -r1.1 pc-fp.exp
> --- pc-fp.exp	25 Sep 2002 20:30:38 -0000	1.1
> +++ pc-fp.exp	5 Nov 2002 22:09:51 -0000
> @@ -62,7 +62,7 @@ proc get_valueofx { fmt exp default } {
>      gdb_expect {
>  	-re "\\$\[0-9\]* = (0x\[0-9a-zA-Z\]+).*$gdb_prompt $" {
>  	    set val $expect_out(1,string)
> -	    pass "get value of ${exp} ($val)"
> +	    pass "get value of ${exp}"
>  	}
>  	timeout {
>  	    set size ${default}
> 


[-- Attachment #2: do-analize-tests --]
[-- Type: text/plain, Size: 6388 bytes --]

#!/bin/sh

if awk --version < /dev/null 2>&1 | grep -i gnu > /dev/null
then
    AWK=awk
else
    AWK=gawk
fi

$AWK '
function trace (msg) {
    print msg >> "/dev/stderr"
}


# returns a SUPSEP separated list created by sorting TABLE.
function qsort (table, middle, tmp, left, nr_left, right, nr_right, result) {
    middle = ""
    for (middle in table) { break; }
    nr_left = 0;
    nr_right = 0;
    for (tmp in table) {
	if (tmp < middle) {
	    nr_left++
	    left[tmp] = tmp
	} else if (tmp > middle) {
	    nr_right++
	    right[tmp] = tmp
	}
    }
    #print "qsort " nr_left " " middle " " nr_right > "/dev/stderr"
    result = ""
    if (nr_left > 0) {
	result = qsort(left) SUBSEP
    }
    result = result middle
    if (nr_right > 0) {
	result = result SUBSEP qsort(right)
    }
    return result
}


function print_headings (html_p, file, title, target, i) {
    if (html_p) {
	printf "<tr><td></td><th>"					>> file
    }
    printf title							>> file
    if (html_p) {
	printf "</th>"							>> file
    }
    for (i = 1; i <= nr_targets; i++) {
	target = sorted_targets[i]
	if (html_p) {
	    printf "<th>" i "<br>" target "</th>"			>> file
	} else {
	    printf "\t" i						>> file
	}
    }
    if (html_p) {
	printf "</tr>"							>> file
    }
    printf "\n"								>> file
}

function print_test (file, test, nr, i) {
    printf "<tr>"							>> file
    printf "<td>%d</td>", nr						>> file
    printf "<th align=left>%s</th>", test				>> file
    for (i = 1; i <= nr_targets; i++) {
	target = sorted_targets[i];
	result = results[target, test]
	if (result == "FAIL") {
	    printf "<th>%s</th>", result				>> file
	} else {
	    printf "<td align=center>%s</td>", result			>> file
	}
    }
    printf "</tr>\n"							>> file
}

function print_totals (html_p, file, status, target, i, j) {
    for (i = 1; i <= nr_statuses; i++) {
	status = sorted_statuses[i];
	if (html_p) {
	    printf "<tr><td></td><th align=left>"			>> file
	}
	printf status							>> file
	if (html_p) {
	    printf "</th>"						>> file
	}
	for (j = 1; j <= nr_targets; j++) {
	    target = sorted_targets[j];
	    if (html_p) {
		printf "<th>"						>> file
	    } else {
		printf "\t"
	    }
	    printf totals[target, status]				>> file
	    if (html_p) {
		printf "</th>"						>> file
	    }
	}
	if (html_p) {
	    printf "</tr>"						>> file
	}
	printf "\n"							>> file
    }
}

function print_table_start (file, title, full_title, i, target) {
    full_title = "Failure Table: " title;
    trace("Generating " file " (" full_title ")")
    print "<html>"							>> file
    print "<head>"							>> file
    print "<title>" full_title "</title>"				>> file
    print "</head>"							>> file

    print "<body>"							>> file
    print "<h1><center>" full_title "</center></h1>"			>> file
    print "<center>"							>> file
    print "<table>"							>> file
    for (i = 1; i <= nr_targets; i++) {
	target = sorted_targets[i]
	print "<tr>"							>> file
	print "<td>" i "</td>"						>> file
	print "<td>" target "</td>"					>> file
	print "<td>" target_file[target] "</td>"			>> file
	print "</tr>"							>> file
    }
    print "</table>"							>> file
    print "</center>"							>> file
    print "<table>"							>> file
}

function print_table_end (file) {
    print "</table>"							>> file
    print "</body>"							>> file
    print "</html>"							>> file
}

function print_opening (file, title) {
    printf "" > file
    print_table_start(file, title);
    print_headings(1, file, title);
    print_totals(1, file);
}

function print_closing (file, title, nr) {
    trace(title ": " nr);
    print_totals(1, file);
    print_headings(1, file, title);
    print_table_end(file);
}

BEGIN {
    nr_runs = 0
}
/Running target/ {
    nr_runs++
    sub (/Running target/, "");
    gsub (/ /, "");
    gsub (/\//, " ");
    nr = 1;
    do {
	target = $0 " (" nr++ ")";
    } while (target in targets)
    targets[target] = target
    target_file[target] = FILENAME
    trace("Parsing " nr_runs ": " target " " (target_file[target]));
    next
}

# Collect up sizeof???

# Collect the results
/^PASS:/ || /^FAIL:/ || /^XFAIL:/ || /^XPASS:/ {
    # accumulate statuses
    status = $0
    sub (/:.*$/, "", status)
    statuses[status] = status
    # accumulate tests
    test = $0
    sub (/^[A-Z]*: /, "", test)
    gsub (/ *\([^)]*\)/, "", test)
    tests[test] = test

    # accumulate result counts
    totals[target, status] += 1;
    results[target, test] = status;
    totals[test, status] += 1;
}


END {

    # sort the test names
    trace("Sorting results")
    nr_tests = split (qsort(tests), sorted_tests, SUBSEP);
    trace("Number of tests " nr_tests)
#    for (i = 0; i < nr_tests; i++) {
#	print i " " sorted_tests[i] > "/dev/stderr"
#    }
    nr_targets = split (qsort(targets), sorted_targets, SUBSEP);
    nr_statuses = split (qsort(statuses), sorted_statuses, SUBSEP);

    print_headings(0, "/dev/stderr", "Summary")
    print_totals(0, "/dev/stderr")
    for (i = 1; i <= nr_targets; i++) {
	target = sorted_targets[i]
	printf "%3d  %s %s\n", i, target, target_file[target]  >> "/dev/stderr"
    }

    ####

    # heading
    file = "diff.html"
    print_opening(file, "Differences");

    # body - differences
    nr = 0
    for (i = 1; i <= nr_tests; i++) {
	test = sorted_tests[i];
	different = 0;
	fail = 0;
	result = "<uninitialized>";
	for (target in targets) {
	    if (result == "<uninitialized>") {
		result = results[target, test];
	    } else if (result != results[target, test]) {
		different = 1
	    }
	    if (results[target, test] == "FAIL") {
		fail = 1;
	    }
#	    print "TEST=" test, "different=" different, "fail=" fail, "result=" result > "/dev/stderr"
	}
	if (different) {
	    print_test(file, test, ++nr);
	}
    }

    print_closing(file, "Differences", nr);

    ####

    file = "fail.html"
    print_opening(file, "Failures");

    # body - failures
    nr = 0
    for (i = 1; i <= nr_tests; i++) {
	test = sorted_tests[i]
	if (totals[test, "FAIL"] > 0) {
	    print_test(file, test, ++nr);
	}
    }

    print_closing(file, "Failures", nr);

    ####

    file = "pass.html"
    print_opening(file, "Passes");

    # body - passes
    nr = 0
    for (i = 1; i <= nr_tests; i++) {
	test = sorted_tests[i]
	if (totals[test, "PASS"] > 0) {
	    print_test(file, test, ++nr);
	}
    }

    print_closing(file, "Passes", nr);
}
' "$@"

Re: [rfa/testsuite] Don't display values in output of pc-fp.exp

[David Carlton]

On Tue, 05 Nov 2002 17:54:27 -0500, Andrew Cagney <ac131313@redhat.com> said:

>> The tests in pc-fp.exp display the actual values of $pc and $fp in
>> their PASS messages.  That makes regression testing a bit more
>> annoying for me: the value of $fp changes with every change I make
>> to GDB.  Is it okay to tweak the PASS messages not to display those
>> values?

> As far as I know, anything in trailing paren should be ignored when
> comparing test results.  You might want to tweak your script (I've
> attached mine) to do this.

Wow: your script is complicated.  I just do

  diff -u (first file) (second file) | grep -v schedlock

I could do something more complicated than that, of course; on the
other hand, I'm still not convinced that I should.  It seems to me
that details like the value of the variables in question shouldn't be
in gdb.sum: if I want that level of information, I'll look in gdb.log.
I think there's a virtue in having gdb.sum in a format that's easily
checkable without worrying about parsing issues: the easier it is to
do regression testing, the more often it will happen.

> Also, why is FP/PC changing?  Your GDB changes shouldn't affect the
> behavior of the target program's $fp / $pc.

That's a good point; I hadn't thought of that.  I'm actually not
entirely sure what it is that leads to the value of $fp changing from
test run to test run.  But I will make the empirical observation that
it does change from test run to test run, and I'd be shocked if those
changes reflected introduction of new bugs into GDB.

David Carlton
carlton@math.stanford.edu

Re: [rfa/testsuite] Don't display values in output of pc-fp.exp

[Andrew Cagney]

> 
> That's a good point; I hadn't thought of that.  I'm actually not
> entirely sure what it is that leads to the value of $fp changing from
> test run to test run.  But I will make the empirical observation that
> it does change from test run to test run, and I'd be shocked if those
> changes reflected introduction of new bugs into GDB.

Hmm, from memory, $fp, is affected by your environment. environ[] is 
pushed onto the stack before calling main.

Andrew

Re: [rfa/testsuite] Don't display values in output of pc-fp.exp

[Andrew Cagney]

> On Tue, 05 Nov 2002 17:54:27 -0500, Andrew Cagney <ac131313@redhat.com> said:
> 
> 
>>> The tests in pc-fp.exp display the actual values of $pc and $fp in
>>> their PASS messages.  That makes regression testing a bit more
>>> annoying for me: the value of $fp changes with every change I make
>>> to GDB.  Is it okay to tweak the PASS messages not to display those
>>> values?
> 
> 
>> As far as I know, anything in trailing paren should be ignored when
>> comparing test results.  You might want to tweak your script (I've
>> attached mine) to do this.
> 
> 
> Wow: your script is complicated.  I just do
> 
>   diff -u (first file) (second file) | grep -v schedlock
> 
> I could do something more complicated than that, of course; on the
> other hand, I'm still not convinced that I should.  It seems to me
> that details like the value of the variables in question shouldn't be
> in gdb.sum: if I want that level of information, I'll look in gdb.log.
> I think there's a virtue in having gdb.sum in a format that's easily
> checkable without worrying about parsing issues: the easier it is to
> do regression testing, the more often it will happen.

Ah, you've never had to cross compare 64 different targets to check that 
their results are equivalent :-)

Andrew

Re: [rfa/testsuite] Don't display values in output of pc-fp.exp

[David Carlton]

On Wed, 06 Nov 2002 13:36:22 -0800, Michael Snyder <msnyder@redhat.com> said:

[ Michael accidentally replied just to me rather than to the whole
list, so others haven't seen his response yet. ]

> David Carlton wrote:
>> On Tue, 05 Nov 2002 17:54:27 -0500, Andrew Cagney <ac131313@redhat.com> said:

>>> As far as I know, anything in trailing paren should be ignored when
>>> comparing test results.  You might want to tweak your script (I've
>>> attached mine) to do this.

>> Wow: your script is complicated.  I just do
>> 
>> diff -u (first file) (second file) | grep -v schedlock
>> 
>> I could do something more complicated than that, of course; on the
>> other hand, I'm still not convinced that I should. 

> You should.  You'll bang your head against this again.  It's a
> convention [Andrew: is this documented?] that the testsuites may
> generate any random output so long as its in parentheses -- scripts
> should ignore it.

Fair enough.  It may well be documented somewhere, actually: I didn't
think to look.

>> It seems to me that details like the value of the variables in
>> question shouldn't be in gdb.sum: if I want that level of
>> information, I'll look in gdb.log.

> You're right.  I don't think they belong there either.

Okay.  So it seems like, on the one hand, I should be more accepting
of stuff in parentheses.  But, on the other hand, tests shouldn't go
out of their way to add stuff in parentheses, either: if it's
something that a human reader is likely to find useful or if it's
something where it's interesting if its value changes even if that
doesn't count as a regression, then it's probably a good idea for it
to be there.  But I don't think that applies in this case.

>>> Also, why is FP/PC changing?  Your GDB changes shouldn't affect the
>>> behavior of the target program's $fp / $pc.

>> That's a good point; I hadn't thought of that.  I'm actually not
>> entirely sure what it is that leads to the value of $fp changing from
>> test run to test run. 

> Is this native linux?

Yup.

>> But I will make the empirical observation that it does change from
>> test run to test run, and I'd be shocked if those changes reflected
>> introduction of new bugs into GDB.

> PROBABLY not... but maybe it's actually a virtue that we've noticed them.
> I'm curious what could account for them too.  Things I can think of 
> include:
>  * running gdb under gdb (or some other debugger-like or prof-like tool)
>  * changing the size of the inferior's (or possibly gdb's) environment.
>  * running from the unix shell one time, but from a shell script another time.

My guess is that it's the environment.  I do 'make check' every once
in a while (well, about every 15 minutes these days, but I won't be
tinkering with linespec.c forever...), and I compare the resulting
gdb.sum against the gdb.sum from the last time that the correct output
changed.  And that last copy could be days or even weeks old, so my
environment could have easily changed in the interim.

David Carlton
carlton@math.stanford.edu