Re: Update dg-extract-results.* from gcc

[Pedro Alves <palves@redhat.com>]

On 08/08/2018 03:36 PM, Rainer Orth wrote:

>> On 07/20/2018 12:02 PM, Rainer Orth wrote:
>>> When looking at the gdb.sum file produced by dg-extract-results.sh on
>>> Solaris 11/x86, I noticed some wrong sorting, like this:
>>>
>>> PASS: gdb.ada/addr_arith.exp: print something'address + 0
>>> PASS: gdb.ada/addr_arith.exp: print 0 + something'address
>>> PASS: gdb.ada/addr_arith.exp: print something'address - 0
>>> PASS: gdb.ada/addr_arith.exp: print 0 - something'address
>>>
>>> Looking closer, I noticed that while dg-extract-results.sh had been
>>> copied over from contrib in the gcc repo, the corresponding
>>> dg-extract-results.py file had not.  The latter not only fixes the
>>> sorting problem I'd observed, but is also way faster than the shell
>>> version (like a factor of 50 faster).
>>
>> We used to have the dg-extract-results.py file, but we deleted it
>> because it caused (funnily enough, sorting) problems.  See:
>>
>>   https://sourceware.org/ml/gdb-patches/2015-02/msg00333.html
>>
>> Has that sorting stability issue been meanwhile fixed upstream?
> 
> not that I can see: between the version of dg-extract-results.py removed
> in early 2015 and the one in current gcc trunk, there's only added
> handling for DejaGnu ERRORs and another minor change to do with
> summaries that doesn't seem to change anything wrt. sorting on first
> blush.

OK.

> Howver, I've just run make -j16 check three times in a row on
> amd64-pc-solaris2.11, followed by make -j48 check, and the only
> differences were to to 200+ racy tests, the vast majority of them in
> gdb.threads.  

Thanks for testing.

> Maybe the prior problems have been due to bugs in older
> versions of python?
Might be.

IIRC, the sorting that would change would be the order that the different
individual gdb.sum results would  be merged (the per-gdb.sum order was
stable).  So comparing two runs, you'd get something like, in one run this:

 gdb.base/foo.exp:PASS: test1
 gdb.base/foo.exp:PASS: test2
 gdb.base/foo.exp:PASS: test3
 gdb.base/bar.exp:PASS: testA
 gdb.base/bar.exp:PASS: testB
 gdb.base/bar.exp:PASS: testC

and another run this:

 gdb.base/bar.exp:PASS: testA
 gdb.base/bar.exp:PASS: testB
 gdb.base/bar.exp:PASS: testC
 gdb.base/foo.exp:PASS: test1
 gdb.base/foo.exp:PASS: test2
 gdb.base/foo.exp:PASS: test3

which would result in all those tests spuriously showing up in a
gdb.sum old/new diff.

I'm not sure whether we were seeing that if you compared runs
of the same tree multiple times.  It could be that it only happened
when comparing the results of different trees, which contained
a slightly different set of tests and testcases, like for example
comparing testresults of a patched master against the testresults
of master from a month or week ago, which is something I frequently
do, for example.

*time passes*

Wait wait wait, can you clarify what you meant by wrong sorting in:

 PASS: gdb.ada/addr_arith.exp: print something'address + 0
 PASS: gdb.ada/addr_arith.exp: print 0 + something'address
 PASS: gdb.ada/addr_arith.exp: print something'address - 0
 PASS: gdb.ada/addr_arith.exp: print 0 - something'address

?

Why do you think those results _should_ be sorted?  And in what order?

Typically, the order/sequence in which the tests of a given exp
file is executed is important.  The order in the gdb.sum file must
be the order in which the fail/pass calls are written/issued in the .exp file.
It'd be absolutely incorrect to alphabetically sort the gdb.sum output.
Is that what the .py version does?  That's not what I recall, though.
I guess I may be confused.

Thanks,
Pedro Alves