From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (qmail 22756 invoked by alias); 6 Apr 2009 19:51:09 -0000 Received: (qmail 22746 invoked by uid 22791); 6 Apr 2009 19:51:08 -0000 X-SWARE-Spam-Status: No, hits=-2.3 required=5.0 tests=AWL,BAYES_00,SPF_HELO_PASS,SPF_PASS X-Spam-Check-By: sourceware.org Received: from mx2.redhat.com (HELO mx2.redhat.com) (66.187.237.31) by sourceware.org (qpsmtpd/0.43rc1) with ESMTP; Mon, 06 Apr 2009 19:50:59 +0000 Received: from int-mx2.corp.redhat.com (int-mx2.corp.redhat.com [172.16.27.26]) by mx2.redhat.com (8.13.8/8.13.8) with ESMTP id n36JoqHE003946; Mon, 6 Apr 2009 15:50:52 -0400 Received: from ns3.rdu.redhat.com (ns3.rdu.redhat.com [10.11.255.199]) by int-mx2.corp.redhat.com (8.13.1/8.13.1) with ESMTP id n36Joqhi004964; Mon, 6 Apr 2009 15:50:53 -0400 Received: from opsy.redhat.com (vpn-12-50.rdu.redhat.com [10.11.12.50]) by ns3.rdu.redhat.com (8.13.8/8.13.8) with ESMTP id n36Jopqu031507; Mon, 6 Apr 2009 15:50:51 -0400 Received: by opsy.redhat.com (Postfix, from userid 500) id 706C537817A; Mon, 6 Apr 2009 13:50:49 -0600 (MDT) To: Thiago Jung Bauermann Cc: Paul Pluzhnikov , gdb-patches@sourceware.org Subject: Re: relying on testsuite results References: <200901121846.51709.pedro@codesourcery.com> <200902062335.17737.pedro@codesourcery.com> <8ac60eac0902082223q2192830cu8b75f6424fca6c68@mail.gmail.com> <200902092216.54762.pedro@codesourcery.com> <8ac60eac0904061159v4deb9d48n1cf791463e587e54@mail.gmail.com> <1239045480.8871.11.camel@localhost.localdomain> From: Tom Tromey Reply-To: tromey@redhat.com Date: Mon, 06 Apr 2009 19:51:00 -0000 In-Reply-To: <1239045480.8871.11.camel@localhost.localdomain> (Thiago Jung Bauermann's message of "Mon\, 06 Apr 2009 16\:18\:00 -0300") Message-ID: User-Agent: Gnus/5.11 (Gnus v5.11) Emacs/22.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Mailing-List: contact gdb-patches-help@sourceware.org; run by ezmlm Precedence: bulk List-Id: List-Subscribe: List-Archive: List-Post: List-Help: , Sender: gdb-patches-owner@sourceware.org X-SW-Source: 2009-04/txt/msg00109.txt.bz2 >>>>> "Thiago" == Thiago Jung Bauermann writes: Thiago> So the most use I get from the testsuite is to run regression Thiago> tests on each patch I submit, and tediously eyeball the diff Thiago> looking to see if any of the PASS<->FAIL flips actuallly mean Thiago> something. Yeah, me too. These non-deterministic tests may add something, in one scenario: if they start failing consistently, and somebody notices that, then they may provide information about a real regression. I'm sure I won't notice if they start failing consistently, though. I already ignore them. I generally do full regression tests for each patch. ("Generally" because I occasionally forget, or something goes wrong and I don't notice.) This is nice, although slow, and unfortunately also open to some kinds of major failure (e.g., a change of compiler negatively affecting baseline results). Tom