From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (qmail 27832 invoked by alias); 4 Aug 2011 20:51:35 -0000 Received: (qmail 27818 invoked by uid 22791); 4 Aug 2011 20:51:34 -0000 X-SWARE-Spam-Status: No, hits=-7.3 required=5.0 tests=AWL,BAYES_00,RCVD_IN_DNSWL_HI,RP_MATCHES_RCVD,SPF_HELO_PASS X-Spam-Check-By: sourceware.org Received: from mx1.redhat.com (HELO mx1.redhat.com) (209.132.183.28) by sourceware.org (qpsmtpd/0.43rc1) with ESMTP; Thu, 04 Aug 2011 20:51:18 +0000 Received: from int-mx02.intmail.prod.int.phx2.redhat.com (int-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.12]) by mx1.redhat.com (8.14.4/8.14.4) with ESMTP id p74KpFWG027475 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK); Thu, 4 Aug 2011 16:51:15 -0400 Received: from ns3.rdu.redhat.com (ns3.rdu.redhat.com [10.11.255.199]) by int-mx02.intmail.prod.int.phx2.redhat.com (8.13.8/8.13.8) with ESMTP id p74KpCQv026130; Thu, 4 Aug 2011 16:51:12 -0400 Received: from barimba (ovpn01.gateway.prod.ext.phx2.redhat.com [10.5.9.1]) by ns3.rdu.redhat.com (8.13.8/8.13.8) with ESMTP id p74KpArH010141; Thu, 4 Aug 2011 16:51:11 -0400 From: Tom Tromey To: Marc Khouzam Cc: "'gdb\@sourceware.org'" Subject: Re: How to properly check for regressions? References: Date: Thu, 04 Aug 2011 20:51:00 -0000 In-Reply-To: (Marc Khouzam's message of "Thu, 4 Aug 2011 16:43:55 -0400") Message-ID: User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/24.0.50 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain Mailing-List: contact gdb-help@sourceware.org; run by ezmlm Precedence: bulk List-Id: List-Subscribe: List-Archive: List-Post: List-Help: , Sender: gdb-owner@sourceware.org X-SW-Source: 2011-08/txt/msg00018.txt.bz2 >>>>> "Marc" == Marc Khouzam writes: Marc> What is the right way to make sure my patch does not cause regressions? What I do is have a script that extracts the relevant lines from gdb.sum and normalizes them; then I compare the normalized files. Our buildbot automates all this -- but I suspect somewhat poorly, because Jan is constantly finding regressions I didn't see. Probably we should all use his script. I think he's sent it to one of the lists before; or maybe it is on the wiki. Marc> And how do I deal with tests that sometimes pass and sometimes fail. The most common approach is to learn what they are and ignore them. This isn't totally satisfactory; fixing them or removing them would be nicer. Tom