From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (qmail 21605 invoked by alias); 4 Aug 2011 20:44:19 -0000 Received: (qmail 21555 invoked by uid 22791); 4 Aug 2011 20:44:18 -0000 X-SWARE-Spam-Status: No, hits=-1.5 required=5.0 tests=AWL,BAYES_00 X-Spam-Check-By: sourceware.org Received: from imr4.ericy.com (HELO imr4.ericy.com) (198.24.6.9) by sourceware.org (qpsmtpd/0.43rc1) with ESMTP; Thu, 04 Aug 2011 20:44:03 +0000 Received: from eusaamw0706.eamcs.ericsson.se ([147.117.20.31]) by imr4.ericy.com (8.14.3/8.14.3/Debian-9.1ubuntu1) with ESMTP id p74KhswY026319 for ; Thu, 4 Aug 2011 15:44:02 -0500 Received: from EUSAACMS0703.eamcs.ericsson.se ([169.254.1.253]) by eusaamw0706.eamcs.ericsson.se ([147.117.20.31]) with mapi; Thu, 4 Aug 2011 16:43:55 -0400 From: Marc Khouzam To: "'gdb@sourceware.org'" Date: Thu, 04 Aug 2011 20:44:00 -0000 Subject: How to properly check for regressions? Message-ID: Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-IsSubscribed: yes Mailing-List: contact gdb-help@sourceware.org; run by ezmlm Precedence: bulk List-Id: List-Subscribe: List-Archive: List-Post: List-Help: , Sender: gdb-owner@sourceware.org X-SW-Source: 2011-08/txt/msg00017.txt.bz2 Hi, I just realized that I might have been checking for regressions wrong! The doc says=20 (http://sourceware.org/gdb/current/onlinedocs/gdbint/Testsuite.html#Testsui= te) "When the testsuite is finished, you'll get a summary that looks like this:" So, I've always looked at that summary, at the end of the console printouts, and made sure it was the same before and after my patches. (note that with -j4, that last summary showed thousands of tests) But now I found the testsuite/gdb.sum file which shows a larger set of tests in the summary that what is shown in the console. Doing a diff of that file before and after my patch showed failures I hadn't seen from the console summary. What is the right way to make sure my patch does not cause regressions? And how do I deal with tests that sometimes pass and sometimes fail. The gdb.sum file does not seem to be constant, event without code changes. Thanks Marc