From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (qmail 119159 invoked by alias); 14 Jul 2015 16:05:18 -0000 Mailing-List: contact gdb-patches-help@sourceware.org; run by ezmlm Precedence: bulk List-Id: List-Subscribe: List-Archive: List-Post: List-Help: , Sender: gdb-patches-owner@sourceware.org Received: (qmail 119150 invoked by uid 89); 14 Jul 2015 16:05:17 -0000 Authentication-Results: sourceware.org; auth=none X-Virus-Found: No X-Spam-SWARE-Status: No, score=-3.3 required=5.0 tests=AWL,BAYES_00,RCVD_IN_DNSWL_LOW,RP_MATCHES_RCVD,SPF_PASS autolearn=ham version=3.3.2 X-HELO: mail-ie0-f179.google.com Received: from mail-ie0-f179.google.com (HELO mail-ie0-f179.google.com) (209.85.223.179) by sourceware.org (qpsmtpd/0.93/v0.84-503-g423c35a) with (AES128-GCM-SHA256 encrypted) ESMTPS; Tue, 14 Jul 2015 16:05:16 +0000 Received: by ietj16 with SMTP id j16so14651345iet.0 for ; Tue, 14 Jul 2015 09:05:14 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc:content-type; bh=hbFog9SufPFhx4MonBT3+6OEA8fp4vKOynz8uG8Q2kM=; b=AjrlzvWsh/qp134yC116stngljlqqqe8lV7LObVZzn/+sNsYejbOvWPIBxaT9o8+Bz 2f4cx49fC2+JwLv+fXDppprBnnRerw8qlXjxu9rWFiOIX05IjxujAwEQnvr5WdoR6ZfP kgzVovrk2zt8QZA3/9pizIoFbpDHPyR+ya617jNApOdBKztFvh/hRmg/AVUOvUHhGGbI ER9V8mKSvOjWcgONLhlYBvC4zAlvQmZls21a7tVB/CA2rP/d/JKVkdqhb2ufG+3g7Toa uWS3vc9SdH5aMSWuHtk5DzrOJEcIaVeacD1w1dFtMrK1ZCA6BuFBZ8nfzxltLq28HJPP k2rQ== X-Gm-Message-State: ALoCoQlTb5Cpprh1WJ6Mri95Wru5BwMX2/QA/f8GuG8yNWuEWsLx/hkzABvy6Z1hFlc6lQbClNam X-Received: by 10.50.66.167 with SMTP id g7mr4472074igt.22.1436889914366; Tue, 14 Jul 2015 09:05:14 -0700 (PDT) MIME-Version: 1.0 Received: by 10.64.93.102 with HTTP; Tue, 14 Jul 2015 09:04:34 -0700 (PDT) In-Reply-To: <55A1665E.2070404@redhat.com> References: <20150709232141.GA7406@adacore.com> <20150710034255.GB7406@adacore.com> <559FD7C6.2060502@redhat.com> <55A1665E.2070404@redhat.com> From: Doug Evans Date: Tue, 14 Jul 2015 16:05:00 -0000 Message-ID: Subject: Re: racy tests To: Pedro Alves Cc: David Edelsohn , Joel Brobecker , GDB Patches Content-Type: text/plain; charset=UTF-8 X-IsSubscribed: yes X-SW-Source: 2015-07/txt/msg00410.txt.bz2 On Sat, Jul 11, 2015 at 11:54 AM, Pedro Alves wrote: > One thing that I'd like is for this to be part of the testsuite > itself, rather than separate machinery the buildbot uses. That way, > everyone benefits from it, and so that we all maintain/evolve it. > I think this is important, because people are often confused that > they do a test run before patch, apply patch, run test, and see > confusing new FAILs their patch can't explain. No disagreement there. I would build it on top of what's there now. [I'd rather build this up in layers, and not have overly complicated lower layers.] A next question that arises is maintaining history. E.g., how does one diff the results of the current run with the current "gold standard"? The way I do it here is to have separate files that augment the XFAIL/KFAIL markers in the test (it's far easier to maintain a few files than editing each test's .exp file) but I'm not sure it scales well. [E.g., I need to keep separate files for different compilers, though there is a #include mechanism for common stuff.] Alternatively, If a test run could take as input the gdb.sum file from a baseline run (e.g., from an unpatched trunk) then that could work. Buildbot could use the previous run, and Joe-Developer could either use as input a buildbot run's output file or run the testsuite twice (e.g., with/without the patch-under-test). [I wouldn't use gdb.sum specifically, I'm just using it here for illustration's sake.]