From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (qmail 55672 invoked by alias); 11 Jul 2015 13:59:02 -0000 Mailing-List: contact gdb-patches-help@sourceware.org; run by ezmlm Precedence: bulk List-Id: List-Subscribe: List-Archive: List-Post: List-Help: , Sender: gdb-patches-owner@sourceware.org Received: (qmail 55659 invoked by uid 89); 11 Jul 2015 13:59:01 -0000 Authentication-Results: sourceware.org; auth=none X-Virus-Found: No X-Spam-SWARE-Status: No, score=-2.8 required=5.0 tests=AWL,BAYES_00,RCVD_IN_DNSWL_LOW,RP_MATCHES_RCVD,SPF_PASS autolearn=ham version=3.3.2 X-HELO: mail-wi0-f175.google.com Received: from mail-wi0-f175.google.com (HELO mail-wi0-f175.google.com) (209.85.212.175) by sourceware.org (qpsmtpd/0.93/v0.84-503-g423c35a) with (AES128-GCM-SHA256 encrypted) ESMTPS; Sat, 11 Jul 2015 13:58:59 +0000 Received: by wiwl6 with SMTP id l6so67762697wiw.0 for ; Sat, 11 Jul 2015 06:58:56 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc:content-type; bh=wzOEKAVHCkM0u65xQGxwC4cjRmHBX400A1pUfV8y7V4=; b=SS5miQl6qQwcD+02ik91d6jt1z+kpt4rU5k0zrrMtGdcE8YywOZXBBqIBwNur3dfdV CKwu1BYY4VrqOY0OW9S2HnFsos+hwNMcVj16HChZEgtUvblhlXg9UCSIOQ4IVhYA0Xqk iZYwTlL3DfxEMA/cginYY2TAVke5i9yIrlp7xPWRKKSl2mRnl/dMH9i1cqbr7tCqi4tw JIjwBpM0sIIUZoPL+JBEXy3hBY2zntu5ewGLrlqWIXeWYkpUwVQmQpQWOc4reNhQCRs8 IEvz3sjFAhPsXuXocYW2EvGdRiVcIn2iSseEhBCC0+21nhV08O1UCT4/WzcLunBg8TsU zvYQ== X-Gm-Message-State: ALoCoQmCGdQMKjCeur34P9p2++lq1319lmjqI/PTleMeEhUNpzVN8akGv+Pf6/ykd/rJsP3qsUGy X-Received: by 10.194.246.105 with SMTP id xv9mr50956026wjc.135.1436623136346; Sat, 11 Jul 2015 06:58:56 -0700 (PDT) MIME-Version: 1.0 Received: by 10.28.157.79 with HTTP; Sat, 11 Jul 2015 06:58:16 -0700 (PDT) In-Reply-To: <559FD7C6.2060502@redhat.com> References: <20150709232141.GA7406@adacore.com> <20150710034255.GB7406@adacore.com> <559FD7C6.2060502@redhat.com> From: Doug Evans Date: Sat, 11 Jul 2015 13:59:00 -0000 Message-ID: Subject: Re: GDB 7.9.90 available for testing To: Pedro Alves Cc: David Edelsohn , Joel Brobecker , GDB Patches Content-Type: text/plain; charset=UTF-8 X-IsSubscribed: yes X-SW-Source: 2015-07/txt/msg00347.txt.bz2 On Fri, Jul 10, 2015 at 9:33 AM, Pedro Alves wrote: > There's no single "base" run, actually. The baseline is dynamically > adjusted at each build; it's a moving baseline, and it's per > test (single PASS/FAIL, not file). As soon as a test PASSes, it's > added to the baseline. That means that if some test is racy, it'll > sometimes FAIL, and then a few builds later it'll PASS, at which point > the PASS is recorded in the baseline, and then a few builds again > later, the test FAIL again, and so the buildbot email report mentions > the regression against the baseline. In sum, if a test goes > FAIL -> PASS -> FAIL -> PASS on and on over builds, you'll constantly > get reports of regressions against the baseline for that racy test. Time for another plug to change how we manage racy tests? E.g., if a test fails, run it again a few times. I can think of various things to do after that. E.g. if any of the additional runs of the test record a PASS then flag the test as RACY, and remember this state for the next run, and rerun the same test multiple times in the next run. If the next time all N runs pass (or all N runs fail) then switch its state to PASS/FAIL. That's not perfect, it's hard to be perfect with racy tests. One can build on that, but there's a pragmatic tradeoff here between being too complex and not doing anything at all. I think we should do something. The above keeps the baseline machine-generated and does minimal work to manage racy tests. A lot of racy tests get exposed during these additional runs for me because I don't run them in parallel and thus the system is under less load, and it's system load that triggers a lot of the racyness. The ultimate goal is of course to remove racy tests, but first we need to be more systematic in identifying them, which is one of the goals of this process.