From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (qmail 60503 invoked by alias); 6 Mar 2016 01:29:50 -0000 Mailing-List: contact gdb-patches-help@sourceware.org; run by ezmlm Precedence: bulk List-Id: List-Subscribe: List-Archive: List-Post: List-Help: , Sender: gdb-patches-owner@sourceware.org Received: (qmail 60471 invoked by uid 89); 6 Mar 2016 01:29:47 -0000 Authentication-Results: sourceware.org; auth=none X-Virus-Found: No X-Spam-SWARE-Status: No, score=-1.9 required=5.0 tests=BAYES_00,RP_MATCHES_RCVD,SPF_HELO_PASS autolearn=ham version=3.3.2 spammy=interpreted, race, watchpoint, advising X-HELO: mx1.redhat.com Received: from mx1.redhat.com (HELO mx1.redhat.com) (209.132.183.28) by sourceware.org (qpsmtpd/0.93/v0.84-503-g423c35a) with (AES256-GCM-SHA384 encrypted) ESMTPS; Sun, 06 Mar 2016 01:29:45 +0000 Received: from int-mx10.intmail.prod.int.phx2.redhat.com (int-mx10.intmail.prod.int.phx2.redhat.com [10.5.11.23]) by mx1.redhat.com (Postfix) with ESMTPS id 460C9C09FA87 for ; Sun, 6 Mar 2016 01:29:44 +0000 (UTC) Received: from localhost (unused-10-15-17-51.yyz.redhat.com [10.15.17.51]) by int-mx10.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u261Thtn017426 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Sat, 5 Mar 2016 20:29:44 -0500 From: Sergio Durigan Junior To: Pedro Alves Cc: GDB Patches Subject: Re: [PATCH] Improve analysis of racy testcases References: <87r3gcgm91.fsf@redhat.com> <56CF3368.5070101@redhat.com> <87h9gszf14.fsf@redhat.com> <56D58376.6020303@redhat.com> X-URL: http://blog.sergiodj.net Date: Sun, 06 Mar 2016 01:29:00 -0000 In-Reply-To: <56D58376.6020303@redhat.com> (Pedro Alves's message of "Tue, 1 Mar 2016 11:56:38 +0000") Message-ID: <874mckwfzs.fsf@redhat.com> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/24.5 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain X-IsSubscribed: yes X-SW-Source: 2016-03/txt/msg00099.txt.bz2 On Tuesday, March 01 2016, Pedro Alves wrote: > On 02/28/2016 09:44 PM, Sergio Durigan Junior wrote: >> On Thursday, February 25 2016, Pedro Alves wrote: > >> >>> I've now run "make check -j8 RACY_ITER=3" and got this: >>> >>> $ cat testsuite/racy.sum >>> gdb.threads/attach-into-signal.exp: nonthreaded: attempt 1: attach (pass 1), pending signal catch >>> gdb.threads/attach-into-signal.exp: nonthreaded: attempt 1: attach (pass 2), pending signal catch >>> gdb.threads/attach-into-signal.exp: nonthreaded: attempt 4: attach (pass 1), pending signal catch >>> gdb.threads/attach-into-signal.exp: nonthreaded: attempt 6: attach (pass 2), pending signal catch >>> gdb.threads/attach-into-signal.exp: threaded: attempt 1: attach (pass 1), pending signal catch >>> gdb.threads/attach-into-signal.exp: threaded: attempt 3: attach (pass 1), pending signal catch >>> gdb.threads/attach-into-signal.exp: threaded: attempt 3: attach (pass 2), pending signal catch >>> gdb.threads/attach-many-short-lived-threads.exp: iter 5: attach >>> gdb.threads/attach-many-short-lived-threads.exp: iter 5: attach (EPERM) >>> gdb.threads/attach-many-short-lived-threads.exp: iter 9: attach >>> gdb.threads/attach-many-short-lived-threads.exp: iter 9: attach (EPERM) >>> gdb.threads/fork-plus-threads.exp: detach-on-fork=off: only inferior 1 left >>> gdb.threads/process-dies-while-handling-bp.exp: non_stop=off: cond_bp_target=0: inferior 1 exited >>> gdb.threads/process-dies-while-handling-bp.exp: non_stop=off: cond_bp_target=0: inferior 1 exited (prompt) (PRMS: gdb/18749) >>> gdb.threads/process-dies-while-handling-bp.exp: non_stop=off: cond_bp_target=0: no threads left >>> gdb.threads/process-dies-while-handling-bp.exp: non_stop=off: cond_bp_target=1: inferior 1 exited >>> gdb.threads/process-dies-while-handling-bp.exp: non_stop=off: cond_bp_target=1: inferior 1 exited (memory error) (PRMS: gdb/18749) >>> gdb.threads/process-dies-while-handling-bp.exp: non_stop=off: cond_bp_target=1: no threads left >>> gdb.threads/process-dies-while-handling-bp.exp: non_stop=on: cond_bp_target=0: inferior 1 exited >>> gdb.threads/process-dies-while-handling-bp.exp: non_stop=on: cond_bp_target=0: inferior 1 exited (prompt) (PRMS: gdb/18749) >>> gdb.threads/process-dies-while-handling-bp.exp: non_stop=on: cond_bp_target=0: no threads left >>> gdb.threads/process-dies-while-handling-bp.exp: non_stop=on: cond_bp_target=1: inferior 1 exited >>> gdb.threads/process-dies-while-handling-bp.exp: non_stop=on: cond_bp_target=1: inferior 1 exited (timeout) (PRMS: gdb/18749) >>> gdb.threads/process-dies-while-handling-bp.exp: non_stop=on: cond_bp_target=1: no threads left >>> gdb.threads/watchpoint-fork.exp: child: multithreaded: finish >>> gdb.threads/watchpoint-fork.exp: child: multithreaded: watchpoint A after the second fork >>> gdb.threads/watchpoint-fork.exp: child: multithreaded: watchpoint B after the second fork >>> > >>> The gdb.threads/process-dies-while-handling-bp.exp ones are actually: >>> >>> -PASS: gdb.threads/process-dies-while-handling-bp.exp: non_stop=on: cond_bp_target=1: inferior 1 exited >>> +KFAIL: gdb.threads/process-dies-while-handling-bp.exp: non_stop=on: cond_bp_target=1: inferior 1 exited (prompt) (PRMS: gdb/18749) >>> >>> Test sum diffing should probably strip away tail ()s, and ignore PASS->KFAIL->PASS. >> >> I thought about stripping tail ()s away before comparing the names, but >> the problem is that maybe we'll miss a test that actually writes >> something meaningful inside the parentheses. There isn't a strong >> convention advising testcase writers to not do that. What do you think? > > I think we should have that convention. We already largely implicitly > have it, exactly because of "(PRMS: gdb/NNN)", "(timeout)" or "(eof)", > etc. > > IOW, I think this should be interpreted as a regression in the > "whatever test" test: > > -PASS: gdb.base/foo.exp: whatever test > +FAIL: gdb.base/foo.exp: whatever test (timeout) > > If that actually strips something meaningful, I'd just file it under > the same bucket as non-unique test names, and fix it by tweaking the > test message. Alright, makes sense to me. I went ahead and also updated the wiki with this rule: >>> The only thing I do wish we should do, is use the fruits of this >>> to somehow mark racy tests in the testsuite itself, instead of only >>> making the buildbot ignore them, so that local development benefits >>> as well. >> >> I totally agree, and also spent some time thinking about this problem, >> but I don't see an easy solution for that. Racy testcases vary wildly >> between targets, GDB vs. gdbserver, CFLAGS, etc. >> We would have to maintain several lists of racy tests, > > I wouldn't want to maintain separate lists at all. Instead, I'd want > to mark testcases themselves with something like setup_kfail. > Testcases already call those depending on target/arch, I see no > difference. > > Perhaps a problem with setup_kfail is that that generates a KPASS when > the race doesn't trigger. We could add a new setup_racy_kfail that > generates a PASS on success and KFAIL on failure, but never a KPASS. > Perhaps we could have a racy_test_scope in the spirit of > with_test_prefix, which would automatically mark all tests in the > scope > as racy, but I'm not sure we do need or want that. I liked the idea of a setup_racy_kfail (not sure about the racy_test_scope thing...). I will take a look at implementing that later. -- Sergio GPG key ID: 237A 54B1 0287 28BF 00EF 31F4 D0EB 7628 65FC 5E36 Please send encrypted e-mail if possible http://sergiodj.net/