From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (qmail 66231 invoked by alias); 1 Mar 2016 11:56:43 -0000 Mailing-List: contact gdb-patches-help@sourceware.org; run by ezmlm Precedence: bulk List-Id: List-Subscribe: List-Archive: List-Post: List-Help: , Sender: gdb-patches-owner@sourceware.org Received: (qmail 66190 invoked by uid 89); 1 Mar 2016 11:56:41 -0000 Authentication-Results: sourceware.org; auth=none X-Virus-Found: No X-Spam-SWARE-Status: No, score=-1.9 required=5.0 tests=BAYES_00,RP_MATCHES_RCVD,SPF_HELO_PASS autolearn=ham version=3.3.2 spammy=varies, Hx-languages-length:5453, UD:watchpoint-fork.exp, sk:watchpo X-HELO: mx1.redhat.com Received: from mx1.redhat.com (HELO mx1.redhat.com) (209.132.183.28) by sourceware.org (qpsmtpd/0.93/v0.84-503-g423c35a) with (AES256-GCM-SHA384 encrypted) ESMTPS; Tue, 01 Mar 2016 11:56:40 +0000 Received: from int-mx10.intmail.prod.int.phx2.redhat.com (int-mx10.intmail.prod.int.phx2.redhat.com [10.5.11.23]) by mx1.redhat.com (Postfix) with ESMTPS id 90A5664380 for ; Tue, 1 Mar 2016 11:56:39 +0000 (UTC) Received: from [127.0.0.1] (ovpn01.gateway.prod.ext.phx2.redhat.com [10.5.9.1]) by int-mx10.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u21BucUr006707; Tue, 1 Mar 2016 06:56:38 -0500 Subject: Re: [PATCH] Improve analysis of racy testcases To: Sergio Durigan Junior References: <87r3gcgm91.fsf@redhat.com> <56CF3368.5070101@redhat.com> <87h9gszf14.fsf@redhat.com> Cc: GDB Patches From: Pedro Alves Message-ID: <56D58376.6020303@redhat.com> Date: Tue, 01 Mar 2016 11:56:00 -0000 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101 Thunderbird/38.6.0 MIME-Version: 1.0 In-Reply-To: <87h9gszf14.fsf@redhat.com> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-SW-Source: 2016-03/txt/msg00005.txt.bz2 On 02/28/2016 09:44 PM, Sergio Durigan Junior wrote: > On Thursday, February 25 2016, Pedro Alves wrote: > >> I've now run "make check -j8 RACY_ITER=3" and got this: >> >> $ cat testsuite/racy.sum >> gdb.threads/attach-into-signal.exp: nonthreaded: attempt 1: attach (pass 1), pending signal catch >> gdb.threads/attach-into-signal.exp: nonthreaded: attempt 1: attach (pass 2), pending signal catch >> gdb.threads/attach-into-signal.exp: nonthreaded: attempt 4: attach (pass 1), pending signal catch >> gdb.threads/attach-into-signal.exp: nonthreaded: attempt 6: attach (pass 2), pending signal catch >> gdb.threads/attach-into-signal.exp: threaded: attempt 1: attach (pass 1), pending signal catch >> gdb.threads/attach-into-signal.exp: threaded: attempt 3: attach (pass 1), pending signal catch >> gdb.threads/attach-into-signal.exp: threaded: attempt 3: attach (pass 2), pending signal catch >> gdb.threads/attach-many-short-lived-threads.exp: iter 5: attach >> gdb.threads/attach-many-short-lived-threads.exp: iter 5: attach (EPERM) >> gdb.threads/attach-many-short-lived-threads.exp: iter 9: attach >> gdb.threads/attach-many-short-lived-threads.exp: iter 9: attach (EPERM) >> gdb.threads/fork-plus-threads.exp: detach-on-fork=off: only inferior 1 left >> gdb.threads/process-dies-while-handling-bp.exp: non_stop=off: cond_bp_target=0: inferior 1 exited >> gdb.threads/process-dies-while-handling-bp.exp: non_stop=off: cond_bp_target=0: inferior 1 exited (prompt) (PRMS: gdb/18749) >> gdb.threads/process-dies-while-handling-bp.exp: non_stop=off: cond_bp_target=0: no threads left >> gdb.threads/process-dies-while-handling-bp.exp: non_stop=off: cond_bp_target=1: inferior 1 exited >> gdb.threads/process-dies-while-handling-bp.exp: non_stop=off: cond_bp_target=1: inferior 1 exited (memory error) (PRMS: gdb/18749) >> gdb.threads/process-dies-while-handling-bp.exp: non_stop=off: cond_bp_target=1: no threads left >> gdb.threads/process-dies-while-handling-bp.exp: non_stop=on: cond_bp_target=0: inferior 1 exited >> gdb.threads/process-dies-while-handling-bp.exp: non_stop=on: cond_bp_target=0: inferior 1 exited (prompt) (PRMS: gdb/18749) >> gdb.threads/process-dies-while-handling-bp.exp: non_stop=on: cond_bp_target=0: no threads left >> gdb.threads/process-dies-while-handling-bp.exp: non_stop=on: cond_bp_target=1: inferior 1 exited >> gdb.threads/process-dies-while-handling-bp.exp: non_stop=on: cond_bp_target=1: inferior 1 exited (timeout) (PRMS: gdb/18749) >> gdb.threads/process-dies-while-handling-bp.exp: non_stop=on: cond_bp_target=1: no threads left >> gdb.threads/watchpoint-fork.exp: child: multithreaded: finish >> gdb.threads/watchpoint-fork.exp: child: multithreaded: watchpoint A after the second fork >> gdb.threads/watchpoint-fork.exp: child: multithreaded: watchpoint B after the second fork >> >> The gdb.threads/process-dies-while-handling-bp.exp ones are actually: >> >> -PASS: gdb.threads/process-dies-while-handling-bp.exp: non_stop=on: cond_bp_target=1: inferior 1 exited >> +KFAIL: gdb.threads/process-dies-while-handling-bp.exp: non_stop=on: cond_bp_target=1: inferior 1 exited (prompt) (PRMS: gdb/18749) >> >> Test sum diffing should probably strip away tail ()s, and ignore PASS->KFAIL->PASS. > > I thought about stripping tail ()s away before comparing the names, but > the problem is that maybe we'll miss a test that actually writes > something meaningful inside the parentheses. There isn't a strong > convention advising testcase writers to not do that. What do you think? I think we should have that convention. We already largely implicitly have it, exactly because of "(PRMS: gdb/NNN)", "(timeout)" or "(eof)", etc. IOW, I think this should be interpreted as a regression in the "whatever test" test: -PASS: gdb.base/foo.exp: whatever test +FAIL: gdb.base/foo.exp: whatever test (timeout) If that actually strips something meaningful, I'd just file it under the same bucket as non-unique test names, and fix it by tweaking the test message. >> The only thing I do wish we should do, is use the fruits of this >> to somehow mark racy tests in the testsuite itself, instead of only >> making the buildbot ignore them, so that local development benefits >> as well. > > I totally agree, and also spent some time thinking about this problem, > but I don't see an easy solution for that. Racy testcases vary wildly > between targets, GDB vs. gdbserver, CFLAGS, etc. > We would have to maintain several lists of racy tests, I wouldn't want to maintain separate lists at all. Instead, I'd want to mark testcases themselves with something like setup_kfail. Testcases already call those depending on target/arch, I see no difference. Perhaps a problem with setup_kfail is that that generates a KPASS when the race doesn't trigger. We could add a new setup_racy_kfail that generates a PASS on success and KFAIL on failure, but never a KPASS. Perhaps we could have a racy_test_scope in the spirit of with_test_prefix, which would automatically mark all tests in the scope as racy, but I'm not sure we do need or want that. > and even this wouldn't be 100% > trustworthy because the racy tests also vary depending on the machine > load... Varying the load does not make the _set_ of racy tests vary. It only varies the chances of a racy test in a set of racy tests actual trigger the race. A test in that set is still racy regardless. Thanks, Pedro Alves