From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from simark.ca by simark.ca with LMTP id MpW3M1wql1+cKQAAWB0awg (envelope-from ) for ; Mon, 26 Oct 2020 15:58:20 -0400 Received: by simark.ca (Postfix, from userid 112) id C61931F08D; Mon, 26 Oct 2020 15:58:20 -0400 (EDT) X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on simark.ca X-Spam-Level: X-Spam-Status: No, score=-1.1 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,MAILING_LIST_MULTI,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.2 Received: from lists.lttng.org (lists.lttng.org [167.114.26.123]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by simark.ca (Postfix) with ESMTPS id BAA921E590 for ; Mon, 26 Oct 2020 15:58:19 -0400 (EDT) Received: from lists-lttng01.efficios.com (localhost [IPv6:::1]) by lists.lttng.org (Postfix) with ESMTP id 4CKly316Hqz17Pq; Mon, 26 Oct 2020 15:58:19 -0400 (EDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=lists.lttng.org; s=default; t=1603742299; bh=njvx7sDT+fAJrCR2gIYtM8ceJvU4GBT3dxZmOocjt7U=; h=Date:To:Cc:In-Reply-To:References:Subject:List-Id: List-Unsubscribe:List-Archive:List-Post:List-Help:List-Subscribe: From:Reply-To:From; b=YFzJjcu79gqFg430x/rdIJ7T+X0UX8mTdnWxd9Hx/DRCSxX3csGKwTD92WDBdJoxN MLC/CKcqrVzSfbQxPNetIQAXYEGvZ9jn77jE5stdu90eXaBf9DauEZFmKDWNbgi6v/ UE2r7+g6x/CMa4FNjs0iqvTz7F67p5nYBujA74bI8BZGyd1dh3ke+LyunafcN2d9d/ DmxVXSNWHKC18m9XRGNljRauAlAw1s2v7MsX9ZRX6Nl6lPJIeuQSgN1nQsPl3KkJwS bWH3UKB33+N5TNBciP6qPeI3yprQXLE9pewPdp5EHWyIDn251JOavkkfkhwFYBdN9c EcFXtto2+2RxA== Received: from mail.efficios.com (mail.efficios.com [167.114.26.124]) by lists.lttng.org (Postfix) with ESMTPS id 4CKly22xylz17Pp for ; Mon, 26 Oct 2020 15:58:18 -0400 (EDT) Received: from localhost (localhost [127.0.0.1]) by mail.efficios.com (Postfix) with ESMTP id 1DBF4243705 for ; Mon, 26 Oct 2020 15:58:12 -0400 (EDT) Received: from mail.efficios.com ([127.0.0.1]) by localhost (mail03.efficios.com [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id ZL3Q0D64S4Pw; Mon, 26 Oct 2020 15:58:11 -0400 (EDT) Received: from localhost (localhost [127.0.0.1]) by mail.efficios.com (Postfix) with ESMTP id BB44024322C; Mon, 26 Oct 2020 15:58:11 -0400 (EDT) DKIM-Filter: OpenDKIM Filter v2.10.3 mail.efficios.com BB44024322C X-Virus-Scanned: amavisd-new at efficios.com Received: from mail.efficios.com ([127.0.0.1]) by localhost (mail03.efficios.com [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id LiksraaO4Ybn; Mon, 26 Oct 2020 15:58:11 -0400 (EDT) Received: from mail03.efficios.com (mail03.efficios.com [167.114.26.124]) by mail.efficios.com (Postfix) with ESMTP id ADC0F243624; Mon, 26 Oct 2020 15:58:11 -0400 (EDT) Date: Mon, 26 Oct 2020 15:58:11 -0400 (EDT) To: paulmck Cc: Stephen Hemminger , Alan Stern , Lai Jiangshan , lttng-dev , linux-kernel Message-ID: <1576751762.38206.1603742291604.JavaMail.zimbra@efficios.com> In-Reply-To: <20201022223021.GA8535@paulmck-ThinkPad-P72> References: <20201022223021.GA8535@paulmck-ThinkPad-P72> MIME-Version: 1.0 X-Originating-IP: [167.114.26.124] X-Mailer: Zimbra 8.8.15_GA_3968 (ZimbraWebClient - FF81 (Linux)/8.8.15_GA_3968) Thread-Topic: call_rcu: Fix race between rcu_barrier() and call_rcu_data_free() Thread-Index: rMNXOOdfGCDrW92dFjPnT/ePYC8HIQ== Subject: Re: [lttng-dev] [PATCH] call_rcu: Fix race between rcu_barrier() and call_rcu_data_free() X-BeenThere: lttng-dev@lists.lttng.org X-Mailman-Version: 2.1.31 Precedence: list List-Id: LTTng development list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , From: Mathieu Desnoyers via lttng-dev Reply-To: Mathieu Desnoyers Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: lttng-dev-bounces@lists.lttng.org Sender: "lttng-dev" ----- On Oct 22, 2020, at 6:30 PM, paulmck paulmck@kernel.org wrote: > The current code can lose RCU callbacks at shutdown time, which can > result in hangs. This lossage can happen as follows: > > o A thread invokes call_rcu_data_free(), which executes up through > the wake_call_rcu_thread(). At this point, the call_rcu_data > structure has been drained of callbacks, but is still on the > call_rcu_data_list. Note that this thread does not hold the > call_rcu_mutex. > > o Another thread invokes rcu_barrier(), which traverses the > call_rcu_data_list under the protection of call_rcu_mutex, > a list which still includes the above newly drained structure. > This thread therefore adds a callback to the newly drained > call_rcu_data structure. It then releases call_rcu_mutex and > enters a mystifying loop that does futex stuff. > > o The first thread finishes executing call_rcu_data_free(), > which acquires call_rcu_mutex just long enough to remove the > newly drained call_rcu_data structure from call_rcu_data_list. > Which causes one of the rcu_barrier() invocation's callbacks to > be leaked. > > o The second thread's rcu_barrier() invocation never returns > resulting in a hang. > > This commit therefore changes call_rcu_data_free() to acquire > call_rcu_mutex before checking the call_rcu_data structure for callbacks. > In the case where there are no callbacks, call_rcu_mutex is held across > both the check and the removal from call_rcu_data_list, thus preventing > rcu_barrier() from adding a callback in the meantime. In the case where > there are callbacks, call_rcu_mutex must be momentarily dropped across > the call to get_default_call_rcu_data(), which can itself acquire > call_rcu_mutex. This momentary drop is not a problem because any > callbacks that rcu_barrier() might queue during that period of time will > be moved to the default call_rcu_data structure, and the lock will be > held across the full time including moving those callbacks and removing > the call_rcu_data structure that was passed into call_rcu_data_free() > from call_rcu_data_list. > > With this fix, a several-hundred-CPU test successfully completes more > than 5,000 executions. Without this fix, it fails within a few tens > of executions. Although the failures happen more quickly on larger > systems, in theory this could happen on a single-CPU system, courtesy > of preemption. I agree with this fix, will merge in liburcu master, stable-0.12, and stable-2.11. Out of curiosity, which test is hanging ? Is it a test which is part of the liburcu tree or some out-of-tree test ? I wonder why we did not catch it in our CI [1]. Thanks, Mathieu [1] https://ci.lttng.org/view/Liburcu/ > > Signed-off-by: Paul E. McKenney > Cc: Mathieu Desnoyers > Cc: Stephen Hemminger > Cc: Alan Stern > Cc: Lai Jiangshan > Cc: > Cc: > > --- > > urcu-call-rcu-impl.h | 7 +++++-- > 1 file changed, 5 insertions(+), 2 deletions(-) > > diff --git a/src/urcu-call-rcu-impl.h b/src/urcu-call-rcu-impl.h > index b6ec6ba..18fd65a 100644 > --- a/src/urcu-call-rcu-impl.h > +++ b/src/urcu-call-rcu-impl.h > @@ -772,9 +772,13 @@ void call_rcu_data_free(struct call_rcu_data *crdp) > while ((uatomic_read(&crdp->flags) & URCU_CALL_RCU_STOPPED) == 0) > (void) poll(NULL, 0, 1); > } > + call_rcu_lock(&call_rcu_mutex); > if (!cds_wfcq_empty(&crdp->cbs_head, &crdp->cbs_tail)) { > - /* Create default call rcu data if need be */ > + call_rcu_unlock(&call_rcu_mutex); > + /* Create default call rcu data if need be. */ > + /* CBs queued here will be handed to the default list. */ > (void) get_default_call_rcu_data(); > + call_rcu_lock(&call_rcu_mutex); > __cds_wfcq_splice_blocking(&default_call_rcu_data->cbs_head, > &default_call_rcu_data->cbs_tail, > &crdp->cbs_head, &crdp->cbs_tail); > @@ -783,7 +787,6 @@ void call_rcu_data_free(struct call_rcu_data *crdp) > wake_call_rcu_thread(default_call_rcu_data); > } > > - call_rcu_lock(&call_rcu_mutex); > cds_list_del(&crdp->list); > call_rcu_unlock(&call_rcu_mutex); -- Mathieu Desnoyers EfficiOS Inc. http://www.efficios.com _______________________________________________ lttng-dev mailing list lttng-dev@lists.lttng.org https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev