From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from simark.ca by simark.ca with LMTP id 5pMXIhcrUGqA/wAAWB0awg (envelope-from ) for ; Thu, 09 Jul 2026 19:13:27 -0400 Authentication-Results: simark.ca; dkim=pass (2048-bit key; unprotected) header.d=lists.lttng.org header.i=@lists.lttng.org header.a=rsa-sha256 header.s=default header.b=GWSjrs4P; dkim-atps=neutral Received: by simark.ca (Postfix, from userid 112) id 740D31E0A3; Thu, 09 Jul 2026 19:13:27 -0400 (EDT) X-Spam-Checker-Version: SpamAssassin 4.0.1 (2024-03-25) on simark.ca X-Spam-Level: X-Spam-Status: No, score=-3.1 required=5.0 tests=ARC_SIGNED,ARC_VALID,BAYES_00, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI autolearn=ham autolearn_force=no version=4.0.1 Received: from lists.lttng.org (lists.lttng.org [158.69.130.29]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519 server-signature ECDSA (prime256v1) server-digest SHA256) (No client certificate requested) by simark.ca (Postfix) with ESMTPS id 56C951E04F for ; Thu, 09 Jul 2026 19:13:26 -0400 (EDT) ARC-Filter: OpenARC Filter v1.2.1 lists.lttng.org 4gx9kF50Vxz2B8Q ARC-Seal: i=1; d=lists.lttng.org; s=arc1; a=rsa-sha256; cv=none; t=1783638806; b=UiG6kiKt9HhW+/d9wmML/nFuivKYCnN3qIORBw1yj0OwtmSWasY513f2+pGBZIw0Og8E h1TB6cz6Wd0SyyCnJnsnDSm+u/i12xpA9UbDe2INUADYIhdJ2Eabk0O71R91iP8uXCfBL xbAWJmZwNdb0ei9++dth3+HM9Rp8y2xesPbxiJdrTRxFVkLUF7ZXd9jXx7op5Wui18I+s HU9pPc4y6u/diMiMm5lpQBuhURk/VxT7hSrxmTRYoK4l5BMV96Aitg1G72YhuQXo0qV7F CqUDQSEGjstPCPqQ50Fx5OxhzZidUS0qy5PXIWAjzYA/5CaD/tWDkZVU7MPlojxhupg== ARC-Message-Signature: i=1; d=lists.lttng.org; s=arc1; a=rsa-sha256; c=relaxed/simple; t=1783638806; h=DKIM-Signature:Date:To:Subject:Message-ID:MIME-Version:From:From; bh=IqZh1zpYlAuWRR795Io5JRyO8lI4JgsNuDPgrkiXb4I=; b=S/Kfkfjnh/khejH7lWjvPdSfNxPHqaqus7goF/Pxib4r5XGkZhrFW9AxYvsVKntucpRg cgg/xj0l33B8Egeczl1I0FrXYY7FuLQtkJsuchhvebCU5Y1063743phrv8LcSulbobU8H Ujmsc6IRj8HVVey0I/Si/gT+tOWEwHyIzm9oA+Y6ZD+a9c6z/Ms325HYH+oEsXs/p4Hnu X06Yehs8bFuttHLXG25HfgAv27SQkn0KFIYvZqu9RjGOJU+XT3n2hKoDK6UCZg+llgB6S fpQj6fR40V36FrRHuFMg1cE8Q/+ufk7tEGC/xAbWY+HQlegWY9nveoMTnjLXOOf0QfA== ARC-Authentication-Results: i=1; lists.lttng.org; arc=none smtp.remote-ip="::1" DKIM-Filter: OpenDKIM Filter v2.11.0 lists.lttng.org 4gx9kF50Vxz2B8Q DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=lists.lttng.org; s=default; t=1783638806; bh=Bml576EhURPuJE44TzbB8PYxHUvo0WzTyw+Px0kuSkk=; h=Date:To:Cc:Subject:References:In-Reply-To:List-Id: List-Unsubscribe:List-Archive:List-Post:List-Help:List-Subscribe: From:Reply-To:From; b=GWSjrs4PuK9nkzhpghggQ5eO8P3gzK6ogvG2mEdf1Y4gML6crm+USSzc0rE083CEK s0AwjpBoh4iMy4ZoRRNWIuvJxHtA+vCxU/DwM4bgQL8ek3QwgIMAM+QX6F/4Q/iquV K5ZmITgYMayh8eQ0TzXVrJPeQ0gXzBPvHK0a/pQFo/uAt6nPTznwIJbaupMjry7w8K v/1iM7gDG8cQn4iLpz8PLVBwiXzy/VdCPC85frloVUZ6+MxJbLkRo8InZ+uktDSvwu +ROGhVqM6xi8RKpAyiKc8dKx4iFaAqAgdY+rmPFAEn1txuU6HOL2rHlJIHkLtO2n3t v3TNPEd7eKvzg== Received: from lists-lttng01.efficios.com (localhost [IPv6:::1]) by lists.lttng.org (Postfix) with ESMTP id 4gx9kF50Vxz2B8Q; Thu, 9 Jul 2026 19:13:25 -0400 (EDT) ARC-Filter: OpenARC Filter v1.2.1 lists.lttng.org 4gx9kC4flpz2B8M DKIM-Filter: OpenDKIM Filter v2.11.0 lists.lttng.org 4gx9kC4flpz2B8M Received: from sea.source.kernel.org (sea.source.kernel.org [IPv6:2600:3c0a:e001:78e:0:1991:8:25]) by lists.lttng.org (Postfix) with ESMTPS id 4gx9kC4flpz2B8M for ; Thu, 9 Jul 2026 19:13:23 -0400 (EDT) Received: from smtp.kernel.org (quasi.space.kernel.org [100.103.45.18]) by sea.source.kernel.org (Postfix) with ESMTP id 9CC4240EFA; Thu, 9 Jul 2026 23:13:21 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 80F731F000E9; Thu, 9 Jul 2026 23:13:21 +0000 (UTC) Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 4FDDCCE1E72; Thu, 9 Jul 2026 16:13:21 -0700 (PDT) Date: Thu, 9 Jul 2026 16:13:21 -0700 To: Mathieu Desnoyers Cc: lttng-dev , Olivier Dion Subject: Re: CPU affinity behavior of liburcu call-rcu per-cpu worker threads Message-ID: <1aaa979b-bad9-47a5-879b-b4fb2379ad20@paulmck-laptop> References: <0600b601-3edf-4897-b319-b4763943809c@efficios.com> <3573944e-3abc-48d7-aedf-3362ebd7ebe4@paulmck-laptop> <38f1f93e-3bfa-425a-96d9-3035c8ecc785@efficios.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <38f1f93e-3bfa-425a-96d9-3035c8ecc785@efficios.com> X-BeenThere: lttng-dev@lists.lttng.org X-Mailman-Version: 2.1.39 Precedence: list List-Id: LTTng development list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , From: "Paul E. McKenney via lttng-dev" Reply-To: paulmck@kernel.org Errors-To: lttng-dev-bounces@lists.lttng.org Sender: "lttng-dev" On Thu, Jul 09, 2026 at 06:01:54PM -0400, Mathieu Desnoyers wrote: > On 2026-07-09 17:45, Paul E. McKenney wrote: > > On Thu, Jul 09, 2026 at 04:00:17PM -0400, Mathieu Desnoyers wrote: > > > On 2026-07-09 14:48, Mathieu Desnoyers wrote: > > > > Hi Paul, > > > > > > > > I have a question related to liburcu per-cpu call-rcu worker threads. > > > > > > > > So far in the current liburcu releases no affinity is set when the > > > > worker threads are started. We have to wait for > > > > SET_AFFINITY_CHECK_PERIOD_MASK grace periods before the affinity is set > > > > (it takes about 2.5s in my benchmarks). > > > > > > > > This means that short-lived test programs will get poor CPU affinity > > > > for call-rcu worker threads at the beginning of their lifetime. So for > > > > short lived programs, this means poor performance. > > > > > > > > OTOH, if we have short-lived applications on a large machine, setting > > > > the affinity immediately when the worker thread is starting means > > > > we use CPU time on CPUs which may never be actually used by the > > > > application (no call-rcu activity), which can be detrimental to other > > > > use-cases as well. > > > > > > > > So I was wondering: is the choice of skipping setting the affinity > > > > on call-rcu worker thread startup done on purpose ? And if so, > > > > should we perhaps consider setting the affinity as soon as the > > > > worker thread is woken up for the first time rather than after > > > > SET_AFFINITY_CHECK_PERIOD_MASK grace periods ? > > > > My approach when I first implemented call_rcu() was that the typical > > application would have lots of reads and very few updates, so the > > default was a single call_rcu() thread shared by all worker threads. > > I allowed for manual configuration and affinity of call_rcu() threads > > for special-case high-update-rate applications. > > I'm currently working on data structures supporting scalable updates, > which highlight a broader need for per-cpu call-rcu worker threads > (at least if update scalability is wanted). > > > My hope was that we would be able to use per-CPU call_rcu() threads > > with something like rseq allowing the worker threads to move around > > arbitrarily but still getting good cache affinity in their interactions > > with the call_rcu() threads. Easy to say, I know! ;-) > > Having a call-rcu worker thread moving around and sitting on a different > node than the call-rcu users on a big NUMA box is somewhat detrimental > to performance unfortunately. No, I mean that the call_rcu() worker threads each stay put on their own CPU. The other threads might (or might not, user's choice) move around, but in the common case, they would queue to the worker thread on the CPU that they were currently running on. > The telltale sign of this behavior (which got fixed by pinning call-rcu > workers early) was that benchmark results were mostly binomial (high > variance, some slower, some faster). Those benchmarks happened to run > many 2s instances of processes, all falling within the missing affinity > window. Understood, been there, and just pinned everything with each thread having its own call_rcu() worker thread. And ran longer benchmarks, so that any initial issues would be lost in the noise. > > > I've implemented this compromise approach here: > > > > > > https://review.lttng.org/c/userspace-rcu/+/18257 call-rcu worker: set CPU affinity on first non-empty dequeue > > > > The idea being that the call_rcu() thread checks its partner and uses > > explicit affinity to follow it around? > > Actually, the per-cpu call-rcu worker thread already has all the cpu > affinity information it needs. It's handed over at creation. It's just > that it applies it rather late (after ~2.5s), and only if it happens > to be running on the "wrong" CPU at that particular moment. > > Those two factors are not really great for benchmark reproducibility. Indeed, 2.5 seconds is a long time, even for longish benchmarks. > > I am reminded of a DYNIX/ptx feature that allowed you to say that a > > pair of userspace threads were related, so that they should be migrated > > together. This idea did not go over well during the initial Linux-kernel > > scheduler discussions a quarter century back. ;-) > > I don't really need to have the call-rcu worker related to specific > tasks, but I do care that it's local to (at least) a core. I have a > high-churn workload which quickly recycles per-cpu slab memory, and > in order to get good performance with it I need: > > - Per-cpu slab cache, > - Per-cpu (or at least per core) call-rcu worker threads, > > Which keeps all the alloc -> call-rcu -> worker -> free churn local > to a CPU. So tcmalloc()? And extensions to rseq to allow something else to use it concurrently. Or modifications to tcmalloc() to play nicely. :-/ > The task pairing you hint at would be great for the per-thread > call-rcu workers we support in liburcu, but as you say it's easier > said than done. It worked great in DYNIX/ptx! Which was admittedly way simpler than Linux currently is. ;-) > Another approach which would be interesting to look into is to > somehow attach the call-rcu workers to rseq mm_cid concurrency > ids. So that a given invocation of call_rcu() queues to the current CPU's call_rcu() worker thread? That sounds like a way to obtain what I was asking for above. As you said, easier said than done. Thanx, Paul > Thanks, > > Mathieu > > -- > Mathieu Desnoyers > EfficiOS Inc. > https://www.efficios.com