CPU affinity behavior of liburcu call-rcu per-cpu worker threads

CPU affinity behavior of liburcu call-rcu per-cpu worker threads

[Mathieu Desnoyers via lttng-dev]

Hi Paul,

I have a question related to liburcu per-cpu call-rcu worker threads.

So far in the current liburcu releases no affinity is set when the
worker threads are started. We have to wait for 
SET_AFFINITY_CHECK_PERIOD_MASK grace periods before the affinity is set
(it takes about 2.5s in my benchmarks).

This means that short-lived test programs will get poor CPU affinity
for call-rcu worker threads at the beginning of their lifetime. So for
short lived programs, this means poor performance.

OTOH, if we have short-lived applications on a large machine, setting
the affinity immediately when the worker thread is starting means
we use CPU time on CPUs which may never be actually used by the
application (no call-rcu activity), which can be detrimental to other
use-cases as well.

So I was wondering: is the choice of skipping setting the affinity
on call-rcu worker thread startup done on purpose ? And if so,
should we perhaps consider setting the affinity as soon as the
worker thread is woken up for the first time rather than after
SET_AFFINITY_CHECK_PERIOD_MASK grace periods ?

Note that there is a commit in liburcu changing this behavior right
now, but it's not part of any release yet. I want to figure out the
wanted behavior before releasing this change:

commit 28f282af1905a1cf50ff9b5835b4cd9de4416ddf
Author: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Date:   Wed Jul 1 12:49:43 2026 -0400

     call_rcu: pin per-CPU worker at thread startup

Thanks,

Mathieu

-- 
Mathieu Desnoyers
EfficiOS Inc.
https://www.efficios.com

Re: CPU affinity behavior of liburcu call-rcu per-cpu worker threads

[Mathieu Desnoyers via lttng-dev]

On 2026-07-09 14:48, Mathieu Desnoyers wrote:
> Hi Paul,
> 
> I have a question related to liburcu per-cpu call-rcu worker threads.
> 
> So far in the current liburcu releases no affinity is set when the
> worker threads are started. We have to wait for 
> SET_AFFINITY_CHECK_PERIOD_MASK grace periods before the affinity is set
> (it takes about 2.5s in my benchmarks).
> 
> This means that short-lived test programs will get poor CPU affinity
> for call-rcu worker threads at the beginning of their lifetime. So for
> short lived programs, this means poor performance.
> 
> OTOH, if we have short-lived applications on a large machine, setting
> the affinity immediately when the worker thread is starting means
> we use CPU time on CPUs which may never be actually used by the
> application (no call-rcu activity), which can be detrimental to other
> use-cases as well.
> 
> So I was wondering: is the choice of skipping setting the affinity
> on call-rcu worker thread startup done on purpose ? And if so,
> should we perhaps consider setting the affinity as soon as the
> worker thread is woken up for the first time rather than after
> SET_AFFINITY_CHECK_PERIOD_MASK grace periods ?

I've implemented this compromise approach here:

https://review.lttng.org/c/userspace-rcu/+/18257 call-rcu worker: set CPU affinity on first non-empty dequeue

Feedback is welcome,

Thanks!

Mathieu

> 
> Note that there is a commit in liburcu changing this behavior right
> now, but it's not part of any release yet. I want to figure out the
> wanted behavior before releasing this change:
> 
> commit 28f282af1905a1cf50ff9b5835b4cd9de4416ddf
> Author: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
> Date:   Wed Jul 1 12:49:43 2026 -0400
> 
>      call_rcu: pin per-CPU worker at thread startup
> 
> Thanks,
> 
> Mathieu
> 


-- 
Mathieu Desnoyers
EfficiOS Inc.
https://www.efficios.com

Re: CPU affinity behavior of liburcu call-rcu per-cpu worker threads

[Paul E. McKenney via lttng-dev]

On Thu, Jul 09, 2026 at 04:00:17PM -0400, Mathieu Desnoyers wrote:
> On 2026-07-09 14:48, Mathieu Desnoyers wrote:
> > Hi Paul,
> > 
> > I have a question related to liburcu per-cpu call-rcu worker threads.
> > 
> > So far in the current liburcu releases no affinity is set when the
> > worker threads are started. We have to wait for
> > SET_AFFINITY_CHECK_PERIOD_MASK grace periods before the affinity is set
> > (it takes about 2.5s in my benchmarks).
> > 
> > This means that short-lived test programs will get poor CPU affinity
> > for call-rcu worker threads at the beginning of their lifetime. So for
> > short lived programs, this means poor performance.
> > 
> > OTOH, if we have short-lived applications on a large machine, setting
> > the affinity immediately when the worker thread is starting means
> > we use CPU time on CPUs which may never be actually used by the
> > application (no call-rcu activity), which can be detrimental to other
> > use-cases as well.
> > 
> > So I was wondering: is the choice of skipping setting the affinity
> > on call-rcu worker thread startup done on purpose ? And if so,
> > should we perhaps consider setting the affinity as soon as the
> > worker thread is woken up for the first time rather than after
> > SET_AFFINITY_CHECK_PERIOD_MASK grace periods ?

My approach when I first implemented call_rcu() was that the typical
application would have lots of reads and very few updates, so the
default was a single call_rcu() thread shared by all worker threads.
I allowed for manual configuration and affinity of call_rcu() threads
for special-case high-update-rate applications.

My hope was that we would be able to use per-CPU call_rcu() threads
with something like rseq allowing the worker threads to move around
arbitrarily but still getting good cache affinity in their interactions
with the call_rcu() threads.  Easy to say, I know!  ;-)

> I've implemented this compromise approach here:
> 
> https://review.lttng.org/c/userspace-rcu/+/18257 call-rcu worker: set CPU affinity on first non-empty dequeue

The idea being that the call_rcu() thread checks its partner and uses
explicit affinity to follow it around?

I am reminded of a DYNIX/ptx feature that allowed you to say that a
pair of userspace threads were related, so that they should be migrated
together.  This idea did not go over well during the initial Linux-kernel
scheduler discussions a quarter century back.  ;-)

I know!  I know!  Use sched_ext!!!  (Sorry, couldn't resist...)

							Thanx, Paul

> Feedback is welcome,
> 
> Thanks!
> 
> Mathieu
> 
> > 
> > Note that there is a commit in liburcu changing this behavior right
> > now, but it's not part of any release yet. I want to figure out the
> > wanted behavior before releasing this change:
> > 
> > commit 28f282af1905a1cf50ff9b5835b4cd9de4416ddf
> > Author: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
> > Date:   Wed Jul 1 12:49:43 2026 -0400
> > 
> >      call_rcu: pin per-CPU worker at thread startup
> > 
> > Thanks,
> > 
> > Mathieu
> > 
> 
> 
> -- 
> Mathieu Desnoyers
> EfficiOS Inc.
> https://www.efficios.com

Re: CPU affinity behavior of liburcu call-rcu per-cpu worker threads

[Mathieu Desnoyers via lttng-dev]

On 2026-07-09 17:45, Paul E. McKenney wrote:
> On Thu, Jul 09, 2026 at 04:00:17PM -0400, Mathieu Desnoyers wrote:
>> On 2026-07-09 14:48, Mathieu Desnoyers wrote:
>>> Hi Paul,
>>>
>>> I have a question related to liburcu per-cpu call-rcu worker threads.
>>>
>>> So far in the current liburcu releases no affinity is set when the
>>> worker threads are started. We have to wait for
>>> SET_AFFINITY_CHECK_PERIOD_MASK grace periods before the affinity is set
>>> (it takes about 2.5s in my benchmarks).
>>>
>>> This means that short-lived test programs will get poor CPU affinity
>>> for call-rcu worker threads at the beginning of their lifetime. So for
>>> short lived programs, this means poor performance.
>>>
>>> OTOH, if we have short-lived applications on a large machine, setting
>>> the affinity immediately when the worker thread is starting means
>>> we use CPU time on CPUs which may never be actually used by the
>>> application (no call-rcu activity), which can be detrimental to other
>>> use-cases as well.
>>>
>>> So I was wondering: is the choice of skipping setting the affinity
>>> on call-rcu worker thread startup done on purpose ? And if so,
>>> should we perhaps consider setting the affinity as soon as the
>>> worker thread is woken up for the first time rather than after
>>> SET_AFFINITY_CHECK_PERIOD_MASK grace periods ?
> 
> My approach when I first implemented call_rcu() was that the typical
> application would have lots of reads and very few updates, so the
> default was a single call_rcu() thread shared by all worker threads.
> I allowed for manual configuration and affinity of call_rcu() threads
> for special-case high-update-rate applications.

I'm currently working on data structures supporting scalable updates,
which highlight a broader need for per-cpu call-rcu worker threads
(at least if update scalability is wanted).

> My hope was that we would be able to use per-CPU call_rcu() threads
> with something like rseq allowing the worker threads to move around
> arbitrarily but still getting good cache affinity in their interactions
> with the call_rcu() threads.  Easy to say, I know!  ;-)

Having a call-rcu worker thread moving around and sitting on a different
node than the call-rcu users on a big NUMA box is somewhat detrimental
to performance unfortunately.

The telltale sign of this behavior (which got fixed by pinning call-rcu
workers early) was that benchmark results were mostly binomial (high
variance, some slower, some faster). Those benchmarks happened to run
many 2s instances of processes, all falling within the missing affinity
window.

> 
>> I've implemented this compromise approach here:
>>
>> https://review.lttng.org/c/userspace-rcu/+/18257 call-rcu worker: set CPU affinity on first non-empty dequeue
> 
> The idea being that the call_rcu() thread checks its partner and uses
> explicit affinity to follow it around?

Actually, the per-cpu call-rcu worker thread already has all the cpu
affinity information it needs. It's handed over at creation. It's just
that it applies it rather late (after ~2.5s), and only if it happens
to be running on the "wrong" CPU at that particular moment.

Those two factors are not really great for benchmark reproducibility.

> 
> I am reminded of a DYNIX/ptx feature that allowed you to say that a
> pair of userspace threads were related, so that they should be migrated
> together.  This idea did not go over well during the initial Linux-kernel
> scheduler discussions a quarter century back.  ;-)

I don't really need to have the call-rcu worker related to specific
tasks, but I do care that it's local to (at least) a core. I have a
high-churn workload which quickly recycles per-cpu slab memory, and
in order to get good performance with it I need:

- Per-cpu slab cache,
- Per-cpu (or at least per core) call-rcu worker threads,

Which keeps all the alloc -> call-rcu -> worker -> free churn local
to a CPU.

The task pairing you hint at would be great for the per-thread
call-rcu workers we support in liburcu, but as you say it's easier
said than done.

Another approach which would be interesting to look into is to
somehow attach the call-rcu workers to rseq mm_cid concurrency
ids.

Thanks,

Mathieu

-- 
Mathieu Desnoyers
EfficiOS Inc.
https://www.efficios.com

Re: CPU affinity behavior of liburcu call-rcu per-cpu worker threads

[Paul E. McKenney via lttng-dev]

On Thu, Jul 09, 2026 at 06:01:54PM -0400, Mathieu Desnoyers wrote:
> On 2026-07-09 17:45, Paul E. McKenney wrote:
> > On Thu, Jul 09, 2026 at 04:00:17PM -0400, Mathieu Desnoyers wrote:
> > > On 2026-07-09 14:48, Mathieu Desnoyers wrote:
> > > > Hi Paul,
> > > > 
> > > > I have a question related to liburcu per-cpu call-rcu worker threads.
> > > > 
> > > > So far in the current liburcu releases no affinity is set when the
> > > > worker threads are started. We have to wait for
> > > > SET_AFFINITY_CHECK_PERIOD_MASK grace periods before the affinity is set
> > > > (it takes about 2.5s in my benchmarks).
> > > > 
> > > > This means that short-lived test programs will get poor CPU affinity
> > > > for call-rcu worker threads at the beginning of their lifetime. So for
> > > > short lived programs, this means poor performance.
> > > > 
> > > > OTOH, if we have short-lived applications on a large machine, setting
> > > > the affinity immediately when the worker thread is starting means
> > > > we use CPU time on CPUs which may never be actually used by the
> > > > application (no call-rcu activity), which can be detrimental to other
> > > > use-cases as well.
> > > > 
> > > > So I was wondering: is the choice of skipping setting the affinity
> > > > on call-rcu worker thread startup done on purpose ? And if so,
> > > > should we perhaps consider setting the affinity as soon as the
> > > > worker thread is woken up for the first time rather than after
> > > > SET_AFFINITY_CHECK_PERIOD_MASK grace periods ?
> > 
> > My approach when I first implemented call_rcu() was that the typical
> > application would have lots of reads and very few updates, so the
> > default was a single call_rcu() thread shared by all worker threads.
> > I allowed for manual configuration and affinity of call_rcu() threads
> > for special-case high-update-rate applications.
> 
> I'm currently working on data structures supporting scalable updates,
> which highlight a broader need for per-cpu call-rcu worker threads
> (at least if update scalability is wanted).
> 
> > My hope was that we would be able to use per-CPU call_rcu() threads
> > with something like rseq allowing the worker threads to move around
> > arbitrarily but still getting good cache affinity in their interactions
> > with the call_rcu() threads.  Easy to say, I know!  ;-)
> 
> Having a call-rcu worker thread moving around and sitting on a different
> node than the call-rcu users on a big NUMA box is somewhat detrimental
> to performance unfortunately.

No, I mean that the call_rcu() worker threads each stay put on their
own CPU.  The other threads might (or might not, user's choice) move
around, but in the common case, they would queue to the worker thread
on the CPU that they were currently running on.

> The telltale sign of this behavior (which got fixed by pinning call-rcu
> workers early) was that benchmark results were mostly binomial (high
> variance, some slower, some faster). Those benchmarks happened to run
> many 2s instances of processes, all falling within the missing affinity
> window.

Understood, been there, and just pinned everything with each thread
having its own call_rcu() worker thread.  And ran longer benchmarks,
so that any initial issues would be lost in the noise.

> > > I've implemented this compromise approach here:
> > > 
> > > https://review.lttng.org/c/userspace-rcu/+/18257 call-rcu worker: set CPU affinity on first non-empty dequeue
> > 
> > The idea being that the call_rcu() thread checks its partner and uses
> > explicit affinity to follow it around?
> 
> Actually, the per-cpu call-rcu worker thread already has all the cpu
> affinity information it needs. It's handed over at creation. It's just
> that it applies it rather late (after ~2.5s), and only if it happens
> to be running on the "wrong" CPU at that particular moment.
> 
> Those two factors are not really great for benchmark reproducibility.

Indeed, 2.5 seconds is a long time, even for longish benchmarks.

> > I am reminded of a DYNIX/ptx feature that allowed you to say that a
> > pair of userspace threads were related, so that they should be migrated
> > together.  This idea did not go over well during the initial Linux-kernel
> > scheduler discussions a quarter century back.  ;-)
> 
> I don't really need to have the call-rcu worker related to specific
> tasks, but I do care that it's local to (at least) a core. I have a
> high-churn workload which quickly recycles per-cpu slab memory, and
> in order to get good performance with it I need:
> 
> - Per-cpu slab cache,
> - Per-cpu (or at least per core) call-rcu worker threads,
> 
> Which keeps all the alloc -> call-rcu -> worker -> free churn local
> to a CPU.

So tcmalloc()?  And extensions to rseq to allow something else to use
it concurrently.  Or modifications to tcmalloc() to play nicely.  :-/

> The task pairing you hint at would be great for the per-thread
> call-rcu workers we support in liburcu, but as you say it's easier
> said than done.

It worked great in DYNIX/ptx!  Which was admittedly way simpler than
Linux currently is.  ;-)

> Another approach which would be interesting to look into is to
> somehow attach the call-rcu workers to rseq mm_cid concurrency
> ids.

So that a given invocation of call_rcu() queues to the current CPU's
call_rcu() worker thread?  That sounds like a way to obtain what I was
asking for above.  As you said, easier said than done.

							Thanx, Paul

> Thanks,
> 
> Mathieu
> 
> -- 
> Mathieu Desnoyers
> EfficiOS Inc.
> https://www.efficios.com

Re: CPU affinity behavior of liburcu call-rcu per-cpu worker threads

[Mathieu Desnoyers via lttng-dev]

On 2026-07-09 19:13, Paul E. McKenney wrote:
[...]

>>> I am reminded of a DYNIX/ptx feature that allowed you to say that a
>>> pair of userspace threads were related, so that they should be migrated
>>> together.  This idea did not go over well during the initial Linux-kernel
>>> scheduler discussions a quarter century back.  ;-)
>>
>> I don't really need to have the call-rcu worker related to specific
>> tasks, but I do care that it's local to (at least) a core. I have a
>> high-churn workload which quickly recycles per-cpu slab memory, and
>> in order to get good performance with it I need:
>>
>> - Per-cpu slab cache,
>> - Per-cpu (or at least per core) call-rcu worker threads,
>>
>> Which keeps all the alloc -> call-rcu -> worker -> free churn local
>> to a CPU.
> 
> So tcmalloc()?  And extensions to rseq to allow something else to use
> it concurrently.  Or modifications to tcmalloc() to play nicely.  :-/

I created my own per-cpu "cache" with a wfstack list in a liburcu
feature branch. It works with all allocators :)

And unfortunately, for me, tcmalloc is really not a viable option,
because it needs to own the RSEQ area registration, and because glibc
cannot use it at the same time as tcmalloc, my benchmarks suffer because
glibc has a slower sched_getcpu() implementation.

So jemalloc it is. tcmalloc is not usable for me because they don't
compose with the rest of the world. I warned the tcmalloc developers
many times, but they did not listen. :-(

> 
>> The task pairing you hint at would be great for the per-thread
>> call-rcu workers we support in liburcu, but as you say it's easier
>> said than done.
> 
> It worked great in DYNIX/ptx!  Which was admittedly way simpler than
> Linux currently is.  ;-)
> 
>> Another approach which would be interesting to look into is to
>> somehow attach the call-rcu workers to rseq mm_cid concurrency
>> ids.
> 
> So that a given invocation of call_rcu() queues to the current CPU's
> call_rcu() worker thread?  That sounds like a way to obtain what I was
> asking for above.  As you said, easier said than done.

Yes, the advantage is that you would only need to keep around one call
rcu worker thread per _concurrently used_ cpu, rather than per-possible
CPUs. And you would have one call-rcu list per concurrency ID.

Currently the max number for concurrency ids in a process is limited by
min(nr_threads, hweight(affinity mask)).

And I still have future plans to implement a cgroup cpu.max.concurrency
file, but I did not get there yet.

Thanks,

Mathieu

-- 
Mathieu Desnoyers
EfficiOS Inc.
https://www.efficios.com