CPU affinity behavior of liburcu call-rcu per-cpu worker threads

CPU affinity behavior of liburcu call-rcu per-cpu worker threads

[Mathieu Desnoyers via lttng-dev]

Hi Paul,

I have a question related to liburcu per-cpu call-rcu worker threads.

So far in the current liburcu releases no affinity is set when the
worker threads are started. We have to wait for 
SET_AFFINITY_CHECK_PERIOD_MASK grace periods before the affinity is set
(it takes about 2.5s in my benchmarks).

This means that short-lived test programs will get poor CPU affinity
for call-rcu worker threads at the beginning of their lifetime. So for
short lived programs, this means poor performance.

OTOH, if we have short-lived applications on a large machine, setting
the affinity immediately when the worker thread is starting means
we use CPU time on CPUs which may never be actually used by the
application (no call-rcu activity), which can be detrimental to other
use-cases as well.

So I was wondering: is the choice of skipping setting the affinity
on call-rcu worker thread startup done on purpose ? And if so,
should we perhaps consider setting the affinity as soon as the
worker thread is woken up for the first time rather than after
SET_AFFINITY_CHECK_PERIOD_MASK grace periods ?

Note that there is a commit in liburcu changing this behavior right
now, but it's not part of any release yet. I want to figure out the
wanted behavior before releasing this change:

commit 28f282af1905a1cf50ff9b5835b4cd9de4416ddf
Author: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Date:   Wed Jul 1 12:49:43 2026 -0400

     call_rcu: pin per-CPU worker at thread startup

Thanks,

Mathieu

-- 
Mathieu Desnoyers
EfficiOS Inc.
https://www.efficios.com

Re: CPU affinity behavior of liburcu call-rcu per-cpu worker threads

[Mathieu Desnoyers via lttng-dev]

On 2026-07-09 14:48, Mathieu Desnoyers wrote:
> Hi Paul,
> 
> I have a question related to liburcu per-cpu call-rcu worker threads.
> 
> So far in the current liburcu releases no affinity is set when the
> worker threads are started. We have to wait for 
> SET_AFFINITY_CHECK_PERIOD_MASK grace periods before the affinity is set
> (it takes about 2.5s in my benchmarks).
> 
> This means that short-lived test programs will get poor CPU affinity
> for call-rcu worker threads at the beginning of their lifetime. So for
> short lived programs, this means poor performance.
> 
> OTOH, if we have short-lived applications on a large machine, setting
> the affinity immediately when the worker thread is starting means
> we use CPU time on CPUs which may never be actually used by the
> application (no call-rcu activity), which can be detrimental to other
> use-cases as well.
> 
> So I was wondering: is the choice of skipping setting the affinity
> on call-rcu worker thread startup done on purpose ? And if so,
> should we perhaps consider setting the affinity as soon as the
> worker thread is woken up for the first time rather than after
> SET_AFFINITY_CHECK_PERIOD_MASK grace periods ?

I've implemented this compromise approach here:

https://review.lttng.org/c/userspace-rcu/+/18257 call-rcu worker: set CPU affinity on first non-empty dequeue

Feedback is welcome,

Thanks!

Mathieu

> 
> Note that there is a commit in liburcu changing this behavior right
> now, but it's not part of any release yet. I want to figure out the
> wanted behavior before releasing this change:
> 
> commit 28f282af1905a1cf50ff9b5835b4cd9de4416ddf
> Author: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
> Date:   Wed Jul 1 12:49:43 2026 -0400
> 
>      call_rcu: pin per-CPU worker at thread startup
> 
> Thanks,
> 
> Mathieu
> 


-- 
Mathieu Desnoyers
EfficiOS Inc.
https://www.efficios.com

Re: CPU affinity behavior of liburcu call-rcu per-cpu worker threads

[Paul E. McKenney via lttng-dev]

On Thu, Jul 09, 2026 at 04:00:17PM -0400, Mathieu Desnoyers wrote:
> On 2026-07-09 14:48, Mathieu Desnoyers wrote:
> > Hi Paul,
> > 
> > I have a question related to liburcu per-cpu call-rcu worker threads.
> > 
> > So far in the current liburcu releases no affinity is set when the
> > worker threads are started. We have to wait for
> > SET_AFFINITY_CHECK_PERIOD_MASK grace periods before the affinity is set
> > (it takes about 2.5s in my benchmarks).
> > 
> > This means that short-lived test programs will get poor CPU affinity
> > for call-rcu worker threads at the beginning of their lifetime. So for
> > short lived programs, this means poor performance.
> > 
> > OTOH, if we have short-lived applications on a large machine, setting
> > the affinity immediately when the worker thread is starting means
> > we use CPU time on CPUs which may never be actually used by the
> > application (no call-rcu activity), which can be detrimental to other
> > use-cases as well.
> > 
> > So I was wondering: is the choice of skipping setting the affinity
> > on call-rcu worker thread startup done on purpose ? And if so,
> > should we perhaps consider setting the affinity as soon as the
> > worker thread is woken up for the first time rather than after
> > SET_AFFINITY_CHECK_PERIOD_MASK grace periods ?

My approach when I first implemented call_rcu() was that the typical
application would have lots of reads and very few updates, so the
default was a single call_rcu() thread shared by all worker threads.
I allowed for manual configuration and affinity of call_rcu() threads
for special-case high-update-rate applications.

My hope was that we would be able to use per-CPU call_rcu() threads
with something like rseq allowing the worker threads to move around
arbitrarily but still getting good cache affinity in their interactions
with the call_rcu() threads.  Easy to say, I know!  ;-)

> I've implemented this compromise approach here:
> 
> https://review.lttng.org/c/userspace-rcu/+/18257 call-rcu worker: set CPU affinity on first non-empty dequeue

The idea being that the call_rcu() thread checks its partner and uses
explicit affinity to follow it around?

I am reminded of a DYNIX/ptx feature that allowed you to say that a
pair of userspace threads were related, so that they should be migrated
together.  This idea did not go over well during the initial Linux-kernel
scheduler discussions a quarter century back.  ;-)

I know!  I know!  Use sched_ext!!!  (Sorry, couldn't resist...)

							Thanx, Paul

> Feedback is welcome,
> 
> Thanks!
> 
> Mathieu
> 
> > 
> > Note that there is a commit in liburcu changing this behavior right
> > now, but it's not part of any release yet. I want to figure out the
> > wanted behavior before releasing this change:
> > 
> > commit 28f282af1905a1cf50ff9b5835b4cd9de4416ddf
> > Author: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
> > Date:   Wed Jul 1 12:49:43 2026 -0400
> > 
> >      call_rcu: pin per-CPU worker at thread startup
> > 
> > Thanks,
> > 
> > Mathieu
> > 
> 
> 
> -- 
> Mathieu Desnoyers
> EfficiOS Inc.
> https://www.efficios.com

Re: CPU affinity behavior of liburcu call-rcu per-cpu worker threads

[Mathieu Desnoyers via lttng-dev]

On 2026-07-09 17:45, Paul E. McKenney wrote:
> On Thu, Jul 09, 2026 at 04:00:17PM -0400, Mathieu Desnoyers wrote:
>> On 2026-07-09 14:48, Mathieu Desnoyers wrote:
>>> Hi Paul,
>>>
>>> I have a question related to liburcu per-cpu call-rcu worker threads.
>>>
>>> So far in the current liburcu releases no affinity is set when the
>>> worker threads are started. We have to wait for
>>> SET_AFFINITY_CHECK_PERIOD_MASK grace periods before the affinity is set
>>> (it takes about 2.5s in my benchmarks).
>>>
>>> This means that short-lived test programs will get poor CPU affinity
>>> for call-rcu worker threads at the beginning of their lifetime. So for
>>> short lived programs, this means poor performance.
>>>
>>> OTOH, if we have short-lived applications on a large machine, setting
>>> the affinity immediately when the worker thread is starting means
>>> we use CPU time on CPUs which may never be actually used by the
>>> application (no call-rcu activity), which can be detrimental to other
>>> use-cases as well.
>>>
>>> So I was wondering: is the choice of skipping setting the affinity
>>> on call-rcu worker thread startup done on purpose ? And if so,
>>> should we perhaps consider setting the affinity as soon as the
>>> worker thread is woken up for the first time rather than after
>>> SET_AFFINITY_CHECK_PERIOD_MASK grace periods ?
> 
> My approach when I first implemented call_rcu() was that the typical
> application would have lots of reads and very few updates, so the
> default was a single call_rcu() thread shared by all worker threads.
> I allowed for manual configuration and affinity of call_rcu() threads
> for special-case high-update-rate applications.

I'm currently working on data structures supporting scalable updates,
which highlight a broader need for per-cpu call-rcu worker threads
(at least if update scalability is wanted).

> My hope was that we would be able to use per-CPU call_rcu() threads
> with something like rseq allowing the worker threads to move around
> arbitrarily but still getting good cache affinity in their interactions
> with the call_rcu() threads.  Easy to say, I know!  ;-)

Having a call-rcu worker thread moving around and sitting on a different
node than the call-rcu users on a big NUMA box is somewhat detrimental
to performance unfortunately.

The telltale sign of this behavior (which got fixed by pinning call-rcu
workers early) was that benchmark results were mostly binomial (high
variance, some slower, some faster). Those benchmarks happened to run
many 2s instances of processes, all falling within the missing affinity
window.

> 
>> I've implemented this compromise approach here:
>>
>> https://review.lttng.org/c/userspace-rcu/+/18257 call-rcu worker: set CPU affinity on first non-empty dequeue
> 
> The idea being that the call_rcu() thread checks its partner and uses
> explicit affinity to follow it around?

Actually, the per-cpu call-rcu worker thread already has all the cpu
affinity information it needs. It's handed over at creation. It's just
that it applies it rather late (after ~2.5s), and only if it happens
to be running on the "wrong" CPU at that particular moment.

Those two factors are not really great for benchmark reproducibility.

> 
> I am reminded of a DYNIX/ptx feature that allowed you to say that a
> pair of userspace threads were related, so that they should be migrated
> together.  This idea did not go over well during the initial Linux-kernel
> scheduler discussions a quarter century back.  ;-)

I don't really need to have the call-rcu worker related to specific
tasks, but I do care that it's local to (at least) a core. I have a
high-churn workload which quickly recycles per-cpu slab memory, and
in order to get good performance with it I need:

- Per-cpu slab cache,
- Per-cpu (or at least per core) call-rcu worker threads,

Which keeps all the alloc -> call-rcu -> worker -> free churn local
to a CPU.

The task pairing you hint at would be great for the per-thread
call-rcu workers we support in liburcu, but as you say it's easier
said than done.

Another approach which would be interesting to look into is to
somehow attach the call-rcu workers to rseq mm_cid concurrency
ids.

Thanks,

Mathieu

-- 
Mathieu Desnoyers
EfficiOS Inc.
https://www.efficios.com

Re: CPU affinity behavior of liburcu call-rcu per-cpu worker threads

[Paul E. McKenney via lttng-dev]

On Thu, Jul 09, 2026 at 06:01:54PM -0400, Mathieu Desnoyers wrote:
> On 2026-07-09 17:45, Paul E. McKenney wrote:
> > On Thu, Jul 09, 2026 at 04:00:17PM -0400, Mathieu Desnoyers wrote:
> > > On 2026-07-09 14:48, Mathieu Desnoyers wrote:
> > > > Hi Paul,
> > > > 
> > > > I have a question related to liburcu per-cpu call-rcu worker threads.
> > > > 
> > > > So far in the current liburcu releases no affinity is set when the
> > > > worker threads are started. We have to wait for
> > > > SET_AFFINITY_CHECK_PERIOD_MASK grace periods before the affinity is set
> > > > (it takes about 2.5s in my benchmarks).
> > > > 
> > > > This means that short-lived test programs will get poor CPU affinity
> > > > for call-rcu worker threads at the beginning of their lifetime. So for
> > > > short lived programs, this means poor performance.
> > > > 
> > > > OTOH, if we have short-lived applications on a large machine, setting
> > > > the affinity immediately when the worker thread is starting means
> > > > we use CPU time on CPUs which may never be actually used by the
> > > > application (no call-rcu activity), which can be detrimental to other
> > > > use-cases as well.
> > > > 
> > > > So I was wondering: is the choice of skipping setting the affinity
> > > > on call-rcu worker thread startup done on purpose ? And if so,
> > > > should we perhaps consider setting the affinity as soon as the
> > > > worker thread is woken up for the first time rather than after
> > > > SET_AFFINITY_CHECK_PERIOD_MASK grace periods ?
> > 
> > My approach when I first implemented call_rcu() was that the typical
> > application would have lots of reads and very few updates, so the
> > default was a single call_rcu() thread shared by all worker threads.
> > I allowed for manual configuration and affinity of call_rcu() threads
> > for special-case high-update-rate applications.
> 
> I'm currently working on data structures supporting scalable updates,
> which highlight a broader need for per-cpu call-rcu worker threads
> (at least if update scalability is wanted).
> 
> > My hope was that we would be able to use per-CPU call_rcu() threads
> > with something like rseq allowing the worker threads to move around
> > arbitrarily but still getting good cache affinity in their interactions
> > with the call_rcu() threads.  Easy to say, I know!  ;-)
> 
> Having a call-rcu worker thread moving around and sitting on a different
> node than the call-rcu users on a big NUMA box is somewhat detrimental
> to performance unfortunately.

No, I mean that the call_rcu() worker threads each stay put on their
own CPU.  The other threads might (or might not, user's choice) move
around, but in the common case, they would queue to the worker thread
on the CPU that they were currently running on.

> The telltale sign of this behavior (which got fixed by pinning call-rcu
> workers early) was that benchmark results were mostly binomial (high
> variance, some slower, some faster). Those benchmarks happened to run
> many 2s instances of processes, all falling within the missing affinity
> window.

Understood, been there, and just pinned everything with each thread
having its own call_rcu() worker thread.  And ran longer benchmarks,
so that any initial issues would be lost in the noise.

> > > I've implemented this compromise approach here:
> > > 
> > > https://review.lttng.org/c/userspace-rcu/+/18257 call-rcu worker: set CPU affinity on first non-empty dequeue
> > 
> > The idea being that the call_rcu() thread checks its partner and uses
> > explicit affinity to follow it around?
> 
> Actually, the per-cpu call-rcu worker thread already has all the cpu
> affinity information it needs. It's handed over at creation. It's just
> that it applies it rather late (after ~2.5s), and only if it happens
> to be running on the "wrong" CPU at that particular moment.
> 
> Those two factors are not really great for benchmark reproducibility.

Indeed, 2.5 seconds is a long time, even for longish benchmarks.

> > I am reminded of a DYNIX/ptx feature that allowed you to say that a
> > pair of userspace threads were related, so that they should be migrated
> > together.  This idea did not go over well during the initial Linux-kernel
> > scheduler discussions a quarter century back.  ;-)
> 
> I don't really need to have the call-rcu worker related to specific
> tasks, but I do care that it's local to (at least) a core. I have a
> high-churn workload which quickly recycles per-cpu slab memory, and
> in order to get good performance with it I need:
> 
> - Per-cpu slab cache,
> - Per-cpu (or at least per core) call-rcu worker threads,
> 
> Which keeps all the alloc -> call-rcu -> worker -> free churn local
> to a CPU.

So tcmalloc()?  And extensions to rseq to allow something else to use
it concurrently.  Or modifications to tcmalloc() to play nicely.  :-/

> The task pairing you hint at would be great for the per-thread
> call-rcu workers we support in liburcu, but as you say it's easier
> said than done.

It worked great in DYNIX/ptx!  Which was admittedly way simpler than
Linux currently is.  ;-)

> Another approach which would be interesting to look into is to
> somehow attach the call-rcu workers to rseq mm_cid concurrency
> ids.

So that a given invocation of call_rcu() queues to the current CPU's
call_rcu() worker thread?  That sounds like a way to obtain what I was
asking for above.  As you said, easier said than done.

							Thanx, Paul

> Thanks,
> 
> Mathieu
> 
> -- 
> Mathieu Desnoyers
> EfficiOS Inc.
> https://www.efficios.com

Re: CPU affinity behavior of liburcu call-rcu per-cpu worker threads

[Mathieu Desnoyers via lttng-dev]

On 2026-07-09 19:13, Paul E. McKenney wrote:
[...]

>>> I am reminded of a DYNIX/ptx feature that allowed you to say that a
>>> pair of userspace threads were related, so that they should be migrated
>>> together.  This idea did not go over well during the initial Linux-kernel
>>> scheduler discussions a quarter century back.  ;-)
>>
>> I don't really need to have the call-rcu worker related to specific
>> tasks, but I do care that it's local to (at least) a core. I have a
>> high-churn workload which quickly recycles per-cpu slab memory, and
>> in order to get good performance with it I need:
>>
>> - Per-cpu slab cache,
>> - Per-cpu (or at least per core) call-rcu worker threads,
>>
>> Which keeps all the alloc -> call-rcu -> worker -> free churn local
>> to a CPU.
> 
> So tcmalloc()?  And extensions to rseq to allow something else to use
> it concurrently.  Or modifications to tcmalloc() to play nicely.  :-/

I created my own per-cpu "cache" with a wfstack list in a liburcu
feature branch. It works with all allocators :)

And unfortunately, for me, tcmalloc is really not a viable option,
because it needs to own the RSEQ area registration, and because glibc
cannot use it at the same time as tcmalloc, my benchmarks suffer because
glibc has a slower sched_getcpu() implementation.

So jemalloc it is. tcmalloc is not usable for me because they don't
compose with the rest of the world. I warned the tcmalloc developers
many times, but they did not listen. :-(

> 
>> The task pairing you hint at would be great for the per-thread
>> call-rcu workers we support in liburcu, but as you say it's easier
>> said than done.
> 
> It worked great in DYNIX/ptx!  Which was admittedly way simpler than
> Linux currently is.  ;-)
> 
>> Another approach which would be interesting to look into is to
>> somehow attach the call-rcu workers to rseq mm_cid concurrency
>> ids.
> 
> So that a given invocation of call_rcu() queues to the current CPU's
> call_rcu() worker thread?  That sounds like a way to obtain what I was
> asking for above.  As you said, easier said than done.

Yes, the advantage is that you would only need to keep around one call
rcu worker thread per _concurrently used_ cpu, rather than per-possible
CPUs. And you would have one call-rcu list per concurrency ID.

Currently the max number for concurrency ids in a process is limited by
min(nr_threads, hweight(affinity mask)).

And I still have future plans to implement a cgroup cpu.max.concurrency
file, but I did not get there yet.

Thanks,

Mathieu

-- 
Mathieu Desnoyers
EfficiOS Inc.
https://www.efficios.com

Re: CPU affinity behavior of liburcu call-rcu per-cpu worker threads

[Paul E. McKenney via lttng-dev]

On Thu, Jul 09, 2026 at 07:31:18PM -0400, Mathieu Desnoyers wrote:
> On 2026-07-09 19:13, Paul E. McKenney wrote:
> [...]
> 
> > > > I am reminded of a DYNIX/ptx feature that allowed you to say that a
> > > > pair of userspace threads were related, so that they should be migrated
> > > > together.  This idea did not go over well during the initial Linux-kernel
> > > > scheduler discussions a quarter century back.  ;-)
> > > 
> > > I don't really need to have the call-rcu worker related to specific
> > > tasks, but I do care that it's local to (at least) a core. I have a
> > > high-churn workload which quickly recycles per-cpu slab memory, and
> > > in order to get good performance with it I need:
> > > 
> > > - Per-cpu slab cache,
> > > - Per-cpu (or at least per core) call-rcu worker threads,
> > > 
> > > Which keeps all the alloc -> call-rcu -> worker -> free churn local
> > > to a CPU.
> > 
> > So tcmalloc()?  And extensions to rseq to allow something else to use
> > it concurrently.  Or modifications to tcmalloc() to play nicely.  :-/
> 
> I created my own per-cpu "cache" with a wfstack list in a liburcu
> feature branch. It works with all allocators :)

Been there, done that!  ;-)

> And unfortunately, for me, tcmalloc is really not a viable option,
> because it needs to own the RSEQ area registration, and because glibc
> cannot use it at the same time as tcmalloc, my benchmarks suffer because
> glibc has a slower sched_getcpu() implementation.
> 
> So jemalloc it is. tcmalloc is not usable for me because they don't
> compose with the rest of the world. I warned the tcmalloc developers
> many times, but they did not listen. :-(

So an alternative rseq for the rest of us?

> > > The task pairing you hint at would be great for the per-thread
> > > call-rcu workers we support in liburcu, but as you say it's easier
> > > said than done.
> > 
> > It worked great in DYNIX/ptx!  Which was admittedly way simpler than
> > Linux currently is.  ;-)
> > 
> > > Another approach which would be interesting to look into is to
> > > somehow attach the call-rcu workers to rseq mm_cid concurrency
> > > ids.
> > 
> > So that a given invocation of call_rcu() queues to the current CPU's
> > call_rcu() worker thread?  That sounds like a way to obtain what I was
> > asking for above.  As you said, easier said than done.
> 
> Yes, the advantage is that you would only need to keep around one call
> rcu worker thread per _concurrently used_ cpu, rather than per-possible
> CPUs. And you would have one call-rcu list per concurrency ID.
> 
> Currently the max number for concurrency ids in a process is limited by
> min(nr_threads, hweight(affinity mask)).
> 
> And I still have future plans to implement a cgroup cpu.max.concurrency
> file, but I did not get there yet.

These do sound like good improvements!

							Thanx, Paul

Re: CPU affinity behavior of liburcu call-rcu per-cpu worker threads

[Mathieu Desnoyers via lttng-dev]

On 2026-07-09 22:53, Paul E. McKenney wrote:
[...]
> 
>> And unfortunately, for me, tcmalloc is really not a viable option,
>> because it needs to own the RSEQ area registration, and because glibc
>> cannot use it at the same time as tcmalloc, my benchmarks suffer because
>> glibc has a slower sched_getcpu() implementation.
>>
>> So jemalloc it is. tcmalloc is not usable for me because they don't
>> compose with the rest of the world. I warned the tcmalloc developers
>> many times, but they did not listen. :-(
> 
> So an alternative rseq for the rest of us?

I'm not quite sure I understand. Since there is only one rseq
registration per thread, this means that tcmalloc require their
users to use a GLIBC tunable to disable rseq registration at the
libc level, leaving rseq solely to tcmalloc.

Thanks,

Mathieu


-- 
Mathieu Desnoyers
EfficiOS Inc.
https://www.efficios.com

Re: CPU affinity behavior of liburcu call-rcu per-cpu worker threads

[Paul E. McKenney via lttng-dev]

On Thu, Jul 09, 2026 at 10:56:51PM -0400, Mathieu Desnoyers wrote:
> On 2026-07-09 22:53, Paul E. McKenney wrote:
> [...]
> > 
> > > And unfortunately, for me, tcmalloc is really not a viable option,
> > > because it needs to own the RSEQ area registration, and because glibc
> > > cannot use it at the same time as tcmalloc, my benchmarks suffer because
> > > glibc has a slower sched_getcpu() implementation.
> > > 
> > > So jemalloc it is. tcmalloc is not usable for me because they don't
> > > compose with the rest of the world. I warned the tcmalloc developers
> > > many times, but they did not listen. :-(
> > 
> > So an alternative rseq for the rest of us?
> 
> I'm not quite sure I understand. Since there is only one rseq
> registration per thread, this means that tcmalloc require their
> users to use a GLIBC tunable to disable rseq registration at the
> libc level, leaving rseq solely to tcmalloc.

Could an independent thing very closely resembling rseq be brought into
being alongside the current rseq, so that tcmalloc gets the existing
one and everyone else plays nice and shares the new one?

							Thanx, Paul

Re: CPU affinity behavior of liburcu call-rcu per-cpu worker threads

[Mathieu Desnoyers via lttng-dev]

On 2026-07-10 00:07, Paul E. McKenney wrote:
> On Thu, Jul 09, 2026 at 10:56:51PM -0400, Mathieu Desnoyers wrote:
>> On 2026-07-09 22:53, Paul E. McKenney wrote:
>> [...]
>>>
>>>> And unfortunately, for me, tcmalloc is really not a viable option,
>>>> because it needs to own the RSEQ area registration, and because glibc
>>>> cannot use it at the same time as tcmalloc, my benchmarks suffer because
>>>> glibc has a slower sched_getcpu() implementation.
>>>>
>>>> So jemalloc it is. tcmalloc is not usable for me because they don't
>>>> compose with the rest of the world. I warned the tcmalloc developers
>>>> many times, but they did not listen. :-(
>>>
>>> So an alternative rseq for the rest of us?
>>
>> I'm not quite sure I understand. Since there is only one rseq
>> registration per thread, this means that tcmalloc require their
>> users to use a GLIBC tunable to disable rseq registration at the
>> libc level, leaving rseq solely to tcmalloc.
> 
> Could an independent thing very closely resembling rseq be brought into
> being alongside the current rseq, so that tcmalloc gets the existing
> one and everyone else plays nice and shares the new one?

That would unfortunately roll the clock backward about 5 years in terms
of glibc integration, which supports rseq out of the box since version
2.35 (Feb 2022).

Moreover, if we add a second rseq area, this means the kernel now need
to have code to support both areas in key performance-critical code
areas (scheduler, interrupt entry/exit).

So no, as far as I am concerned, this is really not an option.

The "simpler" option would be to implement the rseq extension tcmalloc
need, so they can finally stop doing their odd games and become
compatible with the rest of the world. But the last time I offered to
help them on this front I've been told that they would not even have
time to try my patches.

Oh well.

Thanks,

Mathieu

-- 
Mathieu Desnoyers
EfficiOS Inc.
https://www.efficios.com