A question regarding isolated cpus and software interrupts

A question regarding isolated cpus and software interrupts

[David Aldrich via lttng-dev]

Hi

My question isn't directly related to lttng but I would welcome some advice.

Our Linux application is showing 'blips' whereby there are occasional
increases in processing times. This is for a user process that runs on
an isolated core. Using lttng and TraceCompass I have found that the
'blip' sometimes coincides with a TIMER_SOFTIRQ event on the isolated
core.

I was surprised that software interrupts may run on isolated cores. Is
there a way of preventing this?

Are there any techniques using lttng to gain more information related
to the interrupt (e.g. what is causing it)?

Best regards
David

Re: A question regarding isolated cpus and software interrupts

[Mathieu Desnoyers via lttng-dev]

On 2025-05-29 14:06, David Aldrich via lttng-dev wrote:
> Hi
> 
> My question isn't directly related to lttng but I would welcome some advice.
> 
> Our Linux application is showing 'blips' whereby there are occasional
> increases in processing times. This is for a user process that runs on
> an isolated core. Using lttng and TraceCompass I have found that the
> 'blip' sometimes coincides with a TIMER_SOFTIRQ event on the isolated
> core.
> 
> I was surprised that software interrupts may run on isolated cores. Is
> there a way of preventing this?

+CC Frederic Weisbecker.

How are the cores isolated on your system (exact kernel command line
options or commands issued) ? There are various degree of fine-tuning
to prevent disturbance from the tick (nohz full), scheduling, and
irqs that can be done explicitly. So depending on your specific
configuration, you may be missing isolation options that would
prevent the softirq from running on your isolated cpus.

Thanks,

Mathieu

> 
> Are there any techniques using lttng to gain more information related
> to the interrupt (e.g. what is causing it)?
> 
> Best regards
> David


-- 
Mathieu Desnoyers
EfficiOS Inc.
https://www.efficios.com

Re: A question regarding isolated cpus and software interrupts

[David Aldrich via lttng-dev]

Hi Mathieu

Thanks for your reply.

> How are the cores isolated on your system (exact kernel command line
> options or commands issued) ? There are various degree of fine-tuning
> to prevent disturbance from the tick (nohz full), scheduling, and
> irqs that can be done explicitly. So depending on your specific
> configuration, you may be missing isolation options that would
> prevent the softirq from running on your isolated cpus.

Our kernel parameters are:

ro skew_tick=1 rcu_nocb_poll rcu_nocbs=4-31 nohz=on nohz_full=4-31
kthread_cpus=0-3 irqaffinity=0-3 isolcpus=managed_irq,domain,4-31
intel_pstate=passive nosoftlockup tsc=nowatchdog intel_iommu=on
iommu=pt default_hugepagesz=1G hugepagesz=1G hugepages=4
vfio_pci.enable_sriov=1 vfio_pci.disable_idle_d3=1

(the affected thread runs on core 4).

Best regards
David

Re: A question regarding isolated cpus and software interrupts

[Norbert Lange via lttng-dev]

Hello David,

You cant fully isolate a core in Linux, there are some timekeeping
functions that will always run (Linux needs some basic runtime
accounting).

The things you can do is avoiding IRQ handlers and "lazy" subsystems
that depend on background updates via timer or IPI. You already do
this via kernel parameters.
Maybe you can look at disabling membarrier too.

isolcpu just takes the core out of the scheduler, threads wont be
moved in or out automatically.

if you want to find out what causes the interrupts... use ftrace + kernelshark?

Norbert