Understanding SOFT IRQ 1

Understanding SOFT IRQ 1

[David Aldrich via lttng-dev]

Hi

I am just getting started with LTTng and Trace Compass. I can capture
Kernel and UST events.

My purpose is to investigate why a periodic task, running on an
isolated core, has occasional blips i.e. it's period (duration)
extends.

I have inserted tracepoints to mark normal and extended durations. In
Trace Compass the blip tracepoint is preceded by a brown state that is
'SOFT IRQ 1'. This extends down all cores. It lasts 105us and is
likely to be the reason for the blip.

Please could someone explain to me what may be happening here and how
can I find out the reason for this SOFT IRQ?

Best regards
David

Re: Understanding SOFT IRQ 1

[Mathieu Desnoyers via lttng-dev]

On 2025-05-09 12:00, David Aldrich via lttng-dev wrote:
> Hi
> 
> I am just getting started with LTTng and Trace Compass. I can capture
> Kernel and UST events.
> 
> My purpose is to investigate why a periodic task, running on an
> isolated core, has occasional blips i.e. it's period (duration)
> extends.
> 
> I have inserted tracepoints to mark normal and extended durations. In
> Trace Compass the blip tracepoint is preceded by a brown state that is
> 'SOFT IRQ 1'. This extends down all cores. It lasts 105us and is
> likely to be the reason for the blip.
> 
> Please could someone explain to me what may be happening here and how
> can I find out the reason for this SOFT IRQ?

See include/linux/interrupt.h from the Linux kernel:

enum
{
         HI_SOFTIRQ=0,
         TIMER_SOFTIRQ,
         NET_TX_SOFTIRQ,
         NET_RX_SOFTIRQ,
         BLOCK_SOFTIRQ,
         IRQ_POLL_SOFTIRQ,
         TASKLET_SOFTIRQ,
         SCHED_SOFTIRQ,
         HRTIMER_SOFTIRQ,
         RCU_SOFTIRQ,    /* Preferable RCU should always be the last softirq */

         NR_SOFTIRQS
};

So SOFT IRQ 1 would be TIMER_SOFTIRQ.

Note that we have a TODO in lttng-modules:

lttng_dump_softirq_vec()

Implementing this by dumping this softirq table mapping numbers to names
would be a welcome contribution.

Thanks,

Mathieu

> 
> Best regards
> David


-- 
Mathieu Desnoyers
EfficiOS Inc.
https://www.efficios.com

Re: Understanding SOFT IRQ 1

[Mathieu Desnoyers via lttng-dev]

On 2025-05-12 11:18, Mathieu Desnoyers via lttng-dev wrote:
> On 2025-05-09 12:00, David Aldrich via lttng-dev wrote:
>> Hi
>>
>> I am just getting started with LTTng and Trace Compass. I can capture
>> Kernel and UST events.
>>
>> My purpose is to investigate why a periodic task, running on an
>> isolated core, has occasional blips i.e. it's period (duration)
>> extends.
>>
>> I have inserted tracepoints to mark normal and extended durations. In
>> Trace Compass the blip tracepoint is preceded by a brown state that is
>> 'SOFT IRQ 1'. This extends down all cores. It lasts 105us and is
>> likely to be the reason for the blip.
>>
>> Please could someone explain to me what may be happening here and how
>> can I find out the reason for this SOFT IRQ?
> 
> See include/linux/interrupt.h from the Linux kernel:
> 
> enum
> {
>          HI_SOFTIRQ=0,
>          TIMER_SOFTIRQ,
>          NET_TX_SOFTIRQ,
>          NET_RX_SOFTIRQ,
>          BLOCK_SOFTIRQ,
>          IRQ_POLL_SOFTIRQ,
>          TASKLET_SOFTIRQ,
>          SCHED_SOFTIRQ,
>          HRTIMER_SOFTIRQ,
>          RCU_SOFTIRQ,    /* Preferable RCU should always be the last 
> softirq */
> 
>          NR_SOFTIRQS
> };
> 
> So SOFT IRQ 1 would be TIMER_SOFTIRQ.
> 
> Note that we have a TODO in lttng-modules:
> 
> lttng_dump_softirq_vec()
> 
> Implementing this by dumping this softirq table mapping numbers to names
> would be a welcome contribution.

I just implemented it (only compile-tested), see:

https://review.lttng.org/c/lttng-modules/+/14611 Implement softirq number, name statedump

This dumps the list of softirqs into the trace with their associated names.
We'd have to check whether tweaks are needed for older kernels before I can
merge this.

Thanks,

Mathieu

> 
> Thanks,
> 
> Mathieu
> 
>>
>> Best regards
>> David
> 
> 


-- 
Mathieu Desnoyers
EfficiOS Inc.
https://www.efficios.com