[lttng-dev] Can't receive a lot of functions that i registerd as trace-points

[lttng-dev] Can't receive a lot of functions that i registerd as trace-points

[Lin ArcX via lttng-dev]

[-- Attachment #1.1: Type: text/plain, Size: 1550 bytes --]

Hello,

I have a simple udp server/client and I want to trace these functions:

    Event rules:
      netif_rx_ni_entry (loglevel: TRACE_EMERG (0)) (type: tracepoint)
[enabled]
      netif_rx_entry (loglevel: TRACE_EMERG (0)) (type: tracepoint)
[enabled]
      napi_gro_receive_entry (loglevel: TRACE_EMERG (0)) (type: tracepoint)
[enabled]
      softirq_raise (loglevel: TRACE_EMERG (0)) (type: tracepoint) [enabled]
      irq_handler_entry (loglevel: TRACE_EMERG (0)) (type: tracepoint)
[enabled]
      napi_poll (loglevel: TRACE_EMERG (0)) (type: tracepoint) [enabled]
      softirq_entry (loglevel: TRACE_EMERG (0)) (type: tracepoint) [enabled]
      e1000_xmit_frame (loglevel: TRACE_EMERG (0)) (type: tracepoint)
[enabled]
      ndo_start_xmit (loglevel: TRACE_EMERG (0)) (type: tracepoint)
[enabled]
      net_dev_start_xmit (loglevel: TRACE_EMERG (0)) (type: tracepoint)
[enabled]
      dev_queue_xmit (loglevel: TRACE_EMERG (0)) (type: tracepoint)
[enabled]
      net_dev_queue (loglevel: TRACE_EMERG (0)) (type: tracepoint) [enabled]
      netif_receive_skb_list_entry (loglevel: TRACE_EMERG (0)) (type:
tracepoint) [enabled]
      netif_receive_skb (loglevel: TRACE_EMERG (0)) (type: tracepoint)
[enabled]
      inet_sock_set_state (loglevel: TRACE_EMERG (0)) (type: tracepoint)
[enabled]

I start tracing, and when it is finished, I will open my trace directory
into TraceCompass, But I can see just some traces from these functions:

- irq_handler_entry
- napi_poll
- net_dev_queue

I'm using  2.12.14.

Do you have any idea?

Regards,

[-- Attachment #1.2: Type: text/html, Size: 1718 bytes --]

[-- Attachment #2: Type: text/plain, Size: 156 bytes --]

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [lttng-dev] Can't receive a lot of functions that i registerd as trace-points

[Kienan Stewart via lttng-dev]

Hi Lin,

Some of the event rules you have listed are for items that aren't 
instrumented. Others have the wrong name.

I've put some more specific indications below. You can check which 
events are available in the kernel domain with

`lttng list --kernel`

And for syscalls:

`lttng list --kernel --syscall`

There is also documentation available on adding LTTng probes for 
existing ftrace tracepoints at 
https://lttng.org/docs/v2.13/#doc-linux-add-lttng-layer (for 2.13) or 
https://lttng.org/docs/v2.12/#doc-linux-add-lttng-layer


> Hello,
> 
> I have a simple udp server/client and I want to trace these functions:
> 
>      Event rules:
>        netif_rx_ni_entry (loglevel: TRACE_EMERG (0)) (type: tracepoint) 
> [enabled]
>        netif_rx_entry (loglevel: TRACE_EMERG (0)) (type: tracepoint) 

In LTTng, these events start with "net_if_", not "netif_"

> [enabled]
>        napi_gro_receive_entry (loglevel: TRACE_EMERG (0)) (type: 
> tracepoint) [enabled]

Try "net_napi_gro_receive_entry" instead.

>        softirq_raise (loglevel: TRACE_EMERG (0)) (type: tracepoint) 
> [enabled]
>        irq_handler_entry (loglevel: TRACE_EMERG (0)) (type: tracepoint) 
> [enabled]
>        napi_poll (loglevel: TRACE_EMERG (0)) (type: tracepoint) [enabled]
>        softirq_entry (loglevel: TRACE_EMERG (0)) (type: tracepoint) 
> [enabled]
>        e1000_xmit_frame (loglevel: TRACE_EMERG (0)) (type: tracepoint) 
> [enabled]
>        ndo_start_xmit (loglevel: TRACE_EMERG (0)) (type: tracepoint) 
> [enabled]

lttng-modules doesn't provide any specific instrumentation for the e1000 
and ndo drivers. Are you using additional kernel modules that add the 
instrumentation?

If you are, please ensure that the module is loaded, or it can be added 
to the list of probes loaded by the sessiond using `--extra-kmod-probes` 
(c.f. 
https://lttng.org/man/8/lttng-sessiond/v2.13/#doc-opt--extra-kmod-probes)

If not, you will need to implement the probes as mentioned earlier. If 
you need probes within a specific timeline, either patches and/or
financial support are welcome to create and maintain them. If you're
interested in funding this, please feel free to reach out to EfficiOS.

>        net_dev_start_xmit (loglevel: TRACE_EMERG (0)) (type: tracepoint) > [enabled]

This might be "net_dev_xmit" instead of "net_dev_start_xmit"

>        dev_queue_xmit (loglevel: TRACE_EMERG (0)) (type: tracepoint) 
> [enabled]
>        net_dev_queue (loglevel: TRACE_EMERG (0)) (type: tracepoint) 
> [enabled]
>        netif_receive_skb_list_entry (loglevel: TRACE_EMERG (0)) (type: 
> tracepoint) [enabled]
>        netif_receive_skb (loglevel: TRACE_EMERG (0)) (type: tracepoint) 
> [enabled]

These are also prefixed with "net_if_" instead of "netif_".

>        inet_sock_set_state (loglevel: TRACE_EMERG (0)) (type: 
> tracepoint) [enabled]

I don't think this is instrumented in lttng-modules at this time.

> 
> I start tracing, and when it is finished, I will open my trace directory 
> into TraceCompass, But I can see just some traces from these functions:
> 
> - irq_handler_entry
> - napi_poll
> - net_dev_queue
> 
> I'm using  2.12.14.
> 
> Do you have any idea?
> 
> Regards,
> 

Hope this helps.

thanks,
kienan

> _______________________________________________
> lttng-dev mailing list
> lttng-dev@lists.lttng.org
> https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [lttng-dev] Can't receive a lot of functions that i registerd as trace-points

[Kienan Stewart via lttng-dev]

Hi Lin,

could you keep the lttng-dev list in CC on your replies in the future?


On 2023-10-24 09:25, Lin ArcX wrote:
> Thanks for your response. i fixed the names. here is new commands:
> # enable event for traced functions
> sudo lttng enable-event --kernel irq_handler_entry,napi_poll,net_dev_queue
> 
> # different name with kernel
> sudo lttng enable-event --kernel 
> net_if_rx_entry,net_if_rx_ni_entry,net_napi_gro_receive_entry,net_if_receive_skb_list_entry,net_if_receive_skb
> 
> Now In new tracefile i can see a lot of new events come. but still can't 
> see any trace from these:
> 
> net_if_rx_ni_entry
> net_napi_gro_receive_entry
> net_if_receive_skb_list_entry
> 
This looks fine to me.

On a linux 4.19 system, I ran a test with using lttng-modules 2.12.14. I 
saw events for "net_napi_gro_receive_entry".

In parallel with the LTTng tracing session, I ran

   perf stat -e 
net:netif_rx_ni_entry,net:netif_receive_skb_list_entry,net:napi_gro_receive_entry

The above allowed me to confirm that there were 
"net:napi_gro_receive_entry_events", but no "net:netif_rx_ni_entry" or 
"net:netif_receive_skb_list_entry" occurrences in my test.

Are you seeing a situation where another tool is recording occurrences 
of the events that you think aren't being seen during your tracing session?

thanks,
kienan
_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [lttng-dev] Can't receive a lot of functions that i registerd as trace-points

[Lin ArcX via lttng-dev]

[-- Attachment #1.1: Type: text/plain, Size: 1581 bytes --]

On Tue, 24 Oct 2023 at 21:56, Kienan Stewart <kstewart@efficios.com> wrote:

> Hi Lin,
>
> could you keep the lttng-dev list in CC on your replies in the future?
>
Yes, sure.

>
>
> On 2023-10-24 09:25, Lin ArcX wrote:
> > Thanks for your response. i fixed the names. here is new commands:
> > # enable event for traced functions
> > sudo lttng enable-event --kernel
> irq_handler_entry,napi_poll,net_dev_queue
> >
> > # different name with kernel
> > sudo lttng enable-event --kernel
> >
> net_if_rx_entry,net_if_rx_ni_entry,net_napi_gro_receive_entry,net_if_receive_skb_list_entry,net_if_receive_skb
> >
> > Now In new tracefile i can see a lot of new events come. but still can't
> > see any trace from these:
> >
> > net_if_rx_ni_entry
> > net_napi_gro_receive_entry
> > net_if_receive_skb_list_entry
> >
> This looks fine to me.
>
> On a linux 4.19 system, I ran a test with using lttng-modules 2.12.14. I
> saw events for "net_napi_gro_receive_entry".
>
> In parallel with the LTTng tracing session, I ran
>
>    perf stat -e
>
> net:netif_rx_ni_entry,net:netif_receive_skb_list_entry,net:napi_gro_receive_entry
>
> The above allowed me to confirm that there were
> "net:napi_gro_receive_entry_events", but no "net:netif_rx_ni_entry" or
> "net:netif_receive_skb_list_entry" occurrences in my test.
>
> Are you seeing a situation where another tool is recording occurrences
> of the events that you think aren't being seen during your tracing session?
>

No, nothing, I'm running my client/server application inside wsl and no
other tools are running.

>
> thanks,
> kienan
>

[-- Attachment #1.2: Type: text/html, Size: 2423 bytes --]

[-- Attachment #2: Type: text/plain, Size: 156 bytes --]

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [lttng-dev] Can't receive a lot of functions that i registerd as trace-points

[Kienan Stewart via lttng-dev]

Hi Lin,

On 2023-10-25 04:54, Lin ArcX wrote:
>     Are you seeing a situation where another tool is recording occurrences
>     of the events that you think aren't being seen during your tracing
>     session?
> 
> 
> No, nothing, I'm running my client/server application inside wsl and no 
> other tools are running.
> 

Perhaps your test program isn't causing the WSL kernel to hit the 
`net_if_rx_ni_entry`, `net_napi_gro_receive_entry`, and 
`net_if_receive_skb_list_entry tracepoints` during your tracing session?

I recommend using `perf` to double-check that there are indeed none of 
those tracepoints being hit during a tracing session.

thanks,
kienan
_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev