Question about LTTng Kernel Probe Tracking

Question about LTTng Kernel Probe Tracking

[Amir Najafi Zadeh via lttng-dev]

[-- Attachment #1: Type: text/plain, Size: 2341 bytes --]

Hello all,

I hope you are doing well.

I have a quick question about LTTng tracking when using kernel probes. I
mainly use LTTng to trace file system functions in the Linux kernel, and
for that I use the LTTng kernel probe feature, which works very well.
However, when I tried filtering logs for a specific group ID, I noticed
that filters are not supported for kernel probes.

Then I attempted to use the tracking feature to trace only a specific GID,
but this also did not work. From what I see in my logs, it appears that for
kernel probes all events are captured and recorded, regardless of the
tracking options.

I wanted to know if this is a known limitation, and whether there is any
in-progress work or reason why this is not currently supported.

For reference, here is my code. I am using LTTng 2.13.11 with kernel
6.8.0-101-generic:

```
#!/bin/sh
SESSION_NAME="lttng-gid-test"
OUTPUT_DIR="/tmp/lttng/${SESSION_NAME}"
GROUP_ID=1002

# create the lttng session
lttng create "$SESSION_NAME" -o "$OUTPUT_DIR"

# create the lttng ring buffer channel
lttng enable-channel --session="$SESSION_NAME" --kernel channel0 \
  --subbuf-size=16M \
  --num-subbuf=2

# add pid, tid, proc, and gid for tracing collection
lttng add-context --session="$SESSION_NAME" --channel=channel0
--kernel --type pid
lttng add-context --session="$SESSION_NAME" --channel=channel0
--kernel --type gid

# enable the target probe
lttng enable-event --session="$SESSION_NAME" --channel=channel0
--kernel --probe=ext4_get_inode_loc ext4_get_inode_loc

# disable all tracks
# NOTE: untracking doesn't make a difference
# lttng untrack --session="$SESSION_NAME" --kernel --all --gid
# lttng untrack --session="$SESSION_NAME" --kernel --all --pid
# lttng untrack --session="$SESSION_NAME" --kernel --all --vpid
# lttng untrack --session="$SESSION_NAME" --kernel --all --uid
# lttng untrack --session="$SESSION_NAME" --kernel --all --vuid
# lttng untrack --session="$SESSION_NAME" --kernel --all --vgid

# only enable gid 1002 track
lttng track --kernel --gid=1002

# start lttng
lttng start $SESSION_NAME

```

Thank you for your time and guidance.

Best regards,
Amir
-- 
*Amirhossein Najafizadeh*



*PhD Student, Computer Science Department, Stony Brook University, N.Y.File
systems and Storage Lab (FSL)*
najafizadeh21@gmail.com
https://amirhnajafiz.github.io/

[-- Attachment #2: Type: text/html, Size: 4534 bytes --]

Re: Question about LTTng Kernel Probe Tracking

[Kienan Stewart via lttng-dev]

Hi Amir,

On 2026-04-04 21:02, Amir Najafi Zadeh via lttng-dev wrote:
> Hello all,
> 
> I hope you are doing well.
> 
> I have a quick question about LTTng tracking when using kernel probes. I 
> mainly use LTTng to trace file system functions in the Linux kernel, and 
> for that I use the LTTng kernel probe feature, which works very well. 
> However, when I tried filtering logs for a specific group ID, I noticed 
> that filters are not supported for kernel probes.
> 
> Then I attempted to use the tracking feature to trace only a specific 
> GID, but this also did not work. From what I see in my logs, it appears 
> that for kernel probes all events are captured and recorded, regardless 
> of the tracking options.
> 
> I wanted to know if this is a known limitation, and whether there is any 
> in-progress work or reason why this is not currently supported.
> 

I just tried on the master branch (since I had easy access to it), and 
wasn't able to reproduce a situation where I added the gid tracking and 
had results for a different gid in lttng view.

> For reference, here is my code. I am using LTTng 2.13.11 with kernel 
> 6.8.0-101-generic:
> 
> ``` #!/bin/sh
> SESSION_NAME="lttng-gid-test"
> OUTPUT_DIR="/tmp/lttng/${SESSION_NAME}"
> GROUP_ID=1002
> 
> # create the lttng session
> lttng create "$SESSION_NAME" -o "$OUTPUT_DIR"
> 
> # create the lttng ring buffer channel
> lttng enable-channel --session="$SESSION_NAME" --kernel channel0 \
> --subbuf-size=16M \
> --num-subbuf=2
> 
> # add pid, tid, proc, and gid for tracing collection
> lttng add-context --session="$SESSION_NAME" --channel=channel0 --kernel 
> --type pid
> lttng add-context --session="$SESSION_NAME" --channel=channel0 --kernel 
> --type gid
> 
> # enable the target probe
> lttng enable-event --session="$SESSION_NAME" --channel=channel0 --kernel 
> --probe=ext4_get_inode_loc ext4_get_inode_loc
> 
> # disable all tracks # NOTE: untracking doesn't make a difference
> # lttng untrack --session="$SESSION_NAME" --kernel --all --gid
> # lttng untrack --session="$SESSION_NAME" --kernel --all --pid
> # lttng untrack --session="$SESSION_NAME" --kernel --all --vpid
> # lttng untrack --session="$SESSION_NAME" --kernel --all --uid
> # lttng untrack --session="$SESSION_NAME" --kernel --all --vuid
> # lttng untrack --session="$SESSION_NAME" --kernel --all --vgid
> # only enable gid 1002 track
> lttng track --kernel --gid=1002
> 

Could you check the output of your script after "lttng track --kernel 
--gid=1002"?

The rest of your commands specify a session name, but this one does not. 
I wonder if  it's selecting a different default session that the one 
you're expecting?

> # start lttng lttng start $SESSION_NAME
> 
> ```
> 
> Thank you for your time and guidance.
> 
> Best regards,
> Amir
> 
> -- 
> *Amirhossein Najafizadeh*
> *PhD Student, Computer Science Department, Stony Brook University, N.Y.
> File systems and Storage Lab (FSL)
> 
> *
> najafizadeh21@gmail.com <mailto:najafizadeh21@gmail.com>
> https://amirhnajafiz.github.io/ <https://amirhnajafiz.github.io/>

Re: Question about LTTng Kernel Probe Tracking

[Amir Najafi Zadeh via lttng-dev]

[-- Attachment #1: Type: text/plain, Size: 4706 bytes --]

Added LTTng dev to BCC to avoid spamming.

Hi Kienan. Thanks for your response. I tried specifying the session name,
but nothing changed.

Here is the output I get after adding the tracker:

Added 1002 to the group ID tracker inclusion set

I don’t think the session name is the issue, because I also specified the
session name when starting tracking. This is the session state after
creating it:

$ sudo lttng list lttng-gid-test
Recording session lttng-gid-test: [active]
    Trace output: /tmp/lttng/lttng-gid-test

=== Domain: Linux kernel ===

Tracked process attributes
  Process IDs:          none
  Virtual process IDs:  none
  User IDs:             none
  Virtual user IDs:     none
  Group IDs:            1002
  Virtual group IDs:    none

And these are the logs I get:

[11:01:51.222277079] (+0.000004846) dhcp15 ext4_get_inode_loc: {
cpu_id = 3 }, { pid = 403653, tid = 403660, procname =
"lttng-consumerd", gid = 0, uid = 0, vpid = 403653, vuid = 0, vgid = 0
}, { ip = 0xFFFFFFFFAC804940 }
...

There are many logs like this from systemd, kworker, and processes
belonging to other groups. Note that the GID 1002 that I want to trace
currently has no active subjects.

Do you suggest switching to the recent version of LTTng (current version
2.13.11)?

Best,
Amir

On Tue, Apr 7, 2026 at 10:57 AM Kienan Stewart <kstewart@efficios.com>
wrote:

> Hi Amir,
>
> On 2026-04-04 21:02, Amir Najafi Zadeh via lttng-dev wrote:
> > Hello all,
> >
> > I hope you are doing well.
> >
> > I have a quick question about LTTng tracking when using kernel probes. I
> > mainly use LTTng to trace file system functions in the Linux kernel, and
> > for that I use the LTTng kernel probe feature, which works very well.
> > However, when I tried filtering logs for a specific group ID, I noticed
> > that filters are not supported for kernel probes.
> >
> > Then I attempted to use the tracking feature to trace only a specific
> > GID, but this also did not work. From what I see in my logs, it appears
> > that for kernel probes all events are captured and recorded, regardless
> > of the tracking options.
> >
> > I wanted to know if this is a known limitation, and whether there is any
> > in-progress work or reason why this is not currently supported.
> >
>
> I just tried on the master branch (since I had easy access to it), and
> wasn't able to reproduce a situation where I added the gid tracking and
> had results for a different gid in lttng view.
>
> > For reference, here is my code. I am using LTTng 2.13.11 with kernel
> > 6.8.0-101-generic:
> >
> > ``` #!/bin/sh
> > SESSION_NAME="lttng-gid-test"
> > OUTPUT_DIR="/tmp/lttng/${SESSION_NAME}"
> > GROUP_ID=1002
> >
> > # create the lttng session
> > lttng create "$SESSION_NAME" -o "$OUTPUT_DIR"
> >
> > # create the lttng ring buffer channel
> > lttng enable-channel --session="$SESSION_NAME" --kernel channel0 \
> > --subbuf-size=16M \
> > --num-subbuf=2
> >
> > # add pid, tid, proc, and gid for tracing collection
> > lttng add-context --session="$SESSION_NAME" --channel=channel0 --kernel
> > --type pid
> > lttng add-context --session="$SESSION_NAME" --channel=channel0 --kernel
> > --type gid
> >
> > # enable the target probe
> > lttng enable-event --session="$SESSION_NAME" --channel=channel0 --kernel
> > --probe=ext4_get_inode_loc ext4_get_inode_loc
> >
> > # disable all tracks # NOTE: untracking doesn't make a difference
> > # lttng untrack --session="$SESSION_NAME" --kernel --all --gid
> > # lttng untrack --session="$SESSION_NAME" --kernel --all --pid
> > # lttng untrack --session="$SESSION_NAME" --kernel --all --vpid
> > # lttng untrack --session="$SESSION_NAME" --kernel --all --uid
> > # lttng untrack --session="$SESSION_NAME" --kernel --all --vuid
> > # lttng untrack --session="$SESSION_NAME" --kernel --all --vgid
> > # only enable gid 1002 track
> > lttng track --kernel --gid=1002
> >
>
> Could you check the output of your script after "lttng track --kernel
> --gid=1002"?
>
> The rest of your commands specify a session name, but this one does not.
> I wonder if  it's selecting a different default session that the one
> you're expecting?
>
> > # start lttng lttng start $SESSION_NAME
> >
> > ```
> >
> > Thank you for your time and guidance.
> >
> > Best regards,
> > Amir
> >
> > --
> > *Amirhossein Najafizadeh*
> > *PhD Student, Computer Science Department, Stony Brook University, N.Y.
> > File systems and Storage Lab (FSL)
> >
> > *
> > najafizadeh21@gmail.com <mailto:najafizadeh21@gmail.com>
> > https://amirhnajafiz.github.io/ <https://amirhnajafiz.github.io/>
>
>

[-- Attachment #2: Type: text/html, Size: 6659 bytes --]

Re: Question about LTTng Kernel Probe Tracking

[Kienan Stewart via lttng-dev]

Hi Amir,

On 2026-04-07 11:14, Amir Najafi Zadeh wrote:
> Added LTTng dev to BCC to avoid spamming.
> 
> Hi Kienan. Thanks for your response. I tried specifying the session 
> name, but nothing changed.
> 
> Here is the output I get after adding the tracker:
> 
> |Added 1002 to the group ID tracker inclusion set|
> 
> I don’t think the session name is the issue, because I also specified 
> the session name when starting tracking. This is the session state after 
> creating it:
> 
> |$ sudo lttng list lttng-gid-test Recording session lttng-gid-test: 
> [active] Trace output: /tmp/lttng/lttng-gid-test === Domain: Linux 
> kernel === Tracked process attributes Process IDs: none Virtual process 
> IDs: none User IDs: none Virtual user IDs: none Group IDs: 1002 Virtual 
> group IDs: none|
> 
> And these are the logs I get:
> 
> |[11:01:51.222277079] (+0.000004846) dhcp15 ext4_get_inode_loc: { cpu_id 
> = 3 }, { pid = 403653, tid = 403660, procname = "lttng-consumerd", gid = 
> 0, uid = 0, vpid = 403653, vuid = 0, vgid = 0 }, { ip = 
> 0xFFFFFFFFAC804940 } ...|
> 
> There are many logs like this from systemd, kworker, and processes 
> belonging to other groups. Note that the GID 1002 that I want to trace 
> currently has no active subjects.
> 
> Do you suggest switching to the recent version of LTTng (current version 
> 2.13.11)?
> 

I just tested with stable-2.13 and stable-2.14. I was able to reproduce 
the issue you saw in stable-2.13, but not stable-2.14

I recommend updating to 2.14 or later.

thanks,
kienan

> Best,
> Amir
> 
> 
> On Tue, Apr 7, 2026 at 10:57 AM Kienan Stewart <kstewart@efficios.com 
> <mailto:kstewart@efficios.com>> wrote:
> 
>     Hi Amir,
> 
>     On 2026-04-04 21:02, Amir Najafi Zadeh via lttng-dev wrote:
>      > Hello all,
>      >
>      > I hope you are doing well.
>      >
>      > I have a quick question about LTTng tracking when using kernel
>     probes. I
>      > mainly use LTTng to trace file system functions in the Linux
>     kernel, and
>      > for that I use the LTTng kernel probe feature, which works very
>     well.
>      > However, when I tried filtering logs for a specific group ID, I
>     noticed
>      > that filters are not supported for kernel probes.
>      >
>      > Then I attempted to use the tracking feature to trace only a
>     specific
>      > GID, but this also did not work. From what I see in my logs, it
>     appears
>      > that for kernel probes all events are captured and recorded,
>     regardless
>      > of the tracking options.
>      >
>      > I wanted to know if this is a known limitation, and whether there
>     is any
>      > in-progress work or reason why this is not currently supported.
>      >
> 
>     I just tried on the master branch (since I had easy access to it), and
>     wasn't able to reproduce a situation where I added the gid tracking and
>     had results for a different gid in lttng view.
> 
>      > For reference, here is my code. I am using LTTng 2.13.11 with kernel
>      > 6.8.0-101-generic:
>      >
>      > ``` #!/bin/sh
>      > SESSION_NAME="lttng-gid-test"
>      > OUTPUT_DIR="/tmp/lttng/${SESSION_NAME}"
>      > GROUP_ID=1002
>      >
>      > # create the lttng session
>      > lttng create "$SESSION_NAME" -o "$OUTPUT_DIR"
>      >
>      > # create the lttng ring buffer channel
>      > lttng enable-channel --session="$SESSION_NAME" --kernel channel0 \
>      > --subbuf-size=16M \
>      > --num-subbuf=2
>      >
>      > # add pid, tid, proc, and gid for tracing collection
>      > lttng add-context --session="$SESSION_NAME" --channel=channel0 --
>     kernel
>      > --type pid
>      > lttng add-context --session="$SESSION_NAME" --channel=channel0 --
>     kernel
>      > --type gid
>      >
>      > # enable the target probe
>      > lttng enable-event --session="$SESSION_NAME" --channel=channel0
>     --kernel
>      > --probe=ext4_get_inode_loc ext4_get_inode_loc
>      >
>      > # disable all tracks # NOTE: untracking doesn't make a difference
>      > # lttng untrack --session="$SESSION_NAME" --kernel --all --gid
>      > # lttng untrack --session="$SESSION_NAME" --kernel --all --pid
>      > # lttng untrack --session="$SESSION_NAME" --kernel --all --vpid
>      > # lttng untrack --session="$SESSION_NAME" --kernel --all --uid
>      > # lttng untrack --session="$SESSION_NAME" --kernel --all --vuid
>      > # lttng untrack --session="$SESSION_NAME" --kernel --all --vgid
>      > # only enable gid 1002 track
>      > lttng track --kernel --gid=1002
>      >
> 
>     Could you check the output of your script after "lttng track --kernel
>     --gid=1002"?
> 
>     The rest of your commands specify a session name, but this one does
>     not.
>     I wonder if  it's selecting a different default session that the one
>     you're expecting?
> 
>      > # start lttng lttng start $SESSION_NAME
>      >
>      > ```
>      >
>      > Thank you for your time and guidance.
>      >
>      > Best regards,
>      > Amir
>      >
>      > --
>      > *Amirhossein Najafizadeh*
>      > *PhD Student, Computer Science Department, Stony Brook
>     University, N.Y.
>      > File systems and Storage Lab (FSL)
>      >
>      > *
>      > najafizadeh21@gmail.com <mailto:najafizadeh21@gmail.com>
>     <mailto:najafizadeh21@gmail.com <mailto:najafizadeh21@gmail.com>>
>      > https://amirhnajafiz.github.io/ <https://amirhnajafiz.github.io/>
>     <https://amirhnajafiz.github.io/ <https://amirhnajafiz.github.io/>>
>