[lttng-dev] loading modules with modprobe

[lttng-dev] loading modules with modprobe

[François Belias via lttng-dev]

[-- Attachment #1.1: Type: text/plain, Size: 1159 bytes --]

Hello,

I'm currently running Linux kernel version 6.5.0-41-generic and attempting
to install LTTng from source. Here are the steps I followed:

 - I cloned the LTTng git repository to my computer from [
https://github.com/lttng/lttng-modules.git](https://github.com/lttng/lttng-modules.git)
.
- Downloaded the kernel source tree corresponding to my kernel version to
enable compilation.
- Executed `make KERNELDIR=/path/to/my/linux/source`, which was completed
successfully.
- Ran `make KERNELDIR=/path/to/my/linux/source modules_install`, also
without any issues.
- Finally, I executed `sudo depmod -a`, which was completed successfully.

However, when I attempt to load a module with commands like `sudo modprobe
lttng-ketprobes` or "sudo modprobe lttng-tracer", I encounter the following
errors:

- For `lttng-ketprobes`: `modprobe: FATAL: Module lttng-ketprobes not found
in directory /lib/modules/6.5.0-41-generic`.
- For `lttng-tracer`: `modprobe: ERROR: could not insert 'lttng_tracer':
Exec format error`.

I've confirmed that the modules exist but can't load them. I'm unsure how
to resolve these issues. Could someone please assist me?

Thank you.

[-- Attachment #1.2: Type: text/html, Size: 1371 bytes --]

[-- Attachment #2: Type: text/plain, Size: 156 bytes --]

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [lttng-dev] loading modules with modprobe

[Kienan Stewart via lttng-dev]

Hi François,

On 7/15/24 6:10 PM, François Belias via lttng-dev wrote:
> Hello,
> 
> I'm currently running Linux kernel version 6.5.0-41-generic and 
> attempting to install LTTng from source. Here are the steps I followed:
> 
>   - I cloned the LTTng git repository to my computer from 
> [https://github.com/lttng/lttng-modules.git](https://github.com/lttng/lttng-modules.git) <https://github.com/lttng/lttng-modules.git](https://github.com/lttng/lttng-modules.git)>.
> - Downloaded the kernel source tree corresponding to my kernel version 
> to enable compilation.
> - Executed `make KERNELDIR=/path/to/my/linux/source`, which was 
> completed successfully.
> - Ran `make KERNELDIR=/path/to/my/linux/source modules_install`, also 
> without any issues.
> - Finally, I executed `sudo depmod -a`, which was completed successfully.
> 
> However, when I attempt to load a module with commands like `sudo 
> modprobe lttng-ketprobes` or "sudo modprobe lttng-tracer", I encounter 
> the following errors:
> 
> - For `lttng-ketprobes`: `modprobe: FATAL: Module lttng-ketprobes not 
> found in directory /lib/modules/6.5.0-41-generic`.
> - For `lttng-tracer`: `modprobe: ERROR: could not insert 'lttng_tracer': 
> Exec format error`.

If you rebuilt modules on the master branch as of yesterday you may need 
to clean the LTTng modules install directory, rebuild, and reinstall.

This is due to kprobes, kretprobes, and uprobes modules being included 
in the tracer core module. C.f. 
https://github.com/lttng/lttng-modules/commit/0c2c6606eb66cb5222ab7a2a4f8329fd5dfdef70

You may want to stick with running lttng-modules 2.13.x which will not 
have breaking changes merged in.

The exec format error also could be due to modules that are installed 
but built against a different version of the Linux kernel.

A few things to check

- If your kernel has been upgraded recently, ensure the new kernel is 
the one actually running
- If the modules haven't been rebuilt since the upgrade, they should be 
rebuilt and reinstalled
- As I recall, you are playing around with some addon modules - do any 
of those modules define dependencies on the kprobes, kretprobes, or 
uprobes modules which now are no longer produced by builds of the master 
branch

Is there further information in dmesg or journalctl regarding the module 
loading?

hope this helps.

thanks,
kienan

> 
> I've confirmed that the modules exist but can't load them. I'm unsure 
> how to resolve these issues. Could someone please assist me?
> 
> Thank you.
> 
> _______________________________________________
> lttng-dev mailing list
> lttng-dev@lists.lttng.org
> https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [lttng-dev] loading modules with modprobe

[François Belias via lttng-dev]

[-- Attachment #1.1: Type: text/plain, Size: 4418 bytes --]

Hello Kienan,

I've continued investigating my issue and discovered something new that
might shed some light. Here's the rundown:

1. To ensure my add-ons weren't causing problems, deleted all old modules
and I followed the documentation on the LTTng website (
https://lttng.org/docs/v2.13/#doc-building-from-source) and downloaded
`lttng-modules-2.13.13`.

2. Inside the downloaded folder, I successfully executed `make
KERNELDIR=/usr/src/linux-source-6.5.0` without any issues.

3. Subsequently, I ran `make KERNELDIR=/usr/src/linux-source-6.5.0
modules_install`, which also completed successfully.

4. After that, I updated the module dependencies using `sudo depmod -a`,
which went fine.

However, when I tried to load a module, such as `sudo modprobe
lttng-probe-kvm`, it reported that the module couldn't be found at
`/lib/modules/6.5.0-41-generic/`. Upon checking the logs, I confirmed that
step 3 had installed the modules in `/lib/modules/6.5.0/`.

To address this, I modified step 3 to simply `make modules_install`, which
then installed the modules correctly at `/lib/modules/6.5.0-41-generic/`
and ran again step 4. But now, when attempting to load a module, I'm
encountering the error `modprobe: ERROR: could not insert
'lttng_probe_kvm': Exec format error`.

Do you have any insights or ideas on what might be causing this issue?

Thanks again for your assistance.




















Le mar. 16 juill. 2024, à 09 h 05, Kienan Stewart <kstewart@efficios.com> a
écrit :

> Hi François,
>
> On 7/15/24 6:10 PM, François Belias via lttng-dev wrote:
> > Hello,
> >
> > I'm currently running Linux kernel version 6.5.0-41-generic and
> > attempting to install LTTng from source. Here are the steps I followed:
> >
> >   - I cloned the LTTng git repository to my computer from
> > [
> https://github.com/lttng/lttng-modules.git](https://github.com/lttng/lttng-modules.git)
> <
> https://github.com/lttng/lttng-modules.git](https://github.com/lttng/lttng-modules.git)
> >.
> > - Downloaded the kernel source tree corresponding to my kernel version
> > to enable compilation.
> > - Executed `make KERNELDIR=/path/to/my/linux/source`, which was
> > completed successfully.
> > - Ran `make KERNELDIR=/path/to/my/linux/source modules_install`, also
> > without any issues.
> > - Finally, I executed `sudo depmod -a`, which was completed successfully.
> >
> > However, when I attempt to load a module with commands like `sudo
> > modprobe lttng-ketprobes` or "sudo modprobe lttng-tracer", I encounter
> > the following errors:
> >
> > - For `lttng-ketprobes`: `modprobe: FATAL: Module lttng-ketprobes not
> > found in directory /lib/modules/6.5.0-41-generic`.
> > - For `lttng-tracer`: `modprobe: ERROR: could not insert 'lttng_tracer':
> > Exec format error`.
>
> If you rebuilt modules on the master branch as of yesterday you may need
> to clean the LTTng modules install directory, rebuild, and reinstall.
>
> This is due to kprobes, kretprobes, and uprobes modules being included
> in the tracer core module. C.f.
>
> https://github.com/lttng/lttng-modules/commit/0c2c6606eb66cb5222ab7a2a4f8329fd5dfdef70
>
> You may want to stick with running lttng-modules 2.13.x which will not
> have breaking changes merged in.
>
> The exec format error also could be due to modules that are installed
> but built against a different version of the Linux kernel.
>
> A few things to check
>
> - If your kernel has been upgraded recently, ensure the new kernel is
> the one actually running
> - If the modules haven't been rebuilt since the upgrade, they should be
> rebuilt and reinstalled
> - As I recall, you are playing around with some addon modules - do any
> of those modules define dependencies on the kprobes, kretprobes, or
> uprobes modules which now are no longer produced by builds of the master
> branch
>
> Is there further information in dmesg or journalctl regarding the module
> loading?
>
> hope this helps.
>
> thanks,
> kienan
>
> >
> > I've confirmed that the modules exist but can't load them. I'm unsure
> > how to resolve these issues. Could someone please assist me?
> >
> > Thank you.
> >
> > _______________________________________________
> > lttng-dev mailing list
> > lttng-dev@lists.lttng.org
> > https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
>

[-- Attachment #1.2: Type: text/html, Size: 5952 bytes --]

[-- Attachment #2: Type: text/plain, Size: 156 bytes --]

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [lttng-dev] loading modules with modprobe

[Kienan Stewart via lttng-dev]

Hi François,

On 7/16/24 3:01 PM, François Belias wrote:
> Hello Kienan,
> 
> I've continued investigating my issue and discovered something new that 
> might shed some light. Here's the rundown:
> 
> 1. To ensure my add-ons weren't causing problems, deleted all old 
> modules and I followed the documentation on the LTTng website 
> (https://lttng.org/docs/v2.13/#doc-building-from-source 
> <https://lttng.org/docs/v2.13/#doc-building-from-source>) and downloaded 
> `lttng-modules-2.13.13`.
> 
> 2. Inside the downloaded folder, I successfully executed `make 
> KERNELDIR=/usr/src/linux-source-6.5.0` without any issues.
> 
> 3. Subsequently, I ran `make KERNELDIR=/usr/src/linux-source-6.5.0 
> modules_install`, which also completed successfully.
> 
> 4. After that, I updated the module dependencies using `sudo depmod -a`, 
> which went fine.
> 
> However, when I tried to load a module, such as `sudo modprobe 
> lttng-probe-kvm`, it reported that the module couldn't be found at 
> `/lib/modules/6.5.0-41-generic/`. Upon checking the logs, I confirmed 
> that step 3 had installed the modules in `/lib/modules/6.5.0/`.
> 
> To address this, I modified step 3 to simply `make modules_install`, 
> which then installed the modules correctly at 
> `/lib/modules/6.5.0-41-generic/` and ran again step 4. But now, when 
> attempting to load a module, I'm encountering the error `modprobe: 
> ERROR: could not insert 'lttng_probe_kvm': Exec format error`.
> 
> Do you have any insights or ideas on what might be causing this issue?
> 

It sounds like /usr/src/linux-source-6.5.0 is not correct source for the 
kernel that you are currently running.

When running `make modules_install` in your modified step 3 (without 
KERNELDIR set), the modules were installed for your current kernel 
directory as auto-detected by the build script.

What happens if you do the following

```
unset KERNELDIR

# Warning: this removes all untracked content from the lttng-modules 
directory
cd /path/to/lttng-modules-source
git clean -dxf

make
sudo make modules_install
sudo depmod -a
```

KERNELDIR is set based on the current running kernel: `uname -r`. C.f. 
https://github.com/lttng/lttng-modules/blob/434e86eea846b33cad57f1d8105089905f541853/Makefile#L16C1-L16C50

A couple of questions for you:

- Where did /usr/src/linux-source-6.5.0 come from?
- Which linux distribution + version are you using?
- Assuming your distribution has kernel packages, which do you have 
installed, and do you have the corresponding kernel headers package 
installed?

thanks,
kienan

> Thanks again for your assistance.
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> Le mar. 16 juill. 2024, à 09 h 05, Kienan Stewart <kstewart@efficios.com 
> <mailto:kstewart@efficios.com>> a écrit :
> 
>     Hi François,
> 
>     On 7/15/24 6:10 PM, François Belias via lttng-dev wrote:
>      > Hello,
>      >
>      > I'm currently running Linux kernel version 6.5.0-41-generic and
>      > attempting to install LTTng from source. Here are the steps I
>     followed:
>      >
>      >   - I cloned the LTTng git repository to my computer from
>      >
>     [https://github.com/lttng/lttng-modules.git](https://github.com/lttng/lttng-modules.git) <https://github.com/lttng/lttng-modules.git%5D(https://github.com/lttng/lttng-modules.git)> <https://github.com/lttng/lttng-modules.git](https://github.com/lttng/lttng-modules.git) <https://github.com/lttng/lttng-modules.git%5D(https://github.com/lttng/lttng-modules.git)>>.
>      > - Downloaded the kernel source tree corresponding to my kernel
>     version
>      > to enable compilation.
>      > - Executed `make KERNELDIR=/path/to/my/linux/source`, which was
>      > completed successfully.
>      > - Ran `make KERNELDIR=/path/to/my/linux/source modules_install`,
>     also
>      > without any issues.
>      > - Finally, I executed `sudo depmod -a`, which was completed
>     successfully.
>      >
>      > However, when I attempt to load a module with commands like `sudo
>      > modprobe lttng-ketprobes` or "sudo modprobe lttng-tracer", I
>     encounter
>      > the following errors:
>      >
>      > - For `lttng-ketprobes`: `modprobe: FATAL: Module lttng-ketprobes
>     not
>      > found in directory /lib/modules/6.5.0-41-generic`.
>      > - For `lttng-tracer`: `modprobe: ERROR: could not insert
>     'lttng_tracer':
>      > Exec format error`.
> 
>     If you rebuilt modules on the master branch as of yesterday you may
>     need
>     to clean the LTTng modules install directory, rebuild, and reinstall.
> 
>     This is due to kprobes, kretprobes, and uprobes modules being included
>     in the tracer core module. C.f.
>     https://github.com/lttng/lttng-modules/commit/0c2c6606eb66cb5222ab7a2a4f8329fd5dfdef70 <https://github.com/lttng/lttng-modules/commit/0c2c6606eb66cb5222ab7a2a4f8329fd5dfdef70>
> 
>     You may want to stick with running lttng-modules 2.13.x which will not
>     have breaking changes merged in.
> 
>     The exec format error also could be due to modules that are installed
>     but built against a different version of the Linux kernel.
> 
>     A few things to check
> 
>     - If your kernel has been upgraded recently, ensure the new kernel is
>     the one actually running
>     - If the modules haven't been rebuilt since the upgrade, they should be
>     rebuilt and reinstalled
>     - As I recall, you are playing around with some addon modules - do any
>     of those modules define dependencies on the kprobes, kretprobes, or
>     uprobes modules which now are no longer produced by builds of the
>     master
>     branch
> 
>     Is there further information in dmesg or journalctl regarding the
>     module
>     loading?
> 
>     hope this helps.
> 
>     thanks,
>     kienan
> 
>      >
>      > I've confirmed that the modules exist but can't load them. I'm
>     unsure
>      > how to resolve these issues. Could someone please assist me?
>      >
>      > Thank you.
>      >
>      > _______________________________________________
>      > lttng-dev mailing list
>      > lttng-dev@lists.lttng.org <mailto:lttng-dev@lists.lttng.org>
>      > https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
>     <https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev>
> 
_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [lttng-dev] loading modules with modprobe

[François Belias via lttng-dev]

[-- Attachment #1.1: Type: text/plain, Size: 10223 bytes --]

Hello Kienan,

When I proceed with the following steps:

```
# Warning: this removes all untracked content from the lttng-modules
directory
cd /path/to/lttng-modules-source
git clean -dxf

make
sudo make modules_install
sudo depmod -a
```

I encounter a warning in
`/home/philippe/lttng-modules/src/probes/Kbuild:79` stating "File
./arch/x86/kvm/lapic.h not found. Probe 'kvm' x86-specific is disabled. Use
full kernel source tree to enable it." Although everything compiles fine
and I'm able to load modules, when I execute `sudo lttng list -k`, I don't
see the tracepoints `kvm_entry` and `kvm_exit`, which I believe should be
available. Could this be due to not using my full kernel tree during
compilation?

Here are the answers to your questions:

- *Where did `/usr/src/linux-source-6.5.0` come from?*
  I initially attempted `sudo apt-get update && sudo apt-get install
linux-source`, but it gave me this answer *linux-source is already the
newest version (5.15.0.116.116)*`, which doesn't match my current kernel
version. So, I downloaded it directly from kernel.org using this command:
`wget https://cdn.kernel.org/pub/linux/kernel/v6.x/linux-6.5.tar.xz`.

- *Which Linux distribution and version are you using?*
  Running `lsb_release -a` gives me:
  ```
  No LSB modules are available.
  Distributor ID: Ubuntu
  Description:    Ubuntu 22.04.4 LTS
  Release:        22.04
  Codename:       jammy
  ```

- *Assuming your distribution has kernel packages, which do you have
installed, and do you have the corresponding kernel headers package
installed?*

  Running `dpkg -l | grep linux-image` shows these installed packages:
  ```
  ii  linux-image-5.15.0-116-generic           5.15.0-116.126      amd64
Signed kernel image generic
  ii  linux-image-6.5.0-28-generic             6.5.0-28.29~22.04.1  amd64
Signed kernel image generic
  ii  linux-image-6.5.0-41-generic             6.5.0-41.41~22.04.2  amd64
Signed kernel image generic
  ii  linux-image-6.5.0-44-generic             6.5.0-44.44~22.04.1  amd64
Signed kernel image generic
  ii  linux-image-generic                      5.15.0.116.116      amd64
Generic Linux kernel image
  ii  linux-image-generic-hwe-22.04            6.5.0.44.44~22.04.1  amd64
Generic Linux kernel image
  ```

  And `dpkg -l | grep linux-headers` shows these installed header packages:
  ```
  ii  linux-headers-5.15.0-116                 5.15.0-116.126      all
Header files related to Linux kernel version 5.15.0
  ii  linux-headers-5.15.0-116-generic         5.15.0-116.126      amd64
Linux kernel headers for version 5.15.0 on 64 bit x86 SMP
  ii  linux-headers-6.5.0-28-generic           6.5.0-28.29~22.04.1  amd64
Linux kernel headers for version 6.5.0 on 64 bit x86 SMP
  ii  linux-headers-6.5.0-41-generic           6.5.0-41.41~22.04.2  amd64
Linux kernel headers for version 6.5.0 on 64 bit x86 SMP
  ii  linux-headers-6.5.0-44-generic           6.5.0-44.44~22.04.1  amd64
Linux kernel headers for version 6.5.0 on 64 bit x86 SMP
  ii  linux-headers-generic                    5.15.0.116.116      amd64
Generic Linux kernel headers
  ii  linux-headers-generic-hwe-22.04          6.5.0.44.44~22.04.1  amd64
Generic Linux kernel headers
  ```

Thanks.
















Le mar. 16 juill. 2024, à 15 h 41, Kienan Stewart <kstewart@efficios.com> a
écrit :

> Hi François,
>
> On 7/16/24 3:01 PM, François Belias wrote:
> > Hello Kienan,
> >
> > I've continued investigating my issue and discovered something new that
> > might shed some light. Here's the rundown:
> >
> > 1. To ensure my add-ons weren't causing problems, deleted all old
> > modules and I followed the documentation on the LTTng website
> > (https://lttng.org/docs/v2.13/#doc-building-from-source
> > <https://lttng.org/docs/v2.13/#doc-building-from-source>) and
> downloaded
> > `lttng-modules-2.13.13`.
> >
> > 2. Inside the downloaded folder, I successfully executed `make
> > KERNELDIR=/usr/src/linux-source-6.5.0` without any issues.
> >
> > 3. Subsequently, I ran `make KERNELDIR=/usr/src/linux-source-6.5.0
> > modules_install`, which also completed successfully.
> >
> > 4. After that, I updated the module dependencies using `sudo depmod -a`,
> > which went fine.
> >
> > However, when I tried to load a module, such as `sudo modprobe
> > lttng-probe-kvm`, it reported that the module couldn't be found at
> > `/lib/modules/6.5.0-41-generic/`. Upon checking the logs, I confirmed
> > that step 3 had installed the modules in `/lib/modules/6.5.0/`.
> >
> > To address this, I modified step 3 to simply `make modules_install`,
> > which then installed the modules correctly at
> > `/lib/modules/6.5.0-41-generic/` and ran again step 4. But now, when
> > attempting to load a module, I'm encountering the error `modprobe:
> > ERROR: could not insert 'lttng_probe_kvm': Exec format error`.
> >
> > Do you have any insights or ideas on what might be causing this issue?
> >
>
> It sounds like /usr/src/linux-source-6.5.0 is not correct source for the
> kernel that you are currently running.
>
> When running `make modules_install` in your modified step 3 (without
> KERNELDIR set), the modules were installed for your current kernel
> directory as auto-detected by the build script.
>
> What happens if you do the following
>
> ```
> unset KERNELDIR
>
> # Warning: this removes all untracked content from the lttng-modules
> directory
> cd /path/to/lttng-modules-source
> git clean -dxf
>
> make
> sudo make modules_install
> sudo depmod -a
> ```
>
> KERNELDIR is set based on the current running kernel: `uname -r`. C.f.
>
> https://github.com/lttng/lttng-modules/blob/434e86eea846b33cad57f1d8105089905f541853/Makefile#L16C1-L16C50
>
> A couple of questions for you:
>
> - Where did /usr/src/linux-source-6.5.0 come from?
> - Which linux distribution + version are you using?
> - Assuming your distribution has kernel packages, which do you have
> installed, and do you have the corresponding kernel headers package
> installed?
>
> thanks,
> kienan
>
> > Thanks again for your assistance.
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > Le mar. 16 juill. 2024, à 09 h 05, Kienan Stewart <kstewart@efficios.com
> > <mailto:kstewart@efficios.com>> a écrit :
> >
> >     Hi François,
> >
> >     On 7/15/24 6:10 PM, François Belias via lttng-dev wrote:
> >      > Hello,
> >      >
> >      > I'm currently running Linux kernel version 6.5.0-41-generic and
> >      > attempting to install LTTng from source. Here are the steps I
> >     followed:
> >      >
> >      >   - I cloned the LTTng git repository to my computer from
> >      >
> >     [
> https://github.com/lttng/lttng-modules.git](https://github.com/lttng/lttng-modules.git)
> <
> https://github.com/lttng/lttng-modules.git%5D(https://github.com/lttng/lttng-modules.git)>
> <
> https://github.com/lttng/lttng-modules.git](https://github.com/lttng/lttng-modules.git)
> <
> https://github.com/lttng/lttng-modules.git%5D(https://github.com/lttng/lttng-modules.git)
> >>.
> >      > - Downloaded the kernel source tree corresponding to my kernel
> >     version
> >      > to enable compilation.
> >      > - Executed `make KERNELDIR=/path/to/my/linux/source`, which was
> >      > completed successfully.
> >      > - Ran `make KERNELDIR=/path/to/my/linux/source modules_install`,
> >     also
> >      > without any issues.
> >      > - Finally, I executed `sudo depmod -a`, which was completed
> >     successfully.
> >      >
> >      > However, when I attempt to load a module with commands like `sudo
> >      > modprobe lttng-ketprobes` or "sudo modprobe lttng-tracer", I
> >     encounter
> >      > the following errors:
> >      >
> >      > - For `lttng-ketprobes`: `modprobe: FATAL: Module lttng-ketprobes
> >     not
> >      > found in directory /lib/modules/6.5.0-41-generic`.
> >      > - For `lttng-tracer`: `modprobe: ERROR: could not insert
> >     'lttng_tracer':
> >      > Exec format error`.
> >
> >     If you rebuilt modules on the master branch as of yesterday you may
> >     need
> >     to clean the LTTng modules install directory, rebuild, and reinstall.
> >
> >     This is due to kprobes, kretprobes, and uprobes modules being
> included
> >     in the tracer core module. C.f.
> >
> https://github.com/lttng/lttng-modules/commit/0c2c6606eb66cb5222ab7a2a4f8329fd5dfdef70
> <
> https://github.com/lttng/lttng-modules/commit/0c2c6606eb66cb5222ab7a2a4f8329fd5dfdef70
> >
> >
> >     You may want to stick with running lttng-modules 2.13.x which will
> not
> >     have breaking changes merged in.
> >
> >     The exec format error also could be due to modules that are installed
> >     but built against a different version of the Linux kernel.
> >
> >     A few things to check
> >
> >     - If your kernel has been upgraded recently, ensure the new kernel is
> >     the one actually running
> >     - If the modules haven't been rebuilt since the upgrade, they should
> be
> >     rebuilt and reinstalled
> >     - As I recall, you are playing around with some addon modules - do
> any
> >     of those modules define dependencies on the kprobes, kretprobes, or
> >     uprobes modules which now are no longer produced by builds of the
> >     master
> >     branch
> >
> >     Is there further information in dmesg or journalctl regarding the
> >     module
> >     loading?
> >
> >     hope this helps.
> >
> >     thanks,
> >     kienan
> >
> >      >
> >      > I've confirmed that the modules exist but can't load them. I'm
> >     unsure
> >      > how to resolve these issues. Could someone please assist me?
> >      >
> >      > Thank you.
> >      >
> >      > _______________________________________________
> >      > lttng-dev mailing list
> >      > lttng-dev@lists.lttng.org <mailto:lttng-dev@lists.lttng.org>
> >      > https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
> >     <https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev>
> >
>

[-- Attachment #1.2: Type: text/html, Size: 14008 bytes --]

[-- Attachment #2: Type: text/plain, Size: 156 bytes --]

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [lttng-dev] loading modules with modprobe

[Kienan Stewart via lttng-dev]

Hi François,

On 7/16/24 4:34 PM, François Belias wrote:
> Hello Kienan,
> 
> When I proceed with the following steps:
> 
> ```
> # Warning: this removes all untracked content from the lttng-modules 
> directory
> cd /path/to/lttng-modules-source
> git clean -dxf
> 
> make
> sudo make modules_install
> sudo depmod -a
> ```
> 
> I encounter a warning in 
> `/home/philippe/lttng-modules/src/probes/Kbuild:79` stating "File 
> ./arch/x86/kvm/lapic.h not found. Probe 'kvm' x86-specific is disabled. 
> Use full kernel source tree to enable it." Although everything compiles 
> fine and I'm able to load modules, when I execute `sudo lttng list -k`, 
> I don't see the tracepoints `kvm_entry` and `kvm_exit`, which I believe 
> should be available. Could this be due to not using my full kernel tree 
> during compilation?


Exactly. Those probes depend on having the full kernel source as they 
require private headers which aren't shipped as part of the 
`linux-headers` package.

> 
> Here are the answers to your questions:
> 
> - *Where did `/usr/src/linux-source-6.5.0` come from?*
>    I initially attempted `sudo apt-get update && sudo apt-get install 
> linux-source`, but it gave me this answer */linux-source is already the 
> newest version (5.15.0.116.116)/*`, which doesn't match my current 
> kernel version. So, I downloaded it directly from kernel.org 
> <http://kernel.org> using this command: `wget 
> https://cdn.kernel.org/pub/linux/kernel/v6.x/linux-6.5.tar.xz` 
> <https://cdn.kernel.org/pub/linux/kernel/v6.x/linux-6.5.tar.xz`>.
> 

Right. Which kernel are you actually running at this time? You can check 
using: `uname -a`

In any case, Linux upstream 6.5 is not the same as Ubuntus's 6.5.

> - *Which Linux distribution and version are you using?* 
>    Running `lsb_release -a` gives me:
>    ```
>    No LSB modules are available.
>    Distributor ID: Ubuntu
>    Description:    Ubuntu 22.04.4 LTS
>    Release:        22.04
>    Codename:       jammy
>    ```
> 
> - *Assuming your distribution has kernel packages, which do you have 
> installed, and do you have the corresponding kernel headers package 
> installed?*
> 
>    Running `dpkg -l | grep linux-image` shows these installed packages:
>    ```
>    ii  linux-image-5.15.0-116-generic           5.15.0-116.126     
>   amd64   Signed kernel image generic
>    ii  linux-image-6.5.0-28-generic             6.5.0-28.29~22.04.1 
>   amd64   Signed kernel image generic
>    ii  linux-image-6.5.0-41-generic             6.5.0-41.41~22.04.2 
>   amd64   Signed kernel image generic
>    ii  linux-image-6.5.0-44-generic             6.5.0-44.44~22.04.1 
>   amd64   Signed kernel image generic
>    ii  linux-image-generic                      5.15.0.116.116     
>   amd64   Generic Linux kernel image
>    ii  linux-image-generic-hwe-22.04            6.5.0.44.44~22.04.1 
>   amd64   Generic Linux kernel image
>    ```
> 
>    And `dpkg -l | grep linux-headers` shows these installed header packages:
>    ```
>    ii  linux-headers-5.15.0-116                 5.15.0-116.126      all 
>      Header files related to Linux kernel version 5.15.0
>    ii  linux-headers-5.15.0-116-generic         5.15.0-116.126     
>   amd64   Linux kernel headers for version 5.15.0 on 64 bit x86 SMP
>    ii  linux-headers-6.5.0-28-generic           6.5.0-28.29~22.04.1 
>   amd64   Linux kernel headers for version 6.5.0 on 64 bit x86 SMP
>    ii  linux-headers-6.5.0-41-generic           6.5.0-41.41~22.04.2 
>   amd64   Linux kernel headers for version 6.5.0 on 64 bit x86 SMP
>    ii  linux-headers-6.5.0-44-generic           6.5.0-44.44~22.04.1 
>   amd64   Linux kernel headers for version 6.5.0 on 64 bit x86 SMP
>    ii  linux-headers-generic                    5.15.0.116.116     
>   amd64   Generic Linux kernel headers
>    ii  linux-headers-generic-hwe-22.04          6.5.0.44.44~22.04.1 
>   amd64   Generic Linux kernel headers
>    ```
> 

Since you've got a few different images and headers installed, it is 
possible to check that same info the linux-source package: `dpkg -l | 
grep linux-source`.

Maybe one thing tripping this process up is that you don't have an apt 
source that provides linux-source that matches the kernel you are using. 
Could you run either `apt-cache policy` or `apt policy` and `apt-mark 
showhold` in order to know which sources you have configured.

There are also alternative sources for the full source tree:

- You could download the specific package from packages.ubuntu.com. See 
https://packages.ubuntu.com and use `apt install ./downloaded_package.deb`

- You could clone the kernel source for Ubuntu and checkout the 
appropriate tag. See 
https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/jammy/refs/tags

thanks,
kienan
_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [lttng-dev] loading modules with modprobe

[François Belias via lttng-dev]

[-- Attachment #1.1: Type: text/plain, Size: 7966 bytes --]

Hello, Kienan



*Right. Which kernel are you actually running at this time? You can
checkusing: `uname -a`*
Linux philippe 6.5.0-41-generic #41~22.04.2-Ubuntu SMP PREEMPT_DYNAMIC Mon
Jun  3 11:32:55 UTC 2 x86_64 x86_64 x86_64 GNU/Linux



*Since you've got a few different images and headers installed, it
ispossible to check that same info the linux-source package: `dpkg -l |grep
linux-source`.*
ii  linux-source                                   5.15.0.116.116
                         all          Linux kernel source with Ubuntu
patches
ii  linux-source-5.15.0                       5.15.0-116.126
                     all          Linux kernel source for version 5.15.0
with Ubuntu patches
ii  linux-source-6.5.0                      6.5.0-44.44~22.04.1
                  all          Linux kernel source for version 6.5.0 with
Ubuntu patches

*Could you run either `apt-cache policy` or `apt policy` and
`apt-markshowhold` in order to know which sources you have configured.*
For this part, *apt-mark showhold* gave me nothing I don't know what is
means.

I am looking at the links, but nothing seems to match my kernel version.

Thanks for your assistance.

Le mer. 17 juill. 2024, à 11 h 46, François Belias <beliasossim@gmail.com>
a écrit :

> Hello, Kienan
>
>
>
> *Right. Which kernel are you actually running at this time? You can
> checkusing: `uname -a`*
> Linux philippe 6.5.0-41-generic #41~22.04.2-Ubuntu SMP PREEMPT_DYNAMIC Mon
> Jun  3 11:32:55 UTC 2 x86_64 x86_64 x86_64 GNU/Linux
>
>
>
> *Since you've got a few different images and headers installed, it
> ispossible to check that same info the linux-source package: `dpkg -l |grep
> linux-source`.*
> ii  linux-source                                   5.15.0.116.116
>                            all          Linux kernel source with Ubuntu
> patches
> ii  linux-source-5.15.0                       5.15.0-116.126
>                      all          Linux kernel source for version 5.15.0
> with Ubuntu patches
> ii  linux-source-6.5.0                      6.5.0-44.44~22.04.1
>                     all          Linux kernel source for version 6.5.0 with
> Ubuntu patches
>
> *Could you run either `apt-cache policy` or `apt policy` and
> `apt-markshowhold` in order to know which sources you have configured.*
> For this part, *apt-mark showhold* gave me nothing I don't know what is
> means.
>
> I am looking at the links, but nothing seems to match my kernel version.
>
> Thanks for your assistance.
>
> Le mer. 17 juill. 2024, à 10 h 55, Kienan Stewart <kstewart@efficios.com>
> a écrit :
>
>> Hi François,
>>
>> On 7/16/24 4:34 PM, François Belias wrote:
>> > Hello Kienan,
>> >
>> > When I proceed with the following steps:
>> >
>> > ```
>> > # Warning: this removes all untracked content from the lttng-modules
>> > directory
>> > cd /path/to/lttng-modules-source
>> > git clean -dxf
>> >
>> > make
>> > sudo make modules_install
>> > sudo depmod -a
>> > ```
>> >
>> > I encounter a warning in
>> > `/home/philippe/lttng-modules/src/probes/Kbuild:79` stating "File
>> > ./arch/x86/kvm/lapic.h not found. Probe 'kvm' x86-specific is disabled.
>> > Use full kernel source tree to enable it." Although everything compiles
>> > fine and I'm able to load modules, when I execute `sudo lttng list -k`,
>> > I don't see the tracepoints `kvm_entry` and `kvm_exit`, which I believe
>> > should be available. Could this be due to not using my full kernel tree
>> > during compilation?
>>
>>
>> Exactly. Those probes depend on having the full kernel source as they
>> require private headers which aren't shipped as part of the
>> `linux-headers` package.
>>
>> >
>> > Here are the answers to your questions:
>> >
>> > - *Where did `/usr/src/linux-source-6.5.0` come from?*
>> >    I initially attempted `sudo apt-get update && sudo apt-get install
>> > linux-source`, but it gave me this answer */linux-source is already the
>> > newest version (5.15.0.116.116)/*`, which doesn't match my current
>> > kernel version. So, I downloaded it directly from kernel.org
>> > <http://kernel.org> using this command: `wget
>> > https://cdn.kernel.org/pub/linux/kernel/v6.x/linux-6.5.tar.xz`
>> <https://cdn.kernel.org/pub/linux/kernel/v6.x/linux-6.5.tar.xz>
>> > <https://cdn.kernel.org/pub/linux/kernel/v6.x/linux-6.5.tar.xz`
>> <https://cdn.kernel.org/pub/linux/kernel/v6.x/linux-6.5.tar.xz>>.
>> >
>>
>> Right. Which kernel are you actually running at this time? You can check
>> using: `uname -a`
>>
>> In any case, Linux upstream 6.5 is not the same as Ubuntus's 6.5.
>>
>> > - *Which Linux distribution and version are you using?*
>> >    Running `lsb_release -a` gives me:
>> >    ```
>> >    No LSB modules are available.
>> >    Distributor ID: Ubuntu
>> >    Description:    Ubuntu 22.04.4 LTS
>> >    Release:        22.04
>> >    Codename:       jammy
>> >    ```
>> >
>> > - *Assuming your distribution has kernel packages, which do you have
>> > installed, and do you have the corresponding kernel headers package
>> > installed?*
>> >
>> >    Running `dpkg -l | grep linux-image` shows these installed packages:
>> >    ```
>> >    ii  linux-image-5.15.0-116-generic           5.15.0-116.126
>> >   amd64   Signed kernel image generic
>> >    ii  linux-image-6.5.0-28-generic             6.5.0-28.29~22.04.1
>> >   amd64   Signed kernel image generic
>> >    ii  linux-image-6.5.0-41-generic             6.5.0-41.41~22.04.2
>> >   amd64   Signed kernel image generic
>> >    ii  linux-image-6.5.0-44-generic             6.5.0-44.44~22.04.1
>> >   amd64   Signed kernel image generic
>> >    ii  linux-image-generic                      5.15.0.116.116
>> >   amd64   Generic Linux kernel image
>> >    ii  linux-image-generic-hwe-22.04            6.5.0.44.44~22.04.1
>> >   amd64   Generic Linux kernel image
>> >    ```
>> >
>> >    And `dpkg -l | grep linux-headers` shows these installed header
>> packages:
>> >    ```
>> >    ii  linux-headers-5.15.0-116                 5.15.0-116.126      all
>> >      Header files related to Linux kernel version 5.15.0
>> >    ii  linux-headers-5.15.0-116-generic         5.15.0-116.126
>> >   amd64   Linux kernel headers for version 5.15.0 on 64 bit x86 SMP
>> >    ii  linux-headers-6.5.0-28-generic           6.5.0-28.29~22.04.1
>> >   amd64   Linux kernel headers for version 6.5.0 on 64 bit x86 SMP
>> >    ii  linux-headers-6.5.0-41-generic           6.5.0-41.41~22.04.2
>> >   amd64   Linux kernel headers for version 6.5.0 on 64 bit x86 SMP
>> >    ii  linux-headers-6.5.0-44-generic           6.5.0-44.44~22.04.1
>> >   amd64   Linux kernel headers for version 6.5.0 on 64 bit x86 SMP
>> >    ii  linux-headers-generic                    5.15.0.116.116
>> >   amd64   Generic Linux kernel headers
>> >    ii  linux-headers-generic-hwe-22.04          6.5.0.44.44~22.04.1
>> >   amd64   Generic Linux kernel headers
>> >    ```
>> >
>>
>> Since you've got a few different images and headers installed, it is
>> possible to check that same info the linux-source package: `dpkg -l |
>> grep linux-source`.
>>
>> Maybe one thing tripping this process up is that you don't have an apt
>> source that provides linux-source that matches the kernel you are using.
>> Could you run either `apt-cache policy` or `apt policy` and `apt-mark
>> showhold` in order to know which sources you have configured.
>>
>> There are also alternative sources for the full source tree:
>>
>> - You could download the specific package from packages.ubuntu.com. See
>> https://packages.ubuntu.com and use `apt install
>> ./downloaded_package.deb`
>>
>> - You could clone the kernel source for Ubuntu and checkout the
>> appropriate tag. See
>>
>> https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/jammy/refs/tags
>>
>> thanks,
>> kienan
>>
>

[-- Attachment #1.2: Type: text/html, Size: 10527 bytes --]

[-- Attachment #2: Type: text/plain, Size: 156 bytes --]

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [lttng-dev] loading modules with modprobe

[Kienan Stewart via lttng-dev]

Hi François,

On 7/17/24 11:47 AM, François Belias wrote:
> Hello, Kienan
> 
> 
> *Right. Which kernel are you actually running at this time? You can check
> using: `uname -a`*
> Linux philippe 6.5.0-41-generic #41~22.04.2-Ubuntu SMP PREEMPT_DYNAMIC 
> Mon Jun  3 11:32:55 UTC 2 x86_64 x86_64 x86_64 GNU/Linux
> 
> *Since you've got a few different images and headers installed, it is
> possible to check that same info the linux-source package: `dpkg -l |
> grep linux-source`.*
> ii  linux-source                                   5.15.0.116.116       
>                               all          Linux kernel source with 
> Ubuntu patches
> ii  linux-source-5.15.0                       5.15.0-116.126             
>                         all          Linux kernel source for version 
> 5.15.0 with Ubuntu patches
> ii  linux-source-6.5.0                      6.5.0-44.44~22.04.1         
>                        all          Linux kernel source for version 

Your currently running `6.5.0-41-generic #41~22.04.2-Ubuntu`

You have source installed for `6.5.0-44.44~22.04.1`. From your previous 
e-mail, there is also a linux image installed:

```
linux-image-6.5.0-44-generic 6.5.0-44.44~22.04.1
```

Assuming you are using grub, could you verify that one of the options 
for booting in `/boot/grub/grub.cfg` is linux 6.5.0-44.44. Look for the 
lines similar to the following inside a menu entry block:

(Note my example is 5.15.0-116, but you will be lookng for 6.5.0-44 or 
6.5.0-44.44)

```
linux   /boot/vmlinuz-5.15.0-116-generic root=....
initrd  /boot/initrd.img-5.15.0-116-generic
```

If you have a grub entry for linux 6.5.0-44.44, could you reboot to run 
on the linux 6.5.0-44.44.

After rebooting verify with `uname -a` that your kernel version is 
6.5.0-44.44.

Make sure you have unpacked the linux sources:

```
# Approximating the version numbers, since I don't have 6.5.0 installed
cd /usr/src/linux-source-6.5.0
bunzip2 linux-source-6.5.0.tar.bz2
tar -xf linux-source-6.5.0.tar
```

Then retry the build with 
`KERNELDIR=/usr/src/linux-source-6.5.0/linux-source-6.5.0/`

I haven't personally tried the full steps on an Ubuntu installation so 
you may still need to tweak a couple of things, but I think this should 
get you on the right path.

Really the important thing is to make sure that the kernel source code 
matches the kernel you are running.

> 6.5.0 with Ubuntu patches
> *Could you run either `apt-cache policy` or `apt policy` and `apt-mark
> showhold` in order to know which sources you have configured.*
> For this part, /*apt-mark showhold*/ gave me nothing I don't know what 
> is means.

`apt-mark showhold` will show any packages that are on hold - it is a 
way of marking packages that shouldn't be updated.

As you have the linux-source for 6.5.0-44.44~22.04.1, I don't think it 
will be necessary to have the output of `apt-cache policy`.

thanks,
kienan

> 
> I am looking at the links, but nothing seems to match my kernel version.
> 
> Thanks for your assistance.
> 
> Le mer. 17 juill. 2024, à 11 h 46, François Belias 
> <beliasossim@gmail.com <mailto:beliasossim@gmail.com>> a écrit :
> 
>     Hello, Kienan
> 
> 
>     *Right. Which kernel are you actually running at this time? You can
>     check
>     using: `uname -a`*
>     Linux philippe 6.5.0-41-generic #41~22.04.2-Ubuntu SMP
>     PREEMPT_DYNAMIC Mon Jun  3 11:32:55 UTC 2 x86_64 x86_64 x86_64 GNU/Linux
> 
>     *Since you've got a few different images and headers installed, it is
>     possible to check that same info the linux-source package: `dpkg -l |
>     grep linux-source`.*
>     ii  linux-source                                   5.15.0.116.116  
>                                       all          Linux kernel source
>     with Ubuntu patches
>     ii  linux-source-5.15.0                       5.15.0-116.126        
>                                 all          Linux kernel source for
>     version 5.15.0 with Ubuntu patches
>     ii  linux-source-6.5.0                      6.5.0-44.44~22.04.1    
>                                all          Linux kernel source for
>     version 6.5.0 with Ubuntu patches
>     *Could you run either `apt-cache policy` or `apt policy` and `apt-mark
>     showhold` in order to know which sources you have configured.*
>     For this part, /*apt-mark showhold*/ gave me nothing I don't know
>     what is means.
> 
>     I am looking at the links, but nothing seems to match my kernel
>     version.
> 
>     Thanks for your assistance.
> 
>     Le mer. 17 juill. 2024, à 10 h 55, Kienan Stewart
>     <kstewart@efficios.com <mailto:kstewart@efficios.com>> a écrit :
> 
>         Hi François,
> 
>         On 7/16/24 4:34 PM, François Belias wrote:
>          > Hello Kienan,
>          >
>          > When I proceed with the following steps:
>          >
>          > ```
>          > # Warning: this removes all untracked content from the
>         lttng-modules
>          > directory
>          > cd /path/to/lttng-modules-source
>          > git clean -dxf
>          >
>          > make
>          > sudo make modules_install
>          > sudo depmod -a
>          > ```
>          >
>          > I encounter a warning in
>          > `/home/philippe/lttng-modules/src/probes/Kbuild:79` stating
>         "File
>          > ./arch/x86/kvm/lapic.h not found. Probe 'kvm' x86-specific is
>         disabled.
>          > Use full kernel source tree to enable it." Although
>         everything compiles
>          > fine and I'm able to load modules, when I execute `sudo lttng
>         list -k`,
>          > I don't see the tracepoints `kvm_entry` and `kvm_exit`, which
>         I believe
>          > should be available. Could this be due to not using my full
>         kernel tree
>          > during compilation?
> 
> 
>         Exactly. Those probes depend on having the full kernel source as
>         they
>         require private headers which aren't shipped as part of the
>         `linux-headers` package.
> 
>          >
>          > Here are the answers to your questions:
>          >
>          > - *Where did `/usr/src/linux-source-6.5.0` come from?*
>          >    I initially attempted `sudo apt-get update && sudo apt-get
>         install
>          > linux-source`, but it gave me this answer */linux-source is
>         already the
>          > newest version (5.15.0.116.116)/*`, which doesn't match my
>         current
>          > kernel version. So, I downloaded it directly from kernel.org
>         <http://kernel.org>
>          > <http://kernel.org <http://kernel.org>> using this command:
>         `wget
>          >
>         https://cdn.kernel.org/pub/linux/kernel/v6.x/linux-6.5.tar.xz`
>         <https://cdn.kernel.org/pub/linux/kernel/v6.x/linux-6.5.tar.xz>
>          >
>         <https://cdn.kernel.org/pub/linux/kernel/v6.x/linux-6.5.tar.xz`
>         <https://cdn.kernel.org/pub/linux/kernel/v6.x/linux-6.5.tar.xz>>.
>          >
> 
>         Right. Which kernel are you actually running at this time? You
>         can check
>         using: `uname -a`
> 
>         In any case, Linux upstream 6.5 is not the same as Ubuntus's 6.5.
> 
>          > - *Which Linux distribution and version are you using?*
>          >    Running `lsb_release -a` gives me:
>          >    ```
>          >    No LSB modules are available.
>          >    Distributor ID: Ubuntu
>          >    Description:    Ubuntu 22.04.4 LTS
>          >    Release:        22.04
>          >    Codename:       jammy
>          >    ```
>          >
>          > - *Assuming your distribution has kernel packages, which do
>         you have
>          > installed, and do you have the corresponding kernel headers
>         package
>          > installed?*
>          >
>          >    Running `dpkg -l | grep linux-image` shows these installed
>         packages:
>          >    ```
>          >    ii  linux-image-5.15.0-116-generic           5.15.0-116.126
>          >   amd64   Signed kernel image generic
>          >    ii  linux-image-6.5.0-28-generic            
>         6.5.0-28.29~22.04.1
>          >   amd64   Signed kernel image generic
>          >    ii  linux-image-6.5.0-41-generic            
>         6.5.0-41.41~22.04.2
>          >   amd64   Signed kernel image generic
>          >    ii  linux-image-6.5.0-44-generic            
>         6.5.0-44.44~22.04.1
>          >   amd64   Signed kernel image generic
>          >    ii  linux-image-generic                      5.15.0.116.116
>          >   amd64   Generic Linux kernel image
>          >    ii  linux-image-generic-hwe-22.04          
>           6.5.0.44.44~22.04.1
>          >   amd64   Generic Linux kernel image
>          >    ```
>          >
>          >    And `dpkg -l | grep linux-headers` shows these installed
>         header packages:
>          >    ```
>          >    ii  linux-headers-5.15.0-116                
>         5.15.0-116.126      all
>          >      Header files related to Linux kernel version 5.15.0
>          >    ii  linux-headers-5.15.0-116-generic         5.15.0-116.126
>          >   amd64   Linux kernel headers for version 5.15.0 on 64 bit
>         x86 SMP
>          >    ii  linux-headers-6.5.0-28-generic          
>         6.5.0-28.29~22.04.1
>          >   amd64   Linux kernel headers for version 6.5.0 on 64 bit
>         x86 SMP
>          >    ii  linux-headers-6.5.0-41-generic          
>         6.5.0-41.41~22.04.2
>          >   amd64   Linux kernel headers for version 6.5.0 on 64 bit
>         x86 SMP
>          >    ii  linux-headers-6.5.0-44-generic          
>         6.5.0-44.44~22.04.1
>          >   amd64   Linux kernel headers for version 6.5.0 on 64 bit
>         x86 SMP
>          >    ii  linux-headers-generic                    5.15.0.116.116
>          >   amd64   Generic Linux kernel headers
>          >    ii  linux-headers-generic-hwe-22.04        
>           6.5.0.44.44~22.04.1
>          >   amd64   Generic Linux kernel headers
>          >    ```
>          >
> 
>         Since you've got a few different images and headers installed,
>         it is
>         possible to check that same info the linux-source package: `dpkg
>         -l |
>         grep linux-source`.
> 
>         Maybe one thing tripping this process up is that you don't have
>         an apt
>         source that provides linux-source that matches the kernel you
>         are using.
>         Could you run either `apt-cache policy` or `apt policy` and
>         `apt-mark
>         showhold` in order to know which sources you have configured.
> 
>         There are also alternative sources for the full source tree:
> 
>         - You could download the specific package from
>         packages.ubuntu.com <http://packages.ubuntu.com>. See
>         https://packages.ubuntu.com <https://packages.ubuntu.com> and
>         use `apt install ./downloaded_package.deb`
> 
>         - You could clone the kernel source for Ubuntu and checkout the
>         appropriate tag. See
>         https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/jammy/refs/tags <https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/jammy/refs/tags>
> 
>         thanks,
>         kienan
> 
_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [lttng-dev] loading modules with modprobe

[François Belias via lttng-dev]

[-- Attachment #1.1: Type: text/plain, Size: 13084 bytes --]

Hello Kienan,

Thanks to your advice, I managed to load some modules. However, I
encountered errors like these when running `dmesg | grep lttng`:

```
[  218.043672] module lttng_kretprobes: .gnu.linkonce.this_module section
size must match the kernel's built struct module size at run time
[  650.560883] lttng_kretprobes: module verification failed: signature
and/or required key missing - tainting kernel
[ 1468.329988] module lttng_ring_buffer_client_discard:
.gnu.linkonce.this_module section size must match the kernel's built struct
module size at run time
```

I'm unsure of their meaning. After loading my module, how can I be certain
that my tracepoints are active? I read online that people use `sudo lttng
list -k` for this purpose, but when I tried it, I got the error: "Error:
Unable to list kernel events: Kernel tracer not available". Is there
another way?

Thank you for your assistance.

Le mer. 17 juill. 2024, à 12 h 08, Kienan Stewart <kstewart@efficios.com> a
écrit :

> Hi François,
>
> On 7/17/24 11:47 AM, François Belias wrote:
> > Hello, Kienan
> >
> >
> > *Right. Which kernel are you actually running at this time? You can check
> > using: `uname -a`*
> > Linux philippe 6.5.0-41-generic #41~22.04.2-Ubuntu SMP PREEMPT_DYNAMIC
> > Mon Jun  3 11:32:55 UTC 2 x86_64 x86_64 x86_64 GNU/Linux
> >
> > *Since you've got a few different images and headers installed, it is
> > possible to check that same info the linux-source package: `dpkg -l |
> > grep linux-source`.*
> > ii  linux-source                                   5.15.0.116.116
> >                               all          Linux kernel source with
> > Ubuntu patches
> > ii  linux-source-5.15.0                       5.15.0-116.126
> >                         all          Linux kernel source for version
> > 5.15.0 with Ubuntu patches
> > ii  linux-source-6.5.0                      6.5.0-44.44~22.04.1
> >                        all          Linux kernel source for version
>
> Your currently running `6.5.0-41-generic #41~22.04.2-Ubuntu`
>
> You have source installed for `6.5.0-44.44~22.04.1`. From your previous
> e-mail, there is also a linux image installed:
>
> ```
> linux-image-6.5.0-44-generic 6.5.0-44.44~22.04.1
> ```
>
> Assuming you are using grub, could you verify that one of the options
> for booting in `/boot/grub/grub.cfg` is linux 6.5.0-44.44. Look for the
> lines similar to the following inside a menu entry block:
>
> (Note my example is 5.15.0-116, but you will be lookng for 6.5.0-44 or
> 6.5.0-44.44)
>
> ```
> linux   /boot/vmlinuz-5.15.0-116-generic root=....
> initrd  /boot/initrd.img-5.15.0-116-generic
> ```
>
> If you have a grub entry for linux 6.5.0-44.44, could you reboot to run
> on the linux 6.5.0-44.44.
>
> After rebooting verify with `uname -a` that your kernel version is
> 6.5.0-44.44.
>
> Make sure you have unpacked the linux sources:
>
> ```
> # Approximating the version numbers, since I don't have 6.5.0 installed
> cd /usr/src/linux-source-6.5.0
> bunzip2 linux-source-6.5.0.tar.bz2
> tar -xf linux-source-6.5.0.tar
> ```
>
> Then retry the build with
> `KERNELDIR=/usr/src/linux-source-6.5.0/linux-source-6.5.0/`
>
> I haven't personally tried the full steps on an Ubuntu installation so
> you may still need to tweak a couple of things, but I think this should
> get you on the right path.
>
> Really the important thing is to make sure that the kernel source code
> matches the kernel you are running.
>
> > 6.5.0 with Ubuntu patches
> > *Could you run either `apt-cache policy` or `apt policy` and `apt-mark
> > showhold` in order to know which sources you have configured.*
> > For this part, /*apt-mark showhold*/ gave me nothing I don't know what
> > is means.
>
> `apt-mark showhold` will show any packages that are on hold - it is a
> way of marking packages that shouldn't be updated.
>
> As you have the linux-source for 6.5.0-44.44~22.04.1, I don't think it
> will be necessary to have the output of `apt-cache policy`.
>
> thanks,
> kienan
>
> >
> > I am looking at the links, but nothing seems to match my kernel version.
> >
> > Thanks for your assistance.
> >
> > Le mer. 17 juill. 2024, à 11 h 46, François Belias
> > <beliasossim@gmail.com <mailto:beliasossim@gmail.com>> a écrit :
> >
> >     Hello, Kienan
> >
> >
> >     *Right. Which kernel are you actually running at this time? You can
> >     check
> >     using: `uname -a`*
> >     Linux philippe 6.5.0-41-generic #41~22.04.2-Ubuntu SMP
> >     PREEMPT_DYNAMIC Mon Jun  3 11:32:55 UTC 2 x86_64 x86_64 x86_64
> GNU/Linux
> >
> >     *Since you've got a few different images and headers installed, it is
> >     possible to check that same info the linux-source package: `dpkg -l |
> >     grep linux-source`.*
> >     ii  linux-source                                   5.15.0.116.116
> >                                       all          Linux kernel source
> >     with Ubuntu patches
> >     ii  linux-source-5.15.0                       5.15.0-116.126
> >                                 all          Linux kernel source for
> >     version 5.15.0 with Ubuntu patches
> >     ii  linux-source-6.5.0                      6.5.0-44.44~22.04.1
> >                                all          Linux kernel source for
> >     version 6.5.0 with Ubuntu patches
> >     *Could you run either `apt-cache policy` or `apt policy` and
> `apt-mark
> >     showhold` in order to know which sources you have configured.*
> >     For this part, /*apt-mark showhold*/ gave me nothing I don't know
> >     what is means.
> >
> >     I am looking at the links, but nothing seems to match my kernel
> >     version.
> >
> >     Thanks for your assistance.
> >
> >     Le mer. 17 juill. 2024, à 10 h 55, Kienan Stewart
> >     <kstewart@efficios.com <mailto:kstewart@efficios.com>> a écrit :
> >
> >         Hi François,
> >
> >         On 7/16/24 4:34 PM, François Belias wrote:
> >          > Hello Kienan,
> >          >
> >          > When I proceed with the following steps:
> >          >
> >          > ```
> >          > # Warning: this removes all untracked content from the
> >         lttng-modules
> >          > directory
> >          > cd /path/to/lttng-modules-source
> >          > git clean -dxf
> >          >
> >          > make
> >          > sudo make modules_install
> >          > sudo depmod -a
> >          > ```
> >          >
> >          > I encounter a warning in
> >          > `/home/philippe/lttng-modules/src/probes/Kbuild:79` stating
> >         "File
> >          > ./arch/x86/kvm/lapic.h not found. Probe 'kvm' x86-specific is
> >         disabled.
> >          > Use full kernel source tree to enable it." Although
> >         everything compiles
> >          > fine and I'm able to load modules, when I execute `sudo lttng
> >         list -k`,
> >          > I don't see the tracepoints `kvm_entry` and `kvm_exit`, which
> >         I believe
> >          > should be available. Could this be due to not using my full
> >         kernel tree
> >          > during compilation?
> >
> >
> >         Exactly. Those probes depend on having the full kernel source as
> >         they
> >         require private headers which aren't shipped as part of the
> >         `linux-headers` package.
> >
> >          >
> >          > Here are the answers to your questions:
> >          >
> >          > - *Where did `/usr/src/linux-source-6.5.0` come from?*
> >          >    I initially attempted `sudo apt-get update && sudo apt-get
> >         install
> >          > linux-source`, but it gave me this answer */linux-source is
> >         already the
> >          > newest version (5.15.0.116.116)/*`, which doesn't match my
> >         current
> >          > kernel version. So, I downloaded it directly from kernel.org
> >         <http://kernel.org>
> >          > <http://kernel.org <http://kernel.org>> using this command:
> >         `wget
> >          >
> >         https://cdn.kernel.org/pub/linux/kernel/v6.x/linux-6.5.tar.xz`
> <https://cdn.kernel.org/pub/linux/kernel/v6.x/linux-6.5.tar.xz>
> >         <https://cdn.kernel.org/pub/linux/kernel/v6.x/linux-6.5.tar.xz>
> >          >
> >         <https://cdn.kernel.org/pub/linux/kernel/v6.x/linux-6.5.tar.xz`
> <https://cdn.kernel.org/pub/linux/kernel/v6.x/linux-6.5.tar.xz>
> >         <https://cdn.kernel.org/pub/linux/kernel/v6.x/linux-6.5.tar.xz
> >>.
> >          >
> >
> >         Right. Which kernel are you actually running at this time? You
> >         can check
> >         using: `uname -a`
> >
> >         In any case, Linux upstream 6.5 is not the same as Ubuntus's 6.5.
> >
> >          > - *Which Linux distribution and version are you using?*
> >          >    Running `lsb_release -a` gives me:
> >          >    ```
> >          >    No LSB modules are available.
> >          >    Distributor ID: Ubuntu
> >          >    Description:    Ubuntu 22.04.4 LTS
> >          >    Release:        22.04
> >          >    Codename:       jammy
> >          >    ```
> >          >
> >          > - *Assuming your distribution has kernel packages, which do
> >         you have
> >          > installed, and do you have the corresponding kernel headers
> >         package
> >          > installed?*
> >          >
> >          >    Running `dpkg -l | grep linux-image` shows these installed
> >         packages:
> >          >    ```
> >          >    ii  linux-image-5.15.0-116-generic           5.15.0-116.126
> >          >   amd64   Signed kernel image generic
> >          >    ii  linux-image-6.5.0-28-generic
> >         6.5.0-28.29~22.04.1
> >          >   amd64   Signed kernel image generic
> >          >    ii  linux-image-6.5.0-41-generic
> >         6.5.0-41.41~22.04.2
> >          >   amd64   Signed kernel image generic
> >          >    ii  linux-image-6.5.0-44-generic
> >         6.5.0-44.44~22.04.1
> >          >   amd64   Signed kernel image generic
> >          >    ii  linux-image-generic                      5.15.0.116.116
> >          >   amd64   Generic Linux kernel image
> >          >    ii  linux-image-generic-hwe-22.04
> >           6.5.0.44.44~22.04.1
> >          >   amd64   Generic Linux kernel image
> >          >    ```
> >          >
> >          >    And `dpkg -l | grep linux-headers` shows these installed
> >         header packages:
> >          >    ```
> >          >    ii  linux-headers-5.15.0-116
> >         5.15.0-116.126      all
> >          >      Header files related to Linux kernel version 5.15.0
> >          >    ii  linux-headers-5.15.0-116-generic         5.15.0-116.126
> >          >   amd64   Linux kernel headers for version 5.15.0 on 64 bit
> >         x86 SMP
> >          >    ii  linux-headers-6.5.0-28-generic
> >         6.5.0-28.29~22.04.1
> >          >   amd64   Linux kernel headers for version 6.5.0 on 64 bit
> >         x86 SMP
> >          >    ii  linux-headers-6.5.0-41-generic
> >         6.5.0-41.41~22.04.2
> >          >   amd64   Linux kernel headers for version 6.5.0 on 64 bit
> >         x86 SMP
> >          >    ii  linux-headers-6.5.0-44-generic
> >         6.5.0-44.44~22.04.1
> >          >   amd64   Linux kernel headers for version 6.5.0 on 64 bit
> >         x86 SMP
> >          >    ii  linux-headers-generic                    5.15.0.116.116
> >          >   amd64   Generic Linux kernel headers
> >          >    ii  linux-headers-generic-hwe-22.04
> >           6.5.0.44.44~22.04.1
> >          >   amd64   Generic Linux kernel headers
> >          >    ```
> >          >
> >
> >         Since you've got a few different images and headers installed,
> >         it is
> >         possible to check that same info the linux-source package: `dpkg
> >         -l |
> >         grep linux-source`.
> >
> >         Maybe one thing tripping this process up is that you don't have
> >         an apt
> >         source that provides linux-source that matches the kernel you
> >         are using.
> >         Could you run either `apt-cache policy` or `apt policy` and
> >         `apt-mark
> >         showhold` in order to know which sources you have configured.
> >
> >         There are also alternative sources for the full source tree:
> >
> >         - You could download the specific package from
> >         packages.ubuntu.com <http://packages.ubuntu.com>. See
> >         https://packages.ubuntu.com <https://packages.ubuntu.com> and
> >         use `apt install ./downloaded_package.deb`
> >
> >         - You could clone the kernel source for Ubuntu and checkout the
> >         appropriate tag. See
> >
> https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/jammy/refs/tags
> <
> https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/jammy/refs/tags
> >
> >
> >         thanks,
> >         kienan
> >
>

[-- Attachment #1.2: Type: text/html, Size: 17890 bytes --]

[-- Attachment #2: Type: text/plain, Size: 156 bytes --]

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [lttng-dev] loading modules with modprobe

[Kienan Stewart via lttng-dev]

Hi François,

The dmesg excerpts are spaced by a number of minutes each, are they from 
different attempts to build and load the modules?

On 7/17/24 8:44 PM, François Belias wrote:
> Hello Kienan,
> 
> Thanks to your advice, I managed to load some modules. However, I 
> encountered errors like these when running `dmesg | grep lttng`:
> 
> ```
> [  218.043672] module lttng_kretprobes: .gnu.linkonce.this_module 
> section size must match the kernel's built struct module size at run time

This sounds like the version of the module is built against the wrong 
kernel headers or source.

> [  650.560883] lttng_kretprobes: module verification failed: signature 
> and/or required key missing - tainting kernel

Is your system running with secure boot enabled? If so, you either need 
to disable secure boot or sign the built modules with a MOK key that is 
correctly enrolled.

See the following documentation:

- https://lttng.org/docs/v2.13/#doc-linux-kernel-sig
- https://www.kernel.org/doc/html/latest/admin-guide/module-signing.html
- https://github.com/dell/dkms
- 
https://gitlab.com/linux-kernel/stable/-/blob/master/kernel/module/main.c#L2064

Whatever you did for this test is pretty closed to working. The previous 
errors with the section size check happen earlier in the module load 
process. This check seems to be one of the last, c.f. 
https://elixir.bootlin.com/linux/v6.10/source/kernel/module/main.c#L2908

> [ 1468.329988] module lttng_ring_buffer_client_discard: 
> .gnu.linkonce.this_module section size must match the kernel's built 
> struct module size at run time
> ```
> 
> I'm unsure of their meaning. After loading my module, how can I be 
> certain that my tracepoints are active? I read online that people use 
> `sudo lttng list -k` for this purpose, but when I tried it, I got the 
> error: "Error: Unable to list kernel events: Kernel tracer not 
> available". Is there another way?

If you're still getting "kernel tracer not available" then either 
required kernel modules haven't been loaded or you started lttng-session 
with `--no-kernel` option. `lttng-sessiond -vvv`, dmesg, and/or 
journalctl should have information on which modules failed to load.

thanks,
kienan

> 
> Thank you for your assistance.
> 
> Le mer. 17 juill. 2024, à 12 h 08, Kienan Stewart <kstewart@efficios.com 
> <mailto:kstewart@efficios.com>> a écrit :
> 
>     Hi François,
> 
>     On 7/17/24 11:47 AM, François Belias wrote:
>      > Hello, Kienan
>      >
>      >
>      > *Right. Which kernel are you actually running at this time? You
>     can check
>      > using: `uname -a`*
>      > Linux philippe 6.5.0-41-generic #41~22.04.2-Ubuntu SMP
>     PREEMPT_DYNAMIC
>      > Mon Jun  3 11:32:55 UTC 2 x86_64 x86_64 x86_64 GNU/Linux
>      >
>      > *Since you've got a few different images and headers installed, it is
>      > possible to check that same info the linux-source package: `dpkg -l |
>      > grep linux-source`.*
>      > ii  linux-source                                   5.15.0.116.116
>      >                               all          Linux kernel source with
>      > Ubuntu patches
>      > ii  linux-source-5.15.0                       5.15.0-116.126
>      >                         all          Linux kernel source for version
>      > 5.15.0 with Ubuntu patches
>      > ii  linux-source-6.5.0                      6.5.0-44.44~22.04.1
>      >                        all          Linux kernel source for version
> 
>     Your currently running `6.5.0-41-generic #41~22.04.2-Ubuntu`
> 
>     You have source installed for `6.5.0-44.44~22.04.1`. From your previous
>     e-mail, there is also a linux image installed:
> 
>     ```
>     linux-image-6.5.0-44-generic 6.5.0-44.44~22.04.1
>     ```
> 
>     Assuming you are using grub, could you verify that one of the options
>     for booting in `/boot/grub/grub.cfg` is linux 6.5.0-44.44. Look for the
>     lines similar to the following inside a menu entry block:
> 
>     (Note my example is 5.15.0-116, but you will be lookng for 6.5.0-44 or
>     6.5.0-44.44)
> 
>     ```
>     linux   /boot/vmlinuz-5.15.0-116-generic root=....
>     initrd  /boot/initrd.img-5.15.0-116-generic
>     ```
> 
>     If you have a grub entry for linux 6.5.0-44.44, could you reboot to run
>     on the linux 6.5.0-44.44.
> 
>     After rebooting verify with `uname -a` that your kernel version is
>     6.5.0-44.44.
> 
>     Make sure you have unpacked the linux sources:
> 
>     ```
>     # Approximating the version numbers, since I don't have 6.5.0 installed
>     cd /usr/src/linux-source-6.5.0
>     bunzip2 linux-source-6.5.0.tar.bz2
>     tar -xf linux-source-6.5.0.tar
>     ```
> 
>     Then retry the build with
>     `KERNELDIR=/usr/src/linux-source-6.5.0/linux-source-6.5.0/`
> 
>     I haven't personally tried the full steps on an Ubuntu installation so
>     you may still need to tweak a couple of things, but I think this should
>     get you on the right path.
> 
>     Really the important thing is to make sure that the kernel source code
>     matches the kernel you are running.
> 
>      > 6.5.0 with Ubuntu patches
>      > *Could you run either `apt-cache policy` or `apt policy` and
>     `apt-mark
>      > showhold` in order to know which sources you have configured.*
>      > For this part, /*apt-mark showhold*/ gave me nothing I don't know
>     what
>      > is means.
> 
>     `apt-mark showhold` will show any packages that are on hold - it is a
>     way of marking packages that shouldn't be updated.
> 
>     As you have the linux-source for 6.5.0-44.44~22.04.1, I don't think it
>     will be necessary to have the output of `apt-cache policy`.
> 
>     thanks,
>     kienan
> 
>      >
>      > I am looking at the links, but nothing seems to match my kernel
>     version.
>      >
>      > Thanks for your assistance.
>      >
>      > Le mer. 17 juill. 2024, à 11 h 46, François Belias
>      > <beliasossim@gmail.com <mailto:beliasossim@gmail.com>
>     <mailto:beliasossim@gmail.com <mailto:beliasossim@gmail.com>>> a écrit :
>      >
>      >     Hello, Kienan
>      >
>      >
>      >     *Right. Which kernel are you actually running at this time?
>     You can
>      >     check
>      >     using: `uname -a`*
>      >     Linux philippe 6.5.0-41-generic #41~22.04.2-Ubuntu SMP
>      >     PREEMPT_DYNAMIC Mon Jun  3 11:32:55 UTC 2 x86_64 x86_64
>     x86_64 GNU/Linux
>      >
>      >     *Since you've got a few different images and headers
>     installed, it is
>      >     possible to check that same info the linux-source package:
>     `dpkg -l |
>      >     grep linux-source`.*
>      >     ii  linux-source                                 
>       5.15.0.116.116
>      >                                       all          Linux kernel
>     source
>      >     with Ubuntu patches
>      >     ii  linux-source-5.15.0                       5.15.0-116.126
>      >                                 all          Linux kernel source for
>      >     version 5.15.0 with Ubuntu patches
>      >     ii  linux-source-6.5.0                      6.5.0-44.44~22.04.1
>      >                                all          Linux kernel source for
>      >     version 6.5.0 with Ubuntu patches
>      >     *Could you run either `apt-cache policy` or `apt policy` and
>     `apt-mark
>      >     showhold` in order to know which sources you have configured.*
>      >     For this part, /*apt-mark showhold*/ gave me nothing I don't know
>      >     what is means.
>      >
>      >     I am looking at the links, but nothing seems to match my kernel
>      >     version.
>      >
>      >     Thanks for your assistance.
>      >
>      >     Le mer. 17 juill. 2024, à 10 h 55, Kienan Stewart
>      >     <kstewart@efficios.com <mailto:kstewart@efficios.com>
>     <mailto:kstewart@efficios.com <mailto:kstewart@efficios.com>>> a écrit :
>      >
>      >         Hi François,
>      >
>      >         On 7/16/24 4:34 PM, François Belias wrote:
>      >          > Hello Kienan,
>      >          >
>      >          > When I proceed with the following steps:
>      >          >
>      >          > ```
>      >          > # Warning: this removes all untracked content from the
>      >         lttng-modules
>      >          > directory
>      >          > cd /path/to/lttng-modules-source
>      >          > git clean -dxf
>      >          >
>      >          > make
>      >          > sudo make modules_install
>      >          > sudo depmod -a
>      >          > ```
>      >          >
>      >          > I encounter a warning in
>      >          > `/home/philippe/lttng-modules/src/probes/Kbuild:79`
>     stating
>      >         "File
>      >          > ./arch/x86/kvm/lapic.h not found. Probe 'kvm'
>     x86-specific is
>      >         disabled.
>      >          > Use full kernel source tree to enable it." Although
>      >         everything compiles
>      >          > fine and I'm able to load modules, when I execute
>     `sudo lttng
>      >         list -k`,
>      >          > I don't see the tracepoints `kvm_entry` and
>     `kvm_exit`, which
>      >         I believe
>      >          > should be available. Could this be due to not using my
>     full
>      >         kernel tree
>      >          > during compilation?
>      >
>      >
>      >         Exactly. Those probes depend on having the full kernel
>     source as
>      >         they
>      >         require private headers which aren't shipped as part of the
>      >         `linux-headers` package.
>      >
>      >          >
>      >          > Here are the answers to your questions:
>      >          >
>      >          > - *Where did `/usr/src/linux-source-6.5.0` come from?*
>      >          >    I initially attempted `sudo apt-get update && sudo
>     apt-get
>      >         install
>      >          > linux-source`, but it gave me this answer
>     */linux-source is
>      >         already the
>      >          > newest version (5.15.0.116.116)/*`, which doesn't match my
>      >         current
>      >          > kernel version. So, I downloaded it directly from
>     kernel.org <http://kernel.org>
>      >         <http://kernel.org <http://kernel.org>>
>      >          > <http://kernel.org <http://kernel.org>
>     <http://kernel.org <http://kernel.org>>> using this command:
>      >         `wget
>      >          >
>      > https://cdn.kernel.org/pub/linux/kernel/v6.x/linux-6.5.tar.xz`
>     <https://cdn.kernel.org/pub/linux/kernel/v6.x/linux-6.5.tar.xz>
>      >       
>       <https://cdn.kernel.org/pub/linux/kernel/v6.x/linux-6.5.tar.xz
>     <https://cdn.kernel.org/pub/linux/kernel/v6.x/linux-6.5.tar.xz>>
>      >          >
>      >       
>       <https://cdn.kernel.org/pub/linux/kernel/v6.x/linux-6.5.tar.xz`
>     <https://cdn.kernel.org/pub/linux/kernel/v6.x/linux-6.5.tar.xz>
>      >       
>       <https://cdn.kernel.org/pub/linux/kernel/v6.x/linux-6.5.tar.xz
>     <https://cdn.kernel.org/pub/linux/kernel/v6.x/linux-6.5.tar.xz>>>.
>      >          >
>      >
>      >         Right. Which kernel are you actually running at this
>     time? You
>      >         can check
>      >         using: `uname -a`
>      >
>      >         In any case, Linux upstream 6.5 is not the same as
>     Ubuntus's 6.5.
>      >
>      >          > - *Which Linux distribution and version are you using?*
>      >          >    Running `lsb_release -a` gives me:
>      >          >    ```
>      >          >    No LSB modules are available.
>      >          >    Distributor ID: Ubuntu
>      >          >    Description:    Ubuntu 22.04.4 LTS
>      >          >    Release:        22.04
>      >          >    Codename:       jammy
>      >          >    ```
>      >          >
>      >          > - *Assuming your distribution has kernel packages,
>     which do
>      >         you have
>      >          > installed, and do you have the corresponding kernel
>     headers
>      >         package
>      >          > installed?*
>      >          >
>      >          >    Running `dpkg -l | grep linux-image` shows these
>     installed
>      >         packages:
>      >          >    ```
>      >          >    ii  linux-image-5.15.0-116-generic          
>     5.15.0-116.126
>      >          >   amd64   Signed kernel image generic
>      >          >    ii  linux-image-6.5.0-28-generic
>      >         6.5.0-28.29~22.04.1
>      >          >   amd64   Signed kernel image generic
>      >          >    ii  linux-image-6.5.0-41-generic
>      >         6.5.0-41.41~22.04.2
>      >          >   amd64   Signed kernel image generic
>      >          >    ii  linux-image-6.5.0-44-generic
>      >         6.5.0-44.44~22.04.1
>      >          >   amd64   Signed kernel image generic
>      >          >    ii  linux-image-generic                    
>       5.15.0.116.116
>      >          >   amd64   Generic Linux kernel image
>      >          >    ii  linux-image-generic-hwe-22.04
>      >           6.5.0.44.44~22.04.1
>      >          >   amd64   Generic Linux kernel image
>      >          >    ```
>      >          >
>      >          >    And `dpkg -l | grep linux-headers` shows these
>     installed
>      >         header packages:
>      >          >    ```
>      >          >    ii  linux-headers-5.15.0-116
>      >         5.15.0-116.126      all
>      >          >      Header files related to Linux kernel version 5.15.0
>      >          >    ii  linux-headers-5.15.0-116-generic        
>     5.15.0-116.126
>      >          >   amd64   Linux kernel headers for version 5.15.0 on
>     64 bit
>      >         x86 SMP
>      >          >    ii  linux-headers-6.5.0-28-generic
>      >         6.5.0-28.29~22.04.1
>      >          >   amd64   Linux kernel headers for version 6.5.0 on 64 bit
>      >         x86 SMP
>      >          >    ii  linux-headers-6.5.0-41-generic
>      >         6.5.0-41.41~22.04.2
>      >          >   amd64   Linux kernel headers for version 6.5.0 on 64 bit
>      >         x86 SMP
>      >          >    ii  linux-headers-6.5.0-44-generic
>      >         6.5.0-44.44~22.04.1
>      >          >   amd64   Linux kernel headers for version 6.5.0 on 64 bit
>      >         x86 SMP
>      >          >    ii  linux-headers-generic                  
>       5.15.0.116.116
>      >          >   amd64   Generic Linux kernel headers
>      >          >    ii  linux-headers-generic-hwe-22.04
>      >           6.5.0.44.44~22.04.1
>      >          >   amd64   Generic Linux kernel headers
>      >          >    ```
>      >          >
>      >
>      >         Since you've got a few different images and headers
>     installed,
>      >         it is
>      >         possible to check that same info the linux-source
>     package: `dpkg
>      >         -l |
>      >         grep linux-source`.
>      >
>      >         Maybe one thing tripping this process up is that you
>     don't have
>      >         an apt
>      >         source that provides linux-source that matches the kernel you
>      >         are using.
>      >         Could you run either `apt-cache policy` or `apt policy` and
>      >         `apt-mark
>      >         showhold` in order to know which sources you have configured.
>      >
>      >         There are also alternative sources for the full source tree:
>      >
>      >         - You could download the specific package from
>      > packages.ubuntu.com <http://packages.ubuntu.com>
>     <http://packages.ubuntu.com <http://packages.ubuntu.com>>. See
>      > https://packages.ubuntu.com <https://packages.ubuntu.com>
>     <https://packages.ubuntu.com <https://packages.ubuntu.com>> and
>      >         use `apt install ./downloaded_package.deb`
>      >
>      >         - You could clone the kernel source for Ubuntu and
>     checkout the
>      >         appropriate tag. See
>      >
>     https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/jammy/refs/tags <https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/jammy/refs/tags> <https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/jammy/refs/tags <https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/jammy/refs/tags>>
>      >
>      >         thanks,
>      >         kienan
>      >
> 
_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [lttng-dev] loading modules with modprobe

[Kienan Stewart via lttng-dev]

A couple of addendums inline below

On 7/18/24 10:47 AM, Kienan Stewart via lttng-dev wrote:
> Hi François,
> 
> The dmesg excerpts are spaced by a number of minutes each, are they from 
> different attempts to build and load the modules?
> 
> On 7/17/24 8:44 PM, François Belias wrote:
>> Hello Kienan,
>>
>> Thanks to your advice, I managed to load some modules. However, I 
>> encountered errors like these when running `dmesg | grep lttng`:
>>
>> ```
>> [  218.043672] module lttng_kretprobes: .gnu.linkonce.this_module 
>> section size must match the kernel's built struct module size at run time
> 
> This sounds like the version of the module is built against the wrong 
> kernel headers or source.

Also, are you using lttng-modules master or did you swap back to 
stable-2.13?

If you're still on master and have the most recent changes 
lttng_kretprobes isn't required anymore. You will also need lttng-tools 
from master with the corresponding changes so that it doesn't try to 
load lttng_kretprobes.

> 
>> [  650.560883] lttng_kretprobes: module verification failed: signature 
>> and/or required key missing - tainting kernel
> 
> Is your system running with secure boot enabled? If so, you either need 
> to disable secure boot or sign the built modules with a MOK key that is 
> correctly enrolled.
> 
> See the following documentation:
> 
> - https://lttng.org/docs/v2.13/#doc-linux-kernel-sig
> - https://www.kernel.org/doc/html/latest/admin-guide/module-signing.html
> - https://github.com/dell/dkms
> - 
> https://gitlab.com/linux-kernel/stable/-/blob/master/kernel/module/main.c#L2064
> 
> Whatever you did for this test is pretty closed to working. The previous 
> errors with the section size check happen earlier in the module load 
> process. This check seems to be one of the last, c.f. 
> https://elixir.bootlin.com/linux/v6.10/source/kernel/module/main.c#L2908
> 
>> [ 1468.329988] module lttng_ring_buffer_client_discard: 
>> .gnu.linkonce.this_module section size must match the kernel's built 
>> struct module size at run time
>> ```
>>
>> I'm unsure of their meaning. After loading my module, how can I be 
>> certain that my tracepoints are active? I read online that people use 
>> `sudo lttng list -k` for this purpose, but when I tried it, I got the 
>> error: "Error: Unable to list kernel events: Kernel tracer not 
>> available". Is there another way?
> 
> If you're still getting "kernel tracer not available" then either 
> required kernel modules haven't been loaded or you started lttng-session 
> with `--no-kernel` option. `lttng-sessiond -vvv`, dmesg, and/or 
> journalctl should have information on which modules failed to load.
> 

I know you're doing `sudo lttng list -k`, but the basic checks still apply:

- is the lttng-sessiond running as the root user
- (when running without sudo) does the tracing group exist, and is your 
user a member of that group

> kienan
> 
>>
>> Thank you for your assistance.
>>
>> Le mer. 17 juill. 2024, à 12 h 08, Kienan Stewart 
>> <kstewart@efficios.com <mailto:kstewart@efficios.com>> a écrit :
>>
>>     Hi François,
>>
>>     On 7/17/24 11:47 AM, François Belias wrote:
>>      > Hello, Kienan
>>      >
>>      >
>>      > *Right. Which kernel are you actually running at this time? You
>>     can check
>>      > using: `uname -a`*
>>      > Linux philippe 6.5.0-41-generic #41~22.04.2-Ubuntu SMP
>>     PREEMPT_DYNAMIC
>>      > Mon Jun  3 11:32:55 UTC 2 x86_64 x86_64 x86_64 GNU/Linux
>>      >
>>      > *Since you've got a few different images and headers installed, 
>> it is
>>      > possible to check that same info the linux-source package: 
>> `dpkg -l |
>>      > grep linux-source`.*
>>      > ii  linux-source                                   5.15.0.116.116
>>      >                               all          Linux kernel source 
>> with
>>      > Ubuntu patches
>>      > ii  linux-source-5.15.0                       5.15.0-116.126
>>      >                         all          Linux kernel source for 
>> version
>>      > 5.15.0 with Ubuntu patches
>>      > ii  linux-source-6.5.0                      6.5.0-44.44~22.04.1
>>      >                        all          Linux kernel source for 
>> version
>>
>>     Your currently running `6.5.0-41-generic #41~22.04.2-Ubuntu`
>>
>>     You have source installed for `6.5.0-44.44~22.04.1`. From your 
>> previous
>>     e-mail, there is also a linux image installed:
>>
>>     ```
>>     linux-image-6.5.0-44-generic 6.5.0-44.44~22.04.1
>>     ```
>>
>>     Assuming you are using grub, could you verify that one of the options
>>     for booting in `/boot/grub/grub.cfg` is linux 6.5.0-44.44. Look 
>> for the
>>     lines similar to the following inside a menu entry block:
>>
>>     (Note my example is 5.15.0-116, but you will be lookng for 
>> 6.5.0-44 or
>>     6.5.0-44.44)
>>
>>     ```
>>     linux   /boot/vmlinuz-5.15.0-116-generic root=....
>>     initrd  /boot/initrd.img-5.15.0-116-generic
>>     ```
>>
>>     If you have a grub entry for linux 6.5.0-44.44, could you reboot 
>> to run
>>     on the linux 6.5.0-44.44.
>>
>>     After rebooting verify with `uname -a` that your kernel version is
>>     6.5.0-44.44.
>>
>>     Make sure you have unpacked the linux sources:
>>
>>     ```
>>     # Approximating the version numbers, since I don't have 6.5.0 
>> installed
>>     cd /usr/src/linux-source-6.5.0
>>     bunzip2 linux-source-6.5.0.tar.bz2
>>     tar -xf linux-source-6.5.0.tar
>>     ```
>>
>>     Then retry the build with
>>     `KERNELDIR=/usr/src/linux-source-6.5.0/linux-source-6.5.0/`
>>
>>     I haven't personally tried the full steps on an Ubuntu 
>> installation so
>>     you may still need to tweak a couple of things, but I think this 
>> should
>>     get you on the right path.
>>
>>     Really the important thing is to make sure that the kernel source 
>> code
>>     matches the kernel you are running.
>>
>>      > 6.5.0 with Ubuntu patches
>>      > *Could you run either `apt-cache policy` or `apt policy` and
>>     `apt-mark
>>      > showhold` in order to know which sources you have configured.*
>>      > For this part, /*apt-mark showhold*/ gave me nothing I don't know
>>     what
>>      > is means.
>>
>>     `apt-mark showhold` will show any packages that are on hold - it is a
>>     way of marking packages that shouldn't be updated.
>>
>>     As you have the linux-source for 6.5.0-44.44~22.04.1, I don't 
>> think it
>>     will be necessary to have the output of `apt-cache policy`.
>>
>>     thanks,
>>     kienan
>>
>>      >
>>      > I am looking at the links, but nothing seems to match my kernel
>>     version.
>>      >
>>      > Thanks for your assistance.
>>      >
>>      > Le mer. 17 juill. 2024, à 11 h 46, François Belias
>>      > <beliasossim@gmail.com <mailto:beliasossim@gmail.com>
>>     <mailto:beliasossim@gmail.com <mailto:beliasossim@gmail.com>>> a 
>> écrit :
>>      >
>>      >     Hello, Kienan
>>      >
>>      >
>>      >     *Right. Which kernel are you actually running at this time?
>>     You can
>>      >     check
>>      >     using: `uname -a`*
>>      >     Linux philippe 6.5.0-41-generic #41~22.04.2-Ubuntu SMP
>>      >     PREEMPT_DYNAMIC Mon Jun  3 11:32:55 UTC 2 x86_64 x86_64
>>     x86_64 GNU/Linux
>>      >
>>      >     *Since you've got a few different images and headers
>>     installed, it is
>>      >     possible to check that same info the linux-source package:
>>     `dpkg -l |
>>      >     grep linux-source`.*
>>      >     ii  linux-source       5.15.0.116.116
>>      >                                       all          Linux kernel
>>     source
>>      >     with Ubuntu patches
>>      >     ii  linux-source-5.15.0                       5.15.0-116.126
>>      >                                 all          Linux kernel 
>> source for
>>      >     version 5.15.0 with Ubuntu patches
>>      >     ii  linux-source-6.5.0                      
>> 6.5.0-44.44~22.04.1
>>      >                                all          Linux kernel source 
>> for
>>      >     version 6.5.0 with Ubuntu patches
>>      >     *Could you run either `apt-cache policy` or `apt policy` and
>>     `apt-mark
>>      >     showhold` in order to know which sources you have configured.*
>>      >     For this part, /*apt-mark showhold*/ gave me nothing I 
>> don't know
>>      >     what is means.
>>      >
>>      >     I am looking at the links, but nothing seems to match my 
>> kernel
>>      >     version.
>>      >
>>      >     Thanks for your assistance.
>>      >
>>      >     Le mer. 17 juill. 2024, à 10 h 55, Kienan Stewart
>>      >     <kstewart@efficios.com <mailto:kstewart@efficios.com>
>>     <mailto:kstewart@efficios.com <mailto:kstewart@efficios.com>>> a 
>> écrit :
>>      >
>>      >         Hi François,
>>      >
>>      >         On 7/16/24 4:34 PM, François Belias wrote:
>>      >          > Hello Kienan,
>>      >          >
>>      >          > When I proceed with the following steps:
>>      >          >
>>      >          > ```
>>      >          > # Warning: this removes all untracked content from the
>>      >         lttng-modules
>>      >          > directory
>>      >          > cd /path/to/lttng-modules-source
>>      >          > git clean -dxf
>>      >          >
>>      >          > make
>>      >          > sudo make modules_install
>>      >          > sudo depmod -a
>>      >          > ```
>>      >          >
>>      >          > I encounter a warning in
>>      >          > `/home/philippe/lttng-modules/src/probes/Kbuild:79`
>>     stating
>>      >         "File
>>      >          > ./arch/x86/kvm/lapic.h not found. Probe 'kvm'
>>     x86-specific is
>>      >         disabled.
>>      >          > Use full kernel source tree to enable it." Although
>>      >         everything compiles
>>      >          > fine and I'm able to load modules, when I execute
>>     `sudo lttng
>>      >         list -k`,
>>      >          > I don't see the tracepoints `kvm_entry` and
>>     `kvm_exit`, which
>>      >         I believe
>>      >          > should be available. Could this be due to not using my
>>     full
>>      >         kernel tree
>>      >          > during compilation?
>>      >
>>      >
>>      >         Exactly. Those probes depend on having the full kernel
>>     source as
>>      >         they
>>      >         require private headers which aren't shipped as part of 
>> the
>>      >         `linux-headers` package.
>>      >
>>      >          >
>>      >          > Here are the answers to your questions:
>>      >          >
>>      >          > - *Where did `/usr/src/linux-source-6.5.0` come from?*
>>      >          >    I initially attempted `sudo apt-get update && sudo
>>     apt-get
>>      >         install
>>      >          > linux-source`, but it gave me this answer
>>     */linux-source is
>>      >         already the
>>      >          > newest version (5.15.0.116.116)/*`, which doesn't 
>> match my
>>      >         current
>>      >          > kernel version. So, I downloaded it directly from
>>     kernel.org <http://kernel.org>
>>      >         <http://kernel.org <http://kernel.org>>
>>      >          > <http://kernel.org <http://kernel.org>
>>     <http://kernel.org <http://kernel.org>>> using this command:
>>      >         `wget
>>      >          >
>>      > https://cdn.kernel.org/pub/linux/kernel/v6.x/linux-6.5.tar.xz`
>>     <https://cdn.kernel.org/pub/linux/kernel/v6.x/linux-6.5.tar.xz>
>>      >      
>>  <https://cdn.kernel.org/pub/linux/kernel/v6.x/linux-6.5.tar.xz
>>     <https://cdn.kernel.org/pub/linux/kernel/v6.x/linux-6.5.tar.xz>>
>>      >          >
>>      >      
>>  <https://cdn.kernel.org/pub/linux/kernel/v6.x/linux-6.5.tar.xz`
>>     <https://cdn.kernel.org/pub/linux/kernel/v6.x/linux-6.5.tar.xz>
>>      >      
>>  <https://cdn.kernel.org/pub/linux/kernel/v6.x/linux-6.5.tar.xz
>>     <https://cdn.kernel.org/pub/linux/kernel/v6.x/linux-6.5.tar.xz>>>.
>>      >          >
>>      >
>>      >         Right. Which kernel are you actually running at this
>>     time? You
>>      >         can check
>>      >         using: `uname -a`
>>      >
>>      >         In any case, Linux upstream 6.5 is not the same as
>>     Ubuntus's 6.5.
>>      >
>>      >          > - *Which Linux distribution and version are you using?*
>>      >          >    Running `lsb_release -a` gives me:
>>      >          >    ```
>>      >          >    No LSB modules are available.
>>      >          >    Distributor ID: Ubuntu
>>      >          >    Description:    Ubuntu 22.04.4 LTS
>>      >          >    Release:        22.04
>>      >          >    Codename:       jammy
>>      >          >    ```
>>      >          >
>>      >          > - *Assuming your distribution has kernel packages,
>>     which do
>>      >         you have
>>      >          > installed, and do you have the corresponding kernel
>>     headers
>>      >         package
>>      >          > installed?*
>>      >          >
>>      >          >    Running `dpkg -l | grep linux-image` shows these
>>     installed
>>      >         packages:
>>      >          >    ```
>>      >          >    ii  linux-image-5.15.0-116-generic     
>> 5.15.0-116.126
>>      >          >   amd64   Signed kernel image generic
>>      >          >    ii  linux-image-6.5.0-28-generic
>>      >         6.5.0-28.29~22.04.1
>>      >          >   amd64   Signed kernel image generic
>>      >          >    ii  linux-image-6.5.0-41-generic
>>      >         6.5.0-41.41~22.04.2
>>      >          >   amd64   Signed kernel image generic
>>      >          >    ii  linux-image-6.5.0-44-generic
>>      >         6.5.0-44.44~22.04.1
>>      >          >   amd64   Signed kernel image generic
>>      >          >    ii  linux-image-generic       5.15.0.116.116
>>      >          >   amd64   Generic Linux kernel image
>>      >          >    ii  linux-image-generic-hwe-22.04
>>      >           6.5.0.44.44~22.04.1
>>      >          >   amd64   Generic Linux kernel image
>>      >          >    ```
>>      >          >
>>      >          >    And `dpkg -l | grep linux-headers` shows these
>>     installed
>>      >         header packages:
>>      >          >    ```
>>      >          >    ii  linux-headers-5.15.0-116
>>      >         5.15.0-116.126      all
>>      >          >      Header files related to Linux kernel version 
>> 5.15.0
>>      >          >    ii  linux-headers-5.15.0-116-generic     
>> 5.15.0-116.126
>>      >          >   amd64   Linux kernel headers for version 5.15.0 on
>>     64 bit
>>      >         x86 SMP
>>      >          >    ii  linux-headers-6.5.0-28-generic
>>      >         6.5.0-28.29~22.04.1
>>      >          >   amd64   Linux kernel headers for version 6.5.0 on 
>> 64 bit
>>      >         x86 SMP
>>      >          >    ii  linux-headers-6.5.0-41-generic
>>      >         6.5.0-41.41~22.04.2
>>      >          >   amd64   Linux kernel headers for version 6.5.0 on 
>> 64 bit
>>      >         x86 SMP
>>      >          >    ii  linux-headers-6.5.0-44-generic
>>      >         6.5.0-44.44~22.04.1
>>      >          >   amd64   Linux kernel headers for version 6.5.0 on 
>> 64 bit
>>      >         x86 SMP
>>      >          >    ii  linux-headers-generic       5.15.0.116.116
>>      >          >   amd64   Generic Linux kernel headers
>>      >          >    ii  linux-headers-generic-hwe-22.04
>>      >           6.5.0.44.44~22.04.1
>>      >          >   amd64   Generic Linux kernel headers
>>      >          >    ```
>>      >          >
>>      >
>>      >         Since you've got a few different images and headers
>>     installed,
>>      >         it is
>>      >         possible to check that same info the linux-source
>>     package: `dpkg
>>      >         -l |
>>      >         grep linux-source`.
>>      >
>>      >         Maybe one thing tripping this process up is that you
>>     don't have
>>      >         an apt
>>      >         source that provides linux-source that matches the 
>> kernel you
>>      >         are using.
>>      >         Could you run either `apt-cache policy` or `apt policy` 
>> and
>>      >         `apt-mark
>>      >         showhold` in order to know which sources you have 
>> configured.
>>      >
>>      >         There are also alternative sources for the full source 
>> tree:
>>      >
>>      >         - You could download the specific package from
>>      > packages.ubuntu.com <http://packages.ubuntu.com>
>>     <http://packages.ubuntu.com <http://packages.ubuntu.com>>. See
>>      > https://packages.ubuntu.com <https://packages.ubuntu.com>
>>     <https://packages.ubuntu.com <https://packages.ubuntu.com>> and
>>      >         use `apt install ./downloaded_package.deb`
>>      >
>>      >         - You could clone the kernel source for Ubuntu and
>>     checkout the
>>      >         appropriate tag. See
>>      >
>>     
>> https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/jammy/refs/tags <https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/jammy/refs/tags> <https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/jammy/refs/tags <https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/jammy/refs/tags>>
>>      >
>>      >         thanks,
>>      >         kienan
>>      >
>>
> _______________________________________________
> lttng-dev mailing list
> lttng-dev@lists.lttng.org
> https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [lttng-dev] loading modules with modprobe

[François Belias via lttng-dev]

[-- Attachment #1.1: Type: text/plain, Size: 19119 bytes --]

Hello Kienan,

 my issue has been fixed. thanks for your help

Le jeu. 18 juill. 2024, à 10 h 58, Kienan Stewart <kstewart@efficios.com> a
écrit :

> A couple of addendums inline below
>
> On 7/18/24 10:47 AM, Kienan Stewart via lttng-dev wrote:
> > Hi François,
> >
> > The dmesg excerpts are spaced by a number of minutes each, are they from
> > different attempts to build and load the modules?
> >
> > On 7/17/24 8:44 PM, François Belias wrote:
> >> Hello Kienan,
> >>
> >> Thanks to your advice, I managed to load some modules. However, I
> >> encountered errors like these when running `dmesg | grep lttng`:
> >>
> >> ```
> >> [  218.043672] module lttng_kretprobes: .gnu.linkonce.this_module
> >> section size must match the kernel's built struct module size at run
> time
> >
> > This sounds like the version of the module is built against the wrong
> > kernel headers or source.
>
> Also, are you using lttng-modules master or did you swap back to
> stable-2.13?
>
> If you're still on master and have the most recent changes
> lttng_kretprobes isn't required anymore. You will also need lttng-tools
> from master with the corresponding changes so that it doesn't try to
> load lttng_kretprobes.
>
> >
> >> [  650.560883] lttng_kretprobes: module verification failed: signature
> >> and/or required key missing - tainting kernel
> >
> > Is your system running with secure boot enabled? If so, you either need
> > to disable secure boot or sign the built modules with a MOK key that is
> > correctly enrolled.
> >
> > See the following documentation:
> >
> > - https://lttng.org/docs/v2.13/#doc-linux-kernel-sig
> > - https://www.kernel.org/doc/html/latest/admin-guide/module-signing.html
> > - https://github.com/dell/dkms
> > -
> >
> https://gitlab.com/linux-kernel/stable/-/blob/master/kernel/module/main.c#L2064
> >
> > Whatever you did for this test is pretty closed to working. The previous
> > errors with the section size check happen earlier in the module load
> > process. This check seems to be one of the last, c.f.
> > https://elixir.bootlin.com/linux/v6.10/source/kernel/module/main.c#L2908
> >
> >> [ 1468.329988] module lttng_ring_buffer_client_discard:
> >> .gnu.linkonce.this_module section size must match the kernel's built
> >> struct module size at run time
> >> ```
> >>
> >> I'm unsure of their meaning. After loading my module, how can I be
> >> certain that my tracepoints are active? I read online that people use
> >> `sudo lttng list -k` for this purpose, but when I tried it, I got the
> >> error: "Error: Unable to list kernel events: Kernel tracer not
> >> available". Is there another way?
> >
> > If you're still getting "kernel tracer not available" then either
> > required kernel modules haven't been loaded or you started lttng-session
> > with `--no-kernel` option. `lttng-sessiond -vvv`, dmesg, and/or
> > journalctl should have information on which modules failed to load.
> >
>
> I know you're doing `sudo lttng list -k`, but the basic checks still apply:
>
> - is the lttng-sessiond running as the root user
> - (when running without sudo) does the tracing group exist, and is your
> user a member of that group
>
> > kienan
> >
> >>
> >> Thank you for your assistance.
> >>
> >> Le mer. 17 juill. 2024, à 12 h 08, Kienan Stewart
> >> <kstewart@efficios.com <mailto:kstewart@efficios.com>> a écrit :
> >>
> >>     Hi François,
> >>
> >>     On 7/17/24 11:47 AM, François Belias wrote:
> >>      > Hello, Kienan
> >>      >
> >>      >
> >>      > *Right. Which kernel are you actually running at this time? You
> >>     can check
> >>      > using: `uname -a`*
> >>      > Linux philippe 6.5.0-41-generic #41~22.04.2-Ubuntu SMP
> >>     PREEMPT_DYNAMIC
> >>      > Mon Jun  3 11:32:55 UTC 2 x86_64 x86_64 x86_64 GNU/Linux
> >>      >
> >>      > *Since you've got a few different images and headers installed,
> >> it is
> >>      > possible to check that same info the linux-source package:
> >> `dpkg -l |
> >>      > grep linux-source`.*
> >>      > ii  linux-source                                   5.15.0.116.116
> >>      >                               all          Linux kernel source
> >> with
> >>      > Ubuntu patches
> >>      > ii  linux-source-5.15.0                       5.15.0-116.126
> >>      >                         all          Linux kernel source for
> >> version
> >>      > 5.15.0 with Ubuntu patches
> >>      > ii  linux-source-6.5.0                      6.5.0-44.44~22.04.1
> >>      >                        all          Linux kernel source for
> >> version
> >>
> >>     Your currently running `6.5.0-41-generic #41~22.04.2-Ubuntu`
> >>
> >>     You have source installed for `6.5.0-44.44~22.04.1`. From your
> >> previous
> >>     e-mail, there is also a linux image installed:
> >>
> >>     ```
> >>     linux-image-6.5.0-44-generic 6.5.0-44.44~22.04.1
> >>     ```
> >>
> >>     Assuming you are using grub, could you verify that one of the
> options
> >>     for booting in `/boot/grub/grub.cfg` is linux 6.5.0-44.44. Look
> >> for the
> >>     lines similar to the following inside a menu entry block:
> >>
> >>     (Note my example is 5.15.0-116, but you will be lookng for
> >> 6.5.0-44 or
> >>     6.5.0-44.44)
> >>
> >>     ```
> >>     linux   /boot/vmlinuz-5.15.0-116-generic root=....
> >>     initrd  /boot/initrd.img-5.15.0-116-generic
> >>     ```
> >>
> >>     If you have a grub entry for linux 6.5.0-44.44, could you reboot
> >> to run
> >>     on the linux 6.5.0-44.44.
> >>
> >>     After rebooting verify with `uname -a` that your kernel version is
> >>     6.5.0-44.44.
> >>
> >>     Make sure you have unpacked the linux sources:
> >>
> >>     ```
> >>     # Approximating the version numbers, since I don't have 6.5.0
> >> installed
> >>     cd /usr/src/linux-source-6.5.0
> >>     bunzip2 linux-source-6.5.0.tar.bz2
> >>     tar -xf linux-source-6.5.0.tar
> >>     ```
> >>
> >>     Then retry the build with
> >>     `KERNELDIR=/usr/src/linux-source-6.5.0/linux-source-6.5.0/`
> >>
> >>     I haven't personally tried the full steps on an Ubuntu
> >> installation so
> >>     you may still need to tweak a couple of things, but I think this
> >> should
> >>     get you on the right path.
> >>
> >>     Really the important thing is to make sure that the kernel source
> >> code
> >>     matches the kernel you are running.
> >>
> >>      > 6.5.0 with Ubuntu patches
> >>      > *Could you run either `apt-cache policy` or `apt policy` and
> >>     `apt-mark
> >>      > showhold` in order to know which sources you have configured.*
> >>      > For this part, /*apt-mark showhold*/ gave me nothing I don't know
> >>     what
> >>      > is means.
> >>
> >>     `apt-mark showhold` will show any packages that are on hold - it is
> a
> >>     way of marking packages that shouldn't be updated.
> >>
> >>     As you have the linux-source for 6.5.0-44.44~22.04.1, I don't
> >> think it
> >>     will be necessary to have the output of `apt-cache policy`.
> >>
> >>     thanks,
> >>     kienan
> >>
> >>      >
> >>      > I am looking at the links, but nothing seems to match my kernel
> >>     version.
> >>      >
> >>      > Thanks for your assistance.
> >>      >
> >>      > Le mer. 17 juill. 2024, à 11 h 46, François Belias
> >>      > <beliasossim@gmail.com <mailto:beliasossim@gmail.com>
> >>     <mailto:beliasossim@gmail.com <mailto:beliasossim@gmail.com>>> a
> >> écrit :
> >>      >
> >>      >     Hello, Kienan
> >>      >
> >>      >
> >>      >     *Right. Which kernel are you actually running at this time?
> >>     You can
> >>      >     check
> >>      >     using: `uname -a`*
> >>      >     Linux philippe 6.5.0-41-generic #41~22.04.2-Ubuntu SMP
> >>      >     PREEMPT_DYNAMIC Mon Jun  3 11:32:55 UTC 2 x86_64 x86_64
> >>     x86_64 GNU/Linux
> >>      >
> >>      >     *Since you've got a few different images and headers
> >>     installed, it is
> >>      >     possible to check that same info the linux-source package:
> >>     `dpkg -l |
> >>      >     grep linux-source`.*
> >>      >     ii  linux-source       5.15.0.116.116
> >>      >                                       all          Linux kernel
> >>     source
> >>      >     with Ubuntu patches
> >>      >     ii  linux-source-5.15.0                       5.15.0-116.126
> >>      >                                 all          Linux kernel
> >> source for
> >>      >     version 5.15.0 with Ubuntu patches
> >>      >     ii  linux-source-6.5.0
> >> 6.5.0-44.44~22.04.1
> >>      >                                all          Linux kernel source
> >> for
> >>      >     version 6.5.0 with Ubuntu patches
> >>      >     *Could you run either `apt-cache policy` or `apt policy` and
> >>     `apt-mark
> >>      >     showhold` in order to know which sources you have
> configured.*
> >>      >     For this part, /*apt-mark showhold*/ gave me nothing I
> >> don't know
> >>      >     what is means.
> >>      >
> >>      >     I am looking at the links, but nothing seems to match my
> >> kernel
> >>      >     version.
> >>      >
> >>      >     Thanks for your assistance.
> >>      >
> >>      >     Le mer. 17 juill. 2024, à 10 h 55, Kienan Stewart
> >>      >     <kstewart@efficios.com <mailto:kstewart@efficios.com>
> >>     <mailto:kstewart@efficios.com <mailto:kstewart@efficios.com>>> a
> >> écrit :
> >>      >
> >>      >         Hi François,
> >>      >
> >>      >         On 7/16/24 4:34 PM, François Belias wrote:
> >>      >          > Hello Kienan,
> >>      >          >
> >>      >          > When I proceed with the following steps:
> >>      >          >
> >>      >          > ```
> >>      >          > # Warning: this removes all untracked content from the
> >>      >         lttng-modules
> >>      >          > directory
> >>      >          > cd /path/to/lttng-modules-source
> >>      >          > git clean -dxf
> >>      >          >
> >>      >          > make
> >>      >          > sudo make modules_install
> >>      >          > sudo depmod -a
> >>      >          > ```
> >>      >          >
> >>      >          > I encounter a warning in
> >>      >          > `/home/philippe/lttng-modules/src/probes/Kbuild:79`
> >>     stating
> >>      >         "File
> >>      >          > ./arch/x86/kvm/lapic.h not found. Probe 'kvm'
> >>     x86-specific is
> >>      >         disabled.
> >>      >          > Use full kernel source tree to enable it." Although
> >>      >         everything compiles
> >>      >          > fine and I'm able to load modules, when I execute
> >>     `sudo lttng
> >>      >         list -k`,
> >>      >          > I don't see the tracepoints `kvm_entry` and
> >>     `kvm_exit`, which
> >>      >         I believe
> >>      >          > should be available. Could this be due to not using my
> >>     full
> >>      >         kernel tree
> >>      >          > during compilation?
> >>      >
> >>      >
> >>      >         Exactly. Those probes depend on having the full kernel
> >>     source as
> >>      >         they
> >>      >         require private headers which aren't shipped as part of
> >> the
> >>      >         `linux-headers` package.
> >>      >
> >>      >          >
> >>      >          > Here are the answers to your questions:
> >>      >          >
> >>      >          > - *Where did `/usr/src/linux-source-6.5.0` come from?*
> >>      >          >    I initially attempted `sudo apt-get update && sudo
> >>     apt-get
> >>      >         install
> >>      >          > linux-source`, but it gave me this answer
> >>     */linux-source is
> >>      >         already the
> >>      >          > newest version (5.15.0.116.116)/*`, which doesn't
> >> match my
> >>      >         current
> >>      >          > kernel version. So, I downloaded it directly from
> >>     kernel.org <http://kernel.org>
> >>      >         <http://kernel.org <http://kernel.org>>
> >>      >          > <http://kernel.org <http://kernel.org>
> >>     <http://kernel.org <http://kernel.org>>> using this command:
> >>      >         `wget
> >>      >          >
> >>      > https://cdn.kernel.org/pub/linux/kernel/v6.x/linux-6.5.tar.xz`
> <https://cdn.kernel.org/pub/linux/kernel/v6.x/linux-6.5.tar.xz>
> >>     <https://cdn.kernel.org/pub/linux/kernel/v6.x/linux-6.5.tar.xz>
> >>      >
> >>  <https://cdn.kernel.org/pub/linux/kernel/v6.x/linux-6.5.tar.xz
> >>     <https://cdn.kernel.org/pub/linux/kernel/v6.x/linux-6.5.tar.xz>>
> >>      >          >
> >>      >
> >>  <https://cdn.kernel.org/pub/linux/kernel/v6.x/linux-6.5.tar.xz`
> <https://cdn.kernel.org/pub/linux/kernel/v6.x/linux-6.5.tar.xz>
> >>     <https://cdn.kernel.org/pub/linux/kernel/v6.x/linux-6.5.tar.xz>
> >>      >
> >>  <https://cdn.kernel.org/pub/linux/kernel/v6.x/linux-6.5.tar.xz
> >>     <https://cdn.kernel.org/pub/linux/kernel/v6.x/linux-6.5.tar.xz>>>.
> >>      >          >
> >>      >
> >>      >         Right. Which kernel are you actually running at this
> >>     time? You
> >>      >         can check
> >>      >         using: `uname -a`
> >>      >
> >>      >         In any case, Linux upstream 6.5 is not the same as
> >>     Ubuntus's 6.5.
> >>      >
> >>      >          > - *Which Linux distribution and version are you
> using?*
> >>      >          >    Running `lsb_release -a` gives me:
> >>      >          >    ```
> >>      >          >    No LSB modules are available.
> >>      >          >    Distributor ID: Ubuntu
> >>      >          >    Description:    Ubuntu 22.04.4 LTS
> >>      >          >    Release:        22.04
> >>      >          >    Codename:       jammy
> >>      >          >    ```
> >>      >          >
> >>      >          > - *Assuming your distribution has kernel packages,
> >>     which do
> >>      >         you have
> >>      >          > installed, and do you have the corresponding kernel
> >>     headers
> >>      >         package
> >>      >          > installed?*
> >>      >          >
> >>      >          >    Running `dpkg -l | grep linux-image` shows these
> >>     installed
> >>      >         packages:
> >>      >          >    ```
> >>      >          >    ii  linux-image-5.15.0-116-generic
> >> 5.15.0-116.126
> >>      >          >   amd64   Signed kernel image generic
> >>      >          >    ii  linux-image-6.5.0-28-generic
> >>      >         6.5.0-28.29~22.04.1
> >>      >          >   amd64   Signed kernel image generic
> >>      >          >    ii  linux-image-6.5.0-41-generic
> >>      >         6.5.0-41.41~22.04.2
> >>      >          >   amd64   Signed kernel image generic
> >>      >          >    ii  linux-image-6.5.0-44-generic
> >>      >         6.5.0-44.44~22.04.1
> >>      >          >   amd64   Signed kernel image generic
> >>      >          >    ii  linux-image-generic       5.15.0.116.116
> >>      >          >   amd64   Generic Linux kernel image
> >>      >          >    ii  linux-image-generic-hwe-22.04
> >>      >           6.5.0.44.44~22.04.1
> >>      >          >   amd64   Generic Linux kernel image
> >>      >          >    ```
> >>      >          >
> >>      >          >    And `dpkg -l | grep linux-headers` shows these
> >>     installed
> >>      >         header packages:
> >>      >          >    ```
> >>      >          >    ii  linux-headers-5.15.0-116
> >>      >         5.15.0-116.126      all
> >>      >          >      Header files related to Linux kernel version
> >> 5.15.0
> >>      >          >    ii  linux-headers-5.15.0-116-generic
> >> 5.15.0-116.126
> >>      >          >   amd64   Linux kernel headers for version 5.15.0 on
> >>     64 bit
> >>      >         x86 SMP
> >>      >          >    ii  linux-headers-6.5.0-28-generic
> >>      >         6.5.0-28.29~22.04.1
> >>      >          >   amd64   Linux kernel headers for version 6.5.0 on
> >> 64 bit
> >>      >         x86 SMP
> >>      >          >    ii  linux-headers-6.5.0-41-generic
> >>      >         6.5.0-41.41~22.04.2
> >>      >          >   amd64   Linux kernel headers for version 6.5.0 on
> >> 64 bit
> >>      >         x86 SMP
> >>      >          >    ii  linux-headers-6.5.0-44-generic
> >>      >         6.5.0-44.44~22.04.1
> >>      >          >   amd64   Linux kernel headers for version 6.5.0 on
> >> 64 bit
> >>      >         x86 SMP
> >>      >          >    ii  linux-headers-generic       5.15.0.116.116
> >>      >          >   amd64   Generic Linux kernel headers
> >>      >          >    ii  linux-headers-generic-hwe-22.04
> >>      >           6.5.0.44.44~22.04.1
> >>      >          >   amd64   Generic Linux kernel headers
> >>      >          >    ```
> >>      >          >
> >>      >
> >>      >         Since you've got a few different images and headers
> >>     installed,
> >>      >         it is
> >>      >         possible to check that same info the linux-source
> >>     package: `dpkg
> >>      >         -l |
> >>      >         grep linux-source`.
> >>      >
> >>      >         Maybe one thing tripping this process up is that you
> >>     don't have
> >>      >         an apt
> >>      >         source that provides linux-source that matches the
> >> kernel you
> >>      >         are using.
> >>      >         Could you run either `apt-cache policy` or `apt policy`
> >> and
> >>      >         `apt-mark
> >>      >         showhold` in order to know which sources you have
> >> configured.
> >>      >
> >>      >         There are also alternative sources for the full source
> >> tree:
> >>      >
> >>      >         - You could download the specific package from
> >>      > packages.ubuntu.com <http://packages.ubuntu.com>
> >>     <http://packages.ubuntu.com <http://packages.ubuntu.com>>. See
> >>      > https://packages.ubuntu.com <https://packages.ubuntu.com>
> >>     <https://packages.ubuntu.com <https://packages.ubuntu.com>> and
> >>      >         use `apt install ./downloaded_package.deb`
> >>      >
> >>      >         - You could clone the kernel source for Ubuntu and
> >>     checkout the
> >>      >         appropriate tag. See
> >>      >
> >>
> >>
> https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/jammy/refs/tags
> <
> https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/jammy/refs/tags>
> <
> https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/jammy/refs/tags
> <
> https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/jammy/refs/tags
> >>
> >>      >
> >>      >         thanks,
> >>      >         kienan
> >>      >
> >>
> > _______________________________________________
> > lttng-dev mailing list
> > lttng-dev@lists.lttng.org
> > https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
>

[-- Attachment #1.2: Type: text/html, Size: 30127 bytes --]

[-- Attachment #2: Type: text/plain, Size: 156 bytes --]

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev