Re: [lttng-dev] loading modules with modprobe

["François Belias via lttng-dev" <lttng-dev@lists.lttng.org>]

[-- Attachment #1.1: Type: text/plain, Size: 10223 bytes --]

Hello Kienan,

When I proceed with the following steps:

```
# Warning: this removes all untracked content from the lttng-modules
directory
cd /path/to/lttng-modules-source
git clean -dxf

make
sudo make modules_install
sudo depmod -a
```

I encounter a warning in
`/home/philippe/lttng-modules/src/probes/Kbuild:79` stating "File
./arch/x86/kvm/lapic.h not found. Probe 'kvm' x86-specific is disabled. Use
full kernel source tree to enable it." Although everything compiles fine
and I'm able to load modules, when I execute `sudo lttng list -k`, I don't
see the tracepoints `kvm_entry` and `kvm_exit`, which I believe should be
available. Could this be due to not using my full kernel tree during
compilation?

Here are the answers to your questions:

- *Where did `/usr/src/linux-source-6.5.0` come from?*
  I initially attempted `sudo apt-get update && sudo apt-get install
linux-source`, but it gave me this answer *linux-source is already the
newest version (5.15.0.116.116)*`, which doesn't match my current kernel
version. So, I downloaded it directly from kernel.org using this command:
`wget https://cdn.kernel.org/pub/linux/kernel/v6.x/linux-6.5.tar.xz`.

- *Which Linux distribution and version are you using?*
  Running `lsb_release -a` gives me:
  ```
  No LSB modules are available.
  Distributor ID: Ubuntu
  Description:    Ubuntu 22.04.4 LTS
  Release:        22.04
  Codename:       jammy
  ```

- *Assuming your distribution has kernel packages, which do you have
installed, and do you have the corresponding kernel headers package
installed?*

  Running `dpkg -l | grep linux-image` shows these installed packages:
  ```
  ii  linux-image-5.15.0-116-generic           5.15.0-116.126      amd64
Signed kernel image generic
  ii  linux-image-6.5.0-28-generic             6.5.0-28.29~22.04.1  amd64
Signed kernel image generic
  ii  linux-image-6.5.0-41-generic             6.5.0-41.41~22.04.2  amd64
Signed kernel image generic
  ii  linux-image-6.5.0-44-generic             6.5.0-44.44~22.04.1  amd64
Signed kernel image generic
  ii  linux-image-generic                      5.15.0.116.116      amd64
Generic Linux kernel image
  ii  linux-image-generic-hwe-22.04            6.5.0.44.44~22.04.1  amd64
Generic Linux kernel image
  ```

  And `dpkg -l | grep linux-headers` shows these installed header packages:
  ```
  ii  linux-headers-5.15.0-116                 5.15.0-116.126      all
Header files related to Linux kernel version 5.15.0
  ii  linux-headers-5.15.0-116-generic         5.15.0-116.126      amd64
Linux kernel headers for version 5.15.0 on 64 bit x86 SMP
  ii  linux-headers-6.5.0-28-generic           6.5.0-28.29~22.04.1  amd64
Linux kernel headers for version 6.5.0 on 64 bit x86 SMP
  ii  linux-headers-6.5.0-41-generic           6.5.0-41.41~22.04.2  amd64
Linux kernel headers for version 6.5.0 on 64 bit x86 SMP
  ii  linux-headers-6.5.0-44-generic           6.5.0-44.44~22.04.1  amd64
Linux kernel headers for version 6.5.0 on 64 bit x86 SMP
  ii  linux-headers-generic                    5.15.0.116.116      amd64
Generic Linux kernel headers
  ii  linux-headers-generic-hwe-22.04          6.5.0.44.44~22.04.1  amd64
Generic Linux kernel headers
  ```

Thanks.
















Le mar. 16 juill. 2024, à 15 h 41, Kienan Stewart <kstewart@efficios.com> a
écrit :

> Hi François,
>
> On 7/16/24 3:01 PM, François Belias wrote:
> > Hello Kienan,
> >
> > I've continued investigating my issue and discovered something new that
> > might shed some light. Here's the rundown:
> >
> > 1. To ensure my add-ons weren't causing problems, deleted all old
> > modules and I followed the documentation on the LTTng website
> > (https://lttng.org/docs/v2.13/#doc-building-from-source
> > <https://lttng.org/docs/v2.13/#doc-building-from-source>) and
> downloaded
> > `lttng-modules-2.13.13`.
> >
> > 2. Inside the downloaded folder, I successfully executed `make
> > KERNELDIR=/usr/src/linux-source-6.5.0` without any issues.
> >
> > 3. Subsequently, I ran `make KERNELDIR=/usr/src/linux-source-6.5.0
> > modules_install`, which also completed successfully.
> >
> > 4. After that, I updated the module dependencies using `sudo depmod -a`,
> > which went fine.
> >
> > However, when I tried to load a module, such as `sudo modprobe
> > lttng-probe-kvm`, it reported that the module couldn't be found at
> > `/lib/modules/6.5.0-41-generic/`. Upon checking the logs, I confirmed
> > that step 3 had installed the modules in `/lib/modules/6.5.0/`.
> >
> > To address this, I modified step 3 to simply `make modules_install`,
> > which then installed the modules correctly at
> > `/lib/modules/6.5.0-41-generic/` and ran again step 4. But now, when
> > attempting to load a module, I'm encountering the error `modprobe:
> > ERROR: could not insert 'lttng_probe_kvm': Exec format error`.
> >
> > Do you have any insights or ideas on what might be causing this issue?
> >
>
> It sounds like /usr/src/linux-source-6.5.0 is not correct source for the
> kernel that you are currently running.
>
> When running `make modules_install` in your modified step 3 (without
> KERNELDIR set), the modules were installed for your current kernel
> directory as auto-detected by the build script.
>
> What happens if you do the following
>
> ```
> unset KERNELDIR
>
> # Warning: this removes all untracked content from the lttng-modules
> directory
> cd /path/to/lttng-modules-source
> git clean -dxf
>
> make
> sudo make modules_install
> sudo depmod -a
> ```
>
> KERNELDIR is set based on the current running kernel: `uname -r`. C.f.
>
> https://github.com/lttng/lttng-modules/blob/434e86eea846b33cad57f1d8105089905f541853/Makefile#L16C1-L16C50
>
> A couple of questions for you:
>
> - Where did /usr/src/linux-source-6.5.0 come from?
> - Which linux distribution + version are you using?
> - Assuming your distribution has kernel packages, which do you have
> installed, and do you have the corresponding kernel headers package
> installed?
>
> thanks,
> kienan
>
> > Thanks again for your assistance.
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > Le mar. 16 juill. 2024, à 09 h 05, Kienan Stewart <kstewart@efficios.com
> > <mailto:kstewart@efficios.com>> a écrit :
> >
> >     Hi François,
> >
> >     On 7/15/24 6:10 PM, François Belias via lttng-dev wrote:
> >      > Hello,
> >      >
> >      > I'm currently running Linux kernel version 6.5.0-41-generic and
> >      > attempting to install LTTng from source. Here are the steps I
> >     followed:
> >      >
> >      >   - I cloned the LTTng git repository to my computer from
> >      >
> >     [
> https://github.com/lttng/lttng-modules.git](https://github.com/lttng/lttng-modules.git)
> <
> https://github.com/lttng/lttng-modules.git%5D(https://github.com/lttng/lttng-modules.git)>
> <
> https://github.com/lttng/lttng-modules.git](https://github.com/lttng/lttng-modules.git)
> <
> https://github.com/lttng/lttng-modules.git%5D(https://github.com/lttng/lttng-modules.git)
> >>.
> >      > - Downloaded the kernel source tree corresponding to my kernel
> >     version
> >      > to enable compilation.
> >      > - Executed `make KERNELDIR=/path/to/my/linux/source`, which was
> >      > completed successfully.
> >      > - Ran `make KERNELDIR=/path/to/my/linux/source modules_install`,
> >     also
> >      > without any issues.
> >      > - Finally, I executed `sudo depmod -a`, which was completed
> >     successfully.
> >      >
> >      > However, when I attempt to load a module with commands like `sudo
> >      > modprobe lttng-ketprobes` or "sudo modprobe lttng-tracer", I
> >     encounter
> >      > the following errors:
> >      >
> >      > - For `lttng-ketprobes`: `modprobe: FATAL: Module lttng-ketprobes
> >     not
> >      > found in directory /lib/modules/6.5.0-41-generic`.
> >      > - For `lttng-tracer`: `modprobe: ERROR: could not insert
> >     'lttng_tracer':
> >      > Exec format error`.
> >
> >     If you rebuilt modules on the master branch as of yesterday you may
> >     need
> >     to clean the LTTng modules install directory, rebuild, and reinstall.
> >
> >     This is due to kprobes, kretprobes, and uprobes modules being
> included
> >     in the tracer core module. C.f.
> >
> https://github.com/lttng/lttng-modules/commit/0c2c6606eb66cb5222ab7a2a4f8329fd5dfdef70
> <
> https://github.com/lttng/lttng-modules/commit/0c2c6606eb66cb5222ab7a2a4f8329fd5dfdef70
> >
> >
> >     You may want to stick with running lttng-modules 2.13.x which will
> not
> >     have breaking changes merged in.
> >
> >     The exec format error also could be due to modules that are installed
> >     but built against a different version of the Linux kernel.
> >
> >     A few things to check
> >
> >     - If your kernel has been upgraded recently, ensure the new kernel is
> >     the one actually running
> >     - If the modules haven't been rebuilt since the upgrade, they should
> be
> >     rebuilt and reinstalled
> >     - As I recall, you are playing around with some addon modules - do
> any
> >     of those modules define dependencies on the kprobes, kretprobes, or
> >     uprobes modules which now are no longer produced by builds of the
> >     master
> >     branch
> >
> >     Is there further information in dmesg or journalctl regarding the
> >     module
> >     loading?
> >
> >     hope this helps.
> >
> >     thanks,
> >     kienan
> >
> >      >
> >      > I've confirmed that the modules exist but can't load them. I'm
> >     unsure
> >      > how to resolve these issues. Could someone please assist me?
> >      >
> >      > Thank you.
> >      >
> >      > _______________________________________________
> >      > lttng-dev mailing list
> >      > lttng-dev@lists.lttng.org <mailto:lttng-dev@lists.lttng.org>
> >      > https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
> >     <https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev>
> >
>

[-- Attachment #1.2: Type: text/html, Size: 14008 bytes --]

[-- Attachment #2: Type: text/plain, Size: 156 bytes --]

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev