[lttng-dev] lttng-tools-2.10.6: Spawning consumerd failed

[lttng-dev] lttng-tools-2.10.6: Spawning consumerd failed

[Zvi Vered]

Hi Jonathan,

I'm not using buildroot nor Yokto.
In order to build lttng-tools I ran:
./configure --prefix=/64_sw/git/zvi_vered/MY_PROJ/linux4.9.20/boot/rootfs
make
make install

This path contains a complete root file system (based on busybox) that is
later copied to my target.

On the target I ran:
lttng-sessiond -vvv --verbose-consumer
and got:
....
I/O warning : failed to load external entity
"/64_sw/git/zvi_vered/MY_PROJ/linux4.9.20/boot/rootfs/share/xml/lttng/session.xsd"
XML Error: Failed to locate the main schema resource at
'/64_sw/git/zvi_vered/MY_PROJ/linux4.9.20/boot/rootfs/share/xml/lttng/session.xsd'.
Error: XSD parsing failed
Error: Session load failed: Invalid session configuration
DEBUG1 - 00:10:36.746330272 [1476/1618]: Accepting client command ... (in
thread_manage_clients() at main.c:4374)

Then I ran:
lttng create my-kernel-session
and got:
....
DEBUG3 - 00:11:41.869875832 [1628/1628]: Looking for a kernel consumer at
these locations: (in spawn_consumerd() at main.c:2429)
DEBUG3 - 00:11:41.869901726 [1628/1628]:        1) NULL (in
spawn_consumerd() at main.c:2430)
DEBUG3 - 00:11:41.869908520 [1628/1628]:        2)
/64_sw/git/zvi_vered/MY_PROJ/linux4.9.20/boot/rootfs/lib/lttng/libexec/lttng-consumerd
(in spawn_consumerd() at main.c:2431)
DEBUG3 - 00:11:41.869915959 [1628/1628]:        3)
/64_sw/git/zvi_vered/MY_PROJ/linux4.9.20/boot/rootfs/lib/lttng/libexec/lttng-consumerd
(in spawn_consumerd() at main.c:2432)
DEBUG1 - 00:11:41.869971468 [1628/1628]: Could not find any valid consumerd
executable (in spawn_consumerd() at main.c:2444)
Error: Spawning consumerd failed

I double checked: The target has /lib/lttng-libexec/lttng-consumerd
But it looks for it in:
 /64_sw/git/zvi_vered/MY_PROJ/linux4.9.20/boot/rootfs/lib/lttng/libexec ]

What is missing in my configuration ?

Thank you,
Zvika

On Tue, May 14, 2019 at 9:04 PM Jonathan Rajotte-Julien <
jonathan.rajotte-julien at efficios.com> wrote:

> >
> > >
> > >
> > > I'm running under a 32 bit (x86) vanilla, 4.9.20 kernel with busybox
> > > with the following lttng versions:
>
> I took some time to at least test on a 32 bit system (buildroot) just in
> case
> and everything seems to be working fine on a base installation.
>
> The log from lttng-sessiond should provides more information on why we
> fail to
> spawn the consumerd (wrong path, etc.).
>
> Cheers
>
> --
> Jonathan Rajotte-Julien
> EfficiOS
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.lttng.org/pipermail/lttng-dev/attachments/20190515/88b036cc/attachment.html>

[lttng-dev] lttng-tools-2.10.6: Spawning consumerd failed

[Jonathan Rajotte-Julien]

On Wed, May 15, 2019 at 04:42:57PM +0300, Zvi Vered wrote:
> Hi Jonathan,
> 
> I'm not using buildroot nor Yokto.
> In order to build lttng-tools I ran:
> ./configure --prefix=/64_sw/git/zvi_vered/MY_PROJ/linux4.9.20/boot/rootfs

I'm no expert on building/cross compiling but I think you are missing some
options here. with-sysroot is probably one of them.

I think this should do the trick:

./configure --with-sysroot=/64_sw/git/zvi_vered/MY_PROJ/linux4.9.20/boot/rootfs
--prefix /

Another possibility would be using DESTDIR on make install.

> make
> make install
> 
> This path contains a complete root file system (based on busybox) that is
> later copied to my target.
> 
> On the target I ran:
> lttng-sessiond -vvv --verbose-consumer
> and got:
> ....
> I/O warning : failed to load external entity
> "/64_sw/git/zvi_vered/MY_PROJ/linux4.9.20/boot/rootfs/share/xml/lttng/session.xsd"
> XML Error: Failed to locate the main schema resource at
> '/64_sw/git/zvi_vered/MY_PROJ/linux4.9.20/boot/rootfs/share/xml/lttng/session.xsd'.
> Error: XSD parsing failed
> Error: Session load failed: Invalid session configuration
> DEBUG1 - 00:10:36.746330272 [1476/1618]: Accepting client command ... (in
> thread_manage_clients() at main.c:4374)
> 
> Then I ran:
> lttng create my-kernel-session
> and got:
> ....
> DEBUG3 - 00:11:41.869875832 [1628/1628]: Looking for a kernel consumer at
> these locations: (in spawn_consumerd() at main.c:2429)
> DEBUG3 - 00:11:41.869901726 [1628/1628]:        1) NULL (in
> spawn_consumerd() at main.c:2430)
> DEBUG3 - 00:11:41.869908520 [1628/1628]:        2)
> /64_sw/git/zvi_vered/MY_PROJ/linux4.9.20/boot/rootfs/lib/lttng/libexec/lttng-consumerd
> (in spawn_consumerd() at main.c:2431)
> DEBUG3 - 00:11:41.869915959 [1628/1628]:        3)
> /64_sw/git/zvi_vered/MY_PROJ/linux4.9.20/boot/rootfs/lib/lttng/libexec/lttng-consumerd
> (in spawn_consumerd() at main.c:2432)
> DEBUG1 - 00:11:41.869971468 [1628/1628]: Could not find any valid consumerd
> executable (in spawn_consumerd() at main.c:2444)
> Error: Spawning consumerd failed
> 
> I double checked: The target has /lib/lttng-libexec/lttng-consumerd
> But it looks for it in:
>  /64_sw/git/zvi_vered/MY_PROJ/linux4.9.20/boot/rootfs/lib/lttng/libexec ]
> 
> What is missing in my configuration ?

The prefix is used to bake in the path used by lttng-sessiond for
consumerd lookup.

We use libdir as the base path.

  --libdir=DIR            object code libraries [EPREFIX/lib]

Cheers

-- 
Jonathan Rajotte-Julien
EfficiOS

[lttng-dev] lttng-tools-2.10.6: Spawning consumerd failed

[Jonathan Rajotte-Julien]

Hi,

You forgot to cc the mailing list. Adding it.

On Fri, May 17, 2019 at 04:50:02AM +0300, Zvi Vered wrote:
> Hi Janathan, All,
> 
> I solved the path issue with
> 
> export LTTNG_CONSUMERD32_BIN=/lib/lttng/libexec
> export LTTNG_CONSUMERD32_LIBDIR =/lib
> export LTTNG_SESSION_CONFIG_XSD_PATH=/share/lttng/...

This is a way to solve it, I would still recommend you fix it at the source
(proper cross build).

...

> ....
> DEBUG3 - 00:13:48.436226116 [1980/1980]: Looking for a kernel consumer at
> these locations: (in spawn_consumerd() at main.c:2429)
> DEBUG3 - 00:13:48.436252139 [1980/1980]:        1) NULL (in
> spawn_consumerd() at main.c:2430)
> DEBUG3 - 00:13:48.436256414 [1980/1980]:        2)
> /64_sw/git/zvi_vered/MY_PROJ/linux4.9.20/boot/rootfs/lib/lttng/libexec/lttng-consumerd
> (in spawn_consumerd() at main.c:2431)
> DEBUG3 - 00:13:48.436260543 [1980/1980]:        3) /lib/lttng/libexec (in
> spawn_consumerd() at main.c:2432)
> DEBUG3 - 00:13:48.436313492 [1980/1980]: Found location #3 (in
> spawn_consumerd() at main.c:2441)
> DEBUG1 - 00:13:48.436319027 [1980/1980]: Using kernel consumer at:
> /lib/lttng/libexec (in spawn_consumerd() at main.c:2448)
> PERROR - 00:13:48.436341261 [1980/1980]: Consumer execl(): Permission
> denied (in spawn_consumerd() at main.c:2539)

We found the consumerd in /lib/lttng/libexec/lttng-consumerd. Good.

We now try to execl the following:

/lib/lttng/libexec/lttng-consumerd -k --consumerd-cmd-sock <a_path> --consumerd-error-sock <b_path> --group root

And execl returns "Permission Denied".

Clearly, /lib/lttng/libexec/lttng-consumerd have some permissions problem.

"Permission denied" is linked to EACCES on my system.
  asm-generic/errno-base.h
  17:#define    EACCES          13     /* Permission denied */

"man execve" list all the reasons why execl might return EACCES per "man execl":

  EACCES Search permission is denied on a component of the path prefix of filename or the name of a script interpreter.  (See also path_resolution(7).)
  EACCES The file or a script interpreter is not a regular file.
  EACCES Execute permission is denied for the file or a script or ELF interpreter.
  EACCES The filesystem is mounted noexec.

One of them probably fit your current scenario.

Let us know how it goes.

Cheers

-- 
Jonathan Rajotte-Julien
EfficiOS

[lttng-dev] lttng-tools-2.10.6: Spawning consumerd failed

[Zvi Vered]

Hi Jonathan,

I had to run:
export LTTNG_CONSUMERD32_BIN=/lib/lttng/libexec/lttng-consumerd

Thank you very much !
Zvika

On Fri, May 17, 2019 at 10:48 PM Jonathan Rajotte-Julien <
jonathan.rajotte-julien at efficios.com> wrote:

> Hi,
>
> You forgot to cc the mailing list. Adding it.
>
> On Fri, May 17, 2019 at 04:50:02AM +0300, Zvi Vered wrote:
> > Hi Janathan, All,
> >
> > I solved the path issue with
> >
> > export LTTNG_CONSUMERD32_BIN=/lib/lttng/libexec
> > export LTTNG_CONSUMERD32_LIBDIR =/lib
> > export LTTNG_SESSION_CONFIG_XSD_PATH=/share/lttng/...
>
> This is a way to solve it, I would still recommend you fix it at the source
> (proper cross build).
>
> ...
>
> > ....
> > DEBUG3 - 00:13:48.436226116 [1980/1980]: Looking for a kernel consumer at
> > these locations: (in spawn_consumerd() at main.c:2429)
> > DEBUG3 - 00:13:48.436252139 [1980/1980]:        1) NULL (in
> > spawn_consumerd() at main.c:2430)
> > DEBUG3 - 00:13:48.436256414 [1980/1980]:        2)
> >
> /64_sw/git/zvi_vered/MY_PROJ/linux4.9.20/boot/rootfs/lib/lttng/libexec/lttng-consumerd
> > (in spawn_consumerd() at main.c:2431)
> > DEBUG3 - 00:13:48.436260543 [1980/1980]:        3) /lib/lttng/libexec (in
> > spawn_consumerd() at main.c:2432)
> > DEBUG3 - 00:13:48.436313492 [1980/1980]: Found location #3 (in
> > spawn_consumerd() at main.c:2441)
> > DEBUG1 - 00:13:48.436319027 [1980/1980]: Using kernel consumer at:
> > /lib/lttng/libexec (in spawn_consumerd() at main.c:2448)
> > PERROR - 00:13:48.436341261 [1980/1980]: Consumer execl(): Permission
> > denied (in spawn_consumerd() at main.c:2539)
>
> We found the consumerd in /lib/lttng/libexec/lttng-consumerd. Good.
>
> We now try to execl the following:
>
> /lib/lttng/libexec/lttng-consumerd -k --consumerd-cmd-sock <a_path>
> --consumerd-error-sock <b_path> --group root
>
> And execl returns "Permission Denied".
>
> Clearly, /lib/lttng/libexec/lttng-consumerd have some permissions problem.
>
> "Permission denied" is linked to EACCES on my system.
>   asm-generic/errno-base.h
>   17:#define    EACCES          13     /* Permission denied */
>
> "man execve" list all the reasons why execl might return EACCES per "man
> execl":
>
>   EACCES Search permission is denied on a component of the path prefix of
> filename or the name of a script interpreter.  (See also
> path_resolution(7).)
>   EACCES The file or a script interpreter is not a regular file.
>   EACCES Execute permission is denied for the file or a script or ELF
> interpreter.
>   EACCES The filesystem is mounted noexec.
>
> One of them probably fit your current scenario.
>
> Let us know how it goes.
>
> Cheers
>
> --
> Jonathan Rajotte-Julien
> EfficiOS
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.lttng.org/pipermail/lttng-dev/attachments/20190519/515c83c6/attachment.html>