From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from simark.ca by simark.ca with LMTP id 5eg9BxKLkGCoUAAAWB0awg (envelope-from ) for ; Mon, 03 May 2021 19:45:22 -0400 Received: by simark.ca (Postfix, from userid 112) id 0F7511F11C; Mon, 3 May 2021 19:45:22 -0400 (EDT) X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on simark.ca X-Spam-Level: X-Spam-Status: No, score=-1.1 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,HTML_MESSAGE,MAILING_LIST_MULTI,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.2 Received: from lists.lttng.org (lists.lttng.org [167.114.26.123]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by simark.ca (Postfix) with ESMTPS id D4E261E54D for ; Mon, 3 May 2021 19:45:20 -0400 (EDT) Received: from lists-lttng01.efficios.com (localhost [IPv6:::1]) by lists.lttng.org (Postfix) with ESMTP id 4FZ02m1vhqz1grY; Mon, 3 May 2021 19:45:20 -0400 (EDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=lists.lttng.org; s=default; t=1620085520; bh=bUXHHSEsZWlLjU0PZwgs3Staq4Se75K0LcOA1+A/MUA=; h=References:In-Reply-To:Date:To:Cc:Subject:List-Id: List-Unsubscribe:List-Archive:List-Post:List-Help:List-Subscribe: From:Reply-To:From; b=H51q9UEfpVAH6IFFjQGtJ/Wx84zZf1p57aezbDmdksjP/xuTjeNrszqaS9JKkCSxa tEmJEtrBUlpHVPTaE+u4dn4bqJNLl02V15gL5GDujORvSFnZJLO3WrxEkPFLh8XL34 ha4gqR7HIouyhdsHb4Ij7CFdpHr6XKQOcOHgFhrn+lBkh88u/quo4Zpg3RsXkXBPP/ R8XiQoNnActm1GJe64mFIml7Fo3Ca/zL/lxZ9v1SYtf8s8A4SpvxhLeTq3gxmuMjB1 U2ISjhPXK8Hkw3JcEfE814zz8lrP/sTDl3wV/4c/k7eH3iXY6bV1UKCY4ZzbHeLEOD fUgMk/knk2cbA== Received: from mail-oi1-x235.google.com (mail-oi1-x235.google.com [IPv6:2607:f8b0:4864:20::235]) by lists.lttng.org (Postfix) with ESMTPS id 4FZ02l4zblz1grX for ; Mon, 3 May 2021 19:45:19 -0400 (EDT) Received: by mail-oi1-x235.google.com with SMTP id k25so7099060oic.4 for ; Mon, 03 May 2021 16:45:19 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=tWVnFEudJHPRxpnEDSLirQMvwUgenca54wjY7B4uVSs=; b=WE7nnmDSi3LkTlaq3EVCgMjNLLfVky7OdFWD3SonMJDoHFcc7Pscjo5FoA3rKLrww/ ca+4jKt7jXX5zmJYI+phJSg35a5SMf0yOmCpgy68JlAzQLrxgDKgfQbPj/qol3w7gsgY SttfiQZJcQMpuBq3w1mMbWqE435NBudDDMLmzMsLZe3JENoCG42ALKEDk1VBmDLKVZJw P/f7J7l7en33Okuw8vWUDza3xJfgHTT88El39wAYeTcaPX3LvoMcjZyI8pXCK6CxHFQx 2h7PxjX//PNy1H0Dk8gHgfB7bH9l0PXXJt7RFRkqlq4LvsbHkOvWxLvasf5Ma1Ws3q9/ 9/EA== X-Gm-Message-State: AOAM533izzLGwT5AnJXckPLZwMFQk2H5IYaa00SC9oYnueYa100ft/qS LgcoWg9MsmsnAQ7TcOZiH9BucbWoPr6xZetsA2A= X-Google-Smtp-Source: ABdhPJykZxVomKpCzH5woEnZMBTzPQsR34GzLunnStfp5fpaKadnu3SaApPEmdVQVlklfr/kJU8KCXE+RuZODoFyDcQ= X-Received: by 2002:a54:4812:: with SMTP id j18mr15792213oij.14.1620085518775; Mon, 03 May 2021 16:45:18 -0700 (PDT) MIME-Version: 1.0 References: <20210406140753.GE79283@joraj-alpa> In-Reply-To: <20210406140753.GE79283@joraj-alpa> Date: Mon, 3 May 2021 16:45:07 -0700 Message-ID: To: Jonathan Rajotte-Julien Cc: lttng-dev@lists.lttng.org Subject: Re: [lttng-dev] Userspace tracing in docker containers X-BeenThere: lttng-dev@lists.lttng.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: LTTng development list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , From: Eqbal via lttng-dev Reply-To: Eqbal Content-Type: multipart/mixed; boundary="===============4715195462515793040==" Errors-To: lttng-dev-bounces@lists.lttng.org Sender: "lttng-dev" --===============4715195462515793040== Content-Type: multipart/alternative; boundary="000000000000da3a9a05c1758d0c" --000000000000da3a9a05c1758d0c Content-Type: text/plain; charset="UTF-8" Thanks for the responses. The reasoning makes sense. We have decided to run lttng-sessiond on the host. Our trace generating application will run in a container and so will our libbabeltrace based trace consumer app (using live sessions). On Tue, Apr 6, 2021 at 7:07 AM Jonathan Rajotte-Julien < jonathan.rajotte-julien@efficios.com> wrote: > Hi, > > On Mon, Apr 05, 2021 at 11:09:39AM -0700, Eqbal via lttng-dev wrote: > > Hi, > > > > I am trying to get user space tracing working for an application running > in > > a docker container. I am running lttng session daemon in another > container. > > I mounted the unix socket locations (either /var/run/lttng for root or > > $HOME/.lttng for another user). By doing that I can run commands like > lttng > > create or lttng list , but the tracepoint events from the > > application don't get registered and there is no trace output. > > > > I enabled LTTNG_UST_DEBUG an ran lttng-sessiond in verbose mode (-vvv and > > --verbose-consumer) and got the following error message: > > > > "*Unix socket credential pid=0. Refusing application in distinct, > > non-nested pid namespace.*" > > > > It appears that for some calls to the session daemon there is a > getsockopt > > syscall made with *SO_PEERCRED* which returns 0 for pid and the call is > > failed with *LTTNG_UST_ERR_PEERCRED_PID* error (see get_cred call in > > ustctl.c). > > > > If I comment out the getsockopt call, my application tracing starts to > work. > > > > From what I found, docker cannot support getsockopt/SO_PEERCRED call to > get > > peer pid on the unix socket which would make sense as it's in a separate > > namespace. > > > > I have a few questions on this: > > 1. What is the reason for the get_cred/getsockopt call with SO_PEERCRED? > I > > would like to understand why it's required for some and not other calls. > > > More information is found in the introducing commit: > > commit a834901f2890deadb815d7f9e3ab79c3ba673994 > Author: Mathieu Desnoyers > Date: Mon Oct 12 16:52:03 2020 -0400 > > Fix: Use unix socket peercred for pid, uid, gid credentials > > Currently, the session daemon trust the pid, ppid, uid, and gid values > passed by the application, but should really validate the uid using > unix > socket peercred. This fix uses the peercred values rather than the > values provided by the application on registration for: > > - pid, uid and gid on Linux, > - uid and gid on FreeBSD. > > This should improve how the session daemon deals with containerized > applications on Linux as well. Applications are required to be either > in > the same pid namespace, or in a pid namespace nested within the pid > namespace of the lttng-sessiond, so the session daemon can map the > application pid to something meaningful within its own pid namespace. > Applications in a unrelated (disjoint) pid namespace will be refused by > the session daemon. > > About the uid and gid with user namespaces on Linux, those will provide > meaningful IDs if the application user namespace is either the same as > the user namespace of the session daemon, or a nested user namespace. > Otherwise, the IDs will be that of /proc/sys/kernel/overflowuid and > /proc/sys/kernel/overflowgid, which typically maps to nobody.nogroup on > current distributions. > > Given that fetching the parent pid (ppid) of the application would > require to use /proc//status (which is racy wrt pid reuse), expose > the ppid provided by the application on registration instead, but only > in situations where the application sits in the same pid namespace as > the session daemon (on Linux), which is detected by checking if the pid > provided by the application matches the pid obtained using unix socket > credentials. The ppid is only used for logging/debugging purposes in > the > session daemon anyway, so it is OK to use the value provided by the > application for that purpose. > > Fixes: #1286 > Signed-off-by: Mathieu Desnoyers > Change-Id: I94742e57dad642106908d09e2c7e395993c2c48f > > As for "why it's required for some and not other calls.", there is a > difference > between communicating with a lttng-sessiond daemon (using the lttng CLI) > and > userspace application registering. They are essentially two distinct > communication interface. Now, to be honest, I'm not certain of the complete > "security" policy for the lttng-sessiond <-> CLI interface and if we > should be > more strict or not. > > > 2. Is there any workaround for this problem, so that I can get this to > work > > with the container topology I am working with (app in one container and > > lttng daemons in another). > > Based on the commit message, lttng-ust explicitly cannot be used across > non-nested pid namespace. > > Could you give us more information on the goal for the topology you plan > to use? > This could lead to further discussion and/or alternative solution based on > the > goal and constraints of your deployment. > > > 3. Related to 2, are there any gotchas to bypassing the getsockopt call > in > > get_cred? > > Based on the content of the mentioned bug (1286) [1], the principal > concern is: > > " > This means a non-root application could theoretically impersonate a root > application from a tracing perspective, and thus access root tracing > buffers in > a per-uid configuration, which is unwanted. > " > > [1] https://bugs.lttng.org/issues/1286 > > Cheers > > -- > Jonathan Rajotte-Julien > EfficiOS > --000000000000da3a9a05c1758d0c Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Thanks for the responses. The reasoning makes sense. We ha= ve decided to run lttng-sessiond on the host. Our trace generating applicat= ion will run in a container and so will our libbabeltrace based trace consu= mer app (using live sessions).

On Tue, Apr 6, 2021 at 7:07 AM Jonathan R= ajotte-Julien <j= onathan.rajotte-julien@efficios.com> wrote:
Hi,

On Mon, Apr 05, 2021 at 11:09:39AM -0700, Eqbal via lttng-dev wrote:
> Hi,
>
> I am trying to get user space tracing working for an application runni= ng in
> a docker container. I am running lttng session daemon in another conta= iner.
> I mounted the unix socket locations (either /var/run/lttng for root or=
> $HOME/.lttng for another user). By doing that I can run commands like = lttng
> create or lttng list <session-name>, but the tracepoint events f= rom the
> application don't get registered and there is no trace output.
>
> I enabled LTTNG_UST_DEBUG an ran lttng-sessiond in verbose mode (-vvv = and
> --verbose-consumer) and got the following error message:
>
> "*Unix socket credential pid=3D0. Refusing application in distinc= t,
> non-nested pid namespace.*"
>
> It appears that for some calls to the session daemon there is a getsoc= kopt
> syscall made with *SO_PEERCRED* which returns 0 for pid and the call i= s
> failed with *LTTNG_UST_ERR_PEERCRED_PID* error (see get_cred call in > ustctl.c).
>
> If I comment out the getsockopt call, my application tracing starts to= work.
>
> From what I found, docker cannot support getsockopt/SO_PEERCRED call t= o get
> peer pid on the unix socket which would make sense as it's in a se= parate
> namespace.
>
> I have a few questions on this:
> 1. What is the reason for the get_cred/getsockopt call with SO_PEERCRE= D? I
> would like to understand why it's required for some and not other = calls.


More information is found in the introducing commit:

=C2=A0 commit a834901f2890deadb815d7f9e3ab79c3ba673994
=C2=A0 Author: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
=C2=A0 Date:=C2=A0 =C2=A0Mon Oct 12 16:52:03 2020 -0400

=C2=A0 =C2=A0 Fix: Use unix socket peercred for pid, uid, gid credentials
=C2=A0 =C2=A0 Currently, the session daemon trust the pid, ppid, uid, and g= id values
=C2=A0 =C2=A0 passed by the application, but should really validate the uid= using unix
=C2=A0 =C2=A0 socket peercred. This fix uses the peercred values rather tha= n the
=C2=A0 =C2=A0 values provided by the application on registration for:

=C2=A0 =C2=A0 - pid, uid and gid on Linux,
=C2=A0 =C2=A0 - uid and gid on FreeBSD.

=C2=A0 =C2=A0 This should improve how the session daemon deals with contain= erized
=C2=A0 =C2=A0 applications on Linux as well. Applications are required to b= e either in
=C2=A0 =C2=A0 the same pid namespace, or in a pid namespace nested within t= he pid
=C2=A0 =C2=A0 namespace of the lttng-sessiond, so the session daemon can ma= p the
=C2=A0 =C2=A0 application pid to something meaningful within its own pid na= mespace.
=C2=A0 =C2=A0 Applications in a unrelated (disjoint) pid namespace will be = refused by
=C2=A0 =C2=A0 the session daemon.

=C2=A0 =C2=A0 About the uid and gid with user namespaces on Linux, those wi= ll provide
=C2=A0 =C2=A0 meaningful IDs if the application user namespace is either th= e same as
=C2=A0 =C2=A0 the user namespace of the session daemon, or a nested user na= mespace.
=C2=A0 =C2=A0 Otherwise, the IDs will be that of /proc/sys/kernel/overflowu= id and
=C2=A0 =C2=A0 /proc/sys/kernel/overflowgid, which typically maps to nobody.= nogroup on
=C2=A0 =C2=A0 current distributions.

=C2=A0 =C2=A0 Given that fetching the parent pid (ppid) of the application = would
=C2=A0 =C2=A0 require to use /proc/<pid>/status (which is racy wrt pi= d reuse), expose
=C2=A0 =C2=A0 the ppid provided by the application on registration instead,= but only
=C2=A0 =C2=A0 in situations where the application sits in the same pid name= space as
=C2=A0 =C2=A0 the session daemon (on Linux), which is detected by checking = if the pid
=C2=A0 =C2=A0 provided by the application matches the pid obtained using un= ix socket
=C2=A0 =C2=A0 credentials. The ppid is only used for logging/debugging purp= oses in the
=C2=A0 =C2=A0 session daemon anyway, so it is OK to use the value provided = by the
=C2=A0 =C2=A0 application for that purpose.

=C2=A0 =C2=A0 Fixes: #1286
=C2=A0 =C2=A0 Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com<= /a>>
=C2=A0 =C2=A0 Change-Id: I94742e57dad642106908d09e2c7e395993c2c48f

As for "why it's required for some and not other calls.", the= re is a difference
between communicating with a lttng-sessiond daemon (using the lttng CLI) an= d
userspace application registering. They are essentially two distinct
communication interface. Now, to be honest, I'm not certain of the comp= lete
"security" policy for the lttng-sessiond <-> CLI interface = and if we should be
more strict or not.

> 2. Is there any workaround for this problem, so that I can get this to= work
> with the container topology I am working with (app in one container an= d
> lttng daemons in another).

Based on the commit message, lttng-ust explicitly cannot be used across
non-nested pid namespace.

Could you give us more information on the goal for the topology you plan to= use?
This could lead to further discussion and/or alternative solution based on = the
goal and constraints of your deployment.

> 3. Related to 2, are there any gotchas to bypassing the getsockopt cal= l in
> get_cred?

Based on the content of the mentioned bug (1286) [1],=C2=A0 the principal c= oncern is:

"
This means a non-root application could theoretically impersonate a root application from a tracing perspective, and thus access root tracing buffer= s in
a per-uid configuration, which is unwanted.
"

[1] https://bugs.lttng.org/issues/1286

Cheers

--
Jonathan Rajotte-Julien
EfficiOS
--000000000000da3a9a05c1758d0c-- --===============4715195462515793040== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ lttng-dev mailing list lttng-dev@lists.lttng.org https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev --===============4715195462515793040==--