From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from simark.ca by simark.ca with LMTP id ifL2CAxPRmbTFxQAWB0awg (envelope-from ) for ; Thu, 16 May 2024 14:23:08 -0400 Authentication-Results: simark.ca; dkim=pass (2048-bit key; unprotected) header.d=lists.lttng.org header.i=@lists.lttng.org header.a=rsa-sha256 header.s=default header.b=AfNBfEQw; dkim-atps=neutral Received: by simark.ca (Postfix, from userid 112) id 128E01E0C1; Thu, 16 May 2024 14:23:08 -0400 (EDT) Received: from lists.lttng.org (lists.lttng.org [167.114.26.123]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (prime256v1) server-digest SHA256) (No client certificate requested) by simark.ca (Postfix) with ESMTPS id C84B51E030 for ; Thu, 16 May 2024 14:23:05 -0400 (EDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=lists.lttng.org; s=default; t=1715883785; bh=jYhxZcRjdKNnc/StNQCwmcfHcGsmC8WhXEfzPp3KR+I=; h=References:In-Reply-To:Date:To:Cc:Subject:List-Id: List-Unsubscribe:List-Archive:List-Post:List-Help:List-Subscribe: From:Reply-To:From; b=AfNBfEQw68lunMUO7JDjt5mSWMkYE9UelK2KvIZkupEavkABE8WBM6xy78lOD9Y/H kOF928wYOa5qJmGF8bWtkJVTwwKkkV7s+IPekwViunDud3HW+PzNDbxY/T7vhmX/qs 3+ZRffBTM4L80K3giYLYSEm9ltbbTJanUl3S8Cu4dhOcrMTOiGRhvlhqXPvbV6vifZ bWFH6PZG+SqLWNDWF13B1l1OXgax6oQBMxW7DOx81AIgjADVOcRzrXpBUeLGYnJjCS bIM3NMsMlxkBUOGUy8yrIYl7ldgTH580Qpf0Blq7k5pXusrLPmPqfGkgTj+G/LZrPG lyhtpRAlq+SUw== Received: from lists-lttng01.efficios.com (localhost [IPv6:::1]) by lists.lttng.org (Postfix) with ESMTP id 4VgJP51hBwz2nrB; Thu, 16 May 2024 14:23:05 -0400 (EDT) Received: from mail-yw1-x1130.google.com (mail-yw1-x1130.google.com [IPv6:2607:f8b0:4864:20::1130]) by lists.lttng.org (Postfix) with ESMTPS id 4VgJP42S5lz2nr9 for ; Thu, 16 May 2024 14:23:03 -0400 (EDT) Received: by mail-yw1-x1130.google.com with SMTP id 00721157ae682-6277a2bb35fso8648197b3.1 for ; Thu, 16 May 2024 11:23:03 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1715883783; x=1716488583; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=nHlCpezm2uC4wBXQxWtAMNAoocMSbmdMBWrDfvHp0/Q=; b=PoBtSJjskwmBtO7CFfa4tW1Fu/Oz6qxHrlWY4o0G4soy8OVpXXGgfZcuZp1xUYwUIV 91NEODynM0b4AN4tS8VcxgkbnqUWwI0nra2Hgv4G0gRYpyKzMoOvDgoCb9IF4X/HQega dTQ/XTWks1bDv0P/Y8gLXqe8+1zEt7fPxzrP+UcHaQ4qk7CqnQZirtrtgGNR9LOfOufA k80y0MVZ7ibyYx4q4L5l6JPAIHzKapI9AC9C+FJbL3GszwFTcHoU8xMRhDR5Ac0FwA0K FQm3mw95fidySSZS8p9SxEPOd824E6PVgD1vqxDBUOSZv0G5kluhRCvgllbm7DzhUKM+ mUUQ== X-Gm-Message-State: AOJu0YxkP1rvQjtGWvWvo/nQgxEC4Zlc+zfGHf32q/G1DAQNBwIRrKtt lC2I7CYcff6befDdX9Updl6L9uDM5buDDnlVAPSPr1mlgyNf3CJyt2SL6CDzuborCe9oDElQKTT gIQEt1arQUGyo6xnvak3mQfgDW7HMPRLs7KqbOg== X-Google-Smtp-Source: AGHT+IEpyCwKcUAcDcUB3n2xpb/xdCdBSwaJmLvG9qAyCvnFn7S/xlJAgt6aQ4mt8XRnBOGCQC8BV/ZIr6GLJ+0hzRo= X-Received: by 2002:a0d:d545:0:b0:61a:bf86:9d29 with SMTP id 00721157ae682-620993808d8mr169017757b3.8.1715883782983; Thu, 16 May 2024 11:23:02 -0700 (PDT) MIME-Version: 1.0 References: <479fa8db-6d40-49c3-bb4f-8c920ab291ac@efficios.com> <39257e22-f657-4bba-b99d-97d4ee8789d5@efficios.com> In-Reply-To: <39257e22-f657-4bba-b99d-97d4ee8789d5@efficios.com> Date: Thu, 16 May 2024 11:22:52 -0700 Message-ID: To: Kienan Stewart Cc: lttng-dev@lists.lttng.org Subject: Re: [lttng-dev] Capturing snapshot on kernel panic X-BeenThere: lttng-dev@lists.lttng.org X-Mailman-Version: 2.1.39 Precedence: list List-Id: LTTng development list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , From: Damien Berget via lttng-dev Reply-To: Damien Berget Content-Type: multipart/mixed; boundary="===============7043627842448215477==" Errors-To: lttng-dev-bounces@lists.lttng.org Sender: "lttng-dev" --===============7043627842448215477== Content-Type: multipart/alternative; boundary="0000000000005c728106189653e7" --0000000000005c728106189653e7 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Thanks Kienan for these quick suggestions, we'll investigate the pmem route (I was not aware of the lttng-cash utility, it's pretty nice) even if I'm not sure how fast it would burn through our SSD, it might still be worth trying. As for kexec-tool, it's not officially supported on our embedded modules unfortunately, so we might be SOL there. We may have to try to add our own trace-point in kernel to use as trigger. Cheers Damien On Thu, May 16, 2024 at 8:12=E2=80=AFAM Kienan Stewart wrote: > Hi Damien, > > I want to expand on one of the options that could work for your case. > > On 5/16/24 9:37 AM, Kienan Stewart via lttng-dev wrote: > > Hi Damien, > > > > > > On 5/15/24 6:24 PM, Damien Berget via lttng-dev wrote: > >> Good day, > >> we have been using LTTng successfully to capture snapshots on user > >> defined tracepoints and it did provide invaluable to debug our issues. > >> Thanks to all the contributors of this project! > >> > >> We'd like to know if it would be possible to trigger on a kernel > >> panic? I might be dubiously possible as you would still need to have > >> the file-system working to write the results but I should ask. > >> > > > > For userspace tracing, I think the recommendation is usually to use a > > dax/pmem device and have the buffers for the session mapped there. Afte= r > > a panic, the contents of the buffers can be restored using > lttng-crash[1]. > > > > Note that dax/pem isn't supported by the kernel space tracer at this > time. > > > > If I recall, there are other ways to things in the panic sequence (that > > aren't lttng specific), but I'm personally not as familiar with the > > details of that stage of linux. > > > > It's possible to kexec-tools to load a new kernel post-panic[1]. If your > system uses kexec, the contents of RAM aren't necessarily flushed, and > if both the initial kernel and post-panic kernel started by kexec have > the same configuration for an emulated PMEM device using the memmap > paramenter [2,3] that region of memory can have a daxfs created in it > post-clean boot. > > Note: some systems may not flush the memory during a warm reboot, but > this is dependent on the BIOS. > > When your system boots you could do something like the following: > > * If it's a clean boot, create the daxfs > * If it's an "unclean" boot (e.g. the daxfs already exists, or a > kernel parameter informs you that it started post-panic) then you can > copy/move/use lttng-crash to persistent storage for analysis > * Start tracing using a snapshot session and the userspace buffers on > the daxfs. > > In this type of situation the "snapshot" command is never invoked > directly, but the recovery of the buffers to create a snapshot is possibl= e. > > [1]: https://www.kernel.org/doc/html/latest/admin-guide/kdump/kdump.html > [2]: > https://www.kernel.org/doc/html/latest/admin-guide/kernel-parameters.html > [3]: > > https://docs.pmem.io/persistent-memory/getting-started-guide/creating-dev= elopment-environments/linux-environments/linux-memmap > > thanks, > kienan > > >> Looking at available kernel syscall, the "reboot" one seems like a > >> good candidate, however I was not able to capture a snapshot on it. I > >> have tested the setup below with "--name=3Dchdir" syscall and it > >> works, "cd" to a directory will create a trace. But no dice with reboo= t. > >> > > > > The details of how this work will depend on your system. For example, m= y > > installations tend to use systemd as PID 1. The broad strokes seem to > > be: `/usr/sbin/reboot` is actually a link to `systemctl`, which I > > believe then kicks off the reboot.service, the PID 1 is swapped to > > /usr/lib/systemd/systemd-shutdown, sigterm then sigkill are sent to all > > processes, unmounts, syncs, calls the reboot system call [2,3]. > > > > As both the sigterm and the unmounts are done before the syscall, > > lttng-sessiond and the consumers will have already shutdown by the time > > it enters. > > > > While this doesn't necessarily help your original question of panics, i= f > > you want to snapshot before shutdown or reboot and are using systemd, > > it's possible to leave a script or binary in a known directory so that > > it's invoked prior to the rest of the shutdown sequence[4]. > > > > [1]: https://lttng.org/docs/v2.13/#doc-persistent-memory-file-systems > > [2]: > > > https://github.com/systemd/systemd/blob/6533c14997700f74e9ea42121303fc1f5= c63e62b/src/shutdown/shutdown.c > > [3]: > > > https://github.com/systemd/systemd/blob/main/src/shared/reboot-util.c#L77 > > [4]: https://www.systutorials.com/docs/linux/man/8-systemd-reboot/ > > > > hope this helps, > > kienan > > > >> Would you have any suggestions? > >> Thanks for your help, > >> Cheers > >> Damien > >> > >> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D > >> > >> # Prep output dir > >> mkdir /application/trace/ > >> rm -rf /application/trace/* > >> > >> # Create session > >> sudo lttng destroy snapshot-trace-session > >> sudo lttng create snapshot-trace-session --snapshot > >> --output=3D"/application/trace/" > >> sudo lttng enable-channel --kernel --num-subbuf=3D8 channelk > >> sudo lttng enable-channel --userspace --num-subbuf=3D8 channelu > >> > >> # Configure session > >> sudo lttng enable-event --kernel --syscall --all --channel channelk > >> sudo lttng enable-event --kernel --tracepoint "sched*" --channel > channelk > >> sudo lttng enable-event --userspace --all --channel channelu > >> sudo lttng add-context -u -t vtid -t procname > >> sudo lttng remove-trigger trig_reboot > >> sudo lttng add-trigger --name=3Dtrig_reboot \ > >> --condition=3Devent-rule-matches --type=3Dkernel:syscall:entr= y \ > >> --name=3Dreboot\ > >> --action=3Dsnapshot-session snapshot-trace-session \ > >> --rate-policy=3Donce-after:1 > >> > >> # start & list info > >> sudo lttng start > >> sudo lttng list snapshot-trace-session > >> sudo lttng list-triggers > >> > >> #=3D=3D=3D=3D=3D=3D=3D=3D test it... > >> sudo reboot > >> > >> #=3D=3D=3D=3D=3D=3D=3D reconnect and Nothing :( > >> $ ls -alu /application/trace/ > >> drwxr-xr-x 2 u u 4096 May 15 2024 . > >> drwxr-xr-x 10 u u 4096 May 15 2024 .. > >> > >> > >> _______________________________________________ > >> lttng-dev mailing list > >> lttng-dev@lists.lttng.org > >> https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev > > _______________________________________________ > > lttng-dev mailing list > > lttng-dev@lists.lttng.org > > https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev > --=20 *Damien Berget* --0000000000005c728106189653e7 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Thanks=C2=A0Kienan for these quick suggestions,
we'= ;ll investigate the pmem route (I was not aware of the=C2=A0lttng-cash util= ity, it's pretty nice) even if I'm not sure how fast it would burn = through our SSD, it might still be worth=C2=A0trying.
As for kexec-tool,= it's not officially supported=C2=A0on our embedded modules unfortunate= ly, so we might be SOL there. We may have to try to add our own trace-point= in kernel to use as trigger.
Cheers
Damien

On Thu, May 16, 202= 4 at 8:12=E2=80=AFAM Kienan Stewart <kstewart@efficios.com> wrote:
Hi Damien,

I want to expand on one of the options that could work for your case.

On 5/16/24 9:37 AM, Kienan Stewart via lttng-dev wrote:
> Hi Damien,
>
>
> On 5/15/24 6:24 PM, Damien Berget via lttng-dev wrote:
>> Good day,
>> we have been using LTTng successfully=C2=A0to capture snapshots on= user
>> defined tracepoints and it did provide invaluable to debug our iss= ues.
>> Thanks to all the contributors of this project!
>>
>> We'd like to know if it would be possible to trigger on a kern= el
>> panic? I might be dubiously possible as you would still need to ha= ve
>> the file-system working to write the results but I should ask.
>>
>
> For userspace tracing, I think the recommendation is usually to use a =
> dax/pmem device and have the buffers for the session mapped there. Aft= er
> a panic, the contents of the buffers can be restored using lttng-crash= [1].
>
> Note that dax/pem isn't supported by the kernel space tracer at th= is time.
>
> If I recall, there are other ways to things in the panic sequence (tha= t
> aren't lttng specific), but I'm personally not as familiar wit= h the
> details of that stage of linux.
>

It's possible to kexec-tools to load a new kernel post-panic[1]. If you= r
system uses kexec, the contents of RAM aren't necessarily flushed, and =
if both the initial kernel and post-panic kernel started by kexec have
the same configuration for an emulated PMEM device using the memmap
paramenter [2,3] that region of memory can have a daxfs created in it
post-clean boot.

Note: some systems may not flush the memory during a warm reboot, but
this is dependent on the BIOS.

When your system boots you could do something like the following:

=C2=A0 * If it's a clean boot, create the daxfs
=C2=A0 * If it's an "unclean" boot (e.g. the daxfs already ex= ists, or a
kernel parameter informs you that it started post-panic) then you can
copy/move/use lttng-crash to persistent storage for analysis
=C2=A0 * Start tracing using a snapshot session and the userspace buffers o= n
the daxfs.

In this type of situation the "snapshot" command is never invoked=
directly, but the recovery of the buffers to create a snapshot is possible.=

[1]: https://www.kernel.org/doc/h= tml/latest/admin-guide/kdump/kdump.html
[2]:
https://www.kernel.org/doc/= html/latest/admin-guide/kernel-parameters.html
[3]:
https://docs.pmem.io/persistent-memory/getting-st= arted-guide/creating-development-environments/linux-environments/linux-memm= ap

thanks,
kienan

>> Looking at available kernel syscall, the "reboot" one se= ems like a
>> good candidate, however I was not able to capture a snapshot on it= . I
>> have tested the setup below with "--name=3Dchdir" syscal= l and it
>> works,=C2=A0"cd" to a directory will=C2=A0create a trace= . But no dice with reboot.
>>
>
> The details of how this work will depend on your system. For example, = my
> installations tend to use systemd as PID 1. The broad strokes seem to =
> be: `/usr/sbin/reboot` is actually a link to `systemctl`, which I
> believe then kicks off the reboot.service, the PID 1 is swapped to > /usr/lib/systemd/systemd-shutdown, sigterm then sigkill are sent to al= l
> processes, unmounts, syncs, calls the reboot system call [2,3].
>
> As both the sigterm and the unmounts are done before the syscall,
> lttng-sessiond and the consumers will have already shutdown by the tim= e
> it enters.
>
> While this doesn't necessarily help your original question of pani= cs, if
> you want to snapshot before shutdown or reboot and are using systemd, =
> it's possible to leave a script or binary in a known directory so = that
> it's invoked prior to the rest of the shutdown sequence[4].
>
> [1]: https://lttng.org/docs/v2.= 13/#doc-persistent-memory-file-systems
> [2]:
> https://github.com/systemd/systemd/blob/6533c14997700f74e9ea421213= 03fc1f5c63e62b/src/shutdown/shutdown.c
> [3]:
> https://github.com/sys= temd/systemd/blob/main/src/shared/reboot-util.c#L77
> [4]: https://www.systutorials.com/= docs/linux/man/8-systemd-reboot/
>
> hope this helps,
> kienan
>
>> Would you have any suggestions?
>> Thanks for your help,
>> Cheers
>> Damien
>>
>> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D
>>
>> # Prep output dir
>> mkdir /application/trace/
>> rm -rf /application/trace/*
>>
>> # Create session
>> sudo lttng destroy snapshot-trace-session
>> sudo lttng create snapshot-trace-session --snapshot
>> --output=3D"/application/trace/"
>> sudo lttng enable-channel --kernel --num-subbuf=3D8 channelk
>> sudo lttng enable-channel --userspace --num-subbuf=3D8 channelu >>
>> # Configure session
>> sudo lttng enable-event --kernel --syscall --all --channel channel= k
>> sudo lttng enable-event --kernel --tracepoint "sched*" -= -channel channelk
>> sudo lttng enable-event --userspace --all --channel channelu
>> sudo lttng add-context -u -t vtid -t procname
>> sudo lttng remove-trigger trig_reboot
>> sudo lttng add-trigger --name=3Dtrig_reboot \
>> =C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0 --condition=3Devent-rule-matches= --type=3Dkernel:syscall:entry \
>> =C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0 --name=3Dreboot\
>> =C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0 --action=3Dsnapshot-session snap= shot-trace-session \
>> =C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0 --rate-policy=3Donce-after:1
>>
>> # start & list info
>> sudo lttng start
>> sudo lttng list snapshot-trace-session
>> sudo lttng list-triggers
>>
>> #=3D=3D=3D=3D=3D=3D=3D=3D test it...
>> sudo reboot
>>
>> #=3D=3D=3D=3D=3D=3D=3D reconnect and=C2=A0Nothing :(
>> $ ls -alu /application/trace/
>> drwxr-xr-x =C2=A0 =C2=A02 u =C2=A0u =C2=A0 =C2=A0 =C2=A0 4096 May = 15 =C2=A02024 .
>> drwxr-xr-x =C2=A0 10 u =C2=A0u =C2=A0 =C2=A0 =C2=A0 4096 May 15 = =C2=A02024 ..
>>
>>
>> _______________________________________________
>> lttng-dev mailing list
>> ltt= ng-dev@lists.lttng.org
>> https://lists.lttng.org/cgi-bin/m= ailman/listinfo/lttng-dev
> _______________________________________________
> lttng-dev mailing list
> lttng-d= ev@lists.lttng.org
> https://lists.lttng.org/cgi-bin/mailm= an/listinfo/lttng-dev


--
Damien Berget
--0000000000005c728106189653e7-- --===============7043627842448215477== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ lttng-dev mailing list lttng-dev@lists.lttng.org https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev --===============7043627842448215477==--