[ltt-dev] [PATCH 0/2] Add tracepoint of panic and kexec

[ltt-dev] [PATCH 0/2] Add tracepoint of panic and kexec

[Zhaolei]

Hi,

lttng can be widely used in many fields, for example, kernel flight recorder.

We can add tracepoints of panic and kexec for following reasons:
1: By tracing these functions, we can know exactly that panic and kexec are
   called, and when called.
2: We can make sure that the last record until panic is well traced.

Regards
Zhaolei

[ltt-dev] [PATCH 2/1] tracepoints-panic

[Zhaolei]

Instrumentation of panic related events : now include panic event only.
It is useful for build flight-recorder program based on lttng infrastructure.

Signed-off-by: Zhao lei <zhaolei at cn.fujitsu.com>
---
 include/trace/panic.h |   10 ++++++++++
 kernel/panic.c        |    7 +++++++
 2 files changed, 17 insertions(+), 0 deletions(-)
 create mode 100644 include/trace/panic.h

diff --git a/include/trace/panic.h b/include/trace/panic.h
new file mode 100644
index 0000000..eede4ca
--- /dev/null
+++ b/include/trace/panic.h
@@ -0,0 +1,10 @@
+#ifndef _TRACE_PANIC_H
+#define _TRACE_PANIC_H
+
+#include <linux/tracepoint.h>
+
+DECLARE_TRACE(panic_panic,
+	TPPROTO(const char *fmt, va_list args),
+	TPARGS(fmt, args));
+
+#endif
diff --git a/kernel/panic.c b/kernel/panic.c
index 12c5a0a..33b4440 100644
--- a/kernel/panic.c
+++ b/kernel/panic.c
@@ -21,6 +21,9 @@
 #include <linux/debug_locks.h>
 #include <linux/random.h>
 #include <linux/kallsyms.h>
+#include <trace/panic.h>
+
+DEFINE_TRACE(panic_panic);
 
 int panic_on_oops;
 int tainted;
@@ -68,6 +71,10 @@ NORET_TYPE void panic(const char * fmt, ...)
 	unsigned long caller = (unsigned long) __builtin_return_address(0);
 #endif
 
+	va_start(args, fmt);
+	trace_panic_panic(fmt, args);
+	va_end(args);
+
 	/*
 	 * It's possible to come here directly from a panic-assertion and not
 	 * have preempt disabled. Some functions called from here want
-- 
1.5.5.3

[ltt-dev] [PATCH 2/1] tracepoints-panic

[Mathieu Desnoyers]

* Zhaolei (zhaolei at cn.fujitsu.com) wrote:
> Instrumentation of panic related events : now include panic event only.
> It is useful for build flight-recorder program based on lttng infrastructure.
> 
> Signed-off-by: Zhao lei <zhaolei at cn.fujitsu.com>
> ---
>  include/trace/panic.h |   10 ++++++++++
>  kernel/panic.c        |    7 +++++++
>  2 files changed, 17 insertions(+), 0 deletions(-)
>  create mode 100644 include/trace/panic.h
> 
> diff --git a/include/trace/panic.h b/include/trace/panic.h
> new file mode 100644
> index 0000000..eede4ca
> --- /dev/null
> +++ b/include/trace/panic.h
> @@ -0,0 +1,10 @@
> +#ifndef _TRACE_PANIC_H
> +#define _TRACE_PANIC_H
> +
> +#include <linux/tracepoint.h>
> +
> +DECLARE_TRACE(panic_panic,
> +	TPPROTO(const char *fmt, va_list args),
> +	TPARGS(fmt, args));
> +

Interesting idea. Maybe kernel_panic would be more appropriate, in
include/trace/kernel.h ?

Mathieu

> +#endif
> diff --git a/kernel/panic.c b/kernel/panic.c
> index 12c5a0a..33b4440 100644
> --- a/kernel/panic.c
> +++ b/kernel/panic.c
> @@ -21,6 +21,9 @@
>  #include <linux/debug_locks.h>
>  #include <linux/random.h>
>  #include <linux/kallsyms.h>
> +#include <trace/panic.h>
> +
> +DEFINE_TRACE(panic_panic);
>  
>  int panic_on_oops;
>  int tainted;
> @@ -68,6 +71,10 @@ NORET_TYPE void panic(const char * fmt, ...)
>  	unsigned long caller = (unsigned long) __builtin_return_address(0);
>  #endif
>  
> +	va_start(args, fmt);
> +	trace_panic_panic(fmt, args);
> +	va_end(args);
> +
>  	/*
>  	 * It's possible to come here directly from a panic-assertion and not
>  	 * have preempt disabled. Some functions called from here want
> -- 
> 1.5.5.3
> 
> 

-- 
Mathieu Desnoyers
OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68

[ltt-dev] [PATCH 2/1] tracepoints-panic

[Zhaolei]

Mathieu Desnoyers wrote:
> * Zhaolei (zhaolei at cn.fujitsu.com) wrote:
>> Instrumentation of panic related events : now include panic event only.
>> It is useful for build flight-recorder program based on lttng infrastructure.
>>
>> Signed-off-by: Zhao lei <zhaolei at cn.fujitsu.com>
>> ---
>>  include/trace/panic.h |   10 ++++++++++
>>  kernel/panic.c        |    7 +++++++
>>  2 files changed, 17 insertions(+), 0 deletions(-)
>>  create mode 100644 include/trace/panic.h
>>
>> diff --git a/include/trace/panic.h b/include/trace/panic.h
>> new file mode 100644
>> index 0000000..eede4ca
>> --- /dev/null
>> +++ b/include/trace/panic.h
>> @@ -0,0 +1,10 @@
>> +#ifndef _TRACE_PANIC_H
>> +#define _TRACE_PANIC_H
>> +
>> +#include <linux/tracepoint.h>
>> +
>> +DECLARE_TRACE(panic_panic,
>> +	TPPROTO(const char *fmt, va_list args),
>> +	TPARGS(fmt, args));
>> +
> 
> Interesting idea. Maybe kernel_panic would be more appropriate, in
> include/trace/kernel.h ?
Hello, Mathieu

Thanks for your advise.
I will send v2 of this patch.

B.R
Zhaolei
> 
> Mathieu
> 
>> +#endif
>> diff --git a/kernel/panic.c b/kernel/panic.c
>> index 12c5a0a..33b4440 100644
>> --- a/kernel/panic.c
>> +++ b/kernel/panic.c
>> @@ -21,6 +21,9 @@
>>  #include <linux/debug_locks.h>
>>  #include <linux/random.h>
>>  #include <linux/kallsyms.h>
>> +#include <trace/panic.h>
>> +
>> +DEFINE_TRACE(panic_panic);
>>  
>>  int panic_on_oops;
>>  int tainted;
>> @@ -68,6 +71,10 @@ NORET_TYPE void panic(const char * fmt, ...)
>>  	unsigned long caller = (unsigned long) __builtin_return_address(0);
>>  #endif
>>  
>> +	va_start(args, fmt);
>> +	trace_panic_panic(fmt, args);
>> +	va_end(args);
>> +
>>  	/*
>>  	 * It's possible to come here directly from a panic-assertion and not
>>  	 * have preempt disabled. Some functions called from here want
>> -- 
>> 1.5.5.3
>>
>>
> 

[ltt-dev] [PATCH v2] add tracepoints of panic and kexec

[Zhaolei]

Instrumentation of following panic and kexec related events are added:
	panic
	kernel_kexec
	crash_kexec

It is useful for build flight-recorder program based on lttng infrastructure.

Signed-off-by: Zhao Lei <zhaolei at cn.fujitsu.com>
---
 include/trace/kernel.h |   10 ++++++++++
 kernel/kexec.c         |    8 ++++++++
 kernel/panic.c         |    7 +++++++
 3 files changed, 25 insertions(+), 0 deletions(-)

diff --git a/include/trace/kernel.h b/include/trace/kernel.h
index d2c3320..06d585f 100644
--- a/include/trace/kernel.h
+++ b/include/trace/kernel.h
@@ -2,6 +2,7 @@
 #define _TRACE_KERNEL_H
 
 #include <linux/tracepoint.h>
+#include <linux/kexec.h>
 
 DECLARE_TRACE(kernel_printk,
 	TPPROTO(unsigned long retaddr),
@@ -15,5 +16,14 @@ DECLARE_TRACE(kernel_module_free,
 DECLARE_TRACE(kernel_module_load,
 	TPPROTO(struct module *mod),
 	TPARGS(mod));
+DECLARE_TRACE(kernel_panic,
+	TPPROTO(const char *fmt, va_list args),
+	TPARGS(fmt, args));
+DECLARE_TRACE(kernel_kernel_kexec,
+	TPPROTO(struct kimage *image),
+	TPARGS(image));
+DECLARE_TRACE(kernel_crash_kexec,
+	TPPROTO(struct kimage *image, struct pt_regs *regs),
+	TPARGS(image, regs));
 
 #endif
diff --git a/kernel/kexec.c b/kernel/kexec.c
index aef2653..56cdaac 100644
--- a/kernel/kexec.c
+++ b/kernel/kexec.c
@@ -30,6 +30,7 @@
 #include <linux/pm.h>
 #include <linux/cpu.h>
 #include <linux/console.h>
+#include <trace/kernel.h>
 
 #include <asm/page.h>
 #include <asm/uaccess.h>
@@ -37,6 +38,9 @@
 #include <asm/system.h>
 #include <asm/sections.h>
 
+DEFINE_TRACE(kernel_kernel_kexec);
+DEFINE_TRACE(kernel_crash_kexec);
+
 /* Per cpu memory for storing cpu states in case of system crash. */
 note_buf_t* crash_notes;
 
@@ -1062,6 +1066,8 @@ asmlinkage long compat_sys_kexec_load(unsigned long entry,
 
 void crash_kexec(struct pt_regs *regs)
 {
+	trace_kernel_crash_kexec(kexec_crash_image, regs);
+
 	/* Take the kexec_mutex here to prevent sys_kexec_load
 	 * running on one cpu from replacing the crash kernel
 	 * we are using after a panic on a different cpu.
@@ -1428,6 +1434,8 @@ int kernel_kexec(void)
 {
 	int error = 0;
 
+	trace_kernel_kernel_kexec(kexec_image);
+
 	if (!mutex_trylock(&kexec_mutex))
 		return -EBUSY;
 	if (!kexec_image) {
diff --git a/kernel/panic.c b/kernel/panic.c
index 12c5a0a..3fa642e 100644
--- a/kernel/panic.c
+++ b/kernel/panic.c
@@ -21,6 +21,9 @@
 #include <linux/debug_locks.h>
 #include <linux/random.h>
 #include <linux/kallsyms.h>
+#include <trace/kernel.h>
+
+DEFINE_TRACE(kernel_panic);
 
 int panic_on_oops;
 int tainted;
@@ -68,6 +71,10 @@ NORET_TYPE void panic(const char * fmt, ...)
 	unsigned long caller = (unsigned long) __builtin_return_address(0);
 #endif
 
+	va_start(args, fmt);
+	trace_kernel_panic(fmt, args);
+	va_end(args);
+
 	/*
 	 * It's possible to come here directly from a panic-assertion and not
 	 * have preempt disabled. Some functions called from here want
-- 
1.5.5.3

[ltt-dev] [PATCH v2] add tracepoints of panic and kexec

[Mathieu Desnoyers]

* Zhaolei (zhaolei at cn.fujitsu.com) wrote:
> Instrumentation of following panic and kexec related events are added:
> 	panic
> 	kernel_kexec
> 	crash_kexec
> 
> It is useful for build flight-recorder program based on lttng infrastructure.
> 

Hrm, I'm wondering if these are events we are interested to trace or if
you plan to use these tracepoints as hooks into the kernel to connect
the tracer infrastructure to it ?

If you plan to use those as hooks into the kernel, we should probably
consider adding notifiers instead (notifier.h). This is the "blessed"
way to be called upon such events.

Mathieu

> Signed-off-by: Zhao Lei <zhaolei at cn.fujitsu.com>
> ---
>  include/trace/kernel.h |   10 ++++++++++
>  kernel/kexec.c         |    8 ++++++++
>  kernel/panic.c         |    7 +++++++
>  3 files changed, 25 insertions(+), 0 deletions(-)
> 
> diff --git a/include/trace/kernel.h b/include/trace/kernel.h
> index d2c3320..06d585f 100644
> --- a/include/trace/kernel.h
> +++ b/include/trace/kernel.h
> @@ -2,6 +2,7 @@
>  #define _TRACE_KERNEL_H
>  
>  #include <linux/tracepoint.h>
> +#include <linux/kexec.h>
>  
>  DECLARE_TRACE(kernel_printk,
>  	TPPROTO(unsigned long retaddr),
> @@ -15,5 +16,14 @@ DECLARE_TRACE(kernel_module_free,
>  DECLARE_TRACE(kernel_module_load,
>  	TPPROTO(struct module *mod),
>  	TPARGS(mod));
> +DECLARE_TRACE(kernel_panic,
> +	TPPROTO(const char *fmt, va_list args),
> +	TPARGS(fmt, args));
> +DECLARE_TRACE(kernel_kernel_kexec,
> +	TPPROTO(struct kimage *image),
> +	TPARGS(image));
> +DECLARE_TRACE(kernel_crash_kexec,
> +	TPPROTO(struct kimage *image, struct pt_regs *regs),
> +	TPARGS(image, regs));
>  
>  #endif
> diff --git a/kernel/kexec.c b/kernel/kexec.c
> index aef2653..56cdaac 100644
> --- a/kernel/kexec.c
> +++ b/kernel/kexec.c
> @@ -30,6 +30,7 @@
>  #include <linux/pm.h>
>  #include <linux/cpu.h>
>  #include <linux/console.h>
> +#include <trace/kernel.h>
>  
>  #include <asm/page.h>
>  #include <asm/uaccess.h>
> @@ -37,6 +38,9 @@
>  #include <asm/system.h>
>  #include <asm/sections.h>
>  
> +DEFINE_TRACE(kernel_kernel_kexec);
> +DEFINE_TRACE(kernel_crash_kexec);
> +
>  /* Per cpu memory for storing cpu states in case of system crash. */
>  note_buf_t* crash_notes;
>  
> @@ -1062,6 +1066,8 @@ asmlinkage long compat_sys_kexec_load(unsigned long entry,
>  
>  void crash_kexec(struct pt_regs *regs)
>  {
> +	trace_kernel_crash_kexec(kexec_crash_image, regs);
> +
>  	/* Take the kexec_mutex here to prevent sys_kexec_load
>  	 * running on one cpu from replacing the crash kernel
>  	 * we are using after a panic on a different cpu.
> @@ -1428,6 +1434,8 @@ int kernel_kexec(void)
>  {
>  	int error = 0;
>  
> +	trace_kernel_kernel_kexec(kexec_image);
> +
>  	if (!mutex_trylock(&kexec_mutex))
>  		return -EBUSY;
>  	if (!kexec_image) {
> diff --git a/kernel/panic.c b/kernel/panic.c
> index 12c5a0a..3fa642e 100644
> --- a/kernel/panic.c
> +++ b/kernel/panic.c
> @@ -21,6 +21,9 @@
>  #include <linux/debug_locks.h>
>  #include <linux/random.h>
>  #include <linux/kallsyms.h>
> +#include <trace/kernel.h>
> +
> +DEFINE_TRACE(kernel_panic);
>  
>  int panic_on_oops;
>  int tainted;
> @@ -68,6 +71,10 @@ NORET_TYPE void panic(const char * fmt, ...)
>  	unsigned long caller = (unsigned long) __builtin_return_address(0);
>  #endif
>  
> +	va_start(args, fmt);
> +	trace_kernel_panic(fmt, args);
> +	va_end(args);
> +
>  	/*
>  	 * It's possible to come here directly from a panic-assertion and not
>  	 * have preempt disabled. Some functions called from here want
> -- 
> 1.5.5.3
> 
> 
> 
> _______________________________________________
> ltt-dev mailing list
> ltt-dev at lists.casi.polymtl.ca
> http://lists.casi.polymtl.ca/cgi-bin/mailman/listinfo/ltt-dev
> 

-- 
Mathieu Desnoyers
OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68

[ltt-dev] [PATCH v2] add tracepoints of panic and kexec

[Zhaolei]

* From: "Mathieu Desnoyers" <compudj@krystal.dyndns.org>
>* Zhaolei (zhaolei at cn.fujitsu.com) wrote:
>> Instrumentation of following panic and kexec related events are added:
>> panic
>> kernel_kexec
>> crash_kexec
>> 
>> It is useful for build flight-recorder program based on lttng infrastructure.
>> 
> 
> Hrm, I'm wondering if these are events we are interested to trace or if
> you plan to use these tracepoints as hooks into the kernel to connect
> the tracer infrastructure to it ?
Hello, Mathieu

IMHO, both.

As a programmer who want to build  flight-recorder based on lttng infrastructure,
he can:
1: use lttng trace, and get lttng's trace datas from kernel-dump file.
or
2: use lttng's tracepoint, and record events himself.

But as a lttng's programmer, we can make both way possible.

What is your opinion?

B.R.
Zhaolei
> 
> If you plan to use those as hooks into the kernel, we should probably
> consider adding notifiers instead (notifier.h). This is the "blessed"
> way to be called upon such events.
> 
> Mathieu
> 
>> Signed-off-by: Zhao Lei <zhaolei at cn.fujitsu.com>
>> ---
>>  include/trace/kernel.h |   10 ++++++++++
>>  kernel/kexec.c         |    8 ++++++++
>>  kernel/panic.c         |    7 +++++++
>>  3 files changed, 25 insertions(+), 0 deletions(-)
>> 
>> diff --git a/include/trace/kernel.h b/include/trace/kernel.h
>> index d2c3320..06d585f 100644
>> --- a/include/trace/kernel.h
>> +++ b/include/trace/kernel.h
>> @@ -2,6 +2,7 @@
>>  #define _TRACE_KERNEL_H
>>  
>>  #include <linux/tracepoint.h>
>> +#include <linux/kexec.h>
>>  
>>  DECLARE_TRACE(kernel_printk,
>>  TPPROTO(unsigned long retaddr),
>> @@ -15,5 +16,14 @@ DECLARE_TRACE(kernel_module_free,
>>  DECLARE_TRACE(kernel_module_load,
>>  TPPROTO(struct module *mod),
>>  TPARGS(mod));
>> +DECLARE_TRACE(kernel_panic,
>> + TPPROTO(const char *fmt, va_list args),
>> + TPARGS(fmt, args));
>> +DECLARE_TRACE(kernel_kernel_kexec,
>> + TPPROTO(struct kimage *image),
>> + TPARGS(image));
>> +DECLARE_TRACE(kernel_crash_kexec,
>> + TPPROTO(struct kimage *image, struct pt_regs *regs),
>> + TPARGS(image, regs));
>>  
>>  #endif
>> diff --git a/kernel/kexec.c b/kernel/kexec.c
>> index aef2653..56cdaac 100644
>> --- a/kernel/kexec.c
>> +++ b/kernel/kexec.c
>> @@ -30,6 +30,7 @@
>>  #include <linux/pm.h>
>>  #include <linux/cpu.h>
>>  #include <linux/console.h>
>> +#include <trace/kernel.h>
>>  
>>  #include <asm/page.h>
>>  #include <asm/uaccess.h>
>> @@ -37,6 +38,9 @@
>>  #include <asm/system.h>
>>  #include <asm/sections.h>
>>  
>> +DEFINE_TRACE(kernel_kernel_kexec);
>> +DEFINE_TRACE(kernel_crash_kexec);
>> +
>>  /* Per cpu memory for storing cpu states in case of system crash. */
>>  note_buf_t* crash_notes;
>>  
>> @@ -1062,6 +1066,8 @@ asmlinkage long compat_sys_kexec_load(unsigned long entry,
>>  
>>  void crash_kexec(struct pt_regs *regs)
>>  {
>> + trace_kernel_crash_kexec(kexec_crash_image, regs);
>> +
>>  /* Take the kexec_mutex here to prevent sys_kexec_load
>>  * running on one cpu from replacing the crash kernel
>>  * we are using after a panic on a different cpu.
>> @@ -1428,6 +1434,8 @@ int kernel_kexec(void)
>>  {
>>  int error = 0;
>>  
>> + trace_kernel_kernel_kexec(kexec_image);
>> +
>>  if (!mutex_trylock(&kexec_mutex))
>>  return -EBUSY;
>>  if (!kexec_image) {
>> diff --git a/kernel/panic.c b/kernel/panic.c
>> index 12c5a0a..3fa642e 100644
>> --- a/kernel/panic.c
>> +++ b/kernel/panic.c
>> @@ -21,6 +21,9 @@
>>  #include <linux/debug_locks.h>
>>  #include <linux/random.h>
>>  #include <linux/kallsyms.h>
>> +#include <trace/kernel.h>
>> +
>> +DEFINE_TRACE(kernel_panic);
>>  
>>  int panic_on_oops;
>>  int tainted;
>> @@ -68,6 +71,10 @@ NORET_TYPE void panic(const char * fmt, ...)
>>  unsigned long caller = (unsigned long) __builtin_return_address(0);
>>  #endif
>>  
>> + va_start(args, fmt);
>> + trace_kernel_panic(fmt, args);
>> + va_end(args);
>> +
>>  /*
>>  * It's possible to come here directly from a panic-assertion and not
>>  * have preempt disabled. Some functions called from here want
>> -- 
>> 1.5.5.3
>> 
>> 
>> 
>> _______________________________________________
>> ltt-dev mailing list
>> ltt-dev at lists.casi.polymtl.ca
>> http://lists.casi.polymtl.ca/cgi-bin/mailman/listinfo/ltt-dev
>> 
> 
> -- 
> Mathieu Desnoyers
> OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68
> 
>

[ltt-dev] [PATCH v2] add tracepoints of panic and kexec

[Mathieu Desnoyers]

* Zhaolei (zhaolei at cn.fujitsu.com) wrote:
> * From: "Mathieu Desnoyers" <compudj@krystal.dyndns.org>
> >* Zhaolei (zhaolei at cn.fujitsu.com) wrote:
> >> Instrumentation of following panic and kexec related events are added:
> >> panic
> >> kernel_kexec
> >> crash_kexec
> >> 
> >> It is useful for build flight-recorder program based on lttng infrastructure.
> >> 
> > 
> > Hrm, I'm wondering if these are events we are interested to trace or if
> > you plan to use these tracepoints as hooks into the kernel to connect
> > the tracer infrastructure to it ?
> Hello, Mathieu
> 
> IMHO, both.
> 
> As a programmer who want to build  flight-recorder based on lttng infrastructure,
> he can:
> 1: use lttng trace, and get lttng's trace datas from kernel-dump file.
> or
> 2: use lttng's tracepoint, and record events himself.
> 
> But as a lttng's programmer, we can make both way possible.
> 
> What is your opinion?
> 

Yes, I think having both is good. So we could add, separately :

A hook in panic and kexec using include/linux/notifier.h so mechanisms
other than tracers can connect their hook to it.

Also add a tracepoint so tracers can use this as an event source.

I would do both in separate patches though, because they have different
purposes.

Best regards,

Mathieu


> B.R.
> Zhaolei
> > 
> > If you plan to use those as hooks into the kernel, we should probably
> > consider adding notifiers instead (notifier.h). This is the "blessed"
> > way to be called upon such events.
> > 
> > Mathieu
> > 
> >> Signed-off-by: Zhao Lei <zhaolei at cn.fujitsu.com>
> >> ---
> >>  include/trace/kernel.h |   10 ++++++++++
> >>  kernel/kexec.c         |    8 ++++++++
> >>  kernel/panic.c         |    7 +++++++
> >>  3 files changed, 25 insertions(+), 0 deletions(-)
> >> 
> >> diff --git a/include/trace/kernel.h b/include/trace/kernel.h
> >> index d2c3320..06d585f 100644
> >> --- a/include/trace/kernel.h
> >> +++ b/include/trace/kernel.h
> >> @@ -2,6 +2,7 @@
> >>  #define _TRACE_KERNEL_H
> >>  
> >>  #include <linux/tracepoint.h>
> >> +#include <linux/kexec.h>
> >>  
> >>  DECLARE_TRACE(kernel_printk,
> >>  TPPROTO(unsigned long retaddr),
> >> @@ -15,5 +16,14 @@ DECLARE_TRACE(kernel_module_free,
> >>  DECLARE_TRACE(kernel_module_load,
> >>  TPPROTO(struct module *mod),
> >>  TPARGS(mod));
> >> +DECLARE_TRACE(kernel_panic,
> >> + TPPROTO(const char *fmt, va_list args),
> >> + TPARGS(fmt, args));
> >> +DECLARE_TRACE(kernel_kernel_kexec,
> >> + TPPROTO(struct kimage *image),
> >> + TPARGS(image));
> >> +DECLARE_TRACE(kernel_crash_kexec,
> >> + TPPROTO(struct kimage *image, struct pt_regs *regs),
> >> + TPARGS(image, regs));
> >>  
> >>  #endif
> >> diff --git a/kernel/kexec.c b/kernel/kexec.c
> >> index aef2653..56cdaac 100644
> >> --- a/kernel/kexec.c
> >> +++ b/kernel/kexec.c
> >> @@ -30,6 +30,7 @@
> >>  #include <linux/pm.h>
> >>  #include <linux/cpu.h>
> >>  #include <linux/console.h>
> >> +#include <trace/kernel.h>
> >>  
> >>  #include <asm/page.h>
> >>  #include <asm/uaccess.h>
> >> @@ -37,6 +38,9 @@
> >>  #include <asm/system.h>
> >>  #include <asm/sections.h>
> >>  
> >> +DEFINE_TRACE(kernel_kernel_kexec);
> >> +DEFINE_TRACE(kernel_crash_kexec);
> >> +
> >>  /* Per cpu memory for storing cpu states in case of system crash. */
> >>  note_buf_t* crash_notes;
> >>  
> >> @@ -1062,6 +1066,8 @@ asmlinkage long compat_sys_kexec_load(unsigned long entry,
> >>  
> >>  void crash_kexec(struct pt_regs *regs)
> >>  {
> >> + trace_kernel_crash_kexec(kexec_crash_image, regs);
> >> +
> >>  /* Take the kexec_mutex here to prevent sys_kexec_load
> >>  * running on one cpu from replacing the crash kernel
> >>  * we are using after a panic on a different cpu.
> >> @@ -1428,6 +1434,8 @@ int kernel_kexec(void)
> >>  {
> >>  int error = 0;
> >>  
> >> + trace_kernel_kernel_kexec(kexec_image);
> >> +
> >>  if (!mutex_trylock(&kexec_mutex))
> >>  return -EBUSY;
> >>  if (!kexec_image) {
> >> diff --git a/kernel/panic.c b/kernel/panic.c
> >> index 12c5a0a..3fa642e 100644
> >> --- a/kernel/panic.c
> >> +++ b/kernel/panic.c
> >> @@ -21,6 +21,9 @@
> >>  #include <linux/debug_locks.h>
> >>  #include <linux/random.h>
> >>  #include <linux/kallsyms.h>
> >> +#include <trace/kernel.h>
> >> +
> >> +DEFINE_TRACE(kernel_panic);
> >>  
> >>  int panic_on_oops;
> >>  int tainted;
> >> @@ -68,6 +71,10 @@ NORET_TYPE void panic(const char * fmt, ...)
> >>  unsigned long caller = (unsigned long) __builtin_return_address(0);
> >>  #endif
> >>  
> >> + va_start(args, fmt);
> >> + trace_kernel_panic(fmt, args);
> >> + va_end(args);
> >> +
> >>  /*
> >>  * It's possible to come here directly from a panic-assertion and not
> >>  * have preempt disabled. Some functions called from here want
> >> -- 
> >> 1.5.5.3
> >> 
> >> 
> >> 
> >> _______________________________________________
> >> ltt-dev mailing list
> >> ltt-dev at lists.casi.polymtl.ca
> >> http://lists.casi.polymtl.ca/cgi-bin/mailman/listinfo/ltt-dev
> >> 
> > 
> > -- 
> > Mathieu Desnoyers
> > OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68
> > 
> >
> _______________________________________________
> ltt-dev mailing list
> ltt-dev at lists.casi.polymtl.ca
> http://lists.casi.polymtl.ca/cgi-bin/mailman/listinfo/ltt-dev
> 

-- 
Mathieu Desnoyers
OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68

[ltt-dev] [PATCH v2] add tracepoints of panic and kexec

[Zhaolei]

* From: "Mathieu Desnoyers" <compudj@krystal.dyndns.org>
>* Zhaolei (zhaolei at cn.fujitsu.com) wrote:
>> * From: "Mathieu Desnoyers" <compudj@krystal.dyndns.org>
>> >* Zhaolei (zhaolei at cn.fujitsu.com) wrote:
>> >> Instrumentation of following panic and kexec related events are added:
>> >> panic
>> >> kernel_kexec
>> >> crash_kexec
>> >> 
>> >> It is useful for build flight-recorder program based on lttng infrastructure.
>> >> 
>> > 
>> > Hrm, I'm wondering if these are events we are interested to trace or if
>> > you plan to use these tracepoints as hooks into the kernel to connect
>> > the tracer infrastructure to it ?
>> Hello, Mathieu
>> 
>> IMHO, both.
>> 
>> As a programmer who want to build  flight-recorder based on lttng infrastructure,
>> he can:
>> 1: use lttng trace, and get lttng's trace datas from kernel-dump file.
>> or
>> 2: use lttng's tracepoint, and record events himself.
>> 
>> But as a lttng's programmer, we can make both way possible.
>> 
>> What is your opinion?
>> 
> 
> Yes, I think having both is good. So we could add, separately :
> 
> A hook in panic and kexec using include/linux/notifier.h so mechanisms
> other than tracers can connect their hook to it.
> 
> Also add a tracepoint so tracers can use this as an event source.
> 
> I would do both in separate patches though, because they have different
> purposes.
Hello, Mathieu

Are you means you will do both in next version of lttng?
Thanks!

B.R
Zhaolei

> 
> Best regards,
> 
> Mathieu
> 
> 
>> B.R.
>> Zhaolei
>> > 
>> > If you plan to use those as hooks into the kernel, we should probably
>> > consider adding notifiers instead (notifier.h). This is the "blessed"
>> > way to be called upon such events.
>> > 
>> > Mathieu
>> > 
>> >> Signed-off-by: Zhao Lei <zhaolei at cn.fujitsu.com>
>> >> ---
>> >>  include/trace/kernel.h |   10 ++++++++++
>> >>  kernel/kexec.c         |    8 ++++++++
>> >>  kernel/panic.c         |    7 +++++++
>> >>  3 files changed, 25 insertions(+), 0 deletions(-)
>> >> 
>> >> diff --git a/include/trace/kernel.h b/include/trace/kernel.h
>> >> index d2c3320..06d585f 100644
>> >> --- a/include/trace/kernel.h
>> >> +++ b/include/trace/kernel.h
>> >> @@ -2,6 +2,7 @@
>> >>  #define _TRACE_KERNEL_H
>> >>  
>> >>  #include <linux/tracepoint.h>
>> >> +#include <linux/kexec.h>
>> >>  
>> >>  DECLARE_TRACE(kernel_printk,
>> >>  TPPROTO(unsigned long retaddr),
>> >> @@ -15,5 +16,14 @@ DECLARE_TRACE(kernel_module_free,
>> >>  DECLARE_TRACE(kernel_module_load,
>> >>  TPPROTO(struct module *mod),
>> >>  TPARGS(mod));
>> >> +DECLARE_TRACE(kernel_panic,
>> >> + TPPROTO(const char *fmt, va_list args),
>> >> + TPARGS(fmt, args));
>> >> +DECLARE_TRACE(kernel_kernel_kexec,
>> >> + TPPROTO(struct kimage *image),
>> >> + TPARGS(image));
>> >> +DECLARE_TRACE(kernel_crash_kexec,
>> >> + TPPROTO(struct kimage *image, struct pt_regs *regs),
>> >> + TPARGS(image, regs));
>> >>  
>> >>  #endif
>> >> diff --git a/kernel/kexec.c b/kernel/kexec.c
>> >> index aef2653..56cdaac 100644
>> >> --- a/kernel/kexec.c
>> >> +++ b/kernel/kexec.c
>> >> @@ -30,6 +30,7 @@
>> >>  #include <linux/pm.h>
>> >>  #include <linux/cpu.h>
>> >>  #include <linux/console.h>
>> >> +#include <trace/kernel.h>
>> >>  
>> >>  #include <asm/page.h>
>> >>  #include <asm/uaccess.h>
>> >> @@ -37,6 +38,9 @@
>> >>  #include <asm/system.h>
>> >>  #include <asm/sections.h>
>> >>  
>> >> +DEFINE_TRACE(kernel_kernel_kexec);
>> >> +DEFINE_TRACE(kernel_crash_kexec);
>> >> +
>> >>  /* Per cpu memory for storing cpu states in case of system crash. */
>> >>  note_buf_t* crash_notes;
>> >>  
>> >> @@ -1062,6 +1066,8 @@ asmlinkage long compat_sys_kexec_load(unsigned long entry,
>> >>  
>> >>  void crash_kexec(struct pt_regs *regs)
>> >>  {
>> >> + trace_kernel_crash_kexec(kexec_crash_image, regs);
>> >> +
>> >>  /* Take the kexec_mutex here to prevent sys_kexec_load
>> >>  * running on one cpu from replacing the crash kernel
>> >>  * we are using after a panic on a different cpu.
>> >> @@ -1428,6 +1434,8 @@ int kernel_kexec(void)
>> >>  {
>> >>  int error = 0;
>> >>  
>> >> + trace_kernel_kernel_kexec(kexec_image);
>> >> +
>> >>  if (!mutex_trylock(&kexec_mutex))
>> >>  return -EBUSY;
>> >>  if (!kexec_image) {
>> >> diff --git a/kernel/panic.c b/kernel/panic.c
>> >> index 12c5a0a..3fa642e 100644
>> >> --- a/kernel/panic.c
>> >> +++ b/kernel/panic.c
>> >> @@ -21,6 +21,9 @@
>> >>  #include <linux/debug_locks.h>
>> >>  #include <linux/random.h>
>> >>  #include <linux/kallsyms.h>
>> >> +#include <trace/kernel.h>
>> >> +
>> >> +DEFINE_TRACE(kernel_panic);
>> >>  
>> >>  int panic_on_oops;
>> >>  int tainted;
>> >> @@ -68,6 +71,10 @@ NORET_TYPE void panic(const char * fmt, ...)
>> >>  unsigned long caller = (unsigned long) __builtin_return_address(0);
>> >>  #endif
>> >>  
>> >> + va_start(args, fmt);
>> >> + trace_kernel_panic(fmt, args);
>> >> + va_end(args);
>> >> +
>> >>  /*
>> >>  * It's possible to come here directly from a panic-assertion and not
>> >>  * have preempt disabled. Some functions called from here want
>> >> -- 
>> >> 1.5.5.3
>> >> 
>> >> 
>> >> 
>> >> _______________________________________________
>> >> ltt-dev mailing list
>> >> ltt-dev at lists.casi.polymtl.ca
>> >> http://lists.casi.polymtl.ca/cgi-bin/mailman/listinfo/ltt-dev
>> >> 
>> > 
>> > -- 
>> > Mathieu Desnoyers
>> > OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68
>> > 
>> >
>> _______________________________________________
>> ltt-dev mailing list
>> ltt-dev at lists.casi.polymtl.ca
>> http://lists.casi.polymtl.ca/cgi-bin/mailman/listinfo/ltt-dev
>> 
> 
> -- 
> Mathieu Desnoyers
> OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68
>

[ltt-dev] [PATCH v2] add tracepoints of panic and kexec

[Mathieu Desnoyers]

* Zhaolei (zhaolei at cn.fujitsu.com) wrote:
> * From: "Mathieu Desnoyers" <compudj@krystal.dyndns.org>
> >* Zhaolei (zhaolei at cn.fujitsu.com) wrote:
> >> * From: "Mathieu Desnoyers" <compudj@krystal.dyndns.org>
> >> >* Zhaolei (zhaolei at cn.fujitsu.com) wrote:
> >> >> Instrumentation of following panic and kexec related events are added:
> >> >> panic
> >> >> kernel_kexec
> >> >> crash_kexec
> >> >> 
> >> >> It is useful for build flight-recorder program based on lttng infrastructure.
> >> >> 
> >> > 
> >> > Hrm, I'm wondering if these are events we are interested to trace or if
> >> > you plan to use these tracepoints as hooks into the kernel to connect
> >> > the tracer infrastructure to it ?
> >> Hello, Mathieu
> >> 
> >> IMHO, both.
> >> 
> >> As a programmer who want to build  flight-recorder based on lttng infrastructure,
> >> he can:
> >> 1: use lttng trace, and get lttng's trace datas from kernel-dump file.
> >> or
> >> 2: use lttng's tracepoint, and record events himself.
> >> 
> >> But as a lttng's programmer, we can make both way possible.
> >> 
> >> What is your opinion?
> >> 
> > 
> > Yes, I think having both is good. So we could add, separately :
> > 
> > A hook in panic and kexec using include/linux/notifier.h so mechanisms
> > other than tracers can connect their hook to it.
> > 
> > Also add a tracepoint so tracers can use this as an event source.
> > 
> > I would do both in separate patches though, because they have different
> > purposes.
> Hello, Mathieu
> 
> Are you means you will do both in next version of lttng?
> Thanks!
> 

Hi Zhaolei,

I won't create the patches myself because I don't see it as a priority
(Linus said he would refuse any instrumentation patch before we get the
data output mechanism into the kernel), but if someone provides patches
to add notifier feature to kexec and panic so we can plug LTTng into
them to extract traces from a crashed kernel, I would be very interested
to add this. OTOH, about the tracepoints, I am not convinced they are
necessary for now, because we already instrument printk which logs some
information on panic(). Maybe kexec would be better suited for
instrumentation.

Also, it would be good to provide the LTTng probes for the tracepoints
you add in a separate patch within the same patchset
(ltt/probes/*-trace.c).

Thanks,

Mathieu

> B.R
> Zhaolei
> 
> > 
> > Best regards,
> > 
> > Mathieu
> > 
> > 
> >> B.R.
> >> Zhaolei
> >> > 
> >> > If you plan to use those as hooks into the kernel, we should probably
> >> > consider adding notifiers instead (notifier.h). This is the "blessed"
> >> > way to be called upon such events.
> >> > 
> >> > Mathieu
> >> > 
> >> >> Signed-off-by: Zhao Lei <zhaolei at cn.fujitsu.com>
> >> >> ---
> >> >>  include/trace/kernel.h |   10 ++++++++++
> >> >>  kernel/kexec.c         |    8 ++++++++
> >> >>  kernel/panic.c         |    7 +++++++
> >> >>  3 files changed, 25 insertions(+), 0 deletions(-)
> >> >> 
> >> >> diff --git a/include/trace/kernel.h b/include/trace/kernel.h
> >> >> index d2c3320..06d585f 100644
> >> >> --- a/include/trace/kernel.h
> >> >> +++ b/include/trace/kernel.h
> >> >> @@ -2,6 +2,7 @@
> >> >>  #define _TRACE_KERNEL_H
> >> >>  
> >> >>  #include <linux/tracepoint.h>
> >> >> +#include <linux/kexec.h>
> >> >>  
> >> >>  DECLARE_TRACE(kernel_printk,
> >> >>  TPPROTO(unsigned long retaddr),
> >> >> @@ -15,5 +16,14 @@ DECLARE_TRACE(kernel_module_free,
> >> >>  DECLARE_TRACE(kernel_module_load,
> >> >>  TPPROTO(struct module *mod),
> >> >>  TPARGS(mod));
> >> >> +DECLARE_TRACE(kernel_panic,
> >> >> + TPPROTO(const char *fmt, va_list args),
> >> >> + TPARGS(fmt, args));
> >> >> +DECLARE_TRACE(kernel_kernel_kexec,
> >> >> + TPPROTO(struct kimage *image),
> >> >> + TPARGS(image));
> >> >> +DECLARE_TRACE(kernel_crash_kexec,
> >> >> + TPPROTO(struct kimage *image, struct pt_regs *regs),
> >> >> + TPARGS(image, regs));
> >> >>  
> >> >>  #endif
> >> >> diff --git a/kernel/kexec.c b/kernel/kexec.c
> >> >> index aef2653..56cdaac 100644
> >> >> --- a/kernel/kexec.c
> >> >> +++ b/kernel/kexec.c
> >> >> @@ -30,6 +30,7 @@
> >> >>  #include <linux/pm.h>
> >> >>  #include <linux/cpu.h>
> >> >>  #include <linux/console.h>
> >> >> +#include <trace/kernel.h>
> >> >>  
> >> >>  #include <asm/page.h>
> >> >>  #include <asm/uaccess.h>
> >> >> @@ -37,6 +38,9 @@
> >> >>  #include <asm/system.h>
> >> >>  #include <asm/sections.h>
> >> >>  
> >> >> +DEFINE_TRACE(kernel_kernel_kexec);
> >> >> +DEFINE_TRACE(kernel_crash_kexec);
> >> >> +
> >> >>  /* Per cpu memory for storing cpu states in case of system crash. */
> >> >>  note_buf_t* crash_notes;
> >> >>  
> >> >> @@ -1062,6 +1066,8 @@ asmlinkage long compat_sys_kexec_load(unsigned long entry,
> >> >>  
> >> >>  void crash_kexec(struct pt_regs *regs)
> >> >>  {
> >> >> + trace_kernel_crash_kexec(kexec_crash_image, regs);
> >> >> +
> >> >>  /* Take the kexec_mutex here to prevent sys_kexec_load
> >> >>  * running on one cpu from replacing the crash kernel
> >> >>  * we are using after a panic on a different cpu.
> >> >> @@ -1428,6 +1434,8 @@ int kernel_kexec(void)
> >> >>  {
> >> >>  int error = 0;
> >> >>  
> >> >> + trace_kernel_kernel_kexec(kexec_image);
> >> >> +
> >> >>  if (!mutex_trylock(&kexec_mutex))
> >> >>  return -EBUSY;
> >> >>  if (!kexec_image) {
> >> >> diff --git a/kernel/panic.c b/kernel/panic.c
> >> >> index 12c5a0a..3fa642e 100644
> >> >> --- a/kernel/panic.c
> >> >> +++ b/kernel/panic.c
> >> >> @@ -21,6 +21,9 @@
> >> >>  #include <linux/debug_locks.h>
> >> >>  #include <linux/random.h>
> >> >>  #include <linux/kallsyms.h>
> >> >> +#include <trace/kernel.h>
> >> >> +
> >> >> +DEFINE_TRACE(kernel_panic);
> >> >>  
> >> >>  int panic_on_oops;
> >> >>  int tainted;
> >> >> @@ -68,6 +71,10 @@ NORET_TYPE void panic(const char * fmt, ...)
> >> >>  unsigned long caller = (unsigned long) __builtin_return_address(0);
> >> >>  #endif
> >> >>  
> >> >> + va_start(args, fmt);
> >> >> + trace_kernel_panic(fmt, args);
> >> >> + va_end(args);
> >> >> +
> >> >>  /*
> >> >>  * It's possible to come here directly from a panic-assertion and not
> >> >>  * have preempt disabled. Some functions called from here want
> >> >> -- 
> >> >> 1.5.5.3
> >> >> 
> >> >> 
> >> >> 
> >> >> _______________________________________________
> >> >> ltt-dev mailing list
> >> >> ltt-dev at lists.casi.polymtl.ca
> >> >> http://lists.casi.polymtl.ca/cgi-bin/mailman/listinfo/ltt-dev
> >> >> 
> >> > 
> >> > -- 
> >> > Mathieu Desnoyers
> >> > OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68
> >> > 
> >> >
> >> _______________________________________________
> >> ltt-dev mailing list
> >> ltt-dev at lists.casi.polymtl.ca
> >> http://lists.casi.polymtl.ca/cgi-bin/mailman/listinfo/ltt-dev
> >> 
> > 
> > -- 
> > Mathieu Desnoyers
> > OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68
> >
> _______________________________________________
> ltt-dev mailing list
> ltt-dev at lists.casi.polymtl.ca
> http://lists.casi.polymtl.ca/cgi-bin/mailman/listinfo/ltt-dev
> 

-- 
Mathieu Desnoyers
OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68

[ltt-dev] [PATCH v2] add tracepoints of panic and kexec

[Lai Jiangshan]

Mathieu Desnoyers wrote:
> * Zhaolei (zhaolei at cn.fujitsu.com) wrote:
>> * From: "Mathieu Desnoyers" <compudj@krystal.dyndns.org>
>>> * Zhaolei (zhaolei at cn.fujitsu.com) wrote:
>>>> * From: "Mathieu Desnoyers" <compudj@krystal.dyndns.org>
>>>>> * Zhaolei (zhaolei at cn.fujitsu.com) wrote:
>>>>>> Instrumentation of following panic and kexec related events are added:
>>>>>> panic
>>>>>> kernel_kexec
>>>>>> crash_kexec
>>>>>>
>>>>>> It is useful for build flight-recorder program based on lttng infrastructure.
>>>>>>
>>>>> Hrm, I'm wondering if these are events we are interested to trace or if
>>>>> you plan to use these tracepoints as hooks into the kernel to connect
>>>>> the tracer infrastructure to it ?
>>>> Hello, Mathieu
>>>>
>>>> IMHO, both.
>>>>
>>>> As a programmer who want to build  flight-recorder based on lttng infrastructure,
>>>> he can:
>>>> 1: use lttng trace, and get lttng's trace datas from kernel-dump file.
>>>> or
>>>> 2: use lttng's tracepoint, and record events himself.
>>>>
>>>> But as a lttng's programmer, we can make both way possible.
>>>>
>>>> What is your opinion?
>>>>
>>> Yes, I think having both is good. So we could add, separately :
>>>
>>> A hook in panic and kexec using include/linux/notifier.h so mechanisms
>>> other than tracers can connect their hook to it.
>>>
>>> Also add a tracepoint so tracers can use this as an event source.
>>>
>>> I would do both in separate patches though, because they have different
>>> purposes.
>> Hello, Mathieu
>>
>> Are you means you will do both in next version of lttng?
>> Thanks!
>>
> 
> Hi Zhaolei,
> 
> I won't create the patches myself because I don't see it as a priority
> (Linus said he would refuse any instrumentation patch before we get the
> data output mechanism into the kernel), but if someone provides patches

Hi, Mathieu,

You *forgot* one thing: we are lttng *users*, not only developers.
I don't think it's good that you just heard the God only.

lttng is used by her users at the end. User's need is one of the
first class things to concern.

> to add notifier feature to kexec and panic so we can plug LTTng into
> them to extract traces from a crashed kernel, I would be very interested
> to add this. OTOH, about the tracepoints, I am not convinced they are
> necessary for now, because we already instrument printk which logs some
> information on panic(). Maybe kexec would be better suited for
> instrumentation.

We need to know the time and other information that panic happened.
And output message is integrated in lttng with other messages.
I don't think it's bad idea which we use for a long time.

And this is a *key message*, event should be generated efficient/elegant.

Sorry for participated in this discuss with bad manners.

Thanks, Lai.

> 
> Also, it would be good to provide the LTTng probes for the tracepoints
> you add in a separate patch within the same patchset
> (ltt/probes/*-trace.c).
> 
> Thanks,
> 
> Mathieu
> 

[ltt-dev] [PATCH v2] add tracepoints of panic and kexec

[Mathieu Desnoyers]

* Lai Jiangshan (laijs at cn.fujitsu.com) wrote:
> Mathieu Desnoyers wrote:
> > * Zhaolei (zhaolei at cn.fujitsu.com) wrote:
> >> * From: "Mathieu Desnoyers" <compudj@krystal.dyndns.org>
> >>> * Zhaolei (zhaolei at cn.fujitsu.com) wrote:
> >>>> * From: "Mathieu Desnoyers" <compudj@krystal.dyndns.org>
> >>>>> * Zhaolei (zhaolei at cn.fujitsu.com) wrote:
> >>>>>> Instrumentation of following panic and kexec related events are added:
> >>>>>> panic
> >>>>>> kernel_kexec
> >>>>>> crash_kexec
> >>>>>>
> >>>>>> It is useful for build flight-recorder program based on lttng infrastructure.
> >>>>>>
> >>>>> Hrm, I'm wondering if these are events we are interested to trace or if
> >>>>> you plan to use these tracepoints as hooks into the kernel to connect
> >>>>> the tracer infrastructure to it ?
> >>>> Hello, Mathieu
> >>>>
> >>>> IMHO, both.
> >>>>
> >>>> As a programmer who want to build  flight-recorder based on lttng infrastructure,
> >>>> he can:
> >>>> 1: use lttng trace, and get lttng's trace datas from kernel-dump file.
> >>>> or
> >>>> 2: use lttng's tracepoint, and record events himself.
> >>>>
> >>>> But as a lttng's programmer, we can make both way possible.
> >>>>
> >>>> What is your opinion?
> >>>>
> >>> Yes, I think having both is good. So we could add, separately :
> >>>
> >>> A hook in panic and kexec using include/linux/notifier.h so mechanisms
> >>> other than tracers can connect their hook to it.
> >>>
> >>> Also add a tracepoint so tracers can use this as an event source.
> >>>
> >>> I would do both in separate patches though, because they have different
> >>> purposes.
> >> Hello, Mathieu
> >>
> >> Are you means you will do both in next version of lttng?
> >> Thanks!
> >>
> > 
> > Hi Zhaolei,
> > 
> > I won't create the patches myself because I don't see it as a priority
> > (Linus said he would refuse any instrumentation patch before we get the
> > data output mechanism into the kernel), but if someone provides patches
> 
> Hi, Mathieu,
> 
> You *forgot* one thing: we are lttng *users*, not only developers.
> I don't think it's good that you just heard the God only.
> 
> lttng is used by her users at the end. User's need is one of the
> first class things to concern.
> 
> > to add notifier feature to kexec and panic so we can plug LTTng into
> > them to extract traces from a crashed kernel, I would be very interested
> > to add this. OTOH, about the tracepoints, I am not convinced they are
> > necessary for now, because we already instrument printk which logs some
> > information on panic(). Maybe kexec would be better suited for
> > instrumentation.
> 
> We need to know the time and other information that panic happened.
> And output message is integrated in lttng with other messages.
> I don't think it's bad idea which we use for a long time.
> 
> And this is a *key message*, event should be generated efficient/elegant.
> 
> Sorry for participated in this discuss with bad manners.
> 

No no, that's ok :) Well, if the panic event is useful, I have nothing
against it. I just think that if the tracer starts hooking in some parts
of the kernel to do something else than just logging events (e.g.
triggering a full dump), then we should think about using the standard
notifiers interface (mabe in addition to tracepoints).

And about the "I won't create the patches myself", it's just a matter of
priority. I focus on mainline integration currently, but I have nothing
against integrating work done by others in the LTTng project. I hope
this clarifies my point of view a bit.

Mathieu

> Thanks, Lai.
> 
> > 
> > Also, it would be good to provide the LTTng probes for the tracepoints
> > you add in a separate patch within the same patchset
> > (ltt/probes/*-trace.c).
> > 
> > Thanks,
> > 
> > Mathieu
> > 
> 

-- 
Mathieu Desnoyers
OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68

[ltt-dev] [PATCH v2] add tracepoints of panic and kexec

[KOSAKI Motohiro]

Hi Mathieu,

> > We need to know the time and other information that panic happened.
> > And output message is integrated in lttng with other messages.
> > I don't think it's bad idea which we use for a long time.
> > 
> > And this is a *key message*, event should be generated efficient/elegant.
> > 
> > Sorry for participated in this discuss with bad manners.
> > 
> 
> No no, that's ok :) Well, if the panic event is useful, I have nothing
> against it. I just think that if the tracer starts hooking in some parts
> of the kernel to do something else than just logging events (e.g.
> triggering a full dump), then we should think about using the standard
> notifiers interface (mabe in addition to tracepoints).
> 
> And about the "I won't create the patches myself", it's just a matter of
> priority. I focus on mainline integration currently, but I have nothing
> against integrating work done by others in the LTTng project. I hope
> this clarifies my point of view a bit.
> 
> Mathieu

I read this thread and feel interesting.
Thank you mathiu, good explaination.

I agree to Fujitsu have the responsibility to make panic prove consumer.
(eg. integrate lttng directly or generic notifier mechanism)

ok, we receive this responcibility ball.


and, I'd like to talk about fujitsu tracer development team goal.
We also strongly want to merge lttng to upstream. we agree it's top
priority.
and our next priority, we need good flight recorder supporting.
  - low overhead
  - transparent to end-user
    (vendor want to turn on, by default)
  - easy analysis to the support engineer of vendor (include fujitsu).
  et al.

in flight recorder usage, we need to ignore the event of happend
after panic() function.
I believe it's general requirement of flight recorder, not fujitsu 
specific requirement.

and ok, we integrate this patch to lttng after upstream merge.


last week, gui-san did hear your development plan.
I and gui-san is making the plan of fujitsu short term and long term
development now.
So, periodically development plain dumping and/or it's your priority
explanation are very helpful for us, it help to adjust our human resource.

Thanks!



btw, your "[Regression] High latency when doing large I/O" thread is
very interesting and good explain why lttng is useful for lkml guys.
I plan to demonstrate similar thing on MM area. that's my homework.


  - kosaki "double as fujitsu tracing tech-lead/manager as well as MM developer"

[ltt-dev] [PATCH v2] add tracepoints of panic and kexec

[Mathieu Desnoyers]

* KOSAKI Motohiro (kosaki.motohiro at jp.fujitsu.com) wrote:
> Hi Mathieu,
> 
> > > We need to know the time and other information that panic happened.
> > > And output message is integrated in lttng with other messages.
> > > I don't think it's bad idea which we use for a long time.
> > > 
> > > And this is a *key message*, event should be generated efficient/elegant.
> > > 
> > > Sorry for participated in this discuss with bad manners.
> > > 
> > 
> > No no, that's ok :) Well, if the panic event is useful, I have nothing
> > against it. I just think that if the tracer starts hooking in some parts
> > of the kernel to do something else than just logging events (e.g.
> > triggering a full dump), then we should think about using the standard
> > notifiers interface (mabe in addition to tracepoints).
> > 
> > And about the "I won't create the patches myself", it's just a matter of
> > priority. I focus on mainline integration currently, but I have nothing
> > against integrating work done by others in the LTTng project. I hope
> > this clarifies my point of view a bit.
> > 
> > Mathieu
> 
> I read this thread and feel interesting.
> Thank you mathiu, good explaination.
> 
> I agree to Fujitsu have the responsibility to make panic prove consumer.
> (eg. integrate lttng directly or generic notifier mechanism)
> 
> ok, we receive this responcibility ball.
> 
> 
> and, I'd like to talk about fujitsu tracer development team goal.
> We also strongly want to merge lttng to upstream. we agree it's top
> priority.
> and our next priority, we need good flight recorder supporting.
>   - low overhead
>   - transparent to end-user
>     (vendor want to turn on, by default)
>   - easy analysis to the support engineer of vendor (include fujitsu).
>   et al.
> 
> in flight recorder usage, we need to ignore the event of happend
> after panic() function.
> I believe it's general requirement of flight recorder, not fujitsu 
> specific requirement.
> 

I see. This is where you need to have an event saying "the kernel
stopped here". I'd prefer to leave tracing "on" at that point, just in
case something interesting happens within the panic() handler or after
it has run. Therefore having a "panic" tracepoint makes sense.

> and ok, we integrate this patch to lttng after upstream merge.
> 

I'm ok to integrate it now into the LTTng tree. I'll push the patches
upstream piece-by-piece anyway (mostly for instrumentation). The thing I
would like, in addition to a panic tracepoint, would be to add a panic()
notifier in the linux kernel. This notifier chain could call into LTTng
so we could do whatever action we have to do on the trace buffers (e.g.
to send them remotely over the network). Do you think this would be
useful ? Or maybe is there already something avalailable more widely
that we could hook into am I not fully aware of ?

> 
> last week, gui-san did hear your development plan.
> I and gui-san is making the plan of fujitsu short term and long term
> development now.
> So, periodically development plain dumping and/or it's your priority
> explanation are very helpful for us, it help to adjust our human resource.
> 

That's great. I really like the way things are moving forward. It's a
pleasure to work with you guys !

> Thanks!
> 
> 
> 
> btw, your "[Regression] High latency when doing large I/O" thread is
> very interesting and good explain why lttng is useful for lkml guys.
> I plan to demonstrate similar thing on MM area. that's my homework.
> 

Yes, I happen to learn some I/O scheduler basics at the same time. These
are some very interesting problems to tackle, and I hope it will
demonstrate the tracer usefulness to the community. Please CC me on such
MM problems, it's always a lot of fun. :)

> 
>   - kosaki "double as fujitsu tracing tech-lead/manager as well as MM developer"
> 

Mathieu, LTTng lead trying to write a Ph.D. thesis :)


> 

-- 
Mathieu Desnoyers
OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68

[ltt-dev] [PATCH v2] add tracepoints of panic and kexec

[Mathieu Desnoyers]

* Zhaolei (zhaolei at cn.fujitsu.com) wrote:
> * From: "Mathieu Desnoyers" <compudj@krystal.dyndns.org>
> >* Zhaolei (zhaolei at cn.fujitsu.com) wrote:
> >> Instrumentation of following panic and kexec related events are added:
> >> panic
> >> kernel_kexec
> >> crash_kexec
> >> 
> >> It is useful for build flight-recorder program based on lttng infrastructure.
> >> 
> > 
> > Hrm, I'm wondering if these are events we are interested to trace or if
> > you plan to use these tracepoints as hooks into the kernel to connect
> > the tracer infrastructure to it ?
> Hello, Mathieu
> 
> IMHO, both.
> 
> As a programmer who want to build  flight-recorder based on lttng infrastructure,
> he can:
> 1: use lttng trace, and get lttng's trace datas from kernel-dump file.
> or
> 2: use lttng's tracepoint, and record events himself.
> 
> But as a lttng's programmer, we can make both way possible.
> 
> What is your opinion?
> 

Hi Zhaolei,

I think having the tracepoints for panic and kexec would be a good
start. Then whenever we need more specific hooks to send the data out
(for crash data extraction), we can add notification hooks at that time.
Do you have an updated version of panic+kexec instrumentation ?

Thanks,

Mathieu

> B.R.
> Zhaolei
> > 
> > If you plan to use those as hooks into the kernel, we should probably
> > consider adding notifiers instead (notifier.h). This is the "blessed"
> > way to be called upon such events.
> > 
> > Mathieu
> > 
> >> Signed-off-by: Zhao Lei <zhaolei at cn.fujitsu.com>
> >> ---
> >>  include/trace/kernel.h |   10 ++++++++++
> >>  kernel/kexec.c         |    8 ++++++++
> >>  kernel/panic.c         |    7 +++++++
> >>  3 files changed, 25 insertions(+), 0 deletions(-)
> >> 
> >> diff --git a/include/trace/kernel.h b/include/trace/kernel.h
> >> index d2c3320..06d585f 100644
> >> --- a/include/trace/kernel.h
> >> +++ b/include/trace/kernel.h
> >> @@ -2,6 +2,7 @@
> >>  #define _TRACE_KERNEL_H
> >>  
> >>  #include <linux/tracepoint.h>
> >> +#include <linux/kexec.h>
> >>  
> >>  DECLARE_TRACE(kernel_printk,
> >>  TPPROTO(unsigned long retaddr),
> >> @@ -15,5 +16,14 @@ DECLARE_TRACE(kernel_module_free,
> >>  DECLARE_TRACE(kernel_module_load,
> >>  TPPROTO(struct module *mod),
> >>  TPARGS(mod));
> >> +DECLARE_TRACE(kernel_panic,
> >> + TPPROTO(const char *fmt, va_list args),
> >> + TPARGS(fmt, args));
> >> +DECLARE_TRACE(kernel_kernel_kexec,
> >> + TPPROTO(struct kimage *image),
> >> + TPARGS(image));
> >> +DECLARE_TRACE(kernel_crash_kexec,
> >> + TPPROTO(struct kimage *image, struct pt_regs *regs),
> >> + TPARGS(image, regs));
> >>  
> >>  #endif
> >> diff --git a/kernel/kexec.c b/kernel/kexec.c
> >> index aef2653..56cdaac 100644
> >> --- a/kernel/kexec.c
> >> +++ b/kernel/kexec.c
> >> @@ -30,6 +30,7 @@
> >>  #include <linux/pm.h>
> >>  #include <linux/cpu.h>
> >>  #include <linux/console.h>
> >> +#include <trace/kernel.h>
> >>  
> >>  #include <asm/page.h>
> >>  #include <asm/uaccess.h>
> >> @@ -37,6 +38,9 @@
> >>  #include <asm/system.h>
> >>  #include <asm/sections.h>
> >>  
> >> +DEFINE_TRACE(kernel_kernel_kexec);
> >> +DEFINE_TRACE(kernel_crash_kexec);
> >> +
> >>  /* Per cpu memory for storing cpu states in case of system crash. */
> >>  note_buf_t* crash_notes;
> >>  
> >> @@ -1062,6 +1066,8 @@ asmlinkage long compat_sys_kexec_load(unsigned long entry,
> >>  
> >>  void crash_kexec(struct pt_regs *regs)
> >>  {
> >> + trace_kernel_crash_kexec(kexec_crash_image, regs);
> >> +
> >>  /* Take the kexec_mutex here to prevent sys_kexec_load
> >>  * running on one cpu from replacing the crash kernel
> >>  * we are using after a panic on a different cpu.
> >> @@ -1428,6 +1434,8 @@ int kernel_kexec(void)
> >>  {
> >>  int error = 0;
> >>  
> >> + trace_kernel_kernel_kexec(kexec_image);
> >> +
> >>  if (!mutex_trylock(&kexec_mutex))
> >>  return -EBUSY;
> >>  if (!kexec_image) {
> >> diff --git a/kernel/panic.c b/kernel/panic.c
> >> index 12c5a0a..3fa642e 100644
> >> --- a/kernel/panic.c
> >> +++ b/kernel/panic.c
> >> @@ -21,6 +21,9 @@
> >>  #include <linux/debug_locks.h>
> >>  #include <linux/random.h>
> >>  #include <linux/kallsyms.h>
> >> +#include <trace/kernel.h>
> >> +
> >> +DEFINE_TRACE(kernel_panic);
> >>  
> >>  int panic_on_oops;
> >>  int tainted;
> >> @@ -68,6 +71,10 @@ NORET_TYPE void panic(const char * fmt, ...)
> >>  unsigned long caller = (unsigned long) __builtin_return_address(0);
> >>  #endif
> >>  
> >> + va_start(args, fmt);
> >> + trace_kernel_panic(fmt, args);
> >> + va_end(args);
> >> +
> >>  /*
> >>  * It's possible to come here directly from a panic-assertion and not
> >>  * have preempt disabled. Some functions called from here want
> >> -- 
> >> 1.5.5.3
> >> 
> >> 
> >> 
> >> _______________________________________________
> >> ltt-dev mailing list
> >> ltt-dev at lists.casi.polymtl.ca
> >> http://lists.casi.polymtl.ca/cgi-bin/mailman/listinfo/ltt-dev
> >> 
> > 
> > -- 
> > Mathieu Desnoyers
> > OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68
> > 
> >
> 

-- 
Mathieu Desnoyers
OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68

[ltt-dev] [PATCH v2] add tracepoints of panic and kexec

[Zhaolei]

* From: "Mathieu Desnoyers" <mathieu.desnoyers@polymtl.ca>
>* Zhaolei (zhaolei at cn.fujitsu.com) wrote:
>> * From: "Mathieu Desnoyers" <compudj@krystal.dyndns.org>
>> >* Zhaolei (zhaolei at cn.fujitsu.com) wrote:
>> >> Instrumentation of following panic and kexec related events are added:
>> >> panic
>> >> kernel_kexec
>> >> crash_kexec
>> >> 
>> >> It is useful for build flight-recorder program based on lttng infrastructure.
>> >> 
>> > 
>> > Hrm, I'm wondering if these are events we are interested to trace or if
>> > you plan to use these tracepoints as hooks into the kernel to connect
>> > the tracer infrastructure to it ?
>> Hello, Mathieu
>> 
>> IMHO, both.
>> 
>> As a programmer who want to build  flight-recorder based on lttng infrastructure,
>> he can:
>> 1: use lttng trace, and get lttng's trace datas from kernel-dump file.
>> or
>> 2: use lttng's tracepoint, and record events himself.
>> 
>> But as a lttng's programmer, we can make both way possible.
>> 
>> What is your opinion?
>> 
> 
> Hi Zhaolei,
> 
> I think having the tracepoints for panic and kexec would be a good
> start. 
Hello, Mathieu

Glad to hear that.

> Then whenever we need more specific hooks to send the data out
> (for crash data extraction), we can add notification hooks at that time.
> Do you have an updated version of panic+kexec instrumentation ?
Do you means update this patch to top of lttng-git-tree only?

B.R.
Zhaolei

> 
> Thanks,
> 
> Mathieu
> 
>> B.R.
>> Zhaolei
>> > 
>> > If you plan to use those as hooks into the kernel, we should probably
>> > consider adding notifiers instead (notifier.h). This is the "blessed"
>> > way to be called upon such events.
>> > 
>> > Mathieu
>> > 
>> >> Signed-off-by: Zhao Lei <zhaolei at cn.fujitsu.com>
>> >> ---
>> >>  include/trace/kernel.h |   10 ++++++++++
>> >>  kernel/kexec.c         |    8 ++++++++
>> >>  kernel/panic.c         |    7 +++++++
>> >>  3 files changed, 25 insertions(+), 0 deletions(-)
>> >> 
>> >> diff --git a/include/trace/kernel.h b/include/trace/kernel.h
>> >> index d2c3320..06d585f 100644
>> >> --- a/include/trace/kernel.h
>> >> +++ b/include/trace/kernel.h
>> >> @@ -2,6 +2,7 @@
>> >>  #define _TRACE_KERNEL_H
>> >>  
>> >>  #include <linux/tracepoint.h>
>> >> +#include <linux/kexec.h>
>> >>  
>> >>  DECLARE_TRACE(kernel_printk,
>> >>  TPPROTO(unsigned long retaddr),
>> >> @@ -15,5 +16,14 @@ DECLARE_TRACE(kernel_module_free,
>> >>  DECLARE_TRACE(kernel_module_load,
>> >>  TPPROTO(struct module *mod),
>> >>  TPARGS(mod));
>> >> +DECLARE_TRACE(kernel_panic,
>> >> + TPPROTO(const char *fmt, va_list args),
>> >> + TPARGS(fmt, args));
>> >> +DECLARE_TRACE(kernel_kernel_kexec,
>> >> + TPPROTO(struct kimage *image),
>> >> + TPARGS(image));
>> >> +DECLARE_TRACE(kernel_crash_kexec,
>> >> + TPPROTO(struct kimage *image, struct pt_regs *regs),
>> >> + TPARGS(image, regs));
>> >>  
>> >>  #endif
>> >> diff --git a/kernel/kexec.c b/kernel/kexec.c
>> >> index aef2653..56cdaac 100644
>> >> --- a/kernel/kexec.c
>> >> +++ b/kernel/kexec.c
>> >> @@ -30,6 +30,7 @@
>> >>  #include <linux/pm.h>
>> >>  #include <linux/cpu.h>
>> >>  #include <linux/console.h>
>> >> +#include <trace/kernel.h>
>> >>  
>> >>  #include <asm/page.h>
>> >>  #include <asm/uaccess.h>
>> >> @@ -37,6 +38,9 @@
>> >>  #include <asm/system.h>
>> >>  #include <asm/sections.h>
>> >>  
>> >> +DEFINE_TRACE(kernel_kernel_kexec);
>> >> +DEFINE_TRACE(kernel_crash_kexec);
>> >> +
>> >>  /* Per cpu memory for storing cpu states in case of system crash. */
>> >>  note_buf_t* crash_notes;
>> >>  
>> >> @@ -1062,6 +1066,8 @@ asmlinkage long compat_sys_kexec_load(unsigned long entry,
>> >>  
>> >>  void crash_kexec(struct pt_regs *regs)
>> >>  {
>> >> + trace_kernel_crash_kexec(kexec_crash_image, regs);
>> >> +
>> >>  /* Take the kexec_mutex here to prevent sys_kexec_load
>> >>  * running on one cpu from replacing the crash kernel
>> >>  * we are using after a panic on a different cpu.
>> >> @@ -1428,6 +1434,8 @@ int kernel_kexec(void)
>> >>  {
>> >>  int error = 0;
>> >>  
>> >> + trace_kernel_kernel_kexec(kexec_image);
>> >> +
>> >>  if (!mutex_trylock(&kexec_mutex))
>> >>  return -EBUSY;
>> >>  if (!kexec_image) {
>> >> diff --git a/kernel/panic.c b/kernel/panic.c
>> >> index 12c5a0a..3fa642e 100644
>> >> --- a/kernel/panic.c
>> >> +++ b/kernel/panic.c
>> >> @@ -21,6 +21,9 @@
>> >>  #include <linux/debug_locks.h>
>> >>  #include <linux/random.h>
>> >>  #include <linux/kallsyms.h>
>> >> +#include <trace/kernel.h>
>> >> +
>> >> +DEFINE_TRACE(kernel_panic);
>> >>  
>> >>  int panic_on_oops;
>> >>  int tainted;
>> >> @@ -68,6 +71,10 @@ NORET_TYPE void panic(const char * fmt, ...)
>> >>  unsigned long caller = (unsigned long) __builtin_return_address(0);
>> >>  #endif
>> >>  
>> >> + va_start(args, fmt);
>> >> + trace_kernel_panic(fmt, args);
>> >> + va_end(args);
>> >> +
>> >>  /*
>> >>  * It's possible to come here directly from a panic-assertion and not
>> >>  * have preempt disabled. Some functions called from here want
>> >> -- 
>> >> 1.5.5.3
>> >> 
>> >> 
>> >> 
>> >> _______________________________________________
>> >> ltt-dev mailing list
>> >> ltt-dev at lists.casi.polymtl.ca
>> >> http://lists.casi.polymtl.ca/cgi-bin/mailman/listinfo/ltt-dev
>> >> 
>> > 
>> > -- 
>> > Mathieu Desnoyers
>> > OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68
>> > 
>> >
>> 
> 
> -- 
> Mathieu Desnoyers
> OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68
> 
>

[ltt-dev] [PATCH v2] add tracepoints of panic and kexec

[Mathieu Desnoyers]

* Zhaolei (zhaolei at cn.fujitsu.com) wrote:
> * From: "Mathieu Desnoyers" <mathieu.desnoyers@polymtl.ca>
> >* Zhaolei (zhaolei at cn.fujitsu.com) wrote:
> >> * From: "Mathieu Desnoyers" <compudj@krystal.dyndns.org>
> >> >* Zhaolei (zhaolei at cn.fujitsu.com) wrote:
> >> >> Instrumentation of following panic and kexec related events are added:
> >> >> panic
> >> >> kernel_kexec
> >> >> crash_kexec
> >> >> 
> >> >> It is useful for build flight-recorder program based on lttng infrastructure.
> >> >> 
> >> > 
> >> > Hrm, I'm wondering if these are events we are interested to trace or if
> >> > you plan to use these tracepoints as hooks into the kernel to connect
> >> > the tracer infrastructure to it ?
> >> Hello, Mathieu
> >> 
> >> IMHO, both.
> >> 
> >> As a programmer who want to build  flight-recorder based on lttng infrastructure,
> >> he can:
> >> 1: use lttng trace, and get lttng's trace datas from kernel-dump file.
> >> or
> >> 2: use lttng's tracepoint, and record events himself.
> >> 
> >> But as a lttng's programmer, we can make both way possible.
> >> 
> >> What is your opinion?
> >> 
> > 
> > Hi Zhaolei,
> > 
> > I think having the tracepoints for panic and kexec would be a good
> > start. 
> Hello, Mathieu
> 
> Glad to hear that.
> 
> > Then whenever we need more specific hooks to send the data out
> > (for crash data extraction), we can add notification hooks at that time.
> > Do you have an updated version of panic+kexec instrumentation ?
> Do you means update this patch to top of lttng-git-tree only?
> 

Yes,

it's possible that it directly applies, can you confirm and resend ?

Thanks,

Mathieu

> B.R.
> Zhaolei
> 
> > 
> > Thanks,
> > 
> > Mathieu
> > 
> >> B.R.
> >> Zhaolei
> >> > 
> >> > If you plan to use those as hooks into the kernel, we should probably
> >> > consider adding notifiers instead (notifier.h). This is the "blessed"
> >> > way to be called upon such events.
> >> > 
> >> > Mathieu
> >> > 
> >> >> Signed-off-by: Zhao Lei <zhaolei at cn.fujitsu.com>
> >> >> ---
> >> >>  include/trace/kernel.h |   10 ++++++++++
> >> >>  kernel/kexec.c         |    8 ++++++++
> >> >>  kernel/panic.c         |    7 +++++++
> >> >>  3 files changed, 25 insertions(+), 0 deletions(-)
> >> >> 
> >> >> diff --git a/include/trace/kernel.h b/include/trace/kernel.h
> >> >> index d2c3320..06d585f 100644
> >> >> --- a/include/trace/kernel.h
> >> >> +++ b/include/trace/kernel.h
> >> >> @@ -2,6 +2,7 @@
> >> >>  #define _TRACE_KERNEL_H
> >> >>  
> >> >>  #include <linux/tracepoint.h>
> >> >> +#include <linux/kexec.h>
> >> >>  
> >> >>  DECLARE_TRACE(kernel_printk,
> >> >>  TPPROTO(unsigned long retaddr),
> >> >> @@ -15,5 +16,14 @@ DECLARE_TRACE(kernel_module_free,
> >> >>  DECLARE_TRACE(kernel_module_load,
> >> >>  TPPROTO(struct module *mod),
> >> >>  TPARGS(mod));
> >> >> +DECLARE_TRACE(kernel_panic,
> >> >> + TPPROTO(const char *fmt, va_list args),
> >> >> + TPARGS(fmt, args));
> >> >> +DECLARE_TRACE(kernel_kernel_kexec,
> >> >> + TPPROTO(struct kimage *image),
> >> >> + TPARGS(image));
> >> >> +DECLARE_TRACE(kernel_crash_kexec,
> >> >> + TPPROTO(struct kimage *image, struct pt_regs *regs),
> >> >> + TPARGS(image, regs));
> >> >>  
> >> >>  #endif
> >> >> diff --git a/kernel/kexec.c b/kernel/kexec.c
> >> >> index aef2653..56cdaac 100644
> >> >> --- a/kernel/kexec.c
> >> >> +++ b/kernel/kexec.c
> >> >> @@ -30,6 +30,7 @@
> >> >>  #include <linux/pm.h>
> >> >>  #include <linux/cpu.h>
> >> >>  #include <linux/console.h>
> >> >> +#include <trace/kernel.h>
> >> >>  
> >> >>  #include <asm/page.h>
> >> >>  #include <asm/uaccess.h>
> >> >> @@ -37,6 +38,9 @@
> >> >>  #include <asm/system.h>
> >> >>  #include <asm/sections.h>
> >> >>  
> >> >> +DEFINE_TRACE(kernel_kernel_kexec);
> >> >> +DEFINE_TRACE(kernel_crash_kexec);
> >> >> +
> >> >>  /* Per cpu memory for storing cpu states in case of system crash. */
> >> >>  note_buf_t* crash_notes;
> >> >>  
> >> >> @@ -1062,6 +1066,8 @@ asmlinkage long compat_sys_kexec_load(unsigned long entry,
> >> >>  
> >> >>  void crash_kexec(struct pt_regs *regs)
> >> >>  {
> >> >> + trace_kernel_crash_kexec(kexec_crash_image, regs);
> >> >> +
> >> >>  /* Take the kexec_mutex here to prevent sys_kexec_load
> >> >>  * running on one cpu from replacing the crash kernel
> >> >>  * we are using after a panic on a different cpu.
> >> >> @@ -1428,6 +1434,8 @@ int kernel_kexec(void)
> >> >>  {
> >> >>  int error = 0;
> >> >>  
> >> >> + trace_kernel_kernel_kexec(kexec_image);
> >> >> +
> >> >>  if (!mutex_trylock(&kexec_mutex))
> >> >>  return -EBUSY;
> >> >>  if (!kexec_image) {
> >> >> diff --git a/kernel/panic.c b/kernel/panic.c
> >> >> index 12c5a0a..3fa642e 100644
> >> >> --- a/kernel/panic.c
> >> >> +++ b/kernel/panic.c
> >> >> @@ -21,6 +21,9 @@
> >> >>  #include <linux/debug_locks.h>
> >> >>  #include <linux/random.h>
> >> >>  #include <linux/kallsyms.h>
> >> >> +#include <trace/kernel.h>
> >> >> +
> >> >> +DEFINE_TRACE(kernel_panic);
> >> >>  
> >> >>  int panic_on_oops;
> >> >>  int tainted;
> >> >> @@ -68,6 +71,10 @@ NORET_TYPE void panic(const char * fmt, ...)
> >> >>  unsigned long caller = (unsigned long) __builtin_return_address(0);
> >> >>  #endif
> >> >>  
> >> >> + va_start(args, fmt);
> >> >> + trace_kernel_panic(fmt, args);
> >> >> + va_end(args);
> >> >> +
> >> >>  /*
> >> >>  * It's possible to come here directly from a panic-assertion and not
> >> >>  * have preempt disabled. Some functions called from here want
> >> >> -- 
> >> >> 1.5.5.3
> >> >> 
> >> >> 
> >> >> 
> >> >> _______________________________________________
> >> >> ltt-dev mailing list
> >> >> ltt-dev at lists.casi.polymtl.ca
> >> >> http://lists.casi.polymtl.ca/cgi-bin/mailman/listinfo/ltt-dev
> >> >> 
> >> > 
> >> > -- 
> >> > Mathieu Desnoyers
> >> > OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68
> >> > 
> >> >
> >> 
> > 
> > -- 
> > Mathieu Desnoyers
> > OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68
> > 
> >
> _______________________________________________
> ltt-dev mailing list
> ltt-dev at lists.casi.polymtl.ca
> http://lists.casi.polymtl.ca/cgi-bin/mailman/listinfo/ltt-dev
> 

-- 
Mathieu Desnoyers
OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68

[ltt-dev] [PATCH v2] add tracepoints of panic and kexec

[Mathieu Desnoyers]

* Mathieu Desnoyers (compudj at krystal.dyndns.org) wrote:
> * Zhaolei (zhaolei at cn.fujitsu.com) wrote:
> > * From: "Mathieu Desnoyers" <mathieu.desnoyers@polymtl.ca>
> > >* Zhaolei (zhaolei at cn.fujitsu.com) wrote:
> > >> * From: "Mathieu Desnoyers" <compudj@krystal.dyndns.org>
> > >> >* Zhaolei (zhaolei at cn.fujitsu.com) wrote:
> > >> >> Instrumentation of following panic and kexec related events are added:
> > >> >> panic
> > >> >> kernel_kexec
> > >> >> crash_kexec
> > >> >> 
> > >> >> It is useful for build flight-recorder program based on lttng infrastructure.
> > >> >> 
> > >> > 
> > >> > Hrm, I'm wondering if these are events we are interested to trace or if
> > >> > you plan to use these tracepoints as hooks into the kernel to connect
> > >> > the tracer infrastructure to it ?
> > >> Hello, Mathieu
> > >> 
> > >> IMHO, both.
> > >> 
> > >> As a programmer who want to build  flight-recorder based on lttng infrastructure,
> > >> he can:
> > >> 1: use lttng trace, and get lttng's trace datas from kernel-dump file.
> > >> or
> > >> 2: use lttng's tracepoint, and record events himself.
> > >> 
> > >> But as a lttng's programmer, we can make both way possible.
> > >> 
> > >> What is your opinion?
> > >> 
> > > 
> > > Hi Zhaolei,
> > > 
> > > I think having the tracepoints for panic and kexec would be a good
> > > start. 
> > Hello, Mathieu
> > 
> > Glad to hear that.
> > 
> > > Then whenever we need more specific hooks to send the data out
> > > (for crash data extraction), we can add notification hooks at that time.
> > > Do you have an updated version of panic+kexec instrumentation ?
> > Do you means update this patch to top of lttng-git-tree only?
> > 
> 
> Yes,
> 
> it's possible that it directly applies, can you confirm and resend ?
> 
> Thanks,
> 

Hrm, I just had an afterthought :

Could you also provide a probe module in ltt/probes/ that would get the
data identified by these tracepoints into the trace stream ?  This would
help getting the data into the trace. Otherwise, there would be no real
user of these tracepoints.

Thanks,

MAthieu


> Mathieu
> 
> > B.R.
> > Zhaolei
> > 
> > > 
> > > Thanks,
> > > 
> > > Mathieu
> > > 
> > >> B.R.
> > >> Zhaolei
> > >> > 
> > >> > If you plan to use those as hooks into the kernel, we should probably
> > >> > consider adding notifiers instead (notifier.h). This is the "blessed"
> > >> > way to be called upon such events.
> > >> > 
> > >> > Mathieu
> > >> > 
> > >> >> Signed-off-by: Zhao Lei <zhaolei at cn.fujitsu.com>
> > >> >> ---
> > >> >>  include/trace/kernel.h |   10 ++++++++++
> > >> >>  kernel/kexec.c         |    8 ++++++++
> > >> >>  kernel/panic.c         |    7 +++++++
> > >> >>  3 files changed, 25 insertions(+), 0 deletions(-)
> > >> >> 
> > >> >> diff --git a/include/trace/kernel.h b/include/trace/kernel.h
> > >> >> index d2c3320..06d585f 100644
> > >> >> --- a/include/trace/kernel.h
> > >> >> +++ b/include/trace/kernel.h
> > >> >> @@ -2,6 +2,7 @@
> > >> >>  #define _TRACE_KERNEL_H
> > >> >>  
> > >> >>  #include <linux/tracepoint.h>
> > >> >> +#include <linux/kexec.h>
> > >> >>  
> > >> >>  DECLARE_TRACE(kernel_printk,
> > >> >>  TPPROTO(unsigned long retaddr),
> > >> >> @@ -15,5 +16,14 @@ DECLARE_TRACE(kernel_module_free,
> > >> >>  DECLARE_TRACE(kernel_module_load,
> > >> >>  TPPROTO(struct module *mod),
> > >> >>  TPARGS(mod));
> > >> >> +DECLARE_TRACE(kernel_panic,
> > >> >> + TPPROTO(const char *fmt, va_list args),
> > >> >> + TPARGS(fmt, args));
> > >> >> +DECLARE_TRACE(kernel_kernel_kexec,
> > >> >> + TPPROTO(struct kimage *image),
> > >> >> + TPARGS(image));
> > >> >> +DECLARE_TRACE(kernel_crash_kexec,
> > >> >> + TPPROTO(struct kimage *image, struct pt_regs *regs),
> > >> >> + TPARGS(image, regs));
> > >> >>  
> > >> >>  #endif
> > >> >> diff --git a/kernel/kexec.c b/kernel/kexec.c
> > >> >> index aef2653..56cdaac 100644
> > >> >> --- a/kernel/kexec.c
> > >> >> +++ b/kernel/kexec.c
> > >> >> @@ -30,6 +30,7 @@
> > >> >>  #include <linux/pm.h>
> > >> >>  #include <linux/cpu.h>
> > >> >>  #include <linux/console.h>
> > >> >> +#include <trace/kernel.h>
> > >> >>  
> > >> >>  #include <asm/page.h>
> > >> >>  #include <asm/uaccess.h>
> > >> >> @@ -37,6 +38,9 @@
> > >> >>  #include <asm/system.h>
> > >> >>  #include <asm/sections.h>
> > >> >>  
> > >> >> +DEFINE_TRACE(kernel_kernel_kexec);
> > >> >> +DEFINE_TRACE(kernel_crash_kexec);
> > >> >> +
> > >> >>  /* Per cpu memory for storing cpu states in case of system crash. */
> > >> >>  note_buf_t* crash_notes;
> > >> >>  
> > >> >> @@ -1062,6 +1066,8 @@ asmlinkage long compat_sys_kexec_load(unsigned long entry,
> > >> >>  
> > >> >>  void crash_kexec(struct pt_regs *regs)
> > >> >>  {
> > >> >> + trace_kernel_crash_kexec(kexec_crash_image, regs);
> > >> >> +
> > >> >>  /* Take the kexec_mutex here to prevent sys_kexec_load
> > >> >>  * running on one cpu from replacing the crash kernel
> > >> >>  * we are using after a panic on a different cpu.
> > >> >> @@ -1428,6 +1434,8 @@ int kernel_kexec(void)
> > >> >>  {
> > >> >>  int error = 0;
> > >> >>  
> > >> >> + trace_kernel_kernel_kexec(kexec_image);
> > >> >> +
> > >> >>  if (!mutex_trylock(&kexec_mutex))
> > >> >>  return -EBUSY;
> > >> >>  if (!kexec_image) {
> > >> >> diff --git a/kernel/panic.c b/kernel/panic.c
> > >> >> index 12c5a0a..3fa642e 100644
> > >> >> --- a/kernel/panic.c
> > >> >> +++ b/kernel/panic.c
> > >> >> @@ -21,6 +21,9 @@
> > >> >>  #include <linux/debug_locks.h>
> > >> >>  #include <linux/random.h>
> > >> >>  #include <linux/kallsyms.h>
> > >> >> +#include <trace/kernel.h>
> > >> >> +
> > >> >> +DEFINE_TRACE(kernel_panic);
> > >> >>  
> > >> >>  int panic_on_oops;
> > >> >>  int tainted;
> > >> >> @@ -68,6 +71,10 @@ NORET_TYPE void panic(const char * fmt, ...)
> > >> >>  unsigned long caller = (unsigned long) __builtin_return_address(0);
> > >> >>  #endif
> > >> >>  
> > >> >> + va_start(args, fmt);
> > >> >> + trace_kernel_panic(fmt, args);
> > >> >> + va_end(args);
> > >> >> +
> > >> >>  /*
> > >> >>  * It's possible to come here directly from a panic-assertion and not
> > >> >>  * have preempt disabled. Some functions called from here want
> > >> >> -- 
> > >> >> 1.5.5.3
> > >> >> 
> > >> >> 
> > >> >> 
> > >> >> _______________________________________________
> > >> >> ltt-dev mailing list
> > >> >> ltt-dev at lists.casi.polymtl.ca
> > >> >> http://lists.casi.polymtl.ca/cgi-bin/mailman/listinfo/ltt-dev
> > >> >> 
> > >> > 
> > >> > -- 
> > >> > Mathieu Desnoyers
> > >> > OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68
> > >> > 
> > >> >
> > >> 
> > > 
> > > -- 
> > > Mathieu Desnoyers
> > > OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68
> > > 
> > >
> > _______________________________________________
> > ltt-dev mailing list
> > ltt-dev at lists.casi.polymtl.ca
> > http://lists.casi.polymtl.ca/cgi-bin/mailman/listinfo/ltt-dev
> > 
> 
> -- 
> Mathieu Desnoyers
> OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68
> 
> _______________________________________________
> ltt-dev mailing list
> ltt-dev at lists.casi.polymtl.ca
> http://lists.casi.polymtl.ca/cgi-bin/mailman/listinfo/ltt-dev
> 

-- 
Mathieu Desnoyers
OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68

[ltt-dev] [PATCH v2] add tracepoints of panic and kexec

[Zhaolei]

* From: "Mathieu Desnoyers" <compudj@krystal.dyndns.org>
>* Mathieu Desnoyers (compudj at krystal.dyndns.org) wrote:
>> * Zhaolei (zhaolei at cn.fujitsu.com) wrote:
>> > * From: "Mathieu Desnoyers" <mathieu.desnoyers@polymtl.ca>
>> > >* Zhaolei (zhaolei at cn.fujitsu.com) wrote:
>> > >> * From: "Mathieu Desnoyers" <compudj@krystal.dyndns.org>
>> > >> >* Zhaolei (zhaolei at cn.fujitsu.com) wrote:
>> > >> >> Instrumentation of following panic and kexec related events are added:
>> > >> >> panic
>> > >> >> kernel_kexec
>> > >> >> crash_kexec
>> > >> >> 
>> > >> >> It is useful for build flight-recorder program based on lttng infrastructure.
>> > >> >> 
>> > >> > 
>> > >> > Hrm, I'm wondering if these are events we are interested to trace or if
>> > >> > you plan to use these tracepoints as hooks into the kernel to connect
>> > >> > the tracer infrastructure to it ?
>> > >> Hello, Mathieu
>> > >> 
>> > >> IMHO, both.
>> > >> 
>> > >> As a programmer who want to build  flight-recorder based on lttng infrastructure,
>> > >> he can:
>> > >> 1: use lttng trace, and get lttng's trace datas from kernel-dump file.
>> > >> or
>> > >> 2: use lttng's tracepoint, and record events himself.
>> > >> 
>> > >> But as a lttng's programmer, we can make both way possible.
>> > >> 
>> > >> What is your opinion?
>> > >> 
>> > > 
>> > > Hi Zhaolei,
>> > > 
>> > > I think having the tracepoints for panic and kexec would be a good
>> > > start. 
>> > Hello, Mathieu
>> > 
>> > Glad to hear that.
>> > 
>> > > Then whenever we need more specific hooks to send the data out
>> > > (for crash data extraction), we can add notification hooks at that time.
>> > > Do you have an updated version of panic+kexec instrumentation ?
>> > Do you means update this patch to top of lttng-git-tree only?
>> > 
>> 
>> Yes,
>> 
>> it's possible that it directly applies, can you confirm and resend ?
>> 
>> Thanks,
>> 
> 
> Hrm, I just had an afterthought :
> 
> Could you also provide a probe module in ltt/probes/ that would get the
> data identified by these tracepoints into the trace stream ?  This would
> help getting the data into the trace. Otherwise, there would be no real
> user of these tracepoints.
Hello, Mathieu

It is useful to add marker-interface to these event.
I'll do it and send a updated patch.

B.R.
Zhaolei
> 
> Thanks,
> 
> MAthieu
> 
> 
>> Mathieu
>> 
>> > B.R.
>> > Zhaolei
>> > 
>> > > 
>> > > Thanks,
>> > > 
>> > > Mathieu
>> > > 
>> > >> B.R.
>> > >> Zhaolei
>> > >> > 
>> > >> > If you plan to use those as hooks into the kernel, we should probably
>> > >> > consider adding notifiers instead (notifier.h). This is the "blessed"
>> > >> > way to be called upon such events.
>> > >> > 
>> > >> > Mathieu
>> > >> > 
>> > >> >> Signed-off-by: Zhao Lei <zhaolei at cn.fujitsu.com>
>> > >> >> ---
>> > >> >>  include/trace/kernel.h |   10 ++++++++++
>> > >> >>  kernel/kexec.c         |    8 ++++++++
>> > >> >>  kernel/panic.c         |    7 +++++++
>> > >> >>  3 files changed, 25 insertions(+), 0 deletions(-)
>> > >> >> 
>> > >> >> diff --git a/include/trace/kernel.h b/include/trace/kernel.h
>> > >> >> index d2c3320..06d585f 100644
>> > >> >> --- a/include/trace/kernel.h
>> > >> >> +++ b/include/trace/kernel.h
>> > >> >> @@ -2,6 +2,7 @@
>> > >> >>  #define _TRACE_KERNEL_H
>> > >> >>  
>> > >> >>  #include <linux/tracepoint.h>
>> > >> >> +#include <linux/kexec.h>
>> > >> >>  
>> > >> >>  DECLARE_TRACE(kernel_printk,
>> > >> >>  TPPROTO(unsigned long retaddr),
>> > >> >> @@ -15,5 +16,14 @@ DECLARE_TRACE(kernel_module_free,
>> > >> >>  DECLARE_TRACE(kernel_module_load,
>> > >> >>  TPPROTO(struct module *mod),
>> > >> >>  TPARGS(mod));
>> > >> >> +DECLARE_TRACE(kernel_panic,
>> > >> >> + TPPROTO(const char *fmt, va_list args),
>> > >> >> + TPARGS(fmt, args));
>> > >> >> +DECLARE_TRACE(kernel_kernel_kexec,
>> > >> >> + TPPROTO(struct kimage *image),
>> > >> >> + TPARGS(image));
>> > >> >> +DECLARE_TRACE(kernel_crash_kexec,
>> > >> >> + TPPROTO(struct kimage *image, struct pt_regs *regs),
>> > >> >> + TPARGS(image, regs));
>> > >> >>  
>> > >> >>  #endif
>> > >> >> diff --git a/kernel/kexec.c b/kernel/kexec.c
>> > >> >> index aef2653..56cdaac 100644
>> > >> >> --- a/kernel/kexec.c
>> > >> >> +++ b/kernel/kexec.c
>> > >> >> @@ -30,6 +30,7 @@
>> > >> >>  #include <linux/pm.h>
>> > >> >>  #include <linux/cpu.h>
>> > >> >>  #include <linux/console.h>
>> > >> >> +#include <trace/kernel.h>
>> > >> >>  
>> > >> >>  #include <asm/page.h>
>> > >> >>  #include <asm/uaccess.h>
>> > >> >> @@ -37,6 +38,9 @@
>> > >> >>  #include <asm/system.h>
>> > >> >>  #include <asm/sections.h>
>> > >> >>  
>> > >> >> +DEFINE_TRACE(kernel_kernel_kexec);
>> > >> >> +DEFINE_TRACE(kernel_crash_kexec);
>> > >> >> +
>> > >> >>  /* Per cpu memory for storing cpu states in case of system crash. */
>> > >> >>  note_buf_t* crash_notes;
>> > >> >>  
>> > >> >> @@ -1062,6 +1066,8 @@ asmlinkage long compat_sys_kexec_load(unsigned long entry,
>> > >> >>  
>> > >> >>  void crash_kexec(struct pt_regs *regs)
>> > >> >>  {
>> > >> >> + trace_kernel_crash_kexec(kexec_crash_image, regs);
>> > >> >> +
>> > >> >>  /* Take the kexec_mutex here to prevent sys_kexec_load
>> > >> >>  * running on one cpu from replacing the crash kernel
>> > >> >>  * we are using after a panic on a different cpu.
>> > >> >> @@ -1428,6 +1434,8 @@ int kernel_kexec(void)
>> > >> >>  {
>> > >> >>  int error = 0;
>> > >> >>  
>> > >> >> + trace_kernel_kernel_kexec(kexec_image);
>> > >> >> +
>> > >> >>  if (!mutex_trylock(&kexec_mutex))
>> > >> >>  return -EBUSY;
>> > >> >>  if (!kexec_image) {
>> > >> >> diff --git a/kernel/panic.c b/kernel/panic.c
>> > >> >> index 12c5a0a..3fa642e 100644
>> > >> >> --- a/kernel/panic.c
>> > >> >> +++ b/kernel/panic.c
>> > >> >> @@ -21,6 +21,9 @@
>> > >> >>  #include <linux/debug_locks.h>
>> > >> >>  #include <linux/random.h>
>> > >> >>  #include <linux/kallsyms.h>
>> > >> >> +#include <trace/kernel.h>
>> > >> >> +
>> > >> >> +DEFINE_TRACE(kernel_panic);
>> > >> >>  
>> > >> >>  int panic_on_oops;
>> > >> >>  int tainted;
>> > >> >> @@ -68,6 +71,10 @@ NORET_TYPE void panic(const char * fmt, ...)
>> > >> >>  unsigned long caller = (unsigned long) __builtin_return_address(0);
>> > >> >>  #endif
>> > >> >>  
>> > >> >> + va_start(args, fmt);
>> > >> >> + trace_kernel_panic(fmt, args);
>> > >> >> + va_end(args);
>> > >> >> +
>> > >> >>  /*
>> > >> >>  * It's possible to come here directly from a panic-assertion and not
>> > >> >>  * have preempt disabled. Some functions called from here want
>> > >> >> -- 
>> > >> >> 1.5.5.3
>> > >> >> 
>> > >> >> 
>> > >> >> 
>> > >> >> _______________________________________________
>> > >> >> ltt-dev mailing list
>> > >> >> ltt-dev at lists.casi.polymtl.ca
>> > >> >> http://lists.casi.polymtl.ca/cgi-bin/mailman/listinfo/ltt-dev
>> > >> >> 
>> > >> > 
>> > >> > -- 
>> > >> > Mathieu Desnoyers
>> > >> > OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68
>> > >> > 
>> > >> >
>> > >> 
>> > > 
>> > > -- 
>> > > Mathieu Desnoyers
>> > > OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68
>> > > 
>> > >
>> > _______________________________________________
>> > ltt-dev mailing list
>> > ltt-dev at lists.casi.polymtl.ca
>> > http://lists.casi.polymtl.ca/cgi-bin/mailman/listinfo/ltt-dev
>> > 
>> 
>> -- 
>> Mathieu Desnoyers
>> OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68
>> 
>> _______________________________________________
>> ltt-dev mailing list
>> ltt-dev at lists.casi.polymtl.ca
>> http://lists.casi.polymtl.ca/cgi-bin/mailman/listinfo/ltt-dev
>> 
> 
> -- 
> Mathieu Desnoyers
> OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68
> 
>

[ltt-dev] [PATCH v3 0/1] add tracepoints of panic and kexec

[Zhaolei]

* Mathieu Desnoyers wrote:
> * Zhaolei (zhaolei at cn.fujitsu.com) wrote:
>> Instrumentation of following panic and kexec related events are added:
>> 	panic
>> 	kernel_kexec
>> 	crash_kexec
>>
>> It is useful for build flight-recorder program based on lttng infrastructure.
>>
> 
> Hrm, I'm wondering if these are events we are interested to trace or if
> you plan to use these tracepoints as hooks into the kernel to connect
> the tracer infrastructure to it ?
Hello, Mathieu,

I modified this patch to use notifier.h on your suggest.
Is is useful for flight-recorder, I think.

B.R.
Zhaolei
> 
> If you plan to use those as hooks into the kernel, we should probably
> consider adding notifiers instead (notifier.h). This is the "blessed"
> way to be called upon such events.
> 
> Mathieu
> 

[ltt-dev] [PATCH v3 1/1] add tracepoints of panic and kexec

[Zhaolei]

Instrumentation of following panic and kexec related events are added:
	panic
	kernel_kexec
	crash_kexec

It is useful for build flight-recorder program based on lttng infrastructure.

Signed-off-by: Zhao Lei <zhaolei at cn.fujitsu.com>
---
 include/trace/notifier.h |   17 +++++++++++++++++
 kernel/kexec.c           |    8 ++++++++
 kernel/panic.c           |    7 +++++++
 3 files changed, 32 insertions(+), 0 deletions(-)
 create mode 100644 include/trace/notifier.h

diff --git a/include/trace/notifier.h b/include/trace/notifier.h
new file mode 100644
index 0000000..0eb721b
--- /dev/null
+++ b/include/trace/notifier.h
@@ -0,0 +1,17 @@
+#ifndef _TRACE_NOTIFIER_H
+#define _TRACE_NOTIFIER_H
+
+#include <linux/tracepoint.h>
+#include <linux/kexec.h>
+
+DECLARE_TRACE(notifier_panic,
+	TPPROTO(const char *fmt, va_list args),
+	TPARGS(fmt, args));
+DECLARE_TRACE(notifier_kernel_kexec,
+	TPPROTO(struct kimage *image),
+	TPARGS(image));
+DECLARE_TRACE(notifier_crash_kexec,
+	TPPROTO(struct kimage *image, struct pt_regs *regs),
+	TPARGS(image, regs));
+
+#endif
diff --git a/kernel/kexec.c b/kernel/kexec.c
index aef2653..4de87cc 100644
--- a/kernel/kexec.c
+++ b/kernel/kexec.c
@@ -30,6 +30,7 @@
 #include <linux/pm.h>
 #include <linux/cpu.h>
 #include <linux/console.h>
+#include <trace/notifier.h>
 
 #include <asm/page.h>
 #include <asm/uaccess.h>
@@ -37,6 +38,9 @@
 #include <asm/system.h>
 #include <asm/sections.h>
 
+DEFINE_TRACE(notifier_kernel_kexec);
+DEFINE_TRACE(notifier_crash_kexec);
+
 /* Per cpu memory for storing cpu states in case of system crash. */
 note_buf_t* crash_notes;
 
@@ -1062,6 +1066,8 @@ asmlinkage long compat_sys_kexec_load(unsigned long entry,
 
 void crash_kexec(struct pt_regs *regs)
 {
+	trace_notifier_crash_kexec(kexec_crash_image, regs);
+
 	/* Take the kexec_mutex here to prevent sys_kexec_load
 	 * running on one cpu from replacing the crash kernel
 	 * we are using after a panic on a different cpu.
@@ -1428,6 +1434,8 @@ int kernel_kexec(void)
 {
 	int error = 0;
 
+	trace_notifier_kernel_kexec(kexec_image);
+
 	if (!mutex_trylock(&kexec_mutex))
 		return -EBUSY;
 	if (!kexec_image) {
diff --git a/kernel/panic.c b/kernel/panic.c
index 12c5a0a..d1be86c 100644
--- a/kernel/panic.c
+++ b/kernel/panic.c
@@ -21,6 +21,9 @@
 #include <linux/debug_locks.h>
 #include <linux/random.h>
 #include <linux/kallsyms.h>
+#include <trace/notifier.h>
+
+DEFINE_TRACE(notifier_panic);
 
 int panic_on_oops;
 int tainted;
@@ -68,6 +71,10 @@ NORET_TYPE void panic(const char * fmt, ...)
 	unsigned long caller = (unsigned long) __builtin_return_address(0);
 #endif
 
+	va_start(args, fmt);
+	trace_notifier_panic(fmt, args);
+	va_end(args);
+
 	/*
 	 * It's possible to come here directly from a panic-assertion and not
 	 * have preempt disabled. Some functions called from here want
-- 
1.5.5.3

[ltt-dev] [PATCH v3 1/1] add tracepoints of panic and kexec

[Mathieu Desnoyers]

* Zhaolei (zhaolei at cn.fujitsu.com) wrote:
> Instrumentation of following panic and kexec related events are added:
> 	panic
> 	kernel_kexec
> 	crash_kexec
> 
> It is useful for build flight-recorder program based on lttng infrastructure.
> 
> Signed-off-by: Zhao Lei <zhaolei at cn.fujitsu.com>
> ---
>  include/trace/notifier.h |   17 +++++++++++++++++
>  kernel/kexec.c           |    8 ++++++++
>  kernel/panic.c           |    7 +++++++
>  3 files changed, 32 insertions(+), 0 deletions(-)
>  create mode 100644 include/trace/notifier.h
> 
> diff --git a/include/trace/notifier.h b/include/trace/notifier.h

Hi Zhaolei,

By saying "using notifier.h", I meant "using include/linux/notifier.h",
the standard kernel header, with a standard notifier call chain. Sorry
for the misunderstanding.

Mathieu


> new file mode 100644
> index 0000000..0eb721b
> --- /dev/null
> +++ b/include/trace/notifier.h
> @@ -0,0 +1,17 @@
> +#ifndef _TRACE_NOTIFIER_H
> +#define _TRACE_NOTIFIER_H
> +
> +#include <linux/tracepoint.h>
> +#include <linux/kexec.h>
> +
> +DECLARE_TRACE(notifier_panic,
> +	TPPROTO(const char *fmt, va_list args),
> +	TPARGS(fmt, args));
> +DECLARE_TRACE(notifier_kernel_kexec,
> +	TPPROTO(struct kimage *image),
> +	TPARGS(image));
> +DECLARE_TRACE(notifier_crash_kexec,
> +	TPPROTO(struct kimage *image, struct pt_regs *regs),
> +	TPARGS(image, regs));
> +
> +#endif
> diff --git a/kernel/kexec.c b/kernel/kexec.c
> index aef2653..4de87cc 100644
> --- a/kernel/kexec.c
> +++ b/kernel/kexec.c
> @@ -30,6 +30,7 @@
>  #include <linux/pm.h>
>  #include <linux/cpu.h>
>  #include <linux/console.h>
> +#include <trace/notifier.h>
>  
>  #include <asm/page.h>
>  #include <asm/uaccess.h>
> @@ -37,6 +38,9 @@
>  #include <asm/system.h>
>  #include <asm/sections.h>
>  
> +DEFINE_TRACE(notifier_kernel_kexec);
> +DEFINE_TRACE(notifier_crash_kexec);
> +
>  /* Per cpu memory for storing cpu states in case of system crash. */
>  note_buf_t* crash_notes;
>  
> @@ -1062,6 +1066,8 @@ asmlinkage long compat_sys_kexec_load(unsigned long entry,
>  
>  void crash_kexec(struct pt_regs *regs)
>  {
> +	trace_notifier_crash_kexec(kexec_crash_image, regs);
> +
>  	/* Take the kexec_mutex here to prevent sys_kexec_load
>  	 * running on one cpu from replacing the crash kernel
>  	 * we are using after a panic on a different cpu.
> @@ -1428,6 +1434,8 @@ int kernel_kexec(void)
>  {
>  	int error = 0;
>  
> +	trace_notifier_kernel_kexec(kexec_image);
> +
>  	if (!mutex_trylock(&kexec_mutex))
>  		return -EBUSY;
>  	if (!kexec_image) {
> diff --git a/kernel/panic.c b/kernel/panic.c
> index 12c5a0a..d1be86c 100644
> --- a/kernel/panic.c
> +++ b/kernel/panic.c
> @@ -21,6 +21,9 @@
>  #include <linux/debug_locks.h>
>  #include <linux/random.h>
>  #include <linux/kallsyms.h>
> +#include <trace/notifier.h>
> +
> +DEFINE_TRACE(notifier_panic);
>  
>  int panic_on_oops;
>  int tainted;
> @@ -68,6 +71,10 @@ NORET_TYPE void panic(const char * fmt, ...)
>  	unsigned long caller = (unsigned long) __builtin_return_address(0);
>  #endif
>  
> +	va_start(args, fmt);
> +	trace_notifier_panic(fmt, args);
> +	va_end(args);
> +
>  	/*
>  	 * It's possible to come here directly from a panic-assertion and not
>  	 * have preempt disabled. Some functions called from here want
> -- 
> 1.5.5.3
> 
> 
> 
> _______________________________________________
> ltt-dev mailing list
> ltt-dev at lists.casi.polymtl.ca
> http://lists.casi.polymtl.ca/cgi-bin/mailman/listinfo/ltt-dev
> 

-- 
Mathieu Desnoyers
OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68

[ltt-dev] [PATCH 2/2] tracepoints-kexec

[Zhaolei]

Instrumentation of kexec related events : kernel_kexec and crash_kexec.
It is useful for build flight-recorder program based on lttng infrastructure.

Signed-off-by: Zhao lei <zhaolei at cn.fujitsu.com>
---
 include/trace/kexec.h |   14 ++++++++++++++
 kernel/kexec.c        |    8 ++++++++
 2 files changed, 22 insertions(+), 0 deletions(-)
 create mode 100644 include/trace/kexec.h

diff --git a/include/trace/kexec.h b/include/trace/kexec.h
new file mode 100644
index 0000000..928bdd2
--- /dev/null
+++ b/include/trace/kexec.h
@@ -0,0 +1,14 @@
+#ifndef _TRACE_KEXEC_H
+#define _TRACE_KEXEC_H
+
+#include <linux/tracepoint.h>
+#include <linux/kexec.h>
+
+DECLARE_TRACE(kexec_kernel_kexec,
+	TPPROTO(struct kimage *image),
+	TPARGS(image));
+DECLARE_TRACE(kexec_crash_kexec,
+	TPPROTO(struct kimage *image, struct pt_regs *regs),
+	TPARGS(image, regs));
+
+#endif
diff --git a/kernel/kexec.c b/kernel/kexec.c
index aef2653..01ab16b 100644
--- a/kernel/kexec.c
+++ b/kernel/kexec.c
@@ -30,6 +30,7 @@
 #include <linux/pm.h>
 #include <linux/cpu.h>
 #include <linux/console.h>
+#include <trace/kexec.h>
 
 #include <asm/page.h>
 #include <asm/uaccess.h>
@@ -37,6 +38,9 @@
 #include <asm/system.h>
 #include <asm/sections.h>
 
+DEFINE_TRACE(kexec_kernel_kexec);
+DEFINE_TRACE(kexec_crash_kexec);
+
 /* Per cpu memory for storing cpu states in case of system crash. */
 note_buf_t* crash_notes;
 
@@ -1062,6 +1066,8 @@ asmlinkage long compat_sys_kexec_load(unsigned long entry,
 
 void crash_kexec(struct pt_regs *regs)
 {
+	trace_kexec_crash_kexec(kexec_crash_image, regs);
+
 	/* Take the kexec_mutex here to prevent sys_kexec_load
 	 * running on one cpu from replacing the crash kernel
 	 * we are using after a panic on a different cpu.
@@ -1428,6 +1434,8 @@ int kernel_kexec(void)
 {
 	int error = 0;
 
+	trace_kexec_kernel_kexec(kexec_image);
+
 	if (!mutex_trylock(&kexec_mutex))
 		return -EBUSY;
 	if (!kexec_image) {
-- 
1.5.5.3

[ltt-dev] [PATCH 2/2] tracepoints-kexec

[Mathieu Desnoyers]

* Zhaolei (zhaolei at cn.fujitsu.com) wrote:
> Instrumentation of kexec related events : kernel_kexec and crash_kexec.
> It is useful for build flight-recorder program based on lttng infrastructure.
> 
> Signed-off-by: Zhao lei <zhaolei at cn.fujitsu.com>
> ---
>  include/trace/kexec.h |   14 ++++++++++++++
>  kernel/kexec.c        |    8 ++++++++
>  2 files changed, 22 insertions(+), 0 deletions(-)
>  create mode 100644 include/trace/kexec.h
> 
> diff --git a/include/trace/kexec.h b/include/trace/kexec.h
> new file mode 100644
> index 0000000..928bdd2
> --- /dev/null
> +++ b/include/trace/kexec.h
> @@ -0,0 +1,14 @@
> +#ifndef _TRACE_KEXEC_H
> +#define _TRACE_KEXEC_H
> +
> +#include <linux/tracepoint.h>
> +#include <linux/kexec.h>
> +
> +DECLARE_TRACE(kexec_kernel_kexec,
> +	TPPROTO(struct kimage *image),
> +	TPARGS(image));
> +DECLARE_TRACE(kexec_crash_kexec,
> +	TPPROTO(struct kimage *image, struct pt_regs *regs),
> +	TPARGS(image, regs));
> +

Those could fit in include/trace/kernel.h too.

kernel_kexec
and
kernel_kexec_crash

?

My rule of thumb would be this :
- If we instrument a very small kernel file under kernel/, then we use
  kernel_ prefix.
- If we instrument a bigger kernel infrastructure, even if under
  kernel/, like the scheduler, then it may be better to use a sched_
  prefix.

Mathieu

> +#endif
> diff --git a/kernel/kexec.c b/kernel/kexec.c
> index aef2653..01ab16b 100644
> --- a/kernel/kexec.c
> +++ b/kernel/kexec.c
> @@ -30,6 +30,7 @@
>  #include <linux/pm.h>
>  #include <linux/cpu.h>
>  #include <linux/console.h>
> +#include <trace/kexec.h>
>  
>  #include <asm/page.h>
>  #include <asm/uaccess.h>
> @@ -37,6 +38,9 @@
>  #include <asm/system.h>
>  #include <asm/sections.h>
>  
> +DEFINE_TRACE(kexec_kernel_kexec);
> +DEFINE_TRACE(kexec_crash_kexec);
> +
>  /* Per cpu memory for storing cpu states in case of system crash. */
>  note_buf_t* crash_notes;
>  
> @@ -1062,6 +1066,8 @@ asmlinkage long compat_sys_kexec_load(unsigned long entry,
>  
>  void crash_kexec(struct pt_regs *regs)
>  {
> +	trace_kexec_crash_kexec(kexec_crash_image, regs);
> +
>  	/* Take the kexec_mutex here to prevent sys_kexec_load
>  	 * running on one cpu from replacing the crash kernel
>  	 * we are using after a panic on a different cpu.
> @@ -1428,6 +1434,8 @@ int kernel_kexec(void)
>  {
>  	int error = 0;
>  
> +	trace_kexec_kernel_kexec(kexec_image);
> +
>  	if (!mutex_trylock(&kexec_mutex))
>  		return -EBUSY;
>  	if (!kexec_image) {
> -- 
> 1.5.5.3
> 
> 

-- 
Mathieu Desnoyers
OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68