[lttng-dev] [Qemu-devel] [PATCH 0/6] hypertrace: Lightweight guest-to-QEMU trace channel

[lttng-dev] [Qemu-devel] [PATCH 0/6] hypertrace: Lightweight guest-to-QEMU trace channel

[Stefan Hajnoczi]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain, Size: 4270 bytes --]

On Fri, Aug 05, 2016 at 06:59:23PM +0200, Lluís Vilanova wrote:
> The hypertrace channel allows guest code to emit events in QEMU (the host) using
> its tracing infrastructure (see "docs/trace.txt"). This works in both 'system'
> and 'user' modes. That is, hypertrace is to tracing, what hypercalls are to
> system calls.
> 
> You can use this to emit an event on both guest and QEMU (host) traces to easily
> synchronize or correlate them. You could also modify you guest's tracing system
> to emit all events through the hypertrace channel, providing a unified and fully
> synchronized trace log. Another use case is timing the performance of guest code
> when optimizing TCG (QEMU traces have a timestamp).
> 
> See first commit for a full description.
> 
> Signed-off-by: Lluís Vilanova <vilanova at ac.upc.edu>
> ---

CCing Steven Rostedt, Masami Hiramatsu, Luiz Capitulino, and LTTng folks
who have all looked into host/guest tracing solutions.

Here is a mailing list archive link to this email thread:
https://lists.gnu.org/archive/html/qemu-devel/2016-08/msg01081.html

> 
> Lluís Vilanova (6):
>       hypertrace: Add documentation
>       hypertrace: Add tracing event "guest_hypertrace"
>       hypertrace: [*-user] Add QEMU-side proxy to "guest_hypertrace" event
>       hypertrace: [softmmu] Add QEMU-side proxy to "guest_hypertrace" event
>       hypertrace: Add guest-side user-level library
>       hypertrace: Add guest-side Linux module
> 
> 
>  Makefile                                           |    8 -
>  Makefile.objs                                      |    6 
>  bsd-user/main.c                                    |   16 +
>  bsd-user/mmap.c                                    |    2 
>  bsd-user/syscall.c                                 |    4 
>  configure                                          |   42 +++
>  docs/hypertrace.txt                                |  141 ++++++++++
>  docs/tracing.txt                                   |    3 
>  hypertrace/Makefile.objs                           |   20 +
>  hypertrace/guest/linux-module/Kbuild.in            |    5 
>  hypertrace/guest/linux-module/Makefile             |   24 ++
>  .../include/linux/qemu-hypertrace-internal.h       |   46 +++
>  .../linux-module/include/linux/qemu-hypertrace.h   |   71 +++++
>  hypertrace/guest/linux-module/qemu-hypertrace.c    |  133 +++++++++
>  hypertrace/guest/user/Makefile                     |   28 ++
>  hypertrace/guest/user/common.c                     |  221 +++++++++++++++
>  hypertrace/guest/user/qemu-hypertrace.h            |   77 +++++
>  hypertrace/softmmu.c                               |  243 +++++++++++++++++
>  hypertrace/user.c                                  |  292 ++++++++++++++++++++
>  hypertrace/user.h                                  |   52 ++++
>  include/hw/pci/pci.h                               |    2 
>  linux-user/main.c                                  |   19 +
>  linux-user/mmap.c                                  |    2 
>  linux-user/syscall.c                               |    3 
>  trace/Makefile.objs                                |    2 
>  25 files changed, 1460 insertions(+), 2 deletions(-)
>  create mode 100644 docs/hypertrace.txt
>  create mode 100644 hypertrace/Makefile.objs
>  create mode 100644 hypertrace/guest/linux-module/Kbuild.in
>  create mode 100644 hypertrace/guest/linux-module/Makefile
>  create mode 100644 hypertrace/guest/linux-module/include/linux/qemu-hypertrace-internal.h
>  create mode 100644 hypertrace/guest/linux-module/include/linux/qemu-hypertrace.h
>  create mode 100644 hypertrace/guest/linux-module/qemu-hypertrace.c
>  create mode 100644 hypertrace/guest/user/Makefile
>  create mode 100644 hypertrace/guest/user/common.c
>  create mode 100644 hypertrace/guest/user/qemu-hypertrace.h
>  create mode 100644 hypertrace/softmmu.c
>  create mode 100644 hypertrace/user.c
>  create mode 100644 hypertrace/user.h
> 
> 
> To: qemu-devel at nongnu.org
> Cc: Stefan Hajnoczi <stefanha at redhat.com>
> 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 473 bytes
Desc: not available
URL: <https://lists.lttng.org/pipermail/lttng-dev/attachments/20160818/ae62286f/attachment.sig>

[lttng-dev] [Qemu-devel] [PATCH 0/6] hypertrace: Lightweight guest-to-QEMU trace channel

[Lluís Vilanova]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain, Size: 1096 bytes --]

Stefan Hajnoczi writes:

> On Fri, Aug 05, 2016 at 06:59:23PM +0200, Lluís Vilanova wrote:
>> The hypertrace channel allows guest code to emit events in QEMU (the host) using
>> its tracing infrastructure (see "docs/trace.txt"). This works in both 'system'
>> and 'user' modes. That is, hypertrace is to tracing, what hypercalls are to
>> system calls.
>> 
>> You can use this to emit an event on both guest and QEMU (host) traces to easily
>> synchronize or correlate them. You could also modify you guest's tracing system
>> to emit all events through the hypertrace channel, providing a unified and fully
>> synchronized trace log. Another use case is timing the performance of guest code
>> when optimizing TCG (QEMU traces have a timestamp).
>> 
>> See first commit for a full description.
>> 
>> Signed-off-by: Lluís Vilanova <vilanova at ac.upc.edu>
>> ---

> CCing Steven Rostedt, Masami Hiramatsu, Luiz Capitulino, and LTTng folks
> who have all looked into host/guest tracing solutions.
[...]

Oh, I wasn't aware of that. I'm certainly interested in collaborating.


Cheers,
  Lluis

[lttng-dev] [Qemu-devel] [PATCH 0/6] hypertrace: Lightweight guest-to-QEMU trace channel

[Stefan Hajnoczi]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain, Size: 2575 bytes --]

On Fri, Aug 05, 2016 at 06:59:23PM +0200, Lluís Vilanova wrote:
> The hypertrace channel allows guest code to emit events in QEMU (the host) using
> its tracing infrastructure (see "docs/trace.txt"). This works in both 'system'
> and 'user' modes. That is, hypertrace is to tracing, what hypercalls are to
> system calls.
> 
> You can use this to emit an event on both guest and QEMU (host) traces to easily
> synchronize or correlate them. You could also modify you guest's tracing system
> to emit all events through the hypertrace channel, providing a unified and fully
> synchronized trace log. Another use case is timing the performance of guest code
> when optimizing TCG (QEMU traces have a timestamp).
> 
> See first commit for a full description.

This tracing approach has a high performance overhead, particularly for
SMP guests where each trace event requires writing to the global control
register.  All CPUs will be hammering this register (heavyweight vmexit)
for each trace event.

I think the folks CCed on this email all take an asynchronous approach
to avoid this performance overhead.  Synchronous means taking a VM exit
for every event.  Asynchronous means writing trace data to a buffer and
later interleaving guest data with host trace data.

LTTng Userspace Tracer is an example of the asynchronous approach.  The
trace data buffers are in shared memory.  The LTTng process can grab
buffers at appropriate times.

The ftrace virtio-serial approach has been to splice() the ftrace
buffers, resulting in efficient I/O.

Steven is working on a host/guest solution for trace-cmd.  It is also
asynchronous.  No new paravirt hardware is needed and it makes me wonder
whether the hypertrace PCI device is trying to solve the problem at the
wrong layer.

If you want to play around with asynchronous tracing, you could start
with trace/simple.c.  It has a trace buffer that is asynchronously
written out to file by a dedicated "writer" thread.

The one case where hypertrace makes sense to me is for -user tracing.
There QEMU can efficiently interleave guest and QEMU traces, although as
mentioned in the patch, I don't think the SIGSEGV approach should be
used.

I suggest stripping this series down to focus on -user.  Synchronous
tracing is not a good approach for -system emulation.

Stefan
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 473 bytes
Desc: not available
URL: <https://lists.lttng.org/pipermail/lttng-dev/attachments/20160818/e8ce56be/attachment.sig>

[lttng-dev] [Qemu-devel] [PATCH 0/6] hypertrace: Lightweight guest-to-QEMU trace channel

[Luiz Capitulino]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain, Size: 2967 bytes --]

On Thu, 18 Aug 2016 11:54:24 +0100
Stefan Hajnoczi <stefanha at gmail.com> wrote:

> On Fri, Aug 05, 2016 at 06:59:23PM +0200, Lluís Vilanova wrote:
> > The hypertrace channel allows guest code to emit events in QEMU (the host) using
> > its tracing infrastructure (see "docs/trace.txt"). This works in both 'system'
> > and 'user' modes. That is, hypertrace is to tracing, what hypercalls are to
> > system calls.
> > 
> > You can use this to emit an event on both guest and QEMU (host) traces to easily
> > synchronize or correlate them. You could also modify you guest's tracing system
> > to emit all events through the hypertrace channel, providing a unified and fully
> > synchronized trace log. Another use case is timing the performance of guest code
> > when optimizing TCG (QEMU traces have a timestamp).
> > 
> > See first commit for a full description.  

I hate the be the one asking the stupid questions, but what's
the problem this series solves? What are the use-cases? Why
existing solutions don't solve this problem? How does this
compares to existing solutions?

> This tracing approach has a high performance overhead, particularly for
> SMP guests where each trace event requires writing to the global control
> register.  All CPUs will be hammering this register (heavyweight vmexit)
> for each trace event.
> 
> I think the folks CCed on this email all take an asynchronous approach
> to avoid this performance overhead.  Synchronous means taking a VM exit
> for every event.  Asynchronous means writing trace data to a buffer and
> later interleaving guest data with host trace data.
> 
> LTTng Userspace Tracer is an example of the asynchronous approach.  The
> trace data buffers are in shared memory.  The LTTng process can grab
> buffers at appropriate times.
> 
> The ftrace virtio-serial approach has been to splice() the ftrace
> buffers, resulting in efficient I/O.

Yes. However, note that the virtio-serial device is only used to
transfer the tracing data to the host. It has no role in the
tracing process. Also, it's not required. I've been using the
networking and it works OK as long as your tracing data is not big.

> 
> Steven is working on a host/guest solution for trace-cmd.  It is also
> asynchronous.  No new paravirt hardware is needed and it makes me wonder
> whether the hypertrace PCI device is trying to solve the problem at the
> wrong layer.
> 
> If you want to play around with asynchronous tracing, you could start
> with trace/simple.c.  It has a trace buffer that is asynchronously
> written out to file by a dedicated "writer" thread.
> 
> The one case where hypertrace makes sense to me is for -user tracing.
> There QEMU can efficiently interleave guest and QEMU traces, although as
> mentioned in the patch, I don't think the SIGSEGV approach should be
> used.
> 
> I suggest stripping this series down to focus on -user.  Synchronous
> tracing is not a good approach for -system emulation.
> 
> Stefan

[lttng-dev] [Qemu-devel] [PATCH 0/6] hypertrace: Lightweight guest-to-QEMU trace channel

[Stefan Hajnoczi]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain, Size: 1759 bytes --]

On Thu, Aug 18, 2016 at 12:22:18PM +0200, Lluís Vilanova wrote:
> Stefan Hajnoczi writes:
> 
> > On Fri, Aug 05, 2016 at 06:59:23PM +0200, Lluís Vilanova wrote:
> >> The hypertrace channel allows guest code to emit events in QEMU (the host) using
> >> its tracing infrastructure (see "docs/trace.txt"). This works in both 'system'
> >> and 'user' modes. That is, hypertrace is to tracing, what hypercalls are to
> >> system calls.
> >> 
> >> You can use this to emit an event on both guest and QEMU (host) traces to easily
> >> synchronize or correlate them. You could also modify you guest's tracing system
> >> to emit all events through the hypertrace channel, providing a unified and fully
> >> synchronized trace log. Another use case is timing the performance of guest code
> >> when optimizing TCG (QEMU traces have a timestamp).
> >> 
> >> See first commit for a full description.
> >> 
> >> Signed-off-by: Lluís Vilanova <vilanova at ac.upc.edu>
> >> ---
> 
> > CCing Steven Rostedt, Masami Hiramatsu, Luiz Capitulino, and LTTng folks
> > who have all looked into host/guest tracing solutions.
> [...]
> 
> Oh, I wasn't aware of that. I'm certainly interested in collaborating.

They are working on or have worked on different approaches to host/guest
tracing.  Unfortunately there isn't an out-of-the-box solution as far as
I know.

It would be nice if there was a documented host/guest tracing approach
that didn't involve much manual setup and handled most use cases.

Stefan
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 473 bytes
Desc: not available
URL: <https://lists.lttng.org/pipermail/lttng-dev/attachments/20160818/d5b4eef1/attachment-0001.sig>

[lttng-dev] [Qemu-devel] [PATCH 0/6] hypertrace: Lightweight guest-to-QEMU trace channel

[Luiz Capitulino]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain, Size: 2405 bytes --]

On Thu, 18 Aug 2016 14:53:27 +0100
Stefan Hajnoczi <stefanha at redhat.com> wrote:

> On Thu, Aug 18, 2016 at 12:22:18PM +0200, Lluís Vilanova wrote:
> > Stefan Hajnoczi writes:
> >   
> > > On Fri, Aug 05, 2016 at 06:59:23PM +0200, Lluís Vilanova wrote:  
> > >> The hypertrace channel allows guest code to emit events in QEMU (the host) using
> > >> its tracing infrastructure (see "docs/trace.txt"). This works in both 'system'
> > >> and 'user' modes. That is, hypertrace is to tracing, what hypercalls are to
> > >> system calls.
> > >> 
> > >> You can use this to emit an event on both guest and QEMU (host) traces to easily
> > >> synchronize or correlate them. You could also modify you guest's tracing system
> > >> to emit all events through the hypertrace channel, providing a unified and fully
> > >> synchronized trace log. Another use case is timing the performance of guest code
> > >> when optimizing TCG (QEMU traces have a timestamp).
> > >> 
> > >> See first commit for a full description.
> > >> 
> > >> Signed-off-by: Lluís Vilanova <vilanova at ac.upc.edu>
> > >> ---  
> >   
> > > CCing Steven Rostedt, Masami Hiramatsu, Luiz Capitulino, and LTTng folks
> > > who have all looked into host/guest tracing solutions.  
> > [...]
> > 
> > Oh, I wasn't aware of that. I'm certainly interested in collaborating.  
> 
> They are working on or have worked on different approaches to host/guest
> tracing.  Unfortunately there isn't an out-of-the-box solution as far as
> I know.

The ftrace solution is documented here:

 https://lists.nongnu.org/archive/html/qemu-devel/2016-03/msg00887.html

This traces the guest and host kernels. It supports merging the guest
and host traces. It's extremely low latency and has helped us to
find several spikes for real-time KVM (we're talking a few to
a dozen microseconds at most).

Now, our stack actually is:

 - Guest app
 - Guest kernel
 - Host kernel
 - QEMU

QEMU already has its own tracing (which I don't know how it works).
If I had to trace the guest app, I'd certainly start off by using
LTTng. Although, we'd have to write a tool to merge and orchestrate
(wooo, cloud buzzword!) all those traces (if that's what one wants).

> It would be nice if there was a documented host/guest tracing approach
> that didn't involve much manual setup and handled most use cases.

I'd volunteer to do it, although it will take me weeks to do it.

[lttng-dev] [Qemu-devel] [PATCH 0/6] hypertrace: Lightweight guest-to-QEMU trace channel

[Steven Rostedt]

On Thu, 18 Aug 2016 11:54:24 +0100
Stefan Hajnoczi <stefanha at gmail.com> wrote:

> Steven is working on a host/guest solution for trace-cmd.  It is also
> asynchronous.  No new paravirt hardware is needed and it makes me wonder
> whether the hypertrace PCI device is trying to solve the problem at the
> wrong layer.

Yes, and I'm currently working on it again. In fact, if any of you will
be at LinuxCon in Toronto next week, I'll be presenting[1] what I have
and what I plan to complete in the near future.

-- Steve

[1] https://lcccna2016.sched.org/event/7JWL/trace-cmd-virt-server-a-status-update-steven-rostedt-red-hat?iframe=no&w=i:100;&sidebar=yes&bg=no

[lttng-dev] [Qemu-devel] [PATCH 0/6] hypertrace: Lightweight guest-to-QEMU trace channel

[Masami Hiramatsu]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain, Size: 3953 bytes --]

On Thu, 18 Aug 2016 09:37:11 -0400
Luiz Capitulino <lcapitulino at redhat.com> wrote:

> On Thu, 18 Aug 2016 11:54:24 +0100
> Stefan Hajnoczi <stefanha at gmail.com> wrote:
> 
> > On Fri, Aug 05, 2016 at 06:59:23PM +0200, Lluís Vilanova wrote:
> > > The hypertrace channel allows guest code to emit events in QEMU (the host) using
> > > its tracing infrastructure (see "docs/trace.txt"). This works in both 'system'
> > > and 'user' modes. That is, hypertrace is to tracing, what hypercalls are to
> > > system calls.
> > > 
> > > You can use this to emit an event on both guest and QEMU (host) traces to easily
> > > synchronize or correlate them. You could also modify you guest's tracing system
> > > to emit all events through the hypertrace channel, providing a unified and fully
> > > synchronized trace log. Another use case is timing the performance of guest code
> > > when optimizing TCG (QEMU traces have a timestamp).
> > > 
> > > See first commit for a full description.  
> 
> I hate the be the one asking the stupid questions, but what's
> the problem this series solves? What are the use-cases? Why
> existing solutions don't solve this problem? How does this
> compares to existing solutions?
> 
> > This tracing approach has a high performance overhead, particularly for
> > SMP guests where each trace event requires writing to the global control
> > register.  All CPUs will be hammering this register (heavyweight vmexit)
> > for each trace event.

That reminds me xenprobes.
https://www.usenix.org/legacy/event/usenix07/tech/full_papers/quynh/quynh.pdf

The paper also tried to implement kprobe-like feature for tracing events in
guest. Synchronous tracing maybe easy to read the raw trace data, but cause
big slowdown. Moreover, we can merge and synchronize the data easily after
traced.

> > 
> > I think the folks CCed on this email all take an asynchronous approach
> > to avoid this performance overhead.  Synchronous means taking a VM exit
> > for every event.  Asynchronous means writing trace data to a buffer and
> > later interleaving guest data with host trace data.
> > 
> > LTTng Userspace Tracer is an example of the asynchronous approach.  The
> > trace data buffers are in shared memory.  The LTTng process can grab
> > buffers at appropriate times.
> > 
> > The ftrace virtio-serial approach has been to splice() the ftrace
> > buffers, resulting in efficient I/O.
> 
> Yes. However, note that the virtio-serial device is only used to
> transfer the tracing data to the host. It has no role in the
> tracing process. Also, it's not required. I've been using the
> networking and it works OK as long as your tracing data is not big.

Right, the no-copy virtio-serial is for reducing the overhead as small as
possible but it also requires a special programs in both of guest and host.
(since it uses unix domain socket to transfer data without copy)
In normal case, you can just trace it as like as tracing on a remote machine
via network. That's more scalable and flexible. 


> > Steven is working on a host/guest solution for trace-cmd.  It is also
> > asynchronous.  No new paravirt hardware is needed and it makes me wonder
> > whether the hypertrace PCI device is trying to solve the problem at the
> > wrong layer.

Thanks Steven to continue that!

Regards,

> > 
> > If you want to play around with asynchronous tracing, you could start
> > with trace/simple.c.  It has a trace buffer that is asynchronously
> > written out to file by a dedicated "writer" thread.
> > 
> > The one case where hypertrace makes sense to me is for -user tracing.
> > There QEMU can efficiently interleave guest and QEMU traces, although as
> > mentioned in the patch, I don't think the SIGSEGV approach should be
> > used.
> > 
> > I suggest stripping this series down to focus on -user.  Synchronous
> > tracing is not a good approach for -system emulation.
> > 
> > Stefan
> 


-- 
Masami Hiramatsu <mhiramat at kernel.org>

[lttng-dev] [Qemu-devel] [PATCH 0/6] hypertrace: Lightweight guest-to-QEMU trace channel

[Stefan Hajnoczi]

On Thu, Aug 18, 2016 at 12:19:50PM -0400, Steven Rostedt wrote:
> On Thu, 18 Aug 2016 11:54:24 +0100
> Stefan Hajnoczi <stefanha at gmail.com> wrote:
> 
> > Steven is working on a host/guest solution for trace-cmd.  It is also
> > asynchronous.  No new paravirt hardware is needed and it makes me wonder
> > whether the hypertrace PCI device is trying to solve the problem at the
> > wrong layer.
> 
> Yes, and I'm currently working on it again. In fact, if any of you will
> be at LinuxCon in Toronto next week, I'll be presenting[1] what I have
> and what I plan to complete in the near future.
> 
> -- Steve
> 
> [1] https://lcccna2016.sched.org/event/7JWL/trace-cmd-virt-server-a-status-update-steven-rostedt-red-hat?iframe=no&w=i:100;&sidebar=yes&bg=no

Cool, that's very interesting!  I'll try to attend.

I guess LinuxCon will post a video of your talk and the slides will be
available after the talk so people not at LinuxCon will be able to check
it out too.

Stefan
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 473 bytes
Desc: not available
URL: <https://lists.lttng.org/pipermail/lttng-dev/attachments/20160819/1151ca74/attachment.sig>

[lttng-dev] [Qemu-devel] [PATCH 0/6] hypertrace: Lightweight guest-to-QEMU trace channel

[Steven Rostedt]

On Fri, 19 Aug 2016 11:02:07 +0100
Stefan Hajnoczi <stefanha at redhat.com> wrote:


> > [1] https://lcccna2016.sched.org/event/7JWL/trace-cmd-virt-server-a-status-update-steven-rostedt-red-hat?iframe=no&w=i:100;&sidebar=yes&bg=no  
> 
> Cool, that's very interesting!  I'll try to attend.

Good, see you then!

> 
> I guess LinuxCon will post a video of your talk and the slides will be
> available after the talk so people not at LinuxCon will be able to check
> it out too.

Hmm, IIRC the LinuxCon talks are seldom recorded. But the slides will be
available.

-- Steve

[lttng-dev] [Qemu-devel] [PATCH 0/6] hypertrace: Lightweight guest-to-QEMU trace channel

[Lluís Vilanova]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain, Size: 2421 bytes --]

Luiz Capitulino writes:

> On Thu, 18 Aug 2016 14:53:27 +0100
> Stefan Hajnoczi <stefanha at redhat.com> wrote:

>> On Thu, Aug 18, 2016 at 12:22:18PM +0200, Lluís Vilanova wrote:
>> > Stefan Hajnoczi writes:
>> >   
>> > > On Fri, Aug 05, 2016 at 06:59:23PM +0200, Lluís Vilanova wrote:  
>> > >> The hypertrace channel allows guest code to emit events in QEMU (the host) using
>> > >> its tracing infrastructure (see "docs/trace.txt"). This works in both 'system'
>> > >> and 'user' modes. That is, hypertrace is to tracing, what hypercalls are to
>> > >> system calls.
>> > >> 
>> > >> You can use this to emit an event on both guest and QEMU (host) traces to easily
>> > >> synchronize or correlate them. You could also modify you guest's tracing system
>> > >> to emit all events through the hypertrace channel, providing a unified and fully
>> > >> synchronized trace log. Another use case is timing the performance of guest code
>> > >> when optimizing TCG (QEMU traces have a timestamp).
>> > >> 
>> > >> See first commit for a full description.
>> > >> 
>> > >> Signed-off-by: Lluís Vilanova <vilanova at ac.upc.edu>
>> > >> ---  
>> >   
>> > > CCing Steven Rostedt, Masami Hiramatsu, Luiz Capitulino, and LTTng folks
>> > > who have all looked into host/guest tracing solutions.  
>> > [...]
>> > 
>> > Oh, I wasn't aware of that. I'm certainly interested in collaborating.  
>> 
>> They are working on or have worked on different approaches to host/guest
>> tracing.  Unfortunately there isn't an out-of-the-box solution as far as
>> I know.

> The ftrace solution is documented here:

>  https://lists.nongnu.org/archive/html/qemu-devel/2016-03/msg00887.html

> This traces the guest and host kernels. It supports merging the guest
> and host traces. It's extremely low latency and has helped us to
> find several spikes for real-time KVM (we're talking a few to
> a dozen microseconds at most).

> Now, our stack actually is:

>  - Guest app
>  - Guest kernel
>  - Host kernel
>  - QEMU

> QEMU already has its own tracing (which I don't know how it works).
> If I had to trace the guest app, I'd certainly start off by using
> LTTng. Although, we'd have to write a tool to merge and orchestrate
> (wooo, cloud buzzword!) all those traces (if that's what one wants).
[...]

One of my targets was to simplify the merge by providing known reference points
between guest and host traces.


Cheers,
  Lluis

[lttng-dev] [Qemu-devel] [PATCH 0/6] hypertrace: Lightweight guest-to-QEMU trace channel

[Lluís Vilanova]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain, Size: 3385 bytes --]

Stefan Hajnoczi writes:

> On Fri, Aug 05, 2016 at 06:59:23PM +0200, Lluís Vilanova wrote:
>> The hypertrace channel allows guest code to emit events in QEMU (the host) using
>> its tracing infrastructure (see "docs/trace.txt"). This works in both 'system'
>> and 'user' modes. That is, hypertrace is to tracing, what hypercalls are to
>> system calls.
>> 
>> You can use this to emit an event on both guest and QEMU (host) traces to easily
>> synchronize or correlate them. You could also modify you guest's tracing system
>> to emit all events through the hypertrace channel, providing a unified and fully
>> synchronized trace log. Another use case is timing the performance of guest code
>> when optimizing TCG (QEMU traces have a timestamp).
>> 
>> See first commit for a full description.

> This tracing approach has a high performance overhead, particularly for
> SMP guests where each trace event requires writing to the global control
> register.  All CPUs will be hammering this register (heavyweight vmexit)
> for each trace event.

> I think the folks CCed on this email all take an asynchronous approach
> to avoid this performance overhead.  Synchronous means taking a VM exit
> for every event.  Asynchronous means writing trace data to a buffer and
> later interleaving guest data with host trace data.

> LTTng Userspace Tracer is an example of the asynchronous approach.  The
> trace data buffers are in shared memory.  The LTTng process can grab
> buffers at appropriate times.

> The ftrace virtio-serial approach has been to splice() the ftrace
> buffers, resulting in efficient I/O.

> Steven is working on a host/guest solution for trace-cmd.  It is also
> asynchronous.  No new paravirt hardware is needed and it makes me wonder
> whether the hypertrace PCI device is trying to solve the problem at the
> wrong layer.

> If you want to play around with asynchronous tracing, you could start
> with trace/simple.c.  It has a trace buffer that is asynchronously
> written out to file by a dedicated "writer" thread.

> The one case where hypertrace makes sense to me is for -user tracing.
> There QEMU can efficiently interleave guest and QEMU traces, although as
> mentioned in the patch, I don't think the SIGSEGV approach should be
> used.

> I suggest stripping this series down to focus on -user.  Synchronous
> tracing is not a good approach for -system emulation.

As I said, I wanted to implement a simple way to provide common reference points
in guest and host traces. For that, a *synchronous* guest-to-host channel is the
simplest way to do it (otherwise you're back to square one).

Another reason to have this is that I wanted to attach a dtrace/systemtap hook
to the hypertrace event, and from there dynamically control QEMU's tracing. Then
you can insert simple calls to hypertrace on your guest code to tell your dtrace
hooks when to enable/disable tracing of certain QEMU events (like guest instr
traces).

Unfortuntely, I've been unable to to make dtrace recognise QEMU's events (I'm
only able to see the host kernel events). If someone with more experience on it
can help me use dtrace with QEMU's events, I'll also add the supporting library
to let dtrace do the callout to QEMU's moitor interface and control the events,
and add a prperly useful example of that on the hypertrace docs (which was my
original intention).


Thanks,
  Lluis

[lttng-dev] [Qemu-devel] [PATCH 0/6] hypertrace: Lightweight guest-to-QEMU trace channel

[Stefan Hajnoczi]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain, Size: 944 bytes --]

On Sun, Aug 21, 2016 at 02:32:34PM +0200, Lluís Vilanova wrote:
> Unfortuntely, I've been unable to to make dtrace recognise QEMU's events (I'm
> only able to see the host kernel events). If someone with more experience on it
> can help me use dtrace with QEMU's events, I'll also add the supporting library
> to let dtrace do the callout to QEMU's moitor interface and control the events,
> and add a prperly useful example of that on the hypertrace docs (which was my
> original intention).

Which "dtrace" and host OS are you using?

QEMU builds with static user-space probes.  You need to tell DTrace or
SystemTap to enable those probes in order to record trace data.

Stefan
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 473 bytes
Desc: not available
URL: <https://lists.lttng.org/pipermail/lttng-dev/attachments/20160823/90a66580/attachment.sig>

[lttng-dev] [Qemu-devel] [PATCH 0/6] hypertrace: Lightweight guest-to-QEMU trace channel

[Lluís Vilanova]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain, Size: 1553 bytes --]

Stefan Hajnoczi writes:

> On Sun, Aug 21, 2016 at 02:32:34PM +0200, Lluís Vilanova wrote:
>> Unfortuntely, I've been unable to to make dtrace recognise QEMU's events (I'm
>> only able to see the host kernel events). If someone with more experience on it
>> can help me use dtrace with QEMU's events, I'll also add the supporting library
>> to let dtrace do the callout to QEMU's moitor interface and control the events,
>> and add a prperly useful example of that on the hypertrace docs (which was my
>> original intention).

> Which "dtrace" and host OS are you using?

> QEMU builds with static user-space probes.  You need to tell DTrace or
> SystemTap to enable those probes in order to record trace data.

I'm using debian on a 4.6.0-1-amd64 kernel with systemtap 3.0.6.

I just gave it another try, and works if I use probes like:

  process("<binary-path>").mark("<event-name>")

although they don't seem to appear on "stap -l" or anything like that (I cannot
find a "qemu" provider). But I'm still unable to print the event values. This:

  probe process("./install/vanilla/bin/qemu-system-i386").mark("guest_mem_before_exec")
  {
      printf("%p %lx %d\n", $arg1, $arg2, $arg3)
  }

always prints "0x0 0x0 0", which is clearly wrong (other backend on the same
build print the correct values).

Also, I'm still not sure how to interact with QEMU's monitor interface from
within the probe code (probes execute in kernel mode, including "guru mode"
code).

If anybody can shed some like into any of this, I'd appreaciate it.


Cheers,
  Lluis

[lttng-dev] [Qemu-devel] [PATCH 0/6] hypertrace: Lightweight guest-to-QEMU trace channel

[Stefan Hajnoczi]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain, Size: 2153 bytes --]

On Wed, Aug 24, 2016 at 12:25:54PM +0200, Lluís Vilanova wrote:
> Stefan Hajnoczi writes:
> 
> > On Sun, Aug 21, 2016 at 02:32:34PM +0200, Lluís Vilanova wrote:
> >> Unfortuntely, I've been unable to to make dtrace recognise QEMU's events (I'm
> >> only able to see the host kernel events). If someone with more experience on it
> >> can help me use dtrace with QEMU's events, I'll also add the supporting library
> >> to let dtrace do the callout to QEMU's moitor interface and control the events,
> >> and add a prperly useful example of that on the hypertrace docs (which was my
> >> original intention).
> 
> > Which "dtrace" and host OS are you using?
> 
> > QEMU builds with static user-space probes.  You need to tell DTrace or
> > SystemTap to enable those probes in order to record trace data.
> 
> I'm using debian on a 4.6.0-1-amd64 kernel with systemtap 3.0.6.
> 
> I just gave it another try, and works if I use probes like:
> 
>   process("<binary-path>").mark("<event-name>")
> 
> although they don't seem to appear on "stap -l" or anything like that (I cannot
> find a "qemu" provider). But I'm still unable to print the event values. This:

The following enumerates events for me:

$ sudo stap -L 'process("/usr/bin/qemu-system-x86_64").mark("*")'
process("/usr/bin/qemu-system-x86_64").mark("alsa_no_frames") $arg1:long
process("/usr/bin/qemu-system-x86_64").mark("alsa_pollout") $arg1:long $arg2:long
process("/usr/bin/qemu-system-x86_64").mark("alsa_read_zero") $arg1:long
...

You can also use /usr/share/systemtap/tapset/qemu-system-x86_64.stp.
I'm running Fedora 24 with qemu-system-x86 2.6.0-5.fc24 and SystemTap
3.0-3.fc24.

> Also, I'm still not sure how to interact with QEMU's monitor interface from
> within the probe code (probes execute in kernel mode, including "guru mode"
> code).

When SystemTap is used the QEMU monitor interface does nothing.

Stefan
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 473 bytes
Desc: not available
URL: <https://lists.lttng.org/pipermail/lttng-dev/attachments/20160829/b79c3b58/attachment.sig>

[lttng-dev] [Qemu-devel] [PATCH 0/6] hypertrace: Lightweight guest-to-QEMU trace channel

[Lluís Vilanova]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain, Size: 3060 bytes --]

Stefan Hajnoczi writes:

> On Wed, Aug 24, 2016 at 12:25:54PM +0200, Lluís Vilanova wrote:
>> Stefan Hajnoczi writes:
>> 
>> > On Sun, Aug 21, 2016 at 02:32:34PM +0200, Lluís Vilanova wrote:
>> >> Unfortuntely, I've been unable to to make dtrace recognise QEMU's events (I'm
>> >> only able to see the host kernel events). If someone with more experience on it
>> >> can help me use dtrace with QEMU's events, I'll also add the supporting library
>> >> to let dtrace do the callout to QEMU's moitor interface and control the events,
>> >> and add a prperly useful example of that on the hypertrace docs (which was my
>> >> original intention).
>> 
>> > Which "dtrace" and host OS are you using?
>> 
>> > QEMU builds with static user-space probes.  You need to tell DTrace or
>> > SystemTap to enable those probes in order to record trace data.
>> 
>> I'm using debian on a 4.6.0-1-amd64 kernel with systemtap 3.0.6.
>> 
>> I just gave it another try, and works if I use probes like:
>> 
>> process("<binary-path>").mark("<event-name>")
>> 
>> although they don't seem to appear on "stap -l" or anything like that (I cannot
>> find a "qemu" provider). But I'm still unable to print the event values. This:

> The following enumerates events for me:

> $ sudo stap -L 'process("/usr/bin/qemu-system-x86_64").mark("*")'
> process("/usr/bin/qemu-system-x86_64").mark("alsa_no_frames") $arg1:long
> process("/usr/bin/qemu-system-x86_64").mark("alsa_pollout") $arg1:long $arg2:long
> process("/usr/bin/qemu-system-x86_64").mark("alsa_read_zero") $arg1:long
> ...

> You can also use /usr/share/systemtap/tapset/qemu-system-x86_64.stp.
> I'm running Fedora 24 with qemu-system-x86 2.6.0-5.fc24 and SystemTap
> 3.0-3.fc24.

I meant I cannot just use "qemu" as a provider name. Later I discovered that
"process()" does work.


>> Also, I'm still not sure how to interact with QEMU's monitor interface from
>> within the probe code (probes execute in kernel mode, including "guru mode"
>> code).

> When SystemTap is used the QEMU monitor interface does nothing.

That's not what I've experienced. I was able to use a stap script to change the
tracing state of events:

   #!/usr/bin/env stap

   %{
   #include </home/lluis/Projects/qemu-dbi-test/test.h>
   %}

   function event:long(cpu:long, addr:long, info:long)
   %{
       char *argv[4] = {"/bin/sh", "-c", "echo 'trace-event * off' | telnet localhost 1234", NULL};
       call_usermodehelper(argv[0], argv, NULL, UMH_WAIT_EXEC);
       STAP_RETURN(0);
   %}

   probe begin {
       printf("hello\n")
   }
   probe process("./install/vanilla/bin/qemu-system-i386").mark("guest_mem_before_exec")
   {
       printf("%x %d %d\n", $arg1, $arg2, $arg3)
       event($arg1, $arg2, $arg3)
       exit()
   }

The only caveat is that you must pass the "-g" argument to stap.

Also, for some reason the printf in the probe always prints zeros, no matter
what the actual event receives (I've debugged QEMU down to the call to the
auto-generated stap functions). Could this be an error in systemtap?


Thanks,
  Lluis

[lttng-dev] [Qemu-devel] [PATCH 0/6] hypertrace: Lightweight guest-to-QEMU trace channel

[Stefan Hajnoczi]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain, Size: 1876 bytes --]

On Mon, Aug 29, 2016 at 08:46:02PM +0200, Lluís Vilanova wrote:
> >> Also, I'm still not sure how to interact with QEMU's monitor interface from
> >> within the probe code (probes execute in kernel mode, including "guru mode"
> >> code).
> 
> > When SystemTap is used the QEMU monitor interface does nothing.
> 
> That's not what I've experienced. I was able to use a stap script to change the
> tracing state of events:
> 
>    #!/usr/bin/env stap
> 
>    %{
>    #include </home/lluis/Projects/qemu-dbi-test/test.h>
>    %}
> 
>    function event:long(cpu:long, addr:long, info:long)
>    %{
>        char *argv[4] = {"/bin/sh", "-c", "echo 'trace-event * off' | telnet localhost 1234", NULL};
>        call_usermodehelper(argv[0], argv, NULL, UMH_WAIT_EXEC);
>        STAP_RETURN(0);
>    %}
> 
>    probe begin {
>        printf("hello\n")
>    }
>    probe process("./install/vanilla/bin/qemu-system-i386").mark("guest_mem_before_exec")
>    {
>        printf("%x %d %d\n", $arg1, $arg2, $arg3)
>        event($arg1, $arg2, $arg3)
>        exit()
>    }
> 
> The only caveat is that you must pass the "-g" argument to stap.
> 
> Also, for some reason the printf in the probe always prints zeros, no matter
> what the actual event receives (I've debugged QEMU down to the call to the
> auto-generated stap functions). Could this be an error in systemtap?

It's strange that arguments do not have valid values.  Debugging the
stap functions is the next step if you want to figure out what happened.
I've never had this issue before so maybe something with Debian
SystemTap userspace probes is broken.

Stefan
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 473 bytes
Desc: not available
URL: <https://lists.lttng.org/pipermail/lttng-dev/attachments/20160831/b8d7c90c/attachment.sig>