[ltt-dev] Problem on combine kernel trace and userspace trace.

[ltt-dev] Problem on combine kernel trace and userspace trace.

[Sam Liao]

Hi,

I'm trying to combine the kernel trace and userspace trace, and it
seems that the
the userspace events can not be recognized to the right process with the kernel
events. I'm not sure if this is a problem of timestamp between kernel
and userspace.

Here is the environment I'm using:
----
Linux Trace Toolkit Visualizer 0.12.35-20082010
linux kernel 2.6.34.6
ust-0.4. ( I update the clock to use clock_gettime).
----

Is there anything I missed? Also I'm thinking about one problem, if
ust using clock_gettime to
get the nonseconds timestamp which is actually the timestamp that the user
application is doing the clock_gettime system call instead the
userspace event, how ust
deal with such difference and get this fixed when trace combination?

Thanks,
-Sam

[ltt-dev] Problem on combine kernel trace and userspace trace.

[Julien Desfossez]

Hi Sam,

Thanks for posting this, regarding the issue with time synchronization
of the traces, we are actually working exactly on it right now.
The main problem for now is that the events are recorded in cycles in
the trace which means we need to know the TSC frequency to convert back
in sec.nsec in LTTV. From UST we don't have a clear way to access this
information, so we are now working on a new option in clock_gettime to
use the LTTng trace clock infrastructure to make it easy to synchronise
kernel and ust traces. I will post more details soon.

Regarding the possibility of combining the userspace and kernelspace
events for the same process I'm not sure exactly how it works, from what
I can see the PID is 0 in UST traces so the traces cannot really be
merged but I will let the people more familiar with UST answer this one.

Thanks,

Julien


On 10-11-03 05:24 AM, Sam Liao wrote:
> Hi,
> 
> I'm trying to combine the kernel trace and userspace trace, and it
> seems that the
> the userspace events can not be recognized to the right process with the kernel
> events. I'm not sure if this is a problem of timestamp between kernel
> and userspace.
> 
> Here is the environment I'm using:
> ----
> Linux Trace Toolkit Visualizer 0.12.35-20082010
> linux kernel 2.6.34.6
> ust-0.4. ( I update the clock to use clock_gettime).
> ----
> 
> Is there anything I missed? Also I'm thinking about one problem, if
> ust using clock_gettime to
> get the nonseconds timestamp which is actually the timestamp that the user
> application is doing the clock_gettime system call instead the
> userspace event, how ust
> deal with such difference and get this fixed when trace combination?
> 
> Thanks,
> -Sam
> 
> _______________________________________________
> ltt-dev mailing list
> ltt-dev at lists.casi.polymtl.ca
> http://lists.casi.polymtl.ca/cgi-bin/mailman/listinfo/ltt-dev

[ltt-dev] Problem on combine kernel trace and userspace trace.

[Sam Liao]

Hi Julien,

Thanks for your answer.

On Wed, Nov 3, 2010 at 10:04 PM, Julien Desfossez
<julien.desfossez at polymtl.ca> wrote:
> Hi Sam,
>
> Thanks for posting this, regarding the issue with time synchronization
> of the traces, we are actually working exactly on it right now.
> The main problem for now is that the events are recorded in cycles in
> the trace which means we need to know the TSC frequency to convert back
> in sec.nsec in LTTV. From UST we don't have a clear way to access this
> information, so we are now working on a new option in clock_gettime to
> use the LTTng trace clock infrastructure to make it easy to synchronise
> kernel and ust traces. I will post more details soon.
>

So using different clock sources brings this problem.

> Regarding the possibility of combining the userspace and kernelspace
> events for the same process I'm not sure exactly how it works, from what
> I can see the PID is 0 in UST traces so the traces cannot really be
> merged but I will let the people more familiar with UST answer this one.
>

I think that even there is no PID related information in the UST trace, if
the kernel trace's timestamp is consistent with the ust's, then we can get the
ust event's pid context from context switch("sched_change") related kernel
events after combining the kernel and ust trace.

Thanks,
-Sam

> Thanks,
>
> Julien
>
>
> On 10-11-03 05:24 AM, Sam Liao wrote:
>> Hi,
>>
>> I'm trying to combine the kernel trace and userspace trace, and it
>> seems that the
>> the userspace events can not be recognized to the right process with the kernel
>> events. I'm not sure if this is a problem of timestamp between kernel
>> and userspace.
>>
>> Here is the environment I'm using:
>> ----
>> Linux Trace Toolkit Visualizer 0.12.35-20082010
>> linux kernel 2.6.34.6
>> ust-0.4. ( I update the clock to use clock_gettime).
>> ----
>>
>> Is there anything I missed? Also I'm thinking about one problem, if
>> ust using clock_gettime to
>> get the nonseconds timestamp which is actually the timestamp that the user
>> application is doing the clock_gettime system call instead the
>> userspace event, how ust
>> deal with such difference and get this fixed when trace combination?
>>
>> Thanks,
>> -Sam
>>
>> _______________________________________________
>> ltt-dev mailing list
>> ltt-dev at lists.casi.polymtl.ca
>> http://lists.casi.polymtl.ca/cgi-bin/mailman/listinfo/ltt-dev
>

[ltt-dev] Problem on combine kernel trace and userspace trace.

[Mathieu Desnoyers]

* Sam Liao (phyomh at gmail.com) wrote:
> Hi,
> 
> I'm trying to combine the kernel trace and userspace trace, and it
> seems that the
> the userspace events can not be recognized to the right process with the kernel
> events. I'm not sure if this is a problem of timestamp between kernel
> and userspace.
> 
> Here is the environment I'm using:
> ----
> Linux Trace Toolkit Visualizer 0.12.35-20082010
> linux kernel 2.6.34.6
> ust-0.4. ( I update the clock to use clock_gettime).

If you are on x86 or powerpc, using the clock source that does not use
clock_gettime would directly read the timestamp counter value, which
should work. Make sure your architecture (on x86) has synchronized
contant TSCs. LTTng complains about using a work-around in dmesg if it
is not the case. If lttng uses the workaround on x86, then there is
currently no mean to gather synchronized ust-lttng traces. As Julien
said, we're currently working on a vdso solution to make this all work.

Thanks,

Mathieu

> ----
> 
> Is there anything I missed? Also I'm thinking about one problem, if
> ust using clock_gettime to
> get the nonseconds timestamp which is actually the timestamp that the user
> application is doing the clock_gettime system call instead the
> userspace event, how ust
> deal with such difference and get this fixed when trace combination?
> 
> Thanks,
> -Sam
> 
> _______________________________________________
> ltt-dev mailing list
> ltt-dev at lists.casi.polymtl.ca
> http://lists.casi.polymtl.ca/cgi-bin/mailman/listinfo/ltt-dev
> 

-- 
Mathieu Desnoyers
Operating System Efficiency R&D Consultant
EfficiOS Inc.
http://www.efficios.com

[ltt-dev] Problem on combine kernel trace and userspace trace.

[Sam Liao]

On Wed, Nov 3, 2010 at 11:13 PM, Mathieu Desnoyers
<compudj at krystal.dyndns.org> wrote:
> * Sam Liao (phyomh at gmail.com) wrote:
>> Hi,
>>
>> I'm trying to combine the kernel trace and userspace trace, and it
>> seems that the
>> the userspace events can not be recognized to the right process with the kernel
>> events. I'm not sure if this is a problem of timestamp between kernel
>> and userspace.
>>
>> Here is the environment I'm using:
>> ----
>> Linux Trace Toolkit Visualizer 0.12.35-20082010
>> linux kernel 2.6.34.6
>> ust-0.4. ( I update the clock to use clock_gettime).
>
> If you are on x86 or powerpc, using the clock source that does not use
> clock_gettime would directly read the timestamp counter value, which
> should work. Make sure your architecture (on x86) has synchronized
> contant TSCs. LTTng complains about using a work-around in dmesg if it
> is not the case. If lttng uses the workaround on x86, then there is
> currently no mean to gather synchronized ust-lttng traces. As Julien
> said, we're currently working on a vdso solution to make this all work.
>

Thanks, after apply with the "rdtsc" clock source and set up the correct
clock frequency,  the kernel trace and ust trace seems to be consistent
now.

Thanks,
-Sam

> Thanks,
>
> Mathieu
>
>> ----
>>
>> Is there anything I missed? Also I'm thinking about one problem, if
>> ust using clock_gettime to
>> get the nonseconds timestamp which is actually the timestamp that the user
>> application is doing the clock_gettime system call instead the
>> userspace event, how ust
>> deal with such difference and get this fixed when trace combination?
>>
>> Thanks,
>> -Sam
>>
>> _______________________________________________
>> ltt-dev mailing list
>> ltt-dev at lists.casi.polymtl.ca
>> http://lists.casi.polymtl.ca/cgi-bin/mailman/listinfo/ltt-dev
>>
>
> --
> Mathieu Desnoyers
> Operating System Efficiency R&D Consultant
> EfficiOS Inc.
> http://www.efficios.com
>