* [lttng-dev] baddr feature triggers deadlock in lttng-ust
2014-02-27 21:30 ` [lttng-dev] baddr feature triggers deadlock in lttng-ust Mathieu Desnoyers
@ 2014-02-27 23:37 ` Alexandre Montplaisir
2014-02-28 0:07 ` Mathieu Desnoyers
2014-02-28 9:33 ` Woegerer, Paul
2014-02-28 12:38 ` Woegerer, Paul
2 siblings, 1 reply; 6+ messages in thread
From: Alexandre Montplaisir @ 2014-02-27 23:37 UTC (permalink / raw)
Could it be hidden behind a non-default configure option, instead of
completely removed?
Alex
On 14-02-27 04:30 PM, Mathieu Desnoyers wrote:
> Hi Paul,
>
> Please see http://bugs.lttng.org/issues/745, which explains a deadlock
> we just found out.
>
> We might have to disable the baddr dump feature, since our release is
> tomorrow. We'll have to re-think our locking to protect operations
> using the dynamic loader lock against fork, clone, and daemon.
>
> Thanks,
>
> Mathieu
>
^ permalink raw reply [flat|nested] 6+ messages in thread
* [lttng-dev] baddr feature triggers deadlock in lttng-ust
2014-02-27 23:37 ` Alexandre Montplaisir
@ 2014-02-28 0:07 ` Mathieu Desnoyers
0 siblings, 0 replies; 6+ messages in thread
From: Mathieu Desnoyers @ 2014-02-28 0:07 UTC (permalink / raw)
As fixing it will very likely require a major rethinking of the locking strategy
for the entire UST, I'm not very keen to make that kind of change within stable
versions.
Unless an elegant solution shows up by tomorrow, I plan to disable the feature
(leaving the code there), and think up a solution for 2.5.
Thoughts ?
Thanks,
Mathieu
----- Original Message -----
> From: "Alexandre Montplaisir" <alexmonthy@voxpopuli.im>
> To: "Mathieu Desnoyers" <mathieu.desnoyers at efficios.com>
> Cc: "Paul Woegerer" <paul_woegerer at mentor.com>, lttng-dev at lists.lttng.org
> Sent: Thursday, February 27, 2014 6:37:23 PM
> Subject: Re: [lttng-dev] baddr feature triggers deadlock in lttng-ust
>
> Could it be hidden behind a non-default configure option, instead of
> completely removed?
>
> Alex
>
>
> On 14-02-27 04:30 PM, Mathieu Desnoyers wrote:
> > Hi Paul,
> >
> > Please see http://bugs.lttng.org/issues/745, which explains a deadlock
> > we just found out.
> >
> > We might have to disable the baddr dump feature, since our release is
> > tomorrow. We'll have to re-think our locking to protect operations
> > using the dynamic loader lock against fork, clone, and daemon.
> >
> > Thanks,
> >
> > Mathieu
> >
>
>
--
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com
^ permalink raw reply [flat|nested] 6+ messages in thread
* [lttng-dev] baddr feature triggers deadlock in lttng-ust
2014-02-27 21:30 ` [lttng-dev] baddr feature triggers deadlock in lttng-ust Mathieu Desnoyers
2014-02-27 23:37 ` Alexandre Montplaisir
@ 2014-02-28 9:33 ` Woegerer, Paul
2014-02-28 12:38 ` Woegerer, Paul
2 siblings, 0 replies; 6+ messages in thread
From: Woegerer, Paul @ 2014-02-28 9:33 UTC (permalink / raw)
Hi Mathieu,
this is very unfortunate. I tested forks as well but couldn?t find any
issues on x86_64. Ironically, last week we found an issue that matches
your description on PowerPC but since we could not reproduce it on x86 I
blamed it on the peculiarities of that platform and worked on seemingly
higher priority issues instead.
For the sake of damage containment I just posted:
http://lists.lttng.org/pipermail/lttng-dev/2014-February/022494.html
to the list.
Not having that feature in 2.4 (at least in the experimental form
suggested above) would be a disaster for us.
Meanwhile I'm trying to think about a more fundamental solution to the
problem. But I'm afraid your comment about rethinking of the locking
strategy for the entire UST might come true.
Thanks,
Paul
On 02/27/2014 10:30 PM, Mathieu Desnoyers wrote:
> Hi Paul,
>
> Please see http://bugs.lttng.org/issues/745, which explains a deadlock
> we just found out.
>
> We might have to disable the baddr dump feature, since our release is
> tomorrow. We'll have to re-think our locking to protect operations
> using the dynamic loader lock against fork, clone, and daemon.
>
> Thanks,
>
> Mathieu
>
--
Paul Woegerer, SW Development Engineer
Sourcery Analyzer <http://go.mentor.com/sourceryanalyzer>
Mentor Graphics, Embedded Software Division
^ permalink raw reply [flat|nested] 6+ messages in thread
* [lttng-dev] baddr feature triggers deadlock in lttng-ust
2014-02-27 21:30 ` [lttng-dev] baddr feature triggers deadlock in lttng-ust Mathieu Desnoyers
2014-02-27 23:37 ` Alexandre Montplaisir
2014-02-28 9:33 ` Woegerer, Paul
@ 2014-02-28 12:38 ` Woegerer, Paul
2014-02-28 14:05 ` Mathieu Desnoyers
2 siblings, 1 reply; 6+ messages in thread
From: Woegerer, Paul @ 2014-02-28 12:38 UTC (permalink / raw)
I have looked a bit into the issue...
The ust_locking inside the dl_iterate_phdr triggers the deadlock.
If I just collect the base address info inside dl_iterate_phdr and dump
the collected data with trace_baddr afterwards (outside the
dl_iterate_phdr iteration) the deadlock will be prevented.
I'm now going to create a patch that implements this approach.
Thanks,
Paul
On 02/27/2014 10:30 PM, Mathieu Desnoyers wrote:
> Hi Paul,
>
> Please see http://bugs.lttng.org/issues/745, which explains a deadlock
> we just found out.
>
> We might have to disable the baddr dump feature, since our release is
> tomorrow. We'll have to re-think our locking to protect operations
> using the dynamic loader lock against fork, clone, and daemon.
>
> Thanks,
>
> Mathieu
>
--
Paul Woegerer, SW Development Engineer
Sourcery Analyzer <http://go.mentor.com/sourceryanalyzer>
Mentor Graphics, Embedded Software Division
^ permalink raw reply [flat|nested] 6+ messages in thread
* [lttng-dev] baddr feature triggers deadlock in lttng-ust
2014-02-28 12:38 ` Woegerer, Paul
@ 2014-02-28 14:05 ` Mathieu Desnoyers
0 siblings, 0 replies; 6+ messages in thread
From: Mathieu Desnoyers @ 2014-02-28 14:05 UTC (permalink / raw)
----- Original Message -----
> From: "Paul Woegerer" <Paul_Woegerer@mentor.com>
> To: "Mathieu Desnoyers" <mathieu.desnoyers at efficios.com>, lttng-dev at lists.lttng.org, "CS ESD-Analyzer"
> <esd-analyzer at mentor.com>
> Sent: Friday, February 28, 2014 7:38:55 AM
> Subject: Re: baddr feature triggers deadlock in lttng-ust
>
> I have looked a bit into the issue...
>
> The ust_locking inside the dl_iterate_phdr triggers the deadlock.
>
> If I just collect the base address info inside dl_iterate_phdr and dump
> the collected data with trace_baddr afterwards (outside the
> dl_iterate_phdr iteration) the deadlock will be prevented.
>
> I'm now going to create a patch that implements this approach.
I suspect that having the UST lock nested inside the dynamic loader lock
only makes the problem happen more consistently, but that the basic issue
is that the dynamic loader lock should not be held while another thread
concurrently calls fork/clone/daemon. It seems to leave the child process
in a state where the lock state is stale (taken).
Thanks,
Mathieu
>
> Thanks,
> Paul
>
> On 02/27/2014 10:30 PM, Mathieu Desnoyers wrote:
> > Hi Paul,
> >
> > Please see http://bugs.lttng.org/issues/745, which explains a deadlock
> > we just found out.
> >
> > We might have to disable the baddr dump feature, since our release is
> > tomorrow. We'll have to re-think our locking to protect operations
> > using the dynamic loader lock against fork, clone, and daemon.
> >
> > Thanks,
> >
> > Mathieu
> >
>
>
> --
> Paul Woegerer, SW Development Engineer
> Sourcery Analyzer <http://go.mentor.com/sourceryanalyzer>
> Mentor Graphics, Embedded Software Division
>
>
--
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com
^ permalink raw reply [flat|nested] 6+ messages in thread