* [lttng-dev] Deadlock in call_rcu_thread when destroy rculfhash node with nested rculfhash @ 2016-10-19 9:56 Evgeniy Ivanov 2016-10-19 10:03 ` Evgeniy Ivanov 0 siblings, 1 reply; 4+ messages in thread From: Evgeniy Ivanov @ 2016-10-19 9:56 UTC (permalink / raw) Hi, Each node of top level rculfhash has nested rculfhash. Some thread clears the top level map and then uses rcu_barrier() to wait until everything is destroyed (it is done to check leaks). Recently it started to dead lock sometimes with following stacks: Thread1: __poll cds_lfht_destroy <---- nested map ... free_Node(rcu_head*) <----- node of top level map call_rcu_thread Thread2: syscall rcu_barrier_qsbr destroy_all main Did call_rcu_thread dead lock with barrier thread? Or is it some kind of internal deadlock because of nested maps? -- Cheers, Evgeniy -------------- next part -------------- An HTML attachment was scrubbed... URL: <https://lists.lttng.org/pipermail/lttng-dev/attachments/20161019/96d17576/attachment.html> ^ permalink raw reply [flat|nested] 4+ messages in thread
* [lttng-dev] Deadlock in call_rcu_thread when destroy rculfhash node with nested rculfhash 2016-10-19 9:56 [lttng-dev] Deadlock in call_rcu_thread when destroy rculfhash node with nested rculfhash Evgeniy Ivanov @ 2016-10-19 10:03 ` Evgeniy Ivanov 2016-10-19 15:03 ` Mathieu Desnoyers 0 siblings, 1 reply; 4+ messages in thread From: Evgeniy Ivanov @ 2016-10-19 10:03 UTC (permalink / raw) Sorry, found partial answer in docs which state that cds_lfht_destroy should not be called from a call_rcu thread context. Why does this limitation exists? On Wed, Oct 19, 2016 at 12:56 PM, Evgeniy Ivanov <i at eivanov.com> wrote: > Hi, > > Each node of top level rculfhash has nested rculfhash. Some thread clears > the top level map and then uses rcu_barrier() to wait until everything is > destroyed (it is done to check leaks). Recently it started to dead lock > sometimes with following stacks: > > Thread1: > > __poll > cds_lfht_destroy <---- nested map > ... > free_Node(rcu_head*) <----- node of top level map > call_rcu_thread > > Thread2: > > syscall > rcu_barrier_qsbr > destroy_all > main > > > Did call_rcu_thread dead lock with barrier thread? Or is it some kind of > internal deadlock because of nested maps? > > > -- > Cheers, > Evgeniy > -- Cheers, Evgeniy -------------- next part -------------- An HTML attachment was scrubbed... URL: <https://lists.lttng.org/pipermail/lttng-dev/attachments/20161019/cb8771fc/attachment-0001.html> ^ permalink raw reply [flat|nested] 4+ messages in thread
* [lttng-dev] Deadlock in call_rcu_thread when destroy rculfhash node with nested rculfhash 2016-10-19 10:03 ` Evgeniy Ivanov @ 2016-10-19 15:03 ` Mathieu Desnoyers 2016-10-21 8:19 ` Evgeniy Ivanov 0 siblings, 1 reply; 4+ messages in thread From: Mathieu Desnoyers @ 2016-10-19 15:03 UTC (permalink / raw) This is because we use call_rcu internally to trigger the hash table resize. In cds_lfht_destroy, we start by waiting for "in-flight" resize to complete. Unfortunately, this requires that call_rcu worker thread progresses. If cds_lfht_destroy is called from the call_rcu worker thread, it will wait forever. One alternative would be to implement our own worker thread scheme for the rcu HT resize rather than use the call_rcu worker thread. This would simplify cds_lfht_destroy requirements a lot. Ideally I'd like to re-use all the call_rcu work dispatch/worker handling scheme, just as a separate work queue. Thoughts ? Thanks, Mathieu ----- On Oct 19, 2016, at 6:03 AM, Evgeniy Ivanov <i at eivanov.com> wrote: > Sorry, found partial answer in docs which state that cds_lfht_destroy should not > be called from a call_rcu thread context. Why does this limitation exists? > On Wed, Oct 19, 2016 at 12:56 PM, Evgeniy Ivanov < [ mailto:i at eivanov.com | > i at eivanov.com ] > wrote: >> Hi, >> Each node of top level rculfhash has nested rculfhash. Some thread clears the >> top level map and then uses rcu_barrier() to wait until everything is destroyed >> (it is done to check leaks). Recently it started to dead lock sometimes with >> following stacks: >> Thread1: >> __poll >> cds_lfht_destroy <---- nested map >> ... >> free_Node(rcu_head*) <----- node of top level map >> call_rcu_thread >> Thread2: >> syscall >> rcu_barrier_qsbr >> destroy_all >> main >> Did call_rcu_thread dead lock with barrier thread? Or is it some kind of >> internal deadlock because of nested maps? >> -- >> Cheers, >> Evgeniy > -- > Cheers, > Evgeniy > _______________________________________________ > lttng-dev mailing list > lttng-dev at lists.lttng.org > https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev -- Mathieu Desnoyers EfficiOS Inc. http://www.efficios.com -------------- next part -------------- An HTML attachment was scrubbed... URL: <https://lists.lttng.org/pipermail/lttng-dev/attachments/20161019/6522ba98/attachment.html> ^ permalink raw reply [flat|nested] 4+ messages in thread
* [lttng-dev] Deadlock in call_rcu_thread when destroy rculfhash node with nested rculfhash 2016-10-19 15:03 ` Mathieu Desnoyers @ 2016-10-21 8:19 ` Evgeniy Ivanov 0 siblings, 0 replies; 4+ messages in thread From: Evgeniy Ivanov @ 2016-10-21 8:19 UTC (permalink / raw) On Wed, Oct 19, 2016 at 6:03 PM, Mathieu Desnoyers < mathieu.desnoyers at efficios.com> wrote: > This is because we use call_rcu internally to trigger the hash table > resize. > > In cds_lfht_destroy, we start by waiting for "in-flight" resize to > complete. > Unfortunately, this requires that call_rcu worker thread progresses. If > cds_lfht_destroy is called from the call_rcu worker thread, it will wait > forever. > > One alternative would be to implement our own worker thread scheme > for the rcu HT resize rather than use the call_rcu worker thread. This > would simplify cds_lfht_destroy requirements a lot. > > Ideally I'd like to re-use all the call_rcu work dispatch/worker handling > scheme, just as a separate work queue. > > Thoughts ? > Thank you for explaining. Sounds like a plan: in our prod there is no issue with having extra thread for table resizes. And nested tables is important feature. > Thanks, > > Mathieu > > ----- On Oct 19, 2016, at 6:03 AM, Evgeniy Ivanov <i at eivanov.com> wrote: > > Sorry, found partial answer in docs which state that cds_lfht_destroy > should not be called from a call_rcu thread context. Why does this > limitation exists? > > On Wed, Oct 19, 2016 at 12:56 PM, Evgeniy Ivanov <i at eivanov.com> wrote: > >> Hi, >> >> Each node of top level rculfhash has nested rculfhash. Some thread clears >> the top level map and then uses rcu_barrier() to wait until everything is >> destroyed (it is done to check leaks). Recently it started to dead lock >> sometimes with following stacks: >> >> Thread1: >> >> __poll >> cds_lfht_destroy <---- nested map >> ... >> free_Node(rcu_head*) <----- node of top level map >> call_rcu_thread >> >> Thread2: >> >> syscall >> rcu_barrier_qsbr >> destroy_all >> main >> >> >> Did call_rcu_thread dead lock with barrier thread? Or is it some kind of >> internal deadlock because of nested maps? >> >> >> -- >> Cheers, >> Evgeniy >> > > > > -- > Cheers, > Evgeniy > > _______________________________________________ > lttng-dev mailing list > lttng-dev at lists.lttng.org > https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev > > > -- > Mathieu Desnoyers > EfficiOS Inc. > http://www.efficios.com > > _______________________________________________ > lttng-dev mailing list > lttng-dev at lists.lttng.org > https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev > > -- Cheers, Evgeniy -------------- next part -------------- An HTML attachment was scrubbed... URL: <https://lists.lttng.org/pipermail/lttng-dev/attachments/20161021/70022942/attachment.html> ^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2016-10-21 8:19 UTC | newest] Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2016-10-19 9:56 [lttng-dev] Deadlock in call_rcu_thread when destroy rculfhash node with nested rculfhash Evgeniy Ivanov 2016-10-19 10:03 ` Evgeniy Ivanov 2016-10-19 15:03 ` Mathieu Desnoyers 2016-10-21 8:19 ` Evgeniy Ivanov
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox