From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from simark.ca by simark.ca with LMTP id MZK+GBvgqWKNWgIAWB0awg (envelope-from ) for ; Wed, 15 Jun 2022 09:35:23 -0400 Received: by simark.ca (Postfix, from userid 112) id 52FB41E224; Wed, 15 Jun 2022 09:35:23 -0400 (EDT) Authentication-Results: simark.ca; dkim=pass (2048-bit key; unprotected) header.d=lists.lttng.org header.i=@lists.lttng.org header.a=rsa-sha256 header.s=default header.b=0UHtWdE8; dkim-atps=neutral X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on simark.ca X-Spam-Level: X-Spam-Status: No, score=-3.0 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6 Received: from lists.lttng.org (lists.lttng.org [167.114.26.123]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by simark.ca (Postfix) with ESMTPS id AE49A1E143 for ; Wed, 15 Jun 2022 09:35:21 -0400 (EDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=lists.lttng.org; s=default; t=1655300121; bh=I2Ytdm8kx88JqN9ek/hDjQX2Vlc6d4LuaRUDSocj+i4=; h=Date:To:Cc:In-Reply-To:References:Subject:List-Id: List-Unsubscribe:List-Archive:List-Post:List-Help:List-Subscribe: From:Reply-To:From; b=0UHtWdE8NzDFBaBvh1A9SpbFB+3vAL03A2iWfzgAsp/WTA/RrLVcDYCi0uDoUBAu/ V5Q2+4GNPfQN1D+IZACGfX30GRaHtMiNX3ExkP3oZU0JikGKJSf+E0+ueLIh6Vu8vj CNWE8WNReGYAoQN9P4jlShdMUgXScqlwGoykTvI6AN4cYo/kjvFsxRZTnsW722U7h+ 9O15pB3d7BnwIihylaFJY2nKSUU2b8pZ6+7RdLdFWXGuhrYgRZujo1h6gegkCUE3NI uh7aVEwBK5AiNct3rhV6L8uNMnT8stwLXsQH0Ok9X3upPBeBnA5vS1dVa+4v9VzXQS OkJ79MprxIvVw== Received: from lists-lttng01.efficios.com (localhost [IPv6:::1]) by lists.lttng.org (Postfix) with ESMTP id 4LNRBd0wS6z8RY; Wed, 15 Jun 2022 09:35:21 -0400 (EDT) Received: from mail.efficios.com (mail.efficios.com [167.114.26.124]) by lists.lttng.org (Postfix) with ESMTPS id 4LNRBb2pSGz8RX for ; Wed, 15 Jun 2022 09:35:19 -0400 (EDT) Received: from localhost (localhost [127.0.0.1]) by mail.efficios.com (Postfix) with ESMTP id C39E140991A for ; Wed, 15 Jun 2022 09:35:18 -0400 (EDT) Received: from mail.efficios.com ([127.0.0.1]) by localhost (mail03.efficios.com [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id 0tpHqtC9JZdz; Wed, 15 Jun 2022 09:35:18 -0400 (EDT) Received: from localhost (localhost [127.0.0.1]) by mail.efficios.com (Postfix) with ESMTP id 0D72A4091E1; Wed, 15 Jun 2022 09:35:18 -0400 (EDT) DKIM-Filter: OpenDKIM Filter v2.10.3 mail.efficios.com 0D72A4091E1 X-Virus-Scanned: amavisd-new at efficios.com Received: from mail.efficios.com ([127.0.0.1]) by localhost (mail03.efficios.com [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id 0L5728rTrUWW; Wed, 15 Jun 2022 09:35:18 -0400 (EDT) Received: from mail03.efficios.com (mail03.efficios.com [167.114.26.124]) by mail.efficios.com (Postfix) with ESMTP id 02B0C40973C; Wed, 15 Jun 2022 09:35:18 -0400 (EDT) Date: Wed, 15 Jun 2022 09:35:17 -0400 (EDT) To: Minlan Wang Cc: lttng-dev Message-ID: <1314631804.61859.1655300117898.JavaMail.zimbra@efficios.com> In-Reply-To: <20220615034900.GA187807@localhost.localdomain>+5075995E84B158A6> References: <20220614035533.GA174967@localhost.localdomain> <765519142.60031.1655221996546.JavaMail.zimbra@efficios.com> <20220615034900.GA187807@localhost.localdomain> MIME-Version: 1.0 X-Originating-IP: [167.114.26.124] X-Mailer: Zimbra 8.8.15_GA_4272 (ZimbraWebClient - FF100 (Linux)/8.8.15_GA_4257) Thread-Topic: urcu workqueue thread uses 99% of cpu while workqueue is empty Thread-Index: vfakGoFUjUR5ACAPT7S/Ymeqm+T9qQ== Subject: Re: [lttng-dev] urcu workqueue thread uses 99% of cpu while workqueue is empty X-BeenThere: lttng-dev@lists.lttng.org X-Mailman-Version: 2.1.39 Precedence: list List-Id: LTTng development list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , From: Mathieu Desnoyers via lttng-dev Reply-To: Mathieu Desnoyers Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: lttng-dev-bounces@lists.lttng.org Sender: "lttng-dev" ----- On Jun 14, 2022, at 11:49 PM, Minlan Wang wangminlan@szsandstone.com wrote: > Hi, Mathieu, > The commit on branch stable-0.12 correponds to the tarball we downloaded is > this: > > commit d5277e807192178ddb79f56ecbbd5ac3c4994f60 (HEAD -> v0.12.1.b, tag: > v0.12.1) > Author: Mathieu Desnoyers > Date: Wed Apr 22 08:51:41 2020 -0400 > > Version 0.12.1 > > Signed-off-by: Mathieu Desnoyers > > The OS we are using is CentOS Linux release 7.9.2009 (Core), not CentOS 8.2 > as mentioned before. And the kernel version is: 3.10.0-1160.el7.x86_64. Can you reproduce with an up-to-date kernel ? AFAIU, kernel version 3.10.0-1160.el7.x86_64 was built on Mon Oct 19 18:34:26 2020 [1], whereas 3.10.0-1160.66.1.el7 was built on Wed May 18 18:19:46 2022 [2]. [1] https://rpmfind.net/linux/RPM/centos/7.9.2009/x86_64/Packages/kernel-3.10.0-1160.el7.x86_64.html [2] https://rpmfind.net/linux/RPM/centos/updates/7.9.2009/x86_64/Packages/kernel-3.10.0-1160.66.1.el7.x86_64.html I just want to make sure we are not hitting a bug in the futex system call. Can you also try to reproduce with an up-to-date Linux kernel ? (5.18.4) Can you provide a reproducer I could try on my own system ? Thanks, Mathieu > > On Tue, Jun 14, 2022 at 11:53:16AM -0400, Mathieu Desnoyers wrote: >> Also, I notice that you appear to be using an internal liburcu API (not public) >> from outside of the liburcu project, which is not really expected. > We are trying to move some linux kernel module function into userspace, and > found that the urcu internal workqueue.h has all the things we need for a > replace for kernel workqueue, so we decided to give it a try. > >> >> If your process forks without exec, make sure you wire up the equivalent of >> rculfhash pthread_atfork functions which call urcu_workqueue_pause_worker(), >> urcu_workqueue_resume_worker() and urcu_workqueue_create_worker(). > There's no fork/exec in the process who is calling alloc_workqueue, and the > threads who are enqueue work into the workqueue is created by calling > pthread_create. > >> >> Also, can you validate of you have many workqueue worker threads trying to >> dequeue from the same workqueue in parallel ? This is unsupported and would >> cause the kind of issues you are observing here. > The workqueue thread is created by calling urcu_workqueue_create in the code > below, and it is the only thread which will dequeue work from the workqueue. > Though, there are multiple threads who will enqueue work by calling > urcu_workqueue_queue_work(wq, work, work->func). > --- > static void workqueue_init_fn(struct urcu_workqueue *workqueue, void *priv) > { > pthread_t tid; > const char *name; > char thread_name[16] = {0}; > > if (!priv) > return; > > name = (const char *)priv; > tid = pthread_self(); > > memcpy(thread_name, name, 15); > if (pthread_setname_np(tid, thread_name)) { > pr_err("failed to set thread name for workqueue %s\n", name); > } > > urcu_memb_register_thread(); > } > > static void workqueue_finalize_fn(struct urcu_workqueue *workqueue, void > *priv) > { > urcu_memb_unregister_thread(); > if (priv) > free(priv); > } > > struct workqueue_struct *alloc_workqueue(const char *fmt, > unsigned int flags, > int max_active, ...) > { > const char *name; > > name = strdup(fmt); > if (!name) { > pr_err("failed to dup name for workqueue %s\n", fmt); > return NULL; > } > > return urcu_workqueue_create(0, -1, (void *)name, > NULL, /* grace */ > workqueue_init_fn, /* init */ > workqueue_finalize_fn, /* finalize */ > NULL, /* before wait */ > NULL, /* after wake up */ > NULL, /* before pasue */ > NULL); /* after resume */ > } > --- > > B.R > Minlan -- Mathieu Desnoyers EfficiOS Inc. http://www.efficios.com _______________________________________________ lttng-dev mailing list lttng-dev@lists.lttng.org https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev