From patchwork Fri Jun 15 21:06:04 2012 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paul E. McKenney" X-Patchwork-Id: 9351 Return-Path: X-Original-To: patchwork@peony.canonical.com Delivered-To: patchwork@peony.canonical.com Received: from fiordland.canonical.com (fiordland.canonical.com [91.189.94.145]) by peony.canonical.com (Postfix) with ESMTP id DDCAC23E0C for ; Fri, 15 Jun 2012 21:06:34 +0000 (UTC) Received: from mail-gg0-f180.google.com (mail-gg0-f180.google.com [209.85.161.180]) by fiordland.canonical.com (Postfix) with ESMTP id ADCFEA1821D for ; Fri, 15 Jun 2012 21:06:34 +0000 (UTC) Received: by mail-gg0-f180.google.com with SMTP id f1so2962892ggn.11 for ; Fri, 15 Jun 2012 14:06:34 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=x-forwarded-to:x-forwarded-for:delivered-to:received-spf:from:to:cc :subject:date:message-id:x-mailer:in-reply-to:references :x-content-scanned:x-cbid:x-gm-message-state; bh=a3u7ms1iB13xtXTLNJeT88b91uRRc8KfTZKPMC7AvSc=; b=asj0daY82nF/036Ou7zZr8i0lqCxQnXyCVJCilPvk0PAZsNsrFgxZw35ObRI2H01Yw atPhGHnXFstIbRHZ7kD9dHPPA8riVFoWpap/v06kn6v6bBaLkbFKxnGUCdzveSMRcZsB dXVwUNmtJsvXmZnfjGI+XdfzSfexDsfZkJI2oxvitGHBu0RAWLNeOdERRGtHJ/ZA/mdt 3gPi2zyReFfdHstbCQk0IUX4XuGFus9mww8YzZnvTnbKeIxwZ7oSY5JdM00RNneuQe2q PIoTIvdC75WZyiIrcZ6fmU8qsKYwe6oosfkLhC7NnoaBg2Yqwr9bJSoRFidOiEUzIGxW yy4Q== Received: by 10.50.193.196 with SMTP id hq4mr3246950igc.57.1339794394339; Fri, 15 Jun 2012 14:06:34 -0700 (PDT) X-Forwarded-To: linaro-patchwork@canonical.com X-Forwarded-For: patch@linaro.org linaro-patchwork@canonical.com Delivered-To: patches@linaro.org Received: by 10.231.24.148 with SMTP id v20csp169696ibb; Fri, 15 Jun 2012 14:06:33 -0700 (PDT) Received: by 10.68.217.40 with SMTP id ov8mr24001096pbc.131.1339794393623; Fri, 15 Jun 2012 14:06:33 -0700 (PDT) Received: from e36.co.us.ibm.com (e36.co.us.ibm.com. [32.97.110.154]) by mx.google.com with ESMTPS id ss6si17913043pbc.183.2012.06.15.14.06.33 (version=TLSv1/SSLv3 cipher=OTHER); Fri, 15 Jun 2012 14:06:33 -0700 (PDT) Received-SPF: pass (google.com: domain of paulmck@linux.vnet.ibm.com designates 32.97.110.154 as permitted sender) client-ip=32.97.110.154; Authentication-Results: mx.google.com; spf=pass (google.com: domain of paulmck@linux.vnet.ibm.com designates 32.97.110.154 as permitted sender) smtp.mail=paulmck@linux.vnet.ibm.com Received: from /spool/local by e36.co.us.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Fri, 15 Jun 2012 15:06:32 -0600 Received: from d03dlp01.boulder.ibm.com (9.17.202.177) by e36.co.us.ibm.com (192.168.1.136) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; Fri, 15 Jun 2012 15:06:30 -0600 Received: from d03relay01.boulder.ibm.com (d03relay01.boulder.ibm.com [9.17.195.226]) by d03dlp01.boulder.ibm.com (Postfix) with ESMTP id D1C951FF0025; Fri, 15 Jun 2012 21:06:28 +0000 (WET) Received: from d03av01.boulder.ibm.com (d03av01.boulder.ibm.com [9.17.195.167]) by d03relay01.boulder.ibm.com (8.13.8/8.13.8/NCO v10.0) with ESMTP id q5FL6H18103498; Fri, 15 Jun 2012 15:06:21 -0600 Received: from d03av01.boulder.ibm.com (loopback [127.0.0.1]) by d03av01.boulder.ibm.com (8.14.4/8.13.1/NCO v10.0 AVout) with ESMTP id q5FL6DsN002312; Fri, 15 Jun 2012 15:06:16 -0600 Received: from paulmck-ThinkPad-W500 ([9.47.24.152]) by d03av01.boulder.ibm.com (8.14.4/8.13.1/NCO v10.0 AVin) with ESMTP id q5FL6CYJ001984; Fri, 15 Jun 2012 15:06:13 -0600 Received: by paulmck-ThinkPad-W500 (Postfix, from userid 1000) id 8F571E527B; Fri, 15 Jun 2012 14:06:12 -0700 (PDT) From: "Paul E. McKenney" To: linux-kernel@vger.kernel.org Cc: mingo@elte.hu, laijs@cn.fujitsu.com, dipankar@in.ibm.com, akpm@linux-foundation.org, mathieu.desnoyers@polymtl.ca, josh@joshtriplett.org, niv@us.ibm.com, tglx@linutronix.de, peterz@infradead.org, rostedt@goodmis.org, Valdis.Kletnieks@vt.edu, dhowells@redhat.com, eric.dumazet@gmail.com, darren@dvhart.com, fweisbec@gmail.com, patches@linaro.org, "Paul E. McKenney" , "Paul E. McKenney" Subject: [PATCH tip/core/rcu 09/15] rcu: Increasing rcu_barrier() concurrency Date: Fri, 15 Jun 2012 14:06:04 -0700 Message-Id: <1339794370-28119-9-git-send-email-paulmck@linux.vnet.ibm.com> X-Mailer: git-send-email 1.7.8 In-Reply-To: <1339794370-28119-1-git-send-email-paulmck@linux.vnet.ibm.com> References: <20120615210550.GA27506@linux.vnet.ibm.com> <1339794370-28119-1-git-send-email-paulmck@linux.vnet.ibm.com> X-Content-Scanned: Fidelis XPS MAILER x-cbid: 12061521-7606-0000-0000-000001295815 X-Gm-Message-State: ALoCoQldTqgvrHtCrZgXYBXQ48g3N352m01csS69p7wPe6537+jiTYKrjB/e0PuwQ5DLGaiY2Skr From: "Paul E. McKenney" The traditional rcu_barrier() implementation has serialized all requests, regardless of RCU flavor, and also does not coalesce concurrent requests. In the past, this has been good and sufficient. However, systems are getting larger and use of rcu_barrier() has been increasing. This commit therefore introduces a counter-based scheme that allows _rcu_barrier() calls for the same flavor of RCU to take advantage of each others' work. Signed-off-by: Paul E. McKenney Signed-off-by: Paul E. McKenney --- kernel/rcutree.c | 27 ++++++++++++++++++++++++++- kernel/rcutree.h | 2 ++ 2 files changed, 28 insertions(+), 1 deletions(-) diff --git a/kernel/rcutree.c b/kernel/rcutree.c index 93358d4..7c299d3 100644 --- a/kernel/rcutree.c +++ b/kernel/rcutree.c @@ -2291,13 +2291,32 @@ static void _rcu_barrier(struct rcu_state *rsp) unsigned long flags; struct rcu_data *rdp; struct rcu_data rd; + unsigned long snap = ACCESS_ONCE(rsp->n_barrier_done); + unsigned long snap_done; init_rcu_head_on_stack(&rd.barrier_head); /* Take mutex to serialize concurrent rcu_barrier() requests. */ mutex_lock(&rsp->barrier_mutex); - smp_mb(); /* Prevent any prior operations from leaking in. */ + /* + * Ensure tht all prior references, including to ->n_barrier_done, + * are ordered before the _rcu_barrier() machinery. + */ + smp_mb(); /* See above block comment. */ + + /* Recheck ->n_barrier_done to see if others did our work for us. */ + snap_done = ACCESS_ONCE(rsp->n_barrier_done); + if (ULONG_CMP_GE(snap_done, ((snap + 1) & ~0x1) + 2)) { + smp_mb(); + mutex_unlock(&rsp->barrier_mutex); + return; + } + + /* Increment ->n_barrier_done to avoid duplicate work. */ + ACCESS_ONCE(rsp->n_barrier_done)++; + WARN_ON_ONCE((rsp->n_barrier_done & 0x1) != 1); + smp_mb(); /* Order ->n_barrier_done increment with below mechanism. */ /* * Initialize the count to one rather than to zero in order to @@ -2368,6 +2387,12 @@ static void _rcu_barrier(struct rcu_state *rsp) if (atomic_dec_and_test(&rsp->barrier_cpu_count)) complete(&rsp->barrier_completion); + /* Increment ->n_barrier_done to prevent duplicate work. */ + smp_mb(); /* Keep increment after above mechanism. */ + ACCESS_ONCE(rsp->n_barrier_done)++; + WARN_ON_ONCE((rsp->n_barrier_done & 0x1) != 0); + smp_mb(); /* Keep increment before caller's subsequent code. */ + /* Wait for all rcu_barrier_callback() callbacks to be invoked. */ wait_for_completion(&rsp->barrier_completion); diff --git a/kernel/rcutree.h b/kernel/rcutree.h index d9ac82f..a294f7f 100644 --- a/kernel/rcutree.h +++ b/kernel/rcutree.h @@ -389,6 +389,8 @@ struct rcu_state { struct mutex barrier_mutex; /* Guards barrier fields. */ atomic_t barrier_cpu_count; /* # CPUs waiting on. */ struct completion barrier_completion; /* Wake at barrier end. */ + unsigned long n_barrier_done; /* ++ at start and end of */ + /* _rcu_barrier(). */ raw_spinlock_t fqslock; /* Only one task forcing */ /* quiescent states. */ unsigned long jiffies_force_qs; /* Time at which to invoke */