From patchwork Fri Jun 22 15:17:20 2012 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paul E. McKenney" X-Patchwork-Id: 9555 Return-Path: X-Original-To: patchwork@peony.canonical.com Delivered-To: patchwork@peony.canonical.com Received: from fiordland.canonical.com (fiordland.canonical.com [91.189.94.145]) by peony.canonical.com (Postfix) with ESMTP id B9E7223E55 for ; Fri, 22 Jun 2012 15:26:15 +0000 (UTC) Received: from mail-yx0-f180.google.com (mail-yx0-f180.google.com [209.85.213.180]) by fiordland.canonical.com (Postfix) with ESMTP id 89D45A18891 for ; Fri, 22 Jun 2012 15:26:15 +0000 (UTC) Received: by mail-yx0-f180.google.com with SMTP id q6so1742751yen.11 for ; Fri, 22 Jun 2012 08:26:15 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=x-forwarded-to:x-forwarded-for:delivered-to:received-spf:from:to:cc :subject:date:message-id:x-mailer:in-reply-to:references :x-content-scanned:x-cbid:x-gm-message-state; bh=L5VRwt6qxCfqzHPyH0KgINAcSK4o/lWrgrUCDFbdrCo=; b=S5BzmXd0t6sZ+nlSaLd/RIFOg9ogICY3OS5xxqckrhp0R7n2Qd47eIQ0YI4D8rjcM1 RbWFLhrr7K4isEcUkCIHCUeoY1vdgD25UvPEEjV28p7zByuPhOBtCTEb91BP+iRhgulS aqkJcga3xVH+41x8Z8FhWSwsMNKWLLPkif19mFFPzjEJYWgHoN7UOuU5Z6jK7EwhKJPD ZvyW8JOK1r9z8sDNx9JIsdU7VwH4MsT00jub7TQmhb6RpjxBq93rmj/vPiUrd6iSo5L/ aRw7JEnwD98qK6Jy2QOW0Rpnd0EV/K2tHTYtPv8TlXYUSnVqOAF/jQafOjDsMxXni5ta iQDg== Received: by 10.50.40.193 with SMTP id z1mr2073272igk.0.1340378775032; Fri, 22 Jun 2012 08:26:15 -0700 (PDT) X-Forwarded-To: linaro-patchwork@canonical.com X-Forwarded-For: patch@linaro.org linaro-patchwork@canonical.com Delivered-To: patches@linaro.org Received: by 10.231.24.148 with SMTP id v20csp71861ibb; Fri, 22 Jun 2012 08:26:14 -0700 (PDT) Received: by 10.68.225.42 with SMTP id rh10mr11103446pbc.116.1340378774305; Fri, 22 Jun 2012 08:26:14 -0700 (PDT) Received: from e36.co.us.ibm.com (e36.co.us.ibm.com. [32.97.110.154]) by mx.google.com with ESMTPS id qf1si1568973pbc.295.2012.06.22.08.26.13 (version=TLSv1/SSLv3 cipher=OTHER); Fri, 22 Jun 2012 08:26:14 -0700 (PDT) Received-SPF: pass (google.com: domain of paulmck@linux.vnet.ibm.com designates 32.97.110.154 as permitted sender) client-ip=32.97.110.154; Authentication-Results: mx.google.com; spf=pass (google.com: domain of paulmck@linux.vnet.ibm.com designates 32.97.110.154 as permitted sender) smtp.mail=paulmck@linux.vnet.ibm.com Received: from /spool/local by e36.co.us.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Fri, 22 Jun 2012 09:26:12 -0600 Received: from d01dlp03.pok.ibm.com (9.56.224.17) by e36.co.us.ibm.com (192.168.1.136) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; Fri, 22 Jun 2012 09:25:40 -0600 Received: from d01relay05.pok.ibm.com (d01relay05.pok.ibm.com [9.56.227.237]) by d01dlp03.pok.ibm.com (Postfix) with ESMTP id 04B93C900A4 for ; Fri, 22 Jun 2012 11:25:07 -0400 (EDT) Received: from d03av01.boulder.ibm.com (d03av01.boulder.ibm.com [9.17.195.167]) by d01relay05.pok.ibm.com (8.13.8/8.13.8/NCO v10.0) with ESMTP id q5MFP8lW166134 for ; Fri, 22 Jun 2012 11:25:08 -0400 Received: from d03av01.boulder.ibm.com (loopback [127.0.0.1]) by d03av01.boulder.ibm.com (8.14.4/8.13.1/NCO v10.0 AVout) with ESMTP id q5MFObPp019713 for ; Fri, 22 Jun 2012 09:24:47 -0600 Received: from paulmck-ThinkPad-W500 ([9.47.24.152]) by d03av01.boulder.ibm.com (8.14.4/8.13.1/NCO v10.0 AVin) with ESMTP id q5MFOYob019281; Fri, 22 Jun 2012 09:24:34 -0600 Received: by paulmck-ThinkPad-W500 (Postfix, from userid 1000) id 50969E71BC; Fri, 22 Jun 2012 08:17:24 -0700 (PDT) From: "Paul E. McKenney" To: linux-kernel@vger.kernel.org Cc: mingo@elte.hu, laijs@cn.fujitsu.com, dipankar@in.ibm.com, akpm@linux-foundation.org, mathieu.desnoyers@polymtl.ca, josh@joshtriplett.org, niv@us.ibm.com, tglx@linutronix.de, peterz@infradead.org, rostedt@goodmis.org, Valdis.Kletnieks@vt.edu, dhowells@redhat.com, eric.dumazet@gmail.com, darren@dvhart.com, fweisbec@gmail.com, sbw@mit.edu, patches@linaro.org, "Paul E. McKenney" Subject: [PATCH tip/core/rcu 21/22] rcu: Allow RCU grace-period cleanup to be preempted Date: Fri, 22 Jun 2012 08:17:20 -0700 Message-Id: <1340378241-6458-21-git-send-email-paulmck@linux.vnet.ibm.com> X-Mailer: git-send-email 1.7.8 In-Reply-To: <1340378241-6458-1-git-send-email-paulmck@linux.vnet.ibm.com> References: <20120622151655.GA6249@linux.vnet.ibm.com> <1340378241-6458-1-git-send-email-paulmck@linux.vnet.ibm.com> X-Content-Scanned: Fidelis XPS MAILER x-cbid: 12062215-7606-0000-0000-00000160F16B X-Gm-Message-State: ALoCoQmPME3IHZrc1B8+Yq+RixpsDzaMTmEEdPXLl7L/00Tns+rzyGycIlb9JWwS1jajWD8ySFSu From: "Paul E. McKenney" RCU grace-period cleanup is currently carried out with interrupts disabled, which can result in excessive latency spikes on large systems (many hundreds or thousands of CPUs). This patch therefore makes the RCU grace-period cleanup be preemptible, including voluntary preemption points, which should eliminate those latency spikes. Similar spikes from forcing of quiescent states will be dealt with similarly by later patches. Reported-by: Mike Galbraith Reported-by: Dimitri Sivanich Signed-off-by: Paul E. McKenney --- kernel/rcutree.c | 11 +++++------ 1 files changed, 5 insertions(+), 6 deletions(-) diff --git a/kernel/rcutree.c b/kernel/rcutree.c index 4a3f6b0..d21c2d3 100644 --- a/kernel/rcutree.c +++ b/kernel/rcutree.c @@ -1160,7 +1160,7 @@ static int rcu_gp_kthread(void *arg) * completed. */ if (*rdp->nxttail[RCU_WAIT_TAIL] == NULL) { - raw_spin_unlock(&rnp->lock); /* irqs remain disabled. */ + raw_spin_unlock_irqrestore(&rnp->lock, flags); /* * Propagate new ->completed value to rcu_node @@ -1169,14 +1169,13 @@ static int rcu_gp_kthread(void *arg) * to process their callbacks. */ rcu_for_each_node_breadth_first(rsp, rnp) { - /* irqs already disabled. */ - raw_spin_lock(&rnp->lock); + raw_spin_lock_irqsave(&rnp->lock, flags); rnp->completed = rsp->gpnum; - /* irqs remain disabled. */ - raw_spin_unlock(&rnp->lock); + raw_spin_unlock_irqrestore(&rnp->lock, flags); + cond_resched(); } rnp = rcu_get_root(rsp); - raw_spin_lock(&rnp->lock); /* irqs already disabled. */ + raw_spin_lock_irqsave(&rnp->lock, flags); } rsp->completed = rsp->gpnum; /* Declare grace period done. */