From patchwork Thu Aug 30 18:18:19 2012 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paul E. McKenney" X-Patchwork-Id: 11049 Return-Path: X-Original-To: patchwork@peony.canonical.com Delivered-To: patchwork@peony.canonical.com Received: from fiordland.canonical.com (fiordland.canonical.com [91.189.94.145]) by peony.canonical.com (Postfix) with ESMTP id 54DEF23EFE for ; Thu, 30 Aug 2012 18:19:44 +0000 (UTC) Received: from mail-iy0-f180.google.com (mail-iy0-f180.google.com [209.85.210.180]) by fiordland.canonical.com (Postfix) with ESMTP id 1D406A1818E for ; Thu, 30 Aug 2012 18:19:08 +0000 (UTC) Received: by mail-iy0-f180.google.com with SMTP id j25so3403526iaf.11 for ; Thu, 30 Aug 2012 11:19:43 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=x-forwarded-to:x-forwarded-for:delivered-to:received-spf:from:to:cc :subject:date:message-id:x-mailer:in-reply-to:references :x-content-scanned:x-cbid:x-gm-message-state; bh=P5bbc73HuVus/j06tJd3mW2bKMhOtsZZHFyAu+K8fd4=; b=cVkG7Phbr0tjrH6FPMhBhx3oauLfacaTXd5G8MiGjkF2+8PwC9QvNFELilIsaur2Mq weO3aMV/2L2MLAT385+Z+Em89I88BalLpMgxuOCRU6xXVLjL94vsjy4EP0CG5n9+yloP hsBQWb5X1/snvocKqbc+M0cjudWS+w5diQGTgr2C7gnSqDfop3baaZSf1oNQd59YU6WX mvwmPSCC9uBTCuxcaUN37rBCTKvVX5cG4eoCZB8JQ6OItUW5XHkiGX0IMsEkE23V6lTt 2T2m8V+8oKJFzUp+8Z6bDuSlNxd8rb3oYw/av+nrF1oksUDiouVNy1SVgvCseGvIjk6M 2bSA== Received: by 10.50.217.227 with SMTP id pb3mr1806234igc.28.1346350783738; Thu, 30 Aug 2012 11:19:43 -0700 (PDT) X-Forwarded-To: linaro-patchwork@canonical.com X-Forwarded-For: patch@linaro.org linaro-patchwork@canonical.com Delivered-To: patches@linaro.org Received: by 10.50.184.232 with SMTP id ex8csp24731igc; Thu, 30 Aug 2012 11:19:43 -0700 (PDT) Received: by 10.182.50.98 with SMTP id b2mr5711585obo.28.1346350783360; Thu, 30 Aug 2012 11:19:43 -0700 (PDT) Received: from e36.co.us.ibm.com (e36.co.us.ibm.com. [32.97.110.154]) by mx.google.com with ESMTPS id k6si3064499obd.211.2012.08.30.11.19.42 (version=TLSv1/SSLv3 cipher=OTHER); Thu, 30 Aug 2012 11:19:43 -0700 (PDT) Received-SPF: pass (google.com: domain of paulmck@linux.vnet.ibm.com designates 32.97.110.154 as permitted sender) client-ip=32.97.110.154; Authentication-Results: mx.google.com; spf=pass (google.com: domain of paulmck@linux.vnet.ibm.com designates 32.97.110.154 as permitted sender) smtp.mail=paulmck@linux.vnet.ibm.com Received: from /spool/local by e36.co.us.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Thu, 30 Aug 2012 12:19:41 -0600 Received: from d03dlp01.boulder.ibm.com (9.17.202.177) by e36.co.us.ibm.com (192.168.1.136) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; Thu, 30 Aug 2012 12:19:15 -0600 Received: from d03relay03.boulder.ibm.com (d03relay03.boulder.ibm.com [9.17.195.228]) by d03dlp01.boulder.ibm.com (Postfix) with ESMTP id 64D4B1FF003B for ; Thu, 30 Aug 2012 12:19:10 -0600 (MDT) Received: from d03av01.boulder.ibm.com (d03av01.boulder.ibm.com [9.17.195.167]) by d03relay03.boulder.ibm.com (8.13.8/8.13.8/NCO v10.0) with ESMTP id q7UIIw7a075146 for ; Thu, 30 Aug 2012 12:18:59 -0600 Received: from d03av01.boulder.ibm.com (loopback [127.0.0.1]) by d03av01.boulder.ibm.com (8.14.4/8.13.1/NCO v10.0 AVout) with ESMTP id q7UIIfK6018681 for ; Thu, 30 Aug 2012 12:18:45 -0600 Received: from paulmck-ThinkPad-W500 (sig-9-77-132-62.mts.ibm.com [9.77.132.62]) by d03av01.boulder.ibm.com (8.14.4/8.13.1/NCO v10.0 AVin) with ESMTP id q7UIIew0018627; Thu, 30 Aug 2012 12:18:40 -0600 Received: by paulmck-ThinkPad-W500 (Postfix, from userid 1000) id D55F2EA829; Thu, 30 Aug 2012 11:18:39 -0700 (PDT) From: "Paul E. McKenney" To: linux-kernel@vger.kernel.org Cc: mingo@elte.hu, laijs@cn.fujitsu.com, dipankar@in.ibm.com, akpm@linux-foundation.org, mathieu.desnoyers@polymtl.ca, josh@joshtriplett.org, niv@us.ibm.com, tglx@linutronix.de, peterz@infradead.org, rostedt@goodmis.org, Valdis.Kletnieks@vt.edu, dhowells@redhat.com, eric.dumazet@gmail.com, darren@dvhart.com, fweisbec@gmail.com, sbw@mit.edu, patches@linaro.org, "Paul E. McKenney" Subject: [PATCH tip/core/rcu 04/23] rcu: Allow RCU grace-period cleanup to be preempted Date: Thu, 30 Aug 2012 11:18:19 -0700 Message-Id: <1346350718-30937-4-git-send-email-paulmck@linux.vnet.ibm.com> X-Mailer: git-send-email 1.7.8 In-Reply-To: <1346350718-30937-1-git-send-email-paulmck@linux.vnet.ibm.com> References: <20120830181811.GA29154@linux.vnet.ibm.com> <1346350718-30937-1-git-send-email-paulmck@linux.vnet.ibm.com> X-Content-Scanned: Fidelis XPS MAILER x-cbid: 12083018-7606-0000-0000-00000333B98E X-Gm-Message-State: ALoCoQmzFsFQbFheQp4pGFqhtS4/TDK13DC1GifjzyKkZMxeDiFGTmmfBmZQNVT/xWvjo1uLcEJZ From: "Paul E. McKenney" RCU grace-period cleanup is currently carried out with interrupts disabled, which can result in excessive latency spikes on large systems (many hundreds or thousands of CPUs). This patch therefore makes the RCU grace-period cleanup be preemptible, including voluntary preemption points, which should eliminate those latency spikes. Similar spikes from forcing of quiescent states will be dealt with similarly by later patches. Reported-by: Mike Galbraith Reported-by: Dimitri Sivanich Signed-off-by: Paul E. McKenney Reviewed-by: Josh Triplett --- kernel/rcutree.c | 11 +++++------ 1 files changed, 5 insertions(+), 6 deletions(-) diff --git a/kernel/rcutree.c b/kernel/rcutree.c index 9fad21c..300aba6 100644 --- a/kernel/rcutree.c +++ b/kernel/rcutree.c @@ -1170,7 +1170,7 @@ static int rcu_gp_kthread(void *arg) * completed. */ if (*rdp->nxttail[RCU_WAIT_TAIL] == NULL) { - raw_spin_unlock(&rnp->lock); /* irqs remain disabled. */ + raw_spin_unlock_irqrestore(&rnp->lock, flags); /* * Propagate new ->completed value to rcu_node @@ -1179,14 +1179,13 @@ static int rcu_gp_kthread(void *arg) * to process their callbacks. */ rcu_for_each_node_breadth_first(rsp, rnp) { - /* irqs already disabled. */ - raw_spin_lock(&rnp->lock); + raw_spin_lock_irqsave(&rnp->lock, flags); rnp->completed = rsp->gpnum; - /* irqs remain disabled. */ - raw_spin_unlock(&rnp->lock); + raw_spin_unlock_irqrestore(&rnp->lock, flags); + cond_resched(); } rnp = rcu_get_root(rsp); - raw_spin_lock(&rnp->lock); /* irqs already disabled. */ + raw_spin_lock_irqsave(&rnp->lock, flags); } rsp->completed = rsp->gpnum; /* Declare grace period done. */