[tip/core/rcu,04/23] rcu: Allow RCU grace-period cleanup to be preempted

Message ID 1346350718-30937-4-git-send-email-paulmck@linux.vnet.ibm.com
State New
Headers show

Commit Message

Paul E. McKenney Aug. 30, 2012, 6:18 p.m.
From: "Paul E. McKenney" <paulmck@linux.vnet.ibm.com>

RCU grace-period cleanup is currently carried out with interrupts
disabled, which can result in excessive latency spikes on large systems
(many hundreds or thousands of CPUs).  This patch therefore makes the
RCU grace-period cleanup be preemptible, including voluntary preemption
points, which should eliminate those latency spikes.  Similar spikes from
forcing of quiescent states will be dealt with similarly by later patches.

Reported-by: Mike Galbraith <mgalbraith@suse.de>
Reported-by: Dimitri Sivanich <sivanich@sgi.com>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
---
 kernel/rcutree.c |   11 +++++------
 1 files changed, 5 insertions(+), 6 deletions(-)

Comments

Josh Triplett Sept. 2, 2012, 1:36 a.m. | #1
On Thu, Aug 30, 2012 at 11:18:19AM -0700, Paul E. McKenney wrote:
> From: "Paul E. McKenney" <paulmck@linux.vnet.ibm.com>
> 
> RCU grace-period cleanup is currently carried out with interrupts
> disabled, which can result in excessive latency spikes on large systems
> (many hundreds or thousands of CPUs).  This patch therefore makes the
> RCU grace-period cleanup be preemptible, including voluntary preemption
> points, which should eliminate those latency spikes.  Similar spikes from
> forcing of quiescent states will be dealt with similarly by later patches.
> 
> Reported-by: Mike Galbraith <mgalbraith@suse.de>
> Reported-by: Dimitri Sivanich <sivanich@sgi.com>
> Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>

Reviewed-by: Josh Triplett <josh@joshtriplett.org>

>  kernel/rcutree.c |   11 +++++------
>  1 files changed, 5 insertions(+), 6 deletions(-)
> 
> diff --git a/kernel/rcutree.c b/kernel/rcutree.c
> index 9fad21c..300aba6 100644
> --- a/kernel/rcutree.c
> +++ b/kernel/rcutree.c
> @@ -1170,7 +1170,7 @@ static int rcu_gp_kthread(void *arg)
>  		 * completed.
>  		 */
>  		if (*rdp->nxttail[RCU_WAIT_TAIL] == NULL) {
> -			raw_spin_unlock(&rnp->lock); /* irqs remain disabled. */
> +			raw_spin_unlock_irqrestore(&rnp->lock, flags);
>  
>  			/*
>  			 * Propagate new ->completed value to rcu_node
> @@ -1179,14 +1179,13 @@ static int rcu_gp_kthread(void *arg)
>  			 * to process their callbacks.
>  			 */
>  			rcu_for_each_node_breadth_first(rsp, rnp) {
> -				/* irqs already disabled. */
> -				raw_spin_lock(&rnp->lock);
> +				raw_spin_lock_irqsave(&rnp->lock, flags);
>  				rnp->completed = rsp->gpnum;
> -				/* irqs remain disabled. */
> -				raw_spin_unlock(&rnp->lock);
> +				raw_spin_unlock_irqrestore(&rnp->lock, flags);
> +				cond_resched();
>  			}
>  			rnp = rcu_get_root(rsp);
> -			raw_spin_lock(&rnp->lock); /* irqs already disabled. */
> +			raw_spin_lock_irqsave(&rnp->lock, flags);
>  		}
>  
>  		rsp->completed = rsp->gpnum; /* Declare grace period done. */
> -- 
> 1.7.8
>

Patch

diff --git a/kernel/rcutree.c b/kernel/rcutree.c
index 9fad21c..300aba6 100644
--- a/kernel/rcutree.c
+++ b/kernel/rcutree.c
@@ -1170,7 +1170,7 @@  static int rcu_gp_kthread(void *arg)
 		 * completed.
 		 */
 		if (*rdp->nxttail[RCU_WAIT_TAIL] == NULL) {
-			raw_spin_unlock(&rnp->lock); /* irqs remain disabled. */
+			raw_spin_unlock_irqrestore(&rnp->lock, flags);
 
 			/*
 			 * Propagate new ->completed value to rcu_node
@@ -1179,14 +1179,13 @@  static int rcu_gp_kthread(void *arg)
 			 * to process their callbacks.
 			 */
 			rcu_for_each_node_breadth_first(rsp, rnp) {
-				/* irqs already disabled. */
-				raw_spin_lock(&rnp->lock);
+				raw_spin_lock_irqsave(&rnp->lock, flags);
 				rnp->completed = rsp->gpnum;
-				/* irqs remain disabled. */
-				raw_spin_unlock(&rnp->lock);
+				raw_spin_unlock_irqrestore(&rnp->lock, flags);
+				cond_resched();
 			}
 			rnp = rcu_get_root(rsp);
-			raw_spin_lock(&rnp->lock); /* irqs already disabled. */
+			raw_spin_lock_irqsave(&rnp->lock, flags);
 		}
 
 		rsp->completed = rsp->gpnum; /* Declare grace period done. */