| Message ID | 1346350718-30937-4-git-send-email-paulmck@linux.vnet.ibm.com |
|---|---|
| State | New |
| Headers | show |
On Thu, Aug 30, 2012 at 11:18:19AM -0700, Paul E. McKenney wrote: > From: "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> > > RCU grace-period cleanup is currently carried out with interrupts > disabled, which can result in excessive latency spikes on large systems > (many hundreds or thousands of CPUs). This patch therefore makes the > RCU grace-period cleanup be preemptible, including voluntary preemption > points, which should eliminate those latency spikes. Similar spikes from > forcing of quiescent states will be dealt with similarly by later patches. > > Reported-by: Mike Galbraith <mgalbraith@suse.de> > Reported-by: Dimitri Sivanich <sivanich@sgi.com> > Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com> Reviewed-by: Josh Triplett <josh@joshtriplett.org> > kernel/rcutree.c | 11 +++++------ > 1 files changed, 5 insertions(+), 6 deletions(-) > > diff --git a/kernel/rcutree.c b/kernel/rcutree.c > index 9fad21c..300aba6 100644 > --- a/kernel/rcutree.c > +++ b/kernel/rcutree.c > @@ -1170,7 +1170,7 @@ static int rcu_gp_kthread(void *arg) > * completed. > */ > if (*rdp->nxttail[RCU_WAIT_TAIL] == NULL) { > - raw_spin_unlock(&rnp->lock); /* irqs remain disabled. */ > + raw_spin_unlock_irqrestore(&rnp->lock, flags); > > /* > * Propagate new ->completed value to rcu_node > @@ -1179,14 +1179,13 @@ static int rcu_gp_kthread(void *arg) > * to process their callbacks. > */ > rcu_for_each_node_breadth_first(rsp, rnp) { > - /* irqs already disabled. */ > - raw_spin_lock(&rnp->lock); > + raw_spin_lock_irqsave(&rnp->lock, flags); > rnp->completed = rsp->gpnum; > - /* irqs remain disabled. */ > - raw_spin_unlock(&rnp->lock); > + raw_spin_unlock_irqrestore(&rnp->lock, flags); > + cond_resched(); > } > rnp = rcu_get_root(rsp); > - raw_spin_lock(&rnp->lock); /* irqs already disabled. */ > + raw_spin_lock_irqsave(&rnp->lock, flags); > } > > rsp->completed = rsp->gpnum; /* Declare grace period done. */ > -- > 1.7.8 >
diff --git a/kernel/rcutree.c b/kernel/rcutree.c index 9fad21c..300aba6 100644 --- a/kernel/rcutree.c +++ b/kernel/rcutree.c @@ -1170,7 +1170,7 @@ static int rcu_gp_kthread(void *arg) * completed. */ if (*rdp->nxttail[RCU_WAIT_TAIL] == NULL) { - raw_spin_unlock(&rnp->lock); /* irqs remain disabled. */ + raw_spin_unlock_irqrestore(&rnp->lock, flags); /* * Propagate new ->completed value to rcu_node @@ -1179,14 +1179,13 @@ static int rcu_gp_kthread(void *arg) * to process their callbacks. */ rcu_for_each_node_breadth_first(rsp, rnp) { - /* irqs already disabled. */ - raw_spin_lock(&rnp->lock); + raw_spin_lock_irqsave(&rnp->lock, flags); rnp->completed = rsp->gpnum; - /* irqs remain disabled. */ - raw_spin_unlock(&rnp->lock); + raw_spin_unlock_irqrestore(&rnp->lock, flags); + cond_resched(); } rnp = rcu_get_root(rsp); - raw_spin_lock(&rnp->lock); /* irqs already disabled. */ + raw_spin_lock_irqsave(&rnp->lock, flags); } rsp->completed = rsp->gpnum; /* Declare grace period done. */