diff mbox series

[12/13] rcu/nocb: Prepare for finegrained deferred wakeup

Message ID 20210223001011.127063-13-frederic@kernel.org
State New
Headers show
Series rcu/nocb updates v2 | expand

Commit Message

Frederic Weisbecker Feb. 23, 2021, 12:10 a.m. UTC
Provide a way to tune the deferred wakeup level we want to perform from
a safe wakeup point. Currently those sites are:

* nocb_timer
* user/idle/guest entry
* CPU down
* softirq/rcuc

All of these sites perform the wake up for both RCU_NOCB_WAKE and
RCU_NOCB_WAKE_FORCE.

In order to merge nocb_timer and nocb_bypass_timer together, we plan to
add a new RCU_NOCB_WAKE_BYPASS that really want to be deferred until
a timer fires so that we don't wake up the NOCB-gp kthread too early.

To prepare for that, integrate a wake up level/limit that a callsite is
deemed to perform.

Signed-off-by: Frederic Weisbecker <frederic@kernel.org>
Cc: Josh Triplett <josh@joshtriplett.org>
Cc: Lai Jiangshan <jiangshanlai@gmail.com>
Cc: Joel Fernandes <joel@joelfernandes.org>
Cc: Neeraj Upadhyay <neeraju@codeaurora.org>
Cc: Boqun Feng <boqun.feng@gmail.com>
---
 kernel/rcu/tree.c        |  2 +-
 kernel/rcu/tree.h        |  2 +-
 kernel/rcu/tree_plugin.h | 15 ++++++++-------
 3 files changed, 10 insertions(+), 9 deletions(-)

Comments

Paul E. McKenney March 16, 2021, 3:02 a.m. UTC | #1
On Tue, Feb 23, 2021 at 01:10:10AM +0100, Frederic Weisbecker wrote:
> Provide a way to tune the deferred wakeup level we want to perform from

> a safe wakeup point. Currently those sites are:

> 

> * nocb_timer

> * user/idle/guest entry

> * CPU down

> * softirq/rcuc

> 

> All of these sites perform the wake up for both RCU_NOCB_WAKE and

> RCU_NOCB_WAKE_FORCE.

> 

> In order to merge nocb_timer and nocb_bypass_timer together, we plan to

> add a new RCU_NOCB_WAKE_BYPASS that really want to be deferred until

> a timer fires so that we don't wake up the NOCB-gp kthread too early.

> 

> To prepare for that, integrate a wake up level/limit that a callsite is

> deemed to perform.


This appears to need the following in order to build for non-NOCB
configurations.  I will fold it in and am retesting.

							Thanx, Paul

------------------------------------------------------------------------

commit 55f59dd75a11455cf558fd387fbf9011017dcc8a
Author: Paul E. McKenney <paulmck@kernel.org>
Date:   Mon Mar 15 20:00:34 2021 -0700

    squash! rcu/nocb: Prepare for fine-grained deferred wakeup
    
    [ paulmck: Fix non-NOCB rcu_nocb_need_deferred_wakeup() definition. ]
    Signed-off-by: Paul E. McKenney <paulmck@kernel.org>


diff --git a/kernel/rcu/tree_plugin.h b/kernel/rcu/tree_plugin.h
index 0cc7f68..dfb048e 100644
--- a/kernel/rcu/tree_plugin.h
+++ b/kernel/rcu/tree_plugin.h
@@ -2926,7 +2926,7 @@ static void __init rcu_boot_init_nocb_percpu_data(struct rcu_data *rdp)
 {
 }
 
-static int rcu_nocb_need_deferred_wakeup(struct rcu_data *rdp)
+static int rcu_nocb_need_deferred_wakeup(struct rcu_data *rdp, int level)
 {
 	return false;
 }
Frederic Weisbecker March 16, 2021, 11:45 a.m. UTC | #2
On Mon, Mar 15, 2021 at 08:02:39PM -0700, Paul E. McKenney wrote:
> On Tue, Feb 23, 2021 at 01:10:10AM +0100, Frederic Weisbecker wrote:

> > Provide a way to tune the deferred wakeup level we want to perform from

> > a safe wakeup point. Currently those sites are:

> > 

> > * nocb_timer

> > * user/idle/guest entry

> > * CPU down

> > * softirq/rcuc

> > 

> > All of these sites perform the wake up for both RCU_NOCB_WAKE and

> > RCU_NOCB_WAKE_FORCE.

> > 

> > In order to merge nocb_timer and nocb_bypass_timer together, we plan to

> > add a new RCU_NOCB_WAKE_BYPASS that really want to be deferred until

> > a timer fires so that we don't wake up the NOCB-gp kthread too early.

> > 

> > To prepare for that, integrate a wake up level/limit that a callsite is

> > deemed to perform.

> 

> This appears to need the following in order to build for non-NOCB

> configurations.  I will fold it in and am retesting.

> 

> 							Thanx, Paul

> 

> ------------------------------------------------------------------------

> 

> commit 55f59dd75a11455cf558fd387fbf9011017dcc8a

> Author: Paul E. McKenney <paulmck@kernel.org>

> Date:   Mon Mar 15 20:00:34 2021 -0700

> 

>     squash! rcu/nocb: Prepare for fine-grained deferred wakeup

>     

>     [ paulmck: Fix non-NOCB rcu_nocb_need_deferred_wakeup() definition. ]

>     Signed-off-by: Paul E. McKenney <paulmck@kernel.org>

> 

> diff --git a/kernel/rcu/tree_plugin.h b/kernel/rcu/tree_plugin.h

> index 0cc7f68..dfb048e 100644

> --- a/kernel/rcu/tree_plugin.h

> +++ b/kernel/rcu/tree_plugin.h

> @@ -2926,7 +2926,7 @@ static void __init rcu_boot_init_nocb_percpu_data(struct rcu_data *rdp)

>  {

>  }

>  

> -static int rcu_nocb_need_deferred_wakeup(struct rcu_data *rdp)

> +static int rcu_nocb_need_deferred_wakeup(struct rcu_data *rdp, int level)

>  {

>  	return false;

>  }



Oops thanks, I missed that.
Paul E. McKenney March 16, 2021, 2:02 p.m. UTC | #3
On Tue, Mar 16, 2021 at 12:45:26PM +0100, Frederic Weisbecker wrote:
> On Mon, Mar 15, 2021 at 08:02:39PM -0700, Paul E. McKenney wrote:

> > On Tue, Feb 23, 2021 at 01:10:10AM +0100, Frederic Weisbecker wrote:

> > > Provide a way to tune the deferred wakeup level we want to perform from

> > > a safe wakeup point. Currently those sites are:

> > > 

> > > * nocb_timer

> > > * user/idle/guest entry

> > > * CPU down

> > > * softirq/rcuc

> > > 

> > > All of these sites perform the wake up for both RCU_NOCB_WAKE and

> > > RCU_NOCB_WAKE_FORCE.

> > > 

> > > In order to merge nocb_timer and nocb_bypass_timer together, we plan to

> > > add a new RCU_NOCB_WAKE_BYPASS that really want to be deferred until

> > > a timer fires so that we don't wake up the NOCB-gp kthread too early.

> > > 

> > > To prepare for that, integrate a wake up level/limit that a callsite is

> > > deemed to perform.

> > 

> > This appears to need the following in order to build for non-NOCB

> > configurations.  I will fold it in and am retesting.

> > 

> > 							Thanx, Paul

> > 

> > ------------------------------------------------------------------------

> > 

> > commit 55f59dd75a11455cf558fd387fbf9011017dcc8a

> > Author: Paul E. McKenney <paulmck@kernel.org>

> > Date:   Mon Mar 15 20:00:34 2021 -0700

> > 

> >     squash! rcu/nocb: Prepare for fine-grained deferred wakeup

> >     

> >     [ paulmck: Fix non-NOCB rcu_nocb_need_deferred_wakeup() definition. ]

> >     Signed-off-by: Paul E. McKenney <paulmck@kernel.org>

> > 

> > diff --git a/kernel/rcu/tree_plugin.h b/kernel/rcu/tree_plugin.h

> > index 0cc7f68..dfb048e 100644

> > --- a/kernel/rcu/tree_plugin.h

> > +++ b/kernel/rcu/tree_plugin.h

> > @@ -2926,7 +2926,7 @@ static void __init rcu_boot_init_nocb_percpu_data(struct rcu_data *rdp)

> >  {

> >  }

> >  

> > -static int rcu_nocb_need_deferred_wakeup(struct rcu_data *rdp)

> > +static int rcu_nocb_need_deferred_wakeup(struct rcu_data *rdp, int level)

> >  {

> >  	return false;

> >  }

> 

> Oops thanks, I missed that.


And with this, light rcutorture testing passed.  I hope to pound on it
harder later this week.

							Thanx, Paul
diff mbox series

Patch

diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
index 2c9cf4df942c..9951a4bef504 100644
--- a/kernel/rcu/tree.c
+++ b/kernel/rcu/tree.c
@@ -3823,7 +3823,7 @@  static int rcu_pending(int user)
 	check_cpu_stall(rdp);
 
 	/* Does this CPU need a deferred NOCB wakeup? */
-	if (rcu_nocb_need_deferred_wakeup(rdp))
+	if (rcu_nocb_need_deferred_wakeup(rdp, RCU_NOCB_WAKE))
 		return 1;
 
 	/* Is this a nohz_full CPU in userspace or idle?  (Ignore RCU if so.) */
diff --git a/kernel/rcu/tree.h b/kernel/rcu/tree.h
index b280a843bd2c..2510e86265c1 100644
--- a/kernel/rcu/tree.h
+++ b/kernel/rcu/tree.h
@@ -433,7 +433,7 @@  static bool rcu_nocb_try_bypass(struct rcu_data *rdp, struct rcu_head *rhp,
 				bool *was_alldone, unsigned long flags);
 static void __call_rcu_nocb_wake(struct rcu_data *rdp, bool was_empty,
 				 unsigned long flags);
-static int rcu_nocb_need_deferred_wakeup(struct rcu_data *rdp);
+static int rcu_nocb_need_deferred_wakeup(struct rcu_data *rdp, int level);
 static bool do_nocb_deferred_wakeup(struct rcu_data *rdp);
 static void rcu_boot_init_nocb_percpu_data(struct rcu_data *rdp);
 static void rcu_spawn_cpu_nocb_kthread(int cpu);
diff --git a/kernel/rcu/tree_plugin.h b/kernel/rcu/tree_plugin.h
index d8b50ff40e4b..e0420e3b30e6 100644
--- a/kernel/rcu/tree_plugin.h
+++ b/kernel/rcu/tree_plugin.h
@@ -2364,13 +2364,14 @@  static int rcu_nocb_cb_kthread(void *arg)
 }
 
 /* Is a deferred wakeup of rcu_nocb_kthread() required? */
-static int rcu_nocb_need_deferred_wakeup(struct rcu_data *rdp)
+static int rcu_nocb_need_deferred_wakeup(struct rcu_data *rdp, int level)
 {
-	return READ_ONCE(rdp->nocb_defer_wakeup) > RCU_NOCB_WAKE_NOT;
+	return READ_ONCE(rdp->nocb_defer_wakeup) >= level;
 }
 
 /* Do a deferred wakeup of rcu_nocb_kthread(). */
-static bool do_nocb_deferred_wakeup_common(struct rcu_data *rdp)
+static bool do_nocb_deferred_wakeup_common(struct rcu_data *rdp,
+					   int level)
 {
 	unsigned long flags;
 	int ndw;
@@ -2379,7 +2380,7 @@  static bool do_nocb_deferred_wakeup_common(struct rcu_data *rdp)
 
 	raw_spin_lock_irqsave(&rdp_gp->nocb_gp_lock, flags);
 
-	if (!rcu_nocb_need_deferred_wakeup(rdp_gp)) {
+	if (!rcu_nocb_need_deferred_wakeup(rdp_gp, level)) {
 		raw_spin_unlock_irqrestore(&rdp_gp->nocb_gp_lock, flags);
 		return false;
 	}
@@ -2396,7 +2397,7 @@  static void do_nocb_deferred_wakeup_timer(struct timer_list *t)
 {
 	struct rcu_data *rdp = from_timer(rdp, t, nocb_timer);
 
-	do_nocb_deferred_wakeup_common(rdp);
+	do_nocb_deferred_wakeup_common(rdp, RCU_NOCB_WAKE);
 }
 
 /*
@@ -2409,8 +2410,8 @@  static bool do_nocb_deferred_wakeup(struct rcu_data *rdp)
 	if (!rdp->nocb_gp_rdp)
 		return false;
 
-	if (rcu_nocb_need_deferred_wakeup(rdp->nocb_gp_rdp))
-		return do_nocb_deferred_wakeup_common(rdp);
+	if (rcu_nocb_need_deferred_wakeup(rdp->nocb_gp_rdp, RCU_NOCB_WAKE))
+		return do_nocb_deferred_wakeup_common(rdp, RCU_NOCB_WAKE);
 	return false;
 }