| Message ID | 20210223001011.127063-13-frederic@kernel.org |
|---|---|
| State | New |
| Headers | show |
| Series | rcu/nocb updates v2 | expand |
On Tue, Feb 23, 2021 at 01:10:10AM +0100, Frederic Weisbecker wrote: > Provide a way to tune the deferred wakeup level we want to perform from > a safe wakeup point. Currently those sites are: > > * nocb_timer > * user/idle/guest entry > * CPU down > * softirq/rcuc > > All of these sites perform the wake up for both RCU_NOCB_WAKE and > RCU_NOCB_WAKE_FORCE. > > In order to merge nocb_timer and nocb_bypass_timer together, we plan to > add a new RCU_NOCB_WAKE_BYPASS that really want to be deferred until > a timer fires so that we don't wake up the NOCB-gp kthread too early. > > To prepare for that, integrate a wake up level/limit that a callsite is > deemed to perform. This appears to need the following in order to build for non-NOCB configurations. I will fold it in and am retesting. Thanx, Paul ------------------------------------------------------------------------ commit 55f59dd75a11455cf558fd387fbf9011017dcc8a Author: Paul E. McKenney <paulmck@kernel.org> Date: Mon Mar 15 20:00:34 2021 -0700 squash! rcu/nocb: Prepare for fine-grained deferred wakeup [ paulmck: Fix non-NOCB rcu_nocb_need_deferred_wakeup() definition. ] Signed-off-by: Paul E. McKenney <paulmck@kernel.org> diff --git a/kernel/rcu/tree_plugin.h b/kernel/rcu/tree_plugin.h index 0cc7f68..dfb048e 100644 --- a/kernel/rcu/tree_plugin.h +++ b/kernel/rcu/tree_plugin.h @@ -2926,7 +2926,7 @@ static void __init rcu_boot_init_nocb_percpu_data(struct rcu_data *rdp) { } -static int rcu_nocb_need_deferred_wakeup(struct rcu_data *rdp) +static int rcu_nocb_need_deferred_wakeup(struct rcu_data *rdp, int level) { return false; }
On Mon, Mar 15, 2021 at 08:02:39PM -0700, Paul E. McKenney wrote: > On Tue, Feb 23, 2021 at 01:10:10AM +0100, Frederic Weisbecker wrote: > > Provide a way to tune the deferred wakeup level we want to perform from > > a safe wakeup point. Currently those sites are: > > > > * nocb_timer > > * user/idle/guest entry > > * CPU down > > * softirq/rcuc > > > > All of these sites perform the wake up for both RCU_NOCB_WAKE and > > RCU_NOCB_WAKE_FORCE. > > > > In order to merge nocb_timer and nocb_bypass_timer together, we plan to > > add a new RCU_NOCB_WAKE_BYPASS that really want to be deferred until > > a timer fires so that we don't wake up the NOCB-gp kthread too early. > > > > To prepare for that, integrate a wake up level/limit that a callsite is > > deemed to perform. > > This appears to need the following in order to build for non-NOCB > configurations. I will fold it in and am retesting. > > Thanx, Paul > > ------------------------------------------------------------------------ > > commit 55f59dd75a11455cf558fd387fbf9011017dcc8a > Author: Paul E. McKenney <paulmck@kernel.org> > Date: Mon Mar 15 20:00:34 2021 -0700 > > squash! rcu/nocb: Prepare for fine-grained deferred wakeup > > [ paulmck: Fix non-NOCB rcu_nocb_need_deferred_wakeup() definition. ] > Signed-off-by: Paul E. McKenney <paulmck@kernel.org> > > diff --git a/kernel/rcu/tree_plugin.h b/kernel/rcu/tree_plugin.h > index 0cc7f68..dfb048e 100644 > --- a/kernel/rcu/tree_plugin.h > +++ b/kernel/rcu/tree_plugin.h > @@ -2926,7 +2926,7 @@ static void __init rcu_boot_init_nocb_percpu_data(struct rcu_data *rdp) > { > } > > -static int rcu_nocb_need_deferred_wakeup(struct rcu_data *rdp) > +static int rcu_nocb_need_deferred_wakeup(struct rcu_data *rdp, int level) > { > return false; > } Oops thanks, I missed that.
On Tue, Mar 16, 2021 at 12:45:26PM +0100, Frederic Weisbecker wrote: > On Mon, Mar 15, 2021 at 08:02:39PM -0700, Paul E. McKenney wrote: > > On Tue, Feb 23, 2021 at 01:10:10AM +0100, Frederic Weisbecker wrote: > > > Provide a way to tune the deferred wakeup level we want to perform from > > > a safe wakeup point. Currently those sites are: > > > > > > * nocb_timer > > > * user/idle/guest entry > > > * CPU down > > > * softirq/rcuc > > > > > > All of these sites perform the wake up for both RCU_NOCB_WAKE and > > > RCU_NOCB_WAKE_FORCE. > > > > > > In order to merge nocb_timer and nocb_bypass_timer together, we plan to > > > add a new RCU_NOCB_WAKE_BYPASS that really want to be deferred until > > > a timer fires so that we don't wake up the NOCB-gp kthread too early. > > > > > > To prepare for that, integrate a wake up level/limit that a callsite is > > > deemed to perform. > > > > This appears to need the following in order to build for non-NOCB > > configurations. I will fold it in and am retesting. > > > > Thanx, Paul > > > > ------------------------------------------------------------------------ > > > > commit 55f59dd75a11455cf558fd387fbf9011017dcc8a > > Author: Paul E. McKenney <paulmck@kernel.org> > > Date: Mon Mar 15 20:00:34 2021 -0700 > > > > squash! rcu/nocb: Prepare for fine-grained deferred wakeup > > > > [ paulmck: Fix non-NOCB rcu_nocb_need_deferred_wakeup() definition. ] > > Signed-off-by: Paul E. McKenney <paulmck@kernel.org> > > > > diff --git a/kernel/rcu/tree_plugin.h b/kernel/rcu/tree_plugin.h > > index 0cc7f68..dfb048e 100644 > > --- a/kernel/rcu/tree_plugin.h > > +++ b/kernel/rcu/tree_plugin.h > > @@ -2926,7 +2926,7 @@ static void __init rcu_boot_init_nocb_percpu_data(struct rcu_data *rdp) > > { > > } > > > > -static int rcu_nocb_need_deferred_wakeup(struct rcu_data *rdp) > > +static int rcu_nocb_need_deferred_wakeup(struct rcu_data *rdp, int level) > > { > > return false; > > } > > Oops thanks, I missed that. And with this, light rcutorture testing passed. I hope to pound on it harder later this week. Thanx, Paul
diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c index 2c9cf4df942c..9951a4bef504 100644 --- a/kernel/rcu/tree.c +++ b/kernel/rcu/tree.c @@ -3823,7 +3823,7 @@ static int rcu_pending(int user) check_cpu_stall(rdp); /* Does this CPU need a deferred NOCB wakeup? */ - if (rcu_nocb_need_deferred_wakeup(rdp)) + if (rcu_nocb_need_deferred_wakeup(rdp, RCU_NOCB_WAKE)) return 1; /* Is this a nohz_full CPU in userspace or idle? (Ignore RCU if so.) */ diff --git a/kernel/rcu/tree.h b/kernel/rcu/tree.h index b280a843bd2c..2510e86265c1 100644 --- a/kernel/rcu/tree.h +++ b/kernel/rcu/tree.h @@ -433,7 +433,7 @@ static bool rcu_nocb_try_bypass(struct rcu_data *rdp, struct rcu_head *rhp, bool *was_alldone, unsigned long flags); static void __call_rcu_nocb_wake(struct rcu_data *rdp, bool was_empty, unsigned long flags); -static int rcu_nocb_need_deferred_wakeup(struct rcu_data *rdp); +static int rcu_nocb_need_deferred_wakeup(struct rcu_data *rdp, int level); static bool do_nocb_deferred_wakeup(struct rcu_data *rdp); static void rcu_boot_init_nocb_percpu_data(struct rcu_data *rdp); static void rcu_spawn_cpu_nocb_kthread(int cpu); diff --git a/kernel/rcu/tree_plugin.h b/kernel/rcu/tree_plugin.h index d8b50ff40e4b..e0420e3b30e6 100644 --- a/kernel/rcu/tree_plugin.h +++ b/kernel/rcu/tree_plugin.h @@ -2364,13 +2364,14 @@ static int rcu_nocb_cb_kthread(void *arg) } /* Is a deferred wakeup of rcu_nocb_kthread() required? */ -static int rcu_nocb_need_deferred_wakeup(struct rcu_data *rdp) +static int rcu_nocb_need_deferred_wakeup(struct rcu_data *rdp, int level) { - return READ_ONCE(rdp->nocb_defer_wakeup) > RCU_NOCB_WAKE_NOT; + return READ_ONCE(rdp->nocb_defer_wakeup) >= level; } /* Do a deferred wakeup of rcu_nocb_kthread(). */ -static bool do_nocb_deferred_wakeup_common(struct rcu_data *rdp) +static bool do_nocb_deferred_wakeup_common(struct rcu_data *rdp, + int level) { unsigned long flags; int ndw; @@ -2379,7 +2380,7 @@ static bool do_nocb_deferred_wakeup_common(struct rcu_data *rdp) raw_spin_lock_irqsave(&rdp_gp->nocb_gp_lock, flags); - if (!rcu_nocb_need_deferred_wakeup(rdp_gp)) { + if (!rcu_nocb_need_deferred_wakeup(rdp_gp, level)) { raw_spin_unlock_irqrestore(&rdp_gp->nocb_gp_lock, flags); return false; } @@ -2396,7 +2397,7 @@ static void do_nocb_deferred_wakeup_timer(struct timer_list *t) { struct rcu_data *rdp = from_timer(rdp, t, nocb_timer); - do_nocb_deferred_wakeup_common(rdp); + do_nocb_deferred_wakeup_common(rdp, RCU_NOCB_WAKE); } /* @@ -2409,8 +2410,8 @@ static bool do_nocb_deferred_wakeup(struct rcu_data *rdp) if (!rdp->nocb_gp_rdp) return false; - if (rcu_nocb_need_deferred_wakeup(rdp->nocb_gp_rdp)) - return do_nocb_deferred_wakeup_common(rdp); + if (rcu_nocb_need_deferred_wakeup(rdp->nocb_gp_rdp, RCU_NOCB_WAKE)) + return do_nocb_deferred_wakeup_common(rdp, RCU_NOCB_WAKE); return false; }
Provide a way to tune the deferred wakeup level we want to perform from a safe wakeup point. Currently those sites are: * nocb_timer * user/idle/guest entry * CPU down * softirq/rcuc All of these sites perform the wake up for both RCU_NOCB_WAKE and RCU_NOCB_WAKE_FORCE. In order to merge nocb_timer and nocb_bypass_timer together, we plan to add a new RCU_NOCB_WAKE_BYPASS that really want to be deferred until a timer fires so that we don't wake up the NOCB-gp kthread too early. To prepare for that, integrate a wake up level/limit that a callsite is deemed to perform. Signed-off-by: Frederic Weisbecker <frederic@kernel.org> Cc: Josh Triplett <josh@joshtriplett.org> Cc: Lai Jiangshan <jiangshanlai@gmail.com> Cc: Joel Fernandes <joel@joelfernandes.org> Cc: Neeraj Upadhyay <neeraju@codeaurora.org> Cc: Boqun Feng <boqun.feng@gmail.com> --- kernel/rcu/tree.c | 2 +- kernel/rcu/tree.h | 2 +- kernel/rcu/tree_plugin.h | 15 ++++++++------- 3 files changed, 10 insertions(+), 9 deletions(-)