[tip/core/rcu,10/26] rcu: Exit RCU extended QS on kernel preemption after irq/exception

From: Frederic Weisbecker <fweisbec@gmail.com>

When an exception or an irq exits, and we are going to resume into
interrupted kernel code, the low level architecture code calls
preempt_schedule_irq() if there is a need to reschedule.

If the interrupt/exception occured between a call to rcu_user_enter()
(from syscall exit, exception exit, do_notify_resume exit, ...) and
a real resume to userspace (iret,...), preempt_schedule_irq() can be
called whereas RCU thinks we are in userspace. But preempt_schedule_irq()
is going to run kernel code and may be some RCU read side critical
section. We must exit the userspace extended quiescent state before
we call it.

To solve this, just call rcu_user_exit() in the beginning of
preempt_schedule_irq().

Signed-off-by: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Alessio Igor Bogani <abogani@kernel.org>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Avi Kivity <avi@redhat.com>
Cc: Chris Metcalf <cmetcalf@tilera.com>
Cc: Christoph Lameter <cl@linux.com>
Cc: Geoff Levand <geoff@infradead.org>
Cc: Gilad Ben Yossef <gilad@benyossef.com>
Cc: Hakan Akkan <hakanakkan@gmail.com>
Cc: H. Peter Anvin <hpa@zytor.com>
Cc: Ingo Molnar <mingo@kernel.org>
Cc: Josh Triplett <josh@joshtriplett.org>
Cc: Kevin Hilman <khilman@ti.com>
Cc: Max Krasnyansky <maxk@qualcomm.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Stephen Hemminger <shemminger@vyatta.com>
Cc: Steven Rostedt <rostedt@goodmis.org>
Cc: Sven-Thorsten Dietrich <thebigcorporation@gmail.com>
Cc: Thomas Gleixner <tglx@linutronix.de>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
---
 kernel/sched/core.c |    1 +
 1 files changed, 1 insertions(+), 0 deletions(-)

On Thu, Aug 30, 2012 at 02:05:27PM -0700, Paul E. McKenney wrote:
> From: Frederic Weisbecker <fweisbec@gmail.com>
> 
> When an exception or an irq exits, and we are going to resume into
> interrupted kernel code, the low level architecture code calls
> preempt_schedule_irq() if there is a need to reschedule.
> 
> If the interrupt/exception occured between a call to rcu_user_enter()
> (from syscall exit, exception exit, do_notify_resume exit, ...) and
> a real resume to userspace (iret,...), preempt_schedule_irq() can be
> called whereas RCU thinks we are in userspace. But preempt_schedule_irq()
> is going to run kernel code and may be some RCU read side critical
> section. We must exit the userspace extended quiescent state before
> we call it.
> 
> To solve this, just call rcu_user_exit() in the beginning of
> preempt_schedule_irq().
> 
> Signed-off-by: Frederic Weisbecker <fweisbec@gmail.com>
> Cc: Alessio Igor Bogani <abogani@kernel.org>
> Cc: Andrew Morton <akpm@linux-foundation.org>
> Cc: Avi Kivity <avi@redhat.com>
> Cc: Chris Metcalf <cmetcalf@tilera.com>
> Cc: Christoph Lameter <cl@linux.com>
> Cc: Geoff Levand <geoff@infradead.org>
> Cc: Gilad Ben Yossef <gilad@benyossef.com>
> Cc: Hakan Akkan <hakanakkan@gmail.com>
> Cc: H. Peter Anvin <hpa@zytor.com>
> Cc: Ingo Molnar <mingo@kernel.org>
> Cc: Josh Triplett <josh@joshtriplett.org>
> Cc: Kevin Hilman <khilman@ti.com>
> Cc: Max Krasnyansky <maxk@qualcomm.com>
> Cc: Peter Zijlstra <peterz@infradead.org>
> Cc: Stephen Hemminger <shemminger@vyatta.com>
> Cc: Steven Rostedt <rostedt@goodmis.org>
> Cc: Sven-Thorsten Dietrich <thebigcorporation@gmail.com>
> Cc: Thomas Gleixner <tglx@linutronix.de>
> Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>

Reviewed-by: Josh Triplett <josh@joshtriplett.org>

>  kernel/sched/core.c |    1 +
>  1 files changed, 1 insertions(+), 0 deletions(-)
> 
> diff --git a/kernel/sched/core.c b/kernel/sched/core.c
> index 07c6d9a..0bd599b 100644
> --- a/kernel/sched/core.c
> +++ b/kernel/sched/core.c
> @@ -3564,6 +3564,7 @@ asmlinkage void __sched preempt_schedule_irq(void)
>  	/* Catch callers which need to be fixed */
>  	BUG_ON(ti->preempt_count || !irqs_disabled());
>  
> +	rcu_user_exit();
>  	do {
>  		add_preempt_count(PREEMPT_ACTIVE);
>  		local_irq_enable();
> -- 
> 1.7.8
>

On Thu, 2012-08-30 at 14:05 -0700, Paul E. McKenney wrote:
> From: Frederic Weisbecker <fweisbec@gmail.com>
> 
> When an exception or an irq exits, and we are going to resume into
> interrupted kernel code, the low level architecture code calls
> preempt_schedule_irq() if there is a need to reschedule.
> 
> If the interrupt/exception occured between a call to rcu_user_enter()
> (from syscall exit, exception exit, do_notify_resume exit, ...) and
> a real resume to userspace (iret,...), preempt_schedule_irq() can be
> called whereas RCU thinks we are in userspace. But preempt_schedule_irq()
> is going to run kernel code and may be some RCU read side critical
> section. We must exit the userspace extended quiescent state before
> we call it.
> 
> To solve this, just call rcu_user_exit() in the beginning of
> preempt_schedule_irq().

> ---
>  kernel/sched/core.c |    1 +
>  1 files changed, 1 insertions(+), 0 deletions(-)
> 
> diff --git a/kernel/sched/core.c b/kernel/sched/core.c
> index 07c6d9a..0bd599b 100644
> --- a/kernel/sched/core.c
> +++ b/kernel/sched/core.c
> @@ -3564,6 +3564,7 @@ asmlinkage void __sched preempt_schedule_irq(void)
>  	/* Catch callers which need to be fixed */
>  	BUG_ON(ti->preempt_count || !irqs_disabled());
>  
> +	rcu_user_exit();
>  	do {
>  		add_preempt_count(PREEMPT_ACTIVE);
>  		local_irq_enable();

But in this case you want to restart the tick as well, are we going to
add a nohz_user_exit() like call as well, or do we have something that
does both?

On Thu, Sep 06, 2012 at 06:52:44PM +0200, Peter Zijlstra wrote:
> On Thu, 2012-08-30 at 14:05 -0700, Paul E. McKenney wrote:
> > From: Frederic Weisbecker <fweisbec@gmail.com>
> > 
> > When an exception or an irq exits, and we are going to resume into
> > interrupted kernel code, the low level architecture code calls
> > preempt_schedule_irq() if there is a need to reschedule.
> > 
> > If the interrupt/exception occured between a call to rcu_user_enter()
> > (from syscall exit, exception exit, do_notify_resume exit, ...) and
> > a real resume to userspace (iret,...), preempt_schedule_irq() can be
> > called whereas RCU thinks we are in userspace. But preempt_schedule_irq()
> > is going to run kernel code and may be some RCU read side critical
> > section. We must exit the userspace extended quiescent state before
> > we call it.
> > 
> > To solve this, just call rcu_user_exit() in the beginning of
> > preempt_schedule_irq().
> 
> > ---
> >  kernel/sched/core.c |    1 +
> >  1 files changed, 1 insertions(+), 0 deletions(-)
> > 
> > diff --git a/kernel/sched/core.c b/kernel/sched/core.c
> > index 07c6d9a..0bd599b 100644
> > --- a/kernel/sched/core.c
> > +++ b/kernel/sched/core.c
> > @@ -3564,6 +3564,7 @@ asmlinkage void __sched preempt_schedule_irq(void)
> >  	/* Catch callers which need to be fixed */
> >  	BUG_ON(ti->preempt_count || !irqs_disabled());
> >  
> > +	rcu_user_exit();
> >  	do {
> >  		add_preempt_count(PREEMPT_ACTIVE);
> >  		local_irq_enable();
> 
> But in this case you want to restart the tick as well, are we going to
> add a nohz_user_exit() like call as well, or do we have something that
> does both?

Yeah this will soon become user_exit() and that hook will rely on the
nohz engine and RCU APIs accordingly.

For now I'm only focusing on the RCU user mode. In this patchset it
does the RCU user mode unconditionally. I'm just doing this to integrate
the things incrementally.

First RCU, then we turn that into some generic user hooks, then we handle
the cputime with a generic vtime, then we integrate the nohz engine which
drives the user_hooks such that we stop/restart the tick and RCU user
mode accordingly.