From patchwork Thu Aug 30 21:05:22 2012 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paul E. McKenney" X-Patchwork-Id: 11092 Return-Path: X-Original-To: patchwork@peony.canonical.com Delivered-To: patchwork@peony.canonical.com Received: from fiordland.canonical.com (fiordland.canonical.com [91.189.94.145]) by peony.canonical.com (Postfix) with ESMTP id 89B2123E52 for ; Thu, 30 Aug 2012 21:14:28 +0000 (UTC) Received: from mail-ie0-f180.google.com (mail-ie0-f180.google.com [209.85.223.180]) by fiordland.canonical.com (Postfix) with ESMTP id D5F24A18196 for ; Thu, 30 Aug 2012 21:13:51 +0000 (UTC) Received: by mail-ie0-f180.google.com with SMTP id k11so1133172iea.11 for ; Thu, 30 Aug 2012 14:14:27 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=x-forwarded-to:x-forwarded-for:delivered-to:received-spf:from:to:cc :subject:date:message-id:x-mailer:in-reply-to:references :x-content-scanned:x-cbid:x-gm-message-state; bh=isf3BMqQJvAaGZU/PIYOeViHlE8eQ/BBXVVtZQPuySs=; b=QQ9sJwcjY3EgjVt6zZ+gjdoH9nyOhiE/oJXw7R0hFYJ/eCKAcB1cTLPLdQWZD9RdmO 9gFguCNNJhD0gLqvdb05eNiLMBQtZrxideubcKEHYXMR25ErDUMRgg8ozZeB3vV+/RbU 227I+Yn+Hs8NQ3d/fmOsZYGDkDLM/moB0eZYLZwZEzXoslvBDNfOSSMKHwTvFId0882Y EAfhee2cjSA6yn2LO6ace9EmuUntf9G0GM4oCGd7dW807bCTVJ4eLIxXDj+nL7K83EMB bnXROkWLF0Qn4tXVUM4ZVlKL4eG6y6KaHCne3oAJ8N/2oMXSIjF9ra0svKL/BoMSbXyh LXcA== Received: by 10.50.159.196 with SMTP id xe4mr2357043igb.43.1346361267913; Thu, 30 Aug 2012 14:14:27 -0700 (PDT) X-Forwarded-To: linaro-patchwork@canonical.com X-Forwarded-For: patch@linaro.org linaro-patchwork@canonical.com Delivered-To: patches@linaro.org Received: by 10.50.184.232 with SMTP id ex8csp31675igc; Thu, 30 Aug 2012 14:14:27 -0700 (PDT) Received: by 10.50.187.162 with SMTP id ft2mr2478037igc.11.1346361267623; Thu, 30 Aug 2012 14:14:27 -0700 (PDT) Received: from e4.ny.us.ibm.com (e4.ny.us.ibm.com. [32.97.182.144]) by mx.google.com with ESMTPS id cx8si2610734igc.31.2012.08.30.14.14.27 (version=TLSv1/SSLv3 cipher=OTHER); Thu, 30 Aug 2012 14:14:27 -0700 (PDT) Received-SPF: pass (google.com: domain of paulmck@linux.vnet.ibm.com designates 32.97.182.144 as permitted sender) client-ip=32.97.182.144; Authentication-Results: mx.google.com; spf=pass (google.com: domain of paulmck@linux.vnet.ibm.com designates 32.97.182.144 as permitted sender) smtp.mail=paulmck@linux.vnet.ibm.com Received: from /spool/local by e4.ny.us.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Thu, 30 Aug 2012 17:14:25 -0400 Received: from d01dlp01.pok.ibm.com (9.56.250.166) by e4.ny.us.ibm.com (192.168.1.104) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; Thu, 30 Aug 2012 17:13:53 -0400 Received: from d01relay04.pok.ibm.com (d01relay04.pok.ibm.com [9.56.227.236]) by d01dlp01.pok.ibm.com (Postfix) with ESMTP id B617738C803F for ; Thu, 30 Aug 2012 17:13:52 -0400 (EDT) Received: from d03av01.boulder.ibm.com (d03av01.boulder.ibm.com [9.17.195.167]) by d01relay04.pok.ibm.com (8.13.8/8.13.8/NCO v10.0) with ESMTP id q7ULDqqJ194446 for ; Thu, 30 Aug 2012 17:13:52 -0400 Received: from d03av01.boulder.ibm.com (loopback [127.0.0.1]) by d03av01.boulder.ibm.com (8.14.4/8.13.1/NCO v10.0 AVout) with ESMTP id q7ULDZ6d009341 for ; Thu, 30 Aug 2012 15:13:50 -0600 Received: from paulmck-ThinkPad-W500 (sig-9-65-153-225.mts.ibm.com [9.65.153.225]) by d03av01.boulder.ibm.com (8.14.4/8.13.1/NCO v10.0 AVin) with ESMTP id q7ULDYcj009161; Thu, 30 Aug 2012 15:13:34 -0600 Received: by paulmck-ThinkPad-W500 (Postfix, from userid 1000) id 00863EA83E; Thu, 30 Aug 2012 14:05:45 -0700 (PDT) From: "Paul E. McKenney" To: linux-kernel@vger.kernel.org Cc: mingo@elte.hu, laijs@cn.fujitsu.com, dipankar@in.ibm.com, akpm@linux-foundation.org, mathieu.desnoyers@polymtl.ca, josh@joshtriplett.org, niv@us.ibm.com, tglx@linutronix.de, peterz@infradead.org, rostedt@goodmis.org, Valdis.Kletnieks@vt.edu, dhowells@redhat.com, eric.dumazet@gmail.com, darren@dvhart.com, fweisbec@gmail.com, sbw@mit.edu, patches@linaro.org, Alessio Igor Bogani , Avi Kivity , Chris Metcalf , Christoph Lameter , Geoff Levand , Gilad Ben Yossef , Hakan Akkan , "H. Peter Anvin" , Ingo Molnar , Kevin Hilman , Max Krasnyansky , Stephen Hemminger , Sven-Thorsten Dietrich , "Paul E. McKenney" Subject: [PATCH tip/core/rcu 05/26] rcu: Allow rcu_user_enter()/exit() to nest Date: Thu, 30 Aug 2012 14:05:22 -0700 Message-Id: <1346360743-3628-5-git-send-email-paulmck@linux.vnet.ibm.com> X-Mailer: git-send-email 1.7.8 In-Reply-To: <1346360743-3628-1-git-send-email-paulmck@linux.vnet.ibm.com> References: <20120830210520.GA2824@linux.vnet.ibm.com> <1346360743-3628-1-git-send-email-paulmck@linux.vnet.ibm.com> X-Content-Scanned: Fidelis XPS MAILER x-cbid: 12083021-3534-0000-0000-00000C0BE4DA X-Gm-Message-State: ALoCoQk65GTvDWSt4ahMQl0FqIUzU9f7EUrq5GhK/d9AtC6Jwn1VL34rbMkRx+JOJHBoJXXGzX3i From: Frederic Weisbecker Allow calls to rcu_user_enter() even if we are already in userspace (as seen by RCU) and allow calls to rcu_user_exit() even if we are already in the kernel. This makes the APIs more flexible to be called from architectures. Exception entries for example won't need to know if they come from userspace before calling rcu_user_exit(). Signed-off-by: Frederic Weisbecker Cc: Alessio Igor Bogani Cc: Andrew Morton Cc: Avi Kivity Cc: Chris Metcalf Cc: Christoph Lameter Cc: Geoff Levand Cc: Gilad Ben Yossef Cc: Hakan Akkan Cc: H. Peter Anvin Cc: Ingo Molnar Cc: Josh Triplett Cc: Kevin Hilman Cc: Max Krasnyansky Cc: Peter Zijlstra Cc: Stephen Hemminger Cc: Steven Rostedt Cc: Sven-Thorsten Dietrich Cc: Thomas Gleixner Signed-off-by: Paul E. McKenney Reviewed-by: Josh Triplett --- kernel/rcutree.c | 41 +++++++++++++++++++++++++++++++++-------- kernel/rcutree.h | 3 +++ 2 files changed, 36 insertions(+), 8 deletions(-) diff --git a/kernel/rcutree.c b/kernel/rcutree.c index e287c4a..8bbc7fb 100644 --- a/kernel/rcutree.c +++ b/kernel/rcutree.c @@ -390,11 +390,9 @@ static void rcu_eqs_enter_common(struct rcu_dynticks *rdtp, long long oldval, */ static void rcu_eqs_enter(bool user) { - unsigned long flags; long long oldval; struct rcu_dynticks *rdtp; - local_irq_save(flags); rdtp = &__get_cpu_var(rcu_dynticks); oldval = rdtp->dynticks_nesting; WARN_ON_ONCE((oldval & DYNTICK_TASK_NEST_MASK) == 0); @@ -403,7 +401,6 @@ static void rcu_eqs_enter(bool user) else rdtp->dynticks_nesting -= DYNTICK_TASK_NEST_VALUE; rcu_eqs_enter_common(rdtp, oldval, user); - local_irq_restore(flags); } /** @@ -420,7 +417,11 @@ static void rcu_eqs_enter(bool user) */ void rcu_idle_enter(void) { + unsigned long flags; + + local_irq_save(flags); rcu_eqs_enter(0); + local_irq_restore(flags); } EXPORT_SYMBOL_GPL(rcu_idle_enter); @@ -435,7 +436,18 @@ EXPORT_SYMBOL_GPL(rcu_idle_enter); */ void rcu_user_enter(void) { - rcu_eqs_enter(1); + unsigned long flags; + struct rcu_dynticks *rdtp; + + WARN_ON_ONCE(!current->mm); + + local_irq_save(flags); + rdtp = &__get_cpu_var(rcu_dynticks); + if (!rdtp->in_user) { + rdtp->in_user = true; + rcu_eqs_enter(1); + } + local_irq_restore(flags); } EXPORT_SYMBOL_GPL(rcu_user_enter); @@ -530,11 +542,9 @@ static void rcu_eqs_exit_common(struct rcu_dynticks *rdtp, long long oldval, */ static void rcu_eqs_exit(bool user) { - unsigned long flags; struct rcu_dynticks *rdtp; long long oldval; - local_irq_save(flags); rdtp = &__get_cpu_var(rcu_dynticks); oldval = rdtp->dynticks_nesting; WARN_ON_ONCE(oldval < 0); @@ -543,7 +553,6 @@ static void rcu_eqs_exit(bool user) else rdtp->dynticks_nesting = DYNTICK_TASK_EXIT_IDLE; rcu_eqs_exit_common(rdtp, oldval, user); - local_irq_restore(flags); } /** @@ -559,7 +568,11 @@ static void rcu_eqs_exit(bool user) */ void rcu_idle_exit(void) { + unsigned long flags; + + local_irq_save(flags); rcu_eqs_exit(0); + local_irq_restore(flags); } EXPORT_SYMBOL_GPL(rcu_idle_exit); @@ -572,7 +585,16 @@ EXPORT_SYMBOL_GPL(rcu_idle_exit); */ void rcu_user_exit(void) { - rcu_eqs_exit(1); + unsigned long flags; + struct rcu_dynticks *rdtp; + + local_irq_save(flags); + rdtp = &__get_cpu_var(rcu_dynticks); + if (rdtp->in_user) { + rdtp->in_user = false; + rcu_eqs_exit(1); + } + local_irq_restore(flags); } EXPORT_SYMBOL_GPL(rcu_user_exit); @@ -2590,6 +2612,9 @@ rcu_boot_init_percpu_data(int cpu, struct rcu_state *rsp) rdp->dynticks = &per_cpu(rcu_dynticks, cpu); WARN_ON_ONCE(rdp->dynticks->dynticks_nesting != DYNTICK_TASK_EXIT_IDLE); WARN_ON_ONCE(atomic_read(&rdp->dynticks->dynticks) != 1); +#ifdef CONFIG_RCU_USER_QS + WARN_ON_ONCE(rdp->dynticks->in_user); +#endif rdp->cpu = cpu; rdp->rsp = rsp; raw_spin_unlock_irqrestore(&rnp->lock, flags); diff --git a/kernel/rcutree.h b/kernel/rcutree.h index 4d29169..0dd5fd6 100644 --- a/kernel/rcutree.h +++ b/kernel/rcutree.h @@ -102,6 +102,9 @@ struct rcu_dynticks { /* idle-period nonlazy_posted snapshot. */ int tick_nohz_enabled_snap; /* Previously seen value from sysfs. */ #endif /* #ifdef CONFIG_RCU_FAST_NO_HZ */ +#ifdef CONFIG_RCU_USER_QS + bool in_user; /* Is the CPU in userland from RCU POV? */ +#endif }; /* RCU's kthread states for tracing. */