From patchwork Wed Oct 18 13:36:41 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Rutland X-Patchwork-Id: 116299 Delivered-To: patch@linaro.org Received: by 10.80.163.170 with SMTP id s39csp2382143edb; Wed, 18 Oct 2017 06:36:55 -0700 (PDT) X-Google-Smtp-Source: AOwi7QDlBZmoSab7DcUipThf2oP8gh1mvVdlHo9QIowun2kASQfxz++iWfHPtbanrgVj96H16XOr X-Received: by 10.84.178.101 with SMTP id y92mr15102298plb.83.1508333815848; Wed, 18 Oct 2017 06:36:55 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1508333815; cv=none; d=google.com; s=arc-20160816; b=WCZt+PEtqoOCLzeZ72e6FUtDKVAeeO+FJJ4N208fDyvz+xP8nJNRRaPcsq3ZVQWZFp k7+Z7FREZs08VskJVU3zXCm9KhCta8ho1R1/mE4jM2pCvjY9emn2xMuRS5i/2ZnFaDa+ Q/OYkzZ/GOWck1/ne6AIP31DL0zri2I9DTndnO6iupEczEQ4G2emUVxkWiuLW02ZBo2I BKnR9DFPbILXm0jrwNRU4E+a0mllmiJJH2cZJFajj4oGPclV+1SX1UCJfKmjvNfwB3h2 bX2Xbmg5BU2r4JiHssQml0oc0UqOzwZSx0Dvws1T/8IDH+qXEYb+fu7dU8U5pyi8rnwH hIxA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:message-id:date:subject:cc:to:from :arc-authentication-results; bh=V73Edq3EXgb+92U0fljm1Hrwtx8bW+HjJXYuyxExajU=; b=rh0r7HmJ7cZI2x7r5Q/LRe0FZl3WMCFbZPMj8SPBWbOG/SwxpjDhYk4jV5p+kl2SCD J6kNhtvM/ZxOOMF/4aU739PU6D9ZuOTkwCX0i3fTSToDwzaF3L8/rdcQJlXrk8RlU80y leERD6BeiiAWtm8sV1nHKYXRC/bvnC1YlYTv+FLX6kmHlxLs6EivMuK8VkSeVRjFP1aw Od+UZEEHh29m2z3AV6fLCyZmcv9/PStcMzb4qsY2UUohVY/x2F5IgPydvCenzVjTT4pA XyIjZP/bUNzGFFZyOnoN6ivDSkmVcQwpb4NIKLZIuLAcbLCxghpgZKunnCIt2UiBE7wE PhxA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of stable-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=stable-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id y40si2297893pla.545.2017.10.18.06.36.55; Wed, 18 Oct 2017 06:36:55 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of stable-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of stable-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=stable-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1754604AbdJRNgx (ORCPT + 9 others); Wed, 18 Oct 2017 09:36:53 -0400 Received: from usa-sjc-mx-foss1.foss.arm.com ([217.140.101.70]:40796 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1754398AbdJRNgu (ORCPT ); Wed, 18 Oct 2017 09:36:50 -0400 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 35845F; Wed, 18 Oct 2017 06:36:50 -0700 (PDT) Received: from lakrids.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 8914A3F483; Wed, 18 Oct 2017 06:36:48 -0700 (PDT) From: Mark Rutland To: stable@vger.kernel.org Cc: Mark Rutland , Arnd Bergmann , Christoph Lameter , Peter Zijlstra , Pranith Kumar , Tejun Heo , Thomas Gleixner , linux-arch@vger.kernel.org Subject: [PATCH v3.10.y] percpu: make this_cpu_generic_read() atomic w.r.t. interrupts Date: Wed, 18 Oct 2017 14:36:41 +0100 Message-Id: <20171018133641.25078-1-mark.rutland@arm.com> X-Mailer: git-send-email 2.11.0 Sender: stable-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: stable@vger.kernel.org Commit e88d62cd4b2f0b1ae55e9008e79c2794b1fc914d upstream. As raw_cpu_generic_read() is a plain read from a raw_cpu_ptr() address, it's possible (albeit unlikely) that the compiler will split the access across multiple instructions. In this_cpu_generic_read() we disable preemption but not interrupts before calling raw_cpu_generic_read(). Thus, an interrupt could be taken in the middle of the split load instructions. If a this_cpu_write() or RMW this_cpu_*() op is made to the same variable in the interrupt handling path, this_cpu_read() will return a torn value. For native word types, we can avoid tearing using READ_ONCE(), but this won't work in all cases (e.g. 64-bit types on most 32-bit platforms). This patch reworks this_cpu_generic_read() to use READ_ONCE() where possible, otherwise falling back to disabling interrupts. Signed-off-by: Mark Rutland Cc: Arnd Bergmann Cc: Christoph Lameter Cc: Peter Zijlstra Cc: Pranith Kumar Cc: Tejun Heo Cc: Thomas Gleixner Cc: linux-arch@vger.kernel.org Cc: stable@vger.kernel.org Signed-off-by: Tejun Heo [Mark: backport to v3.10.y: folded in __native_word from upstream commits - 47933ad41a86a4a9 ("arch: Introduce smp_load_acquire(), smp_store_release()") - 536fa402221f0963 ("compiler: Allow 1- and 2-byte smp_load_acquire() and smp_store_release()") ] Signed-off-by: Mark Rutland --- include/linux/compiler.h | 5 +++++ include/linux/percpu.h | 29 +++++++++++++++++++++++++---- 2 files changed, 30 insertions(+), 4 deletions(-) -- 2.11.0 diff --git a/include/linux/compiler.h b/include/linux/compiler.h index a2ce6f8871c4..84d36b5ce832 100644 --- a/include/linux/compiler.h +++ b/include/linux/compiler.h @@ -302,6 +302,11 @@ void ftrace_likely_update(struct ftrace_branch_data *f, int val, int expect); # define __same_type(a, b) __builtin_types_compatible_p(typeof(a), typeof(b)) #endif +/* Is this type a native word size -- useful for atomic operations */ +#ifndef __native_word +# define __native_word(t) (sizeof(t) == sizeof(char) || sizeof(t) == sizeof(short) || sizeof(t) == sizeof(int) || sizeof(t) == sizeof(long)) +#endif + /* Compile time object size, -1 for unknown */ #ifndef __compiletime_object_size # define __compiletime_object_size(obj) -1 diff --git a/include/linux/percpu.h b/include/linux/percpu.h index cc88172c7d9a..3c047cd54ca0 100644 --- a/include/linux/percpu.h +++ b/include/linux/percpu.h @@ -267,12 +267,33 @@ do { \ * used. */ -#define _this_cpu_generic_read(pcp) \ -({ typeof(pcp) ret__; \ +#define __this_cpu_generic_read_nopreempt(pcp) \ +({ \ + typeof(pcp) __ret; \ preempt_disable(); \ - ret__ = *this_cpu_ptr(&(pcp)); \ + __ret = ACCESS_ONCE(*this_cpu_ptr(&(pcp))); \ preempt_enable(); \ - ret__; \ + __ret; \ +}) + +#define __this_cpu_generic_read_noirq(pcp) \ +({ \ + typeof(pcp) __ret; \ + unsigned long __flags; \ + raw_local_irq_save(__flags); \ + __ret = *this_cpu_ptr(&(pcp)); \ + raw_local_irq_restore(__flags); \ + __ret; \ +}) + +#define _this_cpu_generic_read(pcp) \ +({ \ + typeof(pcp) __ret; \ + if (__native_word(pcp)) \ + __ret = __this_cpu_generic_read_nopreempt(pcp); \ + else \ + __ret = __this_cpu_generic_read_noirq(pcp); \ + __ret; \ }) #ifndef this_cpu_read