From patchwork Wed Oct 18 13:37:28 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Rutland X-Patchwork-Id: 116300 Delivered-To: patch@linaro.org Received: by 10.80.163.170 with SMTP id s39csp2382773edb; Wed, 18 Oct 2017 06:37:38 -0700 (PDT) X-Google-Smtp-Source: ABhQp+RTJBvtQIAXfA05Ww8xiatI3FSNWC1X66j8G+z6c2BdEi+lTwyj56H2hvuI54ChEPWe6jKU X-Received: by 10.98.67.154 with SMTP id l26mr8807516pfi.212.1508333858121; Wed, 18 Oct 2017 06:37:38 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1508333858; cv=none; d=google.com; s=arc-20160816; b=Q2POqJMQpanKuC02FGIQiNI7UVYiIOpYBJlHT2gsHH3jSkUlrBRh6Rwq5kUk8VYUuZ FAsXV1RPegnF9C6OpiDJdKf3oHg0ChzLGqh4r0lT770Fs1BVYzP3Q1FeU2I05jwGajwN H5Nd1gA6Qtc66VAnO0Mf+NlhGTHqfnUF26ONlSmxrz1Vi8Sd0dZaOJId5/hB7Tn2oCXg Nbz+msd/8XQTPcIeSsN+kVZ5ZX50CYC1Wk87mZD19qDkjP852ZbsuwJV3BMMw3o4DbPq 44k2WwbM4Qt/LAv1limfm4cKoh44N+1sUSuk1cBcm4BHOQtA/Wp57jdXPPlY/humcU61 Gg3Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:message-id:date:subject:cc:to:from :arc-authentication-results; bh=qj4NEz1k752VeD3h5H/g6LBhvsLkoccYFIfsnCeughI=; b=mrS10hGb8UnhlZRlVi+QxHWENRwTGzfRnXKENWBhFqAoYnw2YqYxTHnSZ4VqYDcc72 bXZEtguMu7uSBGvRK7aSxfzWsLFlMKlTguv7VO5V3+2etnntelI4u569tNhBjP3r41rg kOZN2AQFIoJWmr3KGxLqhqASvMSEqz3w+9Wd+0Kw95nPJmAGhGybj/+F/bGqXAnjnYbX +JTvZugG89+7acOhuSRGWDpY9IAeKMB2aIivXhkUNr3tBt1Hdh2XLFFpksrChcUaeVtf H+uldcv5J+f0a/2w57Gsd5u5kF+15HrnwTXPes1c8r4oL3NJWTD5TDTy/IG/UVHuhS1w PepA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of stable-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=stable-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id y40si2297893pla.545.2017.10.18.06.37.37; Wed, 18 Oct 2017 06:37:38 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of stable-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of stable-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=stable-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1754617AbdJRNhh (ORCPT + 9 others); Wed, 18 Oct 2017 09:37:37 -0400 Received: from foss.arm.com ([217.140.101.70]:40862 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1754615AbdJRNhg (ORCPT ); Wed, 18 Oct 2017 09:37:36 -0400 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 0E2C9F; Wed, 18 Oct 2017 06:37:36 -0700 (PDT) Received: from lakrids.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 65A703F483; Wed, 18 Oct 2017 06:37:34 -0700 (PDT) From: Mark Rutland To: stable@vger.kernel.org Cc: Mark Rutland , Arnd Bergmann , Christoph Lameter , Peter Zijlstra , Pranith Kumar , Tejun Heo , Thomas Gleixner , linux-arch@vger.kernel.org Subject: [PATCH v3.2.y] percpu: make this_cpu_generic_read() atomic w.r.t. interrupts Date: Wed, 18 Oct 2017 14:37:28 +0100 Message-Id: <20171018133728.25174-1-mark.rutland@arm.com> X-Mailer: git-send-email 2.11.0 Sender: stable-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: stable@vger.kernel.org Commit e88d62cd4b2f0b1ae55e9008e79c2794b1fc914d upstream. As raw_cpu_generic_read() is a plain read from a raw_cpu_ptr() address, it's possible (albeit unlikely) that the compiler will split the access across multiple instructions. In this_cpu_generic_read() we disable preemption but not interrupts before calling raw_cpu_generic_read(). Thus, an interrupt could be taken in the middle of the split load instructions. If a this_cpu_write() or RMW this_cpu_*() op is made to the same variable in the interrupt handling path, this_cpu_read() will return a torn value. For native word types, we can avoid tearing using READ_ONCE(), but this won't work in all cases (e.g. 64-bit types on most 32-bit platforms). This patch reworks this_cpu_generic_read() to use READ_ONCE() where possible, otherwise falling back to disabling interrupts. Signed-off-by: Mark Rutland Cc: Arnd Bergmann Cc: Christoph Lameter Cc: Peter Zijlstra Cc: Pranith Kumar Cc: Tejun Heo Cc: Thomas Gleixner Cc: linux-arch@vger.kernel.org Cc: stable@vger.kernel.org Signed-off-by: Tejun Heo [Mark: backport to v3.10.y: folded in __native_word from upstream commits - 47933ad41a86a4a9 ("arch: Introduce smp_load_acquire(), smp_store_release()") - 536fa402221f0963 ("compiler: Allow 1- and 2-byte smp_load_acquire() and smp_store_release()") ] Signed-off-by: Mark Rutland --- include/linux/compiler.h | 5 +++++ include/linux/percpu.h | 29 +++++++++++++++++++++++++---- 2 files changed, 30 insertions(+), 4 deletions(-) -- 2.11.0 diff --git a/include/linux/compiler.h b/include/linux/compiler.h index 01b6d065d609..45289f72c35b 100644 --- a/include/linux/compiler.h +++ b/include/linux/compiler.h @@ -287,6 +287,11 @@ void ftrace_likely_update(struct ftrace_branch_data *f, int val, int expect); # define __same_type(a, b) __builtin_types_compatible_p(typeof(a), typeof(b)) #endif +/* Is this type a native word size -- useful for atomic operations */ +#ifndef __native_word +# define __native_word(t) (sizeof(t) == sizeof(char) || sizeof(t) == sizeof(short) || sizeof(t) == sizeof(int) || sizeof(t) == sizeof(long)) +#endif + /* Compile time object size, -1 for unknown */ #ifndef __compiletime_object_size # define __compiletime_object_size(obj) -1 diff --git a/include/linux/percpu.h b/include/linux/percpu.h index 9ca008f0c542..03e0f0d738fb 100644 --- a/include/linux/percpu.h +++ b/include/linux/percpu.h @@ -321,12 +321,33 @@ do { \ * used. */ -#define _this_cpu_generic_read(pcp) \ -({ typeof(pcp) ret__; \ +#define __this_cpu_generic_read_nopreempt(pcp) \ +({ \ + typeof(pcp) __ret; \ preempt_disable(); \ - ret__ = *this_cpu_ptr(&(pcp)); \ + __ret = ACCESS_ONCE(*this_cpu_ptr(&(pcp))); \ preempt_enable(); \ - ret__; \ + __ret; \ +}) + +#define __this_cpu_generic_read_noirq(pcp) \ +({ \ + typeof(pcp) __ret; \ + unsigned long __flags; \ + raw_local_irq_save(__flags); \ + __ret = *this_cpu_ptr(&(pcp)); \ + raw_local_irq_restore(__flags); \ + __ret; \ +}) + +#define _this_cpu_generic_read(pcp) \ +({ \ + typeof(pcp) __ret; \ + if (__native_word(pcp)) \ + __ret = __this_cpu_generic_read_nopreempt(pcp); \ + else \ + __ret = __this_cpu_generic_read_noirq(pcp); \ + __ret; \ }) #ifndef this_cpu_read