From patchwork Wed Oct 18 13:33:43 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Rutland X-Patchwork-Id: 116297 Delivered-To: patch@linaro.org Received: by 10.80.163.170 with SMTP id s39csp2379108edb; Wed, 18 Oct 2017 06:33:56 -0700 (PDT) X-Google-Smtp-Source: ABhQp+SAYF2J8PYziLGen/YL9Ej7IQq7EkUWjv7Je5uieb+wUk5i+upRFCWN8D9eruy8I3J0Hdn1 X-Received: by 10.101.90.73 with SMTP id z9mr1822434pgs.76.1508333636267; Wed, 18 Oct 2017 06:33:56 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1508333636; cv=none; d=google.com; s=arc-20160816; b=kInS5isW/ld2t56Q0fnAUaX3YmupcvwZr8OXZX2cew277kO+48QhsZEoHnxqv34MCB rW5+PmepbI9WtH5uPhMK0IsaNuvF60IGFwzsJz4npC/hGCldxINf4vhm4MAe/oct/wrq sfWWN3+k/OTqz0XD/TmwyQzoOlDqLEXnjogzK101CUvs+YgYMB0Tpr3jCpUtW3p/Vrhx A2DytgK1mPXFy/Dtq7bqhWszoJzczIPbndpCcAonapd6+9qAhPE9ksOCUe5yEUskXErm rlPmRl9sflst8munSmDKi117hYeDYJrk+YJ54Sg/elcINbZfri1rPud0nEJ9bUoUT8VM mlDQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:message-id:date:subject:cc:to:from :arc-authentication-results; bh=DXERmwPGpyrthqQPZVLi3z5sEcC8vz7Sl6QYdXPgmn4=; b=Bf9lhvdyeTP2aIT/6BPszqMY/a0z46YmUZ1n8rf2JfeP/02VLHcfNxIaS+6UDmMcNv kadDULeTmQAU57sQHj2fLuSC5WAONXk16Rku0vBuZIyDT5ZX+TJGSeLdcYwgn99dmPOh 8JwMSmZYIeaFyC+dd5SSkRmI/8TAquso2ojzIoaufdMJbcfNp+iCdUT/6vJmx1XZOqnt xGE84iJPDjXta7cLcEemEHC+B+M5zhrigGU33fw8Yyz7aL4loPSbBBePX8n4T0Yy+RsT nLf6NSuze7zEEDtVVao/2YJr6KJ/5FbMKqOrn7rkYaK9dpkY83CSiTAZU5V8DOhC2Nsr Nbwg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of stable-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=stable-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id g34si7865717pld.207.2017.10.18.06.33.55; Wed, 18 Oct 2017 06:33:56 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of stable-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of stable-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=stable-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1754154AbdJRNdx (ORCPT + 9 others); Wed, 18 Oct 2017 09:33:53 -0400 Received: from usa-sjc-mx-foss1.foss.arm.com ([217.140.101.70]:40672 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752652AbdJRNdw (ORCPT ); Wed, 18 Oct 2017 09:33:52 -0400 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 123CFF; Wed, 18 Oct 2017 06:33:52 -0700 (PDT) Received: from lakrids.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 69BCB3F483; Wed, 18 Oct 2017 06:33:50 -0700 (PDT) From: Mark Rutland To: stable@vger.kernel.org Cc: Mark Rutland , Arnd Bergmann , Christoph Lameter , Peter Zijlstra , Pranith Kumar , Tejun Heo , Thomas Gleixner , linux-arch@vger.kernel.org Subject: [PATCH v4.1.y] percpu: make this_cpu_generic_read() atomic w.r.t. interrupts Date: Wed, 18 Oct 2017 14:33:43 +0100 Message-Id: <20171018133343.55337-1-mark.rutland@arm.com> X-Mailer: git-send-email 2.11.0 Sender: stable-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: stable@vger.kernel.org Commit e88d62cd4b2f0b1ae55e9008e79c2794b1fc914d upstream. As raw_cpu_generic_read() is a plain read from a raw_cpu_ptr() address, it's possible (albeit unlikely) that the compiler will split the access across multiple instructions. In this_cpu_generic_read() we disable preemption but not interrupts before calling raw_cpu_generic_read(). Thus, an interrupt could be taken in the middle of the split load instructions. If a this_cpu_write() or RMW this_cpu_*() op is made to the same variable in the interrupt handling path, this_cpu_read() will return a torn value. For native word types, we can avoid tearing using READ_ONCE(), but this won't work in all cases (e.g. 64-bit types on most 32-bit platforms). This patch reworks this_cpu_generic_read() to use READ_ONCE() where possible, otherwise falling back to disabling interrupts. Signed-off-by: Mark Rutland Cc: Arnd Bergmann Cc: Christoph Lameter Cc: Peter Zijlstra Cc: Pranith Kumar Cc: Tejun Heo Cc: Thomas Gleixner Cc: linux-arch@vger.kernel.org Cc: stable@vger.kernel.org Signed-off-by: Tejun Heo [Mark: backport to v4.1.y] Signed-off-by: Mark Rutland --- include/asm-generic/percpu.h | 24 ++++++++++++++++++++++-- 1 file changed, 22 insertions(+), 2 deletions(-) -- 2.11.0 diff --git a/include/asm-generic/percpu.h b/include/asm-generic/percpu.h index 4d9f233c4ba8..7d58ffdacd62 100644 --- a/include/asm-generic/percpu.h +++ b/include/asm-generic/percpu.h @@ -105,15 +105,35 @@ do { \ (__ret); \ }) -#define this_cpu_generic_read(pcp) \ +#define __this_cpu_generic_read_nopreempt(pcp) \ ({ \ typeof(pcp) __ret; \ preempt_disable(); \ - __ret = *this_cpu_ptr(&(pcp)); \ + __ret = READ_ONCE(*raw_cpu_ptr(&(pcp))); \ preempt_enable(); \ __ret; \ }) +#define __this_cpu_generic_read_noirq(pcp) \ +({ \ + typeof(pcp) __ret; \ + unsigned long __flags; \ + raw_local_irq_save(__flags); \ + __ret = *raw_cpu_ptr(&(pcp)); \ + raw_local_irq_restore(__flags); \ + __ret; \ +}) + +#define this_cpu_generic_read(pcp) \ +({ \ + typeof(pcp) __ret; \ + if (__native_word(pcp)) \ + __ret = __this_cpu_generic_read_nopreempt(pcp); \ + else \ + __ret = __this_cpu_generic_read_noirq(pcp); \ + __ret; \ +}) + #define this_cpu_generic_to_op(pcp, val, op) \ do { \ unsigned long __flags; \