From patchwork Wed Oct 18 13:40:22 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Rutland X-Patchwork-Id: 116301 Delivered-To: patch@linaro.org Received: by 10.80.163.170 with SMTP id s39csp2385451edb; Wed, 18 Oct 2017 06:40:33 -0700 (PDT) X-Google-Smtp-Source: AOwi7QB/m9xOc+4Fl2HrqZvBuNsxjryA/lELoY/MrqIq6C/Cv5kRLUjnmxBuUArlzqTA3K4uPycb X-Received: by 10.99.3.1 with SMTP id 1mr14018644pgd.111.1508334033264; Wed, 18 Oct 2017 06:40:33 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1508334033; cv=none; d=google.com; s=arc-20160816; b=GjVEf90jO2Yf82EfJ2mb46hBmRIRy0KbgGv785HNK2sQM4JYnh7Pe3XTRKnJLCsmGn vK9Ybaxpb/EdMN7ST6t2QelkEZ0WzPo+XZOvBn5RDVdT3RSLxH8kF7iHxtDne1YRK2bw TySa5eAiuL9WAjBbMz409Rqk/VdcDWCvHh8UyX2YbCCHsGQRoERwBbRIkROnKCCL9PlF FhKIVVzaVlSCtR70Ek56vFySoa1wtIgwtZCE6Sd0XIFlIF3nTlXCAX+vjCz7+Vg9l4E/ fAG0gidf72mvN36L3/JFJ9tdqLDbQyaXfprunZi1DnwbUwhVIkg5srHADsvjC5Eu8vbd BCFQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:message-id:date:subject:cc:to:from :arc-authentication-results; bh=qs8kCYtbjTrul6qLltD2bjVXPoMtdNbl/VLRNgzKaPg=; b=msq2f4yeXjT1IACcTlH1CxhVaUUsNHYNuGddIystgTxXgCVD350D/tuUZOaKsBJG7b dziu7BhI0DQgVE/X4fNe/0Vl0jpF4FVNjbgM5lrVDeKg31COKije0I7Tdt4jcIfBVpc2 qN3GlN8EbS4NuD38arhv/KfkC2Xl+Y3qmSgTCIaeRj62yL1ftpf8eli5fP1r97HpQviq gtuOiV2SJf7hLkW2wz3wPX7sz7Tcsb5m0fArcEf9eUnbKaeJ7QbgOE8/UG2ED2GkBaL2 IoKmjvXH1nLFzHqN3ZZw5mSCJSDZ9skfdYQVPWgBzOJIWEsDE2aMl8uFSEUclcBCxPA0 hu7w== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of stable-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=stable-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id z25si6608544pfg.351.2017.10.18.06.40.32; Wed, 18 Oct 2017 06:40:33 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of stable-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of stable-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=stable-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1754652AbdJRNkb (ORCPT + 9 others); Wed, 18 Oct 2017 09:40:31 -0400 Received: from foss.arm.com ([217.140.101.70]:40952 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753823AbdJRNkb (ORCPT ); Wed, 18 Oct 2017 09:40:31 -0400 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 0D3E1F; Wed, 18 Oct 2017 06:40:31 -0700 (PDT) Received: from lakrids.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 639253F483; Wed, 18 Oct 2017 06:40:29 -0700 (PDT) From: Mark Rutland To: stable@vger.kernel.org Cc: Mark Rutland , Arnd Bergmann , Christoph Lameter , Peter Zijlstra , Pranith Kumar , Tejun Heo , Thomas Gleixner , linux-arch@vger.kernel.org Subject: [PATCH v3.16.y] percpu: make this_cpu_generic_read() atomic w.r.t. interrupts Date: Wed, 18 Oct 2017 14:40:22 +0100 Message-Id: <20171018134022.25282-1-mark.rutland@arm.com> X-Mailer: git-send-email 2.11.0 Sender: stable-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: stable@vger.kernel.org Commit e88d62cd4b2f0b1ae55e9008e79c2794b1fc914d upstream. As raw_cpu_generic_read() is a plain read from a raw_cpu_ptr() address, it's possible (albeit unlikely) that the compiler will split the access across multiple instructions. In this_cpu_generic_read() we disable preemption but not interrupts before calling raw_cpu_generic_read(). Thus, an interrupt could be taken in the middle of the split load instructions. If a this_cpu_write() or RMW this_cpu_*() op is made to the same variable in the interrupt handling path, this_cpu_read() will return a torn value. For native word types, we can avoid tearing using READ_ONCE(), but this won't work in all cases (e.g. 64-bit types on most 32-bit platforms). This patch reworks this_cpu_generic_read() to use READ_ONCE() where possible, otherwise falling back to disabling interrupts. Signed-off-by: Mark Rutland Cc: Arnd Bergmann Cc: Christoph Lameter Cc: Peter Zijlstra Cc: Pranith Kumar Cc: Tejun Heo Cc: Thomas Gleixner Cc: linux-arch@vger.kernel.org Cc: stable@vger.kernel.org Signed-off-by: Tejun Heo [Mark: backport to v3.16.y] Signed-off-by: Mark Rutland --- include/linux/percpu.h | 29 +++++++++++++++++++++++++---- 1 file changed, 25 insertions(+), 4 deletions(-) -- 2.11.0 diff --git a/include/linux/percpu.h b/include/linux/percpu.h index 8419053d0f2e..37ff713a7c7f 100644 --- a/include/linux/percpu.h +++ b/include/linux/percpu.h @@ -276,12 +276,33 @@ do { \ * used. */ -#define _this_cpu_generic_read(pcp) \ -({ typeof(pcp) ret__; \ +#define __this_cpu_generic_read_nopreempt(pcp) \ +({ \ + typeof(pcp) __ret; \ preempt_disable(); \ - ret__ = *this_cpu_ptr(&(pcp)); \ + __ret = ACCESS_ONCE(*raw_cpu_ptr(&(pcp))); \ preempt_enable(); \ - ret__; \ + __ret; \ +}) + +#define __this_cpu_generic_read_noirq(pcp) \ +({ \ + typeof(pcp) __ret; \ + unsigned long __flags; \ + raw_local_irq_save(__flags); \ + __ret = *raw_cpu_ptr(&(pcp)); \ + raw_local_irq_restore(__flags); \ + __ret; \ +}) + +#define _this_cpu_generic_read(pcp) \ +({ \ + typeof(pcp) __ret; \ + if (__native_word(pcp)) \ + __ret = __this_cpu_generic_read_nopreempt(pcp); \ + else \ + __ret = __this_cpu_generic_read_noirq(pcp); \ + __ret; \ }) #ifndef this_cpu_read