From patchwork Wed Oct 18 13:35:24 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Rutland X-Patchwork-Id: 116298 Delivered-To: patch@linaro.org Received: by 10.80.163.170 with SMTP id s39csp2380846edb; Wed, 18 Oct 2017 06:35:35 -0700 (PDT) X-Google-Smtp-Source: AOwi7QDMaLaQjZtDCC/R8tFR1Zwhph/lnhSF9YIjYHETzWEthdt1Vk3OZ6V1lZV8btL6kkRW3++e X-Received: by 10.98.155.218 with SMTP id e87mr14826349pfk.96.1508333734925; Wed, 18 Oct 2017 06:35:34 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1508333734; cv=none; d=google.com; s=arc-20160816; b=uDGZ5ehPzDumhf5MUQmn9xwO88e9y75Y1yKwI/KVOBAAdNv0Z7DVBmF6FjyrNqCPU3 22rRKMEaOxN9T7Q+6JBjcUJKdnR+TRnbyUaXqOEziDvXsSb/x7eC93LZ6O8FoYgtZLn9 3Wxd1PgRH/NKKYvIM+tLBbCcV66cp2vn0cecjl3SEvCgufGnKwSI2GsCSm7616ULiGmD TyRrVOhVTQ1l9CJQsBsy2kwdSbJhCeD5N+iKNN0/2UM9d4OSR+SapQq/aUcHjkstf9zE FiF2k1pHUSzV6GDcX8POANF/U9SqfEObd/zpf0jmG3ZalBolHHWf1AP27s8Ix6izijaz K4aw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:message-id:date:subject:cc:to:from :arc-authentication-results; bh=qs8kCYtbjTrul6qLltD2bjVXPoMtdNbl/VLRNgzKaPg=; b=pH2v4YZjGiL1iK71UHb6nQy/LtqCKMIzZXvatfNtgK3YVdqTS6zla12hsYME8T5dZQ hePsw2rjbcWSWlKYAE89oJVCPUzBTFKbUIEY4iAmI3X+ijqtaUuPUXfpgI3lfzAnufS0 8bEOVFP1Of6MYbWFBdJq1r4Ue4lT49V1ikhuHUAlBccinl/ZrbrWOwGAuIeJ93QCfcbx /d8rsOj+xWLJwWrPAc3ui+HjE7szEhHh7vxkanmOBDjUna3F6dDzP8IjZ8SoRXY5PrMX zvtzu9hevY8EvTR4C+OTKeT4oWDfry54b1L0/5GvUVlqgMNL5MYppB0neR2xjD+Ini5G YopA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of stable-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=stable-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id f34si4793752plf.648.2017.10.18.06.35.34; Wed, 18 Oct 2017 06:35:34 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of stable-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of stable-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=stable-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1754055AbdJRNfd (ORCPT + 9 others); Wed, 18 Oct 2017 09:35:33 -0400 Received: from usa-sjc-mx-foss1.foss.arm.com ([217.140.101.70]:40716 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1754030AbdJRNfd (ORCPT ); Wed, 18 Oct 2017 09:35:33 -0400 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id E2A19F; Wed, 18 Oct 2017 06:35:32 -0700 (PDT) Received: from lakrids.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 44F1E3F483; Wed, 18 Oct 2017 06:35:31 -0700 (PDT) From: Mark Rutland To: stable@vger.kernel.org Cc: Mark Rutland , Arnd Bergmann , Christoph Lameter , Peter Zijlstra , Pranith Kumar , Tejun Heo , Thomas Gleixner , linux-arch@vger.kernel.org Subject: [PATCH v3.16.y] percpu: make this_cpu_generic_read() atomic w.r.t. interrupts Date: Wed, 18 Oct 2017 14:35:24 +0100 Message-Id: <20171018133524.24911-1-mark.rutland@arm.com> X-Mailer: git-send-email 2.11.0 Sender: stable-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: stable@vger.kernel.org Commit e88d62cd4b2f0b1ae55e9008e79c2794b1fc914d upstream. As raw_cpu_generic_read() is a plain read from a raw_cpu_ptr() address, it's possible (albeit unlikely) that the compiler will split the access across multiple instructions. In this_cpu_generic_read() we disable preemption but not interrupts before calling raw_cpu_generic_read(). Thus, an interrupt could be taken in the middle of the split load instructions. If a this_cpu_write() or RMW this_cpu_*() op is made to the same variable in the interrupt handling path, this_cpu_read() will return a torn value. For native word types, we can avoid tearing using READ_ONCE(), but this won't work in all cases (e.g. 64-bit types on most 32-bit platforms). This patch reworks this_cpu_generic_read() to use READ_ONCE() where possible, otherwise falling back to disabling interrupts. Signed-off-by: Mark Rutland Cc: Arnd Bergmann Cc: Christoph Lameter Cc: Peter Zijlstra Cc: Pranith Kumar Cc: Tejun Heo Cc: Thomas Gleixner Cc: linux-arch@vger.kernel.org Cc: stable@vger.kernel.org Signed-off-by: Tejun Heo [Mark: backport to v3.16.y] Signed-off-by: Mark Rutland --- include/linux/percpu.h | 29 +++++++++++++++++++++++++---- 1 file changed, 25 insertions(+), 4 deletions(-) -- 2.11.0 diff --git a/include/linux/percpu.h b/include/linux/percpu.h index 8419053d0f2e..37ff713a7c7f 100644 --- a/include/linux/percpu.h +++ b/include/linux/percpu.h @@ -276,12 +276,33 @@ do { \ * used. */ -#define _this_cpu_generic_read(pcp) \ -({ typeof(pcp) ret__; \ +#define __this_cpu_generic_read_nopreempt(pcp) \ +({ \ + typeof(pcp) __ret; \ preempt_disable(); \ - ret__ = *this_cpu_ptr(&(pcp)); \ + __ret = ACCESS_ONCE(*raw_cpu_ptr(&(pcp))); \ preempt_enable(); \ - ret__; \ + __ret; \ +}) + +#define __this_cpu_generic_read_noirq(pcp) \ +({ \ + typeof(pcp) __ret; \ + unsigned long __flags; \ + raw_local_irq_save(__flags); \ + __ret = *raw_cpu_ptr(&(pcp)); \ + raw_local_irq_restore(__flags); \ + __ret; \ +}) + +#define _this_cpu_generic_read(pcp) \ +({ \ + typeof(pcp) __ret; \ + if (__native_word(pcp)) \ + __ret = __this_cpu_generic_read_nopreempt(pcp); \ + else \ + __ret = __this_cpu_generic_read_noirq(pcp); \ + __ret; \ }) #ifndef this_cpu_read