From patchwork Thu Jul 5 16:40:47 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Rutland X-Patchwork-Id: 141216 Delivered-To: patch@linaro.org Received: by 2002:a2e:9754:0:0:0:0:0 with SMTP id f20-v6csp2154806ljj; Thu, 5 Jul 2018 09:41:44 -0700 (PDT) X-Google-Smtp-Source: AAOMgpcMCHeKalmXX4Oc3VkHlJSVWRONFYNyz1Pujetbm9jLsKBtTRGRufDQGkZ8el0zXamYwvNn X-Received: by 2002:a62:bd03:: with SMTP id a3-v6mr7161955pff.138.1530808904761; Thu, 05 Jul 2018 09:41:44 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1530808904; cv=none; d=google.com; s=arc-20160816; b=QMfZFnZorJ1B9K2gA/BCppbBYKU/wUe+6FyVeJ1YDB7lQmNfOLzVuxvSFw3Q25qHvH 4YhzVCh/FU7hG9q8SIEchUBTVtjmKLcNw0SDu/CidOTY4K96DGd+V55ozYzyM6tk5ehp MVPQzUW1JlwyrJEh0OCxAyxvc6wGCGRMmtXWm5/T0uiH0q5B2vllFGR40V1RfZK0X7mY 3kNd0yY3IWt4rbiZlWw5/jyQi4SsTDWl+CmD7dCl0dQWBx4DGnqHQOU3u4lUo760ym8v dYyd0EhxFI0OjaBwwMqZWLUzjITqTGdL7xpd8p45eEI6gSth9vvTDbAxDs0bbehp9REw 3ffA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:arc-authentication-results; bh=0B51UNwI5mZVrcKbZSENpmEKnUv0ufgSzoMJU5P28vo=; b=j4k0zKjzCmnne2Jdj+mQzc/dJ3zk/S0WfyQrDHMGdT4NW1A6DidtRQiK8id6Jm7kPm mrvtuJTPkG/h18LQLRnFPAjbrXNtM27a1wrr0C5k/TSsJ9u9hg5iSBMsKdRr5xnYmDzC I864wuwFzvf1pNOSrTrt2XzLdth0Td0anAXcvn8DbTeIk/4Uk8ShSKsI9ciolK7nZVoC n0erMtUTSKe8r9uYlARQzjuu4vugDqRaFyunSK4Dr7CpCch/luoBntKgBcMJ5pnRO1kc DfiZxQDElSHv3JoS2pXpzBLkq5pu5n+IsuoPPqi7YZV666WRKdkI7Gtb+iGjGTnOnCCY U5jg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id g7-v6si5976231plp.236.2018.07.05.09.41.44; Thu, 05 Jul 2018 09:41:44 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753776AbeGEQlm (ORCPT + 31 others); Thu, 5 Jul 2018 12:41:42 -0400 Received: from usa-sjc-mx-foss1.foss.arm.com ([217.140.101.70]:53480 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753250AbeGEQlh (ORCPT ); Thu, 5 Jul 2018 12:41:37 -0400 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 650471595; Thu, 5 Jul 2018 09:41:37 -0700 (PDT) Received: from lakrids.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 3C6613F5BA; Thu, 5 Jul 2018 09:41:34 -0700 (PDT) From: Mark Rutland To: linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, peterz@infradead.org, will.deacon@arm.com Cc: andy.shevchenko@gmail.com, arnd@arndb.de, aryabinin@virtuozzo.com, boqun.feng@gmail.com, catalin.marinas@arm.com, dvyukov@google.com, glider@google.com, hpa@zytor.com, linuxdrivers@attotech.com, mark.rutland@arm.com, mingo@kernel.org, mingo@redhat.com, parri.andrea@gmail.com, peter@hurleysoftware.com, tglx@linutronix.de Subject: [PATCHv3 06/12] atomics/treewide: rework ordering barriers Date: Thu, 5 Jul 2018 17:40:47 +0100 Message-Id: <20180705164053.10073-7-mark.rutland@arm.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20180705164053.10073-1-mark.rutland@arm.com> References: <20180705164053.10073-1-mark.rutland@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Currently architectures can override __atomic_op_*() to define the barriers used before/after a relaxed atomic when used to build acquire/release/fence variants. This has the unfortunate property of requiring the architecture to define the full wrapper for the atomics, rather than just the barriers they care about, and gets in the way of generating atomics which can be easily read. Instead, this patch has architectures define an optional set of barriers: * __atomic_acquire_fence() * __atomic_release_fence() * __atomic_pre_full_fence() * __atomic_post_full_fence() ... which uses to build the wrappers. It would be nice if we could undef these, along with the __atomic_op_*() wrappers, but that would break the cmpxchg() wrappers, which are written in preprocessor. Undefs would have been nice, but alas. There should be no functional change as a result of this patch. Signed-off-by: Mark Rutland Acked-by: Peter Zijlstra (Intel) Cc: Andrea Parri Cc: Boqun Feng Cc: Ingo Molnar Cc: Will Deacon --- arch/alpha/include/asm/atomic.h | 8 ++++---- arch/powerpc/include/asm/atomic.h | 17 +++++------------ arch/riscv/include/asm/atomic.h | 17 +++++------------ include/linux/atomic.h | 38 ++++++++++++++++++++++---------------- 4 files changed, 36 insertions(+), 44 deletions(-) -- 2.11.0 diff --git a/arch/alpha/include/asm/atomic.h b/arch/alpha/include/asm/atomic.h index 4a6a8f58c9c9..150a1c5d6a2c 100644 --- a/arch/alpha/include/asm/atomic.h +++ b/arch/alpha/include/asm/atomic.h @@ -18,11 +18,11 @@ * To ensure dependency ordering is preserved for the _relaxed and * _release atomics, an smp_read_barrier_depends() is unconditionally * inserted into the _relaxed variants, which are used to build the - * barriered versions. To avoid redundant back-to-back fences, we can - * define the _acquire and _fence versions explicitly. + * barriered versions. Avoid redundant back-to-back fences in the + * _acquire and _fence versions. */ -#define __atomic_op_acquire(op, args...) op##_relaxed(args) -#define __atomic_op_fence __atomic_op_release +#define __atomic_acquire_fence() +#define __atomic_post_full_fence() #define ATOMIC_INIT(i) { (i) } #define ATOMIC64_INIT(i) { (i) } diff --git a/arch/powerpc/include/asm/atomic.h b/arch/powerpc/include/asm/atomic.h index a0156cb43d1f..963abf8bf1c0 100644 --- a/arch/powerpc/include/asm/atomic.h +++ b/arch/powerpc/include/asm/atomic.h @@ -18,18 +18,11 @@ * a "bne-" instruction at the end, so an isync is enough as a acquire barrier * on the platform without lwsync. */ -#define __atomic_op_acquire(op, args...) \ -({ \ - typeof(op##_relaxed(args)) __ret = op##_relaxed(args); \ - __asm__ __volatile__(PPC_ACQUIRE_BARRIER "" : : : "memory"); \ - __ret; \ -}) - -#define __atomic_op_release(op, args...) \ -({ \ - __asm__ __volatile__(PPC_RELEASE_BARRIER "" : : : "memory"); \ - op##_relaxed(args); \ -}) +#define __atomic_acquire_fence() \ + __asm__ __volatile__(PPC_ACQUIRE_BARRIER "" : : : "memory") + +#define __atomic_release_fence() \ + __asm__ __volatile__(PPC_RELEASE_BARRIER "" : : : "memory") static __inline__ int atomic_read(const atomic_t *v) { diff --git a/arch/riscv/include/asm/atomic.h b/arch/riscv/include/asm/atomic.h index 512b89485790..c452359c9cb8 100644 --- a/arch/riscv/include/asm/atomic.h +++ b/arch/riscv/include/asm/atomic.h @@ -25,18 +25,11 @@ #define ATOMIC_INIT(i) { (i) } -#define __atomic_op_acquire(op, args...) \ -({ \ - typeof(op##_relaxed(args)) __ret = op##_relaxed(args); \ - __asm__ __volatile__(RISCV_ACQUIRE_BARRIER "" ::: "memory"); \ - __ret; \ -}) - -#define __atomic_op_release(op, args...) \ -({ \ - __asm__ __volatile__(RISCV_RELEASE_BARRIER "" ::: "memory"); \ - op##_relaxed(args); \ -}) +#define __atomic_acquire_fence() \ + __asm__ __volatile__(RISCV_ACQUIRE_BARRIER "" ::: "memory") + +#define __atomic_release_fence() \ + __asm__ __volatile__(RISCV_RELEASE_BARRIER "" ::: "memory"); static __always_inline int atomic_read(const atomic_t *v) { diff --git a/include/linux/atomic.h b/include/linux/atomic.h index 8e04f1f69bd9..1e8e88bdaf09 100644 --- a/include/linux/atomic.h +++ b/include/linux/atomic.h @@ -38,40 +38,46 @@ * barriers on top of the relaxed variant. In the case where the relaxed * variant is already fully ordered, no additional barriers are needed. * - * Besides, if an arch has a special barrier for acquire/release, it could - * implement its own __atomic_op_* and use the same framework for building - * variants - * - * If an architecture overrides __atomic_op_acquire() it will probably want - * to define smp_mb__after_spinlock(). + * If an architecture overrides __atomic_acquire_fence() it will probably + * want to define smp_mb__after_spinlock(). */ -#ifndef __atomic_op_acquire +#ifndef __atomic_acquire_fence +#define __atomic_acquire_fence smp_mb__after_atomic +#endif + +#ifndef __atomic_release_fence +#define __atomic_release_fence smp_mb__before_atomic +#endif + +#ifndef __atomic_pre_full_fence +#define __atomic_pre_full_fence smp_mb__before_atomic +#endif + +#ifndef __atomic_post_full_fence +#define __atomic_post_full_fence smp_mb__after_atomic +#endif + #define __atomic_op_acquire(op, args...) \ ({ \ typeof(op##_relaxed(args)) __ret = op##_relaxed(args); \ - smp_mb__after_atomic(); \ + __atomic_acquire_fence(); \ __ret; \ }) -#endif -#ifndef __atomic_op_release #define __atomic_op_release(op, args...) \ ({ \ - smp_mb__before_atomic(); \ + __atomic_release_fence(); \ op##_relaxed(args); \ }) -#endif -#ifndef __atomic_op_fence #define __atomic_op_fence(op, args...) \ ({ \ typeof(op##_relaxed(args)) __ret; \ - smp_mb__before_atomic(); \ + __atomic_pre_full_fence(); \ __ret = op##_relaxed(args); \ - smp_mb__after_atomic(); \ + __atomic_post_full_fence(); \ __ret; \ }) -#endif /* atomic_add_return_relaxed */ #ifndef atomic_add_return_relaxed