From patchwork Tue May 29 18:07:41 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Rutland X-Patchwork-Id: 137199 Delivered-To: patch@linaro.org Received: by 2002:a2e:9706:0:0:0:0:0 with SMTP id r6-v6csp4363529lji; Tue, 29 May 2018 11:08:11 -0700 (PDT) X-Google-Smtp-Source: AB8JxZrrZVM7RsPxAWDJ9HXQLiuzeZQZSDhOE+Rh7voFXX7b7KwWgpmKSfHLnAS8kheLrNcEUif7 X-Received: by 2002:a65:4ecc:: with SMTP id w12-v6mr14627949pgq.214.1527617290847; Tue, 29 May 2018 11:08:10 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1527617290; cv=none; d=google.com; s=arc-20160816; b=gZIfHC8qvcDfDF3PW4am4oArEw1AmRt6yRdcpld90qMzKKHm8+rng4As+nwDviG0te UPFUFGbf2p4d9TovReKrSzST9AqqeE6pCbltZR7uzqvHnbBco2TCaS8Iw/r4qyF0uAmT Sf0/6fD1Tirl3roFc2EMzW5S1+RHKpMGCFxsS8SK9Y6Kutqzm7yrjd36bYVCLZq/58u8 3HbOKbgjVYmCD/4U8AjVG9CZVjL2LfdXcHfdu+BMvNaUq033qIWd4tVXJYdyEuPDIQeY +ZJAGSCKbMPWfXeqGLOIr5eohd+Q+FGARo3s9tsW7EKeqc2AhVg/ExjfVru8H4Osj6yQ 3WIg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:arc-authentication-results; bh=dNfbCmpaa3SS8R6TAlYyLLJMb61ITWn/IGfs+RbWsE4=; b=jIUjtnL1W99Vgei3+uNCp3D2ugIH7LDhvp0AWS4QQFEOQ8+Xb0HrumZKfldKkavlDk GyiW3YDxQ5QUyKKH+80r5wRdvPJFRhmNo/M8xhFiaYMO3dn6iDXs6cz5gcqY5THMrc/p o1rc/5ON4fIToLjYa2E6t5nmmBpoE0T6mR7RIyaHjrhOAMHDr1PDuJJ8PSjZTG7FEz8K S9JIWmuMvkHepdzVMCg2Hovh1cfrinRQNlHjZZC1Z3Ke+cxZZlubkMJ0xu+1ZBi7jHq8 8npITK9gmStJegZiCXQ8inXA1Z27GN6OKDtMN/X3M1crL0yvP9lTKe2e+ZuxYbeiNNZ0 DAEA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id b19-v6si32700472pfh.358.2018.05.29.11.08.10; Tue, 29 May 2018 11:08:10 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S965843AbeE2SIJ (ORCPT + 30 others); Tue, 29 May 2018 14:08:09 -0400 Received: from foss.arm.com ([217.140.101.70]:45884 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S965778AbeE2SID (ORCPT ); Tue, 29 May 2018 14:08:03 -0400 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 0362715B2; Tue, 29 May 2018 11:08:03 -0700 (PDT) Received: from lakrids.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id C42723F557; Tue, 29 May 2018 11:08:01 -0700 (PDT) From: Mark Rutland To: linux-kernel@vger.kernel.org Cc: Mark Rutland , Boqun Feng , Peter Zijlstra , Will Deacon , Andrea Parri Subject: [PATCH 2/7] atomics/treewide: rework ordering barriers Date: Tue, 29 May 2018 19:07:41 +0100 Message-Id: <20180529180746.29684-3-mark.rutland@arm.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20180529180746.29684-1-mark.rutland@arm.com> References: <20180529180746.29684-1-mark.rutland@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Currently architectures can override __atomic_op_*() to define the barriers used before/after a relaxed atomic when used to build acquire/release/fence variants. This has the unfortunate property of requiring the architecture to define the full wrapper for the atomics, rather than just the barriers they care about, and gets in the way of generating atomics which can be easily read. Instead, this patch has architectures define an optional set of barriers, __atomic_mb_{before,after}_{acquire,release,fence}(), which uses to build the wrappers. There should be no functional change as a result of this patch. Signed-off-by: Mark Rutland Cc: Boqun Feng Cc: Peter Zijlstra Cc: Will Deacon Cc: Andrea Parri --- arch/alpha/include/asm/atomic.h | 8 ++++---- arch/powerpc/include/asm/atomic.h | 17 +++++------------ arch/riscv/include/asm/atomic.h | 17 +++++------------ include/linux/atomic.h | 38 ++++++++++++++++++++++---------------- 4 files changed, 36 insertions(+), 44 deletions(-) I'm aware that the naming of these barriers is potentially bad, and something like atomic_acquire_mb()/atomic_release_mb() might be better. I only really care that the core code can use the barriers directly rather than this being hidden in an architecture's __atomic_op_*(). Peter/Will, I'll defer to your judgement on naming. Thanks, Mark. -- 2.11.0 diff --git a/arch/alpha/include/asm/atomic.h b/arch/alpha/include/asm/atomic.h index 81bf9d15dcc1..aba549a0b147 100644 --- a/arch/alpha/include/asm/atomic.h +++ b/arch/alpha/include/asm/atomic.h @@ -18,11 +18,11 @@ * To ensure dependency ordering is preserved for the _relaxed and * _release atomics, an smp_read_barrier_depends() is unconditionally * inserted into the _relaxed variants, which are used to build the - * barriered versions. To avoid redundant back-to-back fences, we can - * define the _acquire and _fence versions explicitly. + * barriered versions. Avoid redundant back-to-back fences in the + * _acquire and _fence versions. */ -#define __atomic_op_acquire(op, args...) op##_relaxed(args) -#define __atomic_op_fence __atomic_op_release +#define __atomic_mb__after_acquire() +#define __atomic_mb__after_fence() #define ATOMIC_INIT(i) { (i) } #define ATOMIC64_INIT(i) { (i) } diff --git a/arch/powerpc/include/asm/atomic.h b/arch/powerpc/include/asm/atomic.h index a0156cb43d1f..39342a1602e9 100644 --- a/arch/powerpc/include/asm/atomic.h +++ b/arch/powerpc/include/asm/atomic.h @@ -18,18 +18,11 @@ * a "bne-" instruction at the end, so an isync is enough as a acquire barrier * on the platform without lwsync. */ -#define __atomic_op_acquire(op, args...) \ -({ \ - typeof(op##_relaxed(args)) __ret = op##_relaxed(args); \ - __asm__ __volatile__(PPC_ACQUIRE_BARRIER "" : : : "memory"); \ - __ret; \ -}) - -#define __atomic_op_release(op, args...) \ -({ \ - __asm__ __volatile__(PPC_RELEASE_BARRIER "" : : : "memory"); \ - op##_relaxed(args); \ -}) +#define __atomic_mb__after_acquire() \ + __asm__ __volatile__(PPC_ACQUIRE_BARRIER "" : : : "memory") + +#define __atomic_mb__before_release() \ + __asm__ __volatile__(PPC_RELEASE_BARRIER "" : : : "memory") static __inline__ int atomic_read(const atomic_t *v) { diff --git a/arch/riscv/include/asm/atomic.h b/arch/riscv/include/asm/atomic.h index 512b89485790..1d168857638a 100644 --- a/arch/riscv/include/asm/atomic.h +++ b/arch/riscv/include/asm/atomic.h @@ -25,18 +25,11 @@ #define ATOMIC_INIT(i) { (i) } -#define __atomic_op_acquire(op, args...) \ -({ \ - typeof(op##_relaxed(args)) __ret = op##_relaxed(args); \ - __asm__ __volatile__(RISCV_ACQUIRE_BARRIER "" ::: "memory"); \ - __ret; \ -}) - -#define __atomic_op_release(op, args...) \ -({ \ - __asm__ __volatile__(RISCV_RELEASE_BARRIER "" ::: "memory"); \ - op##_relaxed(args); \ -}) +#define __atomic_mb__after_acquire() \ + __asm__ __volatile__(RISCV_ACQUIRE_BARRIER "" ::: "memory") + +#define __atomic_mb__before_release() \ + __asm__ __volatile__(RISCV_RELEASE_BARRIER "" ::: "memory"); static __always_inline int atomic_read(const atomic_t *v) { diff --git a/include/linux/atomic.h b/include/linux/atomic.h index 9c1b0769a846..580d15b10bfd 100644 --- a/include/linux/atomic.h +++ b/include/linux/atomic.h @@ -36,40 +36,46 @@ * barriers on top of the relaxed variant. In the case where the relaxed * variant is already fully ordered, no additional barriers are needed. * - * Besides, if an arch has a special barrier for acquire/release, it could - * implement its own __atomic_op_* and use the same framework for building - * variants - * - * If an architecture overrides __atomic_op_acquire() it will probably want - * to define smp_mb__after_spinlock(). + * If an architecture overrides __atomic_mb_after_acquire() it will probably + * want to define smp_mb__after_spinlock(). */ -#ifndef __atomic_op_acquire +#ifndef __atomic_mb__after_acquire +#define __atomic_mb__after_acquire smp_mb__after_atomic +#endif + +#ifndef __atomic_mb__before_release +#define __atomic_mb__before_release smp_mb__before_atomic +#endif + +#ifndef __atomic_mb__before_fence +#define __atomic_mb__before_fence smp_mb__before_atomic +#endif + +#ifndef __atomic_mb__after_fence +#define __atomic_mb__after_fence smp_mb__after_atomic +#endif + #define __atomic_op_acquire(op, args...) \ ({ \ typeof(op##_relaxed(args)) __ret = op##_relaxed(args); \ - smp_mb__after_atomic(); \ + __atomic_mb__after_acquire(); \ __ret; \ }) -#endif -#ifndef __atomic_op_release #define __atomic_op_release(op, args...) \ ({ \ - smp_mb__before_atomic(); \ + __atomic_mb__before_release(); \ op##_relaxed(args); \ }) -#endif -#ifndef __atomic_op_fence #define __atomic_op_fence(op, args...) \ ({ \ typeof(op##_relaxed(args)) __ret; \ - smp_mb__before_atomic(); \ + __atomic_mb__before_fence(); \ __ret = op##_relaxed(args); \ - smp_mb__after_atomic(); \ + __atomic_mb__after_fence(); \ __ret; \ }) -#endif /* atomic_add_return_relaxed */ #ifndef atomic_add_return_relaxed