From patchwork Wed Oct 4 19:00:07 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Github ODP bot X-Patchwork-Id: 114816 Delivered-To: patch@linaro.org Received: by 10.80.163.150 with SMTP id s22csp6267168edb; Wed, 4 Oct 2017 12:01:56 -0700 (PDT) X-Google-Smtp-Source: AOwi7QCsQc7BmTjJes5GTiaft81Of0eFR8/XUeHFoJJla3Nu6VPNPOvNERpM0lcXOsKQ5j3Xfxfb X-Received: by 10.55.94.196 with SMTP id s187mr25364921qkb.88.1507143716144; Wed, 04 Oct 2017 12:01:56 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1507143716; cv=none; d=google.com; s=arc-20160816; b=x4BXXCBH2AkWgOXQfjQ9En1147c6HSZaTAsirtsESLSZwKEr8Bk5Hx+temsaOA0SkL tz0+63neg70RFZbqmwGtsGa/6wAZOdZo+bbpDmPBJ5KYwHdzxMN7utsdqsoIj/EMUeer +YeRSqimJvtUnzwpc96ippaX1Jz+vC0Yr7DM+G5SWSLAIKSqtusEeGBah4qwEMwkyc0Y AJzDFEyug45svCicLxXgBLun3Cm6dnGFrMCTT0Orqyvw75w5Ec6jpNcZ8lyNzu8paujI ZmUM7gYGd4u5rOXy7oWBJ5IkxMAJ+YqU7h2GYGAcd4Av6uF5AgozSOzZepWqIWfZgaS8 xNcA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:subject:github-pr-num :references:in-reply-to:message-id:date:to:from:delivered-to :arc-authentication-results; bh=GbuOB1JAYipqYv9SKDLRgzjuNCkCyAuXUvU8iNIuLcY=; b=TkqfxVzwnPIWoykprZuq7PEJIvu92BP5JOAEUiZ665Cqq0i6uQlcMpAGoHPXOGdFQc wr42Hy6DWCtGbk52dwZkPUYIah4gK9fcyGiydo+iDWN3n+ltZNxKvwFPIRzuq7GjSaFC qe6bjGbe/u51m7x2vSfZVy5fFXbpE52ElbhLGfAynVEfCT2d0H1jF2E95hHdBK5NADRy nb0B443DccqKgFZ2DcCELZpROJcztd9SGhUGXGxIo5m/eg1JgAbrD9pUMHXl10MTJBGv cpCl2rWpgKKRpWzO6qpD4aGse8wUVAjNSW/HMbM22KKCF8qR8W/nwKJucgBgYxhheLhZ JWaA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of lng-odp-bounces@lists.linaro.org designates 54.225.227.206 as permitted sender) smtp.mailfrom=lng-odp-bounces@lists.linaro.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=yandex.ru Return-Path: Received: from lists.linaro.org (lists.linaro.org. [54.225.227.206]) by mx.google.com with ESMTP id c62si5200032qkf.208.2017.10.04.12.01.55; Wed, 04 Oct 2017 12:01:56 -0700 (PDT) Received-SPF: pass (google.com: domain of lng-odp-bounces@lists.linaro.org designates 54.225.227.206 as permitted sender) client-ip=54.225.227.206; Authentication-Results: mx.google.com; spf=pass (google.com: domain of lng-odp-bounces@lists.linaro.org designates 54.225.227.206 as permitted sender) smtp.mailfrom=lng-odp-bounces@lists.linaro.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=yandex.ru Received: by lists.linaro.org (Postfix, from userid 109) id D1F5B60996; Wed, 4 Oct 2017 19:01:55 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on ip-10-142-244-252 X-Spam-Level: X-Spam-Status: No, score=-5.4 required=5.0 tests=BAYES_00,FREEMAIL_FROM, RCVD_IN_DNSWL_LOW, RCVD_IN_MSPIKE_H2 autolearn=disabled version=3.4.0 Received: from [127.0.0.1] (localhost [127.0.0.1]) by lists.linaro.org (Postfix) with ESMTP id 2607761BC6; Wed, 4 Oct 2017 19:00:35 +0000 (UTC) X-Original-To: lng-odp@lists.linaro.org Delivered-To: lng-odp@lists.linaro.org Received: by lists.linaro.org (Postfix, from userid 109) id CBE5C60CE4; Wed, 4 Oct 2017 19:00:19 +0000 (UTC) Received: from forward101p.mail.yandex.net (forward101p.mail.yandex.net [77.88.28.101]) by lists.linaro.org (Postfix) with ESMTPS id E21A860CE4 for ; Wed, 4 Oct 2017 19:00:12 +0000 (UTC) Received: from mxback2j.mail.yandex.net (mxback2j.mail.yandex.net [IPv6:2a02:6b8:0:1619::10b]) by forward101p.mail.yandex.net (Yandex) with ESMTP id 769026A84EE8 for ; Wed, 4 Oct 2017 22:00:11 +0300 (MSK) Received: from smtp2o.mail.yandex.net (smtp2o.mail.yandex.net [2a02:6b8:0:1a2d::26]) by mxback2j.mail.yandex.net (nwsmtp/Yandex) with ESMTP id XsHAa70N5R-0BomwsWx; Wed, 04 Oct 2017 22:00:11 +0300 Received: by smtp2o.mail.yandex.net (nwsmtp/Yandex) with ESMTPSA id Ftu7Q4zrbA-0A2Of5rP; Wed, 04 Oct 2017 22:00:10 +0300 (using TLSv1.2 with cipher ECDHE-RSA-AES128-SHA256 (128/128 bits)) (Client certificate not present) From: Github ODP bot To: lng-odp@lists.linaro.org Date: Wed, 4 Oct 2017 22:00:07 +0300 Message-Id: <1507143608-13625-2-git-send-email-odpbot@yandex.ru> X-Mailer: git-send-email 1.9.1 In-Reply-To: <1507143608-13625-1-git-send-email-odpbot@yandex.ru> References: <1507143608-13625-1-git-send-email-odpbot@yandex.ru> Github-pr-num: 211 Subject: [lng-odp] [PATCH API-NEXT v2 1/2] linux-gen: update api-next to follow merged changes X-BeenThere: lng-odp@lists.linaro.org X-Mailman-Version: 2.1.16 Precedence: list List-Id: "The OpenDataPlane \(ODP\) List" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: lng-odp-bounces@lists.linaro.org Sender: "lng-odp" From: Dmitry Eremin-Solenikov There were changes in master branch on the way architecture files are treated. Update api-next after merging in master to follow those changes. Signed-off-by: Dmitry Eremin-Solenikov --- /** Email created from pull request 211 (lumag:fix-api-next-merge) ** https://github.com/Linaro/odp/pull/211 ** Patch: https://github.com/Linaro/odp/pull/211.patch ** Base sha: 75cfc6f70d5b21d6f04082e1d03a7e677a895280 ** Merge commit sha: b5aebf54bd0f3e939b778b754975259df1c44de3 **/ platform/linux-generic/Makefile.am | 28 ++- platform/linux-generic/arch/aarch64/odp_atomic.h | 208 +++++++++++++++++++++ platform/linux-generic/arch/aarch64/odp_cpu.h | 62 ++++++ .../linux-generic/arch/aarch64/odp_cpu_idling.h | 51 +++++ platform/linux-generic/arch/aarch64/odp_llsc.h | 165 ++++++++++++++++ platform/linux-generic/arch/arm/odp_atomic.h | 181 ------------------ platform/linux-generic/arch/arm/odp_cpu.h | 22 +-- platform/linux-generic/arch/arm/odp_llsc.h | 157 ---------------- platform/linux-generic/arch/mips64/odp_cpu.h | 43 ----- platform/linux-generic/arch/powerpc/odp_cpu.h | 43 ----- platform/linux-generic/arch/x86/odp_cpu.h | 43 ----- 11 files changed, 500 insertions(+), 503 deletions(-) create mode 100644 platform/linux-generic/arch/aarch64/odp_atomic.h create mode 100644 platform/linux-generic/arch/aarch64/odp_cpu.h create mode 100644 platform/linux-generic/arch/aarch64/odp_cpu_idling.h create mode 100644 platform/linux-generic/arch/aarch64/odp_llsc.h delete mode 100644 platform/linux-generic/arch/mips64/odp_cpu.h delete mode 100644 platform/linux-generic/arch/powerpc/odp_cpu.h delete mode 100644 platform/linux-generic/arch/x86/odp_cpu.h diff --git a/platform/linux-generic/Makefile.am b/platform/linux-generic/Makefile.am index 8f6c65703..f830d69f2 100644 --- a/platform/linux-generic/Makefile.am +++ b/platform/linux-generic/Makefile.am @@ -9,6 +9,7 @@ AM_CPPFLAGS += -I$(top_srcdir)/include/odp/arch/@ARCH_ABI@ AM_CPPFLAGS += -I$(top_builddir)/include AM_CPPFLAGS += -Iinclude AM_CPPFLAGS += -I$(top_srcdir)/platform/$(with_platform)/arch/$(ARCH_DIR) +AM_CPPFLAGS += -I$(top_srcdir)/platform/$(with_platform)/arch/default AM_CPPFLAGS += -Iinclude AM_CPPFLAGS += -DSYSCONFDIR=\"@sysconfdir@\" @@ -195,22 +196,6 @@ noinst_HEADERS = \ ${srcdir}/include/protocols/thash.h \ ${srcdir}/include/protocols/udp.h -if ARCH_IS_ARM -noinst_HEADERS += ${srcdir}/arch/arm/odp_atomic.h \ - ${srcdir}/arch/arm/odp_cpu.h \ - ${srcdir}/arch/arm/odp_cpu_idling.h \ - ${srcdir}/arch/arm/odp_llsc.h -endif -if ARCH_IS_MIPS64 -noinst_HEADERS += ${srcdir}/arch/mips64/odp_cpu.h -endif -if ARCH_IS_POWERPC -noinst_HEADERS += ${srcdir}/arch/powerpc/odp_cpu.h -endif -if ARCH_IS_X86 -noinst_HEADERS += ${srcdir}/arch/x86/odp_cpu.h -endif - __LIB__libodp_linux_la_SOURCES = \ _fdserver.c \ _ishm.c \ @@ -290,6 +275,10 @@ __LIB__libodp_linux_la_SOURCES += arch/default/odp_cpu_arch.c \ arch/default/odp_global_time.c \ arch/default/odp_sysinfo_parse.c arch_odp_headers = $(srcdir)/arch/arm/odp/api/cpu_arch.h +noinst_HEADERS += ${srcdir}/arch/arm/odp_atomic.h \ + ${srcdir}/arch/arm/odp_cpu.h \ + ${srcdir}/arch/arm/odp_cpu_idling.h \ + ${srcdir}/arch/arm/odp_llsc.h endif if ARCH_IS_AARCH64 __LIB__libodp_linux_la_SOURCES += arch/default/odp_cpu_arch.c \ @@ -297,6 +286,10 @@ __LIB__libodp_linux_la_SOURCES += arch/default/odp_cpu_arch.c \ arch/aarch64/odp_global_time.c \ arch/default/odp_sysinfo_parse.c arch_odp_headers = $(srcdir)/arch/aarch64/odp/api/cpu_arch.h +noinst_HEADERS += ${srcdir}/arch/aarch64/odp_atomic.h \ + ${srcdir}/arch/aarch64/odp_cpu.h \ + ${srcdir}/arch/aarch64/odp_cpu_idling.h \ + ${srcdir}/arch/aarch64/odp_llsc.h endif if ARCH_IS_MIPS64 __LIB__libodp_linux_la_SOURCES += arch/mips64/odp_cpu_arch.c \ @@ -304,6 +297,7 @@ __LIB__libodp_linux_la_SOURCES += arch/mips64/odp_cpu_arch.c \ arch/default/odp_global_time.c \ arch/mips64/odp_sysinfo_parse.c arch_odp_headers = $(srcdir)/arch/mips64/odp/api/cpu_arch.h +noinst_HEADERS += ${srcdir}/arch/default/odp_cpu.h endif if ARCH_IS_POWERPC __LIB__libodp_linux_la_SOURCES += arch/default/odp_cpu_arch.c \ @@ -311,6 +305,7 @@ __LIB__libodp_linux_la_SOURCES += arch/default/odp_cpu_arch.c \ arch/default/odp_global_time.c \ arch/powerpc/odp_sysinfo_parse.c arch_odp_headers = $(srcdir)/arch/powerpc/odp/api/cpu_arch.h +noinst_HEADERS += ${srcdir}/arch/default/odp_cpu.h endif if ARCH_IS_X86 __LIB__libodp_linux_la_SOURCES += arch/x86/cpu_flags.c \ @@ -320,6 +315,7 @@ __LIB__libodp_linux_la_SOURCES += arch/x86/cpu_flags.c \ arch/x86/odp_sysinfo_parse.c arch_odp_headers = $(srcdir)/arch/x86/odp/api/cpu_arch.h noinst_HEADERS += $(srcdir)/arch/x86/cpu_flags.h +noinst_HEADERS += ${srcdir}/arch/default/odp_cpu.h endif noinst_HEADERS += $(srcdir)/arch/default/odp/api/cpu_arch.h diff --git a/platform/linux-generic/arch/aarch64/odp_atomic.h b/platform/linux-generic/arch/aarch64/odp_atomic.h new file mode 100644 index 000000000..23bf96c7e --- /dev/null +++ b/platform/linux-generic/arch/aarch64/odp_atomic.h @@ -0,0 +1,208 @@ +/* Copyright (c) 2017, ARM Limited. All rights reserved. + * + * Copyright (c) 2017, Linaro Limited + * All rights reserved. + * + * SPDX-License-Identifier: BSD-3-Clause + */ + +#ifndef PLATFORM_LINUXGENERIC_ARCH_ARM_ODP_ATOMIC_H +#define PLATFORM_LINUXGENERIC_ARCH_ARM_ODP_ATOMIC_H + +#ifndef PLATFORM_LINUXGENERIC_ARCH_ARM_ODP_CPU_H +#error This file should not be included directly, please include odp_cpu.h +#endif + +#ifdef CONFIG_DMBSTR + +#define atomic_store_release(loc, val, ro) \ +do { \ + _odp_release_barrier(ro); \ + __atomic_store_n(loc, val, __ATOMIC_RELAXED); \ +} while (0) + +#else + +#define atomic_store_release(loc, val, ro) \ + __atomic_store_n(loc, val, __ATOMIC_RELEASE) + +#endif /* CONFIG_DMBSTR */ + +#define HAS_ACQ(mo) ((mo) != __ATOMIC_RELAXED && (mo) != __ATOMIC_RELEASE) +#define HAS_RLS(mo) ((mo) == __ATOMIC_RELEASE || (mo) == __ATOMIC_ACQ_REL || \ + (mo) == __ATOMIC_SEQ_CST) + +#define LL_MO(mo) (HAS_ACQ((mo)) ? __ATOMIC_ACQUIRE : __ATOMIC_RELAXED) +#define SC_MO(mo) (HAS_RLS((mo)) ? __ATOMIC_RELEASE : __ATOMIC_RELAXED) + +#ifndef __ARM_FEATURE_QRDMX /* Feature only available in v8.1a and beyond */ +static inline bool +__lockfree_compare_exchange_16(register __int128 *var, __int128 *exp, + register __int128 neu, bool weak, int mo_success, + int mo_failure) +{ + (void)weak; /* Always do strong CAS or we can't perform atomic read */ + /* Ignore memory ordering for failure, memory order for + * success must be stronger or equal. */ + (void)mo_failure; + register __int128 old; + register __int128 expected; + int ll_mo = LL_MO(mo_success); + int sc_mo = SC_MO(mo_success); + + expected = *exp; + __asm__ volatile("" ::: "memory"); + do { + /* Atomicity of LLD is not guaranteed */ + old = lld(var, ll_mo); + /* Must write back neu or old to verify atomicity of LLD */ + } while (odp_unlikely(scd(var, old == expected ? neu : old, sc_mo))); + *exp = old; /* Always update, atomically read value */ + return old == expected; +} + +static inline __int128 __lockfree_exchange_16(__int128 *var, __int128 neu, + int mo) +{ + register __int128 old; + int ll_mo = LL_MO(mo); + int sc_mo = SC_MO(mo); + + do { + /* Atomicity of LLD is not guaranteed */ + old = lld(var, ll_mo); + /* Must successfully write back to verify atomicity of LLD */ + } while (odp_unlikely(scd(var, neu, sc_mo))); + return old; +} + +static inline __int128 __lockfree_fetch_and_16(__int128 *var, __int128 mask, + int mo) +{ + register __int128 old; + int ll_mo = LL_MO(mo); + int sc_mo = SC_MO(mo); + + do { + /* Atomicity of LLD is not guaranteed */ + old = lld(var, ll_mo); + /* Must successfully write back to verify atomicity of LLD */ + } while (odp_unlikely(scd(var, old & mask, sc_mo))); + return old; +} + +static inline __int128 __lockfree_fetch_or_16(__int128 *var, __int128 mask, + int mo) +{ + register __int128 old; + int ll_mo = LL_MO(mo); + int sc_mo = SC_MO(mo); + + do { + /* Atomicity of LLD is not guaranteed */ + old = lld(var, ll_mo); + /* Must successfully write back to verify atomicity of LLD */ + } while (odp_unlikely(scd(var, old | mask, sc_mo))); + return old; +} + +#else + +static inline __int128 casp(__int128 *var, __int128 old, __int128 neu, int mo) +{ + if (mo == __ATOMIC_RELAXED) { + __asm__ volatile("casp %0, %H0, %1, %H1, [%2]" + : "+r" (old) + : "r" (neu), "r" (var) + : "memory"); + } else if (mo == __ATOMIC_ACQUIRE) { + __asm__ volatile("caspa %0, %H0, %1, %H1, [%2]" + : "+r" (old) + : "r" (neu), "r" (var) + : "memory"); + } else if (mo == __ATOMIC_ACQ_REL) { + __asm__ volatile("caspal %0, %H0, %1, %H1, [%2]" + : "+r" (old) + : "r" (neu), "r" (var) + : "memory"); + } else if (mo == __ATOMIC_RELEASE) { + __asm__ volatile("caspl %0, %H0, %1, %H1, [%2]" + : "+r" (old) + : "r" (neu), "r" (var) + : "memory"); + } else { + abort(); + } + return old; +} + +static inline bool +__lockfree_compare_exchange_16(register __int128 *var, __int128 *exp, + register __int128 neu, bool weak, int mo_success, + int mo_failure) +{ + (void)weak; + (void)mo_failure; + __int128 old; + __int128 expected; + + expected = *exp; + old = casp(var, expected, neu, mo_success); + *exp = old; /* Always update, atomically read value */ + return old == expected; +} + +static inline __int128 __lockfree_exchange_16(__int128 *var, __int128 neu, + int mo) +{ + __int128 old; + __int128 expected; + + do { + expected = *var; + old = casp(var, expected, neu, mo); + } while (old != expected); + return old; +} + +static inline __int128 __lockfree_fetch_and_16(__int128 *var, __int128 mask, + int mo) +{ + __int128 old; + __int128 expected; + + do { + expected = *var; + old = casp(var, expected, expected & mask, mo); + } while (old != expected); + return old; +} + +static inline __int128 __lockfree_fetch_or_16(__int128 *var, __int128 mask, + int mo) +{ + __int128 old; + __int128 expected; + + do { + expected = *var; + old = casp(var, expected, expected | mask, mo); + } while (old != expected); + return old; +} + +#endif /* __ARM_FEATURE_QRDMX */ + +static inline __int128 __lockfree_load_16(__int128 *var, int mo) +{ + __int128 old = *var; /* Possibly torn read */ + + /* Do CAS to ensure atomicity + * Either CAS succeeds (writing back the same value) + * Or CAS fails and returns the old value (atomic read) + */ + (void)__lockfree_compare_exchange_16(var, &old, old, false, mo, mo); + return old; +} + +#endif /* PLATFORM_LINUXGENERIC_ARCH_ARM_ODP_ATOMIC_H */ diff --git a/platform/linux-generic/arch/aarch64/odp_cpu.h b/platform/linux-generic/arch/aarch64/odp_cpu.h new file mode 100644 index 000000000..fc35a4625 --- /dev/null +++ b/platform/linux-generic/arch/aarch64/odp_cpu.h @@ -0,0 +1,62 @@ +/* Copyright (c) 2017, ARM Limited. All rights reserved. + * + * Copyright (c) 2017, Linaro Limited + * All rights reserved. + * + * SPDX-License-Identifier: BSD-3-Clause + */ + +#ifndef PLATFORM_LINUXGENERIC_ARCH_ARM_ODP_CPU_H +#define PLATFORM_LINUXGENERIC_ARCH_ARM_ODP_CPU_H + +#if !defined(__aarch64__) +#error Use this file only when compiling for ARMv8 architecture +#endif + +#include + +/* + * Use LLD/SCD atomic primitives instead of lock-based code path in llqueue + * LLD/SCD is on ARM the fastest way to enqueue and dequeue elements from a + * linked list queue. + */ +#define CONFIG_LLDSCD + +/* + * Use DMB;STR instead of STRL on ARM + * On early ARMv8 implementations (e.g. Cortex-A57) this is noticeably more + * performant than using store-release. + * This also allows for load-only barriers (DMB ISHLD) which are much cheaper + * than a full barrier + */ +#define CONFIG_DMBSTR + +/* + * Use ARM event signalling mechanism + * Event signalling minimises spinning (busy waiting) which decreases + * cache coherency traffic when spinning on shared locations (thus faster and + * more scalable) and enables the CPU to enter a sleep state (lower power + * consumption). + */ +#define CONFIG_WFE + +static inline void dmb(void) +{ + __asm__ volatile("dmb" : : : "memory"); +} + +/* Only ARMv8 supports DMB ISHLD */ +/* A load only barrier is much cheaper than full barrier */ +#define _odp_release_barrier(ro) \ +do { \ + if (ro) \ + __asm__ volatile("dmb ishld" ::: "memory"); \ + else \ + __asm__ volatile("dmb ish" ::: "memory"); \ +} while (0) + +#include "odp_llsc.h" +#include "odp_atomic.h" +#include "odp_cpu_idling.h" + +#endif /* PLATFORM_LINUXGENERIC_ARCH_ARM_ODP_CPU_H */ diff --git a/platform/linux-generic/arch/aarch64/odp_cpu_idling.h b/platform/linux-generic/arch/aarch64/odp_cpu_idling.h new file mode 100644 index 000000000..ab29455bb --- /dev/null +++ b/platform/linux-generic/arch/aarch64/odp_cpu_idling.h @@ -0,0 +1,51 @@ +/* Copyright (c) 2017, ARM Limited. All rights reserved. + * + * Copyright (c) 2017, Linaro Limited + * All rights reserved. + * + * SPDX-License-Identifier: BSD-3-Clause + */ + +#ifndef PLATFORM_LINUXGENERIC_ARCH_ARM_CPU_IDLING_H +#define PLATFORM_LINUXGENERIC_ARCH_ARM_CPU_IDLING_H + +#ifndef PLATFORM_LINUXGENERIC_ARCH_ARM_ODP_CPU_H +#error This file should not be included directly, please include odp_cpu.h +#endif + +static inline void sevl(void) +{ +#ifdef CONFIG_WFE + __asm__ volatile("sevl" : : : ); +#endif +} + +static inline int wfe(void) +{ +#ifdef CONFIG_WFE + __asm__ volatile("wfe" : : : "memory"); +#endif + return 1; +} + +static inline void doze(void) +{ +#ifndef CONFIG_WFE + /* When using WFE do not stall the pipeline using other means */ + odp_cpu_pause(); +#endif +} + +#ifdef CONFIG_WFE +#define monitor128(addr, mo) lld((addr), (mo)) +#define monitor64(addr, mo) ll64((addr), (mo)) +#define monitor32(addr, mo) ll32((addr), (mo)) +#define monitor8(addr, mo) ll8((addr), (mo)) +#else +#define monitor128(addr, mo) __atomic_load_n((addr), (mo)) +#define monitor64(addr, mo) __atomic_load_n((addr), (mo)) +#define monitor32(addr, mo) __atomic_load_n((addr), (mo)) +#define monitor8(addr, mo) __atomic_load_n((addr), (mo)) +#endif + +#endif /* PLATFORM_LINUXGENERIC_ARCH_ARM_CPU_IDLING_H */ diff --git a/platform/linux-generic/arch/aarch64/odp_llsc.h b/platform/linux-generic/arch/aarch64/odp_llsc.h new file mode 100644 index 000000000..f56792930 --- /dev/null +++ b/platform/linux-generic/arch/aarch64/odp_llsc.h @@ -0,0 +1,165 @@ +/* Copyright (c) 2017, ARM Limited. All rights reserved. + * + * Copyright (c) 2017, Linaro Limited + * All rights reserved. + * + * SPDX-License-Identifier: BSD-3-Clause + */ + +#ifndef PLATFORM_LINUXGENERIC_ARCH_ARM_LLSC_H +#define PLATFORM_LINUXGENERIC_ARCH_ARM_LLSC_H + +#ifndef PLATFORM_LINUXGENERIC_ARCH_ARM_ODP_CPU_H +#error This file should not be included directly, please include odp_cpu.h +#endif + +static inline uint16_t ll8(uint8_t *var, int mm) +{ + uint16_t old; + + if (mm == __ATOMIC_ACQUIRE) + __asm__ volatile("ldaxrb %w0, [%1]" + : "=&r" (old) + : "r" (var) + : "memory"); + else if (mm == __ATOMIC_RELAXED) + __asm__ volatile("ldxrb %w0, [%1]" + : "=&r" (old) + : "r" (var) + : ); + else + ODP_ABORT(); + return old; +} + +static inline uint32_t ll32(uint32_t *var, int mm) +{ + uint32_t old; + + if (mm == __ATOMIC_ACQUIRE) + __asm__ volatile("ldaxr %w0, [%1]" + : "=&r" (old) + : "r" (var) + : "memory"); + else if (mm == __ATOMIC_RELAXED) + __asm__ volatile("ldxr %w0, [%1]" + : "=&r" (old) + : "r" (var) + : ); + else + ODP_ABORT(); + return old; +} + +/* Return 0 on success, 1 on failure */ +static inline uint32_t sc32(uint32_t *var, uint32_t neu, int mm) +{ + uint32_t ret; + + if (mm == __ATOMIC_RELEASE) + __asm__ volatile("stlxr %w0, %w1, [%2]" + : "=&r" (ret) + : "r" (neu), "r" (var) + : "memory"); + else if (mm == __ATOMIC_RELAXED) + __asm__ volatile("stxr %w0, %w1, [%2]" + : "=&r" (ret) + : "r" (neu), "r" (var) + : ); + else + ODP_ABORT(); + return ret; +} + +static inline uint64_t ll(uint64_t *var, int mm) +{ + uint64_t old; + + if (mm == __ATOMIC_ACQUIRE) + __asm__ volatile("ldaxr %0, [%1]" + : "=&r" (old) + : "r" (var) + : "memory"); + else if (mm == __ATOMIC_RELAXED) + __asm__ volatile("ldxr %0, [%1]" + : "=&r" (old) + : "r" (var) + : ); + else + ODP_ABORT(); + return old; +} + +#define ll64(a, b) ll((a), (b)) + +/* Return 0 on success, 1 on failure */ +static inline uint32_t sc(uint64_t *var, uint64_t neu, int mm) +{ + uint32_t ret; + + if (mm == __ATOMIC_RELEASE) + __asm__ volatile("stlxr %w0, %1, [%2]" + : "=&r" (ret) + : "r" (neu), "r" (var) + : "memory"); + else if (mm == __ATOMIC_RELAXED) + __asm__ volatile("stxr %w0, %1, [%2]" + : "=&r" (ret) + : "r" (neu), "r" (var) + : ); + else + ODP_ABORT(); + return ret; +} + +#define sc64(a, b, c) sc((a), (b), (c)) + +union i128 { + __int128 i128; + int64_t i64[2]; +}; + +static inline __int128 lld(__int128 *var, int mm) +{ + union i128 old; + + if (mm == __ATOMIC_ACQUIRE) + __asm__ volatile("ldaxp %0, %1, [%2]" + : "=&r" (old.i64[0]), "=&r" (old.i64[1]) + : "r" (var) + : "memory"); + else if (mm == __ATOMIC_RELAXED) + __asm__ volatile("ldxp %0, %1, [%2]" + : "=&r" (old.i64[0]), "=&r" (old.i64[1]) + : "r" (var) + : ); + else + ODP_ABORT(); + return old.i128; +} + +/* Return 0 on success, 1 on failure */ +static inline uint32_t scd(__int128 *var, __int128 neu, int mm) +{ + uint32_t ret; + + if (mm == __ATOMIC_RELEASE) + __asm__ volatile("stlxp %w0, %1, %2, [%3]" + : "=&r" (ret) + : "r" (((union i128)neu).i64[0]), + "r" (((union i128)neu).i64[1]), + "r" (var) + : "memory"); + else if (mm == __ATOMIC_RELAXED) + __asm__ volatile("stxp %w0, %1, %2, [%3]" + : "=&r" (ret) + : "r" (((union i128)neu).i64[0]), + "r" (((union i128)neu).i64[1]), + "r" (var) + : ); + else + ODP_ABORT(); + return ret; +} + +#endif /* PLATFORM_LINUXGENERIC_ARCH_ARM_LLSC_H */ diff --git a/platform/linux-generic/arch/arm/odp_atomic.h b/platform/linux-generic/arch/arm/odp_atomic.h index 3a21a47b7..590bdf4a6 100644 --- a/platform/linux-generic/arch/arm/odp_atomic.h +++ b/platform/linux-generic/arch/arm/odp_atomic.h @@ -28,185 +28,4 @@ do { \ #endif /* CONFIG_DMBSTR */ -#ifdef __aarch64__ - -#define HAS_ACQ(mo) ((mo) != __ATOMIC_RELAXED && (mo) != __ATOMIC_RELEASE) -#define HAS_RLS(mo) ((mo) == __ATOMIC_RELEASE || (mo) == __ATOMIC_ACQ_REL || \ - (mo) == __ATOMIC_SEQ_CST) - -#define LL_MO(mo) (HAS_ACQ((mo)) ? __ATOMIC_ACQUIRE : __ATOMIC_RELAXED) -#define SC_MO(mo) (HAS_RLS((mo)) ? __ATOMIC_RELEASE : __ATOMIC_RELAXED) - -#ifndef __ARM_FEATURE_QRDMX /* Feature only available in v8.1a and beyond */ -static inline bool -__lockfree_compare_exchange_16(register __int128 *var, __int128 *exp, - register __int128 neu, bool weak, int mo_success, - int mo_failure) -{ - (void)weak; /* Always do strong CAS or we can't perform atomic read */ - /* Ignore memory ordering for failure, memory order for - * success must be stronger or equal. */ - (void)mo_failure; - register __int128 old; - register __int128 expected; - int ll_mo = LL_MO(mo_success); - int sc_mo = SC_MO(mo_success); - - expected = *exp; - __asm__ volatile("" ::: "memory"); - do { - /* Atomicity of LLD is not guaranteed */ - old = lld(var, ll_mo); - /* Must write back neu or old to verify atomicity of LLD */ - } while (odp_unlikely(scd(var, old == expected ? neu : old, sc_mo))); - *exp = old; /* Always update, atomically read value */ - return old == expected; -} - -static inline __int128 __lockfree_exchange_16(__int128 *var, __int128 neu, - int mo) -{ - register __int128 old; - int ll_mo = LL_MO(mo); - int sc_mo = SC_MO(mo); - - do { - /* Atomicity of LLD is not guaranteed */ - old = lld(var, ll_mo); - /* Must successfully write back to verify atomicity of LLD */ - } while (odp_unlikely(scd(var, neu, sc_mo))); - return old; -} - -static inline __int128 __lockfree_fetch_and_16(__int128 *var, __int128 mask, - int mo) -{ - register __int128 old; - int ll_mo = LL_MO(mo); - int sc_mo = SC_MO(mo); - - do { - /* Atomicity of LLD is not guaranteed */ - old = lld(var, ll_mo); - /* Must successfully write back to verify atomicity of LLD */ - } while (odp_unlikely(scd(var, old & mask, sc_mo))); - return old; -} - -static inline __int128 __lockfree_fetch_or_16(__int128 *var, __int128 mask, - int mo) -{ - register __int128 old; - int ll_mo = LL_MO(mo); - int sc_mo = SC_MO(mo); - - do { - /* Atomicity of LLD is not guaranteed */ - old = lld(var, ll_mo); - /* Must successfully write back to verify atomicity of LLD */ - } while (odp_unlikely(scd(var, old | mask, sc_mo))); - return old; -} - -#else - -static inline __int128 casp(__int128 *var, __int128 old, __int128 neu, int mo) -{ - if (mo == __ATOMIC_RELAXED) { - __asm__ volatile("casp %0, %H0, %1, %H1, [%2]" - : "+r" (old) - : "r" (neu), "r" (var) - : "memory"); - } else if (mo == __ATOMIC_ACQUIRE) { - __asm__ volatile("caspa %0, %H0, %1, %H1, [%2]" - : "+r" (old) - : "r" (neu), "r" (var) - : "memory"); - } else if (mo == __ATOMIC_ACQ_REL) { - __asm__ volatile("caspal %0, %H0, %1, %H1, [%2]" - : "+r" (old) - : "r" (neu), "r" (var) - : "memory"); - } else if (mo == __ATOMIC_RELEASE) { - __asm__ volatile("caspl %0, %H0, %1, %H1, [%2]" - : "+r" (old) - : "r" (neu), "r" (var) - : "memory"); - } else { - abort(); - } - return old; -} - -static inline bool -__lockfree_compare_exchange_16(register __int128 *var, __int128 *exp, - register __int128 neu, bool weak, int mo_success, - int mo_failure) -{ - (void)weak; - (void)mo_failure; - __int128 old; - __int128 expected; - - expected = *exp; - old = casp(var, expected, neu, mo_success); - *exp = old; /* Always update, atomically read value */ - return old == expected; -} - -static inline __int128 __lockfree_exchange_16(__int128 *var, __int128 neu, - int mo) -{ - __int128 old; - __int128 expected; - - do { - expected = *var; - old = casp(var, expected, neu, mo); - } while (old != expected); - return old; -} - -static inline __int128 __lockfree_fetch_and_16(__int128 *var, __int128 mask, - int mo) -{ - __int128 old; - __int128 expected; - - do { - expected = *var; - old = casp(var, expected, expected & mask, mo); - } while (old != expected); - return old; -} - -static inline __int128 __lockfree_fetch_or_16(__int128 *var, __int128 mask, - int mo) -{ - __int128 old; - __int128 expected; - - do { - expected = *var; - old = casp(var, expected, expected | mask, mo); - } while (old != expected); - return old; -} - -#endif /* __ARM_FEATURE_QRDMX */ - -static inline __int128 __lockfree_load_16(__int128 *var, int mo) -{ - __int128 old = *var; /* Possibly torn read */ - - /* Do CAS to ensure atomicity - * Either CAS succeeds (writing back the same value) - * Or CAS fails and returns the old value (atomic read) - */ - (void)__lockfree_compare_exchange_16(var, &old, old, false, mo, mo); - return old; -} - -#endif /* __aarch64__ */ - #endif /* PLATFORM_LINUXGENERIC_ARCH_ARM_ODP_ATOMIC_H */ diff --git a/platform/linux-generic/arch/arm/odp_cpu.h b/platform/linux-generic/arch/arm/odp_cpu.h index 72c810207..d77c85215 100644 --- a/platform/linux-generic/arch/arm/odp_cpu.h +++ b/platform/linux-generic/arch/arm/odp_cpu.h @@ -9,7 +9,7 @@ #ifndef PLATFORM_LINUXGENERIC_ARCH_ARM_ODP_CPU_H #define PLATFORM_LINUXGENERIC_ARCH_ARM_ODP_CPU_H -#if !defined(__arm__) && !defined(__aarch64__) +#if !defined(__arm__) #error Use this file only when compiling for ARM architecture #endif @@ -38,34 +38,16 @@ * more scalable) and enables the CPU to enter a sleep state (lower power * consumption). */ -#ifdef __aarch64__ -#define CONFIG_WFE -#endif +/* #define CONFIG_WFE */ static inline void dmb(void) { __asm__ volatile("dmb" : : : "memory"); } -#ifdef __aarch64__ - -/* Only ARMv8 supports DMB ISHLD */ -/* A load only barrier is much cheaper than full barrier */ -#define _odp_release_barrier(ro) \ -do { \ - if (ro) \ - __asm__ volatile("dmb ishld" ::: "memory"); \ - else \ - __asm__ volatile("dmb ish" ::: "memory"); \ -} while (0) - -#else - #define _odp_release_barrier(ro) \ __atomic_thread_fence(__ATOMIC_RELEASE) -#endif /* __aarch64__ */ - #include "odp_llsc.h" #include "odp_atomic.h" #include "odp_cpu_idling.h" diff --git a/platform/linux-generic/arch/arm/odp_llsc.h b/platform/linux-generic/arch/arm/odp_llsc.h index 73c4d7b11..b53cedc2e 100644 --- a/platform/linux-generic/arch/arm/odp_llsc.h +++ b/platform/linux-generic/arch/arm/odp_llsc.h @@ -13,8 +13,6 @@ #error This file should not be included directly, please include odp_cpu.h #endif -#ifdef __arm__ - static inline uint32_t ll8(uint8_t *var, int mm) { uint8_t old; @@ -95,159 +93,4 @@ static inline uint32_t scd(uint64_t *var, uint64_t neu, int mm) #define sc64(a, b, c) scd((a), (b), (c)) -#endif /* __arm__ */ - -#ifdef __aarch64__ - -static inline uint16_t ll8(uint8_t *var, int mm) -{ - uint16_t old; - - if (mm == __ATOMIC_ACQUIRE) - __asm__ volatile("ldaxrb %w0, [%1]" - : "=&r" (old) - : "r" (var) - : "memory"); - else if (mm == __ATOMIC_RELAXED) - __asm__ volatile("ldxrb %w0, [%1]" - : "=&r" (old) - : "r" (var) - : ); - else - ODP_ABORT(); - return old; -} - -static inline uint32_t ll32(uint32_t *var, int mm) -{ - uint32_t old; - - if (mm == __ATOMIC_ACQUIRE) - __asm__ volatile("ldaxr %w0, [%1]" - : "=&r" (old) - : "r" (var) - : "memory"); - else if (mm == __ATOMIC_RELAXED) - __asm__ volatile("ldxr %w0, [%1]" - : "=&r" (old) - : "r" (var) - : ); - else - ODP_ABORT(); - return old; -} - -/* Return 0 on success, 1 on failure */ -static inline uint32_t sc32(uint32_t *var, uint32_t neu, int mm) -{ - uint32_t ret; - - if (mm == __ATOMIC_RELEASE) - __asm__ volatile("stlxr %w0, %w1, [%2]" - : "=&r" (ret) - : "r" (neu), "r" (var) - : "memory"); - else if (mm == __ATOMIC_RELAXED) - __asm__ volatile("stxr %w0, %w1, [%2]" - : "=&r" (ret) - : "r" (neu), "r" (var) - : ); - else - ODP_ABORT(); - return ret; -} - -static inline uint64_t ll(uint64_t *var, int mm) -{ - uint64_t old; - - if (mm == __ATOMIC_ACQUIRE) - __asm__ volatile("ldaxr %0, [%1]" - : "=&r" (old) - : "r" (var) - : "memory"); - else if (mm == __ATOMIC_RELAXED) - __asm__ volatile("ldxr %0, [%1]" - : "=&r" (old) - : "r" (var) - : ); - else - ODP_ABORT(); - return old; -} - -#define ll64(a, b) ll((a), (b)) - -/* Return 0 on success, 1 on failure */ -static inline uint32_t sc(uint64_t *var, uint64_t neu, int mm) -{ - uint32_t ret; - - if (mm == __ATOMIC_RELEASE) - __asm__ volatile("stlxr %w0, %1, [%2]" - : "=&r" (ret) - : "r" (neu), "r" (var) - : "memory"); - else if (mm == __ATOMIC_RELAXED) - __asm__ volatile("stxr %w0, %1, [%2]" - : "=&r" (ret) - : "r" (neu), "r" (var) - : ); - else - ODP_ABORT(); - return ret; -} - -#define sc64(a, b, c) sc((a), (b), (c)) - -union i128 { - __int128 i128; - int64_t i64[2]; -}; - -static inline __int128 lld(__int128 *var, int mm) -{ - union i128 old; - - if (mm == __ATOMIC_ACQUIRE) - __asm__ volatile("ldaxp %0, %1, [%2]" - : "=&r" (old.i64[0]), "=&r" (old.i64[1]) - : "r" (var) - : "memory"); - else if (mm == __ATOMIC_RELAXED) - __asm__ volatile("ldxp %0, %1, [%2]" - : "=&r" (old.i64[0]), "=&r" (old.i64[1]) - : "r" (var) - : ); - else - ODP_ABORT(); - return old.i128; -} - -/* Return 0 on success, 1 on failure */ -static inline uint32_t scd(__int128 *var, __int128 neu, int mm) -{ - uint32_t ret; - - if (mm == __ATOMIC_RELEASE) - __asm__ volatile("stlxp %w0, %1, %2, [%3]" - : "=&r" (ret) - : "r" (((union i128)neu).i64[0]), - "r" (((union i128)neu).i64[1]), - "r" (var) - : "memory"); - else if (mm == __ATOMIC_RELAXED) - __asm__ volatile("stxp %w0, %1, %2, [%3]" - : "=&r" (ret) - : "r" (((union i128)neu).i64[0]), - "r" (((union i128)neu).i64[1]), - "r" (var) - : ); - else - ODP_ABORT(); - return ret; -} - -#endif /* __aarch64__ */ - #endif /* PLATFORM_LINUXGENERIC_ARCH_ARM_LLSC_H */ diff --git a/platform/linux-generic/arch/mips64/odp_cpu.h b/platform/linux-generic/arch/mips64/odp_cpu.h deleted file mode 100644 index 1097f7bc2..000000000 --- a/platform/linux-generic/arch/mips64/odp_cpu.h +++ /dev/null @@ -1,43 +0,0 @@ -/* Copyright (c) 2017, ARM Limited. All rights reserved. - * - * Copyright (c) 2017, Linaro Limited - * All rights reserved. - * - * SPDX-License-Identifier: BSD-3-Clause - */ - -#ifndef ODP_MIPS64_CPU_H_ -#define ODP_MIPS64_CPU_H_ - -/****************************************************************************** - * Atomics - *****************************************************************************/ - -#define atomic_store_release(loc, val, ro) \ - __atomic_store_n(loc, val, __ATOMIC_RELEASE) - -/****************************************************************************** - * Idle mgmt - *****************************************************************************/ - -static inline void sevl(void) -{ - /* empty */ -} - -static inline int wfe(void) -{ - return 1; -} - -#define monitor128(addr, mo) __atomic_load_n((addr), (mo)) -#define monitor64(addr, mo) __atomic_load_n((addr), (mo)) -#define monitor32(addr, mo) __atomic_load_n((addr), (mo)) -#define monitor8(addr, mo) __atomic_load_n((addr), (mo)) - -static inline void doze(void) -{ - odp_cpu_pause(); -} - -#endif diff --git a/platform/linux-generic/arch/powerpc/odp_cpu.h b/platform/linux-generic/arch/powerpc/odp_cpu.h deleted file mode 100644 index cd27fe240..000000000 --- a/platform/linux-generic/arch/powerpc/odp_cpu.h +++ /dev/null @@ -1,43 +0,0 @@ -/* Copyright (c) 2017, ARM Limited. All rights reserved. - * - * Copyright (c) 2017, Linaro Limited - * All rights reserved. - * - * SPDX-License-Identifier: BSD-3-Clause - */ - -#ifndef ODP_POWERPC_CPU_H_ -#define ODP_POWERPC_CPU_H_ - -/****************************************************************************** - * Atomics - *****************************************************************************/ - -#define atomic_store_release(loc, val, ro) \ - __atomic_store_n(loc, val, __ATOMIC_RELEASE) - -/****************************************************************************** - * Idle mgmt - *****************************************************************************/ - -static inline void sevl(void) -{ - /* empty */ -} - -static inline int wfe(void) -{ - return 1; -} - -#define monitor128(addr, mo) __atomic_load_n((addr), (mo)) -#define monitor64(addr, mo) __atomic_load_n((addr), (mo)) -#define monitor32(addr, mo) __atomic_load_n((addr), (mo)) -#define monitor8(addr, mo) __atomic_load_n((addr), (mo)) - -static inline void doze(void) -{ - odp_cpu_pause(); -} - -#endif diff --git a/platform/linux-generic/arch/x86/odp_cpu.h b/platform/linux-generic/arch/x86/odp_cpu.h deleted file mode 100644 index b5c89409b..000000000 --- a/platform/linux-generic/arch/x86/odp_cpu.h +++ /dev/null @@ -1,43 +0,0 @@ -/* Copyright (c) 2017, ARM Limited. All rights reserved. - * - * Copyright (c) 2017, Linaro Limited - * All rights reserved. - * - * SPDX-License-Identifier: BSD-3-Clause - */ - -#ifndef ODP_X86_CPU_H_ -#define ODP_X86_CPU_H_ - -/****************************************************************************** - * Atomics - *****************************************************************************/ - -#define atomic_store_release(loc, val, ro) \ - __atomic_store_n(loc, val, __ATOMIC_RELEASE) - -/****************************************************************************** - * Idle mgmt - *****************************************************************************/ - -static inline void sevl(void) -{ - /* empty */ -} - -static inline int wfe(void) -{ - return 1; -} - -#define monitor128(addr, mo) __atomic_load_n((addr), (mo)) -#define monitor64(addr, mo) __atomic_load_n((addr), (mo)) -#define monitor32(addr, mo) __atomic_load_n((addr), (mo)) -#define monitor8(addr, mo) __atomic_load_n((addr), (mo)) - -static inline void doze(void) -{ - odp_cpu_pause(); -} - -#endif