From patchwork Fri Oct 27 12:50:01 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dmitry Eremin-Solenikov X-Patchwork-Id: 117339 Delivered-To: patch@linaro.org Received: by 10.140.22.164 with SMTP id 33csp768793qgn; Fri, 27 Oct 2017 06:23:36 -0700 (PDT) X-Google-Smtp-Source: ABhQp+RutYoc2r8AwWFkM48mVfF9dbuom33EKMt8HkvMfqcYBRTa6wsDC5eG0GHhkZstHEiWgw0s X-Received: by 10.107.148.149 with SMTP id w143mr592038iod.140.1509110616065; Fri, 27 Oct 2017 06:23:36 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1509110616; cv=none; d=google.com; s=arc-20160816; b=0qI1xTp+h2LHMwgiUkXIq7Xr8N9XHug+x6Q5Z6/iZ8c2Oq8GGeY3AluS7i6Wq3tpMS 3ZYnPgcjzlvV/1fhsAH0l7ftroVtI2yWz274YE+fpOvRwsJdZF12FcF40U/kCGtnHs9z v4rzl2eCH3w3jrK/BqkfAiDPatKDNnigrveuBKLsEvosE3BV0KDw66isefwKU29iziH+ fwZVIuxHnnvQHRpoQ2U1e1uEFCeg/xx6Qh3xxiqELZwizscgYUEr+kegc3mDEb9df47m arCRNP+vT5+YVvXJ7SwDcWcAB7v4/sBfwC2+yztwxlRXA2eHjy1Q8Cd1DXkbCm7AyI61 0WvA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:subject:references:in-reply-to :message-id:date:to:from:delivered-to:arc-authentication-results; bh=sW17BP7Hs3QBbS6M2ENpummYFR1taXzSLogxL7ec5QQ=; b=PdJLj0bHBHdCA+wkblluSK5fexd5gpvss5qEigDgMXxlR+t7qjG8uIWoCp+dc75fOm EgyyZbWjS2g44lynm4lp71igSRSJV+8zLy+Ps/6wPykNfQsCIzWz48LUZ/J02j6jjAMw aNLP38ROe87oJ/+r0DAZV0ky4o5XShQ+r/xlaILYzDyT+JIR6AtHf2X+LOFncgSSswx/ v6iHFmnH/qw8KaNp2jJLlJhxx01htwcfeoThw0RiTA67o6BcnY3/vySGYvWOAT+E7m8U 7UvWy1IDVKxKbn80oeYllN6R1mjS/Dm0vIAFTa1GjRL0A45x10dIHU3eV/KcvVnHVaxr /5dg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of lng-odp-bounces@lists.linaro.org designates 54.197.127.237 as permitted sender) smtp.mailfrom=lng-odp-bounces@lists.linaro.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.linaro.org (ec2-54-197-127-237.compute-1.amazonaws.com. [54.197.127.237]) by mx.google.com with ESMTP id j134si1295787ita.1.2017.10.27.06.23.35; Fri, 27 Oct 2017 06:23:36 -0700 (PDT) Received-SPF: pass (google.com: domain of lng-odp-bounces@lists.linaro.org designates 54.197.127.237 as permitted sender) client-ip=54.197.127.237; Authentication-Results: mx.google.com; spf=pass (google.com: domain of lng-odp-bounces@lists.linaro.org designates 54.197.127.237 as permitted sender) smtp.mailfrom=lng-odp-bounces@lists.linaro.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: by lists.linaro.org (Postfix, from userid 109) id 930FE62951; Fri, 27 Oct 2017 13:23:35 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on ip-10-142-244-252 X-Spam-Level: X-Spam-Status: No, score=-1.4 required=5.0 tests=BAYES_00, RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H3, RCVD_IN_MSPIKE_WL, RCVD_IN_SORBS_SPAM, URIBL_BLOCKED autolearn=disabled version=3.4.0 Received: from [127.0.0.1] (localhost [127.0.0.1]) by lists.linaro.org (Postfix) with ESMTP id 987F06299F; Fri, 27 Oct 2017 12:54:16 +0000 (UTC) X-Original-To: lng-odp@lists.linaro.org Delivered-To: lng-odp@lists.linaro.org Received: by lists.linaro.org (Postfix, from userid 109) id 53BBE628FD; Fri, 27 Oct 2017 12:52:54 +0000 (UTC) Received: from mail-lf0-f68.google.com (mail-lf0-f68.google.com [209.85.215.68]) by lists.linaro.org (Postfix) with ESMTPS id 95C936295E for ; Fri, 27 Oct 2017 12:50:42 +0000 (UTC) Received: by mail-lf0-f68.google.com with SMTP id l23so7280043lfk.10 for ; Fri, 27 Oct 2017 05:50:42 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references; bh=sW17BP7Hs3QBbS6M2ENpummYFR1taXzSLogxL7ec5QQ=; b=czR5yRnRlyKk+61nztmQQO78AOAdOvA+1jaYn4v9h9r2XCfF8368pgDvJ7okNXqrgX weYTjpQ+o1EhOPpe8vbPeqWJIzqTpE5Nb7wECqhZqkKgC04CnyUn2nzqXe31L/0BWOP4 4R3HJLvnyH8PBhZJA1BlHR1Ooo9rOSVRjnWs7uvP0uRCaI+XoqNHCi8EJrTi6WpOIxwV 0Mh0NWR28fF6NogAWevI5UQ6Adpkd7sMSppsVzVMsMDDYAtCneK9IfClRXbYkLw/xwjs XsQbqpFR9qh2FTOw4ZM4G51rrh9waHpgAcAHaHcQvP/mRYEZVlI9hTp41W4ayNSbSYW5 8umg== X-Gm-Message-State: AMCzsaXHTzbU0ZfxjPNN/8pkbqyHjpOpT7cknXmv/qUf0AyLNn4wDrOt u+bxeakAvZ4jRXOSaaH/Sd4KMBSCf1vVsA== X-Received: by 10.25.207.140 with SMTP id f134mr144886lfg.40.1509108640965; Fri, 27 Oct 2017 05:50:40 -0700 (PDT) Received: from rhovanion.lumag.spb.ru ([94.25.228.222]) by smtp.gmail.com with ESMTPSA id l194sm1584690lfl.11.2017.10.27.05.50.39 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 27 Oct 2017 05:50:40 -0700 (PDT) From: Dmitry Eremin-Solenikov To: lng-odp@lists.linaro.org Date: Fri, 27 Oct 2017 15:50:01 +0300 Message-Id: <20171027125014.27313-18-dmitry.ereminsolenikov@linaro.org> X-Mailer: git-send-email 2.14.2 In-Reply-To: <20171027125014.27313-1-dmitry.ereminsolenikov@linaro.org> References: <20171027125014.27313-1-dmitry.ereminsolenikov@linaro.org> Subject: [lng-odp] [PATCH v10 17/30] linux-gen: atomic: simplify locked 64-bit support X-BeenThere: lng-odp@lists.linaro.org X-Mailman-Version: 2.1.16 Precedence: list List-Id: "The OpenDataPlane \(ODP\) List" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: lng-odp-bounces@lists.linaro.org Sender: "lng-odp" Rewrite atomic_types.h/atomic_inlines.h to clearly separate simple (common) and locked 64-bit cases. This is allows us to ease switching of atomic header to abi setup. Signed-off-by: Dmitry Eremin-Solenikov --- .../include/odp/api/plat/atomic_inlines.h | 315 +++++++++++++-------- .../include/odp/api/plat/atomic_types.h | 58 ++-- .../linux-generic/include/odp_atomic_internal.h | 208 ++++++++++---- 3 files changed, 364 insertions(+), 217 deletions(-) -- 2.14.2 diff --git a/platform/linux-generic/include/odp/api/plat/atomic_inlines.h b/platform/linux-generic/include/odp/api/plat/atomic_inlines.h index 03b2884fdfca..1c58c77db993 100644 --- a/platform/linux-generic/include/odp/api/plat/atomic_inlines.h +++ b/platform/linux-generic/include/odp/api/plat/atomic_inlines.h @@ -109,177 +109,254 @@ _ODP_INLINE void odp_atomic_min_u32(odp_atomic_u32_t *atom, uint32_t new_min) } } +#ifdef ODP_ATOMIC_U64_LOCK + +/** + * @internal + * CAS operation expression for the ATOMIC_OP macro + */ +#define ATOMIC_CAS_OP(ret_ptr, old_val, new_val) \ +({ \ + if (atom->v == (old_val)) { \ + atom->v = (new_val); \ + *(ret_ptr) = 1; \ + } else { \ + *(ret_ptr) = 0; \ + } \ +}) + +/** + * @internal + * Helper macro for lock-based atomic operations on 64-bit integers + * @param[in,out] atom Pointer to the 64-bit atomic variable + * @param expr Expression used update the variable. + * @return The old value of the variable. + */ +#define ATOMIC_OP(atom, expr) \ +({ \ + uint64_t _old_val; \ + /* Loop while lock is already taken, stop when lock becomes clear */ \ + while (__atomic_test_and_set(&(atom)->lock, __ATOMIC_ACQUIRE)) \ + (void)0; \ + _old_val = (atom)->v; \ + (expr); /* Perform whatever update is desired */ \ + __atomic_clear(&(atom)->lock, __ATOMIC_RELEASE); \ + _old_val; /* Return old value */ \ +}) + _ODP_INLINE void odp_atomic_init_u64(odp_atomic_u64_t *atom, uint64_t val) { atom->v = val; -#if __GCC_ATOMIC_LLONG_LOCK_FREE < 2 __atomic_clear(&atom->lock, __ATOMIC_RELAXED); -#endif } _ODP_INLINE uint64_t odp_atomic_load_u64(odp_atomic_u64_t *atom) { -#if __GCC_ATOMIC_LLONG_LOCK_FREE < 2 return ATOMIC_OP(atom, (void)0); -#else - return __atomic_load_n(&atom->v, __ATOMIC_RELAXED); -#endif } _ODP_INLINE void odp_atomic_store_u64(odp_atomic_u64_t *atom, uint64_t val) { -#if __GCC_ATOMIC_LLONG_LOCK_FREE < 2 (void)ATOMIC_OP(atom, atom->v = val); -#else - __atomic_store_n(&atom->v, val, __ATOMIC_RELAXED); -#endif } _ODP_INLINE uint64_t odp_atomic_fetch_add_u64(odp_atomic_u64_t *atom, uint64_t val) { -#if __GCC_ATOMIC_LLONG_LOCK_FREE < 2 return ATOMIC_OP(atom, atom->v += val); -#else - return __atomic_fetch_add(&atom->v, val, __ATOMIC_RELAXED); -#endif } _ODP_INLINE void odp_atomic_add_u64(odp_atomic_u64_t *atom, uint64_t val) { -#if __GCC_ATOMIC_LLONG_LOCK_FREE < 2 (void)ATOMIC_OP(atom, atom->v += val); -#else - (void)__atomic_fetch_add(&atom->v, val, __ATOMIC_RELAXED); -#endif } _ODP_INLINE uint64_t odp_atomic_fetch_sub_u64(odp_atomic_u64_t *atom, uint64_t val) { -#if __GCC_ATOMIC_LLONG_LOCK_FREE < 2 return ATOMIC_OP(atom, atom->v -= val); -#else - return __atomic_fetch_sub(&atom->v, val, __ATOMIC_RELAXED); -#endif } _ODP_INLINE void odp_atomic_sub_u64(odp_atomic_u64_t *atom, uint64_t val) { -#if __GCC_ATOMIC_LLONG_LOCK_FREE < 2 (void)ATOMIC_OP(atom, atom->v -= val); -#else - (void)__atomic_fetch_sub(&atom->v, val, __ATOMIC_RELAXED); -#endif } _ODP_INLINE uint64_t odp_atomic_fetch_inc_u64(odp_atomic_u64_t *atom) { -#if __GCC_ATOMIC_LLONG_LOCK_FREE < 2 return ATOMIC_OP(atom, atom->v++); -#else - return __atomic_fetch_add(&atom->v, 1, __ATOMIC_RELAXED); -#endif } _ODP_INLINE void odp_atomic_inc_u64(odp_atomic_u64_t *atom) { -#if __GCC_ATOMIC_LLONG_LOCK_FREE < 2 (void)ATOMIC_OP(atom, atom->v++); -#else - (void)__atomic_fetch_add(&atom->v, 1, __ATOMIC_RELAXED); -#endif } _ODP_INLINE uint64_t odp_atomic_fetch_dec_u64(odp_atomic_u64_t *atom) { -#if __GCC_ATOMIC_LLONG_LOCK_FREE < 2 return ATOMIC_OP(atom, atom->v--); -#else - return __atomic_fetch_sub(&atom->v, 1, __ATOMIC_RELAXED); -#endif } _ODP_INLINE void odp_atomic_dec_u64(odp_atomic_u64_t *atom) { -#if __GCC_ATOMIC_LLONG_LOCK_FREE < 2 (void)ATOMIC_OP(atom, atom->v--); -#else - (void)__atomic_fetch_sub(&atom->v, 1, __ATOMIC_RELAXED); -#endif } _ODP_INLINE int odp_atomic_cas_u64(odp_atomic_u64_t *atom, uint64_t *old_val, uint64_t new_val) { -#if __GCC_ATOMIC_LLONG_LOCK_FREE < 2 int ret; *old_val = ATOMIC_OP(atom, ATOMIC_CAS_OP(&ret, *old_val, new_val)); return ret; -#else - return __atomic_compare_exchange_n(&atom->v, old_val, new_val, - 0 /* strong */, - __ATOMIC_RELAXED, - __ATOMIC_RELAXED); -#endif } _ODP_INLINE uint64_t odp_atomic_xchg_u64(odp_atomic_u64_t *atom, uint64_t new_val) { -#if __GCC_ATOMIC_LLONG_LOCK_FREE < 2 return ATOMIC_OP(atom, atom->v = new_val); -#else - return __atomic_exchange_n(&atom->v, new_val, __ATOMIC_RELAXED); -#endif } -_ODP_INLINE void odp_atomic_max_u64(odp_atomic_u64_t *atom, uint64_t new_max) +_ODP_INLINE uint64_t odp_atomic_load_acq_u64(odp_atomic_u64_t *atom) { - uint64_t old_val; + return ATOMIC_OP(atom, (void)0); +} - old_val = odp_atomic_load_u64(atom); +_ODP_INLINE void odp_atomic_store_rel_u64(odp_atomic_u64_t *atom, uint64_t val) +{ + (void)ATOMIC_OP(atom, atom->v = val); +} - while (new_max > old_val) { - if (odp_atomic_cas_u64(atom, &old_val, new_max)) - break; - } +_ODP_INLINE void odp_atomic_add_rel_u64(odp_atomic_u64_t *atom, uint64_t val) +{ + (void)ATOMIC_OP(atom, atom->v += val); } -_ODP_INLINE void odp_atomic_min_u64(odp_atomic_u64_t *atom, uint64_t new_min) +_ODP_INLINE void odp_atomic_sub_rel_u64(odp_atomic_u64_t *atom, uint64_t val) { - uint64_t old_val; + (void)ATOMIC_OP(atom, atom->v -= val); +} - old_val = odp_atomic_load_u64(atom); +_ODP_INLINE int odp_atomic_cas_acq_u64(odp_atomic_u64_t *atom, + uint64_t *old_val, uint64_t new_val) +{ + int ret; + *old_val = ATOMIC_OP(atom, ATOMIC_CAS_OP(&ret, *old_val, new_val)); + return ret; +} - while (new_min < old_val) { - if (odp_atomic_cas_u64(atom, &old_val, new_min)) - break; - } +_ODP_INLINE int odp_atomic_cas_rel_u64(odp_atomic_u64_t *atom, + uint64_t *old_val, uint64_t new_val) +{ + int ret; + *old_val = ATOMIC_OP(atom, ATOMIC_CAS_OP(&ret, *old_val, new_val)); + return ret; } -_ODP_INLINE uint32_t odp_atomic_load_acq_u32(odp_atomic_u32_t *atom) +_ODP_INLINE int odp_atomic_cas_acq_rel_u64(odp_atomic_u64_t *atom, + uint64_t *old_val, + uint64_t new_val) +{ + int ret; + *old_val = ATOMIC_OP(atom, ATOMIC_CAS_OP(&ret, *old_val, new_val)); + return ret; +} + +#else /* !ODP_ATOMIC_U64_LOCK */ + +_ODP_INLINE void odp_atomic_init_u64(odp_atomic_u64_t *atom, uint64_t val) +{ + atom->v = val; +} + +_ODP_INLINE uint64_t odp_atomic_load_u64(odp_atomic_u64_t *atom) +{ + return __atomic_load_n(&atom->v, __ATOMIC_RELAXED); +} + +_ODP_INLINE void odp_atomic_store_u64(odp_atomic_u64_t *atom, uint64_t val) +{ + __atomic_store_n(&atom->v, val, __ATOMIC_RELAXED); +} + +_ODP_INLINE uint64_t odp_atomic_fetch_add_u64(odp_atomic_u64_t *atom, + uint64_t val) +{ + return __atomic_fetch_add(&atom->v, val, __ATOMIC_RELAXED); +} + +_ODP_INLINE void odp_atomic_add_u64(odp_atomic_u64_t *atom, uint64_t val) +{ + (void)__atomic_fetch_add(&atom->v, val, __ATOMIC_RELAXED); +} + +_ODP_INLINE uint64_t odp_atomic_fetch_sub_u64(odp_atomic_u64_t *atom, + uint64_t val) +{ + return __atomic_fetch_sub(&atom->v, val, __ATOMIC_RELAXED); +} + +_ODP_INLINE void odp_atomic_sub_u64(odp_atomic_u64_t *atom, uint64_t val) +{ + (void)__atomic_fetch_sub(&atom->v, val, __ATOMIC_RELAXED); +} + +_ODP_INLINE uint64_t odp_atomic_fetch_inc_u64(odp_atomic_u64_t *atom) +{ + return __atomic_fetch_add(&atom->v, 1, __ATOMIC_RELAXED); +} + +_ODP_INLINE void odp_atomic_inc_u64(odp_atomic_u64_t *atom) +{ + (void)__atomic_fetch_add(&atom->v, 1, __ATOMIC_RELAXED); +} + +_ODP_INLINE uint64_t odp_atomic_fetch_dec_u64(odp_atomic_u64_t *atom) +{ + return __atomic_fetch_sub(&atom->v, 1, __ATOMIC_RELAXED); +} + +_ODP_INLINE void odp_atomic_dec_u64(odp_atomic_u64_t *atom) +{ + (void)__atomic_fetch_sub(&atom->v, 1, __ATOMIC_RELAXED); +} + +_ODP_INLINE int odp_atomic_cas_u64(odp_atomic_u64_t *atom, uint64_t *old_val, + uint64_t new_val) +{ + return __atomic_compare_exchange_n(&atom->v, old_val, new_val, + 0 /* strong */, + __ATOMIC_RELAXED, + __ATOMIC_RELAXED); +} + +_ODP_INLINE uint64_t odp_atomic_xchg_u64(odp_atomic_u64_t *atom, + uint64_t new_val) +{ + return __atomic_exchange_n(&atom->v, new_val, __ATOMIC_RELAXED); +} + +_ODP_INLINE uint64_t odp_atomic_load_acq_u64(odp_atomic_u64_t *atom) { return __atomic_load_n(&atom->v, __ATOMIC_ACQUIRE); } -_ODP_INLINE void odp_atomic_store_rel_u32(odp_atomic_u32_t *atom, uint32_t val) +_ODP_INLINE void odp_atomic_store_rel_u64(odp_atomic_u64_t *atom, uint64_t val) { __atomic_store_n(&atom->v, val, __ATOMIC_RELEASE); } -_ODP_INLINE void odp_atomic_add_rel_u32(odp_atomic_u32_t *atom, uint32_t val) +_ODP_INLINE void odp_atomic_add_rel_u64(odp_atomic_u64_t *atom, uint64_t val) { (void)__atomic_fetch_add(&atom->v, val, __ATOMIC_RELEASE); } -_ODP_INLINE void odp_atomic_sub_rel_u32(odp_atomic_u32_t *atom, uint32_t val) +_ODP_INLINE void odp_atomic_sub_rel_u64(odp_atomic_u64_t *atom, uint64_t val) { (void)__atomic_fetch_sub(&atom->v, val, __ATOMIC_RELEASE); } -_ODP_INLINE int odp_atomic_cas_acq_u32(odp_atomic_u32_t *atom, - uint32_t *old_val, uint32_t new_val) +_ODP_INLINE int odp_atomic_cas_acq_u64(odp_atomic_u64_t *atom, + uint64_t *old_val, uint64_t new_val) { return __atomic_compare_exchange_n(&atom->v, old_val, new_val, 0 /* strong */, @@ -287,8 +364,8 @@ _ODP_INLINE int odp_atomic_cas_acq_u32(odp_atomic_u32_t *atom, __ATOMIC_RELAXED); } -_ODP_INLINE int odp_atomic_cas_rel_u32(odp_atomic_u32_t *atom, - uint32_t *old_val, uint32_t new_val) +_ODP_INLINE int odp_atomic_cas_rel_u64(odp_atomic_u64_t *atom, + uint64_t *old_val, uint64_t new_val) { return __atomic_compare_exchange_n(&atom->v, old_val, new_val, 0 /* strong */, @@ -296,9 +373,9 @@ _ODP_INLINE int odp_atomic_cas_rel_u32(odp_atomic_u32_t *atom, __ATOMIC_RELAXED); } -_ODP_INLINE int odp_atomic_cas_acq_rel_u32(odp_atomic_u32_t *atom, - uint32_t *old_val, - uint32_t new_val) +_ODP_INLINE int odp_atomic_cas_acq_rel_u64(odp_atomic_u64_t *atom, + uint64_t *old_val, + uint64_t new_val) { return __atomic_compare_exchange_n(&atom->v, old_val, new_val, 0 /* strong */, @@ -306,86 +383,78 @@ _ODP_INLINE int odp_atomic_cas_acq_rel_u32(odp_atomic_u32_t *atom, __ATOMIC_RELAXED); } -_ODP_INLINE uint64_t odp_atomic_load_acq_u64(odp_atomic_u64_t *atom) +#endif /* !ODP_ATOMIC_U64_LOCK */ + +_ODP_INLINE void odp_atomic_max_u64(odp_atomic_u64_t *atom, uint64_t new_max) +{ + uint64_t old_val; + + old_val = odp_atomic_load_u64(atom); + + while (new_max > old_val) { + if (odp_atomic_cas_u64(atom, &old_val, new_max)) + break; + } +} + +_ODP_INLINE void odp_atomic_min_u64(odp_atomic_u64_t *atom, uint64_t new_min) +{ + uint64_t old_val; + + old_val = odp_atomic_load_u64(atom); + + while (new_min < old_val) { + if (odp_atomic_cas_u64(atom, &old_val, new_min)) + break; + } +} + +_ODP_INLINE uint32_t odp_atomic_load_acq_u32(odp_atomic_u32_t *atom) { -#if __GCC_ATOMIC_LLONG_LOCK_FREE < 2 - return ATOMIC_OP(atom, (void)0); -#else return __atomic_load_n(&atom->v, __ATOMIC_ACQUIRE); -#endif } -_ODP_INLINE void odp_atomic_store_rel_u64(odp_atomic_u64_t *atom, uint64_t val) +_ODP_INLINE void odp_atomic_store_rel_u32(odp_atomic_u32_t *atom, uint32_t val) { -#if __GCC_ATOMIC_LLONG_LOCK_FREE < 2 - (void)ATOMIC_OP(atom, atom->v = val); -#else __atomic_store_n(&atom->v, val, __ATOMIC_RELEASE); -#endif } -_ODP_INLINE void odp_atomic_add_rel_u64(odp_atomic_u64_t *atom, uint64_t val) +_ODP_INLINE void odp_atomic_add_rel_u32(odp_atomic_u32_t *atom, uint32_t val) { -#if __GCC_ATOMIC_LLONG_LOCK_FREE < 2 - (void)ATOMIC_OP(atom, atom->v += val); -#else (void)__atomic_fetch_add(&atom->v, val, __ATOMIC_RELEASE); -#endif } -_ODP_INLINE void odp_atomic_sub_rel_u64(odp_atomic_u64_t *atom, uint64_t val) +_ODP_INLINE void odp_atomic_sub_rel_u32(odp_atomic_u32_t *atom, uint32_t val) { -#if __GCC_ATOMIC_LLONG_LOCK_FREE < 2 - (void)ATOMIC_OP(atom, atom->v -= val); -#else (void)__atomic_fetch_sub(&atom->v, val, __ATOMIC_RELEASE); -#endif } -_ODP_INLINE int odp_atomic_cas_acq_u64(odp_atomic_u64_t *atom, - uint64_t *old_val, uint64_t new_val) +_ODP_INLINE int odp_atomic_cas_acq_u32(odp_atomic_u32_t *atom, + uint32_t *old_val, uint32_t new_val) { -#if __GCC_ATOMIC_LLONG_LOCK_FREE < 2 - int ret; - *old_val = ATOMIC_OP(atom, ATOMIC_CAS_OP(&ret, *old_val, new_val)); - return ret; -#else return __atomic_compare_exchange_n(&atom->v, old_val, new_val, 0 /* strong */, __ATOMIC_ACQUIRE, __ATOMIC_RELAXED); -#endif } -_ODP_INLINE int odp_atomic_cas_rel_u64(odp_atomic_u64_t *atom, - uint64_t *old_val, uint64_t new_val) +_ODP_INLINE int odp_atomic_cas_rel_u32(odp_atomic_u32_t *atom, + uint32_t *old_val, uint32_t new_val) { -#if __GCC_ATOMIC_LLONG_LOCK_FREE < 2 - int ret; - *old_val = ATOMIC_OP(atom, ATOMIC_CAS_OP(&ret, *old_val, new_val)); - return ret; -#else return __atomic_compare_exchange_n(&atom->v, old_val, new_val, 0 /* strong */, __ATOMIC_RELEASE, __ATOMIC_RELAXED); -#endif } -_ODP_INLINE int odp_atomic_cas_acq_rel_u64(odp_atomic_u64_t *atom, - uint64_t *old_val, - uint64_t new_val) +_ODP_INLINE int odp_atomic_cas_acq_rel_u32(odp_atomic_u32_t *atom, + uint32_t *old_val, + uint32_t new_val) { -#if __GCC_ATOMIC_LLONG_LOCK_FREE < 2 - int ret; - *old_val = ATOMIC_OP(atom, ATOMIC_CAS_OP(&ret, *old_val, new_val)); - return ret; -#else return __atomic_compare_exchange_n(&atom->v, old_val, new_val, 0 /* strong */, __ATOMIC_ACQ_REL, __ATOMIC_RELAXED); -#endif } #endif diff --git a/platform/linux-generic/include/odp/api/plat/atomic_types.h b/platform/linux-generic/include/odp/api/plat/atomic_types.h index a674ac9978c4..c0803bf11f63 100644 --- a/platform/linux-generic/include/odp/api/plat/atomic_types.h +++ b/platform/linux-generic/include/odp/api/plat/atomic_types.h @@ -20,61 +20,39 @@ extern "C" { #include #include -/** - * @internal - * Atomic 64-bit unsigned integer - */ -struct odp_atomic_u64_s { - uint64_t v; /**< Actual storage for the atomic variable */ -#if __GCC_ATOMIC_LLONG_LOCK_FREE < 2 - /* Some architectures do not support lock-free operations on 64-bit - * data types. We use a spin lock to ensure atomicity. */ - char lock; /**< Spin lock (if needed) used to ensure atomic access */ -#endif -} ODP_ALIGNED(sizeof(uint64_t)); /* Enforce alignement! */ - /** * @internal * Atomic 32-bit unsigned integer */ struct odp_atomic_u32_s { uint32_t v; /**< Actual storage for the atomic variable */ -} ODP_ALIGNED(sizeof(uint32_t)); /* Enforce alignement! */ +} ODP_ALIGNED(sizeof(uint32_t)); /* Enforce alignment! */ -#if __GCC_ATOMIC_LLONG_LOCK_FREE < 2 +#if __GCC_ATOMIC_LLONG_LOCK_FREE >= 2 /** * @internal - * CAS operation expression for the ATOMIC_OP macro + * Atomic 64-bit unsigned integer */ -#define ATOMIC_CAS_OP(ret_ptr, old_val, new_val) \ -({ \ - if (atom->v == (old_val)) { \ - atom->v = (new_val); \ - *(ret_ptr) = 1; \ - } else { \ - *(ret_ptr) = 0; \ - } \ -}) +struct odp_atomic_u64_s { + uint64_t v; /**< Actual storage for the atomic variable */ +} ODP_ALIGNED(sizeof(uint64_t)); /* Enforce alignment! */ + +#else + +#define ODP_ATOMIC_U64_LOCK 1 /** * @internal - * Helper macro for lock-based atomic operations on 64-bit integers - * @param[in,out] atom Pointer to the 64-bit atomic variable - * @param expr Expression used update the variable. - * @return The old value of the variable. + * Atomic 64-bit unsigned integer */ -#define ATOMIC_OP(atom, expr) \ -({ \ - uint64_t _old_val; \ - /* Loop while lock is already taken, stop when lock becomes clear */ \ - while (__atomic_test_and_set(&(atom)->lock, __ATOMIC_ACQUIRE)) \ - (void)0; \ - _old_val = (atom)->v; \ - (expr); /* Perform whatever update is desired */ \ - __atomic_clear(&(atom)->lock, __ATOMIC_RELEASE); \ - _old_val; /* Return old value */ \ -}) +struct odp_atomic_u64_s { + uint64_t v; /**< Actual storage for the atomic variable */ + /* Some architectures do not support lock-free operations on 64-bit + * data types. We use a spin lock to ensure atomicity. */ + char lock; /**< Spin lock (if needed) used to ensure atomic access */ +} ODP_ALIGNED(sizeof(uint64_t)); /* Enforce alignment! */ + #endif typedef struct odp_atomic_u64_s odp_atomic_u64_t; diff --git a/platform/linux-generic/include/odp_atomic_internal.h b/platform/linux-generic/include/odp_atomic_internal.h index dca2175cefbf..f3696a99426f 100644 --- a/platform/linux-generic/include/odp_atomic_internal.h +++ b/platform/linux-generic/include/odp_atomic_internal.h @@ -223,7 +223,7 @@ static inline void _odp_atomic_u32_sub_mm(odp_atomic_u32_t *atom, *****************************************************************************/ /* Check if the compiler support lock-less atomic operations on 64-bit types */ -#if !defined __GCC_ATOMIC_LLONG_LOCK_FREE || __GCC_ATOMIC_LLONG_LOCK_FREE < 2 +#ifdef ODP_ATOMIC_U64_LOCK /** * @internal * Helper macro for lock-based atomic operations on 64-bit integers @@ -247,7 +247,6 @@ static inline void _odp_atomic_u32_sub_mm(odp_atomic_u32_t *atom, __ATOMIC_SEQ_CST : __ATOMIC_RELEASE); \ old_val; /* Return old value */ \ }) -#endif /** * Atomic load of 64-bit atomic variable @@ -258,13 +257,9 @@ static inline void _odp_atomic_u32_sub_mm(odp_atomic_u32_t *atom, * @return Value of the variable */ static inline uint64_t _odp_atomic_u64_load_mm(odp_atomic_u64_t *atom, - _odp_memmodel_t mmodel) + _odp_memmodel_t mmodel) { -#if !defined __GCC_ATOMIC_LLONG_LOCK_FREE || __GCC_ATOMIC_LLONG_LOCK_FREE < 2 return ATOMIC_OP_MM(atom, (void)0, mmodel); -#else - return __atomic_load_n(&atom->v, mmodel); -#endif } /** @@ -275,14 +270,10 @@ static inline uint64_t _odp_atomic_u64_load_mm(odp_atomic_u64_t *atom, * @param mmodel Memory order associated with the store operation */ static inline void _odp_atomic_u64_store_mm(odp_atomic_u64_t *atom, - uint64_t val, - _odp_memmodel_t mmodel) + uint64_t val, + _odp_memmodel_t mmodel) { -#if !defined __GCC_ATOMIC_LLONG_LOCK_FREE || __GCC_ATOMIC_LLONG_LOCK_FREE < 2 (void)ATOMIC_OP_MM(atom, atom->v = val, mmodel); -#else - __atomic_store_n(&atom->v, val, mmodel); -#endif } /** @@ -295,15 +286,11 @@ static inline void _odp_atomic_u64_store_mm(odp_atomic_u64_t *atom, * @return Old value of variable */ static inline uint64_t _odp_atomic_u64_xchg_mm(odp_atomic_u64_t *atom, - uint64_t val, - _odp_memmodel_t mmodel) + uint64_t val, + _odp_memmodel_t mmodel) { -#if !defined __GCC_ATOMIC_LLONG_LOCK_FREE || __GCC_ATOMIC_LLONG_LOCK_FREE < 2 return ATOMIC_OP_MM(atom, atom->v = val, mmodel); -#else - return __atomic_exchange_n(&atom->v, val, mmodel); -#endif } /** @@ -322,12 +309,11 @@ static inline uint64_t _odp_atomic_u64_xchg_mm(odp_atomic_u64_t *atom, * @retval 0 exchange failed and '*exp' updated with current value */ static inline int _odp_atomic_u64_cmp_xchg_strong_mm(odp_atomic_u64_t *atom, - uint64_t *exp, - uint64_t val, - _odp_memmodel_t success, - _odp_memmodel_t failure) + uint64_t *exp, + uint64_t val, + _odp_memmodel_t success, + _odp_memmodel_t failure) { -#if !defined __GCC_ATOMIC_LLONG_LOCK_FREE || __GCC_ATOMIC_LLONG_LOCK_FREE < 2 /* Possibly we are a bit pessimistic with the memory models */ odp_bool_t ret_succ; /* Loop while lock is already taken, stop when lock becomes clear */ @@ -346,10 +332,6 @@ static inline int _odp_atomic_u64_cmp_xchg_strong_mm(odp_atomic_u64_t *atom, (ret_succ ? success : failure) == _ODP_MEMMODEL_SC ? __ATOMIC_SEQ_CST : __ATOMIC_RELEASE); return ret_succ; -#else - return __atomic_compare_exchange_n(&atom->v, exp, val, - false/*strong*/, success, failure); -#endif } /** @@ -362,14 +344,10 @@ static inline int _odp_atomic_u64_cmp_xchg_strong_mm(odp_atomic_u64_t *atom, * @return Value of the atomic variable before the addition */ static inline uint64_t _odp_atomic_u64_fetch_add_mm(odp_atomic_u64_t *atom, - uint64_t val, - _odp_memmodel_t mmodel) + uint64_t val, + _odp_memmodel_t mmodel) { -#if !defined __GCC_ATOMIC_LLONG_LOCK_FREE || __GCC_ATOMIC_LLONG_LOCK_FREE < 2 return ATOMIC_OP_MM(atom, atom->v += val, mmodel); -#else - return __atomic_fetch_add(&atom->v, val, mmodel); -#endif } /** @@ -380,15 +358,11 @@ static inline uint64_t _odp_atomic_u64_fetch_add_mm(odp_atomic_u64_t *atom, * @param mmodel Memory order associated with the add operation. */ static inline void _odp_atomic_u64_add_mm(odp_atomic_u64_t *atom, - uint64_t val, - _odp_memmodel_t mmodel) + uint64_t val, + _odp_memmodel_t mmodel) { -#if !defined __GCC_ATOMIC_LLONG_LOCK_FREE || __GCC_ATOMIC_LLONG_LOCK_FREE < 2 (void)ATOMIC_OP_MM(atom, atom->v += val, mmodel); -#else - (void)__atomic_fetch_add(&atom->v, val, mmodel); -#endif } /** @@ -401,14 +375,10 @@ static inline void _odp_atomic_u64_add_mm(odp_atomic_u64_t *atom, * @return Value of the atomic variable before the subtraction */ static inline uint64_t _odp_atomic_u64_fetch_sub_mm(odp_atomic_u64_t *atom, - uint64_t val, - _odp_memmodel_t mmodel) + uint64_t val, + _odp_memmodel_t mmodel) { -#if !defined __GCC_ATOMIC_LLONG_LOCK_FREE || __GCC_ATOMIC_LLONG_LOCK_FREE < 2 return ATOMIC_OP_MM(atom, atom->v -= val, mmodel); -#else - return __atomic_fetch_sub(&atom->v, val, mmodel); -#endif } /** @@ -419,20 +389,150 @@ static inline uint64_t _odp_atomic_u64_fetch_sub_mm(odp_atomic_u64_t *atom, * @param mmodel Memory order associated with the subtract operation */ static inline void _odp_atomic_u64_sub_mm(odp_atomic_u64_t *atom, - uint64_t val, - _odp_memmodel_t mmodel) + uint64_t val, + _odp_memmodel_t mmodel) { -#if !defined __GCC_ATOMIC_LLONG_LOCK_FREE || __GCC_ATOMIC_LLONG_LOCK_FREE < 2 (void)ATOMIC_OP_MM(atom, atom->v -= val, mmodel); -#else - (void)__atomic_fetch_sub(&atom->v, val, mmodel); -#endif } -#if !defined __GCC_ATOMIC_LLONG_LOCK_FREE || __GCC_ATOMIC_LLONG_LOCK_FREE < 2 #undef ATOMIC_OP_MM -#endif + +#else /* ! ODP_ATOMIC_U64_LOCK */ + +/** + * Atomic load of 64-bit atomic variable + * + * @param atom Pointer to a 64-bit atomic variable + * @param mmodel Memory order associated with the load operation + * + * @return Value of the variable + */ +static inline uint64_t _odp_atomic_u64_load_mm(odp_atomic_u64_t *atom, + _odp_memmodel_t mmodel) +{ + return __atomic_load_n(&atom->v, mmodel); +} + +/** + * Atomic store to 64-bit atomic variable + * + * @param[out] atom Pointer to a 64-bit atomic variable + * @param val Value to write to the atomic variable + * @param mmodel Memory order associated with the store operation + */ +static inline void _odp_atomic_u64_store_mm(odp_atomic_u64_t *atom, + uint64_t val, + _odp_memmodel_t mmodel) +{ + __atomic_store_n(&atom->v, val, mmodel); +} + +/** + * Atomic exchange (swap) of 64-bit atomic variable + * + * @param[in,out] atom Pointer to a 64-bit atomic variable + * @param val New value to write to the atomic variable + * @param mmodel Memory order associated with the exchange operation + * + * @return Old value of variable + */ +static inline uint64_t _odp_atomic_u64_xchg_mm(odp_atomic_u64_t *atom, + uint64_t val, + _odp_memmodel_t mmodel) + +{ + return __atomic_exchange_n(&atom->v, val, mmodel); +} + +/** + * Atomic compare and exchange (swap) of 64-bit atomic variable + * "Strong" semantics, will not fail spuriously. + * + * @param[in,out] atom Pointer to a 64-bit atomic variable + * @param[in,out] exp Pointer to expected value (updated on failure) + * @param val New value to write + * @param success Memory order associated with a successful compare-and-swap + * operation + * @param failure Memory order associated with a failed compare-and-swap + * operation + * + * @retval 1 exchange successful + * @retval 0 exchange failed and '*exp' updated with current value + */ +static inline int _odp_atomic_u64_cmp_xchg_strong_mm(odp_atomic_u64_t *atom, + uint64_t *exp, + uint64_t val, + _odp_memmodel_t success, + _odp_memmodel_t failure) +{ + return __atomic_compare_exchange_n(&atom->v, exp, val, + false/*strong*/, success, failure); +} + +/** + * Atomic fetch and add of 64-bit atomic variable + * + * @param[in,out] atom Pointer to a 64-bit atomic variable + * @param val Value to add to the atomic variable + * @param mmodel Memory order associated with the add operation + * + * @return Value of the atomic variable before the addition + */ +static inline uint64_t _odp_atomic_u64_fetch_add_mm(odp_atomic_u64_t *atom, + uint64_t val, + _odp_memmodel_t mmodel) +{ + return __atomic_fetch_add(&atom->v, val, mmodel); +} + +/** + * Atomic add of 64-bit atomic variable + * + * @param[in,out] atom Pointer to a 64-bit atomic variable + * @param val Value to add to the atomic variable + * @param mmodel Memory order associated with the add operation. + */ +static inline void _odp_atomic_u64_add_mm(odp_atomic_u64_t *atom, + uint64_t val, + _odp_memmodel_t mmodel) + +{ + (void)__atomic_fetch_add(&atom->v, val, mmodel); +} + +/** + * Atomic fetch and subtract of 64-bit atomic variable + * + * @param[in,out] atom Pointer to a 64-bit atomic variable + * @param val Value to subtract from the atomic variable + * @param mmodel Memory order associated with the subtract operation + * + * @return Value of the atomic variable before the subtraction + */ +static inline uint64_t _odp_atomic_u64_fetch_sub_mm(odp_atomic_u64_t *atom, + uint64_t val, + _odp_memmodel_t mmodel) +{ + return __atomic_fetch_sub(&atom->v, val, mmodel); +} + +/** + * Atomic subtract of 64-bit atomic variable + * + * @param[in,out] atom Pointer to a 64-bit atomic variable + * @param val Value to subtract from the atomic variable + * @param mmodel Memory order associated with the subtract operation + */ +static inline void _odp_atomic_u64_sub_mm(odp_atomic_u64_t *atom, + uint64_t val, + _odp_memmodel_t mmodel) + +{ + (void)__atomic_fetch_sub(&atom->v, val, mmodel); +} + +#endif /* ! ODP_ATOMIC_U64_LOCK */ /***************************************************************************** * Operations on pointer atomics