From patchwork Tue Oct 2 16:19:12 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 147992 Delivered-To: patch@linaro.org Received: by 2002:a2e:8595:0:0:0:0:0 with SMTP id b21-v6csp5356254lji; Tue, 2 Oct 2018 09:21:36 -0700 (PDT) X-Google-Smtp-Source: ACcGV63z3gvy60LJztJM8Zzs6wlSJ4R5ANuISFjQRUdt5o2mG1QJ8XxUut0SsD+U6ug5ZlF36kmT X-Received: by 2002:a62:2845:: with SMTP id o66-v6mr16999557pfo.17.1538497295953; Tue, 02 Oct 2018 09:21:35 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1538497295; cv=none; d=google.com; s=arc-20160816; b=kpt65aegRu/bQebh+gKHJzznp2Aa1Cbv7y87o1ZjMfJZC1y0WxuVekLAcSrHEyopDd GMhE1zs1mH2RZPKJzY8gAz82EqftzlveSJCvsiXp+FqpX0Z6n4Zt1DNSZJ0R3zgXdfn4 UBfVIZyQS0sXVjmkneHwlV+rE0c6+v/Bvg025Q4IgBCuGDZVFyD0eHSUChaVYaVlyGlU 6icGbTk4mvGF/FFMhhlwyeAJ66/3uiZgNMUI2l/rGN1ZcwYnsdk6yQMQq457jgRA2H1B 8BJ43dTh6N/uj03z3mGYzBI4vhnIbbnyWlQYNMN2a/XYl6wWShJkuAUwsvU4wHzQd9bS svTw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature:delivered-to:sender:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:mailing-list:dkim-signature :domainkey-signature; bh=TnkmUZKwSeiVVvXBlseyaEdP1qX0dRubqwLyf4ACWd4=; b=D2u0YSjh0YSrMYrAOIMBvlg/rvh4lnC/dpi1w+2ziGRJVWGFNIPdH9KVetquGiEA/+ vAs+SWR87lXoKtpqQSZzS7Ljf5LQgZ6jh1g9Q73DVdewOGddzx0GQHW4MRd3/ym/9kEl he6CVsAcXqiHw2aoiJo+EB6x7681F8Dgl9ee/DgdhoR4lhUWOVkd0FXUvAqaDNAK+1E0 2Q01Jay75iAtTk3+Rz1lqkoAuO5Kzu1RIiE1WTi+t02hrF2epuQkXQJwnV4T5kW3hPQS bTG9YC5hXpoBUVI3dWvEBAVrWSYaeNuVmcF3xfaBSyakyDda6eXCUqyBxcDrI0SnHUhu Gdnw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gcc.gnu.org header.s=default header.b=KrhPdOTR; dkim=pass header.i=@linaro.org header.s=google header.b=U0wS9Dp3; spf=pass (google.com: domain of gcc-patches-return-486824-patch=linaro.org@gcc.gnu.org designates 209.132.180.131 as permitted sender) smtp.mailfrom="gcc-patches-return-486824-patch=linaro.org@gcc.gnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from sourceware.org (server1.sourceware.org. [209.132.180.131]) by mx.google.com with ESMTPS id f6-v6si16046766plt.16.2018.10.02.09.21.35 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 02 Oct 2018 09:21:35 -0700 (PDT) Received-SPF: pass (google.com: domain of gcc-patches-return-486824-patch=linaro.org@gcc.gnu.org designates 209.132.180.131 as permitted sender) client-ip=209.132.180.131; Authentication-Results: mx.google.com; dkim=pass header.i=@gcc.gnu.org header.s=default header.b=KrhPdOTR; dkim=pass header.i=@linaro.org header.s=google header.b=U0wS9Dp3; spf=pass (google.com: domain of gcc-patches-return-486824-patch=linaro.org@gcc.gnu.org designates 209.132.180.131 as permitted sender) smtp.mailfrom="gcc-patches-return-486824-patch=linaro.org@gcc.gnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org DomainKey-Signature: a=rsa-sha1; c=nofws; d=gcc.gnu.org; h=list-id :list-unsubscribe:list-archive:list-post:list-help:sender:from :to:cc:subject:date:message-id:in-reply-to:references; q=dns; s= default; b=lb+HdUYuV2O/TtG1whGRFilDwjc3DWUlXYDmvlvWbk1Jf+BSxnNvO sB6d0lBD3xMo/NaNnUl922p9O90MzTwURUEZdwgLcy+OJkrSmx5G9nGY1Ky7wF+k 56Vcb/bBnhjlzLbfLigh0o0pqIBLNtRdLV1GKlf49j5udOIk6UklIU= DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=gcc.gnu.org; h=list-id :list-unsubscribe:list-archive:list-post:list-help:sender:from :to:cc:subject:date:message-id:in-reply-to:references; s= default; bh=B7ZCur5PTJdiamG4KgJeu10MPuU=; b=KrhPdOTRdCgHT6IKqR0V LIREId1Fvi7YG0UYBiaYRyr4D7ogPmt/JDzrg33s7jU5yMMXrfgzKndA6YWx3hzd 78wXVDxonfJi0jgwm5a9/Yk9QOiJa4HkULO2YwIbI8z+XIOmtYOcsxo82WEfgrAK sj1hqFp4uyMLeCKE0u6EPLQ= Received: (qmail 68382 invoked by alias); 2 Oct 2018 16:19:41 -0000 Mailing-List: contact gcc-patches-help@gcc.gnu.org; run by ezmlm Precedence: bulk List-Id: List-Unsubscribe: List-Archive: List-Post: List-Help: Sender: gcc-patches-owner@gcc.gnu.org Delivered-To: mailing list gcc-patches@gcc.gnu.org Received: (qmail 68276 invoked by uid 89); 2 Oct 2018 16:19:40 -0000 Authentication-Results: sourceware.org; auth=none X-Spam-SWARE-Status: No, score=-26.9 required=5.0 tests=BAYES_00, GIT_PATCH_0, GIT_PATCH_1, GIT_PATCH_2, GIT_PATCH_3, RCVD_IN_DNSWL_NONE, SPF_PASS autolearn=ham version=3.3.2 spammy=applicable, NAMES, relax X-HELO: mail-ot1-f41.google.com Received: from mail-ot1-f41.google.com (HELO mail-ot1-f41.google.com) (209.85.210.41) by sourceware.org (qpsmtpd/0.93/v0.84-503-g423c35a) with ESMTP; Tue, 02 Oct 2018 16:19:36 +0000 Received: by mail-ot1-f41.google.com with SMTP id h26-v6so2463784otl.9 for ; Tue, 02 Oct 2018 09:19:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=TnkmUZKwSeiVVvXBlseyaEdP1qX0dRubqwLyf4ACWd4=; b=U0wS9Dp3w3oLRKQCQi/7v7mF4okdf4AM/zdOPzfXIC8/P6CBTY005zH1LcrfGfw8G4 T/9EfdUr2RkZ9Uh9Ph9ZnFGB0n3p/X3cMYyR9aINq1Toc9vZ+nPWcOdzD4MCdbZ9RUYe QpeEVbNSRtdYmfoigiSLQFWPSXKntk8E2P7ls= Return-Path: Received: from cloudburst.twiddle.net ([187.217.227.243]) by smtp.gmail.com with ESMTPSA id j15-v6sm145987oth.27.2018.10.02.09.19.32 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Tue, 02 Oct 2018 09:19:33 -0700 (PDT) From: Richard Henderson To: gcc-patches@gcc.gnu.org Cc: ramana.radhakrishnan@arm.com, agraf@suse.de, marcus.shawcroft@arm.com, james.greenhalgh@arm.com, richard.earnshaw@arm.com Subject: [PATCH, AArch64 v2 08/11] aarch64: Implement -matomic-ool Date: Tue, 2 Oct 2018 11:19:12 -0500 Message-Id: <20181002161915.18843-9-richard.henderson@linaro.org> In-Reply-To: <20181002161915.18843-1-richard.henderson@linaro.org> References: <20181002161915.18843-1-richard.henderson@linaro.org> * config/aarch64/aarch64.opt (-matomic-ool): New. * config/aarch64/aarch64.c (aarch64_atomic_ool_func): New. (aarch64_ool_cas_names, aarch64_ool_swp_names): New. (aarch64_ool_ldadd_names, aarch64_ool_ldset_names): New. (aarch64_ool_ldclr_names, aarch64_ool_ldeor_names): New. (aarch64_ool_stadd_names, aarch64_ool_stset_names): New. (aarch64_ool_stclr_names, aarch64_ool_steor_names): New. (aarch64_expand_compare_and_swap): Honor TARGET_ATOMIC_OOL. * config/aarch64/atomics.md (atomic_exchange): Likewise. (atomic_): Likewise. (atomic_fetch_): Likewise. (atomic__fetch): Likewise. --- gcc/config/aarch64/aarch64-protos.h | 17 ++++ gcc/config/aarch64/aarch64.c | 95 +++++++++++++++++++ .../atomic-comp-swap-release-acquire.c | 2 +- .../gcc.target/aarch64/atomic-op-acq_rel.c | 2 +- .../gcc.target/aarch64/atomic-op-acquire.c | 2 +- .../gcc.target/aarch64/atomic-op-char.c | 2 +- .../gcc.target/aarch64/atomic-op-consume.c | 2 +- .../gcc.target/aarch64/atomic-op-imm.c | 2 +- .../gcc.target/aarch64/atomic-op-int.c | 2 +- .../gcc.target/aarch64/atomic-op-long.c | 2 +- .../gcc.target/aarch64/atomic-op-relaxed.c | 2 +- .../gcc.target/aarch64/atomic-op-release.c | 2 +- .../gcc.target/aarch64/atomic-op-seq_cst.c | 2 +- .../gcc.target/aarch64/atomic-op-short.c | 2 +- .../aarch64/atomic_cmp_exchange_zero_reg_1.c | 2 +- .../atomic_cmp_exchange_zero_strong_1.c | 2 +- .../gcc.target/aarch64/sync-comp-swap.c | 2 +- .../gcc.target/aarch64/sync-op-acquire.c | 2 +- .../gcc.target/aarch64/sync-op-full.c | 2 +- gcc/config/aarch64/aarch64.opt | 4 + gcc/config/aarch64/atomics.md | 94 ++++++++++++++++-- gcc/doc/invoke.texi | 14 ++- 22 files changed, 232 insertions(+), 26 deletions(-) -- 2.17.1 diff --git a/gcc/config/aarch64/aarch64-protos.h b/gcc/config/aarch64/aarch64-protos.h index 1d2f8487d1a..c7b96b12bbe 100644 --- a/gcc/config/aarch64/aarch64-protos.h +++ b/gcc/config/aarch64/aarch64-protos.h @@ -624,4 +624,21 @@ poly_uint64 aarch64_regmode_natural_size (machine_mode); bool aarch64_high_bits_all_ones_p (HOST_WIDE_INT); +struct atomic_ool_names +{ + const char *str[4][4]; +}; + +rtx aarch64_atomic_ool_func(machine_mode mode, rtx model_rtx, + const atomic_ool_names *names); +extern const atomic_ool_names aarch64_ool_swp_names; +extern const atomic_ool_names aarch64_ool_stadd_names; +extern const atomic_ool_names aarch64_ool_stset_names; +extern const atomic_ool_names aarch64_ool_stclr_names; +extern const atomic_ool_names aarch64_ool_steor_names; +extern const atomic_ool_names aarch64_ool_ldadd_names; +extern const atomic_ool_names aarch64_ool_ldset_names; +extern const atomic_ool_names aarch64_ool_ldclr_names; +extern const atomic_ool_names aarch64_ool_ldeor_names; + #endif /* GCC_AARCH64_PROTOS_H */ diff --git a/gcc/config/aarch64/aarch64.c b/gcc/config/aarch64/aarch64.c index 867759f7e80..49b47382b5d 100644 --- a/gcc/config/aarch64/aarch64.c +++ b/gcc/config/aarch64/aarch64.c @@ -14160,6 +14160,90 @@ aarch64_emit_unlikely_jump (rtx insn) add_reg_br_prob_note (jump, profile_probability::very_unlikely ()); } +/* We store the names of the various atomic helpers in a 4x4 array. + Return the libcall function given MODE, MODEL and NAMES. */ + +rtx +aarch64_atomic_ool_func(machine_mode mode, rtx model_rtx, + const atomic_ool_names *names) +{ + memmodel model = memmodel_base (INTVAL (model_rtx)); + int mode_idx, model_idx; + + switch (mode) + { + case E_QImode: + mode_idx = 0; + break; + case E_HImode: + mode_idx = 1; + break; + case E_SImode: + mode_idx = 2; + break; + case E_DImode: + mode_idx = 3; + break; + default: + gcc_unreachable (); + } + + switch (model) + { + case MEMMODEL_RELAXED: + model_idx = 0; + break; + case MEMMODEL_CONSUME: + case MEMMODEL_ACQUIRE: + model_idx = 1; + break; + case MEMMODEL_RELEASE: + model_idx = 2; + break; + case MEMMODEL_ACQ_REL: + case MEMMODEL_SEQ_CST: + model_idx = 3; + break; + default: + gcc_unreachable (); + } + + return init_one_libfunc_visibility (names->str[mode_idx][model_idx], + VISIBILITY_HIDDEN); +} + +#define DEF0(B, N) \ + { "__aa64_" #B #N "_relax", \ + "__aa64_" #B #N "_acq", \ + "__aa64_" #B #N "_rel", \ + "__aa64_" #B #N "_acq_rel" } + +#define DEF4(B) DEF0(B, 1), DEF0(B, 2), DEF0(B, 4), DEF0(B, 8) + +static const atomic_ool_names aarch64_ool_cas_names = { { DEF4(cas) } }; +const atomic_ool_names aarch64_ool_swp_names = { { DEF4(swp) } }; +const atomic_ool_names aarch64_ool_ldadd_names = { { DEF4(ldadd) } }; +const atomic_ool_names aarch64_ool_ldset_names = { { DEF4(ldset) } }; +const atomic_ool_names aarch64_ool_ldclr_names = { { DEF4(ldclr) } }; +const atomic_ool_names aarch64_ool_ldeor_names = { { DEF4(ldeor) } }; + +/* Note that the store forms are only available for relax and release + memory models. For the other models, re-use the load forms. */ +#undef DEF0 +#define DEF0(B, N) \ + { "__aa64_st" #B #N "_relax", \ + "__aa64_ld" #B #N "_acq", \ + "__aa64_st" #B #N "_rel", \ + "__aa64_ld" #B #N "_acq_rel" } + +const atomic_ool_names aarch64_ool_stadd_names = { { DEF4(add) } }; +const atomic_ool_names aarch64_ool_stset_names = { { DEF4(set) } }; +const atomic_ool_names aarch64_ool_stclr_names = { { DEF4(clr) } }; +const atomic_ool_names aarch64_ool_steor_names = { { DEF4(eor) } }; + +#undef DEF0 +#undef DEF4 + /* Expand a compare and swap pattern. */ void @@ -14206,6 +14290,17 @@ aarch64_expand_compare_and_swap (rtx operands[]) newval, mod_s)); cc_reg = aarch64_gen_compare_reg_maybe_ze (NE, rval, oldval, mode); } + else if (TARGET_ATOMIC_OOL) + { + /* Oldval must satisfy compare afterward. */ + if (!aarch64_plus_operand (oldval, mode)) + oldval = force_reg (mode, oldval); + rtx func = aarch64_atomic_ool_func (mode, mod_s, &aarch64_ool_cas_names); + rval = emit_library_call_value (func, NULL_RTX, LCT_NORMAL, r_mode, + oldval, mode, newval, mode, + XEXP (mem, 0), ptr_mode); + cc_reg = aarch64_gen_compare_reg_maybe_ze (NE, rval, oldval, mode); + } else { /* The oldval predicate varies by mode. Test it and force to reg. */ diff --git a/gcc/testsuite/gcc.target/aarch64/atomic-comp-swap-release-acquire.c b/gcc/testsuite/gcc.target/aarch64/atomic-comp-swap-release-acquire.c index 49ca5d0d09c..e92f205c3a8 100644 --- a/gcc/testsuite/gcc.target/aarch64/atomic-comp-swap-release-acquire.c +++ b/gcc/testsuite/gcc.target/aarch64/atomic-comp-swap-release-acquire.c @@ -1,5 +1,5 @@ /* { dg-do compile } */ -/* { dg-options "-march=armv8-a+nolse -O2 -fno-ipa-icf" } */ +/* { dg-options "-march=armv8-a+nolse -O2 -fno-ipa-icf -mno-atomic-ool" } */ #include "atomic-comp-swap-release-acquire.x" diff --git a/gcc/testsuite/gcc.target/aarch64/atomic-op-acq_rel.c b/gcc/testsuite/gcc.target/aarch64/atomic-op-acq_rel.c index 74f26348e42..6965431f7d9 100644 --- a/gcc/testsuite/gcc.target/aarch64/atomic-op-acq_rel.c +++ b/gcc/testsuite/gcc.target/aarch64/atomic-op-acq_rel.c @@ -1,5 +1,5 @@ /* { dg-do compile } */ -/* { dg-options "-march=armv8-a+nolse -O2" } */ +/* { dg-options "-march=armv8-a+nolse -O2 -mno-atomic-ool" } */ #include "atomic-op-acq_rel.x" diff --git a/gcc/testsuite/gcc.target/aarch64/atomic-op-acquire.c b/gcc/testsuite/gcc.target/aarch64/atomic-op-acquire.c index 66c1b1efe20..07dbca49d56 100644 --- a/gcc/testsuite/gcc.target/aarch64/atomic-op-acquire.c +++ b/gcc/testsuite/gcc.target/aarch64/atomic-op-acquire.c @@ -1,5 +1,5 @@ /* { dg-do compile } */ -/* { dg-options "-march=armv8-a+nolse -O2" } */ +/* { dg-options "-march=armv8-a+nolse -O2 -mno-atomic-ool" } */ #include "atomic-op-acquire.x" diff --git a/gcc/testsuite/gcc.target/aarch64/atomic-op-char.c b/gcc/testsuite/gcc.target/aarch64/atomic-op-char.c index c09d0434ecf..73bfbb7afc9 100644 --- a/gcc/testsuite/gcc.target/aarch64/atomic-op-char.c +++ b/gcc/testsuite/gcc.target/aarch64/atomic-op-char.c @@ -1,5 +1,5 @@ /* { dg-do compile } */ -/* { dg-options "-march=armv8-a+nolse -O2" } */ +/* { dg-options "-march=armv8-a+nolse -O2 -mno-atomic-ool" } */ #include "atomic-op-char.x" diff --git a/gcc/testsuite/gcc.target/aarch64/atomic-op-consume.c b/gcc/testsuite/gcc.target/aarch64/atomic-op-consume.c index 5783ab84f5c..c7945b3a22d 100644 --- a/gcc/testsuite/gcc.target/aarch64/atomic-op-consume.c +++ b/gcc/testsuite/gcc.target/aarch64/atomic-op-consume.c @@ -1,5 +1,5 @@ /* { dg-do compile } */ -/* { dg-options "-march=armv8-a+nolse -O2" } */ +/* { dg-options "-march=armv8-a+nolse -O2 -mno-atomic-ool" } */ #include "atomic-op-consume.x" diff --git a/gcc/testsuite/gcc.target/aarch64/atomic-op-imm.c b/gcc/testsuite/gcc.target/aarch64/atomic-op-imm.c index 18b8f0b04e9..e46bb3de7c1 100644 --- a/gcc/testsuite/gcc.target/aarch64/atomic-op-imm.c +++ b/gcc/testsuite/gcc.target/aarch64/atomic-op-imm.c @@ -1,5 +1,5 @@ /* { dg-do compile } */ -/* { dg-options "-march=armv8-a+nolse -O2" } */ +/* { dg-options "-march=armv8-a+nolse -O2 -mno-atomic-ool" } */ int v = 0; diff --git a/gcc/testsuite/gcc.target/aarch64/atomic-op-int.c b/gcc/testsuite/gcc.target/aarch64/atomic-op-int.c index 8520f0839ba..9b55deb5225 100644 --- a/gcc/testsuite/gcc.target/aarch64/atomic-op-int.c +++ b/gcc/testsuite/gcc.target/aarch64/atomic-op-int.c @@ -1,5 +1,5 @@ /* { dg-do compile } */ -/* { dg-options "-march=armv8-a+nolse -O2" } */ +/* { dg-options "-march=armv8-a+nolse -O2 -mno-atomic-ool" } */ #include "atomic-op-int.x" diff --git a/gcc/testsuite/gcc.target/aarch64/atomic-op-long.c b/gcc/testsuite/gcc.target/aarch64/atomic-op-long.c index d011f8c5ce2..2622f75331f 100644 --- a/gcc/testsuite/gcc.target/aarch64/atomic-op-long.c +++ b/gcc/testsuite/gcc.target/aarch64/atomic-op-long.c @@ -1,5 +1,5 @@ /* { dg-do compile } */ -/* { dg-options "-march=armv8-a+nolse -O2" } */ +/* { dg-options "-march=armv8-a+nolse -O2 -mno-atomic-ool" } */ long v = 0; diff --git a/gcc/testsuite/gcc.target/aarch64/atomic-op-relaxed.c b/gcc/testsuite/gcc.target/aarch64/atomic-op-relaxed.c index ed96bfdb978..f118a37a352 100644 --- a/gcc/testsuite/gcc.target/aarch64/atomic-op-relaxed.c +++ b/gcc/testsuite/gcc.target/aarch64/atomic-op-relaxed.c @@ -1,5 +1,5 @@ /* { dg-do compile } */ -/* { dg-options "-march=armv8-a+nolse -O2" } */ +/* { dg-options "-march=armv8-a+nolse -O2 -mno-atomic-ool" } */ #include "atomic-op-relaxed.x" diff --git a/gcc/testsuite/gcc.target/aarch64/atomic-op-release.c b/gcc/testsuite/gcc.target/aarch64/atomic-op-release.c index fc4be17de89..579634b08e8 100644 --- a/gcc/testsuite/gcc.target/aarch64/atomic-op-release.c +++ b/gcc/testsuite/gcc.target/aarch64/atomic-op-release.c @@ -1,5 +1,5 @@ /* { dg-do compile } */ -/* { dg-options "-march=armv8-a+nolse -O2" } */ +/* { dg-options "-march=armv8-a+nolse -O2 -mno-atomic-ool" } */ #include "atomic-op-release.x" diff --git a/gcc/testsuite/gcc.target/aarch64/atomic-op-seq_cst.c b/gcc/testsuite/gcc.target/aarch64/atomic-op-seq_cst.c index 613000fe490..016b0d6619f 100644 --- a/gcc/testsuite/gcc.target/aarch64/atomic-op-seq_cst.c +++ b/gcc/testsuite/gcc.target/aarch64/atomic-op-seq_cst.c @@ -1,5 +1,5 @@ /* { dg-do compile } */ -/* { dg-options "-march=armv8-a+nolse -O2" } */ +/* { dg-options "-march=armv8-a+nolse -O2 -mno-atomic-ool" } */ #include "atomic-op-seq_cst.x" diff --git a/gcc/testsuite/gcc.target/aarch64/atomic-op-short.c b/gcc/testsuite/gcc.target/aarch64/atomic-op-short.c index e82c8118ece..978bd1d8377 100644 --- a/gcc/testsuite/gcc.target/aarch64/atomic-op-short.c +++ b/gcc/testsuite/gcc.target/aarch64/atomic-op-short.c @@ -1,5 +1,5 @@ /* { dg-do compile } */ -/* { dg-options "-march=armv8-a+nolse -O2" } */ +/* { dg-options "-march=armv8-a+nolse -O2 -mno-atomic-ool" } */ #include "atomic-op-short.x" diff --git a/gcc/testsuite/gcc.target/aarch64/atomic_cmp_exchange_zero_reg_1.c b/gcc/testsuite/gcc.target/aarch64/atomic_cmp_exchange_zero_reg_1.c index f2a21ddf2e1..77430ecdbce 100644 --- a/gcc/testsuite/gcc.target/aarch64/atomic_cmp_exchange_zero_reg_1.c +++ b/gcc/testsuite/gcc.target/aarch64/atomic_cmp_exchange_zero_reg_1.c @@ -1,5 +1,5 @@ /* { dg-do compile } */ -/* { dg-options "-O2 -march=armv8-a+nolse" } */ +/* { dg-options "-O2 -march=armv8-a+nolse -mno-atomic-ool" } */ /* { dg-skip-if "" { *-*-* } { "-mcpu=*" } { "" } } */ int diff --git a/gcc/testsuite/gcc.target/aarch64/atomic_cmp_exchange_zero_strong_1.c b/gcc/testsuite/gcc.target/aarch64/atomic_cmp_exchange_zero_strong_1.c index 8d2ae67dfbe..7d58b2f6bd0 100644 --- a/gcc/testsuite/gcc.target/aarch64/atomic_cmp_exchange_zero_strong_1.c +++ b/gcc/testsuite/gcc.target/aarch64/atomic_cmp_exchange_zero_strong_1.c @@ -1,5 +1,5 @@ /* { dg-do compile } */ -/* { dg-options "-O2 -march=armv8-a+nolse" } */ +/* { dg-options "-O2 -march=armv8-a+nolse -mno-atomic-ool" } */ /* { dg-skip-if "" { *-*-* } { "-mcpu=*" } { "" } } */ int diff --git a/gcc/testsuite/gcc.target/aarch64/sync-comp-swap.c b/gcc/testsuite/gcc.target/aarch64/sync-comp-swap.c index e571b2f13b3..7fc5885d0fd 100644 --- a/gcc/testsuite/gcc.target/aarch64/sync-comp-swap.c +++ b/gcc/testsuite/gcc.target/aarch64/sync-comp-swap.c @@ -1,5 +1,5 @@ /* { dg-do compile } */ -/* { dg-options "-march=armv8-a+nolse -O2 -fno-ipa-icf" } */ +/* { dg-options "-march=armv8-a+nolse -O2 -fno-ipa-icf -mno-atomic-ool" } */ #include "sync-comp-swap.x" diff --git a/gcc/testsuite/gcc.target/aarch64/sync-op-acquire.c b/gcc/testsuite/gcc.target/aarch64/sync-op-acquire.c index 357bf1be3b2..6ad0daa8998 100644 --- a/gcc/testsuite/gcc.target/aarch64/sync-op-acquire.c +++ b/gcc/testsuite/gcc.target/aarch64/sync-op-acquire.c @@ -1,5 +1,5 @@ /* { dg-do compile } */ -/* { dg-options "-march=armv8-a+nolse -O2" } */ +/* { dg-options "-march=armv8-a+nolse -O2 -mno-atomic-ool" } */ #include "sync-op-acquire.x" diff --git a/gcc/testsuite/gcc.target/aarch64/sync-op-full.c b/gcc/testsuite/gcc.target/aarch64/sync-op-full.c index c6ba1629965..9a7afeb70d3 100644 --- a/gcc/testsuite/gcc.target/aarch64/sync-op-full.c +++ b/gcc/testsuite/gcc.target/aarch64/sync-op-full.c @@ -1,5 +1,5 @@ /* { dg-do compile } */ -/* { dg-options "-march=armv8-a+nolse -O2" } */ +/* { dg-options "-march=armv8-a+nolse -O2 -mno-atomic-ool" } */ #include "sync-op-full.x" diff --git a/gcc/config/aarch64/aarch64.opt b/gcc/config/aarch64/aarch64.opt index b2e80cbf6f1..83166834165 100644 --- a/gcc/config/aarch64/aarch64.opt +++ b/gcc/config/aarch64/aarch64.opt @@ -218,3 +218,7 @@ Enables verbose cost model dumping in the debug dump files. mtrack-speculation Target Var(aarch64_track_speculation) Generate code to track when the CPU might be speculating incorrectly. + +matomic-ool +Target Report Mask(ATOMIC_OOL) Save +Generate local calls to out-of-line atomic operations. diff --git a/gcc/config/aarch64/atomics.md b/gcc/config/aarch64/atomics.md index ee662a4480e..72f9962fe55 100644 --- a/gcc/config/aarch64/atomics.md +++ b/gcc/config/aarch64/atomics.md @@ -140,16 +140,27 @@ (match_operand:SI 3 "const_int_operand" "")] "" { - rtx (*gen) (rtx, rtx, rtx, rtx); - /* Use an atomic SWP when available. */ if (TARGET_LSE) - gen = gen_aarch64_atomic_exchange_lse; + { + emit_insn (gen_aarch64_atomic_exchange_lse + (operands[0], operands[1], operands[2], operands[3])); + } + else if (TARGET_ATOMIC_OOL) + { + machine_mode mode = mode; + rtx func = aarch64_atomic_ool_func (mode, operands[3], + &aarch64_ool_swp_names); + rtx rval = emit_library_call_value (func, operands[0], LCT_NORMAL, + mode, operands[2], mode, + XEXP (operands[1], 0), ptr_mode); + emit_move_insn (operands[0], rval); + } else - gen = gen_aarch64_atomic_exchange; - - emit_insn (gen (operands[0], operands[1], operands[2], operands[3])); - + { + emit_insn (gen_aarch64_atomic_exchange + (operands[0], operands[1], operands[2], operands[3])); + } DONE; } ) @@ -234,6 +245,39 @@ } operands[1] = force_reg (mode, operands[1]); } + else if (TARGET_ATOMIC_OOL) + { + const atomic_ool_names *names; + switch () + { + case MINUS: + operands[1] = expand_simple_unop (mode, NEG, operands[1], + NULL, 1); + /* fallthru */ + case PLUS: + names = &aarch64_ool_stadd_names; + break; + case IOR: + names = &aarch64_ool_stset_names; + break; + case XOR: + names = &aarch64_ool_steor_names; + break; + case AND: + operands[1] = expand_simple_unop (mode, NOT, operands[1], + NULL, 1); + names = &aarch64_ool_stclr_names; + break; + default: + gcc_unreachable (); + } + machine_mode mode = mode; + rtx func = aarch64_atomic_ool_func (mode, operands[2], names); + emit_library_call_value (func, NULL_RTX, LCT_NORMAL, mode, + operands[1], mode, + XEXP (operands[0], 0), ptr_mode); + DONE; + } else gen = gen_aarch64_atomic_; @@ -350,6 +394,40 @@ } operands[2] = force_reg (mode, operands[2]); } + else if (TARGET_ATOMIC_OOL) + { + const atomic_ool_names *names; + switch () + { + case MINUS: + operands[2] = expand_simple_unop (mode, NEG, operands[2], + NULL, 1); + /* fallthru */ + case PLUS: + names = &aarch64_ool_ldadd_names; + break; + case IOR: + names = &aarch64_ool_ldset_names; + break; + case XOR: + names = &aarch64_ool_ldeor_names; + break; + case AND: + operands[2] = expand_simple_unop (mode, NOT, operands[2], + NULL, 1); + names = &aarch64_ool_ldclr_names; + break; + default: + gcc_unreachable (); + } + machine_mode mode = mode; + rtx func = aarch64_atomic_ool_func (mode, operands[3], names); + rtx rval = emit_library_call_value (func, operands[0], LCT_NORMAL, mode, + operands[2], mode, + XEXP (operands[1], 0), ptr_mode); + emit_move_insn (operands[0], rval); + DONE; + } else gen = gen_aarch64_atomic_fetch_; @@ -439,7 +517,7 @@ { /* Use an atomic load-operate instruction when possible. In this case we will re-compute the result from the original mem value. */ - if (TARGET_LSE) + if (TARGET_LSE || TARGET_ATOMIC_OOL) { rtx tmp = gen_reg_rtx (mode); operands[2] = force_reg (mode, operands[2]); diff --git a/gcc/doc/invoke.texi b/gcc/doc/invoke.texi index 5c95f674e0c..c947abea899 100644 --- a/gcc/doc/invoke.texi +++ b/gcc/doc/invoke.texi @@ -621,7 +621,7 @@ Objective-C and Objective-C++ Dialects}. -mpc-relative-literal-loads @gol -msign-return-address=@var{scope} @gol -march=@var{name} -mcpu=@var{name} -mtune=@var{name} @gol --moverride=@var{string} -mverbose-cost-dump -mtrack-speculation} +-moverride=@var{string} -mverbose-cost-dump -mtrack-speculation -matomic-ool} @emph{Adapteva Epiphany Options} @gccoptlist{-mhalf-reg-file -mprefer-short-insn-regs @gol @@ -15057,6 +15057,18 @@ be used by the compiler when expanding calls to @code{__builtin_speculation_safe_copy} to permit a more efficient code sequence to be generated. +@item -matomic-ool +@itemx -mno-atomic-ool +Enable or disable calls to out-of-line helpers to implement atomic operations. +These helpers will, at runtime, determine if ARMv8.1-Atomics instructions +should be used; if not, they will use the load/store-exclusive instructions +that are present in the base ARMv8.0 ISA. + +This option is only applicable when compiling for the base ARMv8.0 +instruction set. If using a later revision, e.g. @option{-march=armv8.1-a} +or @option{-march=armv8-a+lse}, the ARMv8.1-Atomics instructions will be +used directly. + @item -march=@var{name} @opindex march Specify the name of the target architecture and, optionally, one or