From patchwork Tue Oct 2 16:19:05 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 147984 Delivered-To: patch@linaro.org Received: by 2002:a2e:8595:0:0:0:0:0 with SMTP id b21-v6csp5354314lji; Tue, 2 Oct 2018 09:19:50 -0700 (PDT) X-Google-Smtp-Source: ACcGV63veFQIGDZvYrnjH9wPlLiT6jT/K5Tr4CIaSmymsYE9cadRhB0/iDNOcz3VY6rGHGTQaZ10 X-Received: by 2002:a17:902:d704:: with SMTP id w4-v6mr17840880ply.230.1538497190352; Tue, 02 Oct 2018 09:19:50 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1538497190; cv=none; d=google.com; s=arc-20160816; b=J0uxHnNd7SONYSqOL8OGPHdytdIkoSimGGaV4cmezH1FDjl1h5XcEBxcjyKsr66WNY SV1RAuoIvvM8OhnQ1IkSSkXVki4mL8epeficemKZEWHlrkjcJ2PCbVXdz0CGefEnQFXO j9nSzeuerrBDQRPAVsCrlzWTc+2760xi2kUHHnqtE10cz1/zruEfvnTtPKnYOvkQZfVA qG0wrvwob8MGc51j4Acscy3xvNEOGc3v8alsUWTEOyZxRlfPLXAqR+cVaj9AGjkoxClA o1yRNFb3Xbox6JhLsmfzL68r5/fbCKh+fkkwzSvi7cf/iHQXh0Xz9y99H7jp39BvoiKw 8EQA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature:delivered-to:sender:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:mailing-list:dkim-signature :domainkey-signature; bh=7ovK77TSYJGf3pRLzQpaYFzrirzAS1Ji6nEOngnv1Ro=; b=dRyMgMZPa1hVDSImIh5wB0t5qAl4COvZOnHG8jSrxLoMB9jN8l0RXhAnXhs5yGiW/f RA9v50SKim8vw9204BDzoacDsRVaJIx6UdUEF52spKZ+TMuyhWdCXrZcgovt6O0pqRAu lQ3C5CG/8zxyOOUiTr535udZ/Mjh5R1icNT4u/fV6T1+47zs3Y+NzHuL6TlW1DgMJa53 I2xw9bSfhysswWjvQqJKKT5XYffss26XmGchFqD9fHNBJ8K8yvxUWR55HQ7+O04w3C3z +R0mYuEiSpnUe+51Tlltws0EToff8waJSFwoLEAmU8B0R6aidLE9Pb4FjWp8CWxSd0Lu eK6w== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gcc.gnu.org header.s=default header.b=aYzOKClG; dkim=pass header.i=@linaro.org header.s=google header.b=de6PGty+; spf=pass (google.com: domain of gcc-patches-return-486816-patch=linaro.org@gcc.gnu.org designates 209.132.180.131 as permitted sender) smtp.mailfrom="gcc-patches-return-486816-patch=linaro.org@gcc.gnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from sourceware.org (server1.sourceware.org. [209.132.180.131]) by mx.google.com with ESMTPS id s7-v6si15717362pfm.217.2018.10.02.09.19.49 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 02 Oct 2018 09:19:50 -0700 (PDT) Received-SPF: pass (google.com: domain of gcc-patches-return-486816-patch=linaro.org@gcc.gnu.org designates 209.132.180.131 as permitted sender) client-ip=209.132.180.131; Authentication-Results: mx.google.com; dkim=pass header.i=@gcc.gnu.org header.s=default header.b=aYzOKClG; dkim=pass header.i=@linaro.org header.s=google header.b=de6PGty+; spf=pass (google.com: domain of gcc-patches-return-486816-patch=linaro.org@gcc.gnu.org designates 209.132.180.131 as permitted sender) smtp.mailfrom="gcc-patches-return-486816-patch=linaro.org@gcc.gnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org DomainKey-Signature: a=rsa-sha1; c=nofws; d=gcc.gnu.org; h=list-id :list-unsubscribe:list-archive:list-post:list-help:sender:from :to:cc:subject:date:message-id:in-reply-to:references; q=dns; s= default; b=BLYeZHbtI+qsW+Ig54AeBNJGv6Nn3C4PSrGCKBmRQeyUzPhZQhPuv f6JptvT1Q1uYxZSq0ZbKL+NtWh8mYWPOLqPbacXZiSM4mpWi1zpts21XdcWHX7NO Mdiij9tjBM5nvHWW9MSi2S/vR188ZqKWvD+lOQdCIIZfo145f1YqSg= DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=gcc.gnu.org; h=list-id :list-unsubscribe:list-archive:list-post:list-help:sender:from :to:cc:subject:date:message-id:in-reply-to:references; s= default; bh=UOt61rVjpURIzt1oQyqT35R8i1Y=; b=aYzOKClG68snApYSEa75 qakY5HrVja9kAiAR4pB8hlGOBIpiyEz0D1xP6dKI/pmYNkz4k1Dw19nvO1AImRkq g2llDOLyfo10MiINs7IwB3b2hy4MnMxa4WwUuAH7wYyIqli0U7Q5D1GvCJzdumvt HgMjk9KXc/B5WlIVEzUlIXg= Received: (qmail 66289 invoked by alias); 2 Oct 2018 16:19:26 -0000 Mailing-List: contact gcc-patches-help@gcc.gnu.org; run by ezmlm Precedence: bulk List-Id: List-Unsubscribe: List-Archive: List-Post: List-Help: Sender: gcc-patches-owner@gcc.gnu.org Delivered-To: mailing list gcc-patches@gcc.gnu.org Received: (qmail 66104 invoked by uid 89); 2 Oct 2018 16:19:25 -0000 Authentication-Results: sourceware.org; auth=none X-Spam-SWARE-Status: No, score=-26.9 required=5.0 tests=BAYES_00, GIT_PATCH_0, GIT_PATCH_1, GIT_PATCH_2, GIT_PATCH_3, RCVD_IN_DNSWL_NONE, SPF_PASS autolearn=ham version=3.3.2 spammy=CAS, alli, cas X-HELO: mail-ot1-f51.google.com Received: from mail-ot1-f51.google.com (HELO mail-ot1-f51.google.com) (209.85.210.51) by sourceware.org (qpsmtpd/0.93/v0.84-503-g423c35a) with ESMTP; Tue, 02 Oct 2018 16:19:22 +0000 Received: by mail-ot1-f51.google.com with SMTP id 36-v6so2450480oth.11 for ; Tue, 02 Oct 2018 09:19:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=7ovK77TSYJGf3pRLzQpaYFzrirzAS1Ji6nEOngnv1Ro=; b=de6PGty+HCslDFt2OcINPAqkQVQ0+6MveeFvWe1h9MSmnB7BuucftMkB8F/iPpQMA6 CRMWw7OpXrua8znGSU9r5fc4sGIP28zJiGPrSsjoIYvaQ/gElLmis+WkvTgj1SKMEHf+ 0L2AifI5w5+A2xUGzDmiirochiKqxLqrGe8qc= Return-Path: Received: from cloudburst.twiddle.net ([187.217.227.243]) by smtp.gmail.com with ESMTPSA id j15-v6sm145987oth.27.2018.10.02.09.19.19 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Tue, 02 Oct 2018 09:19:19 -0700 (PDT) From: Richard Henderson To: gcc-patches@gcc.gnu.org Cc: ramana.radhakrishnan@arm.com, agraf@suse.de, marcus.shawcroft@arm.com, james.greenhalgh@arm.com, richard.earnshaw@arm.com Subject: [PATCH, AArch64 v2 01/11] aarch64: Simplify LSE cas generation Date: Tue, 2 Oct 2018 11:19:05 -0500 Message-Id: <20181002161915.18843-2-richard.henderson@linaro.org> In-Reply-To: <20181002161915.18843-1-richard.henderson@linaro.org> References: <20181002161915.18843-1-richard.henderson@linaro.org> The cas insn is a single insn, and if expanded properly need not be split after reload. Use the proper inputs for the insn. * config/aarch64/aarch64.c (aarch64_expand_compare_and_swap): Force oldval into the rval register for TARGET_LSE; emit the compare during initial expansion so that it may be deleted if unused. (aarch64_gen_atomic_cas): Remove. * config/aarch64/atomics.md (@aarch64_compare_and_swap_lse): Change =&r to +r for operand 0; use match_dup for operand 2; remove is_weak and mod_f operands as unused. Drop the split and merge with... (@aarch64_atomic_cas): ... this pattern's output; remove. (@aarch64_compare_and_swap_lse): Similarly. (@aarch64_atomic_cas): Similarly. --- gcc/config/aarch64/aarch64-protos.h | 1 - gcc/config/aarch64/aarch64.c | 46 ++++------- gcc/config/aarch64/atomics.md | 121 ++++++++-------------------- 3 files changed, 49 insertions(+), 119 deletions(-) -- 2.17.1 diff --git a/gcc/config/aarch64/aarch64-protos.h b/gcc/config/aarch64/aarch64-protos.h index caf1d2041f0..3d045cf43be 100644 --- a/gcc/config/aarch64/aarch64-protos.h +++ b/gcc/config/aarch64/aarch64-protos.h @@ -562,7 +562,6 @@ rtx aarch64_load_tp (rtx); void aarch64_expand_compare_and_swap (rtx op[]); void aarch64_split_compare_and_swap (rtx op[]); -void aarch64_gen_atomic_cas (rtx, rtx, rtx, rtx, rtx); bool aarch64_atomic_ldop_supported_p (enum rtx_code); void aarch64_gen_atomic_ldop (enum rtx_code, rtx, rtx, rtx, rtx, rtx); diff --git a/gcc/config/aarch64/aarch64.c b/gcc/config/aarch64/aarch64.c index 12f7dfe9a75..fbec54fe5da 100644 --- a/gcc/config/aarch64/aarch64.c +++ b/gcc/config/aarch64/aarch64.c @@ -14183,16 +14183,27 @@ aarch64_expand_compare_and_swap (rtx operands[]) } if (TARGET_LSE) - emit_insn (gen_aarch64_compare_and_swap_lse (mode, rval, mem, oldval, - newval, is_weak, mod_s, - mod_f)); + { + /* The CAS insn requires oldval and rval overlap, but we need to + have a copy of oldval saved across the operation to tell if + the operation is successful. */ + if (mode == QImode || mode == HImode) + rval = copy_to_mode_reg (SImode, gen_lowpart (SImode, oldval)); + else if (reg_overlap_mentioned_p (rval, oldval)) + rval = copy_to_mode_reg (mode, oldval); + else + emit_move_insn (rval, oldval); + emit_insn (gen_aarch64_compare_and_swap_lse (mode, rval, mem, + newval, mod_s)); + aarch64_gen_compare_reg (EQ, rval, oldval); + } else emit_insn (gen_aarch64_compare_and_swap (mode, rval, mem, oldval, newval, is_weak, mod_s, mod_f)); - if (mode == QImode || mode == HImode) - emit_move_insn (operands[1], gen_lowpart (mode, rval)); + rval = gen_lowpart (mode, rval); + emit_move_insn (operands[1], rval); x = gen_rtx_REG (CCmode, CC_REGNUM); x = gen_rtx_EQ (SImode, x, const0_rtx); @@ -14242,31 +14253,6 @@ aarch64_emit_post_barrier (enum memmodel model) } } -/* Emit an atomic compare-and-swap operation. RVAL is the destination register - for the data in memory. EXPECTED is the value expected to be in memory. - DESIRED is the value to store to memory. MEM is the memory location. MODEL - is the memory ordering to use. */ - -void -aarch64_gen_atomic_cas (rtx rval, rtx mem, - rtx expected, rtx desired, - rtx model) -{ - machine_mode mode; - - mode = GET_MODE (mem); - - /* Move the expected value into the CAS destination register. */ - emit_insn (gen_rtx_SET (rval, expected)); - - /* Emit the CAS. */ - emit_insn (gen_aarch64_atomic_cas (mode, rval, mem, desired, model)); - - /* Compare the expected value with the value loaded by the CAS, to establish - whether the swap was made. */ - aarch64_gen_compare_reg (EQ, rval, expected); -} - /* Split a compare and swap pattern. */ void diff --git a/gcc/config/aarch64/atomics.md b/gcc/config/aarch64/atomics.md index bba8e9e9c8e..22660850af1 100644 --- a/gcc/config/aarch64/atomics.md +++ b/gcc/config/aarch64/atomics.md @@ -85,56 +85,50 @@ } ) -(define_insn_and_split "@aarch64_compare_and_swap_lse" - [(set (reg:CC CC_REGNUM) ;; bool out - (unspec_volatile:CC [(const_int 0)] UNSPECV_ATOMIC_CMPSW)) - (set (match_operand:SI 0 "register_operand" "=&r") ;; val out +(define_insn "@aarch64_compare_and_swap_lse" + [(set (match_operand:SI 0 "register_operand" "+r") ;; val out (zero_extend:SI - (match_operand:SHORT 1 "aarch64_sync_memory_operand" "+Q"))) ;; memory + (match_operand:SHORT 1 "aarch64_sync_memory_operand" "+Q"))) ;; memory (set (match_dup 1) (unspec_volatile:SHORT - [(match_operand:SI 2 "aarch64_plus_operand" "rI") ;; expected - (match_operand:SHORT 3 "aarch64_reg_or_zero" "rZ") ;; desired - (match_operand:SI 4 "const_int_operand") ;; is_weak - (match_operand:SI 5 "const_int_operand") ;; mod_s - (match_operand:SI 6 "const_int_operand")] ;; mod_f + [(match_dup 0) ;; expected + (match_operand:SHORT 2 "aarch64_reg_or_zero" "rZ") ;; desired + (match_operand:SI 3 "const_int_operand")] ;; mod_s UNSPECV_ATOMIC_CMPSW))] "TARGET_LSE" - "#" - "&& reload_completed" - [(const_int 0)] - { - aarch64_gen_atomic_cas (operands[0], operands[1], - operands[2], operands[3], - operands[5]); - DONE; - } -) +{ + enum memmodel model = memmodel_from_int (INTVAL (operands[3])); + if (is_mm_relaxed (model)) + return "cas\t%0, %2, %1"; + else if (is_mm_acquire (model) || is_mm_consume (model)) + return "casa\t%0, %2, %1"; + else if (is_mm_release (model)) + return "casl\t%0, %2, %1"; + else + return "casal\t%0, %2, %1"; +}) -(define_insn_and_split "@aarch64_compare_and_swap_lse" - [(set (reg:CC CC_REGNUM) ;; bool out - (unspec_volatile:CC [(const_int 0)] UNSPECV_ATOMIC_CMPSW)) - (set (match_operand:GPI 0 "register_operand" "=&r") ;; val out +(define_insn "@aarch64_compare_and_swap_lse" + [(set (match_operand:GPI 0 "register_operand" "+r") ;; val out (match_operand:GPI 1 "aarch64_sync_memory_operand" "+Q")) ;; memory (set (match_dup 1) (unspec_volatile:GPI - [(match_operand:GPI 2 "aarch64_plus_operand" "rI") ;; expect - (match_operand:GPI 3 "aarch64_reg_or_zero" "rZ") ;; desired - (match_operand:SI 4 "const_int_operand") ;; is_weak - (match_operand:SI 5 "const_int_operand") ;; mod_s - (match_operand:SI 6 "const_int_operand")] ;; mod_f + [(match_dup 0) ;; expected + (match_operand:GPI 2 "aarch64_reg_or_zero" "rZ") ;; desired + (match_operand:SI 3 "const_int_operand")] ;; mod_s UNSPECV_ATOMIC_CMPSW))] "TARGET_LSE" - "#" - "&& reload_completed" - [(const_int 0)] - { - aarch64_gen_atomic_cas (operands[0], operands[1], - operands[2], operands[3], - operands[5]); - DONE; - } -) +{ + enum memmodel model = memmodel_from_int (INTVAL (operands[3])); + if (is_mm_relaxed (model)) + return "cas\t%0, %2, %1"; + else if (is_mm_acquire (model) || is_mm_consume (model)) + return "casa\t%0, %2, %1"; + else if (is_mm_release (model)) + return "casl\t%0, %2, %1"; + else + return "casal\t%0, %2, %1"; +}) (define_expand "atomic_exchange" [(match_operand:ALLI 0 "register_operand" "") @@ -610,55 +604,6 @@ return "swpal\t%2, %0, %1"; }) -;; Atomic compare-and-swap: HI and smaller modes. - -(define_insn "@aarch64_atomic_cas" - [(set (match_operand:SI 0 "register_operand" "+&r") ;; out - (zero_extend:SI - (match_operand:SHORT 1 "aarch64_sync_memory_operand" "+Q"))) ;; memory. - (set (match_dup 1) - (unspec_volatile:SHORT - [(match_dup 0) - (match_operand:SHORT 2 "aarch64_reg_or_zero" "rZ") ;; value. - (match_operand:SI 3 "const_int_operand" "")] ;; model. - UNSPECV_ATOMIC_CAS))] - "TARGET_LSE && reload_completed" -{ - enum memmodel model = memmodel_from_int (INTVAL (operands[3])); - if (is_mm_relaxed (model)) - return "cas\t%0, %2, %1"; - else if (is_mm_acquire (model) || is_mm_consume (model)) - return "casa\t%0, %2, %1"; - else if (is_mm_release (model)) - return "casl\t%0, %2, %1"; - else - return "casal\t%0, %2, %1"; -}) - -;; Atomic compare-and-swap: SI and larger modes. - -(define_insn "@aarch64_atomic_cas" - [(set (match_operand:GPI 0 "register_operand" "+&r") ;; out - (match_operand:GPI 1 "aarch64_sync_memory_operand" "+Q")) ;; memory. - (set (match_dup 1) - (unspec_volatile:GPI - [(match_dup 0) - (match_operand:GPI 2 "aarch64_reg_or_zero" "rZ") ;; value. - (match_operand:SI 3 "const_int_operand" "")] ;; model. - UNSPECV_ATOMIC_CAS))] - "TARGET_LSE && reload_completed" -{ - enum memmodel model = memmodel_from_int (INTVAL (operands[3])); - if (is_mm_relaxed (model)) - return "cas\t%0, %2, %1"; - else if (is_mm_acquire (model) || is_mm_consume (model)) - return "casa\t%0, %2, %1"; - else if (is_mm_release (model)) - return "casl\t%0, %2, %1"; - else - return "casal\t%0, %2, %1"; -}) - ;; Atomic load-op: Load data, operate, store result, keep data. (define_insn "@aarch64_atomic_load" From patchwork Tue Oct 2 16:19:06 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 147985 Delivered-To: patch@linaro.org Received: by 2002:a2e:8595:0:0:0:0:0 with SMTP id b21-v6csp5354536lji; Tue, 2 Oct 2018 09:20:03 -0700 (PDT) X-Google-Smtp-Source: ACcGV63o+5lF3HN9SBxRWB5F6OlJjyFycjg7Ba6jIQvKDnQfMH9cxD15NnhN1+DIJeX1Ecd782Fr X-Received: by 2002:a63:ea0e:: with SMTP id c14-v6mr14369507pgi.361.1538497203192; Tue, 02 Oct 2018 09:20:03 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1538497203; cv=none; d=google.com; s=arc-20160816; b=S6YB3PVh/AmFzRfEeL0NwZJEAzxNCQ9Gna9iV6IB2Knx1MgAgHQYl/JRUnwPi77174 zKqkYO5qH2Vz76OR70iGK9Dvu+ULnhczCSmVBmv3PtdTu5sXtzg/+gBt01tsBrYYAevx jMt51vV9sKyszAHqBoIJn3WTB86f1IDPbGHRET2WaiKb1uvclnmAZzXzvjaAx9zsuuM9 QYnkeBisIc6bkG9IpjqvuvskaQcgym5w9MYskhXRfZyRgGVOnTA+0EbKi9u2+OghEWml itmdjaPE4Vi8/d48OkQCwdV0oy7W2+5FKQTYbAxo5HC9eUX/PLbr/rH996OA6w9JFred EXTw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature:delivered-to:sender:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:mailing-list:dkim-signature :domainkey-signature; bh=TGLVG0xVzSAyhLqj33D+0MaZay5dtMHj4iPUBnYkoDI=; b=vSO/s7fc25ZsrH5Z6OC/u08umu2cj4w2EQKfPJMNjFhkXlzM7MHnY/fmnq5Y1uFdT7 DEQk2/nX3B0ctAdOlF1dmZe6a74J/NFSBjw7jlCtf2Eb2f9wMxpmUw6Ieqzh0moQ5wTo JoCLgEFc2Z70+DW6KS9Gal9hSNe3SKXQgAOcp+4MAcz6tCIQXAaO3Yj2vrHtGhq3dkJw Z8IRCNd4VoQs9hI+taDswsFxx0qYN1oSvmrFovOYcGT2CJBRf9ANBsNWZI0gjkdYJa7R 6JK1XN3DZK6DkIqF5uAdInf7wlLxfcz00BmWAG0DAU5wpYf9Zwvoxr/GsqEP2sXgRBJI EKvw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gcc.gnu.org header.s=default header.b=iZ8d+1vj; dkim=pass header.i=@linaro.org header.s=google header.b=CCADoETc; spf=pass (google.com: domain of gcc-patches-return-486817-patch=linaro.org@gcc.gnu.org designates 209.132.180.131 as permitted sender) smtp.mailfrom="gcc-patches-return-486817-patch=linaro.org@gcc.gnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from sourceware.org (server1.sourceware.org. [209.132.180.131]) by mx.google.com with ESMTPS id 12-v6si9341227pfb.268.2018.10.02.09.20.02 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 02 Oct 2018 09:20:03 -0700 (PDT) Received-SPF: pass (google.com: domain of gcc-patches-return-486817-patch=linaro.org@gcc.gnu.org designates 209.132.180.131 as permitted sender) client-ip=209.132.180.131; Authentication-Results: mx.google.com; dkim=pass header.i=@gcc.gnu.org header.s=default header.b=iZ8d+1vj; dkim=pass header.i=@linaro.org header.s=google header.b=CCADoETc; spf=pass (google.com: domain of gcc-patches-return-486817-patch=linaro.org@gcc.gnu.org designates 209.132.180.131 as permitted sender) smtp.mailfrom="gcc-patches-return-486817-patch=linaro.org@gcc.gnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org DomainKey-Signature: a=rsa-sha1; c=nofws; d=gcc.gnu.org; h=list-id :list-unsubscribe:list-archive:list-post:list-help:sender:from :to:cc:subject:date:message-id:in-reply-to:references; q=dns; s= default; b=QS/DyjCFJMfmHDe/Vv188Z/5HKrt02FEX/z3tNISHDukYbfnXM+yD 2iNYqgAx+RnFTRg82vWdAP95aX+YwdsqGhIVzXB/WhJfvcLGpQt31esoQ8vfBiwL EEfVCxXybMI7JK5McxXfXGOW5GETO4L2Kj7NEq6Ovs0rtArhwjnkZQ= DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=gcc.gnu.org; h=list-id :list-unsubscribe:list-archive:list-post:list-help:sender:from :to:cc:subject:date:message-id:in-reply-to:references; s= default; bh=0XsKLcfispSe+a3oXsnPg1ILkGI=; b=iZ8d+1vj+cZnIotHoxl+ tqC2SJPPuIsLGl5oyenBbohCcjr63Tsv/wXI3maJGMul35K3HbcE2ausqF5Ms+O/ ynDrS5rg3i8E1e22NPkqC0RJcylCMU5emXo5y9zD1a9R4MJt+g9vSEyrW9+N5QEo O74a5T90JW8WvQJjPWMNAHs= Received: (qmail 66481 invoked by alias); 2 Oct 2018 16:19:27 -0000 Mailing-List: contact gcc-patches-help@gcc.gnu.org; run by ezmlm Precedence: bulk List-Id: List-Unsubscribe: List-Archive: List-Post: List-Help: Sender: gcc-patches-owner@gcc.gnu.org Delivered-To: mailing list gcc-patches@gcc.gnu.org Received: (qmail 66346 invoked by uid 89); 2 Oct 2018 16:19:27 -0000 Authentication-Results: sourceware.org; auth=none X-Spam-SWARE-Status: No, score=-26.9 required=5.0 tests=BAYES_00, GIT_PATCH_0, GIT_PATCH_1, GIT_PATCH_2, GIT_PATCH_3, RCVD_IN_DNSWL_NONE, SPF_PASS autolearn=ham version=3.3.2 spammy=RELEASE, alli X-HELO: mail-oi1-f182.google.com Received: from mail-oi1-f182.google.com (HELO mail-oi1-f182.google.com) (209.85.167.182) by sourceware.org (qpsmtpd/0.93/v0.84-503-g423c35a) with ESMTP; Tue, 02 Oct 2018 16:19:24 +0000 Received: by mail-oi1-f182.google.com with SMTP id e17-v6so1997998oib.4 for ; Tue, 02 Oct 2018 09:19:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=TGLVG0xVzSAyhLqj33D+0MaZay5dtMHj4iPUBnYkoDI=; b=CCADoETcIeKbLHlTvjKli729AQ/rNP4I8wlyfezU/dFa5DnYJwv21JcErgD/XVIxky cpfOZXMzXWsZjVKEnC4OpiKL4Xovv+u1DiIU3+pFfQ0yfQGHhdg8PVK8DHFurmChoOEy m1PR89/dhdmOtHmzrr1fEZLLqKjm6+S9UVvzw= Return-Path: Received: from cloudburst.twiddle.net ([187.217.227.243]) by smtp.gmail.com with ESMTPSA id j15-v6sm145987oth.27.2018.10.02.09.19.20 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Tue, 02 Oct 2018 09:19:21 -0700 (PDT) From: Richard Henderson To: gcc-patches@gcc.gnu.org Cc: ramana.radhakrishnan@arm.com, agraf@suse.de, marcus.shawcroft@arm.com, james.greenhalgh@arm.com, richard.earnshaw@arm.com Subject: [PATCH, AArch64 v2 02/11] aarch64: Improve cas generation Date: Tue, 2 Oct 2018 11:19:06 -0500 Message-Id: <20181002161915.18843-3-richard.henderson@linaro.org> In-Reply-To: <20181002161915.18843-1-richard.henderson@linaro.org> References: <20181002161915.18843-1-richard.henderson@linaro.org> Do not zero-extend the input to the cas for subword operations; instead, use the appropriate zero-extending compare insns. Correct the predicates and constraints for immediate expected operand. * config/aarch64/aarch64.c (aarch64_gen_compare_reg_maybe_ze): New. (aarch64_split_compare_and_swap): Use it. (aarch64_expand_compare_and_swap): Likewise. Remove convert_modes; test oldval against the proper predicate. * config/aarch64/atomics.md (@atomic_compare_and_swap): Use nonmemory_operand for expected. (cas_short_expected_pred): New. (@aarch64_compare_and_swap): Use it; use "rn" not "rI" to match. (@aarch64_compare_and_swap): Use "rn" not "rI" for expected. * config/aarch64/predicates.md (aarch64_plushi_immediate): New. (aarch64_plushi_operand): New. --- gcc/config/aarch64/aarch64.c | 90 +++++++++++++++++++------------- gcc/config/aarch64/atomics.md | 19 ++++--- gcc/config/aarch64/predicates.md | 12 +++++ 3 files changed, 76 insertions(+), 45 deletions(-) -- 2.17.1 diff --git a/gcc/config/aarch64/aarch64.c b/gcc/config/aarch64/aarch64.c index fbec54fe5da..0e2b85de1e3 100644 --- a/gcc/config/aarch64/aarch64.c +++ b/gcc/config/aarch64/aarch64.c @@ -1613,6 +1613,33 @@ aarch64_gen_compare_reg (RTX_CODE code, rtx x, rtx y) return cc_reg; } +/* Similarly, but maybe zero-extend Y if Y_MODE < SImode. */ + +static rtx +aarch64_gen_compare_reg_maybe_ze(RTX_CODE code, rtx x, rtx y, + machine_mode y_mode) +{ + if (y_mode == E_QImode || y_mode == E_HImode) + { + if (CONST_INT_P (y)) + y = GEN_INT (INTVAL (y) & GET_MODE_MASK (y_mode)); + else + { + rtx t, cc_reg; + machine_mode cc_mode; + + t = gen_rtx_ZERO_EXTEND (SImode, y); + t = gen_rtx_COMPARE (CC_SWPmode, t, x); + cc_mode = CC_SWPmode; + cc_reg = gen_rtx_REG (cc_mode, CC_REGNUM); + emit_set_insn (cc_reg, t); + return cc_reg; + } + } + + return aarch64_gen_compare_reg (code, x, y); +} + /* Build the SYMBOL_REF for __tls_get_addr. */ static GTY(()) rtx tls_get_addr_libfunc; @@ -14138,8 +14165,8 @@ aarch64_emit_unlikely_jump (rtx insn) void aarch64_expand_compare_and_swap (rtx operands[]) { - rtx bval, rval, mem, oldval, newval, is_weak, mod_s, mod_f, x; - machine_mode mode, cmp_mode; + rtx bval, rval, mem, oldval, newval, is_weak, mod_s, mod_f, x, cc_reg; + machine_mode mode, r_mode; bval = operands[0]; rval = operands[1]; @@ -14150,36 +14177,19 @@ aarch64_expand_compare_and_swap (rtx operands[]) mod_s = operands[6]; mod_f = operands[7]; mode = GET_MODE (mem); - cmp_mode = mode; /* Normally the succ memory model must be stronger than fail, but in the unlikely event of fail being ACQUIRE and succ being RELEASE we need to promote succ to ACQ_REL so that we don't lose the acquire semantics. */ - if (is_mm_acquire (memmodel_from_int (INTVAL (mod_f))) && is_mm_release (memmodel_from_int (INTVAL (mod_s)))) mod_s = GEN_INT (MEMMODEL_ACQ_REL); - switch (mode) + r_mode = mode; + if (mode == QImode || mode == HImode) { - case E_QImode: - case E_HImode: - /* For short modes, we're going to perform the comparison in SImode, - so do the zero-extension now. */ - cmp_mode = SImode; - rval = gen_reg_rtx (SImode); - oldval = convert_modes (SImode, mode, oldval, true); - /* Fall through. */ - - case E_SImode: - case E_DImode: - /* Force the value into a register if needed. */ - if (!aarch64_plus_operand (oldval, mode)) - oldval = force_reg (cmp_mode, oldval); - break; - - default: - gcc_unreachable (); + r_mode = SImode; + rval = gen_reg_rtx (r_mode); } if (TARGET_LSE) @@ -14187,26 +14197,32 @@ aarch64_expand_compare_and_swap (rtx operands[]) /* The CAS insn requires oldval and rval overlap, but we need to have a copy of oldval saved across the operation to tell if the operation is successful. */ - if (mode == QImode || mode == HImode) - rval = copy_to_mode_reg (SImode, gen_lowpart (SImode, oldval)); - else if (reg_overlap_mentioned_p (rval, oldval)) - rval = copy_to_mode_reg (mode, oldval); - else - emit_move_insn (rval, oldval); + if (reg_overlap_mentioned_p (rval, oldval)) + rval = copy_to_mode_reg (r_mode, oldval); + else + emit_move_insn (rval, gen_lowpart (r_mode, oldval)); + emit_insn (gen_aarch64_compare_and_swap_lse (mode, rval, mem, newval, mod_s)); - aarch64_gen_compare_reg (EQ, rval, oldval); + cc_reg = aarch64_gen_compare_reg_maybe_ze (NE, rval, oldval, mode); } else - emit_insn (gen_aarch64_compare_and_swap (mode, rval, mem, oldval, newval, - is_weak, mod_s, mod_f)); + { + /* The oldval predicate varies by mode. Test it and force to reg. */ + insn_code code = code_for_aarch64_compare_and_swap (mode); + if (!insn_data[code].operand[2].predicate (oldval, mode)) + oldval = force_reg (mode, oldval); - if (mode == QImode || mode == HImode) + emit_insn (GEN_FCN (code) (rval, mem, oldval, newval, + is_weak, mod_s, mod_f)); + cc_reg = gen_rtx_REG (CCmode, CC_REGNUM); + } + + if (r_mode != mode) rval = gen_lowpart (mode, rval); emit_move_insn (operands[1], rval); - x = gen_rtx_REG (CCmode, CC_REGNUM); - x = gen_rtx_EQ (SImode, x, const0_rtx); + x = gen_rtx_EQ (SImode, cc_reg, const0_rtx); emit_insn (gen_rtx_SET (bval, x)); } @@ -14321,10 +14337,10 @@ aarch64_split_compare_and_swap (rtx operands[]) } else { - cond = aarch64_gen_compare_reg (NE, rval, oldval); + cond = aarch64_gen_compare_reg_maybe_ze (NE, rval, oldval, mode); x = gen_rtx_NE (VOIDmode, cond, const0_rtx); x = gen_rtx_IF_THEN_ELSE (VOIDmode, x, - gen_rtx_LABEL_REF (Pmode, label2), pc_rtx); + gen_rtx_LABEL_REF (Pmode, label2), pc_rtx); aarch64_emit_unlikely_jump (gen_rtx_SET (pc_rtx, x)); } diff --git a/gcc/config/aarch64/atomics.md b/gcc/config/aarch64/atomics.md index 22660850af1..e44301b40c7 100644 --- a/gcc/config/aarch64/atomics.md +++ b/gcc/config/aarch64/atomics.md @@ -24,8 +24,8 @@ [(match_operand:SI 0 "register_operand" "") ;; bool out (match_operand:ALLI 1 "register_operand" "") ;; val out (match_operand:ALLI 2 "aarch64_sync_memory_operand" "") ;; memory - (match_operand:ALLI 3 "general_operand" "") ;; expected - (match_operand:ALLI 4 "aarch64_reg_or_zero" "") ;; desired + (match_operand:ALLI 3 "nonmemory_operand" "") ;; expected + (match_operand:ALLI 4 "aarch64_reg_or_zero" "") ;; desired (match_operand:SI 5 "const_int_operand") ;; is_weak (match_operand:SI 6 "const_int_operand") ;; mod_s (match_operand:SI 7 "const_int_operand")] ;; mod_f @@ -36,19 +36,22 @@ } ) +(define_mode_attr cas_short_expected_pred + [(QI "aarch64_reg_or_imm") (HI "aarch64_plushi_operand")]) + (define_insn_and_split "@aarch64_compare_and_swap" [(set (reg:CC CC_REGNUM) ;; bool out (unspec_volatile:CC [(const_int 0)] UNSPECV_ATOMIC_CMPSW)) - (set (match_operand:SI 0 "register_operand" "=&r") ;; val out + (set (match_operand:SI 0 "register_operand" "=&r") ;; val out (zero_extend:SI (match_operand:SHORT 1 "aarch64_sync_memory_operand" "+Q"))) ;; memory (set (match_dup 1) (unspec_volatile:SHORT - [(match_operand:SI 2 "aarch64_plus_operand" "rI") ;; expected + [(match_operand:SHORT 2 "" "rn") ;; expected (match_operand:SHORT 3 "aarch64_reg_or_zero" "rZ") ;; desired - (match_operand:SI 4 "const_int_operand") ;; is_weak - (match_operand:SI 5 "const_int_operand") ;; mod_s - (match_operand:SI 6 "const_int_operand")] ;; mod_f + (match_operand:SI 4 "const_int_operand") ;; is_weak + (match_operand:SI 5 "const_int_operand") ;; mod_s + (match_operand:SI 6 "const_int_operand")] ;; mod_f UNSPECV_ATOMIC_CMPSW)) (clobber (match_scratch:SI 7 "=&r"))] "" @@ -68,7 +71,7 @@ (match_operand:GPI 1 "aarch64_sync_memory_operand" "+Q")) ;; memory (set (match_dup 1) (unspec_volatile:GPI - [(match_operand:GPI 2 "aarch64_plus_operand" "rI") ;; expect + [(match_operand:GPI 2 "aarch64_plus_operand" "rn") ;; expect (match_operand:GPI 3 "aarch64_reg_or_zero" "rZ") ;; desired (match_operand:SI 4 "const_int_operand") ;; is_weak (match_operand:SI 5 "const_int_operand") ;; mod_s diff --git a/gcc/config/aarch64/predicates.md b/gcc/config/aarch64/predicates.md index 5b08b03c586..4c75eff3e5a 100644 --- a/gcc/config/aarch64/predicates.md +++ b/gcc/config/aarch64/predicates.md @@ -114,6 +114,18 @@ (ior (match_operand 0 "register_operand") (match_operand 0 "aarch64_plus_immediate"))) +(define_predicate "aarch64_plushi_immediate" + (match_code "const_int") +{ + HOST_WIDE_INT val = INTVAL (op); + /* The HImode value must be zero-extendable to an SImode plus_operand. */ + return ((val & 0xfff) == val || sext_hwi (val & 0xf000, 16) == val); +}) + +(define_predicate "aarch64_plushi_operand" + (ior (match_operand 0 "register_operand") + (match_operand 0 "aarch64_plushi_immediate"))) + (define_predicate "aarch64_pluslong_immediate" (and (match_code "const_int") (match_test "(INTVAL (op) < 0xffffff && INTVAL (op) > -0xffffff)"))) From patchwork Tue Oct 2 16:19:07 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 147986 Delivered-To: patch@linaro.org Received: by 2002:a2e:8595:0:0:0:0:0 with SMTP id b21-v6csp5354749lji; Tue, 2 Oct 2018 09:20:15 -0700 (PDT) X-Google-Smtp-Source: ACcGV610Rd5Uy/qJ09FsRZO0FzA4VLS8NQ9sG1jUCVGbubYmsoLBd2bvvA6EPfE4R5EgKKG4sUCH X-Received: by 2002:a63:f005:: with SMTP id k5-v6mr15366669pgh.259.1538497215497; Tue, 02 Oct 2018 09:20:15 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1538497215; cv=none; d=google.com; s=arc-20160816; b=ANgRTmdYdhiUcxGUDtMpi93lkg5DioGR4WvdkjSWwkeHjY4vC3Is3qq/KPMLF6DT70 5MwdTWCoj+c1VFrBVu7GwDDh6KlBmGgo39AUCihJPISbjkPWcSwUxfIqWmziW9EN7aqu tvOBz2fRrGWxtiXDPV1PzIeiEAIksIGaoXirrUHRHPnYUyJ56oXYJL8kRj4SKMwu030x neLv/pa3nRMBzGl+yQ0dQmYmZNU5m5/b74bDz/4RuUiAA5Fa3dSzSHXtA/VL7LxrJT1+ 1ceB2SLktjBdkQJ0H5IudDWKb/DFa/y5ztTtvPfazolA5bhaVZkOtWXueiTK7MvzeDnp 0LZQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature:delivered-to:sender:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:mailing-list:dkim-signature :domainkey-signature; bh=aGTDQ/eunUfsLNv/8fKvAW8znwnqcmT6nDVY7wN2D5o=; b=MqIUYZgo1I6sKBUo95urdDCLIkbaACDG9LhPlQVq23syjbhVxwo4x8fUkrzDW42Eaz jjv8KvNnBsKigFZXsoKBcVS2kU39GAkiTbcEhLZH1stPCPKFDZM5OTioaR+PcpRXuSUk wT8pKhZHKe8JivLfixnFEVbuOVfe2uONI2NwEu5gVwf8CZ/xhFc1FMYcl6T1VV893EAU kNGjzR33ugIoktKkl90QCHWmOVBFS8+8pgBHqA0UbklYTsHDnX8TXgGt7MQGdzxjrkpc S+NmpOC3qzzZFchFlSAUP7Ijc+prr4EKIi7QP/xHZLxuzoHphAVT1IEO02Z0TnpwpAZq CwHQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gcc.gnu.org header.s=default header.b=XB8ryt28; dkim=pass header.i=@linaro.org header.s=google header.b="OCnAcCO/"; spf=pass (google.com: domain of gcc-patches-return-486818-patch=linaro.org@gcc.gnu.org designates 209.132.180.131 as permitted sender) smtp.mailfrom="gcc-patches-return-486818-patch=linaro.org@gcc.gnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from sourceware.org (server1.sourceware.org. [209.132.180.131]) by mx.google.com with ESMTPS id e2-v6si17836629pfh.64.2018.10.02.09.20.15 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 02 Oct 2018 09:20:15 -0700 (PDT) Received-SPF: pass (google.com: domain of gcc-patches-return-486818-patch=linaro.org@gcc.gnu.org designates 209.132.180.131 as permitted sender) client-ip=209.132.180.131; Authentication-Results: mx.google.com; dkim=pass header.i=@gcc.gnu.org header.s=default header.b=XB8ryt28; dkim=pass header.i=@linaro.org header.s=google header.b="OCnAcCO/"; spf=pass (google.com: domain of gcc-patches-return-486818-patch=linaro.org@gcc.gnu.org designates 209.132.180.131 as permitted sender) smtp.mailfrom="gcc-patches-return-486818-patch=linaro.org@gcc.gnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org DomainKey-Signature: a=rsa-sha1; c=nofws; d=gcc.gnu.org; h=list-id :list-unsubscribe:list-archive:list-post:list-help:sender:from :to:cc:subject:date:message-id:in-reply-to:references; q=dns; s= default; b=YVg+xIaHzeIWzE9ky7KE5LtTgRACS3FNGk9jvBPd4im5ew1gyfp3C viCsUWIKn4yPOlHqXK0wLKUrozYJxs2pOsC8jg7w5a2v8pblNlgSs9AmVjyTW609 mRq6ixaO8d7l6LyBmVqrwWZj3CFxuxM7bHiNJbOlTqEOvDikA4V7h0= DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=gcc.gnu.org; h=list-id :list-unsubscribe:list-archive:list-post:list-help:sender:from :to:cc:subject:date:message-id:in-reply-to:references; s= default; bh=/IwFuDwKIn6EDFybFNIQm6eTM0A=; b=XB8ryt28mRqXWt51vZE4 s5DGj6Oo4aKIcfxSmmEPnY5tJ82GbxvMM7eNI+SF/lCJnUNoFmpPKpb8fS5/DpE0 3cS2xRZBZwYc9TqjaX1nxu8ImY0R0aO7B/A/a13K+8ytDBVz6irBusHyHSEhTIXY Nxw5hmqQiplVefG0AlltKTQ= Received: (qmail 66704 invoked by alias); 2 Oct 2018 16:19:29 -0000 Mailing-List: contact gcc-patches-help@gcc.gnu.org; run by ezmlm Precedence: bulk List-Id: List-Unsubscribe: List-Archive: List-Post: List-Help: Sender: gcc-patches-owner@gcc.gnu.org Delivered-To: mailing list gcc-patches@gcc.gnu.org Received: (qmail 66600 invoked by uid 89); 2 Oct 2018 16:19:28 -0000 Authentication-Results: sourceware.org; auth=none X-Spam-SWARE-Status: No, score=-26.9 required=5.0 tests=BAYES_00, GIT_PATCH_0, GIT_PATCH_1, GIT_PATCH_2, GIT_PATCH_3, RCVD_IN_DNSWL_NONE, SPF_PASS autolearn=ham version=3.3.2 spammy=PLUS, HX-Received:sk:11-v6mr X-HELO: mail-ot1-f47.google.com Received: from mail-ot1-f47.google.com (HELO mail-ot1-f47.google.com) (209.85.210.47) by sourceware.org (qpsmtpd/0.93/v0.84-503-g423c35a) with ESMTP; Tue, 02 Oct 2018 16:19:26 +0000 Received: by mail-ot1-f47.google.com with SMTP id w67so2464164ota.7 for ; Tue, 02 Oct 2018 09:19:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=aGTDQ/eunUfsLNv/8fKvAW8znwnqcmT6nDVY7wN2D5o=; b=OCnAcCO/x0jjG5slj5m0+S9NZ6iAVJZW7E+6Vx07BOPmQQ3fRzDS6ew8Eu2J9QT9a2 E7CSyfKzxcyXLoAYcPu59MFhzY+zxebh7hNHavM8BRvQN5Wb2L8ENRWGIrUXik5BN4HQ CERFEPkkcxR+lZk+4MT1YS8agwNbQsnZwyiqc= Return-Path: Received: from cloudburst.twiddle.net ([187.217.227.243]) by smtp.gmail.com with ESMTPSA id j15-v6sm145987oth.27.2018.10.02.09.19.22 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Tue, 02 Oct 2018 09:19:23 -0700 (PDT) From: Richard Henderson To: gcc-patches@gcc.gnu.org Cc: ramana.radhakrishnan@arm.com, agraf@suse.de, marcus.shawcroft@arm.com, james.greenhalgh@arm.com, richard.earnshaw@arm.com Subject: [PATCH, AArch64 v2 03/11] aarch64: Improve swp generation Date: Tue, 2 Oct 2018 11:19:07 -0500 Message-Id: <20181002161915.18843-4-richard.henderson@linaro.org> In-Reply-To: <20181002161915.18843-1-richard.henderson@linaro.org> References: <20181002161915.18843-1-richard.henderson@linaro.org> Allow zero as an input; fix constraints; avoid unnecessary split. * config/aarch64/aarch64.c (aarch64_emit_atomic_swap): Remove. (aarch64_gen_atomic_ldop): Don't call it. * config/aarch64/atomics.md (atomic_exchange): Use aarch64_reg_or_zero. (aarch64_atomic_exchange): Likewise. (aarch64_atomic_exchange_lse): Remove split; remove & from operand 0; use aarch64_reg_or_zero for input; merge ... (@aarch64_atomic_swp): ... this and remove. --- gcc/config/aarch64/aarch64.c | 13 ---------- gcc/config/aarch64/atomics.md | 49 +++++++++++------------------------ 2 files changed, 15 insertions(+), 47 deletions(-) -- 2.17.1 diff --git a/gcc/config/aarch64/aarch64.c b/gcc/config/aarch64/aarch64.c index 0e2b85de1e3..f7b0af2589e 100644 --- a/gcc/config/aarch64/aarch64.c +++ b/gcc/config/aarch64/aarch64.c @@ -14403,15 +14403,6 @@ aarch64_emit_bic (machine_mode mode, rtx dst, rtx s1, rtx s2, int shift) emit_insn (gen (dst, s2, shift_rtx, s1)); } -/* Emit an atomic swap. */ - -static void -aarch64_emit_atomic_swap (machine_mode mode, rtx dst, rtx value, - rtx mem, rtx model) -{ - emit_insn (gen_aarch64_atomic_swp (mode, dst, mem, value, model)); -} - /* Emit an atomic load+operate. CODE is the operation. OUT_DATA is the location to store the data read from memory. OUT_RESULT is the location to store the result of the operation. MEM is the memory location to read and @@ -14452,10 +14443,6 @@ aarch64_gen_atomic_ldop (enum rtx_code code, rtx out_data, rtx out_result, a SET then emit a swap instruction and finish. */ switch (code) { - case SET: - aarch64_emit_atomic_swap (mode, out_data, src, mem, model_rtx); - return; - case MINUS: /* Negate the value and treat it as a PLUS. */ { diff --git a/gcc/config/aarch64/atomics.md b/gcc/config/aarch64/atomics.md index e44301b40c7..bc9e396dc96 100644 --- a/gcc/config/aarch64/atomics.md +++ b/gcc/config/aarch64/atomics.md @@ -136,7 +136,7 @@ (define_expand "atomic_exchange" [(match_operand:ALLI 0 "register_operand" "") (match_operand:ALLI 1 "aarch64_sync_memory_operand" "") - (match_operand:ALLI 2 "register_operand" "") + (match_operand:ALLI 2 "aarch64_reg_or_zero" "") (match_operand:SI 3 "const_int_operand" "")] "" { @@ -156,10 +156,10 @@ (define_insn_and_split "aarch64_atomic_exchange" [(set (match_operand:ALLI 0 "register_operand" "=&r") ;; output - (match_operand:ALLI 1 "aarch64_sync_memory_operand" "+Q")) ;; memory + (match_operand:ALLI 1 "aarch64_sync_memory_operand" "+Q")) ;; memory (set (match_dup 1) (unspec_volatile:ALLI - [(match_operand:ALLI 2 "register_operand" "r") ;; input + [(match_operand:ALLI 2 "aarch64_reg_or_zero" "rZ") ;; input (match_operand:SI 3 "const_int_operand" "")] ;; model UNSPECV_ATOMIC_EXCHG)) (clobber (reg:CC CC_REGNUM)) @@ -175,22 +175,25 @@ } ) -(define_insn_and_split "aarch64_atomic_exchange_lse" - [(set (match_operand:ALLI 0 "register_operand" "=&r") +(define_insn "aarch64_atomic_exchange_lse" + [(set (match_operand:ALLI 0 "register_operand" "=r") (match_operand:ALLI 1 "aarch64_sync_memory_operand" "+Q")) (set (match_dup 1) (unspec_volatile:ALLI - [(match_operand:ALLI 2 "register_operand" "r") + [(match_operand:ALLI 2 "aarch64_reg_or_zero" "rZ") (match_operand:SI 3 "const_int_operand" "")] UNSPECV_ATOMIC_EXCHG))] "TARGET_LSE" - "#" - "&& reload_completed" - [(const_int 0)] { - aarch64_gen_atomic_ldop (SET, operands[0], NULL, operands[1], - operands[2], operands[3]); - DONE; + enum memmodel model = memmodel_from_int (INTVAL (operands[3])); + if (is_mm_relaxed (model)) + return "swp\t%2, %0, %1"; + else if (is_mm_acquire (model) || is_mm_consume (model)) + return "swpa\t%2, %0, %1"; + else if (is_mm_release (model)) + return "swpl\t%2, %0, %1"; + else + return "swpal\t%2, %0, %1"; } ) @@ -585,28 +588,6 @@ ;; ARMv8.1-A LSE instructions. -;; Atomic swap with memory. -(define_insn "@aarch64_atomic_swp" - [(set (match_operand:ALLI 0 "register_operand" "+&r") - (match_operand:ALLI 1 "aarch64_sync_memory_operand" "+Q")) - (set (match_dup 1) - (unspec_volatile:ALLI - [(match_operand:ALLI 2 "register_operand" "r") - (match_operand:SI 3 "const_int_operand" "")] - UNSPECV_ATOMIC_SWP))] - "TARGET_LSE && reload_completed" - { - enum memmodel model = memmodel_from_int (INTVAL (operands[3])); - if (is_mm_relaxed (model)) - return "swp\t%2, %0, %1"; - else if (is_mm_acquire (model) || is_mm_consume (model)) - return "swpa\t%2, %0, %1"; - else if (is_mm_release (model)) - return "swpl\t%2, %0, %1"; - else - return "swpal\t%2, %0, %1"; - }) - ;; Atomic load-op: Load data, operate, store result, keep data. (define_insn "@aarch64_atomic_load" From patchwork Tue Oct 2 16:19:08 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 147987 Delivered-To: patch@linaro.org Received: by 2002:a2e:8595:0:0:0:0:0 with SMTP id b21-v6csp5354981lji; Tue, 2 Oct 2018 09:20:28 -0700 (PDT) X-Google-Smtp-Source: ACcGV62+ydhtZS3yZlDYH7yCpdXo6DxqWfJgrdrXLpw1gOAlG0JYyrTb+7gdwjMhFexoQQVljNLD X-Received: by 2002:a62:6643:: with SMTP id a64-v6mr16977102pfc.202.1538497228638; Tue, 02 Oct 2018 09:20:28 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1538497228; cv=none; d=google.com; s=arc-20160816; b=zOo1odAFoQMfAJZ37cBL5tcoGt4hXykUXTta7urnP1Knem5Cc4Z5WKoNR5wMmPoRcb v0zFHRtFhw9RCy62jDRBMqmUIpT5LKCvZSmjc550FJNtDNypeGd48aYwINGNieobL88t NDivqGacqXRFhUH1WZducuOoUPOfqtLB9gBgkdH34RB3NcfpqfYsiCslzRLGIZX7KNCI K1Cn/d9IwScYhQZd+suNh+avin/OvGMnnNyCsmfVXANoLmINo6tTzBXFhg7kfa4eHcxL XUiQmkqB2QkYCtxw6Eu75LoTJ3lIDGKxAnAAULqcF5DzwdH3rrRyTgpdDNszfBv1toMC /N2w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature:delivered-to:sender:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:mailing-list:dkim-signature :domainkey-signature; bh=uIlZ6fYjn8jCEkMixj+KWY283sMiPUK0QgLmt+36YgI=; b=MRHxF00dwhfl5GI1gcuYtM++IaBMx5ZUZaCxgGvaYDjvQaY141S87fYAXejwHAtIdR IrVT2O4iPMoyrd9FqHChCwcrcxluC7Vkmk3E2h6t6YMQQSYIZ2pi3mRxSopVyyJJZeBQ grMxHit+i2ynqf5nNxM/XYIhm5VHFxinyrofA9h/11ZxhhqHCPzOGK1bjK+4mhAxAvFp mC+VXXyBXxkWUYtki3AxBNXMXG8y/wtLxuAVvRLvOO5yR4wdMU6+QO1/D1XL341RJVOB 7c7uUyJ4GLOZI+E8L33kp8vT8rVcto2F9doPqpEpDkvznKb5oJA1o9erPRPHx5Ep+qot +KMg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gcc.gnu.org header.s=default header.b=axond+x5; dkim=pass header.i=@linaro.org header.s=google header.b=f2vPBUkq; spf=pass (google.com: domain of gcc-patches-return-486819-patch=linaro.org@gcc.gnu.org designates 209.132.180.131 as permitted sender) smtp.mailfrom="gcc-patches-return-486819-patch=linaro.org@gcc.gnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from sourceware.org (server1.sourceware.org. [209.132.180.131]) by mx.google.com with ESMTPS id j7-v6si15890922pfk.203.2018.10.02.09.20.28 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 02 Oct 2018 09:20:28 -0700 (PDT) Received-SPF: pass (google.com: domain of gcc-patches-return-486819-patch=linaro.org@gcc.gnu.org designates 209.132.180.131 as permitted sender) client-ip=209.132.180.131; Authentication-Results: mx.google.com; dkim=pass header.i=@gcc.gnu.org header.s=default header.b=axond+x5; dkim=pass header.i=@linaro.org header.s=google header.b=f2vPBUkq; spf=pass (google.com: domain of gcc-patches-return-486819-patch=linaro.org@gcc.gnu.org designates 209.132.180.131 as permitted sender) smtp.mailfrom="gcc-patches-return-486819-patch=linaro.org@gcc.gnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org DomainKey-Signature: a=rsa-sha1; c=nofws; d=gcc.gnu.org; h=list-id :list-unsubscribe:list-archive:list-post:list-help:sender:from :to:cc:subject:date:message-id:in-reply-to:references; q=dns; s= default; b=ETZAvvJGa4cdQFcwEWFB4Mesf5U9z+/SGO0wkA3z58PIEy1pdST08 dLtOtWam3DQTUo9V/nQgI/MGFscu8QaXETUNeX62Fc+45d2Zd/JN7RWFk5qSFA1k YQSNRm82aMF2vBRfFA0b5DE1AnqxhxV7iNaM15o2pmlGFZjDCeXp1E= DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=gcc.gnu.org; h=list-id :list-unsubscribe:list-archive:list-post:list-help:sender:from :to:cc:subject:date:message-id:in-reply-to:references; s= default; bh=Nlg/sUkhn1TudPZ+jy8MbtKyJkc=; b=axond+x5HxNIpwYGcWAB EHbDBOSLVZz1bHilJrr/Yrn7ntnm+lBm0PCs4IdD3No2ZCnvcH2kd//FFmeyXM5H 3rlwe5uLvzxQLdpUYbnxuMpF0UcSw0r1PanNBeDdSllq1qUz64wShXZC9ybm1wTP HBIVUl2Dk0bpiqHCVmzipCk= Received: (qmail 67218 invoked by alias); 2 Oct 2018 16:19:33 -0000 Mailing-List: contact gcc-patches-help@gcc.gnu.org; run by ezmlm Precedence: bulk List-Id: List-Unsubscribe: List-Archive: List-Post: List-Help: Sender: gcc-patches-owner@gcc.gnu.org Delivered-To: mailing list gcc-patches@gcc.gnu.org Received: (qmail 67113 invoked by uid 89); 2 Oct 2018 16:19:32 -0000 Authentication-Results: sourceware.org; auth=none X-Spam-SWARE-Status: No, score=-26.1 required=5.0 tests=BAYES_00, GIT_PATCH_0, GIT_PATCH_1, GIT_PATCH_2, GIT_PATCH_3, KAM_ASCII_DIVIDERS, RCVD_IN_DNSWL_NONE, SPF_PASS autolearn=ham version=3.3.2 spammy= X-HELO: mail-ot1-f46.google.com Received: from mail-ot1-f46.google.com (HELO mail-ot1-f46.google.com) (209.85.210.46) by sourceware.org (qpsmtpd/0.93/v0.84-503-g423c35a) with ESMTP; Tue, 02 Oct 2018 16:19:28 +0000 Received: by mail-ot1-f46.google.com with SMTP id u22so2442759ota.12 for ; Tue, 02 Oct 2018 09:19:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=uIlZ6fYjn8jCEkMixj+KWY283sMiPUK0QgLmt+36YgI=; b=f2vPBUkq6tn14u0+vXRZHB7E/OF3znebjz12T67GeNjMcaLVXkVoeI2NOn14tH7R1q i5rwWvKxcuOU7BLzpz0WIS11YMJlLv43W9JQxbtxkZYQnroNEafdPbW1tD9R2/1C/GzJ 8YRpKxlbtoLZ3mRR/rHGFRJebLlnVSooA1wZE= Return-Path: Received: from cloudburst.twiddle.net ([187.217.227.243]) by smtp.gmail.com with ESMTPSA id j15-v6sm145987oth.27.2018.10.02.09.19.24 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Tue, 02 Oct 2018 09:19:25 -0700 (PDT) From: Richard Henderson To: gcc-patches@gcc.gnu.org Cc: ramana.radhakrishnan@arm.com, agraf@suse.de, marcus.shawcroft@arm.com, james.greenhalgh@arm.com, richard.earnshaw@arm.com Subject: [PATCH, AArch64 v2 04/11] aarch64: Improve atomic-op lse generation Date: Tue, 2 Oct 2018 11:19:08 -0500 Message-Id: <20181002161915.18843-5-richard.henderson@linaro.org> In-Reply-To: <20181002161915.18843-1-richard.henderson@linaro.org> References: <20181002161915.18843-1-richard.henderson@linaro.org> Fix constraints; avoid unnecessary split. Drop the use of the atomic_op iterator in favor of the ATOMIC_LDOP iterator; this is simplier and more logical for ldclr aka bic. * config/aarch64/aarch64.c (aarch64_emit_bic): Remove. (aarch64_atomic_ldop_supported_p): Remove. (aarch64_gen_atomic_ldop): Remove. * config/aarch64/atomic.md (atomic_): Fully expand LSE operations here. (atomic_fetch_): Likewise. (atomic__fetch): Likewise. (aarch64_atomic__lse): Drop atomic_op iterator and use ATOMIC_LDOP instead; use register_operand for the input; drop the split and emit insns directly. (aarch64_atomic_fetch__lse): Likewise. (aarch64_atomic__fetch_lse): Remove. (@aarch64_atomic_load): Remove. --- gcc/config/aarch64/aarch64-protos.h | 2 - gcc/config/aarch64/aarch64.c | 176 ------------------------- gcc/config/aarch64/atomics.md | 197 +++++++++++++++------------- gcc/config/aarch64/iterators.md | 5 +- 4 files changed, 108 insertions(+), 272 deletions(-) -- 2.17.1 diff --git a/gcc/config/aarch64/aarch64-protos.h b/gcc/config/aarch64/aarch64-protos.h index 3d045cf43be..1d2f8487d1a 100644 --- a/gcc/config/aarch64/aarch64-protos.h +++ b/gcc/config/aarch64/aarch64-protos.h @@ -563,8 +563,6 @@ rtx aarch64_load_tp (rtx); void aarch64_expand_compare_and_swap (rtx op[]); void aarch64_split_compare_and_swap (rtx op[]); -bool aarch64_atomic_ldop_supported_p (enum rtx_code); -void aarch64_gen_atomic_ldop (enum rtx_code, rtx, rtx, rtx, rtx, rtx); void aarch64_split_atomic_op (enum rtx_code, rtx, rtx, rtx, rtx, rtx, rtx); bool aarch64_gen_adjusted_ldpstp (rtx *, bool, scalar_mode, RTX_CODE); diff --git a/gcc/config/aarch64/aarch64.c b/gcc/config/aarch64/aarch64.c index f7b0af2589e..867759f7e80 100644 --- a/gcc/config/aarch64/aarch64.c +++ b/gcc/config/aarch64/aarch64.c @@ -14226,32 +14226,6 @@ aarch64_expand_compare_and_swap (rtx operands[]) emit_insn (gen_rtx_SET (bval, x)); } -/* Test whether the target supports using a atomic load-operate instruction. - CODE is the operation and AFTER is TRUE if the data in memory after the - operation should be returned and FALSE if the data before the operation - should be returned. Returns FALSE if the operation isn't supported by the - architecture. */ - -bool -aarch64_atomic_ldop_supported_p (enum rtx_code code) -{ - if (!TARGET_LSE) - return false; - - switch (code) - { - case SET: - case AND: - case IOR: - case XOR: - case MINUS: - case PLUS: - return true; - default: - return false; - } -} - /* Emit a barrier, that is appropriate for memory model MODEL, at the end of a sequence implementing an atomic operation. */ @@ -14384,156 +14358,6 @@ aarch64_split_compare_and_swap (rtx operands[]) aarch64_emit_post_barrier (model); } -/* Emit a BIC instruction. */ - -static void -aarch64_emit_bic (machine_mode mode, rtx dst, rtx s1, rtx s2, int shift) -{ - rtx shift_rtx = GEN_INT (shift); - rtx (*gen) (rtx, rtx, rtx, rtx); - - switch (mode) - { - case E_SImode: gen = gen_and_one_cmpl_lshrsi3; break; - case E_DImode: gen = gen_and_one_cmpl_lshrdi3; break; - default: - gcc_unreachable (); - } - - emit_insn (gen (dst, s2, shift_rtx, s1)); -} - -/* Emit an atomic load+operate. CODE is the operation. OUT_DATA is the - location to store the data read from memory. OUT_RESULT is the location to - store the result of the operation. MEM is the memory location to read and - modify. MODEL_RTX is the memory ordering to use. VALUE is the second - operand for the operation. Either OUT_DATA or OUT_RESULT, but not both, can - be NULL. */ - -void -aarch64_gen_atomic_ldop (enum rtx_code code, rtx out_data, rtx out_result, - rtx mem, rtx value, rtx model_rtx) -{ - machine_mode mode = GET_MODE (mem); - machine_mode wmode = (mode == DImode ? DImode : SImode); - const bool short_mode = (mode < SImode); - int ldop_code; - rtx src; - rtx x; - - if (out_data) - out_data = gen_lowpart (mode, out_data); - - if (out_result) - out_result = gen_lowpart (mode, out_result); - - /* Make sure the value is in a register, putting it into a destination - register if it needs to be manipulated. */ - if (!register_operand (value, mode) - || code == AND || code == MINUS) - { - src = out_result ? out_result : out_data; - emit_move_insn (src, gen_lowpart (mode, value)); - } - else - src = value; - gcc_assert (register_operand (src, mode)); - - /* Preprocess the data for the operation as necessary. If the operation is - a SET then emit a swap instruction and finish. */ - switch (code) - { - case MINUS: - /* Negate the value and treat it as a PLUS. */ - { - rtx neg_src; - - /* Resize the value if necessary. */ - if (short_mode) - src = gen_lowpart (wmode, src); - - neg_src = gen_rtx_NEG (wmode, src); - emit_insn (gen_rtx_SET (src, neg_src)); - - if (short_mode) - src = gen_lowpart (mode, src); - } - /* Fall-through. */ - case PLUS: - ldop_code = UNSPECV_ATOMIC_LDOP_PLUS; - break; - - case IOR: - ldop_code = UNSPECV_ATOMIC_LDOP_OR; - break; - - case XOR: - ldop_code = UNSPECV_ATOMIC_LDOP_XOR; - break; - - case AND: - { - rtx not_src; - - /* Resize the value if necessary. */ - if (short_mode) - src = gen_lowpart (wmode, src); - - not_src = gen_rtx_NOT (wmode, src); - emit_insn (gen_rtx_SET (src, not_src)); - - if (short_mode) - src = gen_lowpart (mode, src); - } - ldop_code = UNSPECV_ATOMIC_LDOP_BIC; - break; - - default: - /* The operation can't be done with atomic instructions. */ - gcc_unreachable (); - } - - emit_insn (gen_aarch64_atomic_load (ldop_code, mode, - out_data, mem, src, model_rtx)); - - /* If necessary, calculate the data in memory after the update by redoing the - operation from values in registers. */ - if (!out_result) - return; - - if (short_mode) - { - src = gen_lowpart (wmode, src); - out_data = gen_lowpart (wmode, out_data); - out_result = gen_lowpart (wmode, out_result); - } - - x = NULL_RTX; - - switch (code) - { - case MINUS: - case PLUS: - x = gen_rtx_PLUS (wmode, out_data, src); - break; - case IOR: - x = gen_rtx_IOR (wmode, out_data, src); - break; - case XOR: - x = gen_rtx_XOR (wmode, out_data, src); - break; - case AND: - aarch64_emit_bic (wmode, out_result, out_data, src, 0); - return; - default: - gcc_unreachable (); - } - - emit_set_insn (out_result, x); - - return; -} - /* Split an atomic operation. */ void diff --git a/gcc/config/aarch64/atomics.md b/gcc/config/aarch64/atomics.md index bc9e396dc96..2198649b1be 100644 --- a/gcc/config/aarch64/atomics.md +++ b/gcc/config/aarch64/atomics.md @@ -207,13 +207,37 @@ rtx (*gen) (rtx, rtx, rtx); /* Use an atomic load-operate instruction when possible. */ - if (aarch64_atomic_ldop_supported_p ()) - gen = gen_aarch64_atomic__lse; + if (TARGET_LSE) + { + switch () + { + case MINUS: + operands[1] = expand_simple_unop (mode, NEG, operands[1], + NULL, 1); + /* fallthru */ + case PLUS: + gen = gen_aarch64_atomic_add_lse; + break; + case IOR: + gen = gen_aarch64_atomic_ior_lse; + break; + case XOR: + gen = gen_aarch64_atomic_xor_lse; + break; + case AND: + operands[1] = expand_simple_unop (mode, NOT, operands[1], + NULL, 1); + gen = gen_aarch64_atomic_bic_lse; + break; + default: + gcc_unreachable (); + } + operands[1] = force_reg (mode, operands[1]); + } else gen = gen_aarch64_atomic_; emit_insn (gen (operands[0], operands[1], operands[2])); - DONE; } ) @@ -239,22 +263,25 @@ } ) -(define_insn_and_split "aarch64_atomic__lse" +(define_insn "aarch64_atomic__lse" [(set (match_operand:ALLI 0 "aarch64_sync_memory_operand" "+Q") - (unspec_volatile:ALLI - [(atomic_op:ALLI (match_dup 0) - (match_operand:ALLI 1 "" "r")) - (match_operand:SI 2 "const_int_operand")] - UNSPECV_ATOMIC_OP)) + (unspec_volatile:ALLI + [(match_dup 0) + (match_operand:ALLI 1 "register_operand" "r") + (match_operand:SI 2 "const_int_operand")] + ATOMIC_LDOP)) (clobber (match_scratch:ALLI 3 "=&r"))] "TARGET_LSE" - "#" - "&& reload_completed" - [(const_int 0)] { - aarch64_gen_atomic_ldop (, operands[3], NULL, operands[0], - operands[1], operands[2]); - DONE; + enum memmodel model = memmodel_from_int (INTVAL (operands[2])); + if (is_mm_relaxed (model)) + return "ld\t%1, %3, %0"; + else if (is_mm_release (model)) + return "ldl\t%1, %3, %0"; + else if (is_mm_acquire (model) || is_mm_consume (model)) + return "lda\t%1, %3, %0"; + else + return "ldal\t%1, %3, %0"; } ) @@ -280,7 +307,7 @@ } ) -;; Load-operate-store, returning the updated memory data. +;; Load-operate-store, returning the original memory data. (define_expand "atomic_fetch_" [(match_operand:ALLI 0 "register_operand" "") @@ -293,13 +320,37 @@ rtx (*gen) (rtx, rtx, rtx, rtx); /* Use an atomic load-operate instruction when possible. */ - if (aarch64_atomic_ldop_supported_p ()) - gen = gen_aarch64_atomic_fetch__lse; + if (TARGET_LSE) + { + switch () + { + case MINUS: + operands[2] = expand_simple_unop (mode, NEG, operands[2], + NULL, 1); + /* fallthru */ + case PLUS: + gen = gen_aarch64_atomic_fetch_add_lse; + break; + case IOR: + gen = gen_aarch64_atomic_fetch_ior_lse; + break; + case XOR: + gen = gen_aarch64_atomic_fetch_xor_lse; + break; + case AND: + operands[2] = expand_simple_unop (mode, NOT, operands[2], + NULL, 1); + gen = gen_aarch64_atomic_fetch_bic_lse; + break; + default: + gcc_unreachable (); + } + operands[2] = force_reg (mode, operands[2]); + } else gen = gen_aarch64_atomic_fetch_; emit_insn (gen (operands[0], operands[1], operands[2], operands[3])); - DONE; }) @@ -326,23 +377,26 @@ } ) -(define_insn_and_split "aarch64_atomic_fetch__lse" - [(set (match_operand:ALLI 0 "register_operand" "=&r") - (match_operand:ALLI 1 "aarch64_sync_memory_operand" "+Q")) +(define_insn "aarch64_atomic_fetch__lse" + [(set (match_operand:ALLI 0 "register_operand" "=r") + (match_operand:ALLI 1 "aarch64_sync_memory_operand" "+Q")) (set (match_dup 1) - (unspec_volatile:ALLI - [(atomic_op:ALLI (match_dup 1) - (match_operand:ALLI 2 "" "r")) - (match_operand:SI 3 "const_int_operand")] - UNSPECV_ATOMIC_LDOP))] + (unspec_volatile:ALLI + [(match_dup 1) + (match_operand:ALLI 2 "register_operand" "r") + (match_operand:SI 3 "const_int_operand")] + ATOMIC_LDOP))] "TARGET_LSE" - "#" - "&& reload_completed" - [(const_int 0)] { - aarch64_gen_atomic_ldop (, operands[0], NULL, operands[1], - operands[2], operands[3]); - DONE; + enum memmodel model = memmodel_from_int (INTVAL (operands[3])); + if (is_mm_relaxed (model)) + return "ld\t%2, %0, %1"; + else if (is_mm_acquire (model) || is_mm_consume (model)) + return "lda\t%2, %0, %1"; + else if (is_mm_release (model)) + return "ldl\t%2, %0, %1"; + else + return "ldal\t%2, %0, %1"; } ) @@ -370,7 +424,7 @@ } ) -;; Load-operate-store, returning the original memory data. +;; Load-operate-store, returning the updated memory data. (define_expand "atomic__fetch" [(match_operand:ALLI 0 "register_operand" "") @@ -380,17 +434,23 @@ (match_operand:SI 3 "const_int_operand")] "" { - rtx (*gen) (rtx, rtx, rtx, rtx); - rtx value = operands[2]; - - /* Use an atomic load-operate instruction when possible. */ - if (aarch64_atomic_ldop_supported_p ()) - gen = gen_aarch64_atomic__fetch_lse; + /* Use an atomic load-operate instruction when possible. In this case + we will re-compute the result from the original mem value. */ + if (TARGET_LSE) + { + rtx tmp = gen_reg_rtx (mode); + operands[2] = force_reg (mode, operands[2]); + emit_insn (gen_atomic_fetch_ + (tmp, operands[1], operands[2], operands[3])); + tmp = expand_simple_binop (mode, , tmp, operands[2], + operands[0], 1, OPTAB_WIDEN); + emit_move_insn (operands[0], tmp); + } else - gen = gen_aarch64_atomic__fetch; - - emit_insn (gen (operands[0], operands[1], value, operands[3])); - + { + emit_insn (gen_aarch64_atomic__fetch + (operands[0], operands[1], operands[2], operands[3])); + } DONE; }) @@ -417,29 +477,6 @@ } ) -(define_insn_and_split "aarch64_atomic__fetch_lse" - [(set (match_operand:ALLI 0 "register_operand" "=&r") - (atomic_op:ALLI - (match_operand:ALLI 1 "aarch64_sync_memory_operand" "+Q") - (match_operand:ALLI 2 "" "r"))) - (set (match_dup 1) - (unspec_volatile:ALLI - [(match_dup 1) - (match_dup 2) - (match_operand:SI 3 "const_int_operand")] - UNSPECV_ATOMIC_LDOP)) - (clobber (match_scratch:ALLI 4 "=&r"))] - "TARGET_LSE" - "#" - "&& reload_completed" - [(const_int 0)] - { - aarch64_gen_atomic_ldop (, operands[4], operands[0], operands[1], - operands[2], operands[3]); - DONE; - } -) - (define_insn_and_split "atomic_nand_fetch" [(set (match_operand:ALLI 0 "register_operand" "=&r") (not:ALLI @@ -585,29 +622,3 @@ return "dmb\\tish"; } ) - -;; ARMv8.1-A LSE instructions. - -;; Atomic load-op: Load data, operate, store result, keep data. - -(define_insn "@aarch64_atomic_load" - [(set (match_operand:ALLI 0 "register_operand" "=r") - (match_operand:ALLI 1 "aarch64_sync_memory_operand" "+Q")) - (set (match_dup 1) - (unspec_volatile:ALLI - [(match_dup 1) - (match_operand:ALLI 2 "register_operand") - (match_operand:SI 3 "const_int_operand")] - ATOMIC_LDOP))] - "TARGET_LSE && reload_completed" - { - enum memmodel model = memmodel_from_int (INTVAL (operands[3])); - if (is_mm_relaxed (model)) - return "ld\t%2, %0, %1"; - else if (is_mm_acquire (model) || is_mm_consume (model)) - return "lda\t%2, %0, %1"; - else if (is_mm_release (model)) - return "ldl\t%2, %0, %1"; - else - return "ldal\t%2, %0, %1"; - }) diff --git a/gcc/config/aarch64/iterators.md b/gcc/config/aarch64/iterators.md index a43956054e8..524e4e6929b 100644 --- a/gcc/config/aarch64/iterators.md +++ b/gcc/config/aarch64/iterators.md @@ -503,7 +503,6 @@ UNSPECV_ATOMIC_CAS ; Represent an atomic CAS. UNSPECV_ATOMIC_SWP ; Represent an atomic SWP. UNSPECV_ATOMIC_OP ; Represent an atomic operation. - UNSPECV_ATOMIC_LDOP ; Represent an atomic load-operation UNSPECV_ATOMIC_LDOP_OR ; Represent an atomic load-or UNSPECV_ATOMIC_LDOP_BIC ; Represent an atomic load-bic UNSPECV_ATOMIC_LDOP_XOR ; Represent an atomic load-xor @@ -1591,6 +1590,10 @@ [(UNSPECV_ATOMIC_LDOP_OR "set") (UNSPECV_ATOMIC_LDOP_BIC "clr") (UNSPECV_ATOMIC_LDOP_XOR "eor") (UNSPECV_ATOMIC_LDOP_PLUS "add")]) +(define_int_attr atomic_ldoptab + [(UNSPECV_ATOMIC_LDOP_OR "ior") (UNSPECV_ATOMIC_LDOP_BIC "bic") + (UNSPECV_ATOMIC_LDOP_XOR "xor") (UNSPECV_ATOMIC_LDOP_PLUS "add")]) + ;; ------------------------------------------------------------------- ;; Int Iterators Attributes. ;; ------------------------------------------------------------------- From patchwork Tue Oct 2 16:19:09 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 147988 Delivered-To: patch@linaro.org Received: by 2002:a2e:8595:0:0:0:0:0 with SMTP id b21-v6csp5355209lji; Tue, 2 Oct 2018 09:20:41 -0700 (PDT) X-Google-Smtp-Source: ACcGV63SBWEYLXX9pUlm1IgdSVZ8tbpIeLc5AG4Htldmw9n4YvVFU2QKP0XBh8uKu/iohipwzH9x X-Received: by 2002:a62:4799:: with SMTP id p25-v6mr17181405pfi.197.1538497241222; Tue, 02 Oct 2018 09:20:41 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1538497241; cv=none; d=google.com; s=arc-20160816; b=IPLzj4JE7Tu0KmUGl01ymnle/e6MS1KRNwu6EUHAwj85fdLp/TDhA3K3W7zgUMjLp9 fQCKUNhngKY1diElCMWxYyqmV0No6oOAgOLMVP6cYCh9cvMsFlBK+OxORaxTjVzT7S/A QlH05mHWFu2y2arYHyPunXxpLNzCd5k9YrGplG/+T0M5ntukYnJ4YCyMCS4WvNafxWfl YGiPkVuQu19cQXUOEYFu+TsFekhCS7loDBpeVhNas0/q4aEN+sAIW7yF23qJHl6TBg5/ 2rXyVevrCDCoKZZOBNikETqhc/PLTXdGhOnWJqdaRqgNgJYZIH6fPRAP2x6bcdu/jln9 m8jg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature:delivered-to:sender:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:mailing-list:dkim-signature :domainkey-signature; bh=7+zp9tJT4bsfDlpvpXN+n8MuwsRBjoesubghhKAcJVU=; b=ZPXMPQmXtWUIDdSWjO+xoms6/end6/4sp/AnF0vjzO7IquJhypC1gjk2fz34BX2+yr viWjMxc/ZPr4g5Z20ABrDkrnjkwmRIK2d2OK5lP/Esv+HJk4bmxos17Yk8ZaN7xuG9+4 8Yjbmm/I+QqojXA2fW+KdQgDmxiU9Ex+UuyJd211OOPfyo6w1yXFJxG5+/FV4pKLBPBN Y3rlgkqdeln73J8941CBKMd2lJK9kwW40YWBor4MqUq94CySUgnRN2lffxnXteRdB+t3 MsP9/n6HlfaNtBcy1r/Ot8CCWJ3Fkw/5/ixjCoPegUe8hlRMQ557XXkEEKp+kD8yTHq8 54nA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gcc.gnu.org header.s=default header.b=h3ZKybD3; dkim=pass header.i=@linaro.org header.s=google header.b=bLZ1P0sN; spf=pass (google.com: domain of gcc-patches-return-486820-patch=linaro.org@gcc.gnu.org designates 209.132.180.131 as permitted sender) smtp.mailfrom="gcc-patches-return-486820-patch=linaro.org@gcc.gnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from sourceware.org (server1.sourceware.org. [209.132.180.131]) by mx.google.com with ESMTPS id p17-v6si15731038pgj.416.2018.10.02.09.20.40 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 02 Oct 2018 09:20:41 -0700 (PDT) Received-SPF: pass (google.com: domain of gcc-patches-return-486820-patch=linaro.org@gcc.gnu.org designates 209.132.180.131 as permitted sender) client-ip=209.132.180.131; Authentication-Results: mx.google.com; dkim=pass header.i=@gcc.gnu.org header.s=default header.b=h3ZKybD3; dkim=pass header.i=@linaro.org header.s=google header.b=bLZ1P0sN; spf=pass (google.com: domain of gcc-patches-return-486820-patch=linaro.org@gcc.gnu.org designates 209.132.180.131 as permitted sender) smtp.mailfrom="gcc-patches-return-486820-patch=linaro.org@gcc.gnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org DomainKey-Signature: a=rsa-sha1; c=nofws; d=gcc.gnu.org; h=list-id :list-unsubscribe:list-archive:list-post:list-help:sender:from :to:cc:subject:date:message-id:in-reply-to:references; q=dns; s= default; b=rl7tYRHZyvqZtfNH+u+ExmQLH+5MA0QpP+T6ou/PVv2U68l3GnuX7 5Jqs5+1YZn9el0U4Tkb+Z+IytV6C3sbJx1ho89NaDaI0986cHn0hZB/q4I16hg56 DDrcRTbqCFTddnjpXRs2ByXqWEp+1hWMxkIKlqC3075ak+YhqC2lT4= DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=gcc.gnu.org; h=list-id :list-unsubscribe:list-archive:list-post:list-help:sender:from :to:cc:subject:date:message-id:in-reply-to:references; s= default; bh=7lLVtfyzbDBSG5aObVV4wYJHVsc=; b=h3ZKybD3sqiTYhLoz7+/ tZWzB9X3c8zb4QEmlK6ZFSJ1qNcvl4LeFMPUpozmab1v9F+H2KrZ9y6SW/L7W8Dr uWFaa4SMK28yzxGSlIiCB6QNYiQJ+84D1jHHR2Z84cfOZvk7yK0U5KkuusRH/Drt o3OEm8mU7DlMfn0fJvoGfq4= Received: (qmail 67270 invoked by alias); 2 Oct 2018 16:19:33 -0000 Mailing-List: contact gcc-patches-help@gcc.gnu.org; run by ezmlm Precedence: bulk List-Id: List-Unsubscribe: List-Archive: List-Post: List-Help: Sender: gcc-patches-owner@gcc.gnu.org Delivered-To: mailing list gcc-patches@gcc.gnu.org Received: (qmail 67159 invoked by uid 89); 2 Oct 2018 16:19:33 -0000 Authentication-Results: sourceware.org; auth=none X-Spam-SWARE-Status: No, score=-26.9 required=5.0 tests=BAYES_00, GIT_PATCH_0, GIT_PATCH_1, GIT_PATCH_2, GIT_PATCH_3, RCVD_IN_DNSWL_NONE, SPF_PASS autolearn=ham version=3.3.2 spammy=HX-Received:sk:21-v6mr X-HELO: mail-oi1-f181.google.com Received: from mail-oi1-f181.google.com (HELO mail-oi1-f181.google.com) (209.85.167.181) by sourceware.org (qpsmtpd/0.93/v0.84-503-g423c35a) with ESMTP; Tue, 02 Oct 2018 16:19:30 +0000 Received: by mail-oi1-f181.google.com with SMTP id j68-v6so1992051oib.7 for ; Tue, 02 Oct 2018 09:19:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=7+zp9tJT4bsfDlpvpXN+n8MuwsRBjoesubghhKAcJVU=; b=bLZ1P0sNnrX7DmmgGKQLCpxt2CHF5OUY4nS198tx/81kbH+hNuryBE+FULeLXBOnKE 3rZvZ99SNO0T7+NbBeAo+OmRR/l/BYOFy0ADxFqshHz2FS6S/nk5KshffIQ5rRM01jbj aYGXHjBxFmybjCl3YJtSG9vAqy/N0hXlcUJgM= Return-Path: Received: from cloudburst.twiddle.net ([187.217.227.243]) by smtp.gmail.com with ESMTPSA id j15-v6sm145987oth.27.2018.10.02.09.19.26 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Tue, 02 Oct 2018 09:19:27 -0700 (PDT) From: Richard Henderson To: gcc-patches@gcc.gnu.org Cc: ramana.radhakrishnan@arm.com, agraf@suse.de, marcus.shawcroft@arm.com, james.greenhalgh@arm.com, richard.earnshaw@arm.com Subject: [PATCH, AArch64 v2 05/11] aarch64: Emit LSE st instructions Date: Tue, 2 Oct 2018 11:19:09 -0500 Message-Id: <20181002161915.18843-6-richard.henderson@linaro.org> In-Reply-To: <20181002161915.18843-1-richard.henderson@linaro.org> References: <20181002161915.18843-1-richard.henderson@linaro.org> When the result of an operation is not used, we can ignore the result by storing to XZR. For two of the memory models, using XZR with LD has a preferred assembler alias, ST. * config/aarch64/atomics.md (aarch64_atomic__lse): Use ST for relaxed and release models; load to XZR otherwise; remove the now unnecessary scratch register. * gcc.target/aarch64/atomic-inst-ldadd.c: Expect stadd{,l}. * gcc.target/aarch64/atomic-inst-ldlogic.c: Similarly. --- .../gcc.target/aarch64/atomic-inst-ldadd.c | 18 ++++--- .../gcc.target/aarch64/atomic-inst-ldlogic.c | 54 ++++++++++++------- gcc/config/aarch64/atomics.md | 15 +++--- 3 files changed, 57 insertions(+), 30 deletions(-) -- 2.17.1 diff --git a/gcc/testsuite/gcc.target/aarch64/atomic-inst-ldadd.c b/gcc/testsuite/gcc.target/aarch64/atomic-inst-ldadd.c index 4b2282c6861..db2206186b4 100644 --- a/gcc/testsuite/gcc.target/aarch64/atomic-inst-ldadd.c +++ b/gcc/testsuite/gcc.target/aarch64/atomic-inst-ldadd.c @@ -67,20 +67,26 @@ TEST (add_load_notreturn, ADD_LOAD_NORETURN) TEST (sub_load, SUB_LOAD) TEST (sub_load_notreturn, SUB_LOAD_NORETURN) -/* { dg-final { scan-assembler-times "ldaddb\t" 16} } */ +/* { dg-final { scan-assembler-times "ldaddb\t" 8} } */ /* { dg-final { scan-assembler-times "ldaddab\t" 32} } */ -/* { dg-final { scan-assembler-times "ldaddlb\t" 16} } */ +/* { dg-final { scan-assembler-times "ldaddlb\t" 8} } */ /* { dg-final { scan-assembler-times "ldaddalb\t" 32} } */ +/* { dg-final { scan-assembler-times "staddb\t" 8} } */ +/* { dg-final { scan-assembler-times "staddlb\t" 8} } */ -/* { dg-final { scan-assembler-times "ldaddh\t" 16} } */ +/* { dg-final { scan-assembler-times "ldaddh\t" 8} } */ /* { dg-final { scan-assembler-times "ldaddah\t" 32} } */ -/* { dg-final { scan-assembler-times "ldaddlh\t" 16} } */ +/* { dg-final { scan-assembler-times "ldaddlh\t" 8} } */ /* { dg-final { scan-assembler-times "ldaddalh\t" 32} } */ +/* { dg-final { scan-assembler-times "staddh\t" 8} } */ +/* { dg-final { scan-assembler-times "staddlh\t" 8} } */ -/* { dg-final { scan-assembler-times "ldadd\t" 32} } */ +/* { dg-final { scan-assembler-times "ldadd\t" 16} } */ /* { dg-final { scan-assembler-times "ldadda\t" 64} } */ -/* { dg-final { scan-assembler-times "ldaddl\t" 32} } */ +/* { dg-final { scan-assembler-times "ldaddl\t" 16} } */ /* { dg-final { scan-assembler-times "ldaddal\t" 64} } */ +/* { dg-final { scan-assembler-times "stadd\t" 16} } */ +/* { dg-final { scan-assembler-times "staddl\t" 16} } */ /* { dg-final { scan-assembler-not "ldaxr\t" } } */ /* { dg-final { scan-assembler-not "stlxr\t" } } */ diff --git a/gcc/testsuite/gcc.target/aarch64/atomic-inst-ldlogic.c b/gcc/testsuite/gcc.target/aarch64/atomic-inst-ldlogic.c index 4879d52b9b4..b8a53e0a676 100644 --- a/gcc/testsuite/gcc.target/aarch64/atomic-inst-ldlogic.c +++ b/gcc/testsuite/gcc.target/aarch64/atomic-inst-ldlogic.c @@ -101,54 +101,72 @@ TEST (xor_load_notreturn, XOR_LOAD_NORETURN) /* Load-OR. */ -/* { dg-final { scan-assembler-times "ldsetb\t" 8} } */ +/* { dg-final { scan-assembler-times "ldsetb\t" 4} } */ /* { dg-final { scan-assembler-times "ldsetab\t" 16} } */ -/* { dg-final { scan-assembler-times "ldsetlb\t" 8} } */ +/* { dg-final { scan-assembler-times "ldsetlb\t" 4} } */ /* { dg-final { scan-assembler-times "ldsetalb\t" 16} } */ +/* { dg-final { scan-assembler-times "stsetb\t" 4} } */ +/* { dg-final { scan-assembler-times "stsetlb\t" 4} } */ -/* { dg-final { scan-assembler-times "ldseth\t" 8} } */ +/* { dg-final { scan-assembler-times "ldseth\t" 4} } */ /* { dg-final { scan-assembler-times "ldsetah\t" 16} } */ -/* { dg-final { scan-assembler-times "ldsetlh\t" 8} } */ +/* { dg-final { scan-assembler-times "ldsetlh\t" 4} } */ /* { dg-final { scan-assembler-times "ldsetalh\t" 16} } */ +/* { dg-final { scan-assembler-times "stseth\t" 4} } */ +/* { dg-final { scan-assembler-times "stsetlh\t" 4} } */ -/* { dg-final { scan-assembler-times "ldset\t" 16} } */ +/* { dg-final { scan-assembler-times "ldset\t" 8} } */ /* { dg-final { scan-assembler-times "ldseta\t" 32} } */ -/* { dg-final { scan-assembler-times "ldsetl\t" 16} } */ +/* { dg-final { scan-assembler-times "ldsetl\t" 8} } */ /* { dg-final { scan-assembler-times "ldsetal\t" 32} } */ +/* { dg-final { scan-assembler-times "stset\t" 8} } */ +/* { dg-final { scan-assembler-times "stsetl\t" 8} } */ /* Load-AND. */ -/* { dg-final { scan-assembler-times "ldclrb\t" 8} } */ +/* { dg-final { scan-assembler-times "ldclrb\t" 4} } */ /* { dg-final { scan-assembler-times "ldclrab\t" 16} } */ -/* { dg-final { scan-assembler-times "ldclrlb\t" 8} } */ +/* { dg-final { scan-assembler-times "ldclrlb\t" 4} } */ /* { dg-final { scan-assembler-times "ldclralb\t" 16} } */ +/* { dg-final { scan-assembler-times "stclrb\t" 4} } */ +/* { dg-final { scan-assembler-times "stclrlb\t" 4} } */ -/* { dg-final { scan-assembler-times "ldclrh\t" 8} } */ +/* { dg-final { scan-assembler-times "ldclrh\t" 4} } */ /* { dg-final { scan-assembler-times "ldclrah\t" 16} } */ -/* { dg-final { scan-assembler-times "ldclrlh\t" 8} } */ +/* { dg-final { scan-assembler-times "ldclrlh\t" 4} } */ /* { dg-final { scan-assembler-times "ldclralh\t" 16} } */ +/* { dg-final { scan-assembler-times "stclrh\t" 4} } */ +/* { dg-final { scan-assembler-times "stclrlh\t" 4} } */ -/* { dg-final { scan-assembler-times "ldclr\t" 16} */ +/* { dg-final { scan-assembler-times "ldclr\t" 8} */ /* { dg-final { scan-assembler-times "ldclra\t" 32} } */ -/* { dg-final { scan-assembler-times "ldclrl\t" 16} } */ +/* { dg-final { scan-assembler-times "ldclrl\t" 8} } */ /* { dg-final { scan-assembler-times "ldclral\t" 32} } */ +/* { dg-final { scan-assembler-times "stclr\t" 8} */ +/* { dg-final { scan-assembler-times "stclrl\t" 8} } */ /* Load-XOR. */ -/* { dg-final { scan-assembler-times "ldeorb\t" 8} } */ +/* { dg-final { scan-assembler-times "ldeorb\t" 4} } */ /* { dg-final { scan-assembler-times "ldeorab\t" 16} } */ -/* { dg-final { scan-assembler-times "ldeorlb\t" 8} } */ +/* { dg-final { scan-assembler-times "ldeorlb\t" 4} } */ /* { dg-final { scan-assembler-times "ldeoralb\t" 16} } */ +/* { dg-final { scan-assembler-times "steorb\t" 4} } */ +/* { dg-final { scan-assembler-times "steorlb\t" 4} } */ -/* { dg-final { scan-assembler-times "ldeorh\t" 8} } */ +/* { dg-final { scan-assembler-times "ldeorh\t" 4} } */ /* { dg-final { scan-assembler-times "ldeorah\t" 16} } */ -/* { dg-final { scan-assembler-times "ldeorlh\t" 8} } */ +/* { dg-final { scan-assembler-times "ldeorlh\t" 4} } */ /* { dg-final { scan-assembler-times "ldeoralh\t" 16} } */ +/* { dg-final { scan-assembler-times "steorh\t" 4} } */ +/* { dg-final { scan-assembler-times "steorlh\t" 4} } */ -/* { dg-final { scan-assembler-times "ldeor\t" 16} */ +/* { dg-final { scan-assembler-times "ldeor\t" 8} */ /* { dg-final { scan-assembler-times "ldeora\t" 32} } */ -/* { dg-final { scan-assembler-times "ldeorl\t" 16} } */ +/* { dg-final { scan-assembler-times "ldeorl\t" 8} } */ /* { dg-final { scan-assembler-times "ldeoral\t" 32} } */ +/* { dg-final { scan-assembler-times "steor\t" 8} */ +/* { dg-final { scan-assembler-times "steorl\t" 8} } */ /* { dg-final { scan-assembler-not "ldaxr\t" } } */ /* { dg-final { scan-assembler-not "stlxr\t" } } */ diff --git a/gcc/config/aarch64/atomics.md b/gcc/config/aarch64/atomics.md index 2198649b1be..ee662a4480e 100644 --- a/gcc/config/aarch64/atomics.md +++ b/gcc/config/aarch64/atomics.md @@ -269,19 +269,22 @@ [(match_dup 0) (match_operand:ALLI 1 "register_operand" "r") (match_operand:SI 2 "const_int_operand")] - ATOMIC_LDOP)) - (clobber (match_scratch:ALLI 3 "=&r"))] + ATOMIC_LDOP))] "TARGET_LSE" { enum memmodel model = memmodel_from_int (INTVAL (operands[2])); + /* Note that while ST is an alias for LD with the second + operand as XZR, the assember only defines them for the RELAXED + and REL models. But there's nothing that prevents us from explicitly + using XZR with LD for the ACQ and ACQ_REL models. */ if (is_mm_relaxed (model)) - return "ld\t%1, %3, %0"; + return "st\t%1, %0"; else if (is_mm_release (model)) - return "ldl\t%1, %3, %0"; + return "stl\t%1, %0"; else if (is_mm_acquire (model) || is_mm_consume (model)) - return "lda\t%1, %3, %0"; + return "lda\t%1, zr, %0"; else - return "ldal\t%1, %3, %0"; + return "ldal\t%1, zr, %0"; } ) From patchwork Tue Oct 2 16:19:10 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 147989 Delivered-To: patch@linaro.org Received: by 2002:a2e:8595:0:0:0:0:0 with SMTP id b21-v6csp5355503lji; Tue, 2 Oct 2018 09:20:57 -0700 (PDT) X-Google-Smtp-Source: ACcGV61+G7wsLJIpDdPvt5t5sM4KECVAS5XjgJtqnAnGEDk/adr+3FBCLlNpZEO6WA2D4eF3G+DW X-Received: by 2002:a63:3c46:: with SMTP id i6-v6mr15183896pgn.286.1538497257034; Tue, 02 Oct 2018 09:20:57 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1538497257; cv=none; d=google.com; s=arc-20160816; b=atHX/PzErwxGsZniiNOBFiDSVgpIY1CVLoJe+rXqrU2axG12TL7z65O89gYxZPq5cL g20c79zrxVhMGY+U85xODIiqwrrhMOxRAWjACT57iVXRb2YnhFR3qrBqQVhQ/BZwwYvh KZ+n0smOUfUEoYsUgwLaA8PTn8q6RUolj+78jP8gGsIahRWHNmsgSnVoS5QJiCKKj1lx IijWDskPz35g93KhuPu6mW0gV7ZfVG1X/SzEtRu8R1ok1AToLk7NAWjGLQl/wI1TvBEv ludhCs3lkogq7wSiYca5/YmTAMKSUc/J9GTtg4NgsDb7PO8NllfVgqX/SuxpL1L+pNAG oLyg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature:delivered-to:sender:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:mailing-list:dkim-signature :domainkey-signature; bh=/98QaPCqynGJZv2MFh4kaB/t90szcMgzhLvjDA/RHRU=; b=sEAnI+DczbU8dyraoWpU/62b/FR+YP84SBCzFifup32Aw3IgY7U2q7MF33yrwGHNWm TM5Q7dVdo2dUST6m7cLIgXgp7uNQAcnQ3sjdBr4htNXebumnRxsttajiOSTk90gjorL7 xvzeQlffn5vcFf9Bfz6m2rAH2P2/qNmCi45Bl5yOCuJRYqWE3b8bzPdInNsDMH7g1L/t s/JEFgFuJM2XITi1KQ+9R7ci/KuphjQbspr+jkK5BZAeWyLqMJkZawa7lJZ1psLnNY2D KLb59BD+0gJILgHgi1D/pld/eBqxK0JStLQxF5EvgKfPgb0d5vhfZZYFhpIP4TivS1EA wX4g== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gcc.gnu.org header.s=default header.b=i1O3vCVx; dkim=pass header.i=@linaro.org header.s=google header.b=cy4Jqj+n; spf=pass (google.com: domain of gcc-patches-return-486821-patch=linaro.org@gcc.gnu.org designates 209.132.180.131 as permitted sender) smtp.mailfrom="gcc-patches-return-486821-patch=linaro.org@gcc.gnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from sourceware.org (server1.sourceware.org. [209.132.180.131]) by mx.google.com with ESMTPS id a64-v6si17112969pfb.224.2018.10.02.09.20.56 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 02 Oct 2018 09:20:57 -0700 (PDT) Received-SPF: pass (google.com: domain of gcc-patches-return-486821-patch=linaro.org@gcc.gnu.org designates 209.132.180.131 as permitted sender) client-ip=209.132.180.131; Authentication-Results: mx.google.com; dkim=pass header.i=@gcc.gnu.org header.s=default header.b=i1O3vCVx; dkim=pass header.i=@linaro.org header.s=google header.b=cy4Jqj+n; spf=pass (google.com: domain of gcc-patches-return-486821-patch=linaro.org@gcc.gnu.org designates 209.132.180.131 as permitted sender) smtp.mailfrom="gcc-patches-return-486821-patch=linaro.org@gcc.gnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org DomainKey-Signature: a=rsa-sha1; c=nofws; d=gcc.gnu.org; h=list-id :list-unsubscribe:list-archive:list-post:list-help:sender:from :to:cc:subject:date:message-id:in-reply-to:references; q=dns; s= default; b=r+DpKJmr5qNXuvMqiG/ov7CSf0gC2+o8iHvnrr3fGTx0+9qshhKiZ hCadxvseoagKM2E7jj9RzZ7PGHgbxBKXN/IvUUBIKspJ5jzOoHL5eMINYZ56ouf1 R29ihym/J+rQ/LgSibNtF/Ml6A5axRtCdq1OI0LJaY8DNV8bwG1gi4= DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=gcc.gnu.org; h=list-id :list-unsubscribe:list-archive:list-post:list-help:sender:from :to:cc:subject:date:message-id:in-reply-to:references; s= default; bh=zDuCiOcm/fX5obuVi5jE2GqUBpE=; b=i1O3vCVxHRUTk25GHNMF 5JOSi5XrMdCHSKi17OINEqTrhOUEbuSvvlH83euDOCVbLuXGYWEEr3NLYXW/Wrfy FGFl1xyGuiIAMlX46D88yA2Wy8LCkocIJxWEfQp4uaP5nFXdN/od3AtfhK49QaW2 qBJREMMDNuIKbnaU2LjFH60= Received: (qmail 67457 invoked by alias); 2 Oct 2018 16:19:34 -0000 Mailing-List: contact gcc-patches-help@gcc.gnu.org; run by ezmlm Precedence: bulk List-Id: List-Unsubscribe: List-Archive: List-Post: List-Help: Sender: gcc-patches-owner@gcc.gnu.org Delivered-To: mailing list gcc-patches@gcc.gnu.org Received: (qmail 67305 invoked by uid 89); 2 Oct 2018 16:19:34 -0000 Authentication-Results: sourceware.org; auth=none X-Spam-SWARE-Status: No, score=-26.9 required=5.0 tests=BAYES_00, GIT_PATCH_0, GIT_PATCH_1, GIT_PATCH_2, GIT_PATCH_3, RCVD_IN_DNSWL_NONE, SPF_PASS autolearn=ham version=3.3.2 spammy= X-HELO: mail-oi1-f194.google.com Received: from mail-oi1-f194.google.com (HELO mail-oi1-f194.google.com) (209.85.167.194) by sourceware.org (qpsmtpd/0.93/v0.84-503-g423c35a) with ESMTP; Tue, 02 Oct 2018 16:19:32 +0000 Received: by mail-oi1-f194.google.com with SMTP id e17-v6so1963977oig.12 for ; Tue, 02 Oct 2018 09:19:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=/98QaPCqynGJZv2MFh4kaB/t90szcMgzhLvjDA/RHRU=; b=cy4Jqj+nTIXsoOpeVXRJh9Gmbr/bvf5J1PzmutcjDJM0NnKOpOCWdVjiHXZcPIPQqU XMrpmV3GRtWfMhWF0XfH6QYF5RNN4SHsgEy1OYo+FQaPUzJf3ScWzY6NscUdgGhyjVFu FRvq43HkY/5cZZjO1Ue1MWquqBLdiSV2gKLDo= Return-Path: Received: from cloudburst.twiddle.net ([187.217.227.243]) by smtp.gmail.com with ESMTPSA id j15-v6sm145987oth.27.2018.10.02.09.19.28 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Tue, 02 Oct 2018 09:19:29 -0700 (PDT) From: Richard Henderson To: gcc-patches@gcc.gnu.org Cc: ramana.radhakrishnan@arm.com, agraf@suse.de, marcus.shawcroft@arm.com, james.greenhalgh@arm.com, richard.earnshaw@arm.com Subject: [PATCH, AArch64 v2 06/11] Add visibility to libfunc constructors Date: Tue, 2 Oct 2018 11:19:10 -0500 Message-Id: <20181002161915.18843-7-richard.henderson@linaro.org> In-Reply-To: <20181002161915.18843-1-richard.henderson@linaro.org> References: <20181002161915.18843-1-richard.henderson@linaro.org> * optabs-libfuncs.c (build_libfunc_function_visibility): New, split out from... (build_libfunc_function): ... here. (init_one_libfunc_visibility): New, split out from ... (init_one_libfunc): ... here. --- gcc/optabs-libfuncs.h | 2 ++ gcc/optabs-libfuncs.c | 26 ++++++++++++++++++++------ 2 files changed, 22 insertions(+), 6 deletions(-) -- 2.17.1 diff --git a/gcc/optabs-libfuncs.h b/gcc/optabs-libfuncs.h index 0669ea1fdd7..cf39da36887 100644 --- a/gcc/optabs-libfuncs.h +++ b/gcc/optabs-libfuncs.h @@ -63,7 +63,9 @@ void gen_satfract_conv_libfunc (convert_optab, const char *, void gen_satfractuns_conv_libfunc (convert_optab, const char *, machine_mode, machine_mode); +tree build_libfunc_function_visibility (const char *, symbol_visibility); tree build_libfunc_function (const char *); +rtx init_one_libfunc_visibility (const char *, symbol_visibility); rtx init_one_libfunc (const char *); rtx set_user_assembler_libfunc (const char *, const char *); diff --git a/gcc/optabs-libfuncs.c b/gcc/optabs-libfuncs.c index bd0df8baa37..73a28e9ca7a 100644 --- a/gcc/optabs-libfuncs.c +++ b/gcc/optabs-libfuncs.c @@ -719,10 +719,10 @@ struct libfunc_decl_hasher : ggc_ptr_hash /* A table of previously-created libfuncs, hashed by name. */ static GTY (()) hash_table *libfunc_decls; -/* Build a decl for a libfunc named NAME. */ +/* Build a decl for a libfunc named NAME with visibility VIS. */ tree -build_libfunc_function (const char *name) +build_libfunc_function_visibility (const char *name, symbol_visibility vis) { /* ??? We don't have any type information; pretend this is "int foo ()". */ tree decl = build_decl (UNKNOWN_LOCATION, FUNCTION_DECL, @@ -731,7 +731,7 @@ build_libfunc_function (const char *name) DECL_EXTERNAL (decl) = 1; TREE_PUBLIC (decl) = 1; DECL_ARTIFICIAL (decl) = 1; - DECL_VISIBILITY (decl) = VISIBILITY_DEFAULT; + DECL_VISIBILITY (decl) = vis; DECL_VISIBILITY_SPECIFIED (decl) = 1; gcc_assert (DECL_ASSEMBLER_NAME (decl)); @@ -742,11 +742,19 @@ build_libfunc_function (const char *name) return decl; } +/* Build a decl for a libfunc named NAME. */ + +tree +build_libfunc_function (const char *name) +{ + return build_libfunc_function_visibility (name, VISIBILITY_DEFAULT); +} + /* Return a libfunc for NAME, creating one if we don't already have one. - The returned rtx is a SYMBOL_REF. */ + The decl is given visibility VIS. The returned rtx is a SYMBOL_REF. */ rtx -init_one_libfunc (const char *name) +init_one_libfunc_visibility (const char *name, symbol_visibility vis) { tree id, decl; hashval_t hash; @@ -763,12 +771,18 @@ init_one_libfunc (const char *name) { /* Create a new decl, so that it can be passed to targetm.encode_section_info. */ - decl = build_libfunc_function (name); + decl = build_libfunc_function_visibility (name, vis); *slot = decl; } return XEXP (DECL_RTL (decl), 0); } +rtx +init_one_libfunc (const char *name) +{ + return init_one_libfunc_visibility (name, VISIBILITY_DEFAULT); +} + /* Adjust the assembler name of libfunc NAME to ASMSPEC. */ rtx From patchwork Tue Oct 2 16:19:11 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 147990 Delivered-To: patch@linaro.org Received: by 2002:a2e:8595:0:0:0:0:0 with SMTP id b21-v6csp5355772lji; Tue, 2 Oct 2018 09:21:11 -0700 (PDT) X-Google-Smtp-Source: ACcGV613UlffcKrtpTxcLXpC8a10eylw8PbVyo/54ClEjL6LcXSl+frGmwjpLmqJPNi083kUCmoX X-Received: by 2002:a63:4563:: with SMTP id u35-v6mr15233789pgk.30.1538497271269; Tue, 02 Oct 2018 09:21:11 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1538497271; cv=none; d=google.com; s=arc-20160816; b=WgqWFvrWOdaCkOTfd51QujOCmEwwP3sGSrLrVXLH9RUdqMf2jDUDw2zrfkAJHr1/71 Pn1++axaJ5lJE0k782PQ4dZ/eOjx2v+Z7sPn2pRqRKwDv3X3ZRsL0zkwrrzowlry4XlF aVsr1WOJGxEArPFXMSd5kzbCqtx5PhHWnCYpzqLRPIeOd/RUHjnwzCpHWDiKbq7qGvCG skBfDUjnPVJxGWhkO00IG6C2pZnQKdcgDOr9G5pJSDj5yJBysOwfmsOzFMEcq0ObMjs+ Cp/kyq9UZPkng78WA+TjPzOF0hPBludANqr4KXcKBa926EfqklfGcFCHQH0fZQnxX3JM JdkQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature:delivered-to:sender:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:mailing-list:dkim-signature :domainkey-signature; bh=dAgDzoC0CFG7e3YGtXRGQe+b8hoswMXYofI4xXw2FiI=; b=btrrUWmlFddSdnzWKaP2iiVzXW9pdvchkBbCFg1C4nyXlrtwgEgGBibAr9P0g+m70+ a5KqV67UzRwqwDEhoqdWl6Or1HywX7BWh65OuighFyGvq6hWbcvMTp4meF7c13aYp70g E16bMiWDNMLhjScEBvAscSW9DPAn6yTN4Mq7FJ+hlUS3GQt1zymHHp8Hm4IIlYjA+iT1 uddhWYtUnuyoc5TsegpV0cylCOPsp3EtMwBhjF1IqG4OWwVa7VPhs/SO/Pg09fGXV1rN xSO0Xlm1SDm6h5JcoXKOeNQDe0Hjf2luOuKZeM25LpbAS12b4lyCdy2vEJmop+XuP96Z IvxQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gcc.gnu.org header.s=default header.b=IiRF73Ha; dkim=pass header.i=@linaro.org header.s=google header.b=fPTyvIz3; spf=pass (google.com: domain of gcc-patches-return-486822-patch=linaro.org@gcc.gnu.org designates 209.132.180.131 as permitted sender) smtp.mailfrom="gcc-patches-return-486822-patch=linaro.org@gcc.gnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from sourceware.org (server1.sourceware.org. [209.132.180.131]) by mx.google.com with ESMTPS id cb8-v6si17163963plb.178.2018.10.02.09.21.10 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 02 Oct 2018 09:21:11 -0700 (PDT) Received-SPF: pass (google.com: domain of gcc-patches-return-486822-patch=linaro.org@gcc.gnu.org designates 209.132.180.131 as permitted sender) client-ip=209.132.180.131; Authentication-Results: mx.google.com; dkim=pass header.i=@gcc.gnu.org header.s=default header.b=IiRF73Ha; dkim=pass header.i=@linaro.org header.s=google header.b=fPTyvIz3; spf=pass (google.com: domain of gcc-patches-return-486822-patch=linaro.org@gcc.gnu.org designates 209.132.180.131 as permitted sender) smtp.mailfrom="gcc-patches-return-486822-patch=linaro.org@gcc.gnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org DomainKey-Signature: a=rsa-sha1; c=nofws; d=gcc.gnu.org; h=list-id :list-unsubscribe:list-archive:list-post:list-help:sender:from :to:cc:subject:date:message-id:in-reply-to:references; q=dns; s= default; b=Tnrf7Ftej6PhLMPxa+4tXNF+zUH8hY63iiSpBK+7Dd1MIQiQtoKEj zDU+9WBfR72jQfxALVvbAjkf0fL5CISEN03Omja3TxLAG2Hhtq080ocJsBbJfQg+ fwPeW7y/UUn4kHNPSFExAHId4SU79frz4/wvh96tNyszrc155ADI+w= DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=gcc.gnu.org; h=list-id :list-unsubscribe:list-archive:list-post:list-help:sender:from :to:cc:subject:date:message-id:in-reply-to:references; s= default; bh=YvCJxDUrIVnWL4ZwyzAgXh6N/0Y=; b=IiRF73HabydkEcjfmYoq YlHEoVs9AZUwvoZQvcvzDeCiXtr0IcUyOUCfMnrltJSKQgnwDJshBFhw53v99eaX aPRe0r7pWS8Y7oPU2ll894eWQjAmWi7y0d1wnBph8KoqHTLgXJciE5VxarZzBO12 wxDT1YL+MKZXGtB1j97QPbw= Received: (qmail 67902 invoked by alias); 2 Oct 2018 16:19:38 -0000 Mailing-List: contact gcc-patches-help@gcc.gnu.org; run by ezmlm Precedence: bulk List-Id: List-Unsubscribe: List-Archive: List-Post: List-Help: Sender: gcc-patches-owner@gcc.gnu.org Delivered-To: mailing list gcc-patches@gcc.gnu.org Received: (qmail 67814 invoked by uid 89); 2 Oct 2018 16:19:37 -0000 Authentication-Results: sourceware.org; auth=none X-Spam-SWARE-Status: No, score=-26.9 required=5.0 tests=BAYES_00, GIT_PATCH_0, GIT_PATCH_1, GIT_PATCH_2, GIT_PATCH_3, KAM_SHORT, RCVD_IN_DNSWL_NONE, SPF_PASS autolearn=ham version=3.3.2 spammy=_, STOP, W4, w4 X-HELO: mail-oi1-f194.google.com Received: from mail-oi1-f194.google.com (HELO mail-oi1-f194.google.com) (209.85.167.194) by sourceware.org (qpsmtpd/0.93/v0.84-503-g423c35a) with ESMTP; Tue, 02 Oct 2018 16:19:34 +0000 Received: by mail-oi1-f194.google.com with SMTP id s69-v6so1980635oie.10 for ; Tue, 02 Oct 2018 09:19:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=dAgDzoC0CFG7e3YGtXRGQe+b8hoswMXYofI4xXw2FiI=; b=fPTyvIz32EzKNd7A58VYYwayVaakhSMO137pjRna1gd34O79rLZdDJoawNKRYBG9ay vDko5YrYzX9pB3juDptgpSGqvfjUz2AiROCI5Fa4QPRS8V7qrbYqM5dJMjwvZfhcv4oJ utQ0wa+io2jwnlF35AH5XwqPCtDwRZLaW9lAU= Return-Path: Received: from cloudburst.twiddle.net ([187.217.227.243]) by smtp.gmail.com with ESMTPSA id j15-v6sm145987oth.27.2018.10.02.09.19.30 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Tue, 02 Oct 2018 09:19:31 -0700 (PDT) From: Richard Henderson To: gcc-patches@gcc.gnu.org Cc: ramana.radhakrishnan@arm.com, agraf@suse.de, marcus.shawcroft@arm.com, james.greenhalgh@arm.com, richard.earnshaw@arm.com Subject: [PATCH, AArch64 v2 07/11] aarch64: Add out-of-line functions for LSE atomics Date: Tue, 2 Oct 2018 11:19:11 -0500 Message-Id: <20181002161915.18843-8-richard.henderson@linaro.org> In-Reply-To: <20181002161915.18843-1-richard.henderson@linaro.org> References: <20181002161915.18843-1-richard.henderson@linaro.org> This is the libgcc part of the interface -- providing the functions. Rationale is provided at the top of libgcc/config/aarch64/lse.c. * config/aarch64/lse.c: New file. * config/aarch64/t-lse: New file. * config.host: Add t-lse to all aarch64 tuples. --- libgcc/config/aarch64/lse.c | 260 ++++++++++++++++++++++++++++++++++++ libgcc/config.host | 4 + libgcc/config/aarch64/t-lse | 44 ++++++ 3 files changed, 308 insertions(+) create mode 100644 libgcc/config/aarch64/lse.c create mode 100644 libgcc/config/aarch64/t-lse -- 2.17.1 diff --git a/libgcc/config/aarch64/lse.c b/libgcc/config/aarch64/lse.c new file mode 100644 index 00000000000..68ca7df667b --- /dev/null +++ b/libgcc/config/aarch64/lse.c @@ -0,0 +1,260 @@ +/* Out-of-line LSE atomics for AArch64 architecture. + Copyright (C) 2018 Free Software Foundation, Inc. + Contributed by Linaro Ltd. + +This file is part of GCC. + +GCC is free software; you can redistribute it and/or modify it under +the terms of the GNU General Public License as published by the Free +Software Foundation; either version 3, or (at your option) any later +version. + +GCC is distributed in the hope that it will be useful, but WITHOUT ANY +WARRANTY; without even the implied warranty of MERCHANTABILITY or +FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License +for more details. + +Under Section 7 of GPL version 3, you are granted additional +permissions described in the GCC Runtime Library Exception, version +3.1, as published by the Free Software Foundation. + +You should have received a copy of the GNU General Public License and +a copy of the GCC Runtime Library Exception along with this program; +see the files COPYING3 and COPYING.RUNTIME respectively. If not, see +. */ + +/* + * The problem that we are trying to solve is operating system deployment + * of ARMv8.1-Atomics, also known as Large System Exensions (LSE). + * + * There are a number of potential solutions for this problem which have + * been proposed and rejected for various reasons. To recap: + * + * (1) Multiple builds. The dynamic linker will examine /lib64/atomics/ + * if HWCAP_ATOMICS is set, allowing entire libraries to be overwritten. + * However, not all Linux distributions are happy with multiple builds, + * and anyway it has no effect on main applications. + * + * (2) IFUNC. We could put these functions into libgcc_s.so, and have + * a single copy of each function for all DSOs. However, ARM is concerned + * that the branch-to-indirect-branch that is implied by using a PLT, + * as required by IFUNC, is too much overhead for smaller cpus. + * + * (3) Statically predicted direct branches. This is the approach that + * is taken here. These functions are linked into every DSO that uses them. + * All of the symbols are hidden, so that the functions are called via a + * direct branch. The choice of LSE vs non-LSE is done via one byte load + * followed by a well-predicted direct branch. The functions are compiled + * separately to minimize code size. + */ + +/* Define or declare the symbol gating the LSE implementations. */ +#ifndef L_have_atomics +extern +#endif +_Bool __aa64_have_atomics __attribute__((visibility("hidden"), nocommon)); + +/* The branch controlled by this test should be easily predicted, in that + it will, after constructors, always branch the same way. The expectation + is that systems that implement ARMv8.1-Atomics are "beefier" than those + that omit the extension. By arranging for the fall-through path to use + load-store-exclusive insns, we aid the branch predictor of the + smallest cpus. */ +#define have_atomics __builtin_expect (__aa64_have_atomics, 0) + +#ifdef L_have_atomics +/* Disable initialization of __aa64_have_atomics during bootstrap. */ +# ifndef inhibit_libc +# include +/* Disable initialization if the system headers are too old. */ +# if defined(AT_HWCAP) && defined(HWCAP_ATOMICS) +static void __attribute__((constructor)) +init_have_atomics (void) +{ + unsigned long hwcap = getauxval (AT_HWCAP); + __aa64_have_atomics = (hwcap & HWCAP_ATOMICS) != 0; +} +# endif /* HWCAP */ +# endif /* inhibit_libc */ +#else + +/* Tell the assembler to accept LSE instructions. */ +asm(".arch armv8-a+lse"); + +/* Turn size and memory model defines into mnemonic fragments. */ +#if SIZE == 1 +# define S "b" +# define MASK ", uxtb" +#elif SIZE == 2 +# define S "h" +# define MASK ", uxth" +#elif SIZE == 4 || SIZE == 8 +# define S "" +# define MASK "" +#else +# error +#endif + +#if SIZE < 8 +# define T unsigned int +# define W "w" +#else +# define T unsigned long long +# define W "" +#endif + +#if MODEL == 1 +# define SUFF _relax +# define A "" +# define L "" +#elif MODEL == 2 +# define SUFF _acq +# define A "a" +# define L "" +#elif MODEL == 3 +# define SUFF _rel +# define A "" +# define L "l" +#elif MODEL == 4 +# define SUFF _acq_rel +# define A "a" +# define L "l" +#else +# error +#endif + +#define NAME2(B, S, X) __aa64_ ## B ## S ## X +#define NAME1(B, S, X) NAME2(B, S, X) +#define NAME(BASE) NAME1(BASE, SIZE, SUFF) + +#define str1(S) #S +#define str(S) str1(S) + +#ifdef L_cas +T NAME(cas) (T cmp, T new, T *ptr) __attribute__((visibility("hidden"))); +T NAME(cas) (T cmp, T new, T *ptr) +{ + T old; + unsigned tmp; + + if (have_atomics) + __asm__("cas" A L S " %"W"0, %"W"2, %1" + : "=r"(old), "+m"(*ptr) : "r"(new), "0"(cmp)); + else + __asm__( + "0: " + "ld" A "xr"S" %"W"0, %1\n\t" + "cmp %"W"0, %"W"4" MASK "\n\t" + "bne 1f\n\t" + "st" L "xr"S" %w2, %"W"3, %1\n\t" + "cbnz %w2, 0b\n" + "1:" + : "=&r"(old), "+m"(*ptr), "=&r"(tmp) : "r"(new), "r"(cmp)); + + return old; +} +#endif + +#ifdef L_swp +T NAME(swp) (T new, T *ptr) __attribute__((visibility("hidden"))); +T NAME(swp) (T new, T *ptr) +{ + T old; + unsigned tmp; + + if (have_atomics) + __asm__("swp" A L S " %"W"2, %"W"0, %1" + : "=r"(old), "+m"(*ptr) : "r"(new)); + else + __asm__( + "0: " + "ld" A "xr"S" %"W"0, %1\n\t" + "st" L "xr"S" %w2, %"W"3, %1\n\t" + "cbnz %w2, 0b\n" + "1:" + : "=&r"(old), "+m"(*ptr), "=&r"(tmp) : "r"(new)); + + return old; +} +#endif + +#if defined(L_ldadd) || defined(L_ldclr) \ + || defined(L_ldeor) || defined(L_ldset) + +#ifdef L_ldadd +#define LDOP ldadd +#define OP add +#elif defined(L_ldclr) +#define LDOP ldclr +#define OP bic +#elif defined(L_ldeor) +#define LDOP ldeor +#define OP eor +#elif defined(L_ldset) +#define LDOP ldset +#define OP orr +#else +#error +#endif + +T NAME(LDOP) (T val, T *ptr) __attribute__((visibility("hidden"))); +T NAME(LDOP) (T val, T *ptr) +{ + T old; + unsigned tmp; + + if (have_atomics) + __asm__(str(LDOP) A L S " %"W"2, %"W"0, %1" + : "=r"(old), "+m"(*ptr) : "r"(val)); + else + __asm__( + "0: " + "ld" A "xr"S" %"W"0, %1\n\t" + str(OP) " %"W"2, %"W"0, %"W"3\n\t" + "st" L "xr"S" %w2, %"W"2, %1\n\t" + "cbnz %w2, 0b\n" + "1:" + : "=&r"(old), "+m"(*ptr), "=&r"(tmp) : "r"(val)); + + return old; +} +#endif + +#if defined(L_stadd) || defined(L_stclr) \ + || defined(L_steor) || defined(L_stset) + +#ifdef L_stadd +#define STOP stadd +#define OP add +#elif defined(L_stclr) +#define STOP stclr +#define OP bic +#elif defined(L_steor) +#define STOP steor +#define OP eor +#elif defined(L_stset) +#define STOP stset +#define OP orr +#else +#error +#endif + +void NAME(STOP) (T val, T *ptr) __attribute__((visibility("hidden"))); +void NAME(STOP) (T val, T *ptr) +{ + unsigned tmp; + + if (have_atomics) + __asm__(str(STOP) L S " %"W"1, %0" : "+m"(*ptr) : "r"(val)); + else + __asm__( + "0: " + "ldxr"S" %"W"1, %0\n\t" + str(OP) " %"W"1, %"W"1, %"W"2\n\t" + "st" L "xr"S" %w1, %"W"1, %0\n\t" + "cbnz %w1, 0b\n" + "1:" + : "+m"(*ptr), "=&r"(tmp) : "r"(val)); +} +#endif +#endif /* L_have_atomics */ diff --git a/libgcc/config.host b/libgcc/config.host index 029f6569caf..7e9a8b6bc8f 100644 --- a/libgcc/config.host +++ b/libgcc/config.host @@ -340,23 +340,27 @@ aarch64*-*-elf | aarch64*-*-rtems*) extra_parts="$extra_parts crtbegin.o crtend.o crti.o crtn.o" extra_parts="$extra_parts crtfastmath.o" tmake_file="${tmake_file} ${cpu_type}/t-aarch64" + tmake_file="${tmake_file} ${cpu_type}/t-lse t-slibgcc-libgcc" tmake_file="${tmake_file} ${cpu_type}/t-softfp t-softfp t-crtfm" md_unwind_header=aarch64/aarch64-unwind.h ;; aarch64*-*-freebsd*) extra_parts="$extra_parts crtfastmath.o" tmake_file="${tmake_file} ${cpu_type}/t-aarch64" + tmake_file="${tmake_file} ${cpu_type}/t-lse t-slibgcc-libgcc" tmake_file="${tmake_file} ${cpu_type}/t-softfp t-softfp t-crtfm" md_unwind_header=aarch64/freebsd-unwind.h ;; aarch64*-*-fuchsia*) tmake_file="${tmake_file} ${cpu_type}/t-aarch64" + tmake_file="${tmake_file} ${cpu_type}/t-lse t-slibgcc-libgcc" tmake_file="${tmake_file} ${cpu_type}/t-softfp t-softfp" ;; aarch64*-*-linux*) extra_parts="$extra_parts crtfastmath.o" md_unwind_header=aarch64/linux-unwind.h tmake_file="${tmake_file} ${cpu_type}/t-aarch64" + tmake_file="${tmake_file} ${cpu_type}/t-lse t-slibgcc-libgcc" tmake_file="${tmake_file} ${cpu_type}/t-softfp t-softfp t-crtfm" ;; alpha*-*-linux*) diff --git a/libgcc/config/aarch64/t-lse b/libgcc/config/aarch64/t-lse new file mode 100644 index 00000000000..e862b0c2448 --- /dev/null +++ b/libgcc/config/aarch64/t-lse @@ -0,0 +1,44 @@ +# Out-of-line LSE atomics for AArch64 architecture. +# Copyright (C) 2018 Free Software Foundation, Inc. +# Contributed by Linaro Ltd. +# +# This file is part of GCC. +# +# GCC is free software; you can redistribute it and/or modify it +# under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 3, or (at your option) +# any later version. +# +# GCC is distributed in the hope that it will be useful, but +# WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with GCC; see the file COPYING3. If not see +# . + +# CAS, Swap, Load-and-operate have 4 sizes and 4 memory models +S1 := $(foreach s, 1 2 4 8, $(addsuffix _$(s), cas swp ldadd ldclr ldeor ldset)) +O1 := $(foreach m, 1 2 3 4, $(addsuffix _$(m)$(objext), $(S1))) + +# Store-and-operate has 4 sizes but only 2 memory models (relaxed, release). +S2 := $(foreach s, 1 2 4 8, $(addsuffix _$(s), stadd stclr steor stset)) +O2 := $(foreach m, 1 3, $(addsuffix _$(m)$(objext), $(S2))) + +LSE_OBJS := $(O1) $(O2) + +libgcc-objects += $(LSE_OBJS) have_atomic$(objext) + +empty = +space = $(empty) $(empty) +PAT_SPLIT = $(subst _,$(space),$(*F)) +PAT_BASE = $(word 1,$(PAT_SPLIT)) +PAT_N = $(word 2,$(PAT_SPLIT)) +PAT_M = $(word 3,$(PAT_SPLIT)) + +have_atomic$(objext): $(srcdir)/config/aarch64/lse.c + $(gcc_compile) -DL_have_atomics -c $< + +$(LSE_OBJS): $(srcdir)/config/aarch64/lse.c + $(gcc_compile) -DL_$(PAT_BASE) -DSIZE=$(PAT_N) -DMODEL=$(PAT_M) -c $< From patchwork Tue Oct 2 16:19:12 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 147992 Delivered-To: patch@linaro.org Received: by 2002:a2e:8595:0:0:0:0:0 with SMTP id b21-v6csp5356254lji; Tue, 2 Oct 2018 09:21:36 -0700 (PDT) X-Google-Smtp-Source: ACcGV63z3gvy60LJztJM8Zzs6wlSJ4R5ANuISFjQRUdt5o2mG1QJ8XxUut0SsD+U6ug5ZlF36kmT X-Received: by 2002:a62:2845:: with SMTP id o66-v6mr16999557pfo.17.1538497295953; Tue, 02 Oct 2018 09:21:35 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1538497295; cv=none; d=google.com; s=arc-20160816; b=kpt65aegRu/bQebh+gKHJzznp2Aa1Cbv7y87o1ZjMfJZC1y0WxuVekLAcSrHEyopDd GMhE1zs1mH2RZPKJzY8gAz82EqftzlveSJCvsiXp+FqpX0Z6n4Zt1DNSZJ0R3zgXdfn4 UBfVIZyQS0sXVjmkneHwlV+rE0c6+v/Bvg025Q4IgBCuGDZVFyD0eHSUChaVYaVlyGlU 6icGbTk4mvGF/FFMhhlwyeAJ66/3uiZgNMUI2l/rGN1ZcwYnsdk6yQMQq457jgRA2H1B 8BJ43dTh6N/uj03z3mGYzBI4vhnIbbnyWlQYNMN2a/XYl6wWShJkuAUwsvU4wHzQd9bS svTw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature:delivered-to:sender:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:mailing-list:dkim-signature :domainkey-signature; bh=TnkmUZKwSeiVVvXBlseyaEdP1qX0dRubqwLyf4ACWd4=; b=D2u0YSjh0YSrMYrAOIMBvlg/rvh4lnC/dpi1w+2ziGRJVWGFNIPdH9KVetquGiEA/+ vAs+SWR87lXoKtpqQSZzS7Ljf5LQgZ6jh1g9Q73DVdewOGddzx0GQHW4MRd3/ym/9kEl he6CVsAcXqiHw2aoiJo+EB6x7681F8Dgl9ee/DgdhoR4lhUWOVkd0FXUvAqaDNAK+1E0 2Q01Jay75iAtTk3+Rz1lqkoAuO5Kzu1RIiE1WTi+t02hrF2epuQkXQJwnV4T5kW3hPQS bTG9YC5hXpoBUVI3dWvEBAVrWSYaeNuVmcF3xfaBSyakyDda6eXCUqyBxcDrI0SnHUhu Gdnw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gcc.gnu.org header.s=default header.b=KrhPdOTR; dkim=pass header.i=@linaro.org header.s=google header.b=U0wS9Dp3; spf=pass (google.com: domain of gcc-patches-return-486824-patch=linaro.org@gcc.gnu.org designates 209.132.180.131 as permitted sender) smtp.mailfrom="gcc-patches-return-486824-patch=linaro.org@gcc.gnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from sourceware.org (server1.sourceware.org. [209.132.180.131]) by mx.google.com with ESMTPS id f6-v6si16046766plt.16.2018.10.02.09.21.35 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 02 Oct 2018 09:21:35 -0700 (PDT) Received-SPF: pass (google.com: domain of gcc-patches-return-486824-patch=linaro.org@gcc.gnu.org designates 209.132.180.131 as permitted sender) client-ip=209.132.180.131; Authentication-Results: mx.google.com; dkim=pass header.i=@gcc.gnu.org header.s=default header.b=KrhPdOTR; dkim=pass header.i=@linaro.org header.s=google header.b=U0wS9Dp3; spf=pass (google.com: domain of gcc-patches-return-486824-patch=linaro.org@gcc.gnu.org designates 209.132.180.131 as permitted sender) smtp.mailfrom="gcc-patches-return-486824-patch=linaro.org@gcc.gnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org DomainKey-Signature: a=rsa-sha1; c=nofws; d=gcc.gnu.org; h=list-id :list-unsubscribe:list-archive:list-post:list-help:sender:from :to:cc:subject:date:message-id:in-reply-to:references; q=dns; s= default; b=lb+HdUYuV2O/TtG1whGRFilDwjc3DWUlXYDmvlvWbk1Jf+BSxnNvO sB6d0lBD3xMo/NaNnUl922p9O90MzTwURUEZdwgLcy+OJkrSmx5G9nGY1Ky7wF+k 56Vcb/bBnhjlzLbfLigh0o0pqIBLNtRdLV1GKlf49j5udOIk6UklIU= DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=gcc.gnu.org; h=list-id :list-unsubscribe:list-archive:list-post:list-help:sender:from :to:cc:subject:date:message-id:in-reply-to:references; s= default; bh=B7ZCur5PTJdiamG4KgJeu10MPuU=; b=KrhPdOTRdCgHT6IKqR0V LIREId1Fvi7YG0UYBiaYRyr4D7ogPmt/JDzrg33s7jU5yMMXrfgzKndA6YWx3hzd 78wXVDxonfJi0jgwm5a9/Yk9QOiJa4HkULO2YwIbI8z+XIOmtYOcsxo82WEfgrAK sj1hqFp4uyMLeCKE0u6EPLQ= Received: (qmail 68382 invoked by alias); 2 Oct 2018 16:19:41 -0000 Mailing-List: contact gcc-patches-help@gcc.gnu.org; run by ezmlm Precedence: bulk List-Id: List-Unsubscribe: List-Archive: List-Post: List-Help: Sender: gcc-patches-owner@gcc.gnu.org Delivered-To: mailing list gcc-patches@gcc.gnu.org Received: (qmail 68276 invoked by uid 89); 2 Oct 2018 16:19:40 -0000 Authentication-Results: sourceware.org; auth=none X-Spam-SWARE-Status: No, score=-26.9 required=5.0 tests=BAYES_00, GIT_PATCH_0, GIT_PATCH_1, GIT_PATCH_2, GIT_PATCH_3, RCVD_IN_DNSWL_NONE, SPF_PASS autolearn=ham version=3.3.2 spammy=applicable, NAMES, relax X-HELO: mail-ot1-f41.google.com Received: from mail-ot1-f41.google.com (HELO mail-ot1-f41.google.com) (209.85.210.41) by sourceware.org (qpsmtpd/0.93/v0.84-503-g423c35a) with ESMTP; Tue, 02 Oct 2018 16:19:36 +0000 Received: by mail-ot1-f41.google.com with SMTP id h26-v6so2463784otl.9 for ; Tue, 02 Oct 2018 09:19:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=TnkmUZKwSeiVVvXBlseyaEdP1qX0dRubqwLyf4ACWd4=; b=U0wS9Dp3w3oLRKQCQi/7v7mF4okdf4AM/zdOPzfXIC8/P6CBTY005zH1LcrfGfw8G4 T/9EfdUr2RkZ9Uh9Ph9ZnFGB0n3p/X3cMYyR9aINq1Toc9vZ+nPWcOdzD4MCdbZ9RUYe QpeEVbNSRtdYmfoigiSLQFWPSXKntk8E2P7ls= Return-Path: Received: from cloudburst.twiddle.net ([187.217.227.243]) by smtp.gmail.com with ESMTPSA id j15-v6sm145987oth.27.2018.10.02.09.19.32 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Tue, 02 Oct 2018 09:19:33 -0700 (PDT) From: Richard Henderson To: gcc-patches@gcc.gnu.org Cc: ramana.radhakrishnan@arm.com, agraf@suse.de, marcus.shawcroft@arm.com, james.greenhalgh@arm.com, richard.earnshaw@arm.com Subject: [PATCH, AArch64 v2 08/11] aarch64: Implement -matomic-ool Date: Tue, 2 Oct 2018 11:19:12 -0500 Message-Id: <20181002161915.18843-9-richard.henderson@linaro.org> In-Reply-To: <20181002161915.18843-1-richard.henderson@linaro.org> References: <20181002161915.18843-1-richard.henderson@linaro.org> * config/aarch64/aarch64.opt (-matomic-ool): New. * config/aarch64/aarch64.c (aarch64_atomic_ool_func): New. (aarch64_ool_cas_names, aarch64_ool_swp_names): New. (aarch64_ool_ldadd_names, aarch64_ool_ldset_names): New. (aarch64_ool_ldclr_names, aarch64_ool_ldeor_names): New. (aarch64_ool_stadd_names, aarch64_ool_stset_names): New. (aarch64_ool_stclr_names, aarch64_ool_steor_names): New. (aarch64_expand_compare_and_swap): Honor TARGET_ATOMIC_OOL. * config/aarch64/atomics.md (atomic_exchange): Likewise. (atomic_): Likewise. (atomic_fetch_): Likewise. (atomic__fetch): Likewise. --- gcc/config/aarch64/aarch64-protos.h | 17 ++++ gcc/config/aarch64/aarch64.c | 95 +++++++++++++++++++ .../atomic-comp-swap-release-acquire.c | 2 +- .../gcc.target/aarch64/atomic-op-acq_rel.c | 2 +- .../gcc.target/aarch64/atomic-op-acquire.c | 2 +- .../gcc.target/aarch64/atomic-op-char.c | 2 +- .../gcc.target/aarch64/atomic-op-consume.c | 2 +- .../gcc.target/aarch64/atomic-op-imm.c | 2 +- .../gcc.target/aarch64/atomic-op-int.c | 2 +- .../gcc.target/aarch64/atomic-op-long.c | 2 +- .../gcc.target/aarch64/atomic-op-relaxed.c | 2 +- .../gcc.target/aarch64/atomic-op-release.c | 2 +- .../gcc.target/aarch64/atomic-op-seq_cst.c | 2 +- .../gcc.target/aarch64/atomic-op-short.c | 2 +- .../aarch64/atomic_cmp_exchange_zero_reg_1.c | 2 +- .../atomic_cmp_exchange_zero_strong_1.c | 2 +- .../gcc.target/aarch64/sync-comp-swap.c | 2 +- .../gcc.target/aarch64/sync-op-acquire.c | 2 +- .../gcc.target/aarch64/sync-op-full.c | 2 +- gcc/config/aarch64/aarch64.opt | 4 + gcc/config/aarch64/atomics.md | 94 ++++++++++++++++-- gcc/doc/invoke.texi | 14 ++- 22 files changed, 232 insertions(+), 26 deletions(-) -- 2.17.1 diff --git a/gcc/config/aarch64/aarch64-protos.h b/gcc/config/aarch64/aarch64-protos.h index 1d2f8487d1a..c7b96b12bbe 100644 --- a/gcc/config/aarch64/aarch64-protos.h +++ b/gcc/config/aarch64/aarch64-protos.h @@ -624,4 +624,21 @@ poly_uint64 aarch64_regmode_natural_size (machine_mode); bool aarch64_high_bits_all_ones_p (HOST_WIDE_INT); +struct atomic_ool_names +{ + const char *str[4][4]; +}; + +rtx aarch64_atomic_ool_func(machine_mode mode, rtx model_rtx, + const atomic_ool_names *names); +extern const atomic_ool_names aarch64_ool_swp_names; +extern const atomic_ool_names aarch64_ool_stadd_names; +extern const atomic_ool_names aarch64_ool_stset_names; +extern const atomic_ool_names aarch64_ool_stclr_names; +extern const atomic_ool_names aarch64_ool_steor_names; +extern const atomic_ool_names aarch64_ool_ldadd_names; +extern const atomic_ool_names aarch64_ool_ldset_names; +extern const atomic_ool_names aarch64_ool_ldclr_names; +extern const atomic_ool_names aarch64_ool_ldeor_names; + #endif /* GCC_AARCH64_PROTOS_H */ diff --git a/gcc/config/aarch64/aarch64.c b/gcc/config/aarch64/aarch64.c index 867759f7e80..49b47382b5d 100644 --- a/gcc/config/aarch64/aarch64.c +++ b/gcc/config/aarch64/aarch64.c @@ -14160,6 +14160,90 @@ aarch64_emit_unlikely_jump (rtx insn) add_reg_br_prob_note (jump, profile_probability::very_unlikely ()); } +/* We store the names of the various atomic helpers in a 4x4 array. + Return the libcall function given MODE, MODEL and NAMES. */ + +rtx +aarch64_atomic_ool_func(machine_mode mode, rtx model_rtx, + const atomic_ool_names *names) +{ + memmodel model = memmodel_base (INTVAL (model_rtx)); + int mode_idx, model_idx; + + switch (mode) + { + case E_QImode: + mode_idx = 0; + break; + case E_HImode: + mode_idx = 1; + break; + case E_SImode: + mode_idx = 2; + break; + case E_DImode: + mode_idx = 3; + break; + default: + gcc_unreachable (); + } + + switch (model) + { + case MEMMODEL_RELAXED: + model_idx = 0; + break; + case MEMMODEL_CONSUME: + case MEMMODEL_ACQUIRE: + model_idx = 1; + break; + case MEMMODEL_RELEASE: + model_idx = 2; + break; + case MEMMODEL_ACQ_REL: + case MEMMODEL_SEQ_CST: + model_idx = 3; + break; + default: + gcc_unreachable (); + } + + return init_one_libfunc_visibility (names->str[mode_idx][model_idx], + VISIBILITY_HIDDEN); +} + +#define DEF0(B, N) \ + { "__aa64_" #B #N "_relax", \ + "__aa64_" #B #N "_acq", \ + "__aa64_" #B #N "_rel", \ + "__aa64_" #B #N "_acq_rel" } + +#define DEF4(B) DEF0(B, 1), DEF0(B, 2), DEF0(B, 4), DEF0(B, 8) + +static const atomic_ool_names aarch64_ool_cas_names = { { DEF4(cas) } }; +const atomic_ool_names aarch64_ool_swp_names = { { DEF4(swp) } }; +const atomic_ool_names aarch64_ool_ldadd_names = { { DEF4(ldadd) } }; +const atomic_ool_names aarch64_ool_ldset_names = { { DEF4(ldset) } }; +const atomic_ool_names aarch64_ool_ldclr_names = { { DEF4(ldclr) } }; +const atomic_ool_names aarch64_ool_ldeor_names = { { DEF4(ldeor) } }; + +/* Note that the store forms are only available for relax and release + memory models. For the other models, re-use the load forms. */ +#undef DEF0 +#define DEF0(B, N) \ + { "__aa64_st" #B #N "_relax", \ + "__aa64_ld" #B #N "_acq", \ + "__aa64_st" #B #N "_rel", \ + "__aa64_ld" #B #N "_acq_rel" } + +const atomic_ool_names aarch64_ool_stadd_names = { { DEF4(add) } }; +const atomic_ool_names aarch64_ool_stset_names = { { DEF4(set) } }; +const atomic_ool_names aarch64_ool_stclr_names = { { DEF4(clr) } }; +const atomic_ool_names aarch64_ool_steor_names = { { DEF4(eor) } }; + +#undef DEF0 +#undef DEF4 + /* Expand a compare and swap pattern. */ void @@ -14206,6 +14290,17 @@ aarch64_expand_compare_and_swap (rtx operands[]) newval, mod_s)); cc_reg = aarch64_gen_compare_reg_maybe_ze (NE, rval, oldval, mode); } + else if (TARGET_ATOMIC_OOL) + { + /* Oldval must satisfy compare afterward. */ + if (!aarch64_plus_operand (oldval, mode)) + oldval = force_reg (mode, oldval); + rtx func = aarch64_atomic_ool_func (mode, mod_s, &aarch64_ool_cas_names); + rval = emit_library_call_value (func, NULL_RTX, LCT_NORMAL, r_mode, + oldval, mode, newval, mode, + XEXP (mem, 0), ptr_mode); + cc_reg = aarch64_gen_compare_reg_maybe_ze (NE, rval, oldval, mode); + } else { /* The oldval predicate varies by mode. Test it and force to reg. */ diff --git a/gcc/testsuite/gcc.target/aarch64/atomic-comp-swap-release-acquire.c b/gcc/testsuite/gcc.target/aarch64/atomic-comp-swap-release-acquire.c index 49ca5d0d09c..e92f205c3a8 100644 --- a/gcc/testsuite/gcc.target/aarch64/atomic-comp-swap-release-acquire.c +++ b/gcc/testsuite/gcc.target/aarch64/atomic-comp-swap-release-acquire.c @@ -1,5 +1,5 @@ /* { dg-do compile } */ -/* { dg-options "-march=armv8-a+nolse -O2 -fno-ipa-icf" } */ +/* { dg-options "-march=armv8-a+nolse -O2 -fno-ipa-icf -mno-atomic-ool" } */ #include "atomic-comp-swap-release-acquire.x" diff --git a/gcc/testsuite/gcc.target/aarch64/atomic-op-acq_rel.c b/gcc/testsuite/gcc.target/aarch64/atomic-op-acq_rel.c index 74f26348e42..6965431f7d9 100644 --- a/gcc/testsuite/gcc.target/aarch64/atomic-op-acq_rel.c +++ b/gcc/testsuite/gcc.target/aarch64/atomic-op-acq_rel.c @@ -1,5 +1,5 @@ /* { dg-do compile } */ -/* { dg-options "-march=armv8-a+nolse -O2" } */ +/* { dg-options "-march=armv8-a+nolse -O2 -mno-atomic-ool" } */ #include "atomic-op-acq_rel.x" diff --git a/gcc/testsuite/gcc.target/aarch64/atomic-op-acquire.c b/gcc/testsuite/gcc.target/aarch64/atomic-op-acquire.c index 66c1b1efe20..07dbca49d56 100644 --- a/gcc/testsuite/gcc.target/aarch64/atomic-op-acquire.c +++ b/gcc/testsuite/gcc.target/aarch64/atomic-op-acquire.c @@ -1,5 +1,5 @@ /* { dg-do compile } */ -/* { dg-options "-march=armv8-a+nolse -O2" } */ +/* { dg-options "-march=armv8-a+nolse -O2 -mno-atomic-ool" } */ #include "atomic-op-acquire.x" diff --git a/gcc/testsuite/gcc.target/aarch64/atomic-op-char.c b/gcc/testsuite/gcc.target/aarch64/atomic-op-char.c index c09d0434ecf..73bfbb7afc9 100644 --- a/gcc/testsuite/gcc.target/aarch64/atomic-op-char.c +++ b/gcc/testsuite/gcc.target/aarch64/atomic-op-char.c @@ -1,5 +1,5 @@ /* { dg-do compile } */ -/* { dg-options "-march=armv8-a+nolse -O2" } */ +/* { dg-options "-march=armv8-a+nolse -O2 -mno-atomic-ool" } */ #include "atomic-op-char.x" diff --git a/gcc/testsuite/gcc.target/aarch64/atomic-op-consume.c b/gcc/testsuite/gcc.target/aarch64/atomic-op-consume.c index 5783ab84f5c..c7945b3a22d 100644 --- a/gcc/testsuite/gcc.target/aarch64/atomic-op-consume.c +++ b/gcc/testsuite/gcc.target/aarch64/atomic-op-consume.c @@ -1,5 +1,5 @@ /* { dg-do compile } */ -/* { dg-options "-march=armv8-a+nolse -O2" } */ +/* { dg-options "-march=armv8-a+nolse -O2 -mno-atomic-ool" } */ #include "atomic-op-consume.x" diff --git a/gcc/testsuite/gcc.target/aarch64/atomic-op-imm.c b/gcc/testsuite/gcc.target/aarch64/atomic-op-imm.c index 18b8f0b04e9..e46bb3de7c1 100644 --- a/gcc/testsuite/gcc.target/aarch64/atomic-op-imm.c +++ b/gcc/testsuite/gcc.target/aarch64/atomic-op-imm.c @@ -1,5 +1,5 @@ /* { dg-do compile } */ -/* { dg-options "-march=armv8-a+nolse -O2" } */ +/* { dg-options "-march=armv8-a+nolse -O2 -mno-atomic-ool" } */ int v = 0; diff --git a/gcc/testsuite/gcc.target/aarch64/atomic-op-int.c b/gcc/testsuite/gcc.target/aarch64/atomic-op-int.c index 8520f0839ba..9b55deb5225 100644 --- a/gcc/testsuite/gcc.target/aarch64/atomic-op-int.c +++ b/gcc/testsuite/gcc.target/aarch64/atomic-op-int.c @@ -1,5 +1,5 @@ /* { dg-do compile } */ -/* { dg-options "-march=armv8-a+nolse -O2" } */ +/* { dg-options "-march=armv8-a+nolse -O2 -mno-atomic-ool" } */ #include "atomic-op-int.x" diff --git a/gcc/testsuite/gcc.target/aarch64/atomic-op-long.c b/gcc/testsuite/gcc.target/aarch64/atomic-op-long.c index d011f8c5ce2..2622f75331f 100644 --- a/gcc/testsuite/gcc.target/aarch64/atomic-op-long.c +++ b/gcc/testsuite/gcc.target/aarch64/atomic-op-long.c @@ -1,5 +1,5 @@ /* { dg-do compile } */ -/* { dg-options "-march=armv8-a+nolse -O2" } */ +/* { dg-options "-march=armv8-a+nolse -O2 -mno-atomic-ool" } */ long v = 0; diff --git a/gcc/testsuite/gcc.target/aarch64/atomic-op-relaxed.c b/gcc/testsuite/gcc.target/aarch64/atomic-op-relaxed.c index ed96bfdb978..f118a37a352 100644 --- a/gcc/testsuite/gcc.target/aarch64/atomic-op-relaxed.c +++ b/gcc/testsuite/gcc.target/aarch64/atomic-op-relaxed.c @@ -1,5 +1,5 @@ /* { dg-do compile } */ -/* { dg-options "-march=armv8-a+nolse -O2" } */ +/* { dg-options "-march=armv8-a+nolse -O2 -mno-atomic-ool" } */ #include "atomic-op-relaxed.x" diff --git a/gcc/testsuite/gcc.target/aarch64/atomic-op-release.c b/gcc/testsuite/gcc.target/aarch64/atomic-op-release.c index fc4be17de89..579634b08e8 100644 --- a/gcc/testsuite/gcc.target/aarch64/atomic-op-release.c +++ b/gcc/testsuite/gcc.target/aarch64/atomic-op-release.c @@ -1,5 +1,5 @@ /* { dg-do compile } */ -/* { dg-options "-march=armv8-a+nolse -O2" } */ +/* { dg-options "-march=armv8-a+nolse -O2 -mno-atomic-ool" } */ #include "atomic-op-release.x" diff --git a/gcc/testsuite/gcc.target/aarch64/atomic-op-seq_cst.c b/gcc/testsuite/gcc.target/aarch64/atomic-op-seq_cst.c index 613000fe490..016b0d6619f 100644 --- a/gcc/testsuite/gcc.target/aarch64/atomic-op-seq_cst.c +++ b/gcc/testsuite/gcc.target/aarch64/atomic-op-seq_cst.c @@ -1,5 +1,5 @@ /* { dg-do compile } */ -/* { dg-options "-march=armv8-a+nolse -O2" } */ +/* { dg-options "-march=armv8-a+nolse -O2 -mno-atomic-ool" } */ #include "atomic-op-seq_cst.x" diff --git a/gcc/testsuite/gcc.target/aarch64/atomic-op-short.c b/gcc/testsuite/gcc.target/aarch64/atomic-op-short.c index e82c8118ece..978bd1d8377 100644 --- a/gcc/testsuite/gcc.target/aarch64/atomic-op-short.c +++ b/gcc/testsuite/gcc.target/aarch64/atomic-op-short.c @@ -1,5 +1,5 @@ /* { dg-do compile } */ -/* { dg-options "-march=armv8-a+nolse -O2" } */ +/* { dg-options "-march=armv8-a+nolse -O2 -mno-atomic-ool" } */ #include "atomic-op-short.x" diff --git a/gcc/testsuite/gcc.target/aarch64/atomic_cmp_exchange_zero_reg_1.c b/gcc/testsuite/gcc.target/aarch64/atomic_cmp_exchange_zero_reg_1.c index f2a21ddf2e1..77430ecdbce 100644 --- a/gcc/testsuite/gcc.target/aarch64/atomic_cmp_exchange_zero_reg_1.c +++ b/gcc/testsuite/gcc.target/aarch64/atomic_cmp_exchange_zero_reg_1.c @@ -1,5 +1,5 @@ /* { dg-do compile } */ -/* { dg-options "-O2 -march=armv8-a+nolse" } */ +/* { dg-options "-O2 -march=armv8-a+nolse -mno-atomic-ool" } */ /* { dg-skip-if "" { *-*-* } { "-mcpu=*" } { "" } } */ int diff --git a/gcc/testsuite/gcc.target/aarch64/atomic_cmp_exchange_zero_strong_1.c b/gcc/testsuite/gcc.target/aarch64/atomic_cmp_exchange_zero_strong_1.c index 8d2ae67dfbe..7d58b2f6bd0 100644 --- a/gcc/testsuite/gcc.target/aarch64/atomic_cmp_exchange_zero_strong_1.c +++ b/gcc/testsuite/gcc.target/aarch64/atomic_cmp_exchange_zero_strong_1.c @@ -1,5 +1,5 @@ /* { dg-do compile } */ -/* { dg-options "-O2 -march=armv8-a+nolse" } */ +/* { dg-options "-O2 -march=armv8-a+nolse -mno-atomic-ool" } */ /* { dg-skip-if "" { *-*-* } { "-mcpu=*" } { "" } } */ int diff --git a/gcc/testsuite/gcc.target/aarch64/sync-comp-swap.c b/gcc/testsuite/gcc.target/aarch64/sync-comp-swap.c index e571b2f13b3..7fc5885d0fd 100644 --- a/gcc/testsuite/gcc.target/aarch64/sync-comp-swap.c +++ b/gcc/testsuite/gcc.target/aarch64/sync-comp-swap.c @@ -1,5 +1,5 @@ /* { dg-do compile } */ -/* { dg-options "-march=armv8-a+nolse -O2 -fno-ipa-icf" } */ +/* { dg-options "-march=armv8-a+nolse -O2 -fno-ipa-icf -mno-atomic-ool" } */ #include "sync-comp-swap.x" diff --git a/gcc/testsuite/gcc.target/aarch64/sync-op-acquire.c b/gcc/testsuite/gcc.target/aarch64/sync-op-acquire.c index 357bf1be3b2..6ad0daa8998 100644 --- a/gcc/testsuite/gcc.target/aarch64/sync-op-acquire.c +++ b/gcc/testsuite/gcc.target/aarch64/sync-op-acquire.c @@ -1,5 +1,5 @@ /* { dg-do compile } */ -/* { dg-options "-march=armv8-a+nolse -O2" } */ +/* { dg-options "-march=armv8-a+nolse -O2 -mno-atomic-ool" } */ #include "sync-op-acquire.x" diff --git a/gcc/testsuite/gcc.target/aarch64/sync-op-full.c b/gcc/testsuite/gcc.target/aarch64/sync-op-full.c index c6ba1629965..9a7afeb70d3 100644 --- a/gcc/testsuite/gcc.target/aarch64/sync-op-full.c +++ b/gcc/testsuite/gcc.target/aarch64/sync-op-full.c @@ -1,5 +1,5 @@ /* { dg-do compile } */ -/* { dg-options "-march=armv8-a+nolse -O2" } */ +/* { dg-options "-march=armv8-a+nolse -O2 -mno-atomic-ool" } */ #include "sync-op-full.x" diff --git a/gcc/config/aarch64/aarch64.opt b/gcc/config/aarch64/aarch64.opt index b2e80cbf6f1..83166834165 100644 --- a/gcc/config/aarch64/aarch64.opt +++ b/gcc/config/aarch64/aarch64.opt @@ -218,3 +218,7 @@ Enables verbose cost model dumping in the debug dump files. mtrack-speculation Target Var(aarch64_track_speculation) Generate code to track when the CPU might be speculating incorrectly. + +matomic-ool +Target Report Mask(ATOMIC_OOL) Save +Generate local calls to out-of-line atomic operations. diff --git a/gcc/config/aarch64/atomics.md b/gcc/config/aarch64/atomics.md index ee662a4480e..72f9962fe55 100644 --- a/gcc/config/aarch64/atomics.md +++ b/gcc/config/aarch64/atomics.md @@ -140,16 +140,27 @@ (match_operand:SI 3 "const_int_operand" "")] "" { - rtx (*gen) (rtx, rtx, rtx, rtx); - /* Use an atomic SWP when available. */ if (TARGET_LSE) - gen = gen_aarch64_atomic_exchange_lse; + { + emit_insn (gen_aarch64_atomic_exchange_lse + (operands[0], operands[1], operands[2], operands[3])); + } + else if (TARGET_ATOMIC_OOL) + { + machine_mode mode = mode; + rtx func = aarch64_atomic_ool_func (mode, operands[3], + &aarch64_ool_swp_names); + rtx rval = emit_library_call_value (func, operands[0], LCT_NORMAL, + mode, operands[2], mode, + XEXP (operands[1], 0), ptr_mode); + emit_move_insn (operands[0], rval); + } else - gen = gen_aarch64_atomic_exchange; - - emit_insn (gen (operands[0], operands[1], operands[2], operands[3])); - + { + emit_insn (gen_aarch64_atomic_exchange + (operands[0], operands[1], operands[2], operands[3])); + } DONE; } ) @@ -234,6 +245,39 @@ } operands[1] = force_reg (mode, operands[1]); } + else if (TARGET_ATOMIC_OOL) + { + const atomic_ool_names *names; + switch () + { + case MINUS: + operands[1] = expand_simple_unop (mode, NEG, operands[1], + NULL, 1); + /* fallthru */ + case PLUS: + names = &aarch64_ool_stadd_names; + break; + case IOR: + names = &aarch64_ool_stset_names; + break; + case XOR: + names = &aarch64_ool_steor_names; + break; + case AND: + operands[1] = expand_simple_unop (mode, NOT, operands[1], + NULL, 1); + names = &aarch64_ool_stclr_names; + break; + default: + gcc_unreachable (); + } + machine_mode mode = mode; + rtx func = aarch64_atomic_ool_func (mode, operands[2], names); + emit_library_call_value (func, NULL_RTX, LCT_NORMAL, mode, + operands[1], mode, + XEXP (operands[0], 0), ptr_mode); + DONE; + } else gen = gen_aarch64_atomic_; @@ -350,6 +394,40 @@ } operands[2] = force_reg (mode, operands[2]); } + else if (TARGET_ATOMIC_OOL) + { + const atomic_ool_names *names; + switch () + { + case MINUS: + operands[2] = expand_simple_unop (mode, NEG, operands[2], + NULL, 1); + /* fallthru */ + case PLUS: + names = &aarch64_ool_ldadd_names; + break; + case IOR: + names = &aarch64_ool_ldset_names; + break; + case XOR: + names = &aarch64_ool_ldeor_names; + break; + case AND: + operands[2] = expand_simple_unop (mode, NOT, operands[2], + NULL, 1); + names = &aarch64_ool_ldclr_names; + break; + default: + gcc_unreachable (); + } + machine_mode mode = mode; + rtx func = aarch64_atomic_ool_func (mode, operands[3], names); + rtx rval = emit_library_call_value (func, operands[0], LCT_NORMAL, mode, + operands[2], mode, + XEXP (operands[1], 0), ptr_mode); + emit_move_insn (operands[0], rval); + DONE; + } else gen = gen_aarch64_atomic_fetch_; @@ -439,7 +517,7 @@ { /* Use an atomic load-operate instruction when possible. In this case we will re-compute the result from the original mem value. */ - if (TARGET_LSE) + if (TARGET_LSE || TARGET_ATOMIC_OOL) { rtx tmp = gen_reg_rtx (mode); operands[2] = force_reg (mode, operands[2]); diff --git a/gcc/doc/invoke.texi b/gcc/doc/invoke.texi index 5c95f674e0c..c947abea899 100644 --- a/gcc/doc/invoke.texi +++ b/gcc/doc/invoke.texi @@ -621,7 +621,7 @@ Objective-C and Objective-C++ Dialects}. -mpc-relative-literal-loads @gol -msign-return-address=@var{scope} @gol -march=@var{name} -mcpu=@var{name} -mtune=@var{name} @gol --moverride=@var{string} -mverbose-cost-dump -mtrack-speculation} +-moverride=@var{string} -mverbose-cost-dump -mtrack-speculation -matomic-ool} @emph{Adapteva Epiphany Options} @gccoptlist{-mhalf-reg-file -mprefer-short-insn-regs @gol @@ -15057,6 +15057,18 @@ be used by the compiler when expanding calls to @code{__builtin_speculation_safe_copy} to permit a more efficient code sequence to be generated. +@item -matomic-ool +@itemx -mno-atomic-ool +Enable or disable calls to out-of-line helpers to implement atomic operations. +These helpers will, at runtime, determine if ARMv8.1-Atomics instructions +should be used; if not, they will use the load/store-exclusive instructions +that are present in the base ARMv8.0 ISA. + +This option is only applicable when compiling for the base ARMv8.0 +instruction set. If using a later revision, e.g. @option{-march=armv8.1-a} +or @option{-march=armv8-a+lse}, the ARMv8.1-Atomics instructions will be +used directly. + @item -march=@var{name} @opindex march Specify the name of the target architecture and, optionally, one or From patchwork Tue Oct 2 16:19:13 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 147991 Delivered-To: patch@linaro.org Received: by 2002:a2e:8595:0:0:0:0:0 with SMTP id b21-v6csp5356016lji; Tue, 2 Oct 2018 09:21:23 -0700 (PDT) X-Google-Smtp-Source: ACcGV63cQjToXfhBjgSfSZYpR4kg0iRKC5agEPVdFBgkNND/fvhWD3mQT+HNg+Yq5vvTCLKCxZCA X-Received: by 2002:a62:1e83:: with SMTP id e125-v6mr17206440pfe.231.1538497283180; Tue, 02 Oct 2018 09:21:23 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1538497283; cv=none; d=google.com; s=arc-20160816; b=p4takhNCEbTRnouGdgpYaGNVrylC/q0GiSjTNCMAQXtFo6jFA4ruxu/W9l5Bl78HKc 9jrpX8Chp3EC99YGh73KMNc+Tvvahbvhzwt94x8cBKXBimtUdzc7t8Lr7ESYJgOVd75k xFXUPJrVunGZcOV65qy6eEUBHaVKJLzfN0HpBKfAECjAg0LWOzEO+HqihV6w2e2Ncigk 16npxw6PB/wIVae0dZ6eAJFcWwTt7O3jbEw+4hnlzS+OfEOqtyiQ4z5tcG9OL1R4d14o CBPLcChPs/8x0t1+4XfikmIz39V5SZalyBr/BN89fT6O8o9O4/Td2NkfGmFtRBovXFoZ Z5SQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature:delivered-to:sender:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:mailing-list:dkim-signature :domainkey-signature; bh=bFXjP9PN/RPdCfNaKD9wfHUaATK/mX8xFl2hNp4yRyY=; b=Ywyz7dxVGulRHey5SvgRbtU2Wc7u5LcY3amqtNwQCblAf29oXFr1LAqy7yhiJN9o1a /WR2nJx7bV2U0gvIcSt38vIMfbsfIv8RBQyaGl8P3ci1GbT/KYQ9GBeh/WVoCh8T7kvi l5NRAWMkpKzeJsNyZn27gdg+UyAp3F0EVrDnu8CxjiKiEwXs4ffVMaksYJRWuqwUzA2P 9J3QX8e2Ox44rJe+hObdaiTcZzVS3azyS/b+OUJjGz0WDd53MaiUji0ESaJTkaZ8bDLD Ftv2CY/rY2IRcMiVueCVKG8Ou7TI9OkO1bBg36OAgzSDUznM27YYNrwaHfGqmTAsHdP0 jFVA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gcc.gnu.org header.s=default header.b=PNyfKQ8T; dkim=pass header.i=@linaro.org header.s=google header.b="CHhG/oAl"; spf=pass (google.com: domain of gcc-patches-return-486823-patch=linaro.org@gcc.gnu.org designates 209.132.180.131 as permitted sender) smtp.mailfrom="gcc-patches-return-486823-patch=linaro.org@gcc.gnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from sourceware.org (server1.sourceware.org. [209.132.180.131]) by mx.google.com with ESMTPS id j16-v6si14913062pgg.350.2018.10.02.09.21.22 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 02 Oct 2018 09:21:23 -0700 (PDT) Received-SPF: pass (google.com: domain of gcc-patches-return-486823-patch=linaro.org@gcc.gnu.org designates 209.132.180.131 as permitted sender) client-ip=209.132.180.131; Authentication-Results: mx.google.com; dkim=pass header.i=@gcc.gnu.org header.s=default header.b=PNyfKQ8T; dkim=pass header.i=@linaro.org header.s=google header.b="CHhG/oAl"; spf=pass (google.com: domain of gcc-patches-return-486823-patch=linaro.org@gcc.gnu.org designates 209.132.180.131 as permitted sender) smtp.mailfrom="gcc-patches-return-486823-patch=linaro.org@gcc.gnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org DomainKey-Signature: a=rsa-sha1; c=nofws; d=gcc.gnu.org; h=list-id :list-unsubscribe:list-archive:list-post:list-help:sender:from :to:cc:subject:date:message-id:in-reply-to:references; q=dns; s= default; b=UgoGW9FN/66RMN9oe+oFfPQZlI6JNb7IcZdMTuTwx+2pmgukXQZU5 EuUHU82p1M/B1RwmN3ltGvFtZ5QW3ogX+/QHzMm7CXADBMJuoSKxkcoLmJxBjJNj GrkN8KvHYZGJCrfoxM9wh/QCJn526S/Z2EuvMWeJln0qbcsOPUPSPk= DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=gcc.gnu.org; h=list-id :list-unsubscribe:list-archive:list-post:list-help:sender:from :to:cc:subject:date:message-id:in-reply-to:references; s= default; bh=/WBIAX1IXESgM+FCpITJNnyWc1c=; b=PNyfKQ8TiwVQ8xwCr9Lv 3XPgT+w7QWoIb0XuZGD6LmLLpOr07ilKwm1F4adWysPnA4vjjQj6PwAv9LX15ouY diT7n5G/Ix41rqQDmUAUZu+tKt35fST8gT4ndk7DqEwCCpr19R7NuYorANvVkEWE EXfCmkUIHjcjZFeULk1KuR8= Received: (qmail 68157 invoked by alias); 2 Oct 2018 16:19:40 -0000 Mailing-List: contact gcc-patches-help@gcc.gnu.org; run by ezmlm Precedence: bulk List-Id: List-Unsubscribe: List-Archive: List-Post: List-Help: Sender: gcc-patches-owner@gcc.gnu.org Delivered-To: mailing list gcc-patches@gcc.gnu.org Received: (qmail 68087 invoked by uid 89); 2 Oct 2018 16:19:39 -0000 Authentication-Results: sourceware.org; auth=none X-Spam-SWARE-Status: No, score=-26.9 required=5.0 tests=BAYES_00, GIT_PATCH_0, GIT_PATCH_1, GIT_PATCH_2, GIT_PATCH_3, RCVD_IN_DNSWL_NONE, SPF_PASS autolearn=ham version=3.3.2 spammy=Hx-languages-length:1294, held X-HELO: mail-ot1-f48.google.com Received: from mail-ot1-f48.google.com (HELO mail-ot1-f48.google.com) (209.85.210.48) by sourceware.org (qpsmtpd/0.93/v0.84-503-g423c35a) with ESMTP; Tue, 02 Oct 2018 16:19:37 +0000 Received: by mail-ot1-f48.google.com with SMTP id i12-v6so2499966otl.1 for ; Tue, 02 Oct 2018 09:19:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=bFXjP9PN/RPdCfNaKD9wfHUaATK/mX8xFl2hNp4yRyY=; b=CHhG/oAlwjia+n4E5pQOg1ETsXlv/nm6gU7YKeCd8TUdrli0TwC0DXB0qLc9nQ8+oM KvnU9Nm037DHA7VALL9kbkBw0AL5v9z2Oj1sRzwUek4KdGLsZamboDkuTtFzjRdPeiMK btgjw7QVd6Nnwp6X9gnZsGjSp9/+9aUM8ICFQ= Return-Path: Received: from cloudburst.twiddle.net ([187.217.227.243]) by smtp.gmail.com with ESMTPSA id j15-v6sm145987oth.27.2018.10.02.09.19.34 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Tue, 02 Oct 2018 09:19:35 -0700 (PDT) From: Richard Henderson To: gcc-patches@gcc.gnu.org Cc: ramana.radhakrishnan@arm.com, agraf@suse.de, marcus.shawcroft@arm.com, james.greenhalgh@arm.com, richard.earnshaw@arm.com Subject: [PATCH, AArch64 v2 09/11] aarch64: Force TImode values into even registers Date: Tue, 2 Oct 2018 11:19:13 -0500 Message-Id: <20181002161915.18843-10-richard.henderson@linaro.org> In-Reply-To: <20181002161915.18843-1-richard.henderson@linaro.org> References: <20181002161915.18843-1-richard.henderson@linaro.org> The LSE CASP instruction requires values to be placed in even register pairs. A solution involving two additional register classes was rejected in favor of the much simpler solution of simply requiring all TImode values to be aligned. * config/aarch64/aarch64.c (aarch64_hard_regno_mode_ok): Force 16-byte modes held in GP registers to use an even regno. --- gcc/config/aarch64/aarch64.c | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) -- 2.17.1 diff --git a/gcc/config/aarch64/aarch64.c b/gcc/config/aarch64/aarch64.c index 49b47382b5d..ce4d7e51d00 100644 --- a/gcc/config/aarch64/aarch64.c +++ b/gcc/config/aarch64/aarch64.c @@ -1451,10 +1451,14 @@ aarch64_hard_regno_mode_ok (unsigned regno, machine_mode mode) if (regno == FRAME_POINTER_REGNUM || regno == ARG_POINTER_REGNUM) return mode == Pmode; - if (GP_REGNUM_P (regno) && known_le (GET_MODE_SIZE (mode), 16)) - return true; - - if (FP_REGNUM_P (regno)) + if (GP_REGNUM_P (regno)) + { + if (known_le (GET_MODE_SIZE (mode), 8)) + return true; + else if (known_le (GET_MODE_SIZE (mode), 16)) + return (regno & 1) == 0; + } + else if (FP_REGNUM_P (regno)) { if (vec_flags & VEC_STRUCT) return end_hard_regno (mode, regno) - 1 <= V31_REGNUM; From patchwork Tue Oct 2 16:19:14 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 147994 Delivered-To: patch@linaro.org Received: by 2002:a2e:8595:0:0:0:0:0 with SMTP id b21-v6csp5356683lji; Tue, 2 Oct 2018 09:22:00 -0700 (PDT) X-Google-Smtp-Source: ACcGV61HJ3V3uDWW7Yax4TA32DgW/0e0UVHutqMB03Nf0W3mImhZ1r9SuYj/y5P3qf+o8fv8kFrn X-Received: by 2002:a17:902:447:: with SMTP id 65-v6mr17276583ple.325.1538497320119; Tue, 02 Oct 2018 09:22:00 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1538497320; cv=none; d=google.com; s=arc-20160816; b=rR5DRJ5l3JOZEb3qmbWSO1wXmDpdK/5hhZCUeChIlvQFXG8FLlupdx6XM25vYn2LMw lNmq3j5hjnbobHj6JhGT6mp5PzZ8jqWgi9kkGXe/1c/PJdn+LwAUosUFKFhgURfRHJ3w jRs87pPdYpxZcug9sEMeMNkxLq+jUztr3r59gOSGKLWFesKHo8Q9E4RlIzs0VvGG6b9N a9OXjLJ0Vk4f0b9wJNNY8Zef9Xv/IwJlsDRxZ7hBSAz7py/vkqm8Or8ohuyj7e56ZJgB L6t5t0agl14KqkfeIuxzzup44hE+rtUmrsTdtd/pjKTC4+DOI4U9fddKCFM6lxg2HxJU 0kbg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature:delivered-to:sender:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:mailing-list:dkim-signature :domainkey-signature; bh=jLEKBSTQiwnh6x3Df2F8hfQNaqEZUzYdavTDNFpCjME=; b=uL+pHZHas0O072gMNoGd3WUrz8hCx9jO1fQkFBKBikrESSNgYtnCtq3CsZg9xWRYdz GzaVzCNh2yPQiyRiqGUmCxEzkWyuyDf9REnNXlAt81+Way+JTHAMI7gZg0drdKXhQpeu pOF/ZDt5sILzL2wQoosAO/qbRUq2XxwgfbJLMbadtpsUpdqbuCTNtPT2+5mxkjCJrZ/P adHa1uJtgs+vHL1stpmirBFzgKQmQyNB+wq2zgSocGAKaoBQLu8Y87aCuXDXTItPpjlL JfwMl4J4Gpc58LV5/6kXrmWn8TFdnZi7pR53d3T8dcvbwMrJpV/kjtWwXzwrlWJF36mF VpLA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gcc.gnu.org header.s=default header.b=NO8pQzTx; dkim=pass header.i=@linaro.org header.s=google header.b=OFMHfVZY; spf=pass (google.com: domain of gcc-patches-return-486826-patch=linaro.org@gcc.gnu.org designates 209.132.180.131 as permitted sender) smtp.mailfrom="gcc-patches-return-486826-patch=linaro.org@gcc.gnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from sourceware.org (server1.sourceware.org. [209.132.180.131]) by mx.google.com with ESMTPS id 11-v6si10748900plc.224.2018.10.02.09.21.59 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 02 Oct 2018 09:22:00 -0700 (PDT) Received-SPF: pass (google.com: domain of gcc-patches-return-486826-patch=linaro.org@gcc.gnu.org designates 209.132.180.131 as permitted sender) client-ip=209.132.180.131; Authentication-Results: mx.google.com; dkim=pass header.i=@gcc.gnu.org header.s=default header.b=NO8pQzTx; dkim=pass header.i=@linaro.org header.s=google header.b=OFMHfVZY; spf=pass (google.com: domain of gcc-patches-return-486826-patch=linaro.org@gcc.gnu.org designates 209.132.180.131 as permitted sender) smtp.mailfrom="gcc-patches-return-486826-patch=linaro.org@gcc.gnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org DomainKey-Signature: a=rsa-sha1; c=nofws; d=gcc.gnu.org; h=list-id :list-unsubscribe:list-archive:list-post:list-help:sender:from :to:cc:subject:date:message-id:in-reply-to:references; q=dns; s= default; b=cbKfoaL1pOKFgu51ohpwzSZ+BoeNU4SMWvI56NNjyMIk/GouI4F/b 5TFRqavQdgcQMg0MrUH54pAeuaGvSkHG2+7ErFYVeB1hybr7KtMsIZDjzqhOUat7 OlbsCA9BTIi/sYyRI4A9UIZ7bM7sELtoXA3JOE5ABU7sLdtsELOrN8= DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=gcc.gnu.org; h=list-id :list-unsubscribe:list-archive:list-post:list-help:sender:from :to:cc:subject:date:message-id:in-reply-to:references; s= default; bh=goAGLbeP7eWZ59gWgQsfAz7PHRs=; b=NO8pQzTxEnScKUp9apqe yY7PjIGzLf9EecpE+0id8QJCGiIHrTsGbe19qDvYdktrtBgWWiG4XDSqr3A4zbkZ FiDr3W+Ma/Y8cndGXOf0q1qXA++5liMhnszCJCKAgdrOYC9q76weu8ZWbbDLjyJE nVqf530lP04nhvdewUvCxaQ= Received: (qmail 68978 invoked by alias); 2 Oct 2018 16:19:45 -0000 Mailing-List: contact gcc-patches-help@gcc.gnu.org; run by ezmlm Precedence: bulk List-Id: List-Unsubscribe: List-Archive: List-Post: List-Help: Sender: gcc-patches-owner@gcc.gnu.org Delivered-To: mailing list gcc-patches@gcc.gnu.org Received: (qmail 68847 invoked by uid 89); 2 Oct 2018 16:19:44 -0000 Authentication-Results: sourceware.org; auth=none X-Spam-SWARE-Status: No, score=-26.9 required=5.0 tests=BAYES_00, GIT_PATCH_0, GIT_PATCH_1, GIT_PATCH_2, GIT_PATCH_3, KAM_SHORT, RCVD_IN_DNSWL_NONE, SPF_PASS autolearn=ham version=3.3.2 spammy=NAMES X-HELO: mail-oi1-f181.google.com Received: from mail-oi1-f181.google.com (HELO mail-oi1-f181.google.com) (209.85.167.181) by sourceware.org (qpsmtpd/0.93/v0.84-503-g423c35a) with ESMTP; Tue, 02 Oct 2018 16:19:40 +0000 Received: by mail-oi1-f181.google.com with SMTP id v69-v6so2015521oif.1 for ; Tue, 02 Oct 2018 09:19:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=jLEKBSTQiwnh6x3Df2F8hfQNaqEZUzYdavTDNFpCjME=; b=OFMHfVZYKt/mAQR9YMeq7Iq77nLlFgog977oX+UUiQZAKUPj4zfFqR57WqfiIDYWce M3T4MQKftDJKxDhFSemOlPE/htSgfmaMfSQLegU6Ps5as87B6eByGvDvAqX1qL/2othG +qAnfyxWeVQsRC2lE48Afau2k+IVUyWqeTPzA= Return-Path: Received: from cloudburst.twiddle.net ([187.217.227.243]) by smtp.gmail.com with ESMTPSA id j15-v6sm145987oth.27.2018.10.02.09.19.36 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Tue, 02 Oct 2018 09:19:36 -0700 (PDT) From: Richard Henderson To: gcc-patches@gcc.gnu.org Cc: ramana.radhakrishnan@arm.com, agraf@suse.de, marcus.shawcroft@arm.com, james.greenhalgh@arm.com, richard.earnshaw@arm.com Subject: [PATCH, AArch64 v2 10/11] aarch64: Implement TImode compare-and-swap Date: Tue, 2 Oct 2018 11:19:14 -0500 Message-Id: <20181002161915.18843-11-richard.henderson@linaro.org> In-Reply-To: <20181002161915.18843-1-richard.henderson@linaro.org> References: <20181002161915.18843-1-richard.henderson@linaro.org> This pattern will only be used with the __sync functions, because we do not yet have a bare TImode atomic load. * config/aarch64/aarch64.c (aarch64_gen_compare_reg): Add support for NE comparison of TImode values. (aarch64_print_operand): Extend %R to handle general registers. (aarch64_emit_load_exclusive): Add support for TImode. (aarch64_emit_store_exclusive): Likewise. (aarch64_atomic_ool_func): Likewise. (aarch64_ool_cas_names): Likewise. * config/aarch64/atomics.md (@atomic_compare_and_swap): Change iterator from ALLI to ALLI_TI. (@atomic_compare_and_swap): New. (@atomic_compare_and_swap_lse): New. (aarch64_load_exclusive_pair): New. (aarch64_store_exclusive_pair): New. * config/aarch64/iterators.md (JUST_TI): New. * config/aarch64/lse.c (cas): Add support for SIZE == 16. * config/aarch64/t-lse (S0, O0): Split out cas. (LSE_OBJS): Include $(O0). --- gcc/config/aarch64/aarch64-protos.h | 2 +- gcc/config/aarch64/aarch64.c | 72 ++++++++++++++++++----- libgcc/config/aarch64/lse.c | 48 ++++++++++----- gcc/config/aarch64/atomics.md | 91 +++++++++++++++++++++++++++-- gcc/config/aarch64/iterators.md | 3 + libgcc/config/aarch64/t-lse | 10 +++- 6 files changed, 189 insertions(+), 37 deletions(-) -- 2.17.1 diff --git a/gcc/config/aarch64/aarch64-protos.h b/gcc/config/aarch64/aarch64-protos.h index c7b96b12bbe..f735c4e5ad8 100644 --- a/gcc/config/aarch64/aarch64-protos.h +++ b/gcc/config/aarch64/aarch64-protos.h @@ -626,7 +626,7 @@ bool aarch64_high_bits_all_ones_p (HOST_WIDE_INT); struct atomic_ool_names { - const char *str[4][4]; + const char *str[5][4]; }; rtx aarch64_atomic_ool_func(machine_mode mode, rtx model_rtx, diff --git a/gcc/config/aarch64/aarch64.c b/gcc/config/aarch64/aarch64.c index ce4d7e51d00..ac2f055a09e 100644 --- a/gcc/config/aarch64/aarch64.c +++ b/gcc/config/aarch64/aarch64.c @@ -1610,10 +1610,33 @@ emit_set_insn (rtx x, rtx y) rtx aarch64_gen_compare_reg (RTX_CODE code, rtx x, rtx y) { - machine_mode mode = SELECT_CC_MODE (code, x, y); - rtx cc_reg = gen_rtx_REG (mode, CC_REGNUM); + machine_mode cmp_mode = GET_MODE (x); + machine_mode cc_mode; + rtx cc_reg; - emit_set_insn (cc_reg, gen_rtx_COMPARE (mode, x, y)); + if (cmp_mode == E_TImode) + { + gcc_assert (code == NE); + + cc_mode = E_CCmode; + cc_reg = gen_rtx_REG (cc_mode, CC_REGNUM); + + rtx x_lo = operand_subword (x, 0, 0, TImode); + rtx y_lo = operand_subword (y, 0, 0, TImode); + emit_set_insn (cc_reg, gen_rtx_COMPARE (cc_mode, x_lo, y_lo)); + + rtx x_hi = operand_subword (x, 1, 0, TImode); + rtx y_hi = operand_subword (y, 1, 0, TImode); + emit_insn (gen_ccmpdi (cc_reg, cc_reg, x_hi, y_hi, + gen_rtx_EQ (cc_mode, cc_reg, const0_rtx), + GEN_INT (AARCH64_EQ))); + } + else + { + cc_mode = SELECT_CC_MODE (code, x, y); + cc_reg = gen_rtx_REG (cc_mode, CC_REGNUM); + emit_set_insn (cc_reg, gen_rtx_COMPARE (cc_mode, x, y)); + } return cc_reg; } @@ -6693,7 +6716,7 @@ sizetochar (int size) 'S/T/U/V': Print a FP/SIMD register name for a register list. The register printed is the FP/SIMD register name of X + 0/1/2/3 for S/T/U/V. - 'R': Print a scalar FP/SIMD register name + 1. + 'R': Print a scalar Integer/FP/SIMD register name + 1. 'X': Print bottom 16 bits of integer constant in hex. 'w/x': Print a general register name or the zero register (32-bit or 64-bit). @@ -6885,12 +6908,13 @@ aarch64_print_operand (FILE *f, rtx x, int code) break; case 'R': - if (!REG_P (x) || !FP_REGNUM_P (REGNO (x))) - { - output_operand_lossage ("incompatible floating point / vector register operand for '%%%c'", code); - return; - } - asm_fprintf (f, "q%d", REGNO (x) - V0_REGNUM + 1); + if (REG_P (x) && FP_REGNUM_P (REGNO (x))) + asm_fprintf (f, "q%d", REGNO (x) - V0_REGNUM + 1); + else if (REG_P (x) && GP_REGNUM_P (REGNO (x))) + asm_fprintf (f, "x%d", REGNO (x) - R0_REGNUM + 1); + else + output_operand_lossage ("incompatible register operand for '%%%c'", + code); break; case 'X': @@ -14143,16 +14167,26 @@ static void aarch64_emit_load_exclusive (machine_mode mode, rtx rval, rtx mem, rtx model_rtx) { - emit_insn (gen_aarch64_load_exclusive (mode, rval, mem, model_rtx)); + if (mode == E_TImode) + emit_insn (gen_aarch64_load_exclusive_pair (gen_lowpart (DImode, rval), + gen_highpart (DImode, rval), + mem, model_rtx)); + else + emit_insn (gen_aarch64_load_exclusive (mode, rval, mem, model_rtx)); } /* Emit store exclusive. */ static void aarch64_emit_store_exclusive (machine_mode mode, rtx bval, - rtx rval, rtx mem, rtx model_rtx) + rtx mem, rtx val, rtx model_rtx) { - emit_insn (gen_aarch64_store_exclusive (mode, bval, rval, mem, model_rtx)); + if (mode == E_TImode) + emit_insn (gen_aarch64_store_exclusive_pair + (bval, mem, operand_subword (val, 0, 0, TImode), + operand_subword (val, 1, 0, TImode), model_rtx)); + else + emit_insn (gen_aarch64_store_exclusive (mode, bval, mem, val, model_rtx)); } /* Mark the previous jump instruction as unlikely. */ @@ -14164,7 +14198,7 @@ aarch64_emit_unlikely_jump (rtx insn) add_reg_br_prob_note (jump, profile_probability::very_unlikely ()); } -/* We store the names of the various atomic helpers in a 4x4 array. +/* We store the names of the various atomic helpers in a 5x4 array. Return the libcall function given MODE, MODEL and NAMES. */ rtx @@ -14188,6 +14222,9 @@ aarch64_atomic_ool_func(machine_mode mode, rtx model_rtx, case E_DImode: mode_idx = 3; break; + case E_TImode: + mode_idx = 4; + break; default: gcc_unreachable (); } @@ -14222,9 +14259,11 @@ aarch64_atomic_ool_func(machine_mode mode, rtx model_rtx, "__aa64_" #B #N "_rel", \ "__aa64_" #B #N "_acq_rel" } -#define DEF4(B) DEF0(B, 1), DEF0(B, 2), DEF0(B, 4), DEF0(B, 8) +#define DEF4(B) DEF0(B, 1), DEF0(B, 2), DEF0(B, 4), DEF0(B, 8), \ + { NULL, NULL, NULL, NULL } +#define DEF5(B) DEF0(B, 1), DEF0(B, 2), DEF0(B, 4), DEF0(B, 8), DEF0(B, 16) -static const atomic_ool_names aarch64_ool_cas_names = { { DEF4(cas) } }; +static const atomic_ool_names aarch64_ool_cas_names = { { DEF5(cas) } }; const atomic_ool_names aarch64_ool_swp_names = { { DEF4(swp) } }; const atomic_ool_names aarch64_ool_ldadd_names = { { DEF4(ldadd) } }; const atomic_ool_names aarch64_ool_ldset_names = { { DEF4(ldset) } }; @@ -14247,6 +14286,7 @@ const atomic_ool_names aarch64_ool_steor_names = { { DEF4(eor) } }; #undef DEF0 #undef DEF4 +#undef DEF5 /* Expand a compare and swap pattern. */ diff --git a/libgcc/config/aarch64/lse.c b/libgcc/config/aarch64/lse.c index 68ca7df667b..f6114add71a 100644 --- a/libgcc/config/aarch64/lse.c +++ b/libgcc/config/aarch64/lse.c @@ -91,6 +91,7 @@ asm(".arch armv8-a+lse"); #elif SIZE == 4 || SIZE == 8 # define S "" # define MASK "" +#elif SIZE == 16 #else # error #endif @@ -98,9 +99,11 @@ asm(".arch armv8-a+lse"); #if SIZE < 8 # define T unsigned int # define W "w" -#else +#elif SIZE == 8 # define T unsigned long long # define W "" +#else +# define T unsigned __int128 #endif #if MODEL == 1 @@ -138,19 +141,38 @@ T NAME(cas) (T cmp, T new, T *ptr) unsigned tmp; if (have_atomics) - __asm__("cas" A L S " %"W"0, %"W"2, %1" - : "=r"(old), "+m"(*ptr) : "r"(new), "0"(cmp)); + { +#if SIZE == 16 + __asm__("casp" A L " %0, %R0, %2, %R2, %1" + : "=r"(old), "+m"(*ptr) : "r"(new), "0"(cmp)); +#else + __asm__("cas" A L S " %"W"0, %"W"2, %1" + : "=r"(old), "+m"(*ptr) : "r"(new), "0"(cmp)); +#endif + } else - __asm__( - "0: " - "ld" A "xr"S" %"W"0, %1\n\t" - "cmp %"W"0, %"W"4" MASK "\n\t" - "bne 1f\n\t" - "st" L "xr"S" %w2, %"W"3, %1\n\t" - "cbnz %w2, 0b\n" - "1:" - : "=&r"(old), "+m"(*ptr), "=&r"(tmp) : "r"(new), "r"(cmp)); - + { +#if SIZE == 16 + __asm__("0: " + "ld" A "xp %0, %R0, %1\n\t" + "cmp %0, %4\n\t" + "ccmp %R0, %R4, #0, eq\n\t" + "bne 1f\n\t" + "st" L "xp %w2, %3, %R3, %1\n\t" + "cbnz %w2, 0b\n" + "1:" + : "=&r"(old), "+m"(*ptr), "=&r"(tmp) : "r"(new), "r"(cmp)); +#else + __asm__("0: " + "ld" A "xr"S" %"W"0, %1\n\t" + "cmp %"W"0, %"W"4" MASK "\n\t" + "bne 1f\n\t" + "st" L "xr"S" %w2, %"W"3, %1\n\t" + "cbnz %w2, 0b\n" + "1:" + : "=&r"(old), "+m"(*ptr), "=&r"(tmp) : "r"(new), "r"(cmp)); +#endif + } return old; } #endif diff --git a/gcc/config/aarch64/atomics.md b/gcc/config/aarch64/atomics.md index 72f9962fe55..fe604606bdd 100644 --- a/gcc/config/aarch64/atomics.md +++ b/gcc/config/aarch64/atomics.md @@ -22,10 +22,10 @@ (define_expand "@atomic_compare_and_swap" [(match_operand:SI 0 "register_operand" "") ;; bool out - (match_operand:ALLI 1 "register_operand" "") ;; val out - (match_operand:ALLI 2 "aarch64_sync_memory_operand" "") ;; memory - (match_operand:ALLI 3 "nonmemory_operand" "") ;; expected - (match_operand:ALLI 4 "aarch64_reg_or_zero" "") ;; desired + (match_operand:ALLI_TI 1 "register_operand" "") ;; val out + (match_operand:ALLI_TI 2 "aarch64_sync_memory_operand" "") ;; memory + (match_operand:ALLI_TI 3 "nonmemory_operand" "") ;; expected + (match_operand:ALLI_TI 4 "aarch64_reg_or_zero" "") ;; desired (match_operand:SI 5 "const_int_operand") ;; is_weak (match_operand:SI 6 "const_int_operand") ;; mod_s (match_operand:SI 7 "const_int_operand")] ;; mod_f @@ -88,6 +88,30 @@ } ) +(define_insn_and_split "@aarch64_compare_and_swap" + [(set (reg:CC CC_REGNUM) ;; bool out + (unspec_volatile:CC [(const_int 0)] UNSPECV_ATOMIC_CMPSW)) + (set (match_operand:JUST_TI 0 "register_operand" "=&r") ;; val out + (match_operand:JUST_TI 1 "aarch64_sync_memory_operand" "+Q")) ;; memory + (set (match_dup 1) + (unspec_volatile:JUST_TI + [(match_operand:JUST_TI 2 "register_operand" "r") ;; expect + (match_operand:JUST_TI 3 "aarch64_reg_or_zero" "rZ") ;; desired + (match_operand:SI 4 "const_int_operand") ;; is_weak + (match_operand:SI 5 "const_int_operand") ;; mod_s + (match_operand:SI 6 "const_int_operand")] ;; mod_f + UNSPECV_ATOMIC_CMPSW)) + (clobber (match_scratch:SI 7 "=&r"))] + "" + "#" + "&& reload_completed" + [(const_int 0)] + { + aarch64_split_compare_and_swap (operands); + DONE; + } +) + (define_insn "@aarch64_compare_and_swap_lse" [(set (match_operand:SI 0 "register_operand" "+r") ;; val out (zero_extend:SI @@ -133,6 +157,28 @@ return "casal\t%0, %2, %1"; }) +(define_insn "@aarch64_compare_and_swap_lse" + [(set (match_operand:JUST_TI 0 "register_operand" "+r") ;; val out + (match_operand:JUST_TI 1 "aarch64_sync_memory_operand" "+Q")) ;; memory + (set (match_dup 1) + (unspec_volatile:JUST_TI + [(match_dup 0) ;; expect + (match_operand:JUST_TI 2 "register_operand" "r") ;; desired + (match_operand:SI 3 "const_int_operand")] ;; mod_s + UNSPECV_ATOMIC_CMPSW))] + "TARGET_LSE" +{ + enum memmodel model = memmodel_from_int (INTVAL (operands[3])); + if (is_mm_relaxed (model)) + return "casp\t%0, %R0, %2, %R2, %1"; + else if (is_mm_acquire (model) || is_mm_consume (model)) + return "caspa\t%0, %R0, %2, %R2, %1"; + else if (is_mm_release (model)) + return "caspl\t%0, %R0, %2, %R2, %1"; + else + return "caspal\t%0, %R0, %2, %R2, %1"; +}) + (define_expand "atomic_exchange" [(match_operand:ALLI 0 "register_operand" "") (match_operand:ALLI 1 "aarch64_sync_memory_operand" "") @@ -650,6 +696,24 @@ } ) +(define_insn "aarch64_load_exclusive_pair" + [(set (match_operand:DI 0 "register_operand" "=r") + (unspec_volatile:DI + [(match_operand:TI 2 "aarch64_sync_memory_operand" "Q") + (match_operand:SI 3 "const_int_operand")] + UNSPECV_LX)) + (set (match_operand:DI 1 "register_operand" "=r") + (unspec_volatile:DI [(match_dup 2) (match_dup 3)] UNSPECV_LX))] + "" + { + enum memmodel model = memmodel_from_int (INTVAL (operands[3])); + if (is_mm_relaxed (model) || is_mm_consume (model) || is_mm_release (model)) + return "ldxp\t%0, %1, %2"; + else + return "ldaxp\t%0, %1, %2"; + } +) + (define_insn "@aarch64_store_exclusive" [(set (match_operand:SI 0 "register_operand" "=&r") (unspec_volatile:SI [(const_int 0)] UNSPECV_SX)) @@ -668,6 +732,25 @@ } ) +(define_insn "aarch64_store_exclusive_pair" + [(set (match_operand:SI 0 "register_operand" "=&r") + (unspec_volatile:SI [(const_int 0)] UNSPECV_SX)) + (set (match_operand:TI 1 "aarch64_sync_memory_operand" "=Q") + (unspec_volatile:TI + [(match_operand:DI 2 "aarch64_reg_or_zero" "rZ") + (match_operand:DI 3 "aarch64_reg_or_zero" "rZ") + (match_operand:SI 4 "const_int_operand")] + UNSPECV_SX))] + "" + { + enum memmodel model = memmodel_from_int (INTVAL (operands[3])); + if (is_mm_relaxed (model) || is_mm_consume (model) || is_mm_acquire (model)) + return "stxp\t%w0, %x2, %x3, %1"; + else + return "stlxp\t%w0, %x2, %x3, %1"; + } +) + (define_expand "mem_thread_fence" [(match_operand:SI 0 "const_int_operand" "")] "" diff --git a/gcc/config/aarch64/iterators.md b/gcc/config/aarch64/iterators.md index 524e4e6929b..dd26bdbbc6b 100644 --- a/gcc/config/aarch64/iterators.md +++ b/gcc/config/aarch64/iterators.md @@ -29,6 +29,9 @@ ;; Iterator for HI, SI, DI, some instructions can only work on these modes. (define_mode_iterator GPI_I16 [(HI "AARCH64_ISA_F16") SI DI]) +;; "Iterator" for just TI -- features like @pattern only work with iterators. +(define_mode_iterator JUST_TI [TI]) + ;; Iterator for QI and HI modes (define_mode_iterator SHORT [QI HI]) diff --git a/libgcc/config/aarch64/t-lse b/libgcc/config/aarch64/t-lse index e862b0c2448..534ff6efea8 100644 --- a/libgcc/config/aarch64/t-lse +++ b/libgcc/config/aarch64/t-lse @@ -18,15 +18,19 @@ # along with GCC; see the file COPYING3. If not see # . -# CAS, Swap, Load-and-operate have 4 sizes and 4 memory models -S1 := $(foreach s, 1 2 4 8, $(addsuffix _$(s), cas swp ldadd ldclr ldeor ldset)) +# Compare-and-swap has 5 sizes and 4 memory models. +S0 := $(foreach s, 1 2 4 8 16, $(addsuffix _$(s), cas)) +O0 := $(foreach m, 1 2 3 4, $(addsuffix _$(m)$(objext), $(S0))) + +# Swap, Load-and-operate have 4 sizes and 4 memory models +S1 := $(foreach s, 1 2 4 8, $(addsuffix _$(s), swp ldadd ldclr ldeor ldset)) O1 := $(foreach m, 1 2 3 4, $(addsuffix _$(m)$(objext), $(S1))) # Store-and-operate has 4 sizes but only 2 memory models (relaxed, release). S2 := $(foreach s, 1 2 4 8, $(addsuffix _$(s), stadd stclr steor stset)) O2 := $(foreach m, 1 3, $(addsuffix _$(m)$(objext), $(S2))) -LSE_OBJS := $(O1) $(O2) +LSE_OBJS := $(O0) $(O1) $(O2) libgcc-objects += $(LSE_OBJS) have_atomic$(objext) From patchwork Tue Oct 2 16:19:15 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 147993 Delivered-To: patch@linaro.org Received: by 2002:a2e:8595:0:0:0:0:0 with SMTP id b21-v6csp5356462lji; Tue, 2 Oct 2018 09:21:47 -0700 (PDT) X-Google-Smtp-Source: ACcGV60O3VmsEVcHdoBTzRPBn3RtqbCCV5EIQO97Vne6wqvt9bGBGjWw4A2WhRrnmGj52cBQjzyn X-Received: by 2002:a17:902:ac1:: with SMTP id 59-v6mr17532752plp.18.1538497307375; Tue, 02 Oct 2018 09:21:47 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1538497307; cv=none; d=google.com; s=arc-20160816; b=JXvgK28TdJNfrxCq9Z7Kk980x/MmxvPXYPdSHODlGBdRsIC6fNDF9xakE1ht42wlL2 pvkzY9GJ2il8WF+eMnkC5o//8a0mOL+m45MfKIh90ZTBsw0JoA1dbvDf6nJkNmu3i6d+ rZlF15Nd4tGC5+MlqMXXT6V2FghXAqvMh7WyQ3UQOzRnxsg1cEEOjlzPFF1/uqqK5OYx dNVoogt3U1ruFYV1BIpVkMCH3Jbk5KK29bsKSb09wi5Hq4JEMqF/CuwUNEDDDYvxG2zJ uG7Pv7PR1krciLXZvjHISS6I3e3TpWec9hh4d5/+BzeJ9oJeKJs2LaVs4PiH+kS4NjE4 s3Jw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature:delivered-to:sender:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:mailing-list:dkim-signature :domainkey-signature; bh=g5WJ9co6zXi1mcpKsZR31vsEQRYAsWozTQC4mRHO+ck=; b=vTsTKKCnOUbGGP5iz3MO3YkE3MMGHIZ0HmXTsXEesI/2yuO2x+gp17L6h8fcouEkV3 FXIHbXdb/cwn3r1p3i5MPLJQn9pIm3A7UKDNY8SK/a2KnZyHU9Xr0a8crI9rKU12CGMG fyilkJ2b59suF6HS3MFuHpWa0SNv2ElwvBLf92UMM94hwWgyoqPFPaN0EEg80IoArSd8 i5JCWeiKStt1tAZ1lnSlYqIiip3U2NOeTsfvolMRRIQkpHj7P75pokPelf7Zl3UPsfye Zg73nsM+TXTMdnHD8zktPOlYCRDccfn006k5zMSs3F8LKzfzSDyj5uoZgqFkIZNAGgu2 XTbQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gcc.gnu.org header.s=default header.b=MMdpP3Sz; dkim=pass header.i=@linaro.org header.s=google header.b=Qmbc+4dE; spf=pass (google.com: domain of gcc-patches-return-486825-patch=linaro.org@gcc.gnu.org designates 209.132.180.131 as permitted sender) smtp.mailfrom="gcc-patches-return-486825-patch=linaro.org@gcc.gnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from sourceware.org (server1.sourceware.org. [209.132.180.131]) by mx.google.com with ESMTPS id b9-v6si11788400plx.20.2018.10.02.09.21.47 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 02 Oct 2018 09:21:47 -0700 (PDT) Received-SPF: pass (google.com: domain of gcc-patches-return-486825-patch=linaro.org@gcc.gnu.org designates 209.132.180.131 as permitted sender) client-ip=209.132.180.131; Authentication-Results: mx.google.com; dkim=pass header.i=@gcc.gnu.org header.s=default header.b=MMdpP3Sz; dkim=pass header.i=@linaro.org header.s=google header.b=Qmbc+4dE; spf=pass (google.com: domain of gcc-patches-return-486825-patch=linaro.org@gcc.gnu.org designates 209.132.180.131 as permitted sender) smtp.mailfrom="gcc-patches-return-486825-patch=linaro.org@gcc.gnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org DomainKey-Signature: a=rsa-sha1; c=nofws; d=gcc.gnu.org; h=list-id :list-unsubscribe:list-archive:list-post:list-help:sender:from :to:cc:subject:date:message-id:in-reply-to:references; q=dns; s= default; b=p8ruwJohPLyZIVUUrfcr83krStIXlgufPbIWHdx0iZ8CapLKNFWwz Ady1kte8BZoy9PqYexTgiDXEOShd9PZKHXYf+CkAaVp0z2mCRyCj6kGIw1a7g2si zN3K6cpsOhFs0KBwylsFjWtwySOF5Rezm6vHvl9LqGXBJ3t3sBK8TA= DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=gcc.gnu.org; h=list-id :list-unsubscribe:list-archive:list-post:list-help:sender:from :to:cc:subject:date:message-id:in-reply-to:references; s= default; bh=6jSE99WjIbSHz63MTeEivqllpzs=; b=MMdpP3Sz9Tg8yGYQeHBH jfrAZ6DQD9geM2XYoQy9MayPYJacj8hFtVTBrIumWLyKhyPNIAqsU4rtvegvrdn7 PrZuKBG4egqJPQ9fvc1Qu2QQl+m7HPyI7BYp8KnGVwXk2FTcWqKQV9uudOoqu9GI VEQIF7vC5mTdXIZFAjLqCQc= Received: (qmail 68778 invoked by alias); 2 Oct 2018 16:19:44 -0000 Mailing-List: contact gcc-patches-help@gcc.gnu.org; run by ezmlm Precedence: bulk List-Id: List-Unsubscribe: List-Archive: List-Post: List-Help: Sender: gcc-patches-owner@gcc.gnu.org Delivered-To: mailing list gcc-patches@gcc.gnu.org Received: (qmail 68662 invoked by uid 89); 2 Oct 2018 16:19:43 -0000 Authentication-Results: sourceware.org; auth=none X-Spam-SWARE-Status: No, score=-26.9 required=5.0 tests=BAYES_00, GIT_PATCH_0, GIT_PATCH_1, GIT_PATCH_2, GIT_PATCH_3, RCVD_IN_DNSWL_NONE, SPF_PASS autolearn=ham version=3.3.2 spammy=his X-HELO: mail-ot1-f44.google.com Received: from mail-ot1-f44.google.com (HELO mail-ot1-f44.google.com) (209.85.210.44) by sourceware.org (qpsmtpd/0.93/v0.84-503-g423c35a) with ESMTP; Tue, 02 Oct 2018 16:19:41 +0000 Received: by mail-ot1-f44.google.com with SMTP id e21-v6so2458911otk.10 for ; Tue, 02 Oct 2018 09:19:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=g5WJ9co6zXi1mcpKsZR31vsEQRYAsWozTQC4mRHO+ck=; b=Qmbc+4dE8eNFDAYPJCagyuRMkXmQ0X0GNCAg8i01a2KdUlJ0HD6sT7BIyZfoOHyuq1 szZ33DsnkwMSs+9ZlckHw0ZHED+FS1R85dU7pdjiafPXbwjlRRE2jMSEjD+UT3T4LUpp IHV9P5Y5t2GJu5TNQ02nGMu1zxkZmgPVSWBcc= Return-Path: Received: from cloudburst.twiddle.net ([187.217.227.243]) by smtp.gmail.com with ESMTPSA id j15-v6sm145987oth.27.2018.10.02.09.19.38 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Tue, 02 Oct 2018 09:19:39 -0700 (PDT) From: Richard Henderson To: gcc-patches@gcc.gnu.org Cc: ramana.radhakrishnan@arm.com, agraf@suse.de, marcus.shawcroft@arm.com, james.greenhalgh@arm.com, richard.earnshaw@arm.com Subject: [PATCH, AArch64 v2 11/11] Enable -matomic-ool by default Date: Tue, 2 Oct 2018 11:19:15 -0500 Message-Id: <20181002161915.18843-12-richard.henderson@linaro.org> In-Reply-To: <20181002161915.18843-1-richard.henderson@linaro.org> References: <20181002161915.18843-1-richard.henderson@linaro.org> Do Not Merge Upstream. This is for agraf and his testing within SLES. --- gcc/common/config/aarch64/aarch64-common.c | 6 ++++-- gcc/config/aarch64/aarch64.c | 6 ++++-- 2 files changed, 8 insertions(+), 4 deletions(-) -- 2.17.1 diff --git a/gcc/common/config/aarch64/aarch64-common.c b/gcc/common/config/aarch64/aarch64-common.c index 292fb818705..3bd1312a3f8 100644 --- a/gcc/common/config/aarch64/aarch64-common.c +++ b/gcc/common/config/aarch64/aarch64-common.c @@ -31,9 +31,11 @@ #include "flags.h" #include "diagnostic.h" -#ifdef TARGET_BIG_ENDIAN_DEFAULT #undef TARGET_DEFAULT_TARGET_FLAGS -#define TARGET_DEFAULT_TARGET_FLAGS (MASK_BIG_END) +#ifdef TARGET_BIG_ENDIAN_DEFAULT +#define TARGET_DEFAULT_TARGET_FLAGS (MASK_BIG_END | MASK_ATOMIC_OOL) +#else +#define TARGET_DEFAULT_TARGET_FLAGS (MASK_ATOMIC_OOL) #endif #undef TARGET_HANDLE_OPTION diff --git a/gcc/config/aarch64/aarch64.c b/gcc/config/aarch64/aarch64.c index ac2f055a09e..b20d8bbf19b 100644 --- a/gcc/config/aarch64/aarch64.c +++ b/gcc/config/aarch64/aarch64.c @@ -17617,9 +17617,11 @@ aarch64_run_selftests (void) #undef TARGET_C_MODE_FOR_SUFFIX #define TARGET_C_MODE_FOR_SUFFIX aarch64_c_mode_for_suffix -#ifdef TARGET_BIG_ENDIAN_DEFAULT #undef TARGET_DEFAULT_TARGET_FLAGS -#define TARGET_DEFAULT_TARGET_FLAGS (MASK_BIG_END) +#ifdef TARGET_BIG_ENDIAN_DEFAULT +#define TARGET_DEFAULT_TARGET_FLAGS (MASK_BIG_END | MASK_ATOMIC_OOL) +#else +#define TARGET_DEFAULT_TARGET_FLAGS (MASK_ATOMIC_OOL) #endif #undef TARGET_CLASS_MAX_NREGS