From patchwork Thu May 24 12:07:55 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Sandiford X-Patchwork-Id: 136741 Delivered-To: patch@linaro.org Received: by 2002:a2e:9706:0:0:0:0:0 with SMTP id r6-v6csp2113548lji; Thu, 24 May 2018 05:08:47 -0700 (PDT) X-Google-Smtp-Source: AB8JxZrdy50X2WhoASeZqHkbKJXiX7xbxBpV6WCf4LWT4GTf4uRRVs0bda/p94l7ae2DbeXaxsv/ X-Received: by 2002:a62:5b02:: with SMTP id p2-v6mr6967929pfb.96.1527163727358; Thu, 24 May 2018 05:08:47 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1527163727; cv=none; d=google.com; s=arc-20160816; b=tDB/yEj/6v8Rwe1xf86DlMORvGNNb5FidZs+SV3lFtsKjunJE8VLAAWzdtq7t14kXq yjeXntGoaThCHbfr3rZY3x/R+sGdZe18aEqeNALcxoUZAqLMQL3EIQxps6Lv4emdX8Ql 0bSa7XsozB+LlKj3seRXsxLFR5dE+nnS3W5jb5kXytAkFolLoFfnKlfT5c09fkv8Rx06 PoKrlpQFcknUD4btz7duufxcTpczEPSPDpx3tr/CtakWkuummGg04Zt/s04ffKCBbooD vgCpZk58zCRIzEXBsFaPzynVswYymg6U6gu7d4G5fvcnEIoDaMZ/A58erbxu7N5Yw/IE fauQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:user-agent:message-id:date:subject:mail-followup-to:to :from:delivered-to:sender:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:mailing-list:dkim-signature :domainkey-signature:arc-authentication-results; bh=AFaVIfpff5eQxOyTXum1pYVxZTOSXZ7bLJ+c/Xv++O0=; b=Svev6qqOacPu3GMlfkyHBr2z5lbuxCZYkHp6qQK3lngCcAfdriU25GW8bLKLZcJk1Y sOM+TY5t16qXCWgtvDYucGYu+0eeUKz9oDZZBaP2tvcrtnwsWA9gjOSZe2IGWurgEoq1 admvum+sPAe/31ke2/FDnKK6TkHVI+dT9laIpMvB/KbLS9+n/BmR3ghr+AmYxvfjcq1e wz9UmiW/9gwY+4cMcyXqY0lNhmXGnYeMrBTzxbqyJJ71wXSNb1Pa0usG4gDG/fkURqy+ LkOVvtZb+bNjtb0OS3jT024Xtb0XBG4O6kygZ5Wg6JHR0PXZBz590O6Fzbu6xV1H5Jej oLTg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gcc.gnu.org header.s=default header.b=oClIHhDU; spf=pass (google.com: domain of gcc-patches-return-478365-patch=linaro.org@gcc.gnu.org designates 209.132.180.131 as permitted sender) smtp.mailfrom=gcc-patches-return-478365-patch=linaro.org@gcc.gnu.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from sourceware.org (server1.sourceware.org. [209.132.180.131]) by mx.google.com with ESMTPS id p12-v6si7778828pgu.358.2018.05.24.05.08.46 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 24 May 2018 05:08:47 -0700 (PDT) Received-SPF: pass (google.com: domain of gcc-patches-return-478365-patch=linaro.org@gcc.gnu.org designates 209.132.180.131 as permitted sender) client-ip=209.132.180.131; Authentication-Results: mx.google.com; dkim=pass header.i=@gcc.gnu.org header.s=default header.b=oClIHhDU; spf=pass (google.com: domain of gcc-patches-return-478365-patch=linaro.org@gcc.gnu.org designates 209.132.180.131 as permitted sender) smtp.mailfrom=gcc-patches-return-478365-patch=linaro.org@gcc.gnu.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org DomainKey-Signature: a=rsa-sha1; c=nofws; d=gcc.gnu.org; h=list-id :list-unsubscribe:list-archive:list-post:list-help:sender:from :to:subject:date:message-id:mime-version:content-type; q=dns; s= default; b=FuTgJP4REO4HawqJ4EwTb/134PEakhxO6PFv+tAGdu4nGfjj7EZIE D2LIcx6qX1py0UIObFxOIIUrB2JKClIpgQAoMURDWX5ou++SfS3GFhqN1MWLOHx4 UvFOeCoUmvh2BaStLhPmDSxj/i6lGrtJ/xwEXJalzpCIuxCm5IQNqA= DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=gcc.gnu.org; h=list-id :list-unsubscribe:list-archive:list-post:list-help:sender:from :to:subject:date:message-id:mime-version:content-type; s= default; bh=xZxbcYHfczKJUuL6a1WRB6QUqI8=; b=oClIHhDUcF8lbnk9xgLa EPJB31z9InPeWpWUH47aKaRRutrE7eH+cu7fSzKSQYzPgMn/j0/mEQYVeGysQiTy 3aq3iFZOq6LWY5GH/UErmwe52Pf3OqTgkQlwd/ZG1tKFMdONkHvM5k31wfYP3aTF 8OIXP/rAZBZbUohVHB9oorU= Received: (qmail 118156 invoked by alias); 24 May 2018 12:08:32 -0000 Mailing-List: contact gcc-patches-help@gcc.gnu.org; run by ezmlm Precedence: bulk List-Id: List-Unsubscribe: List-Archive: List-Post: List-Help: Sender: gcc-patches-owner@gcc.gnu.org Delivered-To: mailing list gcc-patches@gcc.gnu.org Received: (qmail 118127 invoked by uid 89); 24 May 2018 12:08:30 -0000 Authentication-Results: sourceware.org; auth=none X-Virus-Found: No X-Spam-SWARE-Status: No, score=-15.3 required=5.0 tests=AWL, BAYES_00, GIT_PATCH_1, GIT_PATCH_2, GIT_PATCH_3, KAM_ASCII_DIVIDERS, RCVD_IN_DNSWL_NONE, SPF_PASS autolearn=ham version=3.3.2 spammy=va X-HELO: mail-wm0-f44.google.com Received: from mail-wm0-f44.google.com (HELO mail-wm0-f44.google.com) (74.125.82.44) by sourceware.org (qpsmtpd/0.93/v0.84-503-g423c35a) with ESMTP; Thu, 24 May 2018 12:08:25 +0000 Received: by mail-wm0-f44.google.com with SMTP id j4-v6so4446109wme.1 for ; Thu, 24 May 2018 05:08:25 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:mail-followup-to:subject:date:message-id :user-agent:mime-version; bh=AFaVIfpff5eQxOyTXum1pYVxZTOSXZ7bLJ+c/Xv++O0=; b=dnVdcOUtjLae2BwTUrOPQ2JaaL1igvN4M9DK+oyC934FYvPBfL4s6z8JjsXZT3dJZz W9L25sAUmGVkfk+WDWRzBBh/FN1Pup9sbVFRHXJ/0G9wR5eu8K5DxsImaV/CyHd8pHEw IakCn8qNbv/Ce1eL0W7qY3z+4IIJxVRz3vQmeGOAL4n9TxYD+Ym80j62BySQNfavqCx6 isRCPGo0zGKvh4h86EM1MMn8B4LUnXWNaxiZ2Ki/M72L6JVX6LK6DCIo0850RcTluHs4 jJodKqJzIaVdoxTlHMPbYGeqCwfIU0bx/ySWpVOyUI1iddJHxIKYeTIVPybvWKVuf/qr bdMQ== X-Gm-Message-State: ALKqPwf+hJsfcgLVCipDtmkU+C8o3NI0VZzIORiXV2GgdR7X7LNjrgjG ALZf7cXWW4R/Y++XhiNh/k7f1TwPfeQ= X-Received: by 2002:a1c:8a45:: with SMTP id m66-v6mr7927126wmd.125.1527163701678; Thu, 24 May 2018 05:08:21 -0700 (PDT) Received: from localhost ([217.140.96.141]) by smtp.gmail.com with ESMTPSA id x130-v6sm6380013wme.24.2018.05.24.05.08.19 for (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Thu, 24 May 2018 05:08:20 -0700 (PDT) From: Richard Sandiford To: gcc-patches@gcc.gnu.org Mail-Followup-To: gcc-patches@gcc.gnu.org, richard.sandiford@linaro.org Subject: Add IFN_COND_FMA functions Date: Thu, 24 May 2018 13:07:55 +0100 Message-ID: <87r2m1cl44.fsf@linaro.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/25.3 (gnu/linux) MIME-Version: 1.0 This patch adds conditional equivalents of the IFN_FMA built-in functions. Most of it is just a mechanical extension of the binary stuff. Tested on aarch64-linux-gnu (with and without SVE), aarch64_be-elf and x86_64-linux-gnu. OK for the non-AArch64 bits? Richard 2018-05-24 Richard Sandiford gcc/ * doc/md.texi (cond_fma, cond_fms, cond_fnma, cond_fnms): Document. * optabs.def (cond_fma_optab, cond_fms_optab, cond_fnma_optab) (cond_fnms_optab): New optabs. * internal-fn.def (COND_FMA, COND_FMS, COND_FNMA, COND_FNMS): New internal functions. (FMA): Use DEF_INTERNAL_FLT_FN rather than DEF_INTERNAL_FLT_FLOATN_FN. * internal-fn.h (get_conditional_internal_fn): Declare. (get_unconditional_internal_fn): Likewise. * internal-fn.c (cond_ternary_direct): New macro. (expand_cond_ternary_optab_fn): Likewise. (direct_cond_ternary_optab_supported_p): Likewise. (FOR_EACH_COND_FN_PAIR): Likewise. (get_conditional_internal_fn): New function. (get_unconditional_internal_fn): Likewise. * gimple-match.h (gimple_match_op::MAX_NUM_OPS): Bump to 5. (gimple_match_op::gimple_match_op): Add a new overload for 5 operands. (gimple_match_op::set_op): Likewise. (gimple_resimplify5): Declare. * genmatch.c (decision_tree::gen): Generate simplifications for 5 operands. * gimple-match-head.c (gimple_simplify): Define an overload for 5 operands. Handle calls with 5 arguments in the top-level overload. (convert_conditional_op): Handle conversions from unconditional internal functions to conditional ones. (gimple_resimplify5): New function. (build_call_internal): Pass a fifth operand. (maybe_push_res_to_seq): Likewise. (try_conditional_simplification): Try converting conditional internal functions to unconditional internal functions. Handle 3-operand unconditional forms. * match.pd (UNCOND_TERNARY, COND_TERNARY): Operator lists. Define ternary equivalents of the current rules for binary conditional internal functions. * config/aarch64/aarch64.c (aarch64_preferred_else_value): Handle ternary operations. * config/aarch64/aarch64-sve.md (cond_) (*cond_, *cond__acc): New SVE_COND_FP_TERNARY patterns. * config/aarch64/iterators.md (UNSPEC_COND_FMLA, UNSPEC_COND_FMLS) (UNSPEC_COND_FNMLA, UNSPEC_COND_FNMLS): New unspecs. (optab): Handle them. (SVE_COND_FP_TERNARY): New int iterator. (sve_fmla_op, sve_fmad_op): New int attributes. gcc/testsuite/ * gcc.dg/vect/vect-cond-arith-3.c: New test. * gcc.target/aarch64/sve/vcond_13.c: Likewise. * gcc.target/aarch64/sve/vcond_13_run.c: Likewise. * gcc.target/aarch64/sve/vcond_14.c: Likewise. * gcc.target/aarch64/sve/vcond_14_run.c: Likewise. * gcc.target/aarch64/sve/vcond_15.c: Likewise. * gcc.target/aarch64/sve/vcond_15_run.c: Likewise. * gcc.target/aarch64/sve/vcond_16.c: Likewise. * gcc.target/aarch64/sve/vcond_16_run.c: Likewise. Index: gcc/doc/md.texi =================================================================== --- gcc/doc/md.texi 2018-05-24 10:12:10.142352315 +0100 +++ gcc/doc/md.texi 2018-05-24 13:05:46.047607587 +0100 @@ -6386,6 +6386,23 @@ Operands 0, 2, 3 and 4 all have mode @va integer if @var{m} is scalar, otherwise it has the mode returned by @code{TARGET_VECTORIZE_GET_MASK_MODE}. +@cindex @code{cond_fma@var{mode}} instruction pattern +@cindex @code{cond_fms@var{mode}} instruction pattern +@cindex @code{cond_fnma@var{mode}} instruction pattern +@cindex @code{cond_fnms@var{mode}} instruction pattern +@item @samp{cond_fma@var{mode}} +@itemx @samp{cond_fms@var{mode}} +@itemx @samp{cond_fnma@var{mode}} +@itemx @samp{cond_fnms@var{mode}} +Like @samp{cond_add@var{m}}, except that the conditional operation +takes 3 operands rather than two. For example, the vector form of +@samp{cond_fma@var{mode}} is equivalent to: + +@smallexample +for (i = 0; i < GET_MODE_NUNITS (@var{m}); i++) + op0[i] = op1[i] ? fma (op2[i], op3[i], op4[i]) : op5[i]; +@end smallexample + @cindex @code{neg@var{mode}cc} instruction pattern @item @samp{neg@var{mode}cc} Similar to @samp{mov@var{mode}cc} but for conditional negation. Conditionally Index: gcc/optabs.def =================================================================== --- gcc/optabs.def 2018-05-24 10:12:10.146352152 +0100 +++ gcc/optabs.def 2018-05-24 13:05:46.049605128 +0100 @@ -234,6 +234,10 @@ OPTAB_D (cond_smin_optab, "cond_smin$a") OPTAB_D (cond_smax_optab, "cond_smax$a") OPTAB_D (cond_umin_optab, "cond_umin$a") OPTAB_D (cond_umax_optab, "cond_umax$a") +OPTAB_D (cond_fma_optab, "cond_fma$a") +OPTAB_D (cond_fms_optab, "cond_fms$a") +OPTAB_D (cond_fnma_optab, "cond_fnma$a") +OPTAB_D (cond_fnms_optab, "cond_fnms$a") OPTAB_D (cmov_optab, "cmov$a6") OPTAB_D (cstore_optab, "cstore$a4") OPTAB_D (ctrap_optab, "ctrap$a4") Index: gcc/internal-fn.def =================================================================== --- gcc/internal-fn.def 2018-05-24 10:12:10.146352152 +0100 +++ gcc/internal-fn.def 2018-05-24 13:05:46.048606357 +0100 @@ -59,7 +59,8 @@ along with GCC; see the file COPYING3. - binary: a normal binary optab, such as vec_interleave_lo_ - ternary: a normal ternary optab, such as fma4 - - cond_binary: a conditional binary optab, such as addcc + - cond_binary: a conditional binary optab, such as cond_add + - cond_ternary: a conditional ternary optab, such as cond_fma_rev - fold_left: for scalar = FN (scalar, vector), keyed off the vector mode @@ -162,6 +163,11 @@ DEF_INTERNAL_OPTAB_FN (COND_IOR, ECF_CON DEF_INTERNAL_OPTAB_FN (COND_XOR, ECF_CONST | ECF_NOTHROW, cond_xor, cond_binary) +DEF_INTERNAL_OPTAB_FN (COND_FMA, ECF_CONST, cond_fma, cond_ternary) +DEF_INTERNAL_OPTAB_FN (COND_FMS, ECF_CONST, cond_fms, cond_ternary) +DEF_INTERNAL_OPTAB_FN (COND_FNMA, ECF_CONST, cond_fnma, cond_ternary) +DEF_INTERNAL_OPTAB_FN (COND_FNMS, ECF_CONST, cond_fnms, cond_ternary) + DEF_INTERNAL_OPTAB_FN (RSQRT, ECF_CONST, rsqrt, unary) DEF_INTERNAL_OPTAB_FN (REDUC_PLUS, ECF_CONST | ECF_NOTHROW, @@ -230,7 +236,7 @@ DEF_INTERNAL_OPTAB_FN (XORSIGN, ECF_CONS DEF_INTERNAL_FLT_FN (LDEXP, ECF_CONST, ldexp, binary) /* Ternary math functions. */ -DEF_INTERNAL_FLT_FN (FMA, ECF_CONST, fma, ternary) +DEF_INTERNAL_FLT_FLOATN_FN (FMA, ECF_CONST, fma, ternary) /* Unary integer ops. */ DEF_INTERNAL_INT_FN (CLRSB, ECF_CONST | ECF_NOTHROW, clrsb, unary) Index: gcc/internal-fn.h =================================================================== --- gcc/internal-fn.h 2018-05-24 10:33:30.870095164 +0100 +++ gcc/internal-fn.h 2018-05-24 13:05:46.049605128 +0100 @@ -193,7 +193,9 @@ direct_internal_fn_supported_p (internal extern bool set_edom_supported_p (void); extern internal_fn get_conditional_internal_fn (tree_code); +extern internal_fn get_conditional_internal_fn (internal_fn); extern tree_code conditional_internal_fn_code (internal_fn); +extern internal_fn get_unconditional_internal_fn (internal_fn); extern bool internal_load_fn_p (internal_fn); extern bool internal_store_fn_p (internal_fn); Index: gcc/internal-fn.c =================================================================== --- gcc/internal-fn.c 2018-05-24 10:33:30.870095164 +0100 +++ gcc/internal-fn.c 2018-05-24 13:05:46.048606357 +0100 @@ -113,6 +113,7 @@ #define binary_direct { 0, 0, true } #define ternary_direct { 0, 0, true } #define cond_unary_direct { 1, 1, true } #define cond_binary_direct { 1, 1, true } +#define cond_ternary_direct { 1, 1, true } #define while_direct { 0, 2, false } #define fold_extract_direct { 2, 2, false } #define fold_left_direct { 1, 1, false } @@ -2993,6 +2994,9 @@ #define expand_cond_unary_optab_fn(FN, S #define expand_cond_binary_optab_fn(FN, STMT, OPTAB) \ expand_direct_optab_fn (FN, STMT, OPTAB, 4) +#define expand_cond_ternary_optab_fn(FN, STMT, OPTAB) \ + expand_direct_optab_fn (FN, STMT, OPTAB, 5) + #define expand_fold_extract_optab_fn(FN, STMT, OPTAB) \ expand_direct_optab_fn (FN, STMT, OPTAB, 3) @@ -3075,6 +3079,7 @@ #define direct_binary_optab_supported_p #define direct_ternary_optab_supported_p direct_optab_supported_p #define direct_cond_unary_optab_supported_p direct_optab_supported_p #define direct_cond_binary_optab_supported_p direct_optab_supported_p +#define direct_cond_ternary_optab_supported_p direct_optab_supported_p #define direct_mask_load_optab_supported_p direct_optab_supported_p #define direct_load_lanes_optab_supported_p multi_vector_optab_supported_p #define direct_mask_load_lanes_optab_supported_p multi_vector_optab_supported_p @@ -3277,6 +3282,57 @@ #define CASE(CODE, IFN) case IFN: return } } +/* Invoke T(IFN) for each internal function IFN that also has an + IFN_COND_* form. */ +#define FOR_EACH_COND_FN_PAIR(T) \ + T (FMA) \ + T (FMS) \ + T (FNMA) \ + T (FNMS) + +/* Return a function that only performs internal function FN when a + certain condition is met and that uses a given fallback value otherwise. + In other words, the returned function FN' is such that: + + LHS = FN' (COND, A1, ... An, ELSE) + + is equivalent to the C expression: + + LHS = COND ? FN (A1, ..., An) : ELSE; + + operating elementwise if the operands are vectors. + + Return IFN_LAST if no such function exists. */ + +internal_fn +get_conditional_internal_fn (internal_fn fn) +{ + switch (fn) + { +#define CASE(NAME) case IFN_##NAME: return IFN_COND_##NAME; + FOR_EACH_COND_FN_PAIR(CASE) +#undef CASE + default: + return IFN_LAST; + } +} + +/* If IFN implements the conditional form of an unconditional internal + function, return that unconditional function, otherwise return IFN_LAST. */ + +internal_fn +get_unconditional_internal_fn (internal_fn ifn) +{ + switch (ifn) + { +#define CASE(NAME) case IFN_COND_##NAME: return IFN_##NAME; + FOR_EACH_COND_FN_PAIR(CASE) +#undef CASE + default: + return IFN_LAST; + } +} + /* Return true if IFN is some form of load from memory. */ bool Index: gcc/gimple-match.h =================================================================== --- gcc/gimple-match.h 2018-05-24 10:33:30.870095164 +0100 +++ gcc/gimple-match.h 2018-05-24 13:05:46.048606357 +0100 @@ -91,18 +91,21 @@ struct gimple_match_op code_helper, tree, tree, tree, tree); gimple_match_op (const gimple_match_cond &, code_helper, tree, tree, tree, tree, tree); + gimple_match_op (const gimple_match_cond &, + code_helper, tree, tree, tree, tree, tree, tree); void set_op (code_helper, tree, unsigned int); void set_op (code_helper, tree, tree); void set_op (code_helper, tree, tree, tree); void set_op (code_helper, tree, tree, tree, tree); void set_op (code_helper, tree, tree, tree, tree, tree); + void set_op (code_helper, tree, tree, tree, tree, tree, tree); void set_value (tree); tree op_or_null (unsigned int) const; /* The maximum value of NUM_OPS. */ - static const unsigned int MAX_NUM_OPS = 4; + static const unsigned int MAX_NUM_OPS = 5; /* The conditions under which the operation is performed, and the value to use as a fallback. */ @@ -182,6 +185,20 @@ gimple_match_op::gimple_match_op (const ops[3] = op3; } +inline +gimple_match_op::gimple_match_op (const gimple_match_cond &cond_in, + code_helper code_in, tree type_in, + tree op0, tree op1, tree op2, tree op3, + tree op4) + : cond (cond_in), code (code_in), type (type_in), num_ops (5) +{ + ops[0] = op0; + ops[1] = op1; + ops[2] = op2; + ops[3] = op3; + ops[4] = op4; +} + /* Change the operation performed to CODE_IN, the type of the result to TYPE_IN, and the number of operands to NUM_OPS_IN. The caller needs to set the operands itself. */ @@ -242,6 +259,20 @@ gimple_match_op::set_op (code_helper cod ops[3] = op3; } +inline void +gimple_match_op::set_op (code_helper code_in, tree type_in, + tree op0, tree op1, tree op2, tree op3, tree op4) +{ + code = code_in; + type = type_in; + num_ops = 5; + ops[0] = op0; + ops[1] = op1; + ops[2] = op2; + ops[3] = op3; + ops[4] = op4; +} + /* Set the "operation" to be the single value VALUE, such as a constant or SSA_NAME. */ @@ -279,6 +310,7 @@ bool gimple_resimplify1 (gimple_seq *, g bool gimple_resimplify2 (gimple_seq *, gimple_match_op *, tree (*)(tree)); bool gimple_resimplify3 (gimple_seq *, gimple_match_op *, tree (*)(tree)); bool gimple_resimplify4 (gimple_seq *, gimple_match_op *, tree (*)(tree)); +bool gimple_resimplify5 (gimple_seq *, gimple_match_op *, tree (*)(tree)); tree maybe_push_res_to_seq (gimple_match_op *, gimple_seq *, tree res = NULL_TREE); void maybe_build_generic_op (gimple_match_op *); Index: gcc/genmatch.c =================================================================== --- gcc/genmatch.c 2018-05-24 10:33:30.869095197 +0100 +++ gcc/genmatch.c 2018-05-24 13:05:46.048606357 +0100 @@ -3760,7 +3760,7 @@ decision_tree::gen (FILE *f, bool gimple } fprintf (stderr, "removed %u duplicate tails\n", rcnt); - for (unsigned n = 1; n <= 4; ++n) + for (unsigned n = 1; n <= 5; ++n) { /* First generate split-out functions. */ for (unsigned i = 0; i < root->kids.length (); i++) Index: gcc/gimple-match-head.c =================================================================== --- gcc/gimple-match-head.c 2018-05-24 10:33:30.870095164 +0100 +++ gcc/gimple-match-head.c 2018-05-24 13:05:46.048606357 +0100 @@ -54,6 +54,8 @@ static bool gimple_simplify (gimple_matc code_helper, tree, tree, tree, tree); static bool gimple_simplify (gimple_match_op *, gimple_seq *, tree (*)(tree), code_helper, tree, tree, tree, tree, tree); +static bool gimple_simplify (gimple_match_op *, gimple_seq *, tree (*)(tree), + code_helper, tree, tree, tree, tree, tree, tree); const unsigned int gimple_match_op::MAX_NUM_OPS; @@ -80,7 +82,12 @@ convert_conditional_op (gimple_match_op if (orig_op->code.is_tree_code ()) ifn = get_conditional_internal_fn ((tree_code) orig_op->code); else - return false; + { + combined_fn cfn = orig_op->code; + if (!internal_fn_p (cfn)) + return false; + ifn = get_conditional_internal_fn (as_internal_fn (cfn)); + } if (ifn == IFN_LAST) return false; unsigned int num_ops = orig_op->num_ops; @@ -347,6 +354,34 @@ gimple_resimplify4 (gimple_seq *seq, gim return false; } +/* Helper that matches and simplifies the toplevel result from + a gimple_simplify run (where we don't want to build + a stmt in case it's used in in-place folding). Replaces + RES_OP with a simplified and/or canonicalized result and + returns whether any change was made. */ + +bool +gimple_resimplify5 (gimple_seq *seq, gimple_match_op *res_op, + tree (*valueize)(tree)) +{ + /* No constant folding is defined for five-operand functions. */ + + gimple_match_op res_op2 (*res_op); + if (gimple_simplify (&res_op2, seq, valueize, + res_op->code, res_op->type, + res_op->ops[0], res_op->ops[1], res_op->ops[2], + res_op->ops[3], res_op->ops[4])) + { + *res_op = res_op2; + return true; + } + + if (maybe_resimplify_conditional_op (seq, res_op, valueize)) + return true; + + return false; +} + /* If in GIMPLE the operation described by RES_OP should be single-rhs, build a GENERIC tree for that expression and update RES_OP accordingly. */ @@ -388,7 +423,8 @@ build_call_internal (internal_fn fn, gim res_op->op_or_null (0), res_op->op_or_null (1), res_op->op_or_null (2), - res_op->op_or_null (3)); + res_op->op_or_null (3), + res_op->op_or_null (4)); } /* Push the exploded expression described by RES_OP as a statement to @@ -482,7 +518,8 @@ maybe_push_res_to_seq (gimple_match_op * res_op->op_or_null (0), res_op->op_or_null (1), res_op->op_or_null (2), - res_op->op_or_null (3)); + res_op->op_or_null (3), + res_op->op_or_null (4)); } if (!res) { @@ -689,14 +726,22 @@ do_valueize (tree op, tree (*valueize)(t try_conditional_simplification (internal_fn ifn, gimple_match_op *res_op, gimple_seq *seq, tree (*valueize) (tree)) { + code_helper op; tree_code code = conditional_internal_fn_code (ifn); - if (code == ERROR_MARK) - return false; + if (code != ERROR_MARK) + op = code; + else + { + ifn = get_unconditional_internal_fn (ifn); + if (ifn == IFN_LAST) + return false; + op = as_combined_fn (ifn); + } unsigned int num_ops = res_op->num_ops; gimple_match_op cond_op (gimple_match_cond (res_op->ops[0], res_op->ops[num_ops - 1]), - code, res_op->type, num_ops - 2); + op, res_op->type, num_ops - 2); for (unsigned int i = 1; i < num_ops - 1; ++i) cond_op.ops[i - 1] = res_op->ops[i]; switch (num_ops - 2) @@ -705,6 +750,10 @@ try_conditional_simplification (internal if (!gimple_resimplify2 (seq, &cond_op, valueize)) return false; break; + case 3: + if (!gimple_resimplify3 (seq, &cond_op, valueize)) + return false; + break; default: gcc_unreachable (); } @@ -837,7 +886,7 @@ gimple_simplify (gimple *stmt, gimple_ma /* ??? This way we can't simplify calls with side-effects. */ if (gimple_call_lhs (stmt) != NULL_TREE && gimple_call_num_args (stmt) >= 1 - && gimple_call_num_args (stmt) <= 4) + && gimple_call_num_args (stmt) <= 5) { bool valueized = false; combined_fn cfn; @@ -887,6 +936,9 @@ gimple_simplify (gimple *stmt, gimple_ma case 4: return (gimple_resimplify4 (seq, res_op, valueize) || valueized); + case 5: + return (gimple_resimplify5 (seq, res_op, valueize) + || valueized); default: gcc_unreachable (); } Index: gcc/match.pd =================================================================== --- gcc/match.pd 2018-05-24 10:33:30.870095164 +0100 +++ gcc/match.pd 2018-05-24 13:05:46.049605128 +0100 @@ -86,6 +86,12 @@ DEFINE_INT_AND_FLOAT_ROUND_FN (RINT) IFN_COND_MUL IFN_COND_DIV IFN_COND_MOD IFN_COND_RDIV IFN_COND_MIN IFN_COND_MAX IFN_COND_AND IFN_COND_IOR IFN_COND_XOR) + +/* Same for ternary operations. */ +(define_operator_list UNCOND_TERNARY + IFN_FMA IFN_FMS IFN_FNMA IFN_FNMS) +(define_operator_list COND_TERNARY + IFN_COND_FMA IFN_COND_FMS IFN_COND_FNMA IFN_COND_FNMS) /* As opposed to convert?, this still creates a single pattern, so it is not a suitable replacement for convert? in all cases. */ @@ -4798,6 +4804,21 @@ DEFINE_INT_AND_FLOAT_ROUND_FN (RINT) (if (element_precision (type) == element_precision (op_type)) (view_convert (cond_op (bit_not @0) @2 @3 (view_convert:op_type @1))))))) +/* Same for ternary operations. */ +(for uncond_op (UNCOND_TERNARY) + cond_op (COND_TERNARY) + (simplify + (vec_cond @0 (view_convert? (uncond_op@5 @1 @2 @3)) @4) + (with { tree op_type = TREE_TYPE (@5); } + (if (element_precision (type) == element_precision (op_type)) + (view_convert (cond_op @0 @1 @2 @3 (view_convert:op_type @4)))))) + (simplify + (vec_cond @0 @1 (view_convert? (uncond_op@5 @2 @3 @4))) + (with { tree op_type = TREE_TYPE (@5); } + (if (element_precision (type) == element_precision (op_type)) + (view_convert (cond_op (bit_not @0) @2 @3 @4 + (view_convert:op_type @1))))))) + /* Detect cases in which a VEC_COND_EXPR effectively replaces the "else" value of an IFN_COND_*. */ (for cond_op (COND_BINARY) @@ -4806,3 +4827,11 @@ DEFINE_INT_AND_FLOAT_ROUND_FN (RINT) (with { tree op_type = TREE_TYPE (@3); } (if (element_precision (type) == element_precision (op_type)) (view_convert (cond_op @0 @1 @2 (view_convert:op_type @4))))))) + +/* Same for ternary operations. */ +(for cond_op (COND_TERNARY) + (simplify + (vec_cond @0 (view_convert? (cond_op @0 @1 @2 @3 @4)) @5) + (with { tree op_type = TREE_TYPE (@4); } + (if (element_precision (type) == element_precision (op_type)) + (view_convert (cond_op @0 @1 @2 @3 (view_convert:op_type @5))))))) Index: gcc/config/aarch64/aarch64.c =================================================================== --- gcc/config/aarch64/aarch64.c 2018-05-24 10:33:30.867095262 +0100 +++ gcc/config/aarch64/aarch64.c 2018-05-24 13:05:46.046608817 +0100 @@ -1292,14 +1292,18 @@ aarch64_get_mask_mode (poly_uint64 nunit return default_get_mask_mode (nunits, nbytes); } -/* Implement TARGET_PREFERRED_ELSE_VALUE. Prefer to use the first - arithmetic operand as the else value if the else value doesn't matter, - since that exactly matches the SVE destructive merging form. */ +/* Implement TARGET_PREFERRED_ELSE_VALUE. For binary operations, + prefer to use the first arithmetic operand as the else value if + the else value doesn't matter, since that exactly matches the SVE + destructive merging form. For ternary operations we could either + pick the first operand and use FMAD-like instructions or the last + operand and use FMLA-like instructions; the latter seems more + natural. */ static tree -aarch64_preferred_else_value (unsigned, tree, unsigned int, tree *ops) +aarch64_preferred_else_value (unsigned, tree, unsigned int nops, tree *ops) { - return ops[0]; + return nops == 3 ? ops[2] : ops[0]; } /* Implement TARGET_HARD_REGNO_NREGS. */ Index: gcc/config/aarch64/aarch64-sve.md =================================================================== --- gcc/config/aarch64/aarch64-sve.md 2018-05-24 10:12:10.141352356 +0100 +++ gcc/config/aarch64/aarch64-sve.md 2018-05-24 13:05:46.044611277 +0100 @@ -2688,6 +2688,58 @@ (define_insn "*cond_" "r\t%0., %1/m, %0., %2." ) +;; Predicated floating-point ternary operations with select. +(define_expand "cond_" + [(set (match_operand:SVE_F 0 "register_operand") + (unspec:SVE_F + [(match_operand: 1 "register_operand") + (unspec:SVE_F + [(match_dup 1) + (match_operand:SVE_F 2 "register_operand") + (match_operand:SVE_F 3 "register_operand") + (match_operand:SVE_F 4 "register_operand")] + SVE_COND_FP_TERNARY) + (match_operand:SVE_F 5 "register_operand")] + UNSPEC_SEL))] + "TARGET_SVE" +{ + aarch64_sve_prepare_conditional_op (operands, 6, true); +}) + +;; Predicated floating-point ternary operations using the FMAD-like form. +(define_insn "*cond_" + [(set (match_operand:SVE_F 0 "register_operand" "=w") + (unspec:SVE_F + [(match_operand: 1 "register_operand" "Upl") + (unspec:SVE_F + [(match_dup 1) + (match_operand:SVE_F 2 "register_operand" "0") + (match_operand:SVE_F 3 "register_operand" "w") + (match_operand:SVE_F 4 "register_operand" "w")] + SVE_COND_FP_TERNARY) + (match_dup 2)] + UNSPEC_SEL))] + "TARGET_SVE" + "\t%0., %1/m, %3., %4." +) + +;; Predicated floating-point ternary operations using the FMLA-like form. +(define_insn "*cond__acc" + [(set (match_operand:SVE_F 0 "register_operand" "=w") + (unspec:SVE_F + [(match_operand: 1 "register_operand" "Upl") + (unspec:SVE_F + [(match_dup 1) + (match_operand:SVE_F 2 "register_operand" "w") + (match_operand:SVE_F 3 "register_operand" "w") + (match_operand:SVE_F 4 "register_operand" "0")] + SVE_COND_FP_TERNARY) + (match_dup 4)] + UNSPEC_SEL))] + "TARGET_SVE" + "\t%0., %1/m, %2., %3." +) + ;; Shift an SVE vector left and insert a scalar into element 0. (define_insn "vec_shl_insert_" [(set (match_operand:SVE_ALL 0 "register_operand" "=w, w") Index: gcc/config/aarch64/iterators.md =================================================================== --- gcc/config/aarch64/iterators.md 2018-05-24 10:12:10.142352315 +0100 +++ gcc/config/aarch64/iterators.md 2018-05-24 13:05:46.046608817 +0100 @@ -468,6 +468,10 @@ (define_c_enum "unspec" UNSPEC_COND_DIV ; Used in aarch64-sve.md. UNSPEC_COND_MAX ; Used in aarch64-sve.md. UNSPEC_COND_MIN ; Used in aarch64-sve.md. + UNSPEC_COND_FMLA ; Used in aarch64-sve.md. + UNSPEC_COND_FMLS ; Used in aarch64-sve.md. + UNSPEC_COND_FNMLA ; Used in aarch64-sve.md. + UNSPEC_COND_FNMLS ; Used in aarch64-sve.md. UNSPEC_COND_LT ; Used in aarch64-sve.md. UNSPEC_COND_LE ; Used in aarch64-sve.md. UNSPEC_COND_EQ ; Used in aarch64-sve.md. @@ -1549,6 +1553,11 @@ (define_int_iterator SVE_COND_FP_BINARY (define_int_iterator SVE_COND_FP_BINARY_REV [UNSPEC_COND_SUB UNSPEC_COND_DIV]) +(define_int_iterator SVE_COND_FP_TERNARY [UNSPEC_COND_FMLA + UNSPEC_COND_FMLS + UNSPEC_COND_FNMLA + UNSPEC_COND_FNMLS]) + (define_int_iterator SVE_COND_FP_CMP [UNSPEC_COND_LT UNSPEC_COND_LE UNSPEC_COND_EQ UNSPEC_COND_NE UNSPEC_COND_GE UNSPEC_COND_GT]) @@ -1581,7 +1590,11 @@ (define_int_attr optab [(UNSPEC_ANDF "an (UNSPEC_COND_MUL "mul") (UNSPEC_COND_DIV "div") (UNSPEC_COND_MAX "smax") - (UNSPEC_COND_MIN "smin")]) + (UNSPEC_COND_MIN "smin") + (UNSPEC_COND_FMLA "fma") + (UNSPEC_COND_FMLS "fnma") + (UNSPEC_COND_FNMLA "fnms") + (UNSPEC_COND_FNMLS "fms")]) (define_int_attr maxmin_uns [(UNSPEC_UMAXV "umax") (UNSPEC_UMINV "umin") @@ -1799,6 +1812,16 @@ (define_int_attr sve_fp_op [(UNSPEC_COND (UNSPEC_COND_MAX "fmaxnm") (UNSPEC_COND_MIN "fminnm")]) +(define_int_attr sve_fmla_op [(UNSPEC_COND_FMLA "fmla") + (UNSPEC_COND_FMLS "fmls") + (UNSPEC_COND_FNMLA "fnmla") + (UNSPEC_COND_FNMLS "fnmls")]) + +(define_int_attr sve_fmad_op [(UNSPEC_COND_FMLA "fmad") + (UNSPEC_COND_FMLS "fmsb") + (UNSPEC_COND_FNMLA "fnmad") + (UNSPEC_COND_FNMLS "fnmsb")]) + (define_int_attr commutative [(UNSPEC_COND_ADD "true") (UNSPEC_COND_SUB "false") (UNSPEC_COND_MUL "true") Index: gcc/testsuite/gcc.dg/vect/vect-cond-arith-3.c =================================================================== --- /dev/null 2018-04-20 16:19:46.369131350 +0100 +++ gcc/testsuite/gcc.dg/vect/vect-cond-arith-3.c 2018-05-24 13:05:46.049605128 +0100 @@ -0,0 +1,63 @@ +/* { dg-require-effective-target scalar_all_fma } */ +/* { dg-additional-options "-fdump-tree-optimized" } */ + +#include "tree-vect.h" + +#define N (VECTOR_BITS * 11 / 64 + 3) + +#define DEF(INV) \ + void __attribute__ ((noipa)) \ + f_##INV (double *restrict a, double *restrict b, \ + double *restrict c, double *restrict d) \ + { \ + for (int i = 0; i < N; ++i) \ + { \ + double mb = (INV & 1 ? -b[i] : b[i]); \ + double mc = c[i]; \ + double md = (INV & 2 ? -d[i] : d[i]); \ + double fma = __builtin_fma (mb, mc, md); \ + double truev = (INV & 4 ? -fma : fma); \ + a[i] = b[i] < 10 ? truev : 10.0; \ + } \ + } + +#define TEST(INV) \ + { \ + f_##INV (a, b, c, d); \ + for (int i = 0; i < N; ++i) \ + { \ + double mb = (INV & 1 ? -b[i] : b[i]); \ + double mc = c[i]; \ + double md = (INV & 2 ? -d[i] : d[i]); \ + double fma = __builtin_fma (mb, mc, md); \ + double truev = (INV & 4 ? -fma : fma); \ + if (a[i] != (i % 17 < 10 ? truev : 10.0)) \ + __builtin_abort (); \ + asm volatile ("" ::: "memory"); \ + } \ + } + +#define FOR_EACH_INV(T) \ + T (0) T (1) T (2) T (3) T (4) T (5) T (6) T (7) + +FOR_EACH_INV (DEF) + +int +main (void) +{ + double a[N], b[N], c[N], d[N]; + for (int i = 0; i < N; ++i) + { + b[i] = i % 17; + c[i] = i % 9 + 11; + d[i] = i % 13 + 14; + asm volatile ("" ::: "memory"); + } + FOR_EACH_INV (TEST) + return 0; +} + +/* { dg-final { scan-tree-dump-times { = \.COND_FMA } 2 "optimized" { target vect_double_cond_arith } } } */ +/* { dg-final { scan-tree-dump-times { = \.COND_FMS } 2 "optimized" { target vect_double_cond_arith } } } */ +/* { dg-final { scan-tree-dump-times { = \.COND_FNMA } 2 "optimized" { target vect_double_cond_arith } } } */ +/* { dg-final { scan-tree-dump-times { = \.COND_FNMS } 2 "optimized" { target vect_double_cond_arith } } } */ Index: gcc/testsuite/gcc.target/aarch64/sve/vcond_13.c =================================================================== --- /dev/null 2018-04-20 16:19:46.369131350 +0100 +++ gcc/testsuite/gcc.target/aarch64/sve/vcond_13.c 2018-05-24 13:05:46.049605128 +0100 @@ -0,0 +1,58 @@ +/* { dg-do compile } */ +/* { dg-options "-O2 -ftree-vectorize" } */ + +#define N 119 + +#define DEF_LOOP(INV, TYPE, CMPTYPE, SUFFIX) \ + void __attribute__ ((noipa)) \ + f_##INV##_##SUFFIX (TYPE *restrict a, TYPE *restrict b, \ + TYPE *restrict c, TYPE *restrict d, \ + CMPTYPE *restrict cond) \ + { \ + for (int i = 0; i < N; ++i) \ + { \ + TYPE mb = (INV & 1 ? -b[i] : b[i]); \ + TYPE mc = c[i]; \ + TYPE md = (INV & 2 ? -d[i] : d[i]); \ + TYPE fma = __builtin_fma##SUFFIX (mb, mc, md); \ + TYPE truev = (INV & 4 ? -fma : fma); \ + a[i] = cond[i] < 10 ? truev : b[i]; \ + } \ + } + +#define FOR_EACH_TYPE(T, INV) \ + T (INV, _Float16, short, f16) \ + T (INV, float, float, f32) \ + T (INV, double, double, f64) + +#define FOR_EACH_INV(T) \ + FOR_EACH_TYPE (T, 0) \ + FOR_EACH_TYPE (T, 1) \ + FOR_EACH_TYPE (T, 2) \ + FOR_EACH_TYPE (T, 3) \ + FOR_EACH_TYPE (T, 4) \ + FOR_EACH_TYPE (T, 5) \ + FOR_EACH_TYPE (T, 6) \ + FOR_EACH_TYPE (T, 7) + +FOR_EACH_INV (DEF_LOOP) + +/* { dg-final { scan-assembler-not {\tsel\t} } } */ +/* { dg-final { scan-assembler-not {\tmovprfx\t} } } */ +/* { dg-final { scan-assembler-not {\tmov\tz[0-9]+\.., z[0-9]+} } } */ + +/* { dg-final { scan-assembler-times {\tfmad\tz[0-9]+\.h,} 2 } } */ +/* { dg-final { scan-assembler-times {\tfmad\tz[0-9]+\.s,} 2 } } */ +/* { dg-final { scan-assembler-times {\tfmad\tz[0-9]+\.d,} 2 } } */ + +/* { dg-final { scan-assembler-times {\tfmsb\tz[0-9]+\.h,} 2 } } */ +/* { dg-final { scan-assembler-times {\tfmsb\tz[0-9]+\.s,} 2 } } */ +/* { dg-final { scan-assembler-times {\tfmsb\tz[0-9]+\.d,} 2 } } */ + +/* { dg-final { scan-assembler-times {\tfnmad\tz[0-9]+\.h,} 2 } } */ +/* { dg-final { scan-assembler-times {\tfnmad\tz[0-9]+\.s,} 2 } } */ +/* { dg-final { scan-assembler-times {\tfnmad\tz[0-9]+\.d,} 2 } } */ + +/* { dg-final { scan-assembler-times {\tfnmsb\tz[0-9]+\.h,} 2 } } */ +/* { dg-final { scan-assembler-times {\tfnmsb\tz[0-9]+\.s,} 2 } } */ +/* { dg-final { scan-assembler-times {\tfnmsb\tz[0-9]+\.d,} 2 } } */ Index: gcc/testsuite/gcc.target/aarch64/sve/vcond_13_run.c =================================================================== --- /dev/null 2018-04-20 16:19:46.369131350 +0100 +++ gcc/testsuite/gcc.target/aarch64/sve/vcond_13_run.c 2018-05-24 13:05:46.050603898 +0100 @@ -0,0 +1,37 @@ +/* { dg-do run { target aarch64_sve_hw } } */ +/* { dg-options "-O2 -ftree-vectorize" } */ + +#include "vcond_13.c" + +#define TEST_LOOP(INV, TYPE, CMPTYPE, SUFFIX) \ + { \ + TYPE a[N], b[N], c[N], d[N]; \ + CMPTYPE cond[N]; \ + for (int i = 0; i < N; ++i) \ + { \ + b[i] = i % 15; \ + c[i] = i % 9 + 11; \ + d[i] = i % 13 + 14; \ + cond[i] = i % 17; \ + asm volatile ("" ::: "memory"); \ + } \ + f_##INV##_##SUFFIX (a, b, c, d, cond); \ + for (int i = 0; i < N; ++i) \ + { \ + double mb = (INV & 1 ? -b[i] : b[i]); \ + double mc = c[i]; \ + double md = (INV & 2 ? -d[i] : d[i]); \ + double fma = __builtin_fma (mb, mc, md); \ + double truev = (INV & 4 ? -fma : fma); \ + if (a[i] != (i % 17 < 10 ? truev : b[i])) \ + __builtin_abort (); \ + asm volatile ("" ::: "memory"); \ + } \ + } + +int +main (void) +{ + FOR_EACH_INV (TEST_LOOP) + return 0; +} Index: gcc/testsuite/gcc.target/aarch64/sve/vcond_14.c =================================================================== --- /dev/null 2018-04-20 16:19:46.369131350 +0100 +++ gcc/testsuite/gcc.target/aarch64/sve/vcond_14.c 2018-05-24 13:05:46.050603898 +0100 @@ -0,0 +1,58 @@ +/* { dg-do compile } */ +/* { dg-options "-O2 -ftree-vectorize" } */ + +#define N 119 + +#define DEF_LOOP(INV, TYPE, CMPTYPE, SUFFIX) \ + void __attribute__ ((noipa)) \ + f_##INV##_##SUFFIX (TYPE *restrict a, TYPE *restrict b, \ + TYPE *restrict c, TYPE *restrict d, \ + CMPTYPE *restrict cond) \ + { \ + for (int i = 0; i < N; ++i) \ + { \ + TYPE mb = (INV & 1 ? -b[i] : b[i]); \ + TYPE mc = c[i]; \ + TYPE md = (INV & 2 ? -d[i] : d[i]); \ + TYPE fma = __builtin_fma##SUFFIX (mb, mc, md); \ + TYPE truev = (INV & 4 ? -fma : fma); \ + a[i] = cond[i] < 10 ? truev : c[i]; \ + } \ + } + +#define FOR_EACH_TYPE(T, INV) \ + T (INV, _Float16, short, f16) \ + T (INV, float, float, f32) \ + T (INV, double, double, f64) + +#define FOR_EACH_INV(T) \ + FOR_EACH_TYPE (T, 0) \ + FOR_EACH_TYPE (T, 1) \ + FOR_EACH_TYPE (T, 2) \ + FOR_EACH_TYPE (T, 3) \ + FOR_EACH_TYPE (T, 4) \ + FOR_EACH_TYPE (T, 5) \ + FOR_EACH_TYPE (T, 6) \ + FOR_EACH_TYPE (T, 7) + +FOR_EACH_INV (DEF_LOOP) + +/* { dg-final { scan-assembler-not {\tsel\t} } } */ +/* { dg-final { scan-assembler-not {\tmovprfx\t} } } */ +/* { dg-final { scan-assembler-not {\tmov\tz[0-9]+\.., z[0-9]+} } } */ + +/* { dg-final { scan-assembler-times {\tfmad\tz[0-9]+\.h,} 2 } } */ +/* { dg-final { scan-assembler-times {\tfmad\tz[0-9]+\.s,} 2 } } */ +/* { dg-final { scan-assembler-times {\tfmad\tz[0-9]+\.d,} 2 } } */ + +/* { dg-final { scan-assembler-times {\tfmsb\tz[0-9]+\.h,} 2 } } */ +/* { dg-final { scan-assembler-times {\tfmsb\tz[0-9]+\.s,} 2 } } */ +/* { dg-final { scan-assembler-times {\tfmsb\tz[0-9]+\.d,} 2 } } */ + +/* { dg-final { scan-assembler-times {\tfnmad\tz[0-9]+\.h,} 2 } } */ +/* { dg-final { scan-assembler-times {\tfnmad\tz[0-9]+\.s,} 2 } } */ +/* { dg-final { scan-assembler-times {\tfnmad\tz[0-9]+\.d,} 2 } } */ + +/* { dg-final { scan-assembler-times {\tfnmsb\tz[0-9]+\.h,} 2 } } */ +/* { dg-final { scan-assembler-times {\tfnmsb\tz[0-9]+\.s,} 2 } } */ +/* { dg-final { scan-assembler-times {\tfnmsb\tz[0-9]+\.d,} 2 } } */ Index: gcc/testsuite/gcc.target/aarch64/sve/vcond_14_run.c =================================================================== --- /dev/null 2018-04-20 16:19:46.369131350 +0100 +++ gcc/testsuite/gcc.target/aarch64/sve/vcond_14_run.c 2018-05-24 13:05:46.050603898 +0100 @@ -0,0 +1,37 @@ +/* { dg-do run { target aarch64_sve_hw } } */ +/* { dg-options "-O2 -ftree-vectorize" } */ + +#include "vcond_14.c" + +#define TEST_LOOP(INV, TYPE, CMPTYPE, SUFFIX) \ + { \ + TYPE a[N], b[N], c[N], d[N]; \ + CMPTYPE cond[N]; \ + for (int i = 0; i < N; ++i) \ + { \ + b[i] = i % 15; \ + c[i] = i % 9 + 11; \ + d[i] = i % 13 + 14; \ + cond[i] = i % 17; \ + asm volatile ("" ::: "memory"); \ + } \ + f_##INV##_##SUFFIX (a, b, c, d, cond); \ + for (int i = 0; i < N; ++i) \ + { \ + double mb = (INV & 1 ? -b[i] : b[i]); \ + double mc = c[i]; \ + double md = (INV & 2 ? -d[i] : d[i]); \ + double fma = __builtin_fma (mb, mc, md); \ + double truev = (INV & 4 ? -fma : fma); \ + if (a[i] != (i % 17 < 10 ? truev : c[i])) \ + __builtin_abort (); \ + asm volatile ("" ::: "memory"); \ + } \ + } + +int +main (void) +{ + FOR_EACH_INV (TEST_LOOP) + return 0; +} Index: gcc/testsuite/gcc.target/aarch64/sve/vcond_15.c =================================================================== --- /dev/null 2018-04-20 16:19:46.369131350 +0100 +++ gcc/testsuite/gcc.target/aarch64/sve/vcond_15.c 2018-05-24 13:05:46.050603898 +0100 @@ -0,0 +1,58 @@ +/* { dg-do compile } */ +/* { dg-options "-O2 -ftree-vectorize" } */ + +#define N 119 + +#define DEF_LOOP(INV, TYPE, CMPTYPE, SUFFIX) \ + void __attribute__ ((noipa)) \ + f_##INV##_##SUFFIX (TYPE *restrict a, TYPE *restrict b, \ + TYPE *restrict c, TYPE *restrict d, \ + CMPTYPE *restrict cond) \ + { \ + for (int i = 0; i < N; ++i) \ + { \ + TYPE mb = (INV & 1 ? -b[i] : b[i]); \ + TYPE mc = c[i]; \ + TYPE md = (INV & 2 ? -d[i] : d[i]); \ + TYPE fma = __builtin_fma##SUFFIX (mb, mc, md); \ + TYPE truev = (INV & 4 ? -fma : fma); \ + a[i] = cond[i] < 10 ? truev : d[i]; \ + } \ + } + +#define FOR_EACH_TYPE(T, INV) \ + T (INV, _Float16, short, f16) \ + T (INV, float, float, f32) \ + T (INV, double, double, f64) + +#define FOR_EACH_INV(T) \ + FOR_EACH_TYPE (T, 0) \ + FOR_EACH_TYPE (T, 1) \ + FOR_EACH_TYPE (T, 2) \ + FOR_EACH_TYPE (T, 3) \ + FOR_EACH_TYPE (T, 4) \ + FOR_EACH_TYPE (T, 5) \ + FOR_EACH_TYPE (T, 6) \ + FOR_EACH_TYPE (T, 7) + +FOR_EACH_INV (DEF_LOOP) + +/* { dg-final { scan-assembler-not {\tsel\t} } } */ +/* { dg-final { scan-assembler-not {\tmovprfx\t} } } */ +/* { dg-final { scan-assembler-not {\tmov\tz[0-9]+\.., z[0-9]+} } } */ + +/* { dg-final { scan-assembler-times {\tfmla\tz[0-9]+\.h,} 2 } } */ +/* { dg-final { scan-assembler-times {\tfmla\tz[0-9]+\.s,} 2 } } */ +/* { dg-final { scan-assembler-times {\tfmla\tz[0-9]+\.d,} 2 } } */ + +/* { dg-final { scan-assembler-times {\tfmls\tz[0-9]+\.h,} 2 } } */ +/* { dg-final { scan-assembler-times {\tfmls\tz[0-9]+\.s,} 2 } } */ +/* { dg-final { scan-assembler-times {\tfmls\tz[0-9]+\.d,} 2 } } */ + +/* { dg-final { scan-assembler-times {\tfnmla\tz[0-9]+\.h,} 2 } } */ +/* { dg-final { scan-assembler-times {\tfnmla\tz[0-9]+\.s,} 2 } } */ +/* { dg-final { scan-assembler-times {\tfnmla\tz[0-9]+\.d,} 2 } } */ + +/* { dg-final { scan-assembler-times {\tfnmls\tz[0-9]+\.h,} 2 } } */ +/* { dg-final { scan-assembler-times {\tfnmls\tz[0-9]+\.s,} 2 } } */ +/* { dg-final { scan-assembler-times {\tfnmls\tz[0-9]+\.d,} 2 } } */ Index: gcc/testsuite/gcc.target/aarch64/sve/vcond_15_run.c =================================================================== --- /dev/null 2018-04-20 16:19:46.369131350 +0100 +++ gcc/testsuite/gcc.target/aarch64/sve/vcond_15_run.c 2018-05-24 13:05:46.050603898 +0100 @@ -0,0 +1,37 @@ +/* { dg-do run { target aarch64_sve_hw } } */ +/* { dg-options "-O2 -ftree-vectorize" } */ + +#include "vcond_15.c" + +#define TEST_LOOP(INV, TYPE, CMPTYPE, SUFFIX) \ + { \ + TYPE a[N], b[N], c[N], d[N]; \ + CMPTYPE cond[N]; \ + for (int i = 0; i < N; ++i) \ + { \ + b[i] = i % 15; \ + c[i] = i % 9 + 11; \ + d[i] = i % 13 + 14; \ + cond[i] = i % 17; \ + asm volatile ("" ::: "memory"); \ + } \ + f_##INV##_##SUFFIX (a, b, c, d, cond); \ + for (int i = 0; i < N; ++i) \ + { \ + double mb = (INV & 1 ? -b[i] : b[i]); \ + double mc = c[i]; \ + double md = (INV & 2 ? -d[i] : d[i]); \ + double fma = __builtin_fma (mb, mc, md); \ + double truev = (INV & 4 ? -fma : fma); \ + if (a[i] != (i % 17 < 10 ? truev : d[i])) \ + __builtin_abort (); \ + asm volatile ("" ::: "memory"); \ + } \ + } + +int +main (void) +{ + FOR_EACH_INV (TEST_LOOP) + return 0; +} Index: gcc/testsuite/gcc.target/aarch64/sve/vcond_16.c =================================================================== --- /dev/null 2018-04-20 16:19:46.369131350 +0100 +++ gcc/testsuite/gcc.target/aarch64/sve/vcond_16.c 2018-05-24 13:05:46.050603898 +0100 @@ -0,0 +1,58 @@ +/* { dg-do compile } */ +/* { dg-options "-O2 -ftree-vectorize" } */ + +#define N 119 + +#define DEF_LOOP(INV, TYPE, CMPTYPE, SUFFIX) \ + void __attribute__ ((noipa)) \ + f_##INV##_##SUFFIX (TYPE *restrict a, TYPE *restrict b, \ + TYPE *restrict c, TYPE *restrict d, \ + CMPTYPE *restrict cond) \ + { \ + for (int i = 0; i < N; ++i) \ + { \ + TYPE mb = (INV & 1 ? -b[i] : b[i]); \ + TYPE mc = c[i]; \ + TYPE md = (INV & 2 ? -d[i] : d[i]); \ + TYPE fma = __builtin_fma##SUFFIX (mb, mc, md); \ + TYPE truev = (INV & 4 ? -fma : fma); \ + a[i] = cond[i] < 10 ? truev : 10; \ + } \ + } + +#define FOR_EACH_TYPE(T, INV) \ + T (INV, _Float16, short, f16) \ + T (INV, float, float, f32) \ + T (INV, double, double, f64) + +#define FOR_EACH_INV(T) \ + FOR_EACH_TYPE (T, 0) \ + FOR_EACH_TYPE (T, 1) \ + FOR_EACH_TYPE (T, 2) \ + FOR_EACH_TYPE (T, 3) \ + FOR_EACH_TYPE (T, 4) \ + FOR_EACH_TYPE (T, 5) \ + FOR_EACH_TYPE (T, 6) \ + FOR_EACH_TYPE (T, 7) + +FOR_EACH_INV (DEF_LOOP) + +/* { dg-final { scan-assembler-times {\tsel\t} 24 } } */ +/* { dg-final { scan-assembler-not {\tmovprfx\t} } } */ +/* { dg-final { scan-assembler-not {\tmov\tz[0-9]+\.., z[0-9]+} } } */ + +/* { dg-final { scan-assembler-times {\tfmad\tz[0-9]+\.h,} 2 } } */ +/* { dg-final { scan-assembler-times {\tfmad\tz[0-9]+\.s,} 2 } } */ +/* { dg-final { scan-assembler-times {\tfmad\tz[0-9]+\.d,} 2 } } */ + +/* { dg-final { scan-assembler-times {\tfmsb\tz[0-9]+\.h,} 2 } } */ +/* { dg-final { scan-assembler-times {\tfmsb\tz[0-9]+\.s,} 2 } } */ +/* { dg-final { scan-assembler-times {\tfmsb\tz[0-9]+\.d,} 2 } } */ + +/* { dg-final { scan-assembler-times {\tfnmad\tz[0-9]+\.h,} 2 } } */ +/* { dg-final { scan-assembler-times {\tfnmad\tz[0-9]+\.s,} 2 } } */ +/* { dg-final { scan-assembler-times {\tfnmad\tz[0-9]+\.d,} 2 } } */ + +/* { dg-final { scan-assembler-times {\tfnmsb\tz[0-9]+\.h,} 2 } } */ +/* { dg-final { scan-assembler-times {\tfnmsb\tz[0-9]+\.s,} 2 } } */ +/* { dg-final { scan-assembler-times {\tfnmsb\tz[0-9]+\.d,} 2 } } */ Index: gcc/testsuite/gcc.target/aarch64/sve/vcond_16_run.c =================================================================== --- /dev/null 2018-04-20 16:19:46.369131350 +0100 +++ gcc/testsuite/gcc.target/aarch64/sve/vcond_16_run.c 2018-05-24 13:05:46.050603898 +0100 @@ -0,0 +1,37 @@ +/* { dg-do run { target aarch64_sve_hw } } */ +/* { dg-options "-O2 -ftree-vectorize" } */ + +#include "vcond_16.c" + +#define TEST_LOOP(INV, TYPE, CMPTYPE, SUFFIX) \ + { \ + TYPE a[N], b[N], c[N], d[N]; \ + CMPTYPE cond[N]; \ + for (int i = 0; i < N; ++i) \ + { \ + b[i] = i % 15; \ + c[i] = i % 9 + 11; \ + d[i] = i % 13 + 14; \ + cond[i] = i % 17; \ + asm volatile ("" ::: "memory"); \ + } \ + f_##INV##_##SUFFIX (a, b, c, d, cond); \ + for (int i = 0; i < N; ++i) \ + { \ + double mb = (INV & 1 ? -b[i] : b[i]); \ + double mc = c[i]; \ + double md = (INV & 2 ? -d[i] : d[i]); \ + double fma = __builtin_fma (mb, mc, md); \ + double truev = (INV & 4 ? -fma : fma); \ + if (a[i] != (i % 17 < 10 ? truev : 10)) \ + __builtin_abort (); \ + asm volatile ("" ::: "memory"); \ + } \ + } + +int +main (void) +{ + FOR_EACH_INV (TEST_LOOP) + return 0; +}