Extend tree code folds to IFN_COND_*

Message ID 87vabdcs77.fsf@linaro.org
State New
Headers show
Series
  • Extend tree code folds to IFN_COND_*
Related show

Commit Message

Richard Sandiford May 24, 2018, 9:34 a.m.
This patch adds match.pd support for applying normal folds to their
IFN_COND_* forms.  E.g. the rule:

  (plus @0 (negate @1)) -> (minus @0 @1)

also allows the fold:

  (IFN_COND_ADD @0 @1 (negate @2) @3) -> (IFN_COND_SUB @0 @1 @2 @3)

Actually doing this by direct matches in gimple-match.c would
probably lead to combinatorial explosion, so instead, the patch
makes gimple_match_op carry a condition under which the operation
happens ("cond"), and the value to use when the condition is false
("else_value").  Thus in the example above we'd do the following

(a) convert:

      cond:NULL_TREE (IFN_COND_ADD @0 @1 @4 @3) else_value:NULL_TREE

    to:

      cond:@0 (plus @1 @4) else_value:@3

(b) apply gimple_resimplify to (plus @1 @4)

(c) reintroduce cond and else_value when constructing the result.

Nested operations inherit the condition of the outer operation
(so that we don't introduce extra faults) but have a null else_value.
If we try to build such an operation, the target gets to choose what
else_value it can handle efficiently: obvious choices include one of
the operands or a zero constant.  (The alternative would be to have some
representation for an undefined value, but that seems a bit invasive,
and isn't likely to be useful here.)

I've made the condition a mandatory part of the gimple_match_op
constructor so that it doesn't accidentally get dropped.

Tested on aarch64-linux-gnu (with and without SVE), aarch64_be-elf
and x86_64-linux-gnu.  OK to install?

Richard


2018-05-24  Richard Sandiford  <richard.sandiford@linaro.org>

gcc/
	* target.def (preferred_else_value): New target hook.
	* doc/tm.texi.in (TARGET_PREFERRED_ELSE_VALUE): New hook.
	* doc/tm.texi: Regenerate.
	* targhooks.h (default_preferred_else_value): Declare.
	* targhooks.c (default_preferred_else_value): New function.
	* internal-fn.h (conditional_internal_fn_code): Declare.
	* internal-fn.c (FOR_EACH_CODE_MAPPING): New macro.
	(get_conditional_internal_fn): Use it.
	(conditional_internal_fn_code): New function.
	* gimple-match.h (gimple_match_cond): New struct.
	(gimple_match_op): Add a cond member function.
	(gimple_match_op::gimple_match_op): Update all forms to take a
	gimple_match_cond.
	* genmatch.c (expr::gen_transform): Use the same condition as res_op
	for the suboperation, but don't specify a particular else_value.
	* tree-ssa-sccvn.c (vn_nary_simplify, vn_reference_lookup_3)
	(visit_nary_op, visit_reference_op_load): Pass
	gimple_match_cond::UNCOND to the gimple_match_op constructor.
	* gimple-match-head.c: Include tree-eh.h
	(convert_conditional_op): New function.
	(maybe_resimplify_conditional_op): Likewise.
	(gimple_resimplify1): Call maybe_resimplify_conditional_op.
	(gimple_resimplify2): Likewise.
	(gimple_resimplify3): Likewise.
	(gimple_resimplify4): Likewise.
	(maybe_push_res_to_seq): Return null for conditional operations.
	(try_conditional_simplification): New function.
	(gimple_simplify): Call it.  Pass conditions to the gimple_match_op
	constructor.
	* match.pd: Fold VEC_COND_EXPRs of an IFN_COND_* call to a new
	IFN_COND_* call.
	* config/aarch64/aarch64.c (aarch64_preferred_else_value): New
	function.
	(TARGET_PREFERRED_ELSE_VALUE): Redefine.

gcc/testsuite/
	* gcc.dg/vect/vect-cond-arith-2.c: New test.
	* gcc.target/aarch64/sve/loop_add_6.c: Likewise.

Comments

Richard Sandiford June 1, 2018, 12:54 p.m. | #1
Ping for the below, which is the only unreviewed IFN_COND_* patch.
Thanks for the reviews of the others.

Richard Sandiford <richard.sandiford@linaro.org> writes:
> This patch adds match.pd support for applying normal folds to their

> IFN_COND_* forms.  E.g. the rule:

>

>   (plus @0 (negate @1)) -> (minus @0 @1)

>

> also allows the fold:

>

>   (IFN_COND_ADD @0 @1 (negate @2) @3) -> (IFN_COND_SUB @0 @1 @2 @3)

>

> Actually doing this by direct matches in gimple-match.c would

> probably lead to combinatorial explosion, so instead, the patch

> makes gimple_match_op carry a condition under which the operation

> happens ("cond"), and the value to use when the condition is false

> ("else_value").  Thus in the example above we'd do the following

>

> (a) convert:

>

>       cond:NULL_TREE (IFN_COND_ADD @0 @1 @4 @3) else_value:NULL_TREE

>

>     to:

>

>       cond:@0 (plus @1 @4) else_value:@3

>

> (b) apply gimple_resimplify to (plus @1 @4)

>

> (c) reintroduce cond and else_value when constructing the result.

>

> Nested operations inherit the condition of the outer operation

> (so that we don't introduce extra faults) but have a null else_value.

> If we try to build such an operation, the target gets to choose what

> else_value it can handle efficiently: obvious choices include one of

> the operands or a zero constant.  (The alternative would be to have some

> representation for an undefined value, but that seems a bit invasive,

> and isn't likely to be useful here.)

>

> I've made the condition a mandatory part of the gimple_match_op

> constructor so that it doesn't accidentally get dropped.

>

> Tested on aarch64-linux-gnu (with and without SVE), aarch64_be-elf

> and x86_64-linux-gnu.  OK to install?

>

> Richard

>

>

> 2018-05-24  Richard Sandiford  <richard.sandiford@linaro.org>

>

> gcc/

> 	* target.def (preferred_else_value): New target hook.

> 	* doc/tm.texi.in (TARGET_PREFERRED_ELSE_VALUE): New hook.

> 	* doc/tm.texi: Regenerate.

> 	* targhooks.h (default_preferred_else_value): Declare.

> 	* targhooks.c (default_preferred_else_value): New function.

> 	* internal-fn.h (conditional_internal_fn_code): Declare.

> 	* internal-fn.c (FOR_EACH_CODE_MAPPING): New macro.

> 	(get_conditional_internal_fn): Use it.

> 	(conditional_internal_fn_code): New function.

> 	* gimple-match.h (gimple_match_cond): New struct.

> 	(gimple_match_op): Add a cond member function.

> 	(gimple_match_op::gimple_match_op): Update all forms to take a

> 	gimple_match_cond.

> 	* genmatch.c (expr::gen_transform): Use the same condition as res_op

> 	for the suboperation, but don't specify a particular else_value.

> 	* tree-ssa-sccvn.c (vn_nary_simplify, vn_reference_lookup_3)

> 	(visit_nary_op, visit_reference_op_load): Pass

> 	gimple_match_cond::UNCOND to the gimple_match_op constructor.

> 	* gimple-match-head.c: Include tree-eh.h

> 	(convert_conditional_op): New function.

> 	(maybe_resimplify_conditional_op): Likewise.

> 	(gimple_resimplify1): Call maybe_resimplify_conditional_op.

> 	(gimple_resimplify2): Likewise.

> 	(gimple_resimplify3): Likewise.

> 	(gimple_resimplify4): Likewise.

> 	(maybe_push_res_to_seq): Return null for conditional operations.

> 	(try_conditional_simplification): New function.

> 	(gimple_simplify): Call it.  Pass conditions to the gimple_match_op

> 	constructor.

> 	* match.pd: Fold VEC_COND_EXPRs of an IFN_COND_* call to a new

> 	IFN_COND_* call.

> 	* config/aarch64/aarch64.c (aarch64_preferred_else_value): New

> 	function.

> 	(TARGET_PREFERRED_ELSE_VALUE): Redefine.

>

> gcc/testsuite/

> 	* gcc.dg/vect/vect-cond-arith-2.c: New test.

> 	* gcc.target/aarch64/sve/loop_add_6.c: Likewise.

>

> Index: gcc/target.def

> ===================================================================

> --- gcc/target.def	2018-05-01 19:30:30.159632586 +0100

> +++ gcc/target.def	2018-05-24 10:33:30.871095132 +0100

> @@ -2040,6 +2040,25 @@ HOOK_VECTOR_END (vectorize)

>  #define HOOK_PREFIX "TARGET_"

>  

>  DEFHOOK

> +(preferred_else_value,

> + "This hook returns the target's preferred final argument for a call\n\

> +to conditional internal function @var{ifn} (really of type\n\

> +@code{internal_fn}).  @var{type} specifies the return type of the\n\

> +function and @var{ops} are the operands to the conditional operation,\n\

> +of which there are @var{nops}.\n\

> +\n\

> +For example, if @var{ifn} is @code{IFN_COND_ADD}, the hook returns\n\

> +a value of type @var{type} that should be used when @samp{@var{ops}[0]}\n\

> +and @samp{@var{ops}[1]} are conditionally added together.\n\

> +\n\

> +This hook is only relevant if the target supports conditional patterns\n\

> +like @code{cond_add@var{m}}.  The default implementation returns a zero\n\

> +constant of type @var{type}.",

> + tree,

> + (unsigned ifn, tree type, unsigned nops, tree *ops),

> + default_preferred_else_value)

> +

> +DEFHOOK

>  (record_offload_symbol,

>   "Used when offloaded functions are seen in the compilation unit and no named\n\

>  sections are available.  It is called once for each symbol that must be\n\

> Index: gcc/doc/tm.texi.in

> ===================================================================

> --- gcc/doc/tm.texi.in	2018-05-01 19:30:28.730694873 +0100

> +++ gcc/doc/tm.texi.in	2018-05-24 10:33:30.869095197 +0100

> @@ -4149,6 +4149,8 @@ address;  but often a machine-dependent

>  

>  @hook TARGET_GOACC_REDUCTION

>  

> +@hook TARGET_PREFERRED_ELSE_VALUE

> +

>  @node Anchored Addresses

>  @section Anchored Addresses

>  @cindex anchored addresses

> Index: gcc/doc/tm.texi

> ===================================================================

> --- gcc/doc/tm.texi	2018-05-01 19:30:28.722695224 +0100

> +++ gcc/doc/tm.texi	2018-05-24 10:33:30.868095229 +0100

> @@ -6046,6 +6046,22 @@ expanded sequence has been inserted.  Th

>  for allocating any storage for reductions when necessary.

>  @end deftypefn

>  

> +@deftypefn {Target Hook} tree TARGET_PREFERRED_ELSE_VALUE (unsigned @var{ifn}, tree @var{type}, unsigned @var{nops}, tree *@var{ops})

> +This hook returns the target's preferred final argument for a call

> +to conditional internal function @var{ifn} (really of type

> +@code{internal_fn}).  @var{type} specifies the return type of the

> +function and @var{ops} are the operands to the conditional operation,

> +of which there are @var{nops}.

> +

> +For example, if @var{ifn} is @code{IFN_COND_ADD}, the hook returns

> +a value of type @var{type} that should be used when @samp{@var{ops}[0]}

> +and @samp{@var{ops}[1]} are conditionally added together.

> +

> +This hook is only relevant if the target supports conditional patterns

> +like @code{cond_add@var{m}}.  The default implementation returns a zero

> +constant of type @var{type}.

> +@end deftypefn

> +

>  @node Anchored Addresses

>  @section Anchored Addresses

>  @cindex anchored addresses

> Index: gcc/targhooks.h

> ===================================================================

> --- gcc/targhooks.h	2018-05-01 19:30:29.390666052 +0100

> +++ gcc/targhooks.h	2018-05-24 10:33:30.872095099 +0100

> @@ -289,5 +289,6 @@ extern unsigned int default_min_arithmet

>  default_excess_precision (enum excess_precision_type ATTRIBUTE_UNUSED);

>  extern bool default_stack_clash_protection_final_dynamic_probe (rtx);

>  extern void default_select_early_remat_modes (sbitmap);

> +extern tree default_preferred_else_value (unsigned, tree, unsigned, tree *);

>  

>  #endif /* GCC_TARGHOOKS_H */

> Index: gcc/targhooks.c

> ===================================================================

> --- gcc/targhooks.c	2018-05-01 19:30:29.390666052 +0100

> +++ gcc/targhooks.c	2018-05-24 10:33:30.871095132 +0100

> @@ -2345,4 +2345,12 @@ default_select_early_remat_modes (sbitma

>  {

>  }

>  

> +/* The default implementation of TARGET_PREFERRED_ELSE_VALUE.  */

> +

> +tree

> +default_preferred_else_value (unsigned, tree type, unsigned, tree *)

> +{

> +  return build_zero_cst (type);

> +}

> +

>  #include "gt-targhooks.h"

> Index: gcc/internal-fn.h

> ===================================================================

> --- gcc/internal-fn.h	2018-05-17 11:52:13.507173989 +0100

> +++ gcc/internal-fn.h	2018-05-24 10:33:30.870095164 +0100

> @@ -193,6 +193,7 @@ direct_internal_fn_supported_p (internal

>  extern bool set_edom_supported_p (void);

>  

>  extern internal_fn get_conditional_internal_fn (tree_code);

> +extern tree_code conditional_internal_fn_code (internal_fn);

>  

>  extern bool internal_load_fn_p (internal_fn);

>  extern bool internal_store_fn_p (internal_fn);

> Index: gcc/internal-fn.c

> ===================================================================

> --- gcc/internal-fn.c	2018-05-24 10:12:10.146352152 +0100

> +++ gcc/internal-fn.c	2018-05-24 10:33:30.870095164 +0100

> @@ -3219,6 +3219,21 @@ #define DEF_INTERNAL_FN(CODE, FLAGS, FNS

>    0

>  };

>  

> +/* Invoke T(CODE, IFN) for each conditional function IFN that maps to a

> +   tree code CODE.  */

> +#define FOR_EACH_CODE_MAPPING(T) \

> +  T (PLUS_EXPR, IFN_COND_ADD) \

> +  T (MINUS_EXPR, IFN_COND_SUB) \

> +  T (MULT_EXPR, IFN_COND_MUL) \

> +  T (TRUNC_DIV_EXPR, IFN_COND_DIV) \

> +  T (TRUNC_MOD_EXPR, IFN_COND_MOD) \

> +  T (RDIV_EXPR, IFN_COND_RDIV) \

> +  T (MIN_EXPR, IFN_COND_MIN) \

> +  T (MAX_EXPR, IFN_COND_MAX) \

> +  T (BIT_AND_EXPR, IFN_COND_AND) \

> +  T (BIT_IOR_EXPR, IFN_COND_IOR) \

> +  T (BIT_XOR_EXPR, IFN_COND_XOR)

> +

>  /* Return a function that only performs CODE when a certain condition is met

>     and that uses a given fallback value otherwise.  For example, if CODE is

>     a binary operation associated with conditional function FN:

> @@ -3238,31 +3253,30 @@ get_conditional_internal_fn (tree_code c

>  {

>    switch (code)

>      {

> -    case PLUS_EXPR:

> -      return IFN_COND_ADD;

> -    case MINUS_EXPR:

> -      return IFN_COND_SUB;

> -    case MIN_EXPR:

> -      return IFN_COND_MIN;

> -    case MAX_EXPR:

> -      return IFN_COND_MAX;

> -    case TRUNC_DIV_EXPR:

> -      return IFN_COND_DIV;

> -    case TRUNC_MOD_EXPR:

> -      return IFN_COND_MOD;

> -    case RDIV_EXPR:

> -      return IFN_COND_RDIV;

> -    case BIT_AND_EXPR:

> -      return IFN_COND_AND;

> -    case BIT_IOR_EXPR:

> -      return IFN_COND_IOR;

> -    case BIT_XOR_EXPR:

> -      return IFN_COND_XOR;

> +#define CASE(CODE, IFN) case CODE: return IFN;

> +      FOR_EACH_CODE_MAPPING(CASE)

> +#undef CASE

>      default:

>        return IFN_LAST;

>      }

>  }

>  

> +/* If IFN implements the conditional form of a tree code, return that

> +   tree code, otherwise return ERROR_MARK.  */

> +

> +tree_code

> +conditional_internal_fn_code (internal_fn ifn)

> +{

> +  switch (ifn)

> +    {

> +#define CASE(CODE, IFN) case IFN: return CODE;

> +      FOR_EACH_CODE_MAPPING(CASE)

> +#undef CASE

> +    default:

> +      return ERROR_MARK;

> +    }

> +}

> +

>  /* Return true if IFN is some form of load from memory.  */

>  

>  bool

> Index: gcc/gimple-match.h

> ===================================================================

> --- gcc/gimple-match.h	2018-05-24 09:54:37.509451356 +0100

> +++ gcc/gimple-match.h	2018-05-24 10:33:30.870095164 +0100

> @@ -40,16 +40,57 @@ #define GCC_GIMPLE_MATCH_H

>    int rep;

>  };

>  

> +/* Represents the condition under which an operation should happen,

> +   and the value to use otherwise.  The condition applies elementwise

> +   (as for VEC_COND_EXPR) if the values are vectors.  */

> +struct gimple_match_cond

> +{

> +  enum uncond { UNCOND };

> +

> +  /* Build an unconditional op.  */

> +  gimple_match_cond (uncond) : cond (NULL_TREE), else_value (NULL_TREE) {}

> +  gimple_match_cond (tree, tree);

> +

> +  gimple_match_cond any_else () const;

> +

> +  /* The condition under which the operation occurs, or NULL_TREE

> +     if the operation is unconditional.  */

> +  tree cond;

> +

> +  /* The value to use when the condition is false.  This is NULL_TREE if

> +     the operation is unconditional or if the value doesn't matter.  */

> +  tree else_value;

> +};

> +

> +inline

> +gimple_match_cond::gimple_match_cond (tree cond_in, tree else_value_in)

> +  : cond (cond_in), else_value (else_value_in)

> +{

> +}

> +

> +/* Return a gimple_match_cond with the same condition but with an

> +   arbitrary ELSE_VALUE.  */

> +

> +inline gimple_match_cond

> +gimple_match_cond::any_else () const

> +{

> +  return gimple_match_cond (cond, NULL_TREE);

> +}

> +

>  /* Represents an operation to be simplified, or the result of the

>     simplification.  */

>  struct gimple_match_op

>  {

> -  gimple_match_op () : type (NULL_TREE), num_ops (0) {}

> -  gimple_match_op (code_helper, tree, unsigned int);

> -  gimple_match_op (code_helper, tree, tree);

> -  gimple_match_op (code_helper, tree, tree, tree);

> -  gimple_match_op (code_helper, tree, tree, tree, tree);

> -  gimple_match_op (code_helper, tree, tree, tree, tree, tree);

> +  gimple_match_op ();

> +  gimple_match_op (const gimple_match_cond &, code_helper, tree, unsigned int);

> +  gimple_match_op (const gimple_match_cond &,

> +		   code_helper, tree, tree);

> +  gimple_match_op (const gimple_match_cond &,

> +		   code_helper, tree, tree, tree);

> +  gimple_match_op (const gimple_match_cond &,

> +		   code_helper, tree, tree, tree, tree);

> +  gimple_match_op (const gimple_match_cond &,

> +		   code_helper, tree, tree, tree, tree, tree);

>  

>    void set_op (code_helper, tree, unsigned int);

>    void set_op (code_helper, tree, tree);

> @@ -63,6 +104,10 @@ struct gimple_match_op

>    /* The maximum value of NUM_OPS.  */

>    static const unsigned int MAX_NUM_OPS = 4;

>  

> +  /* The conditions under which the operation is performed, and the value to

> +     use as a fallback.  */

> +  gimple_match_cond cond;

> +

>    /* The operation being performed.  */

>    code_helper code;

>  

> @@ -76,39 +121,49 @@ struct gimple_match_op

>    tree ops[MAX_NUM_OPS];

>  };

>  

> -/* Constructor that takes the code, type and number of operands, but leaves

> -   the caller to fill in the operands.  */

> +inline

> +gimple_match_op::gimple_match_op ()

> +  : cond (gimple_match_cond::UNCOND), type (NULL_TREE), num_ops (0)

> +{

> +}

> +

> +/* Constructor that takes the condition, code, type and number of

> +   operands, but leaves the caller to fill in the operands.  */

>  

>  inline

> -gimple_match_op::gimple_match_op (code_helper code_in, tree type_in,

> +gimple_match_op::gimple_match_op (const gimple_match_cond &cond_in,

> +				  code_helper code_in, tree type_in,

>  				  unsigned int num_ops_in)

> -  : code (code_in), type (type_in), num_ops (num_ops_in)

> +  : cond (cond_in), code (code_in), type (type_in), num_ops (num_ops_in)

>  {

>  }

>  

>  /* Constructors for various numbers of operands.  */

>  

>  inline

> -gimple_match_op::gimple_match_op (code_helper code_in, tree type_in,

> +gimple_match_op::gimple_match_op (const gimple_match_cond &cond_in,

> +				  code_helper code_in, tree type_in,

>  				  tree op0)

> -  : code (code_in), type (type_in), num_ops (1)

> +  : cond (cond_in), code (code_in), type (type_in), num_ops (1)

>  {

>    ops[0] = op0;

>  }

>  

>  inline

> -gimple_match_op::gimple_match_op (code_helper code_in, tree type_in,

> +gimple_match_op::gimple_match_op (const gimple_match_cond &cond_in,

> +				  code_helper code_in, tree type_in,

>  				  tree op0, tree op1)

> -  : code (code_in), type (type_in), num_ops (2)

> +  : cond (cond_in), code (code_in), type (type_in), num_ops (2)

>  {

>    ops[0] = op0;

>    ops[1] = op1;

>  }

>  

>  inline

> -gimple_match_op::gimple_match_op (code_helper code_in, tree type_in,

> +gimple_match_op::gimple_match_op (const gimple_match_cond &cond_in,

> +				  code_helper code_in, tree type_in,

>  				  tree op0, tree op1, tree op2)

> -  : code (code_in), type (type_in), num_ops (3)

> +  : cond (cond_in), code (code_in), type (type_in), num_ops (3)

>  {

>    ops[0] = op0;

>    ops[1] = op1;

> @@ -116,9 +171,10 @@ gimple_match_op::gimple_match_op (code_h

>  }

>  

>  inline

> -gimple_match_op::gimple_match_op (code_helper code_in, tree type_in,

> +gimple_match_op::gimple_match_op (const gimple_match_cond &cond_in,

> +				  code_helper code_in, tree type_in,

>  				  tree op0, tree op1, tree op2, tree op3)

> -  : code (code_in), type (type_in), num_ops (4)

> +  : cond (cond_in), code (code_in), type (type_in), num_ops (4)

>  {

>    ops[0] = op0;

>    ops[1] = op1;

> Index: gcc/genmatch.c

> ===================================================================

> --- gcc/genmatch.c	2018-05-24 10:12:10.145352193 +0100

> +++ gcc/genmatch.c	2018-05-24 10:33:30.869095197 +0100

> @@ -2507,8 +2507,8 @@ expr::gen_transform (FILE *f, int indent

>        /* ???  Building a stmt can fail for various reasons here, seq being

>           NULL or the stmt referencing SSA names occuring in abnormal PHIs.

>  	 So if we fail here we should continue matching other patterns.  */

> -      fprintf_indent (f, indent, "gimple_match_op tem_op (%s, %s",

> -		      opr_name, type);

> +      fprintf_indent (f, indent, "gimple_match_op tem_op "

> +		      "(res_op->cond.any_else (), %s, %s", opr_name, type);

>        for (unsigned i = 0; i < ops.length (); ++i)

>  	fprintf (f, ", ops%d[%u]", depth, i);

>        fprintf (f, ");\n");

> Index: gcc/tree-ssa-sccvn.c

> ===================================================================

> --- gcc/tree-ssa-sccvn.c	2018-05-24 09:02:28.765328358 +0100

> +++ gcc/tree-ssa-sccvn.c	2018-05-24 10:33:30.872095099 +0100

> @@ -1804,7 +1804,8 @@ vn_nary_simplify (vn_nary_op_t nary)

>  {

>    if (nary->length > gimple_match_op::MAX_NUM_OPS)

>      return NULL_TREE;

> -  gimple_match_op op (nary->opcode, nary->type, nary->length);

> +  gimple_match_op op (gimple_match_cond::UNCOND, nary->opcode,

> +		      nary->type, nary->length);

>    memcpy (op.ops, nary->op, sizeof (tree) * nary->length);

>    return vn_nary_build_or_lookup_1 (&op, false);

>  }

> @@ -2031,8 +2032,8 @@ vn_reference_lookup_3 (ao_ref *ref, tree

>  	  else if (INTEGRAL_TYPE_P (vr->type)

>  		   && known_eq (ref->size, 8))

>  	    {

> -	      gimple_match_op res_op (NOP_EXPR, vr->type,

> -				      gimple_call_arg (def_stmt, 1));

> +	      gimple_match_op res_op (gimple_match_cond::UNCOND, NOP_EXPR,

> +				      vr->type, gimple_call_arg (def_stmt, 1));

>  	      val = vn_nary_build_or_lookup (&res_op);

>  	      if (!val

>  		  || (TREE_CODE (val) == SSA_NAME

> @@ -2172,7 +2173,8 @@ vn_reference_lookup_3 (ao_ref *ref, tree

>  	      || known_eq (ref->size, TYPE_PRECISION (vr->type)))

>  	  && multiple_p (ref->size, BITS_PER_UNIT))

>  	{

> -	  gimple_match_op op (BIT_FIELD_REF, vr->type,

> +	  gimple_match_op op (gimple_match_cond::UNCOND,

> +			      BIT_FIELD_REF, vr->type,

>  			      SSA_VAL (gimple_assign_rhs1 (def_stmt)),

>  			      bitsize_int (ref->size),

>  			      bitsize_int (offset - offset2));

> @@ -3701,7 +3703,8 @@ visit_nary_op (tree lhs, gassign *stmt)

>  		      unsigned rhs_prec = TYPE_PRECISION (TREE_TYPE (rhs1));

>  		      if (lhs_prec == rhs_prec)

>  			{

> -			  gimple_match_op match_op (NOP_EXPR, type, ops[0]);

> +			  gimple_match_op match_op (gimple_match_cond::UNCOND,

> +						    NOP_EXPR, type, ops[0]);

>  			  result = vn_nary_build_or_lookup (&match_op);

>  			  if (result)

>  			    {

> @@ -3714,7 +3717,8 @@ visit_nary_op (tree lhs, gassign *stmt)

>  			{

>  			  tree mask = wide_int_to_tree

>  			    (type, wi::mask (rhs_prec, false, lhs_prec));

> -			  gimple_match_op match_op (BIT_AND_EXPR,

> +			  gimple_match_op match_op (gimple_match_cond::UNCOND,

> +						    BIT_AND_EXPR,

>  						    TREE_TYPE (lhs),

>  						    ops[0], mask);

>  			  result = vn_nary_build_or_lookup (&match_op);

> @@ -3838,7 +3842,8 @@ visit_reference_op_load (tree lhs, tree

>  	 of VIEW_CONVERT_EXPR <TREE_TYPE (result)> (result).

>  	 So first simplify and lookup this expression to see if it

>  	 is already available.  */

> -      gimple_match_op res_op (VIEW_CONVERT_EXPR, TREE_TYPE (op), result);

> +      gimple_match_op res_op (gimple_match_cond::UNCOND,

> +			      VIEW_CONVERT_EXPR, TREE_TYPE (op), result);

>        result = vn_nary_build_or_lookup (&res_op);

>      }

>  

> Index: gcc/gimple-match-head.c

> ===================================================================

> --- gcc/gimple-match-head.c	2018-05-24 09:54:37.509451356 +0100

> +++ gcc/gimple-match-head.c	2018-05-24 10:33:30.870095164 +0100

> @@ -40,6 +40,7 @@ Software Foundation; either version 3, o

>  #include "case-cfn-macros.h"

>  #include "gimplify.h"

>  #include "optabs-tree.h"

> +#include "tree-eh.h"

>  

>  

>  /* Forward declarations of the private auto-generated matchers.

> @@ -68,6 +69,95 @@ constant_for_folding (tree t)

>  	      && TREE_CODE (TREE_OPERAND (t, 0)) == STRING_CST));

>  }

>  

> +/* Try to convert conditional operation ORIG_OP into an IFN_COND_*

> +   operation.  Return true on success, storing the new operation in NEW_OP.  */

> +

> +static bool

> +convert_conditional_op (gimple_match_op *orig_op,

> +			gimple_match_op *new_op)

> +{

> +  internal_fn ifn;

> +  if (orig_op->code.is_tree_code ())

> +    ifn = get_conditional_internal_fn ((tree_code) orig_op->code);

> +  else

> +    return false;

> +  if (ifn == IFN_LAST)

> +    return false;

> +  unsigned int num_ops = orig_op->num_ops;

> +  new_op->set_op (as_combined_fn (ifn), orig_op->type, num_ops + 2);

> +  new_op->ops[0] = orig_op->cond.cond;

> +  for (unsigned int i = 0; i < num_ops; ++i)

> +    new_op->ops[i + 1] = orig_op->ops[i];

> +  tree else_value = orig_op->cond.else_value;

> +  if (!else_value)

> +    else_value = targetm.preferred_else_value (ifn, orig_op->type,

> +					       num_ops, orig_op->ops);

> +  new_op->ops[num_ops + 1] = else_value;

> +  return true;

> +}

> +

> +/* RES_OP is the result of a simplification.  If it is conditional,

> +   try to replace it with the equivalent UNCOND form, such as an

> +   IFN_COND_* call or a VEC_COND_EXPR.  Also try to resimplify the

> +   result of the replacement if appropriate, adding any new statements to

> +   SEQ and using VALUEIZE as the valueization function.  Return true if

> +   this resimplification occurred and resulted in at least one change.  */

> +

> +static bool

> +maybe_resimplify_conditional_op (gimple_seq *seq, gimple_match_op *res_op,

> +				 tree (*valueize) (tree))

> +{

> +  if (!res_op->cond.cond)

> +    return false;

> +

> +  if (!res_op->cond.else_value

> +      && res_op->code.is_tree_code ())

> +    {

> +      /* The "else" value doesn't matter.  If the "then" value is a

> +	 gimple value, just use it unconditionally.  This isn't a

> +	 simplification in itself, since there was no operation to

> +	 build in the first place.  */

> +      if (gimple_simplified_result_is_gimple_val (res_op))

> +	{

> +	  res_op->cond.cond = NULL_TREE;

> +	  return false;

> +	}

> +

> +      /* Likewise if the operation would not trap.  */

> +      bool honor_trapv = (INTEGRAL_TYPE_P (res_op->type)

> +			  && TYPE_OVERFLOW_TRAPS (res_op->type));

> +      if (!operation_could_trap_p ((tree_code) res_op->code,

> +				   FLOAT_TYPE_P (res_op->type),

> +				   honor_trapv, res_op->op_or_null (1)))

> +	{

> +	  res_op->cond.cond = NULL_TREE;

> +	  return false;

> +	}

> +    }

> +

> +  /* If the "then" value is a gimple value and the "else" value matters,

> +     create a VEC_COND_EXPR between them, then see if it can be further

> +     simplified.  */

> +  gimple_match_op new_op;

> +  if (res_op->cond.else_value

> +      && VECTOR_TYPE_P (res_op->type)

> +      && gimple_simplified_result_is_gimple_val (res_op))

> +    {

> +      new_op.set_op (VEC_COND_EXPR, res_op->type,

> +		     res_op->cond.cond, res_op->ops[0],

> +		     res_op->cond.else_value);

> +      *res_op = new_op;

> +      return gimple_resimplify3 (seq, res_op, valueize);

> +    }

> +

> +  /* Otherwise try rewriting the operation as an IFN_COND_* call.

> +     Again, this isn't a simplification in itself, since it's what

> +     RES_OP already described.  */

> +  if (convert_conditional_op (res_op, &new_op))

> +    *res_op = new_op;

> +

> +  return false;

> +}

>  

>  /* Helper that matches and simplifies the toplevel result from

>     a gimple_simplify run (where we don't want to build

> @@ -93,6 +183,7 @@ gimple_resimplify1 (gimple_seq *seq, gim

>  	  if (TREE_OVERFLOW_P (tem))

>  	    tem = drop_tree_overflow (tem);

>  	  res_op->set_value (tem);

> +	  maybe_resimplify_conditional_op (seq, res_op, valueize);

>  	  return true;

>  	}

>      }

> @@ -105,6 +196,9 @@ gimple_resimplify1 (gimple_seq *seq, gim

>        return true;

>      }

>  

> +  if (maybe_resimplify_conditional_op (seq, res_op, valueize))

> +    return true;

> +

>    return false;

>  }

>  

> @@ -134,6 +228,7 @@ gimple_resimplify2 (gimple_seq *seq, gim

>  	  if (TREE_OVERFLOW_P (tem))

>  	    tem = drop_tree_overflow (tem);

>  	  res_op->set_value (tem);

> +	  maybe_resimplify_conditional_op (seq, res_op, valueize);

>  	  return true;

>  	}

>      }

> @@ -160,6 +255,9 @@ gimple_resimplify2 (gimple_seq *seq, gim

>        return true;

>      }

>  

> +  if (maybe_resimplify_conditional_op (seq, res_op, valueize))

> +    return true;

> +

>    return canonicalized;

>  }

>  

> @@ -191,6 +289,7 @@ gimple_resimplify3 (gimple_seq *seq, gim

>  	  if (TREE_OVERFLOW_P (tem))

>  	    tem = drop_tree_overflow (tem);

>  	  res_op->set_value (tem);

> +	  maybe_resimplify_conditional_op (seq, res_op, valueize);

>  	  return true;

>  	}

>      }

> @@ -214,6 +313,9 @@ gimple_resimplify3 (gimple_seq *seq, gim

>        return true;

>      }

>  

> +  if (maybe_resimplify_conditional_op (seq, res_op, valueize))

> +    return true;

> +

>    return canonicalized;

>  }

>  

> @@ -239,6 +341,9 @@ gimple_resimplify4 (gimple_seq *seq, gim

>        return true;

>      }

>  

> +  if (maybe_resimplify_conditional_op (seq, res_op, valueize))

> +    return true;

> +

>    return false;

>  }

>  

> @@ -297,6 +402,12 @@ maybe_push_res_to_seq (gimple_match_op *

>    tree *ops = res_op->ops;

>    unsigned num_ops = res_op->num_ops;

>  

> +  /* The caller should have converted conditional operations into an UNCOND

> +     form and resimplified as appropriate.  The conditional form only

> +     survives this far if that conversion failed.  */

> +  if (res_op->cond.cond)

> +    return NULL_TREE;

> +

>    if (res_op->code.is_tree_code ())

>      {

>        if (!res

> @@ -558,6 +669,50 @@ do_valueize (tree op, tree (*valueize)(t

>    return op;

>  }

>  

> +/* If RES_OP is a call to a conditional internal function, try simplifying

> +   the associated unconditional operation and using the result to build

> +   a new conditional operation.  For example, if RES_OP is:

> +

> +     IFN_COND_ADD (COND, A, B, ELSE)

> +

> +   try simplifying (plus A B) and using the result to build a replacement

> +   for the whole IFN_COND_ADD.

> +

> +   Return true if this approach led to a simplification, otherwise leave

> +   RES_OP unchanged (and so suitable for other simplifications).  When

> +   returning true, add any new statements to SEQ and use VALUEIZE as the

> +   valueization function.

> +

> +   RES_OP is known to be a call to IFN.  */

> +

> +static bool

> +try_conditional_simplification (internal_fn ifn, gimple_match_op *res_op,

> +				gimple_seq *seq, tree (*valueize) (tree))

> +{

> +  tree_code code = conditional_internal_fn_code (ifn);

> +  if (code == ERROR_MARK)

> +    return false;

> +

> +  unsigned int num_ops = res_op->num_ops;

> +  gimple_match_op cond_op (gimple_match_cond (res_op->ops[0],

> +					      res_op->ops[num_ops - 1]),

> +			   code, res_op->type, num_ops - 2);

> +  for (unsigned int i = 1; i < num_ops - 1; ++i)

> +    cond_op.ops[i - 1] = res_op->ops[i];

> +  switch (num_ops - 2)

> +    {

> +    case 2:

> +      if (!gimple_resimplify2 (seq, &cond_op, valueize))

> +	return false;

> +      break;

> +    default:

> +      gcc_unreachable ();

> +    }

> +  *res_op = cond_op;

> +  maybe_resimplify_conditional_op (seq, res_op, valueize);

> +  return true;

> +}

> +

>  /* The main STMT based simplification entry.  It is used by the fold_stmt

>     and the fold_stmt_to_constant APIs.  */

>  

> @@ -643,7 +798,7 @@ gimple_simplify (gimple *stmt, gimple_ma

>  		      tree rhs = TREE_OPERAND (rhs1, 1);

>  		      lhs = do_valueize (lhs, top_valueize, valueized);

>  		      rhs = do_valueize (rhs, top_valueize, valueized);

> -		      gimple_match_op res_op2 (TREE_CODE (rhs1),

> +		      gimple_match_op res_op2 (res_op->cond, TREE_CODE (rhs1),

>  					       TREE_TYPE (rhs1), lhs, rhs);

>  		      if ((gimple_resimplify2 (seq, &res_op2, valueize)

>  			   || valueized)

> @@ -714,6 +869,10 @@ gimple_simplify (gimple *stmt, gimple_ma

>  	      tree arg = gimple_call_arg (stmt, i);

>  	      res_op->ops[i] = do_valueize (arg, top_valueize, valueized);

>  	    }

> +	  if (internal_fn_p (cfn)

> +	      && try_conditional_simplification (as_internal_fn (cfn),

> +						 res_op, seq, valueize))

> +	    return true;

>  	  switch (num_args)

>  	    {

>  	    case 1:

> Index: gcc/match.pd

> ===================================================================

> --- gcc/match.pd	2018-05-24 10:12:10.146352152 +0100

> +++ gcc/match.pd	2018-05-24 10:33:30.870095164 +0100

> @@ -4797,3 +4797,12 @@ DEFINE_INT_AND_FLOAT_ROUND_FN (RINT)

>    (with { tree op_type = TREE_TYPE (@4); }

>     (if (element_precision (type) == element_precision (op_type))

>      (view_convert (cond_op (bit_not @0) @2 @3 (view_convert:op_type @1)))))))

> +

> +/* Detect cases in which a VEC_COND_EXPR effectively replaces the

> +   "else" value of an IFN_COND_*.  */

> +(for cond_op (COND_BINARY)

> + (simplify

> +  (vec_cond @0 (view_convert? (cond_op @0 @1 @2 @3)) @4)

> +  (with { tree op_type = TREE_TYPE (@3); }

> +   (if (element_precision (type) == element_precision (op_type))

> +    (view_convert (cond_op @0 @1 @2 (view_convert:op_type @4)))))))

> Index: gcc/config/aarch64/aarch64.c

> ===================================================================

> --- gcc/config/aarch64/aarch64.c	2018-05-24 09:54:37.507451418 +0100

> +++ gcc/config/aarch64/aarch64.c	2018-05-24 10:33:30.867095262 +0100

> @@ -1292,6 +1292,16 @@ aarch64_get_mask_mode (poly_uint64 nunit

>    return default_get_mask_mode (nunits, nbytes);

>  }

>  

> +/* Implement TARGET_PREFERRED_ELSE_VALUE.  Prefer to use the first

> +   arithmetic operand as the else value if the else value doesn't matter,

> +   since that exactly matches the SVE destructive merging form.  */

> +

> +static tree

> +aarch64_preferred_else_value (unsigned, tree, unsigned int, tree *ops)

> +{

> +  return ops[0];

> +}

> +

>  /* Implement TARGET_HARD_REGNO_NREGS.  */

>  

>  static unsigned int

> @@ -17980,6 +17990,9 @@ #define TARGET_VECTORIZE_GET_MASK_MODE a

>  #undef TARGET_VECTORIZE_EMPTY_MASK_IS_EXPENSIVE

>  #define TARGET_VECTORIZE_EMPTY_MASK_IS_EXPENSIVE \

>    aarch64_empty_mask_is_expensive

> +#undef TARGET_PREFERRED_ELSE_VALUE

> +#define TARGET_PREFERRED_ELSE_VALUE \

> +  aarch64_preferred_else_value

>  

>  #undef TARGET_INIT_LIBFUNCS

>  #define TARGET_INIT_LIBFUNCS aarch64_init_libfuncs

> Index: gcc/testsuite/gcc.dg/vect/vect-cond-arith-2.c

> ===================================================================

> --- /dev/null	2018-04-20 16:19:46.369131350 +0100

> +++ gcc/testsuite/gcc.dg/vect/vect-cond-arith-2.c	2018-05-24 10:33:30.872095099 +0100

> @@ -0,0 +1,45 @@

> +/* { dg-do compile } */

> +/* { dg-additional-options "-fgimple -fdump-tree-optimized -ffast-math" } */

> +

> +double __GIMPLE (startwith("loop"))

> +neg_xi (double *x)

> +{

> +  int i;

> +  long unsigned int index;

> +  long unsigned int offset;

> +  double * xi_ptr;

> +  double xi;

> +  double neg_xi;

> +  double res;

> +  unsigned int ivtmp;

> +

> + bb_1:

> +  goto bb_2;

> +

> + bb_2:

> +  res_1 = __PHI (bb_1: 0.0, bb_3: res_2);

> +  i_4 = __PHI (bb_1: 0, bb_3: i_5);

> +  ivtmp_6 = __PHI (bb_1: 100U, bb_3: ivtmp_7);

> +  index = (long unsigned int) i_4;

> +  offset = index * 8UL;

> +  xi_ptr = x_8(D) + offset;

> +  xi = *xi_ptr;

> +  neg_xi = -xi;

> +  res_2 = neg_xi + res_1;

> +  i_5 = i_4 + 1;

> +  ivtmp_7 = ivtmp_6 - 1U;

> +  if (ivtmp_7 != 0U)

> +    goto bb_3;

> +  else

> +    goto bb_4;

> +

> + bb_3:

> +  goto bb_2;

> +

> + bb_4:

> +  res_3 = __PHI (bb_2: res_2);

> +  return res_3;

> +}

> +

> +/* { dg-final { scan-tree-dump { = \.COND_ADD} "vect" { target { vect_double_cond_arith && vect_fully_masked } } } } */

> +/* { dg-final { scan-tree-dump { = \.COND_SUB} "optimized" { target { vect_double_cond_arith && vect_fully_masked } } } } */

> Index: gcc/testsuite/gcc.target/aarch64/sve/loop_add_6.c

> ===================================================================

> --- /dev/null	2018-04-20 16:19:46.369131350 +0100

> +++ gcc/testsuite/gcc.target/aarch64/sve/loop_add_6.c	2018-05-24 10:33:30.872095099 +0100

> @@ -0,0 +1,46 @@

> +/* { dg-do compile } */

> +/* { dg-options "-O2 -ftree-vectorize -fgimple -ffast-math" } */

> +

> +double __GIMPLE (startwith("loop"))

> +neg_xi (double *x)

> +{

> +  int i;

> +  long unsigned int index;

> +  long unsigned int offset;

> +  double * xi_ptr;

> +  double xi;

> +  double neg_xi;

> +  double res;

> +  unsigned int ivtmp;

> +

> + bb_1:

> +  goto bb_2;

> +

> + bb_2:

> +  res_1 = __PHI (bb_1: 0.0, bb_3: res_2);

> +  i_4 = __PHI (bb_1: 0, bb_3: i_5);

> +  ivtmp_6 = __PHI (bb_1: 100U, bb_3: ivtmp_7);

> +  index = (long unsigned int) i_4;

> +  offset = index * 8UL;

> +  xi_ptr = x_8(D) + offset;

> +  xi = *xi_ptr;

> +  neg_xi = -xi;

> +  res_2 = neg_xi + res_1;

> +  i_5 = i_4 + 1;

> +  ivtmp_7 = ivtmp_6 - 1U;

> +  if (ivtmp_7 != 0U)

> +    goto bb_3;

> +  else

> +    goto bb_4;

> +

> + bb_3:

> +  goto bb_2;

> +

> + bb_4:

> +  res_3 = __PHI (bb_2: res_2);

> +  return res_3;

> +}

> +

> +/* { dg-final { scan-assembler {\tfsub\tz[0-9]+\.d, p[0-7]/m} } } */

> +/* { dg-final { scan-assembler-not {\tsel\t} } } */

> +/* { dg-final { scan-assembler-not {\tmovprfx\t} } } */
Richard Biener June 6, 2018, 11:37 a.m. | #2
On Thu, May 24, 2018 at 11:36 AM Richard Sandiford
<richard.sandiford@linaro.org> wrote:
>

> This patch adds match.pd support for applying normal folds to their

> IFN_COND_* forms.  E.g. the rule:

>

>   (plus @0 (negate @1)) -> (minus @0 @1)

>

> also allows the fold:

>

>   (IFN_COND_ADD @0 @1 (negate @2) @3) -> (IFN_COND_SUB @0 @1 @2 @3)

>

> Actually doing this by direct matches in gimple-match.c would

> probably lead to combinatorial explosion, so instead, the patch

> makes gimple_match_op carry a condition under which the operation

> happens ("cond"), and the value to use when the condition is false

> ("else_value").  Thus in the example above we'd do the following

>

> (a) convert:

>

>       cond:NULL_TREE (IFN_COND_ADD @0 @1 @4 @3) else_value:NULL_TREE

>

>     to:

>

>       cond:@0 (plus @1 @4) else_value:@3

>

> (b) apply gimple_resimplify to (plus @1 @4)

>

> (c) reintroduce cond and else_value when constructing the result.

>

> Nested operations inherit the condition of the outer operation

> (so that we don't introduce extra faults) but have a null else_value.

> If we try to build such an operation, the target gets to choose what

> else_value it can handle efficiently: obvious choices include one of

> the operands or a zero constant.  (The alternative would be to have some

> representation for an undefined value, but that seems a bit invasive,

> and isn't likely to be useful here.)

>

> I've made the condition a mandatory part of the gimple_match_op

> constructor so that it doesn't accidentally get dropped.

>

> Tested on aarch64-linux-gnu (with and without SVE), aarch64_be-elf

> and x86_64-linux-gnu.  OK to install?


It looks somewhat clever but after looking for a while it doesn't handle
simplifying

 (IFN_COND_ADD @0 @1 (IFN_COND_SUB @0 @2 @1 @3) @3)

to

 (cond @0 @2 @3)

right?  Because while the conditional gimple_match_op is built
by try_conditional_simplification it isn't built when doing
SSA use->def following in the generated matching code?

So it looks like a bit much noise for this very special case?

I suppose you ran into the need of these foldings from looking
at real code - which foldings specifically were appearing here?
Usually code is well optimized before if-conversion/vectorization
so we shouldn't need full-blown handling?

That said, I'm not sure how much work it is to massage

      if (gimple *def_stmt = get_def (valueize, op2))
        {
          if (gassign *def = dyn_cast <gassign *> (def_stmt))
            switch (gimple_assign_rhs_code (def))
              {
              case PLUS_EXPR:

to look like

      if (gimple *def_stmt = get_def (valueize, op2))
        {
           code = ERROR_MARK;
           if (!is_cond_ifn_with_cond (curr_gimple_match_op, &code))
             if (gassign *def dyn_cast <gassign *> (def_stmt))
               code = gimple_assign_rhs_code (def);
           switch (code)
             {
             case PLUS_EXPR:

thus transparently treat the IFN_COND_* as their "code" if the condition
matches that of the context (I'm not sure if we can do anything for
mismatching contexts).

Richard.

> Richard

>

>

> 2018-05-24  Richard Sandiford  <richard.sandiford@linaro.org>

>

> gcc/

>         * target.def (preferred_else_value): New target hook.

>         * doc/tm.texi.in (TARGET_PREFERRED_ELSE_VALUE): New hook.

>         * doc/tm.texi: Regenerate.

>         * targhooks.h (default_preferred_else_value): Declare.

>         * targhooks.c (default_preferred_else_value): New function.

>         * internal-fn.h (conditional_internal_fn_code): Declare.

>         * internal-fn.c (FOR_EACH_CODE_MAPPING): New macro.

>         (get_conditional_internal_fn): Use it.

>         (conditional_internal_fn_code): New function.

>         * gimple-match.h (gimple_match_cond): New struct.

>         (gimple_match_op): Add a cond member function.

>         (gimple_match_op::gimple_match_op): Update all forms to take a

>         gimple_match_cond.

>         * genmatch.c (expr::gen_transform): Use the same condition as res_op

>         for the suboperation, but don't specify a particular else_value.

>         * tree-ssa-sccvn.c (vn_nary_simplify, vn_reference_lookup_3)

>         (visit_nary_op, visit_reference_op_load): Pass

>         gimple_match_cond::UNCOND to the gimple_match_op constructor.

>         * gimple-match-head.c: Include tree-eh.h

>         (convert_conditional_op): New function.

>         (maybe_resimplify_conditional_op): Likewise.

>         (gimple_resimplify1): Call maybe_resimplify_conditional_op.

>         (gimple_resimplify2): Likewise.

>         (gimple_resimplify3): Likewise.

>         (gimple_resimplify4): Likewise.

>         (maybe_push_res_to_seq): Return null for conditional operations.

>         (try_conditional_simplification): New function.

>         (gimple_simplify): Call it.  Pass conditions to the gimple_match_op

>         constructor.

>         * match.pd: Fold VEC_COND_EXPRs of an IFN_COND_* call to a new

>         IFN_COND_* call.

>         * config/aarch64/aarch64.c (aarch64_preferred_else_value): New

>         function.

>         (TARGET_PREFERRED_ELSE_VALUE): Redefine.

>

> gcc/testsuite/

>         * gcc.dg/vect/vect-cond-arith-2.c: New test.

>         * gcc.target/aarch64/sve/loop_add_6.c: Likewise.

>

> Index: gcc/target.def

> ===================================================================

> --- gcc/target.def      2018-05-01 19:30:30.159632586 +0100

> +++ gcc/target.def      2018-05-24 10:33:30.871095132 +0100

> @@ -2040,6 +2040,25 @@ HOOK_VECTOR_END (vectorize)

>  #define HOOK_PREFIX "TARGET_"

>

>  DEFHOOK

> +(preferred_else_value,

> + "This hook returns the target's preferred final argument for a call\n\

> +to conditional internal function @var{ifn} (really of type\n\

> +@code{internal_fn}).  @var{type} specifies the return type of the\n\

> +function and @var{ops} are the operands to the conditional operation,\n\

> +of which there are @var{nops}.\n\

> +\n\

> +For example, if @var{ifn} is @code{IFN_COND_ADD}, the hook returns\n\

> +a value of type @var{type} that should be used when @samp{@var{ops}[0]}\n\

> +and @samp{@var{ops}[1]} are conditionally added together.\n\

> +\n\

> +This hook is only relevant if the target supports conditional patterns\n\

> +like @code{cond_add@var{m}}.  The default implementation returns a zero\n\

> +constant of type @var{type}.",

> + tree,

> + (unsigned ifn, tree type, unsigned nops, tree *ops),

> + default_preferred_else_value)

> +

> +DEFHOOK

>  (record_offload_symbol,

>   "Used when offloaded functions are seen in the compilation unit and no named\n\

>  sections are available.  It is called once for each symbol that must be\n\

> Index: gcc/doc/tm.texi.in

> ===================================================================

> --- gcc/doc/tm.texi.in  2018-05-01 19:30:28.730694873 +0100

> +++ gcc/doc/tm.texi.in  2018-05-24 10:33:30.869095197 +0100

> @@ -4149,6 +4149,8 @@ address;  but often a machine-dependent

>

>  @hook TARGET_GOACC_REDUCTION

>

> +@hook TARGET_PREFERRED_ELSE_VALUE

> +

>  @node Anchored Addresses

>  @section Anchored Addresses

>  @cindex anchored addresses

> Index: gcc/doc/tm.texi

> ===================================================================

> --- gcc/doc/tm.texi     2018-05-01 19:30:28.722695224 +0100

> +++ gcc/doc/tm.texi     2018-05-24 10:33:30.868095229 +0100

> @@ -6046,6 +6046,22 @@ expanded sequence has been inserted.  Th

>  for allocating any storage for reductions when necessary.

>  @end deftypefn

>

> +@deftypefn {Target Hook} tree TARGET_PREFERRED_ELSE_VALUE (unsigned @var{ifn}, tree @var{type}, unsigned @var{nops}, tree *@var{ops})

> +This hook returns the target's preferred final argument for a call

> +to conditional internal function @var{ifn} (really of type

> +@code{internal_fn}).  @var{type} specifies the return type of the

> +function and @var{ops} are the operands to the conditional operation,

> +of which there are @var{nops}.

> +

> +For example, if @var{ifn} is @code{IFN_COND_ADD}, the hook returns

> +a value of type @var{type} that should be used when @samp{@var{ops}[0]}

> +and @samp{@var{ops}[1]} are conditionally added together.

> +

> +This hook is only relevant if the target supports conditional patterns

> +like @code{cond_add@var{m}}.  The default implementation returns a zero

> +constant of type @var{type}.

> +@end deftypefn

> +

>  @node Anchored Addresses

>  @section Anchored Addresses

>  @cindex anchored addresses

> Index: gcc/targhooks.h

> ===================================================================

> --- gcc/targhooks.h     2018-05-01 19:30:29.390666052 +0100

> +++ gcc/targhooks.h     2018-05-24 10:33:30.872095099 +0100

> @@ -289,5 +289,6 @@ extern unsigned int default_min_arithmet

>  default_excess_precision (enum excess_precision_type ATTRIBUTE_UNUSED);

>  extern bool default_stack_clash_protection_final_dynamic_probe (rtx);

>  extern void default_select_early_remat_modes (sbitmap);

> +extern tree default_preferred_else_value (unsigned, tree, unsigned, tree *);

>

>  #endif /* GCC_TARGHOOKS_H */

> Index: gcc/targhooks.c

> ===================================================================

> --- gcc/targhooks.c     2018-05-01 19:30:29.390666052 +0100

> +++ gcc/targhooks.c     2018-05-24 10:33:30.871095132 +0100

> @@ -2345,4 +2345,12 @@ default_select_early_remat_modes (sbitma

>  {

>  }

>

> +/* The default implementation of TARGET_PREFERRED_ELSE_VALUE.  */

> +

> +tree

> +default_preferred_else_value (unsigned, tree type, unsigned, tree *)

> +{

> +  return build_zero_cst (type);

> +}

> +

>  #include "gt-targhooks.h"

> Index: gcc/internal-fn.h

> ===================================================================

> --- gcc/internal-fn.h   2018-05-17 11:52:13.507173989 +0100

> +++ gcc/internal-fn.h   2018-05-24 10:33:30.870095164 +0100

> @@ -193,6 +193,7 @@ direct_internal_fn_supported_p (internal

>  extern bool set_edom_supported_p (void);

>

>  extern internal_fn get_conditional_internal_fn (tree_code);

> +extern tree_code conditional_internal_fn_code (internal_fn);

>

>  extern bool internal_load_fn_p (internal_fn);

>  extern bool internal_store_fn_p (internal_fn);

> Index: gcc/internal-fn.c

> ===================================================================

> --- gcc/internal-fn.c   2018-05-24 10:12:10.146352152 +0100

> +++ gcc/internal-fn.c   2018-05-24 10:33:30.870095164 +0100

> @@ -3219,6 +3219,21 @@ #define DEF_INTERNAL_FN(CODE, FLAGS, FNS

>    0

>  };

>

> +/* Invoke T(CODE, IFN) for each conditional function IFN that maps to a

> +   tree code CODE.  */

> +#define FOR_EACH_CODE_MAPPING(T) \

> +  T (PLUS_EXPR, IFN_COND_ADD) \

> +  T (MINUS_EXPR, IFN_COND_SUB) \

> +  T (MULT_EXPR, IFN_COND_MUL) \

> +  T (TRUNC_DIV_EXPR, IFN_COND_DIV) \

> +  T (TRUNC_MOD_EXPR, IFN_COND_MOD) \

> +  T (RDIV_EXPR, IFN_COND_RDIV) \

> +  T (MIN_EXPR, IFN_COND_MIN) \

> +  T (MAX_EXPR, IFN_COND_MAX) \

> +  T (BIT_AND_EXPR, IFN_COND_AND) \

> +  T (BIT_IOR_EXPR, IFN_COND_IOR) \

> +  T (BIT_XOR_EXPR, IFN_COND_XOR)

> +

>  /* Return a function that only performs CODE when a certain condition is met

>     and that uses a given fallback value otherwise.  For example, if CODE is

>     a binary operation associated with conditional function FN:

> @@ -3238,31 +3253,30 @@ get_conditional_internal_fn (tree_code c

>  {

>    switch (code)

>      {

> -    case PLUS_EXPR:

> -      return IFN_COND_ADD;

> -    case MINUS_EXPR:

> -      return IFN_COND_SUB;

> -    case MIN_EXPR:

> -      return IFN_COND_MIN;

> -    case MAX_EXPR:

> -      return IFN_COND_MAX;

> -    case TRUNC_DIV_EXPR:

> -      return IFN_COND_DIV;

> -    case TRUNC_MOD_EXPR:

> -      return IFN_COND_MOD;

> -    case RDIV_EXPR:

> -      return IFN_COND_RDIV;

> -    case BIT_AND_EXPR:

> -      return IFN_COND_AND;

> -    case BIT_IOR_EXPR:

> -      return IFN_COND_IOR;

> -    case BIT_XOR_EXPR:

> -      return IFN_COND_XOR;

> +#define CASE(CODE, IFN) case CODE: return IFN;

> +      FOR_EACH_CODE_MAPPING(CASE)

> +#undef CASE

>      default:

>        return IFN_LAST;

>      }

>  }

>

> +/* If IFN implements the conditional form of a tree code, return that

> +   tree code, otherwise return ERROR_MARK.  */

> +

> +tree_code

> +conditional_internal_fn_code (internal_fn ifn)

> +{

> +  switch (ifn)

> +    {

> +#define CASE(CODE, IFN) case IFN: return CODE;

> +      FOR_EACH_CODE_MAPPING(CASE)

> +#undef CASE

> +    default:

> +      return ERROR_MARK;

> +    }

> +}

> +

>  /* Return true if IFN is some form of load from memory.  */

>

>  bool

> Index: gcc/gimple-match.h

> ===================================================================

> --- gcc/gimple-match.h  2018-05-24 09:54:37.509451356 +0100

> +++ gcc/gimple-match.h  2018-05-24 10:33:30.870095164 +0100

> @@ -40,16 +40,57 @@ #define GCC_GIMPLE_MATCH_H

>    int rep;

>  };

>

> +/* Represents the condition under which an operation should happen,

> +   and the value to use otherwise.  The condition applies elementwise

> +   (as for VEC_COND_EXPR) if the values are vectors.  */

> +struct gimple_match_cond

> +{

> +  enum uncond { UNCOND };

> +

> +  /* Build an unconditional op.  */

> +  gimple_match_cond (uncond) : cond (NULL_TREE), else_value (NULL_TREE) {}

> +  gimple_match_cond (tree, tree);

> +

> +  gimple_match_cond any_else () const;

> +

> +  /* The condition under which the operation occurs, or NULL_TREE

> +     if the operation is unconditional.  */

> +  tree cond;

> +

> +  /* The value to use when the condition is false.  This is NULL_TREE if

> +     the operation is unconditional or if the value doesn't matter.  */

> +  tree else_value;

> +};

> +

> +inline

> +gimple_match_cond::gimple_match_cond (tree cond_in, tree else_value_in)

> +  : cond (cond_in), else_value (else_value_in)

> +{

> +}

> +

> +/* Return a gimple_match_cond with the same condition but with an

> +   arbitrary ELSE_VALUE.  */

> +

> +inline gimple_match_cond

> +gimple_match_cond::any_else () const

> +{

> +  return gimple_match_cond (cond, NULL_TREE);

> +}

> +

>  /* Represents an operation to be simplified, or the result of the

>     simplification.  */

>  struct gimple_match_op

>  {

> -  gimple_match_op () : type (NULL_TREE), num_ops (0) {}

> -  gimple_match_op (code_helper, tree, unsigned int);

> -  gimple_match_op (code_helper, tree, tree);

> -  gimple_match_op (code_helper, tree, tree, tree);

> -  gimple_match_op (code_helper, tree, tree, tree, tree);

> -  gimple_match_op (code_helper, tree, tree, tree, tree, tree);

> +  gimple_match_op ();

> +  gimple_match_op (const gimple_match_cond &, code_helper, tree, unsigned int);

> +  gimple_match_op (const gimple_match_cond &,

> +                  code_helper, tree, tree);

> +  gimple_match_op (const gimple_match_cond &,

> +                  code_helper, tree, tree, tree);

> +  gimple_match_op (const gimple_match_cond &,

> +                  code_helper, tree, tree, tree, tree);

> +  gimple_match_op (const gimple_match_cond &,

> +                  code_helper, tree, tree, tree, tree, tree);

>

>    void set_op (code_helper, tree, unsigned int);

>    void set_op (code_helper, tree, tree);

> @@ -63,6 +104,10 @@ struct gimple_match_op

>    /* The maximum value of NUM_OPS.  */

>    static const unsigned int MAX_NUM_OPS = 4;

>

> +  /* The conditions under which the operation is performed, and the value to

> +     use as a fallback.  */

> +  gimple_match_cond cond;

> +

>    /* The operation being performed.  */

>    code_helper code;

>

> @@ -76,39 +121,49 @@ struct gimple_match_op

>    tree ops[MAX_NUM_OPS];

>  };

>

> -/* Constructor that takes the code, type and number of operands, but leaves

> -   the caller to fill in the operands.  */

> +inline

> +gimple_match_op::gimple_match_op ()

> +  : cond (gimple_match_cond::UNCOND), type (NULL_TREE), num_ops (0)

> +{

> +}

> +

> +/* Constructor that takes the condition, code, type and number of

> +   operands, but leaves the caller to fill in the operands.  */

>

>  inline

> -gimple_match_op::gimple_match_op (code_helper code_in, tree type_in,

> +gimple_match_op::gimple_match_op (const gimple_match_cond &cond_in,

> +                                 code_helper code_in, tree type_in,

>                                   unsigned int num_ops_in)

> -  : code (code_in), type (type_in), num_ops (num_ops_in)

> +  : cond (cond_in), code (code_in), type (type_in), num_ops (num_ops_in)

>  {

>  }

>

>  /* Constructors for various numbers of operands.  */

>

>  inline

> -gimple_match_op::gimple_match_op (code_helper code_in, tree type_in,

> +gimple_match_op::gimple_match_op (const gimple_match_cond &cond_in,

> +                                 code_helper code_in, tree type_in,

>                                   tree op0)

> -  : code (code_in), type (type_in), num_ops (1)

> +  : cond (cond_in), code (code_in), type (type_in), num_ops (1)

>  {

>    ops[0] = op0;

>  }

>

>  inline

> -gimple_match_op::gimple_match_op (code_helper code_in, tree type_in,

> +gimple_match_op::gimple_match_op (const gimple_match_cond &cond_in,

> +                                 code_helper code_in, tree type_in,

>                                   tree op0, tree op1)

> -  : code (code_in), type (type_in), num_ops (2)

> +  : cond (cond_in), code (code_in), type (type_in), num_ops (2)

>  {

>    ops[0] = op0;

>    ops[1] = op1;

>  }

>

>  inline

> -gimple_match_op::gimple_match_op (code_helper code_in, tree type_in,

> +gimple_match_op::gimple_match_op (const gimple_match_cond &cond_in,

> +                                 code_helper code_in, tree type_in,

>                                   tree op0, tree op1, tree op2)

> -  : code (code_in), type (type_in), num_ops (3)

> +  : cond (cond_in), code (code_in), type (type_in), num_ops (3)

>  {

>    ops[0] = op0;

>    ops[1] = op1;

> @@ -116,9 +171,10 @@ gimple_match_op::gimple_match_op (code_h

>  }

>

>  inline

> -gimple_match_op::gimple_match_op (code_helper code_in, tree type_in,

> +gimple_match_op::gimple_match_op (const gimple_match_cond &cond_in,

> +                                 code_helper code_in, tree type_in,

>                                   tree op0, tree op1, tree op2, tree op3)

> -  : code (code_in), type (type_in), num_ops (4)

> +  : cond (cond_in), code (code_in), type (type_in), num_ops (4)

>  {

>    ops[0] = op0;

>    ops[1] = op1;

> Index: gcc/genmatch.c

> ===================================================================

> --- gcc/genmatch.c      2018-05-24 10:12:10.145352193 +0100

> +++ gcc/genmatch.c      2018-05-24 10:33:30.869095197 +0100

> @@ -2507,8 +2507,8 @@ expr::gen_transform (FILE *f, int indent

>        /* ???  Building a stmt can fail for various reasons here, seq being

>           NULL or the stmt referencing SSA names occuring in abnormal PHIs.

>          So if we fail here we should continue matching other patterns.  */

> -      fprintf_indent (f, indent, "gimple_match_op tem_op (%s, %s",

> -                     opr_name, type);

> +      fprintf_indent (f, indent, "gimple_match_op tem_op "

> +                     "(res_op->cond.any_else (), %s, %s", opr_name, type);

>        for (unsigned i = 0; i < ops.length (); ++i)

>         fprintf (f, ", ops%d[%u]", depth, i);

>        fprintf (f, ");\n");

> Index: gcc/tree-ssa-sccvn.c

> ===================================================================

> --- gcc/tree-ssa-sccvn.c        2018-05-24 09:02:28.765328358 +0100

> +++ gcc/tree-ssa-sccvn.c        2018-05-24 10:33:30.872095099 +0100

> @@ -1804,7 +1804,8 @@ vn_nary_simplify (vn_nary_op_t nary)

>  {

>    if (nary->length > gimple_match_op::MAX_NUM_OPS)

>      return NULL_TREE;

> -  gimple_match_op op (nary->opcode, nary->type, nary->length);

> +  gimple_match_op op (gimple_match_cond::UNCOND, nary->opcode,

> +                     nary->type, nary->length);

>    memcpy (op.ops, nary->op, sizeof (tree) * nary->length);

>    return vn_nary_build_or_lookup_1 (&op, false);

>  }

> @@ -2031,8 +2032,8 @@ vn_reference_lookup_3 (ao_ref *ref, tree

>           else if (INTEGRAL_TYPE_P (vr->type)

>                    && known_eq (ref->size, 8))

>             {

> -             gimple_match_op res_op (NOP_EXPR, vr->type,

> -                                     gimple_call_arg (def_stmt, 1));

> +             gimple_match_op res_op (gimple_match_cond::UNCOND, NOP_EXPR,

> +                                     vr->type, gimple_call_arg (def_stmt, 1));

>               val = vn_nary_build_or_lookup (&res_op);

>               if (!val

>                   || (TREE_CODE (val) == SSA_NAME

> @@ -2172,7 +2173,8 @@ vn_reference_lookup_3 (ao_ref *ref, tree

>               || known_eq (ref->size, TYPE_PRECISION (vr->type)))

>           && multiple_p (ref->size, BITS_PER_UNIT))

>         {

> -         gimple_match_op op (BIT_FIELD_REF, vr->type,

> +         gimple_match_op op (gimple_match_cond::UNCOND,

> +                             BIT_FIELD_REF, vr->type,

>                               SSA_VAL (gimple_assign_rhs1 (def_stmt)),

>                               bitsize_int (ref->size),

>                               bitsize_int (offset - offset2));

> @@ -3701,7 +3703,8 @@ visit_nary_op (tree lhs, gassign *stmt)

>                       unsigned rhs_prec = TYPE_PRECISION (TREE_TYPE (rhs1));

>                       if (lhs_prec == rhs_prec)

>                         {

> -                         gimple_match_op match_op (NOP_EXPR, type, ops[0]);

> +                         gimple_match_op match_op (gimple_match_cond::UNCOND,

> +                                                   NOP_EXPR, type, ops[0]);

>                           result = vn_nary_build_or_lookup (&match_op);

>                           if (result)

>                             {

> @@ -3714,7 +3717,8 @@ visit_nary_op (tree lhs, gassign *stmt)

>                         {

>                           tree mask = wide_int_to_tree

>                             (type, wi::mask (rhs_prec, false, lhs_prec));

> -                         gimple_match_op match_op (BIT_AND_EXPR,

> +                         gimple_match_op match_op (gimple_match_cond::UNCOND,

> +                                                   BIT_AND_EXPR,

>                                                     TREE_TYPE (lhs),

>                                                     ops[0], mask);

>                           result = vn_nary_build_or_lookup (&match_op);

> @@ -3838,7 +3842,8 @@ visit_reference_op_load (tree lhs, tree

>          of VIEW_CONVERT_EXPR <TREE_TYPE (result)> (result).

>          So first simplify and lookup this expression to see if it

>          is already available.  */

> -      gimple_match_op res_op (VIEW_CONVERT_EXPR, TREE_TYPE (op), result);

> +      gimple_match_op res_op (gimple_match_cond::UNCOND,

> +                             VIEW_CONVERT_EXPR, TREE_TYPE (op), result);

>        result = vn_nary_build_or_lookup (&res_op);

>      }

>

> Index: gcc/gimple-match-head.c

> ===================================================================

> --- gcc/gimple-match-head.c     2018-05-24 09:54:37.509451356 +0100

> +++ gcc/gimple-match-head.c     2018-05-24 10:33:30.870095164 +0100

> @@ -40,6 +40,7 @@ Software Foundation; either version 3, o

>  #include "case-cfn-macros.h"

>  #include "gimplify.h"

>  #include "optabs-tree.h"

> +#include "tree-eh.h"

>

>

>  /* Forward declarations of the private auto-generated matchers.

> @@ -68,6 +69,95 @@ constant_for_folding (tree t)

>               && TREE_CODE (TREE_OPERAND (t, 0)) == STRING_CST));

>  }

>

> +/* Try to convert conditional operation ORIG_OP into an IFN_COND_*

> +   operation.  Return true on success, storing the new operation in NEW_OP.  */

> +

> +static bool

> +convert_conditional_op (gimple_match_op *orig_op,

> +                       gimple_match_op *new_op)

> +{

> +  internal_fn ifn;

> +  if (orig_op->code.is_tree_code ())

> +    ifn = get_conditional_internal_fn ((tree_code) orig_op->code);

> +  else

> +    return false;

> +  if (ifn == IFN_LAST)

> +    return false;

> +  unsigned int num_ops = orig_op->num_ops;

> +  new_op->set_op (as_combined_fn (ifn), orig_op->type, num_ops + 2);

> +  new_op->ops[0] = orig_op->cond.cond;

> +  for (unsigned int i = 0; i < num_ops; ++i)

> +    new_op->ops[i + 1] = orig_op->ops[i];

> +  tree else_value = orig_op->cond.else_value;

> +  if (!else_value)

> +    else_value = targetm.preferred_else_value (ifn, orig_op->type,

> +                                              num_ops, orig_op->ops);

> +  new_op->ops[num_ops + 1] = else_value;

> +  return true;

> +}

> +

> +/* RES_OP is the result of a simplification.  If it is conditional,

> +   try to replace it with the equivalent UNCOND form, such as an

> +   IFN_COND_* call or a VEC_COND_EXPR.  Also try to resimplify the

> +   result of the replacement if appropriate, adding any new statements to

> +   SEQ and using VALUEIZE as the valueization function.  Return true if

> +   this resimplification occurred and resulted in at least one change.  */

> +

> +static bool

> +maybe_resimplify_conditional_op (gimple_seq *seq, gimple_match_op *res_op,

> +                                tree (*valueize) (tree))

> +{

> +  if (!res_op->cond.cond)

> +    return false;

> +

> +  if (!res_op->cond.else_value

> +      && res_op->code.is_tree_code ())

> +    {

> +      /* The "else" value doesn't matter.  If the "then" value is a

> +        gimple value, just use it unconditionally.  This isn't a

> +        simplification in itself, since there was no operation to

> +        build in the first place.  */

> +      if (gimple_simplified_result_is_gimple_val (res_op))

> +       {

> +         res_op->cond.cond = NULL_TREE;

> +         return false;

> +       }

> +

> +      /* Likewise if the operation would not trap.  */

> +      bool honor_trapv = (INTEGRAL_TYPE_P (res_op->type)

> +                         && TYPE_OVERFLOW_TRAPS (res_op->type));

> +      if (!operation_could_trap_p ((tree_code) res_op->code,

> +                                  FLOAT_TYPE_P (res_op->type),

> +                                  honor_trapv, res_op->op_or_null (1)))

> +       {

> +         res_op->cond.cond = NULL_TREE;

> +         return false;

> +       }

> +    }

> +

> +  /* If the "then" value is a gimple value and the "else" value matters,

> +     create a VEC_COND_EXPR between them, then see if it can be further

> +     simplified.  */

> +  gimple_match_op new_op;

> +  if (res_op->cond.else_value

> +      && VECTOR_TYPE_P (res_op->type)

> +      && gimple_simplified_result_is_gimple_val (res_op))

> +    {

> +      new_op.set_op (VEC_COND_EXPR, res_op->type,

> +                    res_op->cond.cond, res_op->ops[0],

> +                    res_op->cond.else_value);

> +      *res_op = new_op;

> +      return gimple_resimplify3 (seq, res_op, valueize);

> +    }

> +

> +  /* Otherwise try rewriting the operation as an IFN_COND_* call.

> +     Again, this isn't a simplification in itself, since it's what

> +     RES_OP already described.  */

> +  if (convert_conditional_op (res_op, &new_op))

> +    *res_op = new_op;

> +

> +  return false;

> +}

>

>  /* Helper that matches and simplifies the toplevel result from

>     a gimple_simplify run (where we don't want to build

> @@ -93,6 +183,7 @@ gimple_resimplify1 (gimple_seq *seq, gim

>           if (TREE_OVERFLOW_P (tem))

>             tem = drop_tree_overflow (tem);

>           res_op->set_value (tem);

> +         maybe_resimplify_conditional_op (seq, res_op, valueize);

>           return true;

>         }

>      }

> @@ -105,6 +196,9 @@ gimple_resimplify1 (gimple_seq *seq, gim

>        return true;

>      }

>

> +  if (maybe_resimplify_conditional_op (seq, res_op, valueize))

> +    return true;

> +

>    return false;

>  }

>

> @@ -134,6 +228,7 @@ gimple_resimplify2 (gimple_seq *seq, gim

>           if (TREE_OVERFLOW_P (tem))

>             tem = drop_tree_overflow (tem);

>           res_op->set_value (tem);

> +         maybe_resimplify_conditional_op (seq, res_op, valueize);

>           return true;

>         }

>      }

> @@ -160,6 +255,9 @@ gimple_resimplify2 (gimple_seq *seq, gim

>        return true;

>      }

>

> +  if (maybe_resimplify_conditional_op (seq, res_op, valueize))

> +    return true;

> +

>    return canonicalized;

>  }

>

> @@ -191,6 +289,7 @@ gimple_resimplify3 (gimple_seq *seq, gim

>           if (TREE_OVERFLOW_P (tem))

>             tem = drop_tree_overflow (tem);

>           res_op->set_value (tem);

> +         maybe_resimplify_conditional_op (seq, res_op, valueize);

>           return true;

>         }

>      }

> @@ -214,6 +313,9 @@ gimple_resimplify3 (gimple_seq *seq, gim

>        return true;

>      }

>

> +  if (maybe_resimplify_conditional_op (seq, res_op, valueize))

> +    return true;

> +

>    return canonicalized;

>  }

>

> @@ -239,6 +341,9 @@ gimple_resimplify4 (gimple_seq *seq, gim

>        return true;

>      }

>

> +  if (maybe_resimplify_conditional_op (seq, res_op, valueize))

> +    return true;

> +

>    return false;

>  }

>

> @@ -297,6 +402,12 @@ maybe_push_res_to_seq (gimple_match_op *

>    tree *ops = res_op->ops;

>    unsigned num_ops = res_op->num_ops;

>

> +  /* The caller should have converted conditional operations into an UNCOND

> +     form and resimplified as appropriate.  The conditional form only

> +     survives this far if that conversion failed.  */

> +  if (res_op->cond.cond)

> +    return NULL_TREE;

> +

>    if (res_op->code.is_tree_code ())

>      {

>        if (!res

> @@ -558,6 +669,50 @@ do_valueize (tree op, tree (*valueize)(t

>    return op;

>  }

>

> +/* If RES_OP is a call to a conditional internal function, try simplifying

> +   the associated unconditional operation and using the result to build

> +   a new conditional operation.  For example, if RES_OP is:

> +

> +     IFN_COND_ADD (COND, A, B, ELSE)

> +

> +   try simplifying (plus A B) and using the result to build a replacement

> +   for the whole IFN_COND_ADD.

> +

> +   Return true if this approach led to a simplification, otherwise leave

> +   RES_OP unchanged (and so suitable for other simplifications).  When

> +   returning true, add any new statements to SEQ and use VALUEIZE as the

> +   valueization function.

> +

> +   RES_OP is known to be a call to IFN.  */

> +

> +static bool

> +try_conditional_simplification (internal_fn ifn, gimple_match_op *res_op,

> +                               gimple_seq *seq, tree (*valueize) (tree))

> +{

> +  tree_code code = conditional_internal_fn_code (ifn);

> +  if (code == ERROR_MARK)

> +    return false;

> +

> +  unsigned int num_ops = res_op->num_ops;

> +  gimple_match_op cond_op (gimple_match_cond (res_op->ops[0],

> +                                             res_op->ops[num_ops - 1]),

> +                          code, res_op->type, num_ops - 2);

> +  for (unsigned int i = 1; i < num_ops - 1; ++i)

> +    cond_op.ops[i - 1] = res_op->ops[i];

> +  switch (num_ops - 2)

> +    {

> +    case 2:

> +      if (!gimple_resimplify2 (seq, &cond_op, valueize))

> +       return false;

> +      break;

> +    default:

> +      gcc_unreachable ();

> +    }

> +  *res_op = cond_op;

> +  maybe_resimplify_conditional_op (seq, res_op, valueize);

> +  return true;

> +}

> +

>  /* The main STMT based simplification entry.  It is used by the fold_stmt

>     and the fold_stmt_to_constant APIs.  */

>

> @@ -643,7 +798,7 @@ gimple_simplify (gimple *stmt, gimple_ma

>                       tree rhs = TREE_OPERAND (rhs1, 1);

>                       lhs = do_valueize (lhs, top_valueize, valueized);

>                       rhs = do_valueize (rhs, top_valueize, valueized);

> -                     gimple_match_op res_op2 (TREE_CODE (rhs1),

> +                     gimple_match_op res_op2 (res_op->cond, TREE_CODE (rhs1),

>                                                TREE_TYPE (rhs1), lhs, rhs);

>                       if ((gimple_resimplify2 (seq, &res_op2, valueize)

>                            || valueized)

> @@ -714,6 +869,10 @@ gimple_simplify (gimple *stmt, gimple_ma

>               tree arg = gimple_call_arg (stmt, i);

>               res_op->ops[i] = do_valueize (arg, top_valueize, valueized);

>             }

> +         if (internal_fn_p (cfn)

> +             && try_conditional_simplification (as_internal_fn (cfn),

> +                                                res_op, seq, valueize))

> +           return true;

>           switch (num_args)

>             {

>             case 1:

> Index: gcc/match.pd

> ===================================================================

> --- gcc/match.pd        2018-05-24 10:12:10.146352152 +0100

> +++ gcc/match.pd        2018-05-24 10:33:30.870095164 +0100

> @@ -4797,3 +4797,12 @@ DEFINE_INT_AND_FLOAT_ROUND_FN (RINT)

>    (with { tree op_type = TREE_TYPE (@4); }

>     (if (element_precision (type) == element_precision (op_type))

>      (view_convert (cond_op (bit_not @0) @2 @3 (view_convert:op_type @1)))))))

> +

> +/* Detect cases in which a VEC_COND_EXPR effectively replaces the

> +   "else" value of an IFN_COND_*.  */

> +(for cond_op (COND_BINARY)

> + (simplify

> +  (vec_cond @0 (view_convert? (cond_op @0 @1 @2 @3)) @4)

> +  (with { tree op_type = TREE_TYPE (@3); }

> +   (if (element_precision (type) == element_precision (op_type))

> +    (view_convert (cond_op @0 @1 @2 (view_convert:op_type @4)))))))

> Index: gcc/config/aarch64/aarch64.c

> ===================================================================

> --- gcc/config/aarch64/aarch64.c        2018-05-24 09:54:37.507451418 +0100

> +++ gcc/config/aarch64/aarch64.c        2018-05-24 10:33:30.867095262 +0100

> @@ -1292,6 +1292,16 @@ aarch64_get_mask_mode (poly_uint64 nunit

>    return default_get_mask_mode (nunits, nbytes);

>  }

>

> +/* Implement TARGET_PREFERRED_ELSE_VALUE.  Prefer to use the first

> +   arithmetic operand as the else value if the else value doesn't matter,

> +   since that exactly matches the SVE destructive merging form.  */

> +

> +static tree

> +aarch64_preferred_else_value (unsigned, tree, unsigned int, tree *ops)

> +{

> +  return ops[0];

> +}

> +

>  /* Implement TARGET_HARD_REGNO_NREGS.  */

>

>  static unsigned int

> @@ -17980,6 +17990,9 @@ #define TARGET_VECTORIZE_GET_MASK_MODE a

>  #undef TARGET_VECTORIZE_EMPTY_MASK_IS_EXPENSIVE

>  #define TARGET_VECTORIZE_EMPTY_MASK_IS_EXPENSIVE \

>    aarch64_empty_mask_is_expensive

> +#undef TARGET_PREFERRED_ELSE_VALUE

> +#define TARGET_PREFERRED_ELSE_VALUE \

> +  aarch64_preferred_else_value

>

>  #undef TARGET_INIT_LIBFUNCS

>  #define TARGET_INIT_LIBFUNCS aarch64_init_libfuncs

> Index: gcc/testsuite/gcc.dg/vect/vect-cond-arith-2.c

> ===================================================================

> --- /dev/null   2018-04-20 16:19:46.369131350 +0100

> +++ gcc/testsuite/gcc.dg/vect/vect-cond-arith-2.c       2018-05-24 10:33:30.872095099 +0100

> @@ -0,0 +1,45 @@

> +/* { dg-do compile } */

> +/* { dg-additional-options "-fgimple -fdump-tree-optimized -ffast-math" } */

> +

> +double __GIMPLE (startwith("loop"))

> +neg_xi (double *x)

> +{

> +  int i;

> +  long unsigned int index;

> +  long unsigned int offset;

> +  double * xi_ptr;

> +  double xi;

> +  double neg_xi;

> +  double res;

> +  unsigned int ivtmp;

> +

> + bb_1:

> +  goto bb_2;

> +

> + bb_2:

> +  res_1 = __PHI (bb_1: 0.0, bb_3: res_2);

> +  i_4 = __PHI (bb_1: 0, bb_3: i_5);

> +  ivtmp_6 = __PHI (bb_1: 100U, bb_3: ivtmp_7);

> +  index = (long unsigned int) i_4;

> +  offset = index * 8UL;

> +  xi_ptr = x_8(D) + offset;

> +  xi = *xi_ptr;

> +  neg_xi = -xi;

> +  res_2 = neg_xi + res_1;

> +  i_5 = i_4 + 1;

> +  ivtmp_7 = ivtmp_6 - 1U;

> +  if (ivtmp_7 != 0U)

> +    goto bb_3;

> +  else

> +    goto bb_4;

> +

> + bb_3:

> +  goto bb_2;

> +

> + bb_4:

> +  res_3 = __PHI (bb_2: res_2);

> +  return res_3;

> +}

> +

> +/* { dg-final { scan-tree-dump { = \.COND_ADD} "vect" { target { vect_double_cond_arith && vect_fully_masked } } } } */

> +/* { dg-final { scan-tree-dump { = \.COND_SUB} "optimized" { target { vect_double_cond_arith && vect_fully_masked } } } } */

> Index: gcc/testsuite/gcc.target/aarch64/sve/loop_add_6.c

> ===================================================================

> --- /dev/null   2018-04-20 16:19:46.369131350 +0100

> +++ gcc/testsuite/gcc.target/aarch64/sve/loop_add_6.c   2018-05-24 10:33:30.872095099 +0100

> @@ -0,0 +1,46 @@

> +/* { dg-do compile } */

> +/* { dg-options "-O2 -ftree-vectorize -fgimple -ffast-math" } */

> +

> +double __GIMPLE (startwith("loop"))

> +neg_xi (double *x)

> +{

> +  int i;

> +  long unsigned int index;

> +  long unsigned int offset;

> +  double * xi_ptr;

> +  double xi;

> +  double neg_xi;

> +  double res;

> +  unsigned int ivtmp;

> +

> + bb_1:

> +  goto bb_2;

> +

> + bb_2:

> +  res_1 = __PHI (bb_1: 0.0, bb_3: res_2);

> +  i_4 = __PHI (bb_1: 0, bb_3: i_5);

> +  ivtmp_6 = __PHI (bb_1: 100U, bb_3: ivtmp_7);

> +  index = (long unsigned int) i_4;

> +  offset = index * 8UL;

> +  xi_ptr = x_8(D) + offset;

> +  xi = *xi_ptr;

> +  neg_xi = -xi;

> +  res_2 = neg_xi + res_1;

> +  i_5 = i_4 + 1;

> +  ivtmp_7 = ivtmp_6 - 1U;

> +  if (ivtmp_7 != 0U)

> +    goto bb_3;

> +  else

> +    goto bb_4;

> +

> + bb_3:

> +  goto bb_2;

> +

> + bb_4:

> +  res_3 = __PHI (bb_2: res_2);

> +  return res_3;

> +}

> +

> +/* { dg-final { scan-assembler {\tfsub\tz[0-9]+\.d, p[0-7]/m} } } */

> +/* { dg-final { scan-assembler-not {\tsel\t} } } */

> +/* { dg-final { scan-assembler-not {\tmovprfx\t} } } */
Richard Sandiford June 6, 2018, 8:16 p.m. | #3
> On Thu, May 24, 2018 at 11:36 AM Richard Sandiford

> <richard.sandiford@linaro.org> wrote:

>>

>> This patch adds match.pd support for applying normal folds to their

>> IFN_COND_* forms.  E.g. the rule:

>>

>>   (plus @0 (negate @1)) -> (minus @0 @1)

>>

>> also allows the fold:

>>

>>   (IFN_COND_ADD @0 @1 (negate @2) @3) -> (IFN_COND_SUB @0 @1 @2 @3)

>>

>> Actually doing this by direct matches in gimple-match.c would

>> probably lead to combinatorial explosion, so instead, the patch

>> makes gimple_match_op carry a condition under which the operation

>> happens ("cond"), and the value to use when the condition is false

>> ("else_value").  Thus in the example above we'd do the following

>>

>> (a) convert:

>>

>>       cond:NULL_TREE (IFN_COND_ADD @0 @1 @4 @3) else_value:NULL_TREE

>>

>>     to:

>>

>>       cond:@0 (plus @1 @4) else_value:@3

>>

>> (b) apply gimple_resimplify to (plus @1 @4)

>>

>> (c) reintroduce cond and else_value when constructing the result.

>>

>> Nested operations inherit the condition of the outer operation

>> (so that we don't introduce extra faults) but have a null else_value.

>> If we try to build such an operation, the target gets to choose what

>> else_value it can handle efficiently: obvious choices include one of

>> the operands or a zero constant.  (The alternative would be to have some

>> representation for an undefined value, but that seems a bit invasive,

>> and isn't likely to be useful here.)

>>

>> I've made the condition a mandatory part of the gimple_match_op

>> constructor so that it doesn't accidentally get dropped.

>>

>> Tested on aarch64-linux-gnu (with and without SVE), aarch64_be-elf

>> and x86_64-linux-gnu.  OK to install?

>

> It looks somewhat clever but after looking for a while it doesn't handle

> simplifying

>

>  (IFN_COND_ADD @0 @1 (IFN_COND_SUB @0 @2 @1 @3) @3)

>

> to

>

>  (cond @0 @2 @3)

>

> right?  Because while the conditional gimple_match_op is built

> by try_conditional_simplification it isn't built when doing

> SSA use->def following in the generated matching code?


Right.  This would be easy to add, but there's no motivating case yet.

> So it looks like a bit much noise for this very special case?

>

> I suppose you ran into the need of these foldings from looking

> at real code - which foldings specifically were appearing here?

> Usually code is well optimized before if-conversion/vectorization

> so we shouldn't need full-blown handling?


It's needed to get the FMA, FMS, FNMA and FNMS folds for IFN_COND_* too.
I thought it'd be better to do it "automatically" rather than add specific
folds, since if we don't do it automatically now, it's going to end up
being a precedent for not doing it automatically in future either.

> That said, I'm not sure how much work it is to massage

>

>       if (gimple *def_stmt = get_def (valueize, op2))

>         {

>           if (gassign *def = dyn_cast <gassign *> (def_stmt))

>             switch (gimple_assign_rhs_code (def))

>               {

>               case PLUS_EXPR:

>

> to look like

>

>       if (gimple *def_stmt = get_def (valueize, op2))

>         {

>            code = ERROR_MARK;

>            if (!is_cond_ifn_with_cond (curr_gimple_match_op, &code))

>              if (gassign *def dyn_cast <gassign *> (def_stmt))

>                code = gimple_assign_rhs_code (def);

>            switch (code)

>              {

>              case PLUS_EXPR:

>

> thus transparently treat the IFN_COND_* as their "code" if the condition

> matches that of the context (I'm not sure if we can do anything for

> mismatching contexts).


Yeah, this was one approach I had in mind for the subnodes, if we do
end up needing it.  But at least for the top-level node, we want to try
both as a native IFN_COND_FOO and as a conditional FOO, which is why the
top-level case is handled directly in gimple-match-head.c.

Of course, trying both for subnodes would lead to exponential behaviour
in general.  And like you say, in practice most double-IFN_COND cases
should have been folded before we created the IFN_CONDs, so it's hard
to tell which recursive behaviour would be best.

Thanks,
Richard
Richard Biener June 7, 2018, 12:34 p.m. | #4
On Wed, Jun 6, 2018 at 10:16 PM Richard Sandiford
<richard.sandiford@linaro.org> wrote:
>

> > On Thu, May 24, 2018 at 11:36 AM Richard Sandiford

> > <richard.sandiford@linaro.org> wrote:

> >>

> >> This patch adds match.pd support for applying normal folds to their

> >> IFN_COND_* forms.  E.g. the rule:

> >>

> >>   (plus @0 (negate @1)) -> (minus @0 @1)

> >>

> >> also allows the fold:

> >>

> >>   (IFN_COND_ADD @0 @1 (negate @2) @3) -> (IFN_COND_SUB @0 @1 @2 @3)

> >>

> >> Actually doing this by direct matches in gimple-match.c would

> >> probably lead to combinatorial explosion, so instead, the patch

> >> makes gimple_match_op carry a condition under which the operation

> >> happens ("cond"), and the value to use when the condition is false

> >> ("else_value").  Thus in the example above we'd do the following

> >>

> >> (a) convert:

> >>

> >>       cond:NULL_TREE (IFN_COND_ADD @0 @1 @4 @3) else_value:NULL_TREE

> >>

> >>     to:

> >>

> >>       cond:@0 (plus @1 @4) else_value:@3

> >>

> >> (b) apply gimple_resimplify to (plus @1 @4)

> >>

> >> (c) reintroduce cond and else_value when constructing the result.

> >>

> >> Nested operations inherit the condition of the outer operation

> >> (so that we don't introduce extra faults) but have a null else_value.

> >> If we try to build such an operation, the target gets to choose what

> >> else_value it can handle efficiently: obvious choices include one of

> >> the operands or a zero constant.  (The alternative would be to have some

> >> representation for an undefined value, but that seems a bit invasive,

> >> and isn't likely to be useful here.)

> >>

> >> I've made the condition a mandatory part of the gimple_match_op

> >> constructor so that it doesn't accidentally get dropped.

> >>

> >> Tested on aarch64-linux-gnu (with and without SVE), aarch64_be-elf

> >> and x86_64-linux-gnu.  OK to install?

> >

> > It looks somewhat clever but after looking for a while it doesn't handle

> > simplifying

> >

> >  (IFN_COND_ADD @0 @1 (IFN_COND_SUB @0 @2 @1 @3) @3)

> >

> > to

> >

> >  (cond @0 @2 @3)

> >

> > right?  Because while the conditional gimple_match_op is built

> > by try_conditional_simplification it isn't built when doing

> > SSA use->def following in the generated matching code?

>

> Right.  This would be easy to add, but there's no motivating case yet.


...

> > So it looks like a bit much noise for this very special case?

> >

> > I suppose you ran into the need of these foldings from looking

> > at real code - which foldings specifically were appearing here?

> > Usually code is well optimized before if-conversion/vectorization

> > so we shouldn't need full-blown handling?

>

> It's needed to get the FMA, FMS, FNMA and FNMS folds for IFN_COND_* too.

> I thought it'd be better to do it "automatically" rather than add specific

> folds, since if we don't do it automatically now, it's going to end up

> being a precedent for not doing it automatically in future either.


... not like above isn't a similar precedent ;)  But OK, given...

> > That said, I'm not sure how much work it is to massage

> >

> >       if (gimple *def_stmt = get_def (valueize, op2))

> >         {

> >           if (gassign *def = dyn_cast <gassign *> (def_stmt))

> >             switch (gimple_assign_rhs_code (def))

> >               {

> >               case PLUS_EXPR:

> >

> > to look like

> >

> >       if (gimple *def_stmt = get_def (valueize, op2))

> >         {

> >            code = ERROR_MARK;

> >            if (!is_cond_ifn_with_cond (curr_gimple_match_op, &code))

> >              if (gassign *def dyn_cast <gassign *> (def_stmt))

> >                code = gimple_assign_rhs_code (def);

> >            switch (code)

> >              {

> >              case PLUS_EXPR:

> >

> > thus transparently treat the IFN_COND_* as their "code" if the condition

> > matches that of the context (I'm not sure if we can do anything for

> > mismatching contexts).

>

> Yeah, this was one approach I had in mind for the subnodes, if we do

> end up needing it.  But at least for the top-level node, we want to try

> both as a native IFN_COND_FOO and as a conditional FOO, which is why the

> top-level case is handled directly in gimple-match-head.c.

>

> Of course, trying both for subnodes would lead to exponential behaviour

> in general.  And like you say, in practice most double-IFN_COND cases

> should have been folded before we created the IFN_CONDs, so it's hard

> to tell which recursive behaviour would be best.


... this it probably makes sense to do it "simple" first.

Btw, I'd simply _only_ consider the IFN_ stripped path when looking for
defs if it has matching condition (which is a prerequesite anyway).  So
the get_at_the_def () would first check for IFN_ with matching condition
and then expand to the unconditional operation.  And get_at_the_def ()
for UNCOND context would never look into IFN_s.

Even the toplevel handling probably never will need the outermost
conditional IFN_ handling - at least I can't think of a pattern that
you would be forced to write the outermost conditional IFN_
explicitely?

So maybe modify your patch to never try both toplevel variants either.

Thanks,
Richard.


> Thanks,

> Richard

Patch

Index: gcc/target.def
===================================================================
--- gcc/target.def	2018-05-01 19:30:30.159632586 +0100
+++ gcc/target.def	2018-05-24 10:33:30.871095132 +0100
@@ -2040,6 +2040,25 @@  HOOK_VECTOR_END (vectorize)
 #define HOOK_PREFIX "TARGET_"
 
 DEFHOOK
+(preferred_else_value,
+ "This hook returns the target's preferred final argument for a call\n\
+to conditional internal function @var{ifn} (really of type\n\
+@code{internal_fn}).  @var{type} specifies the return type of the\n\
+function and @var{ops} are the operands to the conditional operation,\n\
+of which there are @var{nops}.\n\
+\n\
+For example, if @var{ifn} is @code{IFN_COND_ADD}, the hook returns\n\
+a value of type @var{type} that should be used when @samp{@var{ops}[0]}\n\
+and @samp{@var{ops}[1]} are conditionally added together.\n\
+\n\
+This hook is only relevant if the target supports conditional patterns\n\
+like @code{cond_add@var{m}}.  The default implementation returns a zero\n\
+constant of type @var{type}.",
+ tree,
+ (unsigned ifn, tree type, unsigned nops, tree *ops),
+ default_preferred_else_value)
+
+DEFHOOK
 (record_offload_symbol,
  "Used when offloaded functions are seen in the compilation unit and no named\n\
 sections are available.  It is called once for each symbol that must be\n\
Index: gcc/doc/tm.texi.in
===================================================================
--- gcc/doc/tm.texi.in	2018-05-01 19:30:28.730694873 +0100
+++ gcc/doc/tm.texi.in	2018-05-24 10:33:30.869095197 +0100
@@ -4149,6 +4149,8 @@  address;  but often a machine-dependent
 
 @hook TARGET_GOACC_REDUCTION
 
+@hook TARGET_PREFERRED_ELSE_VALUE
+
 @node Anchored Addresses
 @section Anchored Addresses
 @cindex anchored addresses
Index: gcc/doc/tm.texi
===================================================================
--- gcc/doc/tm.texi	2018-05-01 19:30:28.722695224 +0100
+++ gcc/doc/tm.texi	2018-05-24 10:33:30.868095229 +0100
@@ -6046,6 +6046,22 @@  expanded sequence has been inserted.  Th
 for allocating any storage for reductions when necessary.
 @end deftypefn
 
+@deftypefn {Target Hook} tree TARGET_PREFERRED_ELSE_VALUE (unsigned @var{ifn}, tree @var{type}, unsigned @var{nops}, tree *@var{ops})
+This hook returns the target's preferred final argument for a call
+to conditional internal function @var{ifn} (really of type
+@code{internal_fn}).  @var{type} specifies the return type of the
+function and @var{ops} are the operands to the conditional operation,
+of which there are @var{nops}.
+
+For example, if @var{ifn} is @code{IFN_COND_ADD}, the hook returns
+a value of type @var{type} that should be used when @samp{@var{ops}[0]}
+and @samp{@var{ops}[1]} are conditionally added together.
+
+This hook is only relevant if the target supports conditional patterns
+like @code{cond_add@var{m}}.  The default implementation returns a zero
+constant of type @var{type}.
+@end deftypefn
+
 @node Anchored Addresses
 @section Anchored Addresses
 @cindex anchored addresses
Index: gcc/targhooks.h
===================================================================
--- gcc/targhooks.h	2018-05-01 19:30:29.390666052 +0100
+++ gcc/targhooks.h	2018-05-24 10:33:30.872095099 +0100
@@ -289,5 +289,6 @@  extern unsigned int default_min_arithmet
 default_excess_precision (enum excess_precision_type ATTRIBUTE_UNUSED);
 extern bool default_stack_clash_protection_final_dynamic_probe (rtx);
 extern void default_select_early_remat_modes (sbitmap);
+extern tree default_preferred_else_value (unsigned, tree, unsigned, tree *);
 
 #endif /* GCC_TARGHOOKS_H */
Index: gcc/targhooks.c
===================================================================
--- gcc/targhooks.c	2018-05-01 19:30:29.390666052 +0100
+++ gcc/targhooks.c	2018-05-24 10:33:30.871095132 +0100
@@ -2345,4 +2345,12 @@  default_select_early_remat_modes (sbitma
 {
 }
 
+/* The default implementation of TARGET_PREFERRED_ELSE_VALUE.  */
+
+tree
+default_preferred_else_value (unsigned, tree type, unsigned, tree *)
+{
+  return build_zero_cst (type);
+}
+
 #include "gt-targhooks.h"
Index: gcc/internal-fn.h
===================================================================
--- gcc/internal-fn.h	2018-05-17 11:52:13.507173989 +0100
+++ gcc/internal-fn.h	2018-05-24 10:33:30.870095164 +0100
@@ -193,6 +193,7 @@  direct_internal_fn_supported_p (internal
 extern bool set_edom_supported_p (void);
 
 extern internal_fn get_conditional_internal_fn (tree_code);
+extern tree_code conditional_internal_fn_code (internal_fn);
 
 extern bool internal_load_fn_p (internal_fn);
 extern bool internal_store_fn_p (internal_fn);
Index: gcc/internal-fn.c
===================================================================
--- gcc/internal-fn.c	2018-05-24 10:12:10.146352152 +0100
+++ gcc/internal-fn.c	2018-05-24 10:33:30.870095164 +0100
@@ -3219,6 +3219,21 @@  #define DEF_INTERNAL_FN(CODE, FLAGS, FNS
   0
 };
 
+/* Invoke T(CODE, IFN) for each conditional function IFN that maps to a
+   tree code CODE.  */
+#define FOR_EACH_CODE_MAPPING(T) \
+  T (PLUS_EXPR, IFN_COND_ADD) \
+  T (MINUS_EXPR, IFN_COND_SUB) \
+  T (MULT_EXPR, IFN_COND_MUL) \
+  T (TRUNC_DIV_EXPR, IFN_COND_DIV) \
+  T (TRUNC_MOD_EXPR, IFN_COND_MOD) \
+  T (RDIV_EXPR, IFN_COND_RDIV) \
+  T (MIN_EXPR, IFN_COND_MIN) \
+  T (MAX_EXPR, IFN_COND_MAX) \
+  T (BIT_AND_EXPR, IFN_COND_AND) \
+  T (BIT_IOR_EXPR, IFN_COND_IOR) \
+  T (BIT_XOR_EXPR, IFN_COND_XOR)
+
 /* Return a function that only performs CODE when a certain condition is met
    and that uses a given fallback value otherwise.  For example, if CODE is
    a binary operation associated with conditional function FN:
@@ -3238,31 +3253,30 @@  get_conditional_internal_fn (tree_code c
 {
   switch (code)
     {
-    case PLUS_EXPR:
-      return IFN_COND_ADD;
-    case MINUS_EXPR:
-      return IFN_COND_SUB;
-    case MIN_EXPR:
-      return IFN_COND_MIN;
-    case MAX_EXPR:
-      return IFN_COND_MAX;
-    case TRUNC_DIV_EXPR:
-      return IFN_COND_DIV;
-    case TRUNC_MOD_EXPR:
-      return IFN_COND_MOD;
-    case RDIV_EXPR:
-      return IFN_COND_RDIV;
-    case BIT_AND_EXPR:
-      return IFN_COND_AND;
-    case BIT_IOR_EXPR:
-      return IFN_COND_IOR;
-    case BIT_XOR_EXPR:
-      return IFN_COND_XOR;
+#define CASE(CODE, IFN) case CODE: return IFN;
+      FOR_EACH_CODE_MAPPING(CASE)
+#undef CASE
     default:
       return IFN_LAST;
     }
 }
 
+/* If IFN implements the conditional form of a tree code, return that
+   tree code, otherwise return ERROR_MARK.  */
+
+tree_code
+conditional_internal_fn_code (internal_fn ifn)
+{
+  switch (ifn)
+    {
+#define CASE(CODE, IFN) case IFN: return CODE;
+      FOR_EACH_CODE_MAPPING(CASE)
+#undef CASE
+    default:
+      return ERROR_MARK;
+    }
+}
+
 /* Return true if IFN is some form of load from memory.  */
 
 bool
Index: gcc/gimple-match.h
===================================================================
--- gcc/gimple-match.h	2018-05-24 09:54:37.509451356 +0100
+++ gcc/gimple-match.h	2018-05-24 10:33:30.870095164 +0100
@@ -40,16 +40,57 @@  #define GCC_GIMPLE_MATCH_H
   int rep;
 };
 
+/* Represents the condition under which an operation should happen,
+   and the value to use otherwise.  The condition applies elementwise
+   (as for VEC_COND_EXPR) if the values are vectors.  */
+struct gimple_match_cond
+{
+  enum uncond { UNCOND };
+
+  /* Build an unconditional op.  */
+  gimple_match_cond (uncond) : cond (NULL_TREE), else_value (NULL_TREE) {}
+  gimple_match_cond (tree, tree);
+
+  gimple_match_cond any_else () const;
+
+  /* The condition under which the operation occurs, or NULL_TREE
+     if the operation is unconditional.  */
+  tree cond;
+
+  /* The value to use when the condition is false.  This is NULL_TREE if
+     the operation is unconditional or if the value doesn't matter.  */
+  tree else_value;
+};
+
+inline
+gimple_match_cond::gimple_match_cond (tree cond_in, tree else_value_in)
+  : cond (cond_in), else_value (else_value_in)
+{
+}
+
+/* Return a gimple_match_cond with the same condition but with an
+   arbitrary ELSE_VALUE.  */
+
+inline gimple_match_cond
+gimple_match_cond::any_else () const
+{
+  return gimple_match_cond (cond, NULL_TREE);
+}
+
 /* Represents an operation to be simplified, or the result of the
    simplification.  */
 struct gimple_match_op
 {
-  gimple_match_op () : type (NULL_TREE), num_ops (0) {}
-  gimple_match_op (code_helper, tree, unsigned int);
-  gimple_match_op (code_helper, tree, tree);
-  gimple_match_op (code_helper, tree, tree, tree);
-  gimple_match_op (code_helper, tree, tree, tree, tree);
-  gimple_match_op (code_helper, tree, tree, tree, tree, tree);
+  gimple_match_op ();
+  gimple_match_op (const gimple_match_cond &, code_helper, tree, unsigned int);
+  gimple_match_op (const gimple_match_cond &,
+		   code_helper, tree, tree);
+  gimple_match_op (const gimple_match_cond &,
+		   code_helper, tree, tree, tree);
+  gimple_match_op (const gimple_match_cond &,
+		   code_helper, tree, tree, tree, tree);
+  gimple_match_op (const gimple_match_cond &,
+		   code_helper, tree, tree, tree, tree, tree);
 
   void set_op (code_helper, tree, unsigned int);
   void set_op (code_helper, tree, tree);
@@ -63,6 +104,10 @@  struct gimple_match_op
   /* The maximum value of NUM_OPS.  */
   static const unsigned int MAX_NUM_OPS = 4;
 
+  /* The conditions under which the operation is performed, and the value to
+     use as a fallback.  */
+  gimple_match_cond cond;
+
   /* The operation being performed.  */
   code_helper code;
 
@@ -76,39 +121,49 @@  struct gimple_match_op
   tree ops[MAX_NUM_OPS];
 };
 
-/* Constructor that takes the code, type and number of operands, but leaves
-   the caller to fill in the operands.  */
+inline
+gimple_match_op::gimple_match_op ()
+  : cond (gimple_match_cond::UNCOND), type (NULL_TREE), num_ops (0)
+{
+}
+
+/* Constructor that takes the condition, code, type and number of
+   operands, but leaves the caller to fill in the operands.  */
 
 inline
-gimple_match_op::gimple_match_op (code_helper code_in, tree type_in,
+gimple_match_op::gimple_match_op (const gimple_match_cond &cond_in,
+				  code_helper code_in, tree type_in,
 				  unsigned int num_ops_in)
-  : code (code_in), type (type_in), num_ops (num_ops_in)
+  : cond (cond_in), code (code_in), type (type_in), num_ops (num_ops_in)
 {
 }
 
 /* Constructors for various numbers of operands.  */
 
 inline
-gimple_match_op::gimple_match_op (code_helper code_in, tree type_in,
+gimple_match_op::gimple_match_op (const gimple_match_cond &cond_in,
+				  code_helper code_in, tree type_in,
 				  tree op0)
-  : code (code_in), type (type_in), num_ops (1)
+  : cond (cond_in), code (code_in), type (type_in), num_ops (1)
 {
   ops[0] = op0;
 }
 
 inline
-gimple_match_op::gimple_match_op (code_helper code_in, tree type_in,
+gimple_match_op::gimple_match_op (const gimple_match_cond &cond_in,
+				  code_helper code_in, tree type_in,
 				  tree op0, tree op1)
-  : code (code_in), type (type_in), num_ops (2)
+  : cond (cond_in), code (code_in), type (type_in), num_ops (2)
 {
   ops[0] = op0;
   ops[1] = op1;
 }
 
 inline
-gimple_match_op::gimple_match_op (code_helper code_in, tree type_in,
+gimple_match_op::gimple_match_op (const gimple_match_cond &cond_in,
+				  code_helper code_in, tree type_in,
 				  tree op0, tree op1, tree op2)
-  : code (code_in), type (type_in), num_ops (3)
+  : cond (cond_in), code (code_in), type (type_in), num_ops (3)
 {
   ops[0] = op0;
   ops[1] = op1;
@@ -116,9 +171,10 @@  gimple_match_op::gimple_match_op (code_h
 }
 
 inline
-gimple_match_op::gimple_match_op (code_helper code_in, tree type_in,
+gimple_match_op::gimple_match_op (const gimple_match_cond &cond_in,
+				  code_helper code_in, tree type_in,
 				  tree op0, tree op1, tree op2, tree op3)
-  : code (code_in), type (type_in), num_ops (4)
+  : cond (cond_in), code (code_in), type (type_in), num_ops (4)
 {
   ops[0] = op0;
   ops[1] = op1;
Index: gcc/genmatch.c
===================================================================
--- gcc/genmatch.c	2018-05-24 10:12:10.145352193 +0100
+++ gcc/genmatch.c	2018-05-24 10:33:30.869095197 +0100
@@ -2507,8 +2507,8 @@  expr::gen_transform (FILE *f, int indent
       /* ???  Building a stmt can fail for various reasons here, seq being
          NULL or the stmt referencing SSA names occuring in abnormal PHIs.
 	 So if we fail here we should continue matching other patterns.  */
-      fprintf_indent (f, indent, "gimple_match_op tem_op (%s, %s",
-		      opr_name, type);
+      fprintf_indent (f, indent, "gimple_match_op tem_op "
+		      "(res_op->cond.any_else (), %s, %s", opr_name, type);
       for (unsigned i = 0; i < ops.length (); ++i)
 	fprintf (f, ", ops%d[%u]", depth, i);
       fprintf (f, ");\n");
Index: gcc/tree-ssa-sccvn.c
===================================================================
--- gcc/tree-ssa-sccvn.c	2018-05-24 09:02:28.765328358 +0100
+++ gcc/tree-ssa-sccvn.c	2018-05-24 10:33:30.872095099 +0100
@@ -1804,7 +1804,8 @@  vn_nary_simplify (vn_nary_op_t nary)
 {
   if (nary->length > gimple_match_op::MAX_NUM_OPS)
     return NULL_TREE;
-  gimple_match_op op (nary->opcode, nary->type, nary->length);
+  gimple_match_op op (gimple_match_cond::UNCOND, nary->opcode,
+		      nary->type, nary->length);
   memcpy (op.ops, nary->op, sizeof (tree) * nary->length);
   return vn_nary_build_or_lookup_1 (&op, false);
 }
@@ -2031,8 +2032,8 @@  vn_reference_lookup_3 (ao_ref *ref, tree
 	  else if (INTEGRAL_TYPE_P (vr->type)
 		   && known_eq (ref->size, 8))
 	    {
-	      gimple_match_op res_op (NOP_EXPR, vr->type,
-				      gimple_call_arg (def_stmt, 1));
+	      gimple_match_op res_op (gimple_match_cond::UNCOND, NOP_EXPR,
+				      vr->type, gimple_call_arg (def_stmt, 1));
 	      val = vn_nary_build_or_lookup (&res_op);
 	      if (!val
 		  || (TREE_CODE (val) == SSA_NAME
@@ -2172,7 +2173,8 @@  vn_reference_lookup_3 (ao_ref *ref, tree
 	      || known_eq (ref->size, TYPE_PRECISION (vr->type)))
 	  && multiple_p (ref->size, BITS_PER_UNIT))
 	{
-	  gimple_match_op op (BIT_FIELD_REF, vr->type,
+	  gimple_match_op op (gimple_match_cond::UNCOND,
+			      BIT_FIELD_REF, vr->type,
 			      SSA_VAL (gimple_assign_rhs1 (def_stmt)),
 			      bitsize_int (ref->size),
 			      bitsize_int (offset - offset2));
@@ -3701,7 +3703,8 @@  visit_nary_op (tree lhs, gassign *stmt)
 		      unsigned rhs_prec = TYPE_PRECISION (TREE_TYPE (rhs1));
 		      if (lhs_prec == rhs_prec)
 			{
-			  gimple_match_op match_op (NOP_EXPR, type, ops[0]);
+			  gimple_match_op match_op (gimple_match_cond::UNCOND,
+						    NOP_EXPR, type, ops[0]);
 			  result = vn_nary_build_or_lookup (&match_op);
 			  if (result)
 			    {
@@ -3714,7 +3717,8 @@  visit_nary_op (tree lhs, gassign *stmt)
 			{
 			  tree mask = wide_int_to_tree
 			    (type, wi::mask (rhs_prec, false, lhs_prec));
-			  gimple_match_op match_op (BIT_AND_EXPR,
+			  gimple_match_op match_op (gimple_match_cond::UNCOND,
+						    BIT_AND_EXPR,
 						    TREE_TYPE (lhs),
 						    ops[0], mask);
 			  result = vn_nary_build_or_lookup (&match_op);
@@ -3838,7 +3842,8 @@  visit_reference_op_load (tree lhs, tree
 	 of VIEW_CONVERT_EXPR <TREE_TYPE (result)> (result).
 	 So first simplify and lookup this expression to see if it
 	 is already available.  */
-      gimple_match_op res_op (VIEW_CONVERT_EXPR, TREE_TYPE (op), result);
+      gimple_match_op res_op (gimple_match_cond::UNCOND,
+			      VIEW_CONVERT_EXPR, TREE_TYPE (op), result);
       result = vn_nary_build_or_lookup (&res_op);
     }
 
Index: gcc/gimple-match-head.c
===================================================================
--- gcc/gimple-match-head.c	2018-05-24 09:54:37.509451356 +0100
+++ gcc/gimple-match-head.c	2018-05-24 10:33:30.870095164 +0100
@@ -40,6 +40,7 @@  Software Foundation; either version 3, o
 #include "case-cfn-macros.h"
 #include "gimplify.h"
 #include "optabs-tree.h"
+#include "tree-eh.h"
 
 
 /* Forward declarations of the private auto-generated matchers.
@@ -68,6 +69,95 @@  constant_for_folding (tree t)
 	      && TREE_CODE (TREE_OPERAND (t, 0)) == STRING_CST));
 }
 
+/* Try to convert conditional operation ORIG_OP into an IFN_COND_*
+   operation.  Return true on success, storing the new operation in NEW_OP.  */
+
+static bool
+convert_conditional_op (gimple_match_op *orig_op,
+			gimple_match_op *new_op)
+{
+  internal_fn ifn;
+  if (orig_op->code.is_tree_code ())
+    ifn = get_conditional_internal_fn ((tree_code) orig_op->code);
+  else
+    return false;
+  if (ifn == IFN_LAST)
+    return false;
+  unsigned int num_ops = orig_op->num_ops;
+  new_op->set_op (as_combined_fn (ifn), orig_op->type, num_ops + 2);
+  new_op->ops[0] = orig_op->cond.cond;
+  for (unsigned int i = 0; i < num_ops; ++i)
+    new_op->ops[i + 1] = orig_op->ops[i];
+  tree else_value = orig_op->cond.else_value;
+  if (!else_value)
+    else_value = targetm.preferred_else_value (ifn, orig_op->type,
+					       num_ops, orig_op->ops);
+  new_op->ops[num_ops + 1] = else_value;
+  return true;
+}
+
+/* RES_OP is the result of a simplification.  If it is conditional,
+   try to replace it with the equivalent UNCOND form, such as an
+   IFN_COND_* call or a VEC_COND_EXPR.  Also try to resimplify the
+   result of the replacement if appropriate, adding any new statements to
+   SEQ and using VALUEIZE as the valueization function.  Return true if
+   this resimplification occurred and resulted in at least one change.  */
+
+static bool
+maybe_resimplify_conditional_op (gimple_seq *seq, gimple_match_op *res_op,
+				 tree (*valueize) (tree))
+{
+  if (!res_op->cond.cond)
+    return false;
+
+  if (!res_op->cond.else_value
+      && res_op->code.is_tree_code ())
+    {
+      /* The "else" value doesn't matter.  If the "then" value is a
+	 gimple value, just use it unconditionally.  This isn't a
+	 simplification in itself, since there was no operation to
+	 build in the first place.  */
+      if (gimple_simplified_result_is_gimple_val (res_op))
+	{
+	  res_op->cond.cond = NULL_TREE;
+	  return false;
+	}
+
+      /* Likewise if the operation would not trap.  */
+      bool honor_trapv = (INTEGRAL_TYPE_P (res_op->type)
+			  && TYPE_OVERFLOW_TRAPS (res_op->type));
+      if (!operation_could_trap_p ((tree_code) res_op->code,
+				   FLOAT_TYPE_P (res_op->type),
+				   honor_trapv, res_op->op_or_null (1)))
+	{
+	  res_op->cond.cond = NULL_TREE;
+	  return false;
+	}
+    }
+
+  /* If the "then" value is a gimple value and the "else" value matters,
+     create a VEC_COND_EXPR between them, then see if it can be further
+     simplified.  */
+  gimple_match_op new_op;
+  if (res_op->cond.else_value
+      && VECTOR_TYPE_P (res_op->type)
+      && gimple_simplified_result_is_gimple_val (res_op))
+    {
+      new_op.set_op (VEC_COND_EXPR, res_op->type,
+		     res_op->cond.cond, res_op->ops[0],
+		     res_op->cond.else_value);
+      *res_op = new_op;
+      return gimple_resimplify3 (seq, res_op, valueize);
+    }
+
+  /* Otherwise try rewriting the operation as an IFN_COND_* call.
+     Again, this isn't a simplification in itself, since it's what
+     RES_OP already described.  */
+  if (convert_conditional_op (res_op, &new_op))
+    *res_op = new_op;
+
+  return false;
+}
 
 /* Helper that matches and simplifies the toplevel result from
    a gimple_simplify run (where we don't want to build
@@ -93,6 +183,7 @@  gimple_resimplify1 (gimple_seq *seq, gim
 	  if (TREE_OVERFLOW_P (tem))
 	    tem = drop_tree_overflow (tem);
 	  res_op->set_value (tem);
+	  maybe_resimplify_conditional_op (seq, res_op, valueize);
 	  return true;
 	}
     }
@@ -105,6 +196,9 @@  gimple_resimplify1 (gimple_seq *seq, gim
       return true;
     }
 
+  if (maybe_resimplify_conditional_op (seq, res_op, valueize))
+    return true;
+
   return false;
 }
 
@@ -134,6 +228,7 @@  gimple_resimplify2 (gimple_seq *seq, gim
 	  if (TREE_OVERFLOW_P (tem))
 	    tem = drop_tree_overflow (tem);
 	  res_op->set_value (tem);
+	  maybe_resimplify_conditional_op (seq, res_op, valueize);
 	  return true;
 	}
     }
@@ -160,6 +255,9 @@  gimple_resimplify2 (gimple_seq *seq, gim
       return true;
     }
 
+  if (maybe_resimplify_conditional_op (seq, res_op, valueize))
+    return true;
+
   return canonicalized;
 }
 
@@ -191,6 +289,7 @@  gimple_resimplify3 (gimple_seq *seq, gim
 	  if (TREE_OVERFLOW_P (tem))
 	    tem = drop_tree_overflow (tem);
 	  res_op->set_value (tem);
+	  maybe_resimplify_conditional_op (seq, res_op, valueize);
 	  return true;
 	}
     }
@@ -214,6 +313,9 @@  gimple_resimplify3 (gimple_seq *seq, gim
       return true;
     }
 
+  if (maybe_resimplify_conditional_op (seq, res_op, valueize))
+    return true;
+
   return canonicalized;
 }
 
@@ -239,6 +341,9 @@  gimple_resimplify4 (gimple_seq *seq, gim
       return true;
     }
 
+  if (maybe_resimplify_conditional_op (seq, res_op, valueize))
+    return true;
+
   return false;
 }
 
@@ -297,6 +402,12 @@  maybe_push_res_to_seq (gimple_match_op *
   tree *ops = res_op->ops;
   unsigned num_ops = res_op->num_ops;
 
+  /* The caller should have converted conditional operations into an UNCOND
+     form and resimplified as appropriate.  The conditional form only
+     survives this far if that conversion failed.  */
+  if (res_op->cond.cond)
+    return NULL_TREE;
+
   if (res_op->code.is_tree_code ())
     {
       if (!res
@@ -558,6 +669,50 @@  do_valueize (tree op, tree (*valueize)(t
   return op;
 }
 
+/* If RES_OP is a call to a conditional internal function, try simplifying
+   the associated unconditional operation and using the result to build
+   a new conditional operation.  For example, if RES_OP is:
+
+     IFN_COND_ADD (COND, A, B, ELSE)
+
+   try simplifying (plus A B) and using the result to build a replacement
+   for the whole IFN_COND_ADD.
+
+   Return true if this approach led to a simplification, otherwise leave
+   RES_OP unchanged (and so suitable for other simplifications).  When
+   returning true, add any new statements to SEQ and use VALUEIZE as the
+   valueization function.
+
+   RES_OP is known to be a call to IFN.  */
+
+static bool
+try_conditional_simplification (internal_fn ifn, gimple_match_op *res_op,
+				gimple_seq *seq, tree (*valueize) (tree))
+{
+  tree_code code = conditional_internal_fn_code (ifn);
+  if (code == ERROR_MARK)
+    return false;
+
+  unsigned int num_ops = res_op->num_ops;
+  gimple_match_op cond_op (gimple_match_cond (res_op->ops[0],
+					      res_op->ops[num_ops - 1]),
+			   code, res_op->type, num_ops - 2);
+  for (unsigned int i = 1; i < num_ops - 1; ++i)
+    cond_op.ops[i - 1] = res_op->ops[i];
+  switch (num_ops - 2)
+    {
+    case 2:
+      if (!gimple_resimplify2 (seq, &cond_op, valueize))
+	return false;
+      break;
+    default:
+      gcc_unreachable ();
+    }
+  *res_op = cond_op;
+  maybe_resimplify_conditional_op (seq, res_op, valueize);
+  return true;
+}
+
 /* The main STMT based simplification entry.  It is used by the fold_stmt
    and the fold_stmt_to_constant APIs.  */
 
@@ -643,7 +798,7 @@  gimple_simplify (gimple *stmt, gimple_ma
 		      tree rhs = TREE_OPERAND (rhs1, 1);
 		      lhs = do_valueize (lhs, top_valueize, valueized);
 		      rhs = do_valueize (rhs, top_valueize, valueized);
-		      gimple_match_op res_op2 (TREE_CODE (rhs1),
+		      gimple_match_op res_op2 (res_op->cond, TREE_CODE (rhs1),
 					       TREE_TYPE (rhs1), lhs, rhs);
 		      if ((gimple_resimplify2 (seq, &res_op2, valueize)
 			   || valueized)
@@ -714,6 +869,10 @@  gimple_simplify (gimple *stmt, gimple_ma
 	      tree arg = gimple_call_arg (stmt, i);
 	      res_op->ops[i] = do_valueize (arg, top_valueize, valueized);
 	    }
+	  if (internal_fn_p (cfn)
+	      && try_conditional_simplification (as_internal_fn (cfn),
+						 res_op, seq, valueize))
+	    return true;
 	  switch (num_args)
 	    {
 	    case 1:
Index: gcc/match.pd
===================================================================
--- gcc/match.pd	2018-05-24 10:12:10.146352152 +0100
+++ gcc/match.pd	2018-05-24 10:33:30.870095164 +0100
@@ -4797,3 +4797,12 @@  DEFINE_INT_AND_FLOAT_ROUND_FN (RINT)
   (with { tree op_type = TREE_TYPE (@4); }
    (if (element_precision (type) == element_precision (op_type))
     (view_convert (cond_op (bit_not @0) @2 @3 (view_convert:op_type @1)))))))
+
+/* Detect cases in which a VEC_COND_EXPR effectively replaces the
+   "else" value of an IFN_COND_*.  */
+(for cond_op (COND_BINARY)
+ (simplify
+  (vec_cond @0 (view_convert? (cond_op @0 @1 @2 @3)) @4)
+  (with { tree op_type = TREE_TYPE (@3); }
+   (if (element_precision (type) == element_precision (op_type))
+    (view_convert (cond_op @0 @1 @2 (view_convert:op_type @4)))))))
Index: gcc/config/aarch64/aarch64.c
===================================================================
--- gcc/config/aarch64/aarch64.c	2018-05-24 09:54:37.507451418 +0100
+++ gcc/config/aarch64/aarch64.c	2018-05-24 10:33:30.867095262 +0100
@@ -1292,6 +1292,16 @@  aarch64_get_mask_mode (poly_uint64 nunit
   return default_get_mask_mode (nunits, nbytes);
 }
 
+/* Implement TARGET_PREFERRED_ELSE_VALUE.  Prefer to use the first
+   arithmetic operand as the else value if the else value doesn't matter,
+   since that exactly matches the SVE destructive merging form.  */
+
+static tree
+aarch64_preferred_else_value (unsigned, tree, unsigned int, tree *ops)
+{
+  return ops[0];
+}
+
 /* Implement TARGET_HARD_REGNO_NREGS.  */
 
 static unsigned int
@@ -17980,6 +17990,9 @@  #define TARGET_VECTORIZE_GET_MASK_MODE a
 #undef TARGET_VECTORIZE_EMPTY_MASK_IS_EXPENSIVE
 #define TARGET_VECTORIZE_EMPTY_MASK_IS_EXPENSIVE \
   aarch64_empty_mask_is_expensive
+#undef TARGET_PREFERRED_ELSE_VALUE
+#define TARGET_PREFERRED_ELSE_VALUE \
+  aarch64_preferred_else_value
 
 #undef TARGET_INIT_LIBFUNCS
 #define TARGET_INIT_LIBFUNCS aarch64_init_libfuncs
Index: gcc/testsuite/gcc.dg/vect/vect-cond-arith-2.c
===================================================================
--- /dev/null	2018-04-20 16:19:46.369131350 +0100
+++ gcc/testsuite/gcc.dg/vect/vect-cond-arith-2.c	2018-05-24 10:33:30.872095099 +0100
@@ -0,0 +1,45 @@ 
+/* { dg-do compile } */
+/* { dg-additional-options "-fgimple -fdump-tree-optimized -ffast-math" } */
+
+double __GIMPLE (startwith("loop"))
+neg_xi (double *x)
+{
+  int i;
+  long unsigned int index;
+  long unsigned int offset;
+  double * xi_ptr;
+  double xi;
+  double neg_xi;
+  double res;
+  unsigned int ivtmp;
+
+ bb_1:
+  goto bb_2;
+
+ bb_2:
+  res_1 = __PHI (bb_1: 0.0, bb_3: res_2);
+  i_4 = __PHI (bb_1: 0, bb_3: i_5);
+  ivtmp_6 = __PHI (bb_1: 100U, bb_3: ivtmp_7);
+  index = (long unsigned int) i_4;
+  offset = index * 8UL;
+  xi_ptr = x_8(D) + offset;
+  xi = *xi_ptr;
+  neg_xi = -xi;
+  res_2 = neg_xi + res_1;
+  i_5 = i_4 + 1;
+  ivtmp_7 = ivtmp_6 - 1U;
+  if (ivtmp_7 != 0U)
+    goto bb_3;
+  else
+    goto bb_4;
+
+ bb_3:
+  goto bb_2;
+
+ bb_4:
+  res_3 = __PHI (bb_2: res_2);
+  return res_3;
+}
+
+/* { dg-final { scan-tree-dump { = \.COND_ADD} "vect" { target { vect_double_cond_arith && vect_fully_masked } } } } */
+/* { dg-final { scan-tree-dump { = \.COND_SUB} "optimized" { target { vect_double_cond_arith && vect_fully_masked } } } } */
Index: gcc/testsuite/gcc.target/aarch64/sve/loop_add_6.c
===================================================================
--- /dev/null	2018-04-20 16:19:46.369131350 +0100
+++ gcc/testsuite/gcc.target/aarch64/sve/loop_add_6.c	2018-05-24 10:33:30.872095099 +0100
@@ -0,0 +1,46 @@ 
+/* { dg-do compile } */
+/* { dg-options "-O2 -ftree-vectorize -fgimple -ffast-math" } */
+
+double __GIMPLE (startwith("loop"))
+neg_xi (double *x)
+{
+  int i;
+  long unsigned int index;
+  long unsigned int offset;
+  double * xi_ptr;
+  double xi;
+  double neg_xi;
+  double res;
+  unsigned int ivtmp;
+
+ bb_1:
+  goto bb_2;
+
+ bb_2:
+  res_1 = __PHI (bb_1: 0.0, bb_3: res_2);
+  i_4 = __PHI (bb_1: 0, bb_3: i_5);
+  ivtmp_6 = __PHI (bb_1: 100U, bb_3: ivtmp_7);
+  index = (long unsigned int) i_4;
+  offset = index * 8UL;
+  xi_ptr = x_8(D) + offset;
+  xi = *xi_ptr;
+  neg_xi = -xi;
+  res_2 = neg_xi + res_1;
+  i_5 = i_4 + 1;
+  ivtmp_7 = ivtmp_6 - 1U;
+  if (ivtmp_7 != 0U)
+    goto bb_3;
+  else
+    goto bb_4;
+
+ bb_3:
+  goto bb_2;
+
+ bb_4:
+  res_3 = __PHI (bb_2: res_2);
+  return res_3;
+}
+
+/* { dg-final { scan-assembler {\tfsub\tz[0-9]+\.d, p[0-7]/m} } } */
+/* { dg-final { scan-assembler-not {\tsel\t} } } */
+/* { dg-final { scan-assembler-not {\tmovprfx\t} } } */