From patchwork Wed Aug 16 13:38:27 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Sandiford X-Patchwork-Id: 110247 Delivered-To: patch@linaro.org Received: by 10.182.109.195 with SMTP id hu3csp853694obb; Wed, 16 Aug 2017 06:38:57 -0700 (PDT) X-Received: by 10.84.232.76 with SMTP id f12mr1874619pln.249.1502890737894; Wed, 16 Aug 2017 06:38:57 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1502890737; cv=none; d=google.com; s=arc-20160816; b=B+SlAFwj9fcmg5xc8D4WrewQrujdyNBF4LfkLZ9Ium9aoi9NXDjvJP/3Lo8LJWDPjk 5kGWK12I7pvgkr6R6/jKelIr6ZcI1QZzKPg2Zfa9wc6dWLOqKlF+dUm1MRG4ezoRB9Uv TyQPwy+muFiu/g0KW0PyJSEWHdppMBAQf2bq/RN+eyVP8C58te4MQTlsGMs++r4X5g53 r1vQKkc4Qyjmj8nOWO+DzSylJtsEJJ8ygHNvtya48lzW5pC5YI5MWR04T6HExakqmE83 u++GT3oY61g0IXctORJbZ7L24I2vYevtHjm686JtwkX8SVr0K8uHMZl8mOdO/OIbDHlB xnTA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:user-agent:message-id:date:subject:mail-followup-to:to :from:delivered-to:sender:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:mailing-list:dkim-signature :domainkey-signature:arc-authentication-results; bh=8yGpOPBs/g5snVGNELpQF4b5pLnxAn7oiyVl4C85q7k=; b=gCPo+wQnDjt5AvO7YP0vVI2XxA4MXieihvKtcovd2N3DKjTPbJXYGnY9BEtIeS1aB9 KFFQxReAaNMFvCKT8D3FolW/o1GzN2BHTZ5CGSxw77it+XwrwJM2rA1ZZ5pZRncEPNw1 CpDETwY+TBdeYH22b9d/G3fr277ujUijAerr/Dfsq04Zp2QaAGG8Lbgrukbkkxq8qiZg u5tnpVFLJ+OLbzq2GpJbBvlPPxO+B8SIkJTCieFKGK6ZFkPuSXhOhwhXHpaplim1Fpyc AQUZOEnFbpQPoLA6Cq7I6SXRMeUj6GKyKTknHCpvSXaCWZU9/gWxpzaASgBLKBwt8zu5 XpHw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gcc.gnu.org header.s=default header.b=GLV6fkFz; spf=pass (google.com: domain of gcc-patches-return-460433-patch=linaro.org@gcc.gnu.org designates 209.132.180.131 as permitted sender) smtp.mailfrom=gcc-patches-return-460433-patch=linaro.org@gcc.gnu.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from sourceware.org (server1.sourceware.org. [209.132.180.131]) by mx.google.com with ESMTPS id s67si558358pfe.97.2017.08.16.06.38.57 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 16 Aug 2017 06:38:57 -0700 (PDT) Received-SPF: pass (google.com: domain of gcc-patches-return-460433-patch=linaro.org@gcc.gnu.org designates 209.132.180.131 as permitted sender) client-ip=209.132.180.131; Authentication-Results: mx.google.com; dkim=pass header.i=@gcc.gnu.org header.s=default header.b=GLV6fkFz; spf=pass (google.com: domain of gcc-patches-return-460433-patch=linaro.org@gcc.gnu.org designates 209.132.180.131 as permitted sender) smtp.mailfrom=gcc-patches-return-460433-patch=linaro.org@gcc.gnu.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org DomainKey-Signature: a=rsa-sha1; c=nofws; d=gcc.gnu.org; h=list-id :list-unsubscribe:list-archive:list-post:list-help:sender:from :to:subject:date:message-id:mime-version:content-type; q=dns; s= default; b=AVBmww/9fmxVSaULB9uUn3MqO593MpHn0xxBcrCUqQmedX7xa5CPr fdWfBw9VMvb38uL3nFkt7aVfKPQhA7VSqYJ5Ye2pn5hEPywRIBP4oW93/yIznhEr Q+oUMQUW4/NI5qa4F88u8cwOa5h4cOE2f3C3LT8RNpsyImvA2EoLpk= DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=gcc.gnu.org; h=list-id :list-unsubscribe:list-archive:list-post:list-help:sender:from :to:subject:date:message-id:mime-version:content-type; s= default; bh=YvT6e0HJ8Si4S62gw/Crslun6kg=; b=GLV6fkFzQKFCaRm6AV8N JU5SBIzBzPjZwr/s6XpxLXHJfUqaKKPWYFhyNTxlbp1NFvHHG3EpnqTMzQnW+rIN uZMjA4ZRccI34hwvegFDLcGKAVtn2Jv0G6tmO+RMBpurV+IUVkuMt3sL7wTAZuBN Q4EOYb63ab1OXHGSVhNc4GE= Received: (qmail 94614 invoked by alias); 16 Aug 2017 13:38:41 -0000 Mailing-List: contact gcc-patches-help@gcc.gnu.org; run by ezmlm Precedence: bulk List-Id: List-Unsubscribe: List-Archive: List-Post: List-Help: Sender: gcc-patches-owner@gcc.gnu.org Delivered-To: mailing list gcc-patches@gcc.gnu.org Received: (qmail 94525 invoked by uid 89); 16 Aug 2017 13:38:38 -0000 Authentication-Results: sourceware.org; auth=none X-Virus-Found: No X-Spam-SWARE-Status: No, score=-11.4 required=5.0 tests=AWL, BAYES_00, GIT_PATCH_2, GIT_PATCH_3, KAM_ASCII_DIVIDERS, RCVD_IN_DNSWL_NONE, SPF_PASS autolearn=ham version=3.3.2 spammy=dra X-HELO: mail-wr0-f175.google.com Received: from mail-wr0-f175.google.com (HELO mail-wr0-f175.google.com) (209.85.128.175) by sourceware.org (qpsmtpd/0.93/v0.84-503-g423c35a) with ESMTP; Wed, 16 Aug 2017 13:38:33 +0000 Received: by mail-wr0-f175.google.com with SMTP id 49so7594209wrw.2 for ; Wed, 16 Aug 2017 06:38:33 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:mail-followup-to:subject:date:message-id :user-agent:mime-version; bh=8yGpOPBs/g5snVGNELpQF4b5pLnxAn7oiyVl4C85q7k=; b=kU97HO6cDQlKQpgDMWReqiwWCjGJ1egQHTsFH5xAoVgb1gDycssSbX5K7qw6StkvF3 v6kYrukzAcj90y1UX520NLx1/geJSRzkoc142N0FC3uIQ4MbFVDx7TV2wl7CncrI5+gk H9XuaYbRwnBWGErgkWqGjQS7OZwzVvNdxAuwdd67AqcQ0Hdvg6aPDMSGpP+KgQyT1r10 /G9+iv/nBiSfRv8SB7rU47/ZCe05H5goiy088kRzCtq0FNPNlLDbBFNTa6haIjoyy8tC sPCW/iEFURXeAyx1QHcz8/46eMIATECJ3wLXaODWpkTrg02CwxWOstp4E+6bn4dEkyBs DkiQ== X-Gm-Message-State: AHYfb5gZxyWIZY9GJGPJjeNSfTMNMp74KkqaZqkHcRYSJQk7vE4HUjFl A2bi5UQOUUG3/KlHeX+KoA== X-Received: by 10.28.10.131 with SMTP id 125mr1534201wmk.132.1502890710946; Wed, 16 Aug 2017 06:38:30 -0700 (PDT) Received: from localhost ([2.25.234.123]) by smtp.gmail.com with ESMTPSA id r76sm13464700wmd.1.2017.08.16.06.38.27 for (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Wed, 16 Aug 2017 06:38:30 -0700 (PDT) From: Richard Sandiford To: gcc-patches@gcc.gnu.org Mail-Followup-To: gcc-patches@gcc.gnu.org, richard.sandiford@linaro.org Subject: PR81635: Use chrecs to help find related data refs Date: Wed, 16 Aug 2017 14:38:27 +0100 Message-ID: <87pobvr5t8.fsf@linaro.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/25.2 (gnu/linux) MIME-Version: 1.0 The first loop in the testcase regressed after my recent changes to dr_analyze_innermost. Previously we would treat "i" as an iv even for bb analysis and end up with: DR_BASE_ADDRESS: p or q DR_OFFSET: 0 DR_INIT: 0 or 4 DR_STEP: 16 We now always keep the step as 0 instead, so for an int "i" we'd have: DR_BASE_ADDRESS: p or q DR_OFFSET: (intptr_t) i DR_INIT: 0 or 4 DR_STEP: 0 This is also what we'd like to have for the unsigned "i", but the problem is that strip_constant_offset thinks that the "i + 1" in "(intptr_t) (i + 1)" could wrap and so doesn't peel off the "+ 1". The [i + 1] accesses therefore have a DR_OFFSET equal to the SSA name that holds "(intptr_t) (i + 1)", meaning that the accesses no longer seem to be related to the [i] ones. There seem to be two main uses of DR_OFFSET and DR_INIT. One type expects DR_OFFSET and DR_INIT to be generic expressions whose sum gives the initial offset from DR_BASE_ADDRESS. The other type uses the pair (DR_BASE_ADDRESS, DR_OFFSET) to identify related data references, with the distance between their start addresses being the difference between their DR_INITs. For this second type, the exact form of DR_OFFSET doesn't really matter, and the more it is able to canonicalise the non-constant offset, the better. The patch fixes the PR by adding another offset/init pair to the innermost loop behaviour for this second group. The new pair use chrecs rather than generic exprs for the offset part, with the chrec being analysed in the innermost loop for which the offset isn't invariant. This allows us to vectorise the second function in the testcase as well, which wasn't possible before the earlier patch. Tested on x86_64-linux-gnu and aarch64-linux-gnu. OK to install? Richard gcc/ PR tree-optimization/81635 * tree-data-ref.h (innermost_loop_behavior): Add chrec_offset and chrec_init. (DR_CHREC_OFFSET, DR_CHREC_INIT): New macros. (dr_equal_offsets_p): Delete. (dr_chrec_offsets_equal_p, dr_chrec_offsets_compare): Declare. * tree-data-ref.c: Include tree-ssa-loop-ivopts.h. (split_constant_offset): Handle POLYNOMIAL_CHREC. (dr_analyze_innermost): Initialize dr_chrec_offset and dr_chrec_init. (operator ==): Use dr_chrec_offsets_equal_p and compare the DR_CHREC_INITs. (prune_runtime_alias_test_list): Likewise. (comp_dr_with_seg_len_pair): Use dr_chrec_offsets_compare and compare the DR_CHREC_INITs. (dr_equal_offsets_p1, dr_equal_offsets_p): Delete. (analyze_offset_scev): New function. (compute_offset_chrecs): Likewise. (dr_chrec_offsets_equal_p): Likewise. (dr_chrec_offsets_compare): Likewise. * tree-loop-distribution.c (compute_alias_check_pairs): Use dr_chrec_offsets_compare. * tree-vect-data-refs.c (vect_find_same_alignment_drs): Use dr_chrec_offsets_compare and compare the DR_CHREC_INITs. (dr_group_sort_cmp): Likewise. (vect_analyze_group_access_1): Use DR_CHREC_INIT instead of DR_INIT. (vect_no_alias_p): Likewise. (vect_analyze_data_ref_accesses): Use dr_chrec_offsets_equal_p and compare the DR_CHREC_INITs. (vect_prune_runtime_alias_test_list): Use dr_chrec_offsets_compare. * tree-vect-stmts.c (vectorizable_load): Use DR_CHREC_INIT instead of DR_INIT. gcc/testsuite/ PR tree-optimization/81635 * gcc.dg/vect/bb-slp-pr81635.c: New test. Index: gcc/tree-data-ref.h =================================================================== --- gcc/tree-data-ref.h 2017-08-04 11:42:45.938134820 +0100 +++ gcc/tree-data-ref.h 2017-08-16 14:34:56.611554296 +0100 @@ -52,6 +52,27 @@ struct innermost_loop_behavior tree init; tree step; + /* The scalar evolution of OFFSET + INIT, split into non-constant and + constant terms in the same way as OFFSET and INIT. For example, + if OFFSET + INIT is {x + 4, +, 3}_1, the fields would be: + + chrec_offset {x, +, 3}_1 + chrec_init 4 + + If OFFSET + INIT cannot be represented as a scalar evolution, + the fields are simply a copy of OFFSET and INIT. + + This split is useful when analyzing the relationship between two + data references with the same base. OFFSET and INIT are generic + expressions that can be used to build related data references, + while CHREC_OFFSET and CHREC_INIT are our best attempt at + canonicalizing the value of OFFSET + INIT. + + These fields are only initialized lazily, by a call to + compute_offset_chrecs. */ + tree chrec_offset; + tree chrec_init; + /* BASE_ADDRESS is known to be misaligned by BASE_MISALIGNMENT bytes from an alignment boundary of BASE_ALIGNMENT bytes. For example, if we had: @@ -187,6 +208,8 @@ #define DR_IS_CONDITIONAL_IN_STMT(DR) (D #define DR_BASE_ADDRESS(DR) (DR)->innermost.base_address #define DR_OFFSET(DR) (DR)->innermost.offset #define DR_INIT(DR) (DR)->innermost.init +#define DR_CHREC_OFFSET(DR) (DR)->innermost.chrec_offset +#define DR_CHREC_INIT(DR) (DR)->innermost.chrec_init #define DR_STEP(DR) (DR)->innermost.step #define DR_PTR_INFO(DR) (DR)->alias.ptr_info #define DR_BASE_ALIGNMENT(DR) (DR)->innermost.base_alignment @@ -466,8 +489,10 @@ dr_alignment (data_reference *dr) extern bool dr_may_alias_p (const struct data_reference *, const struct data_reference *, bool); -extern bool dr_equal_offsets_p (struct data_reference *, - struct data_reference *); +extern bool dr_chrec_offsets_equal_p (struct data_reference *, + struct data_reference *); +extern int dr_chrec_offsets_compare (struct data_reference *, + struct data_reference *); extern bool runtime_alias_check_p (ddr_p, struct loop *, bool); extern int data_ref_compare_tree (tree, tree); Index: gcc/tree-data-ref.c =================================================================== --- gcc/tree-data-ref.c 2017-08-04 11:42:45.938134820 +0100 +++ gcc/tree-data-ref.c 2017-08-16 14:34:56.611554296 +0100 @@ -86,6 +86,7 @@ Software Foundation; either version 3, o #include "expr.h" #include "gimple-iterator.h" #include "tree-ssa-loop-niter.h" +#include "tree-ssa-loop-ivopts.h" #include "tree-ssa-loop.h" #include "tree-ssa.h" #include "cfgloop.h" @@ -730,7 +731,19 @@ split_constant_offset (tree exp, tree *v *off = ssize_int (0); STRIP_NOPS (exp); - if (tree_is_chrec (exp) + if (TREE_CODE (exp) == POLYNOMIAL_CHREC) + { + split_constant_offset (CHREC_LEFT (exp), &op0, &op1); + if (op0 != CHREC_LEFT (exp)) + { + *var = build3 (POLYNOMIAL_CHREC, type, CHREC_VAR (exp), + op0, CHREC_RIGHT (exp)); + *off = op1; + } + return; + } + + if (automatically_generated_chrec_p (exp) || get_gimple_rhs_class (TREE_CODE (exp)) == GIMPLE_TERNARY_RHS) return; @@ -924,6 +937,8 @@ dr_analyze_innermost (innermost_loop_beh drb->offset = fold_convert (ssizetype, offset_iv.base); drb->init = init; drb->step = step; + drb->chrec_offset = NULL_TREE; + drb->chrec_init = NULL_TREE; drb->base_alignment = base_alignment; drb->base_misalignment = base_misalignment & (base_alignment - 1); drb->offset_alignment = highest_pow2_factor (offset_iv.base); @@ -1324,11 +1339,12 @@ runtime_alias_check_p (ddr_p ddr, struct operator == (const dr_with_seg_len& d1, const dr_with_seg_len& d2) { - return operand_equal_p (DR_BASE_ADDRESS (d1.dr), + return (operand_equal_p (DR_BASE_ADDRESS (d1.dr), DR_BASE_ADDRESS (d2.dr), 0) - && data_ref_compare_tree (DR_OFFSET (d1.dr), DR_OFFSET (d2.dr)) == 0 - && data_ref_compare_tree (DR_INIT (d1.dr), DR_INIT (d2.dr)) == 0 - && data_ref_compare_tree (d1.seg_len, d2.seg_len) == 0; + && dr_chrec_offsets_equal_p (d1.dr, d2.dr) + && data_ref_compare_tree (DR_CHREC_INIT (d1.dr), + DR_CHREC_INIT (d2.dr)) == 0 + && data_ref_compare_tree (d1.seg_len, d2.seg_len) == 0); } /* Comparison function for sorting objects of dr_with_seg_len_pair_t @@ -1360,17 +1376,15 @@ comp_dr_with_seg_len_pair (const void *p if ((comp_res = data_ref_compare_tree (DR_STEP (a2.dr), DR_STEP (b2.dr))) != 0) return comp_res; - if ((comp_res = data_ref_compare_tree (DR_OFFSET (a1.dr), - DR_OFFSET (b1.dr))) != 0) + if ((comp_res = dr_chrec_offsets_compare (a1.dr, b1.dr)) != 0) return comp_res; - if ((comp_res = data_ref_compare_tree (DR_INIT (a1.dr), - DR_INIT (b1.dr))) != 0) + if ((comp_res = data_ref_compare_tree (DR_CHREC_INIT (a1.dr), + DR_CHREC_INIT (b1.dr))) != 0) return comp_res; - if ((comp_res = data_ref_compare_tree (DR_OFFSET (a2.dr), - DR_OFFSET (b2.dr))) != 0) + if ((comp_res = dr_chrec_offsets_compare (a2.dr, b2.dr)) != 0) return comp_res; - if ((comp_res = data_ref_compare_tree (DR_INIT (a2.dr), - DR_INIT (b2.dr))) != 0) + if ((comp_res = data_ref_compare_tree (DR_CHREC_INIT (a2.dr), + DR_CHREC_INIT (b2.dr))) != 0) return comp_res; return 0; @@ -1455,10 +1469,9 @@ prune_runtime_alias_test_list (vecdr), DR_BASE_ADDRESS (dr_a2->dr), 0) - || !operand_equal_p (DR_OFFSET (dr_a1->dr), - DR_OFFSET (dr_a2->dr), 0) - || !tree_fits_shwi_p (DR_INIT (dr_a1->dr)) - || !tree_fits_shwi_p (DR_INIT (dr_a2->dr))) + || !dr_chrec_offsets_equal_p (dr_a1->dr, dr_a2->dr) + || !tree_fits_shwi_p (DR_CHREC_INIT (dr_a1->dr)) + || !tree_fits_shwi_p (DR_CHREC_INIT (dr_a2->dr))) continue; /* Only merge const step data references. */ @@ -1484,11 +1497,13 @@ prune_runtime_alias_test_list (vecdr), DR_INIT (dr_a1->dr))) + if (tree_int_cst_lt (DR_CHREC_INIT (dr_a2->dr), + DR_CHREC_INIT (dr_a1->dr))) std::swap (*dr_a1, *dr_a2); bool do_remove = false; - wide_int diff = wi::sub (DR_INIT (dr_a2->dr), DR_INIT (dr_a1->dr)); + wide_int diff = wi::sub (DR_CHREC_INIT (dr_a2->dr), + DR_CHREC_INIT (dr_a1->dr)); wide_int min_seg_len_b; tree new_seg_len; @@ -1826,46 +1841,83 @@ create_runtime_alias_checks (struct loop } } -/* Check if OFFSET1 and OFFSET2 (DR_OFFSETs of some data-refs) are identical - expressions. */ -static bool -dr_equal_offsets_p1 (tree offset1, tree offset2) +/* Analyze the scalar evolution of OFFSET in the innermost parent of + LOOP for which it isn't invariant. */ + +static tree +analyze_offset_scev (struct loop *loop, tree offset) { - bool res; + struct loop *inv_loop = outermost_invariant_loop_for_expr (loop, offset); + if (inv_loop != NULL) + { + if (loop_depth (inv_loop) == 0) + return offset; + loop = loop_outer (inv_loop); + } + return analyze_scalar_evolution (loop, offset); +} - STRIP_NOPS (offset1); - STRIP_NOPS (offset2); +/* Analyze the scalar evolution of DRB->offset + DRB->init in a form + that is valid for use in LOOP. Store the result DRB->chrec_offset + and DRB->chrec_init. */ - if (offset1 == offset2) - return true; +static void +compute_offset_chrecs (struct loop *loop, innermost_loop_behavior *drb) +{ + tree scev = analyze_offset_scev (loop, drb->offset); + if (TREE_CODE (scev) == POLYNOMIAL_CHREC) + { + tree init_term; + split_constant_offset (scev, &drb->chrec_offset, &init_term); + drb->chrec_init = fold_build2 (PLUS_EXPR, ssizetype, init_term, + drb->init); + } + else + { + drb->chrec_offset = drb->offset; + drb->chrec_init = drb->init; + } +} - if (TREE_CODE (offset1) != TREE_CODE (offset2) - || (!BINARY_CLASS_P (offset1) && !UNARY_CLASS_P (offset1))) - return false; +/* Analyze the scalar evolution of DR_OFFSET (DR) + DR_INIT (DR) wrt + the loop that contains DR_STMT (DR), storing the result in + DR_CHREC_OFFSET (DR) and DR_CHREC_INIT (DR). */ - res = dr_equal_offsets_p1 (TREE_OPERAND (offset1, 0), - TREE_OPERAND (offset2, 0)); +static void +compute_offset_chrecs (data_reference *dr) +{ + return compute_offset_chrecs (loop_containing_stmt (DR_STMT (dr)), + &DR_INNERMOST (dr)); +} - if (!res || !BINARY_CLASS_P (offset1)) - return res; +/* Check if DRA and DRB have equal DR_CHREC_OFFSETs, computing them + first if necessary. */ - res = dr_equal_offsets_p1 (TREE_OPERAND (offset1, 1), - TREE_OPERAND (offset2, 1)); +bool +dr_chrec_offsets_equal_p (struct data_reference *dra, + struct data_reference *drb) +{ + if (!DR_CHREC_OFFSET (dra)) + compute_offset_chrecs (dra); + if (!DR_CHREC_OFFSET (drb)) + compute_offset_chrecs (drb); - return res; + return eq_evolutions_p (DR_CHREC_OFFSET (dra), DR_CHREC_OFFSET (drb)); } -/* Check if DRA and DRB have equal offsets. */ -bool -dr_equal_offsets_p (struct data_reference *dra, - struct data_reference *drb) -{ - tree offset1, offset2; +/* Compare the DR_CHREC_OFFSETs of DRA and DRB for sorting purposes, + computing them first if necessary. */ - offset1 = DR_OFFSET (dra); - offset2 = DR_OFFSET (drb); +int +dr_chrec_offsets_compare (struct data_reference *dra, + struct data_reference *drb) +{ + if (!DR_CHREC_OFFSET (dra)) + compute_offset_chrecs (dra); + if (!DR_CHREC_OFFSET (drb)) + compute_offset_chrecs (drb); - return dr_equal_offsets_p1 (offset1, offset2); + return data_ref_compare_tree (DR_CHREC_OFFSET (dra), DR_CHREC_OFFSET (drb)); } /* Returns true if FNA == FNB. */ Index: gcc/tree-loop-distribution.c =================================================================== --- gcc/tree-loop-distribution.c 2017-08-16 08:50:32.415427038 +0100 +++ gcc/tree-loop-distribution.c 2017-08-16 14:34:56.612554255 +0100 @@ -2204,7 +2204,7 @@ compute_alias_check_pairs (struct loop * DR_BASE_ADDRESS (dr_b)); if (comp_res == 0) - comp_res = data_ref_compare_tree (DR_OFFSET (dr_a), DR_OFFSET (dr_b)); + comp_res = dr_chrec_offsets_compare (dr_a, dr_b); gcc_assert (comp_res != 0); if (latch_dominated_by_data_ref (loop, dr_a)) Index: gcc/tree-vect-data-refs.c =================================================================== --- gcc/tree-vect-data-refs.c 2017-08-04 11:42:45.939105152 +0100 +++ gcc/tree-vect-data-refs.c 2017-08-16 14:34:56.613554213 +0100 @@ -2187,13 +2187,13 @@ vect_find_same_alignment_drs (struct dat return; if (!operand_equal_p (DR_BASE_ADDRESS (dra), DR_BASE_ADDRESS (drb), 0) - || !operand_equal_p (DR_OFFSET (dra), DR_OFFSET (drb), 0) + || !dr_chrec_offsets_equal_p (dra, drb) || !operand_equal_p (DR_STEP (dra), DR_STEP (drb), 0)) return; /* Two references with distance zero have the same alignment. */ - offset_int diff = (wi::to_offset (DR_INIT (dra)) - - wi::to_offset (DR_INIT (drb))); + offset_int diff = (wi::to_offset (DR_CHREC_INIT (dra)) + - wi::to_offset (DR_CHREC_INIT (drb))); if (diff != 0) { /* Get the wider of the two alignments. */ @@ -2434,7 +2434,7 @@ vect_analyze_group_access_1 (struct data gimple *next = GROUP_NEXT_ELEMENT (vinfo_for_stmt (stmt)); struct data_reference *data_ref = dr; unsigned int count = 1; - tree prev_init = DR_INIT (data_ref); + tree prev_init = DR_CHREC_INIT (data_ref); gimple *prev = stmt; HOST_WIDE_INT diff, gaps = 0; @@ -2444,9 +2444,9 @@ vect_analyze_group_access_1 (struct data data-ref (supported only for loads), we vectorize only the first stmt, and the rest get their vectorized loads from the first one. */ - if (!tree_int_cst_compare (DR_INIT (data_ref), - DR_INIT (STMT_VINFO_DATA_REF ( - vinfo_for_stmt (next))))) + if (!tree_int_cst_compare (DR_CHREC_INIT (data_ref), + DR_CHREC_INIT (STMT_VINFO_DATA_REF + (vinfo_for_stmt (next))))) { if (DR_IS_WRITE (data_ref)) { @@ -2476,7 +2476,7 @@ vect_analyze_group_access_1 (struct data /* Check that the distance between two accesses is equal to the type size. Otherwise, we have gaps. */ - diff = (TREE_INT_CST_LOW (DR_INIT (data_ref)) + diff = (TREE_INT_CST_LOW (DR_CHREC_INIT (data_ref)) - TREE_INT_CST_LOW (prev_init)) / type_size; if (diff != 1) { @@ -2499,7 +2499,7 @@ vect_analyze_group_access_1 (struct data gap in the access, GROUP_GAP is always 1. */ GROUP_GAP (vinfo_for_stmt (next)) = diff; - prev_init = DR_INIT (data_ref); + prev_init = DR_CHREC_INIT (data_ref); next = GROUP_NEXT_ELEMENT (vinfo_for_stmt (next)); /* Count the number of data-refs in the chain. */ count++; @@ -2715,13 +2715,10 @@ dr_group_sort_cmp (const void *dra_, con return cmp; } - /* And according to DR_OFFSET. */ - if (!dr_equal_offsets_p (dra, drb)) - { - cmp = data_ref_compare_tree (DR_OFFSET (dra), DR_OFFSET (drb)); - if (cmp != 0) - return cmp; - } + /* And according to DR_CHREC_OFFSET. */ + cmp = dr_chrec_offsets_compare (dra, drb); + if (cmp != 0) + return cmp; /* Put reads before writes. */ if (DR_IS_READ (dra) != DR_IS_READ (drb)) @@ -2745,8 +2742,9 @@ dr_group_sort_cmp (const void *dra_, con return cmp; } - /* Then sort after DR_INIT. In case of identical DRs sort after stmt UID. */ - cmp = tree_int_cst_compare (DR_INIT (dra), DR_INIT (drb)); + /* Then sort after DR_CHREC_INIT. In case of identical DRs sort after + stmt UID. */ + cmp = tree_int_cst_compare (DR_CHREC_INIT (dra), DR_CHREC_INIT (drb)); if (cmp == 0) return gimple_uid (DR_STMT (dra)) < gimple_uid (DR_STMT (drb)) ? -1 : 1; return cmp; @@ -2817,7 +2815,7 @@ vect_analyze_data_ref_accesses (vec_info if (DR_IS_READ (dra) != DR_IS_READ (drb) || !operand_equal_p (DR_BASE_ADDRESS (dra), DR_BASE_ADDRESS (drb), 0) - || !dr_equal_offsets_p (dra, drb) + || !dr_chrec_offsets_equal_p (dra, drb) || !gimple_assign_single_p (DR_STMT (dra)) || !gimple_assign_single_p (DR_STMT (drb))) break; @@ -2835,7 +2833,8 @@ vect_analyze_data_ref_accesses (vec_info break; /* Do not place the same access in the interleaving chain twice. */ - if (tree_int_cst_compare (DR_INIT (dra), DR_INIT (drb)) == 0) + if (tree_int_cst_compare (DR_CHREC_INIT (dra), + DR_CHREC_INIT (drb)) == 0) break; /* Check the types are compatible. @@ -2844,9 +2843,10 @@ vect_analyze_data_ref_accesses (vec_info TREE_TYPE (DR_REF (drb)))) break; - /* Sorting has ensured that DR_INIT (dra) <= DR_INIT (drb). */ - HOST_WIDE_INT init_a = TREE_INT_CST_LOW (DR_INIT (dra)); - HOST_WIDE_INT init_b = TREE_INT_CST_LOW (DR_INIT (drb)); + /* Sorting has ensured that + DR_CHREC_INIT (dra) <= DR_CHREC_INIT (drb). */ + HOST_WIDE_INT init_a = TREE_INT_CST_LOW (DR_CHREC_INIT (dra)); + HOST_WIDE_INT init_b = TREE_INT_CST_LOW (DR_CHREC_INIT (drb)); gcc_assert (init_a <= init_b); /* If init_b == init_a + the size of the type * k, we have an @@ -2859,10 +2859,10 @@ vect_analyze_data_ref_accesses (vec_info /* If we have a store, the accesses are adjacent. This splits groups into chunks we support (we don't support vectorization of stores with gaps). */ + HOST_WIDE_INT prev_init + = TREE_INT_CST_LOW (DR_CHREC_INIT (datarefs_copy[i - 1])); if (!DR_IS_READ (dra) - && (init_b - (HOST_WIDE_INT) TREE_INT_CST_LOW - (DR_INIT (datarefs_copy[i-1])) - != type_size_a)) + && (init_b - prev_init) != type_size_a) break; /* If the step (if not zero or non-constant) is greater than the @@ -2974,12 +2974,12 @@ vect_vfa_segment_size (struct data_refer vect_no_alias_p (struct data_reference *a, struct data_reference *b, tree segment_length_a, tree segment_length_b) { - gcc_assert (TREE_CODE (DR_INIT (a)) == INTEGER_CST - && TREE_CODE (DR_INIT (b)) == INTEGER_CST); - if (tree_int_cst_equal (DR_INIT (a), DR_INIT (b))) + gcc_assert (TREE_CODE (DR_CHREC_INIT (a)) == INTEGER_CST + && TREE_CODE (DR_CHREC_INIT (b)) == INTEGER_CST); + if (tree_int_cst_equal (DR_CHREC_INIT (a), DR_CHREC_INIT (b))) return false; - tree seg_a_min = DR_INIT (a); + tree seg_a_min = DR_CHREC_INIT (a); tree seg_a_max = fold_build2 (PLUS_EXPR, TREE_TYPE (seg_a_min), seg_a_min, segment_length_a); /* For negative step, we need to adjust address range by TYPE_SIZE_UNIT @@ -2990,10 +2990,10 @@ vect_no_alias_p (struct data_reference * tree unit_size = TYPE_SIZE_UNIT (TREE_TYPE (DR_REF (a))); seg_a_min = fold_build2 (PLUS_EXPR, TREE_TYPE (seg_a_max), seg_a_max, unit_size); - seg_a_max = fold_build2 (PLUS_EXPR, TREE_TYPE (DR_INIT (a)), - DR_INIT (a), unit_size); + seg_a_max = fold_build2 (PLUS_EXPR, TREE_TYPE (DR_CHREC_INIT (a)), + DR_CHREC_INIT (a), unit_size); } - tree seg_b_min = DR_INIT (b); + tree seg_b_min = DR_CHREC_INIT (b); tree seg_b_max = fold_build2 (PLUS_EXPR, TREE_TYPE (seg_b_min), seg_b_min, segment_length_b); if (tree_int_cst_compare (DR_STEP (b), size_zero_node) < 0) @@ -3001,8 +3001,8 @@ vect_no_alias_p (struct data_reference * tree unit_size = TYPE_SIZE_UNIT (TREE_TYPE (DR_REF (b))); seg_b_min = fold_build2 (PLUS_EXPR, TREE_TYPE (seg_b_max), seg_b_max, unit_size); - seg_b_max = fold_build2 (PLUS_EXPR, TREE_TYPE (DR_INIT (b)), - DR_INIT (b), unit_size); + seg_b_max = fold_build2 (PLUS_EXPR, TREE_TYPE (DR_CHREC_INIT (b)), + DR_CHREC_INIT (b), unit_size); } if (tree_int_cst_le (seg_a_max, seg_b_min) @@ -3148,8 +3148,7 @@ vect_prune_runtime_alias_test_list (loop comp_res = data_ref_compare_tree (DR_BASE_ADDRESS (dr_a), DR_BASE_ADDRESS (dr_b)); if (comp_res == 0) - comp_res = data_ref_compare_tree (DR_OFFSET (dr_a), - DR_OFFSET (dr_b)); + comp_res = dr_chrec_offsets_compare (dr_a, dr_b); /* Alias is known at compilation time. */ if (comp_res == 0 Index: gcc/tree-vect-stmts.c =================================================================== --- gcc/tree-vect-stmts.c 2017-08-03 10:40:55.650125801 +0100 +++ gcc/tree-vect-stmts.c 2017-08-16 14:34:56.614554172 +0100 @@ -7423,8 +7423,8 @@ vectorizable_load (gimple *stmt, gimple_ = STMT_VINFO_DATA_REF (vinfo_for_stmt (first_stmt_for_drptr)); tree diff = fold_convert (sizetype, size_binop (MINUS_EXPR, - DR_INIT (first_dr), - DR_INIT (ptrdr))); + DR_CHREC_INIT (first_dr), + DR_CHREC_INIT (ptrdr))); dataref_ptr = bump_vector_ptr (dataref_ptr, ptr_incr, gsi, stmt, diff); } Index: gcc/testsuite/gcc.dg/vect/bb-slp-pr81635.c =================================================================== --- /dev/null 2017-08-15 18:19:06.926776201 +0100 +++ gcc/testsuite/gcc.dg/vect/bb-slp-pr81635.c 2017-08-16 14:34:56.610554337 +0100 @@ -0,0 +1,32 @@ +/* { dg-do compile } */ +/* { dg-require-effective-target vect_unpack } */ + +double p[1000]; +double q[1000]; + +void +f1 (void) +{ + for (unsigned int i = 0; i < 1000; i += 4) + { + double a = q[i] + p[i]; + double b = q[i + 1] + p[i + 1]; + q[i] = a; + q[i + 1] = b; + } +} + +void +f2 (void) +{ + for (unsigned int i = 0; i < 500; i += 6) + for (unsigned int j = 0; j < 500; j += 4) + { + double a = q[j] + p[i]; + double b = q[j + 1] + p[i + 1]; + q[i] = a; + q[i + 1] = b; + } +} + +/* { dg-final { scan-tree-dump-times "basic block vectorized" 2 "slp1" } } */