From patchwork Sat Jan 20 10:35:12 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Sandiford X-Patchwork-Id: 125246 Delivered-To: patch@linaro.org Received: by 10.46.66.141 with SMTP id h13csp227071ljf; Sat, 20 Jan 2018 02:35:33 -0800 (PST) X-Google-Smtp-Source: AH8x225Tl+qGPOXJGWii9wIBDqV7dMCR6djCAUu3NwQHAni+Hnh2LtZSVKgvCw2+jjoe9iuOYgQb X-Received: by 2002:a17:902:4:: with SMTP id 4-v6mr720092pla.187.1516444533117; Sat, 20 Jan 2018 02:35:33 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1516444533; cv=none; d=google.com; s=arc-20160816; b=vCpN7ZFz2QQVNQJ//hjP+pYzpw8San1yiPkWsWBnDIDhE3Wm8XWRyDEUOaGQy7EOsZ W4PhX9nkQ5263AK2cGPS/yuxl9MT+Bnti29qGRchxr6nAZ92R+snk3gVx2TrTvOGGWkm ko7JO7oW2NXT3FbTST8c1POEW75B+4gVFPVP3XaiGWYmyCF1zKhlSh056PwQpUEoseDr XIBsAhRVFzbHn5M8SkASy7AIvZ9FIQuTFWGixic9ajHqfQfjDmO/C6BTEQEpEx8zOWUp BGU5NOrE6xW+JR54oe3PD9lwst0RgWrQvY4FqrkeuytICYIvN8TE13ybdA1FuJMLcL5C 7twA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:user-agent:message-id:date:subject:mail-followup-to:to :from:delivered-to:sender:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:mailing-list:dkim-signature :domainkey-signature:arc-authentication-results; bh=Md/uIrYl15bM1zF4nc/UJ1ZJkVOwp/nqBfwg6bcAx2g=; b=0hE/pPV19PdWZeMk8uvyasToHF3TT5atWITPpHnmjUzTPQLXsknZcpa9ncmif0ZHZz 9SCRFCON2saD6zU7hIOfeaoF7RBsyOlfd/P1hJSNhzQtvWXHWtgtl9oA+PAstaG0lBCq HXAbQYTJ6G4w8M0NdMP3xnU49oHwpd2mt4FzJQ0mvexPA4mPnQ9muR+MDp6Ejb2N9uv2 khxGLcTX5wceCPDbIqO5qnVUKsB8il3C2rqdTRonpH8zonJwZFZG8UEh1Kw4xVvsgi5t 7jUrKQ6G8jwJ6ENKYrAvCVzJd/1Yh7NEj53PoSp4W1+4QnbR6sEvw6Vglp+XZ37Gq8UU OG+A== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gcc.gnu.org header.s=default header.b=p6OKOcmj; spf=pass (google.com: domain of gcc-patches-return-471729-patch=linaro.org@gcc.gnu.org designates 209.132.180.131 as permitted sender) smtp.mailfrom=gcc-patches-return-471729-patch=linaro.org@gcc.gnu.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from sourceware.org (server1.sourceware.org. [209.132.180.131]) by mx.google.com with ESMTPS id j8si10084988pgc.229.2018.01.20.02.35.32 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 20 Jan 2018 02:35:33 -0800 (PST) Received-SPF: pass (google.com: domain of gcc-patches-return-471729-patch=linaro.org@gcc.gnu.org designates 209.132.180.131 as permitted sender) client-ip=209.132.180.131; Authentication-Results: mx.google.com; dkim=pass header.i=@gcc.gnu.org header.s=default header.b=p6OKOcmj; spf=pass (google.com: domain of gcc-patches-return-471729-patch=linaro.org@gcc.gnu.org designates 209.132.180.131 as permitted sender) smtp.mailfrom=gcc-patches-return-471729-patch=linaro.org@gcc.gnu.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org DomainKey-Signature: a=rsa-sha1; c=nofws; d=gcc.gnu.org; h=list-id :list-unsubscribe:list-archive:list-post:list-help:sender:from :to:subject:date:message-id:mime-version:content-type; q=dns; s= default; b=Ciy8/fPo7sK6A0hhP9ijuCjdOsi52ZBB3Owwvgba5q8j2kSYK8hTI pmLuq89vSxoXu6sLN++luGlnFZEptFrBi+mLAvgpkFsR8tpDqH8nJpnLE+sIdpAY IwBYyY9nF/qK0W8mpbmEq2P8kdIhsi/ffsrya0Spmfu4/IuAJHDWN4= DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=gcc.gnu.org; h=list-id :list-unsubscribe:list-archive:list-post:list-help:sender:from :to:subject:date:message-id:mime-version:content-type; s= default; bh=A7k/ScKtj+0j9jQ0LWRIW1YNSx0=; b=p6OKOcmjrTdBGpKb003y Z1GyNZdS8jmV0SiyFIJe0I+DEZpyEpCt4xVY838qxGwMVDsXLlSwkoXdkqD/lCLI LT119r+m0enJDpl2a5KS+PJwI6pmZwyAckxWidyZvxPtEvnIpPjCZJMIPr+bEaFy 6QhRvNFs8m14jfH8XGB/1HU= Received: (qmail 61288 invoked by alias); 20 Jan 2018 10:35:21 -0000 Mailing-List: contact gcc-patches-help@gcc.gnu.org; run by ezmlm Precedence: bulk List-Id: List-Unsubscribe: List-Archive: List-Post: List-Help: Sender: gcc-patches-owner@gcc.gnu.org Delivered-To: mailing list gcc-patches@gcc.gnu.org Received: (qmail 61278 invoked by uid 89); 20 Jan 2018 10:35:19 -0000 Authentication-Results: sourceware.org; auth=none X-Virus-Found: No X-Spam-SWARE-Status: No, score=-15.7 required=5.0 tests=AWL, BAYES_00, GIT_PATCH_1, GIT_PATCH_2, GIT_PATCH_3, KAM_ASCII_DIVIDERS, RCVD_IN_DNSWL_NONE, SPF_PASS autolearn=ham version=3.3.2 spammy=GSI, caches, (unknown), 61727 X-HELO: mail-wm0-f42.google.com Received: from mail-wm0-f42.google.com (HELO mail-wm0-f42.google.com) (74.125.82.42) by sourceware.org (qpsmtpd/0.93/v0.84-503-g423c35a) with ESMTP; Sat, 20 Jan 2018 10:35:17 +0000 Received: by mail-wm0-f42.google.com with SMTP id v71so8042638wmv.2 for ; Sat, 20 Jan 2018 02:35:16 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:mail-followup-to:subject:date:message-id :user-agent:mime-version; bh=Md/uIrYl15bM1zF4nc/UJ1ZJkVOwp/nqBfwg6bcAx2g=; b=JPRWUPChTsCUJK3oSWtSY7NPTLWD2tMgdcRwSfanuc8oKWtRoUXm/rLaQzIS2t1uHh wMZIkE+kBlasAF8pLPWUpzJVEet8fQx4HS1kI2dTUDjz4gd1GjP4xFbldg6U7RRba9jQ rb9+F19SEfn3AYgYkizffDIcwK+4yViBu/R8UcP38+8fn4Q2+xqAttXl4AU9SepbsnK+ 4kr739bQr/ZmTh5toJ41iqcbOW3lypClfodF7mXEmQNhEO1iUUlkqXgeJyo9euwMTqDf i8GKISieMu8AKH11Wmu+NUxduA2cv9P0g8nO3lJn9Y6KqOAJ0g6/R1nwagxnWkW0t+CV +A3g== X-Gm-Message-State: AKwxytd79/cJi7+jFACP3l2TDOwncUZIOjjLrE9T+ZGb+RBiI8k12etv I11pRlkaIXApyqIvuqAJwSDXkc+Phqw= X-Received: by 10.28.48.10 with SMTP id w10mr897962wmw.93.1516444514630; Sat, 20 Jan 2018 02:35:14 -0800 (PST) Received: from localhost ([95.144.14.233]) by smtp.gmail.com with ESMTPSA id k35sm19173351wrc.2.2018.01.20.02.35.12 for (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Sat, 20 Jan 2018 02:35:13 -0800 (PST) From: Richard Sandiford To: gcc-patches@gcc.gnu.org Mail-Followup-To: gcc-patches@gcc.gnu.org, richard.sandiford@linaro.org Subject: Fix vect_def_type handling in x86 scatter support (PR 83940) Date: Sat, 20 Jan 2018 10:35:12 +0000 Message-ID: <874lngyg0v.fsf@linaro.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/25.3 (gnu/linux) MIME-Version: 1.0 As Jakub says in the PR, the problem here was that the x86/built-in version of the scatter support was using a bogus scatter_src_dt when calling vect_get_vec_def_for_stmt_copy (and had since it was added). The patch uses the vect_def_type from the original call to vect_is_simple_use instead. However, Jakub also pointed out that other parts of the load and store code passed the vector operand rather than the scalar operand to vect_is_simple_use. That probably works most of the time since a constant scalar operand should give a constant vector operand, and likewise for external and internal definitions. But it definitely seems more robust to pass the scalar operand. The patch avoids the issue for gather and scatter offsets by using the cached gs_info.offset_dt. This is safe because gathers and scatters are never grouped, so there's only one statement operand to consider. The patch also caches the vect_def_type for mask operands, which is safe because grouped masked operations share the same mask. That just leaves the store rhs. We still need to recalculate the vect_def_type there since different store values in the group can have different definition types. But since we still have access to the original scalar operand, it seems better to use that instead. Tested on aarch64-linux-gnu, x86_64-linux-gnu and powerpc64le-linux-gnu. OK to install? Richard 2018-01-20 Richard Sandiford gcc/ PR tree-optimization/83940 * tree-vect-stmts.c (vect_truncate_gather_scatter_offset): Set offset_dt to vect_constant_def rather than vect_unknown_def_type. (vect_check_load_store_mask): Add a mask_dt_out parameter and use it to pass back the definition type. (vect_check_store_rhs): Likewise rhs_dt_out. (vect_build_gather_load_calls): Add a mask_dt argument and use it instead of a call to vect_is_simple_use. (vectorizable_store): Update calls to vect_check_load_store_mask and vect_check_store_rhs. Use the dt returned by the latter instead of scatter_src_dt. Use the cached mask_dt and gs_info.offset_dt instead of calls to vect_is_simple_use. Pass the scalar rather than the vector operand to vect_is_simple_use when handling second and subsequent copies of an rhs value. (vectorizable_load): Update calls to vect_check_load_store_mask and vect_build_gather_load_calls. Use the cached mask_dt and gs_info.offset_dt instead of calls to vect_is_simple_use. gcc/testsuite/ PR tree-optimization/83940 * gcc.dg/torture/pr83940.c: New test. Index: gcc/tree-vect-stmts.c =================================================================== --- gcc/tree-vect-stmts.c 2018-01-16 15:13:19.658832080 +0000 +++ gcc/tree-vect-stmts.c 2018-01-20 10:31:16.692599735 +0000 @@ -1932,7 +1932,7 @@ vect_truncate_gather_scatter_offset (gim but we don't need to store that here. */ gs_info->base = NULL_TREE; gs_info->offset = fold_convert (offset_type, step); - gs_info->offset_dt = vect_unknown_def_type; + gs_info->offset_dt = vect_constant_def; gs_info->offset_vectype = NULL_TREE; gs_info->scale = scale; gs_info->memory_type = memory_type; @@ -2374,11 +2374,14 @@ get_load_store_type (gimple *stmt, tree } /* Return true if boolean argument MASK is suitable for vectorizing - conditional load or store STMT. When returning true, store the - type of the vectorized mask in *MASK_VECTYPE_OUT. */ + conditional load or store STMT. When returning true, store the type + of the definition in *MASK_DT_OUT and the type of the vectorized mask + in *MASK_VECTYPE_OUT. */ static bool -vect_check_load_store_mask (gimple *stmt, tree mask, tree *mask_vectype_out) +vect_check_load_store_mask (gimple *stmt, tree mask, + vect_def_type *mask_dt_out, + tree *mask_vectype_out) { if (!VECT_SCALAR_BOOLEAN_TYPE_P (TREE_TYPE (mask))) { @@ -2398,9 +2401,9 @@ vect_check_load_store_mask (gimple *stmt stmt_vec_info stmt_info = vinfo_for_stmt (stmt); gimple *def_stmt; - enum vect_def_type dt; + enum vect_def_type mask_dt; tree mask_vectype; - if (!vect_is_simple_use (mask, stmt_info->vinfo, &def_stmt, &dt, + if (!vect_is_simple_use (mask, stmt_info->vinfo, &def_stmt, &mask_dt, &mask_vectype)) { if (dump_enabled_p ()) @@ -2437,18 +2440,19 @@ vect_check_load_store_mask (gimple *stmt return false; } + *mask_dt_out = mask_dt; *mask_vectype_out = mask_vectype; return true; } /* Return true if stored value RHS is suitable for vectorizing store statement STMT. When returning true, store the type of the - vectorized store value in *RHS_VECTYPE_OUT and the type of the - store in *VLS_TYPE_OUT. */ + definition in *RHS_DT_OUT, the type of the vectorized store value in + *RHS_VECTYPE_OUT and the type of the store in *VLS_TYPE_OUT. */ static bool -vect_check_store_rhs (gimple *stmt, tree rhs, tree *rhs_vectype_out, - vec_load_store_type *vls_type_out) +vect_check_store_rhs (gimple *stmt, tree rhs, vect_def_type *rhs_dt_out, + tree *rhs_vectype_out, vec_load_store_type *vls_type_out) { /* In the case this is a store from a constant make sure native_encode_expr can handle it. */ @@ -2462,9 +2466,9 @@ vect_check_store_rhs (gimple *stmt, tree stmt_vec_info stmt_info = vinfo_for_stmt (stmt); gimple *def_stmt; - enum vect_def_type dt; + enum vect_def_type rhs_dt; tree rhs_vectype; - if (!vect_is_simple_use (rhs, stmt_info->vinfo, &def_stmt, &dt, + if (!vect_is_simple_use (rhs, stmt_info->vinfo, &def_stmt, &rhs_dt, &rhs_vectype)) { if (dump_enabled_p ()) @@ -2482,8 +2486,9 @@ vect_check_store_rhs (gimple *stmt, tree return false; } + *rhs_dt_out = rhs_dt; *rhs_vectype_out = rhs_vectype; - if (dt == vect_constant_def || dt == vect_external_def) + if (rhs_dt == vect_constant_def || rhs_dt == vect_external_def) *vls_type_out = VLS_STORE_INVARIANT; else *vls_type_out = VLS_STORE; @@ -2546,12 +2551,12 @@ vect_build_zero_merge_argument (gimple * /* Build a gather load call while vectorizing STMT. Insert new instructions before GSI and add them to VEC_STMT. GS_INFO describes the gather load operation. If the load is conditional, MASK is the unvectorized - condition, otherwise MASK is null. */ + condition and MASK_DT is its definition type, otherwise MASK is null. */ static void vect_build_gather_load_calls (gimple *stmt, gimple_stmt_iterator *gsi, gimple **vec_stmt, gather_scatter_info *gs_info, - tree mask) + tree mask, vect_def_type mask_dt) { stmt_vec_info stmt_info = vinfo_for_stmt (stmt); loop_vec_info loop_vinfo = STMT_VINFO_LOOP_VINFO (stmt_info); @@ -2682,12 +2687,7 @@ vect_build_gather_load_calls (gimple *st if (j == 0) vec_mask = vect_get_vec_def_for_operand (mask, stmt); else - { - gimple *def_stmt; - enum vect_def_type dt; - vect_is_simple_use (vec_mask, loop_vinfo, &def_stmt, &dt); - vec_mask = vect_get_vec_def_for_stmt_copy (dt, vec_mask); - } + vec_mask = vect_get_vec_def_for_stmt_copy (mask_dt, vec_mask); mask_op = vec_mask; if (!useless_type_conversion_p (masktype, TREE_TYPE (vec_mask))) @@ -6057,7 +6057,8 @@ vectorizable_store (gimple *stmt, gimple tree dummy; enum dr_alignment_support alignment_support_scheme; gimple *def_stmt; - enum vect_def_type dt; + enum vect_def_type rhs_dt = vect_unknown_def_type; + enum vect_def_type mask_dt = vect_unknown_def_type; stmt_vec_info prev_stmt_info = NULL; tree dataref_ptr = NULL_TREE; tree dataref_offset = NULL_TREE; @@ -6078,7 +6079,6 @@ vectorizable_store (gimple *stmt, gimple vec_info *vinfo = stmt_info->vinfo; tree aggr_type; gather_scatter_info gs_info; - enum vect_def_type scatter_src_dt = vect_unknown_def_type; gimple *new_stmt; poly_uint64 vf; vec_load_store_type vls_type; @@ -6131,7 +6131,8 @@ vectorizable_store (gimple *stmt, gimple if (mask_index >= 0) { mask = gimple_call_arg (call, mask_index); - if (!vect_check_load_store_mask (stmt, mask, &mask_vectype)) + if (!vect_check_load_store_mask (stmt, mask, &mask_dt, + &mask_vectype)) return false; } } @@ -6172,7 +6173,7 @@ vectorizable_store (gimple *stmt, gimple return false; } - if (!vect_check_store_rhs (stmt, op, &rhs_vectype, &vls_type)) + if (!vect_check_store_rhs (stmt, op, &rhs_dt, &rhs_vectype, &vls_type)) return false; elem_type = TREE_TYPE (vectype); @@ -6339,7 +6340,7 @@ vectorizable_store (gimple *stmt, gimple if (modifier == WIDEN) { src = vec_oprnd1 - = vect_get_vec_def_for_stmt_copy (scatter_src_dt, vec_oprnd1); + = vect_get_vec_def_for_stmt_copy (rhs_dt, vec_oprnd1); op = permute_vec_elements (vec_oprnd0, vec_oprnd0, perm_mask, stmt, gsi); } @@ -6357,7 +6358,7 @@ vectorizable_store (gimple *stmt, gimple else { src = vec_oprnd1 - = vect_get_vec_def_for_stmt_copy (scatter_src_dt, vec_oprnd1); + = vect_get_vec_def_for_stmt_copy (rhs_dt, vec_oprnd1); op = vec_oprnd0 = vect_get_vec_def_for_stmt_copy (gs_info.offset_dt, vec_oprnd0); @@ -6616,8 +6617,9 @@ vectorizable_store (gimple *stmt, gimple vec_oprnd = vec_oprnds[j]; else { - vect_is_simple_use (vec_oprnd, vinfo, &def_stmt, &dt); - vec_oprnd = vect_get_vec_def_for_stmt_copy (dt, vec_oprnd); + vect_is_simple_use (op, vinfo, &def_stmt, &rhs_dt); + vec_oprnd = vect_get_vec_def_for_stmt_copy (rhs_dt, + vec_oprnd); } } /* Pun the vector to extract from if necessary. */ @@ -6858,26 +6860,19 @@ vectorizable_store (gimple *stmt, gimple for (i = 0; i < group_size; i++) { op = oprnds[i]; - vect_is_simple_use (op, vinfo, &def_stmt, &dt); - vec_oprnd = vect_get_vec_def_for_stmt_copy (dt, op); + vect_is_simple_use (op, vinfo, &def_stmt, &rhs_dt); + vec_oprnd = vect_get_vec_def_for_stmt_copy (rhs_dt, op); dr_chain[i] = vec_oprnd; oprnds[i] = vec_oprnd; } if (mask) - { - vect_is_simple_use (vec_mask, vinfo, &def_stmt, &dt); - vec_mask = vect_get_vec_def_for_stmt_copy (dt, vec_mask); - } + vec_mask = vect_get_vec_def_for_stmt_copy (mask_dt, vec_mask); if (dataref_offset) dataref_offset = int_const_binop (PLUS_EXPR, dataref_offset, bump); else if (STMT_VINFO_GATHER_SCATTER_P (stmt_info)) - { - gimple *def_stmt; - vect_def_type dt; - vect_is_simple_use (vec_offset, loop_vinfo, &def_stmt, &dt); - vec_offset = vect_get_vec_def_for_stmt_copy (dt, vec_offset); - } + vec_offset = vect_get_vec_def_for_stmt_copy (gs_info.offset_dt, + vec_offset); else dataref_ptr = bump_vector_ptr (dataref_ptr, ptr_incr, gsi, stmt, bump); @@ -7238,6 +7233,7 @@ vectorizable_load (gimple *stmt, gimple_ gather_scatter_info gs_info; vec_info *vinfo = stmt_info->vinfo; tree ref_type; + enum vect_def_type mask_dt = vect_unknown_def_type; if (!STMT_VINFO_RELEVANT_P (stmt_info) && !bb_vinfo) return false; @@ -7290,7 +7286,8 @@ vectorizable_load (gimple *stmt, gimple_ if (mask_index >= 0) { mask = gimple_call_arg (call, mask_index); - if (!vect_check_load_store_mask (stmt, mask, &mask_vectype)) + if (!vect_check_load_store_mask (stmt, mask, &mask_dt, + &mask_vectype)) return false; } } @@ -7472,7 +7469,8 @@ vectorizable_load (gimple *stmt, gimple_ if (memory_access_type == VMAT_GATHER_SCATTER && gs_info.decl) { - vect_build_gather_load_calls (stmt, gsi, vec_stmt, &gs_info, mask); + vect_build_gather_load_calls (stmt, gsi, vec_stmt, &gs_info, mask, + mask_dt); return true; } @@ -7999,22 +7997,13 @@ vectorizable_load (gimple *stmt, gimple_ dataref_offset = int_const_binop (PLUS_EXPR, dataref_offset, bump); else if (STMT_VINFO_GATHER_SCATTER_P (stmt_info)) - { - gimple *def_stmt; - vect_def_type dt; - vect_is_simple_use (vec_offset, loop_vinfo, &def_stmt, &dt); - vec_offset = vect_get_vec_def_for_stmt_copy (dt, vec_offset); - } + vec_offset = vect_get_vec_def_for_stmt_copy (gs_info.offset_dt, + vec_offset); else dataref_ptr = bump_vector_ptr (dataref_ptr, ptr_incr, gsi, stmt, bump); if (mask) - { - gimple *def_stmt; - vect_def_type dt; - vect_is_simple_use (vec_mask, vinfo, &def_stmt, &dt); - vec_mask = vect_get_vec_def_for_stmt_copy (dt, vec_mask); - } + vec_mask = vect_get_vec_def_for_stmt_copy (mask_dt, vec_mask); } if (grouped_load || slp_perm) Index: gcc/testsuite/gcc.dg/torture/pr83940.c =================================================================== --- /dev/null 2018-01-20 10:07:57.564678031 +0000 +++ gcc/testsuite/gcc.dg/torture/pr83940.c 2018-01-20 10:31:16.690598830 +0000 @@ -0,0 +1,9 @@ +/* { dg-do compile } */ +/* { dg-options "-mavx512f" { target { i?86-*-* x86_64-*-* } } } */ + +void +foo (double *a[], int b) +{ + for (; b; b++) + a[b][b] = 1.0; +}