From patchwork Fri Nov 17 09:17:13 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Sandiford X-Patchwork-Id: 119102 Delivered-To: patch@linaro.org Received: by 10.140.22.164 with SMTP id 33csp279579qgn; Fri, 17 Nov 2017 01:17:47 -0800 (PST) X-Google-Smtp-Source: AGs4zMZSo2T7j7Gbha6F3XTVpySbrKaGJw7LqB7iN6TqYfQcN1+ZsNeLKzx8BthDTewsbpTvyOna X-Received: by 10.98.155.129 with SMTP id e1mr1390099pfk.119.1510910267726; Fri, 17 Nov 2017 01:17:47 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1510910267; cv=none; d=google.com; s=arc-20160816; b=p2XCB02THd8ZwLE9kGLTjQkNaWItfRq9PTdbXIoKcWSao7BSkJUwfeSK2F/Qm1jpjd uPGivPnkszt+yv8P2DUhzHtxPfmBCnzWnsjTuSHGvqKSMtTkJu+RglRCGz1UGRdfdDl3 VWF1fNY27cTOo1UHe+Itdp7EWwmNZszfwql2dog6d0bwILU5QY/c+DrbLovk9qjw/2PO gzARGFUNEl5oIFfWEPU8gNYAYOiUGCC6G3g3ezDKr7vJSub2bwvde32EatJo4YlcK9RR +xCyCtEWlgUcfF4LhVmSVumiJCz8BhtALjTWqGySDj1+ckCG38g6jVUrVykPH+P3rrkw vdBw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:user-agent:message-id:in-reply-to:date:references :subject:mail-followup-to:to:from:delivered-to:sender:list-help :list-post:list-archive:list-unsubscribe:list-id:precedence :mailing-list:dkim-signature:domainkey-signature :arc-authentication-results; bh=61wdMiA6r44t1ev5nspfk68LJHRGs89yL6ZOPkDjmFw=; b=MHk7aVOcs+3fe6Seva3POyjo1Uu/0d5c/+s6LigbUaR5Atj/0dxCgtWWjXzuGojQ+z qOJypA7mQip6xPO/kstVdID6Lc32vyWTBXF/zaj5QzAqZZZsT36Xf7uy6I720m1fCKx7 1kieaN2lIMs8AHxo2p0gfa1yvnrf/dnhvd6gLCVIyzW+s7gSkhlB2oHx2CxHFsDRP5I3 TlILQtRCgdaG3G4sqwB6KH704V8j1iN+iapmwM1+6IhWn1s8YQss8nLGOjoTTVCxvFUi fNY04OLraEFWVsMGHy55hCwfLdaUP02L1SsicAmBEwlYvgE1IxulbiQC/aS5h570M9s1 if3w== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gcc.gnu.org header.s=default header.b=GHBUtn54; spf=pass (google.com: domain of gcc-patches-return-467093-patch=linaro.org@gcc.gnu.org designates 209.132.180.131 as permitted sender) smtp.mailfrom=gcc-patches-return-467093-patch=linaro.org@gcc.gnu.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from sourceware.org (server1.sourceware.org. [209.132.180.131]) by mx.google.com with ESMTPS id w24si219391pll.440.2017.11.17.01.17.47 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 17 Nov 2017 01:17:47 -0800 (PST) Received-SPF: pass (google.com: domain of gcc-patches-return-467093-patch=linaro.org@gcc.gnu.org designates 209.132.180.131 as permitted sender) client-ip=209.132.180.131; Authentication-Results: mx.google.com; dkim=pass header.i=@gcc.gnu.org header.s=default header.b=GHBUtn54; spf=pass (google.com: domain of gcc-patches-return-467093-patch=linaro.org@gcc.gnu.org designates 209.132.180.131 as permitted sender) smtp.mailfrom=gcc-patches-return-467093-patch=linaro.org@gcc.gnu.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org DomainKey-Signature: a=rsa-sha1; c=nofws; d=gcc.gnu.org; h=list-id :list-unsubscribe:list-archive:list-post:list-help:sender:from :to:subject:references:date:in-reply-to:message-id:mime-version :content-type; q=dns; s=default; b=tac8FFMYa9jFAl3zMsTGCzD2f7asd kHBu4qpOL4C/jHo3Enp93ZjM2/d+IYzgFfCpinpxpYVh3B4+dWUofdTfIVOoAyZC +6sujvUfZxJJl7c6s1spx8t70tIcbONwxhQ3c2kjTAiQIDbscqSWnk4uVoZOMkwp iFuYVL2sT4rYk4= DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=gcc.gnu.org; h=list-id :list-unsubscribe:list-archive:list-post:list-help:sender:from :to:subject:references:date:in-reply-to:message-id:mime-version :content-type; s=default; bh=RShhNvO78mLqbjTgWil49d2W2yI=; b=GHB Utn54fS4h1UXGkDNCau9nhMIZh8HGB5ECxfSY8FUDD87ZxtpD59223jgveJJi6AA y0Uj5POEXUDvxsD6GSRjhQlvIawaxtvDdOLibR8lWUxtgAX5JIl+cwpP3L77BP7O Plh7Kw7CBKpSqkYpDy5/eEaePup1pQ0FfSchJ1QA= Received: (qmail 112224 invoked by alias); 17 Nov 2017 09:17:18 -0000 Mailing-List: contact gcc-patches-help@gcc.gnu.org; run by ezmlm Precedence: bulk List-Id: List-Unsubscribe: List-Archive: List-Post: List-Help: Sender: gcc-patches-owner@gcc.gnu.org Delivered-To: mailing list gcc-patches@gcc.gnu.org Received: (qmail 112214 invoked by uid 89); 17 Nov 2017 09:17:17 -0000 Authentication-Results: sourceware.org; auth=none X-Virus-Found: No X-Spam-SWARE-Status: No, score=-10.7 required=5.0 tests=AWL, BAYES_00, GIT_PATCH_2, GIT_PATCH_3, KAM_ASCII_DIVIDERS, KB_WAM_FROM_NAME_SINGLEWORD, RCVD_IN_DNSWL_NONE, SPF_PASS autolearn=ham version=3.3.2 spammy=HX-Received:Fri, UD:tree-vect-stmts.c, UD:tree-vect-loop.c X-HELO: mail-wr0-f173.google.com Received: from mail-wr0-f173.google.com (HELO mail-wr0-f173.google.com) (209.85.128.173) by sourceware.org (qpsmtpd/0.93/v0.84-503-g423c35a) with ESMTP; Fri, 17 Nov 2017 09:17:16 +0000 Received: by mail-wr0-f173.google.com with SMTP id u98so1490586wrb.13 for ; Fri, 17 Nov 2017 01:17:15 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:mail-followup-to:subject:references:date :in-reply-to:message-id:user-agent:mime-version; bh=61wdMiA6r44t1ev5nspfk68LJHRGs89yL6ZOPkDjmFw=; b=dSGp73v0eIPiZ5TEOGri28C4SlvMsCxSy0UfEl+iNVESfsx9LoosQsMMLkZuJ7AqrD A3v/gTTsN2sL2Fcvox5h2ivbNTZUKJ3CO3aHFuBZHVdRHRm93Cf7Lay+zNcFBbdW4dMx KSg007qBJ5DUBWwJ/y5HEqYIG4AnmGIVvYlx5Z10i9zcc+zV1Qp7Swvm2VIelRtnFiPu 1DRMuYNqeUpR0nWXoJpMTdJfUFYBKWdPRDYHj7pI0wlMuLx8bp0yn+KtHDv2SIK3rINa JghbQjMghjFwm0Dfp4cYSlXOI4NNam9YadonM8NWwikIsK9lElb+H8oqnFqAK0dOWkQy pVjg== X-Gm-Message-State: AJaThX5o1YT3dJuWwxBKoUp3JgXHG2qj/WPz88gbSvR8pli19Bj5b90O 3/T7Z7UhtjKHvvaOsjqyAUQ5tHXqUsQ= X-Received: by 10.223.135.121 with SMTP id 54mr3744955wrz.160.1510910233716; Fri, 17 Nov 2017 01:17:13 -0800 (PST) Received: from localhost ([2.25.234.120]) by smtp.gmail.com with ESMTPSA id c54sm5187711wra.84.2017.11.17.01.17.12 for (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Fri, 17 Nov 2017 01:17:12 -0800 (PST) From: Richard Sandiford To: gcc-patches@gcc.gnu.org Mail-Followup-To: gcc-patches@gcc.gnu.org, richard.sandiford@linaro.org Subject: [1/7] Move code that stubs out IFN_MASK_LOADs References: <87375d2rkr.fsf@linaro.org> Date: Fri, 17 Nov 2017 09:17:13 +0000 In-Reply-To: <87375d2rkr.fsf@linaro.org> (Richard Sandiford's message of "Fri, 17 Nov 2017 09:16:20 +0000") Message-ID: <87y3n51cyu.fsf@linaro.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/25.3 (gnu/linux) MIME-Version: 1.0 vectorizable_mask_load_store replaces scalar IFN_MASK_LOAD calls with dummy assignments, so that they never survive vectorisation. This patch moves the code to vect_transform_loop instead, so that we only change the scalar statements once all of them have been vectorised. This makes it easier to handle other types of functions that need stubbing out, and also makes it easier to handle groups and patterns. Richard 2017-11-17 Richard Sandiford gcc/ * tree-vect-loop.c (vect_transform_loop): Stub out scalar IFN_MASK_LOAD calls here rather than... * tree-vect-stmts.c (vectorizable_mask_load_store): ...here. Index: gcc/tree-vect-loop.c =================================================================== --- gcc/tree-vect-loop.c 2017-11-17 09:06:36.818836579 +0000 +++ gcc/tree-vect-loop.c 2017-11-17 09:06:42.552270752 +0000 @@ -7743,6 +7743,25 @@ vect_transform_loop (loop_vec_info loop_ gsi_next (&si); } } /* stmts in BB */ + + /* Stub out scalar statements that must not survive vectorization. + Doing this here helps with grouped statements, or statements that + are involved in patterns. */ + for (gimple_stmt_iterator gsi = gsi_start_bb (bb); + !gsi_end_p (gsi); gsi_next (&gsi)) + { + gcall *call = dyn_cast (gsi_stmt (gsi)); + if (call && gimple_call_internal_p (call, IFN_MASK_LOAD)) + { + tree lhs = gimple_get_lhs (call); + if (!VECTOR_TYPE_P (TREE_TYPE (lhs))) + { + tree zero = build_zero_cst (TREE_TYPE (lhs)); + gimple *new_stmt = gimple_build_assign (lhs, zero); + gsi_replace (&gsi, new_stmt, true); + } + } + } } /* BBs in loop */ /* The vectorization factor is always > 1, so if we use an IV increment of 1. Index: gcc/tree-vect-stmts.c =================================================================== --- gcc/tree-vect-stmts.c 2017-11-17 09:06:36.818836579 +0000 +++ gcc/tree-vect-stmts.c 2017-11-17 09:06:42.553230162 +0000 @@ -2342,20 +2342,6 @@ vectorizable_mask_load_store (gimple *st STMT_VINFO_RELATED_STMT (prev_stmt_info) = new_stmt; prev_stmt_info = vinfo_for_stmt (new_stmt); } - - /* Ensure that even with -fno-tree-dce the scalar MASK_LOAD is removed - from the IL. */ - if (STMT_VINFO_RELATED_STMT (stmt_info)) - { - stmt = STMT_VINFO_RELATED_STMT (stmt_info); - stmt_info = vinfo_for_stmt (stmt); - } - tree lhs = gimple_call_lhs (stmt); - new_stmt = gimple_build_assign (lhs, build_zero_cst (TREE_TYPE (lhs))); - set_vinfo_for_stmt (new_stmt, stmt_info); - set_vinfo_for_stmt (stmt, NULL); - STMT_VINFO_STMT (stmt_info) = new_stmt; - gsi_replace (gsi, new_stmt, true); return true; } else if (vls_type != VLS_LOAD) @@ -2472,23 +2458,6 @@ vectorizable_mask_load_store (gimple *st } } - if (vls_type == VLS_LOAD) - { - /* Ensure that even with -fno-tree-dce the scalar MASK_LOAD is removed - from the IL. */ - if (STMT_VINFO_RELATED_STMT (stmt_info)) - { - stmt = STMT_VINFO_RELATED_STMT (stmt_info); - stmt_info = vinfo_for_stmt (stmt); - } - tree lhs = gimple_call_lhs (stmt); - new_stmt = gimple_build_assign (lhs, build_zero_cst (TREE_TYPE (lhs))); - set_vinfo_for_stmt (new_stmt, stmt_info); - set_vinfo_for_stmt (stmt, NULL); - STMT_VINFO_STMT (stmt_info) = new_stmt; - gsi_replace (gsi, new_stmt, true); - } - return true; } From patchwork Fri Nov 17 09:17:38 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Sandiford X-Patchwork-Id: 119103 Delivered-To: patch@linaro.org Received: by 10.140.22.164 with SMTP id 33csp280044qgn; Fri, 17 Nov 2017 01:18:25 -0800 (PST) X-Google-Smtp-Source: AGs4zMZD4OgR4RKA7acZnYWcab4vQnjh1YKX9wbHVlunWn9QAqDiu02Jnb66fc7ZoDXqvRErVXrt X-Received: by 10.99.110.3 with SMTP id j3mr4531374pgc.70.1510910305530; Fri, 17 Nov 2017 01:18:25 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1510910305; cv=none; d=google.com; s=arc-20160816; b=NAvfUnVdTyPDwL5GqZh7MUXaR7DQp1KXiBoDIcG9OT7oXAYH6wmsUG5VBFuU7nn+Y7 g/yVPswM22RowYjt3dqE2vWmSegTs7Bf2N7mbMS7ejgLHx6r8oWCarGZ9naZCHfFBRPY OZDeQ1KUbEnimDz2gjv4Atvmrpxi0AYqWRW5d7xCkqsWrx9C63nmBR4k7uIM6auXvrb0 xnFiuCgJxCzRld/+A7QiNW02zDp93aMCnJUZCaNB3qO3vmz5ovv1ryiXiRf2Io5g7Y+I Y1bzKO515UBfF25fiE1pxePVuFh2zP6i3uNxt4YkuVKXHR28sxVCpLAFqXqDo+zs3ys0 dldA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:user-agent:message-id:in-reply-to:date:references :subject:mail-followup-to:to:from:delivered-to:sender:list-help :list-post:list-archive:list-unsubscribe:list-id:precedence :mailing-list:dkim-signature:domainkey-signature :arc-authentication-results; bh=5xpu6IBlTTFJbsxttIa2kxM8Sg3qBYFgikUI32h+eAM=; b=wAne46QFtSAbNX4y+gaQPevXfpZRYzLXMFEdCPz+VK/hgIQikRI1rlLn4a3oPX0Dy3 X0/UXbvQL1cLW9yVJjobIlqcBOPaqJnbuXxybK4Mzs55qh72FTGAzG0DaT3AMDiF91Gm RC9Cg0ubHZhCQtQletv3z8+YJ8CcdSe0Rn0QihU0Ex59pPAgBTvJoBO8cLUi9TltNrF3 eSoqjTBZTSFLt2djoRPRwkCtpAMnoIo4BNkJtnZOEjsAFMcaek6dOlFrrF/nfBvMa/80 L9GT4R0qt/yeMSK2j9k91X8XRgIGYQVtZntkMhYAqv8iNywhzWOpI+30/By3Ec8f3Onn p4wg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gcc.gnu.org header.s=default header.b=RWIRDARC; spf=pass (google.com: domain of gcc-patches-return-467094-patch=linaro.org@gcc.gnu.org designates 209.132.180.131 as permitted sender) smtp.mailfrom=gcc-patches-return-467094-patch=linaro.org@gcc.gnu.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from sourceware.org (server1.sourceware.org. [209.132.180.131]) by mx.google.com with ESMTPS id p189si2680231pfp.251.2017.11.17.01.18.25 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 17 Nov 2017 01:18:25 -0800 (PST) Received-SPF: pass (google.com: domain of gcc-patches-return-467094-patch=linaro.org@gcc.gnu.org designates 209.132.180.131 as permitted sender) client-ip=209.132.180.131; Authentication-Results: mx.google.com; dkim=pass header.i=@gcc.gnu.org header.s=default header.b=RWIRDARC; spf=pass (google.com: domain of gcc-patches-return-467094-patch=linaro.org@gcc.gnu.org designates 209.132.180.131 as permitted sender) smtp.mailfrom=gcc-patches-return-467094-patch=linaro.org@gcc.gnu.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org DomainKey-Signature: a=rsa-sha1; c=nofws; d=gcc.gnu.org; h=list-id :list-unsubscribe:list-archive:list-post:list-help:sender:from :to:subject:references:date:in-reply-to:message-id:mime-version :content-type; q=dns; s=default; b=MqmvuXfTWse4z2BcFiEl+ckbUSFqI XJQR/sV27NkLLIyHR2Qc9f025g1bMQFKXZzdOaXNpfzP/W0UpkY3w/zxgo/QBKJb eib01uPK4aXgNZTCpBTHojueqRbOKnWgujL9aIfdTLy4pdAg9HTjQ9EKWQy10OWf gpwLdEaJJS/gjk= DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=gcc.gnu.org; h=list-id :list-unsubscribe:list-archive:list-post:list-help:sender:from :to:subject:references:date:in-reply-to:message-id:mime-version :content-type; s=default; bh=8QQcWZmyZGTqGxCkI5n57wk5NLY=; b=RWI RDARCAcdBaexvXiAczXNf/GBI8NoCdzl/qGwWObgf10Kxrl8I0lE4AC6eO4qXx5X 2IOZyTLMpJi7Wu3dzj0+32MRznsbzZgqs43+89QFHz4RRceWUggS0388LoWEZkG/ c0AosLxs7eR1VZokRxJFsnHuZZb18l8ojuEtHcR8= Received: (qmail 113584 invoked by alias); 17 Nov 2017 09:17:46 -0000 Mailing-List: contact gcc-patches-help@gcc.gnu.org; run by ezmlm Precedence: bulk List-Id: List-Unsubscribe: List-Archive: List-Post: List-Help: Sender: gcc-patches-owner@gcc.gnu.org Delivered-To: mailing list gcc-patches@gcc.gnu.org Received: (qmail 113516 invoked by uid 89); 17 Nov 2017 09:17:45 -0000 Authentication-Results: sourceware.org; auth=none X-Virus-Found: No X-Spam-SWARE-Status: No, score=-10.7 required=5.0 tests=AWL, BAYES_00, GIT_PATCH_2, GIT_PATCH_3, KAM_ASCII_DIVIDERS, KB_WAM_FROM_NAME_SINGLEWORD, RCVD_IN_DNSWL_NONE, SPF_PASS autolearn=ham version=3.3.2 spammy=wash, 9077, 21697, 12207 X-HELO: mail-wr0-f177.google.com Received: from mail-wr0-f177.google.com (HELO mail-wr0-f177.google.com) (209.85.128.177) by sourceware.org (qpsmtpd/0.93/v0.84-503-g423c35a) with ESMTP; Fri, 17 Nov 2017 09:17:43 +0000 Received: by mail-wr0-f177.google.com with SMTP id z75so165957wrc.5 for ; Fri, 17 Nov 2017 01:17:42 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:mail-followup-to:subject:references:date :in-reply-to:message-id:user-agent:mime-version; bh=5xpu6IBlTTFJbsxttIa2kxM8Sg3qBYFgikUI32h+eAM=; b=BesEGK3xg//HO9TUS5LEfsTkhCqKWKamMNU3D/Tls61SICZt/BJX1VMiHBQXpdO8Rm Tqo0BifsVbqHEBgm6/4N3sLFXrUmTv2YG6xUr3eEQy+Bmv/Rl84aVwf8VhlnEugCT3Yo oLxXABdj+fFwo9sQCHjWBhP/QVYo/dkjdFAgO8cfZUV9gWysr/QNxPeJaim4sV3qQB+f 4vs9cqCsRqS0B/5bJiuCR0IQkz6sGEngwgRv9I68KxlguvnckXe3BuePOkOLxpBD684B iaNh3uBKqyApdTG7uj/j32yJjzYAvca+1IHPULzrIiarxQEzOTnpLzcuAzQcxmPOG8EQ DoRA== X-Gm-Message-State: AJaThX7aTUNIwD1VZFovfa2zKxXJtWi5orpyefMXFjarg9cZAsrsJFyb 6NtBuOH46H2xKLeC/kdmFL4hRE0E2ew= X-Received: by 10.223.163.216 with SMTP id m24mr3709139wrb.107.1510910260970; Fri, 17 Nov 2017 01:17:40 -0800 (PST) Received: from localhost ([2.25.234.120]) by smtp.gmail.com with ESMTPSA id x63sm3330281wma.39.2017.11.17.01.17.40 for (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Fri, 17 Nov 2017 01:17:40 -0800 (PST) From: Richard Sandiford To: gcc-patches@gcc.gnu.org Mail-Followup-To: gcc-patches@gcc.gnu.org, richard.sandiford@linaro.org Subject: [2/7] Make vect_model_store_cost take a vec_load_store_type References: <87375d2rkr.fsf@linaro.org> Date: Fri, 17 Nov 2017 09:17:38 +0000 In-Reply-To: <87375d2rkr.fsf@linaro.org> (Richard Sandiford's message of "Fri, 17 Nov 2017 09:16:20 +0000") Message-ID: <87tvxt1cy5.fsf@linaro.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/25.3 (gnu/linux) MIME-Version: 1.0 This patch makes vect_model_store_cost take a vec_load_store_type instead of a vect_def_type. It's a wash on its own, but it helps with later patches. Richard 2017-11-17 Richard Sandiford gcc/ * tree-vectorizer.h (vec_load_store_type): Moved from tree-vec-stmts.c (vect_model_store_cost): Take a vec_load_store_type instead of a vect_def_type. * tree-vect-stmts.c (vec_load_store_type): Move to tree-vectorizer.h. (vect_model_store_cost): Take a vec_load_store_type instead of a vect_def_type. (vectorizable_mask_load_store): Update accordingly. (vectorizable_store): Likewise. * tree-vect-slp.c (vect_analyze_slp_cost_1): Update accordingly. Index: gcc/tree-vectorizer.h =================================================================== --- gcc/tree-vectorizer.h 2017-11-17 09:06:36.818836579 +0000 +++ gcc/tree-vectorizer.h 2017-11-17 09:06:46.238323981 +0000 @@ -520,6 +520,14 @@ enum slp_vect_type { hybrid }; +/* Says whether a statement is a load, a store of a vectorized statement + result, or a store of an invariant value. */ +enum vec_load_store_type { + VLS_LOAD, + VLS_STORE, + VLS_STORE_INVARIANT +}; + /* Describes how we're going to vectorize an individual load or store, or a group of loads or stores. */ enum vect_memory_access_type { @@ -1220,7 +1228,7 @@ extern void vect_model_simple_cost (stmt int, stmt_vector_for_cost *, stmt_vector_for_cost *); extern void vect_model_store_cost (stmt_vec_info, int, vect_memory_access_type, - enum vect_def_type, slp_tree, + vec_load_store_type, slp_tree, stmt_vector_for_cost *, stmt_vector_for_cost *); extern void vect_model_load_cost (stmt_vec_info, int, vect_memory_access_type, Index: gcc/tree-vect-stmts.c =================================================================== --- gcc/tree-vect-stmts.c 2017-11-17 09:06:42.553230162 +0000 +++ gcc/tree-vect-stmts.c 2017-11-17 09:06:46.238323981 +0000 @@ -52,14 +52,6 @@ Software Foundation; either version 3, o /* For lang_hooks.types.type_for_mode. */ #include "langhooks.h" -/* Says whether a statement is a load, a store of a vectorized statement - result, or a store of an invariant value. */ -enum vec_load_store_type { - VLS_LOAD, - VLS_STORE, - VLS_STORE_INVARIANT -}; - /* Return the vectorized type for the given statement. */ tree @@ -906,7 +898,7 @@ vect_model_promotion_demotion_cost (stmt void vect_model_store_cost (stmt_vec_info stmt_info, int ncopies, vect_memory_access_type memory_access_type, - enum vect_def_type dt, slp_tree slp_node, + vec_load_store_type vls_type, slp_tree slp_node, stmt_vector_for_cost *prologue_cost_vec, stmt_vector_for_cost *body_cost_vec) { @@ -915,7 +907,7 @@ vect_model_store_cost (stmt_vec_info stm gimple *first_stmt = STMT_VINFO_STMT (stmt_info); bool grouped_access_p = STMT_VINFO_GROUPED_ACCESS (stmt_info); - if (dt == vect_constant_def || dt == vect_external_def) + if (vls_type == VLS_STORE_INVARIANT) prologue_cost += record_stmt_cost (prologue_cost_vec, 1, scalar_to_vec, stmt_info, 0, vect_prologue); @@ -2169,7 +2161,7 @@ vectorizable_mask_load_store (gimple *st NULL, NULL, NULL); else vect_model_store_cost (stmt_info, ncopies, memory_access_type, - dt, NULL, NULL, NULL); + vls_type, NULL, NULL, NULL); return true; } gcc_assert (memory_access_type == STMT_VINFO_MEMORY_ACCESS_TYPE (stmt_info)); @@ -5797,8 +5789,8 @@ vectorizable_store (gimple *stmt, gimple STMT_VINFO_TYPE (stmt_info) = store_vec_info_type; /* The SLP costs are calculated during SLP analysis. */ if (!PURE_SLP_STMT (stmt_info)) - vect_model_store_cost (stmt_info, ncopies, memory_access_type, dt, - NULL, NULL, NULL); + vect_model_store_cost (stmt_info, ncopies, memory_access_type, + vls_type, NULL, NULL, NULL); return true; } gcc_assert (memory_access_type == STMT_VINFO_MEMORY_ACCESS_TYPE (stmt_info)); Index: gcc/tree-vect-slp.c =================================================================== --- gcc/tree-vect-slp.c 2017-11-17 09:06:36.818836579 +0000 +++ gcc/tree-vect-slp.c 2017-11-17 09:06:46.237364571 +0000 @@ -1724,7 +1724,7 @@ vect_analyze_slp_cost_1 (slp_instance in : VMAT_CONTIGUOUS); if (DR_IS_WRITE (STMT_VINFO_DATA_REF (stmt_info))) vect_model_store_cost (stmt_info, ncopies_for_cost, - memory_access_type, vect_uninitialized_def, + memory_access_type, VLS_STORE, node, prologue_cost_vec, body_cost_vec); else { From patchwork Fri Nov 17 09:18:19 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Sandiford X-Patchwork-Id: 119104 Delivered-To: patch@linaro.org Received: by 10.140.22.164 with SMTP id 33csp281091qgn; Fri, 17 Nov 2017 01:19:35 -0800 (PST) X-Google-Smtp-Source: AGs4zMbmNgwuyyBh+oUaplQytTpi2lq+soEaFLfD4hVAl8l23vY8hrsJPRij+MR9HSmosrsNQM6I X-Received: by 10.159.198.73 with SMTP id y9mr4682894plt.334.1510910375089; Fri, 17 Nov 2017 01:19:35 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1510910375; cv=none; d=google.com; s=arc-20160816; b=pCmVvDiWWAn8s8U9zEvusAvxbNYotdabVIXZ6L/MZ9Z5UbpRccmmbFaFujF4Ggf1/h ZXfpwzf9Rqp4otC6bLC3V2zW/Grys8svniJhkDRUhK7VbrZLeQv4hH5xaR/4jP7an5xz oOOu4SDMErM9/R2tTrny6oqq2CUidRWM6iFFEhWAPKhO4qpO6N9l54YY9HOHftcj4Bev N6oUrbwYlLIngBgIxpRTVFPOAtFN88cTqOcSOLHTZ2dQQTqpi5rTVW7Uo21lvRrPl1Bs X86/3oLLPXIc5ilh9yi2DkkjXAuPja8ioJBDdh1kF2aN0h4nl/j8HFa5/xWgZuGBXecz 2Mqw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:user-agent:message-id:in-reply-to:date:references :subject:mail-followup-to:to:from:delivered-to:sender:list-help :list-post:list-archive:list-unsubscribe:list-id:precedence :mailing-list:dkim-signature:domainkey-signature :arc-authentication-results; bh=9PYpMZqa9knvQ7ffhXXJpH6aWjgfk9WgoWH5MLgdWHI=; b=PM0mG6lekNEZepI7jQQ51UKDlt1971IO/DDusMS6ULylhKDBrUCwaStaoe6hDTOPB0 BRkcLZQ41SX0xCXq88VMqePZ8k3lY2fVSZpFTDOY+RqEQqhc/nrKSRnq2IAHheCWoxgT bkYo6HY/xCDOktitIpYpUmA8DbsE9JC32VNb2otCA+ZdQyVVR09Ek18BRmI12qjtKWuY 3lYwgFdahQnWUy4ur5nZx+L/6N//Jb7jQnrGO0zj4S29t8jqo0OO5zlFToQncSu3rdHO yXhBIgFw1ItM5mIFAO3TnODDTGa1dxbvgX4hlwr0ANdRccYZ0Y/7GvdZQXRYPv/woELA kxbg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gcc.gnu.org header.s=default header.b=FFJ3LMjh; spf=pass (google.com: domain of gcc-patches-return-467096-patch=linaro.org@gcc.gnu.org designates 209.132.180.131 as permitted sender) smtp.mailfrom=gcc-patches-return-467096-patch=linaro.org@gcc.gnu.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from sourceware.org (server1.sourceware.org. [209.132.180.131]) by mx.google.com with ESMTPS id g192si2694826pfc.29.2017.11.17.01.19.34 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 17 Nov 2017 01:19:35 -0800 (PST) Received-SPF: pass (google.com: domain of gcc-patches-return-467096-patch=linaro.org@gcc.gnu.org designates 209.132.180.131 as permitted sender) client-ip=209.132.180.131; Authentication-Results: mx.google.com; dkim=pass header.i=@gcc.gnu.org header.s=default header.b=FFJ3LMjh; spf=pass (google.com: domain of gcc-patches-return-467096-patch=linaro.org@gcc.gnu.org designates 209.132.180.131 as permitted sender) smtp.mailfrom=gcc-patches-return-467096-patch=linaro.org@gcc.gnu.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org DomainKey-Signature: a=rsa-sha1; c=nofws; d=gcc.gnu.org; h=list-id :list-unsubscribe:list-archive:list-post:list-help:sender:from :to:subject:references:date:in-reply-to:message-id:mime-version :content-type; q=dns; s=default; b=QOJKQLuuhFeCfZ/ZoXfcFHinWd4rw RVbUZrQVwIWrDX8AElNEDJ5RRmSF7lQ6p4W65GqSX0Qm4DDj4gnQglq92PMiPNYy rzpGzEtmGNOvP2EidVKs7buT++DjyMYK0pv2JQJu3trOeN5kmD8AD5Hy+SNyNiFX R4U2tLE5Q/5l9o= DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=gcc.gnu.org; h=list-id :list-unsubscribe:list-archive:list-post:list-help:sender:from :to:subject:references:date:in-reply-to:message-id:mime-version :content-type; s=default; bh=MNrLj6uP9maTynP8UAcWLdIj4ow=; b=FFJ 3LMjhcUWRA6uTecPuEzCA9WDOU6k/N3PmpwD4DcrFtS1PRvPngZp86CPbQ6+I9Ve iaK1xlVuyrWI8zm8+LcIlsD6RUPe+khROoOiyzFyJBH8I8EYDTQaJAmxkZm9OMI2 3K2bGDYFQh2Oae8o8VfxqckSY5TI7PjaA+8U7kJs= Received: (qmail 116380 invoked by alias); 17 Nov 2017 09:18:24 -0000 Mailing-List: contact gcc-patches-help@gcc.gnu.org; run by ezmlm Precedence: bulk List-Id: List-Unsubscribe: List-Archive: List-Post: List-Help: Sender: gcc-patches-owner@gcc.gnu.org Delivered-To: mailing list gcc-patches@gcc.gnu.org Received: (qmail 116345 invoked by uid 89); 17 Nov 2017 09:18:23 -0000 Authentication-Results: sourceware.org; auth=none X-Virus-Found: No X-Spam-SWARE-Status: No, score=-11.5 required=5.0 tests=AWL, BAYES_00, GIT_PATCH_2, GIT_PATCH_3, KB_WAM_FROM_NAME_SINGLEWORD, RCVD_IN_DNSWL_NONE, SPF_PASS autolearn=ham version=3.3.2 spammy= X-HELO: mail-wm0-f43.google.com Received: from mail-wm0-f43.google.com (HELO mail-wm0-f43.google.com) (74.125.82.43) by sourceware.org (qpsmtpd/0.93/v0.84-503-g423c35a) with ESMTP; Fri, 17 Nov 2017 09:18:21 +0000 Received: by mail-wm0-f43.google.com with SMTP id v186so5014545wma.2 for ; Fri, 17 Nov 2017 01:18:21 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:mail-followup-to:subject:references:date :in-reply-to:message-id:user-agent:mime-version; bh=9PYpMZqa9knvQ7ffhXXJpH6aWjgfk9WgoWH5MLgdWHI=; b=gSzvZlaYzOvLnsLBuqNTJLbSl1t/zk/eaEG9HWgb7d7rdKQmX6D03Og8EmKEM8ZB06 v+XR4SlPOPy3gmLZDruLoKYIaLVzzCojSuth3W2AqEywSMDIEOMFU4DqENgDFPVfnqf1 qT9d2IECp7A1YhebIb6kXcm/3GMXgxrmyQckOMJG1Spiz194Gcen0xz2L/yRL17f2GTq VNyWJmieqQYt+Ii2YnB0YdRMQs+TKvF22rZCEDqisZv23f2chScCPMmxzlHOxhOPH8Yp QG/ff7RaJ4gJilsnguuz/hWCFC5TriP3pmbcNeCIicQEPRbMzHpoKpUiRYyvBelDUr4D uK2A== X-Gm-Message-State: AJaThX4IXYVD3iDXVgH686ztrNFCX/OLXA+r37Boe9UFecNZ6iKBPXQJ CVb8RVaVhRqUNpHxipnJ5D3+VWAN/t8= X-Received: by 10.28.247.15 with SMTP id v15mr3446759wmh.152.1510910299222; Fri, 17 Nov 2017 01:18:19 -0800 (PST) Received: from localhost ([2.25.234.120]) by smtp.gmail.com with ESMTPSA id o20sm3114785wro.6.2017.11.17.01.18.18 for (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Fri, 17 Nov 2017 01:18:18 -0800 (PST) From: Richard Sandiford To: gcc-patches@gcc.gnu.org Mail-Followup-To: gcc-patches@gcc.gnu.org, richard.sandiford@linaro.org Subject: [3/7] Split mask checking out of vectorizable_mask_load_store References: <87375d2rkr.fsf@linaro.org> Date: Fri, 17 Nov 2017 09:18:19 +0000 In-Reply-To: <87375d2rkr.fsf@linaro.org> (Richard Sandiford's message of "Fri, 17 Nov 2017 09:16:20 +0000") Message-ID: <87po8h1cx0.fsf@linaro.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/25.3 (gnu/linux) MIME-Version: 1.0 This patch splits the mask argument checking out of vectorizable_mask_load_store, so that a later patch can use it in both vectorizable_load and vectorizable_store. It also adds dump messages for false returns. This is mostly useful for the TYPE_VECTOR_SUBPARTS check, which can fail if pattern recognition didn't convert the mask properly. Richard 2017-11-17 Richard Sandiford gcc/ * tree-vect-stmts.c (vect_check_load_store_mask): New function, split out from... (vectorizable_mask_load_store): ...here. Index: gcc/tree-vect-stmts.c =================================================================== --- gcc/tree-vect-stmts.c 2017-11-17 09:06:46.238323981 +0000 +++ gcc/tree-vect-stmts.c 2017-11-17 09:06:49.784303349 +0000 @@ -2023,6 +2023,74 @@ get_load_store_type (gimple *stmt, tree return true; } +/* Return true if boolean argument MASK is suitable for vectorizing + conditional load or store STMT. When returning true, store the + type of the vectorized mask in *MASK_VECTYPE_OUT. */ + +static bool +vect_check_load_store_mask (gimple *stmt, tree mask, tree *mask_vectype_out) +{ + if (!VECT_SCALAR_BOOLEAN_TYPE_P (TREE_TYPE (mask))) + { + if (dump_enabled_p ()) + dump_printf_loc (MSG_MISSED_OPTIMIZATION, vect_location, + "mask argument is not a boolean.\n"); + return false; + } + + if (TREE_CODE (mask) != SSA_NAME) + { + if (dump_enabled_p ()) + dump_printf_loc (MSG_MISSED_OPTIMIZATION, vect_location, + "mask argument is not an SSA name.\n"); + return false; + } + + stmt_vec_info stmt_info = vinfo_for_stmt (stmt); + gimple *def_stmt; + enum vect_def_type dt; + tree mask_vectype; + if (!vect_is_simple_use (mask, stmt_info->vinfo, &def_stmt, &dt, + &mask_vectype)) + { + if (dump_enabled_p ()) + dump_printf_loc (MSG_MISSED_OPTIMIZATION, vect_location, + "mask use not simple.\n"); + return false; + } + + tree vectype = STMT_VINFO_VECTYPE (stmt_info); + if (!mask_vectype) + mask_vectype = get_mask_type_for_scalar_type (TREE_TYPE (vectype)); + + if (!mask_vectype || !VECTOR_BOOLEAN_TYPE_P (mask_vectype)) + { + if (dump_enabled_p ()) + dump_printf_loc (MSG_MISSED_OPTIMIZATION, vect_location, + "could not find an appropriate vector mask type.\n"); + return false; + } + + if (may_ne (TYPE_VECTOR_SUBPARTS (mask_vectype), + TYPE_VECTOR_SUBPARTS (vectype))) + { + if (dump_enabled_p ()) + { + dump_printf_loc (MSG_MISSED_OPTIMIZATION, vect_location, + "vector mask type "); + dump_generic_expr (MSG_MISSED_OPTIMIZATION, TDF_SLIM, mask_vectype); + dump_printf (MSG_MISSED_OPTIMIZATION, + " does not match vector data type "); + dump_generic_expr (MSG_MISSED_OPTIMIZATION, TDF_SLIM, vectype); + dump_printf (MSG_MISSED_OPTIMIZATION, ".\n"); + } + return false; + } + + *mask_vectype_out = mask_vectype; + return true; +} + /* Function vectorizable_mask_load_store. Check if STMT performs a conditional load or store that can be vectorized. @@ -2065,11 +2133,6 @@ vectorizable_mask_load_store (gimple *st ncopies = vect_get_num_copies (loop_vinfo, vectype); gcc_assert (ncopies >= 1); - mask = gimple_call_arg (stmt, 2); - - if (!VECT_SCALAR_BOOLEAN_TYPE_P (TREE_TYPE (mask))) - return false; - /* FORNOW. This restriction should be relaxed. */ if (nested_in_vect_loop && ncopies > 1) { @@ -2089,21 +2152,11 @@ vectorizable_mask_load_store (gimple *st if (!STMT_VINFO_DATA_REF (stmt_info)) return false; - elem_type = TREE_TYPE (vectype); - - if (TREE_CODE (mask) != SSA_NAME) - return false; - - if (!vect_is_simple_use (mask, loop_vinfo, &def_stmt, &dt, &mask_vectype)) + mask = gimple_call_arg (stmt, 2); + if (!vect_check_load_store_mask (stmt, mask, &mask_vectype)) return false; - if (!mask_vectype) - mask_vectype = get_mask_type_for_scalar_type (TREE_TYPE (vectype)); - - if (!mask_vectype || !VECTOR_BOOLEAN_TYPE_P (mask_vectype) - || may_ne (TYPE_VECTOR_SUBPARTS (mask_vectype), - TYPE_VECTOR_SUBPARTS (vectype))) - return false; + elem_type = TREE_TYPE (vectype); if (gimple_call_internal_fn (stmt) == IFN_MASK_STORE) { From patchwork Fri Nov 17 09:18:58 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Sandiford X-Patchwork-Id: 119105 Delivered-To: patch@linaro.org Received: by 10.140.22.164 with SMTP id 33csp281777qgn; Fri, 17 Nov 2017 01:20:21 -0800 (PST) X-Google-Smtp-Source: AGs4zMbcCs5v52F6PSx7lvuj8t6QxgpmRyLLoMYm3dFt42wOFy8A3oBn5ZhDUX//DQDgx5r/qXIw X-Received: by 10.98.87.13 with SMTP id l13mr1393219pfb.193.1510910421613; Fri, 17 Nov 2017 01:20:21 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1510910421; cv=none; d=google.com; s=arc-20160816; b=d0uE1rZxlaQpslGm/BIlOAN2Xqvn/XrzndueMnbUuHZP5EsNu1wYQKEoHAcouIjb7q HH3z4WGDNMIoI4gK2iqTcgQ7LJSag9NcwzF8URFG98U+fQSQkhVy5rvzlx+O1XZIM3im O1RSWdyTf8sckMktJbgepv0pkvniOceZp4RRniVTSLz5atkh3rY0mZ2qUBQ8aiOcnYMG hLa8cQTWOBN8kII0WKPM+R9OnAZNz6FKw5CJtIAVAIn1+SIe+SAD2yi4OPi9fx49incK cS96bgHPfYN+8qa7XdJ+VnEs4wtohSON3lEgzqCraJOnlW910MNwBoOxfbs6L5Lv/0UP jB+A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:user-agent:message-id:in-reply-to:date:references :subject:mail-followup-to:to:from:delivered-to:sender:list-help :list-post:list-archive:list-unsubscribe:list-id:precedence :mailing-list:dkim-signature:domainkey-signature :arc-authentication-results; bh=rTwoY6TAF9nCZoHtLallF5BOPEV2sCchpPQt2yw2GMU=; b=d7/bmUt/cvci8PnEBeNEI2SWgbze+hYD9b+YwaOO5WyvhmBxqeVgmwx4u2nTfLGHc8 SEXTqC20OL75zL4BS/E0iHwbWjSlO74HMaoMk+CWitDmOLW1ChwLcSOxe0eT32Xwy2g7 wBR9a8mfY0RGwRosYRaWqiBT6oXJyhlY397OPDcrGlfdQdgsgD5nFAI7ttj9UsFbdFBA ODNKUoGZb43FRi9uyzyjnmOHlDlz1bsm0qPkduUcGfofcXwORSQfUuRMh09KGn9+8isL 6OhjgVQVuk009NLpH+Yki5OvRCGndw4PQ8gq2UDgtAxL+haT7SEB+ANIRSU+NnQnQ/1V 9wEA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gcc.gnu.org header.s=default header.b=gZQjm/KA; spf=pass (google.com: domain of gcc-patches-return-467098-patch=linaro.org@gcc.gnu.org designates 209.132.180.131 as permitted sender) smtp.mailfrom=gcc-patches-return-467098-patch=linaro.org@gcc.gnu.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from sourceware.org (server1.sourceware.org. [209.132.180.131]) by mx.google.com with ESMTPS id a90si2428605plc.722.2017.11.17.01.20.21 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 17 Nov 2017 01:20:21 -0800 (PST) Received-SPF: pass (google.com: domain of gcc-patches-return-467098-patch=linaro.org@gcc.gnu.org designates 209.132.180.131 as permitted sender) client-ip=209.132.180.131; Authentication-Results: mx.google.com; dkim=pass header.i=@gcc.gnu.org header.s=default header.b=gZQjm/KA; spf=pass (google.com: domain of gcc-patches-return-467098-patch=linaro.org@gcc.gnu.org designates 209.132.180.131 as permitted sender) smtp.mailfrom=gcc-patches-return-467098-patch=linaro.org@gcc.gnu.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org DomainKey-Signature: a=rsa-sha1; c=nofws; d=gcc.gnu.org; h=list-id :list-unsubscribe:list-archive:list-post:list-help:sender:from :to:subject:references:date:in-reply-to:message-id:mime-version :content-type; q=dns; s=default; b=ITVhl1XvDQ9asXzOb69X7hTCwDzhu zKMGjqzjjrHctrVxPxoDANMTrEHzc2/aN/OsGoH/y+xpPttnMTn4D9eZ4K52GDu1 GHa6/kXfmMU9g+Rt1fjE08/GFGWx47iPY/udmsyq7iIgIqtqh4miwUg3JVDOiZtg EGx9IBQ1LbPArI= DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=gcc.gnu.org; h=list-id :list-unsubscribe:list-archive:list-post:list-help:sender:from :to:subject:references:date:in-reply-to:message-id:mime-version :content-type; s=default; bh=JyFrc62DiAOgwto5nCTrUycAMX8=; b=gZQ jm/KA/IGAX88noet9qFzycs6IZrj2VIQQs9pG00Paut1aEhoTnjWRTuARux8kJL8 o2eP1Hfb/Z4AXT0bYHmCAZ+AqWFYKwX3DjknvCldyZ/duE+KIIi2p/jYq2U8DpDr bKxcUE3trJ/iFSrBJng3NUAl0XP9EycnWiZM+CTw= Received: (qmail 118193 invoked by alias); 17 Nov 2017 09:19:02 -0000 Mailing-List: contact gcc-patches-help@gcc.gnu.org; run by ezmlm Precedence: bulk List-Id: List-Unsubscribe: List-Archive: List-Post: List-Help: Sender: gcc-patches-owner@gcc.gnu.org Delivered-To: mailing list gcc-patches@gcc.gnu.org Received: (qmail 118137 invoked by uid 89); 17 Nov 2017 09:19:01 -0000 Authentication-Results: sourceware.org; auth=none X-Virus-Found: No X-Spam-SWARE-Status: No, score=-11.5 required=5.0 tests=AWL, BAYES_00, GIT_PATCH_2, GIT_PATCH_3, KB_WAM_FROM_NAME_SINGLEWORD, RCVD_IN_DNSWL_NONE, SPF_PASS autolearn=ham version=3.3.2 spammy= X-HELO: mail-wr0-f178.google.com Received: from mail-wr0-f178.google.com (HELO mail-wr0-f178.google.com) (209.85.128.178) by sourceware.org (qpsmtpd/0.93/v0.84-503-g423c35a) with ESMTP; Fri, 17 Nov 2017 09:19:00 +0000 Received: by mail-wr0-f178.google.com with SMTP id u97so1528688wrc.1 for ; Fri, 17 Nov 2017 01:18:59 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:mail-followup-to:subject:references:date :in-reply-to:message-id:user-agent:mime-version; bh=rTwoY6TAF9nCZoHtLallF5BOPEV2sCchpPQt2yw2GMU=; b=O7Lr1Zujk21kaC9kYsxGBNhj4GLXH7iFJUDhNWX8Krwyy/ovDhxnswUpUs/grPbHPl BBKm/8xA1Ry1kRSl4YGomXsZama5odPznZcT7H9/t390oYL5nWXDVMqJhV1eCcMkcZYo aGVs+ph+adI8cXuW4s6ydaC4klMHN8FZoLntQTOJZEsCqC2s9DncEzj5ochMI3AxOgiZ 160tKGH2yyWBfC9YdaV1rbQvhDzzt0rfabeiX29nENVn1hwNr7uoMWy+3RYr8ns/S6GH GG2TJiK0I+KgZngnFv1vacV/PMqS9wVcJbBJCaw9HpPlO5A2zpp0pAC6GQWBtwWzQm1C D3LA== X-Gm-Message-State: AJaThX6ChVOb/6wWqEg6I8WkHVUTpObmpqM8RQc/cHRelgyjDtXT67YC M6kjAbLc9iVKNBjL0V9Sy4XhZylXPkw= X-Received: by 10.223.165.4 with SMTP id i4mr3988153wrb.158.1510910337737; Fri, 17 Nov 2017 01:18:57 -0800 (PST) Received: from localhost ([2.25.234.120]) by smtp.gmail.com with ESMTPSA id n14sm3760094wrg.38.2017.11.17.01.18.56 for (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Fri, 17 Nov 2017 01:18:57 -0800 (PST) From: Richard Sandiford To: gcc-patches@gcc.gnu.org Mail-Followup-To: gcc-patches@gcc.gnu.org, richard.sandiford@linaro.org Subject: [4/7] Split rhs checking out of vectorizable_{,mask_load_}store References: <87375d2rkr.fsf@linaro.org> Date: Fri, 17 Nov 2017 09:18:58 +0000 In-Reply-To: <87375d2rkr.fsf@linaro.org> (Richard Sandiford's message of "Fri, 17 Nov 2017 09:16:20 +0000") Message-ID: <87lgj51cvx.fsf@linaro.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/25.3 (gnu/linux) MIME-Version: 1.0 This patch splits out the rhs checking code that's common to both vectorizable_mask_load_store and vectorizable_store. Richard 2017-11-17 Richard Sandiford gcc/ * tree-vect-stmts.c (vect_check_store_rhs): New function, split out from... (vectorizable_mask_load_store): ...here. (vectorizable_store): ...and here. Index: gcc/tree-vect-stmts.c =================================================================== --- gcc/tree-vect-stmts.c 2017-11-17 09:06:49.784303349 +0000 +++ gcc/tree-vect-stmts.c 2017-11-17 09:06:53.395522598 +0000 @@ -2091,6 +2091,55 @@ vect_check_load_store_mask (gimple *stmt return true; } +/* Return true if stored value RHS is suitable for vectorizing store + statement STMT. When returning true, store the type of the + vectorized store value in *RHS_VECTYPE_OUT and the type of the + store in *VLS_TYPE_OUT. */ + +static bool +vect_check_store_rhs (gimple *stmt, tree rhs, tree *rhs_vectype_out, + vec_load_store_type *vls_type_out) +{ + /* In the case this is a store from a constant make sure + native_encode_expr can handle it. */ + if (CONSTANT_CLASS_P (rhs) && native_encode_expr (rhs, NULL, 64) == 0) + { + if (dump_enabled_p ()) + dump_printf_loc (MSG_MISSED_OPTIMIZATION, vect_location, + "cannot encode constant as a byte sequence.\n"); + return false; + } + + stmt_vec_info stmt_info = vinfo_for_stmt (stmt); + gimple *def_stmt; + enum vect_def_type dt; + tree rhs_vectype; + if (!vect_is_simple_use (rhs, stmt_info->vinfo, &def_stmt, &dt, + &rhs_vectype)) + { + if (dump_enabled_p ()) + dump_printf_loc (MSG_MISSED_OPTIMIZATION, vect_location, + "use not simple.\n"); + return false; + } + + tree vectype = STMT_VINFO_VECTYPE (stmt_info); + if (rhs_vectype && !useless_type_conversion_p (vectype, rhs_vectype)) + { + if (dump_enabled_p ()) + dump_printf_loc (MSG_MISSED_OPTIMIZATION, vect_location, + "incompatible vector types.\n"); + return false; + } + + *rhs_vectype_out = rhs_vectype; + if (dt == vect_constant_def || dt == vect_external_def) + *vls_type_out = VLS_STORE_INVARIANT; + else + *vls_type_out = VLS_STORE; + return true; +} + /* Function vectorizable_mask_load_store. Check if STMT performs a conditional load or store that can be vectorized. @@ -2161,12 +2210,8 @@ vectorizable_mask_load_store (gimple *st if (gimple_call_internal_fn (stmt) == IFN_MASK_STORE) { tree rhs = gimple_call_arg (stmt, 3); - if (!vect_is_simple_use (rhs, loop_vinfo, &def_stmt, &dt, &rhs_vectype)) + if (!vect_check_store_rhs (stmt, rhs, &rhs_vectype, &vls_type)) return false; - if (dt == vect_constant_def || dt == vect_external_def) - vls_type = VLS_STORE_INVARIANT; - else - vls_type = VLS_STORE; } else vls_type = VLS_LOAD; @@ -2200,9 +2245,7 @@ vectorizable_mask_load_store (gimple *st else if (!VECTOR_MODE_P (TYPE_MODE (vectype)) || !can_vec_mask_load_store_p (TYPE_MODE (vectype), TYPE_MODE (mask_vectype), - vls_type == VLS_LOAD) - || (rhs_vectype - && !useless_type_conversion_p (vectype, rhs_vectype))) + vls_type == VLS_LOAD)) return false; if (!vec_stmt) /* transformation not required. */ @@ -5798,26 +5841,7 @@ vectorizable_store (gimple *stmt, gimple } op = gimple_assign_rhs1 (stmt); - - /* In the case this is a store from a constant make sure - native_encode_expr can handle it. */ - if (CONSTANT_CLASS_P (op) && native_encode_expr (op, NULL, 64) == 0) - return false; - - if (!vect_is_simple_use (op, vinfo, &def_stmt, &dt, &rhs_vectype)) - { - if (dump_enabled_p ()) - dump_printf_loc (MSG_MISSED_OPTIMIZATION, vect_location, - "use not simple.\n"); - return false; - } - - if (dt == vect_constant_def || dt == vect_external_def) - vls_type = VLS_STORE_INVARIANT; - else - vls_type = VLS_STORE; - - if (rhs_vectype && !useless_type_conversion_p (vectype, rhs_vectype)) + if (!vect_check_store_rhs (stmt, op, &rhs_vectype, &vls_type)) return false; elem_type = TREE_TYPE (vectype); From patchwork Fri Nov 17 09:19:29 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Sandiford X-Patchwork-Id: 119106 Delivered-To: patch@linaro.org Received: by 10.140.22.164 with SMTP id 33csp282159qgn; Fri, 17 Nov 2017 01:20:49 -0800 (PST) X-Google-Smtp-Source: AGs4zMYzGiS3HsdwQojvevlfCCrGxO+0kMwAZMhH0aQhQZwKlu5CH3jBKe0jBkh4yftfv4FdYX6E X-Received: by 10.84.130.6 with SMTP id 6mr4745609plc.402.1510910449111; Fri, 17 Nov 2017 01:20:49 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1510910449; cv=none; d=google.com; s=arc-20160816; b=xSzAyk1sqb4P9YzBtpI0Nql+6fL26oO+0tq6q33jsUgkRkHnKrDCuwo5pSB4gub57a ID3gr/JzqzzfRGvsgpSmAPiPOIlvMiZwOrjyvyOEGm7LIB5Fc7q46yKm4pYGbQC7LoTu OQszY07Ngcimk0L0H8qLPbLWmCqQchbKFMR3ATVdkBNySxWJYCKbvR062NfsgwSDvDQm 10PevI9KZJemSPmLZoIQ3eSGADOkk/eSKmIodDsCawK559acOOwb4IZq+Q/eDd3Ayrl0 iDcVQkGJlxG0CET+ywvH0f4O+LJtnrYqAjWdXOyiV/EANw1bWcRUANuFOP23McX8SRX6 J2fw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:user-agent:message-id:in-reply-to:date:references :subject:mail-followup-to:to:from:delivered-to:sender:list-help :list-post:list-archive:list-unsubscribe:list-id:precedence :mailing-list:dkim-signature:domainkey-signature :arc-authentication-results; bh=jqExs+fIDINSdmFXJdV1MMhRiaC8MxGnaqrzcUBlQgk=; b=sGSHJ2lJQZbt7+Zj18uNAldLDppOQ5soyFEEkd7m79dfTvhNIV89/ev8uilahvZ63D kB9v5dN+hy7G3mqRu9uY5T6/xKmV2zjEDgNeBTxg1lJ5NMgbuMbxuxQuQ6DSSq8ZSMAa jB+Or3aLvfY6rSdKfjT9s7MR3vLM5+yC1wtg7FoOzZAx2L1n7AcIDB0K3vmZ+Hw6Z/1M j2iViHzBhxPgJiXMb4iI6l0F/TgvcmtcjmOYm0FV7Xo8IDbcF8T4mbOdQUCkngG/P1Fq GVdbEtamEc58bscAV2/61KdEERH0eDIidMxCVhKyxAzw0A/g5uouo+xyDBzmxdNrZ++I XLoA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gcc.gnu.org header.s=default header.b=a8xxiObL; spf=pass (google.com: domain of gcc-patches-return-467099-patch=linaro.org@gcc.gnu.org designates 209.132.180.131 as permitted sender) smtp.mailfrom=gcc-patches-return-467099-patch=linaro.org@gcc.gnu.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from sourceware.org (server1.sourceware.org. [209.132.180.131]) by mx.google.com with ESMTPS id h185si2386612pgc.724.2017.11.17.01.20.48 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 17 Nov 2017 01:20:49 -0800 (PST) Received-SPF: pass (google.com: domain of gcc-patches-return-467099-patch=linaro.org@gcc.gnu.org designates 209.132.180.131 as permitted sender) client-ip=209.132.180.131; Authentication-Results: mx.google.com; dkim=pass header.i=@gcc.gnu.org header.s=default header.b=a8xxiObL; spf=pass (google.com: domain of gcc-patches-return-467099-patch=linaro.org@gcc.gnu.org designates 209.132.180.131 as permitted sender) smtp.mailfrom=gcc-patches-return-467099-patch=linaro.org@gcc.gnu.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org DomainKey-Signature: a=rsa-sha1; c=nofws; d=gcc.gnu.org; h=list-id :list-unsubscribe:list-archive:list-post:list-help:sender:from :to:subject:references:date:in-reply-to:message-id:mime-version :content-type; q=dns; s=default; b=UDRS4VoC88S2eFJ+s7WYXtyPXSUS6 mhNVCbq9mibolGTV3Oi5sSvL3WEIrrgFQZ0gnPgYURDcDDerFEfmFw8Py3uoxU1e wFRed/v1LuQpRnT7EyQMo305FHMriljzbGOoBnydqdqTOZRroiy/JrlrI4pKfh0n f/JnMDeDFL2qus= DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=gcc.gnu.org; h=list-id :list-unsubscribe:list-archive:list-post:list-help:sender:from :to:subject:references:date:in-reply-to:message-id:mime-version :content-type; s=default; bh=8NdxIpWvI/iJVd145zTCASjDb34=; b=a8x xiObLlMgLvP6IK34o2i2LRxd1s2XhR24dpYa3HDG7Ngwf7iAKBuT5S2woqnGtPcP Zhfc6XeLTMtLF8WJ71Pf3oWxMn7pOPZ+vRbMOyu6NdBMitql1uzPhFycZd2PrQ3O HAjUZVhDjcxn0w84P5ZG5RsIWn0gGhP/SGGcornA= Received: (qmail 119579 invoked by alias); 17 Nov 2017 09:19:33 -0000 Mailing-List: contact gcc-patches-help@gcc.gnu.org; run by ezmlm Precedence: bulk List-Id: List-Unsubscribe: List-Archive: List-Post: List-Help: Sender: gcc-patches-owner@gcc.gnu.org Delivered-To: mailing list gcc-patches@gcc.gnu.org Received: (qmail 119531 invoked by uid 89); 17 Nov 2017 09:19:33 -0000 Authentication-Results: sourceware.org; auth=none X-Virus-Found: No X-Spam-SWARE-Status: No, score=-11.5 required=5.0 tests=AWL, BAYES_00, GIT_PATCH_2, GIT_PATCH_3, KB_WAM_FROM_NAME_SINGLEWORD, RCVD_IN_DNSWL_NONE, SPF_PASS autolearn=ham version=3.3.2 spammy= X-HELO: mail-wr0-f172.google.com Received: from mail-wr0-f172.google.com (HELO mail-wr0-f172.google.com) (209.85.128.172) by sourceware.org (qpsmtpd/0.93/v0.84-503-g423c35a) with ESMTP; Fri, 17 Nov 2017 09:19:31 +0000 Received: by mail-wr0-f172.google.com with SMTP id o14so1504312wrf.9 for ; Fri, 17 Nov 2017 01:19:30 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:mail-followup-to:subject:references:date :in-reply-to:message-id:user-agent:mime-version; bh=jqExs+fIDINSdmFXJdV1MMhRiaC8MxGnaqrzcUBlQgk=; b=XTqakcYkhKJmQ/kkwQoSslv8hMTrO3e/d9uVzqen4SrgtvywRVfbwxOKgtlXentJIx FjSpuVymXEQ1uwa/kydjWq2+h/CBZh/bFcNWEICLeQ8ljSSA3C2R1YT5YE/sR9C8YbV2 PaBGdsUkJXb3sNeSdGhjWanr88S9LMsJHj7QPDRxIYaB79TSug2dIJTGEVLnpJ8vBwyF WEAq47X7ovQPLahbCrDSVsW4qg7hlTutsxjEfnLBGpMdejWXdZNvypwiBjIv2P470/NC bGIej2M8xZzK/OW69B7ixguU8oEegvVNbqeG3skU2xFC/Pa0vE3dc+dsFgUOYL/xtsCf jRJQ== X-Gm-Message-State: AJaThX6kpq6TowDI7lElF/YTtUQ+n1gflp7Un4e25/QylG26PXsgVUyf Ftg9m124bdBlPAC8LKCaUrYnWkqYB3s= X-Received: by 10.223.163.143 with SMTP id l15mr4113422wrb.149.1510910368976; Fri, 17 Nov 2017 01:19:28 -0800 (PST) Received: from localhost ([2.25.234.120]) by smtp.gmail.com with ESMTPSA id m8sm2310130wrg.55.2017.11.17.01.19.28 for (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Fri, 17 Nov 2017 01:19:28 -0800 (PST) From: Richard Sandiford To: gcc-patches@gcc.gnu.org Mail-Followup-To: gcc-patches@gcc.gnu.org, richard.sandiford@linaro.org Subject: [5/7] Split out gather load mask building References: <87375d2rkr.fsf@linaro.org> Date: Fri, 17 Nov 2017 09:19:29 +0000 In-Reply-To: <87375d2rkr.fsf@linaro.org> (Richard Sandiford's message of "Fri, 17 Nov 2017 09:16:20 +0000") Message-ID: <87h8tt1cv2.fsf@linaro.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/25.3 (gnu/linux) MIME-Version: 1.0 This patch splits out the code to build an all-bits-one or all-bits-zero input to a gather load. The catch is that both masks can have floating-point type, in which case they are implicitly treated in the same way as an integer bitmask. Richard 2017-11-17 Richard Sandiford gcc/ * tree-vect-stmts.c (vect_build_all_ones_mask) (vect_build_zero_merge_argument): New functions, split out from... (vectorizable_load): ...here. Index: gcc/tree-vect-stmts.c =================================================================== --- gcc/tree-vect-stmts.c 2017-11-17 09:06:53.395522598 +0000 +++ gcc/tree-vect-stmts.c 2017-11-17 09:06:56.444527585 +0000 @@ -2140,6 +2140,59 @@ vect_check_store_rhs (gimple *stmt, tree return true; } +/* Build an all-ones vector mask of type MASKTYPE while vectorizing STMT. + Note that we support masks with floating-point type, in which case the + floats are interpreted as a bitmask. */ + +static tree +vect_build_all_ones_mask (gimple *stmt, tree masktype) +{ + if (TREE_CODE (masktype) == INTEGER_TYPE) + return build_int_cst (masktype, -1); + else if (TREE_CODE (TREE_TYPE (masktype)) == INTEGER_TYPE) + { + tree mask = build_int_cst (TREE_TYPE (masktype), -1); + mask = build_vector_from_val (masktype, mask); + return vect_init_vector (stmt, mask, masktype, NULL); + } + else if (SCALAR_FLOAT_TYPE_P (TREE_TYPE (masktype))) + { + REAL_VALUE_TYPE r; + long tmp[6]; + for (int j = 0; j < 6; ++j) + tmp[j] = -1; + real_from_target (&r, tmp, TYPE_MODE (TREE_TYPE (masktype))); + tree mask = build_real (TREE_TYPE (masktype), r); + mask = build_vector_from_val (masktype, mask); + return vect_init_vector (stmt, mask, masktype, NULL); + } + gcc_unreachable (); +} + +/* Build an all-zero merge value of type VECTYPE while vectorizing + STMT as a gather load. */ + +static tree +vect_build_zero_merge_argument (gimple *stmt, tree vectype) +{ + tree merge; + if (TREE_CODE (TREE_TYPE (vectype)) == INTEGER_TYPE) + merge = build_int_cst (TREE_TYPE (vectype), 0); + else if (SCALAR_FLOAT_TYPE_P (TREE_TYPE (vectype))) + { + REAL_VALUE_TYPE r; + long tmp[6]; + for (int j = 0; j < 6; ++j) + tmp[j] = 0; + real_from_target (&r, tmp, TYPE_MODE (TREE_TYPE (vectype))); + merge = build_real (TREE_TYPE (vectype), r); + } + else + gcc_unreachable (); + merge = build_vector_from_val (vectype, merge); + return vect_init_vector (stmt, merge, vectype, NULL); +} + /* Function vectorizable_mask_load_store. Check if STMT performs a conditional load or store that can be vectorized. @@ -6990,45 +7043,9 @@ vectorizable_load (gimple *stmt, gimple_ /* Currently we support only unconditional gather loads, so mask should be all ones. */ - if (TREE_CODE (masktype) == INTEGER_TYPE) - mask = build_int_cst (masktype, -1); - else if (TREE_CODE (TREE_TYPE (masktype)) == INTEGER_TYPE) - { - mask = build_int_cst (TREE_TYPE (masktype), -1); - mask = build_vector_from_val (masktype, mask); - mask = vect_init_vector (stmt, mask, masktype, NULL); - } - else if (SCALAR_FLOAT_TYPE_P (TREE_TYPE (masktype))) - { - REAL_VALUE_TYPE r; - long tmp[6]; - for (j = 0; j < 6; ++j) - tmp[j] = -1; - real_from_target (&r, tmp, TYPE_MODE (TREE_TYPE (masktype))); - mask = build_real (TREE_TYPE (masktype), r); - mask = build_vector_from_val (masktype, mask); - mask = vect_init_vector (stmt, mask, masktype, NULL); - } - else - gcc_unreachable (); - + mask = vect_build_all_ones_mask (stmt, masktype); scale = build_int_cst (scaletype, gs_info.scale); - - if (TREE_CODE (TREE_TYPE (rettype)) == INTEGER_TYPE) - merge = build_int_cst (TREE_TYPE (rettype), 0); - else if (SCALAR_FLOAT_TYPE_P (TREE_TYPE (rettype))) - { - REAL_VALUE_TYPE r; - long tmp[6]; - for (j = 0; j < 6; ++j) - tmp[j] = 0; - real_from_target (&r, tmp, TYPE_MODE (TREE_TYPE (rettype))); - merge = build_real (TREE_TYPE (rettype), r); - } - else - gcc_unreachable (); - merge = build_vector_from_val (rettype, merge); - merge = vect_init_vector (stmt, merge, rettype, NULL); + merge = vect_build_zero_merge_argument (stmt, rettype); prev_stmt_info = NULL; for (j = 0; j < ncopies; ++j) From patchwork Fri Nov 17 09:20:15 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Sandiford X-Patchwork-Id: 119107 Delivered-To: patch@linaro.org Received: by 10.140.22.164 with SMTP id 33csp282842qgn; Fri, 17 Nov 2017 01:21:36 -0800 (PST) X-Google-Smtp-Source: AGs4zMZQXITwLrmcUMmqrMbvyi+lb5XvvZ/Gtk5qczhPAa70XiNqIwEPING3xKXrV0n9hA/I6HBm X-Received: by 10.159.216.141 with SMTP id s13mr4622771plp.377.1510910496597; Fri, 17 Nov 2017 01:21:36 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1510910496; cv=none; d=google.com; s=arc-20160816; b=FaBj41N67rMXek1zmAsuTMICT7xDcspO349ypblBi5ZrgTuxqsrxWj2Cmpzl4ELzPa 4KmYRoRnY27skU8fFY/2h8D9jCOdqNqPFpzFrRU06lfmaJYKGZuVio6/EjsqA7uEt25Z 8C2k9600QDdhZjsoSQ0/tT+vXa0OcNLtt4Vqg4GJGxwN3g9bGARoBpDr86fQ1HR+TfTk DkKsNvhaVu9U7S52cd+3Su7vZr1UExRcLNzMcNau9K5P+iaRWmhIVJZPtBxXoRDlUskG KHUEj+eHpSFSr078xG+PCcNYY3hzw48PNwTa0bebcfdUph3br232oOjrCySxf9uU42OH JRYA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:user-agent:message-id:in-reply-to:date:references :subject:mail-followup-to:to:from:delivered-to:sender:list-help :list-post:list-archive:list-unsubscribe:list-id:precedence :mailing-list:dkim-signature:domainkey-signature :arc-authentication-results; bh=zm6j6rgksZOrgkaLnP7anExteg+1rW0C5P0fr5AfvvU=; b=d8Z5fV6QA98xPZ1rmp+TmYmVJg+sPl6gwPyrUXpXMINHUU2vD+bNjg8fk6hKVYCDgv 5zWIQy4vPk71MywjK92MxqywS1Q7yJbiAnDV7wmr6iAbCVosrZ+Zt2kj8GQJ/n0OfimT GTvDBdrpX9gscBnR1/W4y0C0UEEt99t5k9hsPeqasyg0W/C1eyMK/M+HyK75rUL33yRG SR3P3BkeF/TMqaxCrdaqbMuBfpBP4RZzLsndBe4nE0SYQKkDP1OEZ/KVwilHirYpPpgq FfjiCXEZRyc+WFdGdWWNRDJ1zsog8mrmwnZ/7ucG4961/DagNtWQEv3Rt7rdPcEQT7jI 249Q== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gcc.gnu.org header.s=default header.b=rmP6UuYN; spf=pass (google.com: domain of gcc-patches-return-467100-patch=linaro.org@gcc.gnu.org designates 209.132.180.131 as permitted sender) smtp.mailfrom=gcc-patches-return-467100-patch=linaro.org@gcc.gnu.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from sourceware.org (server1.sourceware.org. [209.132.180.131]) by mx.google.com with ESMTPS id i8si2384738pgr.690.2017.11.17.01.21.36 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 17 Nov 2017 01:21:36 -0800 (PST) Received-SPF: pass (google.com: domain of gcc-patches-return-467100-patch=linaro.org@gcc.gnu.org designates 209.132.180.131 as permitted sender) client-ip=209.132.180.131; Authentication-Results: mx.google.com; dkim=pass header.i=@gcc.gnu.org header.s=default header.b=rmP6UuYN; spf=pass (google.com: domain of gcc-patches-return-467100-patch=linaro.org@gcc.gnu.org designates 209.132.180.131 as permitted sender) smtp.mailfrom=gcc-patches-return-467100-patch=linaro.org@gcc.gnu.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org DomainKey-Signature: a=rsa-sha1; c=nofws; d=gcc.gnu.org; h=list-id :list-unsubscribe:list-archive:list-post:list-help:sender:from :to:subject:references:date:in-reply-to:message-id:mime-version :content-type; q=dns; s=default; b=L7FmDUNqQm34EOvd83yLyhyimNok2 OXKtGwPpKlLawlftKRQDzRH6rVNLo00gy+wz9tQww/8/+c9kHIRl0BpgmpIuXST2 d0AITftQF2mCPTDpUJHmKtUY+rstJlTTTgvqSiYEIhLYkJrnOq/eqfZiqh3LflcG 11Hi+ZT2HYsffA= DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=gcc.gnu.org; h=list-id :list-unsubscribe:list-archive:list-post:list-help:sender:from :to:subject:references:date:in-reply-to:message-id:mime-version :content-type; s=default; bh=eOYHOVSyG4AvYyTxy/NWNA0TpUc=; b=rmP 6UuYNvezetBZy+85uRdPNl8UPsTHm6UiHXQ78fbvjDFgPN3eLXDLW509BCT1iVW9 7xL/TQA1EldbAig3Wxts+yHUi1CKRe8dcoLFY5Hl7GEnmE1O5IGwGuwgtGnqeJNU E8MZfRV0Om71NPvhpGWBSharrs6l67MdLYHq7S88= Received: (qmail 123432 invoked by alias); 17 Nov 2017 09:20:24 -0000 Mailing-List: contact gcc-patches-help@gcc.gnu.org; run by ezmlm Precedence: bulk List-Id: List-Unsubscribe: List-Archive: List-Post: List-Help: Sender: gcc-patches-owner@gcc.gnu.org Delivered-To: mailing list gcc-patches@gcc.gnu.org Received: (qmail 122640 invoked by uid 89); 17 Nov 2017 09:20:21 -0000 Authentication-Results: sourceware.org; auth=none X-Virus-Found: No X-Spam-SWARE-Status: No, score=-11.5 required=5.0 tests=AWL, BAYES_00, GIT_PATCH_2, GIT_PATCH_3, KB_WAM_FROM_NAME_SINGLEWORD, RCVD_IN_DNSWL_NONE, SPF_PASS autolearn=ham version=3.3.2 spammy= X-HELO: mail-wr0-f177.google.com Received: from mail-wr0-f177.google.com (HELO mail-wr0-f177.google.com) (209.85.128.177) by sourceware.org (qpsmtpd/0.93/v0.84-503-g423c35a) with ESMTP; Fri, 17 Nov 2017 09:20:17 +0000 Received: by mail-wr0-f177.google.com with SMTP id y42so1525101wrd.3 for ; Fri, 17 Nov 2017 01:20:16 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:mail-followup-to:subject:references:date :in-reply-to:message-id:user-agent:mime-version; bh=zm6j6rgksZOrgkaLnP7anExteg+1rW0C5P0fr5AfvvU=; b=jEpM7Yp5SkTIik5ENmZ6au46pfMIGhFXEAvFktW8kaKiLJnv/ZBKt+UOUr9E18qdVc Q7m8WrgIVXZkOfklcj4vnaumb4bZ2p6Aa7MWCh8LbWlHpLIPyRoXuhrmCnCcQEdqCqIa AsMP00zSc7wokxC9/OkJXrnmK6zdQcCUt8skIwfFkCVRcLUbTt6Qvr6SjcIkyP1+xGOW 5AiLe6fZZWmzY0M5kPV95w1WMXAkmCtMBC69fS3lBjKvJmiUklSRiyDcPE5+XWsqh+kK EBnuSphyxiLErnoOAhZH7rFgMBlKF3zsQ8qxw9o/qbFPBWhXn4A/Jzd1BrxnEhGV+ck3 huVQ== X-Gm-Message-State: AJaThX7IDKexE+ZDZgu+/ngADCqTE1v2S46Y8Arv3TfiCBAiDWpGLMOj 1jU8TxlpayCFJtWdOofL5eWId+NzWXw= X-Received: by 10.223.186.71 with SMTP id t7mr3569330wrg.75.1510910414887; Fri, 17 Nov 2017 01:20:14 -0800 (PST) Received: from localhost ([2.25.234.120]) by smtp.gmail.com with ESMTPSA id m187sm2396373wmg.18.2017.11.17.01.20.13 for (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Fri, 17 Nov 2017 01:20:14 -0800 (PST) From: Richard Sandiford To: gcc-patches@gcc.gnu.org Mail-Followup-To: gcc-patches@gcc.gnu.org, richard.sandiford@linaro.org Subject: [6/7] Split gather load handling out of vectorizable_{mask_load_store, load} References: <87375d2rkr.fsf@linaro.org> Date: Fri, 17 Nov 2017 09:20:15 +0000 In-Reply-To: <87375d2rkr.fsf@linaro.org> (Richard Sandiford's message of "Fri, 17 Nov 2017 09:16:20 +0000") Message-ID: <87d14h1cts.fsf@linaro.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/25.3 (gnu/linux) MIME-Version: 1.0 vectorizable_mask_load_store and vectorizable_load used the same code to build a gather load call, except that the former also vectorised a mask argument and used it for both the src and mask inputs. The latter instead used a src input of zero and a mask input of all-ones. This patch splits the code out into a subroutine. Richard 2017-11-17 Richard Sandiford gcc/ * tree-vect-stmts.c (vect_build_gather_load_calls): New function, split out from... (vectorizable_mask_load_store): ...here. (vectorizable_load): ...and here. Index: gcc/tree-vect-stmts.c =================================================================== --- gcc/tree-vect-stmts.c 2017-11-17 09:06:56.444527585 +0000 +++ gcc/tree-vect-stmts.c 2017-11-17 09:07:00.001060463 +0000 @@ -2193,6 +2193,206 @@ vect_build_zero_merge_argument (gimple * return vect_init_vector (stmt, merge, vectype, NULL); } +/* Build a gather load call while vectorizing STMT. Insert new instructions + before GSI and add them to VEC_STMT. GS_INFO describes the gather load + operation. If the load is conditional, MASK is the unvectorized + condition, otherwise MASK is null. */ + +static void +vect_build_gather_load_calls (gimple *stmt, gimple_stmt_iterator *gsi, + gimple **vec_stmt, gather_scatter_info *gs_info, + tree mask) +{ + stmt_vec_info stmt_info = vinfo_for_stmt (stmt); + loop_vec_info loop_vinfo = STMT_VINFO_LOOP_VINFO (stmt_info); + struct loop *loop = LOOP_VINFO_LOOP (loop_vinfo); + tree vectype = STMT_VINFO_VECTYPE (stmt_info); + poly_uint64 nunits = TYPE_VECTOR_SUBPARTS (vectype); + int ncopies = vect_get_num_copies (loop_vinfo, vectype); + edge pe = loop_preheader_edge (loop); + enum { NARROW, NONE, WIDEN } modifier; + poly_uint64 gather_off_nunits + = TYPE_VECTOR_SUBPARTS (gs_info->offset_vectype); + + tree arglist = TYPE_ARG_TYPES (TREE_TYPE (gs_info->decl)); + tree rettype = TREE_TYPE (TREE_TYPE (gs_info->decl)); + tree srctype = TREE_VALUE (arglist); arglist = TREE_CHAIN (arglist); + tree ptrtype = TREE_VALUE (arglist); arglist = TREE_CHAIN (arglist); + tree idxtype = TREE_VALUE (arglist); arglist = TREE_CHAIN (arglist); + tree masktype = TREE_VALUE (arglist); arglist = TREE_CHAIN (arglist); + tree scaletype = TREE_VALUE (arglist); + gcc_checking_assert (types_compatible_p (srctype, rettype) + && (!mask || types_compatible_p (srctype, masktype))); + + tree perm_mask = NULL_TREE; + tree mask_perm_mask = NULL_TREE; + if (must_eq (nunits, gather_off_nunits)) + modifier = NONE; + else if (must_eq (nunits * 2, gather_off_nunits)) + { + modifier = WIDEN; + + /* Currently widening gathers and scatters are only supported for + fixed-length vectors. */ + int count = gather_off_nunits.to_constant (); + auto_vec_perm_indices sel (count); + for (int i = 0; i < count; ++i) + sel.quick_push (i | (count / 2)); + + perm_mask = vect_gen_perm_mask_checked (gs_info->offset_vectype, sel); + } + else if (must_eq (nunits, gather_off_nunits * 2)) + { + modifier = NARROW; + + /* Currently narrowing gathers and scatters are only supported for + fixed-length vectors. */ + int count = nunits.to_constant (); + auto_vec_perm_indices sel (count); + sel.quick_grow (count); + for (int i = 0; i < count; ++i) + sel[i] = i < count / 2 ? i : i + count / 2; + perm_mask = vect_gen_perm_mask_checked (vectype, sel); + + ncopies *= 2; + + if (mask) + { + for (int i = 0; i < count; ++i) + sel[i] = i | (count / 2); + mask_perm_mask = vect_gen_perm_mask_checked (masktype, sel); + } + } + else + gcc_unreachable (); + + tree vec_dest = vect_create_destination_var (gimple_get_lhs (stmt), + vectype); + + tree ptr = fold_convert (ptrtype, gs_info->base); + if (!is_gimple_min_invariant (ptr)) + { + gimple_seq seq; + ptr = force_gimple_operand (ptr, &seq, true, NULL_TREE); + basic_block new_bb = gsi_insert_seq_on_edge_immediate (pe, seq); + gcc_assert (!new_bb); + } + + tree scale = build_int_cst (scaletype, gs_info->scale); + + tree vec_oprnd0 = NULL_TREE; + tree vec_mask = NULL_TREE; + tree src_op = NULL_TREE; + tree mask_op = NULL_TREE; + tree prev_res = NULL_TREE; + stmt_vec_info prev_stmt_info = NULL; + + if (!mask) + { + src_op = vect_build_zero_merge_argument (stmt, rettype); + mask_op = vect_build_all_ones_mask (stmt, masktype); + } + + for (int j = 0; j < ncopies; ++j) + { + tree op, var; + gimple *new_stmt; + if (modifier == WIDEN && (j & 1)) + op = permute_vec_elements (vec_oprnd0, vec_oprnd0, + perm_mask, stmt, gsi); + else if (j == 0) + op = vec_oprnd0 + = vect_get_vec_def_for_operand (gs_info->offset, stmt); + else + op = vec_oprnd0 + = vect_get_vec_def_for_stmt_copy (gs_info->offset_dt, vec_oprnd0); + + if (!useless_type_conversion_p (idxtype, TREE_TYPE (op))) + { + gcc_assert (must_eq (TYPE_VECTOR_SUBPARTS (TREE_TYPE (op)), + TYPE_VECTOR_SUBPARTS (idxtype))); + var = vect_get_new_ssa_name (idxtype, vect_simple_var); + op = build1 (VIEW_CONVERT_EXPR, idxtype, op); + new_stmt = gimple_build_assign (var, VIEW_CONVERT_EXPR, op); + vect_finish_stmt_generation (stmt, new_stmt, gsi); + op = var; + } + + if (mask) + { + if (mask_perm_mask && (j & 1)) + mask_op = permute_vec_elements (mask_op, mask_op, + mask_perm_mask, stmt, gsi); + else + { + if (j == 0) + vec_mask = vect_get_vec_def_for_operand (mask, stmt); + else + { + gimple *def_stmt; + enum vect_def_type dt; + vect_is_simple_use (vec_mask, loop_vinfo, &def_stmt, &dt); + vec_mask = vect_get_vec_def_for_stmt_copy (dt, vec_mask); + } + + mask_op = vec_mask; + if (!useless_type_conversion_p (masktype, TREE_TYPE (vec_mask))) + { + gcc_assert + (must_eq (TYPE_VECTOR_SUBPARTS (TREE_TYPE (mask_op)), + TYPE_VECTOR_SUBPARTS (masktype))); + var = vect_get_new_ssa_name (masktype, vect_simple_var); + mask_op = build1 (VIEW_CONVERT_EXPR, masktype, mask_op); + new_stmt = gimple_build_assign (var, VIEW_CONVERT_EXPR, + mask_op); + vect_finish_stmt_generation (stmt, new_stmt, gsi); + mask_op = var; + } + } + src_op = mask_op; + } + + new_stmt = gimple_build_call (gs_info->decl, 5, src_op, ptr, op, + mask_op, scale); + + if (!useless_type_conversion_p (vectype, rettype)) + { + gcc_assert (must_eq (TYPE_VECTOR_SUBPARTS (vectype), + TYPE_VECTOR_SUBPARTS (rettype))); + op = vect_get_new_ssa_name (rettype, vect_simple_var); + gimple_call_set_lhs (new_stmt, op); + vect_finish_stmt_generation (stmt, new_stmt, gsi); + var = make_ssa_name (vec_dest); + op = build1 (VIEW_CONVERT_EXPR, vectype, op); + new_stmt = gimple_build_assign (var, VIEW_CONVERT_EXPR, op); + } + else + { + var = make_ssa_name (vec_dest, new_stmt); + gimple_call_set_lhs (new_stmt, var); + } + + vect_finish_stmt_generation (stmt, new_stmt, gsi); + + if (modifier == NARROW) + { + if ((j & 1) == 0) + { + prev_res = var; + continue; + } + var = permute_vec_elements (prev_res, var, perm_mask, stmt, gsi); + new_stmt = SSA_NAME_DEF_STMT (var); + } + + if (prev_stmt_info == NULL) + STMT_VINFO_VEC_STMT (stmt_info) = *vec_stmt = new_stmt; + else + STMT_VINFO_RELATED_STMT (prev_stmt_info) = new_stmt; + prev_stmt_info = vinfo_for_stmt (new_stmt); + } +} + /* Function vectorizable_mask_load_store. Check if STMT performs a conditional load or store that can be vectorized. @@ -2219,9 +2419,8 @@ vectorizable_mask_load_store (gimple *st tree dummy; tree dataref_ptr = NULL_TREE; gimple *ptr_incr; - poly_uint64 nunits = TYPE_VECTOR_SUBPARTS (vectype); int ncopies; - int i, j; + int i; bool inv_p; gather_scatter_info gs_info; vec_load_store_type vls_type; @@ -2319,170 +2518,7 @@ vectorizable_mask_load_store (gimple *st if (memory_access_type == VMAT_GATHER_SCATTER) { - tree vec_oprnd0 = NULL_TREE, op; - tree arglist = TYPE_ARG_TYPES (TREE_TYPE (gs_info.decl)); - tree rettype, srctype, ptrtype, idxtype, masktype, scaletype; - tree ptr, vec_mask = NULL_TREE, mask_op = NULL_TREE, var, scale; - tree perm_mask = NULL_TREE, prev_res = NULL_TREE; - tree mask_perm_mask = NULL_TREE; - edge pe = loop_preheader_edge (loop); - gimple_seq seq; - basic_block new_bb; - enum { NARROW, NONE, WIDEN } modifier; - poly_uint64 gather_off_nunits - = TYPE_VECTOR_SUBPARTS (gs_info.offset_vectype); - - rettype = TREE_TYPE (TREE_TYPE (gs_info.decl)); - srctype = TREE_VALUE (arglist); arglist = TREE_CHAIN (arglist); - ptrtype = TREE_VALUE (arglist); arglist = TREE_CHAIN (arglist); - idxtype = TREE_VALUE (arglist); arglist = TREE_CHAIN (arglist); - masktype = TREE_VALUE (arglist); arglist = TREE_CHAIN (arglist); - scaletype = TREE_VALUE (arglist); - gcc_checking_assert (types_compatible_p (srctype, rettype) - && types_compatible_p (srctype, masktype)); - - if (must_eq (nunits, gather_off_nunits)) - modifier = NONE; - else if (must_eq (nunits * 2, gather_off_nunits)) - { - modifier = WIDEN; - - /* Currently widening gathers and scatters are only supported for - fixed-length vectors. */ - int count = gather_off_nunits.to_constant (); - auto_vec_perm_indices sel (count); - for (i = 0; i < count; ++i) - sel.quick_push (i | (count / 2)); - - perm_mask = vect_gen_perm_mask_checked (gs_info.offset_vectype, sel); - } - else if (must_eq (nunits, gather_off_nunits * 2)) - { - modifier = NARROW; - - /* Currently narrowing gathers and scatters are only supported for - fixed-length vectors. */ - int count = nunits.to_constant (); - auto_vec_perm_indices sel (count); - sel.quick_grow (count); - for (i = 0; i < count; ++i) - sel[i] = i < count / 2 ? i : i + count / 2; - - perm_mask = vect_gen_perm_mask_checked (vectype, sel); - ncopies *= 2; - for (i = 0; i < count; ++i) - sel[i] = i | (count / 2); - mask_perm_mask = vect_gen_perm_mask_checked (masktype, sel); - } - else - gcc_unreachable (); - - vec_dest = vect_create_destination_var (gimple_call_lhs (stmt), vectype); - - ptr = fold_convert (ptrtype, gs_info.base); - if (!is_gimple_min_invariant (ptr)) - { - ptr = force_gimple_operand (ptr, &seq, true, NULL_TREE); - new_bb = gsi_insert_seq_on_edge_immediate (pe, seq); - gcc_assert (!new_bb); - } - - scale = build_int_cst (scaletype, gs_info.scale); - - prev_stmt_info = NULL; - for (j = 0; j < ncopies; ++j) - { - if (modifier == WIDEN && (j & 1)) - op = permute_vec_elements (vec_oprnd0, vec_oprnd0, - perm_mask, stmt, gsi); - else if (j == 0) - op = vec_oprnd0 - = vect_get_vec_def_for_operand (gs_info.offset, stmt); - else - op = vec_oprnd0 - = vect_get_vec_def_for_stmt_copy (gs_info.offset_dt, vec_oprnd0); - - if (!useless_type_conversion_p (idxtype, TREE_TYPE (op))) - { - gcc_assert (must_eq (TYPE_VECTOR_SUBPARTS (TREE_TYPE (op)), - TYPE_VECTOR_SUBPARTS (idxtype))); - var = vect_get_new_ssa_name (idxtype, vect_simple_var); - op = build1 (VIEW_CONVERT_EXPR, idxtype, op); - new_stmt - = gimple_build_assign (var, VIEW_CONVERT_EXPR, op); - vect_finish_stmt_generation (stmt, new_stmt, gsi); - op = var; - } - - if (mask_perm_mask && (j & 1)) - mask_op = permute_vec_elements (mask_op, mask_op, - mask_perm_mask, stmt, gsi); - else - { - if (j == 0) - vec_mask = vect_get_vec_def_for_operand (mask, stmt); - else - { - vect_is_simple_use (vec_mask, loop_vinfo, &def_stmt, &dt); - vec_mask = vect_get_vec_def_for_stmt_copy (dt, vec_mask); - } - - mask_op = vec_mask; - if (!useless_type_conversion_p (masktype, TREE_TYPE (vec_mask))) - { - gcc_assert - (must_eq (TYPE_VECTOR_SUBPARTS (TREE_TYPE (mask_op)), - TYPE_VECTOR_SUBPARTS (masktype))); - var = vect_get_new_ssa_name (masktype, vect_simple_var); - mask_op = build1 (VIEW_CONVERT_EXPR, masktype, mask_op); - new_stmt - = gimple_build_assign (var, VIEW_CONVERT_EXPR, mask_op); - vect_finish_stmt_generation (stmt, new_stmt, gsi); - mask_op = var; - } - } - - new_stmt - = gimple_build_call (gs_info.decl, 5, mask_op, ptr, op, mask_op, - scale); - - if (!useless_type_conversion_p (vectype, rettype)) - { - gcc_assert (must_eq (TYPE_VECTOR_SUBPARTS (vectype), - TYPE_VECTOR_SUBPARTS (rettype))); - op = vect_get_new_ssa_name (rettype, vect_simple_var); - gimple_call_set_lhs (new_stmt, op); - vect_finish_stmt_generation (stmt, new_stmt, gsi); - var = make_ssa_name (vec_dest); - op = build1 (VIEW_CONVERT_EXPR, vectype, op); - new_stmt = gimple_build_assign (var, VIEW_CONVERT_EXPR, op); - } - else - { - var = make_ssa_name (vec_dest, new_stmt); - gimple_call_set_lhs (new_stmt, var); - } - - vect_finish_stmt_generation (stmt, new_stmt, gsi); - - if (modifier == NARROW) - { - if ((j & 1) == 0) - { - prev_res = var; - continue; - } - var = permute_vec_elements (prev_res, var, - perm_mask, stmt, gsi); - new_stmt = SSA_NAME_DEF_STMT (var); - } - - if (prev_stmt_info == NULL) - STMT_VINFO_VEC_STMT (stmt_info) = *vec_stmt = new_stmt; - else - STMT_VINFO_RELATED_STMT (prev_stmt_info) = new_stmt; - prev_stmt_info = vinfo_for_stmt (new_stmt); - } + vect_build_gather_load_calls (stmt, gsi, vec_stmt, &gs_info, mask); return true; } else if (vls_type != VLS_LOAD) @@ -6980,139 +7016,7 @@ vectorizable_load (gimple *stmt, gimple_ if (memory_access_type == VMAT_GATHER_SCATTER) { - tree vec_oprnd0 = NULL_TREE, op; - tree arglist = TYPE_ARG_TYPES (TREE_TYPE (gs_info.decl)); - tree rettype, srctype, ptrtype, idxtype, masktype, scaletype; - tree ptr, mask, var, scale, merge, perm_mask = NULL_TREE, prev_res = NULL_TREE; - edge pe = loop_preheader_edge (loop); - gimple_seq seq; - basic_block new_bb; - enum { NARROW, NONE, WIDEN } modifier; - poly_uint64 gather_off_nunits - = TYPE_VECTOR_SUBPARTS (gs_info.offset_vectype); - - if (must_eq (nunits, gather_off_nunits)) - modifier = NONE; - else if (must_eq (nunits * 2, gather_off_nunits)) - { - modifier = WIDEN; - - /* Currently widening gathers are only supported for - fixed-length vectors. */ - int count = gather_off_nunits.to_constant (); - auto_vec_perm_indices sel (count); - for (i = 0; i < count; ++i) - sel.quick_push (i | (count / 2)); - - perm_mask = vect_gen_perm_mask_checked (gs_info.offset_vectype, sel); - } - else if (must_eq (nunits, gather_off_nunits * 2)) - { - modifier = NARROW; - - /* Currently narrowing gathers are only supported for - fixed-length vectors. */ - int count = nunits.to_constant (); - auto_vec_perm_indices sel (count); - for (i = 0; i < count; ++i) - sel.quick_push (i < count / 2 ? i : i + count / 2); - - perm_mask = vect_gen_perm_mask_checked (vectype, sel); - ncopies *= 2; - } - else - gcc_unreachable (); - - rettype = TREE_TYPE (TREE_TYPE (gs_info.decl)); - srctype = TREE_VALUE (arglist); arglist = TREE_CHAIN (arglist); - ptrtype = TREE_VALUE (arglist); arglist = TREE_CHAIN (arglist); - idxtype = TREE_VALUE (arglist); arglist = TREE_CHAIN (arglist); - masktype = TREE_VALUE (arglist); arglist = TREE_CHAIN (arglist); - scaletype = TREE_VALUE (arglist); - gcc_checking_assert (types_compatible_p (srctype, rettype)); - - vec_dest = vect_create_destination_var (scalar_dest, vectype); - - ptr = fold_convert (ptrtype, gs_info.base); - if (!is_gimple_min_invariant (ptr)) - { - ptr = force_gimple_operand (ptr, &seq, true, NULL_TREE); - new_bb = gsi_insert_seq_on_edge_immediate (pe, seq); - gcc_assert (!new_bb); - } - - /* Currently we support only unconditional gather loads, - so mask should be all ones. */ - mask = vect_build_all_ones_mask (stmt, masktype); - scale = build_int_cst (scaletype, gs_info.scale); - merge = vect_build_zero_merge_argument (stmt, rettype); - - prev_stmt_info = NULL; - for (j = 0; j < ncopies; ++j) - { - if (modifier == WIDEN && (j & 1)) - op = permute_vec_elements (vec_oprnd0, vec_oprnd0, - perm_mask, stmt, gsi); - else if (j == 0) - op = vec_oprnd0 - = vect_get_vec_def_for_operand (gs_info.offset, stmt); - else - op = vec_oprnd0 - = vect_get_vec_def_for_stmt_copy (gs_info.offset_dt, vec_oprnd0); - - if (!useless_type_conversion_p (idxtype, TREE_TYPE (op))) - { - gcc_assert (must_eq (TYPE_VECTOR_SUBPARTS (TREE_TYPE (op)), - TYPE_VECTOR_SUBPARTS (idxtype))); - var = vect_get_new_ssa_name (idxtype, vect_simple_var); - op = build1 (VIEW_CONVERT_EXPR, idxtype, op); - new_stmt - = gimple_build_assign (var, VIEW_CONVERT_EXPR, op); - vect_finish_stmt_generation (stmt, new_stmt, gsi); - op = var; - } - - new_stmt - = gimple_build_call (gs_info.decl, 5, merge, ptr, op, mask, scale); - - if (!useless_type_conversion_p (vectype, rettype)) - { - gcc_assert (must_eq (TYPE_VECTOR_SUBPARTS (vectype), - TYPE_VECTOR_SUBPARTS (rettype))); - op = vect_get_new_ssa_name (rettype, vect_simple_var); - gimple_call_set_lhs (new_stmt, op); - vect_finish_stmt_generation (stmt, new_stmt, gsi); - var = make_ssa_name (vec_dest); - op = build1 (VIEW_CONVERT_EXPR, vectype, op); - new_stmt - = gimple_build_assign (var, VIEW_CONVERT_EXPR, op); - } - else - { - var = make_ssa_name (vec_dest, new_stmt); - gimple_call_set_lhs (new_stmt, var); - } - - vect_finish_stmt_generation (stmt, new_stmt, gsi); - - if (modifier == NARROW) - { - if ((j & 1) == 0) - { - prev_res = var; - continue; - } - var = permute_vec_elements (prev_res, var, - perm_mask, stmt, gsi); - new_stmt = SSA_NAME_DEF_STMT (var); - } - - if (prev_stmt_info == NULL) - STMT_VINFO_VEC_STMT (stmt_info) = *vec_stmt = new_stmt; - else - STMT_VINFO_RELATED_STMT (prev_stmt_info) = new_stmt; - prev_stmt_info = vinfo_for_stmt (new_stmt); - } + vect_build_gather_load_calls (stmt, gsi, vec_stmt, &gs_info, NULL_TREE); return true; } From patchwork Fri Nov 17 09:21:09 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Sandiford X-Patchwork-Id: 119108 Delivered-To: patch@linaro.org Received: by 10.140.22.164 with SMTP id 33csp283502qgn; Fri, 17 Nov 2017 01:22:28 -0800 (PST) X-Google-Smtp-Source: AGs4zMa0JGL9Zh0in+GaFA8nKiRdg+gTJhwQFuWHADfLDmNoyCBd6Ojryiy6ExdNI27r2s6jvsmb X-Received: by 10.84.168.132 with SMTP id f4mr4644418plb.234.1510910548530; Fri, 17 Nov 2017 01:22:28 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1510910548; cv=none; d=google.com; s=arc-20160816; b=rd4XC1xSt4jJh0ajaFqJK66EDPPjuiPwBVcLNe335s3aqh24wX0N9wB8kGy190Z3/x CQGWTvgPyk5PPCI6YTCHQ3u/SMJQSkRsq9zwNwMjxhD0BEmZD070TTMTIDo71SEaBbiC aVqjsaRJNTSGaDG0tcZsw4wg2jJw05CrpH+kZbDHCXgY+c5YoelAwg6UOBS2LeAbWLoE N8Jt2H0AmX4xNFuxp0I7C+OzMEV7MJLW15jpyzUWE412wMeWbUlvwQOhrxyGnw2IRmT6 XU0JhRINbidI0ZzxB6BkJfIn+X0NbrV2q9/t7mCA+I1CoJQ5JJDrEYRPst1+71Eh9h1X vGiQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:user-agent:message-id:in-reply-to:date:references :subject:mail-followup-to:to:from:delivered-to:sender:list-help :list-post:list-archive:list-unsubscribe:list-id:precedence :mailing-list:dkim-signature:domainkey-signature :arc-authentication-results; bh=5xrvVroNGqXqp4KLEfrN0cF+CwZEMtRffJB9waQ+dY4=; b=Ftqj7DHYETfpOm/hSXUb6fVHojHmNt9pujN66QYXUkNfW8OtBkNVc/CQS+8e04tTyu oJl1nKxscTmmy0QxtDZMtpVG8sUCE0RYkXgZZ6Pbv06Yr4Ux4QuWczNPKSZWDe+z0p4c 9lX9EsNiy3GmpfkF316/EPXEdvIcgB0IYKcUn0eMy5DypzqJheY05MRHMcOPyDPK2GVt AjFMqtz7eK8+kZr55VQ0rZpR+aAXT4IVHV94FHKU0/YLOps+6dpg9whk0nDk/vhn3sho lLyvncqplhMIeTYrCkQe+/meUzPtffByg7GVX/uWj5PyfJYKPG7yElo3TlVc3ofwpls4 UB2Q== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gcc.gnu.org header.s=default header.b=P6QqMDOa; spf=pass (google.com: domain of gcc-patches-return-467101-patch=linaro.org@gcc.gnu.org designates 209.132.180.131 as permitted sender) smtp.mailfrom=gcc-patches-return-467101-patch=linaro.org@gcc.gnu.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from sourceware.org (server1.sourceware.org. [209.132.180.131]) by mx.google.com with ESMTPS id d7si2708127pfj.22.2017.11.17.01.22.28 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 17 Nov 2017 01:22:28 -0800 (PST) Received-SPF: pass (google.com: domain of gcc-patches-return-467101-patch=linaro.org@gcc.gnu.org designates 209.132.180.131 as permitted sender) client-ip=209.132.180.131; Authentication-Results: mx.google.com; dkim=pass header.i=@gcc.gnu.org header.s=default header.b=P6QqMDOa; spf=pass (google.com: domain of gcc-patches-return-467101-patch=linaro.org@gcc.gnu.org designates 209.132.180.131 as permitted sender) smtp.mailfrom=gcc-patches-return-467101-patch=linaro.org@gcc.gnu.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org DomainKey-Signature: a=rsa-sha1; c=nofws; d=gcc.gnu.org; h=list-id :list-unsubscribe:list-archive:list-post:list-help:sender:from :to:subject:references:date:in-reply-to:message-id:mime-version :content-type; q=dns; s=default; b=REr4BKyAN7scjLOMKDvyvvOyzJ9Ye 2ChoFEs4zj5MIV+zGKlWJEYPAJ2haIQo44EDFpyLzShTed0k/z2UQKaibHHsiwy5 f6ppgOu0bD4AXyjbCxbGxG11cgBazQPMlQGG93d/Fq32qp5MsVoXIvj0Rz39UGLT 7NxWdhB2S0VZIQ= DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=gcc.gnu.org; h=list-id :list-unsubscribe:list-archive:list-post:list-help:sender:from :to:subject:references:date:in-reply-to:message-id:mime-version :content-type; s=default; bh=jTT1yAP5jf0fw5mTovjyhURCGgM=; b=P6Q qMDOaxLkTkhTF+y3F900AJS++le5Jzt9vl+EVxvmo8FHGwweES6+yNfU6pwGbO5g /vACYWOJZhQgXH72UIjzwXyXgDTi2NaPNLZ7jzgbudeSAD1IWQT/s5LlDWl1jqDQ EIika0ShX3ihRKC/IKfWnd/p3E62lGOIJ5RMi7AE= Received: (qmail 127411 invoked by alias); 17 Nov 2017 09:21:21 -0000 Mailing-List: contact gcc-patches-help@gcc.gnu.org; run by ezmlm Precedence: bulk List-Id: List-Unsubscribe: List-Archive: List-Post: List-Help: Sender: gcc-patches-owner@gcc.gnu.org Delivered-To: mailing list gcc-patches@gcc.gnu.org Received: (qmail 127107 invoked by uid 89); 17 Nov 2017 09:21:16 -0000 Authentication-Results: sourceware.org; auth=none X-Virus-Found: No X-Spam-SWARE-Status: No, score=-11.5 required=5.0 tests=AWL, BAYES_00, GIT_PATCH_2, GIT_PATCH_3, KB_WAM_FROM_NAME_SINGLEWORD, RCVD_IN_DNSWL_NONE, SPF_PASS autolearn=ham version=3.3.2 spammy= X-HELO: mail-wm0-f42.google.com Received: from mail-wm0-f42.google.com (HELO mail-wm0-f42.google.com) (74.125.82.42) by sourceware.org (qpsmtpd/0.93/v0.84-503-g423c35a) with ESMTP; Fri, 17 Nov 2017 09:21:12 +0000 Received: by mail-wm0-f42.google.com with SMTP id z3so5048456wme.3 for ; Fri, 17 Nov 2017 01:21:11 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:mail-followup-to:subject:references:date :in-reply-to:message-id:user-agent:mime-version; bh=5xrvVroNGqXqp4KLEfrN0cF+CwZEMtRffJB9waQ+dY4=; b=BbjRaBDKSPgFdlLnYheQQkIhSFGKpDgLhWu0AoMCOB3Rk10HK1EXUSFieL24ahy8L2 mMUykNeB8RIawP+qn3ePIzLpHa6R3hhUkxzmsDzRkyzmIbyRbo8jk0vyQs+32qa74C8J gv5jSjxh9dnsqGJp3Lj7SneszUYsLBXTWj3rJX53MSfOeKHLhNUC1oXpfB7CDmKE6Ldw onxeMHrMhUjlleNFAtZ/amcIKQshlpwA6zktJHQaJtoTIr+G5rCli+4FFG12f0/ZDY80 Z8krciPWkaeronQ37I0+sFFnuShzXxrp7cl1rtVNwhRrdSN61iZbsWGABwYKA7fB9ViC CDIQ== X-Gm-Message-State: AJaThX70SKJje9if7F16FhaduD3GHMVYGoZn9lVoCaL0GooKIxpM9G7T aj/W9ivRqIbKbCMy1Fi4hBU3JrFaMoo= X-Received: by 10.28.112.22 with SMTP id l22mr3417118wmc.35.1510910469216; Fri, 17 Nov 2017 01:21:09 -0800 (PST) Received: from localhost ([2.25.234.120]) by smtp.gmail.com with ESMTPSA id n14sm3767531wrg.38.2017.11.17.01.21.07 for (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Fri, 17 Nov 2017 01:21:08 -0800 (PST) From: Richard Sandiford To: gcc-patches@gcc.gnu.org Mail-Followup-To: gcc-patches@gcc.gnu.org, richard.sandiford@linaro.org Subject: [7/7] Make vectorizable_load/store handle IFN_MASK_LOAD/STORE References: <87375d2rkr.fsf@linaro.org> Date: Fri, 17 Nov 2017 09:21:09 +0000 In-Reply-To: <87375d2rkr.fsf@linaro.org> (Richard Sandiford's message of "Fri, 17 Nov 2017 09:16:20 +0000") Message-ID: <878tf51csa.fsf@linaro.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/25.3 (gnu/linux) MIME-Version: 1.0 After the previous patches, it's easier to see that the remaining inlined transform code in vectorizable_mask_load_store is just a cut-down version of the VMAT_CONTIGUOUS handling in vectorizable_load and vectorizable_store. This patch therefore makes those functions handle masked loads and stores instead. This makes it easier to handle more forms of masked load and store without duplicating logic from the unmasked forms. It also helps with support for fully-masked loops. Richard 2017-11-17 Richard Sandiford gcc/ * tree-vect-stmts.c (vect_get_store_rhs): New function. (vectorizable_mask_load_store): Delete. (vectorizable_call): Return false for masked loads and stores. (vectorizable_store): Handle IFN_MASK_STORE. Use vect_get_store_rhs instead of gimple_assign_rhs1. (vectorizable_load): Handle IFN_MASK_LOAD. (vect_transform_stmt): Don't set is_store for call_vec_info_type. Index: gcc/tree-vect-stmts.c =================================================================== --- gcc/tree-vect-stmts.c 2017-11-17 09:07:00.001060463 +0000 +++ gcc/tree-vect-stmts.c 2017-11-17 09:07:03.706301892 +0000 @@ -1725,6 +1725,26 @@ perm_mask_for_reverse (tree vectype) return vect_gen_perm_mask_checked (vectype, sel); } +/* STMT is either a masked or unconditional store. Return the value + being stored. */ + +static tree +vect_get_store_rhs (gimple *stmt) +{ + if (gassign *assign = dyn_cast (stmt)) + { + gcc_assert (gimple_assign_single_p (assign)); + return gimple_assign_rhs1 (assign); + } + if (gcall *call = dyn_cast (stmt)) + { + internal_fn ifn = gimple_call_internal_fn (call); + gcc_assert (ifn == IFN_MASK_STORE); + return gimple_call_arg (stmt, 3); + } + gcc_unreachable (); +} + /* A subroutine of get_load_store_type, with a subset of the same arguments. Handle the case where STMT is part of a grouped load or store. @@ -2393,251 +2413,6 @@ vect_build_gather_load_calls (gimple *st } } -/* Function vectorizable_mask_load_store. - - Check if STMT performs a conditional load or store that can be vectorized. - If VEC_STMT is also passed, vectorize the STMT: create a vectorized - stmt to replace it, put it in VEC_STMT, and insert it at GSI. - Return FALSE if not a vectorizable STMT, TRUE otherwise. */ - -static bool -vectorizable_mask_load_store (gimple *stmt, gimple_stmt_iterator *gsi, - gimple **vec_stmt, slp_tree slp_node) -{ - tree vec_dest = NULL; - stmt_vec_info stmt_info = vinfo_for_stmt (stmt); - stmt_vec_info prev_stmt_info; - loop_vec_info loop_vinfo = STMT_VINFO_LOOP_VINFO (stmt_info); - struct loop *loop = LOOP_VINFO_LOOP (loop_vinfo); - bool nested_in_vect_loop = nested_in_vect_loop_p (loop, stmt); - struct data_reference *dr = STMT_VINFO_DATA_REF (stmt_info); - tree vectype = STMT_VINFO_VECTYPE (stmt_info); - tree rhs_vectype = NULL_TREE; - tree mask_vectype; - tree elem_type; - gimple *new_stmt; - tree dummy; - tree dataref_ptr = NULL_TREE; - gimple *ptr_incr; - int ncopies; - int i; - bool inv_p; - gather_scatter_info gs_info; - vec_load_store_type vls_type; - tree mask; - gimple *def_stmt; - enum vect_def_type dt; - - if (slp_node != NULL) - return false; - - ncopies = vect_get_num_copies (loop_vinfo, vectype); - gcc_assert (ncopies >= 1); - - /* FORNOW. This restriction should be relaxed. */ - if (nested_in_vect_loop && ncopies > 1) - { - if (dump_enabled_p ()) - dump_printf_loc (MSG_MISSED_OPTIMIZATION, vect_location, - "multiple types in nested loop."); - return false; - } - - if (!STMT_VINFO_RELEVANT_P (stmt_info)) - return false; - - if (STMT_VINFO_DEF_TYPE (stmt_info) != vect_internal_def - && ! vec_stmt) - return false; - - if (!STMT_VINFO_DATA_REF (stmt_info)) - return false; - - mask = gimple_call_arg (stmt, 2); - if (!vect_check_load_store_mask (stmt, mask, &mask_vectype)) - return false; - - elem_type = TREE_TYPE (vectype); - - if (gimple_call_internal_fn (stmt) == IFN_MASK_STORE) - { - tree rhs = gimple_call_arg (stmt, 3); - if (!vect_check_store_rhs (stmt, rhs, &rhs_vectype, &vls_type)) - return false; - } - else - vls_type = VLS_LOAD; - - vect_memory_access_type memory_access_type; - if (!get_load_store_type (stmt, vectype, false, vls_type, ncopies, - &memory_access_type, &gs_info)) - return false; - - if (memory_access_type == VMAT_GATHER_SCATTER) - { - tree arglist = TYPE_ARG_TYPES (TREE_TYPE (gs_info.decl)); - tree masktype - = TREE_VALUE (TREE_CHAIN (TREE_CHAIN (TREE_CHAIN (arglist)))); - if (TREE_CODE (masktype) == INTEGER_TYPE) - { - if (dump_enabled_p ()) - dump_printf_loc (MSG_MISSED_OPTIMIZATION, vect_location, - "masked gather with integer mask not supported."); - return false; - } - } - else if (memory_access_type != VMAT_CONTIGUOUS) - { - if (dump_enabled_p ()) - dump_printf_loc (MSG_MISSED_OPTIMIZATION, vect_location, - "unsupported access type for masked %s.\n", - vls_type == VLS_LOAD ? "load" : "store"); - return false; - } - else if (!VECTOR_MODE_P (TYPE_MODE (vectype)) - || !can_vec_mask_load_store_p (TYPE_MODE (vectype), - TYPE_MODE (mask_vectype), - vls_type == VLS_LOAD)) - return false; - - if (!vec_stmt) /* transformation not required. */ - { - STMT_VINFO_MEMORY_ACCESS_TYPE (stmt_info) = memory_access_type; - STMT_VINFO_TYPE (stmt_info) = call_vec_info_type; - if (vls_type == VLS_LOAD) - vect_model_load_cost (stmt_info, ncopies, memory_access_type, - NULL, NULL, NULL); - else - vect_model_store_cost (stmt_info, ncopies, memory_access_type, - vls_type, NULL, NULL, NULL); - return true; - } - gcc_assert (memory_access_type == STMT_VINFO_MEMORY_ACCESS_TYPE (stmt_info)); - - /* Transform. */ - - if (memory_access_type == VMAT_GATHER_SCATTER) - { - vect_build_gather_load_calls (stmt, gsi, vec_stmt, &gs_info, mask); - return true; - } - else if (vls_type != VLS_LOAD) - { - tree vec_rhs = NULL_TREE, vec_mask = NULL_TREE; - prev_stmt_info = NULL; - LOOP_VINFO_HAS_MASK_STORE (loop_vinfo) = true; - for (i = 0; i < ncopies; i++) - { - unsigned align, misalign; - - if (i == 0) - { - tree rhs = gimple_call_arg (stmt, 3); - vec_rhs = vect_get_vec_def_for_operand (rhs, stmt); - vec_mask = vect_get_vec_def_for_operand (mask, stmt, - mask_vectype); - /* We should have catched mismatched types earlier. */ - gcc_assert (useless_type_conversion_p (vectype, - TREE_TYPE (vec_rhs))); - dataref_ptr = vect_create_data_ref_ptr (stmt, vectype, NULL, - NULL_TREE, &dummy, gsi, - &ptr_incr, false, &inv_p); - gcc_assert (!inv_p); - } - else - { - vect_is_simple_use (vec_rhs, loop_vinfo, &def_stmt, &dt); - vec_rhs = vect_get_vec_def_for_stmt_copy (dt, vec_rhs); - vect_is_simple_use (vec_mask, loop_vinfo, &def_stmt, &dt); - vec_mask = vect_get_vec_def_for_stmt_copy (dt, vec_mask); - dataref_ptr = bump_vector_ptr (dataref_ptr, ptr_incr, gsi, stmt, - TYPE_SIZE_UNIT (vectype)); - } - - align = DR_TARGET_ALIGNMENT (dr); - if (aligned_access_p (dr)) - misalign = 0; - else if (DR_MISALIGNMENT (dr) == -1) - { - align = TYPE_ALIGN_UNIT (elem_type); - misalign = 0; - } - else - misalign = DR_MISALIGNMENT (dr); - set_ptr_info_alignment (get_ptr_info (dataref_ptr), align, - misalign); - tree ptr = build_int_cst (TREE_TYPE (gimple_call_arg (stmt, 1)), - misalign ? least_bit_hwi (misalign) : align); - gcall *call - = gimple_build_call_internal (IFN_MASK_STORE, 4, dataref_ptr, - ptr, vec_mask, vec_rhs); - gimple_call_set_nothrow (call, true); - new_stmt = call; - vect_finish_stmt_generation (stmt, new_stmt, gsi); - if (i == 0) - STMT_VINFO_VEC_STMT (stmt_info) = *vec_stmt = new_stmt; - else - STMT_VINFO_RELATED_STMT (prev_stmt_info) = new_stmt; - prev_stmt_info = vinfo_for_stmt (new_stmt); - } - } - else - { - tree vec_mask = NULL_TREE; - prev_stmt_info = NULL; - vec_dest = vect_create_destination_var (gimple_call_lhs (stmt), vectype); - for (i = 0; i < ncopies; i++) - { - unsigned align, misalign; - - if (i == 0) - { - vec_mask = vect_get_vec_def_for_operand (mask, stmt, - mask_vectype); - dataref_ptr = vect_create_data_ref_ptr (stmt, vectype, NULL, - NULL_TREE, &dummy, gsi, - &ptr_incr, false, &inv_p); - gcc_assert (!inv_p); - } - else - { - vect_is_simple_use (vec_mask, loop_vinfo, &def_stmt, &dt); - vec_mask = vect_get_vec_def_for_stmt_copy (dt, vec_mask); - dataref_ptr = bump_vector_ptr (dataref_ptr, ptr_incr, gsi, stmt, - TYPE_SIZE_UNIT (vectype)); - } - - align = DR_TARGET_ALIGNMENT (dr); - if (aligned_access_p (dr)) - misalign = 0; - else if (DR_MISALIGNMENT (dr) == -1) - { - align = TYPE_ALIGN_UNIT (elem_type); - misalign = 0; - } - else - misalign = DR_MISALIGNMENT (dr); - set_ptr_info_alignment (get_ptr_info (dataref_ptr), align, - misalign); - tree ptr = build_int_cst (TREE_TYPE (gimple_call_arg (stmt, 1)), - misalign ? least_bit_hwi (misalign) : align); - gcall *call - = gimple_build_call_internal (IFN_MASK_LOAD, 3, dataref_ptr, - ptr, vec_mask); - gimple_call_set_lhs (call, make_ssa_name (vec_dest)); - gimple_call_set_nothrow (call, true); - vect_finish_stmt_generation (stmt, call, gsi); - if (i == 0) - STMT_VINFO_VEC_STMT (stmt_info) = *vec_stmt = call; - else - STMT_VINFO_RELATED_STMT (prev_stmt_info) = call; - prev_stmt_info = vinfo_for_stmt (call); - } - } - - return true; -} - /* Check and perform vectorization of BUILT_IN_BSWAP{16,32,64}. */ static bool @@ -2829,8 +2604,8 @@ vectorizable_call (gimple *gs, gimple_st if (gimple_call_internal_p (stmt) && (gimple_call_internal_fn (stmt) == IFN_MASK_LOAD || gimple_call_internal_fn (stmt) == IFN_MASK_STORE)) - return vectorizable_mask_load_store (stmt, gsi, vec_stmt, - slp_node); + /* Handled by vectorizable_load and vectorizable_store. */ + return false; if (gimple_call_lhs (stmt) == NULL_TREE || TREE_CODE (gimple_call_lhs (stmt)) != SSA_NAME) @@ -5828,7 +5603,6 @@ get_group_alias_ptr_type (gimple *first_ vectorizable_store (gimple *stmt, gimple_stmt_iterator *gsi, gimple **vec_stmt, slp_tree slp_node) { - tree scalar_dest; tree data_ref; tree op; tree vec_oprnd = NULL_TREE; @@ -5877,28 +5651,48 @@ vectorizable_store (gimple *stmt, gimple /* Is vectorizable store? */ - if (!is_gimple_assign (stmt)) - return false; + tree mask = NULL_TREE, mask_vectype = NULL_TREE; + if (is_gimple_assign (stmt)) + { + tree scalar_dest = gimple_assign_lhs (stmt); + if (TREE_CODE (scalar_dest) == VIEW_CONVERT_EXPR + && is_pattern_stmt_p (stmt_info)) + scalar_dest = TREE_OPERAND (scalar_dest, 0); + if (TREE_CODE (scalar_dest) != ARRAY_REF + && TREE_CODE (scalar_dest) != BIT_FIELD_REF + && TREE_CODE (scalar_dest) != INDIRECT_REF + && TREE_CODE (scalar_dest) != COMPONENT_REF + && TREE_CODE (scalar_dest) != IMAGPART_EXPR + && TREE_CODE (scalar_dest) != REALPART_EXPR + && TREE_CODE (scalar_dest) != MEM_REF) + return false; + } + else + { + gcall *call = dyn_cast (stmt); + if (!call || !gimple_call_internal_p (call, IFN_MASK_STORE)) + return false; - scalar_dest = gimple_assign_lhs (stmt); - if (TREE_CODE (scalar_dest) == VIEW_CONVERT_EXPR - && is_pattern_stmt_p (stmt_info)) - scalar_dest = TREE_OPERAND (scalar_dest, 0); - if (TREE_CODE (scalar_dest) != ARRAY_REF - && TREE_CODE (scalar_dest) != BIT_FIELD_REF - && TREE_CODE (scalar_dest) != INDIRECT_REF - && TREE_CODE (scalar_dest) != COMPONENT_REF - && TREE_CODE (scalar_dest) != IMAGPART_EXPR - && TREE_CODE (scalar_dest) != REALPART_EXPR - && TREE_CODE (scalar_dest) != MEM_REF) - return false; + if (slp_node != NULL) + { + if (dump_enabled_p ()) + dump_printf_loc (MSG_MISSED_OPTIMIZATION, vect_location, + "SLP of masked stores not supported.\n"); + return false; + } + + ref_type = TREE_TYPE (gimple_call_arg (call, 1)); + mask = gimple_call_arg (call, 2); + if (!vect_check_load_store_mask (stmt, mask, &mask_vectype)) + return false; + } + + op = vect_get_store_rhs (stmt); /* Cannot have hybrid store SLP -- that would mean storing to the same location twice. */ gcc_assert (slp == PURE_SLP_STMT (stmt_info)); - gcc_assert (gimple_assign_single_p (stmt)); - tree vectype = STMT_VINFO_VECTYPE (stmt_info), rhs_vectype = NULL_TREE; poly_uint64 nunits = TYPE_VECTOR_SUBPARTS (vectype); @@ -5929,18 +5723,12 @@ vectorizable_store (gimple *stmt, gimple return false; } - op = gimple_assign_rhs1 (stmt); if (!vect_check_store_rhs (stmt, op, &rhs_vectype, &vls_type)) return false; elem_type = TREE_TYPE (vectype); vec_mode = TYPE_MODE (vectype); - /* FORNOW. In some cases can vectorize even if data-type not supported - (e.g. - array initialization with 0). */ - if (optab_handler (mov_optab, vec_mode) == CODE_FOR_nothing) - return false; - if (!STMT_VINFO_DATA_REF (stmt_info)) return false; @@ -5949,6 +5737,28 @@ vectorizable_store (gimple *stmt, gimple &memory_access_type, &gs_info)) return false; + if (mask) + { + if (memory_access_type != VMAT_CONTIGUOUS) + { + if (dump_enabled_p ()) + dump_printf_loc (MSG_MISSED_OPTIMIZATION, vect_location, + "unsupported access type for masked store.\n"); + return false; + } + if (!VECTOR_MODE_P (vec_mode) + || !can_vec_mask_load_store_p (vec_mode, TYPE_MODE (mask_vectype), + false)) + return false; + } + else + { + /* FORNOW. In some cases can vectorize even if data-type not supported + (e.g. - array initialization with 0). */ + if (optab_handler (mov_optab, vec_mode) == CODE_FOR_nothing) + return false; + } + if (!vec_stmt) /* transformation not required. */ { STMT_VINFO_MEMORY_ACCESS_TYPE (stmt_info) = memory_access_type; @@ -5967,7 +5777,7 @@ vectorizable_store (gimple *stmt, gimple if (memory_access_type == VMAT_GATHER_SCATTER) { - tree vec_oprnd0 = NULL_TREE, vec_oprnd1 = NULL_TREE, op, src; + tree vec_oprnd0 = NULL_TREE, vec_oprnd1 = NULL_TREE, src; tree arglist = TYPE_ARG_TYPES (TREE_TYPE (gs_info.decl)); tree rettype, srctype, ptrtype, idxtype, masktype, scaletype; tree ptr, mask, var, scale, perm_mask = NULL_TREE; @@ -6043,7 +5853,7 @@ vectorizable_store (gimple *stmt, gimple if (j == 0) { src = vec_oprnd1 - = vect_get_vec_def_for_operand (gimple_assign_rhs1 (stmt), stmt); + = vect_get_vec_def_for_operand (op, stmt); op = vec_oprnd0 = vect_get_vec_def_for_operand (gs_info.offset, stmt); } @@ -6143,7 +5953,7 @@ vectorizable_store (gimple *stmt, gimple first_stmt = SLP_TREE_SCALAR_STMTS (slp_node)[0]; gcc_assert (GROUP_FIRST_ELEMENT (vinfo_for_stmt (first_stmt)) == first_stmt); first_dr = STMT_VINFO_DATA_REF (vinfo_for_stmt (first_stmt)); - op = gimple_assign_rhs1 (first_stmt); + op = vect_get_store_rhs (first_stmt); } else /* VEC_NUM is the number of vect stmts to be created for this @@ -6312,7 +6122,7 @@ vectorizable_store (gimple *stmt, gimple elsz = tree_to_uhwi (TYPE_SIZE_UNIT (TREE_TYPE (vectype))); for (j = 0; j < ncopies; j++) { - /* We've set op and dt above, from gimple_assign_rhs1(stmt), + /* We've set op and dt above, from vect_get_store_rhs, and first_stmt == stmt. */ if (j == 0) { @@ -6324,8 +6134,7 @@ vectorizable_store (gimple *stmt, gimple } else { - gcc_assert (gimple_assign_single_p (next_stmt)); - op = gimple_assign_rhs1 (next_stmt); + op = vect_get_store_rhs (next_stmt); vec_oprnd = vect_get_vec_def_for_operand (op, next_stmt); } } @@ -6412,8 +6221,9 @@ vectorizable_store (gimple *stmt, gimple alignment_support_scheme = vect_supportable_dr_alignment (first_dr, false); gcc_assert (alignment_support_scheme); /* Targets with store-lane instructions must not require explicit - realignment. */ - gcc_assert (memory_access_type != VMAT_LOAD_STORE_LANES + realignment. vect_supportable_dr_alignment always returns either + dr_aligned or dr_unaligned_supported for masked operations. */ + gcc_assert ((memory_access_type != VMAT_LOAD_STORE_LANES && !mask) || alignment_support_scheme == dr_aligned || alignment_support_scheme == dr_unaligned_supported); @@ -6426,6 +6236,9 @@ vectorizable_store (gimple *stmt, gimple else aggr_type = vectype; + if (mask) + LOOP_VINFO_HAS_MASK_STORE (loop_vinfo) = true; + /* In case the vectorization factor (VF) is bigger than the number of elements that we can fit in a vectype (nunits), we have to generate more than one vector stmt - i.e - we need to "unroll" the @@ -6466,6 +6279,7 @@ vectorizable_store (gimple *stmt, gimple */ prev_stmt_info = NULL; + tree vec_mask = NULL_TREE; for (j = 0; j < ncopies; j++) { @@ -6496,15 +6310,15 @@ vectorizable_store (gimple *stmt, gimple Therefore, NEXT_STMT can't be NULL_TREE. In case that there is no interleaving, GROUP_SIZE is 1, and only one iteration of the loop will be executed. */ - gcc_assert (next_stmt - && gimple_assign_single_p (next_stmt)); - op = gimple_assign_rhs1 (next_stmt); - + op = vect_get_store_rhs (next_stmt); vec_oprnd = vect_get_vec_def_for_operand (op, next_stmt); dr_chain.quick_push (vec_oprnd); oprnds.quick_push (vec_oprnd); next_stmt = GROUP_NEXT_ELEMENT (vinfo_for_stmt (next_stmt)); } + if (mask) + vec_mask = vect_get_vec_def_for_operand (mask, stmt, + mask_vectype); } /* We should have catched mismatched types earlier. */ @@ -6549,6 +6363,11 @@ vectorizable_store (gimple *stmt, gimple dr_chain[i] = vec_oprnd; oprnds[i] = vec_oprnd; } + if (mask) + { + vect_is_simple_use (vec_mask, vinfo, &def_stmt, &dt); + vec_mask = vect_get_vec_def_for_stmt_copy (dt, vec_mask); + } if (dataref_offset) dataref_offset = int_const_binop (PLUS_EXPR, dataref_offset, @@ -6609,11 +6428,6 @@ vectorizable_store (gimple *stmt, gimple vect_permute_store_chain(). */ vec_oprnd = result_chain[i]; - data_ref = fold_build2 (MEM_REF, vectype, - dataref_ptr, - dataref_offset - ? dataref_offset - : build_int_cst (ref_type, 0)); align = DR_TARGET_ALIGNMENT (first_dr); if (aligned_access_p (first_dr)) misalign = 0; @@ -6621,17 +6435,9 @@ vectorizable_store (gimple *stmt, gimple { align = dr_alignment (vect_dr_behavior (first_dr)); misalign = 0; - TREE_TYPE (data_ref) - = build_aligned_type (TREE_TYPE (data_ref), - align * BITS_PER_UNIT); } else - { - TREE_TYPE (data_ref) - = build_aligned_type (TREE_TYPE (data_ref), - TYPE_ALIGN (elem_type)); - misalign = DR_MISALIGNMENT (first_dr); - } + misalign = DR_MISALIGNMENT (first_dr); if (dataref_offset == NULL_TREE && TREE_CODE (dataref_ptr) == SSA_NAME) set_ptr_info_alignment (get_ptr_info (dataref_ptr), align, @@ -6641,7 +6447,7 @@ vectorizable_store (gimple *stmt, gimple { tree perm_mask = perm_mask_for_reverse (vectype); tree perm_dest - = vect_create_destination_var (gimple_assign_rhs1 (stmt), + = vect_create_destination_var (vect_get_store_rhs (stmt), vectype); tree new_temp = make_ssa_name (perm_dest); @@ -6656,7 +6462,36 @@ vectorizable_store (gimple *stmt, gimple } /* Arguments are ready. Create the new vector stmt. */ - new_stmt = gimple_build_assign (data_ref, vec_oprnd); + if (mask) + { + align = least_bit_hwi (misalign | align); + tree ptr = build_int_cst (ref_type, align); + gcall *call + = gimple_build_call_internal (IFN_MASK_STORE, 4, + dataref_ptr, ptr, + vec_mask, vec_oprnd); + gimple_call_set_nothrow (call, true); + new_stmt = call; + } + else + { + data_ref = fold_build2 (MEM_REF, vectype, + dataref_ptr, + dataref_offset + ? dataref_offset + : build_int_cst (ref_type, 0)); + if (aligned_access_p (first_dr)) + ; + else if (DR_MISALIGNMENT (first_dr) == -1) + TREE_TYPE (data_ref) + = build_aligned_type (TREE_TYPE (data_ref), + align * BITS_PER_UNIT); + else + TREE_TYPE (data_ref) + = build_aligned_type (TREE_TYPE (data_ref), + TYPE_ALIGN (elem_type)); + new_stmt = gimple_build_assign (data_ref, vec_oprnd); + } vect_finish_stmt_generation (stmt, new_stmt, gsi); if (slp) @@ -6847,7 +6682,6 @@ vectorizable_load (gimple *stmt, gimple_ int vec_num; bool slp = (slp_node != NULL); bool slp_perm = false; - enum tree_code code; bb_vec_info bb_vinfo = STMT_VINFO_BB_VINFO (stmt_info); poly_uint64 vf; tree aggr_type; @@ -6862,24 +6696,46 @@ vectorizable_load (gimple *stmt, gimple_ && ! vec_stmt) return false; - /* Is vectorizable load? */ - if (!is_gimple_assign (stmt)) - return false; + tree mask = NULL_TREE, mask_vectype = NULL_TREE; + if (is_gimple_assign (stmt)) + { + scalar_dest = gimple_assign_lhs (stmt); + if (TREE_CODE (scalar_dest) != SSA_NAME) + return false; - scalar_dest = gimple_assign_lhs (stmt); - if (TREE_CODE (scalar_dest) != SSA_NAME) - return false; + tree_code code = gimple_assign_rhs_code (stmt); + if (code != ARRAY_REF + && code != BIT_FIELD_REF + && code != INDIRECT_REF + && code != COMPONENT_REF + && code != IMAGPART_EXPR + && code != REALPART_EXPR + && code != MEM_REF + && TREE_CODE_CLASS (code) != tcc_declaration) + return false; + } + else + { + gcall *call = dyn_cast (stmt); + if (!call || !gimple_call_internal_p (call, IFN_MASK_LOAD)) + return false; - code = gimple_assign_rhs_code (stmt); - if (code != ARRAY_REF - && code != BIT_FIELD_REF - && code != INDIRECT_REF - && code != COMPONENT_REF - && code != IMAGPART_EXPR - && code != REALPART_EXPR - && code != MEM_REF - && TREE_CODE_CLASS (code) != tcc_declaration) - return false; + scalar_dest = gimple_call_lhs (call); + if (!scalar_dest) + return false; + + if (slp_node != NULL) + { + if (dump_enabled_p ()) + dump_printf_loc (MSG_MISSED_OPTIMIZATION, vect_location, + "SLP of masked loads not supported.\n"); + return false; + } + + mask = gimple_call_arg (call, 2); + if (!vect_check_load_store_mask (stmt, mask, &mask_vectype)) + return false; + } if (!STMT_VINFO_DATA_REF (stmt_info)) return false; @@ -6990,6 +6846,38 @@ vectorizable_load (gimple *stmt, gimple_ &memory_access_type, &gs_info)) return false; + if (mask) + { + if (memory_access_type == VMAT_CONTIGUOUS) + { + if (!VECTOR_MODE_P (TYPE_MODE (vectype)) + || !can_vec_mask_load_store_p (TYPE_MODE (vectype), + TYPE_MODE (mask_vectype), true)) + return false; + } + else if (memory_access_type == VMAT_GATHER_SCATTER) + { + tree arglist = TYPE_ARG_TYPES (TREE_TYPE (gs_info.decl)); + tree masktype + = TREE_VALUE (TREE_CHAIN (TREE_CHAIN (TREE_CHAIN (arglist)))); + if (TREE_CODE (masktype) == INTEGER_TYPE) + { + if (dump_enabled_p ()) + dump_printf_loc (MSG_MISSED_OPTIMIZATION, vect_location, + "masked gather with integer mask not" + " supported."); + return false; + } + } + else + { + if (dump_enabled_p ()) + dump_printf_loc (MSG_MISSED_OPTIMIZATION, vect_location, + "unsupported access type for masked load.\n"); + return false; + } + } + if (!vec_stmt) /* transformation not required. */ { if (!slp) @@ -7016,7 +6904,7 @@ vectorizable_load (gimple *stmt, gimple_ if (memory_access_type == VMAT_GATHER_SCATTER) { - vect_build_gather_load_calls (stmt, gsi, vec_stmt, &gs_info, NULL_TREE); + vect_build_gather_load_calls (stmt, gsi, vec_stmt, &gs_info, mask); return true; } @@ -7453,6 +7341,7 @@ vectorizable_load (gimple *stmt, gimple_ else aggr_type = vectype; + tree vec_mask = NULL_TREE; prev_stmt_info = NULL; poly_uint64 group_elt = 0; for (j = 0; j < ncopies; j++) @@ -7500,13 +7389,26 @@ vectorizable_load (gimple *stmt, gimple_ offset, &dummy, gsi, &ptr_incr, simd_lane_access_p, &inv_p, byte_offset); + if (mask) + vec_mask = vect_get_vec_def_for_operand (mask, stmt, + mask_vectype); } - else if (dataref_offset) - dataref_offset = int_const_binop (PLUS_EXPR, dataref_offset, - TYPE_SIZE_UNIT (aggr_type)); else - dataref_ptr = bump_vector_ptr (dataref_ptr, ptr_incr, gsi, stmt, - TYPE_SIZE_UNIT (aggr_type)); + { + if (dataref_offset) + dataref_offset = int_const_binop (PLUS_EXPR, dataref_offset, + TYPE_SIZE_UNIT (aggr_type)); + else + dataref_ptr = bump_vector_ptr (dataref_ptr, ptr_incr, gsi, stmt, + TYPE_SIZE_UNIT (aggr_type)); + if (mask) + { + gimple *def_stmt; + vect_def_type dt; + vect_is_simple_use (vec_mask, vinfo, &def_stmt, &dt); + vec_mask = vect_get_vec_def_for_stmt_copy (dt, vec_mask); + } + } if (grouped_load || slp_perm) dr_chain.create (vec_num); @@ -7554,11 +7456,6 @@ vectorizable_load (gimple *stmt, gimple_ { unsigned int align, misalign; - data_ref - = fold_build2 (MEM_REF, vectype, dataref_ptr, - dataref_offset - ? dataref_offset - : build_int_cst (ref_type, 0)); align = DR_TARGET_ALIGNMENT (dr); if (alignment_support_scheme == dr_aligned) { @@ -7569,21 +7466,44 @@ vectorizable_load (gimple *stmt, gimple_ { align = dr_alignment (vect_dr_behavior (first_dr)); misalign = 0; - TREE_TYPE (data_ref) - = build_aligned_type (TREE_TYPE (data_ref), - align * BITS_PER_UNIT); } else - { - TREE_TYPE (data_ref) - = build_aligned_type (TREE_TYPE (data_ref), - TYPE_ALIGN (elem_type)); - misalign = DR_MISALIGNMENT (first_dr); - } + misalign = DR_MISALIGNMENT (first_dr); if (dataref_offset == NULL_TREE && TREE_CODE (dataref_ptr) == SSA_NAME) set_ptr_info_alignment (get_ptr_info (dataref_ptr), align, misalign); + + if (mask) + { + align = least_bit_hwi (misalign | align); + tree ptr = build_int_cst (ref_type, align); + gcall *call + = gimple_build_call_internal (IFN_MASK_LOAD, 3, + dataref_ptr, ptr, + vec_mask); + gimple_call_set_nothrow (call, true); + new_stmt = call; + data_ref = NULL_TREE; + } + else + { + data_ref + = fold_build2 (MEM_REF, vectype, dataref_ptr, + dataref_offset + ? dataref_offset + : build_int_cst (ref_type, 0)); + if (alignment_support_scheme == dr_aligned) + ; + else if (DR_MISALIGNMENT (first_dr) == -1) + TREE_TYPE (data_ref) + = build_aligned_type (TREE_TYPE (data_ref), + align * BITS_PER_UNIT); + else + TREE_TYPE (data_ref) + = build_aligned_type (TREE_TYPE (data_ref), + TYPE_ALIGN (elem_type)); + } break; } case dr_explicit_realign: @@ -7659,9 +7579,11 @@ vectorizable_load (gimple *stmt, gimple_ gcc_unreachable (); } vec_dest = vect_create_destination_var (scalar_dest, vectype); - new_stmt = gimple_build_assign (vec_dest, data_ref); + /* DATA_REF is null if we've already built the statement. */ + if (data_ref) + new_stmt = gimple_build_assign (vec_dest, data_ref); new_temp = make_ssa_name (vec_dest, new_stmt); - gimple_assign_set_lhs (new_stmt, new_temp); + gimple_set_lhs (new_stmt, new_temp); vect_finish_stmt_generation (stmt, new_stmt, gsi); /* 3. Handle explicit realignment if necessary/supported. @@ -8865,8 +8787,6 @@ vect_transform_stmt (gimple *stmt, gimpl case call_vec_info_type: done = vectorizable_call (stmt, gsi, &vec_stmt, slp_node); stmt = gsi_stmt (*gsi); - if (gimple_call_internal_p (stmt, IFN_MASK_STORE)) - is_store = true; break; case call_simd_clone_vec_info_type: