From patchwork Tue Nov 21 21:25:09 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 119407 Delivered-To: patch@linaro.org Received: by 10.140.22.164 with SMTP id 33csp5815891qgn; Tue, 21 Nov 2017 13:33:08 -0800 (PST) X-Google-Smtp-Source: AGs4zMZlV8xnGdMNb/WQZo3Rv2fDnWPGpYq7CkLyZpVF8bcoOPXQ6EE4HYjVPxt7Sg4KAi4HSDeT X-Received: by 10.37.208.74 with SMTP id h71mr11640752ybg.480.1511299988809; Tue, 21 Nov 2017 13:33:08 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1511299988; cv=none; d=google.com; s=arc-20160816; b=oV3avMISMXJgY+3u8eIX01p5iX2dhpZNM6nOiGbtBJJ229mn8ggMH6hbu8Y22EZt1O GhMMem8UtZqXFoue90ZRlyJ7wTZAvqlI5yCzE9OsrkRs3ktXpkehGGG2eE0aStmg1ftW +qN7ZoUxyI0xU7mx+Vlxt5UvgnjFUNwINDyVPqOhDDfJHwVV/PmfOtbAZTiSiBhoYO8H ZIQ63Z/hJydHEnMxGP50nZgkiLeVOjzmSLUdh2bPjnt+WtHGOIZU0zdKm5j1Ebomp4ef oreUqTqzcHjvcObcM4n3+qWbT+l+3SjGdscSOcfrwEX0yZOJHFsJXGr9c1jYBwqOvZox fJuw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:subject:references:in-reply-to :message-id:date:to:from:dkim-signature:arc-authentication-results; bh=Y+atx2M14YL5KBwzXcmdZ2PYIm+GY9wwqd8Y7EyXQVg=; b=TZbBA7w3Q2xR87UFjNcC6Vh+QSZyuStxSSdezzowuyWYDvMmvQ8r3+PiGDQSu3Ei8X 7wVxdHk5uSfJxCbYBLUGdVP6ZL1Fb+aBhCInQVObSGeEkXTSRosv+fJiLv2OVsk6EZbv LD7iKXA2QyMgiIyzx1T2TF6xxHpRmGImC/6bf1Q2I+gcxChAhwWO79Pc/Y0qmn6VcrGN l7RMsAGpUNKwOoFevN/7+a6PvCzce8U0Ke18fd1TFjCU3T0EcIpr3kF31eGkcMocyhIq 7SjiDJoSArO7jlZd/ix41tG3FnN0Alj8WQVshnol06I43F0vHnSQSnBeAbvtQHilVlSs j6VA== ARC-Authentication-Results: i=1; mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=F0nrFDKZ; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) smtp.mailfrom=qemu-devel-bounces+patch=linaro.org@nongnu.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [2001:4830:134:3::11]) by mx.google.com with ESMTPS id f184si818508ywa.735.2017.11.21.13.33.08 for (version=TLS1 cipher=AES128-SHA bits=128/128); Tue, 21 Nov 2017 13:33:08 -0800 (PST) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) client-ip=2001:4830:134:3::11; Authentication-Results: mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=F0nrFDKZ; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) smtp.mailfrom=qemu-devel-bounces+patch=linaro.org@nongnu.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1]:36598 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1eHGAG-0007hM-7z for patch@linaro.org; Tue, 21 Nov 2017 16:33:08 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:53880) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1eHG51-0004Gc-8b for qemu-devel@nongnu.org; Tue, 21 Nov 2017 16:27:46 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1eHG4y-0007cQ-5E for qemu-devel@nongnu.org; Tue, 21 Nov 2017 16:27:43 -0500 Received: from mail-wm0-x244.google.com ([2a00:1450:400c:c09::244]:44104) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1eHG4x-0007bV-Nu for qemu-devel@nongnu.org; Tue, 21 Nov 2017 16:27:40 -0500 Received: by mail-wm0-x244.google.com with SMTP id r68so6351915wmr.3 for ; Tue, 21 Nov 2017 13:27:39 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:subject:date:message-id:in-reply-to:references; bh=Y+atx2M14YL5KBwzXcmdZ2PYIm+GY9wwqd8Y7EyXQVg=; b=F0nrFDKZ50y6Dx35arkM3tzRmOLABzPYz6aQ0GoGbt3Zcqz3juOpHP64bQVMwL47PJ I/QWZVOYaXt51lxWJW5gevd5mcUYMn8a97Moth982qT1TvxidbdRaGjy+5GpxZ9AJkxM SRpo88B97DLKx9hFZH2YEcPWCKRzbUge56IM8= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references; bh=Y+atx2M14YL5KBwzXcmdZ2PYIm+GY9wwqd8Y7EyXQVg=; b=M15alX9Bj0k4/vURhhFo/jKv+RqjXwserh10AL7nm+RwG6Zsm9h0tkael8S7FCJMCa vTTC2OfLBu/I22hPH+r5vuqYvtJgeuuA3eZ4YejmM3FVYH+vlD9EKHB1MfzSEij5WJKI R1HwyUC78143drD4bHlDtg8o1y9vt9nj1L4xqmzSBst8tp9VNQr5owaCAK7yJHlokcPs CMc28Efo2QEekOMlmlyRyOlfrjbgXNbBDy/d9p/yt7RnQX9mzQGS854kIC2XP2PdjDae JhKexeGevHPd9AkHbIhjWbCdSdmHpiczFtN5MMAqAHy9Be0DbtQ/XLFs+yFauGx6yW1I XJzw== X-Gm-Message-State: AJaThX7hcFjoN0wze0RWzveF0sMu3pgASWE0XgRhPMUWlNVjsKnMNJ8i yCTZdD3Tb+T2HJkGoBcxSjqiUMsLtYw= X-Received: by 10.28.88.65 with SMTP id m62mr2361972wmb.111.1511299657995; Tue, 21 Nov 2017 13:27:37 -0800 (PST) Received: from cloudburst.twiddle.net (70.red-37-158-60.dynamicip.rima-tde.net. [37.158.60.70]) by smtp.gmail.com with ESMTPSA id e124sm706517wmg.34.2017.11.21.13.27.36 for (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Tue, 21 Nov 2017 13:27:37 -0800 (PST) From: Richard Henderson To: qemu-devel@nongnu.org Date: Tue, 21 Nov 2017 22:25:09 +0100 Message-Id: <20171121212534.5177-2-richard.henderson@linaro.org> X-Mailer: git-send-email 2.13.6 In-Reply-To: <20171121212534.5177-1-richard.henderson@linaro.org> References: <20171121212534.5177-1-richard.henderson@linaro.org> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2a00:1450:400c:c09::244 Subject: [Qemu-devel] [PATCH v6 01/26] tcg: Remove TCGV_UNUSED* and TCGV_IS_UNUSED* X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: "Qemu-devel" These are now trivial sets and tests against NULL. Unwrap. Signed-off-by: Richard Henderson --- tcg/tcg-op.h | 4 --- tcg/tcg.h | 9 ------- target/alpha/translate.c | 22 ++++++++-------- target/arm/translate-a64.c | 35 +++++++++++------------- target/arm/translate.c | 29 ++++++++++---------- target/cris/translate.c | 2 +- target/hppa/translate.c | 63 ++++++++++++++++++++++---------------------- target/i386/translate.c | 13 +++++---- target/m68k/translate.c | 14 +++++----- target/mips/translate.c | 2 +- target/nios2/translate.c | 6 ++--- target/ppc/translate.c | 2 +- target/s390x/translate.c | 42 ++++++++++++++--------------- target/sh4/translate.c | 2 +- target/sparc/translate.c | 2 +- target/tilegx/translate.c | 10 +++---- target/unicore32/translate.c | 4 +-- tcg/tcg.c | 4 +-- 18 files changed, 122 insertions(+), 143 deletions(-) -- 2.13.6 diff --git a/tcg/tcg-op.h b/tcg/tcg-op.h index 3129159907..ca07b32b65 100644 --- a/tcg/tcg-op.h +++ b/tcg/tcg-op.h @@ -807,8 +807,6 @@ void tcg_gen_lookup_and_goto_ptr(void); #define tcg_global_mem_new tcg_global_mem_new_i32 #define tcg_temp_local_new() tcg_temp_local_new_i32() #define tcg_temp_free tcg_temp_free_i32 -#define TCGV_UNUSED(x) TCGV_UNUSED_I32(x) -#define TCGV_IS_UNUSED(x) TCGV_IS_UNUSED_I32(x) #define tcg_gen_qemu_ld_tl tcg_gen_qemu_ld_i32 #define tcg_gen_qemu_st_tl tcg_gen_qemu_st_i32 #else @@ -817,8 +815,6 @@ void tcg_gen_lookup_and_goto_ptr(void); #define tcg_global_mem_new tcg_global_mem_new_i64 #define tcg_temp_local_new() tcg_temp_local_new_i64() #define tcg_temp_free tcg_temp_free_i64 -#define TCGV_UNUSED(x) TCGV_UNUSED_I64(x) -#define TCGV_IS_UNUSED(x) TCGV_IS_UNUSED_I64(x) #define tcg_gen_qemu_ld_tl tcg_gen_qemu_ld_i64 #define tcg_gen_qemu_st_tl tcg_gen_qemu_st_i64 #endif diff --git a/tcg/tcg.h b/tcg/tcg.h index cb7b329876..c21194c858 100644 --- a/tcg/tcg.h +++ b/tcg/tcg.h @@ -428,15 +428,6 @@ typedef TCGv_ptr TCGv_env; #error Unhandled TARGET_LONG_BITS value #endif -/* See the comment before tcgv_i32_temp. */ -#define TCGV_UNUSED_I32(x) (x = (TCGv_i32)NULL) -#define TCGV_UNUSED_I64(x) (x = (TCGv_i64)NULL) -#define TCGV_UNUSED_PTR(x) (x = (TCGv_ptr)NULL) - -#define TCGV_IS_UNUSED_I32(x) ((x) == (TCGv_i32)NULL) -#define TCGV_IS_UNUSED_I64(x) ((x) == (TCGv_i64)NULL) -#define TCGV_IS_UNUSED_PTR(x) ((x) == (TCGv_ptr)NULL) - /* call flags */ /* Helper does not read globals (either directly or through an exception). It implies TCG_CALL_NO_WRITE_GLOBALS. */ diff --git a/target/alpha/translate.c b/target/alpha/translate.c index 629f35ec8e..73a1b5e63e 100644 --- a/target/alpha/translate.c +++ b/target/alpha/translate.c @@ -156,7 +156,7 @@ void alpha_translate_init(void) static TCGv load_zero(DisasContext *ctx) { - if (TCGV_IS_UNUSED_I64(ctx->zero)) { + if (!ctx->zero) { ctx->zero = tcg_const_i64(0); } return ctx->zero; @@ -164,7 +164,7 @@ static TCGv load_zero(DisasContext *ctx) static TCGv dest_sink(DisasContext *ctx) { - if (TCGV_IS_UNUSED_I64(ctx->sink)) { + if (!ctx->sink) { ctx->sink = tcg_temp_new(); } return ctx->sink; @@ -172,18 +172,18 @@ static TCGv dest_sink(DisasContext *ctx) static void free_context_temps(DisasContext *ctx) { - if (!TCGV_IS_UNUSED_I64(ctx->sink)) { + if (ctx->sink) { tcg_gen_discard_i64(ctx->sink); tcg_temp_free(ctx->sink); - TCGV_UNUSED_I64(ctx->sink); + ctx->sink = NULL; } - if (!TCGV_IS_UNUSED_I64(ctx->zero)) { + if (ctx->zero) { tcg_temp_free(ctx->zero); - TCGV_UNUSED_I64(ctx->zero); + ctx->zero = NULL; } - if (!TCGV_IS_UNUSED_I64(ctx->lit)) { + if (ctx->lit) { tcg_temp_free(ctx->lit); - TCGV_UNUSED_I64(ctx->lit); + ctx->lit = NULL; } } @@ -2948,9 +2948,9 @@ static int alpha_tr_init_disas_context(DisasContextBase *dcbase, /* Similarly for flush-to-zero. */ ctx->tb_ftz = -1; - TCGV_UNUSED_I64(ctx->zero); - TCGV_UNUSED_I64(ctx->sink); - TCGV_UNUSED_I64(ctx->lit); + ctx->zero = NULL; + ctx->sink = NULL; + ctx->lit = NULL; /* Bound the number of insns to execute to those left on the page. */ if (in_superpage(ctx, ctx->base.pc_first)) { diff --git a/target/arm/translate-a64.c b/target/arm/translate-a64.c index 625ef2dfd2..460bab5987 100644 --- a/target/arm/translate-a64.c +++ b/target/arm/translate-a64.c @@ -405,10 +405,7 @@ static void unallocated_encoding(DisasContext *s) static void init_tmp_a64_array(DisasContext *s) { #ifdef CONFIG_DEBUG_TCG - int i; - for (i = 0; i < ARRAY_SIZE(s->tmp_a64); i++) { - TCGV_UNUSED_I64(s->tmp_a64[i]); - } + memset(s->tmp_a64, 0, sizeof(s->tmp_a64)); #endif s->tmp_a64_count = 0; } @@ -6276,7 +6273,7 @@ static void disas_simd_scalar_pairwise(DisasContext *s, uint32_t insn) return; } - TCGV_UNUSED_PTR(fpst); + fpst = NULL; break; case 0xc: /* FMAXNMP */ case 0xd: /* FADDP */ @@ -6371,7 +6368,7 @@ static void disas_simd_scalar_pairwise(DisasContext *s, uint32_t insn) tcg_temp_free_i32(tcg_res); } - if (!TCGV_IS_UNUSED_PTR(fpst)) { + if (fpst) { tcg_temp_free_ptr(fpst); } } @@ -6387,7 +6384,7 @@ static void handle_shri_with_rndacc(TCGv_i64 tcg_res, TCGv_i64 tcg_src, bool is_u, int size, int shift) { bool extended_result = false; - bool round = !TCGV_IS_UNUSED_I64(tcg_rnd); + bool round = tcg_rnd != NULL; int ext_lshift = 0; TCGv_i64 tcg_src_hi; @@ -6533,7 +6530,7 @@ static void handle_scalar_simd_shri(DisasContext *s, uint64_t round_const = 1ULL << (shift - 1); tcg_round = tcg_const_i64(round_const); } else { - TCGV_UNUSED_I64(tcg_round); + tcg_round = NULL; } tcg_rn = read_fp_dreg(s, rn); @@ -6649,7 +6646,7 @@ static void handle_vec_simd_sqshrn(DisasContext *s, bool is_scalar, bool is_q, uint64_t round_const = 1ULL << (shift - 1); tcg_round = tcg_const_i64(round_const); } else { - TCGV_UNUSED_I64(tcg_round); + tcg_round = NULL; } for (i = 0; i < elements; i++) { @@ -8239,8 +8236,8 @@ static void disas_simd_scalar_two_reg_misc(DisasContext *s, uint32_t insn) gen_helper_set_rmode(tcg_rmode, tcg_rmode, cpu_env); tcg_fpstatus = get_fpstatus_ptr(); } else { - TCGV_UNUSED_I32(tcg_rmode); - TCGV_UNUSED_PTR(tcg_fpstatus); + tcg_rmode = NULL; + tcg_fpstatus = NULL; } if (size == 3) { @@ -8360,7 +8357,7 @@ static void handle_vec_simd_shri(DisasContext *s, bool is_q, bool is_u, uint64_t round_const = 1ULL << (shift - 1); tcg_round = tcg_const_i64(round_const); } else { - TCGV_UNUSED_I64(tcg_round); + tcg_round = NULL; } for (i = 0; i < elements; i++) { @@ -8502,7 +8499,7 @@ static void handle_vec_simd_shrn(DisasContext *s, bool is_q, uint64_t round_const = 1ULL << (shift - 1); tcg_round = tcg_const_i64(round_const); } else { - TCGV_UNUSED_I64(tcg_round); + tcg_round = NULL; } for (i = 0; i < elements; i++) { @@ -9168,7 +9165,7 @@ static void handle_simd_3same_pair(DisasContext *s, int is_q, int u, int opcode, if (opcode >= 0x58) { fpst = get_fpstatus_ptr(); } else { - TCGV_UNUSED_PTR(fpst); + fpst = NULL; } if (!fp_access_check(s)) { @@ -9305,7 +9302,7 @@ static void handle_simd_3same_pair(DisasContext *s, int is_q, int u, int opcode, } } - if (!TCGV_IS_UNUSED_PTR(fpst)) { + if (fpst) { tcg_temp_free_ptr(fpst); } } @@ -10226,13 +10223,13 @@ static void disas_simd_two_reg_misc(DisasContext *s, uint32_t insn) if (need_fpstatus) { tcg_fpstatus = get_fpstatus_ptr(); } else { - TCGV_UNUSED_PTR(tcg_fpstatus); + tcg_fpstatus = NULL; } if (need_rmode) { tcg_rmode = tcg_const_i32(arm_rmode_to_sf(rmode)); gen_helper_set_rmode(tcg_rmode, tcg_rmode, cpu_env); } else { - TCGV_UNUSED_I32(tcg_rmode); + tcg_rmode = NULL; } if (size == 3) { @@ -10593,7 +10590,7 @@ static void disas_simd_indexed(DisasContext *s, uint32_t insn) if (is_fp) { fpst = get_fpstatus_ptr(); } else { - TCGV_UNUSED_PTR(fpst); + fpst = NULL; } if (size == 3) { @@ -10917,7 +10914,7 @@ static void disas_simd_indexed(DisasContext *s, uint32_t insn) } } - if (!TCGV_IS_UNUSED_PTR(fpst)) { + if (fpst) { tcg_temp_free_ptr(fpst); } } diff --git a/target/arm/translate.c b/target/arm/translate.c index 4afb0c86ec..0b74ce7dec 100644 --- a/target/arm/translate.c +++ b/target/arm/translate.c @@ -2165,8 +2165,8 @@ static int disas_iwmmxt_insn(DisasContext *s, uint32_t insn) tmp3 = tcg_const_i32((insn & 1) << 5); break; default: - TCGV_UNUSED_I32(tmp2); - TCGV_UNUSED_I32(tmp3); + tmp2 = NULL; + tmp3 = NULL; } gen_helper_iwmmxt_insr(cpu_M0, cpu_M0, tmp, tmp2, tmp3); tcg_temp_free_i32(tmp3); @@ -4935,7 +4935,7 @@ static int disas_neon_ls_insn(DisasContext *s, uint32_t insn) } } else /* size == 0 */ { if (load) { - TCGV_UNUSED_I32(tmp2); + tmp2 = NULL; for (n = 0; n < 4; n++) { tmp = tcg_temp_new_i32(); gen_aa32_ld8u(s, tmp, addr, get_mem_index(s)); @@ -6639,11 +6639,11 @@ static int disas_neon_data_insn(DisasContext *s, uint32_t insn) tmp = neon_load_reg(rn, 1); neon_store_scratch(2, tmp); } - TCGV_UNUSED_I32(tmp3); + tmp3 = NULL; for (pass = 0; pass < 2; pass++) { if (src1_wide) { neon_load_reg64(cpu_V0, rn + pass); - TCGV_UNUSED_I32(tmp); + tmp = NULL; } else { if (pass == 1 && rd == rn) { tmp = neon_load_scratch(2); @@ -6656,7 +6656,7 @@ static int disas_neon_data_insn(DisasContext *s, uint32_t insn) } if (src2_wide) { neon_load_reg64(cpu_V1, rm + pass); - TCGV_UNUSED_I32(tmp2); + tmp2 = NULL; } else { if (pass == 1 && rd == rm) { tmp2 = neon_load_scratch(2); @@ -7074,7 +7074,7 @@ static int disas_neon_data_insn(DisasContext *s, uint32_t insn) if (rm & 1) { return 1; } - TCGV_UNUSED_I32(tmp2); + tmp2 = NULL; for (pass = 0; pass < 2; pass++) { neon_load_reg64(cpu_V0, rm + pass); tmp = tcg_temp_new_i32(); @@ -7213,7 +7213,7 @@ static int disas_neon_data_insn(DisasContext *s, uint32_t insn) if (neon_2rm_is_float_op(op)) { tcg_gen_ld_f32(cpu_F0s, cpu_env, neon_reg_offset(rm, pass)); - TCGV_UNUSED_I32(tmp); + tmp = NULL; } else { tmp = neon_load_reg(rm, pass); } @@ -8662,7 +8662,7 @@ static void disas_arm_insn(DisasContext *s, unsigned int insn) rn = (insn >> 16) & 0xf; tmp = load_reg(s, rn); } else { - TCGV_UNUSED_I32(tmp); + tmp = NULL; } rd = (insn >> 12) & 0xf; switch(op1) { @@ -9501,7 +9501,7 @@ static void disas_arm_insn(DisasContext *s, unsigned int insn) /* compute total size */ loaded_base = 0; - TCGV_UNUSED_I32(loaded_var); + loaded_var = NULL; n = 0; for(i=0;i<16;i++) { if (insn & (1 << i)) @@ -10043,7 +10043,7 @@ static int disas_thumb2_insn(DisasContext *s, uint32_t insn) tcg_gen_addi_i32(addr, addr, -offset); } - TCGV_UNUSED_I32(loaded_var); + loaded_var = NULL; for (i = 0; i < 16; i++) { if ((insn & (1 << i)) == 0) continue; @@ -11324,7 +11324,7 @@ static void disas_thumb_insn(DisasContext *s, uint32_t insn) } else if (op != 0xf) { /* mvn doesn't read its first operand */ tmp = load_reg(s, rd); } else { - TCGV_UNUSED_I32(tmp); + tmp = NULL; } tmp2 = load_reg(s, rm); @@ -11655,7 +11655,7 @@ static void disas_thumb_insn(DisasContext *s, uint32_t insn) tcg_gen_addi_i32(addr, addr, 4); } } - TCGV_UNUSED_I32(tmp); + tmp = NULL; if (insn & (1 << 8)) { if (insn & (1 << 11)) { /* pop pc */ @@ -11800,8 +11800,7 @@ static void disas_thumb_insn(DisasContext *s, uint32_t insn) case 12: { /* load/store multiple */ - TCGv_i32 loaded_var; - TCGV_UNUSED_I32(loaded_var); + TCGv_i32 loaded_var = NULL; rn = (insn >> 8) & 0x7; addr = load_reg(s, rn); for (i = 0; i < 8; i++) { diff --git a/target/cris/translate.c b/target/cris/translate.c index 2831419845..74822ed31f 100644 --- a/target/cris/translate.c +++ b/target/cris/translate.c @@ -2603,7 +2603,7 @@ static int dec_movem_mr(CPUCRISState *env, DisasContext *dc) tcg_gen_addi_tl(addr, cpu_R[dc->op1], i * 8); gen_load(dc, tmp32, addr, 4, 0); } else { - TCGV_UNUSED(tmp32); + tmp32 = NULL; } tcg_temp_free(addr); diff --git a/target/hppa/translate.c b/target/hppa/translate.c index 53aa1f88c4..31d9a2a31b 100644 --- a/target/hppa/translate.c +++ b/target/hppa/translate.c @@ -125,7 +125,7 @@ void hppa_translate_init(void) int i; - TCGV_UNUSED(cpu_gr[0]); + cpu_gr[0] = NULL; for (i = 1; i < 32; i++) { cpu_gr[i] = tcg_global_mem_new(cpu_env, offsetof(CPUHPPAState, gr[i]), @@ -140,28 +140,31 @@ void hppa_translate_init(void) static DisasCond cond_make_f(void) { - DisasCond r = { .c = TCG_COND_NEVER }; - TCGV_UNUSED(r.a0); - TCGV_UNUSED(r.a1); - return r; + return (DisasCond){ + .c = TCG_COND_NEVER, + .a0 = NULL, + .a1 = NULL, + }; } static DisasCond cond_make_n(void) { - DisasCond r = { .c = TCG_COND_NE, .a0_is_n = true, .a1_is_0 = true }; - r.a0 = cpu_psw_n; - TCGV_UNUSED(r.a1); - return r; + return (DisasCond){ + .c = TCG_COND_NE, + .a0 = cpu_psw_n, + .a0_is_n = true, + .a1 = NULL, + .a1_is_0 = true + }; } static DisasCond cond_make_0(TCGCond c, TCGv a0) { - DisasCond r = { .c = c, .a1_is_0 = true }; + DisasCond r = { .c = c, .a1 = NULL, .a1_is_0 = true }; assert (c != TCG_COND_NEVER && c != TCG_COND_ALWAYS); r.a0 = tcg_temp_new(); tcg_gen_mov_tl(r.a0, a0); - TCGV_UNUSED(r.a1); return r; } @@ -199,8 +202,8 @@ static void cond_free(DisasCond *cond) } cond->a0_is_n = false; cond->a1_is_0 = false; - TCGV_UNUSED(cond->a0); - TCGV_UNUSED(cond->a1); + cond->a0 = NULL; + cond->a1 = NULL; /* fallthru */ case TCG_COND_ALWAYS: cond->c = TCG_COND_NEVER; @@ -716,9 +719,8 @@ static DisasCond do_sed_cond(unsigned orig, TCGv res) static DisasCond do_unit_cond(unsigned cf, TCGv res, TCGv in1, TCGv in2) { DisasCond cond; - TCGv tmp, cb; + TCGv tmp, cb = NULL; - TCGV_UNUSED(cb); if (cf & 8) { /* Since we want to test lots of carry-out bits all at once, do not * do our normal thing and compute carry-in of bit B+1 since that @@ -826,8 +828,8 @@ static DisasJumpType do_add(DisasContext *ctx, unsigned rt, TCGv in1, TCGv in2, DisasCond cond; dest = tcg_temp_new(); - TCGV_UNUSED(cb); - TCGV_UNUSED(cb_msb); + cb = NULL; + cb_msb = NULL; if (shift) { tmp = get_temp(ctx); @@ -856,7 +858,7 @@ static DisasJumpType do_add(DisasContext *ctx, unsigned rt, TCGv in1, TCGv in2, } /* Compute signed overflow if required. */ - TCGV_UNUSED(sv); + sv = NULL; if (is_tsv || c == 6) { sv = do_add_sv(ctx, dest, in1, in2); if (is_tsv) { @@ -919,7 +921,7 @@ static DisasJumpType do_sub(DisasContext *ctx, unsigned rt, TCGv in1, TCGv in2, tcg_temp_free(zero); /* Compute signed overflow if required. */ - TCGV_UNUSED(sv); + sv = NULL; if (is_tsv || c == 6) { sv = do_sub_sv(ctx, dest, in1, in2); if (is_tsv) { @@ -965,7 +967,7 @@ static DisasJumpType do_cmpclr(DisasContext *ctx, unsigned rt, TCGv in1, tcg_gen_sub_tl(dest, in1, in2); /* Compute signed overflow if required. */ - TCGV_UNUSED(sv); + sv = NULL; if ((cf >> 1) == 6) { sv = do_sub_sv(ctx, dest, in1, in2); } @@ -2070,8 +2072,7 @@ static DisasJumpType trans_ds(DisasContext *ctx, uint32_t insn, /* Install the new nullification. */ if (cf) { - TCGv sv; - TCGV_UNUSED(sv); + TCGv sv = NULL; if (cf >> 1 == 6) { /* ??? The lshift is supposed to contribute to overflow. */ sv = do_add_sv(ctx, dest, add1, add2); @@ -2542,7 +2543,7 @@ static DisasJumpType trans_cmpb(DisasContext *ctx, uint32_t insn, tcg_gen_sub_tl(dest, in1, in2); - TCGV_UNUSED(sv); + sv = NULL; if (c == 6) { sv = do_sub_sv(ctx, dest, in1, in2); } @@ -2571,8 +2572,8 @@ static DisasJumpType trans_addb(DisasContext *ctx, uint32_t insn, } in2 = load_gpr(ctx, r); dest = dest_gpr(ctx, r); - TCGV_UNUSED(sv); - TCGV_UNUSED(cb_msb); + sv = NULL; + cb_msb = NULL; switch (c) { default: @@ -3732,18 +3733,16 @@ static int hppa_tr_init_disas_context(DisasContextBase *dcbase, { DisasContext *ctx = container_of(dcbase, DisasContext, base); TranslationBlock *tb = ctx->base.tb; - int i, bound; + int bound; ctx->cs = cs; ctx->iaoq_f = tb->pc; ctx->iaoq_b = tb->cs_base; ctx->iaoq_n = -1; - TCGV_UNUSED(ctx->iaoq_n_var); + ctx->iaoq_n_var = NULL; ctx->ntemps = 0; - for (i = 0; i < ARRAY_SIZE(ctx->temps); ++i) { - TCGV_UNUSED(ctx->temps[i]); - } + memset(ctx->temps, 0, sizeof(ctx->temps)); bound = -(tb->pc | TARGET_PAGE_MASK) / 4; return MIN(max_insns, bound); @@ -3804,7 +3803,7 @@ static void hppa_tr_translate_insn(DisasContextBase *dcbase, CPUState *cs) tcg_gen_addi_tl(ctx->iaoq_n_var, cpu_iaoq_b, 4); } else { ctx->iaoq_n = ctx->iaoq_b + 4; - TCGV_UNUSED(ctx->iaoq_n_var); + ctx->iaoq_n_var = NULL; } if (unlikely(ctx->null_cond.c == TCG_COND_ALWAYS)) { @@ -3819,7 +3818,7 @@ static void hppa_tr_translate_insn(DisasContextBase *dcbase, CPUState *cs) /* Free any temporaries allocated. */ for (i = 0, n = ctx->ntemps; i < n; ++i) { tcg_temp_free(ctx->temps[i]); - TCGV_UNUSED(ctx->temps[i]); + ctx->temps[i] = NULL; } ctx->ntemps = 0; diff --git a/target/i386/translate.c b/target/i386/translate.c index 088a9d9766..8c5f12985a 100644 --- a/target/i386/translate.c +++ b/target/i386/translate.c @@ -689,7 +689,7 @@ static void gen_compute_eflags(DisasContext *s) return; } - TCGV_UNUSED(zero); + zero = NULL; dst = cpu_cc_dst; src1 = cpu_cc_src; src2 = cpu_cc_src2; @@ -2050,9 +2050,8 @@ static AddressParts gen_lea_modrm_0(CPUX86State *env, DisasContext *s, /* Compute the address, with a minimum number of TCG ops. */ static TCGv gen_lea_modrm_1(AddressParts a) { - TCGv ea; + TCGv ea = NULL; - TCGV_UNUSED(ea); if (a.index >= 0) { if (a.scale == 0) { ea = cpu_regs[a.index]; @@ -2067,7 +2066,7 @@ static TCGv gen_lea_modrm_1(AddressParts a) } else if (a.base >= 0) { ea = cpu_regs[a.base]; } - if (TCGV_IS_UNUSED(ea)) { + if (!ea) { tcg_gen_movi_tl(cpu_A0, a.disp); ea = cpu_A0; } else if (a.disp != 0) { @@ -3951,7 +3950,7 @@ static void gen_sse(CPUX86State *env, DisasContext *s, int b, gen_ldst_modrm(env, s, modrm, ot, OR_TMP0, 0); /* Re-use the carry-out from a previous round. */ - TCGV_UNUSED(carry_in); + carry_in = NULL; carry_out = (b == 0x1f6 ? cpu_cc_dst : cpu_cc_src2); switch (s->cc_op) { case CC_OP_ADCX: @@ -3979,7 +3978,7 @@ static void gen_sse(CPUX86State *env, DisasContext *s, int b, break; } /* If we can't reuse carry-out, get it out of EFLAGS. */ - if (TCGV_IS_UNUSED(carry_in)) { + if (!carry_in) { if (s->cc_op != CC_OP_ADCX && s->cc_op != CC_OP_ADOX) { gen_compute_eflags(s); } @@ -7672,7 +7671,7 @@ static target_ulong disas_insn(DisasContext *s, CPUState *cpu) tcg_gen_mov_tl(a0, cpu_A0); } else { gen_op_mov_v_reg(ot, t0, rm); - TCGV_UNUSED(a0); + a0 = NULL; } gen_op_mov_v_reg(ot, t1, reg); tcg_gen_andi_tl(cpu_tmp0, t0, 3); diff --git a/target/m68k/translate.c b/target/m68k/translate.c index b60909222c..e78c9f208b 100644 --- a/target/m68k/translate.c +++ b/target/m68k/translate.c @@ -3956,8 +3956,8 @@ DISAS_INSN(bfop_reg) int ofs = extract32(ext, 6, 5); /* big bit-endian */ TCGv mask, tofs, tlen; - TCGV_UNUSED(tofs); - TCGV_UNUSED(tlen); + tofs = NULL; + tlen = NULL; if ((insn & 0x0f00) == 0x0d00) { /* bfffo */ tofs = tcg_temp_new(); tlen = tcg_temp_new(); @@ -3973,7 +3973,7 @@ DISAS_INSN(bfop_reg) } tcg_gen_andi_i32(QREG_CC_N, QREG_CC_N, ~maski); mask = tcg_const_i32(ror32(maski, ofs)); - if (!TCGV_IS_UNUSED(tofs)) { + if (tofs) { tcg_gen_movi_i32(tofs, ofs); tcg_gen_movi_i32(tlen, len); } @@ -3985,13 +3985,13 @@ DISAS_INSN(bfop_reg) tcg_gen_andi_i32(tmp, tmp, 31); mask = tcg_const_i32(0x7fffffffu); tcg_gen_shr_i32(mask, mask, tmp); - if (!TCGV_IS_UNUSED(tlen)) { + if (tlen) { tcg_gen_addi_i32(tlen, tmp, 1); } } else { /* Immediate width */ mask = tcg_const_i32(0x7fffffffu >> (len - 1)); - if (!TCGV_IS_UNUSED(tlen)) { + if (tlen) { tcg_gen_movi_i32(tlen, len); } } @@ -4001,7 +4001,7 @@ DISAS_INSN(bfop_reg) tcg_gen_rotl_i32(QREG_CC_N, src, tmp); tcg_gen_andc_i32(QREG_CC_N, QREG_CC_N, mask); tcg_gen_rotr_i32(mask, mask, tmp); - if (!TCGV_IS_UNUSED(tofs)) { + if (tofs) { tcg_gen_mov_i32(tofs, tmp); } } else { @@ -4009,7 +4009,7 @@ DISAS_INSN(bfop_reg) tcg_gen_rotli_i32(QREG_CC_N, src, ofs); tcg_gen_andc_i32(QREG_CC_N, QREG_CC_N, mask); tcg_gen_rotri_i32(mask, mask, ofs); - if (!TCGV_IS_UNUSED(tofs)) { + if (tofs) { tcg_gen_movi_i32(tofs, ofs); } } diff --git a/target/mips/translate.c b/target/mips/translate.c index b022f840c9..d05ee67e63 100644 --- a/target/mips/translate.c +++ b/target/mips/translate.c @@ -20453,7 +20453,7 @@ void mips_tcg_init(void) { int i; - TCGV_UNUSED(cpu_gpr[0]); + cpu_gpr[0] = NULL; for (i = 1; i < 32; i++) cpu_gpr[i] = tcg_global_mem_new(cpu_env, offsetof(CPUMIPSState, active_tc.gpr[i]), diff --git a/target/nios2/translate.c b/target/nios2/translate.c index 72329002ac..0afd10f08a 100644 --- a/target/nios2/translate.c +++ b/target/nios2/translate.c @@ -124,7 +124,7 @@ static uint8_t get_opxcode(uint32_t code) static TCGv load_zero(DisasContext *dc) { - if (TCGV_IS_UNUSED_I32(dc->zero)) { + if (!dc->zero) { dc->zero = tcg_const_i32(0); } return dc->zero; @@ -754,12 +754,12 @@ static void handle_instruction(DisasContext *dc, CPUNios2State *env) goto illegal_op; } - TCGV_UNUSED_I32(dc->zero); + dc->zero = NULL; instr = &i_type_instructions[op]; instr->handler(dc, code, instr->flags); - if (!TCGV_IS_UNUSED_I32(dc->zero)) { + if (dc->zero) { tcg_temp_free(dc->zero); } diff --git a/target/ppc/translate.c b/target/ppc/translate.c index 998fbed848..4f7b66f990 100644 --- a/target/ppc/translate.c +++ b/target/ppc/translate.c @@ -3495,7 +3495,7 @@ static inline void gen_bcond(DisasContext *ctx, int type) else tcg_gen_mov_tl(target, cpu_lr); } else { - TCGV_UNUSED(target); + target = NULL; } if (LK(ctx->opcode)) gen_setlr(ctx, ctx->nip); diff --git a/target/s390x/translate.c b/target/s390x/translate.c index 85d0a6c3af..03eca099f6 100644 --- a/target/s390x/translate.c +++ b/target/s390x/translate.c @@ -440,11 +440,9 @@ static void set_cc_static(DisasContext *s) /* calculates cc into cc_op */ static void gen_op_calc_cc(DisasContext *s) { - TCGv_i32 local_cc_op; - TCGv_i64 dummy; + TCGv_i32 local_cc_op = NULL; + TCGv_i64 dummy = NULL; - TCGV_UNUSED_I32(local_cc_op); - TCGV_UNUSED_I64(dummy); switch (s->cc_op) { default: dummy = tcg_const_i64(0); @@ -534,10 +532,10 @@ static void gen_op_calc_cc(DisasContext *s) tcg_abort(); } - if (!TCGV_IS_UNUSED_I32(local_cc_op)) { + if (local_cc_op) { tcg_temp_free_i32(local_cc_op); } - if (!TCGV_IS_UNUSED_I64(dummy)) { + if (dummy) { tcg_temp_free_i64(dummy); } @@ -1195,7 +1193,7 @@ static ExitStatus help_branch(DisasContext *s, DisasCompare *c, goto egress; } } else { - if (TCGV_IS_UNUSED_I64(cdest)) { + if (!cdest) { /* E.g. bcr %r0 -> no branch. */ ret = NO_EXIT; goto egress; @@ -1415,7 +1413,7 @@ static ExitStatus op_andi(DisasContext *s, DisasOps *o) static ExitStatus op_bas(DisasContext *s, DisasOps *o) { tcg_gen_movi_i64(o->out, pc_to_link_info(s, s->next_pc)); - if (!TCGV_IS_UNUSED_I64(o->in2)) { + if (o->in2) { tcg_gen_mov_i64(psw_addr, o->in2); per_branch(s, false); return EXIT_PC_UPDATED; @@ -2997,7 +2995,7 @@ static ExitStatus op_mov2(DisasContext *s, DisasOps *o) { o->out = o->in2; o->g_out = o->g_in2; - TCGV_UNUSED_I64(o->in2); + o->in2 = NULL; o->g_in2 = false; return NO_EXIT; } @@ -3009,7 +3007,7 @@ static ExitStatus op_mov2e(DisasContext *s, DisasOps *o) o->out = o->in2; o->g_out = o->g_in2; - TCGV_UNUSED_I64(o->in2); + o->in2 = NULL; o->g_in2 = false; switch (s->tb->flags & FLAG_MASK_ASC) { @@ -3043,8 +3041,8 @@ static ExitStatus op_movx(DisasContext *s, DisasOps *o) o->out2 = o->in2; o->g_out = o->g_in1; o->g_out2 = o->g_in2; - TCGV_UNUSED_I64(o->in1); - TCGV_UNUSED_I64(o->in2); + o->in1 = NULL; + o->in2 = NULL; o->g_in1 = o->g_in2 = false; return NO_EXIT; } @@ -5801,11 +5799,11 @@ static ExitStatus translate_one(CPUS390XState *env, DisasContext *s) s->insn = insn; s->fields = &f; o.g_out = o.g_out2 = o.g_in1 = o.g_in2 = false; - TCGV_UNUSED_I64(o.out); - TCGV_UNUSED_I64(o.out2); - TCGV_UNUSED_I64(o.in1); - TCGV_UNUSED_I64(o.in2); - TCGV_UNUSED_I64(o.addr1); + o.out = NULL; + o.out2 = NULL; + o.in1 = NULL; + o.in2 = NULL; + o.addr1 = NULL; /* Implement the instruction. */ if (insn->help_in1) { @@ -5828,19 +5826,19 @@ static ExitStatus translate_one(CPUS390XState *env, DisasContext *s) } /* Free any temporaries created by the helpers. */ - if (!TCGV_IS_UNUSED_I64(o.out) && !o.g_out) { + if (o.out && !o.g_out) { tcg_temp_free_i64(o.out); } - if (!TCGV_IS_UNUSED_I64(o.out2) && !o.g_out2) { + if (o.out2 && !o.g_out2) { tcg_temp_free_i64(o.out2); } - if (!TCGV_IS_UNUSED_I64(o.in1) && !o.g_in1) { + if (o.in1 && !o.g_in1) { tcg_temp_free_i64(o.in1); } - if (!TCGV_IS_UNUSED_I64(o.in2) && !o.g_in2) { + if (o.in2 && !o.g_in2) { tcg_temp_free_i64(o.in2); } - if (!TCGV_IS_UNUSED_I64(o.addr1)) { + if (o.addr1) { tcg_temp_free_i64(o.addr1); } diff --git a/target/sh4/translate.c b/target/sh4/translate.c index 703020fe87..48ea0fe7e0 100644 --- a/target/sh4/translate.c +++ b/target/sh4/translate.c @@ -1901,7 +1901,7 @@ static int decode_gusa(DisasContext *ctx, CPUSH4State *env, int *pmax_insns) op_dst = op_src = op_opc = -1; mt_dst = -1; st_src = st_mop = -1; - TCGV_UNUSED(op_arg); + op_arg = NULL; i = 0; #define NEXT_INSN \ diff --git a/target/sparc/translate.c b/target/sparc/translate.c index 849a02aebd..71e0853e43 100644 --- a/target/sparc/translate.c +++ b/target/sparc/translate.c @@ -5922,7 +5922,7 @@ void sparc_tcg_init(void) *rtl[i].ptr = tcg_global_mem_new(cpu_env, rtl[i].off, rtl[i].name); } - TCGV_UNUSED(cpu_regs[0]); + cpu_regs[0] = NULL; for (i = 1; i < 8; ++i) { cpu_regs[i] = tcg_global_mem_new(cpu_env, offsetof(CPUSPARCState, gregs[i]), diff --git a/target/tilegx/translate.c b/target/tilegx/translate.c index d55549dabc..d63bf5bba3 100644 --- a/target/tilegx/translate.c +++ b/target/tilegx/translate.c @@ -143,7 +143,7 @@ static bool check_gr(DisasContext *dc, uint8_t reg) static TCGv load_zero(DisasContext *dc) { - if (TCGV_IS_UNUSED_I64(dc->zero)) { + if (!dc->zero) { dc->zero = tcg_const_i64(0); } return dc->zero; @@ -2324,7 +2324,7 @@ static void translate_one_bundle(DisasContext *dc, uint64_t bundle) for (i = 0; i < ARRAY_SIZE(dc->wb); i++) { DisasContextTemp *wb = &dc->wb[i]; wb->reg = TILEGX_R_NOREG; - TCGV_UNUSED_I64(wb->val); + wb->val = NULL; } dc->num_wb = 0; @@ -2384,9 +2384,9 @@ void gen_intermediate_code(CPUState *cs, struct TranslationBlock *tb) dc->exit_tb = false; dc->atomic_excp = TILEGX_EXCP_NONE; dc->jmp.cond = TCG_COND_NEVER; - TCGV_UNUSED_I64(dc->jmp.dest); - TCGV_UNUSED_I64(dc->jmp.val1); - TCGV_UNUSED_I64(dc->zero); + dc->jmp.dest = NULL; + dc->jmp.val1 = NULL; + dc->zero = NULL; if (qemu_loglevel_mask(CPU_LOG_TB_IN_ASM)) { qemu_log_lock(); diff --git a/target/unicore32/translate.c b/target/unicore32/translate.c index 384aa86027..5b51f2166d 100644 --- a/target/unicore32/translate.c +++ b/target/unicore32/translate.c @@ -1230,7 +1230,7 @@ static void do_datap(CPUUniCore32State *env, DisasContext *s, uint32_t insn) if (UCOP_OPCODES != 0x0f && UCOP_OPCODES != 0x0d) { tmp = load_reg(s, UCOP_REG_N); } else { - TCGV_UNUSED(tmp); + tmp = NULL; } switch (UCOP_OPCODES) { @@ -1652,7 +1652,7 @@ static void do_ldst_m(CPUUniCore32State *env, DisasContext *s, uint32_t insn) /* compute total size */ loaded_base = 0; - TCGV_UNUSED(loaded_var); + loaded_var = NULL; n = 0; for (i = 0; i < 6; i++) { if (UCOP_SET(i)) { diff --git a/tcg/tcg.c b/tcg/tcg.c index c22f1c4441..68bcd2267b 100644 --- a/tcg/tcg.c +++ b/tcg/tcg.c @@ -1358,8 +1358,8 @@ void tcg_gen_callN(void *func, TCGTemp *ret, int nargs, TCGTemp **args) TCGv_i64 retl, reth; TCGTemp *split_args[MAX_OPC_PARAM]; - TCGV_UNUSED_I64(retl); - TCGV_UNUSED_I64(reth); + retl = NULL; + reth = NULL; if (sizemask != 0) { for (i = real_args = 0; i < nargs; ++i) { int is_64bit = sizemask & (1 << (i+1)*2); From patchwork Tue Nov 21 21:25:10 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 119405 Delivered-To: patch@linaro.org Received: by 10.140.22.164 with SMTP id 33csp5813433qgn; Tue, 21 Nov 2017 13:30:39 -0800 (PST) X-Google-Smtp-Source: AGs4zMbWjg/EeQJScWQtIASB/6ypuIzLerAkXwBjZgqWPvNKkvOccB1jreARlWyrkfLxMIyYB0ex X-Received: by 10.37.204.142 with SMTP id l136mr11077563ybf.158.1511299839371; Tue, 21 Nov 2017 13:30:39 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1511299839; cv=none; d=google.com; s=arc-20160816; b=N0w6rPcZ4vyqjsy7Fr4pwWnod+zakEh+AayCyDvdfM1VVRE9UitQ6BL2Djbn4BwRvc cn9qlG7E1Dz9/5ZcX3dSPZPi3WXOrRg38Rw54z5LlrNpOc+dtScflWZ1i8vyP0sgjPe8 3H9IqQo3A1IL/KM9sPitydKJ35DCCvU6pgJZqlsvgiCJnYTWl6jwtianTeDpD8A8Tgi9 LNUIM3SU6liaM+p8yulS+QPYvtFABEA/c9bBoi7nwVpEZRoG8JcIwvBLwkZHlb3uVu2S 39gW1wTLX31K6jZ5AJgVRS66hnhrQ/ccu3oPrxBn+tv+dVTeskTVXvm96wMmel2PfYB+ dr5A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:subject:references:in-reply-to :message-id:date:to:from:dkim-signature:arc-authentication-results; bh=oZeejt/o8hgAegpCo66lb3IU3GzzHtQbgQEvFUDrmEM=; b=0mZxwckEAHyqsWkEYAW2oYZXkTOy9g7hDlV7xiOTt/ZyanAC52nfciNAFRWcdxKfGz StDUm/T2n26wr8UA1SsMYnMT3SWufYpoY3r2L26uCmpI3s/lBXRoD2GM+X9FK6YDGjIO K1UNwhvXV1OMBRlPUS2wg4NO7im6LaeQVcdT9XN6e5wSKtXF5xJ2c7UrsucMi5Xv1RaX Y515UTutJkRbA0CElOLaBB92Y5xxHU+B/PAak1YiK9YevDRLDV+xTq89H9o56hYLfR6c adfia8VsUjV/Jo09EA/OOHNIlZgLbB8M4b2F3XLBIK+ULUueH9NvzNdSKPrIxf4Fh2oZ 6Fbg== ARC-Authentication-Results: i=1; mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=kp1Gj2wY; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) smtp.mailfrom=qemu-devel-bounces+patch=linaro.org@nongnu.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [2001:4830:134:3::11]) by mx.google.com with ESMTPS id t12si1699170ywe.68.2017.11.21.13.30.39 for (version=TLS1 cipher=AES128-SHA bits=128/128); Tue, 21 Nov 2017 13:30:39 -0800 (PST) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) client-ip=2001:4830:134:3::11; Authentication-Results: mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=kp1Gj2wY; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) smtp.mailfrom=qemu-devel-bounces+patch=linaro.org@nongnu.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1]:36586 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1eHG7q-0006QG-NP for patch@linaro.org; Tue, 21 Nov 2017 16:30:38 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:53904) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1eHG51-0004Gv-Up for qemu-devel@nongnu.org; Tue, 21 Nov 2017 16:27:46 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1eHG4z-0007cp-2d for qemu-devel@nongnu.org; Tue, 21 Nov 2017 16:27:43 -0500 Received: from mail-wm0-x22f.google.com ([2a00:1450:400c:c09::22f]:42836) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1eHG4y-0007cM-O7 for qemu-devel@nongnu.org; Tue, 21 Nov 2017 16:27:41 -0500 Received: by mail-wm0-x22f.google.com with SMTP id l188so4522627wma.1 for ; Tue, 21 Nov 2017 13:27:40 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:subject:date:message-id:in-reply-to:references; bh=oZeejt/o8hgAegpCo66lb3IU3GzzHtQbgQEvFUDrmEM=; b=kp1Gj2wYh1MMm4ctHkT/TKFpZjc1TZECLV4U9cS3wp1wCzGtCk6rYlc0wU75mTKM1U 4BAEl6jbrhMgM0mHPzk9Gvzjy4FZ6n/w/hmT9atBhI98fUfYqvE1/wHaq6PSDjPa9Y6P B83TT7rxoXePKDhbD4z30U0INUn7Tx7ozvb/U= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references; bh=oZeejt/o8hgAegpCo66lb3IU3GzzHtQbgQEvFUDrmEM=; b=QJVIX1rvd5209uib297UN7XZMM/RQfF7MP+CvGDagnVzHK8hPx/7LJz9P8ZxlUwxUC A7av7PhD6SJqRgVHIMBi0ow7UHi/JYIf8mnvnJEAcOdSjPs3g4O8R+523XpTRoAAf+WW yySaBOSer/Xj7Davlzs8XQaRv94OOzDFTRgX+T0yBlUJUBjF1DqM5EQ9as60Nqcvl/AD 7FgoYIyIhMAclNuEoxZBczczrFs0qDDeqm/heGMGMY25YUaA05+FbLUTXj4ctYVa14tS PDtSelbrsRHtrk/+XBLy/5k1AVR9mRsfdrrLm/qpShcCuy7Vjb7M0tqNcocU1uT5pQY7 y+mA== X-Gm-Message-State: AJaThX4+Sy7pn4+IBi8C4PF97JHEV1Ds6uh2iXJW5B0ZnZ5iIQN1rqgU H22ajIdUwAISIGddsV7gQwBk+x+8zKI= X-Received: by 10.28.235.18 with SMTP id j18mr2415271wmh.50.1511299659099; Tue, 21 Nov 2017 13:27:39 -0800 (PST) Received: from cloudburst.twiddle.net (70.red-37-158-60.dynamicip.rima-tde.net. [37.158.60.70]) by smtp.gmail.com with ESMTPSA id e124sm706517wmg.34.2017.11.21.13.27.38 for (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Tue, 21 Nov 2017 13:27:38 -0800 (PST) From: Richard Henderson To: qemu-devel@nongnu.org Date: Tue, 21 Nov 2017 22:25:10 +0100 Message-Id: <20171121212534.5177-3-richard.henderson@linaro.org> X-Mailer: git-send-email 2.13.6 In-Reply-To: <20171121212534.5177-1-richard.henderson@linaro.org> References: <20171121212534.5177-1-richard.henderson@linaro.org> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2a00:1450:400c:c09::22f Subject: [Qemu-devel] [PATCH v6 02/26] tcg: Dynamically allocate TCGOps X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: "Qemu-devel" With no fixed array allocation, we can't overflow a buffer. This will be important as optimizations related to host vectors may expand the number of ops used. Use QTAILQ to link the ops together. Signed-off-by: Richard Henderson --- include/exec/gen-icount.h | 9 ++-- include/qemu/queue.h | 5 ++ target/arm/translate.h | 10 ++-- tcg/tcg.h | 35 +++++------- target/arm/translate-a64.c | 2 +- target/arm/translate.c | 2 +- target/cris/translate.c | 2 - target/lm32/translate.c | 2 - target/microblaze/translate.c | 4 -- tcg/optimize.c | 16 ++---- tcg/tcg-op.c | 24 --------- tcg/tcg.c | 123 ++++++++++++++++-------------------------- 12 files changed, 77 insertions(+), 157 deletions(-) -- 2.13.6 diff --git a/include/exec/gen-icount.h b/include/exec/gen-icount.h index 049bba86e9..54aaa61d65 100644 --- a/include/exec/gen-icount.h +++ b/include/exec/gen-icount.h @@ -5,7 +5,7 @@ /* Helpers for instruction counting code generation. */ -static int icount_start_insn_idx; +static TCGOp *icount_start_insn; static inline void gen_tb_start(TranslationBlock *tb) { @@ -26,8 +26,8 @@ static inline void gen_tb_start(TranslationBlock *tb) /* We emit a movi with a dummy immediate argument. Keep the insn index * of the movi so that we later (when we know the actual insn count) * can update the immediate argument with the actual insn count. */ - icount_start_insn_idx = tcg_op_buf_count(); tcg_gen_movi_i32(imm, 0xdeadbeef); + icount_start_insn = tcg_last_op(); tcg_gen_sub_i32(count, count, imm); tcg_temp_free_i32(imm); @@ -48,14 +48,11 @@ static inline void gen_tb_end(TranslationBlock *tb, int num_insns) if (tb_cflags(tb) & CF_USE_ICOUNT) { /* Update the num_insn immediate parameter now that we know * the actual insn count. */ - tcg_set_insn_param(icount_start_insn_idx, 1, num_insns); + tcg_set_insn_param(icount_start_insn, 1, num_insns); } gen_set_label(tcg_ctx->exitreq_label); tcg_gen_exit_tb((uintptr_t)tb + TB_EXIT_REQUESTED); - - /* Terminate the linked list. */ - tcg_ctx->gen_op_buf[tcg_ctx->gen_op_buf[0].prev].next = 0; } static inline void gen_io_start(void) diff --git a/include/qemu/queue.h b/include/qemu/queue.h index 35292c3155..aa270d2b38 100644 --- a/include/qemu/queue.h +++ b/include/qemu/queue.h @@ -425,6 +425,11 @@ struct { \ (var); \ (var) = (*(((struct headname *)((var)->field.tqe_prev))->tqh_last))) +#define QTAILQ_FOREACH_REVERSE_SAFE(var, head, headname, field, prev_var) \ + for ((var) = (*(((struct headname *)((head)->tqh_last))->tqh_last)); \ + (var) && ((prev_var) = (*(((struct headname *)((var)->field.tqe_prev))->tqh_last)), 1); \ + (var) = (prev_var)) + /* * Tail queue access methods. */ diff --git a/target/arm/translate.h b/target/arm/translate.h index 410ba79c0d..cd7313ace7 100644 --- a/target/arm/translate.h +++ b/target/arm/translate.h @@ -66,8 +66,8 @@ typedef struct DisasContext { bool ss_same_el; /* Bottom two bits of XScale c15_cpar coprocessor access control reg */ int c15_cpar; - /* TCG op index of the current insn_start. */ - int insn_start_idx; + /* TCG op of the current insn_start. */ + TCGOp *insn_start; #define TMP_A64_MAX 16 int tmp_a64_count; TCGv_i64 tmp_a64[TMP_A64_MAX]; @@ -117,9 +117,9 @@ static void disas_set_insn_syndrome(DisasContext *s, uint32_t syn) syn >>= ARM_INSN_START_WORD2_SHIFT; /* We check and clear insn_start_idx to catch multiple updates. */ - assert(s->insn_start_idx != 0); - tcg_set_insn_param(s->insn_start_idx, 2, syn); - s->insn_start_idx = 0; + assert(s->insn_start != NULL); + tcg_set_insn_param(s->insn_start, 2, syn); + s->insn_start = NULL; } /* is_jmp field values */ diff --git a/tcg/tcg.h b/tcg/tcg.h index c21194c858..a577447846 100644 --- a/tcg/tcg.h +++ b/tcg/tcg.h @@ -29,6 +29,7 @@ #include "cpu.h" #include "exec/tb-context.h" #include "qemu/bitops.h" +#include "qemu/queue.h" #include "tcg-mo.h" #include "tcg-target.h" @@ -48,8 +49,6 @@ * and up to 4 + N parameters on 64-bit archs * (N = number of input arguments + output arguments). */ #define MAX_OPC_PARAM (4 + (MAX_OPC_PARAM_PER_ARG * MAX_OPC_PARAM_ARGS)) -#define OPC_BUF_SIZE 640 -#define OPC_MAX_SIZE (OPC_BUF_SIZE - MAX_OP_PER_INSTR) #define CPU_TEMP_BUF_NLONGS 128 @@ -572,23 +571,18 @@ typedef struct TCGOp { unsigned callo : 2; /* 14 */ unsigned : 2; /* 16 */ - /* Index of the prev/next op, or 0 for the end of the list. */ - unsigned prev : 16; /* 32 */ - unsigned next : 16; /* 48 */ - /* Lifetime data of the operands. */ - unsigned life : 16; /* 64 */ + unsigned life : 16; /* 32 */ + + /* Next and previous opcodes. */ + QTAILQ_ENTRY(TCGOp) link; /* Arguments for the opcode. */ TCGArg args[MAX_OPC_PARAM]; } TCGOp; -/* Make sure that we don't expand the structure without noticing. */ -QEMU_BUILD_BUG_ON(sizeof(TCGOp) != 8 + sizeof(TCGArg) * MAX_OPC_PARAM); - /* Make sure operands fit in the bitfields above. */ QEMU_BUILD_BUG_ON(NB_OPS > (1 << 8)); -QEMU_BUILD_BUG_ON(OPC_BUF_SIZE > (1 << 16)); typedef struct TCGProfile { int64_t tb_count1; @@ -642,8 +636,6 @@ struct TCGContext { int goto_tb_issue_mask; #endif - int gen_next_op_idx; - /* Code generation. Note that we specifically do not use tcg_insn_unit here, because there's too much arithmetic throughout that relies on addition and subtraction working on bytes. Rely on the GCC @@ -674,12 +666,12 @@ struct TCGContext { TCGTempSet free_temps[TCG_TYPE_COUNT * 2]; TCGTemp temps[TCG_MAX_TEMPS]; /* globals first, temps after */ + QTAILQ_HEAD(TCGOpHead, TCGOp) ops, free_ops; + /* Tells which temporary holds a given register. It does not take into account fixed registers */ TCGTemp *reg_to_temp[TCG_TARGET_NB_REGS]; - TCGOp gen_op_buf[OPC_BUF_SIZE]; - uint16_t gen_insn_end_off[TCG_MAX_INSNS]; target_ulong gen_insn_data[TCG_MAX_INSNS][TARGET_INSN_START_WORDS]; }; @@ -769,21 +761,21 @@ static inline TCGv_i32 TCGV_HIGH(TCGv_i64 t) } #endif -static inline void tcg_set_insn_param(int op_idx, int arg, TCGArg v) +static inline void tcg_set_insn_param(TCGOp *op, int arg, TCGArg v) { - tcg_ctx->gen_op_buf[op_idx].args[arg] = v; + op->args[arg] = v; } -/* The number of opcodes emitted so far. */ -static inline int tcg_op_buf_count(void) +/* The last op that was emitted. */ +static inline TCGOp *tcg_last_op(void) { - return tcg_ctx->gen_next_op_idx; + return QTAILQ_LAST(&tcg_ctx->ops, TCGOpHead); } /* Test for whether to terminate the TB for using too many opcodes. */ static inline bool tcg_op_buf_full(void) { - return tcg_op_buf_count() >= OPC_MAX_SIZE; + return false; } /* pool based memory allocation */ @@ -967,6 +959,7 @@ bool tcg_op_supported(TCGOpcode op); void tcg_gen_callN(void *func, TCGTemp *ret, int nargs, TCGTemp **args); +TCGOp *tcg_emit_op(TCGOpcode opc); void tcg_op_remove(TCGContext *s, TCGOp *op); TCGOp *tcg_op_insert_before(TCGContext *s, TCGOp *op, TCGOpcode opc, int narg); TCGOp *tcg_op_insert_after(TCGContext *s, TCGOp *op, TCGOpcode opc, int narg); diff --git a/target/arm/translate-a64.c b/target/arm/translate-a64.c index 460bab5987..ba94f7d045 100644 --- a/target/arm/translate-a64.c +++ b/target/arm/translate-a64.c @@ -11290,8 +11290,8 @@ static void aarch64_tr_insn_start(DisasContextBase *dcbase, CPUState *cpu) { DisasContext *dc = container_of(dcbase, DisasContext, base); - dc->insn_start_idx = tcg_op_buf_count(); tcg_gen_insn_start(dc->pc, 0, 0); + dc->insn_start = tcg_last_op(); } static bool aarch64_tr_breakpoint_check(DisasContextBase *dcbase, CPUState *cpu, diff --git a/target/arm/translate.c b/target/arm/translate.c index 0b74ce7dec..20c3246912 100644 --- a/target/arm/translate.c +++ b/target/arm/translate.c @@ -12065,10 +12065,10 @@ static void arm_tr_insn_start(DisasContextBase *dcbase, CPUState *cpu) { DisasContext *dc = container_of(dcbase, DisasContext, base); - dc->insn_start_idx = tcg_op_buf_count(); tcg_gen_insn_start(dc->pc, (dc->condexec_cond << 4) | (dc->condexec_mask >> 1), 0); + dc->insn_start = tcg_last_op(); } static bool arm_tr_breakpoint_check(DisasContextBase *dcbase, CPUState *cpu, diff --git a/target/cris/translate.c b/target/cris/translate.c index 74822ed31f..f51a731db9 100644 --- a/target/cris/translate.c +++ b/target/cris/translate.c @@ -3297,8 +3297,6 @@ void gen_intermediate_code(CPUState *cs, struct TranslationBlock *tb) qemu_log("--------------\n"); qemu_log("IN: %s\n", lookup_symbol(pc_start)); log_target_disas(cs, pc_start, dc->pc - pc_start); - qemu_log("\nisize=%d osize=%d\n", - dc->pc - pc_start, tcg_op_buf_count()); qemu_log_unlock(); } #endif diff --git a/target/lm32/translate.c b/target/lm32/translate.c index b8b2b13e36..2e1c5e6d01 100644 --- a/target/lm32/translate.c +++ b/target/lm32/translate.c @@ -1156,8 +1156,6 @@ void gen_intermediate_code(CPUState *cs, struct TranslationBlock *tb) qemu_log_lock(); qemu_log("\n"); log_target_disas(cs, pc_start, dc->pc - pc_start); - qemu_log("\nisize=%d osize=%d\n", - dc->pc - pc_start, tcg_op_buf_count()); qemu_log_unlock(); } #endif diff --git a/target/microblaze/translate.c b/target/microblaze/translate.c index e7b5597c46..7628b0e25b 100644 --- a/target/microblaze/translate.c +++ b/target/microblaze/translate.c @@ -1808,11 +1808,7 @@ void gen_intermediate_code(CPUState *cs, struct TranslationBlock *tb) && qemu_log_in_addr_range(pc_start)) { qemu_log_lock(); qemu_log("--------------\n"); -#if DISAS_GNU log_target_disas(cs, pc_start, dc->pc - pc_start); -#endif - qemu_log("\nisize=%d osize=%d\n", - dc->pc - pc_start, tcg_op_buf_count()); qemu_log_unlock(); } #endif diff --git a/tcg/optimize.c b/tcg/optimize.c index 438321c6cc..e495680e95 100644 --- a/tcg/optimize.c +++ b/tcg/optimize.c @@ -602,8 +602,8 @@ static bool swap_commutative2(TCGArg *p1, TCGArg *p2) /* Propagate constants and copies, fold constant expressions. */ void tcg_optimize(TCGContext *s) { - int oi, oi_next, nb_temps, nb_globals; - TCGOp *prev_mb = NULL; + int nb_temps, nb_globals; + TCGOp *op, *op_next, *prev_mb = NULL; struct tcg_temp_info *infos; TCGTempSet temps_used; @@ -617,17 +617,13 @@ void tcg_optimize(TCGContext *s) bitmap_zero(temps_used.l, nb_temps); infos = tcg_malloc(sizeof(struct tcg_temp_info) * nb_temps); - for (oi = s->gen_op_buf[0].next; oi != 0; oi = oi_next) { + QTAILQ_FOREACH_SAFE(op, &s->ops, link, op_next) { tcg_target_ulong mask, partmask, affected; int nb_oargs, nb_iargs, i; TCGArg tmp; - - TCGOp * const op = &s->gen_op_buf[oi]; TCGOpcode opc = op->opc; const TCGOpDef *def = &tcg_op_defs[opc]; - oi_next = op->next; - /* Count the arguments, and initialize the temps that are going to be used */ if (opc == INDEX_op_call) { @@ -1261,9 +1257,6 @@ void tcg_optimize(TCGContext *s) rh = op->args[1]; tcg_opt_gen_movi(s, op, rl, (int32_t)a); tcg_opt_gen_movi(s, op2, rh, (int32_t)(a >> 32)); - - /* We've done all we need to do with the movi. Skip it. */ - oi_next = op2->next; break; } goto do_default; @@ -1280,9 +1273,6 @@ void tcg_optimize(TCGContext *s) rh = op->args[1]; tcg_opt_gen_movi(s, op, rl, (int32_t)r); tcg_opt_gen_movi(s, op2, rh, (int32_t)(r >> 32)); - - /* We've done all we need to do with the movi. Skip it. */ - oi_next = op2->next; break; } goto do_default; diff --git a/tcg/tcg-op.c b/tcg/tcg-op.c index 3cad30b1f2..0c509bfe46 100644 --- a/tcg/tcg-op.c +++ b/tcg/tcg-op.c @@ -42,30 +42,6 @@ extern TCGv_i32 TCGV_HIGH_link_error(TCGv_i64); #define TCGV_HIGH TCGV_HIGH_link_error #endif -/* Note that this is optimized for sequential allocation during translate. - Up to and including filling in the forward link immediately. We'll do - proper termination of the end of the list after we finish translation. */ - -static inline TCGOp *tcg_emit_op(TCGOpcode opc) -{ - TCGContext *ctx = tcg_ctx; - int oi = ctx->gen_next_op_idx; - int ni = oi + 1; - int pi = oi - 1; - TCGOp *op = &ctx->gen_op_buf[oi]; - - tcg_debug_assert(oi < OPC_BUF_SIZE); - ctx->gen_op_buf[0].prev = oi; - ctx->gen_next_op_idx = ni; - - memset(op, 0, offsetof(TCGOp, args)); - op->opc = opc; - op->prev = pi; - op->next = ni; - - return op; -} - void tcg_gen_op1(TCGOpcode opc, TCGArg a1) { TCGOp *op = tcg_emit_op(opc); diff --git a/tcg/tcg.c b/tcg/tcg.c index 68bcd2267b..f26949a900 100644 --- a/tcg/tcg.c +++ b/tcg/tcg.c @@ -862,9 +862,8 @@ void tcg_func_start(TCGContext *s) s->goto_tb_issue_mask = 0; #endif - s->gen_op_buf[0].next = 1; - s->gen_op_buf[0].prev = 0; - s->gen_next_op_idx = 1; + QTAILQ_INIT(&s->ops); + QTAILQ_INIT(&s->free_ops); } static inline TCGTemp *tcg_temp_alloc(TCGContext *s) @@ -1339,7 +1338,6 @@ bool tcg_op_supported(TCGOpcode op) and endian swap in tcg_reg_alloc_call(). */ void tcg_gen_callN(void *func, TCGTemp *ret, int nargs, TCGTemp **args) { - TCGContext *s = tcg_ctx; int i, real_args, nb_rets, pi; unsigned sizemask, flags; TCGHelperInfo *info; @@ -1395,17 +1393,7 @@ void tcg_gen_callN(void *func, TCGTemp *ret, int nargs, TCGTemp **args) } #endif /* TCG_TARGET_EXTEND_ARGS */ - i = s->gen_next_op_idx; - tcg_debug_assert(i < OPC_BUF_SIZE); - s->gen_op_buf[0].prev = i; - s->gen_next_op_idx = i + 1; - op = &s->gen_op_buf[i]; - - /* Set links for sequential allocation during translation. */ - memset(op, 0, offsetof(TCGOp, args)); - op->opc = INDEX_op_call; - op->prev = i - 1; - op->next = i + 1; + op = tcg_emit_op(INDEX_op_call); pi = 0; if (ret != NULL) { @@ -1622,20 +1610,18 @@ void tcg_dump_ops(TCGContext *s) { char buf[128]; TCGOp *op; - int oi; - for (oi = s->gen_op_buf[0].next; oi != 0; oi = op->next) { + QTAILQ_FOREACH(op, &s->ops, link) { int i, k, nb_oargs, nb_iargs, nb_cargs; const TCGOpDef *def; TCGOpcode c; int col = 0; - op = &s->gen_op_buf[oi]; c = op->opc; def = &tcg_op_defs[c]; if (c == INDEX_op_insn_start) { - col += qemu_log("%s ----", oi != s->gen_op_buf[0].next ? "\n" : ""); + col += qemu_log("\n ----"); for (i = 0; i < TARGET_INSN_START_WORDS; ++i) { target_ulong a; @@ -1898,65 +1884,51 @@ static void process_op_defs(TCGContext *s) void tcg_op_remove(TCGContext *s, TCGOp *op) { - int next = op->next; - int prev = op->prev; - - /* We should never attempt to remove the list terminator. */ - tcg_debug_assert(op != &s->gen_op_buf[0]); - - s->gen_op_buf[next].prev = prev; - s->gen_op_buf[prev].next = next; - - memset(op, 0, sizeof(*op)); + QTAILQ_REMOVE(&s->ops, op, link); + QTAILQ_INSERT_TAIL(&s->free_ops, op, link); #ifdef CONFIG_PROFILER atomic_set(&s->prof.del_op_count, s->prof.del_op_count + 1); #endif } -TCGOp *tcg_op_insert_before(TCGContext *s, TCGOp *old_op, - TCGOpcode opc, int nargs) +static TCGOp *tcg_op_alloc(TCGOpcode opc) { - int oi = s->gen_next_op_idx; - int prev = old_op->prev; - int next = old_op - s->gen_op_buf; - TCGOp *new_op; + TCGContext *s = tcg_ctx; + TCGOp *op; - tcg_debug_assert(oi < OPC_BUF_SIZE); - s->gen_next_op_idx = oi + 1; + if (likely(QTAILQ_EMPTY(&s->free_ops))) { + op = tcg_malloc(sizeof(TCGOp)); + } else { + op = QTAILQ_FIRST(&s->free_ops); + QTAILQ_REMOVE(&s->free_ops, op, link); + } + memset(op, 0, offsetof(TCGOp, link)); + op->opc = opc; - new_op = &s->gen_op_buf[oi]; - *new_op = (TCGOp){ - .opc = opc, - .prev = prev, - .next = next - }; - s->gen_op_buf[prev].next = oi; - old_op->prev = oi; + return op; +} + +TCGOp *tcg_emit_op(TCGOpcode opc) +{ + TCGOp *op = tcg_op_alloc(opc); + QTAILQ_INSERT_TAIL(&tcg_ctx->ops, op, link); + return op; +} +TCGOp *tcg_op_insert_before(TCGContext *s, TCGOp *old_op, + TCGOpcode opc, int nargs) +{ + TCGOp *new_op = tcg_op_alloc(opc); + QTAILQ_INSERT_BEFORE(old_op, new_op, link); return new_op; } TCGOp *tcg_op_insert_after(TCGContext *s, TCGOp *old_op, TCGOpcode opc, int nargs) { - int oi = s->gen_next_op_idx; - int prev = old_op - s->gen_op_buf; - int next = old_op->next; - TCGOp *new_op; - - tcg_debug_assert(oi < OPC_BUF_SIZE); - s->gen_next_op_idx = oi + 1; - - new_op = &s->gen_op_buf[oi]; - *new_op = (TCGOp){ - .opc = opc, - .prev = prev, - .next = next - }; - s->gen_op_buf[next].prev = oi; - old_op->next = oi; - + TCGOp *new_op = tcg_op_alloc(opc); + QTAILQ_INSERT_AFTER(&s->ops, old_op, new_op, link); return new_op; } @@ -2006,23 +1978,19 @@ static void tcg_la_bb_end(TCGContext *s) static void liveness_pass_1(TCGContext *s) { int nb_globals = s->nb_globals; - int oi, oi_prev; + TCGOp *op, *op_prev; tcg_la_func_end(s); - for (oi = s->gen_op_buf[0].prev; oi != 0; oi = oi_prev) { + QTAILQ_FOREACH_REVERSE_SAFE(op, &s->ops, TCGOpHead, link, op_prev) { int i, nb_iargs, nb_oargs; TCGOpcode opc_new, opc_new2; bool have_opc_new2; TCGLifeData arg_life = 0; TCGTemp *arg_ts; - - TCGOp * const op = &s->gen_op_buf[oi]; TCGOpcode opc = op->opc; const TCGOpDef *def = &tcg_op_defs[opc]; - oi_prev = op->prev; - switch (opc) { case INDEX_op_call: { @@ -2233,8 +2201,9 @@ static void liveness_pass_1(TCGContext *s) static bool liveness_pass_2(TCGContext *s) { int nb_globals = s->nb_globals; - int nb_temps, i, oi, oi_next; + int nb_temps, i; bool changes = false; + TCGOp *op, *op_next; /* Create a temporary for each indirect global. */ for (i = 0; i < nb_globals; ++i) { @@ -2256,16 +2225,13 @@ static bool liveness_pass_2(TCGContext *s) its->state = TS_DEAD; } - for (oi = s->gen_op_buf[0].next; oi != 0; oi = oi_next) { - TCGOp *op = &s->gen_op_buf[oi]; + QTAILQ_FOREACH_SAFE(op, &s->ops, link, op_next) { TCGOpcode opc = op->opc; const TCGOpDef *def = &tcg_op_defs[opc]; TCGLifeData arg_life = op->life; int nb_iargs, nb_oargs, call_flags; TCGTemp *arg_ts, *dir_ts; - oi_next = op->next; - if (opc == INDEX_op_call) { nb_oargs = op->callo; nb_iargs = op->calli; @@ -3168,13 +3134,16 @@ int tcg_gen_code(TCGContext *s, TranslationBlock *tb) #ifdef CONFIG_PROFILER TCGProfile *prof = &s->prof; #endif - int i, oi, oi_next, num_insns; + int i, num_insns; + TCGOp *op; #ifdef CONFIG_PROFILER { int n; - n = s->gen_op_buf[0].prev + 1; + QTAILQ_FOREACH(op, &s->ops, link) { + n++; + } atomic_set(&prof->op_count, prof->op_count + n); if (n > prof->op_count_max) { atomic_set(&prof->op_count_max, n); @@ -3260,11 +3229,9 @@ int tcg_gen_code(TCGContext *s, TranslationBlock *tb) #endif num_insns = -1; - for (oi = s->gen_op_buf[0].next; oi != 0; oi = oi_next) { - TCGOp * const op = &s->gen_op_buf[oi]; + QTAILQ_FOREACH(op, &s->ops, link) { TCGOpcode opc = op->opc; - oi_next = op->next; #ifdef CONFIG_PROFILER atomic_set(&prof->table_op_count[opc], prof->table_op_count[opc] + 1); #endif From patchwork Tue Nov 21 21:25:11 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 119403 Delivered-To: patch@linaro.org Received: by 10.140.22.164 with SMTP id 33csp5810879qgn; Tue, 21 Nov 2017 13:28:03 -0800 (PST) X-Google-Smtp-Source: AGs4zMYG9WpMSzvZpsZFaiWnXaGzC3fyChrcIgmJMwDXo1pNKxU28Sd09yGhsXn/w4i3cVMVeLzv X-Received: by 10.129.125.3 with SMTP id y3mr12017219ywc.486.1511299683444; Tue, 21 Nov 2017 13:28:03 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1511299683; cv=none; d=google.com; s=arc-20160816; b=MR/GlKDLdBjpj4nW+iAtDpxot6+HRZgLLGdVgJNsw/o5jSj0RUh/F2s7kqbqeZSsie 25qUcCHEl8T4SrBhdTzzynDXzi1KRgS1G0fUpXi6HWsz+fKjbary6HjI8dh6OWAwrp3u J29PVFlnzHq8iKms73RsoFhSJwPInF74+hFe5Ehg2Q3TWAmvMkUg00n2TRujg7Hzghmf qGoRVgtradcOf9tLaFTH0FV31JZY5nSSpDx3tWmY7I/u00OeJTYU6KqmpoJljamOV1me 8zx2alPF8cQ4VRb4unYW3/FfH+s2fVovywkKAVvCGzd/09R34p4iC6SOW4aNj3FU8+PS Mgug== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:subject:references:in-reply-to :message-id:date:to:from:dkim-signature:arc-authentication-results; bh=JgPyalRRpbRQN5BRX+1C71j5J+ztfUwJIUUYRumX/6I=; b=SovvSOqtaO3cOu+IzHtZbiPqc6Iu/4t1MtN3EnFTGtypMXAfkDICP9HsoANNxJimRG mFveuGFs8k1T8vgRY8afe+gEemEYWGRoL95d/95WGpoThapQCsaQmz7XYO7DyXUi1QSu n8/FX71h6BhwmsIDKsiId+Br2NenstjqDCXCFdXydFxjEocmMm2o27+fW1yTMLSYwZxR ajfWo04GDGgCMdlCMSbbq31jTK4AC7FDbvzq5luvbMpISi7kRsnFU061hBmO/Hr3c8Xz LZXnRfBP+/pPhR6u81NEX9dYIqqLBNRsYZIZevzNJ+CCIBCwqdxFRGKdCS+5lhq1tzkc XSbQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=azRl08gT; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) smtp.mailfrom=qemu-devel-bounces+patch=linaro.org@nongnu.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [2001:4830:134:3::11]) by mx.google.com with ESMTPS id j14si2895020ybg.629.2017.11.21.13.28.03 for (version=TLS1 cipher=AES128-SHA bits=128/128); Tue, 21 Nov 2017 13:28:03 -0800 (PST) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) client-ip=2001:4830:134:3::11; Authentication-Results: mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=azRl08gT; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) smtp.mailfrom=qemu-devel-bounces+patch=linaro.org@nongnu.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1]:36575 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1eHG5K-0004Hr-Pp for patch@linaro.org; Tue, 21 Nov 2017 16:28:02 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:53877) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1eHG51-0004Gb-5q for qemu-devel@nongnu.org; Tue, 21 Nov 2017 16:27:44 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1eHG4z-0007d9-KZ for qemu-devel@nongnu.org; Tue, 21 Nov 2017 16:27:43 -0500 Received: from mail-wm0-x241.google.com ([2a00:1450:400c:c09::241]:46812) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1eHG4z-0007ci-EI for qemu-devel@nongnu.org; Tue, 21 Nov 2017 16:27:41 -0500 Received: by mail-wm0-x241.google.com with SMTP id u83so6327078wmb.5 for ; Tue, 21 Nov 2017 13:27:41 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:subject:date:message-id:in-reply-to:references; bh=JgPyalRRpbRQN5BRX+1C71j5J+ztfUwJIUUYRumX/6I=; b=azRl08gTZ7myVx2A3RlmuY6ujX9t8uywlcc3cvea9PBDfCbk4/sFEm5tdHJ7uPCR0F FE6jfdT8olyX0UNbdV0ikadBzsRP1kiX+LwHfAOm23lnQOiIJ/kJElvDWeOvrOdZm0xO UWES3UgoPV94928OwTqmMtXRgqVs+dTrsHH6M= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references; bh=JgPyalRRpbRQN5BRX+1C71j5J+ztfUwJIUUYRumX/6I=; b=lX6wk4VMBlC/ihonHZaWAYOYbVJbS6agKS0frWMLt3Gu3vLYJ4+Pn7Vr7F3E0tdZQ5 xmbRNUZIrpXuxnH81tP7BSImyLFa/mC+zfj8sA/d0OB1A3Dpo+0zBFsep5mvf+Mtq3/9 qLpaaalKvrbEEN2ItKFEdfAuGy+6+sDRoszzAw5f6xOTSSAyzr/a/mF/4HcYFg/PVqEw 6YmHBebM0R7lNFiMvM6sfZIe1MgkyRWVSyIWB1L+4FC0DMtbVzzIxQyVTjVOg2MXcOZq +fVP1qT5NWtpCr/V7BCnzyw/rC2ELgWrB1sqzAk0zfJ/yHY3pIhZmAOUn9P7dhHCeV1G nmbA== X-Gm-Message-State: AJaThX5TYi3B69fGgZefGE3PZ94gOa/N9FC6YftOxTtJo8i9P4kA9wR8 Xdf2XmfTIulYq+kCJsIUaPGaMSu54ts= X-Received: by 10.28.15.149 with SMTP id 143mr2604895wmp.43.1511299660147; Tue, 21 Nov 2017 13:27:40 -0800 (PST) Received: from cloudburst.twiddle.net (70.red-37-158-60.dynamicip.rima-tde.net. [37.158.60.70]) by smtp.gmail.com with ESMTPSA id e124sm706517wmg.34.2017.11.21.13.27.39 for (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Tue, 21 Nov 2017 13:27:39 -0800 (PST) From: Richard Henderson To: qemu-devel@nongnu.org Date: Tue, 21 Nov 2017 22:25:11 +0100 Message-Id: <20171121212534.5177-4-richard.henderson@linaro.org> X-Mailer: git-send-email 2.13.6 In-Reply-To: <20171121212534.5177-1-richard.henderson@linaro.org> References: <20171121212534.5177-1-richard.henderson@linaro.org> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2a00:1450:400c:c09::241 Subject: [Qemu-devel] [PATCH v6 03/26] tcg: Generalize TCGOp parameters X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: "Qemu-devel" We had two fields specific to INDEX_op_call. Rename these and add some macros so that the fields may be reused for other opcodes. Signed-off-by: Richard Henderson --- tcg/tcg.h | 10 ++++++---- tcg/optimize.c | 4 ++-- tcg/tcg.c | 22 +++++++++++----------- 3 files changed, 19 insertions(+), 17 deletions(-) -- 2.13.6 diff --git a/tcg/tcg.h b/tcg/tcg.h index a577447846..f25efa9795 100644 --- a/tcg/tcg.h +++ b/tcg/tcg.h @@ -566,10 +566,9 @@ typedef uint16_t TCGLifeData; typedef struct TCGOp { TCGOpcode opc : 8; /* 8 */ - /* The number of out and in parameter for a call. */ - unsigned calli : 4; /* 12 */ - unsigned callo : 2; /* 14 */ - unsigned : 2; /* 16 */ + /* Parameters for this opcode. See below. */ + unsigned param1 : 4; /* 12 */ + unsigned param2 : 4; /* 16 */ /* Lifetime data of the operands. */ unsigned life : 16; /* 32 */ @@ -581,6 +580,9 @@ typedef struct TCGOp { TCGArg args[MAX_OPC_PARAM]; } TCGOp; +#define TCGOP_CALLI(X) (X)->param1 +#define TCGOP_CALLO(X) (X)->param2 + /* Make sure operands fit in the bitfields above. */ QEMU_BUILD_BUG_ON(NB_OPS > (1 << 8)); diff --git a/tcg/optimize.c b/tcg/optimize.c index e495680e95..2cbbeefd53 100644 --- a/tcg/optimize.c +++ b/tcg/optimize.c @@ -627,8 +627,8 @@ void tcg_optimize(TCGContext *s) /* Count the arguments, and initialize the temps that are going to be used */ if (opc == INDEX_op_call) { - nb_oargs = op->callo; - nb_iargs = op->calli; + nb_oargs = TCGOP_CALLO(op); + nb_iargs = TCGOP_CALLI(op); for (i = 0; i < nb_oargs + nb_iargs; i++) { TCGTemp *ts = arg_temp(op->args[i]); if (ts) { diff --git a/tcg/tcg.c b/tcg/tcg.c index f26949a900..93caa0be93 100644 --- a/tcg/tcg.c +++ b/tcg/tcg.c @@ -1430,7 +1430,7 @@ void tcg_gen_callN(void *func, TCGTemp *ret, int nargs, TCGTemp **args) } else { nb_rets = 0; } - op->callo = nb_rets; + TCGOP_CALLO(op) = nb_rets; real_args = 0; for (i = 0; i < nargs; i++) { @@ -1469,10 +1469,10 @@ void tcg_gen_callN(void *func, TCGTemp *ret, int nargs, TCGTemp **args) } op->args[pi++] = (uintptr_t)func; op->args[pi++] = flags; - op->calli = real_args; + TCGOP_CALLI(op) = real_args; /* Make sure the fields didn't overflow. */ - tcg_debug_assert(op->calli == real_args); + tcg_debug_assert(TCGOP_CALLI(op) == real_args); tcg_debug_assert(pi <= ARRAY_SIZE(op->args)); #if defined(__sparc__) && !defined(__arch64__) \ @@ -1634,8 +1634,8 @@ void tcg_dump_ops(TCGContext *s) } } else if (c == INDEX_op_call) { /* variable number of arguments */ - nb_oargs = op->callo; - nb_iargs = op->calli; + nb_oargs = TCGOP_CALLO(op); + nb_iargs = TCGOP_CALLI(op); nb_cargs = def->nb_cargs; /* function name, flags, out args */ @@ -1996,8 +1996,8 @@ static void liveness_pass_1(TCGContext *s) { int call_flags; - nb_oargs = op->callo; - nb_iargs = op->calli; + nb_oargs = TCGOP_CALLO(op); + nb_iargs = TCGOP_CALLI(op); call_flags = op->args[nb_oargs + nb_iargs + 1]; /* pure functions can be removed if their result is unused */ @@ -2233,8 +2233,8 @@ static bool liveness_pass_2(TCGContext *s) TCGTemp *arg_ts, *dir_ts; if (opc == INDEX_op_call) { - nb_oargs = op->callo; - nb_iargs = op->calli; + nb_oargs = TCGOP_CALLO(op); + nb_iargs = TCGOP_CALLI(op); call_flags = op->args[nb_oargs + nb_iargs + 1]; } else { nb_iargs = def->nb_iargs; @@ -2915,8 +2915,8 @@ static void tcg_reg_alloc_op(TCGContext *s, const TCGOp *op) static void tcg_reg_alloc_call(TCGContext *s, TCGOp *op) { - const int nb_oargs = op->callo; - const int nb_iargs = op->calli; + const int nb_oargs = TCGOP_CALLO(op); + const int nb_iargs = TCGOP_CALLI(op); const TCGLifeData arg_life = op->life; int flags, nb_regs, i; TCGReg reg; From patchwork Tue Nov 21 21:25:12 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 119418 Delivered-To: patch@linaro.org Received: by 10.140.22.164 with SMTP id 33csp5821279qgn; Tue, 21 Nov 2017 13:38:52 -0800 (PST) X-Google-Smtp-Source: AGs4zMYFsSuORiHSABhg3n/Mk8Ak7+j+rM8JyyFsfpmsrdXW+dEjFHMEXG+bVE0ybuZsLGg9jQca X-Received: by 10.37.76.198 with SMTP id z189mr11513658yba.129.1511300332062; Tue, 21 Nov 2017 13:38:52 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1511300332; cv=none; d=google.com; s=arc-20160816; b=c13Wg6iTrCtjEWjdAv03g1g45ktpY2Tu/qC7TL8TtrG4/L0ujKqYIJwIN5mv/42449 bBIHmuW6RXK61Hkit/pCB3K9pS/xSTeV5z4vvRRcmXT2dcwxmuTnz0UlUSvzLS9dfLs+ pmyHn51n7tjNRroyEdaB+v/jmc9L7hgo4tFWz7Cn6uo3VZHwVCZ6bFpU/kqCJGt/L5mB TofFwnjezSZGYpfapCLEwM4NX9K4yPAPbkEfKacYgc78/XPU0V0gEjiUdLOHUgHAcjZR OpVv57CEtNk5gP2SRtZry516yjN4UvhG/e13IH63EqeUO2Zwv1ePA6Pgzf+6Uq/9KJbe ghqA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:subject:references:in-reply-to :message-id:date:to:from:dkim-signature:arc-authentication-results; bh=WENuCFCRL3Sk/4auN2A/skkqPsLKk3iCNRgUviD4Bl0=; b=VlOcTJDGM/6tHbPdj7F5PEqd+jDzcAlLksJK4brvccNufVizW1c7bGQCjnL9CNynJk IkZxIEbV7g2LI/MOU8aEbEWzHq5KG/jdvViBX6bSENEZumE5M1Vq7B24aa7lF4fef9eF nqMick/unt4pxptjj32YWwvmm3AR9+LG37ed/68Nc1PMIScGr9yyspxn52UXYeNZmWMa DBepKNmN0xGQh61h5QSN+Su2bwQtkR1XWk+Vgh1QQAIW4N4N8Q5+bw6gwNjfI9KL3LC4 0VfqZnAwn8sbCTjDBmbN6mb7SQp4Kc99gwRk4UHZ31D+D5kuZJ5pfsirs6CfvKwbqfFE KSJQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=Y9UEeugf; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) smtp.mailfrom=qemu-devel-bounces+patch=linaro.org@nongnu.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [2001:4830:134:3::11]) by mx.google.com with ESMTPS id p3si3117741ywd.532.2017.11.21.13.38.51 for (version=TLS1 cipher=AES128-SHA bits=128/128); Tue, 21 Nov 2017 13:38:52 -0800 (PST) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) client-ip=2001:4830:134:3::11; Authentication-Results: mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=Y9UEeugf; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) smtp.mailfrom=qemu-devel-bounces+patch=linaro.org@nongnu.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1]:36629 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1eHGFn-0004jS-CV for patch@linaro.org; Tue, 21 Nov 2017 16:38:51 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:53964) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1eHG55-0004Jy-OK for qemu-devel@nongnu.org; Tue, 21 Nov 2017 16:27:51 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1eHG51-0007eZ-R9 for qemu-devel@nongnu.org; Tue, 21 Nov 2017 16:27:47 -0500 Received: from mail-wm0-x243.google.com ([2a00:1450:400c:c09::243]:45456) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1eHG51-0007dq-F0 for qemu-devel@nongnu.org; Tue, 21 Nov 2017 16:27:43 -0500 Received: by mail-wm0-x243.google.com with SMTP id 9so6295789wme.4 for ; Tue, 21 Nov 2017 13:27:43 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:subject:date:message-id:in-reply-to:references; bh=WENuCFCRL3Sk/4auN2A/skkqPsLKk3iCNRgUviD4Bl0=; b=Y9UEeugfwR1DyT9KSendPo2uVyxkh0Cb/3ocU5dd8BNx0fg4rGeRGGoA3Aea2hqvrx VmArdwWLbu+tZuIFu4CIHjwYeeBFLfUBvVJ8pitA6n4gs237WPKAy5SuQa5loWpb+TRS Yi+mMSFSteGIosKJoaUV2gk9n4SfddJfzov7U= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references; bh=WENuCFCRL3Sk/4auN2A/skkqPsLKk3iCNRgUviD4Bl0=; b=dxLlJaJpjQWPgF7YrRQ3XraUdG3xZvQkbk348JKI1fn+vjr0Iwj/S3A0yFz2uNCj4U SY7MhxOf5FuSNggFHXfPjJCSOUXwy73B474ADe8zrJmgo2r5YT6dDvoHn+P/+W5+9oNr wnrGbaNY0WfcX66ZcCFLfl8iTrQ0rVMDRPDuaS8LO7iD5SzQedvgT6UkCwXoC6X825J0 eLGwFksNvEzBBu9vbsYZnbU8ju2J0X+G7Ar3jl8fZyyt5j0o7cBRU3TF+Sb0J0gmCzl4 jHmyQFLLH7fE0l44qgApR1754ruh9YQF3ewqYTZW2KUwCPjX/5sYQaoD4kUXN7N0Xnve rSsw== X-Gm-Message-State: AJaThX7bh8m+2fomkAIBCA8kaySLbHgnyx2lzCRuKvJ896Z0fO2vBEGW 82q6bMMMeJg45Kf9YaH3VZ3MpIBk+Ko= X-Received: by 10.28.141.211 with SMTP id p202mr2273128wmd.61.1511299661616; Tue, 21 Nov 2017 13:27:41 -0800 (PST) Received: from cloudburst.twiddle.net (70.red-37-158-60.dynamicip.rima-tde.net. [37.158.60.70]) by smtp.gmail.com with ESMTPSA id e124sm706517wmg.34.2017.11.21.13.27.40 for (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Tue, 21 Nov 2017 13:27:40 -0800 (PST) From: Richard Henderson To: qemu-devel@nongnu.org Date: Tue, 21 Nov 2017 22:25:12 +0100 Message-Id: <20171121212534.5177-5-richard.henderson@linaro.org> X-Mailer: git-send-email 2.13.6 In-Reply-To: <20171121212534.5177-1-richard.henderson@linaro.org> References: <20171121212534.5177-1-richard.henderson@linaro.org> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2a00:1450:400c:c09::243 Subject: [Qemu-devel] [PATCH v6 04/26] tcg: Add types and basic operations for host vectors X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: "Qemu-devel" Nothing uses or enables them yet. Signed-off-by: Richard Henderson --- Makefile.target | 4 +- tcg/tcg-op.h | 30 +++++ tcg/tcg-opc.h | 26 ++++ tcg/tcg.h | 56 +++++++++ tcg/tcg-op-vec.c | 362 +++++++++++++++++++++++++++++++++++++++++++++++++++++++ tcg/tcg.c | 100 ++++++++++++++- tcg/README | 58 +++++++++ 7 files changed, 630 insertions(+), 6 deletions(-) create mode 100644 tcg/tcg-op-vec.c -- 2.13.6 diff --git a/Makefile.target b/Makefile.target index e4244c188a..5ca758f13c 100644 --- a/Makefile.target +++ b/Makefile.target @@ -93,8 +93,8 @@ all: $(PROGS) stap # cpu emulator library obj-y += exec.o obj-y += accel/ -obj-$(CONFIG_TCG) += tcg/tcg.o tcg/tcg-op.o tcg/optimize.o -obj-$(CONFIG_TCG) += tcg/tcg-common.o +obj-$(CONFIG_TCG) += tcg/tcg.o tcg/tcg-op.o tcg/tcg-op-vec.o +obj-$(CONFIG_TCG) += tcg/tcg-common.o tcg/optimize.o obj-$(CONFIG_TCG_INTERPRETER) += tcg/tci.o obj-$(CONFIG_TCG_INTERPRETER) += disas/tci.o obj-y += fpu/softfloat.o diff --git a/tcg/tcg-op.h b/tcg/tcg-op.h index ca07b32b65..9b0560e4d3 100644 --- a/tcg/tcg-op.h +++ b/tcg/tcg-op.h @@ -35,6 +35,10 @@ void tcg_gen_op4(TCGOpcode, TCGArg, TCGArg, TCGArg, TCGArg); void tcg_gen_op5(TCGOpcode, TCGArg, TCGArg, TCGArg, TCGArg, TCGArg); void tcg_gen_op6(TCGOpcode, TCGArg, TCGArg, TCGArg, TCGArg, TCGArg, TCGArg); +void vec_gen_2(TCGOpcode, TCGType, unsigned, TCGArg, TCGArg); +void vec_gen_3(TCGOpcode, TCGType, unsigned, TCGArg, TCGArg, TCGArg); +void vec_gen_4(TCGOpcode, TCGType, unsigned, TCGArg, TCGArg, TCGArg, TCGArg); + static inline void tcg_gen_op1_i32(TCGOpcode opc, TCGv_i32 a1) { tcg_gen_op1(opc, tcgv_i32_arg(a1)); @@ -903,6 +907,30 @@ void tcg_gen_atomic_or_fetch_i64(TCGv_i64, TCGv, TCGv_i64, TCGArg, TCGMemOp); void tcg_gen_atomic_xor_fetch_i32(TCGv_i32, TCGv, TCGv_i32, TCGArg, TCGMemOp); void tcg_gen_atomic_xor_fetch_i64(TCGv_i64, TCGv, TCGv_i64, TCGArg, TCGMemOp); +void tcg_gen_mov_vec(TCGv_vec, TCGv_vec); +void tcg_gen_dup_i32_vec(unsigned vece, TCGv_vec, TCGv_i32); +void tcg_gen_dup_i64_vec(unsigned vece, TCGv_vec, TCGv_i64); +void tcg_gen_dup8i_vec(TCGv_vec, uint32_t); +void tcg_gen_dup16i_vec(TCGv_vec, uint32_t); +void tcg_gen_dup32i_vec(TCGv_vec, uint32_t); +void tcg_gen_dup64i_vec(TCGv_vec, uint64_t); +void tcg_gen_movi_v64(TCGv_vec, uint64_t); +void tcg_gen_movi_v128(TCGv_vec, uint64_t, uint64_t); +void tcg_gen_movi_v256(TCGv_vec, uint64_t, uint64_t, uint64_t, uint64_t); +void tcg_gen_add_vec(unsigned vece, TCGv_vec r, TCGv_vec a, TCGv_vec b); +void tcg_gen_sub_vec(unsigned vece, TCGv_vec r, TCGv_vec a, TCGv_vec b); +void tcg_gen_and_vec(unsigned vece, TCGv_vec r, TCGv_vec a, TCGv_vec b); +void tcg_gen_or_vec(unsigned vece, TCGv_vec r, TCGv_vec a, TCGv_vec b); +void tcg_gen_xor_vec(unsigned vece, TCGv_vec r, TCGv_vec a, TCGv_vec b); +void tcg_gen_andc_vec(unsigned vece, TCGv_vec r, TCGv_vec a, TCGv_vec b); +void tcg_gen_orc_vec(unsigned vece, TCGv_vec r, TCGv_vec a, TCGv_vec b); +void tcg_gen_not_vec(unsigned vece, TCGv_vec r, TCGv_vec a); +void tcg_gen_neg_vec(unsigned vece, TCGv_vec r, TCGv_vec a); + +void tcg_gen_ld_vec(TCGv_vec r, TCGv_ptr base, TCGArg offset); +void tcg_gen_st_vec(TCGv_vec r, TCGv_ptr base, TCGArg offset); +void tcg_gen_stl_vec(TCGv_vec r, TCGv_ptr base, TCGArg offset, TCGType t); + #if TARGET_LONG_BITS == 64 #define tcg_gen_movi_tl tcg_gen_movi_i64 #define tcg_gen_mov_tl tcg_gen_mov_i64 @@ -1001,6 +1029,7 @@ void tcg_gen_atomic_xor_fetch_i64(TCGv_i64, TCGv, TCGv_i64, TCGArg, TCGMemOp); #define tcg_gen_atomic_and_fetch_tl tcg_gen_atomic_and_fetch_i64 #define tcg_gen_atomic_or_fetch_tl tcg_gen_atomic_or_fetch_i64 #define tcg_gen_atomic_xor_fetch_tl tcg_gen_atomic_xor_fetch_i64 +#define tcg_gen_dup_tl_vec tcg_gen_dup_i64_vec #else #define tcg_gen_movi_tl tcg_gen_movi_i32 #define tcg_gen_mov_tl tcg_gen_mov_i32 @@ -1098,6 +1127,7 @@ void tcg_gen_atomic_xor_fetch_i64(TCGv_i64, TCGv, TCGv_i64, TCGArg, TCGMemOp); #define tcg_gen_atomic_and_fetch_tl tcg_gen_atomic_and_fetch_i32 #define tcg_gen_atomic_or_fetch_tl tcg_gen_atomic_or_fetch_i32 #define tcg_gen_atomic_xor_fetch_tl tcg_gen_atomic_xor_fetch_i32 +#define tcg_gen_dup_tl_vec tcg_gen_dup_i32_vec #endif #if UINTPTR_MAX == UINT32_MAX diff --git a/tcg/tcg-opc.h b/tcg/tcg-opc.h index 956fb1e9f3..4e62eda14b 100644 --- a/tcg/tcg-opc.h +++ b/tcg/tcg-opc.h @@ -204,8 +204,34 @@ DEF(qemu_ld_i64, DATA64_ARGS, TLADDR_ARGS, 1, DEF(qemu_st_i64, 0, TLADDR_ARGS + DATA64_ARGS, 1, TCG_OPF_CALL_CLOBBER | TCG_OPF_SIDE_EFFECTS | TCG_OPF_64BIT) +/* Host vector support. */ + +#define IMPLVEC TCG_OPF_VECTOR | IMPL(TCG_TARGET_MAYBE_vec) + +DEF(mov_vec, 1, 1, 0, TCG_OPF_VECTOR | TCG_OPF_NOT_PRESENT) +DEF(movi_vec, 1, 0, 0, TCG_OPF_VECTOR | TCG_OPF_NOT_PRESENT) /* vecl defines const args */ +DEF(dupi_vec, 1, 0, 1, TCG_OPF_VECTOR | TCG_OPF_NOT_PRESENT) + +DEF(dup_vec, 1, 1, 0, IMPLVEC) +DEF(dup2_vec, 1, 2, 0, IMPLVEC | IMPL(TCG_TARGET_REG_BITS == 32)) + +DEF(ld_vec, 1, 1, 1, IMPLVEC) +DEF(st_vec, 0, 2, 1, IMPLVEC) + +DEF(add_vec, 1, 2, 0, IMPLVEC) +DEF(sub_vec, 1, 2, 0, IMPLVEC) +DEF(neg_vec, 1, 1, 0, IMPLVEC | IMPL(TCG_TARGET_HAS_neg_vec)) + +DEF(and_vec, 1, 2, 0, IMPLVEC) +DEF(or_vec, 1, 2, 0, IMPLVEC) +DEF(xor_vec, 1, 2, 0, IMPLVEC) +DEF(andc_vec, 1, 2, 0, IMPLVEC | IMPL(TCG_TARGET_HAS_andc_vec)) +DEF(orc_vec, 1, 2, 0, IMPLVEC | IMPL(TCG_TARGET_HAS_orc_vec)) +DEF(not_vec, 1, 1, 0, IMPLVEC | IMPL(TCG_TARGET_HAS_not_vec)) + #undef TLADDR_ARGS #undef DATA64_ARGS #undef IMPL #undef IMPL64 +#undef IMPLVEC #undef DEF diff --git a/tcg/tcg.h b/tcg/tcg.h index f25efa9795..2acebd387a 100644 --- a/tcg/tcg.h +++ b/tcg/tcg.h @@ -170,6 +170,27 @@ typedef uint64_t TCGRegSet; # error "Missing unsigned widening multiply" #endif +#if !defined(TCG_TARGET_HAS_v64) \ + && !defined(TCG_TARGET_HAS_v128) \ + && !defined(TCG_TARGET_HAS_v256) +#define TCG_TARGET_MAYBE_vec 0 +#define TCG_TARGET_HAS_neg_vec 0 +#define TCG_TARGET_HAS_not_vec 0 +#define TCG_TARGET_HAS_andc_vec 0 +#define TCG_TARGET_HAS_orc_vec 0 +#else +#define TCG_TARGET_MAYBE_vec 1 +#endif +#ifndef TCG_TARGET_HAS_v64 +#define TCG_TARGET_HAS_v64 0 +#endif +#ifndef TCG_TARGET_HAS_v128 +#define TCG_TARGET_HAS_v128 0 +#endif +#ifndef TCG_TARGET_HAS_v256 +#define TCG_TARGET_HAS_v256 0 +#endif + #ifndef TARGET_INSN_START_EXTRA_WORDS # define TARGET_INSN_START_WORDS 1 #else @@ -246,6 +267,11 @@ typedef struct TCGPool { typedef enum TCGType { TCG_TYPE_I32, TCG_TYPE_I64, + + TCG_TYPE_V64, + TCG_TYPE_V128, + TCG_TYPE_V256, + TCG_TYPE_COUNT, /* number of different types */ /* An alias for the size of the host register. */ @@ -396,6 +422,8 @@ typedef tcg_target_ulong TCGArg; * TCGv_i32 : 32 bit integer type * TCGv_i64 : 64 bit integer type * TCGv_ptr : a host pointer type + * TCGv_vec : a host vector type; the exact size is not exposed + to the CPU front-end code. * TCGv : an integer type the same size as target_ulong (an alias for either TCGv_i32 or TCGv_i64) The compiler's type checking will complain if you mix them @@ -418,6 +446,7 @@ typedef tcg_target_ulong TCGArg; typedef struct TCGv_i32_d *TCGv_i32; typedef struct TCGv_i64_d *TCGv_i64; typedef struct TCGv_ptr_d *TCGv_ptr; +typedef struct TCGv_vec_d *TCGv_vec; typedef TCGv_ptr TCGv_env; #if TARGET_LONG_BITS == 32 #define TCGv TCGv_i32 @@ -583,6 +612,9 @@ typedef struct TCGOp { #define TCGOP_CALLI(X) (X)->param1 #define TCGOP_CALLO(X) (X)->param2 +#define TCGOP_VECL(X) (X)->param1 +#define TCGOP_VECE(X) (X)->param2 + /* Make sure operands fit in the bitfields above. */ QEMU_BUILD_BUG_ON(NB_OPS > (1 << 8)); @@ -720,6 +752,11 @@ static inline TCGTemp *tcgv_ptr_temp(TCGv_ptr v) return tcgv_i32_temp((TCGv_i32)v); } +static inline TCGTemp *tcgv_vec_temp(TCGv_vec v) +{ + return tcgv_i32_temp((TCGv_i32)v); +} + static inline TCGArg tcgv_i32_arg(TCGv_i32 v) { return temp_arg(tcgv_i32_temp(v)); @@ -735,6 +772,11 @@ static inline TCGArg tcgv_ptr_arg(TCGv_ptr v) return temp_arg(tcgv_ptr_temp(v)); } +static inline TCGArg tcgv_vec_arg(TCGv_vec v) +{ + return temp_arg(tcgv_vec_temp(v)); +} + static inline TCGv_i32 temp_tcgv_i32(TCGTemp *t) { (void)temp_idx(t); /* trigger embedded assert */ @@ -751,6 +793,11 @@ static inline TCGv_ptr temp_tcgv_ptr(TCGTemp *t) return (TCGv_ptr)temp_tcgv_i32(t); } +static inline TCGv_vec temp_tcgv_vec(TCGTemp *t) +{ + return (TCGv_vec)temp_tcgv_i32(t); +} + #if TCG_TARGET_REG_BITS == 32 static inline TCGv_i32 TCGV_LOW(TCGv_i64 t) { @@ -826,9 +873,12 @@ TCGTemp *tcg_global_mem_new_internal(TCGType, TCGv_ptr, TCGv_i32 tcg_temp_new_internal_i32(int temp_local); TCGv_i64 tcg_temp_new_internal_i64(int temp_local); +TCGv_vec tcg_temp_new_vec(TCGType type); +TCGv_vec tcg_temp_new_vec_matching(TCGv_vec match); void tcg_temp_free_i32(TCGv_i32 arg); void tcg_temp_free_i64(TCGv_i64 arg); +void tcg_temp_free_vec(TCGv_vec arg); static inline TCGv_i32 tcg_global_mem_new_i32(TCGv_ptr reg, intptr_t offset, const char *name) @@ -910,6 +960,8 @@ enum { /* Instruction is optional and not implemented by the host, or insn is generic and should not be implemened by the host. */ TCG_OPF_NOT_PRESENT = 0x10, + /* Instruction operands are vectors. */ + TCG_OPF_VECTOR = 0x20, }; typedef struct TCGOpDef { @@ -975,6 +1027,10 @@ TCGv_i32 tcg_const_i32(int32_t val); TCGv_i64 tcg_const_i64(int64_t val); TCGv_i32 tcg_const_local_i32(int32_t val); TCGv_i64 tcg_const_local_i64(int64_t val); +TCGv_vec tcg_const_zeros_vec(TCGType); +TCGv_vec tcg_const_ones_vec(TCGType); +TCGv_vec tcg_const_zeros_vec_matching(TCGv_vec); +TCGv_vec tcg_const_ones_vec_matching(TCGv_vec); TCGLabel *gen_new_label(void); diff --git a/tcg/tcg-op-vec.c b/tcg/tcg-op-vec.c new file mode 100644 index 0000000000..dc04c11860 --- /dev/null +++ b/tcg/tcg-op-vec.c @@ -0,0 +1,362 @@ +/* + * Tiny Code Generator for QEMU + * + * Copyright (c) 2008 Fabrice Bellard + * + * Permission is hereby granted, free of charge, to any person obtaining a copy + * of this software and associated documentation files (the "Software"), to deal + * in the Software without restriction, including without limitation the rights + * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell + * copies of the Software, and to permit persons to whom the Software is + * furnished to do so, subject to the following conditions: + * + * The above copyright notice and this permission notice shall be included in + * all copies or substantial portions of the Software. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, + * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL + * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER + * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, + * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN + * THE SOFTWARE. + */ + +#include "qemu/osdep.h" +#include "qemu-common.h" +#include "cpu.h" +#include "exec/exec-all.h" +#include "tcg.h" +#include "tcg-op.h" +#include "tcg-mo.h" + +/* Reduce the number of ifdefs below. This assumes that all uses of + TCGV_HIGH and TCGV_LOW are properly protected by a conditional that + the compiler can eliminate. */ +#if TCG_TARGET_REG_BITS == 64 +extern TCGv_i32 TCGV_LOW_link_error(TCGv_i64); +extern TCGv_i32 TCGV_HIGH_link_error(TCGv_i64); +#define TCGV_LOW TCGV_LOW_link_error +#define TCGV_HIGH TCGV_HIGH_link_error +#endif + +void vec_gen_2(TCGOpcode opc, TCGType type, unsigned vece, TCGArg r, TCGArg a) +{ + TCGOp *op = tcg_emit_op(opc); + TCGOP_VECL(op) = type - TCG_TYPE_V64; + TCGOP_VECE(op) = vece; + op->args[0] = r; + op->args[1] = a; +} + +void vec_gen_3(TCGOpcode opc, TCGType type, unsigned vece, + TCGArg r, TCGArg a, TCGArg b) +{ + TCGOp *op = tcg_emit_op(opc); + TCGOP_VECL(op) = type - TCG_TYPE_V64; + TCGOP_VECE(op) = vece; + op->args[0] = r; + op->args[1] = a; + op->args[2] = b; +} + +void vec_gen_4(TCGOpcode opc, TCGType type, unsigned vece, + TCGArg r, TCGArg a, TCGArg b, TCGArg c) +{ + TCGOp *op = tcg_emit_op(opc); + TCGOP_VECL(op) = type - TCG_TYPE_V64; + TCGOP_VECE(op) = vece; + op->args[0] = r; + op->args[1] = a; + op->args[2] = b; + op->args[3] = c; +} + +static void vec_gen_op2(TCGOpcode opc, unsigned vece, TCGv_vec r, TCGv_vec a) +{ + TCGTemp *rt = tcgv_vec_temp(r); + TCGTemp *at = tcgv_vec_temp(a); + TCGType type = rt->base_type; + + tcg_debug_assert(at->base_type == type); + vec_gen_2(opc, type, vece, temp_arg(rt), temp_arg(at)); +} + +static void vec_gen_op3(TCGOpcode opc, unsigned vece, + TCGv_vec r, TCGv_vec a, TCGv_vec b) +{ + TCGTemp *rt = tcgv_vec_temp(r); + TCGTemp *at = tcgv_vec_temp(a); + TCGTemp *bt = tcgv_vec_temp(b); + TCGType type = rt->base_type; + + tcg_debug_assert(at->base_type == type); + tcg_debug_assert(bt->base_type == type); + vec_gen_3(opc, type, vece, temp_arg(rt), temp_arg(at), temp_arg(bt)); +} + +void tcg_gen_mov_vec(TCGv_vec r, TCGv_vec a) +{ + if (r != a) { + vec_gen_op2(INDEX_op_mov_vec, 0, r, a); + } +} + +#define MO_REG (TCG_TARGET_REG_BITS == 64 ? MO_64 : MO_32) + +static void tcg_gen_dupi_vec(TCGv_vec r, unsigned vece, TCGArg a) +{ + TCGTemp *rt = tcgv_vec_temp(r); + vec_gen_2(INDEX_op_dupi_vec, rt->base_type, vece, temp_arg(rt), a); +} + +TCGv_vec tcg_const_zeros_vec(TCGType type) +{ + TCGv_vec ret = tcg_temp_new_vec(type); + tcg_gen_dupi_vec(ret, MO_REG, 0); + return ret; +} + +TCGv_vec tcg_const_ones_vec(TCGType type) +{ + TCGv_vec ret = tcg_temp_new_vec(type); + tcg_gen_dupi_vec(ret, MO_REG, -1); + return ret; +} + +TCGv_vec tcg_const_zeros_vec_matching(TCGv_vec m) +{ + TCGTemp *t = tcgv_vec_temp(m); + return tcg_const_zeros_vec(t->base_type); +} + +TCGv_vec tcg_const_ones_vec_matching(TCGv_vec m) +{ + TCGTemp *t = tcgv_vec_temp(m); + return tcg_const_ones_vec(t->base_type); +} + +void tcg_gen_dup64i_vec(TCGv_vec r, uint64_t a) +{ + if (TCG_TARGET_REG_BITS == 32 && a == deposit64(a, 32, 32, a)) { + tcg_gen_dupi_vec(r, MO_32, a); + } else if (TCG_TARGET_REG_BITS == 64 || a == (uint64_t)(int32_t)a) { + tcg_gen_dupi_vec(r, MO_64, a); + } else { + TCGv_i64 c = tcg_const_i64(a); + tcg_gen_dup_i64_vec(MO_64, r, c); + tcg_temp_free_i64(c); + } +} + +void tcg_gen_dup32i_vec(TCGv_vec r, uint32_t a) +{ + tcg_gen_dupi_vec(r, MO_REG, ((TCGArg)-1 / 0xffffffffu) * a); +} + +void tcg_gen_dup16i_vec(TCGv_vec r, uint32_t a) +{ + tcg_gen_dupi_vec(r, MO_REG, ((TCGArg)-1 / 0xffff) * (a & 0xffff)); +} + +void tcg_gen_dup8i_vec(TCGv_vec r, uint32_t a) +{ + tcg_gen_dupi_vec(r, MO_REG, ((TCGArg)-1 / 0xff) * (a & 0xff)); +} + +void tcg_gen_movi_v64(TCGv_vec r, uint64_t a) +{ + TCGTemp *rt = tcgv_vec_temp(r); + TCGArg ri = temp_arg(rt); + + tcg_debug_assert(rt->base_type == TCG_TYPE_V64); + if (TCG_TARGET_REG_BITS == 64) { + vec_gen_2(INDEX_op_movi_vec, TCG_TYPE_V64, 0, ri, a); + } else { + vec_gen_3(INDEX_op_movi_vec, TCG_TYPE_V64, 0, ri, a, a >> 32); + } +} + +void tcg_gen_movi_v128(TCGv_vec r, uint64_t a, uint64_t b) +{ + TCGTemp *rt = tcgv_vec_temp(r); + TCGArg ri = temp_arg(rt); + + tcg_debug_assert(rt->base_type == TCG_TYPE_V128); + if (a == b) { + tcg_gen_dup64i_vec(r, a); + } else if (TCG_TARGET_REG_BITS == 64) { + vec_gen_3(INDEX_op_movi_vec, TCG_TYPE_V128, 0, ri, a, b); + } else { + TCGOp *op = tcg_emit_op(INDEX_op_movi_vec); + TCGOP_VECL(op) = TCG_TYPE_V128 - TCG_TYPE_V64; + op->args[0] = ri; + op->args[1] = a; + op->args[2] = a >> 32; + op->args[3] = b; + op->args[4] = b >> 32; + } +} + +void tcg_gen_movi_v256(TCGv_vec r, uint64_t a, uint64_t b, + uint64_t c, uint64_t d) +{ + TCGArg ri = tcgv_vec_arg(r); + TCGTemp *rt = arg_temp(ri); + + tcg_debug_assert(rt->base_type == TCG_TYPE_V256); + if (a == b && a == c && a == d) { + tcg_gen_dup64i_vec(r, a); + } else { + TCGOp *op = tcg_emit_op(INDEX_op_movi_vec); + TCGOP_VECL(op) = TCG_TYPE_V256 - TCG_TYPE_V64; + op->args[0] = ri; + if (TCG_TARGET_REG_BITS == 64) { + op->args[1] = a; + op->args[2] = b; + op->args[3] = c; + op->args[4] = d; + } else { + op->args[1] = a; + op->args[2] = a >> 32; + op->args[3] = b; + op->args[4] = b >> 32; + op->args[5] = c; + op->args[6] = c >> 32; + op->args[7] = d; + op->args[8] = d >> 32; + } + } +} + +void tcg_gen_dup_i64_vec(unsigned vece, TCGv_vec r, TCGv_i64 a) +{ + TCGArg ri = tcgv_vec_arg(r); + TCGTemp *rt = arg_temp(ri); + TCGType type = rt->base_type; + + if (TCG_TARGET_REG_BITS == 64) { + TCGArg ai = tcgv_i64_arg(a); + vec_gen_2(INDEX_op_dup_vec, type, MO_64, ri, ai); + } else if (vece == MO_64) { + TCGArg al = tcgv_i32_arg(TCGV_LOW(a)); + TCGArg ah = tcgv_i32_arg(TCGV_HIGH(a)); + vec_gen_3(INDEX_op_dup2_vec, type, MO_64, ri, al, ah); + } else { + TCGArg ai = tcgv_i32_arg(TCGV_LOW(a)); + vec_gen_2(INDEX_op_dup_vec, type, MO_64, ri, ai); + } +} + +void tcg_gen_dup_i32_vec(unsigned vece, TCGv_vec r, TCGv_i32 a) +{ + TCGArg ri = tcgv_vec_arg(r); + TCGArg ai = tcgv_i32_arg(a); + TCGTemp *rt = arg_temp(ri); + TCGType type = rt->base_type; + + vec_gen_2(INDEX_op_dup_vec, type, vece, ri, ai); +} + +static void vec_gen_ldst(TCGOpcode opc, TCGv_vec r, TCGv_ptr b, TCGArg o) +{ + TCGArg ri = tcgv_vec_arg(r); + TCGArg bi = tcgv_ptr_arg(b); + TCGTemp *rt = arg_temp(ri); + TCGType type = rt->base_type; + + vec_gen_3(opc, type, 0, ri, bi, o); +} + +void tcg_gen_ld_vec(TCGv_vec r, TCGv_ptr b, TCGArg o) +{ + vec_gen_ldst(INDEX_op_ld_vec, r, b, o); +} + +void tcg_gen_st_vec(TCGv_vec r, TCGv_ptr b, TCGArg o) +{ + vec_gen_ldst(INDEX_op_st_vec, r, b, o); +} + +void tcg_gen_stl_vec(TCGv_vec r, TCGv_ptr b, TCGArg o, TCGType low_type) +{ + TCGArg ri = tcgv_vec_arg(r); + TCGArg bi = tcgv_ptr_arg(b); + TCGTemp *rt = arg_temp(ri); + TCGType type = rt->base_type; + + tcg_debug_assert(low_type >= TCG_TYPE_V64); + tcg_debug_assert(low_type <= type); + vec_gen_3(INDEX_op_st_vec, low_type, 0, ri, bi, o); +} + +void tcg_gen_add_vec(unsigned vece, TCGv_vec r, TCGv_vec a, TCGv_vec b) +{ + vec_gen_op3(INDEX_op_add_vec, vece, r, a, b); +} + +void tcg_gen_sub_vec(unsigned vece, TCGv_vec r, TCGv_vec a, TCGv_vec b) +{ + vec_gen_op3(INDEX_op_sub_vec, vece, r, a, b); +} + +void tcg_gen_and_vec(unsigned vece, TCGv_vec r, TCGv_vec a, TCGv_vec b) +{ + vec_gen_op3(INDEX_op_and_vec, 0, r, a, b); +} + +void tcg_gen_or_vec(unsigned vece, TCGv_vec r, TCGv_vec a, TCGv_vec b) +{ + vec_gen_op3(INDEX_op_or_vec, 0, r, a, b); +} + +void tcg_gen_xor_vec(unsigned vece, TCGv_vec r, TCGv_vec a, TCGv_vec b) +{ + vec_gen_op3(INDEX_op_xor_vec, 0, r, a, b); +} + +void tcg_gen_andc_vec(unsigned vece, TCGv_vec r, TCGv_vec a, TCGv_vec b) +{ + if (TCG_TARGET_HAS_andc_vec) { + vec_gen_op3(INDEX_op_andc_vec, 0, r, a, b); + } else { + TCGv_vec t = tcg_temp_new_vec_matching(r); + tcg_gen_not_vec(0, t, b); + tcg_gen_and_vec(0, r, a, t); + tcg_temp_free_vec(t); + } +} + +void tcg_gen_orc_vec(unsigned vece, TCGv_vec r, TCGv_vec a, TCGv_vec b) +{ + if (TCG_TARGET_HAS_orc_vec) { + vec_gen_op3(INDEX_op_orc_vec, 0, r, a, b); + } else { + TCGv_vec t = tcg_temp_new_vec_matching(r); + tcg_gen_not_vec(0, t, b); + tcg_gen_or_vec(0, r, a, t); + tcg_temp_free_vec(t); + } +} + +void tcg_gen_not_vec(unsigned vece, TCGv_vec r, TCGv_vec a) +{ + if (TCG_TARGET_HAS_not_vec) { + vec_gen_op2(INDEX_op_not_vec, 0, r, a); + } else { + TCGv_vec t = tcg_const_ones_vec_matching(r); + tcg_gen_xor_vec(0, r, a, t); + tcg_temp_free_vec(t); + } +} + +void tcg_gen_neg_vec(unsigned vece, TCGv_vec r, TCGv_vec a) +{ + if (TCG_TARGET_HAS_neg_vec) { + vec_gen_op2(INDEX_op_neg_vec, vece, r, a); + } else { + TCGv_vec t = tcg_const_zeros_vec_matching(r); + tcg_gen_sub_vec(vece, r, t, a); + tcg_temp_free_vec(t); + } +} diff --git a/tcg/tcg.c b/tcg/tcg.c index 93caa0be93..16b8faf66f 100644 --- a/tcg/tcg.c +++ b/tcg/tcg.c @@ -106,6 +106,18 @@ static void tcg_out_movi(TCGContext *s, TCGType type, TCGReg ret, tcg_target_long arg); static void tcg_out_op(TCGContext *s, TCGOpcode opc, const TCGArg *args, const int *const_args); +#if TCG_TARGET_MAYBE_vec +static void tcg_out_vec_op(TCGContext *s, TCGOpcode opc, unsigned vecl, + unsigned vece, const TCGArg *args, + const int *const_args); +#else +static inline void tcg_out_vec_op(TCGContext *s, TCGOpcode opc, unsigned vecl, + unsigned vece, const TCGArg *args, + const int *const_args) +{ + g_assert_not_reached(); +} +#endif static void tcg_out_st(TCGContext *s, TCGType type, TCGReg arg, TCGReg arg1, intptr_t arg2); static bool tcg_out_sti(TCGContext *s, TCGType type, TCGArg val, @@ -146,8 +158,7 @@ struct tcg_region_state { }; static struct tcg_region_state region; - -static TCGRegSet tcg_target_available_regs[2]; +static TCGRegSet tcg_target_available_regs[TCG_TYPE_COUNT]; static TCGRegSet tcg_target_call_clobber_regs; #if TCG_TARGET_INSN_UNIT_SIZE == 1 @@ -1026,6 +1037,41 @@ TCGv_i64 tcg_temp_new_internal_i64(int temp_local) return temp_tcgv_i64(t); } +TCGv_vec tcg_temp_new_vec(TCGType type) +{ + TCGTemp *t; + +#ifdef CONFIG_DEBUG_TCG + switch (type) { + case TCG_TYPE_V64: + assert(TCG_TARGET_HAS_v64); + break; + case TCG_TYPE_V128: + assert(TCG_TARGET_HAS_v128); + break; + case TCG_TYPE_V256: + assert(TCG_TARGET_HAS_v256); + break; + default: + g_assert_not_reached(); + } +#endif + + t = tcg_temp_new_internal(type, 0); + return temp_tcgv_vec(t); +} + +/* Create a new temp of the same type as an existing temp. */ +TCGv_vec tcg_temp_new_vec_matching(TCGv_vec match) +{ + TCGTemp *t = tcgv_vec_temp(match); + + tcg_debug_assert(t->temp_allocated != 0); + + t = tcg_temp_new_internal(t->base_type, 0); + return temp_tcgv_vec(t); +} + static void tcg_temp_free_internal(TCGTemp *ts) { TCGContext *s = tcg_ctx; @@ -1057,6 +1103,11 @@ void tcg_temp_free_i64(TCGv_i64 arg) tcg_temp_free_internal(tcgv_i64_temp(arg)); } +void tcg_temp_free_vec(TCGv_vec arg) +{ + tcg_temp_free_internal(tcgv_vec_temp(arg)); +} + TCGv_i32 tcg_const_i32(int32_t val) { TCGv_i32 t0; @@ -1114,6 +1165,9 @@ int tcg_check_temp_count(void) Test the runtime variable that controls each opcode. */ bool tcg_op_supported(TCGOpcode op) { + const bool have_vec + = TCG_TARGET_HAS_v64 | TCG_TARGET_HAS_v128 | TCG_TARGET_HAS_v256; + switch (op) { case INDEX_op_discard: case INDEX_op_set_label: @@ -1327,6 +1381,29 @@ bool tcg_op_supported(TCGOpcode op) case INDEX_op_mulsh_i64: return TCG_TARGET_HAS_mulsh_i64; + case INDEX_op_mov_vec: + case INDEX_op_movi_vec: + case INDEX_op_dup_vec: + case INDEX_op_dupi_vec: + case INDEX_op_ld_vec: + case INDEX_op_st_vec: + case INDEX_op_add_vec: + case INDEX_op_sub_vec: + case INDEX_op_and_vec: + case INDEX_op_or_vec: + case INDEX_op_xor_vec: + return have_vec; + case INDEX_op_dup2_vec: + return have_vec && TCG_TARGET_REG_BITS == 32; + case INDEX_op_not_vec: + return have_vec && TCG_TARGET_HAS_not_vec; + case INDEX_op_neg_vec: + return have_vec && TCG_TARGET_HAS_neg_vec; + case INDEX_op_andc_vec: + return have_vec && TCG_TARGET_HAS_andc_vec; + case INDEX_op_orc_vec: + return have_vec && TCG_TARGET_HAS_orc_vec; + case NB_OPS: break; } @@ -1661,6 +1738,14 @@ void tcg_dump_ops(TCGContext *s) nb_iargs = def->nb_iargs; nb_cargs = def->nb_cargs; + if (c == INDEX_op_movi_vec) { + nb_cargs = (64 / TCG_TARGET_REG_BITS) << TCGOP_VECL(op); + } + if (def->flags & TCG_OPF_VECTOR) { + col += qemu_log("v%d,e%d,", 64 << TCGOP_VECL(op), + 8 << TCGOP_VECE(op)); + } + k = 0; for (i = 0; i < nb_oargs; i++) { if (k != 0) { @@ -2890,8 +2975,13 @@ static void tcg_reg_alloc_op(TCGContext *s, const TCGOp *op) } /* emit instruction */ - tcg_out_op(s, op->opc, new_args, const_args); - + if (def->flags & TCG_OPF_VECTOR) { + tcg_out_vec_op(s, op->opc, TCGOP_VECL(op), TCGOP_VECE(op), + new_args, const_args); + } else { + tcg_out_op(s, op->opc, new_args, const_args); + } + /* move the outputs in the correct register if needed */ for(i = 0; i < nb_oargs; i++) { ts = arg_temp(op->args[i]); @@ -3239,10 +3329,12 @@ int tcg_gen_code(TCGContext *s, TranslationBlock *tb) switch (opc) { case INDEX_op_mov_i32: case INDEX_op_mov_i64: + case INDEX_op_mov_vec: tcg_reg_alloc_mov(s, op); break; case INDEX_op_movi_i32: case INDEX_op_movi_i64: + case INDEX_op_dupi_vec: tcg_reg_alloc_movi(s, op); break; case INDEX_op_insn_start: diff --git a/tcg/README b/tcg/README index 03bfb6acd4..e14990fb9b 100644 --- a/tcg/README +++ b/tcg/README @@ -503,6 +503,64 @@ of the memory access. For a 32-bit host, qemu_ld/st_i64 is guaranteed to only be used with a 64-bit memory access specified in flags. +********* Host vector operations + +All of the vector ops have two parameters, TCGOP_VECL & TCGOP_VECE. +The former specifies the length of the vector in log2 64-bit units; the +later specifies the length of the element (if applicable) in log2 8-bit units. +E.g. VECL=1 -> 64 << 1 -> v128, and VECE=2 -> 1 << 2 -> i32. + +* mov_vec v0, v1 +* ld_vec v0, t1 +* st_vec v0, t1 + + Move, load and store. + +* movi_vec v0, a, b, ... + + Move with constant data. There are arguments to hold the entire + vector value, stored little-endian. Note that the way MAX_OPC_PARAM + is sized for 64- and 32-bit hosts, there are enough slots for v256, + but not a future v512. + + Prefer dupi_vec when possible. + +* dup_vec v0, r1 + + Duplicate the low N bits of R1 into VECL/VECE copies across V0. + +* dupi_vec v0, c + + Similarly, for a constant. + Smaller values will be replicated to host register size by the expanders. + +* dup2_vec v0, r1, r2 + + Duplicate r2:r1 into VECL/64 copies across V0. This opcode is + only present for 32-bit hosts. + +* add_vec v0, v1, v2 + + v0 = v1 + v2, in elements across the vector. + +* sub_vec v0, v1, v2 + + Similarly, v0 = v1 - v2. + +* neg_vec v0, v1 + + Similarly, v0 = -v1. + +* and_vec v0, v1, v2 +* or_vec v0, v1, v2 +* xor_vec v0, v1, v2 +* andc_vec v0, v1, v2 +* orc_vec v0, v1, v2 +* not_vec v0, v1 + + Similarly, logical operations with and without compliment. + Note that VECE is unused. + ********* Note 1: Some shortcuts are defined when the last operand is known to be From patchwork Tue Nov 21 21:25:13 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 119413 Delivered-To: patch@linaro.org Received: by 10.140.22.164 with SMTP id 33csp5819151qgn; Tue, 21 Nov 2017 13:36:35 -0800 (PST) X-Google-Smtp-Source: AGs4zMZfnE4NPys5fn8K3qoKHs06ZgYM/ulhr8vYH1yk1W53LVw7pb1LhB+mi7agW/nwp3CVxnkg X-Received: by 10.37.202.72 with SMTP id a69mr11476285ybg.496.1511300195867; Tue, 21 Nov 2017 13:36:35 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1511300195; cv=none; d=google.com; s=arc-20160816; b=kY/Hm3CAHWlZf7QvGVYyh+c25W2W2NzSvzeCHKLmQZ3b1sX4tyRJ0kD9QO+YibWHA0 X+m8s9I/aFFPP5oMNLdbfXRINZcj8+BA+DpU8vQRHcKK4VfPqZEMdcVA8FUz4O3r/jJC xiPRqYVEHYGPcUh3IHhs3cPxvSqnW1IQuI/voov95fT0qID2XkQ2VtUmeTg+Wvt2UiAd xQInCl4JuOTuSicVm3AWznewBr8b5Sm6Utyv1JHWXK84r/aMvNj9n3r5T7z9rswpcEG7 0cUuH1skKQEQUERQTCA3FpWAGdNCmwMW/o07CeQwOO+zMLhntDuQZhTsPpjslfM9WAFZ ImFA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:subject:references:in-reply-to :message-id:date:to:from:dkim-signature:arc-authentication-results; bh=jWknO0RbzhLByyUe3nJCH3qKKHeeV0jgF0EpsUb2/bg=; b=OunofbQomdShnvLaedj8yv5uqD3kKulJFWn+isNXBKmYYiPCdDvPIszLd765Yx2lbp zzEcX7fLigKEvtBFM7v4aScBE/DaSRzwbqPwsX+ojBh7cdfECI/mr7iPiNA2MdQcAcKR v+SZmZImNEQbNqNVa2vfXJax55hJcPX9weQlq3J3fPd7OrPMzSgskIMUe+KF7LTsC+Nw BDt9TyLFiCEOGVz8kFFuWtmp0U03omAbFCVjnXMwQSYD9eqmS90AB8HEQaGXIc18s1g+ jbzZyvy5qhd8LnEx764yaSpxsWaoBodjEQZvWeqzl7dJHWZR1UVNlXO2adJmr65BiCzT ++Ng== ARC-Authentication-Results: i=1; mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=hps3IEHF; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) smtp.mailfrom=qemu-devel-bounces+patch=linaro.org@nongnu.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [2001:4830:134:3::11]) by mx.google.com with ESMTPS id t12si3163475ybt.71.2017.11.21.13.36.35 for (version=TLS1 cipher=AES128-SHA bits=128/128); Tue, 21 Nov 2017 13:36:35 -0800 (PST) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) client-ip=2001:4830:134:3::11; Authentication-Results: mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=hps3IEHF; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) smtp.mailfrom=qemu-devel-bounces+patch=linaro.org@nongnu.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1]:36622 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1eHGDb-0003E3-7D for patch@linaro.org; Tue, 21 Nov 2017 16:36:35 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:54032) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1eHG59-0004NA-BQ for qemu-devel@nongnu.org; Tue, 21 Nov 2017 16:27:57 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1eHG53-0007fZ-In for qemu-devel@nongnu.org; Tue, 21 Nov 2017 16:27:51 -0500 Received: from mail-wm0-x22d.google.com ([2a00:1450:400c:c09::22d]:39131) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1eHG53-0007et-50 for qemu-devel@nongnu.org; Tue, 21 Nov 2017 16:27:45 -0500 Received: by mail-wm0-x22d.google.com with SMTP id x63so6255167wmf.4 for ; Tue, 21 Nov 2017 13:27:45 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:subject:date:message-id:in-reply-to:references; bh=jWknO0RbzhLByyUe3nJCH3qKKHeeV0jgF0EpsUb2/bg=; b=hps3IEHFn4Q7miN/5wA39XXWjullwavJxzrKXkSc3YGcuOeQgxWLwnvh/V9dFFmiyY tGMtoYaVHIaDNvEEPVbWYN44fo/8bKdhkZZi3/SuMZ/K3D4/YZXv62db5qYt482SG/jg ufBo2jUPfux3ZgLz4HnDPGkqDIR7UWNhBgSsg= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references; bh=jWknO0RbzhLByyUe3nJCH3qKKHeeV0jgF0EpsUb2/bg=; b=EzDwbJmfWcZJ0YGiIiWOJ62nEcE6PFxZEgNHUqcnGS8i0lzEfz+9qa19C7GWlMQ+di TDP8JW8BQjMGshJJGb58xUBMpIAUhFNE4pX1jZz6TsLP9Fif+9xgJmAx5UEKqn071U3T t4n0NqH5T6R7IJN8X8VxqbIprXposT4vRxqD5M+5n1Qkmi64pZpCu+/1Oh3TkL9JxG+P +8DdaZLZjvcRIbghqsa9Te8xEWl3nA+G0y9ANyQXIoEYckWogS60CqqIekKeFL2Ti585 F2Z+Xh5hiDeZYDngVB0uTTC9kxiQHyk0MvJWIyxkZhCMW4GQQcFE9eITHqaYuYrUtmaj w7ng== X-Gm-Message-State: AJaThX4X1EBIAy/JcjzMeCq0Zl3mD0m838P33gu6OswefOYED2mx51lf pkMbJdyHkAdhCz3RPI86jnImtCPus3A= X-Received: by 10.28.234.80 with SMTP id i77mr2237306wmh.76.1511299662949; Tue, 21 Nov 2017 13:27:42 -0800 (PST) Received: from cloudburst.twiddle.net (70.red-37-158-60.dynamicip.rima-tde.net. [37.158.60.70]) by smtp.gmail.com with ESMTPSA id e124sm706517wmg.34.2017.11.21.13.27.41 for (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Tue, 21 Nov 2017 13:27:42 -0800 (PST) From: Richard Henderson To: qemu-devel@nongnu.org Date: Tue, 21 Nov 2017 22:25:13 +0100 Message-Id: <20171121212534.5177-6-richard.henderson@linaro.org> X-Mailer: git-send-email 2.13.6 In-Reply-To: <20171121212534.5177-1-richard.henderson@linaro.org> References: <20171121212534.5177-1-richard.henderson@linaro.org> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2a00:1450:400c:c09::22d Subject: [Qemu-devel] [PATCH v6 05/26] tcg: Add generic vector expanders X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: "Qemu-devel" Signed-off-by: Richard Henderson --- Makefile.target | 2 +- accel/tcg/tcg-runtime.h | 29 ++ tcg/tcg-gvec-desc.h | 49 ++ tcg/tcg-op-gvec.h | 152 +++++++ tcg/tcg-op.h | 1 + accel/tcg/tcg-runtime-gvec.c | 295 ++++++++++++ tcg/tcg-op-gvec.c | 1017 ++++++++++++++++++++++++++++++++++++++++++ tcg/tcg-op-vec.c | 36 +- accel/tcg/Makefile.objs | 2 +- 9 files changed, 1573 insertions(+), 10 deletions(-) create mode 100644 tcg/tcg-gvec-desc.h create mode 100644 tcg/tcg-op-gvec.h create mode 100644 accel/tcg/tcg-runtime-gvec.c create mode 100644 tcg/tcg-op-gvec.c -- 2.13.6 diff --git a/Makefile.target b/Makefile.target index 5ca758f13c..d828cec433 100644 --- a/Makefile.target +++ b/Makefile.target @@ -93,7 +93,7 @@ all: $(PROGS) stap # cpu emulator library obj-y += exec.o obj-y += accel/ -obj-$(CONFIG_TCG) += tcg/tcg.o tcg/tcg-op.o tcg/tcg-op-vec.o +obj-$(CONFIG_TCG) += tcg/tcg.o tcg/tcg-op.o tcg/tcg-op-vec.o tcg/tcg-op-gvec.o obj-$(CONFIG_TCG) += tcg/tcg-common.o tcg/optimize.o obj-$(CONFIG_TCG_INTERPRETER) += tcg/tci.o obj-$(CONFIG_TCG_INTERPRETER) += disas/tci.o diff --git a/accel/tcg/tcg-runtime.h b/accel/tcg/tcg-runtime.h index 1df17d0ba9..76ee41ce58 100644 --- a/accel/tcg/tcg-runtime.h +++ b/accel/tcg/tcg-runtime.h @@ -134,3 +134,32 @@ GEN_ATOMIC_HELPERS(xor_fetch) GEN_ATOMIC_HELPERS(xchg) #undef GEN_ATOMIC_HELPERS + +DEF_HELPER_FLAGS_3(gvec_mov, TCG_CALL_NO_RWG, void, ptr, ptr, i32) + +DEF_HELPER_FLAGS_3(gvec_dup8, TCG_CALL_NO_RWG, void, ptr, i32, i32) +DEF_HELPER_FLAGS_3(gvec_dup16, TCG_CALL_NO_RWG, void, ptr, i32, i32) +DEF_HELPER_FLAGS_3(gvec_dup32, TCG_CALL_NO_RWG, void, ptr, i32, i32) +DEF_HELPER_FLAGS_3(gvec_dup64, TCG_CALL_NO_RWG, void, ptr, i32, i64) + +DEF_HELPER_FLAGS_4(gvec_add8, TCG_CALL_NO_RWG, void, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_4(gvec_add16, TCG_CALL_NO_RWG, void, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_4(gvec_add32, TCG_CALL_NO_RWG, void, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_4(gvec_add64, TCG_CALL_NO_RWG, void, ptr, ptr, ptr, i32) + +DEF_HELPER_FLAGS_4(gvec_sub8, TCG_CALL_NO_RWG, void, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_4(gvec_sub16, TCG_CALL_NO_RWG, void, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_4(gvec_sub32, TCG_CALL_NO_RWG, void, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_4(gvec_sub64, TCG_CALL_NO_RWG, void, ptr, ptr, ptr, i32) + +DEF_HELPER_FLAGS_3(gvec_neg8, TCG_CALL_NO_RWG, void, ptr, ptr, i32) +DEF_HELPER_FLAGS_3(gvec_neg16, TCG_CALL_NO_RWG, void, ptr, ptr, i32) +DEF_HELPER_FLAGS_3(gvec_neg32, TCG_CALL_NO_RWG, void, ptr, ptr, i32) +DEF_HELPER_FLAGS_3(gvec_neg64, TCG_CALL_NO_RWG, void, ptr, ptr, i32) + +DEF_HELPER_FLAGS_3(gvec_not, TCG_CALL_NO_RWG, void, ptr, ptr, i32) +DEF_HELPER_FLAGS_4(gvec_and, TCG_CALL_NO_RWG, void, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_4(gvec_or, TCG_CALL_NO_RWG, void, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_4(gvec_xor, TCG_CALL_NO_RWG, void, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_4(gvec_andc, TCG_CALL_NO_RWG, void, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_4(gvec_orc, TCG_CALL_NO_RWG, void, ptr, ptr, ptr, i32) diff --git a/tcg/tcg-gvec-desc.h b/tcg/tcg-gvec-desc.h new file mode 100644 index 0000000000..8ba9a8168d --- /dev/null +++ b/tcg/tcg-gvec-desc.h @@ -0,0 +1,49 @@ +/* + * Generic vector operation descriptor + * + * Copyright (c) 2017 Linaro + * + * This library is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2 of the License, or (at your option) any later version. + * + * This library is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with this library; if not, see . + */ + +/* ??? These bit widths are set for ARM SVE, maxing out at 256 byte vectors. */ +#define SIMD_OPRSZ_SHIFT 0 +#define SIMD_OPRSZ_BITS 5 + +#define SIMD_MAXSZ_SHIFT (SIMD_OPRSZ_SHIFT + SIMD_OPRSZ_BITS) +#define SIMD_MAXSZ_BITS 5 + +#define SIMD_DATA_SHIFT (SIMD_MAXSZ_SHIFT + SIMD_MAXSZ_BITS) +#define SIMD_DATA_BITS (32 - SIMD_DATA_SHIFT) + +/* Create a descriptor from components. */ +uint32_t simd_desc(uint32_t oprsz, uint32_t maxsz, int32_t data); + +/* Extract the operation size from a descriptor. */ +static inline intptr_t simd_oprsz(uint32_t desc) +{ + return (extract32(desc, SIMD_OPRSZ_SHIFT, SIMD_OPRSZ_BITS) + 1) * 8; +} + +/* Extract the max vector size from a descriptor. */ +static inline intptr_t simd_maxsz(uint32_t desc) +{ + return (extract32(desc, SIMD_MAXSZ_SHIFT, SIMD_MAXSZ_BITS) + 1) * 8; +} + +/* Extract the operation-specific data from a descriptor. */ +static inline int32_t simd_data(uint32_t desc) +{ + return sextract32(desc, SIMD_DATA_SHIFT, SIMD_DATA_BITS); +} diff --git a/tcg/tcg-op-gvec.h b/tcg/tcg-op-gvec.h new file mode 100644 index 0000000000..95739946ff --- /dev/null +++ b/tcg/tcg-op-gvec.h @@ -0,0 +1,152 @@ +/* + * Generic vector operation expansion + * + * Copyright (c) 2017 Linaro + * + * This library is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2 of the License, or (at your option) any later version. + * + * This library is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with this library; if not, see . + */ + +/* + * "Generic" vectors. All operands are given as offsets from ENV, + * and therefore cannot also be allocated via tcg_global_mem_new_*. + * OPRSZ is the byte size of the vector upon which the operation is performed. + * MAXSZ is the byte size of the full vector; bytes beyond OPSZ are cleared. + * + * All sizes must be 8 or any multiple of 16. + * When OPRSZ is 8, the alignment may be 8, otherwise must be 16. + * Operands may completely, but not partially, overlap. + */ + +/* Expand a call to a gvec-style helper, with pointers to two vector + operands, and a descriptor (see tcg-gvec-desc.h). */ +typedef void (gen_helper_gvec_2)(TCGv_ptr, TCGv_ptr, TCGv_i32); +void tcg_gen_gvec_2_ool(uint32_t dofs, uint32_t aofs, + uint32_t oprsz, uint32_t maxsz, int32_t data, + gen_helper_gvec_2 *fn); + +/* Similarly, passing an extra pointer (e.g. env or float_status). */ +typedef void (gen_helper_gvec_2_ptr)(TCGv_ptr, TCGv_ptr, TCGv_ptr, TCGv_i32); +void tcg_gen_gvec_2_ptr(uint32_t dofs, uint32_t aofs, + TCGv_ptr ptr, uint32_t oprsz, uint32_t maxsz, + int32_t data, gen_helper_gvec_2_ptr *fn); + +/* Similarly, with three vector operands. */ +typedef void (gen_helper_gvec_3)(TCGv_ptr, TCGv_ptr, TCGv_ptr, TCGv_i32); +void tcg_gen_gvec_3_ool(uint32_t dofs, uint32_t aofs, uint32_t bofs, + uint32_t oprsz, uint32_t maxsz, int32_t data, + gen_helper_gvec_3 *fn); + +typedef void (gen_helper_gvec_3_ptr)(TCGv_ptr, TCGv_ptr, TCGv_ptr, + TCGv_ptr, TCGv_i32); +void tcg_gen_gvec_3_ptr(uint32_t dofs, uint32_t aofs, uint32_t bofs, + TCGv_ptr ptr, uint32_t oprsz, uint32_t maxsz, + int32_t data, gen_helper_gvec_3_ptr *fn); + +/* Expand a gvec operation. Either inline or out-of-line depending on + the actual vector size and the operations supported by the host. */ +typedef struct { + /* Expand inline as a 64-bit or 32-bit integer. + Only one of these will be non-NULL. */ + void (*fni8)(TCGv_i64, TCGv_i64); + void (*fni4)(TCGv_i32, TCGv_i32); + /* Expand inline with a host vector type. */ + void (*fniv)(unsigned, TCGv_vec, TCGv_vec); + /* Expand out-of-line helper w/descriptor. */ + gen_helper_gvec_2 *fno; + /* The opcode, if any, to which this corresponds. */ + TCGOpcode opc; + /* The vector element size, if applicable. */ + uint8_t vece; + /* Prefer i64 to v64. */ + bool prefer_i64; +} GVecGen2; + +typedef struct { + /* Expand inline as a 64-bit or 32-bit integer. + Only one of these will be non-NULL. */ + void (*fni8)(TCGv_i64, TCGv_i64, TCGv_i64); + void (*fni4)(TCGv_i32, TCGv_i32, TCGv_i32); + /* Expand inline with a host vector type. */ + void (*fniv)(unsigned, TCGv_vec, TCGv_vec, TCGv_vec); + /* Expand out-of-line helper w/descriptor. */ + gen_helper_gvec_3 *fno; + /* The opcode, if any, to which this corresponds. */ + TCGOpcode opc; + /* The vector element size, if applicable. */ + uint8_t vece; + /* Prefer i64 to v64. */ + bool prefer_i64; + /* Load dest as a 3rd source operand. */ + bool load_dest; +} GVecGen3; + +void tcg_gen_gvec_2(uint32_t dofs, uint32_t aofs, + uint32_t opsz, uint32_t clsz, const GVecGen2 *); +void tcg_gen_gvec_3(uint32_t dofs, uint32_t aofs, uint32_t bofs, + uint32_t opsz, uint32_t clsz, const GVecGen3 *); + +/* Expand a specific vector operation. */ + +void tcg_gen_gvec_mov(unsigned vece, uint32_t dofs, uint32_t aofs, + uint32_t opsz, uint32_t clsz); +void tcg_gen_gvec_not(unsigned vece, uint32_t dofs, uint32_t aofs, + uint32_t opsz, uint32_t clsz); +void tcg_gen_gvec_neg(unsigned vece, uint32_t dofs, uint32_t aofs, + uint32_t opsz, uint32_t clsz); + +void tcg_gen_gvec_add(unsigned vece, uint32_t dofs, uint32_t aofs, + uint32_t bofs, uint32_t opsz, uint32_t clsz); +void tcg_gen_gvec_sub(unsigned vece, uint32_t dofs, uint32_t aofs, + uint32_t bofs, uint32_t opsz, uint32_t clsz); + +void tcg_gen_gvec_and(unsigned vece, uint32_t dofs, uint32_t aofs, + uint32_t bofs, uint32_t opsz, uint32_t clsz); +void tcg_gen_gvec_or(unsigned vece, uint32_t dofs, uint32_t aofs, + uint32_t bofs, uint32_t opsz, uint32_t clsz); +void tcg_gen_gvec_xor(unsigned vece, uint32_t dofs, uint32_t aofs, + uint32_t bofs, uint32_t opsz, uint32_t clsz); +void tcg_gen_gvec_andc(unsigned vece, uint32_t dofs, uint32_t aofs, + uint32_t bofs, uint32_t opsz, uint32_t clsz); +void tcg_gen_gvec_orc(unsigned vece, uint32_t dofs, uint32_t aofs, + uint32_t bofs, uint32_t opsz, uint32_t clsz); + +void tcg_gen_gvec_dup_mem(unsigned vece, uint32_t dofs, uint32_t aofs, + uint32_t s, uint32_t m); +void tcg_gen_gvec_dup_i32(unsigned vece, uint32_t dofs, uint32_t s, + uint32_t m, TCGv_i32); +void tcg_gen_gvec_dup_i64(unsigned vece, uint32_t dofs, uint32_t s, + uint32_t m, TCGv_i64); + +void tcg_gen_gvec_dup8i(uint32_t dofs, uint32_t s, uint32_t m, uint8_t x); +void tcg_gen_gvec_dup16i(uint32_t dofs, uint32_t s, uint32_t m, uint16_t x); +void tcg_gen_gvec_dup32i(uint32_t dofs, uint32_t s, uint32_t m, uint32_t x); +void tcg_gen_gvec_dup64i(uint32_t dofs, uint32_t s, uint32_t m, uint64_t x); + +/* + * 64-bit vector operations. Use these when the register has been allocated + * with tcg_global_mem_new_i64, and so we cannot also address it via pointer. + * OPRSZ = MAXSZ = 8. + */ + +void tcg_gen_vec_neg8_i64(TCGv_i64 d, TCGv_i64 a); +void tcg_gen_vec_neg16_i64(TCGv_i64 d, TCGv_i64 a); +void tcg_gen_vec_neg32_i64(TCGv_i64 d, TCGv_i64 a); + +void tcg_gen_vec_add8_i64(TCGv_i64 d, TCGv_i64 a, TCGv_i64 b); +void tcg_gen_vec_add16_i64(TCGv_i64 d, TCGv_i64 a, TCGv_i64 b); +void tcg_gen_vec_add32_i64(TCGv_i64 d, TCGv_i64 a, TCGv_i64 b); + +void tcg_gen_vec_sub8_i64(TCGv_i64 d, TCGv_i64 a, TCGv_i64 b); +void tcg_gen_vec_sub16_i64(TCGv_i64 d, TCGv_i64 a, TCGv_i64 b); +void tcg_gen_vec_sub32_i64(TCGv_i64 d, TCGv_i64 a, TCGv_i64 b); diff --git a/tcg/tcg-op.h b/tcg/tcg-op.h index 9b0560e4d3..5f49785cb3 100644 --- a/tcg/tcg-op.h +++ b/tcg/tcg-op.h @@ -914,6 +914,7 @@ void tcg_gen_dup8i_vec(TCGv_vec, uint32_t); void tcg_gen_dup16i_vec(TCGv_vec, uint32_t); void tcg_gen_dup32i_vec(TCGv_vec, uint32_t); void tcg_gen_dup64i_vec(TCGv_vec, uint64_t); +void tcg_gen_dupi_vec(unsigned vece, TCGv_vec, uint64_t); void tcg_gen_movi_v64(TCGv_vec, uint64_t); void tcg_gen_movi_v128(TCGv_vec, uint64_t, uint64_t); void tcg_gen_movi_v256(TCGv_vec, uint64_t, uint64_t, uint64_t, uint64_t); diff --git a/accel/tcg/tcg-runtime-gvec.c b/accel/tcg/tcg-runtime-gvec.c new file mode 100644 index 0000000000..cd1ce12b7e --- /dev/null +++ b/accel/tcg/tcg-runtime-gvec.c @@ -0,0 +1,295 @@ +/* + * Generic vectorized operation runtime + * + * Copyright (c) 2017 Linaro + * + * This library is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2 of the License, or (at your option) any later version. + * + * This library is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with this library; if not, see . + */ + +#include "qemu/osdep.h" +#include "qemu/host-utils.h" +#include "cpu.h" +#include "exec/helper-proto.h" +#include "tcg-gvec-desc.h" + + +/* Virtually all hosts support 16-byte vectors. Those that don't can emulate + them via GCC's generic vector extension. This turns out to be simpler and + more reliable than getting the compiler to autovectorize. + + In tcg-op-gvec.c, we asserted that both the size and alignment + of the data are multiples of 16. */ + +typedef uint8_t vec8 __attribute__((vector_size(16))); +typedef uint16_t vec16 __attribute__((vector_size(16))); +typedef uint32_t vec32 __attribute__((vector_size(16))); +typedef uint64_t vec64 __attribute__((vector_size(16))); + +static inline void clear_high(void *d, intptr_t oprsz, uint32_t desc) +{ + intptr_t maxsz = simd_maxsz(desc); + intptr_t i; + + if (unlikely(maxsz > oprsz)) { + for (i = oprsz; i < maxsz; i += sizeof(uint64_t)) { + *(uint64_t *)(d + i) = 0; + } + } +} + +void HELPER(gvec_add8)(void *d, void *a, void *b, uint32_t desc) +{ + intptr_t oprsz = simd_oprsz(desc); + intptr_t i; + + for (i = 0; i < oprsz; i += sizeof(vec8)) { + *(vec8 *)(d + i) = *(vec8 *)(a + i) + *(vec8 *)(b + i); + } + clear_high(d, oprsz, desc); +} + +void HELPER(gvec_add16)(void *d, void *a, void *b, uint32_t desc) +{ + intptr_t oprsz = simd_oprsz(desc); + intptr_t i; + + for (i = 0; i < oprsz; i += sizeof(vec16)) { + *(vec16 *)(d + i) = *(vec16 *)(a + i) + *(vec16 *)(b + i); + } + clear_high(d, oprsz, desc); +} + +void HELPER(gvec_add32)(void *d, void *a, void *b, uint32_t desc) +{ + intptr_t oprsz = simd_oprsz(desc); + intptr_t i; + + for (i = 0; i < oprsz; i += sizeof(vec32)) { + *(vec32 *)(d + i) = *(vec32 *)(a + i) + *(vec32 *)(b + i); + } + clear_high(d, oprsz, desc); +} + +void HELPER(gvec_add64)(void *d, void *a, void *b, uint32_t desc) +{ + intptr_t oprsz = simd_oprsz(desc); + intptr_t i; + + for (i = 0; i < oprsz; i += sizeof(vec64)) { + *(vec64 *)(d + i) = *(vec64 *)(a + i) + *(vec64 *)(b + i); + } + clear_high(d, oprsz, desc); +} + +void HELPER(gvec_sub8)(void *d, void *a, void *b, uint32_t desc) +{ + intptr_t oprsz = simd_oprsz(desc); + intptr_t i; + + for (i = 0; i < oprsz; i += sizeof(vec8)) { + *(vec8 *)(d + i) = *(vec8 *)(a + i) - *(vec8 *)(b + i); + } + clear_high(d, oprsz, desc); +} + +void HELPER(gvec_sub16)(void *d, void *a, void *b, uint32_t desc) +{ + intptr_t oprsz = simd_oprsz(desc); + intptr_t i; + + for (i = 0; i < oprsz; i += sizeof(vec16)) { + *(vec16 *)(d + i) = *(vec16 *)(a + i) - *(vec16 *)(b + i); + } + clear_high(d, oprsz, desc); +} + +void HELPER(gvec_sub32)(void *d, void *a, void *b, uint32_t desc) +{ + intptr_t oprsz = simd_oprsz(desc); + intptr_t i; + + for (i = 0; i < oprsz; i += sizeof(vec32)) { + *(vec32 *)(d + i) = *(vec32 *)(a + i) - *(vec32 *)(b + i); + } + clear_high(d, oprsz, desc); +} + +void HELPER(gvec_sub64)(void *d, void *a, void *b, uint32_t desc) +{ + intptr_t oprsz = simd_oprsz(desc); + intptr_t i; + + for (i = 0; i < oprsz; i += sizeof(vec64)) { + *(vec64 *)(d + i) = *(vec64 *)(a + i) - *(vec64 *)(b + i); + } + clear_high(d, oprsz, desc); +} + +void HELPER(gvec_neg8)(void *d, void *a, uint32_t desc) +{ + intptr_t oprsz = simd_oprsz(desc); + intptr_t i; + + for (i = 0; i < oprsz; i += sizeof(vec8)) { + *(vec8 *)(d + i) = -*(vec8 *)(a + i); + } + clear_high(d, oprsz, desc); +} + +void HELPER(gvec_neg16)(void *d, void *a, uint32_t desc) +{ + intptr_t oprsz = simd_oprsz(desc); + intptr_t i; + + for (i = 0; i < oprsz; i += sizeof(vec16)) { + *(vec16 *)(d + i) = -*(vec16 *)(a + i); + } + clear_high(d, oprsz, desc); +} + +void HELPER(gvec_neg32)(void *d, void *a, uint32_t desc) +{ + intptr_t oprsz = simd_oprsz(desc); + intptr_t i; + + for (i = 0; i < oprsz; i += sizeof(vec32)) { + *(vec32 *)(d + i) = -*(vec32 *)(a + i); + } + clear_high(d, oprsz, desc); +} + +void HELPER(gvec_neg64)(void *d, void *a, uint32_t desc) +{ + intptr_t oprsz = simd_oprsz(desc); + intptr_t i; + + for (i = 0; i < oprsz; i += sizeof(vec64)) { + *(vec64 *)(d + i) = -*(vec64 *)(a + i); + } + clear_high(d, oprsz, desc); +} + +void HELPER(gvec_mov)(void *d, void *a, uint32_t desc) +{ + intptr_t oprsz = simd_oprsz(desc); + + memcpy(d, a, oprsz); + clear_high(d, oprsz, desc); +} + +void HELPER(gvec_dup64)(void *d, uint32_t desc, uint64_t c) +{ + intptr_t oprsz = simd_oprsz(desc); + intptr_t i; + + if (c == 0) { + oprsz = 0; + } else { + for (i = 0; i < oprsz; i += sizeof(uint64_t)) { + *(uint64_t *)(d + i) = c; + } + } + clear_high(d, oprsz, desc); +} + +void HELPER(gvec_dup32)(void *d, uint32_t desc, uint32_t c) +{ + intptr_t oprsz = simd_oprsz(desc); + intptr_t i; + + if (c == 0) { + oprsz = 0; + } else { + for (i = 0; i < oprsz; i += sizeof(uint32_t)) { + *(uint32_t *)(d + i) = c; + } + } + clear_high(d, oprsz, desc); +} + +void HELPER(gvec_dup16)(void *d, uint32_t desc, uint32_t c) +{ + HELPER(gvec_dup32)(d, desc, 0x00010001 * (c & 0xffff)); +} + +void HELPER(gvec_dup8)(void *d, uint32_t desc, uint32_t c) +{ + HELPER(gvec_dup32)(d, desc, 0x01010101 * (c & 0xff)); +} + +void HELPER(gvec_not)(void *d, void *a, uint32_t desc) +{ + intptr_t oprsz = simd_oprsz(desc); + intptr_t i; + + for (i = 0; i < oprsz; i += sizeof(vec64)) { + *(vec64 *)(d + i) = ~*(vec64 *)(a + i); + } + clear_high(d, oprsz, desc); +} + +void HELPER(gvec_and)(void *d, void *a, void *b, uint32_t desc) +{ + intptr_t oprsz = simd_oprsz(desc); + intptr_t i; + + for (i = 0; i < oprsz; i += sizeof(vec64)) { + *(vec64 *)(d + i) = *(vec64 *)(a + i) & *(vec64 *)(b + i); + } + clear_high(d, oprsz, desc); +} + +void HELPER(gvec_or)(void *d, void *a, void *b, uint32_t desc) +{ + intptr_t oprsz = simd_oprsz(desc); + intptr_t i; + + for (i = 0; i < oprsz; i += sizeof(vec64)) { + *(vec64 *)(d + i) = *(vec64 *)(a + i) | *(vec64 *)(b + i); + } + clear_high(d, oprsz, desc); +} + +void HELPER(gvec_xor)(void *d, void *a, void *b, uint32_t desc) +{ + intptr_t oprsz = simd_oprsz(desc); + intptr_t i; + + for (i = 0; i < oprsz; i += sizeof(vec64)) { + *(vec64 *)(d + i) = *(vec64 *)(a + i) ^ *(vec64 *)(b + i); + } + clear_high(d, oprsz, desc); +} + +void HELPER(gvec_andc)(void *d, void *a, void *b, uint32_t desc) +{ + intptr_t oprsz = simd_oprsz(desc); + intptr_t i; + + for (i = 0; i < oprsz; i += sizeof(vec64)) { + *(vec64 *)(d + i) = *(vec64 *)(a + i) &~ *(vec64 *)(b + i); + } + clear_high(d, oprsz, desc); +} + +void HELPER(gvec_orc)(void *d, void *a, void *b, uint32_t desc) +{ + intptr_t oprsz = simd_oprsz(desc); + intptr_t i; + + for (i = 0; i < oprsz; i += sizeof(vec64)) { + *(vec64 *)(d + i) = *(vec64 *)(a + i) |~ *(vec64 *)(b + i); + } + clear_high(d, oprsz, desc); +} diff --git a/tcg/tcg-op-gvec.c b/tcg/tcg-op-gvec.c new file mode 100644 index 0000000000..925c293f9c --- /dev/null +++ b/tcg/tcg-op-gvec.c @@ -0,0 +1,1017 @@ +/* + * Generic vector operation expansion + * + * Copyright (c) 2017 Linaro + * + * This library is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2 of the License, or (at your option) any later version. + * + * This library is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with this library; if not, see . + */ + +#include "qemu/osdep.h" +#include "qemu-common.h" +#include "tcg.h" +#include "tcg-op.h" +#include "tcg-op-gvec.h" +#include "tcg-gvec-desc.h" + +#define REP8(x) ((x) * 0x0101010101010101ull) +#define REP16(x) ((x) * 0x0001000100010001ull) + +#define MAX_UNROLL 4 + +/* Verify vector size and alignment rules. OFS should be the OR of all + of the operand offsets so that we can check them all at once. */ +static void check_size_align(uint32_t oprsz, uint32_t maxsz, uint32_t ofs) +{ + uint32_t align = maxsz > 16 || oprsz >= 16 ? 15 : 7; + tcg_debug_assert(oprsz > 0); + tcg_debug_assert(oprsz <= maxsz); + tcg_debug_assert((oprsz & align) == 0); + tcg_debug_assert((maxsz & align) == 0); + tcg_debug_assert((ofs & align) == 0); +} + +/* Verify vector overlap rules for two operands. */ +static void check_overlap_2(uint32_t d, uint32_t a, uint32_t s) +{ + tcg_debug_assert(d == a || d + s <= a || a + s <= d); +} + +/* Verify vector overlap rules for three operands. */ +static void check_overlap_3(uint32_t d, uint32_t a, uint32_t b, uint32_t s) +{ + check_overlap_2(d, a, s); + check_overlap_2(d, b, s); + check_overlap_2(a, b, s); +} + +/* Create a descriptor from components. */ +uint32_t simd_desc(uint32_t oprsz, uint32_t maxsz, int32_t data) +{ + uint32_t desc = 0; + + assert(oprsz % 8 == 0 && oprsz <= (8 << SIMD_OPRSZ_BITS)); + assert(maxsz % 8 == 0 && maxsz <= (8 << SIMD_MAXSZ_BITS)); + assert(data == sextract32(data, 0, SIMD_DATA_BITS)); + + oprsz = (oprsz / 8) - 1; + maxsz = (maxsz / 8) - 1; + desc = deposit32(desc, SIMD_OPRSZ_SHIFT, SIMD_OPRSZ_BITS, oprsz); + desc = deposit32(desc, SIMD_MAXSZ_SHIFT, SIMD_MAXSZ_BITS, maxsz); + desc = deposit32(desc, SIMD_DATA_SHIFT, SIMD_DATA_BITS, data); + + return desc; +} + +/* Generate a call to a gvec-style helper with two vector operands. */ +void tcg_gen_gvec_2_ool(uint32_t dofs, uint32_t aofs, + uint32_t oprsz, uint32_t maxsz, int32_t data, + gen_helper_gvec_2 *fn) +{ + TCGv_ptr a0, a1; + TCGv_i32 desc = tcg_const_i32(simd_desc(oprsz, maxsz, data)); + + a0 = tcg_temp_new_ptr(); + a1 = tcg_temp_new_ptr(); + + tcg_gen_addi_ptr(a0, cpu_env, dofs); + tcg_gen_addi_ptr(a1, cpu_env, aofs); + + fn(a0, a1, desc); + + tcg_temp_free_ptr(a0); + tcg_temp_free_ptr(a1); + tcg_temp_free_i32(desc); +} + +/* Generate a call to a gvec-style helper with three vector operands. */ +void tcg_gen_gvec_3_ool(uint32_t dofs, uint32_t aofs, uint32_t bofs, + uint32_t oprsz, uint32_t maxsz, int32_t data, + gen_helper_gvec_3 *fn) +{ + TCGv_ptr a0, a1, a2; + TCGv_i32 desc = tcg_const_i32(simd_desc(oprsz, maxsz, data)); + + a0 = tcg_temp_new_ptr(); + a1 = tcg_temp_new_ptr(); + a2 = tcg_temp_new_ptr(); + + tcg_gen_addi_ptr(a0, cpu_env, dofs); + tcg_gen_addi_ptr(a1, cpu_env, aofs); + tcg_gen_addi_ptr(a2, cpu_env, bofs); + + fn(a0, a1, a2, desc); + + tcg_temp_free_ptr(a0); + tcg_temp_free_ptr(a1); + tcg_temp_free_ptr(a2); + tcg_temp_free_i32(desc); +} + +/* Generate a call to a gvec-style helper with three vector operands + and an extra pointer operand. */ +void tcg_gen_gvec_2_ptr(uint32_t dofs, uint32_t aofs, + TCGv_ptr ptr, uint32_t oprsz, uint32_t maxsz, + int32_t data, gen_helper_gvec_2_ptr *fn) +{ + TCGv_ptr a0, a1; + TCGv_i32 desc = tcg_const_i32(simd_desc(oprsz, maxsz, data)); + + a0 = tcg_temp_new_ptr(); + a1 = tcg_temp_new_ptr(); + + tcg_gen_addi_ptr(a0, cpu_env, dofs); + tcg_gen_addi_ptr(a1, cpu_env, aofs); + + fn(a0, a1, ptr, desc); + + tcg_temp_free_ptr(a0); + tcg_temp_free_ptr(a1); + tcg_temp_free_i32(desc); +} + +/* Generate a call to a gvec-style helper with three vector operands + and an extra pointer operand. */ +void tcg_gen_gvec_3_ptr(uint32_t dofs, uint32_t aofs, uint32_t bofs, + TCGv_ptr ptr, uint32_t oprsz, uint32_t maxsz, + int32_t data, gen_helper_gvec_3_ptr *fn) +{ + TCGv_ptr a0, a1, a2; + TCGv_i32 desc = tcg_const_i32(simd_desc(oprsz, maxsz, data)); + + a0 = tcg_temp_new_ptr(); + a1 = tcg_temp_new_ptr(); + a2 = tcg_temp_new_ptr(); + + tcg_gen_addi_ptr(a0, cpu_env, dofs); + tcg_gen_addi_ptr(a1, cpu_env, aofs); + tcg_gen_addi_ptr(a2, cpu_env, bofs); + + fn(a0, a1, a2, ptr, desc); + + tcg_temp_free_ptr(a0); + tcg_temp_free_ptr(a1); + tcg_temp_free_ptr(a2); + tcg_temp_free_i32(desc); +} + +/* Return true if we want to implement something of OPRSZ bytes + in units of LNSZ. This limits the expansion of inline code. */ +static inline bool check_size_impl(uint32_t oprsz, uint32_t lnsz) +{ + uint32_t lnct = oprsz / lnsz; + return lnct >= 1 && lnct <= MAX_UNROLL; +} + +static void expand_clr(uint32_t dofs, uint32_t maxsz); + +/* Set OPRSZ bytes at DOFS to replications of IN or IN_C. */ +static void do_dup_i32(unsigned vece, uint32_t dofs, uint32_t oprsz, + uint32_t maxsz, TCGv_i32 in, uint32_t in_c, + void (*ool)(TCGv_ptr, TCGv_i32, TCGv_i32)) +{ + TCGv_vec t_vec; + uint32_t i; + + if (TCG_TARGET_HAS_v256 && check_size_impl(oprsz, 32)) { + t_vec = tcg_temp_new_vec(TCG_TYPE_V256); + } else if (TCG_TARGET_HAS_v128 && check_size_impl(oprsz, 16)) { + t_vec = tcg_temp_new_vec(TCG_TYPE_V128); + } else if (TCG_TARGET_HAS_v64 && check_size_impl(oprsz, 8)) { + t_vec = tcg_temp_new_vec(TCG_TYPE_V64); + } else { + TCGv_i32 t_i32 = in ? in : tcg_const_i32(in_c); + + if (check_size_impl(oprsz, 4)) { + for (i = 0; i < oprsz; i += 4) { + tcg_gen_st_i32(t_i32, cpu_env, dofs + i); + } + if (in == NULL) { + tcg_temp_free_i32(t_i32); + } + goto done; + } else { + TCGv_ptr a0 = tcg_temp_new_ptr(); + TCGv_i32 desc = tcg_const_i32(simd_desc(oprsz, maxsz, 0)); + + tcg_gen_addi_ptr(a0, cpu_env, dofs); + ool(a0, desc, t_i32); + + tcg_temp_free_ptr(a0); + tcg_temp_free_i32(desc); + if (in == NULL) { + tcg_temp_free_i32(t_i32); + } + return; + } + } + + if (in) { + tcg_gen_dup_i32_vec(vece, t_vec, in); + } else { + tcg_gen_dup32i_vec(t_vec, in_c); + } + + i = 0; + if (TCG_TARGET_HAS_v256) { + for (; i + 32 <= oprsz; i += 32) { + tcg_gen_stl_vec(t_vec, cpu_env, dofs + i, TCG_TYPE_V256); + } + } + if (TCG_TARGET_HAS_v128) { + for (; i + 16 <= oprsz; i += 16) { + tcg_gen_stl_vec(t_vec, cpu_env, dofs + i, TCG_TYPE_V128); + } + } + if (TCG_TARGET_HAS_v64) { + for (; i < oprsz; i += 8) { + tcg_gen_stl_vec(t_vec, cpu_env, dofs + i, TCG_TYPE_V64); + } + } + + done: + tcg_debug_assert(i == oprsz); + if (i < maxsz) { + expand_clr(dofs + i, maxsz - i); + } +} + +/* Likewise, but with 64-bit quantities. */ +static void do_dup_i64(unsigned vece, uint32_t dofs, uint32_t oprsz, + uint32_t maxsz, TCGv_i64 in, uint64_t in_c) +{ + TCGv_vec t_vec; + uint32_t i; + + if (TCG_TARGET_HAS_v256 && check_size_impl(oprsz, 32)) { + t_vec = tcg_temp_new_vec(TCG_TYPE_V256); + } else if (TCG_TARGET_HAS_v128 && check_size_impl(oprsz, 16)) { + t_vec = tcg_temp_new_vec(TCG_TYPE_V128); + } else if (TCG_TARGET_HAS_v64 && TCG_TARGET_REG_BITS == 32 + && check_size_impl(oprsz, 8)) { + t_vec = tcg_temp_new_vec(TCG_TYPE_V64); + } else { + TCGv_i64 t_i64 = in ? in : tcg_const_i64(in_c); + + if (check_size_impl(oprsz, 8)) { + for (i = 0; i < oprsz; i += 8) { + tcg_gen_st_i64(t_i64, cpu_env, dofs + i); + } + if (in == NULL) { + tcg_temp_free_i64(t_i64); + } + goto done; + } else { + TCGv_ptr a0 = tcg_temp_new_ptr(); + TCGv_i32 desc = tcg_const_i32(simd_desc(oprsz, maxsz, 0)); + + tcg_gen_addi_ptr(a0, cpu_env, dofs); + gen_helper_gvec_dup64(a0, desc, t_i64); + + tcg_temp_free_ptr(a0); + tcg_temp_free_i32(desc); + if (in == NULL) { + tcg_temp_free_i64(t_i64); + } + return; + } + } + + if (in) { + tcg_gen_dup_i64_vec(vece, t_vec, in); + } else { + tcg_gen_dup64i_vec(t_vec, in_c); + } + + i = 0; + if (TCG_TARGET_HAS_v256) { + for (; i + 32 <= oprsz; i += 32) { + tcg_gen_stl_vec(t_vec, cpu_env, dofs + i, TCG_TYPE_V256); + } + } + if (TCG_TARGET_HAS_v128) { + for (; i + 16 <= oprsz; i += 16) { + tcg_gen_stl_vec(t_vec, cpu_env, dofs + i, TCG_TYPE_V128); + } + } + if (TCG_TARGET_HAS_v64) { + for (; i < oprsz; i += 8) { + tcg_gen_stl_vec(t_vec, cpu_env, dofs + i, TCG_TYPE_V64); + } + } + + done: + tcg_debug_assert(i == oprsz); + if (i < maxsz) { + expand_clr(dofs + i, maxsz - i); + } +} + +/* Likewise, but with zero. */ +static void expand_clr(uint32_t dofs, uint32_t maxsz) +{ + if (TCG_TARGET_REG_BITS == 64) { + do_dup_i64(MO_64, dofs, maxsz, maxsz, NULL, 0); + } else { + do_dup_i32(MO_32, dofs, maxsz, maxsz, NULL, 0, gen_helper_gvec_dup32); + } +} + +/* Expand OPSZ bytes worth of two-operand operations using i32 elements. */ +static void expand_2_i32(uint32_t dofs, uint32_t aofs, uint32_t opsz, + void (*fni)(TCGv_i32, TCGv_i32)) +{ + TCGv_i32 t0 = tcg_temp_new_i32(); + uint32_t i; + + for (i = 0; i < opsz; i += 4) { + tcg_gen_ld_i32(t0, cpu_env, aofs + i); + fni(t0, t0); + tcg_gen_st_i32(t0, cpu_env, dofs + i); + } + tcg_temp_free_i32(t0); +} + +/* Expand OPSZ bytes worth of three-operand operations using i32 elements. */ +static void expand_3_i32(uint32_t dofs, uint32_t aofs, + uint32_t bofs, uint32_t opsz, bool load_dest, + void (*fni)(TCGv_i32, TCGv_i32, TCGv_i32)) +{ + TCGv_i32 t0 = tcg_temp_new_i32(); + TCGv_i32 t1 = tcg_temp_new_i32(); + TCGv_i32 t2 = tcg_temp_new_i32(); + uint32_t i; + + for (i = 0; i < opsz; i += 4) { + tcg_gen_ld_i32(t0, cpu_env, aofs + i); + tcg_gen_ld_i32(t1, cpu_env, bofs + i); + if (load_dest) { + tcg_gen_ld_i32(t2, cpu_env, dofs + i); + } + fni(t2, t0, t1); + tcg_gen_st_i32(t2, cpu_env, dofs + i); + } + tcg_temp_free_i32(t2); + tcg_temp_free_i32(t1); + tcg_temp_free_i32(t0); +} + +/* Expand OPSZ bytes worth of two-operand operations using i64 elements. */ +static void expand_2_i64(uint32_t dofs, uint32_t aofs, uint32_t opsz, + void (*fni)(TCGv_i64, TCGv_i64)) +{ + TCGv_i64 t0 = tcg_temp_new_i64(); + uint32_t i; + + for (i = 0; i < opsz; i += 8) { + tcg_gen_ld_i64(t0, cpu_env, aofs + i); + fni(t0, t0); + tcg_gen_st_i64(t0, cpu_env, dofs + i); + } + tcg_temp_free_i64(t0); +} + +/* Expand OPSZ bytes worth of three-operand operations using i64 elements. */ +static void expand_3_i64(uint32_t dofs, uint32_t aofs, + uint32_t bofs, uint32_t opsz, bool load_dest, + void (*fni)(TCGv_i64, TCGv_i64, TCGv_i64)) +{ + TCGv_i64 t0 = tcg_temp_new_i64(); + TCGv_i64 t1 = tcg_temp_new_i64(); + TCGv_i64 t2 = tcg_temp_new_i64(); + uint32_t i; + + for (i = 0; i < opsz; i += 8) { + tcg_gen_ld_i64(t0, cpu_env, aofs + i); + tcg_gen_ld_i64(t1, cpu_env, bofs + i); + if (load_dest) { + tcg_gen_ld_i64(t2, cpu_env, dofs + i); + } + fni(t2, t0, t1); + tcg_gen_st_i64(t2, cpu_env, dofs + i); + } + tcg_temp_free_i64(t2); + tcg_temp_free_i64(t1); + tcg_temp_free_i64(t0); +} + +/* Expand OPSZ bytes worth of two-operand operations using host vectors. */ +static void expand_2_vec(unsigned vece, uint32_t dofs, uint32_t aofs, + uint32_t opsz, uint32_t tysz, TCGType type, + void (*fni)(unsigned, TCGv_vec, TCGv_vec)) +{ + TCGv_vec t0 = tcg_temp_new_vec(type); + uint32_t i; + + for (i = 0; i < opsz; i += tysz) { + tcg_gen_ld_vec(t0, cpu_env, aofs + i); + fni(vece, t0, t0); + tcg_gen_st_vec(t0, cpu_env, dofs + i); + } + tcg_temp_free_vec(t0); +} + +/* Expand OPSZ bytes worth of three-operand operations using host vectors. */ +static void expand_3_vec(unsigned vece, uint32_t dofs, uint32_t aofs, + uint32_t bofs, uint32_t opsz, + uint32_t tysz, TCGType type, bool load_dest, + void (*fni)(unsigned, TCGv_vec, TCGv_vec, TCGv_vec)) +{ + TCGv_vec t0 = tcg_temp_new_vec(type); + TCGv_vec t1 = tcg_temp_new_vec(type); + TCGv_vec t2 = tcg_temp_new_vec(type); + uint32_t i; + + for (i = 0; i < opsz; i += tysz) { + tcg_gen_ld_vec(t0, cpu_env, aofs + i); + tcg_gen_ld_vec(t1, cpu_env, bofs + i); + if (load_dest) { + tcg_gen_ld_vec(t2, cpu_env, dofs + i); + } + fni(vece, t2, t0, t1); + tcg_gen_st_vec(t2, cpu_env, dofs + i); + } + tcg_temp_free_vec(t2); + tcg_temp_free_vec(t1); + tcg_temp_free_vec(t0); +} + +/* Expand a vector two-operand operation. */ +void tcg_gen_gvec_2(uint32_t dofs, uint32_t aofs, + uint32_t oprsz, uint32_t maxsz, const GVecGen2 *g) +{ + check_size_align(oprsz, maxsz, dofs | aofs); + check_overlap_2(dofs, aofs, maxsz); + + /* Quick check for sizes we won't support inline. */ + if (oprsz > MAX_UNROLL * 32 || maxsz > MAX_UNROLL * 32) { + goto do_ool; + } + + /* Recall that ARM SVE allows vector sizes that are not a power of 2. + Expand with successively smaller host vector sizes. The intent is + that e.g. oprsz == 80 would be expanded with 2x32 + 1x16. */ + /* ??? For maxsz > oprsz, the host may be able to use an op-sized + operation, zeroing the balance of the register. We can then + use a cl-sized store to implement the clearing without an extra + store operation. This is true for aarch64 and x86_64 hosts. */ + + if (TCG_TARGET_HAS_v256 && check_size_impl(oprsz, 32)) { + uint32_t done = QEMU_ALIGN_DOWN(oprsz, 32); + expand_2_vec(g->vece, dofs, aofs, done, 32, TCG_TYPE_V256, g->fniv); + dofs += done; + aofs += done; + oprsz -= done; + maxsz -= done; + } + + if (TCG_TARGET_HAS_v128 && check_size_impl(oprsz, 16)) { + uint32_t done = QEMU_ALIGN_DOWN(oprsz, 16); + expand_2_vec(g->vece, dofs, aofs, done, 16, TCG_TYPE_V128, g->fniv); + dofs += done; + aofs += done; + oprsz -= done; + maxsz -= done; + } + + if (check_size_impl(oprsz, 8)) { + uint32_t done = QEMU_ALIGN_DOWN(oprsz, 8); + if (TCG_TARGET_HAS_v64 && !g->prefer_i64) { + expand_2_vec(g->vece, dofs, aofs, done, 8, TCG_TYPE_V64, g->fniv); + } else if (g->fni8) { + expand_2_i64(dofs, aofs, done, g->fni8); + } else { + done = 0; + } + dofs += done; + aofs += done; + oprsz -= done; + maxsz -= done; + } + + if (g->fni4 && check_size_impl(oprsz, 4)) { + uint32_t done = QEMU_ALIGN_DOWN(oprsz, 4); + expand_2_i32(dofs, aofs, done, g->fni4); + dofs += done; + aofs += done; + oprsz -= done; + maxsz -= done; + } + + if (oprsz == 0) { + if (maxsz != 0) { + expand_clr(dofs, maxsz); + } + return; + } + + do_ool: + tcg_gen_gvec_2_ool(dofs, aofs, oprsz, maxsz, 0, g->fno); +} + +/* Expand a vector three-operand operation. */ +void tcg_gen_gvec_3(uint32_t dofs, uint32_t aofs, uint32_t bofs, + uint32_t oprsz, uint32_t maxsz, const GVecGen3 *g) +{ + check_size_align(oprsz, maxsz, dofs | aofs | bofs); + check_overlap_3(dofs, aofs, bofs, maxsz); + + /* Quick check for sizes we won't support inline. */ + if (oprsz > MAX_UNROLL * 32 || maxsz > MAX_UNROLL * 32) { + goto do_ool; + } + + /* Recall that ARM SVE allows vector sizes that are not a power of 2. + Expand with successively smaller host vector sizes. The intent is + that e.g. oprsz == 80 would be expanded with 2x32 + 1x16. */ + /* ??? For maxsz > oprsz, the host may be able to use an op-sized + operation, zeroing the balance of the register. We can then + use a cl-sized store to implement the clearing without an extra + store operation. This is true for aarch64 and x86_64 hosts. */ + + if (TCG_TARGET_HAS_v256 && check_size_impl(oprsz, 32)) { + uint32_t done = QEMU_ALIGN_DOWN(oprsz, 32); + expand_3_vec(g->vece, dofs, aofs, bofs, done, 32, TCG_TYPE_V256, + g->load_dest, g->fniv); + dofs += done; + aofs += done; + bofs += done; + oprsz -= done; + maxsz -= done; + } + + if (TCG_TARGET_HAS_v128 && check_size_impl(oprsz, 16)) { + uint32_t done = QEMU_ALIGN_DOWN(oprsz, 16); + expand_3_vec(g->vece, dofs, aofs, bofs, done, 16, TCG_TYPE_V128, + g->load_dest, g->fniv); + dofs += done; + aofs += done; + bofs += done; + oprsz -= done; + maxsz -= done; + } + + if (check_size_impl(oprsz, 8)) { + uint32_t done = QEMU_ALIGN_DOWN(oprsz, 8); + if (TCG_TARGET_HAS_v64 && !g->prefer_i64) { + expand_3_vec(g->vece, dofs, aofs, bofs, done, 8, TCG_TYPE_V64, + g->load_dest, g->fniv); + } else if (g->fni8) { + expand_3_i64(dofs, aofs, bofs, done, g->load_dest, g->fni8); + } else { + done = 0; + } + dofs += done; + aofs += done; + bofs += done; + oprsz -= done; + maxsz -= done; + } + + if (g->fni4 && check_size_impl(oprsz, 4)) { + uint32_t done = QEMU_ALIGN_DOWN(oprsz, 4); + expand_3_i32(dofs, aofs, bofs, done, g->load_dest, g->fni4); + dofs += done; + aofs += done; + bofs += done; + oprsz -= done; + maxsz -= done; + } + + if (oprsz == 0) { + if (maxsz != 0) { + expand_clr(dofs, maxsz); + } + return; + } + + do_ool: + tcg_gen_gvec_3_ool(dofs, aofs, bofs, oprsz, maxsz, 0, g->fno); +} + +/* + * Expand specific vector operations. + */ + +static void vec_mov2(unsigned vece, TCGv_vec a, TCGv_vec b) +{ + tcg_gen_mov_vec(a, b); +} + +void tcg_gen_gvec_mov(unsigned vece, uint32_t dofs, uint32_t aofs, + uint32_t opsz, uint32_t maxsz) +{ + static const GVecGen2 g = { + .fni8 = tcg_gen_mov_i64, + .fniv = vec_mov2, + .fno = gen_helper_gvec_mov, + .prefer_i64 = TCG_TARGET_REG_BITS == 64, + }; + tcg_gen_gvec_2(dofs, aofs, opsz, maxsz, &g); +} + +void tcg_gen_gvec_dup_i32(unsigned vece, uint32_t dofs, uint32_t oprsz, + uint32_t maxsz, TCGv_i32 in) +{ + typedef void dup_fn(TCGv_ptr, TCGv_i32, TCGv_i32); + static dup_fn * const fns[3] = { + gen_helper_gvec_dup8, + gen_helper_gvec_dup16, + gen_helper_gvec_dup32 + }; + + check_size_align(oprsz, maxsz, dofs); + tcg_debug_assert(vece <= MO_32); + do_dup_i32(vece, dofs, oprsz, maxsz, in, 0, fns[vece]); +} + +void tcg_gen_gvec_dup_i64(unsigned vece, uint32_t dofs, uint32_t oprsz, + uint32_t maxsz, TCGv_i64 in) +{ + check_size_align(oprsz, maxsz, dofs); + tcg_debug_assert(vece <= MO_64); + if (vece <= MO_32) { + /* This works for both host register sizes. */ + tcg_gen_gvec_dup_i32(vece, dofs, oprsz, maxsz, (TCGv_i32)in); + } else { + do_dup_i64(vece, dofs, oprsz, maxsz, in, 0); + } +} + +void tcg_gen_gvec_dup_mem(unsigned vece, uint32_t dofs, uint32_t aofs, + uint32_t oprsz, uint32_t maxsz) +{ + tcg_debug_assert(vece <= MO_64); + if (vece <= MO_32) { + TCGv_i32 in = tcg_temp_new_i32(); + switch (vece) { + case MO_8: + tcg_gen_ld8u_i32(in, cpu_env, aofs); + break; + case MO_16: + tcg_gen_ld16u_i32(in, cpu_env, aofs); + break; + case MO_32: + tcg_gen_ld_i32(in, cpu_env, aofs); + break; + } + tcg_gen_gvec_dup_i32(vece, dofs, oprsz, maxsz, in); + tcg_temp_free_i32(in); + } else { + TCGv_i64 in = tcg_temp_new_i64(); + tcg_gen_ld_i64(in, cpu_env, aofs); + tcg_gen_gvec_dup_i64(MO_64, dofs, oprsz, maxsz, in); + tcg_temp_free_i64(in); + } +} + +void tcg_gen_gvec_dup64i(uint32_t dofs, uint32_t oprsz, + uint32_t maxsz, uint64_t x) +{ + check_size_align(oprsz, maxsz, dofs); + do_dup_i64(MO_64, dofs, oprsz, maxsz, NULL, x); +} + +void tcg_gen_gvec_dup32i(uint32_t dofs, uint32_t oprsz, + uint32_t maxsz, uint32_t x) +{ + if (TCG_TARGET_REG_BITS == 64) { + do_dup_i64(MO_64, dofs, oprsz, maxsz, NULL, deposit64(x, 32, 32, x)); + } else { + do_dup_i32(MO_32, dofs, oprsz, maxsz, NULL, x, gen_helper_gvec_dup32); + } +} + +void tcg_gen_gvec_dup16i(uint32_t dofs, uint32_t oprsz, + uint32_t maxsz, uint16_t x) +{ + tcg_gen_gvec_dup32i(dofs, oprsz, maxsz, 0x00010001 * x); +} + +void tcg_gen_gvec_dup8i(uint32_t dofs, uint32_t oprsz, + uint32_t maxsz, uint8_t x) +{ + tcg_gen_gvec_dup32i(dofs, oprsz, maxsz, 0x01010101 * x); +} + +void tcg_gen_gvec_not(unsigned vece, uint32_t dofs, uint32_t aofs, + uint32_t opsz, uint32_t maxsz) +{ + static const GVecGen2 g = { + .fni8 = tcg_gen_not_i64, + .fniv = tcg_gen_not_vec, + .fno = gen_helper_gvec_not, + .prefer_i64 = TCG_TARGET_REG_BITS == 64, + }; + tcg_gen_gvec_2(dofs, aofs, opsz, maxsz, &g); +} + +/* Perform a vector addition using normal addition and a mask. The mask + should be the sign bit of each lane. This 6-operation form is more + efficient than separate additions when there are 4 or more lanes in + the 64-bit operation. */ +static void gen_addv_mask(TCGv_i64 d, TCGv_i64 a, TCGv_i64 b, TCGv_i64 m) +{ + TCGv_i64 t1 = tcg_temp_new_i64(); + TCGv_i64 t2 = tcg_temp_new_i64(); + TCGv_i64 t3 = tcg_temp_new_i64(); + + tcg_gen_andc_i64(t1, a, m); + tcg_gen_andc_i64(t2, b, m); + tcg_gen_xor_i64(t3, a, b); + tcg_gen_add_i64(d, t1, t2); + tcg_gen_and_i64(t3, t3, m); + tcg_gen_xor_i64(d, d, t3); + + tcg_temp_free_i64(t1); + tcg_temp_free_i64(t2); + tcg_temp_free_i64(t3); +} + +void tcg_gen_vec_add8_i64(TCGv_i64 d, TCGv_i64 a, TCGv_i64 b) +{ + TCGv_i64 m = tcg_const_i64(REP8(0x80)); + gen_addv_mask(d, a, b, m); + tcg_temp_free_i64(m); +} + +void tcg_gen_vec_add16_i64(TCGv_i64 d, TCGv_i64 a, TCGv_i64 b) +{ + TCGv_i64 m = tcg_const_i64(REP16(0x8000)); + gen_addv_mask(d, a, b, m); + tcg_temp_free_i64(m); +} + +void tcg_gen_vec_add32_i64(TCGv_i64 d, TCGv_i64 a, TCGv_i64 b) +{ + TCGv_i64 t1 = tcg_temp_new_i64(); + TCGv_i64 t2 = tcg_temp_new_i64(); + + tcg_gen_andi_i64(t1, a, ~0xffffffffull); + tcg_gen_add_i64(t2, a, b); + tcg_gen_add_i64(t1, t1, b); + tcg_gen_deposit_i64(d, t1, t2, 0, 32); + + tcg_temp_free_i64(t1); + tcg_temp_free_i64(t2); +} + +void tcg_gen_gvec_add(unsigned vece, uint32_t dofs, uint32_t aofs, + uint32_t bofs, uint32_t opsz, uint32_t maxsz) +{ + static const GVecGen3 g[4] = { + { .fni8 = tcg_gen_vec_add8_i64, + .fniv = tcg_gen_add_vec, + .fno = gen_helper_gvec_add8, + .opc = INDEX_op_add_vec, + .vece = MO_8 }, + { .fni8 = tcg_gen_vec_add16_i64, + .fniv = tcg_gen_add_vec, + .fno = gen_helper_gvec_add16, + .opc = INDEX_op_add_vec, + .vece = MO_16 }, + { .fni4 = tcg_gen_add_i32, + .fniv = tcg_gen_add_vec, + .fno = gen_helper_gvec_add32, + .opc = INDEX_op_add_vec, + .vece = MO_32 }, + { .fni8 = tcg_gen_add_i64, + .fniv = tcg_gen_add_vec, + .fno = gen_helper_gvec_add64, + .opc = INDEX_op_add_vec, + .prefer_i64 = TCG_TARGET_REG_BITS == 64, + .vece = MO_64 }, + }; + + tcg_debug_assert(vece <= MO_64); + tcg_gen_gvec_3(dofs, aofs, bofs, opsz, maxsz, &g[vece]); +} + +/* Perform a vector subtraction using normal subtraction and a mask. + Compare gen_addv_mask above. */ +static void gen_subv_mask(TCGv_i64 d, TCGv_i64 a, TCGv_i64 b, TCGv_i64 m) +{ + TCGv_i64 t1 = tcg_temp_new_i64(); + TCGv_i64 t2 = tcg_temp_new_i64(); + TCGv_i64 t3 = tcg_temp_new_i64(); + + tcg_gen_or_i64(t1, a, m); + tcg_gen_andc_i64(t2, b, m); + tcg_gen_eqv_i64(t3, a, b); + tcg_gen_sub_i64(d, t1, t2); + tcg_gen_and_i64(t3, t3, m); + tcg_gen_xor_i64(d, d, t3); + + tcg_temp_free_i64(t1); + tcg_temp_free_i64(t2); + tcg_temp_free_i64(t3); +} + +void tcg_gen_vec_sub8_i64(TCGv_i64 d, TCGv_i64 a, TCGv_i64 b) +{ + TCGv_i64 m = tcg_const_i64(REP8(0x80)); + gen_subv_mask(d, a, b, m); + tcg_temp_free_i64(m); +} + +void tcg_gen_vec_sub16_i64(TCGv_i64 d, TCGv_i64 a, TCGv_i64 b) +{ + TCGv_i64 m = tcg_const_i64(REP16(0x8000)); + gen_subv_mask(d, a, b, m); + tcg_temp_free_i64(m); +} + +void tcg_gen_vec_sub32_i64(TCGv_i64 d, TCGv_i64 a, TCGv_i64 b) +{ + TCGv_i64 t1 = tcg_temp_new_i64(); + TCGv_i64 t2 = tcg_temp_new_i64(); + + tcg_gen_andi_i64(t1, b, ~0xffffffffull); + tcg_gen_sub_i64(t2, a, b); + tcg_gen_sub_i64(t1, a, t1); + tcg_gen_deposit_i64(d, t1, t2, 0, 32); + + tcg_temp_free_i64(t1); + tcg_temp_free_i64(t2); +} + +void tcg_gen_gvec_sub(unsigned vece, uint32_t dofs, uint32_t aofs, + uint32_t bofs, uint32_t opsz, uint32_t maxsz) +{ + static const GVecGen3 g[4] = { + { .fni8 = tcg_gen_vec_sub8_i64, + .fniv = tcg_gen_sub_vec, + .fno = gen_helper_gvec_sub8, + .opc = INDEX_op_sub_vec, + .vece = MO_8 }, + { .fni8 = tcg_gen_vec_sub16_i64, + .fniv = tcg_gen_sub_vec, + .fno = gen_helper_gvec_sub16, + .opc = INDEX_op_sub_vec, + .vece = MO_16 }, + { .fni4 = tcg_gen_sub_i32, + .fniv = tcg_gen_sub_vec, + .fno = gen_helper_gvec_sub32, + .opc = INDEX_op_sub_vec, + .vece = MO_32 }, + { .fni8 = tcg_gen_sub_i64, + .fniv = tcg_gen_sub_vec, + .fno = gen_helper_gvec_sub64, + .opc = INDEX_op_sub_vec, + .prefer_i64 = TCG_TARGET_REG_BITS == 64, + .vece = MO_64 }, + }; + + tcg_debug_assert(vece <= MO_64); + tcg_gen_gvec_3(dofs, aofs, bofs, opsz, maxsz, &g[vece]); +} + +/* Perform a vector negation using normal negation and a mask. + Compare gen_subv_mask above. */ +static void gen_negv_mask(TCGv_i64 d, TCGv_i64 b, TCGv_i64 m) +{ + TCGv_i64 t2 = tcg_temp_new_i64(); + TCGv_i64 t3 = tcg_temp_new_i64(); + + tcg_gen_andc_i64(t3, m, b); + tcg_gen_andc_i64(t2, b, m); + tcg_gen_sub_i64(d, m, t2); + tcg_gen_xor_i64(d, d, t3); + + tcg_temp_free_i64(t2); + tcg_temp_free_i64(t3); +} + +void tcg_gen_vec_neg8_i64(TCGv_i64 d, TCGv_i64 b) +{ + TCGv_i64 m = tcg_const_i64(REP8(0x80)); + gen_negv_mask(d, b, m); + tcg_temp_free_i64(m); +} + +void tcg_gen_vec_neg16_i64(TCGv_i64 d, TCGv_i64 b) +{ + TCGv_i64 m = tcg_const_i64(REP16(0x8000)); + gen_negv_mask(d, b, m); + tcg_temp_free_i64(m); +} + +void tcg_gen_vec_neg32_i64(TCGv_i64 d, TCGv_i64 b) +{ + TCGv_i64 t1 = tcg_temp_new_i64(); + TCGv_i64 t2 = tcg_temp_new_i64(); + + tcg_gen_andi_i64(t1, b, ~0xffffffffull); + tcg_gen_neg_i64(t2, b); + tcg_gen_neg_i64(t1, t1); + tcg_gen_deposit_i64(d, t1, t2, 0, 32); + + tcg_temp_free_i64(t1); + tcg_temp_free_i64(t2); +} + +void tcg_gen_gvec_neg(unsigned vece, uint32_t dofs, uint32_t aofs, + uint32_t opsz, uint32_t maxsz) +{ + static const GVecGen2 g[4] = { + { .fni8 = tcg_gen_vec_neg8_i64, + .fniv = tcg_gen_neg_vec, + .fno = gen_helper_gvec_neg8, + .opc = INDEX_op_neg_vec, + .vece = MO_8 }, + { .fni8 = tcg_gen_vec_neg16_i64, + .fniv = tcg_gen_neg_vec, + .fno = gen_helper_gvec_neg16, + .opc = INDEX_op_neg_vec, + .vece = MO_16 }, + { .fni4 = tcg_gen_neg_i32, + .fniv = tcg_gen_neg_vec, + .fno = gen_helper_gvec_neg32, + .opc = INDEX_op_neg_vec, + .vece = MO_32 }, + { .fni8 = tcg_gen_neg_i64, + .fniv = tcg_gen_neg_vec, + .fno = gen_helper_gvec_neg64, + .opc = INDEX_op_neg_vec, + .prefer_i64 = TCG_TARGET_REG_BITS == 64, + .vece = MO_64 }, + }; + + tcg_debug_assert(vece <= MO_64); + tcg_gen_gvec_2(dofs, aofs, opsz, maxsz, &g[vece]); +} + +void tcg_gen_gvec_and(unsigned vece, uint32_t dofs, uint32_t aofs, + uint32_t bofs, uint32_t opsz, uint32_t maxsz) +{ + static const GVecGen3 g = { + .fni8 = tcg_gen_and_i64, + .fniv = tcg_gen_and_vec, + .fno = gen_helper_gvec_and, + .opc = INDEX_op_and_vec, + .prefer_i64 = TCG_TARGET_REG_BITS == 64, + }; + tcg_gen_gvec_3(dofs, aofs, bofs, opsz, maxsz, &g); +} + +void tcg_gen_gvec_or(unsigned vece, uint32_t dofs, uint32_t aofs, + uint32_t bofs, uint32_t opsz, uint32_t maxsz) +{ + static const GVecGen3 g = { + .fni8 = tcg_gen_or_i64, + .fniv = tcg_gen_or_vec, + .fno = gen_helper_gvec_or, + .opc = INDEX_op_or_vec, + .prefer_i64 = TCG_TARGET_REG_BITS == 64, + }; + tcg_gen_gvec_3(dofs, aofs, bofs, opsz, maxsz, &g); +} + +void tcg_gen_gvec_xor(unsigned vece, uint32_t dofs, uint32_t aofs, + uint32_t bofs, uint32_t opsz, uint32_t maxsz) +{ + static const GVecGen3 g = { + .fni8 = tcg_gen_xor_i64, + .fniv = tcg_gen_xor_vec, + .fno = gen_helper_gvec_xor, + .opc = INDEX_op_xor_vec, + .prefer_i64 = TCG_TARGET_REG_BITS == 64, + }; + tcg_gen_gvec_3(dofs, aofs, bofs, opsz, maxsz, &g); +} + +void tcg_gen_gvec_andc(unsigned vece, uint32_t dofs, uint32_t aofs, + uint32_t bofs, uint32_t opsz, uint32_t maxsz) +{ + static const GVecGen3 g = { + .fni8 = tcg_gen_andc_i64, + .fniv = tcg_gen_andc_vec, + .fno = gen_helper_gvec_andc, + .opc = INDEX_op_andc_vec, + .prefer_i64 = TCG_TARGET_REG_BITS == 64, + }; + tcg_gen_gvec_3(dofs, aofs, bofs, opsz, maxsz, &g); +} + +void tcg_gen_gvec_orc(unsigned vece, uint32_t dofs, uint32_t aofs, + uint32_t bofs, uint32_t opsz, uint32_t maxsz) +{ + static const GVecGen3 g = { + .fni8 = tcg_gen_orc_i64, + .fniv = tcg_gen_orc_vec, + .fno = gen_helper_gvec_orc, + .opc = INDEX_op_orc_vec, + .prefer_i64 = TCG_TARGET_REG_BITS == 64, + }; + tcg_gen_gvec_3(dofs, aofs, bofs, opsz, maxsz, &g); +} diff --git a/tcg/tcg-op-vec.c b/tcg/tcg-op-vec.c index dc04c11860..5cfe4af6bd 100644 --- a/tcg/tcg-op-vec.c +++ b/tcg/tcg-op-vec.c @@ -104,7 +104,7 @@ void tcg_gen_mov_vec(TCGv_vec r, TCGv_vec a) #define MO_REG (TCG_TARGET_REG_BITS == 64 ? MO_64 : MO_32) -static void tcg_gen_dupi_vec(TCGv_vec r, unsigned vece, TCGArg a) +static void do_dupi_vec(TCGv_vec r, unsigned vece, TCGArg a) { TCGTemp *rt = tcgv_vec_temp(r); vec_gen_2(INDEX_op_dupi_vec, rt->base_type, vece, temp_arg(rt), a); @@ -113,14 +113,14 @@ static void tcg_gen_dupi_vec(TCGv_vec r, unsigned vece, TCGArg a) TCGv_vec tcg_const_zeros_vec(TCGType type) { TCGv_vec ret = tcg_temp_new_vec(type); - tcg_gen_dupi_vec(ret, MO_REG, 0); + do_dupi_vec(ret, MO_REG, 0); return ret; } TCGv_vec tcg_const_ones_vec(TCGType type) { TCGv_vec ret = tcg_temp_new_vec(type); - tcg_gen_dupi_vec(ret, MO_REG, -1); + do_dupi_vec(ret, MO_REG, -1); return ret; } @@ -139,9 +139,9 @@ TCGv_vec tcg_const_ones_vec_matching(TCGv_vec m) void tcg_gen_dup64i_vec(TCGv_vec r, uint64_t a) { if (TCG_TARGET_REG_BITS == 32 && a == deposit64(a, 32, 32, a)) { - tcg_gen_dupi_vec(r, MO_32, a); + do_dupi_vec(r, MO_32, a); } else if (TCG_TARGET_REG_BITS == 64 || a == (uint64_t)(int32_t)a) { - tcg_gen_dupi_vec(r, MO_64, a); + do_dupi_vec(r, MO_64, a); } else { TCGv_i64 c = tcg_const_i64(a); tcg_gen_dup_i64_vec(MO_64, r, c); @@ -151,17 +151,37 @@ void tcg_gen_dup64i_vec(TCGv_vec r, uint64_t a) void tcg_gen_dup32i_vec(TCGv_vec r, uint32_t a) { - tcg_gen_dupi_vec(r, MO_REG, ((TCGArg)-1 / 0xffffffffu) * a); + do_dupi_vec(r, MO_REG, ((TCGArg)-1 / 0xffffffffu) * a); } void tcg_gen_dup16i_vec(TCGv_vec r, uint32_t a) { - tcg_gen_dupi_vec(r, MO_REG, ((TCGArg)-1 / 0xffff) * (a & 0xffff)); + do_dupi_vec(r, MO_REG, ((TCGArg)-1 / 0xffff) * (a & 0xffff)); } void tcg_gen_dup8i_vec(TCGv_vec r, uint32_t a) { - tcg_gen_dupi_vec(r, MO_REG, ((TCGArg)-1 / 0xff) * (a & 0xff)); + do_dupi_vec(r, MO_REG, ((TCGArg)-1 / 0xff) * (a & 0xff)); +} + +void tcg_gen_dupi_vec(unsigned vece, TCGv_vec r, uint64_t a) +{ + switch (vece) { + case MO_8: + tcg_gen_dup8i_vec(r, a); + break; + case MO_16: + tcg_gen_dup16i_vec(r, a); + break; + case MO_32: + tcg_gen_dup32i_vec(r, a); + break; + case MO_64: + tcg_gen_dup64i_vec(r, a); + break; + default: + g_assert_not_reached(); + } } void tcg_gen_movi_v64(TCGv_vec r, uint64_t a) diff --git a/accel/tcg/Makefile.objs b/accel/tcg/Makefile.objs index 228cd84fa4..d381a02f34 100644 --- a/accel/tcg/Makefile.objs +++ b/accel/tcg/Makefile.objs @@ -1,6 +1,6 @@ obj-$(CONFIG_SOFTMMU) += tcg-all.o obj-$(CONFIG_SOFTMMU) += cputlb.o -obj-y += tcg-runtime.o +obj-y += tcg-runtime.o tcg-runtime-gvec.o obj-y += cpu-exec.o cpu-exec-common.o translate-all.o obj-y += translator.o From patchwork Tue Nov 21 21:25:14 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 119409 Delivered-To: patch@linaro.org Received: by 10.140.22.164 with SMTP id 33csp5816274qgn; Tue, 21 Nov 2017 13:33:33 -0800 (PST) X-Google-Smtp-Source: AGs4zMZc80Bil+U3kLOWnIE/oI7XelHLu2FhmY9uET+v3uUIg2Q0AIZXxpUjKUDBBgoQ3idT/7OA X-Received: by 10.13.234.195 with SMTP id t186mr11834124ywe.326.1511300013374; Tue, 21 Nov 2017 13:33:33 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1511300013; cv=none; d=google.com; s=arc-20160816; b=TtVwkstcxigmuee08B8i6rHxCqPCLNSxhmzbWygUU2RyBK+bt+Kv5/z4Z4k5jj1y0Q 8xbsZI4CYCTimHzoh8pG3K9mgdIzhrRv0l+y+9f7e2/9DvsQVQkxnixspbc9WNLJFZ/w zqHRk34lz8uyrmtmB3sVGEBAkQQ3tcKEThEyeB6kyAUwM6SC1F21yFexzQqwBvF34J99 4C1c2J/dbSI/4DYohLGd8cgZMDVHM3OR2DrrrdCeYfO9DdipFYWycZE3hQH/LEwe06Wb fQ3W3sIYAxW62IJiBEyh3Cr7GGL2JzAoWTKSkmEdbZRz3rX/EZXVNi3KVS8xPEN0EYcS I1fA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:subject:references:in-reply-to :message-id:date:to:from:dkim-signature:arc-authentication-results; bh=btHN6BlYfEMHw/3KsRDCBldcyv8BDSFRKPzXoGS7RHA=; b=W4eduaG8WKan9ddzhNoaQ2I45oaShFjGm16c5v/0LAFmAxqdxvpdILFjXj66LywmOE SIkJylJwS94KFb5dJE5xohm8gAVmM8VYLNkNqzBXWExngN4T6vU5iN84h0uCnMgJ+xIG VXWHrdYIboFptFdmS88DlEDDpA3JqTrRVbNt/Ap+aVYXu8Fprhwd7AIXtgiI+eYhPjT7 vHnHmP1o1sOSMSd8sIdJITZFZFXrlQt1VY0wCvlU/j30bVywbVmxrpYGURN5GvJ2RPyK txxo0HQlOi1g9VjdHhffDge4bTzXTu4TGWNnE07NWzj4dTR6QN5Nt12UzOpZf0AJzdsW rzOQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=Vah0ub7s; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) smtp.mailfrom=qemu-devel-bounces+patch=linaro.org@nongnu.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [2001:4830:134:3::11]) by mx.google.com with ESMTPS id n66si3151748ybn.48.2017.11.21.13.33.33 for (version=TLS1 cipher=AES128-SHA bits=128/128); Tue, 21 Nov 2017 13:33:33 -0800 (PST) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) client-ip=2001:4830:134:3::11; Authentication-Results: mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=Vah0ub7s; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) smtp.mailfrom=qemu-devel-bounces+patch=linaro.org@nongnu.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1]:36602 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1eHGAe-0000Ou-QQ for patch@linaro.org; Tue, 21 Nov 2017 16:33:32 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:53943) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1eHG54-0004J8-QM for qemu-devel@nongnu.org; Tue, 21 Nov 2017 16:27:48 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1eHG53-0007fT-GX for qemu-devel@nongnu.org; Tue, 21 Nov 2017 16:27:46 -0500 Received: from mail-wm0-x243.google.com ([2a00:1450:400c:c09::243]:42741) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1eHG53-0007ez-7y for qemu-devel@nongnu.org; Tue, 21 Nov 2017 16:27:45 -0500 Received: by mail-wm0-x243.google.com with SMTP id l188so4522980wma.1 for ; Tue, 21 Nov 2017 13:27:45 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:subject:date:message-id:in-reply-to:references; bh=btHN6BlYfEMHw/3KsRDCBldcyv8BDSFRKPzXoGS7RHA=; b=Vah0ub7sAwA2RojUccd7K0ErmiRDPoTVusby4r/gZyciin0QG94bQIfZEtHOrgy2M8 K6xfN12559Kch+ca6HjfMUTf3vB835FAeo09cm0T+8UTUP7XNvZyT6EmDXRkTc4g6OWc qEK85+WDm0CLon6lynOyVMBpsKMD8gj+9NyAw= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references; bh=btHN6BlYfEMHw/3KsRDCBldcyv8BDSFRKPzXoGS7RHA=; b=MBO7IVK0lCWkIyAS6GZxlU4ltr4+OUAmLFBSGrO+uoXLO4JLolZFr8dJXxAg9KEwv6 PqaFgOmEHd06E5cAj0SpPjeAHBxAitDzgsKE/eAcRhzshBKOzTO0r9FtPbcm4Vtsffgq PM4Vfqdb3WynlN86wJGExhWB9zG7kOd9uxOc+QL39USeIqW9ixZBzUdOCnX/4L0K9VUr 8NJ+jVIjSBWQG5UzFhFb9fIVC4VUJdwAQARt6d1xqspcQlF90Mr08O3uNI33L2/bCzdN x1J20i2VGNrvv2GPD1v497Okebj5KTMzGFA1f+H3zpzSv/A9V5u3lBdIqEzH5PqM1c5z 6zLQ== X-Gm-Message-State: AJaThX5ZVt8bmvLdLV4kySWznkwknZuWuYACVLEq3/MKWmEQHzYt0eLy OsNI28aXJQxUWpAbMVxlP23BHS5Nxkc= X-Received: by 10.28.73.193 with SMTP id w184mr2397763wma.3.1511299664002; Tue, 21 Nov 2017 13:27:44 -0800 (PST) Received: from cloudburst.twiddle.net (70.red-37-158-60.dynamicip.rima-tde.net. [37.158.60.70]) by smtp.gmail.com with ESMTPSA id e124sm706517wmg.34.2017.11.21.13.27.43 for (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Tue, 21 Nov 2017 13:27:43 -0800 (PST) From: Richard Henderson To: qemu-devel@nongnu.org Date: Tue, 21 Nov 2017 22:25:14 +0100 Message-Id: <20171121212534.5177-7-richard.henderson@linaro.org> X-Mailer: git-send-email 2.13.6 In-Reply-To: <20171121212534.5177-1-richard.henderson@linaro.org> References: <20171121212534.5177-1-richard.henderson@linaro.org> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2a00:1450:400c:c09::243 Subject: [Qemu-devel] [PATCH v6 06/26] tcg: Allow multiple word entries into the constant pool X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: "Qemu-devel" This will be required for storing vector constants. Signed-off-by: Richard Henderson --- tcg/tcg-pool.inc.c | 115 +++++++++++++++++++++++++++++++++++++++++++---------- 1 file changed, 93 insertions(+), 22 deletions(-) -- 2.13.6 diff --git a/tcg/tcg-pool.inc.c b/tcg/tcg-pool.inc.c index 8a85131405..0f76e7bee3 100644 --- a/tcg/tcg-pool.inc.c +++ b/tcg/tcg-pool.inc.c @@ -22,39 +22,110 @@ typedef struct TCGLabelPoolData { struct TCGLabelPoolData *next; - tcg_target_ulong data; tcg_insn_unit *label; - intptr_t addend; - int type; + int addend : 32; + int rtype : 16; + int nlong : 16; + tcg_target_ulong data[]; } TCGLabelPoolData; -static void new_pool_label(TCGContext *s, tcg_target_ulong data, int type, - tcg_insn_unit *label, intptr_t addend) +static TCGLabelPoolData *new_pool_alloc(TCGContext *s, int nlong, int rtype, + tcg_insn_unit *label, int addend) { - TCGLabelPoolData *n = tcg_malloc(sizeof(*n)); - TCGLabelPoolData *i, **pp; + TCGLabelPoolData *n = tcg_malloc(sizeof(TCGLabelPoolData) + + sizeof(tcg_target_ulong) * nlong); - n->data = data; n->label = label; - n->type = type; n->addend = addend; + n->rtype = rtype; + n->nlong = nlong; + return n; +} + +static void new_pool_insert(TCGContext *s, TCGLabelPoolData *n) +{ + TCGLabelPoolData *i, **pp; + int nlong = n->nlong; /* Insertion sort on the pool. */ - for (pp = &s->pool_labels; (i = *pp) && i->data < data; pp = &i->next) { - continue; + for (pp = &s->pool_labels; (i = *pp) != NULL; pp = &i->next) { + if (nlong > i->nlong) { + break; + } + if (nlong < i->nlong) { + continue; + } + if (memcmp(n->data, i->data, sizeof(tcg_target_ulong) * nlong) >= 0) { + break; + } } n->next = *pp; *pp = n; } +/* The "usual" for generic integer code. */ +static inline void new_pool_label(TCGContext *s, tcg_target_ulong d, int rtype, + tcg_insn_unit *label, int addend) +{ + TCGLabelPoolData *n = new_pool_alloc(s, 1, rtype, label, addend); + n->data[0] = d; + new_pool_insert(s, n); +} + +/* For v64 or v128, depending on the host. */ +static inline void new_pool_l2(TCGContext *s, int rtype, tcg_insn_unit *label, + int addend, tcg_target_ulong d0, + tcg_target_ulong d1) +{ + TCGLabelPoolData *n = new_pool_alloc(s, 2, rtype, label, addend); + n->data[0] = d0; + n->data[1] = d1; + new_pool_insert(s, n); +} + +/* For v128 or v256, depending on the host. */ +static inline void new_pool_l4(TCGContext *s, int rtype, tcg_insn_unit *label, + int addend, tcg_target_ulong d0, + tcg_target_ulong d1, tcg_target_ulong d2, + tcg_target_ulong d3) +{ + TCGLabelPoolData *n = new_pool_alloc(s, 4, rtype, label, addend); + n->data[0] = d0; + n->data[1] = d1; + n->data[2] = d2; + n->data[3] = d3; + new_pool_insert(s, n); +} + +/* For v256, for 32-bit host. */ +static inline void new_pool_l8(TCGContext *s, int rtype, tcg_insn_unit *label, + int addend, tcg_target_ulong d0, + tcg_target_ulong d1, tcg_target_ulong d2, + tcg_target_ulong d3, tcg_target_ulong d4, + tcg_target_ulong d5, tcg_target_ulong d6, + tcg_target_ulong d7) +{ + TCGLabelPoolData *n = new_pool_alloc(s, 8, rtype, label, addend); + n->data[0] = d0; + n->data[1] = d1; + n->data[2] = d2; + n->data[3] = d3; + n->data[4] = d4; + n->data[5] = d5; + n->data[6] = d6; + n->data[7] = d7; + new_pool_insert(s, n); +} + /* To be provided by cpu/tcg-target.inc.c. */ static void tcg_out_nop_fill(tcg_insn_unit *p, int count); static bool tcg_out_pool_finalize(TCGContext *s) { TCGLabelPoolData *p = s->pool_labels; - tcg_target_ulong d, *a; + TCGLabelPoolData *l = NULL; + void *a; if (p == NULL) { return true; @@ -62,24 +133,24 @@ static bool tcg_out_pool_finalize(TCGContext *s) /* ??? Round up to qemu_icache_linesize, but then do not round again when allocating the next TranslationBlock structure. */ - a = (void *)ROUND_UP((uintptr_t)s->code_ptr, sizeof(tcg_target_ulong)); + a = (void *)ROUND_UP((uintptr_t)s->code_ptr, + sizeof(tcg_target_ulong) * p->nlong); tcg_out_nop_fill(s->code_ptr, (tcg_insn_unit *)a - s->code_ptr); s->data_gen_ptr = a; - /* Ensure the first comparison fails. */ - d = p->data + 1; - for (; p != NULL; p = p->next) { - if (p->data != d) { - d = p->data; - if (unlikely((void *)a > s->code_gen_highwater)) { + size_t size = sizeof(tcg_target_ulong) * p->nlong; + if (!l || l->nlong != p->nlong || memcmp(l->data, p->data, size)) { + if (unlikely(a > s->code_gen_highwater)) { return false; } - *a++ = d; + memcpy(a, p->data, size); + a += size; + l = p; } - patch_reloc(p->label, p->type, (intptr_t)(a - 1), p->addend); + patch_reloc(p->label, p->rtype, (intptr_t)a - size, p->addend); } - s->code_ptr = (void *)a; + s->code_ptr = a; return true; } From patchwork Tue Nov 21 21:25:15 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 119404 Delivered-To: patch@linaro.org Received: by 10.140.22.164 with SMTP id 33csp5811067qgn; Tue, 21 Nov 2017 13:28:15 -0800 (PST) X-Google-Smtp-Source: AGs4zMYQXms62scwKd7xnToapsySywoYBLKWD7uD+0p2Na/qcq2ACM9sp+NTap3waLDQf+Wy+Lo5 X-Received: by 10.129.152.143 with SMTP id p137mr11544719ywg.165.1511299695664; Tue, 21 Nov 2017 13:28:15 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1511299695; cv=none; d=google.com; s=arc-20160816; b=E4bG9fBrYt68X4tDYP0Df/rH1LAo05wAlxdZPQagdcdWi0xhCfBOrj/PWlyglZqY65 sqevS7EPxC3L9IcstWfcPbBor+YfvmFyNlDCENNXcAFxyIwqWCQ36kvP8Un9wIrBV+7x Ytc6idVYdlMwK81FbAblPGXhasxgPx87KlRXdrADNsyR3D61x4rgEXMLp1ZssJ3enTAN 8OaAV1KNr5XImYzsbDPGBzr7pAyRO/0/nDkwD44GORIlgq7Te2o3u4jXhtRRBvfMV4ou f7rDUFZW5S2EWb0IHQs7O5g0M/vdQHwRU5w2vwSVv/gU21kEUk2JCyAfKHbNv/pMwLxJ zqiQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:subject:references:in-reply-to :message-id:date:to:from:dkim-signature:arc-authentication-results; bh=DQqqwXi6krqrAR+MFMh6WBa/XGoGSPQ/T7AOKHY6UYY=; b=htn39LQk9bku5p5+Z2G8KATZuw7FVdwc1GxUiCx3QOsPT5fvIU2pCGKfC18Zmfrlz7 YgbaYRT1yVczp3iWhFA7ay6FfoMTFAu+Rwus7D+aEALygcSZ5E+kV14jLmD2gvMTEVar 5ipUjoKS6CT5IL6qGlC53X/WSigyA63262K0Shyxapy53mIodqKFH05bepaCSC1bvNmL NCz07wR0aD3qqbg4UMxoWNMuTixUOsJg6/n68QSBDgDSbfhZ7vvkaWvdWs8qjcNvPTvr VuqgNT3TnCIXrV1+PloaoF3tOf5zy28kgqdRkf5AoRs5RrbFNMcadw3AsOyd9JH0b+T+ i3Xg== ARC-Authentication-Results: i=1; mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=au4rpKIh; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) smtp.mailfrom=qemu-devel-bounces+patch=linaro.org@nongnu.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [2001:4830:134:3::11]) by mx.google.com with ESMTPS id 204si2222882ywf.423.2017.11.21.13.28.15 for (version=TLS1 cipher=AES128-SHA bits=128/128); Tue, 21 Nov 2017 13:28:15 -0800 (PST) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) client-ip=2001:4830:134:3::11; Authentication-Results: mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=au4rpKIh; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) smtp.mailfrom=qemu-devel-bounces+patch=linaro.org@nongnu.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1]:36577 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1eHG5X-0004O1-5M for patch@linaro.org; Tue, 21 Nov 2017 16:28:15 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:53960) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1eHG55-0004Jl-Hb for qemu-devel@nongnu.org; Tue, 21 Nov 2017 16:27:50 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1eHG54-0007g6-OL for qemu-devel@nongnu.org; Tue, 21 Nov 2017 16:27:47 -0500 Received: from mail-wm0-x244.google.com ([2a00:1450:400c:c09::244]:44105) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1eHG54-0007fj-HT for qemu-devel@nongnu.org; Tue, 21 Nov 2017 16:27:46 -0500 Received: by mail-wm0-x244.google.com with SMTP id r68so6352420wmr.3 for ; Tue, 21 Nov 2017 13:27:46 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:subject:date:message-id:in-reply-to:references; bh=DQqqwXi6krqrAR+MFMh6WBa/XGoGSPQ/T7AOKHY6UYY=; b=au4rpKIhTmnfMFTh1kVaInfvALY5+GYq5qwvdOAuJE+zefJbNUvvlRUr9Us1YCARr5 G3y5q2+SOlpZ1JjEUE/lw7Xz1LNEJi2NKJ+VJION/sdFPf23fMDg0A3KaSTAeNNMKZZ4 P5ctxbBbSu40wryqBr4expE4FueKPh40ATMqQ= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references; bh=DQqqwXi6krqrAR+MFMh6WBa/XGoGSPQ/T7AOKHY6UYY=; b=ngDl9ppCBWD0Xim+FRrVq7sB9tIPVtwIle89LqcnU9MGUUM/fzP94OfXDQ4vWBBahf oGOY97LaLLJHtg1JmLdq7doP4JJRBWdlGXc4zxVJNqSBBLdXZRWNDvYXv/5M5cDTl9Ma FZ+TRwHSonqJA2UG8X5T2fpuPvoAwL5qfZnc+KZSuzIU1rehcpm7WXYYZ3Xbrx3m4ibW U18kEBxfDU0huRi1SClbDgI5jzfFEcrNnlIu5bOHiXpdYmMphjq9lfbzveCIIK0UAvKm fBOgBu9gXH/aM3/8nbIV66a+qC2v9jSC4se479fpHbgZEE7BglbbQI54cQfSBnZMsiZm PIRw== X-Gm-Message-State: AJaThX4k8jAvTG0ug0VpmCeKOosOFJdXNyA+XktR9RbhfBlw87BJicne sY2Nw1gQwQh34DkMNPwF0siajc4EoLc= X-Received: by 10.28.33.136 with SMTP id h130mr2255539wmh.151.1511299665417; Tue, 21 Nov 2017 13:27:45 -0800 (PST) Received: from cloudburst.twiddle.net (70.red-37-158-60.dynamicip.rima-tde.net. [37.158.60.70]) by smtp.gmail.com with ESMTPSA id e124sm706517wmg.34.2017.11.21.13.27.44 for (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Tue, 21 Nov 2017 13:27:44 -0800 (PST) From: Richard Henderson To: qemu-devel@nongnu.org Date: Tue, 21 Nov 2017 22:25:15 +0100 Message-Id: <20171121212534.5177-8-richard.henderson@linaro.org> X-Mailer: git-send-email 2.13.6 In-Reply-To: <20171121212534.5177-1-richard.henderson@linaro.org> References: <20171121212534.5177-1-richard.henderson@linaro.org> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2a00:1450:400c:c09::244 Subject: [Qemu-devel] [PATCH v6 07/26] tcg: Add tcg_signed_cond X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: "Qemu-devel" Complimenting the existing tcg_unsigned_cond. Signed-off-by: Richard Henderson --- tcg/tcg.h | 6 ++++++ 1 file changed, 6 insertions(+) -- 2.13.6 diff --git a/tcg/tcg.h b/tcg/tcg.h index 2acebd387a..49d4c5fe05 100644 --- a/tcg/tcg.h +++ b/tcg/tcg.h @@ -517,6 +517,12 @@ static inline TCGCond tcg_unsigned_cond(TCGCond c) return c & 2 ? (TCGCond)(c ^ 6) : c; } +/* Create a "signed" version of an "unsigned" comparison. */ +static inline TCGCond tcg_signed_cond(TCGCond c) +{ + return c & 4 ? (TCGCond)(c ^ 6) : c; +} + /* Must a comparison be considered unsigned? */ static inline bool is_unsigned_cond(TCGCond c) { From patchwork Tue Nov 21 21:25:16 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 119410 Delivered-To: patch@linaro.org Received: by 10.140.22.164 with SMTP id 33csp5816373qgn; Tue, 21 Nov 2017 13:33:39 -0800 (PST) X-Google-Smtp-Source: AGs4zMa2rtrUyazUFEVSi75e/U9qbLUSwc4taBTgutNfj3HTguG9qOHr4SSOmt9HNcBwAiBXzvR2 X-Received: by 10.129.73.23 with SMTP id w23mr12137242ywa.373.1511300019419; Tue, 21 Nov 2017 13:33:39 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1511300019; cv=none; d=google.com; s=arc-20160816; b=Qg4rGpoxOBKwtrld3GmFnPHqt9wDMvjRVSpVG/ckXTjWfe56KWlQZMQliEHLDrFELz HDc9zPVv3dIyLPoGFHU7wdFVGYr5m7sjBYtovbPL7J/4TV487P4PQxFp5dtXpHVGNwCn DlgLyDz2nApZja7ShwekD7ZzHANbqwDupBQ8QvNz6+FmaRER5uSfwwnH/BTDmhOb0Fhd 1I3syWlTf4yCim/KdaEfuG3p2qcIx0q6wJ2R/gdUY1+IYXsa13Jy0bUU9jSiYsFWGHa7 +bkd2Nd0P10VyPXOVIDFTIvx66BZAJ65f03VN/i/2G+csOm+ncjmNmHcQvpv5fzz97Zl gsCw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:subject :content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:to:from:dkim-signature:arc-authentication-results; bh=GLyvDykI4af6RkGn5xmg6fjMBXV0cxlT5aCt8CXch74=; b=XYO5ucTkSV+81yICrAzwBRui4YDFWH4mQ5FOew2gl5xAYzbSpT/nKkNX6L/g8JXfIV NMfmNk7illlgt7LVwJMqQNSRgtgwM7cxWQoJDYmuzGgJkmzizDLQpeVePjM0t+lmbyYQ pQm3FpfIL5cwUXJi/jSb7/j1JaYCFJY4/YLpqTcsgmuDvlMbvOBJ+fv1sGnTtdeKpbIT EdGov2w7GHukf3cw46lGefzw9llzIOanPnRDZG+N2aVp8dK3ZKx5TR4n9iRgq7+uvkxB AqingO+rZClepedJAM1KbW28swle3HKJeU8leHA/VrqAwlPc2IL8CNvM9ntIgRzkX36C kw0g== ARC-Authentication-Results: i=1; mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=N/9GiBvG; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) smtp.mailfrom=qemu-devel-bounces+patch=linaro.org@nongnu.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [2001:4830:134:3::11]) by mx.google.com with ESMTPS id o5si3326563ywm.285.2017.11.21.13.33.39 for (version=TLS1 cipher=AES128-SHA bits=128/128); Tue, 21 Nov 2017 13:33:39 -0800 (PST) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) client-ip=2001:4830:134:3::11; Authentication-Results: mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=N/9GiBvG; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) smtp.mailfrom=qemu-devel-bounces+patch=linaro.org@nongnu.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1]:36604 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1eHGAk-0000U9-PC for patch@linaro.org; Tue, 21 Nov 2017 16:33:38 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:54006) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1eHG58-0004Mb-MO for qemu-devel@nongnu.org; Tue, 21 Nov 2017 16:27:51 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1eHG55-0007gv-Qw for qemu-devel@nongnu.org; Tue, 21 Nov 2017 16:27:50 -0500 Received: from mail-wm0-x244.google.com ([2a00:1450:400c:c09::244]:38918) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1eHG55-0007gQ-L7 for qemu-devel@nongnu.org; Tue, 21 Nov 2017 16:27:47 -0500 Received: by mail-wm0-x244.google.com with SMTP id x63so6255336wmf.4 for ; Tue, 21 Nov 2017 13:27:47 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=GLyvDykI4af6RkGn5xmg6fjMBXV0cxlT5aCt8CXch74=; b=N/9GiBvGVFewvHQ4J9gl86ZfK2UxqnVTCR4yQnspooEZdWL8Es74vTAt9nhz4FdGeb EmoNPWHU2WD5RYt+VtdIaDeJ3IOPkf/LOOhC7VaOiMaqNYXq/9PsXC48Vr2f28vcRfF3 co+60wv0joXg/Jh63/X0Nw14tShi7+Xy3Trp0= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=GLyvDykI4af6RkGn5xmg6fjMBXV0cxlT5aCt8CXch74=; b=qj9dGPReOBzDkhse8tIZPeFnc1jDFPYwj9n7Pd9stcSmlV9pFY0PRd/tonaYZI+Th/ /zKIL8v0TqD9pRhm+M0MpOVpoBH8El10QagmixDjv6qcowguU0/yPHNiG4x4T6Q6PhqF Xwz40yMWOzRwj3b9+PB9wgHxMDpbDH/4qA6BhUaeXtZAvk4iFC1unbm1aTUqGzOv5Olj luVg5Xz3WK2DFaLODkvuMwYhCmgNLuEW+PtD70a5GCAo7nS6Qdl3m6xCILTELk5J7M7Z OBcqrTFF7qwn2v0xtjFk4VNOX6APipCxyXbehWx09en0ZtOsDQRO0KggalGg8aX1zLTP 7obA== X-Gm-Message-State: AJaThX5YIHHBjxiQWB4lgqvEdbBfHcbmAt8OBNYyvK5ccLlkGr3cdp6l bRcnhsVOqJs2Qpt7dmqFlO6IKpfZewo= X-Received: by 10.28.12.193 with SMTP id 184mr2683027wmm.70.1511299666553; Tue, 21 Nov 2017 13:27:46 -0800 (PST) Received: from cloudburst.twiddle.net (70.red-37-158-60.dynamicip.rima-tde.net. [37.158.60.70]) by smtp.gmail.com with ESMTPSA id e124sm706517wmg.34.2017.11.21.13.27.45 for (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Tue, 21 Nov 2017 13:27:45 -0800 (PST) From: Richard Henderson To: qemu-devel@nongnu.org Date: Tue, 21 Nov 2017 22:25:16 +0100 Message-Id: <20171121212534.5177-9-richard.henderson@linaro.org> X-Mailer: git-send-email 2.13.6 In-Reply-To: <20171121212534.5177-1-richard.henderson@linaro.org> References: <20171121212534.5177-1-richard.henderson@linaro.org> MIME-Version: 1.0 X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2a00:1450:400c:c09::244 Subject: [Qemu-devel] [PATCH v6 08/26] target/arm: Align vector registers X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: "Qemu-devel" Reviewed-by: Alex Bennée Signed-off-by: Richard Henderson --- target/arm/cpu.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) -- 2.13.6 diff --git a/target/arm/cpu.h b/target/arm/cpu.h index 89d49cdcb2..8238edaba9 100644 --- a/target/arm/cpu.h +++ b/target/arm/cpu.h @@ -492,7 +492,7 @@ typedef struct CPUARMState { * the two execution states, and means we do not need to explicitly * map these registers when changing states. */ - float64 regs[64]; + float64 regs[64] QEMU_ALIGNED(16); uint32_t xregs[16]; /* We store these fpcsr fields separately for convenience. */ From patchwork Tue Nov 21 21:25:17 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 119419 Delivered-To: patch@linaro.org Received: by 10.140.22.164 with SMTP id 33csp5821726qgn; Tue, 21 Nov 2017 13:39:23 -0800 (PST) X-Google-Smtp-Source: AGs4zMYQqnG05ELgBAP3GaQH1C9rJ/uLIdXAdCdeF3GbasjuSTuK4r89dkzhs6F2N2cFwZGGTTRF X-Received: by 10.37.70.195 with SMTP id t186mr12151059yba.407.1511300363398; Tue, 21 Nov 2017 13:39:23 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1511300363; cv=none; d=google.com; s=arc-20160816; b=G83ngSsBrv7QJPjZSP5/dB6+f21zus858yw3ZPxE+tTKBIuADuIPJBHU8094MWXMg2 sx8AfyL1p+nhZpdAFC4+BcLUNuiq9Bj1tGNAH2Lw2NGC/7KGLwaf5dBGg3jr0sVLPtWB DXNur/3woYYlQrxh47pnBmLBIJzokowPsX1MykYnw94vD7Rmt2sbvcj/og4iBPo+f8wH kheIY1kvdCHnao3UD3gkj7DmrzYIhrEUOjA636cDS3bJejr/Jw0iQBAaiw+WLTMkf1bX 6Cxgqd8SR2WGzwO08zSw9FLr4YRyQyvFWYZrlC4cdSUSiXbjApn+KDxiGrTzLYgCmjh6 fPvg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:subject :content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:to:from:dkim-signature:arc-authentication-results; bh=Rrd4S97eVhWKdenLugOq5tq9MgHu7wRf2kAbYFyPVy8=; b=chqr/k6EM5LHSQw+IyvL0KFPTdFhic223QofZsqVDqpb6JF6GL0r9XqmkmOetDQKXk KRfwh5B5xgsZW51DscudkPKBkmeStpwmoMn0dgpqMIqfE2cOMHC2Cr5UnJfTC09wYB8D 7nyOoLe/e9pX8M0xb4NUgygevrDfIPyazlxf9shcPJPaxkzyj9zBIf2DRIO98LvP8B8R qDrSpi87xpMLw0EsBurz0Gc6+ylCUlfQxWd9pazU3SyLsecvSamFhNb+Rc8AxnUHUnD7 gq81jRihPSIxGZGDrhHvEvN2AUaR+GPoiHcCwCxCZjq0fbhGeE/TJHlhmeqQckTzAZf+ 8b7Q== ARC-Authentication-Results: i=1; mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=HtykI9Zf; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) smtp.mailfrom=qemu-devel-bounces+patch=linaro.org@nongnu.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [2001:4830:134:3::11]) by mx.google.com with ESMTPS id y77si3060740ybe.590.2017.11.21.13.39.23 for (version=TLS1 cipher=AES128-SHA bits=128/128); Tue, 21 Nov 2017 13:39:23 -0800 (PST) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) client-ip=2001:4830:134:3::11; Authentication-Results: mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=HtykI9Zf; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) smtp.mailfrom=qemu-devel-bounces+patch=linaro.org@nongnu.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1]:36636 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1eHGGI-0005yj-O6 for patch@linaro.org; Tue, 21 Nov 2017 16:39:22 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:54025) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1eHG59-0004Mr-0Z for qemu-devel@nongnu.org; Tue, 21 Nov 2017 16:27:52 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1eHG57-0007ho-Ff for qemu-devel@nongnu.org; Tue, 21 Nov 2017 16:27:50 -0500 Received: from mail-wm0-x241.google.com ([2a00:1450:400c:c09::241]:38057) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1eHG57-0007hL-52 for qemu-devel@nongnu.org; Tue, 21 Nov 2017 16:27:49 -0500 Received: by mail-wm0-x241.google.com with SMTP id 128so6267641wmo.3 for ; Tue, 21 Nov 2017 13:27:49 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=Rrd4S97eVhWKdenLugOq5tq9MgHu7wRf2kAbYFyPVy8=; b=HtykI9Zfzh+eKhbql3LZf4BPx/ybwj056POeUARhUso5xlSz/r/rApVGMYNXgpF45y 8JlVtXN0St+v/f0EdUovVHDjtWiRKkmdUKcItE0o/xb67POaWcX9fHgkTqxjlpxhmNug 5tn4jIRu8ra6fajVeUrTGbfPINmhL6pbdFgbc= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Rrd4S97eVhWKdenLugOq5tq9MgHu7wRf2kAbYFyPVy8=; b=JE/D7TorPtPv3wT8lXVC/vp+DQWgh81GKq7tnoACpGdsgeHuCOso+eYF2VHeMIGIJd quiUwey2SFF0ha8Pw93vnt5K1PUQgVaVPlRxuqaXUpXOiMsDyveCNUnNlyUY0fiaYvYE w8ckUAKIsoFowtpXVR/oDDSBdUj1sUfDCHKDsrRjR3dlsxm5OyBckeJYxJTtloRQzO0s Eeez5SvbOgHfMJ85u6TmO3MjkBEtDC9WQphlcthWDruU8rpkjNs3n3vTcfMunO2VWUQm u0UHZ2eo8K7sR1V/5KxXLphsjS+zZFTftYsWJdeILIL1Js725CcE+xupBo+GLbI1mJJ6 jU5Q== X-Gm-Message-State: AJaThX6+1ll/1g++BYcyUL/WuOcpPSwYZ2QdkjLwsXDUny+TdbaJS/ol ZeQn/uixQ/zdK7VVji3ZGln8Yt5gYdE= X-Received: by 10.28.238.74 with SMTP id m71mr2287517wmh.83.1511299667871; Tue, 21 Nov 2017 13:27:47 -0800 (PST) Received: from cloudburst.twiddle.net (70.red-37-158-60.dynamicip.rima-tde.net. [37.158.60.70]) by smtp.gmail.com with ESMTPSA id e124sm706517wmg.34.2017.11.21.13.27.46 for (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Tue, 21 Nov 2017 13:27:47 -0800 (PST) From: Richard Henderson To: qemu-devel@nongnu.org Date: Tue, 21 Nov 2017 22:25:17 +0100 Message-Id: <20171121212534.5177-10-richard.henderson@linaro.org> X-Mailer: git-send-email 2.13.6 In-Reply-To: <20171121212534.5177-1-richard.henderson@linaro.org> References: <20171121212534.5177-1-richard.henderson@linaro.org> MIME-Version: 1.0 X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2a00:1450:400c:c09::241 Subject: [Qemu-devel] [PATCH v6 09/26] target/arm: Use vector infrastructure for aa64 add/sub/logic X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: "Qemu-devel" Reviewed-by: Alex Bennée Signed-off-by: Richard Henderson --- target/arm/translate-a64.c | 207 +++++++++++++++++++++++++++++---------------- 1 file changed, 134 insertions(+), 73 deletions(-) -- 2.13.6 diff --git a/target/arm/translate-a64.c b/target/arm/translate-a64.c index ba94f7d045..572af456d1 100644 --- a/target/arm/translate-a64.c +++ b/target/arm/translate-a64.c @@ -21,6 +21,7 @@ #include "cpu.h" #include "exec/exec-all.h" #include "tcg-op.h" +#include "tcg-op-gvec.h" #include "qemu/log.h" #include "arm_ldst.h" #include "translate.h" @@ -83,6 +84,10 @@ typedef void NeonGenOneOpFn(TCGv_i64, TCGv_i64); typedef void CryptoTwoOpEnvFn(TCGv_ptr, TCGv_i32, TCGv_i32); typedef void CryptoThreeOpEnvFn(TCGv_ptr, TCGv_i32, TCGv_i32, TCGv_i32); +/* Note that the gvec expanders operate on offsets + sizes. */ +typedef void GVecGen3Fn(unsigned, uint32_t, uint32_t, + uint32_t, uint32_t, uint32_t); + /* initialize TCG globals. */ void a64_translate_init(void) { @@ -535,6 +540,21 @@ static inline int vec_reg_offset(DisasContext *s, int regno, return offs; } +/* Return the offset info CPUARMState of the "whole" vector register Qn. */ +static inline int vec_full_reg_offset(DisasContext *s, int regno) +{ + assert_fp_access_checked(s); + return offsetof(CPUARMState, vfp.regs[regno * 2]); +} + +/* Return the byte size of the "whole" vector register, VL / 8. */ +static inline int vec_full_reg_size(DisasContext *s) +{ + /* FIXME SVE: We should put the composite ZCR_EL* value into tb->flags. + In the meantime this is just the AdvSIMD length of 128. */ + return 128 / 8; +} + /* Return the offset into CPUARMState of a slice (from * the least significant end) of FP register Qn (ie * Dn, Sn, Hn or Bn). @@ -9048,85 +9068,125 @@ static void disas_simd_three_reg_diff(DisasContext *s, uint32_t insn) } } +static void gen_bsl_i64(TCGv_i64 rd, TCGv_i64 rn, TCGv_i64 rm) +{ + tcg_gen_xor_i64(rn, rn, rm); + tcg_gen_and_i64(rn, rn, rd); + tcg_gen_xor_i64(rd, rm, rn); +} + +static void gen_bit_i64(TCGv_i64 rd, TCGv_i64 rn, TCGv_i64 rm) +{ + tcg_gen_xor_i64(rn, rn, rd); + tcg_gen_and_i64(rn, rn, rm); + tcg_gen_xor_i64(rd, rd, rn); +} + +static void gen_bif_i64(TCGv_i64 rd, TCGv_i64 rn, TCGv_i64 rm) +{ + tcg_gen_xor_i64(rn, rn, rd); + tcg_gen_andc_i64(rn, rn, rm); + tcg_gen_xor_i64(rd, rd, rn); +} + +static void gen_bsl_vec(unsigned vece, TCGv_vec rd, TCGv_vec rn, TCGv_vec rm) +{ + tcg_gen_xor_vec(vece, rn, rn, rm); + tcg_gen_and_vec(vece, rn, rn, rd); + tcg_gen_xor_vec(vece, rd, rm, rn); +} + +static void gen_bit_vec(unsigned vece, TCGv_vec rd, TCGv_vec rn, TCGv_vec rm) +{ + tcg_gen_xor_vec(vece, rn, rn, rd); + tcg_gen_and_vec(vece, rn, rn, rm); + tcg_gen_xor_vec(vece, rd, rd, rn); +} + +static void gen_bif_vec(unsigned vece, TCGv_vec rd, TCGv_vec rn, TCGv_vec rm) +{ + tcg_gen_xor_vec(vece, rn, rn, rd); + tcg_gen_andc_vec(vece, rn, rn, rm); + tcg_gen_xor_vec(vece, rd, rd, rn); +} + /* Logic op (opcode == 3) subgroup of C3.6.16. */ static void disas_simd_3same_logic(DisasContext *s, uint32_t insn) { + static const GVecGen3 bsl_op = { + .fni8 = gen_bsl_i64, + .fniv = gen_bsl_vec, + .prefer_i64 = TCG_TARGET_REG_BITS == 64, + .load_dest = true + }; + static const GVecGen3 bit_op = { + .fni8 = gen_bit_i64, + .fniv = gen_bit_vec, + .prefer_i64 = TCG_TARGET_REG_BITS == 64, + .load_dest = true + }; + static const GVecGen3 bif_op = { + .fni8 = gen_bif_i64, + .fniv = gen_bif_vec, + .prefer_i64 = TCG_TARGET_REG_BITS == 64, + .load_dest = true + }; + int rd = extract32(insn, 0, 5); int rn = extract32(insn, 5, 5); int rm = extract32(insn, 16, 5); int size = extract32(insn, 22, 2); bool is_u = extract32(insn, 29, 1); bool is_q = extract32(insn, 30, 1); - TCGv_i64 tcg_op1, tcg_op2, tcg_res[2]; - int pass; + GVecGen3Fn *gvec_fn; + const GVecGen3 *gvec_op; if (!fp_access_check(s)) { return; } - tcg_op1 = tcg_temp_new_i64(); - tcg_op2 = tcg_temp_new_i64(); - tcg_res[0] = tcg_temp_new_i64(); - tcg_res[1] = tcg_temp_new_i64(); - - for (pass = 0; pass < (is_q ? 2 : 1); pass++) { - read_vec_element(s, tcg_op1, rn, pass, MO_64); - read_vec_element(s, tcg_op2, rm, pass, MO_64); - - if (!is_u) { - switch (size) { - case 0: /* AND */ - tcg_gen_and_i64(tcg_res[pass], tcg_op1, tcg_op2); - break; - case 1: /* BIC */ - tcg_gen_andc_i64(tcg_res[pass], tcg_op1, tcg_op2); - break; - case 2: /* ORR */ - tcg_gen_or_i64(tcg_res[pass], tcg_op1, tcg_op2); - break; - case 3: /* ORN */ - tcg_gen_orc_i64(tcg_res[pass], tcg_op1, tcg_op2); - break; - } - } else { - if (size != 0) { - /* B* ops need res loaded to operate on */ - read_vec_element(s, tcg_res[pass], rd, pass, MO_64); - } - - switch (size) { - case 0: /* EOR */ - tcg_gen_xor_i64(tcg_res[pass], tcg_op1, tcg_op2); - break; - case 1: /* BSL bitwise select */ - tcg_gen_xor_i64(tcg_op1, tcg_op1, tcg_op2); - tcg_gen_and_i64(tcg_op1, tcg_op1, tcg_res[pass]); - tcg_gen_xor_i64(tcg_res[pass], tcg_op2, tcg_op1); - break; - case 2: /* BIT, bitwise insert if true */ - tcg_gen_xor_i64(tcg_op1, tcg_op1, tcg_res[pass]); - tcg_gen_and_i64(tcg_op1, tcg_op1, tcg_op2); - tcg_gen_xor_i64(tcg_res[pass], tcg_res[pass], tcg_op1); - break; - case 3: /* BIF, bitwise insert if false */ - tcg_gen_xor_i64(tcg_op1, tcg_op1, tcg_res[pass]); - tcg_gen_andc_i64(tcg_op1, tcg_op1, tcg_op2); - tcg_gen_xor_i64(tcg_res[pass], tcg_res[pass], tcg_op1); - break; - } - } - } + switch (size + 4 * is_u) { + case 0: /* AND */ + gvec_fn = tcg_gen_gvec_and; + goto do_fn; + case 1: /* BIC */ + gvec_fn = tcg_gen_gvec_andc; + goto do_fn; + case 2: /* ORR */ + gvec_fn = tcg_gen_gvec_or; + goto do_fn; + case 3: /* ORN */ + gvec_fn = tcg_gen_gvec_orc; + goto do_fn; + case 4: /* EOR */ + gvec_fn = tcg_gen_gvec_xor; + goto do_fn; + do_fn: + gvec_fn(0, vec_full_reg_offset(s, rd), + vec_full_reg_offset(s, rn), + vec_full_reg_offset(s, rm), + is_q ? 16 : 8, vec_full_reg_size(s)); + return; + + case 5: /* BSL bitwise select */ + gvec_op = &bsl_op; + goto do_op; + case 6: /* BIT, bitwise insert if true */ + gvec_op = &bit_op; + goto do_op; + case 7: /* BIF, bitwise insert if false */ + gvec_op = &bif_op; + goto do_op; + do_op: + tcg_gen_gvec_3(vec_full_reg_offset(s, rd), + vec_full_reg_offset(s, rn), + vec_full_reg_offset(s, rm), + is_q ? 16 : 8, vec_full_reg_size(s), gvec_op); + return; - write_vec_element(s, tcg_res[0], rd, 0, MO_64); - if (!is_q) { - tcg_gen_movi_i64(tcg_res[1], 0); + default: + g_assert_not_reached(); } - write_vec_element(s, tcg_res[1], rd, 1, MO_64); - - tcg_temp_free_i64(tcg_op1); - tcg_temp_free_i64(tcg_op2); - tcg_temp_free_i64(tcg_res[0]); - tcg_temp_free_i64(tcg_res[1]); } /* Helper functions for 32 bit comparisons */ @@ -9387,6 +9447,7 @@ static void disas_simd_3same_int(DisasContext *s, uint32_t insn) int rn = extract32(insn, 5, 5); int rd = extract32(insn, 0, 5); int pass; + GVecGen3Fn *gvec_op; switch (opcode) { case 0x13: /* MUL, PMUL */ @@ -9426,6 +9487,16 @@ static void disas_simd_3same_int(DisasContext *s, uint32_t insn) return; } + switch (opcode) { + case 0x10: /* ADD, SUB */ + gvec_op = u ? tcg_gen_gvec_sub : tcg_gen_gvec_add; + gvec_op(size, vec_full_reg_offset(s, rd), + vec_full_reg_offset(s, rn), + vec_full_reg_offset(s, rm), + is_q ? 16 : 8, vec_full_reg_size(s)); + return; + } + if (size == 3) { assert(is_q); for (pass = 0; pass < 2; pass++) { @@ -9598,16 +9669,6 @@ static void disas_simd_3same_int(DisasContext *s, uint32_t insn) genfn = fns[size][u]; break; } - case 0x10: /* ADD, SUB */ - { - static NeonGenTwoOpFn * const fns[3][2] = { - { gen_helper_neon_add_u8, gen_helper_neon_sub_u8 }, - { gen_helper_neon_add_u16, gen_helper_neon_sub_u16 }, - { tcg_gen_add_i32, tcg_gen_sub_i32 }, - }; - genfn = fns[size][u]; - break; - } case 0x11: /* CMTST, CMEQ */ { static NeonGenTwoOpFn * const fns[3][2] = { From patchwork Tue Nov 21 21:25:18 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 119408 Delivered-To: patch@linaro.org Received: by 10.140.22.164 with SMTP id 33csp5815936qgn; Tue, 21 Nov 2017 13:33:12 -0800 (PST) X-Google-Smtp-Source: AGs4zMbhGDTm2WD7FqeF6YpAaPWUfE4buomC86qy8ObXvNo8cawD1orqJQyCAEXCCW1pAI9kZV4T X-Received: by 10.129.228.73 with SMTP id t9mr12453188ywl.112.1511299992253; Tue, 21 Nov 2017 13:33:12 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1511299992; cv=none; d=google.com; s=arc-20160816; b=WWG8B17QuAzBROJh/tM65eVpBg5f0u8mTOqQeB5FxxBwxf+II/YMeeFkeB5wQzPmR/ ieytOs4t/zdeAfekHsr2776zht0EerNq8nEmGIRH+KSLaLDXxb+Y55B1KBARuzYSgi6d v3Suff8r8Mtt4wpmyEp88CiAtXtPTbKsZzZmQIO2dvN+C9wFjIcKTrVMl2R/Ra2j1VDo dx8LpwqgzlgHDMqMSnp3SADtFba+zvdcvnS6T/LBMH7dIKIDRZ9/chqPv4G5JaUoXJmE XEBeCUub0ZVz1CsUPv7OjSdiuqC/glc1d/zaXEiJ4UeY/rNWLLGA6fGjWrtg493IMQ8U OMdg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:subject:references:in-reply-to :message-id:date:to:from:dkim-signature:arc-authentication-results; bh=09qF8n03XSVo4FEyGsP2EEOlkKHL3L4/cRQSJWq9xv0=; b=DC2JFJSRufBd6L9FqM2So3QNgs/H2gdKXmvCKJbE84SzO7HfXTZV0tjNrqaeSHaEOe zd1+w0sgFRM6n8Nhf+ltauu5qs1InH2wbc7AavU4JBzgSiBSPSnNrbFjwGupG9xiWH6E 0GwOXDbHB92ep3YjjBdLbylUD7ovXia+qrnvxvK1x7nqeM2iviGogA3NixMS2GH/5X02 wGCf1GnjRLH4SZ+1IuGTw5E5go6hxnbU+Xy70gXq84CleUzUtT4Kho7qcyD6Tcdh2tJB e6uRLTGZe6iCTIlUek5hoMROjQuhxG+xwEKGfpa604W7FOvG3TZv3ALZaptMMWD/wXoU Z+CA== ARC-Authentication-Results: i=1; mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=C5x9VPLi; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) smtp.mailfrom=qemu-devel-bounces+patch=linaro.org@nongnu.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [2001:4830:134:3::11]) by mx.google.com with ESMTPS id j17si2911440ybm.556.2017.11.21.13.33.12 for (version=TLS1 cipher=AES128-SHA bits=128/128); Tue, 21 Nov 2017 13:33:12 -0800 (PST) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) client-ip=2001:4830:134:3::11; Authentication-Results: mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=C5x9VPLi; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) smtp.mailfrom=qemu-devel-bounces+patch=linaro.org@nongnu.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1]:36599 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1eHGAJ-0007nZ-Kr for patch@linaro.org; Tue, 21 Nov 2017 16:33:11 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:54036) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1eHG59-0004NJ-GO for qemu-devel@nongnu.org; Tue, 21 Nov 2017 16:27:52 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1eHG58-0007iM-E4 for qemu-devel@nongnu.org; Tue, 21 Nov 2017 16:27:51 -0500 Received: from mail-wm0-x242.google.com ([2a00:1450:400c:c09::242]:38058) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1eHG58-0007hv-8l for qemu-devel@nongnu.org; Tue, 21 Nov 2017 16:27:50 -0500 Received: by mail-wm0-x242.google.com with SMTP id 128so6267720wmo.3 for ; Tue, 21 Nov 2017 13:27:50 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:subject:date:message-id:in-reply-to:references; bh=09qF8n03XSVo4FEyGsP2EEOlkKHL3L4/cRQSJWq9xv0=; b=C5x9VPLi7Frt5tMOoZVph4BCaCr3C6338NYRRS/6RZnwjfHTdK+oSey5YlSXginpxT jG9pewgr8Ic50CjOg6EzMhBJTWtCr1ZpjoWtJKd0/N6j++5T1K43y4euEB7Lc1cgYp16 RM37E2jF7yp5F1wNFc2S2w/rsEpJ+gnFrqek4= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references; bh=09qF8n03XSVo4FEyGsP2EEOlkKHL3L4/cRQSJWq9xv0=; b=r6HRYqZe8Jov4hvYAMXLH9XKh2E6X67l02EdvApFIFe/1ZYSKmjVMVjBP12jLRJL39 qfSVqAbaUrI+4FpIIqcPsTQTudc8H7UPjN9EzkYEu1DrCsVQA1SjXJU0xmKL7zmQqR20 CRpA4LzwYu2O0Vy3Ix58I+TWQL85h3z/qzWCJixbdkRBL+xRR7lG4jHNzmTLfWfQmH2e mFP0Y0nVcGo3w9BuvVkQflpaNPCtGppV5eVTSmUzceV7Dc8ALcYVGXj13OFQlne5bpwj PPSenFVv/YbQxbPqRbpT38qVb+tc9YCiO1aLngeovKDNS2MZ32VZOg+VtUtBJ01qLc9R fErQ== X-Gm-Message-State: AJaThX6CaBVTZavEBibZ83jw6IPJQmYQW/mzLOVxW9mlUwv1/q/LQJP0 b+DP2kgIJPYxz2fhm05kKDmzfeMPeq4= X-Received: by 10.28.146.20 with SMTP id u20mr2620090wmd.49.1511299669023; Tue, 21 Nov 2017 13:27:49 -0800 (PST) Received: from cloudburst.twiddle.net (70.red-37-158-60.dynamicip.rima-tde.net. [37.158.60.70]) by smtp.gmail.com with ESMTPSA id e124sm706517wmg.34.2017.11.21.13.27.47 for (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Tue, 21 Nov 2017 13:27:48 -0800 (PST) From: Richard Henderson To: qemu-devel@nongnu.org Date: Tue, 21 Nov 2017 22:25:18 +0100 Message-Id: <20171121212534.5177-11-richard.henderson@linaro.org> X-Mailer: git-send-email 2.13.6 In-Reply-To: <20171121212534.5177-1-richard.henderson@linaro.org> References: <20171121212534.5177-1-richard.henderson@linaro.org> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2a00:1450:400c:c09::242 Subject: [Qemu-devel] [PATCH v6 10/26] target/arm: Use vector infrastructure for aa64 mov/not/neg X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: "Qemu-devel" Signed-off-by: Richard Henderson --- target/arm/translate-a64.c | 43 ++++++++++++++++++++++++++++++++++++++----- 1 file changed, 38 insertions(+), 5 deletions(-) -- 2.13.6 diff --git a/target/arm/translate-a64.c b/target/arm/translate-a64.c index 572af456d1..bc14c28e71 100644 --- a/target/arm/translate-a64.c +++ b/target/arm/translate-a64.c @@ -85,6 +85,7 @@ typedef void CryptoTwoOpEnvFn(TCGv_ptr, TCGv_i32, TCGv_i32); typedef void CryptoThreeOpEnvFn(TCGv_ptr, TCGv_i32, TCGv_i32, TCGv_i32); /* Note that the gvec expanders operate on offsets + sizes. */ +typedef void GVecGen2Fn(unsigned, uint32_t, uint32_t, uint32_t, uint32_t); typedef void GVecGen3Fn(unsigned, uint32_t, uint32_t, uint32_t, uint32_t, uint32_t); @@ -4579,14 +4580,19 @@ static void handle_fp_1src_double(DisasContext *s, int opcode, int rd, int rn) TCGv_i64 tcg_op; TCGv_i64 tcg_res; + switch (opcode) { + case 0x0: /* FMOV */ + tcg_gen_gvec_mov(0, vec_full_reg_offset(s, rd), + vec_full_reg_offset(s, rn), + 8, vec_full_reg_size(s)); + return; + } + fpst = get_fpstatus_ptr(); tcg_op = read_fp_dreg(s, rn); tcg_res = tcg_temp_new_i64(); switch (opcode) { - case 0x0: /* FMOV */ - tcg_gen_mov_i64(tcg_res, tcg_op); - break; case 0x1: /* FABS */ gen_helper_vfp_absd(tcg_res, tcg_op); break; @@ -9153,6 +9159,12 @@ static void disas_simd_3same_logic(DisasContext *s, uint32_t insn) gvec_fn = tcg_gen_gvec_andc; goto do_fn; case 2: /* ORR */ + if (rn == rm) { /* MOV */ + tcg_gen_gvec_mov(0, vec_full_reg_offset(s, rd), + vec_full_reg_offset(s, rn), + is_q ? 16 : 8, vec_full_reg_size(s)); + return; + } gvec_fn = tcg_gen_gvec_or; goto do_fn; case 3: /* ORN */ @@ -10032,6 +10044,7 @@ static void disas_simd_two_reg_misc(DisasContext *s, uint32_t insn) int rmode = -1; TCGv_i32 tcg_rmode; TCGv_ptr tcg_fpstatus; + GVecGen2Fn *gvec_fn; switch (opcode) { case 0x0: /* REV64, REV32 */ @@ -10040,8 +10053,7 @@ static void disas_simd_two_reg_misc(DisasContext *s, uint32_t insn) return; case 0x5: /* CNT, NOT, RBIT */ if (u && size == 0) { - /* NOT: adjust size so we can use the 64-bits-at-a-time loop. */ - size = 3; + /* NOT */ break; } else if (u && size == 1) { /* RBIT */ @@ -10293,6 +10305,27 @@ static void disas_simd_two_reg_misc(DisasContext *s, uint32_t insn) tcg_rmode = NULL; } + switch (opcode) { + case 0x5: + if (u && size == 0) { /* NOT */ + gvec_fn = tcg_gen_gvec_not; + goto do_fn; + } + break; + case 0xb: + if (u) { /* NEG */ + gvec_fn = tcg_gen_gvec_neg; + goto do_fn; + } + break; + + do_fn: + gvec_fn(size, vec_full_reg_offset(s, rd), + vec_full_reg_offset(s, rn), + is_q ? 16 : 8, vec_full_reg_size(s)); + return; + } + if (size == 3) { /* All 64-bit element operations can be shared with scalar 2misc */ int pass; From patchwork Tue Nov 21 21:25:19 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 119406 Delivered-To: patch@linaro.org Received: by 10.140.22.164 with SMTP id 33csp5814362qgn; Tue, 21 Nov 2017 13:31:33 -0800 (PST) X-Google-Smtp-Source: AGs4zMbhbqxmSl7x5B3htLYjZQqHb0i2eSWOR0kw8IPx+vrzeLCx2ePgIIhqUtdaNaqEanHC+vyL X-Received: by 10.37.113.4 with SMTP id m4mr7470483ybc.206.1511299893622; Tue, 21 Nov 2017 13:31:33 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1511299893; cv=none; d=google.com; s=arc-20160816; b=AEjrQWrIoskDa/hUgwoqCwLBWgRXGenECbUpnE8vVhkBUMUHHtVX5Z0/rNHOuXQuiD pLg3R6PNP5vwhT4yrF3SQ+IRl69OyiOIBmNJvjN5Y4xYf+jo8DsinMJJ8KOMpXczrAm5 4a9N/R6/N0QDarKyA4ywkOPXZQ7UltBe3/9WxhXK4sbLt2JvUKfp2UXfsvZyfPNYQRCB gffWMU6VyRsKhi0VWYGA0p16JuiSYZplEXxBIkixjxoPzhAvTFkjMB6+3dLCDy+f+QW3 e/Yyuo4Ud6G6O4EP4+4/ljUpieoHDUmpbmcCwLst5QF4z9GQKPt7WeeGO/9dBbsBBMvI F0Ow== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:subject:references:in-reply-to :message-id:date:to:from:dkim-signature:arc-authentication-results; bh=hQ/lP9KQaCheo4UPX9JOhyv6XEAkTWCkibaqYs2X2J8=; b=e9UkvyNp/5ddcTkmr7WVFgpDQhj4iqgvnV8CMlwTf8ceyua+QOf14ELnfLzM8KLAH8 BYbfbAoHZJuzGxmgvEICFpGnOY09ackAMfQNe75vcXlbpPpM0g/agkOnYTbKY3dKZaDV P8Ek0PMR+MbsFDJsEpU0JaeZF7vtx6luDnsxXGSjnqz1XvTeG3jy/LfzDUXwOXVr0Ixe vANWaDXDkyM3V0pWuh2txq3xS0agorFopGTE1kT9e6djElN9vbi/I7IAW0C5DJTp2aTR DGF6xuiZxDjdQgGtt5QQAVr+DBAN90pRNnaVHN78NbVJiFdNVd9yo9ePJMlerV4mVwPI +d3Q== ARC-Authentication-Results: i=1; mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=a328D2hY; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) smtp.mailfrom=qemu-devel-bounces+patch=linaro.org@nongnu.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [2001:4830:134:3::11]) by mx.google.com with ESMTPS id e130si3149071ywh.691.2017.11.21.13.31.33 for (version=TLS1 cipher=AES128-SHA bits=128/128); Tue, 21 Nov 2017 13:31:33 -0800 (PST) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) client-ip=2001:4830:134:3::11; Authentication-Results: mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=a328D2hY; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) smtp.mailfrom=qemu-devel-bounces+patch=linaro.org@nongnu.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1]:36587 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1eHG8j-0006Sf-2c for patch@linaro.org; Tue, 21 Nov 2017 16:31:33 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:54058) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1eHG5A-0004Ok-Nk for qemu-devel@nongnu.org; Tue, 21 Nov 2017 16:27:53 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1eHG59-0007jF-KW for qemu-devel@nongnu.org; Tue, 21 Nov 2017 16:27:52 -0500 Received: from mail-wm0-x243.google.com ([2a00:1450:400c:c09::243]:38918) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1eHG59-0007ie-Ci for qemu-devel@nongnu.org; Tue, 21 Nov 2017 16:27:51 -0500 Received: by mail-wm0-x243.google.com with SMTP id x63so6255618wmf.4 for ; Tue, 21 Nov 2017 13:27:51 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:subject:date:message-id:in-reply-to:references; bh=hQ/lP9KQaCheo4UPX9JOhyv6XEAkTWCkibaqYs2X2J8=; b=a328D2hYF+BNV/h1W16BdCIyOruiWN6SE3xTVS0PJZzlbEoZoVp+s5646/4zswHr50 +kRgoGxeB6iUeJm+twcF/4o4qH76M029e/GG0p1zG7LKJtjJpeTDnGVk6lGAToNY8Txy m4SgkFonpW0+SlsddxEwCUKAUKtf3pJ3CQeWM= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references; bh=hQ/lP9KQaCheo4UPX9JOhyv6XEAkTWCkibaqYs2X2J8=; b=akIMAa49JFDmfa9EoyE3+OBbTgnLdivdK3OTj4ZfYivbTYIDci/XU4GrNU+GvX3BLL ZZKyVnZ5SctY6mgCvCwtPGD0vDEuki4rx28RdN1OHXa3S5zYwkmajCJZrQExLFfS9FDR NkOsXmUmEV8Mpe2al3sscLTxOB9zH+J425L6vxlj4atJ0oYMWQ4SjL6nmgW/CCLihinx WbuYRXE4LhpjDnS2ZyXlatGZabLljP5SCW716/6PH4a0AtM/IWLfp47SiK7BK0yy3dgM v+tdqNeK2hWZDlx2ZVICuQmXPB3lQUyj1CFAzaJ4sUaXfwxeR56Bk9SeaI66DHI+kVN/ pX0Q== X-Gm-Message-State: AJaThX632/x9bHc1UCNKOvAoE90TQaWKvW+RmhVaINXd5U38pTbfS49w 2vPE/S1juf2U1/fzhpTRoeINNnOIA6U= X-Received: by 10.28.14.195 with SMTP id 186mr2384135wmo.56.1511299670144; Tue, 21 Nov 2017 13:27:50 -0800 (PST) Received: from cloudburst.twiddle.net (70.red-37-158-60.dynamicip.rima-tde.net. [37.158.60.70]) by smtp.gmail.com with ESMTPSA id e124sm706517wmg.34.2017.11.21.13.27.49 for (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Tue, 21 Nov 2017 13:27:49 -0800 (PST) From: Richard Henderson To: qemu-devel@nongnu.org Date: Tue, 21 Nov 2017 22:25:19 +0100 Message-Id: <20171121212534.5177-12-richard.henderson@linaro.org> X-Mailer: git-send-email 2.13.6 In-Reply-To: <20171121212534.5177-1-richard.henderson@linaro.org> References: <20171121212534.5177-1-richard.henderson@linaro.org> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2a00:1450:400c:c09::243 Subject: [Qemu-devel] [PATCH v6 11/26] target/arm: Use vector infrastructure for aa64 dup/movi X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: "Qemu-devel" Signed-off-by: Richard Henderson --- target/arm/translate-a64.c | 83 +++++++++++++++++++--------------------------- 1 file changed, 34 insertions(+), 49 deletions(-) -- 2.13.6 diff --git a/target/arm/translate-a64.c b/target/arm/translate-a64.c index bc14c28e71..55a4902fc2 100644 --- a/target/arm/translate-a64.c +++ b/target/arm/translate-a64.c @@ -5846,38 +5846,24 @@ static void disas_simd_across_lanes(DisasContext *s, uint32_t insn) * * size: encoded in imm5 (see ARM ARM LowestSetBit()) */ + static void handle_simd_dupe(DisasContext *s, int is_q, int rd, int rn, int imm5) { int size = ctz32(imm5); - int esize = 8 << size; - int elements = (is_q ? 128 : 64) / esize; - int index, i; - TCGv_i64 tmp; + int index = imm5 >> (size + 1); if (size > 3 || (size == 3 && !is_q)) { unallocated_encoding(s); return; } - if (!fp_access_check(s)) { return; } - index = imm5 >> (size + 1); - - tmp = tcg_temp_new_i64(); - read_vec_element(s, tmp, rn, index, size); - - for (i = 0; i < elements; i++) { - write_vec_element(s, tmp, rd, i, size); - } - - if (!is_q) { - clear_vec_high(s, rd); - } - - tcg_temp_free_i64(tmp); + tcg_gen_gvec_dup_mem(size, vec_full_reg_offset(s, rd), + vec_reg_offset(s, rn, index, size), + is_q ? 16 : 8, vec_full_reg_size(s)); } /* DUP (element, scalar) @@ -5926,9 +5912,7 @@ static void handle_simd_dupg(DisasContext *s, int is_q, int rd, int rn, int imm5) { int size = ctz32(imm5); - int esize = 8 << size; - int elements = (is_q ? 128 : 64)/esize; - int i = 0; + uint32_t dofs, oprsz, maxsz; if (size > 3 || ((size == 3) && !is_q)) { unallocated_encoding(s); @@ -5939,12 +5923,11 @@ static void handle_simd_dupg(DisasContext *s, int is_q, int rd, int rn, return; } - for (i = 0; i < elements; i++) { - write_vec_element(s, cpu_reg(s, rn), rd, i, size); - } - if (!is_q) { - clear_vec_high(s, rd); - } + dofs = vec_full_reg_offset(s, rd); + oprsz = is_q ? 16 : 8; + maxsz = vec_full_reg_size(s); + + tcg_gen_gvec_dup_i64(size, dofs, oprsz, maxsz, cpu_reg(s, rn)); } /* INS (Element) @@ -6135,7 +6118,6 @@ static void disas_simd_mod_imm(DisasContext *s, uint32_t insn) bool is_neg = extract32(insn, 29, 1); bool is_q = extract32(insn, 30, 1); uint64_t imm = 0; - TCGv_i64 tcg_rd, tcg_imm; int i; if (o2 != 0 || ((cmode == 0xf) && is_neg && !is_q)) { @@ -6217,32 +6199,35 @@ static void disas_simd_mod_imm(DisasContext *s, uint32_t insn) imm = ~imm; } - tcg_imm = tcg_const_i64(imm); - tcg_rd = new_tmp_a64(s); + if (!((cmode & 0x9) == 0x1 || (cmode & 0xd) == 0x9)) { + /* MOVI or MVNI, with MVNI negation handled above. */ + tcg_gen_gvec_dup64i(vec_full_reg_offset(s, rd), is_q ? 16 : 8, + vec_full_reg_size(s), imm); + } else { + TCGv_i64 tcg_imm = tcg_const_i64(imm); + TCGv_i64 tcg_rd = new_tmp_a64(s); - for (i = 0; i < 2; i++) { - int foffs = i ? fp_reg_hi_offset(s, rd) : fp_reg_offset(s, rd, MO_64); + for (i = 0; i < 2; i++) { + int foffs = vec_reg_offset(s, rd, i, MO_64); - if (i == 1 && !is_q) { - /* non-quad ops clear high half of vector */ - tcg_gen_movi_i64(tcg_rd, 0); - } else if ((cmode & 0x9) == 0x1 || (cmode & 0xd) == 0x9) { - tcg_gen_ld_i64(tcg_rd, cpu_env, foffs); - if (is_neg) { - /* AND (BIC) */ - tcg_gen_and_i64(tcg_rd, tcg_rd, tcg_imm); + if (i == 1 && !is_q) { + /* non-quad ops clear high half of vector */ + tcg_gen_movi_i64(tcg_rd, 0); } else { - /* ORR */ - tcg_gen_or_i64(tcg_rd, tcg_rd, tcg_imm); + tcg_gen_ld_i64(tcg_rd, cpu_env, foffs); + if (is_neg) { + /* AND (BIC) */ + tcg_gen_and_i64(tcg_rd, tcg_rd, tcg_imm); + } else { + /* ORR */ + tcg_gen_or_i64(tcg_rd, tcg_rd, tcg_imm); + } } - } else { - /* MOVI */ - tcg_gen_mov_i64(tcg_rd, tcg_imm); + tcg_gen_st_i64(tcg_rd, cpu_env, foffs); } - tcg_gen_st_i64(tcg_rd, cpu_env, foffs); - } - tcg_temp_free_i64(tcg_imm); + tcg_temp_free_i64(tcg_imm); + } } /* AdvSIMD scalar copy From patchwork Tue Nov 21 21:25:20 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 119421 Delivered-To: patch@linaro.org Received: by 10.140.22.164 with SMTP id 33csp5824044qgn; Tue, 21 Nov 2017 13:42:04 -0800 (PST) X-Google-Smtp-Source: AGs4zMbYa6XtBNPjaXta6IPnS1Z2lHdo3TFYOQ+e2Kf5zMwNmNzdP49IgCYwSoO3W8WxztLWs/Dp X-Received: by 10.37.212.79 with SMTP id m76mr9742827ybf.184.1511300524540; Tue, 21 Nov 2017 13:42:04 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1511300524; cv=none; d=google.com; s=arc-20160816; b=OhXMjqFyLm0oveCTTm1oJyWw0Vm4lFiHr+pwr64a64AvYs/xTQIKw8zsAKxYfouzQZ EmraMyFmU/Hm7eJjOBmwJupCYgnX5t81ILetkWNGi9qOjWa2HsV6E5djs07jFeIwAiIn V36ubP9yxmGaOmZ8G1zLRQPOfgP07fo3TsFW8/rSDLCFt08VvQpBLIwLUwqSddD6LyZI ctzIxbvzwmCm5s2JsnUP/XOFEtc9chZX/LApnaEi9bCRQWPdw+VEDn/1uhGagLIUq7Dn vTnfR12UZKFPXmDRkqgIPT0UvgGwrrLuSPX0XZV0kviYDz9bbJhOM2VlJFsZOZqBfFa6 4Yjw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:subject:references:in-reply-to :message-id:date:to:from:dkim-signature:arc-authentication-results; bh=WLPyy+fACvvR0vbdJcOHb6E8KxcywHrep+6Emw+8Lz4=; b=UJA1W6jeRjQNVJKkf/rc8zSJn8GJFHW3Pct296vTkqWQwI7Ahik87gtTn3TJ5aJMxO yCrIzYYx6W3BnRwsGOpGh6MxQl4/5m73BsRG4fnj4srtS5OCArtskL3/wkO2jQRUnSil gloyUpSJW+KKIAqEX2mnq3rFy/rRUtoT53D+hOj2Iu8aG9v7gw0Pb19FJQyJ1KUc+IXL 4BQTU52DahPwfEBS2doeJOhJoRsGqYwoL7eNxk7EZDHVoVYnX6lzXM01kQg9dzZjZDNe jAQoDfURF3bGevqmrCEb+X0qXqf5JTHQtLfozdYgMpDTMX0SIruZyFz+E5xFlLpSngFj aNAg== ARC-Authentication-Results: i=1; mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=O+aJLNCi; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) smtp.mailfrom=qemu-devel-bounces+patch=linaro.org@nongnu.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [2001:4830:134:3::11]) by mx.google.com with ESMTPS id w136si3037830ywa.168.2017.11.21.13.42.04 for (version=TLS1 cipher=AES128-SHA bits=128/128); Tue, 21 Nov 2017 13:42:04 -0800 (PST) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) client-ip=2001:4830:134:3::11; Authentication-Results: mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=O+aJLNCi; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) smtp.mailfrom=qemu-devel-bounces+patch=linaro.org@nongnu.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1]:36655 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1eHGIu-0008J5-1Q for patch@linaro.org; Tue, 21 Nov 2017 16:42:04 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:54101) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1eHG5E-0004Sp-Ua for qemu-devel@nongnu.org; Tue, 21 Nov 2017 16:27:59 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1eHG5B-0007jr-LT for qemu-devel@nongnu.org; Tue, 21 Nov 2017 16:27:56 -0500 Received: from mail-wm0-x242.google.com ([2a00:1450:400c:c09::242]:36802) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1eHG5B-0007jZ-8X for qemu-devel@nongnu.org; Tue, 21 Nov 2017 16:27:53 -0500 Received: by mail-wm0-x242.google.com with SMTP id r68so6323978wmr.1 for ; Tue, 21 Nov 2017 13:27:53 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:subject:date:message-id:in-reply-to:references; bh=WLPyy+fACvvR0vbdJcOHb6E8KxcywHrep+6Emw+8Lz4=; b=O+aJLNCiOorL60DBfiXoteBQ66i/jZ0nYs4qlgTiBgoTvBepwxJn7pCAdCg3i6whVL 8aNWLy/CxqSXlsG/Ya9VPID9/esyoXEvY4v4l363r7H2Lgb+r5tb14bP/b6nHtQXphBY WCYfsOtqo6v5W5ubKtiysSxEo+uK4BgkyrekM= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references; bh=WLPyy+fACvvR0vbdJcOHb6E8KxcywHrep+6Emw+8Lz4=; b=SkE1x/4zIe87Wa9sNcri0+0+LE1AABgXxqutxqVLthYL9c2f12sI8tK1Vt2ftb0qZL T1OnSMOtljCHtaLc+08YWX4gmJRez7VsfsHIBB8tmNXE+GAEwyjD1DxnmDi7F1PZKCTS 4wDKFsSx1EnoJrhyIAUp96vNust1npcp77fdwbYXluhMADta9HWNzhytjkDeI/t4gkPj H6dJ3esLoy2Ij7TgrGrUwl+ENjS+m1LOs7jgeJSwB5aR21RfXjrFc5XpYpve9mqkaSkH dC1dnViZwDqSOE6QM9TVxlrAbvGukXLHt/KO0kYz+Yieq24I4558IvYcBVgtXDmfyt0o Dd8A== X-Gm-Message-State: AJaThX7bTtfWOo0D6/x+Y2RaXjPuxYnpDCo8GoesoBRnUUBi1ypYhs4w YDImIzO63rPo+tBrphya7f2yWIrLAJg= X-Received: by 10.28.1.14 with SMTP id 14mr2506466wmb.51.1511299671375; Tue, 21 Nov 2017 13:27:51 -0800 (PST) Received: from cloudburst.twiddle.net (70.red-37-158-60.dynamicip.rima-tde.net. [37.158.60.70]) by smtp.gmail.com with ESMTPSA id e124sm706517wmg.34.2017.11.21.13.27.50 for (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Tue, 21 Nov 2017 13:27:50 -0800 (PST) From: Richard Henderson To: qemu-devel@nongnu.org Date: Tue, 21 Nov 2017 22:25:20 +0100 Message-Id: <20171121212534.5177-13-richard.henderson@linaro.org> X-Mailer: git-send-email 2.13.6 In-Reply-To: <20171121212534.5177-1-richard.henderson@linaro.org> References: <20171121212534.5177-1-richard.henderson@linaro.org> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2a00:1450:400c:c09::242 Subject: [Qemu-devel] [PATCH v6 12/26] tcg/i386: Add vector operations X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: "Qemu-devel" The x86 vector instruction set is extremely irregular. With newer editions, Intel has filled in some of the blanks. However, we don't get many 64-bit operations until SSE4.2, introduced in 2009. The subsequent edition was for AVX1, introduced in 2011, which added three-operand addressing, and adjusts how all instructions should be encoded. Given the relatively narrow 2 year window between possible to support and desirable to support, and to vastly simplify code maintainence, I am only planning to support AVX1 and later cpus. Signed-off-by: Richard Henderson --- tcg/i386/tcg-target.h | 36 ++- tcg/i386/tcg-target.inc.c | 561 ++++++++++++++++++++++++++++++++++++++++++---- 2 files changed, 546 insertions(+), 51 deletions(-) -- 2.13.6 diff --git a/tcg/i386/tcg-target.h b/tcg/i386/tcg-target.h index b89dababf4..f9d3fc4a93 100644 --- a/tcg/i386/tcg-target.h +++ b/tcg/i386/tcg-target.h @@ -30,10 +30,10 @@ #ifdef __x86_64__ # define TCG_TARGET_REG_BITS 64 -# define TCG_TARGET_NB_REGS 16 +# define TCG_TARGET_NB_REGS 32 #else # define TCG_TARGET_REG_BITS 32 -# define TCG_TARGET_NB_REGS 8 +# define TCG_TARGET_NB_REGS 24 #endif typedef enum { @@ -56,6 +56,26 @@ typedef enum { TCG_REG_R13, TCG_REG_R14, TCG_REG_R15, + + TCG_REG_XMM0, + TCG_REG_XMM1, + TCG_REG_XMM2, + TCG_REG_XMM3, + TCG_REG_XMM4, + TCG_REG_XMM5, + TCG_REG_XMM6, + TCG_REG_XMM7, + + /* 64-bit registers; likewise always define. */ + TCG_REG_XMM8, + TCG_REG_XMM9, + TCG_REG_XMM10, + TCG_REG_XMM11, + TCG_REG_XMM12, + TCG_REG_XMM13, + TCG_REG_XMM14, + TCG_REG_XMM15, + TCG_REG_RAX = TCG_REG_EAX, TCG_REG_RCX = TCG_REG_ECX, TCG_REG_RDX = TCG_REG_EDX, @@ -77,6 +97,8 @@ typedef enum { extern bool have_bmi1; extern bool have_popcnt; +extern bool have_avx1; +extern bool have_avx2; /* optional instructions */ #define TCG_TARGET_HAS_div2_i32 1 @@ -146,6 +168,16 @@ extern bool have_popcnt; #define TCG_TARGET_HAS_mulsh_i64 0 #endif +/* We do not support older SSE systems, only beginning with AVX1. */ +#define TCG_TARGET_HAS_v64 have_avx1 +#define TCG_TARGET_HAS_v128 have_avx1 +#define TCG_TARGET_HAS_v256 have_avx2 + +#define TCG_TARGET_HAS_andc_vec 1 +#define TCG_TARGET_HAS_orc_vec 0 +#define TCG_TARGET_HAS_not_vec 0 +#define TCG_TARGET_HAS_neg_vec 0 + #define TCG_TARGET_deposit_i32_valid(ofs, len) \ (((ofs) == 0 && (len) == 8) || ((ofs) == 8 && (len) == 8) || \ ((ofs) == 0 && (len) == 16)) diff --git a/tcg/i386/tcg-target.inc.c b/tcg/i386/tcg-target.inc.c index 63d27f10e7..e9a4d92598 100644 --- a/tcg/i386/tcg-target.inc.c +++ b/tcg/i386/tcg-target.inc.c @@ -28,10 +28,15 @@ static const char * const tcg_target_reg_names[TCG_TARGET_NB_REGS] = { #if TCG_TARGET_REG_BITS == 64 "%rax", "%rcx", "%rdx", "%rbx", "%rsp", "%rbp", "%rsi", "%rdi", - "%r8", "%r9", "%r10", "%r11", "%r12", "%r13", "%r14", "%r15", #else "%eax", "%ecx", "%edx", "%ebx", "%esp", "%ebp", "%esi", "%edi", #endif + "%r8", "%r9", "%r10", "%r11", "%r12", "%r13", "%r14", "%r15", + "%xmm0", "%xmm1", "%xmm2", "%xmm3", "%xmm4", "%xmm5", "%xmm6", "%xmm7", +#if TCG_TARGET_REG_BITS == 64 + "%xmm8", "%xmm9", "%xmm10", "%xmm11", + "%xmm12", "%xmm13", "%xmm14", "%xmm15", +#endif }; #endif @@ -61,6 +66,28 @@ static const int tcg_target_reg_alloc_order[] = { TCG_REG_EDX, TCG_REG_EAX, #endif + TCG_REG_XMM0, + TCG_REG_XMM1, + TCG_REG_XMM2, + TCG_REG_XMM3, + TCG_REG_XMM4, + TCG_REG_XMM5, +#ifndef _WIN64 + /* The Win64 ABI has xmm6-xmm15 as caller-saves, and we do not save + any of them. Therefore only allow xmm0-xmm5 to be allocated. */ + TCG_REG_XMM6, + TCG_REG_XMM7, +#if TCG_TARGET_REG_BITS == 64 + TCG_REG_XMM8, + TCG_REG_XMM9, + TCG_REG_XMM10, + TCG_REG_XMM11, + TCG_REG_XMM12, + TCG_REG_XMM13, + TCG_REG_XMM14, + TCG_REG_XMM15, +#endif +#endif }; static const int tcg_target_call_iarg_regs[] = { @@ -94,7 +121,7 @@ static const int tcg_target_call_oarg_regs[] = { #define TCG_CT_CONST_I32 0x400 #define TCG_CT_CONST_WSZ 0x800 -/* Registers used with L constraint, which are the first argument +/* Registers used with L constraint, which are the first argument registers on x86_64, and two random call clobbered registers on i386. */ #if TCG_TARGET_REG_BITS == 64 @@ -125,6 +152,8 @@ static bool have_cmov; it there. Therefore we always define the variable. */ bool have_bmi1; bool have_popcnt; +bool have_avx1; +bool have_avx2; #ifdef CONFIG_CPUID_H static bool have_movbe; @@ -148,6 +177,8 @@ static void patch_reloc(tcg_insn_unit *code_ptr, int type, if (value != (int32_t)value) { tcg_abort(); } + /* FALLTHRU */ + case R_386_32: tcg_patch32(code_ptr, value); break; case R_386_PC8: @@ -162,6 +193,14 @@ static void patch_reloc(tcg_insn_unit *code_ptr, int type, } } +#if TCG_TARGET_REG_BITS == 64 +#define ALL_GENERAL_REGS 0x0000ffffu +#define ALL_VECTOR_REGS 0xffff0000u +#else +#define ALL_GENERAL_REGS 0x000000ffu +#define ALL_VECTOR_REGS 0x00ff0000u +#endif + /* parse target specific constraints */ static const char *target_parse_constraint(TCGArgConstraint *ct, const char *ct_str, TCGType type) @@ -192,21 +231,29 @@ static const char *target_parse_constraint(TCGArgConstraint *ct, tcg_regset_set_reg(ct->u.regs, TCG_REG_EDI); break; case 'q': + /* A register that can be used as a byte operand. */ ct->ct |= TCG_CT_REG; ct->u.regs = TCG_TARGET_REG_BITS == 64 ? 0xffff : 0xf; break; case 'Q': + /* A register with an addressable second byte (e.g. %ah). */ ct->ct |= TCG_CT_REG; ct->u.regs = 0xf; break; case 'r': + /* A general register. */ ct->ct |= TCG_CT_REG; - ct->u.regs = TCG_TARGET_REG_BITS == 64 ? 0xffff : 0xff; + ct->u.regs |= ALL_GENERAL_REGS; break; case 'W': /* With TZCNT/LZCNT, we can have operand-size as an input. */ ct->ct |= TCG_CT_CONST_WSZ; break; + case 'x': + /* A vector register. */ + ct->ct |= TCG_CT_REG; + ct->u.regs |= ALL_VECTOR_REGS; + break; /* qemu_ld/st address constraint */ case 'L': @@ -277,8 +324,9 @@ static inline int tcg_target_const_match(tcg_target_long val, TCGType type, # define P_REXB_RM 0 # define P_GS 0 #endif -#define P_SIMDF3 0x10000 /* 0xf3 opcode prefix */ -#define P_SIMDF2 0x20000 /* 0xf2 opcode prefix */ +#define P_SIMDF3 0x20000 /* 0xf3 opcode prefix */ +#define P_SIMDF2 0x40000 /* 0xf2 opcode prefix */ +#define P_VEXL 0x80000 /* Set VEX.L = 1 */ #define OPC_ARITH_EvIz (0x81) #define OPC_ARITH_EvIb (0x83) @@ -310,11 +358,38 @@ static inline int tcg_target_const_match(tcg_target_long val, TCGType type, #define OPC_MOVL_Iv (0xb8) #define OPC_MOVBE_GyMy (0xf0 | P_EXT38) #define OPC_MOVBE_MyGy (0xf1 | P_EXT38) +#define OPC_MOVD_VyEy (0x6e | P_EXT | P_DATA16) +#define OPC_MOVD_EyVy (0x7e | P_EXT | P_DATA16) +#define OPC_MOVDDUP (0x12 | P_EXT | P_SIMDF2) +#define OPC_MOVDQA_VxWx (0x6f | P_EXT | P_DATA16) +#define OPC_MOVDQA_WxVx (0x7f | P_EXT | P_DATA16) +#define OPC_MOVDQU_VxWx (0x6f | P_EXT | P_SIMDF3) +#define OPC_MOVDQU_WxVx (0x7f | P_EXT | P_SIMDF3) +#define OPC_MOVQ_VqWq (0x7e | P_EXT | P_SIMDF3) +#define OPC_MOVQ_WqVq (0xd6 | P_EXT | P_DATA16) #define OPC_MOVSBL (0xbe | P_EXT) #define OPC_MOVSWL (0xbf | P_EXT) #define OPC_MOVSLQ (0x63 | P_REXW) #define OPC_MOVZBL (0xb6 | P_EXT) #define OPC_MOVZWL (0xb7 | P_EXT) +#define OPC_PADDB (0xfc | P_EXT | P_DATA16) +#define OPC_PADDW (0xfd | P_EXT | P_DATA16) +#define OPC_PADDD (0xfe | P_EXT | P_DATA16) +#define OPC_PADDQ (0xd4 | P_EXT | P_DATA16) +#define OPC_PAND (0xdb | P_EXT | P_DATA16) +#define OPC_PANDN (0xdf | P_EXT | P_DATA16) +#define OPC_PCMPEQB (0x74 | P_EXT | P_DATA16) +#define OPC_POR (0xeb | P_EXT | P_DATA16) +#define OPC_PSHUFD (0x70 | P_EXT | P_DATA16) +#define OPC_PSUBB (0xf8 | P_EXT | P_DATA16) +#define OPC_PSUBW (0xf9 | P_EXT | P_DATA16) +#define OPC_PSUBD (0xfa | P_EXT | P_DATA16) +#define OPC_PSUBQ (0xfb | P_EXT | P_DATA16) +#define OPC_PUNPCKLBW (0x60 | P_EXT | P_DATA16) +#define OPC_PUNPCKLWD (0x61 | P_EXT | P_DATA16) +#define OPC_PUNPCKLDQ (0x62 | P_EXT | P_DATA16) +#define OPC_PUNPCKLQDQ (0x6c | P_EXT | P_DATA16) +#define OPC_PXOR (0xef | P_EXT | P_DATA16) #define OPC_POP_r32 (0x58) #define OPC_POPCNT (0xb8 | P_EXT | P_SIMDF3) #define OPC_PUSH_r32 (0x50) @@ -330,6 +405,11 @@ static inline int tcg_target_const_match(tcg_target_long val, TCGType type, #define OPC_SHRX (0xf7 | P_EXT38 | P_SIMDF2) #define OPC_TESTL (0x85) #define OPC_TZCNT (0xbc | P_EXT | P_SIMDF3) +#define OPC_VPBROADCASTB (0x78 | P_EXT38 | P_DATA16) +#define OPC_VPBROADCASTW (0x79 | P_EXT38 | P_DATA16) +#define OPC_VPBROADCASTD (0x58 | P_EXT38 | P_DATA16) +#define OPC_VPBROADCASTQ (0x59 | P_EXT38 | P_DATA16) +#define OPC_VZEROUPPER (0x77 | P_EXT) #define OPC_XCHG_ax_r32 (0x90) #define OPC_GRP3_Ev (0xf7) @@ -479,11 +559,20 @@ static void tcg_out_modrm(TCGContext *s, int opc, int r, int rm) tcg_out8(s, 0xc0 | (LOWREGMASK(r) << 3) | LOWREGMASK(rm)); } -static void tcg_out_vex_modrm(TCGContext *s, int opc, int r, int v, int rm) +static void tcg_out_vex_opc(TCGContext *s, int opc, int r, int v, + int rm, int index) { int tmp; - if ((opc & (P_REXW | P_EXT | P_EXT38)) || (rm & 8)) { + /* Use the two byte form if possible, which cannot encode + VEX.W, VEX.B, VEX.X, or an m-mmmm field other than P_EXT. */ + if ((opc & (P_EXT | P_EXT38 | P_REXW)) == P_EXT + && ((rm | index) & 8) == 0) { + /* Two byte VEX prefix. */ + tcg_out8(s, 0xc5); + + tmp = (r & 8 ? 0 : 0x80); /* VEX.R */ + } else { /* Three byte VEX prefix. */ tcg_out8(s, 0xc4); @@ -493,20 +582,17 @@ static void tcg_out_vex_modrm(TCGContext *s, int opc, int r, int v, int rm) } else if (opc & P_EXT) { tmp = 1; } else { - tcg_abort(); + g_assert_not_reached(); } - tmp |= 0x40; /* VEX.X */ - tmp |= (r & 8 ? 0 : 0x80); /* VEX.R */ - tmp |= (rm & 8 ? 0 : 0x20); /* VEX.B */ + tmp |= (r & 8 ? 0 : 0x80); /* VEX.R */ + tmp |= (index & 8 ? 0 : 0x40); /* VEX.X */ + tmp |= (rm & 8 ? 0 : 0x20); /* VEX.B */ tcg_out8(s, tmp); - tmp = (opc & P_REXW ? 0x80 : 0); /* VEX.W */ - } else { - /* Two byte VEX prefix. */ - tcg_out8(s, 0xc5); - - tmp = (r & 8 ? 0 : 0x80); /* VEX.R */ + tmp = (opc & P_REXW ? 0x80 : 0); /* VEX.W */ } + + tmp |= (opc & P_VEXL ? 0x04 : 0); /* VEX.L */ /* VEX.pp */ if (opc & P_DATA16) { tmp |= 1; /* 0x66 */ @@ -518,6 +604,11 @@ static void tcg_out_vex_modrm(TCGContext *s, int opc, int r, int v, int rm) tmp |= (~v & 15) << 3; /* VEX.vvvv */ tcg_out8(s, tmp); tcg_out8(s, opc); +} + +static void tcg_out_vex_modrm(TCGContext *s, int opc, int r, int v, int rm) +{ + tcg_out_vex_opc(s, opc, r, v, rm, 0); tcg_out8(s, 0xc0 | (LOWREGMASK(r) << 3) | LOWREGMASK(rm)); } @@ -526,8 +617,8 @@ static void tcg_out_vex_modrm(TCGContext *s, int opc, int r, int v, int rm) mode for absolute addresses, ~RM is the size of the immediate operand that will follow the instruction. */ -static void tcg_out_modrm_sib_offset(TCGContext *s, int opc, int r, int rm, - int index, int shift, intptr_t offset) +static void tcg_out_sib_offset(TCGContext *s, int r, int rm, int index, + int shift, intptr_t offset) { int mod, len; @@ -538,7 +629,6 @@ static void tcg_out_modrm_sib_offset(TCGContext *s, int opc, int r, int rm, intptr_t pc = (intptr_t)s->code_ptr + 5 + ~rm; intptr_t disp = offset - pc; if (disp == (int32_t)disp) { - tcg_out_opc(s, opc, r, 0, 0); tcg_out8(s, (LOWREGMASK(r) << 3) | 5); tcg_out32(s, disp); return; @@ -548,7 +638,6 @@ static void tcg_out_modrm_sib_offset(TCGContext *s, int opc, int r, int rm, use of the MODRM+SIB encoding and is therefore larger than rip-relative addressing. */ if (offset == (int32_t)offset) { - tcg_out_opc(s, opc, r, 0, 0); tcg_out8(s, (LOWREGMASK(r) << 3) | 4); tcg_out8(s, (4 << 3) | 5); tcg_out32(s, offset); @@ -556,10 +645,9 @@ static void tcg_out_modrm_sib_offset(TCGContext *s, int opc, int r, int rm, } /* ??? The memory isn't directly addressable. */ - tcg_abort(); + g_assert_not_reached(); } else { /* Absolute address. */ - tcg_out_opc(s, opc, r, 0, 0); tcg_out8(s, (r << 3) | 5); tcg_out32(s, offset); return; @@ -582,7 +670,6 @@ static void tcg_out_modrm_sib_offset(TCGContext *s, int opc, int r, int rm, that would be used for %esp is the escape to the two byte form. */ if (index < 0 && LOWREGMASK(rm) != TCG_REG_ESP) { /* Single byte MODRM format. */ - tcg_out_opc(s, opc, r, rm, 0); tcg_out8(s, mod | (LOWREGMASK(r) << 3) | LOWREGMASK(rm)); } else { /* Two byte MODRM+SIB format. */ @@ -596,7 +683,6 @@ static void tcg_out_modrm_sib_offset(TCGContext *s, int opc, int r, int rm, tcg_debug_assert(index != TCG_REG_ESP); } - tcg_out_opc(s, opc, r, rm, index); tcg_out8(s, mod | (LOWREGMASK(r) << 3) | 4); tcg_out8(s, (shift << 6) | (LOWREGMASK(index) << 3) | LOWREGMASK(rm)); } @@ -608,6 +694,21 @@ static void tcg_out_modrm_sib_offset(TCGContext *s, int opc, int r, int rm, } } +static void tcg_out_modrm_sib_offset(TCGContext *s, int opc, int r, int rm, + int index, int shift, intptr_t offset) +{ + tcg_out_opc(s, opc, r, rm < 0 ? 0 : rm, index < 0 ? 0 : index); + tcg_out_sib_offset(s, r, rm, index, shift, offset); +} + +static void tcg_out_vex_modrm_sib_offset(TCGContext *s, int opc, int r, int v, + int rm, int index, int shift, + intptr_t offset) +{ + tcg_out_vex_opc(s, opc, r, v, rm < 0 ? 0 : rm, index < 0 ? 0 : index); + tcg_out_sib_offset(s, r, rm, index, shift, offset); +} + /* A simplification of the above with no index or shift. */ static inline void tcg_out_modrm_offset(TCGContext *s, int opc, int r, int rm, intptr_t offset) @@ -615,6 +716,30 @@ static inline void tcg_out_modrm_offset(TCGContext *s, int opc, int r, tcg_out_modrm_sib_offset(s, opc, r, rm, -1, 0, offset); } +static inline void tcg_out_vex_modrm_offset(TCGContext *s, int opc, int r, + int v, int rm, intptr_t offset) +{ + tcg_out_vex_modrm_sib_offset(s, opc, r, v, rm, -1, 0, offset); +} + +/* Output an opcode with an expected reference to the constant pool. */ +static inline void tcg_out_modrm_pool(TCGContext *s, int opc, int r) +{ + tcg_out_opc(s, opc, r, 0, 0); + /* Absolute for 32-bit, pc-relative for 64-bit. */ + tcg_out8(s, LOWREGMASK(r) << 3 | 5); + tcg_out32(s, 0); +} + +/* Output an opcode with an expected reference to the constant pool. */ +static inline void tcg_out_vex_modrm_pool(TCGContext *s, int opc, int r) +{ + tcg_out_vex_opc(s, opc, r, 0, 0, 0); + /* Absolute for 32-bit, pc-relative for 64-bit. */ + tcg_out8(s, LOWREGMASK(r) << 3 | 5); + tcg_out32(s, 0); +} + /* Generate dest op= src. Uses the same ARITH_* codes as tgen_arithi. */ static inline void tgen_arithr(TCGContext *s, int subop, int dest, int src) { @@ -625,12 +750,114 @@ static inline void tgen_arithr(TCGContext *s, int subop, int dest, int src) tcg_out_modrm(s, OPC_ARITH_GvEv + (subop << 3) + ext, dest, src); } -static inline void tcg_out_mov(TCGContext *s, TCGType type, - TCGReg ret, TCGReg arg) +static void tcg_out_mov(TCGContext *s, TCGType type, TCGReg ret, TCGReg arg) +{ + int rexw = 0; + + if (arg == ret) { + return; + } + switch (type) { + case TCG_TYPE_I64: + rexw = P_REXW; + /* fallthru */ + case TCG_TYPE_I32: + if (ret < 16) { + if (arg < 16) { + tcg_out_modrm(s, OPC_MOVL_GvEv + rexw, ret, arg); + } else { + tcg_out_vex_modrm(s, OPC_MOVD_EyVy + rexw, ret, 0, arg); + } + } else { + if (arg < 16) { + tcg_out_vex_modrm(s, OPC_MOVD_VyEy + rexw, ret, 0, arg); + } else { + tcg_out_vex_modrm(s, OPC_MOVQ_VqWq, ret, 0, arg); + } + } + break; + + case TCG_TYPE_V64: + tcg_debug_assert(ret >= 16 && arg >= 16); + tcg_out_vex_modrm(s, OPC_MOVQ_VqWq, ret, 0, arg); + break; + case TCG_TYPE_V128: + tcg_debug_assert(ret >= 16 && arg >= 16); + tcg_out_vex_modrm(s, OPC_MOVDQA_VxWx, ret, 0, arg); + break; + case TCG_TYPE_V256: + tcg_debug_assert(ret >= 16 && arg >= 16); + tcg_out_vex_modrm(s, OPC_MOVDQA_VxWx | P_VEXL, ret, 0, arg); + break; + + default: + g_assert_not_reached(); + } +} + +static void tcg_out_dup_vec(TCGContext *s, TCGType type, unsigned vece, + TCGReg r, TCGReg a) { - if (arg != ret) { - int opc = OPC_MOVL_GvEv + (type == TCG_TYPE_I64 ? P_REXW : 0); - tcg_out_modrm(s, opc, ret, arg); + if (have_avx2) { + static const int dup_insn[4] = { + OPC_VPBROADCASTB, OPC_VPBROADCASTW, + OPC_VPBROADCASTD, OPC_VPBROADCASTQ, + }; + int vex_l = (type == TCG_TYPE_V256 ? P_VEXL : 0); + tcg_out_vex_modrm(s, dup_insn[vece] + vex_l, r, 0, a); + } else { + switch (vece) { + case MO_8: + /* ??? With zero in a register, use PSHUFB. */ + tcg_out_vex_modrm(s, OPC_PUNPCKLBW, r, 0, a); + a = r; + /* FALLTHRU */ + case MO_16: + tcg_out_vex_modrm(s, OPC_PUNPCKLWD, r, 0, a); + a = r; + /* FALLTHRU */ + case MO_32: + tcg_out_vex_modrm(s, OPC_PSHUFD, r, 0, a); + /* imm8 operand: all output lanes selected from input lane 0. */ + tcg_out8(s, 0); + break; + case MO_64: + tcg_out_vex_modrm(s, OPC_PUNPCKLQDQ, r, 0, a); + break; + default: + g_assert_not_reached(); + } + } +} + +static void tcg_out_dupi_vec(TCGContext *s, TCGType type, + TCGReg ret, tcg_target_long arg) +{ + int vex_l = (type == TCG_TYPE_V256 ? P_VEXL : 0); + + if (arg == 0) { + tcg_out_vex_modrm(s, OPC_PXOR, ret, ret, ret); + return; + } + if (arg == -1) { + tcg_out_vex_modrm(s, OPC_PCMPEQB + vex_l, ret, ret, ret); + return; + } + + if (TCG_TARGET_REG_BITS == 64) { + if (have_avx2) { + tcg_out_vex_modrm_pool(s, OPC_VPBROADCASTQ + vex_l, ret); + } else { + tcg_out_vex_modrm_pool(s, OPC_MOVDDUP, ret); + } + new_pool_label(s, arg, R_386_PC32, s->code_ptr - 4, -4); + } else if (have_avx2) { + tcg_out_vex_modrm_pool(s, OPC_VPBROADCASTD + vex_l, ret); + new_pool_label(s, arg, R_386_32, s->code_ptr - 4, 0); + } else { + tcg_out_vex_modrm_pool(s, OPC_MOVD_VyEy, ret); + new_pool_label(s, arg, R_386_32, s->code_ptr - 4, 0); + tcg_out_dup_vec(s, type, MO_32, ret, ret); } } @@ -639,6 +866,25 @@ static void tcg_out_movi(TCGContext *s, TCGType type, { tcg_target_long diff; + switch (type) { + case TCG_TYPE_I32: +#if TCG_TARGET_REG_BITS == 64 + case TCG_TYPE_I64: +#endif + if (ret < 16) { + break; + } + /* fallthru */ + case TCG_TYPE_V64: + case TCG_TYPE_V128: + case TCG_TYPE_V256: + tcg_debug_assert(ret >= 16); + tcg_out_dupi_vec(s, type, ret, arg); + return; + default: + g_assert_not_reached(); + } + if (arg == 0) { tgen_arithr(s, ARITH_XOR, ret, ret); return; @@ -667,6 +913,59 @@ static void tcg_out_movi(TCGContext *s, TCGType type, tcg_out64(s, arg); } +static void tcg_out_movi_vec(TCGContext *s, TCGType type, + TCGReg ret, const TCGArg *a) +{ + int n = (64 / TCG_TARGET_REG_BITS) << (type - TCG_TYPE_V64); + int opc, ofs, rel; + + tcg_debug_assert(ret >= 16); + tcg_debug_assert(type >= TCG_TYPE_V64); + + /* We assume that INDEX_op_dupi could not be used and therefore + we must use a constant pool entry. */ + + switch (type) { + case TCG_TYPE_V64: + opc = OPC_MOVQ_VqWq; + break; + case TCG_TYPE_V128: + opc = OPC_MOVDQU_VxWx; + break; + case TCG_TYPE_V256: + opc = OPC_MOVDQU_VxWx | P_VEXL; + break; + default: + g_assert_not_reached(); + } + tcg_out_vex_modrm_pool(s, opc, ret); + + if (TCG_TARGET_REG_BITS == 64) { + rel = R_386_PC32, ofs = -4; + } else { + rel = R_386_32, ofs = 0; + } + switch (n) { + case 1: + new_pool_label(s, a[0], rel, s->code_ptr - 4, rel); + break; + case 2: + new_pool_l2(s, rel, s->code_ptr - 4, ofs, a[0], a[1]); + break; + case 4: + new_pool_l4(s, rel, s->code_ptr - 4, ofs, a[0], a[1], a[2], a[3]); + break; +#if TCG_TARGET_REG_BITS == 32 + case 8: + new_pool_l8(s, rel, s->code_ptr - 4, ofs, + a[0], a[1], a[2], a[3], a[4], a[5], a[6], a[7]); + break; +#endif + default: + g_assert_not_reached(); + } +} + static inline void tcg_out_pushi(TCGContext *s, tcg_target_long val) { if (val == (int8_t)val) { @@ -702,18 +1001,74 @@ static inline void tcg_out_pop(TCGContext *s, int reg) tcg_out_opc(s, OPC_POP_r32 + LOWREGMASK(reg), 0, reg, 0); } -static inline void tcg_out_ld(TCGContext *s, TCGType type, TCGReg ret, - TCGReg arg1, intptr_t arg2) +static void tcg_out_ld(TCGContext *s, TCGType type, TCGReg ret, + TCGReg arg1, intptr_t arg2) { - int opc = OPC_MOVL_GvEv + (type == TCG_TYPE_I64 ? P_REXW : 0); - tcg_out_modrm_offset(s, opc, ret, arg1, arg2); + switch (type) { + case TCG_TYPE_I32: + if (ret < 16) { + tcg_out_modrm_offset(s, OPC_MOVL_GvEv, ret, arg1, arg2); + } else { + tcg_out_vex_modrm_offset(s, OPC_MOVD_VyEy, ret, 0, arg1, arg2); + } + break; + case TCG_TYPE_I64: + if (ret < 16) { + tcg_out_modrm_offset(s, OPC_MOVL_GvEv | P_REXW, ret, arg1, arg2); + break; + } + /* FALLTHRU */ + case TCG_TYPE_V64: + tcg_debug_assert(ret >= 16); + tcg_out_vex_modrm_offset(s, OPC_MOVQ_VqWq, ret, 0, arg1, arg2); + break; + case TCG_TYPE_V128: + tcg_debug_assert(ret >= 16); + tcg_out_vex_modrm_offset(s, OPC_MOVDQU_VxWx, ret, 0, arg1, arg2); + break; + case TCG_TYPE_V256: + tcg_debug_assert(ret >= 16); + tcg_out_vex_modrm_offset(s, OPC_MOVDQU_VxWx | P_VEXL, + ret, 0, arg1, arg2); + break; + default: + g_assert_not_reached(); + } } -static inline void tcg_out_st(TCGContext *s, TCGType type, TCGReg arg, - TCGReg arg1, intptr_t arg2) +static void tcg_out_st(TCGContext *s, TCGType type, TCGReg arg, + TCGReg arg1, intptr_t arg2) { - int opc = OPC_MOVL_EvGv + (type == TCG_TYPE_I64 ? P_REXW : 0); - tcg_out_modrm_offset(s, opc, arg, arg1, arg2); + switch (type) { + case TCG_TYPE_I32: + if (arg < 16) { + tcg_out_modrm_offset(s, OPC_MOVL_EvGv, arg, arg1, arg2); + } else { + tcg_out_vex_modrm_offset(s, OPC_MOVD_EyVy, arg, 0, arg1, arg2); + } + break; + case TCG_TYPE_I64: + if (arg < 16) { + tcg_out_modrm_offset(s, OPC_MOVL_EvGv | P_REXW, arg, arg1, arg2); + break; + } + /* FALLTHRU */ + case TCG_TYPE_V64: + tcg_debug_assert(arg >= 16); + tcg_out_vex_modrm_offset(s, OPC_MOVQ_WqVq, arg, 0, arg1, arg2); + break; + case TCG_TYPE_V128: + tcg_debug_assert(arg >= 16); + tcg_out_vex_modrm_offset(s, OPC_MOVDQU_WxVx, arg, 0, arg1, arg2); + break; + case TCG_TYPE_V256: + tcg_debug_assert(arg >= 16); + tcg_out_vex_modrm_offset(s, OPC_MOVDQU_WxVx | P_VEXL, + arg, 0, arg1, arg2); + break; + default: + g_assert_not_reached(); + } } static bool tcg_out_sti(TCGContext *s, TCGType type, TCGArg val, @@ -725,6 +1080,8 @@ static bool tcg_out_sti(TCGContext *s, TCGType type, TCGArg val, return false; } rexw = P_REXW; + } else if (type != TCG_TYPE_I32) { + return false; } tcg_out_modrm_offset(s, OPC_MOVL_EvIz | rexw, 0, base, ofs); tcg_out32(s, val); @@ -2259,8 +2616,10 @@ static inline void tcg_out_op(TCGContext *s, TCGOpcode opc, break; case INDEX_op_mov_i32: /* Always emitted via tcg_out_mov. */ case INDEX_op_mov_i64: + case INDEX_op_mov_vec: case INDEX_op_movi_i32: /* Always emitted via tcg_out_movi. */ case INDEX_op_movi_i64: + case INDEX_op_dupi_vec: case INDEX_op_call: /* Always emitted via tcg_out_call. */ default: tcg_abort(); @@ -2269,6 +2628,73 @@ static inline void tcg_out_op(TCGContext *s, TCGOpcode opc, #undef OP_32_64 } +static void tcg_out_vec_op(TCGContext *s, TCGOpcode opc, + unsigned vecl, unsigned vece, + const TCGArg *args, const int *const_args) +{ + static int const add_insn[4] = { + OPC_PADDB, OPC_PADDW, OPC_PADDD, OPC_PADDQ + }; + static int const sub_insn[4] = { + OPC_PSUBB, OPC_PSUBW, OPC_PSUBD, OPC_PSUBQ + }; + + TCGType type = vecl + TCG_TYPE_V64; + int insn; + TCGArg a0, a1, a2; + + a0 = args[0]; + a1 = args[1]; + a2 = args[2]; + + switch (opc) { + case INDEX_op_add_vec: + insn = add_insn[vece]; + goto gen_simd; + case INDEX_op_sub_vec: + insn = sub_insn[vece]; + goto gen_simd; + case INDEX_op_and_vec: + insn = OPC_PAND; + goto gen_simd; + case INDEX_op_or_vec: + insn = OPC_POR; + goto gen_simd; + case INDEX_op_xor_vec: + insn = OPC_PXOR; + gen_simd: + if (type == TCG_TYPE_V256) { + insn |= P_VEXL; + } + tcg_out_vex_modrm(s, insn, a0, a1, a2); + break; + + case INDEX_op_andc_vec: + insn = OPC_PANDN; + if (type == TCG_TYPE_V256) { + insn |= P_VEXL; + } + tcg_out_vex_modrm(s, insn, a0, a2, a1); + break; + + case INDEX_op_ld_vec: + tcg_out_ld(s, type, a0, a1, a2); + break; + case INDEX_op_st_vec: + tcg_out_st(s, type, a0, a1, a2); + break; + case INDEX_op_movi_vec: + tcg_out_movi_vec(s, type, a0, args + 1); + break; + case INDEX_op_dup_vec: + tcg_out_dup_vec(s, type, vece, a0, a1); + break; + + default: + g_assert_not_reached(); + } +} + static const TCGTargetOpDef *tcg_target_op_def(TCGOpcode op) { static const TCGTargetOpDef r = { .args_ct_str = { "r" } }; @@ -2292,6 +2718,9 @@ static const TCGTargetOpDef *tcg_target_op_def(TCGOpcode op) = { .args_ct_str = { "r", "r", "L", "L" } }; static const TCGTargetOpDef L_L_L_L = { .args_ct_str = { "L", "L", "L", "L" } }; + static const TCGTargetOpDef x_x = { .args_ct_str = { "x", "x" } }; + static const TCGTargetOpDef x_x_x = { .args_ct_str = { "x", "x", "x" } }; + static const TCGTargetOpDef x_r = { .args_ct_str = { "x", "r" } }; switch (op) { case INDEX_op_goto_ptr: @@ -2493,6 +2922,20 @@ static const TCGTargetOpDef *tcg_target_op_def(TCGOpcode op) return &s2; } + case INDEX_op_ld_vec: + case INDEX_op_st_vec: + return &x_r; + + case INDEX_op_add_vec: + case INDEX_op_sub_vec: + case INDEX_op_and_vec: + case INDEX_op_or_vec: + case INDEX_op_xor_vec: + case INDEX_op_andc_vec: + return &x_x_x; + case INDEX_op_dup_vec: + return &x_x; + default: break; } @@ -2577,6 +3020,9 @@ static void tcg_target_qemu_prologue(TCGContext *s) tcg_out_addi(s, TCG_REG_CALL_STACK, stack_addend); + if (have_avx2) { + tcg_out_vex_opc(s, OPC_VZEROUPPER, 0, 0, 0, 0); + } for (i = ARRAY_SIZE(tcg_target_callee_save_regs) - 1; i >= 0; i--) { tcg_out_pop(s, tcg_target_callee_save_regs[i]); } @@ -2598,9 +3044,16 @@ static void tcg_out_nop_fill(tcg_insn_unit *p, int count) static void tcg_target_init(TCGContext *s) { #ifdef CONFIG_CPUID_H - unsigned a, b, c, d; + unsigned a, b, c, d, b7 = 0; int max = __get_cpuid_max(0, 0); + if (max >= 7) { + /* BMI1 is available on AMD Piledriver and Intel Haswell CPUs. */ + __cpuid_count(7, 0, a, b7, c, d); + have_bmi1 = (b7 & bit_BMI) != 0; + have_bmi2 = (b7 & bit_BMI2) != 0; + } + if (max >= 1) { __cpuid(1, a, b, c, d); #ifndef have_cmov @@ -2609,17 +3062,22 @@ static void tcg_target_init(TCGContext *s) available, we'll use a small forward branch. */ have_cmov = (d & bit_CMOV) != 0; #endif + /* MOVBE is only available on Intel Atom and Haswell CPUs, so we need to probe for it. */ have_movbe = (c & bit_MOVBE) != 0; have_popcnt = (c & bit_POPCNT) != 0; - } - if (max >= 7) { - /* BMI1 is available on AMD Piledriver and Intel Haswell CPUs. */ - __cpuid_count(7, 0, a, b, c, d); - have_bmi1 = (b & bit_BMI) != 0; - have_bmi2 = (b & bit_BMI2) != 0; + /* There are a number of things we must check before we can be + sure of not hitting invalid opcode. */ + if (c & bit_OSXSAVE) { + unsigned xcrl, xcrh; + asm ("xgetbv" : "=a" (xcrl), "=d" (xcrh) : "c" (0)); + if ((xcrl & 6) == 6) { + have_avx1 = (c & bit_AVX) != 0; + have_avx2 = (b7 & bit_AVX2) != 0; + } + } } max = __get_cpuid_max(0x8000000, 0); @@ -2630,11 +3088,16 @@ static void tcg_target_init(TCGContext *s) } #endif /* CONFIG_CPUID_H */ + tcg_target_available_regs[TCG_TYPE_I32] = ALL_GENERAL_REGS; if (TCG_TARGET_REG_BITS == 64) { - tcg_target_available_regs[TCG_TYPE_I32] = 0xffff; - tcg_target_available_regs[TCG_TYPE_I64] = 0xffff; - } else { - tcg_target_available_regs[TCG_TYPE_I32] = 0xff; + tcg_target_available_regs[TCG_TYPE_I64] = ALL_GENERAL_REGS; + } + if (have_avx1) { + tcg_target_available_regs[TCG_TYPE_V64] = ALL_VECTOR_REGS; + tcg_target_available_regs[TCG_TYPE_V128] = ALL_VECTOR_REGS; + } + if (have_avx2) { + tcg_target_available_regs[TCG_TYPE_V256] = ALL_VECTOR_REGS; } tcg_target_call_clobber_regs = 0; From patchwork Tue Nov 21 21:25:21 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 119412 Delivered-To: patch@linaro.org Received: by 10.140.22.164 with SMTP id 33csp5817090qgn; Tue, 21 Nov 2017 13:34:25 -0800 (PST) X-Google-Smtp-Source: AGs4zMbMC5M88VGjp3S0AUoxNG10JRdRJRDfwmRU+zpMEn4ax56VhImowXivZGVvSHecXNPutP/0 X-Received: by 10.129.1.134 with SMTP id 128mr14279ywb.34.1511300065202; Tue, 21 Nov 2017 13:34:25 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1511300065; cv=none; d=google.com; s=arc-20160816; b=h/KZEkm5by6ShuPSpO2lAmuecWiMUU9nmVLhtDyYgLJAjJlb65bur7Vkxrih9yfkIH h4zcUPpug7jIuvK+iT2DVzy9FuCrl2ASqJEmrVHjg4rsCxcbSursTNfJbkXtitOTwaDZ ZwaB2EtaP9Js/uV1ELniYx+sNUZjyMxq96fWjdozIdsTG/sVQ46wGCoiWx35hgbCP8iL Svvq9K4MPLH84Q14TTu6ZPFIknpbt2QE6jlVqYNEJWv+Fjj8e7GjiArrqbyjX9TwJjQP 7jNyEFLvB/gwpzZrpn7wWsC6L0utzOnIP+jVMX/LIGy4tIDZjtNf5O9F735/A8Siardl ME4Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:subject:references:in-reply-to :message-id:date:to:from:dkim-signature:arc-authentication-results; bh=RaKEgOy5PTcHDYX4YjVz+3lAvMKFpHo7aLXzwCfdXUo=; b=oGxc5r6smVPPWmIjLS/wYsQLahUwJGDPQFvLHPT4YcLCb+7RSyd1Al2T/klzqkSXdh ho9+IF0Uw0WM1rHh21MdJZfpRe5EtBxWWBxM+A3K7nyzmunAxDVchU08hhDRSg4aXC7E OOzHQzQPhEgoNw2eO1IU6m4bd88qyIFx+ZoIS+1MyEU+crnSuo9ZAeXJZeTn1ZQ7liUB UxE1kAKvAMoEBHPzegqCPcR0fVqIDp6ed3I8MjPry/uwcxkEG0rUKm31JiJdua2fqi6J UPfb63Y9FdsQTdaWSleefXGM+6vrX7HRNZfkwwpTcRx6y1/l5xKv89lummVyd5GhpsTY sXww== ARC-Authentication-Results: i=1; mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=am6hClXd; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) smtp.mailfrom=qemu-devel-bounces+patch=linaro.org@nongnu.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [2001:4830:134:3::11]) by mx.google.com with ESMTPS id y77si3058432ybe.590.2017.11.21.13.34.25 for (version=TLS1 cipher=AES128-SHA bits=128/128); Tue, 21 Nov 2017 13:34:25 -0800 (PST) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) client-ip=2001:4830:134:3::11; Authentication-Results: mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=am6hClXd; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) smtp.mailfrom=qemu-devel-bounces+patch=linaro.org@nongnu.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1]:36600 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1eHGBU-0000Ku-LY for patch@linaro.org; Tue, 21 Nov 2017 16:34:24 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:54079) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1eHG5D-0004R6-5q for qemu-devel@nongnu.org; Tue, 21 Nov 2017 16:27:56 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1eHG5C-0007k0-5T for qemu-devel@nongnu.org; Tue, 21 Nov 2017 16:27:55 -0500 Received: from mail-wm0-x242.google.com ([2a00:1450:400c:c09::242]:40248) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1eHG5B-0007jl-VO for qemu-devel@nongnu.org; Tue, 21 Nov 2017 16:27:54 -0500 Received: by mail-wm0-x242.google.com with SMTP id b189so6266478wmd.5 for ; Tue, 21 Nov 2017 13:27:53 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:subject:date:message-id:in-reply-to:references; bh=RaKEgOy5PTcHDYX4YjVz+3lAvMKFpHo7aLXzwCfdXUo=; b=am6hClXdkbyADPYH2LixWtS6gth8+eDzkXaTuKOboWHlZLoDth/Cv40m0iqemaOnuu e8HyRenUddqnn0kSgTJitK0qCIAO+QH/vVcAZllwudk1n1qO7XzZBn4+FKDVI+yOrYjz 7Nrg4gOFYivxl3oafEx2DuW3qY7YZAaHomFTQ= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references; bh=RaKEgOy5PTcHDYX4YjVz+3lAvMKFpHo7aLXzwCfdXUo=; b=DV9+dGrqKp576kax+E8mf5dRf5Oxm/hr1GQ4mkXOk6TmxLxutYVSO2uBeNTXC6psgB oSVT1pXImIN+Sst/vIaLjBr6WlOM76vlt8Uow3G2UlMy791nepUUmiPaJfAESILMYdrB UIy+MVQez8XmWEZ6egkvK6TtGhIFqeTW8gkop8bxUHAPgdpt4IsKWGtIqcxgQPQ9A1bC 2T1j1TBooAiJowi1DtOaSPajyTdOr37bqNWGa0GvHlwg1ttDbdD5ffEUElHtBUrIMhfO jm1x1ZqMuh3YWqyCqQWUfV0ltrUYKyehA6UD+BUh0KUBttDBMo/Hh5ZTWJGpMHNK5qmP 3QIQ== X-Gm-Message-State: AJaThX5mdWW7y1zIu9W+3E237e8p5XRTRDb1dOfrDgf981yqRM/1FcmD HJO9TxQijg7xJY9yJyyIszr/xQeCQ1s= X-Received: by 10.28.225.197 with SMTP id y188mr2568861wmg.12.1511299672775; Tue, 21 Nov 2017 13:27:52 -0800 (PST) Received: from cloudburst.twiddle.net (70.red-37-158-60.dynamicip.rima-tde.net. [37.158.60.70]) by smtp.gmail.com with ESMTPSA id e124sm706517wmg.34.2017.11.21.13.27.51 for (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Tue, 21 Nov 2017 13:27:51 -0800 (PST) From: Richard Henderson To: qemu-devel@nongnu.org Date: Tue, 21 Nov 2017 22:25:21 +0100 Message-Id: <20171121212534.5177-14-richard.henderson@linaro.org> X-Mailer: git-send-email 2.13.6 In-Reply-To: <20171121212534.5177-1-richard.henderson@linaro.org> References: <20171121212534.5177-1-richard.henderson@linaro.org> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2a00:1450:400c:c09::242 Subject: [Qemu-devel] [PATCH v6 13/26] tcg: Add tcg_expand_vec_op and tcg-target.opc.h X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: "Qemu-devel" These will be useful in the next few patches adding shifts, permutes, and multiplication. Signed-off-by: Richard Henderson --- tcg/i386/tcg-target.opc.h | 3 +++ tcg/tcg-opc.h | 6 ++++++ tcg/tcg.h | 11 +++++++++++ tcg/i386/tcg-target.inc.c | 21 +++++++++++++++++++++ tcg/tcg.c | 6 +++--- 5 files changed, 44 insertions(+), 3 deletions(-) create mode 100644 tcg/i386/tcg-target.opc.h -- 2.13.6 diff --git a/tcg/i386/tcg-target.opc.h b/tcg/i386/tcg-target.opc.h new file mode 100644 index 0000000000..4816a6c3d4 --- /dev/null +++ b/tcg/i386/tcg-target.opc.h @@ -0,0 +1,3 @@ +/* Target-specific opcodes for host vector expansion. These will be + emitted by tcg_expand_vec_op. For those familiar with GCC internals, + consider these to be UNSPEC with names. */ diff --git a/tcg/tcg-opc.h b/tcg/tcg-opc.h index 4e62eda14b..b4e16cfbc3 100644 --- a/tcg/tcg-opc.h +++ b/tcg/tcg-opc.h @@ -229,6 +229,12 @@ DEF(andc_vec, 1, 2, 0, IMPLVEC | IMPL(TCG_TARGET_HAS_andc_vec)) DEF(orc_vec, 1, 2, 0, IMPLVEC | IMPL(TCG_TARGET_HAS_orc_vec)) DEF(not_vec, 1, 1, 0, IMPLVEC | IMPL(TCG_TARGET_HAS_not_vec)) +DEF(last_generic, 0, 0, 0, TCG_OPF_NOT_PRESENT) + +#if TCG_TARGET_MAYBE_vec +#include "tcg-target.opc.h" +#endif + #undef TLADDR_ARGS #undef DATA64_ARGS #undef IMPL diff --git a/tcg/tcg.h b/tcg/tcg.h index 49d4c5fe05..2cba208b4c 100644 --- a/tcg/tcg.h +++ b/tcg/tcg.h @@ -1207,6 +1207,17 @@ uintptr_t tcg_qemu_tb_exec(CPUArchState *env, uint8_t *tb_ptr); void tcg_register_jit(void *buf, size_t buf_size); +#if TCG_TARGET_MAYBE_vec +/* Return zero if the tuple (opc, type, vece) is unsupportable; + return > 0 if it is directly supportable; + return < 0 if we must call tcg_expand_vec_op. */ +int tcg_can_emit_vec_op(TCGOpcode, TCGType, unsigned); + +/* Expand the tuple (opc, type, vece) on the given arguments. */ +void tcg_expand_vec_op(TCGOpcode, TCGType, unsigned, TCGArg, ...); +#endif + + /* * Memory helpers that will be used by TCG generated code. */ diff --git a/tcg/i386/tcg-target.inc.c b/tcg/i386/tcg-target.inc.c index e9a4d92598..062cf16607 100644 --- a/tcg/i386/tcg-target.inc.c +++ b/tcg/i386/tcg-target.inc.c @@ -2942,6 +2942,27 @@ static const TCGTargetOpDef *tcg_target_op_def(TCGOpcode op) return NULL; } +int tcg_can_emit_vec_op(TCGOpcode opc, TCGType type, unsigned vece) +{ + switch (opc) { + case INDEX_op_add_vec: + case INDEX_op_sub_vec: + case INDEX_op_and_vec: + case INDEX_op_or_vec: + case INDEX_op_xor_vec: + case INDEX_op_andc_vec: + return true; + + default: + return false; + } +} + +void tcg_expand_vec_op(TCGOpcode opc, TCGType type, unsigned vece, + TCGArg a0, ...) +{ +} + static const int tcg_target_callee_save_regs[] = { #if TCG_TARGET_REG_BITS == 64 TCG_REG_RBP, diff --git a/tcg/tcg.c b/tcg/tcg.c index 16b8faf66f..e725b1818f 100644 --- a/tcg/tcg.c +++ b/tcg/tcg.c @@ -1404,10 +1404,10 @@ bool tcg_op_supported(TCGOpcode op) case INDEX_op_orc_vec: return have_vec && TCG_TARGET_HAS_orc_vec; - case NB_OPS: - break; + default: + tcg_debug_assert(op > INDEX_op_last_generic && op < NB_OPS); + return true; } - g_assert_not_reached(); } /* Note: we convert the 64 bit args to 32 bit and do some alignment From patchwork Tue Nov 21 21:25:22 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 119420 Delivered-To: patch@linaro.org Received: by 10.140.22.164 with SMTP id 33csp5822790qgn; Tue, 21 Nov 2017 13:40:35 -0800 (PST) X-Google-Smtp-Source: AGs4zMZ1ngGSrWvwUzXCz0BCH0t9kHpmVQGIshBGOPMjhxw6cn9LyPwdJBnkiShe53Odzvd7a6mf X-Received: by 10.37.76.5 with SMTP id z5mr11850567yba.257.1511300435520; Tue, 21 Nov 2017 13:40:35 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1511300435; cv=none; d=google.com; s=arc-20160816; b=b0Pl9C2HgV4MFoNadXTRjM4rZjDTNb4mNeMQ5232MsXqjMiRdV0Z0QnptE3SKSXhnq yrjG/DUVRL9nYM7g2FH2X2MxhaGKQPVrXd0wQZ2VHx3aVl4jUsMd2ulR7KyDx8aNhCsO 1geR483bs4F9LEYMN56Owr9lVF6j7z8ms/CRNAeLzJe7nztdjmB0TuszNr1LkrUzlYhQ LlXhkB732VPh/zjNlDyG9HLJLNKH3Bmtu1mNPbdnbyHY2GSWE1wKJpIWccnLpJhUX7Ea 0efMF82M2ur1/vg9IM0A9Q/xqXWsXbemTTV0CN8pR2ng/XmZ6SS/7ikoz3ELbeStGZLO Ma+Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:subject:references:in-reply-to :message-id:date:to:from:dkim-signature:arc-authentication-results; bh=1Qw/kpjFM4PWAW2XEiApN9T8ak9b8Ymmu6dhpNI89UE=; b=CNqFVIsLGuZZWzf63oaIx+ksv/Iov1mP8hhgtVRPkn7DyYAeq0CQQRLE+gRVQknSmO qCCH+ZFmWK42EWl7wkGCovFCcvf+hGBVruaEHDPRkDYVnnxB+uj7rMinIpm+GbgjDK4x hlzwxQ/1Y/NjI9rKZPjZHH5VscJswhW3HInDlqAhLQB+zpsWaK1muaA3cdQUV+sqDEwQ PV+q3ozrlJJkarBfFe34PtmUBJv+E+WbhsUjUj4/KHf1Cx0bRDEjNzGpWdJJznwVhWn7 o2gDrXN5m9GoAd+bV9r50ySEe9K0DOiRq6EnUEA2q6MuZHkD7Dzx2KkS05RdIfToLU0T 1GaQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=HalAK0g3; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) smtp.mailfrom=qemu-devel-bounces+patch=linaro.org@nongnu.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [2001:4830:134:3::11]) by mx.google.com with ESMTPS id z7si2574981yba.761.2017.11.21.13.40.35 for (version=TLS1 cipher=AES128-SHA bits=128/128); Tue, 21 Nov 2017 13:40:35 -0800 (PST) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) client-ip=2001:4830:134:3::11; Authentication-Results: mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=HalAK0g3; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) smtp.mailfrom=qemu-devel-bounces+patch=linaro.org@nongnu.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1]:36641 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1eHGHT-0006zw-1F for patch@linaro.org; Tue, 21 Nov 2017 16:40:35 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:54116) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1eHG5H-0004Vc-Hc for qemu-devel@nongnu.org; Tue, 21 Nov 2017 16:28:02 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1eHG5E-0007l0-AJ for qemu-devel@nongnu.org; Tue, 21 Nov 2017 16:27:59 -0500 Received: from mail-wm0-x242.google.com ([2a00:1450:400c:c09::242]:43987) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1eHG5D-0007kU-Sq for qemu-devel@nongnu.org; Tue, 21 Nov 2017 16:27:56 -0500 Received: by mail-wm0-x242.google.com with SMTP id x63so6294675wmf.2 for ; Tue, 21 Nov 2017 13:27:55 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:subject:date:message-id:in-reply-to:references; bh=1Qw/kpjFM4PWAW2XEiApN9T8ak9b8Ymmu6dhpNI89UE=; b=HalAK0g3zx3eCnvTcBw7yWwyFLfHfRQ6ab8GdfuiSQbG69CeFJCXNUIN+VzMQrnCzg ScFVWoEVcQZsLflCnsL0XdcDFzkogMaacYLSAhGMg9iSdwwbVIH6ug72pEDJbsIG7Y4b vjzDBAH1Mi5FtiTETpwyOdSM+PlxSS0fdHDKo= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references; bh=1Qw/kpjFM4PWAW2XEiApN9T8ak9b8Ymmu6dhpNI89UE=; b=ie7VwS/mO796G718PrMb3+zrJfgGbV6kGxLU/Q10gbJNYh0IL1z3HA2jRk5rp7Szfa C6kKN8DZMOdqUZ4UBBIhCVRmJdZU/bqTZuzkIN3HNv03///yxifkXMHCg2wEwH5cHli+ 0cF8GcupM0fHdsR3W9XaW8HIiDKFZyPP7AXmj2jzz9otk6B8t0hKvYnWmqtTejji17Yh K8i55nmtudNvVU1lDfwiFd6tcjHv3ysSKbq1MMOMYggbAbbd0GQIFbSZgTsAv7AoGBFe CYhny7jIsoGEtvTSnn4KQ0d2/snxfH7GqQHlhzvS3uJ0oug9ArEYxsK5VcfYYIuW9qxc 9LNQ== X-Gm-Message-State: AJaThX5rdHM7gGD41aUjnpItH2S1vmIN1KATWXrcw2zW0mtxk9sIr4cm XFd6D9I0l3hEWEG+htRNjIR3H0n8320= X-Received: by 10.28.236.71 with SMTP id k68mr2289113wmh.120.1511299674250; Tue, 21 Nov 2017 13:27:54 -0800 (PST) Received: from cloudburst.twiddle.net (70.red-37-158-60.dynamicip.rima-tde.net. [37.158.60.70]) by smtp.gmail.com with ESMTPSA id e124sm706517wmg.34.2017.11.21.13.27.52 for (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Tue, 21 Nov 2017 13:27:53 -0800 (PST) From: Richard Henderson To: qemu-devel@nongnu.org Date: Tue, 21 Nov 2017 22:25:22 +0100 Message-Id: <20171121212534.5177-15-richard.henderson@linaro.org> X-Mailer: git-send-email 2.13.6 In-Reply-To: <20171121212534.5177-1-richard.henderson@linaro.org> References: <20171121212534.5177-1-richard.henderson@linaro.org> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2a00:1450:400c:c09::242 Subject: [Qemu-devel] [PATCH v6 14/26] tcg: Add generic vector ops for interleave X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: "Qemu-devel" Includes zip, unzip, and transform. Signed-off-by: Richard Henderson --- accel/tcg/tcg-runtime.h | 15 ++ tcg/i386/tcg-target.h | 3 + tcg/tcg-op-gvec.h | 17 +++ tcg/tcg-op.h | 6 + tcg/tcg-opc.h | 7 + tcg/tcg.h | 3 + accel/tcg/tcg-runtime-gvec.c | 78 ++++++++++ tcg/tcg-op-gvec.c | 337 ++++++++++++++++++++++++++++++++++++++++++- tcg/tcg-op-vec.c | 55 +++++++ tcg/tcg.c | 9 ++ tcg/README | 40 +++++ 11 files changed, 562 insertions(+), 8 deletions(-) -- 2.13.6 diff --git a/accel/tcg/tcg-runtime.h b/accel/tcg/tcg-runtime.h index 76ee41ce58..c6de749134 100644 --- a/accel/tcg/tcg-runtime.h +++ b/accel/tcg/tcg-runtime.h @@ -163,3 +163,18 @@ DEF_HELPER_FLAGS_4(gvec_or, TCG_CALL_NO_RWG, void, ptr, ptr, ptr, i32) DEF_HELPER_FLAGS_4(gvec_xor, TCG_CALL_NO_RWG, void, ptr, ptr, ptr, i32) DEF_HELPER_FLAGS_4(gvec_andc, TCG_CALL_NO_RWG, void, ptr, ptr, ptr, i32) DEF_HELPER_FLAGS_4(gvec_orc, TCG_CALL_NO_RWG, void, ptr, ptr, ptr, i32) + +DEF_HELPER_FLAGS_4(gvec_zip8, TCG_CALL_NO_RWG, void, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_4(gvec_zip16, TCG_CALL_NO_RWG, void, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_4(gvec_zip32, TCG_CALL_NO_RWG, void, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_4(gvec_zip64, TCG_CALL_NO_RWG, void, ptr, ptr, ptr, i32) + +DEF_HELPER_FLAGS_4(gvec_uzp8, TCG_CALL_NO_RWG, void, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_4(gvec_uzp16, TCG_CALL_NO_RWG, void, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_4(gvec_uzp32, TCG_CALL_NO_RWG, void, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_4(gvec_uzp64, TCG_CALL_NO_RWG, void, ptr, ptr, ptr, i32) + +DEF_HELPER_FLAGS_4(gvec_trn8, TCG_CALL_NO_RWG, void, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_4(gvec_trn16, TCG_CALL_NO_RWG, void, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_4(gvec_trn32, TCG_CALL_NO_RWG, void, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_4(gvec_trn64, TCG_CALL_NO_RWG, void, ptr, ptr, ptr, i32) diff --git a/tcg/i386/tcg-target.h b/tcg/i386/tcg-target.h index f9d3fc4a93..ff0ad7dcdb 100644 --- a/tcg/i386/tcg-target.h +++ b/tcg/i386/tcg-target.h @@ -177,6 +177,9 @@ extern bool have_avx2; #define TCG_TARGET_HAS_orc_vec 0 #define TCG_TARGET_HAS_not_vec 0 #define TCG_TARGET_HAS_neg_vec 0 +#define TCG_TARGET_HAS_zip_vec 0 +#define TCG_TARGET_HAS_uzp_vec 0 +#define TCG_TARGET_HAS_trn_vec 0 #define TCG_TARGET_deposit_i32_valid(ofs, len) \ (((ofs) == 0 && (len) == 8) || ((ofs) == 8 && (len) == 8) || \ diff --git a/tcg/tcg-op-gvec.h b/tcg/tcg-op-gvec.h index 95739946ff..64270a3c74 100644 --- a/tcg/tcg-op-gvec.h +++ b/tcg/tcg-op-gvec.h @@ -66,6 +66,8 @@ typedef struct { gen_helper_gvec_2 *fno; /* The opcode, if any, to which this corresponds. */ TCGOpcode opc; + /* The data argument to the out-of-line helper. */ + uint32_t data; /* The vector element size, if applicable. */ uint8_t vece; /* Prefer i64 to v64. */ @@ -83,6 +85,8 @@ typedef struct { gen_helper_gvec_3 *fno; /* The opcode, if any, to which this corresponds. */ TCGOpcode opc; + /* The data argument to the out-of-line helper. */ + uint32_t data; /* The vector element size, if applicable. */ uint8_t vece; /* Prefer i64 to v64. */ @@ -133,6 +137,19 @@ void tcg_gen_gvec_dup16i(uint32_t dofs, uint32_t s, uint32_t m, uint16_t x); void tcg_gen_gvec_dup32i(uint32_t dofs, uint32_t s, uint32_t m, uint32_t x); void tcg_gen_gvec_dup64i(uint32_t dofs, uint32_t s, uint32_t m, uint64_t x); +void tcg_gen_gvec_zipl(unsigned vece, uint32_t dofs, uint32_t aofs, + uint32_t bofs, uint32_t opsz, uint32_t clsz); +void tcg_gen_gvec_ziph(unsigned vece, uint32_t dofs, uint32_t aofs, + uint32_t bofs, uint32_t opsz, uint32_t clsz); +void tcg_gen_gvec_uzpe(unsigned vece, uint32_t dofs, uint32_t aofs, + uint32_t bofs, uint32_t opsz, uint32_t clsz); +void tcg_gen_gvec_uzpo(unsigned vece, uint32_t dofs, uint32_t aofs, + uint32_t bofs, uint32_t opsz, uint32_t clsz); +void tcg_gen_gvec_trne(unsigned vece, uint32_t dofs, uint32_t aofs, + uint32_t bofs, uint32_t opsz, uint32_t clsz); +void tcg_gen_gvec_trno(unsigned vece, uint32_t dofs, uint32_t aofs, + uint32_t bofs, uint32_t opsz, uint32_t clsz); + /* * 64-bit vector operations. Use these when the register has been allocated * with tcg_global_mem_new_i64, and so we cannot also address it via pointer. diff --git a/tcg/tcg-op.h b/tcg/tcg-op.h index 5f49785cb3..733e29b5f8 100644 --- a/tcg/tcg-op.h +++ b/tcg/tcg-op.h @@ -927,6 +927,12 @@ void tcg_gen_andc_vec(unsigned vece, TCGv_vec r, TCGv_vec a, TCGv_vec b); void tcg_gen_orc_vec(unsigned vece, TCGv_vec r, TCGv_vec a, TCGv_vec b); void tcg_gen_not_vec(unsigned vece, TCGv_vec r, TCGv_vec a); void tcg_gen_neg_vec(unsigned vece, TCGv_vec r, TCGv_vec a); +void tcg_gen_zipl_vec(unsigned vece, TCGv_vec r, TCGv_vec a, TCGv_vec b); +void tcg_gen_ziph_vec(unsigned vece, TCGv_vec r, TCGv_vec a, TCGv_vec b); +void tcg_gen_uzpe_vec(unsigned vece, TCGv_vec r, TCGv_vec a, TCGv_vec b); +void tcg_gen_uzpo_vec(unsigned vece, TCGv_vec r, TCGv_vec a, TCGv_vec b); +void tcg_gen_trne_vec(unsigned vece, TCGv_vec r, TCGv_vec a, TCGv_vec b); +void tcg_gen_trno_vec(unsigned vece, TCGv_vec r, TCGv_vec a, TCGv_vec b); void tcg_gen_ld_vec(TCGv_vec r, TCGv_ptr base, TCGArg offset); void tcg_gen_st_vec(TCGv_vec r, TCGv_ptr base, TCGArg offset); diff --git a/tcg/tcg-opc.h b/tcg/tcg-opc.h index b4e16cfbc3..c911d62442 100644 --- a/tcg/tcg-opc.h +++ b/tcg/tcg-opc.h @@ -229,6 +229,13 @@ DEF(andc_vec, 1, 2, 0, IMPLVEC | IMPL(TCG_TARGET_HAS_andc_vec)) DEF(orc_vec, 1, 2, 0, IMPLVEC | IMPL(TCG_TARGET_HAS_orc_vec)) DEF(not_vec, 1, 1, 0, IMPLVEC | IMPL(TCG_TARGET_HAS_not_vec)) +DEF(zipl_vec, 1, 2, 0, IMPLVEC | IMPL(TCG_TARGET_HAS_zip_vec)) +DEF(ziph_vec, 1, 2, 0, IMPLVEC | IMPL(TCG_TARGET_HAS_zip_vec)) +DEF(uzpe_vec, 1, 2, 0, IMPLVEC | IMPL(TCG_TARGET_HAS_uzp_vec)) +DEF(uzpo_vec, 1, 2, 0, IMPLVEC | IMPL(TCG_TARGET_HAS_uzp_vec)) +DEF(trne_vec, 1, 2, 0, IMPLVEC | IMPL(TCG_TARGET_HAS_trn_vec)) +DEF(trno_vec, 1, 2, 0, IMPLVEC | IMPL(TCG_TARGET_HAS_trn_vec)) + DEF(last_generic, 0, 0, 0, TCG_OPF_NOT_PRESENT) #if TCG_TARGET_MAYBE_vec diff --git a/tcg/tcg.h b/tcg/tcg.h index 2cba208b4c..c6f7157c60 100644 --- a/tcg/tcg.h +++ b/tcg/tcg.h @@ -178,6 +178,9 @@ typedef uint64_t TCGRegSet; #define TCG_TARGET_HAS_not_vec 0 #define TCG_TARGET_HAS_andc_vec 0 #define TCG_TARGET_HAS_orc_vec 0 +#define TCG_TARGET_HAS_zip_vec 0 +#define TCG_TARGET_HAS_uzp_vec 0 +#define TCG_TARGET_HAS_trn_vec 0 #else #define TCG_TARGET_MAYBE_vec 1 #endif diff --git a/accel/tcg/tcg-runtime-gvec.c b/accel/tcg/tcg-runtime-gvec.c index cd1ce12b7e..628df811b2 100644 --- a/accel/tcg/tcg-runtime-gvec.c +++ b/accel/tcg/tcg-runtime-gvec.c @@ -293,3 +293,81 @@ void HELPER(gvec_orc)(void *d, void *a, void *b, uint32_t desc) } clear_high(d, oprsz, desc); } + +/* The size of the alloca in the following is currently bounded to 2k. */ + +#define DO_ZIP(NAME, TYPE) \ +void HELPER(NAME)(void *d, void *a, void *b, uint32_t desc) \ +{ \ + intptr_t oprsz = simd_oprsz(desc); \ + intptr_t oprsz_2 = oprsz / 2; \ + intptr_t i; \ + /* We produce output faster than we consume input. \ + Therefore we must be mindful of possible overlap. */ \ + if (unlikely((a - d) < (uintptr_t)oprsz)) { \ + void *a_new = alloca(oprsz_2); \ + memcpy(a_new, a, oprsz_2); \ + a = a_new; \ + } \ + if (unlikely((b - d) < (uintptr_t)oprsz)) { \ + void *b_new = alloca(oprsz_2); \ + memcpy(b_new, b, oprsz_2); \ + b = b_new; \ + } \ + for (i = 0; i < oprsz_2; i += sizeof(TYPE)) { \ + *(TYPE *)(d + 2 * i + 0) = *(TYPE *)(a + i); \ + *(TYPE *)(d + 2 * i + sizeof(TYPE)) = *(TYPE *)(b + i); \ + } \ + clear_high(d, oprsz, desc); \ +} + +DO_ZIP(gvec_zip8, uint8_t) +DO_ZIP(gvec_zip16, uint16_t) +DO_ZIP(gvec_zip32, uint32_t) +DO_ZIP(gvec_zip64, uint64_t) + +#define DO_UZP(NAME, TYPE) \ +void HELPER(NAME)(void *d, void *a, void *b, uint32_t desc) \ +{ \ + intptr_t oprsz = simd_oprsz(desc); \ + intptr_t oprsz_2 = oprsz / 2; \ + intptr_t odd_ofs = simd_data(desc); \ + intptr_t i; \ + if (unlikely((b - d) < (uintptr_t)oprsz)) { \ + void *b_new = alloca(oprsz); \ + memcpy(b_new, b, oprsz); \ + b = b_new; \ + } \ + for (i = 0; i < oprsz_2; i += sizeof(TYPE)) { \ + *(TYPE *)(d + i) = *(TYPE *)(a + 2 * i + odd_ofs); \ + } \ + for (i = 0; i < oprsz_2; i += sizeof(TYPE)) { \ + *(TYPE *)(d + oprsz_2 + i) = *(TYPE *)(b + 2 * i + odd_ofs); \ + } \ + clear_high(d, oprsz, desc); \ +} + +DO_UZP(gvec_uzp8, uint8_t) +DO_UZP(gvec_uzp16, uint16_t) +DO_UZP(gvec_uzp32, uint32_t) +DO_UZP(gvec_uzp64, uint64_t) + +#define DO_TRN(NAME, TYPE) \ +void HELPER(NAME)(void *d, void *a, void *b, uint32_t desc) \ +{ \ + intptr_t oprsz = simd_oprsz(desc); \ + intptr_t odd_ofs = simd_data(desc); \ + intptr_t i; \ + for (i = 0; i < oprsz; i += 2 * sizeof(TYPE)) { \ + TYPE ae = *(TYPE *)(a + i + odd_ofs); \ + TYPE be = *(TYPE *)(b + i + odd_ofs); \ + *(TYPE *)(d + i + 0) = ae; \ + *(TYPE *)(d + i + sizeof(TYPE)) = be; \ + } \ + clear_high(d, oprsz, desc); \ +} + +DO_TRN(gvec_trn8, uint8_t) +DO_TRN(gvec_trn16, uint16_t) +DO_TRN(gvec_trn32, uint32_t) +DO_TRN(gvec_trn64, uint64_t) diff --git a/tcg/tcg-op-gvec.c b/tcg/tcg-op-gvec.c index 925c293f9c..a64baa9dcf 100644 --- a/tcg/tcg-op-gvec.c +++ b/tcg/tcg-op-gvec.c @@ -466,7 +466,8 @@ void tcg_gen_gvec_2(uint32_t dofs, uint32_t aofs, use a cl-sized store to implement the clearing without an extra store operation. This is true for aarch64 and x86_64 hosts. */ - if (TCG_TARGET_HAS_v256 && check_size_impl(oprsz, 32)) { + if (TCG_TARGET_HAS_v256 && check_size_impl(oprsz, 32) + && (!g->opc || tcg_can_emit_vec_op(g->opc, TCG_TYPE_V256, g->vece))) { uint32_t done = QEMU_ALIGN_DOWN(oprsz, 32); expand_2_vec(g->vece, dofs, aofs, done, 32, TCG_TYPE_V256, g->fniv); dofs += done; @@ -475,7 +476,8 @@ void tcg_gen_gvec_2(uint32_t dofs, uint32_t aofs, maxsz -= done; } - if (TCG_TARGET_HAS_v128 && check_size_impl(oprsz, 16)) { + if (TCG_TARGET_HAS_v128 && check_size_impl(oprsz, 16) + && (!g->opc || tcg_can_emit_vec_op(g->opc, TCG_TYPE_V128, g->vece))) { uint32_t done = QEMU_ALIGN_DOWN(oprsz, 16); expand_2_vec(g->vece, dofs, aofs, done, 16, TCG_TYPE_V128, g->fniv); dofs += done; @@ -486,7 +488,9 @@ void tcg_gen_gvec_2(uint32_t dofs, uint32_t aofs, if (check_size_impl(oprsz, 8)) { uint32_t done = QEMU_ALIGN_DOWN(oprsz, 8); - if (TCG_TARGET_HAS_v64 && !g->prefer_i64) { + if (TCG_TARGET_HAS_v64 && !g->prefer_i64 + && (!g->opc + || tcg_can_emit_vec_op(g->opc, TCG_TYPE_V64, g->vece))) { expand_2_vec(g->vece, dofs, aofs, done, 8, TCG_TYPE_V64, g->fniv); } else if (g->fni8) { expand_2_i64(dofs, aofs, done, g->fni8); @@ -516,7 +520,7 @@ void tcg_gen_gvec_2(uint32_t dofs, uint32_t aofs, } do_ool: - tcg_gen_gvec_2_ool(dofs, aofs, oprsz, maxsz, 0, g->fno); + tcg_gen_gvec_2_ool(dofs, aofs, oprsz, maxsz, g->data, g->fno); } /* Expand a vector three-operand operation. */ @@ -539,7 +543,8 @@ void tcg_gen_gvec_3(uint32_t dofs, uint32_t aofs, uint32_t bofs, use a cl-sized store to implement the clearing without an extra store operation. This is true for aarch64 and x86_64 hosts. */ - if (TCG_TARGET_HAS_v256 && check_size_impl(oprsz, 32)) { + if (TCG_TARGET_HAS_v256 && check_size_impl(oprsz, 32) + && (!g->opc || tcg_can_emit_vec_op(g->opc, TCG_TYPE_V256, g->vece))) { uint32_t done = QEMU_ALIGN_DOWN(oprsz, 32); expand_3_vec(g->vece, dofs, aofs, bofs, done, 32, TCG_TYPE_V256, g->load_dest, g->fniv); @@ -550,7 +555,8 @@ void tcg_gen_gvec_3(uint32_t dofs, uint32_t aofs, uint32_t bofs, maxsz -= done; } - if (TCG_TARGET_HAS_v128 && check_size_impl(oprsz, 16)) { + if (TCG_TARGET_HAS_v128 && check_size_impl(oprsz, 16) + && (!g->opc || tcg_can_emit_vec_op(g->opc, TCG_TYPE_V128, g->vece))) { uint32_t done = QEMU_ALIGN_DOWN(oprsz, 16); expand_3_vec(g->vece, dofs, aofs, bofs, done, 16, TCG_TYPE_V128, g->load_dest, g->fniv); @@ -563,7 +569,9 @@ void tcg_gen_gvec_3(uint32_t dofs, uint32_t aofs, uint32_t bofs, if (check_size_impl(oprsz, 8)) { uint32_t done = QEMU_ALIGN_DOWN(oprsz, 8); - if (TCG_TARGET_HAS_v64 && !g->prefer_i64) { + if (TCG_TARGET_HAS_v64 && !g->prefer_i64 + && (!g->opc + || tcg_can_emit_vec_op(g->opc, TCG_TYPE_V64, g->vece))) { expand_3_vec(g->vece, dofs, aofs, bofs, done, 8, TCG_TYPE_V64, g->load_dest, g->fniv); } else if (g->fni8) { @@ -596,7 +604,7 @@ void tcg_gen_gvec_3(uint32_t dofs, uint32_t aofs, uint32_t bofs, } do_ool: - tcg_gen_gvec_3_ool(dofs, aofs, bofs, oprsz, maxsz, 0, g->fno); + tcg_gen_gvec_3_ool(dofs, aofs, bofs, oprsz, maxsz, g->data, g->fno); } /* @@ -1015,3 +1023,316 @@ void tcg_gen_gvec_orc(unsigned vece, uint32_t dofs, uint32_t aofs, }; tcg_gen_gvec_3(dofs, aofs, bofs, opsz, maxsz, &g); } + +static void do_zip(unsigned vece, uint32_t dofs, uint32_t aofs, + uint32_t bofs, uint32_t oprsz, uint32_t maxsz, + bool high) +{ + static gen_helper_gvec_3 * const zip_fns[4] = { + gen_helper_gvec_zip8, + gen_helper_gvec_zip16, + gen_helper_gvec_zip32, + gen_helper_gvec_zip64, + }; + + TCGType type; + uint32_t step, i, n; + TCGOpcode zip_op; + + check_size_align(oprsz, maxsz, dofs | aofs | bofs); + check_overlap_3(dofs, aofs, bofs, oprsz); + tcg_debug_assert(vece <= MO_64); + + /* Quick check for sizes we won't support inline. */ + if (oprsz > 4 * 32 || maxsz > MAX_UNROLL * 32) { + goto do_ool; + } + + zip_op = high ? INDEX_op_ziph_vec : INDEX_op_zipl_vec; + + /* Since these operations don't operate in lock-step lanes, + we must care for overlap. */ + if (TCG_TARGET_HAS_v256 && oprsz % 32 == 0 && oprsz / 32 <= 8 + && tcg_can_emit_vec_op(zip_op, TCG_TYPE_V256, vece)) { + type = TCG_TYPE_V256; + step = 32; + n = oprsz / 32; + } else if (TCG_TARGET_HAS_v128 && oprsz % 16 == 0 && oprsz / 16 <= 8 + && tcg_can_emit_vec_op(zip_op, TCG_TYPE_V128, vece)) { + type = TCG_TYPE_V128; + step = 16; + n = oprsz / 16; + } else if (TCG_TARGET_HAS_v64 && oprsz % 8 == 0 && oprsz / 8 <= 8 + && tcg_can_emit_vec_op(zip_op, TCG_TYPE_V64, vece)) { + type = TCG_TYPE_V64; + step = 8; + n = oprsz / 8; + } else { + goto do_ool; + } + + if (n == 1) { + TCGv_vec t1 = tcg_temp_new_vec(type); + TCGv_vec t2 = tcg_temp_new_vec(type); + + tcg_gen_ld_vec(t1, cpu_env, aofs); + tcg_gen_ld_vec(t2, cpu_env, bofs); + if (high) { + tcg_gen_ziph_vec(vece, t1, t1, t2); + } else { + tcg_gen_zipl_vec(vece, t1, t1, t2); + } + tcg_gen_st_vec(t1, cpu_env, dofs); + tcg_temp_free_vec(t1); + tcg_temp_free_vec(t2); + } else { + TCGv_vec ta[4], tb[4], tmp; + + if (high) { + aofs += oprsz / 2; + bofs += oprsz / 2; + } + + for (i = 0; i < (n / 2 + n % 2); ++i) { + ta[i] = tcg_temp_new_vec(type); + tb[i] = tcg_temp_new_vec(type); + tcg_gen_ld_vec(ta[i], cpu_env, aofs + i * step); + tcg_gen_ld_vec(tb[i], cpu_env, bofs + i * step); + } + + tmp = tcg_temp_new_vec(type); + for (i = 0; i < n; ++i) { + if (i & 1) { + tcg_gen_ziph_vec(vece, tmp, ta[i / 2], tb[i / 2]); + } else { + tcg_gen_zipl_vec(vece, tmp, ta[i / 2], tb[i / 2]); + } + tcg_gen_st_vec(tmp, cpu_env, dofs + i * step); + } + tcg_temp_free_vec(tmp); + + for (i = 0; i < (n / 2 + n % 2); ++i) { + tcg_temp_free_vec(ta[i]); + tcg_temp_free_vec(tb[i]); + } + } + if (oprsz < maxsz) { + expand_clr(dofs + oprsz, maxsz - oprsz); + } + return; + + do_ool: + if (high) { + aofs += oprsz / 2; + bofs += oprsz / 2; + } + tcg_gen_gvec_3_ool(dofs, aofs, bofs, oprsz, maxsz, 0, zip_fns[vece]); +} + +void tcg_gen_gvec_zipl(unsigned vece, uint32_t dofs, uint32_t aofs, + uint32_t bofs, uint32_t oprsz, uint32_t maxsz) +{ + do_zip(vece, dofs, aofs, bofs, oprsz, maxsz, false); +} + +void tcg_gen_gvec_ziph(unsigned vece, uint32_t dofs, uint32_t aofs, + uint32_t bofs, uint32_t oprsz, uint32_t maxsz) +{ + do_zip(vece, dofs, aofs, bofs, oprsz, maxsz, true); +} + +static void do_uzp(unsigned vece, uint32_t dofs, uint32_t aofs, + uint32_t bofs, uint32_t oprsz, uint32_t maxsz, bool odd) +{ + static gen_helper_gvec_3 * const uzp_fns[4] = { + gen_helper_gvec_uzp8, + gen_helper_gvec_uzp16, + gen_helper_gvec_uzp32, + gen_helper_gvec_uzp64, + }; + + TCGType type; + uint32_t step, i, n; + TCGv_vec t[8]; + TCGOpcode uzp_op; + + check_size_align(oprsz, maxsz, dofs | aofs | bofs); + check_overlap_3(dofs, aofs, bofs, oprsz); + tcg_debug_assert(vece <= MO_64); + + /* Quick check for sizes we won't support inline. */ + if (oprsz > 4 * 32 || maxsz > MAX_UNROLL * 32) { + goto do_ool; + } + + uzp_op = odd ? INDEX_op_uzpo_vec : INDEX_op_uzpe_vec; + + /* Since these operations don't operate in lock-step lanes, + we must care for overlap. */ + if (TCG_TARGET_HAS_v256 && oprsz % 32 == 0 && oprsz / 32 <= 4 + && tcg_can_emit_vec_op(uzp_op, TCG_TYPE_V256, vece)) { + type = TCG_TYPE_V256; + step = 32; + n = oprsz / 32; + } else if (TCG_TARGET_HAS_v128 && oprsz % 16 == 0 && oprsz / 16 <= 4 + && tcg_can_emit_vec_op(uzp_op, TCG_TYPE_V128, vece)) { + type = TCG_TYPE_V128; + step = 16; + n = oprsz / 16; + } else if (TCG_TARGET_HAS_v64 && oprsz % 8 == 0 && oprsz / 8 <= 4 + && tcg_can_emit_vec_op(uzp_op, TCG_TYPE_V64, vece)) { + type = TCG_TYPE_V64; + step = 8; + n = oprsz / 8; + } else { + goto do_ool; + } + + for (i = 0; i < n; ++i) { + t[i] = tcg_temp_new_vec(type); + tcg_gen_ld_vec(t[i], cpu_env, aofs + i * step); + } + for (i = 0; i < n; ++i) { + t[n + i] = tcg_temp_new_vec(type); + tcg_gen_ld_vec(t[n + i], cpu_env, bofs + i * step); + } + for (i = 0; i < n; ++i) { + if (odd) { + tcg_gen_uzpo_vec(vece, t[2 * i], t[2 * i], t[2 * i + 1]); + } else { + tcg_gen_uzpe_vec(vece, t[2 * i], t[2 * i], t[2 * i + 1]); + } + tcg_gen_st_vec(t[2 * i], cpu_env, dofs + i * step); + tcg_temp_free_vec(t[2 * i]); + tcg_temp_free_vec(t[2 * i + 1]); + } + if (oprsz < maxsz) { + expand_clr(dofs + oprsz, maxsz - oprsz); + } + return; + + do_ool: + tcg_gen_gvec_3_ool(dofs, aofs, bofs, oprsz, maxsz, + (1 << vece) * odd, uzp_fns[vece]); +} + +void tcg_gen_gvec_uzpe(unsigned vece, uint32_t dofs, uint32_t aofs, + uint32_t bofs, uint32_t oprsz, uint32_t maxsz) +{ + do_uzp(vece, dofs, aofs, bofs, oprsz, maxsz, false); +} + +void tcg_gen_gvec_uzpo(unsigned vece, uint32_t dofs, uint32_t aofs, + uint32_t bofs, uint32_t oprsz, uint32_t maxsz) +{ + do_uzp(vece, dofs, aofs, bofs, oprsz, maxsz, true); +} + +static void gen_trne8_i64(TCGv_i64 d, TCGv_i64 a, TCGv_i64 b) +{ + uint64_t m = 0x00ff00ff00ff00ffull; + tcg_gen_andi_i64(a, a, m); + tcg_gen_andi_i64(b, b, m); + tcg_gen_shli_i64(b, b, 8); + tcg_gen_or_i64(d, a, b); +} + +static void gen_trne16_i64(TCGv_i64 d, TCGv_i64 a, TCGv_i64 b) +{ + uint64_t m = 0x0000ffff0000ffffull; + tcg_gen_andi_i64(a, a, m); + tcg_gen_andi_i64(b, b, m); + tcg_gen_shli_i64(b, b, 16); + tcg_gen_or_i64(d, a, b); +} + +static void gen_trne32_i64(TCGv_i64 d, TCGv_i64 a, TCGv_i64 b) +{ + tcg_gen_deposit_i64(d, a, b, 32, 32); +} + +void tcg_gen_gvec_trne(unsigned vece, uint32_t dofs, uint32_t aofs, + uint32_t bofs, uint32_t opsz, uint32_t maxsz) +{ + static const GVecGen3 g[4] = { + { .fni8 = gen_trne8_i64, + .fniv = tcg_gen_trne_vec, + .fno = gen_helper_gvec_trn8, + .opc = INDEX_op_trne_vec, + .vece = MO_8 }, + { .fni8 = gen_trne16_i64, + .fniv = tcg_gen_trne_vec, + .fno = gen_helper_gvec_trn16, + .opc = INDEX_op_trne_vec, + .vece = MO_16 }, + { .fni8 = gen_trne32_i64, + .fniv = tcg_gen_trne_vec, + .fno = gen_helper_gvec_trn32, + .opc = INDEX_op_trne_vec, + .vece = MO_32 }, + { .fniv = tcg_gen_trne_vec, + .fno = gen_helper_gvec_trn64, + .opc = INDEX_op_trne_vec, + .vece = MO_64 }, + }; + + tcg_debug_assert(vece <= MO_64); + tcg_gen_gvec_3(dofs, aofs, bofs, opsz, maxsz, &g[vece]); +} + +static void gen_trno8_i64(TCGv_i64 d, TCGv_i64 a, TCGv_i64 b) +{ + uint64_t m = 0xff00ff00ff00ff00ull; + tcg_gen_andi_i64(a, a, m); + tcg_gen_andi_i64(b, b, m); + tcg_gen_shri_i64(a, a, 8); + tcg_gen_or_i64(d, a, b); +} + +static void gen_trno16_i64(TCGv_i64 d, TCGv_i64 a, TCGv_i64 b) +{ + uint64_t m = 0xffff0000ffff0000ull; + tcg_gen_andi_i64(a, a, m); + tcg_gen_andi_i64(b, b, m); + tcg_gen_shri_i64(a, a, 16); + tcg_gen_or_i64(d, a, b); +} + +static void gen_trno32_i64(TCGv_i64 d, TCGv_i64 a, TCGv_i64 b) +{ + tcg_gen_shri_i64(a, a, 32); + tcg_gen_deposit_i64(d, b, a, 0, 32); +} + +void tcg_gen_gvec_trno(unsigned vece, uint32_t dofs, uint32_t aofs, + uint32_t bofs, uint32_t opsz, uint32_t maxsz) +{ + static const GVecGen3 g[4] = { + { .fni8 = gen_trno8_i64, + .fniv = tcg_gen_trno_vec, + .fno = gen_helper_gvec_trn8, + .opc = INDEX_op_trno_vec, + .data = 1, + .vece = MO_8 }, + { .fni8 = gen_trno16_i64, + .fniv = tcg_gen_trno_vec, + .fno = gen_helper_gvec_trn16, + .opc = INDEX_op_trno_vec, + .data = 2, + .vece = MO_16 }, + { .fni8 = gen_trno32_i64, + .fniv = tcg_gen_trno_vec, + .fno = gen_helper_gvec_trn32, + .opc = INDEX_op_trno_vec, + .data = 4, + .vece = MO_32 }, + { .fniv = tcg_gen_trno_vec, + .fno = gen_helper_gvec_trn64, + .opc = INDEX_op_trno_vec, + .data = 8, + .vece = MO_64 }, + }; + + tcg_debug_assert(vece <= MO_64); + tcg_gen_gvec_3(dofs, aofs, bofs, opsz, maxsz, &g[vece]); +} diff --git a/tcg/tcg-op-vec.c b/tcg/tcg-op-vec.c index 5cfe4af6bd..a5d0ff89c3 100644 --- a/tcg/tcg-op-vec.c +++ b/tcg/tcg-op-vec.c @@ -380,3 +380,58 @@ void tcg_gen_neg_vec(unsigned vece, TCGv_vec r, TCGv_vec a) tcg_temp_free_vec(t); } } + +static void do_interleave(TCGOpcode opc, unsigned vece, + TCGv_vec r, TCGv_vec a, TCGv_vec b) +{ + TCGTemp *rt = tcgv_vec_temp(r); + TCGTemp *at = tcgv_vec_temp(a); + TCGTemp *bt = tcgv_vec_temp(b); + TCGArg ri = temp_arg(rt); + TCGArg ai = temp_arg(at); + TCGArg bi = temp_arg(bt); + TCGType type = rt->base_type; + unsigned vecl = type - TCG_TYPE_V64; + int can; + + tcg_debug_assert(at->base_type == type); + tcg_debug_assert(bt->base_type == type); + tcg_debug_assert((8 << vece) <= (32 << vecl)); + can = tcg_can_emit_vec_op(opc, type, vece); + if (can > 0) { + vec_gen_3(opc, type, vece, ri, ai, bi); + } else { + tcg_debug_assert(can < 0); + tcg_expand_vec_op(opc, type, vece, ri, ai, bi); + } +} + +void tcg_gen_zipl_vec(unsigned vece, TCGv_vec r, TCGv_vec a, TCGv_vec b) +{ + do_interleave(INDEX_op_zipl_vec, vece, r, a, b); +} + +void tcg_gen_ziph_vec(unsigned vece, TCGv_vec r, TCGv_vec a, TCGv_vec b) +{ + do_interleave(INDEX_op_ziph_vec, vece, r, a, b); +} + +void tcg_gen_uzpe_vec(unsigned vece, TCGv_vec r, TCGv_vec a, TCGv_vec b) +{ + do_interleave(INDEX_op_uzpe_vec, vece, r, a, b); +} + +void tcg_gen_uzpo_vec(unsigned vece, TCGv_vec r, TCGv_vec a, TCGv_vec b) +{ + do_interleave(INDEX_op_uzpo_vec, vece, r, a, b); +} + +void tcg_gen_trne_vec(unsigned vece, TCGv_vec r, TCGv_vec a, TCGv_vec b) +{ + do_interleave(INDEX_op_trne_vec, vece, r, a, b); +} + +void tcg_gen_trno_vec(unsigned vece, TCGv_vec r, TCGv_vec a, TCGv_vec b) +{ + do_interleave(INDEX_op_trno_vec, vece, r, a, b); +} diff --git a/tcg/tcg.c b/tcg/tcg.c index e725b1818f..ec7db4e82d 100644 --- a/tcg/tcg.c +++ b/tcg/tcg.c @@ -1403,6 +1403,15 @@ bool tcg_op_supported(TCGOpcode op) return have_vec && TCG_TARGET_HAS_andc_vec; case INDEX_op_orc_vec: return have_vec && TCG_TARGET_HAS_orc_vec; + case INDEX_op_zipl_vec: + case INDEX_op_ziph_vec: + return have_vec && TCG_TARGET_HAS_zip_vec; + case INDEX_op_uzpe_vec: + case INDEX_op_uzpo_vec: + return have_vec && TCG_TARGET_HAS_uzp_vec; + case INDEX_op_trne_vec: + case INDEX_op_trno_vec: + return have_vec && TCG_TARGET_HAS_trn_vec; default: tcg_debug_assert(op > INDEX_op_last_generic && op < NB_OPS); diff --git a/tcg/README b/tcg/README index e14990fb9b..8ab8d3ab7e 100644 --- a/tcg/README +++ b/tcg/README @@ -561,6 +561,46 @@ E.g. VECL=1 -> 64 << 1 -> v128, and VECE=2 -> 1 << 2 -> i32. Similarly, logical operations with and without compliment. Note that VECE is unused. +* zipl_vec v0, v1, v2 +* ziph_vec v0, v1, v2 + + "Zip" two vectors together, either the low half of v1/v2 or the high half. + The name comes from ARM ARM; the equivalent function in Intel terminology + is the less scrutable "punpck". The effect is + + part = ("high" ? VECL/VECE/2 : 0); + for (i = 0; i < VECL/VECE/2; ++i) { + v0[2i + 0] = v1[i + part]; + v0[2i + 1] = v2[i + part]; + } + +* uzpe_vec v0, v1, v2 +* uzpo_vec v0, v1, v2 + + "Unzip" two vectors, either the even elements or the odd elements. + If v1 and v2 are the result of zipl and ziph, this performs the + inverse operation. The effect is + + part = ("odd" ? 1 : 0) + for (i = 0; i < VECL/VECE/2; ++i) { + v0[i] = v1[2i + part]; + } + for (i = 0; i < VECL/VECE/2; ++i) { + v0[i + VECL/VECE/2] = v1[2i + part]; + } + +* trne_vec v0, v1, v2 +* trno_vec v1, v1, v2 + + "Transpose" two vectors, either the even elements or the odd elements. + The effect is + + part = ("odd" ? 1 : 0) + for (i = 0; i < VECL/VECE/2; ++i) { + v0[2i + 0] = v1[2i + part]; + v0[2i + 1] = v2[2i + part]; + } + ********* Note 1: Some shortcuts are defined when the last operand is known to be From patchwork Tue Nov 21 21:25:23 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 119424 Delivered-To: patch@linaro.org Received: by 10.140.22.164 with SMTP id 33csp5826445qgn; Tue, 21 Nov 2017 13:44:47 -0800 (PST) X-Google-Smtp-Source: AGs4zMaIQCXaF0r4sg1MaMS3jcQAmxcCryQIi/CODAH5ywJSYYXYr9PkLzuBWBrTD0q+1+YjXoVJ X-Received: by 10.129.76.132 with SMTP id z126mr11775090ywa.67.1511300687539; Tue, 21 Nov 2017 13:44:47 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1511300687; cv=none; d=google.com; s=arc-20160816; b=PqxvcurJ5cGNiiXn9Y0SwCCoJExBxOLxNzpXvi7pxK1jTFGFb7DLUQVy7VPE6ETaob dJdh6is28b8gHgxT976XBMvQ4SyVnyWkk9RWzXBQRIB2W0LcbVp8uV0YRstQz8sPCWsl kiP/V8DlGqPwhdL/dXJxVqNK3bgGVTimGRYDmSmXziOYirTW4zLmWWql9zdIXLK4muid 1xw3BdH53d6wBRRRTQ7X3V4Smir0dYVP/LVnuEFlvwt+i6H7xb7DWXEcm7mAKf9pi2x2 9V+ZU1MBez2uYHJh+HeCkpZzlBKiZzTVHvbkKK5mtITgAH3E9Q/KRrhxTDkvgGf3eCGE KtnA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:subject:references:in-reply-to :message-id:date:to:from:dkim-signature:arc-authentication-results; bh=QuBWrKgNGFu9gomUQwOFUeI255DsQ+2ZZs3Ag3qWXg8=; b=xgw3d1XFNFmvGbwlMEJgm5V5Mmz7zBIrgDl4IykRmUm9V6Wy0BuYW30qJwOV9pajAO k48FVPSiYMq2vA2hzGFnNRaGipVskmTwQgvjoNBvFmOX7fR8ZZjSe/FsoFduRvSQ9IEV mEdJTWxKfYxVebot796vLO/ATlqg7fvTokRsWK4hh6TUNvwov29Q/ifK8HWpw6ux+rVt zI1h08OCw2aa9pV0cpAjrvMSy6qI6RRK/My3cMAL7hSo+rkgE2aRjDJVjiQP1zrFpi75 r57nHWix6ED8+BWBmWUINGfBtZRwnaFKt0gzuKedsN720GN42FEvRCd/7xHtwkK5GBUw jpJw== ARC-Authentication-Results: i=1; mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=g3e15Va0; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) smtp.mailfrom=qemu-devel-bounces+patch=linaro.org@nongnu.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [2001:4830:134:3::11]) by mx.google.com with ESMTPS id 64si2935644ybw.344.2017.11.21.13.44.47 for (version=TLS1 cipher=AES128-SHA bits=128/128); Tue, 21 Nov 2017 13:44:47 -0800 (PST) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) client-ip=2001:4830:134:3::11; Authentication-Results: mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=g3e15Va0; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) smtp.mailfrom=qemu-devel-bounces+patch=linaro.org@nongnu.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1]:36668 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1eHGLW-0002sK-Tp for patch@linaro.org; Tue, 21 Nov 2017 16:44:46 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:54114) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1eHG5H-0004Vb-Gi for qemu-devel@nongnu.org; Tue, 21 Nov 2017 16:28:00 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1eHG5E-0007lC-Nv for qemu-devel@nongnu.org; Tue, 21 Nov 2017 16:27:59 -0500 Received: from mail-wm0-x242.google.com ([2a00:1450:400c:c09::242]:44107) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1eHG5E-0007kl-Fh for qemu-devel@nongnu.org; Tue, 21 Nov 2017 16:27:56 -0500 Received: by mail-wm0-x242.google.com with SMTP id r68so6353176wmr.3 for ; Tue, 21 Nov 2017 13:27:56 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:subject:date:message-id:in-reply-to:references; bh=QuBWrKgNGFu9gomUQwOFUeI255DsQ+2ZZs3Ag3qWXg8=; b=g3e15Va0sqQlal7Y1aZXsc5T2qCh+1NdEVN5xtsiizacWtTOGlc4SFD1XjN0dX0HKP v05AgQlM9vX2JYI/Gt1f8H5IuKo1HXqd6xjgY7OHpx7CH/S/A5+j26JbYUBKLwmYKm6X dVcn8TG8tr9YWT6fh9zaWFuiJLgLZd8vXZhwc= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references; bh=QuBWrKgNGFu9gomUQwOFUeI255DsQ+2ZZs3Ag3qWXg8=; b=JonH5iumkig1oTkt7qJhd6so8P6XmYvTQ/WfyWayRYBZDOleoxyBVs54VL1Y96bQxn De+9Z/aAtlf1Q7Riw1KgnR5rBBy8qZZrvdA75W7Vo5MmxLYY+UxZTrmApn/T0jr40t9o JhBHxtlWfNtZrTPqDA9+ELF2Fjmr5YL6qAtpEnt947eSZrEey+JldLGDAvE+gqiQUW1S D7oljLcx2S9hQ7PbUWYkWet3mE4STg98kCczj2wDIHSaD7FS4X8jxHkVLes3S67pRuuL QOD6pCylMagoI5wN+p/G85sbNFdhDZxL95xiyJg4fMA4RDyGR7XpbGFmMOIYzmilo23e kv6A== X-Gm-Message-State: AJaThX7Z8TYjJfgWlPMOf6okwfAQ8lINx+e0TPOBcawsQH2XZpoEocAL GYsHfRDm+Np4kGSwDIL+yrdJY0uA78M= X-Received: by 10.28.127.22 with SMTP id a22mr2260239wmd.12.1511299675300; Tue, 21 Nov 2017 13:27:55 -0800 (PST) Received: from cloudburst.twiddle.net (70.red-37-158-60.dynamicip.rima-tde.net. [37.158.60.70]) by smtp.gmail.com with ESMTPSA id e124sm706517wmg.34.2017.11.21.13.27.54 for (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Tue, 21 Nov 2017 13:27:54 -0800 (PST) From: Richard Henderson To: qemu-devel@nongnu.org Date: Tue, 21 Nov 2017 22:25:23 +0100 Message-Id: <20171121212534.5177-16-richard.henderson@linaro.org> X-Mailer: git-send-email 2.13.6 In-Reply-To: <20171121212534.5177-1-richard.henderson@linaro.org> References: <20171121212534.5177-1-richard.henderson@linaro.org> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2a00:1450:400c:c09::242 Subject: [Qemu-devel] [PATCH v6 15/26] target/arm: Use vector infrastructure for aa64 zip/uzp/trn/xtn X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: "Qemu-devel" Signed-off-by: Richard Henderson --- target/arm/translate-a64.c | 103 +++++++++++++++------------------------------ 1 file changed, 35 insertions(+), 68 deletions(-) -- 2.13.6 diff --git a/target/arm/translate-a64.c b/target/arm/translate-a64.c index 55a4902fc2..8769b4505a 100644 --- a/target/arm/translate-a64.c +++ b/target/arm/translate-a64.c @@ -5576,11 +5576,7 @@ static void disas_simd_zip_trn(DisasContext *s, uint32_t insn) int opcode = extract32(insn, 12, 2); bool part = extract32(insn, 14, 1); bool is_q = extract32(insn, 30, 1); - int esize = 8 << size; - int i, ofs; - int datasize = is_q ? 128 : 64; - int elements = datasize / esize; - TCGv_i64 tcg_res, tcg_resl, tcg_resh; + GVecGen3Fn *gvec_fn; if (opcode == 0 || (size == 3 && !is_q)) { unallocated_encoding(s); @@ -5591,60 +5587,24 @@ static void disas_simd_zip_trn(DisasContext *s, uint32_t insn) return; } - tcg_resl = tcg_const_i64(0); - tcg_resh = tcg_const_i64(0); - tcg_res = tcg_temp_new_i64(); - - for (i = 0; i < elements; i++) { - switch (opcode) { - case 1: /* UZP1/2 */ - { - int midpoint = elements / 2; - if (i < midpoint) { - read_vec_element(s, tcg_res, rn, 2 * i + part, size); - } else { - read_vec_element(s, tcg_res, rm, - 2 * (i - midpoint) + part, size); - } - break; - } - case 2: /* TRN1/2 */ - if (i & 1) { - read_vec_element(s, tcg_res, rm, (i & ~1) + part, size); - } else { - read_vec_element(s, tcg_res, rn, (i & ~1) + part, size); - } - break; - case 3: /* ZIP1/2 */ - { - int base = part * elements / 2; - if (i & 1) { - read_vec_element(s, tcg_res, rm, base + (i >> 1), size); - } else { - read_vec_element(s, tcg_res, rn, base + (i >> 1), size); - } - break; - } - default: - g_assert_not_reached(); - } - - ofs = i * esize; - if (ofs < 64) { - tcg_gen_shli_i64(tcg_res, tcg_res, ofs); - tcg_gen_or_i64(tcg_resl, tcg_resl, tcg_res); - } else { - tcg_gen_shli_i64(tcg_res, tcg_res, ofs - 64); - tcg_gen_or_i64(tcg_resh, tcg_resh, tcg_res); - } + switch (opcode) { + case 1: /* UZP1/2 */ + gvec_fn = part ? tcg_gen_gvec_uzpo : tcg_gen_gvec_uzpe; + break; + case 2: /* TRN1/2 */ + gvec_fn = part ? tcg_gen_gvec_trno : tcg_gen_gvec_trne; + break; + case 3: /* ZIP1/2 */ + gvec_fn = part ? tcg_gen_gvec_ziph : tcg_gen_gvec_zipl; + break; + default: + g_assert_not_reached(); } - tcg_temp_free_i64(tcg_res); - - write_vec_element(s, tcg_resl, rd, 0, MO_64); - tcg_temp_free_i64(tcg_resl); - write_vec_element(s, tcg_resh, rd, 1, MO_64); - tcg_temp_free_i64(tcg_resh); + gvec_fn(size, vec_full_reg_offset(s, rd), + vec_full_reg_offset(s, rn), + vec_full_reg_offset(s, rm), + is_q ? 16 : 8, vec_full_reg_size(s)); } static void do_minmaxop(DisasContext *s, TCGv_i32 tcg_elt1, TCGv_i32 tcg_elt2, @@ -7922,6 +7882,22 @@ static void handle_2misc_narrow(DisasContext *s, bool scalar, int destelt = is_q ? 2 : 0; int passes = scalar ? 1 : 2; + if (opcode == 0x12 && !u) { /* XTN, XTN2 */ + tcg_debug_assert(!scalar); + if (is_q) { /* XTN2 */ + tcg_gen_gvec_uzpe(size, vec_reg_offset(s, rd, 1, MO_64), + vec_reg_offset(s, rn, 0, MO_64), + vec_reg_offset(s, rn, 1, MO_64), + 8, vec_full_reg_size(s) - 8); + } else { + tcg_gen_gvec_uzpe(size, vec_reg_offset(s, rd, 0, MO_64), + vec_reg_offset(s, rn, 0, MO_64), + vec_reg_offset(s, rn, 1, MO_64), + 8, vec_full_reg_size(s)); + } + return; + } + if (scalar) { tcg_res[1] = tcg_const_i32(0); } @@ -7939,23 +7915,14 @@ static void handle_2misc_narrow(DisasContext *s, bool scalar, tcg_res[pass] = tcg_temp_new_i32(); switch (opcode) { - case 0x12: /* XTN, SQXTUN */ + case 0x12: /* , SQXTUN */ { - static NeonGenNarrowFn * const xtnfns[3] = { - gen_helper_neon_narrow_u8, - gen_helper_neon_narrow_u16, - tcg_gen_extrl_i64_i32, - }; static NeonGenNarrowEnvFn * const sqxtunfns[3] = { gen_helper_neon_unarrow_sat8, gen_helper_neon_unarrow_sat16, gen_helper_neon_unarrow_sat32, }; - if (u) { - genenvfn = sqxtunfns[size]; - } else { - genfn = xtnfns[size]; - } + genenvfn = sqxtunfns[size]; break; } case 0x14: /* SQXTN, UQXTN */ From patchwork Tue Nov 21 21:25:24 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 119411 Delivered-To: patch@linaro.org Received: by 10.140.22.164 with SMTP id 33csp5816431qgn; Tue, 21 Nov 2017 13:33:43 -0800 (PST) X-Google-Smtp-Source: AGs4zMbCZFt2o3nmvzsLK+fwEWnlrNJCNo8i/wQ9X5DfJJUk4KR6CJy31WH37Njwyyzb1WlGWDQA X-Received: by 10.129.198.3 with SMTP id l3mr11974035ywi.385.1511300023232; Tue, 21 Nov 2017 13:33:43 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1511300023; cv=none; d=google.com; s=arc-20160816; b=T2DBgBqlrqRRkmJ0T/WgCBX7o+uzmq6yqAY1bicN8SzmClumH35KsdAaYsTpFun5lQ fojn3zzgAUUw1DJoJGIcCF9T+ws7I6KVdAuAltFgP1vH/FGCvb660jWIVtWD0cTU7+de shJKSOusAaoqLi1BTUAoa4X0tn0zy9owdQIXloMLIFENq7qeAF2EfDnK6c9jUGFe22s3 cnCC8oUG4ZtBUdbmq4RvPQGb20dmtYeazPYrAnKmuOmqHqOk1T48CaGzEvkCKXHx4Dxi MwOUUE4Yswfkx4bpXBJFPb9G71FlI0bsA10fn4BTXm+ebFFgLqNWqTip6sul0J7yra12 0UQA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:subject:references:in-reply-to :message-id:date:to:from:dkim-signature:arc-authentication-results; bh=gEP25eYik6fxPR0wtymkd7b3FZO+RSo0Zu2+3q/rHkE=; b=FeI7k5s7y5Tl5cfMtQiQsANZUkDuT15OYavNLLV5s4Umlkxu7ifyAyaEYthmu5wxOI 6VwxK08BIzk2JvsJNbLK6STsqi47BvFdi5LhkBD+tfn6wc5gf5T+hwA6cMwpHO1gzUet fWRIaGww1L2KKgASgLq8xIewbxh3W8W+uF+/iTaDn1J+QRRrKPXtnDAiTQWnIGeRhMZe 1KG6FoUteP22gdQVYyDf8gu8XgC9YwccBGaum2Ogr/8yRqL15a8ms969Zp6FwhhvOOI5 RF/FEp9ltI56Z9wzMPYy5yOhs+Tm0UtVHwl/TuGj/Ge+0O84dnYVsY5KGn0drYsHe3f7 UJuQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=IxsBGpyP; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) smtp.mailfrom=qemu-devel-bounces+patch=linaro.org@nongnu.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [2001:4830:134:3::11]) by mx.google.com with ESMTPS id r129si2229655ywr.779.2017.11.21.13.33.43 for (version=TLS1 cipher=AES128-SHA bits=128/128); Tue, 21 Nov 2017 13:33:43 -0800 (PST) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) client-ip=2001:4830:134:3::11; Authentication-Results: mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=IxsBGpyP; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) smtp.mailfrom=qemu-devel-bounces+patch=linaro.org@nongnu.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1]:36606 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1eHGAo-0000Wz-KU for patch@linaro.org; Tue, 21 Nov 2017 16:33:42 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:54134) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1eHG5J-0004Z4-Od for qemu-devel@nongnu.org; Tue, 21 Nov 2017 16:28:04 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1eHG5G-0007ll-Nx for qemu-devel@nongnu.org; Tue, 21 Nov 2017 16:28:01 -0500 Received: from mail-wr0-x244.google.com ([2a00:1450:400c:c0c::244]:35737) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1eHG5G-0007lS-9t for qemu-devel@nongnu.org; Tue, 21 Nov 2017 16:27:58 -0500 Received: by mail-wr0-x244.google.com with SMTP id w95so12649198wrc.2 for ; Tue, 21 Nov 2017 13:27:58 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:subject:date:message-id:in-reply-to:references; bh=gEP25eYik6fxPR0wtymkd7b3FZO+RSo0Zu2+3q/rHkE=; b=IxsBGpyP703d0vh40qgaUBZbDkPLrMHs3N7Uyzp2/jR7e5e7jWYRBd1SE6T53Subpp RaZhShZhy4x3yrkSdSZBRHM70XpVI7gIRxsNssgE5arGaO1XK8YPtTppiLaiZUI9D5Xu vIzDpB5DPlfowtnlDQKFHs1dZE8R/qn5mgHdw= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references; bh=gEP25eYik6fxPR0wtymkd7b3FZO+RSo0Zu2+3q/rHkE=; b=Ny0uDQsMadD5+UGMJwrkTMxrhZ3t3hkkW5MEIlEBqwj9uOihK7rv5qTRMuioklgqkw T0o7+pV22cX4GENwVGG+IQOZ87dF+61ets8eg+wWiyhqRqj7SNnIGR0NsSHEs5NRrtEX nLvQa1v3cEP6r7jblY2bheWzM/7C5BW7HwMEzsU9f3xmgynW3c2Ehb24Q0yH2USpLADM VVDk1EV6ZXRY69CVuNMPqC1o0CDJcPKH8lpN2BR1Fwcwi/bMgw0N7wlY7KZO8WTOG4RJ 5OGc5bPNwmq+OFuNb2EBMi7coAYzKOgQrRG51Osq40L6vPzp36OdvOoDlPNtV9EhFqBi yFKg== X-Gm-Message-State: AJaThX5cjgkrxxhAtqTwuGcUiwbb3AXgJeJbEGmgi/tGhhXSu4hOVHFO 8Yr8EwVOrgy/1clHKOmG2ENi17KaX70= X-Received: by 10.223.184.42 with SMTP id h39mr16378107wrf.38.1511299676694; Tue, 21 Nov 2017 13:27:56 -0800 (PST) Received: from cloudburst.twiddle.net (70.red-37-158-60.dynamicip.rima-tde.net. [37.158.60.70]) by smtp.gmail.com with ESMTPSA id e124sm706517wmg.34.2017.11.21.13.27.55 for (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Tue, 21 Nov 2017 13:27:55 -0800 (PST) From: Richard Henderson To: qemu-devel@nongnu.org Date: Tue, 21 Nov 2017 22:25:24 +0100 Message-Id: <20171121212534.5177-17-richard.henderson@linaro.org> X-Mailer: git-send-email 2.13.6 In-Reply-To: <20171121212534.5177-1-richard.henderson@linaro.org> References: <20171121212534.5177-1-richard.henderson@linaro.org> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2a00:1450:400c:c0c::244 Subject: [Qemu-devel] [PATCH v6 16/26] tcg: Add generic vector ops for constant shifts X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: "Qemu-devel" Opcodes are added for scalar and vector shifts, but considering the varied semantics of these do not expose them to the front ends. Do go ahead and provide them in case they are needed for backend expansion. Signed-off-by: Richard Henderson --- accel/tcg/tcg-runtime.h | 15 +++ tcg/i386/tcg-target.h | 3 + tcg/tcg-op-gvec.h | 35 ++++++ tcg/tcg-op.h | 5 + tcg/tcg-opc.h | 12 ++ tcg/tcg.h | 3 + accel/tcg/tcg-runtime-gvec.c | 149 ++++++++++++++++++++++ tcg/tcg-op-gvec.c | 291 +++++++++++++++++++++++++++++++++++++++++++ tcg/tcg-op-vec.c | 40 ++++++ tcg/tcg.c | 12 ++ tcg/README | 29 +++++ 11 files changed, 594 insertions(+) -- 2.13.6 diff --git a/accel/tcg/tcg-runtime.h b/accel/tcg/tcg-runtime.h index c6de749134..cb05a755b8 100644 --- a/accel/tcg/tcg-runtime.h +++ b/accel/tcg/tcg-runtime.h @@ -164,6 +164,21 @@ DEF_HELPER_FLAGS_4(gvec_xor, TCG_CALL_NO_RWG, void, ptr, ptr, ptr, i32) DEF_HELPER_FLAGS_4(gvec_andc, TCG_CALL_NO_RWG, void, ptr, ptr, ptr, i32) DEF_HELPER_FLAGS_4(gvec_orc, TCG_CALL_NO_RWG, void, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_3(gvec_shl8i, TCG_CALL_NO_RWG, void, ptr, ptr, i32) +DEF_HELPER_FLAGS_3(gvec_shl16i, TCG_CALL_NO_RWG, void, ptr, ptr, i32) +DEF_HELPER_FLAGS_3(gvec_shl32i, TCG_CALL_NO_RWG, void, ptr, ptr, i32) +DEF_HELPER_FLAGS_3(gvec_shl64i, TCG_CALL_NO_RWG, void, ptr, ptr, i32) + +DEF_HELPER_FLAGS_3(gvec_shr8i, TCG_CALL_NO_RWG, void, ptr, ptr, i32) +DEF_HELPER_FLAGS_3(gvec_shr16i, TCG_CALL_NO_RWG, void, ptr, ptr, i32) +DEF_HELPER_FLAGS_3(gvec_shr32i, TCG_CALL_NO_RWG, void, ptr, ptr, i32) +DEF_HELPER_FLAGS_3(gvec_shr64i, TCG_CALL_NO_RWG, void, ptr, ptr, i32) + +DEF_HELPER_FLAGS_3(gvec_sar8i, TCG_CALL_NO_RWG, void, ptr, ptr, i32) +DEF_HELPER_FLAGS_3(gvec_sar16i, TCG_CALL_NO_RWG, void, ptr, ptr, i32) +DEF_HELPER_FLAGS_3(gvec_sar32i, TCG_CALL_NO_RWG, void, ptr, ptr, i32) +DEF_HELPER_FLAGS_3(gvec_sar64i, TCG_CALL_NO_RWG, void, ptr, ptr, i32) + DEF_HELPER_FLAGS_4(gvec_zip8, TCG_CALL_NO_RWG, void, ptr, ptr, ptr, i32) DEF_HELPER_FLAGS_4(gvec_zip16, TCG_CALL_NO_RWG, void, ptr, ptr, ptr, i32) DEF_HELPER_FLAGS_4(gvec_zip32, TCG_CALL_NO_RWG, void, ptr, ptr, ptr, i32) diff --git a/tcg/i386/tcg-target.h b/tcg/i386/tcg-target.h index ff0ad7dcdb..92d533eb92 100644 --- a/tcg/i386/tcg-target.h +++ b/tcg/i386/tcg-target.h @@ -177,6 +177,9 @@ extern bool have_avx2; #define TCG_TARGET_HAS_orc_vec 0 #define TCG_TARGET_HAS_not_vec 0 #define TCG_TARGET_HAS_neg_vec 0 +#define TCG_TARGET_HAS_shi_vec 0 +#define TCG_TARGET_HAS_shs_vec 0 +#define TCG_TARGET_HAS_shv_vec 0 #define TCG_TARGET_HAS_zip_vec 0 #define TCG_TARGET_HAS_uzp_vec 0 #define TCG_TARGET_HAS_trn_vec 0 diff --git a/tcg/tcg-op-gvec.h b/tcg/tcg-op-gvec.h index 64270a3c74..de2c0e669a 100644 --- a/tcg/tcg-op-gvec.h +++ b/tcg/tcg-op-gvec.h @@ -77,6 +77,25 @@ typedef struct { typedef struct { /* Expand inline as a 64-bit or 32-bit integer. Only one of these will be non-NULL. */ + void (*fni8)(TCGv_i64, TCGv_i64, unsigned); + void (*fni4)(TCGv_i32, TCGv_i32, unsigned); + /* Expand inline with a host vector type. */ + void (*fniv)(unsigned, TCGv_vec, TCGv_vec, unsigned); + /* Expand out-of-line helper w/descriptor. */ + gen_helper_gvec_2 *fno; + /* The opcode, if any, to which this corresponds. */ + TCGOpcode opc; + /* The vector element size, if applicable. */ + uint8_t vece; + /* Prefer i64 to v64. */ + bool prefer_i64; + /* Load dest as a 3rd source operand. */ + bool load_dest; +} GVecGen2i; + +typedef struct { + /* Expand inline as a 64-bit or 32-bit integer. + Only one of these will be non-NULL. */ void (*fni8)(TCGv_i64, TCGv_i64, TCGv_i64); void (*fni4)(TCGv_i32, TCGv_i32, TCGv_i32); /* Expand inline with a host vector type. */ @@ -97,6 +116,8 @@ typedef struct { void tcg_gen_gvec_2(uint32_t dofs, uint32_t aofs, uint32_t opsz, uint32_t clsz, const GVecGen2 *); +void tcg_gen_gvec_2i(uint32_t dofs, uint32_t aofs, uint32_t opsz, + uint32_t clsz, unsigned c, const GVecGen2i *); void tcg_gen_gvec_3(uint32_t dofs, uint32_t aofs, uint32_t bofs, uint32_t opsz, uint32_t clsz, const GVecGen3 *); @@ -137,6 +158,13 @@ void tcg_gen_gvec_dup16i(uint32_t dofs, uint32_t s, uint32_t m, uint16_t x); void tcg_gen_gvec_dup32i(uint32_t dofs, uint32_t s, uint32_t m, uint32_t x); void tcg_gen_gvec_dup64i(uint32_t dofs, uint32_t s, uint32_t m, uint64_t x); +void tcg_gen_gvec_shli(unsigned vece, uint32_t dofs, uint32_t aofs, + uint32_t opsz, uint32_t clsz, unsigned shift); +void tcg_gen_gvec_shri(unsigned vece, uint32_t dofs, uint32_t aofs, + uint32_t opsz, uint32_t clsz, unsigned shift); +void tcg_gen_gvec_sari(unsigned vece, uint32_t dofs, uint32_t aofs, + uint32_t opsz, uint32_t clsz, unsigned shift); + void tcg_gen_gvec_zipl(unsigned vece, uint32_t dofs, uint32_t aofs, uint32_t bofs, uint32_t opsz, uint32_t clsz); void tcg_gen_gvec_ziph(unsigned vece, uint32_t dofs, uint32_t aofs, @@ -167,3 +195,10 @@ void tcg_gen_vec_add32_i64(TCGv_i64 d, TCGv_i64 a, TCGv_i64 b); void tcg_gen_vec_sub8_i64(TCGv_i64 d, TCGv_i64 a, TCGv_i64 b); void tcg_gen_vec_sub16_i64(TCGv_i64 d, TCGv_i64 a, TCGv_i64 b); void tcg_gen_vec_sub32_i64(TCGv_i64 d, TCGv_i64 a, TCGv_i64 b); + +void tcg_gen_vec_shl8i_i64(TCGv_i64 d, TCGv_i64 a, unsigned); +void tcg_gen_vec_shl16i_i64(TCGv_i64 d, TCGv_i64 a, unsigned); +void tcg_gen_vec_shr8i_i64(TCGv_i64 d, TCGv_i64 a, unsigned); +void tcg_gen_vec_shr16i_i64(TCGv_i64 d, TCGv_i64 a, unsigned); +void tcg_gen_vec_sar8i_i64(TCGv_i64 d, TCGv_i64 a, unsigned); +void tcg_gen_vec_sar16i_i64(TCGv_i64 d, TCGv_i64 a, unsigned); diff --git a/tcg/tcg-op.h b/tcg/tcg-op.h index 733e29b5f8..83478ab006 100644 --- a/tcg/tcg-op.h +++ b/tcg/tcg-op.h @@ -927,6 +927,11 @@ void tcg_gen_andc_vec(unsigned vece, TCGv_vec r, TCGv_vec a, TCGv_vec b); void tcg_gen_orc_vec(unsigned vece, TCGv_vec r, TCGv_vec a, TCGv_vec b); void tcg_gen_not_vec(unsigned vece, TCGv_vec r, TCGv_vec a); void tcg_gen_neg_vec(unsigned vece, TCGv_vec r, TCGv_vec a); + +void tcg_gen_shli_vec(unsigned vece, TCGv_vec r, TCGv_vec a, unsigned i); +void tcg_gen_shri_vec(unsigned vece, TCGv_vec r, TCGv_vec a, unsigned i); +void tcg_gen_sari_vec(unsigned vece, TCGv_vec r, TCGv_vec a, unsigned i); + void tcg_gen_zipl_vec(unsigned vece, TCGv_vec r, TCGv_vec a, TCGv_vec b); void tcg_gen_ziph_vec(unsigned vece, TCGv_vec r, TCGv_vec a, TCGv_vec b); void tcg_gen_uzpe_vec(unsigned vece, TCGv_vec r, TCGv_vec a, TCGv_vec b); diff --git a/tcg/tcg-opc.h b/tcg/tcg-opc.h index c911d62442..a085fc077b 100644 --- a/tcg/tcg-opc.h +++ b/tcg/tcg-opc.h @@ -229,6 +229,18 @@ DEF(andc_vec, 1, 2, 0, IMPLVEC | IMPL(TCG_TARGET_HAS_andc_vec)) DEF(orc_vec, 1, 2, 0, IMPLVEC | IMPL(TCG_TARGET_HAS_orc_vec)) DEF(not_vec, 1, 1, 0, IMPLVEC | IMPL(TCG_TARGET_HAS_not_vec)) +DEF(shli_vec, 1, 1, 1, IMPLVEC | IMPL(TCG_TARGET_HAS_shi_vec)) +DEF(shri_vec, 1, 1, 1, IMPLVEC | IMPL(TCG_TARGET_HAS_shi_vec)) +DEF(sari_vec, 1, 1, 1, IMPLVEC | IMPL(TCG_TARGET_HAS_shi_vec)) + +DEF(shls_vec, 1, 2, 0, IMPLVEC | IMPL(TCG_TARGET_HAS_shs_vec)) +DEF(shrs_vec, 1, 2, 0, IMPLVEC | IMPL(TCG_TARGET_HAS_shs_vec)) +DEF(sars_vec, 1, 2, 0, IMPLVEC | IMPL(TCG_TARGET_HAS_shs_vec)) + +DEF(shlv_vec, 1, 2, 0, IMPLVEC | IMPL(TCG_TARGET_HAS_shv_vec)) +DEF(shrv_vec, 1, 2, 0, IMPLVEC | IMPL(TCG_TARGET_HAS_shv_vec)) +DEF(sarv_vec, 1, 2, 0, IMPLVEC | IMPL(TCG_TARGET_HAS_shv_vec)) + DEF(zipl_vec, 1, 2, 0, IMPLVEC | IMPL(TCG_TARGET_HAS_zip_vec)) DEF(ziph_vec, 1, 2, 0, IMPLVEC | IMPL(TCG_TARGET_HAS_zip_vec)) DEF(uzpe_vec, 1, 2, 0, IMPLVEC | IMPL(TCG_TARGET_HAS_uzp_vec)) diff --git a/tcg/tcg.h b/tcg/tcg.h index c6f7157c60..5f414d880e 100644 --- a/tcg/tcg.h +++ b/tcg/tcg.h @@ -178,6 +178,9 @@ typedef uint64_t TCGRegSet; #define TCG_TARGET_HAS_not_vec 0 #define TCG_TARGET_HAS_andc_vec 0 #define TCG_TARGET_HAS_orc_vec 0 +#define TCG_TARGET_HAS_shi_vec 0 +#define TCG_TARGET_HAS_shs_vec 0 +#define TCG_TARGET_HAS_shv_vec 0 #define TCG_TARGET_HAS_zip_vec 0 #define TCG_TARGET_HAS_uzp_vec 0 #define TCG_TARGET_HAS_trn_vec 0 diff --git a/accel/tcg/tcg-runtime-gvec.c b/accel/tcg/tcg-runtime-gvec.c index 628df811b2..fba62f1192 100644 --- a/accel/tcg/tcg-runtime-gvec.c +++ b/accel/tcg/tcg-runtime-gvec.c @@ -36,6 +36,11 @@ typedef uint16_t vec16 __attribute__((vector_size(16))); typedef uint32_t vec32 __attribute__((vector_size(16))); typedef uint64_t vec64 __attribute__((vector_size(16))); +typedef int8_t svec8 __attribute__((vector_size(16))); +typedef int16_t svec16 __attribute__((vector_size(16))); +typedef int32_t svec32 __attribute__((vector_size(16))); +typedef int64_t svec64 __attribute__((vector_size(16))); + static inline void clear_high(void *d, intptr_t oprsz, uint32_t desc) { intptr_t maxsz = simd_maxsz(desc); @@ -294,6 +299,150 @@ void HELPER(gvec_orc)(void *d, void *a, void *b, uint32_t desc) clear_high(d, oprsz, desc); } +void HELPER(gvec_shl8i)(void *d, void *a, uint32_t desc) +{ + intptr_t oprsz = simd_oprsz(desc); + int shift = simd_data(desc); + intptr_t i; + + for (i = 0; i < oprsz; i += sizeof(vec8)) { + *(vec8 *)(d + i) = *(vec8 *)(a + i) << shift; + } + clear_high(d, oprsz, desc); +} + +void HELPER(gvec_shl16i)(void *d, void *a, uint32_t desc) +{ + intptr_t oprsz = simd_oprsz(desc); + int shift = simd_data(desc); + intptr_t i; + + for (i = 0; i < oprsz; i += sizeof(vec16)) { + *(vec16 *)(d + i) = *(vec16 *)(a + i) << shift; + } + clear_high(d, oprsz, desc); +} + +void HELPER(gvec_shl32i)(void *d, void *a, uint32_t desc) +{ + intptr_t oprsz = simd_oprsz(desc); + int shift = simd_data(desc); + intptr_t i; + + for (i = 0; i < oprsz; i += sizeof(vec32)) { + *(vec32 *)(d + i) = *(vec32 *)(a + i) << shift; + } + clear_high(d, oprsz, desc); +} + +void HELPER(gvec_shl64i)(void *d, void *a, uint32_t desc) +{ + intptr_t oprsz = simd_oprsz(desc); + int shift = simd_data(desc); + intptr_t i; + + for (i = 0; i < oprsz; i += sizeof(vec64)) { + *(vec64 *)(d + i) = *(vec64 *)(a + i) << shift; + } + clear_high(d, oprsz, desc); +} + +void HELPER(gvec_shr8i)(void *d, void *a, uint32_t desc) +{ + intptr_t oprsz = simd_oprsz(desc); + int shift = simd_data(desc); + intptr_t i; + + for (i = 0; i < oprsz; i += sizeof(vec8)) { + *(vec8 *)(d + i) = *(vec8 *)(a + i) >> shift; + } + clear_high(d, oprsz, desc); +} + +void HELPER(gvec_shr16i)(void *d, void *a, uint32_t desc) +{ + intptr_t oprsz = simd_oprsz(desc); + int shift = simd_data(desc); + intptr_t i; + + for (i = 0; i < oprsz; i += sizeof(vec16)) { + *(vec16 *)(d + i) = *(vec16 *)(a + i) >> shift; + } + clear_high(d, oprsz, desc); +} + +void HELPER(gvec_shr32i)(void *d, void *a, uint32_t desc) +{ + intptr_t oprsz = simd_oprsz(desc); + int shift = simd_data(desc); + intptr_t i; + + for (i = 0; i < oprsz; i += sizeof(vec32)) { + *(vec32 *)(d + i) = *(vec32 *)(a + i) >> shift; + } + clear_high(d, oprsz, desc); +} + +void HELPER(gvec_shr64i)(void *d, void *a, uint32_t desc) +{ + intptr_t oprsz = simd_oprsz(desc); + int shift = simd_data(desc); + intptr_t i; + + for (i = 0; i < oprsz; i += sizeof(vec64)) { + *(vec64 *)(d + i) = *(vec64 *)(a + i) >> shift; + } + clear_high(d, oprsz, desc); +} + +void HELPER(gvec_sar8i)(void *d, void *a, uint32_t desc) +{ + intptr_t oprsz = simd_oprsz(desc); + int shift = simd_data(desc); + intptr_t i; + + for (i = 0; i < oprsz; i += sizeof(vec8)) { + *(svec8 *)(d + i) = *(svec8 *)(a + i) >> shift; + } + clear_high(d, oprsz, desc); +} + +void HELPER(gvec_sar16i)(void *d, void *a, uint32_t desc) +{ + intptr_t oprsz = simd_oprsz(desc); + int shift = simd_data(desc); + intptr_t i; + + for (i = 0; i < oprsz; i += sizeof(vec16)) { + *(svec16 *)(d + i) = *(svec16 *)(a + i) >> shift; + } + clear_high(d, oprsz, desc); +} + +void HELPER(gvec_sar32i)(void *d, void *a, uint32_t desc) +{ + intptr_t oprsz = simd_oprsz(desc); + int shift = simd_data(desc); + intptr_t i; + + for (i = 0; i < oprsz; i += sizeof(vec32)) { + *(svec32 *)(d + i) = *(svec32 *)(a + i) >> shift; + } + clear_high(d, oprsz, desc); +} + +void HELPER(gvec_sar64i)(void *d, void *a, uint32_t desc) +{ + intptr_t oprsz = simd_oprsz(desc); + int shift = simd_data(desc); + intptr_t i; + + for (i = 0; i < oprsz; i += sizeof(vec64)) { + *(svec64 *)(d + i) = *(svec64 *)(a + i) >> shift; + } + clear_high(d, oprsz, desc); +} + /* The size of the alloca in the following is currently bounded to 2k. */ #define DO_ZIP(NAME, TYPE) \ diff --git a/tcg/tcg-op-gvec.c b/tcg/tcg-op-gvec.c index a64baa9dcf..f8ccb137eb 100644 --- a/tcg/tcg-op-gvec.c +++ b/tcg/tcg-op-gvec.c @@ -342,6 +342,26 @@ static void expand_2_i32(uint32_t dofs, uint32_t aofs, uint32_t opsz, tcg_temp_free_i32(t0); } +static void expand_2i_i32(uint32_t dofs, uint32_t aofs, uint32_t opsz, + unsigned c, bool load_dest, + void (*fni)(TCGv_i32, TCGv_i32, unsigned)) +{ + TCGv_i32 t0 = tcg_temp_new_i32(); + TCGv_i32 t1 = tcg_temp_new_i32(); + uint32_t i; + + for (i = 0; i < opsz; i += 4) { + tcg_gen_ld_i32(t0, cpu_env, aofs + i); + if (load_dest) { + tcg_gen_ld_i32(t1, cpu_env, dofs + i); + } + fni(t1, t0, c); + tcg_gen_st_i32(t1, cpu_env, dofs + i); + } + tcg_temp_free_i32(t0); + tcg_temp_free_i32(t1); +} + /* Expand OPSZ bytes worth of three-operand operations using i32 elements. */ static void expand_3_i32(uint32_t dofs, uint32_t aofs, uint32_t bofs, uint32_t opsz, bool load_dest, @@ -381,6 +401,26 @@ static void expand_2_i64(uint32_t dofs, uint32_t aofs, uint32_t opsz, tcg_temp_free_i64(t0); } +static void expand_2i_i64(uint32_t dofs, uint32_t aofs, uint32_t opsz, + unsigned c, bool load_dest, + void (*fni)(TCGv_i64, TCGv_i64, unsigned)) +{ + TCGv_i64 t0 = tcg_temp_new_i64(); + TCGv_i64 t1 = tcg_temp_new_i64(); + uint32_t i; + + for (i = 0; i < opsz; i += 8) { + tcg_gen_ld_i64(t0, cpu_env, aofs + i); + if (load_dest) { + tcg_gen_ld_i64(t1, cpu_env, dofs + i); + } + fni(t1, t0, c); + tcg_gen_st_i64(t1, cpu_env, dofs + i); + } + tcg_temp_free_i64(t0); + tcg_temp_free_i64(t1); +} + /* Expand OPSZ bytes worth of three-operand operations using i64 elements. */ static void expand_3_i64(uint32_t dofs, uint32_t aofs, uint32_t bofs, uint32_t opsz, bool load_dest, @@ -421,6 +461,29 @@ static void expand_2_vec(unsigned vece, uint32_t dofs, uint32_t aofs, tcg_temp_free_vec(t0); } +/* Expand OPSZ bytes worth of two-vector operands and an immediate operand + using host vectors. */ +static void expand_2i_vec(unsigned vece, uint32_t dofs, uint32_t aofs, + uint32_t opsz, uint32_t tysz, TCGType type, + unsigned c, bool load_dest, + void (*fni)(unsigned, TCGv_vec, TCGv_vec, unsigned)) +{ + TCGv_vec t0 = tcg_temp_new_vec(type); + TCGv_vec t1 = tcg_temp_new_vec(type); + uint32_t i; + + for (i = 0; i < opsz; i += tysz) { + tcg_gen_ld_vec(t0, cpu_env, aofs + i); + if (load_dest) { + tcg_gen_ld_vec(t1, cpu_env, dofs + i); + } + fni(vece, t1, t0, c); + tcg_gen_st_vec(t1, cpu_env, dofs + i); + } + tcg_temp_free_vec(t0); + tcg_temp_free_vec(t1); +} + /* Expand OPSZ bytes worth of three-operand operations using host vectors. */ static void expand_3_vec(unsigned vece, uint32_t dofs, uint32_t aofs, uint32_t bofs, uint32_t opsz, @@ -523,6 +586,85 @@ void tcg_gen_gvec_2(uint32_t dofs, uint32_t aofs, tcg_gen_gvec_2_ool(dofs, aofs, oprsz, maxsz, g->data, g->fno); } +void tcg_gen_gvec_2i(uint32_t dofs, uint32_t aofs, uint32_t oprsz, + uint32_t maxsz, unsigned c, const GVecGen2i *g) +{ + check_size_align(oprsz, maxsz, dofs | aofs); + check_overlap_2(dofs, aofs, maxsz); + + /* Quick check for sizes we won't support inline. */ + if (oprsz > MAX_UNROLL * 32 || maxsz > MAX_UNROLL * 32) { + goto do_ool; + } + + /* Recall that ARM SVE allows vector sizes that are not a power of 2. + Expand with successively smaller host vector sizes. The intent is + that e.g. oprsz == 80 would be expanded with 2x32 + 1x16. */ + /* ??? For maxsz > oprsz, the host may be able to use an op-sized + operation, zeroing the balance of the register. We can then + use a cl-sized store to implement the clearing without an extra + store operation. This is true for aarch64 and x86_64 hosts. */ + + if (TCG_TARGET_HAS_v256 && check_size_impl(oprsz, 32) + && (!g->opc || tcg_can_emit_vec_op(g->opc, TCG_TYPE_V256, g->vece))) { + uint32_t done = QEMU_ALIGN_DOWN(oprsz, 32); + expand_2i_vec(g->vece, dofs, aofs, done, 32, TCG_TYPE_V256, + c, g->load_dest, g->fniv); + dofs += done; + aofs += done; + oprsz -= done; + maxsz -= done; + } + + if (TCG_TARGET_HAS_v128 && check_size_impl(oprsz, 16) + && (!g->opc || tcg_can_emit_vec_op(g->opc, TCG_TYPE_V128, g->vece))) { + uint32_t done = QEMU_ALIGN_DOWN(oprsz, 16); + expand_2i_vec(g->vece, dofs, aofs, done, 16, TCG_TYPE_V128, + c, g->load_dest, g->fniv); + dofs += done; + aofs += done; + oprsz -= done; + maxsz -= done; + } + + if (check_size_impl(oprsz, 8)) { + uint32_t done = QEMU_ALIGN_DOWN(oprsz, 8); + if (TCG_TARGET_HAS_v64 && !g->prefer_i64 + && (!g->opc + || tcg_can_emit_vec_op(g->opc, TCG_TYPE_V64, g->vece))) { + expand_2i_vec(g->vece, dofs, aofs, done, 8, TCG_TYPE_V64, + c, g->load_dest, g->fniv); + } else if (g->fni8) { + expand_2i_i64(dofs, aofs, done, c, g->load_dest, g->fni8); + } else { + done = 0; + } + dofs += done; + aofs += done; + oprsz -= done; + maxsz -= done; + } + + if (g->fni4 && check_size_impl(oprsz, 4)) { + uint32_t done = QEMU_ALIGN_DOWN(oprsz, 4); + expand_2i_i32(dofs, aofs, done, c, g->load_dest, g->fni4); + dofs += done; + aofs += done; + oprsz -= done; + maxsz -= done; + } + + if (oprsz == 0) { + if (maxsz != 0) { + expand_clr(dofs, maxsz); + } + return; + } + + do_ool: + tcg_gen_gvec_2_ool(dofs, aofs, oprsz, maxsz, 0, g->fno); +} + /* Expand a vector three-operand operation. */ void tcg_gen_gvec_3(uint32_t dofs, uint32_t aofs, uint32_t bofs, uint32_t oprsz, uint32_t maxsz, const GVecGen3 *g) @@ -1024,6 +1166,155 @@ void tcg_gen_gvec_orc(unsigned vece, uint32_t dofs, uint32_t aofs, tcg_gen_gvec_3(dofs, aofs, bofs, opsz, maxsz, &g); } +void tcg_gen_vec_shl8i_i64(TCGv_i64 d, TCGv_i64 a, unsigned c) +{ + uint64_t mask = ((0xff << c) & 0xff) * (-1ull / 0xff); + tcg_gen_shli_i64(d, a, c); + tcg_gen_andi_i64(d, d, mask); +} + +void tcg_gen_vec_shl16i_i64(TCGv_i64 d, TCGv_i64 a, unsigned c) +{ + uint64_t mask = ((0xffff << c) & 0xffff) * (-1ull / 0xffff); + tcg_gen_shli_i64(d, a, c); + tcg_gen_andi_i64(d, d, mask); +} + +void tcg_gen_gvec_shli(unsigned vece, uint32_t dofs, uint32_t aofs, + uint32_t opsz, uint32_t clsz, unsigned shift) +{ + static const GVecGen2i g[4] = { + { .fni8 = tcg_gen_vec_shl8i_i64, + .fniv = tcg_gen_shli_vec, + .fno = gen_helper_gvec_shl8i, + .opc = INDEX_op_shli_vec, + .vece = MO_8 }, + { .fni8 = tcg_gen_vec_shl16i_i64, + .fniv = tcg_gen_shli_vec, + .fno = gen_helper_gvec_shl16i, + .opc = INDEX_op_shli_vec, + .vece = MO_16 }, + { .fni4 = tcg_gen_shli_i32, + .fniv = tcg_gen_shli_vec, + .fno = gen_helper_gvec_shl32i, + .opc = INDEX_op_shli_vec, + .vece = MO_32 }, + { .fni8 = tcg_gen_shli_i64, + .fniv = tcg_gen_shli_vec, + .fno = gen_helper_gvec_shl64i, + .opc = INDEX_op_shli_vec, + .prefer_i64 = TCG_TARGET_REG_BITS == 64, + .vece = MO_64 }, + }; + + tcg_debug_assert(vece <= MO_64); + tcg_gen_gvec_2i(dofs, aofs, opsz, clsz, shift, &g[vece]); +} + +void tcg_gen_vec_shr8i_i64(TCGv_i64 d, TCGv_i64 a, unsigned c) +{ + uint64_t mask = (0xff >> c) * (-1ull / 0xff); + tcg_gen_shri_i64(d, a, c); + tcg_gen_andi_i64(d, d, mask); +} + +void tcg_gen_vec_shr16i_i64(TCGv_i64 d, TCGv_i64 a, unsigned c) +{ + uint64_t mask = (0xffff >> c) * (-1ull / 0xffff); + tcg_gen_shri_i64(d, a, c); + tcg_gen_andi_i64(d, d, mask); +} + +void tcg_gen_gvec_shri(unsigned vece, uint32_t dofs, uint32_t aofs, + uint32_t opsz, uint32_t clsz, unsigned shift) +{ + static const GVecGen2i g[4] = { + { .fni8 = tcg_gen_vec_shr8i_i64, + .fniv = tcg_gen_shri_vec, + .fno = gen_helper_gvec_shr8i, + .opc = INDEX_op_shri_vec, + .vece = MO_8 }, + { .fni8 = tcg_gen_vec_shr16i_i64, + .fniv = tcg_gen_shri_vec, + .fno = gen_helper_gvec_shr16i, + .opc = INDEX_op_shri_vec, + .vece = MO_16 }, + { .fni4 = tcg_gen_shri_i32, + .fniv = tcg_gen_shri_vec, + .fno = gen_helper_gvec_shr32i, + .opc = INDEX_op_shri_vec, + .vece = MO_32 }, + { .fni8 = tcg_gen_shri_i64, + .fniv = tcg_gen_shri_vec, + .fno = gen_helper_gvec_shr64i, + .opc = INDEX_op_shri_vec, + .prefer_i64 = TCG_TARGET_REG_BITS == 64, + .vece = MO_64 }, + }; + + tcg_debug_assert(vece <= MO_64); + tcg_gen_gvec_2i(dofs, aofs, opsz, clsz, shift, &g[vece]); +} + +void tcg_gen_vec_sar8i_i64(TCGv_i64 d, TCGv_i64 a, unsigned c) +{ + uint64_t s_mask = (0x80 >> c) * (-1ull / 0xff); + uint64_t c_mask = (0xff >> c) * (-1ull / 0xff); + TCGv_i64 s = tcg_temp_new_i64(); + + tcg_gen_shri_i64(d, a, c); + tcg_gen_andi_i64(s, d, s_mask); /* isolate (shifted) sign bit */ + tcg_gen_muli_i64(s, s, (2 << c) - 2); /* replicate isolated signs */ + tcg_gen_andi_i64(d, d, c_mask); /* clear out bits above sign */ + tcg_gen_or_i64(d, d, s); /* include sign extension */ + tcg_temp_free_i64(s); +} + +void tcg_gen_vec_sar16i_i64(TCGv_i64 d, TCGv_i64 a, unsigned c) +{ + uint64_t s_mask = (0x8000 >> c) * (-1ull / 0xffff); + uint64_t c_mask = (0xffff >> c) * (-1ull / 0xffff); + TCGv_i64 s = tcg_temp_new_i64(); + + tcg_gen_shri_i64(d, a, c); + tcg_gen_andi_i64(s, d, s_mask); /* isolate (shifted) sign bit */ + tcg_gen_andi_i64(d, d, c_mask); /* clear out bits above sign */ + tcg_gen_muli_i64(s, s, (2 << c) - 2); /* replicate isolated signs */ + tcg_gen_or_i64(d, d, s); /* include sign extension */ + tcg_temp_free_i64(s); +} + +void tcg_gen_gvec_sari(unsigned vece, uint32_t dofs, uint32_t aofs, + uint32_t opsz, uint32_t clsz, unsigned shift) +{ + static const GVecGen2i g[4] = { + { .fni8 = tcg_gen_vec_sar8i_i64, + .fniv = tcg_gen_sari_vec, + .fno = gen_helper_gvec_sar8i, + .opc = INDEX_op_sari_vec, + .vece = MO_8 }, + { .fni8 = tcg_gen_vec_sar16i_i64, + .fniv = tcg_gen_sari_vec, + .fno = gen_helper_gvec_sar16i, + .opc = INDEX_op_sari_vec, + .vece = MO_16 }, + { .fni4 = tcg_gen_sari_i32, + .fniv = tcg_gen_sari_vec, + .fno = gen_helper_gvec_sar32i, + .opc = INDEX_op_sari_vec, + .vece = MO_32 }, + { .fni8 = tcg_gen_sari_i64, + .fniv = tcg_gen_sari_vec, + .fno = gen_helper_gvec_sar64i, + .opc = INDEX_op_sari_vec, + .prefer_i64 = TCG_TARGET_REG_BITS == 64, + .vece = MO_64 }, + }; + + tcg_debug_assert(vece <= MO_64); + tcg_gen_gvec_2i(dofs, aofs, opsz, clsz, shift, &g[vece]); +} + static void do_zip(unsigned vece, uint32_t dofs, uint32_t aofs, uint32_t bofs, uint32_t oprsz, uint32_t maxsz, bool high) diff --git a/tcg/tcg-op-vec.c b/tcg/tcg-op-vec.c index a5d0ff89c3..a441193b8e 100644 --- a/tcg/tcg-op-vec.c +++ b/tcg/tcg-op-vec.c @@ -381,6 +381,46 @@ void tcg_gen_neg_vec(unsigned vece, TCGv_vec r, TCGv_vec a) } } +static void do_shifti(TCGOpcode opc, unsigned vece, + TCGv_vec r, TCGv_vec a, unsigned i) +{ + TCGTemp *rt = tcgv_vec_temp(r); + TCGTemp *at = tcgv_vec_temp(a); + TCGArg ri = temp_arg(rt); + TCGArg ai = temp_arg(at); + TCGType type = rt->base_type; + unsigned vecl = type - TCG_TYPE_V64; + int can; + + tcg_debug_assert(at->base_type == type); + tcg_debug_assert(i < (8 << vece)); + can = tcg_can_emit_vec_op(opc, type, vece); + if (can > 0) { + vec_gen_3(opc, type, vece, ri, ai, i); + } else { + /* We leave the choice of expansion via scalar or vector shift + to the target. Often, but not always, dupi can feed a vector + shift easier than a scalar. */ + tcg_debug_assert(can < 0); + tcg_expand_vec_op(opc, vecl, vece, ri, ai, i); + } +} + +void tcg_gen_shli_vec(unsigned vece, TCGv_vec r, TCGv_vec a, unsigned i) +{ + do_shifti(INDEX_op_shli_vec, vece, r, a, i); +} + +void tcg_gen_shri_vec(unsigned vece, TCGv_vec r, TCGv_vec a, unsigned i) +{ + do_shifti(INDEX_op_shri_vec, vece, r, a, i); +} + +void tcg_gen_sari_vec(unsigned vece, TCGv_vec r, TCGv_vec a, unsigned i) +{ + do_shifti(INDEX_op_sari_vec, vece, r, a, i); +} + static void do_interleave(TCGOpcode opc, unsigned vece, TCGv_vec r, TCGv_vec a, TCGv_vec b) { diff --git a/tcg/tcg.c b/tcg/tcg.c index ec7db4e82d..4bde7d6afd 100644 --- a/tcg/tcg.c +++ b/tcg/tcg.c @@ -1403,6 +1403,18 @@ bool tcg_op_supported(TCGOpcode op) return have_vec && TCG_TARGET_HAS_andc_vec; case INDEX_op_orc_vec: return have_vec && TCG_TARGET_HAS_orc_vec; + case INDEX_op_shli_vec: + case INDEX_op_shri_vec: + case INDEX_op_sari_vec: + return have_vec && TCG_TARGET_HAS_shi_vec; + case INDEX_op_shls_vec: + case INDEX_op_shrs_vec: + case INDEX_op_sars_vec: + return have_vec && TCG_TARGET_HAS_shs_vec; + case INDEX_op_shlv_vec: + case INDEX_op_shrv_vec: + case INDEX_op_sarv_vec: + return have_vec && TCG_TARGET_HAS_shv_vec; case INDEX_op_zipl_vec: case INDEX_op_ziph_vec: return have_vec && TCG_TARGET_HAS_zip_vec; diff --git a/tcg/README b/tcg/README index 8ab8d3ab7e..75db47922d 100644 --- a/tcg/README +++ b/tcg/README @@ -561,6 +561,35 @@ E.g. VECL=1 -> 64 << 1 -> v128, and VECE=2 -> 1 << 2 -> i32. Similarly, logical operations with and without compliment. Note that VECE is unused. +* shli_vec v0, v1, i2 +* shls_vec v0, v1, s2 + + Shift all elements from v1 by a scalar i2/s2. I.e. + + for (i = 0; i < VECL/VECE; ++i) { + v0[i] = v1[i] << s2; + } + +* shri_vec v0, v1, i2 +* sari_vec v0, v1, i2 +* shrs_vec v0, v1, s2 +* sars_vec v0, v1, s2 + + Similarly for logical and arithmetic right shift. + +* shlv_vec v0, v1, v2 + + Shift elements from v1 by elements from v2. I.e. + + for (i = 0; i < VECL/VECE; ++i) { + v0[i] = v1[i] << v2[i]; + } + +* shrv_vec v0, v1, v2 +* sarv_vec v0, v1, v2 + + Similarly for logical and arithmetic right shift. + * zipl_vec v0, v1, v2 * ziph_vec v0, v1, v2 From patchwork Tue Nov 21 21:25:25 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 119427 Delivered-To: patch@linaro.org Received: by 10.140.22.164 with SMTP id 33csp5830906qgn; Tue, 21 Nov 2017 13:50:01 -0800 (PST) X-Google-Smtp-Source: AGs4zMazx6/Cw2XIFzkxkBsJAKpyfb+xdMWhMi+1giRvlNkBurzvunCaDraXK/FRTX+ZxbNYdJAl X-Received: by 10.13.208.6 with SMTP id s6mr12128061ywd.200.1511301001140; Tue, 21 Nov 2017 13:50:01 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1511301001; cv=none; d=google.com; s=arc-20160816; b=ZsqZtEs6aER3AprkpGuzWY65v9FqqYqsNL2qAqhdqRPYFr8pF1xQtQxfg3kLCVkWAs qQnBoUDgh2FginMBSr5PKtMQnWj/Ix5toOPsD1wrwV2plNV57//5pGVB57K01gwmhkmo NlQKMMinE6qnCNekBqqYzLZcwiQzJONvZyNYM/VxthyFq2uFu2Lz4F7Y4C0Doc34FxfH 82AKfXqY5eUX5BWLlTB2IOU7Hc0gLmKUw1VQ39TO+uw+FbWGMCH+C04Cd6IoFPvALMbJ /Qdh5yqqNR43bNGRR9xMrf/pmhqPK9f8YxPbjjwvd1SO4y6/8BA+rMNiGFBCQu1HXJzX VLog== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:subject:references:in-reply-to :message-id:date:to:from:dkim-signature:arc-authentication-results; bh=AUCCcLSiloYJGFdkrS7hcMcQv27xMer+AqIbHA6YACE=; b=Fv/VP2vB1SovOX7G3gj9YcMMVjo66cjRQhX3GNHcTgKUSlYhqfa2L2gnX/Or0S2SW/ 749ygGsW5HcnASp8qP/lZYH8AJUtGyqw6cQcO9T0WBcp/MmVTlTBdmloJsDNv2qa+NBx nTbfHiMnKa1uWBmQULeOpKHO7acx9cAcWLEuLCk/PCthRR97fahUh5Ugf0FOaYDura19 GpTpWzWNZckKsIKfF6BsL8fP+bvQcEonx0yrZd9CGlImflOHvGEZd9D4UbIKNi1apzUj AaD8VOfpdbPqpb+tLeBMh1Ft9Gzp5WHodDciXZYUl1xYZsWmN5/RKiL7qoHPsBgfs7B3 AK5w== ARC-Authentication-Results: i=1; mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=YzMcrust; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) smtp.mailfrom=qemu-devel-bounces+patch=linaro.org@nongnu.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [2001:4830:134:3::11]) by mx.google.com with ESMTPS id h188si1491744ywh.755.2017.11.21.13.50.01 for (version=TLS1 cipher=AES128-SHA bits=128/128); Tue, 21 Nov 2017 13:50:01 -0800 (PST) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) client-ip=2001:4830:134:3::11; Authentication-Results: mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=YzMcrust; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) smtp.mailfrom=qemu-devel-bounces+patch=linaro.org@nongnu.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1]:36700 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1eHGQa-00078e-J0 for patch@linaro.org; Tue, 21 Nov 2017 16:50:00 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:54136) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1eHG5J-0004Z5-PC for qemu-devel@nongnu.org; Tue, 21 Nov 2017 16:28:03 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1eHG5H-0007mK-Sp for qemu-devel@nongnu.org; Tue, 21 Nov 2017 16:28:01 -0500 Received: from mail-wr0-x244.google.com ([2a00:1450:400c:c0c::244]:43771) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1eHG5H-0007lw-IW for qemu-devel@nongnu.org; Tue, 21 Nov 2017 16:27:59 -0500 Received: by mail-wr0-x244.google.com with SMTP id u40so12593377wrf.10 for ; Tue, 21 Nov 2017 13:27:59 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:subject:date:message-id:in-reply-to:references; bh=AUCCcLSiloYJGFdkrS7hcMcQv27xMer+AqIbHA6YACE=; b=YzMcrustL45KlU3E9oItFWKQ2/MZZ7p7deWdZP8s6qfd1raO7+tvMU9S7Fn5FcAO92 c7/cNJYlBBimcpzseXAoul+4j8RnOfJlDS10v9JVuRfhWxuwk2hmcsXFzk7HKwf3jXwo hNnE9EWsW7l2qXbRaGVWenBI7NtRkVxBwS55g= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references; bh=AUCCcLSiloYJGFdkrS7hcMcQv27xMer+AqIbHA6YACE=; b=Od9C8VII7u2dQsPHFZo6k1HKuUgYtE0O9DRP+1DaRz//z8BpjSwr5ThqqujxjRG3G4 5m75huFYD1R0B/FmeIIBff5LLhy9KFcFAQCWLj8lTHQEqtP9zkKlOueqvb8XkJLIFTaK pdWaMuJSW+fBPOlymImaeITzahs+KO97upmKMec0WxzeojsNwJskJF4ZDgaq9qnsZ9qN gYAJJc4RS8nGsSrrVGjdJ//3LT1Q/Gf8j+CRo/OOmKB3ieY9L2a3YP96Ua6JlbxbXSCk 4xYnKH/ek6lcgXiFV2eziT7fbON2ZsFwuSNphUizmmxZ02U+SNz7CsHhYTVo73DNAKb9 HDfw== X-Gm-Message-State: AJaThX5znGovJmcNouHsGO3Xz25SVzC4jrrKQm0wmzYSYV1ZARwKDLRC 04xzFptQ/yXlfYo4dH4WB7Hq2aWBKiY= X-Received: by 10.223.184.122 with SMTP id u55mr17394083wrf.61.1511299678158; Tue, 21 Nov 2017 13:27:58 -0800 (PST) Received: from cloudburst.twiddle.net (70.red-37-158-60.dynamicip.rima-tde.net. [37.158.60.70]) by smtp.gmail.com with ESMTPSA id e124sm706517wmg.34.2017.11.21.13.27.56 for (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Tue, 21 Nov 2017 13:27:57 -0800 (PST) From: Richard Henderson To: qemu-devel@nongnu.org Date: Tue, 21 Nov 2017 22:25:25 +0100 Message-Id: <20171121212534.5177-18-richard.henderson@linaro.org> X-Mailer: git-send-email 2.13.6 In-Reply-To: <20171121212534.5177-1-richard.henderson@linaro.org> References: <20171121212534.5177-1-richard.henderson@linaro.org> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2a00:1450:400c:c0c::244 Subject: [Qemu-devel] [PATCH v6 17/26] target/arm: Use vector infrastructure for aa64 constant shifts X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: "Qemu-devel" Signed-off-by: Richard Henderson --- target/arm/translate-a64.c | 386 ++++++++++++++++++++++++++++++++++++++------- tcg/tcg-op-gvec.c | 18 ++- tcg/tcg-op-vec.c | 9 +- 3 files changed, 351 insertions(+), 62 deletions(-) -- 2.13.6 diff --git a/target/arm/translate-a64.c b/target/arm/translate-a64.c index 8769b4505a..c47faa5633 100644 --- a/target/arm/translate-a64.c +++ b/target/arm/translate-a64.c @@ -6432,17 +6432,6 @@ static void handle_shri_with_rndacc(TCGv_i64 tcg_res, TCGv_i64 tcg_src, } } -/* Common SHL/SLI - Shift left with an optional insert */ -static void handle_shli_with_ins(TCGv_i64 tcg_res, TCGv_i64 tcg_src, - bool insert, int shift) -{ - if (insert) { /* SLI */ - tcg_gen_deposit_i64(tcg_res, tcg_res, tcg_src, shift, 64 - shift); - } else { /* SHL */ - tcg_gen_shli_i64(tcg_res, tcg_src, shift); - } -} - /* SRI: shift right with insert */ static void handle_shri_with_ins(TCGv_i64 tcg_res, TCGv_i64 tcg_src, int size, int shift) @@ -6546,7 +6535,11 @@ static void handle_scalar_simd_shli(DisasContext *s, bool insert, tcg_rn = read_fp_dreg(s, rn); tcg_rd = insert ? read_fp_dreg(s, rd) : tcg_temp_new_i64(); - handle_shli_with_ins(tcg_rd, tcg_rn, insert, shift); + if (insert) { + tcg_gen_deposit_i64(tcg_rd, tcg_rd, tcg_rn, shift, 64 - shift); + } else { + tcg_gen_shli_i64(tcg_rd, tcg_rn, shift); + } write_fp_dreg(s, rd, tcg_rd); @@ -8283,16 +8276,195 @@ static void disas_simd_scalar_two_reg_misc(DisasContext *s, uint32_t insn) } } +static void gen_ssra8_i64(TCGv_i64 d, TCGv_i64 a, unsigned shift) +{ + tcg_gen_vec_sar8i_i64(a, a, shift); + tcg_gen_vec_add8_i64(d, d, a); +} + +static void gen_ssra16_i64(TCGv_i64 d, TCGv_i64 a, unsigned shift) +{ + tcg_gen_vec_sar16i_i64(a, a, shift); + tcg_gen_vec_add16_i64(d, d, a); +} + +static void gen_ssra32_i32(TCGv_i32 d, TCGv_i32 a, unsigned shift) +{ + tcg_gen_sari_i32(a, a, shift); + tcg_gen_add_i32(d, d, a); +} + +static void gen_ssra64_i64(TCGv_i64 d, TCGv_i64 a, unsigned shift) +{ + tcg_gen_sari_i64(a, a, shift); + tcg_gen_add_i64(d, d, a); +} + +static void gen_ssra_vec(unsigned vece, TCGv_vec d, TCGv_vec a, unsigned sh) +{ + tcg_gen_sari_vec(vece, a, a, sh); + tcg_gen_add_vec(vece, d, d, a); +} + +static void gen_usra8_i64(TCGv_i64 d, TCGv_i64 a, unsigned shift) +{ + tcg_gen_vec_shr8i_i64(a, a, shift); + tcg_gen_vec_add8_i64(d, d, a); +} + +static void gen_usra16_i64(TCGv_i64 d, TCGv_i64 a, unsigned shift) +{ + tcg_gen_vec_shr16i_i64(a, a, shift); + tcg_gen_vec_add16_i64(d, d, a); +} + +static void gen_usra32_i32(TCGv_i32 d, TCGv_i32 a, unsigned shift) +{ + tcg_gen_shri_i32(a, a, shift); + tcg_gen_add_i32(d, d, a); +} + +static void gen_usra64_i64(TCGv_i64 d, TCGv_i64 a, unsigned shift) +{ + tcg_gen_shri_i64(a, a, shift); + tcg_gen_add_i64(d, d, a); +} + +static void gen_usra_vec(unsigned vece, TCGv_vec d, TCGv_vec a, unsigned sh) +{ + tcg_gen_shri_vec(vece, a, a, sh); + tcg_gen_add_vec(vece, d, d, a); +} + +static void gen_shr8_ins_i64(TCGv_i64 d, TCGv_i64 a, unsigned shift) +{ + uint64_t mask = (0xff >> shift) * (-1ull / 0xff); + TCGv_i64 t = tcg_temp_new_i64(); + + tcg_gen_shri_i64(t, a, shift); + tcg_gen_andi_i64(t, t, mask); + tcg_gen_andi_i64(d, d, ~mask); + tcg_gen_or_i64(d, d, t); + tcg_temp_free_i64(t); +} + +static void gen_shr16_ins_i64(TCGv_i64 d, TCGv_i64 a, unsigned shift) +{ + uint64_t mask = (0xffff >> shift) * (-1ull / 0xffff); + TCGv_i64 t = tcg_temp_new_i64(); + + tcg_gen_shri_i64(t, a, shift); + tcg_gen_andi_i64(t, t, mask); + tcg_gen_andi_i64(d, d, ~mask); + tcg_gen_or_i64(d, d, t); + tcg_temp_free_i64(t); +} + +static void gen_shr32_ins_i32(TCGv_i32 d, TCGv_i32 a, unsigned shift) +{ + tcg_gen_shri_i32(a, a, shift); + tcg_gen_deposit_i32(d, d, a, 0, 32 - shift); +} + +static void gen_shr64_ins_i64(TCGv_i64 d, TCGv_i64 a, unsigned shift) +{ + tcg_gen_shri_i64(a, a, shift); + tcg_gen_deposit_i64(d, d, a, 0, 64 - shift); +} + +static void gen_shr_ins_vec(unsigned vece, TCGv_vec d, TCGv_vec a, unsigned sh) +{ + uint64_t mask = (2ull << ((8 << vece) - 1)) - 1; + TCGv_vec t = tcg_temp_new_vec_matching(d); + TCGv_vec m = tcg_temp_new_vec_matching(d); + + tcg_gen_dupi_vec(vece, m, mask ^ (mask >> sh)); + tcg_gen_shri_vec(vece, t, a, sh); + tcg_gen_and_vec(vece, d, d, m); + tcg_gen_or_vec(vece, d, d, t); + + tcg_temp_free_vec(t); + tcg_temp_free_vec(m); +} + /* SSHR[RA]/USHR[RA] - Vector shift right (optional rounding/accumulate) */ static void handle_vec_simd_shri(DisasContext *s, bool is_q, bool is_u, int immh, int immb, int opcode, int rn, int rd) { + static const GVecGen2i ssra_op[4] = { + { .fni8 = gen_ssra8_i64, + .fniv = gen_ssra_vec, + .load_dest = true, + .opc = INDEX_op_sari_vec, + .vece = MO_8 }, + { .fni8 = gen_ssra16_i64, + .fniv = gen_ssra_vec, + .load_dest = true, + .opc = INDEX_op_sari_vec, + .vece = MO_16 }, + { .fni4 = gen_ssra32_i32, + .fniv = gen_ssra_vec, + .load_dest = true, + .opc = INDEX_op_sari_vec, + .vece = MO_32 }, + { .fni8 = gen_ssra64_i64, + .fniv = gen_ssra_vec, + .prefer_i64 = TCG_TARGET_REG_BITS == 64, + .load_dest = true, + .opc = INDEX_op_sari_vec, + .vece = MO_64 }, + }; + static const GVecGen2i usra_op[4] = { + { .fni8 = gen_usra8_i64, + .fniv = gen_usra_vec, + .load_dest = true, + .opc = INDEX_op_shri_vec, + .vece = MO_8, }, + { .fni8 = gen_usra16_i64, + .fniv = gen_usra_vec, + .load_dest = true, + .opc = INDEX_op_shri_vec, + .vece = MO_16, }, + { .fni4 = gen_usra32_i32, + .fniv = gen_usra_vec, + .load_dest = true, + .opc = INDEX_op_shri_vec, + .vece = MO_32, }, + { .fni8 = gen_usra64_i64, + .fniv = gen_usra_vec, + .prefer_i64 = TCG_TARGET_REG_BITS == 64, + .load_dest = true, + .opc = INDEX_op_shri_vec, + .vece = MO_64, }, + }; + static const GVecGen2i sri_op[4] = { + { .fni8 = gen_shr8_ins_i64, + .fniv = gen_shr_ins_vec, + .load_dest = true, + .opc = INDEX_op_shri_vec, + .vece = MO_8 }, + { .fni8 = gen_shr16_ins_i64, + .fniv = gen_shr_ins_vec, + .load_dest = true, + .opc = INDEX_op_shri_vec, + .vece = MO_16 }, + { .fni4 = gen_shr32_ins_i32, + .fniv = gen_shr_ins_vec, + .load_dest = true, + .opc = INDEX_op_shri_vec, + .vece = MO_32 }, + { .fni8 = gen_shr64_ins_i64, + .fniv = gen_shr_ins_vec, + .prefer_i64 = TCG_TARGET_REG_BITS == 64, + .load_dest = true, + .opc = INDEX_op_shri_vec, + .vece = MO_64 }, + }; + int size = 32 - clz32(immh) - 1; int immhb = immh << 3 | immb; int shift = 2 * (8 << size) - immhb; bool accumulate = false; - bool round = false; - bool insert = false; int dsize = is_q ? 128 : 64; int esize = 8 << size; int elements = dsize/esize; @@ -8300,6 +8472,8 @@ static void handle_vec_simd_shri(DisasContext *s, bool is_q, bool is_u, TCGv_i64 tcg_rn = new_tmp_a64(s); TCGv_i64 tcg_rd = new_tmp_a64(s); TCGv_i64 tcg_round; + uint64_t round_const; + const GVecGen2i *gvec_op; int i; if (extract32(immh, 3, 1) && !is_q) { @@ -8318,64 +8492,141 @@ static void handle_vec_simd_shri(DisasContext *s, bool is_q, bool is_u, switch (opcode) { case 0x02: /* SSRA / USRA (accumulate) */ - accumulate = true; - break; + if (is_u) { + /* Shift count same as element size produces zero to add. */ + if (shift == 8 << size) { + goto done; + } + gvec_op = &usra_op[size]; + } else { + /* Shift count same as element size produces all sign to add. */ + if (shift == 8 << size) { + shift -= 1; + } + gvec_op = &ssra_op[size]; + } + goto do_gvec; + case 0x08: /* SRI */ + /* Shift count same as element size is valid but does nothing. */ + if (shift == 8 << size) { + goto done; + } + gvec_op = &sri_op[size]; + do_gvec: + tcg_gen_gvec_2i(vec_full_reg_offset(s, rd), + vec_full_reg_offset(s, rn), is_q ? 16 : 8, + vec_full_reg_size(s), shift, gvec_op); + return; + + case 0x00: /* SSHR / USHR */ + if (is_u) { + if (shift == 8 << size) { + /* Shift count the same size as element size produces zero. */ + tcg_gen_gvec_dup8i(vec_full_reg_offset(s, rd), + is_q ? 16 : 8, vec_full_reg_size(s), 0); + } else { + tcg_gen_gvec_shri(size, vec_full_reg_offset(s, rd), + vec_full_reg_offset(s, rn), is_q ? 16 : 8, + vec_full_reg_size(s), shift); + } + } else { + /* Shift count the same size as element size produces all sign. */ + if (shift == 8 << size) { + shift -= 1; + } + tcg_gen_gvec_sari(size, vec_full_reg_offset(s, rd), + vec_full_reg_offset(s, rn), is_q ? 16 : 8, + vec_full_reg_size(s), shift); + } + return; + case 0x04: /* SRSHR / URSHR (rounding) */ - round = true; break; case 0x06: /* SRSRA / URSRA (accum + rounding) */ - accumulate = round = true; - break; - case 0x08: /* SRI */ - insert = true; + accumulate = true; break; + default: + g_assert_not_reached(); } - if (round) { - uint64_t round_const = 1ULL << (shift - 1); - tcg_round = tcg_const_i64(round_const); - } else { - tcg_round = NULL; - } + round_const = 1ULL << (shift - 1); + tcg_round = tcg_const_i64(round_const); for (i = 0; i < elements; i++) { read_vec_element(s, tcg_rn, rn, i, memop); - if (accumulate || insert) { + if (accumulate) { read_vec_element(s, tcg_rd, rd, i, memop); } - if (insert) { - handle_shri_with_ins(tcg_rd, tcg_rn, size, shift); - } else { - handle_shri_with_rndacc(tcg_rd, tcg_rn, tcg_round, - accumulate, is_u, size, shift); - } + handle_shri_with_rndacc(tcg_rd, tcg_rn, tcg_round, + accumulate, is_u, size, shift); write_vec_element(s, tcg_rd, rd, i, size); } + tcg_temp_free_i64(tcg_round); + done: if (!is_q) { clear_vec_high(s, rd); } +} - if (round) { - tcg_temp_free_i64(tcg_round); - } +static void gen_shl8_ins_i64(TCGv_i64 d, TCGv_i64 a, unsigned shift) +{ + uint64_t mask = ((0xff << shift) & 0xff) * (-1ull / 0xff); + TCGv_i64 t = tcg_temp_new_i64(); + + tcg_gen_shli_i64(t, a, shift); + tcg_gen_andi_i64(t, t, mask); + tcg_gen_andi_i64(d, d, ~mask); + tcg_gen_or_i64(d, d, t); + tcg_temp_free_i64(t); +} + +static void gen_shl16_ins_i64(TCGv_i64 d, TCGv_i64 a, unsigned shift) +{ + uint64_t mask = ((0xffff << shift) & 0xffff) * (-1ull / 0xffff); + TCGv_i64 t = tcg_temp_new_i64(); + + tcg_gen_shli_i64(t, a, shift); + tcg_gen_andi_i64(t, t, mask); + tcg_gen_andi_i64(d, d, ~mask); + tcg_gen_or_i64(d, d, t); + tcg_temp_free_i64(t); +} + +static void gen_shl32_ins_i32(TCGv_i32 d, TCGv_i32 a, unsigned shift) +{ + tcg_gen_deposit_i32(d, d, a, shift, 32 - shift); +} + +static void gen_shl64_ins_i64(TCGv_i64 d, TCGv_i64 a, unsigned shift) +{ + tcg_gen_deposit_i64(d, d, a, shift, 64 - shift); +} + +static void gen_shl_ins_vec(unsigned vece, TCGv_vec d, TCGv_vec a, unsigned sh) +{ + uint64_t mask = (1ull << sh) - 1; + TCGv_vec t = tcg_temp_new_vec_matching(d); + TCGv_vec m = tcg_temp_new_vec_matching(d); + + tcg_gen_dupi_vec(vece, m, mask); + tcg_gen_shli_vec(vece, t, a, sh); + tcg_gen_and_vec(vece, d, d, m); + tcg_gen_or_vec(vece, d, d, t); + + tcg_temp_free_vec(t); + tcg_temp_free_vec(m); } /* SHL/SLI - Vector shift left */ static void handle_vec_simd_shli(DisasContext *s, bool is_q, bool insert, - int immh, int immb, int opcode, int rn, int rd) + int immh, int immb, int opcode, int rn, int rd) { int size = 32 - clz32(immh) - 1; int immhb = immh << 3 | immb; int shift = immhb - (8 << size); - int dsize = is_q ? 128 : 64; - int esize = 8 << size; - int elements = dsize/esize; - TCGv_i64 tcg_rn = new_tmp_a64(s); - TCGv_i64 tcg_rd = new_tmp_a64(s); - int i; if (extract32(immh, 3, 1) && !is_q) { unallocated_encoding(s); @@ -8391,19 +8642,40 @@ static void handle_vec_simd_shli(DisasContext *s, bool is_q, bool insert, return; } - for (i = 0; i < elements; i++) { - read_vec_element(s, tcg_rn, rn, i, size); - if (insert) { - read_vec_element(s, tcg_rd, rd, i, size); - } - - handle_shli_with_ins(tcg_rd, tcg_rn, insert, shift); - - write_vec_element(s, tcg_rd, rd, i, size); - } - - if (!is_q) { - clear_vec_high(s, rd); + if (insert) { + static const GVecGen2i shi_op[4] = { + { .fni8 = gen_shl8_ins_i64, + .fniv = gen_shl_ins_vec, + .opc = INDEX_op_shli_vec, + .prefer_i64 = TCG_TARGET_REG_BITS == 64, + .load_dest = true, + .vece = MO_8 }, + { .fni8 = gen_shl16_ins_i64, + .fniv = gen_shl_ins_vec, + .opc = INDEX_op_shli_vec, + .prefer_i64 = TCG_TARGET_REG_BITS == 64, + .load_dest = true, + .vece = MO_16 }, + { .fni4 = gen_shl32_ins_i32, + .fniv = gen_shl_ins_vec, + .opc = INDEX_op_shli_vec, + .prefer_i64 = TCG_TARGET_REG_BITS == 64, + .load_dest = true, + .vece = MO_32 }, + { .fni8 = gen_shl64_ins_i64, + .fniv = gen_shl_ins_vec, + .opc = INDEX_op_shli_vec, + .prefer_i64 = TCG_TARGET_REG_BITS == 64, + .load_dest = true, + .vece = MO_64 }, + }; + tcg_gen_gvec_2i(vec_full_reg_offset(s, rd), + vec_full_reg_offset(s, rn), is_q ? 16 : 8, + vec_full_reg_size(s), shift, &shi_op[size]); + } else { + tcg_gen_gvec_shli(size, vec_full_reg_offset(s, rd), + vec_full_reg_offset(s, rn), is_q ? 16 : 8, + vec_full_reg_size(s), shift); } } diff --git a/tcg/tcg-op-gvec.c b/tcg/tcg-op-gvec.c index f8ccb137eb..d91b424dfc 100644 --- a/tcg/tcg-op-gvec.c +++ b/tcg/tcg-op-gvec.c @@ -1208,7 +1208,11 @@ void tcg_gen_gvec_shli(unsigned vece, uint32_t dofs, uint32_t aofs, }; tcg_debug_assert(vece <= MO_64); - tcg_gen_gvec_2i(dofs, aofs, opsz, clsz, shift, &g[vece]); + if (shift == 0) { + tcg_gen_gvec_mov(vece, dofs, aofs, opsz, clsz); + } else { + tcg_gen_gvec_2i(dofs, aofs, opsz, clsz, shift, &g[vece]); + } } void tcg_gen_vec_shr8i_i64(TCGv_i64 d, TCGv_i64 a, unsigned c) @@ -1253,7 +1257,11 @@ void tcg_gen_gvec_shri(unsigned vece, uint32_t dofs, uint32_t aofs, }; tcg_debug_assert(vece <= MO_64); - tcg_gen_gvec_2i(dofs, aofs, opsz, clsz, shift, &g[vece]); + if (shift == 0) { + tcg_gen_gvec_mov(vece, dofs, aofs, opsz, clsz); + } else { + tcg_gen_gvec_2i(dofs, aofs, opsz, clsz, shift, &g[vece]); + } } void tcg_gen_vec_sar8i_i64(TCGv_i64 d, TCGv_i64 a, unsigned c) @@ -1312,7 +1320,11 @@ void tcg_gen_gvec_sari(unsigned vece, uint32_t dofs, uint32_t aofs, }; tcg_debug_assert(vece <= MO_64); - tcg_gen_gvec_2i(dofs, aofs, opsz, clsz, shift, &g[vece]); + if (shift == 0) { + tcg_gen_gvec_mov(vece, dofs, aofs, opsz, clsz); + } else { + tcg_gen_gvec_2i(dofs, aofs, opsz, clsz, shift, &g[vece]); + } } static void do_zip(unsigned vece, uint32_t dofs, uint32_t aofs, diff --git a/tcg/tcg-op-vec.c b/tcg/tcg-op-vec.c index a441193b8e..502c5ba891 100644 --- a/tcg/tcg-op-vec.c +++ b/tcg/tcg-op-vec.c @@ -389,11 +389,16 @@ static void do_shifti(TCGOpcode opc, unsigned vece, TCGArg ri = temp_arg(rt); TCGArg ai = temp_arg(at); TCGType type = rt->base_type; - unsigned vecl = type - TCG_TYPE_V64; int can; tcg_debug_assert(at->base_type == type); tcg_debug_assert(i < (8 << vece)); + + if (i == 0) { + tcg_gen_mov_vec(r, a); + return; + } + can = tcg_can_emit_vec_op(opc, type, vece); if (can > 0) { vec_gen_3(opc, type, vece, ri, ai, i); @@ -402,7 +407,7 @@ static void do_shifti(TCGOpcode opc, unsigned vece, to the target. Often, but not always, dupi can feed a vector shift easier than a scalar. */ tcg_debug_assert(can < 0); - tcg_expand_vec_op(opc, vecl, vece, ri, ai, i); + tcg_expand_vec_op(opc, type, vece, ri, ai, i); } } From patchwork Tue Nov 21 21:25:26 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 119428 Delivered-To: patch@linaro.org Received: by 10.140.22.164 with SMTP id 33csp5831328qgn; Tue, 21 Nov 2017 13:50:31 -0800 (PST) X-Google-Smtp-Source: AGs4zMZHsBpBP0SKmBuQfwMRTZOrHCKx3OyuXzC/YwxuSWCoFEaPmvcMIXR436nvcoJTcDynvPtj X-Received: by 10.129.233.9 with SMTP id d9mr3680352ywm.367.1511301031601; Tue, 21 Nov 2017 13:50:31 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1511301031; cv=none; d=google.com; s=arc-20160816; b=xYEydpsFnNWyH9AtoK7ul/T/T9XeeqRwe924iOmr8Qk/femrZ6hbMMCCPz1Fv6Yy5V L6FC49Y356Nj7g+TwwvyoenqDrN5kmhECxHHAPhhD6/yRL14cMDopOO1EQuIPpnr2uIp jtgrGw5etW0X7wPypMCfHrpVu+A8g+nY5TIcg6AobIU2ieFakIhMyf17PBayeaFgoasz W+iUHk6PLOAKYeV8G8iomaN8pZfYYtgGZGIkqu7PU6gXOJNyjSZhoCEFNrGUH9ctRqDF Yd2zOURs6+x0bPrYkd7Pk6P99lchovxKzj/WCcXh6OoQcLIX5GelhX0DUXO2bzqvz90x 2Ekw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:subject:references:in-reply-to :message-id:date:to:from:dkim-signature:arc-authentication-results; bh=DkO82O/ongsVIKe7ZTZZq9W/QXhL7OlFxH7i0vdfqUg=; b=Ai/LLiiEjHyLVoUzcyDCqfT1wszV73p/6Nj0NrZ65B96MogXi2OaeZ166wnDcVxwMk Mv7iLyixuSYPrMqsj62XayXcqpSrlPbrxohPl6QyK5Jgfn0SN6h00lXEdydo4H5E/Sqo WKJPL6Jw88ETHi2zmbhRSLH/bq0KblG2D9s2JAtDYFGqBAlp5qnTSOjFOCqBjhXGKMqm zYpXGV0f83MMoNuOUvXGFGrVl2R+fXhU6JncUHxg7WfeaXPffkHHNZzv1rIPLN8NfAmi ADqpghfsqqPZ2FMQ+HqqsPsEkS6AKOj1OhZG+5zfF4nu5XaholZWPgxPtR6RS4z4Egpw R93w== ARC-Authentication-Results: i=1; mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=ByHNU/BK; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) smtp.mailfrom=qemu-devel-bounces+patch=linaro.org@nongnu.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [2001:4830:134:3::11]) by mx.google.com with ESMTPS id w1si2923733yba.571.2017.11.21.13.50.31 for (version=TLS1 cipher=AES128-SHA bits=128/128); Tue, 21 Nov 2017 13:50:31 -0800 (PST) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) client-ip=2001:4830:134:3::11; Authentication-Results: mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=ByHNU/BK; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) smtp.mailfrom=qemu-devel-bounces+patch=linaro.org@nongnu.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1]:36715 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1eHGR5-0008GL-4A for patch@linaro.org; Tue, 21 Nov 2017 16:50:31 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:54155) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1eHG5L-0004ax-Lr for qemu-devel@nongnu.org; Tue, 21 Nov 2017 16:28:05 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1eHG5J-0007nc-DH for qemu-devel@nongnu.org; Tue, 21 Nov 2017 16:28:03 -0500 Received: from mail-wr0-x241.google.com ([2a00:1450:400c:c0c::241]:42951) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1eHG5I-0007mS-Vi for qemu-devel@nongnu.org; Tue, 21 Nov 2017 16:28:01 -0500 Received: by mail-wr0-x241.google.com with SMTP id o14so12613525wrf.9 for ; Tue, 21 Nov 2017 13:28:00 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:subject:date:message-id:in-reply-to:references; bh=DkO82O/ongsVIKe7ZTZZq9W/QXhL7OlFxH7i0vdfqUg=; b=ByHNU/BKbja8rbErwRaczsRsbipHUZF9tH3VYzNwxwJ3e8L1uc8CnjexMSSjuzFIby P7gKbQjBEXhtugfZBxzHjodUa1wc0k63Puyg/fJ8F5x6eW3P17tqVti4YN6CDtl4N1wP ylr9Ph2nx/f7/ObW/lV+gdvi8dEa6BeQ57Ibo= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references; bh=DkO82O/ongsVIKe7ZTZZq9W/QXhL7OlFxH7i0vdfqUg=; b=cJ3/cbpWWtaMztUjYBfOFroXq6iM/OYraHRDwspdEVhprdaIm8iHuBaWlqWBMFZCk5 Kt5Vmgp8q8SuBh/caAeZChBHEwII2RZb9LpSDCb5CCuaTm6gh6fgl90ImcVqbrn+XpC7 rNj5ihyUS7f1MpUyGUxthD6R5pFOnMhuIRKFepKCeZgadjFCbUiGfrjomD7TZDpY4nj9 h7ohKenm982q8p7wHqUPNrXTOkviXjVnc3LV0XEyfTdM17XlDC6rxoarrLCEeV1lt1LT opRmE2q6IuxEHcGmaMZltg4NnUOKlSIVhDl87q5o0O5y0oV6HxPVm1LIRXToxjSIQN6V lNqA== X-Gm-Message-State: AJaThX7tPN0szhxp9hCdVGG50r0P9Uye75NzRFK18hyPW6XEEpKOHN5S MlPCgdH80YNxBljIaCtLYUeRLRgRNnM= X-Received: by 10.223.197.141 with SMTP id m13mr16572424wrg.203.1511299679512; Tue, 21 Nov 2017 13:27:59 -0800 (PST) Received: from cloudburst.twiddle.net (70.red-37-158-60.dynamicip.rima-tde.net. [37.158.60.70]) by smtp.gmail.com with ESMTPSA id e124sm706517wmg.34.2017.11.21.13.27.58 for (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Tue, 21 Nov 2017 13:27:58 -0800 (PST) From: Richard Henderson To: qemu-devel@nongnu.org Date: Tue, 21 Nov 2017 22:25:26 +0100 Message-Id: <20171121212534.5177-19-richard.henderson@linaro.org> X-Mailer: git-send-email 2.13.6 In-Reply-To: <20171121212534.5177-1-richard.henderson@linaro.org> References: <20171121212534.5177-1-richard.henderson@linaro.org> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2a00:1450:400c:c0c::241 Subject: [Qemu-devel] [PATCH v6 18/26] tcg: Add generic vector ops for comparisons X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: "Qemu-devel" Signed-off-by: Richard Henderson --- accel/tcg/tcg-runtime.h | 30 +++++++ tcg/i386/tcg-target.h | 1 + tcg/tcg-op-gvec.h | 4 + tcg/tcg-op.h | 3 + tcg/tcg-opc.h | 2 + tcg/tcg.h | 1 + accel/tcg/tcg-runtime-gvec.c | 24 +++++ tcg/tcg-op-gvec.c | 202 +++++++++++++++++++++++++++++++++++++++++++ tcg/tcg-op-vec.c | 23 +++++ tcg/tcg.c | 2 + tcg/README | 4 + 11 files changed, 296 insertions(+) -- 2.13.6 diff --git a/accel/tcg/tcg-runtime.h b/accel/tcg/tcg-runtime.h index cb05a755b8..28abf30d76 100644 --- a/accel/tcg/tcg-runtime.h +++ b/accel/tcg/tcg-runtime.h @@ -193,3 +193,33 @@ DEF_HELPER_FLAGS_4(gvec_trn8, TCG_CALL_NO_RWG, void, ptr, ptr, ptr, i32) DEF_HELPER_FLAGS_4(gvec_trn16, TCG_CALL_NO_RWG, void, ptr, ptr, ptr, i32) DEF_HELPER_FLAGS_4(gvec_trn32, TCG_CALL_NO_RWG, void, ptr, ptr, ptr, i32) DEF_HELPER_FLAGS_4(gvec_trn64, TCG_CALL_NO_RWG, void, ptr, ptr, ptr, i32) + +DEF_HELPER_FLAGS_4(gvec_eq8, TCG_CALL_NO_RWG, void, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_4(gvec_eq16, TCG_CALL_NO_RWG, void, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_4(gvec_eq32, TCG_CALL_NO_RWG, void, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_4(gvec_eq64, TCG_CALL_NO_RWG, void, ptr, ptr, ptr, i32) + +DEF_HELPER_FLAGS_4(gvec_ne8, TCG_CALL_NO_RWG, void, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_4(gvec_ne16, TCG_CALL_NO_RWG, void, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_4(gvec_ne32, TCG_CALL_NO_RWG, void, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_4(gvec_ne64, TCG_CALL_NO_RWG, void, ptr, ptr, ptr, i32) + +DEF_HELPER_FLAGS_4(gvec_lt8, TCG_CALL_NO_RWG, void, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_4(gvec_lt16, TCG_CALL_NO_RWG, void, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_4(gvec_lt32, TCG_CALL_NO_RWG, void, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_4(gvec_lt64, TCG_CALL_NO_RWG, void, ptr, ptr, ptr, i32) + +DEF_HELPER_FLAGS_4(gvec_le8, TCG_CALL_NO_RWG, void, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_4(gvec_le16, TCG_CALL_NO_RWG, void, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_4(gvec_le32, TCG_CALL_NO_RWG, void, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_4(gvec_le64, TCG_CALL_NO_RWG, void, ptr, ptr, ptr, i32) + +DEF_HELPER_FLAGS_4(gvec_ltu8, TCG_CALL_NO_RWG, void, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_4(gvec_ltu16, TCG_CALL_NO_RWG, void, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_4(gvec_ltu32, TCG_CALL_NO_RWG, void, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_4(gvec_ltu64, TCG_CALL_NO_RWG, void, ptr, ptr, ptr, i32) + +DEF_HELPER_FLAGS_4(gvec_leu8, TCG_CALL_NO_RWG, void, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_4(gvec_leu16, TCG_CALL_NO_RWG, void, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_4(gvec_leu32, TCG_CALL_NO_RWG, void, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_4(gvec_leu64, TCG_CALL_NO_RWG, void, ptr, ptr, ptr, i32) diff --git a/tcg/i386/tcg-target.h b/tcg/i386/tcg-target.h index 92d533eb92..46c4dca7be 100644 --- a/tcg/i386/tcg-target.h +++ b/tcg/i386/tcg-target.h @@ -183,6 +183,7 @@ extern bool have_avx2; #define TCG_TARGET_HAS_zip_vec 0 #define TCG_TARGET_HAS_uzp_vec 0 #define TCG_TARGET_HAS_trn_vec 0 +#define TCG_TARGET_HAS_cmp_vec 0 #define TCG_TARGET_deposit_i32_valid(ofs, len) \ (((ofs) == 0 && (len) == 8) || ((ofs) == 8 && (len) == 8) || \ diff --git a/tcg/tcg-op-gvec.h b/tcg/tcg-op-gvec.h index de2c0e669a..308bdc13b4 100644 --- a/tcg/tcg-op-gvec.h +++ b/tcg/tcg-op-gvec.h @@ -178,6 +178,10 @@ void tcg_gen_gvec_trne(unsigned vece, uint32_t dofs, uint32_t aofs, void tcg_gen_gvec_trno(unsigned vece, uint32_t dofs, uint32_t aofs, uint32_t bofs, uint32_t opsz, uint32_t clsz); +void tcg_gen_gvec_cmp(TCGCond cond, unsigned vece, uint32_t dofs, + uint32_t aofs, uint32_t bofs, + uint32_t opsz, uint32_t clsz); + /* * 64-bit vector operations. Use these when the register has been allocated * with tcg_global_mem_new_i64, and so we cannot also address it via pointer. diff --git a/tcg/tcg-op.h b/tcg/tcg-op.h index 83478ab006..b4f73c6048 100644 --- a/tcg/tcg-op.h +++ b/tcg/tcg-op.h @@ -939,6 +939,9 @@ void tcg_gen_uzpo_vec(unsigned vece, TCGv_vec r, TCGv_vec a, TCGv_vec b); void tcg_gen_trne_vec(unsigned vece, TCGv_vec r, TCGv_vec a, TCGv_vec b); void tcg_gen_trno_vec(unsigned vece, TCGv_vec r, TCGv_vec a, TCGv_vec b); +void tcg_gen_cmp_vec(TCGCond cond, unsigned vece, TCGv_vec r, + TCGv_vec a, TCGv_vec b); + void tcg_gen_ld_vec(TCGv_vec r, TCGv_ptr base, TCGArg offset); void tcg_gen_st_vec(TCGv_vec r, TCGv_ptr base, TCGArg offset); void tcg_gen_stl_vec(TCGv_vec r, TCGv_ptr base, TCGArg offset, TCGType t); diff --git a/tcg/tcg-opc.h b/tcg/tcg-opc.h index a085fc077b..d3fa014507 100644 --- a/tcg/tcg-opc.h +++ b/tcg/tcg-opc.h @@ -248,6 +248,8 @@ DEF(uzpo_vec, 1, 2, 0, IMPLVEC | IMPL(TCG_TARGET_HAS_uzp_vec)) DEF(trne_vec, 1, 2, 0, IMPLVEC | IMPL(TCG_TARGET_HAS_trn_vec)) DEF(trno_vec, 1, 2, 0, IMPLVEC | IMPL(TCG_TARGET_HAS_trn_vec)) +DEF(cmp_vec, 1, 2, 1, IMPLVEC) + DEF(last_generic, 0, 0, 0, TCG_OPF_NOT_PRESENT) #if TCG_TARGET_MAYBE_vec diff --git a/tcg/tcg.h b/tcg/tcg.h index 5f414d880e..96760dd2d6 100644 --- a/tcg/tcg.h +++ b/tcg/tcg.h @@ -184,6 +184,7 @@ typedef uint64_t TCGRegSet; #define TCG_TARGET_HAS_zip_vec 0 #define TCG_TARGET_HAS_uzp_vec 0 #define TCG_TARGET_HAS_trn_vec 0 +#define TCG_TARGET_HAS_cmp_vec 0 #else #define TCG_TARGET_MAYBE_vec 1 #endif diff --git a/accel/tcg/tcg-runtime-gvec.c b/accel/tcg/tcg-runtime-gvec.c index fba62f1192..e0cde3216f 100644 --- a/accel/tcg/tcg-runtime-gvec.c +++ b/accel/tcg/tcg-runtime-gvec.c @@ -520,3 +520,27 @@ DO_TRN(gvec_trn8, uint8_t) DO_TRN(gvec_trn16, uint16_t) DO_TRN(gvec_trn32, uint32_t) DO_TRN(gvec_trn64, uint64_t) + +#define DO_CMP1(NAME, TYPE, OP) \ +void HELPER(NAME)(void *d, void *a, void *b, uint32_t desc) \ +{ \ + intptr_t oprsz = simd_oprsz(desc); \ + intptr_t i; \ + for (i = 0; i < oprsz; i += sizeof(vec64)) { \ + *(TYPE *)(d + i) = *(TYPE *)(a + i) OP *(TYPE *)(b + i); \ + } \ + clear_high(d, oprsz, desc); \ +} + +#define DO_CMP2(SZ) \ + DO_CMP1(gvec_eq##SZ, vec##SZ, ==) \ + DO_CMP1(gvec_ne##SZ, vec##SZ, !=) \ + DO_CMP1(gvec_lt##SZ, svec##SZ, <) \ + DO_CMP1(gvec_le##SZ, svec##SZ, <=) \ + DO_CMP1(gvec_ltu##SZ, vec##SZ, <) \ + DO_CMP1(gvec_leu##SZ, vec##SZ, <=) + +DO_CMP2(8) +DO_CMP2(16) +DO_CMP2(32) +DO_CMP2(64) diff --git a/tcg/tcg-op-gvec.c b/tcg/tcg-op-gvec.c index d91b424dfc..1cc28b459f 100644 --- a/tcg/tcg-op-gvec.c +++ b/tcg/tcg-op-gvec.c @@ -1639,3 +1639,205 @@ void tcg_gen_gvec_trno(unsigned vece, uint32_t dofs, uint32_t aofs, tcg_debug_assert(vece <= MO_64); tcg_gen_gvec_3(dofs, aofs, bofs, opsz, maxsz, &g[vece]); } + +/* Expand OPSZ bytes worth of three-operand operations using i32 elements. */ +static void expand_cmp_i32(uint32_t dofs, uint32_t aofs, uint32_t bofs, + uint32_t opsz, TCGCond cond) +{ + TCGv_i32 t0 = tcg_temp_new_i32(); + TCGv_i32 t1 = tcg_temp_new_i32(); + uint32_t i; + + for (i = 0; i < opsz; i += 4) { + tcg_gen_ld_i32(t0, cpu_env, aofs + i); + tcg_gen_ld_i32(t1, cpu_env, bofs + i); + tcg_gen_setcond_i32(cond, t0, t0, t1); + tcg_gen_neg_i32(t0, t0); + tcg_gen_st_i32(t0, cpu_env, dofs + i); + } + tcg_temp_free_i32(t1); + tcg_temp_free_i32(t0); +} + +static void expand_cmp_i64(uint32_t dofs, uint32_t aofs, uint32_t bofs, + uint32_t opsz, TCGCond cond) +{ + TCGv_i64 t0 = tcg_temp_new_i64(); + TCGv_i64 t1 = tcg_temp_new_i64(); + uint32_t i; + + for (i = 0; i < opsz; i += 8) { + tcg_gen_ld_i64(t0, cpu_env, aofs + i); + tcg_gen_ld_i64(t1, cpu_env, bofs + i); + tcg_gen_setcond_i64(cond, t0, t0, t1); + tcg_gen_neg_i64(t0, t0); + tcg_gen_st_i64(t0, cpu_env, dofs + i); + } + tcg_temp_free_i64(t1); + tcg_temp_free_i64(t0); +} + +static void expand_cmp_vec(unsigned vece, uint32_t dofs, uint32_t aofs, + uint32_t bofs, uint32_t opsz, uint32_t tysz, + TCGType type, TCGCond cond) +{ + TCGv_vec t0 = tcg_temp_new_vec(type); + TCGv_vec t1 = tcg_temp_new_vec(type); + uint32_t i; + + for (i = 0; i < opsz; i += tysz) { + tcg_gen_ld_vec(t0, cpu_env, aofs + i); + tcg_gen_ld_vec(t1, cpu_env, bofs + i); + tcg_gen_cmp_vec(cond, vece, t0, t0, t1); + tcg_gen_st_vec(t0, cpu_env, dofs + i); + } + tcg_temp_free_vec(t1); + tcg_temp_free_vec(t0); +} + +void tcg_gen_gvec_cmp(TCGCond cond, unsigned vece, uint32_t dofs, + uint32_t aofs, uint32_t bofs, + uint32_t oprsz, uint32_t maxsz) +{ + static gen_helper_gvec_3 * const eq_fn[4] = { + gen_helper_gvec_eq8, gen_helper_gvec_eq16, + gen_helper_gvec_eq32, gen_helper_gvec_eq64 + }; + static gen_helper_gvec_3 * const ne_fn[4] = { + gen_helper_gvec_ne8, gen_helper_gvec_ne16, + gen_helper_gvec_ne32, gen_helper_gvec_ne64 + }; + static gen_helper_gvec_3 * const lt_fn[4] = { + gen_helper_gvec_lt8, gen_helper_gvec_lt16, + gen_helper_gvec_lt32, gen_helper_gvec_lt64 + }; + static gen_helper_gvec_3 * const le_fn[4] = { + gen_helper_gvec_le8, gen_helper_gvec_le16, + gen_helper_gvec_le32, gen_helper_gvec_le64 + }; + static gen_helper_gvec_3 * const ltu_fn[4] = { + gen_helper_gvec_ltu8, gen_helper_gvec_ltu16, + gen_helper_gvec_ltu32, gen_helper_gvec_ltu64 + }; + static gen_helper_gvec_3 * const leu_fn[4] = { + gen_helper_gvec_leu8, gen_helper_gvec_leu16, + gen_helper_gvec_leu32, gen_helper_gvec_leu64 + }; + gen_helper_gvec_3 *fn; + uint32_t tmp; + + check_size_align(oprsz, maxsz, dofs | aofs | bofs); + check_overlap_3(dofs, aofs, bofs, maxsz); + + if (cond == TCG_COND_NEVER || cond == TCG_COND_ALWAYS) { + tcg_gen_gvec_dup32i(dofs, oprsz, maxsz, -(cond == TCG_COND_ALWAYS)); + return; + } + + /* Quick check for sizes we won't support inline. */ + if (oprsz > MAX_UNROLL * 32 || maxsz > MAX_UNROLL * 32) { + goto do_ool; + } + + /* Recall that ARM SVE allows vector sizes that are not a power of 2. + Expand with successively smaller host vector sizes. The intent is + that e.g. oprsz == 80 would be expanded with 2x32 + 1x16. */ + /* ??? For maxsz > oprsz, the host may be able to use an op-sized + operation, zeroing the balance of the register. We can then + use a cl-sized store to implement the clearing without an extra + store operation. This is true for aarch64 and x86_64 hosts. */ + + if (TCG_TARGET_HAS_v256 && check_size_impl(oprsz, 32) + && tcg_can_emit_vec_op(INDEX_op_cmp_vec, TCG_TYPE_V256, vece)) { + uint32_t done = QEMU_ALIGN_DOWN(oprsz, 32); + expand_cmp_vec(vece, dofs, aofs, bofs, done, 32, TCG_TYPE_V256, cond); + dofs += done; + aofs += done; + bofs += done; + oprsz -= done; + maxsz -= done; + } + + if (TCG_TARGET_HAS_v128 && check_size_impl(oprsz, 16) + && tcg_can_emit_vec_op(INDEX_op_cmp_vec, TCG_TYPE_V128, vece)) { + uint32_t done = QEMU_ALIGN_DOWN(oprsz, 16); + expand_cmp_vec(vece, dofs, aofs, bofs, done, 16, TCG_TYPE_V128, cond); + dofs += done; + aofs += done; + bofs += done; + oprsz -= done; + maxsz -= done; + } + + if (check_size_impl(oprsz, 8)) { + uint32_t done = QEMU_ALIGN_DOWN(oprsz, 8); + if (TCG_TARGET_HAS_v64 + && (TCG_TARGET_REG_BITS == 32 || vece != MO_64) + && tcg_can_emit_vec_op(INDEX_op_cmp_vec, TCG_TYPE_V64, vece)) { + expand_cmp_vec(vece, dofs, aofs, bofs, done, 8, TCG_TYPE_V64, cond); + } else if (vece == MO_64) { + expand_cmp_i64(dofs, aofs, bofs, done, cond); + } else { + done = 0; + } + dofs += done; + aofs += done; + bofs += done; + oprsz -= done; + maxsz -= done; + } + + if (vece == MO_32 && check_size_impl(oprsz, 4)) { + uint32_t done = QEMU_ALIGN_DOWN(oprsz, 4); + expand_cmp_i32(dofs, aofs, bofs, done, cond); + dofs += done; + aofs += done; + bofs += done; + oprsz -= done; + maxsz -= done; + } + + if (oprsz == 0) { + if (maxsz != 0) { + expand_clr(dofs, maxsz); + } + return; + } + + do_ool: + switch (cond) { + case TCG_COND_EQ: + fn = eq_fn[vece]; + break; + case TCG_COND_NE: + fn = ne_fn[vece]; + break; + case TCG_COND_GT: + tmp = aofs, aofs = bofs, bofs = tmp; + /* fallthru */ + case TCG_COND_LT: + fn = lt_fn[vece]; + break; + case TCG_COND_GE: + tmp = aofs, aofs = bofs, bofs = tmp; + /* fallthru */ + case TCG_COND_LE: + fn = le_fn[vece]; + break; + case TCG_COND_GTU: + tmp = aofs, aofs = bofs, bofs = tmp; + /* fallthru */ + case TCG_COND_LTU: + fn = ltu_fn[vece]; + break; + case TCG_COND_GEU: + tmp = aofs, aofs = bofs, bofs = tmp; + /* fallthru */ + case TCG_COND_LEU: + fn = leu_fn[vece]; + break; + default: + g_assert_not_reached(); + } + tcg_gen_gvec_3_ool(dofs, aofs, bofs, oprsz, maxsz, 0, fn); +} diff --git a/tcg/tcg-op-vec.c b/tcg/tcg-op-vec.c index 502c5ba891..2c636ebbd6 100644 --- a/tcg/tcg-op-vec.c +++ b/tcg/tcg-op-vec.c @@ -480,3 +480,26 @@ void tcg_gen_trno_vec(unsigned vece, TCGv_vec r, TCGv_vec a, TCGv_vec b) { do_interleave(INDEX_op_trno_vec, vece, r, a, b); } + +void tcg_gen_cmp_vec(TCGCond cond, unsigned vece, + TCGv_vec r, TCGv_vec a, TCGv_vec b) +{ + TCGTemp *rt = tcgv_vec_temp(r); + TCGTemp *at = tcgv_vec_temp(a); + TCGTemp *bt = tcgv_vec_temp(b); + TCGArg ri = temp_arg(rt); + TCGArg ai = temp_arg(at); + TCGArg bi = temp_arg(bt); + TCGType type = rt->base_type; + int can; + + tcg_debug_assert(at->base_type == type); + tcg_debug_assert(bt->base_type == type); + can = tcg_can_emit_vec_op(INDEX_op_cmp_vec, type, vece); + if (can > 0) { + vec_gen_4(INDEX_op_cmp_vec, type, vece, ri, ai, bi, cond); + } else { + tcg_debug_assert(can < 0); + tcg_expand_vec_op(INDEX_op_cmp_vec, type, vece, ri, ai, bi, cond); + } +} diff --git a/tcg/tcg.c b/tcg/tcg.c index 4bde7d6afd..f35d4a1a47 100644 --- a/tcg/tcg.c +++ b/tcg/tcg.c @@ -1392,6 +1392,7 @@ bool tcg_op_supported(TCGOpcode op) case INDEX_op_and_vec: case INDEX_op_or_vec: case INDEX_op_xor_vec: + case INDEX_op_cmp_vec: return have_vec; case INDEX_op_dup2_vec: return have_vec && TCG_TARGET_REG_BITS == 32; @@ -1791,6 +1792,7 @@ void tcg_dump_ops(TCGContext *s) case INDEX_op_brcond_i64: case INDEX_op_setcond_i64: case INDEX_op_movcond_i64: + case INDEX_op_cmp_vec: if (op->args[k] < ARRAY_SIZE(cond_name) && cond_name[op->args[k]]) { col += qemu_log(",%s", cond_name[op->args[k++]]); diff --git a/tcg/README b/tcg/README index 75db47922d..18b6bbd8f1 100644 --- a/tcg/README +++ b/tcg/README @@ -630,6 +630,10 @@ E.g. VECL=1 -> 64 << 1 -> v128, and VECE=2 -> 1 << 2 -> i32. v0[2i + 1] = v2[2i + part]; } +* cmp_vec v0, v1, v2, cond + + Compare vectors by element, storing -1 for true and 0 for false. + ********* Note 1: Some shortcuts are defined when the last operand is known to be From patchwork Tue Nov 21 21:25:27 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 119425 Delivered-To: patch@linaro.org Received: by 10.140.22.164 with SMTP id 33csp5827044qgn; Tue, 21 Nov 2017 13:45:27 -0800 (PST) X-Google-Smtp-Source: AGs4zMZvHvAUJkN1PPwMAgadyEe+zV1E4Pe3EYLfgPso7LXvFB1K9nzeOiodYoaFheBrexIvrwi+ X-Received: by 10.129.159.72 with SMTP id w69mr12005768ywg.491.1511300727952; Tue, 21 Nov 2017 13:45:27 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1511300727; cv=none; d=google.com; s=arc-20160816; b=BSHMHb9nvpUKsFZHX7LuUeovuK3fQVLOVSQ6mvrMai/GSDAOyx8a0BSrwB3NKwhEGa HgA1TmAq8v9BQJ17CoK1prGQ6qe5Zn27zwko+P0L/d9homOK1ObuiNU2HQtI762goGyT U+Wto9DSe+u9n4DZhidFBUPazoSkipxNx745iVV+0Yahme2sw7BUsMJjjl+ge6BYEJLd 1/I7quwXX2In1WCJEuCKYSvxAAZpnFG7xKrKSeR3f0wCttDqaxbpUM/ZdcwKo5jh27ZE sbF4FG41OK87qmsU0K16+humvVF6e9wUMUgy4/mL8X7pCmXDRGDxHD8QX+3j9vovFBgk i+yA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:subject:references:in-reply-to :message-id:date:to:from:dkim-signature:arc-authentication-results; bh=9nX13rkW5BnudPsgCBbM6M5Hq1UugkSHhy4B6et7qLw=; b=M9b08w+9NlbqThB+Yi3ShtCVvY26G9mIWJP+Zjn7vmSr9wku9//4mBqSuAtycrgbzA 8EDPq9LK9fR4tWzRd/hVv6OC8UIcBDx0ua3f6tb/TzHtsXZtQd4w549kb0U2hm4ZiWkb YIKxnRnqIbNPBWbyTw6N18OrC82nZJhsolpiYMiTz4/cV1b/L39Cd4r/IPMRDSb/rq4T wXZZIMJp0/sTGl7Ya3hrkqojiQn/E2XCohQ3SgLZoIxMieAul7OlZnyLS49HWFk587Hr TST5xOfyH2C5ZSP3yy+QlnzzEQK1VgT7UtAeyzainoGL4O7E4huW0MIhS4GVbR0IEfF3 I3Jg== ARC-Authentication-Results: i=1; mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=HNj226uM; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) smtp.mailfrom=qemu-devel-bounces+patch=linaro.org@nongnu.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [2001:4830:134:3::11]) by mx.google.com with ESMTPS id u1si2984905ybd.714.2017.11.21.13.45.27 for (version=TLS1 cipher=AES128-SHA bits=128/128); Tue, 21 Nov 2017 13:45:27 -0800 (PST) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) client-ip=2001:4830:134:3::11; Authentication-Results: mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=HNj226uM; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) smtp.mailfrom=qemu-devel-bounces+patch=linaro.org@nongnu.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1]:36665 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1eHGMB-0002j4-BM for patch@linaro.org; Tue, 21 Nov 2017 16:45:27 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:54157) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1eHG5L-0004b3-PQ for qemu-devel@nongnu.org; Tue, 21 Nov 2017 16:28:04 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1eHG5K-0007oS-F6 for qemu-devel@nongnu.org; Tue, 21 Nov 2017 16:28:03 -0500 Received: from mail-wm0-x242.google.com ([2a00:1450:400c:c09::242]:37620) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1eHG5K-0007oA-6B for qemu-devel@nongnu.org; Tue, 21 Nov 2017 16:28:02 -0500 Received: by mail-wm0-x242.google.com with SMTP id v186so6270134wma.2 for ; Tue, 21 Nov 2017 13:28:02 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:subject:date:message-id:in-reply-to:references; bh=9nX13rkW5BnudPsgCBbM6M5Hq1UugkSHhy4B6et7qLw=; b=HNj226uMcsK0W7dq9aBFFE0Av0CKkTxXMK9SDbMADir4WMtQcERgfj+Xo+Xm5tbSzq gDlPqtP77AzIF7JqqD3sXn3I7kzZWQk+t+ohBV/zb9Y/eU0kzSgGmddAskdq2pi18HpI dqqdvH4hKrxl0kITSxB7wQhP4P90qnynYzf7g= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references; bh=9nX13rkW5BnudPsgCBbM6M5Hq1UugkSHhy4B6et7qLw=; b=tLiPZBqYKxKKpzkYWm+s9FfLjHKIYLc9StNv4lUPPM6kZg6CXYMs8kQ7mtz+PLZoHJ LoHvDvAhPwc3zaM5fi1ImrcGwTwwsajGk+geVf3x7p1t/4yxiRzvSDEdHIqbaZszWZy/ 7U/ddgTlxf1xiFGmFGeK7t4o3f3cwZhiRXCDP2BjpJS68DnX7DJpXCRNgH+5jymexyQu 7RgMmziVqPfXD4IAuTS/sLTcOAK06k64dWDTlU4bAuKNzYziy43OFm/GlFlpoe24GtPx G1iUfJr5XZ1avWUnEnMC8jEePsgXje0DMPoPDHgkGmRDrgsjYiDTSQKc+VBA3L6BQtRU Sh3A== X-Gm-Message-State: AJaThX4ZtU1gB9W191u73xUunPrjE56LwMRB25Fonc2SBsPck+uIUJ25 FZFran+qXXBZKffKs60Exnt/ggEXY8w= X-Received: by 10.28.125.85 with SMTP id y82mr2440412wmc.25.1511299680955; Tue, 21 Nov 2017 13:28:00 -0800 (PST) Received: from cloudburst.twiddle.net (70.red-37-158-60.dynamicip.rima-tde.net. [37.158.60.70]) by smtp.gmail.com with ESMTPSA id e124sm706517wmg.34.2017.11.21.13.27.59 for (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Tue, 21 Nov 2017 13:27:59 -0800 (PST) From: Richard Henderson To: qemu-devel@nongnu.org Date: Tue, 21 Nov 2017 22:25:27 +0100 Message-Id: <20171121212534.5177-20-richard.henderson@linaro.org> X-Mailer: git-send-email 2.13.6 In-Reply-To: <20171121212534.5177-1-richard.henderson@linaro.org> References: <20171121212534.5177-1-richard.henderson@linaro.org> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2a00:1450:400c:c09::242 Subject: [Qemu-devel] [PATCH v6 19/26] target/arm: Use vector infrastructure for aa64 compares X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: "Qemu-devel" Signed-off-by: Richard Henderson --- target/arm/translate-a64.c | 96 ++++++++++++++++++++++++++++++---------------- 1 file changed, 62 insertions(+), 34 deletions(-) -- 2.13.6 diff --git a/target/arm/translate-a64.c b/target/arm/translate-a64.c index c47faa5633..1ea7e37b03 100644 --- a/target/arm/translate-a64.c +++ b/target/arm/translate-a64.c @@ -7115,6 +7115,28 @@ static void disas_simd_scalar_three_reg_diff(DisasContext *s, uint32_t insn) } } +/* CMTST : test is "if (X & Y != 0)". */ +static void gen_cmtst_i32(TCGv_i32 d, TCGv_i32 a, TCGv_i32 b) +{ + tcg_gen_and_i32(d, a, b); + tcg_gen_setcondi_i32(TCG_COND_NE, d, d, 0); + tcg_gen_neg_i32(d, d); +} + +static void gen_cmtst_i64(TCGv_i64 d, TCGv_i64 a, TCGv_i64 b) +{ + tcg_gen_and_i64(d, a, b); + tcg_gen_setcondi_i64(TCG_COND_NE, d, d, 0); + tcg_gen_neg_i64(d, d); +} + +static void gen_cmtst_vec(unsigned vece, TCGv_vec d, TCGv_vec a, TCGv_vec b) +{ + tcg_gen_and_vec(vece, d, a, b); + tcg_gen_dupi_vec(vece, a, 0); + tcg_gen_cmp_vec(TCG_COND_NE, vece, d, d, a); +} + static void handle_3same_64(DisasContext *s, int opcode, bool u, TCGv_i64 tcg_rd, TCGv_i64 tcg_rn, TCGv_i64 tcg_rm) { @@ -7158,10 +7180,7 @@ static void handle_3same_64(DisasContext *s, int opcode, bool u, cond = TCG_COND_EQ; goto do_cmop; } - /* CMTST : test is "if (X & Y != 0)". */ - tcg_gen_and_i64(tcg_rd, tcg_rn, tcg_rm); - tcg_gen_setcondi_i64(TCG_COND_NE, tcg_rd, tcg_rd, 0); - tcg_gen_neg_i64(tcg_rd, tcg_rd); + gen_cmtst_i64(tcg_rd, tcg_rn, tcg_rm); break; case 0x8: /* SSHL, USHL */ if (u) { @@ -9684,6 +9703,7 @@ static void disas_simd_3same_int(DisasContext *s, uint32_t insn) int rd = extract32(insn, 0, 5); int pass; GVecGen3Fn *gvec_op; + TCGCond cond; switch (opcode) { case 0x13: /* MUL, PMUL */ @@ -9731,6 +9751,44 @@ static void disas_simd_3same_int(DisasContext *s, uint32_t insn) vec_full_reg_offset(s, rm), is_q ? 16 : 8, vec_full_reg_size(s)); return; + case 0x11: + if (u) { /* CMEQ */ + cond = TCG_COND_EQ; + goto do_gvec_cmp; + } else { /* CMTST */ + static const GVecGen3 cmtst_op[4] = { + { .fni4 = gen_helper_neon_tst_u8, + .fniv = gen_cmtst_vec, + .vece = MO_8 }, + { .fni4 = gen_helper_neon_tst_u16, + .fniv = gen_cmtst_vec, + .vece = MO_16 }, + { .fni4 = gen_cmtst_i32, + .fniv = gen_cmtst_vec, + .vece = MO_32 }, + { .fni8 = gen_cmtst_i64, + .fniv = gen_cmtst_vec, + .prefer_i64 = TCG_TARGET_REG_BITS == 64, + .vece = MO_64 }, + }; + tcg_gen_gvec_3(vec_full_reg_offset(s, rd), + vec_full_reg_offset(s, rn), + vec_full_reg_offset(s, rm), + is_q ? 16 : 8, vec_full_reg_size(s), + &cmtst_op[size]); + } + return; + case 0x06: /* CMGT, CMHI */ + cond = u ? TCG_COND_GTU : TCG_COND_GT; + goto do_gvec_cmp; + case 0x07: /* CMGE, CMHS */ + cond = u ? TCG_COND_GEU : TCG_COND_GE; + do_gvec_cmp: + tcg_gen_gvec_cmp(cond, size, vec_full_reg_offset(s, rd), + vec_full_reg_offset(s, rn), + vec_full_reg_offset(s, rm), + is_q ? 16 : 8, vec_full_reg_size(s)); + return; } if (size == 3) { @@ -9813,26 +9871,6 @@ static void disas_simd_3same_int(DisasContext *s, uint32_t insn) genenvfn = fns[size][u]; break; } - case 0x6: /* CMGT, CMHI */ - { - static NeonGenTwoOpFn * const fns[3][2] = { - { gen_helper_neon_cgt_s8, gen_helper_neon_cgt_u8 }, - { gen_helper_neon_cgt_s16, gen_helper_neon_cgt_u16 }, - { gen_helper_neon_cgt_s32, gen_helper_neon_cgt_u32 }, - }; - genfn = fns[size][u]; - break; - } - case 0x7: /* CMGE, CMHS */ - { - static NeonGenTwoOpFn * const fns[3][2] = { - { gen_helper_neon_cge_s8, gen_helper_neon_cge_u8 }, - { gen_helper_neon_cge_s16, gen_helper_neon_cge_u16 }, - { gen_helper_neon_cge_s32, gen_helper_neon_cge_u32 }, - }; - genfn = fns[size][u]; - break; - } case 0x8: /* SSHL, USHL */ { static NeonGenTwoOpFn * const fns[3][2] = { @@ -9905,16 +9943,6 @@ static void disas_simd_3same_int(DisasContext *s, uint32_t insn) genfn = fns[size][u]; break; } - case 0x11: /* CMTST, CMEQ */ - { - static NeonGenTwoOpFn * const fns[3][2] = { - { gen_helper_neon_tst_u8, gen_helper_neon_ceq_u8 }, - { gen_helper_neon_tst_u16, gen_helper_neon_ceq_u16 }, - { gen_helper_neon_tst_u32, gen_helper_neon_ceq_u32 }, - }; - genfn = fns[size][u]; - break; - } case 0x13: /* MUL, PMUL */ if (u) { /* PMUL */ From patchwork Tue Nov 21 21:25:28 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 119417 Delivered-To: patch@linaro.org Received: by 10.140.22.164 with SMTP id 33csp5820237qgn; Tue, 21 Nov 2017 13:37:49 -0800 (PST) X-Google-Smtp-Source: AGs4zMYYZyIKTvnMC9B1lpWUHfBofXPQWsVYmzu1TRyjYWRRusOnYVlJkxARRmIZFIvBQeHAY13g X-Received: by 10.13.247.199 with SMTP id h190mr12850730ywf.235.1511300269572; Tue, 21 Nov 2017 13:37:49 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1511300269; cv=none; d=google.com; s=arc-20160816; b=iFGk1ujuf1ltPIk16dsTW1VwI5vN3wC8WGkyKgL6ehEDhLrFod1DPkC3eIDoXrxFfP NHyrhCulaqEx6eu1m6U93+idAUxMmWIbBDe+sTVlqdMIA8GBhTOHU11oVWP8Gh9L/jOB Gxhz4QRmWWnQRZEVJQKFYrn5yZ0+AXQ9Vp1H536JtyVByrPSdKlsTUfUEsZ9D1tlnJXr eyzbhyyGTIv/88DOgMVThwMb6DY0cSie+b4shs1Hs7eVLX2X1yFZLF2sJHKIH6CHN+lr F9NGt0U4KFQ8UYVKRSvm8EsSG6qKrBk8qleXbZyM0gKSCBDbY5+hWpBk+sAqyR+2qwlx 2RuA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:subject:references:in-reply-to :message-id:date:to:from:dkim-signature:arc-authentication-results; bh=SxsHTrmFL0QP0R8Xwe7DfD3sgRiugKBN4S8d8W/Yb5E=; b=Tjt3A4djj8yQeg0JQ7LyXWYCnV4o9rJSjEu5W+1aIyTICSyoe7NrrvkeRlfvwUNwFa goOA53yTjSWg72ViWwbZ/CqpeJRjuLvaSa1fZfHcec7d/ufSdTWrKuAjuET1uuj1pVYj AtnYZ0E54MzrukS4Jd1l9xcuEElJOVeP6ogmk+vNnqcHEPCO+Hv4XTdYW9Mccx7DP7zU MBjsiq14vJkA+R4jqxcGTgPvr2Q4HZXOIteqK0P1dGb2DWwuWeV+IeIuhhml5TRpuEDB m3qCywnl9str87ORA751KjUsOmEe7pCr7dLvEangqNkCxNEIZj/n1SL7SkljI2oSiGuy WlNg== ARC-Authentication-Results: i=1; mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=T41+7jOR; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) smtp.mailfrom=qemu-devel-bounces+patch=linaro.org@nongnu.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [2001:4830:134:3::11]) by mx.google.com with ESMTPS id 188si3113171ybr.760.2017.11.21.13.37.49 for (version=TLS1 cipher=AES128-SHA bits=128/128); Tue, 21 Nov 2017 13:37:49 -0800 (PST) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) client-ip=2001:4830:134:3::11; Authentication-Results: mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=T41+7jOR; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) smtp.mailfrom=qemu-devel-bounces+patch=linaro.org@nongnu.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1]:36628 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1eHGEn-0004dY-1Q for patch@linaro.org; Tue, 21 Nov 2017 16:37:49 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:54197) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1eHG5P-0004eX-Cw for qemu-devel@nongnu.org; Tue, 21 Nov 2017 16:28:10 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1eHG5M-0007pI-EH for qemu-devel@nongnu.org; Tue, 21 Nov 2017 16:28:07 -0500 Received: from mail-wm0-x242.google.com ([2a00:1450:400c:c09::242]:38919) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1eHG5L-0007og-W9 for qemu-devel@nongnu.org; Tue, 21 Nov 2017 16:28:04 -0500 Received: by mail-wm0-x242.google.com with SMTP id x63so6256437wmf.4 for ; Tue, 21 Nov 2017 13:28:03 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:subject:date:message-id:in-reply-to:references; bh=SxsHTrmFL0QP0R8Xwe7DfD3sgRiugKBN4S8d8W/Yb5E=; b=T41+7jORjh5RLYerhhO7J4GV6Qdg59YSd6yDQ1KFKdpiPlXaOk45E+sR1/021Bg81G 97javp0p1K+VbRBJGkENLjOAIp8IJ8jYmkXlshbLW7NjAV3BVwMOyGHPMQX8Vm7/Im2G UeqJP9cjAW+9t64wsoFzOMkl86Zs/9tkRhyLE= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references; bh=SxsHTrmFL0QP0R8Xwe7DfD3sgRiugKBN4S8d8W/Yb5E=; b=DspOuAGTmwa2Eba/GrwGnDFGE72DyZJ9GNsJvYfEWu2jxQlqzDmnlkkv2PP5yQuMWJ mRojUeL+bYruhQ2i/ab9Bj0HcPj1E0G+1YGZm3kETuUBjYTPl5SVE9t1X7iQlamdS+cR wV72afJiGGukXGUM5cOcmBAztZUJdpyRkMB8+247SumkvF3QsrSVgftTilHdG+VDG9VJ 8kRPH3axa6K/7M7TW1yo+xDypr+H3CyE++zj5Q1eHKjuBOLPxu3ka9gddUyZKA0iLKH7 OqZRdojfsOb/wINKhYVgOA/YKss7DCqZZO4BuMI23Y+wn9BbsH2tJW9b5IDHcIowbiYY kBHQ== X-Gm-Message-State: AJaThX6QsN/IY41N1F5VwCzxKeCzBK4hL+T+3hedfY7papw7l7MO135n wfJbDEVkT/GA9NiedXR1s1wqu0kqruY= X-Received: by 10.28.198.75 with SMTP id w72mr2389886wmf.2.1511299682188; Tue, 21 Nov 2017 13:28:02 -0800 (PST) Received: from cloudburst.twiddle.net (70.red-37-158-60.dynamicip.rima-tde.net. [37.158.60.70]) by smtp.gmail.com with ESMTPSA id e124sm706517wmg.34.2017.11.21.13.28.01 for (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Tue, 21 Nov 2017 13:28:01 -0800 (PST) From: Richard Henderson To: qemu-devel@nongnu.org Date: Tue, 21 Nov 2017 22:25:28 +0100 Message-Id: <20171121212534.5177-21-richard.henderson@linaro.org> X-Mailer: git-send-email 2.13.6 In-Reply-To: <20171121212534.5177-1-richard.henderson@linaro.org> References: <20171121212534.5177-1-richard.henderson@linaro.org> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2a00:1450:400c:c09::242 Subject: [Qemu-devel] [PATCH v6 20/26] tcg/i386: Add vector operations/expansions for shift/cmp/interleave X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: "Qemu-devel" Signed-off-by: Richard Henderson --- tcg/i386/tcg-target.h | 6 +- tcg/i386/tcg-target.opc.h | 7 + tcg/i386/tcg-target.inc.c | 595 +++++++++++++++++++++++++++++++++++++++++++++- 3 files changed, 598 insertions(+), 10 deletions(-) -- 2.13.6 diff --git a/tcg/i386/tcg-target.h b/tcg/i386/tcg-target.h index 46c4dca7be..60d3684750 100644 --- a/tcg/i386/tcg-target.h +++ b/tcg/i386/tcg-target.h @@ -177,13 +177,13 @@ extern bool have_avx2; #define TCG_TARGET_HAS_orc_vec 0 #define TCG_TARGET_HAS_not_vec 0 #define TCG_TARGET_HAS_neg_vec 0 -#define TCG_TARGET_HAS_shi_vec 0 +#define TCG_TARGET_HAS_shi_vec 1 #define TCG_TARGET_HAS_shs_vec 0 #define TCG_TARGET_HAS_shv_vec 0 -#define TCG_TARGET_HAS_zip_vec 0 +#define TCG_TARGET_HAS_zip_vec 1 #define TCG_TARGET_HAS_uzp_vec 0 #define TCG_TARGET_HAS_trn_vec 0 -#define TCG_TARGET_HAS_cmp_vec 0 +#define TCG_TARGET_HAS_cmp_vec 1 #define TCG_TARGET_deposit_i32_valid(ofs, len) \ (((ofs) == 0 && (len) == 8) || ((ofs) == 8 && (len) == 8) || \ diff --git a/tcg/i386/tcg-target.opc.h b/tcg/i386/tcg-target.opc.h index 4816a6c3d4..77125ef818 100644 --- a/tcg/i386/tcg-target.opc.h +++ b/tcg/i386/tcg-target.opc.h @@ -1,3 +1,10 @@ /* Target-specific opcodes for host vector expansion. These will be emitted by tcg_expand_vec_op. For those familiar with GCC internals, consider these to be UNSPEC with names. */ + +DEF(x86_shufps_vec, 1, 2, 1, IMPLVEC) +DEF(x86_vpblendvb_vec, 1, 3, 0, IMPLVEC) +DEF(x86_blend_vec, 1, 2, 1, IMPLVEC) +DEF(x86_packss_vec, 1, 2, 0, IMPLVEC) +DEF(x86_packus_vec, 1, 2, 0, IMPLVEC) +DEF(x86_psrldq_vec, 1, 1, 1, IMPLVEC) diff --git a/tcg/i386/tcg-target.inc.c b/tcg/i386/tcg-target.inc.c index 062cf16607..694d9e5cb5 100644 --- a/tcg/i386/tcg-target.inc.c +++ b/tcg/i386/tcg-target.inc.c @@ -324,6 +324,7 @@ static inline int tcg_target_const_match(tcg_target_long val, TCGType type, # define P_REXB_RM 0 # define P_GS 0 #endif +#define P_EXT3A 0x10000 /* 0x0f 0x3a opcode prefix */ #define P_SIMDF3 0x20000 /* 0xf3 opcode prefix */ #define P_SIMDF2 0x40000 /* 0xf2 opcode prefix */ #define P_VEXL 0x80000 /* Set VEX.L = 1 */ @@ -333,6 +334,7 @@ static inline int tcg_target_const_match(tcg_target_long val, TCGType type, #define OPC_ARITH_GvEv (0x03) /* ... plus (ARITH_FOO << 3) */ #define OPC_ANDN (0xf2 | P_EXT38) #define OPC_ADD_GvEv (OPC_ARITH_GvEv | (ARITH_ADD << 3)) +#define OPC_BLENDPS (0x0c | P_EXT3A | P_DATA16) #define OPC_BSF (0xbc | P_EXT) #define OPC_BSR (0xbd | P_EXT) #define OPC_BSWAP (0xc8 | P_EXT) @@ -372,15 +374,33 @@ static inline int tcg_target_const_match(tcg_target_long val, TCGType type, #define OPC_MOVSLQ (0x63 | P_REXW) #define OPC_MOVZBL (0xb6 | P_EXT) #define OPC_MOVZWL (0xb7 | P_EXT) +#define OPC_PACKSSDW (0x6b | P_EXT | P_DATA16) +#define OPC_PACKSSWB (0x63 | P_EXT | P_DATA16) +#define OPC_PACKUSDW (0x2b | P_EXT38 | P_DATA16) +#define OPC_PACKUSWB (0x67 | P_EXT | P_DATA16) #define OPC_PADDB (0xfc | P_EXT | P_DATA16) #define OPC_PADDW (0xfd | P_EXT | P_DATA16) #define OPC_PADDD (0xfe | P_EXT | P_DATA16) #define OPC_PADDQ (0xd4 | P_EXT | P_DATA16) #define OPC_PAND (0xdb | P_EXT | P_DATA16) #define OPC_PANDN (0xdf | P_EXT | P_DATA16) +#define OPC_PBLENDW (0x0e | P_EXT3A | P_DATA16) #define OPC_PCMPEQB (0x74 | P_EXT | P_DATA16) +#define OPC_PCMPEQW (0x75 | P_EXT | P_DATA16) +#define OPC_PCMPEQD (0x76 | P_EXT | P_DATA16) +#define OPC_PCMPEQQ (0x29 | P_EXT38 | P_DATA16) +#define OPC_PCMPGTB (0x64 | P_EXT | P_DATA16) +#define OPC_PCMPGTW (0x65 | P_EXT | P_DATA16) +#define OPC_PCMPGTD (0x66 | P_EXT | P_DATA16) +#define OPC_PCMPGTQ (0x37 | P_EXT38 | P_DATA16) #define OPC_POR (0xeb | P_EXT | P_DATA16) +#define OPC_PSHUFB (0x00 | P_EXT38 | P_DATA16) #define OPC_PSHUFD (0x70 | P_EXT | P_DATA16) +#define OPC_PSHUFLW (0x70 | P_EXT | P_SIMDF2) +#define OPC_PSHUFHW (0x70 | P_EXT | P_SIMDF3) +#define OPC_PSHIFTW_Ib (0x71 | P_EXT | P_DATA16) /* /2 /6 /4 */ +#define OPC_PSHIFTD_Ib (0x72 | P_EXT | P_DATA16) /* /2 /6 /4 */ +#define OPC_PSHIFTQ_Ib (0x73 | P_EXT | P_DATA16) /* /2 /6 /4 */ #define OPC_PSUBB (0xf8 | P_EXT | P_DATA16) #define OPC_PSUBW (0xf9 | P_EXT | P_DATA16) #define OPC_PSUBD (0xfa | P_EXT | P_DATA16) @@ -389,6 +409,10 @@ static inline int tcg_target_const_match(tcg_target_long val, TCGType type, #define OPC_PUNPCKLWD (0x61 | P_EXT | P_DATA16) #define OPC_PUNPCKLDQ (0x62 | P_EXT | P_DATA16) #define OPC_PUNPCKLQDQ (0x6c | P_EXT | P_DATA16) +#define OPC_PUNPCKHBW (0x68 | P_EXT | P_DATA16) +#define OPC_PUNPCKHWD (0x69 | P_EXT | P_DATA16) +#define OPC_PUNPCKHDQ (0x6a | P_EXT | P_DATA16) +#define OPC_PUNPCKHQDQ (0x6d | P_EXT | P_DATA16) #define OPC_PXOR (0xef | P_EXT | P_DATA16) #define OPC_POP_r32 (0x58) #define OPC_POPCNT (0xb8 | P_EXT | P_SIMDF3) @@ -401,19 +425,26 @@ static inline int tcg_target_const_match(tcg_target_long val, TCGType type, #define OPC_SHIFT_Ib (0xc1) #define OPC_SHIFT_cl (0xd3) #define OPC_SARX (0xf7 | P_EXT38 | P_SIMDF3) +#define OPC_SHUFPS (0xc6 | P_EXT) #define OPC_SHLX (0xf7 | P_EXT38 | P_DATA16) #define OPC_SHRX (0xf7 | P_EXT38 | P_SIMDF2) #define OPC_TESTL (0x85) #define OPC_TZCNT (0xbc | P_EXT | P_SIMDF3) +#define OPC_UD2 (0x0b | P_EXT) +#define OPC_VPBLENDD (0x02 | P_EXT3A | P_DATA16) +#define OPC_VPBLENDVB (0x4c | P_EXT3A | P_DATA16) #define OPC_VPBROADCASTB (0x78 | P_EXT38 | P_DATA16) #define OPC_VPBROADCASTW (0x79 | P_EXT38 | P_DATA16) #define OPC_VPBROADCASTD (0x58 | P_EXT38 | P_DATA16) #define OPC_VPBROADCASTQ (0x59 | P_EXT38 | P_DATA16) +#define OPC_VPERMQ (0x00 | P_EXT3A | P_DATA16 | P_REXW) +#define OPC_VPERM2I128 (0x46 | P_EXT3A | P_DATA16 | P_VEXL) #define OPC_VZEROUPPER (0x77 | P_EXT) #define OPC_XCHG_ax_r32 (0x90) #define OPC_GRP3_Ev (0xf7) #define OPC_GRP5 (0xff) +#define OPC_GRP14 (0x73 | P_EXT | P_DATA16) /* Group 1 opcode extensions for 0x80-0x83. These are also used as modifiers for OPC_ARITH. */ @@ -519,10 +550,12 @@ static void tcg_out_opc(TCGContext *s, int opc, int r, int rm, int x) tcg_out8(s, (uint8_t)(rex | 0x40)); } - if (opc & (P_EXT | P_EXT38)) { + if (opc & (P_EXT | P_EXT38 | P_EXT3A)) { tcg_out8(s, 0x0f); if (opc & P_EXT38) { tcg_out8(s, 0x38); + } else if (opc & P_EXT3A) { + tcg_out8(s, 0x3a); } } @@ -539,10 +572,12 @@ static void tcg_out_opc(TCGContext *s, int opc) } else if (opc & P_SIMDF2) { tcg_out8(s, 0xf2); } - if (opc & (P_EXT | P_EXT38)) { + if (opc & (P_EXT | P_EXT38 | P_EXT3A)) { tcg_out8(s, 0x0f); if (opc & P_EXT38) { tcg_out8(s, 0x38); + } else if (opc & P_EXT3A) { + tcg_out8(s, 0x3a); } } tcg_out8(s, opc); @@ -566,7 +601,7 @@ static void tcg_out_vex_opc(TCGContext *s, int opc, int r, int v, /* Use the two byte form if possible, which cannot encode VEX.W, VEX.B, VEX.X, or an m-mmmm field other than P_EXT. */ - if ((opc & (P_EXT | P_EXT38 | P_REXW)) == P_EXT + if ((opc & (P_EXT | P_EXT38 | P_EXT3A | P_REXW)) == P_EXT && ((rm | index) & 8) == 0) { /* Two byte VEX prefix. */ tcg_out8(s, 0xc5); @@ -577,7 +612,9 @@ static void tcg_out_vex_opc(TCGContext *s, int opc, int r, int v, tcg_out8(s, 0xc4); /* VEX.m-mmmm */ - if (opc & P_EXT38) { + if (opc & P_EXT3A) { + tmp = 3; + } else if (opc & P_EXT38) { tmp = 2; } else if (opc & P_EXT) { tmp = 1; @@ -2638,9 +2675,24 @@ static void tcg_out_vec_op(TCGContext *s, TCGOpcode opc, static int const sub_insn[4] = { OPC_PSUBB, OPC_PSUBW, OPC_PSUBD, OPC_PSUBQ }; + static int const shift_imm_insn[4] = { + OPC_UD2, OPC_PSHIFTW_Ib, OPC_PSHIFTD_Ib, OPC_PSHIFTQ_Ib + }; + static int const cmpeq_insn[4] = { + OPC_PCMPEQB, OPC_PCMPEQW, OPC_PCMPEQD, OPC_PCMPEQQ + }; + static int const cmpgt_insn[4] = { + OPC_PCMPGTB, OPC_PCMPGTW, OPC_PCMPGTD, OPC_PCMPGTQ + }; + static int const punpckl_insn[4] = { + OPC_PUNPCKLBW, OPC_PUNPCKLWD, OPC_PUNPCKLDQ, OPC_PUNPCKLQDQ + }; + static int const punpckh_insn[4] = { + OPC_PUNPCKHBW, OPC_PUNPCKHWD, OPC_PUNPCKHDQ, OPC_PUNPCKHQDQ + }; TCGType type = vecl + TCG_TYPE_V64; - int insn; + int insn, sub; TCGArg a0, a1, a2; a0 = args[0]; @@ -2662,6 +2714,31 @@ static void tcg_out_vec_op(TCGContext *s, TCGOpcode opc, goto gen_simd; case INDEX_op_xor_vec: insn = OPC_PXOR; + goto gen_simd; + case INDEX_op_zipl_vec: + insn = punpckl_insn[vece]; + goto gen_simd; + case INDEX_op_ziph_vec: + insn = punpckh_insn[vece]; + goto gen_simd; + case INDEX_op_x86_packss_vec: + if (vece == MO_8) { + insn = OPC_PACKSSWB; + } else if (vece == MO_16) { + insn = OPC_PACKSSDW; + } else { + g_assert_not_reached(); + } + goto gen_simd; + case INDEX_op_x86_packus_vec: + if (vece == MO_8) { + insn = OPC_PACKUSWB; + } else if (vece == MO_16) { + insn = OPC_PACKUSDW; + } else { + g_assert_not_reached(); + } + goto gen_simd; gen_simd: if (type == TCG_TYPE_V256) { insn |= P_VEXL; @@ -2669,6 +2746,17 @@ static void tcg_out_vec_op(TCGContext *s, TCGOpcode opc, tcg_out_vex_modrm(s, insn, a0, a1, a2); break; + case INDEX_op_cmp_vec: + sub = args[3]; + if (sub == TCG_COND_EQ) { + insn = cmpeq_insn[vece]; + } else if (sub == TCG_COND_GT) { + insn = cmpgt_insn[vece]; + } else { + g_assert_not_reached(); + } + goto gen_simd; + case INDEX_op_andc_vec: insn = OPC_PANDN; if (type == TCG_TYPE_V256) { @@ -2677,6 +2765,25 @@ static void tcg_out_vec_op(TCGContext *s, TCGOpcode opc, tcg_out_vex_modrm(s, insn, a0, a2, a1); break; + case INDEX_op_shli_vec: + sub = 6; + goto gen_shift; + case INDEX_op_shri_vec: + sub = 2; + goto gen_shift; + case INDEX_op_sari_vec: + tcg_debug_assert(vece != MO_64); + sub = 4; + gen_shift: + tcg_debug_assert(vece != MO_8); + insn = shift_imm_insn[vece]; + if (type == TCG_TYPE_V256) { + insn |= P_VEXL; + } + tcg_out_vex_modrm(s, insn, sub, a0, a1); + tcg_out8(s, a2); + break; + case INDEX_op_ld_vec: tcg_out_ld(s, type, a0, a1, a2); break; @@ -2690,6 +2797,42 @@ static void tcg_out_vec_op(TCGContext *s, TCGOpcode opc, tcg_out_dup_vec(s, type, vece, a0, a1); break; + case INDEX_op_x86_shufps_vec: + insn = OPC_SHUFPS; + sub = args[3]; + goto gen_simd_imm8; + case INDEX_op_x86_blend_vec: + if (vece == MO_16) { + insn = OPC_PBLENDW; + } else if (vece == MO_32) { + insn = (have_avx2 ? OPC_VPBLENDD : OPC_BLENDPS); + } else { + g_assert_not_reached(); + } + sub = args[3]; + goto gen_simd_imm8; + gen_simd_imm8: + if (type == TCG_TYPE_V256) { + insn |= P_VEXL; + } + tcg_out_vex_modrm(s, insn, a0, a1, a2); + tcg_out8(s, sub); + break; + + case INDEX_op_x86_vpblendvb_vec: + insn = OPC_VPBLENDVB; + if (type == TCG_TYPE_V256) { + insn |= P_VEXL; + } + tcg_out_vex_modrm(s, insn, a0, a1, a2); + tcg_out8(s, args[3] << 4); + break; + + case INDEX_op_x86_psrldq_vec: + tcg_out_vex_modrm(s, OPC_GRP14, 3, a0, a1); + tcg_out8(s, a2); + break; + default: g_assert_not_reached(); } @@ -2720,6 +2863,8 @@ static const TCGTargetOpDef *tcg_target_op_def(TCGOpcode op) = { .args_ct_str = { "L", "L", "L", "L" } }; static const TCGTargetOpDef x_x = { .args_ct_str = { "x", "x" } }; static const TCGTargetOpDef x_x_x = { .args_ct_str = { "x", "x", "x" } }; + static const TCGTargetOpDef x_x_x_x + = { .args_ct_str = { "x", "x", "x", "x" } }; static const TCGTargetOpDef x_r = { .args_ct_str = { "x", "r" } }; switch (op) { @@ -2932,9 +3077,22 @@ static const TCGTargetOpDef *tcg_target_op_def(TCGOpcode op) case INDEX_op_or_vec: case INDEX_op_xor_vec: case INDEX_op_andc_vec: + case INDEX_op_cmp_vec: + case INDEX_op_zipl_vec: + case INDEX_op_ziph_vec: + case INDEX_op_x86_shufps_vec: + case INDEX_op_x86_blend_vec: + case INDEX_op_x86_packss_vec: + case INDEX_op_x86_packus_vec: return &x_x_x; case INDEX_op_dup_vec: + case INDEX_op_shli_vec: + case INDEX_op_shri_vec: + case INDEX_op_sari_vec: + case INDEX_op_x86_psrldq_vec: return &x_x; + case INDEX_op_x86_vpblendvb_vec: + return &x_x_x_x; default: break; @@ -2951,16 +3109,439 @@ int tcg_can_emit_vec_op(TCGOpcode opc, TCGType type, unsigned vece) case INDEX_op_or_vec: case INDEX_op_xor_vec: case INDEX_op_andc_vec: - return true; + return 1; + case INDEX_op_cmp_vec: + return -1; + + case INDEX_op_shli_vec: + case INDEX_op_shri_vec: + /* We must expand the operation for MO_8. */ + return vece == MO_8 ? -1 : 1; + + case INDEX_op_sari_vec: + /* We must expand the operation for MO_8. */ + if (vece == MO_8) { + return -1; + } + /* We can emulate this for MO_64, but it does not pay off + unless we're producing at least 4 values. */ + if (vece == MO_64) { + return type >= TCG_TYPE_V256 ? -1 : 0; + } + return 1; + + case INDEX_op_zipl_vec: + /* We could support v256, but with 3 insns per opcode. + It is better to expand with v128 instead. */ + return type <= TCG_TYPE_V128; + case INDEX_op_ziph_vec: + if (type == TCG_TYPE_V64) { + return -1; + } + return type == TCG_TYPE_V128; + + case INDEX_op_uzpe_vec: + case INDEX_op_uzpo_vec: + case INDEX_op_trne_vec: + case INDEX_op_trno_vec: + /* ??? Not implemented for V256. */ + return -(type <= TCG_TYPE_V128); default: - return false; + return 0; } } void tcg_expand_vec_op(TCGOpcode opc, TCGType type, unsigned vece, TCGArg a0, ...) { + va_list va; + TCGArg a1, a2; + TCGv_vec v0, v1, v2, t1, t2; + + va_start(va, a0); + v0 = temp_tcgv_vec(arg_temp(a0)); + + switch (opc) { + case INDEX_op_shli_vec: + case INDEX_op_shri_vec: + tcg_debug_assert(vece == MO_8); + a1 = va_arg(va, TCGArg); + a2 = va_arg(va, TCGArg); + /* Unpack to W, shift, and repack. Tricky bits: + (1) Use punpck*bw x,x to produce DDCCBBAA, + i.e. duplicate in other half of the 16-bit lane. + (2) For right-shift, add 8 so that the high half of + the lane becomes zero. For left-shift, we must + shift up and down again. + (3) Step 2 leaves high half zero such that PACKUSWB + (pack with unsigned saturation) does not modify + the quantity. */ + t1 = tcg_temp_new_vec(type); + t2 = tcg_temp_new_vec(type); + vec_gen_3(INDEX_op_zipl_vec, type, MO_8, tcgv_vec_arg(t1), a1, a1); + vec_gen_3(INDEX_op_ziph_vec, type, MO_8, tcgv_vec_arg(t2), a1, a1); + if (opc == INDEX_op_shri_vec) { + vec_gen_3(INDEX_op_shri_vec, type, MO_16, + tcgv_vec_arg(t1), tcgv_vec_arg(t1), a2 + 8); + vec_gen_3(INDEX_op_shri_vec, type, MO_16, + tcgv_vec_arg(t2), tcgv_vec_arg(t2), a2 + 8); + } else { + vec_gen_3(INDEX_op_shli_vec, type, MO_16, + tcgv_vec_arg(t1), tcgv_vec_arg(t1), a2 + 8); + vec_gen_3(INDEX_op_shli_vec, type, MO_16, + tcgv_vec_arg(t2), tcgv_vec_arg(t2), a2 + 8); + vec_gen_3(INDEX_op_shri_vec, type, MO_16, + tcgv_vec_arg(t1), tcgv_vec_arg(t1), 8); + vec_gen_3(INDEX_op_shri_vec, type, MO_16, + tcgv_vec_arg(t2), tcgv_vec_arg(t2), 8); + } + vec_gen_3(INDEX_op_x86_packus_vec, type, MO_8, + a0, tcgv_vec_arg(t1), tcgv_vec_arg(t2)); + tcg_temp_free_vec(t1); + tcg_temp_free_vec(t2); + break; + + case INDEX_op_sari_vec: + a1 = va_arg(va, TCGArg); + a2 = va_arg(va, TCGArg); + if (vece == MO_8) { + /* Unpack to W, shift, and repack, as above. */ + t1 = tcg_temp_new_vec(type); + t2 = tcg_temp_new_vec(type); + vec_gen_3(INDEX_op_zipl_vec, type, MO_8, tcgv_vec_arg(t1), a1, a1); + vec_gen_3(INDEX_op_ziph_vec, type, MO_8, tcgv_vec_arg(t2), a1, a1); + vec_gen_3(INDEX_op_sari_vec, type, MO_16, + tcgv_vec_arg(t1), tcgv_vec_arg(t1), a2 + 8); + vec_gen_3(INDEX_op_sari_vec, type, MO_16, + tcgv_vec_arg(t2), tcgv_vec_arg(t2), a2 + 8); + vec_gen_3(INDEX_op_x86_packss_vec, type, MO_8, + a0, tcgv_vec_arg(t1), tcgv_vec_arg(t2)); + tcg_temp_free_vec(t1); + tcg_temp_free_vec(t2); + break; + } + tcg_debug_assert(vece == MO_64); + /* MO_64: If the shift is <= 32, we can emulate the sign extend by + performing an arithmetic 32-bit shift and overwriting the high + half of the result (note that the ISA says shift of 32 is valid). */ + if (a2 <= 32) { + t1 = tcg_temp_new_vec(type); + vec_gen_3(INDEX_op_sari_vec, type, MO_32, tcgv_vec_arg(t1), a1, a2); + vec_gen_3(INDEX_op_sari_vec, type, MO_64, a0, a1, a2); + vec_gen_4(INDEX_op_x86_blend_vec, type, MO_32, + a0, a0, tcgv_vec_arg(t1), 0xaa); + tcg_temp_free_vec(t1); + break; + } + /* Otherwise we will need to use a compare vs 0 to produce the + sign-extend, shift and merge. */ + t1 = tcg_temp_new_vec(type); + t2 = tcg_const_zeros_vec(type); + vec_gen_4(INDEX_op_cmp_vec, type, MO_64, + tcgv_vec_arg(t1), tcgv_vec_arg(t2), a1, TCG_COND_GT); + tcg_temp_free_vec(t2); + vec_gen_3(INDEX_op_shri_vec, type, MO_64, a0, a1, a2); + vec_gen_3(INDEX_op_shli_vec, type, MO_64, + tcgv_vec_arg(t1), tcgv_vec_arg(t1), 64 - a2); + vec_gen_3(INDEX_op_or_vec, type, MO_64, a0, a0, tcgv_vec_arg(t1)); + tcg_temp_free_vec(t1); + break; + + case INDEX_op_ziph_vec: + tcg_debug_assert(type == TCG_TYPE_V64); + a1 = va_arg(va, TCGArg); + a2 = va_arg(va, TCGArg); + vec_gen_3(INDEX_op_zipl_vec, TCG_TYPE_V128, vece, a0, a1, a2); + vec_gen_3(INDEX_op_x86_psrldq_vec, TCG_TYPE_V128, MO_64, a0, a0, 8); + break; + + case INDEX_op_uzpe_vec: + a1 = va_arg(va, TCGArg); + a2 = va_arg(va, TCGArg); + v1 = temp_tcgv_vec(arg_temp(a1)); + v2 = temp_tcgv_vec(arg_temp(a2)); + + if (type == TCG_TYPE_V128) { + switch (vece) { + case MO_8: + t1 = tcg_temp_new_vec(type); + t2 = tcg_temp_new_vec(type); + tcg_gen_dup16i_vec(t2, 0x00ff); + tcg_gen_and_vec(MO_16, t1, v2, t2); + tcg_gen_and_vec(MO_16, v0, v1, t2); + vec_gen_3(INDEX_op_x86_packus_vec, type, MO_8, + a0, a0, tcgv_vec_arg(t1)); + tcg_temp_free_vec(t1); + tcg_temp_free_vec(t2); + break; + case MO_16: + t1 = tcg_temp_new_vec(type); + t2 = tcg_temp_new_vec(type); + tcg_gen_dup32i_vec(t2, 0x0000ffff); + tcg_gen_and_vec(MO_32, t1, v2, t2); + tcg_gen_and_vec(MO_32, v0, v1, t2); + vec_gen_3(INDEX_op_x86_packus_vec, type, MO_16, + a0, a0, tcgv_vec_arg(t1)); + tcg_temp_free_vec(t1); + tcg_temp_free_vec(t2); + break; + case MO_32: + vec_gen_4(INDEX_op_x86_shufps_vec, type, MO_32, + a0, a1, a2, 0x88); + break; + case MO_64: + tcg_gen_zipl_vec(vece, v0, v1, v2); + break; + default: + g_assert_not_reached(); + } + } else { + tcg_debug_assert(type == TCG_TYPE_V64); + switch (vece) { + case MO_8: + t1 = tcg_temp_new_vec(TCG_TYPE_V128); + vec_gen_3(INDEX_op_zipl_vec, TCG_TYPE_V128, MO_64, + tcgv_vec_arg(t1), a1, a2); + t2 = tcg_temp_new_vec(TCG_TYPE_V128); + tcg_gen_dup16i_vec(t2, 0x00ff); + tcg_gen_and_vec(MO_16, t1, t1, t2); + vec_gen_3(INDEX_op_x86_packus_vec, TCG_TYPE_V128, MO_8, + a0, tcgv_vec_arg(t1), tcgv_vec_arg(t1)); + tcg_temp_free_vec(t1); + tcg_temp_free_vec(t2); + break; + case MO_16: + t1 = tcg_temp_new_vec(TCG_TYPE_V128); + vec_gen_3(INDEX_op_zipl_vec, TCG_TYPE_V128, MO_64, + tcgv_vec_arg(t1), a1, a2); + t2 = tcg_temp_new_vec(TCG_TYPE_V128); + tcg_gen_dup32i_vec(t2, 0x0000ffff); + tcg_gen_and_vec(MO_32, t1, t1, t2); + vec_gen_3(INDEX_op_x86_packus_vec, TCG_TYPE_V128, MO_16, + a0, tcgv_vec_arg(t1), tcgv_vec_arg(t1)); + tcg_temp_free_vec(t1); + tcg_temp_free_vec(t2); + break; + case MO_32: + tcg_gen_zipl_vec(vece, v0, v1, v2); + break; + default: + g_assert_not_reached(); + } + } + break; + + case INDEX_op_uzpo_vec: + a1 = va_arg(va, TCGArg); + a2 = va_arg(va, TCGArg); + v1 = temp_tcgv_vec(arg_temp(a1)); + v2 = temp_tcgv_vec(arg_temp(a2)); + + if (type == TCG_TYPE_V128) { + switch (vece) { + case MO_8: + t1 = tcg_temp_new_vec(type); + tcg_gen_shri_vec(MO_16, t1, v2, 8); + tcg_gen_shri_vec(MO_16, v0, v1, 8); + vec_gen_3(INDEX_op_x86_packus_vec, type, MO_8, + a0, a0, tcgv_vec_arg(t1)); + tcg_temp_free_vec(t1); + break; + case MO_16: + t1 = tcg_temp_new_vec(type); + tcg_gen_shri_vec(MO_32, t1, v2, 16); + tcg_gen_shri_vec(MO_32, v0, v1, 16); + vec_gen_3(INDEX_op_x86_packus_vec, type, MO_16, + a0, a0, tcgv_vec_arg(t1)); + tcg_temp_free_vec(t1); + break; + case MO_32: + vec_gen_4(INDEX_op_x86_shufps_vec, type, MO_32, + a0, a1, a2, 0xdd); + break; + case MO_64: + tcg_gen_ziph_vec(vece, v0, v1, v2); + break; + default: + g_assert_not_reached(); + } + } else { + switch (vece) { + case MO_8: + t1 = tcg_temp_new_vec(TCG_TYPE_V128); + vec_gen_3(INDEX_op_zipl_vec, TCG_TYPE_V128, MO_64, + tcgv_vec_arg(t1), a1, a2); + tcg_gen_shri_vec(MO_16, t1, t1, 8); + vec_gen_3(INDEX_op_x86_packus_vec, TCG_TYPE_V128, MO_8, + a0, tcgv_vec_arg(t1), tcgv_vec_arg(t1)); + tcg_temp_free_vec(t1); + break; + case MO_16: + t1 = tcg_temp_new_vec(TCG_TYPE_V128); + vec_gen_3(INDEX_op_zipl_vec, TCG_TYPE_V128, MO_64, + tcgv_vec_arg(t1), a1, a2); + tcg_gen_shri_vec(MO_32, t1, t1, 16); + vec_gen_3(INDEX_op_x86_packus_vec, TCG_TYPE_V128, MO_16, + a0, tcgv_vec_arg(t1), tcgv_vec_arg(t1)); + tcg_temp_free_vec(t1); + break; + case MO_32: + tcg_gen_ziph_vec(vece, v0, v1, v2); + break; + default: + g_assert_not_reached(); + } + } + break; + + case INDEX_op_trne_vec: + a1 = va_arg(va, TCGArg); + a2 = va_arg(va, TCGArg); + switch (vece) { + case MO_8: + t1 = tcg_temp_new_vec(type); + t2 = tcg_temp_new_vec(type); + vec_gen_3(INDEX_op_shli_vec, type, MO_16, + tcgv_vec_arg(t1), a2, 8); + tcg_gen_dup16i_vec(t2, 0xff00); + vec_gen_4(INDEX_op_x86_vpblendvb_vec, type, MO_8, + a0, a1, tcgv_vec_arg(t1), tcgv_vec_arg(t2)); + tcg_temp_free_vec(t1); + tcg_temp_free_vec(t2); + break; + case MO_16: + t1 = tcg_temp_new_vec(type); + vec_gen_3(INDEX_op_shli_vec, type, MO_32, + tcgv_vec_arg(t1), a2, 16); + vec_gen_4(INDEX_op_x86_blend_vec, type, MO_16, + a0, a1, tcgv_vec_arg(t1), 0xaa); + tcg_temp_free_vec(t1); + break; + case MO_32: + t1 = tcg_temp_new_vec(type); + vec_gen_3(INDEX_op_shli_vec, type, MO_64, + tcgv_vec_arg(t1), a2, 32); + vec_gen_4(INDEX_op_x86_blend_vec, type, MO_32, + a0, a1, tcgv_vec_arg(t1), 0xaa); + tcg_temp_free_vec(t1); + break; + case MO_64: + vec_gen_3(INDEX_op_zipl_vec, type, MO_64, a0, a1, a2); + break; + default: + g_assert_not_reached(); + } + break; + + case INDEX_op_trno_vec: + a1 = va_arg(va, TCGArg); + a2 = va_arg(va, TCGArg); + switch (vece) { + case MO_8: + t1 = tcg_temp_new_vec(type); + t2 = tcg_temp_new_vec(type); + vec_gen_3(INDEX_op_shri_vec, type, MO_16, + tcgv_vec_arg(t1), a1, 8); + tcg_gen_dup16i_vec(t2, 0xff00); + vec_gen_4(INDEX_op_x86_vpblendvb_vec, type, MO_8, + a0, tcgv_vec_arg(t1), a2, tcgv_vec_arg(t2)); + tcg_temp_free_vec(t1); + tcg_temp_free_vec(t2); + break; + case MO_16: + t1 = tcg_temp_new_vec(type); + vec_gen_3(INDEX_op_shri_vec, type, MO_32, + tcgv_vec_arg(t1), a1, 16); + vec_gen_4(INDEX_op_x86_blend_vec, type, MO_16, + a0, tcgv_vec_arg(t1), a2, 0xaa); + tcg_temp_free_vec(t1); + break; + case MO_32: + t1 = tcg_temp_new_vec(type); + vec_gen_3(INDEX_op_shri_vec, type, MO_64, + tcgv_vec_arg(t1), a1, 32); + vec_gen_4(INDEX_op_x86_blend_vec, type, MO_32, + a0, tcgv_vec_arg(t1), a2, 0xaa); + tcg_temp_free_vec(t1); + break; + case MO_64: + vec_gen_3(INDEX_op_ziph_vec, type, MO_64, a0, a1, a2); + break; + default: + g_assert_not_reached(); + } + break; + + case INDEX_op_cmp_vec: + { + enum { + NEED_SWAP = 1, + NEED_INV = 2, + NEED_BIAS = 4 + }; + static const uint8_t fixups[16] = { + [0 ... 15] = -1, + [TCG_COND_EQ] = 0, + [TCG_COND_NE] = NEED_INV, + [TCG_COND_GT] = 0, + [TCG_COND_LT] = NEED_SWAP, + [TCG_COND_LE] = NEED_INV, + [TCG_COND_GE] = NEED_SWAP | NEED_INV, + [TCG_COND_GTU] = NEED_BIAS, + [TCG_COND_LTU] = NEED_BIAS | NEED_SWAP, + [TCG_COND_LEU] = NEED_BIAS | NEED_INV, + [TCG_COND_GEU] = NEED_BIAS | NEED_SWAP | NEED_INV, + }; + + TCGCond cond; + uint8_t fixup; + + a1 = va_arg(va, TCGArg); + a2 = va_arg(va, TCGArg); + cond = va_arg(va, TCGArg); + fixup = fixups[cond & 15]; + tcg_debug_assert(fixup != 0xff); + + if (fixup & NEED_INV) { + cond = tcg_invert_cond(cond); + } + if (fixup & NEED_SWAP) { + TCGArg t; + t = a1, a1 = a2, a2 = t; + cond = tcg_swap_cond(cond); + } + + t1 = t2 = NULL; + if (fixup & NEED_BIAS) { + t1 = tcg_temp_new_vec(type); + t2 = tcg_temp_new_vec(type); + tcg_gen_dupi_vec(vece, t2, 1ull << ((8 << vece) - 1)); + tcg_gen_sub_vec(vece, t1, temp_tcgv_vec(arg_temp(a1)), t2); + tcg_gen_sub_vec(vece, t2, temp_tcgv_vec(arg_temp(a2)), t2); + a1 = tcgv_vec_arg(t1); + a2 = tcgv_vec_arg(t2); + cond = tcg_signed_cond(cond); + } + + tcg_debug_assert(cond == TCG_COND_EQ || cond == TCG_COND_GT); + vec_gen_4(INDEX_op_cmp_vec, type, vece, a0, a1, a2, cond); + + if (fixup & NEED_BIAS) { + tcg_temp_free_vec(t1); + tcg_temp_free_vec(t2); + } + if (fixup & NEED_INV) { + tcg_gen_not_vec(vece, v0, v0); + } + } + break; + + default: + break; + } + + va_end(va); } static const int tcg_target_callee_save_regs[] = { From patchwork Tue Nov 21 21:25:29 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 119429 Delivered-To: patch@linaro.org Received: by 10.140.22.164 with SMTP id 33csp5833937qgn; Tue, 21 Nov 2017 13:53:16 -0800 (PST) X-Google-Smtp-Source: AGs4zMa0NSEPmHvkurggPX4QvOnpkVdsMx5BoiwLxJ/2/NHO2gnXHRH9eqic5b5R6sSrEe+u2Xj1 X-Received: by 10.13.252.69 with SMTP id m66mr12128903ywf.451.1511301196021; Tue, 21 Nov 2017 13:53:16 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1511301196; cv=none; d=google.com; s=arc-20160816; b=nYyLX4yeX88vHO23EOnnHnysvuMcdktIx0LAeXVLMg/ZhzCysxyVWKdFV04uP8xoop fbneR5fUOjzDN0w2b/h6gb6Y9J3GXHTzImAIP0A+ojPYB9rnvwfgzuQGlZosK45iRvkd wkltWL1e8xYpo5MQKPTiYTKo4cTMWD4aGDt9IL6ajeijQtt5AHpt+h69TddN+vtiPiYZ dAxpPfOjrGiBW105Ua7ICYvfr+TtwfbfY2oPlwgS0QF7ZV7GwDIC923TDud6MiJQc2BR F9ZL2bl9NU7KVs6gmgtmKafAfUKvkETf6JPj6tsIwF2Z6nrNn6WNhXY/FypSmYPvv4a/ on+Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:subject:references:in-reply-to :message-id:date:to:from:dkim-signature:arc-authentication-results; bh=nu77O2h7jrWbGP54KYbs3hNiXQ3fIhv9Byd3iuDgXa8=; b=MJu3dywFF5a6rb8BAnKcwzFk/lqN7t1tD0vZd/SAPH5amDGfOTOj30rdcZLaNmN2PD iZ5aHloneY0PAsllARjcEPgxPVMk/p9m55pfL5LMtAtxwM2MZJYN9w9Hq5hGSMPtLHxh 8Tpv/7k1um0flJ3smtbpazT/6TtnTlqRfnqt+X/bBVdHVMGxtCgROJvQ+l+g/u0FKm5M xIakI3jkch5E/AcDT1YsDVOG6jOE3ivERUO8y2W9oeQZCFLUGVThK/6AGVVsOR4vftyx /DP3VM3fq5O2Ia7C21PtpSCmTOtuT+ir5OCJ3ySqUUC8ClLKO0cHCm9HVIjsLXQTSloi J2FQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=kp9nGWv+; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) smtp.mailfrom=qemu-devel-bounces+patch=linaro.org@nongnu.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [2001:4830:134:3::11]) by mx.google.com with ESMTPS id t2si398435ywl.579.2017.11.21.13.53.15 for (version=TLS1 cipher=AES128-SHA bits=128/128); Tue, 21 Nov 2017 13:53:16 -0800 (PST) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) client-ip=2001:4830:134:3::11; Authentication-Results: mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=kp9nGWv+; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) smtp.mailfrom=qemu-devel-bounces+patch=linaro.org@nongnu.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1]:36738 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1eHGTj-0002Ci-II for patch@linaro.org; Tue, 21 Nov 2017 16:53:15 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:54180) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1eHG5O-0004dS-6J for qemu-devel@nongnu.org; Tue, 21 Nov 2017 16:28:09 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1eHG5M-0007pY-Pb for qemu-devel@nongnu.org; Tue, 21 Nov 2017 16:28:06 -0500 Received: from mail-wm0-x242.google.com ([2a00:1450:400c:c09::242]:40250) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1eHG5M-0007pA-Gu for qemu-devel@nongnu.org; Tue, 21 Nov 2017 16:28:04 -0500 Received: by mail-wm0-x242.google.com with SMTP id b189so6267110wmd.5 for ; Tue, 21 Nov 2017 13:28:04 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:subject:date:message-id:in-reply-to:references; bh=nu77O2h7jrWbGP54KYbs3hNiXQ3fIhv9Byd3iuDgXa8=; b=kp9nGWv+EdVWPudxev9RMVEpQFaofTMECWYyVTHZu/XoND4NGAuw4BN2UL6uAa8Aqu cttjiOdS7yGjWYI21cKcr+J4jHABkeIJsPVavNj9T4h16oYyfHF6MGSHUO4kMpUrI36m CBxs+u05mlcfRyYJs85hxGnJ4gyHJd0El68nI= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references; bh=nu77O2h7jrWbGP54KYbs3hNiXQ3fIhv9Byd3iuDgXa8=; b=SIZYpGN2YDmWMR+COCKEx899r1FUkIrHE/P6Z5qjasJ4QWpHxjkZVpZ9p1L0Jf+XDS zBI24dcknlAlQYA4Cftl3XfB666Eko/Urrj63Zo7ZGFbVYJE/GyfwxHnfSx4OSPwUXaH PtqPZz6sswUdmGbwknD9rAFlbnM4xMyGP8isPjsoPHw2RstpS190cs7AIfwEPGriu89l BFh4XL+DoIvSEZQSLZupyeK7Rgi5cmGE2lYmCTTP02tYGMVktMPLe+m2B1PWjS3i6y/q dmAcZLbh9GfZNm/yrDswD8Ws/RovYedL46/wZSGSl6eXbxo8McFpPP2L8B+DQlosjC53 1KNA== X-Gm-Message-State: AJaThX6oME0FWFvzT+p60XN+k/4jzhthyZvrgLlS70IjXj3GIv9MF+D+ 9rlBF2H6mbq9hxPpZMVKXSfMEu2JRkI= X-Received: by 10.28.173.199 with SMTP id w190mr2410509wme.69.1511299683258; Tue, 21 Nov 2017 13:28:03 -0800 (PST) Received: from cloudburst.twiddle.net (70.red-37-158-60.dynamicip.rima-tde.net. [37.158.60.70]) by smtp.gmail.com with ESMTPSA id e124sm706517wmg.34.2017.11.21.13.28.02 for (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Tue, 21 Nov 2017 13:28:02 -0800 (PST) From: Richard Henderson To: qemu-devel@nongnu.org Date: Tue, 21 Nov 2017 22:25:29 +0100 Message-Id: <20171121212534.5177-22-richard.henderson@linaro.org> X-Mailer: git-send-email 2.13.6 In-Reply-To: <20171121212534.5177-1-richard.henderson@linaro.org> References: <20171121212534.5177-1-richard.henderson@linaro.org> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2a00:1450:400c:c09::242 Subject: [Qemu-devel] [PATCH v6 21/26] tcg: Add generic vector ops for multiplication X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: "Qemu-devel" Signed-off-by: Richard Henderson --- accel/tcg/tcg-runtime.h | 5 +++++ tcg/i386/tcg-target.h | 1 + tcg/tcg-op-gvec.h | 2 ++ tcg/tcg-op.h | 1 + tcg/tcg-opc.h | 1 + tcg/tcg.h | 1 + accel/tcg/tcg-runtime-gvec.c | 44 ++++++++++++++++++++++++++++++++++++++++++++ tcg/tcg-op-gvec.c | 29 +++++++++++++++++++++++++++++ tcg/tcg-op-vec.c | 22 ++++++++++++++++++++++ tcg/tcg.c | 2 ++ tcg/README | 4 ++++ 11 files changed, 112 insertions(+) -- 2.13.6 diff --git a/accel/tcg/tcg-runtime.h b/accel/tcg/tcg-runtime.h index 28abf30d76..c4a2e6b215 100644 --- a/accel/tcg/tcg-runtime.h +++ b/accel/tcg/tcg-runtime.h @@ -152,6 +152,11 @@ DEF_HELPER_FLAGS_4(gvec_sub16, TCG_CALL_NO_RWG, void, ptr, ptr, ptr, i32) DEF_HELPER_FLAGS_4(gvec_sub32, TCG_CALL_NO_RWG, void, ptr, ptr, ptr, i32) DEF_HELPER_FLAGS_4(gvec_sub64, TCG_CALL_NO_RWG, void, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_4(gvec_mul8, TCG_CALL_NO_RWG, void, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_4(gvec_mul16, TCG_CALL_NO_RWG, void, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_4(gvec_mul32, TCG_CALL_NO_RWG, void, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_4(gvec_mul64, TCG_CALL_NO_RWG, void, ptr, ptr, ptr, i32) + DEF_HELPER_FLAGS_3(gvec_neg8, TCG_CALL_NO_RWG, void, ptr, ptr, i32) DEF_HELPER_FLAGS_3(gvec_neg16, TCG_CALL_NO_RWG, void, ptr, ptr, i32) DEF_HELPER_FLAGS_3(gvec_neg32, TCG_CALL_NO_RWG, void, ptr, ptr, i32) diff --git a/tcg/i386/tcg-target.h b/tcg/i386/tcg-target.h index 60d3684750..949d138c9d 100644 --- a/tcg/i386/tcg-target.h +++ b/tcg/i386/tcg-target.h @@ -184,6 +184,7 @@ extern bool have_avx2; #define TCG_TARGET_HAS_uzp_vec 0 #define TCG_TARGET_HAS_trn_vec 0 #define TCG_TARGET_HAS_cmp_vec 1 +#define TCG_TARGET_HAS_mul_vec 0 #define TCG_TARGET_deposit_i32_valid(ofs, len) \ (((ofs) == 0 && (len) == 8) || ((ofs) == 8 && (len) == 8) || \ diff --git a/tcg/tcg-op-gvec.h b/tcg/tcg-op-gvec.h index 308bdc13b4..ad5e22e1bf 100644 --- a/tcg/tcg-op-gvec.h +++ b/tcg/tcg-op-gvec.h @@ -134,6 +134,8 @@ void tcg_gen_gvec_add(unsigned vece, uint32_t dofs, uint32_t aofs, uint32_t bofs, uint32_t opsz, uint32_t clsz); void tcg_gen_gvec_sub(unsigned vece, uint32_t dofs, uint32_t aofs, uint32_t bofs, uint32_t opsz, uint32_t clsz); +void tcg_gen_gvec_mul(unsigned vece, uint32_t dofs, uint32_t aofs, + uint32_t bofs, uint32_t opsz, uint32_t clsz); void tcg_gen_gvec_and(unsigned vece, uint32_t dofs, uint32_t aofs, uint32_t bofs, uint32_t opsz, uint32_t clsz); diff --git a/tcg/tcg-op.h b/tcg/tcg-op.h index b4f73c6048..3296a7baa5 100644 --- a/tcg/tcg-op.h +++ b/tcg/tcg-op.h @@ -920,6 +920,7 @@ void tcg_gen_movi_v128(TCGv_vec, uint64_t, uint64_t); void tcg_gen_movi_v256(TCGv_vec, uint64_t, uint64_t, uint64_t, uint64_t); void tcg_gen_add_vec(unsigned vece, TCGv_vec r, TCGv_vec a, TCGv_vec b); void tcg_gen_sub_vec(unsigned vece, TCGv_vec r, TCGv_vec a, TCGv_vec b); +void tcg_gen_mul_vec(unsigned vece, TCGv_vec r, TCGv_vec a, TCGv_vec b); void tcg_gen_and_vec(unsigned vece, TCGv_vec r, TCGv_vec a, TCGv_vec b); void tcg_gen_or_vec(unsigned vece, TCGv_vec r, TCGv_vec a, TCGv_vec b); void tcg_gen_xor_vec(unsigned vece, TCGv_vec r, TCGv_vec a, TCGv_vec b); diff --git a/tcg/tcg-opc.h b/tcg/tcg-opc.h index d3fa014507..b21a30273c 100644 --- a/tcg/tcg-opc.h +++ b/tcg/tcg-opc.h @@ -220,6 +220,7 @@ DEF(st_vec, 0, 2, 1, IMPLVEC) DEF(add_vec, 1, 2, 0, IMPLVEC) DEF(sub_vec, 1, 2, 0, IMPLVEC) +DEF(mul_vec, 1, 2, 0, IMPLVEC | IMPL(TCG_TARGET_HAS_mul_vec)) DEF(neg_vec, 1, 1, 0, IMPLVEC | IMPL(TCG_TARGET_HAS_neg_vec)) DEF(and_vec, 1, 2, 0, IMPLVEC) diff --git a/tcg/tcg.h b/tcg/tcg.h index 96760dd2d6..a729e66b66 100644 --- a/tcg/tcg.h +++ b/tcg/tcg.h @@ -185,6 +185,7 @@ typedef uint64_t TCGRegSet; #define TCG_TARGET_HAS_uzp_vec 0 #define TCG_TARGET_HAS_trn_vec 0 #define TCG_TARGET_HAS_cmp_vec 0 +#define TCG_TARGET_HAS_mul_vec 0 #else #define TCG_TARGET_MAYBE_vec 1 #endif diff --git a/accel/tcg/tcg-runtime-gvec.c b/accel/tcg/tcg-runtime-gvec.c index e0cde3216f..9406ccd769 100644 --- a/accel/tcg/tcg-runtime-gvec.c +++ b/accel/tcg/tcg-runtime-gvec.c @@ -141,6 +141,50 @@ void HELPER(gvec_sub64)(void *d, void *a, void *b, uint32_t desc) clear_high(d, oprsz, desc); } +void HELPER(gvec_mul8)(void *d, void *a, void *b, uint32_t desc) +{ + intptr_t oprsz = simd_oprsz(desc); + intptr_t i; + + for (i = 0; i < oprsz; i += sizeof(vec8)) { + *(vec8 *)(d + i) = *(vec8 *)(a + i) * *(vec8 *)(b + i); + } + clear_high(d, oprsz, desc); +} + +void HELPER(gvec_mul16)(void *d, void *a, void *b, uint32_t desc) +{ + intptr_t oprsz = simd_oprsz(desc); + intptr_t i; + + for (i = 0; i < oprsz; i += sizeof(vec16)) { + *(vec16 *)(d + i) = *(vec16 *)(a + i) * *(vec16 *)(b + i); + } + clear_high(d, oprsz, desc); +} + +void HELPER(gvec_mul32)(void *d, void *a, void *b, uint32_t desc) +{ + intptr_t oprsz = simd_oprsz(desc); + intptr_t i; + + for (i = 0; i < oprsz; i += sizeof(vec32)) { + *(vec32 *)(d + i) = *(vec32 *)(a + i) * *(vec32 *)(b + i); + } + clear_high(d, oprsz, desc); +} + +void HELPER(gvec_mul64)(void *d, void *a, void *b, uint32_t desc) +{ + intptr_t oprsz = simd_oprsz(desc); + intptr_t i; + + for (i = 0; i < oprsz; i += sizeof(vec64)) { + *(vec64 *)(d + i) = *(vec64 *)(a + i) * *(vec64 *)(b + i); + } + clear_high(d, oprsz, desc); +} + void HELPER(gvec_neg8)(void *d, void *a, uint32_t desc) { intptr_t oprsz = simd_oprsz(desc); diff --git a/tcg/tcg-op-gvec.c b/tcg/tcg-op-gvec.c index 1cc28b459f..3f567f0027 100644 --- a/tcg/tcg-op-gvec.c +++ b/tcg/tcg-op-gvec.c @@ -1026,6 +1026,35 @@ void tcg_gen_gvec_sub(unsigned vece, uint32_t dofs, uint32_t aofs, tcg_gen_gvec_3(dofs, aofs, bofs, opsz, maxsz, &g[vece]); } +void tcg_gen_gvec_mul(unsigned vece, uint32_t dofs, uint32_t aofs, + uint32_t bofs, uint32_t opsz, uint32_t maxsz) +{ + static const GVecGen3 g[4] = { + { .fniv = tcg_gen_mul_vec, + .fno = gen_helper_gvec_mul8, + .opc = INDEX_op_mul_vec, + .vece = MO_8 }, + { .fniv = tcg_gen_mul_vec, + .fno = gen_helper_gvec_mul16, + .opc = INDEX_op_mul_vec, + .vece = MO_16 }, + { .fni4 = tcg_gen_mul_i32, + .fniv = tcg_gen_mul_vec, + .fno = gen_helper_gvec_mul32, + .opc = INDEX_op_mul_vec, + .vece = MO_32 }, + { .fni8 = tcg_gen_mul_i64, + .fniv = tcg_gen_mul_vec, + .fno = gen_helper_gvec_mul64, + .opc = INDEX_op_mul_vec, + .prefer_i64 = TCG_TARGET_REG_BITS == 64, + .vece = MO_64 }, + }; + + tcg_debug_assert(vece <= MO_64); + tcg_gen_gvec_3(dofs, aofs, bofs, opsz, maxsz, &g[vece]); +} + /* Perform a vector negation using normal negation and a mask. Compare gen_subv_mask above. */ static void gen_negv_mask(TCGv_i64 d, TCGv_i64 b, TCGv_i64 m) diff --git a/tcg/tcg-op-vec.c b/tcg/tcg-op-vec.c index 2c636ebbd6..160db2e26e 100644 --- a/tcg/tcg-op-vec.c +++ b/tcg/tcg-op-vec.c @@ -503,3 +503,25 @@ void tcg_gen_cmp_vec(TCGCond cond, unsigned vece, tcg_expand_vec_op(INDEX_op_cmp_vec, type, vece, ri, ai, bi, cond); } } + +void tcg_gen_mul_vec(unsigned vece, TCGv_vec r, TCGv_vec a, TCGv_vec b) +{ + TCGTemp *rt = tcgv_vec_temp(r); + TCGTemp *at = tcgv_vec_temp(a); + TCGTemp *bt = tcgv_vec_temp(b); + TCGArg ri = temp_arg(rt); + TCGArg ai = temp_arg(at); + TCGArg bi = temp_arg(bt); + TCGType type = rt->base_type; + int can; + + tcg_debug_assert(at->base_type == type); + tcg_debug_assert(bt->base_type == type); + can = tcg_can_emit_vec_op(INDEX_op_cmp_vec, type, vece); + if (can > 0) { + vec_gen_3(INDEX_op_mul_vec, type, vece, ri, ai, bi); + } else { + tcg_debug_assert(can < 0); + tcg_expand_vec_op(INDEX_op_mul_vec, type, vece, ri, ai, bi); + } +} diff --git a/tcg/tcg.c b/tcg/tcg.c index f35d4a1a47..b0a7cb7d38 100644 --- a/tcg/tcg.c +++ b/tcg/tcg.c @@ -1404,6 +1404,8 @@ bool tcg_op_supported(TCGOpcode op) return have_vec && TCG_TARGET_HAS_andc_vec; case INDEX_op_orc_vec: return have_vec && TCG_TARGET_HAS_orc_vec; + case INDEX_op_mul_vec: + return have_vec && TCG_TARGET_HAS_mul_vec; case INDEX_op_shli_vec: case INDEX_op_shri_vec: case INDEX_op_sari_vec: diff --git a/tcg/README b/tcg/README index 18b6bbd8f1..17695ff7f6 100644 --- a/tcg/README +++ b/tcg/README @@ -547,6 +547,10 @@ E.g. VECL=1 -> 64 << 1 -> v128, and VECE=2 -> 1 << 2 -> i32. Similarly, v0 = v1 - v2. +* mul_vec v0, v1, v2 + + Similarly, v0 = v1 * v2. + * neg_vec v0, v1 Similarly, v0 = -v1. From patchwork Tue Nov 21 21:25:30 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 119415 Delivered-To: patch@linaro.org Received: by 10.140.22.164 with SMTP id 33csp5819739qgn; Tue, 21 Nov 2017 13:37:17 -0800 (PST) X-Google-Smtp-Source: AGs4zMa88fQsXs9kmcImRC7zX2l9/5DBABtfoEVxLb6aA5Ok6qLeqC27qtzWTKUYxArnz1M1MFJY X-Received: by 10.129.172.66 with SMTP id z2mr3617988ywj.397.1511300237023; Tue, 21 Nov 2017 13:37:17 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1511300237; cv=none; d=google.com; s=arc-20160816; b=0S+QWLksLSOlXjemtJd2rt8OWPTVcXECORhjwuXVWCxN8t0BZfedOe8KdltADNv5Hh tvz1GCOsdOpqnRcSX8P2DL41MFw55BVLAIkoqHf25ey/Z7HUIT89iNEnIHeSzXPXjGRK h6cVscS/CiOq6z7BmiIBLVMT+3yExfK5P9OgwwRF8zwiD1PbFnkFKIJf4knMaq3XfBP8 heoxmt+1m7rGUlB31Ocj0WkW/2kiQaIWm7CTILovVxp4GLAyZnOqoSq8J04rsmHKZfqv jtfzP5cswQXxgO5BwSCl0gXQASeSlYcKC77ZQvciEYrjE5Pz14yconit4sRSnt0Y+OMX jMuw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:subject:references:in-reply-to :message-id:date:to:from:dkim-signature:arc-authentication-results; bh=BYEvdLCnHSRMQQOqzqA9lQqG/GgVmtNTMqhFQhpcx8o=; b=dLIYdWI+KaGu3KSNXJOcsR2C2W1MvtqtSHxc2sI06yHx0zRhLRt70qRxXNUhoR3sD4 m7Fsopg87M7imV3FZUUwC/g+kvZdxARANucDnbeqVqmEpuibSm3aLSVv5ZiS5D3TQInV Z1fWH0OttAhGbS/4kFgyg1cxFle/sw5pK4u+ZwRLoAvpFVD+toCoWp9eAeuFe8pGDg2o m6IR88MVGEJ6YWEhO4HZ+P1nM1ZCO59SB+7/0c4MVkiJ4bmskGgUujbwvV21Bbs2YPKy JT0LNpXSGzb2y5NdrVhNDvsoNxv4ovibpngPf+t5xGpzpeEnR/iubLu9K8ogogajp54U FYEQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=DyYej7dR; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) smtp.mailfrom=qemu-devel-bounces+patch=linaro.org@nongnu.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [2001:4830:134:3::11]) by mx.google.com with ESMTPS id k197si3048411ybf.393.2017.11.21.13.37.16 for (version=TLS1 cipher=AES128-SHA bits=128/128); Tue, 21 Nov 2017 13:37:17 -0800 (PST) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) client-ip=2001:4830:134:3::11; Authentication-Results: mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=DyYej7dR; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) smtp.mailfrom=qemu-devel-bounces+patch=linaro.org@nongnu.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1]:36618 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1eHGEG-00035V-Cv for patch@linaro.org; Tue, 21 Nov 2017 16:37:16 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:54213) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1eHG5R-0004gC-6g for qemu-devel@nongnu.org; Tue, 21 Nov 2017 16:28:10 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1eHG5O-0007pv-1B for qemu-devel@nongnu.org; Tue, 21 Nov 2017 16:28:09 -0500 Received: from mail-wm0-x241.google.com ([2a00:1450:400c:c09::241]:38919) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1eHG5N-0007pf-N1 for qemu-devel@nongnu.org; Tue, 21 Nov 2017 16:28:05 -0500 Received: by mail-wm0-x241.google.com with SMTP id x63so6256553wmf.4 for ; Tue, 21 Nov 2017 13:28:05 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:subject:date:message-id:in-reply-to:references; bh=BYEvdLCnHSRMQQOqzqA9lQqG/GgVmtNTMqhFQhpcx8o=; b=DyYej7dRV8HU4MKXP7UZRj/zbXmgr8vk1C4Yan9dp1YRCJbeBlLaO4TAB2fsxkQSWG DuTVUoOQ80yEO/beB5/GvCAUVVSqD1vach+0Nq7Ft2Pbg8DDzqOxj5bkKpidY/ipUfSm rM2S8dh9Sc8adNWLffjvsKLVEBaVB630d/04Q= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references; bh=BYEvdLCnHSRMQQOqzqA9lQqG/GgVmtNTMqhFQhpcx8o=; b=LRT7t5t9OJKq2QHKYaYB6Uka8xk4f5tDJ1TIFOn2zPKKBoG0B3ULAPjPGBw5Pmo70Y f52mkLjfjYbL750LuX3wiiDer5InG2zDGosKGl87wfriYaSy1F0A9/4Y4qPjefsiWVUn 0kw6jvuqZIT9C37C67A0wy16HWTOZfUevmo27+o9RJPyUTioQUR3Ld11LNpdj2O916+I 0jyMYtEW/Si4ISEtUL7qTcm87/4zZ8PHKWUuWUyoRpO4fFk/HrsrzRIxDFvjNWcfRZXE A3NSZLPJ0TjTUcgXYZwzGqA8KpdoWe3kenwds6WaVWsZ91dkmnlz3WPOJ3KJPmBp8JzC efIw== X-Gm-Message-State: AJaThX5fvajjcRRmCXrPyBibkPXlM4vHDfDf3AHb7pH6DVUT+LA2nKxY Hh7HGAtc/TJzsXtZ0oKgu5NgH4HkgGM= X-Received: by 10.28.170.69 with SMTP id t66mr2363735wme.15.1511299684460; Tue, 21 Nov 2017 13:28:04 -0800 (PST) Received: from cloudburst.twiddle.net (70.red-37-158-60.dynamicip.rima-tde.net. [37.158.60.70]) by smtp.gmail.com with ESMTPSA id e124sm706517wmg.34.2017.11.21.13.28.03 for (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Tue, 21 Nov 2017 13:28:03 -0800 (PST) From: Richard Henderson To: qemu-devel@nongnu.org Date: Tue, 21 Nov 2017 22:25:30 +0100 Message-Id: <20171121212534.5177-23-richard.henderson@linaro.org> X-Mailer: git-send-email 2.13.6 In-Reply-To: <20171121212534.5177-1-richard.henderson@linaro.org> References: <20171121212534.5177-1-richard.henderson@linaro.org> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2a00:1450:400c:c09::241 Subject: [Qemu-devel] [PATCH v6 22/26] target/arm: Use vector infrastructure for aa64 multiplies X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: "Qemu-devel" Signed-off-by: Richard Henderson --- target/arm/translate-a64.c | 171 ++++++++++++++++++++++++++++++++++++--------- 1 file changed, 138 insertions(+), 33 deletions(-) -- 2.13.6 diff --git a/target/arm/translate-a64.c b/target/arm/translate-a64.c index 1ea7e37b03..c47d9caa49 100644 --- a/target/arm/translate-a64.c +++ b/target/arm/translate-a64.c @@ -9691,6 +9691,66 @@ static void disas_simd_3same_float(DisasContext *s, uint32_t insn) } } +static void gen_mla8_i32(TCGv_i32 d, TCGv_i32 a, TCGv_i32 b) +{ + gen_helper_neon_mul_u8(a, a, b); + gen_helper_neon_add_u8(d, d, a); +} + +static void gen_mla16_i32(TCGv_i32 d, TCGv_i32 a, TCGv_i32 b) +{ + gen_helper_neon_mul_u16(a, a, b); + gen_helper_neon_add_u16(d, d, a); +} + +static void gen_mla32_i32(TCGv_i32 d, TCGv_i32 a, TCGv_i32 b) +{ + tcg_gen_mul_i32(a, a, b); + tcg_gen_add_i32(d, d, a); +} + +static void gen_mla64_i64(TCGv_i64 d, TCGv_i64 a, TCGv_i64 b) +{ + tcg_gen_mul_i64(a, a, b); + tcg_gen_add_i64(d, d, a); +} + +static void gen_mla_vec(unsigned vece, TCGv_vec d, TCGv_vec a, TCGv_vec b) +{ + tcg_gen_mul_vec(vece, a, a, b); + tcg_gen_add_vec(vece, d, d, a); +} + +static void gen_mls8_i32(TCGv_i32 d, TCGv_i32 a, TCGv_i32 b) +{ + gen_helper_neon_mul_u8(a, a, b); + gen_helper_neon_sub_u8(d, d, a); +} + +static void gen_mls16_i32(TCGv_i32 d, TCGv_i32 a, TCGv_i32 b) +{ + gen_helper_neon_mul_u16(a, a, b); + gen_helper_neon_sub_u16(d, d, a); +} + +static void gen_mls32_i32(TCGv_i32 d, TCGv_i32 a, TCGv_i32 b) +{ + tcg_gen_mul_i32(a, a, b); + tcg_gen_sub_i32(d, d, a); +} + +static void gen_mls64_i64(TCGv_i64 d, TCGv_i64 a, TCGv_i64 b) +{ + tcg_gen_mul_i64(a, a, b); + tcg_gen_sub_i64(d, d, a); +} + +static void gen_mls_vec(unsigned vece, TCGv_vec d, TCGv_vec a, TCGv_vec b) +{ + tcg_gen_mul_vec(vece, a, a, b); + tcg_gen_sub_vec(vece, d, d, a); +} + /* Integer op subgroup of C3.6.16. */ static void disas_simd_3same_int(DisasContext *s, uint32_t insn) { @@ -9702,7 +9762,8 @@ static void disas_simd_3same_int(DisasContext *s, uint32_t insn) int rn = extract32(insn, 5, 5); int rd = extract32(insn, 0, 5); int pass; - GVecGen3Fn *gvec_op; + GVecGen3Fn *gvec_fn; + const GVecGen3 *gvec_op; TCGCond cond; switch (opcode) { @@ -9745,12 +9806,70 @@ static void disas_simd_3same_int(DisasContext *s, uint32_t insn) switch (opcode) { case 0x10: /* ADD, SUB */ - gvec_op = u ? tcg_gen_gvec_sub : tcg_gen_gvec_add; - gvec_op(size, vec_full_reg_offset(s, rd), + gvec_fn = u ? tcg_gen_gvec_sub : tcg_gen_gvec_add; + do_gvec: + gvec_fn(size, vec_full_reg_offset(s, rd), vec_full_reg_offset(s, rn), vec_full_reg_offset(s, rm), is_q ? 16 : 8, vec_full_reg_size(s)); return; + case 0x13: /* MUL, PMUL */ + if (!u) { /* MUL */ + gvec_fn = tcg_gen_gvec_mul; + goto do_gvec; + } + break; + case 0x12: /* MLA, MLS */ + { + static const GVecGen3 mla_op[4] = { + { .fni4 = gen_mla8_i32, + .fniv = gen_mla_vec, + .opc = INDEX_op_mul_vec, + .load_dest = true, + .vece = MO_8 }, + { .fni4 = gen_mla16_i32, + .fniv = gen_mla_vec, + .opc = INDEX_op_mul_vec, + .load_dest = true, + .vece = MO_16 }, + { .fni4 = gen_mla32_i32, + .fniv = gen_mla_vec, + .opc = INDEX_op_mul_vec, + .load_dest = true, + .vece = MO_32 }, + { .fni8 = gen_mla64_i64, + .fniv = gen_mla_vec, + .opc = INDEX_op_mul_vec, + .prefer_i64 = TCG_TARGET_REG_BITS == 64, + .load_dest = true, + .vece = MO_64 }, + }; + static const GVecGen3 mls_op[4] = { + { .fni4 = gen_mls8_i32, + .fniv = gen_mls_vec, + .opc = INDEX_op_mul_vec, + .load_dest = true, + .vece = MO_8 }, + { .fni4 = gen_mls16_i32, + .fniv = gen_mls_vec, + .opc = INDEX_op_mul_vec, + .load_dest = true, + .vece = MO_16 }, + { .fni4 = gen_mls32_i32, + .fniv = gen_mls_vec, + .opc = INDEX_op_mul_vec, + .load_dest = true, + .vece = MO_32 }, + { .fni8 = gen_mls64_i64, + .fniv = gen_mls_vec, + .opc = INDEX_op_mul_vec, + .prefer_i64 = TCG_TARGET_REG_BITS == 64, + .load_dest = true, + .vece = MO_64 }, + }; + gvec_op = (u ? &mls_op[size] : &mla_op[size]); + } + goto do_gvec_op; case 0x11: if (u) { /* CMEQ */ cond = TCG_COND_EQ; @@ -9771,12 +9890,13 @@ static void disas_simd_3same_int(DisasContext *s, uint32_t insn) .prefer_i64 = TCG_TARGET_REG_BITS == 64, .vece = MO_64 }, }; - tcg_gen_gvec_3(vec_full_reg_offset(s, rd), - vec_full_reg_offset(s, rn), - vec_full_reg_offset(s, rm), - is_q ? 16 : 8, vec_full_reg_size(s), - &cmtst_op[size]); + gvec_op = &cmtst_op[size]; } + do_gvec_op: + tcg_gen_gvec_3(vec_full_reg_offset(s, rd), + vec_full_reg_offset(s, rn), + vec_full_reg_offset(s, rm), + is_q ? 16 : 8, vec_full_reg_size(s), gvec_op); return; case 0x06: /* CMGT, CMHI */ cond = u ? TCG_COND_GTU : TCG_COND_GT; @@ -9944,23 +10064,10 @@ static void disas_simd_3same_int(DisasContext *s, uint32_t insn) break; } case 0x13: /* MUL, PMUL */ - if (u) { - /* PMUL */ - assert(size == 0); - genfn = gen_helper_neon_mul_p8; - break; - } - /* fall through : MUL */ - case 0x12: /* MLA, MLS */ - { - static NeonGenTwoOpFn * const fns[3] = { - gen_helper_neon_mul_u8, - gen_helper_neon_mul_u16, - tcg_gen_mul_i32, - }; - genfn = fns[size]; + assert(u); /* PMUL */ + assert(size == 0); + genfn = gen_helper_neon_mul_p8; break; - } case 0x16: /* SQDMULH, SQRDMULH */ { static NeonGenTwoOpEnvFn * const fns[2][2] = { @@ -9981,18 +10088,16 @@ static void disas_simd_3same_int(DisasContext *s, uint32_t insn) genfn(tcg_res, tcg_op1, tcg_op2); } - if (opcode == 0xf || opcode == 0x12) { - /* SABA, UABA, MLA, MLS: accumulating ops */ - static NeonGenTwoOpFn * const fns[3][2] = { - { gen_helper_neon_add_u8, gen_helper_neon_sub_u8 }, - { gen_helper_neon_add_u16, gen_helper_neon_sub_u16 }, - { tcg_gen_add_i32, tcg_gen_sub_i32 }, + if (opcode == 0xf) { + /* SABA, UABA: accumulating ops */ + static NeonGenTwoOpFn * const fns[3] = { + gen_helper_neon_add_u8, + gen_helper_neon_add_u16, + tcg_gen_add_i32, }; - bool is_sub = (opcode == 0x12 && u); /* MLS */ - genfn = fns[size][is_sub]; read_vec_element_i32(s, tcg_op1, rd, pass, MO_32); - genfn(tcg_res, tcg_op1, tcg_res); + fns[size](tcg_res, tcg_op1, tcg_res); } write_vec_element_i32(s, tcg_res, rd, pass, MO_32); From patchwork Tue Nov 21 21:25:31 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 119426 Delivered-To: patch@linaro.org Received: by 10.140.22.164 with SMTP id 33csp5827420qgn; Tue, 21 Nov 2017 13:45:55 -0800 (PST) X-Google-Smtp-Source: AGs4zMYO6zENQbWSvbJsADqKDj/1HWZH+dFgWsXhbVi4BLPFSwaVZJYTU8PTWSci8GfexvlLdp1b X-Received: by 10.37.174.5 with SMTP id a5mr11376331ybj.117.1511300755396; Tue, 21 Nov 2017 13:45:55 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1511300755; cv=none; d=google.com; s=arc-20160816; b=APKUMzfPDgQXabSWlNrMLnXjR4mRySnYu5Y9GozN6OT783uD/TO+WpltolRRLdkYX6 7fT7yrbKPXXmQRMGMpBEr+oaZT8DVG+Fyw5ovmkZqQZDZwwwuNAlOWKOIR9zKUuk+X4p VC13Tri3xXEi5laOR4yQB+C4bnn9un2lC1h+tnic7SQUGsDLhVAko5qXLsVSfxuG6qb/ GbYfxCoT5GTGyGvytLeAoAs+jRxn2XT+vTODXSHxkU144yvldHSJcuGezi3pHPCrEzJf qcfgYN+hcqPCE19eZcHru4nOkldbTLyacBSrcWGXo2YXBanNPIw+hG6AcZBRkRU8KqZa PoDQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:subject:references:in-reply-to :message-id:date:to:from:dkim-signature:arc-authentication-results; bh=gYnP6aABwK31CDcBjJE7j2jOpkRvF+VsSGnxFhhu6xk=; b=0rNJVTlXY5Te+SBz8/7NNLAkrwFcFevNWnx1ipOZYr2w0Ty4QyrpILsveYZgjRimkz qONj3Zdy5Fs5QOP+eCDBNE/YhQ1fOR2rh2+X18sb0qbCkmcpPv7Pq3Pb8HNMkiEI3kl9 5l6vwzMej/GMJyYqhscX6sXMV0mBUaQffyxao59FN07FZwziIouaBwlTnvs1CSE6XDNJ DmCRQeiUtk2XCsNuB0qTx+79kVLys8hOaByQkX6CO+YAVi71pR0gImZLbov/Y7+efp62 Q8bnsr2EVDELX7OnklcmVZ7p1spsrE1z4+XRODpfsiRUIXDpTrIOxLatBC9twcrhsUtn HFUw== ARC-Authentication-Results: i=1; mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=cImrrJGa; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) smtp.mailfrom=qemu-devel-bounces+patch=linaro.org@nongnu.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [2001:4830:134:3::11]) by mx.google.com with ESMTPS id j189si3010453ywc.471.2017.11.21.13.45.55 for (version=TLS1 cipher=AES128-SHA bits=128/128); Tue, 21 Nov 2017 13:45:55 -0800 (PST) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) client-ip=2001:4830:134:3::11; Authentication-Results: mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=cImrrJGa; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) smtp.mailfrom=qemu-devel-bounces+patch=linaro.org@nongnu.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1]:36683 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1eHGMc-0003tG-TY for patch@linaro.org; Tue, 21 Nov 2017 16:45:54 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:54210) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1eHG5R-0004g9-63 for qemu-devel@nongnu.org; Tue, 21 Nov 2017 16:28:11 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1eHG5P-0007qg-Cb for qemu-devel@nongnu.org; Tue, 21 Nov 2017 16:28:09 -0500 Received: from mail-wr0-x241.google.com ([2a00:1450:400c:c0c::241]:35739) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1eHG5P-0007qB-0l for qemu-devel@nongnu.org; Tue, 21 Nov 2017 16:28:07 -0500 Received: by mail-wr0-x241.google.com with SMTP id w95so12649559wrc.2 for ; Tue, 21 Nov 2017 13:28:06 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:subject:date:message-id:in-reply-to:references; bh=gYnP6aABwK31CDcBjJE7j2jOpkRvF+VsSGnxFhhu6xk=; b=cImrrJGaMUIX8HXiMO3UaVkVVNdbwYP2G7m1xiH+xFWAuI68jsHwHUe1rAL6kS/HZs iX0CeQ1TeYHreYVz0wB9p3MuKden2fUGDpn8KjX8BeVLkoXqkcrJjXV3YvvhEEPL0mOP vDhD6cKIpE3lkOSfIlCSJ804S21uZKvdYJHrw= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references; bh=gYnP6aABwK31CDcBjJE7j2jOpkRvF+VsSGnxFhhu6xk=; b=qKWlhz/hXksNxTPpEs8rmNJWCL/DJk+LwgWvi5/RX+bpoVfXWOhqRDlpOl6qQXd/pu y461bkNBDRlKTV5777SSvRnnY1Mq3nHEhZA0c3uGb3QjMLB8S82NTQXIsmFDV/Lokqbj oXgf3k3oJR0ES93HPGpYlMspYc4bu3FcAmnlberONWO1qQ0TATIcJDIiipy1KPWN+81y H3nLw58MNUinsZ7QhK0QlVnHrjwmH2l9J9QrL2YuSAvoiRMCAf5zsTHgU1s0RIxhRcm1 9n5FYFbiiQCfaskyJvsQsH6ngwYn8U8CJvYZQSk8q8L3qieDeYI4ocQWAcCcSyxtVXSz 8pTg== X-Gm-Message-State: AJaThX48zNOeByq1aSsQlecymVZn+t74tur3HxENrDP/v0B/nTz9koJf c+mvWrR8Nfo8f+B51C4X4uixHrD3cJw= X-Received: by 10.223.184.47 with SMTP id h44mr14379361wrf.11.1511299685740; Tue, 21 Nov 2017 13:28:05 -0800 (PST) Received: from cloudburst.twiddle.net (70.red-37-158-60.dynamicip.rima-tde.net. [37.158.60.70]) by smtp.gmail.com with ESMTPSA id e124sm706517wmg.34.2017.11.21.13.28.04 for (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Tue, 21 Nov 2017 13:28:04 -0800 (PST) From: Richard Henderson To: qemu-devel@nongnu.org Date: Tue, 21 Nov 2017 22:25:31 +0100 Message-Id: <20171121212534.5177-24-richard.henderson@linaro.org> X-Mailer: git-send-email 2.13.6 In-Reply-To: <20171121212534.5177-1-richard.henderson@linaro.org> References: <20171121212534.5177-1-richard.henderson@linaro.org> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2a00:1450:400c:c0c::241 Subject: [Qemu-devel] [PATCH v6 23/26] tcg: Add generic vector ops for extension X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: "Qemu-devel" Signed-off-by: Richard Henderson --- accel/tcg/tcg-runtime.h | 8 +++ tcg/i386/tcg-target.h | 2 + tcg/tcg-op-gvec.h | 9 +++ tcg/tcg-op.h | 5 ++ tcg/tcg-opc.h | 5 ++ tcg/tcg.h | 2 + accel/tcg/tcg-runtime-gvec.c | 26 ++++++++ tcg/tcg-op-gvec.c | 138 +++++++++++++++++++++++++++++++++++++++++++ tcg/tcg-op-vec.c | 39 ++++++++++++ tcg/tcg.c | 6 ++ tcg/README | 13 ++++ 11 files changed, 253 insertions(+) -- 2.13.6 diff --git a/accel/tcg/tcg-runtime.h b/accel/tcg/tcg-runtime.h index c4a2e6b215..d1b3542946 100644 --- a/accel/tcg/tcg-runtime.h +++ b/accel/tcg/tcg-runtime.h @@ -199,6 +199,14 @@ DEF_HELPER_FLAGS_4(gvec_trn16, TCG_CALL_NO_RWG, void, ptr, ptr, ptr, i32) DEF_HELPER_FLAGS_4(gvec_trn32, TCG_CALL_NO_RWG, void, ptr, ptr, ptr, i32) DEF_HELPER_FLAGS_4(gvec_trn64, TCG_CALL_NO_RWG, void, ptr, ptr, ptr, i32) +DEF_HELPER_FLAGS_3(gvec_extu8, TCG_CALL_NO_RWG, void, ptr, ptr, i32) +DEF_HELPER_FLAGS_3(gvec_extu16, TCG_CALL_NO_RWG, void, ptr, ptr, i32) +DEF_HELPER_FLAGS_3(gvec_extu32, TCG_CALL_NO_RWG, void, ptr, ptr, i32) + +DEF_HELPER_FLAGS_3(gvec_exts8, TCG_CALL_NO_RWG, void, ptr, ptr, i32) +DEF_HELPER_FLAGS_3(gvec_exts16, TCG_CALL_NO_RWG, void, ptr, ptr, i32) +DEF_HELPER_FLAGS_3(gvec_exts32, TCG_CALL_NO_RWG, void, ptr, ptr, i32) + DEF_HELPER_FLAGS_4(gvec_eq8, TCG_CALL_NO_RWG, void, ptr, ptr, ptr, i32) DEF_HELPER_FLAGS_4(gvec_eq16, TCG_CALL_NO_RWG, void, ptr, ptr, ptr, i32) DEF_HELPER_FLAGS_4(gvec_eq32, TCG_CALL_NO_RWG, void, ptr, ptr, ptr, i32) diff --git a/tcg/i386/tcg-target.h b/tcg/i386/tcg-target.h index 949d138c9d..fedc3449c1 100644 --- a/tcg/i386/tcg-target.h +++ b/tcg/i386/tcg-target.h @@ -185,6 +185,8 @@ extern bool have_avx2; #define TCG_TARGET_HAS_trn_vec 0 #define TCG_TARGET_HAS_cmp_vec 1 #define TCG_TARGET_HAS_mul_vec 0 +#define TCG_TARGET_HAS_extl_vec 0 +#define TCG_TARGET_HAS_exth_vec 0 #define TCG_TARGET_deposit_i32_valid(ofs, len) \ (((ofs) == 0 && (len) == 8) || ((ofs) == 8 && (len) == 8) || \ diff --git a/tcg/tcg-op-gvec.h b/tcg/tcg-op-gvec.h index ad5e22e1bf..188c3368bd 100644 --- a/tcg/tcg-op-gvec.h +++ b/tcg/tcg-op-gvec.h @@ -180,6 +180,15 @@ void tcg_gen_gvec_trne(unsigned vece, uint32_t dofs, uint32_t aofs, void tcg_gen_gvec_trno(unsigned vece, uint32_t dofs, uint32_t aofs, uint32_t bofs, uint32_t opsz, uint32_t clsz); +void tcg_gen_gvec_extul(unsigned vece, uint32_t dofs, uint32_t aofs, + uint32_t opsz, uint32_t clsz); +void tcg_gen_gvec_extuh(unsigned vece, uint32_t dofs, uint32_t aofs, + uint32_t opsz, uint32_t clsz); +void tcg_gen_gvec_extsl(unsigned vece, uint32_t dofs, uint32_t aofs, + uint32_t opsz, uint32_t clsz); +void tcg_gen_gvec_extsh(unsigned vece, uint32_t dofs, uint32_t aofs, + uint32_t opsz, uint32_t clsz); + void tcg_gen_gvec_cmp(TCGCond cond, unsigned vece, uint32_t dofs, uint32_t aofs, uint32_t bofs, uint32_t opsz, uint32_t clsz); diff --git a/tcg/tcg-op.h b/tcg/tcg-op.h index 3296a7baa5..a722c400c2 100644 --- a/tcg/tcg-op.h +++ b/tcg/tcg-op.h @@ -940,6 +940,11 @@ void tcg_gen_uzpo_vec(unsigned vece, TCGv_vec r, TCGv_vec a, TCGv_vec b); void tcg_gen_trne_vec(unsigned vece, TCGv_vec r, TCGv_vec a, TCGv_vec b); void tcg_gen_trno_vec(unsigned vece, TCGv_vec r, TCGv_vec a, TCGv_vec b); +void tcg_gen_extul_vec(unsigned vece, TCGv_vec r, TCGv_vec a); +void tcg_gen_extuh_vec(unsigned vece, TCGv_vec r, TCGv_vec a); +void tcg_gen_extsl_vec(unsigned vece, TCGv_vec r, TCGv_vec a); +void tcg_gen_extsh_vec(unsigned vece, TCGv_vec r, TCGv_vec a); + void tcg_gen_cmp_vec(TCGCond cond, unsigned vece, TCGv_vec r, TCGv_vec a, TCGv_vec b); diff --git a/tcg/tcg-opc.h b/tcg/tcg-opc.h index b21a30273c..3dfd872a0f 100644 --- a/tcg/tcg-opc.h +++ b/tcg/tcg-opc.h @@ -249,6 +249,11 @@ DEF(uzpo_vec, 1, 2, 0, IMPLVEC | IMPL(TCG_TARGET_HAS_uzp_vec)) DEF(trne_vec, 1, 2, 0, IMPLVEC | IMPL(TCG_TARGET_HAS_trn_vec)) DEF(trno_vec, 1, 2, 0, IMPLVEC | IMPL(TCG_TARGET_HAS_trn_vec)) +DEF(extul_vec, 1, 1, 0, IMPLVEC | IMPL(TCG_TARGET_HAS_extl_vec)) +DEF(extuh_vec, 1, 1, 0, IMPLVEC | IMPL(TCG_TARGET_HAS_exth_vec)) +DEF(extsl_vec, 1, 1, 0, IMPLVEC | IMPL(TCG_TARGET_HAS_extl_vec)) +DEF(extsh_vec, 1, 1, 0, IMPLVEC | IMPL(TCG_TARGET_HAS_exth_vec)) + DEF(cmp_vec, 1, 2, 1, IMPLVEC) DEF(last_generic, 0, 0, 0, TCG_OPF_NOT_PRESENT) diff --git a/tcg/tcg.h b/tcg/tcg.h index a729e66b66..b3dae8bcde 100644 --- a/tcg/tcg.h +++ b/tcg/tcg.h @@ -186,6 +186,8 @@ typedef uint64_t TCGRegSet; #define TCG_TARGET_HAS_trn_vec 0 #define TCG_TARGET_HAS_cmp_vec 0 #define TCG_TARGET_HAS_mul_vec 0 +#define TCG_TARGET_HAS_extl_vec 0 +#define TCG_TARGET_HAS_exth_vec 0 #else #define TCG_TARGET_MAYBE_vec 1 #endif diff --git a/accel/tcg/tcg-runtime-gvec.c b/accel/tcg/tcg-runtime-gvec.c index 9406ccd769..ff26be0744 100644 --- a/accel/tcg/tcg-runtime-gvec.c +++ b/accel/tcg/tcg-runtime-gvec.c @@ -588,3 +588,29 @@ DO_CMP2(8) DO_CMP2(16) DO_CMP2(32) DO_CMP2(64) + +#define DO_EXT(NAME, TYPE1, TYPE2) \ +void HELPER(NAME)(void *d, void *a, uint32_t desc) \ +{ \ + intptr_t oprsz = simd_oprsz(desc); \ + intptr_t oprsz_2 = oprsz / 2; \ + intptr_t i; \ + /* We produce output faster than we consume input. \ + Therefore we must be mindful of possible overlap. */ \ + if (unlikely((a - d) < (uintptr_t)oprsz)) { \ + void *a_new = alloca(oprsz_2); \ + memcpy(a_new, a, oprsz_2); \ + a = a_new; \ + } \ + for (i = 0; i < oprsz_2; i += sizeof(TYPE1)) { \ + *(TYPE2 *)(d + 2 * i) = *(TYPE1 *)(a + i); \ + } \ + clear_high(d, oprsz, desc); \ +} + +DO_EXT(gvec_extu8, uint8_t, uint16_t) +DO_EXT(gvec_extu16, uint16_t, uint32_t) +DO_EXT(gvec_extu32, uint32_t, uint64_t) +DO_EXT(gvec_exts8, int8_t, int16_t) +DO_EXT(gvec_exts16, int16_t, int32_t) +DO_EXT(gvec_exts32, int32_t, int64_t) diff --git a/tcg/tcg-op-gvec.c b/tcg/tcg-op-gvec.c index 3f567f0027..ad674693c5 100644 --- a/tcg/tcg-op-gvec.c +++ b/tcg/tcg-op-gvec.c @@ -1870,3 +1870,141 @@ void tcg_gen_gvec_cmp(TCGCond cond, unsigned vece, uint32_t dofs, } tcg_gen_gvec_3_ool(dofs, aofs, bofs, oprsz, maxsz, 0, fn); } + +static void do_ext(unsigned vece, uint32_t dofs, uint32_t aofs, + uint32_t oprsz, uint32_t maxsz, bool high, bool is_sign) +{ + static gen_helper_gvec_2 * const extu_fn[3] = { + gen_helper_gvec_extu8, gen_helper_gvec_extu16, gen_helper_gvec_extu32 + }; + static gen_helper_gvec_2 * const exts_fn[3] = { + gen_helper_gvec_exts8, gen_helper_gvec_exts16, gen_helper_gvec_exts32 + }; + + TCGType type; + uint32_t step, i, n; + TCGOpcode opc; + + check_size_align(oprsz, maxsz, dofs | aofs); + check_overlap_2(dofs, aofs, oprsz); + tcg_debug_assert(vece < MO_64); + + /* Quick check for sizes we won't support inline. */ + if (oprsz > 4 * 32 || maxsz > MAX_UNROLL * 32) { + goto do_ool; + } + + opc = is_sign ? (high ? INDEX_op_extsh_vec : INDEX_op_extsl_vec) + : (high ? INDEX_op_extuh_vec : INDEX_op_extul_vec); + + /* Since these operations don't operate in lock-step lanes, + we must care for overlap. */ + if (TCG_TARGET_HAS_v256 && oprsz % 32 == 0 && oprsz / 32 <= 8 + && tcg_can_emit_vec_op(opc, TCG_TYPE_V256, vece)) { + type = TCG_TYPE_V256; + step = 32; + n = oprsz / 32; + } else if (TCG_TARGET_HAS_v128 && oprsz % 16 == 0 && oprsz / 16 <= 8 + && tcg_can_emit_vec_op(opc, TCG_TYPE_V128, vece)) { + type = TCG_TYPE_V128; + step = 16; + n = oprsz / 16; + } else if (TCG_TARGET_HAS_v64 && oprsz % 8 == 0 && oprsz / 8 <= 8 + && tcg_can_emit_vec_op(opc, TCG_TYPE_V64, vece)) { + type = TCG_TYPE_V64; + step = 8; + n = oprsz / 8; + } else { + goto do_ool; + } + + if (n == 1) { + TCGv_vec t1 = tcg_temp_new_vec(type); + + tcg_gen_ld_vec(t1, cpu_env, aofs); + if (high) { + if (is_sign) { + tcg_gen_extsh_vec(vece, t1, t1); + } else { + tcg_gen_extuh_vec(vece, t1, t1); + } + } else { + if (is_sign) { + tcg_gen_extsl_vec(vece, t1, t1); + } else { + tcg_gen_extul_vec(vece, t1, t1); + } + } + tcg_gen_st_vec(t1, cpu_env, dofs); + tcg_temp_free_vec(t1); + } else { + TCGv_vec ta[4], tmp; + + if (high) { + aofs += oprsz / 2; + } + + for (i = 0; i < (n / 2 + n % 2); ++i) { + ta[i] = tcg_temp_new_vec(type); + tcg_gen_ld_vec(ta[i], cpu_env, aofs + i * step); + } + + tmp = tcg_temp_new_vec(type); + for (i = 0; i < n; ++i) { + if (i & 1) { + if (is_sign) { + tcg_gen_extsh_vec(vece, tmp, ta[i / 2]); + } else { + tcg_gen_extuh_vec(vece, tmp, ta[i / 2]); + } + } else { + if (is_sign) { + tcg_gen_extsl_vec(vece, tmp, ta[i / 2]); + } else { + tcg_gen_extul_vec(vece, tmp, ta[i / 2]); + } + } + tcg_gen_st_vec(tmp, cpu_env, dofs + i * step); + } + tcg_temp_free_vec(tmp); + + for (i = 0; i < (n / 2 + n % 2); ++i) { + tcg_temp_free_vec(ta[i]); + } + } + if (oprsz < maxsz) { + expand_clr(dofs + oprsz, maxsz - oprsz); + } + return; + + do_ool: + if (high) { + aofs += oprsz / 2; + } + tcg_gen_gvec_2_ool(dofs, aofs, oprsz, maxsz, 0, + is_sign ? exts_fn[vece] : extu_fn[vece]); +} + +void tcg_gen_gvec_extul(unsigned vece, uint32_t dofs, uint32_t aofs, + uint32_t oprsz, uint32_t maxsz) +{ + do_ext(vece, dofs, aofs, oprsz, maxsz, false, false); +} + +void tcg_gen_gvec_extuh(unsigned vece, uint32_t dofs, uint32_t aofs, + uint32_t oprsz, uint32_t maxsz) +{ + do_ext(vece, dofs, aofs, oprsz, maxsz, true, false); +} + +void tcg_gen_gvec_extsl(unsigned vece, uint32_t dofs, uint32_t aofs, + uint32_t oprsz, uint32_t maxsz) +{ + do_ext(vece, dofs, aofs, oprsz, maxsz, false, true); +} + +void tcg_gen_gvec_extsh(unsigned vece, uint32_t dofs, uint32_t aofs, + uint32_t oprsz, uint32_t maxsz) +{ + do_ext(vece, dofs, aofs, oprsz, maxsz, true, true); +} diff --git a/tcg/tcg-op-vec.c b/tcg/tcg-op-vec.c index 160db2e26e..c685bb985a 100644 --- a/tcg/tcg-op-vec.c +++ b/tcg/tcg-op-vec.c @@ -525,3 +525,42 @@ void tcg_gen_mul_vec(unsigned vece, TCGv_vec r, TCGv_vec a, TCGv_vec b) tcg_expand_vec_op(INDEX_op_mul_vec, type, vece, ri, ai, bi); } } + +static void do_ext(TCGOpcode opc, unsigned vece, TCGv_vec r, TCGv_vec a) +{ + TCGTemp *rt = tcgv_vec_temp(r); + TCGTemp *at = tcgv_vec_temp(a); + TCGArg ri = temp_arg(rt); + TCGArg ai = temp_arg(at); + TCGType type = rt->base_type; + int can; + + tcg_debug_assert(at->base_type == type); + can = tcg_can_emit_vec_op(opc, type, vece); + if (can > 0) { + vec_gen_2(opc, type, vece, ri, ai); + } else { + tcg_debug_assert(can < 0); + tcg_expand_vec_op(opc, type, vece, ri, ai); + } +} + +void tcg_gen_extul_vec(unsigned vece, TCGv_vec r, TCGv_vec a) +{ + do_ext(INDEX_op_extul_vec, vece, r, a); +} + +void tcg_gen_extuh_vec(unsigned vece, TCGv_vec r, TCGv_vec a) +{ + do_ext(INDEX_op_extuh_vec, vece, r, a); +} + +void tcg_gen_extsl_vec(unsigned vece, TCGv_vec r, TCGv_vec a) +{ + do_ext(INDEX_op_extsl_vec, vece, r, a); +} + +void tcg_gen_extsh_vec(unsigned vece, TCGv_vec r, TCGv_vec a) +{ + do_ext(INDEX_op_extsh_vec, vece, r, a); +} diff --git a/tcg/tcg.c b/tcg/tcg.c index b0a7cb7d38..0130c42994 100644 --- a/tcg/tcg.c +++ b/tcg/tcg.c @@ -1427,6 +1427,12 @@ bool tcg_op_supported(TCGOpcode op) case INDEX_op_trne_vec: case INDEX_op_trno_vec: return have_vec && TCG_TARGET_HAS_trn_vec; + case INDEX_op_extul_vec: + case INDEX_op_extsl_vec: + return have_vec && TCG_TARGET_HAS_extl_vec; + case INDEX_op_extuh_vec: + case INDEX_op_extsh_vec: + return have_vec && TCG_TARGET_HAS_exth_vec; default: tcg_debug_assert(op > INDEX_op_last_generic && op < NB_OPS); diff --git a/tcg/README b/tcg/README index 17695ff7f6..56c70764bc 100644 --- a/tcg/README +++ b/tcg/README @@ -634,6 +634,19 @@ E.g. VECL=1 -> 64 << 1 -> v128, and VECE=2 -> 1 << 2 -> i32. v0[2i + 1] = v2[2i + part]; } +* extul_vec v0, v1 + + Extend unsigned the low VECL/VECE/2 elements of v1 into v0. + +* extuh_vec v0, v1 + + Similarly for the high VECL/VECE/2 elements. + +* extsl_vec v0, v1 +* extsh_vec v0, v1 + + Similarly with signed extension. + * cmp_vec v0, v1, v2, cond Compare vectors by element, storing -1 for true and 0 for false. From patchwork Tue Nov 21 21:25:32 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 119422 Delivered-To: patch@linaro.org Received: by 10.140.22.164 with SMTP id 33csp5824732qgn; Tue, 21 Nov 2017 13:42:47 -0800 (PST) X-Google-Smtp-Source: AGs4zMa0N/J1G45qYEN9uPJJ27Ad9A0PEpSeWa6LeWtl4DLuqNft71kH/1Xw30V5SvQ6SQPUFZnw X-Received: by 10.129.118.17 with SMTP id r17mr12604156ywc.497.1511300567432; Tue, 21 Nov 2017 13:42:47 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1511300567; cv=none; d=google.com; s=arc-20160816; b=t9cjJdU7/SzUADv7wLu5tge97vD3prZdftGQ9OVFKJ9Ul87+3LJBN8MvgQT1lyJS24 ZDhslMDCYm5v/ix8nkbRxeP8mj8s4mLf1OwKfIXXND5mADizI+iWVm1AmHBvtzsnwLXE jTfwg2Rk0erbcmglLIxKmDKUTcIAOiiQ/29odvp3Sxo9wJRkcfcAIdrbQyIRbOSXo4JU 6qhLC4vmJjSGgp3Fia6ENVBeoYKtj/CtKuOEdWNYN2HfBB5RDyElF82lVPZWAu0CJ4xp FCwVS75i1s4i8jbQXe5ygUDviHQ4yPW5nUJCpJL5+GZiZM9pOSZKDOYepZBSV/t83ebG zdhQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:subject:references:in-reply-to :message-id:date:to:from:dkim-signature:arc-authentication-results; bh=UfWLBriMEwBCkjQxTbR502Q73I7ejTcSkyhpjba8iuA=; b=DBtwuEqk6osp+V5Fp5qMRhK4FIsO/61gYpwotd0FVya3OT9KOxHSdO3rlsWRMyVe0U R6LzzkxfG+ah91It94P5U9He4ESsKFZUsyf+3nTVZlgvxQRyGXu65+/IMFlNrRxrG4im 2L24xhEs+48PX3U1Xe9Pa2vl7PS4Tqy+vh853mAB2HI88GaOKV0TFMRHYUB0quxxkbZ3 EJOsN2sBlSf691WcBmmVhAz+TY72y1Z09G5TZw0pXQsEog4Me2GopZMrwMLHaA26dcbp XWQxfFGfoEIMwhnM8PEZNDCjw94njCuGOo+klHW4rWwHbgoQprPCew0Ou247DLAE/0Jz RDgg== ARC-Authentication-Results: i=1; mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=PYVZ553Z; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) smtp.mailfrom=qemu-devel-bounces+patch=linaro.org@nongnu.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [2001:4830:134:3::11]) by mx.google.com with ESMTPS id j17si2962009ybm.170.2017.11.21.13.42.47 for (version=TLS1 cipher=AES128-SHA bits=128/128); Tue, 21 Nov 2017 13:42:47 -0800 (PST) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) client-ip=2001:4830:134:3::11; Authentication-Results: mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=PYVZ553Z; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) smtp.mailfrom=qemu-devel-bounces+patch=linaro.org@nongnu.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1]:36651 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1eHGJa-0008D6-R1 for patch@linaro.org; Tue, 21 Nov 2017 16:42:46 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:54212) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1eHG5R-0004gB-6d for qemu-devel@nongnu.org; Tue, 21 Nov 2017 16:28:10 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1eHG5Q-0007qx-94 for qemu-devel@nongnu.org; Tue, 21 Nov 2017 16:28:09 -0500 Received: from mail-wr0-x244.google.com ([2a00:1450:400c:c0c::244]:34090) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1eHG5Q-0007qm-13 for qemu-devel@nongnu.org; Tue, 21 Nov 2017 16:28:08 -0500 Received: by mail-wr0-x244.google.com with SMTP id k18so7664007wre.1 for ; Tue, 21 Nov 2017 13:28:07 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:subject:date:message-id:in-reply-to:references; bh=UfWLBriMEwBCkjQxTbR502Q73I7ejTcSkyhpjba8iuA=; b=PYVZ553Z7NWoK/QYwEUstXrN8JyPykS6PLa+BlRvrpsx1xTA0KjLESuyw+wJCdBmij Up9zjUpwp0aA9TDvilXlhP040cSN8cuskguFz07hkQ2oG9pTlRY87JDKzYNcL4rx6bBs c8+5dCqWj/jxOpPmFTu8HvRmJkyeogEKZDl6M= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references; bh=UfWLBriMEwBCkjQxTbR502Q73I7ejTcSkyhpjba8iuA=; b=J8eCicVjFmg5yw8nn0jqWfFE1sUUdW+Yo2FrI4Jf2z4iLQLMN0mRteEmWP+nu8qIyZ LXESAhhKu3xUJBh0rANEW0qD8DruxkcQfOeCswsMvdSwFika2Kgv3gjmMqFMPDLY6Jir fPNfgq3LGCubJ5K6kuRSq+M2I0xh7KI9ZJuGpydPmlQOceO4df3QvEpaVqpy2ZyN+lE/ AF7R1JbzbEpdZJB9bTtnfNrb9VtYyETIHMVPhHMBJMH1a+tqZHCmUVIuPt9S572sAF77 ldyV9alrDLg8Jc+KSqJd44lw/GMc6AJFQZHuk8px6HLrrpNt6WwZFdEQmxHstIWpIHCO nfhQ== X-Gm-Message-State: AJaThX7yYucs2KFii9dRnj3vERJ1fiD5AykL7K9XTxUHDgTbpkeUZiQp 2XnyQOpQuZfAqTiuCT5XdCGThfFyH/I= X-Received: by 10.223.142.143 with SMTP id q15mr10089196wrb.6.1511299686836; Tue, 21 Nov 2017 13:28:06 -0800 (PST) Received: from cloudburst.twiddle.net (70.red-37-158-60.dynamicip.rima-tde.net. [37.158.60.70]) by smtp.gmail.com with ESMTPSA id e124sm706517wmg.34.2017.11.21.13.28.05 for (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Tue, 21 Nov 2017 13:28:06 -0800 (PST) From: Richard Henderson To: qemu-devel@nongnu.org Date: Tue, 21 Nov 2017 22:25:32 +0100 Message-Id: <20171121212534.5177-25-richard.henderson@linaro.org> X-Mailer: git-send-email 2.13.6 In-Reply-To: <20171121212534.5177-1-richard.henderson@linaro.org> References: <20171121212534.5177-1-richard.henderson@linaro.org> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2a00:1450:400c:c0c::244 Subject: [Qemu-devel] [PATCH v6 24/26] target/arm: Use vector infrastructure for aa64 widening shifts X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: "Qemu-devel" Signed-off-by: Richard Henderson --- target/arm/translate-a64.c | 29 ++++++++++++----------------- 1 file changed, 12 insertions(+), 17 deletions(-) -- 2.13.6 diff --git a/target/arm/translate-a64.c b/target/arm/translate-a64.c index c47d9caa49..1f7e9c4e19 100644 --- a/target/arm/translate-a64.c +++ b/target/arm/translate-a64.c @@ -8705,12 +8705,7 @@ static void handle_vec_simd_wshli(DisasContext *s, bool is_q, bool is_u, int size = 32 - clz32(immh) - 1; int immhb = immh << 3 | immb; int shift = immhb - (8 << size); - int dsize = 64; - int esize = 8 << size; - int elements = dsize/esize; - TCGv_i64 tcg_rn = new_tmp_a64(s); - TCGv_i64 tcg_rd = new_tmp_a64(s); - int i; + GVecGen2Fn *gvec_fn; if (size >= 3) { unallocated_encoding(s); @@ -8721,18 +8716,18 @@ static void handle_vec_simd_wshli(DisasContext *s, bool is_q, bool is_u, return; } - /* For the LL variants the store is larger than the load, - * so if rd == rn we would overwrite parts of our input. - * So load everything right now and use shifts in the main loop. - */ - read_vec_element(s, tcg_rn, rn, is_q ? 1 : 0, MO_64); - - for (i = 0; i < elements; i++) { - tcg_gen_shri_i64(tcg_rd, tcg_rn, i * esize); - ext_and_shift_reg(tcg_rd, tcg_rd, size | (!is_u << 2), 0); - tcg_gen_shli_i64(tcg_rd, tcg_rd, shift); - write_vec_element(s, tcg_rd, rd, i, size + 1); + if (is_u) { + gvec_fn = is_q ? tcg_gen_gvec_extuh : tcg_gen_gvec_extul; + } else { + gvec_fn = is_q ? tcg_gen_gvec_extsh : tcg_gen_gvec_extsl; } + gvec_fn(size, vec_full_reg_offset(s, rd), + vec_full_reg_offset(s, rn), 16, 16); + + /* Perform the shift in the wider format. */ + tcg_gen_gvec_shli(size + 1, vec_full_reg_offset(s, rd), + vec_full_reg_offset(s, rd), + 16, vec_full_reg_size(s), shift); } /* SHRN/RSHRN - Shift right with narrowing (and potential rounding) */ From patchwork Tue Nov 21 21:25:33 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 119423 Delivered-To: patch@linaro.org Received: by 10.140.22.164 with SMTP id 33csp5825114qgn; Tue, 21 Nov 2017 13:43:14 -0800 (PST) X-Google-Smtp-Source: AGs4zMbcg5XBKgeFjziBKW8/mIz8+aJepruXz8CorrDxjiU1PA/AI30HBA0zWEAARMa8vFc8xvNg X-Received: by 10.129.141.5 with SMTP id d5mr11560759ywg.184.1511300594547; Tue, 21 Nov 2017 13:43:14 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1511300594; cv=none; d=google.com; s=arc-20160816; b=d8twsjr9uVZ3BLKx3EY/g8VlS5X0+8ziIDcpQpOg33vczYzpdRgLFWOExcJQFcyERY Iu2Amfmm3A3F1fXfIEpxVcbqiR2wlxacjnymJ3QyuJ+Uj0V5ksccywJcZDvuP97VeGc+ 6CBacO4mJwz1JJpI9uA5KNsCsuhlEEhzVsUZY06ueaNjtTBeU0wbDpLXKuHIDcwwctfc nMklHJzgsNlr0UGUB0XHsaPik0PO3bMifrPTKVlgLqAvavLz1qVqmJhSzKS5E2NIQHd8 +W/UHF2j+IurBXQGXqVXWfzCFzafWxWEFm5cxapxmrqyJz6N6+0sJLsXo6Q4drEFPedR JhZQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:subject:references:in-reply-to :message-id:date:to:from:dkim-signature:arc-authentication-results; bh=7EK0FNhRILTkyI03yMIiLDt55Nl6NalvbeMzTdjaFsk=; b=zFaIuc44vLxCKjwzLLA5gIk4scFyGb9ErVwwd5bHD0nZCSd1vxhkyX9d1/ILsbY+jR H95yBUWJzbNLX69S6HLh82magsEhj6Yx6CI8LfEjwqG6YCQb/CSilrDHb7ya/obi0Rqb mIAhBsMISegfEarsflJj9t3KkL/XGhIQP6pvNBn5jsuS3uuOs/OyN0oqRsFqo1NR3qhn Y4C+YFXMVjtlg2KRX69AhfF/aaClbG3z+h3vZxML3vo3XP0HWxeS/jfzO2YHRrrqyZxo fmNE2ELdYhBfrVQZbDlpM8BRFRjZLX60ObEu9Wsib/S13X1MCG/0s0/YbS8qYwFgU9A1 jXyA== ARC-Authentication-Results: i=1; mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=OE8H7JqN; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) smtp.mailfrom=qemu-devel-bounces+patch=linaro.org@nongnu.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [2001:4830:134:3::11]) by mx.google.com with ESMTPS id x124si3117651ywg.751.2017.11.21.13.43.14 for (version=TLS1 cipher=AES128-SHA bits=128/128); Tue, 21 Nov 2017 13:43:14 -0800 (PST) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) client-ip=2001:4830:134:3::11; Authentication-Results: mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=OE8H7JqN; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) smtp.mailfrom=qemu-devel-bounces+patch=linaro.org@nongnu.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1]:36659 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1eHGK2-0001ZD-3i for patch@linaro.org; Tue, 21 Nov 2017 16:43:14 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:54252) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1eHG5T-0004iQ-GO for qemu-devel@nongnu.org; Tue, 21 Nov 2017 16:28:17 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1eHG5R-0007ro-Px for qemu-devel@nongnu.org; Tue, 21 Nov 2017 16:28:11 -0500 Received: from mail-wr0-x242.google.com ([2a00:1450:400c:c0c::242]:41309) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1eHG5R-0007rA-Gx for qemu-devel@nongnu.org; Tue, 21 Nov 2017 16:28:09 -0500 Received: by mail-wr0-x242.google.com with SMTP id z14so12610528wrb.8 for ; Tue, 21 Nov 2017 13:28:09 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:subject:date:message-id:in-reply-to:references; bh=7EK0FNhRILTkyI03yMIiLDt55Nl6NalvbeMzTdjaFsk=; b=OE8H7JqNWCnixLgCrSG1xWtrGb1F+BwleqSY+YGgNDfoaahG4BToBJCrllt7qFylvd Dll0BexOu7BkJoEXkCMJwZ0lPblVj3FV4ZozVHt6Dlt4klDY94gQMJBWnjUkQDBnrPYl quMRyalX0TpBcbYdQ+ZjrNy4x2NGygHGjpPe4= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references; bh=7EK0FNhRILTkyI03yMIiLDt55Nl6NalvbeMzTdjaFsk=; b=DrUwiBhWEjOOqByfCmtsdZuNt/rv41EOT0dkZo+IZ4DzsyMdTuHa90SgB/WPvctHFn Ga0INwFs+s1MwE3MCRABI1UlH3zV0GXHipf0l02Lh6Ko04hWMGxie5j2zjpVLnpkysNj lSPii8dyIyZfTCoCD/qZJbdBwb2XL1h89/cIM4FQBqYbLztLYrLCchTjZLo2TsXoebpc eYGh+mxoekSSreiXEVlaFDVrYmXQcp9L56QCEBotIgaQr83S40rpuaRGbCpGJMAB6NaK IjpKymMRNn5lGeaUyI2JqHI5rkhiHWhrH8/xXQ2rXGVTXo5nNGPHjJID8fWK3VNjwS0v FNBA== X-Gm-Message-State: AJaThX6X+goXlhJY7DM4sijBHy3YdROHd7vccNDACNL9wc7A3sZiCva8 v33ke3VD552j50Kw+3rJJ0jobna5vaw= X-Received: by 10.223.185.35 with SMTP id k32mr16276388wrf.50.1511299688216; Tue, 21 Nov 2017 13:28:08 -0800 (PST) Received: from cloudburst.twiddle.net (70.red-37-158-60.dynamicip.rima-tde.net. [37.158.60.70]) by smtp.gmail.com with ESMTPSA id e124sm706517wmg.34.2017.11.21.13.28.07 for (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Tue, 21 Nov 2017 13:28:07 -0800 (PST) From: Richard Henderson To: qemu-devel@nongnu.org Date: Tue, 21 Nov 2017 22:25:33 +0100 Message-Id: <20171121212534.5177-26-richard.henderson@linaro.org> X-Mailer: git-send-email 2.13.6 In-Reply-To: <20171121212534.5177-1-richard.henderson@linaro.org> References: <20171121212534.5177-1-richard.henderson@linaro.org> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2a00:1450:400c:c0c::242 Subject: [Qemu-devel] [PATCH v6 25/26] tcg/i386: Add vector operations/expansions for mul/extend X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: "Qemu-devel" Signed-off-by: Richard Henderson --- tcg/i386/tcg-target.h | 4 +- tcg/i386/tcg-target.opc.h | 1 + tcg/i386/tcg-target.inc.c | 186 ++++++++++++++++++++++++++++++++++++++++++---- 3 files changed, 174 insertions(+), 17 deletions(-) -- 2.13.6 diff --git a/tcg/i386/tcg-target.h b/tcg/i386/tcg-target.h index fedc3449c1..e77b95cc2c 100644 --- a/tcg/i386/tcg-target.h +++ b/tcg/i386/tcg-target.h @@ -184,8 +184,8 @@ extern bool have_avx2; #define TCG_TARGET_HAS_uzp_vec 0 #define TCG_TARGET_HAS_trn_vec 0 #define TCG_TARGET_HAS_cmp_vec 1 -#define TCG_TARGET_HAS_mul_vec 0 -#define TCG_TARGET_HAS_extl_vec 0 +#define TCG_TARGET_HAS_mul_vec 1 +#define TCG_TARGET_HAS_extl_vec 1 #define TCG_TARGET_HAS_exth_vec 0 #define TCG_TARGET_deposit_i32_valid(ofs, len) \ diff --git a/tcg/i386/tcg-target.opc.h b/tcg/i386/tcg-target.opc.h index 77125ef818..5f05df65e0 100644 --- a/tcg/i386/tcg-target.opc.h +++ b/tcg/i386/tcg-target.opc.h @@ -8,3 +8,4 @@ DEF(x86_blend_vec, 1, 2, 1, IMPLVEC) DEF(x86_packss_vec, 1, 2, 0, IMPLVEC) DEF(x86_packus_vec, 1, 2, 0, IMPLVEC) DEF(x86_psrldq_vec, 1, 1, 1, IMPLVEC) +DEF(x86_vperm2i128_vec, 1, 2, 1, IMPLVEC) diff --git a/tcg/i386/tcg-target.inc.c b/tcg/i386/tcg-target.inc.c index 694d9e5cb5..e61aeebf3e 100644 --- a/tcg/i386/tcg-target.inc.c +++ b/tcg/i386/tcg-target.inc.c @@ -393,6 +393,14 @@ static inline int tcg_target_const_match(tcg_target_long val, TCGType type, #define OPC_PCMPGTW (0x65 | P_EXT | P_DATA16) #define OPC_PCMPGTD (0x66 | P_EXT | P_DATA16) #define OPC_PCMPGTQ (0x37 | P_EXT38 | P_DATA16) +#define OPC_PMOVSXBW (0x20 | P_EXT38 | P_DATA16) +#define OPC_PMOVSXWD (0x23 | P_EXT38 | P_DATA16) +#define OPC_PMOVSXDQ (0x25 | P_EXT38 | P_DATA16) +#define OPC_PMOVZXBW (0x30 | P_EXT38 | P_DATA16) +#define OPC_PMOVZXWD (0x33 | P_EXT38 | P_DATA16) +#define OPC_PMOVZXDQ (0x35 | P_EXT38 | P_DATA16) +#define OPC_PMULLW (0xd5 | P_EXT | P_DATA16) +#define OPC_PMULLD (0x40 | P_EXT38 | P_DATA16) #define OPC_POR (0xeb | P_EXT | P_DATA16) #define OPC_PSHUFB (0x00 | P_EXT38 | P_DATA16) #define OPC_PSHUFD (0x70 | P_EXT | P_DATA16) @@ -2675,6 +2683,9 @@ static void tcg_out_vec_op(TCGContext *s, TCGOpcode opc, static int const sub_insn[4] = { OPC_PSUBB, OPC_PSUBW, OPC_PSUBD, OPC_PSUBQ }; + static int const mul_insn[4] = { + OPC_UD2, OPC_PMULLW, OPC_PMULLD, OPC_UD2 + }; static int const shift_imm_insn[4] = { OPC_UD2, OPC_PSHIFTW_Ib, OPC_PSHIFTD_Ib, OPC_PSHIFTQ_Ib }; @@ -2690,6 +2701,18 @@ static void tcg_out_vec_op(TCGContext *s, TCGOpcode opc, static int const punpckh_insn[4] = { OPC_PUNPCKHBW, OPC_PUNPCKHWD, OPC_PUNPCKHDQ, OPC_PUNPCKHQDQ }; + static int const packss_insn[4] = { + OPC_PACKSSWB, OPC_PACKSSDW, OPC_UD2, OPC_UD2 + }; + static int const packus_insn[4] = { + OPC_PACKUSWB, OPC_PACKUSDW, OPC_UD2, OPC_UD2 + }; + static int const pmovsx_insn[3] = { + OPC_PMOVSXBW, OPC_PMOVSXWD, OPC_PMOVSXDQ + }; + static int const pmovzx_insn[3] = { + OPC_PMOVZXBW, OPC_PMOVZXWD, OPC_PMOVZXDQ + }; TCGType type = vecl + TCG_TYPE_V64; int insn, sub; @@ -2706,6 +2729,9 @@ static void tcg_out_vec_op(TCGContext *s, TCGOpcode opc, case INDEX_op_sub_vec: insn = sub_insn[vece]; goto gen_simd; + case INDEX_op_mul_vec: + insn = mul_insn[vece]; + goto gen_simd; case INDEX_op_and_vec: insn = OPC_PAND; goto gen_simd; @@ -2722,30 +2748,33 @@ static void tcg_out_vec_op(TCGContext *s, TCGOpcode opc, insn = punpckh_insn[vece]; goto gen_simd; case INDEX_op_x86_packss_vec: - if (vece == MO_8) { - insn = OPC_PACKSSWB; - } else if (vece == MO_16) { - insn = OPC_PACKSSDW; - } else { - g_assert_not_reached(); - } + insn = packss_insn[vece]; goto gen_simd; case INDEX_op_x86_packus_vec: - if (vece == MO_8) { - insn = OPC_PACKUSWB; - } else if (vece == MO_16) { - insn = OPC_PACKUSDW; - } else { - g_assert_not_reached(); - } + insn = packus_insn[vece]; goto gen_simd; gen_simd: + tcg_debug_assert(insn != OPC_UD2); if (type == TCG_TYPE_V256) { insn |= P_VEXL; } tcg_out_vex_modrm(s, insn, a0, a1, a2); break; + case INDEX_op_extsl_vec: + insn = pmovsx_insn[vece]; + goto gen_simd2; + case INDEX_op_extul_vec: + insn = pmovzx_insn[vece]; + goto gen_simd2; + gen_simd2: + tcg_debug_assert(vece < MO_64); + if (type == TCG_TYPE_V256) { + insn |= P_VEXL; + } + tcg_out_vex_modrm(s, insn, a0, 0, a1); + break; + case INDEX_op_cmp_vec: sub = args[3]; if (sub == TCG_COND_EQ) { @@ -2811,6 +2840,10 @@ static void tcg_out_vec_op(TCGContext *s, TCGOpcode opc, } sub = args[3]; goto gen_simd_imm8; + case INDEX_op_x86_vperm2i128_vec: + insn = OPC_VPERM2I128; + sub = args[3]; + goto gen_simd_imm8; gen_simd_imm8: if (type == TCG_TYPE_V256) { insn |= P_VEXL; @@ -3073,6 +3106,7 @@ static const TCGTargetOpDef *tcg_target_op_def(TCGOpcode op) case INDEX_op_add_vec: case INDEX_op_sub_vec: + case INDEX_op_mul_vec: case INDEX_op_and_vec: case INDEX_op_or_vec: case INDEX_op_xor_vec: @@ -3084,11 +3118,14 @@ static const TCGTargetOpDef *tcg_target_op_def(TCGOpcode op) case INDEX_op_x86_blend_vec: case INDEX_op_x86_packss_vec: case INDEX_op_x86_packus_vec: + case INDEX_op_x86_vperm2i128_vec: return &x_x_x; case INDEX_op_dup_vec: case INDEX_op_shli_vec: case INDEX_op_shri_vec: case INDEX_op_sari_vec: + case INDEX_op_extsl_vec: + case INDEX_op_extul_vec: case INDEX_op_x86_psrldq_vec: return &x_x; case INDEX_op_x86_vpblendvb_vec: @@ -3109,8 +3146,12 @@ int tcg_can_emit_vec_op(TCGOpcode opc, TCGType type, unsigned vece) case INDEX_op_or_vec: case INDEX_op_xor_vec: case INDEX_op_andc_vec: + case INDEX_op_extsl_vec: + case INDEX_op_extul_vec: return 1; case INDEX_op_cmp_vec: + case INDEX_op_extsh_vec: + case INDEX_op_extuh_vec: return -1; case INDEX_op_shli_vec: @@ -3130,6 +3171,16 @@ int tcg_can_emit_vec_op(TCGOpcode opc, TCGType type, unsigned vece) } return 1; + case INDEX_op_mul_vec: + if (vece == MO_8) { + /* We can expand the operation for MO_8. */ + return -1; + } + if (vece == MO_64) { + return 0; + } + return 1; + case INDEX_op_zipl_vec: /* We could support v256, but with 3 insns per opcode. It is better to expand with v128 instead. */ @@ -3157,7 +3208,7 @@ void tcg_expand_vec_op(TCGOpcode opc, TCGType type, unsigned vece, { va_list va; TCGArg a1, a2; - TCGv_vec v0, v1, v2, t1, t2; + TCGv_vec v0, v1, v2, t1, t2, t3, t4; va_start(va, a0); v0 = temp_tcgv_vec(arg_temp(a0)); @@ -3248,6 +3299,91 @@ void tcg_expand_vec_op(TCGOpcode opc, TCGType type, unsigned vece, tcg_temp_free_vec(t1); break; + case INDEX_op_mul_vec: + tcg_debug_assert(vece == MO_8); + a1 = va_arg(va, TCGArg); + a2 = va_arg(va, TCGArg); + switch (type) { + case TCG_TYPE_V64: + t1 = tcg_temp_new_vec(TCG_TYPE_V128); + t2 = tcg_temp_new_vec(TCG_TYPE_V128); + tcg_gen_dup16i_vec(t2, 0); + vec_gen_3(INDEX_op_zipl_vec, TCG_TYPE_V128, MO_8, + tcgv_vec_arg(t1), a1, tcgv_vec_arg(t2)); + vec_gen_3(INDEX_op_zipl_vec, TCG_TYPE_V128, MO_8, + tcgv_vec_arg(t2), tcgv_vec_arg(t2), a2); + tcg_gen_mul_vec(MO_16, t1, t1, t2); + tcg_gen_shri_vec(MO_16, t1, t1, 8); + vec_gen_3(INDEX_op_x86_packus_vec, TCG_TYPE_V128, MO_8, + a0, tcgv_vec_arg(t1), tcgv_vec_arg(t1)); + tcg_temp_free_vec(t1); + tcg_temp_free_vec(t2); + break; + + case TCG_TYPE_V128: + t1 = tcg_temp_new_vec(TCG_TYPE_V128); + t2 = tcg_temp_new_vec(TCG_TYPE_V128); + t3 = tcg_temp_new_vec(TCG_TYPE_V128); + t4 = tcg_temp_new_vec(TCG_TYPE_V128); + tcg_gen_dup16i_vec(t4, 0); + vec_gen_3(INDEX_op_zipl_vec, TCG_TYPE_V128, MO_8, + tcgv_vec_arg(t1), a1, tcgv_vec_arg(t4)); + vec_gen_3(INDEX_op_zipl_vec, TCG_TYPE_V128, MO_8, + tcgv_vec_arg(t2), tcgv_vec_arg(t4), a2); + vec_gen_3(INDEX_op_ziph_vec, TCG_TYPE_V128, MO_8, + tcgv_vec_arg(t3), a1, tcgv_vec_arg(t4)); + vec_gen_3(INDEX_op_ziph_vec, TCG_TYPE_V128, MO_8, + tcgv_vec_arg(t4), tcgv_vec_arg(t4), a2); + tcg_gen_mul_vec(MO_16, t1, t1, t2); + tcg_gen_mul_vec(MO_16, t3, t3, t4); + tcg_gen_shri_vec(MO_16, t1, t1, 8); + tcg_gen_shri_vec(MO_16, t3, t3, 8); + vec_gen_3(INDEX_op_x86_packus_vec, TCG_TYPE_V128, MO_8, + a0, tcgv_vec_arg(t1), tcgv_vec_arg(t3)); + tcg_temp_free_vec(t1); + tcg_temp_free_vec(t2); + tcg_temp_free_vec(t3); + tcg_temp_free_vec(t4); + break; + + case TCG_TYPE_V256: + t1 = tcg_temp_new_vec(TCG_TYPE_V256); + t2 = tcg_temp_new_vec(TCG_TYPE_V256); + t3 = tcg_temp_new_vec(TCG_TYPE_V256); + t4 = tcg_temp_new_vec(TCG_TYPE_V256); + tcg_gen_dup16i_vec(t4, 0); + /* a1: A[0-7] ... D[0-7]; a2: W[0-7] ... Z[0-7] + t1: extends of B[0-7], D[0-7] + t2: extends of X[0-7], Z[0-7] + t3: extends of A[0-7], C[0-7] + t4: extends of W[0-7], Y[0-7]. */ + vec_gen_3(INDEX_op_zipl_vec, TCG_TYPE_V256, MO_8, + tcgv_vec_arg(t1), a1, tcgv_vec_arg(t4)); + vec_gen_3(INDEX_op_zipl_vec, TCG_TYPE_V256, MO_8, + tcgv_vec_arg(t2), tcgv_vec_arg(t4), a2); + vec_gen_3(INDEX_op_ziph_vec, TCG_TYPE_V256, MO_8, + tcgv_vec_arg(t3), a1, tcgv_vec_arg(t4)); + vec_gen_3(INDEX_op_ziph_vec, TCG_TYPE_V256, MO_8, + tcgv_vec_arg(t4), tcgv_vec_arg(t4), a2); + /* t1: BX DZ; t2: AW CY. */ + tcg_gen_mul_vec(MO_16, t1, t1, t2); + tcg_gen_mul_vec(MO_16, t3, t3, t4); + tcg_gen_shri_vec(MO_16, t1, t1, 8); + tcg_gen_shri_vec(MO_16, t3, t3, 8); + /* a0: AW BX CY DZ. */ + vec_gen_3(INDEX_op_x86_packus_vec, TCG_TYPE_V256, MO_8, + a0, tcgv_vec_arg(t1), tcgv_vec_arg(t3)); + tcg_temp_free_vec(t1); + tcg_temp_free_vec(t2); + tcg_temp_free_vec(t3); + tcg_temp_free_vec(t4); + break; + + default: + g_assert_not_reached(); + } + break; + case INDEX_op_ziph_vec: tcg_debug_assert(type == TCG_TYPE_V64); a1 = va_arg(va, TCGArg); @@ -3256,6 +3392,26 @@ void tcg_expand_vec_op(TCGOpcode opc, TCGType type, unsigned vece, vec_gen_3(INDEX_op_x86_psrldq_vec, TCG_TYPE_V128, MO_64, a0, a0, 8); break; + case INDEX_op_extsh_vec: + case INDEX_op_extuh_vec: + a1 = va_arg(va, TCGArg); + switch (type) { + case TCG_TYPE_V64: + vec_gen_3(INDEX_op_x86_psrldq_vec, type, MO_64, a0, a1, 4); + break; + case TCG_TYPE_V128: + vec_gen_3(INDEX_op_x86_psrldq_vec, type, MO_64, a0, a1, 8); + break; + case TCG_TYPE_V256: + vec_gen_4(INDEX_op_x86_vperm2i128_vec, type, 4, a0, a1, a1, 0x81); + break; + default: + g_assert_not_reached(); + } + vec_gen_2(opc == INDEX_op_extsh_vec ? INDEX_op_extsl_vec + : INDEX_op_extul_vec, type, vece, a0, a0); + break; + case INDEX_op_uzpe_vec: a1 = va_arg(va, TCGArg); a2 = va_arg(va, TCGArg); From patchwork Tue Nov 21 21:25:34 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 119416 Delivered-To: patch@linaro.org Received: by 10.140.22.164 with SMTP id 33csp5820231qgn; Tue, 21 Nov 2017 13:37:49 -0800 (PST) X-Google-Smtp-Source: AGs4zMa9I498I1rnoAVtDWP2A7ELdKij0x17h+QduPX5muUzgazW9yvNpd5CP/5zYlSa8ssFUYf1 X-Received: by 10.37.228.193 with SMTP id b184mr11523576ybh.342.1511300269055; Tue, 21 Nov 2017 13:37:49 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1511300269; cv=none; d=google.com; s=arc-20160816; b=KuhYbi+7mF05393AzDolEh9KzsGZZWyFqjFzaXU49UD7ezcx03/yS35DoYjuO8Q+tv Yxqh4wlr/glpjYOtBQzGkeBtDKi6gps7h1f6MF7+2+pKc7oqgiXZwhj04LmjyNP+6Loj Au1ysbmhDbQX9+yW99/eLlxVoMQ4tp9L8Y+ZMDAs2Y8P3/l83r5lz2nITZHNDXVHE+kv AM0/EKVSX5PliW7AOkp+ccGN5qCzEoiwxHj3IuxxNJovb6cyuSPi/IRriDNFJNqIZgPz E/OUbg1uvfmAkwtMPaEyNSAq2yJhBXskgtVKJ5G3j1LgZrR2vjrn8efFq23pgclPoC7Q 6Qmw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:subject:references:in-reply-to :message-id:date:to:from:dkim-signature:arc-authentication-results; bh=qINC7joXp7uw4KPCQ67WFGXLT7ugR58Q6zkus8Zk1hI=; b=0Xu/vT42CAWJMaBpgBTH5Bdxrf4/MYnpFBwjSf8Khu0fdbVmQUv2CPfYgQxBgMRKx6 fyFxu0FyI3iPqJqhca8Gk4Meq1PAPgoSa6wSs6Y2OMFqtSAi7Todq1ifAiXA6i7MF13E d4tokKTAPwCPig/0hTj6iaQkGRIuvO7v8pyq8jOia3JQphqr796bbBr7E5UdXMsy3SFY f8PXgIlaEuYCHTZI8euwW35USbXR9ZxvsMBNbHihAUX0zMRnl7eM1ahc4VwQSC2L0zD7 4VX8A6JXx/SqVMhzWhphEDCCho1QFoQKvl46FXhrUmWHdVwBYg50g4Ra6cyWzuCcuvZl /aBQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=Ar6epBmO; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) smtp.mailfrom=qemu-devel-bounces+patch=linaro.org@nongnu.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [2001:4830:134:3::11]) by mx.google.com with ESMTPS id w13si3318327ywi.223.2017.11.21.13.37.48 for (version=TLS1 cipher=AES128-SHA bits=128/128); Tue, 21 Nov 2017 13:37:49 -0800 (PST) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) client-ip=2001:4830:134:3::11; Authentication-Results: mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=Ar6epBmO; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) smtp.mailfrom=qemu-devel-bounces+patch=linaro.org@nongnu.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1]:36627 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1eHGEm-0004Zb-El for patch@linaro.org; Tue, 21 Nov 2017 16:37:48 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:54281) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1eHG5X-0004lV-Pd for qemu-devel@nongnu.org; Tue, 21 Nov 2017 16:28:19 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1eHG5U-0007sp-3k for qemu-devel@nongnu.org; Tue, 21 Nov 2017 16:28:15 -0500 Received: from mail-wr0-x242.google.com ([2a00:1450:400c:c0c::242]:45850) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1eHG5T-0007sL-Lr for qemu-devel@nongnu.org; Tue, 21 Nov 2017 16:28:12 -0500 Received: by mail-wr0-x242.google.com with SMTP id a63so12599492wrc.12 for ; Tue, 21 Nov 2017 13:28:11 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:subject:date:message-id:in-reply-to:references; bh=qINC7joXp7uw4KPCQ67WFGXLT7ugR58Q6zkus8Zk1hI=; b=Ar6epBmOFW/51C/Xyfx/17TwDJbg7WFqDVdVZg3lDW9sDx7AdnE0vM2lASBRhq5snp own7bJGdWcfTLbxo8epb/9NYjnSL2XZoihQ7xNN1IN9UTNpYBIxQ+4w5V9oY0I25ved3 fdlgn25BbJtskvxFVpCnwFBwpsK3DBv0H2GiI= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references; bh=qINC7joXp7uw4KPCQ67WFGXLT7ugR58Q6zkus8Zk1hI=; b=OJrqI0+IGX5XBg+xF+h7M+7ZtXLXI8PfJX48YbASQFq83llkka9+4zz0iQMt9GKhY1 FrfqVoiRILP7PiVsiyDPrx5h/yob1mmx5gszPiCLzLRvvxIVkTBj894KgtUiskt+7n7h eyeyjXCXg0kNc/Kgvsou0stcc5AYrY/Dy5ugwIPjvCyOgs4VG5X4b3oyow+mR3PiPbJT yfzf73wAzKOHBtjvF0KMeosR7dscu8IBO7Irww8JCxZeOtVXo1/5zHDy+ggwGTQhIai1 o/kmtWmBquQmrZXW+F1avoSRC2Ua8Ex78M7KVMLd4xVngHJkju61iIrQafPpZ1L1jcZ8 CFWw== X-Gm-Message-State: AJaThX78/rSyXRl78GXOu+fijsghBq8BGwB1WGjWHfyx2/OvpgdFIva/ NFAhaSmE3OjvTPjGVAo/lL1whIKoY9Y= X-Received: by 10.223.165.4 with SMTP id i4mr16647589wrb.158.1511299689984; Tue, 21 Nov 2017 13:28:09 -0800 (PST) Received: from cloudburst.twiddle.net (70.red-37-158-60.dynamicip.rima-tde.net. [37.158.60.70]) by smtp.gmail.com with ESMTPSA id e124sm706517wmg.34.2017.11.21.13.28.08 for (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Tue, 21 Nov 2017 13:28:09 -0800 (PST) From: Richard Henderson To: qemu-devel@nongnu.org Date: Tue, 21 Nov 2017 22:25:34 +0100 Message-Id: <20171121212534.5177-27-richard.henderson@linaro.org> X-Mailer: git-send-email 2.13.6 In-Reply-To: <20171121212534.5177-1-richard.henderson@linaro.org> References: <20171121212534.5177-1-richard.henderson@linaro.org> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2a00:1450:400c:c0c::242 Subject: [Qemu-devel] [PATCH v6 26/26] tcg/aarch64: Add vector operations X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: "Qemu-devel" Signed-off-by: Richard Henderson --- tcg/aarch64/tcg-target.h | 30 +- tcg/aarch64/tcg-target.opc.h | 3 + tcg/aarch64/tcg-target.inc.c | 674 ++++++++++++++++++++++++++++++++++++++++--- 3 files changed, 660 insertions(+), 47 deletions(-) create mode 100644 tcg/aarch64/tcg-target.opc.h -- 2.13.6 diff --git a/tcg/aarch64/tcg-target.h b/tcg/aarch64/tcg-target.h index c2525066ab..46434ecca4 100644 --- a/tcg/aarch64/tcg-target.h +++ b/tcg/aarch64/tcg-target.h @@ -31,13 +31,22 @@ typedef enum { TCG_REG_SP = 31, TCG_REG_XZR = 31, + TCG_REG_V0 = 32, TCG_REG_V1, TCG_REG_V2, TCG_REG_V3, + TCG_REG_V4, TCG_REG_V5, TCG_REG_V6, TCG_REG_V7, + TCG_REG_V8, TCG_REG_V9, TCG_REG_V10, TCG_REG_V11, + TCG_REG_V12, TCG_REG_V13, TCG_REG_V14, TCG_REG_V15, + TCG_REG_V16, TCG_REG_V17, TCG_REG_V18, TCG_REG_V19, + TCG_REG_V20, TCG_REG_V21, TCG_REG_V22, TCG_REG_V23, + TCG_REG_V24, TCG_REG_V25, TCG_REG_V26, TCG_REG_V27, + TCG_REG_V28, TCG_REG_V29, TCG_REG_V30, TCG_REG_V31, + /* Aliases. */ TCG_REG_FP = TCG_REG_X29, TCG_REG_LR = TCG_REG_X30, TCG_AREG0 = TCG_REG_X19, } TCGReg; -#define TCG_TARGET_NB_REGS 32 +#define TCG_TARGET_NB_REGS 64 /* used for function call generation */ #define TCG_REG_CALL_STACK TCG_REG_SP @@ -113,6 +122,25 @@ typedef enum { #define TCG_TARGET_HAS_mulsh_i64 1 #define TCG_TARGET_HAS_direct_jump 1 +#define TCG_TARGET_HAS_v64 1 +#define TCG_TARGET_HAS_v128 1 +#define TCG_TARGET_HAS_v256 0 + +#define TCG_TARGET_HAS_andc_vec 1 +#define TCG_TARGET_HAS_orc_vec 1 +#define TCG_TARGET_HAS_not_vec 1 +#define TCG_TARGET_HAS_neg_vec 1 +#define TCG_TARGET_HAS_shi_vec 1 +#define TCG_TARGET_HAS_shs_vec 0 +#define TCG_TARGET_HAS_shv_vec 0 +#define TCG_TARGET_HAS_zip_vec 1 +#define TCG_TARGET_HAS_uzp_vec 1 +#define TCG_TARGET_HAS_trn_vec 1 +#define TCG_TARGET_HAS_cmp_vec 1 +#define TCG_TARGET_HAS_mul_vec 1 +#define TCG_TARGET_HAS_extl_vec 1 +#define TCG_TARGET_HAS_exth_vec 1 + #define TCG_TARGET_DEFAULT_MO (0) static inline void flush_icache_range(uintptr_t start, uintptr_t stop) diff --git a/tcg/aarch64/tcg-target.opc.h b/tcg/aarch64/tcg-target.opc.h new file mode 100644 index 0000000000..4816a6c3d4 --- /dev/null +++ b/tcg/aarch64/tcg-target.opc.h @@ -0,0 +1,3 @@ +/* Target-specific opcodes for host vector expansion. These will be + emitted by tcg_expand_vec_op. For those familiar with GCC internals, + consider these to be UNSPEC with names. */ diff --git a/tcg/aarch64/tcg-target.inc.c b/tcg/aarch64/tcg-target.inc.c index 150530f30e..b2ce818d7c 100644 --- a/tcg/aarch64/tcg-target.inc.c +++ b/tcg/aarch64/tcg-target.inc.c @@ -20,10 +20,15 @@ QEMU_BUILD_BUG_ON(TCG_TYPE_I32 != 0 || TCG_TYPE_I64 != 1); #ifdef CONFIG_DEBUG_TCG static const char * const tcg_target_reg_names[TCG_TARGET_NB_REGS] = { - "%x0", "%x1", "%x2", "%x3", "%x4", "%x5", "%x6", "%x7", - "%x8", "%x9", "%x10", "%x11", "%x12", "%x13", "%x14", "%x15", - "%x16", "%x17", "%x18", "%x19", "%x20", "%x21", "%x22", "%x23", - "%x24", "%x25", "%x26", "%x27", "%x28", "%fp", "%x30", "%sp", + "x0", "x1", "x2", "x3", "x4", "x5", "x6", "x7", + "x8", "x9", "x10", "x11", "x12", "x13", "x14", "x15", + "x16", "x17", "x18", "x19", "x20", "x21", "x22", "x23", + "x24", "x25", "x26", "x27", "x28", "fp", "x30", "sp", + + "v0", "v1", "v2", "v3", "v4", "v5", "v6", "v7", + "v8", "v9", "v10", "v11", "v12", "v13", "v14", "v15", + "v16", "v17", "v18", "v19", "v20", "v21", "v22", "v23", + "v24", "v25", "v26", "v27", "v28", "fp", "v30", "v31", }; #endif /* CONFIG_DEBUG_TCG */ @@ -43,6 +48,14 @@ static const int tcg_target_reg_alloc_order[] = { /* X19 reserved for AREG0 */ /* X29 reserved as fp */ /* X30 reserved as temporary */ + + TCG_REG_V0, TCG_REG_V1, TCG_REG_V2, TCG_REG_V3, + TCG_REG_V4, TCG_REG_V5, TCG_REG_V6, TCG_REG_V7, + /* V8 - V15 are call-saved, and skipped. */ + TCG_REG_V16, TCG_REG_V17, TCG_REG_V18, TCG_REG_V19, + TCG_REG_V20, TCG_REG_V21, TCG_REG_V22, TCG_REG_V23, + TCG_REG_V24, TCG_REG_V25, TCG_REG_V26, TCG_REG_V27, + TCG_REG_V28, TCG_REG_V29, TCG_REG_V30, TCG_REG_V31, }; static const int tcg_target_call_iarg_regs[8] = { @@ -54,6 +67,7 @@ static const int tcg_target_call_oarg_regs[1] = { }; #define TCG_REG_TMP TCG_REG_X30 +#define TCG_VEC_TMP TCG_REG_V31 #ifndef CONFIG_SOFTMMU /* Note that XZR cannot be encoded in the address base register slot, @@ -119,9 +133,13 @@ static const char *target_parse_constraint(TCGArgConstraint *ct, const char *ct_str, TCGType type) { switch (*ct_str++) { - case 'r': + case 'r': /* general registers */ ct->ct |= TCG_CT_REG; - ct->u.regs = 0xffffffffu; + ct->u.regs |= 0xffffffffu; + break; + case 'w': /* advsimd registers */ + ct->ct |= TCG_CT_REG; + ct->u.regs |= 0xffffffff00000000ull; break; case 'l': /* qemu_ld / qemu_st address, data_reg */ ct->ct |= TCG_CT_REG; @@ -178,6 +196,98 @@ static inline bool is_limm(uint64_t val) return (val & (val - 1)) == 0; } +static bool is_fimm(uint64_t v64, int *op, int *cmode, int *imm8) +{ + int i; + + *op = 0; + if (v64 == (-1ull / 0xff) * (v64 & 0xff)) { + *cmode = 0xe; + *imm8 = v64 & 0xff; + return true; + } + if (v64 == (-1ull / 0xffff) * (v64 & 0xffff)) { + uint64_t v16 = v64 & 0xffff; + + if (v16 == (v64 & 0xff)) { + *cmode = 0x8; + *imm8 = v64 & 0xff; + return true; + } else if (v16 == (v64 & 0xff00)) { + *cmode = 0xa; + *imm8 = v16 >> 8; + return true; + } + } + if (v64 == deposit64(v64, 32, 32, v64)) { + uint64_t v32 = (uint32_t)v64; + + if (v32 == (v64 & 0xff)) { + *cmode = 0x0; + *imm8 = v64 & 0xff; + return true; + } else if (v32 == (v32 & 0xff00)) { + *cmode = 0x2; + *imm8 = (v64 >> 8) & 0xff; + return true; + } else if (v32 == (v32 & 0xff0000)) { + *cmode = 0x4; + *imm8 = (v64 >> 16) & 0xff; + return true; + } else if (v32 == (v32 & 0xff000000)) { + *cmode = 0x6; + *imm8 = v32 >> 24; + return true; + } else if ((v32 & 0xffff00ff) == 0xff) { + *cmode = 0xc; + *imm8 = (v64 >> 8) & 0xff; + return true; + } else if ((v32 & 0xff00ffff) == 0xffff) { + *cmode = 0xd; + *imm8 = (v64 >> 16) & 0xff; + return true; + } else if (extract32(v32, 0, 19) == 0 + && (extract32(v32, 25, 6) == 0x20 + || extract32(v32, 25, 6) == 0x1f)) { + *cmode = 0xf; + *imm8 = (extract32(v32, 31, 1) << 7) + | (extract32(v32, 25, 1) << 6) + | extract32(v32, 19, 6); + return true; + } + } + if (extract64(v64, 0, 48) == 0 + && (extract64(v64, 54, 9) == 0x100 + || extract64(v64, 54, 9) == 0x0ff)) { + *cmode = 0xf; + *op = 1; + *imm8 = (extract64(v64, 63, 1) << 7) + | (extract64(v64, 54, 1) << 6) + | extract64(v64, 48, 6); + return true; + } + for (i = 0; i < 64; i += 8) { + uint64_t byte = extract64(v64, i, 8); + if (byte != 0 && byte != 0xff) { + break; + } + } + if (i == 64) { + *cmode = 0xe; + *op = 1; + *imm8 = (extract64(v64, 0, 1) << 0) + | (extract64(v64, 8, 1) << 1) + | (extract64(v64, 16, 1) << 2) + | (extract64(v64, 24, 1) << 3) + | (extract64(v64, 32, 1) << 4) + | (extract64(v64, 40, 1) << 5) + | (extract64(v64, 48, 1) << 6) + | (extract64(v64, 56, 1) << 7); + return true; + } + return false; +} + static int tcg_target_const_match(tcg_target_long val, TCGType type, const TCGArgConstraint *arg_ct) { @@ -271,6 +381,9 @@ typedef enum { /* Load literal for loading the address at pc-relative offset */ I3305_LDR = 0x58000000, + I3305_LDR_v64 = 0x5c000000, + I3305_LDR_v128 = 0x9c000000, + /* Load/store register. Described here as 3.3.12, but the helper that emits them can transform to 3.3.10 or 3.3.13. */ I3312_STRB = 0x38000000 | LDST_ST << 22 | MO_8 << 30, @@ -290,6 +403,15 @@ typedef enum { I3312_LDRSHX = 0x38000000 | LDST_LD_S_X << 22 | MO_16 << 30, I3312_LDRSWX = 0x38000000 | LDST_LD_S_X << 22 | MO_32 << 30, + I3312_LDRVS = 0x3c000000 | LDST_LD << 22 | MO_32 << 30, + I3312_STRVS = 0x3c000000 | LDST_ST << 22 | MO_32 << 30, + + I3312_LDRVD = 0x3c000000 | LDST_LD << 22 | MO_64 << 30, + I3312_STRVD = 0x3c000000 | LDST_ST << 22 | MO_64 << 30, + + I3312_LDRVQ = 0x3c000000 | 3 << 22 | 0 << 30, + I3312_STRVQ = 0x3c000000 | 2 << 22 | 0 << 30, + I3312_TO_I3310 = 0x00200800, I3312_TO_I3313 = 0x01000000, @@ -374,8 +496,58 @@ typedef enum { I3510_EON = 0x4a200000, I3510_ANDS = 0x6a000000, - NOP = 0xd503201f, + /* AdvSIMD zip.uzp/trn */ + I3603_ZIP1 = 0x0e003800, + I3603_UZP1 = 0x0e001800, + I3603_TRN1 = 0x0e002800, + I3603_ZIP2 = 0x0e007800, + I3603_UZP2 = 0x0e005800, + I3603_TRN2 = 0x0e006800, + + /* AdvSIMD copy */ + I3605_DUP = 0x0e000400, + I3605_INS = 0x4e001c00, + I3605_UMOV = 0x0e003c00, + + /* AdvSIMD modified immediate */ + I3606_MOVI = 0x0f000400, + + /* AdvSIMD shift by immediate */ + I3614_SSHR = 0x0f000400, + I3614_SSRA = 0x0f001400, + I3614_SHL = 0x0f005400, + I3614_SSHLL = 0x0f00a400, + I3614_USHR = 0x2f000400, + I3614_USRA = 0x2f001400, + I3614_USHLL = 0x2f00a400, + + /* AdvSIMD three same. */ + I3616_ADD = 0x0e208400, + I3616_AND = 0x0e201c00, + I3616_BIC = 0x0e601c00, + I3616_EOR = 0x2e201c00, + I3616_MUL = 0x0e209c00, + I3616_ORR = 0x0ea01c00, + I3616_ORN = 0x0ee01c00, + I3616_SUB = 0x2e208400, + I3616_CMGT = 0x0e203400, + I3616_CMGE = 0x0e203c00, + I3616_CMTST = 0x0e208c00, + I3616_CMHI = 0x2e203400, + I3616_CMHS = 0x2e203c00, + I3616_CMEQ = 0x2e208c00, + + /* AdvSIMD two-reg misc. */ + I3617_CMGT0 = 0x0e208800, + I3617_CMEQ0 = 0x0e209800, + I3617_CMLT0 = 0x0e20a800, + I3617_CMGE0 = 0x2e208800, + I3617_CMLE0 = 0x2e20a800, + I3617_NOT = 0x2e205800, + I3617_NEG = 0x2e20b800, + /* System instructions. */ + NOP = 0xd503201f, DMB_ISH = 0xd50338bf, DMB_LD = 0x00000100, DMB_ST = 0x00000200, @@ -520,26 +692,71 @@ static void tcg_out_insn_3509(TCGContext *s, AArch64Insn insn, TCGType ext, tcg_out32(s, insn | ext << 31 | rm << 16 | ra << 10 | rn << 5 | rd); } +static void tcg_out_insn_3603(TCGContext *s, AArch64Insn insn, bool q, + unsigned size, TCGReg rd, TCGReg rn, TCGReg rm) +{ + tcg_out32(s, insn | q << 30 | (size << 22) | (rd & 0x1f) + | (rn & 0x1f) << 5 | (rm & 0x1f) << 16); +} + +static void tcg_out_insn_3605(TCGContext *s, AArch64Insn insn, bool q, + TCGReg rd, TCGReg rn, int dst_idx, int src_idx) +{ + /* Note that bit 11 set means general register input. Therefore + we can handle both register sets with one function. */ + tcg_out32(s, insn | q << 30 | (dst_idx << 16) | (src_idx << 11) + | (rd & 0x1f) | (~rn & 0x20) << 6 | (rn & 0x1f) << 5); +} + +static void tcg_out_insn_3606(TCGContext *s, AArch64Insn insn, bool q, + TCGReg rd, bool op, int cmode, uint8_t imm8) +{ + tcg_out32(s, insn | q << 30 | op << 29 | cmode << 12 | (rd & 0x1f) + | (imm8 & 0xe0) << (16 - 5) | (imm8 & 0x1f) << 5); +} + +static void tcg_out_insn_3614(TCGContext *s, AArch64Insn insn, bool q, + TCGReg rd, TCGReg rn, unsigned immhb) +{ + tcg_out32(s, insn | q << 30 | immhb << 16 + | (rn & 0x1f) << 5 | (rd & 0x1f)); +} + +static void tcg_out_insn_3616(TCGContext *s, AArch64Insn insn, bool q, + unsigned size, TCGReg rd, TCGReg rn, TCGReg rm) +{ + tcg_out32(s, insn | q << 30 | (size << 22) | (rm & 0x1f) << 16 + | (rn & 0x1f) << 5 | (rd & 0x1f)); +} + +static void tcg_out_insn_3617(TCGContext *s, AArch64Insn insn, bool q, + unsigned size, TCGReg rd, TCGReg rn) +{ + tcg_out32(s, insn | q << 30 | (size << 22) + | (rn & 0x1f) << 5 | (rd & 0x1f)); +} + static void tcg_out_insn_3310(TCGContext *s, AArch64Insn insn, TCGReg rd, TCGReg base, TCGType ext, TCGReg regoff) { /* Note the AArch64Insn constants above are for C3.3.12. Adjust. */ tcg_out32(s, insn | I3312_TO_I3310 | regoff << 16 | - 0x4000 | ext << 13 | base << 5 | rd); + 0x4000 | ext << 13 | base << 5 | (rd & 0x1f)); } static void tcg_out_insn_3312(TCGContext *s, AArch64Insn insn, TCGReg rd, TCGReg rn, intptr_t offset) { - tcg_out32(s, insn | (offset & 0x1ff) << 12 | rn << 5 | rd); + tcg_out32(s, insn | (offset & 0x1ff) << 12 | rn << 5 | (rd & 0x1f)); } static void tcg_out_insn_3313(TCGContext *s, AArch64Insn insn, TCGReg rd, TCGReg rn, uintptr_t scaled_uimm) { /* Note the AArch64Insn constants above are for C3.3.12. Adjust. */ - tcg_out32(s, insn | I3312_TO_I3313 | scaled_uimm << 10 | rn << 5 | rd); + tcg_out32(s, insn | I3312_TO_I3313 | scaled_uimm << 10 + | rn << 5 | (rd & 0x1f)); } /* Register to register move using ORR (shifted register with no shift). */ @@ -585,6 +802,35 @@ static void tcg_out_logicali(TCGContext *s, AArch64Insn insn, TCGType ext, tcg_out_insn_3404(s, insn, ext, rd, rn, ext, r, c); } +static void tcg_out_dupi_vec(TCGContext *s, TCGType type, + TCGReg rd, uint64_t v64) +{ + int op, cmode, imm8; + + if (is_fimm(v64, &op, &cmode, &imm8)) { + tcg_out_insn(s, 3606, MOVI, type == TCG_TYPE_V128, rd, op, cmode, imm8); + } else if (type == TCG_TYPE_V128) { + new_pool_l2(s, R_AARCH64_CONDBR19, s->code_ptr, 0, v64, v64); + tcg_out_insn(s, 3305, LDR_v128, 0, rd); + } else { + new_pool_label(s, v64, R_AARCH64_CONDBR19, s->code_ptr, 0); + tcg_out_insn(s, 3305, LDR_v64, 0, rd); + } +} + +static void tcg_out_movi_vec(TCGContext *s, TCGType type, + TCGReg ret, const TCGArg *a) +{ + if (type == TCG_TYPE_V128) { + /* We assume that INDEX_op_dupi could not be used and + therefore we must use a constant pool entry. */ + new_pool_l2(s, R_AARCH64_CONDBR19, s->code_ptr, 0, a[0], a[1]); + tcg_out_insn(s, 3305, LDR_v128, 0, ret); + } else { + tcg_out_dupi_vec(s, TCG_TYPE_V64, ret, a[0]); + } +} + static void tcg_out_movi(TCGContext *s, TCGType type, TCGReg rd, tcg_target_long value) { @@ -594,6 +840,22 @@ static void tcg_out_movi(TCGContext *s, TCGType type, TCGReg rd, int s0, s1; AArch64Insn opc; + switch (type) { + case TCG_TYPE_I32: + case TCG_TYPE_I64: + tcg_debug_assert(rd < 32); + break; + + case TCG_TYPE_V64: + case TCG_TYPE_V128: + tcg_debug_assert(rd >= 32); + tcg_out_dupi_vec(s, type, rd, value); + return; + + default: + g_assert_not_reached(); + } + /* For 32-bit values, discard potential garbage in value. For 64-bit values within [2**31, 2**32-1], we can create smaller sequences by interpreting this as a negative 32-bit number, while ensuring that @@ -669,15 +931,13 @@ static void tcg_out_movi(TCGContext *s, TCGType type, TCGReg rd, /* Define something more legible for general use. */ #define tcg_out_ldst_r tcg_out_insn_3310 -static void tcg_out_ldst(TCGContext *s, AArch64Insn insn, - TCGReg rd, TCGReg rn, intptr_t offset) +static void tcg_out_ldst(TCGContext *s, AArch64Insn insn, TCGReg rd, + TCGReg rn, intptr_t offset, int lgsize) { - TCGMemOp size = (uint32_t)insn >> 30; - /* If the offset is naturally aligned and in range, then we can use the scaled uimm12 encoding */ - if (offset >= 0 && !(offset & ((1 << size) - 1))) { - uintptr_t scaled_uimm = offset >> size; + if (offset >= 0 && !(offset & ((1 << lgsize) - 1))) { + uintptr_t scaled_uimm = offset >> lgsize; if (scaled_uimm <= 0xfff) { tcg_out_insn_3313(s, insn, rd, rn, scaled_uimm); return; @@ -695,32 +955,102 @@ static void tcg_out_ldst(TCGContext *s, AArch64Insn insn, tcg_out_ldst_r(s, insn, rd, rn, TCG_TYPE_I64, TCG_REG_TMP); } -static inline void tcg_out_mov(TCGContext *s, - TCGType type, TCGReg ret, TCGReg arg) +static void tcg_out_mov(TCGContext *s, TCGType type, TCGReg ret, TCGReg arg) { - if (ret != arg) { - tcg_out_movr(s, type, ret, arg); + if (ret == arg) { + return; + } + switch (type) { + case TCG_TYPE_I32: + case TCG_TYPE_I64: + if (ret < 32 && arg < 32) { + tcg_out_movr(s, type, ret, arg); + break; + } else if (ret < 32) { + tcg_out_insn(s, 3605, UMOV, type, ret, arg, 0, 0); + break; + } else if (arg < 32) { + tcg_out_insn(s, 3605, INS, 0, ret, arg, 4 << type, 0); + break; + } + /* FALLTHRU */ + + case TCG_TYPE_V64: + tcg_debug_assert(ret >= 32 && arg >= 32); + tcg_out_insn(s, 3616, ORR, 0, 0, ret, arg, arg); + break; + case TCG_TYPE_V128: + tcg_debug_assert(ret >= 32 && arg >= 32); + tcg_out_insn(s, 3616, ORR, 1, 0, ret, arg, arg); + break; + + default: + g_assert_not_reached(); } } -static inline void tcg_out_ld(TCGContext *s, TCGType type, TCGReg arg, - TCGReg arg1, intptr_t arg2) +static void tcg_out_ld(TCGContext *s, TCGType type, TCGReg ret, + TCGReg base, intptr_t ofs) { - tcg_out_ldst(s, type == TCG_TYPE_I32 ? I3312_LDRW : I3312_LDRX, - arg, arg1, arg2); + AArch64Insn insn; + int lgsz; + + switch (type) { + case TCG_TYPE_I32: + insn = (ret < 32 ? I3312_LDRW : I3312_LDRVS); + lgsz = 2; + break; + case TCG_TYPE_I64: + insn = (ret < 32 ? I3312_LDRX : I3312_LDRVD); + lgsz = 3; + break; + case TCG_TYPE_V64: + insn = I3312_LDRVD; + lgsz = 3; + break; + case TCG_TYPE_V128: + insn = I3312_LDRVQ; + lgsz = 4; + break; + default: + g_assert_not_reached(); + } + tcg_out_ldst(s, insn, ret, base, ofs, lgsz); } -static inline void tcg_out_st(TCGContext *s, TCGType type, TCGReg arg, - TCGReg arg1, intptr_t arg2) +static void tcg_out_st(TCGContext *s, TCGType type, TCGReg src, + TCGReg base, intptr_t ofs) { - tcg_out_ldst(s, type == TCG_TYPE_I32 ? I3312_STRW : I3312_STRX, - arg, arg1, arg2); + AArch64Insn insn; + int lgsz; + + switch (type) { + case TCG_TYPE_I32: + insn = (src < 32 ? I3312_STRW : I3312_STRVS); + lgsz = 2; + break; + case TCG_TYPE_I64: + insn = (src < 32 ? I3312_STRX : I3312_STRVD); + lgsz = 3; + break; + case TCG_TYPE_V64: + insn = I3312_STRVD; + lgsz = 3; + break; + case TCG_TYPE_V128: + insn = I3312_STRVQ; + lgsz = 4; + break; + default: + g_assert_not_reached(); + } + tcg_out_ldst(s, insn, src, base, ofs, lgsz); } static inline bool tcg_out_sti(TCGContext *s, TCGType type, TCGArg val, TCGReg base, intptr_t ofs) { - if (val == 0) { + if (type <= TCG_TYPE_I64 && val == 0) { tcg_out_st(s, type, TCG_REG_XZR, base, ofs); return true; } @@ -1210,14 +1540,15 @@ static void tcg_out_tlb_read(TCGContext *s, TCGReg addr_reg, TCGMemOp opc, /* Merge "low bits" from tlb offset, load the tlb comparator into X0. X0 = load [X2 + (tlb_offset & 0x000fff)] */ tcg_out_ldst(s, TARGET_LONG_BITS == 32 ? I3312_LDRW : I3312_LDRX, - TCG_REG_X0, TCG_REG_X2, tlb_offset & 0xfff); + TCG_REG_X0, TCG_REG_X2, tlb_offset & 0xfff, + TARGET_LONG_BITS == 32 ? 2 : 3); /* Load the tlb addend. Do that early to avoid stalling. X1 = load [X2 + (tlb_offset & 0xfff) + offsetof(addend)] */ tcg_out_ldst(s, I3312_LDRX, TCG_REG_X1, TCG_REG_X2, (tlb_offset & 0xfff) + (offsetof(CPUTLBEntry, addend)) - (is_read ? offsetof(CPUTLBEntry, addr_read) - : offsetof(CPUTLBEntry, addr_write))); + : offsetof(CPUTLBEntry, addr_write)), 3); /* Perform the address comparison. */ tcg_out_cmp(s, (TARGET_LONG_BITS == 64), TCG_REG_X0, TCG_REG_X3, 0); @@ -1435,49 +1766,49 @@ static void tcg_out_op(TCGContext *s, TCGOpcode opc, case INDEX_op_ld8u_i32: case INDEX_op_ld8u_i64: - tcg_out_ldst(s, I3312_LDRB, a0, a1, a2); + tcg_out_ldst(s, I3312_LDRB, a0, a1, a2, 0); break; case INDEX_op_ld8s_i32: - tcg_out_ldst(s, I3312_LDRSBW, a0, a1, a2); + tcg_out_ldst(s, I3312_LDRSBW, a0, a1, a2, 0); break; case INDEX_op_ld8s_i64: - tcg_out_ldst(s, I3312_LDRSBX, a0, a1, a2); + tcg_out_ldst(s, I3312_LDRSBX, a0, a1, a2, 0); break; case INDEX_op_ld16u_i32: case INDEX_op_ld16u_i64: - tcg_out_ldst(s, I3312_LDRH, a0, a1, a2); + tcg_out_ldst(s, I3312_LDRH, a0, a1, a2, 1); break; case INDEX_op_ld16s_i32: - tcg_out_ldst(s, I3312_LDRSHW, a0, a1, a2); + tcg_out_ldst(s, I3312_LDRSHW, a0, a1, a2, 1); break; case INDEX_op_ld16s_i64: - tcg_out_ldst(s, I3312_LDRSHX, a0, a1, a2); + tcg_out_ldst(s, I3312_LDRSHX, a0, a1, a2, 1); break; case INDEX_op_ld_i32: case INDEX_op_ld32u_i64: - tcg_out_ldst(s, I3312_LDRW, a0, a1, a2); + tcg_out_ldst(s, I3312_LDRW, a0, a1, a2, 2); break; case INDEX_op_ld32s_i64: - tcg_out_ldst(s, I3312_LDRSWX, a0, a1, a2); + tcg_out_ldst(s, I3312_LDRSWX, a0, a1, a2, 2); break; case INDEX_op_ld_i64: - tcg_out_ldst(s, I3312_LDRX, a0, a1, a2); + tcg_out_ldst(s, I3312_LDRX, a0, a1, a2, 3); break; case INDEX_op_st8_i32: case INDEX_op_st8_i64: - tcg_out_ldst(s, I3312_STRB, REG0(0), a1, a2); + tcg_out_ldst(s, I3312_STRB, REG0(0), a1, a2, 0); break; case INDEX_op_st16_i32: case INDEX_op_st16_i64: - tcg_out_ldst(s, I3312_STRH, REG0(0), a1, a2); + tcg_out_ldst(s, I3312_STRH, REG0(0), a1, a2, 1); break; case INDEX_op_st_i32: case INDEX_op_st32_i64: - tcg_out_ldst(s, I3312_STRW, REG0(0), a1, a2); + tcg_out_ldst(s, I3312_STRW, REG0(0), a1, a2, 2); break; case INDEX_op_st_i64: - tcg_out_ldst(s, I3312_STRX, REG0(0), a1, a2); + tcg_out_ldst(s, I3312_STRX, REG0(0), a1, a2, 3); break; case INDEX_op_add_i32: @@ -1776,25 +2107,230 @@ static void tcg_out_op(TCGContext *s, TCGOpcode opc, case INDEX_op_mov_i32: /* Always emitted via tcg_out_mov. */ case INDEX_op_mov_i64: + case INDEX_op_mov_vec: case INDEX_op_movi_i32: /* Always emitted via tcg_out_movi. */ case INDEX_op_movi_i64: + case INDEX_op_dupi_vec: case INDEX_op_call: /* Always emitted via tcg_out_call. */ default: - tcg_abort(); + g_assert_not_reached(); } #undef REG0 } +static void tcg_out_vec_op(TCGContext *s, TCGOpcode opc, + unsigned vecl, unsigned vece, + const TCGArg *args, const int *const_args) +{ + static const AArch64Insn cmp_insn[16] = { + [TCG_COND_EQ] = I3616_CMEQ, + [TCG_COND_GT] = I3616_CMGT, + [TCG_COND_GE] = I3616_CMGE, + [TCG_COND_GTU] = I3616_CMHI, + [TCG_COND_GEU] = I3616_CMHS, + }; + static const AArch64Insn cmp0_insn[16] = { + [TCG_COND_EQ] = I3617_CMEQ0, + [TCG_COND_GT] = I3617_CMGT0, + [TCG_COND_GE] = I3617_CMGE0, + [TCG_COND_LT] = I3617_CMLT0, + [TCG_COND_LE] = I3617_CMLE0, + }; + + TCGType type = vecl + TCG_TYPE_V64; + unsigned is_q = vecl; + TCGArg a0, a1, a2; + + a0 = args[0]; + a1 = args[1]; + a2 = args[2]; + + switch (opc) { + case INDEX_op_movi_vec: + tcg_out_movi_vec(s, type, a0, args + 1); + break; + case INDEX_op_ld_vec: + tcg_out_ld(s, type, a0, a1, a2); + break; + case INDEX_op_st_vec: + tcg_out_st(s, type, a0, a1, a2); + break; + case INDEX_op_add_vec: + tcg_out_insn(s, 3616, ADD, is_q, vece, a0, a1, a2); + break; + case INDEX_op_sub_vec: + tcg_out_insn(s, 3616, SUB, is_q, vece, a0, a1, a2); + break; + case INDEX_op_mul_vec: + tcg_out_insn(s, 3616, MUL, is_q, vece, a0, a1, a2); + break; + case INDEX_op_neg_vec: + tcg_out_insn(s, 3617, NEG, is_q, vece, a0, a1); + break; + case INDEX_op_and_vec: + tcg_out_insn(s, 3616, AND, is_q, 0, a0, a1, a2); + break; + case INDEX_op_or_vec: + tcg_out_insn(s, 3616, ORR, is_q, 0, a0, a1, a2); + break; + case INDEX_op_xor_vec: + tcg_out_insn(s, 3616, EOR, is_q, 0, a0, a1, a2); + break; + case INDEX_op_andc_vec: + tcg_out_insn(s, 3616, BIC, is_q, 0, a0, a1, a2); + break; + case INDEX_op_orc_vec: + tcg_out_insn(s, 3616, ORN, is_q, 0, a0, a1, a2); + break; + case INDEX_op_not_vec: + tcg_out_insn(s, 3617, NOT, is_q, 0, a0, a1); + break; + case INDEX_op_dup_vec: + tcg_out_insn(s, 3605, DUP, is_q, a0, a1, 1 << vece, 0); + break; + case INDEX_op_shli_vec: + tcg_out_insn(s, 3614, SHL, is_q, a0, a1, a2 + (8 << vece)); + break; + case INDEX_op_shri_vec: + tcg_out_insn(s, 3614, USHR, is_q, a0, a1, (16 << vece) - a2); + break; + case INDEX_op_sari_vec: + tcg_out_insn(s, 3614, SSHR, is_q, a0, a1, (16 << vece) - a2); + break; + case INDEX_op_cmp_vec: + { + TCGCond cond = args[3]; + AArch64Insn insn; + + if (cond == TCG_COND_NE) { + if (const_args[2]) { + tcg_out_insn(s, 3616, CMTST, is_q, vece, a0, a1, a1); + } else { + tcg_out_insn(s, 3616, CMEQ, is_q, vece, a0, a1, a2); + tcg_out_insn(s, 3617, NOT, is_q, 0, a0, a0); + } + } else { + if (const_args[2]) { + insn = cmp0_insn[cond]; + if (insn) { + tcg_out_insn_3617(s, insn, is_q, vece, a0, a1); + break; + } + tcg_out_dupi_vec(s, type, TCG_VEC_TMP, 0); + a2 = TCG_VEC_TMP; + } + insn = cmp_insn[cond]; + if (insn == 0) { + TCGArg t; + t = a1, a1 = a2, a2 = t; + cond = tcg_swap_cond(cond); + insn = cmp_insn[cond]; + tcg_debug_assert(insn != 0); + } + tcg_out_insn_3616(s, insn, is_q, vece, a0, a1, a2); + } + } + break; + case INDEX_op_zipl_vec: + tcg_out_insn(s, 3603, ZIP1, is_q, vece, a0, a1, a2); + break; + case INDEX_op_ziph_vec: + tcg_out_insn(s, 3603, ZIP2, is_q, vece, a0, a1, a2); + break; + case INDEX_op_uzpe_vec: + tcg_out_insn(s, 3603, UZP1, is_q, vece, a0, a1, a2); + break; + case INDEX_op_uzpo_vec: + tcg_out_insn(s, 3603, UZP2, is_q, vece, a0, a1, a2); + break; + case INDEX_op_trne_vec: + tcg_out_insn(s, 3603, TRN1, is_q, vece, a0, a1, a2); + break; + case INDEX_op_trno_vec: + tcg_out_insn(s, 3603, TRN2, is_q, vece, a0, a1, a2); + break; + case INDEX_op_extul_vec: + tcg_out_insn(s, 3614, USHLL, 0, a0, a1, 0 + (8 << vece)); + break; + case INDEX_op_extuh_vec: + if (is_q) { + tcg_out_insn(s, 3614, USHLL, 1, a0, a1, 0 + (8 << vece)); + } else { + tcg_out_insn(s, 3614, USHLL, 0, a0, a1, 0 + (8 << vece)); + tcg_out_insn(s, 3605, INS, 0, a0, a0, 8, 16); + } + break; + case INDEX_op_extsl_vec: + tcg_out_insn(s, 3614, SSHLL, 0, a0, a1, 0 + (8 << vece)); + break; + case INDEX_op_extsh_vec: + if (is_q) { + tcg_out_insn(s, 3614, SSHLL, 1, a0, a1, 0 + (8 << vece)); + } else { + tcg_out_insn(s, 3614, SSHLL, 0, a0, a1, 0 + (8 << vece)); + tcg_out_insn(s, 3605, INS, 0, a0, a0, 8, 16); + } + break; + default: + g_assert_not_reached(); + } +} + +int tcg_can_emit_vec_op(TCGOpcode opc, TCGType type, unsigned vece) +{ + switch (opc) { + case INDEX_op_add_vec: + case INDEX_op_sub_vec: + case INDEX_op_mul_vec: + case INDEX_op_and_vec: + case INDEX_op_or_vec: + case INDEX_op_xor_vec: + case INDEX_op_andc_vec: + case INDEX_op_orc_vec: + case INDEX_op_neg_vec: + case INDEX_op_not_vec: + case INDEX_op_cmp_vec: + case INDEX_op_zipl_vec: + case INDEX_op_ziph_vec: + case INDEX_op_uzpe_vec: + case INDEX_op_uzpo_vec: + case INDEX_op_trne_vec: + case INDEX_op_trno_vec: + case INDEX_op_shli_vec: + case INDEX_op_shri_vec: + case INDEX_op_sari_vec: + case INDEX_op_extul_vec: + case INDEX_op_extuh_vec: + case INDEX_op_extsl_vec: + case INDEX_op_extsh_vec: + return 1; + + default: + return 0; + } +} + +void tcg_expand_vec_op(TCGOpcode opc, TCGType type, unsigned vece, + TCGArg a0, ...) +{ +} + static const TCGTargetOpDef *tcg_target_op_def(TCGOpcode op) { static const TCGTargetOpDef r = { .args_ct_str = { "r" } }; + static const TCGTargetOpDef w = { .args_ct_str = { "w" } }; static const TCGTargetOpDef r_r = { .args_ct_str = { "r", "r" } }; + static const TCGTargetOpDef w_w = { .args_ct_str = { "w", "w" } }; + static const TCGTargetOpDef w_r = { .args_ct_str = { "w", "r" } }; + static const TCGTargetOpDef w_wr = { .args_ct_str = { "w", "wr" } }; static const TCGTargetOpDef r_l = { .args_ct_str = { "r", "l" } }; static const TCGTargetOpDef r_rA = { .args_ct_str = { "r", "rA" } }; static const TCGTargetOpDef rZ_r = { .args_ct_str = { "rZ", "r" } }; static const TCGTargetOpDef lZ_l = { .args_ct_str = { "lZ", "l" } }; static const TCGTargetOpDef r_r_r = { .args_ct_str = { "r", "r", "r" } }; + static const TCGTargetOpDef w_w_w = { .args_ct_str = { "w", "w", "w" } }; + static const TCGTargetOpDef w_w_wZ = { .args_ct_str = { "w", "w", "wZ" } }; static const TCGTargetOpDef r_r_ri = { .args_ct_str = { "r", "r", "ri" } }; static const TCGTargetOpDef r_r_rA = { .args_ct_str = { "r", "r", "rA" } }; static const TCGTargetOpDef r_r_rL = { .args_ct_str = { "r", "r", "rL" } }; @@ -1938,6 +2474,41 @@ static const TCGTargetOpDef *tcg_target_op_def(TCGOpcode op) case INDEX_op_sub2_i64: return &add2; + case INDEX_op_movi_vec: + return &w; + case INDEX_op_add_vec: + case INDEX_op_sub_vec: + case INDEX_op_mul_vec: + case INDEX_op_and_vec: + case INDEX_op_or_vec: + case INDEX_op_xor_vec: + case INDEX_op_andc_vec: + case INDEX_op_orc_vec: + case INDEX_op_zipl_vec: + case INDEX_op_ziph_vec: + case INDEX_op_uzpe_vec: + case INDEX_op_uzpo_vec: + case INDEX_op_trne_vec: + case INDEX_op_trno_vec: + return &w_w_w; + case INDEX_op_not_vec: + case INDEX_op_neg_vec: + case INDEX_op_shli_vec: + case INDEX_op_shri_vec: + case INDEX_op_sari_vec: + case INDEX_op_extul_vec: + case INDEX_op_extuh_vec: + case INDEX_op_extsl_vec: + case INDEX_op_extsh_vec: + return &w_w; + case INDEX_op_ld_vec: + case INDEX_op_st_vec: + return &w_r; + case INDEX_op_dup_vec: + return &w_wr; + case INDEX_op_cmp_vec: + return &w_w_wZ; + default: return NULL; } @@ -1947,8 +2518,10 @@ static void tcg_target_init(TCGContext *s) { tcg_target_available_regs[TCG_TYPE_I32] = 0xffffffffu; tcg_target_available_regs[TCG_TYPE_I64] = 0xffffffffu; + tcg_target_available_regs[TCG_TYPE_V64] = 0xffffffff00000000ull; + tcg_target_available_regs[TCG_TYPE_V128] = 0xffffffff00000000ull; - tcg_target_call_clobber_regs = 0xfffffffu; + tcg_target_call_clobber_regs = -1ull; tcg_regset_reset_reg(tcg_target_call_clobber_regs, TCG_REG_X19); tcg_regset_reset_reg(tcg_target_call_clobber_regs, TCG_REG_X20); tcg_regset_reset_reg(tcg_target_call_clobber_regs, TCG_REG_X21); @@ -1960,12 +2533,21 @@ static void tcg_target_init(TCGContext *s) tcg_regset_reset_reg(tcg_target_call_clobber_regs, TCG_REG_X27); tcg_regset_reset_reg(tcg_target_call_clobber_regs, TCG_REG_X28); tcg_regset_reset_reg(tcg_target_call_clobber_regs, TCG_REG_X29); + tcg_regset_reset_reg(tcg_target_call_clobber_regs, TCG_REG_V8); + tcg_regset_reset_reg(tcg_target_call_clobber_regs, TCG_REG_V9); + tcg_regset_reset_reg(tcg_target_call_clobber_regs, TCG_REG_V10); + tcg_regset_reset_reg(tcg_target_call_clobber_regs, TCG_REG_V11); + tcg_regset_reset_reg(tcg_target_call_clobber_regs, TCG_REG_V12); + tcg_regset_reset_reg(tcg_target_call_clobber_regs, TCG_REG_V13); + tcg_regset_reset_reg(tcg_target_call_clobber_regs, TCG_REG_V14); + tcg_regset_reset_reg(tcg_target_call_clobber_regs, TCG_REG_V15); s->reserved_regs = 0; tcg_regset_set_reg(s->reserved_regs, TCG_REG_SP); tcg_regset_set_reg(s->reserved_regs, TCG_REG_FP); tcg_regset_set_reg(s->reserved_regs, TCG_REG_TMP); tcg_regset_set_reg(s->reserved_regs, TCG_REG_X18); /* platform register */ + tcg_regset_set_reg(s->reserved_regs, TCG_VEC_TMP); } /* Saving pairs: (X19, X20) .. (X27, X28), (X29(fp), X30(lr)). */