From patchwork Thu Aug 9 04:21:47 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 143658 Delivered-To: patch@linaro.org Received: by 2002:a2e:9754:0:0:0:0:0 with SMTP id f20-v6csp1603584ljj; Wed, 8 Aug 2018 21:25:06 -0700 (PDT) X-Google-Smtp-Source: AA+uWPxCP+lsZwPaJFFYieG2SNJZz4eQ/KM8fAkKpxhyKdoXvQ4/9JfRncYg+ReHf/DhKkEA82op X-Received: by 2002:a37:9404:: with SMTP id w4-v6mr413827qkd.50.1533788706809; Wed, 08 Aug 2018 21:25:06 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1533788706; cv=none; d=google.com; s=arc-20160816; b=irqpYJSfx256jpEgzq4QQMFo61bAHH+A2nDpZtFUBPifVMWVuP+iooIVwiPjjFKJkM 8HpN9Qf+2Sp5AvYeXMpc+1hts4KmI1SmcuKow4bG7fj/2AGIvs/cwK/ezXSIr3h7MqdD e7xOnUV2Z4uT4y4kehWm8RkKbMNtc/elMHFKjA1wEK7P+DNBXqI4+Qqn9e0onaAgkpCq 682TGqHXie+A9HLlNtJd1ItKh6Tzd2eojwuaxfYiug4eIkrbTmYZ94ksY+y9JZcBYsbh 2ETuED3shWxzssUDqY1yVxTqfu/ZhedPuPqYmQGMZ4Ag8bbKeddBPx+yQJfw1xozbqIE 4ftw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:cc:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:subject:references:in-reply-to :message-id:date:to:from:dkim-signature:arc-authentication-results; bh=mhyzR8Zi3Rp/BTnv7kPW11xxhgksRqlAPA2BPWvADYw=; b=iM9BeZQyEQ7XGhSizSuV0zo2WkT5JltWZ/U0CXaYO5UcrHMh8bc1zVwk8q0BYjEg/U bQ9cKqWt12ByciU/1qRtqcwvGpx1cYKqsZuX+tKSsxgqpe7tj1sNJ885827On3ugcexi mp1JQwkSUI6ViLWH7k7RKuEcXvTI9CqFePfovemLTaamAtzeFgIMJxAfu6k+2bQP4OBO AvwiW7x2d/JKyuMUOa+YJY0RWBv013uq2C5dS94DpKOu4bbc1eCmTzPYpYO8anJrlm4h NbP0+OEZZFpc5tjL6qMS7sJ/xapBMk5SqKYKJz5hfeNWi24QZqGz9QwGQLf2IIecCS8U XFXA== ARC-Authentication-Results: i=1; mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=fIF0bqpD; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [2001:4830:134:3::11]) by mx.google.com with ESMTPS id e60-v6si1963839qtb.145.2018.08.08.21.25.06 for (version=TLS1 cipher=AES128-SHA bits=128/128); Wed, 08 Aug 2018 21:25:06 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) client-ip=2001:4830:134:3::11; Authentication-Results: mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=fIF0bqpD; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1]:47231 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fncVW-0001vs-7f for patch@linaro.org; Thu, 09 Aug 2018 00:25:06 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:53943) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fncSj-0000i7-WA for qemu-devel@nongnu.org; Thu, 09 Aug 2018 00:22:15 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fncSh-000765-GT for qemu-devel@nongnu.org; Thu, 09 Aug 2018 00:22:13 -0400 Received: from mail-pl0-x22d.google.com ([2607:f8b0:400e:c01::22d]:36213) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fncSh-00075A-86 for qemu-devel@nongnu.org; Thu, 09 Aug 2018 00:22:11 -0400 Received: by mail-pl0-x22d.google.com with SMTP id e11-v6so1983787plb.3 for ; Wed, 08 Aug 2018 21:22:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=mhyzR8Zi3Rp/BTnv7kPW11xxhgksRqlAPA2BPWvADYw=; b=fIF0bqpDFVPskLNwzauX8/faeSLwEVcQH3ZBDNxRZrsIj8wJohhzJ1lg1ljcCewTwI 1emE/bfdHs4nnGIXi2jboOHjLhfw9x1S1cbiRHbUOwqdAoDTuicigKfBEvbcCqNVRUXy 3LAA2nYhXQmGW5icuNdmLyU+ULurspIwo8oWo= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=mhyzR8Zi3Rp/BTnv7kPW11xxhgksRqlAPA2BPWvADYw=; b=cPYouqwux++YO5f0pYN0RrESUmdCiubaMIQ0iXunE/sG+vNmPhj9T28M13L0HymKPo ALaNSnZoT23g22ta5tX9wfpt5a6iJ30A+WMFy8qowhqjbgTaNZH8VzLzmEJ3q30VvIvw Rm+p2KSjSAdTUSb6hG4mi2bkOIdw3LUBk9lUseEmz81W2lb6dSNnIGIFWWQZ471WDyQn E/ByoyguPh5DNn5jdBkvTYKqaR3kJhjZ2AOxECmsPFrwMttE9+j0BSgcgRnZVqnGO8M/ EsVSU3A5Dk7vgDg+Sq4dO1VlBghQ1hnJys7NGLEyE7xAR1fbldtzxNvqMrXH8hyPJo+C 7nJw== X-Gm-Message-State: AOUpUlE+rNONkjtx+tgCNyBOC+I1QcDWP0TFORzO7t8li/LMZU/mVJDW a+gF8SY/C2b+7w+3JiQA55VKQm/EgW0= X-Received: by 2002:a17:902:b784:: with SMTP id e4-v6mr503387pls.185.1533788529963; Wed, 08 Aug 2018 21:22:09 -0700 (PDT) Received: from cloudburst.twiddle.net (97-113-8-179.tukw.qwest.net. [97.113.8.179]) by smtp.gmail.com with ESMTPSA id m30-v6sm7355799pff.121.2018.08.08.21.22.08 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Wed, 08 Aug 2018 21:22:08 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Date: Wed, 8 Aug 2018 21:21:47 -0700 Message-Id: <20180809042206.15726-2-richard.henderson@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180809042206.15726-1-richard.henderson@linaro.org> References: <20180809042206.15726-1-richard.henderson@linaro.org> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c01::22d Subject: [Qemu-devel] [PATCH 01/20] target/arm: Set ISAR bits for -cpu max X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: laurent.desnogues@gmail.com, peter.maydell@linaro.org, alex.bennee@linaro.org Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: "Qemu-devel" For the supported extensions, fill in the appropriate bits in ID_ISAR5, ID_ISAR6, ID_AA64ISAR0, ID_AA64ISAR1. Signed-off-by: Richard Henderson --- target/arm/cpu.c | 24 +++++++++++++++++------- target/arm/cpu64.c | 36 ++++++++++++++++++++++++++++-------- 2 files changed, 45 insertions(+), 15 deletions(-) -- 2.17.1 diff --git a/target/arm/cpu.c b/target/arm/cpu.c index b25898ed4c..71daa39e86 100644 --- a/target/arm/cpu.c +++ b/target/arm/cpu.c @@ -1802,19 +1802,29 @@ static void arm_max_initfn(Object *obj) kvm_arm_set_cpu_features_from_host(cpu); } else { cortex_a15_initfn(obj); + + set_feature(&cpu->env, ARM_FEATURE_V8_AES); + cpu->id_isar5 = deposit32(cpu->id_isar5, 4, 4, 2); + set_feature(&cpu->env, ARM_FEATURE_V8_SHA1); + cpu->id_isar5 = deposit32(cpu->id_isar5, 8, 4, 1); + set_feature(&cpu->env, ARM_FEATURE_V8_SHA256); + cpu->id_isar5 = deposit32(cpu->id_isar5, 12, 4, 1); + set_feature(&cpu->env, ARM_FEATURE_CRC); + cpu->id_isar5 = deposit32(cpu->id_isar5, 16, 4, 1); + set_feature(&cpu->env, ARM_FEATURE_V8_RDM); + cpu->id_isar5 = deposit32(cpu->id_isar5, 24, 4, 1); + set_feature(&cpu->env, ARM_FEATURE_V8_FCMA); + cpu->id_isar5 = deposit32(cpu->id_isar5, 28, 4, 1); + + set_feature(&cpu->env, ARM_FEATURE_V8_DOTPROD); + cpu->id_isar6 = deposit32(cpu->id_isar6, 4, 4, 1); + #ifdef CONFIG_USER_ONLY /* We don't set these in system emulation mode for the moment, * since we don't correctly set the ID registers to advertise them, */ set_feature(&cpu->env, ARM_FEATURE_V8); - set_feature(&cpu->env, ARM_FEATURE_V8_AES); - set_feature(&cpu->env, ARM_FEATURE_V8_SHA1); - set_feature(&cpu->env, ARM_FEATURE_V8_SHA256); set_feature(&cpu->env, ARM_FEATURE_V8_PMULL); - set_feature(&cpu->env, ARM_FEATURE_CRC); - set_feature(&cpu->env, ARM_FEATURE_V8_RDM); - set_feature(&cpu->env, ARM_FEATURE_V8_DOTPROD); - set_feature(&cpu->env, ARM_FEATURE_V8_FCMA); #endif } } diff --git a/target/arm/cpu64.c b/target/arm/cpu64.c index 800bff780e..4d629bb99b 100644 --- a/target/arm/cpu64.c +++ b/target/arm/cpu64.c @@ -254,6 +254,34 @@ static void aarch64_max_initfn(Object *obj) kvm_arm_set_cpu_features_from_host(cpu); } else { aarch64_a57_initfn(obj); + + set_feature(&cpu->env, ARM_FEATURE_V8_SHA512); + cpu->id_aa64isar0 = deposit64(cpu->id_aa64isar0, 12, 4, 2); + + set_feature(&cpu->env, ARM_FEATURE_V8_ATOMICS); + cpu->id_aa64isar0 = deposit64(cpu->id_aa64isar0, 20, 4, 2); + + set_feature(&cpu->env, ARM_FEATURE_V8_RDM); + cpu->id_aa64isar0 = deposit64(cpu->id_aa64isar0, 28, 4, 1); + cpu->id_isar5 = deposit32(cpu->id_isar5, 24, 4, 1); + + set_feature(&cpu->env, ARM_FEATURE_V8_SHA3); + cpu->id_aa64isar0 = deposit64(cpu->id_aa64isar0, 32, 4, 1); + + set_feature(&cpu->env, ARM_FEATURE_V8_SM3); + cpu->id_aa64isar0 = deposit64(cpu->id_aa64isar0, 36, 4, 1); + + set_feature(&cpu->env, ARM_FEATURE_V8_SM4); + cpu->id_aa64isar0 = deposit64(cpu->id_aa64isar0, 40, 4, 1); + + set_feature(&cpu->env, ARM_FEATURE_V8_DOTPROD); + cpu->id_aa64isar0 = deposit64(cpu->id_aa64isar0, 44, 4, 1); + cpu->id_isar6 = deposit32(cpu->id_isar6, 4, 4, 1); + + set_feature(&cpu->env, ARM_FEATURE_V8_FCMA); + cpu->id_aa64isar1 = deposit64(cpu->id_aa64isar1, 16, 4, 1); + cpu->id_isar5 = deposit32(cpu->id_isar5, 28, 4, 1); + #ifdef CONFIG_USER_ONLY /* We don't set these in system emulation mode for the moment, * since we don't correctly set the ID registers to advertise them, @@ -261,15 +289,7 @@ static void aarch64_max_initfn(Object *obj) * whereas the architecture requires them to be present in both if * present in either. */ - set_feature(&cpu->env, ARM_FEATURE_V8_SHA512); - set_feature(&cpu->env, ARM_FEATURE_V8_SHA3); - set_feature(&cpu->env, ARM_FEATURE_V8_SM3); - set_feature(&cpu->env, ARM_FEATURE_V8_SM4); - set_feature(&cpu->env, ARM_FEATURE_V8_ATOMICS); - set_feature(&cpu->env, ARM_FEATURE_V8_RDM); - set_feature(&cpu->env, ARM_FEATURE_V8_DOTPROD); set_feature(&cpu->env, ARM_FEATURE_V8_FP16); - set_feature(&cpu->env, ARM_FEATURE_V8_FCMA); set_feature(&cpu->env, ARM_FEATURE_SVE); /* For usermode -cpu max we can use a larger and more efficient DCZ * blocksize since we don't have to follow what the hardware does. From patchwork Thu Aug 9 04:21:48 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 143656 Delivered-To: patch@linaro.org Received: by 2002:a2e:9754:0:0:0:0:0 with SMTP id f20-v6csp1601791ljj; Wed, 8 Aug 2018 21:22:43 -0700 (PDT) X-Google-Smtp-Source: AA+uWPzuCxRUO2/7o6LyzxY0xh5m1XluLj7Kv0ZKgRi2Oh+o0Og1i+L2pV6YNsFlwcJamywZXuiq X-Received: by 2002:ac8:1987:: with SMTP id u7-v6mr455573qtj.296.1533788563873; Wed, 08 Aug 2018 21:22:43 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1533788563; cv=none; d=google.com; s=arc-20160816; b=xwnd8aui3VPM/c64gVt8Hp8xsyfeBvcX/HxnTq1MvKPu6JVUz/Nst2AzKjyVbesQqo RfVIEAotgFAmm8alTHwRiXqZdvzd53VHMMoqe4udivhh37J51SIgc8rDYIma3mmRFWwB 5gLwhdC6PgIHwABEsvnOz+Lyeo5T4ekMwRCYmBusC+7iuJNJvKSBtj9wWr6tS4qU01oo z+sb4+Wj+oSLsphFuNqkYEF5nT4HWJbN4FrqQiIAJnCJ3tseTEtN4bvGpvrTJ5PPnAy2 /fIhOoVopJLf2CqsaTFlYyWUZZU3UVw+b4v44PPYq04/DtFovd+iw8g6Hlj6UZRNtk+g CbTg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:cc:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:subject:references:in-reply-to :message-id:date:to:from:dkim-signature:arc-authentication-results; bh=megdeTjYGDi91jW7S3Vt/13VC1dSMwSCHADtSBmt828=; b=xuW7cEcZIDRanHSWCmEjAYTiA5R0OzJyTgChI0S76JFKdZEfcqxOd/yMDlko3ES8VR JWnd2E1GR/bI70kPbHE6uk87Txq/toyfCCxqA67xbVnG6tQVXH4tM6O2xJ0eoG6yf15M f9WCcbnCw7XDKGUJ0qiggfQXGeu9WppGv/tY8u0HR98tpUL2XLZN7KOz9KLTxuVug9lW m7EDuVSkhI/3G38tAJVZRYWhb9j+U3Bp/NiKjPS1nCxqNdguq+C889ecvBIrfjuPRRyL +Qt/zgxtPFAt+ei8EbO5jnBVLvnjTHPpIFeNbk5g1LYtuYVnSJTaXMzdgEIBpH6gnLkS fE2A== ARC-Authentication-Results: i=1; mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=OCfbar2s; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [2001:4830:134:3::11]) by mx.google.com with ESMTPS id m24-v6si6088116qkl.241.2018.08.08.21.22.43 for (version=TLS1 cipher=AES128-SHA bits=128/128); Wed, 08 Aug 2018 21:22:43 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) client-ip=2001:4830:134:3::11; Authentication-Results: mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=OCfbar2s; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1]:47226 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fncTD-0000jS-AY for patch@linaro.org; Thu, 09 Aug 2018 00:22:43 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:53946) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fncSk-0000i8-1G for qemu-devel@nongnu.org; Thu, 09 Aug 2018 00:22:14 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fncSi-00077U-VJ for qemu-devel@nongnu.org; Thu, 09 Aug 2018 00:22:14 -0400 Received: from mail-pl0-x244.google.com ([2607:f8b0:400e:c01::244]:43992) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fncSi-00076q-Nh for qemu-devel@nongnu.org; Thu, 09 Aug 2018 00:22:12 -0400 Received: by mail-pl0-x244.google.com with SMTP id x6-v6so1972192plv.10 for ; Wed, 08 Aug 2018 21:22:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=megdeTjYGDi91jW7S3Vt/13VC1dSMwSCHADtSBmt828=; b=OCfbar2s3AvvtrRP2flIyuJebUWgnz4OsnW0S0NCUTAnucXZcf8umReYanVR0GznoY dR7QZedSuwvcne4ZAaoLiKzJOWw/pFipah8nrwnf+d5szluxhEtmOfU047tPHtBy8Y6O IpdENAjxiNrQiSwXCwno1sIqbwaP2enuXtBEQ= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=megdeTjYGDi91jW7S3Vt/13VC1dSMwSCHADtSBmt828=; b=soJ+BSAUNXCdMmpu+YDun/NM0y51iZNPPKbWiXAedCFI63PS8BChxEFVXINXDwtRRx tmr7WzpFkeTCLH/V6n01CNYoyOmP9Rajfpgq2vL5BWFWR6+WQ33u4MdA9TFoVNrrh1ql yQ4DdSTFror2oljfak4cykl9oRKuyI6L1hLwl25kZIVHlg1s+MItQLDodpzjIqxBTOry v+f0GgJVrXEWSCpULwbelmU+vGrSpTh6Ix3h24vFyxWNSS2UC1BO/qEhDPp85r79pRND Z+n1iGJCqxQuBAB3ub5uGb1wFkpHlPmXW6biyvfBYbBIFjF6kJjpz0QualUjOs25Y4Jb YxPA== X-Gm-Message-State: AOUpUlFKmldz7tv4uRYxhK2OG3VdG+KT4K7xl5S2R7wccQY0+FAbfR1k x+IsEBV6YsvBYcH7ovSgrUzUtNtABjg= X-Received: by 2002:a17:902:163:: with SMTP id 90-v6mr501677plb.322.1533788531458; Wed, 08 Aug 2018 21:22:11 -0700 (PDT) Received: from cloudburst.twiddle.net (97-113-8-179.tukw.qwest.net. [97.113.8.179]) by smtp.gmail.com with ESMTPSA id m30-v6sm7355799pff.121.2018.08.08.21.22.10 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Wed, 08 Aug 2018 21:22:10 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Date: Wed, 8 Aug 2018 21:21:48 -0700 Message-Id: <20180809042206.15726-3-richard.henderson@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180809042206.15726-1-richard.henderson@linaro.org> References: <20180809042206.15726-1-richard.henderson@linaro.org> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c01::244 Subject: [Qemu-devel] [PATCH 02/20] target/arm: Set ID_AA64PFR0 bits for SVE for -cpu max X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: laurent.desnogues@gmail.com, peter.maydell@linaro.org, alex.bennee@linaro.org Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: "Qemu-devel" This it a hair out of spec in that we have and advertise, support for fp16 in aarch64 mode, but do not have nor advertise the same in aarch32 mode. Rationale as commented. Signed-off-by: Richard Henderson --- target/arm/cpu64.c | 21 +++++++++++++++------ 1 file changed, 15 insertions(+), 6 deletions(-) -- 2.17.1 diff --git a/target/arm/cpu64.c b/target/arm/cpu64.c index 4d629bb99b..ae650b608e 100644 --- a/target/arm/cpu64.c +++ b/target/arm/cpu64.c @@ -282,15 +282,24 @@ static void aarch64_max_initfn(Object *obj) cpu->id_aa64isar1 = deposit64(cpu->id_aa64isar1, 16, 4, 1); cpu->id_isar5 = deposit32(cpu->id_isar5, 28, 4, 1); -#ifdef CONFIG_USER_ONLY - /* We don't set these in system emulation mode for the moment, - * since we don't correctly set the ID registers to advertise them, - * and in some cases they're only available in AArch64 and not AArch32, - * whereas the architecture requires them to be present in both if - * present in either. + /* TODO: This is not yet implemented for AArch32, whereas the + * architecture requires a feature to be present in both if + * it is present in either. However, it is required by SVE, + * so we don't want to leave it out of AArch64 state. + * + * Practically, the Linux kernel does not query the MVFR1 bit + * nor expose this as a HWCAP bit to AArch32 userland. Thus + * userland, if it wanted to use fp16, would have to probe for + * support by executing an insn and checking for SIGILL. + * At which point it will get the correct answer: unsupported. */ set_feature(&cpu->env, ARM_FEATURE_V8_FP16); + cpu->id_aa64pfr0 = deposit64(cpu->id_aa64pfr0, 20, 4, 1); + set_feature(&cpu->env, ARM_FEATURE_SVE); + cpu->id_aa64pfr0 = deposit64(cpu->id_aa64pfr0, 32, 4, 1); + +#ifdef CONFIG_USER_ONLY /* For usermode -cpu max we can use a larger and more efficient DCZ * blocksize since we don't have to follow what the hardware does. */ From patchwork Thu Aug 9 04:21:49 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 143661 Delivered-To: patch@linaro.org Received: by 2002:a2e:9754:0:0:0:0:0 with SMTP id f20-v6csp1605647ljj; Wed, 8 Aug 2018 21:27:56 -0700 (PDT) X-Google-Smtp-Source: AA+uWPy/TRtzqKbQMHR7qtk0qonyFaXhWJolhxy9vbRjT4T/ZrYGuPw9fUUF6YDmgFMyOSpRNwOo X-Received: by 2002:ac8:247b:: with SMTP id d56-v6mr473761qtd.131.1533788876332; Wed, 08 Aug 2018 21:27:56 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1533788876; cv=none; d=google.com; s=arc-20160816; b=gk28T9aMH1TiwmnkbX4OsM0tgd4HV9CNZ6Yf8wAWeF8rqolO1ZVHYbZLTrO3l2HTUP jiJeGHNvFH0eIeAgp6LO+FVKfhJbYbyEJ90dQZvOz0eWJi67v2hOu1rNAK7kf+E0Opnr Wp+b2G42S1tuhdZs+EvuqO56teqstZ/UpNbpXjpS3pQBOcYL/CGOrF3TXMCpDvCFLLbd 0SKAncR9Lam7VIJL+wLDEm6Tdvpclt5uHjoqxLiHsjCsEpidGcWiyv2R9isZ3T2QszCN Fb7xY7wqw1ajlOGTEqWmampcdudPM+PIvSGBAWVG2bF2Hj76apqmUOQYM9F7TPRByS8U opDQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:cc:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:subject:references:in-reply-to :message-id:date:to:from:dkim-signature:arc-authentication-results; bh=sHtxaTZ3pFPqXL33e8EqAOyUhtc5YlxCG2/QzEfKR8M=; b=SZEAKuPz1FZNxKajUdIM3wWDGfzJgZjZBgXs3F9IT4YzaywfqYRk8zQi2qPvV4DU2r u+i006J1vbcZ2dYfY/dgl2dHIvCRv0knYwiwDemAcefDizw5IilrSk7eDUIq6U7NvXw/ sAkO0Kz5wFDhGb/rcvtL0W17xsb7Es8b9rkllCOxzOvDkTCjugPqj2DgasHxgmgFBVKg hoUsmeahDyNBpulxDvfHRehEOS1K1037NJC5MkV3IE/rErn8XO0T8YzKuQexnq9OnuK8 BgCowpe0YKWXoHBMu9ufnjV9HMoRbqxA+hQv3mPdGSK3y1+9NZz3NsF8AiJWnbZVwQE3 UN8Q== ARC-Authentication-Results: i=1; mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=CGNVDZ3c; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [2001:4830:134:3::11]) by mx.google.com with ESMTPS id g28-v6si1555971qvi.238.2018.08.08.21.27.56 for (version=TLS1 cipher=AES128-SHA bits=128/128); Wed, 08 Aug 2018 21:27:56 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) client-ip=2001:4830:134:3::11; Authentication-Results: mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=CGNVDZ3c; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1]:47248 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fncYF-0004jg-OJ for patch@linaro.org; Thu, 09 Aug 2018 00:27:55 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:53964) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fncSk-0000iA-T5 for qemu-devel@nongnu.org; Thu, 09 Aug 2018 00:22:15 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fncSk-00078h-5S for qemu-devel@nongnu.org; Thu, 09 Aug 2018 00:22:14 -0400 Received: from mail-pf1-x444.google.com ([2607:f8b0:4864:20::444]:35660) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fncSj-00077r-Vo for qemu-devel@nongnu.org; Thu, 09 Aug 2018 00:22:14 -0400 Received: by mail-pf1-x444.google.com with SMTP id p12-v6so2206894pfh.2 for ; Wed, 08 Aug 2018 21:22:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=sHtxaTZ3pFPqXL33e8EqAOyUhtc5YlxCG2/QzEfKR8M=; b=CGNVDZ3cQrMzkwwUBOa6u0eirQFS4XM4iecvxbdEdbVy39MnFQ5ol6q9Uf/gYJQhWm S6XoL+BTUy/Bngb4ReLuT/NDPYaoyLd+IdATFFaGF9NIuVPvMr9pncEnuPO+OrGhbg8m gPIvYuGIJeBxrwFFtDdJV1a9B350kTnUphuyg= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=sHtxaTZ3pFPqXL33e8EqAOyUhtc5YlxCG2/QzEfKR8M=; b=NiqGJzyuXNyDAvAfFdxDHb03ScpvMtLH9Ij1b1A1v6SXiwmNoobT1jGmhJOUF/aOQV HOdnM0RyqG0Gcb8BqhUL54lOvEej/2/ATrkRILODp2HtxCAWKN7PSEIENnimr+F4jbls zIcV3sqmbrYh7VBmLB5NWOz+VOTpwQxgH9lYsv/3Enxf2VzKdrvmqxZ8BCBZS4FDXoh0 h8H4zpP6XjVeip8mSSdLXttTYObJhLM5MBS0eGNs8HbEOAp3r4M/zA7Z9Ye5eCWW3F2X 4nvPpy6FD/WkoqHVZhcPSsCjTV+aoEgOx89xKXBy7gyo7WzfTyxYwhOd5NefB6Uphjfb X5fw== X-Gm-Message-State: AOUpUlGuCHWGncj2PHoVOuidE3Z0B+5JRYcxuezusFcXpRT+GgdSuJLl SFyP0H5yoLbLc8eOx9xrDsnViSlmo6k= X-Received: by 2002:a62:a216:: with SMTP id m22-v6mr542167pff.163.1533788532722; Wed, 08 Aug 2018 21:22:12 -0700 (PDT) Received: from cloudburst.twiddle.net (97-113-8-179.tukw.qwest.net. [97.113.8.179]) by smtp.gmail.com with ESMTPSA id m30-v6sm7355799pff.121.2018.08.08.21.22.11 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Wed, 08 Aug 2018 21:22:11 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Date: Wed, 8 Aug 2018 21:21:49 -0700 Message-Id: <20180809042206.15726-4-richard.henderson@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180809042206.15726-1-richard.henderson@linaro.org> References: <20180809042206.15726-1-richard.henderson@linaro.org> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::444 Subject: [Qemu-devel] [PATCH 03/20] target/arm: Define ID_AA64ZFR0_EL1 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: laurent.desnogues@gmail.com, peter.maydell@linaro.org, alex.bennee@linaro.org Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: "Qemu-devel" Given that the only field defined for this new register may only be 0, we don't actually need to change anything except the name. Signed-off-by: Richard Henderson --- target/arm/helper.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) -- 2.17.1 Reviewed-by: Peter Maydell diff --git a/target/arm/helper.c b/target/arm/helper.c index c24c66d43e..61a79e4c44 100644 --- a/target/arm/helper.c +++ b/target/arm/helper.c @@ -4956,9 +4956,10 @@ void register_cp_regs_for_features(ARMCPU *cpu) .opc0 = 3, .opc1 = 0, .crn = 0, .crm = 4, .opc2 = 3, .access = PL1_R, .type = ARM_CP_CONST, .resetvalue = 0 }, - { .name = "ID_AA64PFR4_EL1_RESERVED", .state = ARM_CP_STATE_AA64, + { .name = "ID_AA64ZFR0_EL1", .state = ARM_CP_STATE_AA64, .opc0 = 3, .opc1 = 0, .crn = 0, .crm = 4, .opc2 = 4, .access = PL1_R, .type = ARM_CP_CONST, + /* At present, only SVEver == 0 is defined anyway. */ .resetvalue = 0 }, { .name = "ID_AA64PFR5_EL1_RESERVED", .state = ARM_CP_STATE_AA64, .opc0 = 3, .opc1 = 0, .crn = 0, .crm = 4, .opc2 = 5, From patchwork Thu Aug 9 04:21:50 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 143657 Delivered-To: patch@linaro.org Received: by 2002:a2e:9754:0:0:0:0:0 with SMTP id f20-v6csp1601831ljj; Wed, 8 Aug 2018 21:22:47 -0700 (PDT) X-Google-Smtp-Source: AA+uWPz1TdqSJ/giPaipp9C6X9hnlvJZGuCsoHfy6zVNtW+nqd6iVSzHBi2bF0c2Mrh1aSCP/pm8 X-Received: by 2002:ac8:2bb3:: with SMTP id m48-v6mr438158qtm.207.1533788567072; Wed, 08 Aug 2018 21:22:47 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1533788567; cv=none; d=google.com; s=arc-20160816; b=wR4bc1gbULHN0IvMc+QIp07MhK9EPYM+xKSX4aCfCY4L2+IZOvj9bLZrwaYnVOjVL0 1DQef+FmQagbRAxFF6rWmZVoB7YMyniU8Cg9ozm+DNGCJbTUXz5Y7LGWhdrMJqh3k+UI fM8RMkwTS1xZcWz3MUeGji1sdET4mG7Y5chxOwULtJjT0JN+XSh+OEwkSJxLsoFmUTUc Ll05JJ+7qs17eIDyABMX0gvt141fuFe9lV8NlICjV4cUPKj5Q8h/Jn8YEpJ2FEOtTzr9 hyaIFx2Mi6RXt/CPs7V7iUn+dZYaHSQTE7RESTCJMhRf7BhMV20aGvWO9HEJrKt/JvfF 4JLw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:cc:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:subject:references:in-reply-to :message-id:date:to:from:dkim-signature:arc-authentication-results; bh=GHNmLdKb7IekLRFejxMlqEkz2+Rx9y/cL35+qY06ZJM=; b=iO/eo8ZnWeZ9Mlx237sMV4D1W4KRrgXmDUvWEg0Tj/38y+/XsAjMjdl8WLrzOzaZ7X VIWoM/cLdW96cVaozsoOc/e/3vPPD/hbjSFaZneUT3fhGLlMAxvOnaN02BjVrJ6kq4v1 Vrkn7vaD58mlmyJhfLMpOZyOhH7IkUyru3vtgdpirWcgVRFm9WwJ8YbASeZJ8nBbJWXs BMV3b+ROFhYfFGlcG3ImDBD68NNYDzdyldjHkUZ1WDPsHIC6cZgDz/qj5Kw54iGcJ5Gb fYly1ZryXt4NNOm6C5ErOp1R3KROdnTudOy1Wi+plQjGFgy8HF0PDBS3I5jH7EVrRHwn 8PHw== ARC-Authentication-Results: i=1; mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=em5yQ7pL; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [2001:4830:134:3::11]) by mx.google.com with ESMTPS id g2-v6si5175034qtj.99.2018.08.08.21.22.46 for (version=TLS1 cipher=AES128-SHA bits=128/128); Wed, 08 Aug 2018 21:22:47 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) client-ip=2001:4830:134:3::11; Authentication-Results: mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=em5yQ7pL; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1]:47227 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fncTG-0000lY-Ce for patch@linaro.org; Thu, 09 Aug 2018 00:22:46 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:53976) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fncSm-0000im-Im for qemu-devel@nongnu.org; Thu, 09 Aug 2018 00:22:17 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fncSl-00079t-Id for qemu-devel@nongnu.org; Thu, 09 Aug 2018 00:22:16 -0400 Received: from mail-pf1-x42f.google.com ([2607:f8b0:4864:20::42f]:36973) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fncSl-00079K-BP for qemu-devel@nongnu.org; Thu, 09 Aug 2018 00:22:15 -0400 Received: by mail-pf1-x42f.google.com with SMTP id a26-v6so2199430pfo.4 for ; Wed, 08 Aug 2018 21:22:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=GHNmLdKb7IekLRFejxMlqEkz2+Rx9y/cL35+qY06ZJM=; b=em5yQ7pLvsBPoszuY7qqe8UwNH/udxcSZv7GQRxhL3qVe+mO0iGmwmePeroWGxB+1j Emiod/sDslAJNNdD7tnZyQfP8YxE+BhcUHRqf7bRptAiH/LBOJSmxXID+eSqe0Bd5Gty BE4Hw8pOu6GwI+glvstznL7oNHAbCiU3BNd64= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=GHNmLdKb7IekLRFejxMlqEkz2+Rx9y/cL35+qY06ZJM=; b=B5a/PZS38Xcu23QzyhLimp4dG9IvvI/dd0Se+oIWb6pZTbJ9FUJFcTsp9MG8Z+l5+t lYYz7CBODR5fC3Qr9fu9Y5wzTzLiCSulRbad+K87u/vBNChVfMi1EbiwnTxx2I5umMNS /zE7A5GzSxaoqVCQ29tLX1/2HnV3mCoNYGqImWKsNhCMHKQ37jVwXLaTLE3p6NNyR6T6 asH9gl1fNC8zrSjKdcpjvjInHxydt8Vyuek6ykMkVszPCU+JMYiBd4z6VdDFx+WsjMHa 1nudVxDuw+dJyJRrab0wAlyB5b8nounnGo6TLsob+NQ8g9I7wdrilQdEmWITInQzvqJX axYw== X-Gm-Message-State: AOUpUlFQoJfShMiSFIPX8Z52JmNBEXSNSFrFjmdYdyz5KN9yGaClsJLy U62THde9hiNL+MUjyG2xlDq6+I/i3Jc= X-Received: by 2002:a62:d45d:: with SMTP id u29-v6mr568852pfl.51.1533788534103; Wed, 08 Aug 2018 21:22:14 -0700 (PDT) Received: from cloudburst.twiddle.net (97-113-8-179.tukw.qwest.net. [97.113.8.179]) by smtp.gmail.com with ESMTPSA id m30-v6sm7355799pff.121.2018.08.08.21.22.12 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Wed, 08 Aug 2018 21:22:13 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Date: Wed, 8 Aug 2018 21:21:50 -0700 Message-Id: <20180809042206.15726-5-richard.henderson@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180809042206.15726-1-richard.henderson@linaro.org> References: <20180809042206.15726-1-richard.henderson@linaro.org> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::42f Subject: [Qemu-devel] [PATCH 04/20] target/arm: Adjust sve_exception_el X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: laurent.desnogues@gmail.com, peter.maydell@linaro.org, alex.bennee@linaro.org Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: "Qemu-devel" Check for EL3 before testing CPTR_EL3.EZ. Return 0 when the exception should be routed via AdvSIMDFPAccessTrap. Mirror the structure of CheckSVEEnabled more closely. Fixes: 5be5e8eda78 Signed-off-by: Richard Henderson --- target/arm/helper.c | 96 ++++++++++++++++++++++----------------------- 1 file changed, 46 insertions(+), 50 deletions(-) -- 2.17.1 Reviewed-by: Peter Maydell diff --git a/target/arm/helper.c b/target/arm/helper.c index 61a79e4c44..26e9098c5f 100644 --- a/target/arm/helper.c +++ b/target/arm/helper.c @@ -4338,67 +4338,63 @@ static const ARMCPRegInfo debug_lpae_cp_reginfo[] = { REGINFO_SENTINEL }; -/* Return the exception level to which SVE-disabled exceptions should - * be taken, or 0 if SVE is enabled. +/* Return the exception level to which exceptions should be taken + * via SVEAccessTrap. If an exception should be routed through + * AArch64.AdvSIMDFPAccessTrap, return 0; fp_exception_el should + * take care of raising that exception. + * C.f. the ARM pseudocode function CheckSVEEnabled. */ static int sve_exception_el(CPUARMState *env) { #ifndef CONFIG_USER_ONLY unsigned current_el = arm_current_el(env); - /* The CPACR.ZEN controls traps to EL1: - * 0, 2 : trap EL0 and EL1 accesses - * 1 : trap only EL0 accesses - * 3 : trap no accesses + if (current_el <= 1) { + bool disabled = false; + + /* The CPACR.ZEN controls traps to EL1: + * 0, 2 : trap EL0 and EL1 accesses + * 1 : trap only EL0 accesses + * 3 : trap no accesses + */ + if (!extract32(env->cp15.cpacr_el1, 16, 1)) { + disabled = true; + } else if (!extract32(env->cp15.cpacr_el1, 17, 1)) { + disabled = current_el == 0; + } + if (disabled) { + /* route_to_el2 */ + return (arm_feature(env, ARM_FEATURE_EL2) + && !arm_is_secure(env) + && (env->cp15.hcr_el2 & HCR_TGE) ? 2 : 1); + } + + /* Check CPACR.FPEN. */ + if (!extract32(env->cp15.cpacr_el1, 20, 1)) { + disabled = true; + } else if (!extract32(env->cp15.cpacr_el1, 21, 1)) { + disabled = current_el == 0; + } + if (disabled) { + return 0; + } + } + + /* CPTR_EL2. Since TZ and TFP are positive, + * they will be zero when EL2 is not present. */ - switch (extract32(env->cp15.cpacr_el1, 16, 2)) { - default: - if (current_el <= 1) { - /* Trap to PL1, which might be EL1 or EL3 */ - if (arm_is_secure(env) && !arm_el_is_aa64(env, 3)) { - return 3; - } - return 1; + if (current_el <= 2 && !arm_is_secure_below_el3(env)) { + if (env->cp15.cptr_el[2] & CPTR_TZ) { + return 2; } - break; - case 1: - if (current_el == 0) { - return 1; + if (env->cp15.cptr_el[2] & CPTR_TFP) { + return 0; } - break; - case 3: - break; } - /* Similarly for CPACR.FPEN, after having checked ZEN. */ - switch (extract32(env->cp15.cpacr_el1, 20, 2)) { - default: - if (current_el <= 1) { - if (arm_is_secure(env) && !arm_el_is_aa64(env, 3)) { - return 3; - } - return 1; - } - break; - case 1: - if (current_el == 0) { - return 1; - } - break; - case 3: - break; - } - - /* CPTR_EL2. Check both TZ and TFP. */ - if (current_el <= 2 - && (env->cp15.cptr_el[2] & (CPTR_TFP | CPTR_TZ)) - && !arm_is_secure_below_el3(env)) { - return 2; - } - - /* CPTR_EL3. Check both EZ and TFP. */ - if (!(env->cp15.cptr_el[3] & CPTR_EZ) - || (env->cp15.cptr_el[3] & CPTR_TFP)) { + /* CPTR_EL3. Since EZ is negative we must check for EL3. */ + if (arm_feature(env, ARM_FEATURE_EL3) + && !(env->cp15.cptr_el[3] & CPTR_EZ)) { return 3; } #endif From patchwork Thu Aug 9 04:21:51 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 143660 Delivered-To: patch@linaro.org Received: by 2002:a2e:9754:0:0:0:0:0 with SMTP id f20-v6csp1605633ljj; Wed, 8 Aug 2018 21:27:55 -0700 (PDT) X-Google-Smtp-Source: AA+uWPxnjbocynL3ZBnWlae186/yycDS3X8Dv/NI4VYExdmk2pnqRCttcm7FMhki5fOLbvanBAmF X-Received: by 2002:a0c:891a:: with SMTP id 26-v6mr410274qvp.195.1533788875424; Wed, 08 Aug 2018 21:27:55 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1533788875; cv=none; d=google.com; s=arc-20160816; b=kICgj3F1CbVCNp3Wwf/OW40Ok/EhZZ8OMkSKzjZCkxWG8/usPMxQQfgZ2QJSBdepDv cqu++KkbeVHtQfEIZEu3GKf55cCPU96CAUqUAJnDpP4YCgyoloHmlPj8ZtvzhwIrll8V Mc/NT1DeLLlEXE8jhYPsYbDQclayEM1T9T3CBTzwyGc455oNf6PA4ysCguYWzbQtep8W YP4v+m9Prd9EKtv3QtDWoTvH1ziUIGHmNbCIRWU/L7FDlO9zxmLyGL9n7h+PSdYMrp77 7Ds5APQlDN1H6SvotRFxZO0l4Ta+s3Kp0pmboHpjGilP+SekHzEn+o5aN9UMpZD3SE+O qjRA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:cc:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:subject:references:in-reply-to :message-id:date:to:from:dkim-signature:arc-authentication-results; bh=sg+bJaDUMi6cidJwpuhz8YqPDcihHRJslrVPGNZ4Mos=; b=hAKFFp11B3ieGuBBPl30mLfcERm5QecjN8ma2qCJ33IZSVV+yLP6R3fFInyKphjnbp /wZG7dQZ0Si1J9n11BuDnn9SSD4uVnUWQi+ZUJhiAzabgUNDPl1pgqbcOwAjX/Wn95Z2 EgnURGLtHu0B4AyXT1BptpaAWsOyfViBZ7Oxzo3if0o9nB6WmLoS+noPpFraaVqdG4k5 ngGpbfMdtDwKMIV1VGvjSsgKYEet/JIXZarq9uokEtxXlle+h+XPCOlAZ7r31g23ZDgF pXitP/8NXwUw+W4w1dFFa4koThnLhiM9pneSazmvqj1NhWDQDtVF3Yd813u7Hx+TLfSv UOCA== ARC-Authentication-Results: i=1; mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=Mcda91sy; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [2001:4830:134:3::11]) by mx.google.com with ESMTPS id s63-v6si3963381qkc.404.2018.08.08.21.27.55 for (version=TLS1 cipher=AES128-SHA bits=128/128); Wed, 08 Aug 2018 21:27:55 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) client-ip=2001:4830:134:3::11; Authentication-Results: mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=Mcda91sy; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1]:47247 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fncYE-0004ic-NG for patch@linaro.org; Thu, 09 Aug 2018 00:27:54 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:53988) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fncSn-0000jl-Nz for qemu-devel@nongnu.org; Thu, 09 Aug 2018 00:22:18 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fncSm-0007BU-S5 for qemu-devel@nongnu.org; Thu, 09 Aug 2018 00:22:17 -0400 Received: from mail-pf1-x431.google.com ([2607:f8b0:4864:20::431]:44430) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fncSm-0007AK-ML for qemu-devel@nongnu.org; Thu, 09 Aug 2018 00:22:16 -0400 Received: by mail-pf1-x431.google.com with SMTP id k21-v6so2183463pff.11 for ; Wed, 08 Aug 2018 21:22:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=sg+bJaDUMi6cidJwpuhz8YqPDcihHRJslrVPGNZ4Mos=; b=Mcda91syyAH9u8d5quYw0bmttmt+OV2kvugDfqhywYgYFHt4pehwttOTFov106bZ8e CkZtCUCVBf/NPw4foPr+pKGtUnw9x9FTp1+mvL8XHLJU9n6m8Jo4eKlFjcHguVl30xMk yKt/DWhmLNRhoL5LNtxF5z8RIG116wjewYPbc= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=sg+bJaDUMi6cidJwpuhz8YqPDcihHRJslrVPGNZ4Mos=; b=bAZZef/QOoYZkmTIwQidYAb65ebM1pUxETJK4rEGSuuK9HZpLY98BZcVIlLx00H2wk ufnv/0aOsZ4IuBFzL39z0Hqydxqy7vsvep71Hx6PlBcb2wOl8U41AIfl7dNey5hF5mNG /4EJXhUsuP80ukNpH1DltuWrX6dr+oYPu6gPMle92AEcs+54wnifJy+UcPQl0w9BUVHc 8QF302bkjxKXosPxBhjS6C8kLM0GrCs9cX1uHJCzEJjq6DqK/qM4TMrtqhIIWII95YEZ r8hO9U2taPhLKgF0RbmEWYlahR1HiXErTQ4BjXCt/g6QGEBrc8R/LgllVSzegOYQG2ap dPfw== X-Gm-Message-State: AOUpUlFkILF+vfm+B+WOcZ50QLZuMoRIc+eJNon1bhL3LJnGAf+1BQ0L 5d3O2/6nAPAKRF/lO6XSaTD8vtSWS3M= X-Received: by 2002:a62:4b48:: with SMTP id y69-v6mr558409pfa.93.1533788535421; Wed, 08 Aug 2018 21:22:15 -0700 (PDT) Received: from cloudburst.twiddle.net (97-113-8-179.tukw.qwest.net. [97.113.8.179]) by smtp.gmail.com with ESMTPSA id m30-v6sm7355799pff.121.2018.08.08.21.22.14 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Wed, 08 Aug 2018 21:22:14 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Date: Wed, 8 Aug 2018 21:21:51 -0700 Message-Id: <20180809042206.15726-6-richard.henderson@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180809042206.15726-1-richard.henderson@linaro.org> References: <20180809042206.15726-1-richard.henderson@linaro.org> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::431 Subject: [Qemu-devel] [PATCH 05/20] target/arm: Fix arm_cpu_data_is_big_endian for aa64 user-only X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: laurent.desnogues@gmail.com, peter.maydell@linaro.org, alex.bennee@linaro.org Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: "Qemu-devel" Unlike aa32, endianness cannot be adjusted by userland in aa64. Signed-off-by: Richard Henderson --- target/arm/cpu.h | 27 +++++++++++++++++---------- 1 file changed, 17 insertions(+), 10 deletions(-) -- 2.17.1 diff --git a/target/arm/cpu.h b/target/arm/cpu.h index 9526ed27cb..2d6d7d03aa 100644 --- a/target/arm/cpu.h +++ b/target/arm/cpu.h @@ -2709,8 +2709,6 @@ static inline bool arm_sctlr_b(CPUARMState *env) /* Return true if the processor is in big-endian mode. */ static inline bool arm_cpu_data_is_big_endian(CPUARMState *env) { - int cur_el; - /* In 32bit endianness is determined by looking at CPSR's E bit */ if (!is_a64(env)) { return @@ -2729,15 +2727,24 @@ static inline bool arm_cpu_data_is_big_endian(CPUARMState *env) arm_sctlr_b(env) || #endif ((env->uncached_cpsr & CPSR_E) ? 1 : 0); + } else { +#ifdef CONFIG_USER_ONLY + /* AArch64 does not have a SETEND instruction; endianness + * for usermode is fixed at compile-time. + */ +# ifdef TARGET_WORDS_BIGENDIAN + return true; +# else + return false; +# endif +#else + int cur_el = arm_current_el(env); + if (cur_el == 0) { + return (env->cp15.sctlr_el[1] & SCTLR_E0E) != 0; + } + return (env->cp15.sctlr_el[cur_el] & SCTLR_EE) != 0; +#endif } - - cur_el = arm_current_el(env); - - if (cur_el == 0) { - return (env->cp15.sctlr_el[1] & SCTLR_E0E) != 0; - } - - return (env->cp15.sctlr_el[cur_el] & SCTLR_EE) != 0; } #include "exec/cpu-all.h" From patchwork Thu Aug 9 04:21:52 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 143659 Delivered-To: patch@linaro.org Received: by 2002:a2e:9754:0:0:0:0:0 with SMTP id f20-v6csp1603899ljj; Wed, 8 Aug 2018 21:25:32 -0700 (PDT) X-Google-Smtp-Source: AA+uWPxAoDySUqROiaR48rzCCSDmEVfMpwBsJNwuucmFTVLM2Vr0dQVnk7JcI6wnfKKVzBdik0NA X-Received: by 2002:ac8:32c7:: with SMTP id a7-v6mr493913qtb.26.1533788732353; Wed, 08 Aug 2018 21:25:32 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1533788732; cv=none; d=google.com; s=arc-20160816; b=TeiuH1CY/UdBUrtmCGOmOEs+zTbSMO9HxFYyMG7BJFme59idJLG9CgTfOYvAcHPShJ /zhfoK42QKGLbXCiy1eXx2oJ1WZRU0XGs7JZrBRNyYjkO043xfne0vvgGe+ti0qKZDji L9yFiGlD0VYVhS2itEGbUbubsIAlNM/46NDpdyAUfZov/oC9oX/P9VPqklcAO9NXuflu d5KSUq/hUCmuPgq1YG+0goROv+SV+9eEgpJv6IiIimx+Ae2iZZQMxGp66rUwjrehRXqe FaS2qimcLPKU351SkBPJNbtBPhWAvzjxTvXtIQQHOw6Y334zJfDoa8Ub4ifVa/iIEl6i /R8w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:cc:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:subject:references:in-reply-to :message-id:date:to:from:dkim-signature:arc-authentication-results; bh=LD/vcIqlnO0xpD+E8Q45wGd4xiipQg1fxpSpB81mnec=; b=hG7kQL6AjBw4rpnOvYvYMToFZnZe8yQSQdZ8lb2n3FHG6iOxQedCGfwQkGOriRtuFk XzBw+50Oa9TeTYao+6d9n2JdoaG90cYGy3R9Ccb4/Gp4BBHaKpCoUOdgjaaTckZlCruy svj+fVaCCCUeXVNuJXovm7LHas7n+LpfzDz4v2zuZLpWzowczOSuvZ9usEBaFvNBjbuv 5Jy8Zd1USt4QkQuSGMRdN+aFChuhb2gzITNaOnEqr+VAPWa8GwrTnPPb7djndUIle5Vq t/1T9COflPllL9Ssm4LvZ6/g1R8dRe8sJJr6svzdUOVI5SDj7rhvfZ+JTj3fzfn1SmfN wx6w== ARC-Authentication-Results: i=1; mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=MwlTbhUN; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [2001:4830:134:3::11]) by mx.google.com with ESMTPS id n19-v6si1069998qtp.63.2018.08.08.21.25.32 for (version=TLS1 cipher=AES128-SHA bits=128/128); Wed, 08 Aug 2018 21:25:32 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) client-ip=2001:4830:134:3::11; Authentication-Results: mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=MwlTbhUN; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1]:47235 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fncVv-0003Ad-NV for patch@linaro.org; Thu, 09 Aug 2018 00:25:31 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:53999) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fncSp-0000lX-Av for qemu-devel@nongnu.org; Thu, 09 Aug 2018 00:22:22 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fncSo-0007D2-HH for qemu-devel@nongnu.org; Thu, 09 Aug 2018 00:22:19 -0400 Received: from mail-pg1-x529.google.com ([2607:f8b0:4864:20::529]:44953) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fncSo-0007CK-Aq for qemu-devel@nongnu.org; Thu, 09 Aug 2018 00:22:18 -0400 Received: by mail-pg1-x529.google.com with SMTP id r1-v6so2099238pgp.11 for ; Wed, 08 Aug 2018 21:22:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=LD/vcIqlnO0xpD+E8Q45wGd4xiipQg1fxpSpB81mnec=; b=MwlTbhUNMbEESZq4YrbjKtslfLV9VhF8S7+hkvSKE9yAc97G/UDoqb4yL5UQ5K15Q/ rP7S5nqWuQf64FzbEbeBKnC6MIy0dBzjBb6eJiKs8OFbNGV8pWpaPOjzIkKZgRX+SCg+ aczV57JKf+66ORPSB8s7+vaI0K8gLG9nG2tZo= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=LD/vcIqlnO0xpD+E8Q45wGd4xiipQg1fxpSpB81mnec=; b=Gs6WzZxbFyeQLnSFlxwnjHRo9KqNPeim7EabC8vozJaTvsrDTCJ3qDA7cF/kIdA1P3 d8xZdoizPThKvuqVfVmC0+HEIxfcRn5HfgOqrzmsFKnptamN0N5ZDEvDNZTw6fApMzx4 igpVFpjKtKrgpPPqmjyQVgLX2jkHzurwBf1+WJy1Ae+6W2j8i+f84aEtcf48eayHayLL 0xw4E+8vEAzpH04o8uEQmY70Bd1Y0n11kdomBqzMxDvtxOyUBpm0KGOtrv8Mnik4Qr3K g03EOSC9Np3FLHVYDHIsZjRU8TM8htKWeiJaHZMOtJ0Wfo1Kx9NTjWLecWI2vvqjrKpI EZ1w== X-Gm-Message-State: AOUpUlFDOkBzHY0OevCw1FRJew5QZ4ZYQT9hN94EHNNMVfrxAQtxlaUU Xx3SETAiFK89T9tZhKS4OcymGTGNDcc= X-Received: by 2002:a62:4dc1:: with SMTP id a184-v6mr565653pfb.5.1533788537140; Wed, 08 Aug 2018 21:22:17 -0700 (PDT) Received: from cloudburst.twiddle.net (97-113-8-179.tukw.qwest.net. [97.113.8.179]) by smtp.gmail.com with ESMTPSA id m30-v6sm7355799pff.121.2018.08.08.21.22.15 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Wed, 08 Aug 2018 21:22:16 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Date: Wed, 8 Aug 2018 21:21:52 -0700 Message-Id: <20180809042206.15726-7-richard.henderson@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180809042206.15726-1-richard.henderson@linaro.org> References: <20180809042206.15726-1-richard.henderson@linaro.org> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::529 Subject: [Qemu-devel] [PATCH 06/20] target/arm: Fix arm_current_el for user-only X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: laurent.desnogues@gmail.com, peter.maydell@linaro.org, alex.bennee@linaro.org Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: "Qemu-devel" Saves about 12k code size in qemu-aarch64. Signed-off-by: Richard Henderson --- target/arm/cpu.h | 4 ++++ 1 file changed, 4 insertions(+) -- 2.17.1 diff --git a/target/arm/cpu.h b/target/arm/cpu.h index 2d6d7d03aa..aedaf2631e 100644 --- a/target/arm/cpu.h +++ b/target/arm/cpu.h @@ -1958,6 +1958,9 @@ static inline bool arm_v7m_is_handler_mode(CPUARMState *env) */ static inline int arm_current_el(CPUARMState *env) { +#ifdef CONFIG_USER_ONLY + return 0; +#else if (arm_feature(env, ARM_FEATURE_M)) { return arm_v7m_is_handler_mode(env) || !(env->v7m.control[env->v7m.secure] & 1); @@ -1984,6 +1987,7 @@ static inline int arm_current_el(CPUARMState *env) return 1; } +#endif } typedef struct ARMCPRegInfo ARMCPRegInfo; From patchwork Thu Aug 9 04:21:53 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 143665 Delivered-To: patch@linaro.org Received: by 2002:a2e:9754:0:0:0:0:0 with SMTP id f20-v6csp1606048ljj; Wed, 8 Aug 2018 21:28:27 -0700 (PDT) X-Google-Smtp-Source: AA+uWPxbbwLi4espnIojn6Lxty2Tuga70TZVvuJSONQClAn5SdXb0y/bKLcnXUfQAKo7T5YtrWVu X-Received: by 2002:a37:1614:: with SMTP id g20-v6mr427171qkh.64.1533788907019; Wed, 08 Aug 2018 21:28:27 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1533788907; cv=none; d=google.com; s=arc-20160816; b=bOwsDQN/9mhGU9KevB8fMbaOwssnf58PJiUF1/S7Jak9PO+/8ZL3YBxo4QIs2vNkzZ ODEoNqrrRggWEOe76oMQegTQJMrBL+X8Q/BlmCC5U+oiENF6/GxOM1FvgOmp/i+LGmWz lEiQ4zKsHhSEyxFIuBvlyzve4vt2kRpncBHIBBFdDyHqNll/TZZDwcHy7xQx6v8KoyHm voD5yxgaCuLLAsIxN8kzVHzmwpRDT5kvCFOWr6sRKEWDKXzvy9KZhLPDIxg99Zd1d03M LAmQzZMQCA3iCDpBe3AEDkYhfeX+FJWX2FjY6VkTpGtRnrtciYcLzWQopLKmD1X4lgER 1WNQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:cc:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:subject:references:in-reply-to :message-id:date:to:from:dkim-signature:arc-authentication-results; bh=ZhqStPaobVNOlrpd9bhYKZkFUowiLMw3HFHpD3qTb9Q=; b=BI3w1XHLMJ7x3DanphglW/KFcyt5H9uHYccN1DQx48J3mDe8nLl5ANqCgqz9NLYXDq 9e09LrMj3o+dNpRRfL4huR/Uw2KOfEOsbseP9OuRi9I5jKkkbYmBbX6uC9UOHZ9euOm/ QS6Si6C4PrGto8KrfGjJ/dh6N8AWvCXazlSE4ELHV1z7FqjOBR9m0lr+kN1I7uNuetcm GWgDHLaqpaMoUB+xJaawu3xft8Pjf1zx4YsHh1Oc7VF6P+aVOLnpaF5y2M7XjA8neicB 909uZ1yC2imxKfvH2K2t5UxzwjPiQlcu+X8C8DmyF2rsfZTq/5ey1YritS77N3lZFJDv op/w== ARC-Authentication-Results: i=1; mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=eUb7FZ5u; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [2001:4830:134:3::11]) by mx.google.com with ESMTPS id a65-v6si3128346qkd.318.2018.08.08.21.28.26 for (version=TLS1 cipher=AES128-SHA bits=128/128); Wed, 08 Aug 2018 21:28:27 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) client-ip=2001:4830:134:3::11; Authentication-Results: mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=eUb7FZ5u; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1]:47257 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fncYk-0006EO-DE for patch@linaro.org; Thu, 09 Aug 2018 00:28:26 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:54016) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fncSs-0000ol-IL for qemu-devel@nongnu.org; Thu, 09 Aug 2018 00:22:23 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fncSq-0007Fy-6y for qemu-devel@nongnu.org; Thu, 09 Aug 2018 00:22:22 -0400 Received: from mail-pl0-x22e.google.com ([2607:f8b0:400e:c01::22e]:40274) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fncSq-0007Er-03 for qemu-devel@nongnu.org; Thu, 09 Aug 2018 00:22:20 -0400 Received: by mail-pl0-x22e.google.com with SMTP id s17-v6so1975141plp.7 for ; Wed, 08 Aug 2018 21:22:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=ZhqStPaobVNOlrpd9bhYKZkFUowiLMw3HFHpD3qTb9Q=; b=eUb7FZ5u6jwRd/Ax8cU+2v9tW9toRSfHbht2Gjp4rH5GAsjOI73GDkSMxdcZj80Xg+ jr+cksN4x+yG39fon7caTPz60vu3fUlwVPsY5+a6QbuF+zL5TquuMLOE7a46VNOEAvai gMmXrKKekt8mB7gTu7A7qDKrXLdt4m4eBexNo= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=ZhqStPaobVNOlrpd9bhYKZkFUowiLMw3HFHpD3qTb9Q=; b=MSuo8A2JVusRDTOAqMIz4TKX+XP6oC6n90B+4ov5yxngvOig9RJzMNQxr54pBVfEpK KliKY9lEIVyRdLKZOMz5xCwRA2O9dNEN9ZNtL/W//isbKCLDxYgVYEj9gHKjSz8ScOxZ MCLT+H1BmqZzgfClxuPjD/9dACuB1HThOAbUVGLZpB3/a7V7PIh7hhBbMrB4BbgZ5UUz av+cAzaj+BYuwGCLtsn3sB61b09g+f6YIsD5FhrEfjvzBmfuZBu5GZL9c1Mb66gXB5We zn5MiUcDZfZeP71s/rDSWZtdqEA0UX/gE44C3PvR2cNguvd4DaCFS3MsS+HxAdxr23Z1 C+zQ== X-Gm-Message-State: AOUpUlE59xjshddut8QPxlpKlb4f/Ne1dMd8naf3ZSqIi4q17QObw+kN CS+2H2R1yXlw6aZhBmR+0sYo+kcKRww= X-Received: by 2002:a17:902:8a90:: with SMTP id p16-v6mr521184plo.106.1533788538763; Wed, 08 Aug 2018 21:22:18 -0700 (PDT) Received: from cloudburst.twiddle.net (97-113-8-179.tukw.qwest.net. [97.113.8.179]) by smtp.gmail.com with ESMTPSA id m30-v6sm7355799pff.121.2018.08.08.21.22.17 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Wed, 08 Aug 2018 21:22:17 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Date: Wed, 8 Aug 2018 21:21:53 -0700 Message-Id: <20180809042206.15726-8-richard.henderson@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180809042206.15726-1-richard.henderson@linaro.org> References: <20180809042206.15726-1-richard.henderson@linaro.org> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c01::22e Subject: [Qemu-devel] [PATCH 07/20] target/arm: Fix is_a64 for user-only X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: laurent.desnogues@gmail.com, peter.maydell@linaro.org, alex.bennee@linaro.org Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: "Qemu-devel" Saves about 8k code size in qemu-aarch64. Signed-off-by: Richard Henderson --- target/arm/cpu.h | 8 ++++++++ 1 file changed, 8 insertions(+) -- 2.17.1 diff --git a/target/arm/cpu.h b/target/arm/cpu.h index aedaf2631e..ed51a2f5aa 100644 --- a/target/arm/cpu.h +++ b/target/arm/cpu.h @@ -918,7 +918,15 @@ void aarch64_sync_64_to_32(CPUARMState *env); static inline bool is_a64(CPUARMState *env) { +#ifdef CONFIG_USER_ONLY +# ifdef TARGET_AARCH64 + return true; +# else + return false; +# endif +#else return env->aarch64; +#endif } /* you can call this signal handler from your SIGBUS and SIGSEGV From patchwork Thu Aug 9 04:21:54 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 143662 Delivered-To: patch@linaro.org Received: by 2002:a2e:9754:0:0:0:0:0 with SMTP id f20-v6csp1605700ljj; Wed, 8 Aug 2018 21:27:59 -0700 (PDT) X-Google-Smtp-Source: AA+uWPx30M+GV3sIfAFVo7CQ6HeCJJzcMQdhDBboOfIGx1ph0JJHf8Qg/ksf9ZtySfjV5mr+FLwy X-Received: by 2002:aed:2a33:: with SMTP id c48-v6mr478209qtd.147.1533788879706; Wed, 08 Aug 2018 21:27:59 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1533788879; cv=none; d=google.com; s=arc-20160816; b=DeHBNCJ9spnHY4PMHndal5IfwTvrynqZ18tAzR0iMiSitHXbaPc8Mly0bJ9zERLOLS crAcIch/7ox5jKFgbLA6H9DMzwz7Lqvf8YiAOztNqNa7gD92bb6FKZzCXNK9nsfW+e/1 aD3C9kj3GsrkdL14R4xduCyYYYP22mlI8gxAz/T0Knr3HI9YEvaL4131iXeJI4mGAKoA 4tYyc6hQFVj1BQS2xiKGc3ttRAeLtStk1lhdrBZw0FXxHzaPaRZES4bkXRl90eMs1JmM H/WC6A7cy5rAD0Gd3kZAZpektQssiF8E25qlhoTXlsSyRzYzldxjPZ5xOoXGArnlpKWG QYDQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:cc:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:subject:references:in-reply-to :message-id:date:to:from:dkim-signature:arc-authentication-results; bh=iPTMcFpTowzzeR3AA8m0CxGcQ3PcWy1tCYbypW7plhs=; b=OU3Vw4p4F9dmDBNsa5UsgeGGZ5a0BaWE/nukNkn9alCWlrwsj3uqv6kJAIZJvZHwuN FiZo1BgPDAwoWmBm2yUNsGgVhtuRLoaJ1nHi+v5Ljyy6+zXVoLVPBvZTjj6YmSGZn4vc Rf+FKc9PVvvgX5ObiMqtWpBkdnauJyBF1zDDar0cW9fM7fhApo9C7QjaNayqFL7BoFln TUBg05oJ8g8q3McYG2a2yK9BnZ9L6u2JdLAnwsdoTUVA/bJTjXcbrOonOY7YXB09HO6m VfVxcjuMCFciaOOBHCHfnmhn6+qrr3/5O8/LXhyG83NbyBZg/gKIY9jpyM+r1J+PCK7U 5jeg== ARC-Authentication-Results: i=1; mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=RnkwxGB9; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [2001:4830:134:3::11]) by mx.google.com with ESMTPS id c40-v6si6091341qtb.208.2018.08.08.21.27.59 for (version=TLS1 cipher=AES128-SHA bits=128/128); Wed, 08 Aug 2018 21:27:59 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) client-ip=2001:4830:134:3::11; Authentication-Results: mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=RnkwxGB9; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1]:47249 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fncYJ-0004nX-3m for patch@linaro.org; Thu, 09 Aug 2018 00:27:59 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:54015) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fncSs-0000oj-IA for qemu-devel@nongnu.org; Thu, 09 Aug 2018 00:22:23 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fncSr-0007Gv-FJ for qemu-devel@nongnu.org; Thu, 09 Aug 2018 00:22:22 -0400 Received: from mail-pl0-x242.google.com ([2607:f8b0:400e:c01::242]:47012) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fncSr-0007GM-8d for qemu-devel@nongnu.org; Thu, 09 Aug 2018 00:22:21 -0400 Received: by mail-pl0-x242.google.com with SMTP id t17-v6so1967497ply.13 for ; Wed, 08 Aug 2018 21:22:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=iPTMcFpTowzzeR3AA8m0CxGcQ3PcWy1tCYbypW7plhs=; b=RnkwxGB9pyCnDTdyOHHhZhDWwbMvV9USJ+alu1vyr2HUx/P4qkLr9YWikNlXDuO6gO w+DXWLEO2UjiLhHqd3qD0YJt5l7De58U+ehfwS1O6YlBh7I3P7GG3FK9yAjHC7AVEgx9 XdI1LccGqmh6Bf1q6X2Q3VG2PJKHL8rKd1cDE= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=iPTMcFpTowzzeR3AA8m0CxGcQ3PcWy1tCYbypW7plhs=; b=nSS8cMvPiqsbs/lXl1mcmnx1UwQR+kXiNfaps5xlPRDWFlXKnV6EjLnFVcatadrrq6 NG0DmgMG5vUA5JDDyIgY7VxsHcP1x3dujp0hP393lNqgE2XJPw5X8fI80ex0/ERJm/gT D+6W7q7DunVMX3u+OaeiK+kLwyZCFz9KUJ/G4FEl4/paq2YRcTm7DlEa65+YomczPZvn 3h9tNrz3+Q/m1QKDzbQLAo1VcAGw+wEU8mxn1MmnVFKtuKjtAzpQ+zawZHrBvtgHkOqw yH8rtBIAwKU7kiWyDzxv9kTF8Ozi38fyHYG1tnbxYDXYUBR8a55RQZ4yzaMvBngWJNaf kzPw== X-Gm-Message-State: AOUpUlH0hlTkB+e2jH47cLTRpluuG8CfTvXv4sx4ow8UVrdqR2UvlfAA R6FYX0x0cKMFUVW5FvsbtLwY1dQf6Yk= X-Received: by 2002:a17:902:26c:: with SMTP id 99-v6mr466624plc.341.1533788539995; Wed, 08 Aug 2018 21:22:19 -0700 (PDT) Received: from cloudburst.twiddle.net (97-113-8-179.tukw.qwest.net. [97.113.8.179]) by smtp.gmail.com with ESMTPSA id m30-v6sm7355799pff.121.2018.08.08.21.22.18 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Wed, 08 Aug 2018 21:22:19 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Date: Wed, 8 Aug 2018 21:21:54 -0700 Message-Id: <20180809042206.15726-9-richard.henderson@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180809042206.15726-1-richard.henderson@linaro.org> References: <20180809042206.15726-1-richard.henderson@linaro.org> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c01::242 Subject: [Qemu-devel] [PATCH 08/20] target/arm: Pass in current_el to fp and sve_exception_el X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: laurent.desnogues@gmail.com, peter.maydell@linaro.org, alex.bennee@linaro.org Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: "Qemu-devel" We are going to want to determine whether sve is enabled for EL than current. Signed-off-by: Richard Henderson --- target/arm/helper.c | 21 +++++++++------------ 1 file changed, 9 insertions(+), 12 deletions(-) -- 2.17.1 diff --git a/target/arm/helper.c b/target/arm/helper.c index 26e9098c5f..290b1a849e 100644 --- a/target/arm/helper.c +++ b/target/arm/helper.c @@ -4344,12 +4344,10 @@ static const ARMCPRegInfo debug_lpae_cp_reginfo[] = { * take care of raising that exception. * C.f. the ARM pseudocode function CheckSVEEnabled. */ -static int sve_exception_el(CPUARMState *env) +static int sve_exception_el(CPUARMState *env, int el) { #ifndef CONFIG_USER_ONLY - unsigned current_el = arm_current_el(env); - - if (current_el <= 1) { + if (el <= 1) { bool disabled = false; /* The CPACR.ZEN controls traps to EL1: @@ -4360,7 +4358,7 @@ static int sve_exception_el(CPUARMState *env) if (!extract32(env->cp15.cpacr_el1, 16, 1)) { disabled = true; } else if (!extract32(env->cp15.cpacr_el1, 17, 1)) { - disabled = current_el == 0; + disabled = el == 0; } if (disabled) { /* route_to_el2 */ @@ -4373,7 +4371,7 @@ static int sve_exception_el(CPUARMState *env) if (!extract32(env->cp15.cpacr_el1, 20, 1)) { disabled = true; } else if (!extract32(env->cp15.cpacr_el1, 21, 1)) { - disabled = current_el == 0; + disabled = el == 0; } if (disabled) { return 0; @@ -4383,7 +4381,7 @@ static int sve_exception_el(CPUARMState *env) /* CPTR_EL2. Since TZ and TFP are positive, * they will be zero when EL2 is not present. */ - if (current_el <= 2 && !arm_is_secure_below_el3(env)) { + if (el <= 2 && !arm_is_secure_below_el3(env)) { if (env->cp15.cptr_el[2] & CPTR_TZ) { return 2; } @@ -12318,11 +12316,10 @@ uint32_t HELPER(crc32c)(uint32_t acc, uint32_t val, uint32_t bytes) /* Return the exception level to which FP-disabled exceptions should * be taken, or 0 if FP is enabled. */ -static inline int fp_exception_el(CPUARMState *env) +static int fp_exception_el(CPUARMState *env, int cur_el) { #ifndef CONFIG_USER_ONLY int fpen; - int cur_el = arm_current_el(env); /* CPACR and the CPTR registers don't exist before v6, so FP is * always accessible @@ -12385,11 +12382,12 @@ void cpu_get_tb_cpu_state(CPUARMState *env, target_ulong *pc, target_ulong *cs_base, uint32_t *pflags) { ARMMMUIdx mmu_idx = core_to_arm_mmu_idx(env, cpu_mmu_index(env, false)); - int fp_el = fp_exception_el(env); + int current_el = arm_current_el(env); + int fp_el = fp_exception_el(env, current_el); uint32_t flags; if (is_a64(env)) { - int sve_el = sve_exception_el(env); + int sve_el = sve_exception_el(env, current_el); uint32_t zcr_len; *pc = env->pc; @@ -12404,7 +12402,6 @@ void cpu_get_tb_cpu_state(CPUARMState *env, target_ulong *pc, if (sve_el != 0 && fp_el == 0) { zcr_len = 0; } else { - int current_el = arm_current_el(env); ARMCPU *cpu = arm_env_get_cpu(env); zcr_len = cpu->sve_max_vq - 1; From patchwork Thu Aug 9 04:21:55 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 143666 Delivered-To: patch@linaro.org Received: by 2002:a2e:9754:0:0:0:0:0 with SMTP id f20-v6csp1607787ljj; Wed, 8 Aug 2018 21:30:45 -0700 (PDT) X-Google-Smtp-Source: AA+uWPyX+/pHH16NpdAAowBbmXT4XNwvsXtsqQKRxDsN7frAPf4eN+SbCvh1pcwZgCVEv9ogf+pr X-Received: by 2002:a37:434a:: with SMTP id q71-v6mr395748qka.277.1533789045153; Wed, 08 Aug 2018 21:30:45 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1533789045; cv=none; d=google.com; s=arc-20160816; b=ZrAKsPdfNOOufFzvnMUDgrWBrfly86FqQcMtFRexcVliEf43luFqfq5tfBFdYZPFhD mHiUPDNz+1I2Xbe2P7iHjIMwr3nh6NhEWHmSpIz2GOrBCxx+OlIIDPhpXQ3uvPA83wZD K4R+SWs0rew5zFjK1OcWFstItMrnHjgsNndG3J4zP7jfCPZ6eCLg5glKWJdYdokRAaOK Svwt9NV0heExOatzL1qlYRl9GaRo3i/8T6GjE2CYuHql0F76BgYladyrG8Humqz9nYQm iryThM99EP48JhZvR3mpzx4gqN3MvupNIYMIuuSCnXtYn8T03wn53I7j0hvJfcb8ryNa zn3w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:cc:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:subject:references:in-reply-to :message-id:date:to:from:dkim-signature:arc-authentication-results; bh=GUayhAnOGzgkh75iWjLeMdxi4TkGN1zqtM9FxRzxZnM=; b=P/0rZBS7rlT27Xis+8Bi2JArV8Qbq4pK4ZwK554I5YpjnTSY9SUKsB1c2PkEnTbyBa N9gN9d0X6BDY/HOlebvY5CbDn+5OVEEpoI8gBeXz8cjX9W552juWVs4xqnfNorqgSSRR 0l/MnE6TDxd5KMr53kXMlQGe3XnwZk/ud56Hc0bqKHf8xlZuBWUXVD+VO19QzxRHoJL3 LqLlhe92oxup7ZkHPraUhU01dHZc2ew8h9dLl1sd6m5Iq1XnwwmEMQBZpkDuaStPw4lm axoph3pa4kZdpyFLW94ifkHb6ywtdIELIhkCJNHRB8R4dISxb/pFuSQ+jJvihdqUuCyX SQNg== ARC-Authentication-Results: i=1; mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=fWwa4dB3; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [2001:4830:134:3::11]) by mx.google.com with ESMTPS id g10-v6si4446560qth.312.2018.08.08.21.30.44 for (version=TLS1 cipher=AES128-SHA bits=128/128); Wed, 08 Aug 2018 21:30:45 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) client-ip=2001:4830:134:3::11; Authentication-Results: mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=fWwa4dB3; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1]:47259 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fncay-0007Hr-If for patch@linaro.org; Thu, 09 Aug 2018 00:30:44 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:54036) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fncSu-0000qh-Jr for qemu-devel@nongnu.org; Thu, 09 Aug 2018 00:22:26 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fncSs-0007IX-Vs for qemu-devel@nongnu.org; Thu, 09 Aug 2018 00:22:24 -0400 Received: from mail-pl0-x232.google.com ([2607:f8b0:400e:c01::232]:39443) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fncSs-0007HQ-Lj for qemu-devel@nongnu.org; Thu, 09 Aug 2018 00:22:22 -0400 Received: by mail-pl0-x232.google.com with SMTP id w14-v6so1981330plp.6 for ; Wed, 08 Aug 2018 21:22:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=GUayhAnOGzgkh75iWjLeMdxi4TkGN1zqtM9FxRzxZnM=; b=fWwa4dB3ntpidb1OsO8S8eT9kuKVX443QD4SsUTH3xI9Emyh0QLQW7pTmYjY5fMnYT hdtAXHNUGN4zP0UhkC4gq9a0AVX6+V1eC95L02TYNbx8HTReRa1HwQovX6AoFm5VlFGM xltxv4a4QodXQ+YiyZzazEaHwF9dTvqi/fN7U= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=GUayhAnOGzgkh75iWjLeMdxi4TkGN1zqtM9FxRzxZnM=; b=LUgWNvIU7/oZIzGeelXXgS3UNWkuR6ioImV2eNNjHMrsDh1eRfobU7M/1trWhgltGt KacD424Lam26qxnXkRQiyhsm4XBtxcoitl+0Lts0Ai/nLL2GNgaokrt23hwAo0u0GK11 kKtW6E83EDszNzDZLz2Vdb2JH07aCOLFfA0i6pa3MTG1nb7qefDwPvdpQyOe+T5pUVEH TFT3xH37iM0ujYFpAt8uqqKEcFHKJnHo9UZepbFDJVQ97isF94Xlhh0j4zH+o7LABYcS h93qVUZCEeKAY4n4PtKsRgDbwk6XKn8dA0IIZBcJkz5FZSZIT1Yjrqt7t1QPmACmFSn6 ai5A== X-Gm-Message-State: AOUpUlGMV4ie9Bcg8g5ZZv0d6jgSPrYw4K9ZTTykNhl1p41E4q1+7KGH ZxPqic0Y5dokrSoIjwCccNdNd6L6JaM= X-Received: by 2002:a17:902:c6:: with SMTP id a64-v6mr514834pla.180.1533788541338; Wed, 08 Aug 2018 21:22:21 -0700 (PDT) Received: from cloudburst.twiddle.net (97-113-8-179.tukw.qwest.net. [97.113.8.179]) by smtp.gmail.com with ESMTPSA id m30-v6sm7355799pff.121.2018.08.08.21.22.20 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Wed, 08 Aug 2018 21:22:20 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Date: Wed, 8 Aug 2018 21:21:55 -0700 Message-Id: <20180809042206.15726-10-richard.henderson@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180809042206.15726-1-richard.henderson@linaro.org> References: <20180809042206.15726-1-richard.henderson@linaro.org> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c01::232 Subject: [Qemu-devel] [PATCH 09/20] target/arm: Handle SVE vector length changes in system mode X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: laurent.desnogues@gmail.com, peter.maydell@linaro.org, alex.bennee@linaro.org Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: "Qemu-devel" SVE vector length can change when changing EL, or when writing to one of the ZCR_ELn registers. For correctness, our implementation requires that predicate bits that are inaccessible are never set. Which means noticing length changes and zeroing the appropriate register bits. Signed-off-by: Richard Henderson --- target/arm/cpu.h | 4 ++ target/arm/cpu64.c | 42 -------------- target/arm/helper.c | 127 ++++++++++++++++++++++++++++++++++++----- target/arm/op_helper.c | 1 + 4 files changed, 119 insertions(+), 55 deletions(-) -- 2.17.1 diff --git a/target/arm/cpu.h b/target/arm/cpu.h index ed51a2f5aa..18b3c92c2e 100644 --- a/target/arm/cpu.h +++ b/target/arm/cpu.h @@ -910,6 +910,10 @@ int arm_cpu_write_elf32_note(WriteCoreDumpFunction f, CPUState *cs, int aarch64_cpu_gdb_read_register(CPUState *cpu, uint8_t *buf, int reg); int aarch64_cpu_gdb_write_register(CPUState *cpu, uint8_t *buf, int reg); void aarch64_sve_narrow_vq(CPUARMState *env, unsigned vq); +void aarch64_sve_change_el(CPUARMState *env, int old_el, int new_el); +#else +static inline void aarch64_sve_narrow_vq(CPUARMState *env, unsigned vq) { } +static inline void aarch64_sve_change_el(CPUARMState *env, int o, int n) { } #endif target_ulong do_arm_semihosting(CPUARMState *env); diff --git a/target/arm/cpu64.c b/target/arm/cpu64.c index ae650b608e..16272f1358 100644 --- a/target/arm/cpu64.c +++ b/target/arm/cpu64.c @@ -439,45 +439,3 @@ static void aarch64_cpu_register_types(void) } type_init(aarch64_cpu_register_types) - -/* The manual says that when SVE is enabled and VQ is widened the - * implementation is allowed to zero the previously inaccessible - * portion of the registers. The corollary to that is that when - * SVE is enabled and VQ is narrowed we are also allowed to zero - * the now inaccessible portion of the registers. - * - * The intent of this is that no predicate bit beyond VQ is ever set. - * Which means that some operations on predicate registers themselves - * may operate on full uint64_t or even unrolled across the maximum - * uint64_t[4]. Performing 4 bits of host arithmetic unconditionally - * may well be cheaper than conditionals to restrict the operation - * to the relevant portion of a uint16_t[16]. - * - * TODO: Need to call this for changes to the real system registers - * and EL state changes. - */ -void aarch64_sve_narrow_vq(CPUARMState *env, unsigned vq) -{ - int i, j; - uint64_t pmask; - - assert(vq >= 1 && vq <= ARM_MAX_VQ); - assert(vq <= arm_env_get_cpu(env)->sve_max_vq); - - /* Zap the high bits of the zregs. */ - for (i = 0; i < 32; i++) { - memset(&env->vfp.zregs[i].d[2 * vq], 0, 16 * (ARM_MAX_VQ - vq)); - } - - /* Zap the high bits of the pregs and ffr. */ - pmask = 0; - if (vq & 3) { - pmask = ~(-1ULL << (16 * (vq & 3))); - } - for (j = vq / 4; j < ARM_MAX_VQ / 4; j++) { - for (i = 0; i < 17; ++i) { - env->vfp.pregs[i].p[j] &= pmask; - } - pmask = 0; - } -} diff --git a/target/arm/helper.c b/target/arm/helper.c index 290b1a849e..fb79b27cf6 100644 --- a/target/arm/helper.c +++ b/target/arm/helper.c @@ -4399,11 +4399,44 @@ static int sve_exception_el(CPUARMState *env, int el) return 0; } +/* + * Given that SVE is enabled, return the vector length for EL. + */ +static uint32_t sve_zcr_len_for_el(CPUARMState *env, int el) +{ + ARMCPU *cpu = arm_env_get_cpu(env); + uint32_t zcr_len = cpu->sve_max_vq - 1; + + if (el <= 1) { + zcr_len = MIN(zcr_len, 0xf & (uint32_t)env->vfp.zcr_el[1]); + } + if (el < 2 && arm_feature(env, ARM_FEATURE_EL2)) { + zcr_len = MIN(zcr_len, 0xf & (uint32_t)env->vfp.zcr_el[2]); + } + if (el < 3 && arm_feature(env, ARM_FEATURE_EL3)) { + zcr_len = MIN(zcr_len, 0xf & (uint32_t)env->vfp.zcr_el[3]); + } + return zcr_len; +} + static void zcr_write(CPUARMState *env, const ARMCPRegInfo *ri, uint64_t value) { + int cur_el = arm_current_el(env); + int old_len = sve_zcr_len_for_el(env, cur_el); + int new_len; + /* Bits other than [3:0] are RAZ/WI. */ raw_write(env, ri, value & 0xf); + + /* + * Because we arrived here, we know both FP and SVE are enabled; + * otherwise we would have trapped access to the ZCR_ELn register. + */ + new_len = sve_zcr_len_for_el(env, cur_el); + if (new_len < old_len) { + aarch64_sve_narrow_vq(env, new_len + 1); + } } static const ARMCPRegInfo zcr_el1_reginfo = { @@ -8100,8 +8133,11 @@ static void arm_cpu_do_interrupt_aarch64(CPUState *cs) unsigned int new_el = env->exception.target_el; target_ulong addr = env->cp15.vbar_el[new_el]; unsigned int new_mode = aarch64_pstate_mode(new_el, true); + unsigned int cur_el = arm_current_el(env); - if (arm_current_el(env) < new_el) { + aarch64_sve_change_el(env, cur_el, new_el); + + if (cur_el < new_el) { /* Entry vector offset depends on whether the implemented EL * immediately lower than the target level is using AArch32 or AArch64 */ @@ -12402,18 +12438,7 @@ void cpu_get_tb_cpu_state(CPUARMState *env, target_ulong *pc, if (sve_el != 0 && fp_el == 0) { zcr_len = 0; } else { - ARMCPU *cpu = arm_env_get_cpu(env); - - zcr_len = cpu->sve_max_vq - 1; - if (current_el <= 1) { - zcr_len = MIN(zcr_len, 0xf & (uint32_t)env->vfp.zcr_el[1]); - } - if (current_el < 2 && arm_feature(env, ARM_FEATURE_EL2)) { - zcr_len = MIN(zcr_len, 0xf & (uint32_t)env->vfp.zcr_el[2]); - } - if (current_el < 3 && arm_feature(env, ARM_FEATURE_EL3)) { - zcr_len = MIN(zcr_len, 0xf & (uint32_t)env->vfp.zcr_el[3]); - } + zcr_len = sve_zcr_len_for_el(env, current_el); } flags |= zcr_len << ARM_TBFLAG_ZCR_LEN_SHIFT; } else { @@ -12467,3 +12492,79 @@ void cpu_get_tb_cpu_state(CPUARMState *env, target_ulong *pc, *pflags = flags; *cs_base = 0; } + +#ifdef TARGET_AARCH64 +/* + * The manual says that when SVE is enabled and VQ is widened the + * implementation is allowed to zero the previously inaccessible + * portion of the registers. The corollary to that is that when + * SVE is enabled and VQ is narrowed we are also allowed to zero + * the now inaccessible portion of the registers. + * + * The intent of this is that no predicate bit beyond VQ is ever set. + * Which means that some operations on predicate registers themselves + * may operate on full uint64_t or even unrolled across the maximum + * uint64_t[4]. Performing 4 bits of host arithmetic unconditionally + * may well be cheaper than conditionals to restrict the operation + * to the relevant portion of a uint16_t[16]. + */ +void aarch64_sve_narrow_vq(CPUARMState *env, unsigned vq) +{ + int i, j; + uint64_t pmask; + + assert(vq >= 1 && vq <= ARM_MAX_VQ); + assert(vq <= arm_env_get_cpu(env)->sve_max_vq); + + /* Zap the high bits of the zregs. */ + for (i = 0; i < 32; i++) { + memset(&env->vfp.zregs[i].d[2 * vq], 0, 16 * (ARM_MAX_VQ - vq)); + } + + /* Zap the high bits of the pregs and ffr. */ + pmask = 0; + if (vq & 3) { + pmask = ~(-1ULL << (16 * (vq & 3))); + } + for (j = vq / 4; j < ARM_MAX_VQ / 4; j++) { + for (i = 0; i < 17; ++i) { + env->vfp.pregs[i].p[j] &= pmask; + } + pmask = 0; + } +} + +/* + * Notice a change in SVE vector size when changing EL. + */ +void aarch64_sve_change_el(CPUARMState *env, int old_el, int new_el) +{ + int old_len, new_len; + + /* Nothing to do if no SVE. */ + if (!arm_feature(env, ARM_FEATURE_SVE)) { + return; + } + + /* Nothing to do if FP is disabled in either EL. */ + if (fp_exception_el(env, old_el) || fp_exception_el(env, new_el)) { + return; + } + + /* + * When FP is enabled, but SVE is disabled, the effective len is 0. + * ??? How should sve_exception_el interact with AArch32 state? + * That isn't included in the CheckSVEEnabled pseudocode, so is the + * host kernel required to explicitly disable SVE for an EL using aa32? + */ + old_len = (sve_exception_el(env, old_el) + ? 0 : sve_zcr_len_for_el(env, old_el)); + new_len = (sve_exception_el(env, new_el) + ? 0 : sve_zcr_len_for_el(env, new_el)); + + /* When changing vector length, clear inaccessible state. */ + if (new_len < old_len) { + aarch64_sve_narrow_vq(env, new_len + 1); + } +} +#endif diff --git a/target/arm/op_helper.c b/target/arm/op_helper.c index f728f25e4b..b9f920b3c4 100644 --- a/target/arm/op_helper.c +++ b/target/arm/op_helper.c @@ -1068,6 +1068,7 @@ void HELPER(exception_return)(CPUARMState *env) "AArch64 EL%d PC 0x%" PRIx64 "\n", cur_el, new_el, env->pc); } + aarch64_sve_change_el(env, cur_el, new_el); qemu_mutex_lock_iothread(); arm_call_el_change_hook(arm_env_get_cpu(env)); From patchwork Thu Aug 9 04:21:56 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 143667 Delivered-To: patch@linaro.org Received: by 2002:a2e:9754:0:0:0:0:0 with SMTP id f20-v6csp1608198ljj; Wed, 8 Aug 2018 21:31:16 -0700 (PDT) X-Google-Smtp-Source: AA+uWPwf3VGGqf+zF/MVcwRhtTMeHG6D/ejoTFuXp1URzqnZmn78ZCVnC0MVxER/CGEEHxOmc2/5 X-Received: by 2002:a0c:aac9:: with SMTP id g9-v6mr435641qvb.78.1533789076400; Wed, 08 Aug 2018 21:31:16 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1533789076; cv=none; d=google.com; s=arc-20160816; b=Khl0qMgfLwho3kQu2/RFmgaHUX+ZyuElV7yiNRVrJLHKY6KSwPJ/719hwyLWQQDD4b OstoXd/vLEPHQL+1AKp1w3oUmdaKMdd/OA0+lYPXbGyd+qWQ1D5yQrkmrLRdINV8481W p7ZzG0w5D3/8ZAWe8jcgD6s0RKOeCDtxv9UMYj3u+pxbLcHI04aEx12nHTLGOW2VOKyS GWp97Des4EiEDmM5jEJ4rXcPOYQ90PVgsGrLYfF0YJg8HXDiX9V0MkK9YU86uGu6XnCX lE6SXHNJiQPzXIVH53MOGiF4j2oNJP9a48ow7W/dWoA7wew98/RaiN/+V8y38ouoZYlR rlwA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:cc:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:subject:references:in-reply-to :message-id:date:to:from:dkim-signature:arc-authentication-results; bh=fNJ5/L5RBIqFhK3h+FZqh+tESQTq1Q27XJeHHblUUck=; b=uNxg8AJ3KgxuXFswuRtnG0aSk5yBTlmTiSj+OhmHrXan7/bjv2XM3iWS2ASgtL3SPd ZCHsyaLfUdut2SJR4U6jiE+ipOLR+PRUckU8bzmRAB8cFMP5+EOnLidYh1yjl/UDOln+ BXYTYfAFV7Kz0XfFDL09LxhO2ufcFeW/KEMBtdkbU2AuBdKK45ZvarZj4ZjboX4apAn5 Rk/+tFzx6YHQYIUDuTVy0ud1zEcCxa7XpMaLv8l2U19GGBMa5NyMcI02urVncFXwGn5Q HWlvKKggpM9LIwfCzdKracRbwtIKtENYu0SaBAvGPMZmEEMdF/VHVz9bo3gZLdiiEzDN uM+A== ARC-Authentication-Results: i=1; mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=kP7KD5nC; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [2001:4830:134:3::11]) by mx.google.com with ESMTPS id k40-v6si1671450qtk.339.2018.08.08.21.31.16 for (version=TLS1 cipher=AES128-SHA bits=128/128); Wed, 08 Aug 2018 21:31:16 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) client-ip=2001:4830:134:3::11; Authentication-Results: mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=kP7KD5nC; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1]:47272 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fncbT-0000xQ-Oy for patch@linaro.org; Thu, 09 Aug 2018 00:31:15 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:54042) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fncSv-0000rI-76 for qemu-devel@nongnu.org; Thu, 09 Aug 2018 00:22:26 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fncSu-0007JP-4G for qemu-devel@nongnu.org; Thu, 09 Aug 2018 00:22:25 -0400 Received: from mail-pf1-x434.google.com ([2607:f8b0:4864:20::434]:44433) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fncSt-0007Iz-UB for qemu-devel@nongnu.org; Thu, 09 Aug 2018 00:22:24 -0400 Received: by mail-pf1-x434.google.com with SMTP id k21-v6so2183604pff.11 for ; Wed, 08 Aug 2018 21:22:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=fNJ5/L5RBIqFhK3h+FZqh+tESQTq1Q27XJeHHblUUck=; b=kP7KD5nCIfkHjDomauBY20QJnAgliTo/LVf3KXl5VCtnJrO/fhxX+G+zcfyLJJY2wp 01etFP3Dsiw7w7LschECc3whELreUESnylJ/MAfdNvtzxrTm8PpJ0OU1ryB1M/TPj+Gl P2IFtfxh0UeGg7mTZRQdYcaAqwpkv2Oi6PRNk= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=fNJ5/L5RBIqFhK3h+FZqh+tESQTq1Q27XJeHHblUUck=; b=ByKfooa+0/res3KSuw/WpWjt/dvcP5HLifiBSN1CjjXaVjmUOkyEyrmhlGm/EC1dYz fF4fgbvpnoCkMsQf36sV9pBLcMxQ5svGuDPIcW2GyTdLqS0I8y+vojL4OJRa7FFPRXs/ w8u+32DAz2VoEe3hngRVyYJzcxtXnq3tGDYXDYwYWgOlZBrAogBD7kZ1cy1ts65W7jtL AMJttTEcpBGA44RNF6qCy5QGpR1VlieYckjQHJtDM2W0XlSznMvGvtZcDDHukA7i7qHk Toc10b6lPTnj2JLE0efWZX3/QDklKEFFXsf2Ha1DjUEl1r0U2DXLgLYiNnvNxcqGkYoH SRvA== X-Gm-Message-State: AOUpUlFvwSVKZEhhEql0uiNT5uPtjyyw2HHZenU86VBcA7UCAX5Pxe7m RViCHsS+rUAnQcmKuIRVmJL3x8qDKyo= X-Received: by 2002:a63:a042:: with SMTP id u2-v6mr528895pgn.80.1533788542675; Wed, 08 Aug 2018 21:22:22 -0700 (PDT) Received: from cloudburst.twiddle.net (97-113-8-179.tukw.qwest.net. [97.113.8.179]) by smtp.gmail.com with ESMTPSA id m30-v6sm7355799pff.121.2018.08.08.21.22.21 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Wed, 08 Aug 2018 21:22:21 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Date: Wed, 8 Aug 2018 21:21:56 -0700 Message-Id: <20180809042206.15726-11-richard.henderson@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180809042206.15726-1-richard.henderson@linaro.org> References: <20180809042206.15726-1-richard.henderson@linaro.org> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::434 Subject: [Qemu-devel] [PATCH 10/20] target/arm: Adjust aarch64_cpu_dump_state for system mode SVE X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: laurent.desnogues@gmail.com, peter.maydell@linaro.org, alex.bennee@linaro.org Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: "Qemu-devel" Use the existing helpers to determine if (1) the fpu is enabled, (2) sve state is enabled, and (3) the current sve vector length. Signed-off-by: Richard Henderson --- target/arm/cpu.h | 4 ++++ target/arm/helper.c | 6 +++--- target/arm/translate-a64.c | 8 ++++++-- 3 files changed, 13 insertions(+), 5 deletions(-) -- 2.17.1 Reviewed-by: Peter Maydell diff --git a/target/arm/cpu.h b/target/arm/cpu.h index 18b3c92c2e..33d06f2340 100644 --- a/target/arm/cpu.h +++ b/target/arm/cpu.h @@ -920,6 +920,10 @@ target_ulong do_arm_semihosting(CPUARMState *env); void aarch64_sync_32_to_64(CPUARMState *env); void aarch64_sync_64_to_32(CPUARMState *env); +int fp_exception_el(CPUARMState *env, int cur_el); +int sve_exception_el(CPUARMState *env, int cur_el); +uint32_t sve_zcr_len_for_el(CPUARMState *env, int el); + static inline bool is_a64(CPUARMState *env) { #ifdef CONFIG_USER_ONLY diff --git a/target/arm/helper.c b/target/arm/helper.c index fb79b27cf6..64ff71b722 100644 --- a/target/arm/helper.c +++ b/target/arm/helper.c @@ -4344,7 +4344,7 @@ static const ARMCPRegInfo debug_lpae_cp_reginfo[] = { * take care of raising that exception. * C.f. the ARM pseudocode function CheckSVEEnabled. */ -static int sve_exception_el(CPUARMState *env, int el) +int sve_exception_el(CPUARMState *env, int el) { #ifndef CONFIG_USER_ONLY if (el <= 1) { @@ -4402,7 +4402,7 @@ static int sve_exception_el(CPUARMState *env, int el) /* * Given that SVE is enabled, return the vector length for EL. */ -static uint32_t sve_zcr_len_for_el(CPUARMState *env, int el) +uint32_t sve_zcr_len_for_el(CPUARMState *env, int el) { ARMCPU *cpu = arm_env_get_cpu(env); uint32_t zcr_len = cpu->sve_max_vq - 1; @@ -12352,7 +12352,7 @@ uint32_t HELPER(crc32c)(uint32_t acc, uint32_t val, uint32_t bytes) /* Return the exception level to which FP-disabled exceptions should * be taken, or 0 if FP is enabled. */ -static int fp_exception_el(CPUARMState *env, int cur_el) +int fp_exception_el(CPUARMState *env, int cur_el) { #ifndef CONFIG_USER_ONLY int fpen; diff --git a/target/arm/translate-a64.c b/target/arm/translate-a64.c index b29dc49c4f..4a0ca8c906 100644 --- a/target/arm/translate-a64.c +++ b/target/arm/translate-a64.c @@ -166,11 +166,15 @@ void aarch64_cpu_dump_state(CPUState *cs, FILE *f, cpu_fprintf(f, "\n"); return; } + if (fp_exception_el(env, el) != 0) { + cpu_fprintf(f, " FPU disabled\n"); + return; + } cpu_fprintf(f, " FPCR=%08x FPSR=%08x\n", vfp_get_fpcr(env), vfp_get_fpsr(env)); - if (arm_feature(env, ARM_FEATURE_SVE)) { - int j, zcr_len = env->vfp.zcr_el[1] & 0xf; /* fix for system mode */ + if (arm_feature(env, ARM_FEATURE_SVE) && sve_exception_el(env, el) == 0) { + int j, zcr_len = sve_zcr_len_for_el(env, el); for (i = 0; i <= FFR_PRED_NUM; i++) { bool eol; From patchwork Thu Aug 9 04:21:57 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 143668 Delivered-To: patch@linaro.org Received: by 2002:a2e:9754:0:0:0:0:0 with SMTP id f20-v6csp1608229ljj; Wed, 8 Aug 2018 21:31:19 -0700 (PDT) X-Google-Smtp-Source: AA+uWPx/HXmSidIRmS4nqx/evzta+szOUUVTcdSwrjQVCQ8L7IwPBGu0iPWHnWpp7TidIoN2MWT3 X-Received: by 2002:a37:3106:: with SMTP id x6-v6mr444535qkx.366.1533789079495; Wed, 08 Aug 2018 21:31:19 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1533789079; cv=none; d=google.com; s=arc-20160816; b=JyAxOy0v4YnwO8mAUm7wPoajUtxX37As7BnVCVNPED49OtQ3TG8Af4hwERjzQIs9ln 0GcaIoX5a9IsTbNdHJMdJIpakgXM2Q5P+nQBarjVb3vRjJbGJU1YWQfIOP2tPYVEpeWQ zXDnRlm/Nv3T3vD7j7m8SrnGRJ1al8js94lVNTLP0AF3O+/Wo0Y8B89c9c6fBZBd/701 +frFpa0PNg6cg7tGxgN4SKr583sUa6lUhRn2csBvftMyOGaBnnf8ZMT+QzqsWi+e23IG m+RXu3lpHtD25JW8vAcOrsdiwwGc/zZgP0RgqLHxakupZOHPmX+jmYmfpN7xW4NSRUxL W7Nw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:cc:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:subject:references:in-reply-to :message-id:date:to:from:dkim-signature:arc-authentication-results; bh=Ze8ZdJDqn6zsWMoBANBrHY4udg1WT+Qa4bIr6PsHaec=; b=iMHgrlT5aMzLoeUyfvmE7DuEVG8eazd12aApCVxRoTaGxPxZMOFpogh1ho310BFEos SisQP/OrBpnH1vFzMktOzLZswquIt+qTV+0Oba3MgS4lYfrQ4mYwIohzGlthvKUd1Ysu BWMJ8y2TYkmY3tlSkUwI5HrhE4NWIr9M6eYKb7i6XJ7dV2FoQX2WzsYxyKUCa8SgHKwS n7+WsboxRj5sSZqsbhKhni5ex1vsnGnnYJ653vDJ9zMsorAGFhl6JwI5bE+3pTzXdy8p LDV78uCASVCPTROTKajxsVeY6wulV5PasIWnACqbn1+aKRmF8EPLcXsmUuCjmWM+8Ggv +tbw== ARC-Authentication-Results: i=1; mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b="e/qei9Cv"; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [2001:4830:134:3::11]) by mx.google.com with ESMTPS id z2-v6si1975527qvi.288.2018.08.08.21.31.19 for (version=TLS1 cipher=AES128-SHA bits=128/128); Wed, 08 Aug 2018 21:31:19 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) client-ip=2001:4830:134:3::11; Authentication-Results: mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b="e/qei9Cv"; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1]:47273 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fncbW-0000zX-Un for patch@linaro.org; Thu, 09 Aug 2018 00:31:18 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:54058) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fncSw-0000sl-GW for qemu-devel@nongnu.org; Thu, 09 Aug 2018 00:22:27 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fncSv-0007Kh-KA for qemu-devel@nongnu.org; Thu, 09 Aug 2018 00:22:26 -0400 Received: from mail-pg1-x52d.google.com ([2607:f8b0:4864:20::52d]:41706) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fncSv-0007K5-DF for qemu-devel@nongnu.org; Thu, 09 Aug 2018 00:22:25 -0400 Received: by mail-pg1-x52d.google.com with SMTP id z8-v6so2107869pgu.8 for ; Wed, 08 Aug 2018 21:22:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=Ze8ZdJDqn6zsWMoBANBrHY4udg1WT+Qa4bIr6PsHaec=; b=e/qei9CvSkQMXtMVhwDal6FZ1VIphr0EApU1biWvNhpHqkEkZyyRT1iHuuRWhzFbXT TzX49Hxg/x46ngiKk7wIfSecasLKnfmP73iZ7YTYJNCmkurTSF/7s1FdeRwsN0Ns4Qhy 5Xfg9+8pcLKXpkZIE+EdJCA+oiiBtq6XhhpwE= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=Ze8ZdJDqn6zsWMoBANBrHY4udg1WT+Qa4bIr6PsHaec=; b=cKvUm1vhxfACQ6M9ls42JQ+TiRLowZYUV428i0FkouIaR5zDfLs6deo+FMlJbx19Gu uXIN2pUgkZoLU5hl673vYe8Q8HI7PZHoEUK5t/sfqUmB9fD91MQhiTWwi+Cfw2h5Az1w IFmh5DJDoe/+b8SQvoyt9cJ/yR9km16nwhkDKh8Un8GOq1amQ5ow6xW5vIlFbB6L4Yoc Qab7K8994NOy4qtyVEEua/20GDTHqU0s7Y3phMSQn0J0HIs/aGwSgYQ21GrcOZqeWtIp k0rWBBjWr9m3TC2Q7UPFTBL50aiIu3VKLhFc70aTuYZ4kengbK5MsbT2ihLMrXew39Sm QP0g== X-Gm-Message-State: AOUpUlEMOb/rlOKJUfRujOYP0Gs6CBbBpe9es2PXi5+y2ibMXSeT2O6v YIF5LB4vrLgF6T1auui/lv8SohlD/E4= X-Received: by 2002:a63:5055:: with SMTP id q21-v6mr466933pgl.397.1533788544191; Wed, 08 Aug 2018 21:22:24 -0700 (PDT) Received: from cloudburst.twiddle.net (97-113-8-179.tukw.qwest.net. [97.113.8.179]) by smtp.gmail.com with ESMTPSA id m30-v6sm7355799pff.121.2018.08.08.21.22.22 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Wed, 08 Aug 2018 21:22:23 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Date: Wed, 8 Aug 2018 21:21:57 -0700 Message-Id: <20180809042206.15726-12-richard.henderson@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180809042206.15726-1-richard.henderson@linaro.org> References: <20180809042206.15726-1-richard.henderson@linaro.org> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::52d Subject: [Qemu-devel] [PATCH 11/20] target/arm: Clear unused predicate bits for LD1RQ X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: laurent.desnogues@gmail.com, peter.maydell@linaro.org, alex.bennee@linaro.org Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: "Qemu-devel" The 16-byte load only uses 16 predicate bits. But while reusing the other load infrastructure, we find other bits that are set and trigger an assert. To avoid this and retain the assert, zero-extend the predicate that we pass to the LD1 helper. Reported-by: Laurent Desnogues Signed-off-by: Richard Henderson --- target/arm/translate-sve.c | 25 +++++++++++++++++++++++-- 1 file changed, 23 insertions(+), 2 deletions(-) -- 2.17.1 Reviewed-by: Peter Maydell diff --git a/target/arm/translate-sve.c b/target/arm/translate-sve.c index d27bc8c946..bef6b8242d 100644 --- a/target/arm/translate-sve.c +++ b/target/arm/translate-sve.c @@ -4765,12 +4765,33 @@ static void do_ldrq(DisasContext *s, int zt, int pg, TCGv_i64 addr, int msz) unsigned vsz = vec_full_reg_size(s); TCGv_ptr t_pg; TCGv_i32 desc; + int poff; /* Load the first quadword using the normal predicated load helpers. */ desc = tcg_const_i32(simd_desc(16, 16, zt)); - t_pg = tcg_temp_new_ptr(); - tcg_gen_addi_ptr(t_pg, cpu_env, pred_full_reg_offset(s, pg)); + poff = pred_full_reg_offset(s, pg); + if (vsz > 16) { + /* + * Zero-extend the first 16 bits of the predicate into a temporary. + * This avoids triggering an assert making sure we don't have bits + * set within a predicate beyond VQ, but we have lowered VQ to 1 + * for this load operation. + */ + TCGv_i64 tmp = tcg_temp_new_i64(); +#ifdef HOST_WORDS_BIGENDIAN + poff += 6; +#endif + tcg_gen_ld16u_i64(tmp, cpu_env, poff); + + poff = offsetof(CPUARMState, vfp.preg_tmp); + tcg_gen_st_i64(tmp, cpu_env, poff); + tcg_temp_free_i64(tmp); + } + + t_pg = tcg_temp_new_ptr(); + tcg_gen_addi_ptr(t_pg, cpu_env, poff); + fns[msz](cpu_env, t_pg, addr, desc); tcg_temp_free_ptr(t_pg); From patchwork Thu Aug 9 04:21:58 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 143670 Delivered-To: patch@linaro.org Received: by 2002:a2e:9754:0:0:0:0:0 with SMTP id f20-v6csp1611285ljj; Wed, 8 Aug 2018 21:35:39 -0700 (PDT) X-Google-Smtp-Source: AA+uWPyWV95c5s/kHjr3BNr1AW6/F6pPdBZN4fLJUFKZ6ulbdRxi94MxUkpMUyfkN7gK5jREseUc X-Received: by 2002:aed:23f8:: with SMTP id k53-v6mr481323qtc.230.1533789339601; Wed, 08 Aug 2018 21:35:39 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1533789339; cv=none; d=google.com; s=arc-20160816; b=q/cYjEGT72epruxFONfSBPW89N2zG69IlccmP0EvozSDXW4G6aPMKiO0b5NEF54yeA OaJtLgkqJtLulzWQ6hTuSuCE5NMlykQifwxwCiyTNjEMhQasU/j73PzwkaGQcBikMaan zT4Eiyjoiy0EGeBetkkba9Kr5fiEf8SlMxC+OFAYHnVccwS0K7XoY9JwYjH8SFEe+S6P lPMxtq2BSenrorkV+98zkvGNzLTu97ync3Q6WYAWPrEglF2JZ2OH268H+2iyHN0tgm5s tak2vUi3qfzn4F1ETTqz22DApLuEjzkx0ohDUgheNnFuv0ZRk7PD5pPLppr2HdBZY7YT LQpA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:cc:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:subject:references:in-reply-to :message-id:date:to:from:dkim-signature:arc-authentication-results; bh=suxGo2uB+X2J7hOEpgGfAbZy9Aiz9wXIbZligPkw1xA=; b=I8fbCsj/3aDZh9ezhDsmukdzg/QkbYCiOU8AkHOR3CfzKPihNJgDvVaRJlWzLiiAJt yyVgFzhlDm4tcA/3j+aMXGsdiQwAPgBnB/xWOm01MgdhMMaluwfiVFRTBAWo1WZsrtNx 1KpdE1ZQdiF4lnb37Rlk2LHAFstfccnxlaNW3nI31RsVCsEEm7Wc1wW8+b3os20Th+Op bZj0aRuOvne6pkXO9Rjyq3J0lK8jn1YiOTHA8XGb9fR9oDkYDJI0dyOGuLgsgcvlLfGG f/skdDv1WGMVOyHSSdg5Em62pau6DBfcn2ri/TshaymwJIhNvQDKL50DZ5oUgCpbtYar DH7w== ARC-Authentication-Results: i=1; mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=WAs3vScU; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [2001:4830:134:3::11]) by mx.google.com with ESMTPS id t13-v6si5169892qto.159.2018.08.08.21.35.39 for (version=TLS1 cipher=AES128-SHA bits=128/128); Wed, 08 Aug 2018 21:35:39 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) client-ip=2001:4830:134:3::11; Authentication-Results: mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=WAs3vScU; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1]:47288 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fncfi-0003Th-Gh for patch@linaro.org; Thu, 09 Aug 2018 00:35:39 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:54079) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fncT0-0000xu-Pa for qemu-devel@nongnu.org; Thu, 09 Aug 2018 00:22:35 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fncSx-0007MS-Gx for qemu-devel@nongnu.org; Thu, 09 Aug 2018 00:22:30 -0400 Received: from mail-pf1-x42d.google.com ([2607:f8b0:4864:20::42d]:32826) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fncSx-0007Le-4Q for qemu-devel@nongnu.org; Thu, 09 Aug 2018 00:22:27 -0400 Received: by mail-pf1-x42d.google.com with SMTP id d4-v6so2209689pfn.0 for ; Wed, 08 Aug 2018 21:22:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=suxGo2uB+X2J7hOEpgGfAbZy9Aiz9wXIbZligPkw1xA=; b=WAs3vScUjdD+Nst/qbDL41lSfsl8sOtocV8r1EUbCd/per39VY52E5DvrHOAj3WBiW SSTNvtOxdQYNn1/goFpFv6BnQ3iKGIh17YGDNRLRt79eqQirZiXVVGgMcoq5ECGbSvSE fHXIe44mBPmqjgXDNdJdSCyxTkeJV+LbaeopI= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=suxGo2uB+X2J7hOEpgGfAbZy9Aiz9wXIbZligPkw1xA=; b=WZlTz5iSsBrgoVIo7v832aSJozXRf/vQpzMmT4PmDVZMSqVRgehM+MNOldL4PfL5xm 9yEg4AIHTyOGkz2+pZyhmRYx+4H8PUaqH14DlGBIR5Vshbm8xDNC7Z8ozjGeRIFd6hA9 6DfJ9CzKd8gdp50ai5T6Bvt2i68ZgS5XHvRgQLCsucv1dhnLNYfchw+z07h4mBxiSLrZ lmJkLyYsqdUO7gNsTjOL529i8sB4YRLlnStd0/OEtG6dUZyqDvzqZJW6sro3sXqs+Yha Dk7yfto4PeMPWO+cc3Wg3uAPPtcXCpXWqfM8vZzWDe1Brl9KXjZe8fieix+BWw8IHw7O zPZg== X-Gm-Message-State: AOUpUlE/WQUp0d4NK+H6StiF1/Ugz2DeWKn1okaXPY28TC2hqYqnf5Ph YzQp0kyFRO2DlRVhRG/wOkQ25TtjFiw= X-Received: by 2002:a63:da04:: with SMTP id c4-v6mr495312pgh.398.1533788545529; Wed, 08 Aug 2018 21:22:25 -0700 (PDT) Received: from cloudburst.twiddle.net (97-113-8-179.tukw.qwest.net. [97.113.8.179]) by smtp.gmail.com with ESMTPSA id m30-v6sm7355799pff.121.2018.08.08.21.22.24 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Wed, 08 Aug 2018 21:22:24 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Date: Wed, 8 Aug 2018 21:21:58 -0700 Message-Id: <20180809042206.15726-13-richard.henderson@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180809042206.15726-1-richard.henderson@linaro.org> References: <20180809042206.15726-1-richard.henderson@linaro.org> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::42d Subject: [Qemu-devel] [PATCH 12/20] target/arm: Rewrite helper_sve_ld1*_r using pages X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: laurent.desnogues@gmail.com, peter.maydell@linaro.org, alex.bennee@linaro.org Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: "Qemu-devel" Uses tlb_vaddr_to_host for correct operation with softmmu. Optimize for accesses within a single page or pair of pages. Perf report comparison for cortex-strings test-strlen with aarch64-linux-user: before: 1.59% qemu-aarch64 qemu-aarch64 [.] do_sve_ld1bb_r 0.86% qemu-aarch64 qemu-aarch64 [.] do_sve_ldff1bb_r after: 0.09% qemu-aarch64 qemu-aarch64 [.] helper_sve_ldff1bb_r 0.01% qemu-aarch64 qemu-aarch64 [.] sve_ld1bb_host Signed-off-by: Richard Henderson --- target/arm/sve_helper.c | 839 ++++++++++++++++++++++++++++++++-------- 1 file changed, 675 insertions(+), 164 deletions(-) -- 2.17.1 diff --git a/target/arm/sve_helper.c b/target/arm/sve_helper.c index e03f954a26..4ca9412e20 100644 --- a/target/arm/sve_helper.c +++ b/target/arm/sve_helper.c @@ -1688,6 +1688,45 @@ static void swap_memmove(void *vd, void *vs, size_t n) } } +/* Similarly for memset of 0. */ +static void swap_memzero(void *vd, size_t n) +{ + uintptr_t d = (uintptr_t)vd; + uintptr_t o = (d | n) & 7; + size_t i; + + if (likely(n == 0)) { + return; + } +#ifndef HOST_WORDS_BIGENDIAN + o = 0; +#endif + switch (o) { + case 0: + memset(vd, 0, n); + break; + + case 4: + for (i = 0; i < n; i += 4) { + *(uint32_t *)H1_4(d + i) = 0; + } + break; + + case 2: + case 6: + for (i = 0; i < n; i += 2) { + *(uint16_t *)H1_2(d + i) = 0; + } + break; + + default: + for (i = 0; i < n; i++) { + *(uint8_t *)H1(d + i) = 0; + } + break; + } +} + void HELPER(sve_ext)(void *vd, void *vn, void *vm, uint32_t desc) { intptr_t opr_sz = simd_oprsz(desc); @@ -3927,32 +3966,438 @@ void HELPER(sve_fcmla_zpzzz_d)(CPUARMState *env, void *vg, uint32_t desc) /* * Load contiguous data, protected by a governing predicate. */ -#define DO_LD1(NAME, FN, TYPEE, TYPEM, H) \ -static void do_##NAME(CPUARMState *env, void *vd, void *vg, \ - target_ulong addr, intptr_t oprsz, \ - uintptr_t ra) \ -{ \ - intptr_t i = 0; \ - do { \ - uint16_t pg = *(uint16_t *)(vg + H1_2(i >> 3)); \ - do { \ - TYPEM m = 0; \ - if (pg & 1) { \ - m = FN(env, addr, ra); \ - } \ - *(TYPEE *)(vd + H(i)) = m; \ - i += sizeof(TYPEE), pg >>= sizeof(TYPEE); \ - addr += sizeof(TYPEM); \ - } while (i & 15); \ - } while (i < oprsz); \ -} \ -void HELPER(NAME)(CPUARMState *env, void *vg, \ - target_ulong addr, uint32_t desc) \ -{ \ - do_##NAME(env, &env->vfp.zregs[simd_data(desc)], vg, \ - addr, simd_oprsz(desc), GETPC()); \ + +/* Load elements into VD, controlled by VG, from HOST+MEM_OFS. + * Memory is valid through MEM_MAX. The register element indicies + * are inferred from MEM_OFS, as modified by the types for which + * the helper is built. Return the MEM_OFS of the first element + * not loaded (which is MEM_MAX if they are all loaded). + * + * For softmmu, we have fully validated the guest page. For user-only, + * we cannot fully validate without taking the mmap lock, but since we + * know the access is within one host page, if any access is valid they + * all must be valid. However, it may be that no access is valid and + * they have all been predicated false. + */ +typedef intptr_t sve_ld1_host_fn(void *vd, void *vg, void *host, + intptr_t mem_ofs, intptr_t mem_max); + +/* Load one element into VD+REG_OFF from (ENV,VADDR,RA). + * The controlling predicate is known to be true. + */ +typedef void sve_ld1_tlb_fn(CPUARMState *env, void *vd, intptr_t reg_off, + target_ulong vaddr, int mmu_idx, uintptr_t ra); + +/* + * Generate the above primitives. + */ + +#define DO_LD_HOST(NAME, H, TYPEE, TYPEM, HOST) \ +static intptr_t sve_##NAME##_host(void *vd, void *vg, void *host, \ + intptr_t mem_off, const intptr_t mem_max) \ +{ \ + intptr_t reg_off = mem_off * (sizeof(TYPEE) / sizeof(TYPEM)); \ + uint64_t *pg = vg; \ + while (mem_off + sizeof(TYPEM) <= mem_max) { \ + TYPEM val = 0; \ + if (likely((pg[reg_off >> 6] >> (reg_off & 63)) & 1)) { \ + val = HOST(host + mem_off); \ + } \ + *(TYPEE *)(vd + H(reg_off)) = val; \ + mem_off += sizeof(TYPEM), reg_off += sizeof(TYPEE); \ + } \ + return mem_off; \ } +#ifdef CONFIG_SOFTMMU +#define DO_LD_TLB(NAME, H, TYPEE, TYPEM, HOST, MOEND, TLB) \ +static void sve_##NAME##_tlb(CPUARMState *env, void *vd, intptr_t reg_off, \ + target_ulong addr, int mmu_idx, uintptr_t ra) \ +{ \ + TCGMemOpIdx oi = make_memop_idx(ctz32(sizeof(TYPEM)) | MOEND, mmu_idx); \ + TYPEM val = TLB(env, addr, oi, ra); \ + *(TYPEE *)(vd + H(reg_off)) = val; \ +} +#else +#define DO_LD_TLB(NAME, H, TYPEE, TYPEM, HOST, MOEND, TLB) \ +static void sve_##NAME##_tlb(CPUARMState *env, void *vd, intptr_t reg_off, \ + target_ulong addr, int mmu_idx, uintptr_t ra) \ +{ \ + TYPEM val = HOST(g2h(addr)); \ + *(TYPEE *)(vd + H(reg_off)) = val; \ +} +#endif + +DO_LD_TLB(ld1bb, H1, uint8_t, uint8_t, ldub_p, 0, helper_ret_ldub_mmu) + +#define DO_LD_PRIM_1(NAME, H, TE, TM) \ + DO_LD_HOST(NAME, H, TE, TM, ldub_p) \ + DO_LD_TLB(NAME, H, TE, TM, ldub_p, 0, helper_ret_ldub_mmu) + +DO_LD_PRIM_1(ld1bhu, H1_2, uint16_t, uint8_t) +DO_LD_PRIM_1(ld1bhs, H1_2, uint16_t, int8_t) +DO_LD_PRIM_1(ld1bsu, H1_4, uint32_t, uint8_t) +DO_LD_PRIM_1(ld1bss, H1_4, uint32_t, int8_t) +DO_LD_PRIM_1(ld1bdu, , uint64_t, uint8_t) +DO_LD_PRIM_1(ld1bds, , uint64_t, int8_t) + +#define DO_LD_PRIM_2(NAME, end, MOEND, H, TE, TM, PH, PT) \ + DO_LD_HOST(NAME##_##end, H, TE, TM, PH##_##end##_p) \ + DO_LD_TLB(NAME##_##end, H, TE, TM, PH##_##end##_p, \ + MOEND, helper_##end##_##PT##_mmu) + +DO_LD_PRIM_2(ld1hh, le, MO_LE, H1_2, uint16_t, uint16_t, lduw, lduw) +DO_LD_PRIM_2(ld1hsu, le, MO_LE, H1_4, uint32_t, uint16_t, lduw, lduw) +DO_LD_PRIM_2(ld1hss, le, MO_LE, H1_4, uint32_t, int16_t, lduw, lduw) +DO_LD_PRIM_2(ld1hdu, le, MO_LE, , uint64_t, uint16_t, lduw, lduw) +DO_LD_PRIM_2(ld1hds, le, MO_LE, , uint64_t, int16_t, lduw, lduw) + +DO_LD_PRIM_2(ld1ss, le, MO_LE, H1_4, uint32_t, uint32_t, ldl, ldul) +DO_LD_PRIM_2(ld1sdu, le, MO_LE, , uint64_t, uint32_t, ldl, ldul) +DO_LD_PRIM_2(ld1sds, le, MO_LE, , uint64_t, int32_t, ldl, ldul) + +DO_LD_PRIM_2(ld1dd, le, MO_LE, , uint64_t, uint64_t, ldq, ldq) + +DO_LD_PRIM_2(ld1hh, be, MO_BE, H1_2, uint16_t, uint16_t, lduw, lduw) +DO_LD_PRIM_2(ld1hsu, be, MO_BE, H1_4, uint32_t, uint16_t, lduw, lduw) +DO_LD_PRIM_2(ld1hss, be, MO_BE, H1_4, uint32_t, int16_t, lduw, lduw) +DO_LD_PRIM_2(ld1hdu, be, MO_BE, , uint64_t, uint16_t, lduw, lduw) +DO_LD_PRIM_2(ld1hds, be, MO_BE, , uint64_t, int16_t, lduw, lduw) + +DO_LD_PRIM_2(ld1ss, be, MO_BE, H1_4, uint32_t, uint32_t, ldl, ldul) +DO_LD_PRIM_2(ld1sdu, be, MO_BE, , uint64_t, uint32_t, ldl, ldul) +DO_LD_PRIM_2(ld1sds, be, MO_BE, , uint64_t, int32_t, ldl, ldul) + +DO_LD_PRIM_2(ld1dd, be, MO_BE, , uint64_t, uint64_t, ldq, ldq) + +#undef DO_LD_TLB +#undef DO_LD_HOST +#undef DO_LD_PRIM_1 +#undef DO_LD_PRIM_2 + +/* + * Special case contiguous loads of bytes to accellerate strings. + * + * The assumption is that the governing predicate will be mostly true. + * When it is not all true, it has been set by whilelo and so has a + * block of true elements followed by a block of false elements. + * Thus anything we can do to handle as many bytes as possible in one + * step will pay dividends. + * + * Because of how vector registers are represented in CPUARMState, + * each block of 8 can be read with a little-endian load to be stored + * into the vector register in host-endian order. + * + * TODO: For LE host and LE guest (by far the most common combination), + * the only difference for other non-extending loads is the controlling + * predicate. Even for other combinations, it might be fastest to use + * this primitive to block load all of the data and then reorder the + * bytes afterward. + */ + +/* For user-only, conditionally load and mask from HOST, returning 0 + * if the predicate is false. This is required because, as described + * above, we have not fully validated the page, and faults are not + * permitted when the predicate is false. + * For softmmu, we never arrive here with invalid host memory; just mask. + */ +static inline uint64_t ldq_le_pred_b(uint8_t pg, void *host) +{ +#ifdef CONFIG_USER_ONLY + if (pg == 0) { + return 0; + } +#endif + return ldq_le_p(host) & expand_pred_b(pg); +} + +static inline uint8_t ldub_pred(uint8_t pg, void *host) +{ +#ifdef CONFIG_USER_ONLY + return pg & 1 ? ldub_p(host) : 0; +#else + return ldub_p(host) & -(pg & 1); +#endif +} + +static intptr_t sve_ld1bb_host(void *vd, void *vg, void *host, + intptr_t off, const intptr_t max) +{ + uint64_t *d = vd; + uint8_t *g = vg; + + /* Assuming OFF and MAX may be misaligned, but also the most common + * case is an entire vector register: OFF == 0, MAX % 16 == 0. + */ + if (likely(off + 8 <= max)) { + const intptr_t max_div_8 = max >> 3; + intptr_t off_div_8 = off >> 3; + uint64_t data; + + if (unlikely(off & 63)) { + /* Align for a loop-of-8. We know from the range check + * above that we have enough remaining to load 8 bytes. + */ + if (unlikely(off & 7)) { + int off_7 = off & 7; + uint8_t pg = g[H1(off_div_8)] >> off_7; + + off_7 *= 8; + data = ldq_le_pred_b(pg, host + off); + data = deposit64(d[off_div_8], off_7, 64 - off_7, data); + d[off_div_8] = data; + + off_div_8 += 1; + } + + /* If there are not sufficient bytes to align for 64 + * and also execute that loop at least once, skip to tail. + */ + if (ROUND_UP(off_div_8, 8) + 8 > max_div_8) { + goto skip_64; + } + + /* Align for the loop-of-64. */ + if (unlikely(off_div_8 & 7)) { + do { + uint8_t pg = g[off_div_8]; + data = ldq_le_pred_b(pg, host + off_div_8 * 8); + d[off_div_8] = data; + } while (++off_div_8 & 7); + } + } + + /* While we have blocks of 64 remaining, we can perform tests + * against large blocks of predicates at once. + */ + for (; off_div_8 + 8 <= max_div_8; off_div_8 += 8) { + uint64_t pg = *(uint64_t *)(g + off_div_8); + if (likely(pg == -1ULL)) { +#ifndef HOST_WORDS_BIGENDIAN + memcpy(d + off_div_8, host + off_div_8 * 8, 64); +#else + intptr_t j; + for (j = 0; j < 8; j++) { + data = ldq_le_p(host + (off_div_8 + j) * 8); + d[off_div_8 + j] = data; + } +#endif + } else if (pg == 0) { + memset(d + off_div_8, 0, 64); + } else { + intptr_t j; + for (j = 0; j < 8; j++) { + data = ldq_le_pred_b(pg >> (j * 8), + host + (off_div_8 + j) * 8); + d[off_div_8 + j] = data; + } + } + } + + skip_64: + /* Final tail or a copy smaller than 64 bytes. */ + for (; off_div_8 < max_div_8; off_div_8++) { + uint8_t pg = g[H1(off_div_8)]; + data = ldq_le_pred_b(pg, host + off_div_8 * 8); + d[off_div_8] = data; + } + + /* Restore using OFF. */ + off = off_div_8 * 8; + } + + /* Final tail or a really small copy. */ + if (unlikely(off < max)) { + do { + uint8_t pg = g[H1(off >> 3)] >> (off & 7); + ((uint8_t *)vd)[H1(off)] = ldub_pred(pg, host + off); + } while (++off < max); + } + + return max; +} + +/* Skip through a sequence of inactive elements in the guarding predicate VG, + * beginning at REG_OFF bounded by REG_MAX. Return the offset of the active + * element >= REG_OFF, or REG_MAX if there were no active elements at all. + */ +static intptr_t find_next_active(uint64_t *vg, intptr_t reg_off, + intptr_t reg_max, int esz) +{ + uint64_t pg_mask = pred_esz_masks[esz]; + uint64_t pg = (vg[reg_off >> 6] & pg_mask) >> (reg_off & 63); + + /* In normal usage, the first element is active. */ + if (likely(pg & 1)) { + return reg_off; + } + + if (pg == 0) { + reg_off &= -64; + do { + reg_off += 64; + if (unlikely(reg_off >= reg_max)) { + /* The entire predicate was false. */ + return reg_max; + } + pg = vg[reg_off >> 6] & pg_mask; + } while (pg == 0); + } + reg_off += ctz64(pg); + + /* We should never see an out of range predicate bit set. */ + tcg_debug_assert(reg_off < reg_max); + return reg_off; +} + +/* Return the maximum offset <= MEM_MAX which is still within the page + * referenced by BASE+MEM_OFF. + */ +static intptr_t max_for_page(target_ulong base, intptr_t mem_off, + intptr_t mem_max) +{ + target_ulong addr = base + mem_off; + intptr_t split = -(intptr_t)(addr | TARGET_PAGE_MASK); + return MIN(split, mem_max - mem_off) + mem_off; +} + +static inline void set_helper_retaddr(uintptr_t ra) +{ +#ifdef CONFIG_USER_ONLY + helper_retaddr = ra; +#endif +} + +static inline bool test_host_page(void *host) +{ +#ifdef CONFIG_USER_ONLY + return true; +#else + return likely(host != NULL); +#endif +} + +/* + * Common helper for all contiguous one-register predicated loads. + */ +static void sve_ld1_r(CPUARMState *env, void *vg, const target_ulong addr, + uint32_t desc, const uintptr_t retaddr, + const int esz, const int msz, + sve_ld1_host_fn *host_fn, + sve_ld1_tlb_fn *tlb_fn) +{ + void *vd = &env->vfp.zregs[simd_data(desc)]; + const int diffsz = esz - msz; + const intptr_t reg_max = simd_oprsz(desc); + const intptr_t mem_max = reg_max >> diffsz; + const int mmu_idx = cpu_mmu_index(env, false); + ARMVectorReg scratch; + void *host, *result; + intptr_t split; + + set_helper_retaddr(retaddr); + + host = tlb_vaddr_to_host(env, addr, MMU_DATA_LOAD, mmu_idx); + if (test_host_page(host)) { + split = max_for_page(addr, 0, mem_max); + if (likely(split == mem_max)) { + /* The load is entirely within a valid page. For softmmu, + * no faults. For user-only, if the first byte does not + * fault then none of them will fault, so Vd will never be + * partially modified. + */ + host_fn(vd, vg, host, 0, mem_max); + set_helper_retaddr(0); + return; + } + } + + /* Perform the predicated read into a temporary, thus ensuring + * if the load of the last element faults, Vd is not modified. + */ + result = &scratch; +#ifdef CONFIG_USER_ONLY + host_fn(vd, vg, host, 0, mem_max); +#else + memset(result, 0, reg_max); + for (intptr_t reg_off = find_next_active(vg, 0, reg_max, esz); + reg_off < reg_max; + reg_off = find_next_active(vg, reg_off, reg_max, esz)) { + intptr_t mem_off = reg_off >> diffsz; + + split = max_for_page(addr, mem_off, mem_max); + if (msz == 0 || split - mem_off >= (1 << msz)) { + /* At least one whole element on this page. */ + host = tlb_vaddr_to_host(env, addr + mem_off, + MMU_DATA_LOAD, mmu_idx); + if (host) { + mem_off = host_fn(result, vg, host - mem_off, mem_off, split); + reg_off = mem_off << diffsz; + continue; + } + } + + /* Perform one normal read. This may fault, longjmping out to the + * main loop in order to raise an exception. It may succeed, and + * as a side-effect load the TLB entry for the next round. Finally, + * in the extremely unlikely case we're performing this operation + * on I/O memory, it may succeed but not bring in the TLB entry. + * But even then we have still made forward progress. + */ + tlb_fn(env, result, reg_off, addr + mem_off, mmu_idx, retaddr); + reg_off += 1 << esz; + } +#endif + + set_helper_retaddr(0); + memcpy(vd, result, reg_max); +} + +#define DO_LD1_1(NAME, ESZ) \ +void HELPER(sve_##NAME##_r)(CPUARMState *env, void *vg, \ + target_ulong addr, uint32_t desc) \ +{ \ + sve_ld1_r(env, vg, addr, desc, GETPC(), ESZ, 0, \ + sve_##NAME##_host, sve_##NAME##_tlb); \ +} + +/* TODO: Propagate the endian check back to the translator. */ +#define DO_LD1_2(NAME, ESZ, MSZ) \ +void HELPER(sve_##NAME##_r)(CPUARMState *env, void *vg, \ + target_ulong addr, uint32_t desc) \ +{ \ + if (arm_cpu_data_is_big_endian(env)) { \ + sve_ld1_r(env, vg, addr, desc, GETPC(), ESZ, MSZ, \ + sve_##NAME##_be_host, sve_##NAME##_be_tlb); \ + } else { \ + sve_ld1_r(env, vg, addr, desc, GETPC(), ESZ, MSZ, \ + sve_##NAME##_le_host, sve_##NAME##_le_tlb); \ + } \ +} + +DO_LD1_1(ld1bb, 0) +DO_LD1_1(ld1bhu, 1) +DO_LD1_1(ld1bhs, 1) +DO_LD1_1(ld1bsu, 2) +DO_LD1_1(ld1bss, 2) +DO_LD1_1(ld1bdu, 3) +DO_LD1_1(ld1bds, 3) + +DO_LD1_2(ld1hh, 1, 1) +DO_LD1_2(ld1hsu, 2, 1) +DO_LD1_2(ld1hss, 2, 1) +DO_LD1_2(ld1hdu, 3, 1) +DO_LD1_2(ld1hds, 3, 1) + +DO_LD1_2(ld1ss, 2, 2) +DO_LD1_2(ld1sdu, 3, 2) +DO_LD1_2(ld1sds, 3, 2) + +DO_LD1_2(ld1dd, 3, 3) + +#undef DO_LD1_1 +#undef DO_LD1_2 + #define DO_LD2(NAME, FN, TYPEE, TYPEM, H) \ void HELPER(NAME)(CPUARMState *env, void *vg, \ target_ulong addr, uint32_t desc) \ @@ -4037,52 +4482,40 @@ void HELPER(NAME)(CPUARMState *env, void *vg, \ } \ } -DO_LD1(sve_ld1bhu_r, cpu_ldub_data_ra, uint16_t, uint8_t, H1_2) -DO_LD1(sve_ld1bhs_r, cpu_ldsb_data_ra, uint16_t, int8_t, H1_2) -DO_LD1(sve_ld1bsu_r, cpu_ldub_data_ra, uint32_t, uint8_t, H1_4) -DO_LD1(sve_ld1bss_r, cpu_ldsb_data_ra, uint32_t, int8_t, H1_4) -DO_LD1(sve_ld1bdu_r, cpu_ldub_data_ra, uint64_t, uint8_t, ) -DO_LD1(sve_ld1bds_r, cpu_ldsb_data_ra, uint64_t, int8_t, ) - -DO_LD1(sve_ld1hsu_r, cpu_lduw_data_ra, uint32_t, uint16_t, H1_4) -DO_LD1(sve_ld1hss_r, cpu_ldsw_data_ra, uint32_t, int16_t, H1_4) -DO_LD1(sve_ld1hdu_r, cpu_lduw_data_ra, uint64_t, uint16_t, ) -DO_LD1(sve_ld1hds_r, cpu_ldsw_data_ra, uint64_t, int16_t, ) - -DO_LD1(sve_ld1sdu_r, cpu_ldl_data_ra, uint64_t, uint32_t, ) -DO_LD1(sve_ld1sds_r, cpu_ldl_data_ra, uint64_t, int32_t, ) - -DO_LD1(sve_ld1bb_r, cpu_ldub_data_ra, uint8_t, uint8_t, H1) DO_LD2(sve_ld2bb_r, cpu_ldub_data_ra, uint8_t, uint8_t, H1) DO_LD3(sve_ld3bb_r, cpu_ldub_data_ra, uint8_t, uint8_t, H1) DO_LD4(sve_ld4bb_r, cpu_ldub_data_ra, uint8_t, uint8_t, H1) -DO_LD1(sve_ld1hh_r, cpu_lduw_data_ra, uint16_t, uint16_t, H1_2) DO_LD2(sve_ld2hh_r, cpu_lduw_data_ra, uint16_t, uint16_t, H1_2) DO_LD3(sve_ld3hh_r, cpu_lduw_data_ra, uint16_t, uint16_t, H1_2) DO_LD4(sve_ld4hh_r, cpu_lduw_data_ra, uint16_t, uint16_t, H1_2) -DO_LD1(sve_ld1ss_r, cpu_ldl_data_ra, uint32_t, uint32_t, H1_4) DO_LD2(sve_ld2ss_r, cpu_ldl_data_ra, uint32_t, uint32_t, H1_4) DO_LD3(sve_ld3ss_r, cpu_ldl_data_ra, uint32_t, uint32_t, H1_4) DO_LD4(sve_ld4ss_r, cpu_ldl_data_ra, uint32_t, uint32_t, H1_4) -DO_LD1(sve_ld1dd_r, cpu_ldq_data_ra, uint64_t, uint64_t, ) DO_LD2(sve_ld2dd_r, cpu_ldq_data_ra, uint64_t, uint64_t, ) DO_LD3(sve_ld3dd_r, cpu_ldq_data_ra, uint64_t, uint64_t, ) DO_LD4(sve_ld4dd_r, cpu_ldq_data_ra, uint64_t, uint64_t, ) -#undef DO_LD1 #undef DO_LD2 #undef DO_LD3 #undef DO_LD4 /* * Load contiguous data, first-fault and no-fault. + * + * For user-only, one could argue that we should hold the mmap_lock during + * the operation so that there is no race between page_check_range and the + * load operation. However, unmapping pages out from under operating thread + * is extrodinarily unlikely. This theoretical race condition also affects + * linux-user/ in its get_user/put_user macros. + * + * TODO: Construct some helpers, written in assembly, that interact with + * handle_cpu_signal to produce memory ops which can properly report errors + * without racing. */ -#ifdef CONFIG_USER_ONLY - /* Fault on byte I. All bits in FFR from I are cleared. The vector * result from I is CONSTRAINED UNPREDICTABLE; we choose the MERGE * option, which leaves subsequent data unchanged. @@ -4092,147 +4525,225 @@ static void record_fault(CPUARMState *env, uintptr_t i, uintptr_t oprsz) uint64_t *ffr = env->vfp.pregs[FFR_PRED_NUM].p; if (i & 63) { - ffr[i / 64] &= MAKE_64BIT_MASK(0, i & 63); + ffr[i >> 6] &= MAKE_64BIT_MASK(0, i & 63); i = ROUND_UP(i, 64); } for (; i < oprsz; i += 64) { - ffr[i / 64] = 0; + ffr[i >> 6] = 0; } } -/* Hold the mmap lock during the operation so that there is no race - * between page_check_range and the load operation. We expect the - * usual case to have no faults at all, so we check the whole range - * first and if successful defer to the normal load operation. - * - * TODO: Change mmap_lock to a rwlock so that multiple readers - * can run simultaneously. This will probably help other uses - * within QEMU as well. +/* + * Common helper for all contiguous first-fault loads. */ -#define DO_LDFF1(PART, FN, TYPEE, TYPEM, H) \ -static void do_sve_ldff1##PART(CPUARMState *env, void *vd, void *vg, \ - target_ulong addr, intptr_t oprsz, \ - bool first, uintptr_t ra) \ -{ \ - intptr_t i = 0; \ - do { \ - uint16_t pg = *(uint16_t *)(vg + H1_2(i >> 3)); \ - do { \ - TYPEM m = 0; \ - if (pg & 1) { \ - if (!first && \ - unlikely(page_check_range(addr, sizeof(TYPEM), \ - PAGE_READ))) { \ - record_fault(env, i, oprsz); \ - return; \ - } \ - m = FN(env, addr, ra); \ - first = false; \ - } \ - *(TYPEE *)(vd + H(i)) = m; \ - i += sizeof(TYPEE), pg >>= sizeof(TYPEE); \ - addr += sizeof(TYPEM); \ - } while (i & 15); \ - } while (i < oprsz); \ -} \ -void HELPER(sve_ldff1##PART)(CPUARMState *env, void *vg, \ - target_ulong addr, uint32_t desc) \ -{ \ - intptr_t oprsz = simd_oprsz(desc); \ - unsigned rd = simd_data(desc); \ - void *vd = &env->vfp.zregs[rd]; \ - mmap_lock(); \ - if (likely(page_check_range(addr, oprsz, PAGE_READ) == 0)) { \ - do_sve_ld1##PART(env, vd, vg, addr, oprsz, GETPC()); \ - } else { \ - do_sve_ldff1##PART(env, vd, vg, addr, oprsz, true, GETPC()); \ - } \ - mmap_unlock(); \ -} +static void sve_ldff1_r(CPUARMState *env, void *vg, const target_ulong addr, + uint32_t desc, const uintptr_t retaddr, + const int esz, const int msz, + sve_ld1_host_fn *host_fn, + sve_ld1_tlb_fn *tlb_fn) +{ + void *vd = &env->vfp.zregs[simd_data(desc)]; + const int diffsz = esz - msz; + const intptr_t reg_max = simd_oprsz(desc); + const intptr_t mem_max = reg_max >> diffsz; + const int mmu_idx = cpu_mmu_index(env, false); + intptr_t split, reg_off, mem_off; + void *host; -/* No-fault loads are like first-fault loads without the - * first faulting special case. - */ -#define DO_LDNF1(PART) \ -void HELPER(sve_ldnf1##PART)(CPUARMState *env, void *vg, \ - target_ulong addr, uint32_t desc) \ -{ \ - intptr_t oprsz = simd_oprsz(desc); \ - unsigned rd = simd_data(desc); \ - void *vd = &env->vfp.zregs[rd]; \ - mmap_lock(); \ - if (likely(page_check_range(addr, oprsz, PAGE_READ) == 0)) { \ - do_sve_ld1##PART(env, vd, vg, addr, oprsz, GETPC()); \ - } else { \ - do_sve_ldff1##PART(env, vd, vg, addr, oprsz, false, GETPC()); \ - } \ - mmap_unlock(); \ -} + set_helper_retaddr(retaddr); + split = max_for_page(addr, 0, mem_max); + if (likely(split == mem_max)) { + /* The entire operation is within one page. */ + host = tlb_vaddr_to_host(env, addr, MMU_DATA_LOAD, mmu_idx); + if (test_host_page(host)) { + mem_off = host_fn(vd, vg, host, 0, mem_max); + tcg_debug_assert(mem_off == mem_max); + set_helper_retaddr(0); + return; + } + } + + /* Skip to the first true predicate. */ + reg_off = find_next_active(vg, 0, reg_max, esz); + if (unlikely(reg_off == reg_max)) { + /* The entire predicate was false; no load occurs. */ + set_helper_retaddr(0); + memset(vd, 0, reg_max); + return; + } + mem_off = reg_off >> diffsz; + +#ifdef CONFIG_USER_ONLY + /* The page(s) containing this first element at ADDR+MEM_OFF must + * be valid. Considering that this first element may be misaligned + * and cross a page boundary itself, take the rest of the page from + * the last byte of the element. + */ + split = max_for_page(addr, mem_off + (1 << msz) - 1, mem_max); + mem_off = host_fn(vd, vg, g2h(addr), mem_off, split); + + /* After any fault, zero any leading predicated false elts. */ + swap_memzero(vd, reg_off); + reg_off = mem_off << diffsz; #else + /* Perform one normal read, which will fault or not. + * But it is likely to bring the page into the tlb. + */ + tlb_fn(env, vd, reg_off, addr + mem_off, mmu_idx, retaddr); -/* TODO: System mode is not yet supported. - * This would probably use tlb_vaddr_to_host. - */ -#define DO_LDFF1(PART, FN, TYPEE, TYPEM, H) \ -void HELPER(sve_ldff1##PART)(CPUARMState *env, void *vg, \ - target_ulong addr, uint32_t desc) \ -{ \ - g_assert_not_reached(); \ -} - -#define DO_LDNF1(PART) \ -void HELPER(sve_ldnf1##PART)(CPUARMState *env, void *vg, \ - target_ulong addr, uint32_t desc) \ -{ \ - g_assert_not_reached(); \ -} + /* After any fault, zero any leading predicated false elts. */ + swap_memzero(vd, reg_off); + mem_off += 1 << msz; + reg_off += 1 << esz; + /* Try again to read the balance of the page. */ + split = max_for_page(addr, mem_off - 1, mem_max); + if (split >= (1 << msz)) { + host = tlb_vaddr_to_host(env, addr + mem_off, MMU_DATA_LOAD, mmu_idx); + if (host) { + mem_off = host_fn(vd, vg, host - mem_off, mem_off, split); + reg_off = mem_off << diffsz; + } + } #endif -DO_LDFF1(bb_r, cpu_ldub_data_ra, uint8_t, uint8_t, H1) -DO_LDFF1(bhu_r, cpu_ldub_data_ra, uint16_t, uint8_t, H1_2) -DO_LDFF1(bhs_r, cpu_ldsb_data_ra, uint16_t, int8_t, H1_2) -DO_LDFF1(bsu_r, cpu_ldub_data_ra, uint32_t, uint8_t, H1_4) -DO_LDFF1(bss_r, cpu_ldsb_data_ra, uint32_t, int8_t, H1_4) -DO_LDFF1(bdu_r, cpu_ldub_data_ra, uint64_t, uint8_t, ) -DO_LDFF1(bds_r, cpu_ldsb_data_ra, uint64_t, int8_t, ) + set_helper_retaddr(0); + record_fault(env, reg_off, reg_max); +} -DO_LDFF1(hh_r, cpu_lduw_data_ra, uint16_t, uint16_t, H1_2) -DO_LDFF1(hsu_r, cpu_lduw_data_ra, uint32_t, uint16_t, H1_4) -DO_LDFF1(hss_r, cpu_ldsw_data_ra, uint32_t, int8_t, H1_4) -DO_LDFF1(hdu_r, cpu_lduw_data_ra, uint64_t, uint16_t, ) -DO_LDFF1(hds_r, cpu_ldsw_data_ra, uint64_t, int16_t, ) +/* + * Common helper for all contiguous no-fault loads. + */ +static void sve_ldnf1_r(CPUARMState *env, void *vg, const target_ulong addr, + uint32_t desc, const int esz, const int msz, + sve_ld1_host_fn *host_fn) +{ + void *vd = &env->vfp.zregs[simd_data(desc)]; + const int diffsz = esz - msz; + const intptr_t reg_max = simd_oprsz(desc); + const intptr_t mem_max = reg_max >> diffsz; + intptr_t split, reg_off, mem_off; + void *host; -DO_LDFF1(ss_r, cpu_ldl_data_ra, uint32_t, uint32_t, H1_4) -DO_LDFF1(sdu_r, cpu_ldl_data_ra, uint64_t, uint32_t, ) -DO_LDFF1(sds_r, cpu_ldl_data_ra, uint64_t, int32_t, ) +#ifdef CONFIG_USER_ONLY + /* Do not set helper_retaddr as there should be no fault. */ + host = g2h(addr); + if (likely(page_check_range(addr, mem_max, PAGE_READ) == 0)) { + /* The entire operation is valid. */ + host_fn(vd, vg, host, 0, mem_max); + return; + } +#else + const int mmu_idx = extract32(desc, SIMD_DATA_SHIFT, 4); + /* Unless we can load the entire vector from the same page, + * we need to search for the first active element. + */ + split = max_for_page(addr, 0, mem_max); + if (likely(split == mem_max)) { + host = tlb_vaddr_to_host(env, addr, MMU_DATA_LOAD, mmu_idx); + if (host) { + host_fn(vd, vg, host, 0, mem_max); + return; + } + } +#endif -DO_LDFF1(dd_r, cpu_ldq_data_ra, uint64_t, uint64_t, ) + /* There will be no fault, so we may modify in advance. */ + memset(vd, 0, reg_max); -#undef DO_LDFF1 + /* Skip to the first true predicate. */ + reg_off = find_next_active(vg, 0, reg_max, esz); + if (unlikely(reg_off == reg_max)) { + /* The entire predicate was false; no load occurs. */ + return; + } + mem_off = reg_off >> diffsz; -DO_LDNF1(bb_r) -DO_LDNF1(bhu_r) -DO_LDNF1(bhs_r) -DO_LDNF1(bsu_r) -DO_LDNF1(bss_r) -DO_LDNF1(bdu_r) -DO_LDNF1(bds_r) +#ifdef CONFIG_USER_ONLY + if (page_check_range(addr + mem_off, 1 << msz, PAGE_READ) == 0) { + /* At least one load is valid; take the rest of the page. */ + split = max_for_page(addr, mem_off + (1 << msz) - 1, mem_max); + mem_off = host_fn(vd, vg, host, mem_off, split); + reg_off = mem_off << diffsz; + } +#else + /* If the address is not in the TLB, we have no way to bring the + * entry into the TLB without also risking a fault. Note that + * the corollary is that we never load from an address not in RAM. + * ??? This last may be out of spec. + */ + host = tlb_vaddr_to_host(env, addr + mem_off, MMU_DATA_LOAD, mmu_idx); + split = max_for_page(addr, mem_off, mem_max); + if (host && split >= (1 << msz)) { + mem_off = host_fn(vd, vg, host - mem_off, mem_off, split); + reg_off = mem_off << diffsz; + } +#endif -DO_LDNF1(hh_r) -DO_LDNF1(hsu_r) -DO_LDNF1(hss_r) -DO_LDNF1(hdu_r) -DO_LDNF1(hds_r) + record_fault(env, reg_off, reg_max); +} -DO_LDNF1(ss_r) -DO_LDNF1(sdu_r) -DO_LDNF1(sds_r) +#define DO_LDFF1_LDNF1_1(PART, ESZ) \ +void HELPER(sve_ldff1##PART##_r)(CPUARMState *env, void *vg, \ + target_ulong addr, uint32_t desc) \ +{ \ + sve_ldff1_r(env, vg, addr, desc, GETPC(), ESZ, 0, \ + sve_ld1##PART##_host, sve_ld1##PART##_tlb); \ +} \ +void HELPER(sve_ldnf1##PART##_r)(CPUARMState *env, void *vg, \ + target_ulong addr, uint32_t desc) \ +{ \ + sve_ldnf1_r(env, vg, addr, desc, ESZ, 0, sve_ld1##PART##_host); \ +} -DO_LDNF1(dd_r) +/* TODO: Propagate the endian check back to the translator. */ +#define DO_LDFF1_LDNF1_2(PART, ESZ, MSZ) \ +void HELPER(sve_ldff1##PART##_r)(CPUARMState *env, void *vg, \ + target_ulong addr, uint32_t desc) \ +{ \ + if (arm_cpu_data_is_big_endian(env)) { \ + sve_ldff1_r(env, vg, addr, desc, GETPC(), ESZ, MSZ, \ + sve_ld1##PART##_be_host, sve_ld1##PART##_be_tlb); \ + } else { \ + sve_ldff1_r(env, vg, addr, desc, GETPC(), ESZ, MSZ, \ + sve_ld1##PART##_le_host, sve_ld1##PART##_le_tlb); \ + } \ +} \ +void HELPER(sve_ldnf1##PART##_r)(CPUARMState *env, void *vg, \ + target_ulong addr, uint32_t desc) \ +{ \ + if (arm_cpu_data_is_big_endian(env)) { \ + sve_ldnf1_r(env, vg, addr, desc, ESZ, MSZ, \ + sve_ld1##PART##_be_host); \ + } else { \ + sve_ldnf1_r(env, vg, addr, desc, ESZ, MSZ, \ + sve_ld1##PART##_le_host); \ + } \ +} -#undef DO_LDNF1 +DO_LDFF1_LDNF1_1(bb, 0) +DO_LDFF1_LDNF1_1(bhu, 1) +DO_LDFF1_LDNF1_1(bhs, 1) +DO_LDFF1_LDNF1_1(bsu, 2) +DO_LDFF1_LDNF1_1(bss, 2) +DO_LDFF1_LDNF1_1(bdu, 3) +DO_LDFF1_LDNF1_1(bds, 3) + +DO_LDFF1_LDNF1_2(hh, 1, 1) +DO_LDFF1_LDNF1_2(hsu, 2, 1) +DO_LDFF1_LDNF1_2(hss, 2, 1) +DO_LDFF1_LDNF1_2(hdu, 3, 1) +DO_LDFF1_LDNF1_2(hds, 3, 1) + +DO_LDFF1_LDNF1_2(ss, 2, 2) +DO_LDFF1_LDNF1_2(sdu, 3, 2) +DO_LDFF1_LDNF1_2(sds, 3, 2) + +DO_LDFF1_LDNF1_2(dd, 3, 3) + +#undef DO_LDFF1_LDNF1_1 +#undef DO_LDFF1_LDNF1_2 /* * Store contiguous data, protected by a governing predicate. From patchwork Thu Aug 9 04:21:59 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 143663 Delivered-To: patch@linaro.org Received: by 2002:a2e:9754:0:0:0:0:0 with SMTP id f20-v6csp1605945ljj; Wed, 8 Aug 2018 21:28:21 -0700 (PDT) X-Google-Smtp-Source: AA+uWPwl9odD60WV6ePuGl2XmfKWWhrK4w1FPzSWzJnAzo6Xwv9p7KF5NJrCVPXIu3mvRMEoRA8n X-Received: by 2002:a0c:f611:: with SMTP id r17-v6mr415109qvm.103.1533788900977; Wed, 08 Aug 2018 21:28:20 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1533788900; cv=none; d=google.com; s=arc-20160816; b=gbXF0H84xYEatEdy4La3wFyllPQrIUChCBtFFcumnvKkleGOXod149cax5pI0zWqhb ey5SEhCuPdCS3r6aGiQleYbb79qDrSC/m0o14WRm7078AyDbxXkR0SatT7HvYHbxi/kn GedtJLJH1I2wXuGUMYuqppxyi6YEZFf/G352a0aekfiyy+/3/gjTTNOBFtRX+9dGu4VH VyLx8cB5ubHrmrAHo8ZFHXavDiCa8kO8iZXkGFKNpbrUPM9Id+AqLVbmj4s1hrLKJCoU 4KriRgDFifEAtx9c05OUWOXOX2InXc2hnAr21E9Hly8mZ1FO3zJWem0S4Q7kRgD9KEpu Lgkg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:cc:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:subject:references:in-reply-to :message-id:date:to:from:dkim-signature:arc-authentication-results; bh=g6wk8E63hRi1UpPz695juSTD3aMNhCCM/dwAKrSVuhg=; b=DTwsGbIdh5jRgP45TTgF0/aCbifKHJAT+wYGZ76Uu+W3dIWpNq8+nUXcwKLpl5nOWC j4hkILiydP53QESlHyF0B5b4FYbnAvlXcP+bwbtge0BK/X923tS9MJVTPzD3o0cwD2z1 8iG96/9NZbiCZPHjKGaScct9DGPY5eq2reeYkyUcneY59uzAc8ygzJzG0aMaIcs1jqsf 3D7FgLEplAd+mtz5a7ZGtIrjgZG+JVWL+l8Hzhwr9Ld0mOgkFSLfEDsa38uOXUahY2D0 WW4n4n0B6Jtt2lzCdAaEBR49vR3Efsj20z/Fv3V70SPxZaH8qGdwpdG9ULH0G3f8eNyi ychw== ARC-Authentication-Results: i=1; mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=LiUBaX8s; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [2001:4830:134:3::11]) by mx.google.com with ESMTPS id u25-v6si1551709qva.197.2018.08.08.21.28.20 for (version=TLS1 cipher=AES128-SHA bits=128/128); Wed, 08 Aug 2018 21:28:20 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) client-ip=2001:4830:134:3::11; Authentication-Results: mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=LiUBaX8s; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1]:47252 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fncYe-00068i-93 for patch@linaro.org; Thu, 09 Aug 2018 00:28:20 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:54087) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fncT1-0000xw-KT for qemu-devel@nongnu.org; Thu, 09 Aug 2018 00:22:33 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fncSy-0007N7-BM for qemu-devel@nongnu.org; Thu, 09 Aug 2018 00:22:31 -0400 Received: from mail-pg1-x52f.google.com ([2607:f8b0:4864:20::52f]:41708) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fncSy-0007MW-28 for qemu-devel@nongnu.org; Thu, 09 Aug 2018 00:22:28 -0400 Received: by mail-pg1-x52f.google.com with SMTP id z8-v6so2107922pgu.8 for ; Wed, 08 Aug 2018 21:22:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=g6wk8E63hRi1UpPz695juSTD3aMNhCCM/dwAKrSVuhg=; b=LiUBaX8sarivR3S6lGs6Sw+1cTOa+LNHH6ROWn2C6nUiu6Y1gDJkNSkAc1nCQGvPzl /7NkCk7MwYrwKggw49pdo+KWsAM6SPJodo8WANgZEWQiizW4zHJZ3f1o+FIWlZ77DM02 3MfBEpxNRnVZwvCe6MLda8J3ZyXgHRHCW0U28= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=g6wk8E63hRi1UpPz695juSTD3aMNhCCM/dwAKrSVuhg=; b=CMxTzDpsg7wGmQ/WIzxdRY3vKoinfvyRgfPolwjg+VSG3zpPydbc9eDc6qGF8c9kTM bgiEvUq/SsQghFHq5WZ2wWnrThB9l3YNYNuCtTyTUSkTkHMyNYJQeFtqleD/fenKCi3D uS3QOo+YDg8rSXigeM9BOwXy32w+4tCwossXsX65Re3zbb+i5YmkLTFJmWYBgsMV61JV b+q4s6Z4qDwMmGAPqD3gAKrhA0yKQaXc1WiOKKn9sGPZzh3ly8u6A0BRdgWv/untjsbM lda7ltMQ1Eo3kq9COEylesbdKT/mqEVMbwQLwR5yHIbfXR0Sd6VkzQ98hr2fV+5Ejxr5 I8Jw== X-Gm-Message-State: AOUpUlHrxQgUMInO/uybAPydbXJj1eMa4boRdGiMNexg49pgKFHxSR3f mHJWwS7VRvxc+7JJ1Di5acks8GDwioE= X-Received: by 2002:a63:e14a:: with SMTP id h10-v6mr526936pgk.358.1533788546788; Wed, 08 Aug 2018 21:22:26 -0700 (PDT) Received: from cloudburst.twiddle.net (97-113-8-179.tukw.qwest.net. [97.113.8.179]) by smtp.gmail.com with ESMTPSA id m30-v6sm7355799pff.121.2018.08.08.21.22.25 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Wed, 08 Aug 2018 21:22:25 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Date: Wed, 8 Aug 2018 21:21:59 -0700 Message-Id: <20180809042206.15726-14-richard.henderson@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180809042206.15726-1-richard.henderson@linaro.org> References: <20180809042206.15726-1-richard.henderson@linaro.org> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::52f Subject: [Qemu-devel] [PATCH 13/20] target/arm: Rewrite helper_sve_ld[234]*_r X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: laurent.desnogues@gmail.com, peter.maydell@linaro.org, alex.bennee@linaro.org Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: "Qemu-devel" Use the same *_tlb primitives as we use for ld1. This is not a significant change, but does (for linux-user) hoist the set of helper_retaddr, and (for softmmu) hoist the computation of the current mmu_idx outside the loop. This does fix the endianness problem for softmmu, and does move the main loop out of a macro and into an inlined function. Signed-off-by: Richard Henderson --- target/arm/sve_helper.c | 210 ++++++++++++++++++++++------------------ 1 file changed, 117 insertions(+), 93 deletions(-) -- 2.17.1 Reviewed-by: Peter Maydell diff --git a/target/arm/sve_helper.c b/target/arm/sve_helper.c index 4ca9412e20..5cc7de5077 100644 --- a/target/arm/sve_helper.c +++ b/target/arm/sve_helper.c @@ -4398,109 +4398,133 @@ DO_LD1_2(ld1dd, 3, 3) #undef DO_LD1_1 #undef DO_LD1_2 -#define DO_LD2(NAME, FN, TYPEE, TYPEM, H) \ -void HELPER(NAME)(CPUARMState *env, void *vg, \ - target_ulong addr, uint32_t desc) \ -{ \ - intptr_t i, oprsz = simd_oprsz(desc); \ - intptr_t ra = GETPC(); \ - unsigned rd = simd_data(desc); \ - void *d1 = &env->vfp.zregs[rd]; \ - void *d2 = &env->vfp.zregs[(rd + 1) & 31]; \ - for (i = 0; i < oprsz; ) { \ - uint16_t pg = *(uint16_t *)(vg + H1_2(i >> 3)); \ - do { \ - TYPEM m1 = 0, m2 = 0; \ - if (pg & 1) { \ - m1 = FN(env, addr, ra); \ - m2 = FN(env, addr + sizeof(TYPEM), ra); \ - } \ - *(TYPEE *)(d1 + H(i)) = m1; \ - *(TYPEE *)(d2 + H(i)) = m2; \ - i += sizeof(TYPEE), pg >>= sizeof(TYPEE); \ - addr += 2 * sizeof(TYPEM); \ - } while (i & 15); \ - } \ +/* + * Common helpers for all contiguous 2,3,4-register predicated loads. + */ +static void sve_ld2_r(CPUARMState *env, void *vg, target_ulong addr, + uint32_t desc, int size, uintptr_t ra, + sve_ld1_tlb_fn *tlb_fn) +{ + const int mmu_idx = cpu_mmu_index(env, false); + intptr_t i, oprsz = simd_oprsz(desc); + unsigned rd = simd_data(desc); + ARMVectorReg scratch[2] = { }; + + set_helper_retaddr(ra); + for (i = 0; i < oprsz; ) { + uint16_t pg = *(uint16_t *)(vg + H1_2(i >> 3)); + do { + if (pg & 1) { + tlb_fn(env, &scratch[0], i, addr, mmu_idx, ra); + tlb_fn(env, &scratch[1], i, addr + size, mmu_idx, ra); + } + i += size, pg >>= size; + addr += 2 * size; + } while (i & 15); + } + set_helper_retaddr(0); + + /* Wait until all exceptions have been raised to write back. */ + memcpy(&env->vfp.zregs[rd], &scratch[0], oprsz); + memcpy(&env->vfp.zregs[(rd + 1) & 31], &scratch[1], oprsz); } -#define DO_LD3(NAME, FN, TYPEE, TYPEM, H) \ -void HELPER(NAME)(CPUARMState *env, void *vg, \ - target_ulong addr, uint32_t desc) \ -{ \ - intptr_t i, oprsz = simd_oprsz(desc); \ - intptr_t ra = GETPC(); \ - unsigned rd = simd_data(desc); \ - void *d1 = &env->vfp.zregs[rd]; \ - void *d2 = &env->vfp.zregs[(rd + 1) & 31]; \ - void *d3 = &env->vfp.zregs[(rd + 2) & 31]; \ - for (i = 0; i < oprsz; ) { \ - uint16_t pg = *(uint16_t *)(vg + H1_2(i >> 3)); \ - do { \ - TYPEM m1 = 0, m2 = 0, m3 = 0; \ - if (pg & 1) { \ - m1 = FN(env, addr, ra); \ - m2 = FN(env, addr + sizeof(TYPEM), ra); \ - m3 = FN(env, addr + 2 * sizeof(TYPEM), ra); \ - } \ - *(TYPEE *)(d1 + H(i)) = m1; \ - *(TYPEE *)(d2 + H(i)) = m2; \ - *(TYPEE *)(d3 + H(i)) = m3; \ - i += sizeof(TYPEE), pg >>= sizeof(TYPEE); \ - addr += 3 * sizeof(TYPEM); \ - } while (i & 15); \ - } \ +static void sve_ld3_r(CPUARMState *env, void *vg, target_ulong addr, + uint32_t desc, int size, uintptr_t ra, + sve_ld1_tlb_fn *tlb_fn) +{ + const int mmu_idx = cpu_mmu_index(env, false); + intptr_t i, oprsz = simd_oprsz(desc); + unsigned rd = simd_data(desc); + ARMVectorReg scratch[3] = { }; + + set_helper_retaddr(ra); + for (i = 0; i < oprsz; ) { + uint16_t pg = *(uint16_t *)(vg + H1_2(i >> 3)); + do { + if (pg & 1) { + tlb_fn(env, &scratch[0], i, addr, mmu_idx, ra); + tlb_fn(env, &scratch[1], i, addr + size, mmu_idx, ra); + tlb_fn(env, &scratch[2], i, addr + 2 * size, mmu_idx, ra); + } + i += size, pg >>= size; + addr += 3 * size; + } while (i & 15); + } + set_helper_retaddr(0); + + /* Wait until all exceptions have been raised to write back. */ + memcpy(&env->vfp.zregs[rd], &scratch[0], oprsz); + memcpy(&env->vfp.zregs[(rd + 1) & 31], &scratch[1], oprsz); + memcpy(&env->vfp.zregs[(rd + 2) & 31], &scratch[2], oprsz); } -#define DO_LD4(NAME, FN, TYPEE, TYPEM, H) \ -void HELPER(NAME)(CPUARMState *env, void *vg, \ - target_ulong addr, uint32_t desc) \ -{ \ - intptr_t i, oprsz = simd_oprsz(desc); \ - intptr_t ra = GETPC(); \ - unsigned rd = simd_data(desc); \ - void *d1 = &env->vfp.zregs[rd]; \ - void *d2 = &env->vfp.zregs[(rd + 1) & 31]; \ - void *d3 = &env->vfp.zregs[(rd + 2) & 31]; \ - void *d4 = &env->vfp.zregs[(rd + 3) & 31]; \ - for (i = 0; i < oprsz; ) { \ - uint16_t pg = *(uint16_t *)(vg + H1_2(i >> 3)); \ - do { \ - TYPEM m1 = 0, m2 = 0, m3 = 0, m4 = 0; \ - if (pg & 1) { \ - m1 = FN(env, addr, ra); \ - m2 = FN(env, addr + sizeof(TYPEM), ra); \ - m3 = FN(env, addr + 2 * sizeof(TYPEM), ra); \ - m4 = FN(env, addr + 3 * sizeof(TYPEM), ra); \ - } \ - *(TYPEE *)(d1 + H(i)) = m1; \ - *(TYPEE *)(d2 + H(i)) = m2; \ - *(TYPEE *)(d3 + H(i)) = m3; \ - *(TYPEE *)(d4 + H(i)) = m4; \ - i += sizeof(TYPEE), pg >>= sizeof(TYPEE); \ - addr += 4 * sizeof(TYPEM); \ - } while (i & 15); \ - } \ +static void sve_ld4_r(CPUARMState *env, void *vg, target_ulong addr, + uint32_t desc, int size, uintptr_t ra, + sve_ld1_tlb_fn *tlb_fn) +{ + const int mmu_idx = cpu_mmu_index(env, false); + intptr_t i, oprsz = simd_oprsz(desc); + unsigned rd = simd_data(desc); + ARMVectorReg scratch[4] = { }; + + set_helper_retaddr(ra); + for (i = 0; i < oprsz; ) { + uint16_t pg = *(uint16_t *)(vg + H1_2(i >> 3)); + do { + if (pg & 1) { + tlb_fn(env, &scratch[0], i, addr, mmu_idx, ra); + tlb_fn(env, &scratch[1], i, addr + size, mmu_idx, ra); + tlb_fn(env, &scratch[2], i, addr + 2 * size, mmu_idx, ra); + tlb_fn(env, &scratch[3], i, addr + 3 * size, mmu_idx, ra); + } + i += size, pg >>= size; + addr += 4 * size; + } while (i & 15); + } + set_helper_retaddr(0); + + /* Wait until all exceptions have been raised to write back. */ + memcpy(&env->vfp.zregs[rd], &scratch[0], oprsz); + memcpy(&env->vfp.zregs[(rd + 1) & 31], &scratch[1], oprsz); + memcpy(&env->vfp.zregs[(rd + 2) & 31], &scratch[2], oprsz); + memcpy(&env->vfp.zregs[(rd + 3) & 31], &scratch[3], oprsz); } -DO_LD2(sve_ld2bb_r, cpu_ldub_data_ra, uint8_t, uint8_t, H1) -DO_LD3(sve_ld3bb_r, cpu_ldub_data_ra, uint8_t, uint8_t, H1) -DO_LD4(sve_ld4bb_r, cpu_ldub_data_ra, uint8_t, uint8_t, H1) +#define DO_LDN_1(N) \ +void __attribute__((flatten)) HELPER(sve_ld##N##bb_r) \ + (CPUARMState *env, void *vg, target_ulong addr, uint32_t desc) \ +{ \ + sve_ld##N##_r(env, vg, addr, desc, 1, GETPC(), sve_ld1bb_tlb); \ +} -DO_LD2(sve_ld2hh_r, cpu_lduw_data_ra, uint16_t, uint16_t, H1_2) -DO_LD3(sve_ld3hh_r, cpu_lduw_data_ra, uint16_t, uint16_t, H1_2) -DO_LD4(sve_ld4hh_r, cpu_lduw_data_ra, uint16_t, uint16_t, H1_2) +#define DO_LDN_2(N, SUFF, SIZE) \ +void __attribute__((flatten)) HELPER(sve_ld##N##SUFF##_r) \ + (CPUARMState *env, void *vg, target_ulong addr, uint32_t desc) \ +{ \ + sve_ld##N##_r(env, vg, addr, desc, SIZE, GETPC(), \ + arm_cpu_data_is_big_endian(env) \ + ? sve_ld1##SUFF##_be_tlb : sve_ld1##SUFF##_le_tlb); \ +} -DO_LD2(sve_ld2ss_r, cpu_ldl_data_ra, uint32_t, uint32_t, H1_4) -DO_LD3(sve_ld3ss_r, cpu_ldl_data_ra, uint32_t, uint32_t, H1_4) -DO_LD4(sve_ld4ss_r, cpu_ldl_data_ra, uint32_t, uint32_t, H1_4) +DO_LDN_1(2) +DO_LDN_1(3) +DO_LDN_1(4) -DO_LD2(sve_ld2dd_r, cpu_ldq_data_ra, uint64_t, uint64_t, ) -DO_LD3(sve_ld3dd_r, cpu_ldq_data_ra, uint64_t, uint64_t, ) -DO_LD4(sve_ld4dd_r, cpu_ldq_data_ra, uint64_t, uint64_t, ) +DO_LDN_2(2, hh, 2) +DO_LDN_2(3, hh, 2) +DO_LDN_2(4, hh, 2) -#undef DO_LD2 -#undef DO_LD3 -#undef DO_LD4 +DO_LDN_2(2, ss, 4) +DO_LDN_2(3, ss, 4) +DO_LDN_2(4, ss, 4) + +DO_LDN_2(2, dd, 8) +DO_LDN_2(3, dd, 8) +DO_LDN_2(4, dd, 8) + +#undef DO_LDN_1 +#undef DO_LDN_2 /* * Load contiguous data, first-fault and no-fault. From patchwork Thu Aug 9 04:22:00 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 143669 Delivered-To: patch@linaro.org Received: by 2002:a2e:9754:0:0:0:0:0 with SMTP id f20-v6csp1608917ljj; Wed, 8 Aug 2018 21:32:15 -0700 (PDT) X-Google-Smtp-Source: AA+uWPxMl4f6zjqKJiA58+kbQZaa4SPi6MN3WN5lGibNdIR04F+5kBuMNOUCmPb+/7TOIFksQRsW X-Received: by 2002:ac8:5042:: with SMTP id h2-v6mr475762qtm.71.1533789135504; Wed, 08 Aug 2018 21:32:15 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1533789135; cv=none; d=google.com; s=arc-20160816; b=D4DDxpYounM5Cjfb9JLjITryM1GqlBCWQIBxLdAU02VHvYUpIEQoaa2thBJGlWqZHi 86MrTSkK5/VeBedgJIibtsTwLFQee86ZtJ9W6dDw/bpHZ+FR16ZU5wUwxXSMfaL7fMFJ gWooXbO8xflbKwOjmpWZYa56U5nsCfpKBfi8yF7dnIzTkn0BoATgt0Eu0EqsR3t8oIpH ZAtYcch3yxEDB/BE1zV8ysnV1l2lNe5M+tT/hxMLEAAcH6aAi0PcE68uDIMLIIE81pko 9P/oK2mmcG/Jwzb7k0vIB+QIyaxPWDjCznqneFAmco85sCHrL6ZPgtFZJymF/t0STjhx rJ3Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:cc:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:subject:references:in-reply-to :message-id:date:to:from:dkim-signature:arc-authentication-results; bh=5u7PSIK4SfdKyW2wEuMRslvK2inqMUFtXHvO/DU45T8=; b=f4nshCHTyLcTgox/oQhk1R1z+FiTW3kBJz0bjj/mYixlnzXQhKwQ3zO77rXVaFgWbv 3Dik2isTdrNYjLBO2mF0KouP9mcGv4csA7ww08gvFVz+fp1qUSVPI5HLBQDQmwVs6yWu KdCh2PHapQhckCa1lhxbxGHbTajjoHRS2xvXimqv1MOkJpbcMLjiinUFsXp6JdAqZXbC hnLPr6F8Kyn1CrPANti3lG41gjkNFqz3o472sJci0rX36LEk73vTCjkJs6Cfn5bHBHOX vpBmtRWxUqKB5k0ZNVXxZqP0vvK/Rv28KfGYfRCzSSQEy5XnPqwU2lhKa7ZKi74YoQYR /tzg== ARC-Authentication-Results: i=1; mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=SY0le7pN; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [2001:4830:134:3::11]) by mx.google.com with ESMTPS id f59-v6si4874442qva.48.2018.08.08.21.32.15 for (version=TLS1 cipher=AES128-SHA bits=128/128); Wed, 08 Aug 2018 21:32:15 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) client-ip=2001:4830:134:3::11; Authentication-Results: mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=SY0le7pN; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1]:47276 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fnccQ-0001ek-Tp for patch@linaro.org; Thu, 09 Aug 2018 00:32:14 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:54089) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fncT1-0000xx-Rg for qemu-devel@nongnu.org; Thu, 09 Aug 2018 00:22:35 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fncSz-0007OS-W1 for qemu-devel@nongnu.org; Thu, 09 Aug 2018 00:22:31 -0400 Received: from mail-pg1-x535.google.com ([2607:f8b0:4864:20::535]:35957) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fncSz-0007Nq-L7 for qemu-devel@nongnu.org; Thu, 09 Aug 2018 00:22:29 -0400 Received: by mail-pg1-x535.google.com with SMTP id h12-v6so2119569pgs.3 for ; Wed, 08 Aug 2018 21:22:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=5u7PSIK4SfdKyW2wEuMRslvK2inqMUFtXHvO/DU45T8=; b=SY0le7pNhqOJcPPSLUr+SSwvM5pKitfwNsJIenjPejsn4bMsglc681agj8cDToQ7vt 6J5UH4nxxJ8VEFCnSL2EBTiYjEGM/QJuQk7KdQl1/Yqev2zoXHzSfOaW1PjuH2JkzQsJ ONmV3D0XljtGs8RGkG5RVa5MZ1fE8WB9gO188= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=5u7PSIK4SfdKyW2wEuMRslvK2inqMUFtXHvO/DU45T8=; b=hpp6Vpwq/fhOzohQTFjGZQByfY39/UdGmL0T/FbDxoxMetr0BGuPSgevxeT/Isc7g1 9yXKsqewJWunYXJqCkRuxcUTTVsDJmO77a37HFVfKQJ8+BvL70lhioxM6OJqqcw7wMsA pAAadJmqWrjk3UQM39fR+t1xgc1FKNXw1iYIBrIXVtVz+0ejQXT6uxSkU/0GbEbmvhgf 02l2pPX3yCFWeDbE+TAUjmlqBtQdVS/eBIfU5YmKqj01xPGhCLv5LuSJr9zZUAB0YZP+ O1Aeg943MCwqMfUpw+5YBkQeA3g4syybcdBWChRJyOmEyAGgzHM7GIuUTVJ/Exp1TC9o NK/Q== X-Gm-Message-State: AOUpUlGa4QfT5l94o+/BQWyXNlMFu+zIFrpVOrbGihYxGN2VNnKEWLH/ 24VlyizWnx1k7iTFaTdRS9ZY562dihc= X-Received: by 2002:a63:1e08:: with SMTP id e8-v6mr483431pge.281.1533788548190; Wed, 08 Aug 2018 21:22:28 -0700 (PDT) Received: from cloudburst.twiddle.net (97-113-8-179.tukw.qwest.net. [97.113.8.179]) by smtp.gmail.com with ESMTPSA id m30-v6sm7355799pff.121.2018.08.08.21.22.26 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Wed, 08 Aug 2018 21:22:27 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Date: Wed, 8 Aug 2018 21:22:00 -0700 Message-Id: <20180809042206.15726-15-richard.henderson@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180809042206.15726-1-richard.henderson@linaro.org> References: <20180809042206.15726-1-richard.henderson@linaro.org> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::535 Subject: [Qemu-devel] [PATCH 14/20] target/arm: Rewrite helper_sve_st[1234]*_r X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: laurent.desnogues@gmail.com, peter.maydell@linaro.org, alex.bennee@linaro.org Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: "Qemu-devel" This fixes the endianness problem for softmmu, and does move the main loop out of a macro and into an inlined function. Signed-off-by: Richard Henderson --- target/arm/sve_helper.c | 351 ++++++++++++++++++++-------------------- 1 file changed, 172 insertions(+), 179 deletions(-) -- 2.17.1 Reviewed-by: Peter Maydell diff --git a/target/arm/sve_helper.c b/target/arm/sve_helper.c index 5cc7de5077..4eae6569cc 100644 --- a/target/arm/sve_helper.c +++ b/target/arm/sve_helper.c @@ -3987,6 +3987,7 @@ typedef intptr_t sve_ld1_host_fn(void *vd, void *vg, void *host, */ typedef void sve_ld1_tlb_fn(CPUARMState *env, void *vd, intptr_t reg_off, target_ulong vaddr, int mmu_idx, uintptr_t ra); +typedef sve_ld1_tlb_fn sve_st1_tlb_fn; /* * Generate the above primitives. @@ -4772,214 +4773,206 @@ DO_LDFF1_LDNF1_2(dd, 3, 3) /* * Store contiguous data, protected by a governing predicate. */ -#define DO_ST1(NAME, FN, TYPEE, TYPEM, H) \ -void HELPER(NAME)(CPUARMState *env, void *vg, \ - target_ulong addr, uint32_t desc) \ -{ \ - intptr_t i, oprsz = simd_oprsz(desc); \ - intptr_t ra = GETPC(); \ - unsigned rd = simd_data(desc); \ - void *vd = &env->vfp.zregs[rd]; \ - for (i = 0; i < oprsz; ) { \ - uint16_t pg = *(uint16_t *)(vg + H1_2(i >> 3)); \ - do { \ - if (pg & 1) { \ - TYPEM m = *(TYPEE *)(vd + H(i)); \ - FN(env, addr, m, ra); \ - } \ - i += sizeof(TYPEE), pg >>= sizeof(TYPEE); \ - addr += sizeof(TYPEM); \ - } while (i & 15); \ - } \ + +#ifdef CONFIG_SOFTMMU +#define DO_ST_TLB(NAME, H, TYPEM, HOST, MOEND, TLB) \ +static void sve_##NAME##_tlb(CPUARMState *env, void *vd, intptr_t reg_off, \ + target_ulong addr, int mmu_idx, uintptr_t ra) \ +{ \ + TCGMemOpIdx oi = make_memop_idx(ctz32(sizeof(TYPEM)) | MOEND, mmu_idx); \ + TLB(env, addr, *(TYPEM *)(vd + H(reg_off)), oi, ra); \ } - -#define DO_ST1_D(NAME, FN, TYPEM) \ -void HELPER(NAME)(CPUARMState *env, void *vg, \ - target_ulong addr, uint32_t desc) \ -{ \ - intptr_t i, oprsz = simd_oprsz(desc) / 8; \ - intptr_t ra = GETPC(); \ - unsigned rd = simd_data(desc); \ - uint64_t *d = &env->vfp.zregs[rd].d[0]; \ - uint8_t *pg = vg; \ - for (i = 0; i < oprsz; i += 1) { \ - if (pg[H1(i)] & 1) { \ - FN(env, addr, d[i], ra); \ - } \ - addr += sizeof(TYPEM); \ - } \ +#else +#define DO_ST_TLB(NAME, H, TYPEM, HOST, MOEND, TLB) \ +static void sve_##NAME##_tlb(CPUARMState *env, void *vd, intptr_t reg_off, \ + target_ulong addr, int mmu_idx, uintptr_t ra) \ +{ \ + HOST(g2h(addr), *(TYPEM *)(vd + H(reg_off))); \ } +#endif -#define DO_ST2(NAME, FN, TYPEE, TYPEM, H) \ -void HELPER(NAME)(CPUARMState *env, void *vg, \ - target_ulong addr, uint32_t desc) \ -{ \ - intptr_t i, oprsz = simd_oprsz(desc); \ - intptr_t ra = GETPC(); \ - unsigned rd = simd_data(desc); \ - void *d1 = &env->vfp.zregs[rd]; \ - void *d2 = &env->vfp.zregs[(rd + 1) & 31]; \ - for (i = 0; i < oprsz; ) { \ - uint16_t pg = *(uint16_t *)(vg + H1_2(i >> 3)); \ - do { \ - if (pg & 1) { \ - TYPEM m1 = *(TYPEE *)(d1 + H(i)); \ - TYPEM m2 = *(TYPEE *)(d2 + H(i)); \ - FN(env, addr, m1, ra); \ - FN(env, addr + sizeof(TYPEM), m2, ra); \ - } \ - i += sizeof(TYPEE), pg >>= sizeof(TYPEE); \ - addr += 2 * sizeof(TYPEM); \ - } while (i & 15); \ - } \ -} +DO_ST_TLB(st1bb, H1, uint8_t, stb_p, 0, helper_ret_stb_mmu) +DO_ST_TLB(st1bh, H1_2, uint16_t, stb_p, 0, helper_ret_stb_mmu) +DO_ST_TLB(st1bs, H1_4, uint32_t, stb_p, 0, helper_ret_stb_mmu) +DO_ST_TLB(st1bd, , uint64_t, stb_p, 0, helper_ret_stb_mmu) -#define DO_ST3(NAME, FN, TYPEE, TYPEM, H) \ -void HELPER(NAME)(CPUARMState *env, void *vg, \ - target_ulong addr, uint32_t desc) \ -{ \ - intptr_t i, oprsz = simd_oprsz(desc); \ - intptr_t ra = GETPC(); \ - unsigned rd = simd_data(desc); \ - void *d1 = &env->vfp.zregs[rd]; \ - void *d2 = &env->vfp.zregs[(rd + 1) & 31]; \ - void *d3 = &env->vfp.zregs[(rd + 2) & 31]; \ - for (i = 0; i < oprsz; ) { \ - uint16_t pg = *(uint16_t *)(vg + H1_2(i >> 3)); \ - do { \ - if (pg & 1) { \ - TYPEM m1 = *(TYPEE *)(d1 + H(i)); \ - TYPEM m2 = *(TYPEE *)(d2 + H(i)); \ - TYPEM m3 = *(TYPEE *)(d3 + H(i)); \ - FN(env, addr, m1, ra); \ - FN(env, addr + sizeof(TYPEM), m2, ra); \ - FN(env, addr + 2 * sizeof(TYPEM), m3, ra); \ - } \ - i += sizeof(TYPEE), pg >>= sizeof(TYPEE); \ - addr += 3 * sizeof(TYPEM); \ - } while (i & 15); \ - } \ -} +DO_ST_TLB(st1hh_le, H1_2, uint16_t, stw_le_p, MO_LE, helper_le_stw_mmu) +DO_ST_TLB(st1hs_le, H1_4, uint32_t, stw_le_p, MO_LE, helper_le_stw_mmu) +DO_ST_TLB(st1hd_le, , uint64_t, stw_le_p, MO_LE, helper_le_stw_mmu) -#define DO_ST4(NAME, FN, TYPEE, TYPEM, H) \ -void HELPER(NAME)(CPUARMState *env, void *vg, \ - target_ulong addr, uint32_t desc) \ -{ \ - intptr_t i, oprsz = simd_oprsz(desc); \ - intptr_t ra = GETPC(); \ - unsigned rd = simd_data(desc); \ - void *d1 = &env->vfp.zregs[rd]; \ - void *d2 = &env->vfp.zregs[(rd + 1) & 31]; \ - void *d3 = &env->vfp.zregs[(rd + 2) & 31]; \ - void *d4 = &env->vfp.zregs[(rd + 3) & 31]; \ - for (i = 0; i < oprsz; ) { \ - uint16_t pg = *(uint16_t *)(vg + H1_2(i >> 3)); \ - do { \ - if (pg & 1) { \ - TYPEM m1 = *(TYPEE *)(d1 + H(i)); \ - TYPEM m2 = *(TYPEE *)(d2 + H(i)); \ - TYPEM m3 = *(TYPEE *)(d3 + H(i)); \ - TYPEM m4 = *(TYPEE *)(d4 + H(i)); \ - FN(env, addr, m1, ra); \ - FN(env, addr + sizeof(TYPEM), m2, ra); \ - FN(env, addr + 2 * sizeof(TYPEM), m3, ra); \ - FN(env, addr + 3 * sizeof(TYPEM), m4, ra); \ - } \ - i += sizeof(TYPEE), pg >>= sizeof(TYPEE); \ - addr += 4 * sizeof(TYPEM); \ - } while (i & 15); \ - } \ -} +DO_ST_TLB(st1ss_le, H1_4, uint32_t, stl_le_p, MO_LE, helper_le_stl_mmu) +DO_ST_TLB(st1sd_le, , uint64_t, stl_le_p, MO_LE, helper_le_stl_mmu) -DO_ST1(sve_st1bh_r, cpu_stb_data_ra, uint16_t, uint8_t, H1_2) -DO_ST1(sve_st1bs_r, cpu_stb_data_ra, uint32_t, uint8_t, H1_4) -DO_ST1_D(sve_st1bd_r, cpu_stb_data_ra, uint8_t) +DO_ST_TLB(st1dd_le, , uint64_t, stq_le_p, MO_LE, helper_le_stq_mmu) -DO_ST1(sve_st1hs_r, cpu_stw_data_ra, uint32_t, uint16_t, H1_4) -DO_ST1_D(sve_st1hd_r, cpu_stw_data_ra, uint16_t) +DO_ST_TLB(st1hh_be, H1_2, uint16_t, stw_be_p, MO_BE, helper_be_stw_mmu) +DO_ST_TLB(st1hs_be, H1_4, uint32_t, stw_be_p, MO_BE, helper_be_stw_mmu) +DO_ST_TLB(st1hd_be, , uint64_t, stw_be_p, MO_BE, helper_be_stw_mmu) -DO_ST1_D(sve_st1sd_r, cpu_stl_data_ra, uint32_t) +DO_ST_TLB(st1ss_be, H1_4, uint32_t, stl_be_p, MO_BE, helper_be_stl_mmu) +DO_ST_TLB(st1sd_be, , uint64_t, stl_be_p, MO_BE, helper_be_stl_mmu) -DO_ST1(sve_st1bb_r, cpu_stb_data_ra, uint8_t, uint8_t, H1) -DO_ST2(sve_st2bb_r, cpu_stb_data_ra, uint8_t, uint8_t, H1) -DO_ST3(sve_st3bb_r, cpu_stb_data_ra, uint8_t, uint8_t, H1) -DO_ST4(sve_st4bb_r, cpu_stb_data_ra, uint8_t, uint8_t, H1) +DO_ST_TLB(st1dd_be, , uint64_t, stq_be_p, MO_BE, helper_be_stq_mmu) -DO_ST1(sve_st1hh_r, cpu_stw_data_ra, uint16_t, uint16_t, H1_2) -DO_ST2(sve_st2hh_r, cpu_stw_data_ra, uint16_t, uint16_t, H1_2) -DO_ST3(sve_st3hh_r, cpu_stw_data_ra, uint16_t, uint16_t, H1_2) -DO_ST4(sve_st4hh_r, cpu_stw_data_ra, uint16_t, uint16_t, H1_2) +#undef DO_ST_TLB -DO_ST1(sve_st1ss_r, cpu_stl_data_ra, uint32_t, uint32_t, H1_4) -DO_ST2(sve_st2ss_r, cpu_stl_data_ra, uint32_t, uint32_t, H1_4) -DO_ST3(sve_st3ss_r, cpu_stl_data_ra, uint32_t, uint32_t, H1_4) -DO_ST4(sve_st4ss_r, cpu_stl_data_ra, uint32_t, uint32_t, H1_4) - -DO_ST1_D(sve_st1dd_r, cpu_stq_data_ra, uint64_t) - -void HELPER(sve_st2dd_r)(CPUARMState *env, void *vg, - target_ulong addr, uint32_t desc) +/* + * Common helpers for all contiguous 1,2,3,4-register predicated stores. + */ +static void sve_st1_r(CPUARMState *env, void *vg, target_ulong addr, + uint32_t desc, const uintptr_t ra, + const int esize, const int msize, + sve_st1_tlb_fn *tlb_fn) { - intptr_t i, oprsz = simd_oprsz(desc) / 8; - intptr_t ra = GETPC(); + const int mmu_idx = cpu_mmu_index(env, false); + intptr_t i, oprsz = simd_oprsz(desc); unsigned rd = simd_data(desc); - uint64_t *d1 = &env->vfp.zregs[rd].d[0]; - uint64_t *d2 = &env->vfp.zregs[(rd + 1) & 31].d[0]; - uint8_t *pg = vg; + void *vd = &env->vfp.zregs[rd]; - for (i = 0; i < oprsz; i += 1) { - if (pg[H1(i)] & 1) { - cpu_stq_data_ra(env, addr, d1[i], ra); - cpu_stq_data_ra(env, addr + 8, d2[i], ra); - } - addr += 2 * 8; + set_helper_retaddr(ra); + for (i = 0; i < oprsz; ) { + uint16_t pg = *(uint16_t *)(vg + H1_2(i >> 3)); + do { + if (pg & 1) { + tlb_fn(env, vd, i, addr, mmu_idx, ra); + } + i += esize, pg >>= esize; + addr += msize; + } while (i & 15); } + set_helper_retaddr(0); } -void HELPER(sve_st3dd_r)(CPUARMState *env, void *vg, - target_ulong addr, uint32_t desc) +static void sve_st2_r(CPUARMState *env, void *vg, target_ulong addr, + uint32_t desc, const uintptr_t ra, + const int esize, const int msize, + sve_st1_tlb_fn *tlb_fn) { - intptr_t i, oprsz = simd_oprsz(desc) / 8; - intptr_t ra = GETPC(); + const int mmu_idx = cpu_mmu_index(env, false); + intptr_t i, oprsz = simd_oprsz(desc); unsigned rd = simd_data(desc); - uint64_t *d1 = &env->vfp.zregs[rd].d[0]; - uint64_t *d2 = &env->vfp.zregs[(rd + 1) & 31].d[0]; - uint64_t *d3 = &env->vfp.zregs[(rd + 2) & 31].d[0]; - uint8_t *pg = vg; + void *d1 = &env->vfp.zregs[rd]; + void *d2 = &env->vfp.zregs[(rd + 1) & 31]; - for (i = 0; i < oprsz; i += 1) { - if (pg[H1(i)] & 1) { - cpu_stq_data_ra(env, addr, d1[i], ra); - cpu_stq_data_ra(env, addr + 8, d2[i], ra); - cpu_stq_data_ra(env, addr + 16, d3[i], ra); - } - addr += 3 * 8; + set_helper_retaddr(ra); + for (i = 0; i < oprsz; ) { + uint16_t pg = *(uint16_t *)(vg + H1_2(i >> 3)); + do { + if (pg & 1) { + tlb_fn(env, d1, i, addr, mmu_idx, ra); + tlb_fn(env, d2, i, addr + msize, mmu_idx, ra); + } + i += esize, pg >>= esize; + addr += 2 * msize; + } while (i & 15); } + set_helper_retaddr(0); } -void HELPER(sve_st4dd_r)(CPUARMState *env, void *vg, - target_ulong addr, uint32_t desc) +static void sve_st3_r(CPUARMState *env, void *vg, target_ulong addr, + uint32_t desc, const uintptr_t ra, + const int esize, const int msize, + sve_st1_tlb_fn *tlb_fn) { - intptr_t i, oprsz = simd_oprsz(desc) / 8; - intptr_t ra = GETPC(); + const int mmu_idx = cpu_mmu_index(env, false); + intptr_t i, oprsz = simd_oprsz(desc); unsigned rd = simd_data(desc); - uint64_t *d1 = &env->vfp.zregs[rd].d[0]; - uint64_t *d2 = &env->vfp.zregs[(rd + 1) & 31].d[0]; - uint64_t *d3 = &env->vfp.zregs[(rd + 2) & 31].d[0]; - uint64_t *d4 = &env->vfp.zregs[(rd + 3) & 31].d[0]; - uint8_t *pg = vg; + void *d1 = &env->vfp.zregs[rd]; + void *d2 = &env->vfp.zregs[(rd + 1) & 31]; + void *d3 = &env->vfp.zregs[(rd + 2) & 31]; - for (i = 0; i < oprsz; i += 1) { - if (pg[H1(i)] & 1) { - cpu_stq_data_ra(env, addr, d1[i], ra); - cpu_stq_data_ra(env, addr + 8, d2[i], ra); - cpu_stq_data_ra(env, addr + 16, d3[i], ra); - cpu_stq_data_ra(env, addr + 24, d4[i], ra); - } - addr += 4 * 8; + set_helper_retaddr(ra); + for (i = 0; i < oprsz; ) { + uint16_t pg = *(uint16_t *)(vg + H1_2(i >> 3)); + do { + if (pg & 1) { + tlb_fn(env, d1, i, addr, mmu_idx, ra); + tlb_fn(env, d2, i, addr + msize, mmu_idx, ra); + tlb_fn(env, d3, i, addr + 2 * msize, mmu_idx, ra); + } + i += esize, pg >>= esize; + addr += 3 * msize; + } while (i & 15); } + set_helper_retaddr(0); } +static void sve_st4_r(CPUARMState *env, void *vg, target_ulong addr, + uint32_t desc, const uintptr_t ra, + const int esize, const int msize, + sve_st1_tlb_fn *tlb_fn) +{ + const int mmu_idx = cpu_mmu_index(env, false); + intptr_t i, oprsz = simd_oprsz(desc); + unsigned rd = simd_data(desc); + void *d1 = &env->vfp.zregs[rd]; + void *d2 = &env->vfp.zregs[(rd + 1) & 31]; + void *d3 = &env->vfp.zregs[(rd + 2) & 31]; + void *d4 = &env->vfp.zregs[(rd + 3) & 31]; + + set_helper_retaddr(ra); + for (i = 0; i < oprsz; ) { + uint16_t pg = *(uint16_t *)(vg + H1_2(i >> 3)); + do { + if (pg & 1) { + tlb_fn(env, d1, i, addr, mmu_idx, ra); + tlb_fn(env, d2, i, addr + msize, mmu_idx, ra); + tlb_fn(env, d3, i, addr + 2 * msize, mmu_idx, ra); + tlb_fn(env, d4, i, addr + 3 * msize, mmu_idx, ra); + } + i += esize, pg >>= esize; + addr += 4 * msize; + } while (i & 15); + } + set_helper_retaddr(0); +} + +#define DO_STN_1(N, NAME, ESIZE) \ +void __attribute__((flatten)) HELPER(sve_st##N##NAME##_r) \ + (CPUARMState *env, void *vg, target_ulong addr, uint32_t desc) \ +{ \ + sve_st##N##_r(env, vg, addr, desc, GETPC(), ESIZE, 1, \ + sve_st1##NAME##_tlb); \ +} + +#define DO_STN_2(N, NAME, ESIZE, MSIZE) \ +void __attribute__((flatten)) HELPER(sve_st##N##NAME##_r) \ + (CPUARMState *env, void *vg, target_ulong addr, uint32_t desc) \ +{ \ + sve_st##N##_r(env, vg, addr, desc, GETPC(), ESIZE, MSIZE, \ + arm_cpu_data_is_big_endian(env) \ + ? sve_st1##NAME##_be_tlb : sve_st1##NAME##_le_tlb); \ +} + +DO_STN_1(1, bb, 1) +DO_STN_1(1, bh, 2) +DO_STN_1(1, bs, 4) +DO_STN_1(1, bd, 8) +DO_STN_1(2, bb, 1) +DO_STN_1(3, bb, 1) +DO_STN_1(4, bb, 1) + +DO_STN_2(1, hh, 2, 2) +DO_STN_2(1, hs, 4, 2) +DO_STN_2(1, hd, 8, 2) +DO_STN_2(2, hh, 2, 2) +DO_STN_2(3, hh, 2, 2) +DO_STN_2(4, hh, 2, 2) + +DO_STN_2(1, ss, 4, 4) +DO_STN_2(1, sd, 8, 4) +DO_STN_2(2, ss, 4, 4) +DO_STN_2(3, ss, 4, 4) +DO_STN_2(4, ss, 4, 4) + +DO_STN_2(1, dd, 8, 8) +DO_STN_2(2, dd, 8, 8) +DO_STN_2(3, dd, 8, 8) +DO_STN_2(4, dd, 8, 8) + +#undef DO_STN_1 +#undef DO_STN_2 + /* Loads with a vector index. */ #define DO_LD1_ZPZ_S(NAME, TYPEI, TYPEM, FN) \ From patchwork Thu Aug 9 04:22:01 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 143673 Delivered-To: patch@linaro.org Received: by 2002:a2e:9754:0:0:0:0:0 with SMTP id f20-v6csp1612617ljj; Wed, 8 Aug 2018 21:37:30 -0700 (PDT) X-Google-Smtp-Source: AA+uWPyp3wZzbK/NPGErr5WVRkjyH0NV7FybOOwxzxoxKxAZo6qQh+ZmIalu9FqDJr8D1NecTM0E X-Received: by 2002:a37:d98f:: with SMTP id q15-v6mr432200qkl.167.1533789450603; Wed, 08 Aug 2018 21:37:30 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1533789450; cv=none; d=google.com; s=arc-20160816; b=RlD+1TW4DhDv9nIS0JwNk79+YStbkcicr22qF6Pw1hBmajk16N1E/3mVaGGhjPGIKD IxEUFBHssNn7W7/eAq8g8rtYLCQsiBzZOhSCNkm2Niu4oJQHQz4poLSzy1Iw70+5Or3v bUs9IxpRN+QYv7HayoTqQ4P8p9XRWnwB60Avz6bL3hIs5b7flZh3QrQ+SqCL3RiGI9ub GPLun5pKQEdHjgXrp7390jk3NhZNJgHojNdeXZi9Kh7YJ/MUNvl9In8ROP/Pb9wYQIcq EHpDG4DuNcqOY4mxGtTns4iOTyZ8Q8SQUloMazQyiDZQb/pKnPO+EEqAG3EcbI9SKK93 O28g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:cc:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:subject:references:in-reply-to :message-id:date:to:from:dkim-signature:arc-authentication-results; bh=TKl2+vY+FvUE0FrPaTF9gdJTO3MM+Z0r21MXYMD4wQE=; b=qVcCDNQ83AsZPc9mVkZ8br2Mm2vAyjHEp0W3Hx+Bk4mORpAj4x6xAAk1FeqdtAuSqw u3HP2wGuVRU8X3cWjxEw1etnZwPs9V0uZLSv3wSA1AGrv5n19/1Zja/rSmdKQO+aeCFP b2QbLKGWGLDNXs7XVyOjP+qumxs3Dnvk71liQsuq+WJPKDj7Bq4TlgxTHw3XCkpkFz4d /n7wGGVmc0YrLCBSAAjVcQ0MJdf/E20g8mLHSDBghM+F/HnOANJuK9pph8fLQQy4Up72 QTY6Hh+CgvfRuCxkz3H9WuKwOTLnYZ1in4Gb9an5QUz2sSDfwpDahci1LNdlQ8ayAhuL oMuA== ARC-Authentication-Results: i=1; mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=bzhv7hgJ; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [2001:4830:134:3::11]) by mx.google.com with ESMTPS id p53-v6si4985962qvc.131.2018.08.08.21.37.30 for (version=TLS1 cipher=AES128-SHA bits=128/128); Wed, 08 Aug 2018 21:37:30 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) client-ip=2001:4830:134:3::11; Authentication-Results: mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=bzhv7hgJ; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1]:47301 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fnchW-0005AO-2q for patch@linaro.org; Thu, 09 Aug 2018 00:37:30 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:54112) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fncT4-0000zL-CJ for qemu-devel@nongnu.org; Thu, 09 Aug 2018 00:22:37 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fncT1-0007Pt-Hf for qemu-devel@nongnu.org; Thu, 09 Aug 2018 00:22:34 -0400 Received: from mail-pf1-x42b.google.com ([2607:f8b0:4864:20::42b]:40058) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fncT1-0007P0-67 for qemu-devel@nongnu.org; Thu, 09 Aug 2018 00:22:31 -0400 Received: by mail-pf1-x42b.google.com with SMTP id e13-v6so2191099pff.7 for ; Wed, 08 Aug 2018 21:22:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=TKl2+vY+FvUE0FrPaTF9gdJTO3MM+Z0r21MXYMD4wQE=; b=bzhv7hgJ4f5Z/gN2L8midW7M5kMMO9r0UWss+iVL26pD4+YTg22T73n21548z+ro78 YwmdWfgEZP0OrfsVYZh6jqcx1mrZ+OBiL7g1NqD40I3YjQkxYDmGWkh0dSPnS//+53cF tFGRERoDYoUfcZ2tKENDgfBPnOYSJiwkGCi4I= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=TKl2+vY+FvUE0FrPaTF9gdJTO3MM+Z0r21MXYMD4wQE=; b=SRM+c5caF5SXl86KzxEbljx4VyBjFDgX+7jDKpX9NI4d0CcUVvj563Ktehhz0MIeYl +yFqmwS3wJpI3OXLxi/vt3HcSeFjOf+En6nUI+N33UbJchI9qfSP1RZV5OkT34tJiXQB waUkEY2O/1OibAXWIqOrru4f/tOqcPQ6VuPAGscRj4367fidCKfLJcFhH9rStW8KMWoN ploZvj3uCoKOtKV8HI7ZtCnMjyoWrOWEwIeiU99tQwE5SK/rOWYtNn/5l0oPTO/gFsCV JBmtStL7ur5Rrt0NTHt0Wo4rK2UIrWSGlAJq0ZIfitIK27wCwEf5MprrpEiko/jdm+w7 mTqQ== X-Gm-Message-State: AOUpUlHbe6RbsV9wxZrncRDNRLpfmD3QEVmSBXbGd+ouOkV/oaGwW1BX XkVEoyCwKkmUi7bPQ0+x46lRAUjo1XE= X-Received: by 2002:a63:416:: with SMTP id 22-v6mr499392pge.229.1533788549670; Wed, 08 Aug 2018 21:22:29 -0700 (PDT) Received: from cloudburst.twiddle.net (97-113-8-179.tukw.qwest.net. [97.113.8.179]) by smtp.gmail.com with ESMTPSA id m30-v6sm7355799pff.121.2018.08.08.21.22.28 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Wed, 08 Aug 2018 21:22:28 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Date: Wed, 8 Aug 2018 21:22:01 -0700 Message-Id: <20180809042206.15726-16-richard.henderson@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180809042206.15726-1-richard.henderson@linaro.org> References: <20180809042206.15726-1-richard.henderson@linaro.org> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::42b Subject: [Qemu-devel] [PATCH 15/20] target/arm: Split contiguous loads for endianness X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: laurent.desnogues@gmail.com, peter.maydell@linaro.org, alex.bennee@linaro.org Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: "Qemu-devel" We can choose the endianness at translation time, rather than re-computing it at execution time. Signed-off-by: Richard Henderson --- target/arm/helper-sve.h | 117 +++++++++++++++------- target/arm/sve_helper.c | 70 ++++++------- target/arm/translate-sve.c | 196 +++++++++++++++++++++++++------------ 3 files changed, 252 insertions(+), 131 deletions(-) -- 2.17.1 Reviewed-by: Philippe Mathieu-Daudé diff --git a/target/arm/helper-sve.h b/target/arm/helper-sve.h index 023952a9a4..526caec8da 100644 --- a/target/arm/helper-sve.h +++ b/target/arm/helper-sve.h @@ -1128,20 +1128,35 @@ DEF_HELPER_FLAGS_4(sve_ld2bb_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) DEF_HELPER_FLAGS_4(sve_ld3bb_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) DEF_HELPER_FLAGS_4(sve_ld4bb_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) -DEF_HELPER_FLAGS_4(sve_ld1hh_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) -DEF_HELPER_FLAGS_4(sve_ld2hh_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) -DEF_HELPER_FLAGS_4(sve_ld3hh_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) -DEF_HELPER_FLAGS_4(sve_ld4hh_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) +DEF_HELPER_FLAGS_4(sve_ld1hh_le_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) +DEF_HELPER_FLAGS_4(sve_ld2hh_le_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) +DEF_HELPER_FLAGS_4(sve_ld3hh_le_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) +DEF_HELPER_FLAGS_4(sve_ld4hh_le_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) -DEF_HELPER_FLAGS_4(sve_ld1ss_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) -DEF_HELPER_FLAGS_4(sve_ld2ss_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) -DEF_HELPER_FLAGS_4(sve_ld3ss_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) -DEF_HELPER_FLAGS_4(sve_ld4ss_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) +DEF_HELPER_FLAGS_4(sve_ld1hh_be_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) +DEF_HELPER_FLAGS_4(sve_ld2hh_be_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) +DEF_HELPER_FLAGS_4(sve_ld3hh_be_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) +DEF_HELPER_FLAGS_4(sve_ld4hh_be_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) -DEF_HELPER_FLAGS_4(sve_ld1dd_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) -DEF_HELPER_FLAGS_4(sve_ld2dd_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) -DEF_HELPER_FLAGS_4(sve_ld3dd_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) -DEF_HELPER_FLAGS_4(sve_ld4dd_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) +DEF_HELPER_FLAGS_4(sve_ld1ss_le_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) +DEF_HELPER_FLAGS_4(sve_ld2ss_le_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) +DEF_HELPER_FLAGS_4(sve_ld3ss_le_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) +DEF_HELPER_FLAGS_4(sve_ld4ss_le_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) + +DEF_HELPER_FLAGS_4(sve_ld1ss_be_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) +DEF_HELPER_FLAGS_4(sve_ld2ss_be_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) +DEF_HELPER_FLAGS_4(sve_ld3ss_be_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) +DEF_HELPER_FLAGS_4(sve_ld4ss_be_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) + +DEF_HELPER_FLAGS_4(sve_ld1dd_le_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) +DEF_HELPER_FLAGS_4(sve_ld2dd_le_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) +DEF_HELPER_FLAGS_4(sve_ld3dd_le_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) +DEF_HELPER_FLAGS_4(sve_ld4dd_le_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) + +DEF_HELPER_FLAGS_4(sve_ld1dd_be_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) +DEF_HELPER_FLAGS_4(sve_ld2dd_be_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) +DEF_HELPER_FLAGS_4(sve_ld3dd_be_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) +DEF_HELPER_FLAGS_4(sve_ld4dd_be_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) DEF_HELPER_FLAGS_4(sve_ld1bhu_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) DEF_HELPER_FLAGS_4(sve_ld1bsu_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) @@ -1150,13 +1165,21 @@ DEF_HELPER_FLAGS_4(sve_ld1bhs_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) DEF_HELPER_FLAGS_4(sve_ld1bss_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) DEF_HELPER_FLAGS_4(sve_ld1bds_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) -DEF_HELPER_FLAGS_4(sve_ld1hsu_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) -DEF_HELPER_FLAGS_4(sve_ld1hdu_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) -DEF_HELPER_FLAGS_4(sve_ld1hss_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) -DEF_HELPER_FLAGS_4(sve_ld1hds_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) +DEF_HELPER_FLAGS_4(sve_ld1hsu_le_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) +DEF_HELPER_FLAGS_4(sve_ld1hdu_le_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) +DEF_HELPER_FLAGS_4(sve_ld1hss_le_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) +DEF_HELPER_FLAGS_4(sve_ld1hds_le_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) -DEF_HELPER_FLAGS_4(sve_ld1sdu_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) -DEF_HELPER_FLAGS_4(sve_ld1sds_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) +DEF_HELPER_FLAGS_4(sve_ld1hsu_be_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) +DEF_HELPER_FLAGS_4(sve_ld1hdu_be_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) +DEF_HELPER_FLAGS_4(sve_ld1hss_be_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) +DEF_HELPER_FLAGS_4(sve_ld1hds_be_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) + +DEF_HELPER_FLAGS_4(sve_ld1sdu_le_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) +DEF_HELPER_FLAGS_4(sve_ld1sds_le_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) + +DEF_HELPER_FLAGS_4(sve_ld1sdu_be_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) +DEF_HELPER_FLAGS_4(sve_ld1sds_be_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) DEF_HELPER_FLAGS_4(sve_ldff1bb_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) DEF_HELPER_FLAGS_4(sve_ldff1bhu_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) @@ -1166,17 +1189,28 @@ DEF_HELPER_FLAGS_4(sve_ldff1bhs_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) DEF_HELPER_FLAGS_4(sve_ldff1bss_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) DEF_HELPER_FLAGS_4(sve_ldff1bds_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) -DEF_HELPER_FLAGS_4(sve_ldff1hh_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) -DEF_HELPER_FLAGS_4(sve_ldff1hsu_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) -DEF_HELPER_FLAGS_4(sve_ldff1hdu_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) -DEF_HELPER_FLAGS_4(sve_ldff1hss_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) -DEF_HELPER_FLAGS_4(sve_ldff1hds_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) +DEF_HELPER_FLAGS_4(sve_ldff1hh_le_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) +DEF_HELPER_FLAGS_4(sve_ldff1hsu_le_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) +DEF_HELPER_FLAGS_4(sve_ldff1hdu_le_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) +DEF_HELPER_FLAGS_4(sve_ldff1hss_le_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) +DEF_HELPER_FLAGS_4(sve_ldff1hds_le_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) -DEF_HELPER_FLAGS_4(sve_ldff1ss_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) -DEF_HELPER_FLAGS_4(sve_ldff1sdu_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) -DEF_HELPER_FLAGS_4(sve_ldff1sds_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) +DEF_HELPER_FLAGS_4(sve_ldff1hh_be_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) +DEF_HELPER_FLAGS_4(sve_ldff1hsu_be_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) +DEF_HELPER_FLAGS_4(sve_ldff1hdu_be_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) +DEF_HELPER_FLAGS_4(sve_ldff1hss_be_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) +DEF_HELPER_FLAGS_4(sve_ldff1hds_be_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) -DEF_HELPER_FLAGS_4(sve_ldff1dd_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) +DEF_HELPER_FLAGS_4(sve_ldff1ss_le_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) +DEF_HELPER_FLAGS_4(sve_ldff1sdu_le_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) +DEF_HELPER_FLAGS_4(sve_ldff1sds_le_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) + +DEF_HELPER_FLAGS_4(sve_ldff1ss_be_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) +DEF_HELPER_FLAGS_4(sve_ldff1sdu_be_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) +DEF_HELPER_FLAGS_4(sve_ldff1sds_be_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) + +DEF_HELPER_FLAGS_4(sve_ldff1dd_le_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) +DEF_HELPER_FLAGS_4(sve_ldff1dd_be_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) DEF_HELPER_FLAGS_4(sve_ldnf1bb_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) DEF_HELPER_FLAGS_4(sve_ldnf1bhu_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) @@ -1186,17 +1220,28 @@ DEF_HELPER_FLAGS_4(sve_ldnf1bhs_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) DEF_HELPER_FLAGS_4(sve_ldnf1bss_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) DEF_HELPER_FLAGS_4(sve_ldnf1bds_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) -DEF_HELPER_FLAGS_4(sve_ldnf1hh_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) -DEF_HELPER_FLAGS_4(sve_ldnf1hsu_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) -DEF_HELPER_FLAGS_4(sve_ldnf1hdu_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) -DEF_HELPER_FLAGS_4(sve_ldnf1hss_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) -DEF_HELPER_FLAGS_4(sve_ldnf1hds_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) +DEF_HELPER_FLAGS_4(sve_ldnf1hh_le_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) +DEF_HELPER_FLAGS_4(sve_ldnf1hsu_le_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) +DEF_HELPER_FLAGS_4(sve_ldnf1hdu_le_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) +DEF_HELPER_FLAGS_4(sve_ldnf1hss_le_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) +DEF_HELPER_FLAGS_4(sve_ldnf1hds_le_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) -DEF_HELPER_FLAGS_4(sve_ldnf1ss_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) -DEF_HELPER_FLAGS_4(sve_ldnf1sdu_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) -DEF_HELPER_FLAGS_4(sve_ldnf1sds_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) +DEF_HELPER_FLAGS_4(sve_ldnf1hh_be_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) +DEF_HELPER_FLAGS_4(sve_ldnf1hsu_be_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) +DEF_HELPER_FLAGS_4(sve_ldnf1hdu_be_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) +DEF_HELPER_FLAGS_4(sve_ldnf1hss_be_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) +DEF_HELPER_FLAGS_4(sve_ldnf1hds_be_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) -DEF_HELPER_FLAGS_4(sve_ldnf1dd_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) +DEF_HELPER_FLAGS_4(sve_ldnf1ss_le_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) +DEF_HELPER_FLAGS_4(sve_ldnf1sdu_le_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) +DEF_HELPER_FLAGS_4(sve_ldnf1sds_le_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) + +DEF_HELPER_FLAGS_4(sve_ldnf1ss_be_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) +DEF_HELPER_FLAGS_4(sve_ldnf1sdu_be_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) +DEF_HELPER_FLAGS_4(sve_ldnf1sds_be_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) + +DEF_HELPER_FLAGS_4(sve_ldnf1dd_le_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) +DEF_HELPER_FLAGS_4(sve_ldnf1dd_be_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) DEF_HELPER_FLAGS_4(sve_st1bb_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) DEF_HELPER_FLAGS_4(sve_st2bb_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) diff --git a/target/arm/sve_helper.c b/target/arm/sve_helper.c index 4eae6569cc..56e2f523c5 100644 --- a/target/arm/sve_helper.c +++ b/target/arm/sve_helper.c @@ -4362,18 +4362,18 @@ void HELPER(sve_##NAME##_r)(CPUARMState *env, void *vg, \ sve_##NAME##_host, sve_##NAME##_tlb); \ } -/* TODO: Propagate the endian check back to the translator. */ #define DO_LD1_2(NAME, ESZ, MSZ) \ -void HELPER(sve_##NAME##_r)(CPUARMState *env, void *vg, \ - target_ulong addr, uint32_t desc) \ -{ \ - if (arm_cpu_data_is_big_endian(env)) { \ - sve_ld1_r(env, vg, addr, desc, GETPC(), ESZ, MSZ, \ - sve_##NAME##_be_host, sve_##NAME##_be_tlb); \ - } else { \ - sve_ld1_r(env, vg, addr, desc, GETPC(), ESZ, MSZ, \ - sve_##NAME##_le_host, sve_##NAME##_le_tlb); \ - } \ +void HELPER(sve_##NAME##_le_r)(CPUARMState *env, void *vg, \ + target_ulong addr, uint32_t desc) \ +{ \ + sve_ld1_r(env, vg, addr, desc, GETPC(), ESZ, MSZ, \ + sve_##NAME##_le_host, sve_##NAME##_le_tlb); \ +} \ +void HELPER(sve_##NAME##_be_r)(CPUARMState *env, void *vg, \ + target_ulong addr, uint32_t desc) \ +{ \ + sve_ld1_r(env, vg, addr, desc, GETPC(), ESZ, MSZ, \ + sve_##NAME##_be_host, sve_##NAME##_be_tlb); \ } DO_LD1_1(ld1bb, 0) @@ -4500,12 +4500,17 @@ void __attribute__((flatten)) HELPER(sve_ld##N##bb_r) \ } #define DO_LDN_2(N, SUFF, SIZE) \ -void __attribute__((flatten)) HELPER(sve_ld##N##SUFF##_r) \ +void __attribute__((flatten)) HELPER(sve_ld##N##SUFF##_le_r) \ (CPUARMState *env, void *vg, target_ulong addr, uint32_t desc) \ { \ sve_ld##N##_r(env, vg, addr, desc, SIZE, GETPC(), \ - arm_cpu_data_is_big_endian(env) \ - ? sve_ld1##SUFF##_be_tlb : sve_ld1##SUFF##_le_tlb); \ + sve_ld1##SUFF##_le_tlb); \ +} \ +void __attribute__((flatten)) HELPER(sve_ld##N##SUFF##_be_r) \ + (CPUARMState *env, void *vg, target_ulong addr, uint32_t desc) \ +{ \ + sve_ld##N##_r(env, vg, addr, desc, SIZE, GETPC(), \ + sve_ld1##SUFF##_be_tlb); \ } DO_LDN_1(2) @@ -4722,29 +4727,28 @@ void HELPER(sve_ldnf1##PART##_r)(CPUARMState *env, void *vg, \ sve_ldnf1_r(env, vg, addr, desc, ESZ, 0, sve_ld1##PART##_host); \ } -/* TODO: Propagate the endian check back to the translator. */ #define DO_LDFF1_LDNF1_2(PART, ESZ, MSZ) \ -void HELPER(sve_ldff1##PART##_r)(CPUARMState *env, void *vg, \ - target_ulong addr, uint32_t desc) \ +void HELPER(sve_ldff1##PART##_le_r)(CPUARMState *env, void *vg, \ + target_ulong addr, uint32_t desc) \ { \ - if (arm_cpu_data_is_big_endian(env)) { \ - sve_ldff1_r(env, vg, addr, desc, GETPC(), ESZ, MSZ, \ - sve_ld1##PART##_be_host, sve_ld1##PART##_be_tlb); \ - } else { \ - sve_ldff1_r(env, vg, addr, desc, GETPC(), ESZ, MSZ, \ - sve_ld1##PART##_le_host, sve_ld1##PART##_le_tlb); \ - } \ + sve_ldff1_r(env, vg, addr, desc, GETPC(), ESZ, MSZ, \ + sve_ld1##PART##_le_host, sve_ld1##PART##_le_tlb); \ } \ -void HELPER(sve_ldnf1##PART##_r)(CPUARMState *env, void *vg, \ - target_ulong addr, uint32_t desc) \ +void HELPER(sve_ldnf1##PART##_le_r)(CPUARMState *env, void *vg, \ + target_ulong addr, uint32_t desc) \ { \ - if (arm_cpu_data_is_big_endian(env)) { \ - sve_ldnf1_r(env, vg, addr, desc, ESZ, MSZ, \ - sve_ld1##PART##_be_host); \ - } else { \ - sve_ldnf1_r(env, vg, addr, desc, ESZ, MSZ, \ - sve_ld1##PART##_le_host); \ - } \ + sve_ldnf1_r(env, vg, addr, desc, ESZ, MSZ, sve_ld1##PART##_le_host); \ +} \ +void HELPER(sve_ldff1##PART##_be_r)(CPUARMState *env, void *vg, \ + target_ulong addr, uint32_t desc) \ +{ \ + sve_ldff1_r(env, vg, addr, desc, GETPC(), ESZ, MSZ, \ + sve_ld1##PART##_be_host, sve_ld1##PART##_be_tlb); \ +} \ +void HELPER(sve_ldnf1##PART##_be_r)(CPUARMState *env, void *vg, \ + target_ulong addr, uint32_t desc) \ +{ \ + sve_ldnf1_r(env, vg, addr, desc, ESZ, MSZ, sve_ld1##PART##_be_host); \ } DO_LDFF1_LDNF1_1(bb, 0) diff --git a/target/arm/translate-sve.c b/target/arm/translate-sve.c index bef6b8242d..de12c01e7d 100644 --- a/target/arm/translate-sve.c +++ b/target/arm/translate-sve.c @@ -4624,32 +4624,58 @@ static void do_mem_zpa(DisasContext *s, int zt, int pg, TCGv_i64 addr, static void do_ld_zpa(DisasContext *s, int zt, int pg, TCGv_i64 addr, int dtype, int nreg) { - static gen_helper_gvec_mem * const fns[16][4] = { - { gen_helper_sve_ld1bb_r, gen_helper_sve_ld2bb_r, - gen_helper_sve_ld3bb_r, gen_helper_sve_ld4bb_r }, - { gen_helper_sve_ld1bhu_r, NULL, NULL, NULL }, - { gen_helper_sve_ld1bsu_r, NULL, NULL, NULL }, - { gen_helper_sve_ld1bdu_r, NULL, NULL, NULL }, + static gen_helper_gvec_mem * const fns[2][16][4] = { + /* Little-endian */ + { { gen_helper_sve_ld1bb_r, gen_helper_sve_ld2bb_r, + gen_helper_sve_ld3bb_r, gen_helper_sve_ld4bb_r }, + { gen_helper_sve_ld1bhu_r, NULL, NULL, NULL }, + { gen_helper_sve_ld1bsu_r, NULL, NULL, NULL }, + { gen_helper_sve_ld1bdu_r, NULL, NULL, NULL }, - { gen_helper_sve_ld1sds_r, NULL, NULL, NULL }, - { gen_helper_sve_ld1hh_r, gen_helper_sve_ld2hh_r, - gen_helper_sve_ld3hh_r, gen_helper_sve_ld4hh_r }, - { gen_helper_sve_ld1hsu_r, NULL, NULL, NULL }, - { gen_helper_sve_ld1hdu_r, NULL, NULL, NULL }, + { gen_helper_sve_ld1sds_le_r, NULL, NULL, NULL }, + { gen_helper_sve_ld1hh_le_r, gen_helper_sve_ld2hh_le_r, + gen_helper_sve_ld3hh_le_r, gen_helper_sve_ld4hh_le_r }, + { gen_helper_sve_ld1hsu_le_r, NULL, NULL, NULL }, + { gen_helper_sve_ld1hdu_le_r, NULL, NULL, NULL }, - { gen_helper_sve_ld1hds_r, NULL, NULL, NULL }, - { gen_helper_sve_ld1hss_r, NULL, NULL, NULL }, - { gen_helper_sve_ld1ss_r, gen_helper_sve_ld2ss_r, - gen_helper_sve_ld3ss_r, gen_helper_sve_ld4ss_r }, - { gen_helper_sve_ld1sdu_r, NULL, NULL, NULL }, + { gen_helper_sve_ld1hds_le_r, NULL, NULL, NULL }, + { gen_helper_sve_ld1hss_le_r, NULL, NULL, NULL }, + { gen_helper_sve_ld1ss_le_r, gen_helper_sve_ld2ss_le_r, + gen_helper_sve_ld3ss_le_r, gen_helper_sve_ld4ss_le_r }, + { gen_helper_sve_ld1sdu_le_r, NULL, NULL, NULL }, - { gen_helper_sve_ld1bds_r, NULL, NULL, NULL }, - { gen_helper_sve_ld1bss_r, NULL, NULL, NULL }, - { gen_helper_sve_ld1bhs_r, NULL, NULL, NULL }, - { gen_helper_sve_ld1dd_r, gen_helper_sve_ld2dd_r, - gen_helper_sve_ld3dd_r, gen_helper_sve_ld4dd_r }, + { gen_helper_sve_ld1bds_r, NULL, NULL, NULL }, + { gen_helper_sve_ld1bss_r, NULL, NULL, NULL }, + { gen_helper_sve_ld1bhs_r, NULL, NULL, NULL }, + { gen_helper_sve_ld1dd_le_r, gen_helper_sve_ld2dd_le_r, + gen_helper_sve_ld3dd_le_r, gen_helper_sve_ld4dd_le_r } }, + + /* Big-endian */ + { { gen_helper_sve_ld1bb_r, gen_helper_sve_ld2bb_r, + gen_helper_sve_ld3bb_r, gen_helper_sve_ld4bb_r }, + { gen_helper_sve_ld1bhu_r, NULL, NULL, NULL }, + { gen_helper_sve_ld1bsu_r, NULL, NULL, NULL }, + { gen_helper_sve_ld1bdu_r, NULL, NULL, NULL }, + + { gen_helper_sve_ld1sds_be_r, NULL, NULL, NULL }, + { gen_helper_sve_ld1hh_be_r, gen_helper_sve_ld2hh_be_r, + gen_helper_sve_ld3hh_be_r, gen_helper_sve_ld4hh_be_r }, + { gen_helper_sve_ld1hsu_be_r, NULL, NULL, NULL }, + { gen_helper_sve_ld1hdu_be_r, NULL, NULL, NULL }, + + { gen_helper_sve_ld1hds_be_r, NULL, NULL, NULL }, + { gen_helper_sve_ld1hss_be_r, NULL, NULL, NULL }, + { gen_helper_sve_ld1ss_be_r, gen_helper_sve_ld2ss_be_r, + gen_helper_sve_ld3ss_be_r, gen_helper_sve_ld4ss_be_r }, + { gen_helper_sve_ld1sdu_be_r, NULL, NULL, NULL }, + + { gen_helper_sve_ld1bds_r, NULL, NULL, NULL }, + { gen_helper_sve_ld1bss_r, NULL, NULL, NULL }, + { gen_helper_sve_ld1bhs_r, NULL, NULL, NULL }, + { gen_helper_sve_ld1dd_be_r, gen_helper_sve_ld2dd_be_r, + gen_helper_sve_ld3dd_be_r, gen_helper_sve_ld4dd_be_r } } }; - gen_helper_gvec_mem *fn = fns[dtype][nreg]; + gen_helper_gvec_mem *fn = fns[s->be_data == MO_BE][dtype][nreg]; /* While there are holes in the table, they are not * accessible via the instruction encoding. @@ -4689,59 +4715,103 @@ static bool trans_LD_zpri(DisasContext *s, arg_rpri_load *a, uint32_t insn) static bool trans_LDFF1_zprr(DisasContext *s, arg_rprr_load *a, uint32_t insn) { - static gen_helper_gvec_mem * const fns[16] = { - gen_helper_sve_ldff1bb_r, - gen_helper_sve_ldff1bhu_r, - gen_helper_sve_ldff1bsu_r, - gen_helper_sve_ldff1bdu_r, + static gen_helper_gvec_mem * const fns[2][16] = { + /* Little-endian */ + { gen_helper_sve_ldff1bb_r, + gen_helper_sve_ldff1bhu_r, + gen_helper_sve_ldff1bsu_r, + gen_helper_sve_ldff1bdu_r, - gen_helper_sve_ldff1sds_r, - gen_helper_sve_ldff1hh_r, - gen_helper_sve_ldff1hsu_r, - gen_helper_sve_ldff1hdu_r, + gen_helper_sve_ldff1sds_le_r, + gen_helper_sve_ldff1hh_le_r, + gen_helper_sve_ldff1hsu_le_r, + gen_helper_sve_ldff1hdu_le_r, - gen_helper_sve_ldff1hds_r, - gen_helper_sve_ldff1hss_r, - gen_helper_sve_ldff1ss_r, - gen_helper_sve_ldff1sdu_r, + gen_helper_sve_ldff1hds_le_r, + gen_helper_sve_ldff1hss_le_r, + gen_helper_sve_ldff1ss_le_r, + gen_helper_sve_ldff1sdu_le_r, - gen_helper_sve_ldff1bds_r, - gen_helper_sve_ldff1bss_r, - gen_helper_sve_ldff1bhs_r, - gen_helper_sve_ldff1dd_r, + gen_helper_sve_ldff1bds_r, + gen_helper_sve_ldff1bss_r, + gen_helper_sve_ldff1bhs_r, + gen_helper_sve_ldff1dd_le_r }, + + /* Big-endian */ + { gen_helper_sve_ldff1bb_r, + gen_helper_sve_ldff1bhu_r, + gen_helper_sve_ldff1bsu_r, + gen_helper_sve_ldff1bdu_r, + + gen_helper_sve_ldff1sds_be_r, + gen_helper_sve_ldff1hh_be_r, + gen_helper_sve_ldff1hsu_be_r, + gen_helper_sve_ldff1hdu_be_r, + + gen_helper_sve_ldff1hds_be_r, + gen_helper_sve_ldff1hss_be_r, + gen_helper_sve_ldff1ss_be_r, + gen_helper_sve_ldff1sdu_be_r, + + gen_helper_sve_ldff1bds_r, + gen_helper_sve_ldff1bss_r, + gen_helper_sve_ldff1bhs_r, + gen_helper_sve_ldff1dd_be_r }, }; if (sve_access_check(s)) { TCGv_i64 addr = new_tmp_a64(s); tcg_gen_shli_i64(addr, cpu_reg(s, a->rm), dtype_msz(a->dtype)); tcg_gen_add_i64(addr, addr, cpu_reg_sp(s, a->rn)); - do_mem_zpa(s, a->rd, a->pg, addr, fns[a->dtype]); + do_mem_zpa(s, a->rd, a->pg, addr, fns[s->be_data == MO_BE][a->dtype]); } return true; } static bool trans_LDNF1_zpri(DisasContext *s, arg_rpri_load *a, uint32_t insn) { - static gen_helper_gvec_mem * const fns[16] = { - gen_helper_sve_ldnf1bb_r, - gen_helper_sve_ldnf1bhu_r, - gen_helper_sve_ldnf1bsu_r, - gen_helper_sve_ldnf1bdu_r, + static gen_helper_gvec_mem * const fns[2][16] = { + /* Little-endian */ + { gen_helper_sve_ldnf1bb_r, + gen_helper_sve_ldnf1bhu_r, + gen_helper_sve_ldnf1bsu_r, + gen_helper_sve_ldnf1bdu_r, - gen_helper_sve_ldnf1sds_r, - gen_helper_sve_ldnf1hh_r, - gen_helper_sve_ldnf1hsu_r, - gen_helper_sve_ldnf1hdu_r, + gen_helper_sve_ldnf1sds_le_r, + gen_helper_sve_ldnf1hh_le_r, + gen_helper_sve_ldnf1hsu_le_r, + gen_helper_sve_ldnf1hdu_le_r, - gen_helper_sve_ldnf1hds_r, - gen_helper_sve_ldnf1hss_r, - gen_helper_sve_ldnf1ss_r, - gen_helper_sve_ldnf1sdu_r, + gen_helper_sve_ldnf1hds_le_r, + gen_helper_sve_ldnf1hss_le_r, + gen_helper_sve_ldnf1ss_le_r, + gen_helper_sve_ldnf1sdu_le_r, - gen_helper_sve_ldnf1bds_r, - gen_helper_sve_ldnf1bss_r, - gen_helper_sve_ldnf1bhs_r, - gen_helper_sve_ldnf1dd_r, + gen_helper_sve_ldnf1bds_r, + gen_helper_sve_ldnf1bss_r, + gen_helper_sve_ldnf1bhs_r, + gen_helper_sve_ldnf1dd_le_r }, + + /* Big-endian */ + { gen_helper_sve_ldnf1bb_r, + gen_helper_sve_ldnf1bhu_r, + gen_helper_sve_ldnf1bsu_r, + gen_helper_sve_ldnf1bdu_r, + + gen_helper_sve_ldnf1sds_be_r, + gen_helper_sve_ldnf1hh_be_r, + gen_helper_sve_ldnf1hsu_be_r, + gen_helper_sve_ldnf1hdu_be_r, + + gen_helper_sve_ldnf1hds_be_r, + gen_helper_sve_ldnf1hss_be_r, + gen_helper_sve_ldnf1ss_be_r, + gen_helper_sve_ldnf1sdu_be_r, + + gen_helper_sve_ldnf1bds_r, + gen_helper_sve_ldnf1bss_r, + gen_helper_sve_ldnf1bhs_r, + gen_helper_sve_ldnf1dd_be_r }, }; if (sve_access_check(s)) { @@ -4751,16 +4821,18 @@ static bool trans_LDNF1_zpri(DisasContext *s, arg_rpri_load *a, uint32_t insn) TCGv_i64 addr = new_tmp_a64(s); tcg_gen_addi_i64(addr, cpu_reg_sp(s, a->rn), off); - do_mem_zpa(s, a->rd, a->pg, addr, fns[a->dtype]); + do_mem_zpa(s, a->rd, a->pg, addr, fns[s->be_data == MO_BE][a->dtype]); } return true; } static void do_ldrq(DisasContext *s, int zt, int pg, TCGv_i64 addr, int msz) { - static gen_helper_gvec_mem * const fns[4] = { - gen_helper_sve_ld1bb_r, gen_helper_sve_ld1hh_r, - gen_helper_sve_ld1ss_r, gen_helper_sve_ld1dd_r, + static gen_helper_gvec_mem * const fns[2][4] = { + { gen_helper_sve_ld1bb_r, gen_helper_sve_ld1hh_le_r, + gen_helper_sve_ld1ss_le_r, gen_helper_sve_ld1dd_le_r }, + { gen_helper_sve_ld1bb_r, gen_helper_sve_ld1hh_be_r, + gen_helper_sve_ld1ss_be_r, gen_helper_sve_ld1dd_be_r }, }; unsigned vsz = vec_full_reg_size(s); TCGv_ptr t_pg; @@ -4792,7 +4864,7 @@ static void do_ldrq(DisasContext *s, int zt, int pg, TCGv_i64 addr, int msz) t_pg = tcg_temp_new_ptr(); tcg_gen_addi_ptr(t_pg, cpu_env, poff); - fns[msz](cpu_env, t_pg, addr, desc); + fns[s->be_data == MO_BE][msz](cpu_env, t_pg, addr, desc); tcg_temp_free_ptr(t_pg); tcg_temp_free_i32(desc); From patchwork Thu Aug 9 04:22:02 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 143671 Delivered-To: patch@linaro.org Received: by 2002:a2e:9754:0:0:0:0:0 with SMTP id f20-v6csp1611753ljj; Wed, 8 Aug 2018 21:36:20 -0700 (PDT) X-Google-Smtp-Source: AA+uWPyHAcm+n4B+xCPSatcDTslOKnFcagk/y0w4nFoA9qVPxqiVmsPiRzYLLaU7GSfwl6c2/g1L X-Received: by 2002:a0c:8745:: with SMTP id 5-v6mr411432qvi.223.1533789380819; Wed, 08 Aug 2018 21:36:20 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1533789380; cv=none; d=google.com; s=arc-20160816; b=tjGAlM6Xma90sOVK+o+bq1j3nnijadI7TJTFOgHKW9gGsab0ZRmtSVCKNDKrCUEnLb hrcv4YfI+iHdZXKz3WRz3odla2VvfThs9OxKu3O3gk7m59888aqpuC66BRiwrZvc/4+e JIB74jF1iwtRgoQUW/xcNC7EKQ3m/V8UEst6KrjCiX2CNehn2g2llFTlYev6Kf6pSZWl ZUh5HqhgAP3HzlymzPMvWyon5e2l1JzAhJJ8eGiWCAPPxhdTknGahA9PKJxzKKM7ApB+ s4Y8WYANm/WmqTakiqavw/QW9QnjXx204oIUxTMPKqq72VvvaM4i+M3Z2IwU/7jamESm 6WdA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:cc:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:subject:references:in-reply-to :message-id:date:to:from:dkim-signature:arc-authentication-results; bh=ZP9LjvZ7RdzvXKxbJAUmEFUz9QE9Ls0QkL+ef9F8j6s=; b=biipJzuE7nM5llOjMT/TR0EAOt8ZKjIExC2jPUDqAXj+zyquVjTNPJ8eTjfjU6rfJ0 MjVYtWETpaN/gOpCn1JQaors5poqBJITnEK3zXo14VYM+aKrluHHGNXtkcwgC2zVO7N6 FMb753XMRjMhLPFnYAOL11Ny8aGND/oopbi43meUYLIHt03KURN7skCkTE5vEsEv9s+T IVeiohJ1iUsMNQZMf9DlRTQfRPuIb1t71tV073xLXDyPcIL9QJGsk52JmOZbxa+oXhKk oUPgUABdRDNYJ8TxnXMAiLhaZGhmHwt2eG5vQg4TykDrfRF12dRRf3aYVyxs1KS5dZb1 6zCA== ARC-Authentication-Results: i=1; mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=Z+ScdScp; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [2001:4830:134:3::11]) by mx.google.com with ESMTPS id m54-v6si5623633qvh.220.2018.08.08.21.36.20 for (version=TLS1 cipher=AES128-SHA bits=128/128); Wed, 08 Aug 2018 21:36:20 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) client-ip=2001:4830:134:3::11; Authentication-Results: mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=Z+ScdScp; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1]:47296 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fncgO-0004PF-AN for patch@linaro.org; Thu, 09 Aug 2018 00:36:20 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:54114) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fncT4-0000zM-DH for qemu-devel@nongnu.org; Thu, 09 Aug 2018 00:22:36 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fncT2-0007Qy-NI for qemu-devel@nongnu.org; Thu, 09 Aug 2018 00:22:34 -0400 Received: from mail-pf1-x42b.google.com ([2607:f8b0:4864:20::42b]:42675) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fncT2-0007QV-FG for qemu-devel@nongnu.org; Thu, 09 Aug 2018 00:22:32 -0400 Received: by mail-pf1-x42b.google.com with SMTP id l9-v6so2185758pff.9 for ; Wed, 08 Aug 2018 21:22:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=ZP9LjvZ7RdzvXKxbJAUmEFUz9QE9Ls0QkL+ef9F8j6s=; b=Z+ScdScp02YmJHt9jgooXturoDGA1vtzi6icBgflm0nDxXiGjCB+HXJBjBJ+2Hdw3t cNnMV2IPft5lpmO/yeBykozCY5xnSikWwEqVnE6XqmumuIMEoAsJiZjh2hg24EN7y9gn VIUTSfDfLD3ap1kDufDk/7SmuJQOqeYy45qdE= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=ZP9LjvZ7RdzvXKxbJAUmEFUz9QE9Ls0QkL+ef9F8j6s=; b=ckYfIda46FMb3BsO34BSdZNHyu/F9RlvcZ8Br+YC7gTPqCIHLj4+bos4Yolog9x5Zy YCq23SSm7phIY0W9gs/EKwDgknx5WHwyjLIYyU3vClPU2InZ7DX4cQA5PeBg3l3di2vW V1spuxy1WCy1qR2QQmUfc+ENgo0yeONlCAyQCQdepjz9H3eFQ+93utnEhBeqwNOBkgLD zoUEReaq9R/Vcm1m8r34tLnH8o5MCjbRn85i6Eb7QR4aA3r8WZdQQos8vjXU4HxHrHm9 pS3lm1NyPWtC2mkJKtz2byaQNECWWYOhpqBIMA5reKKV9k4fhImtmMZCrigorm/eBgjb UcRQ== X-Gm-Message-State: AOUpUlFaa7e9WZ8IPVySUlfs//sJv7/Kr0S9iUd0Gno/hv2tnzRyKf1V 6n5n/Gv5xo3bOYWT7dYaNm92GJZLZw4= X-Received: by 2002:a63:d20e:: with SMTP id a14-v6mr525613pgg.226.1533788551226; Wed, 08 Aug 2018 21:22:31 -0700 (PDT) Received: from cloudburst.twiddle.net (97-113-8-179.tukw.qwest.net. [97.113.8.179]) by smtp.gmail.com with ESMTPSA id m30-v6sm7355799pff.121.2018.08.08.21.22.29 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Wed, 08 Aug 2018 21:22:30 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Date: Wed, 8 Aug 2018 21:22:02 -0700 Message-Id: <20180809042206.15726-17-richard.henderson@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180809042206.15726-1-richard.henderson@linaro.org> References: <20180809042206.15726-1-richard.henderson@linaro.org> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::42b Subject: [Qemu-devel] [PATCH 16/20] target/arm: Split contiguous stores for endianness X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: laurent.desnogues@gmail.com, peter.maydell@linaro.org, alex.bennee@linaro.org Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: "Qemu-devel" We can choose the endianness at translation time, rather than re-computing it at execution time. Signed-off-by: Richard Henderson --- target/arm/helper-sve.h | 48 +++++++++++++++++-------- target/arm/sve_helper.c | 11 ++++-- target/arm/translate-sve.c | 72 +++++++++++++++++++++++++++++--------- 3 files changed, 96 insertions(+), 35 deletions(-) -- 2.17.1 Reviewed-by: Philippe Mathieu-Daudé diff --git a/target/arm/helper-sve.h b/target/arm/helper-sve.h index 526caec8da..1ad043101a 100644 --- a/target/arm/helper-sve.h +++ b/target/arm/helper-sve.h @@ -1248,29 +1248,47 @@ DEF_HELPER_FLAGS_4(sve_st2bb_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) DEF_HELPER_FLAGS_4(sve_st3bb_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) DEF_HELPER_FLAGS_4(sve_st4bb_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) -DEF_HELPER_FLAGS_4(sve_st1hh_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) -DEF_HELPER_FLAGS_4(sve_st2hh_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) -DEF_HELPER_FLAGS_4(sve_st3hh_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) -DEF_HELPER_FLAGS_4(sve_st4hh_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) +DEF_HELPER_FLAGS_4(sve_st1hh_le_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) +DEF_HELPER_FLAGS_4(sve_st2hh_le_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) +DEF_HELPER_FLAGS_4(sve_st3hh_le_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) +DEF_HELPER_FLAGS_4(sve_st4hh_le_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) -DEF_HELPER_FLAGS_4(sve_st1ss_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) -DEF_HELPER_FLAGS_4(sve_st2ss_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) -DEF_HELPER_FLAGS_4(sve_st3ss_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) -DEF_HELPER_FLAGS_4(sve_st4ss_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) +DEF_HELPER_FLAGS_4(sve_st1hh_be_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) +DEF_HELPER_FLAGS_4(sve_st2hh_be_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) +DEF_HELPER_FLAGS_4(sve_st3hh_be_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) +DEF_HELPER_FLAGS_4(sve_st4hh_be_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) -DEF_HELPER_FLAGS_4(sve_st1dd_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) -DEF_HELPER_FLAGS_4(sve_st2dd_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) -DEF_HELPER_FLAGS_4(sve_st3dd_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) -DEF_HELPER_FLAGS_4(sve_st4dd_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) +DEF_HELPER_FLAGS_4(sve_st1ss_le_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) +DEF_HELPER_FLAGS_4(sve_st2ss_le_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) +DEF_HELPER_FLAGS_4(sve_st3ss_le_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) +DEF_HELPER_FLAGS_4(sve_st4ss_le_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) + +DEF_HELPER_FLAGS_4(sve_st1ss_be_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) +DEF_HELPER_FLAGS_4(sve_st2ss_be_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) +DEF_HELPER_FLAGS_4(sve_st3ss_be_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) +DEF_HELPER_FLAGS_4(sve_st4ss_be_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) + +DEF_HELPER_FLAGS_4(sve_st1dd_le_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) +DEF_HELPER_FLAGS_4(sve_st2dd_le_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) +DEF_HELPER_FLAGS_4(sve_st3dd_le_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) +DEF_HELPER_FLAGS_4(sve_st4dd_le_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) + +DEF_HELPER_FLAGS_4(sve_st1dd_be_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) +DEF_HELPER_FLAGS_4(sve_st2dd_be_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) +DEF_HELPER_FLAGS_4(sve_st3dd_be_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) +DEF_HELPER_FLAGS_4(sve_st4dd_be_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) DEF_HELPER_FLAGS_4(sve_st1bh_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) DEF_HELPER_FLAGS_4(sve_st1bs_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) DEF_HELPER_FLAGS_4(sve_st1bd_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) -DEF_HELPER_FLAGS_4(sve_st1hs_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) -DEF_HELPER_FLAGS_4(sve_st1hd_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) +DEF_HELPER_FLAGS_4(sve_st1hs_le_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) +DEF_HELPER_FLAGS_4(sve_st1hd_le_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) +DEF_HELPER_FLAGS_4(sve_st1hs_be_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) +DEF_HELPER_FLAGS_4(sve_st1hd_be_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) -DEF_HELPER_FLAGS_4(sve_st1sd_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) +DEF_HELPER_FLAGS_4(sve_st1sd_le_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) +DEF_HELPER_FLAGS_4(sve_st1sd_be_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) DEF_HELPER_FLAGS_6(sve_ldbsu_zsu, TCG_CALL_NO_WG, void, env, ptr, ptr, ptr, tl, i32) diff --git a/target/arm/sve_helper.c b/target/arm/sve_helper.c index 56e2f523c5..92c0e961a9 100644 --- a/target/arm/sve_helper.c +++ b/target/arm/sve_helper.c @@ -4940,12 +4940,17 @@ void __attribute__((flatten)) HELPER(sve_st##N##NAME##_r) \ } #define DO_STN_2(N, NAME, ESIZE, MSIZE) \ -void __attribute__((flatten)) HELPER(sve_st##N##NAME##_r) \ +void __attribute__((flatten)) HELPER(sve_st##N##NAME##_le_r) \ (CPUARMState *env, void *vg, target_ulong addr, uint32_t desc) \ { \ sve_st##N##_r(env, vg, addr, desc, GETPC(), ESIZE, MSIZE, \ - arm_cpu_data_is_big_endian(env) \ - ? sve_st1##NAME##_be_tlb : sve_st1##NAME##_le_tlb); \ + sve_st1##NAME##_le_tlb); \ +} \ +void __attribute__((flatten)) HELPER(sve_st##N##NAME##_be_r) \ + (CPUARMState *env, void *vg, target_ulong addr, uint32_t desc) \ +{ \ + sve_st##N##_r(env, vg, addr, desc, GETPC(), ESIZE, MSIZE, \ + sve_st1##NAME##_be_tlb); \ } DO_STN_1(1, bb, 1) diff --git a/target/arm/translate-sve.c b/target/arm/translate-sve.c index de12c01e7d..acb85731f8 100644 --- a/target/arm/translate-sve.c +++ b/target/arm/translate-sve.c @@ -4953,32 +4953,70 @@ static bool trans_LD1R_zpri(DisasContext *s, arg_rpri_load *a, uint32_t insn) static void do_st_zpa(DisasContext *s, int zt, int pg, TCGv_i64 addr, int msz, int esz, int nreg) { - static gen_helper_gvec_mem * const fn_single[4][4] = { - { gen_helper_sve_st1bb_r, gen_helper_sve_st1bh_r, - gen_helper_sve_st1bs_r, gen_helper_sve_st1bd_r }, - { NULL, gen_helper_sve_st1hh_r, - gen_helper_sve_st1hs_r, gen_helper_sve_st1hd_r }, - { NULL, NULL, - gen_helper_sve_st1ss_r, gen_helper_sve_st1sd_r }, - { NULL, NULL, NULL, gen_helper_sve_st1dd_r }, + static gen_helper_gvec_mem * const fn_single[2][4][4] = { + { { gen_helper_sve_st1bb_r, + gen_helper_sve_st1bh_r, + gen_helper_sve_st1bs_r, + gen_helper_sve_st1bd_r }, + { NULL, + gen_helper_sve_st1hh_le_r, + gen_helper_sve_st1hs_le_r, + gen_helper_sve_st1hd_le_r }, + { NULL, NULL, + gen_helper_sve_st1ss_le_r, + gen_helper_sve_st1sd_le_r }, + { NULL, NULL, NULL, + gen_helper_sve_st1dd_le_r } }, + { { gen_helper_sve_st1bb_r, + gen_helper_sve_st1bh_r, + gen_helper_sve_st1bs_r, + gen_helper_sve_st1bd_r }, + { NULL, + gen_helper_sve_st1hh_be_r, + gen_helper_sve_st1hs_be_r, + gen_helper_sve_st1hd_be_r }, + { NULL, NULL, + gen_helper_sve_st1ss_be_r, + gen_helper_sve_st1sd_be_r }, + { NULL, NULL, NULL, + gen_helper_sve_st1dd_be_r } }, }; - static gen_helper_gvec_mem * const fn_multiple[3][4] = { - { gen_helper_sve_st2bb_r, gen_helper_sve_st2hh_r, - gen_helper_sve_st2ss_r, gen_helper_sve_st2dd_r }, - { gen_helper_sve_st3bb_r, gen_helper_sve_st3hh_r, - gen_helper_sve_st3ss_r, gen_helper_sve_st3dd_r }, - { gen_helper_sve_st4bb_r, gen_helper_sve_st4hh_r, - gen_helper_sve_st4ss_r, gen_helper_sve_st4dd_r }, + static gen_helper_gvec_mem * const fn_multiple[2][3][4] = { + { { gen_helper_sve_st2bb_r, + gen_helper_sve_st2hh_le_r, + gen_helper_sve_st2ss_le_r, + gen_helper_sve_st2dd_le_r }, + { gen_helper_sve_st3bb_r, + gen_helper_sve_st3hh_le_r, + gen_helper_sve_st3ss_le_r, + gen_helper_sve_st3dd_le_r }, + { gen_helper_sve_st4bb_r, + gen_helper_sve_st4hh_le_r, + gen_helper_sve_st4ss_le_r, + gen_helper_sve_st4dd_le_r } }, + { { gen_helper_sve_st2bb_r, + gen_helper_sve_st2hh_be_r, + gen_helper_sve_st2ss_be_r, + gen_helper_sve_st2dd_be_r }, + { gen_helper_sve_st3bb_r, + gen_helper_sve_st3hh_be_r, + gen_helper_sve_st3ss_be_r, + gen_helper_sve_st3dd_be_r }, + { gen_helper_sve_st4bb_r, + gen_helper_sve_st4hh_be_r, + gen_helper_sve_st4ss_be_r, + gen_helper_sve_st4dd_be_r } }, }; gen_helper_gvec_mem *fn; + int be = s->be_data == MO_BE; if (nreg == 0) { /* ST1 */ - fn = fn_single[msz][esz]; + fn = fn_single[be][msz][esz]; } else { /* ST2, ST3, ST4 -- msz == esz, enforced by encoding */ assert(msz == esz); - fn = fn_multiple[nreg - 1][msz]; + fn = fn_multiple[be][nreg - 1][msz]; } assert(fn != NULL); do_mem_zpa(s, zt, pg, addr, fn); From patchwork Thu Aug 9 04:22:03 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 143674 Delivered-To: patch@linaro.org Received: by 2002:a2e:9754:0:0:0:0:0 with SMTP id f20-v6csp1613630ljj; Wed, 8 Aug 2018 21:38:59 -0700 (PDT) X-Google-Smtp-Source: AA+uWPzT06Pcoezz9iUkOFFlaDCTNVIab4WddpzNkrnCdfm2bsRiYj4utTZO4fkUXJF2/T5sTeve X-Received: by 2002:ac8:3644:: with SMTP id n4-v6mr463382qtb.323.1533789538906; Wed, 08 Aug 2018 21:38:58 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1533789538; cv=none; d=google.com; s=arc-20160816; b=0XJu3+OtVEZcbhyFbRljBBbj+3Q7lHB2Zg5umaKuC9gTP6nSMtgwJ8gKaMfpgrvc0H uw/FoLZzhxYnLfCQgeYenC+zNDFIQ9bYqT6slkpYcsBvwpPSKedsczlnQIIMxP1hDPR4 FfxNMR5PVw3ZLqHuXrgfkE18ApNEY+Mzs+nPSb4QT5dzNdHuKLfUB89/WAuvF90dgXMI yDYvCgJKqkZnQz+us3uXS3ndUWY0uW4eCPaFipb2fY4gnOnSbjABocppXIEDauI2QtPp zFnjwXcXD0IejoPABB0bx1wo/g+JXcnhPaJVihctgJSzYm7yefc7iW63iMEpTQyV79/K toqg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:cc:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:subject:references:in-reply-to :message-id:date:to:from:dkim-signature:arc-authentication-results; bh=MYhgJucgZ+U6nSJDfeKc+2O2o0Gq7XMWGUI3jo+jjRM=; b=hArZnUbWI4YfqwjbascjRWt2VoJgfpCcla1MFzmygL7VKlwND9cWa72qm5W8hmT5eH n1YPdZGBMoysaZ1uC42AWC+y24qs/fyv4OPTIoCpKO6AXPD2A5nGEa3P1a6Uhs8tblhE 7yO/pM4s85uKOz7cImnAQqIRttBamfyRYSFvzDkoWz9pw4cC+KBf14hvJWggO9+gTc17 dtu7htxhUd2Ob4DHTVK+W/DuVD99Ak+6wSsJO1fgJj9SCncXvYgLd6+w/+Od9//nCJ+e t7BiIJCVn5wk2k+Z7tGcQk/LeFRToGmPW6ZdniswNGw786dWgnzEZVBU+RKaOosOG3i2 c1lA== ARC-Authentication-Results: i=1; mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=AoKV7ib9; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [2001:4830:134:3::11]) by mx.google.com with ESMTPS id y15-v6si4961816qto.192.2018.08.08.21.38.58 for (version=TLS1 cipher=AES128-SHA bits=128/128); Wed, 08 Aug 2018 21:38:58 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) client-ip=2001:4830:134:3::11; Authentication-Results: mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=AoKV7ib9; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1]:47307 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fnciw-0006D4-Ca for patch@linaro.org; Thu, 09 Aug 2018 00:38:58 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:54139) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fncT7-000128-N2 for qemu-devel@nongnu.org; Thu, 09 Aug 2018 00:22:40 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fncT4-0007Su-ON for qemu-devel@nongnu.org; Thu, 09 Aug 2018 00:22:37 -0400 Received: from mail-pf1-x432.google.com ([2607:f8b0:4864:20::432]:34670) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fncT4-0007Rl-DS for qemu-devel@nongnu.org; Thu, 09 Aug 2018 00:22:34 -0400 Received: by mail-pf1-x432.google.com with SMTP id k19-v6so2208678pfi.1 for ; Wed, 08 Aug 2018 21:22:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=MYhgJucgZ+U6nSJDfeKc+2O2o0Gq7XMWGUI3jo+jjRM=; b=AoKV7ib9qW0+WTnSISN/3yJNKiMNgj122pJhHB1uD1mlsng4halNLOxt1cENI7RVVS pl+Pd2EgvaSx5ejdDtCMIifIeqmsUNxRLlKbre1qlcRiKS5N9ZLkUtQa84D0prqytqmV xJ9S04Snx4HYbQtljsVkfuI7p6sH3Dw4Rc/+0= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=MYhgJucgZ+U6nSJDfeKc+2O2o0Gq7XMWGUI3jo+jjRM=; b=ihz7lhOlsh0edosquR4icfAuEbvmgyZZAsJvT5QN68gwPGi8VtjVITt07Pbyo2oZIy H4C4JjRnC3cSg3r7mR8JdJMgv3WbRrtv8BCgmDTFBUUO+lgTkYRd9lAhOlMkzxgp0iwN BAD2UR9KSpnoU+NeGPuw3u5otcSlVP4tnZkpepCZIbdN3PDWnRmlZohcD9Sh5M4ogRgf os2fcRQ77orgCP3s2T4dpH13VvfjPA1dCTa+rY99CjiyLnErUCLPwFwT3n13HQGi+NGZ F+Fu+XN6yiG6N9rmF5b7JfZjNqH6ucgAI/zCLKKaoyfy9UfmFBgpIJRE7TGVdvg6iXJh bDMA== X-Gm-Message-State: AOUpUlGSncQz1tjfrD3XXudr7JXL0U/o6L60I0CuFGvdOe3CF0naQHFo FxgVXVQBZ1PC2uplJSdqp+gXDqKC3Ys= X-Received: by 2002:a65:5907:: with SMTP id f7-v6mr497734pgu.83.1533788552598; Wed, 08 Aug 2018 21:22:32 -0700 (PDT) Received: from cloudburst.twiddle.net (97-113-8-179.tukw.qwest.net. [97.113.8.179]) by smtp.gmail.com with ESMTPSA id m30-v6sm7355799pff.121.2018.08.08.21.22.31 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Wed, 08 Aug 2018 21:22:31 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Date: Wed, 8 Aug 2018 21:22:03 -0700 Message-Id: <20180809042206.15726-18-richard.henderson@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180809042206.15726-1-richard.henderson@linaro.org> References: <20180809042206.15726-1-richard.henderson@linaro.org> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::432 Subject: [Qemu-devel] [PATCH 17/20] target/arm: Rewrite vector gather loads X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: laurent.desnogues@gmail.com, peter.maydell@linaro.org, alex.bennee@linaro.org Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: "Qemu-devel" This fixes the endianness problem for softmmu, and does move the main loop out of a macro and into an inlined function. Signed-off-by: Richard Henderson --- target/arm/helper-sve.h | 84 +++++++++---- target/arm/sve_helper.c | 218 +++++++++++++++++++++++---------- target/arm/translate-sve.c | 244 +++++++++++++++++++++++++------------ 3 files changed, 380 insertions(+), 166 deletions(-) -- 2.17.1 Reviewed-by: Peter Maydell diff --git a/target/arm/helper-sve.h b/target/arm/helper-sve.h index 1ad043101a..49d1c09e30 100644 --- a/target/arm/helper-sve.h +++ b/target/arm/helper-sve.h @@ -1292,69 +1292,111 @@ DEF_HELPER_FLAGS_4(sve_st1sd_be_r, TCG_CALL_NO_WG, void, env, ptr, tl, i32) DEF_HELPER_FLAGS_6(sve_ldbsu_zsu, TCG_CALL_NO_WG, void, env, ptr, ptr, ptr, tl, i32) -DEF_HELPER_FLAGS_6(sve_ldhsu_zsu, TCG_CALL_NO_WG, +DEF_HELPER_FLAGS_6(sve_ldhsu_le_zsu, TCG_CALL_NO_WG, void, env, ptr, ptr, ptr, tl, i32) -DEF_HELPER_FLAGS_6(sve_ldssu_zsu, TCG_CALL_NO_WG, +DEF_HELPER_FLAGS_6(sve_ldhsu_be_zsu, TCG_CALL_NO_WG, + void, env, ptr, ptr, ptr, tl, i32) +DEF_HELPER_FLAGS_6(sve_ldss_le_zsu, TCG_CALL_NO_WG, + void, env, ptr, ptr, ptr, tl, i32) +DEF_HELPER_FLAGS_6(sve_ldss_be_zsu, TCG_CALL_NO_WG, void, env, ptr, ptr, ptr, tl, i32) DEF_HELPER_FLAGS_6(sve_ldbss_zsu, TCG_CALL_NO_WG, void, env, ptr, ptr, ptr, tl, i32) -DEF_HELPER_FLAGS_6(sve_ldhss_zsu, TCG_CALL_NO_WG, +DEF_HELPER_FLAGS_6(sve_ldhss_le_zsu, TCG_CALL_NO_WG, + void, env, ptr, ptr, ptr, tl, i32) +DEF_HELPER_FLAGS_6(sve_ldhss_be_zsu, TCG_CALL_NO_WG, void, env, ptr, ptr, ptr, tl, i32) DEF_HELPER_FLAGS_6(sve_ldbsu_zss, TCG_CALL_NO_WG, void, env, ptr, ptr, ptr, tl, i32) -DEF_HELPER_FLAGS_6(sve_ldhsu_zss, TCG_CALL_NO_WG, +DEF_HELPER_FLAGS_6(sve_ldhsu_le_zss, TCG_CALL_NO_WG, void, env, ptr, ptr, ptr, tl, i32) -DEF_HELPER_FLAGS_6(sve_ldssu_zss, TCG_CALL_NO_WG, +DEF_HELPER_FLAGS_6(sve_ldhsu_be_zss, TCG_CALL_NO_WG, + void, env, ptr, ptr, ptr, tl, i32) +DEF_HELPER_FLAGS_6(sve_ldss_le_zss, TCG_CALL_NO_WG, + void, env, ptr, ptr, ptr, tl, i32) +DEF_HELPER_FLAGS_6(sve_ldss_be_zss, TCG_CALL_NO_WG, void, env, ptr, ptr, ptr, tl, i32) DEF_HELPER_FLAGS_6(sve_ldbss_zss, TCG_CALL_NO_WG, void, env, ptr, ptr, ptr, tl, i32) -DEF_HELPER_FLAGS_6(sve_ldhss_zss, TCG_CALL_NO_WG, +DEF_HELPER_FLAGS_6(sve_ldhss_le_zss, TCG_CALL_NO_WG, + void, env, ptr, ptr, ptr, tl, i32) +DEF_HELPER_FLAGS_6(sve_ldhss_be_zss, TCG_CALL_NO_WG, void, env, ptr, ptr, ptr, tl, i32) DEF_HELPER_FLAGS_6(sve_ldbdu_zsu, TCG_CALL_NO_WG, void, env, ptr, ptr, ptr, tl, i32) -DEF_HELPER_FLAGS_6(sve_ldhdu_zsu, TCG_CALL_NO_WG, +DEF_HELPER_FLAGS_6(sve_ldhdu_le_zsu, TCG_CALL_NO_WG, void, env, ptr, ptr, ptr, tl, i32) -DEF_HELPER_FLAGS_6(sve_ldsdu_zsu, TCG_CALL_NO_WG, +DEF_HELPER_FLAGS_6(sve_ldhdu_be_zsu, TCG_CALL_NO_WG, void, env, ptr, ptr, ptr, tl, i32) -DEF_HELPER_FLAGS_6(sve_ldddu_zsu, TCG_CALL_NO_WG, +DEF_HELPER_FLAGS_6(sve_ldsdu_le_zsu, TCG_CALL_NO_WG, + void, env, ptr, ptr, ptr, tl, i32) +DEF_HELPER_FLAGS_6(sve_ldsdu_be_zsu, TCG_CALL_NO_WG, + void, env, ptr, ptr, ptr, tl, i32) +DEF_HELPER_FLAGS_6(sve_lddd_le_zsu, TCG_CALL_NO_WG, + void, env, ptr, ptr, ptr, tl, i32) +DEF_HELPER_FLAGS_6(sve_lddd_be_zsu, TCG_CALL_NO_WG, void, env, ptr, ptr, ptr, tl, i32) DEF_HELPER_FLAGS_6(sve_ldbds_zsu, TCG_CALL_NO_WG, void, env, ptr, ptr, ptr, tl, i32) -DEF_HELPER_FLAGS_6(sve_ldhds_zsu, TCG_CALL_NO_WG, +DEF_HELPER_FLAGS_6(sve_ldhds_le_zsu, TCG_CALL_NO_WG, void, env, ptr, ptr, ptr, tl, i32) -DEF_HELPER_FLAGS_6(sve_ldsds_zsu, TCG_CALL_NO_WG, +DEF_HELPER_FLAGS_6(sve_ldhds_be_zsu, TCG_CALL_NO_WG, + void, env, ptr, ptr, ptr, tl, i32) +DEF_HELPER_FLAGS_6(sve_ldsds_le_zsu, TCG_CALL_NO_WG, + void, env, ptr, ptr, ptr, tl, i32) +DEF_HELPER_FLAGS_6(sve_ldsds_be_zsu, TCG_CALL_NO_WG, void, env, ptr, ptr, ptr, tl, i32) DEF_HELPER_FLAGS_6(sve_ldbdu_zss, TCG_CALL_NO_WG, void, env, ptr, ptr, ptr, tl, i32) -DEF_HELPER_FLAGS_6(sve_ldhdu_zss, TCG_CALL_NO_WG, +DEF_HELPER_FLAGS_6(sve_ldhdu_le_zss, TCG_CALL_NO_WG, void, env, ptr, ptr, ptr, tl, i32) -DEF_HELPER_FLAGS_6(sve_ldsdu_zss, TCG_CALL_NO_WG, +DEF_HELPER_FLAGS_6(sve_ldhdu_be_zss, TCG_CALL_NO_WG, void, env, ptr, ptr, ptr, tl, i32) -DEF_HELPER_FLAGS_6(sve_ldddu_zss, TCG_CALL_NO_WG, +DEF_HELPER_FLAGS_6(sve_ldsdu_le_zss, TCG_CALL_NO_WG, + void, env, ptr, ptr, ptr, tl, i32) +DEF_HELPER_FLAGS_6(sve_ldsdu_be_zss, TCG_CALL_NO_WG, + void, env, ptr, ptr, ptr, tl, i32) +DEF_HELPER_FLAGS_6(sve_lddd_le_zss, TCG_CALL_NO_WG, + void, env, ptr, ptr, ptr, tl, i32) +DEF_HELPER_FLAGS_6(sve_lddd_be_zss, TCG_CALL_NO_WG, void, env, ptr, ptr, ptr, tl, i32) DEF_HELPER_FLAGS_6(sve_ldbds_zss, TCG_CALL_NO_WG, void, env, ptr, ptr, ptr, tl, i32) -DEF_HELPER_FLAGS_6(sve_ldhds_zss, TCG_CALL_NO_WG, +DEF_HELPER_FLAGS_6(sve_ldhds_le_zss, TCG_CALL_NO_WG, void, env, ptr, ptr, ptr, tl, i32) -DEF_HELPER_FLAGS_6(sve_ldsds_zss, TCG_CALL_NO_WG, +DEF_HELPER_FLAGS_6(sve_ldhds_be_zss, TCG_CALL_NO_WG, + void, env, ptr, ptr, ptr, tl, i32) +DEF_HELPER_FLAGS_6(sve_ldsds_le_zss, TCG_CALL_NO_WG, + void, env, ptr, ptr, ptr, tl, i32) +DEF_HELPER_FLAGS_6(sve_ldsds_be_zss, TCG_CALL_NO_WG, void, env, ptr, ptr, ptr, tl, i32) DEF_HELPER_FLAGS_6(sve_ldbdu_zd, TCG_CALL_NO_WG, void, env, ptr, ptr, ptr, tl, i32) -DEF_HELPER_FLAGS_6(sve_ldhdu_zd, TCG_CALL_NO_WG, +DEF_HELPER_FLAGS_6(sve_ldhdu_le_zd, TCG_CALL_NO_WG, void, env, ptr, ptr, ptr, tl, i32) -DEF_HELPER_FLAGS_6(sve_ldsdu_zd, TCG_CALL_NO_WG, +DEF_HELPER_FLAGS_6(sve_ldhdu_be_zd, TCG_CALL_NO_WG, void, env, ptr, ptr, ptr, tl, i32) -DEF_HELPER_FLAGS_6(sve_ldddu_zd, TCG_CALL_NO_WG, +DEF_HELPER_FLAGS_6(sve_ldsdu_le_zd, TCG_CALL_NO_WG, + void, env, ptr, ptr, ptr, tl, i32) +DEF_HELPER_FLAGS_6(sve_ldsdu_be_zd, TCG_CALL_NO_WG, + void, env, ptr, ptr, ptr, tl, i32) +DEF_HELPER_FLAGS_6(sve_lddd_le_zd, TCG_CALL_NO_WG, + void, env, ptr, ptr, ptr, tl, i32) +DEF_HELPER_FLAGS_6(sve_lddd_be_zd, TCG_CALL_NO_WG, void, env, ptr, ptr, ptr, tl, i32) DEF_HELPER_FLAGS_6(sve_ldbds_zd, TCG_CALL_NO_WG, void, env, ptr, ptr, ptr, tl, i32) -DEF_HELPER_FLAGS_6(sve_ldhds_zd, TCG_CALL_NO_WG, +DEF_HELPER_FLAGS_6(sve_ldhds_le_zd, TCG_CALL_NO_WG, void, env, ptr, ptr, ptr, tl, i32) -DEF_HELPER_FLAGS_6(sve_ldsds_zd, TCG_CALL_NO_WG, +DEF_HELPER_FLAGS_6(sve_ldhds_be_zd, TCG_CALL_NO_WG, + void, env, ptr, ptr, ptr, tl, i32) +DEF_HELPER_FLAGS_6(sve_ldsds_le_zd, TCG_CALL_NO_WG, + void, env, ptr, ptr, ptr, tl, i32) +DEF_HELPER_FLAGS_6(sve_ldsds_be_zd, TCG_CALL_NO_WG, void, env, ptr, ptr, ptr, tl, i32) DEF_HELPER_FLAGS_6(sve_ldffbsu_zsu, TCG_CALL_NO_WG, diff --git a/target/arm/sve_helper.c b/target/arm/sve_helper.c index 92c0e961a9..76d3f021e4 100644 --- a/target/arm/sve_helper.c +++ b/target/arm/sve_helper.c @@ -4984,80 +4984,166 @@ DO_STN_2(4, dd, 8, 8) /* Loads with a vector index. */ -#define DO_LD1_ZPZ_S(NAME, TYPEI, TYPEM, FN) \ -void HELPER(NAME)(CPUARMState *env, void *vd, void *vg, void *vm, \ - target_ulong base, uint32_t desc) \ -{ \ - intptr_t i, oprsz = simd_oprsz(desc); \ - unsigned scale = simd_data(desc); \ - uintptr_t ra = GETPC(); \ - for (i = 0; i < oprsz; ) { \ - uint16_t pg = *(uint16_t *)(vg + H1_2(i >> 3)); \ - do { \ - TYPEM m = 0; \ - if (pg & 1) { \ - target_ulong off = *(TYPEI *)(vm + H1_4(i)); \ - m = FN(env, base + (off << scale), ra); \ - } \ - *(uint32_t *)(vd + H1_4(i)) = m; \ - i += 4, pg >>= 4; \ - } while (i & 15); \ - } \ +typedef target_ulong zreg_off_fn(void *reg, intptr_t reg_ofs); + +static target_ulong off_zsu_s(void *reg, intptr_t reg_ofs) +{ + return *(uint32_t *)(reg + H1_4(reg_ofs)); } -#define DO_LD1_ZPZ_D(NAME, TYPEI, TYPEM, FN) \ -void HELPER(NAME)(CPUARMState *env, void *vd, void *vg, void *vm, \ - target_ulong base, uint32_t desc) \ -{ \ - intptr_t i, oprsz = simd_oprsz(desc) / 8; \ - unsigned scale = simd_data(desc); \ - uintptr_t ra = GETPC(); \ - uint64_t *d = vd, *m = vm; uint8_t *pg = vg; \ - for (i = 0; i < oprsz; i++) { \ - TYPEM mm = 0; \ - if (pg[H1(i)] & 1) { \ - target_ulong off = (TYPEI)m[i]; \ - mm = FN(env, base + (off << scale), ra); \ - } \ - d[i] = mm; \ - } \ +static target_ulong off_zss_s(void *reg, intptr_t reg_ofs) +{ + return *(int32_t *)(reg + H1_4(reg_ofs)); } -DO_LD1_ZPZ_S(sve_ldbsu_zsu, uint32_t, uint8_t, cpu_ldub_data_ra) -DO_LD1_ZPZ_S(sve_ldhsu_zsu, uint32_t, uint16_t, cpu_lduw_data_ra) -DO_LD1_ZPZ_S(sve_ldssu_zsu, uint32_t, uint32_t, cpu_ldl_data_ra) -DO_LD1_ZPZ_S(sve_ldbss_zsu, uint32_t, int8_t, cpu_ldub_data_ra) -DO_LD1_ZPZ_S(sve_ldhss_zsu, uint32_t, int16_t, cpu_lduw_data_ra) +static target_ulong off_zsu_d(void *reg, intptr_t reg_ofs) +{ + return (uint32_t)*(uint64_t *)(reg + reg_ofs); +} -DO_LD1_ZPZ_S(sve_ldbsu_zss, int32_t, uint8_t, cpu_ldub_data_ra) -DO_LD1_ZPZ_S(sve_ldhsu_zss, int32_t, uint16_t, cpu_lduw_data_ra) -DO_LD1_ZPZ_S(sve_ldssu_zss, int32_t, uint32_t, cpu_ldl_data_ra) -DO_LD1_ZPZ_S(sve_ldbss_zss, int32_t, int8_t, cpu_ldub_data_ra) -DO_LD1_ZPZ_S(sve_ldhss_zss, int32_t, int16_t, cpu_lduw_data_ra) +static target_ulong off_zss_d(void *reg, intptr_t reg_ofs) +{ + return (int32_t)*(uint64_t *)(reg + reg_ofs); +} -DO_LD1_ZPZ_D(sve_ldbdu_zsu, uint32_t, uint8_t, cpu_ldub_data_ra) -DO_LD1_ZPZ_D(sve_ldhdu_zsu, uint32_t, uint16_t, cpu_lduw_data_ra) -DO_LD1_ZPZ_D(sve_ldsdu_zsu, uint32_t, uint32_t, cpu_ldl_data_ra) -DO_LD1_ZPZ_D(sve_ldddu_zsu, uint32_t, uint64_t, cpu_ldq_data_ra) -DO_LD1_ZPZ_D(sve_ldbds_zsu, uint32_t, int8_t, cpu_ldub_data_ra) -DO_LD1_ZPZ_D(sve_ldhds_zsu, uint32_t, int16_t, cpu_lduw_data_ra) -DO_LD1_ZPZ_D(sve_ldsds_zsu, uint32_t, int32_t, cpu_ldl_data_ra) +static target_ulong off_zd_d(void *reg, intptr_t reg_ofs) +{ + return *(uint64_t *)(reg + reg_ofs); +} -DO_LD1_ZPZ_D(sve_ldbdu_zss, int32_t, uint8_t, cpu_ldub_data_ra) -DO_LD1_ZPZ_D(sve_ldhdu_zss, int32_t, uint16_t, cpu_lduw_data_ra) -DO_LD1_ZPZ_D(sve_ldsdu_zss, int32_t, uint32_t, cpu_ldl_data_ra) -DO_LD1_ZPZ_D(sve_ldddu_zss, int32_t, uint64_t, cpu_ldq_data_ra) -DO_LD1_ZPZ_D(sve_ldbds_zss, int32_t, int8_t, cpu_ldub_data_ra) -DO_LD1_ZPZ_D(sve_ldhds_zss, int32_t, int16_t, cpu_lduw_data_ra) -DO_LD1_ZPZ_D(sve_ldsds_zss, int32_t, int32_t, cpu_ldl_data_ra) +static void sve_ld1_zs(CPUARMState *env, void *vd, void *vg, void *vm, + target_ulong base, uint32_t desc, uintptr_t ra, + zreg_off_fn *off_fn, sve_ld1_tlb_fn *tlb_fn) +{ + const int mmu_idx = cpu_mmu_index(env, false); + intptr_t i, oprsz = simd_oprsz(desc); + unsigned scale = simd_data(desc); + ARMVectorReg scratch = { }; -DO_LD1_ZPZ_D(sve_ldbdu_zd, uint64_t, uint8_t, cpu_ldub_data_ra) -DO_LD1_ZPZ_D(sve_ldhdu_zd, uint64_t, uint16_t, cpu_lduw_data_ra) -DO_LD1_ZPZ_D(sve_ldsdu_zd, uint64_t, uint32_t, cpu_ldl_data_ra) -DO_LD1_ZPZ_D(sve_ldddu_zd, uint64_t, uint64_t, cpu_ldq_data_ra) -DO_LD1_ZPZ_D(sve_ldbds_zd, uint64_t, int8_t, cpu_ldub_data_ra) -DO_LD1_ZPZ_D(sve_ldhds_zd, uint64_t, int16_t, cpu_lduw_data_ra) -DO_LD1_ZPZ_D(sve_ldsds_zd, uint64_t, int32_t, cpu_ldl_data_ra) + set_helper_retaddr(ra); + for (i = 0; i < oprsz; ) { + uint16_t pg = *(uint16_t *)(vg + H1_2(i >> 3)); + do { + if (pg & 1) { + target_ulong off = off_fn(vm, i); + tlb_fn(env, &scratch, i, base + (off << scale), mmu_idx, ra); + } + i += 4, pg >>= 4; + } while (i & 15); + } + set_helper_retaddr(0); + + /* Wait until all exceptions have been raised to write back. */ + memcpy(vd, &scratch, oprsz); +} + +static void sve_ld1_zd(CPUARMState *env, void *vd, void *vg, void *vm, + target_ulong base, uint32_t desc, uintptr_t ra, + zreg_off_fn *off_fn, sve_ld1_tlb_fn *tlb_fn) +{ + const int mmu_idx = cpu_mmu_index(env, false); + intptr_t i, oprsz = simd_oprsz(desc) / 8; + unsigned scale = simd_data(desc); + ARMVectorReg scratch = { }; + + set_helper_retaddr(ra); + for (i = 0; i < oprsz; i++) { + uint8_t pg = *(uint8_t *)(vg + H1(i)); + if (pg & 1) { + target_ulong off = off_fn(vm, i * 8); + tlb_fn(env, &scratch, i * 8, base + (off << scale), mmu_idx, ra); + } + } + set_helper_retaddr(0); + + /* Wait until all exceptions have been raised to write back. */ + memcpy(vd, &scratch, oprsz * 8); +} + +#define DO_LD1_ZPZ_S(MEM, OFS) \ +void __attribute__((flatten)) HELPER(sve_ld##MEM##_##OFS) \ + (CPUARMState *env, void *vd, void *vg, void *vm, \ + target_ulong base, uint32_t desc) \ +{ \ + sve_ld1_zs(env, vd, vg, vm, base, desc, GETPC(), \ + off_##OFS##_s, sve_ld1##MEM##_tlb); \ +} + +#define DO_LD1_ZPZ_D(MEM, OFS) \ +void __attribute__((flatten)) HELPER(sve_ld##MEM##_##OFS) \ + (CPUARMState *env, void *vd, void *vg, void *vm, \ + target_ulong base, uint32_t desc) \ +{ \ + sve_ld1_zd(env, vd, vg, vm, base, desc, GETPC(), \ + off_##OFS##_d, sve_ld1##MEM##_tlb); \ +} + +DO_LD1_ZPZ_S(bsu, zsu) +DO_LD1_ZPZ_S(bsu, zss) +DO_LD1_ZPZ_D(bdu, zsu) +DO_LD1_ZPZ_D(bdu, zss) +DO_LD1_ZPZ_D(bdu, zd) + +DO_LD1_ZPZ_S(bss, zsu) +DO_LD1_ZPZ_S(bss, zss) +DO_LD1_ZPZ_D(bds, zsu) +DO_LD1_ZPZ_D(bds, zss) +DO_LD1_ZPZ_D(bds, zd) + +DO_LD1_ZPZ_S(hsu_le, zsu) +DO_LD1_ZPZ_S(hsu_le, zss) +DO_LD1_ZPZ_D(hdu_le, zsu) +DO_LD1_ZPZ_D(hdu_le, zss) +DO_LD1_ZPZ_D(hdu_le, zd) + +DO_LD1_ZPZ_S(hsu_be, zsu) +DO_LD1_ZPZ_S(hsu_be, zss) +DO_LD1_ZPZ_D(hdu_be, zsu) +DO_LD1_ZPZ_D(hdu_be, zss) +DO_LD1_ZPZ_D(hdu_be, zd) + +DO_LD1_ZPZ_S(hss_le, zsu) +DO_LD1_ZPZ_S(hss_le, zss) +DO_LD1_ZPZ_D(hds_le, zsu) +DO_LD1_ZPZ_D(hds_le, zss) +DO_LD1_ZPZ_D(hds_le, zd) + +DO_LD1_ZPZ_S(hss_be, zsu) +DO_LD1_ZPZ_S(hss_be, zss) +DO_LD1_ZPZ_D(hds_be, zsu) +DO_LD1_ZPZ_D(hds_be, zss) +DO_LD1_ZPZ_D(hds_be, zd) + +DO_LD1_ZPZ_S(ss_le, zsu) +DO_LD1_ZPZ_S(ss_le, zss) +DO_LD1_ZPZ_D(sdu_le, zsu) +DO_LD1_ZPZ_D(sdu_le, zss) +DO_LD1_ZPZ_D(sdu_le, zd) + +DO_LD1_ZPZ_S(ss_be, zsu) +DO_LD1_ZPZ_S(ss_be, zss) +DO_LD1_ZPZ_D(sdu_be, zsu) +DO_LD1_ZPZ_D(sdu_be, zss) +DO_LD1_ZPZ_D(sdu_be, zd) + +DO_LD1_ZPZ_D(sds_le, zsu) +DO_LD1_ZPZ_D(sds_le, zss) +DO_LD1_ZPZ_D(sds_le, zd) + +DO_LD1_ZPZ_D(sds_be, zsu) +DO_LD1_ZPZ_D(sds_be, zss) +DO_LD1_ZPZ_D(sds_be, zd) + +DO_LD1_ZPZ_D(dd_le, zsu) +DO_LD1_ZPZ_D(dd_le, zss) +DO_LD1_ZPZ_D(dd_le, zd) + +DO_LD1_ZPZ_D(dd_be, zsu) +DO_LD1_ZPZ_D(dd_be, zss) +DO_LD1_ZPZ_D(dd_be, zd) + +#undef DO_LD1_ZPZ_S +#undef DO_LD1_ZPZ_D /* First fault loads with a vector index. */ diff --git a/target/arm/translate-sve.c b/target/arm/translate-sve.c index acb85731f8..d4d7e9d3ae 100644 --- a/target/arm/translate-sve.c +++ b/target/arm/translate-sve.c @@ -5077,91 +5077,176 @@ static void do_mem_zpz(DisasContext *s, int zt, int pg, int zm, int scale, tcg_temp_free_i32(desc); } -/* Indexed by [ff][xs][u][msz]. */ -static gen_helper_gvec_mem_scatter * const gather_load_fn32[2][2][2][3] = { - { { { gen_helper_sve_ldbss_zsu, - gen_helper_sve_ldhss_zsu, - NULL, }, - { gen_helper_sve_ldbsu_zsu, - gen_helper_sve_ldhsu_zsu, - gen_helper_sve_ldssu_zsu, } }, - { { gen_helper_sve_ldbss_zss, - gen_helper_sve_ldhss_zss, - NULL, }, - { gen_helper_sve_ldbsu_zss, - gen_helper_sve_ldhsu_zss, - gen_helper_sve_ldssu_zss, } } }, +/* Indexed by [be][ff][xs][u][msz]. */ +static gen_helper_gvec_mem_scatter * const gather_load_fn32[2][2][2][2][3] = { + /* Little-endian */ + { { { { gen_helper_sve_ldbss_zsu, + gen_helper_sve_ldhss_le_zsu, + NULL, }, + { gen_helper_sve_ldbsu_zsu, + gen_helper_sve_ldhsu_le_zsu, + gen_helper_sve_ldss_le_zsu, } }, + { { gen_helper_sve_ldbss_zss, + gen_helper_sve_ldhss_le_zss, + NULL, }, + { gen_helper_sve_ldbsu_zss, + gen_helper_sve_ldhsu_le_zss, + gen_helper_sve_ldss_le_zss, } } }, - { { { gen_helper_sve_ldffbss_zsu, - gen_helper_sve_ldffhss_zsu, - NULL, }, - { gen_helper_sve_ldffbsu_zsu, - gen_helper_sve_ldffhsu_zsu, - gen_helper_sve_ldffssu_zsu, } }, - { { gen_helper_sve_ldffbss_zss, - gen_helper_sve_ldffhss_zss, - NULL, }, - { gen_helper_sve_ldffbsu_zss, - gen_helper_sve_ldffhsu_zss, - gen_helper_sve_ldffssu_zss, } } } + /* First-fault */ + { { { gen_helper_sve_ldffbss_zsu, + gen_helper_sve_ldffhss_zsu, + NULL, }, + { gen_helper_sve_ldffbsu_zsu, + gen_helper_sve_ldffhsu_zsu, + gen_helper_sve_ldffssu_zsu, } }, + { { gen_helper_sve_ldffbss_zss, + gen_helper_sve_ldffhss_zss, + NULL, }, + { gen_helper_sve_ldffbsu_zss, + gen_helper_sve_ldffhsu_zss, + gen_helper_sve_ldffssu_zss, } } } }, + + /* Big-endian */ + { { { { gen_helper_sve_ldbss_zsu, + gen_helper_sve_ldhss_be_zsu, + NULL, }, + { gen_helper_sve_ldbsu_zsu, + gen_helper_sve_ldhsu_be_zsu, + gen_helper_sve_ldss_be_zsu, } }, + { { gen_helper_sve_ldbss_zss, + gen_helper_sve_ldhss_be_zss, + NULL, }, + { gen_helper_sve_ldbsu_zss, + gen_helper_sve_ldhsu_be_zss, + gen_helper_sve_ldss_be_zss, } } }, + + /* First-fault */ + { { { gen_helper_sve_ldffbss_zsu, + gen_helper_sve_ldffhss_zsu, + NULL, }, + { gen_helper_sve_ldffbsu_zsu, + gen_helper_sve_ldffhsu_zsu, + gen_helper_sve_ldffssu_zsu, } }, + { { gen_helper_sve_ldffbss_zss, + gen_helper_sve_ldffhss_zss, + NULL, }, + { gen_helper_sve_ldffbsu_zss, + gen_helper_sve_ldffhsu_zss, + gen_helper_sve_ldffssu_zss, } } } }, }; /* Note that we overload xs=2 to indicate 64-bit offset. */ -static gen_helper_gvec_mem_scatter * const gather_load_fn64[2][3][2][4] = { - { { { gen_helper_sve_ldbds_zsu, - gen_helper_sve_ldhds_zsu, - gen_helper_sve_ldsds_zsu, - NULL, }, - { gen_helper_sve_ldbdu_zsu, - gen_helper_sve_ldhdu_zsu, - gen_helper_sve_ldsdu_zsu, - gen_helper_sve_ldddu_zsu, } }, - { { gen_helper_sve_ldbds_zss, - gen_helper_sve_ldhds_zss, - gen_helper_sve_ldsds_zss, - NULL, }, - { gen_helper_sve_ldbdu_zss, - gen_helper_sve_ldhdu_zss, - gen_helper_sve_ldsdu_zss, - gen_helper_sve_ldddu_zss, } }, - { { gen_helper_sve_ldbds_zd, - gen_helper_sve_ldhds_zd, - gen_helper_sve_ldsds_zd, - NULL, }, - { gen_helper_sve_ldbdu_zd, - gen_helper_sve_ldhdu_zd, - gen_helper_sve_ldsdu_zd, - gen_helper_sve_ldddu_zd, } } }, +static gen_helper_gvec_mem_scatter * const gather_load_fn64[2][2][3][2][4] = { + /* Little-endian */ + { { { { gen_helper_sve_ldbds_zsu, + gen_helper_sve_ldhds_le_zsu, + gen_helper_sve_ldsds_le_zsu, + NULL, }, + { gen_helper_sve_ldbdu_zsu, + gen_helper_sve_ldhdu_le_zsu, + gen_helper_sve_ldsdu_le_zsu, + gen_helper_sve_lddd_le_zsu, } }, + { { gen_helper_sve_ldbds_zss, + gen_helper_sve_ldhds_le_zss, + gen_helper_sve_ldsds_le_zss, + NULL, }, + { gen_helper_sve_ldbdu_zss, + gen_helper_sve_ldhdu_le_zss, + gen_helper_sve_ldsdu_le_zss, + gen_helper_sve_lddd_le_zss, } }, + { { gen_helper_sve_ldbds_zd, + gen_helper_sve_ldhds_le_zd, + gen_helper_sve_ldsds_le_zd, + NULL, }, + { gen_helper_sve_ldbdu_zd, + gen_helper_sve_ldhdu_le_zd, + gen_helper_sve_ldsdu_le_zd, + gen_helper_sve_lddd_le_zd, } } }, - { { { gen_helper_sve_ldffbds_zsu, - gen_helper_sve_ldffhds_zsu, - gen_helper_sve_ldffsds_zsu, - NULL, }, - { gen_helper_sve_ldffbdu_zsu, - gen_helper_sve_ldffhdu_zsu, - gen_helper_sve_ldffsdu_zsu, - gen_helper_sve_ldffddu_zsu, } }, - { { gen_helper_sve_ldffbds_zss, - gen_helper_sve_ldffhds_zss, - gen_helper_sve_ldffsds_zss, - NULL, }, - { gen_helper_sve_ldffbdu_zss, - gen_helper_sve_ldffhdu_zss, - gen_helper_sve_ldffsdu_zss, - gen_helper_sve_ldffddu_zss, } }, - { { gen_helper_sve_ldffbds_zd, - gen_helper_sve_ldffhds_zd, - gen_helper_sve_ldffsds_zd, - NULL, }, - { gen_helper_sve_ldffbdu_zd, - gen_helper_sve_ldffhdu_zd, - gen_helper_sve_ldffsdu_zd, - gen_helper_sve_ldffddu_zd, } } } + /* First-fault */ + { { { gen_helper_sve_ldffbds_zsu, + gen_helper_sve_ldffhds_zsu, + gen_helper_sve_ldffsds_zsu, + NULL, }, + { gen_helper_sve_ldffbdu_zsu, + gen_helper_sve_ldffhdu_zsu, + gen_helper_sve_ldffsdu_zsu, + gen_helper_sve_ldffddu_zsu, } }, + { { gen_helper_sve_ldffbds_zss, + gen_helper_sve_ldffhds_zss, + gen_helper_sve_ldffsds_zss, + NULL, }, + { gen_helper_sve_ldffbdu_zss, + gen_helper_sve_ldffhdu_zss, + gen_helper_sve_ldffsdu_zss, + gen_helper_sve_ldffddu_zss, } }, + { { gen_helper_sve_ldffbds_zd, + gen_helper_sve_ldffhds_zd, + gen_helper_sve_ldffsds_zd, + NULL, }, + { gen_helper_sve_ldffbdu_zd, + gen_helper_sve_ldffhdu_zd, + gen_helper_sve_ldffsdu_zd, + gen_helper_sve_ldffddu_zd, } } } }, + + /* Big-endian */ + { { { { gen_helper_sve_ldbds_zsu, + gen_helper_sve_ldhds_be_zsu, + gen_helper_sve_ldsds_be_zsu, + NULL, }, + { gen_helper_sve_ldbdu_zsu, + gen_helper_sve_ldhdu_be_zsu, + gen_helper_sve_ldsdu_be_zsu, + gen_helper_sve_lddd_be_zsu, } }, + { { gen_helper_sve_ldbds_zss, + gen_helper_sve_ldhds_be_zss, + gen_helper_sve_ldsds_be_zss, + NULL, }, + { gen_helper_sve_ldbdu_zss, + gen_helper_sve_ldhdu_be_zss, + gen_helper_sve_ldsdu_be_zss, + gen_helper_sve_lddd_be_zss, } }, + { { gen_helper_sve_ldbds_zd, + gen_helper_sve_ldhds_be_zd, + gen_helper_sve_ldsds_be_zd, + NULL, }, + { gen_helper_sve_ldbdu_zd, + gen_helper_sve_ldhdu_be_zd, + gen_helper_sve_ldsdu_be_zd, + gen_helper_sve_lddd_be_zd, } } }, + + /* First-fault */ + { { { gen_helper_sve_ldffbds_zsu, + gen_helper_sve_ldffhds_zsu, + gen_helper_sve_ldffsds_zsu, + NULL, }, + { gen_helper_sve_ldffbdu_zsu, + gen_helper_sve_ldffhdu_zsu, + gen_helper_sve_ldffsdu_zsu, + gen_helper_sve_ldffddu_zsu, } }, + { { gen_helper_sve_ldffbds_zss, + gen_helper_sve_ldffhds_zss, + gen_helper_sve_ldffsds_zss, + NULL, }, + { gen_helper_sve_ldffbdu_zss, + gen_helper_sve_ldffhdu_zss, + gen_helper_sve_ldffsdu_zss, + gen_helper_sve_ldffddu_zss, } }, + { { gen_helper_sve_ldffbds_zd, + gen_helper_sve_ldffhds_zd, + gen_helper_sve_ldffsds_zd, + NULL, }, + { gen_helper_sve_ldffbdu_zd, + gen_helper_sve_ldffhdu_zd, + gen_helper_sve_ldffsdu_zd, + gen_helper_sve_ldffddu_zd, } } } }, }; static bool trans_LD1_zprz(DisasContext *s, arg_LD1_zprz *a, uint32_t insn) { gen_helper_gvec_mem_scatter *fn = NULL; + int be = s->be_data == MO_BE; if (!sve_access_check(s)) { return true; @@ -5169,10 +5254,10 @@ static bool trans_LD1_zprz(DisasContext *s, arg_LD1_zprz *a, uint32_t insn) switch (a->esz) { case MO_32: - fn = gather_load_fn32[a->ff][a->xs][a->u][a->msz]; + fn = gather_load_fn32[be][a->ff][a->xs][a->u][a->msz]; break; case MO_64: - fn = gather_load_fn64[a->ff][a->xs][a->u][a->msz]; + fn = gather_load_fn64[be][a->ff][a->xs][a->u][a->msz]; break; } assert(fn != NULL); @@ -5185,6 +5270,7 @@ static bool trans_LD1_zprz(DisasContext *s, arg_LD1_zprz *a, uint32_t insn) static bool trans_LD1_zpiz(DisasContext *s, arg_LD1_zpiz *a, uint32_t insn) { gen_helper_gvec_mem_scatter *fn = NULL; + int be = s->be_data == MO_BE; TCGv_i64 imm; if (a->esz < a->msz || (a->esz == a->msz && !a->u)) { @@ -5196,10 +5282,10 @@ static bool trans_LD1_zpiz(DisasContext *s, arg_LD1_zpiz *a, uint32_t insn) switch (a->esz) { case MO_32: - fn = gather_load_fn32[a->ff][0][a->u][a->msz]; + fn = gather_load_fn32[be][a->ff][0][a->u][a->msz]; break; case MO_64: - fn = gather_load_fn64[a->ff][2][a->u][a->msz]; + fn = gather_load_fn64[be][a->ff][2][a->u][a->msz]; break; } assert(fn != NULL); From patchwork Thu Aug 9 04:22:04 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 143672 Delivered-To: patch@linaro.org Received: by 2002:a2e:9754:0:0:0:0:0 with SMTP id f20-v6csp1611983ljj; Wed, 8 Aug 2018 21:36:40 -0700 (PDT) X-Google-Smtp-Source: AA+uWPzUvbr51eQ9Fn39A19MCEWhuoRK8fskvVzHWvnB0o8Al2eVLgtQt5qlXFMLFfGMgCKCVM3R X-Received: by 2002:a37:188c:: with SMTP id 12-v6mr459334qky.72.1533789400434; Wed, 08 Aug 2018 21:36:40 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1533789400; cv=none; d=google.com; s=arc-20160816; b=Yds2YUE4g55sjfAaKamMrIdixbnyIez96HzUuY4Fb2+aww9KgTbAgl/M8KZSt5BMp6 sZoJEzME67s7xAw+nw70PCEGYbQvG+aR7hDNEMjKmX12H6GO+yKRBQ3D23o2R82B5Lbw E813HoDOTSjqb2/gNm0N0EfMVSjcs0YkXS5SKOC8aXvCD/sgDRq4I8b4op8w7YM3gkxz H4c5ArpK79XL0y8y3wqHg0ceJlpY8nH2ApNDNqNZ4htquzk6A1uidti2Bo2aYGDWzkOD XoUxSqroVGb3v+zvymL+14jSz1hjmqMUTfshtTk23NAdgtjRWefeQwX5ZFG4Kf7Ydrig w3bQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:cc:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:subject:references:in-reply-to :message-id:date:to:from:dkim-signature:arc-authentication-results; bh=paXDRnlhkiadwTlPfvFhVFOePEDhp/iJOK0Jyc9qWfU=; b=kBuLOz6rTZFKbmRAakfXHBFbckYJ8CkLmXomdJyZh9qav87j89JNmJ6J3QhhFlCM7c xJCS+IR8jrMS39NDtC5F830QcqPenNm4UjNXJ0nxp+QN44WEi1t2/uNKEKTv3Ebw64TY 7pWZFa94QR13ZG78mKdETJ+Clv3DLEJqL/IfhFsBOs2TNFXKUIu+tGgHGcA8HjRKqt5S yLA6XHyGSFdWgqrsnSjQq228fonshEE3cQhn41ImD+0+2oMSX+3ZmOeTMInHl7MwlSjQ 2g0lpbeqn3ZWbucF1D5ayJyWYIa+QowfKt55r0Md4+7Ko03ub3YL8xB3PUXD/2qyqZMV reYw== ARC-Authentication-Results: i=1; mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=hXe7WrD8; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [2001:4830:134:3::11]) by mx.google.com with ESMTPS id v17-v6si6034113qtc.331.2018.08.08.21.36.40 for (version=TLS1 cipher=AES128-SHA bits=128/128); Wed, 08 Aug 2018 21:36:40 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) client-ip=2001:4830:134:3::11; Authentication-Results: mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=hXe7WrD8; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1]:47300 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fncgh-0004zG-TW for patch@linaro.org; Thu, 09 Aug 2018 00:36:39 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:54137) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fncT7-00011t-FH for qemu-devel@nongnu.org; Thu, 09 Aug 2018 00:22:40 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fncT5-0007Te-G3 for qemu-devel@nongnu.org; Thu, 09 Aug 2018 00:22:37 -0400 Received: from mail-pf1-x42f.google.com ([2607:f8b0:4864:20::42f]:43248) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fncT5-0007Sq-6B for qemu-devel@nongnu.org; Thu, 09 Aug 2018 00:22:35 -0400 Received: by mail-pf1-x42f.google.com with SMTP id j26-v6so2186491pfi.10 for ; Wed, 08 Aug 2018 21:22:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=paXDRnlhkiadwTlPfvFhVFOePEDhp/iJOK0Jyc9qWfU=; b=hXe7WrD8SyFm+0h9BFHzCmW/rfO/hT+2eMbCf0hzhe2XSVRIVW4ICbS5Ai7NDJ+iED bY526NwnGw/e6k6Ce0PoaKM7crAe349EF2F8sN+WvkWBczaRGU5D4Kt0Y+q//Pw/dE6x baRrRIqV1nYDx298hlFF17DkiF3ny2RKXc8Gc= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=paXDRnlhkiadwTlPfvFhVFOePEDhp/iJOK0Jyc9qWfU=; b=qY9LvTagFQY2V4RI2O0hImtha0jaQDnN54YG7crXopjnbFpVjy+7+WNO4Kdqi4drfZ 0u7fQyr33juZxrXrS0wuCKvwK4nbGDvHyA/CjuD/MsmHqPjDlwq/4xAWeZQsArL6m2NF YnXyTWSIJQU+WpqgjiTtEg7GCqidCwHo4/8awJSxgzOjXZn8qSTiKzFS9ECRrsjiaqIT 53GhWa7M+XJmSJb+llDtgv9LKhiwLJ77JkRTpRJyGFQc2nuXeuq3LMdqWVvjuWP8aMYw e4RXU9Bq0LSot9zFT50+avpe2JIN51d/5gvMiCfm/j3g1v4btjF6lplDJYd0vfewirM+ JzDA== X-Gm-Message-State: AOUpUlGl5fxBOzXkk8Foa1rpq1Yd7g0pQzFEnN2XhiZD2IaAJIo6Z7Lq 7QftMSZrsgriCWxepnnknkTD989x2/s= X-Received: by 2002:a63:82c7:: with SMTP id w190-v6mr493678pgd.253.1533788553886; Wed, 08 Aug 2018 21:22:33 -0700 (PDT) Received: from cloudburst.twiddle.net (97-113-8-179.tukw.qwest.net. [97.113.8.179]) by smtp.gmail.com with ESMTPSA id m30-v6sm7355799pff.121.2018.08.08.21.22.32 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Wed, 08 Aug 2018 21:22:32 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Date: Wed, 8 Aug 2018 21:22:04 -0700 Message-Id: <20180809042206.15726-19-richard.henderson@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180809042206.15726-1-richard.henderson@linaro.org> References: <20180809042206.15726-1-richard.henderson@linaro.org> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::42f Subject: [Qemu-devel] [PATCH 18/20] target/arm: Rewrite vector gather stores X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: laurent.desnogues@gmail.com, peter.maydell@linaro.org, alex.bennee@linaro.org Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: "Qemu-devel" This fixes the endianness problem for softmmu, and does move the main loop out of a macro and into an inlined function. Signed-off-by: Richard Henderson --- target/arm/helper-sve.h | 52 ++++++++++---- target/arm/sve_helper.c | 139 ++++++++++++++++++++++++------------- target/arm/translate-sve.c | 74 +++++++++++++------- 3 files changed, 177 insertions(+), 88 deletions(-) -- 2.17.1 Reviewed-by: Peter Maydell diff --git a/target/arm/helper-sve.h b/target/arm/helper-sve.h index 49d1c09e30..6b9b93af45 100644 --- a/target/arm/helper-sve.h +++ b/target/arm/helper-sve.h @@ -1468,41 +1468,67 @@ DEF_HELPER_FLAGS_6(sve_ldffsds_zd, TCG_CALL_NO_WG, DEF_HELPER_FLAGS_6(sve_stbs_zsu, TCG_CALL_NO_WG, void, env, ptr, ptr, ptr, tl, i32) -DEF_HELPER_FLAGS_6(sve_sths_zsu, TCG_CALL_NO_WG, +DEF_HELPER_FLAGS_6(sve_sths_le_zsu, TCG_CALL_NO_WG, void, env, ptr, ptr, ptr, tl, i32) -DEF_HELPER_FLAGS_6(sve_stss_zsu, TCG_CALL_NO_WG, +DEF_HELPER_FLAGS_6(sve_sths_be_zsu, TCG_CALL_NO_WG, + void, env, ptr, ptr, ptr, tl, i32) +DEF_HELPER_FLAGS_6(sve_stss_le_zsu, TCG_CALL_NO_WG, + void, env, ptr, ptr, ptr, tl, i32) +DEF_HELPER_FLAGS_6(sve_stss_be_zsu, TCG_CALL_NO_WG, void, env, ptr, ptr, ptr, tl, i32) DEF_HELPER_FLAGS_6(sve_stbs_zss, TCG_CALL_NO_WG, void, env, ptr, ptr, ptr, tl, i32) -DEF_HELPER_FLAGS_6(sve_sths_zss, TCG_CALL_NO_WG, +DEF_HELPER_FLAGS_6(sve_sths_le_zss, TCG_CALL_NO_WG, void, env, ptr, ptr, ptr, tl, i32) -DEF_HELPER_FLAGS_6(sve_stss_zss, TCG_CALL_NO_WG, +DEF_HELPER_FLAGS_6(sve_sths_be_zss, TCG_CALL_NO_WG, + void, env, ptr, ptr, ptr, tl, i32) +DEF_HELPER_FLAGS_6(sve_stss_le_zss, TCG_CALL_NO_WG, + void, env, ptr, ptr, ptr, tl, i32) +DEF_HELPER_FLAGS_6(sve_stss_be_zss, TCG_CALL_NO_WG, void, env, ptr, ptr, ptr, tl, i32) DEF_HELPER_FLAGS_6(sve_stbd_zsu, TCG_CALL_NO_WG, void, env, ptr, ptr, ptr, tl, i32) -DEF_HELPER_FLAGS_6(sve_sthd_zsu, TCG_CALL_NO_WG, +DEF_HELPER_FLAGS_6(sve_sthd_le_zsu, TCG_CALL_NO_WG, void, env, ptr, ptr, ptr, tl, i32) -DEF_HELPER_FLAGS_6(sve_stsd_zsu, TCG_CALL_NO_WG, +DEF_HELPER_FLAGS_6(sve_sthd_be_zsu, TCG_CALL_NO_WG, void, env, ptr, ptr, ptr, tl, i32) -DEF_HELPER_FLAGS_6(sve_stdd_zsu, TCG_CALL_NO_WG, +DEF_HELPER_FLAGS_6(sve_stsd_le_zsu, TCG_CALL_NO_WG, + void, env, ptr, ptr, ptr, tl, i32) +DEF_HELPER_FLAGS_6(sve_stsd_be_zsu, TCG_CALL_NO_WG, + void, env, ptr, ptr, ptr, tl, i32) +DEF_HELPER_FLAGS_6(sve_stdd_le_zsu, TCG_CALL_NO_WG, + void, env, ptr, ptr, ptr, tl, i32) +DEF_HELPER_FLAGS_6(sve_stdd_be_zsu, TCG_CALL_NO_WG, void, env, ptr, ptr, ptr, tl, i32) DEF_HELPER_FLAGS_6(sve_stbd_zss, TCG_CALL_NO_WG, void, env, ptr, ptr, ptr, tl, i32) -DEF_HELPER_FLAGS_6(sve_sthd_zss, TCG_CALL_NO_WG, +DEF_HELPER_FLAGS_6(sve_sthd_le_zss, TCG_CALL_NO_WG, void, env, ptr, ptr, ptr, tl, i32) -DEF_HELPER_FLAGS_6(sve_stsd_zss, TCG_CALL_NO_WG, +DEF_HELPER_FLAGS_6(sve_sthd_be_zss, TCG_CALL_NO_WG, void, env, ptr, ptr, ptr, tl, i32) -DEF_HELPER_FLAGS_6(sve_stdd_zss, TCG_CALL_NO_WG, +DEF_HELPER_FLAGS_6(sve_stsd_le_zss, TCG_CALL_NO_WG, + void, env, ptr, ptr, ptr, tl, i32) +DEF_HELPER_FLAGS_6(sve_stsd_be_zss, TCG_CALL_NO_WG, + void, env, ptr, ptr, ptr, tl, i32) +DEF_HELPER_FLAGS_6(sve_stdd_le_zss, TCG_CALL_NO_WG, + void, env, ptr, ptr, ptr, tl, i32) +DEF_HELPER_FLAGS_6(sve_stdd_be_zss, TCG_CALL_NO_WG, void, env, ptr, ptr, ptr, tl, i32) DEF_HELPER_FLAGS_6(sve_stbd_zd, TCG_CALL_NO_WG, void, env, ptr, ptr, ptr, tl, i32) -DEF_HELPER_FLAGS_6(sve_sthd_zd, TCG_CALL_NO_WG, +DEF_HELPER_FLAGS_6(sve_sthd_le_zd, TCG_CALL_NO_WG, void, env, ptr, ptr, ptr, tl, i32) -DEF_HELPER_FLAGS_6(sve_stsd_zd, TCG_CALL_NO_WG, +DEF_HELPER_FLAGS_6(sve_sthd_be_zd, TCG_CALL_NO_WG, void, env, ptr, ptr, ptr, tl, i32) -DEF_HELPER_FLAGS_6(sve_stdd_zd, TCG_CALL_NO_WG, +DEF_HELPER_FLAGS_6(sve_stsd_le_zd, TCG_CALL_NO_WG, + void, env, ptr, ptr, ptr, tl, i32) +DEF_HELPER_FLAGS_6(sve_stsd_be_zd, TCG_CALL_NO_WG, + void, env, ptr, ptr, ptr, tl, i32) +DEF_HELPER_FLAGS_6(sve_stdd_le_zd, TCG_CALL_NO_WG, + void, env, ptr, ptr, ptr, tl, i32) +DEF_HELPER_FLAGS_6(sve_stdd_be_zd, TCG_CALL_NO_WG, void, env, ptr, ptr, ptr, tl, i32) diff --git a/target/arm/sve_helper.c b/target/arm/sve_helper.c index 76d3f021e4..0a4756bff9 100644 --- a/target/arm/sve_helper.c +++ b/target/arm/sve_helper.c @@ -5235,61 +5235,100 @@ DO_LDFF1_ZPZ_D(sve_ldffsds_zd, uint64_t, int32_t, cpu_ldl_data_ra) /* Stores with a vector index. */ -#define DO_ST1_ZPZ_S(NAME, TYPEI, FN) \ -void HELPER(NAME)(CPUARMState *env, void *vd, void *vg, void *vm, \ - target_ulong base, uint32_t desc) \ -{ \ - intptr_t i, oprsz = simd_oprsz(desc); \ - unsigned scale = simd_data(desc); \ - uintptr_t ra = GETPC(); \ - for (i = 0; i < oprsz; ) { \ - uint16_t pg = *(uint16_t *)(vg + H1_2(i >> 3)); \ - do { \ - if (likely(pg & 1)) { \ - target_ulong off = *(TYPEI *)(vm + H1_4(i)); \ - uint32_t d = *(uint32_t *)(vd + H1_4(i)); \ - FN(env, base + (off << scale), d, ra); \ - } \ - i += sizeof(uint32_t), pg >>= sizeof(uint32_t); \ - } while (i & 15); \ - } \ +static void sve_st1_zs(CPUARMState *env, void *vd, void *vg, void *vm, + target_ulong base, uint32_t desc, uintptr_t ra, + zreg_off_fn *off_fn, sve_ld1_tlb_fn *tlb_fn) +{ + const int mmu_idx = cpu_mmu_index(env, false); + intptr_t i, oprsz = simd_oprsz(desc); + unsigned scale = simd_data(desc); + + set_helper_retaddr(ra); + for (i = 0; i < oprsz; ) { + uint16_t pg = *(uint16_t *)(vg + H1_2(i >> 3)); + do { + if (pg & 1) { + target_ulong off = off_fn(vm, i); + tlb_fn(env, vd, i, base + (off << scale), mmu_idx, ra); + } + i += 4, pg >>= 4; + } while (i & 15); + } + set_helper_retaddr(0); } -#define DO_ST1_ZPZ_D(NAME, TYPEI, FN) \ -void HELPER(NAME)(CPUARMState *env, void *vd, void *vg, void *vm, \ - target_ulong base, uint32_t desc) \ -{ \ - intptr_t i, oprsz = simd_oprsz(desc) / 8; \ - unsigned scale = simd_data(desc); \ - uintptr_t ra = GETPC(); \ - uint64_t *d = vd, *m = vm; uint8_t *pg = vg; \ - for (i = 0; i < oprsz; i++) { \ - if (likely(pg[H1(i)] & 1)) { \ - target_ulong off = (target_ulong)(TYPEI)m[i] << scale; \ - FN(env, base + off, d[i], ra); \ - } \ - } \ +static void sve_st1_zd(CPUARMState *env, void *vd, void *vg, void *vm, + target_ulong base, uint32_t desc, uintptr_t ra, + zreg_off_fn *off_fn, sve_ld1_tlb_fn *tlb_fn) +{ + const int mmu_idx = cpu_mmu_index(env, false); + intptr_t i, oprsz = simd_oprsz(desc) / 8; + unsigned scale = simd_data(desc); + + set_helper_retaddr(ra); + for (i = 0; i < oprsz; i++) { + uint8_t pg = *(uint8_t *)(vg + H1(i)); + if (pg & 1) { + target_ulong off = off_fn(vm, i * 8); + tlb_fn(env, vd, i * 8, base + (off << scale), mmu_idx, ra); + } + } + set_helper_retaddr(0); } -DO_ST1_ZPZ_S(sve_stbs_zsu, uint32_t, cpu_stb_data_ra) -DO_ST1_ZPZ_S(sve_sths_zsu, uint32_t, cpu_stw_data_ra) -DO_ST1_ZPZ_S(sve_stss_zsu, uint32_t, cpu_stl_data_ra) +#define DO_ST1_ZPZ_S(MEM, OFS) \ +void __attribute__((flatten)) HELPER(sve_st##MEM##_##OFS) \ + (CPUARMState *env, void *vd, void *vg, void *vm, \ + target_ulong base, uint32_t desc) \ +{ \ + sve_st1_zs(env, vd, vg, vm, base, desc, GETPC(), \ + off_##OFS##_s, sve_st1##MEM##_tlb); \ +} -DO_ST1_ZPZ_S(sve_stbs_zss, int32_t, cpu_stb_data_ra) -DO_ST1_ZPZ_S(sve_sths_zss, int32_t, cpu_stw_data_ra) -DO_ST1_ZPZ_S(sve_stss_zss, int32_t, cpu_stl_data_ra) +#define DO_ST1_ZPZ_D(MEM, OFS) \ +void __attribute__((flatten)) HELPER(sve_st##MEM##_##OFS) \ + (CPUARMState *env, void *vd, void *vg, void *vm, \ + target_ulong base, uint32_t desc) \ +{ \ + sve_st1_zd(env, vd, vg, vm, base, desc, GETPC(), \ + off_##OFS##_d, sve_st1##MEM##_tlb); \ +} -DO_ST1_ZPZ_D(sve_stbd_zsu, uint32_t, cpu_stb_data_ra) -DO_ST1_ZPZ_D(sve_sthd_zsu, uint32_t, cpu_stw_data_ra) -DO_ST1_ZPZ_D(sve_stsd_zsu, uint32_t, cpu_stl_data_ra) -DO_ST1_ZPZ_D(sve_stdd_zsu, uint32_t, cpu_stq_data_ra) +DO_ST1_ZPZ_S(bs, zsu) +DO_ST1_ZPZ_S(hs_le, zsu) +DO_ST1_ZPZ_S(hs_be, zsu) +DO_ST1_ZPZ_S(ss_le, zsu) +DO_ST1_ZPZ_S(ss_be, zsu) -DO_ST1_ZPZ_D(sve_stbd_zss, int32_t, cpu_stb_data_ra) -DO_ST1_ZPZ_D(sve_sthd_zss, int32_t, cpu_stw_data_ra) -DO_ST1_ZPZ_D(sve_stsd_zss, int32_t, cpu_stl_data_ra) -DO_ST1_ZPZ_D(sve_stdd_zss, int32_t, cpu_stq_data_ra) +DO_ST1_ZPZ_S(bs, zss) +DO_ST1_ZPZ_S(hs_le, zss) +DO_ST1_ZPZ_S(hs_be, zss) +DO_ST1_ZPZ_S(ss_le, zss) +DO_ST1_ZPZ_S(ss_be, zss) -DO_ST1_ZPZ_D(sve_stbd_zd, uint64_t, cpu_stb_data_ra) -DO_ST1_ZPZ_D(sve_sthd_zd, uint64_t, cpu_stw_data_ra) -DO_ST1_ZPZ_D(sve_stsd_zd, uint64_t, cpu_stl_data_ra) -DO_ST1_ZPZ_D(sve_stdd_zd, uint64_t, cpu_stq_data_ra) +DO_ST1_ZPZ_D(bd, zsu) +DO_ST1_ZPZ_D(hd_le, zsu) +DO_ST1_ZPZ_D(hd_be, zsu) +DO_ST1_ZPZ_D(sd_le, zsu) +DO_ST1_ZPZ_D(sd_be, zsu) +DO_ST1_ZPZ_D(dd_le, zsu) +DO_ST1_ZPZ_D(dd_be, zsu) + +DO_ST1_ZPZ_D(bd, zss) +DO_ST1_ZPZ_D(hd_le, zss) +DO_ST1_ZPZ_D(hd_be, zss) +DO_ST1_ZPZ_D(sd_le, zss) +DO_ST1_ZPZ_D(sd_be, zss) +DO_ST1_ZPZ_D(dd_le, zss) +DO_ST1_ZPZ_D(dd_be, zss) + +DO_ST1_ZPZ_D(bd, zd) +DO_ST1_ZPZ_D(hd_le, zd) +DO_ST1_ZPZ_D(hd_be, zd) +DO_ST1_ZPZ_D(sd_le, zd) +DO_ST1_ZPZ_D(sd_be, zd) +DO_ST1_ZPZ_D(dd_le, zd) +DO_ST1_ZPZ_D(dd_be, zd) + +#undef DO_ST1_ZPZ_S +#undef DO_ST1_ZPZ_D diff --git a/target/arm/translate-sve.c b/target/arm/translate-sve.c index d4d7e9d3ae..fdd9b9b3a0 100644 --- a/target/arm/translate-sve.c +++ b/target/arm/translate-sve.c @@ -5299,35 +5299,58 @@ static bool trans_LD1_zpiz(DisasContext *s, arg_LD1_zpiz *a, uint32_t insn) return true; } -/* Indexed by [xs][msz]. */ -static gen_helper_gvec_mem_scatter * const scatter_store_fn32[2][3] = { - { gen_helper_sve_stbs_zsu, - gen_helper_sve_sths_zsu, - gen_helper_sve_stss_zsu, }, - { gen_helper_sve_stbs_zss, - gen_helper_sve_sths_zss, - gen_helper_sve_stss_zss, }, +/* Indexed by [be][xs][msz]. */ +static gen_helper_gvec_mem_scatter * const scatter_store_fn32[2][2][3] = { + /* Little-endian */ + { { gen_helper_sve_stbs_zsu, + gen_helper_sve_sths_le_zsu, + gen_helper_sve_stss_le_zsu, }, + { gen_helper_sve_stbs_zss, + gen_helper_sve_sths_le_zss, + gen_helper_sve_stss_le_zss, } }, + /* Big-endian */ + { { gen_helper_sve_stbs_zsu, + gen_helper_sve_sths_be_zsu, + gen_helper_sve_stss_be_zsu, }, + { gen_helper_sve_stbs_zss, + gen_helper_sve_sths_be_zss, + gen_helper_sve_stss_be_zss, } }, }; /* Note that we overload xs=2 to indicate 64-bit offset. */ -static gen_helper_gvec_mem_scatter * const scatter_store_fn64[3][4] = { - { gen_helper_sve_stbd_zsu, - gen_helper_sve_sthd_zsu, - gen_helper_sve_stsd_zsu, - gen_helper_sve_stdd_zsu, }, - { gen_helper_sve_stbd_zss, - gen_helper_sve_sthd_zss, - gen_helper_sve_stsd_zss, - gen_helper_sve_stdd_zss, }, - { gen_helper_sve_stbd_zd, - gen_helper_sve_sthd_zd, - gen_helper_sve_stsd_zd, - gen_helper_sve_stdd_zd, }, +static gen_helper_gvec_mem_scatter * const scatter_store_fn64[2][3][4] = { + /* Little-endian */ + { { gen_helper_sve_stbd_zsu, + gen_helper_sve_sthd_le_zsu, + gen_helper_sve_stsd_le_zsu, + gen_helper_sve_stdd_le_zsu, }, + { gen_helper_sve_stbd_zss, + gen_helper_sve_sthd_le_zss, + gen_helper_sve_stsd_le_zss, + gen_helper_sve_stdd_le_zss, }, + { gen_helper_sve_stbd_zd, + gen_helper_sve_sthd_le_zd, + gen_helper_sve_stsd_le_zd, + gen_helper_sve_stdd_le_zd, } }, + /* Big-endian */ + { { gen_helper_sve_stbd_zsu, + gen_helper_sve_sthd_be_zsu, + gen_helper_sve_stsd_be_zsu, + gen_helper_sve_stdd_be_zsu, }, + { gen_helper_sve_stbd_zss, + gen_helper_sve_sthd_be_zss, + gen_helper_sve_stsd_be_zss, + gen_helper_sve_stdd_be_zss, }, + { gen_helper_sve_stbd_zd, + gen_helper_sve_sthd_be_zd, + gen_helper_sve_stsd_be_zd, + gen_helper_sve_stdd_be_zd, } }, }; static bool trans_ST1_zprz(DisasContext *s, arg_ST1_zprz *a, uint32_t insn) { gen_helper_gvec_mem_scatter *fn; + int be = s->be_data == MO_BE; if (a->esz < a->msz || (a->msz == 0 && a->scale)) { return false; @@ -5337,10 +5360,10 @@ static bool trans_ST1_zprz(DisasContext *s, arg_ST1_zprz *a, uint32_t insn) } switch (a->esz) { case MO_32: - fn = scatter_store_fn32[a->xs][a->msz]; + fn = scatter_store_fn32[be][a->xs][a->msz]; break; case MO_64: - fn = scatter_store_fn64[a->xs][a->msz]; + fn = scatter_store_fn64[be][a->xs][a->msz]; break; default: g_assert_not_reached(); @@ -5353,6 +5376,7 @@ static bool trans_ST1_zprz(DisasContext *s, arg_ST1_zprz *a, uint32_t insn) static bool trans_ST1_zpiz(DisasContext *s, arg_ST1_zpiz *a, uint32_t insn) { gen_helper_gvec_mem_scatter *fn = NULL; + int be = s->be_data == MO_BE; TCGv_i64 imm; if (a->esz < a->msz) { @@ -5364,10 +5388,10 @@ static bool trans_ST1_zpiz(DisasContext *s, arg_ST1_zpiz *a, uint32_t insn) switch (a->esz) { case MO_32: - fn = scatter_store_fn32[0][a->msz]; + fn = scatter_store_fn32[be][0][a->msz]; break; case MO_64: - fn = scatter_store_fn64[2][a->msz]; + fn = scatter_store_fn64[be][2][a->msz]; break; } assert(fn != NULL); From patchwork Thu Aug 9 04:22:05 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 143675 Delivered-To: patch@linaro.org Received: by 2002:a2e:9754:0:0:0:0:0 with SMTP id f20-v6csp1614107ljj; Wed, 8 Aug 2018 21:39:38 -0700 (PDT) X-Google-Smtp-Source: AA+uWPwoG1FfPsvradbYODbdxWjrl9z1wZajMF51fLZaYbgTy5McXFsAFiCt3hnG0ijFe4tnp4+2 X-Received: by 2002:aed:3d3d:: with SMTP id g58-v6mr512280qtf.368.1533789578684; Wed, 08 Aug 2018 21:39:38 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1533789578; cv=none; d=google.com; s=arc-20160816; b=QFDhE9BNgdVv6rb6sust1HxiVie9deu09CrY/QnS1bvvhki+ll8k5VOP/QqK7oY1RK bnsn9PrNh/slJDzqR+QSAWix5Ns6sNoTaI252IvCwbiiPXtBd33CmEgHuHGuPksizoj4 urMpQPson9mAGbvm0mSPjcsTq3a32j9lx7oPUC/RCJVRgkDrBQhdydVhvrc4C95OI1f3 sY0hlqIsP/qzSvUDQgik3Q2T7XWkzoXRD/f7gfjPPJEb1sV+cUTEI+uO2tkKe44YApXs O73KCxhAR1ZLnGsVznAXZFECwyk8LwO6szImJLzspSOCtHL0oecmdfkX3j8QP4ANxZuG nJIA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:cc:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:subject:references:in-reply-to :message-id:date:to:from:dkim-signature:arc-authentication-results; bh=NJppe39DEVzDwFw+k2+WeIwy63OovxKFQOz0UayTB2U=; b=egiZzItaiDylkeWalbqsWl4g+YE7SN5HEfqE8CEvAIiJaY+9iJFdAoLqVlr31Bw26a +/8sMFMdzR6vtuen1XqWFuXtChvPIEe4w6CB9ACRBsU+9u6cHDjgmGoeG5B+Ivr4I0R7 EF84Wzm8BZVe8wpwqgcOr8QewEOwUshD+4AAr+0ObWhK/QiviLhjzh63PFHEvd1JO/24 gUQoaY+nH1DnI1IQO7GRVhllFHskNexbyOS0+XRqwcmc5SilnVLejiH+BytK4vbjAu6V Xob9S1yxSniHa8c1jiC9XWPcJFDDkTKJwnL77JYZ8Vu7Sb2/FV+xOZlHPhgdhgP8G+4Q jBPw== ARC-Authentication-Results: i=1; mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=bbA9EESW; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [2001:4830:134:3::11]) by mx.google.com with ESMTPS id p20-v6si1182969qta.33.2018.08.08.21.39.38 for (version=TLS1 cipher=AES128-SHA bits=128/128); Wed, 08 Aug 2018 21:39:38 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) client-ip=2001:4830:134:3::11; Authentication-Results: mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=bbA9EESW; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1]:47310 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fncja-0006uq-5w for patch@linaro.org; Thu, 09 Aug 2018 00:39:38 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:54160) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fncTA-000151-HL for qemu-devel@nongnu.org; Thu, 09 Aug 2018 00:22:43 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fncT7-0007VA-BG for qemu-devel@nongnu.org; Thu, 09 Aug 2018 00:22:40 -0400 Received: from mail-pg1-x52a.google.com ([2607:f8b0:4864:20::52a]:41704) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fncT6-0007UM-W1 for qemu-devel@nongnu.org; Thu, 09 Aug 2018 00:22:37 -0400 Received: by mail-pg1-x52a.google.com with SMTP id z8-v6so2108090pgu.8 for ; Wed, 08 Aug 2018 21:22:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=NJppe39DEVzDwFw+k2+WeIwy63OovxKFQOz0UayTB2U=; b=bbA9EESWYZBs627pNVtlPRdSX4+uT9UCHIyaCKLcp7DapFHm0pOq4Bzucp7VyLkFWo WBKuv39Me9PY01kUv/F1YFkxKN8EKJT4Fk+BMGtPfNxCTRWsTA+TTqAaNY24JtbcuVr/ JY4eKBsh3XNiZi6qbxcUatkgTt+kE2Bda7XgQ= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=NJppe39DEVzDwFw+k2+WeIwy63OovxKFQOz0UayTB2U=; b=kuy/G8l4VeRLj8suKTEf/huad0piRkgWHHxYq0Kea6pEwibCdLvA1evGY0cbJnFYCC 2whiZQ7A6aNRIiDlZBqcoDhtkmit+nhegAVPPeIwLMmH5zsJp0WvhSlZhVIUtAmcf1r1 YlmAMd4i57y/L3iJilIJWhn661S2sGjqGhbPRzURs9bQe98lBh4SXaAqdUxzaRE8D190 M/+hJ4rszccFYmOmTcjHtqVn490FPdWfizkJ5ZSHa3W3b8wF/AmOvyb//KKh68cNdl2Y DwLZ5wur+dyFc0lEoi+c2txfI7tl3aS6WEILvAK3u2pXiRE2cwNu8Dv9gWmkqbiSQiYq FNlA== X-Gm-Message-State: AOUpUlFZLqGwv3gF73bLF95xVnW59Cy/X9si8jP3weD+1zlxcgpWenM7 NfmsrFwjPD3of5d9FN6l7p3X25cbSZQ= X-Received: by 2002:a63:4506:: with SMTP id s6-v6mr522329pga.422.1533788555409; Wed, 08 Aug 2018 21:22:35 -0700 (PDT) Received: from cloudburst.twiddle.net (97-113-8-179.tukw.qwest.net. [97.113.8.179]) by smtp.gmail.com with ESMTPSA id m30-v6sm7355799pff.121.2018.08.08.21.22.34 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Wed, 08 Aug 2018 21:22:34 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Date: Wed, 8 Aug 2018 21:22:05 -0700 Message-Id: <20180809042206.15726-20-richard.henderson@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180809042206.15726-1-richard.henderson@linaro.org> References: <20180809042206.15726-1-richard.henderson@linaro.org> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::52a Subject: [Qemu-devel] [PATCH 19/20] target/arm: Rewrite vector gather first-fault loads X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: laurent.desnogues@gmail.com, peter.maydell@linaro.org, alex.bennee@linaro.org Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: "Qemu-devel" This implements the feature for softmmu, and moves the main loop out of a macro and into a function. Signed-off-by: Richard Henderson --- target/arm/helper-sve.h | 84 ++++++++--- target/arm/sve_helper.c | 290 +++++++++++++++++++++++++++---------- target/arm/translate-sve.c | 84 +++++------ 3 files changed, 321 insertions(+), 137 deletions(-) -- 2.17.1 Reviewed-by: Peter Maydell diff --git a/target/arm/helper-sve.h b/target/arm/helper-sve.h index 6b9b93af45..9e79182ab4 100644 --- a/target/arm/helper-sve.h +++ b/target/arm/helper-sve.h @@ -1401,69 +1401,111 @@ DEF_HELPER_FLAGS_6(sve_ldsds_be_zd, TCG_CALL_NO_WG, DEF_HELPER_FLAGS_6(sve_ldffbsu_zsu, TCG_CALL_NO_WG, void, env, ptr, ptr, ptr, tl, i32) -DEF_HELPER_FLAGS_6(sve_ldffhsu_zsu, TCG_CALL_NO_WG, +DEF_HELPER_FLAGS_6(sve_ldffhsu_le_zsu, TCG_CALL_NO_WG, void, env, ptr, ptr, ptr, tl, i32) -DEF_HELPER_FLAGS_6(sve_ldffssu_zsu, TCG_CALL_NO_WG, +DEF_HELPER_FLAGS_6(sve_ldffhsu_be_zsu, TCG_CALL_NO_WG, + void, env, ptr, ptr, ptr, tl, i32) +DEF_HELPER_FLAGS_6(sve_ldffss_le_zsu, TCG_CALL_NO_WG, + void, env, ptr, ptr, ptr, tl, i32) +DEF_HELPER_FLAGS_6(sve_ldffss_be_zsu, TCG_CALL_NO_WG, void, env, ptr, ptr, ptr, tl, i32) DEF_HELPER_FLAGS_6(sve_ldffbss_zsu, TCG_CALL_NO_WG, void, env, ptr, ptr, ptr, tl, i32) -DEF_HELPER_FLAGS_6(sve_ldffhss_zsu, TCG_CALL_NO_WG, +DEF_HELPER_FLAGS_6(sve_ldffhss_le_zsu, TCG_CALL_NO_WG, + void, env, ptr, ptr, ptr, tl, i32) +DEF_HELPER_FLAGS_6(sve_ldffhss_be_zsu, TCG_CALL_NO_WG, void, env, ptr, ptr, ptr, tl, i32) DEF_HELPER_FLAGS_6(sve_ldffbsu_zss, TCG_CALL_NO_WG, void, env, ptr, ptr, ptr, tl, i32) -DEF_HELPER_FLAGS_6(sve_ldffhsu_zss, TCG_CALL_NO_WG, +DEF_HELPER_FLAGS_6(sve_ldffhsu_le_zss, TCG_CALL_NO_WG, void, env, ptr, ptr, ptr, tl, i32) -DEF_HELPER_FLAGS_6(sve_ldffssu_zss, TCG_CALL_NO_WG, +DEF_HELPER_FLAGS_6(sve_ldffhsu_be_zss, TCG_CALL_NO_WG, + void, env, ptr, ptr, ptr, tl, i32) +DEF_HELPER_FLAGS_6(sve_ldffss_le_zss, TCG_CALL_NO_WG, + void, env, ptr, ptr, ptr, tl, i32) +DEF_HELPER_FLAGS_6(sve_ldffss_be_zss, TCG_CALL_NO_WG, void, env, ptr, ptr, ptr, tl, i32) DEF_HELPER_FLAGS_6(sve_ldffbss_zss, TCG_CALL_NO_WG, void, env, ptr, ptr, ptr, tl, i32) -DEF_HELPER_FLAGS_6(sve_ldffhss_zss, TCG_CALL_NO_WG, +DEF_HELPER_FLAGS_6(sve_ldffhss_le_zss, TCG_CALL_NO_WG, + void, env, ptr, ptr, ptr, tl, i32) +DEF_HELPER_FLAGS_6(sve_ldffhss_be_zss, TCG_CALL_NO_WG, void, env, ptr, ptr, ptr, tl, i32) DEF_HELPER_FLAGS_6(sve_ldffbdu_zsu, TCG_CALL_NO_WG, void, env, ptr, ptr, ptr, tl, i32) -DEF_HELPER_FLAGS_6(sve_ldffhdu_zsu, TCG_CALL_NO_WG, +DEF_HELPER_FLAGS_6(sve_ldffhdu_le_zsu, TCG_CALL_NO_WG, void, env, ptr, ptr, ptr, tl, i32) -DEF_HELPER_FLAGS_6(sve_ldffsdu_zsu, TCG_CALL_NO_WG, +DEF_HELPER_FLAGS_6(sve_ldffhdu_be_zsu, TCG_CALL_NO_WG, void, env, ptr, ptr, ptr, tl, i32) -DEF_HELPER_FLAGS_6(sve_ldffddu_zsu, TCG_CALL_NO_WG, +DEF_HELPER_FLAGS_6(sve_ldffsdu_le_zsu, TCG_CALL_NO_WG, + void, env, ptr, ptr, ptr, tl, i32) +DEF_HELPER_FLAGS_6(sve_ldffsdu_be_zsu, TCG_CALL_NO_WG, + void, env, ptr, ptr, ptr, tl, i32) +DEF_HELPER_FLAGS_6(sve_ldffdd_le_zsu, TCG_CALL_NO_WG, + void, env, ptr, ptr, ptr, tl, i32) +DEF_HELPER_FLAGS_6(sve_ldffdd_be_zsu, TCG_CALL_NO_WG, void, env, ptr, ptr, ptr, tl, i32) DEF_HELPER_FLAGS_6(sve_ldffbds_zsu, TCG_CALL_NO_WG, void, env, ptr, ptr, ptr, tl, i32) -DEF_HELPER_FLAGS_6(sve_ldffhds_zsu, TCG_CALL_NO_WG, +DEF_HELPER_FLAGS_6(sve_ldffhds_le_zsu, TCG_CALL_NO_WG, void, env, ptr, ptr, ptr, tl, i32) -DEF_HELPER_FLAGS_6(sve_ldffsds_zsu, TCG_CALL_NO_WG, +DEF_HELPER_FLAGS_6(sve_ldffhds_be_zsu, TCG_CALL_NO_WG, + void, env, ptr, ptr, ptr, tl, i32) +DEF_HELPER_FLAGS_6(sve_ldffsds_le_zsu, TCG_CALL_NO_WG, + void, env, ptr, ptr, ptr, tl, i32) +DEF_HELPER_FLAGS_6(sve_ldffsds_be_zsu, TCG_CALL_NO_WG, void, env, ptr, ptr, ptr, tl, i32) DEF_HELPER_FLAGS_6(sve_ldffbdu_zss, TCG_CALL_NO_WG, void, env, ptr, ptr, ptr, tl, i32) -DEF_HELPER_FLAGS_6(sve_ldffhdu_zss, TCG_CALL_NO_WG, +DEF_HELPER_FLAGS_6(sve_ldffhdu_le_zss, TCG_CALL_NO_WG, void, env, ptr, ptr, ptr, tl, i32) -DEF_HELPER_FLAGS_6(sve_ldffsdu_zss, TCG_CALL_NO_WG, +DEF_HELPER_FLAGS_6(sve_ldffhdu_be_zss, TCG_CALL_NO_WG, void, env, ptr, ptr, ptr, tl, i32) -DEF_HELPER_FLAGS_6(sve_ldffddu_zss, TCG_CALL_NO_WG, +DEF_HELPER_FLAGS_6(sve_ldffsdu_le_zss, TCG_CALL_NO_WG, + void, env, ptr, ptr, ptr, tl, i32) +DEF_HELPER_FLAGS_6(sve_ldffsdu_be_zss, TCG_CALL_NO_WG, + void, env, ptr, ptr, ptr, tl, i32) +DEF_HELPER_FLAGS_6(sve_ldffdd_le_zss, TCG_CALL_NO_WG, + void, env, ptr, ptr, ptr, tl, i32) +DEF_HELPER_FLAGS_6(sve_ldffdd_be_zss, TCG_CALL_NO_WG, void, env, ptr, ptr, ptr, tl, i32) DEF_HELPER_FLAGS_6(sve_ldffbds_zss, TCG_CALL_NO_WG, void, env, ptr, ptr, ptr, tl, i32) -DEF_HELPER_FLAGS_6(sve_ldffhds_zss, TCG_CALL_NO_WG, +DEF_HELPER_FLAGS_6(sve_ldffhds_le_zss, TCG_CALL_NO_WG, void, env, ptr, ptr, ptr, tl, i32) -DEF_HELPER_FLAGS_6(sve_ldffsds_zss, TCG_CALL_NO_WG, +DEF_HELPER_FLAGS_6(sve_ldffhds_be_zss, TCG_CALL_NO_WG, + void, env, ptr, ptr, ptr, tl, i32) +DEF_HELPER_FLAGS_6(sve_ldffsds_le_zss, TCG_CALL_NO_WG, + void, env, ptr, ptr, ptr, tl, i32) +DEF_HELPER_FLAGS_6(sve_ldffsds_be_zss, TCG_CALL_NO_WG, void, env, ptr, ptr, ptr, tl, i32) DEF_HELPER_FLAGS_6(sve_ldffbdu_zd, TCG_CALL_NO_WG, void, env, ptr, ptr, ptr, tl, i32) -DEF_HELPER_FLAGS_6(sve_ldffhdu_zd, TCG_CALL_NO_WG, +DEF_HELPER_FLAGS_6(sve_ldffhdu_le_zd, TCG_CALL_NO_WG, void, env, ptr, ptr, ptr, tl, i32) -DEF_HELPER_FLAGS_6(sve_ldffsdu_zd, TCG_CALL_NO_WG, +DEF_HELPER_FLAGS_6(sve_ldffhdu_be_zd, TCG_CALL_NO_WG, void, env, ptr, ptr, ptr, tl, i32) -DEF_HELPER_FLAGS_6(sve_ldffddu_zd, TCG_CALL_NO_WG, +DEF_HELPER_FLAGS_6(sve_ldffsdu_le_zd, TCG_CALL_NO_WG, + void, env, ptr, ptr, ptr, tl, i32) +DEF_HELPER_FLAGS_6(sve_ldffsdu_be_zd, TCG_CALL_NO_WG, + void, env, ptr, ptr, ptr, tl, i32) +DEF_HELPER_FLAGS_6(sve_ldffdd_le_zd, TCG_CALL_NO_WG, + void, env, ptr, ptr, ptr, tl, i32) +DEF_HELPER_FLAGS_6(sve_ldffdd_be_zd, TCG_CALL_NO_WG, void, env, ptr, ptr, ptr, tl, i32) DEF_HELPER_FLAGS_6(sve_ldffbds_zd, TCG_CALL_NO_WG, void, env, ptr, ptr, ptr, tl, i32) -DEF_HELPER_FLAGS_6(sve_ldffhds_zd, TCG_CALL_NO_WG, +DEF_HELPER_FLAGS_6(sve_ldffhds_le_zd, TCG_CALL_NO_WG, void, env, ptr, ptr, ptr, tl, i32) -DEF_HELPER_FLAGS_6(sve_ldffsds_zd, TCG_CALL_NO_WG, +DEF_HELPER_FLAGS_6(sve_ldffhds_be_zd, TCG_CALL_NO_WG, + void, env, ptr, ptr, ptr, tl, i32) +DEF_HELPER_FLAGS_6(sve_ldffsds_le_zd, TCG_CALL_NO_WG, + void, env, ptr, ptr, ptr, tl, i32) +DEF_HELPER_FLAGS_6(sve_ldffsds_be_zd, TCG_CALL_NO_WG, void, env, ptr, ptr, ptr, tl, i32) DEF_HELPER_FLAGS_6(sve_stbs_zsu, TCG_CALL_NO_WG, diff --git a/target/arm/sve_helper.c b/target/arm/sve_helper.c index 0a4756bff9..6728862326 100644 --- a/target/arm/sve_helper.c +++ b/target/arm/sve_helper.c @@ -5147,91 +5147,233 @@ DO_LD1_ZPZ_D(dd_be, zd) /* First fault loads with a vector index. */ -#ifdef CONFIG_USER_ONLY +/* Load one element into VD+REG_OFF from (ENV,VADDR) without faulting. + * The controlling predicate is known to be true. Return true if the + * load was successful. + */ +typedef bool sve_ld1_nf_fn(CPUARMState *env, void *vd, intptr_t reg_off, + target_ulong vaddr, int mmu_idx); -#define DO_LDFF1_ZPZ(NAME, TYPEE, TYPEI, TYPEM, FN, H) \ -void HELPER(NAME)(CPUARMState *env, void *vd, void *vg, void *vm, \ - target_ulong base, uint32_t desc) \ -{ \ - intptr_t i, oprsz = simd_oprsz(desc); \ - unsigned scale = simd_data(desc); \ - uintptr_t ra = GETPC(); \ - bool first = true; \ - mmap_lock(); \ - for (i = 0; i < oprsz; ) { \ - uint16_t pg = *(uint16_t *)(vg + H1_2(i >> 3)); \ - do { \ - TYPEM m = 0; \ - if (pg & 1) { \ - target_ulong off = *(TYPEI *)(vm + H(i)); \ - target_ulong addr = base + (off << scale); \ - if (!first && \ - page_check_range(addr, sizeof(TYPEM), PAGE_READ)) { \ - record_fault(env, i, oprsz); \ - goto exit; \ - } \ - m = FN(env, addr, ra); \ - first = false; \ - } \ - *(TYPEE *)(vd + H(i)) = m; \ - i += sizeof(TYPEE), pg >>= sizeof(TYPEE); \ - } while (i & 15); \ - } \ - exit: \ - mmap_unlock(); \ +#ifdef CONFIG_SOFTMMU +#define DO_LD_NF(NAME, H, TYPEE, TYPEM, HOST) \ +static bool sve_ld##NAME##_nf(CPUARMState *env, void *vd, intptr_t reg_off, \ + target_ulong addr, int mmu_idx) \ +{ \ + target_ulong next_page = -(addr | TARGET_PAGE_MASK); \ + if (likely(next_page - addr >= sizeof(TYPEM))) { \ + void *host = tlb_vaddr_to_host(env, addr, MMU_DATA_LOAD, mmu_idx); \ + if (likely(host)) { \ + TYPEM val = HOST(host); \ + *(TYPEE *)(vd + H(reg_off)) = val; \ + return true; \ + } \ + } \ + return false; \ } - #else - -#define DO_LDFF1_ZPZ(NAME, TYPEE, TYPEI, TYPEM, FN, H) \ -void HELPER(NAME)(CPUARMState *env, void *vd, void *vg, void *vm, \ - target_ulong base, uint32_t desc) \ -{ \ - g_assert_not_reached(); \ +#define DO_LD_NF(NAME, H, TYPEE, TYPEM, HOST) \ +static bool sve_ld##NAME##_nf(CPUARMState *env, void *vd, intptr_t reg_off, \ + target_ulong addr, int mmu_idx) \ +{ \ + if (likely(page_check_range(addr, sizeof(TYPEM), PAGE_READ))) { \ + TYPEM val = HOST(g2h(addr)); \ + *(TYPEE *)(vd + H(reg_off)) = val; \ + return true; \ + } \ + return false; \ } - #endif -#define DO_LDFF1_ZPZ_S(NAME, TYPEI, TYPEM, FN) \ - DO_LDFF1_ZPZ(NAME, uint32_t, TYPEI, TYPEM, FN, H1_4) -#define DO_LDFF1_ZPZ_D(NAME, TYPEI, TYPEM, FN) \ - DO_LDFF1_ZPZ(NAME, uint64_t, TYPEI, TYPEM, FN, ) +DO_LD_NF(bsu, H1_4, uint32_t, uint8_t, ldub_p) +DO_LD_NF(bss, H1_4, uint32_t, int8_t, ldsb_p) +DO_LD_NF(bdu, , uint64_t, uint8_t, ldub_p) +DO_LD_NF(bds, , uint64_t, int8_t, ldsb_p) -DO_LDFF1_ZPZ_S(sve_ldffbsu_zsu, uint32_t, uint8_t, cpu_ldub_data_ra) -DO_LDFF1_ZPZ_S(sve_ldffhsu_zsu, uint32_t, uint16_t, cpu_lduw_data_ra) -DO_LDFF1_ZPZ_S(sve_ldffssu_zsu, uint32_t, uint32_t, cpu_ldl_data_ra) -DO_LDFF1_ZPZ_S(sve_ldffbss_zsu, uint32_t, int8_t, cpu_ldub_data_ra) -DO_LDFF1_ZPZ_S(sve_ldffhss_zsu, uint32_t, int16_t, cpu_lduw_data_ra) +DO_LD_NF(hsu_le, H1_4, uint32_t, uint16_t, lduw_le_p) +DO_LD_NF(hss_le, H1_4, uint32_t, int16_t, ldsw_le_p) +DO_LD_NF(hsu_be, H1_4, uint32_t, uint16_t, lduw_be_p) +DO_LD_NF(hss_be, H1_4, uint32_t, int16_t, ldsw_be_p) +DO_LD_NF(hdu_le, , uint64_t, uint16_t, lduw_le_p) +DO_LD_NF(hds_le, , uint64_t, int16_t, ldsw_le_p) +DO_LD_NF(hdu_be, , uint64_t, uint16_t, lduw_be_p) +DO_LD_NF(hds_be, , uint64_t, int16_t, ldsw_be_p) -DO_LDFF1_ZPZ_S(sve_ldffbsu_zss, int32_t, uint8_t, cpu_ldub_data_ra) -DO_LDFF1_ZPZ_S(sve_ldffhsu_zss, int32_t, uint16_t, cpu_lduw_data_ra) -DO_LDFF1_ZPZ_S(sve_ldffssu_zss, int32_t, uint32_t, cpu_ldl_data_ra) -DO_LDFF1_ZPZ_S(sve_ldffbss_zss, int32_t, int8_t, cpu_ldub_data_ra) -DO_LDFF1_ZPZ_S(sve_ldffhss_zss, int32_t, int16_t, cpu_lduw_data_ra) +DO_LD_NF(ss_le, H1_4, uint32_t, uint32_t, ldl_le_p) +DO_LD_NF(ss_be, H1_4, uint32_t, uint32_t, ldl_be_p) +DO_LD_NF(sdu_le, , uint64_t, uint32_t, ldl_le_p) +DO_LD_NF(sds_le, , uint64_t, int32_t, ldl_le_p) +DO_LD_NF(sdu_be, , uint64_t, uint32_t, ldl_be_p) +DO_LD_NF(sds_be, , uint64_t, int32_t, ldl_be_p) -DO_LDFF1_ZPZ_D(sve_ldffbdu_zsu, uint32_t, uint8_t, cpu_ldub_data_ra) -DO_LDFF1_ZPZ_D(sve_ldffhdu_zsu, uint32_t, uint16_t, cpu_lduw_data_ra) -DO_LDFF1_ZPZ_D(sve_ldffsdu_zsu, uint32_t, uint32_t, cpu_ldl_data_ra) -DO_LDFF1_ZPZ_D(sve_ldffddu_zsu, uint32_t, uint64_t, cpu_ldq_data_ra) -DO_LDFF1_ZPZ_D(sve_ldffbds_zsu, uint32_t, int8_t, cpu_ldub_data_ra) -DO_LDFF1_ZPZ_D(sve_ldffhds_zsu, uint32_t, int16_t, cpu_lduw_data_ra) -DO_LDFF1_ZPZ_D(sve_ldffsds_zsu, uint32_t, int32_t, cpu_ldl_data_ra) +DO_LD_NF(dd_le, , uint64_t, uint64_t, ldq_le_p) +DO_LD_NF(dd_be, , uint64_t, uint64_t, ldq_be_p) -DO_LDFF1_ZPZ_D(sve_ldffbdu_zss, int32_t, uint8_t, cpu_ldub_data_ra) -DO_LDFF1_ZPZ_D(sve_ldffhdu_zss, int32_t, uint16_t, cpu_lduw_data_ra) -DO_LDFF1_ZPZ_D(sve_ldffsdu_zss, int32_t, uint32_t, cpu_ldl_data_ra) -DO_LDFF1_ZPZ_D(sve_ldffddu_zss, int32_t, uint64_t, cpu_ldq_data_ra) -DO_LDFF1_ZPZ_D(sve_ldffbds_zss, int32_t, int8_t, cpu_ldub_data_ra) -DO_LDFF1_ZPZ_D(sve_ldffhds_zss, int32_t, int16_t, cpu_lduw_data_ra) -DO_LDFF1_ZPZ_D(sve_ldffsds_zss, int32_t, int32_t, cpu_ldl_data_ra) +/* + * Common helper for all gather first-faulting loads. + */ +static inline void sve_ldff1_zs(CPUARMState *env, void *vd, void *vg, void *vm, + target_ulong base, uint32_t desc, uintptr_t ra, + zreg_off_fn *off_fn, sve_ld1_tlb_fn *tlb_fn, + sve_ld1_nf_fn *nonfault_fn) +{ + const int mmu_idx = cpu_mmu_index(env, false); + intptr_t reg_off, reg_max = simd_oprsz(desc); + unsigned scale = simd_data(desc); + target_ulong addr; -DO_LDFF1_ZPZ_D(sve_ldffbdu_zd, uint64_t, uint8_t, cpu_ldub_data_ra) -DO_LDFF1_ZPZ_D(sve_ldffhdu_zd, uint64_t, uint16_t, cpu_lduw_data_ra) -DO_LDFF1_ZPZ_D(sve_ldffsdu_zd, uint64_t, uint32_t, cpu_ldl_data_ra) -DO_LDFF1_ZPZ_D(sve_ldffddu_zd, uint64_t, uint64_t, cpu_ldq_data_ra) -DO_LDFF1_ZPZ_D(sve_ldffbds_zd, uint64_t, int8_t, cpu_ldub_data_ra) -DO_LDFF1_ZPZ_D(sve_ldffhds_zd, uint64_t, int16_t, cpu_lduw_data_ra) -DO_LDFF1_ZPZ_D(sve_ldffsds_zd, uint64_t, int32_t, cpu_ldl_data_ra) + /* Skip to the first true predicate. */ + reg_off = find_next_active(vg, 0, reg_max, MO_32); + if (likely(reg_off < reg_max)) { + /* Perform one normal read, which will fault or not. */ + set_helper_retaddr(ra); + addr = off_fn(vm, reg_off); + addr = base + (addr << scale); + tlb_fn(env, vd, reg_off, addr, mmu_idx, ra); + + /* The rest of the reads will be non-faulting. */ + set_helper_retaddr(0); + } + + /* After any fault, zero the leading predicated false elements. */ + swap_memzero(vd, reg_off); + + while (likely((reg_off += 4) < reg_max)) { + uint64_t pg = *(uint64_t *)(vg + (reg_off >> 6) * 8); + if (likely((pg >> (reg_off & 63)) & 1)) { + addr = off_fn(vm, reg_off); + addr = base + (addr << scale); + if (!nonfault_fn(env, vd, reg_off, addr, mmu_idx)) { + record_fault(env, reg_off, reg_max); + break; + } + } else { + *(uint32_t *)(vd + H1_4(reg_off)) = 0; + } + } +} + +static inline void sve_ldff1_zd(CPUARMState *env, void *vd, void *vg, void *vm, + target_ulong base, uint32_t desc, uintptr_t ra, + zreg_off_fn *off_fn, sve_ld1_tlb_fn *tlb_fn, + sve_ld1_nf_fn *nonfault_fn) +{ + const int mmu_idx = cpu_mmu_index(env, false); + intptr_t reg_off, reg_max = simd_oprsz(desc); + unsigned scale = simd_data(desc); + target_ulong addr; + + /* Skip to the first true predicate. */ + reg_off = find_next_active(vg, 0, reg_max, MO_64); + if (likely(reg_off < reg_max)) { + /* Perform one normal read, which will fault or not. */ + set_helper_retaddr(ra); + addr = off_fn(vm, reg_off); + addr = base + (addr << scale); + tlb_fn(env, vd, reg_off, addr, mmu_idx, ra); + + /* The rest of the reads will be non-faulting. */ + set_helper_retaddr(0); + } + + /* After any fault, zero the leading predicated false elements. */ + swap_memzero(vd, reg_off); + + while (likely((reg_off += 8) < reg_max)) { + uint8_t pg = *(uint8_t *)(vg + H1(reg_off >> 3)); + if (likely(pg & 1)) { + addr = off_fn(vm, reg_off); + addr = base + (addr << scale); + if (!nonfault_fn(env, vd, reg_off, addr, mmu_idx)) { + record_fault(env, reg_off, reg_max); + break; + } + } else { + *(uint64_t *)(vd + reg_off) = 0; + } + } +} + +#define DO_LDFF1_ZPZ_S(MEM, OFS) \ +void HELPER(sve_ldff##MEM##_##OFS) \ + (CPUARMState *env, void *vd, void *vg, void *vm, \ + target_ulong base, uint32_t desc) \ +{ \ + sve_ldff1_zs(env, vd, vg, vm, base, desc, GETPC(), \ + off_##OFS##_s, sve_ld1##MEM##_tlb, sve_ld##MEM##_nf); \ +} + +#define DO_LDFF1_ZPZ_D(MEM, OFS) \ +void HELPER(sve_ldff##MEM##_##OFS) \ + (CPUARMState *env, void *vd, void *vg, void *vm, \ + target_ulong base, uint32_t desc) \ +{ \ + sve_ldff1_zd(env, vd, vg, vm, base, desc, GETPC(), \ + off_##OFS##_d, sve_ld1##MEM##_tlb, sve_ld##MEM##_nf); \ +} + +DO_LDFF1_ZPZ_S(bsu, zsu) +DO_LDFF1_ZPZ_S(bsu, zss) +DO_LDFF1_ZPZ_D(bdu, zsu) +DO_LDFF1_ZPZ_D(bdu, zss) +DO_LDFF1_ZPZ_D(bdu, zd) + +DO_LDFF1_ZPZ_S(bss, zsu) +DO_LDFF1_ZPZ_S(bss, zss) +DO_LDFF1_ZPZ_D(bds, zsu) +DO_LDFF1_ZPZ_D(bds, zss) +DO_LDFF1_ZPZ_D(bds, zd) + +DO_LDFF1_ZPZ_S(hsu_le, zsu) +DO_LDFF1_ZPZ_S(hsu_le, zss) +DO_LDFF1_ZPZ_D(hdu_le, zsu) +DO_LDFF1_ZPZ_D(hdu_le, zss) +DO_LDFF1_ZPZ_D(hdu_le, zd) + +DO_LDFF1_ZPZ_S(hsu_be, zsu) +DO_LDFF1_ZPZ_S(hsu_be, zss) +DO_LDFF1_ZPZ_D(hdu_be, zsu) +DO_LDFF1_ZPZ_D(hdu_be, zss) +DO_LDFF1_ZPZ_D(hdu_be, zd) + +DO_LDFF1_ZPZ_S(hss_le, zsu) +DO_LDFF1_ZPZ_S(hss_le, zss) +DO_LDFF1_ZPZ_D(hds_le, zsu) +DO_LDFF1_ZPZ_D(hds_le, zss) +DO_LDFF1_ZPZ_D(hds_le, zd) + +DO_LDFF1_ZPZ_S(hss_be, zsu) +DO_LDFF1_ZPZ_S(hss_be, zss) +DO_LDFF1_ZPZ_D(hds_be, zsu) +DO_LDFF1_ZPZ_D(hds_be, zss) +DO_LDFF1_ZPZ_D(hds_be, zd) + +DO_LDFF1_ZPZ_S(ss_le, zsu) +DO_LDFF1_ZPZ_S(ss_le, zss) +DO_LDFF1_ZPZ_D(sdu_le, zsu) +DO_LDFF1_ZPZ_D(sdu_le, zss) +DO_LDFF1_ZPZ_D(sdu_le, zd) + +DO_LDFF1_ZPZ_S(ss_be, zsu) +DO_LDFF1_ZPZ_S(ss_be, zss) +DO_LDFF1_ZPZ_D(sdu_be, zsu) +DO_LDFF1_ZPZ_D(sdu_be, zss) +DO_LDFF1_ZPZ_D(sdu_be, zd) + +DO_LDFF1_ZPZ_D(sds_le, zsu) +DO_LDFF1_ZPZ_D(sds_le, zss) +DO_LDFF1_ZPZ_D(sds_le, zd) + +DO_LDFF1_ZPZ_D(sds_be, zsu) +DO_LDFF1_ZPZ_D(sds_be, zss) +DO_LDFF1_ZPZ_D(sds_be, zd) + +DO_LDFF1_ZPZ_D(dd_le, zsu) +DO_LDFF1_ZPZ_D(dd_le, zss) +DO_LDFF1_ZPZ_D(dd_le, zd) + +DO_LDFF1_ZPZ_D(dd_be, zsu) +DO_LDFF1_ZPZ_D(dd_be, zss) +DO_LDFF1_ZPZ_D(dd_be, zd) /* Stores with a vector index. */ diff --git a/target/arm/translate-sve.c b/target/arm/translate-sve.c index fdd9b9b3a0..20492e9b8b 100644 --- a/target/arm/translate-sve.c +++ b/target/arm/translate-sve.c @@ -5095,17 +5095,17 @@ static gen_helper_gvec_mem_scatter * const gather_load_fn32[2][2][2][2][3] = { /* First-fault */ { { { gen_helper_sve_ldffbss_zsu, - gen_helper_sve_ldffhss_zsu, + gen_helper_sve_ldffhss_le_zsu, NULL, }, { gen_helper_sve_ldffbsu_zsu, - gen_helper_sve_ldffhsu_zsu, - gen_helper_sve_ldffssu_zsu, } }, + gen_helper_sve_ldffhsu_le_zsu, + gen_helper_sve_ldffss_le_zsu, } }, { { gen_helper_sve_ldffbss_zss, - gen_helper_sve_ldffhss_zss, + gen_helper_sve_ldffhss_le_zss, NULL, }, { gen_helper_sve_ldffbsu_zss, - gen_helper_sve_ldffhsu_zss, - gen_helper_sve_ldffssu_zss, } } } }, + gen_helper_sve_ldffhsu_le_zss, + gen_helper_sve_ldffss_le_zss, } } } }, /* Big-endian */ { { { { gen_helper_sve_ldbss_zsu, @@ -5123,17 +5123,17 @@ static gen_helper_gvec_mem_scatter * const gather_load_fn32[2][2][2][2][3] = { /* First-fault */ { { { gen_helper_sve_ldffbss_zsu, - gen_helper_sve_ldffhss_zsu, + gen_helper_sve_ldffhss_be_zsu, NULL, }, { gen_helper_sve_ldffbsu_zsu, - gen_helper_sve_ldffhsu_zsu, - gen_helper_sve_ldffssu_zsu, } }, + gen_helper_sve_ldffhsu_be_zsu, + gen_helper_sve_ldffss_be_zsu, } }, { { gen_helper_sve_ldffbss_zss, - gen_helper_sve_ldffhss_zss, + gen_helper_sve_ldffhss_be_zss, NULL, }, { gen_helper_sve_ldffbsu_zss, - gen_helper_sve_ldffhsu_zss, - gen_helper_sve_ldffssu_zss, } } } }, + gen_helper_sve_ldffhsu_be_zss, + gen_helper_sve_ldffss_be_zss, } } } }, }; /* Note that we overload xs=2 to indicate 64-bit offset. */ @@ -5166,29 +5166,29 @@ static gen_helper_gvec_mem_scatter * const gather_load_fn64[2][2][3][2][4] = { /* First-fault */ { { { gen_helper_sve_ldffbds_zsu, - gen_helper_sve_ldffhds_zsu, - gen_helper_sve_ldffsds_zsu, + gen_helper_sve_ldffhds_le_zsu, + gen_helper_sve_ldffsds_le_zsu, NULL, }, { gen_helper_sve_ldffbdu_zsu, - gen_helper_sve_ldffhdu_zsu, - gen_helper_sve_ldffsdu_zsu, - gen_helper_sve_ldffddu_zsu, } }, + gen_helper_sve_ldffhdu_le_zsu, + gen_helper_sve_ldffsdu_le_zsu, + gen_helper_sve_ldffdd_le_zsu, } }, { { gen_helper_sve_ldffbds_zss, - gen_helper_sve_ldffhds_zss, - gen_helper_sve_ldffsds_zss, + gen_helper_sve_ldffhds_le_zss, + gen_helper_sve_ldffsds_le_zss, NULL, }, { gen_helper_sve_ldffbdu_zss, - gen_helper_sve_ldffhdu_zss, - gen_helper_sve_ldffsdu_zss, - gen_helper_sve_ldffddu_zss, } }, + gen_helper_sve_ldffhdu_le_zss, + gen_helper_sve_ldffsdu_le_zss, + gen_helper_sve_ldffdd_le_zss, } }, { { gen_helper_sve_ldffbds_zd, - gen_helper_sve_ldffhds_zd, - gen_helper_sve_ldffsds_zd, + gen_helper_sve_ldffhds_le_zd, + gen_helper_sve_ldffsds_le_zd, NULL, }, { gen_helper_sve_ldffbdu_zd, - gen_helper_sve_ldffhdu_zd, - gen_helper_sve_ldffsdu_zd, - gen_helper_sve_ldffddu_zd, } } } }, + gen_helper_sve_ldffhdu_le_zd, + gen_helper_sve_ldffsdu_le_zd, + gen_helper_sve_ldffdd_le_zd, } } } }, /* Big-endian */ { { { { gen_helper_sve_ldbds_zsu, @@ -5218,29 +5218,29 @@ static gen_helper_gvec_mem_scatter * const gather_load_fn64[2][2][3][2][4] = { /* First-fault */ { { { gen_helper_sve_ldffbds_zsu, - gen_helper_sve_ldffhds_zsu, - gen_helper_sve_ldffsds_zsu, + gen_helper_sve_ldffhds_be_zsu, + gen_helper_sve_ldffsds_be_zsu, NULL, }, { gen_helper_sve_ldffbdu_zsu, - gen_helper_sve_ldffhdu_zsu, - gen_helper_sve_ldffsdu_zsu, - gen_helper_sve_ldffddu_zsu, } }, + gen_helper_sve_ldffhdu_be_zsu, + gen_helper_sve_ldffsdu_be_zsu, + gen_helper_sve_ldffdd_be_zsu, } }, { { gen_helper_sve_ldffbds_zss, - gen_helper_sve_ldffhds_zss, - gen_helper_sve_ldffsds_zss, + gen_helper_sve_ldffhds_be_zss, + gen_helper_sve_ldffsds_be_zss, NULL, }, { gen_helper_sve_ldffbdu_zss, - gen_helper_sve_ldffhdu_zss, - gen_helper_sve_ldffsdu_zss, - gen_helper_sve_ldffddu_zss, } }, + gen_helper_sve_ldffhdu_be_zss, + gen_helper_sve_ldffsdu_be_zss, + gen_helper_sve_ldffdd_be_zss, } }, { { gen_helper_sve_ldffbds_zd, - gen_helper_sve_ldffhds_zd, - gen_helper_sve_ldffsds_zd, + gen_helper_sve_ldffhds_be_zd, + gen_helper_sve_ldffsds_be_zd, NULL, }, { gen_helper_sve_ldffbdu_zd, - gen_helper_sve_ldffhdu_zd, - gen_helper_sve_ldffsdu_zd, - gen_helper_sve_ldffddu_zd, } } } }, + gen_helper_sve_ldffhdu_be_zd, + gen_helper_sve_ldffsdu_be_zd, + gen_helper_sve_ldffdd_be_zd, } } } }, }; static bool trans_LD1_zprz(DisasContext *s, arg_LD1_zprz *a, uint32_t insn) From patchwork Thu Aug 9 04:22:06 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 143664 Delivered-To: patch@linaro.org Received: by 2002:a2e:9754:0:0:0:0:0 with SMTP id f20-v6csp1605985ljj; Wed, 8 Aug 2018 21:28:23 -0700 (PDT) X-Google-Smtp-Source: AA+uWPz6uGdhd3L8xU279u5W7Wthy+FEBAsZb+b1Zg3YFQhJKoGOA/VaqFkKg0wLyHX8n2GcANR9 X-Received: by 2002:ac8:33c5:: with SMTP id d5-v6mr482363qtb.142.1533788903337; Wed, 08 Aug 2018 21:28:23 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1533788903; cv=none; d=google.com; s=arc-20160816; b=XtjyhomvKBcRVG8COJSnAuRgX/ejWuPA78990MoB3Wm7Pwg4HT1W0k/rbriDB3j8bS Sp7fgcKQDmmWiEY9wX5JmSubZgSQhPWDhxwUOaRAmg8r2agRv7iEpN2jirFgR5b4BzCi XTwxNjBev1kjApPKgODZjYGpLs4FvFOLxyWp1uZfy/SE/2kyu/KQ+qT4xtpqkJq23iIC cW0PzRFbK3YWAQuzvEregn2KQa6NZuLRnJfKXtIaIC7EiGCwsAroN29HLCdMNm00GAfd P8uNl3hJi0o1s5SOh4JXZ1J+dlTqDzjpMHrjGTVCMXMjGCeJjklMifbVOqqn8qSgdX2P /FPQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:cc:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:subject:references:in-reply-to :message-id:date:to:from:dkim-signature:arc-authentication-results; bh=m4fzpXRRelRwa/5HNltn2wYR+ly+65acDwTBZQuGGx0=; b=X1ocvcahrfxOfJ7wfv5MaHdODq3ON4ZctDLVreTv7M/gibzRGxVPhnuG2y8J5ogD/h mVksNo0EDq6MRkhPm9c/kSp3h6eB9qWS2H5SRHgrmIrWlY9k995gj3H5AFDJjY6a5Wel cOvkDFbaFQ5gsJYs9fLRz2bgHl0UbfIwTT1mboHRWSSBExiOaeUfjrvm0o0PHqlP/H0o 40M6ERYHzSMPY49qmBjLczDR7aFYkppxm4+NOJi/OEBrULj2E6adxzwp34SDzegyf/su VWQPaUvw0jwBWHBKQqh1arVYhaSEOGmGgtfFdYvgdKfpoC7iSptjzMf6zqBQASzLEP8E fydg== ARC-Authentication-Results: i=1; mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=UvQa25tj; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [2001:4830:134:3::11]) by mx.google.com with ESMTPS id z2-v6si2684384qvc.154.2018.08.08.21.28.23 for (version=TLS1 cipher=AES128-SHA bits=128/128); Wed, 08 Aug 2018 21:28:23 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) client-ip=2001:4830:134:3::11; Authentication-Results: mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=UvQa25tj; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1]:47254 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fncYg-0006AY-M6 for patch@linaro.org; Thu, 09 Aug 2018 00:28:22 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:54166) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fncTB-00016T-VQ for qemu-devel@nongnu.org; Thu, 09 Aug 2018 00:22:44 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fncT8-0007X5-Th for qemu-devel@nongnu.org; Thu, 09 Aug 2018 00:22:41 -0400 Received: from mail-pf1-x42a.google.com ([2607:f8b0:4864:20::42a]:35014) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fncT8-0007WP-Hm for qemu-devel@nongnu.org; Thu, 09 Aug 2018 00:22:38 -0400 Received: by mail-pf1-x42a.google.com with SMTP id p12-v6so2207430pfh.2 for ; Wed, 08 Aug 2018 21:22:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=m4fzpXRRelRwa/5HNltn2wYR+ly+65acDwTBZQuGGx0=; b=UvQa25tjs7HaNzUiDQd2qhmf6LbhRe5MtoyyAEBsaqoYSxEBxi4hVuz//5SF88mVME fxcadNJndjpxZmw+OxvXbA+te5LZKgQsBYCGpJk4+sohjCuzguYaUp0/OQYBOfQJfldH Ht2cikfIQN0PQs5niBRUvfxCwmK5lHLX3Gq8M= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=m4fzpXRRelRwa/5HNltn2wYR+ly+65acDwTBZQuGGx0=; b=ZDVuawKHFLcWLeAvIR1of9RSalyscUNRKM0wZ7qto3hoYTM5YUPIpJtFAqwAAOxfRp s0M/LpKzp949SdjjgKXhcHJYbjnZXhaU0i46Bk5/HPfFICwOjZPfeRBBF67Y+4AkgEtq Ofrf3joPQGtkMEdZVi4uOoiIxmovP3FVW6C+AmLtD7Ju2Z9klpT031Xb7N1ZB/TNaZ4g XtXzHksRLHeomnOoWgMIgQDZe0xwp1y/KBgHaxV1OTU28eWDPqyf728fx5C/55xvt/DH VgvYMbeJaKcmTvcjdJSTHjRbTjrodVVz14DeTEQO5yuUc0kyM8AFpuooVqx9WXoeMM7m MHMw== X-Gm-Message-State: AOUpUlFIzzZe/JAPMTJUnjJpZ1BzfGXfZnKp+J+vef+kzZ0h510kzyWF OE0Cjk9NS/hZ9pN9QNi5FIHr5CktzIg= X-Received: by 2002:a63:710d:: with SMTP id m13-v6mr530237pgc.66.1533788556993; Wed, 08 Aug 2018 21:22:36 -0700 (PDT) Received: from cloudburst.twiddle.net (97-113-8-179.tukw.qwest.net. [97.113.8.179]) by smtp.gmail.com with ESMTPSA id m30-v6sm7355799pff.121.2018.08.08.21.22.35 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Wed, 08 Aug 2018 21:22:35 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Date: Wed, 8 Aug 2018 21:22:06 -0700 Message-Id: <20180809042206.15726-21-richard.henderson@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180809042206.15726-1-richard.henderson@linaro.org> References: <20180809042206.15726-1-richard.henderson@linaro.org> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::42a Subject: [Qemu-devel] [PATCH 20/20] target/arm: Pass TCGMemOpIdx to sve memory helpers X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: laurent.desnogues@gmail.com, peter.maydell@linaro.org, alex.bennee@linaro.org Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: "Qemu-devel" There is quite a lot of code required to compute cpu_mem_index, or even put together the full TCGMemOpIdx. This can easily be done at translation time. Signed-off-by: Richard Henderson --- target/arm/internals.h | 5 ++ target/arm/sve_helper.c | 138 +++++++++++++++++++------------------ target/arm/translate-sve.c | 67 +++++++++++------- 3 files changed, 121 insertions(+), 89 deletions(-) -- 2.17.1 Reviewed-by: Peter Maydell diff --git a/target/arm/internals.h b/target/arm/internals.h index dc9357766c..24c0444c8d 100644 --- a/target/arm/internals.h +++ b/target/arm/internals.h @@ -796,4 +796,9 @@ static inline uint32_t arm_debug_exception_fsr(CPUARMState *env) } } +/* Note make_memop_idx reserves 4 bits for mmu_idx, and MO_BSWAP is bit 3. + * Thus a TCGMemOpIdx, without any MO_ALIGN bits, fits in 8 bits. + */ +#define MEMOPIDX_SHIFT 8 + #endif diff --git a/target/arm/sve_helper.c b/target/arm/sve_helper.c index 6728862326..5bae600d17 100644 --- a/target/arm/sve_helper.c +++ b/target/arm/sve_helper.c @@ -19,6 +19,7 @@ #include "qemu/osdep.h" #include "cpu.h" +#include "internals.h" #include "exec/exec-all.h" #include "exec/cpu_ldst.h" #include "exec/helper-proto.h" @@ -3986,7 +3987,7 @@ typedef intptr_t sve_ld1_host_fn(void *vd, void *vg, void *host, * The controlling predicate is known to be true. */ typedef void sve_ld1_tlb_fn(CPUARMState *env, void *vd, intptr_t reg_off, - target_ulong vaddr, int mmu_idx, uintptr_t ra); + target_ulong vaddr, TCGMemOpIdx oi, uintptr_t ra); typedef sve_ld1_tlb_fn sve_st1_tlb_fn; /* @@ -4013,16 +4014,15 @@ static intptr_t sve_##NAME##_host(void *vd, void *vg, void *host, \ #ifdef CONFIG_SOFTMMU #define DO_LD_TLB(NAME, H, TYPEE, TYPEM, HOST, MOEND, TLB) \ static void sve_##NAME##_tlb(CPUARMState *env, void *vd, intptr_t reg_off, \ - target_ulong addr, int mmu_idx, uintptr_t ra) \ + target_ulong addr, TCGMemOpIdx oi, uintptr_t ra) \ { \ - TCGMemOpIdx oi = make_memop_idx(ctz32(sizeof(TYPEM)) | MOEND, mmu_idx); \ TYPEM val = TLB(env, addr, oi, ra); \ *(TYPEE *)(vd + H(reg_off)) = val; \ } #else #define DO_LD_TLB(NAME, H, TYPEE, TYPEM, HOST, MOEND, TLB) \ static void sve_##NAME##_tlb(CPUARMState *env, void *vd, intptr_t reg_off, \ - target_ulong addr, int mmu_idx, uintptr_t ra) \ + target_ulong addr, TCGMemOpIdx oi, uintptr_t ra) \ { \ TYPEM val = HOST(g2h(addr)); \ *(TYPEE *)(vd + H(reg_off)) = val; \ @@ -4287,11 +4287,13 @@ static void sve_ld1_r(CPUARMState *env, void *vg, const target_ulong addr, sve_ld1_host_fn *host_fn, sve_ld1_tlb_fn *tlb_fn) { - void *vd = &env->vfp.zregs[simd_data(desc)]; + const TCGMemOpIdx oi = extract32(desc, SIMD_DATA_SHIFT, MEMOPIDX_SHIFT); + const int mmu_idx = get_mmuidx(oi); + const unsigned rd = extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT, 5); + void *vd = &env->vfp.zregs[rd]; const int diffsz = esz - msz; const intptr_t reg_max = simd_oprsz(desc); const intptr_t mem_max = reg_max >> diffsz; - const int mmu_idx = cpu_mmu_index(env, false); ARMVectorReg scratch; void *host, *result; intptr_t split; @@ -4345,7 +4347,7 @@ static void sve_ld1_r(CPUARMState *env, void *vg, const target_ulong addr, * on I/O memory, it may succeed but not bring in the TLB entry. * But even then we have still made forward progress. */ - tlb_fn(env, result, reg_off, addr + mem_off, mmu_idx, retaddr); + tlb_fn(env, result, reg_off, addr + mem_off, oi, retaddr); reg_off += 1 << esz; } #endif @@ -4406,9 +4408,9 @@ static void sve_ld2_r(CPUARMState *env, void *vg, target_ulong addr, uint32_t desc, int size, uintptr_t ra, sve_ld1_tlb_fn *tlb_fn) { - const int mmu_idx = cpu_mmu_index(env, false); + const TCGMemOpIdx oi = extract32(desc, SIMD_DATA_SHIFT, MEMOPIDX_SHIFT); + const unsigned rd = extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT, 5); intptr_t i, oprsz = simd_oprsz(desc); - unsigned rd = simd_data(desc); ARMVectorReg scratch[2] = { }; set_helper_retaddr(ra); @@ -4416,8 +4418,8 @@ static void sve_ld2_r(CPUARMState *env, void *vg, target_ulong addr, uint16_t pg = *(uint16_t *)(vg + H1_2(i >> 3)); do { if (pg & 1) { - tlb_fn(env, &scratch[0], i, addr, mmu_idx, ra); - tlb_fn(env, &scratch[1], i, addr + size, mmu_idx, ra); + tlb_fn(env, &scratch[0], i, addr, oi, ra); + tlb_fn(env, &scratch[1], i, addr + size, oi, ra); } i += size, pg >>= size; addr += 2 * size; @@ -4434,9 +4436,9 @@ static void sve_ld3_r(CPUARMState *env, void *vg, target_ulong addr, uint32_t desc, int size, uintptr_t ra, sve_ld1_tlb_fn *tlb_fn) { - const int mmu_idx = cpu_mmu_index(env, false); + const TCGMemOpIdx oi = extract32(desc, SIMD_DATA_SHIFT, MEMOPIDX_SHIFT); + const unsigned rd = extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT, 5); intptr_t i, oprsz = simd_oprsz(desc); - unsigned rd = simd_data(desc); ARMVectorReg scratch[3] = { }; set_helper_retaddr(ra); @@ -4444,9 +4446,9 @@ static void sve_ld3_r(CPUARMState *env, void *vg, target_ulong addr, uint16_t pg = *(uint16_t *)(vg + H1_2(i >> 3)); do { if (pg & 1) { - tlb_fn(env, &scratch[0], i, addr, mmu_idx, ra); - tlb_fn(env, &scratch[1], i, addr + size, mmu_idx, ra); - tlb_fn(env, &scratch[2], i, addr + 2 * size, mmu_idx, ra); + tlb_fn(env, &scratch[0], i, addr, oi, ra); + tlb_fn(env, &scratch[1], i, addr + size, oi, ra); + tlb_fn(env, &scratch[2], i, addr + 2 * size, oi, ra); } i += size, pg >>= size; addr += 3 * size; @@ -4464,9 +4466,9 @@ static void sve_ld4_r(CPUARMState *env, void *vg, target_ulong addr, uint32_t desc, int size, uintptr_t ra, sve_ld1_tlb_fn *tlb_fn) { - const int mmu_idx = cpu_mmu_index(env, false); + const TCGMemOpIdx oi = extract32(desc, SIMD_DATA_SHIFT, MEMOPIDX_SHIFT); + const unsigned rd = extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT, 5); intptr_t i, oprsz = simd_oprsz(desc); - unsigned rd = simd_data(desc); ARMVectorReg scratch[4] = { }; set_helper_retaddr(ra); @@ -4474,10 +4476,10 @@ static void sve_ld4_r(CPUARMState *env, void *vg, target_ulong addr, uint16_t pg = *(uint16_t *)(vg + H1_2(i >> 3)); do { if (pg & 1) { - tlb_fn(env, &scratch[0], i, addr, mmu_idx, ra); - tlb_fn(env, &scratch[1], i, addr + size, mmu_idx, ra); - tlb_fn(env, &scratch[2], i, addr + 2 * size, mmu_idx, ra); - tlb_fn(env, &scratch[3], i, addr + 3 * size, mmu_idx, ra); + tlb_fn(env, &scratch[0], i, addr, oi, ra); + tlb_fn(env, &scratch[1], i, addr + size, oi, ra); + tlb_fn(env, &scratch[2], i, addr + 2 * size, oi, ra); + tlb_fn(env, &scratch[3], i, addr + 3 * size, oi, ra); } i += size, pg >>= size; addr += 4 * size; @@ -4572,11 +4574,13 @@ static void sve_ldff1_r(CPUARMState *env, void *vg, const target_ulong addr, sve_ld1_host_fn *host_fn, sve_ld1_tlb_fn *tlb_fn) { - void *vd = &env->vfp.zregs[simd_data(desc)]; + const TCGMemOpIdx oi = extract32(desc, SIMD_DATA_SHIFT, MEMOPIDX_SHIFT); + const int mmu_idx = get_mmuidx(oi); + const unsigned rd = extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT, 5); + void *vd = &env->vfp.zregs[rd]; const int diffsz = esz - msz; const intptr_t reg_max = simd_oprsz(desc); const intptr_t mem_max = reg_max >> diffsz; - const int mmu_idx = cpu_mmu_index(env, false); intptr_t split, reg_off, mem_off; void *host; @@ -4620,7 +4624,7 @@ static void sve_ldff1_r(CPUARMState *env, void *vg, const target_ulong addr, /* Perform one normal read, which will fault or not. * But it is likely to bring the page into the tlb. */ - tlb_fn(env, vd, reg_off, addr + mem_off, mmu_idx, retaddr); + tlb_fn(env, vd, reg_off, addr + mem_off, oi, retaddr); /* After any fault, zero any leading predicated false elts. */ swap_memzero(vd, reg_off); @@ -4649,7 +4653,8 @@ static void sve_ldnf1_r(CPUARMState *env, void *vg, const target_ulong addr, uint32_t desc, const int esz, const int msz, sve_ld1_host_fn *host_fn) { - void *vd = &env->vfp.zregs[simd_data(desc)]; + const unsigned rd = extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT, 5); + void *vd = &env->vfp.zregs[rd]; const int diffsz = esz - msz; const intptr_t reg_max = simd_oprsz(desc); const intptr_t mem_max = reg_max >> diffsz; @@ -4781,15 +4786,14 @@ DO_LDFF1_LDNF1_2(dd, 3, 3) #ifdef CONFIG_SOFTMMU #define DO_ST_TLB(NAME, H, TYPEM, HOST, MOEND, TLB) \ static void sve_##NAME##_tlb(CPUARMState *env, void *vd, intptr_t reg_off, \ - target_ulong addr, int mmu_idx, uintptr_t ra) \ + target_ulong addr, TCGMemOpIdx oi, uintptr_t ra) \ { \ - TCGMemOpIdx oi = make_memop_idx(ctz32(sizeof(TYPEM)) | MOEND, mmu_idx); \ TLB(env, addr, *(TYPEM *)(vd + H(reg_off)), oi, ra); \ } #else #define DO_ST_TLB(NAME, H, TYPEM, HOST, MOEND, TLB) \ static void sve_##NAME##_tlb(CPUARMState *env, void *vd, intptr_t reg_off, \ - target_ulong addr, int mmu_idx, uintptr_t ra) \ + target_ulong addr, TCGMemOpIdx oi, uintptr_t ra) \ { \ HOST(g2h(addr), *(TYPEM *)(vd + H(reg_off))); \ } @@ -4828,9 +4832,9 @@ static void sve_st1_r(CPUARMState *env, void *vg, target_ulong addr, const int esize, const int msize, sve_st1_tlb_fn *tlb_fn) { - const int mmu_idx = cpu_mmu_index(env, false); + const TCGMemOpIdx oi = extract32(desc, SIMD_DATA_SHIFT, MEMOPIDX_SHIFT); + const unsigned rd = extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT, 5); intptr_t i, oprsz = simd_oprsz(desc); - unsigned rd = simd_data(desc); void *vd = &env->vfp.zregs[rd]; set_helper_retaddr(ra); @@ -4838,7 +4842,7 @@ static void sve_st1_r(CPUARMState *env, void *vg, target_ulong addr, uint16_t pg = *(uint16_t *)(vg + H1_2(i >> 3)); do { if (pg & 1) { - tlb_fn(env, vd, i, addr, mmu_idx, ra); + tlb_fn(env, vd, i, addr, oi, ra); } i += esize, pg >>= esize; addr += msize; @@ -4852,9 +4856,9 @@ static void sve_st2_r(CPUARMState *env, void *vg, target_ulong addr, const int esize, const int msize, sve_st1_tlb_fn *tlb_fn) { - const int mmu_idx = cpu_mmu_index(env, false); + const TCGMemOpIdx oi = extract32(desc, SIMD_DATA_SHIFT, MEMOPIDX_SHIFT); + const unsigned rd = extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT, 5); intptr_t i, oprsz = simd_oprsz(desc); - unsigned rd = simd_data(desc); void *d1 = &env->vfp.zregs[rd]; void *d2 = &env->vfp.zregs[(rd + 1) & 31]; @@ -4863,8 +4867,8 @@ static void sve_st2_r(CPUARMState *env, void *vg, target_ulong addr, uint16_t pg = *(uint16_t *)(vg + H1_2(i >> 3)); do { if (pg & 1) { - tlb_fn(env, d1, i, addr, mmu_idx, ra); - tlb_fn(env, d2, i, addr + msize, mmu_idx, ra); + tlb_fn(env, d1, i, addr, oi, ra); + tlb_fn(env, d2, i, addr + msize, oi, ra); } i += esize, pg >>= esize; addr += 2 * msize; @@ -4878,9 +4882,9 @@ static void sve_st3_r(CPUARMState *env, void *vg, target_ulong addr, const int esize, const int msize, sve_st1_tlb_fn *tlb_fn) { - const int mmu_idx = cpu_mmu_index(env, false); + const TCGMemOpIdx oi = extract32(desc, SIMD_DATA_SHIFT, MEMOPIDX_SHIFT); + const unsigned rd = extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT, 5); intptr_t i, oprsz = simd_oprsz(desc); - unsigned rd = simd_data(desc); void *d1 = &env->vfp.zregs[rd]; void *d2 = &env->vfp.zregs[(rd + 1) & 31]; void *d3 = &env->vfp.zregs[(rd + 2) & 31]; @@ -4890,9 +4894,9 @@ static void sve_st3_r(CPUARMState *env, void *vg, target_ulong addr, uint16_t pg = *(uint16_t *)(vg + H1_2(i >> 3)); do { if (pg & 1) { - tlb_fn(env, d1, i, addr, mmu_idx, ra); - tlb_fn(env, d2, i, addr + msize, mmu_idx, ra); - tlb_fn(env, d3, i, addr + 2 * msize, mmu_idx, ra); + tlb_fn(env, d1, i, addr, oi, ra); + tlb_fn(env, d2, i, addr + msize, oi, ra); + tlb_fn(env, d3, i, addr + 2 * msize, oi, ra); } i += esize, pg >>= esize; addr += 3 * msize; @@ -4906,9 +4910,9 @@ static void sve_st4_r(CPUARMState *env, void *vg, target_ulong addr, const int esize, const int msize, sve_st1_tlb_fn *tlb_fn) { - const int mmu_idx = cpu_mmu_index(env, false); + const TCGMemOpIdx oi = extract32(desc, SIMD_DATA_SHIFT, MEMOPIDX_SHIFT); + const unsigned rd = extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT, 5); intptr_t i, oprsz = simd_oprsz(desc); - unsigned rd = simd_data(desc); void *d1 = &env->vfp.zregs[rd]; void *d2 = &env->vfp.zregs[(rd + 1) & 31]; void *d3 = &env->vfp.zregs[(rd + 2) & 31]; @@ -4919,10 +4923,10 @@ static void sve_st4_r(CPUARMState *env, void *vg, target_ulong addr, uint16_t pg = *(uint16_t *)(vg + H1_2(i >> 3)); do { if (pg & 1) { - tlb_fn(env, d1, i, addr, mmu_idx, ra); - tlb_fn(env, d2, i, addr + msize, mmu_idx, ra); - tlb_fn(env, d3, i, addr + 2 * msize, mmu_idx, ra); - tlb_fn(env, d4, i, addr + 3 * msize, mmu_idx, ra); + tlb_fn(env, d1, i, addr, oi, ra); + tlb_fn(env, d2, i, addr + msize, oi, ra); + tlb_fn(env, d3, i, addr + 2 * msize, oi, ra); + tlb_fn(env, d4, i, addr + 3 * msize, oi, ra); } i += esize, pg >>= esize; addr += 4 * msize; @@ -5015,9 +5019,9 @@ static void sve_ld1_zs(CPUARMState *env, void *vd, void *vg, void *vm, target_ulong base, uint32_t desc, uintptr_t ra, zreg_off_fn *off_fn, sve_ld1_tlb_fn *tlb_fn) { - const int mmu_idx = cpu_mmu_index(env, false); + const TCGMemOpIdx oi = extract32(desc, SIMD_DATA_SHIFT, MEMOPIDX_SHIFT); + const int scale = extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT, 2); intptr_t i, oprsz = simd_oprsz(desc); - unsigned scale = simd_data(desc); ARMVectorReg scratch = { }; set_helper_retaddr(ra); @@ -5026,7 +5030,7 @@ static void sve_ld1_zs(CPUARMState *env, void *vd, void *vg, void *vm, do { if (pg & 1) { target_ulong off = off_fn(vm, i); - tlb_fn(env, &scratch, i, base + (off << scale), mmu_idx, ra); + tlb_fn(env, &scratch, i, base + (off << scale), oi, ra); } i += 4, pg >>= 4; } while (i & 15); @@ -5041,9 +5045,9 @@ static void sve_ld1_zd(CPUARMState *env, void *vd, void *vg, void *vm, target_ulong base, uint32_t desc, uintptr_t ra, zreg_off_fn *off_fn, sve_ld1_tlb_fn *tlb_fn) { - const int mmu_idx = cpu_mmu_index(env, false); + const TCGMemOpIdx oi = extract32(desc, SIMD_DATA_SHIFT, MEMOPIDX_SHIFT); + const int scale = extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT, 2); intptr_t i, oprsz = simd_oprsz(desc) / 8; - unsigned scale = simd_data(desc); ARMVectorReg scratch = { }; set_helper_retaddr(ra); @@ -5051,7 +5055,7 @@ static void sve_ld1_zd(CPUARMState *env, void *vd, void *vg, void *vm, uint8_t pg = *(uint8_t *)(vg + H1(i)); if (pg & 1) { target_ulong off = off_fn(vm, i * 8); - tlb_fn(env, &scratch, i * 8, base + (off << scale), mmu_idx, ra); + tlb_fn(env, &scratch, i * 8, base + (off << scale), oi, ra); } } set_helper_retaddr(0); @@ -5157,7 +5161,7 @@ typedef bool sve_ld1_nf_fn(CPUARMState *env, void *vd, intptr_t reg_off, #ifdef CONFIG_SOFTMMU #define DO_LD_NF(NAME, H, TYPEE, TYPEM, HOST) \ static bool sve_ld##NAME##_nf(CPUARMState *env, void *vd, intptr_t reg_off, \ - target_ulong addr, int mmu_idx) \ + target_ulong addr, int mmu_idx) \ { \ target_ulong next_page = -(addr | TARGET_PAGE_MASK); \ if (likely(next_page - addr >= sizeof(TYPEM))) { \ @@ -5216,9 +5220,10 @@ static inline void sve_ldff1_zs(CPUARMState *env, void *vd, void *vg, void *vm, zreg_off_fn *off_fn, sve_ld1_tlb_fn *tlb_fn, sve_ld1_nf_fn *nonfault_fn) { - const int mmu_idx = cpu_mmu_index(env, false); + const TCGMemOpIdx oi = extract32(desc, SIMD_DATA_SHIFT, MEMOPIDX_SHIFT); + const int mmu_idx = get_mmuidx(oi); + const int scale = extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT, 2); intptr_t reg_off, reg_max = simd_oprsz(desc); - unsigned scale = simd_data(desc); target_ulong addr; /* Skip to the first true predicate. */ @@ -5228,7 +5233,7 @@ static inline void sve_ldff1_zs(CPUARMState *env, void *vd, void *vg, void *vm, set_helper_retaddr(ra); addr = off_fn(vm, reg_off); addr = base + (addr << scale); - tlb_fn(env, vd, reg_off, addr, mmu_idx, ra); + tlb_fn(env, vd, reg_off, addr, oi, ra); /* The rest of the reads will be non-faulting. */ set_helper_retaddr(0); @@ -5257,9 +5262,10 @@ static inline void sve_ldff1_zd(CPUARMState *env, void *vd, void *vg, void *vm, zreg_off_fn *off_fn, sve_ld1_tlb_fn *tlb_fn, sve_ld1_nf_fn *nonfault_fn) { - const int mmu_idx = cpu_mmu_index(env, false); + const TCGMemOpIdx oi = extract32(desc, SIMD_DATA_SHIFT, MEMOPIDX_SHIFT); + const int mmu_idx = get_mmuidx(oi); + const int scale = extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT, 2); intptr_t reg_off, reg_max = simd_oprsz(desc); - unsigned scale = simd_data(desc); target_ulong addr; /* Skip to the first true predicate. */ @@ -5269,7 +5275,7 @@ static inline void sve_ldff1_zd(CPUARMState *env, void *vd, void *vg, void *vm, set_helper_retaddr(ra); addr = off_fn(vm, reg_off); addr = base + (addr << scale); - tlb_fn(env, vd, reg_off, addr, mmu_idx, ra); + tlb_fn(env, vd, reg_off, addr, oi, ra); /* The rest of the reads will be non-faulting. */ set_helper_retaddr(0); @@ -5381,9 +5387,9 @@ static void sve_st1_zs(CPUARMState *env, void *vd, void *vg, void *vm, target_ulong base, uint32_t desc, uintptr_t ra, zreg_off_fn *off_fn, sve_ld1_tlb_fn *tlb_fn) { - const int mmu_idx = cpu_mmu_index(env, false); + const TCGMemOpIdx oi = extract32(desc, SIMD_DATA_SHIFT, MEMOPIDX_SHIFT); + const int scale = extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT, 2); intptr_t i, oprsz = simd_oprsz(desc); - unsigned scale = simd_data(desc); set_helper_retaddr(ra); for (i = 0; i < oprsz; ) { @@ -5391,7 +5397,7 @@ static void sve_st1_zs(CPUARMState *env, void *vd, void *vg, void *vm, do { if (pg & 1) { target_ulong off = off_fn(vm, i); - tlb_fn(env, vd, i, base + (off << scale), mmu_idx, ra); + tlb_fn(env, vd, i, base + (off << scale), oi, ra); } i += 4, pg >>= 4; } while (i & 15); @@ -5403,16 +5409,16 @@ static void sve_st1_zd(CPUARMState *env, void *vd, void *vg, void *vm, target_ulong base, uint32_t desc, uintptr_t ra, zreg_off_fn *off_fn, sve_ld1_tlb_fn *tlb_fn) { - const int mmu_idx = cpu_mmu_index(env, false); + const TCGMemOpIdx oi = extract32(desc, SIMD_DATA_SHIFT, MEMOPIDX_SHIFT); + const int scale = extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT, 2); intptr_t i, oprsz = simd_oprsz(desc) / 8; - unsigned scale = simd_data(desc); set_helper_retaddr(ra); for (i = 0; i < oprsz; i++) { uint8_t pg = *(uint8_t *)(vg + H1(i)); if (pg & 1) { target_ulong off = off_fn(vm, i * 8); - tlb_fn(env, vd, i * 8, base + (off << scale), mmu_idx, ra); + tlb_fn(env, vd, i * 8, base + (off << scale), oi, ra); } } set_helper_retaddr(0); diff --git a/target/arm/translate-sve.c b/target/arm/translate-sve.c index 20492e9b8b..05ba0518c8 100644 --- a/target/arm/translate-sve.c +++ b/target/arm/translate-sve.c @@ -4600,25 +4600,34 @@ static const uint8_t dtype_esz[16] = { 3, 2, 1, 3 }; +static TCGMemOpIdx sve_memopidx(DisasContext *s, int dtype) +{ + return make_memop_idx(s->be_data | dtype_mop[dtype], get_mem_index(s)); +} + static void do_mem_zpa(DisasContext *s, int zt, int pg, TCGv_i64 addr, - gen_helper_gvec_mem *fn) + int dtype, gen_helper_gvec_mem *fn) { unsigned vsz = vec_full_reg_size(s); TCGv_ptr t_pg; - TCGv_i32 desc; + TCGv_i32 t_desc; + int desc; /* For e.g. LD4, there are not enough arguments to pass all 4 * registers as pointers, so encode the regno into the data field. * For consistency, do this even for LD1. */ - desc = tcg_const_i32(simd_desc(vsz, vsz, zt)); + desc = sve_memopidx(s, dtype); + desc |= zt << MEMOPIDX_SHIFT; + desc = simd_desc(vsz, vsz, desc); + t_desc = tcg_const_i32(desc); t_pg = tcg_temp_new_ptr(); tcg_gen_addi_ptr(t_pg, cpu_env, pred_full_reg_offset(s, pg)); - fn(cpu_env, t_pg, addr, desc); + fn(cpu_env, t_pg, addr, t_desc); tcg_temp_free_ptr(t_pg); - tcg_temp_free_i32(desc); + tcg_temp_free_i32(t_desc); } static void do_ld_zpa(DisasContext *s, int zt, int pg, @@ -4681,7 +4690,7 @@ static void do_ld_zpa(DisasContext *s, int zt, int pg, * accessible via the instruction encoding. */ assert(fn != NULL); - do_mem_zpa(s, zt, pg, addr, fn); + do_mem_zpa(s, zt, pg, addr, dtype, fn); } static bool trans_LD_zprr(DisasContext *s, arg_rprr_load *a, uint32_t insn) @@ -4763,7 +4772,8 @@ static bool trans_LDFF1_zprr(DisasContext *s, arg_rprr_load *a, uint32_t insn) TCGv_i64 addr = new_tmp_a64(s); tcg_gen_shli_i64(addr, cpu_reg(s, a->rm), dtype_msz(a->dtype)); tcg_gen_add_i64(addr, addr, cpu_reg_sp(s, a->rn)); - do_mem_zpa(s, a->rd, a->pg, addr, fns[s->be_data == MO_BE][a->dtype]); + do_mem_zpa(s, a->rd, a->pg, addr, a->dtype, + fns[s->be_data == MO_BE][a->dtype]); } return true; } @@ -4821,7 +4831,8 @@ static bool trans_LDNF1_zpri(DisasContext *s, arg_rpri_load *a, uint32_t insn) TCGv_i64 addr = new_tmp_a64(s); tcg_gen_addi_i64(addr, cpu_reg_sp(s, a->rn), off); - do_mem_zpa(s, a->rd, a->pg, addr, fns[s->be_data == MO_BE][a->dtype]); + do_mem_zpa(s, a->rd, a->pg, addr, a->dtype, + fns[s->be_data == MO_BE][a->dtype]); } return true; } @@ -4836,11 +4847,14 @@ static void do_ldrq(DisasContext *s, int zt, int pg, TCGv_i64 addr, int msz) }; unsigned vsz = vec_full_reg_size(s); TCGv_ptr t_pg; - TCGv_i32 desc; - int poff; + TCGv_i32 t_desc; + int desc, poff; /* Load the first quadword using the normal predicated load helpers. */ - desc = tcg_const_i32(simd_desc(16, 16, zt)); + desc = sve_memopidx(s, msz_dtype(msz)); + desc |= zt << MEMOPIDX_SHIFT; + desc = simd_desc(16, 16, desc); + t_desc = tcg_const_i32(desc); poff = pred_full_reg_offset(s, pg); if (vsz > 16) { @@ -4864,10 +4878,10 @@ static void do_ldrq(DisasContext *s, int zt, int pg, TCGv_i64 addr, int msz) t_pg = tcg_temp_new_ptr(); tcg_gen_addi_ptr(t_pg, cpu_env, poff); - fns[s->be_data == MO_BE][msz](cpu_env, t_pg, addr, desc); + fns[s->be_data == MO_BE][msz](cpu_env, t_pg, addr, t_desc); tcg_temp_free_ptr(t_pg); - tcg_temp_free_i32(desc); + tcg_temp_free_i32(t_desc); /* Replicate that first quadword. */ if (vsz > 16) { @@ -5019,7 +5033,7 @@ static void do_st_zpa(DisasContext *s, int zt, int pg, TCGv_i64 addr, fn = fn_multiple[be][nreg - 1][msz]; } assert(fn != NULL); - do_mem_zpa(s, zt, pg, addr, fn); + do_mem_zpa(s, zt, pg, addr, msz_dtype(msz), fn); } static bool trans_ST_zprr(DisasContext *s, arg_rprr_store *a, uint32_t insn) @@ -5057,24 +5071,31 @@ static bool trans_ST_zpri(DisasContext *s, arg_rpri_store *a, uint32_t insn) *** SVE gather loads / scatter stores */ -static void do_mem_zpz(DisasContext *s, int zt, int pg, int zm, int scale, - TCGv_i64 scalar, gen_helper_gvec_mem_scatter *fn) +static void do_mem_zpz(DisasContext *s, int zt, int pg, int zm, + int scale, TCGv_i64 scalar, int msz, + gen_helper_gvec_mem_scatter *fn) { unsigned vsz = vec_full_reg_size(s); - TCGv_i32 desc = tcg_const_i32(simd_desc(vsz, vsz, scale)); TCGv_ptr t_zm = tcg_temp_new_ptr(); TCGv_ptr t_pg = tcg_temp_new_ptr(); TCGv_ptr t_zt = tcg_temp_new_ptr(); + TCGv_i32 t_desc; + int desc; + + desc = sve_memopidx(s, msz_dtype(msz)); + desc |= scale << MEMOPIDX_SHIFT; + desc = simd_desc(vsz, vsz, desc); + t_desc = tcg_const_i32(desc); tcg_gen_addi_ptr(t_pg, cpu_env, pred_full_reg_offset(s, pg)); tcg_gen_addi_ptr(t_zm, cpu_env, vec_full_reg_offset(s, zm)); tcg_gen_addi_ptr(t_zt, cpu_env, vec_full_reg_offset(s, zt)); - fn(cpu_env, t_zt, t_pg, t_zm, scalar, desc); + fn(cpu_env, t_zt, t_pg, t_zm, scalar, t_desc); tcg_temp_free_ptr(t_zt); tcg_temp_free_ptr(t_zm); tcg_temp_free_ptr(t_pg); - tcg_temp_free_i32(desc); + tcg_temp_free_i32(t_desc); } /* Indexed by [be][ff][xs][u][msz]. */ @@ -5263,7 +5284,7 @@ static bool trans_LD1_zprz(DisasContext *s, arg_LD1_zprz *a, uint32_t insn) assert(fn != NULL); do_mem_zpz(s, a->rd, a->pg, a->rm, a->scale * a->msz, - cpu_reg_sp(s, a->rn), fn); + cpu_reg_sp(s, a->rn), a->msz, fn); return true; } @@ -5294,7 +5315,7 @@ static bool trans_LD1_zpiz(DisasContext *s, arg_LD1_zpiz *a, uint32_t insn) * by loading the immediate into the scalar parameter. */ imm = tcg_const_i64(a->imm << a->msz); - do_mem_zpz(s, a->rd, a->pg, a->rn, 0, imm, fn); + do_mem_zpz(s, a->rd, a->pg, a->rn, 0, imm, a->msz, fn); tcg_temp_free_i64(imm); return true; } @@ -5369,7 +5390,7 @@ static bool trans_ST1_zprz(DisasContext *s, arg_ST1_zprz *a, uint32_t insn) g_assert_not_reached(); } do_mem_zpz(s, a->rd, a->pg, a->rm, a->scale * a->msz, - cpu_reg_sp(s, a->rn), fn); + cpu_reg_sp(s, a->rn), a->msz, fn); return true; } @@ -5400,7 +5421,7 @@ static bool trans_ST1_zpiz(DisasContext *s, arg_ST1_zpiz *a, uint32_t insn) * by loading the immediate into the scalar parameter. */ imm = tcg_const_i64(a->imm << a->msz); - do_mem_zpz(s, a->rd, a->pg, a->rn, 0, imm, fn); + do_mem_zpz(s, a->rd, a->pg, a->rn, 0, imm, a->msz, fn); tcg_temp_free_i64(imm); return true; }