From patchwork Wed Apr 22 04:32:52 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 185655 Delivered-To: patch@linaro.org Received: by 2002:a92:3d9a:0:0:0:0:0 with SMTP id k26csp296842ilf; Tue, 21 Apr 2020 21:33:33 -0700 (PDT) X-Google-Smtp-Source: APiQypL4LjGBaIzKsNU0jnQmkx0pawzuIX2obn2451jQqt7NVUDvmF9EOI/blA7+aCTSb1UN2scM X-Received: by 2002:ac8:6d2c:: with SMTP id r12mr24783923qtu.355.1587530012930; Tue, 21 Apr 2020 21:33:32 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1587530012; cv=none; d=google.com; s=arc-20160816; b=mb7BYybrWROuojkf4fWxZkLuQ3A4x8T3VvXRTjDzW6O8BhH3i83uL0Du6fkhm4/o5a ETE1Z4LCKWpelA3V/JK5v5tH++vLkbV59X+grSQBtAdPklPf0tgcoJbd7YsZ4SK6jdhh RJYAOWqd545gnBoW2AF+T4VhpnqWca/+Vnioxyj7mrE8//zb699RoN94AMGCaZ2ZVps9 nZBUIuuH6lj5uQU5MVofsFacOStbtWAD2WMuCKG8zqaDqtIoDHZj/outrACGlCB2DUNS 9iCo8EW5Z/NcrLriae/2tXbeq3UVN1erMVm6J59O9H9TmlQYoirchP0VBx8D9/zPNQJC sQag== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:cc:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:content-transfer-encoding :mime-version:references:in-reply-to:message-id:date:subject:to:from :dkim-signature; bh=wfD3qzrTNTYWcIz1faBmR4+bRJgKxyAneWjssHLJ1no=; b=Vu1cmu5UsvspqfO72hBuNG2nR5opE7SlW2cz1xAQWL1nNVhGWapi0RTrRXZed2+zUw KVqzYeoFgHK94HwTC2eF4Ui+pY0P5SUdo43nJjpqLpLEzo0ho43ANuUbC4o/xSRunZNZ SzW5z+nO9fvtyQ53L9iOuEzWBjj3AB50l/pLY/iTFPted2n2b6S7QDz89vURhLi28fW+ htxumBmoME8T+E/9EvhF7fsPLSbeOBAEhnd/T1Ty3hf4Koy8i+tsXWccK1Q1dLbjP+Vs kQyzT5zjUOYdkUJE0pPwmWWaSdhQuxZQc7CDARYgUJzJrSXqn6u8/EyTtUDL4QxhVKON e+QQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=fOwcxgGf; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [209.51.188.17]) by mx.google.com with ESMTPS id c76si2630585qkg.222.2020.04.21.21.33.32 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Tue, 21 Apr 2020 21:33:32 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=fOwcxgGf; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1]:41130 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jR74m-0000Qg-0w for patch@linaro.org; Wed, 22 Apr 2020 00:33:32 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:50000) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jR74V-0000Nf-Uj for qemu-devel@nongnu.org; Wed, 22 Apr 2020 00:33:16 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.90_1) (envelope-from ) id 1jR74V-0004dD-Dn for qemu-devel@nongnu.org; Wed, 22 Apr 2020 00:33:15 -0400 Received: from mail-pf1-x444.google.com ([2607:f8b0:4864:20::444]:43317) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1jR74U-0004ZV-S4 for qemu-devel@nongnu.org; Wed, 22 Apr 2020 00:33:15 -0400 Received: by mail-pf1-x444.google.com with SMTP id v63so434008pfb.10 for ; Tue, 21 Apr 2020 21:33:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=wfD3qzrTNTYWcIz1faBmR4+bRJgKxyAneWjssHLJ1no=; b=fOwcxgGfIr5FBkD3yUmUPjqtuEW1Da+0ABHfYwu+6a3XUr9+WZaH9sFdq4ZRZNUR5N odFKdauq5u/AybXpMNThlWxPRVyzv7RdzEQMgxwm+fe0YHw29PP+jHZL/ARxqSkSXPXl F49HXdt6yh9snFAdIe2AgMKZZAzL8I+qNZcskD7fzWQOxpmtC90Dc+l3hqHUcIGEW9RQ iGI9D5b5PuSNHzmbceGiKYQg3EGCbhQxWsu8C690ZNZbNF7Zd/FnQAtxyQzV7nWH+A/J gAO4THKysdhZ0a7M1RwaoZNxWLCzofTiVNWGi0aPBxA/vyXKjaRaMz1wkoXC0yHxSxRA 2jXw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=wfD3qzrTNTYWcIz1faBmR4+bRJgKxyAneWjssHLJ1no=; b=UKPnn/WpoF0UhaeFaaPs6CveOzkIHSSrV8IwhO0tM3WX2SsNYB/LeMhQTxH5lKrhB1 3X1miN6QOfEL8UC9qtiH/r8V1xCdkabH4nxlosOx+cvfqaQpOQkNhvc/YqkxWDxJfnkF tCnP1h4I9JnfOxv1Pgz5UtHGV1rB4/ey5MTkH9/97WjN9uj08vu+XHg8i1EO6hPMzGb2 iz22j4Kmr2DwGGQ53l5xyRir3drmZwuY2NF8nkh8PgptBZyoXk3GMW/WenFy8ZAgl38F kumyOAm9YfRZt+HKN+wl64lwYZmv9cbSxuYAxv3pEL02HY60fgVUZR3MyHiZdRvv10dg AzKQ== X-Gm-Message-State: AGi0PuZO1RBbo4X4TQn0n4wz7CBst4GOoKLDCReohyubCj4rDBR1CZ0g bN1hd/yC9Z1ItTqv0+68Mbd3lT9iXPE= X-Received: by 2002:a63:4646:: with SMTP id v6mr25769727pgk.426.1587529992763; Tue, 21 Apr 2020 21:33:12 -0700 (PDT) Received: from localhost.localdomain (174-21-149-226.tukw.qwest.net. [174.21.149.226]) by smtp.gmail.com with ESMTPSA id l137sm4129613pfd.107.2020.04.21.21.33.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 21 Apr 2020 21:33:12 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Subject: [PATCH v3 01/18] exec: Add block comments for watchpoint routines Date: Tue, 21 Apr 2020 21:32:52 -0700 Message-Id: <20200422043309.18430-2-richard.henderson@linaro.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200422043309.18430-1-richard.henderson@linaro.org> References: <20200422043309.18430-1-richard.henderson@linaro.org> MIME-Version: 1.0 Received-SPF: pass client-ip=2607:f8b0:4864:20::444; envelope-from=richard.henderson@linaro.org; helo=mail-pf1-x444.google.com X-detected-operating-system: by eggs.gnu.org: Error: [-] PROGRAM ABORT : Malformed IPv6 address (bad octet value). Location : parse_addr6(), p0f-client.c:67 X-Received-From: 2607:f8b0:4864:20::444 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: peter.maydell@linaro.org, qemu-arm@nongnu.org Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: "Qemu-devel" Signed-off-by: Richard Henderson --- include/hw/core/cpu.h | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) -- 2.20.1 Reviewed-by: Peter Maydell diff --git a/include/hw/core/cpu.h b/include/hw/core/cpu.h index 5bf94d28cf..07f7698155 100644 --- a/include/hw/core/cpu.h +++ b/include/hw/core/cpu.h @@ -1100,8 +1100,31 @@ int cpu_watchpoint_remove(CPUState *cpu, vaddr addr, vaddr len, int flags); void cpu_watchpoint_remove_by_ref(CPUState *cpu, CPUWatchpoint *watchpoint); void cpu_watchpoint_remove_all(CPUState *cpu, int mask); + +/** + * cpu_check_watchpoint: + * @cpu: cpu context + * @addr: guest virtual address + * @len: access length + * @attrs: memory access attributes + * @flags: watchpoint access type + * @ra: unwind return address + * + * Check for a watchpoint hit in [addr, addr+len) of the type + * specified by @flags. Exit via exception with a hit. + */ void cpu_check_watchpoint(CPUState *cpu, vaddr addr, vaddr len, MemTxAttrs attrs, int flags, uintptr_t ra); + +/** + * cpu_watchpoint_address_matches: + * @cpu: cpu context + * @addr: guest virtual address + * @len: access length + * + * Return the watchpoint flags that apply to [addr, addr+len). + * If no watchpoint is registered for the range, the result is 0. + */ int cpu_watchpoint_address_matches(CPUState *cpu, vaddr addr, vaddr len); #endif From patchwork Wed Apr 22 04:32:53 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 185656 Delivered-To: patch@linaro.org Received: by 2002:a92:3d9a:0:0:0:0:0 with SMTP id k26csp296998ilf; Tue, 21 Apr 2020 21:33:46 -0700 (PDT) X-Google-Smtp-Source: APiQypL3bGoaDGMqGDX89433PfVFCRw6/CJmy4TDSc12CABXNPNHT2LdhiH8my/gl6X7lzmQBHO8 X-Received: by 2002:ac8:4e88:: with SMTP id 8mr24692368qtp.82.1587530026507; Tue, 21 Apr 2020 21:33:46 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1587530026; cv=none; d=google.com; s=arc-20160816; b=ooqtY7T3Vg+5qbM1S0jzzPLzJ78cNoF2BFNGgJUY1IgoYClTBUBOun9BUUTo8HTwYa URi04oJnWKGPkj0kKudWCSvxSWvwtoLRUYAvdktSKsVvCUIgnSGH8vs/4Z5P+/DfELsc VRbsjs4EWWn49AQ8Pq65Hwvt7Li8AACGA7ASMtnVZ2jGdUylwLm2sgXQzuw/rj8nMAmC MZ1W/on6iPQWA/pyT0Z8Ke8vZ4dbJtgriBG1NuluvKmCPc4vajPrKvvsxKojs+8fFTt2 jtnazFRgaP/75voKcSTEahMVwzEZIpfV41qLNIFB/I9laHl1sdbYKKr9erLVosqIKacF NtKg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:cc:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:content-transfer-encoding :mime-version:references:in-reply-to:message-id:date:subject:to:from :dkim-signature; bh=iKZvtr9IPAzyq9GBvX8e6ybeQtJDWGAh30ejEfYLqNo=; b=nUnC1JTWKLi0HaujGbwXhtNVed+VJyf7BWTDxjstLARpLUVsfqigdJWIlDV+/WgNZq Q+yEFkjbcB1ni98kYTO5REIX3n0qYC/em4dXFFikCuOCWSIWQKFX/1BVUMIvPSDlp+T9 kI6Os5OD7z47sMnPeuBZFSfmG7moIcsDqW8Q69bTxbS91mDOGat/4QLV1ye5qGMuaf4z ThdH7nDbSdNjN1s25xZpxE2jzCYAYtqtoO+ErO0V3H2QJxSK1DHc6LCdvVzVgSzUHuvT 2wF+FKMAUTFHnfRJA29DOzKlRC2Z9NE3+jPi3lqcLy6AtCiaIJpZZSSJ7AoldT5+l34T 8jLA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=O8YdVlUF; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [209.51.188.17]) by mx.google.com with ESMTPS id b16si2415120qvk.183.2020.04.21.21.33.46 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Tue, 21 Apr 2020 21:33:46 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=O8YdVlUF; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1]:41202 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jR74z-0000qt-Sw for patch@linaro.org; Wed, 22 Apr 2020 00:33:45 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:50050) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jR74X-0000PO-0I for qemu-devel@nongnu.org; Wed, 22 Apr 2020 00:33:17 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.90_1) (envelope-from ) id 1jR74W-0004g3-4W for qemu-devel@nongnu.org; Wed, 22 Apr 2020 00:33:16 -0400 Received: from mail-pj1-x1041.google.com ([2607:f8b0:4864:20::1041]:34975) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1jR74V-0004c7-Nd for qemu-devel@nongnu.org; Wed, 22 Apr 2020 00:33:15 -0400 Received: by mail-pj1-x1041.google.com with SMTP id ms17so378984pjb.0 for ; Tue, 21 Apr 2020 21:33:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=iKZvtr9IPAzyq9GBvX8e6ybeQtJDWGAh30ejEfYLqNo=; b=O8YdVlUFMeAyiQckNOrUjTA+kG9wxt/Vpotnkg7s2jSHuklfDef4zCNAvnL92Kigk9 /YcwhcFjzqs7MJlzEy+8erOYdtH/uYgiWm60rrvtmf1M2rAzGQqJQTtQvXAw9JbgpUHE cw4CI0vWTkGYmmsVrve8uUfTWPLHT33mmXnmPdqIBsw3gXBbPrRwGx8Wo8fjLZzAg9R/ EewiN/3oJH4I0oSwcEs0aTyDlJ16LOQSgE6q0OHnfPwafr8yX6Mb2dde1AzkvfNqgckp spvobVVvQHyo1JF1wa3lsT9ANXCRqJPfmTxyIOdSR5+MZXGfpHUtMFF+9IY4+Bd5iWuF 2ZRg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=iKZvtr9IPAzyq9GBvX8e6ybeQtJDWGAh30ejEfYLqNo=; b=cZ7nUdH8Bp9WxqumKcqMtVXTJjSmTyZoK0VvrQcxGNw/u4iRaG+HBKz47OBypXQJAl jggbjWwTpVF+pAheMniyhjlefjld8gdHgOHVlsF9kUpXi+UfQ+olsF03O+28Qk2D7PG7 1IwYtUhpcToH4aytpButDs3fbkx8WMB18F+RmH88RZuLuh8lsZ6pkOawyxRwKvsIqX3C qRLp736+x3YlPrLuvU9yji+nxvX8YWCnKF6olnlqh598V1aaD603MJgMUrSJQW1llIhm jtlBFasV7cANuvxr/48d1lEZTLBBxxsNs8e5MvnSNice363LsrL8l1AxXoXVtLuovaE7 dFbg== X-Gm-Message-State: AGi0PuYBPqD8tr72Rhex55aKDM+TsFZLodA4ZM74JUvgnYefKAJAuBkT JEsSy2pRO4nsH475XT76V815270kyh0= X-Received: by 2002:a17:902:d201:: with SMTP id t1mr3566571ply.328.1587529993856; Tue, 21 Apr 2020 21:33:13 -0700 (PDT) Received: from localhost.localdomain (174-21-149-226.tukw.qwest.net. [174.21.149.226]) by smtp.gmail.com with ESMTPSA id l137sm4129613pfd.107.2020.04.21.21.33.12 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 21 Apr 2020 21:33:13 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Subject: [PATCH v3 02/18] exec: Fix cpu_watchpoint_address_matches address length Date: Tue, 21 Apr 2020 21:32:53 -0700 Message-Id: <20200422043309.18430-3-richard.henderson@linaro.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200422043309.18430-1-richard.henderson@linaro.org> References: <20200422043309.18430-1-richard.henderson@linaro.org> MIME-Version: 1.0 Received-SPF: pass client-ip=2607:f8b0:4864:20::1041; envelope-from=richard.henderson@linaro.org; helo=mail-pj1-x1041.google.com X-detected-operating-system: by eggs.gnu.org: Error: [-] PROGRAM ABORT : Malformed IPv6 address (bad octet value). Location : parse_addr6(), p0f-client.c:67 X-Received-From: 2607:f8b0:4864:20::1041 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: peter.maydell@linaro.org, qemu-arm@nongnu.org Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: "Qemu-devel" The only caller of cpu_watchpoint_address_matches passes TARGET_PAGE_SIZE, so the bug is not currently visible. Signed-off-by: Richard Henderson --- exec.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) -- 2.20.1 Reviewed-by: Peter Maydell diff --git a/exec.c b/exec.c index 2874bb5088..5162f0d12f 100644 --- a/exec.c +++ b/exec.c @@ -1127,7 +1127,7 @@ int cpu_watchpoint_address_matches(CPUState *cpu, vaddr addr, vaddr len) int ret = 0; QTAILQ_FOREACH(wp, &cpu->watchpoints, entry) { - if (watchpoint_address_matches(wp, addr, TARGET_PAGE_SIZE)) { + if (watchpoint_address_matches(wp, addr, len)) { ret |= wp->flags; } } From patchwork Wed Apr 22 04:32:54 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 185658 Delivered-To: patch@linaro.org Received: by 2002:a92:3d9a:0:0:0:0:0 with SMTP id k26csp297976ilf; Tue, 21 Apr 2020 21:35:09 -0700 (PDT) X-Google-Smtp-Source: APiQypJ6hJjXdn1+Vd6Ri7nazKGv9BkA5dz7GWH6oh30KoLBvVVYyezcGPC6Vvdc/0m9z6b97ffQ X-Received: by 2002:ac8:6d3b:: with SMTP id r27mr25694391qtu.55.1587530109814; Tue, 21 Apr 2020 21:35:09 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1587530109; cv=none; d=google.com; s=arc-20160816; b=xV1IkeLNOSXJ3nP26+pBiAlgM0ssNWLib3WwddmPWuBZqEXCNY7nWFx5Rupv+WxsU+ JsgvlVRT0SnUgaVsCkhgG2YJ6EYm6p33hsy04v0FqsFxe0TMNf2XjhxY0x2hb3vGbFPw Q9TDMdt5AGrLe24hh3J2pKcXQ7fQ7VWsnpWUk/Xov5+ImFSOcrOsU1kgPnWwwdXRJ83y aWrG5jRQ/QDmhqtMo/WkL+jFoLQJowzzYc1c7uUuSSJMyg5Lcf++qgh5ksset9IUvVKO nV6hIw+mkjAvvfnCB65bNMdIQBGnMtAU7X5sDYNexFMGm5/hJkwDdML3HKtNtTIdvJJY M0AA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:cc:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:content-transfer-encoding :mime-version:references:in-reply-to:message-id:date:subject:to:from :dkim-signature; bh=Ar4ktbgi8ho4D2K0AlAA7ABuossIOEQqKwU/YvDdwbY=; b=HWcd7KbxPjwxZSZaLYe85zU1L3BN13/3972v22x56KSCDQmeeq3Rec17tD3afxwrny qlki3Zy8tQgFHRUUf86OvPQP2ygklgL0MjtRcw6OWQ6tR8I7aXspm6CL90+Mlo7VS3ex 1d6Z2+Z3+YtfEBg9eWIfYJH1zzMxrdC7IrEvCg47WPD2yx93lW2AxQWspNWh+EJ+b/3s 9DdXXuFddZKI5Lq8GUi6j3W6d6oMBSv8+rStzy0F9WFhZu5ORpUoq6SYk8OpUDk7p63e zvSWkcUBD3bzXvHa/Oyz8nBmKozQA1Oy4UAHuMMjFQRlEK6yRaMCySjRWonF7WOQ7lxX Xp4Q== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=KKHmN9DU; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [209.51.188.17]) by mx.google.com with ESMTPS id e3si2532258qvx.39.2020.04.21.21.35.09 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Tue, 21 Apr 2020 21:35:09 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=KKHmN9DU; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1]:41298 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jR76L-0003MG-5C for patch@linaro.org; Wed, 22 Apr 2020 00:35:09 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:50072) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jR74X-0000QJ-OA for qemu-devel@nongnu.org; Wed, 22 Apr 2020 00:33:18 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.90_1) (envelope-from ) id 1jR74X-0004j5-4w for qemu-devel@nongnu.org; Wed, 22 Apr 2020 00:33:17 -0400 Received: from mail-pf1-x443.google.com ([2607:f8b0:4864:20::443]:33825) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1jR74W-0004fG-Mm for qemu-devel@nongnu.org; Wed, 22 Apr 2020 00:33:16 -0400 Received: by mail-pf1-x443.google.com with SMTP id x15so457990pfa.1 for ; Tue, 21 Apr 2020 21:33:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=Ar4ktbgi8ho4D2K0AlAA7ABuossIOEQqKwU/YvDdwbY=; b=KKHmN9DUtNl/895jM8s3jff7LEzmFeqpQ8DWMpXM1qONzT3m9ARgcZsMgP+XdscICt HZLMNtv/ZqoDWF8cXeHk+2bSlQs/4PWjWY5m0f5+6wiIyah2Z+B4dd2NmFckQr7NJ6HO FJ2+o+DHRGk8N+afxBcsGByNqu6GDXbOc6issTgN7yM3kAgSlrKXj2oY6o/6HzchDLLo iH6AsFkzi9ygAU/ahiyH9fYPWIhJi4eQgbeJH9nfG4wPaO08AUWKlOGemDpIwU7+crxI eXUdP952xqi7ovID9SZ5VX81/jeMI2wy+AP48xd3inj5+XoViGfrnA0WWgdTVEceMl3X kM3Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Ar4ktbgi8ho4D2K0AlAA7ABuossIOEQqKwU/YvDdwbY=; b=pKp4HhKOe3UZkAp1jcPrX/7FgkrhVQUzdOlqJ8FcUwITiakSyIb3CJbH7EB1kC9WGo Go2nMCRhHSgro+KOMQzGcnAi3fHf1o2WT20QrdG2RlDAYqffQMwqQGN4q/KHvwJclZ4+ LLdlS1kaYPSV+z54WA2a87bZgYiaE2v6sbxLrGl0KYT9mHYR0twK5eRYDilt0qsT/xto CKy2q2n+V11dvSLyyM3jXQPKQIDbRGpsyCP+UFpSlxP9ZHcaIww2cbAZYVIHdcXThVDT i7B+rLL6MBqqnh5uSX8pUTd1aJ2q6qut0GR6UDu+eSi+8EYJOVo3D2v16a/FaKhhbFHJ SJtg== X-Gm-Message-State: AGi0PuYRUjPCbt6N+meKuufYobEMfw1dEnIezrp13MBI4AY3uxQKYpEn ipf2YhkG6EKvX6+qUPDl9G6ZyX+e2cE= X-Received: by 2002:a63:5853:: with SMTP id i19mr22901678pgm.288.1587529994937; Tue, 21 Apr 2020 21:33:14 -0700 (PDT) Received: from localhost.localdomain (174-21-149-226.tukw.qwest.net. [174.21.149.226]) by smtp.gmail.com with ESMTPSA id l137sm4129613pfd.107.2020.04.21.21.33.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 21 Apr 2020 21:33:14 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Subject: [PATCH v3 03/18] accel/tcg: Add block comment for probe_access Date: Tue, 21 Apr 2020 21:32:54 -0700 Message-Id: <20200422043309.18430-4-richard.henderson@linaro.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200422043309.18430-1-richard.henderson@linaro.org> References: <20200422043309.18430-1-richard.henderson@linaro.org> MIME-Version: 1.0 Received-SPF: pass client-ip=2607:f8b0:4864:20::443; envelope-from=richard.henderson@linaro.org; helo=mail-pf1-x443.google.com X-detected-operating-system: by eggs.gnu.org: Error: [-] PROGRAM ABORT : Malformed IPv6 address (bad octet value). Location : parse_addr6(), p0f-client.c:67 X-Received-From: 2607:f8b0:4864:20::443 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: peter.maydell@linaro.org, qemu-arm@nongnu.org Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: "Qemu-devel" Reviewed-by: Peter Maydell Signed-off-by: Richard Henderson --- include/exec/exec-all.h | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) -- 2.20.1 diff --git a/include/exec/exec-all.h b/include/exec/exec-all.h index 350c4b451b..d656a1f05c 100644 --- a/include/exec/exec-all.h +++ b/include/exec/exec-all.h @@ -330,6 +330,23 @@ static inline void tlb_flush_by_mmuidx_all_cpus_synced(CPUState *cpu, { } #endif +/** + * probe_access: + * @env: CPUArchState + * @addr: guest virtual address to look up + * @size: size of the access + * @access_type: read, write or execute permission + * @mmu_idx: MMU index to use for lookup + * @retaddr: return address for unwinding + * + * Look up the guest virtual address @addr. Raise an exception if the + * page does not satisfy @access_type. Raise an exception if the + * access (@addr, @size) hits a watchpoint. For writes, mark a clean + * page as dirty. + * + * Finally, return the host address for a page that is backed by RAM, + * or NULL if the page requires I/O. + */ void *probe_access(CPUArchState *env, target_ulong addr, int size, MMUAccessType access_type, int mmu_idx, uintptr_t retaddr); From patchwork Wed Apr 22 04:32:55 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 185659 Delivered-To: patch@linaro.org Received: by 2002:a92:3d9a:0:0:0:0:0 with SMTP id k26csp298516ilf; Tue, 21 Apr 2020 21:35:56 -0700 (PDT) X-Google-Smtp-Source: APiQypL5YaEdAV1si3dhSRa1vV+iDndA3sF8/0MUSv8oh5ax1NhMf4BZGAEmNosHmGO27006YauB X-Received: by 2002:ac8:6d06:: with SMTP id o6mr25236840qtt.165.1587530156649; Tue, 21 Apr 2020 21:35:56 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1587530156; cv=none; d=google.com; s=arc-20160816; b=Yqc9YOoau2Fzi4R0wrYz6aCvuSoq4MWof4KEjOr4H/bWKA6F8aCe6xoMIyNlG/LFyI 0B3khuxW5jgYxuIGqeNChM1A+AW994aqZi8jW7/ek/VRY2zU2EXPe3TLihoMl6K8etwE eG4cKSdH3+UMT0nYT9Un0LuwJdzddXL3FrGW1xJ/MV3oGOb53yjvEXSAmNUqeEFGxgNG bR3ETIDlUj1tnQN74txoGuccaR7EwVrJQucNRrHqx+lgqQANdqs15+Q4J0lHnpd3g2u7 KSpOKLeKaERprsa3mrnuLNacE+WZKw9h8mRjlWaEv+pLS/sOLSjxq4d8OZYIeAqkdU+s bA1w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:cc:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:content-transfer-encoding :mime-version:references:in-reply-to:message-id:date:subject:to:from :dkim-signature; bh=eMvCKk7QhyRjEslWtBZnVbQxlAek5NbJgVMiopqtZ1M=; b=uU1qM8vxDdQ/D3eaRzwyzeFb9812ustb2wyOoK0KSaYuPjJiV+MZzCb3HRwMysylet ky8RoNcBJUD7IAhybPiu4aOuIoiVIB2k8Hd9IlWAuRVrbCzY/yepjBKY/t3CPJprLo7+ EaxWW//Co1WNWIluoKLi4/i9YO681TKp9/HLmvCSZR8oS+ILuBrsMEt8+h6bgFQqMs1a dRmSSUdGlr08uYSXre2phOnlrUW6D/Z6Tp8jEZKbhgyoJHD/vqdk+8zykQt7cvmfdHI9 bw80b0idA+W2/1ptiM3PnZftDCfMFazlkmdwRo9KxgrOHSqsR/oQA8etyxqKrg4DMy6Z DIRw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=IfvFN+lJ; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [209.51.188.17]) by mx.google.com with ESMTPS id e7si2522705qvs.68.2020.04.21.21.35.56 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Tue, 21 Apr 2020 21:35:56 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=IfvFN+lJ; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1]:41328 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jR776-0004l8-0k for patch@linaro.org; Wed, 22 Apr 2020 00:35:56 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:50256) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jR74c-0000Yl-6m for qemu-devel@nongnu.org; Wed, 22 Apr 2020 00:33:23 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.90_1) (envelope-from ) id 1jR74b-00052U-97 for qemu-devel@nongnu.org; Wed, 22 Apr 2020 00:33:21 -0400 Received: from mail-pj1-x1044.google.com ([2607:f8b0:4864:20::1044]:50661) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1jR74Z-0004rH-Re for qemu-devel@nongnu.org; Wed, 22 Apr 2020 00:33:20 -0400 Received: by mail-pj1-x1044.google.com with SMTP id t9so354426pjw.0 for ; Tue, 21 Apr 2020 21:33:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=eMvCKk7QhyRjEslWtBZnVbQxlAek5NbJgVMiopqtZ1M=; b=IfvFN+lJrvtTDotJPyBa6VErBwxxm+0SZdAJ14GqzrL+sBWq5Uat62ylrAAlC5NQBF ngqIVbNsMWZbADqVoPb0BO29k0EYQ0VLD1QQ/7sgDmfa/X90uID+3EN7aAdMmdGjKGXN s5PUiw1Kw/LKKLj7XHhfUy+r0jJepuRN86FOGFYFKmd7726PZZ0EDoQY5DOI6CqfQ+Hp +hxDgKpmlM+rYsYYA/leHzYZfxZWS3PbrdM934B2+vK51G92I0CxEhTMhj9o97H2BMOZ 63mS46VNmlH3oygUO50QL8PYcmtuOVHG8YTaB7MXPE6Q3mRYLKwUFruvAiN1GCbEGfTf YMyw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=eMvCKk7QhyRjEslWtBZnVbQxlAek5NbJgVMiopqtZ1M=; b=WwzG5TAwbEBhf8T4TlpW7UpBK5eZd/LG1fbtmT4iIY6/BXlBtGW3T+1UBeUciUIRqa 2TD8C65SHPnJyFuoGvGcqGfeQ9IGDoG+jvlMTrR3ao4HxXSPa/vNE6OjVCrMA2EG/DkN wQpdx5mww1ZxKRbwrygtCyWe6yxhRLKRbDN9yKwX/o4mxbsyl5S1sdq8Mxjgsx88TBFi j1ujJOwDD5++VtLXXpt9D/Y5CZwX8QB9leuaoH72mVPZn+vUhb2CuZeM8XfMXz2lwZ53 7LGOTN6FxMLL1iWS8WeXKsIMPi3C13iIFO6daWnUwu48gSu+4a3VgDqdw0WUBp5nhRkP WUNg== X-Gm-Message-State: AGi0PuYuZ3KhY2fMzjj84eIHa5GDzsrKTN+TZyATRPZMmUVDCahhZC1p JzqlWmF0O8yPzNoKAX8x2LVTUFnCY8E= X-Received: by 2002:a17:90a:bc4a:: with SMTP id t10mr9710263pjv.104.1587529996162; Tue, 21 Apr 2020 21:33:16 -0700 (PDT) Received: from localhost.localdomain (174-21-149-226.tukw.qwest.net. [174.21.149.226]) by smtp.gmail.com with ESMTPSA id l137sm4129613pfd.107.2020.04.21.21.33.15 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 21 Apr 2020 21:33:15 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Subject: [PATCH v3 04/18] accel/tcg: Add probe_access_flags Date: Tue, 21 Apr 2020 21:32:55 -0700 Message-Id: <20200422043309.18430-5-richard.henderson@linaro.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200422043309.18430-1-richard.henderson@linaro.org> References: <20200422043309.18430-1-richard.henderson@linaro.org> MIME-Version: 1.0 Received-SPF: pass client-ip=2607:f8b0:4864:20::1044; envelope-from=richard.henderson@linaro.org; helo=mail-pj1-x1044.google.com X-detected-operating-system: by eggs.gnu.org: Error: [-] PROGRAM ABORT : Malformed IPv6 address (bad octet value). Location : parse_addr6(), p0f-client.c:67 X-Received-From: 2607:f8b0:4864:20::1044 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: peter.maydell@linaro.org, qemu-arm@nongnu.org Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: "Qemu-devel" This new interface will allow targets to probe for a page and then handle watchpoints themselves. This will be most useful for vector predicated memory operations, where one page lookup can be used for many operations, and one test can avoid many watchpoint checks. Signed-off-by: Richard Henderson --- v2: Fix return of host pointer in softmmu probe_access_flags. --- include/exec/cpu-all.h | 13 ++- include/exec/exec-all.h | 22 +++++ accel/tcg/cputlb.c | 177 ++++++++++++++++++++-------------------- accel/tcg/user-exec.c | 36 +++++--- 4 files changed, 149 insertions(+), 99 deletions(-) -- 2.20.1 diff --git a/include/exec/cpu-all.h b/include/exec/cpu-all.h index 49384bb66a..43ddcf024c 100644 --- a/include/exec/cpu-all.h +++ b/include/exec/cpu-all.h @@ -328,7 +328,18 @@ CPUArchState *cpu_copy(CPUArchState *env); | CPU_INTERRUPT_TGT_EXT_3 \ | CPU_INTERRUPT_TGT_EXT_4) -#if !defined(CONFIG_USER_ONLY) +#ifdef CONFIG_USER_ONLY + +/* + * Allow some level of source compatibility with softmmu. We do not + * support any of the more exotic features, so only invalid pages may + * be signaled by probe_access_flags(). + */ +#define TLB_INVALID_MASK (1 << (TARGET_PAGE_BITS_MIN - 1)) +#define TLB_MMIO 0 +#define TLB_WATCHPOINT 0 + +#else /* * Flags stored in the low bits of the TLB virtual address. diff --git a/include/exec/exec-all.h b/include/exec/exec-all.h index d656a1f05c..8792bea07a 100644 --- a/include/exec/exec-all.h +++ b/include/exec/exec-all.h @@ -362,6 +362,28 @@ static inline void *probe_read(CPUArchState *env, target_ulong addr, int size, return probe_access(env, addr, size, MMU_DATA_LOAD, mmu_idx, retaddr); } +/** + * probe_access_flags: + * @env: CPUArchState + * @addr: guest virtual address to look up + * @access_type: read, write or execute permission + * @mmu_idx: MMU index to use for lookup + * @nonfault: suppress the fault + * @phost: return value for host address + * @retaddr: return address for unwinding + * + * Similar to probe_access, loosely returning the TLB_FLAGS_MASK for + * the page, and storing the host address for RAM in @phost. + * + * If @nonfault is set, do not raise an exception but return TLB_INVALID_MASK. + * Do not handle watchpoints, but include TLB_WATCHPOINT in the returned flags. + * Do handle clean pages, so exclude TLB_NOTDIRY from the returned flags. + * For simplicity, all "mmio-like" flags are folded to TLB_MMIO. + */ +int probe_access_flags(CPUArchState *env, target_ulong addr, + MMUAccessType access_type, int mmu_idx, + bool nonfault, void **phost, uintptr_t retaddr); + #define CODE_GEN_ALIGN 16 /* must be >= of the size of a icache line */ /* Estimated block size for TB allocation. */ diff --git a/accel/tcg/cputlb.c b/accel/tcg/cputlb.c index e3b5750c3b..bbe265ce28 100644 --- a/accel/tcg/cputlb.c +++ b/accel/tcg/cputlb.c @@ -1231,86 +1231,16 @@ static void notdirty_write(CPUState *cpu, vaddr mem_vaddr, unsigned size, } } -/* - * Probe for whether the specified guest access is permitted. If it is not - * permitted then an exception will be taken in the same way as if this - * were a real access (and we will not return). - * If the size is 0 or the page requires I/O access, returns NULL; otherwise, - * returns the address of the host page similar to tlb_vaddr_to_host(). - */ -void *probe_access(CPUArchState *env, target_ulong addr, int size, - MMUAccessType access_type, int mmu_idx, uintptr_t retaddr) +static int probe_access_internal(CPUArchState *env, target_ulong addr, + int fault_size, MMUAccessType access_type, + int mmu_idx, bool nonfault, + void **phost, uintptr_t retaddr) { uintptr_t index = tlb_index(env, mmu_idx, addr); CPUTLBEntry *entry = tlb_entry(env, mmu_idx, addr); - target_ulong tlb_addr; - size_t elt_ofs; - int wp_access; - - g_assert(-(addr | TARGET_PAGE_MASK) >= size); - - switch (access_type) { - case MMU_DATA_LOAD: - elt_ofs = offsetof(CPUTLBEntry, addr_read); - wp_access = BP_MEM_READ; - break; - case MMU_DATA_STORE: - elt_ofs = offsetof(CPUTLBEntry, addr_write); - wp_access = BP_MEM_WRITE; - break; - case MMU_INST_FETCH: - elt_ofs = offsetof(CPUTLBEntry, addr_code); - wp_access = BP_MEM_READ; - break; - default: - g_assert_not_reached(); - } - tlb_addr = tlb_read_ofs(entry, elt_ofs); - - if (unlikely(!tlb_hit(tlb_addr, addr))) { - if (!victim_tlb_hit(env, mmu_idx, index, elt_ofs, - addr & TARGET_PAGE_MASK)) { - tlb_fill(env_cpu(env), addr, size, access_type, mmu_idx, retaddr); - /* TLB resize via tlb_fill may have moved the entry. */ - index = tlb_index(env, mmu_idx, addr); - entry = tlb_entry(env, mmu_idx, addr); - } - tlb_addr = tlb_read_ofs(entry, elt_ofs); - } - - if (!size) { - return NULL; - } - - if (unlikely(tlb_addr & TLB_FLAGS_MASK)) { - CPUIOTLBEntry *iotlbentry = &env_tlb(env)->d[mmu_idx].iotlb[index]; - - /* Reject I/O access, or other required slow-path. */ - if (tlb_addr & (TLB_MMIO | TLB_BSWAP | TLB_DISCARD_WRITE)) { - return NULL; - } - - /* Handle watchpoints. */ - if (tlb_addr & TLB_WATCHPOINT) { - cpu_check_watchpoint(env_cpu(env), addr, size, - iotlbentry->attrs, wp_access, retaddr); - } - - /* Handle clean RAM pages. */ - if (tlb_addr & TLB_NOTDIRTY) { - notdirty_write(env_cpu(env), addr, size, iotlbentry, retaddr); - } - } - - return (void *)((uintptr_t)addr + entry->addend); -} - -void *tlb_vaddr_to_host(CPUArchState *env, abi_ptr addr, - MMUAccessType access_type, int mmu_idx) -{ - CPUTLBEntry *entry = tlb_entry(env, mmu_idx, addr); - target_ulong tlb_addr, page; + target_ulong tlb_addr, page_addr; size_t elt_ofs; + int flags; switch (access_type) { case MMU_DATA_LOAD: @@ -1325,20 +1255,19 @@ void *tlb_vaddr_to_host(CPUArchState *env, abi_ptr addr, default: g_assert_not_reached(); } - - page = addr & TARGET_PAGE_MASK; tlb_addr = tlb_read_ofs(entry, elt_ofs); - if (!tlb_hit_page(tlb_addr, page)) { - uintptr_t index = tlb_index(env, mmu_idx, addr); - - if (!victim_tlb_hit(env, mmu_idx, index, elt_ofs, page)) { + page_addr = addr & TARGET_PAGE_MASK; + if (!tlb_hit_page(tlb_addr, page_addr)) { + if (!victim_tlb_hit(env, mmu_idx, index, elt_ofs, page_addr)) { CPUState *cs = env_cpu(env); CPUClass *cc = CPU_GET_CLASS(cs); - if (!cc->tlb_fill(cs, addr, 0, access_type, mmu_idx, true, 0)) { + if (!cc->tlb_fill(cs, addr, fault_size, access_type, + mmu_idx, nonfault, retaddr)) { /* Non-faulting page table read failed. */ - return NULL; + *phost = NULL; + return TLB_INVALID_MASK; } /* TLB resize via tlb_fill may have moved the entry. */ @@ -1346,15 +1275,89 @@ void *tlb_vaddr_to_host(CPUArchState *env, abi_ptr addr, } tlb_addr = tlb_read_ofs(entry, elt_ofs); } + flags = tlb_addr & TLB_FLAGS_MASK; - if (tlb_addr & ~TARGET_PAGE_MASK) { - /* IO access */ + /* Fold all "mmio-like" bits into TLB_MMIO. This is not RAM. */ + if (unlikely(flags & ~(TLB_WATCHPOINT | TLB_NOTDIRTY))) { + *phost = NULL; + return TLB_MMIO; + } + + /* Everything else is RAM. */ + *phost = (void *)((uintptr_t)addr + entry->addend); + return flags; +} + +int probe_access_flags(CPUArchState *env, target_ulong addr, + MMUAccessType access_type, int mmu_idx, + bool nonfault, void **phost, uintptr_t retaddr) +{ + int flags; + + flags = probe_access_internal(env, addr, 0, access_type, mmu_idx, + nonfault, phost, retaddr); + + /* Handle clean RAM pages. */ + if (unlikely(flags & TLB_NOTDIRTY)) { + uintptr_t index = tlb_index(env, mmu_idx, addr); + CPUIOTLBEntry *iotlbentry = &env_tlb(env)->d[mmu_idx].iotlb[index]; + + notdirty_write(env_cpu(env), addr, 1, iotlbentry, retaddr); + flags &= ~TLB_NOTDIRTY; + } + + return flags; +} + +void *probe_access(CPUArchState *env, target_ulong addr, int size, + MMUAccessType access_type, int mmu_idx, uintptr_t retaddr) +{ + void *host; + int flags; + + g_assert(-(addr | TARGET_PAGE_MASK) >= size); + + flags = probe_access_internal(env, addr, size, access_type, mmu_idx, + false, &host, retaddr); + + /* Per the interface, size == 0 merely faults the access. */ + if (size == 0) { return NULL; } - return (void *)((uintptr_t)addr + entry->addend); + if (unlikely(flags & (TLB_NOTDIRTY | TLB_WATCHPOINT))) { + uintptr_t index = tlb_index(env, mmu_idx, addr); + CPUIOTLBEntry *iotlbentry = &env_tlb(env)->d[mmu_idx].iotlb[index]; + + /* Handle clean RAM pages. */ + if (flags & TLB_NOTDIRTY) { + notdirty_write(env_cpu(env), addr, 1, iotlbentry, retaddr); + } + + /* Handle watchpoints. */ + if (flags & TLB_WATCHPOINT) { + int wp_access = (access_type == MMU_DATA_STORE + ? BP_MEM_WRITE : BP_MEM_READ); + cpu_check_watchpoint(env_cpu(env), addr, size, + iotlbentry->attrs, wp_access, retaddr); + } + } + + return host; } +void *tlb_vaddr_to_host(CPUArchState *env, abi_ptr addr, + MMUAccessType access_type, int mmu_idx) +{ + void *host; + int flags; + + flags = probe_access_internal(env, addr, 0, access_type, + mmu_idx, true, &host, 0); + + /* No combination of flags are expected by the caller. */ + return flags ? NULL : host; +} #ifdef CONFIG_PLUGIN /* diff --git a/accel/tcg/user-exec.c b/accel/tcg/user-exec.c index 4be78eb9b3..91c2dc46dd 100644 --- a/accel/tcg/user-exec.c +++ b/accel/tcg/user-exec.c @@ -190,13 +190,12 @@ static inline int handle_cpu_signal(uintptr_t pc, siginfo_t *info, g_assert_not_reached(); } -void *probe_access(CPUArchState *env, target_ulong addr, int size, - MMUAccessType access_type, int mmu_idx, uintptr_t retaddr) +int probe_access_flags(CPUArchState *env, target_ulong addr, + MMUAccessType access_type, int mmu_idx, + bool nonfault, void **phost, uintptr_t ra) { int flags; - g_assert(-(addr | TARGET_PAGE_MASK) >= size); - switch (access_type) { case MMU_DATA_STORE: flags = PAGE_WRITE; @@ -211,15 +210,30 @@ void *probe_access(CPUArchState *env, target_ulong addr, int size, g_assert_not_reached(); } - if (!guest_addr_valid(addr) || page_check_range(addr, size, flags) < 0) { - CPUState *cpu = env_cpu(env); - CPUClass *cc = CPU_GET_CLASS(cpu); - cc->tlb_fill(cpu, addr, size, access_type, MMU_USER_IDX, false, - retaddr); - g_assert_not_reached(); + if (!guest_addr_valid(addr) || page_check_range(addr, 1, flags) < 0) { + if (nonfault) { + *phost = NULL; + return TLB_INVALID_MASK; + } else { + CPUState *cpu = env_cpu(env); + CPUClass *cc = CPU_GET_CLASS(cpu); + cc->tlb_fill(cpu, addr, 0, access_type, MMU_USER_IDX, false, ra); + g_assert_not_reached(); + } } - return size ? g2h(addr) : NULL; + *phost = g2h(addr); + return 0; +} + +void *probe_access(CPUArchState *env, target_ulong addr, int size, + MMUAccessType access_type, int mmu_idx, uintptr_t retaddr) +{ + void *host; + + g_assert(-(addr | TARGET_PAGE_MASK) >= size); + probe_access_flags(env, addr, access_type, mmu_idx, false, &host, retaddr); + return host; } #if defined(__i386__) From patchwork Wed Apr 22 04:32:56 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 185665 Delivered-To: patch@linaro.org Received: by 2002:a92:3d9a:0:0:0:0:0 with SMTP id k26csp301285ilf; Tue, 21 Apr 2020 21:40:25 -0700 (PDT) X-Google-Smtp-Source: APiQypJMDHR56G5vqC9VOwJO9obO+5Z1HeHXOpFmu+rpyvi1xY3WYcEWu8XMOM4094DuqpF+UzJ4 X-Received: by 2002:aed:2061:: with SMTP id 88mr5514399qta.47.1587530425083; Tue, 21 Apr 2020 21:40:25 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1587530425; cv=none; d=google.com; s=arc-20160816; b=id1E33TaSwfbFHswIsiYXDuXZuYSsHCWoT+9WHngcEYtTbLrlbS1g0UvTp8T6DeUk3 1Dnw43T9RNBzeGh1lPmYQ/m0cnYY82g6WH9iVef0PCL1O/qyYxlg1QnXCHablq+hbg+J uZVkBX2jimNDJSCb+haz7HiwFd5QCYSQ9xOsX8p0kB/SnIt9CiQYavmO7cfLJ2oefhmZ 4Jn8G1dVAFTxjxI9mE3ZmCvFIGVriO5UYi/dwwWGUFa0GLg1aX/KuwTY1JaAFHo5wH1X FeRWmMLzl9kx293D7SQJMXSxQBUvJmh7UE31faL+xWYFeljcYCoppnOomjoIIrMuTXqS FyqQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:cc:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:content-transfer-encoding :mime-version:references:in-reply-to:message-id:date:subject:to:from :dkim-signature; bh=TKqQ4LL/EXD07c3oky97BGHTG17Hr8gQbPOCsIsLoeU=; b=F6wyGTXnH+pzoxyj/WW/KaGE7WS1wUsaP6/hwIeifPufUHwnWWrJEollK22FDIIQ7a 0wvVjrkzgB+la8l7VfDcHtD9YG05W0HMsQG6MldnunPJpCkihCwTjq9dwwKEZQMt0d9N RlNqMf70UsKil8IGnbZ6F1doDUobfFSFcJ4C2lIA6fA6pbuKpNThF4nLUzrKOHBcp3If x47Qc8pOLR7B3hPEKpUJhySiRTYuOWm00Pc4NG/Gt+7YSn+ldyCIT//jPY28d7ePTUJN fVR6PIw76uarA/YzQn0TiheOUSzk49cEuZGGLiADxPTVZ1tQdWTw8U6L5kEVTShnvX8N u2+Q== ARC-Authentication-Results: i=1; mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b="Td/lw+3R"; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [209.51.188.17]) by mx.google.com with ESMTPS id i2si2565002qtq.359.2020.04.21.21.40.24 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Tue, 21 Apr 2020 21:40:25 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; Authentication-Results: mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b="Td/lw+3R"; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1]:41466 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jR7BP-0003pm-P3 for patch@linaro.org; Wed, 22 Apr 2020 00:40:24 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:50368) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jR74f-0000fN-7R for qemu-devel@nongnu.org; Wed, 22 Apr 2020 00:33:28 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.90_1) (envelope-from ) id 1jR74c-00058t-AU for qemu-devel@nongnu.org; Wed, 22 Apr 2020 00:33:24 -0400 Received: from mail-pg1-x541.google.com ([2607:f8b0:4864:20::541]:35657) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1jR74b-00050d-PX for qemu-devel@nongnu.org; Wed, 22 Apr 2020 00:33:21 -0400 Received: by mail-pg1-x541.google.com with SMTP id t11so477438pgg.2 for ; Tue, 21 Apr 2020 21:33:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=TKqQ4LL/EXD07c3oky97BGHTG17Hr8gQbPOCsIsLoeU=; b=Td/lw+3RkEwNrEd2m9K1iuxg5+KZd4LmnfLsqyGAmfyPxdKU0Nrg3rvgZ1rVhshQHm rFJll/RGyfdDLi9MxKYJcba1hTOuzLlUmshcdSeYX4xMRMrkX92lzSt3j5PncSM0FiiL tPzecQOfQv17S5hdHR0meR5rAmhJPLeLQTz6tRxv3Snn9ehzTdnM41CV7uRYcOhxr9fZ fp9j75M7W93/ihoxUY7nKlnyOyTc37nAUA8kl5lwxaBeSaSe2Sl8KJ5OwDSYOK/B6K9k bf2tirqraYmspatzyiF5eT3wGOHkoQvSPpeK4YWG6T0IzFhWs/rV1a83SE5+eVn/mSa2 iZ+A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=TKqQ4LL/EXD07c3oky97BGHTG17Hr8gQbPOCsIsLoeU=; b=tPb6iyIBolOXZHtMc6iS59Bbm6iyIWe4H2+6Z0CnXP+qBiihm75GZeNAiyJp/KYv5W nox2z1SNtbLm2fsWjlsXxJdWWInWAMGCO/c3mp+FKtZKujihhTDHby274GTkXOrKmh+H Lsv99GAyAtJisCPhaeoBNOIPTXBP8Yknl7d7BUga0rGeeiBix5nIsStxDdCjZjQf+rJ/ QdaC7saAAVhg+2vOJocv9I+Q63cYAX3YSv87GU8iqBevp4kQT9uAbWZyN2LuXrsD4SB2 g1q90MKrTKSk+VI83D0tJugct3WGsrG1HOrtvsamx9bM++93o4trjBoshqNS/y910BTQ dfeA== X-Gm-Message-State: AGi0PuaPTpabeXfOeHzIurfO2jdfWCO80awunXEMhCD8ujVdo8vOAnsu fcfqrNmYtLXj0rsWL4aYUA5lW+gDnC8= X-Received: by 2002:a63:5907:: with SMTP id n7mr22857778pgb.439.1587529999135; Tue, 21 Apr 2020 21:33:19 -0700 (PDT) Received: from localhost.localdomain (174-21-149-226.tukw.qwest.net. [174.21.149.226]) by smtp.gmail.com with ESMTPSA id l137sm4129613pfd.107.2020.04.21.21.33.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 21 Apr 2020 21:33:18 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Subject: [PATCH v3 05/18] accel/tcg: Add endian-specific cpu_{ld, st}* operations Date: Tue, 21 Apr 2020 21:32:56 -0700 Message-Id: <20200422043309.18430-6-richard.henderson@linaro.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200422043309.18430-1-richard.henderson@linaro.org> References: <20200422043309.18430-1-richard.henderson@linaro.org> MIME-Version: 1.0 Received-SPF: pass client-ip=2607:f8b0:4864:20::541; envelope-from=richard.henderson@linaro.org; helo=mail-pg1-x541.google.com X-detected-operating-system: by eggs.gnu.org: Error: [-] PROGRAM ABORT : Malformed IPv6 address (bad octet value). Location : parse_addr6(), p0f-client.c:67 X-Received-From: 2607:f8b0:4864:20::541 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: peter.maydell@linaro.org, qemu-arm@nongnu.org Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: "Qemu-devel" We currently have target-endian versions of these operations, but no easy way to force a specific endianness. This can be helpful if the target has endian-specific operations, or a mode that swaps endianness. Signed-off-by: Richard Henderson --- docs/devel/loads-stores.rst | 39 +++-- include/exec/cpu_ldst.h | 277 +++++++++++++++++++++++++++--------- accel/tcg/cputlb.c | 236 ++++++++++++++++++++++-------- accel/tcg/user-exec.c | 211 ++++++++++++++++++++++----- 4 files changed, 587 insertions(+), 176 deletions(-) -- 2.20.1 Reviewed-by: Peter Maydell diff --git a/docs/devel/loads-stores.rst b/docs/devel/loads-stores.rst index 0d99eb24c1..9a944ef1af 100644 --- a/docs/devel/loads-stores.rst +++ b/docs/devel/loads-stores.rst @@ -97,9 +97,9 @@ function, which is a return address into the generated code. Function names follow the pattern: -load: ``cpu_ld{sign}{size}_mmuidx_ra(env, ptr, mmuidx, retaddr)`` +load: ``cpu_ld{sign}{size}{end}_mmuidx_ra(env, ptr, mmuidx, retaddr)`` -store: ``cpu_st{size}_mmuidx_ra(env, ptr, val, mmuidx, retaddr)`` +store: ``cpu_st{size}{end}_mmuidx_ra(env, ptr, val, mmuidx, retaddr)`` ``sign`` - (empty) : for 32 or 64 bit sizes @@ -112,9 +112,14 @@ store: ``cpu_st{size}_mmuidx_ra(env, ptr, val, mmuidx, retaddr)`` - ``l`` : 32 bits - ``q`` : 64 bits +``end`` + - (empty) : for target endian, or 8 bit sizes + - ``_be`` : big endian + - ``_le`` : little endian + Regexes for git grep: - - ``\`` - - ``\`` + - ``\`` + - ``\`` ``cpu_{ld,st}*_data_ra`` ~~~~~~~~~~~~~~~~~~~~~~~~ @@ -129,9 +134,9 @@ be performed with a context other than the default. Function names follow the pattern: -load: ``cpu_ld{sign}{size}_data_ra(env, ptr, ra)`` +load: ``cpu_ld{sign}{size}{end}_data_ra(env, ptr, ra)`` -store: ``cpu_st{size}_data_ra(env, ptr, val, ra)`` +store: ``cpu_st{size}{end}_data_ra(env, ptr, val, ra)`` ``sign`` - (empty) : for 32 or 64 bit sizes @@ -144,9 +149,14 @@ store: ``cpu_st{size}_data_ra(env, ptr, val, ra)`` - ``l`` : 32 bits - ``q`` : 64 bits +``end`` + - (empty) : for target endian, or 8 bit sizes + - ``_be`` : big endian + - ``_le`` : little endian + Regexes for git grep: - - ``\`` - - ``\`` + - ``\`` + - ``\`` ``cpu_{ld,st}*_data`` ~~~~~~~~~~~~~~~~~~~~~ @@ -163,9 +173,9 @@ the CPU state anyway. Function names follow the pattern: -load: ``cpu_ld{sign}{size}_data(env, ptr)`` +load: ``cpu_ld{sign}{size}{end}_data(env, ptr)`` -store: ``cpu_st{size}_data(env, ptr, val)`` +store: ``cpu_st{size}{end}_data(env, ptr, val)`` ``sign`` - (empty) : for 32 or 64 bit sizes @@ -178,9 +188,14 @@ store: ``cpu_st{size}_data(env, ptr, val)`` - ``l`` : 32 bits - ``q`` : 64 bits +``end`` + - (empty) : for target endian, or 8 bit sizes + - ``_be`` : big endian + - ``_le`` : little endian + Regexes for git grep - - ``\`` - - ``\`` + - ``\`` + - ``\`` ``cpu_ld*_code`` ~~~~~~~~~~~~~~~~ diff --git a/include/exec/cpu_ldst.h b/include/exec/cpu_ldst.h index 53de19753a..1ba515bfcc 100644 --- a/include/exec/cpu_ldst.h +++ b/include/exec/cpu_ldst.h @@ -26,12 +26,18 @@ * The syntax for the accessors is: * * load: cpu_ld{sign}{size}_{mmusuffix}(env, ptr) + * cpu_ld{sign}{size}{end}_{mmusuffix}(env, ptr) * cpu_ld{sign}{size}_{mmusuffix}_ra(env, ptr, retaddr) + * cpu_ld{sign}{size}{end}_{mmusuffix}_ra(env, ptr, retaddr) * cpu_ld{sign}{size}_mmuidx_ra(env, ptr, mmu_idx, retaddr) + * cpu_ld{sign}{size}{end}_mmuidx_ra(env, ptr, mmu_idx, retaddr) * * store: cpu_st{size}_{mmusuffix}(env, ptr, val) + * cpu_st{size}{end}_{mmusuffix}(env, ptr, val) * cpu_st{size}_{mmusuffix}_ra(env, ptr, val, retaddr) + * cpu_st{size}{end}_{mmusuffix}_ra(env, ptr, val, retaddr) * cpu_st{size}_mmuidx_ra(env, ptr, val, mmu_idx, retaddr) + * cpu_st{size}{end}_mmuidx_ra(env, ptr, val, mmu_idx, retaddr) * * sign is: * (empty): for 32 and 64 bit sizes @@ -44,6 +50,11 @@ * l: 32 bits * q: 64 bits * + * end is: + * (empty): for target native endian, or for 8 bit access + * _be: for forced big endian + * _le: for forced little endian + * * mmusuffix is one of the generic suffixes "data" or "code", or "mmuidx". * The "mmuidx" suffix carries an extra mmu_idx argument that specifies * the index to use; the "data" and "code" suffixes take the index from @@ -95,32 +106,57 @@ typedef target_ulong abi_ptr; #endif uint32_t cpu_ldub_data(CPUArchState *env, abi_ptr ptr); -uint32_t cpu_lduw_data(CPUArchState *env, abi_ptr ptr); -uint32_t cpu_ldl_data(CPUArchState *env, abi_ptr ptr); -uint64_t cpu_ldq_data(CPUArchState *env, abi_ptr ptr); int cpu_ldsb_data(CPUArchState *env, abi_ptr ptr); -int cpu_ldsw_data(CPUArchState *env, abi_ptr ptr); -uint32_t cpu_ldub_data_ra(CPUArchState *env, abi_ptr ptr, uintptr_t retaddr); -uint32_t cpu_lduw_data_ra(CPUArchState *env, abi_ptr ptr, uintptr_t retaddr); -uint32_t cpu_ldl_data_ra(CPUArchState *env, abi_ptr ptr, uintptr_t retaddr); -uint64_t cpu_ldq_data_ra(CPUArchState *env, abi_ptr ptr, uintptr_t retaddr); -int cpu_ldsb_data_ra(CPUArchState *env, abi_ptr ptr, uintptr_t retaddr); -int cpu_ldsw_data_ra(CPUArchState *env, abi_ptr ptr, uintptr_t retaddr); +uint32_t cpu_lduw_be_data(CPUArchState *env, abi_ptr ptr); +int cpu_ldsw_be_data(CPUArchState *env, abi_ptr ptr); +uint32_t cpu_ldl_be_data(CPUArchState *env, abi_ptr ptr); +uint64_t cpu_ldq_be_data(CPUArchState *env, abi_ptr ptr); + +uint32_t cpu_lduw_le_data(CPUArchState *env, abi_ptr ptr); +int cpu_ldsw_le_data(CPUArchState *env, abi_ptr ptr); +uint32_t cpu_ldl_le_data(CPUArchState *env, abi_ptr ptr); +uint64_t cpu_ldq_le_data(CPUArchState *env, abi_ptr ptr); + +uint32_t cpu_ldub_data_ra(CPUArchState *env, abi_ptr ptr, uintptr_t ra); +int cpu_ldsb_data_ra(CPUArchState *env, abi_ptr ptr, uintptr_t ra); + +uint32_t cpu_lduw_be_data_ra(CPUArchState *env, abi_ptr ptr, uintptr_t ra); +int cpu_ldsw_be_data_ra(CPUArchState *env, abi_ptr ptr, uintptr_t ra); +uint32_t cpu_ldl_be_data_ra(CPUArchState *env, abi_ptr ptr, uintptr_t ra); +uint64_t cpu_ldq_be_data_ra(CPUArchState *env, abi_ptr ptr, uintptr_t ra); + +uint32_t cpu_lduw_le_data_ra(CPUArchState *env, abi_ptr ptr, uintptr_t ra); +int cpu_ldsw_le_data_ra(CPUArchState *env, abi_ptr ptr, uintptr_t ra); +uint32_t cpu_ldl_le_data_ra(CPUArchState *env, abi_ptr ptr, uintptr_t ra); +uint64_t cpu_ldq_le_data_ra(CPUArchState *env, abi_ptr ptr, uintptr_t ra); void cpu_stb_data(CPUArchState *env, abi_ptr ptr, uint32_t val); -void cpu_stw_data(CPUArchState *env, abi_ptr ptr, uint32_t val); -void cpu_stl_data(CPUArchState *env, abi_ptr ptr, uint32_t val); -void cpu_stq_data(CPUArchState *env, abi_ptr ptr, uint64_t val); + +void cpu_stw_be_data(CPUArchState *env, abi_ptr ptr, uint32_t val); +void cpu_stl_be_data(CPUArchState *env, abi_ptr ptr, uint32_t val); +void cpu_stq_be_data(CPUArchState *env, abi_ptr ptr, uint64_t val); + +void cpu_stw_le_data(CPUArchState *env, abi_ptr ptr, uint32_t val); +void cpu_stl_le_data(CPUArchState *env, abi_ptr ptr, uint32_t val); +void cpu_stq_le_data(CPUArchState *env, abi_ptr ptr, uint64_t val); void cpu_stb_data_ra(CPUArchState *env, abi_ptr ptr, - uint32_t val, uintptr_t retaddr); -void cpu_stw_data_ra(CPUArchState *env, abi_ptr ptr, - uint32_t val, uintptr_t retaddr); -void cpu_stl_data_ra(CPUArchState *env, abi_ptr ptr, - uint32_t val, uintptr_t retaddr); -void cpu_stq_data_ra(CPUArchState *env, abi_ptr ptr, - uint64_t val, uintptr_t retaddr); + uint32_t val, uintptr_t ra); + +void cpu_stw_be_data_ra(CPUArchState *env, abi_ptr ptr, + uint32_t val, uintptr_t ra); +void cpu_stl_be_data_ra(CPUArchState *env, abi_ptr ptr, + uint32_t val, uintptr_t ra); +void cpu_stq_be_data_ra(CPUArchState *env, abi_ptr ptr, + uint64_t val, uintptr_t ra); + +void cpu_stw_le_data_ra(CPUArchState *env, abi_ptr ptr, + uint32_t val, uintptr_t ra); +void cpu_stl_le_data_ra(CPUArchState *env, abi_ptr ptr, + uint32_t val, uintptr_t ra); +void cpu_stq_le_data_ra(CPUArchState *env, abi_ptr ptr, + uint64_t val, uintptr_t ra); #if defined(CONFIG_USER_ONLY) @@ -157,34 +193,58 @@ static inline uint32_t cpu_ldub_mmuidx_ra(CPUArchState *env, abi_ptr addr, return cpu_ldub_data_ra(env, addr, ra); } -static inline uint32_t cpu_lduw_mmuidx_ra(CPUArchState *env, abi_ptr addr, - int mmu_idx, uintptr_t ra) -{ - return cpu_lduw_data_ra(env, addr, ra); -} - -static inline uint32_t cpu_ldl_mmuidx_ra(CPUArchState *env, abi_ptr addr, - int mmu_idx, uintptr_t ra) -{ - return cpu_ldl_data_ra(env, addr, ra); -} - -static inline uint64_t cpu_ldq_mmuidx_ra(CPUArchState *env, abi_ptr addr, - int mmu_idx, uintptr_t ra) -{ - return cpu_ldq_data_ra(env, addr, ra); -} - static inline int cpu_ldsb_mmuidx_ra(CPUArchState *env, abi_ptr addr, int mmu_idx, uintptr_t ra) { return cpu_ldsb_data_ra(env, addr, ra); } -static inline int cpu_ldsw_mmuidx_ra(CPUArchState *env, abi_ptr addr, - int mmu_idx, uintptr_t ra) +static inline uint32_t cpu_lduw_be_mmuidx_ra(CPUArchState *env, abi_ptr addr, + int mmu_idx, uintptr_t ra) { - return cpu_ldsw_data_ra(env, addr, ra); + return cpu_lduw_be_data_ra(env, addr, ra); +} + +static inline int cpu_ldsw_be_mmuidx_ra(CPUArchState *env, abi_ptr addr, + int mmu_idx, uintptr_t ra) +{ + return cpu_ldsw_be_data_ra(env, addr, ra); +} + +static inline uint32_t cpu_ldl_be_mmuidx_ra(CPUArchState *env, abi_ptr addr, + int mmu_idx, uintptr_t ra) +{ + return cpu_ldl_be_data_ra(env, addr, ra); +} + +static inline uint64_t cpu_ldq_be_mmuidx_ra(CPUArchState *env, abi_ptr addr, + int mmu_idx, uintptr_t ra) +{ + return cpu_ldq_be_data_ra(env, addr, ra); +} + +static inline uint32_t cpu_lduw_le_mmuidx_ra(CPUArchState *env, abi_ptr addr, + int mmu_idx, uintptr_t ra) +{ + return cpu_lduw_le_data_ra(env, addr, ra); +} + +static inline int cpu_ldsw_le_mmuidx_ra(CPUArchState *env, abi_ptr addr, + int mmu_idx, uintptr_t ra) +{ + return cpu_ldsw_le_data_ra(env, addr, ra); +} + +static inline uint32_t cpu_ldl_le_mmuidx_ra(CPUArchState *env, abi_ptr addr, + int mmu_idx, uintptr_t ra) +{ + return cpu_ldl_le_data_ra(env, addr, ra); +} + +static inline uint64_t cpu_ldq_le_mmuidx_ra(CPUArchState *env, abi_ptr addr, + int mmu_idx, uintptr_t ra) +{ + return cpu_ldq_le_data_ra(env, addr, ra); } static inline void cpu_stb_mmuidx_ra(CPUArchState *env, abi_ptr addr, @@ -193,22 +253,46 @@ static inline void cpu_stb_mmuidx_ra(CPUArchState *env, abi_ptr addr, cpu_stb_data_ra(env, addr, val, ra); } -static inline void cpu_stw_mmuidx_ra(CPUArchState *env, abi_ptr addr, - uint32_t val, int mmu_idx, uintptr_t ra) +static inline void cpu_stw_be_mmuidx_ra(CPUArchState *env, abi_ptr addr, + uint32_t val, int mmu_idx, + uintptr_t ra) { - cpu_stw_data_ra(env, addr, val, ra); + cpu_stw_be_data_ra(env, addr, val, ra); } -static inline void cpu_stl_mmuidx_ra(CPUArchState *env, abi_ptr addr, - uint32_t val, int mmu_idx, uintptr_t ra) +static inline void cpu_stl_be_mmuidx_ra(CPUArchState *env, abi_ptr addr, + uint32_t val, int mmu_idx, + uintptr_t ra) { - cpu_stl_data_ra(env, addr, val, ra); + cpu_stl_be_data_ra(env, addr, val, ra); } -static inline void cpu_stq_mmuidx_ra(CPUArchState *env, abi_ptr addr, - uint64_t val, int mmu_idx, uintptr_t ra) +static inline void cpu_stq_be_mmuidx_ra(CPUArchState *env, abi_ptr addr, + uint64_t val, int mmu_idx, + uintptr_t ra) { - cpu_stq_data_ra(env, addr, val, ra); + cpu_stq_be_data_ra(env, addr, val, ra); +} + +static inline void cpu_stw_le_mmuidx_ra(CPUArchState *env, abi_ptr addr, + uint32_t val, int mmu_idx, + uintptr_t ra) +{ + cpu_stw_le_data_ra(env, addr, val, ra); +} + +static inline void cpu_stl_le_mmuidx_ra(CPUArchState *env, abi_ptr addr, + uint32_t val, int mmu_idx, + uintptr_t ra) +{ + cpu_stl_le_data_ra(env, addr, val, ra); +} + +static inline void cpu_stq_le_mmuidx_ra(CPUArchState *env, abi_ptr addr, + uint64_t val, int mmu_idx, + uintptr_t ra) +{ + cpu_stq_le_data_ra(env, addr, val, ra); } #else @@ -243,29 +327,92 @@ static inline CPUTLBEntry *tlb_entry(CPUArchState *env, uintptr_t mmu_idx, uint32_t cpu_ldub_mmuidx_ra(CPUArchState *env, abi_ptr addr, int mmu_idx, uintptr_t ra); -uint32_t cpu_lduw_mmuidx_ra(CPUArchState *env, abi_ptr addr, - int mmu_idx, uintptr_t ra); -uint32_t cpu_ldl_mmuidx_ra(CPUArchState *env, abi_ptr addr, - int mmu_idx, uintptr_t ra); -uint64_t cpu_ldq_mmuidx_ra(CPUArchState *env, abi_ptr addr, - int mmu_idx, uintptr_t ra); - int cpu_ldsb_mmuidx_ra(CPUArchState *env, abi_ptr addr, int mmu_idx, uintptr_t ra); -int cpu_ldsw_mmuidx_ra(CPUArchState *env, abi_ptr addr, - int mmu_idx, uintptr_t ra); + +uint32_t cpu_lduw_be_mmuidx_ra(CPUArchState *env, abi_ptr addr, + int mmu_idx, uintptr_t ra); +int cpu_ldsw_be_mmuidx_ra(CPUArchState *env, abi_ptr addr, + int mmu_idx, uintptr_t ra); +uint32_t cpu_ldl_be_mmuidx_ra(CPUArchState *env, abi_ptr addr, + int mmu_idx, uintptr_t ra); +uint64_t cpu_ldq_be_mmuidx_ra(CPUArchState *env, abi_ptr addr, + int mmu_idx, uintptr_t ra); + +uint32_t cpu_lduw_le_mmuidx_ra(CPUArchState *env, abi_ptr addr, + int mmu_idx, uintptr_t ra); +int cpu_ldsw_le_mmuidx_ra(CPUArchState *env, abi_ptr addr, + int mmu_idx, uintptr_t ra); +uint32_t cpu_ldl_le_mmuidx_ra(CPUArchState *env, abi_ptr addr, + int mmu_idx, uintptr_t ra); +uint64_t cpu_ldq_le_mmuidx_ra(CPUArchState *env, abi_ptr addr, + int mmu_idx, uintptr_t ra); void cpu_stb_mmuidx_ra(CPUArchState *env, abi_ptr addr, uint32_t val, int mmu_idx, uintptr_t retaddr); -void cpu_stw_mmuidx_ra(CPUArchState *env, abi_ptr addr, uint32_t val, - int mmu_idx, uintptr_t retaddr); -void cpu_stl_mmuidx_ra(CPUArchState *env, abi_ptr addr, uint32_t val, - int mmu_idx, uintptr_t retaddr); -void cpu_stq_mmuidx_ra(CPUArchState *env, abi_ptr addr, uint64_t val, - int mmu_idx, uintptr_t retaddr); + +void cpu_stw_be_mmuidx_ra(CPUArchState *env, abi_ptr addr, uint32_t val, + int mmu_idx, uintptr_t retaddr); +void cpu_stl_be_mmuidx_ra(CPUArchState *env, abi_ptr addr, uint32_t val, + int mmu_idx, uintptr_t retaddr); +void cpu_stq_be_mmuidx_ra(CPUArchState *env, abi_ptr addr, uint64_t val, + int mmu_idx, uintptr_t retaddr); + +void cpu_stw_le_mmuidx_ra(CPUArchState *env, abi_ptr addr, uint32_t val, + int mmu_idx, uintptr_t retaddr); +void cpu_stl_le_mmuidx_ra(CPUArchState *env, abi_ptr addr, uint32_t val, + int mmu_idx, uintptr_t retaddr); +void cpu_stq_le_mmuidx_ra(CPUArchState *env, abi_ptr addr, uint64_t val, + int mmu_idx, uintptr_t retaddr); #endif /* defined(CONFIG_USER_ONLY) */ +#ifdef TARGET_WORDS_BIGENDIAN +# define cpu_lduw_data cpu_lduw_be_data +# define cpu_ldsw_data cpu_ldsw_be_data +# define cpu_ldl_data cpu_ldl_be_data +# define cpu_ldq_data cpu_ldq_be_data +# define cpu_lduw_data_ra cpu_lduw_be_data_ra +# define cpu_ldsw_data_ra cpu_ldsw_be_data_ra +# define cpu_ldl_data_ra cpu_ldl_be_data_ra +# define cpu_ldq_data_ra cpu_ldq_be_data_ra +# define cpu_lduw_mmuidx_ra cpu_lduw_be_mmuidx_ra +# define cpu_ldsw_mmuidx_ra cpu_ldsw_be_mmuidx_ra +# define cpu_ldl_mmuidx_ra cpu_ldl_be_mmuidx_ra +# define cpu_ldq_mmuidx_ra cpu_ldq_be_mmuidx_ra +# define cpu_stw_data cpu_stw_be_data +# define cpu_stl_data cpu_stl_be_data +# define cpu_stq_data cpu_stq_be_data +# define cpu_stw_data_ra cpu_stw_be_data_ra +# define cpu_stl_data_ra cpu_stl_be_data_ra +# define cpu_stq_data_ra cpu_stq_be_data_ra +# define cpu_stw_mmuidx_ra cpu_stw_be_mmuidx_ra +# define cpu_stl_mmuidx_ra cpu_stl_be_mmuidx_ra +# define cpu_stq_mmuidx_ra cpu_stq_be_mmuidx_ra +#else +# define cpu_lduw_data cpu_lduw_le_data +# define cpu_ldsw_data cpu_ldsw_le_data +# define cpu_ldl_data cpu_ldl_le_data +# define cpu_ldq_data cpu_ldq_le_data +# define cpu_lduw_data_ra cpu_lduw_le_data_ra +# define cpu_ldsw_data_ra cpu_ldsw_le_data_ra +# define cpu_ldl_data_ra cpu_ldl_le_data_ra +# define cpu_ldq_data_ra cpu_ldq_le_data_ra +# define cpu_lduw_mmuidx_ra cpu_lduw_le_mmuidx_ra +# define cpu_ldsw_mmuidx_ra cpu_ldsw_le_mmuidx_ra +# define cpu_ldl_mmuidx_ra cpu_ldl_le_mmuidx_ra +# define cpu_ldq_mmuidx_ra cpu_ldq_le_mmuidx_ra +# define cpu_stw_data cpu_stw_le_data +# define cpu_stl_data cpu_stl_le_data +# define cpu_stq_data cpu_stq_le_data +# define cpu_stw_data_ra cpu_stw_le_data_ra +# define cpu_stl_data_ra cpu_stl_le_data_ra +# define cpu_stq_data_ra cpu_stq_le_data_ra +# define cpu_stw_mmuidx_ra cpu_stw_le_mmuidx_ra +# define cpu_stl_mmuidx_ra cpu_stl_le_mmuidx_ra +# define cpu_stq_mmuidx_ra cpu_stq_le_mmuidx_ra +#endif + uint32_t cpu_ldub_code(CPUArchState *env, abi_ptr addr); uint32_t cpu_lduw_code(CPUArchState *env, abi_ptr addr); uint32_t cpu_ldl_code(CPUArchState *env, abi_ptr addr); diff --git a/accel/tcg/cputlb.c b/accel/tcg/cputlb.c index bbe265ce28..ca18543463 100644 --- a/accel/tcg/cputlb.c +++ b/accel/tcg/cputlb.c @@ -1772,36 +1772,54 @@ int cpu_ldsb_mmuidx_ra(CPUArchState *env, abi_ptr addr, full_ldub_mmu); } -uint32_t cpu_lduw_mmuidx_ra(CPUArchState *env, abi_ptr addr, - int mmu_idx, uintptr_t ra) +uint32_t cpu_lduw_be_mmuidx_ra(CPUArchState *env, abi_ptr addr, + int mmu_idx, uintptr_t ra) { - return cpu_load_helper(env, addr, mmu_idx, ra, MO_TEUW, - MO_TE == MO_LE - ? full_le_lduw_mmu : full_be_lduw_mmu); + return cpu_load_helper(env, addr, mmu_idx, ra, MO_BEUW, full_be_lduw_mmu); } -int cpu_ldsw_mmuidx_ra(CPUArchState *env, abi_ptr addr, - int mmu_idx, uintptr_t ra) +int cpu_ldsw_be_mmuidx_ra(CPUArchState *env, abi_ptr addr, + int mmu_idx, uintptr_t ra) { - return (int16_t)cpu_load_helper(env, addr, mmu_idx, ra, MO_TESW, - MO_TE == MO_LE - ? full_le_lduw_mmu : full_be_lduw_mmu); + return (int16_t)cpu_load_helper(env, addr, mmu_idx, ra, MO_BESW, + full_be_lduw_mmu); } -uint32_t cpu_ldl_mmuidx_ra(CPUArchState *env, abi_ptr addr, - int mmu_idx, uintptr_t ra) +uint32_t cpu_ldl_be_mmuidx_ra(CPUArchState *env, abi_ptr addr, + int mmu_idx, uintptr_t ra) { - return cpu_load_helper(env, addr, mmu_idx, ra, MO_TEUL, - MO_TE == MO_LE - ? full_le_ldul_mmu : full_be_ldul_mmu); + return cpu_load_helper(env, addr, mmu_idx, ra, MO_BEUL, full_be_ldul_mmu); } -uint64_t cpu_ldq_mmuidx_ra(CPUArchState *env, abi_ptr addr, - int mmu_idx, uintptr_t ra) +uint64_t cpu_ldq_be_mmuidx_ra(CPUArchState *env, abi_ptr addr, + int mmu_idx, uintptr_t ra) { - return cpu_load_helper(env, addr, mmu_idx, ra, MO_TEQ, - MO_TE == MO_LE - ? helper_le_ldq_mmu : helper_be_ldq_mmu); + return cpu_load_helper(env, addr, mmu_idx, ra, MO_BEQ, helper_be_ldq_mmu); +} + +uint32_t cpu_lduw_le_mmuidx_ra(CPUArchState *env, abi_ptr addr, + int mmu_idx, uintptr_t ra) +{ + return cpu_load_helper(env, addr, mmu_idx, ra, MO_LEUW, full_le_lduw_mmu); +} + +int cpu_ldsw_le_mmuidx_ra(CPUArchState *env, abi_ptr addr, + int mmu_idx, uintptr_t ra) +{ + return (int16_t)cpu_load_helper(env, addr, mmu_idx, ra, MO_LESW, + full_le_lduw_mmu); +} + +uint32_t cpu_ldl_le_mmuidx_ra(CPUArchState *env, abi_ptr addr, + int mmu_idx, uintptr_t ra) +{ + return cpu_load_helper(env, addr, mmu_idx, ra, MO_LEUL, full_le_ldul_mmu); +} + +uint64_t cpu_ldq_le_mmuidx_ra(CPUArchState *env, abi_ptr addr, + int mmu_idx, uintptr_t ra) +{ + return cpu_load_helper(env, addr, mmu_idx, ra, MO_LEQ, helper_le_ldq_mmu); } uint32_t cpu_ldub_data_ra(CPUArchState *env, target_ulong ptr, @@ -1815,25 +1833,50 @@ int cpu_ldsb_data_ra(CPUArchState *env, target_ulong ptr, uintptr_t retaddr) return cpu_ldsb_mmuidx_ra(env, ptr, cpu_mmu_index(env, false), retaddr); } -uint32_t cpu_lduw_data_ra(CPUArchState *env, target_ulong ptr, - uintptr_t retaddr) +uint32_t cpu_lduw_be_data_ra(CPUArchState *env, target_ulong ptr, + uintptr_t retaddr) { - return cpu_lduw_mmuidx_ra(env, ptr, cpu_mmu_index(env, false), retaddr); + return cpu_lduw_be_mmuidx_ra(env, ptr, cpu_mmu_index(env, false), retaddr); } -int cpu_ldsw_data_ra(CPUArchState *env, target_ulong ptr, uintptr_t retaddr) +int cpu_ldsw_be_data_ra(CPUArchState *env, target_ulong ptr, uintptr_t retaddr) { - return cpu_ldsw_mmuidx_ra(env, ptr, cpu_mmu_index(env, false), retaddr); + return cpu_ldsw_be_mmuidx_ra(env, ptr, cpu_mmu_index(env, false), retaddr); } -uint32_t cpu_ldl_data_ra(CPUArchState *env, target_ulong ptr, uintptr_t retaddr) +uint32_t cpu_ldl_be_data_ra(CPUArchState *env, target_ulong ptr, + uintptr_t retaddr) { - return cpu_ldl_mmuidx_ra(env, ptr, cpu_mmu_index(env, false), retaddr); + return cpu_ldl_be_mmuidx_ra(env, ptr, cpu_mmu_index(env, false), retaddr); } -uint64_t cpu_ldq_data_ra(CPUArchState *env, target_ulong ptr, uintptr_t retaddr) +uint64_t cpu_ldq_be_data_ra(CPUArchState *env, target_ulong ptr, + uintptr_t retaddr) { - return cpu_ldq_mmuidx_ra(env, ptr, cpu_mmu_index(env, false), retaddr); + return cpu_ldq_be_mmuidx_ra(env, ptr, cpu_mmu_index(env, false), retaddr); +} + +uint32_t cpu_lduw_le_data_ra(CPUArchState *env, target_ulong ptr, + uintptr_t retaddr) +{ + return cpu_lduw_le_mmuidx_ra(env, ptr, cpu_mmu_index(env, false), retaddr); +} + +int cpu_ldsw_le_data_ra(CPUArchState *env, target_ulong ptr, uintptr_t retaddr) +{ + return cpu_ldsw_le_mmuidx_ra(env, ptr, cpu_mmu_index(env, false), retaddr); +} + +uint32_t cpu_ldl_le_data_ra(CPUArchState *env, target_ulong ptr, + uintptr_t retaddr) +{ + return cpu_ldl_le_mmuidx_ra(env, ptr, cpu_mmu_index(env, false), retaddr); +} + +uint64_t cpu_ldq_le_data_ra(CPUArchState *env, target_ulong ptr, + uintptr_t retaddr) +{ + return cpu_ldq_le_mmuidx_ra(env, ptr, cpu_mmu_index(env, false), retaddr); } uint32_t cpu_ldub_data(CPUArchState *env, target_ulong ptr) @@ -1846,24 +1889,44 @@ int cpu_ldsb_data(CPUArchState *env, target_ulong ptr) return cpu_ldsb_data_ra(env, ptr, 0); } -uint32_t cpu_lduw_data(CPUArchState *env, target_ulong ptr) +uint32_t cpu_lduw_be_data(CPUArchState *env, target_ulong ptr) { - return cpu_lduw_data_ra(env, ptr, 0); + return cpu_lduw_be_data_ra(env, ptr, 0); } -int cpu_ldsw_data(CPUArchState *env, target_ulong ptr) +int cpu_ldsw_be_data(CPUArchState *env, target_ulong ptr) { - return cpu_ldsw_data_ra(env, ptr, 0); + return cpu_ldsw_be_data_ra(env, ptr, 0); } -uint32_t cpu_ldl_data(CPUArchState *env, target_ulong ptr) +uint32_t cpu_ldl_be_data(CPUArchState *env, target_ulong ptr) { - return cpu_ldl_data_ra(env, ptr, 0); + return cpu_ldl_be_data_ra(env, ptr, 0); } -uint64_t cpu_ldq_data(CPUArchState *env, target_ulong ptr) +uint64_t cpu_ldq_be_data(CPUArchState *env, target_ulong ptr) { - return cpu_ldq_data_ra(env, ptr, 0); + return cpu_ldq_be_data_ra(env, ptr, 0); +} + +uint32_t cpu_lduw_le_data(CPUArchState *env, target_ulong ptr) +{ + return cpu_lduw_le_data_ra(env, ptr, 0); +} + +int cpu_ldsw_le_data(CPUArchState *env, target_ulong ptr) +{ + return cpu_ldsw_le_data_ra(env, ptr, 0); +} + +uint32_t cpu_ldl_le_data(CPUArchState *env, target_ulong ptr) +{ + return cpu_ldl_le_data_ra(env, ptr, 0); +} + +uint64_t cpu_ldq_le_data(CPUArchState *env, target_ulong ptr) +{ + return cpu_ldq_le_data_ra(env, ptr, 0); } /* @@ -2121,22 +2184,40 @@ void cpu_stb_mmuidx_ra(CPUArchState *env, target_ulong addr, uint32_t val, cpu_store_helper(env, addr, val, mmu_idx, retaddr, MO_UB); } -void cpu_stw_mmuidx_ra(CPUArchState *env, target_ulong addr, uint32_t val, - int mmu_idx, uintptr_t retaddr) +void cpu_stw_be_mmuidx_ra(CPUArchState *env, target_ulong addr, uint32_t val, + int mmu_idx, uintptr_t retaddr) { - cpu_store_helper(env, addr, val, mmu_idx, retaddr, MO_TEUW); + cpu_store_helper(env, addr, val, mmu_idx, retaddr, MO_BEUW); } -void cpu_stl_mmuidx_ra(CPUArchState *env, target_ulong addr, uint32_t val, - int mmu_idx, uintptr_t retaddr) +void cpu_stl_be_mmuidx_ra(CPUArchState *env, target_ulong addr, uint32_t val, + int mmu_idx, uintptr_t retaddr) { - cpu_store_helper(env, addr, val, mmu_idx, retaddr, MO_TEUL); + cpu_store_helper(env, addr, val, mmu_idx, retaddr, MO_BEUL); } -void cpu_stq_mmuidx_ra(CPUArchState *env, target_ulong addr, uint64_t val, - int mmu_idx, uintptr_t retaddr) +void cpu_stq_be_mmuidx_ra(CPUArchState *env, target_ulong addr, uint64_t val, + int mmu_idx, uintptr_t retaddr) { - cpu_store_helper(env, addr, val, mmu_idx, retaddr, MO_TEQ); + cpu_store_helper(env, addr, val, mmu_idx, retaddr, MO_BEQ); +} + +void cpu_stw_le_mmuidx_ra(CPUArchState *env, target_ulong addr, uint32_t val, + int mmu_idx, uintptr_t retaddr) +{ + cpu_store_helper(env, addr, val, mmu_idx, retaddr, MO_LEUW); +} + +void cpu_stl_le_mmuidx_ra(CPUArchState *env, target_ulong addr, uint32_t val, + int mmu_idx, uintptr_t retaddr) +{ + cpu_store_helper(env, addr, val, mmu_idx, retaddr, MO_LEUL); +} + +void cpu_stq_le_mmuidx_ra(CPUArchState *env, target_ulong addr, uint64_t val, + int mmu_idx, uintptr_t retaddr) +{ + cpu_store_helper(env, addr, val, mmu_idx, retaddr, MO_LEQ); } void cpu_stb_data_ra(CPUArchState *env, target_ulong ptr, @@ -2145,22 +2226,40 @@ void cpu_stb_data_ra(CPUArchState *env, target_ulong ptr, cpu_stb_mmuidx_ra(env, ptr, val, cpu_mmu_index(env, false), retaddr); } -void cpu_stw_data_ra(CPUArchState *env, target_ulong ptr, - uint32_t val, uintptr_t retaddr) +void cpu_stw_be_data_ra(CPUArchState *env, target_ulong ptr, + uint32_t val, uintptr_t retaddr) { - cpu_stw_mmuidx_ra(env, ptr, val, cpu_mmu_index(env, false), retaddr); + cpu_stw_be_mmuidx_ra(env, ptr, val, cpu_mmu_index(env, false), retaddr); } -void cpu_stl_data_ra(CPUArchState *env, target_ulong ptr, - uint32_t val, uintptr_t retaddr) +void cpu_stl_be_data_ra(CPUArchState *env, target_ulong ptr, + uint32_t val, uintptr_t retaddr) { - cpu_stl_mmuidx_ra(env, ptr, val, cpu_mmu_index(env, false), retaddr); + cpu_stl_be_mmuidx_ra(env, ptr, val, cpu_mmu_index(env, false), retaddr); } -void cpu_stq_data_ra(CPUArchState *env, target_ulong ptr, - uint64_t val, uintptr_t retaddr) +void cpu_stq_be_data_ra(CPUArchState *env, target_ulong ptr, + uint64_t val, uintptr_t retaddr) { - cpu_stq_mmuidx_ra(env, ptr, val, cpu_mmu_index(env, false), retaddr); + cpu_stq_be_mmuidx_ra(env, ptr, val, cpu_mmu_index(env, false), retaddr); +} + +void cpu_stw_le_data_ra(CPUArchState *env, target_ulong ptr, + uint32_t val, uintptr_t retaddr) +{ + cpu_stw_le_mmuidx_ra(env, ptr, val, cpu_mmu_index(env, false), retaddr); +} + +void cpu_stl_le_data_ra(CPUArchState *env, target_ulong ptr, + uint32_t val, uintptr_t retaddr) +{ + cpu_stl_le_mmuidx_ra(env, ptr, val, cpu_mmu_index(env, false), retaddr); +} + +void cpu_stq_le_data_ra(CPUArchState *env, target_ulong ptr, + uint64_t val, uintptr_t retaddr) +{ + cpu_stq_le_mmuidx_ra(env, ptr, val, cpu_mmu_index(env, false), retaddr); } void cpu_stb_data(CPUArchState *env, target_ulong ptr, uint32_t val) @@ -2168,19 +2267,34 @@ void cpu_stb_data(CPUArchState *env, target_ulong ptr, uint32_t val) cpu_stb_data_ra(env, ptr, val, 0); } -void cpu_stw_data(CPUArchState *env, target_ulong ptr, uint32_t val) +void cpu_stw_be_data(CPUArchState *env, target_ulong ptr, uint32_t val) { - cpu_stw_data_ra(env, ptr, val, 0); + cpu_stw_be_data_ra(env, ptr, val, 0); } -void cpu_stl_data(CPUArchState *env, target_ulong ptr, uint32_t val) +void cpu_stl_be_data(CPUArchState *env, target_ulong ptr, uint32_t val) { - cpu_stl_data_ra(env, ptr, val, 0); + cpu_stl_be_data_ra(env, ptr, val, 0); } -void cpu_stq_data(CPUArchState *env, target_ulong ptr, uint64_t val) +void cpu_stq_be_data(CPUArchState *env, target_ulong ptr, uint64_t val) { - cpu_stq_data_ra(env, ptr, val, 0); + cpu_stq_be_data_ra(env, ptr, val, 0); +} + +void cpu_stw_le_data(CPUArchState *env, target_ulong ptr, uint32_t val) +{ + cpu_stw_le_data_ra(env, ptr, val, 0); +} + +void cpu_stl_le_data(CPUArchState *env, target_ulong ptr, uint32_t val) +{ + cpu_stl_le_data_ra(env, ptr, val, 0); +} + +void cpu_stq_le_data(CPUArchState *env, target_ulong ptr, uint64_t val) +{ + cpu_stq_le_data_ra(env, ptr, val, 0); } /* First set of helpers allows passing in of OI and RETADDR. This makes diff --git a/accel/tcg/user-exec.c b/accel/tcg/user-exec.c index 91c2dc46dd..ac45dd118e 100644 --- a/accel/tcg/user-exec.c +++ b/accel/tcg/user-exec.c @@ -772,46 +772,90 @@ int cpu_ldsb_data(CPUArchState *env, abi_ptr ptr) return ret; } -uint32_t cpu_lduw_data(CPUArchState *env, abi_ptr ptr) +uint32_t cpu_lduw_be_data(CPUArchState *env, abi_ptr ptr) { uint32_t ret; - uint16_t meminfo = trace_mem_get_info(MO_TEUW, MMU_USER_IDX, false); + uint16_t meminfo = trace_mem_get_info(MO_BEUW, MMU_USER_IDX, false); trace_guest_mem_before_exec(env_cpu(env), ptr, meminfo); - ret = lduw_p(g2h(ptr)); + ret = lduw_be_p(g2h(ptr)); qemu_plugin_vcpu_mem_cb(env_cpu(env), ptr, meminfo); return ret; } -int cpu_ldsw_data(CPUArchState *env, abi_ptr ptr) +int cpu_ldsw_be_data(CPUArchState *env, abi_ptr ptr) { int ret; - uint16_t meminfo = trace_mem_get_info(MO_TESW, MMU_USER_IDX, false); + uint16_t meminfo = trace_mem_get_info(MO_BESW, MMU_USER_IDX, false); trace_guest_mem_before_exec(env_cpu(env), ptr, meminfo); - ret = ldsw_p(g2h(ptr)); + ret = ldsw_be_p(g2h(ptr)); qemu_plugin_vcpu_mem_cb(env_cpu(env), ptr, meminfo); return ret; } -uint32_t cpu_ldl_data(CPUArchState *env, abi_ptr ptr) +uint32_t cpu_ldl_be_data(CPUArchState *env, abi_ptr ptr) { uint32_t ret; - uint16_t meminfo = trace_mem_get_info(MO_TEUL, MMU_USER_IDX, false); + uint16_t meminfo = trace_mem_get_info(MO_BEUL, MMU_USER_IDX, false); trace_guest_mem_before_exec(env_cpu(env), ptr, meminfo); - ret = ldl_p(g2h(ptr)); + ret = ldl_be_p(g2h(ptr)); qemu_plugin_vcpu_mem_cb(env_cpu(env), ptr, meminfo); return ret; } -uint64_t cpu_ldq_data(CPUArchState *env, abi_ptr ptr) +uint64_t cpu_ldq_be_data(CPUArchState *env, abi_ptr ptr) { uint64_t ret; - uint16_t meminfo = trace_mem_get_info(MO_TEQ, MMU_USER_IDX, false); + uint16_t meminfo = trace_mem_get_info(MO_BEQ, MMU_USER_IDX, false); trace_guest_mem_before_exec(env_cpu(env), ptr, meminfo); - ret = ldq_p(g2h(ptr)); + ret = ldq_be_p(g2h(ptr)); + qemu_plugin_vcpu_mem_cb(env_cpu(env), ptr, meminfo); + return ret; +} + +uint32_t cpu_lduw_le_data(CPUArchState *env, abi_ptr ptr) +{ + uint32_t ret; + uint16_t meminfo = trace_mem_get_info(MO_LEUW, MMU_USER_IDX, false); + + trace_guest_mem_before_exec(env_cpu(env), ptr, meminfo); + ret = lduw_le_p(g2h(ptr)); + qemu_plugin_vcpu_mem_cb(env_cpu(env), ptr, meminfo); + return ret; +} + +int cpu_ldsw_le_data(CPUArchState *env, abi_ptr ptr) +{ + int ret; + uint16_t meminfo = trace_mem_get_info(MO_LESW, MMU_USER_IDX, false); + + trace_guest_mem_before_exec(env_cpu(env), ptr, meminfo); + ret = ldsw_le_p(g2h(ptr)); + qemu_plugin_vcpu_mem_cb(env_cpu(env), ptr, meminfo); + return ret; +} + +uint32_t cpu_ldl_le_data(CPUArchState *env, abi_ptr ptr) +{ + uint32_t ret; + uint16_t meminfo = trace_mem_get_info(MO_LEUL, MMU_USER_IDX, false); + + trace_guest_mem_before_exec(env_cpu(env), ptr, meminfo); + ret = ldl_le_p(g2h(ptr)); + qemu_plugin_vcpu_mem_cb(env_cpu(env), ptr, meminfo); + return ret; +} + +uint64_t cpu_ldq_le_data(CPUArchState *env, abi_ptr ptr) +{ + uint64_t ret; + uint16_t meminfo = trace_mem_get_info(MO_LEQ, MMU_USER_IDX, false); + + trace_guest_mem_before_exec(env_cpu(env), ptr, meminfo); + ret = ldq_le_p(g2h(ptr)); qemu_plugin_vcpu_mem_cb(env_cpu(env), ptr, meminfo); return ret; } @@ -836,42 +880,82 @@ int cpu_ldsb_data_ra(CPUArchState *env, abi_ptr ptr, uintptr_t retaddr) return ret; } -uint32_t cpu_lduw_data_ra(CPUArchState *env, abi_ptr ptr, uintptr_t retaddr) +uint32_t cpu_lduw_be_data_ra(CPUArchState *env, abi_ptr ptr, uintptr_t retaddr) { uint32_t ret; set_helper_retaddr(retaddr); - ret = cpu_lduw_data(env, ptr); + ret = cpu_lduw_be_data(env, ptr); clear_helper_retaddr(); return ret; } -int cpu_ldsw_data_ra(CPUArchState *env, abi_ptr ptr, uintptr_t retaddr) +int cpu_ldsw_be_data_ra(CPUArchState *env, abi_ptr ptr, uintptr_t retaddr) { int ret; set_helper_retaddr(retaddr); - ret = cpu_ldsw_data(env, ptr); + ret = cpu_ldsw_be_data(env, ptr); clear_helper_retaddr(); return ret; } -uint32_t cpu_ldl_data_ra(CPUArchState *env, abi_ptr ptr, uintptr_t retaddr) +uint32_t cpu_ldl_be_data_ra(CPUArchState *env, abi_ptr ptr, uintptr_t retaddr) { uint32_t ret; set_helper_retaddr(retaddr); - ret = cpu_ldl_data(env, ptr); + ret = cpu_ldl_be_data(env, ptr); clear_helper_retaddr(); return ret; } -uint64_t cpu_ldq_data_ra(CPUArchState *env, abi_ptr ptr, uintptr_t retaddr) +uint64_t cpu_ldq_be_data_ra(CPUArchState *env, abi_ptr ptr, uintptr_t retaddr) { uint64_t ret; set_helper_retaddr(retaddr); - ret = cpu_ldq_data(env, ptr); + ret = cpu_ldq_be_data(env, ptr); + clear_helper_retaddr(); + return ret; +} + +uint32_t cpu_lduw_le_data_ra(CPUArchState *env, abi_ptr ptr, uintptr_t retaddr) +{ + uint32_t ret; + + set_helper_retaddr(retaddr); + ret = cpu_lduw_le_data(env, ptr); + clear_helper_retaddr(); + return ret; +} + +int cpu_ldsw_le_data_ra(CPUArchState *env, abi_ptr ptr, uintptr_t retaddr) +{ + int ret; + + set_helper_retaddr(retaddr); + ret = cpu_ldsw_le_data(env, ptr); + clear_helper_retaddr(); + return ret; +} + +uint32_t cpu_ldl_le_data_ra(CPUArchState *env, abi_ptr ptr, uintptr_t retaddr) +{ + uint32_t ret; + + set_helper_retaddr(retaddr); + ret = cpu_ldl_le_data(env, ptr); + clear_helper_retaddr(); + return ret; +} + +uint64_t cpu_ldq_le_data_ra(CPUArchState *env, abi_ptr ptr, uintptr_t retaddr) +{ + uint64_t ret; + + set_helper_retaddr(retaddr); + ret = cpu_ldq_le_data(env, ptr); clear_helper_retaddr(); return ret; } @@ -885,30 +969,57 @@ void cpu_stb_data(CPUArchState *env, abi_ptr ptr, uint32_t val) qemu_plugin_vcpu_mem_cb(env_cpu(env), ptr, meminfo); } -void cpu_stw_data(CPUArchState *env, abi_ptr ptr, uint32_t val) +void cpu_stw_be_data(CPUArchState *env, abi_ptr ptr, uint32_t val) { - uint16_t meminfo = trace_mem_get_info(MO_TEUW, MMU_USER_IDX, true); + uint16_t meminfo = trace_mem_get_info(MO_BEUW, MMU_USER_IDX, true); trace_guest_mem_before_exec(env_cpu(env), ptr, meminfo); - stw_p(g2h(ptr), val); + stw_be_p(g2h(ptr), val); qemu_plugin_vcpu_mem_cb(env_cpu(env), ptr, meminfo); } -void cpu_stl_data(CPUArchState *env, abi_ptr ptr, uint32_t val) +void cpu_stl_be_data(CPUArchState *env, abi_ptr ptr, uint32_t val) { - uint16_t meminfo = trace_mem_get_info(MO_TEUL, MMU_USER_IDX, true); + uint16_t meminfo = trace_mem_get_info(MO_BEUL, MMU_USER_IDX, true); trace_guest_mem_before_exec(env_cpu(env), ptr, meminfo); - stl_p(g2h(ptr), val); + stl_be_p(g2h(ptr), val); qemu_plugin_vcpu_mem_cb(env_cpu(env), ptr, meminfo); } -void cpu_stq_data(CPUArchState *env, abi_ptr ptr, uint64_t val) +void cpu_stq_be_data(CPUArchState *env, abi_ptr ptr, uint64_t val) { - uint16_t meminfo = trace_mem_get_info(MO_TEQ, MMU_USER_IDX, true); + uint16_t meminfo = trace_mem_get_info(MO_BEQ, MMU_USER_IDX, true); trace_guest_mem_before_exec(env_cpu(env), ptr, meminfo); - stq_p(g2h(ptr), val); + stq_be_p(g2h(ptr), val); + qemu_plugin_vcpu_mem_cb(env_cpu(env), ptr, meminfo); +} + +void cpu_stw_le_data(CPUArchState *env, abi_ptr ptr, uint32_t val) +{ + uint16_t meminfo = trace_mem_get_info(MO_LEUW, MMU_USER_IDX, true); + + trace_guest_mem_before_exec(env_cpu(env), ptr, meminfo); + stw_le_p(g2h(ptr), val); + qemu_plugin_vcpu_mem_cb(env_cpu(env), ptr, meminfo); +} + +void cpu_stl_le_data(CPUArchState *env, abi_ptr ptr, uint32_t val) +{ + uint16_t meminfo = trace_mem_get_info(MO_LEUL, MMU_USER_IDX, true); + + trace_guest_mem_before_exec(env_cpu(env), ptr, meminfo); + stl_le_p(g2h(ptr), val); + qemu_plugin_vcpu_mem_cb(env_cpu(env), ptr, meminfo); +} + +void cpu_stq_le_data(CPUArchState *env, abi_ptr ptr, uint64_t val) +{ + uint16_t meminfo = trace_mem_get_info(MO_LEQ, MMU_USER_IDX, true); + + trace_guest_mem_before_exec(env_cpu(env), ptr, meminfo); + stq_le_p(g2h(ptr), val); qemu_plugin_vcpu_mem_cb(env_cpu(env), ptr, meminfo); } @@ -920,27 +1031,51 @@ void cpu_stb_data_ra(CPUArchState *env, abi_ptr ptr, clear_helper_retaddr(); } -void cpu_stw_data_ra(CPUArchState *env, abi_ptr ptr, - uint32_t val, uintptr_t retaddr) +void cpu_stw_be_data_ra(CPUArchState *env, abi_ptr ptr, + uint32_t val, uintptr_t retaddr) { set_helper_retaddr(retaddr); - cpu_stw_data(env, ptr, val); + cpu_stw_be_data(env, ptr, val); clear_helper_retaddr(); } -void cpu_stl_data_ra(CPUArchState *env, abi_ptr ptr, - uint32_t val, uintptr_t retaddr) +void cpu_stl_be_data_ra(CPUArchState *env, abi_ptr ptr, + uint32_t val, uintptr_t retaddr) { set_helper_retaddr(retaddr); - cpu_stl_data(env, ptr, val); + cpu_stl_be_data(env, ptr, val); clear_helper_retaddr(); } -void cpu_stq_data_ra(CPUArchState *env, abi_ptr ptr, - uint64_t val, uintptr_t retaddr) +void cpu_stq_be_data_ra(CPUArchState *env, abi_ptr ptr, + uint64_t val, uintptr_t retaddr) { set_helper_retaddr(retaddr); - cpu_stq_data(env, ptr, val); + cpu_stq_be_data(env, ptr, val); + clear_helper_retaddr(); +} + +void cpu_stw_le_data_ra(CPUArchState *env, abi_ptr ptr, + uint32_t val, uintptr_t retaddr) +{ + set_helper_retaddr(retaddr); + cpu_stw_le_data(env, ptr, val); + clear_helper_retaddr(); +} + +void cpu_stl_le_data_ra(CPUArchState *env, abi_ptr ptr, + uint32_t val, uintptr_t retaddr) +{ + set_helper_retaddr(retaddr); + cpu_stl_le_data(env, ptr, val); + clear_helper_retaddr(); +} + +void cpu_stq_le_data_ra(CPUArchState *env, abi_ptr ptr, + uint64_t val, uintptr_t retaddr) +{ + set_helper_retaddr(retaddr); + cpu_stq_le_data(env, ptr, val); clear_helper_retaddr(); } From patchwork Wed Apr 22 04:32:57 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 185662 Delivered-To: patch@linaro.org Received: by 2002:a92:3d9a:0:0:0:0:0 with SMTP id k26csp299823ilf; Tue, 21 Apr 2020 21:38:00 -0700 (PDT) X-Google-Smtp-Source: APiQypJa2rtiAlnyQuOUcghSucPP/RSG4JX51BGp2b1unqJaDitum0qGGr5qaEO6BZ/PvgeKmNbH X-Received: by 2002:aed:2b43:: with SMTP id p61mr18704356qtd.298.1587530280184; Tue, 21 Apr 2020 21:38:00 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1587530280; cv=none; d=google.com; s=arc-20160816; b=LqATr38WYIiUFC/sOrKgflGOa7ChUhwwFa8bnK/bIt7arVgFMoWo9BuSYb9aHKSU1C L42cbJuT0X4eYCTg6nVrxmnCESBLrzqEa0WN1AYEeAgMM8gzMW0dOmWraIkd5o7yYYKg PVORQpeUmqSkmymUEtnX4U8NgkvJqEfT3SncpYZL56LqDqJNB3uv3TtfpITlkD+UJYNJ 5N3OeBbnSf7O1OxAqkANYaYJwBAuo6ukDXbksn65klJ3WHk7Z4Enxc7q/AW4bzixJkyP N/61Ppw4Kq2LADFezTkfglr/fKr25EtRMavhzjjAuQlTpII1iGUOUoLx21IUEit4ETKf sc8g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:cc:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:content-transfer-encoding :mime-version:references:in-reply-to:message-id:date:subject:to:from :dkim-signature; bh=DksJMGZT4oW5xG6q7nxfDY9bzvBc9Nkm5tzoCILYIaw=; b=y7O5yf0bLscNdzR/wMZrhZIHjVNUSD1Q2FtNEIghFN/Q1C6aYi/kdmQXICjKyHE+rT s77P1UeePiAyXDgOSzi5IZTGRYAosU5q4SgSRdoA6YQ8MsHWQQ17yq23kzc0o7/TGD9I DtSnQgbkP4ZXFKwgVh/zmGi8Xms7hLTVb3f2vvHgQYwKfNf5gg3Xeuq8GRI/NmbLyO5S 9w6geKQuoqRqSVaxRUi35LoUtAr54PKKFWo84y4VaEg3TREiuq0+RMP9lvsbZYhMF3M+ ajO7fPPnEaU9IDF1n6biwJWe65DGWi3uUhxGTAXq3AvjZLZWcQ+3kiBP7B+XvGmf8lb8 fgNQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=EpzfvuRI; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [209.51.188.17]) by mx.google.com with ESMTPS id z21si2555100qtn.342.2020.04.21.21.38.00 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Tue, 21 Apr 2020 21:38:00 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=EpzfvuRI; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1]:41396 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jR795-0007wD-EU for patch@linaro.org; Wed, 22 Apr 2020 00:37:59 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:50356) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jR74f-0000es-18 for qemu-devel@nongnu.org; Wed, 22 Apr 2020 00:33:26 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.90_1) (envelope-from ) id 1jR74d-0005D3-5g for qemu-devel@nongnu.org; Wed, 22 Apr 2020 00:33:24 -0400 Received: from mail-pg1-x541.google.com ([2607:f8b0:4864:20::541]:43654) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1jR74c-000575-LY for qemu-devel@nongnu.org; Wed, 22 Apr 2020 00:33:22 -0400 Received: by mail-pg1-x541.google.com with SMTP id x26so455397pgc.10 for ; Tue, 21 Apr 2020 21:33:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=DksJMGZT4oW5xG6q7nxfDY9bzvBc9Nkm5tzoCILYIaw=; b=EpzfvuRI9G8wJcTozZX5ynKEUktwMTqHsRnqdRgR7SoN7Ew6XzMGtRwTyLsE2exE1L ayWks7HxdT10mjhddOf5GwZTkYQ6dxr9gSsLb3RGvRgnFdesYVKEHI/P4tF1Q/FhXEwS pRVaPJYL2S705dj6iomH88LvtbuAryN9nAaXlii6DUmjDsC0gVRuFT+el4vzGmiGJkuv QHID53a2gvrtyFNDQ0RfIoqz2XYyN5+woSV49ICm3iYE2xd/y4dv6ToN5BTuv2mruBv8 a13PDEcOw5u3dL7W8nGiBie4JINTW18NuZh/UnhxAgrxyNxGRuiQwxkmztmMPhJRBlDp Zlhg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=DksJMGZT4oW5xG6q7nxfDY9bzvBc9Nkm5tzoCILYIaw=; b=IAE36Pwuk3KnTQxRgytXMVwGbyj788Bv8QogLdWpSg5c0xRCGqXZQp3gCi38rG8Scp fnkE1buU4e9KTOMMzNxS4KbmThc1eZ46OZ9puPY6yFE74s2zZl3BVMqZ14RRUEqBAZ6D 3WwrJ54bb/lrSRA/RPeURZ1o1WaY4cm1pLXsfGcAO2C5OEQ0l0k5+caXZKyt7Sm1nUgg z8klW4BHGA55LzBJiV1M5U/fha8o4EAIo4c+W9ecg9t80xonrsgLU4EjUVxaWrDZWN3q rSyyatvB5OwAzmPYyZeQ0n7bA7MWSpJHPzuUh9pDnO+0eji3kypOkCo8kn5rkbxSOtN1 qv0Q== X-Gm-Message-State: AGi0PuZIyJ72qzZWO2R2FNI5ke3KENnJ0vu0wiNs8ZmwfnGHQAHlt0tW Qy3uEsuLS2NKMEiR1kjjcQI1ao3IdgQ= X-Received: by 2002:a63:f50a:: with SMTP id w10mr23869011pgh.181.1587530000303; Tue, 21 Apr 2020 21:33:20 -0700 (PDT) Received: from localhost.localdomain (174-21-149-226.tukw.qwest.net. [174.21.149.226]) by smtp.gmail.com with ESMTPSA id l137sm4129613pfd.107.2020.04.21.21.33.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 21 Apr 2020 21:33:19 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Subject: [PATCH v3 06/18] target/arm: Use cpu_*_data_ra for sve_ldst_tlb_fn Date: Tue, 21 Apr 2020 21:32:57 -0700 Message-Id: <20200422043309.18430-7-richard.henderson@linaro.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200422043309.18430-1-richard.henderson@linaro.org> References: <20200422043309.18430-1-richard.henderson@linaro.org> MIME-Version: 1.0 Received-SPF: pass client-ip=2607:f8b0:4864:20::541; envelope-from=richard.henderson@linaro.org; helo=mail-pg1-x541.google.com X-detected-operating-system: by eggs.gnu.org: Error: [-] PROGRAM ABORT : Malformed IPv6 address (bad octet value). Location : parse_addr6(), p0f-client.c:67 X-Received-From: 2607:f8b0:4864:20::541 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: peter.maydell@linaro.org, qemu-arm@nongnu.org Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: "Qemu-devel" Use the "normal" memory access functions, rather than the softmmu internal helper functions directly. Since fb901c905dc3, cpu_mem_index is now a simple extract from env->hflags and not a large computation. Which means that it's now more work to pass around this value than it is to recompute it. This only adjusts the primitives, and does not clean up all of the uses within sve_helper.c. Signed-off-by: Richard Henderson --- target/arm/sve_helper.c | 221 ++++++++++++++++------------------------ 1 file changed, 86 insertions(+), 135 deletions(-) -- 2.20.1 Reviewed-by: Peter Maydell diff --git a/target/arm/sve_helper.c b/target/arm/sve_helper.c index fdfa652094..655bc9476f 100644 --- a/target/arm/sve_helper.c +++ b/target/arm/sve_helper.c @@ -3991,9 +3991,8 @@ typedef intptr_t sve_ld1_host_fn(void *vd, void *vg, void *host, * Load one element into @vd + @reg_off from (@env, @vaddr, @ra). * The controlling predicate is known to be true. */ -typedef void sve_ld1_tlb_fn(CPUARMState *env, void *vd, intptr_t reg_off, - target_ulong vaddr, TCGMemOpIdx oi, uintptr_t ra); -typedef sve_ld1_tlb_fn sve_st1_tlb_fn; +typedef void sve_ldst1_tlb_fn(CPUARMState *env, void *vd, intptr_t reg_off, + target_ulong vaddr, uintptr_t retaddr); /* * Generate the above primitives. @@ -4016,27 +4015,23 @@ static intptr_t sve_##NAME##_host(void *vd, void *vg, void *host, \ return mem_off; \ } -#ifdef CONFIG_SOFTMMU -#define DO_LD_TLB(NAME, H, TYPEE, TYPEM, HOST, MOEND, TLB) \ +#define DO_LD_TLB(NAME, H, TYPEE, TYPEM, TLB) \ static void sve_##NAME##_tlb(CPUARMState *env, void *vd, intptr_t reg_off, \ - target_ulong addr, TCGMemOpIdx oi, uintptr_t ra) \ + target_ulong addr, uintptr_t ra) \ { \ - TYPEM val = TLB(env, addr, oi, ra); \ - *(TYPEE *)(vd + H(reg_off)) = val; \ + *(TYPEE *)(vd + H(reg_off)) = (TYPEM)TLB(env, addr, ra); \ } -#else -#define DO_LD_TLB(NAME, H, TYPEE, TYPEM, HOST, MOEND, TLB) \ + +#define DO_ST_TLB(NAME, H, TYPEE, TYPEM, TLB) \ static void sve_##NAME##_tlb(CPUARMState *env, void *vd, intptr_t reg_off, \ - target_ulong addr, TCGMemOpIdx oi, uintptr_t ra) \ + target_ulong addr, uintptr_t ra) \ { \ - TYPEM val = HOST(g2h(addr)); \ - *(TYPEE *)(vd + H(reg_off)) = val; \ + TLB(env, addr, (TYPEM)*(TYPEE *)(vd + H(reg_off)), ra); \ } -#endif #define DO_LD_PRIM_1(NAME, H, TE, TM) \ DO_LD_HOST(NAME, H, TE, TM, ldub_p) \ - DO_LD_TLB(NAME, H, TE, TM, ldub_p, 0, helper_ret_ldub_mmu) + DO_LD_TLB(NAME, H, TE, TM, cpu_ldub_data_ra) DO_LD_PRIM_1(ld1bb, H1, uint8_t, uint8_t) DO_LD_PRIM_1(ld1bhu, H1_2, uint16_t, uint8_t) @@ -4046,39 +4041,51 @@ DO_LD_PRIM_1(ld1bss, H1_4, uint32_t, int8_t) DO_LD_PRIM_1(ld1bdu, , uint64_t, uint8_t) DO_LD_PRIM_1(ld1bds, , uint64_t, int8_t) -#define DO_LD_PRIM_2(NAME, end, MOEND, H, TE, TM, PH, PT) \ - DO_LD_HOST(NAME##_##end, H, TE, TM, PH##_##end##_p) \ - DO_LD_TLB(NAME##_##end, H, TE, TM, PH##_##end##_p, \ - MOEND, helper_##end##_##PT##_mmu) +#define DO_ST_PRIM_1(NAME, H, TE, TM) \ + DO_ST_TLB(st1##NAME, H, TE, TM, cpu_stb_data_ra) -DO_LD_PRIM_2(ld1hh, le, MO_LE, H1_2, uint16_t, uint16_t, lduw, lduw) -DO_LD_PRIM_2(ld1hsu, le, MO_LE, H1_4, uint32_t, uint16_t, lduw, lduw) -DO_LD_PRIM_2(ld1hss, le, MO_LE, H1_4, uint32_t, int16_t, lduw, lduw) -DO_LD_PRIM_2(ld1hdu, le, MO_LE, , uint64_t, uint16_t, lduw, lduw) -DO_LD_PRIM_2(ld1hds, le, MO_LE, , uint64_t, int16_t, lduw, lduw) +DO_ST_PRIM_1(bb, H1, uint8_t, uint8_t) +DO_ST_PRIM_1(bh, H1_2, uint16_t, uint8_t) +DO_ST_PRIM_1(bs, H1_4, uint32_t, uint8_t) +DO_ST_PRIM_1(bd, , uint64_t, uint8_t) -DO_LD_PRIM_2(ld1ss, le, MO_LE, H1_4, uint32_t, uint32_t, ldl, ldul) -DO_LD_PRIM_2(ld1sdu, le, MO_LE, , uint64_t, uint32_t, ldl, ldul) -DO_LD_PRIM_2(ld1sds, le, MO_LE, , uint64_t, int32_t, ldl, ldul) +#define DO_LD_PRIM_2(NAME, H, TE, TM, LD) \ + DO_LD_HOST(ld1##NAME##_be, H, TE, TM, LD##_be_p) \ + DO_LD_HOST(ld1##NAME##_le, H, TE, TM, LD##_le_p) \ + DO_LD_TLB(ld1##NAME##_be, H, TE, TM, cpu_##LD##_be_data_ra) \ + DO_LD_TLB(ld1##NAME##_le, H, TE, TM, cpu_##LD##_le_data_ra) -DO_LD_PRIM_2(ld1dd, le, MO_LE, , uint64_t, uint64_t, ldq, ldq) +#define DO_ST_PRIM_2(NAME, H, TE, TM, ST) \ + DO_ST_TLB(st1##NAME##_be, H, TE, TM, cpu_##ST##_be_data_ra) \ + DO_ST_TLB(st1##NAME##_le, H, TE, TM, cpu_##ST##_le_data_ra) -DO_LD_PRIM_2(ld1hh, be, MO_BE, H1_2, uint16_t, uint16_t, lduw, lduw) -DO_LD_PRIM_2(ld1hsu, be, MO_BE, H1_4, uint32_t, uint16_t, lduw, lduw) -DO_LD_PRIM_2(ld1hss, be, MO_BE, H1_4, uint32_t, int16_t, lduw, lduw) -DO_LD_PRIM_2(ld1hdu, be, MO_BE, , uint64_t, uint16_t, lduw, lduw) -DO_LD_PRIM_2(ld1hds, be, MO_BE, , uint64_t, int16_t, lduw, lduw) +DO_LD_PRIM_2(hh, H1_2, uint16_t, uint16_t, lduw) +DO_LD_PRIM_2(hsu, H1_4, uint32_t, uint16_t, lduw) +DO_LD_PRIM_2(hss, H1_4, uint32_t, int16_t, lduw) +DO_LD_PRIM_2(hdu, , uint64_t, uint16_t, lduw) +DO_LD_PRIM_2(hds, , uint64_t, int16_t, lduw) -DO_LD_PRIM_2(ld1ss, be, MO_BE, H1_4, uint32_t, uint32_t, ldl, ldul) -DO_LD_PRIM_2(ld1sdu, be, MO_BE, , uint64_t, uint32_t, ldl, ldul) -DO_LD_PRIM_2(ld1sds, be, MO_BE, , uint64_t, int32_t, ldl, ldul) +DO_ST_PRIM_2(hh, H1_2, uint16_t, uint16_t, stw) +DO_ST_PRIM_2(hs, H1_4, uint32_t, uint16_t, stw) +DO_ST_PRIM_2(hd, , uint64_t, uint16_t, stw) -DO_LD_PRIM_2(ld1dd, be, MO_BE, , uint64_t, uint64_t, ldq, ldq) +DO_LD_PRIM_2(ss, H1_4, uint32_t, uint32_t, ldl) +DO_LD_PRIM_2(sdu, , uint64_t, uint32_t, ldl) +DO_LD_PRIM_2(sds, , uint64_t, int32_t, ldl) + +DO_ST_PRIM_2(ss, H1_4, uint32_t, uint32_t, stl) +DO_ST_PRIM_2(sd, , uint64_t, uint32_t, stl) + +DO_LD_PRIM_2(dd, , uint64_t, uint64_t, ldq) +DO_ST_PRIM_2(dd, , uint64_t, uint64_t, stq) #undef DO_LD_TLB +#undef DO_ST_TLB #undef DO_LD_HOST #undef DO_LD_PRIM_1 +#undef DO_ST_PRIM_1 #undef DO_LD_PRIM_2 +#undef DO_ST_PRIM_2 /* * Skip through a sequence of inactive elements in the guarding predicate @vg, @@ -4152,7 +4159,7 @@ static void sve_ld1_r(CPUARMState *env, void *vg, const target_ulong addr, uint32_t desc, const uintptr_t retaddr, const int esz, const int msz, sve_ld1_host_fn *host_fn, - sve_ld1_tlb_fn *tlb_fn) + sve_ldst1_tlb_fn *tlb_fn) { const TCGMemOpIdx oi = extract32(desc, SIMD_DATA_SHIFT, MEMOPIDX_SHIFT); const int mmu_idx = get_mmuidx(oi); @@ -4234,7 +4241,7 @@ static void sve_ld1_r(CPUARMState *env, void *vg, const target_ulong addr, * on I/O memory, it may succeed but not bring in the TLB entry. * But even then we have still made forward progress. */ - tlb_fn(env, &scratch, reg_off, addr + mem_off, oi, retaddr); + tlb_fn(env, &scratch, reg_off, addr + mem_off, retaddr); reg_off += 1 << esz; } #endif @@ -4293,9 +4300,8 @@ DO_LD1_2(ld1dd, 3, 3) */ static void sve_ld2_r(CPUARMState *env, void *vg, target_ulong addr, uint32_t desc, int size, uintptr_t ra, - sve_ld1_tlb_fn *tlb_fn) + sve_ldst1_tlb_fn *tlb_fn) { - const TCGMemOpIdx oi = extract32(desc, SIMD_DATA_SHIFT, MEMOPIDX_SHIFT); const unsigned rd = extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT, 5); intptr_t i, oprsz = simd_oprsz(desc); ARMVectorReg scratch[2] = { }; @@ -4305,8 +4311,8 @@ static void sve_ld2_r(CPUARMState *env, void *vg, target_ulong addr, uint16_t pg = *(uint16_t *)(vg + H1_2(i >> 3)); do { if (pg & 1) { - tlb_fn(env, &scratch[0], i, addr, oi, ra); - tlb_fn(env, &scratch[1], i, addr + size, oi, ra); + tlb_fn(env, &scratch[0], i, addr, ra); + tlb_fn(env, &scratch[1], i, addr + size, ra); } i += size, pg >>= size; addr += 2 * size; @@ -4321,9 +4327,8 @@ static void sve_ld2_r(CPUARMState *env, void *vg, target_ulong addr, static void sve_ld3_r(CPUARMState *env, void *vg, target_ulong addr, uint32_t desc, int size, uintptr_t ra, - sve_ld1_tlb_fn *tlb_fn) + sve_ldst1_tlb_fn *tlb_fn) { - const TCGMemOpIdx oi = extract32(desc, SIMD_DATA_SHIFT, MEMOPIDX_SHIFT); const unsigned rd = extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT, 5); intptr_t i, oprsz = simd_oprsz(desc); ARMVectorReg scratch[3] = { }; @@ -4333,9 +4338,9 @@ static void sve_ld3_r(CPUARMState *env, void *vg, target_ulong addr, uint16_t pg = *(uint16_t *)(vg + H1_2(i >> 3)); do { if (pg & 1) { - tlb_fn(env, &scratch[0], i, addr, oi, ra); - tlb_fn(env, &scratch[1], i, addr + size, oi, ra); - tlb_fn(env, &scratch[2], i, addr + 2 * size, oi, ra); + tlb_fn(env, &scratch[0], i, addr, ra); + tlb_fn(env, &scratch[1], i, addr + size, ra); + tlb_fn(env, &scratch[2], i, addr + 2 * size, ra); } i += size, pg >>= size; addr += 3 * size; @@ -4351,9 +4356,8 @@ static void sve_ld3_r(CPUARMState *env, void *vg, target_ulong addr, static void sve_ld4_r(CPUARMState *env, void *vg, target_ulong addr, uint32_t desc, int size, uintptr_t ra, - sve_ld1_tlb_fn *tlb_fn) + sve_ldst1_tlb_fn *tlb_fn) { - const TCGMemOpIdx oi = extract32(desc, SIMD_DATA_SHIFT, MEMOPIDX_SHIFT); const unsigned rd = extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT, 5); intptr_t i, oprsz = simd_oprsz(desc); ARMVectorReg scratch[4] = { }; @@ -4363,10 +4367,10 @@ static void sve_ld4_r(CPUARMState *env, void *vg, target_ulong addr, uint16_t pg = *(uint16_t *)(vg + H1_2(i >> 3)); do { if (pg & 1) { - tlb_fn(env, &scratch[0], i, addr, oi, ra); - tlb_fn(env, &scratch[1], i, addr + size, oi, ra); - tlb_fn(env, &scratch[2], i, addr + 2 * size, oi, ra); - tlb_fn(env, &scratch[3], i, addr + 3 * size, oi, ra); + tlb_fn(env, &scratch[0], i, addr, ra); + tlb_fn(env, &scratch[1], i, addr + size, ra); + tlb_fn(env, &scratch[2], i, addr + 2 * size, ra); + tlb_fn(env, &scratch[3], i, addr + 3 * size, ra); } i += size, pg >>= size; addr += 4 * size; @@ -4459,7 +4463,7 @@ static void sve_ldff1_r(CPUARMState *env, void *vg, const target_ulong addr, uint32_t desc, const uintptr_t retaddr, const int esz, const int msz, sve_ld1_host_fn *host_fn, - sve_ld1_tlb_fn *tlb_fn) + sve_ldst1_tlb_fn *tlb_fn) { const TCGMemOpIdx oi = extract32(desc, SIMD_DATA_SHIFT, MEMOPIDX_SHIFT); const int mmu_idx = get_mmuidx(oi); @@ -4519,7 +4523,7 @@ static void sve_ldff1_r(CPUARMState *env, void *vg, const target_ulong addr, * Perform one normal read, which will fault or not. * But it is likely to bring the page into the tlb. */ - tlb_fn(env, vd, reg_off, addr + mem_off, oi, retaddr); + tlb_fn(env, vd, reg_off, addr + mem_off, retaddr); /* After any fault, zero any leading predicated false elts. */ swap_memzero(vd, reg_off); @@ -4671,60 +4675,14 @@ DO_LDFF1_LDNF1_2(dd, 3, 3) #undef DO_LDFF1_LDNF1_1 #undef DO_LDFF1_LDNF1_2 -/* - * Store contiguous data, protected by a governing predicate. - */ - -#ifdef CONFIG_SOFTMMU -#define DO_ST_TLB(NAME, H, TYPEM, HOST, MOEND, TLB) \ -static void sve_##NAME##_tlb(CPUARMState *env, void *vd, intptr_t reg_off, \ - target_ulong addr, TCGMemOpIdx oi, uintptr_t ra) \ -{ \ - TLB(env, addr, *(TYPEM *)(vd + H(reg_off)), oi, ra); \ -} -#else -#define DO_ST_TLB(NAME, H, TYPEM, HOST, MOEND, TLB) \ -static void sve_##NAME##_tlb(CPUARMState *env, void *vd, intptr_t reg_off, \ - target_ulong addr, TCGMemOpIdx oi, uintptr_t ra) \ -{ \ - HOST(g2h(addr), *(TYPEM *)(vd + H(reg_off))); \ -} -#endif - -DO_ST_TLB(st1bb, H1, uint8_t, stb_p, 0, helper_ret_stb_mmu) -DO_ST_TLB(st1bh, H1_2, uint16_t, stb_p, 0, helper_ret_stb_mmu) -DO_ST_TLB(st1bs, H1_4, uint32_t, stb_p, 0, helper_ret_stb_mmu) -DO_ST_TLB(st1bd, , uint64_t, stb_p, 0, helper_ret_stb_mmu) - -DO_ST_TLB(st1hh_le, H1_2, uint16_t, stw_le_p, MO_LE, helper_le_stw_mmu) -DO_ST_TLB(st1hs_le, H1_4, uint32_t, stw_le_p, MO_LE, helper_le_stw_mmu) -DO_ST_TLB(st1hd_le, , uint64_t, stw_le_p, MO_LE, helper_le_stw_mmu) - -DO_ST_TLB(st1ss_le, H1_4, uint32_t, stl_le_p, MO_LE, helper_le_stl_mmu) -DO_ST_TLB(st1sd_le, , uint64_t, stl_le_p, MO_LE, helper_le_stl_mmu) - -DO_ST_TLB(st1dd_le, , uint64_t, stq_le_p, MO_LE, helper_le_stq_mmu) - -DO_ST_TLB(st1hh_be, H1_2, uint16_t, stw_be_p, MO_BE, helper_be_stw_mmu) -DO_ST_TLB(st1hs_be, H1_4, uint32_t, stw_be_p, MO_BE, helper_be_stw_mmu) -DO_ST_TLB(st1hd_be, , uint64_t, stw_be_p, MO_BE, helper_be_stw_mmu) - -DO_ST_TLB(st1ss_be, H1_4, uint32_t, stl_be_p, MO_BE, helper_be_stl_mmu) -DO_ST_TLB(st1sd_be, , uint64_t, stl_be_p, MO_BE, helper_be_stl_mmu) - -DO_ST_TLB(st1dd_be, , uint64_t, stq_be_p, MO_BE, helper_be_stq_mmu) - -#undef DO_ST_TLB - /* * Common helpers for all contiguous 1,2,3,4-register predicated stores. */ static void sve_st1_r(CPUARMState *env, void *vg, target_ulong addr, uint32_t desc, const uintptr_t ra, const int esize, const int msize, - sve_st1_tlb_fn *tlb_fn) + sve_ldst1_tlb_fn *tlb_fn) { - const TCGMemOpIdx oi = extract32(desc, SIMD_DATA_SHIFT, MEMOPIDX_SHIFT); const unsigned rd = extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT, 5); intptr_t i, oprsz = simd_oprsz(desc); void *vd = &env->vfp.zregs[rd]; @@ -4734,7 +4692,7 @@ static void sve_st1_r(CPUARMState *env, void *vg, target_ulong addr, uint16_t pg = *(uint16_t *)(vg + H1_2(i >> 3)); do { if (pg & 1) { - tlb_fn(env, vd, i, addr, oi, ra); + tlb_fn(env, vd, i, addr, ra); } i += esize, pg >>= esize; addr += msize; @@ -4746,9 +4704,8 @@ static void sve_st1_r(CPUARMState *env, void *vg, target_ulong addr, static void sve_st2_r(CPUARMState *env, void *vg, target_ulong addr, uint32_t desc, const uintptr_t ra, const int esize, const int msize, - sve_st1_tlb_fn *tlb_fn) + sve_ldst1_tlb_fn *tlb_fn) { - const TCGMemOpIdx oi = extract32(desc, SIMD_DATA_SHIFT, MEMOPIDX_SHIFT); const unsigned rd = extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT, 5); intptr_t i, oprsz = simd_oprsz(desc); void *d1 = &env->vfp.zregs[rd]; @@ -4759,8 +4716,8 @@ static void sve_st2_r(CPUARMState *env, void *vg, target_ulong addr, uint16_t pg = *(uint16_t *)(vg + H1_2(i >> 3)); do { if (pg & 1) { - tlb_fn(env, d1, i, addr, oi, ra); - tlb_fn(env, d2, i, addr + msize, oi, ra); + tlb_fn(env, d1, i, addr, ra); + tlb_fn(env, d2, i, addr + msize, ra); } i += esize, pg >>= esize; addr += 2 * msize; @@ -4772,9 +4729,8 @@ static void sve_st2_r(CPUARMState *env, void *vg, target_ulong addr, static void sve_st3_r(CPUARMState *env, void *vg, target_ulong addr, uint32_t desc, const uintptr_t ra, const int esize, const int msize, - sve_st1_tlb_fn *tlb_fn) + sve_ldst1_tlb_fn *tlb_fn) { - const TCGMemOpIdx oi = extract32(desc, SIMD_DATA_SHIFT, MEMOPIDX_SHIFT); const unsigned rd = extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT, 5); intptr_t i, oprsz = simd_oprsz(desc); void *d1 = &env->vfp.zregs[rd]; @@ -4786,9 +4742,9 @@ static void sve_st3_r(CPUARMState *env, void *vg, target_ulong addr, uint16_t pg = *(uint16_t *)(vg + H1_2(i >> 3)); do { if (pg & 1) { - tlb_fn(env, d1, i, addr, oi, ra); - tlb_fn(env, d2, i, addr + msize, oi, ra); - tlb_fn(env, d3, i, addr + 2 * msize, oi, ra); + tlb_fn(env, d1, i, addr, ra); + tlb_fn(env, d2, i, addr + msize, ra); + tlb_fn(env, d3, i, addr + 2 * msize, ra); } i += esize, pg >>= esize; addr += 3 * msize; @@ -4800,9 +4756,8 @@ static void sve_st3_r(CPUARMState *env, void *vg, target_ulong addr, static void sve_st4_r(CPUARMState *env, void *vg, target_ulong addr, uint32_t desc, const uintptr_t ra, const int esize, const int msize, - sve_st1_tlb_fn *tlb_fn) + sve_ldst1_tlb_fn *tlb_fn) { - const TCGMemOpIdx oi = extract32(desc, SIMD_DATA_SHIFT, MEMOPIDX_SHIFT); const unsigned rd = extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT, 5); intptr_t i, oprsz = simd_oprsz(desc); void *d1 = &env->vfp.zregs[rd]; @@ -4815,10 +4770,10 @@ static void sve_st4_r(CPUARMState *env, void *vg, target_ulong addr, uint16_t pg = *(uint16_t *)(vg + H1_2(i >> 3)); do { if (pg & 1) { - tlb_fn(env, d1, i, addr, oi, ra); - tlb_fn(env, d2, i, addr + msize, oi, ra); - tlb_fn(env, d3, i, addr + 2 * msize, oi, ra); - tlb_fn(env, d4, i, addr + 3 * msize, oi, ra); + tlb_fn(env, d1, i, addr, ra); + tlb_fn(env, d2, i, addr + msize, ra); + tlb_fn(env, d3, i, addr + 2 * msize, ra); + tlb_fn(env, d4, i, addr + 3 * msize, ra); } i += esize, pg >>= esize; addr += 4 * msize; @@ -4914,9 +4869,8 @@ static target_ulong off_zd_d(void *reg, intptr_t reg_ofs) static void sve_ld1_zs(CPUARMState *env, void *vd, void *vg, void *vm, target_ulong base, uint32_t desc, uintptr_t ra, - zreg_off_fn *off_fn, sve_ld1_tlb_fn *tlb_fn) + zreg_off_fn *off_fn, sve_ldst1_tlb_fn *tlb_fn) { - const TCGMemOpIdx oi = extract32(desc, SIMD_DATA_SHIFT, MEMOPIDX_SHIFT); const int scale = extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT, 2); intptr_t i, oprsz = simd_oprsz(desc); ARMVectorReg scratch = { }; @@ -4927,7 +4881,7 @@ static void sve_ld1_zs(CPUARMState *env, void *vd, void *vg, void *vm, do { if (likely(pg & 1)) { target_ulong off = off_fn(vm, i); - tlb_fn(env, &scratch, i, base + (off << scale), oi, ra); + tlb_fn(env, &scratch, i, base + (off << scale), ra); } i += 4, pg >>= 4; } while (i & 15); @@ -4940,9 +4894,8 @@ static void sve_ld1_zs(CPUARMState *env, void *vd, void *vg, void *vm, static void sve_ld1_zd(CPUARMState *env, void *vd, void *vg, void *vm, target_ulong base, uint32_t desc, uintptr_t ra, - zreg_off_fn *off_fn, sve_ld1_tlb_fn *tlb_fn) + zreg_off_fn *off_fn, sve_ldst1_tlb_fn *tlb_fn) { - const TCGMemOpIdx oi = extract32(desc, SIMD_DATA_SHIFT, MEMOPIDX_SHIFT); const int scale = extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT, 2); intptr_t i, oprsz = simd_oprsz(desc) / 8; ARMVectorReg scratch = { }; @@ -4952,7 +4905,7 @@ static void sve_ld1_zd(CPUARMState *env, void *vd, void *vg, void *vm, uint8_t pg = *(uint8_t *)(vg + H1(i)); if (likely(pg & 1)) { target_ulong off = off_fn(vm, i * 8); - tlb_fn(env, &scratch, i * 8, base + (off << scale), oi, ra); + tlb_fn(env, &scratch, i * 8, base + (off << scale), ra); } } clear_helper_retaddr(); @@ -5114,7 +5067,7 @@ DO_LD_NF(dd_be, , uint64_t, uint64_t, ldq_be_p) */ static inline void sve_ldff1_zs(CPUARMState *env, void *vd, void *vg, void *vm, target_ulong base, uint32_t desc, uintptr_t ra, - zreg_off_fn *off_fn, sve_ld1_tlb_fn *tlb_fn, + zreg_off_fn *off_fn, sve_ldst1_tlb_fn *tlb_fn, sve_ld1_nf_fn *nonfault_fn) { const TCGMemOpIdx oi = extract32(desc, SIMD_DATA_SHIFT, MEMOPIDX_SHIFT); @@ -5130,7 +5083,7 @@ static inline void sve_ldff1_zs(CPUARMState *env, void *vd, void *vg, void *vm, set_helper_retaddr(ra); addr = off_fn(vm, reg_off); addr = base + (addr << scale); - tlb_fn(env, vd, reg_off, addr, oi, ra); + tlb_fn(env, vd, reg_off, addr, ra); /* The rest of the reads will be non-faulting. */ clear_helper_retaddr(); @@ -5156,7 +5109,7 @@ static inline void sve_ldff1_zs(CPUARMState *env, void *vd, void *vg, void *vm, static inline void sve_ldff1_zd(CPUARMState *env, void *vd, void *vg, void *vm, target_ulong base, uint32_t desc, uintptr_t ra, - zreg_off_fn *off_fn, sve_ld1_tlb_fn *tlb_fn, + zreg_off_fn *off_fn, sve_ldst1_tlb_fn *tlb_fn, sve_ld1_nf_fn *nonfault_fn) { const TCGMemOpIdx oi = extract32(desc, SIMD_DATA_SHIFT, MEMOPIDX_SHIFT); @@ -5172,7 +5125,7 @@ static inline void sve_ldff1_zd(CPUARMState *env, void *vd, void *vg, void *vm, set_helper_retaddr(ra); addr = off_fn(vm, reg_off); addr = base + (addr << scale); - tlb_fn(env, vd, reg_off, addr, oi, ra); + tlb_fn(env, vd, reg_off, addr, ra); /* The rest of the reads will be non-faulting. */ clear_helper_retaddr(); @@ -5282,9 +5235,8 @@ DO_LDFF1_ZPZ_D(dd_be, zd) static void sve_st1_zs(CPUARMState *env, void *vd, void *vg, void *vm, target_ulong base, uint32_t desc, uintptr_t ra, - zreg_off_fn *off_fn, sve_ld1_tlb_fn *tlb_fn) + zreg_off_fn *off_fn, sve_ldst1_tlb_fn *tlb_fn) { - const TCGMemOpIdx oi = extract32(desc, SIMD_DATA_SHIFT, MEMOPIDX_SHIFT); const int scale = extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT, 2); intptr_t i, oprsz = simd_oprsz(desc); @@ -5294,7 +5246,7 @@ static void sve_st1_zs(CPUARMState *env, void *vd, void *vg, void *vm, do { if (likely(pg & 1)) { target_ulong off = off_fn(vm, i); - tlb_fn(env, vd, i, base + (off << scale), oi, ra); + tlb_fn(env, vd, i, base + (off << scale), ra); } i += 4, pg >>= 4; } while (i & 15); @@ -5304,9 +5256,8 @@ static void sve_st1_zs(CPUARMState *env, void *vd, void *vg, void *vm, static void sve_st1_zd(CPUARMState *env, void *vd, void *vg, void *vm, target_ulong base, uint32_t desc, uintptr_t ra, - zreg_off_fn *off_fn, sve_ld1_tlb_fn *tlb_fn) + zreg_off_fn *off_fn, sve_ldst1_tlb_fn *tlb_fn) { - const TCGMemOpIdx oi = extract32(desc, SIMD_DATA_SHIFT, MEMOPIDX_SHIFT); const int scale = extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT, 2); intptr_t i, oprsz = simd_oprsz(desc) / 8; @@ -5315,7 +5266,7 @@ static void sve_st1_zd(CPUARMState *env, void *vd, void *vg, void *vm, uint8_t pg = *(uint8_t *)(vg + H1(i)); if (likely(pg & 1)) { target_ulong off = off_fn(vm, i * 8); - tlb_fn(env, vd, i * 8, base + (off << scale), oi, ra); + tlb_fn(env, vd, i * 8, base + (off << scale), ra); } } clear_helper_retaddr(); From patchwork Wed Apr 22 04:32:58 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 185657 Delivered-To: patch@linaro.org Received: by 2002:a92:3d9a:0:0:0:0:0 with SMTP id k26csp297003ilf; Tue, 21 Apr 2020 21:33:46 -0700 (PDT) X-Google-Smtp-Source: APiQypLUxImNNjmjnqqzo2ixEF2DuNvOuf8lwKq2nwsAzpLlGUpNPAvoL7HazoJUpmM5+018HVgN X-Received: by 2002:ac8:7581:: with SMTP id s1mr10992361qtq.77.1587530026763; Tue, 21 Apr 2020 21:33:46 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1587530026; cv=none; d=google.com; s=arc-20160816; b=lMm6fPWmG9pB1E381G24B1AQOO2L/Hbwiqo7fGpwyACyFV2nbTO8CX2TtcPbKpWuI0 hzRJ55Z4Ub+r2B7+qdBiN0NKD1FwiC/9B6Wh9yJ29ziKBT+19ZNEew7ti6Kt8pmGpIRi fPmB+9Um3GzCJWzUBk34DEre8WUYAhEvvnb+RgLroiqPcGD7cucXvg16DxwKqS7g6Gae 7lOqRU/TD2NtAkICRZCLw4sk3b8XgIpSFpl3Y2diZaHF2R3sE3b1yw+RcbK7nNcmXenO d5+aGFCR00tS99Vp9bjsTql6cQmvv0+KPX3A1g7hYLLpl8IcYm3O9nE6sWHwkbERI/wm /nbg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:cc:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:content-transfer-encoding :mime-version:references:in-reply-to:message-id:date:subject:to:from :dkim-signature; bh=fF+5wwLOen+npTtt/gQ1qVYGPHz8onoyiX2hDw1RdH0=; b=DxDqSM4Hpkai170N/oFriaYyn3BXsu+tJyhcItvq3hQzGo/SN/aybW75INDPO2diSr 4irizKMDCDuO5yk9mPzbdl7hzvXh6imoJJJaeYC5zKOhrQnp9JPQZVmPzAUK9XOV9qgK qwDW9lB81nmPZ68vpQ7WSesrt7vl+6KJ6z8AKRzWkHJ4VPnC1KB9rvg0GnVn77hRMqB9 4tNZdSYFoTzly92ibl9X44ZXglMdaYnhLq2oxTC4LYVS5AvLTD62McrS7dxv/iet9m6Q LZJFmhvVhbq2rN+yEwgKy4OBW9N2i9hBGcRaLi6YcqvXhqKr6G/wG72i9+TiQDrjomxY gpyw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=G0gPa6Da; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [209.51.188.17]) by mx.google.com with ESMTPS id n22si2413979qtp.366.2020.04.21.21.33.46 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Tue, 21 Apr 2020 21:33:46 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=G0gPa6Da; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1]:41186 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jR750-0000ms-4I for patch@linaro.org; Wed, 22 Apr 2020 00:33:46 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:50388) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jR74g-0000gG-Mh for qemu-devel@nongnu.org; Wed, 22 Apr 2020 00:33:27 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.90_1) (envelope-from ) id 1jR74e-0005Gf-3E for qemu-devel@nongnu.org; Wed, 22 Apr 2020 00:33:25 -0400 Received: from mail-pl1-x643.google.com ([2607:f8b0:4864:20::643]:45311) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1jR74d-0005AK-Gf for qemu-devel@nongnu.org; Wed, 22 Apr 2020 00:33:23 -0400 Received: by mail-pl1-x643.google.com with SMTP id t4so423775plq.12 for ; Tue, 21 Apr 2020 21:33:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=fF+5wwLOen+npTtt/gQ1qVYGPHz8onoyiX2hDw1RdH0=; b=G0gPa6DatS4Qs7B0xQ6cnq+NzGzS0+WBV33MPPCI7JiY/q6XWwYjBSdPvtqayA1pyk 52Ut1Ro4moyOICFIkKnM94O5oFpCxZq4iAB9jli/QZ0tPzx2Fq/oxJYxzhJ7cwmQBIzK RzZdLOIwpaeazlF0JcJL/GK5AsA+c4ZJj/5S/K3POHXh89M/RsXmUQWph4RFQ9gfoCsu 4CBcPJgkstDyb/2IY1qm2pamLMoA/jkCXTJx3BwmX6dKlOCVBaVJUZhHrWb0j35K61+b ns8EXK3tHxTtQyz9eoA0f04QGfz+Ov5oVfjEtVThW8GR3B0gRAkD5eCsMW/Q7+6q+1GN 5luw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=fF+5wwLOen+npTtt/gQ1qVYGPHz8onoyiX2hDw1RdH0=; b=AO6Rmi7Pd4SoP3yQLNeDZk8Durw9Fo+2M7CQI3R2bU6IKG3ZbQQhRPUVBcRLouGJEP KgsKjX6L8yexxXmvawDfksBI4djVHtiVN0xTDpYZm7WHnqYPQnNsgG7SlBnrsBMPGRwo OHkW9Jo5NnjcKtOXcVwD7vM4mZ5UDPOcYiIEN8u3yzEQ9Sj/03XxtncskKrUrue13NXG 73jHJATJ52MUWqYlx7AO+sfzsf7wZ05JhjGZBiILAPp2En20pmCo9PFNG+n4RhFLP1Bq X5hmOvkv25HM0JWDpNk1X4BAL4FqjmWGKt1gOh1+gV7HqhfUNSfnSzJ1RJ3wH78uK/eZ QUiA== X-Gm-Message-State: AGi0PubMmeGsnQcDIwBktsA7N/eSUEv9CQ298xaDP7WMt1ipW1+OM8ib wPjz9ze3HpDjvlbLqMOls0smCzhyl8Q= X-Received: by 2002:a17:90a:6581:: with SMTP id k1mr9513665pjj.50.1587530001301; Tue, 21 Apr 2020 21:33:21 -0700 (PDT) Received: from localhost.localdomain (174-21-149-226.tukw.qwest.net. [174.21.149.226]) by smtp.gmail.com with ESMTPSA id l137sm4129613pfd.107.2020.04.21.21.33.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 21 Apr 2020 21:33:20 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Subject: [PATCH v3 07/18] target/arm: Drop manual handling of set/clear_helper_retaddr Date: Tue, 21 Apr 2020 21:32:58 -0700 Message-Id: <20200422043309.18430-8-richard.henderson@linaro.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200422043309.18430-1-richard.henderson@linaro.org> References: <20200422043309.18430-1-richard.henderson@linaro.org> MIME-Version: 1.0 Received-SPF: pass client-ip=2607:f8b0:4864:20::643; envelope-from=richard.henderson@linaro.org; helo=mail-pl1-x643.google.com X-detected-operating-system: by eggs.gnu.org: Error: [-] PROGRAM ABORT : Malformed IPv6 address (bad octet value). Location : parse_addr6(), p0f-client.c:67 X-Received-From: 2607:f8b0:4864:20::643 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: peter.maydell@linaro.org, qemu-arm@nongnu.org Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: "Qemu-devel" Since we converted back to cpu_*_data_ra, we do not need to do this ourselves. Reviewed-by: Peter Maydell Signed-off-by: Richard Henderson --- target/arm/sve_helper.c | 38 -------------------------------------- 1 file changed, 38 deletions(-) -- 2.20.1 diff --git a/target/arm/sve_helper.c b/target/arm/sve_helper.c index 655bc9476f..aad2c8c237 100644 --- a/target/arm/sve_helper.c +++ b/target/arm/sve_helper.c @@ -4133,12 +4133,6 @@ static intptr_t max_for_page(target_ulong base, intptr_t mem_off, return MIN(split, mem_max - mem_off) + mem_off; } -#ifndef CONFIG_USER_ONLY -/* These are normally defined only for CONFIG_USER_ONLY in */ -static inline void set_helper_retaddr(uintptr_t ra) { } -static inline void clear_helper_retaddr(void) { } -#endif - /* * The result of tlb_vaddr_to_host for user-only is just g2h(x), * which is always non-null. Elide the useless test. @@ -4180,7 +4174,6 @@ static void sve_ld1_r(CPUARMState *env, void *vg, const target_ulong addr, return; } mem_off = reg_off >> diffsz; - set_helper_retaddr(retaddr); /* * If the (remaining) load is entirely within a single page, then: @@ -4195,7 +4188,6 @@ static void sve_ld1_r(CPUARMState *env, void *vg, const target_ulong addr, if (test_host_page(host)) { mem_off = host_fn(vd, vg, host - mem_off, mem_off, mem_max); tcg_debug_assert(mem_off == mem_max); - clear_helper_retaddr(); /* After having taken any fault, zero leading inactive elements. */ swap_memzero(vd, reg_off); return; @@ -4246,7 +4238,6 @@ static void sve_ld1_r(CPUARMState *env, void *vg, const target_ulong addr, } #endif - clear_helper_retaddr(); memcpy(vd, &scratch, reg_max); } @@ -4306,7 +4297,6 @@ static void sve_ld2_r(CPUARMState *env, void *vg, target_ulong addr, intptr_t i, oprsz = simd_oprsz(desc); ARMVectorReg scratch[2] = { }; - set_helper_retaddr(ra); for (i = 0; i < oprsz; ) { uint16_t pg = *(uint16_t *)(vg + H1_2(i >> 3)); do { @@ -4318,7 +4308,6 @@ static void sve_ld2_r(CPUARMState *env, void *vg, target_ulong addr, addr += 2 * size; } while (i & 15); } - clear_helper_retaddr(); /* Wait until all exceptions have been raised to write back. */ memcpy(&env->vfp.zregs[rd], &scratch[0], oprsz); @@ -4333,7 +4322,6 @@ static void sve_ld3_r(CPUARMState *env, void *vg, target_ulong addr, intptr_t i, oprsz = simd_oprsz(desc); ARMVectorReg scratch[3] = { }; - set_helper_retaddr(ra); for (i = 0; i < oprsz; ) { uint16_t pg = *(uint16_t *)(vg + H1_2(i >> 3)); do { @@ -4346,7 +4334,6 @@ static void sve_ld3_r(CPUARMState *env, void *vg, target_ulong addr, addr += 3 * size; } while (i & 15); } - clear_helper_retaddr(); /* Wait until all exceptions have been raised to write back. */ memcpy(&env->vfp.zregs[rd], &scratch[0], oprsz); @@ -4362,7 +4349,6 @@ static void sve_ld4_r(CPUARMState *env, void *vg, target_ulong addr, intptr_t i, oprsz = simd_oprsz(desc); ARMVectorReg scratch[4] = { }; - set_helper_retaddr(ra); for (i = 0; i < oprsz; ) { uint16_t pg = *(uint16_t *)(vg + H1_2(i >> 3)); do { @@ -4376,7 +4362,6 @@ static void sve_ld4_r(CPUARMState *env, void *vg, target_ulong addr, addr += 4 * size; } while (i & 15); } - clear_helper_retaddr(); /* Wait until all exceptions have been raised to write back. */ memcpy(&env->vfp.zregs[rd], &scratch[0], oprsz); @@ -4483,7 +4468,6 @@ static void sve_ldff1_r(CPUARMState *env, void *vg, const target_ulong addr, return; } mem_off = reg_off >> diffsz; - set_helper_retaddr(retaddr); /* * If the (remaining) load is entirely within a single page, then: @@ -4498,7 +4482,6 @@ static void sve_ldff1_r(CPUARMState *env, void *vg, const target_ulong addr, if (test_host_page(host)) { mem_off = host_fn(vd, vg, host - mem_off, mem_off, mem_max); tcg_debug_assert(mem_off == mem_max); - clear_helper_retaddr(); /* After any fault, zero any leading inactive elements. */ swap_memzero(vd, reg_off); return; @@ -4541,7 +4524,6 @@ static void sve_ldff1_r(CPUARMState *env, void *vg, const target_ulong addr, } #endif - clear_helper_retaddr(); record_fault(env, reg_off, reg_max); } @@ -4687,7 +4669,6 @@ static void sve_st1_r(CPUARMState *env, void *vg, target_ulong addr, intptr_t i, oprsz = simd_oprsz(desc); void *vd = &env->vfp.zregs[rd]; - set_helper_retaddr(ra); for (i = 0; i < oprsz; ) { uint16_t pg = *(uint16_t *)(vg + H1_2(i >> 3)); do { @@ -4698,7 +4679,6 @@ static void sve_st1_r(CPUARMState *env, void *vg, target_ulong addr, addr += msize; } while (i & 15); } - clear_helper_retaddr(); } static void sve_st2_r(CPUARMState *env, void *vg, target_ulong addr, @@ -4711,7 +4691,6 @@ static void sve_st2_r(CPUARMState *env, void *vg, target_ulong addr, void *d1 = &env->vfp.zregs[rd]; void *d2 = &env->vfp.zregs[(rd + 1) & 31]; - set_helper_retaddr(ra); for (i = 0; i < oprsz; ) { uint16_t pg = *(uint16_t *)(vg + H1_2(i >> 3)); do { @@ -4723,7 +4702,6 @@ static void sve_st2_r(CPUARMState *env, void *vg, target_ulong addr, addr += 2 * msize; } while (i & 15); } - clear_helper_retaddr(); } static void sve_st3_r(CPUARMState *env, void *vg, target_ulong addr, @@ -4737,7 +4715,6 @@ static void sve_st3_r(CPUARMState *env, void *vg, target_ulong addr, void *d2 = &env->vfp.zregs[(rd + 1) & 31]; void *d3 = &env->vfp.zregs[(rd + 2) & 31]; - set_helper_retaddr(ra); for (i = 0; i < oprsz; ) { uint16_t pg = *(uint16_t *)(vg + H1_2(i >> 3)); do { @@ -4750,7 +4727,6 @@ static void sve_st3_r(CPUARMState *env, void *vg, target_ulong addr, addr += 3 * msize; } while (i & 15); } - clear_helper_retaddr(); } static void sve_st4_r(CPUARMState *env, void *vg, target_ulong addr, @@ -4765,7 +4741,6 @@ static void sve_st4_r(CPUARMState *env, void *vg, target_ulong addr, void *d3 = &env->vfp.zregs[(rd + 2) & 31]; void *d4 = &env->vfp.zregs[(rd + 3) & 31]; - set_helper_retaddr(ra); for (i = 0; i < oprsz; ) { uint16_t pg = *(uint16_t *)(vg + H1_2(i >> 3)); do { @@ -4779,7 +4754,6 @@ static void sve_st4_r(CPUARMState *env, void *vg, target_ulong addr, addr += 4 * msize; } while (i & 15); } - clear_helper_retaddr(); } #define DO_STN_1(N, NAME, ESIZE) \ @@ -4875,7 +4849,6 @@ static void sve_ld1_zs(CPUARMState *env, void *vd, void *vg, void *vm, intptr_t i, oprsz = simd_oprsz(desc); ARMVectorReg scratch = { }; - set_helper_retaddr(ra); for (i = 0; i < oprsz; ) { uint16_t pg = *(uint16_t *)(vg + H1_2(i >> 3)); do { @@ -4886,7 +4859,6 @@ static void sve_ld1_zs(CPUARMState *env, void *vd, void *vg, void *vm, i += 4, pg >>= 4; } while (i & 15); } - clear_helper_retaddr(); /* Wait until all exceptions have been raised to write back. */ memcpy(vd, &scratch, oprsz); @@ -4900,7 +4872,6 @@ static void sve_ld1_zd(CPUARMState *env, void *vd, void *vg, void *vm, intptr_t i, oprsz = simd_oprsz(desc) / 8; ARMVectorReg scratch = { }; - set_helper_retaddr(ra); for (i = 0; i < oprsz; i++) { uint8_t pg = *(uint8_t *)(vg + H1(i)); if (likely(pg & 1)) { @@ -4908,7 +4879,6 @@ static void sve_ld1_zd(CPUARMState *env, void *vd, void *vg, void *vm, tlb_fn(env, &scratch, i * 8, base + (off << scale), ra); } } - clear_helper_retaddr(); /* Wait until all exceptions have been raised to write back. */ memcpy(vd, &scratch, oprsz * 8); @@ -5080,13 +5050,11 @@ static inline void sve_ldff1_zs(CPUARMState *env, void *vd, void *vg, void *vm, reg_off = find_next_active(vg, 0, reg_max, MO_32); if (likely(reg_off < reg_max)) { /* Perform one normal read, which will fault or not. */ - set_helper_retaddr(ra); addr = off_fn(vm, reg_off); addr = base + (addr << scale); tlb_fn(env, vd, reg_off, addr, ra); /* The rest of the reads will be non-faulting. */ - clear_helper_retaddr(); } /* After any fault, zero the leading predicated false elements. */ @@ -5122,13 +5090,11 @@ static inline void sve_ldff1_zd(CPUARMState *env, void *vd, void *vg, void *vm, reg_off = find_next_active(vg, 0, reg_max, MO_64); if (likely(reg_off < reg_max)) { /* Perform one normal read, which will fault or not. */ - set_helper_retaddr(ra); addr = off_fn(vm, reg_off); addr = base + (addr << scale); tlb_fn(env, vd, reg_off, addr, ra); /* The rest of the reads will be non-faulting. */ - clear_helper_retaddr(); } /* After any fault, zero the leading predicated false elements. */ @@ -5240,7 +5206,6 @@ static void sve_st1_zs(CPUARMState *env, void *vd, void *vg, void *vm, const int scale = extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT, 2); intptr_t i, oprsz = simd_oprsz(desc); - set_helper_retaddr(ra); for (i = 0; i < oprsz; ) { uint16_t pg = *(uint16_t *)(vg + H1_2(i >> 3)); do { @@ -5251,7 +5216,6 @@ static void sve_st1_zs(CPUARMState *env, void *vd, void *vg, void *vm, i += 4, pg >>= 4; } while (i & 15); } - clear_helper_retaddr(); } static void sve_st1_zd(CPUARMState *env, void *vd, void *vg, void *vm, @@ -5261,7 +5225,6 @@ static void sve_st1_zd(CPUARMState *env, void *vd, void *vg, void *vm, const int scale = extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT, 2); intptr_t i, oprsz = simd_oprsz(desc) / 8; - set_helper_retaddr(ra); for (i = 0; i < oprsz; i++) { uint8_t pg = *(uint8_t *)(vg + H1(i)); if (likely(pg & 1)) { @@ -5269,7 +5232,6 @@ static void sve_st1_zd(CPUARMState *env, void *vd, void *vg, void *vm, tlb_fn(env, vd, i * 8, base + (off << scale), ra); } } - clear_helper_retaddr(); } #define DO_ST1_ZPZ_S(MEM, OFS) \ From patchwork Wed Apr 22 04:32:59 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 185664 Delivered-To: patch@linaro.org Received: by 2002:a92:3d9a:0:0:0:0:0 with SMTP id k26csp300749ilf; Tue, 21 Apr 2020 21:39:30 -0700 (PDT) X-Google-Smtp-Source: APiQypKpas4th1qtP7IEPmwNon7xKxltnMwcqw7BL9x40ZR8uHcGTgevhMHLj8b0JMB4f8PUz8SI X-Received: by 2002:a0c:8045:: with SMTP id 63mr13389936qva.68.1587530370078; Tue, 21 Apr 2020 21:39:30 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1587530370; cv=none; d=google.com; s=arc-20160816; b=euctUuMPfhVi5/iUp4tXJGHMEDt/O612ylOQnZiK0Nb7SaJDBBmdg0fXKcSRpMCvdd BsxIbM0QeNkvBM8o2Gypj2YDrPyqBqsydd3q+3wafnhdMO+K9MK9XlEOfhP/6vMvUeEw Q+We8Y9VbXgoCYs7/2B1eb/rjHtH7HAA4nun3sVJ50kCgB3UmntSs03l23WrXXDcX7sT 0IQWAYR7/GSr2t+jm3KArzQK+qQkk/E1TZiDJLY4nM8k5bb5TDwPDZ7HMRlGs2m7lm1Y HCAT/ne5/5AuyX1ehUDby1hwWVls/ZFnbNrtWDA2mRewVRx6v+RXeEgIXp9F6UdZaV6p RnxQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:cc:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:content-transfer-encoding :mime-version:references:in-reply-to:message-id:date:subject:to:from :dkim-signature; bh=tL5ieek7M74DoYpRz2/QiRNptpBgZR28dRwW48lo168=; b=t3SMkucAzUcc+f0QQa4wdKlR2Iim20xhbVEeKL0Y+JLp/4IOfBzzrTWZ66WsA4dA8c KXIB+wW+gQ2la/EPSlQKOne1WCAnvLlrZrL4qMstR46JOErwXFRxRuAbN74aLmLG223T 34a9TzFxjrKaHv7tzgjqjYD3M+oTxiEgajguho3m5ZxLRE5EFseHC5BqVAv7M30AssPa LDjAu4ajp7Mpl+ck/t6kZ0rTpjlolQ1N4HUB/+s4Oujg38oU3aIgh9USZ7LEjbwvmRu4 zm4vJfngECKX167AvFajwMOnX40MvyzNmWmwRa9ViYcdVFAygvZ9cUXzdPwgHUa/ctf4 y5Ag== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=Tq6uNrEq; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [209.51.188.17]) by mx.google.com with ESMTPS id r40si2544620qtb.286.2020.04.21.21.39.29 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Tue, 21 Apr 2020 21:39:30 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=Tq6uNrEq; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1]:41434 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jR7AX-0000yM-EX for patch@linaro.org; Wed, 22 Apr 2020 00:39:29 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:50442) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jR74h-0000i8-FU for qemu-devel@nongnu.org; Wed, 22 Apr 2020 00:33:29 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.90_1) (envelope-from ) id 1jR74f-0005Ny-Gq for qemu-devel@nongnu.org; Wed, 22 Apr 2020 00:33:27 -0400 Received: from mail-pj1-x1041.google.com ([2607:f8b0:4864:20::1041]:54595) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1jR74e-0005Gu-Vh for qemu-devel@nongnu.org; Wed, 22 Apr 2020 00:33:25 -0400 Received: by mail-pj1-x1041.google.com with SMTP id y6so345015pjc.4 for ; Tue, 21 Apr 2020 21:33:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=tL5ieek7M74DoYpRz2/QiRNptpBgZR28dRwW48lo168=; b=Tq6uNrEqUUQVb1p2TU7+4TSxDp06OtKaO9BfJB8OcAUX4bPpAXSkj9FV1HA0e628OV BiBXJHpfdv9yLRcFc6kzfdnecGyhwz3QYZ1jHD4FJ4xj+JsGWxRY7vlfvlotZxxGy/Jx MLbwk3MbknNgS/bI9U/jv8sG6GiW01bBivjTT38SWhanghHGctSHCNV8zr0DvkFNxs16 0NnR3ivubE3zVvhG2eNsXrvX2wDE4F/Kz3FaMYk1PfH8HrFAzauG++jZt6LozCm7x+GO 2XiVscBGLtrbuMIPXJS2p6h9mEvJ2EERLd78bgkNQWxw9RTd972WbSj6GGYxindwpkXw 6fyg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=tL5ieek7M74DoYpRz2/QiRNptpBgZR28dRwW48lo168=; b=Jo2OzNR/yh2Te5Vts0UNw2Jr+1KU18AMoYmt+Xl1nm9Hv3nTa3Y4JEm3pl3Eg68D0Y d2kftRXAZKYxc0EDdouNVUDhN/ACx5UVEl0nLIJSxSRpBHmmTo3loZi7Jeml7zW2+WG+ lY0ulhwLxIYM8j8bttuZgJ4CKCC3m0VNAiOnOB0qVNBARKuHu0gMqaAnrezS4UVYeS/v V5MhIYuAmVlgU1hrKrXm06p74m5b49X4QEHhurEwut8DP9gSIGsK8lRkFl0GII6BmwjW eLuj4IYmKMqnkpGIIu46UPqLqOYurKDvocU83VtugM8NoPsrx2yTrJI/wva+1neaMRK5 FQLQ== X-Gm-Message-State: AGi0PuZ87gBYIgOvj0tOlQ3BbTUoouyH6UxwRmP3vw27SxmsulUdiUJf QSkNvapedsaIrrDxE0rjPxId2+astks= X-Received: by 2002:a17:902:6b86:: with SMTP id p6mr24755683plk.150.1587530002691; Tue, 21 Apr 2020 21:33:22 -0700 (PDT) Received: from localhost.localdomain (174-21-149-226.tukw.qwest.net. [174.21.149.226]) by smtp.gmail.com with ESMTPSA id l137sm4129613pfd.107.2020.04.21.21.33.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 21 Apr 2020 21:33:22 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Subject: [PATCH v3 08/18] target/arm: Add sve infrastructure for page lookup Date: Tue, 21 Apr 2020 21:32:59 -0700 Message-Id: <20200422043309.18430-9-richard.henderson@linaro.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200422043309.18430-1-richard.henderson@linaro.org> References: <20200422043309.18430-1-richard.henderson@linaro.org> MIME-Version: 1.0 Received-SPF: pass client-ip=2607:f8b0:4864:20::1041; envelope-from=richard.henderson@linaro.org; helo=mail-pj1-x1041.google.com X-detected-operating-system: by eggs.gnu.org: Error: [-] PROGRAM ABORT : Malformed IPv6 address (bad octet value). Location : parse_addr6(), p0f-client.c:67 X-Received-From: 2607:f8b0:4864:20::1041 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: peter.maydell@linaro.org, qemu-arm@nongnu.org Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: "Qemu-devel" For contiguous predicated memory operations, we want to minimize the number of tlb lookups performed. We have open-coded this for sve_ld1_r, but for correctness with MTE we will need this for all of the memory operations. Create a structure that holds the bounds of active elements, and metadata for two pages. Add routines to find those active elements, lookup the pages, and run watchpoints for those pages. Temporarily mark the functions unused to avoid Werror. Signed-off-by: Richard Henderson --- target/arm/sve_helper.c | 263 +++++++++++++++++++++++++++++++++++++++- 1 file changed, 261 insertions(+), 2 deletions(-) -- 2.20.1 Reviewed-by: Peter Maydell diff --git a/target/arm/sve_helper.c b/target/arm/sve_helper.c index aad2c8c237..2f053a9152 100644 --- a/target/arm/sve_helper.c +++ b/target/arm/sve_helper.c @@ -1630,7 +1630,7 @@ void HELPER(sve_cpy_z_d)(void *vd, void *vg, uint64_t val, uint32_t desc) } } -/* Big-endian hosts need to frob the byte indicies. If the copy +/* Big-endian hosts need to frob the byte indices. If the copy * happens to be 8-byte aligned, then no frobbing necessary. */ static void swap_memmove(void *vd, void *vs, size_t n) @@ -3974,7 +3974,7 @@ void HELPER(sve_fcmla_zpzzz_d)(CPUARMState *env, void *vg, uint32_t desc) /* * Load elements into @vd, controlled by @vg, from @host + @mem_ofs. * Memory is valid through @host + @mem_max. The register element - * indicies are inferred from @mem_ofs, as modified by the types for + * indices are inferred from @mem_ofs, as modified by the types for * which the helper is built. Return the @mem_ofs of the first element * not loaded (which is @mem_max if they are all loaded). * @@ -4133,6 +4133,265 @@ static intptr_t max_for_page(target_ulong base, intptr_t mem_off, return MIN(split, mem_max - mem_off) + mem_off; } +/* + * Resolve the guest virtual address to info->host and info->flags. + * If @nofault, return false if the page is invalid, otherwise + * exit via page fault exception. + */ + +typedef struct { + void *host; + int flags; + MemTxAttrs attrs; +} SVEHostPage; + +static bool sve_probe_page(SVEHostPage *info, bool nofault, + CPUARMState *env, target_ulong addr, + int mem_off, MMUAccessType access_type, + int mmu_idx, uintptr_t retaddr) +{ + int flags; + + addr += mem_off; + flags = probe_access_flags(env, addr, access_type, mmu_idx, nofault, + &info->host, retaddr); + info->flags = flags; + + if (flags & TLB_INVALID_MASK) { + g_assert(nofault); + return false; + } + + /* Ensure that info->host[] is relative to addr, not addr + mem_off. */ + info->host -= mem_off; + +#ifdef CONFIG_USER_ONLY + memset(&info->attrs, 0, sizeof(info->attrs)); +#else + /* + * Find the iotlbentry for addr and return the transaction attributes. + * This *must* be present in the TLB because we just found the mapping. + */ + { + uintptr_t index = tlb_index(env, mmu_idx, addr); + +# ifdef CONFIG_DEBUG_TCG + CPUTLBEntry *entry = tlb_entry(env, mmu_idx, addr); + target_ulong comparator = (access_type == MMU_DATA_LOAD + ? entry->addr_read + : tlb_addr_write(entry)); + g_assert(tlb_hit(comparator, addr)); +# endif + + CPUIOTLBEntry *iotlbentry = &env_tlb(env)->d[mmu_idx].iotlb[index]; + info->attrs = iotlbentry->attrs; + } +#endif + + return true; +} + + +/* + * Analyse contiguous data, protected by a governing predicate. + */ + +typedef enum { + FAULT_NO, + FAULT_FIRST, + FAULT_ALL, +} SVEContFault; + +typedef struct { + /* + * First and last element wholly contained within the two pages. + * mem_off_first[0] and reg_off_first[0] are always set >= 0. + * reg_off_last[0] may be < 0 if the first element crosses pages. + * All of mem_off_first[1], reg_off_first[1] and reg_off_last[1] + * are set >= 0 only if there are complete elements on a second page. + * + * The reg_off_* offsets are relative to the internal vector register. + * The mem_off_first offset is relative to the memory address; the + * two offsets are different when a load operation extends, a store + * operation truncates, or for multi-register operations. + */ + int16_t mem_off_first[2]; + int16_t reg_off_first[2]; + int16_t reg_off_last[2]; + + /* + * One element that is misaligned and spans both pages, + * or -1 if there is no such active element. + */ + int16_t mem_off_split; + int16_t reg_off_split; + + /* + * The byte offset at which the entire operation crosses a page boundary. + * Set >= 0 if and only if the entire operation spans two pages. + */ + int16_t page_split; + + /* TLB data for the two pages. */ + SVEHostPage page[2]; +} SVEContLdSt; + +/* + * Find first active element on each page, and a loose bound for the + * final element on each page. Identify any single element that spans + * the page boundary. Return true if there are any active elements. + */ +static bool __attribute__((unused)) +sve_cont_ldst_elements(SVEContLdSt *info, target_ulong addr, uint64_t *vg, + intptr_t reg_max, int esz, int msize) +{ + const int esize = 1 << esz; + const uint64_t pg_mask = pred_esz_masks[esz]; + intptr_t reg_off_first = -1, reg_off_last = -1, reg_off_split; + intptr_t mem_off_last, mem_off_split; + intptr_t page_split, elt_split; + intptr_t i; + + /* Set all of the element indices to -1, and the TLB data to 0. */ + memset(info, -1, offsetof(SVEContLdSt, page)); + memset(info->page, 0, sizeof(info->page)); + + /* Gross scan over the entire predicate to find bounds. */ + i = 0; + do { + uint64_t pg = vg[i] & pg_mask; + if (pg) { + reg_off_last = i * 64 + 63 - clz64(pg); + if (reg_off_first < 0) { + reg_off_first = i * 64 + ctz64(pg); + } + } + } while (++i * 64 < reg_max); + + if (unlikely(reg_off_first < 0)) { + /* No active elements, no pages touched. */ + return false; + } + tcg_debug_assert(reg_off_last >= 0 && reg_off_last < reg_max); + + info->reg_off_first[0] = reg_off_first; + info->mem_off_first[0] = (reg_off_first >> esz) * msize; + mem_off_last = (reg_off_last >> esz) * msize; + + page_split = -(addr | TARGET_PAGE_MASK); + if (likely(mem_off_last + msize <= page_split)) { + /* The entire operation fits within a single page. */ + info->reg_off_last[0] = reg_off_last; + return true; + } + + info->page_split = page_split; + elt_split = page_split / msize; + reg_off_split = elt_split << esz; + mem_off_split = elt_split * msize; + + /* + * This is the last full element on the first page, but it is not + * necessarily active. If there is no full element, i.e. the first + * active element is the one that's split, this value remains -1. + * It is useful as iteration bounds. + */ + if (elt_split != 0) { + info->reg_off_last[0] = reg_off_split - esize; + } + + /* Determine if an unaligned element spans the pages. */ + if (page_split % msize != 0) { + /* It is helpful to know if the split element is active. */ + if ((vg[reg_off_split >> 6] >> (reg_off_split & 63)) & 1) { + info->reg_off_split = reg_off_split; + info->mem_off_split = mem_off_split; + + if (reg_off_split == reg_off_last) { + /* The page crossing element is last. */ + return true; + } + } + reg_off_split += esize; + mem_off_split += msize; + } + + /* + * We do want the first active element on the second page, because + * this may affect the address reported in an exception. + */ + reg_off_split = find_next_active(vg, reg_off_split, reg_max, esz); + tcg_debug_assert(reg_off_split <= reg_off_last); + info->reg_off_first[1] = reg_off_split; + info->mem_off_first[1] = (reg_off_split >> esz) * msize; + info->reg_off_last[1] = reg_off_last; + return true; +} + +/* + * Resolve the guest virtual addresses to info->page[]. + * Control the generation of page faults with @fault. Return false if + * there is no work to do, which can only happen with @fault == FAULT_NO. + */ +static bool __attribute__((unused)) +sve_cont_ldst_pages(SVEContLdSt *info, SVEContFault fault, CPUARMState *env, + target_ulong addr, MMUAccessType access_type, + uintptr_t retaddr) +{ + int mmu_idx = cpu_mmu_index(env, false); + int mem_off = info->mem_off_first[0]; + bool nofault = fault == FAULT_NO; + bool have_work = true; + + if (!sve_probe_page(&info->page[0], nofault, env, addr, mem_off, + access_type, mmu_idx, retaddr)) { + /* No work to be done. */ + return false; + } + + if (likely(info->page_split < 0)) { + /* The entire operation was on the one page. */ + return true; + } + + /* + * If the second page is invalid, then we want the fault address to be + * the first byte on that page which is accessed. + */ + if (info->mem_off_split >= 0) { + /* + * There is an element split across the pages. The fault address + * should be the first byte of the second page. + */ + mem_off = info->page_split; + /* + * If the split element is also the first active element + * of the vector, then: For first-fault we should continue + * to generate faults for the second page. For no-fault, + * we have work only if the second page is valid. + */ + if (info->mem_off_first[0] < info->mem_off_split) { + nofault = FAULT_FIRST; + have_work = false; + } + } else { + /* + * There is no element split across the pages. The fault address + * should be the first active element on the second page. + */ + mem_off = info->mem_off_first[1]; + /* + * There must have been one active element on the first page, + * so we're out of first-fault territory. + */ + nofault = fault != FAULT_ALL; + } + + have_work |= sve_probe_page(&info->page[1], nofault, env, addr, mem_off, + access_type, mmu_idx, retaddr); + return have_work; +} + /* * The result of tlb_vaddr_to_host for user-only is just g2h(x), * which is always non-null. Elide the useless test. From patchwork Wed Apr 22 04:33:00 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 185667 Delivered-To: patch@linaro.org Received: by 2002:a92:3d9a:0:0:0:0:0 with SMTP id k26csp301758ilf; Tue, 21 Apr 2020 21:41:14 -0700 (PDT) X-Google-Smtp-Source: APiQypJ7JT+b+YJ/nOPmuMz3gMLsjNASeTL0LCs6EKYnOs80iVP9o6OyWQ25W1MsXJGXO42WmpRM X-Received: by 2002:ad4:4d50:: with SMTP id m16mr21959295qvm.222.1587530474331; Tue, 21 Apr 2020 21:41:14 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1587530474; cv=none; d=google.com; s=arc-20160816; b=W06E6G+NFWnVgSldBCDRjxRG5j1rUSqRVE/s4RxiBAm1HjZjqS8B5nd18BOe/MeeWG VUb7xiWaclV0R3Lf8OO8VL/WhYBeIgIwPNQ8BmaCTN2JIij5taTxTOK91KS4ftYFA0W5 dL88P02h2x8hDmSSvVaFNzRGM7Q3Plckkmj6F4eb2iJlnKnhKZkJo5uGTg8y8/SPXD8E nZqXwsoDoeuaPltvoZlV07r6ooetksCuUTEzIduy/XYS0mhRXyfzUanaiBcgPYZqZOWb StlfDzxf5/j8+Zi7d0UNPM1k7l6dp85UWc/VoXkNaGQZkTrQkOy/Nikq/CwAFtLmWrjO D0Sw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:cc:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:content-transfer-encoding :mime-version:references:in-reply-to:message-id:date:subject:to:from :dkim-signature; bh=suwrl9RFQOuVoDr4Q1KFh+iDEpTs3nZzUnaA8dPRNoc=; b=FMms8K4vhcSjFtVoHZcO+9Wnrw2CToyGjnRBOdLYJE3HII8oute6KWV3fa6GxOu2/i Rk+o+oEglz1UO9OJ5xBSo7G/uaquu2/+/gqyunFjhCMTpAiqbhvJ2WjTwMtLgnU1crWB n39c+dN4P6VSQiwi87rWidZBP9+X+im5QzdyJc9biLpvf4Xw/CC38UO/yBqBMhQfrWOb 0skWJiypGYZQC4NsjYzQpifpjBUX7xbxQz3spQdLegUFySoL6sWElaQlXj7BJtyQLWd4 Ea+qh+6FCM6Zvc0d+zFSMo+Lsbm86sruCxtK8Z/nnlmAay+XBKohIAMVKCDazG6N0W1D +KxQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=EE8ORcH7; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [209.51.188.17]) by mx.google.com with ESMTPS id h2si2444828qvk.162.2020.04.21.21.41.14 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Tue, 21 Apr 2020 21:41:14 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=EE8ORcH7; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1]:41498 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jR7CD-00055M-Mx for patch@linaro.org; Wed, 22 Apr 2020 00:41:13 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:50502) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jR74i-0000lT-Tz for qemu-devel@nongnu.org; Wed, 22 Apr 2020 00:33:31 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.90_1) (envelope-from ) id 1jR74h-0005TG-GO for qemu-devel@nongnu.org; Wed, 22 Apr 2020 00:33:28 -0400 Received: from mail-pl1-x644.google.com ([2607:f8b0:4864:20::644]:45312) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1jR74g-0005LI-SN for qemu-devel@nongnu.org; Wed, 22 Apr 2020 00:33:26 -0400 Received: by mail-pl1-x644.google.com with SMTP id t4so423816plq.12 for ; Tue, 21 Apr 2020 21:33:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=suwrl9RFQOuVoDr4Q1KFh+iDEpTs3nZzUnaA8dPRNoc=; b=EE8ORcH7c25Y8idjVFFCe+3g3o7NQku+RgtNpE7HrHokOkOD0Ehzy9Lwk6Cxl/AWIi GNRTlQ39PPoU4QMQCdEybfxTNdasF4ei11yii9d/6jlP7VOGHNv5jIs4usEErirJwdB3 Qpcexe14v7YtgFo0Ukyky+I3ryZjODaXHOUr+wpNVYGSioLJ35z3j0HqlIAM07O552A4 BZWNy1PBTqEqRd/lDz640NzV8VWJukewQ/kEKtBrnEu8L8d1rxH0cRdvC+zrJfrFN/WY JRfbOF4DMG2dCzSg+vKbknqZDEyWUt00yC/SBg/BWXzSUqt0/HId1vlDXk5PyJrwTtHf ucDw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=suwrl9RFQOuVoDr4Q1KFh+iDEpTs3nZzUnaA8dPRNoc=; b=sS/fxoJ0491rCvEsEyzVbKKa3dXJF38IVPke8cgpPxAIw4s5uumRVQzQkHc/G01pJk OJYGuwQgOmnm6hZNg/rHwcc+yScFvlMvakDyecvhcgsCq9w3zdRtr+tGcGTyMu9mnhcM QAOYWNhgqFxbCcChO0nDKxLncBg9+EeNhmqSUfnMdWzs4SEN5cnl1T7u0jzgaRYaq3iX 9UkhAl4Tc8bq6NY+J44N9OmBKSid2Ib1YbB8JYDUVeVO1TRzQK2ECr+BX+hexWv8zf1S U3liOwHdOXQfT6WSYN7gLdFtdK0NXnfSUqFdqnvnF7LRUNo61+Z4TY9OHyDnsZGoeA5/ 9fcg== X-Gm-Message-State: AGi0PubVjEGqKtm30r0DhrRjZ5+LBk7f1LWiFqvDSk3nGqvd0Q5gom5m +EtIB9KFoloHXYRmMpKEvn75jiFn/wE= X-Received: by 2002:a17:90a:2fc8:: with SMTP id n8mr9086745pjm.159.1587530003767; Tue, 21 Apr 2020 21:33:23 -0700 (PDT) Received: from localhost.localdomain (174-21-149-226.tukw.qwest.net. [174.21.149.226]) by smtp.gmail.com with ESMTPSA id l137sm4129613pfd.107.2020.04.21.21.33.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 21 Apr 2020 21:33:23 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Subject: [PATCH v3 09/18] target/arm: Adjust interface of sve_ld1_host_fn Date: Tue, 21 Apr 2020 21:33:00 -0700 Message-Id: <20200422043309.18430-10-richard.henderson@linaro.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200422043309.18430-1-richard.henderson@linaro.org> References: <20200422043309.18430-1-richard.henderson@linaro.org> MIME-Version: 1.0 Received-SPF: pass client-ip=2607:f8b0:4864:20::644; envelope-from=richard.henderson@linaro.org; helo=mail-pl1-x644.google.com X-detected-operating-system: by eggs.gnu.org: Error: [-] PROGRAM ABORT : Malformed IPv6 address (bad octet value). Location : parse_addr6(), p0f-client.c:67 X-Received-From: 2607:f8b0:4864:20::644 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: peter.maydell@linaro.org, qemu-arm@nongnu.org Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: "Qemu-devel" The current interface includes a loop; change it to load a single element. We will then be able to use the function for ld{2,3,4} where individual vector elements are not adjacent. Replace each call with the simplest possible loop over active elements. Reviewed-by: Peter Maydell Signed-off-by: Richard Henderson --- target/arm/sve_helper.c | 124 ++++++++++++++++++++-------------------- 1 file changed, 63 insertions(+), 61 deletions(-) -- 2.20.1 diff --git a/target/arm/sve_helper.c b/target/arm/sve_helper.c index 2f053a9152..d007137735 100644 --- a/target/arm/sve_helper.c +++ b/target/arm/sve_helper.c @@ -3972,20 +3972,10 @@ void HELPER(sve_fcmla_zpzzz_d)(CPUARMState *env, void *vg, uint32_t desc) */ /* - * Load elements into @vd, controlled by @vg, from @host + @mem_ofs. - * Memory is valid through @host + @mem_max. The register element - * indices are inferred from @mem_ofs, as modified by the types for - * which the helper is built. Return the @mem_ofs of the first element - * not loaded (which is @mem_max if they are all loaded). - * - * For softmmu, we have fully validated the guest page. For user-only, - * we cannot fully validate without taking the mmap lock, but since we - * know the access is within one host page, if any access is valid they - * all must be valid. However, when @vg is all false, it may be that - * no access is valid. + * Load one element into @vd + @reg_off from @host. + * The controlling predicate is known to be true. */ -typedef intptr_t sve_ld1_host_fn(void *vd, void *vg, void *host, - intptr_t mem_ofs, intptr_t mem_max); +typedef void sve_ldst1_host_fn(void *vd, intptr_t reg_off, void *host); /* * Load one element into @vd + @reg_off from (@env, @vaddr, @ra). @@ -3999,20 +3989,10 @@ typedef void sve_ldst1_tlb_fn(CPUARMState *env, void *vd, intptr_t reg_off, */ #define DO_LD_HOST(NAME, H, TYPEE, TYPEM, HOST) \ -static intptr_t sve_##NAME##_host(void *vd, void *vg, void *host, \ - intptr_t mem_off, const intptr_t mem_max) \ -{ \ - intptr_t reg_off = mem_off * (sizeof(TYPEE) / sizeof(TYPEM)); \ - uint64_t *pg = vg; \ - while (mem_off + sizeof(TYPEM) <= mem_max) { \ - TYPEM val = 0; \ - if (likely((pg[reg_off >> 6] >> (reg_off & 63)) & 1)) { \ - val = HOST(host + mem_off); \ - } \ - *(TYPEE *)(vd + H(reg_off)) = val; \ - mem_off += sizeof(TYPEM), reg_off += sizeof(TYPEE); \ - } \ - return mem_off; \ +static void sve_##NAME##_host(void *vd, intptr_t reg_off, void *host) \ +{ \ + TYPEM val = HOST(host); \ + *(TYPEE *)(vd + H(reg_off)) = val; \ } #define DO_LD_TLB(NAME, H, TYPEE, TYPEM, TLB) \ @@ -4411,7 +4391,7 @@ static inline bool test_host_page(void *host) static void sve_ld1_r(CPUARMState *env, void *vg, const target_ulong addr, uint32_t desc, const uintptr_t retaddr, const int esz, const int msz, - sve_ld1_host_fn *host_fn, + sve_ldst1_host_fn *host_fn, sve_ldst1_tlb_fn *tlb_fn) { const TCGMemOpIdx oi = extract32(desc, SIMD_DATA_SHIFT, MEMOPIDX_SHIFT); @@ -4445,8 +4425,12 @@ static void sve_ld1_r(CPUARMState *env, void *vg, const target_ulong addr, if (likely(split == mem_max)) { host = tlb_vaddr_to_host(env, addr + mem_off, MMU_DATA_LOAD, mmu_idx); if (test_host_page(host)) { - mem_off = host_fn(vd, vg, host - mem_off, mem_off, mem_max); - tcg_debug_assert(mem_off == mem_max); + intptr_t i = reg_off; + host -= mem_off; + do { + host_fn(vd, i, host + (i >> diffsz)); + i = find_next_active(vg, i + (1 << esz), reg_max, esz); + } while (i < reg_max); /* After having taken any fault, zero leading inactive elements. */ swap_memzero(vd, reg_off); return; @@ -4459,7 +4443,12 @@ static void sve_ld1_r(CPUARMState *env, void *vg, const target_ulong addr, */ #ifdef CONFIG_USER_ONLY swap_memzero(&scratch, reg_off); - host_fn(&scratch, vg, g2h(addr), mem_off, mem_max); + host = g2h(addr); + do { + host_fn(&scratch, reg_off, host + (reg_off >> diffsz)); + reg_off += 1 << esz; + reg_off = find_next_active(vg, reg_off, reg_max, esz); + } while (reg_off < reg_max); #else memset(&scratch, 0, reg_max); goto start; @@ -4477,9 +4466,13 @@ static void sve_ld1_r(CPUARMState *env, void *vg, const target_ulong addr, host = tlb_vaddr_to_host(env, addr + mem_off, MMU_DATA_LOAD, mmu_idx); if (host) { - mem_off = host_fn(&scratch, vg, host - mem_off, - mem_off, split); - reg_off = mem_off << diffsz; + host -= mem_off; + do { + host_fn(&scratch, reg_off, host + mem_off); + reg_off += 1 << esz; + reg_off = find_next_active(vg, reg_off, reg_max, esz); + mem_off = reg_off >> diffsz; + } while (split - mem_off >= (1 << msz)); continue; } } @@ -4706,7 +4699,7 @@ static void record_fault(CPUARMState *env, uintptr_t i, uintptr_t oprsz) static void sve_ldff1_r(CPUARMState *env, void *vg, const target_ulong addr, uint32_t desc, const uintptr_t retaddr, const int esz, const int msz, - sve_ld1_host_fn *host_fn, + sve_ldst1_host_fn *host_fn, sve_ldst1_tlb_fn *tlb_fn) { const TCGMemOpIdx oi = extract32(desc, SIMD_DATA_SHIFT, MEMOPIDX_SHIFT); @@ -4716,7 +4709,7 @@ static void sve_ldff1_r(CPUARMState *env, void *vg, const target_ulong addr, const int diffsz = esz - msz; const intptr_t reg_max = simd_oprsz(desc); const intptr_t mem_max = reg_max >> diffsz; - intptr_t split, reg_off, mem_off; + intptr_t split, reg_off, mem_off, i; void *host; /* Skip to the first active element. */ @@ -4739,28 +4732,18 @@ static void sve_ldff1_r(CPUARMState *env, void *vg, const target_ulong addr, if (likely(split == mem_max)) { host = tlb_vaddr_to_host(env, addr + mem_off, MMU_DATA_LOAD, mmu_idx); if (test_host_page(host)) { - mem_off = host_fn(vd, vg, host - mem_off, mem_off, mem_max); - tcg_debug_assert(mem_off == mem_max); + i = reg_off; + host -= mem_off; + do { + host_fn(vd, i, host + (i >> diffsz)); + i = find_next_active(vg, i + (1 << esz), reg_max, esz); + } while (i < reg_max); /* After any fault, zero any leading inactive elements. */ swap_memzero(vd, reg_off); return; } } -#ifdef CONFIG_USER_ONLY - /* - * The page(s) containing this first element at ADDR+MEM_OFF must - * be valid. Considering that this first element may be misaligned - * and cross a page boundary itself, take the rest of the page from - * the last byte of the element. - */ - split = max_for_page(addr, mem_off + (1 << msz) - 1, mem_max); - mem_off = host_fn(vd, vg, g2h(addr), mem_off, split); - - /* After any fault, zero any leading inactive elements. */ - swap_memzero(vd, reg_off); - reg_off = mem_off << diffsz; -#else /* * Perform one normal read, which will fault or not. * But it is likely to bring the page into the tlb. @@ -4777,11 +4760,15 @@ static void sve_ldff1_r(CPUARMState *env, void *vg, const target_ulong addr, if (split >= (1 << msz)) { host = tlb_vaddr_to_host(env, addr + mem_off, MMU_DATA_LOAD, mmu_idx); if (host) { - mem_off = host_fn(vd, vg, host - mem_off, mem_off, split); - reg_off = mem_off << diffsz; + host -= mem_off; + do { + host_fn(vd, reg_off, host + mem_off); + reg_off += 1 << esz; + reg_off = find_next_active(vg, reg_off, reg_max, esz); + mem_off = reg_off >> diffsz; + } while (split - mem_off >= (1 << msz)); } } -#endif record_fault(env, reg_off, reg_max); } @@ -4791,7 +4778,7 @@ static void sve_ldff1_r(CPUARMState *env, void *vg, const target_ulong addr, */ static void sve_ldnf1_r(CPUARMState *env, void *vg, const target_ulong addr, uint32_t desc, const int esz, const int msz, - sve_ld1_host_fn *host_fn) + sve_ldst1_host_fn *host_fn) { const unsigned rd = extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT, 5); void *vd = &env->vfp.zregs[rd]; @@ -4806,7 +4793,13 @@ static void sve_ldnf1_r(CPUARMState *env, void *vg, const target_ulong addr, host = tlb_vaddr_to_host(env, addr, MMU_DATA_LOAD, mmu_idx); if (likely(page_check_range(addr, mem_max, PAGE_READ) == 0)) { /* The entire operation is valid and will not fault. */ - host_fn(vd, vg, host, 0, mem_max); + reg_off = 0; + do { + mem_off = reg_off >> diffsz; + host_fn(vd, reg_off, host + mem_off); + reg_off += 1 << esz; + reg_off = find_next_active(vg, reg_off, reg_max, esz); + } while (reg_off < reg_max); return; } #endif @@ -4826,8 +4819,12 @@ static void sve_ldnf1_r(CPUARMState *env, void *vg, const target_ulong addr, if (page_check_range(addr + mem_off, 1 << msz, PAGE_READ) == 0) { /* At least one load is valid; take the rest of the page. */ split = max_for_page(addr, mem_off + (1 << msz) - 1, mem_max); - mem_off = host_fn(vd, vg, host, mem_off, split); - reg_off = mem_off << diffsz; + do { + host_fn(vd, reg_off, host + mem_off); + reg_off += 1 << esz; + reg_off = find_next_active(vg, reg_off, reg_max, esz); + mem_off = reg_off >> diffsz; + } while (split - mem_off >= (1 << msz)); } #else /* @@ -4848,8 +4845,13 @@ static void sve_ldnf1_r(CPUARMState *env, void *vg, const target_ulong addr, host = tlb_vaddr_to_host(env, addr + mem_off, MMU_DATA_LOAD, mmu_idx); split = max_for_page(addr, mem_off, mem_max); if (host && split >= (1 << msz)) { - mem_off = host_fn(vd, vg, host - mem_off, mem_off, split); - reg_off = mem_off << diffsz; + host -= mem_off; + do { + host_fn(vd, reg_off, host + mem_off); + reg_off += 1 << esz; + reg_off = find_next_active(vg, reg_off, reg_max, esz); + mem_off = reg_off >> diffsz; + } while (split - mem_off >= (1 << msz)); } #endif From patchwork Wed Apr 22 04:33:01 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 185668 Delivered-To: patch@linaro.org Received: by 2002:a92:3d9a:0:0:0:0:0 with SMTP id k26csp302824ilf; Tue, 21 Apr 2020 21:42:50 -0700 (PDT) X-Google-Smtp-Source: APiQypJNdYm4NhEzYzq2HI7b8f4G2EOFA9SMC5N2IdwKn2brfmrptsV35ZwSUTh2b8z9ANNwn+sN X-Received: by 2002:ac8:2f86:: with SMTP id l6mr468523qta.377.1587530570238; Tue, 21 Apr 2020 21:42:50 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1587530570; cv=none; d=google.com; s=arc-20160816; b=SY22PFXgHG3ThXAf3FB+i+tY+yXTvSYRft12W/3K8t8ImIsQArBXH5/ZSknpLzPnih 3Po9+X02xnH4ZjhP2WYyP8uZ+ylIaFJEYxfJMt/B43MnwuzEk/t8KfJpbHKe8nOM/kDk SuqixcsGw3mY04cvRhYmMZ4L6KxUZIkrmtJZKd+VT1al//YQpsHqVPne/m20osKCEMdx Actvp5Mm8x1j7QxAhmZ+fIUSAHyfhbR/zrnXXUMLXFig62B77MHLoKPg0f/h7EXup56y 7W/cqnP/rl2BeUh06gTAJ2ypF2w/+1YrJPbhV26PQz95qDOjhVRmLuYHpqZS6SIO/dhG IIuw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:cc:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:content-transfer-encoding :mime-version:references:in-reply-to:message-id:date:subject:to:from :dkim-signature; bh=7ZJAf0ODZypJUEx/2hOu0Olpz0sJ7Rm6b/UcyBKBG+Y=; b=wfV5C+dNYxt6IP4MlBcOB45eryOjRNKnQvq6d7OHFGAt2YPo0kDKDCavFCmRUNX50P Pf3UksuCq9Vum/gP+1o5rwFy5+mMmyPZzUac/jdo/1o9bg8Tfz5uyPNY1zVXqlebuAjg 6CE6fG1461w6Z8AGIqf085CPhgedKrP8wszs7h5tZIJnefIgjWiTpNQDJ8u+Xtn6sL+k Crcmin2S/YFxo/LVxFcDpXT734YsAaorqYt8Mx0MHyFTIRZq9E9tXqCgo1FQgoBeZpkJ y2mQR73lvOwrsFnoPgcIOkcLVELrny4SJgCSLqnGLw0sUKFcGPbjRItHoJSYaZCYhdxl 0Uew== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=lK+6xDsS; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [209.51.188.17]) by mx.google.com with ESMTPS id q13si2560134qta.209.2020.04.21.21.42.50 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Tue, 21 Apr 2020 21:42:50 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=lK+6xDsS; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1]:41550 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jR7Dl-0007zA-Ku for patch@linaro.org; Wed, 22 Apr 2020 00:42:49 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:50516) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jR74j-0000ma-Ex for qemu-devel@nongnu.org; Wed, 22 Apr 2020 00:33:31 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.90_1) (envelope-from ) id 1jR74h-0005US-Lv for qemu-devel@nongnu.org; Wed, 22 Apr 2020 00:33:29 -0400 Received: from mail-pl1-x641.google.com ([2607:f8b0:4864:20::641]:38670) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1jR74g-0005PV-WC for qemu-devel@nongnu.org; Wed, 22 Apr 2020 00:33:27 -0400 Received: by mail-pl1-x641.google.com with SMTP id w3so438374plz.5 for ; Tue, 21 Apr 2020 21:33:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=7ZJAf0ODZypJUEx/2hOu0Olpz0sJ7Rm6b/UcyBKBG+Y=; b=lK+6xDsS9QNiMchdBTw4XEb1+4hXtUPwBgPukHKrkE4+Re3FHUvt9jxCHn+HmpWmmp EIMqiQXHo8sFVCQcYB2iuub6D8Nb4Htb0OHIWXqF5jdIgpHixHBI1Z992x/a8myz475g qDcJIBQDHsYVBXjkBo6Tkt+VMkw5N3kvqXtgovYAkNIVIOcXU+6x6eUyHKFG5YbO3S+b kT36zYM5ph3/nUOjE227/O3zXCMGZNOWImXOEvcGQWHkLu3wLyNDwjkHiCUj4nhHTj0y /mfucr7MthuexsWjRXmthSecmVrjbHRxVtY6epUgaFXlCXQX9aJOx4rrCoxy9H16Y2ff V/2w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=7ZJAf0ODZypJUEx/2hOu0Olpz0sJ7Rm6b/UcyBKBG+Y=; b=Y1CAfLXai4byvBH6+jozS21x6BspWrP2cKBFVnXfgK/h9wOM9wMByhBm3jkgaoyYSi TMXXWoan81PyDW+QNJIksSJVW2303c8Y+a/1xELaNTjNObgZuWCyJHEw9AQnhPtQe50Z cIsxtWLM3t3VxIWi0fPYmlekMUd5PCDYqRnX8LteTSlPhU7/NaEubFc37sH7poIn39BV ffPLOxLV5l3W7UxQlcOQOQd0mwI5JJHlHpttQraO+pp6DQ6Cg1118r1AZaPLT5XjZI6V KVYvl+bbCfnyWEhW8ZFm5a64BPHaE43+wu1dTtWEe3CKb4X97A9bQ9i3p2iOzznbIsbP laFg== X-Gm-Message-State: AGi0PuaH2WejpnlNtCTw4dCQThoUiTdXQuRZTlVpxJw5q2CpWinCWAxC /OfTI6MTcxinMJYC4FurOaQzxHDRrjE= X-Received: by 2002:a17:902:7795:: with SMTP id o21mr3505790pll.202.1587530005129; Tue, 21 Apr 2020 21:33:25 -0700 (PDT) Received: from localhost.localdomain (174-21-149-226.tukw.qwest.net. [174.21.149.226]) by smtp.gmail.com with ESMTPSA id l137sm4129613pfd.107.2020.04.21.21.33.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 21 Apr 2020 21:33:24 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Subject: [PATCH v3 10/18] target/arm: Use SVEContLdSt in sve_ld1_r Date: Tue, 21 Apr 2020 21:33:01 -0700 Message-Id: <20200422043309.18430-11-richard.henderson@linaro.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200422043309.18430-1-richard.henderson@linaro.org> References: <20200422043309.18430-1-richard.henderson@linaro.org> MIME-Version: 1.0 Received-SPF: pass client-ip=2607:f8b0:4864:20::641; envelope-from=richard.henderson@linaro.org; helo=mail-pl1-x641.google.com X-detected-operating-system: by eggs.gnu.org: Error: [-] PROGRAM ABORT : Malformed IPv6 address (bad octet value). Location : parse_addr6(), p0f-client.c:67 X-Received-From: 2607:f8b0:4864:20::641 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: peter.maydell@linaro.org, qemu-arm@nongnu.org Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: "Qemu-devel" First use of the new helper functions, so we can remove the unused markup. No longer need a scratch for user-only, as we completely probe the page set before reading; system mode still requires a scratch for MMIO. Reviewed-by: Peter Maydell Signed-off-by: Richard Henderson --- target/arm/sve_helper.c | 188 +++++++++++++++++++++------------------- 1 file changed, 97 insertions(+), 91 deletions(-) -- 2.20.1 diff --git a/target/arm/sve_helper.c b/target/arm/sve_helper.c index d007137735..6bae342a17 100644 --- a/target/arm/sve_helper.c +++ b/target/arm/sve_helper.c @@ -4221,9 +4221,9 @@ typedef struct { * final element on each page. Identify any single element that spans * the page boundary. Return true if there are any active elements. */ -static bool __attribute__((unused)) -sve_cont_ldst_elements(SVEContLdSt *info, target_ulong addr, uint64_t *vg, - intptr_t reg_max, int esz, int msize) +static bool sve_cont_ldst_elements(SVEContLdSt *info, target_ulong addr, + uint64_t *vg, intptr_t reg_max, + int esz, int msize) { const int esize = 1 << esz; const uint64_t pg_mask = pred_esz_masks[esz]; @@ -4313,10 +4313,9 @@ sve_cont_ldst_elements(SVEContLdSt *info, target_ulong addr, uint64_t *vg, * Control the generation of page faults with @fault. Return false if * there is no work to do, which can only happen with @fault == FAULT_NO. */ -static bool __attribute__((unused)) -sve_cont_ldst_pages(SVEContLdSt *info, SVEContFault fault, CPUARMState *env, - target_ulong addr, MMUAccessType access_type, - uintptr_t retaddr) +static bool sve_cont_ldst_pages(SVEContLdSt *info, SVEContFault fault, + CPUARMState *env, target_ulong addr, + MMUAccessType access_type, uintptr_t retaddr) { int mmu_idx = cpu_mmu_index(env, false); int mem_off = info->mem_off_first[0]; @@ -4388,109 +4387,116 @@ static inline bool test_host_page(void *host) /* * Common helper for all contiguous one-register predicated loads. */ -static void sve_ld1_r(CPUARMState *env, void *vg, const target_ulong addr, - uint32_t desc, const uintptr_t retaddr, - const int esz, const int msz, - sve_ldst1_host_fn *host_fn, - sve_ldst1_tlb_fn *tlb_fn) +static inline QEMU_ALWAYS_INLINE +void sve_ld1_r(CPUARMState *env, uint64_t *vg, const target_ulong addr, + uint32_t desc, const uintptr_t retaddr, + const int esz, const int msz, + sve_ldst1_host_fn *host_fn, + sve_ldst1_tlb_fn *tlb_fn) { - const TCGMemOpIdx oi = extract32(desc, SIMD_DATA_SHIFT, MEMOPIDX_SHIFT); - const int mmu_idx = get_mmuidx(oi); const unsigned rd = extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT, 5); void *vd = &env->vfp.zregs[rd]; - const int diffsz = esz - msz; const intptr_t reg_max = simd_oprsz(desc); - const intptr_t mem_max = reg_max >> diffsz; - ARMVectorReg scratch; + intptr_t reg_off, reg_last, mem_off; + SVEContLdSt info; void *host; - intptr_t split, reg_off, mem_off; + int flags; - /* Find the first active element. */ - reg_off = find_next_active(vg, 0, reg_max, esz); - if (unlikely(reg_off == reg_max)) { + /* Find the active elements. */ + if (!sve_cont_ldst_elements(&info, addr, vg, reg_max, esz, 1 << msz)) { /* The entire predicate was false; no load occurs. */ memset(vd, 0, reg_max); return; } - mem_off = reg_off >> diffsz; - /* - * If the (remaining) load is entirely within a single page, then: - * For softmmu, and the tlb hits, then no faults will occur; - * For user-only, either the first load will fault or none will. - * We can thus perform the load directly to the destination and - * Vd will be unmodified on any exception path. - */ - split = max_for_page(addr, mem_off, mem_max); - if (likely(split == mem_max)) { - host = tlb_vaddr_to_host(env, addr + mem_off, MMU_DATA_LOAD, mmu_idx); - if (test_host_page(host)) { - intptr_t i = reg_off; - host -= mem_off; - do { - host_fn(vd, i, host + (i >> diffsz)); - i = find_next_active(vg, i + (1 << esz), reg_max, esz); - } while (i < reg_max); - /* After having taken any fault, zero leading inactive elements. */ - swap_memzero(vd, reg_off); - return; - } - } + /* Probe the page(s). Exit with exception for any invalid page. */ + sve_cont_ldst_pages(&info, FAULT_ALL, env, addr, MMU_DATA_LOAD, retaddr); - /* - * Perform the predicated read into a temporary, thus ensuring - * if the load of the last element faults, Vd is not modified. - */ + flags = info.page[0].flags | info.page[1].flags; + if (unlikely(flags != 0)) { #ifdef CONFIG_USER_ONLY - swap_memzero(&scratch, reg_off); - host = g2h(addr); - do { - host_fn(&scratch, reg_off, host + (reg_off >> diffsz)); - reg_off += 1 << esz; - reg_off = find_next_active(vg, reg_off, reg_max, esz); - } while (reg_off < reg_max); + g_assert_not_reached(); #else - memset(&scratch, 0, reg_max); - goto start; - while (1) { - reg_off = find_next_active(vg, reg_off, reg_max, esz); - if (reg_off >= reg_max) { - break; - } - mem_off = reg_off >> diffsz; - split = max_for_page(addr, mem_off, mem_max); + /* + * At least one page includes MMIO (or watchpoints). + * Any bus operation can fail with cpu_transaction_failed, + * which for ARM will raise SyncExternal. Perform the load + * into scratch memory to preserve register state until the end. + */ + ARMVectorReg scratch; - start: - if (split - mem_off >= (1 << msz)) { - /* At least one whole element on this page. */ - host = tlb_vaddr_to_host(env, addr + mem_off, - MMU_DATA_LOAD, mmu_idx); - if (host) { - host -= mem_off; - do { - host_fn(&scratch, reg_off, host + mem_off); - reg_off += 1 << esz; - reg_off = find_next_active(vg, reg_off, reg_max, esz); - mem_off = reg_off >> diffsz; - } while (split - mem_off >= (1 << msz)); - continue; + memset(&scratch, 0, reg_max); + mem_off = info.mem_off_first[0]; + reg_off = info.reg_off_first[0]; + reg_last = info.reg_off_last[1]; + if (reg_last < 0) { + reg_last = info.reg_off_split; + if (reg_last < 0) { + reg_last = info.reg_off_last[0]; } } - /* - * Perform one normal read. This may fault, longjmping out to the - * main loop in order to raise an exception. It may succeed, and - * as a side-effect load the TLB entry for the next round. Finally, - * in the extremely unlikely case we're performing this operation - * on I/O memory, it may succeed but not bring in the TLB entry. - * But even then we have still made forward progress. - */ - tlb_fn(env, &scratch, reg_off, addr + mem_off, retaddr); - reg_off += 1 << esz; - } -#endif + do { + uint64_t pg = vg[reg_off >> 6]; + do { + if ((pg >> (reg_off & 63)) & 1) { + tlb_fn(env, &scratch, reg_off, addr + mem_off, retaddr); + } + reg_off += 1 << esz; + mem_off += 1 << msz; + } while (reg_off & 63); + } while (reg_off <= reg_last); - memcpy(vd, &scratch, reg_max); + memcpy(vd, &scratch, reg_max); + return; +#endif + } + + /* The entire operation is in RAM, on valid pages. */ + + memset(vd, 0, reg_max); + mem_off = info.mem_off_first[0]; + reg_off = info.reg_off_first[0]; + reg_last = info.reg_off_last[0]; + host = info.page[0].host; + + while (reg_off <= reg_last) { + uint64_t pg = vg[reg_off >> 6]; + do { + if ((pg >> (reg_off & 63)) & 1) { + host_fn(vd, reg_off, host + mem_off); + } + reg_off += 1 << esz; + mem_off += 1 << msz; + } while (reg_off <= reg_last && (reg_off & 63)); + } + + /* + * Use the slow path to manage the cross-page misalignment. + * But we know this is RAM and cannot trap. + */ + mem_off = info.mem_off_split; + if (unlikely(mem_off >= 0)) { + tlb_fn(env, vd, info.reg_off_split, addr + mem_off, retaddr); + } + + mem_off = info.mem_off_first[1]; + if (unlikely(mem_off >= 0)) { + reg_off = info.reg_off_first[1]; + reg_last = info.reg_off_last[1]; + host = info.page[1].host; + + do { + uint64_t pg = vg[reg_off >> 6]; + do { + if ((pg >> (reg_off & 63)) & 1) { + host_fn(vd, reg_off, host + mem_off); + } + reg_off += 1 << esz; + mem_off += 1 << msz; + } while (reg_off & 63); + } while (reg_off <= reg_last); + } } #define DO_LD1_1(NAME, ESZ) \ From patchwork Wed Apr 22 04:33:02 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 185670 Delivered-To: patch@linaro.org Received: by 2002:a92:3d9a:0:0:0:0:0 with SMTP id k26csp304165ilf; Tue, 21 Apr 2020 21:45:13 -0700 (PDT) X-Google-Smtp-Source: APiQypIdyJNzBRfQDmjkF9U1oiQc0Z6oEBx/kEOJRlbgTHDh/0jbjhBRmmljbLRze96eVzkqrIGq X-Received: by 2002:ac8:31ac:: with SMTP id h41mr24484812qte.139.1587530713628; Tue, 21 Apr 2020 21:45:13 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1587530713; cv=none; d=google.com; s=arc-20160816; b=ZbQtm88OXe6j7XFbcRsmh+JF7OI6Ef41HGwd5vX2oPKDhvcaOElBh4VSYQgzV1QGGP psEbFEBIrwuvIaZ7OD7MJYnpbGvNV71Ig2UqKJT91d/XUGwfccJ1lSXi6C+n05k/cACk kbIOhmhMJtrsr9PUV0c1kY7kzzwGh4LWQycV0MyO/SCoPrzzqbrxPCawnpmtt/gyHUcz iNOLoT4jjGWIqVGBMJPRH5dnwJdeLYPGJzQfNM7HkiM12f8ru3TvIXsMSFsJrYNRq2/1 27m7Kz+IdL8W2tlgywi144ny2djm9YA/YX+s/ieSvpQ4lLEI1vZ3dwTmOU9JqXGabp4+ 3VMQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:cc:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:content-transfer-encoding :mime-version:references:in-reply-to:message-id:date:subject:to:from :dkim-signature; bh=4tSyrjuk3Om51Zcagt3t6yh4FkYKfOOT3yWLv6tJ2Vs=; b=G6xVGL1ZX1HLMsy43Ftvyv1kBNjEC0CP2n6gAXr48myNLF33YVvnWstofVAf5vm3bZ d4/Ch3EVjqUNx2ovPyfahIMLoDH5CePMKSoYLPCG0w/A582oimjYpZFdwGnB+DzjfU7N 3EKdY7okO7fJl/MpoxVKYcR7++9oobku3VG0YDVG3fEdtGo2KaOQ4kYVLaZU4qM+Vs+2 ESc7SRUIRdHgL76qwAPIn+1UPpb67cCpXZimdwRGOIvBTRnt3mZ4KRS/WfX27SATeWue wkoXHWEBH6PvVJscp6rzOovT8f9Hbg3SSyA5Bn0m1O3ygktt8Q9fiLiI0dWcpgCkHqL/ 4H2w== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=tdEcW4t6; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [209.51.188.17]) by mx.google.com with ESMTPS id a22si2782095qtn.144.2020.04.21.21.45.13 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Tue, 21 Apr 2020 21:45:13 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=tdEcW4t6; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1]:41602 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jR7G4-0001eb-Hu for patch@linaro.org; Wed, 22 Apr 2020 00:45:12 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:50554) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jR74k-0000oC-ON for qemu-devel@nongnu.org; Wed, 22 Apr 2020 00:33:33 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.90_1) (envelope-from ) id 1jR74i-0005aI-Pg for qemu-devel@nongnu.org; Wed, 22 Apr 2020 00:33:30 -0400 Received: from mail-pf1-x443.google.com ([2607:f8b0:4864:20::443]:35405) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1jR74i-0005Ux-Bj for qemu-devel@nongnu.org; Wed, 22 Apr 2020 00:33:28 -0400 Received: by mail-pf1-x443.google.com with SMTP id r14so454096pfg.2 for ; Tue, 21 Apr 2020 21:33:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=4tSyrjuk3Om51Zcagt3t6yh4FkYKfOOT3yWLv6tJ2Vs=; b=tdEcW4t6iHlzfHZT7MTd+nowW120rfdwA2OWXW5lqAGzmrZSFDwmt+9n3UmPfoAos4 eqUEb+ZQd+XyrEp7acgd9JM1ZfU10GoY8T9KRHdujmvKCfxVwVHyBKljXimnCsJRCUM+ xCWBT9TDkC47pnwCghsS8QNiqmwZnNzev7VjiPJJ3dHwVw82nOq9FpLDIWkHQpOdIWWu T9upG/pVuPXfen905x9u1FgVNC0eLnPpms3RNdCoRhZApYoaDM0yPI8gkh9BgWNxDf3J rm6iksS38MZLlCGNFKp0trbfMT2g/SYSqIxl9QGoKQxZli1skpOMGuD0jSz/wDOqTp9i TWpA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=4tSyrjuk3Om51Zcagt3t6yh4FkYKfOOT3yWLv6tJ2Vs=; b=MhozP3gsdWlFNehjaVvbDPyeV57xzq30+8YSIT/6zTXWF4ySuYFI3PAPBOu1deqPHZ iUQarw18q+r8M6NbqWRZs0BT+z7pEfRoj5FDnd6omz6t1RG810Dhgrqglt3OB1O9oQHM b+j1OeM0DsgDuPhwr+d1M24p03X+aF5LpOJYfk6gLUWocIfGJyU6Z2WatjZxV2ookXbt mu6Wm2KnsMIBsbgcE1/opQXFBZJIQzDffUU3+/q+Oqjf7MNLn4rBxweOwCCsMRjov7P0 BTHeX7cUUnmDTjNTDouqHOafsypGBnjN7pkRDe7EubcmwDZu8pMMgn18E7gpxnL9fSml W2Lw== X-Gm-Message-State: AGi0PuZgbRhY60NhlJiigkEnpE1E0TNy+v6TeADPL5E8J3ppLy6pffJw SIhuRNT3YK6WcFQIhpXv5LOjtz/Fux8= X-Received: by 2002:aa7:9724:: with SMTP id k4mr10964808pfg.309.1587530006550; Tue, 21 Apr 2020 21:33:26 -0700 (PDT) Received: from localhost.localdomain (174-21-149-226.tukw.qwest.net. [174.21.149.226]) by smtp.gmail.com with ESMTPSA id l137sm4129613pfd.107.2020.04.21.21.33.25 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 21 Apr 2020 21:33:25 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Subject: [PATCH v3 11/18] target/arm: Handle watchpoints in sve_ld1_r Date: Tue, 21 Apr 2020 21:33:02 -0700 Message-Id: <20200422043309.18430-12-richard.henderson@linaro.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200422043309.18430-1-richard.henderson@linaro.org> References: <20200422043309.18430-1-richard.henderson@linaro.org> MIME-Version: 1.0 Received-SPF: pass client-ip=2607:f8b0:4864:20::443; envelope-from=richard.henderson@linaro.org; helo=mail-pf1-x443.google.com X-detected-operating-system: by eggs.gnu.org: Error: [-] PROGRAM ABORT : Malformed IPv6 address (bad octet value). Location : parse_addr6(), p0f-client.c:67 X-Received-From: 2607:f8b0:4864:20::443 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: peter.maydell@linaro.org, qemu-arm@nongnu.org Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: "Qemu-devel" Handle all of the watchpoints for active elements all at once, before we've modified the vector register. This removes the TLB_WATCHPOINT bit from page[].flags, which means that we can use the normal fast path via RAM. Reviewed-by: Peter Maydell Signed-off-by: Richard Henderson --- target/arm/sve_helper.c | 72 ++++++++++++++++++++++++++++++++++++++++- 1 file changed, 71 insertions(+), 1 deletion(-) -- 2.20.1 diff --git a/target/arm/sve_helper.c b/target/arm/sve_helper.c index 6bae342a17..7992a569b0 100644 --- a/target/arm/sve_helper.c +++ b/target/arm/sve_helper.c @@ -4371,6 +4371,70 @@ static bool sve_cont_ldst_pages(SVEContLdSt *info, SVEContFault fault, return have_work; } +static void sve_cont_ldst_watchpoints(SVEContLdSt *info, CPUARMState *env, + uint64_t *vg, target_ulong addr, + int esize, int msize, int wp_access, + uintptr_t retaddr) +{ +#ifndef CONFIG_USER_ONLY + intptr_t mem_off, reg_off, reg_last; + int flags0 = info->page[0].flags; + int flags1 = info->page[1].flags; + + if (likely(!((flags0 | flags1) & TLB_WATCHPOINT))) { + return; + } + + /* Indicate that watchpoints are handled. */ + info->page[0].flags = flags0 & ~TLB_WATCHPOINT; + info->page[1].flags = flags1 & ~TLB_WATCHPOINT; + + if (flags0 & TLB_WATCHPOINT) { + mem_off = info->mem_off_first[0]; + reg_off = info->reg_off_first[0]; + reg_last = info->reg_off_last[0]; + + while (reg_off <= reg_last) { + uint64_t pg = vg[reg_off >> 6]; + do { + if ((pg >> (reg_off & 63)) & 1) { + cpu_check_watchpoint(env_cpu(env), addr + mem_off, + msize, info->page[0].attrs, + wp_access, retaddr); + } + reg_off += esize; + mem_off += msize; + } while (reg_off <= reg_last && (reg_off & 63)); + } + } + + mem_off = info->mem_off_split; + if (mem_off >= 0) { + cpu_check_watchpoint(env_cpu(env), addr + mem_off, msize, + info->page[0].attrs, wp_access, retaddr); + } + + mem_off = info->mem_off_first[1]; + if ((flags1 & TLB_WATCHPOINT) && mem_off >= 0) { + reg_off = info->reg_off_first[1]; + reg_last = info->reg_off_last[1]; + + do { + uint64_t pg = vg[reg_off >> 6]; + do { + if ((pg >> (reg_off & 63)) & 1) { + cpu_check_watchpoint(env_cpu(env), addr + mem_off, + msize, info->page[1].attrs, + wp_access, retaddr); + } + reg_off += esize; + mem_off += msize; + } while (reg_off & 63); + } while (reg_off <= reg_last); + } +#endif +} + /* * The result of tlb_vaddr_to_host for user-only is just g2h(x), * which is always non-null. Elide the useless test. @@ -4412,13 +4476,19 @@ void sve_ld1_r(CPUARMState *env, uint64_t *vg, const target_ulong addr, /* Probe the page(s). Exit with exception for any invalid page. */ sve_cont_ldst_pages(&info, FAULT_ALL, env, addr, MMU_DATA_LOAD, retaddr); + /* Handle watchpoints for all active elements. */ + sve_cont_ldst_watchpoints(&info, env, vg, addr, 1 << esz, 1 << msz, + BP_MEM_READ, retaddr); + + /* TODO: MTE check. */ + flags = info.page[0].flags | info.page[1].flags; if (unlikely(flags != 0)) { #ifdef CONFIG_USER_ONLY g_assert_not_reached(); #else /* - * At least one page includes MMIO (or watchpoints). + * At least one page includes MMIO. * Any bus operation can fail with cpu_transaction_failed, * which for ARM will raise SyncExternal. Perform the load * into scratch memory to preserve register state until the end. From patchwork Wed Apr 22 04:33:03 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 185669 Delivered-To: patch@linaro.org Received: by 2002:a92:3d9a:0:0:0:0:0 with SMTP id k26csp302915ilf; Tue, 21 Apr 2020 21:42:59 -0700 (PDT) X-Google-Smtp-Source: APiQypIv5AVJJUTB9OjsP9mIWbGa/19M1IJEgYw7qX2UebIpFHM74yU5uyi/CdGnFN/sShphuMNg X-Received: by 2002:ac8:162f:: with SMTP id p44mr23839769qtj.75.1587530579076; Tue, 21 Apr 2020 21:42:59 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1587530579; cv=none; d=google.com; s=arc-20160816; b=iBpEoby8RMh5hCByNrYHR1tLhAd8o+QjUISE4uc7UAFDWAh0ZC+0oJvMswv+Ppmj/Z 8UDzJ/9AFyWxcQAmQPn2sPz4L6lmLigYPXAIAwhMWh99CGkzQWfeFZfHg0AH1SEE1ty5 1ff0ivFYeeVm7MDuM1ChUGKfiYuBdOQ9TJ5UBXpvlBR2ZS5rpZgxmli7r4jJy0QEYn6q LeMgislgxosM8OSY6lTTucJvDC1BO9zLjzyO0Cpqf/g5ds/93Jgf4KAjsQjbkNdRGoJh gnqW9RbSbgQ/leEhC6tbO97MhFYQL0WBmBNQPMthSenwMem4jLXHkwOEbCrUP6irVrH7 hlWg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:cc:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:content-transfer-encoding :mime-version:references:in-reply-to:message-id:date:subject:to:from :dkim-signature; bh=9amWqshE7k/NAq9a276bjEryeqReUJxYJNXk6wnm0Gs=; b=LBY7oB5goIY0GALof6VXvRZJjlX0G70HqeLvLgxV0+wDeCy9qONxbKDzgtUzOf/0yV JMyLGlvC3NNBGGHkhlNAISgPKO2iBq+p/DLU7MH/GwZWyhcrZEcQv5WHw2AAdOrJT6B0 Wy3NYf1BCDIxY81rNJYLn2CCDpqpLMRY0V6+UqBAfS5VLzaSDgMNhTWSvG/oQ1vzi1ZF 2+doQ4nrfr3xIm/HQV5ntxg4G+zFmWsBPHyKv6Qi1dYt0qKeFmn4vQXuoQ8uoNYVsean FLxPcfLSRZScpYvBBlzJcm1ICwZJRFpp3MRlUbAtTIHhNsOOUjOneQnAPshFK65KIvYr ljbQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b="L6aI8/Aa"; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [209.51.188.17]) by mx.google.com with ESMTPS id p9si2488986qkh.125.2020.04.21.21.42.58 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Tue, 21 Apr 2020 21:42:59 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b="L6aI8/Aa"; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1]:41560 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jR7Du-0008E1-EL for patch@linaro.org; Wed, 22 Apr 2020 00:42:58 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:50674) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jR74n-0000v5-JR for qemu-devel@nongnu.org; Wed, 22 Apr 2020 00:33:38 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.90_1) (envelope-from ) id 1jR74k-0005gj-8o for qemu-devel@nongnu.org; Wed, 22 Apr 2020 00:33:33 -0400 Received: from mail-pl1-x641.google.com ([2607:f8b0:4864:20::641]:44020) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1jR74j-0005cB-PS for qemu-devel@nongnu.org; Wed, 22 Apr 2020 00:33:29 -0400 Received: by mail-pl1-x641.google.com with SMTP id z6so427160plk.10 for ; Tue, 21 Apr 2020 21:33:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=9amWqshE7k/NAq9a276bjEryeqReUJxYJNXk6wnm0Gs=; b=L6aI8/AaePKUmLWkH5eJ5ORLDsUfOSxjqUkGmV1d5iecMehEqtqzbzVWEM0INGaFht b+hNdyq5yecu2WYRVpK7WcBSch99brb3memU+hc6J7LEurbG/VG6I8z/A3hyRVEgkR4r zp9d3r0cEwWfJbbtWbzSDFwGqN0piODENwe5pxaaGe3D29SB3BKx+511gZrZ0+PIpF0w J13qS15xS3xwXEJTO35S1I9Q2WLYB99UMnBsOmb4vIWETVUA5UcR6HuEwag4EQ0hs5eA gcJUpjdz4dqukuCameZY8vi8aoGkCu9FNvrGM6skhqkiF6uTREDSxuVAZ5/Zm8jftOBQ VHoA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=9amWqshE7k/NAq9a276bjEryeqReUJxYJNXk6wnm0Gs=; b=o60PsqbSm+6eq0apFFfyUreAaWQtJxmeOPxI1+h0iknhHTAfU4JDo9RwC75jZ8KvpN n0GovQ/Z0SGb036Qhjv64UGHksISlJ8xYDMAa7wMBwYHFmnK3rFFwBt1ZvQK9LeT3LNG t8Pt/+0fqLoLVZq39gxuKWMUcxLnvVdZaNwdqPy0HCNs7rLcm5wtRjk0tEtPiw8ayl6i lM2ZkV5ZYrJmqa1ZExObyOk0gIvymCTIQ6qRzqmsLFSrMZQ9Uc3HkIALp3eRADINcU/O Q03ZbLHHzfv0GVeedJHPng/hObT7kYC43tfYvLGKZGx147hhQwFK1KDkDeHUwC3nF3Ed UUSg== X-Gm-Message-State: AGi0Pubhk+6si54GZTvKHCnPh2SGKSKS1VoSqmqpLhRbu5JXd6Igs1Qh CsxwTFaMcH5mBunVC9hnaMttqrAn/sI= X-Received: by 2002:a17:90a:1da6:: with SMTP id v35mr10044180pjv.44.1587530007896; Tue, 21 Apr 2020 21:33:27 -0700 (PDT) Received: from localhost.localdomain (174-21-149-226.tukw.qwest.net. [174.21.149.226]) by smtp.gmail.com with ESMTPSA id l137sm4129613pfd.107.2020.04.21.21.33.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 21 Apr 2020 21:33:26 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Subject: [PATCH v3 12/18] target/arm: Use SVEContLdSt for multi-register contiguous loads Date: Tue, 21 Apr 2020 21:33:03 -0700 Message-Id: <20200422043309.18430-13-richard.henderson@linaro.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200422043309.18430-1-richard.henderson@linaro.org> References: <20200422043309.18430-1-richard.henderson@linaro.org> MIME-Version: 1.0 Received-SPF: pass client-ip=2607:f8b0:4864:20::641; envelope-from=richard.henderson@linaro.org; helo=mail-pl1-x641.google.com X-detected-operating-system: by eggs.gnu.org: Error: [-] PROGRAM ABORT : Malformed IPv6 address (bad octet value). Location : parse_addr6(), p0f-client.c:67 X-Received-From: 2607:f8b0:4864:20::641 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: peter.maydell@linaro.org, qemu-arm@nongnu.org Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: "Qemu-devel" Reviewed-by: Peter Maydell Signed-off-by: Richard Henderson --- target/arm/sve_helper.c | 223 ++++++++++++++-------------------------- 1 file changed, 79 insertions(+), 144 deletions(-) -- 2.20.1 diff --git a/target/arm/sve_helper.c b/target/arm/sve_helper.c index 7992a569b0..9365e32646 100644 --- a/target/arm/sve_helper.c +++ b/target/arm/sve_helper.c @@ -4449,27 +4449,28 @@ static inline bool test_host_page(void *host) } /* - * Common helper for all contiguous one-register predicated loads. + * Common helper for all contiguous 1,2,3,4-register predicated stores. */ static inline QEMU_ALWAYS_INLINE -void sve_ld1_r(CPUARMState *env, uint64_t *vg, const target_ulong addr, +void sve_ldN_r(CPUARMState *env, uint64_t *vg, const target_ulong addr, uint32_t desc, const uintptr_t retaddr, - const int esz, const int msz, + const int esz, const int msz, const int N, sve_ldst1_host_fn *host_fn, sve_ldst1_tlb_fn *tlb_fn) { const unsigned rd = extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT, 5); - void *vd = &env->vfp.zregs[rd]; const intptr_t reg_max = simd_oprsz(desc); intptr_t reg_off, reg_last, mem_off; SVEContLdSt info; void *host; - int flags; + int flags, i; /* Find the active elements. */ - if (!sve_cont_ldst_elements(&info, addr, vg, reg_max, esz, 1 << msz)) { + if (!sve_cont_ldst_elements(&info, addr, vg, reg_max, esz, N << msz)) { /* The entire predicate was false; no load occurs. */ - memset(vd, 0, reg_max); + for (i = 0; i < N; ++i) { + memset(&env->vfp.zregs[(rd + i) & 31], 0, reg_max); + } return; } @@ -4477,7 +4478,7 @@ void sve_ld1_r(CPUARMState *env, uint64_t *vg, const target_ulong addr, sve_cont_ldst_pages(&info, FAULT_ALL, env, addr, MMU_DATA_LOAD, retaddr); /* Handle watchpoints for all active elements. */ - sve_cont_ldst_watchpoints(&info, env, vg, addr, 1 << esz, 1 << msz, + sve_cont_ldst_watchpoints(&info, env, vg, addr, 1 << esz, N << msz, BP_MEM_READ, retaddr); /* TODO: MTE check. */ @@ -4493,9 +4494,8 @@ void sve_ld1_r(CPUARMState *env, uint64_t *vg, const target_ulong addr, * which for ARM will raise SyncExternal. Perform the load * into scratch memory to preserve register state until the end. */ - ARMVectorReg scratch; + ARMVectorReg scratch[4] = { }; - memset(&scratch, 0, reg_max); mem_off = info.mem_off_first[0]; reg_off = info.reg_off_first[0]; reg_last = info.reg_off_last[1]; @@ -4510,21 +4510,29 @@ void sve_ld1_r(CPUARMState *env, uint64_t *vg, const target_ulong addr, uint64_t pg = vg[reg_off >> 6]; do { if ((pg >> (reg_off & 63)) & 1) { - tlb_fn(env, &scratch, reg_off, addr + mem_off, retaddr); + for (i = 0; i < N; ++i) { + tlb_fn(env, &scratch[i], reg_off, + addr + mem_off + (i << msz), retaddr); + } } reg_off += 1 << esz; - mem_off += 1 << msz; + mem_off += N << msz; } while (reg_off & 63); } while (reg_off <= reg_last); - memcpy(vd, &scratch, reg_max); + for (i = 0; i < N; ++i) { + memcpy(&env->vfp.zregs[(rd + i) & 31], &scratch[i], reg_max); + } return; #endif } /* The entire operation is in RAM, on valid pages. */ - memset(vd, 0, reg_max); + for (i = 0; i < N; ++i) { + memset(&env->vfp.zregs[(rd + i) & 31], 0, reg_max); + } + mem_off = info.mem_off_first[0]; reg_off = info.reg_off_first[0]; reg_last = info.reg_off_last[0]; @@ -4534,10 +4542,13 @@ void sve_ld1_r(CPUARMState *env, uint64_t *vg, const target_ulong addr, uint64_t pg = vg[reg_off >> 6]; do { if ((pg >> (reg_off & 63)) & 1) { - host_fn(vd, reg_off, host + mem_off); + for (i = 0; i < N; ++i) { + host_fn(&env->vfp.zregs[(rd + i) & 31], reg_off, + host + mem_off + (i << msz)); + } } reg_off += 1 << esz; - mem_off += 1 << msz; + mem_off += N << msz; } while (reg_off <= reg_last && (reg_off & 63)); } @@ -4547,7 +4558,11 @@ void sve_ld1_r(CPUARMState *env, uint64_t *vg, const target_ulong addr, */ mem_off = info.mem_off_split; if (unlikely(mem_off >= 0)) { - tlb_fn(env, vd, info.reg_off_split, addr + mem_off, retaddr); + reg_off = info.reg_off_split; + for (i = 0; i < N; ++i) { + tlb_fn(env, &env->vfp.zregs[(rd + i) & 31], reg_off, + addr + mem_off + (i << msz), retaddr); + } } mem_off = info.mem_off_first[1]; @@ -4560,10 +4575,13 @@ void sve_ld1_r(CPUARMState *env, uint64_t *vg, const target_ulong addr, uint64_t pg = vg[reg_off >> 6]; do { if ((pg >> (reg_off & 63)) & 1) { - host_fn(vd, reg_off, host + mem_off); + for (i = 0; i < N; ++i) { + host_fn(&env->vfp.zregs[(rd + i) & 31], reg_off, + host + mem_off + (i << msz)); + } } reg_off += 1 << esz; - mem_off += 1 << msz; + mem_off += N << msz; } while (reg_off & 63); } while (reg_off <= reg_last); } @@ -4573,7 +4591,7 @@ void sve_ld1_r(CPUARMState *env, uint64_t *vg, const target_ulong addr, void HELPER(sve_##NAME##_r)(CPUARMState *env, void *vg, \ target_ulong addr, uint32_t desc) \ { \ - sve_ld1_r(env, vg, addr, desc, GETPC(), ESZ, 0, \ + sve_ldN_r(env, vg, addr, desc, GETPC(), ESZ, MO_8, 1, \ sve_##NAME##_host, sve_##NAME##_tlb); \ } @@ -4581,159 +4599,76 @@ void HELPER(sve_##NAME##_r)(CPUARMState *env, void *vg, \ void HELPER(sve_##NAME##_le_r)(CPUARMState *env, void *vg, \ target_ulong addr, uint32_t desc) \ { \ - sve_ld1_r(env, vg, addr, desc, GETPC(), ESZ, MSZ, \ + sve_ldN_r(env, vg, addr, desc, GETPC(), ESZ, MSZ, 1, \ sve_##NAME##_le_host, sve_##NAME##_le_tlb); \ } \ void HELPER(sve_##NAME##_be_r)(CPUARMState *env, void *vg, \ target_ulong addr, uint32_t desc) \ { \ - sve_ld1_r(env, vg, addr, desc, GETPC(), ESZ, MSZ, \ + sve_ldN_r(env, vg, addr, desc, GETPC(), ESZ, MSZ, 1, \ sve_##NAME##_be_host, sve_##NAME##_be_tlb); \ } -DO_LD1_1(ld1bb, 0) -DO_LD1_1(ld1bhu, 1) -DO_LD1_1(ld1bhs, 1) -DO_LD1_1(ld1bsu, 2) -DO_LD1_1(ld1bss, 2) -DO_LD1_1(ld1bdu, 3) -DO_LD1_1(ld1bds, 3) +DO_LD1_1(ld1bb, MO_8) +DO_LD1_1(ld1bhu, MO_16) +DO_LD1_1(ld1bhs, MO_16) +DO_LD1_1(ld1bsu, MO_32) +DO_LD1_1(ld1bss, MO_32) +DO_LD1_1(ld1bdu, MO_64) +DO_LD1_1(ld1bds, MO_64) -DO_LD1_2(ld1hh, 1, 1) -DO_LD1_2(ld1hsu, 2, 1) -DO_LD1_2(ld1hss, 2, 1) -DO_LD1_2(ld1hdu, 3, 1) -DO_LD1_2(ld1hds, 3, 1) +DO_LD1_2(ld1hh, MO_16, MO_16) +DO_LD1_2(ld1hsu, MO_32, MO_16) +DO_LD1_2(ld1hss, MO_32, MO_16) +DO_LD1_2(ld1hdu, MO_64, MO_16) +DO_LD1_2(ld1hds, MO_64, MO_16) -DO_LD1_2(ld1ss, 2, 2) -DO_LD1_2(ld1sdu, 3, 2) -DO_LD1_2(ld1sds, 3, 2) +DO_LD1_2(ld1ss, MO_32, MO_32) +DO_LD1_2(ld1sdu, MO_64, MO_32) +DO_LD1_2(ld1sds, MO_64, MO_32) -DO_LD1_2(ld1dd, 3, 3) +DO_LD1_2(ld1dd, MO_64, MO_64) #undef DO_LD1_1 #undef DO_LD1_2 -/* - * Common helpers for all contiguous 2,3,4-register predicated loads. - */ -static void sve_ld2_r(CPUARMState *env, void *vg, target_ulong addr, - uint32_t desc, int size, uintptr_t ra, - sve_ldst1_tlb_fn *tlb_fn) -{ - const unsigned rd = extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT, 5); - intptr_t i, oprsz = simd_oprsz(desc); - ARMVectorReg scratch[2] = { }; - - for (i = 0; i < oprsz; ) { - uint16_t pg = *(uint16_t *)(vg + H1_2(i >> 3)); - do { - if (pg & 1) { - tlb_fn(env, &scratch[0], i, addr, ra); - tlb_fn(env, &scratch[1], i, addr + size, ra); - } - i += size, pg >>= size; - addr += 2 * size; - } while (i & 15); - } - - /* Wait until all exceptions have been raised to write back. */ - memcpy(&env->vfp.zregs[rd], &scratch[0], oprsz); - memcpy(&env->vfp.zregs[(rd + 1) & 31], &scratch[1], oprsz); -} - -static void sve_ld3_r(CPUARMState *env, void *vg, target_ulong addr, - uint32_t desc, int size, uintptr_t ra, - sve_ldst1_tlb_fn *tlb_fn) -{ - const unsigned rd = extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT, 5); - intptr_t i, oprsz = simd_oprsz(desc); - ARMVectorReg scratch[3] = { }; - - for (i = 0; i < oprsz; ) { - uint16_t pg = *(uint16_t *)(vg + H1_2(i >> 3)); - do { - if (pg & 1) { - tlb_fn(env, &scratch[0], i, addr, ra); - tlb_fn(env, &scratch[1], i, addr + size, ra); - tlb_fn(env, &scratch[2], i, addr + 2 * size, ra); - } - i += size, pg >>= size; - addr += 3 * size; - } while (i & 15); - } - - /* Wait until all exceptions have been raised to write back. */ - memcpy(&env->vfp.zregs[rd], &scratch[0], oprsz); - memcpy(&env->vfp.zregs[(rd + 1) & 31], &scratch[1], oprsz); - memcpy(&env->vfp.zregs[(rd + 2) & 31], &scratch[2], oprsz); -} - -static void sve_ld4_r(CPUARMState *env, void *vg, target_ulong addr, - uint32_t desc, int size, uintptr_t ra, - sve_ldst1_tlb_fn *tlb_fn) -{ - const unsigned rd = extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT, 5); - intptr_t i, oprsz = simd_oprsz(desc); - ARMVectorReg scratch[4] = { }; - - for (i = 0; i < oprsz; ) { - uint16_t pg = *(uint16_t *)(vg + H1_2(i >> 3)); - do { - if (pg & 1) { - tlb_fn(env, &scratch[0], i, addr, ra); - tlb_fn(env, &scratch[1], i, addr + size, ra); - tlb_fn(env, &scratch[2], i, addr + 2 * size, ra); - tlb_fn(env, &scratch[3], i, addr + 3 * size, ra); - } - i += size, pg >>= size; - addr += 4 * size; - } while (i & 15); - } - - /* Wait until all exceptions have been raised to write back. */ - memcpy(&env->vfp.zregs[rd], &scratch[0], oprsz); - memcpy(&env->vfp.zregs[(rd + 1) & 31], &scratch[1], oprsz); - memcpy(&env->vfp.zregs[(rd + 2) & 31], &scratch[2], oprsz); - memcpy(&env->vfp.zregs[(rd + 3) & 31], &scratch[3], oprsz); -} - #define DO_LDN_1(N) \ -void QEMU_FLATTEN HELPER(sve_ld##N##bb_r) \ - (CPUARMState *env, void *vg, target_ulong addr, uint32_t desc) \ -{ \ - sve_ld##N##_r(env, vg, addr, desc, 1, GETPC(), sve_ld1bb_tlb); \ +void HELPER(sve_ld##N##bb_r)(CPUARMState *env, void *vg, \ + target_ulong addr, uint32_t desc) \ +{ \ + sve_ldN_r(env, vg, addr, desc, GETPC(), MO_8, MO_8, N, \ + sve_ld1bb_host, sve_ld1bb_tlb); \ } -#define DO_LDN_2(N, SUFF, SIZE) \ -void QEMU_FLATTEN HELPER(sve_ld##N##SUFF##_le_r) \ - (CPUARMState *env, void *vg, target_ulong addr, uint32_t desc) \ +#define DO_LDN_2(N, SUFF, ESZ) \ +void HELPER(sve_ld##N##SUFF##_le_r)(CPUARMState *env, void *vg, \ + target_ulong addr, uint32_t desc) \ { \ - sve_ld##N##_r(env, vg, addr, desc, SIZE, GETPC(), \ - sve_ld1##SUFF##_le_tlb); \ + sve_ldN_r(env, vg, addr, desc, GETPC(), ESZ, ESZ, N, \ + sve_ld1##SUFF##_le_host, sve_ld1##SUFF##_le_tlb); \ } \ -void QEMU_FLATTEN HELPER(sve_ld##N##SUFF##_be_r) \ - (CPUARMState *env, void *vg, target_ulong addr, uint32_t desc) \ +void HELPER(sve_ld##N##SUFF##_be_r)(CPUARMState *env, void *vg, \ + target_ulong addr, uint32_t desc) \ { \ - sve_ld##N##_r(env, vg, addr, desc, SIZE, GETPC(), \ - sve_ld1##SUFF##_be_tlb); \ + sve_ldN_r(env, vg, addr, desc, GETPC(), ESZ, ESZ, N, \ + sve_ld1##SUFF##_be_host, sve_ld1##SUFF##_be_tlb); \ } DO_LDN_1(2) DO_LDN_1(3) DO_LDN_1(4) -DO_LDN_2(2, hh, 2) -DO_LDN_2(3, hh, 2) -DO_LDN_2(4, hh, 2) +DO_LDN_2(2, hh, MO_16) +DO_LDN_2(3, hh, MO_16) +DO_LDN_2(4, hh, MO_16) -DO_LDN_2(2, ss, 4) -DO_LDN_2(3, ss, 4) -DO_LDN_2(4, ss, 4) +DO_LDN_2(2, ss, MO_32) +DO_LDN_2(3, ss, MO_32) +DO_LDN_2(4, ss, MO_32) -DO_LDN_2(2, dd, 8) -DO_LDN_2(3, dd, 8) -DO_LDN_2(4, dd, 8) +DO_LDN_2(2, dd, MO_64) +DO_LDN_2(3, dd, MO_64) +DO_LDN_2(4, dd, MO_64) #undef DO_LDN_1 #undef DO_LDN_2 From patchwork Wed Apr 22 04:33:04 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 185661 Delivered-To: patch@linaro.org Received: by 2002:a92:3d9a:0:0:0:0:0 with SMTP id k26csp299643ilf; Tue, 21 Apr 2020 21:37:40 -0700 (PDT) X-Google-Smtp-Source: APiQypLvcl488McIpC2BOVCcpBqOpeM5BVVGEYlIvZozO21oTmSywsKnwRv2BHhTznjVji5g2pSb X-Received: by 2002:ac8:2a70:: with SMTP id l45mr10531784qtl.232.1587530260390; Tue, 21 Apr 2020 21:37:40 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1587530260; cv=none; d=google.com; s=arc-20160816; b=bmP+9OhZ+aKB0Z0FaX/bNih+03fDdc0WzLawlm306PM/Iei6Y8RhhIngVpQlnZ1MJt F2oyJFt5EZ4Uvhppi0vmw3cby1Vbfa5v+DzZFrbi66UyYsZ9sQoKGN2bt0xypxYiXpzP colT8q5vu5/tZ15p9LAiiBLEN5356Q7OtgT9bH+3AvCdsxQ9CZ0q1FZ5hQIojQ3TGyMR pP/V0TIrOYiynzTPDtUYFTBqhpEZ7KvWJehT21W0AE3vLJJ4jXDHIWXQNeb+eVRpzYYt KEueCycfzNiJBDEanGH15okzbbusyZI23UUtoJvjrURVKp5KdpkdugEYuknPw1mZybjK fytQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:cc:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:content-transfer-encoding :mime-version:references:in-reply-to:message-id:date:subject:to:from :dkim-signature; bh=sb5xzDqSBN+FVoQJfSff78l7QFMsPU1ClJ/gV59ihKk=; b=FBxYK7ZAZYCwKpsKqdt+FhF3OYodEyM21B0CqvHuJO3JDlB6dE0NM4PV4zVYw1zZO3 mUyk2NYu/uQkmIsP/Bi33QDDqVh4PaVgwr/IbK6uKXuQ3y0Q6TZQEn4a9eHOf5O24zcZ K+kTVLz4gifb2ANtNxUpSXLxd130zICLMeNrDEaKaU30V7mGTiUCzeGuxUW4/Ag5G2ta 3EbGjjl95lyHy78/WXEoX2ZdZ8yWn/Hg03ot42WYsRbJkZV/6bTegcZGQTmoHZB0QyBM +QE+454A5s1rhehjmu3UmvryIGUTQmwKOADuq6lKfMfSxod3zXQPPq/6Ua7Mgwi4wiYq HFAQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=wZC1YO1n; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [209.51.188.17]) by mx.google.com with ESMTPS id b16si2086932qkc.371.2020.04.21.21.37.40 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Tue, 21 Apr 2020 21:37:40 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=wZC1YO1n; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1]:41376 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jR78l-0007Iz-MV for patch@linaro.org; Wed, 22 Apr 2020 00:37:39 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:50764) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jR74p-0000zJ-LZ for qemu-devel@nongnu.org; Wed, 22 Apr 2020 00:33:39 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.90_1) (envelope-from ) id 1jR74m-0005tO-44 for qemu-devel@nongnu.org; Wed, 22 Apr 2020 00:33:35 -0400 Received: from mail-pf1-x444.google.com ([2607:f8b0:4864:20::444]:35406) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1jR74l-0005kf-JY for qemu-devel@nongnu.org; Wed, 22 Apr 2020 00:33:31 -0400 Received: by mail-pf1-x444.google.com with SMTP id r14so454135pfg.2 for ; Tue, 21 Apr 2020 21:33:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=sb5xzDqSBN+FVoQJfSff78l7QFMsPU1ClJ/gV59ihKk=; b=wZC1YO1nig7XSf+CfGjuopiUd1nQOJumiEXuUjCkVyi9atCHNN6ABZZ4xKAV5VxIKU 7Qe5akEKrKJiC3VAuI/xElih81+tmqfX9UCAfbi8wEjEpOShXuZ/SrrQglgf9wpm4MtE plrswaYwOjD9OQkyXbnNuw2SRuyVvBHCY0SsF6U0canjJS5L4pUYbDCLfyjBWgN4euko D3BsqzZ9712BKzaRP60lNX/SP76k7WAPUPinRAVXTmlBpILrg2LGapC+4Lz9E880ELM2 9mauf6Ys8VxFh3j83EinxaoCWR1P/UfILXZvuRnk/dZA850d77ems+QOEJbSDwobc1ix ZBMQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=sb5xzDqSBN+FVoQJfSff78l7QFMsPU1ClJ/gV59ihKk=; b=hQcXjcfbT9DzUIVfO36kEzQS3VCO1G5LqD8d/DvN7ekbLzil99e568erU8q6rU9OQz m5i8UVZs5B1RlULenhltbuTOL+MYQqWq6XAJQi6SQ1q5RTTmxAZQ0lxUAA8g3/aL1R1Y E7M89YqcvQiiktZE0BeUfOB9iGLXJVOsv8Jl4Jb4gKpXN6YbHBWKHRQnI5Z5jDoZ0u3+ w6UMx9TlLYcEtDjyRGU0OUlKeZWgAq2yexwu12KTvlse+QpPr5J6+mUQIAJ+VL3vvKZg w3oIKOGh5NlGxSuTcxq3kD85KOfkPdLxEDVg/WY2ODINCnT8MaStoH8JPXmIKd24Jgef ZDOg== X-Gm-Message-State: AGi0PuZTMTcVaDkq6l1BtExBIxz8/F6bIkVeCqQDxr4tsxOiIVGQNUxF QpS4NjlSorzqCvVIjYufChvJqkK0B4E= X-Received: by 2002:a62:18c8:: with SMTP id 191mr25786838pfy.255.1587530009325; Tue, 21 Apr 2020 21:33:29 -0700 (PDT) Received: from localhost.localdomain (174-21-149-226.tukw.qwest.net. [174.21.149.226]) by smtp.gmail.com with ESMTPSA id l137sm4129613pfd.107.2020.04.21.21.33.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 21 Apr 2020 21:33:28 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Subject: [PATCH v3 13/18] target/arm: Update contiguous first-fault and no-fault loads Date: Tue, 21 Apr 2020 21:33:04 -0700 Message-Id: <20200422043309.18430-14-richard.henderson@linaro.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200422043309.18430-1-richard.henderson@linaro.org> References: <20200422043309.18430-1-richard.henderson@linaro.org> MIME-Version: 1.0 Received-SPF: pass client-ip=2607:f8b0:4864:20::444; envelope-from=richard.henderson@linaro.org; helo=mail-pf1-x444.google.com X-detected-operating-system: by eggs.gnu.org: Error: [-] PROGRAM ABORT : Malformed IPv6 address (bad octet value). Location : parse_addr6(), p0f-client.c:67 X-Received-From: 2607:f8b0:4864:20::444 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: peter.maydell@linaro.org, qemu-arm@nongnu.org Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: "Qemu-devel" With sve_cont_ldst_pages, the differences between first-fault and no-fault are minimal, so unify the routines. With cpu_probe_watchpoint, we are able to make progress through pages with TLB_WATCHPOINT set when the watchpoint does not actually fire. Signed-off-by: Richard Henderson --- target/arm/sve_helper.c | 342 +++++++++++++++++++--------------------- 1 file changed, 158 insertions(+), 184 deletions(-) -- 2.20.1 diff --git a/target/arm/sve_helper.c b/target/arm/sve_helper.c index 9365e32646..3473b53928 100644 --- a/target/arm/sve_helper.c +++ b/target/arm/sve_helper.c @@ -4101,18 +4101,6 @@ static intptr_t find_next_active(uint64_t *vg, intptr_t reg_off, return reg_off; } -/* - * Return the maximum offset <= @mem_max which is still within the page - * referenced by @base + @mem_off. - */ -static intptr_t max_for_page(target_ulong base, intptr_t mem_off, - intptr_t mem_max) -{ - target_ulong addr = base + mem_off; - intptr_t split = -(intptr_t)(addr | TARGET_PAGE_MASK); - return MIN(split, mem_max - mem_off) + mem_off; -} - /* * Resolve the guest virtual address to info->host and info->flags. * If @nofault, return false if the page is invalid, otherwise @@ -4435,19 +4423,6 @@ static void sve_cont_ldst_watchpoints(SVEContLdSt *info, CPUARMState *env, #endif } -/* - * The result of tlb_vaddr_to_host for user-only is just g2h(x), - * which is always non-null. Elide the useless test. - */ -static inline bool test_host_page(void *host) -{ -#ifdef CONFIG_USER_ONLY - return true; -#else - return likely(host != NULL); -#endif -} - /* * Common helper for all contiguous 1,2,3,4-register predicated stores. */ @@ -4705,167 +4680,163 @@ static void record_fault(CPUARMState *env, uintptr_t i, uintptr_t oprsz) } /* - * Common helper for all contiguous first-fault loads. + * Common helper for all contiguous no-fault and first-fault loads. */ -static void sve_ldff1_r(CPUARMState *env, void *vg, const target_ulong addr, - uint32_t desc, const uintptr_t retaddr, - const int esz, const int msz, - sve_ldst1_host_fn *host_fn, - sve_ldst1_tlb_fn *tlb_fn) +static inline QEMU_ALWAYS_INLINE +void sve_ldnfff1_r(CPUARMState *env, void *vg, const target_ulong addr, + uint32_t desc, const uintptr_t retaddr, + const int esz, const int msz, const SVEContFault fault, + sve_ldst1_host_fn *host_fn, + sve_ldst1_tlb_fn *tlb_fn) { - const TCGMemOpIdx oi = extract32(desc, SIMD_DATA_SHIFT, MEMOPIDX_SHIFT); - const int mmu_idx = get_mmuidx(oi); const unsigned rd = extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT, 5); void *vd = &env->vfp.zregs[rd]; - const int diffsz = esz - msz; const intptr_t reg_max = simd_oprsz(desc); - const intptr_t mem_max = reg_max >> diffsz; - intptr_t split, reg_off, mem_off, i; + intptr_t reg_off, mem_off, reg_last; + SVEContLdSt info; + int flags; void *host; - /* Skip to the first active element. */ - reg_off = find_next_active(vg, 0, reg_max, esz); - if (unlikely(reg_off == reg_max)) { + /* Find the active elements. */ + if (!sve_cont_ldst_elements(&info, addr, vg, reg_max, esz, 1 << msz)) { /* The entire predicate was false; no load occurs. */ memset(vd, 0, reg_max); return; } - mem_off = reg_off >> diffsz; + reg_off = info.reg_off_first[0]; - /* - * If the (remaining) load is entirely within a single page, then: - * For softmmu, and the tlb hits, then no faults will occur; - * For user-only, either the first load will fault or none will. - * We can thus perform the load directly to the destination and - * Vd will be unmodified on any exception path. - */ - split = max_for_page(addr, mem_off, mem_max); - if (likely(split == mem_max)) { - host = tlb_vaddr_to_host(env, addr + mem_off, MMU_DATA_LOAD, mmu_idx); - if (test_host_page(host)) { - i = reg_off; - host -= mem_off; - do { - host_fn(vd, i, host + (i >> diffsz)); - i = find_next_active(vg, i + (1 << esz), reg_max, esz); - } while (i < reg_max); - /* After any fault, zero any leading inactive elements. */ + /* Probe the page(s). */ + if (!sve_cont_ldst_pages(&info, fault, env, addr, MMU_DATA_LOAD, retaddr)) { + /* Fault on first element. */ + tcg_debug_assert(fault == FAULT_NO); + memset(vd, 0, reg_max); + goto do_fault; + } + + mem_off = info.mem_off_first[0]; + flags = info.page[0].flags; + + if (fault == FAULT_FIRST) { + /* + * Special handling of the first active element, + * if it crosses a page boundary or is MMIO. + */ + bool is_split = mem_off == info.mem_off_split; + /* TODO: MTE check. */ + if (unlikely(flags != 0) || unlikely(is_split)) { + /* + * Use the slow path for cross-page handling. + * Might trap for MMIO or watchpoints. + */ + tlb_fn(env, vd, reg_off, addr + mem_off, retaddr); + + /* After any fault, zero the other elements. */ swap_memzero(vd, reg_off); - return; + reg_off += 1 << esz; + mem_off += 1 << msz; + swap_memzero(vd + reg_off, reg_max - reg_off); + + if (is_split) { + goto second_page; + } + } else { + memset(vd, 0, reg_max); + } + } else { + memset(vd, 0, reg_max); + if (unlikely(mem_off == info.mem_off_split)) { + /* The first active element crosses a page boundary. */ + flags |= info.page[1].flags; + if (unlikely(flags & TLB_MMIO)) { + /* Some page is MMIO, see below. */ + goto do_fault; + } + if (unlikely(flags & TLB_WATCHPOINT) && + (cpu_watchpoint_address_matches + (env_cpu(env), addr + mem_off, 1 << msz) + & BP_MEM_READ)) { + /* Watchpoint hit, see below. */ + goto do_fault; + } + /* TODO: MTE check. */ + /* + * Use the slow path for cross-page handling. + * This is RAM, without a watchpoint, and will not trap. + */ + tlb_fn(env, vd, reg_off, addr + mem_off, retaddr); + goto second_page; } } /* - * Perform one normal read, which will fault or not. - * But it is likely to bring the page into the tlb. + * From this point on, all memory operations are MemSingleNF. + * + * Per the MemSingleNF pseudocode, a no-fault load from Device memory + * must not actually hit the bus -- it returns (UNKNOWN, FAULT) instead. + * If you map non-RAM with Normal memory attributes and do a NF + * load then it should access the bus -- but doing so is illegal. + * + * While we do not have access to the memory attributes from the PTE + * to tell Device memory from Normal memory, we can validly assume that + * non-RAM has been mapped as Device memory. Thus we indicate fault + * on all MMIO. + * + * Similarly, CPU_BP breakpoints would raise exceptions, and so + * return (UNKNOWN, FAULT). For simplicity, we consider gdb and + * architectural breakpoints the same. */ - tlb_fn(env, vd, reg_off, addr + mem_off, retaddr); + if (unlikely(flags & TLB_MMIO)) { + goto do_fault; + } - /* After any fault, zero any leading predicated false elts. */ - swap_memzero(vd, reg_off); - mem_off += 1 << msz; - reg_off += 1 << esz; + reg_last = info.reg_off_last[0]; + host = info.page[0].host; - /* Try again to read the balance of the page. */ - split = max_for_page(addr, mem_off - 1, mem_max); - if (split >= (1 << msz)) { - host = tlb_vaddr_to_host(env, addr + mem_off, MMU_DATA_LOAD, mmu_idx); - if (host) { - host -= mem_off; - do { + do { + uint64_t pg = *(uint64_t *)(vg + (reg_off >> 3)); + do { + if ((pg >> (reg_off & 63)) & 1) { + if (unlikely(flags & TLB_WATCHPOINT) && + (cpu_watchpoint_address_matches + (env_cpu(env), addr + mem_off, 1 << msz) + & BP_MEM_READ)) { + goto do_fault; + } + /* TODO: MTE check. */ host_fn(vd, reg_off, host + mem_off); - reg_off += 1 << esz; - reg_off = find_next_active(vg, reg_off, reg_max, esz); - mem_off = reg_off >> diffsz; - } while (split - mem_off >= (1 << msz)); - } - } - - record_fault(env, reg_off, reg_max); -} - -/* - * Common helper for all contiguous no-fault loads. - */ -static void sve_ldnf1_r(CPUARMState *env, void *vg, const target_ulong addr, - uint32_t desc, const int esz, const int msz, - sve_ldst1_host_fn *host_fn) -{ - const unsigned rd = extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT, 5); - void *vd = &env->vfp.zregs[rd]; - const int diffsz = esz - msz; - const intptr_t reg_max = simd_oprsz(desc); - const intptr_t mem_max = reg_max >> diffsz; - const int mmu_idx = cpu_mmu_index(env, false); - intptr_t split, reg_off, mem_off; - void *host; - -#ifdef CONFIG_USER_ONLY - host = tlb_vaddr_to_host(env, addr, MMU_DATA_LOAD, mmu_idx); - if (likely(page_check_range(addr, mem_max, PAGE_READ) == 0)) { - /* The entire operation is valid and will not fault. */ - reg_off = 0; - do { - mem_off = reg_off >> diffsz; - host_fn(vd, reg_off, host + mem_off); + } reg_off += 1 << esz; - reg_off = find_next_active(vg, reg_off, reg_max, esz); - } while (reg_off < reg_max); - return; - } -#endif + mem_off += 1 << msz; + } while (reg_off <= reg_last && (reg_off & 63)); + } while (reg_off <= reg_last); - /* There will be no fault, so we may modify in advance. */ - memset(vd, 0, reg_max); - - /* Skip to the first active element. */ - reg_off = find_next_active(vg, 0, reg_max, esz); - if (unlikely(reg_off == reg_max)) { - /* The entire predicate was false; no load occurs. */ - return; - } - mem_off = reg_off >> diffsz; - -#ifdef CONFIG_USER_ONLY - if (page_check_range(addr + mem_off, 1 << msz, PAGE_READ) == 0) { - /* At least one load is valid; take the rest of the page. */ - split = max_for_page(addr, mem_off + (1 << msz) - 1, mem_max); - do { - host_fn(vd, reg_off, host + mem_off); - reg_off += 1 << esz; - reg_off = find_next_active(vg, reg_off, reg_max, esz); - mem_off = reg_off >> diffsz; - } while (split - mem_off >= (1 << msz)); - } -#else /* - * If the address is not in the TLB, we have no way to bring the - * entry into the TLB without also risking a fault. Note that - * the corollary is that we never load from an address not in RAM. - * - * This last is out of spec, in a weird corner case. - * Per the MemNF/MemSingleNF pseudocode, a NF load from Device memory - * must not actually hit the bus -- it returns UNKNOWN data instead. - * But if you map non-RAM with Normal memory attributes and do a NF - * load then it should access the bus. (Nobody ought actually do this - * in the real world, obviously.) - * - * Then there are the annoying special cases with watchpoints... - * TODO: Add a form of non-faulting loads using cc->tlb_fill(probe=true). + * MemSingleNF is allowed to fail for any reason. We have special + * code above to handle the first element crossing a page boundary. + * As an implementation choice, decline to handle a cross-page element + * in any other position. */ - host = tlb_vaddr_to_host(env, addr + mem_off, MMU_DATA_LOAD, mmu_idx); - split = max_for_page(addr, mem_off, mem_max); - if (host && split >= (1 << msz)) { - host -= mem_off; - do { - host_fn(vd, reg_off, host + mem_off); - reg_off += 1 << esz; - reg_off = find_next_active(vg, reg_off, reg_max, esz); - mem_off = reg_off >> diffsz; - } while (split - mem_off >= (1 << msz)); + reg_off = info.reg_off_split; + if (reg_off >= 0) { + goto do_fault; } -#endif + second_page: + reg_off = info.reg_off_first[1]; + if (likely(reg_off < 0)) { + /* No active elements on the second page. All done. */ + return; + } + + /* + * MemSingleNF is allowed to fail for any reason. As an implementation + * choice, decline to handle elements on the second page. This should + * be low frequency as the guest walks through memory -- the next + * iteration of the guest's loop should be aligned on the page boundary, + * and then all following iterations will stay aligned. + */ + + do_fault: record_fault(env, reg_off, reg_max); } @@ -4873,58 +4844,61 @@ static void sve_ldnf1_r(CPUARMState *env, void *vg, const target_ulong addr, void HELPER(sve_ldff1##PART##_r)(CPUARMState *env, void *vg, \ target_ulong addr, uint32_t desc) \ { \ - sve_ldff1_r(env, vg, addr, desc, GETPC(), ESZ, 0, \ - sve_ld1##PART##_host, sve_ld1##PART##_tlb); \ + sve_ldnfff1_r(env, vg, addr, desc, GETPC(), ESZ, MO_8, FAULT_FIRST, \ + sve_ld1##PART##_host, sve_ld1##PART##_tlb); \ } \ void HELPER(sve_ldnf1##PART##_r)(CPUARMState *env, void *vg, \ target_ulong addr, uint32_t desc) \ { \ - sve_ldnf1_r(env, vg, addr, desc, ESZ, 0, sve_ld1##PART##_host); \ + sve_ldnfff1_r(env, vg, addr, desc, GETPC(), ESZ, MO_8, FAULT_NO, \ + sve_ld1##PART##_host, sve_ld1##PART##_tlb); \ } #define DO_LDFF1_LDNF1_2(PART, ESZ, MSZ) \ void HELPER(sve_ldff1##PART##_le_r)(CPUARMState *env, void *vg, \ target_ulong addr, uint32_t desc) \ { \ - sve_ldff1_r(env, vg, addr, desc, GETPC(), ESZ, MSZ, \ - sve_ld1##PART##_le_host, sve_ld1##PART##_le_tlb); \ + sve_ldnfff1_r(env, vg, addr, desc, GETPC(), ESZ, MSZ, FAULT_FIRST, \ + sve_ld1##PART##_le_host, sve_ld1##PART##_le_tlb); \ } \ void HELPER(sve_ldnf1##PART##_le_r)(CPUARMState *env, void *vg, \ target_ulong addr, uint32_t desc) \ { \ - sve_ldnf1_r(env, vg, addr, desc, ESZ, MSZ, sve_ld1##PART##_le_host); \ + sve_ldnfff1_r(env, vg, addr, desc, GETPC(), ESZ, MSZ, FAULT_NO, \ + sve_ld1##PART##_le_host, sve_ld1##PART##_le_tlb); \ } \ void HELPER(sve_ldff1##PART##_be_r)(CPUARMState *env, void *vg, \ target_ulong addr, uint32_t desc) \ { \ - sve_ldff1_r(env, vg, addr, desc, GETPC(), ESZ, MSZ, \ - sve_ld1##PART##_be_host, sve_ld1##PART##_be_tlb); \ + sve_ldnfff1_r(env, vg, addr, desc, GETPC(), ESZ, MSZ, FAULT_FIRST, \ + sve_ld1##PART##_be_host, sve_ld1##PART##_be_tlb); \ } \ void HELPER(sve_ldnf1##PART##_be_r)(CPUARMState *env, void *vg, \ target_ulong addr, uint32_t desc) \ { \ - sve_ldnf1_r(env, vg, addr, desc, ESZ, MSZ, sve_ld1##PART##_be_host); \ + sve_ldnfff1_r(env, vg, addr, desc, GETPC(), ESZ, MSZ, FAULT_NO, \ + sve_ld1##PART##_be_host, sve_ld1##PART##_be_tlb); \ } -DO_LDFF1_LDNF1_1(bb, 0) -DO_LDFF1_LDNF1_1(bhu, 1) -DO_LDFF1_LDNF1_1(bhs, 1) -DO_LDFF1_LDNF1_1(bsu, 2) -DO_LDFF1_LDNF1_1(bss, 2) -DO_LDFF1_LDNF1_1(bdu, 3) -DO_LDFF1_LDNF1_1(bds, 3) +DO_LDFF1_LDNF1_1(bb, MO_8) +DO_LDFF1_LDNF1_1(bhu, MO_16) +DO_LDFF1_LDNF1_1(bhs, MO_16) +DO_LDFF1_LDNF1_1(bsu, MO_32) +DO_LDFF1_LDNF1_1(bss, MO_32) +DO_LDFF1_LDNF1_1(bdu, MO_64) +DO_LDFF1_LDNF1_1(bds, MO_64) -DO_LDFF1_LDNF1_2(hh, 1, 1) -DO_LDFF1_LDNF1_2(hsu, 2, 1) -DO_LDFF1_LDNF1_2(hss, 2, 1) -DO_LDFF1_LDNF1_2(hdu, 3, 1) -DO_LDFF1_LDNF1_2(hds, 3, 1) +DO_LDFF1_LDNF1_2(hh, MO_16, MO_16) +DO_LDFF1_LDNF1_2(hsu, MO_32, MO_16) +DO_LDFF1_LDNF1_2(hss, MO_32, MO_16) +DO_LDFF1_LDNF1_2(hdu, MO_64, MO_16) +DO_LDFF1_LDNF1_2(hds, MO_64, MO_16) -DO_LDFF1_LDNF1_2(ss, 2, 2) -DO_LDFF1_LDNF1_2(sdu, 3, 2) -DO_LDFF1_LDNF1_2(sds, 3, 2) +DO_LDFF1_LDNF1_2(ss, MO_32, MO_32) +DO_LDFF1_LDNF1_2(sdu, MO_64, MO_32) +DO_LDFF1_LDNF1_2(sds, MO_64, MO_32) -DO_LDFF1_LDNF1_2(dd, 3, 3) +DO_LDFF1_LDNF1_2(dd, MO_64, MO_64) #undef DO_LDFF1_LDNF1_1 #undef DO_LDFF1_LDNF1_2 From patchwork Wed Apr 22 04:33:05 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 185666 Delivered-To: patch@linaro.org Received: by 2002:a92:3d9a:0:0:0:0:0 with SMTP id k26csp301405ilf; Tue, 21 Apr 2020 21:40:39 -0700 (PDT) X-Google-Smtp-Source: APiQypKlGN7TkNabS8OfjN0D6j56IXgVuHOZ1aCUr71nqBcb8L8F94Mv6dZiKRnUpeD8l2DJrmo3 X-Received: by 2002:a37:9b4f:: with SMTP id d76mr24520572qke.268.1587530439167; Tue, 21 Apr 2020 21:40:39 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1587530439; cv=none; d=google.com; s=arc-20160816; b=mX0ShHlRVJBrOnI7U9UffBgGUlIOlouoI2lxlgADjVDlTlMMZ4wy0oaoKYZjO8UAcq o5tO2lVdlj5VehsyzpGz32IUPBECsizJGM5+uCVIRhTDcqacx0pxq0rZ1eRlvMuN4p3H Y50HpMa4FKr5c+da7Q+cPxDAOpdjh90N7QN3IvGN2zFS7oIE4ret4QtFIiEAbTF0qUX3 f/feOetaXgWxVnf02SBzanz/r9/89+wezxGl+PTqmbLEu8diqUfz2iZfFdSMFe0gj+2P heV1kxSX0a8tkZVT3ewzUBx1+EIPw8FeZM+AIChx3qJA6HN1pJoFRlBaI4LybdfW3svB B0Gw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:cc:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:content-transfer-encoding :mime-version:references:in-reply-to:message-id:date:subject:to:from :dkim-signature; bh=WM1aUyfyRs93urrEY6+L5X21LUQrhWBFyRqQaG2TVqs=; b=bXb+Uy8yCPtLrqP16qzm4kV9Uv9fcTKH3B/aqu4UvKDbGB/ZsRPIiR/JbvrhMJ0qJZ sLCrrU/6QlYexaVyN15KvjIGy6nKfP/KvUyYwIvXR5g4wNs/Udf0EVFh5S5KyFMXLtUI CTUKYxK+CD9OdXkR801DvJfNL1hgPQRMw06D9mEQ3XQmevDLV3vGN49atSNUVnkDjGVl /pLOklOErgA/uGv/2cVsOW4kOsv4m/+T+FOlesWm3q0zni15eJfQ86VBr2JlPTn9Mz+C x1xa0YldA7dKkwJsP4shr9vTXhvgSO/j0FQG4UbcKyYIrJn93719ooDA52lHaEIBYf09 r7wQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=iPt7j29M; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [209.51.188.17]) by mx.google.com with ESMTPS id h26si2487301qtn.284.2020.04.21.21.40.39 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Tue, 21 Apr 2020 21:40:39 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=iPt7j29M; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1]:41450 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jR7Be-0002sU-F0 for patch@linaro.org; Wed, 22 Apr 2020 00:40:38 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:50772) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jR74p-0000zY-OW for qemu-devel@nongnu.org; Wed, 22 Apr 2020 00:33:39 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.90_1) (envelope-from ) id 1jR74n-0005z6-2n for qemu-devel@nongnu.org; Wed, 22 Apr 2020 00:33:35 -0400 Received: from mail-pl1-x642.google.com ([2607:f8b0:4864:20::642]:36935) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1jR74m-0005rH-Ie for qemu-devel@nongnu.org; Wed, 22 Apr 2020 00:33:32 -0400 Received: by mail-pl1-x642.google.com with SMTP id w20so439850pll.4 for ; Tue, 21 Apr 2020 21:33:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=WM1aUyfyRs93urrEY6+L5X21LUQrhWBFyRqQaG2TVqs=; b=iPt7j29Mm9xm7zu1akU3+XKCRrUKIjJmvmGmwrr71vTbgNgESFpHdwMwy69fsTspbs cWjZJfLAbo7uRmzmBqk+odhJKzx/V7Y8x3WJ69liKV8rVwrP6Ql26Ru78qsu9oe20NZa qiOlkpQj1j1PW+rsVPfaEeekMErAJypIM7re2fPYTNMnUcW1DH41yFCzW5A0dJHpwJXO 7pQ4PqdtiO3yIQbZNMnBqR9YMLU6MaEwO9OG7KMYLalFuY/9k5e3iOiMVphMA6gCNiap KgC1yNtoCpqr9ky5oRAUhoAZ2Cd2u3uJbWJsfPnhmD4nmUBQByLxcvpOrY0yZHJzUXq6 867Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=WM1aUyfyRs93urrEY6+L5X21LUQrhWBFyRqQaG2TVqs=; b=gWdSh/iXoMCXgM1qH/aszYB3QnbQ9dm0h2i+SNd3nZ4yvTbrkl4Yv5o4Y8uJhOUe7A gQqetCsO7bOGiGv3uHOD29IxNgYcJbtfZLBw4hF12ZhtTdYkN8rQ3fLuB/q2N9JuLOfq cynpQFfQmdGNHn6plA4rdZgzGMJ7gDSimMJCTX66VPWunURyep/4DvOuWlUXQPtjkvoM SeoLPZ9fNubXcuvq+DMESKtdll2A3mn3loU1lgKhC37qok7tKHUhJB5+AiBXi0gkBdUV xDpRXaOuIZATVJjycXnEQ1lEfW6ypsu0I0CS8+GmxKnmk8TlR1zJpKFxUB3gaELNlofG vfEA== X-Gm-Message-State: AGi0PuZv3x+6zCYEZhyMih0+FjXlA1mpKIDpzF1m1VT5ku7ypiZ++T/o VMihX/FWDptIkRwdXYMPQH0ATnlXXAo= X-Received: by 2002:a17:90a:9f0b:: with SMTP id n11mr8347214pjp.99.1587530010318; Tue, 21 Apr 2020 21:33:30 -0700 (PDT) Received: from localhost.localdomain (174-21-149-226.tukw.qwest.net. [174.21.149.226]) by smtp.gmail.com with ESMTPSA id l137sm4129613pfd.107.2020.04.21.21.33.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 21 Apr 2020 21:33:29 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Subject: [PATCH v3 14/18] target/arm: Use SVEContLdSt for contiguous stores Date: Tue, 21 Apr 2020 21:33:05 -0700 Message-Id: <20200422043309.18430-15-richard.henderson@linaro.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200422043309.18430-1-richard.henderson@linaro.org> References: <20200422043309.18430-1-richard.henderson@linaro.org> MIME-Version: 1.0 Received-SPF: pass client-ip=2607:f8b0:4864:20::642; envelope-from=richard.henderson@linaro.org; helo=mail-pl1-x642.google.com X-detected-operating-system: by eggs.gnu.org: Error: [-] PROGRAM ABORT : Malformed IPv6 address (bad octet value). Location : parse_addr6(), p0f-client.c:67 X-Received-From: 2607:f8b0:4864:20::642 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: peter.maydell@linaro.org, qemu-arm@nongnu.org Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: "Qemu-devel" Follow the model set up for contiguous loads. This handles watchpoints correctly for contiguous stores, recognizing the exception before any changes to memory. Reviewed-by: Peter Maydell Signed-off-by: Richard Henderson --- target/arm/sve_helper.c | 285 ++++++++++++++++++++++------------------ 1 file changed, 159 insertions(+), 126 deletions(-) -- 2.20.1 diff --git a/target/arm/sve_helper.c b/target/arm/sve_helper.c index 3473b53928..9389c7c76d 100644 --- a/target/arm/sve_helper.c +++ b/target/arm/sve_helper.c @@ -3995,6 +3995,10 @@ static void sve_##NAME##_host(void *vd, intptr_t reg_off, void *host) \ *(TYPEE *)(vd + H(reg_off)) = val; \ } +#define DO_ST_HOST(NAME, H, TYPEE, TYPEM, HOST) \ +static void sve_##NAME##_host(void *vd, intptr_t reg_off, void *host) \ +{ HOST(host, (TYPEM)*(TYPEE *)(vd + H(reg_off))); } + #define DO_LD_TLB(NAME, H, TYPEE, TYPEM, TLB) \ static void sve_##NAME##_tlb(CPUARMState *env, void *vd, intptr_t reg_off, \ target_ulong addr, uintptr_t ra) \ @@ -4022,6 +4026,7 @@ DO_LD_PRIM_1(ld1bdu, , uint64_t, uint8_t) DO_LD_PRIM_1(ld1bds, , uint64_t, int8_t) #define DO_ST_PRIM_1(NAME, H, TE, TM) \ + DO_ST_HOST(st1##NAME, H, TE, TM, stb_p) \ DO_ST_TLB(st1##NAME, H, TE, TM, cpu_stb_data_ra) DO_ST_PRIM_1(bb, H1, uint8_t, uint8_t) @@ -4036,6 +4041,8 @@ DO_ST_PRIM_1(bd, , uint64_t, uint8_t) DO_LD_TLB(ld1##NAME##_le, H, TE, TM, cpu_##LD##_le_data_ra) #define DO_ST_PRIM_2(NAME, H, TE, TM, ST) \ + DO_ST_HOST(st1##NAME##_be, H, TE, TM, ST##_be_p) \ + DO_ST_HOST(st1##NAME##_le, H, TE, TM, ST##_le_p) \ DO_ST_TLB(st1##NAME##_be, H, TE, TM, cpu_##ST##_be_data_ra) \ DO_ST_TLB(st1##NAME##_le, H, TE, TM, cpu_##ST##_le_data_ra) @@ -4904,151 +4911,177 @@ DO_LDFF1_LDNF1_2(dd, MO_64, MO_64) #undef DO_LDFF1_LDNF1_2 /* - * Common helpers for all contiguous 1,2,3,4-register predicated stores. + * Common helper for all contiguous 1,2,3,4-register predicated stores. */ -static void sve_st1_r(CPUARMState *env, void *vg, target_ulong addr, - uint32_t desc, const uintptr_t ra, - const int esize, const int msize, - sve_ldst1_tlb_fn *tlb_fn) + +static inline QEMU_ALWAYS_INLINE +void sve_stN_r(CPUARMState *env, uint64_t *vg, target_ulong addr, uint32_t desc, + const uintptr_t retaddr, const int esz, + const int msz, const int N, + sve_ldst1_host_fn *host_fn, + sve_ldst1_tlb_fn *tlb_fn) { const unsigned rd = extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT, 5); - intptr_t i, oprsz = simd_oprsz(desc); - void *vd = &env->vfp.zregs[rd]; + const intptr_t reg_max = simd_oprsz(desc); + intptr_t reg_off, reg_last, mem_off; + SVEContLdSt info; + void *host; + int i, flags; - for (i = 0; i < oprsz; ) { - uint16_t pg = *(uint16_t *)(vg + H1_2(i >> 3)); - do { - if (pg & 1) { - tlb_fn(env, vd, i, addr, ra); + /* Find the active elements. */ + if (!sve_cont_ldst_elements(&info, addr, vg, reg_max, esz, N << msz)) { + /* The entire predicate was false; no store occurs. */ + return; + } + + /* Probe the page(s). Exit with exception for any invalid page. */ + sve_cont_ldst_pages(&info, FAULT_ALL, env, addr, MMU_DATA_STORE, retaddr); + + /* Handle watchpoints for all active elements. */ + sve_cont_ldst_watchpoints(&info, env, vg, addr, 1 << esz, N << msz, + BP_MEM_WRITE, retaddr); + + /* TODO: MTE check. */ + + flags = info.page[0].flags | info.page[1].flags; + if (unlikely(flags != 0)) { +#ifdef CONFIG_USER_ONLY + g_assert_not_reached(); +#else + /* + * At least one page includes MMIO. + * Any bus operation can fail with cpu_transaction_failed, + * which for ARM will raise SyncExternal. We cannot avoid + * this fault and will leave with the store incomplete. + */ + mem_off = info.mem_off_first[0]; + reg_off = info.reg_off_first[0]; + reg_last = info.reg_off_last[1]; + if (reg_last < 0) { + reg_last = info.reg_off_split; + if (reg_last < 0) { + reg_last = info.reg_off_last[0]; } - i += esize, pg >>= esize; - addr += msize; - } while (i & 15); + } + + do { + uint64_t pg = vg[reg_off >> 6]; + do { + if ((pg >> (reg_off & 63)) & 1) { + for (i = 0; i < N; ++i) { + tlb_fn(env, &env->vfp.zregs[(rd + i) & 31], reg_off, + addr + mem_off + (i << msz), retaddr); + } + } + reg_off += 1 << esz; + mem_off += N << msz; + } while (reg_off & 63); + } while (reg_off <= reg_last); + return; +#endif + } + + mem_off = info.mem_off_first[0]; + reg_off = info.reg_off_first[0]; + reg_last = info.reg_off_last[0]; + host = info.page[0].host; + + while (reg_off <= reg_last) { + uint64_t pg = vg[reg_off >> 6]; + do { + if ((pg >> (reg_off & 63)) & 1) { + for (i = 0; i < N; ++i) { + host_fn(&env->vfp.zregs[(rd + i) & 31], reg_off, + host + mem_off + (i << msz)); + } + } + reg_off += 1 << esz; + mem_off += N << msz; + } while (reg_off <= reg_last && (reg_off & 63)); + } + + /* + * Use the slow path to manage the cross-page misalignment. + * But we know this is RAM and cannot trap. + */ + mem_off = info.mem_off_split; + if (unlikely(mem_off >= 0)) { + reg_off = info.reg_off_split; + for (i = 0; i < N; ++i) { + tlb_fn(env, &env->vfp.zregs[(rd + i) & 31], reg_off, + addr + mem_off + (i << msz), retaddr); + } + } + + mem_off = info.mem_off_first[1]; + if (unlikely(mem_off >= 0)) { + reg_off = info.reg_off_first[1]; + reg_last = info.reg_off_last[1]; + host = info.page[1].host; + + do { + uint64_t pg = vg[reg_off >> 6]; + do { + if ((pg >> (reg_off & 63)) & 1) { + for (i = 0; i < N; ++i) { + host_fn(&env->vfp.zregs[(rd + i) & 31], reg_off, + host + mem_off + (i << msz)); + } + } + reg_off += 1 << esz; + mem_off += N << msz; + } while (reg_off & 63); + } while (reg_off <= reg_last); } } -static void sve_st2_r(CPUARMState *env, void *vg, target_ulong addr, - uint32_t desc, const uintptr_t ra, - const int esize, const int msize, - sve_ldst1_tlb_fn *tlb_fn) -{ - const unsigned rd = extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT, 5); - intptr_t i, oprsz = simd_oprsz(desc); - void *d1 = &env->vfp.zregs[rd]; - void *d2 = &env->vfp.zregs[(rd + 1) & 31]; - - for (i = 0; i < oprsz; ) { - uint16_t pg = *(uint16_t *)(vg + H1_2(i >> 3)); - do { - if (pg & 1) { - tlb_fn(env, d1, i, addr, ra); - tlb_fn(env, d2, i, addr + msize, ra); - } - i += esize, pg >>= esize; - addr += 2 * msize; - } while (i & 15); - } -} - -static void sve_st3_r(CPUARMState *env, void *vg, target_ulong addr, - uint32_t desc, const uintptr_t ra, - const int esize, const int msize, - sve_ldst1_tlb_fn *tlb_fn) -{ - const unsigned rd = extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT, 5); - intptr_t i, oprsz = simd_oprsz(desc); - void *d1 = &env->vfp.zregs[rd]; - void *d2 = &env->vfp.zregs[(rd + 1) & 31]; - void *d3 = &env->vfp.zregs[(rd + 2) & 31]; - - for (i = 0; i < oprsz; ) { - uint16_t pg = *(uint16_t *)(vg + H1_2(i >> 3)); - do { - if (pg & 1) { - tlb_fn(env, d1, i, addr, ra); - tlb_fn(env, d2, i, addr + msize, ra); - tlb_fn(env, d3, i, addr + 2 * msize, ra); - } - i += esize, pg >>= esize; - addr += 3 * msize; - } while (i & 15); - } -} - -static void sve_st4_r(CPUARMState *env, void *vg, target_ulong addr, - uint32_t desc, const uintptr_t ra, - const int esize, const int msize, - sve_ldst1_tlb_fn *tlb_fn) -{ - const unsigned rd = extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT, 5); - intptr_t i, oprsz = simd_oprsz(desc); - void *d1 = &env->vfp.zregs[rd]; - void *d2 = &env->vfp.zregs[(rd + 1) & 31]; - void *d3 = &env->vfp.zregs[(rd + 2) & 31]; - void *d4 = &env->vfp.zregs[(rd + 3) & 31]; - - for (i = 0; i < oprsz; ) { - uint16_t pg = *(uint16_t *)(vg + H1_2(i >> 3)); - do { - if (pg & 1) { - tlb_fn(env, d1, i, addr, ra); - tlb_fn(env, d2, i, addr + msize, ra); - tlb_fn(env, d3, i, addr + 2 * msize, ra); - tlb_fn(env, d4, i, addr + 3 * msize, ra); - } - i += esize, pg >>= esize; - addr += 4 * msize; - } while (i & 15); - } -} - -#define DO_STN_1(N, NAME, ESIZE) \ -void QEMU_FLATTEN HELPER(sve_st##N##NAME##_r) \ - (CPUARMState *env, void *vg, target_ulong addr, uint32_t desc) \ +#define DO_STN_1(N, NAME, ESZ) \ +void HELPER(sve_st##N##NAME##_r)(CPUARMState *env, void *vg, \ + target_ulong addr, uint32_t desc) \ { \ - sve_st##N##_r(env, vg, addr, desc, GETPC(), ESIZE, 1, \ - sve_st1##NAME##_tlb); \ + sve_stN_r(env, vg, addr, desc, GETPC(), ESZ, MO_8, N, \ + sve_st1##NAME##_host, sve_st1##NAME##_tlb); \ } -#define DO_STN_2(N, NAME, ESIZE, MSIZE) \ -void QEMU_FLATTEN HELPER(sve_st##N##NAME##_le_r) \ - (CPUARMState *env, void *vg, target_ulong addr, uint32_t desc) \ +#define DO_STN_2(N, NAME, ESZ, MSZ) \ +void HELPER(sve_st##N##NAME##_le_r)(CPUARMState *env, void *vg, \ + target_ulong addr, uint32_t desc) \ { \ - sve_st##N##_r(env, vg, addr, desc, GETPC(), ESIZE, MSIZE, \ - sve_st1##NAME##_le_tlb); \ + sve_stN_r(env, vg, addr, desc, GETPC(), ESZ, MSZ, N, \ + sve_st1##NAME##_le_host, sve_st1##NAME##_le_tlb); \ } \ -void QEMU_FLATTEN HELPER(sve_st##N##NAME##_be_r) \ - (CPUARMState *env, void *vg, target_ulong addr, uint32_t desc) \ +void HELPER(sve_st##N##NAME##_be_r)(CPUARMState *env, void *vg, \ + target_ulong addr, uint32_t desc) \ { \ - sve_st##N##_r(env, vg, addr, desc, GETPC(), ESIZE, MSIZE, \ - sve_st1##NAME##_be_tlb); \ + sve_stN_r(env, vg, addr, desc, GETPC(), ESZ, MSZ, N, \ + sve_st1##NAME##_be_host, sve_st1##NAME##_be_tlb); \ } -DO_STN_1(1, bb, 1) -DO_STN_1(1, bh, 2) -DO_STN_1(1, bs, 4) -DO_STN_1(1, bd, 8) -DO_STN_1(2, bb, 1) -DO_STN_1(3, bb, 1) -DO_STN_1(4, bb, 1) +DO_STN_1(1, bb, MO_8) +DO_STN_1(1, bh, MO_16) +DO_STN_1(1, bs, MO_32) +DO_STN_1(1, bd, MO_64) +DO_STN_1(2, bb, MO_8) +DO_STN_1(3, bb, MO_8) +DO_STN_1(4, bb, MO_8) -DO_STN_2(1, hh, 2, 2) -DO_STN_2(1, hs, 4, 2) -DO_STN_2(1, hd, 8, 2) -DO_STN_2(2, hh, 2, 2) -DO_STN_2(3, hh, 2, 2) -DO_STN_2(4, hh, 2, 2) +DO_STN_2(1, hh, MO_16, MO_16) +DO_STN_2(1, hs, MO_32, MO_16) +DO_STN_2(1, hd, MO_64, MO_16) +DO_STN_2(2, hh, MO_16, MO_16) +DO_STN_2(3, hh, MO_16, MO_16) +DO_STN_2(4, hh, MO_16, MO_16) -DO_STN_2(1, ss, 4, 4) -DO_STN_2(1, sd, 8, 4) -DO_STN_2(2, ss, 4, 4) -DO_STN_2(3, ss, 4, 4) -DO_STN_2(4, ss, 4, 4) +DO_STN_2(1, ss, MO_32, MO_32) +DO_STN_2(1, sd, MO_64, MO_32) +DO_STN_2(2, ss, MO_32, MO_32) +DO_STN_2(3, ss, MO_32, MO_32) +DO_STN_2(4, ss, MO_32, MO_32) -DO_STN_2(1, dd, 8, 8) -DO_STN_2(2, dd, 8, 8) -DO_STN_2(3, dd, 8, 8) -DO_STN_2(4, dd, 8, 8) +DO_STN_2(1, dd, MO_64, MO_64) +DO_STN_2(2, dd, MO_64, MO_64) +DO_STN_2(3, dd, MO_64, MO_64) +DO_STN_2(4, dd, MO_64, MO_64) #undef DO_STN_1 #undef DO_STN_2 From patchwork Wed Apr 22 04:33:06 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 185671 Delivered-To: patch@linaro.org Received: by 2002:a92:3d9a:0:0:0:0:0 with SMTP id k26csp304249ilf; Tue, 21 Apr 2020 21:45:20 -0700 (PDT) X-Google-Smtp-Source: APiQypKbbkJmwsbbUke0BIVS78OhcCekrAQPB8VSG3/NfMZBst5r3IWg4RRd1AqY96xS85xKTLYM X-Received: by 2002:ac8:3403:: with SMTP id u3mr25295806qtb.274.1587530720655; Tue, 21 Apr 2020 21:45:20 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1587530720; cv=none; d=google.com; s=arc-20160816; b=tXckubQJQNQSnTgbhwAk7EVF5CTfdcQsum7A9QfmIYzzelckehtNsA9mq4UKWc4X4t 9nC81lB5rjXK3tdnU71nmAXgy0DVhguxioUseeTqRydO5AbBVhgkq5xqc8t3/GFHvxIg 6MrpiljITD282VYzDolK8n8RCKi9dyTQ2xwqJ4AAFNwFO+3dt1plf7yPjrU15qPR2USM ruO3UnoROBOtApJZSE87nAZIJBeZ1ZAEgW5JtlKbLKUI/c6NjccMnMazKlDEpEPnaJO/ d3fa4LmfZ6F3m79X7rDkqaG51VyTnLPKZMAmC3XoqGdjogV1Sf8d2baTle65ZDpscgl0 FqTQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:cc:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:content-transfer-encoding :mime-version:references:in-reply-to:message-id:date:subject:to:from :dkim-signature; bh=35NWN1irvYiuaoiJYNZapdYKg3V0iP81B2LwiXyBGUY=; b=zYCgVxP8laBRAbEmOpnIInFqaihArMd19e28Sz36WXLxlfE/iuSuTSoyDy2OTWvgqt JsIi97veelA3dlnnrf+uA9rBXLTFfWK4359RtnzLVGQbLgcy9jfgUD9MzzHIoYNP0T+S +oeeEqkqzPplFneaGrOIUhMzZ0EObLhDl3eQicA9QWiDlA4fEb1kifR0RZ9V2KzssdZN 5UrMmMxycn9G8YdkU91o8Fc32EDCWnMupm9w3tcl3ru+qN7mCLfJWOQrfMhJ3bNleb/7 pBpPzYSe3TZXQhGwkn8rUOvbL57a0DtyMWzbUEoWleCy84VB73wEvLCWZXJQdLCPDu8B jFfA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=FZqbA0JU; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [209.51.188.17]) by mx.google.com with ESMTPS id a1si2586083qkk.301.2020.04.21.21.45.20 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Tue, 21 Apr 2020 21:45:20 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=FZqbA0JU; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1]:41624 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jR7GB-0002I3-VH for patch@linaro.org; Wed, 22 Apr 2020 00:45:19 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:50882) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jR74t-00016f-5z for qemu-devel@nongnu.org; Wed, 22 Apr 2020 00:33:42 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.90_1) (envelope-from ) id 1jR74n-00062l-SE for qemu-devel@nongnu.org; Wed, 22 Apr 2020 00:33:38 -0400 Received: from mail-pj1-x1043.google.com ([2607:f8b0:4864:20::1043]:33727) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1jR74n-0005wZ-Do for qemu-devel@nongnu.org; Wed, 22 Apr 2020 00:33:33 -0400 Received: by mail-pj1-x1043.google.com with SMTP id 7so2013736pjo.0 for ; Tue, 21 Apr 2020 21:33:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=35NWN1irvYiuaoiJYNZapdYKg3V0iP81B2LwiXyBGUY=; b=FZqbA0JUkwdhCxOIhLjbuQgHhmSCGJwX27BCr3k6fQE/42saBtMTFfrsnbGG4g6vFl 0FRdORAVevp2wlHvlYNbjLasYWV7eVLTXPKdqpe89Ww2kkofb798WuPtcSKTKFhfpG66 cUUUTIIZWIpff8lw+I4SlMM1VLWrxhMGoHhh11Jqeww6CqDzypOI4KeZ1TSu3MiF50DN iPVfZnDF0YZj9Z3Q/0gFy+4ELVqKGD5tpIo+OknxZzX76DZEO3egreqEC2byVpF9QrUF 7Fwxb3vJB5A0RBOLBeYB3l49abdIicyMODqhyizvpTv6bknx1FpzXm4rAQ/HMJN4ZHqY APNg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=35NWN1irvYiuaoiJYNZapdYKg3V0iP81B2LwiXyBGUY=; b=BNN9z+r4k/OoBHG2ieCsndYJjiIKClHY2tQB6SeHFfg2FJL31NpkuRLjrazjsrkG18 vrqder0K6IxQNs3sY+SIz4LdxQms4YTio2P7anYnnXxJGIvOsFaA4OEptjTQu5Z9sVJt ezIS+KG70kIDfg+QLnMdV28ssoiLl2vwqj86jkHuINVTQ1fOjAWzvVHg4yt3cu1ofufX cZDKBnupaP0Np5/uo3+krby55DunIswtE7l1UDehWMYmzaQrXTdJj/WfVVWKst9bSQwY 3jXXp9T4gXSackRXxGOkdXkqmfPRd1CgMjbsYRnzcq9at1TEoz9G7RawI4HIGsFe8Z2j kEUg== X-Gm-Message-State: AGi0PuYJIYuBbi4s3YFLzw9M0qeAJlHQ6gX3zzqTu1gFNwdrXWAGLwZT QRGknZ+fG5/d7jZki1ahXR9PEDH3UF4= X-Received: by 2002:a17:90a:718c:: with SMTP id i12mr9646155pjk.142.1587530011413; Tue, 21 Apr 2020 21:33:31 -0700 (PDT) Received: from localhost.localdomain (174-21-149-226.tukw.qwest.net. [174.21.149.226]) by smtp.gmail.com with ESMTPSA id l137sm4129613pfd.107.2020.04.21.21.33.30 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 21 Apr 2020 21:33:30 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Subject: [PATCH v3 15/18] target/arm: Reuse sve_probe_page for gather first-fault loads Date: Tue, 21 Apr 2020 21:33:06 -0700 Message-Id: <20200422043309.18430-16-richard.henderson@linaro.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200422043309.18430-1-richard.henderson@linaro.org> References: <20200422043309.18430-1-richard.henderson@linaro.org> MIME-Version: 1.0 Received-SPF: pass client-ip=2607:f8b0:4864:20::1043; envelope-from=richard.henderson@linaro.org; helo=mail-pj1-x1043.google.com X-detected-operating-system: by eggs.gnu.org: Error: [-] PROGRAM ABORT : Malformed IPv6 address (bad octet value). Location : parse_addr6(), p0f-client.c:67 X-Received-From: 2607:f8b0:4864:20::1043 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: peter.maydell@linaro.org, qemu-arm@nongnu.org Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: "Qemu-devel" This avoids the need for a separate set of helpers to implement no-fault semantics, and will enable MTE in the future. Reviewed-by: Peter Maydell Signed-off-by: Richard Henderson --- target/arm/sve_helper.c | 323 ++++++++++++++++------------------------ 1 file changed, 127 insertions(+), 196 deletions(-) -- 2.20.1 diff --git a/target/arm/sve_helper.c b/target/arm/sve_helper.c index 9389c7c76d..6229ea65c0 100644 --- a/target/arm/sve_helper.c +++ b/target/arm/sve_helper.c @@ -5250,231 +5250,162 @@ DO_LD1_ZPZ_D(dd_be, zd) /* First fault loads with a vector index. */ -/* Load one element into VD+REG_OFF from (ENV,VADDR) without faulting. - * The controlling predicate is known to be true. Return true if the - * load was successful. - */ -typedef bool sve_ld1_nf_fn(CPUARMState *env, void *vd, intptr_t reg_off, - target_ulong vaddr, int mmu_idx); - -#ifdef CONFIG_SOFTMMU -#define DO_LD_NF(NAME, H, TYPEE, TYPEM, HOST) \ -static bool sve_ld##NAME##_nf(CPUARMState *env, void *vd, intptr_t reg_off, \ - target_ulong addr, int mmu_idx) \ -{ \ - target_ulong next_page = -(addr | TARGET_PAGE_MASK); \ - if (likely(next_page - addr >= sizeof(TYPEM))) { \ - void *host = tlb_vaddr_to_host(env, addr, MMU_DATA_LOAD, mmu_idx); \ - if (likely(host)) { \ - TYPEM val = HOST(host); \ - *(TYPEE *)(vd + H(reg_off)) = val; \ - return true; \ - } \ - } \ - return false; \ -} -#else -#define DO_LD_NF(NAME, H, TYPEE, TYPEM, HOST) \ -static bool sve_ld##NAME##_nf(CPUARMState *env, void *vd, intptr_t reg_off, \ - target_ulong addr, int mmu_idx) \ -{ \ - if (likely(page_check_range(addr, sizeof(TYPEM), PAGE_READ))) { \ - TYPEM val = HOST(g2h(addr)); \ - *(TYPEE *)(vd + H(reg_off)) = val; \ - return true; \ - } \ - return false; \ -} -#endif - -DO_LD_NF(bsu, H1_4, uint32_t, uint8_t, ldub_p) -DO_LD_NF(bss, H1_4, uint32_t, int8_t, ldsb_p) -DO_LD_NF(bdu, , uint64_t, uint8_t, ldub_p) -DO_LD_NF(bds, , uint64_t, int8_t, ldsb_p) - -DO_LD_NF(hsu_le, H1_4, uint32_t, uint16_t, lduw_le_p) -DO_LD_NF(hss_le, H1_4, uint32_t, int16_t, ldsw_le_p) -DO_LD_NF(hsu_be, H1_4, uint32_t, uint16_t, lduw_be_p) -DO_LD_NF(hss_be, H1_4, uint32_t, int16_t, ldsw_be_p) -DO_LD_NF(hdu_le, , uint64_t, uint16_t, lduw_le_p) -DO_LD_NF(hds_le, , uint64_t, int16_t, ldsw_le_p) -DO_LD_NF(hdu_be, , uint64_t, uint16_t, lduw_be_p) -DO_LD_NF(hds_be, , uint64_t, int16_t, ldsw_be_p) - -DO_LD_NF(ss_le, H1_4, uint32_t, uint32_t, ldl_le_p) -DO_LD_NF(ss_be, H1_4, uint32_t, uint32_t, ldl_be_p) -DO_LD_NF(sdu_le, , uint64_t, uint32_t, ldl_le_p) -DO_LD_NF(sds_le, , uint64_t, int32_t, ldl_le_p) -DO_LD_NF(sdu_be, , uint64_t, uint32_t, ldl_be_p) -DO_LD_NF(sds_be, , uint64_t, int32_t, ldl_be_p) - -DO_LD_NF(dd_le, , uint64_t, uint64_t, ldq_le_p) -DO_LD_NF(dd_be, , uint64_t, uint64_t, ldq_be_p) - /* - * Common helper for all gather first-faulting loads. + * Common helpers for all gather first-faulting loads. */ -static inline void sve_ldff1_zs(CPUARMState *env, void *vd, void *vg, void *vm, - target_ulong base, uint32_t desc, uintptr_t ra, - zreg_off_fn *off_fn, sve_ldst1_tlb_fn *tlb_fn, - sve_ld1_nf_fn *nonfault_fn) + +static inline QEMU_ALWAYS_INLINE +void sve_ldff1_z(CPUARMState *env, void *vd, uint64_t *vg, void *vm, + target_ulong base, uint32_t desc, uintptr_t retaddr, + const int esz, const int msz, zreg_off_fn *off_fn, + sve_ldst1_host_fn *host_fn, + sve_ldst1_tlb_fn *tlb_fn) { - const TCGMemOpIdx oi = extract32(desc, SIMD_DATA_SHIFT, MEMOPIDX_SHIFT); - const int mmu_idx = get_mmuidx(oi); + const int mmu_idx = cpu_mmu_index(env, false); const int scale = extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT, 2); - intptr_t reg_off, reg_max = simd_oprsz(desc); - target_ulong addr; + const int esize = 1 << esz; + const int msize = 1 << msz; + const intptr_t reg_max = simd_oprsz(desc); + intptr_t reg_off; + SVEHostPage info; + target_ulong addr, in_page; /* Skip to the first true predicate. */ - reg_off = find_next_active(vg, 0, reg_max, MO_32); - if (likely(reg_off < reg_max)) { - /* Perform one normal read, which will fault or not. */ - addr = off_fn(vm, reg_off); - addr = base + (addr << scale); - tlb_fn(env, vd, reg_off, addr, ra); - - /* The rest of the reads will be non-faulting. */ + reg_off = find_next_active(vg, 0, reg_max, esz); + if (unlikely(reg_off >= reg_max)) { + /* The entire predicate was false; no load occurs. */ + memset(vd, 0, reg_max); + return; } - /* After any fault, zero the leading predicated false elements. */ + /* + * Probe the first element, allowing faults. + */ + addr = base + (off_fn(vm, reg_off) << scale); + tlb_fn(env, vd, reg_off, addr, retaddr); + + /* After any fault, zero the other elements. */ swap_memzero(vd, reg_off); + reg_off += esize; + swap_memzero(vd + reg_off, reg_max - reg_off); - while (likely((reg_off += 4) < reg_max)) { - uint64_t pg = *(uint64_t *)(vg + (reg_off >> 6) * 8); - if (likely((pg >> (reg_off & 63)) & 1)) { - addr = off_fn(vm, reg_off); - addr = base + (addr << scale); - if (!nonfault_fn(env, vd, reg_off, addr, mmu_idx)) { - record_fault(env, reg_off, reg_max); - break; + /* + * Probe the remaining elements, not allowing faults. + */ + while (reg_off < reg_max) { + uint64_t pg = vg[reg_off >> 6]; + do { + if (likely((pg >> (reg_off & 63)) & 1)) { + addr = base + (off_fn(vm, reg_off) << scale); + in_page = -(addr | TARGET_PAGE_MASK); + + if (unlikely(in_page < msize)) { + /* Stop if the element crosses a page boundary. */ + goto fault; + } + + sve_probe_page(&info, true, env, addr, 0, MMU_DATA_LOAD, + mmu_idx, retaddr); + if (unlikely(info.flags & (TLB_INVALID_MASK | TLB_MMIO))) { + goto fault; + } + if (unlikely(info.flags & TLB_WATCHPOINT) && + (cpu_watchpoint_address_matches + (env_cpu(env), addr, msize) & BP_MEM_READ)) { + goto fault; + } + /* TODO: MTE check. */ + + host_fn(vd, reg_off, info.host); } - } else { - *(uint32_t *)(vd + H1_4(reg_off)) = 0; - } + reg_off += esize; + } while (reg_off & 63); } + return; + + fault: + record_fault(env, reg_off, reg_max); } -static inline void sve_ldff1_zd(CPUARMState *env, void *vd, void *vg, void *vm, - target_ulong base, uint32_t desc, uintptr_t ra, - zreg_off_fn *off_fn, sve_ldst1_tlb_fn *tlb_fn, - sve_ld1_nf_fn *nonfault_fn) -{ - const TCGMemOpIdx oi = extract32(desc, SIMD_DATA_SHIFT, MEMOPIDX_SHIFT); - const int mmu_idx = get_mmuidx(oi); - const int scale = extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT, 2); - intptr_t reg_off, reg_max = simd_oprsz(desc); - target_ulong addr; - - /* Skip to the first true predicate. */ - reg_off = find_next_active(vg, 0, reg_max, MO_64); - if (likely(reg_off < reg_max)) { - /* Perform one normal read, which will fault or not. */ - addr = off_fn(vm, reg_off); - addr = base + (addr << scale); - tlb_fn(env, vd, reg_off, addr, ra); - - /* The rest of the reads will be non-faulting. */ - } - - /* After any fault, zero the leading predicated false elements. */ - swap_memzero(vd, reg_off); - - while (likely((reg_off += 8) < reg_max)) { - uint8_t pg = *(uint8_t *)(vg + H1(reg_off >> 3)); - if (likely(pg & 1)) { - addr = off_fn(vm, reg_off); - addr = base + (addr << scale); - if (!nonfault_fn(env, vd, reg_off, addr, mmu_idx)) { - record_fault(env, reg_off, reg_max); - break; - } - } else { - *(uint64_t *)(vd + reg_off) = 0; - } - } +#define DO_LDFF1_ZPZ_S(MEM, OFS, MSZ) \ +void HELPER(sve_ldff##MEM##_##OFS)(CPUARMState *env, void *vd, void *vg, \ + void *vm, target_ulong base, uint32_t desc) \ +{ \ + sve_ldff1_z(env, vd, vg, vm, base, desc, GETPC(), MO_32, MSZ, \ + off_##OFS##_s, sve_ld1##MEM##_host, sve_ld1##MEM##_tlb); \ } -#define DO_LDFF1_ZPZ_S(MEM, OFS) \ -void HELPER(sve_ldff##MEM##_##OFS) \ - (CPUARMState *env, void *vd, void *vg, void *vm, \ - target_ulong base, uint32_t desc) \ -{ \ - sve_ldff1_zs(env, vd, vg, vm, base, desc, GETPC(), \ - off_##OFS##_s, sve_ld1##MEM##_tlb, sve_ld##MEM##_nf); \ +#define DO_LDFF1_ZPZ_D(MEM, OFS, MSZ) \ +void HELPER(sve_ldff##MEM##_##OFS)(CPUARMState *env, void *vd, void *vg, \ + void *vm, target_ulong base, uint32_t desc) \ +{ \ + sve_ldff1_z(env, vd, vg, vm, base, desc, GETPC(), MO_64, MSZ, \ + off_##OFS##_d, sve_ld1##MEM##_host, sve_ld1##MEM##_tlb); \ } -#define DO_LDFF1_ZPZ_D(MEM, OFS) \ -void HELPER(sve_ldff##MEM##_##OFS) \ - (CPUARMState *env, void *vd, void *vg, void *vm, \ - target_ulong base, uint32_t desc) \ -{ \ - sve_ldff1_zd(env, vd, vg, vm, base, desc, GETPC(), \ - off_##OFS##_d, sve_ld1##MEM##_tlb, sve_ld##MEM##_nf); \ -} +DO_LDFF1_ZPZ_S(bsu, zsu, MO_8) +DO_LDFF1_ZPZ_S(bsu, zss, MO_8) +DO_LDFF1_ZPZ_D(bdu, zsu, MO_8) +DO_LDFF1_ZPZ_D(bdu, zss, MO_8) +DO_LDFF1_ZPZ_D(bdu, zd, MO_8) -DO_LDFF1_ZPZ_S(bsu, zsu) -DO_LDFF1_ZPZ_S(bsu, zss) -DO_LDFF1_ZPZ_D(bdu, zsu) -DO_LDFF1_ZPZ_D(bdu, zss) -DO_LDFF1_ZPZ_D(bdu, zd) +DO_LDFF1_ZPZ_S(bss, zsu, MO_8) +DO_LDFF1_ZPZ_S(bss, zss, MO_8) +DO_LDFF1_ZPZ_D(bds, zsu, MO_8) +DO_LDFF1_ZPZ_D(bds, zss, MO_8) +DO_LDFF1_ZPZ_D(bds, zd, MO_8) -DO_LDFF1_ZPZ_S(bss, zsu) -DO_LDFF1_ZPZ_S(bss, zss) -DO_LDFF1_ZPZ_D(bds, zsu) -DO_LDFF1_ZPZ_D(bds, zss) -DO_LDFF1_ZPZ_D(bds, zd) +DO_LDFF1_ZPZ_S(hsu_le, zsu, MO_16) +DO_LDFF1_ZPZ_S(hsu_le, zss, MO_16) +DO_LDFF1_ZPZ_D(hdu_le, zsu, MO_16) +DO_LDFF1_ZPZ_D(hdu_le, zss, MO_16) +DO_LDFF1_ZPZ_D(hdu_le, zd, MO_16) -DO_LDFF1_ZPZ_S(hsu_le, zsu) -DO_LDFF1_ZPZ_S(hsu_le, zss) -DO_LDFF1_ZPZ_D(hdu_le, zsu) -DO_LDFF1_ZPZ_D(hdu_le, zss) -DO_LDFF1_ZPZ_D(hdu_le, zd) +DO_LDFF1_ZPZ_S(hsu_be, zsu, MO_16) +DO_LDFF1_ZPZ_S(hsu_be, zss, MO_16) +DO_LDFF1_ZPZ_D(hdu_be, zsu, MO_16) +DO_LDFF1_ZPZ_D(hdu_be, zss, MO_16) +DO_LDFF1_ZPZ_D(hdu_be, zd, MO_16) -DO_LDFF1_ZPZ_S(hsu_be, zsu) -DO_LDFF1_ZPZ_S(hsu_be, zss) -DO_LDFF1_ZPZ_D(hdu_be, zsu) -DO_LDFF1_ZPZ_D(hdu_be, zss) -DO_LDFF1_ZPZ_D(hdu_be, zd) +DO_LDFF1_ZPZ_S(hss_le, zsu, MO_16) +DO_LDFF1_ZPZ_S(hss_le, zss, MO_16) +DO_LDFF1_ZPZ_D(hds_le, zsu, MO_16) +DO_LDFF1_ZPZ_D(hds_le, zss, MO_16) +DO_LDFF1_ZPZ_D(hds_le, zd, MO_16) -DO_LDFF1_ZPZ_S(hss_le, zsu) -DO_LDFF1_ZPZ_S(hss_le, zss) -DO_LDFF1_ZPZ_D(hds_le, zsu) -DO_LDFF1_ZPZ_D(hds_le, zss) -DO_LDFF1_ZPZ_D(hds_le, zd) +DO_LDFF1_ZPZ_S(hss_be, zsu, MO_16) +DO_LDFF1_ZPZ_S(hss_be, zss, MO_16) +DO_LDFF1_ZPZ_D(hds_be, zsu, MO_16) +DO_LDFF1_ZPZ_D(hds_be, zss, MO_16) +DO_LDFF1_ZPZ_D(hds_be, zd, MO_16) -DO_LDFF1_ZPZ_S(hss_be, zsu) -DO_LDFF1_ZPZ_S(hss_be, zss) -DO_LDFF1_ZPZ_D(hds_be, zsu) -DO_LDFF1_ZPZ_D(hds_be, zss) -DO_LDFF1_ZPZ_D(hds_be, zd) +DO_LDFF1_ZPZ_S(ss_le, zsu, MO_32) +DO_LDFF1_ZPZ_S(ss_le, zss, MO_32) +DO_LDFF1_ZPZ_D(sdu_le, zsu, MO_32) +DO_LDFF1_ZPZ_D(sdu_le, zss, MO_32) +DO_LDFF1_ZPZ_D(sdu_le, zd, MO_32) -DO_LDFF1_ZPZ_S(ss_le, zsu) -DO_LDFF1_ZPZ_S(ss_le, zss) -DO_LDFF1_ZPZ_D(sdu_le, zsu) -DO_LDFF1_ZPZ_D(sdu_le, zss) -DO_LDFF1_ZPZ_D(sdu_le, zd) +DO_LDFF1_ZPZ_S(ss_be, zsu, MO_32) +DO_LDFF1_ZPZ_S(ss_be, zss, MO_32) +DO_LDFF1_ZPZ_D(sdu_be, zsu, MO_32) +DO_LDFF1_ZPZ_D(sdu_be, zss, MO_32) +DO_LDFF1_ZPZ_D(sdu_be, zd, MO_32) -DO_LDFF1_ZPZ_S(ss_be, zsu) -DO_LDFF1_ZPZ_S(ss_be, zss) -DO_LDFF1_ZPZ_D(sdu_be, zsu) -DO_LDFF1_ZPZ_D(sdu_be, zss) -DO_LDFF1_ZPZ_D(sdu_be, zd) +DO_LDFF1_ZPZ_D(sds_le, zsu, MO_32) +DO_LDFF1_ZPZ_D(sds_le, zss, MO_32) +DO_LDFF1_ZPZ_D(sds_le, zd, MO_32) -DO_LDFF1_ZPZ_D(sds_le, zsu) -DO_LDFF1_ZPZ_D(sds_le, zss) -DO_LDFF1_ZPZ_D(sds_le, zd) +DO_LDFF1_ZPZ_D(sds_be, zsu, MO_32) +DO_LDFF1_ZPZ_D(sds_be, zss, MO_32) +DO_LDFF1_ZPZ_D(sds_be, zd, MO_32) -DO_LDFF1_ZPZ_D(sds_be, zsu) -DO_LDFF1_ZPZ_D(sds_be, zss) -DO_LDFF1_ZPZ_D(sds_be, zd) +DO_LDFF1_ZPZ_D(dd_le, zsu, MO_64) +DO_LDFF1_ZPZ_D(dd_le, zss, MO_64) +DO_LDFF1_ZPZ_D(dd_le, zd, MO_64) -DO_LDFF1_ZPZ_D(dd_le, zsu) -DO_LDFF1_ZPZ_D(dd_le, zss) -DO_LDFF1_ZPZ_D(dd_le, zd) - -DO_LDFF1_ZPZ_D(dd_be, zsu) -DO_LDFF1_ZPZ_D(dd_be, zss) -DO_LDFF1_ZPZ_D(dd_be, zd) +DO_LDFF1_ZPZ_D(dd_be, zsu, MO_64) +DO_LDFF1_ZPZ_D(dd_be, zss, MO_64) +DO_LDFF1_ZPZ_D(dd_be, zd, MO_64) /* Stores with a vector index. */ From patchwork Wed Apr 22 04:33:07 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 185663 Delivered-To: patch@linaro.org Received: by 2002:a92:3d9a:0:0:0:0:0 with SMTP id k26csp300416ilf; Tue, 21 Apr 2020 21:38:56 -0700 (PDT) X-Google-Smtp-Source: APiQypL7F2O7WhVCZWq090QiFkOUX1/30+jJcWrGonZnMUTjDJI0rjCIGYCk7OpVN3rCWtmLN9lF X-Received: by 2002:a0c:a122:: with SMTP id d31mr23247519qva.109.1587530336589; Tue, 21 Apr 2020 21:38:56 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1587530336; cv=none; d=google.com; s=arc-20160816; b=wh0yVj1hwdCL7mcoQTXezUJPVPKzVnH85/7fi68zh4zXICdO0KYiTncDZuOaVERzjH oB43QT06K6MIhqjwDopXF+hgLSX8RmST9QYLz2wiLnEcr11Rx0EoFQE6pW3llaTNUfCj 1IT2aJSUpdCzNa+mZBVf9oi3u7eS45NGI6hvcsX4CJHCsXGXCJEyO7HfMuAGb4XOMwjs PshBDl5LvdYKRUSNoxqaBmMwjvkyEoqco78EqbC3XjzxOAwdcviQIHOJk/mwwBM9rJB/ g1UL02mkS4x1DAXH0De2HsuK9IW75VE6qMC4lmUwEEgsRk2bexGTpZ8rG4IlKVeCLrSV l52w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:cc:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:content-transfer-encoding :mime-version:references:in-reply-to:message-id:date:subject:to:from :dkim-signature; bh=I8JvQT8TOXMCxEmIUxHpTVpmQjSRp16/JfuG+I4HlbU=; b=xPU7sw36EBJuYhbDopyJ5XnESFn+FDSzbyZRQ2tUJOwptS8tsZ1WfrxOeiW95BmgTr qqPA1GTx893DXGy607Ewf6o9zuXeqtoXjxJ66t0jBSi9/CSz3j+457x/MuxVTSdMVviH xh1oWcALspIK7YV2xyfkNXxT13vgjxtuof8a+SC4dmz0U9W/PFjiyft+38ZN3PmGcl8U Ya1X+HLr3oeU5olZEnXESxeJcSQtEkUrHJhAC7PmtIV4EtIDocZpDXVCDriQh/RJpxNd R3KckdKakRZuPD6PCAb5XDRtb32GrfbpZjEvt4cczhN0wUO5byx86hhR4j2t+h/r97DC NENw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b="J/kUOpS9"; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [209.51.188.17]) by mx.google.com with ESMTPS id ei17si2413511qvb.82.2020.04.21.21.38.56 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Tue, 21 Apr 2020 21:38:56 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b="J/kUOpS9"; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1]:41438 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jR79z-00014g-TO for patch@linaro.org; Wed, 22 Apr 2020 00:38:55 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:50870) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jR74s-00016G-NC for qemu-devel@nongnu.org; Wed, 22 Apr 2020 00:33:41 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.90_1) (envelope-from ) id 1jR74o-00067f-V8 for qemu-devel@nongnu.org; Wed, 22 Apr 2020 00:33:38 -0400 Received: from mail-pl1-x643.google.com ([2607:f8b0:4864:20::643]:42412) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1jR74o-00062b-Eq for qemu-devel@nongnu.org; Wed, 22 Apr 2020 00:33:34 -0400 Received: by mail-pl1-x643.google.com with SMTP id v2so428003plp.9 for ; Tue, 21 Apr 2020 21:33:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=I8JvQT8TOXMCxEmIUxHpTVpmQjSRp16/JfuG+I4HlbU=; b=J/kUOpS9/EHzyPrlDxCyPAO6h+tRJZ8og3NA7almJqODsAEjqqzrB226rwf5tudWNK GpizkGfnLbUkRG2AyCbzwGm7iydowJUWr4MU2ALFqhryKlrTrQoCn1inCjOAR0LMvoo4 MjI4AhI4guI8722jwmFG7L0rScd/0xrhrSnbSvD9/2xw8XtjuSmjDRAkg8VBgas2XQPB 73jHl1bAr9qBd6LPCj7/7UGb3tE6F3S8v0g16OPh4IJAb97BEHiUwvbFkGYBBcx2lpkT 6jvbIPVYzC5T7rynoP3OSqSr1wtB7jzMM/mg2SYWd/pH6AiOXdCVfqjAk9oXR483S+ZF XFIQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=I8JvQT8TOXMCxEmIUxHpTVpmQjSRp16/JfuG+I4HlbU=; b=aERybOealWf0K+4LhqJCVXcur7yKwPUJz0gUzpNoO05fPj30f70ceBcOmohvMqJGRE omngKx0/5tVx7/s7MaPM2nf/TNz6MonlWCNdtIXtWceFC5bB3QB67J0eKjTCY8VU80Cg CWzlH12uTfaithQNtEkoPkExdBK0kntbTqwlaeeN3ivg46kibzUjUix1hIEeQyo71ZMu UDhiHJiqqxXQH4QlsRfawElKHYDlI7qgcCd0d0UIntLmrx5nHwPwxYWUZRDF4E0oKqAB AhC1yYWOyX0IIWi0efVfcSyTya27f2ev/uiAAQqA5b/g4rgkut2LNI+Cq639MtODShEf ldlg== X-Gm-Message-State: AGi0Pub9nvXLFn9K/XPuTwxOlxNLW4K81VmHHwYhM/PuPrCwjPVffBFq 5W1KN4JYeIMGQuqRIEJD3d3CxF6+MeQ= X-Received: by 2002:a17:90a:2b8f:: with SMTP id u15mr9965726pjd.137.1587530012758; Tue, 21 Apr 2020 21:33:32 -0700 (PDT) Received: from localhost.localdomain (174-21-149-226.tukw.qwest.net. [174.21.149.226]) by smtp.gmail.com with ESMTPSA id l137sm4129613pfd.107.2020.04.21.21.33.31 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 21 Apr 2020 21:33:32 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Subject: [PATCH v3 16/18] target/arm: Reuse sve_probe_page for scatter stores Date: Tue, 21 Apr 2020 21:33:07 -0700 Message-Id: <20200422043309.18430-17-richard.henderson@linaro.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200422043309.18430-1-richard.henderson@linaro.org> References: <20200422043309.18430-1-richard.henderson@linaro.org> MIME-Version: 1.0 Received-SPF: pass client-ip=2607:f8b0:4864:20::643; envelope-from=richard.henderson@linaro.org; helo=mail-pl1-x643.google.com X-detected-operating-system: by eggs.gnu.org: Error: [-] PROGRAM ABORT : Malformed IPv6 address (bad octet value). Location : parse_addr6(), p0f-client.c:67 X-Received-From: 2607:f8b0:4864:20::643 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: peter.maydell@linaro.org, qemu-arm@nongnu.org Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: "Qemu-devel" Reviewed-by: Peter Maydell Signed-off-by: Richard Henderson --- target/arm/sve_helper.c | 182 ++++++++++++++++++++++++---------------- 1 file changed, 111 insertions(+), 71 deletions(-) -- 2.20.1 diff --git a/target/arm/sve_helper.c b/target/arm/sve_helper.c index 6229ea65c0..f4cdeecdcb 100644 --- a/target/arm/sve_helper.c +++ b/target/arm/sve_helper.c @@ -5409,94 +5409,134 @@ DO_LDFF1_ZPZ_D(dd_be, zd, MO_64) /* Stores with a vector index. */ -static void sve_st1_zs(CPUARMState *env, void *vd, void *vg, void *vm, - target_ulong base, uint32_t desc, uintptr_t ra, - zreg_off_fn *off_fn, sve_ldst1_tlb_fn *tlb_fn) +static inline QEMU_ALWAYS_INLINE +void sve_st1_z(CPUARMState *env, void *vd, uint64_t *vg, void *vm, + target_ulong base, uint32_t desc, uintptr_t retaddr, + int esize, int msize, zreg_off_fn *off_fn, + sve_ldst1_host_fn *host_fn, + sve_ldst1_tlb_fn *tlb_fn) { const int scale = extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT, 2); - intptr_t i, oprsz = simd_oprsz(desc); + const int mmu_idx = cpu_mmu_index(env, false); + const intptr_t reg_max = simd_oprsz(desc); + void *host[ARM_MAX_VQ * 4]; + intptr_t reg_off, i; + SVEHostPage info, info2; - for (i = 0; i < oprsz; ) { - uint16_t pg = *(uint16_t *)(vg + H1_2(i >> 3)); + /* + * Probe all of the elements for host addresses and flags. + */ + i = reg_off = 0; + do { + uint64_t pg = vg[reg_off >> 6]; do { - if (likely(pg & 1)) { - target_ulong off = off_fn(vm, i); - tlb_fn(env, vd, i, base + (off << scale), ra); + target_ulong addr = base + (off_fn(vm, reg_off) << scale); + target_ulong in_page = -(addr | TARGET_PAGE_MASK); + + host[i] = NULL; + if (likely((pg >> (reg_off & 63)) & 1)) { + if (likely(in_page >= msize)) { + sve_probe_page(&info, false, env, addr, 0, MMU_DATA_STORE, + mmu_idx, retaddr); + host[i] = info.host; + } else { + /* + * Element crosses the page boundary. + * Probe both pages, but do not record the host address, + * so that we use the slow path. + */ + sve_probe_page(&info, false, env, addr, 0, + MMU_DATA_STORE, mmu_idx, retaddr); + sve_probe_page(&info2, false, env, addr + in_page, 0, + MMU_DATA_STORE, mmu_idx, retaddr); + info.flags |= info2.flags; + } + + if (unlikely(info.flags & TLB_WATCHPOINT)) { + cpu_check_watchpoint(env_cpu(env), addr, msize, + info.attrs, BP_MEM_WRITE, retaddr); + } + /* TODO: MTE check. */ } - i += 4, pg >>= 4; - } while (i & 15); - } -} + i += 1; + reg_off += esize; + } while (reg_off & 63); + } while (reg_off < reg_max); -static void sve_st1_zd(CPUARMState *env, void *vd, void *vg, void *vm, - target_ulong base, uint32_t desc, uintptr_t ra, - zreg_off_fn *off_fn, sve_ldst1_tlb_fn *tlb_fn) -{ - const int scale = extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT, 2); - intptr_t i, oprsz = simd_oprsz(desc) / 8; - - for (i = 0; i < oprsz; i++) { - uint8_t pg = *(uint8_t *)(vg + H1(i)); - if (likely(pg & 1)) { - target_ulong off = off_fn(vm, i * 8); - tlb_fn(env, vd, i * 8, base + (off << scale), ra); + /* + * Now that we have recognized all exceptions except SyncExternal + * (from TLB_MMIO), which we cannot avoid, perform all of the stores. + * + * Note for the common case of an element in RAM, not crossing a page + * boundary, we have stored the host address in host[]. This doubles + * as a first-level check against the predicate, since only enabled + * elements have non-null host addresses. + */ + i = reg_off = 0; + do { + void *h = host[i]; + if (likely(h != NULL)) { + host_fn(vd, reg_off, h); + } else if ((vg[reg_off >> 6] >> (reg_off & 63)) & 1) { + target_ulong addr = base + (off_fn(vm, reg_off) << scale); + tlb_fn(env, vd, reg_off, addr, retaddr); } - } + i += 1; + reg_off += esize; + } while (reg_off < reg_max); } -#define DO_ST1_ZPZ_S(MEM, OFS) \ -void QEMU_FLATTEN HELPER(sve_st##MEM##_##OFS) \ - (CPUARMState *env, void *vd, void *vg, void *vm, \ - target_ulong base, uint32_t desc) \ -{ \ - sve_st1_zs(env, vd, vg, vm, base, desc, GETPC(), \ - off_##OFS##_s, sve_st1##MEM##_tlb); \ +#define DO_ST1_ZPZ_S(MEM, OFS, MSZ) \ +void HELPER(sve_st##MEM##_##OFS)(CPUARMState *env, void *vd, void *vg, \ + void *vm, target_ulong base, uint32_t desc) \ +{ \ + sve_st1_z(env, vd, vg, vm, base, desc, GETPC(), 4, 1 << MSZ, \ + off_##OFS##_s, sve_st1##MEM##_host, sve_st1##MEM##_tlb); \ } -#define DO_ST1_ZPZ_D(MEM, OFS) \ -void QEMU_FLATTEN HELPER(sve_st##MEM##_##OFS) \ - (CPUARMState *env, void *vd, void *vg, void *vm, \ - target_ulong base, uint32_t desc) \ -{ \ - sve_st1_zd(env, vd, vg, vm, base, desc, GETPC(), \ - off_##OFS##_d, sve_st1##MEM##_tlb); \ +#define DO_ST1_ZPZ_D(MEM, OFS, MSZ) \ +void HELPER(sve_st##MEM##_##OFS)(CPUARMState *env, void *vd, void *vg, \ + void *vm, target_ulong base, uint32_t desc) \ +{ \ + sve_st1_z(env, vd, vg, vm, base, desc, GETPC(), 8, 1 << MSZ, \ + off_##OFS##_d, sve_st1##MEM##_host, sve_st1##MEM##_tlb); \ } -DO_ST1_ZPZ_S(bs, zsu) -DO_ST1_ZPZ_S(hs_le, zsu) -DO_ST1_ZPZ_S(hs_be, zsu) -DO_ST1_ZPZ_S(ss_le, zsu) -DO_ST1_ZPZ_S(ss_be, zsu) +DO_ST1_ZPZ_S(bs, zsu, MO_8) +DO_ST1_ZPZ_S(hs_le, zsu, MO_16) +DO_ST1_ZPZ_S(hs_be, zsu, MO_16) +DO_ST1_ZPZ_S(ss_le, zsu, MO_32) +DO_ST1_ZPZ_S(ss_be, zsu, MO_32) -DO_ST1_ZPZ_S(bs, zss) -DO_ST1_ZPZ_S(hs_le, zss) -DO_ST1_ZPZ_S(hs_be, zss) -DO_ST1_ZPZ_S(ss_le, zss) -DO_ST1_ZPZ_S(ss_be, zss) +DO_ST1_ZPZ_S(bs, zss, MO_8) +DO_ST1_ZPZ_S(hs_le, zss, MO_16) +DO_ST1_ZPZ_S(hs_be, zss, MO_16) +DO_ST1_ZPZ_S(ss_le, zss, MO_32) +DO_ST1_ZPZ_S(ss_be, zss, MO_32) -DO_ST1_ZPZ_D(bd, zsu) -DO_ST1_ZPZ_D(hd_le, zsu) -DO_ST1_ZPZ_D(hd_be, zsu) -DO_ST1_ZPZ_D(sd_le, zsu) -DO_ST1_ZPZ_D(sd_be, zsu) -DO_ST1_ZPZ_D(dd_le, zsu) -DO_ST1_ZPZ_D(dd_be, zsu) +DO_ST1_ZPZ_D(bd, zsu, MO_8) +DO_ST1_ZPZ_D(hd_le, zsu, MO_16) +DO_ST1_ZPZ_D(hd_be, zsu, MO_16) +DO_ST1_ZPZ_D(sd_le, zsu, MO_32) +DO_ST1_ZPZ_D(sd_be, zsu, MO_32) +DO_ST1_ZPZ_D(dd_le, zsu, MO_64) +DO_ST1_ZPZ_D(dd_be, zsu, MO_64) -DO_ST1_ZPZ_D(bd, zss) -DO_ST1_ZPZ_D(hd_le, zss) -DO_ST1_ZPZ_D(hd_be, zss) -DO_ST1_ZPZ_D(sd_le, zss) -DO_ST1_ZPZ_D(sd_be, zss) -DO_ST1_ZPZ_D(dd_le, zss) -DO_ST1_ZPZ_D(dd_be, zss) +DO_ST1_ZPZ_D(bd, zss, MO_8) +DO_ST1_ZPZ_D(hd_le, zss, MO_16) +DO_ST1_ZPZ_D(hd_be, zss, MO_16) +DO_ST1_ZPZ_D(sd_le, zss, MO_32) +DO_ST1_ZPZ_D(sd_be, zss, MO_32) +DO_ST1_ZPZ_D(dd_le, zss, MO_64) +DO_ST1_ZPZ_D(dd_be, zss, MO_64) -DO_ST1_ZPZ_D(bd, zd) -DO_ST1_ZPZ_D(hd_le, zd) -DO_ST1_ZPZ_D(hd_be, zd) -DO_ST1_ZPZ_D(sd_le, zd) -DO_ST1_ZPZ_D(sd_be, zd) -DO_ST1_ZPZ_D(dd_le, zd) -DO_ST1_ZPZ_D(dd_be, zd) +DO_ST1_ZPZ_D(bd, zd, MO_8) +DO_ST1_ZPZ_D(hd_le, zd, MO_16) +DO_ST1_ZPZ_D(hd_be, zd, MO_16) +DO_ST1_ZPZ_D(sd_le, zd, MO_32) +DO_ST1_ZPZ_D(sd_be, zd, MO_32) +DO_ST1_ZPZ_D(dd_le, zd, MO_64) +DO_ST1_ZPZ_D(dd_be, zd, MO_64) #undef DO_ST1_ZPZ_S #undef DO_ST1_ZPZ_D From patchwork Wed Apr 22 04:33:08 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 185672 Delivered-To: patch@linaro.org Received: by 2002:a92:3d9a:0:0:0:0:0 with SMTP id k26csp305005ilf; Tue, 21 Apr 2020 21:46:31 -0700 (PDT) X-Google-Smtp-Source: APiQypLgwT7gMXlU3zjreWcMO8Kn6RaagTbgCj129EDjQOU2sSpf4tEtSvArKPoAoS0Yo77MR4ee X-Received: by 2002:ac8:6ec8:: with SMTP id f8mr23356864qtv.330.1587530791742; Tue, 21 Apr 2020 21:46:31 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1587530791; cv=none; d=google.com; s=arc-20160816; b=QD43+LQWZeEdjLPWFhGCArvg5z0MRVNB/Wp+snpL/Kj0QDqYPo1LYJlTXNqtzNESku NpEQeD41ataG070DykFH00TbQA24oU0CQ64Lm0aQF31QwViIWFXTmRHTZ3uk+F1OB1X4 IwNXouDxPPix24OZtBlBiQox9E/XQoJxef9n54x6+9aKWZJbC/Nz9La6r6eHVp/0fpiO RkmkL4JT8K2fvBOmaRvVApwoKVi7T49ZrZPNQi36mHNsmF8h2JkIMFFV6jEs1rUXzheg Ln74arI+fMl3NyCtTMg6q3l8MBdDFNY74yIRnEVhwd5s4YUZ4YkRcpVhDSP0eqxfHfGX H0Uw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:cc:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:content-transfer-encoding :mime-version:references:in-reply-to:message-id:date:subject:to:from :dkim-signature; bh=AuL6vKFq5L1zc6YYFULSpnJ4zpW3ZJD/n4vMjhxr+d8=; b=jHTj+tEnXBWLsJ8k117aN7BixKP+Wmf8wnYK7iOTRQ5x2OeO4fqQhK0QQqPLwFx7ob n9j6qawNkznMibvxnpY5VlROwEVRAN/avXzyr168pZVPIEupJqFXA1LvhiFCySxUqhiw nz1pONZ1kVocvkjauUeRMHzzx8XKCw2FbUidqSGZTU+Jb2FBDE/igBQNm1atJZYbz4xu EBu4QuAXzoMASuGjvdmnEtU2iRweemOi03Wlg4trdowNCwueCpe/0XGxxlab36h7jL3u JbHeNNN+2FyY0OVcTrK2b9IquE7M3qMc+hF6BqqVG5GcDqrVBc8a+pDe1sw1lAft7FZE feFQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b="t6NUJdc/"; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [209.51.188.17]) by mx.google.com with ESMTPS id e7si2548136qtg.389.2020.04.21.21.46.31 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Tue, 21 Apr 2020 21:46:31 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b="t6NUJdc/"; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1]:41662 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jR7HL-0003vj-3n for patch@linaro.org; Wed, 22 Apr 2020 00:46:31 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:50948) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jR74u-0001AO-MF for qemu-devel@nongnu.org; Wed, 22 Apr 2020 00:33:42 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.90_1) (envelope-from ) id 1jR74q-0006FF-Bc for qemu-devel@nongnu.org; Wed, 22 Apr 2020 00:33:40 -0400 Received: from mail-pl1-x643.google.com ([2607:f8b0:4864:20::643]:36506) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1jR74p-00068K-S0 for qemu-devel@nongnu.org; Wed, 22 Apr 2020 00:33:35 -0400 Received: by mail-pl1-x643.google.com with SMTP id g2so443090plo.3 for ; Tue, 21 Apr 2020 21:33:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=AuL6vKFq5L1zc6YYFULSpnJ4zpW3ZJD/n4vMjhxr+d8=; b=t6NUJdc/ggm0HNO1q2NuaF+pD6OV7JMnVZ4edXz9/kODDBDl57af6tnaEr5svX3RxP K5sEXiK2S84wx9O2Y4BNJK7Ix/AImjZZ+YdqdzXRHZnX/+Xz1comzZQIsEH4aETENn3b T7fLS2XhXo0RApffAH+XpsRzSfdTDXaYm2T9WfwLDNczsCxrVwVpr58MH81/qTQ1ZwSx g0Af76Tv/9Fz/4LqS9PpG3lIj7GZTZN/QFNs/kklq1INX/A0It/wrXCUTVsz/YD2j8dO k0SfHSYUF+WedjtqAl8VXuMm1U7Tb5FMJRKtqEsmfFjKVuOzJjnGIm6KNj7yVnALEpW7 pbiQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=AuL6vKFq5L1zc6YYFULSpnJ4zpW3ZJD/n4vMjhxr+d8=; b=mZs4Die50L3xSFxRgtsOwUK2Vz1wgrV+TaE2G+crb7KvnAgGZDpeEknSNG19Zy2HLK 5zsQuFNeTfDlSSIL+ZGWN8FUl18zEfyYaO+xvRiFhhnI/3m9iSSxQHumDqlg77g98KdU 5b2BUuWMg+kpKbSui/eIWosovcaeE1bZULl+Z5fnjR61fv9rCKFBuwnmJxaaroYB9tkS LyAmTmhY3++LhPzFktDI4jpgDHmN1sjXyD8Zr1fXgqWtWZWuEGCdGZ+aIhq2wdLgTNuZ b1V1HiHEZja20QtKCDiAZf9T2chsdrIWehVvff5QBK3iYyn2W4W1mSyTEqn9Q9t3ajMZ oNbQ== X-Gm-Message-State: AGi0PubKYpIE/iv4uXkpWV8O9apepOPhFLxv8WY3gdG2edsmVKgdAMhD ZKuJE5u0VlHLoHDHaPDk2mZLuyj+qQ0= X-Received: by 2002:a17:90a:8d0f:: with SMTP id c15mr9915901pjo.100.1587530013993; Tue, 21 Apr 2020 21:33:33 -0700 (PDT) Received: from localhost.localdomain (174-21-149-226.tukw.qwest.net. [174.21.149.226]) by smtp.gmail.com with ESMTPSA id l137sm4129613pfd.107.2020.04.21.21.33.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 21 Apr 2020 21:33:33 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Subject: [PATCH v3 17/18] target/arm: Reuse sve_probe_page for gather loads Date: Tue, 21 Apr 2020 21:33:08 -0700 Message-Id: <20200422043309.18430-18-richard.henderson@linaro.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200422043309.18430-1-richard.henderson@linaro.org> References: <20200422043309.18430-1-richard.henderson@linaro.org> MIME-Version: 1.0 Received-SPF: pass client-ip=2607:f8b0:4864:20::643; envelope-from=richard.henderson@linaro.org; helo=mail-pl1-x643.google.com X-detected-operating-system: by eggs.gnu.org: Error: [-] PROGRAM ABORT : Malformed IPv6 address (bad octet value). Location : parse_addr6(), p0f-client.c:67 X-Received-From: 2607:f8b0:4864:20::643 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: peter.maydell@linaro.org, qemu-arm@nongnu.org Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: "Qemu-devel" Reviewed-by: Peter Maydell Signed-off-by: Richard Henderson --- target/arm/sve_helper.c | 208 +++++++++++++++++++++------------------- 1 file changed, 109 insertions(+), 99 deletions(-) -- 2.20.1 diff --git a/target/arm/sve_helper.c b/target/arm/sve_helper.c index f4cdeecdcb..fffde4b6ec 100644 --- a/target/arm/sve_helper.c +++ b/target/arm/sve_helper.c @@ -5120,130 +5120,140 @@ static target_ulong off_zd_d(void *reg, intptr_t reg_ofs) return *(uint64_t *)(reg + reg_ofs); } -static void sve_ld1_zs(CPUARMState *env, void *vd, void *vg, void *vm, - target_ulong base, uint32_t desc, uintptr_t ra, - zreg_off_fn *off_fn, sve_ldst1_tlb_fn *tlb_fn) +static inline QEMU_ALWAYS_INLINE +void sve_ld1_z(CPUARMState *env, void *vd, uint64_t *vg, void *vm, + target_ulong base, uint32_t desc, uintptr_t retaddr, + int esize, int msize, zreg_off_fn *off_fn, + sve_ldst1_host_fn *host_fn, + sve_ldst1_tlb_fn *tlb_fn) { const int scale = extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT, 2); - intptr_t i, oprsz = simd_oprsz(desc); - ARMVectorReg scratch = { }; + const int mmu_idx = cpu_mmu_index(env, false); + const intptr_t reg_max = simd_oprsz(desc); + ARMVectorReg scratch; + intptr_t reg_off; + SVEHostPage info, info2; - for (i = 0; i < oprsz; ) { - uint16_t pg = *(uint16_t *)(vg + H1_2(i >> 3)); + memset(&scratch, 0, reg_max); + reg_off = 0; + do { + uint64_t pg = vg[reg_off >> 6]; do { if (likely(pg & 1)) { - target_ulong off = off_fn(vm, i); - tlb_fn(env, &scratch, i, base + (off << scale), ra); + target_ulong addr = base + (off_fn(vm, reg_off) << scale); + target_ulong in_page = -(addr | TARGET_PAGE_MASK); + + sve_probe_page(&info, false, env, addr, 0, MMU_DATA_LOAD, + mmu_idx, retaddr); + + if (likely(in_page >= msize)) { + if (unlikely(info.flags & TLB_WATCHPOINT)) { + cpu_check_watchpoint(env_cpu(env), addr, msize, + info.attrs, BP_MEM_READ, retaddr); + } + /* TODO: MTE check */ + host_fn(&scratch, reg_off, info.host); + } else { + /* Element crosses the page boundary. */ + sve_probe_page(&info2, false, env, addr + in_page, 0, + MMU_DATA_LOAD, mmu_idx, retaddr); + if (unlikely((info.flags | info2.flags) & TLB_WATCHPOINT)) { + cpu_check_watchpoint(env_cpu(env), addr, + msize, info.attrs, + BP_MEM_READ, retaddr); + } + /* TODO: MTE check */ + tlb_fn(env, &scratch, reg_off, addr, retaddr); + } } - i += 4, pg >>= 4; - } while (i & 15); - } + reg_off += esize; + pg >>= esize; + } while (reg_off & 63); + } while (reg_off < reg_max); /* Wait until all exceptions have been raised to write back. */ - memcpy(vd, &scratch, oprsz); + memcpy(vd, &scratch, reg_max); } -static void sve_ld1_zd(CPUARMState *env, void *vd, void *vg, void *vm, - target_ulong base, uint32_t desc, uintptr_t ra, - zreg_off_fn *off_fn, sve_ldst1_tlb_fn *tlb_fn) -{ - const int scale = extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT, 2); - intptr_t i, oprsz = simd_oprsz(desc) / 8; - ARMVectorReg scratch = { }; - - for (i = 0; i < oprsz; i++) { - uint8_t pg = *(uint8_t *)(vg + H1(i)); - if (likely(pg & 1)) { - target_ulong off = off_fn(vm, i * 8); - tlb_fn(env, &scratch, i * 8, base + (off << scale), ra); - } - } - - /* Wait until all exceptions have been raised to write back. */ - memcpy(vd, &scratch, oprsz * 8); +#define DO_LD1_ZPZ_S(MEM, OFS, MSZ) \ +void HELPER(sve_ld##MEM##_##OFS)(CPUARMState *env, void *vd, void *vg, \ + void *vm, target_ulong base, uint32_t desc) \ +{ \ + sve_ld1_z(env, vd, vg, vm, base, desc, GETPC(), 4, 1 << MSZ, \ + off_##OFS##_s, sve_ld1##MEM##_host, sve_ld1##MEM##_tlb); \ } -#define DO_LD1_ZPZ_S(MEM, OFS) \ -void QEMU_FLATTEN HELPER(sve_ld##MEM##_##OFS) \ - (CPUARMState *env, void *vd, void *vg, void *vm, \ - target_ulong base, uint32_t desc) \ -{ \ - sve_ld1_zs(env, vd, vg, vm, base, desc, GETPC(), \ - off_##OFS##_s, sve_ld1##MEM##_tlb); \ +#define DO_LD1_ZPZ_D(MEM, OFS, MSZ) \ +void HELPER(sve_ld##MEM##_##OFS)(CPUARMState *env, void *vd, void *vg, \ + void *vm, target_ulong base, uint32_t desc) \ +{ \ + sve_ld1_z(env, vd, vg, vm, base, desc, GETPC(), 8, 1 << MSZ, \ + off_##OFS##_d, sve_ld1##MEM##_host, sve_ld1##MEM##_tlb); \ } -#define DO_LD1_ZPZ_D(MEM, OFS) \ -void QEMU_FLATTEN HELPER(sve_ld##MEM##_##OFS) \ - (CPUARMState *env, void *vd, void *vg, void *vm, \ - target_ulong base, uint32_t desc) \ -{ \ - sve_ld1_zd(env, vd, vg, vm, base, desc, GETPC(), \ - off_##OFS##_d, sve_ld1##MEM##_tlb); \ -} +DO_LD1_ZPZ_S(bsu, zsu, MO_8) +DO_LD1_ZPZ_S(bsu, zss, MO_8) +DO_LD1_ZPZ_D(bdu, zsu, MO_8) +DO_LD1_ZPZ_D(bdu, zss, MO_8) +DO_LD1_ZPZ_D(bdu, zd, MO_8) -DO_LD1_ZPZ_S(bsu, zsu) -DO_LD1_ZPZ_S(bsu, zss) -DO_LD1_ZPZ_D(bdu, zsu) -DO_LD1_ZPZ_D(bdu, zss) -DO_LD1_ZPZ_D(bdu, zd) +DO_LD1_ZPZ_S(bss, zsu, MO_8) +DO_LD1_ZPZ_S(bss, zss, MO_8) +DO_LD1_ZPZ_D(bds, zsu, MO_8) +DO_LD1_ZPZ_D(bds, zss, MO_8) +DO_LD1_ZPZ_D(bds, zd, MO_8) -DO_LD1_ZPZ_S(bss, zsu) -DO_LD1_ZPZ_S(bss, zss) -DO_LD1_ZPZ_D(bds, zsu) -DO_LD1_ZPZ_D(bds, zss) -DO_LD1_ZPZ_D(bds, zd) +DO_LD1_ZPZ_S(hsu_le, zsu, MO_16) +DO_LD1_ZPZ_S(hsu_le, zss, MO_16) +DO_LD1_ZPZ_D(hdu_le, zsu, MO_16) +DO_LD1_ZPZ_D(hdu_le, zss, MO_16) +DO_LD1_ZPZ_D(hdu_le, zd, MO_16) -DO_LD1_ZPZ_S(hsu_le, zsu) -DO_LD1_ZPZ_S(hsu_le, zss) -DO_LD1_ZPZ_D(hdu_le, zsu) -DO_LD1_ZPZ_D(hdu_le, zss) -DO_LD1_ZPZ_D(hdu_le, zd) +DO_LD1_ZPZ_S(hsu_be, zsu, MO_16) +DO_LD1_ZPZ_S(hsu_be, zss, MO_16) +DO_LD1_ZPZ_D(hdu_be, zsu, MO_16) +DO_LD1_ZPZ_D(hdu_be, zss, MO_16) +DO_LD1_ZPZ_D(hdu_be, zd, MO_16) -DO_LD1_ZPZ_S(hsu_be, zsu) -DO_LD1_ZPZ_S(hsu_be, zss) -DO_LD1_ZPZ_D(hdu_be, zsu) -DO_LD1_ZPZ_D(hdu_be, zss) -DO_LD1_ZPZ_D(hdu_be, zd) +DO_LD1_ZPZ_S(hss_le, zsu, MO_16) +DO_LD1_ZPZ_S(hss_le, zss, MO_16) +DO_LD1_ZPZ_D(hds_le, zsu, MO_16) +DO_LD1_ZPZ_D(hds_le, zss, MO_16) +DO_LD1_ZPZ_D(hds_le, zd, MO_16) -DO_LD1_ZPZ_S(hss_le, zsu) -DO_LD1_ZPZ_S(hss_le, zss) -DO_LD1_ZPZ_D(hds_le, zsu) -DO_LD1_ZPZ_D(hds_le, zss) -DO_LD1_ZPZ_D(hds_le, zd) +DO_LD1_ZPZ_S(hss_be, zsu, MO_16) +DO_LD1_ZPZ_S(hss_be, zss, MO_16) +DO_LD1_ZPZ_D(hds_be, zsu, MO_16) +DO_LD1_ZPZ_D(hds_be, zss, MO_16) +DO_LD1_ZPZ_D(hds_be, zd, MO_16) -DO_LD1_ZPZ_S(hss_be, zsu) -DO_LD1_ZPZ_S(hss_be, zss) -DO_LD1_ZPZ_D(hds_be, zsu) -DO_LD1_ZPZ_D(hds_be, zss) -DO_LD1_ZPZ_D(hds_be, zd) +DO_LD1_ZPZ_S(ss_le, zsu, MO_32) +DO_LD1_ZPZ_S(ss_le, zss, MO_32) +DO_LD1_ZPZ_D(sdu_le, zsu, MO_32) +DO_LD1_ZPZ_D(sdu_le, zss, MO_32) +DO_LD1_ZPZ_D(sdu_le, zd, MO_32) -DO_LD1_ZPZ_S(ss_le, zsu) -DO_LD1_ZPZ_S(ss_le, zss) -DO_LD1_ZPZ_D(sdu_le, zsu) -DO_LD1_ZPZ_D(sdu_le, zss) -DO_LD1_ZPZ_D(sdu_le, zd) +DO_LD1_ZPZ_S(ss_be, zsu, MO_32) +DO_LD1_ZPZ_S(ss_be, zss, MO_32) +DO_LD1_ZPZ_D(sdu_be, zsu, MO_32) +DO_LD1_ZPZ_D(sdu_be, zss, MO_32) +DO_LD1_ZPZ_D(sdu_be, zd, MO_32) -DO_LD1_ZPZ_S(ss_be, zsu) -DO_LD1_ZPZ_S(ss_be, zss) -DO_LD1_ZPZ_D(sdu_be, zsu) -DO_LD1_ZPZ_D(sdu_be, zss) -DO_LD1_ZPZ_D(sdu_be, zd) +DO_LD1_ZPZ_D(sds_le, zsu, MO_32) +DO_LD1_ZPZ_D(sds_le, zss, MO_32) +DO_LD1_ZPZ_D(sds_le, zd, MO_32) -DO_LD1_ZPZ_D(sds_le, zsu) -DO_LD1_ZPZ_D(sds_le, zss) -DO_LD1_ZPZ_D(sds_le, zd) +DO_LD1_ZPZ_D(sds_be, zsu, MO_32) +DO_LD1_ZPZ_D(sds_be, zss, MO_32) +DO_LD1_ZPZ_D(sds_be, zd, MO_32) -DO_LD1_ZPZ_D(sds_be, zsu) -DO_LD1_ZPZ_D(sds_be, zss) -DO_LD1_ZPZ_D(sds_be, zd) +DO_LD1_ZPZ_D(dd_le, zsu, MO_64) +DO_LD1_ZPZ_D(dd_le, zss, MO_64) +DO_LD1_ZPZ_D(dd_le, zd, MO_64) -DO_LD1_ZPZ_D(dd_le, zsu) -DO_LD1_ZPZ_D(dd_le, zss) -DO_LD1_ZPZ_D(dd_le, zd) - -DO_LD1_ZPZ_D(dd_be, zsu) -DO_LD1_ZPZ_D(dd_be, zss) -DO_LD1_ZPZ_D(dd_be, zd) +DO_LD1_ZPZ_D(dd_be, zsu, MO_64) +DO_LD1_ZPZ_D(dd_be, zss, MO_64) +DO_LD1_ZPZ_D(dd_be, zd, MO_64) #undef DO_LD1_ZPZ_S #undef DO_LD1_ZPZ_D From patchwork Wed Apr 22 04:33:09 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 185673 Delivered-To: patch@linaro.org Received: by 2002:a92:3d9a:0:0:0:0:0 with SMTP id k26csp305872ilf; Tue, 21 Apr 2020 21:47:55 -0700 (PDT) X-Google-Smtp-Source: APiQypI+By1K7/+25h4K/kBV9+yL9uqcema4W5eogBVAYB4ibAOBQ7dywzs+DGIANLsLTCz+P+/U X-Received: by 2002:ae9:ebc8:: with SMTP id b191mr23712826qkg.1.1587530874961; Tue, 21 Apr 2020 21:47:54 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1587530874; cv=none; d=google.com; s=arc-20160816; b=Hw5z1N6Xy+ffe1zluknGN+yGl/NmMcwh63HoHIjH4cixj48pOEa0KboEGvBCXpu9mr R8kecXJZmZgwywjrHRNj7taXz5+RMrprahMhLSnwgEmLQQAAeroN0iwmRhjVH7CPufWr vycXwu7lhnWHTu8lRh/EsIyKlC9RHhbJZAar6g1ew8xHEMLruh0aE/7SuuPqt24CSUI0 oX/wXinzX4pbFYxVnjXaouJ2dE+Dgdr7jvOLEbk9ieVUG1ETGxqf4YcYqYm48qWlcPjt RshJ5SDpt4D5LcdfcK6h/EGl/NbbRKbyPyiqVvPhF0TR5K5h2TrcecAfBlYFsmBSC19u jdZQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:cc:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:content-transfer-encoding :mime-version:references:in-reply-to:message-id:date:subject:to:from :dkim-signature; bh=kAjJuC8E6QocqpgcZFzGlZ3iBSLxUs4KOHUk+pvyTWc=; b=oV5F+uKFKP/WjN/jsiKzurCVVembGX6SvJ7gcNeWxsIg/YSUOlC1ihZxnEIqpRudAc 68tDImRwf2o6xcyZjfyNt6TWfTpxMIicmBag549Y4JHUrU9EZppM6EJOZdV9DkYcnja3 PLAGWx5I1uXV0La9CJ+21yLXc0ZBMVWUSI7PPY8Offm7QY44UKZGfuh2VAXFU8zLtbv3 wNB6owurUvFCZi58h+2wZKxaH8wDLoxWbgJoV9oQu4W88WrMxShokMCTSimgXnsGVbrl rAslpV/dsgl6Pd3hNUGmSdBoR/jw/guVfm383aLuDvuO4caXCvSyGKodP0JKKQZbgo6t lnUQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=mPea3epQ; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [209.51.188.17]) by mx.google.com with ESMTPS id m12si2540920qkg.200.2020.04.21.21.47.54 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Tue, 21 Apr 2020 21:47:54 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=mPea3epQ; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1]:41674 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jR7Ig-00055U-9R for patch@linaro.org; Wed, 22 Apr 2020 00:47:54 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:50980) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jR74v-0001CF-GA for qemu-devel@nongnu.org; Wed, 22 Apr 2020 00:33:43 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.90_1) (envelope-from ) id 1jR74s-0006OG-DS for qemu-devel@nongnu.org; Wed, 22 Apr 2020 00:33:41 -0400 Received: from mail-pl1-x641.google.com ([2607:f8b0:4864:20::641]:46418) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1jR74r-0006JU-Ug for qemu-devel@nongnu.org; Wed, 22 Apr 2020 00:33:38 -0400 Received: by mail-pl1-x641.google.com with SMTP id n24so422250plp.13 for ; Tue, 21 Apr 2020 21:33:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=kAjJuC8E6QocqpgcZFzGlZ3iBSLxUs4KOHUk+pvyTWc=; b=mPea3epQd8fn6acRviE37rsYywYW8IdPLb5a2WnFJSrPd/POhHRq3OnVP4kFXelPdq kIhpiuAbJXRUAo00TPNNQVnBQDKvB+fFMcF3FpvW5rpE5orWST6aPx8APTmANdrrRAxo 6BQPUjJa+mHrN87y4ShMQGq4mCOu4kJ7iQLqyOFGOgvqSJxGDrPXkJBsRpKIiQ8xZi/i 8rrpUEDClG9VNZc5spxmH7/4YgauFGur5ndvanGvszDNp+NiiQMT76Rr3bNOyCQ0lDJ5 nwStRZChUjUzrYC0gS3zLwI7uhdN3WeYnSNAQ9mfu9wf3xLeOaMCMSn6spiakzvSCWJw Sepw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=kAjJuC8E6QocqpgcZFzGlZ3iBSLxUs4KOHUk+pvyTWc=; b=cYFf9lIEba/Tv95pCP+68rAMMq2KzUuXvX/aQy99lyrWh51RcQiQlHkWwfX/9rfEtf 0VYZeDwbDC8CxCzmIXsRacCx2SjgpqXyGsWLwm4L34SyldcFIlR6HkJkAffIcIAM3hwI LDmTixg2dLejk4HlwU4J8P69sMFokrFG/ctH8zYjWzmVGZZDsr/EVMS7Lk/285GWc69j ygWEGvrWeP32S2ZAuprjmcAMIp3np2p+EMx8Zlz7YKTOSbSC50qUfCELpgJFXVjGfDOQ h3ZNzC7UdjgI2H5sRv/cjQVZI5Ec6Pc4OwlhuYg3Q04qfWyk8ypVVbcoBZeXWmJhgGOU 3+BA== X-Gm-Message-State: AGi0PuYrouWl5aSrVMs2HaLRjorqCg8CqFcPVu6sWeO8618VkyP1dAth BqCaNZiU//L1r7ciOyJ6MGAzBhLju28= X-Received: by 2002:a17:90b:3110:: with SMTP id gc16mr9347810pjb.155.1587530015239; Tue, 21 Apr 2020 21:33:35 -0700 (PDT) Received: from localhost.localdomain (174-21-149-226.tukw.qwest.net. [174.21.149.226]) by smtp.gmail.com with ESMTPSA id l137sm4129613pfd.107.2020.04.21.21.33.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 21 Apr 2020 21:33:34 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Subject: [PATCH v3 18/18] target/arm: Remove sve_memopidx Date: Tue, 21 Apr 2020 21:33:09 -0700 Message-Id: <20200422043309.18430-19-richard.henderson@linaro.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200422043309.18430-1-richard.henderson@linaro.org> References: <20200422043309.18430-1-richard.henderson@linaro.org> MIME-Version: 1.0 Received-SPF: pass client-ip=2607:f8b0:4864:20::641; envelope-from=richard.henderson@linaro.org; helo=mail-pl1-x641.google.com X-detected-operating-system: by eggs.gnu.org: Error: [-] PROGRAM ABORT : Malformed IPv6 address (bad octet value). Location : parse_addr6(), p0f-client.c:67 X-Received-From: 2607:f8b0:4864:20::641 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: peter.maydell@linaro.org, qemu-arm@nongnu.org Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: "Qemu-devel" None of the sve helpers use TCGMemOpIdx any longer, so we can stop passing it. Reviewed-by: Peter Maydell Signed-off-by: Richard Henderson --- target/arm/internals.h | 5 ----- target/arm/sve_helper.c | 14 +++++++------- target/arm/translate-sve.c | 17 +++-------------- 3 files changed, 10 insertions(+), 26 deletions(-) -- 2.20.1 diff --git a/target/arm/internals.h b/target/arm/internals.h index e633aff36e..a833e3941d 100644 --- a/target/arm/internals.h +++ b/target/arm/internals.h @@ -979,11 +979,6 @@ static inline int arm_num_ctx_cmps(ARMCPU *cpu) } } -/* Note make_memop_idx reserves 4 bits for mmu_idx, and MO_BSWAP is bit 3. - * Thus a TCGMemOpIdx, without any MO_ALIGN bits, fits in 8 bits. - */ -#define MEMOPIDX_SHIFT 8 - /** * v7m_using_psp: Return true if using process stack pointer * Return true if the CPU is currently using the process stack diff --git a/target/arm/sve_helper.c b/target/arm/sve_helper.c index fffde4b6ec..f482fdd285 100644 --- a/target/arm/sve_helper.c +++ b/target/arm/sve_helper.c @@ -4440,7 +4440,7 @@ void sve_ldN_r(CPUARMState *env, uint64_t *vg, const target_ulong addr, sve_ldst1_host_fn *host_fn, sve_ldst1_tlb_fn *tlb_fn) { - const unsigned rd = extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT, 5); + const unsigned rd = simd_data(desc); const intptr_t reg_max = simd_oprsz(desc); intptr_t reg_off, reg_last, mem_off; SVEContLdSt info; @@ -4696,7 +4696,7 @@ void sve_ldnfff1_r(CPUARMState *env, void *vg, const target_ulong addr, sve_ldst1_host_fn *host_fn, sve_ldst1_tlb_fn *tlb_fn) { - const unsigned rd = extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT, 5); + const unsigned rd = simd_data(desc); void *vd = &env->vfp.zregs[rd]; const intptr_t reg_max = simd_oprsz(desc); intptr_t reg_off, mem_off, reg_last; @@ -4921,7 +4921,7 @@ void sve_stN_r(CPUARMState *env, uint64_t *vg, target_ulong addr, uint32_t desc, sve_ldst1_host_fn *host_fn, sve_ldst1_tlb_fn *tlb_fn) { - const unsigned rd = extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT, 5); + const unsigned rd = simd_data(desc); const intptr_t reg_max = simd_oprsz(desc); intptr_t reg_off, reg_last, mem_off; SVEContLdSt info; @@ -5127,9 +5127,9 @@ void sve_ld1_z(CPUARMState *env, void *vd, uint64_t *vg, void *vm, sve_ldst1_host_fn *host_fn, sve_ldst1_tlb_fn *tlb_fn) { - const int scale = extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT, 2); const int mmu_idx = cpu_mmu_index(env, false); const intptr_t reg_max = simd_oprsz(desc); + const int scale = simd_data(desc); ARMVectorReg scratch; intptr_t reg_off; SVEHostPage info, info2; @@ -5272,10 +5272,10 @@ void sve_ldff1_z(CPUARMState *env, void *vd, uint64_t *vg, void *vm, sve_ldst1_tlb_fn *tlb_fn) { const int mmu_idx = cpu_mmu_index(env, false); - const int scale = extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT, 2); + const intptr_t reg_max = simd_oprsz(desc); + const int scale = simd_data(desc); const int esize = 1 << esz; const int msize = 1 << msz; - const intptr_t reg_max = simd_oprsz(desc); intptr_t reg_off; SVEHostPage info; target_ulong addr, in_page; @@ -5426,9 +5426,9 @@ void sve_st1_z(CPUARMState *env, void *vd, uint64_t *vg, void *vm, sve_ldst1_host_fn *host_fn, sve_ldst1_tlb_fn *tlb_fn) { - const int scale = extract32(desc, SIMD_DATA_SHIFT + MEMOPIDX_SHIFT, 2); const int mmu_idx = cpu_mmu_index(env, false); const intptr_t reg_max = simd_oprsz(desc); + const int scale = simd_data(desc); void *host[ARM_MAX_VQ * 4]; intptr_t reg_off, i; SVEHostPage info, info2; diff --git a/target/arm/translate-sve.c b/target/arm/translate-sve.c index b35bad245e..7bd7de80e6 100644 --- a/target/arm/translate-sve.c +++ b/target/arm/translate-sve.c @@ -4584,11 +4584,6 @@ static const uint8_t dtype_esz[16] = { 3, 2, 1, 3 }; -static TCGMemOpIdx sve_memopidx(DisasContext *s, int dtype) -{ - return make_memop_idx(s->be_data | dtype_mop[dtype], get_mem_index(s)); -} - static void do_mem_zpa(DisasContext *s, int zt, int pg, TCGv_i64 addr, int dtype, gen_helper_gvec_mem *fn) { @@ -4601,9 +4596,7 @@ static void do_mem_zpa(DisasContext *s, int zt, int pg, TCGv_i64 addr, * registers as pointers, so encode the regno into the data field. * For consistency, do this even for LD1. */ - desc = sve_memopidx(s, dtype); - desc |= zt << MEMOPIDX_SHIFT; - desc = simd_desc(vsz, vsz, desc); + desc = simd_desc(vsz, vsz, zt); t_desc = tcg_const_i32(desc); t_pg = tcg_temp_new_ptr(); @@ -4835,9 +4828,7 @@ static void do_ldrq(DisasContext *s, int zt, int pg, TCGv_i64 addr, int msz) int desc, poff; /* Load the first quadword using the normal predicated load helpers. */ - desc = sve_memopidx(s, msz_dtype(s, msz)); - desc |= zt << MEMOPIDX_SHIFT; - desc = simd_desc(16, 16, desc); + desc = simd_desc(16, 16, zt); t_desc = tcg_const_i32(desc); poff = pred_full_reg_offset(s, pg); @@ -5066,9 +5057,7 @@ static void do_mem_zpz(DisasContext *s, int zt, int pg, int zm, TCGv_i32 t_desc; int desc; - desc = sve_memopidx(s, msz_dtype(s, msz)); - desc |= scale << MEMOPIDX_SHIFT; - desc = simd_desc(vsz, vsz, desc); + desc = simd_desc(vsz, vsz, scale); t_desc = tcg_const_i32(desc); tcg_gen_addi_ptr(t_pg, cpu_env, pred_full_reg_offset(s, pg));