From patchwork Wed Apr 3 03:43:58 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 161674 Delivered-To: patch@linaro.org Received: by 2002:a02:c6d8:0:0:0:0:0 with SMTP id r24csp2523448jan; Tue, 2 Apr 2019 21:04:13 -0700 (PDT) X-Google-Smtp-Source: APXvYqyjNpKVN3LzPYFIkDnB0hFg/GciiW2gNg5RRIazbf2TBK1C7tyd2RNi3+ANIr+jDuL4bjTd X-Received: by 2002:a81:234a:: with SMTP id j71mr64193918ywj.352.1554264253306; Tue, 02 Apr 2019 21:04:13 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1554264253; cv=none; d=google.com; s=arc-20160816; b=tY5MkNVO6c5Sru1M1Ug4ludwcV1XN0q5QcmH5L1Y8Jqpuqut5kSp7C7EX5yZkHeGsO EdYQjNSJ78qFyUaLdM0H+8wt0zDT3HI/oo58/NKumow5DpxXwAmVUhHlvBwZKpaTRj2y P/Vps4zC2kXwrU3nOJxsuFF2QCGVDB0RcgoSJzPpmm999rue32zdv8bY+OUqmKB3eCK+ gK7zrBC9p38huj+ZDoAOWhv7woMlvgL2pNrHDlPn9tj3rQjIGfdgNMiHGiBQPHghonW1 uSb9BtnOQpAV8WqZBLQooPF1pg4duPOEclscNIfYfBRqFnUZNzg3MACSRQe79fBTeMib 27fg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:subject:references:in-reply-to :message-id:date:to:from:dkim-signature; bh=eSmixxW4uQkmpintF6uPGofmBAlQammZR2m74ngSc30=; b=ZLfQPIT7kQTlcm5nnysdQkXESdL3JEBZN0NbcU680BL3DETrbz5hTJp8K9ng71Oh9F zV35YUINiHKgmaPN50SglxZ7D5U3lGSYAJgpMNTka8VrbciHWvuqZHR89iY3RqUM3Tuh vK3sl8MrBISI1XVN2AHUYMbBtYRqCg5AjUuNz1zsueB+8L1r6MEB55q6iyGlOdFBA54d G/kcTjVV/5Bjy9ALYonDtqMns4VvVyf5ne67Ci9xSHqZ+PXABppHcFL2TkSUj2BSVR1T wEV5dytIm5P4iooPAYZbv2JwOsNiGU3Ydwt8fOmKJeX//SfQAMcxkCPiEFJfFCYzjapE WrSw== ARC-Authentication-Results: i=1; mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=AgoLDyYK; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [209.51.188.17]) by mx.google.com with ESMTPS id q128si9749391ywb.51.2019.04.02.21.04.13 for (version=TLS1 cipher=AES128-SHA bits=128/128); Tue, 02 Apr 2019 21:04:13 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; Authentication-Results: mx.google.com; dkim=fail header.i=@linaro.org header.s=google header.b=AgoLDyYK; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([127.0.0.1]:56274 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hBX8G-0002jh-Qc for patch@linaro.org; Wed, 03 Apr 2019 00:04:12 -0400 Received: from eggs.gnu.org ([209.51.188.92]:48904) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hBWpe-0003Ky-2U for qemu-devel@nongnu.org; Tue, 02 Apr 2019 23:44:59 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1hBWpc-0000VU-Ib for qemu-devel@nongnu.org; Tue, 02 Apr 2019 23:44:58 -0400 Received: from mail-pg1-x544.google.com ([2607:f8b0:4864:20::544]:38823) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1hBWpc-0000Uk-8H for qemu-devel@nongnu.org; Tue, 02 Apr 2019 23:44:56 -0400 Received: by mail-pg1-x544.google.com with SMTP id j26so7590224pgl.5 for ; Tue, 02 Apr 2019 20:44:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:subject:date:message-id:in-reply-to:references; bh=eSmixxW4uQkmpintF6uPGofmBAlQammZR2m74ngSc30=; b=AgoLDyYKfXcmkyqBht0yChfxRyCqfM+thujBfEG5aQVxMroKMgJsoM9rffGJCb7675 Wl8X+h7ryvSLyI+PfGWX7W0Ey5bTWFxk/vvz77ae6l45FzPodxX4Y0D4zfvHv1telFxF cg57BW5P5Mq5hdghJxIxvp9UqYE1fqyByRmS322M22ikJRBvA++8U/yNImrq0x+ihyGK D9z/TjypT64NZrn5aSkMkLOzdd2ce8qEZPqfgPg1Mb6btHSHbKKjEkkAVQoba8yjM48w gGQbZQIE/T5FPVHYjIZWxDJ+Vmqo4WUeRP0MCQo7WheHWMTIJTXafzQjUN+/ct/6NiRc RkXA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references; bh=eSmixxW4uQkmpintF6uPGofmBAlQammZR2m74ngSc30=; b=g6iMz6l3xh+QJZajdvJyB8keB0kKae0vpYXybHX1Zsh8yO+8xosfgG9RNG9aqelNnK nqFcdJi9VB7lWiVr56pRa+koEWz+sRVFrYx/9f0wrj3oha08bxu2NRBOvQjMsTfivvdB XLKkRcM6iE2jUME1xkFb0PxaEt7Z7N8qCLLzCyAZTyY8lI3d6y97nsp9qB8bvCsF511D 9JPvph/pVuaM07gQjw+jJHPargcrFN4x/n16BlAEdhRZaUK4eRmNAJnMM51/1IeSItm+ It3r9FxguMjr2TEx+aoTIBzgv47mNPBAmVAxPIm+GROZXQH+f5ExnBM30Q0QXD2Lk/1A oTog== X-Gm-Message-State: APjAAAUqgCBwS8XZ92V5m3s5wr+wqAOt/n4R5uPSctWQvYekY+P74h8x J3nGblOI1zCy4m+E0gkpnCHbi02+ev9b9Q== X-Received: by 2002:a62:62c3:: with SMTP id w186mr41924483pfb.73.1554263094953; Tue, 02 Apr 2019 20:44:54 -0700 (PDT) Received: from cloudburst.imgcgcw.net ([147.50.13.10]) by smtp.gmail.com with ESMTPSA id z6sm26753214pgo.31.2019.04.02.20.44.53 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 02 Apr 2019 20:44:54 -0700 (PDT) From: Richard Henderson To: qemu-devel@nongnu.org Date: Wed, 3 Apr 2019 10:43:58 +0700 Message-Id: <20190403034358.21999-27-richard.henderson@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190403034358.21999-1-richard.henderson@linaro.org> References: <20190403034358.21999-1-richard.henderson@linaro.org> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::544 Subject: [Qemu-devel] [PATCH 26/26] tcg: Use tlb_fill probe from tlb_vaddr_to_host X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: "Qemu-devel" Most of the existing users would continue around a loop which would fault the tlb entry in via a normal load/store. But for SVE we have a true non-faulting case which requires the new probing form of tlb_fill. Signed-off-by: Richard Henderson --- include/exec/cpu_ldst.h | 40 ++++-------------------- accel/tcg/cputlb.c | 69 ++++++++++++++++++++++++++++++++++++----- target/arm/sve_helper.c | 6 +--- 3 files changed, 68 insertions(+), 47 deletions(-) -- 2.17.1 Reviewed-by: Peter Maydell diff --git a/include/exec/cpu_ldst.h b/include/exec/cpu_ldst.h index d78041d7a0..be8c3f4da2 100644 --- a/include/exec/cpu_ldst.h +++ b/include/exec/cpu_ldst.h @@ -440,43 +440,15 @@ static inline CPUTLBEntry *tlb_entry(CPUArchState *env, uintptr_t mmu_idx, * This is the equivalent of the initial fast-path code used by * TCG backends for guest load and store accesses. */ +#ifdef CONFIG_USER_ONLY static inline void *tlb_vaddr_to_host(CPUArchState *env, abi_ptr addr, - int access_type, int mmu_idx) + MMUAccessType access_type, int mmu_idx) { -#if defined(CONFIG_USER_ONLY) return g2h(addr); -#else - CPUTLBEntry *tlbentry = tlb_entry(env, mmu_idx, addr); - abi_ptr tlb_addr; - uintptr_t haddr; - - switch (access_type) { - case 0: - tlb_addr = tlbentry->addr_read; - break; - case 1: - tlb_addr = tlb_addr_write(tlbentry); - break; - case 2: - tlb_addr = tlbentry->addr_code; - break; - default: - g_assert_not_reached(); - } - - if (!tlb_hit(tlb_addr, addr)) { - /* TLB entry is for a different page */ - return NULL; - } - - if (tlb_addr & ~TARGET_PAGE_MASK) { - /* IO access */ - return NULL; - } - - haddr = addr + tlbentry->addend; - return (void *)haddr; -#endif /* defined(CONFIG_USER_ONLY) */ } +#else +void *tlb_vaddr_to_host(CPUArchState *env, abi_ptr addr, + MMUAccessType access_type, int mmu_idx); +#endif #endif /* CPU_LDST_H */ diff --git a/accel/tcg/cputlb.c b/accel/tcg/cputlb.c index 7f59d815db..959b6d4ded 100644 --- a/accel/tcg/cputlb.c +++ b/accel/tcg/cputlb.c @@ -1006,6 +1006,16 @@ static void io_writex(CPUArchState *env, CPUIOTLBEntry *iotlbentry, } } +static inline target_ulong tlb_read_ofs(CPUTLBEntry *entry, size_t ofs) +{ +#if TCG_OVERSIZED_GUEST + return *(target_ulong *)((uintptr_t)entry + ofs); +#else + /* ofs might correspond to .addr_write, so use atomic_read */ + return atomic_read((target_ulong *)((uintptr_t)entry + ofs)); +#endif +} + /* Return true if ADDR is present in the victim tlb, and has been copied back to the main tlb. */ static bool victim_tlb_hit(CPUArchState *env, size_t mmu_idx, size_t index, @@ -1016,14 +1026,7 @@ static bool victim_tlb_hit(CPUArchState *env, size_t mmu_idx, size_t index, assert_cpu_is_self(ENV_GET_CPU(env)); for (vidx = 0; vidx < CPU_VTLB_SIZE; ++vidx) { CPUTLBEntry *vtlb = &env->tlb_v_table[mmu_idx][vidx]; - target_ulong cmp; - - /* elt_ofs might correspond to .addr_write, so use atomic_read */ -#if TCG_OVERSIZED_GUEST - cmp = *(target_ulong *)((uintptr_t)vtlb + elt_ofs); -#else - cmp = atomic_read((target_ulong *)((uintptr_t)vtlb + elt_ofs)); -#endif + target_ulong cmp = tlb_read_ofs(vtlb, elt_ofs); if (cmp == page) { /* Found entry in victim tlb, swap tlb and iotlb. */ @@ -1107,6 +1110,56 @@ void probe_write(CPUArchState *env, target_ulong addr, int size, int mmu_idx, } } +void *tlb_vaddr_to_host(CPUArchState *env, abi_ptr addr, + MMUAccessType access_type, int mmu_idx) +{ + CPUTLBEntry *entry = tlb_entry(env, mmu_idx, addr); + uintptr_t tlb_addr, page; + size_t elt_ofs; + + switch (access_type) { + case MMU_DATA_LOAD: + elt_ofs = offsetof(CPUTLBEntry, addr_read); + break; + case MMU_DATA_STORE: + elt_ofs = offsetof(CPUTLBEntry, addr_write); + break; + case MMU_INST_FETCH: + elt_ofs = offsetof(CPUTLBEntry, addr_code); + break; + default: + g_assert_not_reached(); + } + + page = addr & TARGET_PAGE_MASK; + tlb_addr = tlb_read_ofs(entry, elt_ofs); + + if (!tlb_hit_page(tlb_addr, page)) { + uintptr_t index = tlb_index(env, mmu_idx, addr); + + if (!victim_tlb_hit(env, mmu_idx, index, elt_ofs, page)) { + CPUState *cs = ENV_GET_CPU(env); + CPUClass *cc = CPU_GET_CLASS(cs); + + if (!cc->tlb_fill(cs, addr, 0, access_type, mmu_idx, true, 0)) { + /* Non-faulting page table read failed. */ + return NULL; + } + + /* TLB resize via tlb_fill may have moved the entry. */ + entry = tlb_entry(env, mmu_idx, addr); + } + tlb_addr = tlb_read_ofs(entry, elt_ofs); + } + + if (tlb_addr & ~TARGET_PAGE_MASK) { + /* IO access */ + return NULL; + } + + return (void *)(addr + entry->addend); +} + /* Probe for a read-modify-write atomic operation. Do not allow unaligned * operations, or io operations to proceed. Return the host address. */ static void *atomic_mmu_lookup(CPUArchState *env, target_ulong addr, diff --git a/target/arm/sve_helper.c b/target/arm/sve_helper.c index bc847250dd..fd434c66ea 100644 --- a/target/arm/sve_helper.c +++ b/target/arm/sve_helper.c @@ -4598,11 +4598,7 @@ static void sve_ldnf1_r(CPUARMState *env, void *vg, const target_ulong addr, * in the real world, obviously.) * * Then there are the annoying special cases with watchpoints... - * - * TODO: Add a form of tlb_fill that does not raise an exception, - * with a form of tlb_vaddr_to_host and a set of loads to match. - * The non_fault_vaddr_to_host would handle everything, usually, - * and the loads would handle the iomem path for watchpoints. + * TODO: Add a form of non-faulting loads using cc->tlb_fill(probe=true). */ host = tlb_vaddr_to_host(env, addr + mem_off, MMU_DATA_LOAD, mmu_idx); split = max_for_page(addr, mem_off, mem_max);