From patchwork Tue Oct 3 03:10:45 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jintack Lim X-Patchwork-Id: 114642 Delivered-To: patch@linaro.org Received: by 10.140.22.163 with SMTP id 32csp1384073qgn; Mon, 2 Oct 2017 20:11:29 -0700 (PDT) X-Received: by 10.99.160.71 with SMTP id u7mr7077644pgn.22.1507000288974; Mon, 02 Oct 2017 20:11:28 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1507000288; cv=none; d=google.com; s=arc-20160816; b=WGyXJVueFvNHyVNt+Rc4ebmt67k7yNQq53Nz3u48JBhDFK0yv3klBjyBO0TuslGYEo SLeDeKXhjPPpxldPgUQnKtWIxbdAFFUMeI/cU5xa3hFJHd8uM11RnCkz133P5qIhXhyL J4iRBxGPMj/rFD8Ig7aPWiSYxsan8wPrdBdXiJelkdv62UFggsfQKRFP51HBPh2yjgWH R2dmn4XEhGEIpf52IBVmqVDM6Pid/yCpU6TuWzklEpYpmz5LFveGHgAoB0SioZ34FDSw grsHIHDsgr8MhLWtEwZu4g9Nk0R48Z4DrkWwdORkXcZkVQa4bNUYy+hl7H/3nSTIhcIo 3G/A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:message-id:date:subject:cc:to:from :dkim-signature:arc-authentication-results; bh=OEkTNtQgNIwDEjdGAErVSDgoMPhru2aCaJh1dsrpkW4=; b=bvCZ0XLwVjTjBh9h6no7rSOUJYLW7y7n9iylvZUr0gb662Wf7sl/MEvwWH461SnmEI /TOufu/ywIFomZVnCOFb5Ezivz5O4PCwp+Q4YO0EOPDqdmW0DDIxv3G61IJbVDsBGNwT cQFdtGoWcBgthuRlS3O0d9kqbEcTiSTOxlhRbxK/+cFD1zA97239kcG6YeBklIqscKgK li7LKbULfoVWKPNnUoksZ2b2ZRAzgMMONrxJ2ds2kfP6f9KVLmz3vGhqxkFMafDDFrQV GSTCHMdiANM3r4vXxePto+cTRYkS7QnczMI/H+gNOMFYiuytJ2Vn3/3yuVDfWXFNeMJW rbDQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=IlGSazM2; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id p11si232867pfl.538.2017.10.02.20.11.28; Mon, 02 Oct 2017 20:11:28 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=IlGSazM2; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751534AbdJCDL0 (ORCPT + 26 others); Mon, 2 Oct 2017 23:11:26 -0400 Received: from mail-it0-f49.google.com ([209.85.214.49]:55264 "EHLO mail-it0-f49.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751158AbdJCDLY (ORCPT ); Mon, 2 Oct 2017 23:11:24 -0400 Received: by mail-it0-f49.google.com with SMTP id e134so9711369ite.3 for ; Mon, 02 Oct 2017 20:11:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id; bh=OEkTNtQgNIwDEjdGAErVSDgoMPhru2aCaJh1dsrpkW4=; b=IlGSazM239ku5xv/YG5RQa7fk83k6tjgT59XcQ9iF/b42LG8+CEmexAXmDP118jNiq AU1JBDbZdD2GEQji/zi7wIw7vlRPLCBDYtYNDHxmJHj32aAsOEpg88dRdEnzCMWoFZ6+ 2VgHNehoBFF+lDyxcP5hnohcOJurS1sM+lg7k= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id; bh=OEkTNtQgNIwDEjdGAErVSDgoMPhru2aCaJh1dsrpkW4=; b=p6zrjOK3KdyqrCN/Bfk4iZ3t1g88cibQ73itbieFzishkXQrgVMHeYUn8IOPmL0NDc 1W5NC+6IJVM7PVYFqFX1L8lSkFc1puysF6UGDjlhRx9FEkR82YeHYXGQ4Ooj4d4YLQ8v x9FfvvjPs2XmdDIHogm8KibwQt+3gzRb7UeQru+hsI7ftggVgaVMpdnirGZdcU5q5roA eqA6bX5mejxki31No81gQjt0hz6d/6NN0hasm/FJyxcr5uguWLY4089uDT5MjOQd+ovH 215QkUKNWQRCOQswYTvnsWTcdAV+cfgpD+aGzPLjPGQETqyVm/y/UPmeMtmyWiZX8O03 VTOA== X-Gm-Message-State: AMCzsaXDxy6MAy1QxdBz1RoKR9B2DbbC9URggLhw7E+fnvc1PvJLyk3y TVwEdY6bM5/7NGpfPivmelmUGQ== X-Google-Smtp-Source: AOwi7QD4PP1OfYOEPk6RecY7Dn05QoWYdJa396LI9ir2z+r/hV9YAeAyiE++cGiLdDzHU0GsvW/aUA== X-Received: by 10.36.14.200 with SMTP id 191mr22472184ite.133.1507000284307; Mon, 02 Oct 2017 20:11:24 -0700 (PDT) Received: from node.jintackl-qv28633.kvmarm-pg0.wisc.cloudlab.us (c220g1-031126.wisc.cloudlab.us. [128.104.222.76]) by smtp.gmail.com with ESMTPSA id h84sm5367193iod.72.2017.10.02.20.11.23 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Mon, 02 Oct 2017 20:11:23 -0700 (PDT) From: Jintack Lim To: christoffer.dall@linaro.org, marc.zyngier@arm.com, kvmarm@lists.cs.columbia.edu Cc: jintack@cs.columbia.edu, pbonzini@redhat.com, rkrcmar@redhat.com, catalin.marinas@arm.com, will.deacon@arm.com, linux@armlinux.org.uk, mark.rutland@arm.com, linux-arm-kernel@lists.infradead.org, kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Jintack Lim Subject: [RFC PATCH v2 03/31] KVM: arm/arm64: Remove unused params in mmu functions Date: Mon, 2 Oct 2017 22:10:45 -0500 Message-Id: <1507000273-3735-1-git-send-email-jintack.lim@linaro.org> X-Mailer: git-send-email 1.9.1 Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org From: Christoffer Dall stage2_flush_xxx functions take a pointer to the kvm struct as the first parameter but they are never used. Clean this up before modifying mmu code for nested virtualization support. Signed-off-by: Christoffer Dall Signed-off-by: Jintack Lim --- virt/kvm/arm/mmu.c | 15 ++++++--------- 1 file changed, 6 insertions(+), 9 deletions(-) -- 1.9.1 diff --git a/virt/kvm/arm/mmu.c b/virt/kvm/arm/mmu.c index f2d5b6c..0a5f5ca 100644 --- a/virt/kvm/arm/mmu.c +++ b/virt/kvm/arm/mmu.c @@ -315,8 +315,7 @@ static void unmap_stage2_range(struct kvm *kvm, phys_addr_t start, u64 size) } while (pgd++, addr = next, addr != end); } -static void stage2_flush_ptes(struct kvm *kvm, pmd_t *pmd, - phys_addr_t addr, phys_addr_t end) +static void stage2_flush_ptes(pmd_t *pmd, phys_addr_t addr, phys_addr_t end) { pte_t *pte; @@ -327,8 +326,7 @@ static void stage2_flush_ptes(struct kvm *kvm, pmd_t *pmd, } while (pte++, addr += PAGE_SIZE, addr != end); } -static void stage2_flush_pmds(struct kvm *kvm, pud_t *pud, - phys_addr_t addr, phys_addr_t end) +static void stage2_flush_pmds(pud_t *pud, phys_addr_t addr, phys_addr_t end) { pmd_t *pmd; phys_addr_t next; @@ -340,13 +338,12 @@ static void stage2_flush_pmds(struct kvm *kvm, pud_t *pud, if (pmd_thp_or_huge(*pmd)) kvm_flush_dcache_pmd(*pmd); else - stage2_flush_ptes(kvm, pmd, addr, next); + stage2_flush_ptes(pmd, addr, next); } } while (pmd++, addr = next, addr != end); } -static void stage2_flush_puds(struct kvm *kvm, pgd_t *pgd, - phys_addr_t addr, phys_addr_t end) +static void stage2_flush_puds(pgd_t *pgd, phys_addr_t addr, phys_addr_t end) { pud_t *pud; phys_addr_t next; @@ -358,7 +355,7 @@ static void stage2_flush_puds(struct kvm *kvm, pgd_t *pgd, if (stage2_pud_huge(*pud)) kvm_flush_dcache_pud(*pud); else - stage2_flush_pmds(kvm, pud, addr, next); + stage2_flush_pmds(pud, addr, next); } } while (pud++, addr = next, addr != end); } @@ -374,7 +371,7 @@ static void stage2_flush_memslot(struct kvm *kvm, pgd = kvm->arch.pgd + stage2_pgd_index(addr); do { next = stage2_pgd_addr_end(addr, end); - stage2_flush_puds(kvm, pgd, addr, next); + stage2_flush_puds(pgd, addr, next); } while (pgd++, addr = next, addr != end); } From patchwork Tue Oct 3 03:10:46 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jintack Lim X-Patchwork-Id: 114652 Delivered-To: patch@linaro.org Received: by 10.140.22.163 with SMTP id 32csp1389414qgn; Mon, 2 Oct 2017 20:19:08 -0700 (PDT) X-Received: by 10.84.252.143 with SMTP id y15mr5623454pll.80.1507000748006; Mon, 02 Oct 2017 20:19:08 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1507000747; cv=none; d=google.com; s=arc-20160816; b=jdlASBE39Yd3ekHbosioP7F+y2Xs5TmDt7AKhNshHUylG+9lTt+kG0CvOdXVGr9rvv mIRr5AUcYmbWte/5KMqESgM+EXmpl8iP0ZbZ3ohYo8VuV82hM5JfhRwvwxmAjCU8uXRx dZxwaR3p8w5oqOJTlZBta03NBfIQ53CxX76L+ZEk0DmW2SQryYk/JTU2NLGGQ87p9RRw wfAgpzgSUIn3QE9OyKy36yTQkXWx3BJXEp8+KGF4mn5st0ajWk/Jf/sWzTlscrFKGsAD 0Ilxoxali1aMg3zCxLHXdOmLgFymLuNyVU5m8vX+PhiUunLQmpNeHtJ1E9gHPDKnHc9y QTbQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature:arc-authentication-results; bh=Nr8sDV6nQiJGavOYM7VW9hXtKu1vTPVRjeDWk/MOEEA=; b=X+AA4UQukvq2EJgkdnxHfSvsJDf1l7kQgOHt81RXNjnc9sBgj7Q2wdPZA/8AcjISJk DFbv9LRVt0cx2kjXQq1SShU8X/OQ47TFh4uutTM9gGfHA0z3XpBct/9wGNLJGVo1QLAo 2j75/3Iu1SPHkceScPetAFzvaJHTUTglJioML8i6NMnGqXVgA4YkBvtyd9qm9u45f0sT HJctV788KDPS7KivjhD3x9gcTXjoJJQBYONagEy2wYDSPl8iveCe170Gyb2Hz0g+VAkZ 1ErsVZpvJRyNCoFdHWDf+sNBZOLxglNDgww3G2Vis/kfZlfsdfhcsG7JZkuHcHIY0429 O45g== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=Ot8v+n40; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id v2si8920935plp.87.2017.10.02.20.19.07; Mon, 02 Oct 2017 20:19:07 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=Ot8v+n40; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751776AbdJCDLp (ORCPT + 26 others); Mon, 2 Oct 2017 23:11:45 -0400 Received: from mail-io0-f175.google.com ([209.85.223.175]:49204 "EHLO mail-io0-f175.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751478AbdJCDLm (ORCPT ); Mon, 2 Oct 2017 23:11:42 -0400 Received: by mail-io0-f175.google.com with SMTP id 21so6448095iof.6 for ; Mon, 02 Oct 2017 20:11:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=Nr8sDV6nQiJGavOYM7VW9hXtKu1vTPVRjeDWk/MOEEA=; b=Ot8v+n40N3LhxU8F7z7UrmY4T/mlfU+QdJrhMGHxJAcMdDfBzSCdIFE6kHwy8TsyjU ALUoEtLQI+5XrSBaXe8f/qrss17C7aB1BceqdyooBmVuUFWF1iV49m+COiW+qHNWLAi+ izsD7qYiB9rc1PFwO+JG9pMHDGq/por8+wkq4= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=Nr8sDV6nQiJGavOYM7VW9hXtKu1vTPVRjeDWk/MOEEA=; b=qshTQJR13auI9UOf/WR73xUCcIltG2buhuTrFwGx2jF8K9P40iw7N2C83w4bW093bW fE4xlGy732p91gf+Uf1DRw3o2rwwNewmbX5xS2j9DAHmOOgIkd68gKocl+VoiWw1kyMM KVfgLZb/3ybt5mw09N6abGT4UB55QDQ1cYtBGW0HPfoE/MA6C4vkEj48TyZ0hoewaDma fSPXMcTwmVXa4lXgJ0a/cyHE15D1QDVm9Jh6tEb9CBspzLAgPIgKjB0xIyTaKE6/ZINW y+0fTgiL7hBJELR4NZ0HaBkA1SCqa6H9guoCrfujIv1C2v0UUJnjk8DYDXyKNLSGA3al vwAg== X-Gm-Message-State: AMCzsaWyggm1aj1GxmlTXVGFWJO5bPJY34PjIm+1En1LOotBtDX0KXDJ dq7EYj+njEbsNF8V28UsPq+VSw== X-Google-Smtp-Source: AOwi7QAjlCsycDl1Vd0YYu9k+6ikCZdZwv3NR/etmFSWjN3k3wUSL7W7rig0+GCzZIztPyE8Dvwyew== X-Received: by 10.107.173.34 with SMTP id w34mr27554716ioe.14.1507000300915; Mon, 02 Oct 2017 20:11:40 -0700 (PDT) Received: from node.jintackl-qv28633.kvmarm-pg0.wisc.cloudlab.us (c220g1-031126.wisc.cloudlab.us. [128.104.222.76]) by smtp.gmail.com with ESMTPSA id h84sm5367193iod.72.2017.10.02.20.11.39 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Mon, 02 Oct 2017 20:11:40 -0700 (PDT) From: Jintack Lim To: christoffer.dall@linaro.org, marc.zyngier@arm.com, kvmarm@lists.cs.columbia.edu Cc: jintack@cs.columbia.edu, pbonzini@redhat.com, rkrcmar@redhat.com, catalin.marinas@arm.com, will.deacon@arm.com, linux@armlinux.org.uk, mark.rutland@arm.com, linux-arm-kernel@lists.infradead.org, kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Jintack Lim Subject: [RFC PATCH v2 04/31] KVM: arm/arm64: Abstract stage-2 MMU state into a separate structure Date: Mon, 2 Oct 2017 22:10:46 -0500 Message-Id: <1507000273-3735-2-git-send-email-jintack.lim@linaro.org> X-Mailer: git-send-email 1.9.1 In-Reply-To: <1507000273-3735-1-git-send-email-jintack.lim@linaro.org> References: <1507000273-3735-1-git-send-email-jintack.lim@linaro.org> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org From: Christoffer Dall Abstract stage-2 MMU state into a separate structure and change all callers referring to page tables, VMIDs, and the VTTBR to use this new indirection. This is about to become very handy when using shadow stage-2 page tables. Signed-off-by: Christoffer Dall Signed-off-by: Jintack Lim --- arch/arm/include/asm/kvm_asm.h | 7 +- arch/arm/include/asm/kvm_host.h | 26 +++++--- arch/arm/kvm/hyp/switch.c | 5 +- arch/arm/kvm/hyp/tlb.c | 18 ++--- arch/arm64/include/asm/kvm_asm.h | 7 +- arch/arm64/include/asm/kvm_host.h | 10 ++- arch/arm64/kvm/hyp/switch.c | 5 +- arch/arm64/kvm/hyp/tlb.c | 38 +++++------ virt/kvm/arm/arm.c | 34 +++++----- virt/kvm/arm/mmu.c | 137 +++++++++++++++++++++----------------- 10 files changed, 163 insertions(+), 124 deletions(-) -- 1.9.1 diff --git a/arch/arm/include/asm/kvm_asm.h b/arch/arm/include/asm/kvm_asm.h index 14d68a4..71b7255 100644 --- a/arch/arm/include/asm/kvm_asm.h +++ b/arch/arm/include/asm/kvm_asm.h @@ -57,6 +57,7 @@ #ifndef __ASSEMBLY__ struct kvm; struct kvm_vcpu; +struct kvm_s2_mmu; extern char __kvm_hyp_init[]; extern char __kvm_hyp_init_end[]; @@ -64,9 +65,9 @@ extern char __kvm_hyp_vector[]; extern void __kvm_flush_vm_context(void); -extern void __kvm_tlb_flush_vmid_ipa(struct kvm *kvm, phys_addr_t ipa); -extern void __kvm_tlb_flush_vmid(struct kvm *kvm); -extern void __kvm_tlb_flush_local_vmid(struct kvm_vcpu *vcpu); +extern void __kvm_tlb_flush_vmid_ipa(struct kvm_s2_mmu *mmu, phys_addr_t ipa); +extern void __kvm_tlb_flush_vmid(struct kvm_s2_mmu *mmu); +extern void __kvm_tlb_flush_local_vmid(struct kvm_s2_mmu *mmu); extern int __kvm_vcpu_run(struct kvm_vcpu *vcpu); diff --git a/arch/arm/include/asm/kvm_host.h b/arch/arm/include/asm/kvm_host.h index 7e9e6c8..78d826e 100644 --- a/arch/arm/include/asm/kvm_host.h +++ b/arch/arm/include/asm/kvm_host.h @@ -53,9 +53,21 @@ int kvm_reset_vcpu(struct kvm_vcpu *vcpu); void kvm_reset_coprocs(struct kvm_vcpu *vcpu); -struct kvm_arch { - /* VTTBR value associated with below pgd and vmid */ +struct kvm_s2_mmu { + /* The VMID generation used for the virt. memory system */ + u64 vmid_gen; + u32 vmid; + + /* Stage-2 page table */ + pgd_t *pgd; + + /* VTTBR value associated with above pgd and vmid */ u64 vttbr; +}; + +struct kvm_arch { + /* Stage 2 paging state for the VM */ + struct kvm_s2_mmu mmu; /* The last vcpu id that ran on each physical CPU */ int __percpu *last_vcpu_ran; @@ -65,13 +77,6 @@ struct kvm_arch { * here. */ - /* The VMID generation used for the virt. memory system */ - u64 vmid_gen; - u32 vmid; - - /* Stage-2 page table */ - pgd_t *pgd; - /* Interrupt controller */ struct vgic_dist vgic; int max_vcpus; @@ -185,6 +190,9 @@ struct kvm_vcpu_arch { /* Detect first run of a vcpu */ bool has_run_once; + + /* Stage 2 paging state used by the hardware on next switch */ + struct kvm_s2_mmu *hw_mmu; }; struct kvm_vm_stat { diff --git a/arch/arm/kvm/hyp/switch.c b/arch/arm/kvm/hyp/switch.c index ebd2dd4..4814671 100644 --- a/arch/arm/kvm/hyp/switch.c +++ b/arch/arm/kvm/hyp/switch.c @@ -75,8 +75,9 @@ static void __hyp_text __deactivate_traps(struct kvm_vcpu *vcpu) static void __hyp_text __activate_vm(struct kvm_vcpu *vcpu) { - struct kvm *kvm = kern_hyp_va(vcpu->kvm); - write_sysreg(kvm->arch.vttbr, VTTBR); + struct kvm_s2_mmu *mmu = kern_hyp_va(vcpu->arch.hw_mmu); + + write_sysreg(mmu->vttbr, VTTBR); write_sysreg(vcpu->arch.midr, VPIDR); } diff --git a/arch/arm/kvm/hyp/tlb.c b/arch/arm/kvm/hyp/tlb.c index 6d810af..56f0a49 100644 --- a/arch/arm/kvm/hyp/tlb.c +++ b/arch/arm/kvm/hyp/tlb.c @@ -34,13 +34,13 @@ * As v7 does not support flushing per IPA, just nuke the whole TLB * instead, ignoring the ipa value. */ -void __hyp_text __kvm_tlb_flush_vmid(struct kvm *kvm) +void __hyp_text __kvm_tlb_flush_vmid(struct kvm_s2_mmu *mmu) { dsb(ishst); /* Switch to requested VMID */ - kvm = kern_hyp_va(kvm); - write_sysreg(kvm->arch.vttbr, VTTBR); + mmu = kern_hyp_va(mmu); + write_sysreg(mmu->vttbr, VTTBR); isb(); write_sysreg(0, TLBIALLIS); @@ -50,17 +50,17 @@ void __hyp_text __kvm_tlb_flush_vmid(struct kvm *kvm) write_sysreg(0, VTTBR); } -void __hyp_text __kvm_tlb_flush_vmid_ipa(struct kvm *kvm, phys_addr_t ipa) +void __hyp_text __kvm_tlb_flush_vmid_ipa(struct kvm_s2_mmu *mmu, + phys_addr_t ipa) { - __kvm_tlb_flush_vmid(kvm); + __kvm_tlb_flush_vmid(mmu); } -void __hyp_text __kvm_tlb_flush_local_vmid(struct kvm_vcpu *vcpu) +void __hyp_text __kvm_tlb_flush_local_vmid(struct kvm_s2_mmu *mmu) { - struct kvm *kvm = kern_hyp_va(kern_hyp_va(vcpu)->kvm); - /* Switch to requested VMID */ - write_sysreg(kvm->arch.vttbr, VTTBR); + mmu = kern_hyp_va(mmu); + write_sysreg(mmu->vttbr, VTTBR); isb(); write_sysreg(0, TLBIALL); diff --git a/arch/arm64/include/asm/kvm_asm.h b/arch/arm64/include/asm/kvm_asm.h index 26a64d0..ff6244f 100644 --- a/arch/arm64/include/asm/kvm_asm.h +++ b/arch/arm64/include/asm/kvm_asm.h @@ -44,6 +44,7 @@ #ifndef __ASSEMBLY__ struct kvm; struct kvm_vcpu; +struct kvm_s2_mmu; extern char __kvm_hyp_init[]; extern char __kvm_hyp_init_end[]; @@ -51,9 +52,9 @@ extern char __kvm_hyp_vector[]; extern void __kvm_flush_vm_context(void); -extern void __kvm_tlb_flush_vmid_ipa(struct kvm *kvm, phys_addr_t ipa); -extern void __kvm_tlb_flush_vmid(struct kvm *kvm); -extern void __kvm_tlb_flush_local_vmid(struct kvm_vcpu *vcpu); +extern void __kvm_tlb_flush_vmid_ipa(struct kvm_s2_mmu *mmu, phys_addr_t ipa); +extern void __kvm_tlb_flush_vmid(struct kvm_s2_mmu *mmu); +extern void __kvm_tlb_flush_local_vmid(struct kvm_s2_mmu *mmu); extern int __kvm_vcpu_run(struct kvm_vcpu *vcpu); diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h index 373235c..e7e9f70 100644 --- a/arch/arm64/include/asm/kvm_host.h +++ b/arch/arm64/include/asm/kvm_host.h @@ -50,7 +50,7 @@ int kvm_arch_dev_ioctl_check_extension(struct kvm *kvm, long ext); void __extended_idmap_trampoline(phys_addr_t boot_pgd, phys_addr_t idmap_start); -struct kvm_arch { +struct kvm_s2_mmu { /* The VMID generation used for the virt. memory system */ u64 vmid_gen; u32 vmid; @@ -61,6 +61,11 @@ struct kvm_arch { /* VTTBR value associated with above pgd and vmid */ u64 vttbr; +}; + +struct kvm_arch { + /* Stage 2 paging state for the VM */ + struct kvm_s2_mmu mmu; /* The last vcpu id that ran on each physical CPU */ int __percpu *last_vcpu_ran; @@ -329,6 +334,9 @@ struct kvm_vcpu_arch { /* Detect first run of a vcpu */ bool has_run_once; + + /* Stage 2 paging state used by the hardware on next switch */ + struct kvm_s2_mmu *hw_mmu; }; #define vcpu_gp_regs(v) (&(v)->arch.ctxt.gp_regs) diff --git a/arch/arm64/kvm/hyp/switch.c b/arch/arm64/kvm/hyp/switch.c index 2a64a5c..8b1b3e9 100644 --- a/arch/arm64/kvm/hyp/switch.c +++ b/arch/arm64/kvm/hyp/switch.c @@ -181,8 +181,9 @@ static void __hyp_text __deactivate_traps(struct kvm_vcpu *vcpu) static void __hyp_text __activate_vm(struct kvm_vcpu *vcpu) { - struct kvm *kvm = kern_hyp_va(vcpu->kvm); - write_sysreg(kvm->arch.vttbr, vttbr_el2); + struct kvm_s2_mmu *mmu = kern_hyp_va(vcpu->arch.hw_mmu); + + write_sysreg(mmu->vttbr, vttbr_el2); } static void __hyp_text __deactivate_vm(struct kvm_vcpu *vcpu) diff --git a/arch/arm64/kvm/hyp/tlb.c b/arch/arm64/kvm/hyp/tlb.c index 73464a9..0897678 100644 --- a/arch/arm64/kvm/hyp/tlb.c +++ b/arch/arm64/kvm/hyp/tlb.c @@ -18,7 +18,7 @@ #include #include -static void __hyp_text __tlb_switch_to_guest_vhe(struct kvm *kvm) +static void __hyp_text __tlb_switch_to_guest_vhe(struct kvm_s2_mmu *mmu) { u64 val; @@ -29,16 +29,16 @@ static void __hyp_text __tlb_switch_to_guest_vhe(struct kvm *kvm) * bits. Changing E2H is impossible (goodbye TTBR1_EL2), so * let's flip TGE before executing the TLB operation. */ - write_sysreg(kvm->arch.vttbr, vttbr_el2); + write_sysreg(mmu->vttbr, vttbr_el2); val = read_sysreg(hcr_el2); val &= ~HCR_TGE; write_sysreg(val, hcr_el2); isb(); } -static void __hyp_text __tlb_switch_to_guest_nvhe(struct kvm *kvm) +static void __hyp_text __tlb_switch_to_guest_nvhe(struct kvm_s2_mmu *mmu) { - write_sysreg(kvm->arch.vttbr, vttbr_el2); + write_sysreg(mmu->vttbr, vttbr_el2); isb(); } @@ -47,7 +47,7 @@ static hyp_alternate_select(__tlb_switch_to_guest, __tlb_switch_to_guest_vhe, ARM64_HAS_VIRT_HOST_EXTN); -static void __hyp_text __tlb_switch_to_host_vhe(struct kvm *kvm) +static void __hyp_text __tlb_switch_to_host_vhe(struct kvm_s2_mmu *mmu) { /* * We're done with the TLB operation, let's restore the host's @@ -57,7 +57,7 @@ static void __hyp_text __tlb_switch_to_host_vhe(struct kvm *kvm) write_sysreg(HCR_HOST_VHE_FLAGS, hcr_el2); } -static void __hyp_text __tlb_switch_to_host_nvhe(struct kvm *kvm) +static void __hyp_text __tlb_switch_to_host_nvhe(struct kvm_s2_mmu *mmu) { write_sysreg(0, vttbr_el2); } @@ -67,13 +67,14 @@ static hyp_alternate_select(__tlb_switch_to_host, __tlb_switch_to_host_vhe, ARM64_HAS_VIRT_HOST_EXTN); -void __hyp_text __kvm_tlb_flush_vmid_ipa(struct kvm *kvm, phys_addr_t ipa) +void __hyp_text __kvm_tlb_flush_vmid_ipa(struct kvm_s2_mmu *mmu, + phys_addr_t ipa) { dsb(ishst); /* Switch to requested VMID */ - kvm = kern_hyp_va(kvm); - __tlb_switch_to_guest()(kvm); + mmu = kern_hyp_va(mmu); + __tlb_switch_to_guest()(mmu); /* * We could do so much better if we had the VA as well. @@ -116,36 +117,35 @@ void __hyp_text __kvm_tlb_flush_vmid_ipa(struct kvm *kvm, phys_addr_t ipa) if (!has_vhe() && icache_is_vpipt()) __flush_icache_all(); - __tlb_switch_to_host()(kvm); + __tlb_switch_to_host()(mmu); } -void __hyp_text __kvm_tlb_flush_vmid(struct kvm *kvm) +void __hyp_text __kvm_tlb_flush_vmid(struct kvm_s2_mmu *mmu) { dsb(ishst); /* Switch to requested VMID */ - kvm = kern_hyp_va(kvm); - __tlb_switch_to_guest()(kvm); + mmu = kern_hyp_va(mmu); + __tlb_switch_to_guest()(mmu); __tlbi(vmalls12e1is); dsb(ish); isb(); - __tlb_switch_to_host()(kvm); + __tlb_switch_to_host()(mmu); } -void __hyp_text __kvm_tlb_flush_local_vmid(struct kvm_vcpu *vcpu) +void __hyp_text __kvm_tlb_flush_local_vmid(struct kvm_s2_mmu *mmu) { - struct kvm *kvm = kern_hyp_va(kern_hyp_va(vcpu)->kvm); - /* Switch to requested VMID */ - __tlb_switch_to_guest()(kvm); + mmu = kern_hyp_va(mmu); + __tlb_switch_to_guest()(mmu); __tlbi(vmalle1); dsb(nsh); isb(); - __tlb_switch_to_host()(kvm); + __tlb_switch_to_host()(mmu); } void __hyp_text __kvm_flush_vm_context(void) diff --git a/virt/kvm/arm/arm.c b/virt/kvm/arm/arm.c index 0ff2997..bee27bb 100644 --- a/virt/kvm/arm/arm.c +++ b/virt/kvm/arm/arm.c @@ -138,7 +138,7 @@ int kvm_arch_init_vm(struct kvm *kvm, unsigned long type) kvm_vgic_early_init(kvm); /* Mark the initial VMID generation invalid */ - kvm->arch.vmid_gen = 0; + kvm->arch.mmu.vmid_gen = 0; /* The maximum number of VCPUs is limited by the host's GIC model */ kvm->arch.max_vcpus = vgic_present ? @@ -334,6 +334,8 @@ int kvm_arch_vcpu_init(struct kvm_vcpu *vcpu) kvm_arm_reset_debug_ptr(vcpu); + vcpu->arch.hw_mmu = &vcpu->kvm->arch.mmu; + return kvm_vgic_vcpu_init(vcpu); } @@ -348,7 +350,7 @@ void kvm_arch_vcpu_load(struct kvm_vcpu *vcpu, int cpu) * over-invalidation doesn't affect correctness. */ if (*last_ran != vcpu->vcpu_id) { - kvm_call_hyp(__kvm_tlb_flush_local_vmid, vcpu); + kvm_call_hyp(__kvm_tlb_flush_local_vmid, &vcpu->kvm->arch.mmu); *last_ran = vcpu->vcpu_id; } @@ -442,25 +444,26 @@ void force_vm_exit(const cpumask_t *mask) * VMID for the new generation, we must flush necessary caches and TLBs on all * CPUs. */ -static bool need_new_vmid_gen(struct kvm *kvm) +static bool need_new_vmid_gen(struct kvm_s2_mmu *mmu) { - return unlikely(kvm->arch.vmid_gen != atomic64_read(&kvm_vmid_gen)); + return unlikely(mmu->vmid_gen != atomic64_read(&kvm_vmid_gen)); } /** * update_vttbr - Update the VTTBR with a valid VMID before the guest runs - * @kvm The guest that we are about to run + * @kvm: The guest that we are about to run + * @mmu: The stage-2 translation context to update * * Called from kvm_arch_vcpu_ioctl_run before entering the guest to ensure the * VM has a valid VMID, otherwise assigns a new one and flushes corresponding * caches and TLBs. */ -static void update_vttbr(struct kvm *kvm) +static void update_vttbr(struct kvm *kvm, struct kvm_s2_mmu *mmu) { phys_addr_t pgd_phys; u64 vmid; - if (!need_new_vmid_gen(kvm)) + if (!need_new_vmid_gen(mmu)) return; spin_lock(&kvm_vmid_lock); @@ -470,7 +473,7 @@ static void update_vttbr(struct kvm *kvm) * already allocated a valid vmid for this vm, then this vcpu should * use the same vmid. */ - if (!need_new_vmid_gen(kvm)) { + if (!need_new_vmid_gen(mmu)) { spin_unlock(&kvm_vmid_lock); return; } @@ -494,16 +497,17 @@ static void update_vttbr(struct kvm *kvm) kvm_call_hyp(__kvm_flush_vm_context); } - kvm->arch.vmid_gen = atomic64_read(&kvm_vmid_gen); - kvm->arch.vmid = kvm_next_vmid; + mmu->vmid_gen = atomic64_read(&kvm_vmid_gen); + mmu->vmid = kvm_next_vmid; kvm_next_vmid++; kvm_next_vmid &= (1 << kvm_vmid_bits) - 1; /* update vttbr to be used with the new vmid */ - pgd_phys = virt_to_phys(kvm->arch.pgd); + pgd_phys = virt_to_phys(mmu->pgd); BUG_ON(pgd_phys & ~VTTBR_BADDR_MASK); - vmid = ((u64)(kvm->arch.vmid) << VTTBR_VMID_SHIFT) & VTTBR_VMID_MASK(kvm_vmid_bits); - kvm->arch.vttbr = pgd_phys | vmid; + vmid = ((u64)(mmu->vmid) << VTTBR_VMID_SHIFT) & + VTTBR_VMID_MASK(kvm_vmid_bits); + mmu->vttbr = pgd_phys | vmid; spin_unlock(&kvm_vmid_lock); } @@ -638,7 +642,7 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu, struct kvm_run *run) */ cond_resched(); - update_vttbr(vcpu->kvm); + update_vttbr(vcpu->kvm, vcpu->arch.hw_mmu); check_vcpu_requests(vcpu); @@ -677,7 +681,7 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu, struct kvm_run *run) */ smp_store_mb(vcpu->mode, IN_GUEST_MODE); - if (ret <= 0 || need_new_vmid_gen(vcpu->kvm) || + if (ret <= 0 || need_new_vmid_gen(vcpu->arch.hw_mmu) || kvm_request_pending(vcpu)) { vcpu->mode = OUTSIDE_GUEST_MODE; local_irq_enable(); diff --git a/virt/kvm/arm/mmu.c b/virt/kvm/arm/mmu.c index 0a5f5ca..d8ea1f9 100644 --- a/virt/kvm/arm/mmu.c +++ b/virt/kvm/arm/mmu.c @@ -64,9 +64,9 @@ void kvm_flush_remote_tlbs(struct kvm *kvm) kvm_call_hyp(__kvm_tlb_flush_vmid, kvm); } -static void kvm_tlb_flush_vmid_ipa(struct kvm *kvm, phys_addr_t ipa) +static void kvm_tlb_flush_vmid_ipa(struct kvm_s2_mmu *mmu, phys_addr_t ipa) { - kvm_call_hyp(__kvm_tlb_flush_vmid_ipa, kvm, ipa); + kvm_call_hyp(__kvm_tlb_flush_vmid_ipa, mmu, ipa); } /* @@ -103,13 +103,14 @@ static bool kvm_is_device_pfn(unsigned long pfn) * Function clears a PMD entry, flushes addr 1st and 2nd stage TLBs. Marks all * pages in the range dirty. */ -static void stage2_dissolve_pmd(struct kvm *kvm, phys_addr_t addr, pmd_t *pmd) +static void stage2_dissolve_pmd(struct kvm_s2_mmu *mmu, phys_addr_t addr, + pmd_t *pmd) { if (!pmd_thp_or_huge(*pmd)) return; pmd_clear(pmd); - kvm_tlb_flush_vmid_ipa(kvm, addr); + kvm_tlb_flush_vmid_ipa(mmu, addr); put_page(virt_to_page(pmd)); } @@ -145,31 +146,34 @@ static void *mmu_memory_cache_alloc(struct kvm_mmu_memory_cache *mc) return p; } -static void clear_stage2_pgd_entry(struct kvm *kvm, pgd_t *pgd, phys_addr_t addr) +static void clear_stage2_pgd_entry(struct kvm_s2_mmu *mmu, + pgd_t *pgd, phys_addr_t addr) { pud_t *pud_table __maybe_unused = stage2_pud_offset(pgd, 0UL); stage2_pgd_clear(pgd); - kvm_tlb_flush_vmid_ipa(kvm, addr); + kvm_tlb_flush_vmid_ipa(mmu, addr); stage2_pud_free(pud_table); put_page(virt_to_page(pgd)); } -static void clear_stage2_pud_entry(struct kvm *kvm, pud_t *pud, phys_addr_t addr) +static void clear_stage2_pud_entry(struct kvm_s2_mmu *mmu, + pud_t *pud, phys_addr_t addr) { pmd_t *pmd_table __maybe_unused = stage2_pmd_offset(pud, 0); VM_BUG_ON(stage2_pud_huge(*pud)); stage2_pud_clear(pud); - kvm_tlb_flush_vmid_ipa(kvm, addr); + kvm_tlb_flush_vmid_ipa(mmu, addr); stage2_pmd_free(pmd_table); put_page(virt_to_page(pud)); } -static void clear_stage2_pmd_entry(struct kvm *kvm, pmd_t *pmd, phys_addr_t addr) +static void clear_stage2_pmd_entry(struct kvm_s2_mmu *mmu, + pmd_t *pmd, phys_addr_t addr) { pte_t *pte_table = pte_offset_kernel(pmd, 0); VM_BUG_ON(pmd_thp_or_huge(*pmd)); pmd_clear(pmd); - kvm_tlb_flush_vmid_ipa(kvm, addr); + kvm_tlb_flush_vmid_ipa(mmu, addr); pte_free_kernel(NULL, pte_table); put_page(virt_to_page(pmd)); } @@ -194,7 +198,7 @@ static void clear_stage2_pmd_entry(struct kvm *kvm, pmd_t *pmd, phys_addr_t addr * the corresponding TLBs, we call kvm_flush_dcache_p*() to make sure * the IO subsystem will never hit in the cache. */ -static void unmap_stage2_ptes(struct kvm *kvm, pmd_t *pmd, +static void unmap_stage2_ptes(struct kvm_s2_mmu *mmu, pmd_t *pmd, phys_addr_t addr, phys_addr_t end) { phys_addr_t start_addr = addr; @@ -206,7 +210,7 @@ static void unmap_stage2_ptes(struct kvm *kvm, pmd_t *pmd, pte_t old_pte = *pte; kvm_set_pte(pte, __pte(0)); - kvm_tlb_flush_vmid_ipa(kvm, addr); + kvm_tlb_flush_vmid_ipa(mmu, addr); /* No need to invalidate the cache for device mappings */ if (!kvm_is_device_pfn(pte_pfn(old_pte))) @@ -217,10 +221,10 @@ static void unmap_stage2_ptes(struct kvm *kvm, pmd_t *pmd, } while (pte++, addr += PAGE_SIZE, addr != end); if (stage2_pte_table_empty(start_pte)) - clear_stage2_pmd_entry(kvm, pmd, start_addr); + clear_stage2_pmd_entry(mmu, pmd, start_addr); } -static void unmap_stage2_pmds(struct kvm *kvm, pud_t *pud, +static void unmap_stage2_pmds(struct kvm_s2_mmu *mmu, pud_t *pud, phys_addr_t addr, phys_addr_t end) { phys_addr_t next, start_addr = addr; @@ -234,22 +238,22 @@ static void unmap_stage2_pmds(struct kvm *kvm, pud_t *pud, pmd_t old_pmd = *pmd; pmd_clear(pmd); - kvm_tlb_flush_vmid_ipa(kvm, addr); + kvm_tlb_flush_vmid_ipa(mmu, addr); kvm_flush_dcache_pmd(old_pmd); put_page(virt_to_page(pmd)); } else { - unmap_stage2_ptes(kvm, pmd, addr, next); + unmap_stage2_ptes(mmu, pmd, addr, next); } } } while (pmd++, addr = next, addr != end); if (stage2_pmd_table_empty(start_pmd)) - clear_stage2_pud_entry(kvm, pud, start_addr); + clear_stage2_pud_entry(mmu, pud, start_addr); } -static void unmap_stage2_puds(struct kvm *kvm, pgd_t *pgd, +static void unmap_stage2_puds(struct kvm_s2_mmu *mmu, pgd_t *pgd, phys_addr_t addr, phys_addr_t end) { phys_addr_t next, start_addr = addr; @@ -263,17 +267,17 @@ static void unmap_stage2_puds(struct kvm *kvm, pgd_t *pgd, pud_t old_pud = *pud; stage2_pud_clear(pud); - kvm_tlb_flush_vmid_ipa(kvm, addr); + kvm_tlb_flush_vmid_ipa(mmu, addr); kvm_flush_dcache_pud(old_pud); put_page(virt_to_page(pud)); } else { - unmap_stage2_pmds(kvm, pud, addr, next); + unmap_stage2_pmds(mmu, pud, addr, next); } } } while (pud++, addr = next, addr != end); if (stage2_pud_table_empty(start_pud)) - clear_stage2_pgd_entry(kvm, pgd, start_addr); + clear_stage2_pgd_entry(mmu, pgd, start_addr); } /** @@ -292,20 +296,21 @@ static void unmap_stage2_range(struct kvm *kvm, phys_addr_t start, u64 size) pgd_t *pgd; phys_addr_t addr = start, end = start + size; phys_addr_t next; + struct kvm_s2_mmu *mmu = &kvm->arch.mmu; assert_spin_locked(&kvm->mmu_lock); - pgd = kvm->arch.pgd + stage2_pgd_index(addr); + pgd = mmu->pgd + stage2_pgd_index(addr); do { /* * Make sure the page table is still active, as another thread * could have possibly freed the page table, while we released * the lock. */ - if (!READ_ONCE(kvm->arch.pgd)) + if (!READ_ONCE(mmu->pgd)) break; next = stage2_pgd_addr_end(addr, end); if (!stage2_pgd_none(*pgd)) - unmap_stage2_puds(kvm, pgd, addr, next); + unmap_stage2_puds(mmu, pgd, addr, next); /* * If the range is too large, release the kvm->mmu_lock * to prevent starvation and lockup detector warnings. @@ -360,7 +365,7 @@ static void stage2_flush_puds(pgd_t *pgd, phys_addr_t addr, phys_addr_t end) } while (pud++, addr = next, addr != end); } -static void stage2_flush_memslot(struct kvm *kvm, +static void stage2_flush_memslot(struct kvm_s2_mmu *mmu, struct kvm_memory_slot *memslot) { phys_addr_t addr = memslot->base_gfn << PAGE_SHIFT; @@ -368,7 +373,7 @@ static void stage2_flush_memslot(struct kvm *kvm, phys_addr_t next; pgd_t *pgd; - pgd = kvm->arch.pgd + stage2_pgd_index(addr); + pgd = mmu->pgd + stage2_pgd_index(addr); do { next = stage2_pgd_addr_end(addr, end); stage2_flush_puds(pgd, addr, next); @@ -393,7 +398,7 @@ static void stage2_flush_vm(struct kvm *kvm) slots = kvm_memslots(kvm); kvm_for_each_memslot(memslot, slots) - stage2_flush_memslot(kvm, memslot); + stage2_flush_memslot(&kvm->arch.mmu, memslot); spin_unlock(&kvm->mmu_lock); srcu_read_unlock(&kvm->srcu, idx); @@ -745,8 +750,9 @@ int create_hyp_io_mappings(void *from, void *to, phys_addr_t phys_addr) int kvm_alloc_stage2_pgd(struct kvm *kvm) { pgd_t *pgd; + struct kvm_s2_mmu *mmu = &kvm->arch.mmu; - if (kvm->arch.pgd != NULL) { + if (mmu->pgd != NULL) { kvm_err("kvm_arch already initialized?\n"); return -EINVAL; } @@ -756,7 +762,8 @@ int kvm_alloc_stage2_pgd(struct kvm *kvm) if (!pgd) return -ENOMEM; - kvm->arch.pgd = pgd; + mmu->pgd = pgd; + return 0; } @@ -831,19 +838,20 @@ void stage2_unmap_vm(struct kvm *kvm) * kvm_free_stage2_pgd - free all stage-2 tables * @kvm: The KVM struct pointer for the VM. * - * Walks the level-1 page table pointed to by kvm->arch.pgd and frees all + * Walks the level-1 page table pointed to by kvm->arch.mmu.pgd and frees all * underlying level-2 and level-3 tables before freeing the actual level-1 table * and setting the struct pointer to NULL. */ void kvm_free_stage2_pgd(struct kvm *kvm) { void *pgd = NULL; + struct kvm_s2_mmu *mmu = &kvm->arch.mmu; spin_lock(&kvm->mmu_lock); - if (kvm->arch.pgd) { + if (mmu->pgd) { unmap_stage2_range(kvm, 0, KVM_PHYS_SIZE); - pgd = READ_ONCE(kvm->arch.pgd); - kvm->arch.pgd = NULL; + pgd = READ_ONCE(mmu->pgd); + mmu->pgd = NULL; } spin_unlock(&kvm->mmu_lock); @@ -852,13 +860,14 @@ void kvm_free_stage2_pgd(struct kvm *kvm) free_pages_exact(pgd, S2_PGD_SIZE); } -static pud_t *stage2_get_pud(struct kvm *kvm, struct kvm_mmu_memory_cache *cache, +static pud_t *stage2_get_pud(struct kvm_s2_mmu *mmu, + struct kvm_mmu_memory_cache *cache, phys_addr_t addr) { pgd_t *pgd; pud_t *pud; - pgd = kvm->arch.pgd + stage2_pgd_index(addr); + pgd = mmu->pgd + stage2_pgd_index(addr); if (WARN_ON(stage2_pgd_none(*pgd))) { if (!cache) return NULL; @@ -870,13 +879,14 @@ static pud_t *stage2_get_pud(struct kvm *kvm, struct kvm_mmu_memory_cache *cache return stage2_pud_offset(pgd, addr); } -static pmd_t *stage2_get_pmd(struct kvm *kvm, struct kvm_mmu_memory_cache *cache, +static pmd_t *stage2_get_pmd(struct kvm_s2_mmu *mmu, + struct kvm_mmu_memory_cache *cache, phys_addr_t addr) { pud_t *pud; pmd_t *pmd; - pud = stage2_get_pud(kvm, cache, addr); + pud = stage2_get_pud(mmu, cache, addr); if (!pud) return NULL; @@ -891,12 +901,13 @@ static pmd_t *stage2_get_pmd(struct kvm *kvm, struct kvm_mmu_memory_cache *cache return stage2_pmd_offset(pud, addr); } -static int stage2_set_pmd_huge(struct kvm *kvm, struct kvm_mmu_memory_cache +static int stage2_set_pmd_huge(struct kvm_s2_mmu *mmu, + struct kvm_mmu_memory_cache *cache, phys_addr_t addr, const pmd_t *new_pmd) { pmd_t *pmd, old_pmd; - pmd = stage2_get_pmd(kvm, cache, addr); + pmd = stage2_get_pmd(mmu, cache, addr); VM_BUG_ON(!pmd); /* @@ -913,7 +924,7 @@ static int stage2_set_pmd_huge(struct kvm *kvm, struct kvm_mmu_memory_cache old_pmd = *pmd; if (pmd_present(old_pmd)) { pmd_clear(pmd); - kvm_tlb_flush_vmid_ipa(kvm, addr); + kvm_tlb_flush_vmid_ipa(mmu, addr); } else { get_page(virt_to_page(pmd)); } @@ -922,7 +933,8 @@ static int stage2_set_pmd_huge(struct kvm *kvm, struct kvm_mmu_memory_cache return 0; } -static int stage2_set_pte(struct kvm *kvm, struct kvm_mmu_memory_cache *cache, +static int stage2_set_pte(struct kvm_s2_mmu *mmu, + struct kvm_mmu_memory_cache *cache, phys_addr_t addr, const pte_t *new_pte, unsigned long flags) { @@ -934,7 +946,7 @@ static int stage2_set_pte(struct kvm *kvm, struct kvm_mmu_memory_cache *cache, VM_BUG_ON(logging_active && !cache); /* Create stage-2 page table mapping - Levels 0 and 1 */ - pmd = stage2_get_pmd(kvm, cache, addr); + pmd = stage2_get_pmd(mmu, cache, addr); if (!pmd) { /* * Ignore calls from kvm_set_spte_hva for unallocated @@ -948,7 +960,7 @@ static int stage2_set_pte(struct kvm *kvm, struct kvm_mmu_memory_cache *cache, * allocate page. */ if (logging_active) - stage2_dissolve_pmd(kvm, addr, pmd); + stage2_dissolve_pmd(mmu, addr, pmd); /* Create stage-2 page mappings - Level 2 */ if (pmd_none(*pmd)) { @@ -968,7 +980,7 @@ static int stage2_set_pte(struct kvm *kvm, struct kvm_mmu_memory_cache *cache, old_pte = *pte; if (pte_present(old_pte)) { kvm_set_pte(pte, __pte(0)); - kvm_tlb_flush_vmid_ipa(kvm, addr); + kvm_tlb_flush_vmid_ipa(mmu, addr); } else { get_page(virt_to_page(pte)); } @@ -1028,7 +1040,7 @@ int kvm_phys_addr_ioremap(struct kvm *kvm, phys_addr_t guest_ipa, if (ret) goto out; spin_lock(&kvm->mmu_lock); - ret = stage2_set_pte(kvm, &cache, addr, &pte, + ret = stage2_set_pte(&kvm->arch.mmu, &cache, addr, &pte, KVM_S2PTE_FLAG_IS_IOMAP); spin_unlock(&kvm->mmu_lock); if (ret) @@ -1166,12 +1178,13 @@ static void stage2_wp_puds(pgd_t *pgd, phys_addr_t addr, phys_addr_t end) * @addr: Start address of range * @end: End address of range */ -static void stage2_wp_range(struct kvm *kvm, phys_addr_t addr, phys_addr_t end) +static void stage2_wp_range(struct kvm *kvm, struct kvm_s2_mmu *mmu, + phys_addr_t addr, phys_addr_t end) { pgd_t *pgd; phys_addr_t next; - pgd = kvm->arch.pgd + stage2_pgd_index(addr); + pgd = mmu->pgd + stage2_pgd_index(addr); do { /* * Release kvm_mmu_lock periodically if the memory region is @@ -1183,7 +1196,7 @@ static void stage2_wp_range(struct kvm *kvm, phys_addr_t addr, phys_addr_t end) * the lock. */ cond_resched_lock(&kvm->mmu_lock); - if (!READ_ONCE(kvm->arch.pgd)) + if (!READ_ONCE(mmu->pgd)) break; next = stage2_pgd_addr_end(addr, end); if (stage2_pgd_present(*pgd)) @@ -1212,7 +1225,7 @@ void kvm_mmu_wp_memory_region(struct kvm *kvm, int slot) phys_addr_t end = (memslot->base_gfn + memslot->npages) << PAGE_SHIFT; spin_lock(&kvm->mmu_lock); - stage2_wp_range(kvm, start, end); + stage2_wp_range(kvm, &kvm->arch.mmu, start, end); spin_unlock(&kvm->mmu_lock); kvm_flush_remote_tlbs(kvm); } @@ -1236,7 +1249,7 @@ static void kvm_mmu_write_protect_pt_masked(struct kvm *kvm, phys_addr_t start = (base_gfn + __ffs(mask)) << PAGE_SHIFT; phys_addr_t end = (base_gfn + __fls(mask) + 1) << PAGE_SHIFT; - stage2_wp_range(kvm, start, end); + stage2_wp_range(kvm, &kvm->arch.mmu, start, end); } /* @@ -1292,6 +1305,7 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa, pgprot_t mem_type = PAGE_S2; bool logging_active = memslot_is_logging(memslot); unsigned long flags = 0; + struct kvm_s2_mmu *mmu = vcpu->arch.hw_mmu; write_fault = kvm_is_write_fault(vcpu); if (fault_status == FSC_PERM && !write_fault) { @@ -1388,7 +1402,7 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa, kvm_set_pfn_dirty(pfn); } coherent_cache_guest_page(vcpu, pfn, PMD_SIZE); - ret = stage2_set_pmd_huge(kvm, memcache, fault_ipa, &new_pmd); + ret = stage2_set_pmd_huge(mmu, memcache, fault_ipa, &new_pmd); } else { pte_t new_pte = pfn_pte(pfn, mem_type); @@ -1398,7 +1412,7 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa, mark_page_dirty(kvm, gfn); } coherent_cache_guest_page(vcpu, pfn, PAGE_SIZE); - ret = stage2_set_pte(kvm, memcache, fault_ipa, &new_pte, flags); + ret = stage2_set_pte(mmu, memcache, fault_ipa, &new_pte, flags); } out_unlock: @@ -1426,7 +1440,7 @@ static void handle_access_fault(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa) spin_lock(&vcpu->kvm->mmu_lock); - pmd = stage2_get_pmd(vcpu->kvm, NULL, fault_ipa); + pmd = stage2_get_pmd(vcpu->arch.hw_mmu, NULL, fault_ipa); if (!pmd || pmd_none(*pmd)) /* Nothing there */ goto out; @@ -1594,7 +1608,7 @@ int kvm_unmap_hva(struct kvm *kvm, unsigned long hva) { unsigned long end = hva + PAGE_SIZE; - if (!kvm->arch.pgd) + if (!kvm->arch.mmu.pgd) return 0; trace_kvm_unmap_hva(hva); @@ -1605,7 +1619,7 @@ int kvm_unmap_hva(struct kvm *kvm, unsigned long hva) int kvm_unmap_hva_range(struct kvm *kvm, unsigned long start, unsigned long end) { - if (!kvm->arch.pgd) + if (!kvm->arch.mmu.pgd) return 0; trace_kvm_unmap_hva_range(start, end); @@ -1625,7 +1639,7 @@ static int kvm_set_spte_handler(struct kvm *kvm, gpa_t gpa, u64 size, void *data * therefore stage2_set_pte() never needs to clear out a huge PMD * through this calling path. */ - stage2_set_pte(kvm, NULL, gpa, pte, 0); + stage2_set_pte(&kvm->arch.mmu, NULL, gpa, pte, 0); return 0; } @@ -1635,7 +1649,7 @@ void kvm_set_spte_hva(struct kvm *kvm, unsigned long hva, pte_t pte) unsigned long end = hva + PAGE_SIZE; pte_t stage2_pte; - if (!kvm->arch.pgd) + if (!kvm->arch.mmu.pgd) return; trace_kvm_set_spte_hva(hva); @@ -1649,7 +1663,7 @@ static int kvm_age_hva_handler(struct kvm *kvm, gpa_t gpa, u64 size, void *data) pte_t *pte; WARN_ON(size != PAGE_SIZE && size != PMD_SIZE); - pmd = stage2_get_pmd(kvm, NULL, gpa); + pmd = stage2_get_pmd(&kvm->arch.mmu, NULL, gpa); if (!pmd || pmd_none(*pmd)) /* Nothing there */ return 0; @@ -1669,7 +1683,7 @@ static int kvm_test_age_hva_handler(struct kvm *kvm, gpa_t gpa, u64 size, void * pte_t *pte; WARN_ON(size != PAGE_SIZE && size != PMD_SIZE); - pmd = stage2_get_pmd(kvm, NULL, gpa); + pmd = stage2_get_pmd(&kvm->arch.mmu, NULL, gpa); if (!pmd || pmd_none(*pmd)) /* Nothing there */ return 0; @@ -1898,9 +1912,10 @@ int kvm_arch_prepare_memory_region(struct kvm *kvm, spin_lock(&kvm->mmu_lock); if (ret) - unmap_stage2_range(kvm, mem->guest_phys_addr, mem->memory_size); + unmap_stage2_range(kvm, mem->guest_phys_addr, + mem->memory_size); else - stage2_flush_memslot(kvm, memslot); + stage2_flush_memslot(&kvm->arch.mmu, memslot); spin_unlock(&kvm->mmu_lock); out: up_read(¤t->mm->mmap_sem); From patchwork Tue Oct 3 03:10:47 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jintack Lim X-Patchwork-Id: 114643 Delivered-To: patch@linaro.org Received: by 10.140.22.163 with SMTP id 32csp1384387qgn; Mon, 2 Oct 2017 20:11:50 -0700 (PDT) X-Received: by 10.84.212.144 with SMTP id e16mr10753940pli.205.1507000310693; Mon, 02 Oct 2017 20:11:50 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1507000310; cv=none; d=google.com; s=arc-20160816; b=rqqLe6x1lsiw4pGn1dVer+3HVDAmeral5p7KNTN9sInUxQiCsyMVYl+xq7VB07Xser XuNKPv3T2CotU5QAGtReQvUKHpBweB6ccuasRJoOBccFAroo0p0CNyuNGbnbLs4e7III xTtbW8F1pQvcIRaMBQYXcACMstcxAN/WsGYYPOg7M7A1ja6QuBIyuUXe4RSfNqCkN519 JZC1ij84qUkp5BAys4wozqk2pNMCg5uzuHTGdElGUVuq4wnRFIlfi6BTn0T7h7qDAdfI Ms5OLuPge9bsE7zGMepcxMK6tEQH7j7hCGsDPsFFB985c6RnhnSks4pnA9SqKeE6WhWe UxKQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature:arc-authentication-results; bh=eAqx2xLxCExiluCRkqw3a6k08S4JY17Xyb7qWUk8dxk=; b=NSYXEK13j/DV1feeTdDdjhIduAPFtljMucF92j+MUTlLoQ3+xa9HEQR2NfUejERd3j NewhCTfGBa0+P5XsxSHwWP2UWWuXdx3helxC+RhSvhxDMpWP+7sQJQ1+qL5XGqsJyOQ2 RxzshK3BmJ33hWIAd+RioHXdT1IK+YLlzPtrE7b/JeS5PZfViOd7sfHIp2hiC12dcb13 jq9YjqOEvBgkXO8NpEjDjebv6w34K5G36FvY03cGfoenaxMwRz/sdS4f3/k4yOpuhik/ 5FC3u2WeWkYFeaK0WU1NeCL23L1sP1qOQczhd9fIKv3e1oSX5mRu1ATuo0VrYh/S0hLb Rzlg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=S8xzPk2W; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id b9si7652472plx.666.2017.10.02.20.11.50; Mon, 02 Oct 2017 20:11:50 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=S8xzPk2W; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751893AbdJCDLr (ORCPT + 26 others); Mon, 2 Oct 2017 23:11:47 -0400 Received: from mail-io0-f178.google.com ([209.85.223.178]:46983 "EHLO mail-io0-f178.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751469AbdJCDLn (ORCPT ); Mon, 2 Oct 2017 23:11:43 -0400 Received: by mail-io0-f178.google.com with SMTP id d16so6450703ioj.3 for ; Mon, 02 Oct 2017 20:11:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=eAqx2xLxCExiluCRkqw3a6k08S4JY17Xyb7qWUk8dxk=; b=S8xzPk2Wq31DEaTbSDQYeG4CWBCMPuKFYDuzz2HU9FXhvPmUtg5DHHyrbPkeu3aiQt 6rrnLYvufcAiHpiAlzodUICx4aOuhnJcvPVNw3iX1bWWq0lzJvjzNefhP9BmuMsBv4YM YrC7L2efBwhGBFZTgv0oF0s+6fxC2DJZgkM94= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=eAqx2xLxCExiluCRkqw3a6k08S4JY17Xyb7qWUk8dxk=; b=hJTta4ozvbaoMcXobY28+ym21gX52Etv/HC1pWOHeLwHQVXuA0VghDY2EoZUwH6Rk9 aTf65Mt2r5dpj1mkVy7RktGm3uFWozy0jsxxcxYLwV1/x5n7w9pFXCCms7Vy/TII+y6x uoW+0NZoyY2MYJKGY+J232L4OV5ubHZFZsTqtkEhawtuAkPxk1eNMA9/ALM/fV4OomUD JteOODDp7S5LmKs1H+l6KVKjzEP2tbG6yhStPubDpaacPqK8JdHlvvkF/6WI16Abnzyw WAWOO6fwdfrgI4XE3J2+nSskmLuZA0QKj79MZSd513Zk6QLOurIw3+9Dx6w724mh79AR kUiQ== X-Gm-Message-State: AMCzsaXRbrz5BbjqwM3q+7gWa6p1OoVJaDZ0FRcM6McWDldVVvImInmp Xf7sqIJuQX5kVGOvPVj+AhlZMg== X-Google-Smtp-Source: AOwi7QAIBwULKw9b0zd/HR8w5S8KEvT4t3oAigcFqJ7oB2XwtvUhoMEGnvWO1tz0o1vhUTdqx9R4Bg== X-Received: by 10.107.204.5 with SMTP id c5mr26146495iog.66.1507000302318; Mon, 02 Oct 2017 20:11:42 -0700 (PDT) Received: from node.jintackl-qv28633.kvmarm-pg0.wisc.cloudlab.us (c220g1-031126.wisc.cloudlab.us. [128.104.222.76]) by smtp.gmail.com with ESMTPSA id h84sm5367193iod.72.2017.10.02.20.11.40 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Mon, 02 Oct 2017 20:11:41 -0700 (PDT) From: Jintack Lim To: christoffer.dall@linaro.org, marc.zyngier@arm.com, kvmarm@lists.cs.columbia.edu Cc: jintack@cs.columbia.edu, pbonzini@redhat.com, rkrcmar@redhat.com, catalin.marinas@arm.com, will.deacon@arm.com, linux@armlinux.org.uk, mark.rutland@arm.com, linux-arm-kernel@lists.infradead.org, kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Jintack Lim Subject: [RFC PATCH v2 05/31] KVM: arm/arm64: Support mmu for the virtual EL2 execution Date: Mon, 2 Oct 2017 22:10:47 -0500 Message-Id: <1507000273-3735-3-git-send-email-jintack.lim@linaro.org> X-Mailer: git-send-email 1.9.1 In-Reply-To: <1507000273-3735-1-git-send-email-jintack.lim@linaro.org> References: <1507000273-3735-1-git-send-email-jintack.lim@linaro.org> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org From: Christoffer Dall When running a guest hypervisor in virtual EL2, the translation context has to be separate from the rest of the system, including the guest EL1/0 translation regime, so we allocate a separate VMID for this mode. Considering that we have two different vttbr values due to separate VMIDs, it's racy to keep a vttbr value in a struct (kvm_s2_mmu) and share it between multiple vcpus. So, remove the shared vttbr field, and set up per-vcpu hw_vttbr field. Hypercalls to flush tlb now have vttbr as a parameter instead of mmu, since mmu structure does not have vttbr any more. Signed-off-by: Christoffer Dall Signed-off-by: Jintack Lim --- Notes: v1-->v2: Fixed a bug that hw_vttbr was not initialized correctly in kvm_arch_vcpu_init() where vmid is not allocated yet. This prevented the guest from booting on 32bit arm; hw_vttbr is set on each entry on aarch64, so it was fine. arch/arm/include/asm/kvm_asm.h | 6 ++-- arch/arm/include/asm/kvm_emulate.h | 4 +++ arch/arm/include/asm/kvm_host.h | 14 +++++--- arch/arm/include/asm/kvm_mmu.h | 11 ++++++ arch/arm/kvm/hyp/switch.c | 4 +-- arch/arm/kvm/hyp/tlb.c | 15 ++++----- arch/arm64/include/asm/kvm_asm.h | 6 ++-- arch/arm64/include/asm/kvm_emulate.h | 8 +++++ arch/arm64/include/asm/kvm_host.h | 14 +++++--- arch/arm64/include/asm/kvm_mmu.h | 11 ++++++ arch/arm64/kvm/hyp/switch.c | 4 +-- arch/arm64/kvm/hyp/tlb.c | 34 +++++++++---------- virt/kvm/arm/arm.c | 65 +++++++++++++++++++++--------------- virt/kvm/arm/mmu.c | 9 +++-- 14 files changed, 128 insertions(+), 77 deletions(-) -- 1.9.1 diff --git a/arch/arm/include/asm/kvm_asm.h b/arch/arm/include/asm/kvm_asm.h index 71b7255..23a79bd 100644 --- a/arch/arm/include/asm/kvm_asm.h +++ b/arch/arm/include/asm/kvm_asm.h @@ -65,9 +65,9 @@ extern char __kvm_hyp_vector[]; extern void __kvm_flush_vm_context(void); -extern void __kvm_tlb_flush_vmid_ipa(struct kvm_s2_mmu *mmu, phys_addr_t ipa); -extern void __kvm_tlb_flush_vmid(struct kvm_s2_mmu *mmu); -extern void __kvm_tlb_flush_local_vmid(struct kvm_s2_mmu *mmu); +extern void __kvm_tlb_flush_vmid_ipa(u64 vttbr, phys_addr_t ipa); +extern void __kvm_tlb_flush_vmid(u64 vttbr); +extern void __kvm_tlb_flush_local_vmid(u64 vttbr); extern int __kvm_vcpu_run(struct kvm_vcpu *vcpu); diff --git a/arch/arm/include/asm/kvm_emulate.h b/arch/arm/include/asm/kvm_emulate.h index 29a4dec..24a3fbf 100644 --- a/arch/arm/include/asm/kvm_emulate.h +++ b/arch/arm/include/asm/kvm_emulate.h @@ -293,4 +293,8 @@ static inline unsigned long vcpu_data_host_to_guest(struct kvm_vcpu *vcpu, } } +static inline struct kvm_s2_vmid *vcpu_get_active_vmid(struct kvm_vcpu *vcpu) +{ + return &vcpu->kvm->arch.mmu.vmid; +} #endif /* __ARM_KVM_EMULATE_H__ */ diff --git a/arch/arm/include/asm/kvm_host.h b/arch/arm/include/asm/kvm_host.h index 78d826e..33ccdbe 100644 --- a/arch/arm/include/asm/kvm_host.h +++ b/arch/arm/include/asm/kvm_host.h @@ -53,16 +53,18 @@ int kvm_reset_vcpu(struct kvm_vcpu *vcpu); void kvm_reset_coprocs(struct kvm_vcpu *vcpu); -struct kvm_s2_mmu { +struct kvm_s2_vmid { /* The VMID generation used for the virt. memory system */ u64 vmid_gen; u32 vmid; +}; + +struct kvm_s2_mmu { + struct kvm_s2_vmid vmid; + struct kvm_s2_vmid el2_vmid; /* Stage-2 page table */ pgd_t *pgd; - - /* VTTBR value associated with above pgd and vmid */ - u64 vttbr; }; struct kvm_arch { @@ -193,6 +195,9 @@ struct kvm_vcpu_arch { /* Stage 2 paging state used by the hardware on next switch */ struct kvm_s2_mmu *hw_mmu; + + /* VTTBR value used by the hardware on next switch */ + u64 hw_vttbr; }; struct kvm_vm_stat { @@ -239,6 +244,7 @@ static inline void kvm_arch_mmu_notifier_invalidate_page(struct kvm *kvm, { } +unsigned int get_kvm_vmid_bits(void); struct kvm_vcpu *kvm_arm_get_running_vcpu(void); struct kvm_vcpu __percpu **kvm_get_running_vcpus(void); void kvm_arm_halt_guest(struct kvm *kvm); diff --git a/arch/arm/include/asm/kvm_mmu.h b/arch/arm/include/asm/kvm_mmu.h index fa6f217..86fdc70 100644 --- a/arch/arm/include/asm/kvm_mmu.h +++ b/arch/arm/include/asm/kvm_mmu.h @@ -221,6 +221,17 @@ static inline unsigned int kvm_get_vmid_bits(void) return 8; } +static inline u64 kvm_get_vttbr(struct kvm_s2_vmid *vmid, + struct kvm_s2_mmu *mmu) +{ + u64 vmid_field, baddr; + + baddr = virt_to_phys(mmu->pgd); + vmid_field = ((u64)vmid->vmid << VTTBR_VMID_SHIFT) & + VTTBR_VMID_MASK(get_kvm_vmid_bits()); + return baddr | vmid_field; +} + #endif /* !__ASSEMBLY__ */ #endif /* __ARM_KVM_MMU_H__ */ diff --git a/arch/arm/kvm/hyp/switch.c b/arch/arm/kvm/hyp/switch.c index 4814671..4798e39 100644 --- a/arch/arm/kvm/hyp/switch.c +++ b/arch/arm/kvm/hyp/switch.c @@ -75,9 +75,7 @@ static void __hyp_text __deactivate_traps(struct kvm_vcpu *vcpu) static void __hyp_text __activate_vm(struct kvm_vcpu *vcpu) { - struct kvm_s2_mmu *mmu = kern_hyp_va(vcpu->arch.hw_mmu); - - write_sysreg(mmu->vttbr, VTTBR); + write_sysreg(vcpu->arch.hw_vttbr, VTTBR); write_sysreg(vcpu->arch.midr, VPIDR); } diff --git a/arch/arm/kvm/hyp/tlb.c b/arch/arm/kvm/hyp/tlb.c index 56f0a49..562ad0b 100644 --- a/arch/arm/kvm/hyp/tlb.c +++ b/arch/arm/kvm/hyp/tlb.c @@ -34,13 +34,12 @@ * As v7 does not support flushing per IPA, just nuke the whole TLB * instead, ignoring the ipa value. */ -void __hyp_text __kvm_tlb_flush_vmid(struct kvm_s2_mmu *mmu) +void __hyp_text __kvm_tlb_flush_vmid(u64 vttbr) { dsb(ishst); /* Switch to requested VMID */ - mmu = kern_hyp_va(mmu); - write_sysreg(mmu->vttbr, VTTBR); + write_sysreg(vttbr, VTTBR); isb(); write_sysreg(0, TLBIALLIS); @@ -50,17 +49,15 @@ void __hyp_text __kvm_tlb_flush_vmid(struct kvm_s2_mmu *mmu) write_sysreg(0, VTTBR); } -void __hyp_text __kvm_tlb_flush_vmid_ipa(struct kvm_s2_mmu *mmu, - phys_addr_t ipa) +void __hyp_text __kvm_tlb_flush_vmid_ipa(u64 vttbr, phys_addr_t ipa) { - __kvm_tlb_flush_vmid(mmu); + __kvm_tlb_flush_vmid(vttbr); } -void __hyp_text __kvm_tlb_flush_local_vmid(struct kvm_s2_mmu *mmu) +void __hyp_text __kvm_tlb_flush_local_vmid(u64 vttbr) { /* Switch to requested VMID */ - mmu = kern_hyp_va(mmu); - write_sysreg(mmu->vttbr, VTTBR); + write_sysreg(vttbr, VTTBR); isb(); write_sysreg(0, TLBIALL); diff --git a/arch/arm64/include/asm/kvm_asm.h b/arch/arm64/include/asm/kvm_asm.h index ff6244f..e492749 100644 --- a/arch/arm64/include/asm/kvm_asm.h +++ b/arch/arm64/include/asm/kvm_asm.h @@ -52,9 +52,9 @@ extern char __kvm_hyp_vector[]; extern void __kvm_flush_vm_context(void); -extern void __kvm_tlb_flush_vmid_ipa(struct kvm_s2_mmu *mmu, phys_addr_t ipa); -extern void __kvm_tlb_flush_vmid(struct kvm_s2_mmu *mmu); -extern void __kvm_tlb_flush_local_vmid(struct kvm_s2_mmu *mmu); +extern void __kvm_tlb_flush_vmid_ipa(u64 vttbr, phys_addr_t ipa); +extern void __kvm_tlb_flush_vmid(u64 vttbr); +extern void __kvm_tlb_flush_local_vmid(u64 vttbr); extern int __kvm_vcpu_run(struct kvm_vcpu *vcpu); diff --git a/arch/arm64/include/asm/kvm_emulate.h b/arch/arm64/include/asm/kvm_emulate.h index 4776bfc..71a3a04 100644 --- a/arch/arm64/include/asm/kvm_emulate.h +++ b/arch/arm64/include/asm/kvm_emulate.h @@ -385,4 +385,12 @@ static inline unsigned long vcpu_data_host_to_guest(struct kvm_vcpu *vcpu, return data; /* Leave LE untouched */ } +static inline struct kvm_s2_vmid *vcpu_get_active_vmid(struct kvm_vcpu *vcpu) +{ + if (unlikely(is_hyp_ctxt(vcpu))) + return &vcpu->kvm->arch.mmu.el2_vmid; + + return &vcpu->kvm->arch.mmu.vmid; +} + #endif /* __ARM64_KVM_EMULATE_H__ */ diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h index e7e9f70..a7edf0e 100644 --- a/arch/arm64/include/asm/kvm_host.h +++ b/arch/arm64/include/asm/kvm_host.h @@ -50,17 +50,19 @@ int kvm_arch_dev_ioctl_check_extension(struct kvm *kvm, long ext); void __extended_idmap_trampoline(phys_addr_t boot_pgd, phys_addr_t idmap_start); -struct kvm_s2_mmu { +struct kvm_s2_vmid { /* The VMID generation used for the virt. memory system */ u64 vmid_gen; u32 vmid; +}; + +struct kvm_s2_mmu { + struct kvm_s2_vmid vmid; + struct kvm_s2_vmid el2_vmid; /* 1-level 2nd stage table and lock */ spinlock_t pgd_lock; pgd_t *pgd; - - /* VTTBR value associated with above pgd and vmid */ - u64 vttbr; }; struct kvm_arch { @@ -337,6 +339,9 @@ struct kvm_vcpu_arch { /* Stage 2 paging state used by the hardware on next switch */ struct kvm_s2_mmu *hw_mmu; + + /* VTTBR value used by the hardware on next switch */ + u64 hw_vttbr; }; #define vcpu_gp_regs(v) (&(v)->arch.ctxt.gp_regs) @@ -394,6 +399,7 @@ static inline void kvm_arch_mmu_notifier_invalidate_page(struct kvm *kvm, { } +unsigned int get_kvm_vmid_bits(void); struct kvm_vcpu *kvm_arm_get_running_vcpu(void); struct kvm_vcpu * __percpu *kvm_get_running_vcpus(void); void kvm_arm_halt_guest(struct kvm *kvm); diff --git a/arch/arm64/include/asm/kvm_mmu.h b/arch/arm64/include/asm/kvm_mmu.h index a89cc22..21c0299 100644 --- a/arch/arm64/include/asm/kvm_mmu.h +++ b/arch/arm64/include/asm/kvm_mmu.h @@ -312,5 +312,16 @@ static inline unsigned int kvm_get_vmid_bits(void) return (cpuid_feature_extract_unsigned_field(reg, ID_AA64MMFR1_VMIDBITS_SHIFT) == 2) ? 16 : 8; } +static inline u64 kvm_get_vttbr(struct kvm_s2_vmid *vmid, + struct kvm_s2_mmu *mmu) +{ + u64 vmid_field, baddr; + + baddr = virt_to_phys(mmu->pgd); + vmid_field = ((u64)vmid->vmid << VTTBR_VMID_SHIFT) & + VTTBR_VMID_MASK(get_kvm_vmid_bits()); + return baddr | vmid_field; +} + #endif /* __ASSEMBLY__ */ #endif /* __ARM64_KVM_MMU_H__ */ diff --git a/arch/arm64/kvm/hyp/switch.c b/arch/arm64/kvm/hyp/switch.c index 8b1b3e9..3626e76 100644 --- a/arch/arm64/kvm/hyp/switch.c +++ b/arch/arm64/kvm/hyp/switch.c @@ -181,9 +181,7 @@ static void __hyp_text __deactivate_traps(struct kvm_vcpu *vcpu) static void __hyp_text __activate_vm(struct kvm_vcpu *vcpu) { - struct kvm_s2_mmu *mmu = kern_hyp_va(vcpu->arch.hw_mmu); - - write_sysreg(mmu->vttbr, vttbr_el2); + write_sysreg(vcpu->arch.hw_vttbr, vttbr_el2); } static void __hyp_text __deactivate_vm(struct kvm_vcpu *vcpu) diff --git a/arch/arm64/kvm/hyp/tlb.c b/arch/arm64/kvm/hyp/tlb.c index 0897678..680b960 100644 --- a/arch/arm64/kvm/hyp/tlb.c +++ b/arch/arm64/kvm/hyp/tlb.c @@ -18,7 +18,7 @@ #include #include -static void __hyp_text __tlb_switch_to_guest_vhe(struct kvm_s2_mmu *mmu) +static void __hyp_text __tlb_switch_to_guest_vhe(u64 vttbr) { u64 val; @@ -29,16 +29,16 @@ static void __hyp_text __tlb_switch_to_guest_vhe(struct kvm_s2_mmu *mmu) * bits. Changing E2H is impossible (goodbye TTBR1_EL2), so * let's flip TGE before executing the TLB operation. */ - write_sysreg(mmu->vttbr, vttbr_el2); + write_sysreg(vttbr, vttbr_el2); val = read_sysreg(hcr_el2); val &= ~HCR_TGE; write_sysreg(val, hcr_el2); isb(); } -static void __hyp_text __tlb_switch_to_guest_nvhe(struct kvm_s2_mmu *mmu) +static void __hyp_text __tlb_switch_to_guest_nvhe(u64 vttbr) { - write_sysreg(mmu->vttbr, vttbr_el2); + write_sysreg(vttbr, vttbr_el2); isb(); } @@ -47,7 +47,7 @@ static hyp_alternate_select(__tlb_switch_to_guest, __tlb_switch_to_guest_vhe, ARM64_HAS_VIRT_HOST_EXTN); -static void __hyp_text __tlb_switch_to_host_vhe(struct kvm_s2_mmu *mmu) +static void __hyp_text __tlb_switch_to_host_vhe(void) { /* * We're done with the TLB operation, let's restore the host's @@ -57,7 +57,7 @@ static void __hyp_text __tlb_switch_to_host_vhe(struct kvm_s2_mmu *mmu) write_sysreg(HCR_HOST_VHE_FLAGS, hcr_el2); } -static void __hyp_text __tlb_switch_to_host_nvhe(struct kvm_s2_mmu *mmu) +static void __hyp_text __tlb_switch_to_host_nvhe(void) { write_sysreg(0, vttbr_el2); } @@ -67,14 +67,12 @@ static hyp_alternate_select(__tlb_switch_to_host, __tlb_switch_to_host_vhe, ARM64_HAS_VIRT_HOST_EXTN); -void __hyp_text __kvm_tlb_flush_vmid_ipa(struct kvm_s2_mmu *mmu, - phys_addr_t ipa) +void __hyp_text __kvm_tlb_flush_vmid_ipa(u64 vttbr, phys_addr_t ipa) { dsb(ishst); /* Switch to requested VMID */ - mmu = kern_hyp_va(mmu); - __tlb_switch_to_guest()(mmu); + __tlb_switch_to_guest()(vttbr); /* * We could do so much better if we had the VA as well. @@ -117,35 +115,33 @@ void __hyp_text __kvm_tlb_flush_vmid_ipa(struct kvm_s2_mmu *mmu, if (!has_vhe() && icache_is_vpipt()) __flush_icache_all(); - __tlb_switch_to_host()(mmu); + __tlb_switch_to_host()(); } -void __hyp_text __kvm_tlb_flush_vmid(struct kvm_s2_mmu *mmu) +void __hyp_text __kvm_tlb_flush_vmid(u64 vttbr) { dsb(ishst); /* Switch to requested VMID */ - mmu = kern_hyp_va(mmu); - __tlb_switch_to_guest()(mmu); + __tlb_switch_to_guest()(vttbr); __tlbi(vmalls12e1is); dsb(ish); isb(); - __tlb_switch_to_host()(mmu); + __tlb_switch_to_host()(); } -void __hyp_text __kvm_tlb_flush_local_vmid(struct kvm_s2_mmu *mmu) +void __hyp_text __kvm_tlb_flush_local_vmid(u64 vttbr) { /* Switch to requested VMID */ - mmu = kern_hyp_va(mmu); - __tlb_switch_to_guest()(mmu); + __tlb_switch_to_guest()(vttbr); __tlbi(vmalle1); dsb(nsh); isb(); - __tlb_switch_to_host()(mmu); + __tlb_switch_to_host()(); } void __hyp_text __kvm_flush_vm_context(void) diff --git a/virt/kvm/arm/arm.c b/virt/kvm/arm/arm.c index bee27bb..41e0654 100644 --- a/virt/kvm/arm/arm.c +++ b/virt/kvm/arm/arm.c @@ -75,6 +75,11 @@ static void kvm_arm_set_running_vcpu(struct kvm_vcpu *vcpu) __this_cpu_write(kvm_arm_running_vcpu, vcpu); } +unsigned int get_kvm_vmid_bits(void) +{ + return kvm_vmid_bits; +} + /** * kvm_arm_get_running_vcpu - get the vcpu running on the current CPU. * Must be called from non-preemptible context @@ -138,7 +143,8 @@ int kvm_arch_init_vm(struct kvm *kvm, unsigned long type) kvm_vgic_early_init(kvm); /* Mark the initial VMID generation invalid */ - kvm->arch.mmu.vmid_gen = 0; + kvm->arch.mmu.vmid.vmid_gen = 0; + kvm->arch.mmu.el2_vmid.vmid_gen = 0; /* The maximum number of VCPUs is limited by the host's GIC model */ kvm->arch.max_vcpus = vgic_present ? @@ -325,6 +331,8 @@ void kvm_arch_vcpu_unblocking(struct kvm_vcpu *vcpu) int kvm_arch_vcpu_init(struct kvm_vcpu *vcpu) { + struct kvm_s2_mmu *mmu = &vcpu->kvm->arch.mmu; + /* Force users to call KVM_ARM_VCPU_INIT */ vcpu->arch.target = -1; bitmap_zero(vcpu->arch.features, KVM_VCPU_MAX_FEATURES); @@ -334,7 +342,7 @@ int kvm_arch_vcpu_init(struct kvm_vcpu *vcpu) kvm_arm_reset_debug_ptr(vcpu); - vcpu->arch.hw_mmu = &vcpu->kvm->arch.mmu; + vcpu->arch.hw_mmu = mmu; return kvm_vgic_vcpu_init(vcpu); } @@ -350,7 +358,10 @@ void kvm_arch_vcpu_load(struct kvm_vcpu *vcpu, int cpu) * over-invalidation doesn't affect correctness. */ if (*last_ran != vcpu->vcpu_id) { - kvm_call_hyp(__kvm_tlb_flush_local_vmid, &vcpu->kvm->arch.mmu); + struct kvm_s2_mmu *mmu = &vcpu->kvm->arch.mmu; + u64 vttbr = kvm_get_vttbr(&mmu->vmid, mmu); + + kvm_call_hyp(__kvm_tlb_flush_local_vmid, vttbr); *last_ran = vcpu->vcpu_id; } @@ -434,36 +445,38 @@ void force_vm_exit(const cpumask_t *mask) /** * need_new_vmid_gen - check that the VMID is still valid - * @kvm: The VM's VMID to check + * @vmid: The VMID to check * * return true if there is a new generation of VMIDs being used * - * The hardware supports only 256 values with the value zero reserved for the - * host, so we check if an assigned value belongs to a previous generation, - * which which requires us to assign a new value. If we're the first to use a - * VMID for the new generation, we must flush necessary caches and TLBs on all - * CPUs. + * The hardware supports a limited set of values with the value zero reserved + * for the host, so we check if an assigned value belongs to a previous + * generation, which which requires us to assign a new value. If we're the + * first to use a VMID for the new generation, we must flush necessary caches + * and TLBs on all CPUs. */ -static bool need_new_vmid_gen(struct kvm_s2_mmu *mmu) +static bool need_new_vmid_gen(struct kvm_s2_vmid *vmid) { - return unlikely(mmu->vmid_gen != atomic64_read(&kvm_vmid_gen)); + return unlikely(vmid->vmid_gen != atomic64_read(&kvm_vmid_gen)); } /** * update_vttbr - Update the VTTBR with a valid VMID before the guest runs * @kvm: The guest that we are about to run - * @mmu: The stage-2 translation context to update + * @vmid: The stage-2 VMID information struct * * Called from kvm_arch_vcpu_ioctl_run before entering the guest to ensure the * VM has a valid VMID, otherwise assigns a new one and flushes corresponding * caches and TLBs. */ -static void update_vttbr(struct kvm *kvm, struct kvm_s2_mmu *mmu) +static void update_vttbr(struct kvm *kvm, struct kvm_s2_vmid *vmid) { - phys_addr_t pgd_phys; - u64 vmid; + struct kvm_s2_mmu *mmu = &kvm->arch.mmu; + struct kvm_vcpu *vcpu; + int i = 0; + u64 new_vttbr; - if (!need_new_vmid_gen(mmu)) + if (!need_new_vmid_gen(vmid)) return; spin_lock(&kvm_vmid_lock); @@ -473,7 +486,7 @@ static void update_vttbr(struct kvm *kvm, struct kvm_s2_mmu *mmu) * already allocated a valid vmid for this vm, then this vcpu should * use the same vmid. */ - if (!need_new_vmid_gen(mmu)) { + if (!need_new_vmid_gen(vmid)) { spin_unlock(&kvm_vmid_lock); return; } @@ -497,17 +510,15 @@ static void update_vttbr(struct kvm *kvm, struct kvm_s2_mmu *mmu) kvm_call_hyp(__kvm_flush_vm_context); } - mmu->vmid_gen = atomic64_read(&kvm_vmid_gen); - mmu->vmid = kvm_next_vmid; + vmid->vmid_gen = atomic64_read(&kvm_vmid_gen); + vmid->vmid = kvm_next_vmid; kvm_next_vmid++; kvm_next_vmid &= (1 << kvm_vmid_bits) - 1; - /* update vttbr to be used with the new vmid */ - pgd_phys = virt_to_phys(mmu->pgd); - BUG_ON(pgd_phys & ~VTTBR_BADDR_MASK); - vmid = ((u64)(mmu->vmid) << VTTBR_VMID_SHIFT) & - VTTBR_VMID_MASK(kvm_vmid_bits); - mmu->vttbr = pgd_phys | vmid; + new_vttbr = kvm_get_vttbr(&mmu->vmid, mmu); + kvm_for_each_vcpu(i, vcpu, kvm) { + vcpu->arch.hw_vttbr = new_vttbr; + } spin_unlock(&kvm_vmid_lock); } @@ -642,7 +653,7 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu, struct kvm_run *run) */ cond_resched(); - update_vttbr(vcpu->kvm, vcpu->arch.hw_mmu); + update_vttbr(vcpu->kvm, vcpu_get_active_vmid(vcpu)); check_vcpu_requests(vcpu); @@ -681,7 +692,7 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu, struct kvm_run *run) */ smp_store_mb(vcpu->mode, IN_GUEST_MODE); - if (ret <= 0 || need_new_vmid_gen(vcpu->arch.hw_mmu) || + if (ret <= 0 || need_new_vmid_gen(vcpu_get_active_vmid(vcpu)) || kvm_request_pending(vcpu)) { vcpu->mode = OUTSIDE_GUEST_MODE; local_irq_enable(); diff --git a/virt/kvm/arm/mmu.c b/virt/kvm/arm/mmu.c index d8ea1f9..0edcf23 100644 --- a/virt/kvm/arm/mmu.c +++ b/virt/kvm/arm/mmu.c @@ -61,12 +61,17 @@ static bool memslot_is_logging(struct kvm_memory_slot *memslot) */ void kvm_flush_remote_tlbs(struct kvm *kvm) { - kvm_call_hyp(__kvm_tlb_flush_vmid, kvm); + struct kvm_s2_mmu *mmu = &kvm->arch.mmu; + u64 vttbr = kvm_get_vttbr(&mmu->vmid, mmu); + + kvm_call_hyp(__kvm_tlb_flush_vmid, vttbr); } static void kvm_tlb_flush_vmid_ipa(struct kvm_s2_mmu *mmu, phys_addr_t ipa) { - kvm_call_hyp(__kvm_tlb_flush_vmid_ipa, mmu, ipa); + u64 vttbr = kvm_get_vttbr(&mmu->vmid, mmu); + + kvm_call_hyp(__kvm_tlb_flush_vmid_ipa, vttbr, ipa); } /* From patchwork Tue Oct 3 03:10:48 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jintack Lim X-Patchwork-Id: 114653 Delivered-To: patch@linaro.org Received: by 10.140.22.163 with SMTP id 32csp1389503qgn; Mon, 2 Oct 2017 20:19:17 -0700 (PDT) X-Received: by 10.99.124.91 with SMTP id l27mr675552pgn.49.1507000757854; Mon, 02 Oct 2017 20:19:17 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1507000757; cv=none; d=google.com; s=arc-20160816; b=vVUG3xnZIkL/1LaF0K7s8C+m/axr2Wm89BC/Yd3Aet+5y4QE0Va0UWjO5H9LYpkVQx IH8k+QstSMMLFnCBuZgmSBMqwLWnMiTALIZrGUGweYtDG/Y8xqDjlVAHKu2YHYau3j8a vWHvoebHoW3AYESn98iYuEmQI4NXsUvzYxvmFRtowCU4TIX3qvmvprTvv/W2PPhPbeA8 pNmkprCf/Ped37CUQCjFff6KuLJ8bA2ZN5NikvwMrbjcDyL9zP22rYBf6AGuHl30Yn7C 2HBqJq4DxkPmckTx5USdpGrrpxBe1JrMeP0aCIyV9NItvlfcRiV9OESmh3TIHMhuWAmh Cwuw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature:arc-authentication-results; bh=1Ucz1J4P5XUKPSNzgO4sG8gFNOHEthQaf/MHvQEb4Vo=; b=FHtrfIjCX/rrGzwp6rKYO6/lpSJ9wZNmUwJDe9ON2vYKwzhoZu+yCuC6gU6uzIknTZ Q44qikcMFcci+WhjsGqQ+KUks+kPb/f1K/uJyuDQBkG6gQL/BTYhCGQkW3oKWuYIhLsk ZMF5oiI8iw1VQ9AzOsOjDX+e1i3QHLnoPZ0ntRNvPVW5xj6bHLy8QiCYvtTPTftyD2O3 P4b/q+05nD2uj95Ena3Tiv5qRzXA5rdPS7+Poa6l0er5hF0GatOWi9QJdsSv1vyJDeH1 +D7dNrrgNxV53RGkidh1GpUKeez/7lEdL/SB0S/QnxIQnHI32Y17NyvNU5lXtYb5wJjG Sa3A== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=VS9z7cHc; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id a1si9503927pld.153.2017.10.02.20.19.17; Mon, 02 Oct 2017 20:19:17 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=VS9z7cHc; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751534AbdJCDTP (ORCPT + 26 others); Mon, 2 Oct 2017 23:19:15 -0400 Received: from mail-it0-f52.google.com ([209.85.214.52]:53430 "EHLO mail-it0-f52.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751667AbdJCDLo (ORCPT ); Mon, 2 Oct 2017 23:11:44 -0400 Received: by mail-it0-f52.google.com with SMTP id 85so9735246ith.2 for ; Mon, 02 Oct 2017 20:11:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=1Ucz1J4P5XUKPSNzgO4sG8gFNOHEthQaf/MHvQEb4Vo=; b=VS9z7cHcBpErAH9DZjxcZLAvGgEKw+0TDzVFYBAMV9bg1e2GmaHy8xMJgtsNutaCdv NeZmVFJL8Y/5Q7/UvxeqQ9JYUh7LHXIhWgNAEHc3DVqeRKmkvk18lXGUeRLnr5r84KSP X2Eb7Hfs1oDhkPUEeB3tGesKYVSn19KqWf7Ng= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=1Ucz1J4P5XUKPSNzgO4sG8gFNOHEthQaf/MHvQEb4Vo=; b=NNB38af0lTlgGPmB5joOowjAh+AJDjLbH/tWy01/n0EaYA/ZO0T+2aThkkpjbeUwtg 6pkBsprxHv6xCASKagF4OkWvVdZsciJRtQkfMYJyGiO5aGoJaf7oYBfwOk1COyTxt/zf w+m/ETGxCivH+Z0e94C3M7RMK7bzYdnBmiJSa26c3Cr5Bv/rIFithNrLTBYE3a34VHLh eyctp2TKG4Ia/9X7wlqHcHpE3PYScRqNFZZJXkBoOZXFXjQ7ZDyoW5iUqknag6PMA65O 92I1RqS89yj1fUl2CcittryJkvcDjIkJ5VKGijX1LhJ8+Gp9tC1Iz50F5EHHMC9seayS UM+A== X-Gm-Message-State: AMCzsaXxoVIkq2gbhCjdBteHQXimpny7R8P3yP5gYlzmTNqVnuR4jie1 T3p9TL+lCAbtHS7DBpcqwnfvLg== X-Google-Smtp-Source: AOwi7QCsZQo6NhiQgypAsAOVhrCYWZuKSJ8bCa+1lqxBpGBdKtfw2+x65A9Q3cUN0IwWcW7PT/W59g== X-Received: by 10.36.83.146 with SMTP id n140mr24509145itb.104.1507000303557; Mon, 02 Oct 2017 20:11:43 -0700 (PDT) Received: from node.jintackl-qv28633.kvmarm-pg0.wisc.cloudlab.us (c220g1-031126.wisc.cloudlab.us. [128.104.222.76]) by smtp.gmail.com with ESMTPSA id h84sm5367193iod.72.2017.10.02.20.11.42 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Mon, 02 Oct 2017 20:11:42 -0700 (PDT) From: Jintack Lim To: christoffer.dall@linaro.org, marc.zyngier@arm.com, kvmarm@lists.cs.columbia.edu Cc: jintack@cs.columbia.edu, pbonzini@redhat.com, rkrcmar@redhat.com, catalin.marinas@arm.com, will.deacon@arm.com, linux@armlinux.org.uk, mark.rutland@arm.com, linux-arm-kernel@lists.infradead.org, kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Jintack Lim Subject: [RFC PATCH v2 06/31] KVM: arm64: Invalidate virtual EL2 TLB entries when needed Date: Mon, 2 Oct 2017 22:10:48 -0500 Message-Id: <1507000273-3735-4-git-send-email-jintack.lim@linaro.org> X-Mailer: git-send-email 1.9.1 In-Reply-To: <1507000273-3735-1-git-send-email-jintack.lim@linaro.org> References: <1507000273-3735-1-git-send-email-jintack.lim@linaro.org> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org From: Christoffer Dall Sometimes when we are invalidating the TLB for a certain S2 MMU context, this context can also have EL2 context associated with it and we have to invalidate this too. Signed-off-by: Christoffer Dall Signed-off-by: Jintack Lim --- virt/kvm/arm/arm.c | 5 +++++ virt/kvm/arm/mmu.c | 23 ++++++++++++++++++++++- 2 files changed, 27 insertions(+), 1 deletion(-) -- 1.9.1 diff --git a/virt/kvm/arm/arm.c b/virt/kvm/arm/arm.c index 41e0654..63dd897 100644 --- a/virt/kvm/arm/arm.c +++ b/virt/kvm/arm/arm.c @@ -362,6 +362,11 @@ void kvm_arch_vcpu_load(struct kvm_vcpu *vcpu, int cpu) u64 vttbr = kvm_get_vttbr(&mmu->vmid, mmu); kvm_call_hyp(__kvm_tlb_flush_local_vmid, vttbr); + + if (mmu->el2_vmid.vmid) { + vttbr = kvm_get_vttbr(&mmu->el2_vmid, mmu); + kvm_call_hyp(__kvm_tlb_flush_local_vmid, vttbr); + } *last_ran = vcpu->vcpu_id; } diff --git a/virt/kvm/arm/mmu.c b/virt/kvm/arm/mmu.c index 0edcf23..184cdc9 100644 --- a/virt/kvm/arm/mmu.c +++ b/virt/kvm/arm/mmu.c @@ -64,7 +64,21 @@ void kvm_flush_remote_tlbs(struct kvm *kvm) struct kvm_s2_mmu *mmu = &kvm->arch.mmu; u64 vttbr = kvm_get_vttbr(&mmu->vmid, mmu); - kvm_call_hyp(__kvm_tlb_flush_vmid, vttbr); + if (!mmu->el2_vmid.vmid) { + /* + * For a normal (i.e. non-nested) guest, flush entries for the + * given VMID * + */ + kvm_call_hyp(__kvm_tlb_flush_vmid, vttbr); + } else { + /* + * When supporting nested virtualization, we can have multiple + * VMIDs in play for each VCPU in the VM, so it's really not + * worth it to try to quiesce the system and flush all the + * VMIDs that may be in use, instead just nuke the whole thing. + */ + kvm_call_hyp(__kvm_flush_vm_context); + } } static void kvm_tlb_flush_vmid_ipa(struct kvm_s2_mmu *mmu, phys_addr_t ipa) @@ -72,6 +86,13 @@ static void kvm_tlb_flush_vmid_ipa(struct kvm_s2_mmu *mmu, phys_addr_t ipa) u64 vttbr = kvm_get_vttbr(&mmu->vmid, mmu); kvm_call_hyp(__kvm_tlb_flush_vmid_ipa, vttbr, ipa); + + if (!mmu->el2_vmid.vmid) { + /* Nothing to do more for a non-nested guest */ + return; + } + vttbr = kvm_get_vttbr(&mmu->el2_vmid, mmu); + kvm_call_hyp(__kvm_tlb_flush_vmid_ipa, vttbr, ipa); } /* From patchwork Tue Oct 3 03:10:49 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jintack Lim X-Patchwork-Id: 114651 Delivered-To: patch@linaro.org Received: by 10.140.22.163 with SMTP id 32csp1389270qgn; Mon, 2 Oct 2017 20:18:54 -0700 (PDT) X-Received: by 10.84.128.35 with SMTP id 32mr16081851pla.385.1507000734156; Mon, 02 Oct 2017 20:18:54 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1507000734; cv=none; d=google.com; s=arc-20160816; b=vzirSctsGmDrql6THX0PkcbkjrO2vd6Zk6HCnMxENGVgfW1J0Pw3FTXh/h6buWH9Hg aFA5YETKO/NjDXUBTsSFVEWhNNwZ0P7stPuyt7Mj4EYZXA8AwUae3qpl43HdYnKVVIVh DbG9MOgpNEgkTtSp4XCifVSX7MWdNE1duoFgi3UHsFuVSDh9y/uo+sA28dTfSPTz3E3j E8Bu/FAouiYlF0Il/j/PGYsYfS6fHINe7s13OcTSqh2CJH/1izs1qOBUGYdfBaSPsnIq dw+4pYP5CLOjIoaXeS/e69O/biSwMRGkvOdM7CzQV4oQjdeg64BD4sCaiFYtsrYsJLFb J55A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature:arc-authentication-results; bh=udZLXUtAMAMuq7BHsx5r5JKmFrn6ke9gBszBXuj7Bi0=; b=yzi8WwiQyQ9PmeR2T/e+POgPGfTnE0fRKbu14YrD/s50qkmIX1nxS9NxerKclQpzDw okeqFkp0miLH2ky+RT8IOIQwKOztYTX+CHefGZLREOCYrQukz9McEwpcVWfyt8F6mdHg TrFB9MJnd+UvxurzqCO4CR387RI2b5wMJkA/VJ8otpluoWp4Wfq3R3gS8ai3HGobDnXw MampgcAgigSnFFIyMRcDTiFHWsVk+XSokDReV3iZAsTb/vV+q4kZaDqPYDdkFC/uvUJX BQKCtZBfIM3o00f/hWQ0NQ5MSX7WEvMNI80iMpKZA3CoGBJP+79pnlSEwOy8+IoYzhc0 8Hvg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=GXoYUNOO; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id e1si9611507pln.792.2017.10.02.20.18.53; Mon, 02 Oct 2017 20:18:54 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=GXoYUNOO; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751484AbdJCDSw (ORCPT + 26 others); Mon, 2 Oct 2017 23:18:52 -0400 Received: from mail-it0-f50.google.com ([209.85.214.50]:46896 "EHLO mail-it0-f50.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751625AbdJCDLp (ORCPT ); Mon, 2 Oct 2017 23:11:45 -0400 Received: by mail-it0-f50.google.com with SMTP id d192so9963696itd.1 for ; Mon, 02 Oct 2017 20:11:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=udZLXUtAMAMuq7BHsx5r5JKmFrn6ke9gBszBXuj7Bi0=; b=GXoYUNOOwK/aja3g+KvIaCQDn+S+TgqWv9gOuvmFiwYRZUqVw6Ta8tafS7DuiK52m+ wwMqdLMtzipeLz2WBQdaFa34LtiEByuPzhCtjSKA17B/vlTi5HbuJoPHg5KT/ZET0NMV aPAsdvppfKKAcZ4RdWfMogPUHJu/bWpcZrypo= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=udZLXUtAMAMuq7BHsx5r5JKmFrn6ke9gBszBXuj7Bi0=; b=bv5jKmytNexFpI1csrt0Ju1qSxTjYfA+ZGKMTZHDqar1Vus/Gg6thmspO8DUf1wA3O te5hjdqZhiWRXmW1Xu9YoO2HrKjpTM5uQd48Ku5DN+EzkRhSIId2E24BOrSNuaNr1hy7 iz3Q+kczFgs8nwjITtRfEb2YA11bguW2ncmVojaLTquGXqvfcbpch3NrYNOG6O9uCm7q rhIdbaodNbtSW2yIlb3j38byfJIvK/9SDuONlSUGrM2DUnILiiQLvG+4oqJnjPowe1sR FB54Fqth70dwI1ZwmwOHhIxUPGhsuqkjkZRE9bpnJdfbSb0BzZ/dRuOQhBB8dOz/ONC/ SYGQ== X-Gm-Message-State: AMCzsaVEwOuPaaQOMtxeACkLWMIxsaD4d+xd+AXQOF1xG3bYtEiQptxO A25DHRSMEVA5VvedWH3mCI4FgA== X-Google-Smtp-Source: AOwi7QAJJMKPq9kNTAuS+QsefdBFW9MaBw/+BKlMPrTGIelJPqfvDTyd/gwrv+yrrhmbYafoh5D4hQ== X-Received: by 10.36.213.133 with SMTP id a127mr5508027itg.76.1507000304924; Mon, 02 Oct 2017 20:11:44 -0700 (PDT) Received: from node.jintackl-qv28633.kvmarm-pg0.wisc.cloudlab.us (c220g1-031126.wisc.cloudlab.us. [128.104.222.76]) by smtp.gmail.com with ESMTPSA id h84sm5367193iod.72.2017.10.02.20.11.43 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Mon, 02 Oct 2017 20:11:44 -0700 (PDT) From: Jintack Lim To: christoffer.dall@linaro.org, marc.zyngier@arm.com, kvmarm@lists.cs.columbia.edu Cc: jintack@cs.columbia.edu, pbonzini@redhat.com, rkrcmar@redhat.com, catalin.marinas@arm.com, will.deacon@arm.com, linux@armlinux.org.uk, mark.rutland@arm.com, linux-arm-kernel@lists.infradead.org, kvm@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [RFC PATCH v2 07/31] KVM: arm64: Setup vttbr_el2 on each VM entry Date: Mon, 2 Oct 2017 22:10:49 -0500 Message-Id: <1507000273-3735-5-git-send-email-jintack.lim@linaro.org> X-Mailer: git-send-email 1.9.1 In-Reply-To: <1507000273-3735-1-git-send-email-jintack.lim@linaro.org> References: <1507000273-3735-1-git-send-email-jintack.lim@linaro.org> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org From: Christoffer Dall Now that the vttbr value will be different depending on the VM's exception level, we set it on each VM entry. We only have one mmu instance at this point, but there will be multiple of them if we come to run nested VMs. Signed-off-by: Christoffer Dall Signed-off-by: Jintack Lim --- arch/arm64/kvm/context.c | 12 ++++++++++++ 1 file changed, 12 insertions(+) -- 1.9.1 diff --git a/arch/arm64/kvm/context.c b/arch/arm64/kvm/context.c index a7811e1..afd1702 100644 --- a/arch/arm64/kvm/context.c +++ b/arch/arm64/kvm/context.c @@ -18,6 +18,7 @@ #include #include #include +#include struct el1_el2_map { enum vcpu_sysreg el1; @@ -174,6 +175,15 @@ static void flush_shadow_el1_sysregs(struct kvm_vcpu *vcpu) flush_shadow_el1_sysregs_nvhe(vcpu); } +static void setup_s2_mmu(struct kvm_vcpu *vcpu) +{ + struct kvm_s2_mmu *mmu = &vcpu->kvm->arch.mmu; + struct kvm_s2_vmid *vmid = vcpu_get_active_vmid(vcpu); + + vcpu->arch.hw_vttbr = kvm_get_vttbr(vmid, mmu); + vcpu->arch.hw_mmu = mmu; +} + /* * List of EL0 and EL1 registers which we allow the virtual EL2 mode to access * directly without trapping. This is possible because the impact of @@ -323,6 +333,8 @@ void kvm_arm_setup_shadow_state(struct kvm_vcpu *vcpu) setup_mpidr_el1(vcpu); ctxt->hw_sys_regs = ctxt->sys_regs; } + + setup_s2_mmu(vcpu); } /** From patchwork Tue Oct 3 03:10:50 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jintack Lim X-Patchwork-Id: 114650 Delivered-To: patch@linaro.org Received: by 10.140.22.163 with SMTP id 32csp1389038qgn; Mon, 2 Oct 2017 20:18:32 -0700 (PDT) X-Received: by 10.159.229.201 with SMTP id t9mr9649824plq.380.1507000712769; Mon, 02 Oct 2017 20:18:32 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1507000712; cv=none; d=google.com; s=arc-20160816; b=SaByom7V7Ws2skirQChfCYyEmqzK67IpzatAAIU4Ee06275kc21X+eX5QLzRcKMs4k AiqlUVsnYyjN8Z22h7AMJem0NrU97gXkzPTM2E2eM8RjopLADVfEVDdgry18LfI2DB8p kLzFGNoJJGLtZ5RkNKDa3cr2Gi629s/qmY35r2EbXEPKCmoiznys/XimehZBaE3aZm9b odfkBQxX7IiYeHiSnhB3qa+OmdGMVGHK8o3gFk0ZeHA2Pf9mgLGccaA0OHdb31gG+o/U kbhjQSmkPlgW+82eEjH+7PPtag8vbolWsue+Zl8+ox6C1IudVC3WZXcZSKjOxpL4aKcX 8GYQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature:arc-authentication-results; bh=qlrdtPzi6aVzAfRLhoDrvonU78QNaCw34syxUIf/eW8=; b=iLZGZzQrHONVGkx6V9WYxMfS3TX9IwPXl0TZogsJ3ibrf4rmJZCOJGbPh6AvQmcA5H HsupE0FJBUjL1hXmJCxDue+VcHpgizfCtADzSpG7ChtiOcdtXRKSsW24OQUmYSV8z/92 CGOtsXXd615SMv0L82lIXGk0m2QJh3uhQ6dbmONxoKV1oNzPx3IAGT2fOof+qQAfjYsj BEyhPvyF127enkCF7CaDndfQqDRws05ZUeuWXgrhCgJyjNzPGRBfKY/acFH+u6tZFtSi VVDtRjjacuLtalYI5CwncoIb1yGlxKjVBqkp7+bazxKvbdKGxz6e9xUfF7pV8x8edsWD JiTw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=SwXZHzmS; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id e1si9611507pln.792.2017.10.02.20.18.32; Mon, 02 Oct 2017 20:18:32 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=SwXZHzmS; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751339AbdJCDSb (ORCPT + 26 others); Mon, 2 Oct 2017 23:18:31 -0400 Received: from mail-it0-f41.google.com ([209.85.214.41]:55822 "EHLO mail-it0-f41.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751879AbdJCDLq (ORCPT ); Mon, 2 Oct 2017 23:11:46 -0400 Received: by mail-it0-f41.google.com with SMTP id 4so9725254itv.4 for ; Mon, 02 Oct 2017 20:11:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=qlrdtPzi6aVzAfRLhoDrvonU78QNaCw34syxUIf/eW8=; b=SwXZHzmS2tNud0+Hr/EA7W0ToTUNHFy0FnABe2/q9jAODPGL+csf5Pm5386+bWmv2j oyuqJu+NHpDeFLj79IahZWlNMmxDzyd+8ssHDTzuo7GgWQx+siQSByBuFIzKwC8nn/Az g7XtwiOWphejeIurUU4F2kOtRa65KCVkwJDe4= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=qlrdtPzi6aVzAfRLhoDrvonU78QNaCw34syxUIf/eW8=; b=VTD2R5uhrvH9xIi7UyhseN1veJZ/u10cHytsoOHs4Hb74cUesKXH3M8WDPuvixV1Up vyt6UyJj2z7YfA1ardzY/swtxOEW+RLAjNvHKlW2PyP07wR3O+GuyLusgwQNbPkbzPEu JtRIGFxYhfSq1S/yRBjlpSdeeVtJsYcGwyN1OIML6zyZ3d/lcuvUPYTYl6bhUQo3KdAr poHHFact4HIErmAGqs6ukVVzJAlEoxxM25gHCeITVeMPgEfT8aBDUAPeLJXHGO28XFhq HHUJAX8MMwCqVegjcQ1ATAnYkJFRGymtXvks+9rmsylPauTZZaJrO7AmRqoOkTtQf4AV cMzg== X-Gm-Message-State: AMCzsaXHeDocY90aD/r3dxMa/xg3Um1+pC9VDZrK1g/YKLT+4lb3oWNr u9UOpi3yHRNqMGawHWaVUNsDzg== X-Google-Smtp-Source: AOwi7QDQ7NUypN+TkgKF21KXAJ7q5076zKpxcI33zdYsE52KmdpgOyGx8BBLG777/vwGbCF8C7dmKw== X-Received: by 10.36.32.71 with SMTP id t68mr19442724itt.128.1507000306058; Mon, 02 Oct 2017 20:11:46 -0700 (PDT) Received: from node.jintackl-qv28633.kvmarm-pg0.wisc.cloudlab.us (c220g1-031126.wisc.cloudlab.us. [128.104.222.76]) by smtp.gmail.com with ESMTPSA id h84sm5367193iod.72.2017.10.02.20.11.44 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Mon, 02 Oct 2017 20:11:45 -0700 (PDT) From: Jintack Lim To: christoffer.dall@linaro.org, marc.zyngier@arm.com, kvmarm@lists.cs.columbia.edu Cc: jintack@cs.columbia.edu, pbonzini@redhat.com, rkrcmar@redhat.com, catalin.marinas@arm.com, will.deacon@arm.com, linux@armlinux.org.uk, mark.rutland@arm.com, linux-arm-kernel@lists.infradead.org, kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Jintack Lim Subject: [RFC PATCH v2 08/31] KVM: arm/arm64: Make mmu functions non-static Date: Mon, 2 Oct 2017 22:10:50 -0500 Message-Id: <1507000273-3735-6-git-send-email-jintack.lim@linaro.org> X-Mailer: git-send-email 1.9.1 In-Reply-To: <1507000273-3735-1-git-send-email-jintack.lim@linaro.org> References: <1507000273-3735-1-git-send-email-jintack.lim@linaro.org> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org From: Christoffer Dall Make mmu functions non-static so that we can reuse those functions to support mmu for the nested VMs. Signed-off-by: Christoffer Dall Signed-off-by: Jintack Lim --- arch/arm64/include/asm/kvm_mmu.h | 9 ++++ virt/kvm/arm/mmu.c | 94 +++++++++++++++++++++++----------------- 2 files changed, 64 insertions(+), 39 deletions(-) -- 1.9.1 diff --git a/arch/arm64/include/asm/kvm_mmu.h b/arch/arm64/include/asm/kvm_mmu.h index 21c0299..bceaec1 100644 --- a/arch/arm64/include/asm/kvm_mmu.h +++ b/arch/arm64/include/asm/kvm_mmu.h @@ -145,9 +145,18 @@ static inline unsigned long __kern_hyp_va(unsigned long v) void stage2_unmap_vm(struct kvm *kvm); int kvm_alloc_stage2_pgd(struct kvm *kvm); +int __kvm_alloc_stage2_pgd(struct kvm_s2_mmu *mmu); void kvm_free_stage2_pgd(struct kvm *kvm); +void __kvm_free_stage2_pgd(struct kvm *kvm, struct kvm_s2_mmu *mmu); int kvm_phys_addr_ioremap(struct kvm *kvm, phys_addr_t guest_ipa, phys_addr_t pa, unsigned long size, bool writable); +void kvm_unmap_stage2_range(struct kvm *kvm, struct kvm_s2_mmu *mmu, + phys_addr_t start, u64 size); +void kvm_stage2_wp_range(struct kvm *kvm, struct kvm_s2_mmu *mmu, + phys_addr_t addr, phys_addr_t end); +void kvm_stage2_flush_range(struct kvm_s2_mmu *mmu, + phys_addr_t start, phys_addr_t end); + int kvm_handle_guest_abort(struct kvm_vcpu *vcpu, struct kvm_run *run); diff --git a/virt/kvm/arm/mmu.c b/virt/kvm/arm/mmu.c index 184cdc9..ca10799 100644 --- a/virt/kvm/arm/mmu.c +++ b/virt/kvm/arm/mmu.c @@ -307,7 +307,7 @@ static void unmap_stage2_puds(struct kvm_s2_mmu *mmu, pgd_t *pgd, } /** - * unmap_stage2_range -- Clear stage2 page table entries to unmap a range + * kvm_unmap_stage2_range -- Clear stage2 page table entries to unmap a range * @kvm: The VM pointer * @start: The intermediate physical base address of the range to unmap * @size: The size of the area to unmap @@ -317,12 +317,12 @@ static void unmap_stage2_puds(struct kvm_s2_mmu *mmu, pgd_t *pgd, * destroying the VM), otherwise another faulting VCPU may come in and mess * with things behind our backs. */ -static void unmap_stage2_range(struct kvm *kvm, phys_addr_t start, u64 size) +void kvm_unmap_stage2_range(struct kvm *kvm, struct kvm_s2_mmu *mmu, + phys_addr_t start, u64 size) { pgd_t *pgd; phys_addr_t addr = start, end = start + size; phys_addr_t next; - struct kvm_s2_mmu *mmu = &kvm->arch.mmu; assert_spin_locked(&kvm->mmu_lock); pgd = mmu->pgd + stage2_pgd_index(addr); @@ -391,11 +391,10 @@ static void stage2_flush_puds(pgd_t *pgd, phys_addr_t addr, phys_addr_t end) } while (pud++, addr = next, addr != end); } -static void stage2_flush_memslot(struct kvm_s2_mmu *mmu, - struct kvm_memory_slot *memslot) +void kvm_stage2_flush_range(struct kvm_s2_mmu *mmu, + phys_addr_t start, phys_addr_t end) { - phys_addr_t addr = memslot->base_gfn << PAGE_SHIFT; - phys_addr_t end = addr + PAGE_SIZE * memslot->npages; + phys_addr_t addr = start; phys_addr_t next; pgd_t *pgd; @@ -406,6 +405,15 @@ static void stage2_flush_memslot(struct kvm_s2_mmu *mmu, } while (pgd++, addr = next, addr != end); } +static void stage2_flush_memslot(struct kvm_s2_mmu *mmu, + struct kvm_memory_slot *memslot) +{ + phys_addr_t start = memslot->base_gfn << PAGE_SHIFT; + phys_addr_t end = start + PAGE_SIZE * memslot->npages; + + kvm_stage2_flush_range(mmu, start, end); +} + /** * stage2_flush_vm - Invalidate cache for pages mapped in stage 2 * @kvm: The struct kvm pointer @@ -762,21 +770,9 @@ int create_hyp_io_mappings(void *from, void *to, phys_addr_t phys_addr) __phys_to_pfn(phys_addr), PAGE_HYP_DEVICE); } -/** - * kvm_alloc_stage2_pgd - allocate level-1 table for stage-2 translation. - * @kvm: The KVM struct pointer for the VM. - * - * Allocates only the stage-2 HW PGD level table(s) (can support either full - * 40-bit input addresses or limited to 32-bit input addresses). Clears the - * allocated pages. - * - * Note we don't need locking here as this is only called when the VM is - * created, which can only be done once. - */ -int kvm_alloc_stage2_pgd(struct kvm *kvm) +int __kvm_alloc_stage2_pgd(struct kvm_s2_mmu *mmu) { pgd_t *pgd; - struct kvm_s2_mmu *mmu = &kvm->arch.mmu; if (mmu->pgd != NULL) { kvm_err("kvm_arch already initialized?\n"); @@ -793,6 +789,22 @@ int kvm_alloc_stage2_pgd(struct kvm *kvm) return 0; } +/** + * kvm_alloc_stage2_pgd - allocate level-1 table for stage-2 translation. + * @kvm: The KVM struct pointer for the VM. + * + * Allocates only the stage-2 HW PGD level table(s) (can support either full + * 40-bit input addresses or limited to 32-bit input addresses). Clears the + * allocated pages. + * + * Note we don't need locking here as this is only called when the VM is + * created, which can only be done once. + */ +int kvm_alloc_stage2_pgd(struct kvm *kvm) +{ + return __kvm_alloc_stage2_pgd(&kvm->arch.mmu); +} + static void stage2_unmap_memslot(struct kvm *kvm, struct kvm_memory_slot *memslot) { @@ -828,7 +840,8 @@ static void stage2_unmap_memslot(struct kvm *kvm, if (!(vma->vm_flags & VM_PFNMAP)) { gpa_t gpa = addr + (vm_start - memslot->userspace_addr); - unmap_stage2_range(kvm, gpa, vm_end - vm_start); + kvm_unmap_stage2_range(kvm, &kvm->arch.mmu, gpa, + vm_end - vm_start); } hva = vm_end; } while (hva < reg_end); @@ -860,22 +873,13 @@ void stage2_unmap_vm(struct kvm *kvm) srcu_read_unlock(&kvm->srcu, idx); } -/** - * kvm_free_stage2_pgd - free all stage-2 tables - * @kvm: The KVM struct pointer for the VM. - * - * Walks the level-1 page table pointed to by kvm->arch.mmu.pgd and frees all - * underlying level-2 and level-3 tables before freeing the actual level-1 table - * and setting the struct pointer to NULL. - */ -void kvm_free_stage2_pgd(struct kvm *kvm) +void __kvm_free_stage2_pgd(struct kvm *kvm, struct kvm_s2_mmu *mmu) { void *pgd = NULL; - struct kvm_s2_mmu *mmu = &kvm->arch.mmu; spin_lock(&kvm->mmu_lock); if (mmu->pgd) { - unmap_stage2_range(kvm, 0, KVM_PHYS_SIZE); + kvm_unmap_stage2_range(kvm, mmu, 0, KVM_PHYS_SIZE); pgd = READ_ONCE(mmu->pgd); mmu->pgd = NULL; } @@ -885,6 +889,18 @@ void kvm_free_stage2_pgd(struct kvm *kvm) if (pgd) free_pages_exact(pgd, S2_PGD_SIZE); } +/** + * kvm_free_stage2_pgd - free all stage-2 tables + * @kvm: The KVM struct pointer for the VM. + * + * Walks the level-1 page table pointed to by kvm->arch.mmu.pgd and frees all + * underlying level-2 and level-3 tables before freeing the actual level-1 table + * and setting the struct pointer to NULL. + */ +void kvm_free_stage2_pgd(struct kvm *kvm) +{ + __kvm_free_stage2_pgd(kvm, &kvm->arch.mmu); +} static pud_t *stage2_get_pud(struct kvm_s2_mmu *mmu, struct kvm_mmu_memory_cache *cache, @@ -1204,7 +1220,7 @@ static void stage2_wp_puds(pgd_t *pgd, phys_addr_t addr, phys_addr_t end) * @addr: Start address of range * @end: End address of range */ -static void stage2_wp_range(struct kvm *kvm, struct kvm_s2_mmu *mmu, +void kvm_stage2_wp_range(struct kvm *kvm, struct kvm_s2_mmu *mmu, phys_addr_t addr, phys_addr_t end) { pgd_t *pgd; @@ -1251,7 +1267,7 @@ void kvm_mmu_wp_memory_region(struct kvm *kvm, int slot) phys_addr_t end = (memslot->base_gfn + memslot->npages) << PAGE_SHIFT; spin_lock(&kvm->mmu_lock); - stage2_wp_range(kvm, &kvm->arch.mmu, start, end); + kvm_stage2_wp_range(kvm, &kvm->arch.mmu, start, end); spin_unlock(&kvm->mmu_lock); kvm_flush_remote_tlbs(kvm); } @@ -1275,7 +1291,7 @@ static void kvm_mmu_write_protect_pt_masked(struct kvm *kvm, phys_addr_t start = (base_gfn + __ffs(mask)) << PAGE_SHIFT; phys_addr_t end = (base_gfn + __fls(mask) + 1) << PAGE_SHIFT; - stage2_wp_range(kvm, &kvm->arch.mmu, start, end); + kvm_stage2_wp_range(kvm, &kvm->arch.mmu, start, end); } /* @@ -1626,7 +1642,7 @@ static int handle_hva_to_gpa(struct kvm *kvm, static int kvm_unmap_hva_handler(struct kvm *kvm, gpa_t gpa, u64 size, void *data) { - unmap_stage2_range(kvm, gpa, size); + kvm_unmap_stage2_range(kvm, &kvm->arch.mmu, gpa, size); return 0; } @@ -1938,8 +1954,8 @@ int kvm_arch_prepare_memory_region(struct kvm *kvm, spin_lock(&kvm->mmu_lock); if (ret) - unmap_stage2_range(kvm, mem->guest_phys_addr, - mem->memory_size); + kvm_unmap_stage2_range(kvm, &kvm->arch.mmu, + mem->guest_phys_addr, mem->memory_size); else stage2_flush_memslot(&kvm->arch.mmu, memslot); spin_unlock(&kvm->mmu_lock); @@ -1975,7 +1991,7 @@ void kvm_arch_flush_shadow_memslot(struct kvm *kvm, phys_addr_t size = slot->npages << PAGE_SHIFT; spin_lock(&kvm->mmu_lock); - unmap_stage2_range(kvm, gpa, size); + kvm_unmap_stage2_range(kvm, &kvm->arch.mmu, gpa, size); spin_unlock(&kvm->mmu_lock); } From patchwork Tue Oct 3 03:10:52 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jintack Lim X-Patchwork-Id: 114645 Delivered-To: patch@linaro.org Received: by 10.140.22.163 with SMTP id 32csp1387655qgn; Mon, 2 Oct 2017 20:16:30 -0700 (PDT) X-Received: by 10.101.72.132 with SMTP id n4mr14455369pgs.118.1507000590539; Mon, 02 Oct 2017 20:16:30 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1507000590; cv=none; d=google.com; s=arc-20160816; b=FAWBUwo/x4RahOflWmf3xoY+x9m+xRxrAZM90ZPJZ07lHrIGDtOfP4el71ysd00bvl I24sQq/k8uFKxnWOT3/lPTSPSYssNNgGlhwnB8ueA/kRJDEVaa42Eyka6eqYNIkbdpxu 7oTW7LQeUE4N5iEfuofZUwmMUzwUlsquNhQmaqN4AowATHuatWVMG0AsQgEkjHeV/tsT rYwE2LclPpe1/l3Nb5OscCcVTtMbmMJh5Jy258gyTIU07XzluF4WgxmUccIPgT5E5/+V 7bR70gYLpRTrW3WoSj6nKRebp3fUH94tjNQuk3NPHHDparJlPHDGuoFeVGQ5Bv+SzVTz ljbQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature:arc-authentication-results; bh=MQcM1FcXOvspeWAnNJawmA0pVa8r9+tl5UYC5/lqzjE=; b=Z3WQ46Ma8skvyEma+Oa+03dm+hPCzT2/uDtwm87SS+pJUvqdbeLMfXPvSwi19JrGpc Z8d9alhPOIQka/yszEWCBY+ZunADHqYtEDpw9HWqBIyaMBISJhubpk9GpOJk8+239DJA Xm4xJt5f0JQwQ83psk0qcoSm0hZlox6Cb2r5tiiu0JBIkpkzwqodRk/1E35jAtCHcuKZ v5HuRVeLvGDHWoF/BEMtc15pZC67gEbHAx0/UKJHfLOGF2/BiyzQXlhjnzMexr8Ec7lY Q0Rh8Zj0nk2npGao+PKGE8VmkcF4Po9UD/IKYIlPCF48HDh5H4TCO2zFkpNT0XAJDd0s wtpg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=TRpn3e+2; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id y92si374480plb.588.2017.10.02.20.16.30; Mon, 02 Oct 2017 20:16:30 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=TRpn3e+2; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751952AbdJCDL5 (ORCPT + 26 others); Mon, 2 Oct 2017 23:11:57 -0400 Received: from mail-it0-f45.google.com ([209.85.214.45]:51565 "EHLO mail-it0-f45.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751896AbdJCDLt (ORCPT ); Mon, 2 Oct 2017 23:11:49 -0400 Received: by mail-it0-f45.google.com with SMTP id w1so283155itd.0 for ; Mon, 02 Oct 2017 20:11:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=MQcM1FcXOvspeWAnNJawmA0pVa8r9+tl5UYC5/lqzjE=; b=TRpn3e+2rCiow3pZdAdv+YO53xI1ool+eZKvZrcbYOY97xrR1us3q1GuKSdJUeCtqe og8FwBNSR4KkQa30Obtk/LIkw8xb+n9ToMl5jG8u5byAz7tjztYXtsCH+XKop6QhJaWj QQ8ZBhVhsYj9TWqWMuxnElJE/WArDh3YnvQwo= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=MQcM1FcXOvspeWAnNJawmA0pVa8r9+tl5UYC5/lqzjE=; b=EbzetxzzKp85fmTIoCL/ltWFulQTVREXp0/cvcDP3nyzuZ2jLsExuDQZPetisgx/vL HwrC1I3BuWkFlQtMtuKL1SQ2By7iHbrz3QRMOcEAKXIQiMj37CQlPjeFZno+zGEBYUGA 5P017kw7wD0o8Tn6Hj17BNeZcBD4P+/SGif0iN7H3zpP9UBkir43kn2LvW+adwa78vlh qWWhNqzz1Gx7inLNckx0M3DX/gKekwSTNJ/FP2tP3MUNLD2pic07QF2E0OfSD7TYzvVQ 0714QedUaLOvncHnz1h6ob/ewu0+lr5NSc845GmeEzzexCvHHiwJ6xZRCjkxk/b5QsBx i7YQ== X-Gm-Message-State: AHPjjUhUGAMdzL0tWqx0RZWZ6zdAHWpaZts6HsGhc0P9KT/ne7Avnr5x +K+zMuTLM+5eHOF7wlefPMdZeg== X-Google-Smtp-Source: AOwi7QBs3oYX0GYs1RFm66nB92LhQLT3RJU2q1cBjLSi3Yk3I8zaVK0BC6Pw0w7m76ENmoDGHQpa2A== X-Received: by 10.36.77.66 with SMTP id l63mr22561239itb.49.1507000308630; Mon, 02 Oct 2017 20:11:48 -0700 (PDT) Received: from node.jintackl-qv28633.kvmarm-pg0.wisc.cloudlab.us (c220g1-031126.wisc.cloudlab.us. [128.104.222.76]) by smtp.gmail.com with ESMTPSA id h84sm5367193iod.72.2017.10.02.20.11.47 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Mon, 02 Oct 2017 20:11:48 -0700 (PDT) From: Jintack Lim To: christoffer.dall@linaro.org, marc.zyngier@arm.com, kvmarm@lists.cs.columbia.edu Cc: jintack@cs.columbia.edu, pbonzini@redhat.com, rkrcmar@redhat.com, catalin.marinas@arm.com, will.deacon@arm.com, linux@armlinux.org.uk, mark.rutland@arm.com, linux-arm-kernel@lists.infradead.org, kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Jintack Lim Subject: [RFC PATCH v2 10/31] KVM: arm/arm64: Unmap/flush shadow stage 2 page tables Date: Mon, 2 Oct 2017 22:10:52 -0500 Message-Id: <1507000273-3735-8-git-send-email-jintack.lim@linaro.org> X-Mailer: git-send-email 1.9.1 In-Reply-To: <1507000273-3735-1-git-send-email-jintack.lim@linaro.org> References: <1507000273-3735-1-git-send-email-jintack.lim@linaro.org> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org From: Christoffer Dall Unmap/flush shadow stage 2 page tables for the nested VMs as well as the stage 2 page table for the guest hypervisor. Note: A bunch of the code in mmu.c relating to MMU notifiers is currently dealt with in an extremely abrupt way, for example by clearing out an entire shadow stage-2 table. This will be handled in a more efficient way using the reverse mapping feature in a later version of the patch series. Signed-off-by: Christoffer Dall Signed-off-by: Jintack Lim --- Notes: v1-->v2: - Removed an unnecessary iteration for each vcpu in kvm_nested_s2_all_vcpus_*() functions and remove all_vcpus in the function names; a list of nested mmu is per VM, not per vcpu. - Renamed kvm_nested_s2_unmap() to kvm_nested_s2_clear() - Renamed kvm_nested_s2_teardown() to kvm_nested_s2_free() - Removed the unused kvm_nested_s2_init() function. arch/arm/include/asm/kvm_mmu.h | 6 ++++++ arch/arm64/include/asm/kvm_mmu.h | 5 +++++ arch/arm64/kvm/mmu-nested.c | 40 ++++++++++++++++++++++++++++++++++++++++ virt/kvm/arm/arm.c | 6 +++++- virt/kvm/arm/mmu.c | 17 +++++++++++++++++ 5 files changed, 73 insertions(+), 1 deletion(-) -- 1.9.1 diff --git a/arch/arm/include/asm/kvm_mmu.h b/arch/arm/include/asm/kvm_mmu.h index 86fdc70..d3eafc5 100644 --- a/arch/arm/include/asm/kvm_mmu.h +++ b/arch/arm/include/asm/kvm_mmu.h @@ -221,6 +221,12 @@ static inline unsigned int kvm_get_vmid_bits(void) return 8; } +static inline void kvm_nested_s2_unmap(struct kvm_vcpu *vcpu) { } +static inline void kvm_nested_s2_free(struct kvm *kvm) { } +static inline void kvm_nested_s2_wp(struct kvm *kvm) { } +static inline void kvm_nested_s2_clear(struct kvm *kvm) { } +static inline void kvm_nested_s2_flush(struct kvm *kvm) { } + static inline u64 kvm_get_vttbr(struct kvm_s2_vmid *vmid, struct kvm_s2_mmu *mmu) { diff --git a/arch/arm64/include/asm/kvm_mmu.h b/arch/arm64/include/asm/kvm_mmu.h index 452912f..7fc7a83 100644 --- a/arch/arm64/include/asm/kvm_mmu.h +++ b/arch/arm64/include/asm/kvm_mmu.h @@ -325,6 +325,11 @@ static inline unsigned int kvm_get_vmid_bits(void) struct kvm_nested_s2_mmu *get_nested_mmu(struct kvm_vcpu *vcpu, u64 vttbr); struct kvm_s2_mmu *vcpu_get_active_s2_mmu(struct kvm_vcpu *vcpu); void update_nested_s2_mmu(struct kvm_vcpu *vcpu); +void kvm_nested_s2_unmap(struct kvm_vcpu *vcpu); +void kvm_nested_s2_free(struct kvm *kvm); +void kvm_nested_s2_wp(struct kvm *kvm); +void kvm_nested_s2_clear(struct kvm *kvm); +void kvm_nested_s2_flush(struct kvm *kvm); static inline u64 kvm_get_vttbr(struct kvm_s2_vmid *vmid, struct kvm_s2_mmu *mmu) diff --git a/arch/arm64/kvm/mmu-nested.c b/arch/arm64/kvm/mmu-nested.c index c436daf..3ee20f2 100644 --- a/arch/arm64/kvm/mmu-nested.c +++ b/arch/arm64/kvm/mmu-nested.c @@ -1,6 +1,7 @@ /* * Copyright (C) 2017 - Columbia University and Linaro Ltd. * Author: Jintack Lim + * Author: Christoffer Dall * * This program is free software; you can redistribute it and/or modify * it under the terms of the GNU General Public License version 2 as @@ -21,6 +22,45 @@ #include #include +/* expects kvm->mmu_lock to be held */ +void kvm_nested_s2_wp(struct kvm *kvm) +{ + struct kvm_nested_s2_mmu *nested_mmu; + struct list_head *nested_mmu_list = &kvm->arch.nested_mmu_list; + + list_for_each_entry_rcu(nested_mmu, nested_mmu_list, list) + kvm_stage2_wp_range(kvm, &nested_mmu->mmu, 0, KVM_PHYS_SIZE); +} + +/* expects kvm->mmu_lock to be held */ +void kvm_nested_s2_clear(struct kvm *kvm) +{ + struct kvm_nested_s2_mmu *nested_mmu; + struct list_head *nested_mmu_list = &kvm->arch.nested_mmu_list; + + list_for_each_entry_rcu(nested_mmu, nested_mmu_list, list) + kvm_unmap_stage2_range(kvm, &nested_mmu->mmu, 0, KVM_PHYS_SIZE); +} + +/* expects kvm->mmu_lock to be held */ +void kvm_nested_s2_flush(struct kvm *kvm) +{ + struct kvm_nested_s2_mmu *nested_mmu; + struct list_head *nested_mmu_list = &kvm->arch.nested_mmu_list; + + list_for_each_entry_rcu(nested_mmu, nested_mmu_list, list) + kvm_stage2_flush_range(&nested_mmu->mmu, 0, KVM_PHYS_SIZE); +} + +void kvm_nested_s2_free(struct kvm *kvm) +{ + struct kvm_nested_s2_mmu *nested_mmu; + struct list_head *nested_mmu_list = &kvm->arch.nested_mmu_list; + + list_for_each_entry_rcu(nested_mmu, nested_mmu_list, list) + __kvm_free_stage2_pgd(kvm, &nested_mmu->mmu); +} + static struct kvm_nested_s2_mmu *lookup_nested_mmu(struct kvm_vcpu *vcpu, u64 vttbr) { diff --git a/virt/kvm/arm/arm.c b/virt/kvm/arm/arm.c index 4548d77..08706f8 100644 --- a/virt/kvm/arm/arm.c +++ b/virt/kvm/arm/arm.c @@ -187,6 +187,8 @@ void kvm_arch_destroy_vm(struct kvm *kvm) free_percpu(kvm->arch.last_vcpu_ran); kvm->arch.last_vcpu_ran = NULL; + kvm_nested_s2_free(kvm); + for (i = 0; i < KVM_MAX_VCPUS; ++i) { if (kvm->vcpus[i]) { kvm_arch_vcpu_free(kvm->vcpus[i]); @@ -926,8 +928,10 @@ static int kvm_arch_vcpu_ioctl_vcpu_init(struct kvm_vcpu *vcpu, * Ensure a rebooted VM will fault in RAM pages and detect if the * guest MMU is turned off and flush the caches as needed. */ - if (vcpu->arch.has_run_once) + if (vcpu->arch.has_run_once) { stage2_unmap_vm(vcpu->kvm); + kvm_nested_s2_clear(vcpu->kvm); + } vcpu_reset_hcr(vcpu); diff --git a/virt/kvm/arm/mmu.c b/virt/kvm/arm/mmu.c index ca10799..3143f81 100644 --- a/virt/kvm/arm/mmu.c +++ b/virt/kvm/arm/mmu.c @@ -434,6 +434,8 @@ static void stage2_flush_vm(struct kvm *kvm) kvm_for_each_memslot(memslot, slots) stage2_flush_memslot(&kvm->arch.mmu, memslot); + kvm_nested_s2_flush(kvm); + spin_unlock(&kvm->mmu_lock); srcu_read_unlock(&kvm->srcu, idx); } @@ -1268,6 +1270,7 @@ void kvm_mmu_wp_memory_region(struct kvm *kvm, int slot) spin_lock(&kvm->mmu_lock); kvm_stage2_wp_range(kvm, &kvm->arch.mmu, start, end); + kvm_nested_s2_wp(kvm); spin_unlock(&kvm->mmu_lock); kvm_flush_remote_tlbs(kvm); } @@ -1306,6 +1309,7 @@ void kvm_arch_mmu_enable_log_dirty_pt_masked(struct kvm *kvm, gfn_t gfn_offset, unsigned long mask) { kvm_mmu_write_protect_pt_masked(kvm, slot, gfn_offset, mask); + kvm_nested_s2_wp(kvm); } static void coherent_cache_guest_page(struct kvm_vcpu *vcpu, kvm_pfn_t pfn, @@ -1643,6 +1647,7 @@ static int handle_hva_to_gpa(struct kvm *kvm, static int kvm_unmap_hva_handler(struct kvm *kvm, gpa_t gpa, u64 size, void *data) { kvm_unmap_stage2_range(kvm, &kvm->arch.mmu, gpa, size); + kvm_nested_s2_clear(kvm); return 0; } @@ -1682,6 +1687,7 @@ static int kvm_set_spte_handler(struct kvm *kvm, gpa_t gpa, u64 size, void *data * through this calling path. */ stage2_set_pte(&kvm->arch.mmu, NULL, gpa, pte, 0); + kvm_nested_s2_clear(kvm); return 0; } @@ -1716,6 +1722,11 @@ static int kvm_age_hva_handler(struct kvm *kvm, gpa_t gpa, u64 size, void *data) if (pte_none(*pte)) return 0; + /* + * TODO: Handle nested_mmu structures here using the reverse mapping in + * a later version of patch series. + */ + return stage2_ptep_test_and_clear_young(pte); } @@ -1736,6 +1747,11 @@ static int kvm_test_age_hva_handler(struct kvm *kvm, gpa_t gpa, u64 size, void * if (!pte_none(*pte)) /* Just a page... */ return pte_young(*pte); + /* + * TODO: Handle nested_mmu structures here using the reverse mapping in + * a later version of patch series. + */ + return 0; } @@ -1992,6 +2008,7 @@ void kvm_arch_flush_shadow_memslot(struct kvm *kvm, spin_lock(&kvm->mmu_lock); kvm_unmap_stage2_range(kvm, &kvm->arch.mmu, gpa, size); + kvm_nested_s2_clear(kvm); spin_unlock(&kvm->mmu_lock); } From patchwork Tue Oct 3 03:10:53 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jintack Lim X-Patchwork-Id: 114647 Delivered-To: patch@linaro.org Received: by 10.140.22.163 with SMTP id 32csp1388526qgn; Mon, 2 Oct 2017 20:17:45 -0700 (PDT) X-Received: by 10.98.141.28 with SMTP id z28mr16036922pfd.92.1507000664919; Mon, 02 Oct 2017 20:17:44 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1507000664; cv=none; d=google.com; s=arc-20160816; b=bamt+mh8VYGNG3P2FoMtMYHMeDTfn3IWbGDqdTO5X0gRSccW0et5NYpuBmLG2uq5+6 kAAmSCpt/7rl+cILbUEz7mEfm587FSWYSBlUanldByJtXAL5cH49d2tgDtsozEhJMw1V DHD6Qwfoz+As6EpKQxPH9JSDfjYxKno+ka4jIQCaPdV07b7MCn9fdZHTl2apw2Qn9CQo vYspQrR7in1BOjHUA7kAOlVG5u8/aeMxSdc3Z9xPTYU+JS/E8maENoT0mFCGJ75kzkv2 ADH0OnZgpXypPkWa3oy4RtJfZ3uxrhAqlIev7lYyk9SzOhTfSYJ+HkRXO/cd5AHVE91I tUpw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature:arc-authentication-results; bh=VWbCXob9IDcnI5U4QVHByko8cqaKfMTbw2HAwPif6Cc=; b=RaiCYSv/fjXwiYTRKQ7K9CFcgsBro/Pd8ODQPhuzeuae+EkLFHkzopjvbRi723/vWX md5SPoeU3PyVsEiJRVHdDD+LM0NvU6G2rQbbQpdZTNy9lQZhgtZFkQEjHQs1ys6ojF56 YSAQDwmr3cxU9J/1CsrGCIsT5gEdOxQQy0BVdMUpr5lFjnRap6zF4FG9LLzHnLSMfuCd yPBPpO5GGmuxV8pKxKYnDYcaYR66K9GhXF7PjTfWvXkVKY2SC+X8PfYXe/CE0wpdSKsa 1YisscM2S4hFj1oZYzTY3uQubRTbz/RFKTYqaNGaF9utAKC84aMtTrDJBogmb6o7nfMu yL0w== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=eAxNHGLX; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id a59si9473234plc.638.2017.10.02.20.17.44; Mon, 02 Oct 2017 20:17:44 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=eAxNHGLX; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751998AbdJCDRm (ORCPT + 26 others); Mon, 2 Oct 2017 23:17:42 -0400 Received: from mail-it0-f52.google.com ([209.85.214.52]:43193 "EHLO mail-it0-f52.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751816AbdJCDLu (ORCPT ); Mon, 2 Oct 2017 23:11:50 -0400 Received: by mail-it0-f52.google.com with SMTP id v62so7439322itd.0 for ; Mon, 02 Oct 2017 20:11:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=VWbCXob9IDcnI5U4QVHByko8cqaKfMTbw2HAwPif6Cc=; b=eAxNHGLX+rbGBX2bjdqqfZnGnW/RL4AWQlStu+MTEAsvBTMRCqxRsM8DgvejaqfcD7 2c5VOHnS+R6uD0JO4HaAgz7HyineG8FpxYu0ZyjtqTXmIE4xEmY3i6F3TwHvqwzFO9/T S2RaXe1Cd/9UWngxhptLYmoofwc7zjlRtwfes= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=VWbCXob9IDcnI5U4QVHByko8cqaKfMTbw2HAwPif6Cc=; b=npOKDtUrZYiDtIYJ0HrU+q5JaFtVqjHx48E1etPFioD/vHSFOf/MeUrkiGwXFTVg2s Ho7dSXuJ6jZPzAceCgEFDNAu5dz0rtolhQmquUOWxEQX9unh3hAEQ59jsAQBvyHcKLeX n8C5oI88iTgflXuESDFqZ20RY8m7atYacwpNeMc9zvvggxYQfyC/mc0DL+SIHkcQi7ff bW1t04ROz5KV8ECOeqy7uiYHgtCTlUYmHbHiT84T/MCLKgyT8U02ySMzfdt3Rzb7s31D 9mOuDDS4GOPgry8Hqwc2x4wTSwY08C+Pzuy1NjiIeHfNZtzXqXpRtjvlwrVjzhWNzOMZ Q6EQ== X-Gm-Message-State: AHPjjUiiQwNLub7xR2LDpBn2MfvxWnSugBoU5f01sLZtfjpLOzBP8bk4 8ARVNjXMs1XoLxLZ2Q/zqpHNqQ== X-Google-Smtp-Source: AOwi7QA0DafUQrROLZ23jeB2HxXxTUqePUptEvncsNe79ajg6Mr6ZTX2SfYMOToAJTcgORbAjY/UYA== X-Received: by 10.36.172.88 with SMTP id m24mr21713949iti.134.1507000309845; Mon, 02 Oct 2017 20:11:49 -0700 (PDT) Received: from node.jintackl-qv28633.kvmarm-pg0.wisc.cloudlab.us (c220g1-031126.wisc.cloudlab.us. [128.104.222.76]) by smtp.gmail.com with ESMTPSA id h84sm5367193iod.72.2017.10.02.20.11.48 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Mon, 02 Oct 2017 20:11:49 -0700 (PDT) From: Jintack Lim To: christoffer.dall@linaro.org, marc.zyngier@arm.com, kvmarm@lists.cs.columbia.edu Cc: jintack@cs.columbia.edu, pbonzini@redhat.com, rkrcmar@redhat.com, catalin.marinas@arm.com, will.deacon@arm.com, linux@armlinux.org.uk, mark.rutland@arm.com, linux-arm-kernel@lists.infradead.org, kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Jintack Lim Subject: [RFC PATCH v2 11/31] KVM: arm64: Implement nested Stage-2 page table walk logic Date: Mon, 2 Oct 2017 22:10:53 -0500 Message-Id: <1507000273-3735-9-git-send-email-jintack.lim@linaro.org> X-Mailer: git-send-email 1.9.1 In-Reply-To: <1507000273-3735-1-git-send-email-jintack.lim@linaro.org> References: <1507000273-3735-1-git-send-email-jintack.lim@linaro.org> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org From: Christoffer Dall Based on the pseudo-code in the ARM ARM, implement a stage 2 software page table walker. Signed-off-by: Christoffer Dall Signed-off-by: Jintack Lim --- Notes: v1-->v2: - Handled different endianness between the host and the guest hypervisor - Decoupled the stage-2 PTW from injecting exceptions. This will come in handy when we just want to walk the page table. - Added esr and upper_attr fields in kvm_s2_trans struct - Reworked pa_max() to have KVM_PHYS_SHIFT - Updated comment about the continuous bits arch/arm/include/asm/kvm_mmu.h | 16 +++ arch/arm64/include/asm/esr.h | 1 + arch/arm64/include/asm/kvm_arm.h | 3 + arch/arm64/include/asm/kvm_mmu.h | 12 ++ arch/arm64/kvm/mmu-nested.c | 241 +++++++++++++++++++++++++++++++++++++++ 5 files changed, 273 insertions(+) -- 1.9.1 diff --git a/arch/arm/include/asm/kvm_mmu.h b/arch/arm/include/asm/kvm_mmu.h index d3eafc5..5fab21a 100644 --- a/arch/arm/include/asm/kvm_mmu.h +++ b/arch/arm/include/asm/kvm_mmu.h @@ -221,6 +221,22 @@ static inline unsigned int kvm_get_vmid_bits(void) return 8; } +struct kvm_s2_trans { + phys_addr_t output; + phys_addr_t block_size; + bool writable; + bool readable; + int level; + u32 esr; + u64 upper_attr; +}; + +static inline int kvm_walk_nested_s2(struct kvm_vcpu *vcpu, phys_addr_t gipa, + struct kvm_s2_trans *result) +{ + return 0; +} + static inline void kvm_nested_s2_unmap(struct kvm_vcpu *vcpu) { } static inline void kvm_nested_s2_free(struct kvm *kvm) { } static inline void kvm_nested_s2_wp(struct kvm *kvm) { } diff --git a/arch/arm64/include/asm/esr.h b/arch/arm64/include/asm/esr.h index 210fde6..bc6610b 100644 --- a/arch/arm64/include/asm/esr.h +++ b/arch/arm64/include/asm/esr.h @@ -108,6 +108,7 @@ #define ESR_ELx_CM (UL(1) << 8) /* ISS field definitions for exceptions taken in to Hyp */ +#define ESR_ELx_FSC_ADDRSZ (0x00) #define ESR_ELx_CV (UL(1) << 24) #define ESR_ELx_COND_SHIFT (20) #define ESR_ELx_COND_MASK (UL(0xF) << ESR_ELx_COND_SHIFT) diff --git a/arch/arm64/include/asm/kvm_arm.h b/arch/arm64/include/asm/kvm_arm.h index a1274b7..3993703 100644 --- a/arch/arm64/include/asm/kvm_arm.h +++ b/arch/arm64/include/asm/kvm_arm.h @@ -104,6 +104,7 @@ #define VTCR_EL2_RES1 (1 << 31) #define VTCR_EL2_HD (1 << 22) #define VTCR_EL2_HA (1 << 21) +#define VTCR_EL2_PS_SHIFT TCR_EL2_PS_SHIFT #define VTCR_EL2_PS_MASK TCR_EL2_PS_MASK #define VTCR_EL2_TG0_MASK TCR_TG0_MASK #define VTCR_EL2_TG0_4K TCR_TG0_4K @@ -177,6 +178,8 @@ #define VTTBR_VMID_SHIFT (UL(48)) #define VTTBR_VMID_MASK(size) (_AT(u64, (1 << size) - 1) << VTTBR_VMID_SHIFT) +#define SCTLR_EE (UL(1) << 25) + /* Hyp System Trap Register */ #define HSTR_EL2_T(x) (1 << x) diff --git a/arch/arm64/include/asm/kvm_mmu.h b/arch/arm64/include/asm/kvm_mmu.h index 7fc7a83..c4efcd5 100644 --- a/arch/arm64/include/asm/kvm_mmu.h +++ b/arch/arm64/include/asm/kvm_mmu.h @@ -322,9 +322,21 @@ static inline unsigned int kvm_get_vmid_bits(void) return (cpuid_feature_extract_unsigned_field(reg, ID_AA64MMFR1_VMIDBITS_SHIFT) == 2) ? 16 : 8; } +struct kvm_s2_trans { + phys_addr_t output; + phys_addr_t block_size; + bool writable; + bool readable; + int level; + u32 esr; + u64 upper_attr; +}; + struct kvm_nested_s2_mmu *get_nested_mmu(struct kvm_vcpu *vcpu, u64 vttbr); struct kvm_s2_mmu *vcpu_get_active_s2_mmu(struct kvm_vcpu *vcpu); void update_nested_s2_mmu(struct kvm_vcpu *vcpu); +int kvm_walk_nested_s2(struct kvm_vcpu *vcpu, phys_addr_t gipa, + struct kvm_s2_trans *result); void kvm_nested_s2_unmap(struct kvm_vcpu *vcpu); void kvm_nested_s2_free(struct kvm *kvm); void kvm_nested_s2_wp(struct kvm *kvm); diff --git a/arch/arm64/kvm/mmu-nested.c b/arch/arm64/kvm/mmu-nested.c index 3ee20f2..fb694b7 100644 --- a/arch/arm64/kvm/mmu-nested.c +++ b/arch/arm64/kvm/mmu-nested.c @@ -22,6 +22,247 @@ #include #include +struct s2_walk_info { + unsigned int pgshift; + unsigned int pgsize; + unsigned int ps; + unsigned int sl; + unsigned int t0sz; +}; + +static unsigned int ps_to_output_size(unsigned int ps) +{ + switch (ps) { + case 0: return 32; + case 1: return 36; + case 2: return 40; + case 3: return 42; + case 4: return 44; + case 5: + default: + return 48; + } +} + +static unsigned int pa_max(void) +{ + /* We always emulate a VM with maximum PA size of KVM_PHYS_SIZE. */ + return KVM_PHYS_SHIFT; +} + +static int esr_s2_fault(struct kvm_vcpu *vcpu, int level, u32 fsc) +{ + u32 esr; + + esr = kvm_vcpu_get_hsr(vcpu) & ~ESR_ELx_FSC; + esr |= fsc; + esr |= level & 0x3; + return esr; +} + +static int check_base_s2_limits(struct kvm_vcpu *vcpu, struct s2_walk_info *wi, + int level, int input_size, int stride) +{ + int start_size; + + /* Check translation limits */ + switch (wi->pgsize) { + case SZ_64K: + if (level == 0 || (level == 1 && pa_max() <= 42)) + return -EFAULT; + break; + case SZ_16K: + if (level == 0 || (level == 1 && pa_max() <= 40)) + return -EFAULT; + break; + case SZ_4K: + if (level < 0 || (level == 0 && pa_max() <= 42)) + return -EFAULT; + break; + } + + /* Check input size limits */ + if (input_size > pa_max() && + (!vcpu_mode_is_32bit(vcpu) || input_size > 40)) + return -EFAULT; + + /* Check number of entries in starting level table */ + start_size = input_size - ((3 - level) * stride + wi->pgshift); + if (start_size < 1 || start_size > stride + 4) + return -EFAULT; + + return 0; +} + +/* Check if output is within boundaries */ +static int check_output_size(struct kvm_vcpu *vcpu, struct s2_walk_info *wi, + phys_addr_t output) +{ + unsigned int output_size = ps_to_output_size(wi->ps); + + if (output_size > pa_max()) + output_size = pa_max(); + + if (output_size != 48 && (output & GENMASK_ULL(47, output_size))) + return -1; + + return 0; +} + +/* + * This is essentially a C-version of the pseudo code from the ARM ARM + * AArch64.TranslationTableWalk function. I strongly recommend looking at + * that pseudocode in trying to understand this. + * + * Must be called with the kvm->srcy read lock held + */ +static int walk_nested_s2_pgd(struct kvm_vcpu *vcpu, phys_addr_t ipa, + struct s2_walk_info *wi, struct kvm_s2_trans *out) +{ + u64 vttbr = vcpu->arch.ctxt.sys_regs[VTTBR_EL2]; + int first_block_level, level, stride, input_size, base_lower_bound; + phys_addr_t base_addr; + unsigned int addr_top, addr_bottom; + u64 desc; /* page table entry */ + int ret; + phys_addr_t paddr; + + switch (wi->pgsize) { + case SZ_64K: + case SZ_16K: + level = 3 - wi->sl; + first_block_level = 2; + break; + case SZ_4K: + level = 2 - wi->sl; + first_block_level = 1; + break; + default: + /* GCC is braindead */ + WARN(1, "Page size is none of 4K, 16K or 64K"); + } + + stride = wi->pgshift - 3; + input_size = 64 - wi->t0sz; + if (input_size > 48 || input_size < 25) + return -EFAULT; + + ret = check_base_s2_limits(vcpu, wi, level, input_size, stride); + if (WARN_ON(ret)) + return ret; + + if (check_output_size(vcpu, wi, vttbr)) { + out->esr = esr_s2_fault(vcpu, level, ESR_ELx_FSC_ADDRSZ); + return 1; + } + + base_lower_bound = 3 + input_size - ((3 - level) * stride + + wi->pgshift); + base_addr = vttbr & GENMASK_ULL(47, base_lower_bound); + + addr_top = input_size - 1; + + while (1) { + phys_addr_t index; + + addr_bottom = (3 - level) * stride + wi->pgshift; + index = (ipa & GENMASK_ULL(addr_top, addr_bottom)) + >> (addr_bottom - 3); + + paddr = base_addr | index; + ret = kvm_read_guest(vcpu->kvm, paddr, &desc, sizeof(desc)); + if (ret < 0) + return ret; + + /* + * Handle reversedescriptors if endianness differs between the + * host and the guest hypervisor. + */ + if (vcpu_sys_reg(vcpu, SCTLR_EL2) & SCTLR_EE) + desc = be64_to_cpu(desc); + else + desc = le64_to_cpu(desc); + + /* Check for valid descriptor at this point */ + if (!(desc & 1) || ((desc & 3) == 1 && level == 3)) { + out->esr = esr_s2_fault(vcpu, level, ESR_ELx_FSC_FAULT); + return 1; + } + + /* We're at the final level or block translation level */ + if ((desc & 3) == 1 || level == 3) + break; + + if (check_output_size(vcpu, wi, desc)) { + out->esr = esr_s2_fault(vcpu, level, ESR_ELx_FSC_ADDRSZ); + return 1; + } + + base_addr = desc & GENMASK_ULL(47, wi->pgshift); + + level += 1; + addr_top = addr_bottom - 1; + } + + if (level < first_block_level) { + out->esr = esr_s2_fault(vcpu, level, ESR_ELx_FSC_FAULT); + return 1; + } + + /* + * We don't use the contiguous bit in the stage-2 ptes, so skip check + * for misprogramming of the contiguous bit. + */ + + if (check_output_size(vcpu, wi, desc)) { + out->esr = esr_s2_fault(vcpu, level, ESR_ELx_FSC_ADDRSZ); + return 1; + } + + if (!(desc & BIT(10))) { + out->esr = esr_s2_fault(vcpu, level, ESR_ELx_FSC_ACCESS); + return 1; + } + + /* Calculate and return the result */ + paddr = (desc & GENMASK_ULL(47, addr_bottom)) | + (ipa & GENMASK_ULL(addr_bottom - 1, 0)); + out->output = paddr; + out->block_size = 1UL << ((3 - level) * stride + wi->pgshift); + out->readable = desc & (0b01 << 6); + out->writable = desc & (0b10 << 6); + out->level = level; + out->upper_attr = desc & GENMASK_ULL(63, 52); + return 0; +} + +int kvm_walk_nested_s2(struct kvm_vcpu *vcpu, phys_addr_t gipa, + struct kvm_s2_trans *result) +{ + u64 vtcr = vcpu->arch.ctxt.sys_regs[VTCR_EL2]; + struct s2_walk_info wi; + + if (!nested_virt_in_use(vcpu)) + return 0; + + wi.t0sz = vtcr & TCR_EL2_T0SZ_MASK; + + switch (vtcr & VTCR_EL2_TG0_MASK) { + case VTCR_EL2_TG0_4K: + wi.pgshift = 12; break; + case VTCR_EL2_TG0_16K: + wi.pgshift = 14; break; + case VTCR_EL2_TG0_64K: + default: + wi.pgshift = 16; break; + } + wi.pgsize = 1UL << wi.pgshift; + wi.ps = (vtcr & VTCR_EL2_PS_MASK) >> VTCR_EL2_PS_SHIFT; + wi.sl = (vtcr & VTCR_EL2_SL0_MASK) >> VTCR_EL2_SL0_SHIFT; + + return walk_nested_s2_pgd(vcpu, gipa, &wi, result); +} + /* expects kvm->mmu_lock to be held */ void kvm_nested_s2_wp(struct kvm *kvm) { From patchwork Tue Oct 3 03:10:54 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jintack Lim X-Patchwork-Id: 114648 Delivered-To: patch@linaro.org Received: by 10.140.22.163 with SMTP id 32csp1388658qgn; Mon, 2 Oct 2017 20:17:56 -0700 (PDT) X-Received: by 10.99.61.77 with SMTP id k74mr8362548pga.105.1507000676187; Mon, 02 Oct 2017 20:17:56 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1507000676; cv=none; d=google.com; s=arc-20160816; b=EcgOKCzZL5dz2mZy8kuwpqQLWUsi4QvatjjoCQfSP3C54FylqxCytkfaFAGG4lKL2/ 4m6kRYouixxhMKVyAQPfcYdCuhbaSgWSso6RwpMITHMiVIoBPZ9k8BlBhDkVT8tRJKLv ftR27EFQArb1kOjSkg9b1g4OybJzkMKjaUc49Uesyn5AMh65gaE2MYEAGOjNB53hqyfF O89jrLz4J+2USZvHXPB4GdfcidyyudyMuKUqKcXLtRBWCZcAPTo8Mx65UXRWP+wvCCMF aT1AtF8E5hyBgGfn0aaq2KOKqfW9IcAo1dLW50ZSdqkclHjhFVw7XrBfybSXakOSMdPN rT6g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature:arc-authentication-results; bh=BtGwFuK3p9VeVHhpyZTNcwAe4jYasIJNJwevSLVezNY=; b=LwsvJ9wqZC3gIN2vlF/059fLCxKDTlc//Rpzgbh2JXOHVy7Qv4Nh980vfJMaJbtFYR SYYMUuyIIm9Ug+YXdr3THWf/83YrKqmrKT20HrB1wLeDt7m7CGE8FmNWxvSFF3iZq/Yv ui/szn4r5cjmIZYZskCEMTMQelvN/72UoN4WWQh5a5KSVFOYQUBvp/uZR2GptV1pKGkg pZKQLXuWT6E56SK22CKdm8ITv+1GLO9TsgQ6gQH34iOFJgddpnbqpqgWM5sISyxPVqCb vKYTU8ekOQaCen3bEcZ5gfeTImJU2Hl/aXtCfkErZj9c42lnnUcuORHiv9+3ehGsAg3m SoqA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=AtcOiyF7; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id a59si9473234plc.638.2017.10.02.20.17.55; Mon, 02 Oct 2017 20:17:56 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=AtcOiyF7; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751981AbdJCDRk (ORCPT + 26 others); Mon, 2 Oct 2017 23:17:40 -0400 Received: from mail-it0-f49.google.com ([209.85.214.49]:45076 "EHLO mail-it0-f49.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751477AbdJCDLw (ORCPT ); Mon, 2 Oct 2017 23:11:52 -0400 Received: by mail-it0-f49.google.com with SMTP id x15so9995729itb.0 for ; Mon, 02 Oct 2017 20:11:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=BtGwFuK3p9VeVHhpyZTNcwAe4jYasIJNJwevSLVezNY=; b=AtcOiyF7dlB7F5K06L96mgbPxq35K3FpdlERJs44tHQ1QjeikQpPOqjdwIwMrYS00X GGCXmRveeWWwGxWORneqNxX2PtYy0dNaXeuhGYq98VfCWBxbw49U+C6e5ZRFpKjIei0T XgWKKOitl30dqItHpjAhL3cyODiNkbL4pgU8k= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=BtGwFuK3p9VeVHhpyZTNcwAe4jYasIJNJwevSLVezNY=; b=Z2iTn89WZ7GzkI8ElRWvNQ8BcYWSXo7ELaXtYhfpjGtQ7ZYQcPNKUUWY8z01tku9m6 fK8p0T67E0+p42YmWdlUEobbtd1GDNMQk1bulIfHG1DEcDEJAjeDXpxsDedCM7Zs0ALH FQ+U7SSc+4Mlxd3+6SCdDAy5kPDf9WPXix0OM7DekIOzkmrdrVoGMvWn5mQdT05vZExY FFVxPgrk5YFarFeHZz6RmP5Gj/jkIgLNF3tV81qdnE0lhnoXBKs8vnAfjMzQRLiHLkZP IVPSjIugfBlnUx16Ln7V5DHg/tEsImbuO6XiMHfwFytrm0r7F3FcE6fUVPyhjFriKmE1 NamQ== X-Gm-Message-State: AHPjjUjxEeRV6LWBT1JAQLQF+bzPARaQXMaa7B1t72slpELKgNneIZVv KCY5n8f+8Yqw0J8D+awC0gME+g== X-Google-Smtp-Source: AOwi7QCyZbEpm3FGjG9igGd+MUzKGTYaJIt3ZCmLDGRswDgERji89dDLRs53MtrR9YSHHht49vkbyA== X-Received: by 10.36.13.82 with SMTP id 79mr23709277itx.129.1507000311339; Mon, 02 Oct 2017 20:11:51 -0700 (PDT) Received: from node.jintackl-qv28633.kvmarm-pg0.wisc.cloudlab.us (c220g1-031126.wisc.cloudlab.us. [128.104.222.76]) by smtp.gmail.com with ESMTPSA id h84sm5367193iod.72.2017.10.02.20.11.49 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Mon, 02 Oct 2017 20:11:50 -0700 (PDT) From: Jintack Lim To: christoffer.dall@linaro.org, marc.zyngier@arm.com, kvmarm@lists.cs.columbia.edu Cc: jintack@cs.columbia.edu, pbonzini@redhat.com, rkrcmar@redhat.com, catalin.marinas@arm.com, will.deacon@arm.com, linux@armlinux.org.uk, mark.rutland@arm.com, linux-arm-kernel@lists.infradead.org, kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Jintack Lim Subject: [RFC PATCH v2 12/31] KVM: arm/arm64: Handle shadow stage 2 page faults Date: Mon, 2 Oct 2017 22:10:54 -0500 Message-Id: <1507000273-3735-10-git-send-email-jintack.lim@linaro.org> X-Mailer: git-send-email 1.9.1 In-Reply-To: <1507000273-3735-1-git-send-email-jintack.lim@linaro.org> References: <1507000273-3735-1-git-send-email-jintack.lim@linaro.org> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org From: Christoffer Dall If we are faulting on a shadow stage 2 translation, we first walk the guest hypervisor's stage 2 page table to see if it has a mapping. If not, we inject a stage 2 page fault to the virtual EL2. Otherwise, we create a mapping in the shadow stage 2 page table. Note that we have to deal with two IPAs when we got a showdow stage 2 page fault. One is the address we faulted on, and is in the L2 guest phys space. The other is from the guest stage-2 page table walk, and is in the L1 guest phys space. To differentiate them, we rename variable names so that fault_ipa is used for the former and ipa is used for the latter. Signed-off-by: Christoffer Dall Signed-off-by: Jintack Lim --- Notes: v1-->v2: - Added a common function to inject s2 faults. - Align L1 IPA as well as L2 IPA in transparent_hugepage_adjust(). This will come in handy when creating a rmap entry with both IPAs. arch/arm/include/asm/kvm_emulate.h | 7 ++++ arch/arm/include/asm/kvm_mmu.h | 4 ++ arch/arm64/include/asm/kvm_emulate.h | 5 +++ arch/arm64/include/asm/kvm_mmu.h | 1 + arch/arm64/kvm/mmu-nested.c | 8 ++++ virt/kvm/arm/mmio.c | 12 +++--- virt/kvm/arm/mmu.c | 75 +++++++++++++++++++++++++++++------- 7 files changed, 92 insertions(+), 20 deletions(-) -- 1.9.1 diff --git a/arch/arm/include/asm/kvm_emulate.h b/arch/arm/include/asm/kvm_emulate.h index 24a3fbf..8136464 100644 --- a/arch/arm/include/asm/kvm_emulate.h +++ b/arch/arm/include/asm/kvm_emulate.h @@ -297,4 +297,11 @@ static inline struct kvm_s2_vmid *vcpu_get_active_vmid(struct kvm_vcpu *vcpu) { return &vcpu->kvm->arch.mmu.vmid; } + +/* arm architecture doesn't support the nesting */ +static inline bool kvm_is_shadow_s2_fault(struct kvm_vcpu *vcpu) +{ + return false; +} + #endif /* __ARM_KVM_EMULATE_H__ */ diff --git a/arch/arm/include/asm/kvm_mmu.h b/arch/arm/include/asm/kvm_mmu.h index 5fab21a..6a22846 100644 --- a/arch/arm/include/asm/kvm_mmu.h +++ b/arch/arm/include/asm/kvm_mmu.h @@ -242,6 +242,10 @@ static inline void kvm_nested_s2_free(struct kvm *kvm) { } static inline void kvm_nested_s2_wp(struct kvm *kvm) { } static inline void kvm_nested_s2_clear(struct kvm *kvm) { } static inline void kvm_nested_s2_flush(struct kvm *kvm) { } +static inline int kvm_inject_s2_fault(struct kvm_vcpu *vcpu, u64 esr_el2) +{ + return 0; +} static inline u64 kvm_get_vttbr(struct kvm_s2_vmid *vmid, struct kvm_s2_mmu *mmu) diff --git a/arch/arm64/include/asm/kvm_emulate.h b/arch/arm64/include/asm/kvm_emulate.h index f476576..c66554b 100644 --- a/arch/arm64/include/asm/kvm_emulate.h +++ b/arch/arm64/include/asm/kvm_emulate.h @@ -390,4 +390,9 @@ static inline unsigned long vcpu_data_host_to_guest(struct kvm_vcpu *vcpu, return data; /* Leave LE untouched */ } +static inline bool kvm_is_shadow_s2_fault(struct kvm_vcpu *vcpu) +{ + return vcpu_nested_stage2_enabled(vcpu) && !is_hyp_ctxt(vcpu); +} + #endif /* __ARM64_KVM_EMULATE_H__ */ diff --git a/arch/arm64/include/asm/kvm_mmu.h b/arch/arm64/include/asm/kvm_mmu.h index c4efcd5..425e4a2 100644 --- a/arch/arm64/include/asm/kvm_mmu.h +++ b/arch/arm64/include/asm/kvm_mmu.h @@ -342,6 +342,7 @@ int kvm_walk_nested_s2(struct kvm_vcpu *vcpu, phys_addr_t gipa, void kvm_nested_s2_wp(struct kvm *kvm); void kvm_nested_s2_clear(struct kvm *kvm); void kvm_nested_s2_flush(struct kvm *kvm); +int kvm_inject_s2_fault(struct kvm_vcpu *vcpu, u64 esr_el2); static inline u64 kvm_get_vttbr(struct kvm_s2_vmid *vmid, struct kvm_s2_mmu *mmu) diff --git a/arch/arm64/kvm/mmu-nested.c b/arch/arm64/kvm/mmu-nested.c index fb694b7..75570cc 100644 --- a/arch/arm64/kvm/mmu-nested.c +++ b/arch/arm64/kvm/mmu-nested.c @@ -60,6 +60,14 @@ static int esr_s2_fault(struct kvm_vcpu *vcpu, int level, u32 fsc) return esr; } +int kvm_inject_s2_fault(struct kvm_vcpu *vcpu, u64 esr_el2) +{ + vcpu->arch.ctxt.sys_regs[FAR_EL2] = vcpu->arch.fault.far_el2; + vcpu->arch.ctxt.sys_regs[HPFAR_EL2] = vcpu->arch.fault.hpfar_el2; + + return kvm_inject_nested_sync(vcpu, esr_el2); +} + static int check_base_s2_limits(struct kvm_vcpu *vcpu, struct s2_walk_info *wi, int level, int input_size, int stride) { diff --git a/virt/kvm/arm/mmio.c b/virt/kvm/arm/mmio.c index b6e715f..a1009c2 100644 --- a/virt/kvm/arm/mmio.c +++ b/virt/kvm/arm/mmio.c @@ -153,7 +153,7 @@ static int decode_hsr(struct kvm_vcpu *vcpu, bool *is_write, int *len) } int io_mem_abort(struct kvm_vcpu *vcpu, struct kvm_run *run, - phys_addr_t fault_ipa) + phys_addr_t ipa) { unsigned long data; unsigned long rt; @@ -182,22 +182,22 @@ int io_mem_abort(struct kvm_vcpu *vcpu, struct kvm_run *run, data = vcpu_data_guest_to_host(vcpu, vcpu_get_reg(vcpu, rt), len); - trace_kvm_mmio(KVM_TRACE_MMIO_WRITE, len, fault_ipa, data); + trace_kvm_mmio(KVM_TRACE_MMIO_WRITE, len, ipa, data); kvm_mmio_write_buf(data_buf, len, data); - ret = kvm_io_bus_write(vcpu, KVM_MMIO_BUS, fault_ipa, len, + ret = kvm_io_bus_write(vcpu, KVM_MMIO_BUS, ipa, len, data_buf); } else { trace_kvm_mmio(KVM_TRACE_MMIO_READ_UNSATISFIED, len, - fault_ipa, 0); + ipa, 0); - ret = kvm_io_bus_read(vcpu, KVM_MMIO_BUS, fault_ipa, len, + ret = kvm_io_bus_read(vcpu, KVM_MMIO_BUS, ipa, len, data_buf); } /* Now prepare kvm_run for the potential return to userland. */ run->mmio.is_write = is_write; - run->mmio.phys_addr = fault_ipa; + run->mmio.phys_addr = ipa; run->mmio.len = len; if (!ret) { diff --git a/virt/kvm/arm/mmu.c b/virt/kvm/arm/mmu.c index 3143f81..25d3d73 100644 --- a/virt/kvm/arm/mmu.c +++ b/virt/kvm/arm/mmu.c @@ -1098,7 +1098,8 @@ int kvm_phys_addr_ioremap(struct kvm *kvm, phys_addr_t guest_ipa, return ret; } -static bool transparent_hugepage_adjust(kvm_pfn_t *pfnp, phys_addr_t *ipap) +static bool transparent_hugepage_adjust(kvm_pfn_t *pfnp, phys_addr_t *ipap, + phys_addr_t *fault_ipap) { kvm_pfn_t pfn = *pfnp; gfn_t gfn = *ipap >> PAGE_SHIFT; @@ -1126,6 +1127,7 @@ static bool transparent_hugepage_adjust(kvm_pfn_t *pfnp, phys_addr_t *ipap) mask = PTRS_PER_PMD - 1; VM_BUG_ON((gfn & mask) != (pfn & mask)); if (pfn & mask) { + *fault_ipap &= PMD_MASK; *ipap &= PMD_MASK; kvm_release_pfn_clean(pfn); pfn &= ~mask; @@ -1337,13 +1339,15 @@ static void kvm_send_hwpoison_signal(unsigned long address, } static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa, - struct kvm_memory_slot *memslot, unsigned long hva, - unsigned long fault_status) + struct kvm_s2_trans *nested, + struct kvm_memory_slot *memslot, + unsigned long hva, unsigned long fault_status) { int ret; bool write_fault, writable, hugetlb = false, force_pte = false; unsigned long mmu_seq; - gfn_t gfn = fault_ipa >> PAGE_SHIFT; + phys_addr_t ipa = fault_ipa; + gfn_t gfn; struct kvm *kvm = vcpu->kvm; struct kvm_mmu_memory_cache *memcache = &vcpu->arch.mmu_page_cache; struct vm_area_struct *vma; @@ -1368,9 +1372,23 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa, return -EFAULT; } - if (is_vm_hugetlb_page(vma) && !logging_active) { + if (kvm_is_shadow_s2_fault(vcpu)) { + ipa = nested->output; + + /* + * If we're about to create a shadow stage 2 entry, then we + * can only create huge mappings if the guest hypervisor also + * uses a huge mapping. + */ + if (nested->block_size != PMD_SIZE) + force_pte = true; + } + gfn = ipa >> PAGE_SHIFT; + + + if (!force_pte && is_vm_hugetlb_page(vma) && !logging_active) { hugetlb = true; - gfn = (fault_ipa & PMD_MASK) >> PAGE_SHIFT; + gfn = (ipa & PMD_MASK) >> PAGE_SHIFT; } else { /* * Pages belonging to memslots that don't have the same @@ -1438,7 +1456,7 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa, goto out_unlock; if (!hugetlb && !force_pte) - hugetlb = transparent_hugepage_adjust(&pfn, &fault_ipa); + hugetlb = transparent_hugepage_adjust(&pfn, &ipa, &fault_ipa); if (hugetlb) { pmd_t new_pmd = pfn_pmd(pfn, mem_type); @@ -1525,8 +1543,10 @@ static void handle_access_fault(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa) int kvm_handle_guest_abort(struct kvm_vcpu *vcpu, struct kvm_run *run) { unsigned long fault_status; - phys_addr_t fault_ipa; + phys_addr_t fault_ipa; /* The address we faulted on */ + phys_addr_t ipa; /* Always the IPA in the L1 guest phys space */ struct kvm_memory_slot *memslot; + struct kvm_s2_trans nested_trans; unsigned long hva; bool is_iabt, write_fault, writable; gfn_t gfn; @@ -1538,7 +1558,7 @@ int kvm_handle_guest_abort(struct kvm_vcpu *vcpu, struct kvm_run *run) return 1; } - fault_ipa = kvm_vcpu_get_fault_ipa(vcpu); + ipa = fault_ipa = kvm_vcpu_get_fault_ipa(vcpu); trace_kvm_guest_fault(*vcpu_pc(vcpu), kvm_vcpu_get_hsr(vcpu), kvm_vcpu_get_hfar(vcpu), fault_ipa); @@ -1547,6 +1567,12 @@ int kvm_handle_guest_abort(struct kvm_vcpu *vcpu, struct kvm_run *run) fault_status = kvm_vcpu_trap_get_fault_type(vcpu); if (fault_status != FSC_FAULT && fault_status != FSC_PERM && fault_status != FSC_ACCESS) { + /* + * We must never see an address size fault on shadow stage 2 + * page table walk, because we would have injected an addr + * size fault when we walked the nested s2 page and not + * create the shadow entry. + */ kvm_err("Unsupported FSC: EC=%#x xFSC=%#lx ESR_EL2=%#lx\n", kvm_vcpu_trap_get_class(vcpu), (unsigned long)kvm_vcpu_trap_get_fault(vcpu), @@ -1556,7 +1582,27 @@ int kvm_handle_guest_abort(struct kvm_vcpu *vcpu, struct kvm_run *run) idx = srcu_read_lock(&vcpu->kvm->srcu); - gfn = fault_ipa >> PAGE_SHIFT; + /* + * We may have faulted on a shadow stage 2 page table if we are + * running a nested guest. In this case, we have to resovle the L2 + * IPA to the L1 IPA first, before knowing what kind of memory should + * back the L1 IPA. + * + * If the shadow stage 2 page table walk faults, then we simply inject + * this to the guest and carry on. + */ + if (kvm_is_shadow_s2_fault(vcpu)) { + nested_trans.esr = 0; + ret = kvm_walk_nested_s2(vcpu, fault_ipa, &nested_trans); + if (nested_trans.esr) + kvm_inject_s2_fault(vcpu, nested_trans.esr); + if (ret) + goto out_unlock; + + ipa = nested_trans.output; + } + + gfn = ipa >> PAGE_SHIFT; memslot = gfn_to_memslot(vcpu->kvm, gfn); hva = gfn_to_hva_memslot_prot(memslot, gfn, &writable); write_fault = kvm_is_write_fault(vcpu); @@ -1590,13 +1636,13 @@ int kvm_handle_guest_abort(struct kvm_vcpu *vcpu, struct kvm_run *run) * faulting VA. This is always 12 bits, irrespective * of the page size. */ - fault_ipa |= kvm_vcpu_get_hfar(vcpu) & ((1 << 12) - 1); - ret = io_mem_abort(vcpu, run, fault_ipa); + ipa |= kvm_vcpu_get_hfar(vcpu) & ((1 << 12) - 1); + ret = io_mem_abort(vcpu, run, ipa); goto out_unlock; } /* Userspace should not be able to register out-of-bounds IPAs */ - VM_BUG_ON(fault_ipa >= KVM_PHYS_SIZE); + VM_BUG_ON(ipa >= KVM_PHYS_SIZE); if (fault_status == FSC_ACCESS) { handle_access_fault(vcpu, fault_ipa); @@ -1604,7 +1650,8 @@ int kvm_handle_guest_abort(struct kvm_vcpu *vcpu, struct kvm_run *run) goto out_unlock; } - ret = user_mem_abort(vcpu, fault_ipa, memslot, hva, fault_status); + ret = user_mem_abort(vcpu, fault_ipa, &nested_trans, + memslot, hva, fault_status); if (ret == 0) ret = 1; out_unlock: From patchwork Tue Oct 3 03:10:55 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jintack Lim X-Patchwork-Id: 114649 Delivered-To: patch@linaro.org Received: by 10.140.22.163 with SMTP id 32csp1388810qgn; Mon, 2 Oct 2017 20:18:12 -0700 (PDT) X-Received: by 10.84.199.194 with SMTP id d2mr16222328plh.346.1507000691940; Mon, 02 Oct 2017 20:18:11 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1507000691; cv=none; d=google.com; s=arc-20160816; b=IwA9atb/nXPzroMuDUEyzmKU7U/gT4wC9O+lrxkWRwE3mAioQIetLFCfmRe6r4dZQM Y4Cg2sdMXSeizfZn1KJpGBdV06lHVvpMEQuaL8YuwgWtGkOiS9GADFieKE+WfZXy+waU couwmac3M4tKGFSrF+46idxYBZpBRs+KIQF4BHOMx3ogGZdGsk1XzNxMP2lajrs1v2mw xy5n7wKFn3rY3HrnGvqQzzhKexD/C8JjTfJw8Ded0dq3QVgEHyQ1Bwp3sYU38y4ahkB0 VD0lrbMD5Ng1qv3zFJhPxoqcfWyXnBS/X6shoxTPfRJaDUGKbfBla75oE2rjrz+Ghyfd CeAA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature:arc-authentication-results; bh=xS0UTzPa5rAUSRFYeXyRrNnLF/7hbYHNe7PYTIOBDV4=; b=t/Ium4wP1+UYvrO/Ofy1qzzxom8ZomGkBikM+RjClppYyttkm/L5Ml3NZmrBEG7eTc 3+rDL3XMagL1FDQMxIIABiQ4QTYq4K8vPQERN+iPPd2FfWCBIuGcIw1xqcy5mjw5WS3R NwuCti2qmtj/PhdArdXbK283V+7IKUGutvmtp2LFHjXuuBH33wSK2TJv2/drbzR/q6jU mJCVF2hMXc7Xw03RTBvawZsSMQXEEiumyJvFxTG5pVV5Kr8vqzOIpJ2TeSDdNovEtY4r OM8Nqk7q/d5NA44KC0BywNf7R78wVe/qmfqI9VSX24MJKI4Qr0zVUS83YZ3b0XK9Ud3i v8zw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=JbaGGV7m; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id z61si6938684plb.575.2017.10.02.20.18.11; Mon, 02 Oct 2017 20:18:11 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=JbaGGV7m; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751921AbdJCDRi (ORCPT + 26 others); Mon, 2 Oct 2017 23:17:38 -0400 Received: from mail-it0-f54.google.com ([209.85.214.54]:50813 "EHLO mail-it0-f54.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751911AbdJCDLx (ORCPT ); Mon, 2 Oct 2017 23:11:53 -0400 Received: by mail-it0-f54.google.com with SMTP id y138so10005533itc.5 for ; Mon, 02 Oct 2017 20:11:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=xS0UTzPa5rAUSRFYeXyRrNnLF/7hbYHNe7PYTIOBDV4=; b=JbaGGV7mk/KLEw0fSyPR20HWiBbc5K8iEb+wT5hFDx70vhiYRxiZsFuzMmZFUBm2CQ HyTSF4rI9sD9iJqA8ZFxFHUbf1Zp74u2BjHUNCuB6UKZYgPCwdEElDF5zgNiwxKQiUl/ 8oWnzPgJHODO0rxWBvkM1C5Utny7fipW0cfA8= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=xS0UTzPa5rAUSRFYeXyRrNnLF/7hbYHNe7PYTIOBDV4=; b=CoQB4b8Vh4tZRYqk4lhWsd2oAHxm5H7IuWTu/SpQ8zhd+d+bQyHM1do7TxVHTdtK1c dY5ToS66rfRLQdzE+SsCqA+4JmPsfXUj6sm8XpYG5bdE9BgijvTJJRz4ZcRN8NrUq605 /o1dFACl2vd0//j3EiRysz2rRhHxRr+ES67B/eJHpAhgRFF5r6NJsK2QZv/KqTYGordE Kbycv/yn6yDAypNOL7/3L7iZM7OojAo3oz2/ITYhh2nU3fwK1QEIrzrD+RYZghvwnOfh JPTRuLkEX309V80r9Sw/sDMMU0xEqJTOylSK4xNR8IxEPKmuRLC+VdOZ9UK2/ZTZ/F9R lMug== X-Gm-Message-State: AHPjjUhfURJyxjcY63oBxGut8aBj3dDgpaMTEQEu8Gt5QVZeMWUL9Vfu Far3TWpDGswZE0Eky8mvGbstdw== X-Google-Smtp-Source: AOwi7QDgcbJeOhys7JB5rg1UlzB/NOMLyzsIjVEvQuhQfoo5ZhtKMiszmHcjHULca9wKinueaROdmw== X-Received: by 10.36.61.21 with SMTP id n21mr22637144itn.121.1507000312785; Mon, 02 Oct 2017 20:11:52 -0700 (PDT) Received: from node.jintackl-qv28633.kvmarm-pg0.wisc.cloudlab.us (c220g1-031126.wisc.cloudlab.us. [128.104.222.76]) by smtp.gmail.com with ESMTPSA id h84sm5367193iod.72.2017.10.02.20.11.51 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Mon, 02 Oct 2017 20:11:52 -0700 (PDT) From: Jintack Lim To: christoffer.dall@linaro.org, marc.zyngier@arm.com, kvmarm@lists.cs.columbia.edu Cc: jintack@cs.columbia.edu, pbonzini@redhat.com, rkrcmar@redhat.com, catalin.marinas@arm.com, will.deacon@arm.com, linux@armlinux.org.uk, mark.rutland@arm.com, linux-arm-kernel@lists.infradead.org, kvm@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [RFC PATCH v2 13/31] KVM: arm/arm64: Move kvm_is_write_fault to header file Date: Mon, 2 Oct 2017 22:10:55 -0500 Message-Id: <1507000273-3735-11-git-send-email-jintack.lim@linaro.org> X-Mailer: git-send-email 1.9.1 In-Reply-To: <1507000273-3735-1-git-send-email-jintack.lim@linaro.org> References: <1507000273-3735-1-git-send-email-jintack.lim@linaro.org> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org From: Christoffer Dall Move this little function to the header files for arm/arm64 so other code can make use of it directly. Signed-off-by: Christoffer Dall --- arch/arm/include/asm/kvm_emulate.h | 8 ++++++++ arch/arm64/include/asm/kvm_emulate.h | 8 ++++++++ virt/kvm/arm/mmu.c | 8 -------- 3 files changed, 16 insertions(+), 8 deletions(-) -- 1.9.1 diff --git a/arch/arm/include/asm/kvm_emulate.h b/arch/arm/include/asm/kvm_emulate.h index 8136464..9b745d9 100644 --- a/arch/arm/include/asm/kvm_emulate.h +++ b/arch/arm/include/asm/kvm_emulate.h @@ -223,6 +223,14 @@ static inline u8 kvm_vcpu_trap_get_fault_type(struct kvm_vcpu *vcpu) return kvm_vcpu_get_hsr(vcpu) & HSR_FSC_TYPE; } +static inline bool kvm_is_write_fault(struct kvm_vcpu *vcpu) +{ + if (kvm_vcpu_trap_is_iabt(vcpu)) + return false; + + return kvm_vcpu_dabt_iswrite(vcpu); +} + static inline u32 kvm_vcpu_hvc_get_imm(struct kvm_vcpu *vcpu) { return kvm_vcpu_get_hsr(vcpu) & HSR_HVC_IMM_MASK; diff --git a/arch/arm64/include/asm/kvm_emulate.h b/arch/arm64/include/asm/kvm_emulate.h index c66554b..4c47bc7 100644 --- a/arch/arm64/include/asm/kvm_emulate.h +++ b/arch/arm64/include/asm/kvm_emulate.h @@ -307,6 +307,14 @@ static inline int kvm_vcpu_sys_get_rt(struct kvm_vcpu *vcpu) return (esr & ESR_ELx_SYS64_ISS_RT_MASK) >> ESR_ELx_SYS64_ISS_RT_SHIFT; } +static inline bool kvm_is_write_fault(struct kvm_vcpu *vcpu) +{ + if (kvm_vcpu_trap_is_iabt(vcpu)) + return false; + + return kvm_vcpu_dabt_iswrite(vcpu); +} + static inline unsigned long kvm_vcpu_get_mpidr_aff(struct kvm_vcpu *vcpu) { return vcpu_sys_reg(vcpu, MPIDR_EL1) & MPIDR_HWID_BITMASK; diff --git a/virt/kvm/arm/mmu.c b/virt/kvm/arm/mmu.c index 25d3d73..74941ad 100644 --- a/virt/kvm/arm/mmu.c +++ b/virt/kvm/arm/mmu.c @@ -1141,14 +1141,6 @@ static bool transparent_hugepage_adjust(kvm_pfn_t *pfnp, phys_addr_t *ipap, return false; } -static bool kvm_is_write_fault(struct kvm_vcpu *vcpu) -{ - if (kvm_vcpu_trap_is_iabt(vcpu)) - return false; - - return kvm_vcpu_dabt_iswrite(vcpu); -} - /** * stage2_wp_ptes - write protect PMD range * @pmd: pointer to pmd entry From patchwork Tue Oct 3 03:10:56 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jintack Lim X-Patchwork-Id: 114646 Delivered-To: patch@linaro.org Received: by 10.140.22.163 with SMTP id 32csp1388289qgn; Mon, 2 Oct 2017 20:17:22 -0700 (PDT) X-Received: by 10.98.133.17 with SMTP id u17mr16182960pfd.235.1507000642333; Mon, 02 Oct 2017 20:17:22 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1507000642; cv=none; d=google.com; s=arc-20160816; b=NOsZYMJTYcRvb4KANSYyA653I1N/Gpnq2KIwPpb4/l7vdvHkttxWT9+TQHVvmaqm/6 TLm2MM1j4OaHQYixdMu0x9kwP12UzRuWkgdAp0QA6V3ZEhxqkBtrcomq+vdQoshQqePp 7oTxKS8WMF6xEUS4XkTZaZG/S19wAyMzkL1o72fNw9xX9oz128lyU1E0kxy0A4iwoyNE 7q+lYYsbNEXdIandRXbZoQV2bcNwRWpHTA7TN1UtuO+C24/+55GAnxg4igVW0mSBh5ed +95Zwvgy4Edc6jl72DE5QLQApVYR1pfMtpucehB3q9dKgcikvdfPbi+Q0xRHslsz6OmW /3QQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature:arc-authentication-results; bh=8pHfRnT3Ca2lCPhMaDfK5riO0YKJi4Xl6ZAuana/B4Y=; b=gjPtflC6a/jOxi+fMUfccRPZl2WkCOSEFJ/WJY9sVk2JIo/Fggl0eVJPRekYvI+pOw BOyOA26ZdOahP9a8g8RsN3QvhPsceY+0FFjEh+HAmpHv74N6wVwkPK9AkIs5ErcTohgN fQwvZOpBL9P11TvNBhzQ2QM0z8tBq91Xu+KNyDrhUVdAkJeEz9sd/Mqko8y1p14nyI5E 8g9PnGp0HP5Ae26Xzw3jYZ3uvGPz+/r20LheJRnBF0SvgC4IGwGiRRmPc4qc4CqJ8gSy Wkv20NSqhx2v/czcatZ4HY8Av80MkQkNO5nroDW0jOPgh4KXITWluO8164KQR+HulciI NEYg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=B2aCHyWw; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id f8si5484908plk.475.2017.10.02.20.17.22; Mon, 02 Oct 2017 20:17:22 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=B2aCHyWw; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751741AbdJCDRT (ORCPT + 26 others); Mon, 2 Oct 2017 23:17:19 -0400 Received: from mail-it0-f47.google.com ([209.85.214.47]:51570 "EHLO mail-it0-f47.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751895AbdJCDLy (ORCPT ); Mon, 2 Oct 2017 23:11:54 -0400 Received: by mail-it0-f47.google.com with SMTP id w1so283290itd.0 for ; Mon, 02 Oct 2017 20:11:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=8pHfRnT3Ca2lCPhMaDfK5riO0YKJi4Xl6ZAuana/B4Y=; b=B2aCHyWw5gmSrK6M/Kha/Ukrre6yUlffBR+hBZGE61IW07VdnZmMRr9Rv4zWlxrOeh g30txjIrSO50QZxy+eljEV4imxwvO0IxCmGGM5gpk/iBZ7Cb3S7naHWnWOYLJZNXqi3X /qwk7KbLPsefBgPbYjqIkHMq33RnJiIJOP5+k= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=8pHfRnT3Ca2lCPhMaDfK5riO0YKJi4Xl6ZAuana/B4Y=; b=nk/D5Fkj/oK3IXPLFb6FKrkmqmjJTD1cbOfy706uMUARRYQbYoAemjTjlRvunoknq1 8YP0YNxBLys/ZOMB73qmDFmB4h5xMbJEorHZyBqU3AVWg7XjYwZcTWaTf84fIi04/Q+f Ssju+g7Trr9XFcvR7/PK/7F6yNnBE0LdJMD+RT0D4JQ0PXI9+ku6Cg9dINcEp3D4CutA VtJBTvWLv85FV8axR8wHqwWrHamgvB+on6A9XyLN8pkFzL3u4SfCdE6heIaQ+qED9YwA lJTtkjrodDU0FEbpWuEIMXOKmJXo0b51YQnka8RfKmr3D/hB2kIBT1X7cJeNMB6h8O0t 2YDg== X-Gm-Message-State: AMCzsaWtnjBI3IUr7dOIlkYQCmHzGrt7IFsOxJkQCCPi0CpbwGnaskHd f+UvQWS3LgJOZ58N1CCv5uAGhw== X-Google-Smtp-Source: AOwi7QCOD66O7K6UPWs+EeraEyE50orMOq/y7o9+ZMMi/1lx/gcTjeGIImUXclR2+0JKPAG+BpkrVg== X-Received: by 10.36.84.82 with SMTP id t79mr22627575ita.98.1507000314041; Mon, 02 Oct 2017 20:11:54 -0700 (PDT) Received: from node.jintackl-qv28633.kvmarm-pg0.wisc.cloudlab.us (c220g1-031126.wisc.cloudlab.us. [128.104.222.76]) by smtp.gmail.com with ESMTPSA id h84sm5367193iod.72.2017.10.02.20.11.52 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Mon, 02 Oct 2017 20:11:53 -0700 (PDT) From: Jintack Lim To: christoffer.dall@linaro.org, marc.zyngier@arm.com, kvmarm@lists.cs.columbia.edu Cc: jintack@cs.columbia.edu, pbonzini@redhat.com, rkrcmar@redhat.com, catalin.marinas@arm.com, will.deacon@arm.com, linux@armlinux.org.uk, mark.rutland@arm.com, linux-arm-kernel@lists.infradead.org, kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Jintack Lim Subject: [RFC PATCH v2 14/31] KVM: arm/arm64: Forward the guest hypervisor's stage 2 permission faults Date: Mon, 2 Oct 2017 22:10:56 -0500 Message-Id: <1507000273-3735-12-git-send-email-jintack.lim@linaro.org> X-Mailer: git-send-email 1.9.1 In-Reply-To: <1507000273-3735-1-git-send-email-jintack.lim@linaro.org> References: <1507000273-3735-1-git-send-email-jintack.lim@linaro.org> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org From: Christoffer Dall We can have discrepancies between the nested stage 2 page table and the shadow one in a couple of cases. For example, the guest hypervisor can mark a page writable but the host hypervisor maps the page read-only in the shadow page table, if using something like KSM on the host level. In this case, a write fault is handled directly by the host hypervisor. But we could also simply have a read-only page mapped read-only in both tables, in which case the host hypervisor cannot do anything else than telling the guest hypervisor about the fault. Signed-off-by: Christoffer Dall Signed-off-by: Jintack Lim --- arch/arm/include/asm/kvm_mmu.h | 7 +++++++ arch/arm64/include/asm/kvm_mmu.h | 2 ++ arch/arm64/kvm/mmu-nested.c | 22 ++++++++++++++++++++++ virt/kvm/arm/mmu.c | 7 +++++++ 4 files changed, 38 insertions(+) -- 1.9.1 diff --git a/arch/arm/include/asm/kvm_mmu.h b/arch/arm/include/asm/kvm_mmu.h index 6a22846..1c5b652 100644 --- a/arch/arm/include/asm/kvm_mmu.h +++ b/arch/arm/include/asm/kvm_mmu.h @@ -237,6 +237,13 @@ static inline int kvm_walk_nested_s2(struct kvm_vcpu *vcpu, phys_addr_t gipa, return 0; } +static inline int kvm_s2_handle_perm_fault(struct kvm_vcpu *vcpu, + phys_addr_t fault_ipa, + struct kvm_s2_trans *trans) +{ + return 0; +} + static inline void kvm_nested_s2_unmap(struct kvm_vcpu *vcpu) { } static inline void kvm_nested_s2_free(struct kvm *kvm) { } static inline void kvm_nested_s2_wp(struct kvm *kvm) { } diff --git a/arch/arm64/include/asm/kvm_mmu.h b/arch/arm64/include/asm/kvm_mmu.h index 425e4a2..239bb89 100644 --- a/arch/arm64/include/asm/kvm_mmu.h +++ b/arch/arm64/include/asm/kvm_mmu.h @@ -337,6 +337,8 @@ struct kvm_s2_trans { void update_nested_s2_mmu(struct kvm_vcpu *vcpu); int kvm_walk_nested_s2(struct kvm_vcpu *vcpu, phys_addr_t gipa, struct kvm_s2_trans *result); +int kvm_s2_handle_perm_fault(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa, + struct kvm_s2_trans *trans); void kvm_nested_s2_unmap(struct kvm_vcpu *vcpu); void kvm_nested_s2_free(struct kvm *kvm); void kvm_nested_s2_wp(struct kvm *kvm); diff --git a/arch/arm64/kvm/mmu-nested.c b/arch/arm64/kvm/mmu-nested.c index 75570cc..a440d7b 100644 --- a/arch/arm64/kvm/mmu-nested.c +++ b/arch/arm64/kvm/mmu-nested.c @@ -271,6 +271,28 @@ int kvm_walk_nested_s2(struct kvm_vcpu *vcpu, phys_addr_t gipa, return walk_nested_s2_pgd(vcpu, gipa, &wi, result); } +/* + * Returns non-zero if permission fault is handled by injecting it to the next + * level hypervisor. + */ +int kvm_s2_handle_perm_fault(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa, + struct kvm_s2_trans *trans) +{ + unsigned long fault_status = kvm_vcpu_trap_get_fault_type(vcpu); + bool write_fault = kvm_is_write_fault(vcpu); + + if (fault_status != FSC_PERM) + return 0; + + if ((write_fault && !trans->writable) || + (!write_fault && !trans->readable)) { + trans->esr = esr_s2_fault(vcpu, trans->level, ESR_ELx_FSC_PERM); + return 1; + } + + return 0; +} + /* expects kvm->mmu_lock to be held */ void kvm_nested_s2_wp(struct kvm *kvm) { diff --git a/virt/kvm/arm/mmu.c b/virt/kvm/arm/mmu.c index 74941ad..4fb7b3b 100644 --- a/virt/kvm/arm/mmu.c +++ b/virt/kvm/arm/mmu.c @@ -1591,6 +1591,13 @@ int kvm_handle_guest_abort(struct kvm_vcpu *vcpu, struct kvm_run *run) if (ret) goto out_unlock; + nested_trans.esr = 0; + ret = kvm_s2_handle_perm_fault(vcpu, fault_ipa, &nested_trans); + if (nested_trans.esr) + kvm_inject_s2_fault(vcpu, nested_trans.esr); + if (ret) + goto out_unlock; + ipa = nested_trans.output; } From patchwork Tue Oct 3 03:11:13 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jintack Lim X-Patchwork-Id: 114644 Delivered-To: patch@linaro.org Received: by 10.140.22.163 with SMTP id 32csp1384964qgn; Mon, 2 Oct 2017 20:12:34 -0700 (PDT) X-Received: by 10.99.42.141 with SMTP id q135mr14105169pgq.102.1507000354829; Mon, 02 Oct 2017 20:12:34 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1507000354; cv=none; d=google.com; s=arc-20160816; b=Ma+OiBpuoFy/PyJEQ4N73RzIdh49FgVoD5okNaZdXYm5hd/dpWI5ouJ2nUO3LnmeEj bH3vM6r8OAgWPl9z4EKMEJzHntIqMyNmmaM4freVQBlZUwbu4CqnoGhhA+NmvkXX3Hkc ZCOh711Mn77s3O40oIAOjrHpsxLlhq7fa5SSY3Nbe2efeCxiBttr2KnCTrDdQ4G9yQiA cr7uEQLya+MWNBj0h4cpuyRyGqXDpM8llC8iET2ANr+E8zB+gD6bpm8xmafOP2cR1I9y 5PL9Bw/d+BBIhwSZ0+/0RgV9knP+WC36gjpgx1ibc7lIvMEMcOUS0fW0ZIXFpg+ApEoW QJDw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature:arc-authentication-results; bh=UKqqZH0PQQRDqhFJrUnEN3msoL2mkBMjdYBdeAHkGYU=; b=jZD2vkeknm1WaDVI2etNxpxBIKkJ9Ld5Ikzvtn/gc4yz9CBCCtF5Ca0WZxFKuuKaHO 7WDno//h+xVAsEpHHwN2xZ+54QiMBhUxGfSUcDk0eTWcWGBCf4v7P5OOVcc64Wb6TKni HbgGyJsXNxMDbYo+UwuZ7PcVUMASBI7pJ3FeYIob3QSX9XJZi8SKocQ56BrGAoccrdpx HvdEJp2MuV6btuYwabQsPRSdcCzotS+sqfK0GBlFFSluvamfWiershdF8Y/T+Kb+NgPQ 6E3l4Rl9Tmi+GlLcS+qdNlCDQKVDuAbxIDUvS7WElF2FSRqd1/A1/aC7lvcGAr+YhMEV 95sQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=UzrcLgg6; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id k10si8810618pgc.320.2017.10.02.20.12.34; Mon, 02 Oct 2017 20:12:34 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=UzrcLgg6; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752085AbdJCDMb (ORCPT + 26 others); Mon, 2 Oct 2017 23:12:31 -0400 Received: from mail-io0-f174.google.com ([209.85.223.174]:47017 "EHLO mail-io0-f174.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752033AbdJCDMO (ORCPT ); Mon, 2 Oct 2017 23:12:14 -0400 Received: by mail-io0-f174.google.com with SMTP id d16so6451371ioj.3 for ; Mon, 02 Oct 2017 20:12:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=UKqqZH0PQQRDqhFJrUnEN3msoL2mkBMjdYBdeAHkGYU=; b=UzrcLgg6kMknyUlhHctCWi6o9FqV5MhvBMavDfBBp7IvDlFr/U7grT0MezPcBdeRzq HJHSvOKlnzXzavxI9M67ChuQLoIzSJjPAFdeO3F+UvBpVm5RHZtxp0GBK1wPWeB24rFB DuAwvIVO17kC/HktuxgiqbP2zFPU2T/JPrPac= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=UKqqZH0PQQRDqhFJrUnEN3msoL2mkBMjdYBdeAHkGYU=; b=nTLLu41PK5g7tSUNDv6QOaAd5KAUMj7aHq7Ijol7YLyd3VZRg1Y8DYWBcykkmJkymT FXujFJQ9vVKM94jDIvmsxk+NFWPzDGw4toShGFZHrFNVG9sHfl4L9eX1HHIkVNuy8xeC ehTz7jJ1uOh9yNM7svAppU5iSfczQoGYNbQrq4zyvVtGsT7qzgVy6BTF9MPiEPiXQj0m /rfGFGAOR/6Sglb4PSBGzDDHeW7m1OktKTr4BIv4c0QmvAFnbHMhvKupCiE52r5xGVhI +V+vMLscRIYBVj2Uc/ZUC/e37QbxPCS4iK3Aukxm8WWM+d8tMLOF6JW78d8PtFWQDxMB 7UpQ== X-Gm-Message-State: AMCzsaVe/uZQfyV0rOZP9Uj9cTaGkFVqYNR8hnsIW8cszfTPNL2maQW7 JS0ufR8y407LR9HVeJkBgk2l1g== X-Google-Smtp-Source: AOwi7QCKyvxBzMMBQgd3U2cEOA1B48pYsWI2hlaCq7sjJauE0eMTIEP/KHXtAM7E6Z5ckAUSMT+0Ug== X-Received: by 10.107.137.164 with SMTP id t36mr2055438ioi.198.1507000333522; Mon, 02 Oct 2017 20:12:13 -0700 (PDT) Received: from node.jintackl-qv28633.kvmarm-pg0.wisc.cloudlab.us (c220g1-031126.wisc.cloudlab.us. [128.104.222.76]) by smtp.gmail.com with ESMTPSA id h84sm5367193iod.72.2017.10.02.20.12.12 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Mon, 02 Oct 2017 20:12:12 -0700 (PDT) From: Jintack Lim To: christoffer.dall@linaro.org, marc.zyngier@arm.com, kvmarm@lists.cs.columbia.edu Cc: jintack@cs.columbia.edu, pbonzini@redhat.com, rkrcmar@redhat.com, catalin.marinas@arm.com, will.deacon@arm.com, linux@armlinux.org.uk, mark.rutland@arm.com, linux-arm-kernel@lists.infradead.org, kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Jintack Lim Subject: [RFC PATCH v2 31/31] KVM: arm64: Fixes to toggle_cache for nesting Date: Mon, 2 Oct 2017 22:11:13 -0500 Message-Id: <1507000273-3735-29-git-send-email-jintack.lim@linaro.org> X-Mailer: git-send-email 1.9.1 In-Reply-To: <1507000273-3735-1-git-send-email-jintack.lim@linaro.org> References: <1507000273-3735-1-git-send-email-jintack.lim@linaro.org> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org From: Christoffer Dall So far we were flushing almost the entire universe whenever a VM would load/unload the SCTLR_EL1 and the two versions of that register had different MMU enabled settings. This turned out to be so slow that it prevented forward progress for a nested VM, because a scheduler timer tick interrupt would always be pending when we reached the nested VM. To avoid this problem, we consider the SCTLR_EL2 when evaluating if caches are on or off when entering virtual EL2 (because this is the value that we end up shadowing onto the hardware EL1 register). Signed-off-by: Christoffer Dall Signed-off-by: Jintack Lim --- arch/arm64/include/asm/kvm_mmu.h | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) -- 1.9.1 diff --git a/arch/arm64/include/asm/kvm_mmu.h b/arch/arm64/include/asm/kvm_mmu.h index 601f431..7a1c581 100644 --- a/arch/arm64/include/asm/kvm_mmu.h +++ b/arch/arm64/include/asm/kvm_mmu.h @@ -240,7 +240,10 @@ static inline bool kvm_page_empty(void *ptr) static inline bool vcpu_has_cache_enabled(struct kvm_vcpu *vcpu) { - return (vcpu_sys_reg(vcpu, SCTLR_EL1) & 0b101) == 0b101; + if (vcpu_mode_el2(vcpu)) + return (vcpu_sys_reg(vcpu, SCTLR_EL2) & 0b101) == 0b101; + else + return (vcpu_sys_reg(vcpu, SCTLR_EL1) & 0b101) == 0b101; } static inline void __coherent_cache_guest_page(struct kvm_vcpu *vcpu,