From patchwork Tue Mar 19 08:30:34 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Amit Kachhap X-Patchwork-Id: 160532 Delivered-To: patch@linaro.org Received: by 2002:ac9:18c7:0:0:0:0:0 with SMTP id i7csp3756821oce; Tue, 19 Mar 2019 01:31:17 -0700 (PDT) X-Google-Smtp-Source: APXvYqztZ3k/G1zjl/tj14IEn2rFJPlY7YuRBnwzv+69DYEFn/Cy2ot/fpSLxA52nN3Bv4Sjs9sA X-Received: by 2002:aa7:8a92:: with SMTP id a18mr754187pfc.218.1552984277529; Tue, 19 Mar 2019 01:31:17 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1552984277; cv=none; d=google.com; s=arc-20160816; b=HW8upebl2avIdYMQIpfhUDf0seN1lEF2XSliMZZ8gczOumdw86h1QUxRI+SqpGNhMM EM/ZTmag3PYa2L+l/SXCWJ6UOA+P3QoBIL+OL2SWZHNN0R/nyymvNTutq06Ntn80UQ3O K0Nm32RHURWDaJDa1dfu0N8uMqIiVZoxp1l7+UVIShxzs0S8TOMg/a6ssJwD/qWFL+8z P0QhbYByct8njdf4Mi6DB1/Wf9RVX3KjK3B5ND9hHv4rK/lTE4QExMKV4mUpeaOYIzg5 6ZELtvwGPbvacydUDkUJTaN3SPz8WecsoW6tClV4C3KY4RefLuLyIoPkZm2lJa2/0r8P IHVw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from; bh=80oEx7EXOWYe9lNm8FtZEQLWb//Yg1syvNwrmVCjoq8=; b=c0iwq9xYPskbnIMjLJ5Mj2xraByMC6XsyHv1Ab1XnySMHAjSXR+bVdq4Au55gg5J0J CWwqjzdSTYwy9iqhzFpPLic47fpTd2teH/kqvG20t9JB35jxmd3nVxYwT6HmglYe6TV1 O4StSMxf+PFo+GE3WCR29AahzFoyepeJpIZxAXdHgMZTFLNiWBKtP0XuZGDhEaNr4w23 7X+hiOv9HThxzuW16Zgqjky1YvZrbuJf5Evh78kwzbJ2Xkeju84rfBhREFdvHq2gY+Gi I9WZYyeT6sN44L057CxoRiOrXE9P3rzqXwH7YB8lICzar8doJT0PZRWtf8WLBMpQbnkg bJhg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id 38si12050670pln.90.2019.03.19.01.31.17; Tue, 19 Mar 2019 01:31:17 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727414AbfCSIbP (ORCPT + 31 others); Tue, 19 Mar 2019 04:31:15 -0400 Received: from foss.arm.com ([217.140.101.70]:46948 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725934AbfCSIbP (ORCPT ); Tue, 19 Mar 2019 04:31:15 -0400 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id DFEEA1596; Tue, 19 Mar 2019 01:31:14 -0700 (PDT) Received: from a075553-lin.blr.arm.com (a075553-lin.blr.arm.com [10.162.0.144]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id A7D953F59C; Tue, 19 Mar 2019 01:31:10 -0700 (PDT) From: Amit Daniel Kachhap To: linux-arm-kernel@lists.infradead.org Cc: Christoffer Dall , Marc Zyngier , Catalin Marinas , Will Deacon , Andrew Jones , Dave Martin , Ramana Radhakrishnan , kvmarm@lists.cs.columbia.edu, Kristina Martsenko , linux-kernel@vger.kernel.org, Amit Daniel Kachhap , Mark Rutland , James Morse , Julien Thierry Subject: [PATCH v7 1/10] KVM: arm64: Propagate vcpu into read_id_reg() Date: Tue, 19 Mar 2019 14:00:34 +0530 Message-Id: <1552984243-7689-2-git-send-email-amit.kachhap@arm.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1552984243-7689-1-git-send-email-amit.kachhap@arm.com> References: <1552984243-7689-1-git-send-email-amit.kachhap@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org From: Dave Martin Architecture features that are conditionally visible to the guest will require run-time checks in the ID register accessor functions. In particular, read_id_reg() will need to perform checks in order to generate the correct emulated value for certain ID register fields such as ID_AA64PFR0_EL1.SVE for example. This patch propagates vcpu into read_id_reg() so that future patches can add run-time checks on the guest configuration here. For now, there is no functional change. Signed-off-by: Dave Martin Reviewed-by: Alex Bennee --- arch/arm64/kvm/sys_regs.c | 23 +++++++++++++---------- 1 file changed, 13 insertions(+), 10 deletions(-) -- 2.7.4 diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c index 539feec..a5d14b5 100644 --- a/arch/arm64/kvm/sys_regs.c +++ b/arch/arm64/kvm/sys_regs.c @@ -1044,7 +1044,8 @@ static bool access_arch_timer(struct kvm_vcpu *vcpu, } /* Read a sanitised cpufeature ID register by sys_reg_desc */ -static u64 read_id_reg(struct sys_reg_desc const *r, bool raz) +static u64 read_id_reg(const struct kvm_vcpu *vcpu, + struct sys_reg_desc const *r, bool raz) { u32 id = sys_reg((u32)r->Op0, (u32)r->Op1, (u32)r->CRn, (u32)r->CRm, (u32)r->Op2); @@ -1078,7 +1079,7 @@ static bool __access_id_reg(struct kvm_vcpu *vcpu, if (p->is_write) return write_to_read_only(vcpu, p, r); - p->regval = read_id_reg(r, raz); + p->regval = read_id_reg(vcpu, r, raz); return true; } @@ -1107,16 +1108,18 @@ static u64 sys_reg_to_index(const struct sys_reg_desc *reg); * are stored, and for set_id_reg() we don't allow the effective value * to be changed. */ -static int __get_id_reg(const struct sys_reg_desc *rd, void __user *uaddr, +static int __get_id_reg(const struct kvm_vcpu *vcpu, + const struct sys_reg_desc *rd, void __user *uaddr, bool raz) { const u64 id = sys_reg_to_index(rd); - const u64 val = read_id_reg(rd, raz); + const u64 val = read_id_reg(vcpu, rd, raz); return reg_to_user(uaddr, &val, id); } -static int __set_id_reg(const struct sys_reg_desc *rd, void __user *uaddr, +static int __set_id_reg(const struct kvm_vcpu *vcpu, + const struct sys_reg_desc *rd, void __user *uaddr, bool raz) { const u64 id = sys_reg_to_index(rd); @@ -1128,7 +1131,7 @@ static int __set_id_reg(const struct sys_reg_desc *rd, void __user *uaddr, return err; /* This is what we mean by invariant: you can't change it. */ - if (val != read_id_reg(rd, raz)) + if (val != read_id_reg(vcpu, rd, raz)) return -EINVAL; return 0; @@ -1137,25 +1140,25 @@ static int __set_id_reg(const struct sys_reg_desc *rd, void __user *uaddr, static int get_id_reg(struct kvm_vcpu *vcpu, const struct sys_reg_desc *rd, const struct kvm_one_reg *reg, void __user *uaddr) { - return __get_id_reg(rd, uaddr, false); + return __get_id_reg(vcpu, rd, uaddr, false); } static int set_id_reg(struct kvm_vcpu *vcpu, const struct sys_reg_desc *rd, const struct kvm_one_reg *reg, void __user *uaddr) { - return __set_id_reg(rd, uaddr, false); + return __set_id_reg(vcpu, rd, uaddr, false); } static int get_raz_id_reg(struct kvm_vcpu *vcpu, const struct sys_reg_desc *rd, const struct kvm_one_reg *reg, void __user *uaddr) { - return __get_id_reg(rd, uaddr, true); + return __get_id_reg(vcpu, rd, uaddr, true); } static int set_raz_id_reg(struct kvm_vcpu *vcpu, const struct sys_reg_desc *rd, const struct kvm_one_reg *reg, void __user *uaddr) { - return __set_id_reg(rd, uaddr, true); + return __set_id_reg(vcpu, rd, uaddr, true); } static bool access_ctr(struct kvm_vcpu *vcpu, struct sys_reg_params *p, From patchwork Tue Mar 19 08:30:35 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Amit Kachhap X-Patchwork-Id: 160533 Delivered-To: patch@linaro.org Received: by 2002:ac9:18c7:0:0:0:0:0 with SMTP id i7csp3756899oce; Tue, 19 Mar 2019 01:31:22 -0700 (PDT) X-Google-Smtp-Source: APXvYqwRri6js9+pDyVEt0/RFxLcVYnTXync707wOlUuWcg46B0+qHGsssEqmkvRcJBIlUpzFkim X-Received: by 2002:a63:3244:: with SMTP id y65mr696545pgy.417.1552984282392; Tue, 19 Mar 2019 01:31:22 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1552984282; cv=none; d=google.com; s=arc-20160816; b=0yyWWOubCmJpzFtHmKC8t94xXQyWAkglVWptwKLjsA/VoIuPVysMPXxFk8YrN84KU6 o5V4Deh9DSfv90RKdilHlIJlatmzreiSHjl5BQwX0SvcRQMki+McWMIphdgyv9VX3xk9 RdJxYo4/YNM9lYxVkUBUGvBsf7GRqVbfYVTrnzUgtf3j0E8KwFChi2iIjevz4QXsBF5N Jv492U1kZryShLiKPcEu3oj9FK0XzhLZ529I9iU0NYMc/26S1ZLmqlFxUNBojbgDqfrr A02HEiIUz0GYo2vCHjrV1Jiejmh7htiev69w2UH2N6f98XB8ut2NYaKhubtfvkK8WEE5 zu0w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from; bh=yttHqopey9catRXJKAhbVKttOklLUvmT5xoITOYkqt0=; b=JSmfE7wJ9zod7Qkyw3C60Rs5M025wWQtodZnTmA1kpUCKeDaeZRZkXsr4cvKiwW/pC DGI5UGNJ+z0bqP/4A0GHCyqHW6+ko6J2OM+UMbib8vlPvsVClAz/EOtoa0fOa+Cb7rMh WnU+A+07rTUlbREQnetRH8+YCqM2oBMOOmMCRi3NeWAGZ2YXNBIyMXghszw2VH9hw/pc h7gwUSuYVqWQ2c/f6C/WlCPQd1XDVxKBFvJHhNta+dwzlM2pKicLWPDWokvkNl8BaCg4 dGosIcS+2tLaxcwrxrZAnO7DlKxBDFUpVukM0la3MyCwvFQ+2XH93HQYbm4wsEec+lln KBhw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id 38si12050670pln.90.2019.03.19.01.31.21; Tue, 19 Mar 2019 01:31:22 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727521AbfCSIbU (ORCPT + 31 others); Tue, 19 Mar 2019 04:31:20 -0400 Received: from usa-sjc-mx-foss1.foss.arm.com ([217.140.101.70]:46968 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725934AbfCSIbU (ORCPT ); Tue, 19 Mar 2019 04:31:20 -0400 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 93FFE174E; Tue, 19 Mar 2019 01:31:19 -0700 (PDT) Received: from a075553-lin.blr.arm.com (a075553-lin.blr.arm.com [10.162.0.144]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 5BCED3F59C; Tue, 19 Mar 2019 01:31:15 -0700 (PDT) From: Amit Daniel Kachhap To: linux-arm-kernel@lists.infradead.org Cc: Christoffer Dall , Marc Zyngier , Catalin Marinas , Will Deacon , Andrew Jones , Dave Martin , Ramana Radhakrishnan , kvmarm@lists.cs.columbia.edu, Kristina Martsenko , linux-kernel@vger.kernel.org, Amit Daniel Kachhap , Mark Rutland , James Morse , Julien Thierry Subject: [PATCH v7 2/10] KVM: arm64: Support runtime sysreg visibility filtering Date: Tue, 19 Mar 2019 14:00:35 +0530 Message-Id: <1552984243-7689-3-git-send-email-amit.kachhap@arm.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1552984243-7689-1-git-send-email-amit.kachhap@arm.com> References: <1552984243-7689-1-git-send-email-amit.kachhap@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org From: Dave Martin Some optional features of the Arm architecture add new system registers that are not present in the base architecture. Where these features are optional for the guest, the visibility of these registers may need to depend on some runtime configuration, such as a flag passed to KVM_ARM_VCPU_INIT. For example, ZCR_EL1 and ID_AA64ZFR0_EL1 need to be hidden if SVE is not enabled for the guest, even though these registers may be present in the hardware and visible to the host at EL2. Adding special-case checks all over the place for individual registers is going to get messy as the number of conditionally- visible registers grows. In order to help solve this problem, this patch adds a new sysreg method restrictions() that can be used to hook in any needed runtime visibility checks. This method can currently return REG_NO_USER to inhibit enumeration and ioctl access to the register for userspace, and REG_NO_GUEST to inhibit runtime access by the guest using MSR/MRS. This allows a conditionally modified view of individual system registers such as the CPU ID registers, in addition to completely hiding register where appropriate. Signed-off-by: Dave Martin --- arch/arm64/kvm/sys_regs.c | 24 +++++++++++++++++++++--- arch/arm64/kvm/sys_regs.h | 13 +++++++++++++ 2 files changed, 34 insertions(+), 3 deletions(-) -- 2.7.4 diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c index a5d14b5..75942f6 100644 --- a/arch/arm64/kvm/sys_regs.c +++ b/arch/arm64/kvm/sys_regs.c @@ -1927,6 +1927,12 @@ static void perform_access(struct kvm_vcpu *vcpu, { trace_kvm_sys_access(*vcpu_pc(vcpu), params, r); + /* Check for regs disabled by runtime config */ + if (restrictions(vcpu, r) & REG_NO_GUEST) { + kvm_inject_undefined(vcpu); + return; + } + /* * Not having an accessor means that we have configured a trap * that we don't know how to handle. This certainly qualifies @@ -2438,6 +2444,10 @@ int kvm_arm_sys_reg_get_reg(struct kvm_vcpu *vcpu, const struct kvm_one_reg *reg if (!r) return get_invariant_sys_reg(reg->id, uaddr); + /* Check for regs disabled by runtime config */ + if (restrictions(vcpu, r) & REG_NO_USER) + return -ENOENT; + if (r->get_user) return (r->get_user)(vcpu, r, reg, uaddr); @@ -2459,6 +2469,10 @@ int kvm_arm_sys_reg_set_reg(struct kvm_vcpu *vcpu, const struct kvm_one_reg *reg if (!r) return set_invariant_sys_reg(reg->id, uaddr); + /* Check for regs disabled by runtime config */ + if (restrictions(vcpu, r) & REG_NO_USER) + return -ENOENT; + if (r->set_user) return (r->set_user)(vcpu, r, reg, uaddr); @@ -2515,7 +2529,8 @@ static bool copy_reg_to_user(const struct sys_reg_desc *reg, u64 __user **uind) return true; } -static int walk_one_sys_reg(const struct sys_reg_desc *rd, +static int walk_one_sys_reg(const struct kvm_vcpu *vcpu, + const struct sys_reg_desc *rd, u64 __user **uind, unsigned int *total) { @@ -2526,6 +2541,9 @@ static int walk_one_sys_reg(const struct sys_reg_desc *rd, if (!(rd->reg || rd->get_user)) return 0; + if (restrictions(vcpu, rd) & REG_NO_USER) + return 0; + if (!copy_reg_to_user(rd, uind)) return -EFAULT; @@ -2554,9 +2572,9 @@ static int walk_sys_regs(struct kvm_vcpu *vcpu, u64 __user *uind) int cmp = cmp_sys_reg(i1, i2); /* target-specific overrides generic entry. */ if (cmp <= 0) - err = walk_one_sys_reg(i1, &uind, &total); + err = walk_one_sys_reg(vcpu, i1, &uind, &total); else - err = walk_one_sys_reg(i2, &uind, &total); + err = walk_one_sys_reg(vcpu, i2, &uind, &total); if (err) return err; diff --git a/arch/arm64/kvm/sys_regs.h b/arch/arm64/kvm/sys_regs.h index 3b1bc7f..b9dd4a9 100644 --- a/arch/arm64/kvm/sys_regs.h +++ b/arch/arm64/kvm/sys_regs.h @@ -64,8 +64,15 @@ struct sys_reg_desc { const struct kvm_one_reg *reg, void __user *uaddr); int (*set_user)(struct kvm_vcpu *vcpu, const struct sys_reg_desc *rd, const struct kvm_one_reg *reg, void __user *uaddr); + + /* Return mask of REG_* runtime restriction flags */ + unsigned int (*restrictions)(const struct kvm_vcpu *vcpu, + const struct sys_reg_desc *rd); }; +#define REG_NO_USER (1 << 0) /* hidden from userspace ioctl interface */ +#define REG_NO_GUEST (1 << 1) /* hidden from guest */ + static inline void print_sys_reg_instr(const struct sys_reg_params *p) { /* Look, we even formatted it for you to paste into the table! */ @@ -102,6 +109,12 @@ static inline void reset_val(struct kvm_vcpu *vcpu, const struct sys_reg_desc *r __vcpu_sys_reg(vcpu, r->reg) = r->val; } +static inline unsigned int restrictions(const struct kvm_vcpu *vcpu, + const struct sys_reg_desc *r) +{ + return unlikely(r->restrictions) ? r->restrictions(vcpu, r) : 0; +} + static inline int cmp_sys_reg(const struct sys_reg_desc *i1, const struct sys_reg_desc *i2) { From patchwork Tue Mar 19 08:30:37 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Amit Kachhap X-Patchwork-Id: 160534 Delivered-To: patch@linaro.org Received: by 2002:ac9:18c7:0:0:0:0:0 with SMTP id i7csp3757048oce; Tue, 19 Mar 2019 01:31:32 -0700 (PDT) X-Google-Smtp-Source: APXvYqwBF2Du0fRBWiFjnZ6msTQiTQtlcZ5Lhbx4pJgO3jrQT6k0QskWKBZphnYyTcH0XJJi8iMc X-Received: by 2002:a63:f407:: with SMTP id g7mr705501pgi.413.1552984292448; Tue, 19 Mar 2019 01:31:32 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1552984292; cv=none; d=google.com; s=arc-20160816; b=VCoW67aL+xQ5TfyPfefBp0o1kvcDesAKCdUMdBi0eNTsxuugtuYAzG6p38LUjn1g3C hHVYponZSHvYbusZrTxUf1eeVRQEiK630/PbXWMKFTTC184WhmU7jacb10yCGZb6mNvl JRywohys1iCFE5dLHYrZK3Fw+sOUwFSKji1XKEZXROXHNUbdR2tF2FIAmlAwzpiyuNAA GGVRGLxZR2BkvqgJuUtMQ2NdHBFqj9Qkn9hoMyEKbGJ05Pe7kMw8ZLSjpdsG6lrncaZX qAJJ7z6VdDqWmkxdza2tH1bAsUyQJgmPfXcK5mJk2OopT8/LQmTVwrgEQpLkt2Rcivid fuYg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from; bh=RmF15WcFJAfZ0zCrDZr2oPK145fZbBpnu/usZ9Mpxrk=; b=Lv1i0m06Hveybj9FjmHvM4PFKipTGwdF8mKP+iJs9T4HFUrl/fwyPBvH1Lxmbh1m2+ JfbY+gF8SI/r7wEvcOsvRGR15bj61Q+/xZPgvO/z8rxV4nqPIrQ0mXU9WTo2Eqwqj7u8 JOSMBHmKQMVrWFXzviRfwrgHNEUcpy10NNIDXMKTiD3yXGrIitT2SisHWqt/Uhcz6mlx /s101Omo97Hlgf0sjkSg6b1D5FmYnZQoUd/BIUPsCG05WIlQr6b1GiXrVYKiNevlV8P8 iEi+YtwtZ+H6SPo/Y02de/lzBSVb8Ah9MqOTOBxtfLdQO85MG4l0ZgTcxG10jMupwWfn VfUQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id l11si3709350plb.159.2019.03.19.01.31.32; Tue, 19 Mar 2019 01:31:32 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727600AbfCSIba (ORCPT + 31 others); Tue, 19 Mar 2019 04:31:30 -0400 Received: from usa-sjc-mx-foss1.foss.arm.com ([217.140.101.70]:46996 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725934AbfCSIb3 (ORCPT ); Tue, 19 Mar 2019 04:31:29 -0400 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id F27771596; Tue, 19 Mar 2019 01:31:28 -0700 (PDT) Received: from a075553-lin.blr.arm.com (a075553-lin.blr.arm.com [10.162.0.144]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id BA4E53F59C; Tue, 19 Mar 2019 01:31:24 -0700 (PDT) From: Amit Daniel Kachhap To: linux-arm-kernel@lists.infradead.org Cc: Christoffer Dall , Marc Zyngier , Catalin Marinas , Will Deacon , Andrew Jones , Dave Martin , Ramana Radhakrishnan , kvmarm@lists.cs.columbia.edu, Kristina Martsenko , linux-kernel@vger.kernel.org, Amit Daniel Kachhap , Mark Rutland , James Morse , Julien Thierry Subject: [PATCH v7 4/10] KVM: arm/arm64: preserve host HCR_EL2 value Date: Tue, 19 Mar 2019 14:00:37 +0530 Message-Id: <1552984243-7689-5-git-send-email-amit.kachhap@arm.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1552984243-7689-1-git-send-email-amit.kachhap@arm.com> References: <1552984243-7689-1-git-send-email-amit.kachhap@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org From: Mark Rutland When restoring HCR_EL2 for the host, KVM uses HCR_HOST_VHE_FLAGS, which is a constant value. This works today, as the host HCR_EL2 value is always the same, but this will get in the way of supporting extensions that require HCR_EL2 bits to be set conditionally for the host. To allow such features to work without KVM having to explicitly handle every possible host feature combination, this patch has KVM save/restore for the host HCR when switching to/from a guest HCR. The saving of the register is done once during cpu hypervisor initialization state and is just restored after switch from guest. For fetching HCR_EL2 during kvm initialisation, a hyp call is made using kvm_call_hyp and is helpful in non-VHE case. For the hyp TLB maintenance code, __tlb_switch_to_host_vhe() is updated to toggle the TGE bit with a RMW sequence, as we already do in __tlb_switch_to_guest_vhe(). The value of hcr_el2 is now stored in struct kvm_cpu_context as both host and guest can now use this field in a common way. Signed-off-by: Mark Rutland [Added cpu_init_host_ctxt, hcr_el2 field in struct kvm_cpu_context, save hcr_el2 in hyp init stage] Signed-off-by: Amit Daniel Kachhap Reviewed-by: James Morse Cc: Marc Zyngier Cc: Christoffer Dall Cc: kvmarm@lists.cs.columbia.edu --- arch/arm/include/asm/kvm_host.h | 2 ++ arch/arm64/include/asm/kvm_asm.h | 2 ++ arch/arm64/include/asm/kvm_emulate.h | 24 ++++++++++++------------ arch/arm64/include/asm/kvm_host.h | 15 ++++++++++++++- arch/arm64/kvm/guest.c | 2 +- arch/arm64/kvm/hyp/switch.c | 22 +++++++++++++--------- arch/arm64/kvm/hyp/sysreg-sr.c | 14 ++++++++++++++ arch/arm64/kvm/hyp/tlb.c | 6 +++++- virt/kvm/arm/arm.c | 1 + 9 files changed, 64 insertions(+), 24 deletions(-) -- 2.7.4 diff --git a/arch/arm/include/asm/kvm_host.h b/arch/arm/include/asm/kvm_host.h index 770d732..6d0aac4 100644 --- a/arch/arm/include/asm/kvm_host.h +++ b/arch/arm/include/asm/kvm_host.h @@ -322,6 +322,8 @@ static inline void __cpu_init_stage2(void) kvm_call_hyp(__init_stage2_translation); } +static inline void cpu_init_host_ctxt(void) {} + static inline int kvm_arch_vm_ioctl_check_extension(struct kvm *kvm, long ext) { return 0; diff --git a/arch/arm64/include/asm/kvm_asm.h b/arch/arm64/include/asm/kvm_asm.h index 57a07e8..a68205c 100644 --- a/arch/arm64/include/asm/kvm_asm.h +++ b/arch/arm64/include/asm/kvm_asm.h @@ -80,6 +80,8 @@ extern void __vgic_v3_init_lrs(void); extern u32 __kvm_get_mdcr_el2(void); +extern void __kvm_populate_host_regs(void); + /* * Obtain the PC-relative address of a kernel symbol * s: symbol diff --git a/arch/arm64/include/asm/kvm_emulate.h b/arch/arm64/include/asm/kvm_emulate.h index d384279..426815d 100644 --- a/arch/arm64/include/asm/kvm_emulate.h +++ b/arch/arm64/include/asm/kvm_emulate.h @@ -50,25 +50,25 @@ void kvm_inject_pabt32(struct kvm_vcpu *vcpu, unsigned long addr); static inline bool vcpu_el1_is_32bit(struct kvm_vcpu *vcpu) { - return !(vcpu->arch.hcr_el2 & HCR_RW); + return !(vcpu->arch.ctxt.hcr_el2 & HCR_RW); } static inline void vcpu_reset_hcr(struct kvm_vcpu *vcpu) { - vcpu->arch.hcr_el2 = HCR_GUEST_FLAGS; + vcpu->arch.ctxt.hcr_el2 = HCR_GUEST_FLAGS; if (is_kernel_in_hyp_mode()) - vcpu->arch.hcr_el2 |= HCR_E2H; + vcpu->arch.ctxt.hcr_el2 |= HCR_E2H; if (cpus_have_const_cap(ARM64_HAS_RAS_EXTN)) { /* route synchronous external abort exceptions to EL2 */ - vcpu->arch.hcr_el2 |= HCR_TEA; + vcpu->arch.ctxt.hcr_el2 |= HCR_TEA; /* trap error record accesses */ - vcpu->arch.hcr_el2 |= HCR_TERR; + vcpu->arch.ctxt.hcr_el2 |= HCR_TERR; } if (cpus_have_const_cap(ARM64_HAS_STAGE2_FWB)) - vcpu->arch.hcr_el2 |= HCR_FWB; + vcpu->arch.ctxt.hcr_el2 |= HCR_FWB; if (test_bit(KVM_ARM_VCPU_EL1_32BIT, vcpu->arch.features)) - vcpu->arch.hcr_el2 &= ~HCR_RW; + vcpu->arch.ctxt.hcr_el2 &= ~HCR_RW; /* * TID3: trap feature register accesses that we virtualise. @@ -76,26 +76,26 @@ static inline void vcpu_reset_hcr(struct kvm_vcpu *vcpu) * are currently virtualised. */ if (!vcpu_el1_is_32bit(vcpu)) - vcpu->arch.hcr_el2 |= HCR_TID3; + vcpu->arch.ctxt.hcr_el2 |= HCR_TID3; if (cpus_have_const_cap(ARM64_MISMATCHED_CACHE_TYPE) || vcpu_el1_is_32bit(vcpu)) - vcpu->arch.hcr_el2 |= HCR_TID2; + vcpu->arch.ctxt.hcr_el2 |= HCR_TID2; } static inline unsigned long *vcpu_hcr(struct kvm_vcpu *vcpu) { - return (unsigned long *)&vcpu->arch.hcr_el2; + return (unsigned long *)&vcpu->arch.ctxt.hcr_el2; } static inline void vcpu_clear_wfe_traps(struct kvm_vcpu *vcpu) { - vcpu->arch.hcr_el2 &= ~HCR_TWE; + vcpu->arch.ctxt.hcr_el2 &= ~HCR_TWE; } static inline void vcpu_set_wfe_traps(struct kvm_vcpu *vcpu) { - vcpu->arch.hcr_el2 |= HCR_TWE; + vcpu->arch.ctxt.hcr_el2 |= HCR_TWE; } static inline unsigned long vcpu_get_vsesr(struct kvm_vcpu *vcpu) diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h index a01fe087..3b09fd0 100644 --- a/arch/arm64/include/asm/kvm_host.h +++ b/arch/arm64/include/asm/kvm_host.h @@ -209,6 +209,8 @@ struct kvm_cpu_context { u32 copro[NR_COPRO_REGS]; }; + /* HYP host/guest configuration */ + u64 hcr_el2; struct kvm_vcpu *__hyp_running_vcpu; }; @@ -225,7 +227,6 @@ struct kvm_vcpu_arch { struct kvm_cpu_context ctxt; /* HYP configuration */ - u64 hcr_el2; u32 mdcr_el2; /* Exception Information */ @@ -510,6 +511,18 @@ int kvm_arm_vcpu_arch_has_attr(struct kvm_vcpu *vcpu, static inline void __cpu_init_stage2(void) {} +/** + * cpu_init_host_ctxt - save the boot hyp configuration registers + * + * It is called per-cpu during CPU hyp initialisation and the + * registers context saved may be used during host/guest context + * switch. + */ +static inline void cpu_init_host_ctxt(void) +{ + kvm_call_hyp(__kvm_populate_host_regs); +} + /* Guest/host FPSIMD coordination helpers */ int kvm_arch_vcpu_run_map_fp(struct kvm_vcpu *vcpu); void kvm_arch_vcpu_load_fp(struct kvm_vcpu *vcpu); diff --git a/arch/arm64/kvm/guest.c b/arch/arm64/kvm/guest.c index dd436a5..e2f0268 100644 --- a/arch/arm64/kvm/guest.c +++ b/arch/arm64/kvm/guest.c @@ -345,7 +345,7 @@ int kvm_arch_vcpu_ioctl_set_sregs(struct kvm_vcpu *vcpu, int __kvm_arm_vcpu_get_events(struct kvm_vcpu *vcpu, struct kvm_vcpu_events *events) { - events->exception.serror_pending = !!(vcpu->arch.hcr_el2 & HCR_VSE); + events->exception.serror_pending = !!(vcpu->arch.ctxt.hcr_el2 & HCR_VSE); events->exception.serror_has_esr = cpus_have_const_cap(ARM64_HAS_RAS_EXTN); if (events->exception.serror_pending && events->exception.serror_has_esr) diff --git a/arch/arm64/kvm/hyp/switch.c b/arch/arm64/kvm/hyp/switch.c index 3563fe6..f5cefa1 100644 --- a/arch/arm64/kvm/hyp/switch.c +++ b/arch/arm64/kvm/hyp/switch.c @@ -129,7 +129,7 @@ static void __hyp_text __activate_traps_nvhe(struct kvm_vcpu *vcpu) static void __hyp_text __activate_traps(struct kvm_vcpu *vcpu) { - u64 hcr = vcpu->arch.hcr_el2; + u64 hcr = vcpu->arch.ctxt.hcr_el2; write_sysreg(hcr, hcr_el2); @@ -142,10 +142,10 @@ static void __hyp_text __activate_traps(struct kvm_vcpu *vcpu) __activate_traps_nvhe(vcpu); } -static void deactivate_traps_vhe(void) +static void deactivate_traps_vhe(struct kvm_cpu_context *host_ctxt) { extern char vectors[]; /* kernel exception vectors */ - write_sysreg(HCR_HOST_VHE_FLAGS, hcr_el2); + write_sysreg(host_ctxt->hcr_el2, hcr_el2); /* * ARM erratum 1165522 requires the actual execution of the above @@ -159,9 +159,10 @@ static void deactivate_traps_vhe(void) } NOKPROBE_SYMBOL(deactivate_traps_vhe); -static void __hyp_text __deactivate_traps_nvhe(void) +static void __hyp_text __deactivate_traps_nvhe(struct kvm_cpu_context *host_ctxt) { u64 mdcr_el2 = read_sysreg(mdcr_el2); + struct kvm_cpu_context *hyp_host_ctxt = kern_hyp_va(host_ctxt); __deactivate_traps_common(); @@ -169,25 +170,28 @@ static void __hyp_text __deactivate_traps_nvhe(void) mdcr_el2 |= MDCR_EL2_E2PB_MASK << MDCR_EL2_E2PB_SHIFT; write_sysreg(mdcr_el2, mdcr_el2); - write_sysreg(HCR_HOST_NVHE_FLAGS, hcr_el2); + write_sysreg(hyp_host_ctxt->hcr_el2, hcr_el2); write_sysreg(CPTR_EL2_DEFAULT, cptr_el2); } static void __hyp_text __deactivate_traps(struct kvm_vcpu *vcpu) { + struct kvm_cpu_context *host_ctxt; + + host_ctxt = vcpu->arch.host_cpu_context; /* * If we pended a virtual abort, preserve it until it gets * cleared. See D1.14.3 (Virtual Interrupts) for details, but * the crucial bit is "On taking a vSError interrupt, * HCR_EL2.VSE is cleared to 0." */ - if (vcpu->arch.hcr_el2 & HCR_VSE) - vcpu->arch.hcr_el2 = read_sysreg(hcr_el2); + if (vcpu->arch.ctxt.hcr_el2 & HCR_VSE) + vcpu->arch.ctxt.hcr_el2 = read_sysreg(hcr_el2); if (has_vhe()) - deactivate_traps_vhe(); + deactivate_traps_vhe(host_ctxt); else - __deactivate_traps_nvhe(); + __deactivate_traps_nvhe(host_ctxt); } void activate_traps_vhe_load(struct kvm_vcpu *vcpu) diff --git a/arch/arm64/kvm/hyp/sysreg-sr.c b/arch/arm64/kvm/hyp/sysreg-sr.c index c52a845..277f82b 100644 --- a/arch/arm64/kvm/hyp/sysreg-sr.c +++ b/arch/arm64/kvm/hyp/sysreg-sr.c @@ -320,3 +320,17 @@ void __hyp_text __kvm_enable_ssbs(void) "msr sctlr_el2, %0" : "=&r" (tmp) : "L" (SCTLR_ELx_DSSBS)); } + +/** + * __kvm_populate_host_regs - Fetches host register values + * + * This function acts as a function handler parameter for kvm_call_hyp and + * may be called from EL1 exception level to fetch the host register values. + */ +void __hyp_text __kvm_populate_host_regs(void) +{ + struct kvm_cpu_context *host_ctxt; + + host_ctxt = __hyp_this_cpu_ptr(kvm_host_cpu_state); + host_ctxt->hcr_el2 = read_sysreg(hcr_el2); +} diff --git a/arch/arm64/kvm/hyp/tlb.c b/arch/arm64/kvm/hyp/tlb.c index 76c3086..c5e7144 100644 --- a/arch/arm64/kvm/hyp/tlb.c +++ b/arch/arm64/kvm/hyp/tlb.c @@ -86,12 +86,16 @@ static hyp_alternate_select(__tlb_switch_to_guest, static void __hyp_text __tlb_switch_to_host_vhe(struct kvm *kvm, struct tlb_inv_context *cxt) { + u64 val; + /* * We're done with the TLB operation, let's restore the host's * view of HCR_EL2. */ write_sysreg(0, vttbr_el2); - write_sysreg(HCR_HOST_VHE_FLAGS, hcr_el2); + val = read_sysreg(hcr_el2); + val |= HCR_TGE; + write_sysreg(val, hcr_el2); isb(); if (cpus_have_const_cap(ARM64_WORKAROUND_1165522)) { diff --git a/virt/kvm/arm/arm.c b/virt/kvm/arm/arm.c index 99c3738..e8c2ee6 100644 --- a/virt/kvm/arm/arm.c +++ b/virt/kvm/arm/arm.c @@ -1323,6 +1323,7 @@ static void cpu_hyp_reinit(void) cpu_init_hyp_mode(NULL); kvm_arm_init_debug(); + cpu_init_host_ctxt(); if (vgic_present) kvm_vgic_init_cpu_hardware(); From patchwork Tue Mar 19 08:30:40 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Amit Kachhap X-Patchwork-Id: 160535 Delivered-To: patch@linaro.org Received: by 2002:ac9:18c7:0:0:0:0:0 with SMTP id i7csp3757243oce; Tue, 19 Mar 2019 01:31:46 -0700 (PDT) X-Google-Smtp-Source: APXvYqwepRJ3XQ4yhD0JY5v8yJmps0l2CkNRoZOUXN3MHUFPRBUbHqfTEWEzqg02ikVvWjml6l5/ X-Received: by 2002:a63:5b4b:: with SMTP id l11mr1184967pgm.82.1552984306415; Tue, 19 Mar 2019 01:31:46 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1552984306; cv=none; d=google.com; s=arc-20160816; b=hHRuCsnRyAL9O+rhKu/trjRCf0dVjtIr1eWUjD3cID45HshktKBG+Yk9q0IBJcd3PF zkJzdTNrOH4sTABJwVMxJ3EezavCZ5lIvMUpEj2P0y/l3eGyUKhErguHLIuxz2r9uUCR Fibn37+jHv3hteIP2nIGZT1KJKiKPzqNg/lVUeu8+isJys/PO9HRo489ehw9/rumHMQ7 CHzIQNACcoCWuJBd+82/qjrZrpwZqJC6DZ4hp7VAY//ed7xkq06C00U7u6b1xlBGc5N4 yFqkmCm0tVJIhlQb63vFlYBZ7+9iP0H8GFB4ifUwv6M4pBM+e9SnwGDvgX1RLzfnTibK kR3w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from; bh=fsxwBAWTu4asJc9HcBqVO2e6xKsq33DLm0CHKH4Nt20=; b=pDNdRv8rSkcoodmwnsQg5JyMLLS7OZJisL4pYjbD2OlVS7b/n1AjF6iWIXQevAec0G tVPtv5NMeztPf7SFSyDOGIuwnjCvesSF+8jjZ9JM9WLRU/dWlGbyXvQ/xAF3iOovhs8i QCBOZzlp60kBQ7jPDo4hA/QXTO8WF0hxBZQrAEV5JVS6OvmJAYV4WH2yqwfSdkHnjfbv WLstl8FdBTq1FNJsVTanEWSOzHgu6XtCgx7X/JTVRxoNp9Chr4IT0DPV7XHIfwkwvii8 z7TKr1PzpFQDKGbJNd34TZ/oeiJmDrf7058doVEO2ALqiYF1SIy4fuzzAnDrHtmrY0Pp TgwQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id n20si10744046pgb.78.2019.03.19.01.31.45; Tue, 19 Mar 2019 01:31:46 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727685AbfCSIbo (ORCPT + 31 others); Tue, 19 Mar 2019 04:31:44 -0400 Received: from usa-sjc-mx-foss1.foss.arm.com ([217.140.101.70]:47054 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725934AbfCSIbn (ORCPT ); Tue, 19 Mar 2019 04:31:43 -0400 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 4F5EA1596; Tue, 19 Mar 2019 01:31:43 -0700 (PDT) Received: from a075553-lin.blr.arm.com (a075553-lin.blr.arm.com [10.162.0.144]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id CEE683F59C; Tue, 19 Mar 2019 01:31:38 -0700 (PDT) From: Amit Daniel Kachhap To: linux-arm-kernel@lists.infradead.org Cc: Christoffer Dall , Marc Zyngier , Catalin Marinas , Will Deacon , Andrew Jones , Dave Martin , Ramana Radhakrishnan , kvmarm@lists.cs.columbia.edu, Kristina Martsenko , linux-kernel@vger.kernel.org, Amit Daniel Kachhap , Mark Rutland , James Morse , Julien Thierry Subject: [PATCH v7 7/10] KVM: arm/arm64: context-switch ptrauth registers Date: Tue, 19 Mar 2019 14:00:40 +0530 Message-Id: <1552984243-7689-8-git-send-email-amit.kachhap@arm.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1552984243-7689-1-git-send-email-amit.kachhap@arm.com> References: <1552984243-7689-1-git-send-email-amit.kachhap@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org From: Mark Rutland When pointer authentication is supported, a guest may wish to use it. This patch adds the necessary KVM infrastructure for this to work, with a semi-lazy context switch of the pointer auth state. Pointer authentication feature is only enabled when VHE is built in the kernel and present in the CPU implementation so only VHE code paths are modified. When we schedule a vcpu, we disable guest usage of pointer authentication instructions and accesses to the keys. While these are disabled, we avoid context-switching the keys. When we trap the guest trying to use pointer authentication functionality, we change to eagerly context-switching the keys, and enable the feature. The next time the vcpu is scheduled out/in, we start again. However the host key save is optimized and implemented inside ptrauth instruction/register access trap. Pointer authentication consists of address authentication and generic authentication, and CPUs in a system might have varied support for either. Where support for either feature is not uniform, it is hidden from guests via ID register emulation, as a result of the cpufeature framework in the host. Unfortunately, address authentication and generic authentication cannot be trapped separately, as the architecture provides a single EL2 trap covering both. If we wish to expose one without the other, we cannot prevent a (badly-written) guest from intermittently using a feature which is not uniformly supported (when scheduled on a physical CPU which supports the relevant feature). Hence, this patch expects both type of authentication to be present in a cpu. This switch of key is done from guest enter/exit assembly as preperation for the upcoming in-kernel pointer authentication support. Hence, these key switching routines are not implemented in C code as they may cause pointer authentication key signing error in some situations. Signed-off-by: Mark Rutland [Only VHE, key switch in full assembly, vcpu_has_ptrauth checks , save host key in ptrauth exception trap] Signed-off-by: Amit Daniel Kachhap Reviewed-by: Julien Thierry Cc: Marc Zyngier Cc: Christoffer Dall Cc: kvmarm@lists.cs.columbia.edu --- arch/arm64/include/asm/kvm_host.h | 17 ++++++ arch/arm64/include/asm/kvm_ptrauth_asm.h | 100 +++++++++++++++++++++++++++++++ arch/arm64/kernel/asm-offsets.c | 6 ++ arch/arm64/kvm/guest.c | 14 +++++ arch/arm64/kvm/handle_exit.c | 24 +++++--- arch/arm64/kvm/hyp/entry.S | 7 +++ arch/arm64/kvm/reset.c | 7 +++ arch/arm64/kvm/sys_regs.c | 46 +++++++++++++- virt/kvm/arm/arm.c | 2 + 9 files changed, 212 insertions(+), 11 deletions(-) create mode 100644 arch/arm64/include/asm/kvm_ptrauth_asm.h -- 2.7.4 diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h index 9dd2918..61239a6 100644 --- a/arch/arm64/include/asm/kvm_host.h +++ b/arch/arm64/include/asm/kvm_host.h @@ -152,6 +152,18 @@ enum vcpu_sysreg { PMSWINC_EL0, /* Software Increment Register */ PMUSERENR_EL0, /* User Enable Register */ + /* Pointer Authentication Registers */ + APIAKEYLO_EL1, + APIAKEYHI_EL1, + APIBKEYLO_EL1, + APIBKEYHI_EL1, + APDAKEYLO_EL1, + APDAKEYHI_EL1, + APDBKEYLO_EL1, + APDBKEYHI_EL1, + APGAKEYLO_EL1, + APGAKEYHI_EL1, + /* 32bit specific registers. Keep them at the end of the range */ DACR32_EL2, /* Domain Access Control Register */ IFSR32_EL2, /* Instruction Fault Status Register */ @@ -497,6 +509,11 @@ static inline bool kvm_arch_requires_vhe(void) test_bit(KVM_ARM_VCPU_PTRAUTH_ADDRESS, vcpu->arch.features) && \ test_bit(KVM_ARM_VCPU_PTRAUTH_GENERIC, vcpu->arch.features)) +void kvm_arm_vcpu_ptrauth_enable(struct kvm_vcpu *vcpu); +void kvm_arm_vcpu_ptrauth_disable(struct kvm_vcpu *vcpu); +void kvm_arm_vcpu_ptrauth_setup_lazy(struct kvm_vcpu *vcpu); +void kvm_arm_vcpu_ptrauth_trap(struct kvm_vcpu *vcpu); + static inline void kvm_arch_hardware_unsetup(void) {} static inline void kvm_arch_sync_events(struct kvm *kvm) {} static inline void kvm_arch_vcpu_uninit(struct kvm_vcpu *vcpu) {} diff --git a/arch/arm64/include/asm/kvm_ptrauth_asm.h b/arch/arm64/include/asm/kvm_ptrauth_asm.h new file mode 100644 index 0000000..97bb040 --- /dev/null +++ b/arch/arm64/include/asm/kvm_ptrauth_asm.h @@ -0,0 +1,100 @@ +/* SPDX-License-Identifier: GPL-2.0 + * arch/arm64/include/asm/kvm_ptrauth_asm.h: Guest/host ptrauth save/restore + * Copyright 2019 Arm Limited + * Author: Mark Rutland + * Amit Daniel Kachhap + */ + +#ifndef __ASM_KVM_ASM_PTRAUTH_H +#define __ASM_KVM_ASM_PTRAUTH_H + +#ifndef __ASSEMBLY__ + +#define __ptrauth_save_key(regs, key) \ +({ \ + regs[key ## KEYLO_EL1] = read_sysreg_s(SYS_ ## key ## KEYLO_EL1); \ + regs[key ## KEYHI_EL1] = read_sysreg_s(SYS_ ## key ## KEYHI_EL1); \ +}) + +#define __ptrauth_save_state(ctxt) \ +({ \ + __ptrauth_save_key(ctxt->sys_regs, APIA); \ + __ptrauth_save_key(ctxt->sys_regs, APIB); \ + __ptrauth_save_key(ctxt->sys_regs, APDA); \ + __ptrauth_save_key(ctxt->sys_regs, APDB); \ + __ptrauth_save_key(ctxt->sys_regs, APGA); \ +}) + +#else /* __ASSEMBLY__ */ + +#include + +#ifdef CONFIG_ARM64_PTR_AUTH + +#define PTRAUTH_REG_OFFSET(x) (x - CPU_APIAKEYLO_EL1) + +.macro ptrauth_save_state base, reg1, reg2 + mrs_s \reg1, SYS_APIAKEYLO_EL1 + mrs_s \reg2, SYS_APIAKEYHI_EL1 + stp \reg1, \reg2, [\base, #PTRAUTH_REG_OFFSET(CPU_APIAKEYLO_EL1)] + mrs_s \reg1, SYS_APIBKEYLO_EL1 + mrs_s \reg2, SYS_APIBKEYHI_EL1 + stp \reg1, \reg2, [\base, #PTRAUTH_REG_OFFSET(CPU_APIBKEYLO_EL1)] + mrs_s \reg1, SYS_APDAKEYLO_EL1 + mrs_s \reg2, SYS_APDAKEYHI_EL1 + stp \reg1, \reg2, [\base, #PTRAUTH_REG_OFFSET(CPU_APDAKEYLO_EL1)] + mrs_s \reg1, SYS_APDBKEYLO_EL1 + mrs_s \reg2, SYS_APDBKEYHI_EL1 + stp \reg1, \reg2, [\base, #PTRAUTH_REG_OFFSET(CPU_APDBKEYLO_EL1)] + mrs_s \reg1, SYS_APGAKEYLO_EL1 + mrs_s \reg2, SYS_APGAKEYHI_EL1 + stp \reg1, \reg2, [\base, #PTRAUTH_REG_OFFSET(CPU_APGAKEYLO_EL1)] +.endm + +.macro ptrauth_restore_state base, reg1, reg2 + ldp \reg1, \reg2, [\base, #PTRAUTH_REG_OFFSET(CPU_APIAKEYLO_EL1)] + msr_s SYS_APIAKEYLO_EL1, \reg1 + msr_s SYS_APIAKEYHI_EL1, \reg2 + ldp \reg1, \reg2, [\base, #PTRAUTH_REG_OFFSET(CPU_APIBKEYLO_EL1)] + msr_s SYS_APIBKEYLO_EL1, \reg1 + msr_s SYS_APIBKEYHI_EL1, \reg2 + ldp \reg1, \reg2, [\base, #PTRAUTH_REG_OFFSET(CPU_APDAKEYLO_EL1)] + msr_s SYS_APDAKEYLO_EL1, \reg1 + msr_s SYS_APDAKEYHI_EL1, \reg2 + ldp \reg1, \reg2, [\base, #PTRAUTH_REG_OFFSET(CPU_APDBKEYLO_EL1)] + msr_s SYS_APDBKEYLO_EL1, \reg1 + msr_s SYS_APDBKEYHI_EL1, \reg2 + ldp \reg1, \reg2, [\base, #PTRAUTH_REG_OFFSET(CPU_APGAKEYLO_EL1)] + msr_s SYS_APGAKEYLO_EL1, \reg1 + msr_s SYS_APGAKEYHI_EL1, \reg2 + .endm + +.macro ptrauth_switch_to_guest g_ctxt, reg1, reg2, reg3 + ldr \reg1, [\g_ctxt, #CPU_HCR_EL2] + and \reg1, \reg1, #(HCR_API | HCR_APK) + cbz \reg1, skip_switch_to_guest + add \reg1, \g_ctxt, #CPU_APIAKEYLO_EL1 + ptrauth_restore_state \reg1, \reg2, \reg3 +skip_switch_to_guest: +.endm + +.macro ptrauth_switch_to_host g_ctxt, h_ctxt, reg1, reg2, reg3 + ldr \reg1, [\g_ctxt, #CPU_HCR_EL2] + and \reg1, \reg1, #(HCR_API | HCR_APK) + cbz \reg1, skip_switch_to_host + add \reg1, \g_ctxt, #CPU_APIAKEYLO_EL1 + ptrauth_save_state \reg1, \reg2, \reg3 + add \reg1, \h_ctxt, #CPU_APIAKEYLO_EL1 + ptrauth_restore_state \reg1, \reg2, \reg3 + isb +skip_switch_to_host: +.endm + +#else /* !CONFIG_ARM64_PTR_AUTH */ +.macro ptrauth_switch_to_guest g_ctxt, reg1, reg2, reg3 +.endm +.macro ptrauth_switch_to_host g_ctxt, h_ctxt, reg1, reg2, reg3 +.endm +#endif /* CONFIG_ARM64_PTR_AUTH */ +#endif /* __ASSEMBLY__ */ +#endif /* __ASM_KVM_ASM_PTRAUTH_H */ diff --git a/arch/arm64/kernel/asm-offsets.c b/arch/arm64/kernel/asm-offsets.c index 7f40dcb..12ca916 100644 --- a/arch/arm64/kernel/asm-offsets.c +++ b/arch/arm64/kernel/asm-offsets.c @@ -126,6 +126,12 @@ int main(void) DEFINE(VCPU_FAULT_DISR, offsetof(struct kvm_vcpu, arch.fault.disr_el1)); DEFINE(VCPU_WORKAROUND_FLAGS, offsetof(struct kvm_vcpu, arch.workaround_flags)); DEFINE(CPU_GP_REGS, offsetof(struct kvm_cpu_context, gp_regs)); + DEFINE(CPU_APIAKEYLO_EL1, offsetof(struct kvm_cpu_context, sys_regs[APIAKEYLO_EL1])); + DEFINE(CPU_APIBKEYLO_EL1, offsetof(struct kvm_cpu_context, sys_regs[APIBKEYLO_EL1])); + DEFINE(CPU_APDAKEYLO_EL1, offsetof(struct kvm_cpu_context, sys_regs[APDAKEYLO_EL1])); + DEFINE(CPU_APDBKEYLO_EL1, offsetof(struct kvm_cpu_context, sys_regs[APDBKEYLO_EL1])); + DEFINE(CPU_APGAKEYLO_EL1, offsetof(struct kvm_cpu_context, sys_regs[APGAKEYLO_EL1])); + DEFINE(CPU_HCR_EL2, offsetof(struct kvm_cpu_context, hcr_el2)); DEFINE(CPU_USER_PT_REGS, offsetof(struct kvm_regs, regs)); DEFINE(HOST_CONTEXT_VCPU, offsetof(struct kvm_cpu_context, __hyp_running_vcpu)); #endif diff --git a/arch/arm64/kvm/guest.c b/arch/arm64/kvm/guest.c index e2f0268..9f591ad 100644 --- a/arch/arm64/kvm/guest.c +++ b/arch/arm64/kvm/guest.c @@ -544,3 +544,17 @@ int kvm_arm_vcpu_arch_has_attr(struct kvm_vcpu *vcpu, return ret; } + +/** + * kvm_arm_vcpu_ptrauth_setup_lazy - setup lazy ptrauth for vcpu schedule + * + * @vcpu: The VCPU pointer + * + * This function may be used to disable ptrauth and use it in a lazy context + * via traps. + */ +void kvm_arm_vcpu_ptrauth_setup_lazy(struct kvm_vcpu *vcpu) +{ + if (vcpu_has_ptrauth(vcpu)) + kvm_arm_vcpu_ptrauth_disable(vcpu); +} diff --git a/arch/arm64/kvm/handle_exit.c b/arch/arm64/kvm/handle_exit.c index 0b79834..5838ff9 100644 --- a/arch/arm64/kvm/handle_exit.c +++ b/arch/arm64/kvm/handle_exit.c @@ -30,6 +30,7 @@ #include #include #include +#include #include #include @@ -174,19 +175,26 @@ static int handle_sve(struct kvm_vcpu *vcpu, struct kvm_run *run) } /* + * Handle the guest trying to use a ptrauth instruction, or trying to access a + * ptrauth register. + */ +void kvm_arm_vcpu_ptrauth_trap(struct kvm_vcpu *vcpu) +{ + if (vcpu_has_ptrauth(vcpu)) { + kvm_arm_vcpu_ptrauth_enable(vcpu); + __ptrauth_save_state(vcpu->arch.host_cpu_context); + } else { + kvm_inject_undefined(vcpu); + } +} + +/* * Guest usage of a ptrauth instruction (which the guest EL1 did not turn into * a NOP). */ static int kvm_handle_ptrauth(struct kvm_vcpu *vcpu, struct kvm_run *run) { - /* - * We don't currently support ptrauth in a guest, and we mask the ID - * registers to prevent well-behaved guests from trying to make use of - * it. - * - * Inject an UNDEF, as if the feature really isn't present. - */ - kvm_inject_undefined(vcpu); + kvm_arm_vcpu_ptrauth_trap(vcpu); return 1; } diff --git a/arch/arm64/kvm/hyp/entry.S b/arch/arm64/kvm/hyp/entry.S index 675fdc1..3a70213 100644 --- a/arch/arm64/kvm/hyp/entry.S +++ b/arch/arm64/kvm/hyp/entry.S @@ -24,6 +24,7 @@ #include #include #include +#include #define CPU_GP_REG_OFFSET(x) (CPU_GP_REGS + x) #define CPU_XREG_OFFSET(x) CPU_GP_REG_OFFSET(CPU_USER_PT_REGS + 8*x) @@ -64,6 +65,9 @@ ENTRY(__guest_enter) add x18, x0, #VCPU_CONTEXT + // Macro ptrauth_switch_to_guest(guest cxt, tmp1, tmp2, tmp3). + ptrauth_switch_to_guest x18, x0, x1, x2 + // Restore guest regs x0-x17 ldp x0, x1, [x18, #CPU_XREG_OFFSET(0)] ldp x2, x3, [x18, #CPU_XREG_OFFSET(2)] @@ -118,6 +122,9 @@ ENTRY(__guest_exit) get_host_ctxt x2, x3 + // Macro ptrauth_switch_to_host(guest cxt, host cxt, tmp1, tmp2, tmp3). + ptrauth_switch_to_host x1, x2, x3, x4, x5 + // Now restore the host regs restore_callee_saved_regs x2 diff --git a/arch/arm64/kvm/reset.c b/arch/arm64/kvm/reset.c index f16a5f8..00f0639 100644 --- a/arch/arm64/kvm/reset.c +++ b/arch/arm64/kvm/reset.c @@ -128,6 +128,13 @@ int kvm_reset_vcpu(struct kvm_vcpu *vcpu) if (loaded) kvm_arch_vcpu_put(vcpu); + if (test_bit(KVM_ARM_VCPU_PTRAUTH_ADDRESS, vcpu->arch.features) || + test_bit(KVM_ARM_VCPU_PTRAUTH_GENERIC, vcpu->arch.features)) { + /* Verify that KVM startup matches the conditions for ptrauth */ + if (WARN_ON(!vcpu_has_ptrauth(vcpu))) + return -EINVAL; + } + switch (vcpu->arch.target) { default: if (test_bit(KVM_ARM_VCPU_EL1_32BIT, vcpu->arch.features)) { diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c index 75942f6..ed6613e 100644 --- a/arch/arm64/kvm/sys_regs.c +++ b/arch/arm64/kvm/sys_regs.c @@ -1007,6 +1007,38 @@ static bool access_pmuserenr(struct kvm_vcpu *vcpu, struct sys_reg_params *p, { SYS_DESC(SYS_PMEVTYPERn_EL0(n)), \ access_pmu_evtyper, reset_unknown, (PMEVTYPER0_EL0 + n), } +void kvm_arm_vcpu_ptrauth_enable(struct kvm_vcpu *vcpu) +{ + vcpu->arch.ctxt.hcr_el2 |= (HCR_API | HCR_APK); +} + +void kvm_arm_vcpu_ptrauth_disable(struct kvm_vcpu *vcpu) +{ + vcpu->arch.ctxt.hcr_el2 &= ~(HCR_API | HCR_APK); +} + +static bool trap_ptrauth(struct kvm_vcpu *vcpu, + struct sys_reg_params *p, + const struct sys_reg_desc *rd) +{ + kvm_arm_vcpu_ptrauth_trap(vcpu); + return false; +} + +static unsigned int ptrauth_restrictions(const struct kvm_vcpu *vcpu, + const struct sys_reg_desc *rd) +{ + return vcpu_has_ptrauth(vcpu) ? 0 : REG_NO_USER | REG_NO_GUEST; +} + +#define __PTRAUTH_KEY(k) \ + { SYS_DESC(SYS_## k), trap_ptrauth, reset_unknown, k, \ + .restrictions = ptrauth_restrictions} + +#define PTRAUTH_KEY(k) \ + __PTRAUTH_KEY(k ## KEYLO_EL1), \ + __PTRAUTH_KEY(k ## KEYHI_EL1) + static bool access_arch_timer(struct kvm_vcpu *vcpu, struct sys_reg_params *p, const struct sys_reg_desc *r) @@ -1061,9 +1093,11 @@ static u64 read_id_reg(const struct kvm_vcpu *vcpu, (0xfUL << ID_AA64ISAR1_API_SHIFT) | (0xfUL << ID_AA64ISAR1_GPA_SHIFT) | (0xfUL << ID_AA64ISAR1_GPI_SHIFT); - if (val & ptrauth_mask) - kvm_debug("ptrauth unsupported for guests, suppressing\n"); - val &= ~ptrauth_mask; + if (!vcpu_has_ptrauth(vcpu)) { + if (val & ptrauth_mask) + kvm_debug("ptrauth unsupported for guests, suppressing\n"); + val &= ~ptrauth_mask; + } } return val; @@ -1387,6 +1421,12 @@ static const struct sys_reg_desc sys_reg_descs[] = { { SYS_DESC(SYS_TTBR1_EL1), access_vm_reg, reset_unknown, TTBR1_EL1 }, { SYS_DESC(SYS_TCR_EL1), access_vm_reg, reset_val, TCR_EL1, 0 }, + PTRAUTH_KEY(APIA), + PTRAUTH_KEY(APIB), + PTRAUTH_KEY(APDA), + PTRAUTH_KEY(APDB), + PTRAUTH_KEY(APGA), + { SYS_DESC(SYS_AFSR0_EL1), access_vm_reg, reset_unknown, AFSR0_EL1 }, { SYS_DESC(SYS_AFSR1_EL1), access_vm_reg, reset_unknown, AFSR1_EL1 }, { SYS_DESC(SYS_ESR_EL1), access_vm_reg, reset_unknown, ESR_EL1 }, diff --git a/virt/kvm/arm/arm.c b/virt/kvm/arm/arm.c index 58de0ca..50ae066 100644 --- a/virt/kvm/arm/arm.c +++ b/virt/kvm/arm/arm.c @@ -385,6 +385,8 @@ void kvm_arch_vcpu_load(struct kvm_vcpu *vcpu, int cpu) vcpu_clear_wfe_traps(vcpu); else vcpu_set_wfe_traps(vcpu); + + kvm_arm_vcpu_ptrauth_setup_lazy(vcpu); } void kvm_arch_vcpu_put(struct kvm_vcpu *vcpu)