From patchwork Fri Jan 5 13:12:31 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Will Deacon X-Patchwork-Id: 123514 Delivered-To: patch@linaro.org Received: by 10.140.22.227 with SMTP id 90csp800293qgn; Fri, 5 Jan 2018 05:13:00 -0800 (PST) X-Google-Smtp-Source: ACJfBosXXItqidwTkoysypV6VxD6hc1fLZXIYTAXtFIGFfcH/ZzelMLPpR2X91WqNyEunGC4uZqm X-Received: by 10.101.82.138 with SMTP id y10mr2512014pgp.165.1515157980592; Fri, 05 Jan 2018 05:13:00 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1515157980; cv=none; d=google.com; s=arc-20160816; b=tEOrJ4VjKvayA5y5HvfU+/KQyfGs54rlNmtbDXysqWToX3bJERWoqYELOABPnPoVws mR4Ra1q7Jq91nPiJChROWNqGAVSJs8wPz8Fnc9UXzWK9ilYcygdEqwqBB06LGU9p1ZyY SMsw0kTKDdi4EC65I5Bt8lmEAcNMfd6oI4PzcxMGtngoL4ENGfaBQZsm+z2HCY7kDZ85 u6ZA1yIrh5qN1mbcgp2jO4RrUFtf4XUpXyDVR9jIbDJPKX/ODA35aqSPXE3JJRh7q0zB j/i7E5pKRNVSyg9Uu9ZqQI3bOXqYTmVGIlLxZlbARr7gJT39uTaQNlZSQLtTIb5AqEAb axkg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:arc-authentication-results; bh=awGtDl62glKTvTBCE4i4fEhmTFdGWhWh/KYVIWG24Uk=; b=EGYNWTVEQ8PsCpmQhwLsVGYC9DQKwgUuT5nWPwJeKQGktLttXbNZoG4QCSGFWFXbIm 7YHFCh1LxRXUFUBNAxZpeaJcusi3wwNAlK6LuVMyMOwbFvAVMAOTMT9ULX7K8rySdJX3 W6Hb+lUKNmoIcBRs9pLPaefBu0gnV5Fk4vX0+Sfbt6PmhnVkoNVy73yqXRHePoDF79Oy v9uTGRrpc8hI6p57v7ogw3v5WQ/JN/wE0xU1/WHntOA5RoL8vdf/lJbXIPHWNt7VGdg3 85gwfIohVHUmuOzNRz75YJisaVcgEoQxq5CPkSls+9fl4M9e0umwkcqJft0AIs6/TrNS 9afw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id a2si4097383plt.229.2018.01.05.05.13.00; Fri, 05 Jan 2018 05:13:00 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751895AbeAENMr (ORCPT + 27 others); Fri, 5 Jan 2018 08:12:47 -0500 Received: from usa-sjc-mx-foss1.foss.arm.com ([217.140.101.70]:44724 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1750880AbeAENMn (ORCPT ); Fri, 5 Jan 2018 08:12:43 -0500 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id B08B91596; Fri, 5 Jan 2018 05:12:42 -0800 (PST) Received: from edgewater-inn.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 7EDDB3F6CF; Fri, 5 Jan 2018 05:12:42 -0800 (PST) Received: by edgewater-inn.cambridge.arm.com (Postfix, from userid 1000) id EB2101AE17AB; Fri, 5 Jan 2018 13:12:42 +0000 (GMT) From: Will Deacon To: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com, ard.biesheuvel@linaro.org, marc.zyngier@arm.com, lorenzo.pieralisi@arm.com, christoffer.dall@linaro.org, linux-kernel@vger.kernel.org, labbott@redhat.com, Will Deacon Subject: [PATCH v2 01/11] arm64: use RET instruction for exiting the trampoline Date: Fri, 5 Jan 2018 13:12:31 +0000 Message-Id: <1515157961-20963-2-git-send-email-will.deacon@arm.com> X-Mailer: git-send-email 2.1.4 In-Reply-To: <1515157961-20963-1-git-send-email-will.deacon@arm.com> References: <1515157961-20963-1-git-send-email-will.deacon@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Speculation attacks against the entry trampoline can potentially resteer the speculative instruction stream through the indirect branch and into arbitrary gadgets within the kernel. This patch defends against these attacks by forcing a misprediction through the return stack: a dummy BL instruction loads an entry into the stack, so that the predicted program flow of the subsequent RET instruction is to a branch-to-self instruction which is finally resolved as a branch to the kernel vectors with speculation suppressed. Signed-off-by: Will Deacon --- arch/arm64/kernel/entry.S | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) -- 2.1.4 diff --git a/arch/arm64/kernel/entry.S b/arch/arm64/kernel/entry.S index 031392ee5f47..71092ee09b6b 100644 --- a/arch/arm64/kernel/entry.S +++ b/arch/arm64/kernel/entry.S @@ -1029,6 +1029,14 @@ alternative_else_nop_endif .if \regsize == 64 msr tpidrro_el0, x30 // Restored in kernel_ventry .endif + /* + * Defend against branch aliasing attacks by pushing a dummy + * entry onto the return stack and using a RET instruction to + * entr the full-fat kernel vectors. + */ + bl 2f + b . +2: tramp_map_kernel x30 #ifdef CONFIG_RANDOMIZE_BASE adr x30, tramp_vectors + PAGE_SIZE @@ -1041,7 +1049,7 @@ alternative_insn isb, nop, ARM64_WORKAROUND_QCOM_FALKOR_E1003 msr vbar_el1, x30 add x30, x30, #(1b - tramp_vectors) isb - br x30 + ret .endm .macro tramp_exit, regsize = 64 From patchwork Fri Jan 5 13:12:32 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Will Deacon X-Patchwork-Id: 123513 Delivered-To: patch@linaro.org Received: by 10.140.22.227 with SMTP id 90csp800113qgn; Fri, 5 Jan 2018 05:12:50 -0800 (PST) X-Google-Smtp-Source: ACJfBovAJtRZv1oBSIhpRaYiwd0nL4KxMGLja4ioNs61TO9jwnwZQjjhjKeLNKYhG6G56pAdhP19 X-Received: by 10.84.211.39 with SMTP id b36mr3128818pli.217.1515157970719; Fri, 05 Jan 2018 05:12:50 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1515157970; cv=none; d=google.com; s=arc-20160816; b=FcAy9Gis79D0LkSkF3a3w/G+JX21q4qS1q96jw/b4AYXCPQaLmd+jlgRiFmyoPoMEg SPuivD0ms7CvnOQmtkeYF/aEVWm0V43nlyD2q9f4RNG6zhhMUACM7slXDr/cPGTGB1Vf F7Zk0kkNs9l7LVuPNaxYIphJrTU8QPiikg9at7u8DU3nNNf8pQPtH46zaGpx0c7l9zZA 8XHdTiu5qDAjMn2+EHKsuGlqQipd/+a11+pRTf5cvbAyzRJk3MPbAuPpBUSXA5aUZS1a THbDTq+YpfoD/NJ9BnM4bx0vpjjd2mJTkkWRx+IoaUlMpvExqgiFg7PODdpiWvKXLFqx 2u2w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:arc-authentication-results; bh=fyr47MyjiLpFN9wd4C3cjBopnAx0Q8tHK0n2/CJh4PE=; b=S5YzlAxzPiBHubv4xUP+781h8CpF/iJQkdR9nKEWhsUjO1tIq0uO+ERw0TIo/1pd9l eNxIngxBuEPXtJVi7uDkFrgxLa1ORcQA2CP/ZS3kTyw1Ll+ujRZjAANP1/+6SjtiZ3mc gNyJkrrUlBlS/pOtoPf9dwHEKrNnBDh0ytcWZAc48JOJw7xDxFn8Bh5UujardNpevZTY BQqxKUfSh7YqbKMU0jkW95A2LUV0OxBxvoXtkXRbqa/cSDxoO6L0nqiMfur/NJISoTfM 8sXoHn6hDzIC5+G3II4FPE7S5WqUcIoKmYeBcLGgoLJUjeiN7SaPZdDjjl2rC6QtuNDe cWcQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id f30si3999185plf.468.2018.01.05.05.12.50; Fri, 05 Jan 2018 05:12:50 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751932AbeAENMs (ORCPT + 27 others); Fri, 5 Jan 2018 08:12:48 -0500 Received: from usa-sjc-mx-foss1.foss.arm.com ([217.140.101.70]:44732 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1750915AbeAENMn (ORCPT ); Fri, 5 Jan 2018 08:12:43 -0500 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id C1C2215BE; Fri, 5 Jan 2018 05:12:42 -0800 (PST) Received: from edgewater-inn.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 901D83F77C; Fri, 5 Jan 2018 05:12:42 -0800 (PST) Received: by edgewater-inn.cambridge.arm.com (Postfix, from userid 1000) id 07E2F1AE17D1; Fri, 5 Jan 2018 13:12:42 +0000 (GMT) From: Will Deacon To: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com, ard.biesheuvel@linaro.org, marc.zyngier@arm.com, lorenzo.pieralisi@arm.com, christoffer.dall@linaro.org, linux-kernel@vger.kernel.org, labbott@redhat.com, Will Deacon Subject: [PATCH v2 02/11] arm64: Kconfig: Reword UNMAP_KERNEL_AT_EL0 kconfig entry Date: Fri, 5 Jan 2018 13:12:32 +0000 Message-Id: <1515157961-20963-3-git-send-email-will.deacon@arm.com> X-Mailer: git-send-email 2.1.4 In-Reply-To: <1515157961-20963-1-git-send-email-will.deacon@arm.com> References: <1515157961-20963-1-git-send-email-will.deacon@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Although CONFIG_UNMAP_KERNEL_AT_EL0 does make KASLR more robust, it's actually more useful as a mitigation against speculation attacks that can leak arbitrary kernel data to userspace through speculation. Reword the Kconfig help message to reflect this, and make the option depend on EXPERT so that it is on by default for the majority of users. Signed-off-by: Will Deacon --- arch/arm64/Kconfig | 13 ++++++------- 1 file changed, 6 insertions(+), 7 deletions(-) -- 2.1.4 diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index 3af1657fcac3..efaaa3a66b95 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -834,15 +834,14 @@ config FORCE_MAX_ZONEORDER 4M allocations matching the default size used by generic code. config UNMAP_KERNEL_AT_EL0 - bool "Unmap kernel when running in userspace (aka \"KAISER\")" + bool "Unmap kernel when running in userspace (aka \"KAISER\")" if EXPERT default y help - Some attacks against KASLR make use of the timing difference between - a permission fault which could arise from a page table entry that is - present in the TLB, and a translation fault which always requires a - page table walk. This option defends against these attacks by unmapping - the kernel whilst running in userspace, therefore forcing translation - faults for all of kernel space. + Speculation attacks against some high-performance processors can + be used to bypass MMU permission checks and leak kernel data to + userspace. This can be defended against by unmapping the kernel + when running in userspace, mapping it back in on exception entry + via a trampoline page in the vector table. If unsure, say Y. From patchwork Fri Jan 5 13:12:33 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Will Deacon X-Patchwork-Id: 123522 Delivered-To: patch@linaro.org Received: by 10.140.22.227 with SMTP id 90csp802055qgn; Fri, 5 Jan 2018 05:14:39 -0800 (PST) X-Google-Smtp-Source: ACJfBovCAnq/upzUrksqDQFT7fmSNKwD2cMuChpgjVgd3SrpDraLSL3BJOU2+zGMtiNmg4DTYcAa X-Received: by 10.98.49.134 with SMTP id x128mr2909727pfx.163.1515158079553; Fri, 05 Jan 2018 05:14:39 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1515158079; cv=none; d=google.com; s=arc-20160816; b=0BPmQJ76UIYpCdUE8TE/cav1yCiuA8ekQAcH4lSXPyG54wFELSxTYNrQi3WV7JRq2P fNXW+0rJhI65OJ24U3KhxC3jQeTyYs8bmWye/flI7249/bSaiKB2LOKsFhCYKwRqlVZL ifRmTGkF7TCiDyP6kI9/QWQml8znPlJ2uscE/cD1CDfSBxM4HY5mTdDvNcL+vj3y+Mj1 HGDd44gw1RvcUzOcFHmPtHpq8ITnqa+N0yMRI7UxunqDa8DTtd/hVm8QiZ2ggpbMyGQI mNfDscLEEZy7svUCxtKYQGoaRGEorWhKyCEJ/S4q1z7AVddnud6xrGEcMewgFH0l3N5S FmYA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:arc-authentication-results; bh=wncafycWI4aj1SK8RTDAAXsSzBOiMpoTGc+li5wkM84=; b=HraeoomFrA8WdTl1oqBv7Mm82sNkYQDNc36dcsP8LgOhT2lzcQVJRAkrFniWmSUGAe lTHTEFvG9SNS8u2qXY26LALQFyPJvnT09A+LRpP+BCes1J9lH9sghM/RdGu/obkfMojF DU8e1SZo0Q9DKjqefElambxK6Q/iBmSR2lx1z3JZAZh1yX4cVDl36z3FYDRpNs/alYTL /gHEGE0eCNNVq1GaHbDKqCoPZzvOIu0cMOgzuHi273CRWrWJakyHMqGVINiZVvtfiktz Z3L0guQjUXZMfJhKJXnW9GO7rMwvRHIulY9OHcgwG0UMi7sSjkYlRSgG/i19b3yXBsMP StdA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id g23si3555725pgn.58.2018.01.05.05.14.39; Fri, 05 Jan 2018 05:14:39 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752040AbeAENOZ (ORCPT + 27 others); Fri, 5 Jan 2018 08:14:25 -0500 Received: from usa-sjc-mx-foss1.foss.arm.com ([217.140.101.70]:44736 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751221AbeAENMn (ORCPT ); Fri, 5 Jan 2018 08:12:43 -0500 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id D140615BF; Fri, 5 Jan 2018 05:12:42 -0800 (PST) Received: from edgewater-inn.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 9C88C3F7BD; Fri, 5 Jan 2018 05:12:42 -0800 (PST) Received: by edgewater-inn.cambridge.arm.com (Postfix, from userid 1000) id 182801AE17F0; Fri, 5 Jan 2018 13:12:43 +0000 (GMT) From: Will Deacon To: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com, ard.biesheuvel@linaro.org, marc.zyngier@arm.com, lorenzo.pieralisi@arm.com, christoffer.dall@linaro.org, linux-kernel@vger.kernel.org, labbott@redhat.com, Will Deacon Subject: [PATCH v2 03/11] arm64: Take into account ID_AA64PFR0_EL1.CSV3 Date: Fri, 5 Jan 2018 13:12:33 +0000 Message-Id: <1515157961-20963-4-git-send-email-will.deacon@arm.com> X-Mailer: git-send-email 2.1.4 In-Reply-To: <1515157961-20963-1-git-send-email-will.deacon@arm.com> References: <1515157961-20963-1-git-send-email-will.deacon@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org For non-KASLR kernels where the KPTI behaviour has not been overridden on the command line we can use ID_AA64PFR0_EL1.CSV3 to determine whether or not we should unmap the kernel whilst running at EL0. Reviewed-by: Suzuki K Poulose Signed-off-by: Will Deacon --- arch/arm64/include/asm/sysreg.h | 1 + arch/arm64/kernel/cpufeature.c | 8 +++++++- 2 files changed, 8 insertions(+), 1 deletion(-) -- 2.1.4 diff --git a/arch/arm64/include/asm/sysreg.h b/arch/arm64/include/asm/sysreg.h index 08cc88574659..ae519bbd3f9e 100644 --- a/arch/arm64/include/asm/sysreg.h +++ b/arch/arm64/include/asm/sysreg.h @@ -437,6 +437,7 @@ #define ID_AA64ISAR1_DPB_SHIFT 0 /* id_aa64pfr0 */ +#define ID_AA64PFR0_CSV3_SHIFT 60 #define ID_AA64PFR0_SVE_SHIFT 32 #define ID_AA64PFR0_GIC_SHIFT 24 #define ID_AA64PFR0_ASIMD_SHIFT 20 diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c index 9f0545dfe497..d723fc071f39 100644 --- a/arch/arm64/kernel/cpufeature.c +++ b/arch/arm64/kernel/cpufeature.c @@ -145,6 +145,7 @@ static const struct arm64_ftr_bits ftr_id_aa64isar1[] = { }; static const struct arm64_ftr_bits ftr_id_aa64pfr0[] = { + ARM64_FTR_BITS(FTR_HIDDEN, FTR_NONSTRICT, FTR_LOWER_SAFE, ID_AA64PFR0_CSV3_SHIFT, 4, 0), ARM64_FTR_BITS(FTR_VISIBLE, FTR_STRICT, FTR_LOWER_SAFE, ID_AA64PFR0_SVE_SHIFT, 4, 0), ARM64_FTR_BITS(FTR_HIDDEN, FTR_STRICT, FTR_LOWER_SAFE, ID_AA64PFR0_GIC_SHIFT, 4, 0), S_ARM64_FTR_BITS(FTR_VISIBLE, FTR_STRICT, FTR_LOWER_SAFE, ID_AA64PFR0_ASIMD_SHIFT, 4, ID_AA64PFR0_ASIMD_NI), @@ -851,6 +852,8 @@ static int __kpti_forced; /* 0: not forced, >0: forced on, <0: forced off */ static bool unmap_kernel_at_el0(const struct arm64_cpu_capabilities *entry, int __unused) { + u64 pfr0 = read_sanitised_ftr_reg(SYS_ID_AA64PFR0_EL1); + /* Forced on command line? */ if (__kpti_forced) { pr_info_once("kernel page table isolation forced %s by command line option\n", @@ -862,7 +865,9 @@ static bool unmap_kernel_at_el0(const struct arm64_cpu_capabilities *entry, if (IS_ENABLED(CONFIG_RANDOMIZE_BASE)) return true; - return false; + /* Defer to CPU feature registers */ + return !cpuid_feature_extract_unsigned_field(pfr0, + ID_AA64PFR0_CSV3_SHIFT); } static int __init parse_kpti(char *str) @@ -967,6 +972,7 @@ static const struct arm64_cpu_capabilities arm64_features[] = { }, #ifdef CONFIG_UNMAP_KERNEL_AT_EL0 { + .desc = "Kernel page table isolation (KPTI)", .capability = ARM64_UNMAP_KERNEL_AT_EL0, .def_scope = SCOPE_SYSTEM, .matches = unmap_kernel_at_el0, From patchwork Fri Jan 5 13:12:34 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Will Deacon X-Patchwork-Id: 123521 Delivered-To: patch@linaro.org Received: by 10.140.22.227 with SMTP id 90csp801892qgn; Fri, 5 Jan 2018 05:14:28 -0800 (PST) X-Google-Smtp-Source: ACJfBoumaU+EQDPWHB5jO+RPHKGRxpYPer3cAKXH5n+QxPrTGMq1svCOXfIw3cHIkAaB8EiLgScC X-Received: by 10.98.49.134 with SMTP id x128mr2909247pfx.163.1515158068733; Fri, 05 Jan 2018 05:14:28 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1515158068; cv=none; d=google.com; s=arc-20160816; b=Xpxc4taGRMZxsXzaYYVbgfOftzjY0EdvL1oRCTa10q6vPR/PkTe0a2cC0HveVMgO32 xSdFKEYSzZ8A7mReP/Lu0+fkxRtmS2ipw4kQCXnbyJAUKOuJKKn72rZ2lUz8kin/omWz i3U/gO6MPScQNnZG6RtgpTpdRfygIF2TbLYrq+sTxL53b0FR7Tl4fbpm2EaHSMOr6cGY tVkl9VDJgqJLNxCPkbvfQsOdpw3ySA9y1ah21Iku74fM0vaSAaSxZifYxWtwHeWFExzH Xh27nbv7OBwmlZqhqL1zaRiOMEXIdnIwua7R3YGCYJO/c6EUcSpDAYX6xJM78Sr7sP7a zkBA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:arc-authentication-results; bh=Ogeaj8H+I2683zmZkIu4HLy9rjtftg+S19HDtTIQeOs=; b=G5sRcL97zARitHyf9Ft/S/F91AKWfUo9HS9gfd+SFp16e02Jn1qkxgbvLmwY1kmwpT e0AtZviKDEmrVU41DqGQaKTbaOqi/BNIwuUVVo63XbABuffRBHsabSNs9eBonTNfDbMb T7Y3am12lDpmQPlKIAVhGyIIH8rcbGzH5bUDvMLGsFgBGXtgyFJPV4n22QfcGg8kibhn ceChPQgdS6Dw29h5iWOqTp/5aez+IWeSYqFcJtIV38U2UswtGl4e2F6VTRrxYrCzWXNo 5Y2NOb3ZgkNEJPLLAZYOZ2TWRRzQOuyXC5UhD8Z2kxn+Yhx2WD9iw5B8ZsAFTqlahmyV PLqA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id 31si3988278ply.145.2018.01.05.05.14.28; Fri, 05 Jan 2018 05:14:28 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752057AbeAENO0 (ORCPT + 27 others); Fri, 5 Jan 2018 08:14:26 -0500 Received: from usa-sjc-mx-foss1.foss.arm.com ([217.140.101.70]:44740 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751278AbeAENMn (ORCPT ); Fri, 5 Jan 2018 08:12:43 -0500 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id DD1F41610; Fri, 5 Jan 2018 05:12:42 -0800 (PST) Received: from edgewater-inn.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id AB6263F53D; Fri, 5 Jan 2018 05:12:42 -0800 (PST) Received: by edgewater-inn.cambridge.arm.com (Postfix, from userid 1000) id 26A951AE180B; Fri, 5 Jan 2018 13:12:43 +0000 (GMT) From: Will Deacon To: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com, ard.biesheuvel@linaro.org, marc.zyngier@arm.com, lorenzo.pieralisi@arm.com, christoffer.dall@linaro.org, linux-kernel@vger.kernel.org, labbott@redhat.com, Will Deacon Subject: [PATCH v2 04/11] arm64: cpufeature: Pass capability structure to ->enable callback Date: Fri, 5 Jan 2018 13:12:34 +0000 Message-Id: <1515157961-20963-5-git-send-email-will.deacon@arm.com> X-Mailer: git-send-email 2.1.4 In-Reply-To: <1515157961-20963-1-git-send-email-will.deacon@arm.com> References: <1515157961-20963-1-git-send-email-will.deacon@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org In order to invoke the CPU capability ->matches callback from the ->enable callback for applying local-CPU workarounds, we need a handle on the capability structure. This patch passes a pointer to the capability structure to the ->enable callback. Reviewed-by: Suzuki K Poulose Signed-off-by: Will Deacon --- arch/arm64/kernel/cpufeature.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) -- 2.1.4 diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c index d723fc071f39..55712ab4e3bf 100644 --- a/arch/arm64/kernel/cpufeature.c +++ b/arch/arm64/kernel/cpufeature.c @@ -1152,7 +1152,7 @@ void __init enable_cpu_capabilities(const struct arm64_cpu_capabilities *caps) * uses an IPI, giving us a PSTATE that disappears when * we return. */ - stop_machine(caps->enable, NULL, cpu_online_mask); + stop_machine(caps->enable, (void *)caps, cpu_online_mask); } } } @@ -1195,7 +1195,7 @@ verify_local_cpu_features(const struct arm64_cpu_capabilities *caps) cpu_die_early(); } if (caps->enable) - caps->enable(NULL); + caps->enable((void *)caps); } } From patchwork Fri Jan 5 13:12:35 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Will Deacon X-Patchwork-Id: 123523 Delivered-To: patch@linaro.org Received: by 10.140.22.227 with SMTP id 90csp802316qgn; Fri, 5 Jan 2018 05:14:55 -0800 (PST) X-Google-Smtp-Source: ACJfBosLiwU8FF0MF/t2VU++kKqBpA6aD8xDHGVCa4ssnYXnnKm32SkRf76wGTtw6iJQQkWUJ+FJ X-Received: by 10.99.143.5 with SMTP id n5mr2673508pgd.14.1515158094969; Fri, 05 Jan 2018 05:14:54 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1515158094; cv=none; d=google.com; s=arc-20160816; b=YSSBGmwvf0i71LkuR+nGQLiW65clP34YulVbW95jm84R1DY7ZoxdWQ3Oks0+yLMai5 hQTC/7ZGzEjCaJyHjCre0jOQLMtVt4L3lq4bsBOZQuViCsQnluRHP0Z1B0AEedbn72+x vfYrO0lE8+4P2PG6hXPv6mnFzDZHfHi8ovcEM0MCdTpwhd7udddDRc422QhQom6+TrKW 3lto7khiOE5mCs8L/mv/GMmXkG2wndhFcdJBaXYatxNjmacunrHxfJj74icWFYB46h+P 16FO4J1J4/wR3HrO9K6uh1Sv3QkZGYk2ZSEjhfni93cQLZlCOQeFO5u2Xex9hXCRhmv3 eEjw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:arc-authentication-results; bh=UykPvai+4kuWW7i6WcE8r17hxZr4Xa8ijpKJ/9ZJaPA=; b=DV/vfBm9elmjvjkhV5srGNs7vBBggFfIl4EW7xoBjloKcRZVViAgVDKoEDBU4vjXoW gy/zVsTggj1gdhRrOPCApc9a+TB1vg4MC3frYcrefp+c2Sg+uO/Mi8wm8BhQmEVePH2h A5mM7c3QxDTIybctAAT1OMOAx9yIpOchA7zM4xDzf0YDcSvSt72NOzYVE0hk0e8TMdYM cPzW23wIiwC6IoVKstwEUFEmAKCnaB0lxm+4t+J8JFcBDcItIynnYV32YYuE0V7NuGDN X778oH5fWeokg5NehdQrJ+55PifwppRKfe/l+CB7w8LNM7SSZeegEvZCSi9dYbOx2BsY M++A== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id g23si3555725pgn.58.2018.01.05.05.14.54; Fri, 05 Jan 2018 05:14:54 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752023AbeAENOY (ORCPT + 27 others); Fri, 5 Jan 2018 08:14:24 -0500 Received: from usa-sjc-mx-foss1.foss.arm.com ([217.140.101.70]:44774 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751523AbeAENMn (ORCPT ); Fri, 5 Jan 2018 08:12:43 -0500 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id AAFD9164F; Fri, 5 Jan 2018 05:12:43 -0800 (PST) Received: from edgewater-inn.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 798E43F53D; Fri, 5 Jan 2018 05:12:43 -0800 (PST) Received: by edgewater-inn.cambridge.arm.com (Postfix, from userid 1000) id 34A421AE1894; Fri, 5 Jan 2018 13:12:43 +0000 (GMT) From: Will Deacon To: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com, ard.biesheuvel@linaro.org, marc.zyngier@arm.com, lorenzo.pieralisi@arm.com, christoffer.dall@linaro.org, linux-kernel@vger.kernel.org, labbott@redhat.com, Will Deacon Subject: [PATCH v2 05/11] drivers/firmware: Expose psci_get_version through psci_ops structure Date: Fri, 5 Jan 2018 13:12:35 +0000 Message-Id: <1515157961-20963-6-git-send-email-will.deacon@arm.com> X-Mailer: git-send-email 2.1.4 In-Reply-To: <1515157961-20963-1-git-send-email-will.deacon@arm.com> References: <1515157961-20963-1-git-send-email-will.deacon@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Entry into recent versions of ARM Trusted Firmware will invalidate the CPU branch predictor state in order to protect against aliasing attacks. This patch exposes the PSCI "VERSION" function via psci_ops, so that it can be invoked outside of the PSCI driver where necessary. Acked-by: Lorenzo Pieralisi Signed-off-by: Will Deacon --- drivers/firmware/psci.c | 2 ++ include/linux/psci.h | 1 + 2 files changed, 3 insertions(+) -- 2.1.4 diff --git a/drivers/firmware/psci.c b/drivers/firmware/psci.c index d687ca3d5049..8b25d31e8401 100644 --- a/drivers/firmware/psci.c +++ b/drivers/firmware/psci.c @@ -496,6 +496,8 @@ static void __init psci_init_migrate(void) static void __init psci_0_2_set_functions(void) { pr_info("Using standard PSCI v0.2 function IDs\n"); + psci_ops.get_version = psci_get_version; + psci_function_id[PSCI_FN_CPU_SUSPEND] = PSCI_FN_NATIVE(0_2, CPU_SUSPEND); psci_ops.cpu_suspend = psci_cpu_suspend; diff --git a/include/linux/psci.h b/include/linux/psci.h index bdea1cb5e1db..6306ab10af18 100644 --- a/include/linux/psci.h +++ b/include/linux/psci.h @@ -26,6 +26,7 @@ int psci_cpu_init_idle(unsigned int cpu); int psci_cpu_suspend_enter(unsigned long index); struct psci_operations { + u32 (*get_version)(void); int (*cpu_suspend)(u32 state, unsigned long entry_point); int (*cpu_off)(u32 state); int (*cpu_on)(unsigned long cpuid, unsigned long entry_point); From patchwork Fri Jan 5 13:12:36 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Will Deacon X-Patchwork-Id: 123524 Delivered-To: patch@linaro.org Received: by 10.140.22.227 with SMTP id 90csp802553qgn; Fri, 5 Jan 2018 05:15:09 -0800 (PST) X-Google-Smtp-Source: ACJfBosmqn2pclZ1vhMJ1Nl3ojzAC3uwwpscHAwzsDBzKCNUvFG1QikvM2xsLCYuPjWip0SKtlRT X-Received: by 10.159.242.198 with SMTP id x6mr3093437plw.85.1515158109252; Fri, 05 Jan 2018 05:15:09 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1515158109; cv=none; d=google.com; s=arc-20160816; b=t4hpPJQJMZ6nheTA+L87Ig6/bJrdvlxdZFXE1qKVa6eiuiHDBx0xoTTUigkdC53iyu H+1QKjcI/8aCZb1Xi/RjsMPlcJkU97WAVeURM0B1VMX0k4J2G5vvDBfDp042KBx1TH61 DHNJlnfO7zkys/fZLMei3aCc9VKG35PQN8fpwbSjiryHgw6kSXrJIaI7Zi/Zp8i3iPf5 T9b4mG1ni+UBUe0L+qTS4jQelwqH8/stkig4CTCdMh6Z6yf5ldDopbdtQVYpLprF/jBP bAGUZ8iwlh0yyJ0ohNotHBPmqYAs1RloXbpR/+FvW5/ZuNYd4WMK6o65Uaix6z+N5I4K XZaA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:arc-authentication-results; bh=XxJIX19e1/CLyfFspwtK0ieb7ju70HgiknwjiyE2oH0=; b=o4MDjMCnOEHgdQAfB+uCDBEtQSBXt2IdmoYBYHX1znAxtXqYgK7+wfZZ8SDSaAfe2L CRsCp2XsnSxl4m2PwuDIONm0UFpQznxAblHs6PYQ7E9MDCNdeRynHgTnTnKLQ/JBoh3h izxjN3SE9KpsoAZ/sZAvXoMCCOVuspP6heXwTcqibDTEJ5aiPuWBJNizww63Woqk1mja XSPqD1m6YMaQJElhYnU5iHpOBaok1HKlxo/wLthW1aepEwzxV9PVTDNFcTuGusKVwwyu 8RMR1E5pYXS5jOY3o50aMPzEy3OiwVB7I28Nrxx/GBIgGAdmxxpqQ/nxlVeUWMYvEPGo sm8Q== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id z6si4033614pfi.345.2018.01.05.05.15.08; Fri, 05 Jan 2018 05:15:09 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752008AbeAENOX (ORCPT + 27 others); Fri, 5 Jan 2018 08:14:23 -0500 Received: from usa-sjc-mx-foss1.foss.arm.com ([217.140.101.70]:44776 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751551AbeAENMn (ORCPT ); Fri, 5 Jan 2018 08:12:43 -0500 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id B6EF2165D; Fri, 5 Jan 2018 05:12:43 -0800 (PST) Received: from edgewater-inn.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 829C83F6CF; Fri, 5 Jan 2018 05:12:43 -0800 (PST) Received: by edgewater-inn.cambridge.arm.com (Postfix, from userid 1000) id 4BF511AE2D78; Fri, 5 Jan 2018 13:12:43 +0000 (GMT) From: Will Deacon To: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com, ard.biesheuvel@linaro.org, marc.zyngier@arm.com, lorenzo.pieralisi@arm.com, christoffer.dall@linaro.org, linux-kernel@vger.kernel.org, labbott@redhat.com, Will Deacon Subject: [PATCH v2 06/11] arm64: Move post_ttbr_update_workaround to C code Date: Fri, 5 Jan 2018 13:12:36 +0000 Message-Id: <1515157961-20963-7-git-send-email-will.deacon@arm.com> X-Mailer: git-send-email 2.1.4 In-Reply-To: <1515157961-20963-1-git-send-email-will.deacon@arm.com> References: <1515157961-20963-1-git-send-email-will.deacon@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org From: Marc Zyngier We will soon need to invoke a CPU-specific function pointer after changing page tables, so move post_ttbr_update_workaround out into C code to make this possible. Signed-off-by: Marc Zyngier Signed-off-by: Will Deacon --- arch/arm64/include/asm/assembler.h | 13 ------------- arch/arm64/kernel/entry.S | 2 +- arch/arm64/mm/context.c | 9 +++++++++ arch/arm64/mm/proc.S | 3 +-- 4 files changed, 11 insertions(+), 16 deletions(-) -- 2.1.4 diff --git a/arch/arm64/include/asm/assembler.h b/arch/arm64/include/asm/assembler.h index c45bc94f15d0..cee60ce0da52 100644 --- a/arch/arm64/include/asm/assembler.h +++ b/arch/arm64/include/asm/assembler.h @@ -476,17 +476,4 @@ alternative_endif mrs \rd, sp_el0 .endm -/* - * Errata workaround post TTBRx_EL1 update. - */ - .macro post_ttbr_update_workaround -#ifdef CONFIG_CAVIUM_ERRATUM_27456 -alternative_if ARM64_WORKAROUND_CAVIUM_27456 - ic iallu - dsb nsh - isb -alternative_else_nop_endif -#endif - .endm - #endif /* __ASM_ASSEMBLER_H */ diff --git a/arch/arm64/kernel/entry.S b/arch/arm64/kernel/entry.S index 71092ee09b6b..62500d371b06 100644 --- a/arch/arm64/kernel/entry.S +++ b/arch/arm64/kernel/entry.S @@ -277,7 +277,7 @@ alternative_else_nop_endif * Cavium erratum 27456 (broadcast TLBI instructions may cause I-cache * corruption). */ - post_ttbr_update_workaround + bl post_ttbr_update_workaround .endif 1: .if \el != 0 diff --git a/arch/arm64/mm/context.c b/arch/arm64/mm/context.c index 1cb3bc92ae5c..5f7097d0cd12 100644 --- a/arch/arm64/mm/context.c +++ b/arch/arm64/mm/context.c @@ -239,6 +239,15 @@ void check_and_switch_context(struct mm_struct *mm, unsigned int cpu) cpu_switch_mm(mm->pgd, mm); } +/* Errata workaround post TTBRx_EL1 update. */ +asmlinkage void post_ttbr_update_workaround(void) +{ + asm(ALTERNATIVE("nop; nop; nop", + "ic iallu; dsb nsh; isb", + ARM64_WORKAROUND_CAVIUM_27456, + CONFIG_CAVIUM_ERRATUM_27456)); +} + static int asids_init(void) { asid_bits = get_cpu_asid_bits(); diff --git a/arch/arm64/mm/proc.S b/arch/arm64/mm/proc.S index 3146dc96f05b..6affb68a9a14 100644 --- a/arch/arm64/mm/proc.S +++ b/arch/arm64/mm/proc.S @@ -145,8 +145,7 @@ ENTRY(cpu_do_switch_mm) isb msr ttbr0_el1, x0 // now update TTBR0 isb - post_ttbr_update_workaround - ret + b post_ttbr_update_workaround // Back to C code... ENDPROC(cpu_do_switch_mm) .pushsection ".idmap.text", "ax" From patchwork Fri Jan 5 13:12:37 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Will Deacon X-Patchwork-Id: 123516 Delivered-To: patch@linaro.org Received: by 10.140.22.227 with SMTP id 90csp800692qgn; Fri, 5 Jan 2018 05:13:20 -0800 (PST) X-Google-Smtp-Source: ACJfBovjBzF3w8K0fHdRMZvV72Z/Pm9IRKyDTy9k7f56gHGXxIMRPAqistZZUF6BcVlhZDVV6xQ0 X-Received: by 10.98.133.8 with SMTP id u8mr2976773pfd.26.1515158000842; Fri, 05 Jan 2018 05:13:20 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1515158000; cv=none; d=google.com; s=arc-20160816; b=vmoB+1hE1Od/fJoM9oHtuLF2aSXMS2TgpFy74xLHJCwjsK97ftwl1lDEDbEljiCmlj LB+PC4rdyBOsAGQ0+sJ2YYio2oU6gO7V4l3323u6H9dMXk3fDaSfx+044AEAIyPElp34 H/0dNewiEyepqoEfkcLH0neVJrtYNRF7kWDPYeN4imsSIA2zOm6DzG8j2J7sj1qvlIgV HKC7HcsqGmYLthZvRm27+VMQ1V3/lyeJYWOVOlq3ZY9W2+fo/aGcna386pBoMUYQ4teq ithCE/pI2kvaiWHrUDznhLWCOX49WDq2NFYjvlXsPAJoKr2pfTjeNLa2f118UTtuSHsn yJOA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:arc-authentication-results; bh=T6LMh7PHwzB3iKcPYAlMmolSJZ/GAmRuKEKtCRH4dmk=; b=bRFJyVnM9RGYIRtnQzOBd6fhwFT5AqjjKTUltappuXa/Debk0k9Cr5m344KQmS4WeV UcfL7BvHrZ0IR1C+BaKKFFdn6eLR1VciiJUU5k9vxTMsprsYzLiHv1rZkFGGmP3jvzpZ sWxe2t/qnB6aYnk4AfoF1njUdZJVFl6VeMzX1aT7cvGZaCx0yiK4/z3eZVWA9lilpW8z TaoBIyfDNRMMmTUvJ35gwyaeuc1eOQ1NZRbFK3iJlJnfMSIa1NIftono8GIChfFvt3Fp Y49ymFrKzkjBegjQjF5Gbe2/LBxm0jESTu6/xWmEnpg6IYnw9bD/hUQLigkN5FihK7VR k1ag== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id o33si3952676plb.489.2018.01.05.05.13.20; Fri, 05 Jan 2018 05:13:20 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752005AbeAENNT (ORCPT + 27 others); Fri, 5 Jan 2018 08:13:19 -0500 Received: from foss.arm.com ([217.140.101.70]:44780 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751562AbeAENMo (ORCPT ); Fri, 5 Jan 2018 08:12:44 -0500 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id BFBD01682; Fri, 5 Jan 2018 05:12:43 -0800 (PST) Received: from edgewater-inn.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 8B2053F77C; Fri, 5 Jan 2018 05:12:43 -0800 (PST) Received: by edgewater-inn.cambridge.arm.com (Postfix, from userid 1000) id 5F6871AE2DAC; Fri, 5 Jan 2018 13:12:43 +0000 (GMT) From: Will Deacon To: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com, ard.biesheuvel@linaro.org, marc.zyngier@arm.com, lorenzo.pieralisi@arm.com, christoffer.dall@linaro.org, linux-kernel@vger.kernel.org, labbott@redhat.com, Will Deacon Subject: [PATCH v2 07/11] arm64: Add skeleton to harden the branch predictor against aliasing attacks Date: Fri, 5 Jan 2018 13:12:37 +0000 Message-Id: <1515157961-20963-8-git-send-email-will.deacon@arm.com> X-Mailer: git-send-email 2.1.4 In-Reply-To: <1515157961-20963-1-git-send-email-will.deacon@arm.com> References: <1515157961-20963-1-git-send-email-will.deacon@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Aliasing attacks against CPU branch predictors can allow an attacker to redirect speculative control flow on some CPUs and potentially divulge information from one context to another. This patch adds initial skeleton code behind a new Kconfig option to enable implementation-specific mitigations against these attacks for CPUs that are affected. Signed-off-by: Marc Zyngier Signed-off-by: Will Deacon --- arch/arm64/Kconfig | 17 +++++++++ arch/arm64/include/asm/cpucaps.h | 3 +- arch/arm64/include/asm/mmu.h | 37 ++++++++++++++++++++ arch/arm64/include/asm/sysreg.h | 1 + arch/arm64/kernel/Makefile | 4 +++ arch/arm64/kernel/bpi.S | 55 +++++++++++++++++++++++++++++ arch/arm64/kernel/cpu_errata.c | 74 ++++++++++++++++++++++++++++++++++++++++ arch/arm64/kernel/cpufeature.c | 1 + arch/arm64/mm/context.c | 2 ++ arch/arm64/mm/fault.c | 1 + 10 files changed, 194 insertions(+), 1 deletion(-) create mode 100644 arch/arm64/kernel/bpi.S -- 2.1.4 diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index efaaa3a66b95..cea44b95187c 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -845,6 +845,23 @@ config UNMAP_KERNEL_AT_EL0 If unsure, say Y. +config HARDEN_BRANCH_PREDICTOR + bool "Harden the branch predictor against aliasing attacks" if EXPERT + default y + help + Speculation attacks against some high-performance processors rely on + being able to manipulate the branch predictor for a victim context by + executing aliasing branches in the attacker context. Such attacks + can be partially mitigated against by clearing internal branch + predictor state and limiting the prediction logic in some situations. + + This config option will take CPU-specific actions to harden the + branch predictor against aliasing attacks and may rely on specific + instruction sequences or control bits being set by the system + firmware. + + If unsure, say Y. + menuconfig ARMV8_DEPRECATED bool "Emulate deprecated/obsolete ARMv8 instructions" depends on COMPAT diff --git a/arch/arm64/include/asm/cpucaps.h b/arch/arm64/include/asm/cpucaps.h index b4537ffd1018..51616e77fe6b 100644 --- a/arch/arm64/include/asm/cpucaps.h +++ b/arch/arm64/include/asm/cpucaps.h @@ -42,7 +42,8 @@ #define ARM64_HAS_DCPOP 21 #define ARM64_SVE 22 #define ARM64_UNMAP_KERNEL_AT_EL0 23 +#define ARM64_HARDEN_BRANCH_PREDICTOR 24 -#define ARM64_NCAPS 24 +#define ARM64_NCAPS 25 #endif /* __ASM_CPUCAPS_H */ diff --git a/arch/arm64/include/asm/mmu.h b/arch/arm64/include/asm/mmu.h index 6f7bdb89817f..6dd83d75b82a 100644 --- a/arch/arm64/include/asm/mmu.h +++ b/arch/arm64/include/asm/mmu.h @@ -41,6 +41,43 @@ static inline bool arm64_kernel_unmapped_at_el0(void) cpus_have_const_cap(ARM64_UNMAP_KERNEL_AT_EL0); } +typedef void (*bp_hardening_cb_t)(void); + +struct bp_hardening_data { + int hyp_vectors_slot; + bp_hardening_cb_t fn; +}; + +#ifdef CONFIG_HARDEN_BRANCH_PREDICTOR +extern char __bp_harden_hyp_vecs_start[], __bp_harden_hyp_vecs_end[]; + +DECLARE_PER_CPU_READ_MOSTLY(struct bp_hardening_data, bp_hardening_data); + +static inline struct bp_hardening_data *arm64_get_bp_hardening_data(void) +{ + return this_cpu_ptr(&bp_hardening_data); +} + +static inline void arm64_apply_bp_hardening(void) +{ + struct bp_hardening_data *d; + + if (!cpus_have_const_cap(ARM64_HARDEN_BRANCH_PREDICTOR)) + return; + + d = arm64_get_bp_hardening_data(); + if (d->fn) + d->fn(); +} +#else +static inline struct bp_hardening_data *arm64_get_bp_hardening_data(void) +{ + return NULL; +} + +static inline void arm64_apply_bp_hardening(void) { } +#endif /* CONFIG_HARDEN_BRANCH_PREDICTOR */ + extern void paging_init(void); extern void bootmem_init(void); extern void __iomem *early_io_map(phys_addr_t phys, unsigned long virt); diff --git a/arch/arm64/include/asm/sysreg.h b/arch/arm64/include/asm/sysreg.h index ae519bbd3f9e..871744973ece 100644 --- a/arch/arm64/include/asm/sysreg.h +++ b/arch/arm64/include/asm/sysreg.h @@ -438,6 +438,7 @@ /* id_aa64pfr0 */ #define ID_AA64PFR0_CSV3_SHIFT 60 +#define ID_AA64PFR0_CSV2_SHIFT 56 #define ID_AA64PFR0_SVE_SHIFT 32 #define ID_AA64PFR0_GIC_SHIFT 24 #define ID_AA64PFR0_ASIMD_SHIFT 20 diff --git a/arch/arm64/kernel/Makefile b/arch/arm64/kernel/Makefile index 067baace74a0..0c760db04858 100644 --- a/arch/arm64/kernel/Makefile +++ b/arch/arm64/kernel/Makefile @@ -53,6 +53,10 @@ arm64-obj-$(CONFIG_ARM64_RELOC_TEST) += arm64-reloc-test.o arm64-reloc-test-y := reloc_test_core.o reloc_test_syms.o arm64-obj-$(CONFIG_CRASH_DUMP) += crash_dump.o +ifeq ($(CONFIG_KVM),y) +arm64-obj-$(CONFIG_HARDEN_BRANCH_PREDICTOR) += bpi.o +endif + obj-y += $(arm64-obj-y) vdso/ probes/ obj-m += $(arm64-obj-m) head-y := head.o diff --git a/arch/arm64/kernel/bpi.S b/arch/arm64/kernel/bpi.S new file mode 100644 index 000000000000..06a931eb2673 --- /dev/null +++ b/arch/arm64/kernel/bpi.S @@ -0,0 +1,55 @@ +/* + * Contains CPU specific branch predictor invalidation sequences + * + * Copyright (C) 2018 ARM Ltd. + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License version 2 as + * published by the Free Software Foundation. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program. If not, see . + */ + +#include + +.macro ventry target + .rept 31 + nop + .endr + b \target +.endm + +.macro vectors target + ventry \target + 0x000 + ventry \target + 0x080 + ventry \target + 0x100 + ventry \target + 0x180 + + ventry \target + 0x200 + ventry \target + 0x280 + ventry \target + 0x300 + ventry \target + 0x380 + + ventry \target + 0x400 + ventry \target + 0x480 + ventry \target + 0x500 + ventry \target + 0x580 + + ventry \target + 0x600 + ventry \target + 0x680 + ventry \target + 0x700 + ventry \target + 0x780 +.endm + + .align 11 +ENTRY(__bp_harden_hyp_vecs_start) + .rept 4 + vectors __kvm_hyp_vector + .endr +ENTRY(__bp_harden_hyp_vecs_end) diff --git a/arch/arm64/kernel/cpu_errata.c b/arch/arm64/kernel/cpu_errata.c index 0e27f86ee709..16ea5c6f314e 100644 --- a/arch/arm64/kernel/cpu_errata.c +++ b/arch/arm64/kernel/cpu_errata.c @@ -46,6 +46,80 @@ static int cpu_enable_trap_ctr_access(void *__unused) return 0; } +#ifdef CONFIG_HARDEN_BRANCH_PREDICTOR +#include +#include + +DEFINE_PER_CPU_READ_MOSTLY(struct bp_hardening_data, bp_hardening_data); + +#ifdef CONFIG_KVM +static void __copy_hyp_vect_bpi(int slot, const char *hyp_vecs_start, + const char *hyp_vecs_end) +{ + void *dst = lm_alias(__bp_harden_hyp_vecs_start + slot * SZ_2K); + int i; + + for (i = 0; i < SZ_2K; i += 0x80) + memcpy(dst + i, hyp_vecs_start, hyp_vecs_end - hyp_vecs_start); + + flush_icache_range((uintptr_t)dst, (uintptr_t)dst + SZ_2K); +} + +static void __install_bp_hardening_cb(bp_hardening_cb_t fn, + const char *hyp_vecs_start, + const char *hyp_vecs_end) +{ + static int last_slot = -1; + static DEFINE_SPINLOCK(bp_lock); + int cpu, slot = -1; + + spin_lock(&bp_lock); + for_each_possible_cpu(cpu) { + if (per_cpu(bp_hardening_data.fn, cpu) == fn) { + slot = per_cpu(bp_hardening_data.hyp_vectors_slot, cpu); + break; + } + } + + if (slot == -1) { + last_slot++; + BUG_ON(((__bp_harden_hyp_vecs_end - __bp_harden_hyp_vecs_start) + / SZ_2K) <= last_slot); + slot = last_slot; + __copy_hyp_vect_bpi(slot, hyp_vecs_start, hyp_vecs_end); + } + + __this_cpu_write(bp_hardening_data.hyp_vectors_slot, slot); + __this_cpu_write(bp_hardening_data.fn, fn); + spin_unlock(&bp_lock); +} +#else +static void __install_bp_hardening_cb(bp_hardening_cb_t fn, + const char *hyp_vecs_start, + const char *hyp_vecs_end) +{ + __this_cpu_write(bp_hardening_data.fn, fn); +} +#endif /* CONFIG_KVM */ + +static void install_bp_hardening_cb(const struct arm64_cpu_capabilities *entry, + bp_hardening_cb_t fn, + const char *hyp_vecs_start, + const char *hyp_vecs_end) +{ + u64 pfr0; + + if (!entry->matches(entry, SCOPE_LOCAL_CPU)) + return; + + pfr0 = read_cpuid(ID_AA64PFR0_EL1); + if (cpuid_feature_extract_unsigned_field(pfr0, ID_AA64PFR0_CSV2_SHIFT)) + return; + + __install_bp_hardening_cb(fn, hyp_vecs_start, hyp_vecs_end); +} +#endif /* CONFIG_HARDEN_BRANCH_PREDICTOR */ + #define MIDR_RANGE(model, min, max) \ .def_scope = SCOPE_LOCAL_CPU, \ .matches = is_affected_midr_range, \ diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c index 55712ab4e3bf..9d4d82c11528 100644 --- a/arch/arm64/kernel/cpufeature.c +++ b/arch/arm64/kernel/cpufeature.c @@ -146,6 +146,7 @@ static const struct arm64_ftr_bits ftr_id_aa64isar1[] = { static const struct arm64_ftr_bits ftr_id_aa64pfr0[] = { ARM64_FTR_BITS(FTR_HIDDEN, FTR_NONSTRICT, FTR_LOWER_SAFE, ID_AA64PFR0_CSV3_SHIFT, 4, 0), + ARM64_FTR_BITS(FTR_HIDDEN, FTR_NONSTRICT, FTR_LOWER_SAFE, ID_AA64PFR0_CSV2_SHIFT, 4, 0), ARM64_FTR_BITS(FTR_VISIBLE, FTR_STRICT, FTR_LOWER_SAFE, ID_AA64PFR0_SVE_SHIFT, 4, 0), ARM64_FTR_BITS(FTR_HIDDEN, FTR_STRICT, FTR_LOWER_SAFE, ID_AA64PFR0_GIC_SHIFT, 4, 0), S_ARM64_FTR_BITS(FTR_VISIBLE, FTR_STRICT, FTR_LOWER_SAFE, ID_AA64PFR0_ASIMD_SHIFT, 4, ID_AA64PFR0_ASIMD_NI), diff --git a/arch/arm64/mm/context.c b/arch/arm64/mm/context.c index 5f7097d0cd12..d99b36555a16 100644 --- a/arch/arm64/mm/context.c +++ b/arch/arm64/mm/context.c @@ -246,6 +246,8 @@ asmlinkage void post_ttbr_update_workaround(void) "ic iallu; dsb nsh; isb", ARM64_WORKAROUND_CAVIUM_27456, CONFIG_CAVIUM_ERRATUM_27456)); + + arm64_apply_bp_hardening(); } static int asids_init(void) diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c index 22168cd0dde7..5203b6040cb6 100644 --- a/arch/arm64/mm/fault.c +++ b/arch/arm64/mm/fault.c @@ -318,6 +318,7 @@ static void __do_user_fault(struct task_struct *tsk, unsigned long addr, lsb = PAGE_SHIFT; si.si_addr_lsb = lsb; + arm64_apply_bp_hardening(); force_sig_info(sig, &si, tsk); } From patchwork Fri Jan 5 13:12:38 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Will Deacon X-Patchwork-Id: 123517 Delivered-To: patch@linaro.org Received: by 10.140.22.227 with SMTP id 90csp800901qgn; Fri, 5 Jan 2018 05:13:32 -0800 (PST) X-Google-Smtp-Source: ACJfBouQ4Qg3AbXosq++QMlhsZkmffeIuuNAxRizJmFfjZ7qOyYqB5LzMizkhIxdjgPRxHQ1iraq X-Received: by 10.99.53.139 with SMTP id c133mr2596959pga.374.1515158012445; Fri, 05 Jan 2018 05:13:32 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1515158012; cv=none; d=google.com; s=arc-20160816; b=KFUJ/qb2KGNerhEgwp+0HHlBQtwNb4fGGknh39fDnvRvaW4aIGnMQfZBjdfMKZMAzN EqcpAY86lg1w/jrHDzH+C3wcNN5RtYJaa7zf+mLjr31A6qn82UI/juGi7oLNO9/ZkDmZ iZdaPIFHfGIm81A8DHD4umMV/OxNXKdiRNPvfXAsQYtDdRhR6/3oVyKo4a2Da0PEdGrC y48+i4zjb9Of0E9SMaUPmcDu/uaAT6KsBVVmsnw3usqt3VQrmGVKX3DCMvH1Q0h+cNZV k1zIG2C6KfWqF52UDFA+3Vzg8hOm213NvDEGW8Z8kn0w90k0rrifmb/+Q8A1adrPSfvQ bbpw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:arc-authentication-results; bh=915c8OiLgEOnnbwBmKtBZ72s59g0W7ngYkythx4yXgs=; b=Xi+CIEGt0wcv1XUtXLtfYkFOdiI8F9QCRwNOGZboz0O0dHYaPOWM3/rSbNpGTEJw7L Oztx5iY/uWkDvQ3ndzvWjj5Et1AUmonb4I25vtu5j3act84Q8IGMib/F/AUsRhje3V7L w77EF+80w9xxVjv05NeBZ3GjfG9DEEuV28oQnxG6MVKK8KbZQf2O+ZpQL+bE++NBQeKK ybL9hSoLWsc53/8SuWypcZXeaxr/RmY3OPTcNPSvKYqoZSWl3ipNjoEw42JUKmqWAHOU RlChzaObvOX69P4tvPihNGzfGH15jERgCGYb2FpWrNlQd5EGB2xkb+gqDL3T86KwTc/v wSwg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id w24si3923319plp.808.2018.01.05.05.13.32; Fri, 05 Jan 2018 05:13:32 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751989AbeAENNR (ORCPT + 27 others); Fri, 5 Jan 2018 08:13:17 -0500 Received: from usa-sjc-mx-foss1.foss.arm.com ([217.140.101.70]:44786 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751601AbeAENMo (ORCPT ); Fri, 5 Jan 2018 08:12:44 -0500 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id C97B31688; Fri, 5 Jan 2018 05:12:43 -0800 (PST) Received: from edgewater-inn.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 950093F7BD; Fri, 5 Jan 2018 05:12:43 -0800 (PST) Received: by edgewater-inn.cambridge.arm.com (Postfix, from userid 1000) id 6DA4F1AE2E0B; Fri, 5 Jan 2018 13:12:43 +0000 (GMT) From: Will Deacon To: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com, ard.biesheuvel@linaro.org, marc.zyngier@arm.com, lorenzo.pieralisi@arm.com, christoffer.dall@linaro.org, linux-kernel@vger.kernel.org, labbott@redhat.com, Will Deacon Subject: [PATCH v2 08/11] arm64: KVM: Use per-CPU vector when BP hardening is enabled Date: Fri, 5 Jan 2018 13:12:38 +0000 Message-Id: <1515157961-20963-9-git-send-email-will.deacon@arm.com> X-Mailer: git-send-email 2.1.4 In-Reply-To: <1515157961-20963-1-git-send-email-will.deacon@arm.com> References: <1515157961-20963-1-git-send-email-will.deacon@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org From: Marc Zyngier Now that we have per-CPU vectors, let's plug then in the KVM/arm64 code. Signed-off-by: Marc Zyngier Signed-off-by: Will Deacon --- arch/arm/include/asm/kvm_mmu.h | 10 ++++++++++ arch/arm64/include/asm/kvm_mmu.h | 38 ++++++++++++++++++++++++++++++++++++++ arch/arm64/kvm/hyp/switch.c | 2 +- virt/kvm/arm/arm.c | 8 +++++++- 4 files changed, 56 insertions(+), 2 deletions(-) -- 2.1.4 diff --git a/arch/arm/include/asm/kvm_mmu.h b/arch/arm/include/asm/kvm_mmu.h index fa6f2174276b..eb46fc81a440 100644 --- a/arch/arm/include/asm/kvm_mmu.h +++ b/arch/arm/include/asm/kvm_mmu.h @@ -221,6 +221,16 @@ static inline unsigned int kvm_get_vmid_bits(void) return 8; } +static inline void *kvm_get_hyp_vector(void) +{ + return kvm_ksym_ref(__kvm_hyp_vector); +} + +static inline int kvm_map_vectors(void) +{ + return 0; +} + #endif /* !__ASSEMBLY__ */ #endif /* __ARM_KVM_MMU_H__ */ diff --git a/arch/arm64/include/asm/kvm_mmu.h b/arch/arm64/include/asm/kvm_mmu.h index 672c8684d5c2..2d6d4bd9de52 100644 --- a/arch/arm64/include/asm/kvm_mmu.h +++ b/arch/arm64/include/asm/kvm_mmu.h @@ -309,5 +309,43 @@ static inline unsigned int kvm_get_vmid_bits(void) return (cpuid_feature_extract_unsigned_field(reg, ID_AA64MMFR1_VMIDBITS_SHIFT) == 2) ? 16 : 8; } +#ifdef CONFIG_HARDEN_BRANCH_PREDICTOR +#include + +static inline void *kvm_get_hyp_vector(void) +{ + struct bp_hardening_data *data = arm64_get_bp_hardening_data(); + void *vect = kvm_ksym_ref(__kvm_hyp_vector); + + if (data->fn) { + vect = __bp_harden_hyp_vecs_start + + data->hyp_vectors_slot * SZ_2K; + + if (!has_vhe()) + vect = lm_alias(vect); + } + + return vect; +} + +static inline int kvm_map_vectors(void) +{ + return create_hyp_mappings(kvm_ksym_ref(__bp_harden_hyp_vecs_start), + kvm_ksym_ref(__bp_harden_hyp_vecs_end), + PAGE_HYP_EXEC); +} + +#else +static inline void *kvm_get_hyp_vector(void) +{ + return kvm_ksym_ref(__kvm_hyp_vector); +} + +static inline int kvm_map_vectors(void) +{ + return 0; +} +#endif + #endif /* __ASSEMBLY__ */ #endif /* __ARM64_KVM_MMU_H__ */ diff --git a/arch/arm64/kvm/hyp/switch.c b/arch/arm64/kvm/hyp/switch.c index f7c651f3a8c0..8d4f3c9d6dc4 100644 --- a/arch/arm64/kvm/hyp/switch.c +++ b/arch/arm64/kvm/hyp/switch.c @@ -52,7 +52,7 @@ static void __hyp_text __activate_traps_vhe(void) val &= ~(CPACR_EL1_FPEN | CPACR_EL1_ZEN); write_sysreg(val, cpacr_el1); - write_sysreg(__kvm_hyp_vector, vbar_el1); + write_sysreg(kvm_get_hyp_vector(), vbar_el1); } static void __hyp_text __activate_traps_nvhe(void) diff --git a/virt/kvm/arm/arm.c b/virt/kvm/arm/arm.c index 6b60c98a6e22..1c9fdb6db124 100644 --- a/virt/kvm/arm/arm.c +++ b/virt/kvm/arm/arm.c @@ -1158,7 +1158,7 @@ static void cpu_init_hyp_mode(void *dummy) pgd_ptr = kvm_mmu_get_httbr(); stack_page = __this_cpu_read(kvm_arm_hyp_stack_page); hyp_stack_ptr = stack_page + PAGE_SIZE; - vector_ptr = (unsigned long)kvm_ksym_ref(__kvm_hyp_vector); + vector_ptr = (unsigned long)kvm_get_hyp_vector(); __cpu_init_hyp_mode(pgd_ptr, hyp_stack_ptr, vector_ptr); __cpu_init_stage2(); @@ -1403,6 +1403,12 @@ static int init_hyp_mode(void) goto out_err; } + err = kvm_map_vectors(); + if (err) { + kvm_err("Cannot map vectors\n"); + goto out_err; + } + /* * Map the Hyp stack pages */ From patchwork Fri Jan 5 13:12:39 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Will Deacon X-Patchwork-Id: 123519 Delivered-To: patch@linaro.org Received: by 10.140.22.227 with SMTP id 90csp801310qgn; Fri, 5 Jan 2018 05:13:55 -0800 (PST) X-Google-Smtp-Source: ACJfBovNfwKMh2BXs59GAQ1/IZc5C4swchGBwfIOeeRqyxphSHoGyrQwzW3HxXqfn01SqNGlQT45 X-Received: by 10.84.231.130 with SMTP id g2mr3025837plk.309.1515158035274; Fri, 05 Jan 2018 05:13:55 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1515158035; cv=none; d=google.com; s=arc-20160816; b=KFJr6G5Qk+YJ9zYwZc8M55ifcbWJXpGgziR21uqgazNfNmC3hosahekoAOjSLIjHAP SW4GRfP2U1FavKJf64uhD0XK/Sc0Y/5bcn6kswOoDaQv3h/GmnnM5NM7vMEoCogB3K5y 00AYyYlEMMxFRIqfQLbCTX/UUXuwBeICYXHUyA9ECc7fUKBYJmfGL0ZjfUvIJqHjEaGL qcFM2o2Pg6e6Fan9mvRYH84dJdt0hxe+S2tA160lUlPlFUSb/Bh1hE4hcK5Mh47Hdm1v kNtKUJQBu/uVaRYo9VKHjrM7Fh58YupjY+cJ10hF7CO2118eWhm21QIFGXTu0MPlugKe AHew== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:arc-authentication-results; bh=aWAAohPCL/d8SZkGZOFj+DcqDFMfevwgGwnkpe8hBm8=; b=JsF4gC1mWcqXmLfM6HCkJ1fa5Dl2Koy9hauuMXsfdAarrI2hieY4Z6jBHwBuA31T7d dK8aDkEkWEsmG+3Sic64rv9NCf/E7taWGpGfKu7pj5SJFCysLpH5tVtdAIgILoCMHQc/ jyTXojBJd8tkOAQebWrOeDivm6FGLSStMbSPMzWbCPlfz20JMJUvNKsICNp/3INHh2X9 9FBJ4zRB8m3gKzOPUdhaYHvcViZTkWl0yHy2tfrug4m3MfI7g8sW3l+XZJ97WMrymjEg f06NJyJJWfhQy/L8ry6kxoa9Xwkdi9BR3Kx5Sa2Sp5admwq+msD4Ff1K0IzkoUnOxVLY f2Iw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id v11si1185307pgf.326.2018.01.05.05.13.55; Fri, 05 Jan 2018 05:13:55 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751947AbeAENNO (ORCPT + 27 others); Fri, 5 Jan 2018 08:13:14 -0500 Received: from foss.arm.com ([217.140.101.70]:44792 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751658AbeAENMo (ORCPT ); Fri, 5 Jan 2018 08:12:44 -0500 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id D7C4E169F; Fri, 5 Jan 2018 05:12:43 -0800 (PST) Received: from edgewater-inn.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id A697D3F7C7; Fri, 5 Jan 2018 05:12:43 -0800 (PST) Received: by edgewater-inn.cambridge.arm.com (Postfix, from userid 1000) id 7DEE51AE2F47; Fri, 5 Jan 2018 13:12:43 +0000 (GMT) From: Will Deacon To: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com, ard.biesheuvel@linaro.org, marc.zyngier@arm.com, lorenzo.pieralisi@arm.com, christoffer.dall@linaro.org, linux-kernel@vger.kernel.org, labbott@redhat.com, Will Deacon Subject: [PATCH v2 09/11] arm64: KVM: Make PSCI_VERSION a fast path Date: Fri, 5 Jan 2018 13:12:39 +0000 Message-Id: <1515157961-20963-10-git-send-email-will.deacon@arm.com> X-Mailer: git-send-email 2.1.4 In-Reply-To: <1515157961-20963-1-git-send-email-will.deacon@arm.com> References: <1515157961-20963-1-git-send-email-will.deacon@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org From: Marc Zyngier For those CPUs that require PSCI to perform a BP invalidation, going all the way to the PSCI code for not much is a waste of precious cycles. Let's terminate that call as early as possible. Signed-off-by: Marc Zyngier Signed-off-by: Will Deacon --- arch/arm64/kvm/hyp/switch.c | 13 +++++++++++++ 1 file changed, 13 insertions(+) -- 2.1.4 diff --git a/arch/arm64/kvm/hyp/switch.c b/arch/arm64/kvm/hyp/switch.c index 8d4f3c9d6dc4..4d273f6d0e69 100644 --- a/arch/arm64/kvm/hyp/switch.c +++ b/arch/arm64/kvm/hyp/switch.c @@ -17,6 +17,7 @@ #include #include +#include #include #include @@ -341,6 +342,18 @@ int __hyp_text __kvm_vcpu_run(struct kvm_vcpu *vcpu) if (exit_code == ARM_EXCEPTION_TRAP && !__populate_fault_info(vcpu)) goto again; + if (exit_code == ARM_EXCEPTION_TRAP && + (kvm_vcpu_trap_get_class(vcpu) == ESR_ELx_EC_HVC64 || + kvm_vcpu_trap_get_class(vcpu) == ESR_ELx_EC_HVC32) && + vcpu_get_reg(vcpu, 0) == PSCI_0_2_FN_PSCI_VERSION) { + u64 val = PSCI_RET_NOT_SUPPORTED; + if (test_bit(KVM_ARM_VCPU_PSCI_0_2, vcpu->arch.features)) + val = 2; + + vcpu_set_reg(vcpu, 0, val); + goto again; + } + if (static_branch_unlikely(&vgic_v2_cpuif_trap) && exit_code == ARM_EXCEPTION_TRAP) { bool valid; From patchwork Fri Jan 5 13:12:40 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Will Deacon X-Patchwork-Id: 123518 Delivered-To: patch@linaro.org Received: by 10.140.22.227 with SMTP id 90csp801073qgn; Fri, 5 Jan 2018 05:13:43 -0800 (PST) X-Google-Smtp-Source: ACJfBovez6r65cIrbai047Wtx+aDgyhOXk5q0I7wwshWNxdsLS6+ekL1e0U8WYuRpD6NKZ7ba7qO X-Received: by 10.98.133.137 with SMTP id m9mr2903012pfk.97.1515158023318; Fri, 05 Jan 2018 05:13:43 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1515158023; cv=none; d=google.com; s=arc-20160816; b=gAuHLKBwERpZUKe923b5ObP/SsJwnHQia53ZwKhyr994giiccMz9H4uCJ/nMcPSTOZ wiuyLi4Kem0pJH94BiPq3xchyxQ4MXhHcMoVV/rF7znoeIvqKiODgRnhoRWlNp5uFqvG w0MFNgUfV+TOKMDrmbZxQHr9B4ntcBPLFnM/J3lmU2/kx7nXjp0SbgezVdT28725FBkX YWRyOJnFP9NtcW1p4hL1kQxvICZt66ctUUYGmBXd7ejsaW1Qq9q4GRIC8G50lODBTYPP xSfV7KxEuFaB7logqzyKB616pKfFBom0bE8VdjdXFd+s7gzLIk1Gw3E9E+MjKyX2PpzY uikg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:arc-authentication-results; bh=jpUYmNEx7VU+s0l/ndx3SHVIXhCGbNw/RkyqJRiZOVc=; b=GljKerlRrwDXHd1p28VLSSQQUGeAmbAs3EXtApGEE1TTcloZ+2Kk2n7hPdrbr9XRGl AwIUBt7l3hHbL7VCwPAhVjf1TawGnIm96W4hUGjM5qfxccusxDlJcQ9oLEPmGxNCOz82 gyT+TIiqxyTs1KX8MkQB5e9tTQx+3lJLl0LmfZhezUtgohgRbFQ4suipL7FFkRBDM6m/ PL3nUd0xVf+lRiqZkDGlpFRI6QBsutPKWW8UuAGAcP3BE1ElNj05RVjqHIq3bO+g6YKO Z6NAa3ONiOmj8j/bO9YRP4SnvtHTvq+YF9gCi9QpAvr2PHrD9qMa/hM6yE0RjXKoxLtC hFpg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id w24si3923319plp.808.2018.01.05.05.13.43; Fri, 05 Jan 2018 05:13:43 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751970AbeAENNQ (ORCPT + 27 others); Fri, 5 Jan 2018 08:13:16 -0500 Received: from usa-sjc-mx-foss1.foss.arm.com ([217.140.101.70]:44790 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751646AbeAENMo (ORCPT ); Fri, 5 Jan 2018 08:12:44 -0500 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id CECB5169E; Fri, 5 Jan 2018 05:12:43 -0800 (PST) Received: from edgewater-inn.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 9DB8D3F7C5; Fri, 5 Jan 2018 05:12:43 -0800 (PST) Received: by edgewater-inn.cambridge.arm.com (Postfix, from userid 1000) id 8C2B71AE2FF7; Fri, 5 Jan 2018 13:12:43 +0000 (GMT) From: Will Deacon To: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com, ard.biesheuvel@linaro.org, marc.zyngier@arm.com, lorenzo.pieralisi@arm.com, christoffer.dall@linaro.org, linux-kernel@vger.kernel.org, labbott@redhat.com, Will Deacon Subject: [PATCH v2 10/11] arm64: cputype: Add missing MIDR values for Cortex-A72 and Cortex-A75 Date: Fri, 5 Jan 2018 13:12:40 +0000 Message-Id: <1515157961-20963-11-git-send-email-will.deacon@arm.com> X-Mailer: git-send-email 2.1.4 In-Reply-To: <1515157961-20963-1-git-send-email-will.deacon@arm.com> References: <1515157961-20963-1-git-send-email-will.deacon@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Hook up MIDR values for the Cortex-A72 and Cortex-A75 CPUs, since they will soon need MIDR matches for hardening the branch predictor. Signed-off-by: Will Deacon --- arch/arm64/include/asm/cputype.h | 4 ++++ 1 file changed, 4 insertions(+) -- 2.1.4 diff --git a/arch/arm64/include/asm/cputype.h b/arch/arm64/include/asm/cputype.h index 235e77d98261..84385b94e70b 100644 --- a/arch/arm64/include/asm/cputype.h +++ b/arch/arm64/include/asm/cputype.h @@ -79,8 +79,10 @@ #define ARM_CPU_PART_AEM_V8 0xD0F #define ARM_CPU_PART_FOUNDATION 0xD00 #define ARM_CPU_PART_CORTEX_A57 0xD07 +#define ARM_CPU_PART_CORTEX_A72 0xD08 #define ARM_CPU_PART_CORTEX_A53 0xD03 #define ARM_CPU_PART_CORTEX_A73 0xD09 +#define ARM_CPU_PART_CORTEX_A75 0xD0A #define APM_CPU_PART_POTENZA 0x000 @@ -94,7 +96,9 @@ #define MIDR_CORTEX_A53 MIDR_CPU_MODEL(ARM_CPU_IMP_ARM, ARM_CPU_PART_CORTEX_A53) #define MIDR_CORTEX_A57 MIDR_CPU_MODEL(ARM_CPU_IMP_ARM, ARM_CPU_PART_CORTEX_A57) +#define MIDR_CORTEX_A72 MIDR_CPU_MODEL(ARM_CPU_IMP_ARM, ARM_CPU_PART_CORTEX_A72) #define MIDR_CORTEX_A73 MIDR_CPU_MODEL(ARM_CPU_IMP_ARM, ARM_CPU_PART_CORTEX_A73) +#define MIDR_CORTEX_A75 MIDR_CPU_MODEL(ARM_CPU_IMP_ARM, ARM_CPU_PART_CORTEX_A75) #define MIDR_THUNDERX MIDR_CPU_MODEL(ARM_CPU_IMP_CAVIUM, CAVIUM_CPU_PART_THUNDERX) #define MIDR_THUNDERX_81XX MIDR_CPU_MODEL(ARM_CPU_IMP_CAVIUM, CAVIUM_CPU_PART_THUNDERX_81XX) #define MIDR_THUNDERX_83XX MIDR_CPU_MODEL(ARM_CPU_IMP_CAVIUM, CAVIUM_CPU_PART_THUNDERX_83XX) From patchwork Fri Jan 5 13:12:41 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Will Deacon X-Patchwork-Id: 123520 Delivered-To: patch@linaro.org Received: by 10.140.22.227 with SMTP id 90csp801469qgn; Fri, 5 Jan 2018 05:14:06 -0800 (PST) X-Google-Smtp-Source: ACJfBospkuz8K4PDoWEyEj1GTCbXez5HZ9eijbLgjBUqhrXG7y4ZfM5F/nIKlsqRHILyhLzlFGcs X-Received: by 10.159.247.6 with SMTP id d6mr3152790pls.196.1515158045965; Fri, 05 Jan 2018 05:14:05 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1515158045; cv=none; d=google.com; s=arc-20160816; b=Vnkr5GQEoRf6cO6A282O703FU7MCadKn4UCGXbG9zhtVqBieY2U/pOci+HZNnqmWe4 XSen0kh8Gmna0pZ0eVpmbeWYXfawxD2azCPxEs8xGEGn6RDHrOzHCBnZB8fAI/iY7V3z 5yIKirR6+nd45dQ7TO4q60PyPsgp3vGc/+eTPAzlvJdyQcg4NeeA89ZQEyVVQ48PbaZ3 YA+U5UHDn/zVB9UkrLsg0qVpTbB9DF7Cwup1hUolj5UQfUOdRtdAozyVdyMWoVqDTAae BFltsTSRVrQsPQr7vExfzjtshl2SXTDwq55Iyej2R8qr5MstYIwILIilQ+fbvxQOXER0 ooYw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:arc-authentication-results; bh=P4s2mpSrxr4K1rxQmGfW8V77MYYIKRtuIS0e8Z2vGxE=; b=d2NpP3lGe607P9Z8D0Jr4nykLuTHcxWgd8ffyxiYLuC9XcWq+o1HByWlJv9JCm1lfI 3MEK4ZWBbdp2n8G/TpvzwREsL9iBfv9pjXaOhuV54c1YI52VMi52UdmnfhB7rMHpeLJa oQpEwpkLfipDGG/RdbN9ksSZc8WUZmSrTl1IAMDon8gOQh5ZQsldhdu7kEYzLD/RawBD aiPiuf2iB8vwbQTsUUhr5EzP8+JYMvePKq5YNg0281jIJr8Y0x0uhAFoVyY5gaRQ8Kni aupurw37PgXEYEr05c2du4qnB86Oxk75N+3jFaCeH6FMRC5a+HyVBgnN+uZzDYehwoOP 83rw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id 31si3988278ply.145.2018.01.05.05.14.05; Fri, 05 Jan 2018 05:14:05 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751791AbeAENNN (ORCPT + 27 others); Fri, 5 Jan 2018 08:13:13 -0500 Received: from usa-sjc-mx-foss1.foss.arm.com ([217.140.101.70]:44812 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751723AbeAENMo (ORCPT ); Fri, 5 Jan 2018 08:12:44 -0500 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id E34B816A3; Fri, 5 Jan 2018 05:12:43 -0800 (PST) Received: from edgewater-inn.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id AF0253F53D; Fri, 5 Jan 2018 05:12:43 -0800 (PST) Received: by edgewater-inn.cambridge.arm.com (Postfix, from userid 1000) id 9A6F41AE300C; Fri, 5 Jan 2018 13:12:43 +0000 (GMT) From: Will Deacon To: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com, ard.biesheuvel@linaro.org, marc.zyngier@arm.com, lorenzo.pieralisi@arm.com, christoffer.dall@linaro.org, linux-kernel@vger.kernel.org, labbott@redhat.com, Will Deacon Subject: [PATCH v2 11/11] arm64: Implement branch predictor hardening for affected Cortex-A CPUs Date: Fri, 5 Jan 2018 13:12:41 +0000 Message-Id: <1515157961-20963-12-git-send-email-will.deacon@arm.com> X-Mailer: git-send-email 2.1.4 In-Reply-To: <1515157961-20963-1-git-send-email-will.deacon@arm.com> References: <1515157961-20963-1-git-send-email-will.deacon@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Cortex-A57, A72, A73 and A75 are susceptible to branch predictor aliasing and can theoretically be attacked by malicious code. This patch implements a PSCI-based mitigation for these CPUs when available. The call into firmware will invalidate the branch predictor state, preventing any malicious entries from affecting other victim contexts. Signed-off-by: Marc Zyngier Signed-off-by: Will Deacon --- arch/arm64/kernel/bpi.S | 24 ++++++++++++++++++++++++ arch/arm64/kernel/cpu_errata.c | 42 ++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 66 insertions(+) -- 2.1.4 Acked-by: Will Deacon Acked-by: Will Deacon diff --git a/arch/arm64/kernel/bpi.S b/arch/arm64/kernel/bpi.S index 06a931eb2673..2e9146534174 100644 --- a/arch/arm64/kernel/bpi.S +++ b/arch/arm64/kernel/bpi.S @@ -53,3 +53,27 @@ ENTRY(__bp_harden_hyp_vecs_start) vectors __kvm_hyp_vector .endr ENTRY(__bp_harden_hyp_vecs_end) +ENTRY(__psci_hyp_bp_inval_start) + sub sp, sp, #(8 * 18) + stp x16, x17, [sp, #(16 * 0)] + stp x14, x15, [sp, #(16 * 1)] + stp x12, x13, [sp, #(16 * 2)] + stp x10, x11, [sp, #(16 * 3)] + stp x8, x9, [sp, #(16 * 4)] + stp x6, x7, [sp, #(16 * 5)] + stp x4, x5, [sp, #(16 * 6)] + stp x2, x3, [sp, #(16 * 7)] + stp x0, x1, [sp, #(18 * 8)] + mov x0, #0x84000000 + smc #0 + ldp x16, x17, [sp, #(16 * 0)] + ldp x14, x15, [sp, #(16 * 1)] + ldp x12, x13, [sp, #(16 * 2)] + ldp x10, x11, [sp, #(16 * 3)] + ldp x8, x9, [sp, #(16 * 4)] + ldp x6, x7, [sp, #(16 * 5)] + ldp x4, x5, [sp, #(16 * 6)] + ldp x2, x3, [sp, #(16 * 7)] + ldp x0, x1, [sp, #(18 * 8)] + add sp, sp, #(8 * 18) +ENTRY(__psci_hyp_bp_inval_end) diff --git a/arch/arm64/kernel/cpu_errata.c b/arch/arm64/kernel/cpu_errata.c index 16ea5c6f314e..cb0fb3796bb8 100644 --- a/arch/arm64/kernel/cpu_errata.c +++ b/arch/arm64/kernel/cpu_errata.c @@ -53,6 +53,8 @@ static int cpu_enable_trap_ctr_access(void *__unused) DEFINE_PER_CPU_READ_MOSTLY(struct bp_hardening_data, bp_hardening_data); #ifdef CONFIG_KVM +extern char __psci_hyp_bp_inval_start[], __psci_hyp_bp_inval_end[]; + static void __copy_hyp_vect_bpi(int slot, const char *hyp_vecs_start, const char *hyp_vecs_end) { @@ -94,6 +96,9 @@ static void __install_bp_hardening_cb(bp_hardening_cb_t fn, spin_unlock(&bp_lock); } #else +#define __psci_hyp_bp_inval_start NULL +#define __psci_hyp_bp_inval_end NULL + static void __install_bp_hardening_cb(bp_hardening_cb_t fn, const char *hyp_vecs_start, const char *hyp_vecs_end) @@ -118,6 +123,21 @@ static void install_bp_hardening_cb(const struct arm64_cpu_capabilities *entry, __install_bp_hardening_cb(fn, hyp_vecs_start, hyp_vecs_end); } + +#include + +static int enable_psci_bp_hardening(void *data) +{ + const struct arm64_cpu_capabilities *entry = data; + + if (psci_ops.get_version) + install_bp_hardening_cb(entry, + (bp_hardening_cb_t)psci_ops.get_version, + __psci_hyp_bp_inval_start, + __psci_hyp_bp_inval_end); + + return 0; +} #endif /* CONFIG_HARDEN_BRANCH_PREDICTOR */ #define MIDR_RANGE(model, min, max) \ @@ -261,6 +281,28 @@ const struct arm64_cpu_capabilities arm64_errata[] = { MIDR_ALL_VERSIONS(MIDR_CORTEX_A73), }, #endif +#ifdef CONFIG_HARDEN_BRANCH_PREDICTOR + { + .capability = ARM64_HARDEN_BRANCH_PREDICTOR, + MIDR_ALL_VERSIONS(MIDR_CORTEX_A57), + .enable = enable_psci_bp_hardening, + }, + { + .capability = ARM64_HARDEN_BRANCH_PREDICTOR, + MIDR_ALL_VERSIONS(MIDR_CORTEX_A72), + .enable = enable_psci_bp_hardening, + }, + { + .capability = ARM64_HARDEN_BRANCH_PREDICTOR, + MIDR_ALL_VERSIONS(MIDR_CORTEX_A73), + .enable = enable_psci_bp_hardening, + }, + { + .capability = ARM64_HARDEN_BRANCH_PREDICTOR, + MIDR_ALL_VERSIONS(MIDR_CORTEX_A75), + .enable = enable_psci_bp_hardening, + }, +#endif { } };