From patchwork Thu Jan 4 15:08:25 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Will Deacon X-Patchwork-Id: 123432 Delivered-To: patch@linaro.org Received: by 10.80.135.92 with SMTP id 28csp6819787edv; Thu, 4 Jan 2018 07:09:41 -0800 (PST) X-Google-Smtp-Source: ACJfBotUyjMQMoDbPE15iQn3v7ReG+49Rgkx/4yOyxECHwTe6s2gnXqlUH7WWqP69V8TXpAwGA3D X-Received: by 10.84.242.69 with SMTP id c5mr4880706pll.73.1515078581774; Thu, 04 Jan 2018 07:09:41 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1515078581; cv=none; d=google.com; s=arc-20160816; b=MIc2yrt7zkzfko+bG7Y8N+RptVySEM7JbsnOeEJiJ1BVPNvBZaid6d/y7cd1VDR9Gd sdt5170VMBFp/DFAKC22Ddugc+b+H0hOHsC5FzSDBPgjaCpOuChNvXBIrbjMPY0kIvyh QcG9SKBjb/EvDne25Sru2VFlqgkjpTN8uRkEgn+dtBM5BUcDeyVsOl8oRjM2g+n3vTbO DG7Ip+O6H34YvofQ4ZloXGdXQYOnW62McA+Yah1osllKzBVNkXa9o+NNaWUc5ATnlomb z6T+ODwk89hA94BbR/mcUhgoL59DLqMv/cqDlKRMaKom614hRF1JMpNkq1sAJbiIxjlx HbUw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:arc-authentication-results; bh=YOWLIab6KwQkp+2B2TNNGytIvGOlJ4grGQAbPvM/Yo0=; b=rWkibeyua1moCu6kQDogEV/N7IhqKSa/Wow7+QUh2Qu+RKwtIj5M+CP+bdjPGEjKX2 OJP4/3QVj+jghzCG3ETQWCRKO8LYl8UlAnDH6Gy6W1Zsml9onnmt2UNcWbn2e7ztufv1 aMPDH2rg6Jlu5D7zteeYJegD+uVZwOy4rg0xrUct4t1X5LvGj2glRgyOsJq7AfUABUJC Z9py3WKaQ9Ka5ezxQ8AczJVk2n3s0E4w2DR/HcMkCRs7U5MNX4krXYftTmWtSJDMTOqD ABYbfxjpzQT75KhnhvgljYn6mrQso5MOS1/VLd+8q3PRVlDkPVZownlwFpQQFEGX7O4t uNvA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id h12si2467499pln.288.2018.01.04.07.09.41; Thu, 04 Jan 2018 07:09:41 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S932129AbeADPIu (ORCPT + 22 others); Thu, 4 Jan 2018 10:08:50 -0500 Received: from foss.arm.com ([217.140.101.70]:33786 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753366AbeADPIk (ORCPT ); Thu, 4 Jan 2018 10:08:40 -0500 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 7F15415A2; Thu, 4 Jan 2018 07:08:40 -0800 (PST) Received: from edgewater-inn.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 506CD3F6CF; Thu, 4 Jan 2018 07:08:40 -0800 (PST) Received: by edgewater-inn.cambridge.arm.com (Postfix, from userid 1000) id 76E301AE0D6A; Thu, 4 Jan 2018 15:08:40 +0000 (GMT) From: Will Deacon To: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com, ard.biesheuvel@linaro.org, marc.zyngier@arm.com, lorenzo.pieralisi@arm.com, christoffer.dall@linaro.org, linux-kernel@vger.kernel.org, Will Deacon Subject: [PATCH 01/11] arm64: use RET instruction for exiting the trampoline Date: Thu, 4 Jan 2018 15:08:25 +0000 Message-Id: <1515078515-13723-2-git-send-email-will.deacon@arm.com> X-Mailer: git-send-email 2.1.4 In-Reply-To: <1515078515-13723-1-git-send-email-will.deacon@arm.com> References: <1515078515-13723-1-git-send-email-will.deacon@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Speculation attacks against the entry trampoline can potentially resteer the speculative instruction stream through the indirect branch and into arbitrary gadgets within the kernel. This patch defends against these attacks by forcing a misprediction through the return stack: a dummy BL instruction loads an entry into the stack, so that the predicted program flow of the subsequent RET instruction is to a branch-to-self instruction which is finally resolved as a branch to the kernel vectors with speculation suppressed. Signed-off-by: Will Deacon --- arch/arm64/kernel/entry.S | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) -- 2.1.4 diff --git a/arch/arm64/kernel/entry.S b/arch/arm64/kernel/entry.S index 031392ee5f47..b9feb587294d 100644 --- a/arch/arm64/kernel/entry.S +++ b/arch/arm64/kernel/entry.S @@ -1029,6 +1029,9 @@ alternative_else_nop_endif .if \regsize == 64 msr tpidrro_el0, x30 // Restored in kernel_ventry .endif + bl 2f + b . +2: tramp_map_kernel x30 #ifdef CONFIG_RANDOMIZE_BASE adr x30, tramp_vectors + PAGE_SIZE @@ -1041,7 +1044,7 @@ alternative_insn isb, nop, ARM64_WORKAROUND_QCOM_FALKOR_E1003 msr vbar_el1, x30 add x30, x30, #(1b - tramp_vectors) isb - br x30 + ret .endm .macro tramp_exit, regsize = 64 From patchwork Thu Jan 4 15:08:26 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Will Deacon X-Patchwork-Id: 123434 Delivered-To: patch@linaro.org Received: by 10.80.135.92 with SMTP id 28csp6820264edv; Thu, 4 Jan 2018 07:10:05 -0800 (PST) X-Google-Smtp-Source: ACJfBotbqlckS3gRgu11qKF9eLWCeCFzW+16FUsZJqkhKjLfQD3w65/D3bvKtR+rmIBdACTC9Y+8 X-Received: by 10.99.121.207 with SMTP id u198mr4354042pgc.32.1515078605353; Thu, 04 Jan 2018 07:10:05 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1515078605; cv=none; d=google.com; s=arc-20160816; b=IodOJmNJPCSow94cnqQHAX+A1a5V8NRW8b2qGb3b3XDa9tMBDU0I2kITKAZQljP0z+ XzsV6PdMqTQJsuLq2ag6ciJrM7PkIBd+swS9U1O/A8jdZDeYES6ShrDgRnZXMsxRw5jB 5l4HAd4cnpv/zmGyYIzaXTdCCOcJ12nh2xg+jp5YWFGFqpC+xyRvVFbdQvn0aEWY6Jf/ 93Gan+nkN/naz2e/chRbQjdnl2ZdbZQdnrOYjMw/nH6zka3E0sS+aAMdn8TaCqn4f6hp VgbWtc8NxYZaevj0iWE8/01oIjUE1IA6cnRafN+C3CBIAUXQIL8exKFqC/iac3WSxCiO tLFg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:arc-authentication-results; bh=fyr47MyjiLpFN9wd4C3cjBopnAx0Q8tHK0n2/CJh4PE=; b=yebH9oW6K0AvF4dhNy0R1xCbNPfdrip9eT9h+t+45tk/hl+uLqbj8c5DzHdwUvW2e1 MRHXIE7j9T94NQQZtmmkbFwQiacEOBng29mCy26K40xuLWj8iBQo5WAob2DqYWspLzO6 AqNrz3/vS/etEimedbXjt2Eu+U1KerrKdM8NMpC4pBPH0XtHTmgWxw4kliAChIVZ7cvP xVEh2Zd1jZ9qgCrq5cE3eqPDlx5OJ5lHrBjoQOFGjZyhAjDN8UBbFn8Sv0iB2HvqN+XI nKPAPMpQQw8YvnrfqvD8KIYCQ8IxIdwb5xe7ixvMuo70odNdzv+3cG/BOi+gFxBKoDlx b5bg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id o61si2399653pld.181.2018.01.04.07.10.05; Thu, 04 Jan 2018 07:10:05 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S932079AbeADPIr (ORCPT + 22 others); Thu, 4 Jan 2018 10:08:47 -0500 Received: from foss.arm.com ([217.140.101.70]:33790 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753368AbeADPIk (ORCPT ); Thu, 4 Jan 2018 10:08:40 -0500 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 8CB9A15AD; Thu, 4 Jan 2018 07:08:40 -0800 (PST) Received: from edgewater-inn.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 5EBDF3F77C; Thu, 4 Jan 2018 07:08:40 -0800 (PST) Received: by edgewater-inn.cambridge.arm.com (Postfix, from userid 1000) id 852641AE0E43; Thu, 4 Jan 2018 15:08:40 +0000 (GMT) From: Will Deacon To: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com, ard.biesheuvel@linaro.org, marc.zyngier@arm.com, lorenzo.pieralisi@arm.com, christoffer.dall@linaro.org, linux-kernel@vger.kernel.org, Will Deacon Subject: [PATCH 02/11] arm64: Kconfig: Reword UNMAP_KERNEL_AT_EL0 kconfig entry Date: Thu, 4 Jan 2018 15:08:26 +0000 Message-Id: <1515078515-13723-3-git-send-email-will.deacon@arm.com> X-Mailer: git-send-email 2.1.4 In-Reply-To: <1515078515-13723-1-git-send-email-will.deacon@arm.com> References: <1515078515-13723-1-git-send-email-will.deacon@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Although CONFIG_UNMAP_KERNEL_AT_EL0 does make KASLR more robust, it's actually more useful as a mitigation against speculation attacks that can leak arbitrary kernel data to userspace through speculation. Reword the Kconfig help message to reflect this, and make the option depend on EXPERT so that it is on by default for the majority of users. Signed-off-by: Will Deacon --- arch/arm64/Kconfig | 13 ++++++------- 1 file changed, 6 insertions(+), 7 deletions(-) -- 2.1.4 diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index 3af1657fcac3..efaaa3a66b95 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -834,15 +834,14 @@ config FORCE_MAX_ZONEORDER 4M allocations matching the default size used by generic code. config UNMAP_KERNEL_AT_EL0 - bool "Unmap kernel when running in userspace (aka \"KAISER\")" + bool "Unmap kernel when running in userspace (aka \"KAISER\")" if EXPERT default y help - Some attacks against KASLR make use of the timing difference between - a permission fault which could arise from a page table entry that is - present in the TLB, and a translation fault which always requires a - page table walk. This option defends against these attacks by unmapping - the kernel whilst running in userspace, therefore forcing translation - faults for all of kernel space. + Speculation attacks against some high-performance processors can + be used to bypass MMU permission checks and leak kernel data to + userspace. This can be defended against by unmapping the kernel + when running in userspace, mapping it back in on exception entry + via a trampoline page in the vector table. If unsure, say Y. From patchwork Thu Jan 4 15:08:27 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Will Deacon X-Patchwork-Id: 123433 Delivered-To: patch@linaro.org Received: by 10.80.135.92 with SMTP id 28csp6820006edv; Thu, 4 Jan 2018 07:09:53 -0800 (PST) X-Google-Smtp-Source: ACJfBosMFFS9tWtUSF8BgsXew+nz9bILjwdJtdQ81etlTJPzbVTm6z+iXWXLmorAY6ALIrr9XdVD X-Received: by 10.99.115.93 with SMTP id d29mr4375627pgn.345.1515078593335; Thu, 04 Jan 2018 07:09:53 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1515078593; cv=none; d=google.com; s=arc-20160816; b=rOExtstpLVyiy9WqjnPDQOY3FxhUanBbEpJoTUscBGeVE3S98vrLNBMva9eyB4jcvt 6vRx4VM7CRO39iGxCydVccZRBIVcbBS6scEpWZwOQSpSXGV8axv+CQI8OZ6xEZ+Q4SnY 3hgnK1fZcpt4xLXvfDkxGt+kDOyeQw8i2M2VACwNXo7cKdUDZ7Gld6fFxiSbTqmmI9ak nSuVsj/izaCNyl0gC6cxuMQbd5ceW9Z6qTNU8vla4Gqc2hHhx7v11NRp6/SCa7ACeWAu F5MRwkUfNpoZhL/mxwsCAX7aTpJ62MazSvQ40bav1qrOZcLlh4SE3NQLo3Pxu12jdvLZ eimg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:arc-authentication-results; bh=MUOsmZ4kx9+UNLQwsx2AhZiJBk7Zlkesegp7dE63NLY=; b=QN4M/8/QP1Kzd23BwGix64QOfIbfqUR7b3BF0LL0P/t3+sJlRR7G957z9vmIxobzn+ /vAt2Fbv5wTpHepfsN9Dpd84odMklN1vBOAiEy4G4uTDpRWeovEP96Fj4DAYBlgzjxru RM+B9zkOkZ/r9mGd3XKWnmf17Q9steDKUVr2H0W7cWn1oN+SJX0/+T4/c8bQfDDdI0Um yM22lQHED0FDivHdg8p/58xLeImfvGkk3/n/PdbfjA27sY5dxVF0m4uLyKj5fwXFMXK4 uhybAcnPz6aPMh/Dx7NEd8B28mZmeRIrLIAd59BJnzdIGJl9DoysuETGRVilo9EfCazr A61A== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id h12si2467499pln.288.2018.01.04.07.09.53; Thu, 04 Jan 2018 07:09:53 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S932102AbeADPIt (ORCPT + 22 others); Thu, 4 Jan 2018 10:08:49 -0500 Received: from usa-sjc-mx-foss1.foss.arm.com ([217.140.101.70]:33788 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753370AbeADPIk (ORCPT ); Thu, 4 Jan 2018 10:08:40 -0500 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 9C12E15BE; Thu, 4 Jan 2018 07:08:40 -0800 (PST) Received: from edgewater-inn.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 6E15F3F7BD; Thu, 4 Jan 2018 07:08:40 -0800 (PST) Received: by edgewater-inn.cambridge.arm.com (Postfix, from userid 1000) id 93A7B1AE0E88; Thu, 4 Jan 2018 15:08:40 +0000 (GMT) From: Will Deacon To: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com, ard.biesheuvel@linaro.org, marc.zyngier@arm.com, lorenzo.pieralisi@arm.com, christoffer.dall@linaro.org, linux-kernel@vger.kernel.org, Will Deacon Subject: [PATCH 03/11] arm64: Take into account ID_AA64PFR0_EL1.CSV3 Date: Thu, 4 Jan 2018 15:08:27 +0000 Message-Id: <1515078515-13723-4-git-send-email-will.deacon@arm.com> X-Mailer: git-send-email 2.1.4 In-Reply-To: <1515078515-13723-1-git-send-email-will.deacon@arm.com> References: <1515078515-13723-1-git-send-email-will.deacon@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org For non-KASLR kernels where the KPTI behaviour has not been overridden on the command line we can use ID_AA64PFR0_EL1.CSV3 to determine whether or not we should unmap the kernel whilst running at EL0. Signed-off-by: Will Deacon --- arch/arm64/include/asm/sysreg.h | 1 + arch/arm64/kernel/cpufeature.c | 7 ++++++- 2 files changed, 7 insertions(+), 1 deletion(-) -- 2.1.4 Reviewed-by: Suzuki K Poulose diff --git a/arch/arm64/include/asm/sysreg.h b/arch/arm64/include/asm/sysreg.h index 08cc88574659..ae519bbd3f9e 100644 --- a/arch/arm64/include/asm/sysreg.h +++ b/arch/arm64/include/asm/sysreg.h @@ -437,6 +437,7 @@ #define ID_AA64ISAR1_DPB_SHIFT 0 /* id_aa64pfr0 */ +#define ID_AA64PFR0_CSV3_SHIFT 60 #define ID_AA64PFR0_SVE_SHIFT 32 #define ID_AA64PFR0_GIC_SHIFT 24 #define ID_AA64PFR0_ASIMD_SHIFT 20 diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c index 9f0545dfe497..e11c11bb5b02 100644 --- a/arch/arm64/kernel/cpufeature.c +++ b/arch/arm64/kernel/cpufeature.c @@ -145,6 +145,7 @@ static const struct arm64_ftr_bits ftr_id_aa64isar1[] = { }; static const struct arm64_ftr_bits ftr_id_aa64pfr0[] = { + ARM64_FTR_BITS(FTR_HIDDEN, FTR_NONSTRICT, FTR_LOWER_SAFE, ID_AA64PFR0_CSV3_SHIFT, 4, 0), ARM64_FTR_BITS(FTR_VISIBLE, FTR_STRICT, FTR_LOWER_SAFE, ID_AA64PFR0_SVE_SHIFT, 4, 0), ARM64_FTR_BITS(FTR_HIDDEN, FTR_STRICT, FTR_LOWER_SAFE, ID_AA64PFR0_GIC_SHIFT, 4, 0), S_ARM64_FTR_BITS(FTR_VISIBLE, FTR_STRICT, FTR_LOWER_SAFE, ID_AA64PFR0_ASIMD_SHIFT, 4, ID_AA64PFR0_ASIMD_NI), @@ -851,6 +852,8 @@ static int __kpti_forced; /* 0: not forced, >0: forced on, <0: forced off */ static bool unmap_kernel_at_el0(const struct arm64_cpu_capabilities *entry, int __unused) { + u64 pfr0 = read_sanitised_ftr_reg(SYS_ID_AA64PFR0_EL1); + /* Forced on command line? */ if (__kpti_forced) { pr_info_once("kernel page table isolation forced %s by command line option\n", @@ -862,7 +865,9 @@ static bool unmap_kernel_at_el0(const struct arm64_cpu_capabilities *entry, if (IS_ENABLED(CONFIG_RANDOMIZE_BASE)) return true; - return false; + /* Defer to CPU feature registers */ + return !cpuid_feature_extract_unsigned_field(pfr0, + ID_AA64PFR0_CSV3_SHIFT); } static int __init parse_kpti(char *str) From patchwork Thu Jan 4 15:08:28 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Will Deacon X-Patchwork-Id: 123435 Delivered-To: patch@linaro.org Received: by 10.80.135.92 with SMTP id 28csp6820510edv; Thu, 4 Jan 2018 07:10:15 -0800 (PST) X-Google-Smtp-Source: ACJfBovEIQmeViZSHKeFoI+t5LycLjdEtCgejq4p+4GDzsxG6sduzuBws53gBk9iFB0JE/q2ole8 X-Received: by 10.98.75.68 with SMTP id y65mr5049162pfa.78.1515078615244; Thu, 04 Jan 2018 07:10:15 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1515078615; cv=none; d=google.com; s=arc-20160816; b=HAcIYy9ngY5E35ukPdoLQIjyYmgDI+fs0E1IVIXpDs/W640ExBMlb1uRa8CZUtraeM MJkX2BKLGB9eQ365IYZpzEOsTgmcp1phJkIhcrMVv0HyPDnPqPZW+faI68uavDghvo1o lGzBuI5Wzkbsmu8nWwJCvl7b39+9N/vd6m8KIQfjLJxBzchRRVu5VzBxI0UmO4FNo5f4 Q2AxljL6Gy3JsMAab6+0WO15bttOy8mWuu+7qIk42inxv+c9Ct+X01qAyZndRFWEOIOE iJulawFwn3GAOIcjK2qDI7YH4JhPAOJ8NpJdZ5lUnYRRvBPCJg12CVMlB7jfO267yCQ4 efag== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:arc-authentication-results; bh=SUcJO51daVmvnuEkXd5ASXGrt61w8W75hcSYH5squ1c=; b=tOwii+Q6KFHBDQdbIZYME0oqFG6+N/RYJ7P5zANivTZgFKQ2tTOerhKiltRLQDSv7e 1/dstl8QvUl/xs6B7pUkNd6pRj22safmitzIjPyWJriGBoG+JL8lFPFuzb/zJDIa8I1R 7HIILPYoRbytbDM4smMUr0fw1gY0EyYJlUu4VH3q5zDrIjm7yLFqwXDAEuTmSGFbY6dk NprlWWtynNlL0UmcX8E/p24owyzb4UifD556ydD77SEHXheVV2rN/Lhmm+tjDjXCQU/p OhCAnCgdKvztOklV9fQG19YrYOlGua2AbbDC1/xwSYWFwRLcZC0wfhQvlbmpnqd3aLPC 3/Fw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id o61si2399653pld.181.2018.01.04.07.10.14; Thu, 04 Jan 2018 07:10:15 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753406AbeADPIp (ORCPT + 22 others); Thu, 4 Jan 2018 10:08:45 -0500 Received: from foss.arm.com ([217.140.101.70]:33796 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753372AbeADPIk (ORCPT ); Thu, 4 Jan 2018 10:08:40 -0500 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id AC6B11610; Thu, 4 Jan 2018 07:08:40 -0800 (PST) Received: from edgewater-inn.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 7E7743F41F; Thu, 4 Jan 2018 07:08:40 -0800 (PST) Received: by edgewater-inn.cambridge.arm.com (Postfix, from userid 1000) id A3A341AE0EB6; Thu, 4 Jan 2018 15:08:40 +0000 (GMT) From: Will Deacon To: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com, ard.biesheuvel@linaro.org, marc.zyngier@arm.com, lorenzo.pieralisi@arm.com, christoffer.dall@linaro.org, linux-kernel@vger.kernel.org, Will Deacon Subject: [PATCH 04/11] arm64: cpufeature: Pass capability structure to ->enable callback Date: Thu, 4 Jan 2018 15:08:28 +0000 Message-Id: <1515078515-13723-5-git-send-email-will.deacon@arm.com> X-Mailer: git-send-email 2.1.4 In-Reply-To: <1515078515-13723-1-git-send-email-will.deacon@arm.com> References: <1515078515-13723-1-git-send-email-will.deacon@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org In order to invoke the CPU capability ->matches callback from the ->enable callback for applying local-CPU workarounds, we need a handle on the capability structure. This patch passes a pointer to the capability structure to the ->enable callback. Signed-off-by: Will Deacon --- arch/arm64/kernel/cpufeature.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) -- 2.1.4 Reviewed-by: Suzuki K Poulose diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c index e11c11bb5b02..6133c14b9b01 100644 --- a/arch/arm64/kernel/cpufeature.c +++ b/arch/arm64/kernel/cpufeature.c @@ -1151,7 +1151,7 @@ void __init enable_cpu_capabilities(const struct arm64_cpu_capabilities *caps) * uses an IPI, giving us a PSTATE that disappears when * we return. */ - stop_machine(caps->enable, NULL, cpu_online_mask); + stop_machine(caps->enable, (void *)caps, cpu_online_mask); } } } @@ -1194,7 +1194,7 @@ verify_local_cpu_features(const struct arm64_cpu_capabilities *caps) cpu_die_early(); } if (caps->enable) - caps->enable(NULL); + caps->enable((void *)caps); } } From patchwork Thu Jan 4 15:08:29 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Will Deacon X-Patchwork-Id: 123437 Delivered-To: patch@linaro.org Received: by 10.80.135.92 with SMTP id 28csp6822174edv; Thu, 4 Jan 2018 07:11:29 -0800 (PST) X-Google-Smtp-Source: ACJfBovhSyx2jUBQhyaxfiOnqjpdRvp1NbApa8UEwAV+uZz/GHWg70Bmh0apaB0rDXaiEg9S/D19 X-Received: by 10.159.242.10 with SMTP id t10mr5020276plr.227.1515078689476; Thu, 04 Jan 2018 07:11:29 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1515078689; cv=none; d=google.com; s=arc-20160816; b=Yws+i4En9McLnukjUPwFCmJtX9TwKlapIVTETZ4wAP/ROlG06yhwo90fhUM8ruoRxq MQ9AvfNEa7/hcfLD6bybyTMtj81vpR880kbGZlmKZT+/3pfSDHnQhKq0cWqO8jFE/5os 5FYIlKB36d+79qFfjFjlsyivQVZSx9Wz5bPueUceLlFkl3mwIwj7fOARLMOaBfejdgxe k9oaiq9CTfmkvlsR5LmGVL073S8yJnDcJyBjmPa06xfdID9ywUqia9C5rYDoQEQGe6ar KToPAt4SfvhgpfZEirYj9rUhCCzZuToeth5E3U16bOSimdjyCPMbpGUD+PpepEu2hU0f tn1w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:arc-authentication-results; bh=3dp83kp+XAmhZmVTTrvNkPHAmeiXMAdfLmDtrYafVdU=; b=byAb2PYG1mfEhKFzdDisCLWIMCn6R75yzYVn+YZxJlxWEY132OAyRlYjloDpGdwkOx bXcHoSqTrWR7Sotvxn6sUVUFn69fSFm/Ru5zftyi26OoJq7LebLkinFo3e4Xj8khJwOV vupuPxFFmkcNxd6PlEi/sLVRc8TQDRiWMUXphcoxhyBGhztI1pGe0PZ7IUV+WwG74TZ5 I2EbthmuCEuVK24RAxjYY4ba7XnKCIeMuQcXrP7RJn1HZ3AyJT5LAfaugZiS66z16SCL 3Vp8LtjblFOEqms6XDoLPgexpSayF7x+UcvKd+7JCgFx6QxISTO7iwiUTPFK2/ow8LUf y4XQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id g62si2148912pgc.428.2018.01.04.07.11.29; Thu, 04 Jan 2018 07:11:29 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753538AbeADPKe (ORCPT + 22 others); Thu, 4 Jan 2018 10:10:34 -0500 Received: from usa-sjc-mx-foss1.foss.arm.com ([217.140.101.70]:33808 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753374AbeADPIl (ORCPT ); Thu, 4 Jan 2018 10:08:41 -0500 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 5DACD164F; Thu, 4 Jan 2018 07:08:41 -0800 (PST) Received: from edgewater-inn.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 2FA0C3F41F; Thu, 4 Jan 2018 07:08:41 -0800 (PST) Received: by edgewater-inn.cambridge.arm.com (Postfix, from userid 1000) id B3EF31AE0EF6; Thu, 4 Jan 2018 15:08:40 +0000 (GMT) From: Will Deacon To: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com, ard.biesheuvel@linaro.org, marc.zyngier@arm.com, lorenzo.pieralisi@arm.com, christoffer.dall@linaro.org, linux-kernel@vger.kernel.org, Will Deacon Subject: [PATCH 05/11] drivers/firmware: Expose psci_get_version through psci_ops structure Date: Thu, 4 Jan 2018 15:08:29 +0000 Message-Id: <1515078515-13723-6-git-send-email-will.deacon@arm.com> X-Mailer: git-send-email 2.1.4 In-Reply-To: <1515078515-13723-1-git-send-email-will.deacon@arm.com> References: <1515078515-13723-1-git-send-email-will.deacon@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Entry into recent versions of ARM Trusted Firmware will invalidate the CPU branch predictor state in order to protect against aliasing attacks. This patch exposes the PSCI "VERSION" function via psci_ops, so that it can be invoked outside of the PSCI driver where necessary. Signed-off-by: Will Deacon --- drivers/firmware/psci.c | 2 ++ include/linux/psci.h | 1 + 2 files changed, 3 insertions(+) -- 2.1.4 Acked-by: Lorenzo Pieralisi diff --git a/drivers/firmware/psci.c b/drivers/firmware/psci.c index d687ca3d5049..8b25d31e8401 100644 --- a/drivers/firmware/psci.c +++ b/drivers/firmware/psci.c @@ -496,6 +496,8 @@ static void __init psci_init_migrate(void) static void __init psci_0_2_set_functions(void) { pr_info("Using standard PSCI v0.2 function IDs\n"); + psci_ops.get_version = psci_get_version; + psci_function_id[PSCI_FN_CPU_SUSPEND] = PSCI_FN_NATIVE(0_2, CPU_SUSPEND); psci_ops.cpu_suspend = psci_cpu_suspend; diff --git a/include/linux/psci.h b/include/linux/psci.h index bdea1cb5e1db..6306ab10af18 100644 --- a/include/linux/psci.h +++ b/include/linux/psci.h @@ -26,6 +26,7 @@ int psci_cpu_init_idle(unsigned int cpu); int psci_cpu_suspend_enter(unsigned long index); struct psci_operations { + u32 (*get_version)(void); int (*cpu_suspend)(u32 state, unsigned long entry_point); int (*cpu_off)(u32 state); int (*cpu_on)(unsigned long cpuid, unsigned long entry_point); From patchwork Thu Jan 4 15:08:30 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Will Deacon X-Patchwork-Id: 123438 Delivered-To: patch@linaro.org Received: by 10.80.135.92 with SMTP id 28csp6822436edv; Thu, 4 Jan 2018 07:11:41 -0800 (PST) X-Google-Smtp-Source: ACJfBouiXIHgTWCIdb5nqSZjOqDBKIO6va4FbUd3iylHEMvCIILsYt2vh1lGIff5G1fMaIpbDNkO X-Received: by 10.159.207.132 with SMTP id z4mr4903647plo.440.1515078701497; Thu, 04 Jan 2018 07:11:41 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1515078701; cv=none; d=google.com; s=arc-20160816; b=zKzx+wbSDwCWPu1clz4ZKkxpdlO6Tz4nwBW0Xa6i58iLwNvxMVn2Goi0e1xe/VfWG0 6uQ6PfdqwTYMohItMlh/lknkgRoj3LOSxiRDr9IU1DF+UxRSGFMNOwxS4tSqoBWkEcsL IWbjJRne5D14+XPVolyS4qVToj/CUCF7oVyNbztqGBPBpvBft/oVbEu3tLTeqPcNKm1o IX9MKuhp9VHoaK7EtTL0wfVQ+izWH+dsfYU4FE4TUBU5MaEgaw1EStieTLo08QjFPu+c Kz7nsh0+WHfOKsLTgpeC5pSepRy7LqhDvtq7tOqZ0m7mfkkchOYNr4BiVy/TqekaJQWu i7pQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:arc-authentication-results; bh=nttO8VxrACQgRB9QLPa8QR6+GM0DElU11YOsB77CT1w=; b=Qlzb7ayqK8qBverD9t7HMkHLdOYpWqPYKyeclyWvlbjRef1d0fdqs4+il0ijQ9E5lF qDhZ1fWDfgGB4b19hF9pj7+cBVHfsbmmMvwIaLGzcJr5Dg7jy3xW/2QWvYyZqNYfPReZ R4PWUoXE1ssGi0pMXRemJUUrFz+H2vBX9RZaeKXOfg+AFmRfUdjpVw+vn6jomhn5jAd5 LM/NQk1CZfKfJtlZCpIG3kCxBRLaPAQBI+A8/yJnt6Bs+y9fK9d/UTJZTKN7oF/krzMk /el5UUs+8hgd6qaStTxLkArrbiwXym36hGFxKQQa1vqEQR/3fnSQF7MsPvhYHEIFPoR8 XjAQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id g62si2148912pgc.428.2018.01.04.07.11.41; Thu, 04 Jan 2018 07:11:41 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753522AbeADPKd (ORCPT + 22 others); Thu, 4 Jan 2018 10:10:33 -0500 Received: from usa-sjc-mx-foss1.foss.arm.com ([217.140.101.70]:33810 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753375AbeADPIl (ORCPT ); Thu, 4 Jan 2018 10:08:41 -0500 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 6660F1650; Thu, 4 Jan 2018 07:08:41 -0800 (PST) Received: from edgewater-inn.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 389FC3F6CF; Thu, 4 Jan 2018 07:08:41 -0800 (PST) Received: by edgewater-inn.cambridge.arm.com (Postfix, from userid 1000) id C479E1AE0F28; Thu, 4 Jan 2018 15:08:40 +0000 (GMT) From: Will Deacon To: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com, ard.biesheuvel@linaro.org, marc.zyngier@arm.com, lorenzo.pieralisi@arm.com, christoffer.dall@linaro.org, linux-kernel@vger.kernel.org, Will Deacon Subject: [PATCH 06/11] arm64: Move post_ttbr_update_workaround to C code Date: Thu, 4 Jan 2018 15:08:30 +0000 Message-Id: <1515078515-13723-7-git-send-email-will.deacon@arm.com> X-Mailer: git-send-email 2.1.4 In-Reply-To: <1515078515-13723-1-git-send-email-will.deacon@arm.com> References: <1515078515-13723-1-git-send-email-will.deacon@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org From: Marc Zyngier We will soon need to invoke a CPU-specific function pointer after changing page tables, so move post_ttbr_update_workaround out into C code to make this possible. Signed-off-by: Marc Zyngier Signed-off-by: Will Deacon --- arch/arm64/include/asm/assembler.h | 13 ------------- arch/arm64/kernel/entry.S | 2 +- arch/arm64/mm/context.c | 9 +++++++++ arch/arm64/mm/proc.S | 3 +-- 4 files changed, 11 insertions(+), 16 deletions(-) -- 2.1.4 diff --git a/arch/arm64/include/asm/assembler.h b/arch/arm64/include/asm/assembler.h index c45bc94f15d0..cee60ce0da52 100644 --- a/arch/arm64/include/asm/assembler.h +++ b/arch/arm64/include/asm/assembler.h @@ -476,17 +476,4 @@ alternative_endif mrs \rd, sp_el0 .endm -/* - * Errata workaround post TTBRx_EL1 update. - */ - .macro post_ttbr_update_workaround -#ifdef CONFIG_CAVIUM_ERRATUM_27456 -alternative_if ARM64_WORKAROUND_CAVIUM_27456 - ic iallu - dsb nsh - isb -alternative_else_nop_endif -#endif - .endm - #endif /* __ASM_ASSEMBLER_H */ diff --git a/arch/arm64/kernel/entry.S b/arch/arm64/kernel/entry.S index b9feb587294d..6aa112baf601 100644 --- a/arch/arm64/kernel/entry.S +++ b/arch/arm64/kernel/entry.S @@ -277,7 +277,7 @@ alternative_else_nop_endif * Cavium erratum 27456 (broadcast TLBI instructions may cause I-cache * corruption). */ - post_ttbr_update_workaround + bl post_ttbr_update_workaround .endif 1: .if \el != 0 diff --git a/arch/arm64/mm/context.c b/arch/arm64/mm/context.c index 1cb3bc92ae5c..c1e3b6479c8f 100644 --- a/arch/arm64/mm/context.c +++ b/arch/arm64/mm/context.c @@ -239,6 +239,15 @@ void check_and_switch_context(struct mm_struct *mm, unsigned int cpu) cpu_switch_mm(mm->pgd, mm); } +/* Errata workaround post TTBRx_EL1 update. */ +asmlinkage void post_ttbr_update_workaround(void) +{ + asm volatile(ALTERNATIVE("nop; nop; nop", + "ic iallu; dsb nsh; isb", + ARM64_WORKAROUND_CAVIUM_27456, + CONFIG_CAVIUM_ERRATUM_27456)); +} + static int asids_init(void) { asid_bits = get_cpu_asid_bits(); diff --git a/arch/arm64/mm/proc.S b/arch/arm64/mm/proc.S index 3146dc96f05b..6affb68a9a14 100644 --- a/arch/arm64/mm/proc.S +++ b/arch/arm64/mm/proc.S @@ -145,8 +145,7 @@ ENTRY(cpu_do_switch_mm) isb msr ttbr0_el1, x0 // now update TTBR0 isb - post_ttbr_update_workaround - ret + b post_ttbr_update_workaround // Back to C code... ENDPROC(cpu_do_switch_mm) .pushsection ".idmap.text", "ax" From patchwork Thu Jan 4 15:08:31 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Will Deacon X-Patchwork-Id: 123439 Delivered-To: patch@linaro.org Received: by 10.80.135.92 with SMTP id 28csp6822950edv; Thu, 4 Jan 2018 07:12:06 -0800 (PST) X-Google-Smtp-Source: ACJfBotj0TNaF0m9dDMqRMmxVROCl86BNJ6lkgp17U0bXh9fJ/01d+NMLqGQB4IxRbV2AWuvggFY X-Received: by 10.99.106.132 with SMTP id f126mr4272474pgc.167.1515078726107; Thu, 04 Jan 2018 07:12:06 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1515078726; cv=none; d=google.com; s=arc-20160816; b=IiNI6zbS1ou0o5xln3T54gagUPsn4X60IoHCMl2p/TxJmF745E3H1ptvvbX/cjrJKu nZnt9FjCN2UG3op9LH90vI0kx54chwNoZ0giwjIUrlOKPpYK2flLseIIq+LQJEEsiAw0 JofDs8Wdz4aEJvG495t3wUjKotnk5IvHRwwNxasnqEkXuCjDaMjefOZmbAahPXOwRkD9 qi7nkmrH7SvItJipKS5/PIIaq/FIF1dnWAM44spJJZpxnrVhuwJLBrKeK76Hnj28fQPi Zvyy91SBZUzRdFINKIVl0fWH000u7N/S0vNDqT4e33oFUKvaqg7K4YsP3eqA1EJp4Twi xsCQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:arc-authentication-results; bh=q9V6RqLz3IFcF8FdhyuEu5RyPYQ9bSoMzBqBP40E8Vo=; b=n4Nqps387NkkHJDhTNz2UtzMVaHJevzNtgq2o7HNudXyt0uj7jivSP4BmIcs7mPSLZ 8oEePpRinqF1UnyemwKtho0DP8YhcWcaBSFR5SC7su2iHVHbXhJqJS5DVx+s0lrv+A3Q 8fnJYfFrXRxuvHrOkK+Sj3NL+Roqgh/TDIRtyr5nj6k4e01+vGlCxH/G/12yqYkBfuTW WdWwcteS2Ulv2lxr0FvWbQNBw2CX0ebCmFY6xnzFWRWdmCGUaO6c0fzlE+1Dkh6CwOd1 4G6LCg4PanUgoSIPrj8jKjO7jVuI80jRCjl78p8rLJl+uMwsXWopXBCJsctervuyh9kN UmjQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id g62si2148912pgc.428.2018.01.04.07.12.05; Thu, 04 Jan 2018 07:12:06 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753491AbeADPKa (ORCPT + 22 others); Thu, 4 Jan 2018 10:10:30 -0500 Received: from foss.arm.com ([217.140.101.70]:33816 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753376AbeADPIl (ORCPT ); Thu, 4 Jan 2018 10:08:41 -0500 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 6FFBD165C; Thu, 4 Jan 2018 07:08:41 -0800 (PST) Received: from edgewater-inn.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 4184E3F77C; Thu, 4 Jan 2018 07:08:41 -0800 (PST) Received: by edgewater-inn.cambridge.arm.com (Postfix, from userid 1000) id D4CBA1AE0F35; Thu, 4 Jan 2018 15:08:40 +0000 (GMT) From: Will Deacon To: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com, ard.biesheuvel@linaro.org, marc.zyngier@arm.com, lorenzo.pieralisi@arm.com, christoffer.dall@linaro.org, linux-kernel@vger.kernel.org, Will Deacon Subject: [PATCH 07/11] arm64: Add skeleton to harden the branch predictor against aliasing attacks Date: Thu, 4 Jan 2018 15:08:31 +0000 Message-Id: <1515078515-13723-8-git-send-email-will.deacon@arm.com> X-Mailer: git-send-email 2.1.4 In-Reply-To: <1515078515-13723-1-git-send-email-will.deacon@arm.com> References: <1515078515-13723-1-git-send-email-will.deacon@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Aliasing attacks against CPU branch predictors can allow an attacker to redirect speculative control flow on some CPUs and potentially divulge information from one context to another. This patch adds initial skeleton code behind a new Kconfig option to enable implementation-specific mitigations against these attacks for CPUs that are affected. Signed-off-by: Marc Zyngier Signed-off-by: Will Deacon --- arch/arm64/Kconfig | 17 +++++++++ arch/arm64/include/asm/cpucaps.h | 3 +- arch/arm64/include/asm/mmu.h | 37 ++++++++++++++++++++ arch/arm64/include/asm/sysreg.h | 1 + arch/arm64/kernel/Makefile | 4 +++ arch/arm64/kernel/bpi.S | 55 +++++++++++++++++++++++++++++ arch/arm64/kernel/cpu_errata.c | 74 ++++++++++++++++++++++++++++++++++++++++ arch/arm64/kernel/cpufeature.c | 1 + arch/arm64/mm/context.c | 2 ++ arch/arm64/mm/fault.c | 1 + 10 files changed, 194 insertions(+), 1 deletion(-) create mode 100644 arch/arm64/kernel/bpi.S -- 2.1.4 diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index efaaa3a66b95..cea44b95187c 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -845,6 +845,23 @@ config UNMAP_KERNEL_AT_EL0 If unsure, say Y. +config HARDEN_BRANCH_PREDICTOR + bool "Harden the branch predictor against aliasing attacks" if EXPERT + default y + help + Speculation attacks against some high-performance processors rely on + being able to manipulate the branch predictor for a victim context by + executing aliasing branches in the attacker context. Such attacks + can be partially mitigated against by clearing internal branch + predictor state and limiting the prediction logic in some situations. + + This config option will take CPU-specific actions to harden the + branch predictor against aliasing attacks and may rely on specific + instruction sequences or control bits being set by the system + firmware. + + If unsure, say Y. + menuconfig ARMV8_DEPRECATED bool "Emulate deprecated/obsolete ARMv8 instructions" depends on COMPAT diff --git a/arch/arm64/include/asm/cpucaps.h b/arch/arm64/include/asm/cpucaps.h index b4537ffd1018..51616e77fe6b 100644 --- a/arch/arm64/include/asm/cpucaps.h +++ b/arch/arm64/include/asm/cpucaps.h @@ -42,7 +42,8 @@ #define ARM64_HAS_DCPOP 21 #define ARM64_SVE 22 #define ARM64_UNMAP_KERNEL_AT_EL0 23 +#define ARM64_HARDEN_BRANCH_PREDICTOR 24 -#define ARM64_NCAPS 24 +#define ARM64_NCAPS 25 #endif /* __ASM_CPUCAPS_H */ diff --git a/arch/arm64/include/asm/mmu.h b/arch/arm64/include/asm/mmu.h index 6f7bdb89817f..6dd83d75b82a 100644 --- a/arch/arm64/include/asm/mmu.h +++ b/arch/arm64/include/asm/mmu.h @@ -41,6 +41,43 @@ static inline bool arm64_kernel_unmapped_at_el0(void) cpus_have_const_cap(ARM64_UNMAP_KERNEL_AT_EL0); } +typedef void (*bp_hardening_cb_t)(void); + +struct bp_hardening_data { + int hyp_vectors_slot; + bp_hardening_cb_t fn; +}; + +#ifdef CONFIG_HARDEN_BRANCH_PREDICTOR +extern char __bp_harden_hyp_vecs_start[], __bp_harden_hyp_vecs_end[]; + +DECLARE_PER_CPU_READ_MOSTLY(struct bp_hardening_data, bp_hardening_data); + +static inline struct bp_hardening_data *arm64_get_bp_hardening_data(void) +{ + return this_cpu_ptr(&bp_hardening_data); +} + +static inline void arm64_apply_bp_hardening(void) +{ + struct bp_hardening_data *d; + + if (!cpus_have_const_cap(ARM64_HARDEN_BRANCH_PREDICTOR)) + return; + + d = arm64_get_bp_hardening_data(); + if (d->fn) + d->fn(); +} +#else +static inline struct bp_hardening_data *arm64_get_bp_hardening_data(void) +{ + return NULL; +} + +static inline void arm64_apply_bp_hardening(void) { } +#endif /* CONFIG_HARDEN_BRANCH_PREDICTOR */ + extern void paging_init(void); extern void bootmem_init(void); extern void __iomem *early_io_map(phys_addr_t phys, unsigned long virt); diff --git a/arch/arm64/include/asm/sysreg.h b/arch/arm64/include/asm/sysreg.h index ae519bbd3f9e..871744973ece 100644 --- a/arch/arm64/include/asm/sysreg.h +++ b/arch/arm64/include/asm/sysreg.h @@ -438,6 +438,7 @@ /* id_aa64pfr0 */ #define ID_AA64PFR0_CSV3_SHIFT 60 +#define ID_AA64PFR0_CSV2_SHIFT 56 #define ID_AA64PFR0_SVE_SHIFT 32 #define ID_AA64PFR0_GIC_SHIFT 24 #define ID_AA64PFR0_ASIMD_SHIFT 20 diff --git a/arch/arm64/kernel/Makefile b/arch/arm64/kernel/Makefile index 067baace74a0..0c760db04858 100644 --- a/arch/arm64/kernel/Makefile +++ b/arch/arm64/kernel/Makefile @@ -53,6 +53,10 @@ arm64-obj-$(CONFIG_ARM64_RELOC_TEST) += arm64-reloc-test.o arm64-reloc-test-y := reloc_test_core.o reloc_test_syms.o arm64-obj-$(CONFIG_CRASH_DUMP) += crash_dump.o +ifeq ($(CONFIG_KVM),y) +arm64-obj-$(CONFIG_HARDEN_BRANCH_PREDICTOR) += bpi.o +endif + obj-y += $(arm64-obj-y) vdso/ probes/ obj-m += $(arm64-obj-m) head-y := head.o diff --git a/arch/arm64/kernel/bpi.S b/arch/arm64/kernel/bpi.S new file mode 100644 index 000000000000..06a931eb2673 --- /dev/null +++ b/arch/arm64/kernel/bpi.S @@ -0,0 +1,55 @@ +/* + * Contains CPU specific branch predictor invalidation sequences + * + * Copyright (C) 2018 ARM Ltd. + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License version 2 as + * published by the Free Software Foundation. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program. If not, see . + */ + +#include + +.macro ventry target + .rept 31 + nop + .endr + b \target +.endm + +.macro vectors target + ventry \target + 0x000 + ventry \target + 0x080 + ventry \target + 0x100 + ventry \target + 0x180 + + ventry \target + 0x200 + ventry \target + 0x280 + ventry \target + 0x300 + ventry \target + 0x380 + + ventry \target + 0x400 + ventry \target + 0x480 + ventry \target + 0x500 + ventry \target + 0x580 + + ventry \target + 0x600 + ventry \target + 0x680 + ventry \target + 0x700 + ventry \target + 0x780 +.endm + + .align 11 +ENTRY(__bp_harden_hyp_vecs_start) + .rept 4 + vectors __kvm_hyp_vector + .endr +ENTRY(__bp_harden_hyp_vecs_end) diff --git a/arch/arm64/kernel/cpu_errata.c b/arch/arm64/kernel/cpu_errata.c index 0e27f86ee709..16ea5c6f314e 100644 --- a/arch/arm64/kernel/cpu_errata.c +++ b/arch/arm64/kernel/cpu_errata.c @@ -46,6 +46,80 @@ static int cpu_enable_trap_ctr_access(void *__unused) return 0; } +#ifdef CONFIG_HARDEN_BRANCH_PREDICTOR +#include +#include + +DEFINE_PER_CPU_READ_MOSTLY(struct bp_hardening_data, bp_hardening_data); + +#ifdef CONFIG_KVM +static void __copy_hyp_vect_bpi(int slot, const char *hyp_vecs_start, + const char *hyp_vecs_end) +{ + void *dst = lm_alias(__bp_harden_hyp_vecs_start + slot * SZ_2K); + int i; + + for (i = 0; i < SZ_2K; i += 0x80) + memcpy(dst + i, hyp_vecs_start, hyp_vecs_end - hyp_vecs_start); + + flush_icache_range((uintptr_t)dst, (uintptr_t)dst + SZ_2K); +} + +static void __install_bp_hardening_cb(bp_hardening_cb_t fn, + const char *hyp_vecs_start, + const char *hyp_vecs_end) +{ + static int last_slot = -1; + static DEFINE_SPINLOCK(bp_lock); + int cpu, slot = -1; + + spin_lock(&bp_lock); + for_each_possible_cpu(cpu) { + if (per_cpu(bp_hardening_data.fn, cpu) == fn) { + slot = per_cpu(bp_hardening_data.hyp_vectors_slot, cpu); + break; + } + } + + if (slot == -1) { + last_slot++; + BUG_ON(((__bp_harden_hyp_vecs_end - __bp_harden_hyp_vecs_start) + / SZ_2K) <= last_slot); + slot = last_slot; + __copy_hyp_vect_bpi(slot, hyp_vecs_start, hyp_vecs_end); + } + + __this_cpu_write(bp_hardening_data.hyp_vectors_slot, slot); + __this_cpu_write(bp_hardening_data.fn, fn); + spin_unlock(&bp_lock); +} +#else +static void __install_bp_hardening_cb(bp_hardening_cb_t fn, + const char *hyp_vecs_start, + const char *hyp_vecs_end) +{ + __this_cpu_write(bp_hardening_data.fn, fn); +} +#endif /* CONFIG_KVM */ + +static void install_bp_hardening_cb(const struct arm64_cpu_capabilities *entry, + bp_hardening_cb_t fn, + const char *hyp_vecs_start, + const char *hyp_vecs_end) +{ + u64 pfr0; + + if (!entry->matches(entry, SCOPE_LOCAL_CPU)) + return; + + pfr0 = read_cpuid(ID_AA64PFR0_EL1); + if (cpuid_feature_extract_unsigned_field(pfr0, ID_AA64PFR0_CSV2_SHIFT)) + return; + + __install_bp_hardening_cb(fn, hyp_vecs_start, hyp_vecs_end); +} +#endif /* CONFIG_HARDEN_BRANCH_PREDICTOR */ + #define MIDR_RANGE(model, min, max) \ .def_scope = SCOPE_LOCAL_CPU, \ .matches = is_affected_midr_range, \ diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c index 6133c14b9b01..19ed09b0bb24 100644 --- a/arch/arm64/kernel/cpufeature.c +++ b/arch/arm64/kernel/cpufeature.c @@ -146,6 +146,7 @@ static const struct arm64_ftr_bits ftr_id_aa64isar1[] = { static const struct arm64_ftr_bits ftr_id_aa64pfr0[] = { ARM64_FTR_BITS(FTR_HIDDEN, FTR_NONSTRICT, FTR_LOWER_SAFE, ID_AA64PFR0_CSV3_SHIFT, 4, 0), + ARM64_FTR_BITS(FTR_HIDDEN, FTR_NONSTRICT, FTR_LOWER_SAFE, ID_AA64PFR0_CSV2_SHIFT, 4, 0), ARM64_FTR_BITS(FTR_VISIBLE, FTR_STRICT, FTR_LOWER_SAFE, ID_AA64PFR0_SVE_SHIFT, 4, 0), ARM64_FTR_BITS(FTR_HIDDEN, FTR_STRICT, FTR_LOWER_SAFE, ID_AA64PFR0_GIC_SHIFT, 4, 0), S_ARM64_FTR_BITS(FTR_VISIBLE, FTR_STRICT, FTR_LOWER_SAFE, ID_AA64PFR0_ASIMD_SHIFT, 4, ID_AA64PFR0_ASIMD_NI), diff --git a/arch/arm64/mm/context.c b/arch/arm64/mm/context.c index c1e3b6479c8f..f1d99ffc77d1 100644 --- a/arch/arm64/mm/context.c +++ b/arch/arm64/mm/context.c @@ -246,6 +246,8 @@ asmlinkage void post_ttbr_update_workaround(void) "ic iallu; dsb nsh; isb", ARM64_WORKAROUND_CAVIUM_27456, CONFIG_CAVIUM_ERRATUM_27456)); + + arm64_apply_bp_hardening(); } static int asids_init(void) diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c index 22168cd0dde7..5203b6040cb6 100644 --- a/arch/arm64/mm/fault.c +++ b/arch/arm64/mm/fault.c @@ -318,6 +318,7 @@ static void __do_user_fault(struct task_struct *tsk, unsigned long addr, lsb = PAGE_SHIFT; si.si_addr_lsb = lsb; + arm64_apply_bp_hardening(); force_sig_info(sig, &si, tsk); } From patchwork Thu Jan 4 15:08:32 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Will Deacon X-Patchwork-Id: 123436 Delivered-To: patch@linaro.org Received: by 10.80.135.92 with SMTP id 28csp6821116edv; Thu, 4 Jan 2018 07:10:36 -0800 (PST) X-Google-Smtp-Source: ACJfBoupIzFbT6dLGmT0goBr3zXv7TP9SG3Lvmm/q6TUo4vAKQvbfGMqcSUioUxht6aZuae6T9ls X-Received: by 10.84.224.206 with SMTP id k14mr4967429pln.403.1515078636087; Thu, 04 Jan 2018 07:10:36 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1515078636; cv=none; d=google.com; s=arc-20160816; b=I9SciQw0A4Q3Rk+eQyx6TbgrleFTs37a4F+EtRDWQC/Px+5P3MKs9QqNyv2ttIQ3mw 6XOS/LvBxdvQ5UNcxxfHcVhKmkuxe3v6hKAWeKEHB7mtohLs1uWKZDyJ4IISchTqXr7J 2+QYZTjcXJXQK7/+Hnz2W3rUTAEKwb2qA9pQOTnHcdWJN+UYOgefSgN+WGsicub0+8kq hYzOp4cX9AJNXspHI9/ddQhOoSTzeaqipFkzoIWe0QyoA/14/mIxOAHauEJFb5Y1iGs4 yCZoWKly8gNfLFeMK731j3DqFwZZVu3dqdOkHbpK+WIV5cVrA1PaXF+ELK/iXP7AyAPm /i+g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:arc-authentication-results; bh=915c8OiLgEOnnbwBmKtBZ72s59g0W7ngYkythx4yXgs=; b=eC7SZa+5WPip5Yu8FS7pZGhskoBjtsIZRlgSuhMGp3tEWj6tj8cmlBtpOpnu/3TdLU 5Bg9fvWuqb69VkUe7fEz+cauvzVbRR8sBOINf7T/nIWnHrfX0THlfwNDH8JcgFarE5jy QcWcltXOZNkv4eECKFX+krsjtohxUyZ3zxDhrlCbAOBicNLssh9dEvAYfQN6ODMHevNk mYg/EfKQ+ZJkH9S2XCztAwDf6XjX2thSWzhLHielQGj6/WW4+DSFOU7HOcydyAr78wUr hOQr3PgMz/Dh7Ln4W4PwYeAAmX2kC+KpcCkMlgulMUyo9bTi7GDw8AXPr7amRr8qtrNj 1dDQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id r9si2140612pgp.393.2018.01.04.07.10.35; Thu, 04 Jan 2018 07:10:36 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753507AbeADPKb (ORCPT + 22 others); Thu, 4 Jan 2018 10:10:31 -0500 Received: from foss.arm.com ([217.140.101.70]:33818 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753377AbeADPIl (ORCPT ); Thu, 4 Jan 2018 10:08:41 -0500 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 78D671688; Thu, 4 Jan 2018 07:08:41 -0800 (PST) Received: from edgewater-inn.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 4B1103F7BD; Thu, 4 Jan 2018 07:08:41 -0800 (PST) Received: by edgewater-inn.cambridge.arm.com (Postfix, from userid 1000) id E0C731AE1018; Thu, 4 Jan 2018 15:08:40 +0000 (GMT) From: Will Deacon To: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com, ard.biesheuvel@linaro.org, marc.zyngier@arm.com, lorenzo.pieralisi@arm.com, christoffer.dall@linaro.org, linux-kernel@vger.kernel.org, Will Deacon Subject: [PATCH 08/11] arm64: KVM: Use per-CPU vector when BP hardening is enabled Date: Thu, 4 Jan 2018 15:08:32 +0000 Message-Id: <1515078515-13723-9-git-send-email-will.deacon@arm.com> X-Mailer: git-send-email 2.1.4 In-Reply-To: <1515078515-13723-1-git-send-email-will.deacon@arm.com> References: <1515078515-13723-1-git-send-email-will.deacon@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org From: Marc Zyngier Now that we have per-CPU vectors, let's plug then in the KVM/arm64 code. Signed-off-by: Marc Zyngier Signed-off-by: Will Deacon --- arch/arm/include/asm/kvm_mmu.h | 10 ++++++++++ arch/arm64/include/asm/kvm_mmu.h | 38 ++++++++++++++++++++++++++++++++++++++ arch/arm64/kvm/hyp/switch.c | 2 +- virt/kvm/arm/arm.c | 8 +++++++- 4 files changed, 56 insertions(+), 2 deletions(-) -- 2.1.4 diff --git a/arch/arm/include/asm/kvm_mmu.h b/arch/arm/include/asm/kvm_mmu.h index fa6f2174276b..eb46fc81a440 100644 --- a/arch/arm/include/asm/kvm_mmu.h +++ b/arch/arm/include/asm/kvm_mmu.h @@ -221,6 +221,16 @@ static inline unsigned int kvm_get_vmid_bits(void) return 8; } +static inline void *kvm_get_hyp_vector(void) +{ + return kvm_ksym_ref(__kvm_hyp_vector); +} + +static inline int kvm_map_vectors(void) +{ + return 0; +} + #endif /* !__ASSEMBLY__ */ #endif /* __ARM_KVM_MMU_H__ */ diff --git a/arch/arm64/include/asm/kvm_mmu.h b/arch/arm64/include/asm/kvm_mmu.h index 672c8684d5c2..2d6d4bd9de52 100644 --- a/arch/arm64/include/asm/kvm_mmu.h +++ b/arch/arm64/include/asm/kvm_mmu.h @@ -309,5 +309,43 @@ static inline unsigned int kvm_get_vmid_bits(void) return (cpuid_feature_extract_unsigned_field(reg, ID_AA64MMFR1_VMIDBITS_SHIFT) == 2) ? 16 : 8; } +#ifdef CONFIG_HARDEN_BRANCH_PREDICTOR +#include + +static inline void *kvm_get_hyp_vector(void) +{ + struct bp_hardening_data *data = arm64_get_bp_hardening_data(); + void *vect = kvm_ksym_ref(__kvm_hyp_vector); + + if (data->fn) { + vect = __bp_harden_hyp_vecs_start + + data->hyp_vectors_slot * SZ_2K; + + if (!has_vhe()) + vect = lm_alias(vect); + } + + return vect; +} + +static inline int kvm_map_vectors(void) +{ + return create_hyp_mappings(kvm_ksym_ref(__bp_harden_hyp_vecs_start), + kvm_ksym_ref(__bp_harden_hyp_vecs_end), + PAGE_HYP_EXEC); +} + +#else +static inline void *kvm_get_hyp_vector(void) +{ + return kvm_ksym_ref(__kvm_hyp_vector); +} + +static inline int kvm_map_vectors(void) +{ + return 0; +} +#endif + #endif /* __ASSEMBLY__ */ #endif /* __ARM64_KVM_MMU_H__ */ diff --git a/arch/arm64/kvm/hyp/switch.c b/arch/arm64/kvm/hyp/switch.c index f7c651f3a8c0..8d4f3c9d6dc4 100644 --- a/arch/arm64/kvm/hyp/switch.c +++ b/arch/arm64/kvm/hyp/switch.c @@ -52,7 +52,7 @@ static void __hyp_text __activate_traps_vhe(void) val &= ~(CPACR_EL1_FPEN | CPACR_EL1_ZEN); write_sysreg(val, cpacr_el1); - write_sysreg(__kvm_hyp_vector, vbar_el1); + write_sysreg(kvm_get_hyp_vector(), vbar_el1); } static void __hyp_text __activate_traps_nvhe(void) diff --git a/virt/kvm/arm/arm.c b/virt/kvm/arm/arm.c index 6b60c98a6e22..1c9fdb6db124 100644 --- a/virt/kvm/arm/arm.c +++ b/virt/kvm/arm/arm.c @@ -1158,7 +1158,7 @@ static void cpu_init_hyp_mode(void *dummy) pgd_ptr = kvm_mmu_get_httbr(); stack_page = __this_cpu_read(kvm_arm_hyp_stack_page); hyp_stack_ptr = stack_page + PAGE_SIZE; - vector_ptr = (unsigned long)kvm_ksym_ref(__kvm_hyp_vector); + vector_ptr = (unsigned long)kvm_get_hyp_vector(); __cpu_init_hyp_mode(pgd_ptr, hyp_stack_ptr, vector_ptr); __cpu_init_stage2(); @@ -1403,6 +1403,12 @@ static int init_hyp_mode(void) goto out_err; } + err = kvm_map_vectors(); + if (err) { + kvm_err("Cannot map vectors\n"); + goto out_err; + } + /* * Map the Hyp stack pages */ From patchwork Thu Jan 4 15:08:33 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Will Deacon X-Patchwork-Id: 123441 Delivered-To: patch@linaro.org Received: by 10.80.135.92 with SMTP id 28csp6823539edv; Thu, 4 Jan 2018 07:12:31 -0800 (PST) X-Google-Smtp-Source: ACJfBotP1a/qddA+c9H42djhihtw4YUhiv5nDOO6dBTg2EqLiDByKUleLcmxIWc+XV6jzRZaPrTc X-Received: by 10.98.171.12 with SMTP id p12mr5072022pff.201.1515078751643; Thu, 04 Jan 2018 07:12:31 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1515078751; cv=none; d=google.com; s=arc-20160816; b=p5zk7UKbt2f38FYxnwHyv7bnuI2cFfHru7NYfXcjSuIGmwvzV0xrFuuAwEmAfh/Oww u5KeVPjTMoQEcVNP/nEPFWDCxbbE4JMVqImKkNE2eJGHyaeGnn+X9RFnsZeUAkfzihpT Sg+Xw33e31VNXyb4R4hbfN7akSyAALfY5uM/rAYJN8FEBhjoXP0a3+QWT8ro0OKT6Vta CRhuvRqNcfD+eApzsema/yXi5iz8J6i/7+wH+qxSOsTGAf49O6m15aKQBeR0ha5NpyaM tfnG9GdVHSvmx+iAk6PR6crYhHyI7A2VgLKgh/+QsoPhLstBjEhSgeD5NATcEgpXe6vc A40Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:arc-authentication-results; bh=aWAAohPCL/d8SZkGZOFj+DcqDFMfevwgGwnkpe8hBm8=; b=up/KcAlbgddJzDz+B/YNnyOWoB93SdXZzoBzKyiugF8ZdefXEYv/0ujuY+gXK9GMUa jm1ua8hzix/yPlQNIPDt1sCVyIxqVrcUJgKE3t+Lnmm/DJfL2fSdFK+2jXV3cqb6lwO0 9TvleQQt3IbDIyJhaEdhCJMQO3I4TkQlMNGmEmyUgpQWe6wFz4M1XPNN3+BA+tZML2rk +VmXGNtgzcvUS6Z4ikZNftkRO+yA7hvF2KykNxdRJkPkE5ZjaqM6mJimk4gYoqMYGDyz Nulfn2TbTiY8LfgOvfjc8DsKC0uZD/LDVhWVKM3oOjUlPz+msOOORjlXCtdSCV8ezEdJ +ROg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id f62si2399590plb.121.2018.01.04.07.12.31; Thu, 04 Jan 2018 07:12:31 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753462AbeADPK1 (ORCPT + 22 others); Thu, 4 Jan 2018 10:10:27 -0500 Received: from usa-sjc-mx-foss1.foss.arm.com ([217.140.101.70]:33832 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753380AbeADPIm (ORCPT ); Thu, 4 Jan 2018 10:08:42 -0500 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 8337A168F; Thu, 4 Jan 2018 07:08:41 -0800 (PST) Received: from edgewater-inn.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 53D8D3F7C5; Thu, 4 Jan 2018 07:08:41 -0800 (PST) Received: by edgewater-inn.cambridge.arm.com (Postfix, from userid 1000) id ECF1F1AE1195; Thu, 4 Jan 2018 15:08:40 +0000 (GMT) From: Will Deacon To: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com, ard.biesheuvel@linaro.org, marc.zyngier@arm.com, lorenzo.pieralisi@arm.com, christoffer.dall@linaro.org, linux-kernel@vger.kernel.org, Will Deacon Subject: [PATCH 09/11] arm64: KVM: Make PSCI_VERSION a fast path Date: Thu, 4 Jan 2018 15:08:33 +0000 Message-Id: <1515078515-13723-10-git-send-email-will.deacon@arm.com> X-Mailer: git-send-email 2.1.4 In-Reply-To: <1515078515-13723-1-git-send-email-will.deacon@arm.com> References: <1515078515-13723-1-git-send-email-will.deacon@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org From: Marc Zyngier For those CPUs that require PSCI to perform a BP invalidation, going all the way to the PSCI code for not much is a waste of precious cycles. Let's terminate that call as early as possible. Signed-off-by: Marc Zyngier Signed-off-by: Will Deacon --- arch/arm64/kvm/hyp/switch.c | 13 +++++++++++++ 1 file changed, 13 insertions(+) -- 2.1.4 diff --git a/arch/arm64/kvm/hyp/switch.c b/arch/arm64/kvm/hyp/switch.c index 8d4f3c9d6dc4..4d273f6d0e69 100644 --- a/arch/arm64/kvm/hyp/switch.c +++ b/arch/arm64/kvm/hyp/switch.c @@ -17,6 +17,7 @@ #include #include +#include #include #include @@ -341,6 +342,18 @@ int __hyp_text __kvm_vcpu_run(struct kvm_vcpu *vcpu) if (exit_code == ARM_EXCEPTION_TRAP && !__populate_fault_info(vcpu)) goto again; + if (exit_code == ARM_EXCEPTION_TRAP && + (kvm_vcpu_trap_get_class(vcpu) == ESR_ELx_EC_HVC64 || + kvm_vcpu_trap_get_class(vcpu) == ESR_ELx_EC_HVC32) && + vcpu_get_reg(vcpu, 0) == PSCI_0_2_FN_PSCI_VERSION) { + u64 val = PSCI_RET_NOT_SUPPORTED; + if (test_bit(KVM_ARM_VCPU_PSCI_0_2, vcpu->arch.features)) + val = 2; + + vcpu_set_reg(vcpu, 0, val); + goto again; + } + if (static_branch_unlikely(&vgic_v2_cpuif_trap) && exit_code == ARM_EXCEPTION_TRAP) { bool valid; From patchwork Thu Jan 4 15:08:34 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Will Deacon X-Patchwork-Id: 123440 Delivered-To: patch@linaro.org Received: by 10.80.135.92 with SMTP id 28csp6823248edv; Thu, 4 Jan 2018 07:12:19 -0800 (PST) X-Google-Smtp-Source: ACJfBotDiYpBQsxbODrX8KKFEDMVvQvqpo4ftKmsyBtGWzp/0Np7raEnZtx0b3KseZXnZSYFVNf0 X-Received: by 10.101.81.76 with SMTP id g12mr82041pgq.150.1515078739677; Thu, 04 Jan 2018 07:12:19 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1515078739; cv=none; d=google.com; s=arc-20160816; b=ZrhRatelMsUb/v3Z7A1YgkyiGPhY7OYfvNL+XMC+QkGntnP71LuxFTNIzZXof01fE1 uv0J61B0Y804O8cwrRoV8CWFmRYJABAqOXFdQXtLRFlXKWcEP49LJc+wXpQb4MZ9Xbn2 MtE5l4bVpPUp7E24DN/MVTBBd2jG4Wiy88EjmkzzGTA/M3/rUsGKYzMQjAedoxtUvY46 E3dgmyLOF43+kYKyjJ2eiolcqMO/K1IIyg5ia8AVjqazlQV8z4Q4WPoUyEzLP+K/fMmg QCzuVELfELRvTzBwc2PRo7A1pZkwzEMESk7cX/7zGAc5zaWtgM3teRDRj1kqVgx71ta5 EQhQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:arc-authentication-results; bh=jpUYmNEx7VU+s0l/ndx3SHVIXhCGbNw/RkyqJRiZOVc=; b=TWTLTFY5uWO9WbvrVFu3KoHO2FZ/yemVhJNI+CId+PLLgY5vTa9YVlRpE3slFuFDGl TFBCAEjvMuJQ3avKdjMcw010PcEtjD8g4gJp6zH8DrEkmE8Zd1JYAAv3Fb7tDH5OzsxV L33fEXN5dfyH+mflw3ZnpU9jLaus9I0K7bdcLiTGTzsfHQfdWBDTjdJkSCryaXyw0phB f+H8THQ9DJGFPGQHzpTtFQPSkIp4+AtYaju12TXNSoc+NuQR1H2zUvgB3nnxFNR8JFLT eoYeXHpkqHBQ510c1E5VYbBTrXDFgwMyuNWgRY+lxcH7a4G/hS4dJckmoC33tT0Xz9Jr +KLw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id g62si2148912pgc.428.2018.01.04.07.12.19; Thu, 04 Jan 2018 07:12:19 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753475AbeADPK3 (ORCPT + 22 others); Thu, 4 Jan 2018 10:10:29 -0500 Received: from foss.arm.com ([217.140.101.70]:33830 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753379AbeADPIm (ORCPT ); Thu, 4 Jan 2018 10:08:42 -0500 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 8AEDB169F; Thu, 4 Jan 2018 07:08:41 -0800 (PST) Received: from edgewater-inn.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 5C7943F7C7; Thu, 4 Jan 2018 07:08:41 -0800 (PST) Received: by edgewater-inn.cambridge.arm.com (Postfix, from userid 1000) id 0D28C1AE125D; Thu, 4 Jan 2018 15:08:41 +0000 (GMT) From: Will Deacon To: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com, ard.biesheuvel@linaro.org, marc.zyngier@arm.com, lorenzo.pieralisi@arm.com, christoffer.dall@linaro.org, linux-kernel@vger.kernel.org, Will Deacon Subject: [PATCH 10/11] arm64: cputype: Add missing MIDR values for Cortex-A72 and Cortex-A75 Date: Thu, 4 Jan 2018 15:08:34 +0000 Message-Id: <1515078515-13723-11-git-send-email-will.deacon@arm.com> X-Mailer: git-send-email 2.1.4 In-Reply-To: <1515078515-13723-1-git-send-email-will.deacon@arm.com> References: <1515078515-13723-1-git-send-email-will.deacon@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Hook up MIDR values for the Cortex-A72 and Cortex-A75 CPUs, since they will soon need MIDR matches for hardening the branch predictor. Signed-off-by: Will Deacon --- arch/arm64/include/asm/cputype.h | 4 ++++ 1 file changed, 4 insertions(+) -- 2.1.4 diff --git a/arch/arm64/include/asm/cputype.h b/arch/arm64/include/asm/cputype.h index 235e77d98261..84385b94e70b 100644 --- a/arch/arm64/include/asm/cputype.h +++ b/arch/arm64/include/asm/cputype.h @@ -79,8 +79,10 @@ #define ARM_CPU_PART_AEM_V8 0xD0F #define ARM_CPU_PART_FOUNDATION 0xD00 #define ARM_CPU_PART_CORTEX_A57 0xD07 +#define ARM_CPU_PART_CORTEX_A72 0xD08 #define ARM_CPU_PART_CORTEX_A53 0xD03 #define ARM_CPU_PART_CORTEX_A73 0xD09 +#define ARM_CPU_PART_CORTEX_A75 0xD0A #define APM_CPU_PART_POTENZA 0x000 @@ -94,7 +96,9 @@ #define MIDR_CORTEX_A53 MIDR_CPU_MODEL(ARM_CPU_IMP_ARM, ARM_CPU_PART_CORTEX_A53) #define MIDR_CORTEX_A57 MIDR_CPU_MODEL(ARM_CPU_IMP_ARM, ARM_CPU_PART_CORTEX_A57) +#define MIDR_CORTEX_A72 MIDR_CPU_MODEL(ARM_CPU_IMP_ARM, ARM_CPU_PART_CORTEX_A72) #define MIDR_CORTEX_A73 MIDR_CPU_MODEL(ARM_CPU_IMP_ARM, ARM_CPU_PART_CORTEX_A73) +#define MIDR_CORTEX_A75 MIDR_CPU_MODEL(ARM_CPU_IMP_ARM, ARM_CPU_PART_CORTEX_A75) #define MIDR_THUNDERX MIDR_CPU_MODEL(ARM_CPU_IMP_CAVIUM, CAVIUM_CPU_PART_THUNDERX) #define MIDR_THUNDERX_81XX MIDR_CPU_MODEL(ARM_CPU_IMP_CAVIUM, CAVIUM_CPU_PART_THUNDERX_81XX) #define MIDR_THUNDERX_83XX MIDR_CPU_MODEL(ARM_CPU_IMP_CAVIUM, CAVIUM_CPU_PART_THUNDERX_83XX) From patchwork Thu Jan 4 15:08:35 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Will Deacon X-Patchwork-Id: 123442 Delivered-To: patch@linaro.org Received: by 10.80.135.92 with SMTP id 28csp6823773edv; Thu, 4 Jan 2018 07:12:44 -0800 (PST) X-Google-Smtp-Source: ACJfBossUD6qCDORozQGGYSOaXkajZZzdsFRI3gYWyU4vbDp9P/YGHU/J52wcL9uG2I+NMtKltJp X-Received: by 10.159.194.197 with SMTP id u5mr4929583plz.448.1515078764119; Thu, 04 Jan 2018 07:12:44 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1515078764; cv=none; d=google.com; s=arc-20160816; b=ed7xAU6D1yg5SMnuvsLemGtmvyrnEI35THLi4ETCsG8n3OSGEs/RuktTib8rvNSwz6 XolfB5UJbhlWPYPpNkYj47BxgVOLiDmbyzEZ7b8fP1EcXWMoUFAtXmaYFeODhu7ebaSY l/rKa1ns7cyu9znftsLDdjTR+f8DaHSWm7SaDptZp0ZNlM8XINc/L4n/z7F56ANAjoXx jZ8478javx4/s9kR5w73MxDU82FjjWR+DdCKIDlA1hBNy+udz8SD5IsonxqnkezAddw+ Kyy4Jqd7xLCpoCHe0LZS+pJfkfulJeAmwJCTcGdTBj2sfZQuNXKALzRkEg57HDzkqAZ9 mGNQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:arc-authentication-results; bh=Xc//0WbePMQ+pVs2gqq96AzbPTzeiI1THU6AXINmrdA=; b=acRjFhWK8mmmA106Buxt2M1D+bdv9/IMkEvaNgGdn01ElX8UpKhBJc1uGFIPT4tX0U yyAywgN3H+iV9tR/0oAzJFECSuXhElWvfCEyOfGgiwsKodsBFMVtc8RkCJl6Yj13lS/j l02WA2kvpaSVHYIgVFUDDWjeUo8gTWhC5Nw/J5WEBMFSjvjKeEQZxnRZbxFkcTxFfCQV DABjQenG3FFRSYRr5/73dbJlGuZ+PaXweLI4/g3t/3eOH1k9vKt1wOhNseyfBD0NkG6p DoXJ5lPAezxuWsJReIyhrwswyL5GxLXFDOcwly+tIRKq/uccCOq6N3N5+cbYQFkLenCK uRwg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id f62si2399590plb.121.2018.01.04.07.12.43; Thu, 04 Jan 2018 07:12:44 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753326AbeADPK0 (ORCPT + 22 others); Thu, 4 Jan 2018 10:10:26 -0500 Received: from usa-sjc-mx-foss1.foss.arm.com ([217.140.101.70]:33846 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753382AbeADPIm (ORCPT ); Thu, 4 Jan 2018 10:08:42 -0500 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 9352B16BA; Thu, 4 Jan 2018 07:08:41 -0800 (PST) Received: from edgewater-inn.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 658943F41F; Thu, 4 Jan 2018 07:08:41 -0800 (PST) Received: by edgewater-inn.cambridge.arm.com (Postfix, from userid 1000) id 1D6E41AE12E1; Thu, 4 Jan 2018 15:08:41 +0000 (GMT) From: Will Deacon To: linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com, ard.biesheuvel@linaro.org, marc.zyngier@arm.com, lorenzo.pieralisi@arm.com, christoffer.dall@linaro.org, linux-kernel@vger.kernel.org, Will Deacon Subject: [PATCH 11/11] arm64: Implement branch predictor hardening for affected Cortex-A CPUs Date: Thu, 4 Jan 2018 15:08:35 +0000 Message-Id: <1515078515-13723-12-git-send-email-will.deacon@arm.com> X-Mailer: git-send-email 2.1.4 In-Reply-To: <1515078515-13723-1-git-send-email-will.deacon@arm.com> References: <1515078515-13723-1-git-send-email-will.deacon@arm.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Cortex-A57, A72, A73 and A75 are susceptible to branch predictor aliasing and can theoretically be attacked by malicious code. This patch implements a PSCI-based mitigation for these CPUs when available. The call into firmware will invalidate the branch predictor state, preventing any malicious entries from affecting other victim contexts. Signed-off-by: Marc Zyngier Signed-off-by: Will Deacon --- arch/arm64/kernel/bpi.S | 24 ++++++++++++++++++++++++ arch/arm64/kernel/cpu_errata.c | 42 ++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 66 insertions(+) -- 2.1.4 diff --git a/arch/arm64/kernel/bpi.S b/arch/arm64/kernel/bpi.S index 06a931eb2673..2b10d52a0321 100644 --- a/arch/arm64/kernel/bpi.S +++ b/arch/arm64/kernel/bpi.S @@ -53,3 +53,27 @@ ENTRY(__bp_harden_hyp_vecs_start) vectors __kvm_hyp_vector .endr ENTRY(__bp_harden_hyp_vecs_end) +ENTRY(__psci_hyp_bp_inval_start) + stp x0, x1, [sp, #-16]! + stp x2, x3, [sp, #-16]! + stp x4, x5, [sp, #-16]! + stp x6, x7, [sp, #-16]! + stp x8, x9, [sp, #-16]! + stp x10, x11, [sp, #-16]! + stp x12, x13, [sp, #-16]! + stp x14, x15, [sp, #-16]! + stp x16, x17, [sp, #-16]! + stp x18, x19, [sp, #-16]! + mov x0, #0x84000000 + smc #0 + ldp x18, x19, [sp], #16 + ldp x16, x17, [sp], #16 + ldp x14, x15, [sp], #16 + ldp x12, x13, [sp], #16 + ldp x10, x11, [sp], #16 + ldp x8, x9, [sp], #16 + ldp x6, x7, [sp], #16 + ldp x4, x5, [sp], #16 + ldp x2, x3, [sp], #16 + ldp x0, x1, [sp], #16 +ENTRY(__psci_hyp_bp_inval_end) diff --git a/arch/arm64/kernel/cpu_errata.c b/arch/arm64/kernel/cpu_errata.c index 16ea5c6f314e..cb0fb3796bb8 100644 --- a/arch/arm64/kernel/cpu_errata.c +++ b/arch/arm64/kernel/cpu_errata.c @@ -53,6 +53,8 @@ static int cpu_enable_trap_ctr_access(void *__unused) DEFINE_PER_CPU_READ_MOSTLY(struct bp_hardening_data, bp_hardening_data); #ifdef CONFIG_KVM +extern char __psci_hyp_bp_inval_start[], __psci_hyp_bp_inval_end[]; + static void __copy_hyp_vect_bpi(int slot, const char *hyp_vecs_start, const char *hyp_vecs_end) { @@ -94,6 +96,9 @@ static void __install_bp_hardening_cb(bp_hardening_cb_t fn, spin_unlock(&bp_lock); } #else +#define __psci_hyp_bp_inval_start NULL +#define __psci_hyp_bp_inval_end NULL + static void __install_bp_hardening_cb(bp_hardening_cb_t fn, const char *hyp_vecs_start, const char *hyp_vecs_end) @@ -118,6 +123,21 @@ static void install_bp_hardening_cb(const struct arm64_cpu_capabilities *entry, __install_bp_hardening_cb(fn, hyp_vecs_start, hyp_vecs_end); } + +#include + +static int enable_psci_bp_hardening(void *data) +{ + const struct arm64_cpu_capabilities *entry = data; + + if (psci_ops.get_version) + install_bp_hardening_cb(entry, + (bp_hardening_cb_t)psci_ops.get_version, + __psci_hyp_bp_inval_start, + __psci_hyp_bp_inval_end); + + return 0; +} #endif /* CONFIG_HARDEN_BRANCH_PREDICTOR */ #define MIDR_RANGE(model, min, max) \ @@ -261,6 +281,28 @@ const struct arm64_cpu_capabilities arm64_errata[] = { MIDR_ALL_VERSIONS(MIDR_CORTEX_A73), }, #endif +#ifdef CONFIG_HARDEN_BRANCH_PREDICTOR + { + .capability = ARM64_HARDEN_BRANCH_PREDICTOR, + MIDR_ALL_VERSIONS(MIDR_CORTEX_A57), + .enable = enable_psci_bp_hardening, + }, + { + .capability = ARM64_HARDEN_BRANCH_PREDICTOR, + MIDR_ALL_VERSIONS(MIDR_CORTEX_A72), + .enable = enable_psci_bp_hardening, + }, + { + .capability = ARM64_HARDEN_BRANCH_PREDICTOR, + MIDR_ALL_VERSIONS(MIDR_CORTEX_A73), + .enable = enable_psci_bp_hardening, + }, + { + .capability = ARM64_HARDEN_BRANCH_PREDICTOR, + MIDR_ALL_VERSIONS(MIDR_CORTEX_A75), + .enable = enable_psci_bp_hardening, + }, +#endif { } };