From patchwork Wed Mar 30 15:41:50 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 555178 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2CC24C433FE for ; Wed, 30 Mar 2022 15:42:22 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S239041AbiC3PoF (ORCPT ); Wed, 30 Mar 2022 11:44:05 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54642 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1348341AbiC3PoE (ORCPT ); Wed, 30 Mar 2022 11:44:04 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2261433E3E for ; Wed, 30 Mar 2022 08:42:19 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id B12CA616D1 for ; Wed, 30 Mar 2022 15:42:18 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id B788DC34110; Wed, 30 Mar 2022 15:42:16 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1648654938; bh=2XVgmsF56y1icz9ml/x4C3CrLOrJjhf8ku8guxGluyA=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=EzTtGjSSNvHI2GN6o7kSVeISyPYNXiEfA5iW4Op4l0Ak/nOLSj4irpY/hDPmDUMxB Zny1gPOWbZou0P24310VSW05VK0w1DTlqc1ZPU0GqOsJkrTpKe7dLX7jj/gIkbm/g8 G25UedE/pq1buAJrAKl69SO7OUG2kJpFEE3RVrFSc3O36ObKe0c/+g2nIzVNID9CTZ x9oTAk7IgcFHQTpi2GBC0CS1B6oTZqqhi3hR9EhZ3d/o82Nrj+EDR+8tMBakJz4RVO 5nVegoB8LRwE8cWdHssqV4fxQtgw+0Fm/rcyiR43f9SEc/ryinTA7pF/7JmH6RmqqF 3K3qOQONyh7iw== From: Ard Biesheuvel To: linux-efi@vger.kernel.org Cc: linux-arm-kernel@lists.infradead.org, keescook@chromium.org, mark.rutland@arm.com, catalin.marinas@arm.com, Ard Biesheuvel Subject: [RFC PATCH v2 03/18] arm64: kernel: drop unnecessary PoC cache clean+invalidate Date: Wed, 30 Mar 2022 17:41:50 +0200 Message-Id: <20220330154205.2483167-4-ardb@kernel.org> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220330154205.2483167-1-ardb@kernel.org> References: <20220330154205.2483167-1-ardb@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=2088; h=from:subject; bh=2XVgmsF56y1icz9ml/x4C3CrLOrJjhf8ku8guxGluyA=; b=owEB7QES/pANAwAKAcNPIjmS2Y8kAcsmYgBiRHoz8v9i+97hjHtxztia+HkW+5AUge3p9D73EZan Mu7BTKeJAbMEAAEKAB0WIQT72WJ8QGnJQhU3VynDTyI5ktmPJAUCYkR6MwAKCRDDTyI5ktmPJCH8DA Crv17f82H/LBfBNhyAP7Ff0OA+mBKOavC5sA4sidYX3jSjI8V0+QsCNDuZ3S8Of2xTjmyc07KA6qlS +4AV6GDeMeubToqyZoqBlc2g85JO00uJ/zfeHEMxlVF2UgFCDsVbgzIlh2S5hfXyqntYVH2KYemYDn a8vg6r3RfST12ADhWeQt137oXcWlmnC1w9EyuR8f2HmlZT0n4dlRsqzLSngVgcggTO4K/VlK5IzNVa M/odVNT52hi7fT3KPTSp3MAJ6soW1AZ9sWGXSY4xf4tE8NwBB+8lZqe1XakXr0Ujj36TkLhNp4hpC5 q4XaWDD13Nn2/HBzv9swfkXZtMw+ePrdBimcL/0q6QO9WMXQkNbWqdjf5ThmdW1vzB2q5+qRChp+Mx 14ggKzA5lrflpQgVzen9Dpudx+/RZowOXmQdF0cQ4q0VVxGO/mPMm1aqNCvf7JLLXoRjnkROcLQy8G Dl91IBdiiAlBNTA0MnHaxYyjs5GLYbo6/ck3KGTdRGCR8= X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 Precedence: bulk List-ID: X-Mailing-List: linux-efi@vger.kernel.org Some early boot code runs before the virtual placement of the kernel is finalized, and we used to go back to the very start and recreate the ID map along with the page tables describing the virtual kernel mapping, and this involved setting some global variables with the caches off. In order to ensure that global state created by the KASLR code is not corrupted by the cache invalidation that occurs in that case, we needed to clean those global variables to the PoC explicitly. This is no longer needed now that the ID map is created only once (and the associated global variable updates are no longer repeated). So drop the cache maintenance that is no longer necessary. Signed-off-by: Ard Biesheuvel --- arch/arm64/kernel/kaslr.c | 11 ----------- 1 file changed, 11 deletions(-) diff --git a/arch/arm64/kernel/kaslr.c b/arch/arm64/kernel/kaslr.c index 418b2bba1521..d5542666182f 100644 --- a/arch/arm64/kernel/kaslr.c +++ b/arch/arm64/kernel/kaslr.c @@ -13,7 +13,6 @@ #include #include -#include #include #include #include @@ -72,9 +71,6 @@ u64 __init kaslr_early_init(void) * we end up running with module randomization disabled. */ module_alloc_base = (u64)_etext - MODULES_VSIZE; - dcache_clean_inval_poc((unsigned long)&module_alloc_base, - (unsigned long)&module_alloc_base + - sizeof(module_alloc_base)); /* * Try to map the FDT early. If this fails, we simply bail, @@ -174,13 +170,6 @@ u64 __init kaslr_early_init(void) module_alloc_base += (module_range * (seed & ((1 << 21) - 1))) >> 21; module_alloc_base &= PAGE_MASK; - dcache_clean_inval_poc((unsigned long)&module_alloc_base, - (unsigned long)&module_alloc_base + - sizeof(module_alloc_base)); - dcache_clean_inval_poc((unsigned long)&memstart_offset_seed, - (unsigned long)&memstart_offset_seed + - sizeof(memstart_offset_seed)); - return offset; } From patchwork Wed Mar 30 15:41:51 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 555177 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 80AE0C433EF for ; Wed, 30 Mar 2022 15:42:25 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1348341AbiC3PoJ (ORCPT ); Wed, 30 Mar 2022 11:44:09 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54854 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1348339AbiC3PoG (ORCPT ); Wed, 30 Mar 2022 11:44:06 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 06C2533E3E for ; Wed, 30 Mar 2022 08:42:21 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 9111F61562 for ; Wed, 30 Mar 2022 15:42:20 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 906F7C340F2; Wed, 30 Mar 2022 15:42:18 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1648654940; bh=c6iRYtAbo4zUXLvlyPL5mdWMCH5xxMM/KqJwOLjBhFA=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=hGM4OE8iUs2dTV3Li+2QCehHoWLE5Hfp/1lNKWqux4NMiIXtNUtcvuSHwJ0cithwq lBpHzfqCeeFlWckA1d6Hz8m3PD1kNtgRu7Ln16vG7vXA7Wsj/jFD2lF0gbqSvMUfO3 NP0pXqEFszKZhRNVWzxKFNdwu25HT4MjjwWo3U9pIL16Mj3vj2b+QPvH2jzATecI4u W/8BNBN6iTI4q/NXuOB7V6NQHXNR5ciB64r7LOvKFI5LbshM4h985NLNql0vtrcclV ARjGOAq2QFTsCtiJFUIqr4uE+aFX0bUxYgy+znPGVGnXqmdlY/SpCYdbhy3TSrSTjb UpVR5+SAYh94g== From: Ard Biesheuvel To: linux-efi@vger.kernel.org Cc: linux-arm-kernel@lists.infradead.org, keescook@chromium.org, mark.rutland@arm.com, catalin.marinas@arm.com, Ard Biesheuvel Subject: [RFC PATCH v2 04/18] arm64: head: cover entire kernel image in ID map Date: Wed, 30 Mar 2022 17:41:51 +0200 Message-Id: <20220330154205.2483167-5-ardb@kernel.org> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220330154205.2483167-1-ardb@kernel.org> References: <20220330154205.2483167-1-ardb@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=2885; h=from:subject; bh=c6iRYtAbo4zUXLvlyPL5mdWMCH5xxMM/KqJwOLjBhFA=; b=owEB7QES/pANAwAKAcNPIjmS2Y8kAcsmYgBiRHo1bwvcFKr9aJYAFLwhx72X6WP77BHK5U7s1aXY iI6OdOuJAbMEAAEKAB0WIQT72WJ8QGnJQhU3VynDTyI5ktmPJAUCYkR6NQAKCRDDTyI5ktmPJFfBC/ 9ioYg7jfchoEqb2LfzWB/OTkgGcnJdBDUVkCn1V4WqhDm4AjW/bJ9tTwdrkzzngucwbo+czZdezmzd nDJ8AMsh2Fa11XJ65JGrDrU9zhCoFOD0ThFUa+rCBFaOWxFc+pZJbsCTWzKFxbufsz9GnM048UnBs2 XSrAl2rkGctFqe45M5/lLJbSNhGIgEmI23uNlFkgjDB2MHas6pdOvDoTSc2114tY9xhmXeY8+W2n4c /XUplmxgQ863qNFgnnVrJfdBjg4M7IQLvFY2ReT34kFXFyL0PnafJ7mQXEeBx72OYgpoQrkDYoIQbc YOqGgfhNMQB4pgNNgD3d/euCEumC5daNmLPXo45foz5FleZQdd1C4v4TP9OHhXj4ihgJJlH0WSamSw ortlOB1XPSgUlfTVolRLcjEdHq5sRq00C2w6IvxnEhTwg+UVe86kfObwZJ4HHkVfgTIPgefQN7MZuR 1ies9362rUr7Iit6hW5lpwAG0AJ9wj/JzzYIAXX0SRmA8= X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 Precedence: bulk List-ID: X-Mailing-List: linux-efi@vger.kernel.org As a first step towards avoiding the need to create, tear down and recreate the kernel virtual mapping with MMU and caches disabled, start by expanding the ID map so it covers the page tables as well as all executable code. This will allow us to populate the page tables with the MMU and caches on, and call KASLR init code before setting up the virtual mapping. Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/kernel-pgtable.h | 2 +- arch/arm64/kernel/head.S | 10 +++++----- 2 files changed, 6 insertions(+), 6 deletions(-) diff --git a/arch/arm64/include/asm/kernel-pgtable.h b/arch/arm64/include/asm/kernel-pgtable.h index 96dc0f7da258..b62200a9456e 100644 --- a/arch/arm64/include/asm/kernel-pgtable.h +++ b/arch/arm64/include/asm/kernel-pgtable.h @@ -87,7 +87,7 @@ + EARLY_PUDS((vstart), (vend)) /* each PUD needs a next level page table */ \ + EARLY_PMDS((vstart), (vend))) /* each PMD needs a next level page table */ #define INIT_DIR_SIZE (PAGE_SIZE * EARLY_PAGES(KIMAGE_VADDR, _end)) -#define IDMAP_DIR_SIZE (IDMAP_PGTABLE_LEVELS * PAGE_SIZE) +#define IDMAP_DIR_SIZE INIT_DIR_SIZE /* Initial memory map size */ #if ARM64_KERNEL_USES_PMD_MAPS diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S index 275cd14a70c2..727561972e4a 100644 --- a/arch/arm64/kernel/head.S +++ b/arch/arm64/kernel/head.S @@ -287,7 +287,7 @@ SYM_FUNC_END(clear_page_tables) SYM_FUNC_START_LOCAL(create_idmap) adrp x0, idmap_pg_dir - adrp x3, __idmap_text_start // __pa(__idmap_text_start) + adrp x3, _text // __pa(_text) #ifdef CONFIG_ARM64_VA_BITS_52 mrs_s x6, SYS_ID_AA64MMFR2_EL1 @@ -312,10 +312,10 @@ SYM_FUNC_START_LOCAL(create_idmap) * Calculate the maximum allowed value for TCR_EL1.T0SZ so that the * entire ID map region can be mapped. As T0SZ == (64 - #bits used), * this number conveniently equals the number of leading zeroes in - * the physical address of __idmap_text_end. + * the physical address of _end. */ mov x4, PTRS_PER_PGD - adrp x5, __idmap_text_end + adrp x5, _end clz x5, x5 cmp x5, TCR_T0SZ(VA_BITS_MIN) // default T0SZ small enough? b.ge 1f // .. then skip VA range extension @@ -351,7 +351,7 @@ SYM_FUNC_START_LOCAL(create_idmap) mov x4, #1 << (PHYS_MASK_SHIFT - PGDIR_SHIFT) #endif 1: - adr_l x6, __idmap_text_end // __pa(__idmap_text_end) + adr_l x6, _end // __pa(_end) mov x7, SWAPPER_MM_MMUFLAGS map_memory x0, x1, x3, x6, x7, x3, x4, x10, x11, x12, x13, x14 @@ -884,7 +884,7 @@ SYM_FUNC_START_LOCAL(__primary_switch) msr sctlr_el1, x20 // disable the MMU isb bl clear_page_tables - bl create_kernel_mapping // recreate kernel mapping + bl create_kernel_mapping // Recreate kernel mapping tlbi vmalle1 // Remove any stale TLB entries dsb nsh From patchwork Wed Mar 30 15:41:53 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 555176 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C59C3C43217 for ; Wed, 30 Mar 2022 15:42:26 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1348339AbiC3PoK (ORCPT ); Wed, 30 Mar 2022 11:44:10 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55108 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1348347AbiC3PoJ (ORCPT ); Wed, 30 Mar 2022 11:44:09 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9ACDD33E9E for ; Wed, 30 Mar 2022 08:42:24 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 37EFB61727 for ; Wed, 30 Mar 2022 15:42:24 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 428FBC34118; Wed, 30 Mar 2022 15:42:22 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1648654943; bh=mlE7in6kp8AxPXbRmgxis3u7A1nYOdCcJHiVv3g+5g4=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=iI810+obTsHfHWryaVinmb3+le8TZnhFjBfncMoIT0yDFWcQFZYOycFaMX11Kk6aa XG/A37UEta5mciqPXVwe20q2tTKoYQd7wufzuiqRxmmJty5g/5qpPMUqqmUbMw2FsW FcH5ekjHILZQ2zsdaoxEMihH/8s2c9y/xNYzJ8DefIGIHyePr/1gNxYf4r1SdskDEc DV0mE8xqq/JeFffpUp2LA3h9CFt+2m94seaTHVTh5V+Gc6DluLHFEpQoQ7d2DGfAju PbzNxjxQDiJUWF0Y0plebCbl/tl8VVDkq1JFZuLn1tb82dGlrt2yA7aOdt08JDQ23y OvITNoUzRj5uA== From: Ard Biesheuvel To: linux-efi@vger.kernel.org Cc: linux-arm-kernel@lists.infradead.org, keescook@chromium.org, mark.rutland@arm.com, catalin.marinas@arm.com, Ard Biesheuvel Subject: [RFC PATCH v2 06/18] arm64: head: populate kernel page tables with MMU and caches on Date: Wed, 30 Mar 2022 17:41:53 +0200 Message-Id: <20220330154205.2483167-7-ardb@kernel.org> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220330154205.2483167-1-ardb@kernel.org> References: <20220330154205.2483167-1-ardb@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=3911; h=from:subject; bh=mlE7in6kp8AxPXbRmgxis3u7A1nYOdCcJHiVv3g+5g4=; b=owEB7QES/pANAwAKAcNPIjmS2Y8kAcsmYgBiRHo4/2sEZOXX/GDehtECXU7Si/1bIpoAUjef9zQC AV2Eo6iJAbMEAAEKAB0WIQT72WJ8QGnJQhU3VynDTyI5ktmPJAUCYkR6OAAKCRDDTyI5ktmPJO9qC/ 0Zh/uF0Y7I/Ah3uDuqKfksmeg1Yn05fO9wAGuPIVUfCE6zwkQ/VXkHtkgdgAEuxS3zI6zgihZaeMEZ BrpQiX+zmyj9HAGht0VmX0ZFNB9Yn4F/pt9hRxc3z7LcrzvgRedRGXQrcqTkE3XEVrul5xAu/dDeEr ZhOc9xRgpXb8YlerEb843qOdhlj/8r6n56N0Q/jeztCBNh5vcC2LYQVxSlcri181O8JWj3Z4cpxCO5 jmQLOZqmXl26K1Rgtw0Gt/N4K2z1xLtpj/hhgW4iiXLLOpdnf2Ayl6ZgA5jBQ4BeEMb+O7iNzi7GvR DhDtC23VE9KO6DuGeocye8R4Ihc/JBHLIwo0Wx7MBNK4JFcVpBrk1GbJ+Ctg4XconaX8Oxpvnvx4tH uFYY7tc/YLNTpBA8hmUOUIYDL5ZsetameFU09YtC0REEMi8DqQwn8pm/SnTVRydn1qj/7VpokherFa ptUKR1luRZBzb4RqoOVIb75AA8kHmakhVpu0WgD3Q31/g= X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 Precedence: bulk List-ID: X-Mailing-List: linux-efi@vger.kernel.org Now that we can access the entire kernel image via the ID map, we can execute the page table population code with the MMU and caches enabled. The only thing we need to ensure is that translations via TTBR1 remain disabled while we are updating the page tables the second time around, in case KASLR wants them to be randomized. Signed-off-by: Ard Biesheuvel --- arch/arm64/kernel/head.S | 48 +++++--------------- 1 file changed, 11 insertions(+), 37 deletions(-) diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S index 7c4aefacf6c2..5d4cb481e42f 100644 --- a/arch/arm64/kernel/head.S +++ b/arch/arm64/kernel/head.S @@ -84,8 +84,6 @@ * Register Scope Purpose * x21 primary_entry() .. start_kernel() FDT pointer passed at boot in x0 * x23 primary_entry() .. start_kernel() physical misalignment/KASLR offset - * x28 clear_page_tables() callee preserved temp register - * x19/x20 __primary_switch() callee preserved temp registers * x24 __primary_switch() .. relocate_kernel() current RELR displacement */ SYM_CODE_START(primary_entry) @@ -94,9 +92,7 @@ SYM_CODE_START(primary_entry) adrp x23, __PHYS_OFFSET and x23, x23, MIN_KIMG_ALIGN - 1 // KASLR offset, defaults to 0 bl set_cpu_boot_mode_flag - bl clear_page_tables bl create_idmap - bl create_kernel_mapping /* * The following calls CPU setup code, see arch/arm64/mm/proc.S for @@ -126,18 +122,6 @@ SYM_CODE_START_LOCAL(preserve_boot_args) SYM_CODE_END(preserve_boot_args) SYM_FUNC_START_LOCAL(clear_page_tables) - mov x28, lr - - /* - * Invalidate the init page tables to avoid potential dirty cache lines - * being evicted. Other page tables are allocated in rodata as part of - * the kernel image, and thus are clean to the PoC per the boot - * protocol. - */ - adrp x0, init_pg_dir - adrp x1, init_pg_end - bl dcache_inval_poc - /* * Clear the init page tables. */ @@ -151,7 +135,7 @@ SYM_FUNC_START_LOCAL(clear_page_tables) subs x1, x1, #64 b.ne 1b - ret x28 + ret SYM_FUNC_END(clear_page_tables) /* @@ -381,16 +365,7 @@ SYM_FUNC_START_LOCAL(create_kernel_mapping) map_memory x0, x1, x5, x6, x7, x3, x4, x10, x11, x12, x13, x14 - /* - * Since the page tables have been populated with non-cacheable - * accesses (MMU disabled), invalidate those tables again to - * remove any speculatively loaded cache lines. - */ - dmb sy - - adrp x0, init_pg_dir - adrp x1, init_pg_end - b dcache_inval_poc // tail call + ret SYM_FUNC_END(create_kernel_mapping) /* @@ -862,13 +837,13 @@ SYM_FUNC_END(__relocate_kernel) #endif SYM_FUNC_START_LOCAL(__primary_switch) -#ifdef CONFIG_RANDOMIZE_BASE - mov x19, x0 // preserve new SCTLR_EL1 value - mrs x20, sctlr_el1 // preserve old SCTLR_EL1 value -#endif + adrp x1, reserved_pg_dir + bl __enable_mmu + bl clear_page_tables + bl create_kernel_mapping adrp x1, init_pg_dir - bl __enable_mmu + load_ttbr1 x1, x2 #ifdef CONFIG_RELOCATABLE #ifdef CONFIG_RELR mov x24, #0 // no RELR displacement yet @@ -884,9 +859,8 @@ SYM_FUNC_START_LOCAL(__primary_switch) * to take into account by discarding the current kernel mapping and * creating a new one. */ - pre_disable_mmu_workaround - msr sctlr_el1, x20 // disable the MMU - isb + adrp x1, reserved_pg_dir // Disable translations via TTBR1 + load_ttbr1 x1, x2 bl clear_page_tables bl create_kernel_mapping // Recreate kernel mapping @@ -894,8 +868,8 @@ SYM_FUNC_START_LOCAL(__primary_switch) dsb nsh isb - set_sctlr_el1 x19 // re-enable the MMU - + adrp x1, init_pg_dir // Re-enable translations via TTBR1 + load_ttbr1 x1, x2 bl __relocate_kernel #endif #endif From patchwork Wed Mar 30 15:41:55 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 555175 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8B168C433F5 for ; Wed, 30 Mar 2022 15:42:30 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1348357AbiC3PoO (ORCPT ); Wed, 30 Mar 2022 11:44:14 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55458 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1348355AbiC3PoN (ORCPT ); Wed, 30 Mar 2022 11:44:13 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4B8AD35AB8 for ; Wed, 30 Mar 2022 08:42:28 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id DE244616D1 for ; Wed, 30 Mar 2022 15:42:27 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id E8654C34114; Wed, 30 Mar 2022 15:42:25 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1648654947; bh=sBwYXDl/hoZE0hxvjqpVZXzIWdBo8d5kBq1sUC2nsKw=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=dNolZDKDy9F54pqyHqgpUg2bp0uFrkda8Vmf8yI0wjoo9ntPFWCfAnvpVr8qzmj1a EysDjTylz8GC+v8XdbCjOnucAnwRQeDTk/GbyUCFO/aPMgJNTj/gD0HMljYzgIXa2b lqB0xjTNvigcFPxaUemzW2HR119+PVY2TwmQ6pd4fd7G07VIgNNmyi2lcugOMKkRa5 n70FE/Cy7724IDM5iA7oH+J8LZBknnpW1UZOSVnHLSBZ0QdjxLk4S8d1vAj12ixJeE 1wNJe0AitR+wcsdaMY4xMFuktcL3tKR/qMwovHdWoGsZsTV4zchpg4IcaxteXfO8vE jvo5QEzKaEDpA== From: Ard Biesheuvel To: linux-efi@vger.kernel.org Cc: linux-arm-kernel@lists.infradead.org, keescook@chromium.org, mark.rutland@arm.com, catalin.marinas@arm.com, Ard Biesheuvel Subject: [RFC PATCH v2 08/18] arm64: setup: defer R/O remapping of FDT Date: Wed, 30 Mar 2022 17:41:55 +0200 Message-Id: <20220330154205.2483167-9-ardb@kernel.org> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220330154205.2483167-1-ardb@kernel.org> References: <20220330154205.2483167-1-ardb@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=2415; h=from:subject; bh=sBwYXDl/hoZE0hxvjqpVZXzIWdBo8d5kBq1sUC2nsKw=; b=owEB7QES/pANAwAKAcNPIjmS2Y8kAcsmYgBiRHo7LjHU/WqxAPJBoP1hgK1CAW+hGG/fCgWsnqJe XaOBABuJAbMEAAEKAB0WIQT72WJ8QGnJQhU3VynDTyI5ktmPJAUCYkR6OwAKCRDDTyI5ktmPJF7QC/ 0cKBuOKD5rPYxq9NXKCVEUttLdOprhElSpoGU9dCbudzJqintq2//ePXn8gMeknIn/7NNiR0FidaYc dbhaFqSS2+fcScZSC6+v/+iQJ+usGFcB+J92n7TUVn/3JaWsWpg6YqQhBHKXppGSglnEFmAphTA9jO ClYqbUlC4KVnk1XEgVLH73Z1aprg5Ev4ew/iEaQ8yoXTvQCwG1N5s7DtywfuxdbjnDrVuAv3k5wVdS CeAEpE0BpghGl52z3gPHKGrNCX+p2DtLnfECO8JOrFs512lbSDGcpfoO4NSiMQqbFo4+uOHcuO+JSV hchue5g7BikNqeVWowNMXy9/NPIAkkGAeA+BQ1q1q3sqL6+2QnNn3lyemR3U0X8BZXY0wLG1VzuZHe u11vuE8keFg92lg7l2GBBgdOrOUE062Nd1bDRhwc0gjlwDJxrpFOxPH77DLcMuRx8GgSoU1UjVLwsK b0djZUT65+NOa84pXt6KyJ0yYIUoat3EDX6szrxinbKUg= X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 Precedence: bulk List-ID: X-Mailing-List: linux-efi@vger.kernel.org We will be moving the call to kaslr_init() into setup_arch() in an upcoming patch, and this needs the FDT to be writable so the KASLR seed can be wiped from it. So break out the R/O remapping of the FDT from setup_machine_fdt() and call it explicitly from setup_arch(). Signed-off-by: Ard Biesheuvel --- arch/arm64/kernel/setup.c | 6 +++--- arch/arm64/mm/mmu.c | 12 +++++++----- 2 files changed, 10 insertions(+), 8 deletions(-) diff --git a/arch/arm64/kernel/setup.c b/arch/arm64/kernel/setup.c index 3505789cf4bd..ebf69312eabf 100644 --- a/arch/arm64/kernel/setup.c +++ b/arch/arm64/kernel/setup.c @@ -203,9 +203,6 @@ static void __init setup_machine_fdt(phys_addr_t dt_phys) cpu_relax(); } - /* Early fixups are done, map the FDT as read-only now */ - fixmap_remap_fdt(dt_phys, &size, PAGE_KERNEL_RO); - name = of_flat_dt_get_machine_name(); if (!name) return; @@ -316,6 +313,9 @@ void __init __no_sanitize_address setup_arch(char **cmdline_p) setup_machine_fdt(__fdt_pointer); + /* Early fixups are done, map the FDT as read-only now */ + fixmap_remap_fdt(__fdt_pointer, NULL, PAGE_KERNEL_RO); + /* * Initialise the static keys early as they may be enabled by the * cpufeature code and early parameters. diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index e74a6453cb14..20dd95a750bc 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -1324,7 +1324,7 @@ void __set_fixmap(enum fixed_addresses idx, void *__init fixmap_remap_fdt(phys_addr_t dt_phys, int *size, pgprot_t prot) { const u64 dt_virt_base = __fix_to_virt(FIX_FDT); - int offset; + int offset, dt_size; void *dt_virt; /* @@ -1363,13 +1363,15 @@ void *__init fixmap_remap_fdt(phys_addr_t dt_phys, int *size, pgprot_t prot) if (fdt_magic(dt_virt) != FDT_MAGIC) return NULL; - *size = fdt_totalsize(dt_virt); - if (*size > MAX_FDT_SIZE) + dt_size = fdt_totalsize(dt_virt); + if (size) + *size = dt_size; + if (dt_size > MAX_FDT_SIZE) return NULL; - if (offset + *size > SWAPPER_BLOCK_SIZE) + if (offset + dt_size > SWAPPER_BLOCK_SIZE) create_mapping_noalloc(round_down(dt_phys, SWAPPER_BLOCK_SIZE), dt_virt_base, - round_up(offset + *size, SWAPPER_BLOCK_SIZE), prot); + round_up(offset + dt_size, SWAPPER_BLOCK_SIZE), prot); return dt_virt; } From patchwork Wed Mar 30 15:41:57 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 555174 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id A9BA0C433EF for ; Wed, 30 Mar 2022 15:42:39 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236569AbiC3PoW (ORCPT ); Wed, 30 Mar 2022 11:44:22 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55918 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1348356AbiC3PoU (ORCPT ); Wed, 30 Mar 2022 11:44:20 -0400 Received: from ams.source.kernel.org (ams.source.kernel.org [145.40.68.75]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B774533E9E for ; Wed, 30 Mar 2022 08:42:33 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 6C9BEB81D69 for ; Wed, 30 Mar 2022 15:42:32 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id C7C4CC3410F; Wed, 30 Mar 2022 15:42:29 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1648654951; bh=VnakrP68jMnJ7VVKYfr1DkuBhjm1zYhf09fUkYU6tZU=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=kdgtQYJqkSbJnuPBFalasAzfOTUx1wAITPJ+pY/7TjyD3WxqwT8NKr4Yj8UilIgi9 lcq0SADqc60yQgYXXlo0dDOjowsJzRya1V+zFhjOfaMOYp19/Rdj7UhcsODSceMZm1 HGxJlU/etR1o3MddqJxs9Soq0bRxvUJBmoGD5/ew+tRhItA2t+EOwGL6V4BVLK7dmF Fu/rH/za9+DB+nMEfVNSDSzeq46knU8nBfQ8yc4b82Xdz+dWDp//WelDDu3dmA28DF 9IHyIVyi3c0AcmWAgQAwKIIrAGYuN8WybUByqJAsHKr6RrLVvaU3KFAOLwFBwiVHOX g7fJAEDqDGKaw== From: Ard Biesheuvel To: linux-efi@vger.kernel.org Cc: linux-arm-kernel@lists.infradead.org, keescook@chromium.org, mark.rutland@arm.com, catalin.marinas@arm.com, Ard Biesheuvel Subject: [RFC PATCH v2 10/18] arm64: head: record the MMU state at primary entry Date: Wed, 30 Mar 2022 17:41:57 +0200 Message-Id: <20220330154205.2483167-11-ardb@kernel.org> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220330154205.2483167-1-ardb@kernel.org> References: <20220330154205.2483167-1-ardb@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=2220; h=from:subject; bh=VnakrP68jMnJ7VVKYfr1DkuBhjm1zYhf09fUkYU6tZU=; b=owEB7QES/pANAwAKAcNPIjmS2Y8kAcsmYgBiRHo/CuCO1CDC3F1jfOi5U0g6OytEgCjUpRq73M4e itcfC8+JAbMEAAEKAB0WIQT72WJ8QGnJQhU3VynDTyI5ktmPJAUCYkR6PwAKCRDDTyI5ktmPJOhGC/ wLFqlyboSaMzIoukbL6OLiJ3OnHkaSboDffWF0+ubx9bTG+iFNXqYQBxNzf3yhQtTFwJxtc8LhsXG6 9ZlmIqPDh+G9/mxpBLNQ3ZqCA0h3t/nw4EXPBYxVOD+n1lkgTl2umjU6T8kgTqB5uOYjFLgPDBBO0a npXhBQ93uGI1oyGERO/CTulm6qIOm12P8QSi6t7KY891PVNWfYDD+7mz72ILJIkFHt+BINs6MzUmIq 38yKERoUjE051YcDAnE7wJO6agxlyGrv5kp+HXF+FQQuVEDoWoalGKdjtyzpeLKrGFb+nxTWpeRTs6 cx+GwH9bZj2TBp+xTjM7bNOkD9KyGW/IYdJn/A0+kCgunQGOJTfUr5fBQJ2A7VtJiPfSZ5AVKRH8Eh 2lJ93E0cdyn8owuQWclmXDFuUd5We1Mc/lO80HfFDXX6wwNFByoiRhkS6KhfByHceMspO5VLr+nJ2l mwwihh0Sjyk8tu9V0yB10BwYIPZ/1LXBW6GXLIPJmuZTw= X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 Precedence: bulk List-ID: X-Mailing-List: linux-efi@vger.kernel.org Prepare for being able to deal with primary entry with the MMU and caches enabled, by recording whether or not we entered with the MMU on in register x22. While at it, add disable_mmu_workaround macro invocations to init_kernel_el, as its manipulation of SCTLR_ELx may come down to disabling of the MMU after subsequent patches. Signed-off-by: Ard Biesheuvel --- arch/arm64/kernel/head.S | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S index f3b096daf1c5..44e2e39046a9 100644 --- a/arch/arm64/kernel/head.S +++ b/arch/arm64/kernel/head.S @@ -83,9 +83,11 @@ * * Register Scope Purpose * x21 primary_entry() .. start_kernel() FDT pointer passed at boot in x0 + * x22 primary_entry() .. start_kernel() whether we entered with the MMU on * x23 __primary_switch() .. relocate_kernel() physical misalignment/KASLR offset */ SYM_CODE_START(primary_entry) + bl record_mmu_state bl preserve_boot_args bl init_kernel_el // w0=cpu_boot_mode bl set_cpu_boot_mode_flag @@ -101,6 +103,17 @@ SYM_CODE_START(primary_entry) b __primary_switch SYM_CODE_END(primary_entry) +SYM_CODE_START_LOCAL(record_mmu_state) + mrs x22, CurrentEL + cmp x22, #CurrentEL_EL2 + mrs x22, sctlr_el1 + b.ne 0f + mrs x22, sctlr_el2 +0: tst x22, #SCTLR_ELx_M + cset w22, ne + ret +SYM_CODE_END(record_mmu_state) + /* * Preserve the arguments passed by the bootloader in x0 .. x3 */ @@ -485,6 +498,7 @@ SYM_FUNC_START(init_kernel_el) SYM_INNER_LABEL(init_el1, SYM_L_LOCAL) mov_q x0, INIT_SCTLR_EL1_MMU_OFF + pre_disable_mmu_workaround msr sctlr_el1, x0 isb mov_q x0, INIT_PSTATE_EL1 @@ -516,6 +530,7 @@ SYM_INNER_LABEL(init_el2, SYM_L_LOCAL) /* Switching to VHE requires a sane SCTLR_EL1 as a start */ mov_q x0, INIT_SCTLR_EL1_MMU_OFF + pre_disable_mmu_workaround msr_s SYS_SCTLR_EL12, x0 /* @@ -531,6 +546,7 @@ SYM_INNER_LABEL(init_el2, SYM_L_LOCAL) 1: mov_q x0, INIT_SCTLR_EL1_MMU_OFF + pre_disable_mmu_workaround msr sctlr_el1, x0 msr elr_el2, lr From patchwork Wed Mar 30 15:41:59 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 555172 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 68E8AC433EF for ; Wed, 30 Mar 2022 15:42:42 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1348362AbiC3PoZ (ORCPT ); Wed, 30 Mar 2022 11:44:25 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56042 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1348370AbiC3PoV (ORCPT ); Wed, 30 Mar 2022 11:44:21 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 16C443615F for ; Wed, 30 Mar 2022 08:42:35 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 6F36261729 for ; Wed, 30 Mar 2022 15:42:35 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 79648C340F2; Wed, 30 Mar 2022 15:42:33 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1648654954; bh=NKARMdFIpPuUO+fngR3uplP5bKMzLEZRMWLNCU6j1kE=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=R8Rd9RJut7Nn9lpz7AetrlSRRW8rA+5kZYVD+70C+DVhrLO19BNe9/EKVVgRSCQqQ a36mO0etxVjm0FUUEP2r5YkfAq7/CwaCWC7hrvSWLyGpvTqy3nAurrJLt4CiZ4VC+D I4jx3XbZSzmqwX5Cxwk1sO5Loqb2zWtswPQSi0z4LNwZrS2otIp+mRQe0vF45O7BD5 2pHuvh+6BNOM/ani7lzBI+jkzYg2NGvumlSkXqjmP9stSvIWaxYqH+dXK3F5gh0Adm 4Vt/X6lbxETjACk5U3vx6uz1iB5QW20nAF8PJa6vnNvfspgTeO0+5UsqTyZCMuPIc9 0o3zXiMB5hHxw== From: Ard Biesheuvel To: linux-efi@vger.kernel.org Cc: linux-arm-kernel@lists.infradead.org, keescook@chromium.org, mark.rutland@arm.com, catalin.marinas@arm.com, Ard Biesheuvel Subject: [RFC PATCH v2 12/18] arm64: head: avoid cache invalidation when entering with the MMU on Date: Wed, 30 Mar 2022 17:41:59 +0200 Message-Id: <20220330154205.2483167-13-ardb@kernel.org> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220330154205.2483167-1-ardb@kernel.org> References: <20220330154205.2483167-1-ardb@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=2876; h=from:subject; bh=NKARMdFIpPuUO+fngR3uplP5bKMzLEZRMWLNCU6j1kE=; b=owEB7QES/pANAwAKAcNPIjmS2Y8kAcsmYgBiRHpCqqsO4VFBa2X4f5zJeq/GZgYbbyd4r4kj7Kwu SyJa3WOJAbMEAAEKAB0WIQT72WJ8QGnJQhU3VynDTyI5ktmPJAUCYkR6QgAKCRDDTyI5ktmPJO4LDA CNR59DspwJN3JhnhbtoSuLvXaz04oPBpP7bhp8CgIzzQ4RE7flSCs6C5xdA+Zuz6XrMir0r7+wgroG YvynNADmkE1gcQJWMQ0E2a8vrMoiYr5U2GQ0I95cEnITN1WCR2vbUNtH4GHGx3cs7Q87lKLFmYwg9i CZJvnOoFkYJO9g7IJ9xPlNxAWGZTDRVu9WKm1w6opMnCVRqmF8yWx9ytRrJIiU/A9FHgbveayokBXW AZYN9jQuR8qK4ZMaaM2MN9Y/ZWFFxwii51jYk8k4OhjUOj+pLmIVzLmgPNE6hpffcp0OWpLuwifu7O pNZw2+8e5E34JYzu21A2tzO2VFb50v1qjHhYDrwxWcXN705iWbhXEyXgjFt1SpD/9YUdK6i6/dCOLR LqEMgq1UU5yq+3tePK2KNSqGOZtPhQlBCS9BZ2U3FwJUVpY/l6/WPVex3/yKQdFPM6FlbI02WOviae pCVSjBWNnrb9FhqtKpEPhKqotNzk24uLGWWZRYURA8uYM= X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 Precedence: bulk List-ID: X-Mailing-List: linux-efi@vger.kernel.org If we enter with the MMU on, there is no need for explicit cache invalidation for stores to memory, as they will be coherent with the caches. Let's take advantage of this, and create the ID map with the MMU still enabled if that is how we entered, and avoid any cache invalidation calls in that case. Signed-off-by: Ard Biesheuvel --- arch/arm64/kernel/head.S | 15 +++++++++++---- 1 file changed, 11 insertions(+), 4 deletions(-) diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S index 836237289ffb..db315129f15d 100644 --- a/arch/arm64/kernel/head.S +++ b/arch/arm64/kernel/head.S @@ -89,9 +89,9 @@ SYM_CODE_START(primary_entry) bl record_mmu_state bl preserve_boot_args + bl create_idmap bl init_kernel_el // w0=cpu_boot_mode bl set_cpu_boot_mode_flag - bl create_idmap /* * The following calls CPU setup code, see arch/arm64/mm/proc.S for @@ -124,11 +124,13 @@ SYM_CODE_START_LOCAL(preserve_boot_args) stp x21, x1, [x0] // x0 .. x3 at kernel entry stp x2, x3, [x0, #16] + cbnz x22, 0f // skip cache invalidation if MMU is on dmb sy // needed before dc ivac with // MMU off add x1, x0, #0x20 // 4 x 8 bytes b dcache_inval_poc // tail call +0: ret SYM_CODE_END(preserve_boot_args) SYM_FUNC_START_LOCAL(clear_page_tables) @@ -292,8 +294,10 @@ SYM_FUNC_START_LOCAL(create_idmap) 1: adr_l x6, vabits_actual str x5, [x6] + cbnz x22, 2f // skip cache invalidation if MMU is on dmb sy dc ivac, x6 // Invalidate potentially stale cache line +2: #endif /* * VA_BITS may be too small to allow for an ID mapping to be created @@ -311,13 +315,14 @@ SYM_FUNC_START_LOCAL(create_idmap) adrp x5, _end clz x5, x5 cmp x5, TCR_T0SZ(VA_BITS_MIN) // default T0SZ small enough? - b.ge 1f // .. then skip VA range extension + b.ge 4f // .. then skip VA range extension adr_l x6, idmap_t0sz str x5, [x6] + cbnz x22, 3f // skip cache invalidation if MMU is on dmb sy dc ivac, x6 // Invalidate potentially stale cache line - +3: #if (VA_BITS < 48) #define EXTRA_SHIFT (PGDIR_SHIFT + PAGE_SHIFT - 3) #define EXTRA_PTRS (1 << (PHYS_MASK_SHIFT - EXTRA_SHIFT)) @@ -343,7 +348,7 @@ SYM_FUNC_START_LOCAL(create_idmap) */ mov x4, #1 << (PHYS_MASK_SHIFT - PGDIR_SHIFT) #endif -1: +4: adr_l x6, _end // __pa(_end) mov x7, SWAPPER_MM_MMUFLAGS @@ -354,11 +359,13 @@ SYM_FUNC_START_LOCAL(create_idmap) * accesses (MMU disabled), invalidate those tables again to * remove any speculatively loaded cache lines. */ + cbnz x22, 5f // skip cache invalidation if MMU is on dmb sy adrp x0, idmap_pg_dir adrp x1, idmap_pg_end b dcache_inval_poc // tail call +5: ret SYM_FUNC_END(create_idmap) SYM_FUNC_START_LOCAL(create_kernel_mapping) From patchwork Wed Mar 30 15:42:00 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 555173 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0FE63C4332F for ; Wed, 30 Mar 2022 15:42:41 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1348358AbiC3PoY (ORCPT ); Wed, 30 Mar 2022 11:44:24 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56044 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1348363AbiC3PoX (ORCPT ); Wed, 30 Mar 2022 11:44:23 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B100337A33 for ; Wed, 30 Mar 2022 08:42:37 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 4F83461727 for ; Wed, 30 Mar 2022 15:42:37 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 51DBBC340F0; Wed, 30 Mar 2022 15:42:35 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1648654956; bh=AsUy16QUY67g3cS9+sxlqLA5RhJmuo9X0eKR6mPfwig=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=sc++pzOgPSOxWcZ51a1lE51OLcOiq9zPJcM+d258YEHKBpggGgk+mBNxgHtTAgNm+ arrqjO3CkZOyEqvzc/MTFPZmwTjhmdMjGyOCata8fWUYQdkKyiZWPZncxIr4/6RMc7 VXg/AIi9fRhIvUZ62XynQ07EbnMgqwPjZE44bgnvBePhPxZ2fCxjH4iAl0KlneH9sd pww1NhxwtaygJX3jdNAsB6+bvVSxXg21oLsG0/zvmMpkofBbczWGMAe8CqMMf1xY9j nkdDXq/y8aMjU0CO345P44hba+p/aSzJBF3CcPJ7+X3/wBi+Kv0wUfigxM4WL9rgUD HhMeMYrHJELtQ== From: Ard Biesheuvel To: linux-efi@vger.kernel.org Cc: linux-arm-kernel@lists.infradead.org, keescook@chromium.org, mark.rutland@arm.com, catalin.marinas@arm.com, Ard Biesheuvel Subject: [RFC PATCH v2 13/18] arm64: head: record CPU boot mode after enabling the MMU Date: Wed, 30 Mar 2022 17:42:00 +0200 Message-Id: <20220330154205.2483167-14-ardb@kernel.org> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220330154205.2483167-1-ardb@kernel.org> References: <20220330154205.2483167-1-ardb@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=2705; h=from:subject; bh=AsUy16QUY67g3cS9+sxlqLA5RhJmuo9X0eKR6mPfwig=; b=owEB7QES/pANAwAKAcNPIjmS2Y8kAcsmYgBiRHpEqXtN+pBQ/DQKM/BAqICQqlF1Cql9agBDHxba aB1kIWSJAbMEAAEKAB0WIQT72WJ8QGnJQhU3VynDTyI5ktmPJAUCYkR6RAAKCRDDTyI5ktmPJItgDA C9DxMVEU5LifGT1wvhh15YlNaqztd3h3VkMMG/Qb9WXtg4zqafbH/uwtRO9S87KYIFkuPBh399oKLI uzwn//fDmT5gI2Fo6QVxBh+1kO9qzYd4UjOYuMtlESha/Mkc5hfTbmgsiR0qDNWLGJec6s11Lre0EN QYc88PBu9Oazx34VQeEgSQeXrFeFpt/zYiQEe4dtscIbylYlHAYEdQ9JiSf5awKP0Q/Pt8XLjnihHh lNn14Cj4gPoY4C7yPfQLIp2WexswvnMuwQU2pazJG8qk3sCZmnlQ3UsvrMmWfmQkDc+9LtfJysvAjd aqXBaLCoVQKs5dVi/3YXTkSwut3w4/zEhUoyPrAsKyKO5efpZzHz6JQVj+/Qk4OwTtKw4Xp60pEVFJ 8ffy7vKz0cHT/ZnB5cVGzivxGhf8BKAjbZVsc9LH08omBbGJ5ZI58XAedj5vLBllFc00ZHxxRuIIg9 3U3bKzqPzBS0ZZa8v3ZAvS2APpcXrzcIQsF1zH+tMeFcU= X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 Precedence: bulk List-ID: X-Mailing-List: linux-efi@vger.kernel.org In order to avoid having to touch memory with the MMU and caches disabled, and therefore having to invalidate it from the caches explicitly, just defer storing the boot mode until after the MMU has been turned on. Signed-off-by: Ard Biesheuvel --- arch/arm64/kernel/head.S | 16 ++++++++++------ 1 file changed, 10 insertions(+), 6 deletions(-) diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S index db315129f15d..ec57a29f3f43 100644 --- a/arch/arm64/kernel/head.S +++ b/arch/arm64/kernel/head.S @@ -81,7 +81,8 @@ * The following callee saved general purpose registers are used on the * primary lowlevel boot path: * - * Register Scope Purpose + * Register Scope Purpose + * x20 primary_entry() .. __primary_switch() CPU boot mode * x21 primary_entry() .. start_kernel() FDT pointer passed at boot in x0 * x22 primary_entry() .. start_kernel() whether we entered with the MMU on * x23 __primary_switch() .. relocate_kernel() physical misalignment/KASLR offset @@ -91,7 +92,7 @@ SYM_CODE_START(primary_entry) bl preserve_boot_args bl create_idmap bl init_kernel_el // w0=cpu_boot_mode - bl set_cpu_boot_mode_flag + mov x20, x0 /* * The following calls CPU setup code, see arch/arm64/mm/proc.S for @@ -576,8 +577,6 @@ SYM_FUNC_START_LOCAL(set_cpu_boot_mode_flag) b.ne 1f add x1, x1, #4 1: str w0, [x1] // Save CPU boot mode - dmb sy - dc ivac, x1 // Invalidate potentially stale cache line ret SYM_FUNC_END(set_cpu_boot_mode_flag) @@ -615,7 +614,7 @@ SYM_DATA_END(__early_cpu_boot_status) */ SYM_FUNC_START(secondary_holding_pen) bl init_kernel_el // w0=cpu_boot_mode - bl set_cpu_boot_mode_flag + mov x20, x0 mrs x0, mpidr_el1 mov_q x1, MPIDR_HWID_BITMASK and x0, x0, x1 @@ -633,7 +632,7 @@ SYM_FUNC_END(secondary_holding_pen) */ SYM_FUNC_START(secondary_entry) bl init_kernel_el // w0=cpu_boot_mode - bl set_cpu_boot_mode_flag + mov x20, x0 b secondary_startup SYM_FUNC_END(secondary_entry) @@ -646,6 +645,8 @@ SYM_FUNC_START_LOCAL(secondary_startup) bl __cpu_setup // initialise processor adrp x1, swapper_pg_dir bl __enable_mmu + mov x0, x20 + bl set_cpu_boot_mode_flag ldr x8, =__secondary_switched br x8 SYM_FUNC_END(secondary_startup) @@ -861,6 +862,9 @@ SYM_FUNC_START_LOCAL(__primary_switch) bl __enable_mmu bl clear_page_tables + mov x0, x20 + bl set_cpu_boot_mode_flag + #ifdef CONFIG_RELOCATABLE adrp x23, __PHYS_OFFSET and x23, x23, MIN_KIMG_ALIGN - 1 From patchwork Wed Mar 30 15:42:03 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 555171 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 345D8C433EF for ; Wed, 30 Mar 2022 15:42:47 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1348364AbiC3Pob (ORCPT ); Wed, 30 Mar 2022 11:44:31 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56714 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1348366AbiC3Poa (ORCPT ); Wed, 30 Mar 2022 11:44:30 -0400 Received: from ams.source.kernel.org (ams.source.kernel.org [145.40.68.75]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DCD3633EAA for ; Wed, 30 Mar 2022 08:42:44 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 8CB50B81D6E for ; Wed, 30 Mar 2022 15:42:43 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id D1273C34114; Wed, 30 Mar 2022 15:42:40 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1648654962; bh=1CikZ2pxQ07LdCOKvbfGEUejKhtp/ShmFjaqIdNlSrM=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=t7qqqBkDqWwgnhKrpIhK1f7Mlu9bU/Bs01mF9513Q3GzaQ5500NmSbt44FeqbNoD4 AMyyk0MLNc8020bWm4tOpRJcjhedZ+rNwDn2+WYSiKlHljjjUgWQHu9Fpvy9fZH07J Btcu8RV/0c7OYhpOkr8R1hXnLne1Ch23rzkxXG0o5dwa2ooDCbaVooLgi8UoHeMYvw XTetGnooHi/ZnH8eRXx/sFiK/Rzb2kBIMtX2x5+11v08TwygVAukHkwHXMFzsmT7Ft YLpEk5tIxFTC59Ub9SKWat509BnnR3Q1XIgNLuEzo5ulDb8B1mGGsjLyQUaeHEvYqx tvAKi5KcP8O9w== From: Ard Biesheuvel To: linux-efi@vger.kernel.org Cc: linux-arm-kernel@lists.infradead.org, keescook@chromium.org, mark.rutland@arm.com, catalin.marinas@arm.com, Ard Biesheuvel Subject: [RFC PATCH v2 16/18] efi: libstub: pass image handle to handle_kernel_image() Date: Wed, 30 Mar 2022 17:42:03 +0200 Message-Id: <20220330154205.2483167-17-ardb@kernel.org> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220330154205.2483167-1-ardb@kernel.org> References: <20220330154205.2483167-1-ardb@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=3709; h=from:subject; bh=1CikZ2pxQ07LdCOKvbfGEUejKhtp/ShmFjaqIdNlSrM=; b=owEB7QES/pANAwAKAcNPIjmS2Y8kAcsmYgBiRHpJZUSNQ0i79wjfMTAXX35/YMEC9rq9fAqQAUIR lU19mgKJAbMEAAEKAB0WIQT72WJ8QGnJQhU3VynDTyI5ktmPJAUCYkR6SQAKCRDDTyI5ktmPJBrzC/ 9NQzeLdD6VWj4JurFsTuMeHrzidzBJPJePxmiVRuswOgN338vQD6DXeBURklOftjkahHz8qvlT9GBs vZTQR+a7f9HzBagn5pJWexbstuN48i6tEbCkTQZQlihXGaixA4l18gAQGktg/QRVU44Gu0x4FlPTqV ToH30Q1wz+HGU4GHeZEBEDiiAr5gwshvngdvpKRX43Gj48SfIldYHNN+C+boVILiHA7oV0j6dhX8OC ZLisY/mTySnoUes3giKiQSRm8VX2c1H6DtpCl+6/l4TgwM7fnI9T8EMkQvOKJx9O9k6X5msz4uDpMz yoZn+/Sk8aGaMW+MebCg7+RiS+1bfMgmZhId7SMwIsXuonBXJ9RPaj8sDVZUoA1pg6O0Y+UktPx0Pc G4Q50G9ZS7xJxD1olq92LSW6UpCCuKA3C3JFk8CIdVYSY9KVSvT2ofRCbJ3To4hln9c5SQIt1r79YE XBjdxFKVUtiHbTYbQOV0LkK27KUgh073ApDrnDVZglVbQ= X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 Precedence: bulk List-ID: X-Mailing-List: linux-efi@vger.kernel.org In a future patch, arm64's implementation of handle_kernel_image() will omit randomizing the placement of the kernel if the load address was chosen randomly by the loader. In order to do this, it needs to locate a protocol on the image handle, so pass it to handle_kernel_image(). Signed-off-by: Ard Biesheuvel --- drivers/firmware/efi/libstub/arm32-stub.c | 3 ++- drivers/firmware/efi/libstub/arm64-stub.c | 3 ++- drivers/firmware/efi/libstub/efi-stub.c | 2 +- drivers/firmware/efi/libstub/efistub.h | 3 ++- drivers/firmware/efi/libstub/riscv-stub.c | 3 ++- 5 files changed, 9 insertions(+), 5 deletions(-) diff --git a/drivers/firmware/efi/libstub/arm32-stub.c b/drivers/firmware/efi/libstub/arm32-stub.c index 4b5b2403b3a0..0131e3aaa605 100644 --- a/drivers/firmware/efi/libstub/arm32-stub.c +++ b/drivers/firmware/efi/libstub/arm32-stub.c @@ -117,7 +117,8 @@ efi_status_t handle_kernel_image(unsigned long *image_addr, unsigned long *image_size, unsigned long *reserve_addr, unsigned long *reserve_size, - efi_loaded_image_t *image) + efi_loaded_image_t *image, + efi_handle_t image_handle) { const int slack = TEXT_OFFSET - 5 * PAGE_SIZE; int alloc_size = MAX_UNCOMP_KERNEL_SIZE + EFI_PHYS_ALIGN; diff --git a/drivers/firmware/efi/libstub/arm64-stub.c b/drivers/firmware/efi/libstub/arm64-stub.c index 9cc556013d08..00c91a3807ea 100644 --- a/drivers/firmware/efi/libstub/arm64-stub.c +++ b/drivers/firmware/efi/libstub/arm64-stub.c @@ -83,7 +83,8 @@ efi_status_t handle_kernel_image(unsigned long *image_addr, unsigned long *image_size, unsigned long *reserve_addr, unsigned long *reserve_size, - efi_loaded_image_t *image) + efi_loaded_image_t *image, + efi_handle_t image_handle) { efi_status_t status; unsigned long kernel_size, kernel_memsize = 0; diff --git a/drivers/firmware/efi/libstub/efi-stub.c b/drivers/firmware/efi/libstub/efi-stub.c index da93864d7abc..f515394cce6e 100644 --- a/drivers/firmware/efi/libstub/efi-stub.c +++ b/drivers/firmware/efi/libstub/efi-stub.c @@ -198,7 +198,7 @@ efi_status_t __efiapi efi_pe_entry(efi_handle_t handle, status = handle_kernel_image(&image_addr, &image_size, &reserve_addr, &reserve_size, - image); + image, handle); if (status != EFI_SUCCESS) { efi_err("Failed to relocate kernel\n"); goto fail_free_screeninfo; diff --git a/drivers/firmware/efi/libstub/efistub.h b/drivers/firmware/efi/libstub/efistub.h index edb77b0621ea..c4f4f078087d 100644 --- a/drivers/firmware/efi/libstub/efistub.h +++ b/drivers/firmware/efi/libstub/efistub.h @@ -865,7 +865,8 @@ efi_status_t handle_kernel_image(unsigned long *image_addr, unsigned long *image_size, unsigned long *reserve_addr, unsigned long *reserve_size, - efi_loaded_image_t *image); + efi_loaded_image_t *image, + efi_handle_t image_handle); asmlinkage void __noreturn efi_enter_kernel(unsigned long entrypoint, unsigned long fdt_addr, diff --git a/drivers/firmware/efi/libstub/riscv-stub.c b/drivers/firmware/efi/libstub/riscv-stub.c index 9c460843442f..eec043873354 100644 --- a/drivers/firmware/efi/libstub/riscv-stub.c +++ b/drivers/firmware/efi/libstub/riscv-stub.c @@ -80,7 +80,8 @@ efi_status_t handle_kernel_image(unsigned long *image_addr, unsigned long *image_size, unsigned long *reserve_addr, unsigned long *reserve_size, - efi_loaded_image_t *image) + efi_loaded_image_t *image, + efi_handle_t image_handle) { unsigned long kernel_size = 0; unsigned long preferred_addr; From patchwork Wed Mar 30 15:42:04 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 555170 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 817A5C433FE for ; Wed, 30 Mar 2022 15:42:49 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1348366AbiC3Pod (ORCPT ); Wed, 30 Mar 2022 11:44:33 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56914 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1348368AbiC3Poc (ORCPT ); Wed, 30 Mar 2022 11:44:32 -0400 Received: from sin.source.kernel.org (sin.source.kernel.org [IPv6:2604:1380:40e1:4800::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4FED534B91 for ; Wed, 30 Mar 2022 08:42:47 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sin.source.kernel.org (Postfix) with ESMTPS id C39F7CE1D4F for ; Wed, 30 Mar 2022 15:42:45 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id AA1E1C340EC; Wed, 30 Mar 2022 15:42:42 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1648654964; bh=nY5MgIoq4fA9USqK31dnu7E3Bd7d2UqYghLsLSuHjOs=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=PVNx9yDboKjpdbBoDAyyWEf14Ym8nauJXpH4+srvJIyJ/Mp1vZhuwIaq4dW3IRd6a 8uQpOkSKq8JniYsgFcJF6iymtSsg8EhyfXXc21V1xqSMyJaOZ3Poj93oZH0pJRmunC pqn7uZSp7iWm8T52TAdEfb/sdBsJPeY3MKnM+xpgN9YMHFU+QCv4ALwl/8tnQ7fDI8 moa8pglsHOk0BMYAZNU0vzlDG4fkAWlg0/SIRa19v0CQDZ+RiLBkITJFfxotTn+UCs gv1yK4i2IDJIJk6RbQHsjOgxi2in0z6omFbDGaCKm+fDbh7WP2oExngy1PmWVji2ng wJp1m9HTzT0mw== From: Ard Biesheuvel To: linux-efi@vger.kernel.org Cc: linux-arm-kernel@lists.infradead.org, keescook@chromium.org, mark.rutland@arm.com, catalin.marinas@arm.com, Ard Biesheuvel Subject: [RFC PATCH v2 17/18] efi/arm64: libstub: run image in place if randomized by the loader Date: Wed, 30 Mar 2022 17:42:04 +0200 Message-Id: <20220330154205.2483167-18-ardb@kernel.org> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220330154205.2483167-1-ardb@kernel.org> References: <20220330154205.2483167-1-ardb@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=2987; h=from:subject; bh=nY5MgIoq4fA9USqK31dnu7E3Bd7d2UqYghLsLSuHjOs=; b=owEB7QES/pANAwAKAcNPIjmS2Y8kAcsmYgBiRHpKzSWWWLuFC4/Pg5EMCFfx8VQCtcysoeuYuZ+r VfZf23yJAbMEAAEKAB0WIQT72WJ8QGnJQhU3VynDTyI5ktmPJAUCYkR6SgAKCRDDTyI5ktmPJCm3C/ 0eCxz2zED1vCT8/NEwbBohrpW33i7Rjl/8u+X/NyoIGreLpx7UOIPFX4xGtd56C+LvDxMO5vHUp6L/ ZH8cOhk39bRdBD4Mcv4chuBVyGwui1rGLlpytJbGaLaCYaBc2G5TtuVtckJ370UX8auuzCuTvMV7i6 4S9RIl8aqtfgDaC8NGE4d08ujYa2Yr0sbugmx/SuYQ38yD73gSGaz8MiOUIlCI0LePgXuj+W1C6DnO JTO+8TbQooGrXwbDsb1MMC+c4XUYoZEaDkz9YwzHaU1kD7l8NRjlfiF7LsYvDcDpXkTXw8GWKnWaCS i1n4pmGSr+6KE5gkUUyPxE+8BKS83gSinh0LBgv46SckZF3uDqphRVRZykQWUghoeHIvPxzpRE/+7v EhftGRFxp4K0XtPIzCy8srG+2cTp+hx/oujlAYce/o3EBrGa9dMgJVoAzZoEElHDtwSSfYBWnRFObG CFiE/x00VLme2RLdkUSVKm3b/pkDj3drJdHhmjmPk3wM0= X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 Precedence: bulk List-ID: X-Mailing-List: linux-efi@vger.kernel.org If the loader has already placed the EFI kernel image randomly in physical memory, and indicates having done so by installing the 'fixed placement' protocol onto the image handle, don't bother randomizing the placement again in the EFI stub. Signed-off-by: Ard Biesheuvel --- drivers/firmware/efi/libstub/arm64-stub.c | 12 +++++++++--- include/linux/efi.h | 11 +++++++++++ 2 files changed, 20 insertions(+), 3 deletions(-) diff --git a/drivers/firmware/efi/libstub/arm64-stub.c b/drivers/firmware/efi/libstub/arm64-stub.c index 00c91a3807ea..577173ee1f83 100644 --- a/drivers/firmware/efi/libstub/arm64-stub.c +++ b/drivers/firmware/efi/libstub/arm64-stub.c @@ -101,7 +101,15 @@ efi_status_t handle_kernel_image(unsigned long *image_addr, u64 min_kimg_align = efi_nokaslr ? MIN_KIMG_ALIGN : EFI_KIMG_ALIGN; if (IS_ENABLED(CONFIG_RANDOMIZE_BASE)) { - if (!efi_nokaslr) { + efi_guid_t li_fixed_proto = LINUX_EFI_LOADED_IMAGE_FIXED_GUID; + void *p; + + if (efi_nokaslr) { + efi_info("KASLR disabled on kernel command line\n"); + } else if (efi_bs_call(handle_protocol, image_handle, + &li_fixed_proto, &p) == EFI_SUCCESS) { + efi_info("Image placement fixed by loader\n"); + } else { status = efi_get_random_bytes(sizeof(phys_seed), (u8 *)&phys_seed); if (status == EFI_NOT_FOUND) { @@ -112,8 +120,6 @@ efi_status_t handle_kernel_image(unsigned long *image_addr, status); efi_nokaslr = true; } - } else { - efi_info("KASLR disabled on kernel command line\n"); } } diff --git a/include/linux/efi.h b/include/linux/efi.h index ccd4d3f91c98..d7567006e151 100644 --- a/include/linux/efi.h +++ b/include/linux/efi.h @@ -406,6 +406,17 @@ void efi_native_runtime_setup(void); #define LINUX_EFI_INITRD_MEDIA_GUID EFI_GUID(0x5568e427, 0x68fc, 0x4f3d, 0xac, 0x74, 0xca, 0x55, 0x52, 0x31, 0xcc, 0x68) #define LINUX_EFI_MOK_VARIABLE_TABLE_GUID EFI_GUID(0xc451ed2b, 0x9694, 0x45d3, 0xba, 0xba, 0xed, 0x9f, 0x89, 0x88, 0xa3, 0x89) +/* + * This GUID may be installed onto the kernel image's handle as a NULL protocol + * to signal to the stub that the placement of the image should be respected, + * and moving the image in physical memory is undesirable. To ensure + * compatibility with 64k pages kernels with virtually mapped stacks, and to + * avoid defeating physical randomization, this protocol should only be + * installed if the image was placed at a randomized 128k aligned address in + * memory. + */ +#define LINUX_EFI_LOADED_IMAGE_FIXED_GUID EFI_GUID(0xf5a37b6d, 0x3344, 0x42a5, 0xb6, 0xbb, 0x97, 0x86, 0x48, 0xc1, 0x89, 0x0a) + /* OEM GUIDs */ #define DELLEMC_EFI_RCI2_TABLE_GUID EFI_GUID(0x2d9f28a2, 0xa886, 0x456a, 0x97, 0xa8, 0xf1, 0x1e, 0xf2, 0x4f, 0xf4, 0x55) #define AMD_SEV_MEM_ENCRYPT_GUID EFI_GUID(0x0cf29b71, 0x9e51, 0x433a, 0xa3, 0xb7, 0x81, 0xf3, 0xab, 0x16, 0xb8, 0x75)