From patchwork Thu Nov 22 20:47:18 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Julien Grall X-Patchwork-Id: 151828 Delivered-To: patch@linaro.org Received: by 2002:a2e:299d:0:0:0:0:0 with SMTP id p29-v6csp1202557ljp; Thu, 22 Nov 2018 12:49:30 -0800 (PST) X-Google-Smtp-Source: AJdET5erzQT2wCDCPVZdaDspLmwHEtogB2eyWxU+7NrzFKrXHQZxqGJ81wxRfSzxhO0Ow1eMxufU X-Received: by 2002:a0d:c887:: with SMTP id k129-v6mr13088014ywd.485.1542919770699; Thu, 22 Nov 2018 12:49:30 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1542919770; cv=none; d=google.com; s=arc-20160816; b=F9V34Trpl5+N7o3q0iawrw/C177wmjv3eFkNzo0vWLF8wGrMRXYOHIXX4bCYSiC1oT 6emPfrG4TJC8tRYWL1YL9DvjNuSKsQ4KLK4NSDR4ol7g6BxMDvE76xItWsi9+dZJHOm5 L0w3BnBh0PM8idL6vr2g6b9I+2F4jJYTwe81sU2prndeYBGxondX1IVdxI4Z3Op/oSmu gKukFttWbTSJ85kK4IF57W4i4m8Ikw5tuBlGGMYyk2YDoTIxjKKT7LR0rBWTlGOPm4kN ATKzpuYXqtnTCSWAwXfHGQ8kEis0v/nJekzr28nfpYyLyIpV5YvFW6OjxhbCk5YKPw8v RREQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:mime-version:cc :list-subscribe:list-help:list-post:list-unsubscribe:list-id :precedence:subject:message-id:date:to:from; bh=C2A69q3riJFymZWeaTcXmcXmwyhuvm8ZWW4Vrbaulv0=; b=D3ysa2dS8COGdY52QyuuqzxU40SH/W6CfMVOdsq8ApQGAKJ0ERmCUvgEZ8LNbXjlzd qGsrAzqEBpdvrb8Sxq2L1pkFMkngOvCEQsOi308Ig/yGrQ3F7+jzErxuDpyyJ78t8aPf 0xgoTsadc+1UaxmTmNlNTe4kD1w9WmO6OdNzUX5EEMoAABYkVCaNksEk2qJk3yL8Yiqd w9K0srn4cS9LpSBT6Wxda79ts3tpDiA9uC1T1BJ1PuvubSWFoxzNqQihW33i+DU08iq7 QJRb6bu+Ajh+VbppVc5aiTAoZoTG16IFQ6XsISng5SpJiXXrIBLgO8IAfhDXq4KeEp07 2hDQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of xen-devel-bounces@lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Return-Path: Received: from lists.xenproject.org (lists.xenproject.org. [192.237.175.120]) by mx.google.com with ESMTPS id q9-v6si31827840ywq.282.2018.11.22.12.49.30 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Thu, 22 Nov 2018 12:49:30 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of xen-devel-bounces@lists.xenproject.org designates 192.237.175.120 as permitted sender) client-ip=192.237.175.120; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of xen-devel-bounces@lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.89) (envelope-from ) id 1gPvsl-0003JC-0Z; Thu, 22 Nov 2018 20:47:27 +0000 Received: from us1-rack-dfw2.inumbo.com ([104.130.134.6]) by lists.xenproject.org with esmtp (Exim 4.89) (envelope-from ) id 1gPvsj-0003J5-TF for xen-devel@lists.xen.org; Thu, 22 Nov 2018 20:47:25 +0000 X-Inumbo-ID: d1361350-ee97-11e8-9a16-bc764e045a96 Received: from foss.arm.com (unknown [217.140.101.70]) by us1-rack-dfw2.inumbo.com (Halon) with ESMTP id d1361350-ee97-11e8-9a16-bc764e045a96; Thu, 22 Nov 2018 20:47:25 +0000 (UTC) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 10B8F32AD; Thu, 22 Nov 2018 12:47:25 -0800 (PST) Received: from e108454-lin.cambridge.arm.com (e108454-lin.cambridge.arm.com [10.1.196.50]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 005F63F575; Thu, 22 Nov 2018 12:47:23 -0800 (PST) From: Julien Grall To: xen-devel@lists.xen.org Date: Thu, 22 Nov 2018 20:47:18 +0000 Message-Id: <20181122204718.30972-1-julien.grall@arm.com> X-Mailer: git-send-email 2.11.0 Subject: [Xen-devel] [PATCH] xen/arm: Stop relocating Xen X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Cc: andre.przywara@arm.com, Julien Grall , sstabellini@kernel.org MIME-Version: 1.0 Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" At the moment, Xen is relocated towards the end of the memory. While this has the advantage to free space in low memory, the code is not compliant with the break-before-make because it requires to switch between two sets of page-table. This is not entirely trivial to fix as it would require us to go through an identity mapping and disabling MMU. Furtheremore, it looks like that some platform (such as the Hikey960) may not be able to bring-up secondary CPUs if the entry is too high. I don't believe the low memory is an issue because Xen is quite tiny (< 2MB). So the best solution is to stop relocating Xen. This has the advantage to simplify the code. Note that the break-before-make issue is not fixed by this patch. Signed-off-by: Julien Grall --- xen/arch/arm/arm32/head.S | 54 +++------------------------------------ xen/arch/arm/arm64/head.S | 50 +++++------------------------------- xen/arch/arm/mm.c | 22 +++++----------- xen/arch/arm/setup.c | 65 +++-------------------------------------------- xen/include/asm-arm/mm.h | 2 +- 5 files changed, 19 insertions(+), 174 deletions(-) diff --git a/xen/arch/arm/arm32/head.S b/xen/arch/arm/arm32/head.S index 93b51e9ef2..390a505e05 100644 --- a/xen/arch/arm/arm32/head.S +++ b/xen/arch/arm/arm32/head.S @@ -469,58 +469,12 @@ fail: PRINT("- Boot failed -\r\n") GLOBAL(_end_boot) /* - * Copy Xen to new location and switch TTBR + * Switch TTBR * r1:r0 ttbr - * r2 source address - * r3 destination address - * [sp]=>r4 length * - * Source and destination must be word aligned, length is rounded up - * to a 16 byte boundary. - * - * MUST BE VERY CAREFUL when saving things to RAM over the copy + * TODO: This code does not comply with break-before-make. */ -ENTRY(relocate_xen) - push {r4,r5,r6,r7,r8,r9,r10,r11} - - ldr r4, [sp, #8*4] /* Get 4th argument from stack */ - - /* Copy 16 bytes at a time using: - * r5: counter - * r6: data - * r7: data - * r8: data - * r9: data - * r10: source - * r11: destination - */ - mov r5, r4 - mov r10, r2 - mov r11, r3 -1: ldmia r10!, {r6, r7, r8, r9} - stmia r11!, {r6, r7, r8, r9} - - subs r5, r5, #16 - bgt 1b - - /* Flush destination from dcache using: - * r5: counter - * r6: step - * r7: vaddr - */ - dsb /* So the CPU issues all writes to the range */ - - mov r5, r4 - ldr r6, =dcache_line_bytes /* r6 := step */ - ldr r6, [r6] - mov r7, r3 - -1: mcr CP32(r7, DCCMVAC) - - add r7, r7, r6 - subs r5, r5, r6 - bgt 1b - +ENTRY(switch_ttbr) dsb /* Ensure the flushes happen before * continuing */ isb /* Ensure synchronization with previous @@ -543,8 +497,6 @@ ENTRY(relocate_xen) dsb /* Ensure completion of TLB+BP flush */ isb - pop {r4, r5,r6,r7,r8,r9,r10,r11} - mov pc, lr #ifdef CONFIG_EARLY_PRINTK diff --git a/xen/arch/arm/arm64/head.S b/xen/arch/arm/arm64/head.S index 9428c3f5a2..4589a37874 100644 --- a/xen/arch/arm/arm64/head.S +++ b/xen/arch/arm/arm64/head.S @@ -605,52 +605,14 @@ fail: PRINT("- Boot failed -\r\n") GLOBAL(_end_boot) -/* Copy Xen to new location and switch TTBR - * x0 ttbr - * x1 source address - * x2 destination address - * x3 length +/* + * Switch TTBR * - * Source and destination must be word aligned, length is rounded up - * to a 16 byte boundary. + * x0 ttbr * - * MUST BE VERY CAREFUL when saving things to RAM over the copy */ -ENTRY(relocate_xen) - /* Copy 16 bytes at a time using: - * x9: counter - * x10: data - * x11: data - * x12: source - * x13: destination - */ - mov x9, x3 - mov x12, x1 - mov x13, x2 - -1: ldp x10, x11, [x12], #16 - stp x10, x11, [x13], #16 - - subs x9, x9, #16 - bgt 1b - - /* Flush destination from dcache using: - * x9: counter - * x10: step - * x11: vaddr - */ - dsb sy /* So the CPU issues all writes to the range */ - - mov x9, x3 - ldr x10, =dcache_line_bytes /* x10 := step */ - ldr x10, [x10] - mov x11, x2 - -1: dc cvac, x11 - - add x11, x11, x10 - subs x9, x9, x10 - bgt 1b - + * TODO: This code does not comply with break-before-make. + */ +ENTRY(switch_ttbr) dsb sy /* Ensure the flushes happen before * continuing */ isb /* Ensure synchronization with previous diff --git a/xen/arch/arm/mm.c b/xen/arch/arm/mm.c index 987fcb9162..5f4fb09257 100644 --- a/xen/arch/arm/mm.c +++ b/xen/arch/arm/mm.c @@ -601,7 +601,7 @@ void __init remove_early_mappings(void) flush_xen_data_tlb_range_va(BOOT_FDT_VIRT_START, BOOT_FDT_SLOT_SIZE); } -extern void relocate_xen(uint64_t ttbr, void *src, void *dst, size_t len); +extern void switch_ttbr(uint64_t ttbr); /* Clear a translation table and clean & invalidate the cache */ static void clear_table(void *table) @@ -612,15 +612,13 @@ static void clear_table(void *table) /* Boot-time pagetable setup. * Changes here may need matching changes in head.S */ -void __init setup_pagetables(unsigned long boot_phys_offset, paddr_t xen_paddr) +void __init setup_pagetables(unsigned long boot_phys_offset) { uint64_t ttbr; - unsigned long dest_va; lpae_t pte, *p; int i; - /* Calculate virt-to-phys offset for the new location */ - phys_offset = xen_paddr - (unsigned long) _start; + phys_offset = boot_phys_offset; #ifdef CONFIG_ARM_64 p = (void *) xen_pgtable; @@ -652,7 +650,7 @@ void __init setup_pagetables(unsigned long boot_phys_offset, paddr_t xen_paddr) /* Initialise xen second level entries ... */ /* ... Xen's text etc */ - pte = mfn_to_xen_entry(maddr_to_mfn(xen_paddr), MT_NORMAL); + pte = pte_of_xenaddr((vaddr_t)_start); pte.pt.xn = 0;/* Contains our text mapping! */ xen_second[second_table_offset(XEN_VIRT_START)] = pte; @@ -667,21 +665,13 @@ void __init setup_pagetables(unsigned long boot_phys_offset, paddr_t xen_paddr) pte = boot_second[second_table_offset(BOOT_FDT_VIRT_START + SZ_2M)]; xen_second[second_table_offset(BOOT_FDT_VIRT_START + SZ_2M)] = pte; - /* ... Boot Misc area for xen relocation */ - dest_va = BOOT_RELOC_VIRT_START; - pte = mfn_to_xen_entry(maddr_to_mfn(xen_paddr), MT_NORMAL); - /* Map the destination in xen_second. */ - xen_second[second_table_offset(dest_va)] = pte; - /* Map the destination in boot_second. */ - write_pte(boot_second + second_table_offset(dest_va), pte); - flush_xen_data_tlb_range_va_local(dest_va, SECOND_SIZE); #ifdef CONFIG_ARM_64 ttbr = (uintptr_t) xen_pgtable + phys_offset; #else ttbr = (uintptr_t) cpu0_pgtable + phys_offset; #endif - relocate_xen(ttbr, _start, (void*)dest_va, _end - _start); + switch_ttbr(ttbr); /* Clear the copy of the boot pagetables. Each secondary CPU * rebuilds these itself (see head.S) */ @@ -696,7 +686,7 @@ void __init setup_pagetables(unsigned long boot_phys_offset, paddr_t xen_paddr) /* Break up the Xen mapping into 4k pages and protect them separately. */ for ( i = 0; i < LPAE_ENTRIES; i++ ) { - mfn_t mfn = mfn_add(maddr_to_mfn(xen_paddr), i); + mfn_t mfn = mfn_add(maddr_to_mfn((vaddr_t)_start + phys_offset), i); unsigned long va = XEN_VIRT_START + (i << PAGE_SHIFT); if ( !is_kernel(va) ) break; diff --git a/xen/arch/arm/setup.c b/xen/arch/arm/setup.c index aec53f30d3..e84172fbc3 100644 --- a/xen/arch/arm/setup.c +++ b/xen/arch/arm/setup.c @@ -374,6 +374,7 @@ void __init discard_initial_modules(void) remove_early_mappings(); } +#ifdef CONFIG_ARM_32 /* * Returns the end address of the highest region in the range s..e * with required size and alignment that does not conflict with the @@ -440,6 +441,7 @@ static paddr_t __init consider_modules(paddr_t s, paddr_t e, } return e; } +#endif /* * Return the end of the non-module region starting at s. In other @@ -475,59 +477,6 @@ static paddr_t __init next_module(paddr_t s, paddr_t *end) return lowest; } - -/** - * get_xen_paddr - get physical address to relocate Xen to - * - * Xen is relocated to as near to the top of RAM as possible and - * aligned to a XEN_PADDR_ALIGN boundary. - */ -static paddr_t __init get_xen_paddr(void) -{ - struct meminfo *mi = &bootinfo.mem; - paddr_t min_size; - paddr_t paddr = 0; - int i; - - min_size = (_end - _start + (XEN_PADDR_ALIGN-1)) & ~(XEN_PADDR_ALIGN-1); - - /* Find the highest bank with enough space. */ - for ( i = 0; i < mi->nr_banks; i++ ) - { - const struct membank *bank = &mi->bank[i]; - paddr_t s, e; - - if ( bank->size >= min_size ) - { - e = consider_modules(bank->start, bank->start + bank->size, - min_size, XEN_PADDR_ALIGN, 0); - if ( !e ) - continue; - -#ifdef CONFIG_ARM_32 - /* Xen must be under 4GB */ - if ( e > 0x100000000ULL ) - e = 0x100000000ULL; - if ( e < bank->start ) - continue; -#endif - - s = e - min_size; - - if ( s > paddr ) - paddr = s; - } - } - - if ( !paddr ) - panic("Not enough memory to relocate Xen\n"); - - printk("Placing Xen at 0x%"PRIpaddr"-0x%"PRIpaddr"\n", - paddr, paddr + min_size); - - return paddr; -} - static void __init init_pdx(void) { paddr_t bank_start, bank_size, bank_end; @@ -783,7 +732,6 @@ void __init start_xen(unsigned long boot_phys_offset, { size_t fdt_size; int cpus, i; - paddr_t xen_paddr; const char *cmdline; struct bootmodule *xen_bootmodule; struct domain *dom0; @@ -827,14 +775,7 @@ void __init start_xen(unsigned long boot_phys_offset, (paddr_t)(uintptr_t)(_end - _start + 1), false); BUG_ON(!xen_bootmodule); - xen_paddr = get_xen_paddr(); - setup_pagetables(boot_phys_offset, xen_paddr); - - /* Update Xen's address now that we have relocated. */ - printk("Update BOOTMOD_XEN from %"PRIpaddr"-%"PRIpaddr" => %"PRIpaddr"-%"PRIpaddr"\n", - xen_bootmodule->start, xen_bootmodule->start + xen_bootmodule->size, - xen_paddr, xen_paddr + xen_bootmodule->size); - xen_bootmodule->start = xen_paddr; + setup_pagetables(boot_phys_offset); setup_mm(fdt_paddr, fdt_size); diff --git a/xen/include/asm-arm/mm.h b/xen/include/asm-arm/mm.h index 940b74be33..129404b196 100644 --- a/xen/include/asm-arm/mm.h +++ b/xen/include/asm-arm/mm.h @@ -169,7 +169,7 @@ extern unsigned long total_pages; #define PDX_GROUP_SHIFT SECOND_SHIFT /* Boot-time pagetable setup */ -extern void setup_pagetables(unsigned long boot_phys_offset, paddr_t xen_paddr); +extern void setup_pagetables(unsigned long boot_phys_offset); /* Map FDT in boot pagetable */ extern void *early_fdt_map(paddr_t fdt_paddr); /* Remove early mappings */