From patchwork Tue Mar 19 10:33:35 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Julien Grall X-Patchwork-Id: 160537 Delivered-To: patch@linaro.org Received: by 2002:a02:5cc1:0:0:0:0:0 with SMTP id w62csp3705639jad; Tue, 19 Mar 2019 03:35:09 -0700 (PDT) X-Google-Smtp-Source: APXvYqz3vun9FLiLO1B07aN6+6jZjEojaocpEdrCsnux7JjxhzMAWKgNBCcx1vQKDuZl79dSNy5n X-Received: by 2002:a25:5041:: with SMTP id e62mr1188548ybb.271.1552991709534; Tue, 19 Mar 2019 03:35:09 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1552991709; cv=none; d=google.com; s=arc-20160816; b=RkPinRB/fRVPPzNfUlx/CS9+Z9fzeapfMj4ka2b4z1aS88nVRrKSXnwStO4mibDbHO vRxK31wWifvdYrATcUAkiG4oZmeoX0rQ3DzHM3Xdj0XgIMT6rpkyIfOi6maLVOlnvQl1 ruoqMKvveTQIGXqjhop0Hlgwpnhdk4wPv9qGM+1Wgxk9JDUffnnhe0/sSoRRsa3M/ACD /sKaiFFj8rN2FfP+taQBE4XMTkox2R0nnPjDn7PlHHqTuGcZD6Mxn0AI+BL/mjF0ddYu CCdz4uVWqqNxBMaiUv4tltOsMuXn21fiAp/hM8EKAIHPZE75N8HekylPPoS8CB64h3y4 D8zg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:content-transfer-encoding:cc:list-subscribe :list-help:list-post:list-unsubscribe:list-id:precedence:subject :content-language:in-reply-to:mime-version:user-agent:date :message-id:from:references:to; bh=XPKt68stFnkuveZA31bityfKUptBIsbrXON3s4oDZIU=; b=kJ9FhsIBrczM/7Tr5TnCSc2uM8RV1LtlTSW1Ud//ZNN94rpGtIzE93yn03vlOpw3Z2 ywNMz1/+neIHnum3QGLpPm4DUIghsLXdES6rwel86smDf9BNJY2Ci2ZLxy1iIHm9zOvG CC6fBE5UnPX+2Ya6vGiD41/6keaIQbcNOwU7YT2TlXdtHYJ/TwzH1mG+uvPdwPNDzSfN X3TXH0a+nIY8VxXcjebHJfJXMaMntxMhetRY5GxnjbHtOu/1TQ3Uq2ruGw53GTZsUXaY DRKVrREqM7UKjXbmTKZ2UN4nN4pv6xTNVRYfbAdfrZLm3bCCHFKafi1lLvrlJHVFLZ1+ /RsA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of xen-devel-bounces@lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Return-Path: Received: from lists.xenproject.org (lists.xenproject.org. [192.237.175.120]) by mx.google.com with ESMTPS id p130si7621440ywg.245.2019.03.19.03.35.09 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Tue, 19 Mar 2019 03:35:09 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of xen-devel-bounces@lists.xenproject.org designates 192.237.175.120 as permitted sender) client-ip=192.237.175.120; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of xen-devel-bounces@lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.89) (envelope-from ) id 1h6C3y-0003tx-Gj; Tue, 19 Mar 2019 10:33:42 +0000 Received: from us1-rack-dfw2.inumbo.com ([104.130.134.6]) by lists.xenproject.org with esmtp (Exim 4.89) (envelope-from ) id 1h6C3w-0003ts-V7 for xen-devel@lists.xenproject.org; Tue, 19 Mar 2019 10:33:40 +0000 X-Inumbo-ID: 755af00c-4a32-11e9-bc90-bc764e045a96 Received: from foss.arm.com (unknown [217.140.101.70]) by us1-rack-dfw2.inumbo.com (Halon) with ESMTP id 755af00c-4a32-11e9-bc90-bc764e045a96; Tue, 19 Mar 2019 10:33:39 +0000 (UTC) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id C0BF41650; Tue, 19 Mar 2019 03:33:38 -0700 (PDT) Received: from [10.37.10.136] (unknown [10.37.10.136]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 350C73F614; Tue, 19 Mar 2019 03:33:37 -0700 (PDT) To: Amit Tomer References: <6b603d55-5823-b203-45eb-a0dd61ef81ee@arm.com> <18d6524d-8dd9-2c0b-34b3-bdb7316bebae@arm.com> From: Julien Grall Message-ID: Date: Tue, 19 Mar 2019 10:33:35 +0000 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Thunderbird/60.5.1 MIME-Version: 1.0 In-Reply-To: Content-Language: en-US Subject: [Xen-devel] Arm boot regression with Xen 4.12 X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Cc: Juergen Gross , xen-devel@lists.xenproject.org, peng.fan@nxp.com, Stefano Stabellini , Andre Przywara Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" (+ Juergen) Hi Amit, On 3/18/19 3:12 PM, Amit Tomer wrote: >> It will be difficult to help without any log. You probably want to try with >> Stefano series instead. However ... > > If we comment out GPU node(gpu@38000000) , we don't see this issue and > Dom0 kernel is > loaded into memory but we following crash: > > Starting kernel ... > > - UART enabled - > - CPU 00000000 booting - > - Current EL 00000008 - > - Xen starting at EL2 - > - Zero BSS - > - Setting up control registers - > - Turning on paging - > - Ready - > (XEN) Checking for initrd in /chosen > (XEN) RAM: 0000000040000000 - 00000000bfffffff > (XEN) > (XEN) MODULE[0]: 00000000be511000 - 00000000be51d000 Device Tree > (XEN) MODULE[1]: 0000000040480000 - 0000000042680000 Kernel > (XEN) RESVD[0]: 0000000043000000 - 000000004300c000 > (XEN) RESVD[1]: 00000000be511000 - 00000000be51d000 [...] > (XEN) *** Serial input to DOM0 (type 'CTRL-a' three times to switch input) > (XEN) Data Abort Trap. Syndrome=0x6 > (XEN) Walking Hypervisor VA 0x8 on CPU0 via TTBR 0x0000000042114000 > (XEN) 0TH[0x0] = 0x0000000042113f7f > (XEN) 1ST[0x0] = 0x0000000042110f7f > (XEN) 2ND[0x0] = 0x0000000000000000 > (XEN) CPU0: Unexpected Trap: Data Abort > (XEN) ----[ Xen-4.12.0-rc arm64 debug=y Not tainted ]---- > (XEN) CPU: 0 > (XEN) PC: 000000000021c220 page_alloc.c#free_heap_pages+0x3b0/0x58c [...] > (XEN) Xen call trace: > (XEN) [<000000000021c220>] page_alloc.c#free_heap_pages+0x3b0/0x58c (PC) > (XEN) [<000000000021c20c>] page_alloc.c#free_heap_pages+0x39c/0x58c (LR) > (XEN) [<000000000021e5f4>] page_alloc.c#init_heap_pages+0x334/0x4ec > (XEN) [<000000000021e840>] init_domheap_pages+0x94/0x9c > (XEN) [<000000000024e178>] free_init_memory+0xac/0xe0 > (XEN) [<0000000000252580>] setup.c#init_done+0x14/0x20 > (XEN) [<000000000029daa8>] 000000000029daa8 > (XEN) > (XEN) > (XEN) **************************************** > (XEN) Panic on CPU 0: > (XEN) CPU0: Unexpected Trap: Data Abort > (XEN) **************************************** > (XEN) > (XEN) Reboot in five seconds... Could you give a try to the below patch? Now the long answer. Unfortunately, in a recent page, I removed the log telling where Xen lives in memory, so I am not 100% sure this is your problem. >From my own testing, I think the problem is Xen will try to hand reserved memory (the old fashion /memreserve/ and not /reserved-regions) to the allocator. This happen when freeing the init regions (see free_init_memory). We do handle correctly all the others modules (see discard_initial_modules). On my setup this does not crash Xen, instead it happily hand the page to the allocator which is not good. The difference in behavior may be because on how the PDX is setup (I need to investigate that). So by luck, I have a struct page_info backing the reserved-memory region. This does not mean it is better :). This regression was introduced by commit f60658c6ae "xen/arm: Stop relocating Xen". Before hand, Xen was always relocated so the original Xen was left untouched. The relocated version would always live in non-reserved area. On my setup, Xen was not in the reserved region area by default. I had to modify the Device-Tree. I don't know how many platform are putting Xen in /memreserve/ region. Amit, assuming the patch above works for you, could you tell who created the /memreserve/? Cheers, diff --git a/xen/arch/arm/mm.c b/xen/arch/arm/mm.c index 01ae2cccc0..2c34138bbd 100644 --- a/xen/arch/arm/mm.c +++ b/xen/arch/arm/mm.c @@ -1139,7 +1139,7 @@ void free_init_memory(void) *(p + i) = insn; set_pte_flags_on_range(__init_begin, len, mg_clear); - init_domheap_pages(pa, pa + len); + dt_unreserved_regions(pa, pa + len, init_domheap_pages, 0); printk("Freed %ldkB init memory.\n", (long)(__init_end-__init_begin)>>10); } diff --git a/xen/arch/arm/setup.c b/xen/arch/arm/setup.c index 444857a967..8dbc4f819b 100644 --- a/xen/arch/arm/setup.c +++ b/xen/arch/arm/setup.c @@ -764,18 +764,18 @@ void __init start_xen(unsigned long boot_phys_offset, "Please check your bootloader.\n", fdt_paddr); - fdt_size = boot_fdt_info(device_tree_flattened, fdt_paddr); - - cmdline = boot_fdt_cmdline(device_tree_flattened); - printk("Command line: %s\n", cmdline); - cmdline_parse(cmdline); - /* Register Xen's load address as a boot module. */ xen_bootmodule = add_boot_module(BOOTMOD_XEN, (paddr_t)(uintptr_t)(_start + boot_phys_offset), (paddr_t)(uintptr_t)(_end - _start + 1), false); BUG_ON(!xen_bootmodule); + fdt_size = boot_fdt_info(device_tree_flattened, fdt_paddr); + + cmdline = boot_fdt_cmdline(device_tree_flattened); + printk("Command line: %s\n", cmdline); + cmdline_parse(cmdline); + setup_pagetables(boot_phys_offset); setup_mm(fdt_paddr, fdt_size);