From patchwork Wed Nov 4 17:10:49 2015 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Rutland X-Patchwork-Id: 56000 Delivered-To: patch@linaro.org Received: by 10.112.61.134 with SMTP id p6csp2532530lbr; Wed, 4 Nov 2015 09:13:10 -0800 (PST) X-Received: by 10.50.4.98 with SMTP id j2mr4488152igj.13.1446657190514; Wed, 04 Nov 2015 09:13:10 -0800 (PST) Return-Path: Received: from bombadil.infradead.org (bombadil.infradead.org. [2001:1868:205::9]) by mx.google.com with ESMTPS id e6si3042443ioe.27.2015.11.04.09.13.10 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 04 Nov 2015 09:13:10 -0800 (PST) Received-SPF: pass (google.com: domain of linux-arm-kernel-bounces+patch=linaro.org@lists.infradead.org designates 2001:1868:205::9 as permitted sender) client-ip=2001:1868:205::9; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-arm-kernel-bounces+patch=linaro.org@lists.infradead.org designates 2001:1868:205::9 as permitted sender) smtp.mailfrom=linux-arm-kernel-bounces+patch=linaro.org@lists.infradead.org Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.80.1 #2 (Red Hat Linux)) id 1Zu1ap-0005Us-0m; Wed, 04 Nov 2015 17:11:27 +0000 Received: from foss.arm.com ([217.140.101.70]) by bombadil.infradead.org with esmtp (Exim 4.80.1 #2 (Red Hat Linux)) id 1Zu1al-0005Hb-8E for linux-arm-kernel@lists.infradead.org; Wed, 04 Nov 2015 17:11:24 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 8BE144A; Wed, 4 Nov 2015 09:10:50 -0800 (PST) Received: from leverpostej (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id BE5653F21A; Wed, 4 Nov 2015 09:10:59 -0800 (PST) Date: Wed, 4 Nov 2015 17:10:49 +0000 From: Mark Rutland To: Sharma Bhupesh Subject: Re: PSCI_CPU_ON flow Message-ID: <20151104171048.GE23860@leverpostej> References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.21 (2010-09-15) X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20151104_091123_435371_46A604CD X-CRM114-Status: GOOD ( 30.15 ) X-Spam-Score: -6.9 (------) X-Spam-Report: SpamAssassin version 3.4.0 on bombadil.infradead.org summary: Content analysis details: (-6.9 points) pts rule name description ---- ---------------------- -------------------------------------------------- -5.0 RCVD_IN_DNSWL_HI RBL: Sender listed at http://www.dnswl.org/, high trust [217.140.101.70 listed in list.dnswl.org] -0.0 SPF_PASS SPF: sender matches SPF record -0.0 T_RP_MATCHES_RCVD Envelope sender domain matches handover relay domain -1.9 BAYES_00 BODY: Bayes spam probability is 0 to 1% [score: 0.0000] X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: "lorenzo.pieralisi@arm.com" , "will.deacon@arm.com" , "linux-arm-kernel@lists.infradead.org" Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patch=linaro.org@lists.infradead.org On Wed, Nov 04, 2015 at 04:18:40PM +0000, Sharma Bhupesh wrote: > Hi, Hi, > I am trying to build and verify a custom ARM trusted firmware like implementation with > the linux-next to verify if I can use the PSCI interface to get the CPU hotplug > to work (CPU_OFF and CPU_ON sequence). > > My EL3 firmware implements the PSCI v0.2 specifications and I enable the same > as the CPU enable_method in the DTS. > > Now, while I can get the secondary CPU to shutdown properly via the PSCI implementing > firmware, I cannot seem to get them back-on: > > # echo 0 > /sys/devices/system/cpu/cpu2/online > CPU2: shutdown > psci: CPU2 killed. > > # echo 1 > /sys/devices/system/cpu/cpu2/online > CPU2: failed to come online > > I notice that the call to PSCI CPU_ON from kernel returns successfully, > so the EL3 FW returns PSCI_SUCCESS: > > static int cpu_psci_cpu_boot(unsigned int cpu) > { > int err = psci_ops.cpu_on(cpu_logical_map(cpu), __pa(secondary_entry)); > if (err) > pr_err("failed to boot CPU%d (%d)\n", cpu, err); > > return err; > } > > However, I cannot seem to get the secondaries to enter the 'secondary_entry' > leg, i.e. I cannot see the normal 'CPU%u: Booted secondary processor' prints > from here: > > /* > * This is the secondary CPU boot entry. We're using this CPUs > * idle thread stack, but a set of temporary page tables. > */ > asmlinkage void secondary_start_kernel(void) > { > struct mm_struct *mm = &init_mm; > unsigned int cpu = smp_processor_id(); > > /* > * All kernel threads share the same mm context; grab a > * reference and switch to it. > */ > atomic_inc(&mm->mm_count); > current->active_mm = mm; > cpumask_set_cpu(cpu, mm_cpumask(mm)); > > set_my_cpu_offset(per_cpu_offset(smp_processor_id())); > printk("CPU%u: Booted secondary processor\n", cpu); I see this isn't the same as next-20151104, though the difference is a trivial change to the logging. Which linux-next version are you using? > So, I am wondering if my understanding is correct, as to whether the > secondaries will start execution from __pa(secondary_entry), i.e. > secondary_start_kernel Yes, the CPU should start at __pa(secondary_entry) in head.S. From there it will branch to secondary_startup: ENTRY(secondary_entry) bl el2_setup // Drop to EL1 bl set_cpu_boot_mode_flag b secondary_startup ENDPROC(secondary_entry) > and I should see the above prints in the PSCI CPU_ON case as well, Yes, the print should occur for any PSCI CPU_ON of any CPU. > or is this path only taken in case of a cold boot. The same path is taken regardless of cold/warm boot. I don't see issues on Juno with next-20151104: root@ribbensteg:/home/nanook# echo 0 > /sys/devices/system/cpu/cpu2/online CPU2: shutdown psci: CPU2 killed. root@ribbensteg:/home/nanook# echo 1 > /sys/devices/system/cpu/cpu2/online Detected PIPT I-cache on CPU2 CPU2: Booted secondary processor [410fd070] Given that, and the successful cold boot, it looks like the issue is with the FW rather than the kernel. I guess that either the CPU is stuck in the FW, or that it left the FW in a bad state, and things went wrong before it could execute secondary_start_kernel. The following patch may be able to tell you if the CPU is stuck in the FW (the CPU will likely never leave state 2 / ON_PENDING in this case). Thanks, Mark. ---->8---- >From 0573953e0426b2990a71d9993b550d542bf36d87 Mon Sep 17 00:00:00 2001 From: Mark Rutland Date: Wed, 4 Nov 2015 17:01:45 +0000 Subject: [PATCH] arm64: psci: HACK: query AFFINTIY_INFO after CPU_ON Buggy PSCI firmware can report a successful CPU_ON, yet the CPU may never leave ON_PENDING. Query AFFINITY_INFO after a successful CPU_ON to aid debugging such issues. This is rather noisy, and is only useful as a debugging tool. Signed-off-by: Mark Rutland --- arch/arm64/kernel/psci.c | 28 ++++++++++++++++++++++++++-- 1 file changed, 26 insertions(+), 2 deletions(-) -- 1.9.1 _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel diff --git a/arch/arm64/kernel/psci.c b/arch/arm64/kernel/psci.c index f67f35b..8dd9e0a 100644 --- a/arch/arm64/kernel/psci.c +++ b/arch/arm64/kernel/psci.c @@ -111,13 +111,37 @@ static int __init cpu_psci_cpu_prepare(unsigned int cpu) return 0; } +static int cpu_psci_check_online(int cpu) +{ + int err, i; + + if (!psci_ops.affinity_info) + return 0; + + for (i = 0; i < 10; i++) { + err = psci_ops.affinity_info(cpu_logical_map(cpu), 0); + if (err == PSCI_0_2_AFFINITY_LEVEL_ON) { + pr_info("CPU%d reported online\n", cpu); + return 0; + } + msleep(10); + pr_info("CPU%d reported in state %d", cpu, err); + } + + pr_warn("CPU%d was not reported online in a timely fashion\n", cpu); + + return -ETIMEDOUT; +} + static int cpu_psci_cpu_boot(unsigned int cpu) { int err = psci_ops.cpu_on(cpu_logical_map(cpu), __pa(secondary_entry)); - if (err) + if (err) { pr_err("failed to boot CPU%d (%d)\n", cpu, err); + return err; + } - return err; + return cpu_psci_check_online(cpu); } #ifdef CONFIG_HOTPLUG_CPU