[v8,24/26] drivers: firmware: psci: Deal with CPU hotplug when using OSI mode

Message ID 20180620172226.15012-25-ulf.hansson@linaro.org
State New
Headers show
Series
  • PM / Domains: Support hierarchical CPU arrangement (PSCI/ARM)
Related show

Commit Message

Ulf Hansson June 20, 2018, 5:22 p.m.
To deal with CPU hotplug when OSI mode is used, the CPU device needs to be
detached from its PM domain (genpd) when putting it offline, otherwise the
CPU becomes considered as being in use from genpd and runtime PM point of
view. Obviously, then we also need to re-attach the CPU device when bring
the CPU back online, so let's do this.

Cc: Lina Iyer <ilina@codeaurora.org>
Signed-off-by: Ulf Hansson <ulf.hansson@linaro.org>

---
 drivers/firmware/psci/psci.c | 8 ++++++++
 1 file changed, 8 insertions(+)

-- 
2.17.1

Comments

Raju P L S S S N Nov. 19, 2018, 7:50 p.m. | #1
Hi Ulf,

Got one issue in hotplug path where of_genpd_detach_cpu calls 
dev_pm_qos_remove_notifier which can be sleeping as per below call 
stack. I think it should be applicable for current patch as well right? 
Please let me know what am I missing? why didn't you see this issue with 
this patch?


[ 8103.221387] BUG: sleeping function called from invalid context at 
/mnt/host/source/src/third_party/kernel/v4.14/kernel/locking/mutex.c:238
[ 8103.221455] in_atomic(): 1, irqs_disabled(): 128, pid: 11, name: 
migration/0
[ 8103.221487] Preemption disabled at:
[ 8103.221529] [<ffffff800814dfb0>] cpu_stopper_thread+0x98/0x118
[ 8103.221600] ------------[ cut here ]------------
[ 8103.221636] kernel BUG at 
/mnt/host/source/src/third_party/kernel/v4.14/kernel/sched/core.c:6102!
[ 8103.221678] Internal error: Oops - BUG: 0 [#1] PREEMPT SMP
[ 8103.222396] CPU: 0 PID: 11 Comm: migration/0 Tainted: G        W 
  4.14.72 #1
[ 8103.222428] Hardware name: Google Cheza (rev1) (DT)
[ 8103.222460] task: ffffffc0f842d580 task.stack: ffffff8009c18000
[ 8103.222504] PC is at ___might_sleep+0x138/0x140
[ 8103.222542] LR is at ___might_sleep+0x138/0x140
[ 8103.222577] pc : [<ffffff80080d8f04>] lr : [<ffffff80080d8f04>] 
pstate: 60c001c9
[ 8103.222605] sp : ffffff8009c1bb40
….
[ 8103.223924] [<ffffff80080d8f04>] ___might_sleep+0x138/0x140
[ 8103.223965] [<ffffff80080d8d98>] __might_sleep+0x4c/0x80
[ 8103.224009] [<ffffff80088e4258>] mutex_lock+0x28/0x60
[ 8103.224054] [<ffffff800850fa2c>] dev_pm_qos_remove_notifier+0x1c/0x54
[ 8103.224097] [<ffffff8008517814>] genpd_remove_device+0x3c/0x10c
[ 8103.224140] [<ffffff800851949c>] genpd_dev_pm_detach+0x48/0x108
[ 8103.224183] [<ffffff80085193e0>] of_genpd_detach_cpu+0x48/0xbc
[ 8103.224227] [<ffffff80083edea4>] cpu_pd_dying+0x28/0x38
[ 8103.224268] [<ffffff80080ab2c0>] cpuhp_invoke_callback+0x254/0x5f0
[ 8103.224308] [<ffffff80080acdec>] take_cpu_down+0x60/0x9c
[ 8103.224346] [<ffffff800814d898>] multi_cpu_stop+0xac/0x104
[ 8103.224385] [<ffffff800814dfb8>] cpu_stopper_thread+0xa0/0x118
[ 8103.224427] [<ffffff80080cff74>] smpboot_thread_fn+0x19c/0x278
[ 8103.224472] [<ffffff80080cc0c4>] kthread+0x120/0x130
[ 8103.224513] [<ffffff8008084608>] ret_from_fork+0x10/0x18

Thanks,
Raju

On 6/20/2018 10:52 PM, Ulf Hansson wrote:
> To deal with CPU hotplug when OSI mode is used, the CPU device needs to be

> detached from its PM domain (genpd) when putting it offline, otherwise the

> CPU becomes considered as being in use from genpd and runtime PM point of

> view. Obviously, then we also need to re-attach the CPU device when bring

> the CPU back online, so let's do this.

> 

> Cc: Lina Iyer <ilina@codeaurora.org>

> Signed-off-by: Ulf Hansson <ulf.hansson@linaro.org>

> ---

>   drivers/firmware/psci/psci.c | 8 ++++++++

>   1 file changed, 8 insertions(+)

> 

> diff --git a/drivers/firmware/psci/psci.c b/drivers/firmware/psci/psci.c

> index 700e0e995871..e649673d71f0 100644

> --- a/drivers/firmware/psci/psci.c

> +++ b/drivers/firmware/psci/psci.c

> @@ -190,6 +190,10 @@ static int psci_cpu_off(u32 state)

>   	int err;

>   	u32 fn;

>   

> +	/* If running OSI mode, detach the CPU device from its PM domain. */

> +	if (psci_osi_mode_enabled)

> +		of_genpd_detach_cpu(smp_processor_id());

> +

>   	fn = psci_function_id[PSCI_FN_CPU_OFF];

>   	err = invoke_psci_fn(fn, state, 0, 0);

>   	return psci_to_linux_errno(err);

> @@ -204,6 +208,10 @@ static int psci_cpu_on(unsigned long cpuid, unsigned long entry_point)

>   	err = invoke_psci_fn(fn, cpuid, entry_point, 0);

>   	/* Clear the domain state to start fresh. */

>   	psci_set_domain_state(0);

> +

> +	if (!err && psci_osi_mode_enabled)

> +		of_genpd_attach_cpu(cpuid);

> +

>   	return psci_to_linux_errno(err);

>   }

>   

>
Ulf Hansson Nov. 20, 2018, 9:50 a.m. | #2
On 19 November 2018 at 20:50, Raju P L S S S N <rplsssn@codeaurora.org> wrote:
> Hi Ulf,

>

> Got one issue in hotplug path where of_genpd_detach_cpu calls

> dev_pm_qos_remove_notifier which can be sleeping as per below call stack. I

> think it should be applicable for current patch as well right? Please let me

> know what am I missing? why didn't you see this issue with this patch?


Weird.

>

>

> [ 8103.221387] BUG: sleeping function called from invalid context at

> /mnt/host/source/src/third_party/kernel/v4.14/kernel/locking/mutex.c:238


Could it be due to some other patch in your v.4.14 kernel?

> [ 8103.221455] in_atomic(): 1, irqs_disabled(): 128, pid: 11, name:

> migration/0

> [ 8103.221487] Preemption disabled at:

> [ 8103.221529] [<ffffff800814dfb0>] cpu_stopper_thread+0x98/0x118

> [ 8103.221600] ------------[ cut here ]------------

> [ 8103.221636] kernel BUG at

> /mnt/host/source/src/third_party/kernel/v4.14/kernel/sched/core.c:6102!

> [ 8103.221678] Internal error: Oops - BUG: 0 [#1] PREEMPT SMP

> [ 8103.222396] CPU: 0 PID: 11 Comm: migration/0 Tainted: G        W  4.14.72

> #1

> [ 8103.222428] Hardware name: Google Cheza (rev1) (DT)

> [ 8103.222460] task: ffffffc0f842d580 task.stack: ffffff8009c18000

> [ 8103.222504] PC is at ___might_sleep+0x138/0x140

> [ 8103.222542] LR is at ___might_sleep+0x138/0x140

> [ 8103.222577] pc : [<ffffff80080d8f04>] lr : [<ffffff80080d8f04>] pstate:

> 60c001c9

> [ 8103.222605] sp : ffffff8009c1bb40

> ….

> [ 8103.223924] [<ffffff80080d8f04>] ___might_sleep+0x138/0x140

> [ 8103.223965] [<ffffff80080d8d98>] __might_sleep+0x4c/0x80

> [ 8103.224009] [<ffffff80088e4258>] mutex_lock+0x28/0x60

> [ 8103.224054] [<ffffff800850fa2c>] dev_pm_qos_remove_notifier+0x1c/0x54

> [ 8103.224097] [<ffffff8008517814>] genpd_remove_device+0x3c/0x10c

> [ 8103.224140] [<ffffff800851949c>] genpd_dev_pm_detach+0x48/0x108

> [ 8103.224183] [<ffffff80085193e0>] of_genpd_detach_cpu+0x48/0xbc

> [ 8103.224227] [<ffffff80083edea4>] cpu_pd_dying+0x28/0x38

> [ 8103.224268] [<ffffff80080ab2c0>] cpuhp_invoke_callback+0x254/0x5f0

> [ 8103.224308] [<ffffff80080acdec>] take_cpu_down+0x60/0x9c

> [ 8103.224346] [<ffffff800814d898>] multi_cpu_stop+0xac/0x104

> [ 8103.224385] [<ffffff800814dfb8>] cpu_stopper_thread+0xa0/0x118

> [ 8103.224427] [<ffffff80080cff74>] smpboot_thread_fn+0x19c/0x278

> [ 8103.224472] [<ffffff80080cc0c4>] kthread+0x120/0x130

> [ 8103.224513] [<ffffff8008084608>] ret_from_fork+0x10/0x18


Thanks for the report, I will double check my series before I post the
new version of my series. If nothing unexpected shows up, that should
be in a couple of days from now.

I keep you cc.

[...]

Kind regards
Uffe

Patch

diff --git a/drivers/firmware/psci/psci.c b/drivers/firmware/psci/psci.c
index 700e0e995871..e649673d71f0 100644
--- a/drivers/firmware/psci/psci.c
+++ b/drivers/firmware/psci/psci.c
@@ -190,6 +190,10 @@  static int psci_cpu_off(u32 state)
 	int err;
 	u32 fn;
 
+	/* If running OSI mode, detach the CPU device from its PM domain. */
+	if (psci_osi_mode_enabled)
+		of_genpd_detach_cpu(smp_processor_id());
+
 	fn = psci_function_id[PSCI_FN_CPU_OFF];
 	err = invoke_psci_fn(fn, state, 0, 0);
 	return psci_to_linux_errno(err);
@@ -204,6 +208,10 @@  static int psci_cpu_on(unsigned long cpuid, unsigned long entry_point)
 	err = invoke_psci_fn(fn, cpuid, entry_point, 0);
 	/* Clear the domain state to start fresh. */
 	psci_set_domain_state(0);
+
+	if (!err && psci_osi_mode_enabled)
+		of_genpd_attach_cpu(cpuid);
+
 	return psci_to_linux_errno(err);
 }