From patchwork Fri Mar 15 09:13:07 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Viresh Kumar X-Patchwork-Id: 160412 Delivered-To: patch@linaro.org Received: by 2002:a02:5cc1:0:0:0:0:0 with SMTP id w62csp16511926jad; Fri, 15 Mar 2019 02:13:21 -0700 (PDT) X-Google-Smtp-Source: APXvYqwx7q7TspAzjwmsKFpN0IwIqBaPUShGl5G4JfBehnVP9mR4hHfgGDpDFZVfZIkSGWpOcWr5 X-Received: by 2002:aa7:8ac8:: with SMTP id b8mr1978834pfd.87.1552641201749; Fri, 15 Mar 2019 02:13:21 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1552641201; cv=none; d=google.com; s=arc-20160816; b=efTf9LakE2z/q3a1V1U22jSWUmyjgPvNamQ2mps9Zkqq48grHX8u64M2tj8yPfGRGV 9sAdnWUXBejyEQ3UHGebT+yjhErzC+IH4KX1hdE5ihO74aDv7dxnG4cdM/BHhpz6jrNV 6wyFtH9NwW10IiZ5ayrCOa2Kt8N2q3Jo33XL01KZE+sZwiPI45h/aN/A4c16YRUTxlio o2s8VuNU/4jeGT2TVwalRY3u+tTNPXSYqQTYnzu+fhss7Q6IJmjJY7dqubw9fibQc0sZ EA9Lj6obICmuCiIZeqBFUWy4GtwvYcZ40zCadi2rtZ6Oyis2bWTz2/IbQPVLIM6qpAuI FxJg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:content-transfer-encoding:mime-version :message-id:date:subject:cc:to:from:dkim-signature; bh=UkeUa0P2JUNBNYzTRx3rdebmZF7ZweVBbm012prHiz8=; b=l11wqVoX8fWzwSyJqpIfOUWObD1cV1ob7bkNOIO5428igxQJ6XDG8SQbnEn80LsX7g a94vOLvqujTROzDNPNlzN0Wawl3yWN3MMyaYmy5x2EQbFZi5gWgCBXk+tHRtJtonjmMy z1yXgKwnIz5ub9MlWPSZVQ5UYNqWpWrTRXRG0GWHI0Agt7uDyFjuknf2mHtKdoPpAeq2 VBn4wEVPn5e/t7SBixae9x0cxz4VX3RqLkDAEhBpjulSfYV3hFEkUVXObrEnZ/qPs/Pw JsC84zSzjW2gloVOHp7y+NBse8yXKzAKKCcMeUrD8xSTNEixSgcO6U45JgGNyO9NLOrE kcZw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=lHEMOiHb; spf=pass (google.com: best guess record for domain of linux-pm-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-pm-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id o12si1356031plg.221.2019.03.15.02.13.21; Fri, 15 Mar 2019 02:13:21 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-pm-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=lHEMOiHb; spf=pass (google.com: best guess record for domain of linux-pm-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-pm-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728637AbfCOJNU (ORCPT + 11 others); Fri, 15 Mar 2019 05:13:20 -0400 Received: from mail-pg1-f194.google.com ([209.85.215.194]:43248 "EHLO mail-pg1-f194.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728552AbfCOJNU (ORCPT ); Fri, 15 Mar 2019 05:13:20 -0400 Received: by mail-pg1-f194.google.com with SMTP id l11so6003334pgq.10 for ; Fri, 15 Mar 2019 02:13:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=UkeUa0P2JUNBNYzTRx3rdebmZF7ZweVBbm012prHiz8=; b=lHEMOiHbFe0BNTK2ItHwbExU/Tvu6EIKyx/jO19kFW7dSZxhWK9hApaI9weYVRAJoP n+det+TdAJ1b0GtDbFfCLYG+6xv/zf9q9ul4z0+nyvP9T49UzcekknAQYLdnfP6BHl+2 WVp90Mhvpen6RALvShkCkdvJESbXOkEeSLJRVE0OvqZFKCfnkvKTaDChvSCDnhe7M8H1 T38O+M85nYVHNSbn6H1Ei2i8rehgdMW+ZkacA2r7VzLkoxL4JtKTfjFZassCM2vD4f97 WF+wnJnU3ST0rC2nPISkGbPJ1J6MoekasHgs7trMHd4/3aW5FuppwbhxwPXYZFYOYs7h 3FFA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=UkeUa0P2JUNBNYzTRx3rdebmZF7ZweVBbm012prHiz8=; b=tstNCmUdP4u+EkEr1quVRUZvrDPFOLomupv/tg+hkmFo3FnQFUF0Ej5Mx8dEGYjg8W Y+b+Zw1otiOamyZotVzM1XbfSro9xcQVWpQ8x2iou4B9eDw/U3f1nXAJqJq5yrcBemk2 y/C4vwXy9pOWFNnX45GpfjdLxZvST01Yuocrl5lhKRDxHQ5H88ne+2F9jv1Ea0+VinXB v7IZUMFW4qRDPICzZG84J0kQ+Q0kUUuyOnf8JOr08BEWdouvo8jKboRJlqEcjapOPBgN vugXp2frtENeu9HiGBGdnKq1B6uQyJdqa/pa7/63le+XbHai/rLZqTXysnobQ10i5jCP vdxg== X-Gm-Message-State: APjAAAX15+GSTfPP2Y/kOn1QUpxi8GH5iUiJYJKZxSQrTyWZSLo3xFRg 4n7TjWBofcW9XAdN5fr9e2xpuw== X-Received: by 2002:a17:902:b402:: with SMTP id x2mr3066111plr.53.1552641198787; Fri, 15 Mar 2019 02:13:18 -0700 (PDT) Received: from localhost ([122.166.134.37]) by smtp.gmail.com with ESMTPSA id f125sm2640436pfc.91.2019.03.15.02.13.17 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 15 Mar 2019 02:13:17 -0700 (PDT) From: Viresh Kumar To: Rafael Wysocki , Russell King , "David S. Miller" , Thomas Gleixner , Ingo Molnar , Borislav Petkov , "H. Peter Anvin" , x86@kernel.org, Paolo Bonzini , =?utf-8?b?UmFkaW0gS3LEjW3DocWZ?= Cc: Viresh Kumar , linux-pm@vger.kernel.org, Vincent Guittot , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, sparclinux@vger.kernel.org, kvm@vger.kernel.org Subject: [PATCH V2] cpufreq: Call transition notifier only once for each policy Date: Fri, 15 Mar 2019 14:43:07 +0530 Message-Id: X-Mailer: git-send-email 2.21.0.rc0.269.g1a574e7a288b MIME-Version: 1.0 Sender: linux-pm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pm@vger.kernel.org Currently we call these notifiers once for each CPU of the policy->cpus cpumask. It would be more optimal if the notifier can be called only once and all the relevant information be provided to it. Out of the 24 drivers that register for the transition notifiers today, only 5 of them do per-cpu updates and the callback for the rest can be called only once for the policy without any impact. This would also avoid multiple function calls to the notifier callbacks and reduce multiple iterations of notifier core's code (which does locking as well). This patch adds pointer to the cpufreq policy to the struct cpufreq_freqs, so the notifier callback has all the information available to it with a single call. The five drivers which perform per-cpu updates are updated to use the cpufreq policy. The freqs->cpu field is redundant now and is removed. Acked-by: David S. Miller (sparc) Signed-off-by: Viresh Kumar --- V1->V2: - Add cpufreq policy instead of cpus in struct cpufreq_freqs. - Use policy->cpus instead of related_cpus everywhere in order not to change the existing behavior. - Merged all 7 patches into a single patch. - Updated changlog and included Ack from David. arch/arm/kernel/smp.c | 24 +++++++++++++++--------- arch/arm/kernel/smp_twd.c | 9 ++++++--- arch/sparc/kernel/time_64.c | 28 ++++++++++++++++------------ arch/x86/kernel/tsc.c | 32 +++++++++++++++++++++----------- arch/x86/kvm/x86.c | 31 ++++++++++++++++++++----------- drivers/cpufreq/cpufreq.c | 19 ++++++++++--------- include/linux/cpufreq.h | 14 +++++++------- 7 files changed, 95 insertions(+), 62 deletions(-) -- 2.21.0.rc0.269.g1a574e7a288b diff --git a/arch/arm/kernel/smp.c b/arch/arm/kernel/smp.c index 1d6f5ea522f4..6f6b981fecda 100644 --- a/arch/arm/kernel/smp.c +++ b/arch/arm/kernel/smp.c @@ -758,15 +758,20 @@ static int cpufreq_callback(struct notifier_block *nb, unsigned long val, void *data) { struct cpufreq_freqs *freq = data; - int cpu = freq->cpu; + struct cpumask *cpus = freq->policy->cpus; + int cpu, first = cpumask_first(cpus); + unsigned int lpj; if (freq->flags & CPUFREQ_CONST_LOOPS) return NOTIFY_OK; - if (!per_cpu(l_p_j_ref, cpu)) { - per_cpu(l_p_j_ref, cpu) = - per_cpu(cpu_data, cpu).loops_per_jiffy; - per_cpu(l_p_j_ref_freq, cpu) = freq->old; + if (!per_cpu(l_p_j_ref, first)) { + for_each_cpu(cpu, cpus) { + per_cpu(l_p_j_ref, cpu) = + per_cpu(cpu_data, cpu).loops_per_jiffy; + per_cpu(l_p_j_ref_freq, cpu) = freq->old; + } + if (!global_l_p_j_ref) { global_l_p_j_ref = loops_per_jiffy; global_l_p_j_ref_freq = freq->old; @@ -778,10 +783,11 @@ static int cpufreq_callback(struct notifier_block *nb, loops_per_jiffy = cpufreq_scale(global_l_p_j_ref, global_l_p_j_ref_freq, freq->new); - per_cpu(cpu_data, cpu).loops_per_jiffy = - cpufreq_scale(per_cpu(l_p_j_ref, cpu), - per_cpu(l_p_j_ref_freq, cpu), - freq->new); + + lpj = cpufreq_scale(per_cpu(l_p_j_ref, first), + per_cpu(l_p_j_ref_freq, first), freq->new); + for_each_cpu(cpu, cpus) + per_cpu(cpu_data, cpu).loops_per_jiffy = lpj; } return NOTIFY_OK; } diff --git a/arch/arm/kernel/smp_twd.c b/arch/arm/kernel/smp_twd.c index b30eafeef096..495cc7282096 100644 --- a/arch/arm/kernel/smp_twd.c +++ b/arch/arm/kernel/smp_twd.c @@ -162,15 +162,18 @@ static int twd_cpufreq_transition(struct notifier_block *nb, unsigned long state, void *data) { struct cpufreq_freqs *freqs = data; + int cpu; /* * The twd clock events must be reprogrammed to account for the new * frequency. The timer is local to a cpu, so cross-call to the * changing cpu. */ - if (state == CPUFREQ_POSTCHANGE) - smp_call_function_single(freqs->cpu, twd_update_frequency, - NULL, 1); + if (state != CPUFREQ_POSTCHANGE) + return NOTIFY_OK; + + for_each_cpu(cpu, freqs->policy->cpus) + smp_call_function_single(cpu, twd_update_frequency, NULL, 1); return NOTIFY_OK; } diff --git a/arch/sparc/kernel/time_64.c b/arch/sparc/kernel/time_64.c index 3eb77943ce12..89fb05f90609 100644 --- a/arch/sparc/kernel/time_64.c +++ b/arch/sparc/kernel/time_64.c @@ -653,19 +653,23 @@ static int sparc64_cpufreq_notifier(struct notifier_block *nb, unsigned long val void *data) { struct cpufreq_freqs *freq = data; - unsigned int cpu = freq->cpu; - struct freq_table *ft = &per_cpu(sparc64_freq_table, cpu); + unsigned int cpu; + struct freq_table *ft; - if (!ft->ref_freq) { - ft->ref_freq = freq->old; - ft->clock_tick_ref = cpu_data(cpu).clock_tick; - } - if ((val == CPUFREQ_PRECHANGE && freq->old < freq->new) || - (val == CPUFREQ_POSTCHANGE && freq->old > freq->new)) { - cpu_data(cpu).clock_tick = - cpufreq_scale(ft->clock_tick_ref, - ft->ref_freq, - freq->new); + for_each_cpu(cpu, freq->policy->cpus) { + ft = &per_cpu(sparc64_freq_table, cpu); + + if (!ft->ref_freq) { + ft->ref_freq = freq->old; + ft->clock_tick_ref = cpu_data(cpu).clock_tick; + } + + if ((val == CPUFREQ_PRECHANGE && freq->old < freq->new) || + (val == CPUFREQ_POSTCHANGE && freq->old > freq->new)) { + cpu_data(cpu).clock_tick = + cpufreq_scale(ft->clock_tick_ref, ft->ref_freq, + freq->new); + } } return 0; diff --git a/arch/x86/kernel/tsc.c b/arch/x86/kernel/tsc.c index 3fae23834069..cff8779fc0d2 100644 --- a/arch/x86/kernel/tsc.c +++ b/arch/x86/kernel/tsc.c @@ -956,28 +956,38 @@ static int time_cpufreq_notifier(struct notifier_block *nb, unsigned long val, void *data) { struct cpufreq_freqs *freq = data; - unsigned long *lpj; - - lpj = &boot_cpu_data.loops_per_jiffy; -#ifdef CONFIG_SMP - if (!(freq->flags & CPUFREQ_CONST_LOOPS)) - lpj = &cpu_data(freq->cpu).loops_per_jiffy; -#endif + struct cpumask *cpus = freq->policy->cpus; + bool boot_cpu = !IS_ENABLED(CONFIG_SMP) || freq->flags & CPUFREQ_CONST_LOOPS; + unsigned long lpj; + int cpu; if (!ref_freq) { ref_freq = freq->old; - loops_per_jiffy_ref = *lpj; tsc_khz_ref = tsc_khz; + + if (boot_cpu) + loops_per_jiffy_ref = boot_cpu_data.loops_per_jiffy; + else + loops_per_jiffy_ref = cpu_data(cpumask_first(cpus)).loops_per_jiffy; } + if ((val == CPUFREQ_PRECHANGE && freq->old < freq->new) || (val == CPUFREQ_POSTCHANGE && freq->old > freq->new)) { - *lpj = cpufreq_scale(loops_per_jiffy_ref, ref_freq, freq->new); - + lpj = cpufreq_scale(loops_per_jiffy_ref, ref_freq, freq->new); tsc_khz = cpufreq_scale(tsc_khz_ref, ref_freq, freq->new); + if (!(freq->flags & CPUFREQ_CONST_LOOPS)) mark_tsc_unstable("cpufreq changes"); - set_cyc2ns_scale(tsc_khz, freq->cpu, rdtsc()); + if (boot_cpu) { + boot_cpu_data.loops_per_jiffy = lpj; + } else { + for_each_cpu(cpu, cpus) + cpu_data(cpu).loops_per_jiffy = lpj; + } + + for_each_cpu(cpu, cpus) + set_cyc2ns_scale(tsc_khz, cpu, rdtsc()); } return 0; diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index 941f932373d0..653c7da11647 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -6648,10 +6648,8 @@ static void kvm_hyperv_tsc_notifier(void) } #endif -static int kvmclock_cpufreq_notifier(struct notifier_block *nb, unsigned long val, - void *data) +static void __kvmclock_cpufreq_notifier(struct cpufreq_freqs *freq, int cpu) { - struct cpufreq_freqs *freq = data; struct kvm *kvm; struct kvm_vcpu *vcpu; int i, send_ipi = 0; @@ -6695,17 +6693,12 @@ static int kvmclock_cpufreq_notifier(struct notifier_block *nb, unsigned long va * */ - if (val == CPUFREQ_PRECHANGE && freq->old > freq->new) - return 0; - if (val == CPUFREQ_POSTCHANGE && freq->old < freq->new) - return 0; - - smp_call_function_single(freq->cpu, tsc_khz_changed, freq, 1); + smp_call_function_single(cpu, tsc_khz_changed, freq, 1); spin_lock(&kvm_lock); list_for_each_entry(kvm, &vm_list, vm_list) { kvm_for_each_vcpu(i, vcpu, kvm) { - if (vcpu->cpu != freq->cpu) + if (vcpu->cpu != cpu) continue; kvm_make_request(KVM_REQ_CLOCK_UPDATE, vcpu); if (vcpu->cpu != smp_processor_id()) @@ -6727,8 +6720,24 @@ static int kvmclock_cpufreq_notifier(struct notifier_block *nb, unsigned long va * guest context is entered kvmclock will be updated, * so the guest will not see stale values. */ - smp_call_function_single(freq->cpu, tsc_khz_changed, freq, 1); + smp_call_function_single(cpu, tsc_khz_changed, freq, 1); } +} + +static int kvmclock_cpufreq_notifier(struct notifier_block *nb, unsigned long val, + void *data) +{ + struct cpufreq_freqs *freq = data; + int cpu; + + if (val == CPUFREQ_PRECHANGE && freq->old > freq->new) + return 0; + if (val == CPUFREQ_POSTCHANGE && freq->old < freq->new) + return 0; + + for_each_cpu(cpu, freq->policy->cpus) + __kvmclock_cpufreq_notifier(freq, cpu); + return 0; } diff --git a/drivers/cpufreq/cpufreq.c b/drivers/cpufreq/cpufreq.c index e10922709d13..fba38bf27d26 100644 --- a/drivers/cpufreq/cpufreq.c +++ b/drivers/cpufreq/cpufreq.c @@ -300,11 +300,14 @@ static void cpufreq_notify_transition(struct cpufreq_policy *policy, struct cpufreq_freqs *freqs, unsigned int state) { + int cpu; + BUG_ON(irqs_disabled()); if (cpufreq_disabled()) return; + freqs->policy = policy; freqs->flags = cpufreq_driver->flags; pr_debug("notification %u of frequency transition to %u kHz\n", state, freqs->new); @@ -324,10 +327,8 @@ static void cpufreq_notify_transition(struct cpufreq_policy *policy, } } - for_each_cpu(freqs->cpu, policy->cpus) { - srcu_notifier_call_chain(&cpufreq_transition_notifier_list, - CPUFREQ_PRECHANGE, freqs); - } + srcu_notifier_call_chain(&cpufreq_transition_notifier_list, + CPUFREQ_PRECHANGE, freqs); adjust_jiffies(CPUFREQ_PRECHANGE, freqs); break; @@ -337,11 +338,11 @@ static void cpufreq_notify_transition(struct cpufreq_policy *policy, pr_debug("FREQ: %u - CPUs: %*pbl\n", freqs->new, cpumask_pr_args(policy->cpus)); - for_each_cpu(freqs->cpu, policy->cpus) { - trace_cpu_frequency(freqs->new, freqs->cpu); - srcu_notifier_call_chain(&cpufreq_transition_notifier_list, - CPUFREQ_POSTCHANGE, freqs); - } + for_each_cpu(cpu, policy->cpus) + trace_cpu_frequency(freqs->new, cpu); + + srcu_notifier_call_chain(&cpufreq_transition_notifier_list, + CPUFREQ_POSTCHANGE, freqs); cpufreq_stats_record_transition(policy, freqs->new); policy->cur = freqs->new; diff --git a/include/linux/cpufreq.h b/include/linux/cpufreq.h index b160e98076e3..e327523ddff2 100644 --- a/include/linux/cpufreq.h +++ b/include/linux/cpufreq.h @@ -42,13 +42,6 @@ enum cpufreq_table_sorting { CPUFREQ_TABLE_SORTED_DESCENDING }; -struct cpufreq_freqs { - unsigned int cpu; /* cpu nr */ - unsigned int old; - unsigned int new; - u8 flags; /* flags of cpufreq_driver, see below. */ -}; - struct cpufreq_cpuinfo { unsigned int max_freq; unsigned int min_freq; @@ -156,6 +149,13 @@ struct cpufreq_policy { struct thermal_cooling_device *cdev; }; +struct cpufreq_freqs { + struct cpufreq_policy *policy; + unsigned int old; + unsigned int new; + u8 flags; /* flags of cpufreq_driver, see below. */ +}; + /* Only for ACPI */ #define CPUFREQ_SHARED_TYPE_NONE (0) /* None */ #define CPUFREQ_SHARED_TYPE_HW (1) /* HW does needed coordination */