From patchwork Wed Mar 20 04:52:23 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Viresh Kumar X-Patchwork-Id: 160622 Delivered-To: patch@linaro.org Received: by 2002:a02:5cc1:0:0:0:0:0 with SMTP id w62csp367747jad; Tue, 19 Mar 2019 21:52:44 -0700 (PDT) X-Google-Smtp-Source: APXvYqxIB2AuljJWRNvyArt5nRAUidN737lkHPzngZXqUKJJ3f2tHtCz63Qj7T5LfULrR8E4Pb3E X-Received: by 2002:a62:2e07:: with SMTP id u7mr5529949pfu.176.1553057564640; Tue, 19 Mar 2019 21:52:44 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1553057564; cv=none; d=google.com; s=arc-20160816; b=fGcp+7jfEugX1gxI0TKDmauEbShNs9P1ovr62ZrB4nSjq+KMNqi61vpPmAd9xMngT3 DKgURIXjrgZ0wgec13vceLK+Fn0J68+lcvlB8OYiV6akDdUSTYcD+C3yb//7gAN+E9Z1 pGIvlWhLWkccIgvJYyAPd0zJCOY+PfW6uXLx91/gb6Cz9nKgj4lCi0IGB20rCYclSm9g g9Xq4mju0s1stC8Tm61KMKEkW5hFlWFFIFrvyfAWjK3fV0WwAXa3yI/h+9vMRjx8Eu4W /qPQl7p9AkkR1mS/g2QBxGkyI040hZBF1oR2taINFRCRq3O7I+A4WVdlg4FL07EAVCdP I43Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:content-transfer-encoding:mime-version :message-id:date:subject:cc:to:from:dkim-signature; bh=rLbMvVYD9xo2PePz/vY4EcBjG9UXoqRREwseQKk9Xbk=; b=Mplzn6LJIqfapQlsfxzKYAIa+qISHsxnVYH3u8TfKfLQkFSYJ9hIA+owZUUbVeT48X kLWkhx/Yp+8jp2A5Z5OjWPFXv7O5j7O6pCSebbNiFnvxMlHb55gsgjzgy30O+gM9U5j2 Onv4cqsDZtFyFfMsgBsYxunm7sGrrumm18L5DUzU7wW4hgB8EC0l7jH2X438ioT5sYX/ pIpEcQT7OMVCjDVfDM7IOzv22XoNMPTVCMj1ZD9MPeVMeO0AdZ9//F45JqkSiECkXDuY 6cS/s+kNPXV8XrGSYsZIOcorCYwuau/z561Mo4Y1IjQ6hbaGw8Pxgbs9X/zLDJ+I72CD i4vA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=uOuNHEhC; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id e130si765873pfc.264.2019.03.19.21.52.43; Tue, 19 Mar 2019 21:52:44 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=uOuNHEhC; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726492AbfCTEwl (ORCPT + 31 others); Wed, 20 Mar 2019 00:52:41 -0400 Received: from mail-pf1-f195.google.com ([209.85.210.195]:34701 "EHLO mail-pf1-f195.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725996AbfCTEwk (ORCPT ); Wed, 20 Mar 2019 00:52:40 -0400 Received: by mail-pf1-f195.google.com with SMTP id v64so1008866pfb.1 for ; Tue, 19 Mar 2019 21:52:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=rLbMvVYD9xo2PePz/vY4EcBjG9UXoqRREwseQKk9Xbk=; b=uOuNHEhCO6727iMMQlA5YbP7AfYNkROruI6tnTrn5wBEvnQ6it3BYIBS5XkV/9TaYA hkCmH9J9cSg3gaduutp1NKwmUgttpujp99egQaQrBvpC1zMm1Ze5ahHg5Zhnbtbl6q1M ohwbglCuGJFe+76YvUqvw6w4/EfsJGzJjP0YdO5RRX7EEuStoI7fr1bRs4kH4DrhZFCz 0jiZ8hyyXbj+jIGsYpq7SEjqp3tiEAu8fprgpBiFupZW8ltPpZDy1zSDD3HmZ3HwmCw1 i01wRpF3h5YtIitnIpQSM2SVM/z5lkhBt2dKil5PptG/qvKo8AZngrsPK/L4d6QQ5sxJ vLTA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=rLbMvVYD9xo2PePz/vY4EcBjG9UXoqRREwseQKk9Xbk=; b=lszDRWVokc0Ogdofkg3qg4lGd5LCOlnS7TTGUn61gyB/Cho9DoomO2Z3/HEvMfN7jc ef/QdS3j9Fz/SDx9tMqe6M+QuBIJDAEyO/Nddm4+M3Xf4Trnm8nA90QlmRxkgOc42NTD gzRqpry6DpE6dYecCNaPhwt1tMnbgwjvn+A14xrETqdzL0sMotROqpbx/1BMSHiz+Mlw ljTdFeBe8Ys/1pGa3tv8+m2dP+Z42piJjEuXpdJAQsQS8tlUs6+DKHIWrrGP/hLprZXZ RJk+ZQoOqezH6aioI1ja3D6OuHWGtl8Q4gxqmS/8d4fOXymNwSk94ZdhtCCnqjBOjb1a 95lQ== X-Gm-Message-State: APjAAAW3Bvfkx65y55HquozZdPMnUuSMCClH+OHshqo14PqEEJPYpvxE tTRs2PsxjEeq+XaBwQ/vt73qNQ== X-Received: by 2002:a65:6299:: with SMTP id f25mr5457014pgv.376.1553057559839; Tue, 19 Mar 2019 21:52:39 -0700 (PDT) Received: from localhost ([122.166.134.37]) by smtp.gmail.com with ESMTPSA id k17sm836763pfk.166.2019.03.19.21.52.37 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 19 Mar 2019 21:52:38 -0700 (PDT) From: Viresh Kumar To: Rafael Wysocki , Peter Zijlstra , Russell King , "David S. Miller" , Thomas Gleixner , Ingo Molnar , Borislav Petkov , "H. Peter Anvin" , x86@kernel.org, Paolo Bonzini , =?utf-8?q?Radim_Kr=C4=8Dm?= =?utf-8?b?w6HFmQ==?= Cc: Viresh Kumar , linux-pm@vger.kernel.org, Vincent Guittot , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, sparclinux@vger.kernel.org, kvm@vger.kernel.org Subject: [PATCH V3] cpufreq: Call transition notifier only once for each policy Date: Wed, 20 Mar 2019 10:22:23 +0530 Message-Id: <1eb27e3bbcbb2c67e6eadc0893c9b41e5d76894b.1553057341.git.viresh.kumar@linaro.org> X-Mailer: git-send-email 2.21.0.rc0.269.g1a574e7a288b MIME-Version: 1.0 Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Currently we call these notifiers once for each CPU of the policy->cpus cpumask. It would be more optimal if the notifier can be called only once and all the relevant information be provided to it. Out of the 23 drivers that register for the transition notifiers today, only 4 of them do per-cpu updates and the callback for the rest can be called only once for the policy without any impact. This would also avoid multiple function calls to the notifier callbacks and reduce multiple iterations of notifier core's code (which does locking as well). This patch adds pointer to the cpufreq policy to the struct cpufreq_freqs, so the notifier callback has all the information available to it with a single call. The five drivers which perform per-cpu updates are updated to use the cpufreq policy. The freqs->cpu field is redundant now and is removed. Acked-by: David S. Miller (sparc) Signed-off-by: Viresh Kumar --- V2->V3: - Drop changes for arch/arm/kernel/smp_twd.c as the notifier is removed in 5.1-rc1. - Changed implementation in tsc.c as suggested by Rafael and Peterz. We now WARN if more than one CPU is present in the policy. - Rebased over 5.1-rc1. arch/arm/kernel/smp.c | 24 +++++++++++++++--------- arch/sparc/kernel/time_64.c | 28 ++++++++++++++++------------ arch/x86/kernel/tsc.c | 9 +++++++-- arch/x86/kvm/x86.c | 31 ++++++++++++++++++++----------- drivers/cpufreq/cpufreq.c | 19 ++++++++++--------- include/linux/cpufreq.h | 14 +++++++------- 6 files changed, 75 insertions(+), 50 deletions(-) -- 2.21.0.rc0.269.g1a574e7a288b diff --git a/arch/arm/kernel/smp.c b/arch/arm/kernel/smp.c index facd4240ca02..c6d37563610a 100644 --- a/arch/arm/kernel/smp.c +++ b/arch/arm/kernel/smp.c @@ -754,15 +754,20 @@ static int cpufreq_callback(struct notifier_block *nb, unsigned long val, void *data) { struct cpufreq_freqs *freq = data; - int cpu = freq->cpu; + struct cpumask *cpus = freq->policy->cpus; + int cpu, first = cpumask_first(cpus); + unsigned int lpj; if (freq->flags & CPUFREQ_CONST_LOOPS) return NOTIFY_OK; - if (!per_cpu(l_p_j_ref, cpu)) { - per_cpu(l_p_j_ref, cpu) = - per_cpu(cpu_data, cpu).loops_per_jiffy; - per_cpu(l_p_j_ref_freq, cpu) = freq->old; + if (!per_cpu(l_p_j_ref, first)) { + for_each_cpu(cpu, cpus) { + per_cpu(l_p_j_ref, cpu) = + per_cpu(cpu_data, cpu).loops_per_jiffy; + per_cpu(l_p_j_ref_freq, cpu) = freq->old; + } + if (!global_l_p_j_ref) { global_l_p_j_ref = loops_per_jiffy; global_l_p_j_ref_freq = freq->old; @@ -774,10 +779,11 @@ static int cpufreq_callback(struct notifier_block *nb, loops_per_jiffy = cpufreq_scale(global_l_p_j_ref, global_l_p_j_ref_freq, freq->new); - per_cpu(cpu_data, cpu).loops_per_jiffy = - cpufreq_scale(per_cpu(l_p_j_ref, cpu), - per_cpu(l_p_j_ref_freq, cpu), - freq->new); + + lpj = cpufreq_scale(per_cpu(l_p_j_ref, first), + per_cpu(l_p_j_ref_freq, first), freq->new); + for_each_cpu(cpu, cpus) + per_cpu(cpu_data, cpu).loops_per_jiffy = lpj; } return NOTIFY_OK; } diff --git a/arch/sparc/kernel/time_64.c b/arch/sparc/kernel/time_64.c index 3eb77943ce12..89fb05f90609 100644 --- a/arch/sparc/kernel/time_64.c +++ b/arch/sparc/kernel/time_64.c @@ -653,19 +653,23 @@ static int sparc64_cpufreq_notifier(struct notifier_block *nb, unsigned long val void *data) { struct cpufreq_freqs *freq = data; - unsigned int cpu = freq->cpu; - struct freq_table *ft = &per_cpu(sparc64_freq_table, cpu); + unsigned int cpu; + struct freq_table *ft; - if (!ft->ref_freq) { - ft->ref_freq = freq->old; - ft->clock_tick_ref = cpu_data(cpu).clock_tick; - } - if ((val == CPUFREQ_PRECHANGE && freq->old < freq->new) || - (val == CPUFREQ_POSTCHANGE && freq->old > freq->new)) { - cpu_data(cpu).clock_tick = - cpufreq_scale(ft->clock_tick_ref, - ft->ref_freq, - freq->new); + for_each_cpu(cpu, freq->policy->cpus) { + ft = &per_cpu(sparc64_freq_table, cpu); + + if (!ft->ref_freq) { + ft->ref_freq = freq->old; + ft->clock_tick_ref = cpu_data(cpu).clock_tick; + } + + if ((val == CPUFREQ_PRECHANGE && freq->old < freq->new) || + (val == CPUFREQ_POSTCHANGE && freq->old > freq->new)) { + cpu_data(cpu).clock_tick = + cpufreq_scale(ft->clock_tick_ref, ft->ref_freq, + freq->new); + } } return 0; diff --git a/arch/x86/kernel/tsc.c b/arch/x86/kernel/tsc.c index 3fae23834069..b2fe665878f7 100644 --- a/arch/x86/kernel/tsc.c +++ b/arch/x86/kernel/tsc.c @@ -958,10 +958,15 @@ static int time_cpufreq_notifier(struct notifier_block *nb, unsigned long val, struct cpufreq_freqs *freq = data; unsigned long *lpj; + if (WARN_ON_ONCE(cpumask_weight(freq->policy->related_cpus) != 1)) { + mark_tsc_unstable("cpufreq changes: related CPUs affected"); + return 0; + } + lpj = &boot_cpu_data.loops_per_jiffy; #ifdef CONFIG_SMP if (!(freq->flags & CPUFREQ_CONST_LOOPS)) - lpj = &cpu_data(freq->cpu).loops_per_jiffy; + lpj = &cpu_data(freq->policy->cpu).loops_per_jiffy; #endif if (!ref_freq) { @@ -977,7 +982,7 @@ static int time_cpufreq_notifier(struct notifier_block *nb, unsigned long val, if (!(freq->flags & CPUFREQ_CONST_LOOPS)) mark_tsc_unstable("cpufreq changes"); - set_cyc2ns_scale(tsc_khz, freq->cpu, rdtsc()); + set_cyc2ns_scale(tsc_khz, freq->policy->cpu, rdtsc()); } return 0; diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index 65e4559eef2f..1ac8c710cccc 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -6649,10 +6649,8 @@ static void kvm_hyperv_tsc_notifier(void) } #endif -static int kvmclock_cpufreq_notifier(struct notifier_block *nb, unsigned long val, - void *data) +static void __kvmclock_cpufreq_notifier(struct cpufreq_freqs *freq, int cpu) { - struct cpufreq_freqs *freq = data; struct kvm *kvm; struct kvm_vcpu *vcpu; int i, send_ipi = 0; @@ -6696,17 +6694,12 @@ static int kvmclock_cpufreq_notifier(struct notifier_block *nb, unsigned long va * */ - if (val == CPUFREQ_PRECHANGE && freq->old > freq->new) - return 0; - if (val == CPUFREQ_POSTCHANGE && freq->old < freq->new) - return 0; - - smp_call_function_single(freq->cpu, tsc_khz_changed, freq, 1); + smp_call_function_single(cpu, tsc_khz_changed, freq, 1); spin_lock(&kvm_lock); list_for_each_entry(kvm, &vm_list, vm_list) { kvm_for_each_vcpu(i, vcpu, kvm) { - if (vcpu->cpu != freq->cpu) + if (vcpu->cpu != cpu) continue; kvm_make_request(KVM_REQ_CLOCK_UPDATE, vcpu); if (vcpu->cpu != smp_processor_id()) @@ -6728,8 +6721,24 @@ static int kvmclock_cpufreq_notifier(struct notifier_block *nb, unsigned long va * guest context is entered kvmclock will be updated, * so the guest will not see stale values. */ - smp_call_function_single(freq->cpu, tsc_khz_changed, freq, 1); + smp_call_function_single(cpu, tsc_khz_changed, freq, 1); } +} + +static int kvmclock_cpufreq_notifier(struct notifier_block *nb, unsigned long val, + void *data) +{ + struct cpufreq_freqs *freq = data; + int cpu; + + if (val == CPUFREQ_PRECHANGE && freq->old > freq->new) + return 0; + if (val == CPUFREQ_POSTCHANGE && freq->old < freq->new) + return 0; + + for_each_cpu(cpu, freq->policy->cpus) + __kvmclock_cpufreq_notifier(freq, cpu); + return 0; } diff --git a/drivers/cpufreq/cpufreq.c b/drivers/cpufreq/cpufreq.c index e10922709d13..fba38bf27d26 100644 --- a/drivers/cpufreq/cpufreq.c +++ b/drivers/cpufreq/cpufreq.c @@ -300,11 +300,14 @@ static void cpufreq_notify_transition(struct cpufreq_policy *policy, struct cpufreq_freqs *freqs, unsigned int state) { + int cpu; + BUG_ON(irqs_disabled()); if (cpufreq_disabled()) return; + freqs->policy = policy; freqs->flags = cpufreq_driver->flags; pr_debug("notification %u of frequency transition to %u kHz\n", state, freqs->new); @@ -324,10 +327,8 @@ static void cpufreq_notify_transition(struct cpufreq_policy *policy, } } - for_each_cpu(freqs->cpu, policy->cpus) { - srcu_notifier_call_chain(&cpufreq_transition_notifier_list, - CPUFREQ_PRECHANGE, freqs); - } + srcu_notifier_call_chain(&cpufreq_transition_notifier_list, + CPUFREQ_PRECHANGE, freqs); adjust_jiffies(CPUFREQ_PRECHANGE, freqs); break; @@ -337,11 +338,11 @@ static void cpufreq_notify_transition(struct cpufreq_policy *policy, pr_debug("FREQ: %u - CPUs: %*pbl\n", freqs->new, cpumask_pr_args(policy->cpus)); - for_each_cpu(freqs->cpu, policy->cpus) { - trace_cpu_frequency(freqs->new, freqs->cpu); - srcu_notifier_call_chain(&cpufreq_transition_notifier_list, - CPUFREQ_POSTCHANGE, freqs); - } + for_each_cpu(cpu, policy->cpus) + trace_cpu_frequency(freqs->new, cpu); + + srcu_notifier_call_chain(&cpufreq_transition_notifier_list, + CPUFREQ_POSTCHANGE, freqs); cpufreq_stats_record_transition(policy, freqs->new); policy->cur = freqs->new; diff --git a/include/linux/cpufreq.h b/include/linux/cpufreq.h index b160e98076e3..e327523ddff2 100644 --- a/include/linux/cpufreq.h +++ b/include/linux/cpufreq.h @@ -42,13 +42,6 @@ enum cpufreq_table_sorting { CPUFREQ_TABLE_SORTED_DESCENDING }; -struct cpufreq_freqs { - unsigned int cpu; /* cpu nr */ - unsigned int old; - unsigned int new; - u8 flags; /* flags of cpufreq_driver, see below. */ -}; - struct cpufreq_cpuinfo { unsigned int max_freq; unsigned int min_freq; @@ -156,6 +149,13 @@ struct cpufreq_policy { struct thermal_cooling_device *cdev; }; +struct cpufreq_freqs { + struct cpufreq_policy *policy; + unsigned int old; + unsigned int new; + u8 flags; /* flags of cpufreq_driver, see below. */ +}; + /* Only for ACPI */ #define CPUFREQ_SHARED_TYPE_NONE (0) /* None */ #define CPUFREQ_SHARED_TYPE_HW (1) /* HW does needed coordination */