From patchwork Tue Apr 15 10:04:21 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Rafael J. Wysocki" X-Patchwork-Id: 881589 Received: from cloudserver094114.home.pl (cloudserver094114.home.pl [79.96.170.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A1D7A1CAA6D; Tue, 15 Apr 2025 10:17:37 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=79.96.170.134 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1744712259; cv=none; b=pchg2b5jiuaKqIKxBNfxuyWPBC2fHECcWIjdhxK/w06XJc1V50+sukESBbr9jBTPUSesdKqWtAQ73yCzWg4r6FpBwoXAV1GLQ9IqJX+0rn5eUZbj9sxdrbkMEFoB3geNKbviixAVHEAoiKsFqNCqpwSOqF9cb3cRLOpsX7RtKss= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1744712259; c=relaxed/simple; bh=yfxZC1j8MoskcJ1HEXZgO0OGbDvcF6mnmjMlnsHGWp4=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=ijXDHOVu9inin73hpz7HnxlnQyQEf8NodbG+CTddo7FWhdDNKi7ew9fX2CAucdj91JyK6LiylelEw4tLUp9DNxIk+1Rrsi5XeJchQl8G7C9P/XhwnxHAgNFoDwCd1xrfFt05nka1aXv8gmyIJqNcPsHnkqqTu0J4mVS/73HADOc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=rjwysocki.net; spf=pass smtp.mailfrom=rjwysocki.net; dkim=pass (2048-bit key) header.d=rjwysocki.net header.i=@rjwysocki.net header.b=VpC8rBD1; arc=none smtp.client-ip=79.96.170.134 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=rjwysocki.net Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=rjwysocki.net Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=rjwysocki.net header.i=@rjwysocki.net header.b="VpC8rBD1" Received: from kreacher.localnet (unknown [195.136.19.94]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by cloudserver094114.home.pl (Postfix) with ESMTPSA id B67D9662668; Tue, 15 Apr 2025 12:17:33 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=rjwysocki.net; s=dkim; t=1744712254; bh=yfxZC1j8MoskcJ1HEXZgO0OGbDvcF6mnmjMlnsHGWp4=; h=From:Subject:Date; b=VpC8rBD12t96jhMkhy2V+uTSZ3rVq/tMV1xZgQNWQL90Y8KwqmgCGkLUUFyR6UEX5 axUs2zOvvi3qzx83+OzXK3+QjqG3DFtMBS+uxsJl5yfvUaFwaHxTKt942eWOknexDQ WLus4fJSTe9zZTXSenrRAvuDuov/yIMeojCCaMogWfXk/z6qBIcJvVL7QcKTB8y6J5 7mzPPHf1V5szZHp9c7GgXWEAt0nO3a6JWINpZsPvDg7ZWYYrDZeX4LlR9NAgKsZ/vN 9HTvynGmYH1vYIxB0ruMeKC4KlR/7nzQ+9/Bq6qCif3H7XAwK0IqeDyE8yZ1Kyi6ma y9raQwC4SufEA== From: "Rafael J. Wysocki" To: Linux PM Cc: LKML , Viresh Kumar , Srinivas Pandruvada , Mario Limonciello , Vincent Guittot , Christian Loehle , Sultan Alsawaf , Peter Zijlstra , Valentin Schneider , Ingo Molnar Subject: [PATCH v2 5/6] cpufreq: Avoid using inconsistent policy->min and policy->max Date: Tue, 15 Apr 2025 12:04:21 +0200 Message-ID: <9458818.CDJkKcVGEf@rjwysocki.net> In-Reply-To: <6171293.lOV4Wx5bFT@rjwysocki.net> References: <6171293.lOV4Wx5bFT@rjwysocki.net> Precedence: bulk X-Mailing-List: linux-pm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-CLIENT-IP: 195.136.19.94 X-CLIENT-HOSTNAME: 195.136.19.94 X-VADE-SPAMSTATE: clean X-VADE-SPAMCAUSE: gggruggvucftvghtrhhoucdtuddrgeefvddrtddtgddvvdefvdefucetufdoteggodetrfdotffvucfrrhhofhhilhgvmecujffqoffgrffnpdggtffipffknecuuegrihhlohhuthemucduhedtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenucfjughrpefhvfevufffkfgjfhgggfgtsehtufertddttdejnecuhfhrohhmpedftfgrfhgrvghlucflrdcuhgihshhotghkihdfuceorhhjfiesrhhjfiihshhotghkihdrnhgvtheqnecuggftrfgrthhtvghrnhepvdffueeitdfgvddtudegueejtdffteetgeefkeffvdeftddttdeuhfegfedvjefhnecukfhppeduleehrddufeeirdduledrleegnecuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehinhgvthepudelhedrudefiedrudelrdelgedphhgvlhhopehkrhgvrggthhgvrhdrlhhotggrlhhnvghtpdhmrghilhhfrhhomheprhhjfiesrhhjfiihshhotghkihdrnhgvthdpnhgspghrtghpthhtohepuddupdhrtghpthhtoheplhhinhhugidqphhmsehvghgvrhdrkhgvrhhnvghlrdhorhhgpdhrtghpthhtoheplhhinhhugidqkhgvrhhnvghlsehvghgvrhdrkhgvrhhnvghlrdhorhhgpdhrtghpthhtohepvhhirhgvshhhrdhkuhhmrghrsehlihhnrghrohdrohhrghdprhgtphhtthhopehsrhhinhhivhgrshdrphgrnhgurhhuvhgruggrsehlihhnuhigrdhinhhtvghlrdgtohhmpdhrtghpthhtohepmhgrrhhiohdrlhh X-DCC--Metrics: v370.home.net.pl 1024; Body=11 Fuz1=11 Fuz2=11 From: Rafael J. Wysocki Since cpufreq_driver_resolve_freq() can run in parallel with cpufreq_set_policy() and there is no synchronization between them, the former may access policy->min and policy->max while the latter is updating them and it may see intermediate values of them due to the way the update is carried out. Also the compiler is free to apply any optimizations it wants both to the stores in cpufreq_set_policy() and to the loads in cpufreq_driver_resolve_freq() which may result in additional inconsistencies. To address this, use WRITE_ONCE() when updating policy->min and policy->max in cpufreq_set_policy() and use READ_ONCE() for reading them in cpufreq_driver_resolve_freq(). Moreover, rearrange the update in cpufreq_set_policy() to avoid storing intermediate values in policy->min and policy->max with the help of the observation that their new values are expected to be properly ordered upfront. Also modify cpufreq_driver_resolve_freq() to take the possible reverse ordering of policy->min and policy->max, which may happen depending on the ordering of operations when this function and cpufreq_set_policy() run concurrently, into account by always honoring the max when it turns out to be less than the min (in case it comes from thermal throttling or similar). Fixes: 151717690694 ("cpufreq: Make policy min/max hard requirements") Cc: 5.16+ # 5.16+ Signed-off-by: Rafael J. Wysocki --- v1 -> v2: Minor edit in the subject --- drivers/cpufreq/cpufreq.c | 46 ++++++++++++++++++++++++++++++++++++---------- 1 file changed, 36 insertions(+), 10 deletions(-) --- a/drivers/cpufreq/cpufreq.c +++ b/drivers/cpufreq/cpufreq.c @@ -490,14 +490,12 @@ } EXPORT_SYMBOL_GPL(cpufreq_disable_fast_switch); -static unsigned int clamp_and_resolve_freq(struct cpufreq_policy *policy, - unsigned int target_freq, - unsigned int relation) +static unsigned int __resolve_freq(struct cpufreq_policy *policy, + unsigned int target_freq, + unsigned int relation) { unsigned int idx; - target_freq = clamp_val(target_freq, policy->min, policy->max); - if (!policy->freq_table) return target_freq; @@ -507,6 +505,15 @@ return policy->freq_table[idx].frequency; } +static unsigned int clamp_and_resolve_freq(struct cpufreq_policy *policy, + unsigned int target_freq, + unsigned int relation) +{ + target_freq = clamp_val(target_freq, policy->min, policy->max); + + return __resolve_freq(policy, target_freq, relation); +} + /** * cpufreq_driver_resolve_freq - Map a target frequency to a driver-supported * one. @@ -521,7 +528,22 @@ unsigned int cpufreq_driver_resolve_freq(struct cpufreq_policy *policy, unsigned int target_freq) { - return clamp_and_resolve_freq(policy, target_freq, CPUFREQ_RELATION_LE); + unsigned int min = READ_ONCE(policy->min); + unsigned int max = READ_ONCE(policy->max); + + /* + * If this function runs in parallel with cpufreq_set_policy(), it may + * read policy->min before the update and policy->max after the update + * or the other way around, so there is no ordering guarantee. + * + * Resolve this by always honoring the max (in case it comes from + * thermal throttling or similar). + */ + if (unlikely(min > max)) + min = max; + + return __resolve_freq(policy, clamp_val(target_freq, min, max), + CPUFREQ_RELATION_LE); } EXPORT_SYMBOL_GPL(cpufreq_driver_resolve_freq); @@ -2632,11 +2654,15 @@ * Resolve policy min/max to available frequencies. It ensures * no frequency resolution will neither overshoot the requested maximum * nor undershoot the requested minimum. + * + * Avoid storing intermediate values in policy->max or policy->min and + * compiler optimizations around them because them may be accessed + * concurrently by cpufreq_driver_resolve_freq() during the update. */ - policy->min = new_data.min; - policy->max = new_data.max; - policy->min = clamp_and_resolve_freq(policy, policy->min, CPUFREQ_RELATION_L); - policy->max = clamp_and_resolve_freq(policy, policy->max, CPUFREQ_RELATION_H); + WRITE_ONCE(policy->max, __resolve_freq(policy, new_data.max, CPUFREQ_RELATION_H)); + new_data.min = __resolve_freq(policy, new_data.min, CPUFREQ_RELATION_L); + WRITE_ONCE(policy->min, new_data.min > policy->max ? policy->max : new_data.min); + trace_cpu_frequency_limits(policy); cpufreq_update_pressure(policy);