mbox series

[v7,00/23] Introduce runtime modifiable Energy Model

Message ID 20240117095714.1524808-1-lukasz.luba@arm.com
Headers show
Series Introduce runtime modifiable Energy Model | expand

Message

Lukasz Luba Jan. 17, 2024, 9:56 a.m. UTC
Hi all,

This patch set adds a new feature which allows to modify Energy Model (EM)
power values at runtime. It will allow to better reflect power model of
a recent SoCs and silicon. Different characteristics of the power usage
can be leveraged and thus better decisions made during task placement in EAS.

It also optimizes the EAS code hot path, by removing 2 division and 1
multiplication operations in the em_cpu_energy(). Speed-up results:
the em_cpu_energy() should run faster on the Big CPU by 1.43x and on the
Little CPU by 1.69x (mainline board RockPi 4B).

This patch set is part of feature set known as Dynamic Energy Model. It has been
presented and discussed recently at OSPM2023 [3].


The concepts:
1. The CPU power usage can vary due to the workload that it's running or due
to the temperature of the SoC. The same workload can use more power when the
temperature of the silicon has increased (e.g. due to hot GPU or ISP).
In such situation the EM can be adjusted and reflect the fact of increased
power usage. That power increase is due to static power
(sometimes called simply: leakage). The CPUs in recent SoCs are different.
We have heterogeneous SoCs with 3 (or even 4) different microarchitectures.
They are also built differently with High Performance (HP) cells or
Low Power (LP) cells. They are affected by the temperature increase
differently: HP cells have bigger leakage. The SW model can leverage that
knowledge.

2. It is also possible to change the EM to better reflect the currently
running workload. Usually the EM is derived from some average power values
taken from experiments with benchmark (e.g. Dhrystone). The model derived
from such scenario might not represent properly the workloads usually running
on the device. Therefore, runtime modification of the EM allows to switch to
a different model, when there is a need.

3. The EM can be adjusted after boot, when all the modules are loaded and
more information about the SoC is available e.g. chip binning. This would help
to better reflect the silicon characteristics. Thus, this EM modification
API allows it now. It wasn't possible in the past and the EM had to be
'set in stone'.

Example of such runtime modification after boot can be found in a follow-up
patch set. It adds the OPP API and usage in Exynos5 SoC driver after the
voltage values has been adjusted and power changes [5].

More detailed explanation and background can be found in presentations
during LPC2022 [1][2] or in the documentation patches.

Some test results:
The EM can be updated to fit better the workload type. In the case below the EM
has been updated for the Jankbench test on Pixel6 (running v5.18 w/ mainline backports
for the scheduler bits). The Jankbench was run 10 times for those two configurations,
to get more reliable data.

1. Janky frames percentage
+--------+-----------------+---------------------+-------+-----------+
| metric |    variable     |       kernel        | value | perc_diff |
+--------+-----------------+---------------------+-------+-----------+
| gmean  | jank_percentage | EM_default          |  2.0  |   0.0%    |
| gmean  | jank_percentage | EM_modified_runtime |  1.3  |  -35.33%  |
+--------+-----------------+---------------------+-------+-----------+

2. Avg frame render time duration
+--------+---------------------+---------------------+-------+-----------+
| metric |      variable       |       kernel        | value | perc_diff |
+--------+---------------------+---------------------+-------+-----------+
| gmean  | mean_frame_duration | EM_default          | 10.5  |   0.0%    |
| gmean  | mean_frame_duration | EM_modified_runtime |  9.6  |  -8.52%   |
+--------+---------------------+---------------------+-------+-----------+

3. Max frame render time duration
+--------+--------------------+---------------------+-------+-----------+
| metric |      variable      |       kernel        | value | perc_diff |
+--------+--------------------+---------------------+-------+-----------+
| gmean  | max_frame_duration | EM_default          | 251.6 |   0.0%    |
| gmean  | max_frame_duration | EM_modified_runtime | 115.5 |  -54.09%  |
+--------+--------------------+---------------------+-------+-----------+

4. OS overutilized state percentage (when EAS is not working)
+--------------+---------------------+------+------------+------------+
|    metric    |       wa_path       | time | total_time | percentage |
+--------------+---------------------+------+------------+------------+
| overutilized | EM_default          | 1.65 |   253.38   |    0.65    |
| overutilized | EM_modified_runtime | 1.4  |   277.5    |    0.51    |
+--------------+---------------------+------+------------+------------+

5. All CPUs (Little+Mid+Big) power values in mW
+------------+--------+---------------------+-------+-----------+
|  channel   | metric |       kernel        | value | perc_diff |
+------------+--------+---------------------+-------+-----------+
|    CPU     | gmean  | EM_default          | 142.1 |   0.0%    |
|    CPU     | gmean  | EM_modified_runtime | 131.8 |  -7.27%   |
+------------+--------+---------------------+-------+-----------+

The time cost to update the EM decreased in this v5 vs v4:
big: 5us vs 2us -> 2.6x faster
mid: 9us vs 3us -> 3x faster
little: 16us vs 16us -> no change

We still have to update the inefficiency in the cpufreq framework, thus
a bit of overhead will be there.

These series is based on linux next tree, tag 'next-20240112', since there
are changes and revert which touch em_cpu_energy().

Changelog:
v7:
- dropped em_table_get/put() (Rafael)
- renamed memory function to em_table_alloc/free() (Rafael)
- use explicit rcu_read_lock/unlock() instead of wrappers and aligned
  frameworks & drivers using EM (Rafael)
- adjusted documentation to the new functions
- fixed doxygen comments (Rafael)
- renamed 'refcount' to 'kref' (Rafael)
- changed patch headers according to comments (Rafael)
- rebased on 'next-20240112' to get Ingo's revert affecting energy_model.h
v6 [6]:
- renamed 'runtime_table' to 'em_table' (Dietmar, Rafael)
- dropped kref increment during allocation (Qais)
- renamed em_inc/dec_usage() to em_table_inc/dec() (Qais)
- fixed comment description and left old comment block with small
  adjustment in em_cpu_energy() patch 15/23 (Dietmar)
- added platform name which was used for speed-up testing (Dietmar)
- changed patch header description keep it small not repeating the in-code
  comment describing 'cost' in em_cpu_energy() patch 15/23 (Dietmar)
- added check and warning in em_cpu_energy() about RCU lock held (Qais, Xuewen)
- changed nr_perf_states usage in the patch 7/23 (Dietmar)
- changed documentation according to comments (Dietmar)
- changed in-code comment in patch 11/23 according to comments (Dietmar)
- changed example driver function 'ctx' argument in the documentation (Xuewen)
- changed the example driver in documentation, dropped module_exit and
  added em_free_table() explicit in the update function
- fixed comments in various patch headers (Dietmar)
- fixed Doxygen comment s/@state/@table patch 4/23 (Dietmar)
- added information in the cover letter about:
-- optimization in EAS hot code path
-- follow-up patch set which adds OPP support and modifies EM for Exynos5
- rebased on 'next-20240104' to avoid collision with other code touching
  em_cpu_energy()
v5 changes are here [4]

Regards,
Lukasz Luba

[1] https://lpc.events/event/16/contributions/1341/attachments/955/1873/Dynamic_Energy_Model_to_handle_leakage_power.pdf
[2] https://lpc.events/event/16/contributions/1194/attachments/1114/2139/LPC2022_Energy_model_accuracy.pdf
[3] https://www.youtube.com/watch?v=2C-5uikSbtM&list=PL0fKordpLTjKsBOUcZqnzlHShri4YBL1H
[4] https://lore.kernel.org/lkml/20231129110853.94344-1-lukasz.luba@arm.com/
[5] https://lore.kernel.org/lkml/20231220110339.1065505-1-lukasz.luba@arm.com/
[6] https://lore.kernel.org/lkml/20240104171553.2080674-1-lukasz.luba@arm.com/


Lukasz Luba (23):
  PM: EM: Add missing newline for the message log
  PM: EM: Extend em_cpufreq_update_efficiencies() argument list
  PM: EM: Find first CPU active while updating OPP efficiency
  PM: EM: Refactor em_pd_get_efficient_state() to be more flexible
  PM: EM: Introduce em_compute_costs()
  PM: EM: Check if the get_cost() callback is present in
    em_compute_costs()
  PM: EM: Split the allocation and initialization of the EM table
  PM: EM: Introduce runtime modifiable table
  PM: EM: Use runtime modified EM for CPUs energy estimation in EAS
  PM: EM: Add functions for memory allocations for new EM tables
  PM: EM: Introduce em_dev_update_perf_domain() for EM updates
  PM: EM: Add em_perf_state_from_pd() to get performance states table
  PM: EM: Add performance field to struct em_perf_state and optimize
  PM: EM: Support late CPUs booting and capacity adjustment
  PM: EM: Optimize em_cpu_energy() and remove division
  powercap/dtpm_cpu: Use new Energy Model interface to get table
  powercap/dtpm_devfreq: Use new Energy Model interface to get table
  drivers/thermal/cpufreq_cooling: Use new Energy Model interface
  drivers/thermal/devfreq_cooling: Use new Energy Model interface
  PM: EM: Change debugfs configuration to use runtime EM table data
  PM: EM: Remove old table
  PM: EM: Add em_dev_compute_costs()
  Documentation: EM: Update with runtime modification design

 Documentation/power/energy-model.rst | 183 ++++++++++-
 drivers/powercap/dtpm_cpu.c          |  39 ++-
 drivers/powercap/dtpm_devfreq.c      |  34 +-
 drivers/thermal/cpufreq_cooling.c    |  45 ++-
 drivers/thermal/devfreq_cooling.c    |  49 ++-
 include/linux/energy_model.h         | 165 ++++++----
 kernel/power/energy_model.c          | 472 +++++++++++++++++++++++----
 7 files changed, 819 insertions(+), 168 deletions(-)

Comments

Hongyan Xia Jan. 17, 2024, 11:02 a.m. UTC | #1
On 17/01/2024 09:56, Lukasz Luba wrote:
> Fix missing newline for the string long in the error code path.
> 
> Signed-off-by: Lukasz Luba <lukasz.luba@arm.com>
> ---
>   kernel/power/energy_model.c | 2 +-
>   1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/kernel/power/energy_model.c b/kernel/power/energy_model.c
> index 7b44f5b89fa1..8b9dd4a39f63 100644
> --- a/kernel/power/energy_model.c
> +++ b/kernel/power/energy_model.c
> @@ -250,7 +250,7 @@ static void em_cpufreq_update_efficiencies(struct device *dev)
>   
>   	policy = cpufreq_cpu_get(cpumask_first(em_span_cpus(pd)));
>   	if (!policy) {
> -		dev_warn(dev, "EM: Access to CPUFreq policy failed");
> +		dev_warn(dev, "EM: Access to CPUFreq policy failed\n");
>   		return;
>   	}
>   

Reviewed-by: Hongyan Xia <hongyan.xia2@arm.com>
Hongyan Xia Jan. 17, 2024, 12:05 p.m. UTC | #2
On 17/01/2024 09:56, Lukasz Luba wrote:
> The Energy Model might be updated at runtime and the energy efficiency
> for each OPP may change. Thus, there is a need to update also the
> cpufreq framework and make it aligned to the new values. In order to
> do that, use a first active CPU from the Performance Domain. This is
> needed since the first CPU in the cpumask might be offline when we
> run this code path.
> 
> Signed-off-by: Lukasz Luba <lukasz.luba@arm.com>
> ---
>   kernel/power/energy_model.c | 11 +++++++++--
>   1 file changed, 9 insertions(+), 2 deletions(-)
> 
> diff --git a/kernel/power/energy_model.c b/kernel/power/energy_model.c
> index 42486674b834..aa7c89f9e115 100644
> --- a/kernel/power/energy_model.c
> +++ b/kernel/power/energy_model.c
> @@ -243,12 +243,19 @@ em_cpufreq_update_efficiencies(struct device *dev, struct em_perf_state *table)
>   	struct em_perf_domain *pd = dev->em_pd;
>   	struct cpufreq_policy *policy;
>   	int found = 0;
> -	int i;
> +	int i, cpu;
>   
>   	if (!_is_cpu_device(dev) || !pd)
>   		return;
>   
> -	policy = cpufreq_cpu_get(cpumask_first(em_span_cpus(pd)));
> +	/* Try to get a CPU which is active and in this PD */
> +	cpu = cpumask_first_and(em_span_cpus(pd), cpu_active_mask);
> +	if (cpu >= nr_cpu_ids) {
> +		dev_warn(dev, "EM: No online CPU for CPUFreq policy\n");
> +		return;
> +	}
> +
> +	policy = cpufreq_cpu_get(cpu);
>   	if (!policy) {
>   		dev_warn(dev, "EM: Access to CPUFreq policy failed\n");
>   		return;

Reviewed-by: Hongyan Xia <hongyan.xia2@arm.com>
Lukasz Luba Feb. 7, 2024, 9:15 a.m. UTC | #3
Hi Rafael,

On 1/17/24 09:56, Lukasz Luba wrote:
> Hi all,
> 
> This patch set adds a new feature which allows to modify Energy Model (EM)
> power values at runtime. It will allow to better reflect power model of
> a recent SoCs and silicon. Different characteristics of the power usage
> can be leveraged and thus better decisions made during task placement in EAS.
> 

[snip]

> 
> 
> Lukasz Luba (23):
>    PM: EM: Add missing newline for the message log
>    PM: EM: Extend em_cpufreq_update_efficiencies() argument list
>    PM: EM: Find first CPU active while updating OPP efficiency
>    PM: EM: Refactor em_pd_get_efficient_state() to be more flexible
>    PM: EM: Introduce em_compute_costs()
>    PM: EM: Check if the get_cost() callback is present in
>      em_compute_costs()
>    PM: EM: Split the allocation and initialization of the EM table
>    PM: EM: Introduce runtime modifiable table
>    PM: EM: Use runtime modified EM for CPUs energy estimation in EAS
>    PM: EM: Add functions for memory allocations for new EM tables
>    PM: EM: Introduce em_dev_update_perf_domain() for EM updates
>    PM: EM: Add em_perf_state_from_pd() to get performance states table
>    PM: EM: Add performance field to struct em_perf_state and optimize
>    PM: EM: Support late CPUs booting and capacity adjustment
>    PM: EM: Optimize em_cpu_energy() and remove division
>    powercap/dtpm_cpu: Use new Energy Model interface to get table
>    powercap/dtpm_devfreq: Use new Energy Model interface to get table
>    drivers/thermal/cpufreq_cooling: Use new Energy Model interface
>    drivers/thermal/devfreq_cooling: Use new Energy Model interface
>    PM: EM: Change debugfs configuration to use runtime EM table data
>    PM: EM: Remove old table
>    PM: EM: Add em_dev_compute_costs()
>    Documentation: EM: Update with runtime modification design
> 
>   Documentation/power/energy-model.rst | 183 ++++++++++-
>   drivers/powercap/dtpm_cpu.c          |  39 ++-
>   drivers/powercap/dtpm_devfreq.c      |  34 +-
>   drivers/thermal/cpufreq_cooling.c    |  45 ++-
>   drivers/thermal/devfreq_cooling.c    |  49 ++-
>   include/linux/energy_model.h         | 165 ++++++----
>   kernel/power/energy_model.c          | 472 +++++++++++++++++++++++----
>   7 files changed, 819 insertions(+), 168 deletions(-)
> 

The patch set went through decent review. If you don't have any issues,
I will collect the tags and send the v8 which will be re-based on some
recent linux next (or please tell me your preferred branch).

Regards,
Lukasz
Rafael J. Wysocki Feb. 7, 2024, 10:31 a.m. UTC | #4
Hi Lukasz,

On Wed, Feb 7, 2024 at 10:15 AM Lukasz Luba <lukasz.luba@arm.com> wrote:
>
> Hi Rafael,
>
> On 1/17/24 09:56, Lukasz Luba wrote:
> > Hi all,
> >
> > This patch set adds a new feature which allows to modify Energy Model (EM)
> > power values at runtime. It will allow to better reflect power model of
> > a recent SoCs and silicon. Different characteristics of the power usage
> > can be leveraged and thus better decisions made during task placement in EAS.
> >
>
> [snip]
>
> >
> >
> > Lukasz Luba (23):
> >    PM: EM: Add missing newline for the message log
> >    PM: EM: Extend em_cpufreq_update_efficiencies() argument list
> >    PM: EM: Find first CPU active while updating OPP efficiency
> >    PM: EM: Refactor em_pd_get_efficient_state() to be more flexible
> >    PM: EM: Introduce em_compute_costs()
> >    PM: EM: Check if the get_cost() callback is present in
> >      em_compute_costs()
> >    PM: EM: Split the allocation and initialization of the EM table
> >    PM: EM: Introduce runtime modifiable table
> >    PM: EM: Use runtime modified EM for CPUs energy estimation in EAS
> >    PM: EM: Add functions for memory allocations for new EM tables
> >    PM: EM: Introduce em_dev_update_perf_domain() for EM updates
> >    PM: EM: Add em_perf_state_from_pd() to get performance states table
> >    PM: EM: Add performance field to struct em_perf_state and optimize
> >    PM: EM: Support late CPUs booting and capacity adjustment
> >    PM: EM: Optimize em_cpu_energy() and remove division
> >    powercap/dtpm_cpu: Use new Energy Model interface to get table
> >    powercap/dtpm_devfreq: Use new Energy Model interface to get table
> >    drivers/thermal/cpufreq_cooling: Use new Energy Model interface
> >    drivers/thermal/devfreq_cooling: Use new Energy Model interface
> >    PM: EM: Change debugfs configuration to use runtime EM table data
> >    PM: EM: Remove old table
> >    PM: EM: Add em_dev_compute_costs()
> >    Documentation: EM: Update with runtime modification design
> >
> >   Documentation/power/energy-model.rst | 183 ++++++++++-
> >   drivers/powercap/dtpm_cpu.c          |  39 ++-
> >   drivers/powercap/dtpm_devfreq.c      |  34 +-
> >   drivers/thermal/cpufreq_cooling.c    |  45 ++-
> >   drivers/thermal/devfreq_cooling.c    |  49 ++-
> >   include/linux/energy_model.h         | 165 ++++++----
> >   kernel/power/energy_model.c          | 472 +++++++++++++++++++++++----
> >   7 files changed, 819 insertions(+), 168 deletions(-)
> >
>
> The patch set went through decent review. If you don't have any issues,
> I will collect the tags and send the v8 which will be re-based on some
> recent linux next (or please tell me your preferred branch).

Blease base it on 6.8-rc3.
Lukasz Luba Feb. 7, 2024, 11:49 a.m. UTC | #5
On 2/7/24 10:31, Rafael J. Wysocki wrote:
> Hi Lukasz,
> 
> On Wed, Feb 7, 2024 at 10:15 AM Lukasz Luba <lukasz.luba@arm.com> wrote:
>>
>> Hi Rafael,
>>
>> On 1/17/24 09:56, Lukasz Luba wrote:
>>> Hi all,
>>>
>>> This patch set adds a new feature which allows to modify Energy Model (EM)
>>> power values at runtime. It will allow to better reflect power model of
>>> a recent SoCs and silicon. Different characteristics of the power usage
>>> can be leveraged and thus better decisions made during task placement in EAS.
>>>
>>
>> [snip]
>>
>>>
>>>
>>> Lukasz Luba (23):
>>>     PM: EM: Add missing newline for the message log
>>>     PM: EM: Extend em_cpufreq_update_efficiencies() argument list
>>>     PM: EM: Find first CPU active while updating OPP efficiency
>>>     PM: EM: Refactor em_pd_get_efficient_state() to be more flexible
>>>     PM: EM: Introduce em_compute_costs()
>>>     PM: EM: Check if the get_cost() callback is present in
>>>       em_compute_costs()
>>>     PM: EM: Split the allocation and initialization of the EM table
>>>     PM: EM: Introduce runtime modifiable table
>>>     PM: EM: Use runtime modified EM for CPUs energy estimation in EAS
>>>     PM: EM: Add functions for memory allocations for new EM tables
>>>     PM: EM: Introduce em_dev_update_perf_domain() for EM updates
>>>     PM: EM: Add em_perf_state_from_pd() to get performance states table
>>>     PM: EM: Add performance field to struct em_perf_state and optimize
>>>     PM: EM: Support late CPUs booting and capacity adjustment
>>>     PM: EM: Optimize em_cpu_energy() and remove division
>>>     powercap/dtpm_cpu: Use new Energy Model interface to get table
>>>     powercap/dtpm_devfreq: Use new Energy Model interface to get table
>>>     drivers/thermal/cpufreq_cooling: Use new Energy Model interface
>>>     drivers/thermal/devfreq_cooling: Use new Energy Model interface
>>>     PM: EM: Change debugfs configuration to use runtime EM table data
>>>     PM: EM: Remove old table
>>>     PM: EM: Add em_dev_compute_costs()
>>>     Documentation: EM: Update with runtime modification design
>>>
>>>    Documentation/power/energy-model.rst | 183 ++++++++++-
>>>    drivers/powercap/dtpm_cpu.c          |  39 ++-
>>>    drivers/powercap/dtpm_devfreq.c      |  34 +-
>>>    drivers/thermal/cpufreq_cooling.c    |  45 ++-
>>>    drivers/thermal/devfreq_cooling.c    |  49 ++-
>>>    include/linux/energy_model.h         | 165 ++++++----
>>>    kernel/power/energy_model.c          | 472 +++++++++++++++++++++++----
>>>    7 files changed, 819 insertions(+), 168 deletions(-)
>>>
>>
>> The patch set went through decent review. If you don't have any issues,
>> I will collect the tags and send the v8 which will be re-based on some
>> recent linux next (or please tell me your preferred branch).
> 
> Blease base it on 6.8-rc3.

OK, thanks