[2/6] sched/fair: Convert arch_scale_cpu_capacity() from weak function to #define

Hi Vincent,

On 02/09/15 10:31, Vincent Guittot wrote:
> Hi Morten,
> 
> On 14 August 2015 at 18:23, Morten Rasmussen <morten.rasmussen@arm.com> wrote:
>> Bring arch_scale_cpu_capacity() in line with the recent change of its
>> arch_scale_freq_capacity() sibling in commit dfbca41f3479 ("sched:
>> Optimize freq invariant accounting") from weak function to #define to
>> allow inlining of the function.
>>
>> While at it, remove the ARCH_CAPACITY sched_feature as well. With the
>> change to #define there isn't a straightforward way to allow runtime
>> switch between an arch implementation and the default implementation of
>> arch_scale_cpu_capacity() using sched_feature. The default was to use
>> the arch-specific implementation, but only the arm architecture provides
>> one and that is essentially equivalent to the default implementation.

[...]

> 
> So you change the way to declare arch_scale_cpu_capacity but i don't
> see the update of the arm arch which declare a
> arch_scale_cpu_capacity to reflect this change in your series.

We were reluctant to do this because this functionality makes only sense
for ARCH=arm big.Little systems w/ cortex-a{15|7} cores and only if the
clock-frequency property is set in the dts file. 

Are you planning to push for a 'struct cpu_efficiency/clock-frequency
property' solution for ARCH=arm64 as well?

I'm asking because for ARCH=arm64 systems today (JUNO, Hi6220) we use the
capacity value of the last entry of the capacity_state vector for the cores
(e.g. cortex-a{57|53). 
    
To connect the cpu invariant engine (scale_cpu_capacity()
[arch/arm/kernel/topology.c]) with the scheduler, something like this is
missing:

[...]

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

On 3 September 2015 at 21:58, Dietmar Eggemann <dietmar.eggemann@arm.com> wrote:
> Hi Vincent,
>
> On 02/09/15 10:31, Vincent Guittot wrote:
>> Hi Morten,
>>
>> On 14 August 2015 at 18:23, Morten Rasmussen <morten.rasmussen@arm.com> wrote:
>>> Bring arch_scale_cpu_capacity() in line with the recent change of its
>>> arch_scale_freq_capacity() sibling in commit dfbca41f3479 ("sched:
>>> Optimize freq invariant accounting") from weak function to #define to
>>> allow inlining of the function.
>>>
>>> While at it, remove the ARCH_CAPACITY sched_feature as well. With the
>>> change to #define there isn't a straightforward way to allow runtime
>>> switch between an arch implementation and the default implementation of
>>> arch_scale_cpu_capacity() using sched_feature. The default was to use
>>> the arch-specific implementation, but only the arm architecture provides
>>> one and that is essentially equivalent to the default implementation.
>
> [...]
>
>>
>> So you change the way to declare arch_scale_cpu_capacity but i don't
>> see the update of the arm arch which declare a
>> arch_scale_cpu_capacity to reflect this change in your series.
>
> We were reluctant to do this because this functionality makes only sense
> for ARCH=arm big.Little systems w/ cortex-a{15|7} cores and only if the
> clock-frequency property is set in the dts file.

IMO, we should maintain the compatibility of current implementation
instead of breaking the link and creating a dead code.
Your proposal below fits the requirement

>
> Are you planning to push for a 'struct cpu_efficiency/clock-frequency
> property' solution for ARCH=arm64 as well?

I know that there has been some discussions aorund that but i didn't
follow the thread in details

>
> I'm asking because for ARCH=arm64 systems today (JUNO, Hi6220) we use the
> capacity value of the last entry of the capacity_state vector for the cores
> (e.g. cortex-a{57|53).

This is a struct of the eas feature ? Not sure that we should link the
definition of  the cpu capacity to an internal struct of a feature; DT
seems a better way to define it.
So if you want to revisit the way, we set the capacity of CPU for arm
and/or arm64, I'm fully open to the discussion but this should happen
in another thread than this one which has for only purpose the
alignment of the arch_scale_cpu_capacity interface declaration with
arch_scale_freq_capacity one.

So, with the patch below that updates the arm definition of
arch_scale_cpu_capacity, you can add my Acked-by: Vincent Guittot
<vincent.guittot@linaro.org> on this patch and the additional one
below

Regards,
Vincent

>
> To connect the cpu invariant engine (scale_cpu_capacity()
> [arch/arm/kernel/topology.c]) with the scheduler, something like this is
> missing:
>
> diff --git a/arch/arm/include/asm/topology.h b/arch/arm/include/asm/topology.h
> index 370f7a732900..17c6b3243196 100644
> --- a/arch/arm/include/asm/topology.h
> +++ b/arch/arm/include/asm/topology.h
> @@ -24,6 +24,10 @@ void init_cpu_topology(void);
>  void store_cpu_topology(unsigned int cpuid);
>  const struct cpumask *cpu_coregroup_mask(int cpu);
>
> +#define arch_scale_cpu_capacity scale_cpu_capacity
> +struct sched_domain;
> +extern unsigned long scale_cpu_capacity(struct sched_domain *sd, int cpu);
> +
>  #else
>
>  static inline void init_cpu_topology(void) { }
> diff --git a/arch/arm/kernel/topology.c b/arch/arm/kernel/topology.c
> index 08b7847bf912..907e0d2d9b82 100644
> --- a/arch/arm/kernel/topology.c
> +++ b/arch/arm/kernel/topology.c
> @@ -42,7 +42,7 @@
>   */
>  static DEFINE_PER_CPU(unsigned long, cpu_scale);
>
> -unsigned long arch_scale_cpu_capacity(struct sched_domain *sd, int cpu)
> +unsigned long scale_cpu_capacity(struct sched_domain *sd, int cpu)
>  {
>         return per_cpu(cpu_scale, cpu);
>  }
> @@ -166,7 +166,7 @@ static void update_cpu_capacity(unsigned int cpu)
>         set_capacity_scale(cpu, cpu_capacity(cpu) / middle_capacity);
>
>         pr_info("CPU%u: update cpu_capacity %lu\n",
> -               cpu, arch_scale_cpu_capacity(NULL, cpu));
> +               cpu, scale_cpu_capacity(NULL, cpu));
>  }
>
> -- Dietmar
>
> [...]
>
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

On 04/09/15 08:26, Vincent Guittot wrote:
> On 3 September 2015 at 21:58, Dietmar Eggemann <dietmar.eggemann@arm.com> wrote:

[...]

>>> So you change the way to declare arch_scale_cpu_capacity but i don't
>>> see the update of the arm arch which declare a
>>> arch_scale_cpu_capacity to reflect this change in your series.
>>
>> We were reluctant to do this because this functionality makes only sense
>> for ARCH=arm big.Little systems w/ cortex-a{15|7} cores and only if the
>> clock-frequency property is set in the dts file.
> 
> IMO, we should maintain the compatibility of current implementation
> instead of breaking the link and creating a dead code.
> Your proposal below fits the requirement

The only problem with this solution is that now we got a call to
arch_scale_cpu_capacity() in the hotpath whereas before it is only
called in update_cpu_capacity(). An implementation of
scale_cpu_capacity() in arch/arm/kernel/topology.c leads to a function
call in __update_load_avg. I'm in the middle of doing some performance
tests on TC2 w/ and w/o the cpu invariant implementation.

> 
>>
>> Are you planning to push for a 'struct cpu_efficiency/clock-frequency
>> property' solution for ARCH=arm64 as well?
> 
> I know that there has been some discussions aorund that but i didn't
> follow the thread in details
> 
>>
>> I'm asking because for ARCH=arm64 systems today (JUNO, Hi6220) we use the
>> capacity value of the last entry of the capacity_state vector for the cores
>> (e.g. cortex-a{57|53).
> 
> This is a struct of the eas feature ? Not sure that we should link the
> definition of  the cpu capacity to an internal struct of a feature; DT
> seems a better way to define it.

Yeah, the cpu invariant functionality should not base on EAS. We just
use the short-cut in EAS RFCv5 to get it working on ARM64.

> So if you want to revisit the way, we set the capacity of CPU for arm
> and/or arm64, I'm fully open to the discussion but this should happen
> in another thread than this one which has for only purpose the
> alignment of the arch_scale_cpu_capacity interface declaration with
> arch_scale_freq_capacity one.

Agreed.

> 
> So, with the patch below that updates the arm definition of
> arch_scale_cpu_capacity, you can add my Acked-by: Vincent Guittot
> <vincent.guittot@linaro.org> on this patch and the additional one
> below
> 
> Regards,
> Vincent

[...]

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

On 04/09/15 08:26, Vincent Guittot wrote:
> On 3 September 2015 at 21:58, Dietmar Eggemann <dietmar.eggemann@arm.com> wrote:

[...]

> So, with the patch below that updates the arm definition of
> arch_scale_cpu_capacity, you can add my Acked-by: Vincent Guittot
> <vincent.guittot@linaro.org> on this patch and the additional one
> below

My tests on ARM TC2 (only the 2 A15's) show that the influence of the
non-default arch_scale_cpu_capacity function in
arch/arm/kernel/topology.c and with it the extra function call to it
from __update_load_avg() has no measurable influence on performance:

perf stat --null --repeat 10 -- perf bench sched messaging --g 50 -l 200

- default arch_scale_cpu_capacity function [kernel/sched/sched.h]

16.006976251 seconds time elapsed ( +- 0.29% )
16.063814914 seconds time elapsed ( +- 0.37% )
16.088199252 seconds time elapsed ( +- 0.39% )

- arch_scale_cpu_capacity function [arch/arm/kernel/topology.c]

15.945975308 seconds time elapsed ( +- 0.10% )
16.131203074 seconds time elapsed ( +- 0.21% )
16.108302562 seconds time elapsed ( +- 0.41% )

If I force the function to be inline, the result is slightly worse:

- arch_scale_cpu_capacity function [arch/arm/include/asm/topology.h]

16.122545216 seconds time elapsed ( +- 0.09% )
16.285819258 seconds time elapsed ( +- 0.35% )
16.157454024 seconds time elapsed ( +- 0.15% )

So I think we can connect the arch_scale_cpu_capacity function
[arch/arm/kernel/topology.c] to the CFS scheduler for ARCH=arm so that
people get cpu scale different to 1024 on arm big.little machines w/
A15/A7 in case the specify clock-frequency properties in their dtb file.

Can we still have your 'Acked-by' for this patch and 3/6 even though we
now scale weight (by frequency) and scale_freq (by cpu) instead of the
time related values (delta_w, contrib, delta)?

Thanks,

-- Dietmar

> 
> Regards,
> Vincent
> 
>>
>> To connect the cpu invariant engine (scale_cpu_capacity()
>> [arch/arm/kernel/topology.c]) with the scheduler, something like this is
>> missing:
>>
>> diff --git a/arch/arm/include/asm/topology.h b/arch/arm/include/asm/topology.h
>> index 370f7a732900..17c6b3243196 100644
>> --- a/arch/arm/include/asm/topology.h
>> +++ b/arch/arm/include/asm/topology.h
>> @@ -24,6 +24,10 @@ void init_cpu_topology(void);
>>  void store_cpu_topology(unsigned int cpuid);
>>  const struct cpumask *cpu_coregroup_mask(int cpu);
>>
>> +#define arch_scale_cpu_capacity scale_cpu_capacity
>> +struct sched_domain;
>> +extern unsigned long scale_cpu_capacity(struct sched_domain *sd, int cpu);
>> +
>>  #else
>>
>>  static inline void init_cpu_topology(void) { }
>> diff --git a/arch/arm/kernel/topology.c b/arch/arm/kernel/topology.c
>> index 08b7847bf912..907e0d2d9b82 100644
>> --- a/arch/arm/kernel/topology.c
>> +++ b/arch/arm/kernel/topology.c
>> @@ -42,7 +42,7 @@
>>   */
>>  static DEFINE_PER_CPU(unsigned long, cpu_scale);
>>
>> -unsigned long arch_scale_cpu_capacity(struct sched_domain *sd, int cpu)
>> +unsigned long scale_cpu_capacity(struct sched_domain *sd, int cpu)
>>  {
>>         return per_cpu(cpu_scale, cpu);
>>  }
>> @@ -166,7 +166,7 @@ static void update_cpu_capacity(unsigned int cpu)
>>         set_capacity_scale(cpu, cpu_capacity(cpu) / middle_capacity);
>>
>>         pr_info("CPU%u: update cpu_capacity %lu\n",
>> -               cpu, arch_scale_cpu_capacity(NULL, cpu));
>> +               cpu, scale_cpu_capacity(NULL, cpu));
>>  }
>>
>> -- Dietmar
>>
>> [...]

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

On 11 September 2015 at 15:21, Dietmar Eggemann
<dietmar.eggemann@arm.com> wrote:
> On 04/09/15 08:26, Vincent Guittot wrote:
>> On 3 September 2015 at 21:58, Dietmar Eggemann <dietmar.eggemann@arm.com> wrote:
>
> [...]
>
>> So, with the patch below that updates the arm definition of
>> arch_scale_cpu_capacity, you can add my Acked-by: Vincent Guittot
>> <vincent.guittot@linaro.org> on this patch and the additional one
>> below
>
> My tests on ARM TC2 (only the 2 A15's) show that the influence of the
> non-default arch_scale_cpu_capacity function in
> arch/arm/kernel/topology.c and with it the extra function call to it
> from __update_load_avg() has no measurable influence on performance:
>
> perf stat --null --repeat 10 -- perf bench sched messaging --g 50 -l 200
>
> - default arch_scale_cpu_capacity function [kernel/sched/sched.h]
>
> 16.006976251 seconds time elapsed ( +- 0.29% )
> 16.063814914 seconds time elapsed ( +- 0.37% )
> 16.088199252 seconds time elapsed ( +- 0.39% )
>
> - arch_scale_cpu_capacity function [arch/arm/kernel/topology.c]
>
> 15.945975308 seconds time elapsed ( +- 0.10% )
> 16.131203074 seconds time elapsed ( +- 0.21% )
> 16.108302562 seconds time elapsed ( +- 0.41% )
>
> If I force the function to be inline, the result is slightly worse:
>
> - arch_scale_cpu_capacity function [arch/arm/include/asm/topology.h]
>
> 16.122545216 seconds time elapsed ( +- 0.09% )
> 16.285819258 seconds time elapsed ( +- 0.35% )
> 16.157454024 seconds time elapsed ( +- 0.15% )
>
> So I think we can connect the arch_scale_cpu_capacity function
> [arch/arm/kernel/topology.c] to the CFS scheduler for ARCH=arm so that
> people get cpu scale different to 1024 on arm big.little machines w/
> A15/A7 in case the specify clock-frequency properties in their dtb file.
>
> Can we still have your 'Acked-by' for this patch and 3/6 even though we
> now scale weight (by frequency) and scale_freq (by cpu) instead of the
> time related values (delta_w, contrib, delta)?

Yes, Please add my Acked-by

Vincent

>
> Thanks,
>
> -- Dietmar
>
>>
>> Regards,
>> Vincent
>>
>>>
>>> To connect the cpu invariant engine (scale_cpu_capacity()
>>> [arch/arm/kernel/topology.c]) with the scheduler, something like this is
>>> missing:
>>>
>>> diff --git a/arch/arm/include/asm/topology.h b/arch/arm/include/asm/topology.h
>>> index 370f7a732900..17c6b3243196 100644
>>> --- a/arch/arm/include/asm/topology.h
>>> +++ b/arch/arm/include/asm/topology.h
>>> @@ -24,6 +24,10 @@ void init_cpu_topology(void);
>>>  void store_cpu_topology(unsigned int cpuid);
>>>  const struct cpumask *cpu_coregroup_mask(int cpu);
>>>
>>> +#define arch_scale_cpu_capacity scale_cpu_capacity
>>> +struct sched_domain;
>>> +extern unsigned long scale_cpu_capacity(struct sched_domain *sd, int cpu);
>>> +
>>>  #else
>>>
>>>  static inline void init_cpu_topology(void) { }
>>> diff --git a/arch/arm/kernel/topology.c b/arch/arm/kernel/topology.c
>>> index 08b7847bf912..907e0d2d9b82 100644
>>> --- a/arch/arm/kernel/topology.c
>>> +++ b/arch/arm/kernel/topology.c
>>> @@ -42,7 +42,7 @@
>>>   */
>>>  static DEFINE_PER_CPU(unsigned long, cpu_scale);
>>>
>>> -unsigned long arch_scale_cpu_capacity(struct sched_domain *sd, int cpu)
>>> +unsigned long scale_cpu_capacity(struct sched_domain *sd, int cpu)
>>>  {
>>>         return per_cpu(cpu_scale, cpu);
>>>  }
>>> @@ -166,7 +166,7 @@ static void update_cpu_capacity(unsigned int cpu)
>>>         set_capacity_scale(cpu, cpu_capacity(cpu) / middle_capacity);
>>>
>>>         pr_info("CPU%u: update cpu_capacity %lu\n",
>>> -               cpu, arch_scale_cpu_capacity(NULL, cpu));
>>> +               cpu, scale_cpu_capacity(NULL, cpu));
>>>  }
>>>
>>> -- Dietmar
>>>
>>> [...]
>
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/