[v2,3/4] PM: domains: Drop/restore performance state votes for devices at runtime PM

Message ID 20210603093438.138705-4-ulf.hansson@linaro.org
State New
Headers show
Series
  • PM: domains: Avoid boilerplate code for DVFS in subsystem/drivers
Related show

Commit Message

Ulf Hansson June 3, 2021, 9:34 a.m.
A subsystem/driver that need to manage OPPs for its device, should
typically drop its vote for the OPP when the device becomes runtime
suspended. In this way, the corresponding aggregation of the performance
state votes that is managed in genpd for the attached PM domain, may find
that the aggregated vote can be decreased. Hence, it may allow genpd to set
the lower performance state for the PM domain, thus avoiding to waste
energy.

To accomplish this, typically a subsystem/driver would need to call
dev_pm_opp_set_rate|opp() for its device from its ->runtime_suspend()
callback, to drop the vote for the OPP. Accordingly, it needs another call
to dev_pm_opp_set_rate|opp() to restore the vote for the OPP from its
->runtime_resume() callback.

To avoid boilerplate code in subsystems/driver to deal with these things,
let's instead manage this internally in genpd.

Signed-off-by: Ulf Hansson <ulf.hansson@linaro.org>

---

Changes in v2:
	- Rebased.
	- A few minor cosmetic changes.
	- Deal with the error path in genpd_runtime_resume().

---
 drivers/base/power/domain.c | 27 +++++++++++++++++++++++++--
 include/linux/pm_domain.h   |  1 +
 2 files changed, 26 insertions(+), 2 deletions(-)

-- 
2.25.1

Comments

Viresh Kumar June 3, 2021, 9:55 a.m. | #1
On 03-06-21, 11:34, Ulf Hansson wrote:
> A subsystem/driver that need to manage OPPs for its device, should

> typically drop its vote for the OPP when the device becomes runtime

> suspended. In this way, the corresponding aggregation of the performance

> state votes that is managed in genpd for the attached PM domain, may find

> that the aggregated vote can be decreased. Hence, it may allow genpd to set

> the lower performance state for the PM domain, thus avoiding to waste

> energy.

> 

> To accomplish this, typically a subsystem/driver would need to call

> dev_pm_opp_set_rate|opp() for its device from its ->runtime_suspend()

> callback, to drop the vote for the OPP. Accordingly, it needs another call

> to dev_pm_opp_set_rate|opp() to restore the vote for the OPP from its

> ->runtime_resume() callback.

> 

> To avoid boilerplate code in subsystems/driver to deal with these things,

> let's instead manage this internally in genpd.

> 

> Signed-off-by: Ulf Hansson <ulf.hansson@linaro.org>

> ---

> 

> Changes in v2:

> 	- Rebased.

> 	- A few minor cosmetic changes.

> 	- Deal with the error path in genpd_runtime_resume().

> 

> ---

>  drivers/base/power/domain.c | 27 +++++++++++++++++++++++++--

>  include/linux/pm_domain.h   |  1 +

>  2 files changed, 26 insertions(+), 2 deletions(-)

> 

> diff --git a/drivers/base/power/domain.c b/drivers/base/power/domain.c

> index ef25a5b18587..e5d97174c254 100644

> --- a/drivers/base/power/domain.c

> +++ b/drivers/base/power/domain.c

> @@ -400,6 +400,23 @@ static int genpd_set_performance_state(struct device *dev, unsigned int state)

>  	return ret;

>  }

>  

> +static int genpd_drop_performance_state(struct device *dev)


What about passing the state pointer here? that will simplify the
callers to just a call.

> +{

> +	unsigned int prev_state = dev_gpd_data(dev)->performance_state;

> +

> +	if (!genpd_set_performance_state(dev, 0))

> +		return prev_state;

> +

> +	return 0;

> +}

> +

> +static void genpd_restore_performance_state(struct device *dev,

> +					    unsigned int state)

> +{

> +	if (state)


I will skip this check, as we are checking it in
genpd_set_performance_state() anyway ?

> +		genpd_set_performance_state(dev, state);

> +}

> +

>  /**

>   * dev_pm_genpd_set_performance_state- Set performance state of device's power

>   * domain.

> @@ -842,7 +859,8 @@ static int genpd_runtime_suspend(struct device *dev)

>  {

>  	struct generic_pm_domain *genpd;

>  	bool (*suspend_ok)(struct device *__dev);

> -	struct gpd_timing_data *td = &dev_gpd_data(dev)->td;

> +	struct generic_pm_domain_data *gpd_data = dev_gpd_data(dev);

> +	struct gpd_timing_data *td = &gpd_data->td;

>  	bool runtime_pm = pm_runtime_enabled(dev);

>  	ktime_t time_start;

>  	s64 elapsed_ns;

> @@ -899,6 +917,7 @@ static int genpd_runtime_suspend(struct device *dev)

>  		return 0;

>  

>  	genpd_lock(genpd);

> +	gpd_data->rpm_pstate = genpd_drop_performance_state(dev);


So this will become:

	genpd_drop_performance_state(dev, &gpd_data->rpm_pstate);

and it can have return type of void.

-- 
viresh
Ulf Hansson June 3, 2021, 10:31 a.m. | #2
On Thu, 3 Jun 2021 at 11:55, Viresh Kumar <viresh.kumar@linaro.org> wrote:
>

> On 03-06-21, 11:34, Ulf Hansson wrote:

> > A subsystem/driver that need to manage OPPs for its device, should

> > typically drop its vote for the OPP when the device becomes runtime

> > suspended. In this way, the corresponding aggregation of the performance

> > state votes that is managed in genpd for the attached PM domain, may find

> > that the aggregated vote can be decreased. Hence, it may allow genpd to set

> > the lower performance state for the PM domain, thus avoiding to waste

> > energy.

> >

> > To accomplish this, typically a subsystem/driver would need to call

> > dev_pm_opp_set_rate|opp() for its device from its ->runtime_suspend()

> > callback, to drop the vote for the OPP. Accordingly, it needs another call

> > to dev_pm_opp_set_rate|opp() to restore the vote for the OPP from its

> > ->runtime_resume() callback.

> >

> > To avoid boilerplate code in subsystems/driver to deal with these things,

> > let's instead manage this internally in genpd.

> >

> > Signed-off-by: Ulf Hansson <ulf.hansson@linaro.org>

> > ---

> >

> > Changes in v2:

> >       - Rebased.

> >       - A few minor cosmetic changes.

> >       - Deal with the error path in genpd_runtime_resume().

> >

> > ---

> >  drivers/base/power/domain.c | 27 +++++++++++++++++++++++++--

> >  include/linux/pm_domain.h   |  1 +

> >  2 files changed, 26 insertions(+), 2 deletions(-)

> >

> > diff --git a/drivers/base/power/domain.c b/drivers/base/power/domain.c

> > index ef25a5b18587..e5d97174c254 100644

> > --- a/drivers/base/power/domain.c

> > +++ b/drivers/base/power/domain.c

> > @@ -400,6 +400,23 @@ static int genpd_set_performance_state(struct device *dev, unsigned int state)

> >       return ret;

> >  }

> >

> > +static int genpd_drop_performance_state(struct device *dev)

>

> What about passing the state pointer here? that will simplify the

> callers to just a call.


Not sure I get that. Can you elaborate a bit more?

>

> > +{

> > +     unsigned int prev_state = dev_gpd_data(dev)->performance_state;

> > +

> > +     if (!genpd_set_performance_state(dev, 0))

> > +             return prev_state;

> > +

> > +     return 0;

> > +}

> > +

> > +static void genpd_restore_performance_state(struct device *dev,

> > +                                         unsigned int state)

> > +{

> > +     if (state)

>

> I will skip this check, as we are checking it in

> genpd_set_performance_state() anyway ?


I don't want us to override OPP votes made by the subsystem/driver
level runtime PM callbacks. For example, if the drivers manage this
thing themselves, that should be preserved.

That said, by the check above I want to avoid setting the state to
zero internally by genpd, if the driver level ->runtime_resume()
callback has already restored the state.

>

> > +             genpd_set_performance_state(dev, state);

> > +}

> > +

> >  /**

> >   * dev_pm_genpd_set_performance_state- Set performance state of device's power

> >   * domain.

> > @@ -842,7 +859,8 @@ static int genpd_runtime_suspend(struct device *dev)

> >  {

> >       struct generic_pm_domain *genpd;

> >       bool (*suspend_ok)(struct device *__dev);

> > -     struct gpd_timing_data *td = &dev_gpd_data(dev)->td;

> > +     struct generic_pm_domain_data *gpd_data = dev_gpd_data(dev);

> > +     struct gpd_timing_data *td = &gpd_data->td;

> >       bool runtime_pm = pm_runtime_enabled(dev);

> >       ktime_t time_start;

> >       s64 elapsed_ns;

> > @@ -899,6 +917,7 @@ static int genpd_runtime_suspend(struct device *dev)

> >               return 0;

> >

> >       genpd_lock(genpd);

> > +     gpd_data->rpm_pstate = genpd_drop_performance_state(dev);

>

> So this will become:

>

>         genpd_drop_performance_state(dev, &gpd_data->rpm_pstate);

>

> and it can have return type of void.


See more above, about the reason why it looks like this. Hopefully
that explains it.

Kind regards
Uffe
Ulf Hansson June 3, 2021, 11:17 a.m. | #3
On Thu, 3 Jun 2021 at 12:31, Ulf Hansson <ulf.hansson@linaro.org> wrote:
>

> On Thu, 3 Jun 2021 at 11:55, Viresh Kumar <viresh.kumar@linaro.org> wrote:

> >

> > On 03-06-21, 11:34, Ulf Hansson wrote:

> > > A subsystem/driver that need to manage OPPs for its device, should

> > > typically drop its vote for the OPP when the device becomes runtime

> > > suspended. In this way, the corresponding aggregation of the performance

> > > state votes that is managed in genpd for the attached PM domain, may find

> > > that the aggregated vote can be decreased. Hence, it may allow genpd to set

> > > the lower performance state for the PM domain, thus avoiding to waste

> > > energy.

> > >

> > > To accomplish this, typically a subsystem/driver would need to call

> > > dev_pm_opp_set_rate|opp() for its device from its ->runtime_suspend()

> > > callback, to drop the vote for the OPP. Accordingly, it needs another call

> > > to dev_pm_opp_set_rate|opp() to restore the vote for the OPP from its

> > > ->runtime_resume() callback.

> > >

> > > To avoid boilerplate code in subsystems/driver to deal with these things,

> > > let's instead manage this internally in genpd.

> > >

> > > Signed-off-by: Ulf Hansson <ulf.hansson@linaro.org>

> > > ---

> > >

> > > Changes in v2:

> > >       - Rebased.

> > >       - A few minor cosmetic changes.

> > >       - Deal with the error path in genpd_runtime_resume().

> > >

> > > ---

> > >  drivers/base/power/domain.c | 27 +++++++++++++++++++++++++--

> > >  include/linux/pm_domain.h   |  1 +

> > >  2 files changed, 26 insertions(+), 2 deletions(-)

> > >

> > > diff --git a/drivers/base/power/domain.c b/drivers/base/power/domain.c

> > > index ef25a5b18587..e5d97174c254 100644

> > > --- a/drivers/base/power/domain.c

> > > +++ b/drivers/base/power/domain.c

> > > @@ -400,6 +400,23 @@ static int genpd_set_performance_state(struct device *dev, unsigned int state)

> > >       return ret;

> > >  }

> > >

> > > +static int genpd_drop_performance_state(struct device *dev)

> >

> > What about passing the state pointer here? that will simplify the

> > callers to just a call.

>

> Not sure I get that. Can you elaborate a bit more?

>

> >

> > > +{

> > > +     unsigned int prev_state = dev_gpd_data(dev)->performance_state;

> > > +

> > > +     if (!genpd_set_performance_state(dev, 0))

> > > +             return prev_state;

> > > +

> > > +     return 0;

> > > +}

> > > +

> > > +static void genpd_restore_performance_state(struct device *dev,

> > > +                                         unsigned int state)

> > > +{

> > > +     if (state)

> >

> > I will skip this check, as we are checking it in

> > genpd_set_performance_state() anyway ?

>

> I don't want us to override OPP votes made by the subsystem/driver

> level runtime PM callbacks. For example, if the drivers manage this

> thing themselves, that should be preserved.

>

> That said, by the check above I want to avoid setting the state to

> zero internally by genpd, if the driver level ->runtime_resume()

> callback has already restored the state.


Ehh, forget about what I said about the ->runtime_resume() callback.

I am mostly trying to avoid restoring a state that is zero, just to be
sure nobody else on some different level outside gendp, have decided
to set a new OPP in-between our calls to
genpd_drop|restore_performance state.

[...]

Kind regards
Uffe
Dmitry Osipenko June 3, 2021, 7:02 p.m. | #4
03.06.2021 12:34, Ulf Hansson пишет:
> A subsystem/driver that need to manage OPPs for its device, should

> typically drop its vote for the OPP when the device becomes runtime

> suspended. In this way, the corresponding aggregation of the performance

> state votes that is managed in genpd for the attached PM domain, may find

> that the aggregated vote can be decreased. Hence, it may allow genpd to set

> the lower performance state for the PM domain, thus avoiding to waste

> energy.

> 

> To accomplish this, typically a subsystem/driver would need to call

> dev_pm_opp_set_rate|opp() for its device from its ->runtime_suspend()

> callback, to drop the vote for the OPP. Accordingly, it needs another call

> to dev_pm_opp_set_rate|opp() to restore the vote for the OPP from its

> ->runtime_resume() callback.

> 

> To avoid boilerplate code in subsystems/driver to deal with these things,

> let's instead manage this internally in genpd.

> 

> Signed-off-by: Ulf Hansson <ulf.hansson@linaro.org>

> ---

> 

> Changes in v2:

> 	- Rebased.

> 	- A few minor cosmetic changes.

> 	- Deal with the error path in genpd_runtime_resume().


I tested this on NVIDIA Tegra by removing the boilerplate code from
drivers' RPM and haven't noticed any problems, the performance state is
dropped/restored as expected. Thank you.
Dmitry Osipenko June 3, 2021, 7:08 p.m. | #5
03.06.2021 22:02, Dmitry Osipenko пишет:
> 03.06.2021 12:34, Ulf Hansson пишет:

>> A subsystem/driver that need to manage OPPs for its device, should

>> typically drop its vote for the OPP when the device becomes runtime

>> suspended. In this way, the corresponding aggregation of the performance

>> state votes that is managed in genpd for the attached PM domain, may find

>> that the aggregated vote can be decreased. Hence, it may allow genpd to set

>> the lower performance state for the PM domain, thus avoiding to waste

>> energy.

>>

>> To accomplish this, typically a subsystem/driver would need to call

>> dev_pm_opp_set_rate|opp() for its device from its ->runtime_suspend()

>> callback, to drop the vote for the OPP. Accordingly, it needs another call

>> to dev_pm_opp_set_rate|opp() to restore the vote for the OPP from its

>> ->runtime_resume() callback.

>>

>> To avoid boilerplate code in subsystems/driver to deal with these things,

>> let's instead manage this internally in genpd.

>>

>> Signed-off-by: Ulf Hansson <ulf.hansson@linaro.org>

>> ---

>>

>> Changes in v2:

>> 	- Rebased.

>> 	- A few minor cosmetic changes.

>> 	- Deal with the error path in genpd_runtime_resume().

> 

> I tested this on NVIDIA Tegra by removing the boilerplate code from

> drivers' RPM and haven't noticed any problems, the performance state is

> dropped/restored as expected. Thank you.

> 


Tested-by: Dmitry Osipenko <digetx@gmail.com>
Viresh Kumar June 4, 2021, 3:53 a.m. | #6
On 03-06-21, 13:17, Ulf Hansson wrote:
> On Thu, 3 Jun 2021 at 12:31, Ulf Hansson <ulf.hansson@linaro.org> wrote:

> > > > +static int genpd_drop_performance_state(struct device *dev)

> > > > +{

> > > > +     unsigned int prev_state = dev_gpd_data(dev)->performance_state;

> > > > +

> > > > +     if (!genpd_set_performance_state(dev, 0))

> > > > +             return prev_state;

> > > > +

> > > > +     return 0;

> > > > +}

> > > > +

> > > > +static void genpd_restore_performance_state(struct device *dev,

> > > > +                                         unsigned int state)

> > > > +{

> > > > +     if (state)

> > >

> > > I will skip this check, as we are checking it in

> > > genpd_set_performance_state() anyway ?

> >

> > I don't want us to override OPP votes made by the subsystem/driver

> > level runtime PM callbacks. For example, if the drivers manage this

> > thing themselves, that should be preserved.

> >

> > That said, by the check above I want to avoid setting the state to

> > zero internally by genpd, if the driver level ->runtime_resume()

> > callback has already restored the state.

> 

> Ehh, forget about what I said about the ->runtime_resume() callback.

> 

> I am mostly trying to avoid restoring a state that is zero, just to be

> sure nobody else on some different level outside gendp, have decided

> to set a new OPP in-between our calls to

> genpd_drop|restore_performance state.


What stops the core to call genpd_drop_performance_state() in the
first place here, if the driver was doing its own thing ? If that gets
called, then restore should be without any checks IMO. The state
should already be 0 at this point of time, I don't know why this will
get called again with state 0, but it will have no effect.

Can you give some sort of flow sequence where I can see the problem a
bit more clearly ?

-- 
viresh
Ulf Hansson June 4, 2021, 7:20 a.m. | #7
On Thu, 3 Jun 2021 at 21:08, Dmitry Osipenko <digetx@gmail.com> wrote:
>

> 03.06.2021 22:02, Dmitry Osipenko пишет:

> > 03.06.2021 12:34, Ulf Hansson пишет:

> >> A subsystem/driver that need to manage OPPs for its device, should

> >> typically drop its vote for the OPP when the device becomes runtime

> >> suspended. In this way, the corresponding aggregation of the performance

> >> state votes that is managed in genpd for the attached PM domain, may find

> >> that the aggregated vote can be decreased. Hence, it may allow genpd to set

> >> the lower performance state for the PM domain, thus avoiding to waste

> >> energy.

> >>

> >> To accomplish this, typically a subsystem/driver would need to call

> >> dev_pm_opp_set_rate|opp() for its device from its ->runtime_suspend()

> >> callback, to drop the vote for the OPP. Accordingly, it needs another call

> >> to dev_pm_opp_set_rate|opp() to restore the vote for the OPP from its

> >> ->runtime_resume() callback.

> >>

> >> To avoid boilerplate code in subsystems/driver to deal with these things,

> >> let's instead manage this internally in genpd.

> >>

> >> Signed-off-by: Ulf Hansson <ulf.hansson@linaro.org>

> >> ---

> >>

> >> Changes in v2:

> >>      - Rebased.

> >>      - A few minor cosmetic changes.

> >>      - Deal with the error path in genpd_runtime_resume().

> >

> > I tested this on NVIDIA Tegra by removing the boilerplate code from

> > drivers' RPM and haven't noticed any problems, the performance state is

> > dropped/restored as expected. Thank you.

> >

>

> Tested-by: Dmitry Osipenko <digetx@gmail.com>


Thanks a lot, much appreciated!

Kind regards
Uffe
Ulf Hansson June 4, 2021, 7:45 a.m. | #8
On Fri, 4 Jun 2021 at 05:53, Viresh Kumar <viresh.kumar@linaro.org> wrote:
>

> On 03-06-21, 13:17, Ulf Hansson wrote:

> > On Thu, 3 Jun 2021 at 12:31, Ulf Hansson <ulf.hansson@linaro.org> wrote:

> > > > > +static int genpd_drop_performance_state(struct device *dev)

> > > > > +{

> > > > > +     unsigned int prev_state = dev_gpd_data(dev)->performance_state;

> > > > > +

> > > > > +     if (!genpd_set_performance_state(dev, 0))

> > > > > +             return prev_state;

> > > > > +

> > > > > +     return 0;

> > > > > +}

> > > > > +

> > > > > +static void genpd_restore_performance_state(struct device *dev,

> > > > > +                                         unsigned int state)

> > > > > +{

> > > > > +     if (state)

> > > >

> > > > I will skip this check, as we are checking it in

> > > > genpd_set_performance_state() anyway ?

> > >

> > > I don't want us to override OPP votes made by the subsystem/driver

> > > level runtime PM callbacks. For example, if the drivers manage this

> > > thing themselves, that should be preserved.

> > >

> > > That said, by the check above I want to avoid setting the state to

> > > zero internally by genpd, if the driver level ->runtime_resume()

> > > callback has already restored the state.

> >

> > Ehh, forget about what I said about the ->runtime_resume() callback.

> >

> > I am mostly trying to avoid restoring a state that is zero, just to be

> > sure nobody else on some different level outside gendp, have decided

> > to set a new OPP in-between our calls to

> > genpd_drop|restore_performance state.

>

> What stops the core to call genpd_drop_performance_state() in the

> first place here, if the driver was doing its own thing ? If that gets

> called, then restore should be without any checks IMO. The state

> should already be 0 at this point of time, I don't know why this will

> get called again with state 0, but it will have no effect.

>

> Can you give some sort of flow sequence where I can see the problem a

> bit more clearly ?


Starting calls from the subsystem/driver:

------
dev_pm_genpd_set_performance_state(dev, 100);
"run a use case with device runtime resumed"
...
"use case ends"
dev_pm_genpd_set_performance_state(dev, 0);
pm_runtime_put()
    ->genpd_runtime_suspend()
    gpd_data->performance_state == 0, -> gpd_data->rpm_pstate = 0;
...
"new use case start"
dev_pm_genpd_set_performance_state(dev, 100);
pm_runtime_get_sync()
    ->genpd_runtime_resume()
    gpd_data->performance_state == 100, -> gpd_data->rpm_pstate = 0;
(This is where we need to check for "zero" to not override the value)
.....
------

I wouldn't say that the above is the way how I see the calls to
dev_pm_genpd_set_performance_state (or actually
dev_pm_opp_set_rate|opp()) being deployed. The calls should rather be
done from the subsystem/driver's ->runtime_suspend|resume() callback,
then the path above would work in the way you suggest.

Although, as we currently treat performance states and power states in
genpd orthogonally, I wanted to make sure we could cope with both
situations.

Did this help? :-)

Kind regards
Uffe
Viresh Kumar June 7, 2021, 4:47 a.m. | #9
On 04-06-21, 09:45, Ulf Hansson wrote:
> Starting calls from the subsystem/driver:

> 

> ------

> dev_pm_genpd_set_performance_state(dev, 100);

> "run a use case with device runtime resumed"

> ...

> "use case ends"

> dev_pm_genpd_set_performance_state(dev, 0);

> pm_runtime_put()

>     ->genpd_runtime_suspend()

>     gpd_data->performance_state == 0, -> gpd_data->rpm_pstate = 0;

> ...

> "new use case start"

> dev_pm_genpd_set_performance_state(dev, 100);

> pm_runtime_get_sync()

>     ->genpd_runtime_resume()

>     gpd_data->performance_state == 100, -> gpd_data->rpm_pstate = 0;

> (This is where we need to check for "zero" to not override the value)

> .....

> ------

> 

> I wouldn't say that the above is the way how I see the calls to

> dev_pm_genpd_set_performance_state (or actually

> dev_pm_opp_set_rate|opp()) being deployed. The calls should rather be

> done from the subsystem/driver's ->runtime_suspend|resume() callback,

> then the path above would work in the way you suggest.

> 

> Although, as we currently treat performance states and power states in

> genpd orthogonally, I wanted to make sure we could cope with both

> situations.


I think letting the drivers to call
dev_pm_genpd_set_performance_state(dev, 0) from suspend/resume makes
it really ugly/racy as both depend on the gpd_data->performance_state
for this. It doesn't look nice. And we shouldn't try to protect such
drivers.

Anyway, your call :)

> Did this help? :-)


Yes :)

-- 
viresh
Ulf Hansson June 9, 2021, 12:25 p.m. | #10
On Mon, 7 Jun 2021 at 06:47, Viresh Kumar <viresh.kumar@linaro.org> wrote:
>

> On 04-06-21, 09:45, Ulf Hansson wrote:

> > Starting calls from the subsystem/driver:

> >

> > ------

> > dev_pm_genpd_set_performance_state(dev, 100);

> > "run a use case with device runtime resumed"

> > ...

> > "use case ends"

> > dev_pm_genpd_set_performance_state(dev, 0);

> > pm_runtime_put()

> >     ->genpd_runtime_suspend()

> >     gpd_data->performance_state == 0, -> gpd_data->rpm_pstate = 0;

> > ...

> > "new use case start"

> > dev_pm_genpd_set_performance_state(dev, 100);

> > pm_runtime_get_sync()

> >     ->genpd_runtime_resume()

> >     gpd_data->performance_state == 100, -> gpd_data->rpm_pstate = 0;

> > (This is where we need to check for "zero" to not override the value)

> > .....

> > ------

> >

> > I wouldn't say that the above is the way how I see the calls to

> > dev_pm_genpd_set_performance_state (or actually

> > dev_pm_opp_set_rate|opp()) being deployed. The calls should rather be

> > done from the subsystem/driver's ->runtime_suspend|resume() callback,

> > then the path above would work in the way you suggest.

> >

> > Although, as we currently treat performance states and power states in

> > genpd orthogonally, I wanted to make sure we could cope with both

> > situations.

>

> I think letting the drivers to call

> dev_pm_genpd_set_performance_state(dev, 0) from suspend/resume makes

> it really ugly/racy as both depend on the gpd_data->performance_state

> for this. It doesn't look nice. And we shouldn't try to protect such

> drivers.

>

> Anyway, your call :)


Well, I am not sure we have an option at this point. As long as we
allow performance states to be managed orthogonally to on/off states
in genpd, the check in genpd_restore_performance_state() is needed.

I have started to prepare a new version of the series - and will add a
comment about this in the code to try to clarify this.

[...]

Kind regards
Uffe

Patch

diff --git a/drivers/base/power/domain.c b/drivers/base/power/domain.c
index ef25a5b18587..e5d97174c254 100644
--- a/drivers/base/power/domain.c
+++ b/drivers/base/power/domain.c
@@ -400,6 +400,23 @@  static int genpd_set_performance_state(struct device *dev, unsigned int state)
 	return ret;
 }
 
+static int genpd_drop_performance_state(struct device *dev)
+{
+	unsigned int prev_state = dev_gpd_data(dev)->performance_state;
+
+	if (!genpd_set_performance_state(dev, 0))
+		return prev_state;
+
+	return 0;
+}
+
+static void genpd_restore_performance_state(struct device *dev,
+					    unsigned int state)
+{
+	if (state)
+		genpd_set_performance_state(dev, state);
+}
+
 /**
  * dev_pm_genpd_set_performance_state- Set performance state of device's power
  * domain.
@@ -842,7 +859,8 @@  static int genpd_runtime_suspend(struct device *dev)
 {
 	struct generic_pm_domain *genpd;
 	bool (*suspend_ok)(struct device *__dev);
-	struct gpd_timing_data *td = &dev_gpd_data(dev)->td;
+	struct generic_pm_domain_data *gpd_data = dev_gpd_data(dev);
+	struct gpd_timing_data *td = &gpd_data->td;
 	bool runtime_pm = pm_runtime_enabled(dev);
 	ktime_t time_start;
 	s64 elapsed_ns;
@@ -899,6 +917,7 @@  static int genpd_runtime_suspend(struct device *dev)
 		return 0;
 
 	genpd_lock(genpd);
+	gpd_data->rpm_pstate = genpd_drop_performance_state(dev);
 	genpd_power_off(genpd, true, 0);
 	genpd_unlock(genpd);
 
@@ -916,7 +935,8 @@  static int genpd_runtime_suspend(struct device *dev)
 static int genpd_runtime_resume(struct device *dev)
 {
 	struct generic_pm_domain *genpd;
-	struct gpd_timing_data *td = &dev_gpd_data(dev)->td;
+	struct generic_pm_domain_data *gpd_data = dev_gpd_data(dev);
+	struct gpd_timing_data *td = &gpd_data->td;
 	bool runtime_pm = pm_runtime_enabled(dev);
 	ktime_t time_start;
 	s64 elapsed_ns;
@@ -940,6 +960,8 @@  static int genpd_runtime_resume(struct device *dev)
 
 	genpd_lock(genpd);
 	ret = genpd_power_on(genpd, 0);
+	if (!ret)
+		genpd_restore_performance_state(dev, gpd_data->rpm_pstate);
 	genpd_unlock(genpd);
 
 	if (ret)
@@ -978,6 +1000,7 @@  static int genpd_runtime_resume(struct device *dev)
 err_poweroff:
 	if (!pm_runtime_is_irq_safe(dev) || genpd_is_irq_safe(genpd)) {
 		genpd_lock(genpd);
+		gpd_data->rpm_pstate = genpd_drop_performance_state(dev);
 		genpd_power_off(genpd, true, 0);
 		genpd_unlock(genpd);
 	}
diff --git a/include/linux/pm_domain.h b/include/linux/pm_domain.h
index dfcfbcecc34b..21a0577305ef 100644
--- a/include/linux/pm_domain.h
+++ b/include/linux/pm_domain.h
@@ -198,6 +198,7 @@  struct generic_pm_domain_data {
 	struct notifier_block *power_nb;
 	int cpu;
 	unsigned int performance_state;
+	unsigned int rpm_pstate;
 	ktime_t	next_wakeup;
 	void *data;
 };