diff mbox series

[1/3] arm64: dts: qcom: sc8280xp: Fix PCIe PHY power-domains

Message ID 20231227-topic-8280_pcie_dts-v1-1-13d12b1698ff@linaro.org
State New
Headers show
Series SC8280XP preparatory PCIe fixes | expand

Commit Message

Konrad Dybcio Dec. 27, 2023, 10:28 p.m. UTC
The PCIe GDSCs are only related to the RCs. The PCIe PHYs on the other
hand, are powered by VDD_MX and their specific VDDA_PHY/PLL regulators.

Fix the power-domains assignment to stop potentially toggling the GDSC
unnecessarily.

Fixes: 813e83157001 ("arm64: dts: qcom: sc8280xp/sa8540p: add PCIe2-4 nodes")
Signed-off-by: Konrad Dybcio <konrad.dybcio@linaro.org>
---
 arch/arm64/boot/dts/qcom/sc8280xp.dtsi | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)

Comments

Konrad Dybcio Dec. 29, 2023, 1:08 p.m. UTC | #1
On 29.12.2023 12:24, Johan Hovold wrote:
> On Wed, Dec 27, 2023 at 11:28:26PM +0100, Konrad Dybcio wrote:
>> The PCIe GDSCs are only related to the RCs. The PCIe PHYs on the other
>> hand, are powered by VDD_MX and their specific VDDA_PHY/PLL regulators.
> 
> No, that does not seem to be entirely correct. I added the power-domains
> here precisely because they were needed to enable the PHYs.
> 
> This is something I stumbled over when trying to figure out how to
> add support for the second lane pair (i.e. four-lane mode), and I just
> went back and confirmed that this is still the case.
> 
> If you try to enable one of these PHYs without the corresponding GDSC
> being enabled, you end up with:
> 
> [   37.709324] ------------[ cut here ]------------
> [   37.718196] gcc_pcie_3b_aux_clk status stuck at 'off'
> [   37.718205] WARNING: CPU: 4 PID: 482 at drivers/clk/qcom/clk-branch.c:86 clk_branch_wait+0x144/0x15c
> 	
> Now, you may or may not want to describe the above in the devicetree,
> but this makes it sound like you're trying to work around an issue with
> the current Linux implementation.
Could you please recheck this with patch 1 from [1] applied?

Konrad

[1] https://lore.kernel.org/linux-arm-msm/20231227-topic-8280_pcie-v1-1-095491baf9e4@linaro.org/T/#u
Johan Hovold Dec. 29, 2023, 1:20 p.m. UTC | #2
On Fri, Dec 29, 2023 at 02:08:25PM +0100, Konrad Dybcio wrote:
> On 29.12.2023 12:24, Johan Hovold wrote:
> > On Wed, Dec 27, 2023 at 11:28:26PM +0100, Konrad Dybcio wrote:
> >> The PCIe GDSCs are only related to the RCs. The PCIe PHYs on the other
> >> hand, are powered by VDD_MX and their specific VDDA_PHY/PLL regulators.
> > 
> > No, that does not seem to be entirely correct. I added the power-domains
> > here precisely because they were needed to enable the PHYs.

> > If you try to enable one of these PHYs without the corresponding GDSC
> > being enabled, you end up with:
> > 
> > [   37.709324] ------------[ cut here ]------------
> > [   37.718196] gcc_pcie_3b_aux_clk status stuck at 'off'
> > [   37.718205] WARNING: CPU: 4 PID: 482 at drivers/clk/qcom/clk-branch.c:86 clk_branch_wait+0x144/0x15c
> > 	
> > Now, you may or may not want to describe the above in the devicetree,
> > but this makes it sound like you're trying to work around an issue with
> > the current Linux implementation.

> Could you please recheck this with patch 1 from [1] applied?

As expected that makes no difference as I'm powering on the PHY without
the corresponding controller being enabled (which otherwise guarantees
the GDSC to be on).

> [1] https://lore.kernel.org/linux-arm-msm/20231227-topic-8280_pcie-v1-1-095491baf9e4@linaro.org/T/#u

Johan
Manivannan Sadhasivam Dec. 29, 2023, 5:03 p.m. UTC | #3
On Fri, Dec 29, 2023 at 12:24:55PM +0100, Johan Hovold wrote:
> On Wed, Dec 27, 2023 at 11:28:26PM +0100, Konrad Dybcio wrote:
> > The PCIe GDSCs are only related to the RCs. The PCIe PHYs on the other
> > hand, are powered by VDD_MX and their specific VDDA_PHY/PLL regulators.
> 
> No, that does not seem to be entirely correct. I added the power-domains
> here precisely because they were needed to enable the PHYs.
> 
> This is something I stumbled over when trying to figure out how to
> add support for the second lane pair (i.e. four-lane mode), and I just
> went back and confirmed that this is still the case.
> 
> If you try to enable one of these PHYs without the corresponding GDSC
> being enabled, you end up with:
> 
> [   37.709324] ------------[ cut here ]------------
> [   37.718196] gcc_pcie_3b_aux_clk status stuck at 'off'
> [   37.718205] WARNING: CPU: 4 PID: 482 at drivers/clk/qcom/clk-branch.c:86 clk_branch_wait+0x144/0x15c
> 	

Technically this patch is correct. PHYs are backed by MX domain only and not
GDSCs. Only the controllers (PCIe, UFS, USB) are backed by GDSCs. The fact that
you are seeing issue with PCIe Aux clock suggests me that this clock may not be
applicable to the PHY but it needs to be enabled for working of the PHY somehow.
I'll try to find the details on how exactly it is needed.

But if I get the answer like, "This clock is also sourced to PHY directly", then
we may need to add dual power domain for PHY (both GDSC and MX).

> Now, you may or may not want to describe the above in the devicetree,
> but this makes it sound like you're trying to work around an issue with
> the current Linux implementation.
> 

Adding MX domain to PHY in devicetree is definitely not a workaround. It is the
actual hardware representation. MX is the always on domain, and when CX collapse
happens during suspend state, it will ensure that all the analog components
(like PHY) are kept powered on. Otherwise, we will see link down issues.

But, I heard from Qcom that _only_ on this platform, MX is not backing the PCIe
PHY. I can correlate that with my encounter with PCIe issues after forcing CX
power collapse.

I haven't looked in detail on how this series fixes that issue though.

- Mani

> > Fix the power-domains assignment to stop potentially toggling the GDSC
> > unnecessarily.
> 
> Nothing is being toggled unnecessarily, and generally this is just
> another use count increment.
> 
> > Fixes: 813e83157001 ("arm64: dts: qcom: sc8280xp/sa8540p: add PCIe2-4 nodes")
> 
> So not sure a Fixes tag is warranted either.
> 
> > @@ -1895,7 +1895,7 @@ pcie3b_phy: phy@1c0e000 {
> >  			assigned-clocks = <&gcc GCC_PCIE3B_PHY_RCHNG_CLK>;
> >  			assigned-clock-rates = <100000000>;
> >  
> > -			power-domains = <&gcc PCIE_3B_GDSC>;
> > +			power-domains = <&rpmhpd SC8280XP_MX>;
> >  
> >  			resets = <&gcc GCC_PCIE_3B_PHY_BCR>;
> >  			reset-names = "phy";
> 
> Johan
>
Konrad Dybcio Dec. 30, 2023, 12:28 p.m. UTC | #4
On 29.12.2023 18:03, Manivannan Sadhasivam wrote:
> On Fri, Dec 29, 2023 at 12:24:55PM +0100, Johan Hovold wrote:
>> On Wed, Dec 27, 2023 at 11:28:26PM +0100, Konrad Dybcio wrote:
>>> The PCIe GDSCs are only related to the RCs. The PCIe PHYs on the other
>>> hand, are powered by VDD_MX and their specific VDDA_PHY/PLL regulators.
>>
>> No, that does not seem to be entirely correct. I added the power-domains
>> here precisely because they were needed to enable the PHYs.
>>
>> This is something I stumbled over when trying to figure out how to
>> add support for the second lane pair (i.e. four-lane mode), and I just
>> went back and confirmed that this is still the case.
>>
>> If you try to enable one of these PHYs without the corresponding GDSC
>> being enabled, you end up with:
>>
>> [   37.709324] ------------[ cut here ]------------
>> [   37.718196] gcc_pcie_3b_aux_clk status stuck at 'off'
>> [   37.718205] WARNING: CPU: 4 PID: 482 at drivers/clk/qcom/clk-branch.c:86 clk_branch_wait+0x144/0x15c
>> 	
> 
> Technically this patch is correct. PHYs are backed by MX domain only and not
> GDSCs. Only the controllers (PCIe, UFS, USB) are backed by GDSCs. The fact that
> you are seeing issue with PCIe Aux clock suggests me that this clock may not be
> applicable to the PHY but it needs to be enabled for working of the PHY somehow.
> I'll try to find the details on how exactly it is needed.
> 
> But if I get the answer like, "This clock is also sourced to PHY directly", then
> we may need to add dual power domain for PHY (both GDSC and MX).
> 
>> Now, you may or may not want to describe the above in the devicetree,
>> but this makes it sound like you're trying to work around an issue with
>> the current Linux implementation.

I did a bit of experimentation, and.. I think that the PHY itself doesn't
need the GDSC to be enabled.

However.

The AUX clock requires the GDSC to be enabled and the PHY will fail to
power on if this clock is disabled.

That makes me wonder if representing the PCIe PHY as a wholly separate
device (instead of e.g. it being a subdev of PCIe RC) is even correct..

>>
> 
> Adding MX domain to PHY in devicetree is definitely not a workaround. It is the
> actual hardware representation. MX is the always on domain, and when CX collapse
> happens during suspend state, it will ensure that all the analog components
> (like PHY) are kept powered on. Otherwise, we will see link down issues.
> 
> But, I heard from Qcom that _only_ on this platform, MX is not backing the PCIe
> PHY. I can correlate that with my encounter with PCIe issues after forcing CX
> power collapse.
I've heard otherwise, the PHY itself is powered by MX, but CX needs
to be (should be?) enabled for communication with the RC (which itself
needs CX to be up to function).

Konrad
Johan Hovold Jan. 22, 2024, 5:36 p.m. UTC | #5
On Mon, Jan 22, 2024 at 10:55:28PM +0530, Manivannan Sadhasivam wrote:
> On Fri, Dec 29, 2023 at 10:33:34PM +0530, Manivannan Sadhasivam wrote:
> > On Fri, Dec 29, 2023 at 12:24:55PM +0100, Johan Hovold wrote:
> > > On Wed, Dec 27, 2023 at 11:28:26PM +0100, Konrad Dybcio wrote:
> > > > The PCIe GDSCs are only related to the RCs. The PCIe PHYs on the other
> > > > hand, are powered by VDD_MX and their specific VDDA_PHY/PLL regulators.
> > > 
> > > No, that does not seem to be entirely correct. I added the power-domains
> > > here precisely because they were needed to enable the PHYs.
> > > 
> > > This is something I stumbled over when trying to figure out how to
> > > add support for the second lane pair (i.e. four-lane mode), and I just
> > > went back and confirmed that this is still the case.
> > > 
> > > If you try to enable one of these PHYs without the corresponding GDSC
> > > being enabled, you end up with:
> > > 
> > > [   37.709324] ------------[ cut here ]------------
> > > [   37.718196] gcc_pcie_3b_aux_clk status stuck at 'off'
> > > [   37.718205] WARNING: CPU: 4 PID: 482 at drivers/clk/qcom/clk-branch.c:86 clk_branch_wait+0x144/0x15c
> > > 	
> > 
> > Technically this patch is correct. PHYs are backed by MX domain only and not
> > GDSCs. Only the controllers (PCIe, UFS, USB) are backed by GDSCs. The fact that
> > you are seeing issue with PCIe Aux clock suggests me that this clock may not be
> > applicable to the PHY but it needs to be enabled for working of the PHY somehow.
> > I'll try to find the details on how exactly it is needed.
> > 
> > But if I get the answer like, "This clock is also sourced to PHY directly", then
> > we may need to add dual power domain for PHY (both GDSC and MX).
> > 
> 
> So I answer I got from Qcom is that this clock is only applicable to the PCIe
> controller and not PHYs. On some platforms, there is a separate PCIE_PHY_AUX_CLK
> coming from GCC that is used during L1SS state. I think that caused confusion
> while adding PHY support for followup platforms and folks just used PCIE_AUX_CLK
> since they couldn't find the actual PCIE_PHY_AUX_CLK.

Thanks for sorting that out.

> I've prepared a series to fix this mess, but I want to know how you end up
> seeing the above "clk status stuck at off" issue. Is there an actual usecase for
> powering up PHY without controller or you just experimented with it?

As I mentioned, I ran into this when experimenting with how to enable
the "companion" PHY for four-lane support. There shouldn't be any use
case for it (apart from using it to determine that the current
description of the PHY resources is incomplete or incorrect).

Johan
Manivannan Sadhasivam Jan. 23, 2024, 5:06 p.m. UTC | #6
On Mon, Jan 22, 2024 at 06:36:51PM +0100, Johan Hovold wrote:
> On Mon, Jan 22, 2024 at 10:55:28PM +0530, Manivannan Sadhasivam wrote:
> > On Fri, Dec 29, 2023 at 10:33:34PM +0530, Manivannan Sadhasivam wrote:
> > > On Fri, Dec 29, 2023 at 12:24:55PM +0100, Johan Hovold wrote:
> > > > On Wed, Dec 27, 2023 at 11:28:26PM +0100, Konrad Dybcio wrote:
> > > > > The PCIe GDSCs are only related to the RCs. The PCIe PHYs on the other
> > > > > hand, are powered by VDD_MX and their specific VDDA_PHY/PLL regulators.
> > > > 
> > > > No, that does not seem to be entirely correct. I added the power-domains
> > > > here precisely because they were needed to enable the PHYs.
> > > > 
> > > > This is something I stumbled over when trying to figure out how to
> > > > add support for the second lane pair (i.e. four-lane mode), and I just
> > > > went back and confirmed that this is still the case.
> > > > 
> > > > If you try to enable one of these PHYs without the corresponding GDSC
> > > > being enabled, you end up with:
> > > > 
> > > > [   37.709324] ------------[ cut here ]------------
> > > > [   37.718196] gcc_pcie_3b_aux_clk status stuck at 'off'
> > > > [   37.718205] WARNING: CPU: 4 PID: 482 at drivers/clk/qcom/clk-branch.c:86 clk_branch_wait+0x144/0x15c
> > > > 	
> > > 
> > > Technically this patch is correct. PHYs are backed by MX domain only and not
> > > GDSCs. Only the controllers (PCIe, UFS, USB) are backed by GDSCs. The fact that
> > > you are seeing issue with PCIe Aux clock suggests me that this clock may not be
> > > applicable to the PHY but it needs to be enabled for working of the PHY somehow.
> > > I'll try to find the details on how exactly it is needed.
> > > 
> > > But if I get the answer like, "This clock is also sourced to PHY directly", then
> > > we may need to add dual power domain for PHY (both GDSC and MX).
> > > 
> > 
> > So I answer I got from Qcom is that this clock is only applicable to the PCIe
> > controller and not PHYs. On some platforms, there is a separate PCIE_PHY_AUX_CLK
> > coming from GCC that is used during L1SS state. I think that caused confusion
> > while adding PHY support for followup platforms and folks just used PCIE_AUX_CLK
> > since they couldn't find the actual PCIE_PHY_AUX_CLK.
> 
> Thanks for sorting that out.
> 
> > I've prepared a series to fix this mess, but I want to know how you end up
> > seeing the above "clk status stuck at off" issue. Is there an actual usecase for
> > powering up PHY without controller or you just experimented with it?
> 
> As I mentioned, I ran into this when experimenting with how to enable
> the "companion" PHY for four-lane support. There shouldn't be any use
> case for it (apart from using it to determine that the current
> description of the PHY resources is incomplete or incorrect).
> 

Ok. I tested by enabling the PHY clocks during qmp_pcie_clk_init() without
PCIE_GDSC. It worked for one instance of the PHY which doesn't have
PCIE_PHY_AUX_CLK, but for the PHY instance with this clock, I saw the same "clk
stuck" issue. Then checking the internal documentation revealed that this clock
needs PCIE_GDSC to become functional >.<

So to conclude, PCIE_AUX_CLK belongs to the controller and it needs GDSC. And
PCIE_PHY_AUX_CLK belongs to the PHY and it also needs GDSC.

I will just submit a series to remove the PCIE_AUX_CLK from PHY nodes. Then
in another series, I'll remove the GDSC for PHY instances that do not require
PCIE_PHY_AUX_CLK.

Hope this makes sense.

- Mani
Konrad Dybcio Jan. 23, 2024, 6:41 p.m. UTC | #7
On 1/23/24 18:06, Manivannan Sadhasivam wrote:
> On Mon, Jan 22, 2024 at 06:36:51PM +0100, Johan Hovold wrote:
>> On Mon, Jan 22, 2024 at 10:55:28PM +0530, Manivannan Sadhasivam wrote:
>>> On Fri, Dec 29, 2023 at 10:33:34PM +0530, Manivannan Sadhasivam wrote:
>>>> On Fri, Dec 29, 2023 at 12:24:55PM +0100, Johan Hovold wrote:
>>>>> On Wed, Dec 27, 2023 at 11:28:26PM +0100, Konrad Dybcio wrote:
>>>>>> The PCIe GDSCs are only related to the RCs. The PCIe PHYs on the other
>>>>>> hand, are powered by VDD_MX and their specific VDDA_PHY/PLL regulators.
>>>>>
>>>>> No, that does not seem to be entirely correct. I added the power-domains
>>>>> here precisely because they were needed to enable the PHYs.
>>>>>
>>>>> This is something I stumbled over when trying to figure out how to
>>>>> add support for the second lane pair (i.e. four-lane mode), and I just
>>>>> went back and confirmed that this is still the case.
>>>>>
>>>>> If you try to enable one of these PHYs without the corresponding GDSC
>>>>> being enabled, you end up with:
>>>>>
>>>>> [   37.709324] ------------[ cut here ]------------
>>>>> [   37.718196] gcc_pcie_3b_aux_clk status stuck at 'off'
>>>>> [   37.718205] WARNING: CPU: 4 PID: 482 at drivers/clk/qcom/clk-branch.c:86 clk_branch_wait+0x144/0x15c
>>>>> 	
>>>>
>>>> Technically this patch is correct. PHYs are backed by MX domain only and not
>>>> GDSCs. Only the controllers (PCIe, UFS, USB) are backed by GDSCs. The fact that
>>>> you are seeing issue with PCIe Aux clock suggests me that this clock may not be
>>>> applicable to the PHY but it needs to be enabled for working of the PHY somehow.
>>>> I'll try to find the details on how exactly it is needed.
>>>>
>>>> But if I get the answer like, "This clock is also sourced to PHY directly", then
>>>> we may need to add dual power domain for PHY (both GDSC and MX).
>>>>
>>>
>>> So I answer I got from Qcom is that this clock is only applicable to the PCIe
>>> controller and not PHYs. On some platforms, there is a separate PCIE_PHY_AUX_CLK
>>> coming from GCC that is used during L1SS state. I think that caused confusion
>>> while adding PHY support for followup platforms and folks just used PCIE_AUX_CLK
>>> since they couldn't find the actual PCIE_PHY_AUX_CLK.
>>
>> Thanks for sorting that out.
>>
>>> I've prepared a series to fix this mess, but I want to know how you end up
>>> seeing the above "clk status stuck at off" issue. Is there an actual usecase for
>>> powering up PHY without controller or you just experimented with it?
>>
>> As I mentioned, I ran into this when experimenting with how to enable
>> the "companion" PHY for four-lane support. There shouldn't be any use
>> case for it (apart from using it to determine that the current
>> description of the PHY resources is incomplete or incorrect).
>>
> 
> Ok. I tested by enabling the PHY clocks during qmp_pcie_clk_init() without
> PCIE_GDSC. It worked for one instance of the PHY which doesn't have
> PCIE_PHY_AUX_CLK, but for the PHY instance with this clock, I saw the same "clk
> stuck" issue. Then checking the internal documentation revealed that this clock
> needs PCIE_GDSC to become functional >.<
> 
> So to conclude, PCIE_AUX_CLK belongs to the controller and it needs GDSC. And
> PCIE_PHY_AUX_CLK belongs to the PHY and it also needs GDSC.
> 
> I will just submit a series to remove the PCIE_AUX_CLK from PHY nodes. Then
> in another series, I'll remove the GDSC for PHY instances that do not require
> PCIE_PHY_AUX_CLK.
> 
> Hope this makes sense.

Thanks, Mani

Konrad
Johan Hovold Jan. 24, 2024, 7:31 a.m. UTC | #8
On Tue, Jan 23, 2024 at 10:36:14PM +0530, Manivannan Sadhasivam wrote:
> On Mon, Jan 22, 2024 at 06:36:51PM +0100, Johan Hovold wrote:

> Ok. I tested by enabling the PHY clocks during qmp_pcie_clk_init() without
> PCIE_GDSC. It worked for one instance of the PHY which doesn't have
> PCIE_PHY_AUX_CLK, but for the PHY instance with this clock, I saw the same "clk
> stuck" issue. Then checking the internal documentation revealed that this clock
> needs PCIE_GDSC to become functional >.<
> 
> So to conclude, PCIE_AUX_CLK belongs to the controller and it needs GDSC. And
> PCIE_PHY_AUX_CLK belongs to the PHY and it also needs GDSC.
> 
> I will just submit a series to remove the PCIE_AUX_CLK from PHY nodes. Then
> in another series, I'll remove the GDSC for PHY instances that do not require
> PCIE_PHY_AUX_CLK.

Sounds good, thanks.

Johan
diff mbox series

Patch

diff --git a/arch/arm64/boot/dts/qcom/sc8280xp.dtsi b/arch/arm64/boot/dts/qcom/sc8280xp.dtsi
index febf28356ff8..72c5818b67f2 100644
--- a/arch/arm64/boot/dts/qcom/sc8280xp.dtsi
+++ b/arch/arm64/boot/dts/qcom/sc8280xp.dtsi
@@ -1797,7 +1797,7 @@  pcie4_phy: phy@1c06000 {
 			assigned-clocks = <&gcc GCC_PCIE4_PHY_RCHNG_CLK>;
 			assigned-clock-rates = <100000000>;
 
-			power-domains = <&gcc PCIE_4_GDSC>;
+			power-domains = <&rpmhpd SC8280XP_MX>;
 
 			resets = <&gcc GCC_PCIE_4_PHY_BCR>;
 			reset-names = "phy";
@@ -1895,7 +1895,7 @@  pcie3b_phy: phy@1c0e000 {
 			assigned-clocks = <&gcc GCC_PCIE3B_PHY_RCHNG_CLK>;
 			assigned-clock-rates = <100000000>;
 
-			power-domains = <&gcc PCIE_3B_GDSC>;
+			power-domains = <&rpmhpd SC8280XP_MX>;
 
 			resets = <&gcc GCC_PCIE_3B_PHY_BCR>;
 			reset-names = "phy";
@@ -1994,7 +1994,7 @@  pcie3a_phy: phy@1c14000 {
 			assigned-clocks = <&gcc GCC_PCIE3A_PHY_RCHNG_CLK>;
 			assigned-clock-rates = <100000000>;
 
-			power-domains = <&gcc PCIE_3A_GDSC>;
+			power-domains = <&rpmhpd SC8280XP_MX>;
 
 			resets = <&gcc GCC_PCIE_3A_PHY_BCR>;
 			reset-names = "phy";
@@ -2094,7 +2094,7 @@  pcie2b_phy: phy@1c1e000 {
 			assigned-clocks = <&gcc GCC_PCIE2B_PHY_RCHNG_CLK>;
 			assigned-clock-rates = <100000000>;
 
-			power-domains = <&gcc PCIE_2B_GDSC>;
+			power-domains = <&rpmhpd SC8280XP_MX>;
 
 			resets = <&gcc GCC_PCIE_2B_PHY_BCR>;
 			reset-names = "phy";
@@ -2193,7 +2193,7 @@  pcie2a_phy: phy@1c24000 {
 			assigned-clocks = <&gcc GCC_PCIE2A_PHY_RCHNG_CLK>;
 			assigned-clock-rates = <100000000>;
 
-			power-domains = <&gcc PCIE_2A_GDSC>;
+			power-domains = <&rpmhpd SC8280XP_MX>;
 
 			resets = <&gcc GCC_PCIE_2A_PHY_BCR>;
 			reset-names = "phy";