| Message ID | 20231227-topic-8280_pcie_dts-v1-1-13d12b1698ff@linaro.org |
|---|---|
| State | New |
| Headers | show |
| Series | SC8280XP preparatory PCIe fixes | expand |
On Wed, Dec 27, 2023 at 11:28:26PM +0100, Konrad Dybcio wrote: > The PCIe GDSCs are only related to the RCs. The PCIe PHYs on the other > hand, are powered by VDD_MX and their specific VDDA_PHY/PLL regulators. No, that does not seem to be entirely correct. I added the power-domains here precisely because they were needed to enable the PHYs. This is something I stumbled over when trying to figure out how to add support for the second lane pair (i.e. four-lane mode), and I just went back and confirmed that this is still the case. If you try to enable one of these PHYs without the corresponding GDSC being enabled, you end up with: [ 37.709324] ------------[ cut here ]------------ [ 37.718196] gcc_pcie_3b_aux_clk status stuck at 'off' [ 37.718205] WARNING: CPU: 4 PID: 482 at drivers/clk/qcom/clk-branch.c:86 clk_branch_wait+0x144/0x15c Now, you may or may not want to describe the above in the devicetree, but this makes it sound like you're trying to work around an issue with the current Linux implementation. > Fix the power-domains assignment to stop potentially toggling the GDSC > unnecessarily. Nothing is being toggled unnecessarily, and generally this is just another use count increment. > Fixes: 813e83157001 ("arm64: dts: qcom: sc8280xp/sa8540p: add PCIe2-4 nodes") So not sure a Fixes tag is warranted either. > @@ -1895,7 +1895,7 @@ pcie3b_phy: phy@1c0e000 { > assigned-clocks = <&gcc GCC_PCIE3B_PHY_RCHNG_CLK>; > assigned-clock-rates = <100000000>; > > - power-domains = <&gcc PCIE_3B_GDSC>; > + power-domains = <&rpmhpd SC8280XP_MX>; > > resets = <&gcc GCC_PCIE_3B_PHY_BCR>; > reset-names = "phy"; Johan
On 29.12.2023 12:24, Johan Hovold wrote: > On Wed, Dec 27, 2023 at 11:28:26PM +0100, Konrad Dybcio wrote: >> The PCIe GDSCs are only related to the RCs. The PCIe PHYs on the other >> hand, are powered by VDD_MX and their specific VDDA_PHY/PLL regulators. > > No, that does not seem to be entirely correct. I added the power-domains > here precisely because they were needed to enable the PHYs. > > This is something I stumbled over when trying to figure out how to > add support for the second lane pair (i.e. four-lane mode), and I just > went back and confirmed that this is still the case. > > If you try to enable one of these PHYs without the corresponding GDSC > being enabled, you end up with: > > [ 37.709324] ------------[ cut here ]------------ > [ 37.718196] gcc_pcie_3b_aux_clk status stuck at 'off' > [ 37.718205] WARNING: CPU: 4 PID: 482 at drivers/clk/qcom/clk-branch.c:86 clk_branch_wait+0x144/0x15c > > Now, you may or may not want to describe the above in the devicetree, > but this makes it sound like you're trying to work around an issue with > the current Linux implementation. Could you please recheck this with patch 1 from [1] applied? Konrad [1] https://lore.kernel.org/linux-arm-msm/20231227-topic-8280_pcie-v1-1-095491baf9e4@linaro.org/T/#u
On Fri, Dec 29, 2023 at 02:08:25PM +0100, Konrad Dybcio wrote: > On 29.12.2023 12:24, Johan Hovold wrote: > > On Wed, Dec 27, 2023 at 11:28:26PM +0100, Konrad Dybcio wrote: > >> The PCIe GDSCs are only related to the RCs. The PCIe PHYs on the other > >> hand, are powered by VDD_MX and their specific VDDA_PHY/PLL regulators. > > > > No, that does not seem to be entirely correct. I added the power-domains > > here precisely because they were needed to enable the PHYs. > > If you try to enable one of these PHYs without the corresponding GDSC > > being enabled, you end up with: > > > > [ 37.709324] ------------[ cut here ]------------ > > [ 37.718196] gcc_pcie_3b_aux_clk status stuck at 'off' > > [ 37.718205] WARNING: CPU: 4 PID: 482 at drivers/clk/qcom/clk-branch.c:86 clk_branch_wait+0x144/0x15c > > > > Now, you may or may not want to describe the above in the devicetree, > > but this makes it sound like you're trying to work around an issue with > > the current Linux implementation. > Could you please recheck this with patch 1 from [1] applied? As expected that makes no difference as I'm powering on the PHY without the corresponding controller being enabled (which otherwise guarantees the GDSC to be on). > [1] https://lore.kernel.org/linux-arm-msm/20231227-topic-8280_pcie-v1-1-095491baf9e4@linaro.org/T/#u Johan
On Fri, Dec 29, 2023 at 12:24:55PM +0100, Johan Hovold wrote: > On Wed, Dec 27, 2023 at 11:28:26PM +0100, Konrad Dybcio wrote: > > The PCIe GDSCs are only related to the RCs. The PCIe PHYs on the other > > hand, are powered by VDD_MX and their specific VDDA_PHY/PLL regulators. > > No, that does not seem to be entirely correct. I added the power-domains > here precisely because they were needed to enable the PHYs. > > This is something I stumbled over when trying to figure out how to > add support for the second lane pair (i.e. four-lane mode), and I just > went back and confirmed that this is still the case. > > If you try to enable one of these PHYs without the corresponding GDSC > being enabled, you end up with: > > [ 37.709324] ------------[ cut here ]------------ > [ 37.718196] gcc_pcie_3b_aux_clk status stuck at 'off' > [ 37.718205] WARNING: CPU: 4 PID: 482 at drivers/clk/qcom/clk-branch.c:86 clk_branch_wait+0x144/0x15c > Technically this patch is correct. PHYs are backed by MX domain only and not GDSCs. Only the controllers (PCIe, UFS, USB) are backed by GDSCs. The fact that you are seeing issue with PCIe Aux clock suggests me that this clock may not be applicable to the PHY but it needs to be enabled for working of the PHY somehow. I'll try to find the details on how exactly it is needed. But if I get the answer like, "This clock is also sourced to PHY directly", then we may need to add dual power domain for PHY (both GDSC and MX). > Now, you may or may not want to describe the above in the devicetree, > but this makes it sound like you're trying to work around an issue with > the current Linux implementation. > Adding MX domain to PHY in devicetree is definitely not a workaround. It is the actual hardware representation. MX is the always on domain, and when CX collapse happens during suspend state, it will ensure that all the analog components (like PHY) are kept powered on. Otherwise, we will see link down issues. But, I heard from Qcom that _only_ on this platform, MX is not backing the PCIe PHY. I can correlate that with my encounter with PCIe issues after forcing CX power collapse. I haven't looked in detail on how this series fixes that issue though. - Mani > > Fix the power-domains assignment to stop potentially toggling the GDSC > > unnecessarily. > > Nothing is being toggled unnecessarily, and generally this is just > another use count increment. > > > Fixes: 813e83157001 ("arm64: dts: qcom: sc8280xp/sa8540p: add PCIe2-4 nodes") > > So not sure a Fixes tag is warranted either. > > > @@ -1895,7 +1895,7 @@ pcie3b_phy: phy@1c0e000 { > > assigned-clocks = <&gcc GCC_PCIE3B_PHY_RCHNG_CLK>; > > assigned-clock-rates = <100000000>; > > > > - power-domains = <&gcc PCIE_3B_GDSC>; > > + power-domains = <&rpmhpd SC8280XP_MX>; > > > > resets = <&gcc GCC_PCIE_3B_PHY_BCR>; > > reset-names = "phy"; > > Johan >
On 29.12.2023 18:03, Manivannan Sadhasivam wrote: > On Fri, Dec 29, 2023 at 12:24:55PM +0100, Johan Hovold wrote: >> On Wed, Dec 27, 2023 at 11:28:26PM +0100, Konrad Dybcio wrote: >>> The PCIe GDSCs are only related to the RCs. The PCIe PHYs on the other >>> hand, are powered by VDD_MX and their specific VDDA_PHY/PLL regulators. >> >> No, that does not seem to be entirely correct. I added the power-domains >> here precisely because they were needed to enable the PHYs. >> >> This is something I stumbled over when trying to figure out how to >> add support for the second lane pair (i.e. four-lane mode), and I just >> went back and confirmed that this is still the case. >> >> If you try to enable one of these PHYs without the corresponding GDSC >> being enabled, you end up with: >> >> [ 37.709324] ------------[ cut here ]------------ >> [ 37.718196] gcc_pcie_3b_aux_clk status stuck at 'off' >> [ 37.718205] WARNING: CPU: 4 PID: 482 at drivers/clk/qcom/clk-branch.c:86 clk_branch_wait+0x144/0x15c >> > > Technically this patch is correct. PHYs are backed by MX domain only and not > GDSCs. Only the controllers (PCIe, UFS, USB) are backed by GDSCs. The fact that > you are seeing issue with PCIe Aux clock suggests me that this clock may not be > applicable to the PHY but it needs to be enabled for working of the PHY somehow. > I'll try to find the details on how exactly it is needed. > > But if I get the answer like, "This clock is also sourced to PHY directly", then > we may need to add dual power domain for PHY (both GDSC and MX). > >> Now, you may or may not want to describe the above in the devicetree, >> but this makes it sound like you're trying to work around an issue with >> the current Linux implementation. I did a bit of experimentation, and.. I think that the PHY itself doesn't need the GDSC to be enabled. However. The AUX clock requires the GDSC to be enabled and the PHY will fail to power on if this clock is disabled. That makes me wonder if representing the PCIe PHY as a wholly separate device (instead of e.g. it being a subdev of PCIe RC) is even correct.. >> > > Adding MX domain to PHY in devicetree is definitely not a workaround. It is the > actual hardware representation. MX is the always on domain, and when CX collapse > happens during suspend state, it will ensure that all the analog components > (like PHY) are kept powered on. Otherwise, we will see link down issues. > > But, I heard from Qcom that _only_ on this platform, MX is not backing the PCIe > PHY. I can correlate that with my encounter with PCIe issues after forcing CX > power collapse. I've heard otherwise, the PHY itself is powered by MX, but CX needs to be (should be?) enabled for communication with the RC (which itself needs CX to be up to function). Konrad
On Fri, Dec 29, 2023 at 10:33:34PM +0530, Manivannan Sadhasivam wrote: > On Fri, Dec 29, 2023 at 12:24:55PM +0100, Johan Hovold wrote: > > On Wed, Dec 27, 2023 at 11:28:26PM +0100, Konrad Dybcio wrote: > > > The PCIe GDSCs are only related to the RCs. The PCIe PHYs on the other > > > hand, are powered by VDD_MX and their specific VDDA_PHY/PLL regulators. > > > > No, that does not seem to be entirely correct. I added the power-domains > > here precisely because they were needed to enable the PHYs. > > > > This is something I stumbled over when trying to figure out how to > > add support for the second lane pair (i.e. four-lane mode), and I just > > went back and confirmed that this is still the case. > > > > If you try to enable one of these PHYs without the corresponding GDSC > > being enabled, you end up with: > > > > [ 37.709324] ------------[ cut here ]------------ > > [ 37.718196] gcc_pcie_3b_aux_clk status stuck at 'off' > > [ 37.718205] WARNING: CPU: 4 PID: 482 at drivers/clk/qcom/clk-branch.c:86 clk_branch_wait+0x144/0x15c > > > > Technically this patch is correct. PHYs are backed by MX domain only and not > GDSCs. Only the controllers (PCIe, UFS, USB) are backed by GDSCs. The fact that > you are seeing issue with PCIe Aux clock suggests me that this clock may not be > applicable to the PHY but it needs to be enabled for working of the PHY somehow. > I'll try to find the details on how exactly it is needed. Sounds good, thanks. > But if I get the answer like, "This clock is also sourced to PHY directly", then > we may need to add dual power domain for PHY (both GDSC and MX). Right. > > Now, you may or may not want to describe the above in the devicetree, > > but this makes it sound like you're trying to work around an issue with > > the current Linux implementation. > > > > Adding MX domain to PHY in devicetree is definitely not a workaround. I was referring to the fact that the GDSC domain also appears to be needed even if that may possible get in the way when trying to implement suspend. > It is the > actual hardware representation. MX is the always on domain, and when CX collapse > happens during suspend state, it will ensure that all the analog components > (like PHY) are kept powered on. Otherwise, we will see link down issues. But if it's an always-on domain as you say, it should not be shut down, right? Perhaps you still want it described in DT for some other reason. > But, I heard from Qcom that _only_ on this platform, MX is not backing the PCIe > PHY. I can correlate that with my encounter with PCIe issues after forcing CX > power collapse. Ok, but that seems to imply that this patch is definitely *not* correct (for sc8280xp)? Johan
On Mon, Jan 22, 2024 at 10:55:28PM +0530, Manivannan Sadhasivam wrote: > On Fri, Dec 29, 2023 at 10:33:34PM +0530, Manivannan Sadhasivam wrote: > > On Fri, Dec 29, 2023 at 12:24:55PM +0100, Johan Hovold wrote: > > > On Wed, Dec 27, 2023 at 11:28:26PM +0100, Konrad Dybcio wrote: > > > > The PCIe GDSCs are only related to the RCs. The PCIe PHYs on the other > > > > hand, are powered by VDD_MX and their specific VDDA_PHY/PLL regulators. > > > > > > No, that does not seem to be entirely correct. I added the power-domains > > > here precisely because they were needed to enable the PHYs. > > > > > > This is something I stumbled over when trying to figure out how to > > > add support for the second lane pair (i.e. four-lane mode), and I just > > > went back and confirmed that this is still the case. > > > > > > If you try to enable one of these PHYs without the corresponding GDSC > > > being enabled, you end up with: > > > > > > [ 37.709324] ------------[ cut here ]------------ > > > [ 37.718196] gcc_pcie_3b_aux_clk status stuck at 'off' > > > [ 37.718205] WARNING: CPU: 4 PID: 482 at drivers/clk/qcom/clk-branch.c:86 clk_branch_wait+0x144/0x15c > > > > > > > Technically this patch is correct. PHYs are backed by MX domain only and not > > GDSCs. Only the controllers (PCIe, UFS, USB) are backed by GDSCs. The fact that > > you are seeing issue with PCIe Aux clock suggests me that this clock may not be > > applicable to the PHY but it needs to be enabled for working of the PHY somehow. > > I'll try to find the details on how exactly it is needed. > > > > But if I get the answer like, "This clock is also sourced to PHY directly", then > > we may need to add dual power domain for PHY (both GDSC and MX). > > > > So I answer I got from Qcom is that this clock is only applicable to the PCIe > controller and not PHYs. On some platforms, there is a separate PCIE_PHY_AUX_CLK > coming from GCC that is used during L1SS state. I think that caused confusion > while adding PHY support for followup platforms and folks just used PCIE_AUX_CLK > since they couldn't find the actual PCIE_PHY_AUX_CLK. Thanks for sorting that out. > I've prepared a series to fix this mess, but I want to know how you end up > seeing the above "clk status stuck at off" issue. Is there an actual usecase for > powering up PHY without controller or you just experimented with it? As I mentioned, I ran into this when experimenting with how to enable the "companion" PHY for four-lane support. There shouldn't be any use case for it (apart from using it to determine that the current description of the PHY resources is incomplete or incorrect). Johan
On Mon, Jan 22, 2024 at 06:36:51PM +0100, Johan Hovold wrote: > On Mon, Jan 22, 2024 at 10:55:28PM +0530, Manivannan Sadhasivam wrote: > > On Fri, Dec 29, 2023 at 10:33:34PM +0530, Manivannan Sadhasivam wrote: > > > On Fri, Dec 29, 2023 at 12:24:55PM +0100, Johan Hovold wrote: > > > > On Wed, Dec 27, 2023 at 11:28:26PM +0100, Konrad Dybcio wrote: > > > > > The PCIe GDSCs are only related to the RCs. The PCIe PHYs on the other > > > > > hand, are powered by VDD_MX and their specific VDDA_PHY/PLL regulators. > > > > > > > > No, that does not seem to be entirely correct. I added the power-domains > > > > here precisely because they were needed to enable the PHYs. > > > > > > > > This is something I stumbled over when trying to figure out how to > > > > add support for the second lane pair (i.e. four-lane mode), and I just > > > > went back and confirmed that this is still the case. > > > > > > > > If you try to enable one of these PHYs without the corresponding GDSC > > > > being enabled, you end up with: > > > > > > > > [ 37.709324] ------------[ cut here ]------------ > > > > [ 37.718196] gcc_pcie_3b_aux_clk status stuck at 'off' > > > > [ 37.718205] WARNING: CPU: 4 PID: 482 at drivers/clk/qcom/clk-branch.c:86 clk_branch_wait+0x144/0x15c > > > > > > > > > > Technically this patch is correct. PHYs are backed by MX domain only and not > > > GDSCs. Only the controllers (PCIe, UFS, USB) are backed by GDSCs. The fact that > > > you are seeing issue with PCIe Aux clock suggests me that this clock may not be > > > applicable to the PHY but it needs to be enabled for working of the PHY somehow. > > > I'll try to find the details on how exactly it is needed. > > > > > > But if I get the answer like, "This clock is also sourced to PHY directly", then > > > we may need to add dual power domain for PHY (both GDSC and MX). > > > > > > > So I answer I got from Qcom is that this clock is only applicable to the PCIe > > controller and not PHYs. On some platforms, there is a separate PCIE_PHY_AUX_CLK > > coming from GCC that is used during L1SS state. I think that caused confusion > > while adding PHY support for followup platforms and folks just used PCIE_AUX_CLK > > since they couldn't find the actual PCIE_PHY_AUX_CLK. > > Thanks for sorting that out. > > > I've prepared a series to fix this mess, but I want to know how you end up > > seeing the above "clk status stuck at off" issue. Is there an actual usecase for > > powering up PHY without controller or you just experimented with it? > > As I mentioned, I ran into this when experimenting with how to enable > the "companion" PHY for four-lane support. There shouldn't be any use > case for it (apart from using it to determine that the current > description of the PHY resources is incomplete or incorrect). > Ok. I tested by enabling the PHY clocks during qmp_pcie_clk_init() without PCIE_GDSC. It worked for one instance of the PHY which doesn't have PCIE_PHY_AUX_CLK, but for the PHY instance with this clock, I saw the same "clk stuck" issue. Then checking the internal documentation revealed that this clock needs PCIE_GDSC to become functional >.< So to conclude, PCIE_AUX_CLK belongs to the controller and it needs GDSC. And PCIE_PHY_AUX_CLK belongs to the PHY and it also needs GDSC. I will just submit a series to remove the PCIE_AUX_CLK from PHY nodes. Then in another series, I'll remove the GDSC for PHY instances that do not require PCIE_PHY_AUX_CLK. Hope this makes sense. - Mani
On Tue, Jan 23, 2024 at 10:36:14PM +0530, Manivannan Sadhasivam wrote: > On Mon, Jan 22, 2024 at 06:36:51PM +0100, Johan Hovold wrote: > Ok. I tested by enabling the PHY clocks during qmp_pcie_clk_init() without > PCIE_GDSC. It worked for one instance of the PHY which doesn't have > PCIE_PHY_AUX_CLK, but for the PHY instance with this clock, I saw the same "clk > stuck" issue. Then checking the internal documentation revealed that this clock > needs PCIE_GDSC to become functional >.< > > So to conclude, PCIE_AUX_CLK belongs to the controller and it needs GDSC. And > PCIE_PHY_AUX_CLK belongs to the PHY and it also needs GDSC. > > I will just submit a series to remove the PCIE_AUX_CLK from PHY nodes. Then > in another series, I'll remove the GDSC for PHY instances that do not require > PCIE_PHY_AUX_CLK. Sounds good, thanks. Johan
diff --git a/arch/arm64/boot/dts/qcom/sc8280xp.dtsi b/arch/arm64/boot/dts/qcom/sc8280xp.dtsi index febf28356ff8..72c5818b67f2 100644 --- a/arch/arm64/boot/dts/qcom/sc8280xp.dtsi +++ b/arch/arm64/boot/dts/qcom/sc8280xp.dtsi @@ -1797,7 +1797,7 @@ pcie4_phy: phy@1c06000 { assigned-clocks = <&gcc GCC_PCIE4_PHY_RCHNG_CLK>; assigned-clock-rates = <100000000>; - power-domains = <&gcc PCIE_4_GDSC>; + power-domains = <&rpmhpd SC8280XP_MX>; resets = <&gcc GCC_PCIE_4_PHY_BCR>; reset-names = "phy"; @@ -1895,7 +1895,7 @@ pcie3b_phy: phy@1c0e000 { assigned-clocks = <&gcc GCC_PCIE3B_PHY_RCHNG_CLK>; assigned-clock-rates = <100000000>; - power-domains = <&gcc PCIE_3B_GDSC>; + power-domains = <&rpmhpd SC8280XP_MX>; resets = <&gcc GCC_PCIE_3B_PHY_BCR>; reset-names = "phy"; @@ -1994,7 +1994,7 @@ pcie3a_phy: phy@1c14000 { assigned-clocks = <&gcc GCC_PCIE3A_PHY_RCHNG_CLK>; assigned-clock-rates = <100000000>; - power-domains = <&gcc PCIE_3A_GDSC>; + power-domains = <&rpmhpd SC8280XP_MX>; resets = <&gcc GCC_PCIE_3A_PHY_BCR>; reset-names = "phy"; @@ -2094,7 +2094,7 @@ pcie2b_phy: phy@1c1e000 { assigned-clocks = <&gcc GCC_PCIE2B_PHY_RCHNG_CLK>; assigned-clock-rates = <100000000>; - power-domains = <&gcc PCIE_2B_GDSC>; + power-domains = <&rpmhpd SC8280XP_MX>; resets = <&gcc GCC_PCIE_2B_PHY_BCR>; reset-names = "phy"; @@ -2193,7 +2193,7 @@ pcie2a_phy: phy@1c24000 { assigned-clocks = <&gcc GCC_PCIE2A_PHY_RCHNG_CLK>; assigned-clock-rates = <100000000>; - power-domains = <&gcc PCIE_2A_GDSC>; + power-domains = <&rpmhpd SC8280XP_MX>; resets = <&gcc GCC_PCIE_2A_PHY_BCR>; reset-names = "phy";
The PCIe GDSCs are only related to the RCs. The PCIe PHYs on the other hand, are powered by VDD_MX and their specific VDDA_PHY/PLL regulators. Fix the power-domains assignment to stop potentially toggling the GDSC unnecessarily. Fixes: 813e83157001 ("arm64: dts: qcom: sc8280xp/sa8540p: add PCIe2-4 nodes") Signed-off-by: Konrad Dybcio <konrad.dybcio@linaro.org> --- arch/arm64/boot/dts/qcom/sc8280xp.dtsi | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-)