mbox series

[0/4] Qualcomm PCIe RC shutdown & reinit

Message ID 20231227-topic-8280_pcie-v1-0-095491baf9e4@linaro.org
Headers show
Series Qualcomm PCIe RC shutdown & reinit | expand

Message

Konrad Dybcio Dec. 27, 2023, 10:17 p.m. UTC
This series implements shutdown & reinitialization of the PCIe RC on
system suspend. Tested on 8280-crd.

Signed-off-by: Konrad Dybcio <konrad.dybcio@linaro.org>
---
Konrad Dybcio (4):
      PCI: qcom: Reshuffle reset logic in 2_7_0 .init
      PCI: qcom: Cache last icc bandwidth
      PCI: qcom: Read back PARF_LTSSM register
      PCI: qcom: Implement RC shutdown/power up

 drivers/pci/controller/dwc/Kconfig     |   1 +
 drivers/pci/controller/dwc/pcie-qcom.c | 159 ++++++++++++++++++++++++---------
 2 files changed, 120 insertions(+), 40 deletions(-)
---
base-commit: 39676dfe52331dba909c617f213fdb21015c8d10
change-id: 20231227-topic-8280_pcie-811c0f92fc1c

Best regards,

Comments

Johan Hovold Dec. 29, 2023, 2:04 p.m. UTC | #1
On Wed, Dec 27, 2023 at 11:17:19PM +0100, Konrad Dybcio wrote:
> At least on SC8280XP, if the PCIe reset is asserted, the corresponding
> AUX_CLK will be stuck at 'off'.

No, this path is exercised on every boot without the aux clock ever
being stuck at off. So something is clearly missing in this description.
 
> Assert the reset (which may end up being a NOP if it was previously
> asserted) and de-assert it back *before* turning on the clocks to avoid
> such cases.
> 
> In addition to that, in case the clock bulk enable fails, assert the
> RC reset back, as the hardware is in an unknown state at best.

This is arguably a separate change, and not necessarily one that is
correct either, so should at least go in a separate patch if it should
be done at all.

> Fixes: ed8cc3b1fc84 ("PCI: qcom: Add support for SDM845 PCIe controller")

I think you're being way to liberal with your use of Fixes tags. To
claim that this is a bug, you need to make a more convincing case for
why you think so.

Also note Qualcomm's vendor driver is similarly asserting reset after
enabling the clocks.

That driver does not seem to reset the controller on resume, though, in
case that is relevant for your current experiments.

Johan
Konrad Dybcio Dec. 29, 2023, 3:01 p.m. UTC | #2
On 29.12.2023 15:04, Johan Hovold wrote:
> On Wed, Dec 27, 2023 at 11:17:19PM +0100, Konrad Dybcio wrote:
>> At least on SC8280XP, if the PCIe reset is asserted, the corresponding
>> AUX_CLK will be stuck at 'off'.
> 
> No, this path is exercised on every boot without the aux clock ever
> being stuck at off. So something is clearly missing in this description.
That's likely because the hardware has been initialized and not cleanly
shut down by your bootloader. When you reset it, or your bootloader
wasn't so kind, you need to start initialization from scratch.

>> Assert the reset (which may end up being a NOP if it was previously
>> asserted) and de-assert it back *before* turning on the clocks to avoid
>> such cases.
>>
>> In addition to that, in case the clock bulk enable fails, assert the
>> RC reset back, as the hardware is in an unknown state at best.
> 
> This is arguably a separate change, and not necessarily one that is
> correct either
If the clock enable fails, the PCIe hw is not in reset state, ergo it
may be doing "something", and that "something" would eat non-zero power.
It's just cleaning up after yourself.

> so should at least go in a separate patch if it should
> be done at all.
I'll grumpily comply..

>> Fixes: ed8cc3b1fc84 ("PCI: qcom: Add support for SDM845 PCIe controller")
> 
> I think you're being way to liberal with your use of Fixes tags. To
> claim that this is a bug, you need to make a more convincing case for
> why you think so.
The first paragraph describes the issue that this patch fixes.

> Also note Qualcomm's vendor driver is similarly asserting reset after
> enabling the clocks.
It's also not asserting the reset on suspend, see below.

> That driver does not seem to reset the controller on resume, though, in
> case that is relevant for your current experiments.
I know, the vendor driver doesn't fully shut down the controller. This
is however the only sequence that we (partially) have upstream, and the
only one that is going to work on SC8280XP (due to hw design).

On other platforms, a "soft shutdown" (i.e. dropping the link, cutting
clocks but not fully resetting the RC state) should be possible, but
that's not what this patchset concerns.

Konrad
Johan Hovold Dec. 29, 2023, 3:29 p.m. UTC | #3
[ Again, please remember to add a newline before you inline comments to
make you replies readable. ]

On Fri, Dec 29, 2023 at 04:01:27PM +0100, Konrad Dybcio wrote:
> On 29.12.2023 15:04, Johan Hovold wrote:
> > On Wed, Dec 27, 2023 at 11:17:19PM +0100, Konrad Dybcio wrote:
> >> At least on SC8280XP, if the PCIe reset is asserted, the corresponding
> >> AUX_CLK will be stuck at 'off'.
> > 
> > No, this path is exercised on every boot without the aux clock ever
> > being stuck at off. So something is clearly missing in this description.

> That's likely because the hardware has been initialized and not cleanly
> shut down by your bootloader. When you reset it, or your bootloader
> wasn't so kind, you need to start initialization from scratch.

What does that even mean? I'm telling you that this reset is asserted on
each boot, on all sc8280xp platforms I have access to, and never have I
seen the aux clk stuck at off.

So clearly your claim above is too broad and the commit message is
incorrect or incomplete.
 
> >> Assert the reset (which may end up being a NOP if it was previously
> >> asserted) and de-assert it back *before* turning on the clocks to avoid
> >> such cases.
> >>
> >> In addition to that, in case the clock bulk enable fails, assert the
> >> RC reset back, as the hardware is in an unknown state at best.
> > 
> > This is arguably a separate change, and not necessarily one that is
> > correct either

> If the clock enable fails, the PCIe hw is not in reset state, ergo it
> may be doing "something", and that "something" would eat non-zero power.
> It's just cleaning up after yourself.

How can it do something without power and clocks? And leaving reset
asserted for non-powered devices is generally not a good idea.
 
> > so should at least go in a separate patch if it should
> > be done at all.

> I'll grumpily comply..

I suggest you leave it deasserted unless you have documentation
suggesting that the opposite is safe and recommended for this piece of
hardware.
 
> >> Fixes: ed8cc3b1fc84 ("PCI: qcom: Add support for SDM845 PCIe controller")
> > 
> > I think you're being way to liberal with your use of Fixes tags. To
> > claim that this is a bug, you need to make a more convincing case for
> > why you think so.

> The first paragraph describes the issue that this patch fixes.

Yes, but this is all very hand-wavy so far. With a complete commit
message I may agree, but you still haven't convinced me that this is a
bug and not just a workaround from some not fully-understood issue on
one particular platform.
 
> > Also note Qualcomm's vendor driver is similarly asserting reset after
> > enabling the clocks.

> It's also not asserting the reset on suspend, see below.

Right, as I mentioned.
 
> > That driver does not seem to reset the controller on resume, though, in
> > case that is relevant for your current experiments.

> I know, the vendor driver doesn't fully shut down the controller. This
> is however the only sequence that we (partially) have upstream, and the
> only one that is going to work on SC8280XP (due to hw design).
> 
> On other platforms, a "soft shutdown" (i.e. dropping the link, cutting
> clocks but not fully resetting the RC state) should be possible, but
> that's not what this patchset concerns.

The commit message does not even mention suspend, it just makes a
clearly false general claim about a clock being stuck unless you reorder
things.

Johan
Konrad Dybcio Dec. 30, 2023, 1:16 a.m. UTC | #4
On 29.12.2023 16:29, Johan Hovold wrote:
> [ Again, please remember to add a newline before you inline comments to
> make you replies readable. ]
> 
> On Fri, Dec 29, 2023 at 04:01:27PM +0100, Konrad Dybcio wrote:
>> On 29.12.2023 15:04, Johan Hovold wrote:
>>> On Wed, Dec 27, 2023 at 11:17:19PM +0100, Konrad Dybcio wrote:
>>>> At least on SC8280XP, if the PCIe reset is asserted, the corresponding
>>>> AUX_CLK will be stuck at 'off'.
>>>
>>> No, this path is exercised on every boot without the aux clock ever
>>> being stuck at off. So something is clearly missing in this description.
> 
>> That's likely because the hardware has been initialized and not cleanly
>> shut down by your bootloader. When you reset it, or your bootloader
>> wasn't so kind, you need to start initialization from scratch.
> 
> What does that even mean? I'm telling you that this reset is asserted on
> each boot, on all sc8280xp platforms I have access to, and never have I
> seen the aux clk stuck at off.
> 
> So clearly your claim above is too broad and the commit message is
> incorrect or incomplete.

diff --git a/drivers/clk/qcom/gcc-sc8280xp.c b/drivers/clk/qcom/gcc-sc8280xp.c
index 0b7801971dc1..6650bd6af5e3 100644
--- a/drivers/clk/qcom/gcc-sc8280xp.c
+++ b/drivers/clk/qcom/gcc-sc8280xp.c
@@ -7566,6 +7566,18 @@ static int gcc_sc8280xp_probe(struct platform_device *pdev)
        if (ret)
                goto err_put_rpm;
 
+       int val;
+       regmap_read(regmap, 0xa0000, &val);
+       pr_err("GCC_PCIE_3A_BCR = 0x%x\n", val);
+       regmap_read(regmap, 0xa00f0, &val);
+       pr_err("GCC_PCIE_3A_LINK_DOWN_BCR = 0x%x\n", val);
+       regmap_read(regmap, 0xa00fc, &val);
+       pr_err("GCC_PCIE_3A_NOCSR_COM_PHY_BCR = 0x%x\n", val);
+       regmap_read(regmap, 0xa00e0, &val);
+       pr_err("GCC_PCIE_3A_PHY_BCR = 0x%x\n", val);
+       regmap_read(regmap, 0xa00e4, &val);
+       pr_err("GCC_PCIE_3A_PHY_NOCSR_COM_PHY_BCR = 0x%x\n", val);
+
        pm_runtime_put(&pdev->dev);
 
        return 0;


[root@sc8280xp-crd ~]# dmesg | grep BCR
[    2.500245] GCC_PCIE_3A_BCR = 0x0
[    2.500250] GCC_PCIE_3A_LINK_DOWN_BCR = 0x0
[    2.500253] GCC_PCIE_3A_NOCSR_COM_PHY_BCR = 0x0
[    2.500255] GCC_PCIE_3A_PHY_BCR = 0x0
[    2.500257] GCC_PCIE_3A_PHY_NOCSR_COM_PHY_BCR = 0x0


0 meaning "not asserted".

Adding the read in the GCC driver .probe ensures we get the
unmodified data, as all GCC consumers must wait for it to probe. 

PCIE3A is used for WLAN on the CRD, btw.

>  
>>>> Assert the reset (which may end up being a NOP if it was previously
>>>> asserted) and de-assert it back *before* turning on the clocks to avoid
>>>> such cases.
>>>>
>>>> In addition to that, in case the clock bulk enable fails, assert the
>>>> RC reset back, as the hardware is in an unknown state at best.
>>>
>>> This is arguably a separate change, and not necessarily one that is
>>> correct either
> 
>> If the clock enable fails, the PCIe hw is not in reset state, ergo it
>> may be doing "something", and that "something" would eat non-zero power.
>> It's just cleaning up after yourself.
> 
> How can it do something without power and clocks?

Fair point.

As far as power goes, the RC hangs off CX, which is on whenever the
system is not in power collapse. As for clocks, at least parts of it
use the crystal oscillator, not sure if directly.

> And leaving reset
> asserted for non-powered devices is generally not a good idea.

Depends on the hw.

>  
>>> so should at least go in a separate patch if it should
>>> be done at all.
> 
>> I'll grumpily comply..
> 
> I suggest you leave it deasserted unless you have documentation
> suggesting that the opposite is safe and recommended for this piece of
> hardware.
>  
>>>> Fixes: ed8cc3b1fc84 ("PCI: qcom: Add support for SDM845 PCIe controller")
>>>
>>> I think you're being way to liberal with your use of Fixes tags. To
>>> claim that this is a bug, you need to make a more convincing case for
>>> why you think so.
> 
>> The first paragraph describes the issue that this patch fixes.
> 
> Yes, but this is all very hand-wavy so far. With a complete commit
> message I may agree, but you still haven't convinced me that this is a
> bug and not just a workaround from some not fully-understood issue on
> one particular platform.

Right, reading it again, it doesn't really tell the whole story.

>  
>>> Also note Qualcomm's vendor driver is similarly asserting reset after
>>> enabling the clocks.
> 
>> It's also not asserting the reset on suspend, see below.
> 
> Right, as I mentioned.
>  
>>> That driver does not seem to reset the controller on resume, though, in
>>> case that is relevant for your current experiments.
> 
>> I know, the vendor driver doesn't fully shut down the controller. This
>> is however the only sequence that we (partially) have upstream, and the
>> only one that is going to work on SC8280XP (due to hw design).
>>
>> On other platforms, a "soft shutdown" (i.e. dropping the link, cutting
>> clocks but not fully resetting the RC state) should be possible, but
>> that's not what this patchset concerns.
> 
> The commit message does not even mention suspend, it just makes a
> clearly false general claim about a clock being stuck unless you reorder
> things.

No, I insist that this general statement, while indeed lacking a full
description of the problem, is provably true. The AUX clock will not
turn on if the PCIe reset is asserted, at least on SC8280XP.

Konrad
Manivannan Sadhasivam Jan. 1, 2024, 5:31 p.m. UTC | #5
On Sat, Dec 30, 2023 at 02:16:18AM +0100, Konrad Dybcio wrote:
> On 29.12.2023 16:29, Johan Hovold wrote:
> > [ Again, please remember to add a newline before you inline comments to
> > make you replies readable. ]
> > 
> > On Fri, Dec 29, 2023 at 04:01:27PM +0100, Konrad Dybcio wrote:
> >> On 29.12.2023 15:04, Johan Hovold wrote:
> >>> On Wed, Dec 27, 2023 at 11:17:19PM +0100, Konrad Dybcio wrote:
> >>>> At least on SC8280XP, if the PCIe reset is asserted, the corresponding
> >>>> AUX_CLK will be stuck at 'off'.
> >>>
> >>> No, this path is exercised on every boot without the aux clock ever
> >>> being stuck at off. So something is clearly missing in this description.
> > 
> >> That's likely because the hardware has been initialized and not cleanly
> >> shut down by your bootloader. When you reset it, or your bootloader
> >> wasn't so kind, you need to start initialization from scratch.
> > 
> > What does that even mean? I'm telling you that this reset is asserted on
> > each boot, on all sc8280xp platforms I have access to, and never have I
> > seen the aux clk stuck at off.
> > 
> > So clearly your claim above is too broad and the commit message is
> > incorrect or incomplete.
> 
> diff --git a/drivers/clk/qcom/gcc-sc8280xp.c b/drivers/clk/qcom/gcc-sc8280xp.c
> index 0b7801971dc1..6650bd6af5e3 100644
> --- a/drivers/clk/qcom/gcc-sc8280xp.c
> +++ b/drivers/clk/qcom/gcc-sc8280xp.c
> @@ -7566,6 +7566,18 @@ static int gcc_sc8280xp_probe(struct platform_device *pdev)
>         if (ret)
>                 goto err_put_rpm;
>  
> +       int val;
> +       regmap_read(regmap, 0xa0000, &val);
> +       pr_err("GCC_PCIE_3A_BCR = 0x%x\n", val);
> +       regmap_read(regmap, 0xa00f0, &val);
> +       pr_err("GCC_PCIE_3A_LINK_DOWN_BCR = 0x%x\n", val);
> +       regmap_read(regmap, 0xa00fc, &val);
> +       pr_err("GCC_PCIE_3A_NOCSR_COM_PHY_BCR = 0x%x\n", val);
> +       regmap_read(regmap, 0xa00e0, &val);
> +       pr_err("GCC_PCIE_3A_PHY_BCR = 0x%x\n", val);
> +       regmap_read(regmap, 0xa00e4, &val);
> +       pr_err("GCC_PCIE_3A_PHY_NOCSR_COM_PHY_BCR = 0x%x\n", val);
> +
>         pm_runtime_put(&pdev->dev);
>  
>         return 0;
> 
> 
> [root@sc8280xp-crd ~]# dmesg | grep BCR
> [    2.500245] GCC_PCIE_3A_BCR = 0x0
> [    2.500250] GCC_PCIE_3A_LINK_DOWN_BCR = 0x0
> [    2.500253] GCC_PCIE_3A_NOCSR_COM_PHY_BCR = 0x0
> [    2.500255] GCC_PCIE_3A_PHY_BCR = 0x0
> [    2.500257] GCC_PCIE_3A_PHY_NOCSR_COM_PHY_BCR = 0x0
> 
> 
> 0 meaning "not asserted".
> 
> Adding the read in the GCC driver .probe ensures we get the
> unmodified data, as all GCC consumers must wait for it to probe. 
> 
> PCIE3A is used for WLAN on the CRD, btw.
> 

I get what you are trying to do, but I should say that your justification so far
didn't do the justice.

Your point is that, if the PCIe BCR is asserted during boot (which can happen if
the bootloader didn't initialize PCIe for things like storage), then trying to
enable clocks will result in the "clk stuck" error from GCC. But in the case of
sc8280xp, bootloader would've already initialized PCIe during boot as it is uses
NVMe for things like firmware. But when you do a full power down during suspend
(CX power collapse), then while reinitializing the controller during resume, you
are hitting the "clk stuck" error because at that time, the BCR reset would be
asserted (POR value).

But I really suspect if that is the case... Because, the same init function is
being used by other SoCs (sm8150, sm8250, etc... even sdx55) and I'm pretty sure
that in those SoCs, the bootloader wouldn't have initialized PCIe during boot as
there is no use case.

So I cross checked it on SM8450, but I saw the BCR status being "0" during boot
(same as on sc8280xp). Then I checked the HPG and came to know that when the
PCIe GDSC is uncollapsed, some of the BCRs would be deasserted in the back.
Though it mentioned only PHY_BCR and not the PCIE_n_BCR. But I think the
behavior might be same for both. You can verify it by printing the state of all
BCRs during resume from suspend. This will give us some clue...

> >  
> >>>> Assert the reset (which may end up being a NOP if it was previously
> >>>> asserted) and de-assert it back *before* turning on the clocks to avoid
> >>>> such cases.
> >>>>
> >>>> In addition to that, in case the clock bulk enable fails, assert the
> >>>> RC reset back, as the hardware is in an unknown state at best.
> >>>
> >>> This is arguably a separate change, and not necessarily one that is
> >>> correct either
> > 
> >> If the clock enable fails, the PCIe hw is not in reset state, ergo it
> >> may be doing "something", and that "something" would eat non-zero power.
> >> It's just cleaning up after yourself.
> > 
> > How can it do something without power and clocks?
> 
> Fair point.
> 
> As far as power goes, the RC hangs off CX, which is on whenever the
> system is not in power collapse. As for clocks, at least parts of it
> use the crystal oscillator, not sure if directly.
> 
> > And leaving reset
> > asserted for non-powered devices is generally not a good idea.
> 
> Depends on the hw.
> 

I do not have any strong argument here as there are too many things happening
that determines whether the controller is properly powered or not. So I'd say
that if you do not have any power numbers, it is best to leave it as it is.

- Mani

> >  
> >>> so should at least go in a separate patch if it should
> >>> be done at all.
> > 
> >> I'll grumpily comply..
> > 
> > I suggest you leave it deasserted unless you have documentation
> > suggesting that the opposite is safe and recommended for this piece of
> > hardware.
> >  
> >>>> Fixes: ed8cc3b1fc84 ("PCI: qcom: Add support for SDM845 PCIe controller")
> >>>
> >>> I think you're being way to liberal with your use of Fixes tags. To
> >>> claim that this is a bug, you need to make a more convincing case for
> >>> why you think so.
> > 
> >> The first paragraph describes the issue that this patch fixes.
> > 
> > Yes, but this is all very hand-wavy so far. With a complete commit
> > message I may agree, but you still haven't convinced me that this is a
> > bug and not just a workaround from some not fully-understood issue on
> > one particular platform.
> 
> Right, reading it again, it doesn't really tell the whole story.
> 
> >  
> >>> Also note Qualcomm's vendor driver is similarly asserting reset after
> >>> enabling the clocks.
> > 
> >> It's also not asserting the reset on suspend, see below.
> > 
> > Right, as I mentioned.
> >  
> >>> That driver does not seem to reset the controller on resume, though, in
> >>> case that is relevant for your current experiments.
> > 
> >> I know, the vendor driver doesn't fully shut down the controller. This
> >> is however the only sequence that we (partially) have upstream, and the
> >> only one that is going to work on SC8280XP (due to hw design).
> >>
> >> On other platforms, a "soft shutdown" (i.e. dropping the link, cutting
> >> clocks but not fully resetting the RC state) should be possible, but
> >> that's not what this patchset concerns.
> > 
> > The commit message does not even mention suspend, it just makes a
> > clearly false general claim about a clock being stuck unless you reorder
> > things.
> 
> No, I insist that this general statement, while indeed lacking a full
> description of the problem, is provably true. The AUX clock will not
> turn on if the PCIe reset is asserted, at least on SC8280XP.
> 
> Konrad
Johan Hovold Jan. 2, 2024, 10:17 a.m. UTC | #6
On Sat, Dec 30, 2023 at 02:16:18AM +0100, Konrad Dybcio wrote:
> On 29.12.2023 16:29, Johan Hovold wrote:
> > [ Again, please remember to add a newline before you inline comments to
> > make you replies readable. ]
> > 
> > On Fri, Dec 29, 2023 at 04:01:27PM +0100, Konrad Dybcio wrote:
> >> On 29.12.2023 15:04, Johan Hovold wrote:
> >>> On Wed, Dec 27, 2023 at 11:17:19PM +0100, Konrad Dybcio wrote:
> >>>> At least on SC8280XP, if the PCIe reset is asserted, the corresponding
> >>>> AUX_CLK will be stuck at 'off'.
> >>>
> >>> No, this path is exercised on every boot without the aux clock ever
> >>> being stuck at off. So something is clearly missing in this description.
> > 
> >> That's likely because the hardware has been initialized and not cleanly
> >> shut down by your bootloader. When you reset it, or your bootloader
> >> wasn't so kind, you need to start initialization from scratch.
> > 
> > What does that even mean? I'm telling you that this reset is asserted on
> > each boot, on all sc8280xp platforms I have access to, and never have I
> > seen the aux clk stuck at off.
> > 
> > So clearly your claim above is too broad and the commit message is
> > incorrect or incomplete.
> 
> diff --git a/drivers/clk/qcom/gcc-sc8280xp.c b/drivers/clk/qcom/gcc-sc8280xp.c
> index 0b7801971dc1..6650bd6af5e3 100644
> --- a/drivers/clk/qcom/gcc-sc8280xp.c
> +++ b/drivers/clk/qcom/gcc-sc8280xp.c
> @@ -7566,6 +7566,18 @@ static int gcc_sc8280xp_probe(struct platform_device *pdev)
>         if (ret)
>                 goto err_put_rpm;
>  
> +       int val;
> +       regmap_read(regmap, 0xa0000, &val);
> +       pr_err("GCC_PCIE_3A_BCR = 0x%x\n", val);
> +       regmap_read(regmap, 0xa00f0, &val);
> +       pr_err("GCC_PCIE_3A_LINK_DOWN_BCR = 0x%x\n", val);
> +       regmap_read(regmap, 0xa00fc, &val);
> +       pr_err("GCC_PCIE_3A_NOCSR_COM_PHY_BCR = 0x%x\n", val);
> +       regmap_read(regmap, 0xa00e0, &val);
> +       pr_err("GCC_PCIE_3A_PHY_BCR = 0x%x\n", val);
> +       regmap_read(regmap, 0xa00e4, &val);
> +       pr_err("GCC_PCIE_3A_PHY_NOCSR_COM_PHY_BCR = 0x%x\n", val);
> +
>         pm_runtime_put(&pdev->dev);
>  
>         return 0;
> 
> 
> [root@sc8280xp-crd ~]# dmesg | grep BCR
> [    2.500245] GCC_PCIE_3A_BCR = 0x0
> [    2.500250] GCC_PCIE_3A_LINK_DOWN_BCR = 0x0
> [    2.500253] GCC_PCIE_3A_NOCSR_COM_PHY_BCR = 0x0
> [    2.500255] GCC_PCIE_3A_PHY_BCR = 0x0
> [    2.500257] GCC_PCIE_3A_PHY_NOCSR_COM_PHY_BCR = 0x0
> 
> 
> 0 meaning "not asserted".

We're clearly talking past each other. When I'm saying reset is asserted
on each boot, I'm referring to reset being asserted in
qcom_pcie_init_2_7_0(), whereas you appear to be referring to whether
the reset has been left asserted by the bootloader when the driver
probes.

I understand what you meant to say now, but I think you should rephrase:

	At least on SC8280XP, if the PCIe reset is asserted, the
	corresponding AUX_CLK will be stuck at 'off'.

because as it stands, it sounds like the problem happens when the driver
asserts reset.

> PCIE3A is used for WLAN on the CRD, btw.

You meant to say WWAN (modem).

> >>>> Assert the reset (which may end up being a NOP if it was previously
> >>>> asserted) and de-assert it back *before* turning on the clocks to avoid
> >>>> such cases.
> >>>>
> >>>> In addition to that, in case the clock bulk enable fails, assert the
> >>>> RC reset back, as the hardware is in an unknown state at best.
> >>>
> >>> This is arguably a separate change, and not necessarily one that is
> >>> correct either
> > 
> >> If the clock enable fails, the PCIe hw is not in reset state, ergo it
> >> may be doing "something", and that "something" would eat non-zero power.
> >> It's just cleaning up after yourself.
> > 
> > How can it do something without power and clocks?
> 
> Fair point.
> 
> As far as power goes, the RC hangs off CX, which is on whenever the
> system is not in power collapse. As for clocks, at least parts of it
> use the crystal oscillator, not sure if directly.
> 
> > And leaving reset
> > asserted for non-powered devices is generally not a good idea.
> 
> Depends on the hw.

That's why I said "generally".

> > The commit message does not even mention suspend, it just makes a
> > clearly false general claim about a clock being stuck unless you reorder
> > things.
> 
> No, I insist that this general statement, while indeed lacking a full
> description of the problem, is provably true. The AUX clock will not
> turn on if the PCIe reset is asserted, at least on SC8280XP.

I agree, I was misinterpreting the commit message.

Johan
Konrad Dybcio Jan. 2, 2024, 5:03 p.m. UTC | #7
On 2.01.2024 11:17, Johan Hovold wrote:
> On Sat, Dec 30, 2023 at 02:16:18AM +0100, Konrad Dybcio wrote:
>> On 29.12.2023 16:29, Johan Hovold wrote:
>>> [ Again, please remember to add a newline before you inline comments to
>>> make you replies readable. ]
>>>
>>> On Fri, Dec 29, 2023 at 04:01:27PM +0100, Konrad Dybcio wrote:
>>>> On 29.12.2023 15:04, Johan Hovold wrote:
>>>>> On Wed, Dec 27, 2023 at 11:17:19PM +0100, Konrad Dybcio wrote:
>>>>>> At least on SC8280XP, if the PCIe reset is asserted, the corresponding
>>>>>> AUX_CLK will be stuck at 'off'.
>>>>>
>>>>> No, this path is exercised on every boot without the aux clock ever
>>>>> being stuck at off. So something is clearly missing in this description.
>>>
>>>> That's likely because the hardware has been initialized and not cleanly
>>>> shut down by your bootloader. When you reset it, or your bootloader
>>>> wasn't so kind, you need to start initialization from scratch.
>>>
>>> What does that even mean? I'm telling you that this reset is asserted on
>>> each boot, on all sc8280xp platforms I have access to, and never have I
>>> seen the aux clk stuck at off.
>>>
>>> So clearly your claim above is too broad and the commit message is
>>> incorrect or incomplete.
>>
>> diff --git a/drivers/clk/qcom/gcc-sc8280xp.c b/drivers/clk/qcom/gcc-sc8280xp.c
>> index 0b7801971dc1..6650bd6af5e3 100644
>> --- a/drivers/clk/qcom/gcc-sc8280xp.c
>> +++ b/drivers/clk/qcom/gcc-sc8280xp.c
>> @@ -7566,6 +7566,18 @@ static int gcc_sc8280xp_probe(struct platform_device *pdev)
>>         if (ret)
>>                 goto err_put_rpm;
>>  
>> +       int val;
>> +       regmap_read(regmap, 0xa0000, &val);
>> +       pr_err("GCC_PCIE_3A_BCR = 0x%x\n", val);
>> +       regmap_read(regmap, 0xa00f0, &val);
>> +       pr_err("GCC_PCIE_3A_LINK_DOWN_BCR = 0x%x\n", val);
>> +       regmap_read(regmap, 0xa00fc, &val);
>> +       pr_err("GCC_PCIE_3A_NOCSR_COM_PHY_BCR = 0x%x\n", val);
>> +       regmap_read(regmap, 0xa00e0, &val);
>> +       pr_err("GCC_PCIE_3A_PHY_BCR = 0x%x\n", val);
>> +       regmap_read(regmap, 0xa00e4, &val);
>> +       pr_err("GCC_PCIE_3A_PHY_NOCSR_COM_PHY_BCR = 0x%x\n", val);
>> +
>>         pm_runtime_put(&pdev->dev);
>>  
>>         return 0;
>>
>>
>> [root@sc8280xp-crd ~]# dmesg | grep BCR
>> [    2.500245] GCC_PCIE_3A_BCR = 0x0
>> [    2.500250] GCC_PCIE_3A_LINK_DOWN_BCR = 0x0
>> [    2.500253] GCC_PCIE_3A_NOCSR_COM_PHY_BCR = 0x0
>> [    2.500255] GCC_PCIE_3A_PHY_BCR = 0x0
>> [    2.500257] GCC_PCIE_3A_PHY_NOCSR_COM_PHY_BCR = 0x0
>>
>>
>> 0 meaning "not asserted".
> 
> We're clearly talking past each other. When I'm saying reset is asserted
> on each boot, I'm referring to reset being asserted in
> qcom_pcie_init_2_7_0(), whereas you appear to be referring to whether
> the reset has been left asserted by the bootloader when the driver
> probes.

OK, "boot" meant "booting the device" to me, not the PCIe controller.

> 
> I understand what you meant to say now, but I think you should rephrase:
> 
> 	At least on SC8280XP, if the PCIe reset is asserted, the
> 	corresponding AUX_CLK will be stuck at 'off'.
> 
> because as it stands, it sounds like the problem happens when the driver
> asserts reset.

Does this sound good?

"At least on SC8280XP, trying to enable the AUX_CLK associated with
a PCIe host fails if the corresponding PCIe reset is asserted."

> 
>> PCIE3A is used for WLAN on the CRD, btw.
> 
> You meant to say WWAN (modem).

Right :)

> 
>>>>>> Assert the reset (which may end up being a NOP if it was previously
>>>>>> asserted) and de-assert it back *before* turning on the clocks to avoid
>>>>>> such cases.
>>>>>>
>>>>>> In addition to that, in case the clock bulk enable fails, assert the
>>>>>> RC reset back, as the hardware is in an unknown state at best.
>>>>>
>>>>> This is arguably a separate change, and not necessarily one that is
>>>>> correct either
>>>
>>>> If the clock enable fails, the PCIe hw is not in reset state, ergo it
>>>> may be doing "something", and that "something" would eat non-zero power.
>>>> It's just cleaning up after yourself.
>>>
>>> How can it do something without power and clocks?
>>
>> Fair point.
>>
>> As far as power goes, the RC hangs off CX, which is on whenever the
>> system is not in power collapse. As for clocks, at least parts of it
>> use the crystal oscillator, not sure if directly.
>>
>>> And leaving reset
>>> asserted for non-powered devices is generally not a good idea.
>>
>> Depends on the hw.
> 
> That's why I said "generally".

I'll try to get a proper answer for this, or otherwise see if there's
any change in power draw / functionality.

Konrad
Johan Hovold Jan. 3, 2024, 10:40 a.m. UTC | #8
On Tue, Jan 02, 2024 at 06:03:36PM +0100, Konrad Dybcio wrote:
> On 2.01.2024 11:17, Johan Hovold wrote:
> > On Sat, Dec 30, 2023 at 02:16:18AM +0100, Konrad Dybcio wrote:
> >> On 29.12.2023 16:29, Johan Hovold wrote:
> >>> On Fri, Dec 29, 2023 at 04:01:27PM +0100, Konrad Dybcio wrote:
> >>>> On 29.12.2023 15:04, Johan Hovold wrote:
> >>>>> On Wed, Dec 27, 2023 at 11:17:19PM +0100, Konrad Dybcio wrote:
> >>>>>> At least on SC8280XP, if the PCIe reset is asserted, the corresponding
> >>>>>> AUX_CLK will be stuck at 'off'.
> >>>>>
> >>>>> No, this path is exercised on every boot without the aux clock ever
> >>>>> being stuck at off. So something is clearly missing in this description.
> >>>
> >>>> That's likely because the hardware has been initialized and not cleanly
> >>>> shut down by your bootloader. When you reset it, or your bootloader
> >>>> wasn't so kind, you need to start initialization from scratch.
> >>>
> >>> What does that even mean? I'm telling you that this reset is asserted on
> >>> each boot, on all sc8280xp platforms I have access to, and never have I
> >>> seen the aux clk stuck at off.
> >>>
> >>> So clearly your claim above is too broad and the commit message is
> >>> incorrect or incomplete.

> > We're clearly talking past each other. When I'm saying reset is asserted
> > on each boot, I'm referring to reset being asserted in
> > qcom_pcie_init_2_7_0(), whereas you appear to be referring to whether
> > the reset has been left asserted by the bootloader when the driver
> > probes.
> 
> OK, "boot" meant "booting the device" to me, not the PCIe controller.

Still not getting across to you apparently.

Again, the code in question is exercised on every boot and not once have
I seen a stuck clock due to reset being asserted *in*
qcom_pcie_init_2_7_0().

Now that's not what you were trying to describe as you were thinking of
reset having been left asserted *before* the driver probes (or before
qcom_pcie_init_2_7_0() is called).

See? Do you understand now what I was trying to say and why my
misinterpretation of your terse commit message lead me to claim that it
was clearly false?

> > I understand what you meant to say now, but I think you should rephrase:
> > 
> > 	At least on SC8280XP, if the PCIe reset is asserted, the
> > 	corresponding AUX_CLK will be stuck at 'off'.
> > 
> > because as it stands, it sounds like the problem happens when the driver
> > asserts reset.
> 
> Does this sound good?
> 
> "At least on SC8280XP, trying to enable the AUX_CLK associated with
> a PCIe host fails if the corresponding PCIe reset is asserted."

Yes, but you need to also say something about how this would happen, for
example, your hypothetical bootloader leaving it asserted and your actual
motivation for this change which is that it appears to be needed after
suspend. 

A commit message should be clear and self-contained and not force
reviewers to have to try to interpret what it means and guess what the
motivation for the change really is.

Johan
Konrad Dybcio Jan. 3, 2024, 12:11 p.m. UTC | #9
On 3.01.2024 11:40, Johan Hovold wrote:
> On Tue, Jan 02, 2024 at 06:03:36PM +0100, Konrad Dybcio wrote:
>> On 2.01.2024 11:17, Johan Hovold wrote:
>>> On Sat, Dec 30, 2023 at 02:16:18AM +0100, Konrad Dybcio wrote:
>>>> On 29.12.2023 16:29, Johan Hovold wrote:
>>>>> On Fri, Dec 29, 2023 at 04:01:27PM +0100, Konrad Dybcio wrote:
>>>>>> On 29.12.2023 15:04, Johan Hovold wrote:
>>>>>>> On Wed, Dec 27, 2023 at 11:17:19PM +0100, Konrad Dybcio wrote:
>>>>>>>> At least on SC8280XP, if the PCIe reset is asserted, the corresponding
>>>>>>>> AUX_CLK will be stuck at 'off'.
>>>>>>>
>>>>>>> No, this path is exercised on every boot without the aux clock ever
>>>>>>> being stuck at off. So something is clearly missing in this description.
>>>>>
>>>>>> That's likely because the hardware has been initialized and not cleanly
>>>>>> shut down by your bootloader. When you reset it, or your bootloader
>>>>>> wasn't so kind, you need to start initialization from scratch.
>>>>>
>>>>> What does that even mean? I'm telling you that this reset is asserted on
>>>>> each boot, on all sc8280xp platforms I have access to, and never have I
>>>>> seen the aux clk stuck at off.
>>>>>
>>>>> So clearly your claim above is too broad and the commit message is
>>>>> incorrect or incomplete.
> 
>>> We're clearly talking past each other. When I'm saying reset is asserted
>>> on each boot, I'm referring to reset being asserted in
>>> qcom_pcie_init_2_7_0(), whereas you appear to be referring to whether
>>> the reset has been left asserted by the bootloader when the driver
>>> probes.
>>
>> OK, "boot" meant "booting the device" to me, not the PCIe controller.
> 
> Still not getting across to you apparently.
> 
> Again, the code in question is exercised on every boot and not once have
> I seen a stuck clock due to reset being asserted *in*
> qcom_pcie_init_2_7_0().
> 
> Now that's not what you were trying to describe as you were thinking of
> reset having been left asserted *before* the driver probes (or before
> qcom_pcie_init_2_7_0() is called).
> 
> See? Do you understand now what I was trying to say and why my
> misinterpretation of your terse commit message lead me to claim that it
> was clearly false?

No, my response was an acknowledgement of having understood you. Maybe
it's a direct translation of some Polish idiom that's not obvious to
others, but I definitely tried to say that "we were talking about
different things, I had been previously thinking of something else,
but now we're on the same page".


> 
>>> I understand what you meant to say now, but I think you should rephrase:
>>>
>>> 	At least on SC8280XP, if the PCIe reset is asserted, the
>>> 	corresponding AUX_CLK will be stuck at 'off'.
>>>
>>> because as it stands, it sounds like the problem happens when the driver
>>> asserts reset.
>>
>> Does this sound good?
>>
>> "At least on SC8280XP, trying to enable the AUX_CLK associated with
>> a PCIe host fails if the corresponding PCIe reset is asserted."
> 
> Yes, but you need to also say something about how this would happen, for
> example, your hypothetical bootloader leaving it asserted and your actual
> motivation for this change which is that it appears to be needed after
> suspend. 
> 
> A commit message should be clear and self-contained and not force
> reviewers to have to try to interpret what it means and guess what the
> motivation for the change really is.

Got it

Konrad