| Message ID | 20240724022719.2868490-3-quic_pyarlaga@quicinc.com |
|---|---|
| State | New |
| Headers | show |
| Series | PCI: qcom: Avoid DBI and ATU register space mirroring | expand |
Hi Manivannan, Thanks for the review comments. On 7/24/2024 7:10 AM, Manivannan Sadhasivam wrote: > Subject: > > PCI: qcom: Disable mirroring of DBI and iATU register space in BAR/MMIO region > ACK. I will update it in the next patch. > On Tue, Jul 23, 2024 at 07:27:19PM -0700, Prudhvi Yarlagadda wrote: >> PARF hardware block which is a wrapper on top of DWC PCIe controller >> mirrors the DBI and ATU register space. It uses PARF_SLV_ADDR_SPACE_SIZE >> register to get the size of the memory block to be mirrored and uses >> PARF_DBI_BASE_ADDR, PARF_ATU_BASE_ADDR registers to determine the base >> address of DBI and ATU space inside the memory block that is being >> mirrored. >> >> When a memory region which is located above the SLV_ADDR_SPACE_SIZE >> boundary is used for BAR region then there could be an overlap of DBI and >> ATU address space that is getting mirrored and the BAR region. This >> results in DBI and ATU address space contents getting updated when a PCIe >> function driver tries updating the BAR/MMIO memory region. Reference >> memory map of the PCIe memory region with DBI and ATU address space >> overlapping BAR region is as below. >> >> |---------------| >> | | >> | | >> ------- --------|---------------| >> | | |---------------| >> | | | DBI | >> | | |---------------|---->DBI_BASE_ADDR >> | | | | >> | | | | >> | PCIe | |---->2*SLV_ADDR_SPACE_SIZE >> | BAR/MMIO|---------------| >> | Region | ATU | >> | | |---------------|---->ATU_BASE_ADDR >> | | | | >> PCIe | |---------------| >> Memory | | DBI | >> Region | |---------------|---->DBI_BASE_ADDR >> | | | | >> | --------| | >> | | |---->SLV_ADDR_SPACE_SIZE >> | |---------------| >> | | ATU | >> | |---------------|---->ATU_BASE_ADDR >> | | | >> | |---------------| >> | | DBI | >> | |---------------|---->DBI_BASE_ADDR >> | | | >> | | | >> ----------------|---------------| >> | | >> | | >> | | >> |---------------| >> >> Currently memory region beyond the SLV_ADDR_SPACE_SIZE boundary is not >> used for BAR region which is why the above mentioned issue is not >> encountered. This issue is discovered as part of internal testing when we >> tried moving the BAR region beyond the SLV_ADDR_SPACE_SIZE boundary. Hence >> we are trying to fix this. >> >> As PARF hardware block mirrors DBI and ATU register space after every >> PARF_SLV_ADDR_SPACE_SIZE (default 0x1000000) boundary multiple, write >> U32_MAX (all 0xFF's) to PARF_SLV_ADDR_SPACE_SIZE register to avoid >> mirroring DBI and ATU to BAR/MMIO region. Write the physical base address >> of DBI and ATU register blocks to PARF_DBI_BASE_ADDR (default 0x0) and >> PARF_ATU_BASE_ADDR (default 0x1000) respectively to make sure DBI and ATU >> blocks are at expected memory locations. >> >> The register offsets PARF_DBI_BASE_ADDR_V2, PARF_SLV_ADDR_SPACE_SIZE_V2 >> and PARF_ATU_BASE_ADDR are applicable for platforms that use PARF > > There is no 'PARF Qcom IP', just 'Qcom IP'. Here and below. > ACK. I will update it in the next patch. >> Qcom IP rev 1.9.0, 2.7.0 and 2.9.0. PARF_DBI_BASE_ADDR_V2 and >> PARF_SLV_ADDR_SPACE_SIZE_V2 are applicable for PARF Qcom IP rev 2.3.3. >> PARF_DBI_BASE_ADDR and PARF_SLV_ADDR_SPACE_SIZE are applicable for PARF >> Qcom IP rev 1.0.0, 2.3.2 and 2.4.0. Updating the init()/post_init() > > Use imperative tone in commit message. s/Updating/Update > ACK. I will update it in the next patch. >> functions of the respective PARF versions to program applicable >> PARF_DBI_BASE_ADDR, PARF_SLV_ADDR_SPACE_SIZE and PARF_ATU_BASE_ADDR >> register offsets. And remove the unused SLV_ADDR_SPACE_SZ macro. >> >> Signed-off-by: Prudhvi Yarlagadda <quic_pyarlaga@quicinc.com> >> Reviewed-by: Mayank Rana <quic_mrana@quicinc.com> >> --- >> drivers/pci/controller/dwc/pcie-qcom.c | 62 +++++++++++++++++++------- >> 1 file changed, 45 insertions(+), 17 deletions(-) >> >> diff --git a/drivers/pci/controller/dwc/pcie-qcom.c b/drivers/pci/controller/dwc/pcie-qcom.c >> index 0180edf3310e..6976efb8e2f0 100644 >> --- a/drivers/pci/controller/dwc/pcie-qcom.c >> +++ b/drivers/pci/controller/dwc/pcie-qcom.c >> @@ -45,6 +45,7 @@ >> #define PARF_PHY_REFCLK 0x4c >> #define PARF_CONFIG_BITS 0x50 >> #define PARF_DBI_BASE_ADDR 0x168 >> +#define PARF_SLV_ADDR_SPACE_SIZE 0x16C > > Use lowercase for hex. > ACK. I will update it in the next patch. Thanks, Prudhvi > Rest LGTM! With above mentioned changes, > > Reviewed-by: Manivannan Sadhasivam <manivannan.sadhasivam@linaro.org> > > - Mani >
On Tue, Jul 23, 2024 at 07:27:19PM -0700, Prudhvi Yarlagadda wrote: > PARF hardware block which is a wrapper on top of DWC PCIe controller > mirrors the DBI and ATU register space. It uses PARF_SLV_ADDR_SPACE_SIZE > register to get the size of the memory block to be mirrored and uses > PARF_DBI_BASE_ADDR, PARF_ATU_BASE_ADDR registers to determine the base > address of DBI and ATU space inside the memory block that is being > mirrored. > > When a memory region which is located above the SLV_ADDR_SPACE_SIZE > boundary is used for BAR region then there could be an overlap of DBI and > ATU address space that is getting mirrored and the BAR region. This > results in DBI and ATU address space contents getting updated when a PCIe > function driver tries updating the BAR/MMIO memory region. Reference > memory map of the PCIe memory region with DBI and ATU address space > overlapping BAR region is as below. > > |---------------| > | | > | | > ------- --------|---------------| > | | |---------------| > | | | DBI | > | | |---------------|---->DBI_BASE_ADDR > | | | | > | | | | > | PCIe | |---->2*SLV_ADDR_SPACE_SIZE > | BAR/MMIO|---------------| > | Region | ATU | > | | |---------------|---->ATU_BASE_ADDR > | | | | > PCIe | |---------------| > Memory | | DBI | > Region | |---------------|---->DBI_BASE_ADDR > | | | | > | --------| | > | | |---->SLV_ADDR_SPACE_SIZE > | |---------------| > | | ATU | > | |---------------|---->ATU_BASE_ADDR > | | | > | |---------------| > | | DBI | > | |---------------|---->DBI_BASE_ADDR > | | | > | | | > ----------------|---------------| > | | > | | > | | > |---------------| > > Currently memory region beyond the SLV_ADDR_SPACE_SIZE boundary is not > used for BAR region which is why the above mentioned issue is not > encountered. This issue is discovered as part of internal testing when we > tried moving the BAR region beyond the SLV_ADDR_SPACE_SIZE boundary. Hence > we are trying to fix this. > > As PARF hardware block mirrors DBI and ATU register space after every > PARF_SLV_ADDR_SPACE_SIZE (default 0x1000000) boundary multiple, write > U32_MAX (all 0xFF's) to PARF_SLV_ADDR_SPACE_SIZE register to avoid > mirroring DBI and ATU to BAR/MMIO region. Write the physical base address > of DBI and ATU register blocks to PARF_DBI_BASE_ADDR (default 0x0) and > PARF_ATU_BASE_ADDR (default 0x1000) respectively to make sure DBI and ATU > blocks are at expected memory locations. > > The register offsets PARF_DBI_BASE_ADDR_V2, PARF_SLV_ADDR_SPACE_SIZE_V2 > and PARF_ATU_BASE_ADDR are applicable for platforms that use PARF > Qcom IP rev 1.9.0, 2.7.0 and 2.9.0. PARF_DBI_BASE_ADDR_V2 and > PARF_SLV_ADDR_SPACE_SIZE_V2 are applicable for PARF Qcom IP rev 2.3.3. > PARF_DBI_BASE_ADDR and PARF_SLV_ADDR_SPACE_SIZE are applicable for PARF > Qcom IP rev 1.0.0, 2.3.2 and 2.4.0. Updating the init()/post_init() > functions of the respective PARF versions to program applicable > PARF_DBI_BASE_ADDR, PARF_SLV_ADDR_SPACE_SIZE and PARF_ATU_BASE_ADDR > register offsets. And remove the unused SLV_ADDR_SPACE_SZ macro. > > Signed-off-by: Prudhvi Yarlagadda <quic_pyarlaga@quicinc.com> > Reviewed-by: Mayank Rana <quic_mrana@quicinc.com> > --- > drivers/pci/controller/dwc/pcie-qcom.c | 62 +++++++++++++++++++------- > 1 file changed, 45 insertions(+), 17 deletions(-) > > diff --git a/drivers/pci/controller/dwc/pcie-qcom.c b/drivers/pci/controller/dwc/pcie-qcom.c > index 0180edf3310e..6976efb8e2f0 100644 > --- a/drivers/pci/controller/dwc/pcie-qcom.c > +++ b/drivers/pci/controller/dwc/pcie-qcom.c > @@ -45,6 +45,7 @@ > #define PARF_PHY_REFCLK 0x4c > #define PARF_CONFIG_BITS 0x50 > #define PARF_DBI_BASE_ADDR 0x168 > +#define PARF_SLV_ADDR_SPACE_SIZE 0x16C > #define PARF_MHI_CLOCK_RESET_CTRL 0x174 > #define PARF_AXI_MSTR_WR_ADDR_HALT 0x178 > #define PARF_AXI_MSTR_WR_ADDR_HALT_V2 0x1a8 > @@ -52,8 +53,13 @@ > #define PARF_LTSSM 0x1b0 > #define PARF_SID_OFFSET 0x234 > #define PARF_BDF_TRANSLATE_CFG 0x24c > -#define PARF_SLV_ADDR_SPACE_SIZE 0x358 > +#define PARF_DBI_BASE_ADDR_V2 0x350 > +#define PARF_DBI_BASE_ADDR_V2_HI 0x354 > +#define PARF_SLV_ADDR_SPACE_SIZE_V2 0x358 > +#define PARF_SLV_ADDR_SPACE_SIZE_V2_HI 0x35C > #define PARF_NO_SNOOP_OVERIDE 0x3d4 > +#define PARF_ATU_BASE_ADDR 0x634 > +#define PARF_ATU_BASE_ADDR_HI 0x638 > #define PARF_DEVICE_TYPE 0x1000 > #define PARF_BDF_TO_SID_TABLE_N 0x2000 > #define PARF_BDF_TO_SID_CFG 0x2c00 > @@ -107,9 +113,6 @@ > /* PARF_CONFIG_BITS register fields */ > #define PHY_RX0_EQ(x) FIELD_PREP(GENMASK(26, 24), x) > > -/* PARF_SLV_ADDR_SPACE_SIZE register value */ > -#define SLV_ADDR_SPACE_SZ 0x10000000 > - > /* PARF_MHI_CLOCK_RESET_CTRL register fields */ > #define AHB_CLK_EN BIT(0) > #define MSTR_AXI_CLK_EN BIT(1) > @@ -324,6 +327,39 @@ static void qcom_pcie_clear_hpc(struct dw_pcie *pci) > dw_pcie_dbi_ro_wr_dis(pci); > } > > +static void qcom_pcie_configure_dbi_base(struct qcom_pcie *pcie) > +{ > + struct dw_pcie *pci = pcie->pci; > + > + if (pci->dbi_phys_addr) { > + writel(lower_32_bits(pci->dbi_phys_addr), pcie->parf + > + PARF_DBI_BASE_ADDR); > + writel(U32_MAX, pcie->parf + PARF_SLV_ADDR_SPACE_SIZE); > + } > +} > + > +static void qcom_pcie_configure_dbi_atu_base(struct qcom_pcie *pcie) > +{ > + struct dw_pcie *pci = pcie->pci; > + > + if (pci->dbi_phys_addr) { > + writel(lower_32_bits(pci->dbi_phys_addr), pcie->parf + > + PARF_DBI_BASE_ADDR_V2); > + writel(upper_32_bits(pci->dbi_phys_addr), pcie->parf + > + PARF_DBI_BASE_ADDR_V2_HI); > + > + if (pci->atu_phys_addr) { > + writel(lower_32_bits(pci->atu_phys_addr), pcie->parf + > + PARF_ATU_BASE_ADDR); > + writel(upper_32_bits(pci->atu_phys_addr), pcie->parf + > + PARF_ATU_BASE_ADDR_HI); > + } > + > + writel(U32_MAX, pcie->parf + PARF_SLV_ADDR_SPACE_SIZE_V2); > + writel(U32_MAX, pcie->parf + PARF_SLV_ADDR_SPACE_SIZE_V2_HI); > + } > +} These functions write CPU physical addresses into registers. I don't know where these registers live. If they are on the PCI side of the world, they most likely should contain PCI bus addresses, not CPU physical addresses. In some systems, PCI bus addresses are the same as CPU physical addresses, but on many systems they are not the same, so it's better if we don't make implicit assumptions that they are the same. Bjorn
Hi Bjorn, Thanks for the review comments. On 7/24/2024 11:43 AM, Bjorn Helgaas wrote: > On Tue, Jul 23, 2024 at 07:27:19PM -0700, Prudhvi Yarlagadda wrote: >> PARF hardware block which is a wrapper on top of DWC PCIe controller >> mirrors the DBI and ATU register space. It uses PARF_SLV_ADDR_SPACE_SIZE >> register to get the size of the memory block to be mirrored and uses >> PARF_DBI_BASE_ADDR, PARF_ATU_BASE_ADDR registers to determine the base >> address of DBI and ATU space inside the memory block that is being >> mirrored. >> >> When a memory region which is located above the SLV_ADDR_SPACE_SIZE >> boundary is used for BAR region then there could be an overlap of DBI and >> ATU address space that is getting mirrored and the BAR region. This >> results in DBI and ATU address space contents getting updated when a PCIe >> function driver tries updating the BAR/MMIO memory region. Reference >> memory map of the PCIe memory region with DBI and ATU address space >> overlapping BAR region is as below. >> >> |---------------| >> | | >> | | >> ------- --------|---------------| >> | | |---------------| >> | | | DBI | >> | | |---------------|---->DBI_BASE_ADDR >> | | | | >> | | | | >> | PCIe | |---->2*SLV_ADDR_SPACE_SIZE >> | BAR/MMIO|---------------| >> | Region | ATU | >> | | |---------------|---->ATU_BASE_ADDR >> | | | | >> PCIe | |---------------| >> Memory | | DBI | >> Region | |---------------|---->DBI_BASE_ADDR >> | | | | >> | --------| | >> | | |---->SLV_ADDR_SPACE_SIZE >> | |---------------| >> | | ATU | >> | |---------------|---->ATU_BASE_ADDR >> | | | >> | |---------------| >> | | DBI | >> | |---------------|---->DBI_BASE_ADDR >> | | | >> | | | >> ----------------|---------------| >> | | >> | | >> | | >> |---------------| >> >> Currently memory region beyond the SLV_ADDR_SPACE_SIZE boundary is not >> used for BAR region which is why the above mentioned issue is not >> encountered. This issue is discovered as part of internal testing when we >> tried moving the BAR region beyond the SLV_ADDR_SPACE_SIZE boundary. Hence >> we are trying to fix this. >> >> As PARF hardware block mirrors DBI and ATU register space after every >> PARF_SLV_ADDR_SPACE_SIZE (default 0x1000000) boundary multiple, write >> U32_MAX (all 0xFF's) to PARF_SLV_ADDR_SPACE_SIZE register to avoid >> mirroring DBI and ATU to BAR/MMIO region. Write the physical base address >> of DBI and ATU register blocks to PARF_DBI_BASE_ADDR (default 0x0) and >> PARF_ATU_BASE_ADDR (default 0x1000) respectively to make sure DBI and ATU >> blocks are at expected memory locations. >> >> The register offsets PARF_DBI_BASE_ADDR_V2, PARF_SLV_ADDR_SPACE_SIZE_V2 >> and PARF_ATU_BASE_ADDR are applicable for platforms that use PARF >> Qcom IP rev 1.9.0, 2.7.0 and 2.9.0. PARF_DBI_BASE_ADDR_V2 and >> PARF_SLV_ADDR_SPACE_SIZE_V2 are applicable for PARF Qcom IP rev 2.3.3. >> PARF_DBI_BASE_ADDR and PARF_SLV_ADDR_SPACE_SIZE are applicable for PARF >> Qcom IP rev 1.0.0, 2.3.2 and 2.4.0. Updating the init()/post_init() >> functions of the respective PARF versions to program applicable >> PARF_DBI_BASE_ADDR, PARF_SLV_ADDR_SPACE_SIZE and PARF_ATU_BASE_ADDR >> register offsets. And remove the unused SLV_ADDR_SPACE_SZ macro. >> >> Signed-off-by: Prudhvi Yarlagadda <quic_pyarlaga@quicinc.com> >> Reviewed-by: Mayank Rana <quic_mrana@quicinc.com> >> --- >> drivers/pci/controller/dwc/pcie-qcom.c | 62 +++++++++++++++++++------- >> 1 file changed, 45 insertions(+), 17 deletions(-) >> >> diff --git a/drivers/pci/controller/dwc/pcie-qcom.c b/drivers/pci/controller/dwc/pcie-qcom.c >> index 0180edf3310e..6976efb8e2f0 100644 >> --- a/drivers/pci/controller/dwc/pcie-qcom.c >> +++ b/drivers/pci/controller/dwc/pcie-qcom.c >> @@ -45,6 +45,7 @@ >> #define PARF_PHY_REFCLK 0x4c >> #define PARF_CONFIG_BITS 0x50 >> #define PARF_DBI_BASE_ADDR 0x168 >> +#define PARF_SLV_ADDR_SPACE_SIZE 0x16C >> #define PARF_MHI_CLOCK_RESET_CTRL 0x174 >> #define PARF_AXI_MSTR_WR_ADDR_HALT 0x178 >> #define PARF_AXI_MSTR_WR_ADDR_HALT_V2 0x1a8 >> @@ -52,8 +53,13 @@ >> #define PARF_LTSSM 0x1b0 >> #define PARF_SID_OFFSET 0x234 >> #define PARF_BDF_TRANSLATE_CFG 0x24c >> -#define PARF_SLV_ADDR_SPACE_SIZE 0x358 >> +#define PARF_DBI_BASE_ADDR_V2 0x350 >> +#define PARF_DBI_BASE_ADDR_V2_HI 0x354 >> +#define PARF_SLV_ADDR_SPACE_SIZE_V2 0x358 >> +#define PARF_SLV_ADDR_SPACE_SIZE_V2_HI 0x35C >> #define PARF_NO_SNOOP_OVERIDE 0x3d4 >> +#define PARF_ATU_BASE_ADDR 0x634 >> +#define PARF_ATU_BASE_ADDR_HI 0x638 >> #define PARF_DEVICE_TYPE 0x1000 >> #define PARF_BDF_TO_SID_TABLE_N 0x2000 >> #define PARF_BDF_TO_SID_CFG 0x2c00 >> @@ -107,9 +113,6 @@ >> /* PARF_CONFIG_BITS register fields */ >> #define PHY_RX0_EQ(x) FIELD_PREP(GENMASK(26, 24), x) >> >> -/* PARF_SLV_ADDR_SPACE_SIZE register value */ >> -#define SLV_ADDR_SPACE_SZ 0x10000000 >> - >> /* PARF_MHI_CLOCK_RESET_CTRL register fields */ >> #define AHB_CLK_EN BIT(0) >> #define MSTR_AXI_CLK_EN BIT(1) >> @@ -324,6 +327,39 @@ static void qcom_pcie_clear_hpc(struct dw_pcie *pci) >> dw_pcie_dbi_ro_wr_dis(pci); >> } >> >> +static void qcom_pcie_configure_dbi_base(struct qcom_pcie *pcie) >> +{ >> + struct dw_pcie *pci = pcie->pci; >> + >> + if (pci->dbi_phys_addr) { >> + writel(lower_32_bits(pci->dbi_phys_addr), pcie->parf + >> + PARF_DBI_BASE_ADDR); >> + writel(U32_MAX, pcie->parf + PARF_SLV_ADDR_SPACE_SIZE); >> + } >> +} >> + >> +static void qcom_pcie_configure_dbi_atu_base(struct qcom_pcie *pcie) >> +{ >> + struct dw_pcie *pci = pcie->pci; >> + >> + if (pci->dbi_phys_addr) { >> + writel(lower_32_bits(pci->dbi_phys_addr), pcie->parf + >> + PARF_DBI_BASE_ADDR_V2); >> + writel(upper_32_bits(pci->dbi_phys_addr), pcie->parf + >> + PARF_DBI_BASE_ADDR_V2_HI); >> + >> + if (pci->atu_phys_addr) { >> + writel(lower_32_bits(pci->atu_phys_addr), pcie->parf + >> + PARF_ATU_BASE_ADDR); >> + writel(upper_32_bits(pci->atu_phys_addr), pcie->parf + >> + PARF_ATU_BASE_ADDR_HI); >> + } >> + >> + writel(U32_MAX, pcie->parf + PARF_SLV_ADDR_SPACE_SIZE_V2); >> + writel(U32_MAX, pcie->parf + PARF_SLV_ADDR_SPACE_SIZE_V2_HI); >> + } >> +} > > These functions write CPU physical addresses into registers. I don't > know where these registers live. If they are on the PCI side of the > world, they most likely should contain PCI bus addresses, not CPU > physical addresses. > > In some systems, PCI bus addresses are the same as CPU physical > addresses, but on many systems they are not the same, so it's better > if we don't make implicit assumptions that they are the same. > > Bjorn On Qualcomm platforms, CPU physical address and PCI bus addresses for DBI and ATU registers are same. PARF registers live outside the PCI address space in the system memory. There is a mapping logic in the QCOM PARF wrapper which detects any incoming read/write transactions from the NOC towards PCIe controller and checks its addresses against PARF_DBI_BASE_ADDR and PARF_ATU_BASE_ADDR registers so that these transactions can be routed to DBI and ATU registers inside the PCIe controller. So, these PARF registers needs to be programmed with base CPU physical addresses of DBI and ATU as the incoming DBI and ATU transactions from the NOC contain CPU physical adresses. Thanks, Prudhvi
On Thu, Jul 25, 2024 at 04:03:56PM -0700, Prudhvi Yarlagadda wrote: > Hi Bjorn, > > Thanks for the review comments. > > On 7/24/2024 11:43 AM, Bjorn Helgaas wrote: > > On Tue, Jul 23, 2024 at 07:27:19PM -0700, Prudhvi Yarlagadda wrote: > >> PARF hardware block which is a wrapper on top of DWC PCIe controller > >> mirrors the DBI and ATU register space. It uses PARF_SLV_ADDR_SPACE_SIZE > >> register to get the size of the memory block to be mirrored and uses > >> PARF_DBI_BASE_ADDR, PARF_ATU_BASE_ADDR registers to determine the base > >> address of DBI and ATU space inside the memory block that is being > >> mirrored. > >> > >> When a memory region which is located above the SLV_ADDR_SPACE_SIZE > >> boundary is used for BAR region then there could be an overlap of DBI and > >> ATU address space that is getting mirrored and the BAR region. This > >> results in DBI and ATU address space contents getting updated when a PCIe > >> function driver tries updating the BAR/MMIO memory region. Reference > >> memory map of the PCIe memory region with DBI and ATU address space > >> overlapping BAR region is as below. > >> > >> |---------------| > >> | | > >> | | > >> ------- --------|---------------| > >> | | |---------------| > >> | | | DBI | > >> | | |---------------|---->DBI_BASE_ADDR > >> | | | | > >> | | | | > >> | PCIe | |---->2*SLV_ADDR_SPACE_SIZE > >> | BAR/MMIO|---------------| > >> | Region | ATU | > >> | | |---------------|---->ATU_BASE_ADDR > >> | | | | > >> PCIe | |---------------| > >> Memory | | DBI | > >> Region | |---------------|---->DBI_BASE_ADDR > >> | | | | > >> | --------| | > >> | | |---->SLV_ADDR_SPACE_SIZE > >> | |---------------| > >> | | ATU | > >> | |---------------|---->ATU_BASE_ADDR > >> | | | > >> | |---------------| > >> | | DBI | > >> | |---------------|---->DBI_BASE_ADDR > >> | | | > >> | | | > >> ----------------|---------------| > >> | | > >> | | > >> | | > >> |---------------| > >> > >> Currently memory region beyond the SLV_ADDR_SPACE_SIZE boundary is not > >> used for BAR region which is why the above mentioned issue is not > >> encountered. This issue is discovered as part of internal testing when we > >> tried moving the BAR region beyond the SLV_ADDR_SPACE_SIZE boundary. Hence > >> we are trying to fix this. > >> > >> As PARF hardware block mirrors DBI and ATU register space after every > >> PARF_SLV_ADDR_SPACE_SIZE (default 0x1000000) boundary multiple, write > >> U32_MAX (all 0xFF's) to PARF_SLV_ADDR_SPACE_SIZE register to avoid > >> mirroring DBI and ATU to BAR/MMIO region. Write the physical base address > >> of DBI and ATU register blocks to PARF_DBI_BASE_ADDR (default 0x0) and > >> PARF_ATU_BASE_ADDR (default 0x1000) respectively to make sure DBI and ATU > >> blocks are at expected memory locations. > >> > >> The register offsets PARF_DBI_BASE_ADDR_V2, PARF_SLV_ADDR_SPACE_SIZE_V2 > >> and PARF_ATU_BASE_ADDR are applicable for platforms that use PARF > >> Qcom IP rev 1.9.0, 2.7.0 and 2.9.0. PARF_DBI_BASE_ADDR_V2 and > >> PARF_SLV_ADDR_SPACE_SIZE_V2 are applicable for PARF Qcom IP rev 2.3.3. > >> PARF_DBI_BASE_ADDR and PARF_SLV_ADDR_SPACE_SIZE are applicable for PARF > >> Qcom IP rev 1.0.0, 2.3.2 and 2.4.0. Updating the init()/post_init() > >> functions of the respective PARF versions to program applicable > >> PARF_DBI_BASE_ADDR, PARF_SLV_ADDR_SPACE_SIZE and PARF_ATU_BASE_ADDR > >> register offsets. And remove the unused SLV_ADDR_SPACE_SZ macro. > >> > >> Signed-off-by: Prudhvi Yarlagadda <quic_pyarlaga@quicinc.com> > >> Reviewed-by: Mayank Rana <quic_mrana@quicinc.com> > >> --- > >> drivers/pci/controller/dwc/pcie-qcom.c | 62 +++++++++++++++++++------- > >> 1 file changed, 45 insertions(+), 17 deletions(-) > >> > >> diff --git a/drivers/pci/controller/dwc/pcie-qcom.c b/drivers/pci/controller/dwc/pcie-qcom.c > >> index 0180edf3310e..6976efb8e2f0 100644 > >> --- a/drivers/pci/controller/dwc/pcie-qcom.c > >> +++ b/drivers/pci/controller/dwc/pcie-qcom.c > >> @@ -45,6 +45,7 @@ > >> #define PARF_PHY_REFCLK 0x4c > >> #define PARF_CONFIG_BITS 0x50 > >> #define PARF_DBI_BASE_ADDR 0x168 > >> +#define PARF_SLV_ADDR_SPACE_SIZE 0x16C > >> #define PARF_MHI_CLOCK_RESET_CTRL 0x174 > >> #define PARF_AXI_MSTR_WR_ADDR_HALT 0x178 > >> #define PARF_AXI_MSTR_WR_ADDR_HALT_V2 0x1a8 > >> @@ -52,8 +53,13 @@ > >> #define PARF_LTSSM 0x1b0 > >> #define PARF_SID_OFFSET 0x234 > >> #define PARF_BDF_TRANSLATE_CFG 0x24c > >> -#define PARF_SLV_ADDR_SPACE_SIZE 0x358 > >> +#define PARF_DBI_BASE_ADDR_V2 0x350 > >> +#define PARF_DBI_BASE_ADDR_V2_HI 0x354 > >> +#define PARF_SLV_ADDR_SPACE_SIZE_V2 0x358 > >> +#define PARF_SLV_ADDR_SPACE_SIZE_V2_HI 0x35C > >> #define PARF_NO_SNOOP_OVERIDE 0x3d4 > >> +#define PARF_ATU_BASE_ADDR 0x634 > >> +#define PARF_ATU_BASE_ADDR_HI 0x638 > >> #define PARF_DEVICE_TYPE 0x1000 > >> #define PARF_BDF_TO_SID_TABLE_N 0x2000 > >> #define PARF_BDF_TO_SID_CFG 0x2c00 > >> @@ -107,9 +113,6 @@ > >> /* PARF_CONFIG_BITS register fields */ > >> #define PHY_RX0_EQ(x) FIELD_PREP(GENMASK(26, 24), x) > >> > >> -/* PARF_SLV_ADDR_SPACE_SIZE register value */ > >> -#define SLV_ADDR_SPACE_SZ 0x10000000 > >> - > >> /* PARF_MHI_CLOCK_RESET_CTRL register fields */ > >> #define AHB_CLK_EN BIT(0) > >> #define MSTR_AXI_CLK_EN BIT(1) > >> @@ -324,6 +327,39 @@ static void qcom_pcie_clear_hpc(struct dw_pcie *pci) > >> dw_pcie_dbi_ro_wr_dis(pci); > >> } > >> > >> +static void qcom_pcie_configure_dbi_base(struct qcom_pcie *pcie) > >> +{ > >> + struct dw_pcie *pci = pcie->pci; > >> + > >> + if (pci->dbi_phys_addr) { > >> + writel(lower_32_bits(pci->dbi_phys_addr), pcie->parf + > >> + PARF_DBI_BASE_ADDR); > >> + writel(U32_MAX, pcie->parf + PARF_SLV_ADDR_SPACE_SIZE); > >> + } > >> +} > >> + > >> +static void qcom_pcie_configure_dbi_atu_base(struct qcom_pcie *pcie) > >> +{ > >> + struct dw_pcie *pci = pcie->pci; > >> + > >> + if (pci->dbi_phys_addr) { > >> + writel(lower_32_bits(pci->dbi_phys_addr), pcie->parf + > >> + PARF_DBI_BASE_ADDR_V2); > >> + writel(upper_32_bits(pci->dbi_phys_addr), pcie->parf + > >> + PARF_DBI_BASE_ADDR_V2_HI); > >> + > >> + if (pci->atu_phys_addr) { > >> + writel(lower_32_bits(pci->atu_phys_addr), pcie->parf + > >> + PARF_ATU_BASE_ADDR); > >> + writel(upper_32_bits(pci->atu_phys_addr), pcie->parf + > >> + PARF_ATU_BASE_ADDR_HI); > >> + } > >> + > >> + writel(U32_MAX, pcie->parf + PARF_SLV_ADDR_SPACE_SIZE_V2); > >> + writel(U32_MAX, pcie->parf + PARF_SLV_ADDR_SPACE_SIZE_V2_HI); > >> + } > >> +} > > > > These functions write CPU physical addresses into registers. I don't > > know where these registers live. If they are on the PCI side of the > > world, they most likely should contain PCI bus addresses, not CPU > > physical addresses. > > > > In some systems, PCI bus addresses are the same as CPU physical > > addresses, but on many systems they are not the same, so it's better > > if we don't make implicit assumptions that they are the same. > > On Qualcomm platforms, CPU physical address and PCI bus addresses > for DBI and ATU registers are same. PARF registers live outside the > PCI address space in the system memory. > > There is a mapping logic in the QCOM PARF wrapper which detects any > incoming read/write transactions from the NOC towards PCIe > controller and checks its addresses against PARF_DBI_BASE_ADDR and > PARF_ATU_BASE_ADDR registers so that these transactions can be > routed to DBI and ATU registers inside the PCIe controller. > > So, these PARF registers needs to be programmed with base CPU > physical addresses of DBI and ATU as the incoming DBI and ATU > transactions from the NOC contain CPU physical adresses. Can you add a comment to this effect that these registers are effectively in the CPU address domain, not the PCI bus domain? Otherwise the next person who reviews this will have the same question, and somebody may even try to "fix" this by converting it to a PCI bus address. Bjorn
Hi Bjorn, On 7/29/2024 3:51 PM, Bjorn Helgaas wrote: > On Thu, Jul 25, 2024 at 04:03:56PM -0700, Prudhvi Yarlagadda wrote: >> Hi Bjorn, >> >> Thanks for the review comments. >> >> On 7/24/2024 11:43 AM, Bjorn Helgaas wrote: >>> On Tue, Jul 23, 2024 at 07:27:19PM -0700, Prudhvi Yarlagadda wrote: >>>> PARF hardware block which is a wrapper on top of DWC PCIe controller >>>> mirrors the DBI and ATU register space. It uses PARF_SLV_ADDR_SPACE_SIZE >>>> register to get the size of the memory block to be mirrored and uses >>>> PARF_DBI_BASE_ADDR, PARF_ATU_BASE_ADDR registers to determine the base >>>> address of DBI and ATU space inside the memory block that is being >>>> mirrored. >>>> >>>> When a memory region which is located above the SLV_ADDR_SPACE_SIZE >>>> boundary is used for BAR region then there could be an overlap of DBI and >>>> ATU address space that is getting mirrored and the BAR region. This >>>> results in DBI and ATU address space contents getting updated when a PCIe >>>> function driver tries updating the BAR/MMIO memory region. Reference >>>> memory map of the PCIe memory region with DBI and ATU address space >>>> overlapping BAR region is as below. >>>> >>>> |---------------| >>>> | | >>>> | | >>>> ------- --------|---------------| >>>> | | |---------------| >>>> | | | DBI | >>>> | | |---------------|---->DBI_BASE_ADDR >>>> | | | | >>>> | | | | >>>> | PCIe | |---->2*SLV_ADDR_SPACE_SIZE >>>> | BAR/MMIO|---------------| >>>> | Region | ATU | >>>> | | |---------------|---->ATU_BASE_ADDR >>>> | | | | >>>> PCIe | |---------------| >>>> Memory | | DBI | >>>> Region | |---------------|---->DBI_BASE_ADDR >>>> | | | | >>>> | --------| | >>>> | | |---->SLV_ADDR_SPACE_SIZE >>>> | |---------------| >>>> | | ATU | >>>> | |---------------|---->ATU_BASE_ADDR >>>> | | | >>>> | |---------------| >>>> | | DBI | >>>> | |---------------|---->DBI_BASE_ADDR >>>> | | | >>>> | | | >>>> ----------------|---------------| >>>> | | >>>> | | >>>> | | >>>> |---------------| >>>> >>>> Currently memory region beyond the SLV_ADDR_SPACE_SIZE boundary is not >>>> used for BAR region which is why the above mentioned issue is not >>>> encountered. This issue is discovered as part of internal testing when we >>>> tried moving the BAR region beyond the SLV_ADDR_SPACE_SIZE boundary. Hence >>>> we are trying to fix this. >>>> >>>> As PARF hardware block mirrors DBI and ATU register space after every >>>> PARF_SLV_ADDR_SPACE_SIZE (default 0x1000000) boundary multiple, write >>>> U32_MAX (all 0xFF's) to PARF_SLV_ADDR_SPACE_SIZE register to avoid >>>> mirroring DBI and ATU to BAR/MMIO region. Write the physical base address >>>> of DBI and ATU register blocks to PARF_DBI_BASE_ADDR (default 0x0) and >>>> PARF_ATU_BASE_ADDR (default 0x1000) respectively to make sure DBI and ATU >>>> blocks are at expected memory locations. >>>> >>>> The register offsets PARF_DBI_BASE_ADDR_V2, PARF_SLV_ADDR_SPACE_SIZE_V2 >>>> and PARF_ATU_BASE_ADDR are applicable for platforms that use PARF >>>> Qcom IP rev 1.9.0, 2.7.0 and 2.9.0. PARF_DBI_BASE_ADDR_V2 and >>>> PARF_SLV_ADDR_SPACE_SIZE_V2 are applicable for PARF Qcom IP rev 2.3.3. >>>> PARF_DBI_BASE_ADDR and PARF_SLV_ADDR_SPACE_SIZE are applicable for PARF >>>> Qcom IP rev 1.0.0, 2.3.2 and 2.4.0. Updating the init()/post_init() >>>> functions of the respective PARF versions to program applicable >>>> PARF_DBI_BASE_ADDR, PARF_SLV_ADDR_SPACE_SIZE and PARF_ATU_BASE_ADDR >>>> register offsets. And remove the unused SLV_ADDR_SPACE_SZ macro. >>>> >>>> Signed-off-by: Prudhvi Yarlagadda <quic_pyarlaga@quicinc.com> >>>> Reviewed-by: Mayank Rana <quic_mrana@quicinc.com> >>>> --- >>>> drivers/pci/controller/dwc/pcie-qcom.c | 62 +++++++++++++++++++------- >>>> 1 file changed, 45 insertions(+), 17 deletions(-) >>>> >>>> diff --git a/drivers/pci/controller/dwc/pcie-qcom.c b/drivers/pci/controller/dwc/pcie-qcom.c >>>> index 0180edf3310e..6976efb8e2f0 100644 >>>> --- a/drivers/pci/controller/dwc/pcie-qcom.c >>>> +++ b/drivers/pci/controller/dwc/pcie-qcom.c >>>> @@ -45,6 +45,7 @@ >>>> #define PARF_PHY_REFCLK 0x4c >>>> #define PARF_CONFIG_BITS 0x50 >>>> #define PARF_DBI_BASE_ADDR 0x168 >>>> +#define PARF_SLV_ADDR_SPACE_SIZE 0x16C >>>> #define PARF_MHI_CLOCK_RESET_CTRL 0x174 >>>> #define PARF_AXI_MSTR_WR_ADDR_HALT 0x178 >>>> #define PARF_AXI_MSTR_WR_ADDR_HALT_V2 0x1a8 >>>> @@ -52,8 +53,13 @@ >>>> #define PARF_LTSSM 0x1b0 >>>> #define PARF_SID_OFFSET 0x234 >>>> #define PARF_BDF_TRANSLATE_CFG 0x24c >>>> -#define PARF_SLV_ADDR_SPACE_SIZE 0x358 >>>> +#define PARF_DBI_BASE_ADDR_V2 0x350 >>>> +#define PARF_DBI_BASE_ADDR_V2_HI 0x354 >>>> +#define PARF_SLV_ADDR_SPACE_SIZE_V2 0x358 >>>> +#define PARF_SLV_ADDR_SPACE_SIZE_V2_HI 0x35C >>>> #define PARF_NO_SNOOP_OVERIDE 0x3d4 >>>> +#define PARF_ATU_BASE_ADDR 0x634 >>>> +#define PARF_ATU_BASE_ADDR_HI 0x638 >>>> #define PARF_DEVICE_TYPE 0x1000 >>>> #define PARF_BDF_TO_SID_TABLE_N 0x2000 >>>> #define PARF_BDF_TO_SID_CFG 0x2c00 >>>> @@ -107,9 +113,6 @@ >>>> /* PARF_CONFIG_BITS register fields */ >>>> #define PHY_RX0_EQ(x) FIELD_PREP(GENMASK(26, 24), x) >>>> >>>> -/* PARF_SLV_ADDR_SPACE_SIZE register value */ >>>> -#define SLV_ADDR_SPACE_SZ 0x10000000 >>>> - >>>> /* PARF_MHI_CLOCK_RESET_CTRL register fields */ >>>> #define AHB_CLK_EN BIT(0) >>>> #define MSTR_AXI_CLK_EN BIT(1) >>>> @@ -324,6 +327,39 @@ static void qcom_pcie_clear_hpc(struct dw_pcie *pci) >>>> dw_pcie_dbi_ro_wr_dis(pci); >>>> } >>>> >>>> +static void qcom_pcie_configure_dbi_base(struct qcom_pcie *pcie) >>>> +{ >>>> + struct dw_pcie *pci = pcie->pci; >>>> + >>>> + if (pci->dbi_phys_addr) { >>>> + writel(lower_32_bits(pci->dbi_phys_addr), pcie->parf + >>>> + PARF_DBI_BASE_ADDR); >>>> + writel(U32_MAX, pcie->parf + PARF_SLV_ADDR_SPACE_SIZE); >>>> + } >>>> +} >>>> + >>>> +static void qcom_pcie_configure_dbi_atu_base(struct qcom_pcie *pcie) >>>> +{ >>>> + struct dw_pcie *pci = pcie->pci; >>>> + >>>> + if (pci->dbi_phys_addr) { >>>> + writel(lower_32_bits(pci->dbi_phys_addr), pcie->parf + >>>> + PARF_DBI_BASE_ADDR_V2); >>>> + writel(upper_32_bits(pci->dbi_phys_addr), pcie->parf + >>>> + PARF_DBI_BASE_ADDR_V2_HI); >>>> + >>>> + if (pci->atu_phys_addr) { >>>> + writel(lower_32_bits(pci->atu_phys_addr), pcie->parf + >>>> + PARF_ATU_BASE_ADDR); >>>> + writel(upper_32_bits(pci->atu_phys_addr), pcie->parf + >>>> + PARF_ATU_BASE_ADDR_HI); >>>> + } >>>> + >>>> + writel(U32_MAX, pcie->parf + PARF_SLV_ADDR_SPACE_SIZE_V2); >>>> + writel(U32_MAX, pcie->parf + PARF_SLV_ADDR_SPACE_SIZE_V2_HI); >>>> + } >>>> +} >>> >>> These functions write CPU physical addresses into registers. I don't >>> know where these registers live. If they are on the PCI side of the >>> world, they most likely should contain PCI bus addresses, not CPU >>> physical addresses. >>> >>> In some systems, PCI bus addresses are the same as CPU physical >>> addresses, but on many systems they are not the same, so it's better >>> if we don't make implicit assumptions that they are the same. >> >> On Qualcomm platforms, CPU physical address and PCI bus addresses >> for DBI and ATU registers are same. PARF registers live outside the >> PCI address space in the system memory. >> >> There is a mapping logic in the QCOM PARF wrapper which detects any >> incoming read/write transactions from the NOC towards PCIe >> controller and checks its addresses against PARF_DBI_BASE_ADDR and >> PARF_ATU_BASE_ADDR registers so that these transactions can be >> routed to DBI and ATU registers inside the PCIe controller. >> >> So, these PARF registers needs to be programmed with base CPU >> physical addresses of DBI and ATU as the incoming DBI and ATU >> transactions from the NOC contain CPU physical adresses. > > Can you add a comment to this effect that these registers are > effectively in the CPU address domain, not the PCI bus domain? > Otherwise the next person who reviews this will have the same > question, and somebody may even try to "fix" this by converting it to > a PCI bus address. > > Bjorn ACK. I will add a comment in the next patch. Thanks, Prudhvi
diff --git a/drivers/pci/controller/dwc/pcie-qcom.c b/drivers/pci/controller/dwc/pcie-qcom.c index 0180edf3310e..6976efb8e2f0 100644 --- a/drivers/pci/controller/dwc/pcie-qcom.c +++ b/drivers/pci/controller/dwc/pcie-qcom.c @@ -45,6 +45,7 @@ #define PARF_PHY_REFCLK 0x4c #define PARF_CONFIG_BITS 0x50 #define PARF_DBI_BASE_ADDR 0x168 +#define PARF_SLV_ADDR_SPACE_SIZE 0x16C #define PARF_MHI_CLOCK_RESET_CTRL 0x174 #define PARF_AXI_MSTR_WR_ADDR_HALT 0x178 #define PARF_AXI_MSTR_WR_ADDR_HALT_V2 0x1a8 @@ -52,8 +53,13 @@ #define PARF_LTSSM 0x1b0 #define PARF_SID_OFFSET 0x234 #define PARF_BDF_TRANSLATE_CFG 0x24c -#define PARF_SLV_ADDR_SPACE_SIZE 0x358 +#define PARF_DBI_BASE_ADDR_V2 0x350 +#define PARF_DBI_BASE_ADDR_V2_HI 0x354 +#define PARF_SLV_ADDR_SPACE_SIZE_V2 0x358 +#define PARF_SLV_ADDR_SPACE_SIZE_V2_HI 0x35C #define PARF_NO_SNOOP_OVERIDE 0x3d4 +#define PARF_ATU_BASE_ADDR 0x634 +#define PARF_ATU_BASE_ADDR_HI 0x638 #define PARF_DEVICE_TYPE 0x1000 #define PARF_BDF_TO_SID_TABLE_N 0x2000 #define PARF_BDF_TO_SID_CFG 0x2c00 @@ -107,9 +113,6 @@ /* PARF_CONFIG_BITS register fields */ #define PHY_RX0_EQ(x) FIELD_PREP(GENMASK(26, 24), x) -/* PARF_SLV_ADDR_SPACE_SIZE register value */ -#define SLV_ADDR_SPACE_SZ 0x10000000 - /* PARF_MHI_CLOCK_RESET_CTRL register fields */ #define AHB_CLK_EN BIT(0) #define MSTR_AXI_CLK_EN BIT(1) @@ -324,6 +327,39 @@ static void qcom_pcie_clear_hpc(struct dw_pcie *pci) dw_pcie_dbi_ro_wr_dis(pci); } +static void qcom_pcie_configure_dbi_base(struct qcom_pcie *pcie) +{ + struct dw_pcie *pci = pcie->pci; + + if (pci->dbi_phys_addr) { + writel(lower_32_bits(pci->dbi_phys_addr), pcie->parf + + PARF_DBI_BASE_ADDR); + writel(U32_MAX, pcie->parf + PARF_SLV_ADDR_SPACE_SIZE); + } +} + +static void qcom_pcie_configure_dbi_atu_base(struct qcom_pcie *pcie) +{ + struct dw_pcie *pci = pcie->pci; + + if (pci->dbi_phys_addr) { + writel(lower_32_bits(pci->dbi_phys_addr), pcie->parf + + PARF_DBI_BASE_ADDR_V2); + writel(upper_32_bits(pci->dbi_phys_addr), pcie->parf + + PARF_DBI_BASE_ADDR_V2_HI); + + if (pci->atu_phys_addr) { + writel(lower_32_bits(pci->atu_phys_addr), pcie->parf + + PARF_ATU_BASE_ADDR); + writel(upper_32_bits(pci->atu_phys_addr), pcie->parf + + PARF_ATU_BASE_ADDR_HI); + } + + writel(U32_MAX, pcie->parf + PARF_SLV_ADDR_SPACE_SIZE_V2); + writel(U32_MAX, pcie->parf + PARF_SLV_ADDR_SPACE_SIZE_V2_HI); + } +} + static void qcom_pcie_2_1_0_ltssm_enable(struct qcom_pcie *pcie) { u32 val; @@ -540,8 +576,7 @@ static int qcom_pcie_init_1_0_0(struct qcom_pcie *pcie) static int qcom_pcie_post_init_1_0_0(struct qcom_pcie *pcie) { - /* change DBI base address */ - writel(0, pcie->parf + PARF_DBI_BASE_ADDR); + qcom_pcie_configure_dbi_base(pcie); if (IS_ENABLED(CONFIG_PCI_MSI)) { u32 val = readl(pcie->parf + PARF_AXI_MSTR_WR_ADDR_HALT); @@ -628,8 +663,7 @@ static int qcom_pcie_post_init_2_3_2(struct qcom_pcie *pcie) val &= ~PHY_TEST_PWR_DOWN; writel(val, pcie->parf + PARF_PHY_CTRL); - /* change DBI base address */ - writel(0, pcie->parf + PARF_DBI_BASE_ADDR); + qcom_pcie_configure_dbi_base(pcie); /* MAC PHY_POWERDOWN MUX DISABLE */ val = readl(pcie->parf + PARF_SYS_CTRL); @@ -811,13 +845,11 @@ static int qcom_pcie_post_init_2_3_3(struct qcom_pcie *pcie) u16 offset = dw_pcie_find_capability(pci, PCI_CAP_ID_EXP); u32 val; - writel(SLV_ADDR_SPACE_SZ, pcie->parf + PARF_SLV_ADDR_SPACE_SIZE); - val = readl(pcie->parf + PARF_PHY_CTRL); val &= ~PHY_TEST_PWR_DOWN; writel(val, pcie->parf + PARF_PHY_CTRL); - writel(0, pcie->parf + PARF_DBI_BASE_ADDR); + qcom_pcie_configure_dbi_atu_base(pcie); writel(MST_WAKEUP_EN | SLV_WAKEUP_EN | MSTR_ACLK_CGC_DIS | SLV_ACLK_CGC_DIS | CORE_CLK_CGC_DIS | @@ -913,8 +945,7 @@ static int qcom_pcie_init_2_7_0(struct qcom_pcie *pcie) val &= ~PHY_TEST_PWR_DOWN; writel(val, pcie->parf + PARF_PHY_CTRL); - /* change DBI base address */ - writel(0, pcie->parf + PARF_DBI_BASE_ADDR); + qcom_pcie_configure_dbi_atu_base(pcie); /* MAC PHY_POWERDOWN MUX DISABLE */ val = readl(pcie->parf + PARF_SYS_CTRL); @@ -1123,14 +1154,11 @@ static int qcom_pcie_post_init_2_9_0(struct qcom_pcie *pcie) u32 val; int i; - writel(SLV_ADDR_SPACE_SZ, - pcie->parf + PARF_SLV_ADDR_SPACE_SIZE); - val = readl(pcie->parf + PARF_PHY_CTRL); val &= ~PHY_TEST_PWR_DOWN; writel(val, pcie->parf + PARF_PHY_CTRL); - writel(0, pcie->parf + PARF_DBI_BASE_ADDR); + qcom_pcie_configure_dbi_atu_base(pcie); writel(DEVICE_TYPE_RC, pcie->parf + PARF_DEVICE_TYPE); writel(BYPASS | MSTR_AXI_CLK_EN | AHB_CLK_EN,
PARF hardware block which is a wrapper on top of DWC PCIe controller mirrors the DBI and ATU register space. It uses PARF_SLV_ADDR_SPACE_SIZE register to get the size of the memory block to be mirrored and uses PARF_DBI_BASE_ADDR, PARF_ATU_BASE_ADDR registers to determine the base address of DBI and ATU space inside the memory block that is being mirrored. When a memory region which is located above the SLV_ADDR_SPACE_SIZE boundary is used for BAR region then there could be an overlap of DBI and ATU address space that is getting mirrored and the BAR region. This results in DBI and ATU address space contents getting updated when a PCIe function driver tries updating the BAR/MMIO memory region. Reference memory map of the PCIe memory region with DBI and ATU address space overlapping BAR region is as below. |---------------| | | | | ------- --------|---------------| | | |---------------| | | | DBI | | | |---------------|---->DBI_BASE_ADDR | | | | | | | | | PCIe | |---->2*SLV_ADDR_SPACE_SIZE | BAR/MMIO|---------------| | Region | ATU | | | |---------------|---->ATU_BASE_ADDR | | | | PCIe | |---------------| Memory | | DBI | Region | |---------------|---->DBI_BASE_ADDR | | | | | --------| | | | |---->SLV_ADDR_SPACE_SIZE | |---------------| | | ATU | | |---------------|---->ATU_BASE_ADDR | | | | |---------------| | | DBI | | |---------------|---->DBI_BASE_ADDR | | | | | | ----------------|---------------| | | | | | | |---------------| Currently memory region beyond the SLV_ADDR_SPACE_SIZE boundary is not used for BAR region which is why the above mentioned issue is not encountered. This issue is discovered as part of internal testing when we tried moving the BAR region beyond the SLV_ADDR_SPACE_SIZE boundary. Hence we are trying to fix this. As PARF hardware block mirrors DBI and ATU register space after every PARF_SLV_ADDR_SPACE_SIZE (default 0x1000000) boundary multiple, write U32_MAX (all 0xFF's) to PARF_SLV_ADDR_SPACE_SIZE register to avoid mirroring DBI and ATU to BAR/MMIO region. Write the physical base address of DBI and ATU register blocks to PARF_DBI_BASE_ADDR (default 0x0) and PARF_ATU_BASE_ADDR (default 0x1000) respectively to make sure DBI and ATU blocks are at expected memory locations. The register offsets PARF_DBI_BASE_ADDR_V2, PARF_SLV_ADDR_SPACE_SIZE_V2 and PARF_ATU_BASE_ADDR are applicable for platforms that use PARF Qcom IP rev 1.9.0, 2.7.0 and 2.9.0. PARF_DBI_BASE_ADDR_V2 and PARF_SLV_ADDR_SPACE_SIZE_V2 are applicable for PARF Qcom IP rev 2.3.3. PARF_DBI_BASE_ADDR and PARF_SLV_ADDR_SPACE_SIZE are applicable for PARF Qcom IP rev 1.0.0, 2.3.2 and 2.4.0. Updating the init()/post_init() functions of the respective PARF versions to program applicable PARF_DBI_BASE_ADDR, PARF_SLV_ADDR_SPACE_SIZE and PARF_ATU_BASE_ADDR register offsets. And remove the unused SLV_ADDR_SPACE_SZ macro. Signed-off-by: Prudhvi Yarlagadda <quic_pyarlaga@quicinc.com> Reviewed-by: Mayank Rana <quic_mrana@quicinc.com> --- drivers/pci/controller/dwc/pcie-qcom.c | 62 +++++++++++++++++++------- 1 file changed, 45 insertions(+), 17 deletions(-)