From patchwork Tue Dec 31 11:35:28 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Subject: [1/7] Documentation: PCI: endpoint/pci-endpoint-cfs: Guide to use SR-IOV X-Patchwork-Submitter: Kishon Vijay Abraham I X-Patchwork-Id: 182674 Message-Id: <20191231113534.30405-2-kishon@ti.com> To: Kishon Vijay Abraham I , Lorenzo Pieralisi , Bjorn Helgaas , Andrew Murray , Tom Joseph , Rob Herring , Arnd Bergmann , Jingoo Han , Gustavo Pimentel , Shawn Lin , Heiko Stuebner Cc: , , Jonathan Corbet , Greg Kroah-Hartman , , , , Date: Tue, 31 Dec 2019 17:05:28 +0530 From: Kishon Vijay Abraham I List-Id: Add Documentation to help users use PCI endpoint to create virtual functions using configfs. An endpoint function is designated as a virtual endpoint function device when it is linked to a physical endpoint function device (instead of a endpoint controller). Signed-off-by: Kishon Vijay Abraham I --- Documentation/PCI/endpoint/pci-endpoint-cfs.rst | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) -- 2.17.1 diff --git a/Documentation/PCI/endpoint/pci-endpoint-cfs.rst b/Documentation/PCI/endpoint/pci-endpoint-cfs.rst index b6d39cdec56e..0f55c5511c6c 100644 --- a/Documentation/PCI/endpoint/pci-endpoint-cfs.rst +++ b/Documentation/PCI/endpoint/pci-endpoint-cfs.rst @@ -43,6 +43,7 @@ entries corresponding to EPF driver will be created by the EPF core. .. / ... / ... / + ... / .. / ... / ... / @@ -68,6 +69,14 @@ created) ... subsys_vendor_id ... subsys_id ... interrupt_pin + ... / + +The directory can have a list of symbolic links to +other . These symbolic links should be created by the user +to represent the virtual functions that are bound to the physical +function. In the above directory structure is a physical +function and is a virtual function. A EPF device once it's +linked to another EPF device, cannot be linked to a EPC device. EPC Device ========== @@ -88,7 +97,8 @@ entries corresponding to EPC device will be created by the EPC core. The directory will have a list of symbolic links to . These symbolic links should be created by the user to -represent the functions present in the endpoint device. +represent the functions present in the endpoint device. Only +that represents a physical function can be linked to a EPC device. The directory will also have a *start* field. Once "1" is written to this field, the endpoint device will be ready to From patchwork Tue Dec 31 11:35:29 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Subject: [2/7] dt-bindings: PCI: cadence: Add binding to specify max virtual functions X-Patchwork-Submitter: Kishon Vijay Abraham I X-Patchwork-Id: 182673 Message-Id: <20191231113534.30405-3-kishon@ti.com> To: Kishon Vijay Abraham I , Lorenzo Pieralisi , Bjorn Helgaas , Andrew Murray , Tom Joseph , Rob Herring , Arnd Bergmann , Jingoo Han , Gustavo Pimentel , Shawn Lin , Heiko Stuebner Cc: , , Jonathan Corbet , Greg Kroah-Hartman , , , , Date: Tue, 31 Dec 2019 17:05:29 +0530 From: Kishon Vijay Abraham I List-Id: Add binding to specify maximum number of virtual functions that can be associated with each physical function. Signed-off-by: Kishon Vijay Abraham I --- .../devicetree/bindings/pci/cdns,cdns-pcie-ep.txt | 2 ++ .../devicetree/bindings/pci/ti,j721e-pci-ep.yaml | 8 ++++++++ 2 files changed, 10 insertions(+) -- 2.17.1 diff --git a/Documentation/devicetree/bindings/pci/cdns,cdns-pcie-ep.txt b/Documentation/devicetree/bindings/pci/cdns,cdns-pcie-ep.txt index 4a0475e2ba7e..432578202733 100644 --- a/Documentation/devicetree/bindings/pci/cdns,cdns-pcie-ep.txt +++ b/Documentation/devicetree/bindings/pci/cdns,cdns-pcie-ep.txt @@ -9,6 +9,8 @@ Required properties: Optional properties: - max-functions: Maximum number of functions that can be configured (default 1). +- max-virtual-functions: Maximum number of virtual functions that can be + associated with each physical function. - phys: From PHY bindings: List of Generic PHY phandles. One per lane if more than one in the list. If only one PHY listed it must manage all lanes. - phy-names: List of names to identify the PHY. diff --git a/Documentation/devicetree/bindings/pci/ti,j721e-pci-ep.yaml b/Documentation/devicetree/bindings/pci/ti,j721e-pci-ep.yaml index 4621c62016c7..1d4964ba494f 100644 --- a/Documentation/devicetree/bindings/pci/ti,j721e-pci-ep.yaml +++ b/Documentation/devicetree/bindings/pci/ti,j721e-pci-ep.yaml @@ -61,6 +61,12 @@ properties: minimum: 1 maximum: 6 + max-virtual-functions: + minItems: 1 + maxItems: 6 + description: As defined in + Documentation/devicetree/bindings/pci/cdns,cdns-pcie-ep.txt + dma-coherent: description: Indicates that the PCIe IP block can ensure the coherency @@ -85,6 +91,7 @@ required: - cdns,max-outbound-regions - dma-coherent - max-functions + - max-virtual-functions - phys - phy-names @@ -107,6 +114,7 @@ examples: clock-names = "fck"; cdns,max-outbound-regions = <16>; max-functions = /bits/ 8 <6>; + max-virtual-functions = /bits/ 16 <4 4 4 4 0 0>; dma-coherent; phys = <&serdes0_pcie_link>; phy-names = "pcie_phy"; From patchwork Tue Dec 31 11:35:30 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Subject: [3/7] PCI: endpoint: Add support to add virtual function in endpoint core X-Patchwork-Submitter: Kishon Vijay Abraham I X-Patchwork-Id: 182675 Message-Id: <20191231113534.30405-4-kishon@ti.com> To: Kishon Vijay Abraham I , Lorenzo Pieralisi , Bjorn Helgaas , Andrew Murray , Tom Joseph , Rob Herring , Arnd Bergmann , Jingoo Han , Gustavo Pimentel , Shawn Lin , Heiko Stuebner Cc: , , Jonathan Corbet , Greg Kroah-Hartman , , , , Date: Tue, 31 Dec 2019 17:05:30 +0530 From: Kishon Vijay Abraham I List-Id: Add support to add virtual function in endpoint core. The virtual function can only be associated with a physical function instead of a endpoint controller. Provide APIs to associate a virtual function with a physical function here. Signed-off-by: Kishon Vijay Abraham I --- drivers/pci/endpoint/pci-epc-core.c | 2 +- drivers/pci/endpoint/pci-epf-core.c | 92 ++++++++++++++++++++++++++++- include/linux/pci-epf.h | 16 ++++- 3 files changed, 107 insertions(+), 3 deletions(-) -- 2.17.1 diff --git a/drivers/pci/endpoint/pci-epc-core.c b/drivers/pci/endpoint/pci-epc-core.c index 13c03ccb39ac..fef8a212cbd9 100644 --- a/drivers/pci/endpoint/pci-epc-core.c +++ b/drivers/pci/endpoint/pci-epc-core.c @@ -477,7 +477,7 @@ int pci_epc_add_epf(struct pci_epc *epc, struct pci_epf *epf) u32 func_no; int ret = 0; - if (epf->epc) + if (epf->epc || epf->is_vf) return -EBUSY; if (IS_ERR(epc)) diff --git a/drivers/pci/endpoint/pci-epf-core.c b/drivers/pci/endpoint/pci-epf-core.c index 244e00f48c5c..3e0f94b47901 100644 --- a/drivers/pci/endpoint/pci-epf-core.c +++ b/drivers/pci/endpoint/pci-epf-core.c @@ -30,13 +30,20 @@ static const struct device_type pci_epf_type; */ void pci_epf_unbind(struct pci_epf *epf) { + struct pci_epf *epf_vf; + if (!epf->driver) { dev_WARN(&epf->dev, "epf device not bound to driver\n"); return; } mutex_lock(&epf->lock); - epf->driver->ops->unbind(epf); + list_for_each_entry(epf_vf, &epf->pci_vepf, list) { + if (epf_vf->is_bound) + epf_vf->driver->ops->unbind(epf_vf); + } + if (epf->is_bound) + epf->driver->ops->unbind(epf); mutex_unlock(&epf->lock); module_put(epf->driver->owner); } @@ -51,6 +58,7 @@ EXPORT_SYMBOL_GPL(pci_epf_unbind); */ int pci_epf_bind(struct pci_epf *epf) { + struct pci_epf *epf_vf; int ret; if (!epf->driver) { @@ -62,13 +70,91 @@ int pci_epf_bind(struct pci_epf *epf) return -EAGAIN; mutex_lock(&epf->lock); + list_for_each_entry(epf_vf, &epf->pci_vepf, list) { + epf_vf->func_no = epf->func_no; + epf_vf->epc = epf->epc; + ret = epf_vf->driver->ops->bind(epf_vf); + if (ret) + goto ret; + epf_vf->is_bound = true; + } + ret = epf->driver->ops->bind(epf); + if (ret) + goto ret; + epf->is_bound = true; + + mutex_unlock(&epf->lock); + return 0; + +ret: mutex_unlock(&epf->lock); + pci_epf_unbind(epf); return ret; } EXPORT_SYMBOL_GPL(pci_epf_bind); +/** + * pci_epf_add_vepf() - associate virtual EP function to physical EP function + * @epf_pf: the physical EP function to which the virtual EP function should be + * associated + * @epf_vf: the virtual EP function to be added + * + * A physical endpoint function can be associated with multiple virtual + * endpoint functions. Invoke pci_epf_add_epf() to add a virtual PCI endpoint + * function to a physical PCI endpoint function. + */ +int pci_epf_add_vepf(struct pci_epf *epf_pf, struct pci_epf *epf_vf) +{ + u32 vfunc_no; + + if (IS_ERR_OR_NULL(epf_pf) || IS_ERR_OR_NULL(epf_vf)) + return -EINVAL; + + if (epf_pf->epc || epf_vf->epc || epf_vf->epf_pf) + return -EBUSY; + + mutex_lock(&epf_pf->lock); + vfunc_no = find_first_zero_bit(&epf_pf->vfunction_num_map, + BITS_PER_LONG); + if (vfunc_no >= BITS_PER_LONG) + return -EINVAL; + + set_bit(vfunc_no, &epf_pf->vfunction_num_map); + epf_vf->vfunc_no = vfunc_no; + + epf_vf->epf_pf = epf_pf; + epf_vf->is_vf = true; + + list_add_tail(&epf_vf->list, &epf_pf->pci_vepf); + mutex_unlock(&epf_pf->lock); + + return 0; +} +EXPORT_SYMBOL_GPL(pci_epf_add_vepf); + +/** + * pci_epf_remove_vepf() - remove virtual EP function from physical EP function + * @epf_pf: the physical EP function from which the virtual EP function should + * be removed + * @epf_vf: the virtual EP function to be removed + * + * Invoke to remove a virtual endpoint function from the physcial endpoint + * function. + */ +void pci_epf_remove_vepf(struct pci_epf *epf_pf, struct pci_epf *epf_vf) +{ + if (IS_ERR_OR_NULL(epf_pf) || IS_ERR_OR_NULL(epf_vf)) + return; + + mutex_lock(&epf_pf->lock); + clear_bit(epf_vf->vfunc_no, &epf_pf->vfunction_num_map); + list_del(&epf_vf->list); + mutex_unlock(&epf_pf->lock); +} +EXPORT_SYMBOL_GPL(pci_epf_remove_vepf); + /** * pci_epf_free_space() - free the allocated PCI EPF register space * @addr: the virtual address of the PCI EPF register space @@ -258,6 +344,10 @@ struct pci_epf *pci_epf_create(const char *name) return ERR_PTR(-ENOMEM); } + /* VFs are numbered starting with 1. So set BIT(0) by default */ + epf->vfunction_num_map = 1; + INIT_LIST_HEAD(&epf->pci_vepf); + dev = &epf->dev; device_initialize(dev); dev->bus = &pci_epf_bus_type; diff --git a/include/linux/pci-epf.h b/include/linux/pci-epf.h index efbc08a153ff..9544ce81d8fb 100644 --- a/include/linux/pci-epf.h +++ b/include/linux/pci-epf.h @@ -107,12 +107,18 @@ struct pci_epf_bar { * @header: represents standard configuration header * @bar: represents the BAR of EPF device * @msi_interrupts: number of MSI interrupts required by this function - * @func_no: unique function number within this endpoint device + * @func_no: unique (physical) function number within this endpoint device + * @vfunc_no: unique virtual function number within a physical function * @epc: the EPC device to which this EPF device is bound + * @epf_pf: the physical EPF device to which this virtual EPF device is bound * @driver: the EPF driver to which this EPF device is bound * @list: to add pci_epf as a list of PCI endpoint functions to pci_epc * @nb: notifier block to notify EPF of any EPC events (like linkup) * @lock: mutex to protect pci_epf_ops + * @is_bound: indicates if bind notification to function driver has been invoked + * @is_vf: true - virtual function, false - physical function + * @vfunction_num_map: bitmap to manage virtual function number + * @pci_vepf: list of virtual endpoint functions associated with this function */ struct pci_epf { struct device dev; @@ -122,13 +128,19 @@ struct pci_epf { u8 msi_interrupts; u16 msix_interrupts; u8 func_no; + u8 vfunc_no; struct pci_epc *epc; + struct pci_epf *epf_pf; struct pci_epf_driver *driver; struct list_head list; struct notifier_block nb; /* mutex to protect against concurrent access of pci_epf_ops */ struct mutex lock; + unsigned int is_bound; + unsigned int is_vf; + unsigned long vfunction_num_map; + struct list_head pci_vepf; }; /** @@ -171,4 +183,6 @@ void *pci_epf_alloc_space(struct pci_epf *epf, size_t size, enum pci_barno bar, void pci_epf_free_space(struct pci_epf *epf, void *addr, enum pci_barno bar); int pci_epf_bind(struct pci_epf *epf); void pci_epf_unbind(struct pci_epf *epf); +int pci_epf_add_vepf(struct pci_epf *epf_pf, struct pci_epf *epf_vf); +void pci_epf_remove_vepf(struct pci_epf *epf_pf, struct pci_epf *epf_vf); #endif /* __LINUX_PCI_EPF_H */ From patchwork Tue Dec 31 11:35:31 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Subject: [4/7] PCI: endpoint: Add support to link a physical function to a virtual function X-Patchwork-Submitter: Kishon Vijay Abraham I X-Patchwork-Id: 182676 Message-Id: <20191231113534.30405-5-kishon@ti.com> To: Kishon Vijay Abraham I , Lorenzo Pieralisi , Bjorn Helgaas , Andrew Murray , Tom Joseph , Rob Herring , Arnd Bergmann , Jingoo Han , Gustavo Pimentel , Shawn Lin , Heiko Stuebner Cc: , , Jonathan Corbet , Greg Kroah-Hartman , , , , Date: Tue, 31 Dec 2019 17:05:31 +0530 From: Kishon Vijay Abraham I List-Id: While the physical function has to be linked to endpoint controller, the virtual function has to be linked to a physical function. Add support to link a physical function to a virtual function in pci-ep-cfs. Signed-off-by: Kishon Vijay Abraham I --- drivers/pci/endpoint/pci-ep-cfs.c | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) -- 2.17.1 diff --git a/drivers/pci/endpoint/pci-ep-cfs.c b/drivers/pci/endpoint/pci-ep-cfs.c index e7e8367eead1..c18ef626ada5 100644 --- a/drivers/pci/endpoint/pci-ep-cfs.c +++ b/drivers/pci/endpoint/pci-ep-cfs.c @@ -350,6 +350,28 @@ static struct configfs_attribute *pci_epf_attrs[] = { NULL, }; +static int pci_epf_vepf_link(struct config_item *epf_pf_item, + struct config_item *epf_vf_item) +{ + struct pci_epf_group *epf_vf_group = to_pci_epf_group(epf_vf_item); + struct pci_epf_group *epf_pf_group = to_pci_epf_group(epf_pf_item); + struct pci_epf *epf_pf = epf_pf_group->epf; + struct pci_epf *epf_vf = epf_vf_group->epf; + + return pci_epf_add_vepf(epf_pf, epf_vf); +} + +static void pci_epf_vepf_unlink(struct config_item *epf_pf_item, + struct config_item *epf_vf_item) +{ + struct pci_epf_group *epf_vf_group = to_pci_epf_group(epf_vf_item); + struct pci_epf_group *epf_pf_group = to_pci_epf_group(epf_pf_item); + struct pci_epf *epf_pf = epf_pf_group->epf; + struct pci_epf *epf_vf = epf_vf_group->epf; + + pci_epf_remove_vepf(epf_pf, epf_vf); +} + static void pci_epf_release(struct config_item *item) { struct pci_epf_group *epf_group = to_pci_epf_group(item); @@ -362,6 +384,8 @@ static void pci_epf_release(struct config_item *item) } static struct configfs_item_operations pci_epf_ops = { + .allow_link = pci_epf_vepf_link, + .drop_link = pci_epf_vepf_unlink, .release = pci_epf_release, }; From patchwork Tue Dec 31 11:35:33 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Subject: [6/7] PCI: cadence: Add support to configure virtual functions X-Patchwork-Submitter: Kishon Vijay Abraham I X-Patchwork-Id: 182678 Message-Id: <20191231113534.30405-7-kishon@ti.com> To: Kishon Vijay Abraham I , Lorenzo Pieralisi , Bjorn Helgaas , Andrew Murray , Tom Joseph , Rob Herring , Arnd Bergmann , Jingoo Han , Gustavo Pimentel , Shawn Lin , Heiko Stuebner Cc: , , Jonathan Corbet , Greg Kroah-Hartman , , , , Date: Tue, 31 Dec 2019 17:05:33 +0530 From: Kishon Vijay Abraham I List-Id: Now that support for SR-IOV is added in PCIe endpoint core, add support to configure virtual functions in the Cadence PCIe EP driver. Signed-off-by: Kishon Vijay Abraham I --- .../pci/controller/cadence/pcie-cadence-ep.c | 201 ++++++++++++++++-- drivers/pci/controller/cadence/pcie-cadence.h | 7 + 2 files changed, 189 insertions(+), 19 deletions(-) -- 2.17.1 diff --git a/drivers/pci/controller/cadence/pcie-cadence-ep.c b/drivers/pci/controller/cadence/pcie-cadence-ep.c index 673224e2e314..78c25635c4e1 100644 --- a/drivers/pci/controller/cadence/pcie-cadence-ep.c +++ b/drivers/pci/controller/cadence/pcie-cadence-ep.c @@ -20,7 +20,18 @@ static int cdns_pcie_ep_write_header(struct pci_epc *epc, u8 fn, u8 vfn, struct pci_epf_header *hdr) { struct cdns_pcie_ep *ep = epc_get_drvdata(epc); + u32 cap = CDNS_PCIE_EP_FUNC_SRIOV_CAP_OFFSET; struct cdns_pcie *pcie = &ep->pcie; + u32 reg; + + if (vfn > 1) { + dev_dbg(&epc->dev, "Only Virtual Function #1 has deviceID\n"); + return 0; + } else if (vfn == 1) { + reg = cap + PCI_SRIOV_VF_DID; + cdns_pcie_ep_fn_writew(pcie, fn, reg, hdr->deviceid); + return 0; + } cdns_pcie_ep_fn_writew(pcie, fn, PCI_DEVICE_ID, hdr->deviceid); cdns_pcie_ep_fn_writeb(pcie, fn, PCI_REVISION_ID, hdr->revid); @@ -51,12 +62,14 @@ static int cdns_pcie_ep_set_bar(struct pci_epc *epc, u8 fn, u8 vfn, struct pci_epf_bar *epf_bar) { struct cdns_pcie_ep *ep = epc_get_drvdata(epc); + u32 cap = CDNS_PCIE_EP_FUNC_SRIOV_CAP_OFFSET; struct cdns_pcie_epf *epf = &ep->epf[fn]; struct cdns_pcie *pcie = &ep->pcie; dma_addr_t bar_phys = epf_bar->phys_addr; enum pci_barno bar = epf_bar->barno; int flags = epf_bar->flags; u32 addr0, addr1, reg, cfg, b, aperture, ctrl; + u32 first_vf_offset, stride; u64 sz; /* BAR size is 2^(aperture + 7) */ @@ -92,19 +105,39 @@ static int cdns_pcie_ep_set_bar(struct pci_epc *epc, u8 fn, u8 vfn, addr0 = lower_32_bits(bar_phys); addr1 = upper_32_bits(bar_phys); + + if (vfn == 1) { + if (bar < BAR_4) { + reg = CDNS_PCIE_LM_EP_VFUNC_BAR_CFG0(fn); + b = bar; + } else { + reg = CDNS_PCIE_LM_EP_VFUNC_BAR_CFG1(fn); + b = bar - BAR_4; + } + } else { + if (bar < BAR_4) { + reg = CDNS_PCIE_LM_EP_FUNC_BAR_CFG0(fn); + b = bar; + } else { + reg = CDNS_PCIE_LM_EP_FUNC_BAR_CFG1(fn); + b = bar - BAR_4; + } + } + + if (vfn > 0) { + first_vf_offset = cdns_pcie_ep_fn_readw(pcie, fn, cap + + PCI_SRIOV_VF_OFFSET); + stride = cdns_pcie_ep_fn_readw(pcie, fn, cap + + PCI_SRIOV_VF_STRIDE); + fn = fn + first_vf_offset + ((vfn - 1) * stride); + epf = &epf->epf[vfn - 1]; + } + cdns_pcie_writel(pcie, CDNS_PCIE_AT_IB_EP_FUNC_BAR_ADDR0(fn, bar), addr0); cdns_pcie_writel(pcie, CDNS_PCIE_AT_IB_EP_FUNC_BAR_ADDR1(fn, bar), addr1); - if (bar < BAR_4) { - reg = CDNS_PCIE_LM_EP_FUNC_BAR_CFG0(fn); - b = bar; - } else { - reg = CDNS_PCIE_LM_EP_FUNC_BAR_CFG1(fn); - b = bar - BAR_4; - } - cfg = cdns_pcie_readl(pcie, reg); cfg &= ~(CDNS_PCIE_LM_EP_FUNC_BAR_CFG_BAR_APERTURE_MASK(b) | CDNS_PCIE_LM_EP_FUNC_BAR_CFG_BAR_CTRL_MASK(b)); @@ -121,17 +154,38 @@ static void cdns_pcie_ep_clear_bar(struct pci_epc *epc, u8 fn, u8 vfn, struct pci_epf_bar *epf_bar) { struct cdns_pcie_ep *ep = epc_get_drvdata(epc); + u32 cap = CDNS_PCIE_EP_FUNC_SRIOV_CAP_OFFSET; struct cdns_pcie_epf *epf = &ep->epf[fn]; struct cdns_pcie *pcie = &ep->pcie; enum pci_barno bar = epf_bar->barno; + u32 first_vf_offset, stride; u32 reg, cfg, b, ctrl; - if (bar < BAR_4) { - reg = CDNS_PCIE_LM_EP_FUNC_BAR_CFG0(fn); - b = bar; + if (vfn == 1) { + if (bar < BAR_4) { + reg = CDNS_PCIE_LM_EP_VFUNC_BAR_CFG0(fn); + b = bar; + } else { + reg = CDNS_PCIE_LM_EP_VFUNC_BAR_CFG1(fn); + b = bar - BAR_4; + } } else { - reg = CDNS_PCIE_LM_EP_FUNC_BAR_CFG1(fn); - b = bar - BAR_4; + if (bar < BAR_4) { + reg = CDNS_PCIE_LM_EP_FUNC_BAR_CFG0(fn); + b = bar; + } else { + reg = CDNS_PCIE_LM_EP_FUNC_BAR_CFG1(fn); + b = bar - BAR_4; + } + } + + if (vfn > 0) { + first_vf_offset = cdns_pcie_ep_fn_readw(pcie, fn, cap + + PCI_SRIOV_VF_OFFSET); + stride = cdns_pcie_ep_fn_readw(pcie, fn, cap + + PCI_SRIOV_VF_STRIDE); + fn = fn + first_vf_offset + ((vfn - 1) * stride); + epf = &epf->epf[vfn - 1]; } ctrl = CDNS_PCIE_LM_BAR_CFG_CTRL_DISABLED; @@ -152,8 +206,18 @@ static int cdns_pcie_ep_map_addr(struct pci_epc *epc, u8 fn, u8 vfn, { struct cdns_pcie_ep *ep = epc_get_drvdata(epc); struct cdns_pcie *pcie = &ep->pcie; + u32 cap = CDNS_PCIE_EP_FUNC_SRIOV_CAP_OFFSET; + u32 first_vf_offset, stride; u32 r; + if (vfn > 0) { + first_vf_offset = cdns_pcie_ep_fn_readw(pcie, fn, cap + + PCI_SRIOV_VF_OFFSET); + stride = cdns_pcie_ep_fn_readw(pcie, fn, cap + + PCI_SRIOV_VF_STRIDE); + fn = fn + first_vf_offset + ((vfn - 1) * stride); + } + r = find_first_zero_bit(&ep->ob_region_map, sizeof(ep->ob_region_map) * BITS_PER_LONG); if (r >= ep->max_regions - 1) { @@ -193,9 +257,19 @@ static int cdns_pcie_ep_set_msi(struct pci_epc *epc, u8 fn, u8 vfn, u8 mmc) { struct cdns_pcie_ep *ep = epc_get_drvdata(epc); struct cdns_pcie *pcie = &ep->pcie; + u32 sriov_cap = CDNS_PCIE_EP_FUNC_SRIOV_CAP_OFFSET; u32 cap = CDNS_PCIE_EP_FUNC_MSI_CAP_OFFSET; + u32 first_vf_offset, stride; u16 flags; + if (vfn > 0) { + first_vf_offset = cdns_pcie_ep_fn_readw(pcie, fn, sriov_cap + + PCI_SRIOV_VF_OFFSET); + stride = cdns_pcie_ep_fn_readw(pcie, fn, sriov_cap + + PCI_SRIOV_VF_STRIDE); + fn = fn + first_vf_offset + ((vfn - 1) * stride); + } + /* * Set the Multiple Message Capable bitfield into the Message Control * register. @@ -213,9 +287,19 @@ static int cdns_pcie_ep_get_msi(struct pci_epc *epc, u8 fn, u8 vfn) { struct cdns_pcie_ep *ep = epc_get_drvdata(epc); struct cdns_pcie *pcie = &ep->pcie; + u32 sriov_cap = CDNS_PCIE_EP_FUNC_SRIOV_CAP_OFFSET; u32 cap = CDNS_PCIE_EP_FUNC_MSI_CAP_OFFSET; + u32 first_vf_offset, stride; u16 flags, mme; + if (vfn > 0) { + first_vf_offset = cdns_pcie_ep_fn_readw(pcie, fn, sriov_cap + + PCI_SRIOV_VF_OFFSET); + stride = cdns_pcie_ep_fn_readw(pcie, fn, sriov_cap + + PCI_SRIOV_VF_STRIDE); + fn = fn + first_vf_offset + ((vfn - 1) * stride); + } + /* Validate that the MSI feature is actually enabled. */ flags = cdns_pcie_ep_fn_readw(pcie, fn, cap + PCI_MSI_FLAGS); if (!(flags & PCI_MSI_FLAGS_ENABLE)) @@ -232,11 +316,21 @@ static int cdns_pcie_ep_get_msi(struct pci_epc *epc, u8 fn, u8 vfn) static int cdns_pcie_ep_get_msix(struct pci_epc *epc, u8 func_no, u8 vfunc_no) { + u32 sriov_cap = CDNS_PCIE_EP_FUNC_SRIOV_CAP_OFFSET; struct cdns_pcie_ep *ep = epc_get_drvdata(epc); struct cdns_pcie *pcie = &ep->pcie; u32 cap = CDNS_PCIE_EP_FUNC_MSIX_CAP_OFFSET; + u32 first_vf_offset, stride; u32 val, reg; + if (vfunc_no > 0) { + first_vf_offset = cdns_pcie_ep_fn_readw(pcie, func_no, sriov_cap + + PCI_SRIOV_VF_OFFSET); + stride = cdns_pcie_ep_fn_readw(pcie, func_no, sriov_cap + + PCI_SRIOV_VF_STRIDE); + func_no = func_no + first_vf_offset + ((vfunc_no - 1) * stride); + } + reg = cap + PCI_MSIX_FLAGS; val = cdns_pcie_ep_fn_readw(pcie, func_no, reg); if (!(val & PCI_MSIX_FLAGS_ENABLE)) @@ -251,11 +345,21 @@ static int cdns_pcie_ep_set_msix(struct pci_epc *epc, u8 fn, u8 vfn, u16 interrupts, enum pci_barno bir, u32 offset) { + u32 sriov_cap = CDNS_PCIE_EP_FUNC_SRIOV_CAP_OFFSET; struct cdns_pcie_ep *ep = epc_get_drvdata(epc); struct cdns_pcie *pcie = &ep->pcie; u32 cap = CDNS_PCIE_EP_FUNC_MSIX_CAP_OFFSET; + u32 first_vf_offset, stride; u32 val, reg; + if (vfn > 0) { + first_vf_offset = cdns_pcie_ep_fn_readw(pcie, fn, sriov_cap + + PCI_SRIOV_VF_OFFSET); + stride = cdns_pcie_ep_fn_readw(pcie, fn, sriov_cap + + PCI_SRIOV_VF_STRIDE); + fn = fn + first_vf_offset + ((vfn - 1) * stride); + } + reg = cap + PCI_MSIX_FLAGS; val = cdns_pcie_ep_fn_readw(pcie, fn, reg); val &= ~PCI_MSIX_FLAGS_QSIZE; @@ -275,8 +379,8 @@ static int cdns_pcie_ep_set_msix(struct pci_epc *epc, u8 fn, u8 vfn, return 0; } -static void cdns_pcie_ep_assert_intx(struct cdns_pcie_ep *ep, u8 fn, - u8 intx, bool is_asserted) +static void cdns_pcie_ep_assert_intx(struct cdns_pcie_ep *ep, u8 fn, u8 intx, + bool is_asserted) { struct cdns_pcie *pcie = &ep->pcie; u32 offset; @@ -337,10 +441,20 @@ static int cdns_pcie_ep_send_msi_irq(struct cdns_pcie_ep *ep, u8 fn, u8 vfn, u8 interrupt_num) { struct cdns_pcie *pcie = &ep->pcie; + u32 sriov_cap = CDNS_PCIE_EP_FUNC_SRIOV_CAP_OFFSET; u32 cap = CDNS_PCIE_EP_FUNC_MSI_CAP_OFFSET; u16 flags, mme, data, data_mask; u8 msi_count; u64 pci_addr, pci_addr_mask = 0xff; + u32 first_vf_offset, stride; + + if (vfn > 0) { + first_vf_offset = cdns_pcie_ep_fn_readw(pcie, fn, sriov_cap + + PCI_SRIOV_VF_OFFSET); + stride = cdns_pcie_ep_fn_readw(pcie, fn, sriov_cap + + PCI_SRIOV_VF_STRIDE); + fn = fn + first_vf_offset + ((vfn - 1) * stride); + } /* Check whether the MSI feature has been enabled by the PCI host. */ flags = cdns_pcie_ep_fn_readw(pcie, fn, cap + PCI_MSI_FLAGS); @@ -381,13 +495,15 @@ static int cdns_pcie_ep_send_msi_irq(struct cdns_pcie_ep *ep, u8 fn, u8 vfn, return 0; } -static int cdns_pcie_ep_send_msix_irq(struct cdns_pcie_ep *ep, u8 fn, +static int cdns_pcie_ep_send_msix_irq(struct cdns_pcie_ep *ep, u8 fn, u8 vfn, u16 interrupt_num) { + u32 sriov_cap = CDNS_PCIE_EP_FUNC_SRIOV_CAP_OFFSET; u32 cap = CDNS_PCIE_EP_FUNC_MSIX_CAP_OFFSET; u32 tbl_offset, msg_data, reg, vec_ctrl; struct cdns_pcie *pcie = &ep->pcie; struct pci_epf_msix_tbl *msix_tbl; + u32 first_vf_offset, stride; struct pci_epf_bar *epf_bar; struct cdns_pcie_epf *epf; u64 pci_addr_mask = 0xff; @@ -395,6 +511,17 @@ static int cdns_pcie_ep_send_msix_irq(struct cdns_pcie_ep *ep, u8 fn, u16 flags; u8 bir; + epf = &ep->epf[fn]; + + if (vfn > 0) { + first_vf_offset = cdns_pcie_ep_fn_readw(pcie, fn, sriov_cap + + PCI_SRIOV_VF_OFFSET); + stride = cdns_pcie_ep_fn_readw(pcie, fn, sriov_cap + + PCI_SRIOV_VF_STRIDE); + fn = fn + first_vf_offset + ((vfn - 1) * stride); + epf = &epf->epf[vfn - 1]; + } + /* Check whether the MSI-X feature has been enabled by the PCI host. */ flags = cdns_pcie_ep_fn_readw(pcie, fn, cap + PCI_MSIX_FLAGS); if (!(flags & PCI_MSIX_FLAGS_ENABLE)) @@ -405,7 +532,6 @@ static int cdns_pcie_ep_send_msix_irq(struct cdns_pcie_ep *ep, u8 fn, bir = tbl_offset & PCI_MSIX_TABLE_BIR; tbl_offset &= PCI_MSIX_TABLE_OFFSET; - epf = &ep->epf[fn]; epf_bar = epf->epf_bar[bir]; msix_tbl = epf_bar->addr; msix_tbl = (struct pci_epf_msix_tbl *)((char *)msix_tbl + tbl_offset); @@ -436,16 +562,22 @@ static int cdns_pcie_ep_raise_irq(struct pci_epc *epc, u8 fn, u8 vfn, u16 interrupt_num) { struct cdns_pcie_ep *ep = epc_get_drvdata(epc); + struct cdns_pcie *pcie = &ep->pcie; + struct device *dev = pcie->dev; switch (type) { case PCI_EPC_IRQ_LEGACY: + if (vfn > 0) { + dev_err(dev, "Cannot raise legacy interrupts for VF\n"); + return -EINVAL; + } return cdns_pcie_ep_send_legacy_irq(ep, fn, 0); case PCI_EPC_IRQ_MSI: return cdns_pcie_ep_send_msi_irq(ep, fn, vfn, interrupt_num); case PCI_EPC_IRQ_MSIX: - return cdns_pcie_ep_send_msix_irq(ep, fn, interrupt_num); + return cdns_pcie_ep_send_msix_irq(ep, fn, vfn, interrupt_num); default: break; @@ -481,6 +613,13 @@ static int cdns_pcie_ep_start(struct pci_epc *epc) return 0; } +static const struct pci_epc_features cdns_pcie_epc_vf_features = { + .linkup_notifier = false, + .msi_capable = true, + .msix_capable = true, + .align = 65536, +}; + static const struct pci_epc_features cdns_pcie_epc_features = { .linkup_notifier = false, .msi_capable = true, @@ -490,7 +629,10 @@ static const struct pci_epc_features cdns_pcie_epc_features = { static const struct pci_epc_features* cdns_pcie_ep_get_features(struct pci_epc *epc, u8 func_no, u8 vfunc_no) { - return &cdns_pcie_epc_features; + if (!vfunc_no) + return &cdns_pcie_epc_features; + + return &cdns_pcie_epc_vf_features; } static const struct pci_epc_ops cdns_pcie_epc_ops = { @@ -515,9 +657,11 @@ int cdns_pcie_ep_setup(struct cdns_pcie_ep *ep) struct platform_device *pdev = to_platform_device(dev); struct device_node *np = dev->of_node; struct cdns_pcie *pcie = &ep->pcie; + struct cdns_pcie_epf *epf; struct resource *res; struct pci_epc *epc; int ret; + int i; pcie->is_rc = false; @@ -566,6 +710,25 @@ int cdns_pcie_ep_setup(struct cdns_pcie_ep *ep) if (!ep->epf) return -ENOMEM; + epc->max_vfs = devm_kcalloc(dev, epc->max_functions, + sizeof(*epc->max_vfs), GFP_KERNEL); + if (!epc->max_vfs) + return -ENOMEM; + + ret = of_property_read_u16_array(np, "max-virtual-functions", + epc->max_vfs, epc->max_functions); + if (ret == 0) { + for (i = 0; i < epc->max_functions; i++) { + epf = &ep->epf[i]; + if (epc->max_vfs[i] == 0) + continue; + epf->epf = devm_kcalloc(dev, epc->max_vfs[i], + sizeof(*ep->epf), GFP_KERNEL); + if (!epf->epf) + return -ENOMEM; + } + } + ret = pci_epc_mem_init(epc, pcie->mem_res->start, resource_size(pcie->mem_res)); if (ret < 0) { diff --git a/drivers/pci/controller/cadence/pcie-cadence.h b/drivers/pci/controller/cadence/pcie-cadence.h index 2f688b9d205b..285d3c5a715b 100644 --- a/drivers/pci/controller/cadence/pcie-cadence.h +++ b/drivers/pci/controller/cadence/pcie-cadence.h @@ -50,6 +50,10 @@ (CDNS_PCIE_LM_BASE + 0x0240 + (fn) * 0x0008) #define CDNS_PCIE_LM_EP_FUNC_BAR_CFG1(fn) \ (CDNS_PCIE_LM_BASE + 0x0244 + (fn) * 0x0008) +#define CDNS_PCIE_LM_EP_VFUNC_BAR_CFG0(fn) \ + (CDNS_PCIE_LM_BASE + 0x0280 + (fn) * 0x0008) +#define CDNS_PCIE_LM_EP_VFUNC_BAR_CFG1(fn) \ + (CDNS_PCIE_LM_BASE + 0x0284 + (fn) * 0x0008) #define CDNS_PCIE_LM_EP_FUNC_BAR_CFG_BAR_APERTURE_MASK(b) \ (GENMASK(4, 0) << ((b) * 8)) #define CDNS_PCIE_LM_EP_FUNC_BAR_CFG_BAR_APERTURE(b, a) \ @@ -100,6 +104,7 @@ #define CDNS_PCIE_EP_FUNC_MSI_CAP_OFFSET 0x90 #define CDNS_PCIE_EP_FUNC_MSIX_CAP_OFFSET 0xb0 +#define CDNS_PCIE_EP_FUNC_SRIOV_CAP_OFFSET 0x200 /* * Root Port Registers (PCI configuration space for the root port function) @@ -285,9 +290,11 @@ struct cdns_pcie_rc { /** * struct cdns_pcie_epf - Structure to hold info about endpoint function + * @epf: Info about virtual functions attached to the physical function * @epf_bar: reference to the pci_epf_bar for the six Base Address Registers */ struct cdns_pcie_epf { + struct cdns_pcie_epf *epf; struct pci_epf_bar *epf_bar[PCI_STD_NUM_BARS]; }; From patchwork Tue Dec 31 11:35:34 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Subject: [7/7] misc: pci_endpoint_test: Populate sriov_configure ops to configure SR-IOV device X-Patchwork-Submitter: Kishon Vijay Abraham I X-Patchwork-Id: 182679 Message-Id: <20191231113534.30405-8-kishon@ti.com> To: Kishon Vijay Abraham I , Lorenzo Pieralisi , Bjorn Helgaas , Andrew Murray , Tom Joseph , Rob Herring , Arnd Bergmann , Jingoo Han , Gustavo Pimentel , Shawn Lin , Heiko Stuebner Cc: , , Jonathan Corbet , Greg Kroah-Hartman , , , , Date: Tue, 31 Dec 2019 17:05:34 +0530 From: Kishon Vijay Abraham I List-Id: Populate sriov_configure ops with pci_sriov_configure_simple to configure SR-IOV device. Signed-off-by: Kishon Vijay Abraham I --- drivers/misc/pci_endpoint_test.c | 1 + 1 file changed, 1 insertion(+) -- 2.17.1 diff --git a/drivers/misc/pci_endpoint_test.c b/drivers/misc/pci_endpoint_test.c index b2458988939e..0f448c33f45c 100644 --- a/drivers/misc/pci_endpoint_test.c +++ b/drivers/misc/pci_endpoint_test.c @@ -840,6 +840,7 @@ static struct pci_driver pci_endpoint_test_driver = { .id_table = pci_endpoint_test_tbl, .probe = pci_endpoint_test_probe, .remove = pci_endpoint_test_remove, + .sriov_configure = pci_sriov_configure_simple, }; module_pci_driver(pci_endpoint_test_driver);