diff mbox series

[20/20] bus/fslmc: add function to map any addr via VFIO

Message ID 20181227062233.30781-21-hemant.agrawal@nxp.com
State New
Headers show
Series NXP DPAA2 fixes and enhancements | expand

Commit Message

Hemant Agrawal Dec. 27, 2018, 6:23 a.m. UTC
From: Pankaj Chauhan <pankaj.chauhan@nxp.com>


This is required to map any accelerator memory
and PCI address to VFIO using QDMA.

Signed-off-by: Minghuan Lian <Minghuan.Lian@nxp.com>

Signed-off-by: Pankaj Chauhan <pankaj.chauhan@nxp.com>

---
 drivers/bus/fslmc/fslmc_vfio.c              | 42 +++++++++++++++++++++
 drivers/bus/fslmc/fslmc_vfio.h              |  1 +
 drivers/bus/fslmc/rte_bus_fslmc_version.map |  7 ++++
 3 files changed, 50 insertions(+)

-- 
2.17.1

Comments

Ferruh Yigit Jan. 8, 2019, 2:10 p.m. UTC | #1
On 12/27/2018 6:23 AM, Hemant Agrawal wrote:
> From: Pankaj Chauhan <pankaj.chauhan@nxp.com>

> 

> This is required to map any accelerator memory

> and PCI address to VFIO using QDMA.

> 

> Signed-off-by: Minghuan Lian <Minghuan.Lian@nxp.com>

> Signed-off-by: Pankaj Chauhan <pankaj.chauhan@nxp.com>


This requires either sign-off or ack from 'bus/fslmc' maintainers, which are:
	M: Hemant Agrawal <hemant.agrawal@nxp.com>



	M: Shreyansh Jain <shreyansh.jain@nxp.com>

I think Hemant sending the patchset implies that this has been reviewed by him
but this information get lost in git, so better to explicitly provide review/ack
tags whenever appropriate.

<...>

> +	printf("PCIe vfio map 0x%llx:0x%llx, size 0x%llx\n", dma_map.vaddr,

> +		dma_map.iova, dma_map.size);


This is causing build error [1], but why at first place using 'printf()' instead
of logging macros?

[1]
.../drivers/bus/fslmc/fslmc_vfio.c: In function ‘rte_fslmc_vfio_mem_dmamap’:
.../drivers/bus/fslmc/fslmc_vfio.c:376:29: error: format ‘%llx’ expects argument
of type ‘long long unsigned int’, but argument 2 has type ‘__u64’ {aka ‘long
unsigned int’} [-Werror=format=]
  printf("PCIe vfio map 0x%llx:0x%llx, size 0x%llx\n", dma_map.vaddr,
                          ~~~^                         ~~~~~~~~~~~~~
                          %lx

<...>

> +DPDK_19.02 {

> +	global:

> +

> +	rte_fslmc_vfio_mem_dmamap;


Is this need to be an API? Who is the consumer of this API, I don't see anyone
calls this function?
Shreyansh Jain Jan. 10, 2019, 9:58 a.m. UTC | #2
Hello Ferruh,

Replying on behalf of Hemant...

On 08/01/19 7:40 PM, Ferruh Yigit wrote:
> On 12/27/2018 6:23 AM, Hemant Agrawal wrote:

>> From: Pankaj Chauhan <pankaj.chauhan@nxp.com>

>>

>> This is required to map any accelerator memory

>> and PCI address to VFIO using QDMA.

>>

>> Signed-off-by: Minghuan Lian <Minghuan.Lian@nxp.com>

>> Signed-off-by: Pankaj Chauhan <pankaj.chauhan@nxp.com>

> 

> This requires either sign-off or ack from 'bus/fslmc' maintainers, which are:

> 	M: Hemant Agrawal <hemant.agrawal@nxp.com>

> 

> 

> 

> 	M: Shreyansh Jain <shreyansh.jain@nxp.com>

> 

> I think Hemant sending the patchset implies that this has been reviewed by him

> but this information get lost in git, so better to explicitly provide review/ack

> tags whenever appropriate.


What I will do is, re-spin the complete series (with error below 
fixed...) and send again. All the patches are on FSLMC/DPAA2 drivers 
only so probably Hemant's or my Acks would work directly.

> 

> <...>

> 

>> +	printf("PCIe vfio map 0x%llx:0x%llx, size 0x%llx\n", dma_map.vaddr,

>> +		dma_map.iova, dma_map.size);

> 

> This is causing build error [1], but why at first place using 'printf()' instead

> of logging macros?


I will fix this.

> 

> [1]

> .../drivers/bus/fslmc/fslmc_vfio.c: In function ‘rte_fslmc_vfio_mem_dmamap’:

> .../drivers/bus/fslmc/fslmc_vfio.c:376:29: error: format ‘%llx’ expects argument

> of type ‘long long unsigned int’, but argument 2 has type ‘__u64’ {aka ‘long

> unsigned int’} [-Werror=format=]

>    printf("PCIe vfio map 0x%llx:0x%llx, size 0x%llx\n", dma_map.vaddr,

>                            ~~~^                         ~~~~~~~~~~~~~

>                            %lx

> 

> <...>

> 

>> +DPDK_19.02 {

>> +	global:

>> +

>> +	rte_fslmc_vfio_mem_dmamap;

> 

> Is this need to be an API? Who is the consumer of this API, I don't see anyone

> calls this function?

> 


This API (internal to FSLMC) was added for one of NXP's internal 
software stack over DPDK. As this is an internal API, I don't think it 
would pollute the outer namespace - isn't it? And, I think its consumers 
won't necessarily be within DPDK stack.

Or, if this is conflicting case, I will remove this patch (it is 
independent) and send again. Let me know your reservation.

-
Shreyansh
Ferruh Yigit Jan. 11, 2019, 11:58 a.m. UTC | #3
On 1/10/2019 9:58 AM, Shreyansh Jain wrote:
> Hello Ferruh,

> 

> Replying on behalf of Hemant...

> 

> On 08/01/19 7:40 PM, Ferruh Yigit wrote:

>> On 12/27/2018 6:23 AM, Hemant Agrawal wrote:

>>> From: Pankaj Chauhan <pankaj.chauhan@nxp.com>

>>>

>>> This is required to map any accelerator memory

>>> and PCI address to VFIO using QDMA.

>>>

>>> Signed-off-by: Minghuan Lian <Minghuan.Lian@nxp.com>

>>> Signed-off-by: Pankaj Chauhan <pankaj.chauhan@nxp.com>

>>

>> This requires either sign-off or ack from 'bus/fslmc' maintainers, which are:

>> 	M: Hemant Agrawal <hemant.agrawal@nxp.com>

>>

>>

>>

>> 	M: Shreyansh Jain <shreyansh.jain@nxp.com>

>>

>> I think Hemant sending the patchset implies that this has been reviewed by him

>> but this information get lost in git, so better to explicitly provide review/ack

>> tags whenever appropriate.

> 

> What I will do is, re-spin the complete series (with error below 

> fixed...) and send again. All the patches are on FSLMC/DPAA2 drivers 

> only so probably Hemant's or my Acks would work directly.

> 

>>

>> <...>

>>

>>> +	printf("PCIe vfio map 0x%llx:0x%llx, size 0x%llx\n", dma_map.vaddr,

>>> +		dma_map.iova, dma_map.size);

>>

>> This is causing build error [1], but why at first place using 'printf()' instead

>> of logging macros?

> 

> I will fix this.

> 

>>

>> [1]

>> .../drivers/bus/fslmc/fslmc_vfio.c: In function ‘rte_fslmc_vfio_mem_dmamap’:

>> .../drivers/bus/fslmc/fslmc_vfio.c:376:29: error: format ‘%llx’ expects argument

>> of type ‘long long unsigned int’, but argument 2 has type ‘__u64’ {aka ‘long

>> unsigned int’} [-Werror=format=]

>>    printf("PCIe vfio map 0x%llx:0x%llx, size 0x%llx\n", dma_map.vaddr,

>>                            ~~~^                         ~~~~~~~~~~~~~

>>                            %lx

>>

>> <...>

>>

>>> +DPDK_19.02 {

>>> +	global:

>>> +

>>> +	rte_fslmc_vfio_mem_dmamap;

>>

>> Is this need to be an API? Who is the consumer of this API, I don't see anyone

>> calls this function?

>>

> 

> This API (internal to FSLMC) was added for one of NXP's internal 

> software stack over DPDK. As this is an internal API, I don't think it 

> would pollute the outer namespace - isn't it? And, I think its consumers 

> won't necessarily be within DPDK stack.

> 

> Or, if this is conflicting case, I will remove this patch (it is 

> independent) and send again. Let me know your reservation.


API consumers doesn't have to be in DPDK but than it is not an internal API.

And for API we recently decided to have an implementation for new APIs, it can
be unit test, sample application, or app like testpmd, so that we can detect
when it is broken.

Perhaps it can be good idea to separate patch, as you suggested, to not block
the rest of the patchset.
Shreyansh Jain Jan. 11, 2019, 12:16 p.m. UTC | #4
On 11/01/19 5:28 PM, Ferruh Yigit wrote:
> On 1/10/2019 9:58 AM, Shreyansh Jain wrote:

>> Hello Ferruh,

>>

>> Replying on behalf of Hemant...

>>

>> On 08/01/19 7:40 PM, Ferruh Yigit wrote:

>>> On 12/27/2018 6:23 AM, Hemant Agrawal wrote:

>>>> From: Pankaj Chauhan <pankaj.chauhan@nxp.com>


[...]

>>>> +DPDK_19.02 {

>>>> +	global:

>>>> +

>>>> +	rte_fslmc_vfio_mem_dmamap;

>>>

>>> Is this need to be an API? Who is the consumer of this API, I don't see anyone

>>> calls this function?

>>>

>>

>> This API (internal to FSLMC) was added for one of NXP's internal

>> software stack over DPDK. As this is an internal API, I don't think it

>> would pollute the outer namespace - isn't it? And, I think its consumers

>> won't necessarily be within DPDK stack.

>>

>> Or, if this is conflicting case, I will remove this patch (it is

>> independent) and send again. Let me know your reservation.

> 

> API consumers doesn't have to be in DPDK but than it is not an internal API.

> 

> And for API we recently decided to have an implementation for new APIs, it can

> be unit test, sample application, or app like testpmd, so that we can detect

> when it is broken.

> 

> Perhaps it can be good idea to separate patch, as you suggested, to not block

> the rest of the patchset.

> 


Oops, I think I sent my series just about the time you wrote this.
I will send it again (v3) without the last patch. I am OK with that API 
not making it in right now.
diff mbox series

Patch

diff --git a/drivers/bus/fslmc/fslmc_vfio.c b/drivers/bus/fslmc/fslmc_vfio.c
index 1aae56fa9..9befc8087 100644
--- a/drivers/bus/fslmc/fslmc_vfio.c
+++ b/drivers/bus/fslmc/fslmc_vfio.c
@@ -50,6 +50,7 @@  static struct fslmc_vfio_group vfio_group;
 static struct fslmc_vfio_container vfio_container;
 static int container_device_fd;
 static char *g_container;
+static int fslmc_iommu_type;
 static uint32_t *msi_intr_vaddr;
 void *(*rte_mcp_ptr_list);
 
@@ -90,6 +91,9 @@  fslmc_get_container_group(int *groupid)
 		}
 	}
 
+	fslmc_iommu_type = (rte_vfio_noiommu_is_enabled() == 1) ?
+		RTE_VFIO_NOIOMMU : VFIO_TYPE1_IOMMU;
+
 	/* get group number */
 	ret = rte_vfio_get_group_num(SYSFS_FSL_MC_DEVICES,
 				     g_container, groupid);
@@ -344,6 +348,44 @@  fslmc_dmamap_seg(const struct rte_memseg_list *msl __rte_unused,
 	return ret;
 }
 
+int rte_fslmc_vfio_mem_dmamap(uint64_t vaddr, uint64_t iova, uint64_t size)
+{
+	int ret;
+	struct fslmc_vfio_group *group;
+	struct vfio_iommu_type1_dma_map dma_map = {
+		.argsz = sizeof(struct vfio_iommu_type1_dma_map),
+		.flags = VFIO_DMA_MAP_FLAG_READ | VFIO_DMA_MAP_FLAG_WRITE,
+	};
+
+	if (fslmc_iommu_type == RTE_VFIO_NOIOMMU) {
+		DPAA2_BUS_DEBUG("Running in NOIOMMU mode");
+		return 0;
+	}
+
+	/* SET DMA MAP for IOMMU */
+	group = &vfio_group;
+	if (!group->container) {
+		DPAA2_BUS_ERR("Container is not connected");
+		return -1;
+	}
+
+	dma_map.size = size;
+	dma_map.vaddr = vaddr;
+	dma_map.iova = iova;
+
+	printf("PCIe vfio map 0x%llx:0x%llx, size 0x%llx\n", dma_map.vaddr,
+		dma_map.iova, dma_map.size);
+	ret = ioctl(group->container->fd, VFIO_IOMMU_MAP_DMA,
+		    &dma_map);
+	if (ret) {
+		printf("Unable to map DMA address (errno = %d)\n",
+			errno);
+		return ret;
+	}
+
+	return 0;
+}
+
 int rte_fslmc_vfio_dmamap(void)
 {
 	int i = 0, ret;
diff --git a/drivers/bus/fslmc/fslmc_vfio.h b/drivers/bus/fslmc/fslmc_vfio.h
index 9e2c4feef..4e750d623 100644
--- a/drivers/bus/fslmc/fslmc_vfio.h
+++ b/drivers/bus/fslmc/fslmc_vfio.h
@@ -50,5 +50,6 @@  int fslmc_vfio_process_group(void);
 char *fslmc_get_container(void);
 int fslmc_get_container_group(int *gropuid);
 int rte_fslmc_vfio_dmamap(void);
+int rte_fslmc_vfio_mem_dmamap(uint64_t vaddr, uint64_t iova, uint64_t size);
 
 #endif /* _FSLMC_VFIO_H_ */
diff --git a/drivers/bus/fslmc/rte_bus_fslmc_version.map b/drivers/bus/fslmc/rte_bus_fslmc_version.map
index dcc4e082e..c4192d978 100644
--- a/drivers/bus/fslmc/rte_bus_fslmc_version.map
+++ b/drivers/bus/fslmc/rte_bus_fslmc_version.map
@@ -129,3 +129,10 @@  DPDK_18.11 {
 	dpci_set_opr;
 
 } DPDK_18.05;
+
+DPDK_19.02 {
+	global:
+
+	rte_fslmc_vfio_mem_dmamap;
+
+} DPDK_18.11;