From patchwork Tue Aug 27 07:07:27 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sachin Saxena X-Patchwork-Id: 172268 Delivered-To: patch@linaro.org Received: by 2002:a92:d204:0:0:0:0:0 with SMTP id y4csp5400062ily; Tue, 27 Aug 2019 00:13:41 -0700 (PDT) X-Google-Smtp-Source: APXvYqwfdlO/LjJYgbNjAxX/Yl2gpNd/SNl9kWmIxY0HBIWy+wQiVlKsAO38T9KOw363GpbqhwDf X-Received: by 2002:a50:ba81:: with SMTP id x1mr22263255ede.257.1566890021030; Tue, 27 Aug 2019 00:13:41 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1566890021; cv=none; d=google.com; s=arc-20160816; b=KDl9ObG7FmamulYHWaCaw1fDQ7AEqi7dDMAIu1QtrqpARbLr1Gl6zNlkdtvxwG17Pp 7tZebpPcDdbI/u+ESDnwoxJOKCMNNtLZdfmABGgc2VVLvPHHNMrTCOhBKMsUAI5iFcEc bU077O6ZEdB96ooNpVTxXaYIEAjLnaylFDTix9C8Yfj2So1NjVoX1OqbKtmS0hZXWx/5 TObzfVKGAEhevt60DUtHClDVdySZi4Ke9XjRIKPMv66jCa+/88BeeYTc8tJdL9xrLIfw f2EYZ240sLafbjfq/PcpkwXqFzAoktQqa8FqulQMaNk9CSfg55dehI7thNGE0eF/HRMs w0XQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:subject:references:in-reply-to :message-id:date:cc:to:from; bh=IVGEE/uHG0aBqpwR/fdFEGNJGzPcQOSUx1yVRHtdRT0=; b=0pXzXqlEb0fxiltvlb6Q042XhiguozzgXbgACs+bsWQ1EJ7Yq51NZJgDKvv5IVNDNi lHFyCTG6Tqv/JlkBg3W5kB1hnuuV7+nKJj5m0LVU76KWnIBGc/d/Y5xkTVs0qqhK9875 m9qp1vYTgzuUC5XTdNIUkmeK/Pg1B3/Yxzas7Cbzh3Lw+F9nJJjlfTtz/CzvCdG3Ux+4 wCzmHpYxq9k2j/czovuV3XFhN6Xg1aIIZ7OxqsoYKrTOO99+cCAON44e2W4oAIkceR2X NBk9KGr+oqbPm9kjJpX0ggfAGSf+AJ56ikfsdyqQMRy+2q9KHDzjhq3U1UoZDyyz+LUK rbpQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of dev-bounces@dpdk.org designates 92.243.14.124 as permitted sender) smtp.mailfrom=dev-bounces@dpdk.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=nxp.com Return-Path: Received: from dpdk.org (dpdk.org. [92.243.14.124]) by mx.google.com with ESMTP id l23si7360277edv.296.2019.08.27.00.13.40; Tue, 27 Aug 2019 00:13:41 -0700 (PDT) Received-SPF: pass (google.com: domain of dev-bounces@dpdk.org designates 92.243.14.124 as permitted sender) client-ip=92.243.14.124; Authentication-Results: mx.google.com; spf=pass (google.com: domain of dev-bounces@dpdk.org designates 92.243.14.124 as permitted sender) smtp.mailfrom=dev-bounces@dpdk.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=nxp.com Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id B24E91C1DB; Tue, 27 Aug 2019 09:10:37 +0200 (CEST) Received: from inva020.nxp.com (inva020.nxp.com [92.121.34.13]) by dpdk.org (Postfix) with ESMTP id 2598F1C034 for ; Tue, 27 Aug 2019 09:09:34 +0200 (CEST) Received: from inva020.nxp.com (localhost [127.0.0.1]) by inva020.eu-rdc02.nxp.com (Postfix) with ESMTP id 0B4A21A020A; Tue, 27 Aug 2019 09:09:34 +0200 (CEST) Received: from invc005.ap-rdc01.nxp.com (invc005.ap-rdc01.nxp.com [165.114.16.14]) by inva020.eu-rdc02.nxp.com (Postfix) with ESMTP id C688F1A021C; Tue, 27 Aug 2019 09:09:31 +0200 (CEST) Received: from bf-netperf1.ap.freescale.net (bf-netperf1.ap.freescale.net [10.232.133.63]) by invc005.ap-rdc01.nxp.com (Postfix) with ESMTP id B427B402EB; Tue, 27 Aug 2019 15:09:28 +0800 (SGT) From: Sachin Saxena To: dev@dpdk.org Cc: thomas@monjalon.net, Hemant Agrawal Date: Tue, 27 Aug 2019 12:37:27 +0530 Message-Id: <20190827070730.11206-28-sachin.saxena@nxp.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190827070730.11206-1-sachin.saxena@nxp.com> References: <20190827070730.11206-1-sachin.saxena@nxp.com> X-Virus-Scanned: ClamAV using ClamSMTP Subject: [dpdk-dev] [PATCH v1 27/30] bus/fslmc: support multi vfio group X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" From: Hemant Agrawal DPAA2 support VFIO device passthrough in VM. However in this case, each device is associated with different vfio group. This code required different container id for each group. On using the same container fd the second time, ioctl calls are returning error. Signed-off-by: Hemant Agrawal --- drivers/bus/fslmc/fslmc_bus.c | 7 +- drivers/bus/fslmc/fslmc_vfio.c | 160 ++++++++++++++++++++++++++------- drivers/bus/fslmc/fslmc_vfio.h | 3 + 3 files changed, 133 insertions(+), 37 deletions(-) -- 2.17.1 diff --git a/drivers/bus/fslmc/fslmc_bus.c b/drivers/bus/fslmc/fslmc_bus.c index 9226d5b62..eff15f25d 100644 --- a/drivers/bus/fslmc/fslmc_bus.c +++ b/drivers/bus/fslmc/fslmc_bus.c @@ -1,6 +1,6 @@ /* SPDX-License-Identifier: BSD-3-Clause * - * Copyright 2016,2018 NXP + * Copyright 2016,2018-2019 NXP * */ @@ -325,8 +325,7 @@ rte_fslmc_scan(void) goto scan_fail; /* Scan devices on the group */ - snprintf(fslmc_dirpath, sizeof(fslmc_dirpath), "%s/%d/devices", - VFIO_IOMMU_GROUP_PATH, groupid); + sprintf(fslmc_dirpath, "%s/%s", SYSFS_FSL_MC_DEVICES, fslmc_container); dir = opendir(fslmc_dirpath); if (!dir) { DPAA2_BUS_ERR("Unable to open VFIO group directory"); @@ -334,7 +333,7 @@ rte_fslmc_scan(void) } while ((entry = readdir(dir)) != NULL) { - if (entry->d_name[0] == '.' || entry->d_type != DT_LNK) + if (entry->d_name[0] == '.' || entry->d_type != DT_DIR) continue; ret = scan_one_fslmc_device(entry->d_name); diff --git a/drivers/bus/fslmc/fslmc_vfio.c b/drivers/bus/fslmc/fslmc_vfio.c index 60c51d80e..970969d2b 100644 --- a/drivers/bus/fslmc/fslmc_vfio.c +++ b/drivers/bus/fslmc/fslmc_vfio.c @@ -40,16 +40,13 @@ #include "portal/dpaa2_hw_pvt.h" #include "portal/dpaa2_hw_dpio.h" -/** Pathname of FSL-MC devices directory. */ -#define SYSFS_FSL_MC_DEVICES "/sys/bus/fsl-mc/devices" - #define FSLMC_CONTAINER_MAX_LEN 8 /**< Of the format dprc.XX */ /* Number of VFIO containers & groups with in */ static struct fslmc_vfio_group vfio_group; static struct fslmc_vfio_container vfio_container; static int container_device_fd; -static char *fslmc_container; +char *fslmc_container; static int fslmc_iommu_type; static uint32_t *msi_intr_vaddr; void *(*rte_mcp_ptr_list); @@ -435,28 +432,136 @@ int rte_fslmc_vfio_dmamap(void) return 0; } -static int64_t vfio_map_mcp_obj(struct fslmc_vfio_group *group, char *mcp_obj) +static int +fslmc_vfio_setup_device(const char *sysfs_base, const char *dev_addr, + int *vfio_dev_fd, struct vfio_device_info *device_info) +{ + struct vfio_group_status group_status = { + .argsz = sizeof(group_status) + }; + int vfio_group_fd, vfio_container_fd, iommu_group_no, ret; + + /* get group number */ + ret = rte_vfio_get_group_num(sysfs_base, dev_addr, &iommu_group_no); + if (ret < 0) + return -1; + + /* get the actual group fd */ + vfio_group_fd = rte_vfio_get_group_fd(iommu_group_no); + if (vfio_group_fd < 0) + return -1; + + /* if group_fd == 0, that means the device isn't managed by VFIO */ + if (vfio_group_fd == 0) { + RTE_LOG(WARNING, EAL, " %s not managed by VFIO driver, skipping\n", + dev_addr); + return 1; + } + + /* Opens main vfio file descriptor which represents the "container" */ + vfio_container_fd = rte_vfio_get_container_fd(); + if (vfio_container_fd < 0) { + DPAA2_BUS_ERR("Failed to open VFIO container"); + return -errno; + } + + /* check if the group is viable */ + ret = ioctl(vfio_group_fd, VFIO_GROUP_GET_STATUS, &group_status); + if (ret) { + DPAA2_BUS_ERR(" %s cannot get group status, " + "error %i (%s)\n", dev_addr, + errno, strerror(errno)); + close(vfio_group_fd); + rte_vfio_clear_group(vfio_group_fd); + return -1; + } else if (!(group_status.flags & VFIO_GROUP_FLAGS_VIABLE)) { + DPAA2_BUS_ERR(" %s VFIO group is not viable!\n", dev_addr); + close(vfio_group_fd); + rte_vfio_clear_group(vfio_group_fd); + return -1; + } + /* At this point, we know that this group is viable (meaning, + * all devices are either bound to VFIO or not bound to anything) + */ + + /* check if group does not have a container yet */ + if (!(group_status.flags & VFIO_GROUP_FLAGS_CONTAINER_SET)) { + + /* add group to a container */ + ret = ioctl(vfio_group_fd, VFIO_GROUP_SET_CONTAINER, + &vfio_container_fd); + if (ret) { + DPAA2_BUS_ERR(" %s cannot add VFIO group to container, " + "error %i (%s)\n", dev_addr, + errno, strerror(errno)); + close(vfio_group_fd); + close(vfio_container_fd); + rte_vfio_clear_group(vfio_group_fd); + return -1; + } + + /* + * set an IOMMU type for container + * + */ + if (ioctl(vfio_container_fd, VFIO_CHECK_EXTENSION, + fslmc_iommu_type)) { + ret = ioctl(vfio_container_fd, VFIO_SET_IOMMU, + fslmc_iommu_type); + if (ret) { + DPAA2_BUS_ERR("Failed to setup VFIO iommu"); + close(vfio_group_fd); + close(vfio_container_fd); + return -errno; + } + } else { + DPAA2_BUS_ERR("No supported IOMMU available"); + close(vfio_group_fd); + close(vfio_container_fd); + return -EINVAL; + } + } + + /* get a file descriptor for the device */ + *vfio_dev_fd = ioctl(vfio_group_fd, VFIO_GROUP_GET_DEVICE_FD, dev_addr); + if (*vfio_dev_fd < 0) { + /* if we cannot get a device fd, this implies a problem with + * the VFIO group or the container not having IOMMU configured. + */ + + DPAA2_BUS_WARN("Getting a vfio_dev_fd for %s failed", dev_addr); + close(vfio_group_fd); + close(vfio_container_fd); + rte_vfio_clear_group(vfio_group_fd); + return -1; + } + + /* test and setup the device */ + ret = ioctl(*vfio_dev_fd, VFIO_DEVICE_GET_INFO, device_info); + if (ret) { + DPAA2_BUS_ERR(" %s cannot get device info, error %i (%s)", + dev_addr, errno, strerror(errno)); + close(*vfio_dev_fd); + close(vfio_group_fd); + close(vfio_container_fd); + rte_vfio_clear_group(vfio_group_fd); + return -1; + } + + return 0; +} + +static intptr_t vfio_map_mcp_obj(const char *mcp_obj) { intptr_t v_addr = (intptr_t)MAP_FAILED; int32_t ret, mc_fd; + struct vfio_group_status status = { .argsz = sizeof(status) }; struct vfio_device_info d_info = { .argsz = sizeof(d_info) }; struct vfio_region_info reg_info = { .argsz = sizeof(reg_info) }; - /* getting the mcp object's fd*/ - mc_fd = ioctl(group->fd, VFIO_GROUP_GET_DEVICE_FD, mcp_obj); - if (mc_fd < 0) { - DPAA2_BUS_ERR("Error in VFIO get dev %s fd from group %d", - mcp_obj, group->fd); - return v_addr; - } - - /* getting device info*/ - ret = ioctl(mc_fd, VFIO_DEVICE_GET_INFO, &d_info); - if (ret < 0) { - DPAA2_BUS_ERR("Error in VFIO getting DEVICE_INFO"); - goto MC_FAILURE; - } + fslmc_vfio_setup_device(SYSFS_FSL_MC_DEVICES, mcp_obj, + &mc_fd, &d_info); /* getting device region info*/ ret = ioctl(mc_fd, VFIO_DEVICE_GET_REGION_INFO, ®_info); @@ -589,19 +694,8 @@ fslmc_process_iodevices(struct rte_dpaa2_device *dev) struct vfio_device_info device_info = { .argsz = sizeof(device_info) }; struct rte_dpaa2_object *object = NULL; - dev_fd = ioctl(vfio_group.fd, VFIO_GROUP_GET_DEVICE_FD, - dev->device.name); - if (dev_fd <= 0) { - DPAA2_BUS_ERR("Unable to obtain device FD for device:%s", - dev->device.name); - return -1; - } - - if (ioctl(dev_fd, VFIO_DEVICE_GET_INFO, &device_info)) { - DPAA2_BUS_ERR("Unable to obtain information for device:%s", - dev->device.name); - return -1; - } + fslmc_vfio_setup_device(SYSFS_FSL_MC_DEVICES, dev->device.name, + &dev_fd, &device_info); switch (dev->dev_type) { case DPAA2_ETH: @@ -654,7 +748,7 @@ fslmc_process_mcp(struct rte_dpaa2_device *dev) goto cleanup; } - v_addr = vfio_map_mcp_obj(&vfio_group, dev_name); + v_addr = vfio_map_mcp_obj(dev->device.name); if (v_addr == (intptr_t)MAP_FAILED) { DPAA2_BUS_ERR("Error mapping region (errno = %d)", errno); ret = -1; diff --git a/drivers/bus/fslmc/fslmc_vfio.h b/drivers/bus/fslmc/fslmc_vfio.h index e877255ea..c98812129 100644 --- a/drivers/bus/fslmc/fslmc_vfio.h +++ b/drivers/bus/fslmc/fslmc_vfio.h @@ -10,6 +10,8 @@ #include +/* Pathname of FSL-MC devices directory. */ +#define SYSFS_FSL_MC_DEVICES "/sys/bus/fsl-mc/devices" #define DPAA2_MC_DPNI_DEVID 7 #define DPAA2_MC_DPSECI_DEVID 3 #define DPAA2_MC_DPCON_DEVID 5 @@ -38,6 +40,7 @@ typedef struct fslmc_vfio_container { struct fslmc_vfio_group *group; } fslmc_vfio_container; +extern char *fslmc_container; int rte_dpaa2_intr_enable(struct rte_intr_handle *intr_handle, int index); int rte_dpaa2_intr_disable(struct rte_intr_handle *intr_handle, int index);